* [PATCH v6 1/6] riscv: probes: reject probes inside an LR/SC sequence
2026-11-10 13:20 [PATCH v6 0/6] riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO Vladimir Kondratiev
@ 2026-11-10 13:20 ` Vladimir Kondratiev
2026-11-10 13:20 ` [PATCH v6 2/6] riscv: build kernel atomics from LR/SC when Zaamo is disabled Vladimir Kondratiev
` (4 subsequent siblings)
5 siblings, 0 replies; 8+ messages in thread
From: Vladimir Kondratiev @ 2026-11-10 13:20 UTC (permalink / raw)
To: Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Will Deacon, Peter Zijlstra, Mark Rutland, Gary Guo, Yury Norov,
Rasmus Villemoes, Chao-ying Fu, Conor Dooley, torvalds, olof,
aleksa.paunovic, arikalo, Boqun Feng, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
Puranjay Mohan, Masami Hiramatsu, Oleg Nesterov, Miguel Ojeda,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Shuah Khan, Randy Dunlap,
Boqun Feng
Cc: linux-riscv, linux-kernel, Vladimir.Kondratiev, bpf,
linux-trace-kernel, linux-doc, Jonathan Corbet, Charlie Jenkins,
Nam Cao, rust-for-linux, Vladimir Kondratiev
A kprobe or uprobe replaces an instruction with a breakpoint. The trap
clears the hart's reservation, so if the breakpoint sits on the lr, on
the sc, or on any instruction between them, the sc fails, the loop
branches back to the lr and the probe fires again. The sequence never
completes.
This has always been possible on RISC-V, because cmpxchg and the
conditional atomics are LR/SC loops. Once Zaamo is disabled every
kernel atomic is such a loop, and so is every atomic in userspace built
for the same hart, so the chance of a probe landing in one grows
accordingly. arm64 has the same problem and rejects probes inside an
exclusive-load/store-exclusive region.
Teach the shared instruction decoder to reject lr and sc themselves.
This covers both kprobes and uprobes.
For the instructions between them, use the rules for a constrained
LR/SC sequence: at most 16 instructions between lr and sc, all base
integer instructions other than loads, stores, JALR, FENCE, SYSTEM and
backward jumps or branches, or their compressed forms. Scan forward
from the probed instruction. Reaching an sc through permitted
instructions only means the probe sits inside a sequence; any other
instruction, an lr, the end of the window or the end of the readable
text means it does not. Only a sequence that follows these rules is
guaranteed to make progress, so a probe inside any other one is
already on unsupported ground. Scanning forward rather than tracking
state from the start of the function also keeps an lr that has no sc,
such as the one in __cmpwait() before wrs.nto, from affecting probes
later in the same function.
kprobes scan the kernel text up to the end of the enclosing symbol.
uprobes have only the probed instruction copied, so they read the
text that follows it from the task, which the caller's mmap_lock
allows.
This is independent of the rest of the series. cmpxchg is already an
LR/SC loop today, so the problem exists on every kernel; it only
becomes easier to hit once every atomic is such a loop.
Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@mobileye.com>
---
arch/riscv/include/asm/insn.h | 28 ++++++++++
arch/riscv/kernel/probes/decode-insn.c | 94 ++++++++++++++++++++++++++++++++++
arch/riscv/kernel/probes/decode-insn.h | 7 +++
arch/riscv/kernel/probes/kprobes.c | 15 ++++--
arch/riscv/kernel/probes/uprobes.c | 31 +++++++++++
5 files changed, 172 insertions(+), 3 deletions(-)
diff --git a/arch/riscv/include/asm/insn.h b/arch/riscv/include/asm/insn.h
index c3005573e8c9..f5de81753bec 100644
--- a/arch/riscv/include/asm/insn.h
+++ b/arch/riscv/include/asm/insn.h
@@ -10,11 +10,16 @@
#define RV_INSN_FUNCT3_MASK GENMASK(14, 12)
#define RV_INSN_FUNCT3_OPOFF 12
+#define RV_INSN_FUNCT5_MASK GENMASK(31, 27)
+#define RV_INSN_FUNCT5_OPOFF 27
+#define RV_INSN_FUNCT7_MASK GENMASK(31, 25)
+#define RV_INSN_FUNCT7_OPOFF 25
#define RV_INSN_OPCODE_MASK GENMASK(6, 0)
#define RV_INSN_OPCODE_OPOFF 0
#define RV_INSN_FUNCT12_OPOFF 20
#define RV_ENCODE_FUNCT3(f_) (RVG_FUNCT3_##f_ << RV_INSN_FUNCT3_OPOFF)
+#define RV_ENCODE_FUNCT5(f_) (RVG_FUNCT5_##f_ << RV_INSN_FUNCT5_OPOFF)
#define RV_ENCODE_FUNCT12(f_) (RVG_FUNCT12_##f_ << RV_INSN_FUNCT12_OPOFF)
/* The bit field of immediate value in I-type instruction */
@@ -136,7 +141,13 @@
/* parts of opcode for RVG*/
#define RVG_OPCODE_FENCE 0x0f
+#define RVG_OPCODE_OP_IMM 0x13
#define RVG_OPCODE_AUIPC 0x17
+#define RVG_OPCODE_OP_IMM_32 0x1b
+#define RVG_OPCODE_AMO 0x2f
+#define RVG_OPCODE_OP 0x33
+#define RVG_OPCODE_LUI 0x37
+#define RVG_OPCODE_OP_32 0x3b
#define RVG_OPCODE_BRANCH 0x63
#define RVG_OPCODE_JALR 0x67
#define RVG_OPCODE_JAL 0x6f
@@ -188,6 +199,17 @@
#define RVG_FUNCT12_EBREAK 0x1
#define RVG_FUNCT12_SRET 0x102
+/* parts of funct5 code for A extension */
+#define RVG_FUNCT5_LR 0x02
+#define RVG_FUNCT5_SC 0x03
+
+/*
+ * The AMO funct3 field holds the access width: 010 for .w, 011 for .d.
+ * Matching bits 14:13 alone accepts both and nothing else.
+ */
+#define RVG_AMO_WIDTH_MASK GENMASK(14, 13)
+#define RVG_AMO_WIDTH_WD BIT(13)
+
#define RVG_MATCH_AUIPC (RVG_OPCODE_AUIPC)
#define RVG_MATCH_JALR (RV_ENCODE_FUNCT3(JALR) | RVG_OPCODE_JALR)
#define RVG_MATCH_JAL (RVG_OPCODE_JAL)
@@ -200,6 +222,8 @@
#define RVG_MATCH_BGEU (RV_ENCODE_FUNCT3(BGEU) | RVG_OPCODE_BRANCH)
#define RVG_MATCH_EBREAK (RV_ENCODE_FUNCT12(EBREAK) | RVG_OPCODE_SYSTEM)
#define RVG_MATCH_SRET (RV_ENCODE_FUNCT12(SRET) | RVG_OPCODE_SYSTEM)
+#define RVG_MATCH_LR (RV_ENCODE_FUNCT5(LR) | RVG_AMO_WIDTH_WD | RVG_OPCODE_AMO)
+#define RVG_MATCH_SC (RV_ENCODE_FUNCT5(SC) | RVG_AMO_WIDTH_WD | RVG_OPCODE_AMO)
#define RVC_MATCH_C_BEQZ (RVC_ENCODE_FUNCT3(C_BEQZ) | RVC_OPCODE_C1)
#define RVC_MATCH_C_BNEZ (RVC_ENCODE_FUNCT3(C_BNEZ) | RVC_OPCODE_C1)
#define RVC_MATCH_C_J (RVC_ENCODE_FUNCT3(C_J) | RVC_OPCODE_C1)
@@ -227,6 +251,8 @@
#define RVC_MASK_C_EBREAK 0xffff
#define RVG_MASK_EBREAK 0xffffffff
#define RVG_MASK_SRET 0xffffffff
+#define RVG_MASK_LR (RV_INSN_FUNCT5_MASK | RVG_AMO_WIDTH_MASK | RV_INSN_OPCODE_MASK)
+#define RVG_MASK_SC (RV_INSN_FUNCT5_MASK | RVG_AMO_WIDTH_MASK | RV_INSN_OPCODE_MASK)
#define __INSN_LENGTH_MASK _UL(0x3)
#define __INSN_LENGTH_GE_32 _UL(0x3)
@@ -262,6 +288,8 @@ __RISCV_INSN_FUNCS(c_ebreak, RVC_MASK_C_EBREAK, RVC_MATCH_C_EBREAK)
__RISCV_INSN_FUNCS(ebreak, RVG_MASK_EBREAK, RVG_MATCH_EBREAK)
__RISCV_INSN_FUNCS(sret, RVG_MASK_SRET, RVG_MATCH_SRET)
__RISCV_INSN_FUNCS(fence, RVG_MASK_FENCE, RVG_MATCH_FENCE);
+__RISCV_INSN_FUNCS(lr, RVG_MASK_LR, RVG_MATCH_LR)
+__RISCV_INSN_FUNCS(sc, RVG_MASK_SC, RVG_MATCH_SC)
/* special case to catch _any_ system instruction */
static __always_inline bool riscv_insn_is_system(u32 code)
diff --git a/arch/riscv/kernel/probes/decode-insn.c b/arch/riscv/kernel/probes/decode-insn.c
index 433d9035b888..6a4f55b07d04 100644
--- a/arch/riscv/kernel/probes/decode-insn.c
+++ b/arch/riscv/kernel/probes/decode-insn.c
@@ -4,11 +4,99 @@
#include <linux/kprobes.h>
#include <linux/module.h>
#include <linux/kallsyms.h>
+#include <linux/bug.h>
#include <asm/sections.h>
#include "decode-insn.h"
#include "simulate-insn.h"
+/*
+ * Can insn sit between the lr and the sc of a constrained LR/SC sequence?
+ * The ISA allows base integer instructions there, except loads, stores,
+ * JALR, FENCE, SYSTEM, backward jumps and taken backward branches, plus
+ * their compressed forms. Backward branches are treated as not allowed:
+ * the retry branch follows the sc, so one inside the body is not a
+ * constrained sequence.
+ */
+static bool __kprobes riscv_insn_is_lrsc_body(u32 insn)
+{
+ u32 funct3;
+
+ if (GET_INSN_LENGTH(insn) == 2) {
+ switch (insn & RVC_INSN_OPCODE_MASK) {
+ case RVC_OPCODE_C0:
+ /* c.addi4spn; the rest of the quadrant is loads and stores */
+ return (insn & RVC_INSN_FUNCT3_MASK) == 0;
+ case RVC_OPCODE_C1:
+ /* Arithmetic, c.j, c.jal and branches, bit 12 is the offset sign */
+ if (riscv_insn_is_c_j(insn) || riscv_insn_is_c_jal(insn) ||
+ riscv_insn_is_c_beqz(insn) || riscv_insn_is_c_bnez(insn))
+ return !(insn & BIT(12));
+ return true;
+ case RVC_OPCODE_C2:
+ funct3 = (insn & RVC_INSN_FUNCT3_MASK) >> RVC_INSN_FUNCT3_OPOFF;
+ if (funct3 == 0) /* c.slli */
+ return true;
+ if (funct3 == 4) /* c.mv, c.add, but not c.jr, c.jalr, c.ebreak */
+ return !riscv_insn_is_c_jr(insn) &&
+ !riscv_insn_is_c_jalr(insn) &&
+ !riscv_insn_is_c_ebreak(insn);
+ return false; /* stack loads and stores */
+ }
+ return false;
+ }
+
+ switch (insn & RV_INSN_OPCODE_MASK) {
+ case RVG_OPCODE_OP_IMM:
+ case RVG_OPCODE_OP_IMM_32:
+ case RVG_OPCODE_LUI:
+ case RVG_OPCODE_AUIPC:
+ return true;
+ case RVG_OPCODE_OP:
+ case RVG_OPCODE_OP_32:
+ /* funct7 0 or 0x20 is base I; anything else is M, B or another extension */
+ return !((insn & RV_INSN_FUNCT7_MASK) & ~(0x20U << RV_INSN_FUNCT7_OPOFF));
+ case RVG_OPCODE_BRANCH:
+ case RVG_OPCODE_JAL:
+ /* forward only, bit 31 is the offset sign */
+ return !(insn & BIT(31));
+ }
+ return false;
+}
+
+/*
+ * Does the instruction at the start of buf lie inside a constrained LR/SC
+ * sequence? Such a sequence holds at most RISCV_LRSC_MAX_BODY instructions
+ * between the lr and the sc, all from the set above. Walk forward from the
+ * probed instruction: reaching an sc through permitted instructions only
+ * means the probe sits in the body; anything else, including an lr that
+ * opens a new sequence, means it does not. The walk stops at the end of
+ * buf, so a short buffer only makes the answer more permissive.
+ */
+bool __kprobes riscv_probe_in_lrsc_body(const u16 *buf, size_t len)
+{
+ size_t off = 0;
+ u32 insn;
+ int n;
+
+ for (n = 0; n <= RISCV_LRSC_MAX_BODY; n++) {
+ if (off + 2 > len)
+ return false;
+ insn = buf[off / 2];
+ if (GET_INSN_LENGTH(insn) == 4) {
+ if (off + 4 > len)
+ return false;
+ insn |= (u32)buf[off / 2 + 1] << 16;
+ }
+ if (riscv_insn_is_sc(insn))
+ return true;
+ if (!riscv_insn_is_lrsc_body(insn))
+ return false;
+ off += GET_INSN_LENGTH(insn);
+ }
+ return false;
+}
+
/* Return:
* INSN_REJECTED If instruction is one not allowed to kprobe,
* INSN_GOOD_NO_SLOT If instruction is supported but doesn't use its slot.
@@ -23,6 +111,12 @@ riscv_probe_decode_insn(probe_opcode_t *addr, struct arch_probe_insn *api)
*/
RISCV_INSN_REJECTED(system, insn);
RISCV_INSN_REJECTED(fence, insn);
+ /*
+ * The breakpoint trap drops the reservation, so a probe on either
+ * end of an LR/SC sequence makes the SC fail on every retry.
+ */
+ RISCV_INSN_REJECTED(lr, insn);
+ RISCV_INSN_REJECTED(sc, insn);
/*
* Simulate instructions list:
diff --git a/arch/riscv/kernel/probes/decode-insn.h b/arch/riscv/kernel/probes/decode-insn.h
index 42269a7d676d..aa1b0d363545 100644
--- a/arch/riscv/kernel/probes/decode-insn.h
+++ b/arch/riscv/kernel/probes/decode-insn.h
@@ -15,4 +15,11 @@ enum probe_insn {
enum probe_insn __kprobes
riscv_probe_decode_insn(probe_opcode_t *addr, struct arch_probe_insn *asi);
+/* A constrained LR/SC sequence has at most this many instructions between lr and sc */
+#define RISCV_LRSC_MAX_BODY 16
+/* Bytes from the probed instruction through the farthest possible sc */
+#define RISCV_LRSC_SCAN_BYTES ((RISCV_LRSC_MAX_BODY + 1) * 4)
+
+bool __kprobes riscv_probe_in_lrsc_body(const u16 *buf, size_t len);
+
#endif /* _RISCV_KERNEL_KPROBES_DECODE_INSN_H */
diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c
index 9e2afabf94e8..1ade363910c9 100644
--- a/arch/riscv/kernel/probes/kprobes.c
+++ b/arch/riscv/kernel/probes/kprobes.c
@@ -49,19 +49,28 @@ static void __kprobes arch_simulate_insn(struct kprobe *p, struct pt_regs *regs)
post_kprobe_handler(p, kcb, regs);
}
+/*
+ * Walk the instructions from the enclosing symbol up to addr to check that
+ * addr is an instruction boundary. Then reject a probe inside an LR/SC
+ * sequence: the breakpoint trap drops the reservation, so the SC would fail
+ * and the loop would hit the probe again on every retry. The sequence is
+ * recognised by a forward scan with the constrained-sequence rules, as for
+ * uprobes, so an lr that is not followed by an sc (as in __cmpwait()) does
+ * not taint the rest of the function.
+ */
static bool __kprobes arch_check_kprobe(unsigned long addr)
{
- unsigned long tmp, offset;
+ unsigned long tmp, size, offset;
/* start iterating at the closest preceding symbol */
- if (!kallsyms_lookup_size_offset(addr, NULL, &offset))
+ if (!kallsyms_lookup_size_offset(addr, &size, &offset))
return false;
tmp = addr - offset;
while (tmp <= addr) {
if (tmp == addr)
- return true;
+ return !riscv_probe_in_lrsc_body((u16 *)addr, size - offset);
tmp += GET_INSN_LENGTH(*(u16 *)tmp);
}
diff --git a/arch/riscv/kernel/probes/uprobes.c b/arch/riscv/kernel/probes/uprobes.c
index eb177d0ce8ab..d97a10262166 100644
--- a/arch/riscv/kernel/probes/uprobes.c
+++ b/arch/riscv/kernel/probes/uprobes.c
@@ -28,6 +28,34 @@ unsigned long uprobe_get_swbp_addr(struct pt_regs *regs)
return instruction_pointer(regs);
}
+/*
+ * Is the instruction at addr inside a constrained LR/SC sequence? Only the
+ * probed instruction was copied, so read the text that follows it from the
+ * task. The caller holds mmap_lock, as uprobe_write_opcode() requires too.
+ * Text that cannot be read ends the scan early, which accepts the probe.
+ */
+static bool uprobe_in_lrsc_body(struct mm_struct *mm, unsigned long addr)
+{
+ u16 buf[RISCV_LRSC_SCAN_BYTES / sizeof(u16)];
+ struct vm_area_struct *vma;
+ struct page *page;
+ size_t got = 0;
+ int len;
+
+ while (got < sizeof(buf)) {
+ len = min_t(size_t, sizeof(buf) - got,
+ PAGE_SIZE - offset_in_page(addr + got));
+ page = get_user_page_vma_remote(mm, addr + got, FOLL_FORCE, &vma);
+ if (IS_ERR(page))
+ break;
+ uprobe_copy_from_page(page, addr + got, (void *)buf + got, len);
+ put_page(page);
+ got += len;
+ }
+
+ return riscv_probe_in_lrsc_body(buf, got);
+}
+
int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
unsigned long addr)
{
@@ -53,6 +81,9 @@ int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm,
return -EINVAL;
}
+ if (uprobe_in_lrsc_body(mm, addr))
+ return -EINVAL;
+
return 0;
}
--
2.43.0
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH v6 2/6] riscv: build kernel atomics from LR/SC when Zaamo is disabled
2026-11-10 13:20 [PATCH v6 0/6] riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO Vladimir Kondratiev
2026-11-10 13:20 ` [PATCH v6 1/6] riscv: probes: reject probes inside an LR/SC sequence Vladimir Kondratiev
@ 2026-11-10 13:20 ` Vladimir Kondratiev
2026-11-10 19:02 ` Conor Dooley
2026-11-10 13:21 ` [PATCH v6 3/6] riscv: accept the atomic extensions this kernel was built with Vladimir Kondratiev
` (3 subsequent siblings)
5 siblings, 1 reply; 8+ messages in thread
From: Vladimir Kondratiev @ 2026-11-10 13:20 UTC (permalink / raw)
To: Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Will Deacon, Peter Zijlstra, Mark Rutland, Gary Guo, Yury Norov,
Rasmus Villemoes, Chao-ying Fu, Conor Dooley, torvalds, olof,
aleksa.paunovic, arikalo, Boqun Feng, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
Puranjay Mohan, Masami Hiramatsu, Oleg Nesterov, Miguel Ojeda,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Shuah Khan, Randy Dunlap,
Boqun Feng
Cc: linux-riscv, linux-kernel, Vladimir.Kondratiev, bpf,
linux-trace-kernel, linux-doc, Jonathan Corbet, Charlie Jenkins,
Nam Cao, rust-for-linux, Vladimir Kondratiev
The A extension is Zaamo plus Zalrsc. On some CPUs Zaamo is present
only formally. MIPS P8700 traps every AMO instruction into M-mode and
emulates it there, so the extension is supported but very slow. A
typical uncontended atomic increment costs:
native AMO 2..5 cycles
LR/SC, private L1 reservation monitor 6..12 cycles
LR/SC, PoC (L2) reservation monitor about 40 cycles
AMO emulated by an M-mode trap about 600 cycles
The last two were measured on the eyeq7h hardware. On such a CPU the
kernel must use LR/SC for every atomic.
Add RISCV_ISA_ZAAMO, default y. The default keeps "a" in -march and
the generated code unchanged. When it is disabled, -march carries
_zalrsc instead of "a", and atomic_*, bitops, xchg, futex and the
early boot code use LR/SC loops. The sequence is chosen with the
preprocessor, on CONFIG_RISCV_ISA_ZAAMO like the other ISA options in
the headers, because the assembler rejects AMO mnemonics once -march
lacks Zaamo. A kernel built this way runs on any hart that implements
A; it just does not use AMO. It is therefore a plain option, not a
NONPORTABLE one, and the hart keeps advertising "a" to userspace,
which is true.
The LR/SC loops follow the ordering convention of the existing ones
in cmpxchg.h, from commit 5ce6c1f3535f ("riscv/atomic: Strengthen
implementations with fences") and commit 1d84afaf0252 ("riscv: Fix
fully ordered LR/SC xchg[8|16]() implementations"): a fully ordered
operation is lr, sc.rl and a trailing fence rw,rw; acquire is lr.aq
with a plain sc; release is a plain lr with sc.rl. The futex loop is
bounded to 128 retries and returns -EAGAIN after that, as arm64 does,
because userspace can keep the word changing from another thread and
futex_wake_op() retries on -EAGAIN anyway.
Zabha, Zacas and the ssamoswap instruction of Zicfiss are AMO
instructions, so RISCV_ISA_ZABHA, RISCV_ISA_ZACAS and RISCV_USER_CFI
depend on RISCV_ISA_ZAAMO. The riscv64imac Rust target emits AMO too,
so RUSTC_SUPPORTS_RISCV depends on it as well.
The choice is made at compile time rather than with alternatives.
Nearly every CPU has a native A extension, and an alternative would
inflate all those kernels with the nops needed to pad one AMO to the
size of an LR/SC loop, at every atomic site, for the benefit of a few.
Turning Zaamo off needs a compiler that accepts _zaamo and _zalrsc in
-march. GCC 15.1 is the first release that does.
TOOLCHAIN_HAS_ZALRSC_ZAAMO checks for it; with an older compiler the
option stays at y.
Three sets of extensions provide every atomic Linux needs. "a" is what
the kernel supports today. Zalrsc plus Ziccrse is what this patch adds:
every atomic is a constrained LR/SC loop, and Ziccrse is the
forward-progress guarantee for such loops. Zaamo plus Zacas would do
as well, with CAS loops in place of LR/SC as on x86, and is left for
whoever has such hardware.
Suggested-by: Chao-ying Fu <chao-ying.fu@globalfoundries.com>
Nacked-by: Conor Dooley <conor.dooley@microchip.com>
Link: https://lore.kernel.org/linux-riscv/20261106-smock-headband-d6a7d568c228@spud/
Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@mobileye.com>
---
arch/riscv/Kconfig | 36 +++++++++++++++++++
arch/riscv/Makefile | 15 +++++---
arch/riscv/include/asm/atomic.h | 53 +++++++++++++++++++++++++++-
arch/riscv/include/asm/bitops.h | 76 ++++++++++++++++++++++++++++++++++++++--
arch/riscv/include/asm/cmpxchg.h | 39 +++++++++++++++++----
arch/riscv/include/asm/futex.h | 58 ++++++++++++++++++++++++++++++
arch/riscv/kernel/entry.S | 7 ++++
arch/riscv/kernel/head.S | 12 +++++++
8 files changed, 281 insertions(+), 15 deletions(-)
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 5874ca30f4f6..3c3679701a60 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -237,6 +237,8 @@ config RISCV
config RUSTC_SUPPORTS_RISCV
def_bool y
depends on 64BIT
+ # The riscv64imac target emits AMO instructions
+ depends on RISCV_ISA_ZAAMO
config CLANG_SUPPORTS_DYNAMIC_FTRACE
def_bool CC_IS_CLANG
@@ -706,6 +708,7 @@ config RISCV_ISA_ZABHA
bool "Zabha extension support for atomic byte/halfword operations"
depends on TOOLCHAIN_HAS_ZABHA
depends on RISCV_ALTERNATIVE
+ depends on RISCV_ISA_ZAAMO
default y
help
Enable the use of the Zabha ISA-extension to implement kernel
@@ -723,6 +726,7 @@ config TOOLCHAIN_HAS_ZACAS
config RISCV_ISA_ZACAS
bool "Zacas extension support for atomic CAS"
depends on RISCV_ALTERNATIVE
+ depends on RISCV_ISA_ZAAMO
default y
help
Enable the use of the Zacas ISA-extension to implement kernel atomic
@@ -730,6 +734,36 @@ config RISCV_ISA_ZACAS
If you don't know what to do here, say Y.
+config TOOLCHAIN_HAS_ZALRSC_ZAAMO
+ bool
+ default y
+ depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64im_zaamo_zalrsc)
+ depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32im_zaamo_zalrsc)
+
+config RISCV_ISA_ZAAMO
+ bool "Zaamo extension support for AMO atomics" if TOOLCHAIN_HAS_ZALRSC_ZAAMO
+ default y
+ help
+ Use AMO instructions for kernel atomic memory operations. When
+ disabled, those operations are built from LR/SC and the kernel
+ is compiled with zalrsc instead of the A extension.
+
+ Disable this for a CPU where AMO is much slower than LR/SC. That
+ is the case when AMO instructions are emulated by a trap into
+ M-mode, as on the MIPS P8700: an emulated atomic costs hundreds
+ of cycles where LR/SC costs tens.
+
+ A kernel built without Zaamo runs on any CPU that implements the
+ A extension. Zabha, Zacas, userspace control flow integrity and
+ Rust need Zaamo and are unavailable without it.
+
+ The compiler must accept _zaamo and _zalrsc in -march to make
+ this choice. GCC 15.1 is the first release that does. With an
+ older compiler the option stays enabled and -march keeps the A
+ extension.
+
+ If you don't know what to do here, say Y.
+
config TOOLCHAIN_HAS_ZBB
bool
default y
@@ -1206,6 +1240,8 @@ config RISCV_USER_CFI
depends on 64BIT && MMU && \
$(cc-option,-mabi=lp64 -march=rv64ima_zicfiss_zicfilp -fcf-protection=full)
depends on RISCV_ALTERNATIVE
+ # ssamoswap is an AMO instruction
+ depends on RISCV_ISA_ZAAMO
select RISCV_SBI
select ARCH_HAS_USER_SHADOW_STACK
select ARCH_USES_HIGH_VMA_FLAGS
diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile
index 44f5a439741a..59c5123c5485 100644
--- a/arch/riscv/Makefile
+++ b/arch/riscv/Makefile
@@ -58,8 +58,9 @@ ifeq ($(CONFIG_SHADOW_CALL_STACK),y)
endif
# ISA string setting
-riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima
-riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima
+riscv-march-$(CONFIG_ARCH_RV32I) := rv32im
+riscv-march-$(CONFIG_ARCH_RV64I) := rv64im
+riscv-march-$(CONFIG_RISCV_ISA_ZAAMO) := $(riscv-march-y)a
riscv-march-$(CONFIG_FPU) := $(riscv-march-y)fd
riscv-march-$(CONFIG_RISCV_ISA_C) := $(riscv-march-y)c
riscv-march-$(CONFIG_RISCV_ISA_V) := $(riscv-march-y)v
@@ -75,9 +76,15 @@ else
riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei
endif
-KBUILD_BASE_ISA = -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')
+KBUILD_BASE_ISA = -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima?|rv64ima?)fd([^v_]*)v?/\1\2/')
export KBUILD_BASE_ISA
+# Without Zaamo only the LR/SC half of "a" is used. The compiler accepts
+# _zalrsc here, see TOOLCHAIN_HAS_ZALRSC_ZAAMO.
+ifneq ($(CONFIG_RISCV_ISA_ZAAMO),y)
+riscv-march-y := $(riscv-march-y)_zalrsc
+endif
+
# Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by
# matching non-v and non-multi-letter extensions out with the filter ([^v_]*)
KBUILD_CFLAGS += $(KBUILD_BASE_ISA)
@@ -85,7 +92,7 @@ KBUILD_CFLAGS += $(KBUILD_BASE_ISA)
KBUILD_AFLAGS += -march=$(riscv-march-y)
# For C code built with floating-point support, exclude V but keep F and D.
-CC_FLAGS_FPU := -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)([^v_]*)v?/\1\2/')
+CC_FLAGS_FPU := -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima?|rv64ima?)([^v_]*)v?/\1\2/')
KBUILD_CFLAGS += -mno-save-restore
diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h
index 616b8b332ac5..cca1fc3a68ff 100644
--- a/arch/riscv/include/asm/atomic.h
+++ b/arch/riscv/include/asm/atomic.h
@@ -50,6 +50,7 @@ static __always_inline void arch_atomic64_set(atomic64_t *v, s64 i)
* have the AQ or RL bits set. These don't return anything, so there's only
* one version to worry about.
*/
+#ifdef CONFIG_RISCV_ISA_ZAAMO
#define ATOMIC_OP(op, asm_op, I, asm_type, c_type, prefix) \
static __always_inline \
void arch_atomic##prefix##_##op(c_type i, atomic##prefix##_t *v) \
@@ -59,7 +60,23 @@ void arch_atomic##prefix##_##op(c_type i, atomic##prefix##_t *v) \
: "+A" (v->counter) \
: "r" (I) \
: "memory"); \
-} \
+}
+#else
+#define ATOMIC_OP(op, asm_op, I, asm_type, c_type, prefix) \
+static __always_inline \
+void arch_atomic##prefix##_##op(c_type i, atomic##prefix##_t *v) \
+{ \
+ register c_type ret, temp; \
+ __asm__ __volatile__ ( \
+ "1: lr." #asm_type " %1, %0\n" \
+ " " #asm_op " %2, %1, %3\n" \
+ " sc." #asm_type " %2, %2, %0\n" \
+ " bnez %2, 1b\n" \
+ : "+A" (v->counter), "=&r" (ret), "=&r" (temp) \
+ : "r" (I) \
+ : "memory"); \
+}
+#endif
#ifdef CONFIG_GENERIC_ATOMIC64
#define ATOMIC_OPS(op, asm_op, I) \
@@ -84,6 +101,7 @@ ATOMIC_OPS(xor, xor, i)
* There's two flavors of these: the arithmetic ops have both fetch and return
* versions, while the logical ops only have fetch versions.
*/
+#ifdef CONFIG_RISCV_ISA_ZAAMO
#define ATOMIC_FETCH_OP(op, asm_op, I, asm_type, c_type, prefix) \
static __always_inline \
c_type arch_atomic##prefix##_fetch_##op##_relaxed(c_type i, \
@@ -108,6 +126,39 @@ c_type arch_atomic##prefix##_fetch_##op(c_type i, atomic##prefix##_t *v) \
: "memory"); \
return ret; \
}
+#else
+#define ATOMIC_FETCH_OP(op, asm_op, I, asm_type, c_type, prefix) \
+static __always_inline \
+c_type arch_atomic##prefix##_fetch_##op##_relaxed(c_type i, \
+ atomic##prefix##_t *v) \
+{ \
+ register c_type ret, temp; \
+ __asm__ __volatile__ ( \
+ "1: lr." #asm_type " %1, %0\n" \
+ " " #asm_op " %2, %1, %3\n" \
+ " sc." #asm_type " %2, %2, %0\n" \
+ " bnez %2, 1b\n" \
+ : "+A" (v->counter), "=&r" (ret), "=&r" (temp) \
+ : "r" (I) \
+ : "memory"); \
+ return ret; \
+} \
+static __always_inline \
+c_type arch_atomic##prefix##_fetch_##op(c_type i, atomic##prefix##_t *v) \
+{ \
+ register c_type ret, temp; \
+ __asm__ __volatile__ ( \
+ "1: lr." #asm_type " %1, %0\n" \
+ " " #asm_op " %2, %1, %3\n" \
+ " sc." #asm_type ".rl %2, %2, %0\n" \
+ " bnez %2, 1b\n" \
+ RISCV_FULL_BARRIER \
+ : "+A" (v->counter), "=&r" (ret), "=&r" (temp) \
+ : "r" (I) \
+ : "memory"); \
+ return ret; \
+}
+#endif
#define ATOMIC_OP_RETURN(op, asm_op, c_op, I, asm_type, c_type, prefix) \
static __always_inline \
diff --git a/arch/riscv/include/asm/bitops.h b/arch/riscv/include/asm/bitops.h
index 3c1a15be54d8..adcd70525774 100644
--- a/arch/riscv/include/asm/bitops.h
+++ b/arch/riscv/include/asm/bitops.h
@@ -170,12 +170,17 @@ static __always_inline int variable_fls(unsigned int x)
#if (BITS_PER_LONG == 64)
#define __AMO(op) "amo" #op ".d"
+#define __LR "lr.d"
+#define __SC "sc.d"
#elif (BITS_PER_LONG == 32)
#define __AMO(op) "amo" #op ".w"
+#define __LR "lr.w"
+#define __SC "sc.w"
#else
#error "Unexpected BITS_PER_LONG"
#endif
+#ifdef CONFIG_RISCV_ISA_ZAAMO
#define __test_and_op_bit_ord(op, mod, nr, addr, ord) \
({ \
unsigned long __res, __mask; \
@@ -195,10 +200,58 @@ static __always_inline int variable_fls(unsigned int x)
: "r" (mod(BIT_MASK(nr))) \
: "memory");
-#define __test_and_op_bit(op, mod, nr, addr) \
+#define __test_and_op_bit(op, mod, nr, addr) \
__test_and_op_bit_ord(op, mod, nr, addr, .aqrl)
+#define __test_and_op_bit_acquire(op, mod, nr, addr) \
+ __test_and_op_bit_ord(op, mod, nr, addr, .aq)
#define __op_bit(op, mod, nr, addr) \
__op_bit_ord(op, mod, nr, addr, )
+#define __op_bit_release(op, mod, nr, addr) \
+ __op_bit_ord(op, mod, nr, addr, .rl)
+#else
+/*
+ * Fully ordered LR/SC is lr, sc.rl and a full fence; acquire is lr.aq and
+ * release is sc.rl, with no ordering on the other instruction. This is how
+ * the LR/SC loops in asm/atomic.h and asm/cmpxchg.h are built.
+ */
+#define __test_and_op_bit_lrsc(op, mod, nr, addr, lr_sfx, sc_sfx, append) \
+({ \
+ unsigned long __res, __mask, __temp; \
+ __mask = BIT_MASK(nr); \
+ __asm__ __volatile__ ( \
+ "1: " __LR lr_sfx " %0, %1\n" \
+ #op " %2, %0, %3\n" \
+ __SC sc_sfx " %2, %2, %1\n" \
+ "bnez %2, 1b\n" \
+ append \
+ : "=&r" (__res), "+A" (addr[BIT_WORD(nr)]), "=&r" (__temp) \
+ : "r" (mod(__mask)) \
+ : "memory"); \
+ ((__res & __mask) != 0); \
+})
+
+#define __op_bit_lrsc(op, mod, nr, addr, lr_sfx, sc_sfx) \
+do { \
+ unsigned long __res, __temp; \
+ __asm__ __volatile__ ( \
+ "1: " __LR lr_sfx " %0, %1\n" \
+ #op " %2, %0, %3\n" \
+ __SC sc_sfx " %2, %2, %1\n" \
+ "bnez %2, 1b\n" \
+ : "=&r" (__res), "+A" (addr[BIT_WORD(nr)]), "=&r" (__temp) \
+ : "r" (mod(BIT_MASK(nr))) \
+ : "memory"); \
+} while (0)
+
+#define __test_and_op_bit(op, mod, nr, addr) \
+ __test_and_op_bit_lrsc(op, mod, nr, addr, "", ".rl", RISCV_FULL_BARRIER)
+#define __test_and_op_bit_acquire(op, mod, nr, addr) \
+ __test_and_op_bit_lrsc(op, mod, nr, addr, ".aq", "", "")
+#define __op_bit(op, mod, nr, addr) \
+ __op_bit_lrsc(op, mod, nr, addr, "", "")
+#define __op_bit_release(op, mod, nr, addr) \
+ __op_bit_lrsc(op, mod, nr, addr, "", ".rl")
+#endif
/* Bitmask modifiers */
#define __NOP(x) (x)
@@ -297,7 +350,7 @@ static __always_inline void arch_change_bit(int nr, volatile unsigned long *addr
static __always_inline int arch_test_and_set_bit_lock(
unsigned long nr, volatile unsigned long *addr)
{
- return __test_and_op_bit_ord(or, __NOP, nr, addr, .aq);
+ return __test_and_op_bit_acquire(or, __NOP, nr, addr);
}
/**
@@ -310,7 +363,7 @@ static __always_inline int arch_test_and_set_bit_lock(
static __always_inline void arch_clear_bit_unlock(
unsigned long nr, volatile unsigned long *addr)
{
- __op_bit_ord(and, __NOT, nr, addr, .rl);
+ __op_bit_release(and, __NOT, nr, addr);
}
/**
@@ -337,20 +390,37 @@ static __always_inline void arch___clear_bit_unlock(
static __always_inline bool arch_xor_unlock_is_negative_byte(unsigned long mask,
volatile unsigned long *addr)
{
+#ifdef CONFIG_RISCV_ISA_ZAAMO
unsigned long res;
__asm__ __volatile__ (
__AMO(xor) ".rl %0, %2, %1"
: "=r" (res), "+A" (*addr)
: "r" (__NOP(mask))
: "memory");
+#else
+ unsigned long res, temp;
+
+ __asm__ __volatile__ (
+ "1: " __LR " %0, %1\n"
+ "xor %2, %0, %3\n"
+ __SC ".rl %2, %2, %1\n"
+ "bnez %2, 1b\n"
+ : "=&r" (res), "+A" (*addr), "=&r" (temp)
+ : "r" (__NOP(mask))
+ : "memory");
+#endif
return (res & BIT(7)) != 0;
}
#undef __test_and_op_bit
+#undef __test_and_op_bit_acquire
#undef __op_bit
+#undef __op_bit_release
#undef __NOP
#undef __NOT
#undef __AMO
+#undef __LR
+#undef __SC
#include <asm-generic/bitops/instrumented-atomic.h>
#include <asm-generic/bitops/instrumented-lock.h>
diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h
index 662e160b0522..f0199432948a 100644
--- a/arch/riscv/include/asm/cmpxchg.h
+++ b/arch/riscv/include/asm/cmpxchg.h
@@ -57,16 +57,39 @@
} \
})
-#define __arch_xchg(sfx, prepend, append, r, p, n) \
+/*
+ * Word and double-word xchg is one amoswap, or an LR/SC loop without
+ * Zaamo. Each form uses its own pair of ordering arguments, as in
+ * __arch_xchg_masked() above.
+ */
+#ifdef CONFIG_RISCV_ISA_ZAAMO
+#define __arch_xchg(w, sc_sfx, swap_sfx, prepend, sc_append, \
+ swap_append, r, p, n) \
({ \
__asm__ __volatile__ ( \
prepend \
- " amoswap" sfx " %0, %2, %1\n" \
- append \
+ " amoswap" w swap_sfx " %0, %2, %1\n" \
+ swap_append \
: "=r" (r), "+A" (*(p)) \
: "r" (n) \
: "memory"); \
})
+#else
+#define __arch_xchg(w, sc_sfx, swap_sfx, prepend, sc_append, \
+ swap_append, r, p, n) \
+({ \
+ __typeof__(*(p)) __temp; \
+ __asm__ __volatile__ ( \
+ prepend \
+ "1: lr" w " %0, %1\n" \
+ " sc" w sc_sfx " %2, %3, %1\n" \
+ " bnez %2, 1b\n" \
+ sc_append \
+ : "=&r" (r), "+A" (*(p)), "=&r" (__temp) \
+ : "r" (n) \
+ : "memory"); \
+})
+#endif
#define _arch_xchg(ptr, new, sc_sfx, swap_sfx, prepend, \
sc_append, swap_append) \
@@ -87,12 +110,14 @@
__ret, __ptr, __new); \
break; \
case 4: \
- __arch_xchg(".w" swap_sfx, prepend, swap_append, \
- __ret, __ptr, __new); \
+ __arch_xchg(".w", sc_sfx, swap_sfx, \
+ prepend, sc_append, swap_append, \
+ __ret, __ptr, __new); \
break; \
case 8: \
- __arch_xchg(".d" swap_sfx, prepend, swap_append, \
- __ret, __ptr, __new); \
+ __arch_xchg(".d", sc_sfx, swap_sfx, \
+ prepend, sc_append, swap_append, \
+ __ret, __ptr, __new); \
break; \
default: \
BUILD_BUG(); \
diff --git a/arch/riscv/include/asm/futex.h b/arch/riscv/include/asm/futex.h
index 6f3bc9bdb85e..e79d28af55cd 100644
--- a/arch/riscv/include/asm/futex.h
+++ b/arch/riscv/include/asm/futex.h
@@ -12,6 +12,7 @@
#include <linux/errno.h>
#include <asm/asm.h>
#include <asm/asm-extable.h>
+#include <asm/fence.h>
/* We don't even really need the extable code, but for now keep it simple */
#ifndef CONFIG_MMU
@@ -19,6 +20,7 @@
#define __disable_user_access() do { } while (0)
#endif
+#ifdef CONFIG_RISCV_ISA_ZAAMO
#define __futex_atomic_op(insn, ret, oldval, uaddr, oparg) \
{ \
__enable_user_access(); \
@@ -32,6 +34,39 @@
: "memory"); \
__disable_user_access(); \
}
+#else
+/*
+ * Bound the number of LR/SC retries, as arm64 does for LL/SC: a user can
+ * keep the word changing from another thread forever. futex_wake_op()
+ * retries on -EAGAIN after dropping the hash bucket locks.
+ */
+#define FUTEX_MAX_LOOPS 128
+
+#define __futex_atomic_op(insn, ret, oldval, uaddr, oparg) \
+{ \
+ int temp, loops = FUTEX_MAX_LOOPS; \
+ \
+ __enable_user_access(); \
+ __asm__ __volatile__ ( \
+ "1: lr.w %[ov], %[u] \n" \
+ " " insn " \n" \
+ "2: sc.w.rl %[t], %[t], %[u] \n" \
+ " beqz %[t], 3f \n" \
+ " addi %[l], %[l], -1 \n" \
+ " bnez %[l], 1b \n" \
+ " li %[r], %[ea] \n" \
+ "3: \n" \
+ RISCV_FULL_BARRIER \
+ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %[r]) \
+ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %[r]) \
+ : [r] "+r" (ret), [ov] "=&r" (oldval), \
+ [t] "=&r" (temp), [l] "+&r" (loops), \
+ [u] "+m" (*(uaddr)) \
+ : [op] "Jr" (oparg), [ea] "I" (-EAGAIN) \
+ : "memory"); \
+ __disable_user_access(); \
+}
+#endif
static inline int
arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
@@ -43,6 +78,7 @@ arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
uaddr = untagged_addr(uaddr);
switch (op) {
+#ifdef CONFIG_RISCV_ISA_ZAAMO
case FUTEX_OP_SET:
__futex_atomic_op("amoswap.w.aqrl %[ov],%z[op],%[u]",
ret, oldval, uaddr, oparg);
@@ -63,6 +99,28 @@ arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
__futex_atomic_op("amoxor.w.aqrl %[ov],%z[op],%[u]",
ret, oldval, uaddr, oparg);
break;
+#else
+ case FUTEX_OP_SET:
+ __futex_atomic_op("mv %[t], %z[op]",
+ ret, oldval, uaddr, oparg);
+ break;
+ case FUTEX_OP_ADD:
+ __futex_atomic_op("add %[t], %[ov], %z[op]",
+ ret, oldval, uaddr, oparg);
+ break;
+ case FUTEX_OP_OR:
+ __futex_atomic_op("or %[t], %[ov], %z[op]",
+ ret, oldval, uaddr, oparg);
+ break;
+ case FUTEX_OP_ANDN:
+ __futex_atomic_op("and %[t], %[ov], %z[op]",
+ ret, oldval, uaddr, ~oparg);
+ break;
+ case FUTEX_OP_XOR:
+ __futex_atomic_op("xor %[t], %[ov], %z[op]",
+ ret, oldval, uaddr, oparg);
+ break;
+#endif
default:
ret = -ENOSYS;
}
diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
index 61b7a7716697..ba2db6804a6c 100644
--- a/arch/riscv/kernel/entry.S
+++ b/arch/riscv/kernel/entry.S
@@ -73,7 +73,14 @@
beq a2, zero, .Lnew_valid_map_cpus_restore_context
/* Atomically reset the current cpu bit in new_valid_map_cpus */
+#ifdef CONFIG_RISCV_ISA_ZAAMO
amoxor.d a0, a1, (a0)
+#else
+1: lr.d a2, (a0)
+ xor a2, a1, a2
+ sc.d a2, a2, (a0)
+ bnez a2, 1b
+#endif
/*
* A sfence.vma is required here. Even if we had Svvptc, there's no
diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S
index f6a8ca49e627..dc3862ed29c4 100644
--- a/arch/riscv/kernel/head.S
+++ b/arch/riscv/kernel/head.S
@@ -265,7 +265,19 @@ SYM_CODE_START(_start_kernel)
/* Pick one hart to run the main boot sequence */
la a3, hart_lottery
li a2, 1
+#ifdef CONFIG_RISCV_ISA_ZAAMO
amoadd.w a3, a2, (a3)
+#else
+ /*
+ * hart_lottery is a small 32-bit counter. add is valid on RV32
+ * and RV64; sc.w stores the low 32 bits.
+ */
+1: lr.w t0, (a3)
+ add t1, t0, a2
+ sc.w t1, t1, (a3)
+ bnez t1, 1b
+ mv a3, t0
+#endif
bnez a3, .Lsecondary_start
#endif /* CONFIG_RISCV_BOOT_SPINWAIT */
--
2.43.0
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH v6 2/6] riscv: build kernel atomics from LR/SC when Zaamo is disabled
2026-11-10 13:20 ` [PATCH v6 2/6] riscv: build kernel atomics from LR/SC when Zaamo is disabled Vladimir Kondratiev
@ 2026-11-10 19:02 ` Conor Dooley
0 siblings, 0 replies; 8+ messages in thread
From: Conor Dooley @ 2026-11-10 19:02 UTC (permalink / raw)
To: Vladimir Kondratiev
Cc: Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Will Deacon, Peter Zijlstra, Mark Rutland, Gary Guo, Yury Norov,
Rasmus Villemoes, Chao-ying Fu, Conor Dooley, torvalds, olof,
aleksa.paunovic, arikalo, Boqun Feng, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
Puranjay Mohan, Masami Hiramatsu, Oleg Nesterov, Miguel Ojeda,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Shuah Khan, Randy Dunlap,
linux-riscv, linux-kernel, bpf, linux-trace-kernel, linux-doc,
Jonathan Corbet, Charlie Jenkins, Nam Cao, rust-for-linux
[-- Attachment #1: Type: text/plain, Size: 3687 bytes --]
On Tue, Nov 10, 2026 at 03:20:59PM +0200, Vladimir Kondratiev wrote:
> The A extension is Zaamo plus Zalrsc. On some CPUs Zaamo is present
> only formally. MIPS P8700 traps every AMO instruction into M-mode and
> emulates it there, so the extension is supported but very slow. A
> typical uncontended atomic increment costs:
>
> native AMO 2..5 cycles
> LR/SC, private L1 reservation monitor 6..12 cycles
> LR/SC, PoC (L2) reservation monitor about 40 cycles
> AMO emulated by an M-mode trap about 600 cycles
>
> The last two were measured on the eyeq7h hardware. On such a CPU the
> kernel must use LR/SC for every atomic.
>
> Add RISCV_ISA_ZAAMO, default y. The default keeps "a" in -march and
> the generated code unchanged. When it is disabled, -march carries
> _zalrsc instead of "a", and atomic_*, bitops, xchg, futex and the
> early boot code use LR/SC loops. The sequence is chosen with the
> preprocessor, on CONFIG_RISCV_ISA_ZAAMO like the other ISA options in
> the headers, because the assembler rejects AMO mnemonics once -march
> lacks Zaamo. A kernel built this way runs on any hart that implements
> A; it just does not use AMO. It is therefore a plain option, not a
> NONPORTABLE one, and the hart keeps advertising "a" to userspace,
> which is true.
>
> The LR/SC loops follow the ordering convention of the existing ones
> in cmpxchg.h, from commit 5ce6c1f3535f ("riscv/atomic: Strengthen
> implementations with fences") and commit 1d84afaf0252 ("riscv: Fix
> fully ordered LR/SC xchg[8|16]() implementations"): a fully ordered
> operation is lr, sc.rl and a trailing fence rw,rw; acquire is lr.aq
> with a plain sc; release is a plain lr with sc.rl. The futex loop is
> bounded to 128 retries and returns -EAGAIN after that, as arm64 does,
> because userspace can keep the word changing from another thread and
> futex_wake_op() retries on -EAGAIN anyway.
>
> Zabha, Zacas and the ssamoswap instruction of Zicfiss are AMO
> instructions, so RISCV_ISA_ZABHA, RISCV_ISA_ZACAS and RISCV_USER_CFI
> depend on RISCV_ISA_ZAAMO. The riscv64imac Rust target emits AMO too,
> so RUSTC_SUPPORTS_RISCV depends on it as well.
>
> The choice is made at compile time rather than with alternatives.
> Nearly every CPU has a native A extension, and an alternative would
> inflate all those kernels with the nops needed to pad one AMO to the
> size of an LR/SC loop, at every atomic site, for the benefit of a few.
>
> Turning Zaamo off needs a compiler that accepts _zaamo and _zalrsc in
> -march. GCC 15.1 is the first release that does.
> TOOLCHAIN_HAS_ZALRSC_ZAAMO checks for it; with an older compiler the
> option stays at y.
>
> Three sets of extensions provide every atomic Linux needs. "a" is what
> the kernel supports today. Zalrsc plus Ziccrse is what this patch adds:
> every atomic is a constrained LR/SC loop, and Ziccrse is the
> forward-progress guarantee for such loops. Zaamo plus Zacas would do
> as well, with CAS loops in place of LR/SC as on x86, and is left for
> whoever has such hardware.
>
> Suggested-by: Chao-ying Fu <chao-ying.fu@globalfoundries.com>
> Nacked-by: Conor Dooley <conor.dooley@microchip.com>
> Link: https://lore.kernel.org/linux-riscv/20261106-smock-headband-d6a7d568c228@spud/
I had a quick glance at this revision, and I think you've at least tried
to resolve the things I was pointing out. Did I miss something, or why
have you kept the NAK after having resolved the things that I was
objecting to?
Thanks,
Conor.
> Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@mobileye.com>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v6 3/6] riscv: accept the atomic extensions this kernel was built with
2026-11-10 13:20 [PATCH v6 0/6] riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO Vladimir Kondratiev
2026-11-10 13:20 ` [PATCH v6 1/6] riscv: probes: reject probes inside an LR/SC sequence Vladimir Kondratiev
2026-11-10 13:20 ` [PATCH v6 2/6] riscv: build kernel atomics from LR/SC when Zaamo is disabled Vladimir Kondratiev
@ 2026-11-10 13:21 ` Vladimir Kondratiev
2026-11-10 13:21 ` [PATCH v6 4/6] riscv: report the A extension when Zaamo and Zalrsc are both present Vladimir Kondratiev
` (2 subsequent siblings)
5 siblings, 0 replies; 8+ messages in thread
From: Vladimir Kondratiev @ 2026-11-10 13:21 UTC (permalink / raw)
To: Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Will Deacon, Peter Zijlstra, Mark Rutland, Gary Guo, Yury Norov,
Rasmus Villemoes, Chao-ying Fu, Conor Dooley, torvalds, olof,
aleksa.paunovic, arikalo, Boqun Feng, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
Puranjay Mohan, Masami Hiramatsu, Oleg Nesterov, Miguel Ojeda,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Shuah Khan, Randy Dunlap,
Boqun Feng
Cc: linux-riscv, linux-kernel, Vladimir.Kondratiev, bpf,
linux-trace-kernel, linux-doc, Jonathan Corbet, Charlie Jenkins,
Nam Cao, rust-for-linux, Vladimir Kondratiev
riscv_early_of_processor_hartid() rejects a hart unless its
riscv,isa-extensions property contains "a". A hart that lists "zaamo"
and "zalrsc" instead of "a" is rejected, and so is one that lists only
"zalrsc", even when the kernel was built to use only LR/SC.
Accept "a" as before. Without it, require "zalrsc", which every kernel
uses for cmpxchg, and "zaamo" when RISCV_ISA_ZAAMO is enabled. The
default build therefore needs "a", or "zaamo" together with "zalrsc";
a kernel built without Zaamo needs only "zalrsc".
The legacy "riscv,isa" fallback keeps its rv{32,64}ima check.
Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@mobileye.com>
---
arch/riscv/kernel/cpu.c | 23 ++++++++++++++++++++---
1 file changed, 20 insertions(+), 3 deletions(-)
diff --git a/arch/riscv/kernel/cpu.c b/arch/riscv/kernel/cpu.c
index 3dbc8cc557dd..9ba0d49c87bc 100644
--- a/arch/riscv/kernel/cpu.c
+++ b/arch/riscv/kernel/cpu.c
@@ -82,12 +82,29 @@ int __init riscv_early_of_processor_hartid(struct device_node *node, unsigned lo
return -ENODEV;
if (of_property_match_string(node, "riscv,isa-extensions", "i") < 0 ||
- of_property_match_string(node, "riscv,isa-extensions", "m") < 0 ||
- of_property_match_string(node, "riscv,isa-extensions", "a") < 0) {
- pr_warn("CPU with hartid=%lu does not support ima", *hart);
+ of_property_match_string(node, "riscv,isa-extensions", "m") < 0) {
+ pr_warn("CPU with hartid=%lu does not support im", *hart);
return -ENODEV;
}
+ /*
+ * Require the atomic extensions this kernel was built with. "a"
+ * covers both; otherwise LR/SC is always needed and AMO is needed
+ * when the kernel uses it.
+ */
+ if (of_property_match_string(node, "riscv,isa-extensions", "a") < 0) {
+ if (of_property_match_string(node, "riscv,isa-extensions", "zalrsc") < 0) {
+ pr_warn("CPU with hartid=%lu does not support zalrsc", *hart);
+ return -ENODEV;
+ }
+
+ if (IS_ENABLED(CONFIG_RISCV_ISA_ZAAMO) &&
+ of_property_match_string(node, "riscv,isa-extensions", "zaamo") < 0) {
+ pr_warn("CPU with hartid=%lu does not support zaamo", *hart);
+ return -ENODEV;
+ }
+ }
+
return 0;
old_interface:
--
2.43.0
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH v6 4/6] riscv: report the A extension when Zaamo and Zalrsc are both present
2026-11-10 13:20 [PATCH v6 0/6] riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO Vladimir Kondratiev
` (2 preceding siblings ...)
2026-11-10 13:21 ` [PATCH v6 3/6] riscv: accept the atomic extensions this kernel was built with Vladimir Kondratiev
@ 2026-11-10 13:21 ` Vladimir Kondratiev
2026-11-10 13:21 ` [PATCH v6 5/6] riscv, bpf: emit LR/SC loops when Zaamo is disabled Vladimir Kondratiev
2026-11-10 13:21 ` [PATCH v6 6/6] riscv: hwprobe: report whether AMO is faster than LR/SC Vladimir Kondratiev
5 siblings, 0 replies; 8+ messages in thread
From: Vladimir Kondratiev @ 2026-11-10 13:21 UTC (permalink / raw)
To: Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Will Deacon, Peter Zijlstra, Mark Rutland, Gary Guo, Yury Norov,
Rasmus Villemoes, Chao-ying Fu, Conor Dooley, torvalds, olof,
aleksa.paunovic, arikalo, Boqun Feng, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
Puranjay Mohan, Masami Hiramatsu, Oleg Nesterov, Miguel Ojeda,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Shuah Khan, Randy Dunlap,
Boqun Feng
Cc: linux-riscv, linux-kernel, Vladimir.Kondratiev, bpf,
linux-trace-kernel, linux-doc, Jonathan Corbet, Charlie Jenkins,
Nam Cao, rust-for-linux, Vladimir Kondratiev
A hart may describe its atomics as "a", or as "zaamo" and "zalrsc".
The two are the same thing: the A extension is defined as Zaamo plus
Zalrsc. The kernel already expands "a" into both sub-extensions, but
not the other way round, so a hart that lists the sub-extensions ends
up without RISCV_ISA_EXT_A. /proc/cpuinfo then shows rv64imfdc... and
the A bit is missing from ELF hwcap, although the hardware is the
same as one that lists "a".
Set A in the source ISA bitmap when both Zaamo and Zalrsc are there,
before the bitmap is resolved, so that hwcap and cpuinfo do not depend
on how the firmware spelled it. This covers the "riscv,isa" string and
ACPI as well as "riscv,isa-extensions".
Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@mobileye.com>
---
arch/riscv/kernel/cpufeature.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c
index 7329792978cc..45684e129be1 100644
--- a/arch/riscv/kernel/cpufeature.c
+++ b/arch/riscv/kernel/cpufeature.c
@@ -682,6 +682,15 @@ static void __init riscv_resolve_isa(unsigned long *source_isa,
int max_loop_count = riscv_isa_ext_count, ret;
unsigned int bit;
+ /*
+ * "a" is exactly Zaamo plus Zalrsc. A hart that lists both has A,
+ * however the firmware spelled it, so report it the same way in
+ * ELF hwcap and /proc/cpuinfo.
+ */
+ if (test_bit(RISCV_ISA_EXT_ZAAMO, source_isa) &&
+ test_bit(RISCV_ISA_EXT_ZALRSC, source_isa))
+ set_bit(RISCV_ISA_EXT_A, source_isa);
+
do {
loop = false;
if (max_loop_count-- < 0) {
--
2.43.0
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH v6 5/6] riscv, bpf: emit LR/SC loops when Zaamo is disabled
2026-11-10 13:20 [PATCH v6 0/6] riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO Vladimir Kondratiev
` (3 preceding siblings ...)
2026-11-10 13:21 ` [PATCH v6 4/6] riscv: report the A extension when Zaamo and Zalrsc are both present Vladimir Kondratiev
@ 2026-11-10 13:21 ` Vladimir Kondratiev
2026-11-10 13:21 ` [PATCH v6 6/6] riscv: hwprobe: report whether AMO is faster than LR/SC Vladimir Kondratiev
5 siblings, 0 replies; 8+ messages in thread
From: Vladimir Kondratiev @ 2026-11-10 13:21 UTC (permalink / raw)
To: Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Will Deacon, Peter Zijlstra, Mark Rutland, Gary Guo, Yury Norov,
Rasmus Villemoes, Chao-ying Fu, Conor Dooley, torvalds, olof,
aleksa.paunovic, arikalo, Boqun Feng, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
Puranjay Mohan, Masami Hiramatsu, Oleg Nesterov, Miguel Ojeda,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Shuah Khan, Randy Dunlap,
Boqun Feng
Cc: linux-riscv, linux-kernel, Vladimir.Kondratiev, bpf,
linux-trace-kernel, linux-doc, Jonathan Corbet, Charlie Jenkins,
Nam Cao, rust-for-linux, Vladimir Kondratiev
A kernel built with CONFIG_RISCV_ISA_ZAAMO disabled is meant for a
hart where AMO instructions are emulated by an M-mode trap. The BPF JIT
still encodes amo* instructions for the BPF atomic read-modify-write
operations, so every such operation takes that trap.
Emit an LR/SC loop for BPF_ADD, BPF_AND, BPF_OR, BPF_XOR, their
BPF_FETCH forms and BPF_XCHG when Zaamo is disabled, on RV32 and RV64.
The sequences follow the Zalrsc variants in
arch/riscv/include/asm/atomic.h: a relaxed op discards the old value,
a fetch op uses .aqrl and returns the old value. BPF_CMPXCHG is
unchanged; it already uses amocas with Zacas and an LR/SC loop
otherwise.
The JIT encodes instructions as integers, so no assembler constraint
applies and both paths stay compiled. The choice is a plain
IS_ENABLED() test, the same compile-time decision the rest of the
kernel makes. The hart advertises "a" either way, so a runtime
extension check would not tell the two cases apart.
An arena (BPF_PROBE_ATOMIC) access needs an exception table entry for
each instruction that may fault. The verifier reserves one entry per
atomic instruction, but an LR/SC loop can fault on either the lr or the
sc. Count the extra entries once the instruction offsets converge,
before the image is allocated, and add an entry for the sc that shares
the fixup of the lr, so a fault skips the retry and the result move.
Co-developed-by: Chao-ying Fu <chao-ying.fu@globalfoundries.com>
Signed-off-by: Chao-ying Fu <chao-ying.fu@globalfoundries.com>
Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@mobileye.com>
---
arch/riscv/net/bpf_jit.h | 1 +
arch/riscv/net/bpf_jit_comp32.c | 81 ++++++++++++++++++++++++++++--
arch/riscv/net/bpf_jit_comp64.c | 108 +++++++++++++++++++++++++++++++++++++++-
arch/riscv/net/bpf_jit_core.c | 29 +++++++++++
4 files changed, 213 insertions(+), 6 deletions(-)
diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h
index 419b9d795f2a..8c1301289ea8 100644
--- a/arch/riscv/net/bpf_jit.h
+++ b/arch/riscv/net/bpf_jit.h
@@ -79,6 +79,7 @@ struct rv_jit_context {
int nexentries;
int ex_insn_off;
int ex_jmp_off;
+ int ex_sc_off; /* sc in an lr/sc pair; 0 if unused */
unsigned long flags;
int stack_size;
int tcc_offset;
diff --git a/arch/riscv/net/bpf_jit_comp32.c b/arch/riscv/net/bpf_jit_comp32.c
index a9e0bd5cc81d..ea1779a23634 100644
--- a/arch/riscv/net/bpf_jit_comp32.c
+++ b/arch/riscv/net/bpf_jit_comp32.c
@@ -874,11 +874,67 @@ static int emit_load_r64(const s8 *dst, const s8 *src, s16 off,
return 0;
}
-static int emit_bpf_atomic(s8 dst, const s8 *src, const s8 *rs,
- struct rv_jit_context *ctx,
- const struct bpf_insn *insn)
+/*
+ * Read-modify-write via lr/sc for kernels built without Zaamo. T0 holds
+ * the address. T1/T2 are not BPF registers; the address was copied out of
+ * lo(rd) before this is called, so clobbering T2 is fine.
+ */
+static int emit_bpf_atomic_lrsc(s8 dst, const s8 *rs, s32 imm,
+ struct rv_jit_context *ctx)
+{
+ bool is_fetch = (imm & BPF_FETCH) || (imm == BPF_XCHG);
+ bool xchg = imm == BPF_XCHG;
+ u8 aq = is_fetch ? 1 : 0;
+ u8 rl = is_fetch ? 1 : 0;
+ u8 old = RV_REG_T1;
+ u8 tmp = RV_REG_T2;
+ int lr_off, jmp_off;
+ s32 op = xchg ? BPF_XCHG : imm & ~BPF_FETCH;
+
+ switch (op) {
+ case BPF_ADD:
+ case BPF_AND:
+ case BPF_OR:
+ case BPF_XOR:
+ case BPF_XCHG:
+ break;
+ default:
+ return -1;
+ }
+
+ lr_off = ctx->ninsns;
+ emit(rv_lr_w(old, 0, dst, aq, rl), ctx);
+
+ switch (op) {
+ case BPF_ADD:
+ emit(rv_add(tmp, old, lo(rs)), ctx);
+ break;
+ case BPF_AND:
+ emit(rv_and(tmp, old, lo(rs)), ctx);
+ break;
+ case BPF_OR:
+ emit(rv_or(tmp, old, lo(rs)), ctx);
+ break;
+ case BPF_XOR:
+ emit(rv_xor(tmp, old, lo(rs)), ctx);
+ break;
+ }
+
+ if (xchg)
+ emit(rv_sc_w(tmp, lo(rs), dst, aq, rl), ctx);
+ else
+ emit(rv_sc_w(tmp, tmp, dst, aq, rl), ctx);
+
+ jmp_off = ninsns_rvoff(lr_off - ctx->ninsns);
+ emit(rv_bne(tmp, RV_REG_ZERO, jmp_off >> 1), ctx);
+ if (is_fetch)
+ emit(rv_addi(lo(rs), old, 0), ctx);
+ return 0;
+}
+
+static int emit_bpf_atomic_amo(s8 dst, const s8 *rs, s32 imm,
+ struct rv_jit_context *ctx)
{
- s32 imm = insn->imm;
bool is_fetch = (imm & BPF_FETCH) || (imm == BPF_XCHG);
s8 fetch_reg = is_fetch ? lo(rs) : RV_REG_ZERO;
int aq = is_fetch ? 1 : 0;
@@ -907,6 +963,23 @@ static int emit_bpf_atomic(s8 dst, const s8 *src, const s8 *rs,
default:
return -1;
}
+ return 0;
+}
+
+static int emit_bpf_atomic(s8 dst, const s8 *src, const s8 *rs,
+ struct rv_jit_context *ctx,
+ const struct bpf_insn *insn)
+{
+ s32 imm = insn->imm;
+ bool is_fetch = (imm & BPF_FETCH) || (imm == BPF_XCHG);
+ int ret;
+
+ if (IS_ENABLED(CONFIG_RISCV_ISA_ZAAMO))
+ ret = emit_bpf_atomic_amo(dst, rs, imm, ctx);
+ else
+ ret = emit_bpf_atomic_lrsc(dst, rs, imm, ctx);
+ if (ret)
+ return ret;
if (is_fetch) {
emit(rv_addi(hi(rs), RV_REG_ZERO, 0), ctx);
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 01fe66774f02..94e4ddc6d965 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -573,6 +573,79 @@ static int emit_atomic_ld_st(u8 rd, u8 rs, const struct bpf_insn *insn,
return 0;
}
+/*
+ * Read-modify-write via lr/sc for kernels built without Zaamo, matching
+ * the Zalrsc sequences in arch/riscv/include/asm/atomic.h. A relaxed op
+ * discards the old value. A fetch op (including XCHG) uses .aqrl and
+ * writes that value back to val. T2/T3 are not BPF registers; addr may
+ * already be T1.
+ */
+static void emit_atomic_lrsc(u8 addr, u8 val, bool is64, s32 imm,
+ struct rv_jit_context *ctx)
+{
+ bool fetch = imm & BPF_FETCH;
+ u8 aq = fetch ? 1 : 0;
+ u8 rl = fetch ? 1 : 0;
+ u8 old = RV_REG_T2;
+ u8 tmp = RV_REG_T3;
+ int lr_off, jmp_off;
+ s32 op = imm == BPF_XCHG ? BPF_XCHG : imm & ~BPF_FETCH;
+
+ lr_off = ctx->ninsns;
+ ctx->ex_insn_off = ctx->ninsns;
+ emit(is64 ? rv_lr_d(old, 0, addr, aq, rl) :
+ rv_lr_w(old, 0, addr, aq, rl), ctx);
+
+ if (op != BPF_XCHG) {
+ if (!is64 && op == BPF_ADD)
+ emit(rv_addw(tmp, old, val), ctx);
+ else if (op == BPF_ADD)
+ emit_add(tmp, old, val, ctx);
+ else if (op == BPF_AND)
+ emit_and(tmp, old, val, ctx);
+ else if (op == BPF_OR)
+ emit_or(tmp, old, val, ctx);
+ else if (op == BPF_XOR)
+ emit_xor(tmp, old, val, ctx);
+
+ ctx->ex_sc_off = ctx->ninsns;
+ emit(is64 ? rv_sc_d(tmp, tmp, addr, aq, rl) :
+ rv_sc_w(tmp, tmp, addr, aq, rl), ctx);
+ } else {
+ ctx->ex_sc_off = ctx->ninsns;
+ emit(is64 ? rv_sc_d(tmp, val, addr, aq, rl) :
+ rv_sc_w(tmp, val, addr, aq, rl), ctx);
+ }
+
+ /* Branch offset is relative to this instruction, back to the lr. */
+ jmp_off = ninsns_rvoff(lr_off - ctx->ninsns);
+ emit(rv_bne(tmp, RV_REG_ZERO, jmp_off >> 1), ctx);
+
+ if (fetch) {
+ emit_mv(val, old, ctx);
+ if (!is64)
+ emit_zextw(val, val, ctx);
+ }
+ ctx->ex_jmp_off = ctx->ninsns;
+}
+
+static bool is_atomic_lrsc_op(s32 imm)
+{
+ switch (imm) {
+ case BPF_ADD:
+ case BPF_AND:
+ case BPF_OR:
+ case BPF_XOR:
+ case BPF_ADD | BPF_FETCH:
+ case BPF_AND | BPF_FETCH:
+ case BPF_OR | BPF_FETCH:
+ case BPF_XOR | BPF_FETCH:
+ case BPF_XCHG:
+ return true;
+ }
+ return false;
+}
+
static int emit_atomic_rmw(u8 rd, u8 rs, const struct bpf_insn *insn,
struct rv_jit_context *ctx)
{
@@ -601,6 +674,12 @@ static int emit_atomic_rmw(u8 rd, u8 rs, const struct bpf_insn *insn,
rd = RV_REG_T1;
}
+ /* Without Zaamo, every op except cmpxchg is an lr/sc loop. */
+ if (!IS_ENABLED(CONFIG_RISCV_ISA_ZAAMO) && is_atomic_lrsc_op(imm)) {
+ emit_atomic_lrsc(rd, rs, is64, imm, ctx);
+ return 0;
+ }
+
switch (imm) {
/* lock *(u32/u64 *)(dst_reg + off16) <op>= src_reg */
case BPF_ADD:
@@ -732,8 +811,8 @@ bool ex_handler_bpf(const struct exception_table_entry *ex,
}
/* For accesses to BTF pointers, add an entry to the exception table */
-static int add_exception_handler(const struct bpf_insn *insn, int dst_reg,
- struct rv_jit_context *ctx)
+static int add_exception_handler_one(const struct bpf_insn *insn, int dst_reg,
+ struct rv_jit_context *ctx)
{
struct exception_table_entry *ex;
unsigned long pc;
@@ -804,6 +883,31 @@ static int add_exception_handler(const struct bpf_insn *insn, int dst_reg,
return 0;
}
+/*
+ * An lr/sc pair faults on either instruction. Both entries share the fixup
+ * that follows the loop, so a probe fault skips the retry and the result move.
+ */
+static int add_exception_handler(const struct bpf_insn *insn, int dst_reg,
+ struct rv_jit_context *ctx)
+{
+ int sc_off = ctx->ex_sc_off;
+ int jmp_off = ctx->ex_jmp_off;
+ int ret;
+
+ ctx->ex_sc_off = 0;
+ /* Offset discovery runs before the image exists. Entries are added later. */
+ if (!ctx->insns)
+ return 0;
+
+ ret = add_exception_handler_one(insn, dst_reg, ctx);
+ if (ret || sc_off <= 0)
+ return ret;
+
+ ctx->ex_insn_off = sc_off;
+ ctx->ex_jmp_off = jmp_off;
+ return add_exception_handler_one(insn, dst_reg, ctx);
+}
+
static int gen_jump_or_nops(void *target, void *ip, u32 *insns, bool is_call)
{
s64 rvoff;
diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c
index cbfcd287ea16..cba0d34c3e6f 100644
--- a/arch/riscv/net/bpf_jit_core.c
+++ b/arch/riscv/net/bpf_jit_core.c
@@ -16,6 +16,34 @@
/* Number of iterations to try until offsets converge. */
#define NR_JIT_ITERATIONS 32
+/*
+ * The verifier reserves one extable slot per BPF_PROBE_ATOMIC. Without
+ * Zaamo a read-modify-write atomic is an lr/sc loop that can fault on
+ * either instruction, so reserve a second slot for the sc.
+ */
+static void rv_jit_add_lrsc_exentries(struct bpf_prog *prog)
+{
+ const struct bpf_insn *insn = prog->insnsi;
+ int i;
+
+ if (IS_ENABLED(CONFIG_RISCV_ISA_ZAAMO) || __riscv_xlen != 64)
+ return;
+
+ for (i = 0; i < prog->len; i++, insn++) {
+ if (BPF_CLASS(insn->code) != BPF_STX ||
+ BPF_MODE(insn->code) != BPF_PROBE_ATOMIC ||
+ bpf_atomic_is_load_store(insn))
+ continue;
+ if (BPF_SIZE(insn->code) != BPF_W &&
+ BPF_SIZE(insn->code) != BPF_DW)
+ continue;
+ /* amocas is a single faulting instruction. */
+ if (insn->imm == BPF_CMPXCHG && rv_ext_enabled(ZACAS))
+ continue;
+ prog->aux->num_exentries++;
+ }
+}
+
static int build_body(struct rv_jit_context *ctx, bool extra_pass, int *offset)
{
const struct bpf_prog *prog = ctx->prog;
@@ -100,6 +128,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
if (ctx->ninsns == prev_ninsns) {
if (jit_data->header)
break;
+ rv_jit_add_lrsc_exentries(prog);
/* obtain the actual image size */
extable_size = prog->aux->num_exentries *
sizeof(struct exception_table_entry);
--
2.43.0
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH v6 6/6] riscv: hwprobe: report whether AMO is faster than LR/SC
2026-11-10 13:20 [PATCH v6 0/6] riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO Vladimir Kondratiev
` (4 preceding siblings ...)
2026-11-10 13:21 ` [PATCH v6 5/6] riscv, bpf: emit LR/SC loops when Zaamo is disabled Vladimir Kondratiev
@ 2026-11-10 13:21 ` Vladimir Kondratiev
5 siblings, 0 replies; 8+ messages in thread
From: Vladimir Kondratiev @ 2026-11-10 13:21 UTC (permalink / raw)
To: Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Will Deacon, Peter Zijlstra, Mark Rutland, Gary Guo, Yury Norov,
Rasmus Villemoes, Chao-ying Fu, Conor Dooley, torvalds, olof,
aleksa.paunovic, arikalo, Boqun Feng, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Song Liu,
Yonghong Song, Jiri Olsa, Emil Tsalapatis, Ihor Solodrai,
Luke Nelson, Xi Wang, Björn Töpel, Pu Lehui,
Puranjay Mohan, Masami Hiramatsu, Oleg Nesterov, Miguel Ojeda,
Björn Roy Baron, Benno Lossin, Andreas Hindborg, Alice Ryhl,
Trevor Gross, Danilo Krummrich, Daniel Almeida, Tamir Duberstein,
Alexandre Courbot, Onur Özkan, Shuah Khan, Randy Dunlap,
Boqun Feng
Cc: linux-riscv, linux-kernel, Vladimir.Kondratiev, bpf,
linux-trace-kernel, linux-doc, Jonathan Corbet, Charlie Jenkins,
Nam Cao, rust-for-linux, Vladimir Kondratiev
On a CPU that traps AMO instructions into firmware and emulates them
there, such as MIPS P8700, an uncontended AMO costs about 600 cycles
where the equivalent LR/SC sequence costs about 40. Userspace built for
the A extension uses AMO for every atomic and pays that on every one.
It cannot tell from the ISA string: such a CPU still has Zaamo, and
correctly so.
Give userspace the same kind of hint the kernel gives for misaligned
access. At boot, time AMO_SPEED_ITERS amoadd.w against the same number
of lr.w/addi/sc.w loops on each CPU, take the best sample from an 8 ms
window of each, and report the result through a new hwprobe key,
RISCV_HWPROBE_KEY_AMO_PERF:
RISCV_HWPROBE_AMO_FAST AMO is at least as fast as LR/SC
RISCV_HWPROBE_AMO_SLOW LR/SC is faster; use it where possible
RISCV_HWPROBE_AMO_UNSUPPORTED the CPU has no Zaamo
RISCV_HWPROBE_AMO_UNKNOWN not measured, or differs across CPUs
Only the direction of the difference is reported, not its size: the
question a program can act on is which sequence to pick. Each CPU logs
its ratio, and a kernel whose own atomics are AMO warns once when the
CPU turns out to be slow, pointing at RISCV_ISA_ZAAMO.
The amoadd.w is spelled with .insn through a new AMOADD_W() in
insn-def.h, so the file assembles when -march has no Zaamo, and the
probe is skipped on a CPU whose ISA lacks the extension. CPUs that come
online later are measured by a hotplug callback. amo_speed={slow|fast}
on the command line skips the measurement, as unaligned_scalar_speed=
does. The probe runs at the same initcall level as the misaligned
access probe, for the same reasons.
Tested on the eyeq7h hardware, with the DT advertising "a":
[ 0.163714] cpu10: AMO speed is 0.07x LR/SC speed (slow)
[ 0.163727] cpu11: AMO speed is 0.07x LR/SC speed (slow)
[ 0.163724] cpu6: AMO speed is 0.07x LR/SC speed (slow)
and, with CONFIG_RISCV_ISA_ZAAMO set:
[ 0.163729] AMO is slow but kernel atomics use it, consider disabling RISCV_ISA_ZAAMO
Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@mobileye.com>
---
Documentation/admin-guide/kernel-parameters.txt | 7 +
Documentation/arch/riscv/hwprobe.rst | 18 +++
arch/riscv/Kconfig | 13 ++
arch/riscv/include/asm/cpufeature.h | 4 +
arch/riscv/include/asm/hwprobe.h | 2 +-
arch/riscv/include/asm/insn-def.h | 5 +
arch/riscv/include/uapi/asm/hwprobe.h | 5 +
arch/riscv/kernel/Makefile | 1 +
arch/riscv/kernel/amo_speed.c | 183 ++++++++++++++++++++++++
arch/riscv/kernel/sys_hwprobe.c | 37 +++++
10 files changed, 274 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index e75344f4e0cd..9bc5baff20ff 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -513,6 +513,13 @@ Kernel parameters
Format: <a>,<b>
See also Documentation/input/joydev/joystick.rst
+ amo_speed= [RISCV]
+ Format: {slow | fast}
+ Allow skipping the AMO instruction speed test. This is
+ useful for testing alternative code paths and to skip
+ the test in environments where it runs too slowly. All
+ CPUs must have the same AMO speed.
+
analog.map= [HW,JOY] Analog joystick and gamepad support
Specifies type or capabilities of an analog joystick
connected to one of 16 gameports
diff --git a/Documentation/arch/riscv/hwprobe.rst b/Documentation/arch/riscv/hwprobe.rst
index bb1e0cbab36f..51ecb836f491 100644
--- a/Documentation/arch/riscv/hwprobe.rst
+++ b/Documentation/arch/riscv/hwprobe.rst
@@ -423,3 +423,21 @@ The following keys are defined:
* :c:macro:`RISCV_HWPROBE_EXT_ZA64RS`: The Za64rs extension is supported,
as defined in the RISC-V Profiles specification starting from commit
b1d80660 ("Updated to ratified state.")
+
+* :c:macro:`RISCV_HWPROBE_KEY_AMO_PERF`: An enum value describing the
+ performance of the AMO instructions of the Zaamo extension, relative to
+ the equivalent LR/SC sequence, on the selected set of processors.
+
+ * :c:macro:`RISCV_HWPROBE_AMO_UNKNOWN`: The performance of AMO instructions
+ is unknown.
+
+ * :c:macro:`RISCV_HWPROBE_AMO_SLOW`: An uncontended AMO instruction takes
+ longer than the equivalent LR/SC sequence. This is the case when AMO
+ instructions are trapped and emulated by firmware. Code that can choose
+ should use LR/SC.
+
+ * :c:macro:`RISCV_HWPROBE_AMO_FAST`: An uncontended AMO instruction is at
+ least as fast as the equivalent LR/SC sequence.
+
+ * :c:macro:`RISCV_HWPROBE_AMO_UNSUPPORTED`: AMO instructions are not
+ supported, the Zaamo extension is absent.
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 3c3679701a60..dfd6237f879b 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -764,6 +764,19 @@ config RISCV_ISA_ZAAMO
If you don't know what to do here, say Y.
+config RISCV_PROBE_AMO_SPEED
+ bool "Probe AMO instruction speed at boot"
+ default y
+ help
+ During boot, time an uncontended AMO instruction against the
+ equivalent LR/SC sequence on every CPU, and report which one is
+ faster to userspace through the hwprobe syscall, as
+ RISCV_HWPROBE_KEY_AMO_PERF. On a CPU that traps AMO instructions
+ into firmware and emulates them there, LR/SC is much faster, and
+ code that can choose should use it.
+
+ If you don't know what to do here, say Y.
+
config TOOLCHAIN_HAS_ZBB
bool
default y
diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h
index 5efa72823475..92cab4a7a0f1 100644
--- a/arch/riscv/include/asm/cpufeature.h
+++ b/arch/riscv/include/asm/cpufeature.h
@@ -83,6 +83,10 @@ static inline bool unaligned_ctl_available(void)
}
#endif
+#ifdef CONFIG_RISCV_PROBE_AMO_SPEED
+DECLARE_PER_CPU(long, amo_speed);
+#endif
+
#if defined(CONFIG_RISCV_MISALIGNED)
DECLARE_PER_CPU(long, misaligned_access_speed);
bool misaligned_traps_can_delegate(void);
diff --git a/arch/riscv/include/asm/hwprobe.h b/arch/riscv/include/asm/hwprobe.h
index 9b04377c0f98..b700852ac42f 100644
--- a/arch/riscv/include/asm/hwprobe.h
+++ b/arch/riscv/include/asm/hwprobe.h
@@ -8,7 +8,7 @@
#include <uapi/asm/hwprobe.h>
-#define RISCV_HWPROBE_MAX_KEY 16
+#define RISCV_HWPROBE_MAX_KEY 17
static inline bool riscv_hwprobe_key_is_valid(__s64 key)
{
diff --git a/arch/riscv/include/asm/insn-def.h b/arch/riscv/include/asm/insn-def.h
index 8fcc848c196e..5ca5c0b8ad66 100644
--- a/arch/riscv/include/asm/insn-def.h
+++ b/arch/riscv/include/asm/insn-def.h
@@ -219,6 +219,11 @@ ALTERNATIVE("nop", "fence rw, rw", SIFIVE_VENDOR_ID, \
__ASM_STR(.error "hlv.d requires 64-bit support")
#endif
+/* Spelled out so that it assembles when -march lacks Zaamo */
+#define AMOADD_W(dest, addr, val) \
+ INSN_R(OPCODE_AMO, FUNC3(2), FUNC7(0), \
+ RD(dest), RS1(addr), RS2(val))
+
#define LB_AQ(dest, addr) \
INSN_R(OPCODE_AMO, FUNC3(0), FUNC7(26), \
RD(dest), RS1(addr), __RS2(0))
diff --git a/arch/riscv/include/uapi/asm/hwprobe.h b/arch/riscv/include/uapi/asm/hwprobe.h
index e6a394fbe7fc..a5f726f94204 100644
--- a/arch/riscv/include/uapi/asm/hwprobe.h
+++ b/arch/riscv/include/uapi/asm/hwprobe.h
@@ -122,6 +122,11 @@ struct riscv_hwprobe {
#define RISCV_HWPROBE_EXT_ZICCIF _BITULL(3)
#define RISCV_HWPROBE_EXT_ZICCRSE _BITULL(4)
#define RISCV_HWPROBE_EXT_ZA64RS _BITULL(5)
+#define RISCV_HWPROBE_KEY_AMO_PERF 17
+#define RISCV_HWPROBE_AMO_UNKNOWN 0
+#define RISCV_HWPROBE_AMO_SLOW 1
+#define RISCV_HWPROBE_AMO_FAST 2
+#define RISCV_HWPROBE_AMO_UNSUPPORTED 3
/* Increase RISCV_HWPROBE_MAX_KEY when adding items. */
diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile
index ebe1c3588177..fc935c877e82 100644
--- a/arch/riscv/kernel/Makefile
+++ b/arch/riscv/kernel/Makefile
@@ -79,6 +79,7 @@ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o
obj-$(CONFIG_RISCV_MISALIGNED) += unaligned_access_speed.o
obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) += copy-unaligned.o
obj-$(CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS) += vec-copy-unaligned.o
+obj-$(CONFIG_RISCV_PROBE_AMO_SPEED) += amo_speed.o
obj-$(CONFIG_FPU) += fpu.o
obj-$(CONFIG_FPU) += kernel_mode_fpu.o
diff --git a/arch/riscv/kernel/amo_speed.c b/arch/riscv/kernel/amo_speed.c
new file mode 100644
index 000000000000..933891b69db6
--- /dev/null
+++ b/arch/riscv/kernel/amo_speed.c
@@ -0,0 +1,183 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Measure whether AMO instructions are faster than the equivalent LR/SC
+ * sequence. They are not on a CPU that traps them into firmware and
+ * emulates them there, such as MIPS P8700, where an AMO costs hundreds of
+ * cycles and LR/SC tens. The result goes to userspace through hwprobe,
+ * so code that can choose between the two knows which one to use.
+ */
+
+#include <linux/cpu.h>
+#include <linux/smp.h>
+#include <linux/timekeeping.h>
+#include <asm/cpufeature.h>
+#include <asm/hwprobe.h>
+#include <asm/insn-def.h>
+#include <asm/timex.h>
+
+/* Time window for each measurement, and atomics per sample within it */
+#define AMO_SPEED_NS 8000000
+#define AMO_SPEED_ITERS 1024
+
+DEFINE_PER_CPU(long, amo_speed) = RISCV_HWPROBE_AMO_UNKNOWN;
+
+static DEFINE_PER_CPU_ALIGNED(u32, amo_speed_word);
+static long amo_speed_param = RISCV_HWPROBE_AMO_UNKNOWN;
+
+static void amo_speed_amo(u32 *word, unsigned long n)
+{
+ unsigned long tmp;
+
+ asm volatile(
+ "1:\n"
+ AMOADD_W(%[tmp], %[word], %[one])
+ " addi %[n], %[n], -1\n"
+ " bnez %[n], 1b\n"
+ : [tmp] "=&r" (tmp), [n] "+&r" (n), "+m" (*word)
+ : [word] "r" (word), [one] "r" (1UL)
+ : "memory");
+}
+
+static void amo_speed_lrsc(u32 *word, unsigned long n)
+{
+ unsigned long tmp;
+
+ asm volatile(
+ "1: lr.w %[tmp], (%[word])\n"
+ " addi %[tmp], %[tmp], 1\n"
+ " sc.w %[tmp], %[tmp], (%[word])\n"
+ " bnez %[tmp], 1b\n"
+ " addi %[n], %[n], -1\n"
+ " bnez %[n], 1b\n"
+ : [tmp] "=&r" (tmp), [n] "+&r" (n), "+m" (*word)
+ : [word] "r" (word)
+ : "memory");
+}
+
+/*
+ * For a fixed amount of time, repeatedly run func and take the best time
+ * in cycles as the measurement.
+ */
+static u64 amo_speed_measure(void (*func)(u32 *word, unsigned long n),
+ u32 *word)
+{
+ u64 start_cycles, end_cycles, cycles = -1ULL;
+ u64 start_ns;
+
+ /* Do a warmup. */
+ func(word, AMO_SPEED_ITERS);
+
+ preempt_disable();
+
+ start_ns = ktime_get_mono_fast_ns();
+ while (ktime_get_mono_fast_ns() < start_ns + AMO_SPEED_NS) {
+ start_cycles = get_cycles64();
+ /* Ensure the CSR read can't reorder WRT to the atomics. */
+ mb();
+ func(word, AMO_SPEED_ITERS);
+ /* Ensure the atomics end before the end time is snapped. */
+ mb();
+ end_cycles = get_cycles64();
+ if ((end_cycles - start_cycles) < cycles)
+ cycles = end_cycles - start_cycles;
+ }
+
+ preempt_enable();
+
+ return cycles;
+}
+
+static void check_amo_speed(void *unused)
+{
+ int cpu = smp_processor_id();
+ u32 *word = this_cpu_ptr(&amo_speed_word);
+ u64 amo_cycles, lrsc_cycles;
+ long speed;
+ int ratio;
+
+ if (per_cpu(amo_speed, cpu) != RISCV_HWPROBE_AMO_UNKNOWN)
+ return;
+
+ if (!riscv_cpu_has_extension_unlikely(cpu, RISCV_ISA_EXT_ZAAMO)) {
+ per_cpu(amo_speed, cpu) = RISCV_HWPROBE_AMO_UNSUPPORTED;
+ return;
+ }
+
+ amo_cycles = amo_speed_measure(amo_speed_amo, word);
+ lrsc_cycles = amo_speed_measure(amo_speed_lrsc, word);
+
+ /* Don't divide by zero. */
+ if (!amo_cycles || !lrsc_cycles) {
+ pr_warn("cpu%d: rdtime lacks granularity needed to measure AMO speed\n",
+ cpu);
+ return;
+ }
+
+ if (amo_cycles <= lrsc_cycles)
+ speed = RISCV_HWPROBE_AMO_FAST;
+ else
+ speed = RISCV_HWPROBE_AMO_SLOW;
+
+ ratio = div64_u64(lrsc_cycles * 100, amo_cycles);
+ pr_info("cpu%d: AMO speed is %d.%02dx LR/SC speed (%s)\n",
+ cpu, ratio / 100, ratio % 100,
+ speed == RISCV_HWPROBE_AMO_FAST ? "fast" : "slow");
+
+ per_cpu(amo_speed, cpu) = speed;
+
+ if (speed == RISCV_HWPROBE_AMO_SLOW && IS_ENABLED(CONFIG_RISCV_ISA_ZAAMO))
+ pr_warn_once("AMO is slow but kernel atomics use it, consider disabling RISCV_ISA_ZAAMO\n");
+}
+
+static int amo_speed_online_cpu(unsigned int cpu)
+{
+ if (per_cpu(amo_speed, cpu) != RISCV_HWPROBE_AMO_UNKNOWN)
+ return 0;
+
+ if (amo_speed_param != RISCV_HWPROBE_AMO_UNKNOWN)
+ per_cpu(amo_speed, cpu) = amo_speed_param;
+ else
+ check_amo_speed(NULL);
+
+ return 0;
+}
+
+static int __init set_amo_speed_param(char *str)
+{
+ if (!strcmp(str, "slow"))
+ amo_speed_param = RISCV_HWPROBE_AMO_SLOW;
+ else if (!strcmp(str, "fast"))
+ amo_speed_param = RISCV_HWPROBE_AMO_FAST;
+ else
+ return -EINVAL;
+
+ return 1;
+}
+__setup("amo_speed=", set_amo_speed_param);
+
+static int __init check_amo_speed_all_cpus(void)
+{
+ int cpu;
+
+ if (amo_speed_param != RISCV_HWPROBE_AMO_UNKNOWN) {
+ pr_info("AMO speed set to '%s' (%lu) by command line\n",
+ amo_speed_param == RISCV_HWPROBE_AMO_FAST ? "fast" : "slow",
+ amo_speed_param);
+ for_each_online_cpu(cpu)
+ per_cpu(amo_speed, cpu) = amo_speed_param;
+ } else {
+ on_each_cpu(check_amo_speed, NULL, 1);
+ }
+
+ cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "riscv:amo_speed",
+ amo_speed_online_cpu, NULL);
+
+ return 0;
+}
+
+/*
+ * Same level as the unaligned access probe: after clocksource_done_booting()
+ * so that amo_speed_measure() uses a stable clocksource, but before
+ * rootfs_initcall() enables usermode helpers that can reach hwprobe.
+ */
+fs_initcall_sync(check_amo_speed_all_cpus);
diff --git a/arch/riscv/kernel/sys_hwprobe.c b/arch/riscv/kernel/sys_hwprobe.c
index 7818e1d32622..47836ecbd9af 100644
--- a/arch/riscv/kernel/sys_hwprobe.c
+++ b/arch/riscv/kernel/sys_hwprobe.c
@@ -257,6 +257,39 @@ static u64 hwprobe_misaligned(const struct cpumask *cpus)
}
#endif
+#ifdef CONFIG_RISCV_PROBE_AMO_SPEED
+static u64 hwprobe_amo(const struct cpumask *cpus)
+{
+ int cpu;
+ u64 perf = -1ULL;
+
+ for_each_cpu(cpu, cpus) {
+ int this_perf = per_cpu(amo_speed, cpu);
+
+ if (perf == -1ULL)
+ perf = this_perf;
+
+ if (perf != this_perf) {
+ perf = RISCV_HWPROBE_AMO_UNKNOWN;
+ break;
+ }
+ }
+
+ if (perf == -1ULL)
+ return RISCV_HWPROBE_AMO_UNKNOWN;
+
+ return perf;
+}
+#else
+static u64 hwprobe_amo(const struct cpumask *cpus)
+{
+ if (!hwprobe_ext0_has(cpus, RISCV_HWPROBE_EXT_ZAAMO))
+ return RISCV_HWPROBE_AMO_UNSUPPORTED;
+
+ return RISCV_HWPROBE_AMO_UNKNOWN;
+}
+#endif
+
#ifdef CONFIG_RISCV_VECTOR_MISALIGNED
static u64 hwprobe_vec_misaligned(const struct cpumask *cpus)
{
@@ -332,6 +365,10 @@ static void hwprobe_one_pair(struct riscv_hwprobe *pair,
pair->value = hwprobe_vec_misaligned(cpus);
break;
+ case RISCV_HWPROBE_KEY_AMO_PERF:
+ pair->value = hwprobe_amo(cpus);
+ break;
+
case RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE:
if (hwprobe_ext0_has(cpus, RISCV_HWPROBE_EXT_ZICBOZ))
pair->value = riscv_cboz_block_size;
--
2.43.0
^ permalink raw reply [flat|nested] 8+ messages in thread