* [PATCH v3 1/7] scripts/sorttable: Make the arm64 before_func offset configurable
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
@ 2026-10-29 16:53 ` Jose Fernandez (Anthropic)
2026-10-29 16:57 ` Steven Rostedt
2026-10-29 16:54 ` [PATCH v3 2/7] compiler_types: Let notrace keep the function prefix NOPs Jose Fernandez (Anthropic)
` (6 subsequent siblings)
7 siblings, 1 reply; 10+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-10-29 16:53 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Sami Tolvanen, Kees Cook, Alexei Starovoitov,
Daniel Borkmann, Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier
Cc: Ben Cressey, Florent Revest, Bill Wendling, linux-arm-kernel,
linux-kernel, linux-trace-kernel, bpf, Jose Fernandez (Anthropic)
sorttable hardcodes before_func to 8 bytes on arm64 (two prefix NOPs).
Make it configurable from CONFIG_ARM64_FUNCTION_PREFIX_NOPS instead.
kCFI kernels with CALL_OPS need three NOPs. The symbol does not exist
yet, so no functional change.
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
scripts/Makefile | 4 ++++
scripts/sorttable.c | 7 ++++++-
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile b/scripts/Makefile
index d7fbb7ca098b..7ef1c84cf79d 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -54,6 +54,10 @@ endif
ifdef CONFIG_BUILDTIME_MCOUNT_SORT
HOSTCFLAGS_sorttable.o += -DMCOUNT_SORT_ENABLED
+# 0 or unset (no prefix NOPs) leaves sorttable's built-in default in place.
+ifneq ($(filter-out 0,$(CONFIG_ARM64_FUNCTION_PREFIX_NOPS)),)
+HOSTCFLAGS_sorttable.o += -DFUNCTION_PREFIX_BYTES=$(shell expr 4 \* $(CONFIG_ARM64_FUNCTION_PREFIX_NOPS))
+endif
endif
hostprogs += gen_init_cpio
diff --git a/scripts/sorttable.c b/scripts/sorttable.c
index 88e49a5c4251..2e951f2f60e8 100644
--- a/scripts/sorttable.c
+++ b/scripts/sorttable.c
@@ -281,6 +281,11 @@ static int add_field(uint64_t addr, uint64_t size)
/* Used for when mcount/fentry is before the function entry */
static int before_func;
+/* How far before the function entry a patchable entry may be recorded */
+#ifndef FUNCTION_PREFIX_BYTES
+#define FUNCTION_PREFIX_BYTES 8
+#endif
+
/* Only return match if the address lies inside the function size */
static int cmp_func_addr(const void *K, const void *A)
{
@@ -912,7 +917,7 @@ static int do_file(char const *const fname, void *addr)
/* fallthrough */
case EM_RISCV:
/* arm64 and RISC-V place patchable entries before the function. */
- before_func = 8;
+ before_func = FUNCTION_PREFIX_BYTES;
#else
case EM_AARCH64:
case EM_RISCV:
--
2.54.0
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 1/7] scripts/sorttable: Make the arm64 before_func offset configurable
2026-10-29 16:53 ` [PATCH v3 1/7] scripts/sorttable: Make the arm64 before_func offset configurable Jose Fernandez (Anthropic)
@ 2026-10-29 16:57 ` Steven Rostedt
0 siblings, 0 replies; 10+ messages in thread
From: Steven Rostedt @ 2026-10-29 16:57 UTC (permalink / raw)
To: Jose Fernandez (Anthropic)
Cc: Catalin Marinas, Will Deacon, Mark Rutland, Masami Hiramatsu,
Sami Tolvanen, Kees Cook, Alexei Starovoitov, Daniel Borkmann,
Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda, Nathan Chancellor,
Nicolas Schier, Ben Cressey, Florent Revest, Bill Wendling,
linux-arm-kernel, linux-kernel, linux-trace-kernel, bpf
On Thu, 29 Oct 2026 16:53:58 +0000
"Jose Fernandez (Anthropic)" <jose.fernandez@linux.dev> wrote:
> sorttable hardcodes before_func to 8 bytes on arm64 (two prefix NOPs).
> Make it configurable from CONFIG_ARM64_FUNCTION_PREFIX_NOPS instead.
> kCFI kernels with CALL_OPS need three NOPs. The symbol does not exist
> yet, so no functional change.
>
> Reviewed-by: Ben Cressey <ben@cressey.dev>
> Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
> Acked-by: Alexei Starovoitov <ast@kernel.org>
> Assisted-by: LLM
> Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
-- Steve
> ---
> scripts/Makefile | 4 ++++
> scripts/sorttable.c | 7 ++++++-
> 2 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile b/scripts/Makefile
> index d7fbb7ca098b..7ef1c84cf79d 100644
> --- a/scripts/Makefile
> +++ b/scripts/Makefile
> @@ -54,6 +54,10 @@ endif
>
> ifdef CONFIG_BUILDTIME_MCOUNT_SORT
> HOSTCFLAGS_sorttable.o += -DMCOUNT_SORT_ENABLED
> +# 0 or unset (no prefix NOPs) leaves sorttable's built-in default in place.
> +ifneq ($(filter-out 0,$(CONFIG_ARM64_FUNCTION_PREFIX_NOPS)),)
> +HOSTCFLAGS_sorttable.o += -DFUNCTION_PREFIX_BYTES=$(shell expr 4 \* $(CONFIG_ARM64_FUNCTION_PREFIX_NOPS))
> +endif
> endif
>
> hostprogs += gen_init_cpio
> diff --git a/scripts/sorttable.c b/scripts/sorttable.c
> index 88e49a5c4251..2e951f2f60e8 100644
> --- a/scripts/sorttable.c
> +++ b/scripts/sorttable.c
> @@ -281,6 +281,11 @@ static int add_field(uint64_t addr, uint64_t size)
> /* Used for when mcount/fentry is before the function entry */
> static int before_func;
>
> +/* How far before the function entry a patchable entry may be recorded */
> +#ifndef FUNCTION_PREFIX_BYTES
> +#define FUNCTION_PREFIX_BYTES 8
> +#endif
> +
> /* Only return match if the address lies inside the function size */
> static int cmp_func_addr(const void *K, const void *A)
> {
> @@ -912,7 +917,7 @@ static int do_file(char const *const fname, void *addr)
> /* fallthrough */
> case EM_RISCV:
> /* arm64 and RISC-V place patchable entries before the function. */
> - before_func = 8;
> + before_func = FUNCTION_PREFIX_BYTES;
> #else
> case EM_AARCH64:
> case EM_RISCV:
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 2/7] compiler_types: Let notrace keep the function prefix NOPs
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
2026-10-29 16:53 ` [PATCH v3 1/7] scripts/sorttable: Make the arm64 before_func offset configurable Jose Fernandez (Anthropic)
@ 2026-10-29 16:54 ` Jose Fernandez (Anthropic)
2026-10-29 16:55 ` [PATCH v3 3/7] arm64: ftrace: Make the CALL_OPS prefix layout configurable Jose Fernandez (Anthropic)
` (5 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-10-29 16:54 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Sami Tolvanen, Kees Cook, Alexei Starovoitov,
Daniel Borkmann, Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier
Cc: Ben Cressey, Florent Revest, Bill Wendling, linux-arm-kernel,
linux-kernel, linux-trace-kernel, bpf, Jose Fernandez (Anthropic)
Let notrace keep the prefix NOPs when the architecture defines
CC_USING_PATCHABLE_FUNCTION_PREFIX. kCFI callers check the type hash
at a fixed offset before the entry point, so a notrace function needs
the same prefix NOPs as any other. Record the entry in a section the
linker drops, so ftrace never sees it. Clang 21 takes a section name
as a third argument of the attribute. Nothing defines the macro yet,
so no functional change.
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
include/linux/compiler_types.h | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index c5921f139007..43dafaa06194 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -211,6 +211,15 @@ struct ftrace_likely_data {
#if defined(CC_USING_HOTPATCH)
#define notrace __attribute__((hotpatch(0, 0)))
+#elif defined(CC_USING_PATCHABLE_FUNCTION_PREFIX) && !defined(__BINDGEN__)
+/*
+ * Not for bindgen: its libclang may not support the section argument
+ * (Clang >= 21), and leaving the attribute out does not change the ABI.
+ */
+#define notrace \
+ __attribute__((patchable_function_entry(CC_USING_PATCHABLE_FUNCTION_PREFIX, \
+ CC_USING_PATCHABLE_FUNCTION_PREFIX, \
+ ".discard.patchable_function_entries")))
#elif defined(CC_USING_PATCHABLE_FUNCTION_ENTRY)
#define notrace __attribute__((patchable_function_entry(0, 0)))
#else
--
2.54.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 3/7] arm64: ftrace: Make the CALL_OPS prefix layout configurable
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
2026-10-29 16:53 ` [PATCH v3 1/7] scripts/sorttable: Make the arm64 before_func offset configurable Jose Fernandez (Anthropic)
2026-10-29 16:54 ` [PATCH v3 2/7] compiler_types: Let notrace keep the function prefix NOPs Jose Fernandez (Anthropic)
@ 2026-10-29 16:55 ` Jose Fernandez (Anthropic)
2026-10-29 16:56 ` [PATCH v3 4/7] arm64: cfi: Use CONFIG_ARM64_FUNCTION_PREFIX_NOPS for the type hash offset Jose Fernandez (Anthropic)
` (4 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-10-29 16:55 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Sami Tolvanen, Kees Cook, Alexei Starovoitov,
Daniel Borkmann, Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier
Cc: Ben Cressey, Florent Revest, Bill Wendling, linux-arm-kernel,
linux-kernel, linux-trace-kernel, bpf, Jose Fernandez (Anthropic)
The arm64 ftrace code assumes two NOPs before every function in
several places. Replace the hardcoded count with a Kconfig value named
CONFIG_ARM64_FUNCTION_PREFIX_NOPS that defaults to 2. Compute the ops
literal address with one formula in both the assembly that reads it
and the C code that writes it. kCFI kernels with CALL_OPS need three
NOPs, or five with ThinLTO and BTI, and the reader and the writer stay
in sync either way.
Also make ftrace_call_adjust() look at each site's bytes and refuse
the ones that do not match the expected layout instead of patching
them. The two instructions to patch must be NOPs and the literal must
fall within the prefix NOPs. Log a refused site and leave it untraced.
No functional change for the layouts compilers generate today.
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
arch/arm64/Kconfig | 5 ++
arch/arm64/include/asm/ftrace.h | 16 ++++++
arch/arm64/kernel/entry-ftrace.S | 16 +++---
arch/arm64/kernel/ftrace.c | 120 ++++++++++++++++++++++-----------------
4 files changed, 98 insertions(+), 59 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index cfef22aee709..aa7d5e1859af 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -271,6 +271,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS
def_bool CC_IS_GCC
depends on $(cc-option,-fpatchable-function-entry=2)
+config ARM64_FUNCTION_PREFIX_NOPS
+ int
+ default 2 if DYNAMIC_FTRACE_WITH_CALL_OPS
+ default 0
+
config 64BIT
def_bool y
diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h
index 1621c84f44b3..5a7a515bf7cd 100644
--- a/arch/arm64/include/asm/ftrace.h
+++ b/arch/arm64/include/asm/ftrace.h
@@ -24,6 +24,22 @@
#define FTRACE_PLT_IDX 0
#define NR_FTRACE_PLTS 1
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
+/*
+ * ftrace_caller reads the callsite's ftrace_ops literal at
+ * ALIGN_DOWN(LR, 8) - FTRACE_CALL_OPS_BIAS. See ftrace_call_ops_adjust()
+ * for the layouts this serves.
+ */
+#define FTRACE_PREFIX_NOPS CONFIG_ARM64_FUNCTION_PREFIX_NOPS
+#if FTRACE_PREFIX_NOPS == 2 || FTRACE_PREFIX_NOPS == 3
+#define FTRACE_CALL_OPS_BIAS 16
+#elif FTRACE_PREFIX_NOPS == 5
+#define FTRACE_CALL_OPS_BIAS 24
+#else
+#error "Unsupported number of function prefix NOPs"
+#endif
+#endif /* CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS */
+
/*
* Currently, gcc tends to save the link register after the local variables
* on the stack. This causes the max stack tracer to report the function
diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S
index 025140caafe7..95727fee3cc5 100644
--- a/arch/arm64/kernel/entry-ftrace.S
+++ b/arch/arm64/kernel/entry-ftrace.S
@@ -39,17 +39,17 @@ SYM_CODE_START(ftrace_caller)
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
/*
* The literal pointer to the ops is at an 8-byte aligned boundary
- * which is either 12 or 16 bytes before the BL instruction in the call
- * site. See ftrace_call_adjust() for details.
+ * before the BL instruction in the call site. See ftrace_call_adjust()
+ * and <asm/ftrace.h> for details.
*
- * Therefore here the LR points at `literal + 16` or `literal + 20`,
- * and we can find the address of the literal in either case by
- * aligning to an 8-byte boundary and subtracting 16. We do the
- * alignment first as this allows us to fold the subtraction into the
- * LDR.
+ * Therefore here the LR points at `literal + FTRACE_CALL_OPS_BIAS` or
+ * `literal + FTRACE_CALL_OPS_BIAS + 4`, and we can find the address of
+ * the literal in either case by aligning to an 8-byte boundary and
+ * subtracting FTRACE_CALL_OPS_BIAS. We do the alignment first as this
+ * allows us to fold the subtraction into the LDR.
*/
bic x11, x30, 0x7
- ldr x11, [x11, #-(4 * AARCH64_INSN_SIZE)] // op
+ ldr x11, [x11, #-FTRACE_CALL_OPS_BIAS] // op
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
/*
diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
index e1a3c0b3a051..0d922fa63f71 100644
--- a/arch/arm64/kernel/ftrace.c
+++ b/arch/arm64/kernel/ftrace.c
@@ -60,6 +60,73 @@ int ftrace_regs_query_register_offset(const char *name)
}
#endif
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
+/*
+ * The address of the ops literal for a callsite whose `BL <caller>` is at
+ * @pc, exactly as ftrace_caller derives it from the return address.
+ */
+static unsigned long ftrace_call_ops_literal(unsigned long pc)
+{
+ return ALIGN_DOWN(pc + AARCH64_INSN_SIZE, 8) - FTRACE_CALL_OPS_BIAS;
+}
+
+/*
+ * addr is the first of the FTRACE_PREFIX_NOPS NOPs before a function:
+ *
+ * addr-04: .word // kCFI type hash, address-taken and non-local only
+ * addr+00: NOP // FTRACE_PREFIX_NOPS of these, two hold the literal
+ * func+00: func: BTI C // only if BTI and the function may be called indirectly
+ * func+04: NOP // patched to MOV X9, LR
+ * func+08: NOP // patched to BL <caller>
+ *
+ * The compiler aligns whatever comes first to 8 bytes. ftrace_caller reads
+ * the literal at ALIGN_DOWN(LR, 8) - FTRACE_CALL_OPS_BIAS, with LR at
+ * func+8, or func+12 after a BTI:
+ *
+ * NOPs BIAS literal at
+ * 2 16 func - 8
+ * 3 16 func - 8, or func - 12 if func % 8 == 4 (no hash, no BTI)
+ * 5 24 func - 16, or func - 12 / func - 20 if func % 8 == 4
+ *
+ * Return the address of the NOP to be patched to BL, or 0 when the two
+ * NOPs are not there or the literal would fall outside the prefix.
+ */
+static unsigned long ftrace_call_ops_adjust(unsigned long addr)
+{
+ unsigned long func = addr + FTRACE_PREFIX_NOPS * AARCH64_INSN_SIZE;
+ unsigned long pc = func + AARCH64_INSN_SIZE;
+ unsigned long literal;
+ u32 nop = aarch64_insn_gen_nop();
+
+ if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) &&
+ aarch64_insn_is_bti(le32_to_cpu(*(__le32 *)func)))
+ pc += AARCH64_INSN_SIZE;
+
+ /* Objects built with only the prefix NOPs have nothing to patch. */
+ if (le32_to_cpu(*(__le32 *)(pc - AARCH64_INSN_SIZE)) != nop ||
+ le32_to_cpu(*(__le32 *)pc) != nop) {
+ pr_warn_ratelimited("ftrace: no patchable entry at %ps\n",
+ (void *)func);
+ return 0;
+ }
+
+ /* The literal would overlap the type hash or the entry point. */
+ literal = ftrace_call_ops_literal(pc);
+ if (literal < addr || literal + sizeof(u64) > func) {
+ pr_warn_ratelimited("ftrace: cannot trace %ps: no room for the ops literal\n",
+ (void *)func);
+ return 0;
+ }
+
+ return pc;
+}
+#else
+static unsigned long ftrace_call_ops_adjust(unsigned long addr)
+{
+ return 0;
+}
+#endif /* CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS */
+
unsigned long ftrace_call_adjust(unsigned long addr)
{
/*
@@ -91,56 +158,7 @@ unsigned long ftrace_call_adjust(unsigned long addr)
if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS))
return addr + AARCH64_INSN_SIZE;
- /*
- * When using patchable-function-entry with pre-function NOPs, addr is
- * the address of the first pre-function NOP.
- *
- * Starting from an 8-byte aligned base, the compiler has either
- * generated:
- *
- * addr+00: NOP // Literal (first 32 bits)
- * addr+04: NOP // Literal (last 32 bits)
- * addr+08: func: NOP // To be patched to MOV X9, LR
- * addr+12: NOP // To be patched to BL <caller>
- *
- * Or:
- *
- * addr+00: NOP // Literal (first 32 bits)
- * addr+04: NOP // Literal (last 32 bits)
- * addr+08: func: BTI C
- * addr+12: NOP // To be patched to MOV X9, LR
- * addr+16: NOP // To be patched to BL <caller>
- *
- * We must adjust addr to the address of the NOP which will be patched
- * to `BL <caller>`, which is at either addr+12 or addr+16 depending on
- * whether there is a BTI.
- */
-
- if (!IS_ALIGNED(addr, sizeof(unsigned long))) {
- WARN_RATELIMIT(1, "Misaligned patch-site %pS\n",
- (void *)(addr + 8));
- return 0;
- }
-
- /* Skip the NOPs placed before the function entry point */
- addr += 2 * AARCH64_INSN_SIZE;
-
- /* Skip any BTI */
- if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) {
- u32 insn = le32_to_cpu(*(__le32 *)addr);
-
- if (aarch64_insn_is_bti(insn)) {
- addr += AARCH64_INSN_SIZE;
- } else if (insn != aarch64_insn_gen_nop()) {
- WARN_RATELIMIT(1, "unexpected insn in patch-site %pS: 0x%08x\n",
- (void *)addr, insn);
- }
- }
-
- /* Skip the first NOP after function entry */
- addr += AARCH64_INSN_SIZE;
-
- return addr;
+ return ftrace_call_ops_adjust(addr);
}
/* Convert fentry_ip to the symbol address without kallsyms */
@@ -368,7 +386,7 @@ static const struct ftrace_ops *arm64_rec_get_ops(struct dyn_ftrace *rec)
static int ftrace_rec_set_ops(const struct dyn_ftrace *rec,
const struct ftrace_ops *ops)
{
- unsigned long literal = ALIGN_DOWN(rec->ip - 12, 8);
+ unsigned long literal = ftrace_call_ops_literal(rec->ip);
return aarch64_insn_write_literal_u64((void *)literal,
(unsigned long)ops);
}
--
2.54.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 4/7] arm64: cfi: Use CONFIG_ARM64_FUNCTION_PREFIX_NOPS for the type hash offset
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
` (2 preceding siblings ...)
2026-10-29 16:55 ` [PATCH v3 3/7] arm64: ftrace: Make the CALL_OPS prefix layout configurable Jose Fernandez (Anthropic)
@ 2026-10-29 16:56 ` Jose Fernandez (Anthropic)
2026-10-29 16:56 ` [PATCH v3 5/7] arm64: ftrace: Support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
` (3 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-10-29 16:56 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Sami Tolvanen, Kees Cook, Alexei Starovoitov,
Daniel Borkmann, Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier
Cc: Ben Cressey, Florent Revest, Bill Wendling, linux-arm-kernel,
linux-kernel, linux-trace-kernel, bpf, Jose Fernandez (Anthropic)
Make cfi_get_offset(), SYM_TYPED_FUNC_START and the BPF JIT use
CONFIG_ARM64_FUNCTION_PREFIX_NOPS instead of hardcoding four bytes. kCFI
kernels with CALL_OPS need three NOPs, or five with ThinLTO and BTI.
CONFIG_ARM64_FUNCTION_PREFIX_NOPS is 0 on every CFI kernel today, so no
functional change.
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
arch/arm64/include/asm/cfi.h | 11 +++++++++++
arch/arm64/include/asm/linkage.h | 11 +++++++++++
arch/arm64/net/bpf_jit_comp.c | 8 ++++++--
3 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/cfi.h b/arch/arm64/include/asm/cfi.h
index ab90f0351b7a..f3c04a2a6a95 100644
--- a/arch/arm64/include/asm/cfi.h
+++ b/arch/arm64/include/asm/cfi.h
@@ -2,6 +2,17 @@
#ifndef _ASM_ARM64_CFI_H
#define _ASM_ARM64_CFI_H
+#include <asm/insn-def.h>
+
#define __bpfcall
+#ifdef CONFIG_CFI
+static inline int cfi_get_offset(void)
+{
+ /* The 4-byte type hash sits before the prefix NOPs. */
+ return 4 + CONFIG_ARM64_FUNCTION_PREFIX_NOPS * AARCH64_INSN_SIZE;
+}
+#define cfi_get_offset cfi_get_offset
+#endif /* CONFIG_CFI */
+
#endif /* _ASM_ARM64_CFI_H */
diff --git a/arch/arm64/include/asm/linkage.h b/arch/arm64/include/asm/linkage.h
index d1f7a16729d2..14bc3c67c38b 100644
--- a/arch/arm64/include/asm/linkage.h
+++ b/arch/arm64/include/asm/linkage.h
@@ -39,6 +39,17 @@
SYM_START(name, SYM_L_WEAK, SYM_A_NONE) \
bti c ;
+/*
+ * The compiler emits CONFIG_ARM64_FUNCTION_PREFIX_NOPS NOPs between a C
+ * function's kCFI type hash and its entry point. Callers check the hash
+ * at that offset.
+ */
+#define __CFI_TYPE(name) \
+ .4byte __kcfi_typeid_##name ASM_NL \
+ .rept CONFIG_ARM64_FUNCTION_PREFIX_NOPS ASM_NL \
+ nop ASM_NL \
+ .endr
+
#define SYM_TYPED_FUNC_START(name) \
SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) \
bti c ;
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index aaa34c739004..42cb2d446680 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -185,8 +185,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx)
static inline void emit_kcfi(u32 hash, struct jit_ctx *ctx)
{
- if (IS_ENABLED(CONFIG_CFI))
- emit_u32_data(hash, ctx);
+ if (!IS_ENABLED(CONFIG_CFI))
+ return;
+
+ emit_u32_data(hash, ctx);
+ for (int i = 0; i < CONFIG_ARM64_FUNCTION_PREFIX_NOPS; i++)
+ emit(A64_NOP, ctx);
}
/*
--
2.54.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 5/7] arm64: ftrace: Support CALL_OPS on kernels built with kCFI
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
` (3 preceding siblings ...)
2026-10-29 16:56 ` [PATCH v3 4/7] arm64: cfi: Use CONFIG_ARM64_FUNCTION_PREFIX_NOPS for the type hash offset Jose Fernandez (Anthropic)
@ 2026-10-29 16:56 ` Jose Fernandez (Anthropic)
2026-10-29 16:57 ` [PATCH v3 6/7] arm64: ftrace: Use five prefix NOPs on ThinLTO kernels with BTI Jose Fernandez (Anthropic)
` (2 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-10-29 16:56 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Sami Tolvanen, Kees Cook, Alexei Starovoitov,
Daniel Borkmann, Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier
Cc: Ben Cressey, Florent Revest, Bill Wendling, linux-arm-kernel,
linux-kernel, linux-trace-kernel, bpf, Jose Fernandez (Anthropic)
CALL_OPS and kCFI have been mutually exclusive on arm64 since commit
baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS").
kCFI checks the type hash at a fixed offset before the entry point, so
every function needs the same number of prefix NOPs. Before clang 21,
prefix NOPs came only with an entry in ftrace's table, and untraced
functions had none. Since commit 9315e22b0c0a ("arm64: ftrace: allow
DIRECT_CALLS without CALL_OPS") a kCFI kernel can attach BPF
trampolines. But without CALL_OPS, a trampoline out of BL range is
reached through ftrace_caller, ~10 ns more per call than on a CALL_OPS
kernel, plus ~4 ns for every extra ftrace_ops registered.
Clang 21 added a third argument to -fpatchable-function-entry that names
the section the entries go to [1]. Point it at a section the linker
scripts already drop and a function gets prefix NOPs with no entry in
ftrace's table. When CFI and CALL_OPS are both enabled, compile every
object with
-fpatchable-function-entry=M,M,.discard.patchable_function_entries,
where M is CONFIG_ARM64_FUNCTION_PREFIX_NOPS.
Gate CALL_OPS under CFI on ARM64_CFI_PERMITS_CALL_OPS, which needs
clang 21 or newer and RUST off. rustc has its own flag for the prefix
NOPs and this patch does not pass it, so Rust functions would have no
prefix and a different hash offset.
M is 3 because the ops pointer needs an aligned 8-byte slot inside the
prefix NOPs, and the 4-byte hash puts the NOPs at offset 4. A traced
function, with the two entry NOPs from CC_FLAGS_FTRACE, then looks
like this:
func-16: .word <hash> // kCFI type hash
func-12: NOP
func-08: NOP // ops pointer, 8 bytes
func-04: NOP //
func+00: func: NOP // To be patched to MOV X9, LR
func+04: NOP // To be patched to BL <caller>
Size of .text in vmlinux on a defconfig-based config with clang 21.1.4:
Kernel || .text bytes | Delta | Delta
=======================++=============+=============+=======
kCFI, base || 21,023,888 | |
kCFI, M = 3 || 22,031,504 | +1,007,616 | +4.8%
-----------------------++-------------+-------------+-------
Numbers below are from a KVM guest on a Graviton4 host, v7.3-rc1 with
and without this series. The workload is a fentry program on
__arm64_sys_getpid, 20M calls per pass, six passes per boot, six
boots, each cell the call with the program attached minus the call
without it. The in-range rows are the same kernels booted with nokaslr.
Kernel | Trampoline | Extra || Per-call cost (ns)
| vs. BL range | ftrace_ops || Median | Min | Max
==================+==============+============++========+======+======
kCFI, base | out of range | 0 || 32.9 | 32.6 | 33.4
kCFI, base | out of range | 16 || 92.6 | 92.4 | 96.3
kCFI, base | in range | 0 || 21.3 | 21.0 | 21.8
------------------+--------------+------------++--------+------+------
kCFI, this series | out of range | 0 || 22.7 | 22.0 | 22.9
kCFI, this series | out of range | 16 || 22.1 | 21.9 | 22.4
kCFI, this series | in range | 0 || 22.4 | 22.1 | 22.6
------------------+--------------+------------++--------+------+------
Out of range this saves ~10 ns per call and the cost stops growing
with extra ftrace_ops. In range both kernels branch straight to
the trampoline and the numbers roughly match.
ftracetest and the BPF trampoline, struct_ops and kprobe_multi
selftests pass with CFI enforcing.
[1] https://github.com/llvm/llvm-project/pull/131230
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
arch/arm64/Kconfig | 15 ++++++++++++++-
arch/arm64/Makefile | 12 +++++++++++-
arch/arm64/kernel/pi/Makefile | 2 ++
arch/arm64/kernel/vdso/Makefile | 2 +-
drivers/firmware/efi/libstub/Makefile | 2 +-
5 files changed, 29 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index aa7d5e1859af..6715adcae38b 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -190,7 +190,8 @@ config ARM64
select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
if DYNAMIC_FTRACE_WITH_ARGS
select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
- if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \
+ if (DYNAMIC_FTRACE_WITH_ARGS && \
+ (!CFI || ARM64_CFI_PERMITS_CALL_OPS) && \
(CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY \
if DYNAMIC_FTRACE_WITH_ARGS
@@ -271,8 +272,20 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS
def_bool CC_IS_GCC
depends on $(cc-option,-fpatchable-function-entry=2)
+config CC_HAS_PATCHABLE_FUNCTION_ENTRY_SECTION
+ # The section argument needs clang 21 or later:
+ # https://github.com/llvm/llvm-project/pull/131230
+ def_bool $(cc-option,-fpatchable-function-entry=1$(comma)1$(comma).discard.patchable_function_entries)
+
+config ARM64_CFI_PERMITS_CALL_OPS
+ def_bool y
+ depends on CFI
+ depends on CC_HAS_PATCHABLE_FUNCTION_ENTRY_SECTION
+ depends on !RUST
+
config ARM64_FUNCTION_PREFIX_NOPS
int
+ default 3 if DYNAMIC_FTRACE_WITH_CALL_OPS && CFI
default 2 if DYNAMIC_FTRACE_WITH_CALL_OPS
default 0
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index ed28f94be428..9fbfe5ab3724 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -137,7 +137,17 @@ CHECKFLAGS += -D__aarch64__
ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS),y)
KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY
- CC_FLAGS_FTRACE := -fpatchable-function-entry=4,2
+ prefix_nops := $(CONFIG_ARM64_FUNCTION_PREFIX_NOPS)
+ CC_FLAGS_FTRACE := -fpatchable-function-entry=$(shell expr $(prefix_nops) + 2),$(prefix_nops)
+ ifeq ($(CONFIG_CFI),y)
+ # Every object gets the prefix NOPs and no ftrace entry by default.
+ # CC_FLAGS_FTRACE comes later for traced objects and overrides the
+ # -fpatchable-function-entry.
+ CFI_PREFIX_CFLAGS := -fpatchable-function-entry=$(prefix_nops),$(prefix_nops),.discard.patchable_function_entries \
+ -DCC_USING_PATCHABLE_FUNCTION_PREFIX=$(prefix_nops)
+ KBUILD_CFLAGS += $(CFI_PREFIX_CFLAGS)
+ export CFI_PREFIX_CFLAGS
+ endif
else ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_ARGS),y)
KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY
CC_FLAGS_FTRACE := -fpatchable-function-entry=2
diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile
index a273ab5af805..0ccae4ff00a0 100644
--- a/arch/arm64/kernel/pi/Makefile
+++ b/arch/arm64/kernel/pi/Makefile
@@ -18,6 +18,8 @@ CFLAGS_map_range.o += -mstrict-align
KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))
# disable LTO
KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS))
+# the discard section holds absolute relocations, which relacheck rejects
+KBUILD_CFLAGS := $(filter-out $(CFI_PREFIX_CFLAGS), $(KBUILD_CFLAGS))
hostprogs := relacheck
diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index f6619e1cb2ce..334ea3fac812 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -38,7 +38,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO
CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) \
$(RANDSTRUCT_CFLAGS) $(KSTACK_ERASE_CFLAGS) \
$(GCC_PLUGINS_CFLAGS) \
- $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) \
+ $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(CFI_PREFIX_CFLAGS) \
-Wmissing-prototypes -Wmissing-declarations
CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables $(CONFIG_CC_OPT_INLINE_MEMSET)
diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile
index bbab12b8eb4b..5bbb57c64fe0 100644
--- a/drivers/firmware/efi/libstub/Makefile
+++ b/drivers/firmware/efi/libstub/Makefile
@@ -54,7 +54,7 @@ KBUILD_CFLAGS := $(filter-out $(RANDSTRUCT_CFLAGS), $(KBUILD_CFLAGS))
# remove SCS flags from all objects in this directory
KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))
# disable CFI
-KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_CFI), $(KBUILD_CFLAGS))
+KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_CFI) $(CFI_PREFIX_CFLAGS), $(KBUILD_CFLAGS))
# disable LTO
KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS))
--
2.54.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 6/7] arm64: ftrace: Use five prefix NOPs on ThinLTO kernels with BTI
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
` (4 preceding siblings ...)
2026-10-29 16:56 ` [PATCH v3 5/7] arm64: ftrace: Support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
@ 2026-10-29 16:57 ` Jose Fernandez (Anthropic)
2026-10-29 16:57 ` [PATCH v3 7/7] arm64: ftrace: Allow CALL_OPS on kCFI kernels built with Rust Jose Fernandez (Anthropic)
2026-10-29 21:58 ` [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Sami Tolvanen
7 siblings, 0 replies; 10+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-10-29 16:57 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Sami Tolvanen, Kees Cook, Alexei Starovoitov,
Daniel Borkmann, Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier
Cc: Ben Cressey, Florent Revest, Bill Wendling, linux-arm-kernel,
linux-kernel, linux-trace-kernel, bpf, Jose Fernandez (Anthropic)
Default CONFIG_ARM64_FUNCTION_PREFIX_NOPS to 5 on ThinLTO kernels with
BTI. Many functions there have a landing pad but no hash. That moves the
entry point 4 bytes. The 8-byte slot lands on the landing pad and ftrace
refuses the site. With five NOPs every function has a slot.
Commit d3359af21fc9e ("arm64: bti: Disable in-kernel BTI with recent
versions of Clang") keeps BTI off on clang 21 or later, so this default
does not trigger on a mainline build today. Josh Poimboeuf's series to
bring it back is under review [1].
With five NOPs the series costs 7.4% of .text on a defconfig-based
ThinLTO kernel with clang 21.1.4:
Kernel || .text bytes | Delta | Delta
=======================++=============+=============+=======
ThinLTO kCFI, base || 21,466,256 | |
ThinLTO kCFI, M = 5 || 23,047,312 | +1,581,056 | +7.4%
-----------------------++-------------+-------------+-------
ftracetest results match with and without this series. The three
failures on a ThinLTO kernel are pre-existing. The BPF trampoline,
struct_ops and kprobe_multi selftests pass with CFI enforcing.
[1] https://lore.kernel.org/all/cover.1786768375.git.jpoimboe@kernel.org/
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
arch/arm64/Kconfig | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6715adcae38b..e8cb44d881d0 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -285,6 +285,8 @@ config ARM64_CFI_PERMITS_CALL_OPS
config ARM64_FUNCTION_PREFIX_NOPS
int
+ default 5 if DYNAMIC_FTRACE_WITH_CALL_OPS && CFI && \
+ (LTO_CLANG_THIN || LTO_CLANG_THIN_DIST) && ARM64_BTI_KERNEL
default 3 if DYNAMIC_FTRACE_WITH_CALL_OPS && CFI
default 2 if DYNAMIC_FTRACE_WITH_CALL_OPS
default 0
--
2.54.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v3 7/7] arm64: ftrace: Allow CALL_OPS on kCFI kernels built with Rust
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
` (5 preceding siblings ...)
2026-10-29 16:57 ` [PATCH v3 6/7] arm64: ftrace: Use five prefix NOPs on ThinLTO kernels with BTI Jose Fernandez (Anthropic)
@ 2026-10-29 16:57 ` Jose Fernandez (Anthropic)
2026-10-29 21:58 ` [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Sami Tolvanen
7 siblings, 0 replies; 10+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-10-29 16:57 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Sami Tolvanen, Kees Cook, Alexei Starovoitov,
Daniel Borkmann, Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier
Cc: Ben Cressey, Florent Revest, Bill Wendling, linux-arm-kernel,
linux-kernel, linux-trace-kernel, bpf, Jose Fernandez (Anthropic)
Pass -Zpatchable-function-entry=M,M,.discard.patchable_function_entries
to rustc when CFI and CALL_OPS are both enabled, so Rust functions get
the same prefix NOPs as C functions. The section argument needs
Rust 1.98 [1]. Probe for it, and let ARM64_CFI_PERMITS_CALL_OPS
accept RUST=y when the probe passes.
Filter -fpatchable-function-entry out of bindgen's C flags on every
architecture, not only on arm64 with CFI, since removing the flag does
not change the ABI.
[1] https://github.com/rust-lang/rust/pull/157445
Suggested-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Ben Cressey <ben@cressey.dev>
Reviewed-by: Florent Revest (Anthropic) <florent.revest@linux.dev>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Assisted-by: LLM
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
arch/arm64/Kconfig | 7 ++++++-
arch/arm64/Makefile | 3 +++
rust/Makefile | 7 ++++++-
3 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index e8cb44d881d0..e64d053e1854 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -277,11 +277,16 @@ config CC_HAS_PATCHABLE_FUNCTION_ENTRY_SECTION
# https://github.com/llvm/llvm-project/pull/131230
def_bool $(cc-option,-fpatchable-function-entry=1$(comma)1$(comma).discard.patchable_function_entries)
+config RUSTC_HAS_PATCHABLE_FUNCTION_ENTRY_SECTION
+ # The section argument needs Rust 1.98.0 or later:
+ # https://github.com/rust-lang/rust/pull/157445
+ def_bool $(rustc-option,-Zpatchable-function-entry=1$(comma)1$(comma).discard.patchable_function_entries)
+
config ARM64_CFI_PERMITS_CALL_OPS
def_bool y
depends on CFI
depends on CC_HAS_PATCHABLE_FUNCTION_ENTRY_SECTION
- depends on !RUST
+ depends on !RUST || RUSTC_HAS_PATCHABLE_FUNCTION_ENTRY_SECTION
config ARM64_FUNCTION_PREFIX_NOPS
int
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 9fbfe5ab3724..b47733b585b5 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -147,6 +147,9 @@ ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS),y)
-DCC_USING_PATCHABLE_FUNCTION_PREFIX=$(prefix_nops)
KBUILD_CFLAGS += $(CFI_PREFIX_CFLAGS)
export CFI_PREFIX_CFLAGS
+ # Rust functions are not patched by ftrace, but they are called indirectly
+ # from C and make indirect calls themselves, so they need the same prefix.
+ KBUILD_RUSTFLAGS += -Zpatchable-function-entry=$(prefix_nops),$(prefix_nops),.discard.patchable_function_entries
endif
else ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_ARGS),y)
KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY
diff --git a/rust/Makefile b/rust/Makefile
index 0ac9ab28cc2b..686964c826f7 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -511,7 +511,12 @@ endif
# prototypes for functions like `memcpy` -- if this flag is not passed,
# `bindgen`-generated prototypes use `c_ulong` or `c_uint` depending on
# architecture instead of generating `usize`.
-bindgen_c_flags_final = $(bindgen_c_flags_lto) -fno-builtin -D__BINDGEN__
+#
+# Do not pass -fpatchable-function-entry to bindgen: its libclang may not
+# support the section argument (Clang >= 21), and removing the flag does not
+# change the ABI.
+bindgen_c_flags_final = $(filter-out -fpatchable-function-entry=%, \
+ $(bindgen_c_flags_lto)) -fno-builtin -D__BINDGEN__
# `--rust-target` points to our minimum supported Rust version.
quiet_cmd_bindgen = BINDGEN $@
--
2.54.0
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI
2026-10-29 16:53 [PATCH v3 0/7] arm64: ftrace: support CALL_OPS on kernels built with kCFI Jose Fernandez (Anthropic)
` (6 preceding siblings ...)
2026-10-29 16:57 ` [PATCH v3 7/7] arm64: ftrace: Allow CALL_OPS on kCFI kernels built with Rust Jose Fernandez (Anthropic)
@ 2026-10-29 21:58 ` Sami Tolvanen
7 siblings, 0 replies; 10+ messages in thread
From: Sami Tolvanen @ 2026-10-29 21:58 UTC (permalink / raw)
To: Jose Fernandez (Anthropic)
Cc: Catalin Marinas, Will Deacon, Mark Rutland, Steven Rostedt,
Masami Hiramatsu, Kees Cook, Alexei Starovoitov, Daniel Borkmann,
Puranjay Mohan, Ard Biesheuvel, Miguel Ojeda, Nathan Chancellor,
Nicolas Schier, Ben Cressey, Florent Revest, Bill Wendling,
linux-arm-kernel, linux-kernel, linux-trace-kernel, bpf
Hi Jose,
On Thu, Oct 29, 2026 at 9:53 AM Jose Fernandez (Anthropic)
<jose.fernandez@linux.dev> wrote:
>
> CALL_OPS and kCFI have been mutually exclusive on arm64 since commit
> baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS").
> kCFI checks the type hash at a fixed offset before the entry point, so
> every function needs the same number of prefix NOPs. Before clang 21 the
> compiler could not emit prefix NOPs without also adding the function to
> ftrace's table. Since commit 9315e22b0c0a ("arm64: ftrace: allow
> DIRECT_CALLS without CALL_OPS") a kCFI kernel can attach BPF
> trampolines. A trampoline out of BL range is reached through
> ftrace_caller. That path costs ~10 ns more per call than on a CALL_OPS
> kernel, plus ~4 ns for every extra ftrace_ops registered. Clang 21 takes
> a section name as a third argument of -fpatchable-function-entry [1].
> This series uses it to give every function three prefix NOPs while
> keeping untraced functions out of ftrace's table, and enables CALL_OPS
> under kCFI. The per-call cost drops by ~10 ns and stops growing with the
> number of ftrace_ops registered.
Thanks for implementing this! For the series:
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Sami
^ permalink raw reply [flat|nested] 10+ messages in thread