[PATCH v2 00/16] Move notrace from inline to __always_inline

[PATCH v2 00/16] Move notrace from inline to __always_inline

[Song Liu]

Since commit 45959ee7aa64 ("ftrace: Do not function trace inlined
functions"), every function marked inline is also notrace. When the
compiler does not inline such a function, the out-of-line copy cannot
be traced with ftrace, cannot be a BPF fentry/fexit target, and cannot
be livepatched. This series moves notrace from inline to
__always_inline, so these copies become traceable. __always_inline
functions, which are mostly low-level helpers, stay untraced.

v1 [1] removed notrace from inline in one patch. Sashiko's review
showed that some code relies on inline implying notrace, and Steven
suggested keeping notrace on __always_inline and annotating the rest.
Patches 1-15 add notrace explicitly to the inline helpers that need it.
While inline still implies notrace, they do not change the generated
code: on x86_64 with gcc 11.5, vmlinux .text is byte-identical with and
without them.

 - 1-2: x86 paravirt native helpers, and the helpers used by resume and
   kexec.
 - 3-6: helpers used by bpf_spin_lock, the BPF trampoline and
   bpf_mem_alloc, where an fentry program on one of them could
   deadlock, recurse without bound or corrupt the free lists.
 - 7-8: tracepoint and SRCU helpers used by tracing.
 - 9-12: helpers used by ftrace recursion protection: the accessors
   behind preempt_count() and current on arm64, powerpc, csky, mips and
   sh, the Tiny RCU rcu_is_watching(), and the tracing_gen_ctx helpers.
 - 13-15: helpers called by notrace code in the trace clocks,
   core_kernel_text() and the nofault accessors.

To find these, I compared x86_64 builds with and without patch 16, and
listed the functions that become traceable and are called from code
without an fentry call (notrace, noinstr, or built without -pg),
including calls through pv_ops. To also cover helpers that gcc happens
to inline today, I repeated this with -fno-inline. The changes for
other architectures and Tiny RCU come from reading the code that ftrace
recursion protection uses.

On an x86_64 config with CONFIG_PARAVIRT_XXL and gcc 11.5, the series
makes 678 more functions traceable, and none become untraceable. In
QEMU, with ftrace filtered to native_load_idt(), a kernel with v1 dies
on resume from S3, as Sashiko predicted. With this series, S3
suspend/resume, CPU hotplug and hibernation test mode pass with ftrace
on all native_* functions, and repeated kexec passes with ftrace on all
functions. The series also builds for i386, and for arm64 and powerpc
with clang.

Changes v1 => v2:
 - Keep notrace on __always_inline. (Steven Rostedt)
 - Add patches 1-15 to mark notrace the helpers that relied on inline
   implying notrace.
 - Rework the commit message of the last patch.

v1: https://lore.kernel.org/all/20261021222255.3124940-1-song@kernel.org/

Song Liu (16):
  x86/paravirt: Mark native pv_ops helpers notrace
  x86: Mark CPU state helpers used by resume and kexec notrace
  bpf: Mark bpf_spin_lock helpers notrace
  bpf: Mark the BPF trampoline helpers notrace
  bpf: Mark the bpf_mem_alloc fast path helpers notrace
  u64_stats: Mark the update helpers used by BPF notrace
  tracepoint: Mark __do_trace_##name() notrace
  srcu: Mark the fast notrace read-side helpers notrace
  arm64: Mark the preempt count helpers notrace
  powerpc, csky, mips, sh: Mark the accessors behind current notrace
  rcu: Mark the Tiny RCU rcu_is_watching() notrace
  tracing: Mark the tracing_gen_ctx helpers notrace
  ktime: Mark ktime_to_ns() notrace
  kallsyms: Mark the kernel text address checks notrace
  uaccess: Mark pagefault_disable() and pagefault_enable() notrace
  compiler_types: Move notrace from inline to __always_inline

 arch/arm64/include/asm/preempt.h         | 12 +++++-----
 arch/csky/include/asm/thread_info.h      |  2 +-
 arch/mips/include/asm/thread_info.h      |  2 +-
 arch/powerpc/include/asm/current.h       |  2 +-
 arch/sh/include/asm/thread_info.h        |  2 +-
 arch/x86/include/asm/cpuid/api.h         |  4 ++--
 arch/x86/include/asm/desc.h              | 17 +++++++-------
 arch/x86/include/asm/gsseg.h             |  4 ++--
 arch/x86/include/asm/io.h                |  2 +-
 arch/x86/include/asm/io_bitmap.h         |  2 +-
 arch/x86/include/asm/local.h             |  8 +++----
 arch/x86/include/asm/msr.h               |  8 +++----
 arch/x86/include/asm/paravirt-spinlock.h |  2 +-
 arch/x86/include/asm/paravirt.h          | 20 ++++++++--------
 arch/x86/include/asm/pgalloc.h           |  2 +-
 arch/x86/include/asm/pgtable_64.h        | 10 ++++----
 arch/x86/include/asm/processor.h         |  2 +-
 arch/x86/include/asm/segment.h           |  4 ++--
 arch/x86/include/asm/special_insns.h     |  8 +++----
 include/asm-generic/sections.h           |  4 ++--
 include/linux/bpf.h                      |  8 +++----
 include/linux/compiler_attributes.h      |  6 ++++-
 include/linux/compiler_types.h           |  3 +--
 include/linux/kallsyms.h                 |  2 +-
 include/linux/ktime.h                    |  2 +-
 include/linux/llist.h                    | 16 ++++++-------
 include/linux/percpu-refcount.h          | 12 +++++-----
 include/linux/pgtable.h                  | 12 +++++-----
 include/linux/rcupdate.h                 | 14 +++++------
 include/linux/rcupdate_trace.h           |  4 ++--
 include/linux/rcutiny.h                  |  2 +-
 include/linux/sched.h                    |  8 +++----
 include/linux/srcu.h                     | 10 ++++----
 include/linux/srcutiny.h                 |  5 ++--
 include/linux/srcutree.h                 |  2 +-
 include/linux/trace_events.h             |  6 ++---
 include/linux/tracepoint.h               |  4 ++--
 include/linux/u64_stats_sync.h           | 30 ++++++++++++------------
 include/linux/uaccess.h                  |  4 ++--
 kernel/bpf/helpers.c                     | 12 +++++-----
 40 files changed, 143 insertions(+), 136 deletions(-)

--
2.53.0-Meta

[PATCH v2 01/16] x86/paravirt: Mark native pv_ops helpers notrace

[Song Liu]

The native implementations of several pv_ops are static inline functions
in headers. pv_ops stores their address, so with CONFIG_PARAVIRT_XXL the
compiler emits an out-of-line copy of each one, and pv_ops calls go to
that copy. Today these copies are not traced only because the inline
macro implies notrace.

Some callers of these helpers must not be traced:
restore_processor_state() is notrace and runs during resume,
__switch_to() is __notrace_funcgraph, and arch/x86/kernel/cpu/common.c
and arch/x86/kernel/tsc.c, which call several of them, are built
without -pg.

Mark these helpers notrace explicitly, like native_write_msr() and
native_write_msr_safe() already are, so they stay untraced if inline no
longer implies notrace. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 arch/x86/include/asm/cpuid/api.h     |  4 ++--
 arch/x86/include/asm/desc.h          | 17 +++++++++--------
 arch/x86/include/asm/gsseg.h         |  2 +-
 arch/x86/include/asm/io_bitmap.h     |  2 +-
 arch/x86/include/asm/msr.h           |  6 +++---
 arch/x86/include/asm/pgalloc.h       |  2 +-
 arch/x86/include/asm/pgtable_64.h    | 10 +++++-----
 arch/x86/include/asm/processor.h     |  2 +-
 arch/x86/include/asm/special_insns.h |  2 +-
 include/linux/pgtable.h              | 12 ++++++------
 10 files changed, 30 insertions(+), 29 deletions(-)

diff --git a/arch/x86/include/asm/cpuid/api.h b/arch/x86/include/asm/cpuid/api.h
index 82eddfa2347b..8322792c1208 100644
--- a/arch/x86/include/asm/cpuid/api.h
+++ b/arch/x86/include/asm/cpuid/api.h
@@ -23,8 +23,8 @@ static inline bool cpuid_feature(void)
 }
 #endif
 
-static inline void native_cpuid(u32 *eax, u32 *ebx,
-				u32 *ecx, u32 *edx)
+static inline void notrace native_cpuid(u32 *eax, u32 *ebx,
+					u32 *ecx, u32 *edx)
 {
 	/* ecx is often an input as well as an output. */
 	asm volatile("cpuid"
diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index 00aeae843529..9a01dc9c191a 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -134,17 +134,18 @@ static inline void paravirt_free_ldt(struct desc_struct *ldt, unsigned entries)
 
 #define store_ldt(ldt) asm("sldt %0" : "=m"(ldt))
 
-static inline void native_write_idt_entry(gate_desc *idt, int entry, const gate_desc *gate)
+static inline void notrace native_write_idt_entry(gate_desc *idt, int entry, const gate_desc *gate)
 {
 	memcpy(&idt[entry], gate, sizeof(*gate));
 }
 
-static inline void native_write_ldt_entry(struct desc_struct *ldt, int entry, const void *desc)
+static inline void notrace
+native_write_ldt_entry(struct desc_struct *ldt, int entry, const void *desc)
 {
 	memcpy(&ldt[entry], desc, 8);
 }
 
-static inline void
+static inline void notrace
 native_write_gdt_entry(struct desc_struct *gdt, int entry, const void *desc, int type)
 {
 	unsigned int size;
@@ -189,7 +190,7 @@ static inline void __set_tss_desc(unsigned cpu, unsigned int entry, struct x86_h
 
 #define set_tss_desc(cpu, addr) __set_tss_desc(cpu, GDT_ENTRY_TSS, addr)
 
-static inline void native_set_ldt(const void *addr, unsigned int entries)
+static inline void notrace native_set_ldt(const void *addr, unsigned int entries)
 {
 	if (likely(entries == 0))
 		asm volatile("lldt %w0"::"q" (0));
@@ -205,7 +206,7 @@ static inline void native_set_ldt(const void *addr, unsigned int entries)
 	}
 }
 
-static inline void native_load_gdt(const struct desc_ptr *dtr)
+static inline void notrace native_load_gdt(const struct desc_ptr *dtr)
 {
 	asm volatile("lgdt %0"::"m" (*dtr));
 }
@@ -251,7 +252,7 @@ static inline void native_idt_invalidate(void)
  * original writeable version when needed.
  */
 #ifdef CONFIG_X86_64
-static inline void native_load_tr_desc(void)
+static inline void notrace native_load_tr_desc(void)
 {
 	struct desc_ptr gdt;
 	int cpu = raw_smp_processor_id();
@@ -280,7 +281,7 @@ static inline void native_load_tr_desc(void)
 }
 #endif
 
-static inline unsigned long native_store_tr(void)
+static inline unsigned long notrace native_store_tr(void)
 {
 	unsigned long tr;
 
@@ -289,7 +290,7 @@ static inline unsigned long native_store_tr(void)
 	return tr;
 }
 
-static inline void native_load_tls(struct thread_struct *t, unsigned int cpu)
+static inline void notrace native_load_tls(struct thread_struct *t, unsigned int cpu)
 {
 	struct desc_struct *gdt = get_cpu_gdt_rw(cpu);
 	unsigned int i;
diff --git a/arch/x86/include/asm/gsseg.h b/arch/x86/include/asm/gsseg.h
index ab6a595cea70..14596425f36a 100644
--- a/arch/x86/include/asm/gsseg.h
+++ b/arch/x86/include/asm/gsseg.h
@@ -25,7 +25,7 @@ static inline void native_lkgs(unsigned int selector)
 			    : [sel] "+D" (sel));
 }
 
-static inline void native_load_gs_index(unsigned int selector)
+static inline void notrace native_load_gs_index(unsigned int selector)
 {
 	if (cpu_feature_enabled(X86_FEATURE_LKGS)) {
 		native_lkgs(selector);
diff --git a/arch/x86/include/asm/io_bitmap.h b/arch/x86/include/asm/io_bitmap.h
index 7f080f5c7def..af3b08a685e6 100644
--- a/arch/x86/include/asm/io_bitmap.h
+++ b/arch/x86/include/asm/io_bitmap.h
@@ -19,7 +19,7 @@ struct task_struct;
 void io_bitmap_share(struct task_struct *tsk);
 void io_bitmap_exit(struct task_struct *tsk);
 
-static inline void native_tss_invalidate_io_bitmap(void)
+static inline void notrace native_tss_invalidate_io_bitmap(void)
 {
 	/*
 	 * Invalidate the I/O bitmap by moving io_bitmap_base outside the
diff --git a/arch/x86/include/asm/msr.h b/arch/x86/include/asm/msr.h
index 2f41a9289d79..ca138498a00b 100644
--- a/arch/x86/include/asm/msr.h
+++ b/arch/x86/include/asm/msr.h
@@ -101,7 +101,7 @@ static __always_inline u64 native_rdmsrq(u32 msr)
 #define native_wrmsrq(msr, val)				\
 	__wrmsrq((msr), (val))
 
-static inline u64 native_read_msr(u32 msr)
+static inline u64 notrace native_read_msr(u32 msr)
 {
 	u64 val;
 
@@ -113,7 +113,7 @@ static inline u64 native_read_msr(u32 msr)
 	return val;
 }
 
-static inline int native_read_msr_safe(u32 msr, u64 *p)
+static inline int notrace native_read_msr_safe(u32 msr, u64 *p)
 {
 	int err;
 	EAX_EDX_DECLARE_ARGS(val, low, high);
@@ -159,7 +159,7 @@ static inline int notrace native_write_msr_safe(u32 msr, u64 val)
 extern int rdmsr_safe_regs(u32 regs[8]);
 extern int wrmsr_safe_regs(u32 regs[8]);
 
-static inline u64 native_read_pmc(int counter)
+static inline u64 notrace native_read_pmc(int counter)
 {
 	EAX_EDX_DECLARE_ARGS(val, low, high);
 
diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h
index c88691b15f3c..8d233e2e72bf 100644
--- a/arch/x86/include/asm/pgalloc.h
+++ b/arch/x86/include/asm/pgalloc.h
@@ -12,7 +12,7 @@
 #define __HAVE_ARCH_PGD_FREE
 #include <asm-generic/pgalloc.h>
 
-static inline int  __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; }
+static inline int notrace __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; }
 
 #ifdef CONFIG_PARAVIRT_XXL
 #include <asm/paravirt.h>
diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h
index ce45882ccd07..8399ff79941f 100644
--- a/arch/x86/include/asm/pgtable_64.h
+++ b/arch/x86/include/asm/pgtable_64.h
@@ -58,7 +58,7 @@ static inline bool mm_p4d_folded(struct mm_struct *mm)
 void set_pte_vaddr_p4d(p4d_t *p4d_page, unsigned long vaddr, pte_t new_pte);
 void set_pte_vaddr_pud(pud_t *pud_page, unsigned long vaddr, pte_t new_pte);
 
-static inline void native_set_pte(pte_t *ptep, pte_t pte)
+static inline void notrace native_set_pte(pte_t *ptep, pte_t pte)
 {
 	WRITE_ONCE(*ptep, pte);
 }
@@ -74,7 +74,7 @@ static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte)
 	native_set_pte(ptep, pte);
 }
 
-static inline void native_set_pmd(pmd_t *pmdp, pmd_t pmd)
+static inline void notrace native_set_pmd(pmd_t *pmdp, pmd_t pmd)
 {
 	WRITE_ONCE(*pmdp, pmd);
 }
@@ -110,7 +110,7 @@ static inline pmd_t native_pmdp_get_and_clear(pmd_t *xp)
 #endif
 }
 
-static inline void native_set_pud(pud_t *pudp, pud_t pud)
+static inline void notrace native_set_pud(pud_t *pudp, pud_t pud)
 {
 	WRITE_ONCE(*pudp, pud);
 }
@@ -135,7 +135,7 @@ static inline pud_t native_pudp_get_and_clear(pud_t *xp)
 #endif
 }
 
-static inline void native_set_p4d(p4d_t *p4dp, p4d_t p4d)
+static inline void notrace native_set_p4d(p4d_t *p4dp, p4d_t p4d)
 {
 	pgd_t pgd;
 
@@ -155,7 +155,7 @@ static inline void native_p4d_clear(p4d_t *p4d)
 	native_set_p4d(p4d, native_make_p4d(0));
 }
 
-static inline void native_set_pgd(pgd_t *pgdp, pgd_t pgd)
+static inline void notrace native_set_pgd(pgd_t *pgdp, pgd_t pgd)
 {
 	WRITE_ONCE(*pgdp, pti_set_user_pgtbl(pgdp, pgd));
 }
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 87b1d4c0727e..e25f709720df 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -536,7 +536,7 @@ static inline void arch_thread_struct_whitelist(unsigned long *offset,
 	fpu_thread_struct_whitelist(offset, size);
 }
 
-static inline void
+static inline void notrace
 native_load_sp0(unsigned long sp0)
 {
 	this_cpu_write(cpu_tss_rw.x86_tss.sp0, sp0);
diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
index 46aa2c9c1bda..810c992f3a0d 100644
--- a/arch/x86/include/asm/special_insns.h
+++ b/arch/x86/include/asm/special_insns.h
@@ -12,7 +12,7 @@
 
 void native_write_cr0(unsigned long val);
 
-static inline unsigned long native_read_cr0(void)
+static inline unsigned long notrace native_read_cr0(void)
 {
 	unsigned long val;
 	asm volatile("mov %%cr0,%0" : "=r" (val));
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 2981e386da7b..bfdd19ae3194 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1602,9 +1602,9 @@ static inline int pmd_none_or_clear_bad(pmd_t *pmd)
 	return 0;
 }
 
-static inline pte_t __ptep_modify_prot_start(struct vm_area_struct *vma,
-					     unsigned long addr,
-					     pte_t *ptep)
+static inline pte_t notrace __ptep_modify_prot_start(struct vm_area_struct *vma,
+						     unsigned long addr,
+						     pte_t *ptep)
 {
 	/*
 	 * Get the current pte state, but zero it out to make it
@@ -1614,9 +1614,9 @@ static inline pte_t __ptep_modify_prot_start(struct vm_area_struct *vma,
 	return ptep_get_and_clear(vma->vm_mm, addr, ptep);
 }
 
-static inline void __ptep_modify_prot_commit(struct vm_area_struct *vma,
-					     unsigned long addr,
-					     pte_t *ptep, pte_t pte)
+static inline void notrace __ptep_modify_prot_commit(struct vm_area_struct *vma,
+						     unsigned long addr,
+						     pte_t *ptep, pte_t pte)
 {
 	/*
 	 * The pte is non-present, so there's no hardware state to
-- 
2.53.0-Meta

[PATCH v2 02/16] x86: Mark CPU state helpers used by resume and kexec notrace

[Song Liu]

__restore_processor_state() is notrace because it starts running before
%gs and the IDT are restored, when the function tracer cannot run yet.
Before it restores MSR_GS_BASE, it calls wrmsrq(), __write_cr4(),
write_cr3(), write_cr0(), load_idt() and loadsegment(), and after that
it calls load_gs_index(). With CONFIG_PARAVIRT_XXL, wrmsrq() calls
paravirt_write_msr(). machine_kexec() calls virt_to_phys() after
load_segments() has reset %gs. All of these are plain inline functions.
If the compiler emits one of them out of line, that copy is untraced
only because inline implies notrace.

Mark them notrace explicitly, together with the other paravirt MSR
accessors, so they stay consistent with paravirt_write_msr(). No
functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 arch/x86/include/asm/gsseg.h         |  2 +-
 arch/x86/include/asm/io.h            |  2 +-
 arch/x86/include/asm/msr.h           |  2 +-
 arch/x86/include/asm/paravirt.h      | 20 ++++++++++----------
 arch/x86/include/asm/segment.h       |  4 ++--
 arch/x86/include/asm/special_insns.h |  6 +++---
 6 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/arch/x86/include/asm/gsseg.h b/arch/x86/include/asm/gsseg.h
index 14596425f36a..d46d6effb631 100644
--- a/arch/x86/include/asm/gsseg.h
+++ b/arch/x86/include/asm/gsseg.h
@@ -52,7 +52,7 @@ static inline void __init lkgs_init(void)
 
 #ifndef CONFIG_PARAVIRT_XXL
 
-static inline void load_gs_index(unsigned int selector)
+static inline void notrace load_gs_index(unsigned int selector)
 {
 #ifdef CONFIG_X86_64
 	native_load_gs_index(selector);
diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h
index 7f4847b2b904..721ce7878966 100644
--- a/arch/x86/include/asm/io.h
+++ b/arch/x86/include/asm/io.h
@@ -126,7 +126,7 @@ extern int valid_mmap_phys_addr_range(unsigned long pfn, size_t size);
  *	this function
  */
 
-static inline phys_addr_t virt_to_phys(volatile void *address)
+static inline phys_addr_t notrace virt_to_phys(volatile void *address)
 {
 	return __pa(address);
 }
diff --git a/arch/x86/include/asm/msr.h b/arch/x86/include/asm/msr.h
index ca138498a00b..7c311dbcd0bf 100644
--- a/arch/x86/include/asm/msr.h
+++ b/arch/x86/include/asm/msr.h
@@ -194,7 +194,7 @@ static inline void wrmsr(u32 msr, u32 low, u32 high)
 #define rdmsrq(msr, val)			\
 	((val) = native_read_msr((msr)))
 
-static inline void wrmsrq(u32 msr, u64 val)
+static inline void notrace wrmsrq(u32 msr, u64 val)
 {
 	native_write_msr(msr, val);
 }
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 0591aa38fd85..c30af937103b 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -98,7 +98,7 @@ static inline unsigned long read_cr0(void)
 	return PVOP_CALL0(unsigned long, pv_ops, cpu.read_cr0);
 }
 
-static inline void write_cr0(unsigned long x)
+static inline void notrace write_cr0(unsigned long x)
 {
 	PVOP_VCALL1(pv_ops, cpu.write_cr0, x);
 }
@@ -120,32 +120,32 @@ static inline unsigned long __read_cr3(void)
 			      "mov %%cr3, %%rax", ALT_NOT_XEN);
 }
 
-static inline void write_cr3(unsigned long x)
+static inline void notrace write_cr3(unsigned long x)
 {
 	PVOP_ALT_VCALL1(pv_ops, mmu.write_cr3, x, "mov %%rdi, %%cr3", ALT_NOT_XEN);
 }
 
-static inline void __write_cr4(unsigned long x)
+static inline void notrace __write_cr4(unsigned long x)
 {
 	PVOP_VCALL1(pv_ops, cpu.write_cr4, x);
 }
 
-static inline u64 paravirt_read_msr(u32 msr)
+static inline u64 notrace paravirt_read_msr(u32 msr)
 {
 	return PVOP_CALL1(u64, pv_ops, cpu.read_msr, msr);
 }
 
-static inline void paravirt_write_msr(u32 msr, u64 val)
+static inline void notrace paravirt_write_msr(u32 msr, u64 val)
 {
 	PVOP_VCALL2(pv_ops, cpu.write_msr, msr, val);
 }
 
-static inline int paravirt_read_msr_safe(u32 msr, u64 *val)
+static inline int notrace paravirt_read_msr_safe(u32 msr, u64 *val)
 {
 	return PVOP_CALL2(int, pv_ops, cpu.read_msr_safe, msr, val);
 }
 
-static inline int paravirt_write_msr_safe(u32 msr, u64 val)
+static inline int notrace paravirt_write_msr_safe(u32 msr, u64 val)
 {
 	return PVOP_CALL2(int, pv_ops, cpu.write_msr_safe, msr, val);
 }
@@ -167,7 +167,7 @@ do {						\
 	val = paravirt_read_msr(msr);		\
 } while (0)
 
-static inline void wrmsrq(u32 msr, u64 val)
+static inline void notrace wrmsrq(u32 msr, u64 val)
 {
 	paravirt_write_msr(msr, val);
 }
@@ -215,7 +215,7 @@ static inline void load_gdt(const struct desc_ptr *dtr)
 {
 	PVOP_VCALL1(pv_ops, cpu.load_gdt, dtr);
 }
-static inline void load_idt(const struct desc_ptr *dtr)
+static inline void notrace load_idt(const struct desc_ptr *dtr)
 {
 	PVOP_VCALL1(pv_ops, cpu.load_idt, dtr);
 }
@@ -234,7 +234,7 @@ static inline void load_TLS(struct thread_struct *t, unsigned cpu)
 	PVOP_VCALL2(pv_ops, cpu.load_tls, t, cpu);
 }
 
-static inline void load_gs_index(unsigned int gs)
+static inline void notrace load_gs_index(unsigned int gs)
 {
 	PVOP_VCALL1(pv_ops, cpu.load_gs_index, gs);
 }
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index dbd90fede5e7..01ed6f1f7952 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -303,7 +303,7 @@ extern const char xen_early_idt_handler_array[NUM_EXCEPTION_VECTORS][XEN_EARLY_I
  * FS and GS.
  */
 #define LOAD_SEGMENT(seg)						\
-static inline void __loadsegment_##seg(u16 value)			\
+static inline void notrace __loadsegment_##seg(u16 value)		\
 {									\
 	asm volatile("1:	movl %k0,%%" #seg "\n"			\
 		     _ASM_EXTABLE_TYPE_REG(1b, 1b, EX_TYPE_ZERO_REG, %k0)\
@@ -325,7 +325,7 @@ LOAD_SEGMENT(gs)
 
 #else
 
-static inline void __loadsegment_fs(u16 value)
+static inline void notrace __loadsegment_fs(u16 value)
 {
 	asm volatile("1:	movw %0, %%fs\n"
 		     "2:\n"
diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
index 810c992f3a0d..f854eb3333d0 100644
--- a/arch/x86/include/asm/special_insns.h
+++ b/arch/x86/include/asm/special_insns.h
@@ -147,7 +147,7 @@ static inline unsigned long read_cr0(void)
 	return native_read_cr0();
 }
 
-static inline void write_cr0(unsigned long x)
+static inline void notrace write_cr0(unsigned long x)
 {
 	native_write_cr0(x);
 }
@@ -171,12 +171,12 @@ static inline unsigned long __read_cr3(void)
 	return __native_read_cr3();
 }
 
-static inline void write_cr3(unsigned long x)
+static inline void notrace write_cr3(unsigned long x)
 {
 	native_write_cr3(x);
 }
 
-static inline void __write_cr4(unsigned long x)
+static inline void notrace __write_cr4(unsigned long x)
 {
 	native_write_cr4(x);
 }
-- 
2.53.0-Meta

[PATCH v2 03/16] bpf: Mark bpf_spin_lock helpers notrace

[Song Liu]

Commit 178c54666f9c ("bpf: Mark bpf_spin_{lock,unlock}() helpers with
notrace correctly") made bpf_spin_lock() and bpf_spin_unlock()
untraceable, so that an fentry program that also takes the lock cannot
deadlock. These helpers call the static inline functions
__bpf_spin_lock_irqsave(), __bpf_spin_unlock_irqrestore(),
__bpf_spin_lock() and __bpf_spin_unlock(). __bpf_spin_unlock() releases
the lock with queued_spin_unlock(), which is a plain inline function on
x86 with CONFIG_PARAVIRT_SPINLOCKS. The generic version is
__always_inline. __bpf_spin_unlock_irqrestore(), __bpf_spin_unlock()
and queued_spin_unlock() run with the lock held, so an fentry program
attached to an out-of-line copy of one of them can hit the same
deadlock. These copies are untraced today only because inline implies
notrace.

Mark all of these helpers notrace explicitly, so that nothing called by
bpf_spin_lock() and bpf_spin_unlock() is traceable. No functional
change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 arch/x86/include/asm/paravirt-spinlock.h |  2 +-
 kernel/bpf/helpers.c                     | 12 ++++++------
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/paravirt-spinlock.h b/arch/x86/include/asm/paravirt-spinlock.h
index 7beffcb08ed6..dd194646dd9a 100644
--- a/arch/x86/include/asm/paravirt-spinlock.h
+++ b/arch/x86/include/asm/paravirt-spinlock.h
@@ -66,7 +66,7 @@ static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
 	pv_queued_spin_lock_slowpath(lock, val);
 }
 
-static inline void queued_spin_unlock(struct qspinlock *lock)
+static inline void notrace queued_spin_unlock(struct qspinlock *lock)
 {
 	kcsan_release();
 	pv_queued_spin_unlock(lock);
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index c18f1e16edee..66d5005f2260 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -283,7 +283,7 @@ const struct bpf_func_proto bpf_get_current_comm_proto = {
 
 #if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
 
-static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
+static inline void notrace __bpf_spin_lock(struct bpf_spin_lock *lock)
 {
 	arch_spinlock_t *l = (void *)lock;
 	union {
@@ -298,7 +298,7 @@ static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
 	arch_spin_lock(l);
 }
 
-static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
+static inline void notrace __bpf_spin_unlock(struct bpf_spin_lock *lock)
 {
 	arch_spinlock_t *l = (void *)lock;
 
@@ -308,7 +308,7 @@ static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
 
 #else
 
-static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
+static inline void notrace __bpf_spin_lock(struct bpf_spin_lock *lock)
 {
 	atomic_t *l = (void *)lock;
 
@@ -318,7 +318,7 @@ static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
 	} while (atomic_xchg(l, 1));
 }
 
-static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
+static inline void notrace __bpf_spin_unlock(struct bpf_spin_lock *lock)
 {
 	atomic_t *l = (void *)lock;
 
@@ -329,7 +329,7 @@ static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
 
 static DEFINE_PER_CPU(unsigned long, irqsave_flags);
 
-static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
+static inline void notrace __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
 {
 	unsigned long flags;
 
@@ -352,7 +352,7 @@ const struct bpf_func_proto bpf_spin_lock_proto = {
 	.arg1_btf_id    = BPF_PTR_POISON,
 };
 
-static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
+static inline void notrace __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
 {
 	unsigned long flags;
 
-- 
2.53.0-Meta

[PATCH v2 04/16] bpf: Mark the BPF trampoline helpers notrace

[Song Liu]

The BPF trampoline calls __bpf_prog_enter*() before each program and
__bpf_prog_exit*() after it, and calls __bpf_tramp_enter() and
__bpf_tramp_exit() around the traced function. These functions are
notrace: if anything they call could run a BPF program, attaching a
program there would recurse without bound. The recursion check in
__bpf_prog_enter_recur() does not prevent this, because the trampoline
calls the enter and exit functions even for a program it then skips.
For the same reason, btf_id_deny already rejects attaching to the
out-of-line migrate_disable() and migrate_enable(), and, with
CONFIG_PREEMPT_RCU, to __rcu_read_lock() and __rcu_read_unlock().

Many of the helpers they call are plain inline functions:

 - bpf_set_run_ctx(), bpf_reset_run_ctx(),
   bpf_prog_get_recursion_context() and
   bpf_prog_put_recursion_context();
 - __migrate_disable() and __migrate_enable();
 - the !CONFIG_PREEMPT_RCU __rcu_read_lock() and __rcu_read_unlock(),
   rcu_read_unlock() and rcu_read_unlock_migrate(), and the lockdep
   annotations rcu_lock_acquire(), rcu_try_lock_acquire() and
   rcu_lock_release();
 - rcu_read_lock_trace(), rcu_read_unlock_trace() and
   srcu_lock_release();
 - percpu_ref_get(), percpu_ref_put(), percpu_ref_get_many(),
   percpu_ref_put_many() and __ref_is_percpu();
 - local_inc() and local_add() on x86, called through u64_stats_inc()
   and u64_stats_add() from bpf_prog_inc_misses_counter() and
   __update_prog_stats().

An out-of-line copy of any of them is untraced only because inline
implies notrace. Mark them notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 arch/x86/include/asm/local.h    |  4 ++--
 include/linux/bpf.h             |  8 ++++----
 include/linux/percpu-refcount.h | 12 ++++++------
 include/linux/rcupdate.h        | 14 +++++++-------
 include/linux/rcupdate_trace.h  |  4 ++--
 include/linux/sched.h           |  8 ++++----
 include/linux/srcu.h            |  2 +-
 7 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/arch/x86/include/asm/local.h b/arch/x86/include/asm/local.h
index 4957018fef3e..1749c4481000 100644
--- a/arch/x86/include/asm/local.h
+++ b/arch/x86/include/asm/local.h
@@ -16,7 +16,7 @@ typedef struct {
 #define local_read(l)	atomic_long_read(&(l)->a)
 #define local_set(l, i)	atomic_long_set(&(l)->a, (i))
 
-static inline void local_inc(local_t *l)
+static inline void notrace local_inc(local_t *l)
 {
 	asm volatile(_ASM_INC "%0"
 		     : "+m" (l->a.counter));
@@ -28,7 +28,7 @@ static inline void local_dec(local_t *l)
 		     : "+m" (l->a.counter));
 }
 
-static inline void local_add(long i, local_t *l)
+static inline void notrace local_add(long i, local_t *l)
 {
 	asm volatile(_ASM_ADD "%1,%0"
 		     : "+m" (l->a.counter)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 7719f6528445..22bf8e285117 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2156,7 +2156,7 @@ struct bpf_struct_ops_common_value {
 	enum bpf_struct_ops_state state;
 };
 
-static inline bool bpf_prog_get_recursion_context(struct bpf_prog *prog)
+static inline bool notrace bpf_prog_get_recursion_context(struct bpf_prog *prog)
 {
 #ifdef CONFIG_ARM64
 	u8 rctx = interrupt_context_level();
@@ -2176,7 +2176,7 @@ static inline bool bpf_prog_get_recursion_context(struct bpf_prog *prog)
 #endif
 }
 
-static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog)
+static inline void notrace bpf_prog_put_recursion_context(struct bpf_prog *prog)
 {
 #ifdef CONFIG_ARM64
 	u8 rctx = interrupt_context_level();
@@ -2547,7 +2547,7 @@ struct bpf_tramp_run_ctx {
 	struct bpf_run_ctx *saved_run_ctx;
 };
 
-static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
+static inline notrace struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
 {
 	struct bpf_run_ctx *old_ctx = NULL;
 
@@ -2558,7 +2558,7 @@ static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
 	return old_ctx;
 }
 
-static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
+static inline void notrace bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
 {
 #ifdef CONFIG_BPF_SYSCALL
 	current->bpf_ctx = old_ctx;
diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h
index d73a1c08c3e3..41db7d25b802 100644
--- a/include/linux/percpu-refcount.h
+++ b/include/linux/percpu-refcount.h
@@ -155,8 +155,8 @@ static inline void percpu_ref_kill(struct percpu_ref *ref)
  * because doing so forces the compiler to generate two conditional
  * branches as it can't assume that @ref->percpu_count is not NULL.
  */
-static inline bool __ref_is_percpu(struct percpu_ref *ref,
-					  unsigned long __percpu **percpu_countp)
+static inline bool notrace __ref_is_percpu(struct percpu_ref *ref,
+					   unsigned long __percpu **percpu_countp)
 {
 	unsigned long percpu_ptr;
 
@@ -195,7 +195,7 @@ static inline bool __ref_is_percpu(struct percpu_ref *ref,
  *
  * This function is safe to call as long as @ref is between init and exit.
  */
-static inline void percpu_ref_get_many(struct percpu_ref *ref, unsigned long nr)
+static inline void notrace percpu_ref_get_many(struct percpu_ref *ref, unsigned long nr)
 {
 	unsigned long __percpu *percpu_count;
 
@@ -217,7 +217,7 @@ static inline void percpu_ref_get_many(struct percpu_ref *ref, unsigned long nr)
  *
  * This function is safe to call as long as @ref is between init and exit.
  */
-static inline void percpu_ref_get(struct percpu_ref *ref)
+static inline void notrace percpu_ref_get(struct percpu_ref *ref)
 {
 	percpu_ref_get_many(ref, 1);
 }
@@ -323,7 +323,7 @@ static inline bool percpu_ref_tryget_live(struct percpu_ref *ref)
  *
  * This function is safe to call as long as @ref is between init and exit.
  */
-static inline void percpu_ref_put_many(struct percpu_ref *ref, unsigned long nr)
+static inline void notrace percpu_ref_put_many(struct percpu_ref *ref, unsigned long nr)
 {
 	unsigned long __percpu *percpu_count;
 
@@ -346,7 +346,7 @@ static inline void percpu_ref_put_many(struct percpu_ref *ref, unsigned long nr)
  *
  * This function is safe to call as long as @ref is between init and exit.
  */
-static inline void percpu_ref_put(struct percpu_ref *ref)
+static inline void notrace percpu_ref_put(struct percpu_ref *ref)
 {
 	percpu_ref_put_many(ref, 1);
 }
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 5e95acc33989..1d26aee3c71c 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -98,12 +98,12 @@ void __rcu_read_unlock(void);
 void rcu_read_unlock_strict(void);
 #endif
 
-static inline void __rcu_read_lock(void)
+static inline void notrace __rcu_read_lock(void)
 {
 	preempt_disable();
 }
 
-static inline void __rcu_read_unlock(void)
+static inline void notrace __rcu_read_unlock(void)
 {
 	if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD))
 		rcu_read_unlock_strict();
@@ -295,17 +295,17 @@ extern struct lockdep_map rcu_callback_map;
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 
-static inline void rcu_lock_acquire(struct lockdep_map *map)
+static inline void notrace rcu_lock_acquire(struct lockdep_map *map)
 {
 	lock_acquire(map, 0, 0, 2, 0, NULL, _THIS_IP_);
 }
 
-static inline void rcu_try_lock_acquire(struct lockdep_map *map)
+static inline void notrace rcu_try_lock_acquire(struct lockdep_map *map)
 {
 	lock_acquire(map, 0, 1, 2, 0, NULL, _THIS_IP_);
 }
 
-static inline void rcu_lock_release(struct lockdep_map *map)
+static inline void notrace rcu_lock_release(struct lockdep_map *map)
 {
 	lock_release(map, _THIS_IP_);
 }
@@ -863,7 +863,7 @@ static __always_inline void rcu_read_lock(void)
  *
  * See rcu_read_lock() for more information.
  */
-static inline void rcu_read_unlock(void)
+static inline void notrace rcu_read_unlock(void)
 	__releases_shared(RCU)
 {
 	RCU_LOCKDEP_WARN(!rcu_is_watching(),
@@ -982,7 +982,7 @@ static __always_inline void rcu_read_lock_dont_migrate(void)
 	rcu_read_lock();
 }
 
-static inline void rcu_read_unlock_migrate(void)
+static inline void notrace rcu_read_unlock_migrate(void)
 	__releases_shared(RCU)
 {
 	rcu_read_unlock();
diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h
index cee89e51e45c..94a44773b87d 100644
--- a/include/linux/rcupdate_trace.h
+++ b/include/linux/rcupdate_trace.h
@@ -93,7 +93,7 @@ static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *scp)
  *
  * For more details, please see the documentation for rcu_read_lock().
  */
-static inline void rcu_read_lock_trace(void)
+static inline void notrace rcu_read_lock_trace(void)
 {
 	struct task_struct *t = current;
 
@@ -117,7 +117,7 @@ static inline void rcu_read_lock_trace(void)
  *
  * For more details, please see the documentation for rcu_read_unlock().
  */
-static inline void rcu_read_unlock_trace(void)
+static inline void notrace rcu_read_unlock_trace(void)
 {
 	struct srcu_ctr __percpu *scp;
 	struct task_struct *t = current;
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 373bcc0598d1..8ec1f4172180 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2429,7 +2429,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
 #endif
 #define this_rq_pinned() (*(unsigned int *)((void *)this_rq_raw() + RQ_nr_pinned))
 
-static inline void __migrate_enable(void)
+static inline void notrace __migrate_enable(void)
 {
 	struct task_struct *p = current;
 
@@ -2464,7 +2464,7 @@ static inline void __migrate_enable(void)
 	this_rq_pinned()--;
 }
 
-static inline void __migrate_disable(void)
+static inline void notrace __migrate_disable(void)
 {
 	struct task_struct *p = current;
 
@@ -2484,8 +2484,8 @@ static inline void __migrate_disable(void)
 	p->migration_disabled = 1;
 }
 #else /* !COMPILE_OFFSETS */
-static inline void __migrate_disable(void) { }
-static inline void __migrate_enable(void) { }
+static inline void notrace __migrate_disable(void) { }
+static inline void notrace __migrate_enable(void) { }
 #endif /* !COMPILE_OFFSETS */
 
 /*
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index a54ce9e808b9..bce8cac45474 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -188,7 +188,7 @@ static inline void srcu_lock_acquire(struct lockdep_map *map)
 }
 
 /* Annotates a srcu_read_lock() */
-static inline void srcu_lock_release(struct lockdep_map *map)
+static inline void notrace srcu_lock_release(struct lockdep_map *map)
 {
 	lock_map_release(map);
 }
-- 
2.53.0-Meta

[PATCH v2 05/16] bpf: Mark the bpf_mem_alloc fast path helpers notrace

[Song Liu]

unit_alloc(), unit_free() and unit_free_rcu() are notrace so that BPF
programs cannot attach to them and corrupt the per-CPU free lists, as
the comment above unit_alloc() says. To take and release c->active and
to update the free lists, they call local_inc_return()
(local_add_return() on x86), local_dec(), __llist_add() and
llist_add(). These are plain inline functions, as are
__llist_add_batch() and llist_add_batch(), which the llist helpers call.
An out-of-line copy of any of them is untraced only because inline
implies notrace.

Mark them notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 arch/x86/include/asm/local.h |  4 ++--
 include/linux/llist.h        | 16 ++++++++--------
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/arch/x86/include/asm/local.h b/arch/x86/include/asm/local.h
index 1749c4481000..66f2cd57160c 100644
--- a/arch/x86/include/asm/local.h
+++ b/arch/x86/include/asm/local.h
@@ -22,7 +22,7 @@ static inline void notrace local_inc(local_t *l)
 		     : "+m" (l->a.counter));
 }
 
-static inline void local_dec(local_t *l)
+static inline void notrace local_dec(local_t *l)
 {
 	asm volatile(_ASM_DEC "%0"
 		     : "+m" (l->a.counter));
@@ -103,7 +103,7 @@ static inline bool local_add_negative(long i, local_t *l)
  *
  * Atomically adds @i to @l and returns @i + @l
  */
-static inline long local_add_return(long i, local_t *l)
+static inline long notrace local_add_return(long i, local_t *l)
 {
 	long __i = i;
 	asm volatile(_ASM_XADD "%0, %1"
diff --git a/include/linux/llist.h b/include/linux/llist.h
index 8846b7709669..53bc48f6b956 100644
--- a/include/linux/llist.h
+++ b/include/linux/llist.h
@@ -231,9 +231,9 @@ static inline struct llist_node *llist_next(struct llist_node *node)
  *
  * Return whether list is empty before adding.
  */
-static inline bool llist_add_batch(struct llist_node *new_first,
-				   struct llist_node *new_last,
-				   struct llist_head *head)
+static inline bool notrace llist_add_batch(struct llist_node *new_first,
+					   struct llist_node *new_last,
+					   struct llist_head *head)
 {
 	struct llist_node *first = READ_ONCE(head->first);
 
@@ -244,9 +244,9 @@ static inline bool llist_add_batch(struct llist_node *new_first,
 	return !first;
 }
 
-static inline bool __llist_add_batch(struct llist_node *new_first,
-				     struct llist_node *new_last,
-				     struct llist_head *head)
+static inline bool notrace __llist_add_batch(struct llist_node *new_first,
+					     struct llist_node *new_last,
+					     struct llist_head *head)
 {
 	new_last->next = head->first;
 	head->first = new_first;
@@ -260,12 +260,12 @@ static inline bool __llist_add_batch(struct llist_node *new_first,
  *
  * Returns true if the list was empty prior to adding this entry.
  */
-static inline bool llist_add(struct llist_node *new, struct llist_head *head)
+static inline bool notrace llist_add(struct llist_node *new, struct llist_head *head)
 {
 	return llist_add_batch(new, new, head);
 }
 
-static inline bool __llist_add(struct llist_node *new, struct llist_head *head)
+static inline bool notrace __llist_add(struct llist_node *new, struct llist_head *head)
 {
 	return __llist_add_batch(new, new, head);
 }
-- 
2.53.0-Meta

[PATCH v2 06/16] u64_stats: Mark the update helpers used by BPF notrace

[Song Liu]

bpf_prog_inc_misses_counter() is notrace because it runs when a BPF
program was not run due to recursion. If it could call traced code, an
fentry program attached to that code would recurse forever. It calls
u64_stats_update_begin_irqsave(), u64_stats_inc() and
u64_stats_update_end_irqrestore(). The BPF trampoline calls
u64_stats_add() and u64_stats_inc() from __update_prog_stats() in the
same way. These are plain inline helpers, so an out-of-line copy is
untraced only because inline implies notrace.

Mark these helpers, and the __u64_stats_*() helpers they call, notrace
explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/u64_stats_sync.h | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/include/linux/u64_stats_sync.h b/include/linux/u64_stats_sync.h
index f47cf4d78ad7..06e5bffd0287 100644
--- a/include/linux/u64_stats_sync.h
+++ b/include/linux/u64_stats_sync.h
@@ -92,7 +92,7 @@ static inline void u64_stats_set(u64_stats_t *p, u64 val)
 	local64_set(&p->v, val);
 }
 
-static inline void u64_stats_add(u64_stats_t *p, unsigned long val)
+static inline void notrace u64_stats_add(u64_stats_t *p, unsigned long val)
 {
 	local64_add(val, &p->v);
 }
@@ -102,16 +102,16 @@ static inline void u64_stats_sub(u64_stats_t *p, s64 val)
 	local64_sub(val, &p->v);
 }
 
-static inline void u64_stats_inc(u64_stats_t *p)
+static inline void notrace u64_stats_inc(u64_stats_t *p)
 {
 	local64_inc(&p->v);
 }
 
 static inline void u64_stats_init(struct u64_stats_sync *syncp) { }
-static inline void __u64_stats_update_begin(struct u64_stats_sync *syncp) { }
-static inline void __u64_stats_update_end(struct u64_stats_sync *syncp) { }
-static inline unsigned long __u64_stats_irqsave(void) { return 0; }
-static inline void __u64_stats_irqrestore(unsigned long flags) { }
+static inline void notrace __u64_stats_update_begin(struct u64_stats_sync *syncp) { }
+static inline void notrace __u64_stats_update_end(struct u64_stats_sync *syncp) { }
+static inline unsigned long notrace __u64_stats_irqsave(void) { return 0; }
+static inline void notrace __u64_stats_irqrestore(unsigned long flags) { }
 static inline unsigned int __u64_stats_fetch_begin(const struct u64_stats_sync *syncp)
 {
 	return 0;
@@ -145,7 +145,7 @@ static inline void u64_stats_set(u64_stats_t *p, u64 val)
 	p->v = val;
 }
 
-static inline void u64_stats_add(u64_stats_t *p, unsigned long val)
+static inline void notrace u64_stats_add(u64_stats_t *p, unsigned long val)
 {
 	p->v += val;
 }
@@ -155,7 +155,7 @@ static inline void u64_stats_sub(u64_stats_t *p, s64 val)
 	p->v -= val;
 }
 
-static inline void u64_stats_inc(u64_stats_t *p)
+static inline void notrace u64_stats_inc(u64_stats_t *p)
 {
 	p->v++;
 }
@@ -166,19 +166,19 @@ static inline void u64_stats_inc(u64_stats_t *p)
 		seqcount_init(&__s->seq);		\
 	} while (0)
 
-static inline void __u64_stats_update_begin(struct u64_stats_sync *syncp)
+static inline void notrace __u64_stats_update_begin(struct u64_stats_sync *syncp)
 {
 	preempt_disable_nested();
 	write_seqcount_begin(&syncp->seq);
 }
 
-static inline void __u64_stats_update_end(struct u64_stats_sync *syncp)
+static inline void notrace __u64_stats_update_end(struct u64_stats_sync *syncp)
 {
 	write_seqcount_end(&syncp->seq);
 	preempt_enable_nested();
 }
 
-static inline unsigned long __u64_stats_irqsave(void)
+static inline unsigned long notrace __u64_stats_irqsave(void)
 {
 	unsigned long flags;
 
@@ -186,7 +186,7 @@ static inline unsigned long __u64_stats_irqsave(void)
 	return flags;
 }
 
-static inline void __u64_stats_irqrestore(unsigned long flags)
+static inline void notrace __u64_stats_irqrestore(unsigned long flags)
 {
 	local_irq_restore(flags);
 }
@@ -213,7 +213,7 @@ static inline void u64_stats_update_end(struct u64_stats_sync *syncp)
 	__u64_stats_update_end(syncp);
 }
 
-static inline unsigned long u64_stats_update_begin_irqsave(struct u64_stats_sync *syncp)
+static inline unsigned long notrace u64_stats_update_begin_irqsave(struct u64_stats_sync *syncp)
 {
 	unsigned long flags = __u64_stats_irqsave();
 
@@ -221,8 +221,8 @@ static inline unsigned long u64_stats_update_begin_irqsave(struct u64_stats_sync
 	return flags;
 }
 
-static inline void u64_stats_update_end_irqrestore(struct u64_stats_sync *syncp,
-						   unsigned long flags)
+static inline void notrace u64_stats_update_end_irqrestore(struct u64_stats_sync *syncp,
+							   unsigned long flags)
 {
 	__u64_stats_update_end(syncp);
 	__u64_stats_irqrestore(flags);
-- 
2.53.0-Meta

[PATCH v2 07/16] tracepoint: Mark __do_trace_##name() notrace

[Song Liu]

__do_trace_##name() takes the tracepoint SRCU read lock and calls the
probes. It is called from trace_##name(), trace_call__##name(), and the
notrace rust_do_trace_##name(). It is a plain inline function, so if
the compiler emits it out of line, that copy is untraced only because
inline implies notrace.

Mark it notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/tracepoint.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h
index e0d838c9ce93..1cd18b56c0a8 100644
--- a/include/linux/tracepoint.h
+++ b/include/linux/tracepoint.h
@@ -308,7 +308,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 
 #define __DECLARE_TRACE(name, proto, args, cond, data_proto)			\
 	__DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto)) \
-	static inline void __do_trace_##name(proto)			\
+	static inline void notrace __do_trace_##name(proto)		\
 	{								\
 		TRACEPOINT_CHECK(name)					\
 		if (cond) {						\
@@ -332,7 +332,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 
 #define __DECLARE_TRACE_SYSCALL(name, proto, args, data_proto)		\
 	__DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto)) \
-	static inline void __do_trace_##name(proto)			\
+	static inline void notrace __do_trace_##name(proto)		\
 	{								\
 		TRACEPOINT_CHECK(name)					\
 		guard(rcu_tasks_trace)();				\
-- 
2.53.0-Meta

[PATCH v2 08/16] srcu: Mark the fast notrace read-side helpers notrace

[Song Liu]

srcu_read_lock_fast_notrace() and srcu_read_unlock_fast_notrace() are
used by tracepoints, and their comments say they cannot be traced. They
are plain inline functions, as are srcu_check_read_flavor() and the
Tiny SRCU versions of __srcu_read_lock_fast() and
__srcu_read_unlock_fast() that they call. An out-of-line copy of any of
them is untraced only because inline implies notrace.

Mark them notrace explicitly, as srcu_read_lock_notrace() and the Tree
SRCU __srcu_read_lock_fast() and __srcu_read_unlock_fast() already are.
No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/srcu.h     | 8 +++++---
 include/linux/srcutiny.h | 5 +++--
 include/linux/srcutree.h | 2 +-
 3 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index bce8cac45474..1ed3fc50b77f 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -377,7 +377,8 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_fast_updown(struct srcu_s
  * Used by tracing, cannot be traced and cannot call lockdep.
  * See srcu_read_lock_fast() for more information.
  */
-static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct srcu_struct *ssp)
+static inline struct srcu_ctr __percpu notrace *
+srcu_read_lock_fast_notrace(struct srcu_struct *ssp)
 	__acquires_shared(ssp)
 {
 	struct srcu_ctr __percpu *retval;
@@ -524,8 +525,9 @@ srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __percpu *
  * Used by tracing, cannot be traced and cannot call lockdep.
  * See srcu_read_unlock_fast() for more information.
  */
-static inline void srcu_read_unlock_fast_notrace(struct srcu_struct *ssp,
-						 struct srcu_ctr __percpu *scp) __releases_shared(ssp)
+static inline void notrace
+srcu_read_unlock_fast_notrace(struct srcu_struct *ssp,
+			      struct srcu_ctr __percpu *scp) __releases_shared(ssp)
 {
 	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);
 	__srcu_read_unlock_fast(ssp, scp);
diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h
index 905b629e8fa3..fdb988d41167 100644
--- a/include/linux/srcutiny.h
+++ b/include/linux/srcutiny.h
@@ -101,13 +101,14 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_ptr(struct srcu_struct *ss
 	return (struct srcu_ctr __percpu *)(intptr_t)idx;
 }
 
-static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_struct *ssp)
+static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast(struct srcu_struct *ssp)
 	__acquires_shared(ssp)
 {
 	return __srcu_ctr_to_ptr(ssp, __srcu_read_lock(ssp));
 }
 
-static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)
+static inline void notrace
+__srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)
 	__releases_shared(ssp)
 {
 	__srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp));
diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h
index fd1a9270cb9a..8328218a34fa 100644
--- a/include/linux/srcutree.h
+++ b/include/linux/srcutree.h
@@ -368,7 +368,7 @@ __srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __percpu
 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor);
 
 // Record SRCU-reader usage type only for CONFIG_PROVE_RCU=y kernels.
-static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor)
+static inline void notrace srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor)
 {
 	if (IS_ENABLED(CONFIG_PROVE_RCU))
 		__srcu_check_read_flavor(ssp, read_flavor);
-- 
2.53.0-Meta

[PATCH v2 09/16] arm64: Mark the preempt count helpers notrace

[Song Liu]

ftrace_test_recursion_trylock() reads preempt_count() to find the
current context, and preempt_disable_notrace() and
preempt_enable_notrace() use __preempt_count_add(),
__preempt_count_sub(), __preempt_count_dec_and_test() and
should_resched(). On arm64 these are plain inline functions. If the
compiler emits one of them out of line, that copy is untraced only
because inline implies notrace, and tracing it would recurse before the
recursion check takes effect.

Mark them, and preempt_count_set(), notrace explicitly. The x86 and
generic versions are __always_inline. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 arch/arm64/include/asm/preempt.h | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index 932ea4b62042..0441938c9e7e 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -7,12 +7,12 @@
 #define PREEMPT_NEED_RESCHED	BIT(32)
 #define PREEMPT_ENABLED	(PREEMPT_NEED_RESCHED)
 
-static inline int preempt_count(void)
+static inline int notrace preempt_count(void)
 {
 	return READ_ONCE(current_thread_info()->preempt.count);
 }
 
-static inline void preempt_count_set(u64 pc)
+static inline void notrace preempt_count_set(u64 pc)
 {
 	/* Preserve existing value of PREEMPT_NEED_RESCHED */
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
@@ -41,21 +41,21 @@ static inline bool test_preempt_need_resched(void)
 	return !current_thread_info()->preempt.need_resched;
 }
 
-static inline void __preempt_count_add(int val)
+static inline void notrace __preempt_count_add(int val)
 {
 	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
 	pc += val;
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
 }
 
-static inline void __preempt_count_sub(int val)
+static inline void notrace __preempt_count_sub(int val)
 {
 	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
 	pc -= val;
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
 }
 
-static inline bool __preempt_count_dec_and_test(void)
+static inline bool notrace __preempt_count_dec_and_test(void)
 {
 	struct thread_info *ti = current_thread_info();
 	u64 pc = READ_ONCE(ti->preempt_count);
@@ -73,7 +73,7 @@ static inline bool __preempt_count_dec_and_test(void)
 	return !pc || !READ_ONCE(ti->preempt_count);
 }
 
-static inline bool should_resched(int preempt_offset)
+static inline bool notrace should_resched(int preempt_offset)
 {
 	u64 pc = READ_ONCE(current_thread_info()->preempt_count);
 	return pc == preempt_offset;
-- 
2.53.0-Meta

[PATCH v2 10/16] powerpc, csky, mips, sh: Mark the accessors behind current notrace

[Song Liu]

ftrace_test_recursion_trylock() reads current->trace_recursion before
it knows whether it is safe to trace, so reading current must never be
a traced call. On powerpc, get_current() is a plain inline function. On
csky, mips and sh, current comes from the generic get_current(), which
calls current_thread_info(), and current_thread_info() is a plain inline
function. An out-of-line copy of any of these is untraced only because
inline implies notrace.

Mark them notrace explicitly. On the other architectures that support
the function tracer, get_current() is already __always_inline. No
functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 arch/csky/include/asm/thread_info.h | 2 +-
 arch/mips/include/asm/thread_info.h | 2 +-
 arch/powerpc/include/asm/current.h  | 2 +-
 arch/sh/include/asm/thread_info.h   | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/csky/include/asm/thread_info.h b/arch/csky/include/asm/thread_info.h
index fdd4f8ad45ac..d39498883c9a 100644
--- a/arch/csky/include/asm/thread_info.h
+++ b/arch/csky/include/asm/thread_info.h
@@ -42,7 +42,7 @@ struct thread_info {
 #define thread_saved_lr(tsk) \
 	((unsigned long)(((struct switch_stack *)(tsk->thread.sp))->r15))
 
-static inline struct thread_info *current_thread_info(void)
+static inline notrace struct thread_info *current_thread_info(void)
 {
 	unsigned long sp;
 
diff --git a/arch/mips/include/asm/thread_info.h b/arch/mips/include/asm/thread_info.h
index 2707dad260dd..4d531efd89ec 100644
--- a/arch/mips/include/asm/thread_info.h
+++ b/arch/mips/include/asm/thread_info.h
@@ -64,7 +64,7 @@ extern struct thread_info *__current_thread_info;
 register struct thread_info *__current_thread_info __asm__("$28");
 #endif
 
-static inline struct thread_info *current_thread_info(void)
+static inline notrace struct thread_info *current_thread_info(void)
 {
 	return __current_thread_info;
 }
diff --git a/arch/powerpc/include/asm/current.h b/arch/powerpc/include/asm/current.h
index bbfb94800415..9663d3737d5c 100644
--- a/arch/powerpc/include/asm/current.h
+++ b/arch/powerpc/include/asm/current.h
@@ -12,7 +12,7 @@ struct task_struct;
 #include <linux/stddef.h>
 #include <asm/paca.h>
 
-static inline struct task_struct *get_current(void)
+static inline notrace struct task_struct *get_current(void)
 {
 	struct task_struct *task;
 
diff --git a/arch/sh/include/asm/thread_info.h b/arch/sh/include/asm/thread_info.h
index 471db5173036..9e1ccdd70a7d 100644
--- a/arch/sh/include/asm/thread_info.h
+++ b/arch/sh/include/asm/thread_info.h
@@ -63,7 +63,7 @@ struct thread_info {
 register unsigned long current_stack_pointer asm("r15") __used;
 
 /* how to get the thread information struct from C */
-static inline struct thread_info *current_thread_info(void)
+static inline notrace struct thread_info *current_thread_info(void)
 {
 	struct thread_info *ti;
 #if defined(CONFIG_CPU_HAS_SR_RB)
-- 
2.53.0-Meta

[PATCH v2 11/16] rcu: Mark the Tiny RCU rcu_is_watching() notrace

[Song Liu]

ftrace_test_recursion_trylock() calls rcu_is_watching() before it has
taken the recursion lock. The Tree RCU version is a notrace function,
but the Tiny RCU version is a plain inline function. An out-of-line copy
of it is untraced only because inline implies notrace.

Mark it notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/rcutiny.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index f519cd680228..2a61d17e0e4e 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -127,7 +127,7 @@ static inline void rcu_preempt_deferred_qs(struct task_struct *t) { }
 void rcu_scheduler_starting(void);
 static inline void rcu_end_inkernel_boot(void) { }
 static inline bool rcu_inkernel_boot_has_ended(void) { return true; }
-static inline bool rcu_is_watching(void) { return true; }
+static inline bool notrace rcu_is_watching(void) { return true; }
 static inline void rcu_momentary_eqs(void) { }
 
 /* Avoid RCU read-side critical sections leaking across. */
-- 
2.53.0-Meta

[PATCH v2 12/16] tracing: Mark the tracing_gen_ctx helpers notrace

[Song Liu]

tracing_gen_ctx(), tracing_gen_ctx_flags() and tracing_gen_ctx_dec()
are called by ftrace callbacks before they take any recursion
protection, for example function_test_events_call() in
kernel/trace/trace_events.c, which is built with -pg when
CONFIG_FUNCTION_SELF_TRACING=y. They are plain inline functions, so an
out-of-line copy is untraced only because inline implies notrace.

Mark them notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/trace_events.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h
index 5cbd09c8be8d..15990e23a662 100644
--- a/include/linux/trace_events.h
+++ b/include/linux/trace_events.h
@@ -197,13 +197,13 @@ enum trace_flag_type {
 	TRACE_FLAG_BH_OFF		= 0x80,
 };
 
-static inline unsigned int tracing_gen_ctx_flags(unsigned long irqflags)
+static inline unsigned int notrace tracing_gen_ctx_flags(unsigned long irqflags)
 {
 	unsigned int irq_status = irqs_disabled_flags(irqflags) ?
 		TRACE_FLAG_IRQS_OFF : 0;
 	return tracing_gen_ctx_irq_test(irq_status);
 }
-static inline unsigned int tracing_gen_ctx(void)
+static inline unsigned int notrace tracing_gen_ctx(void)
 {
 	unsigned long irqflags;
 
@@ -211,7 +211,7 @@ static inline unsigned int tracing_gen_ctx(void)
 	return tracing_gen_ctx_flags(irqflags);
 }
 
-static inline unsigned int tracing_gen_ctx_dec(void)
+static inline unsigned int notrace tracing_gen_ctx_dec(void)
 {
 	unsigned int trace_ctx;
 
-- 
2.53.0-Meta

[PATCH v2 13/16] ktime: Mark ktime_to_ns() notrace

[Song Liu]

The NMI safe accessors used as trace clocks, such as
ktime_get_mono_fast_ns() and ktime_get_boot_fast_ns(), are notrace and
call ktime_to_ns(). ktime_to_ns() is a plain inline function, so an
out-of-line copy is untraced only because inline implies notrace, and
tracing it would recurse through the trace clock.

Mark it notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/ktime.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/ktime.h b/include/linux/ktime.h
index f247e564602f..ed56774f68ea 100644
--- a/include/linux/ktime.h
+++ b/include/linux/ktime.h
@@ -75,7 +75,7 @@ static inline ktime_t timespec64_to_ktime(struct timespec64 ts)
 #define ktime_to_timespec64(kt)		ns_to_timespec64((kt))
 
 /* Convert ktime_t to nanoseconds */
-static inline s64 ktime_to_ns(const ktime_t kt)
+static inline s64 notrace ktime_to_ns(const ktime_t kt)
 {
 	return kt;
 }
-- 
2.53.0-Meta

[PATCH v2 14/16] kallsyms: Mark the kernel text address checks notrace

[Song Liu]

Commit c0d80ddab899 ("kernel/extable.c: mark core_kernel_text notrace")
made core_kernel_text() notrace, because the MIPS function graph tracer
calls it and tracing it recursed without bound. core_kernel_text()
calls is_kernel_text() and is_kernel_inittext(), and is_kernel_text()
calls __is_kernel_text(). These are plain inline functions, so an
out-of-line copy of any of them is untraced only because inline implies
notrace.

Mark them notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/asm-generic/sections.h | 4 ++--
 include/linux/kallsyms.h       | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h
index 0755bc39b0d8..d478ddc02316 100644
--- a/include/asm-generic/sections.h
+++ b/include/asm-generic/sections.h
@@ -193,7 +193,7 @@ static inline bool is_kernel_ro_after_init(unsigned long addr)
  *
  * Returns: true if the address is located in .init.text, false otherwise.
  */
-static inline bool is_kernel_inittext(unsigned long addr)
+static inline bool notrace is_kernel_inittext(unsigned long addr)
 {
 	return addr >= (unsigned long)_sinittext &&
 	       addr < (unsigned long)_einittext;
@@ -208,7 +208,7 @@ static inline bool is_kernel_inittext(unsigned long addr)
  * Returns: true if the address is located in .text, false otherwise.
  * Note: an internal helper, only check the range of _stext to _etext.
  */
-static inline bool __is_kernel_text(unsigned long addr)
+static inline bool notrace __is_kernel_text(unsigned long addr)
 {
 	return addr >= (unsigned long)_stext &&
 	       addr < (unsigned long)_etext;
diff --git a/include/linux/kallsyms.h b/include/linux/kallsyms.h
index d5dd54c53ace..93c830b35aeb 100644
--- a/include/linux/kallsyms.h
+++ b/include/linux/kallsyms.h
@@ -24,7 +24,7 @@
 struct cred;
 struct module;
 
-static inline int is_kernel_text(unsigned long addr)
+static inline int notrace is_kernel_text(unsigned long addr)
 {
 	if (__is_kernel_text(addr))
 		return 1;
-- 
2.53.0-Meta

[PATCH v2 15/16] uaccess: Mark pagefault_disable() and pagefault_enable() notrace

[Song Liu]

copy_to_kernel_nofault() and copy_to_user_nofault() are declared
notrace in <linux/uaccess.h>. They call pagefault_disable() and
pagefault_enable(), which are plain inline functions, so an out-of-line
copy is untraced only because inline implies notrace.

Mark them notrace explicitly. No functional change.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/uaccess.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index eddbbb65ccc4..ee4966ba9c1e 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -260,7 +260,7 @@ static __always_inline void pagefault_disabled_dec(void)
  * User access methods will not sleep when called from a pagefault_disabled()
  * environment.
  */
-static inline void pagefault_disable(void)
+static inline void notrace pagefault_disable(void)
 {
 	pagefault_disabled_inc();
 	/*
@@ -270,7 +270,7 @@ static inline void pagefault_disable(void)
 	barrier();
 }
 
-static inline void pagefault_enable(void)
+static inline void notrace pagefault_enable(void)
 {
 	/*
 	 * make sure to issue those last loads/stores before enabling
-- 
2.53.0-Meta

[PATCH v2 16/16] compiler_types: Move notrace from inline to __always_inline

[Song Liu]

Commit 45959ee7aa64 ("ftrace: Do not function trace inlined functions")
added notrace to the inline macro. At the time, inline was only a hint
with CONFIG_OPTIMIZE_INLINING, and whether a given inline function got
an mcount call depended on the gcc version. Some functions that must
not be traced, such as the paravirt native_*() helpers, were marked
inline and relied on being inlined, so a different gcc could silently
break the function tracer. Making all inline functions notrace removed
that dependency on the compiler.

That trade-off no longer makes sense. CONFIG_OPTIMIZE_INLINING was made
always on by commit ac7c3e4ff401 ("compiler: enable
CONFIG_OPTIMIZE_INLINING forcibly") and removed by commit 889b3c1245de
("compiler: remove CONFIG_OPTIMIZE_INLINING entirely"). The compiler
also inlines static functions that are not marked inline, and creates
.constprop, .isra and .part clones, so the set of traceable functions
already depends on the compiler. notrace on inline does not make it
consistent. It only hides the inline functions that the compiler chose
not to inline. Hidden functions cannot be traced with ftrace or used
as BPF fentry/fexit targets. They also cannot be livepatched, because
livepatch redirects functions through ftrace, so a fix to one of them
has to be shipped by livepatching its callers instead.

A function that must not be traced should say so. The previous patches
mark notrace the inline helpers that relied on inline implying notrace:
the paravirt native helpers, the helpers used by resume and kexec, the
helpers used by bpf_spin_lock, the BPF trampoline and bpf_mem_alloc,
and the helpers used by ftrace recursion protection, tracepoints, SRCU,
the trace clocks, core_kernel_text() and the nofault accessors.

Keep notrace on __always_inline. Such functions are meant to be inlined
into their callers. An out-of-line copy exists only when the address of
the function is taken or when it has external linkage, and those copies
are mostly low-level helpers used by entry, context switch and tracing
code.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
---
 include/linux/compiler_attributes.h | 6 +++++-
 include/linux/compiler_types.h      | 3 +--
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h
index 476c4c560d17..5fdc07b3e380 100644
--- a/include/linux/compiler_attributes.h
+++ b/include/linux/compiler_attributes.h
@@ -49,10 +49,14 @@
  * to its docs (and also "warning: always_inline function might not be
  * inlinable [-Wattributes]" is emitted).
  *
+ * The compiler still emits an out-of-line copy of an __always_inline function
+ * when its address is taken or when it has external linkage. These functions
+ * are often used by code that must not be traced, so mark them notrace.
+ *
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-always_005finline-function-attribute
  * clang: mentioned
  */
-#define __always_inline                 inline __attribute__((__always_inline__))
+#define __always_inline                 inline __attribute__((__always_inline__)) notrace
 
 /*
  * The second argument is optional (default 0), so we use a variadic macro
diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index c5921f139007..ea7f2a928b4d 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -230,9 +230,8 @@ struct ftrace_likely_data {
  * externally visible function. This makes extern inline behave as per gnu89
  * semantics rather than c99. This prevents multiple symbol definition errors
  * of extern inline functions at link time.
- * A lot of inline functions can cause havoc with function tracing.
  */
-#define inline inline __gnu_inline __inline_maybe_unused notrace
+#define inline inline __gnu_inline __inline_maybe_unused
 
 /*
  * gcc provides both __inline__ and __inline as alternate spellings of
-- 
2.53.0-Meta