[PATCH v2 00/16] Move notrace from inline to __always_inline

[Song Liu <song@kernel.org>]

Since commit 45959ee7aa64 ("ftrace: Do not function trace inlined
functions"), every function marked inline is also notrace. When the
compiler does not inline such a function, the out-of-line copy cannot
be traced with ftrace, cannot be a BPF fentry/fexit target, and cannot
be livepatched. This series moves notrace from inline to
__always_inline, so these copies become traceable. __always_inline
functions, which are mostly low-level helpers, stay untraced.

v1 [1] removed notrace from inline in one patch. Sashiko's review
showed that some code relies on inline implying notrace, and Steven
suggested keeping notrace on __always_inline and annotating the rest.
Patches 1-15 add notrace explicitly to the inline helpers that need it.
While inline still implies notrace, they do not change the generated
code: on x86_64 with gcc 11.5, vmlinux .text is byte-identical with and
without them.

 - 1-2: x86 paravirt native helpers, and the helpers used by resume and
   kexec.
 - 3-6: helpers used by bpf_spin_lock, the BPF trampoline and
   bpf_mem_alloc, where an fentry program on one of them could
   deadlock, recurse without bound or corrupt the free lists.
 - 7-8: tracepoint and SRCU helpers used by tracing.
 - 9-12: helpers used by ftrace recursion protection: the accessors
   behind preempt_count() and current on arm64, powerpc, csky, mips and
   sh, the Tiny RCU rcu_is_watching(), and the tracing_gen_ctx helpers.
 - 13-15: helpers called by notrace code in the trace clocks,
   core_kernel_text() and the nofault accessors.

To find these, I compared x86_64 builds with and without patch 16, and
listed the functions that become traceable and are called from code
without an fentry call (notrace, noinstr, or built without -pg),
including calls through pv_ops. To also cover helpers that gcc happens
to inline today, I repeated this with -fno-inline. The changes for
other architectures and Tiny RCU come from reading the code that ftrace
recursion protection uses.

On an x86_64 config with CONFIG_PARAVIRT_XXL and gcc 11.5, the series
makes 678 more functions traceable, and none become untraceable. In
QEMU, with ftrace filtered to native_load_idt(), a kernel with v1 dies
on resume from S3, as Sashiko predicted. With this series, S3
suspend/resume, CPU hotplug and hibernation test mode pass with ftrace
on all native_* functions, and repeated kexec passes with ftrace on all
functions. The series also builds for i386, and for arm64 and powerpc
with clang.

Changes v1 => v2:
 - Keep notrace on __always_inline. (Steven Rostedt)
 - Add patches 1-15 to mark notrace the helpers that relied on inline
   implying notrace.
 - Rework the commit message of the last patch.

v1: https://lore.kernel.org/all/20261021222255.3124940-1-song@kernel.org/

Song Liu (16):
  x86/paravirt: Mark native pv_ops helpers notrace
  x86: Mark CPU state helpers used by resume and kexec notrace
  bpf: Mark bpf_spin_lock helpers notrace
  bpf: Mark the BPF trampoline helpers notrace
  bpf: Mark the bpf_mem_alloc fast path helpers notrace
  u64_stats: Mark the update helpers used by BPF notrace
  tracepoint: Mark __do_trace_##name() notrace
  srcu: Mark the fast notrace read-side helpers notrace
  arm64: Mark the preempt count helpers notrace
  powerpc, csky, mips, sh: Mark the accessors behind current notrace
  rcu: Mark the Tiny RCU rcu_is_watching() notrace
  tracing: Mark the tracing_gen_ctx helpers notrace
  ktime: Mark ktime_to_ns() notrace
  kallsyms: Mark the kernel text address checks notrace
  uaccess: Mark pagefault_disable() and pagefault_enable() notrace
  compiler_types: Move notrace from inline to __always_inline

 arch/arm64/include/asm/preempt.h         | 12 +++++-----
 arch/csky/include/asm/thread_info.h      |  2 +-
 arch/mips/include/asm/thread_info.h      |  2 +-
 arch/powerpc/include/asm/current.h       |  2 +-
 arch/sh/include/asm/thread_info.h        |  2 +-
 arch/x86/include/asm/cpuid/api.h         |  4 ++--
 arch/x86/include/asm/desc.h              | 17 +++++++-------
 arch/x86/include/asm/gsseg.h             |  4 ++--
 arch/x86/include/asm/io.h                |  2 +-
 arch/x86/include/asm/io_bitmap.h         |  2 +-
 arch/x86/include/asm/local.h             |  8 +++----
 arch/x86/include/asm/msr.h               |  8 +++----
 arch/x86/include/asm/paravirt-spinlock.h |  2 +-
 arch/x86/include/asm/paravirt.h          | 20 ++++++++--------
 arch/x86/include/asm/pgalloc.h           |  2 +-
 arch/x86/include/asm/pgtable_64.h        | 10 ++++----
 arch/x86/include/asm/processor.h         |  2 +-
 arch/x86/include/asm/segment.h           |  4 ++--
 arch/x86/include/asm/special_insns.h     |  8 +++----
 include/asm-generic/sections.h           |  4 ++--
 include/linux/bpf.h                      |  8 +++----
 include/linux/compiler_attributes.h      |  6 ++++-
 include/linux/compiler_types.h           |  3 +--
 include/linux/kallsyms.h                 |  2 +-
 include/linux/ktime.h                    |  2 +-
 include/linux/llist.h                    | 16 ++++++-------
 include/linux/percpu-refcount.h          | 12 +++++-----
 include/linux/pgtable.h                  | 12 +++++-----
 include/linux/rcupdate.h                 | 14 +++++------
 include/linux/rcupdate_trace.h           |  4 ++--
 include/linux/rcutiny.h                  |  2 +-
 include/linux/sched.h                    |  8 +++----
 include/linux/srcu.h                     | 10 ++++----
 include/linux/srcutiny.h                 |  5 ++--
 include/linux/srcutree.h                 |  2 +-
 include/linux/trace_events.h             |  6 ++---
 include/linux/tracepoint.h               |  4 ++--
 include/linux/u64_stats_sync.h           | 30 ++++++++++++------------
 include/linux/uaccess.h                  |  4 ++--
 kernel/bpf/helpers.c                     | 12 +++++-----
 40 files changed, 143 insertions(+), 136 deletions(-)

--
2.53.0-Meta