mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace
@ 2026-11-17  4:34 Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 01/12] arm64, unwind: build with .eh_frame and add gen_lr_unwind skeleton Dylan Hatch
                   ` (11 more replies)
  0 siblings, 12 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Implement a build-time .eh_frame-to-.lr_unwind table generator and
in-kernel unwinder for arm64. The primary goal is to support reliable
stacktrace (arch_stack_walk_reliable()) across in-kernel exception
boundaries (pt_regs).

On arm64, normal stack unwinding follows frame pointer (x29) chains.
However, when an asynchronous exception or interrupt occurs inside the
kernel, the frame pointer chain alone is insufficient to unwind reliably
from the interrupted context (pt_regs):

  1. Before the prologue updates x29 (mov x29, sp), after the epilogue
     restores x29/x30, or throughout a leaf function, the interrupted
     function's return address is live in x30 (pt_regs->regs[30]) while
     x29 still points to the caller's frame record.
  2. Between saving the frame record on the stack (e.g.
     stp x29, x30, [sp, #-N]!) and updating x29 (mov x29, sp), or in
     functions where the compiler restores x29/x30 before adjusting sp,
     the interrupted function's frame record resides at [sp + sp_off]
     while x29 still points to the caller's frame record.
  3. Inside the function body after mov x29, sp, [x29] points to the
     interrupted function's own frame record and x30 may have been
     clobbered by nested calls.

Without knowing which state holds at the interrupted PC, the unwinder
cannot determine whether pt_regs->regs[30] is a valid frame or a
duplicate of [x29 + 8], nor can it recover a frame record at
[sp + sp_off] before x29 is updated, forcing arch_stack_walk_reliable()
to reject stack traces that cross an in-kernel exception boundary.

This RFC is proposed as an alternative to an SFrame [1] based solution:
[2]. While SFrame works well with recent GNU binutils (2.46+), it is not
yet supported by LLVM/Clang toolchains. Conversely, compiler-generated
.eh_frame tables (-fasynchronous-unwind-tables, enabled by
CONFIG_UNWIND_TABLES) are supported today by both GCC and Clang/LLVM,
and arm64 already includes a DWARF .eh_frame parser in
arch/arm64/kernel/pi/patch-scs.c for dynamic Shadow Call Stack patching.

Rather than retaining bulky raw .eh_frame sections in kernel memory and
interpreting variable-length DWARF CFI opcodes during stack unwinding,
this series introduces a build-time host tool (scripts/gen_lr_unwind)
that reuses and extends arm64's .eh_frame parser (<asm/eh_frame.h>) to
distill .eh_frame into compact, fixed-width .lr_unwind_fde and
.lr_unwind_ranges sections, and strips .eh_frame from the final vmlinux
and module binaries.

Each annotated function is described by an 8-byte struct lr_fde entry
(32-bit PC-relative start_offset + 32-bit range_desc), sorted at build
time by scripts/sorttable (and at module load time) for O(log N) lookup.
Each instruction range within a function is classified into one of three
exception-boundary unwind modes:
  - LR_UNWIND_LIVE: x30 holds the return address, x29 is the caller's FP
  - LR_UNWIND_FP:   [x29] holds the saved {x29, x30} frame record
  - LR_UNWIND_SP:   [sp + sp_off] holds the saved {x29, x30} frame record

Functions with non-standard x30 spills (such as frameless leaf functions
spilling x30 without x29) are omitted from .lr_unwind_fde so the
unwinder treats them as unreliable.

For functions following the canonical 4-phase LIVE -> FP/SP -> SP -> LIVE
layout (prologue <= 31 insns, body <= 4095 insns, epilogue/return <= 15
insns each, and sp_off <= 496 bytes), the ranges and sp_off are packed
directly into the 32-bit range_desc word of struct lr_fde; functions
with more ranges or larger lengths reference 2-byte lr_range_t entries
in .lr_unwind_ranges. Across vmlinux, .lr_unwind_fde and
.lr_unwind_ranges total ~800-935 KiB (~4x smaller than raw .eh_frame).

This series is available on github: [3].

Structure of the series:
  - Patches 1-6: Enable .eh_frame generation for CONFIG_ARM64_LR_UNWIND,
    factor the .eh_frame parser out of patch-scs.c into <asm/eh_frame.h>,
    extend it to track CFA and x29/x30 state, and implement
    scripts/gen_lr_unwind (including --dump) to emit .lr_unwind sections.
  - Patch 7: Teach scripts/sorttable to sort .lr_unwind_fde at build time.
  - Patches 8-9: Add in-kernel .lr_unwind lookup, optional boot/module
    validation (CONFIG_ARM64_LR_UNWIND_VALIDATION), and module support.
  - Patches 10-11: Annotate call_on_irq_stack() and arm64 leaf assembly
    functions (SYM_LEAF_FUNC_START / SYM_LEAF_FUNC_END) with CFI
    directives so they are covered by .lr_unwind.
  - Patch 12: Use .lr_unwind in arch/arm64/kernel/stacktrace.c to
    reliably unwind across in-kernel exception boundaries.

Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/all/20261117020648.323464-1-dylanbhatch@google.com/
[3]: https://github.com/dylanbhatch/linux/tree/eh_frame

Dylan Hatch (11):
  arm64, unwind: build with .eh_frame and add gen_lr_unwind skeleton
  arm64: factor .eh_frame parser out of patch-scs.c into
    <asm/eh_frame.h>
  arm64: eh_frame: track CFA and x29/x30 register state in CFI parser
  arm64, unwind: parse .eh_frame into .lr_unwind entries in
    gen_lr_unwind
  scripts/gen_lr_unwind: emit .lr_unwind_fde and .lr_unwind_ranges
  scripts/gen_lr_unwind: add --dump option to inspect .lr_unwind tables
  scripts/sorttable: sort .lr_unwind_fde by start_offset
  arm64, unwind: add in-kernel .lr_unwind lookup and validation
  arm64/module, unwind: add .lr_unwind support for modules
  arm64, crypto/lib: annotate leaf functions with CFI info
  unwind: arm64: use .lr_unwind to unwind interrupt frames

Weinan Liu (1):
  arm64: entry: add unwind info for call_on_irq_stack()

 MAINTAINERS                            |   1 +
 arch/arm64/Kconfig                     |  20 +
 arch/arm64/crypto/aes-ce-ccm-core.S    |  12 +-
 arch/arm64/crypto/aes-neonbs-core.S    |  16 +-
 arch/arm64/crypto/ghash-ce-core.S      |  20 +-
 arch/arm64/crypto/sm4-ce-ccm-core.S    |  16 +-
 arch/arm64/crypto/sm4-ce-cipher-core.S |   4 +-
 arch/arm64/crypto/sm4-ce-core.S        |  44 +-
 arch/arm64/crypto/sm4-ce-gcm-core.S    |  16 +-
 arch/arm64/crypto/sm4-neon-core.S      |  12 +-
 arch/arm64/include/asm/eh_frame.h      | 552 +++++++++++++++++
 arch/arm64/include/asm/linkage.h       |  33 +
 arch/arm64/include/asm/lr_unwind.h     | 149 +++++
 arch/arm64/include/asm/module.h        |   6 +
 arch/arm64/include/asm/module.lds.h    |   2 +-
 arch/arm64/include/asm/scs.h           |   7 -
 arch/arm64/include/asm/sections.h      |   3 +
 arch/arm64/kernel/Makefile             |   5 +
 arch/arm64/kernel/entry.S              |  12 +-
 arch/arm64/kernel/lr_unwind.c          | 395 ++++++++++++
 arch/arm64/kernel/module.c             |  11 +
 arch/arm64/kernel/pi/patch-scs.c       | 259 +-------
 arch/arm64/kernel/setup.c              |   3 +
 arch/arm64/kernel/stacktrace.c         |  85 ++-
 arch/arm64/kernel/vmlinux.lds.S        |  27 +-
 arch/arm64/lib/clear_user.S            |   4 +-
 arch/arm64/lib/copy_from_user.S        |   4 +-
 arch/arm64/lib/copy_page.S             |   4 +-
 arch/arm64/lib/copy_to_user.S          |   4 +-
 arch/arm64/lib/memchr.S                |   4 +-
 arch/arm64/lib/memcmp.S                |   4 +-
 arch/arm64/lib/memcpy.S                |   8 +-
 arch/arm64/lib/memset.S                |   8 +-
 arch/arm64/lib/mte.S                   |  28 +-
 arch/arm64/lib/strchr.S                |   4 +-
 arch/arm64/lib/strcmp.S                |   4 +-
 arch/arm64/lib/strlen.S                |   4 +-
 arch/arm64/lib/strncmp.S               |   4 +-
 arch/arm64/lib/strnlen.S               |   4 +-
 arch/arm64/lib/strrchr.S               |   4 +-
 arch/arm64/lib/tishift.S               |  12 +-
 scripts/.gitignore                     |   1 +
 scripts/Makefile                       |   2 +
 scripts/Makefile.modfinal              |  10 +-
 scripts/Makefile.vmlinux_o             |   8 +-
 scripts/gen_lr_unwind.c                | 821 +++++++++++++++++++++++++
 scripts/link-vmlinux.sh                |   2 +-
 scripts/sorttable.c                    |  50 +-
 48 files changed, 2325 insertions(+), 383 deletions(-)
 create mode 100644 arch/arm64/include/asm/eh_frame.h
 create mode 100644 arch/arm64/include/asm/lr_unwind.h
 create mode 100644 arch/arm64/kernel/lr_unwind.c
 create mode 100644 scripts/gen_lr_unwind.c


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 01/12] arm64, unwind: build with .eh_frame and add gen_lr_unwind skeleton
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
@ 2026-11-17  4:34 ` Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 02/12] arm64: factor .eh_frame parser out of patch-scs.c into <asm/eh_frame.h> Dylan Hatch
                   ` (10 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Add CONFIG_ARM64_LR_UNWIND to build the kernel with .eh_frame unwind
tables while discarding .eh_frame in the final vmlinux and module linker
scripts unless CONFIG_UNWIND_PATCH_PAC_INTO_SCS is enabled. Wire up a
skeleton scripts/gen_lr_unwind host tool invoked on vmlinux.o.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 MAINTAINERS                         |   1 +
 arch/arm64/Kconfig                  |  11 +++
 arch/arm64/include/asm/module.lds.h |   2 +-
 arch/arm64/kernel/vmlinux.lds.S     |  25 ++++-
 scripts/.gitignore                  |   1 +
 scripts/Makefile                    |   2 +
 scripts/Makefile.vmlinux_o          |   8 +-
 scripts/gen_lr_unwind.c             | 137 ++++++++++++++++++++++++++++
 scripts/link-vmlinux.sh             |   2 +-
 9 files changed, 185 insertions(+), 4 deletions(-)
 create mode 100644 scripts/gen_lr_unwind.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 93abbb315065d..c7c8673291702 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4047,6 +4047,7 @@ F:	include/linux/arm-rmi-cmds.h
 F:	include/linux/arm-rsi-cmds.h
 F:	include/linux/arm-smccc-rmi.h
 F:	include/linux/arm-smccc-rsi.h
+F:	scripts/gen_lr_unwind.c
 F:	tools/testing/selftests/arm64/
 X:	arch/arm64/boot/dts/
 X:	arch/arm64/configs/defconfig
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 1075bc6618330..aa09c7ea61be3 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2506,6 +2506,17 @@ config UNWIND_PATCH_PAC_INTO_SCS
 	select UNWIND_TABLES
 	select DYNAMIC_SCS
 
+config ARM64_LR_UNWIND
+	bool "Generate exception-boundary unwind tables from .eh_frame"
+	default y
+	select UNWIND_TABLES
+	help
+	  Generate compact .lr_unwind_fde and .lr_unwind_ranges sections at
+	  build time from DWARF .eh_frame unwind information to determine how
+	  to unwind across exception boundaries (whether the return address is
+	  live in x30, saved in the frame record at [x29], or saved in the
+	  frame record at an SP-relative offset).
+
 config ARM64_CONTPTE
 	bool "Contiguous PTE mappings for user memory" if EXPERT
 	depends on TRANSPARENT_HUGEPAGE
diff --git a/arch/arm64/include/asm/module.lds.h b/arch/arm64/include/asm/module.lds.h
index 603f92a8a73ed..84e5fa0250435 100644
--- a/arch/arm64/include/asm/module.lds.h
+++ b/arch/arm64/include/asm/module.lds.h
@@ -4,7 +4,7 @@ SECTIONS {
 	.text.ftrace_trampoline 0 : { BYTE(0) }
 	.init.text.ftrace_trampoline 0 : { BYTE(0) }
 
-#ifdef CONFIG_UNWIND_TABLES
+#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS
 	/*
 	 * Currently, we only use unwind info at module load time, so we can
 	 * put it into the .init allocation.
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 1869b44ae1909..b8f2963d10030 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -144,15 +144,35 @@ jiffies = jiffies_64;
 #define TRAMP_TEXT
 #endif
 
-#ifdef CONFIG_UNWIND_TABLES
+#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS
 #define UNWIND_DATA_SECTIONS				\
 	.eh_frame : {					\
 		__pi___eh_frame_start = .;		\
 		*(.eh_frame)				\
 		__pi___eh_frame_end = .;		\
 	}
+#define EH_FRAME_DISCARDS
 #else
 #define UNWIND_DATA_SECTIONS
+#define EH_FRAME_DISCARDS	*(.eh_frame)
+#endif
+
+#ifdef CONFIG_ARM64_LR_UNWIND
+#define LR_UNWIND_SECTIONS				\
+	. = ALIGN(4);					\
+	.lr_unwind_fde : {				\
+		__start_lr_unwind_fde = .;		\
+		KEEP(*(.lr_unwind_fde))			\
+		__stop_lr_unwind_fde = .;		\
+	}						\
+	. = ALIGN(2);					\
+	.lr_unwind_ranges : {				\
+		__start_lr_unwind_ranges = .;		\
+		KEEP(*(.lr_unwind_ranges))		\
+		__stop_lr_unwind_ranges = .;		\
+	}
+#else
+#define LR_UNWIND_SECTIONS
 #endif
 
 /*
@@ -184,6 +204,7 @@ SECTIONS
 		*(.interp .dynamic)
 		*(.dynsym .dynstr .hash .gnu.hash)
 		*(.ARM.attributes)
+		EH_FRAME_DISCARDS
 	}
 
 	. = KIMAGE_VADDR;
@@ -212,6 +233,8 @@ SECTIONS
 	/* everything from this point to __init_begin will be marked RO NX */
 	RO_DATA(PAGE_SIZE)
 
+	LR_UNWIND_SECTIONS
+
 	HYPERVISOR_RODATA_SECTIONS
 
 #ifdef CONFIG_NVHE_EL2_TRACING
diff --git a/scripts/.gitignore b/scripts/.gitignore
index 7004565439107..a56a49fee309c 100644
--- a/scripts/.gitignore
+++ b/scripts/.gitignore
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 /asn1_compiler
 /gen_init_cpio
+/gen_lr_unwind
 /gen_packed_field_checks
 /generate_rust_target
 /insert-sys-cert
diff --git a/scripts/Makefile b/scripts/Makefile
index d7fbb7ca098b1..cb5af9339e4ac 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -12,6 +12,7 @@ hostprogs-always-$(CONFIG_SYSTEM_EXTRA_CERTIFICATE)	+= insert-sys-cert
 hostprogs-always-$(CONFIG_RUST_KERNEL_DOCTESTS)		+= rustdoc_test_builder
 hostprogs-always-$(CONFIG_RUST_KERNEL_DOCTESTS)		+= rustdoc_test_gen
 hostprogs-always-$(CONFIG_TRACEPOINTS)			+= tracepoint-update
+hostprogs-always-$(CONFIG_ARM64_LR_UNWIND)		+= gen_lr_unwind
 
 kallsyms-objs := kallsyms.o kallsyms-sysmap.o elf-parse.o
 sorttable-objs := sorttable.o elf-parse.o
@@ -31,6 +32,7 @@ rustdoc_test_builder-rust := y
 rustdoc_test_gen-rust := y
 
 HOSTCFLAGS_tracepoint-update.o = -I$(srctree)/tools/include
+HOSTCFLAGS_gen_lr_unwind.o = -I$(srctree)/tools/include
 HOSTCFLAGS_kallsyms.o = -I$(srctree)/tools/include
 HOSTCFLAGS_kallsyms-sysmap.o = -I$(srctree)/tools/include
 HOSTCFLAGS_elf-parse.o = -I$(srctree)/tools/include
diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
index 0b37bd5dc3a49..362382ec58b15 100644
--- a/scripts/Makefile.vmlinux_o
+++ b/scripts/Makefile.vmlinux_o
@@ -57,6 +57,10 @@ objtool-args = $(vmlinux-objtool-args-y) --link
 
 vmlinux-o-ld-args-$(CONFIG_BUILTIN_MODULE_RANGES)	+= -Map=$@.map
 
+gen_lr_unwind := $(objtree)/scripts/gen_lr_unwind
+cmd_gen_lr_unwind = $(if $(CONFIG_ARM64_LR_UNWIND), ; $(gen_lr_unwind) $@)
+cmd_gen_lr_unwinddep = $(if $(CONFIG_ARM64_LR_UNWIND), { echo ; echo '$@: $$(wildcard $(gen_lr_unwind))' ; } >> $(dot-target).cmd)
+
 quiet_cmd_ld_vmlinux.o = LD      $@
       cmd_ld_vmlinux.o = \
 	$(LD) ${KBUILD_LDFLAGS} -r -o $@ \
@@ -64,13 +68,15 @@ quiet_cmd_ld_vmlinux.o = LD      $@
 	$(addprefix -T , $(initcalls-lds)) \
 	--whole-archive vmlinux.a --no-whole-archive \
 	--start-group $(KBUILD_VMLINUX_LIBS) --end-group \
-	$(cmd_objtool)
+	$(cmd_objtool) \
+	$(cmd_gen_lr_unwind)
 
 cmd_check_function_names = $(srctree)/scripts/check-function-names.sh $@
 
 define rule_ld_vmlinux.o
 	$(call cmd_and_savecmd,ld_vmlinux.o)
 	$(call cmd,gen_objtooldep)
+	$(call cmd,gen_lr_unwinddep)
 	$(call cmd,check_function_names)
 endef
 
diff --git a/scripts/gen_lr_unwind.c b/scripts/gen_lr_unwind.c
new file mode 100644
index 0000000000000..1b52662838653
--- /dev/null
+++ b/scripts/gen_lr_unwind.c
@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Generate .lr_unwind_fde and .lr_unwind_ranges sections from .eh_frame
+ * for arm64 exception-boundary unwinding.
+ */
+
+#include <elf.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+struct lr_unwind_ctx {
+	const void	*map;
+	size_t		map_size;
+
+	const Elf64_Shdr *eh_sec;
+	const Elf64_Shdr *rela_eh_sec;
+	const Elf64_Shdr *fde_sec;
+	unsigned int	symtab_shndx;
+};
+
+static int parse_eh_frame(struct lr_unwind_ctx *ctx)
+{
+	/* TODO: Parse CIEs/FDEs from .eh_frame and .rela.eh_frame */
+	(void)ctx;
+	return 0;
+}
+
+static int emit_lr_unwind_sections(const char *fname, struct lr_unwind_ctx *ctx,
+				   unsigned int orig_shnum,
+				   unsigned int shstrndx)
+{
+	/* TODO: Emit .lr_unwind_fde, .rela.lr_unwind_fde, and .lr_unwind_ranges */
+	(void)fname;
+	(void)ctx;
+	(void)orig_shnum;
+	(void)shstrndx;
+	return 0;
+}
+
+int main(int argc, char *argv[])
+{
+	struct lr_unwind_ctx ctx = { 0 };
+	unsigned int shnum, shstrndx;
+	const Elf64_Ehdr *ehdr;
+	const Elf64_Shdr *shdrs;
+	const char *shstrtab;
+	struct stat sb;
+	int fd, ret = 1;
+
+	if (argc != 2) {
+		fprintf(stderr, "Usage: %s <file>\n", argv[0]);
+		return 1;
+	}
+
+	fd = open(argv[1], O_RDONLY);
+	if (fd < 0) {
+		fprintf(stderr, "gen_lr_unwind: cannot open %s: %s\n",
+			argv[1], strerror(errno));
+		return 1;
+	}
+	if (fstat(fd, &sb) < 0 || (size_t)sb.st_size < sizeof(Elf64_Ehdr)) {
+		fprintf(stderr, "gen_lr_unwind: invalid file %s\n", argv[1]);
+		close(fd);
+		return 1;
+	}
+
+	ctx.map_size = sb.st_size;
+	ctx.map = mmap(NULL, ctx.map_size, PROT_READ, MAP_PRIVATE, fd, 0);
+	close(fd);
+	if (ctx.map == MAP_FAILED) {
+		fprintf(stderr, "gen_lr_unwind: mmap failed on %s: %s\n",
+			argv[1], strerror(errno));
+		return 1;
+	}
+
+	ehdr = ctx.map;
+	if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) ||
+	    ehdr->e_ident[EI_CLASS] != ELFCLASS64 ||
+	    ehdr->e_ident[EI_DATA] != ELFDATA2LSB ||
+	    ehdr->e_type != ET_REL ||
+	    ehdr->e_machine != EM_AARCH64) {
+		fprintf(stderr, "gen_lr_unwind: %s is not an AArch64 ET_REL ELF\n",
+			argv[1]);
+		goto out;
+	}
+
+	shdrs = ctx.map + ehdr->e_shoff;
+	shnum = ehdr->e_shnum == SHN_UNDEF ? shdrs[0].sh_size : ehdr->e_shnum;
+	shstrndx = ehdr->e_shstrndx == SHN_XINDEX ?
+		   shdrs[0].sh_link : ehdr->e_shstrndx;
+	shstrtab = ctx.map + shdrs[shstrndx].sh_offset;
+
+	for (unsigned int i = 0; i < shnum; i++) {
+		const char *name = shstrtab + shdrs[i].sh_name;
+
+		if (!strcmp(name, ".lr_unwind_fde"))
+			ctx.fde_sec = &shdrs[i];
+		else if (!strcmp(name, ".eh_frame") ||
+			 !strcmp(name, ".init.eh_frame"))
+			ctx.eh_sec = &shdrs[i];
+		else if (!strcmp(name, ".rela.eh_frame") ||
+			 !strcmp(name, ".rela.init.eh_frame"))
+			ctx.rela_eh_sec = &shdrs[i];
+		else if (!strcmp(name, ".symtab"))
+			ctx.symtab_shndx = i;
+	}
+
+	if (ctx.fde_sec) {
+		ret = 0;
+		goto out;
+	}
+
+	if (!ctx.eh_sec || !ctx.rela_eh_sec || !ctx.symtab_shndx) {
+		ret = 0;
+		goto out;
+	}
+
+	if (!parse_eh_frame(&ctx) &&
+	    !emit_lr_unwind_sections(argv[1], &ctx, shnum, shstrndx))
+		ret = 0;
+	else
+		fprintf(stderr, "gen_lr_unwind: failed to generate .lr_unwind for %s\n",
+			argv[1]);
+
+out:
+	if (ctx.map)
+		munmap((void *)ctx.map, ctx.map_size);
+	return ret;
+}
diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index d5710e900bd40..9c8bfb6ff05e0 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -61,7 +61,7 @@ vmlinux_link()
 	shift
 
 	if is_enabled CONFIG_LTO_CLANG || is_enabled CONFIG_X86_KERNEL_IBT ||
-	   is_enabled CONFIG_KLP_BUILD; then
+	   is_enabled CONFIG_KLP_BUILD || is_enabled CONFIG_ARM64_LR_UNWIND; then
 		# Use vmlinux.o instead of performing the slow LTO link again.
 		objs=vmlinux.o
 		libs=
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 02/12] arm64: factor .eh_frame parser out of patch-scs.c into <asm/eh_frame.h>
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 01/12] arm64, unwind: build with .eh_frame and add gen_lr_unwind skeleton Dylan Hatch
@ 2026-11-17  4:34 ` Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 03/12] arm64: eh_frame: track CFA and x29/x30 register state in CFI parser Dylan Hatch
                   ` (9 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Move the DWARF .eh_frame CIE/FDE iterator and opcode stepper out of
arch/arm64/kernel/pi/patch-scs.c into a shared <asm/eh_frame.h> header
so it can be reused by build-time unwind table generation.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/include/asm/eh_frame.h | 329 ++++++++++++++++++++++++++++++
 arch/arm64/include/asm/scs.h      |   7 -
 arch/arm64/kernel/pi/patch-scs.c  | 258 +++--------------------
 3 files changed, 358 insertions(+), 236 deletions(-)
 create mode 100644 arch/arm64/include/asm/eh_frame.h

diff --git a/arch/arm64/include/asm/eh_frame.h b/arch/arm64/include/asm/eh_frame.h
new file mode 100644
index 0000000000000..6c17fed1860f1
--- /dev/null
+++ b/arch/arm64/include/asm/eh_frame.h
@@ -0,0 +1,329 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Minimal DWARF .eh_frame CFI parser for arm64.
+ *
+ * This minimal DWARF CFI parser is partially based on the code in
+ * arch/arc/kernel/unwind.c, and on the document below:
+ * https://refspecs.linuxbase.org/LSB_4.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html
+ */
+#ifndef __ASM_EH_FRAME_H
+#define __ASM_EH_FRAME_H
+
+#include <linux/bits.h>
+#include <linux/compiler.h>
+#include <linux/stddef.h>
+#include <linux/string.h>
+#include <linux/types.h>
+
+#ifndef U32_MAX
+#define U32_MAX				((u32)~0U)
+#endif
+
+#define AARCH64_DWARF_LR		30
+
+#define DW_CFA_nop			0x00
+#define DW_CFA_advance_loc1		0x02
+#define DW_CFA_advance_loc2		0x03
+#define DW_CFA_advance_loc4		0x04
+#define DW_CFA_offset_extended		0x05
+#define DW_CFA_restore_extended		0x06
+#define DW_CFA_same_value		0x08
+#define DW_CFA_remember_state		0x0a
+#define DW_CFA_restore_state		0x0b
+#define DW_CFA_def_cfa			0x0c
+#define DW_CFA_def_cfa_register		0x0d
+#define DW_CFA_def_cfa_offset		0x0e
+#define DW_CFA_def_cfa_offset_sf	0x13
+#define DW_CFA_negate_ra_state		0x2d
+
+#define DW_EH_PE_sdata4			0x0b
+#define DW_EH_PE_sdata8			0x0c
+#define DW_EH_PE_pcrel			0x10
+
+enum {
+	EDYNSCS_INVALID_CIE_HEADER		= 1,
+	EDYNSCS_INVALID_CIE_SDATA_SIZE		= 2,
+	EDYNSCS_INVALID_FDE_AUGM_DATA_SIZE	= 3,
+	EDYNSCS_INVALID_CFA_OPCODE		= 4,
+};
+
+struct eh_frame_cfi_state {
+	u64	loc;
+	bool	ra_negated;
+};
+
+struct eh_frame {
+	/*
+	 * The size of this frame if 0 < size < U32_MAX, 0 terminates the list.
+	 */
+	u32	size;
+
+	/*
+	 * The first frame is a Common Information Entry (CIE) frame, followed
+	 * by one or more Frame Description Entry (FDE) frames. In the former
+	 * case, this field is 0, otherwise it is the negated offset relative
+	 * to the associated CIE frame.
+	 */
+	u32	cie_id_or_pointer;
+
+	union {
+		struct { /* CIE */
+			u8	version;
+			char	augmentation_string[3];
+			u8	code_alignment_factor;
+			u8	data_alignment_factor;
+			u8	return_address_register;
+			u8	augmentation_data_size;
+			u8	fde_pointer_format;
+		};
+
+		struct { /* FDE (sdata4) */
+			s32	initial_loc;
+			s32	range;
+			u8	opcodes[];
+		};
+
+		struct { /* FDE (sdata8) */
+			s64	initial_loc64;
+			s64	range64;
+			u8	opcodes64[];
+		};
+	};
+};
+
+struct eh_frame_fde_iter {
+	const u8		*start;
+	const u8		*p;
+	int			size;
+	int			code_align;
+	bool			use_sdata8;
+	const struct eh_frame	*cie;
+	const struct eh_frame	*fde;
+};
+
+#define EH_FRAME_FDE_ITER_INIT(buf, len) \
+	{ .start = (buf), .p = (buf), .size = (len), .code_align = 1 }
+
+/*
+ * Skip one uleb128/sleb128 encoded quantity from the opcode stream. All bytes
+ * except the last one have bit #7 set.
+ */
+static __always_inline int skip_xleb128(const u8 **opcode, int *size)
+{
+	u8 c;
+
+	do {
+		if (*size <= 0)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		c = *(*opcode)++;
+		(*size)--;
+	} while (c & BIT(7));
+
+	return 0;
+}
+
+static __always_inline int eh_frame_step_opcode(const u8 **opcode, int *size,
+						const struct eh_frame_fde_iter *it,
+						struct eh_frame_cfi_state *st)
+{
+	u8 op;
+
+	if (*size <= 0)
+		return EDYNSCS_INVALID_CFA_OPCODE;
+
+	op = *(*opcode)++;
+	(*size)--;
+	st->ra_negated = false;
+
+	switch (op) {
+	case DW_CFA_nop:
+	case DW_CFA_remember_state:
+	case DW_CFA_restore_state:
+		break;
+
+	case DW_CFA_advance_loc1:
+		if (*size < 1)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->loc += *(*opcode)++ * it->code_align;
+		(*size)--;
+		break;
+
+	case DW_CFA_advance_loc2:
+		if (*size < 2)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->loc += ((*opcode)[0] | ((*opcode)[1] << 8)) *
+			   it->code_align;
+		*opcode += 2;
+		*size -= 2;
+		break;
+
+	case DW_CFA_advance_loc4:
+		if (*size < 4)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->loc += (u64)((*opcode)[0] | ((*opcode)[1] << 8) |
+				 ((*opcode)[2] << 16) |
+				 ((u32)(*opcode)[3] << 24)) *
+			   it->code_align;
+		*opcode += 4;
+		*size -= 4;
+		break;
+
+	case DW_CFA_def_cfa:
+	case DW_CFA_offset_extended:
+		if (skip_xleb128(opcode, size))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		fallthrough;
+	case DW_CFA_def_cfa_offset:
+	case DW_CFA_def_cfa_offset_sf:
+	case DW_CFA_def_cfa_register:
+	case DW_CFA_same_value:
+	case DW_CFA_restore_extended:
+	case 0x80 ... 0xbf: /* DW_CFA_offset */
+		if (skip_xleb128(opcode, size))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		break;
+
+	case DW_CFA_negate_ra_state:
+		st->ra_negated = true;
+		break;
+
+	case 0x40 ... 0x7f: /* DW_CFA_advance_loc (delta in low 6 bits) */
+		st->loc += (op & 0x3f) * it->code_align;
+		break;
+
+	case 0xc0 ... 0xff: /* DW_CFA_restore (reg in low 6 bits) */
+		break;
+
+	default:
+		return EDYNSCS_INVALID_CFA_OPCODE;
+	}
+
+	return 0;
+}
+
+static __always_inline int eh_frame_parse_cie(const struct eh_frame *frame,
+					      struct eh_frame_fde_iter *it)
+{
+	if (sizeof(frame->size) + frame->size < sizeof(*frame))
+		return EDYNSCS_INVALID_CIE_HEADER;
+
+	/*
+	 * Require presence of augmentation data (z) with a specifier for the
+	 * size of the FDE initial_loc and range fields (R), and nothing else.
+	 */
+	if (memcmp(frame->augmentation_string, "zR", 3))
+		return EDYNSCS_INVALID_CIE_HEADER;
+
+	/*
+	 * Require single-byte uleb128/sleb128 alignment factors, LR as the
+	 * return address register, and a 1-byte augmentation payload.
+	 */
+	if ((frame->code_alignment_factor & BIT(7)) ||
+	    (frame->data_alignment_factor & BIT(7)) ||
+	    frame->return_address_register != AARCH64_DWARF_LR ||
+	    frame->augmentation_data_size != 1)
+		return EDYNSCS_INVALID_CIE_HEADER;
+
+	it->code_align = frame->code_alignment_factor;
+
+	switch (frame->fde_pointer_format) {
+	case DW_EH_PE_pcrel | DW_EH_PE_sdata4:
+		it->use_sdata8 = false;
+		break;
+	case DW_EH_PE_pcrel | DW_EH_PE_sdata8:
+		it->use_sdata8 = true;
+		break;
+	default:
+		return EDYNSCS_INVALID_CIE_SDATA_SIZE;
+	}
+
+	it->cie = frame;
+	return 0;
+}
+
+static __always_inline int eh_frame_next_fde(struct eh_frame_fde_iter *it)
+{
+	while (it->size > 4) {
+		const struct eh_frame *frame = (const void *)it->p;
+		const struct eh_frame *cie;
+		const u8 *cie_ptr;
+		size_t min_size;
+		int step, ret;
+
+		if (frame->size == 0 || frame->size == U32_MAX ||
+		    frame->size > (u32)(it->size - sizeof(frame->size)))
+			break;
+
+		if (frame->size < sizeof(frame->cie_id_or_pointer))
+			return EDYNSCS_INVALID_CIE_HEADER;
+
+		step = sizeof(frame->size) + frame->size;
+		it->p += step;
+		it->size -= step;
+
+		if (frame->cie_id_or_pointer == 0) {
+			ret = eh_frame_parse_cie(frame, it);
+			if (ret)
+				return ret;
+			continue;
+		}
+
+		cie_ptr = (const u8 *)&frame->cie_id_or_pointer;
+		if (frame->cie_id_or_pointer > (u32)(cie_ptr - it->start))
+			return EDYNSCS_INVALID_CIE_HEADER;
+
+		cie = (const void *)(cie_ptr - frame->cie_id_or_pointer);
+		if (cie != it->cie) {
+			if (cie->cie_id_or_pointer != 0 ||
+			    cie->size == 0 || cie->size == U32_MAX ||
+			    cie->size > (u32)(cie_ptr - (const u8 *)cie -
+					      sizeof(cie->size)))
+				return EDYNSCS_INVALID_CIE_HEADER;
+			ret = eh_frame_parse_cie(cie, it);
+			if (ret)
+				return ret;
+		}
+
+		min_size = it->use_sdata8 ?
+			   offsetof(struct eh_frame, opcodes64) :
+			   offsetof(struct eh_frame, opcodes);
+		if ((size_t)step < min_size)
+			return EDYNSCS_INVALID_FDE_AUGM_DATA_SIZE;
+
+		it->fde = frame;
+		return 0;
+	}
+
+	it->fde = NULL;
+	return 0;
+}
+
+static __always_inline int eh_frame_fde_opcodes(const struct eh_frame *frame,
+						bool use_sdata8,
+						const u8 **opcode_out,
+						int *size_out)
+{
+	size_t hdr_size = use_sdata8 ?
+			  offsetof(struct eh_frame, opcodes64) :
+			  offsetof(struct eh_frame, opcodes);
+	size_t total = sizeof(frame->size) + frame->size;
+	const u8 *opcode = use_sdata8 ? frame->opcodes64 : frame->opcodes;
+	int size, l;
+
+	if (total < hdr_size + 1 || (*opcode & BIT(7)))
+		return EDYNSCS_INVALID_FDE_AUGM_DATA_SIZE;
+
+	size = (int)(total - hdr_size);
+	l = *opcode++;
+	if (l > size - 1)
+		return EDYNSCS_INVALID_FDE_AUGM_DATA_SIZE;
+
+	opcode += l;
+	size -= l + 1;
+
+	*opcode_out = opcode;
+	*size_out = size;
+	return 0;
+}
+
+#endif /* __ASM_EH_FRAME_H */
diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h
index a15a2968e7b66..f49fc27a8a9b2 100644
--- a/arch/arm64/include/asm/scs.h
+++ b/arch/arm64/include/asm/scs.h
@@ -54,13 +54,6 @@ static inline void dynamic_scs_init(void)
 static inline void dynamic_scs_init(void) {}
 #endif
 
-enum {
-	EDYNSCS_INVALID_CIE_HEADER		= 1,
-	EDYNSCS_INVALID_CIE_SDATA_SIZE		= 2,
-	EDYNSCS_INVALID_FDE_AUGM_DATA_SIZE	= 3,
-	EDYNSCS_INVALID_CFA_OPCODE		= 4,
-};
-
 int __pi_scs_patch(const u8 eh_frame[], int size, bool skip_dry_run);
 
 #endif /* __ASSEMBLER__ */
diff --git a/arch/arm64/kernel/pi/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c
index 3944ad899021c..b240144f7d80e 100644
--- a/arch/arm64/kernel/pi/patch-scs.c
+++ b/arch/arm64/kernel/pi/patch-scs.c
@@ -9,51 +9,13 @@
 #include <linux/linkage.h>
 #include <linux/types.h>
 
+#include <asm/eh_frame.h>
 #include <asm/scs.h>
 
 #include "pi.h"
 
 bool dynamic_scs_is_enabled;
 
-//
-// This minimal DWARF CFI parser is partially based on the code in
-// arch/arc/kernel/unwind.c, and on the document below:
-// https://refspecs.linuxbase.org/LSB_4.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html
-//
-
-#define DW_CFA_nop                          0x00
-#define DW_CFA_set_loc                      0x01
-#define DW_CFA_advance_loc1                 0x02
-#define DW_CFA_advance_loc2                 0x03
-#define DW_CFA_advance_loc4                 0x04
-#define DW_CFA_offset_extended              0x05
-#define DW_CFA_restore_extended             0x06
-#define DW_CFA_undefined                    0x07
-#define DW_CFA_same_value                   0x08
-#define DW_CFA_register                     0x09
-#define DW_CFA_remember_state               0x0a
-#define DW_CFA_restore_state                0x0b
-#define DW_CFA_def_cfa                      0x0c
-#define DW_CFA_def_cfa_register             0x0d
-#define DW_CFA_def_cfa_offset               0x0e
-#define DW_CFA_def_cfa_expression           0x0f
-#define DW_CFA_expression                   0x10
-#define DW_CFA_offset_extended_sf           0x11
-#define DW_CFA_def_cfa_sf                   0x12
-#define DW_CFA_def_cfa_offset_sf            0x13
-#define DW_CFA_val_offset                   0x14
-#define DW_CFA_val_offset_sf                0x15
-#define DW_CFA_val_expression               0x16
-#define DW_CFA_lo_user                      0x1c
-#define DW_CFA_negate_ra_state              0x2d
-#define DW_CFA_GNU_args_size                0x2e
-#define DW_CFA_GNU_negative_offset_extended 0x2f
-#define DW_CFA_hi_user                      0x3f
-
-#define DW_EH_PE_sdata4                     0x0b
-#define DW_EH_PE_sdata8                     0x0c
-#define DW_EH_PE_pcrel                      0x10
-
 enum {
 	PACIASP		= 0xd503233f,
 	AUTIASP		= 0xd50323bf,
@@ -61,7 +23,7 @@ enum {
 	SCS_POP		= 0xf85f8e5e,
 };
 
-static void __always_inline scs_patch_loc(u64 loc)
+static __always_inline void scs_patch_loc(u64 loc)
 {
 	u32 insn = le32_to_cpup((void *)loc);
 
@@ -91,211 +53,49 @@ static void __always_inline scs_patch_loc(u64 loc)
 		    :: "r"(loc));
 }
 
-/*
- * Skip one uleb128/sleb128 encoded quantity from the opcode stream. All bytes
- * except the last one have bit #7 set.
- */
-static int __always_inline skip_xleb128(const u8 **opcode, int size)
-{
-	u8 c;
-
-	do {
-		c = *(*opcode)++;
-		size--;
-	} while (c & BIT(7));
-
-	return size;
-}
-
-struct eh_frame {
-	/*
-	 * The size of this frame if 0 < size < U32_MAX, 0 terminates the list.
-	 */
-	u32	size;
-
-	/*
-	 * The first frame is a Common Information Entry (CIE) frame, followed
-	 * by one or more Frame Description Entry (FDE) frames. In the former
-	 * case, this field is 0, otherwise it is the negated offset relative
-	 * to the associated CIE frame.
-	 */
-	u32	cie_id_or_pointer;
-
-	union {
-		struct { // CIE
-			u8	version;
-			u8	augmentation_string[3];
-			u8	code_alignment_factor;
-			u8	data_alignment_factor;
-			u8	return_address_register;
-			u8	augmentation_data_size;
-			u8	fde_pointer_format;
-		};
-
-		struct { // FDE
-			s32	initial_loc;
-			s32	range;
-			u8	opcodes[];
-		};
-
-		struct { // FDE
-			s64	initial_loc64;
-			s64	range64;
-			u8	opcodes64[];
-		};
-	};
-};
-
-static int scs_handle_fde_frame(const struct eh_frame *frame,
-				int code_alignment_factor,
-				bool use_sdata8,
+static int scs_handle_fde_frame(const struct eh_frame_fde_iter *it,
 				bool dry_run)
 {
-	int size = frame->size - offsetof(struct eh_frame, opcodes) + 4;
-	u64 loc = (u64)offset_to_ptr(&frame->initial_loc);
-	const u8 *opcode = frame->opcodes;
-	int l;
-
-	if (use_sdata8) {
-		loc = (u64)&frame->initial_loc64 + frame->initial_loc64;
-		opcode = frame->opcodes64;
-		size -= 8;
-	}
-
-	// assume single byte uleb128_t for augmentation data size
-	if (*opcode & BIT(7))
-		return EDYNSCS_INVALID_FDE_AUGM_DATA_SIZE;
+	const struct eh_frame *frame = it->fde;
+	struct eh_frame_cfi_state st = {
+		.loc = it->use_sdata8 ?
+		       (u64)&frame->initial_loc64 + frame->initial_loc64 :
+		       (u64)offset_to_ptr(&frame->initial_loc),
+	};
+	const u8 *opcode;
+	int size, ret;
 
-	l = *opcode++;
-	opcode += l;
-	size -= l + 1;
+	ret = eh_frame_fde_opcodes(frame, it->use_sdata8, &opcode, &size);
+	if (ret)
+		return ret;
 
 	/*
 	 * Starting from 'loc', apply the CFA opcodes that advance the location
 	 * pointer, and identify the locations of the PAC instructions.
 	 */
-	while (size-- > 0) {
-		switch (*opcode++) {
-		case DW_CFA_nop:
-		case DW_CFA_remember_state:
-		case DW_CFA_restore_state:
-			break;
-
-		case DW_CFA_advance_loc1:
-			loc += *opcode++ * code_alignment_factor;
-			size--;
-			break;
-
-		case DW_CFA_advance_loc2:
-			loc += *opcode++ * code_alignment_factor;
-			loc += (*opcode++ << 8) * code_alignment_factor;
-			size -= 2;
-			break;
-
-		case DW_CFA_advance_loc4:
-			loc += *opcode++ * code_alignment_factor;
-			loc += (*opcode++ << 8) * code_alignment_factor;
-			loc += (*opcode++ << 16) * code_alignment_factor;
-			loc += ((u64)*opcode++ << 24) * code_alignment_factor;
-			size -= 4;
-			break;
-
-		case DW_CFA_def_cfa:
-		case DW_CFA_offset_extended:
-			size = skip_xleb128(&opcode, size);
-			fallthrough;
-		case DW_CFA_def_cfa_offset:
-		case DW_CFA_def_cfa_offset_sf:
-		case DW_CFA_def_cfa_register:
-		case DW_CFA_same_value:
-		case DW_CFA_restore_extended:
-		case 0x80 ... 0xbf:
-			size = skip_xleb128(&opcode, size);
-			break;
-
-		case DW_CFA_negate_ra_state:
-			if (!dry_run)
-				scs_patch_loc(loc - 4);
-			break;
+	while (size > 0) {
+		ret = eh_frame_step_opcode(&opcode, &size, it, &st);
+		if (ret)
+			return ret;
 
-		case 0x40 ... 0x7f:
-			// advance loc
-			loc += (opcode[-1] & 0x3f) * code_alignment_factor;
-			break;
-
-		case 0xc0 ... 0xff:
-			break;
-
-		default:
-			return EDYNSCS_INVALID_CFA_OPCODE;
-		}
+		if (st.ra_negated && !dry_run)
+			scs_patch_loc(st.loc - 4);
 	}
 	return 0;
 }
 
 int scs_patch(const u8 eh_frame[], int size, bool skip_dry_run)
 {
-	int code_alignment_factor = 1;
-	bool fde_use_sdata8 = false;
-	const u8 *p = eh_frame;
-
-	while (size > 4) {
-		const struct eh_frame *frame = (const void *)p;
-		int ret;
-
-		if (frame->size == 0 ||
-		    frame->size == U32_MAX ||
-		    frame->size > size)
-			break;
-
-		if (frame->cie_id_or_pointer == 0) {
-			/*
-			 * Require presence of augmentation data (z) with a
-			 * specifier for the size of the FDE initial_loc and
-			 * range fields (R), and nothing else.
-			 */
-			if (strcmp(frame->augmentation_string, "zR"))
-				return EDYNSCS_INVALID_CIE_HEADER;
+	struct eh_frame_fde_iter it = EH_FRAME_FDE_ITER_INIT(eh_frame, size);
+	int ret;
 
-			/*
-			 * The code alignment factor is a uleb128 encoded field
-			 * but given that the only sensible values are 1 or 4,
-			 * there is no point in decoding the whole thing.  Also
-			 * sanity check the size of the data alignment factor
-			 * field, and the values of the return address register
-			 * and augmentation data size fields.
-			 */
-			if ((frame->code_alignment_factor & BIT(7)) ||
-			    (frame->data_alignment_factor & BIT(7)) ||
-			    frame->return_address_register != 30 ||
-			    frame->augmentation_data_size != 1)
-				return EDYNSCS_INVALID_CIE_HEADER;
+	while (!(ret = eh_frame_next_fde(&it)) && it.fde) {
+		ret = scs_handle_fde_frame(&it, !skip_dry_run);
+		if (ret)
+			return ret;
 
-			code_alignment_factor = frame->code_alignment_factor;
-
-			switch (frame->fde_pointer_format) {
-			case DW_EH_PE_pcrel | DW_EH_PE_sdata4:
-				fde_use_sdata8 = false;
-				break;
-			case DW_EH_PE_pcrel | DW_EH_PE_sdata8:
-				fde_use_sdata8 = true;
-				break;
-			default:
-				return EDYNSCS_INVALID_CIE_SDATA_SIZE;
-			}
-		} else {
-			ret = scs_handle_fde_frame(frame, code_alignment_factor,
-						   fde_use_sdata8, !skip_dry_run);
-			if (ret)
-				return ret;
-
-			if (!skip_dry_run)
-				scs_handle_fde_frame(frame, code_alignment_factor,
-						     fde_use_sdata8, false);
-		}
-
-		p += sizeof(frame->size) + frame->size;
-		size -= sizeof(frame->size) + frame->size;
+		if (!skip_dry_run)
+			scs_handle_fde_frame(&it, false);
 	}
-	return 0;
+	return ret;
 }
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 03/12] arm64: eh_frame: track CFA and x29/x30 register state in CFI parser
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 01/12] arm64, unwind: build with .eh_frame and add gen_lr_unwind skeleton Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 02/12] arm64: factor .eh_frame parser out of patch-scs.c into <asm/eh_frame.h> Dylan Hatch
@ 2026-11-17  4:34 ` Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 04/12] arm64, unwind: parse .eh_frame into .lr_unwind entries in gen_lr_unwind Dylan Hatch
                   ` (8 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Extend <asm/eh_frame.h> to decode ULEB128/SLEB128 operands, evaluate
initial CIE opcodes, and track CFA (cfa_reg, cfa_off) and x29/x30
frame-record save rules (fp_rule/fp_off, lr_rule/lr_off) across FDE
instructions, including DW_CFA_remember_state and DW_CFA_restore_state.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/include/asm/eh_frame.h | 273 +++++++++++++++++++++++++++---
 arch/arm64/kernel/pi/patch-scs.c  |   1 +
 2 files changed, 249 insertions(+), 25 deletions(-)

diff --git a/arch/arm64/include/asm/eh_frame.h b/arch/arm64/include/asm/eh_frame.h
index 6c17fed1860f1..1dcb5b6151bbc 100644
--- a/arch/arm64/include/asm/eh_frame.h
+++ b/arch/arm64/include/asm/eh_frame.h
@@ -2,6 +2,10 @@
 /*
  * Minimal DWARF .eh_frame CFI parser for arm64.
  *
+ * Shared between early boot/module dynamic SCS patching
+ * (arch/arm64/kernel/pi/patch-scs.c) and build-time .lr_unwind generation
+ * (scripts/gen_lr_unwind.c).
+ *
  * This minimal DWARF CFI parser is partially based on the code in
  * arch/arc/kernel/unwind.c, and on the document below:
  * https://refspecs.linuxbase.org/LSB_4.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html
@@ -19,7 +23,11 @@
 #define U32_MAX				((u32)~0U)
 #endif
 
+#define AARCH64_DWARF_FP		29
 #define AARCH64_DWARF_LR		30
+#define AARCH64_DWARF_SP		31
+
+#define EH_FRAME_STATE_STACK_MAX	8
 
 #define DW_CFA_nop			0x00
 #define DW_CFA_advance_loc1		0x02
@@ -27,14 +35,22 @@
 #define DW_CFA_advance_loc4		0x04
 #define DW_CFA_offset_extended		0x05
 #define DW_CFA_restore_extended		0x06
+#define DW_CFA_undefined		0x07
 #define DW_CFA_same_value		0x08
+#define DW_CFA_register			0x09
 #define DW_CFA_remember_state		0x0a
 #define DW_CFA_restore_state		0x0b
 #define DW_CFA_def_cfa			0x0c
 #define DW_CFA_def_cfa_register		0x0d
 #define DW_CFA_def_cfa_offset		0x0e
+#define DW_CFA_def_cfa_expression	0x0f
+#define DW_CFA_expression		0x10
+#define DW_CFA_offset_extended_sf	0x11
+#define DW_CFA_def_cfa_sf		0x12
 #define DW_CFA_def_cfa_offset_sf	0x13
+#define DW_CFA_val_expression		0x16
 #define DW_CFA_negate_ra_state		0x2d
+#define DW_CFA_GNU_args_size		0x2e
 
 #define DW_EH_PE_sdata4			0x0b
 #define DW_EH_PE_sdata8			0x0c
@@ -47,9 +63,27 @@ enum {
 	EDYNSCS_INVALID_CFA_OPCODE		= 4,
 };
 
+enum eh_frame_reg_rule {
+	EH_REG_UNMODIFIED	= 0,
+	EH_REG_OFFSET		= 1,
+	EH_REG_UNSUPPORTED	= 2,
+};
+
+struct eh_frame_reg_state {
+	s64	cfa_off;
+	s64	fp_off;
+	s64	lr_off;
+	u32	cfa_reg;
+	u8	fp_rule;
+	u8	lr_rule;
+};
+
 struct eh_frame_cfi_state {
-	u64	loc;
-	bool	ra_negated;
+	u64				loc;
+	struct eh_frame_reg_state	reg;
+	struct eh_frame_reg_state	stack[EH_FRAME_STATE_STACK_MAX];
+	unsigned int			stack_depth;
+	bool				ra_negated;
 };
 
 struct eh_frame {
@@ -75,6 +109,7 @@ struct eh_frame {
 			u8	return_address_register;
 			u8	augmentation_data_size;
 			u8	fde_pointer_format;
+			u8	cie_opcodes[];
 		};
 
 		struct { /* FDE (sdata4) */
@@ -92,40 +127,100 @@ struct eh_frame {
 };
 
 struct eh_frame_fde_iter {
-	const u8		*start;
-	const u8		*p;
-	int			size;
-	int			code_align;
-	bool			use_sdata8;
-	const struct eh_frame	*cie;
-	const struct eh_frame	*fde;
+	const u8			*start;
+	const u8			*p;
+	int				size;
+	int				code_align;
+	int				data_align;
+	bool				use_sdata8;
+	struct eh_frame_reg_state	cie_reg;
+	const struct eh_frame		*cie;
+	const struct eh_frame		*fde;
 };
 
 #define EH_FRAME_FDE_ITER_INIT(buf, len) \
-	{ .start = (buf), .p = (buf), .size = (len), .code_align = 1 }
+	{ .start = (buf), .p = (buf), .size = (len), \
+	  .code_align = 1, .data_align = -4, \
+	  .cie_reg = { .cfa_reg = AARCH64_DWARF_SP } }
 
-/*
- * Skip one uleb128/sleb128 encoded quantity from the opcode stream. All bytes
- * except the last one have bit #7 set.
- */
-static __always_inline int skip_xleb128(const u8 **opcode, int *size)
+static __always_inline int eh_frame_read_uleb128(const u8 **opcode, int *size,
+						 u64 *val)
 {
+	unsigned int shift = 0;
+	u64 res = 0;
 	u8 c;
 
 	do {
-		if (*size <= 0)
+		if (*size <= 0 || shift >= 64)
 			return EDYNSCS_INVALID_CFA_OPCODE;
 		c = *(*opcode)++;
 		(*size)--;
+		if (shift == 63 && c > 1)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		res |= (u64)(c & 0x7f) << shift;
+		shift += 7;
 	} while (c & BIT(7));
 
+	*val = res;
+	return 0;
+}
+
+static __always_inline int eh_frame_read_sleb128(const u8 **opcode, int *size,
+						 s64 *val)
+{
+	unsigned int shift = 0;
+	u64 res = 0;
+	u8 c;
+
+	do {
+		if (*size <= 0 || shift >= 64)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		c = *(*opcode)++;
+		(*size)--;
+		if (shift == 63 && c != 0x00 && c != 0x7f)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		res |= (u64)(c & 0x7f) << shift;
+		shift += 7;
+	} while (c & BIT(7));
+
+	if (shift < 64 && (c & BIT(6)))
+		res |= ~0ULL << shift;
+
+	*val = (s64)res;
 	return 0;
 }
 
+static __always_inline void eh_frame_set_reg_rule(struct eh_frame_reg_state *r,
+						  u64 reg, u8 rule, s64 off)
+{
+	if (reg == AARCH64_DWARF_FP) {
+		r->fp_rule = rule;
+		r->fp_off = off;
+	} else if (reg == AARCH64_DWARF_LR) {
+		r->lr_rule = rule;
+		r->lr_off = off;
+	}
+}
+
+static __always_inline void eh_frame_restore_reg(struct eh_frame_reg_state *r,
+						 const struct eh_frame_reg_state *cie,
+						 u64 reg)
+{
+	if (reg == AARCH64_DWARF_FP) {
+		r->fp_rule = cie->fp_rule;
+		r->fp_off = cie->fp_off;
+	} else if (reg == AARCH64_DWARF_LR) {
+		r->lr_rule = cie->lr_rule;
+		r->lr_off = cie->lr_off;
+	}
+}
+
 static __always_inline int eh_frame_step_opcode(const u8 **opcode, int *size,
 						const struct eh_frame_fde_iter *it,
 						struct eh_frame_cfi_state *st)
 {
+	u64 ureg, uoff;
+	s64 soff;
 	u8 op;
 
 	if (*size <= 0)
@@ -137,8 +232,18 @@ static __always_inline int eh_frame_step_opcode(const u8 **opcode, int *size,
 
 	switch (op) {
 	case DW_CFA_nop:
+		break;
+
 	case DW_CFA_remember_state:
+		if (st->stack_depth >= EH_FRAME_STATE_STACK_MAX)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->stack[st->stack_depth++] = st->reg;
+		break;
+
 	case DW_CFA_restore_state:
+		if (!st->stack_depth)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->reg = st->stack[--st->stack_depth];
 		break;
 
 	case DW_CFA_advance_loc1:
@@ -168,18 +273,106 @@ static __always_inline int eh_frame_step_opcode(const u8 **opcode, int *size,
 		*size -= 4;
 		break;
 
-	case DW_CFA_def_cfa:
 	case DW_CFA_offset_extended:
-		if (skip_xleb128(opcode, size))
+		if (eh_frame_read_uleb128(opcode, size, &ureg) ||
+		    eh_frame_read_uleb128(opcode, size, &uoff))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		eh_frame_set_reg_rule(&st->reg, ureg, EH_REG_OFFSET,
+				      (s64)uoff * it->data_align);
+		break;
+
+	case DW_CFA_offset_extended_sf:
+		if (eh_frame_read_uleb128(opcode, size, &ureg) ||
+		    eh_frame_read_sleb128(opcode, size, &soff))
 			return EDYNSCS_INVALID_CFA_OPCODE;
-		fallthrough;
+		eh_frame_set_reg_rule(&st->reg, ureg, EH_REG_OFFSET,
+				      soff * it->data_align);
+		break;
+
+	case DW_CFA_restore_extended:
+		if (eh_frame_read_uleb128(opcode, size, &ureg))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		eh_frame_restore_reg(&st->reg, &it->cie_reg, ureg);
+		break;
+
+	case DW_CFA_same_value:
+		if (eh_frame_read_uleb128(opcode, size, &ureg))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		eh_frame_set_reg_rule(&st->reg, ureg, EH_REG_UNMODIFIED, 0);
+		break;
+
+	case DW_CFA_undefined:
+		if (eh_frame_read_uleb128(opcode, size, &ureg))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		eh_frame_set_reg_rule(&st->reg, ureg, EH_REG_UNSUPPORTED, 0);
+		break;
+
+	case DW_CFA_register:
+		if (eh_frame_read_uleb128(opcode, size, &ureg) ||
+		    eh_frame_read_uleb128(opcode, size, &uoff))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		eh_frame_set_reg_rule(&st->reg, ureg, EH_REG_UNSUPPORTED, 0);
+		break;
+
+	case DW_CFA_def_cfa:
+		if (eh_frame_read_uleb128(opcode, size, &ureg) ||
+		    eh_frame_read_uleb128(opcode, size, &uoff) ||
+		    ureg >= U32_MAX)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->reg.cfa_reg = ureg;
+		st->reg.cfa_off = (s64)uoff;
+		break;
+
+	case DW_CFA_def_cfa_sf:
+		if (eh_frame_read_uleb128(opcode, size, &ureg) ||
+		    eh_frame_read_sleb128(opcode, size, &soff) ||
+		    ureg >= U32_MAX)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->reg.cfa_reg = ureg;
+		st->reg.cfa_off = soff * it->data_align;
+		break;
+
+	case DW_CFA_def_cfa_register:
+		if (eh_frame_read_uleb128(opcode, size, &ureg) ||
+		    ureg >= U32_MAX)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->reg.cfa_reg = ureg;
+		break;
+
 	case DW_CFA_def_cfa_offset:
+		if (eh_frame_read_uleb128(opcode, size, &uoff))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->reg.cfa_off = (s64)uoff;
+		break;
+
 	case DW_CFA_def_cfa_offset_sf:
-	case DW_CFA_def_cfa_register:
-	case DW_CFA_same_value:
-	case DW_CFA_restore_extended:
-	case 0x80 ... 0xbf: /* DW_CFA_offset */
-		if (skip_xleb128(opcode, size))
+		if (eh_frame_read_sleb128(opcode, size, &soff))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		st->reg.cfa_off = soff * it->data_align;
+		break;
+
+	case DW_CFA_def_cfa_expression:
+		if (eh_frame_read_uleb128(opcode, size, &uoff) ||
+		    uoff > (u64)*size)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		*opcode += uoff;
+		*size -= uoff;
+		st->reg.cfa_reg = U32_MAX;
+		break;
+
+	case DW_CFA_expression:
+	case DW_CFA_val_expression:
+		if (eh_frame_read_uleb128(opcode, size, &ureg) ||
+		    eh_frame_read_uleb128(opcode, size, &uoff) ||
+		    uoff > (u64)*size)
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		*opcode += uoff;
+		*size -= uoff;
+		eh_frame_set_reg_rule(&st->reg, ureg, EH_REG_UNSUPPORTED, 0);
+		break;
+
+	case DW_CFA_GNU_args_size:
+		if (eh_frame_read_uleb128(opcode, size, &uoff))
 			return EDYNSCS_INVALID_CFA_OPCODE;
 		break;
 
@@ -191,7 +384,15 @@ static __always_inline int eh_frame_step_opcode(const u8 **opcode, int *size,
 		st->loc += (op & 0x3f) * it->code_align;
 		break;
 
+	case 0x80 ... 0xbf: /* DW_CFA_offset (reg in low 6 bits) */
+		if (eh_frame_read_uleb128(opcode, size, &uoff))
+			return EDYNSCS_INVALID_CFA_OPCODE;
+		eh_frame_set_reg_rule(&st->reg, op & 0x3f, EH_REG_OFFSET,
+				      (s64)uoff * it->data_align);
+		break;
+
 	case 0xc0 ... 0xff: /* DW_CFA_restore (reg in low 6 bits) */
+		eh_frame_restore_reg(&st->reg, &it->cie_reg, op & 0x3f);
 		break;
 
 	default:
@@ -204,7 +405,14 @@ static __always_inline int eh_frame_step_opcode(const u8 **opcode, int *size,
 static __always_inline int eh_frame_parse_cie(const struct eh_frame *frame,
 					      struct eh_frame_fde_iter *it)
 {
-	if (sizeof(frame->size) + frame->size < sizeof(*frame))
+	struct eh_frame_cfi_state cie_st = {
+		.reg = { .cfa_reg = AARCH64_DWARF_SP },
+	};
+	const u8 *opcode;
+	int size, ret;
+
+	if (sizeof(frame->size) + frame->size <
+	    offsetof(struct eh_frame, cie_opcodes))
 		return EDYNSCS_INVALID_CIE_HEADER;
 
 	/*
@@ -225,6 +433,9 @@ static __always_inline int eh_frame_parse_cie(const struct eh_frame *frame,
 		return EDYNSCS_INVALID_CIE_HEADER;
 
 	it->code_align = frame->code_alignment_factor;
+	it->data_align = (frame->data_alignment_factor & BIT(6)) ?
+			 (int)frame->data_alignment_factor - 128 :
+			 frame->data_alignment_factor;
 
 	switch (frame->fde_pointer_format) {
 	case DW_EH_PE_pcrel | DW_EH_PE_sdata4:
@@ -237,6 +448,18 @@ static __always_inline int eh_frame_parse_cie(const struct eh_frame *frame,
 		return EDYNSCS_INVALID_CIE_SDATA_SIZE;
 	}
 
+	it->cie_reg = cie_st.reg;
+	opcode = frame->cie_opcodes;
+	size = (int)(sizeof(frame->size) + frame->size -
+		     offsetof(struct eh_frame, cie_opcodes));
+
+	while (size > 0) {
+		ret = eh_frame_step_opcode(&opcode, &size, it, &cie_st);
+		if (ret)
+			return ret;
+	}
+
+	it->cie_reg = cie_st.reg;
 	it->cie = frame;
 	return 0;
 }
diff --git a/arch/arm64/kernel/pi/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c
index b240144f7d80e..f4264dda828f7 100644
--- a/arch/arm64/kernel/pi/patch-scs.c
+++ b/arch/arm64/kernel/pi/patch-scs.c
@@ -61,6 +61,7 @@ static int scs_handle_fde_frame(const struct eh_frame_fde_iter *it,
 		.loc = it->use_sdata8 ?
 		       (u64)&frame->initial_loc64 + frame->initial_loc64 :
 		       (u64)offset_to_ptr(&frame->initial_loc),
+		.reg = it->cie_reg,
 	};
 	const u8 *opcode;
 	int size, ret;
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 04/12] arm64, unwind: parse .eh_frame into .lr_unwind entries in gen_lr_unwind
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (2 preceding siblings ...)
  2026-11-17  4:34 ` [RFC PATCH 03/12] arm64: eh_frame: track CFA and x29/x30 register state in CFI parser Dylan Hatch
@ 2026-11-17  4:34 ` Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 05/12] scripts/gen_lr_unwind: emit .lr_unwind_fde and .lr_unwind_ranges Dylan Hatch
                   ` (7 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Define the .lr_unwind_fde and .lr_unwind_ranges binary format in
<asm/lr_unwind.h> and implement parse_eh_frame() in
scripts/gen_lr_unwind to classify each FDE instruction range into one of
three exception-boundary unwind modes derived from DWARF CFI:
  - LR_UNWIND_LIVE: x30 holds the live return address and x29 holds the
    caller's frame pointer.
  - LR_UNWIND_FP:   the {x29, x30} frame record resides at [x29].
  - LR_UNWIND_SP:   the {x29, x30} frame record resides at
    [sp + sp_off], handling fixed-frame GCC functions and Clang
    epilogues where CFA uses SP before or after x29 is updated.

Functions with non-standard x30 spills (e.g. frameless leaf functions
spilling x30 without x29) are omitted so the unwinder treats them as
unreliable.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/include/asm/lr_unwind.h | 111 ++++++++++
 scripts/gen_lr_unwind.c            | 318 ++++++++++++++++++++++++++++-
 2 files changed, 425 insertions(+), 4 deletions(-)
 create mode 100644 arch/arm64/include/asm/lr_unwind.h

diff --git a/arch/arm64/include/asm/lr_unwind.h b/arch/arm64/include/asm/lr_unwind.h
new file mode 100644
index 0000000000000..81c743994c4d5
--- /dev/null
+++ b/arch/arm64/include/asm/lr_unwind.h
@@ -0,0 +1,111 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Binary format definitions for .lr_unwind_fde and .lr_unwind_ranges.
+ *
+ * Each function (FDE) is described by an 8-byte struct lr_fde containing a
+ * 32-bit PC-relative start_offset and a 32-bit range_desc encoding instruction
+ * ranges and how to unwind across an exception boundary at each range:
+ *   - LR_UNWIND_LIVE: return address is live in x30; caller FP is in x29
+ *   - LR_UNWIND_FP:   frame record {caller_fp, caller_lr} is at [x29]
+ *   - LR_UNWIND_SP:   frame record {caller_fp, caller_lr} is at [sp + sp_off]
+ */
+#ifndef __ASM_LR_UNWIND_H
+#define __ASM_LR_UNWIND_H
+
+#include <linux/bits.h>
+#include <linux/compiler.h>
+#include <linux/types.h>
+
+enum lr_unwind_mode {
+	LR_UNWIND_LIVE	= 0,
+	LR_UNWIND_FP	= 1,
+	LR_UNWIND_SP	= 2,
+};
+
+struct lr_fde {
+	s32 start_offset;	/* PC-relative offset from &fde->start_offset */
+	u32 range_desc;		/* Inline (bit 31 = 1) or OOL (bit 31 = 0) */
+};
+
+/*
+ * Out-of-line range entry:
+ *   [15:14] mode (2 bits): enum lr_unwind_mode
+ *   [13:0]  len (14 bits): length in 4-byte instructions (1..16,383)
+ */
+typedef u16 lr_range_t;
+
+#define LR_RANGE_LEN_BITS	14
+#define LR_RANGE_LEN_MAX	((1U << LR_RANGE_LEN_BITS) - 1)
+
+#define LR_RANGE_MAKE(mode, len) \
+	((lr_range_t)((((u32)(mode)) << LR_RANGE_LEN_BITS) | \
+		      ((u32)(len) & LR_RANGE_LEN_MAX)))
+#define LR_RANGE_MODE(r)	((enum lr_unwind_mode)((r) >> LR_RANGE_LEN_BITS))
+#define LR_RANGE_LEN(r)		((u32)((r) & LR_RANGE_LEN_MAX))
+
+/*
+ * Common range_desc header bits (both inline and out-of-line):
+ *   [31]    inline flag (1 = inline, 0 = out-of-line)
+ *   [30:26] sp_off (5 bits): frame record offset from SP in 16-byte units
+ */
+#define LR_FDE_INLINE		BIT(31)
+#define LR_FDE_SP_OFF_SHIFT	26
+#define LR_FDE_SP_OFF_MAX	0x1fU
+#define LR_FDE_SP_OFF_BYTES(d)	((((d) >> LR_FDE_SP_OFF_SHIFT) & LR_FDE_SP_OFF_MAX) << 4)
+
+/*
+ * Inline range_desc layout (bit 31 == 1):
+ *   [25]    body_sp (1 bit): r1 mode (0 = LR_UNWIND_FP, 1 = LR_UNWIND_SP)
+ *   [24:20] r0 (5 bits):     prologue LR_UNWIND_LIVE insns
+ *   [19:8]  r1 (12 bits):    body LR_UNWIND_FP / LR_UNWIND_SP insns
+ *   [7:4]   r2 (4 bits):     epilogue LR_UNWIND_SP insns
+ *   [3:0]   r3 (4 bits):     return LR_UNWIND_LIVE insns
+ */
+#define LR_INLINE_BODY_SP	BIT(25)
+#define LR_INLINE_R0_MAX	0x1fU
+#define LR_INLINE_R1_MAX	0xfffU
+#define LR_INLINE_R2_MAX	0xfU
+#define LR_INLINE_R3_MAX	0xfU
+
+#define LR_INLINE_MAKE(sp_off, body_sp, r0, r1, r2, r3) \
+	(LR_FDE_INLINE | \
+	 (((u32)(sp_off) & LR_FDE_SP_OFF_MAX) << LR_FDE_SP_OFF_SHIFT) | \
+	 ((body_sp) ? LR_INLINE_BODY_SP : 0U) | \
+	 (((u32)(r0) & LR_INLINE_R0_MAX) << 20) | \
+	 (((u32)(r1) & LR_INLINE_R1_MAX) << 8) | \
+	 (((u32)(r2) & LR_INLINE_R2_MAX) << 4) | \
+	 ((u32)(r3) & LR_INLINE_R3_MAX))
+
+#define LR_INLINE_R0(d)		(((d) >> 20) & LR_INLINE_R0_MAX)
+#define LR_INLINE_R1(d)		(((d) >> 8) & LR_INLINE_R1_MAX)
+#define LR_INLINE_R2(d)		(((d) >> 4) & LR_INLINE_R2_MAX)
+#define LR_INLINE_R3(d)		((d) & LR_INLINE_R3_MAX)
+
+static __always_inline void lr_fde_unpack_inline(u32 desc, lr_range_t ranges[4])
+{
+	enum lr_unwind_mode body_mode = (desc & LR_INLINE_BODY_SP) ?
+					LR_UNWIND_SP : LR_UNWIND_FP;
+
+	ranges[0] = LR_RANGE_MAKE(LR_UNWIND_LIVE, LR_INLINE_R0(desc));
+	ranges[1] = LR_RANGE_MAKE(body_mode, LR_INLINE_R1(desc));
+	ranges[2] = LR_RANGE_MAKE(LR_UNWIND_SP, LR_INLINE_R2(desc));
+	ranges[3] = LR_RANGE_MAKE(LR_UNWIND_LIVE, LR_INLINE_R3(desc));
+}
+
+/*
+ * Out-of-line range_desc layout (bit 31 == 0):
+ *   [25:6] idx (20 bits): starting index into .lr_unwind_ranges
+ *   [5:0]  cnt (6 bits):  number of lr_range_t entries
+ */
+#define LR_OOL_IDX_MAX		0xfffffU
+#define LR_OOL_COUNT_MAX	0x3fU
+
+#define LR_OOL_MAKE(sp_off, idx, cnt) \
+	((((u32)(sp_off) & LR_FDE_SP_OFF_MAX) << LR_FDE_SP_OFF_SHIFT) | \
+	 (((u32)(idx) & LR_OOL_IDX_MAX) << 6) | \
+	 ((u32)(cnt) & LR_OOL_COUNT_MAX))
+
+#define LR_OOL_IDX(d)		(((d) >> 6) & LR_OOL_IDX_MAX)
+#define LR_OOL_COUNT(d)		((d) & LR_OOL_COUNT_MAX)
+
+#endif /* __ASM_LR_UNWIND_H */
diff --git a/scripts/gen_lr_unwind.c b/scripts/gen_lr_unwind.c
index 1b52662838653..118f995b2c48b 100644
--- a/scripts/gen_lr_unwind.c
+++ b/scripts/gen_lr_unwind.c
@@ -16,6 +16,18 @@
 #include <sys/stat.h>
 #include <unistd.h>
 
+#include "../arch/arm64/include/asm/eh_frame.h"
+#include "../arch/arm64/include/asm/lr_unwind.h"
+
+#ifndef R_AARCH64_PREL32
+#define R_AARCH64_PREL32	261
+#endif
+
+struct lr_run {
+	enum lr_unwind_mode	mode;
+	u32			len;
+};
+
 struct lr_unwind_ctx {
 	const void	*map;
 	size_t		map_size;
@@ -24,15 +36,308 @@ struct lr_unwind_ctx {
 	const Elf64_Shdr *rela_eh_sec;
 	const Elf64_Shdr *fde_sec;
 	unsigned int	symtab_shndx;
+
+	const Elf64_Rela *eh_relas;
+	size_t		nr_eh_relas;
+	size_t		rela_cursor;
+
+	struct lr_fde	*fdes;
+	Elf64_Rela	*fde_relas;
+	size_t		nr_fdes;
+	size_t		cap_fdes;
+
+	lr_range_t	*ranges;
+	size_t		nr_ranges;
+	size_t		cap_ranges;
+
+	struct lr_run	*runs;
+	size_t		cap_runs;
 };
 
-static int parse_eh_frame(struct lr_unwind_ctx *ctx)
+static int append_fde_entry(struct lr_unwind_ctx *ctx,
+			    const Elf64_Rela *eh_rela,
+			    s64 byte_off, u32 desc)
 {
-	/* TODO: Parse CIEs/FDEs from .eh_frame and .rela.eh_frame */
-	(void)ctx;
+	size_t idx;
+
+	if (ctx->nr_fdes == ctx->cap_fdes) {
+		size_t cap = ctx->cap_fdes ? ctx->cap_fdes * 2 : 1024;
+		struct lr_fde *fdes = realloc(ctx->fdes, cap * sizeof(*fdes));
+		Elf64_Rela *relas;
+
+		if (!fdes)
+			return -1;
+		ctx->fdes = fdes;
+
+		relas = realloc(ctx->fde_relas, cap * sizeof(*relas));
+		if (!relas)
+			return -1;
+		ctx->fde_relas = relas;
+		ctx->cap_fdes = cap;
+	}
+
+	idx = ctx->nr_fdes++;
+	ctx->fdes[idx].start_offset = 0;
+	ctx->fdes[idx].range_desc = desc;
+
+	ctx->fde_relas[idx].r_offset = idx * sizeof(struct lr_fde);
+	ctx->fde_relas[idx].r_info = ELF64_R_INFO(ELF64_R_SYM(eh_rela->r_info),
+						  R_AARCH64_PREL32);
+	ctx->fde_relas[idx].r_addend = eh_rela->r_addend + byte_off;
+	return 0;
+}
+
+static int append_ool_range(struct lr_unwind_ctx *ctx, lr_range_t entry)
+{
+	if (ctx->nr_ranges == ctx->cap_ranges) {
+		size_t cap = ctx->cap_ranges ? ctx->cap_ranges * 2 : 65536;
+		lr_range_t *tmp = realloc(ctx->ranges, cap * sizeof(*tmp));
+
+		if (!tmp)
+			return -1;
+		ctx->ranges = tmp;
+		ctx->cap_ranges = cap;
+	}
+
+	ctx->ranges[ctx->nr_ranges++] = entry;
 	return 0;
 }
 
+static int push_run(struct lr_unwind_ctx *ctx, size_t *nr_runs,
+		    enum lr_unwind_mode mode, u32 len)
+{
+	if (*nr_runs == ctx->cap_runs) {
+		size_t cap = ctx->cap_runs ? ctx->cap_runs * 2 : 64;
+		struct lr_run *tmp = realloc(ctx->runs, cap * sizeof(*tmp));
+
+		if (!tmp)
+			return -1;
+		ctx->runs = tmp;
+		ctx->cap_runs = cap;
+	}
+
+	ctx->runs[(*nr_runs)++] = (struct lr_run){ .mode = mode, .len = len };
+	return 0;
+}
+
+static bool encode_inline(const struct lr_run *runs, size_t nr_runs,
+			  u32 sp_off, u32 *desc_out)
+{
+	u32 r0 = 0, r1 = 0, r2 = 0, r3 = 0;
+	bool body_sp = false;
+	size_t k = 0;
+
+	if (k < nr_runs && runs[k].mode == LR_UNWIND_LIVE) {
+		if (runs[k].len > LR_INLINE_R0_MAX)
+			return false;
+		r0 = runs[k++].len;
+	}
+	if (k < nr_runs && (runs[k].mode == LR_UNWIND_FP ||
+			    runs[k].mode == LR_UNWIND_SP)) {
+		if (runs[k].len > LR_INLINE_R1_MAX)
+			return false;
+		body_sp = (runs[k].mode == LR_UNWIND_SP);
+		r1 = runs[k++].len;
+	}
+	if (k < nr_runs && runs[k].mode == LR_UNWIND_SP) {
+		if (runs[k].len > LR_INLINE_R2_MAX)
+			return false;
+		r2 = runs[k++].len;
+	}
+	if (k < nr_runs && runs[k].mode == LR_UNWIND_LIVE) {
+		if (runs[k].len > LR_INLINE_R3_MAX)
+			return false;
+		r3 = runs[k++].len;
+	}
+	if (k != nr_runs)
+		return false;
+
+	*desc_out = LR_INLINE_MAKE(sp_off, body_sp, r0, r1, r2, r3);
+	return true;
+}
+
+static int emit_fde(struct lr_unwind_ctx *ctx, const Elf64_Rela *eh_rela,
+		    size_t nr_runs, u32 sp_off)
+{
+	size_t start_idx = ctx->nr_ranges;
+	s64 byte_off = 0;
+	size_t total_ool;
+	u32 desc;
+
+	if (encode_inline(ctx->runs, nr_runs, sp_off, &desc))
+		return append_fde_entry(ctx, eh_rela, 0, desc);
+
+	for (size_t k = 0; k < nr_runs; k++) {
+		u32 rem = ctx->runs[k].len;
+
+		while (rem > 0) {
+			u32 chunk = rem > LR_RANGE_LEN_MAX ?
+				    LR_RANGE_LEN_MAX : rem;
+
+			if (append_ool_range(ctx, LR_RANGE_MAKE(ctx->runs[k].mode,
+								chunk)) < 0)
+				return -1;
+			rem -= chunk;
+		}
+	}
+
+	total_ool = ctx->nr_ranges - start_idx;
+	for (size_t pos = 0; pos < total_ool; pos += LR_OOL_COUNT_MAX) {
+		size_t slice_idx = start_idx + pos;
+		size_t slice_cnt = total_ool - pos;
+		u64 slice_insns = 0;
+
+		if (slice_cnt > LR_OOL_COUNT_MAX)
+			slice_cnt = LR_OOL_COUNT_MAX;
+		if (slice_idx > LR_OOL_IDX_MAX)
+			return -1;
+
+		desc = LR_OOL_MAKE(sp_off, slice_idx, slice_cnt);
+		if (append_fde_entry(ctx, eh_rela, byte_off, desc) < 0)
+			return -1;
+
+		for (size_t j = 0; j < slice_cnt; j++)
+			slice_insns += LR_RANGE_LEN(ctx->ranges[slice_idx + j]);
+		byte_off += (s64)(slice_insns << 2);
+	}
+
+	return 0;
+}
+
+static int classify_cfi_state(const struct eh_frame_reg_state *r,
+			      enum lr_unwind_mode *mode, int *sp_off)
+{
+	s64 disp;
+
+	if (r->fp_rule == EH_REG_UNSUPPORTED ||
+	    r->lr_rule == EH_REG_UNSUPPORTED)
+		return -1;
+
+	if (r->lr_rule == EH_REG_UNMODIFIED) {
+		if (r->fp_rule != EH_REG_UNMODIFIED)
+			return -1;
+		*mode = LR_UNWIND_LIVE;
+		return 0;
+	}
+
+	/*
+	 * When LR (x30) is saved on the stack, require a standard arm64 frame
+	 * record {x29, x30} with x30 at fp_addr + 8. Functions that spill x30
+	 * without x29 (e.g. leaf functions using x30 as a scratch GPR) are
+	 * omitted so the unwinder treats them as unreliable.
+	 */
+	if (r->fp_rule != EH_REG_OFFSET || r->lr_off != r->fp_off + 8)
+		return -1;
+
+	disp = r->cfa_off + r->fp_off;
+
+	if (r->cfa_reg == AARCH64_DWARF_FP) {
+		if (disp != 0)
+			return -1;
+		*mode = LR_UNWIND_FP;
+		return 0;
+	}
+
+	if (r->cfa_reg == AARCH64_DWARF_SP) {
+		if (disp < 0 || (disp & 0xf) || (disp >> 4) > LR_FDE_SP_OFF_MAX)
+			return -1;
+		if (*sp_off >= 0 && *sp_off != (disp >> 4))
+			return -1;
+		*sp_off = disp >> 4;
+		*mode = LR_UNWIND_SP;
+		return 0;
+	}
+
+	return -1;
+}
+
+static int process_fde(struct lr_unwind_ctx *ctx,
+		       const struct eh_frame_fde_iter *it, u64 fde_off)
+{
+	s64 raw_range = it->use_sdata8 ? it->fde->range64 : it->fde->range;
+	struct eh_frame_cfi_state st = { .reg = it->cie_reg };
+	const Elf64_Rela *eh_rela;
+	size_t nr_runs = 0;
+	int sp_off = -1;
+	u32 range_bytes;
+	u32 prev_loc = 0;
+	const u8 *opcode;
+	int size;
+
+	while (ctx->rela_cursor < ctx->nr_eh_relas &&
+	       ctx->eh_relas[ctx->rela_cursor].r_offset < fde_off + 8)
+		ctx->rela_cursor++;
+
+	if (ctx->rela_cursor >= ctx->nr_eh_relas ||
+	    ctx->eh_relas[ctx->rela_cursor].r_offset != fde_off + 8)
+		return -1;
+	eh_rela = &ctx->eh_relas[ctx->rela_cursor++];
+
+	if (raw_range < 4 || raw_range > U32_MAX)
+		return 0;
+	range_bytes = (u32)raw_range;
+
+	if (eh_frame_fde_opcodes(it->fde, it->use_sdata8, &opcode, &size))
+		return -1;
+
+	for (;;) {
+		struct eh_frame_reg_state span_reg = st.reg;
+		bool done = (size <= 0);
+		u32 clamped, insns;
+
+		if (done)
+			st.loc = range_bytes;
+		else if (eh_frame_step_opcode(&opcode, &size, it, &st))
+			return -1;
+
+		clamped = st.loc < range_bytes ? (u32)st.loc : range_bytes;
+		insns = clamped > prev_loc ? (clamped - prev_loc) >> 2 : 0;
+
+		if (insns) {
+			enum lr_unwind_mode mode;
+
+			if (classify_cfi_state(&span_reg, &mode, &sp_off) < 0)
+				return 0;
+
+			if (nr_runs && ctx->runs[nr_runs - 1].mode == mode)
+				ctx->runs[nr_runs - 1].len += insns;
+			else if (push_run(ctx, &nr_runs, mode, insns) < 0)
+				return -1;
+			prev_loc += insns << 2;
+		}
+
+		if (done)
+			break;
+	}
+
+	if (st.stack_depth != 0)
+		return -1;
+	if (!nr_runs)
+		return 0;
+
+	return emit_fde(ctx, eh_rela, nr_runs, sp_off < 0 ? 0 : (u32)sp_off);
+}
+
+static int parse_eh_frame(struct lr_unwind_ctx *ctx)
+{
+	const u8 *eh_base = ctx->map + ctx->eh_sec->sh_offset;
+	struct eh_frame_fde_iter it =
+		EH_FRAME_FDE_ITER_INIT(eh_base, ctx->eh_sec->sh_size);
+	int ret;
+
+	ctx->eh_relas = ctx->map + ctx->rela_eh_sec->sh_offset;
+	ctx->nr_eh_relas = ctx->rela_eh_sec->sh_size / sizeof(Elf64_Rela);
+
+	while (!(ret = eh_frame_next_fde(&it)) && it.fde) {
+		u64 fde_off = (const u8 *)it.fde - eh_base;
+
+		if (process_fde(ctx, &it, fde_off) < 0)
+			return -1;
+	}
+
+	return ret ? -1 : 0;
+}
+
 static int emit_lr_unwind_sections(const char *fname, struct lr_unwind_ctx *ctx,
 				   unsigned int orig_shnum,
 				   unsigned int shstrndx)
@@ -124,7 +429,8 @@ int main(int argc, char *argv[])
 	}
 
 	if (!parse_eh_frame(&ctx) &&
-	    !emit_lr_unwind_sections(argv[1], &ctx, shnum, shstrndx))
+	    (!ctx.nr_fdes ||
+	     !emit_lr_unwind_sections(argv[1], &ctx, shnum, shstrndx)))
 		ret = 0;
 	else
 		fprintf(stderr, "gen_lr_unwind: failed to generate .lr_unwind for %s\n",
@@ -133,5 +439,9 @@ int main(int argc, char *argv[])
 out:
 	if (ctx.map)
 		munmap((void *)ctx.map, ctx.map_size);
+	free(ctx.fdes);
+	free(ctx.fde_relas);
+	free(ctx.ranges);
+	free(ctx.runs);
 	return ret;
 }
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 05/12] scripts/gen_lr_unwind: emit .lr_unwind_fde and .lr_unwind_ranges
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (3 preceding siblings ...)
  2026-11-17  4:34 ` [RFC PATCH 04/12] arm64, unwind: parse .eh_frame into .lr_unwind entries in gen_lr_unwind Dylan Hatch
@ 2026-11-17  4:34 ` Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 06/12] scripts/gen_lr_unwind: add --dump option to inspect .lr_unwind tables Dylan Hatch
                   ` (6 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Implement emit_lr_unwind_sections() in scripts/gen_lr_unwind to append
.lr_unwind_fde, .rela.lr_unwind_fde, and .lr_unwind_ranges sections to
the target ELF relocatable object and update its .shstrtab and section
header table.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 scripts/gen_lr_unwind.c | 140 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 134 insertions(+), 6 deletions(-)

diff --git a/scripts/gen_lr_unwind.c b/scripts/gen_lr_unwind.c
index 118f995b2c48b..f65edd86ac78c 100644
--- a/scripts/gen_lr_unwind.c
+++ b/scripts/gen_lr_unwind.c
@@ -54,6 +54,19 @@ struct lr_unwind_ctx {
 	size_t		cap_runs;
 };
 
+#define LR_SHSTRTAB_EXTRA \
+	(sizeof(".lr_unwind_fde") + sizeof(".rela.lr_unwind_fde") + \
+	 sizeof(".lr_unwind_ranges"))
+
+struct elf_writer {
+	int		fd;
+	u64		off;
+	Elf64_Shdr	*shdrs;
+	unsigned int	shnum;
+	char		*shstrtab;
+	size_t		str_sz;
+};
+
 static int append_fde_entry(struct lr_unwind_ctx *ctx,
 			    const Elf64_Rela *eh_rela,
 			    s64 byte_off, u32 desc)
@@ -338,16 +351,131 @@ static int parse_eh_frame(struct lr_unwind_ctx *ctx)
 	return ret ? -1 : 0;
 }
 
+static u64 align_up(u64 val, u64 align)
+{
+	return (val + align - 1) & ~(align - 1);
+}
+
+static int pwrite_all(int fd, const void *buf, size_t count, off_t offset)
+{
+	const u8 *p = buf;
+
+	while (count > 0) {
+		ssize_t n = pwrite(fd, p, count, offset);
+
+		if (n < 0) {
+			if (errno == EINTR)
+				continue;
+			return -1;
+		}
+		p += n;
+		count -= n;
+		offset += n;
+	}
+	return 0;
+}
+
+static int append_section(struct elf_writer *w, const char *name,
+			  const void *data, u64 size, u64 align,
+			  u32 type, u64 flags, u32 link, u32 info, u64 entsize)
+{
+	size_t name_len = strlen(name) + 1;
+	u64 sec_off = align_up(w->off, align);
+
+	if (size && pwrite_all(w->fd, data, size, sec_off) < 0)
+		return -1;
+
+	memcpy(w->shstrtab + w->str_sz, name, name_len);
+	w->shdrs[w->shnum++] = (Elf64_Shdr){
+		.sh_name	= w->str_sz,
+		.sh_type	= type,
+		.sh_flags	= flags,
+		.sh_offset	= sec_off,
+		.sh_size	= size,
+		.sh_link	= link,
+		.sh_info	= info,
+		.sh_addralign	= align,
+		.sh_entsize	= entsize,
+	};
+	w->str_sz += name_len;
+	w->off = sec_off + size;
+	return 0;
+}
+
 static int emit_lr_unwind_sections(const char *fname, struct lr_unwind_ctx *ctx,
 				   unsigned int orig_shnum,
 				   unsigned int shstrndx)
 {
-	/* TODO: Emit .lr_unwind_fde, .rela.lr_unwind_fde, and .lr_unwind_ranges */
-	(void)fname;
-	(void)ctx;
-	(void)orig_shnum;
-	(void)shstrndx;
-	return 0;
+	const Elf64_Shdr *orig_shdrs =
+		ctx->map + ((const Elf64_Ehdr *)ctx->map)->e_shoff;
+	size_t orig_str_sz = orig_shdrs[shstrndx].sh_size;
+	Elf64_Ehdr ehdr = *(const Elf64_Ehdr *)ctx->map;
+	struct elf_writer w = {
+		.fd = -1,
+		.shnum = orig_shnum,
+		.str_sz = orig_str_sz,
+		.shdrs = calloc(orig_shnum + 3, sizeof(Elf64_Shdr)),
+		.shstrtab = malloc(orig_str_sz + LR_SHSTRTAB_EXTRA),
+	};
+	unsigned int fde_shndx = orig_shnum;
+	int ret = -1;
+
+	if (!w.shdrs || !w.shstrtab)
+		goto out;
+
+	memcpy(w.shdrs, orig_shdrs, orig_shnum * sizeof(Elf64_Shdr));
+	memcpy(w.shstrtab, ctx->map + orig_shdrs[shstrndx].sh_offset,
+	       orig_str_sz);
+
+	w.off = (ehdr.e_shoff + (u64)orig_shnum * sizeof(Elf64_Shdr) ==
+		 ctx->map_size) ? ehdr.e_shoff : ctx->map_size;
+
+	munmap((void *)ctx->map, ctx->map_size);
+	ctx->map = NULL;
+
+	w.fd = open(fname, O_RDWR);
+	if (w.fd < 0)
+		goto out;
+
+	if (append_section(&w, ".lr_unwind_fde", ctx->fdes,
+			   ctx->nr_fdes * sizeof(*ctx->fdes), 4,
+			   SHT_PROGBITS, SHF_ALLOC, 0, 0, 0) < 0 ||
+	    append_section(&w, ".rela.lr_unwind_fde", ctx->fde_relas,
+			   ctx->nr_fdes * sizeof(*ctx->fde_relas), 8,
+			   SHT_RELA, SHF_INFO_LINK, ctx->symtab_shndx,
+			   fde_shndx, sizeof(Elf64_Rela)) < 0 ||
+	    (ctx->nr_ranges &&
+	     append_section(&w, ".lr_unwind_ranges", ctx->ranges,
+			    ctx->nr_ranges * sizeof(*ctx->ranges), 2,
+			    SHT_PROGBITS, SHF_ALLOC, 0, 0, 0) < 0))
+		goto out;
+
+	if (pwrite_all(w.fd, w.shstrtab, w.str_sz, w.off) < 0)
+		goto out;
+	w.shdrs[shstrndx].sh_offset = w.off;
+	w.shdrs[shstrndx].sh_size = w.str_sz;
+	w.off = align_up(w.off + w.str_sz, 8);
+
+	if (ehdr.e_shnum == SHN_UNDEF || w.shnum >= SHN_LORESERVE) {
+		ehdr.e_shnum = SHN_UNDEF;
+		w.shdrs[0].sh_size = w.shnum;
+	} else {
+		ehdr.e_shnum = w.shnum;
+	}
+	ehdr.e_shoff = w.off;
+
+	if (pwrite_all(w.fd, w.shdrs, w.shnum * sizeof(Elf64_Shdr), w.off) < 0 ||
+	    pwrite_all(w.fd, &ehdr, sizeof(ehdr), 0) < 0 ||
+	    ftruncate(w.fd, w.off + w.shnum * sizeof(Elf64_Shdr)) < 0)
+		goto out;
+
+	ret = 0;
+out:
+	if (w.fd >= 0)
+		close(w.fd);
+	free(w.shstrtab);
+	free(w.shdrs);
+	return ret;
 }
 
 int main(int argc, char *argv[])
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 06/12] scripts/gen_lr_unwind: add --dump option to inspect .lr_unwind tables
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (4 preceding siblings ...)
  2026-11-17  4:34 ` [RFC PATCH 05/12] scripts/gen_lr_unwind: emit .lr_unwind_fde and .lr_unwind_ranges Dylan Hatch
@ 2026-11-17  4:34 ` Dylan Hatch
  2026-11-17  4:34 ` [RFC PATCH 07/12] scripts/sorttable: sort .lr_unwind_fde by start_offset Dylan Hatch
                   ` (5 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Add a --dump (-d) flag to scripts/gen_lr_unwind to dump .lr_unwind FDEs
and instruction ranges in human-readable format. Support dumping both
linked/post-processed objects containing .lr_unwind_fde and individual
relocatable .o files containing only .eh_frame.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 scripts/gen_lr_unwind.c | 268 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 257 insertions(+), 11 deletions(-)

diff --git a/scripts/gen_lr_unwind.c b/scripts/gen_lr_unwind.c
index f65edd86ac78c..2058601ec6135 100644
--- a/scripts/gen_lr_unwind.c
+++ b/scripts/gen_lr_unwind.c
@@ -35,6 +35,8 @@ struct lr_unwind_ctx {
 	const Elf64_Shdr *eh_sec;
 	const Elf64_Shdr *rela_eh_sec;
 	const Elf64_Shdr *fde_sec;
+	const Elf64_Shdr *rela_fde_sec;
+	const Elf64_Shdr *ranges_sec;
 	unsigned int	symtab_shndx;
 
 	const Elf64_Rela *eh_relas;
@@ -67,6 +69,13 @@ struct elf_writer {
 	size_t		str_sz;
 };
 
+struct dump_sym {
+	u64		addr;
+	unsigned int	shndx;
+	bool		is_func;
+	const char	*name;
+};
+
 static int append_fde_entry(struct lr_unwind_ctx *ctx,
 			    const Elf64_Rela *eh_rela,
 			    s64 byte_off, u32 desc)
@@ -478,6 +487,225 @@ static int emit_lr_unwind_sections(const char *fname, struct lr_unwind_ctx *ctx,
 	return ret;
 }
 
+static int cmp_dump_sym(const void *a, const void *b)
+{
+	const struct dump_sym *sa = a, *sb = b;
+
+	if (sa->shndx != sb->shndx)
+		return sa->shndx < sb->shndx ? -1 : 1;
+	if (sa->addr != sb->addr)
+		return sa->addr < sb->addr ? -1 : 1;
+	if (sa->is_func != sb->is_func)
+		return sa->is_func ? 1 : -1;
+	return 0;
+}
+
+static const struct dump_sym *find_dump_sym(const struct dump_sym *syms,
+					    size_t nr_syms,
+					    unsigned int shndx, u64 addr)
+{
+	size_t lo = 0, hi = nr_syms;
+	const struct dump_sym *best = NULL;
+
+	while (lo < hi) {
+		size_t mid = lo + (hi - lo) / 2;
+		const struct dump_sym *s = &syms[mid];
+
+		if ((shndx && s->shndx < shndx) ||
+		    ((!shndx || s->shndx == shndx) && s->addr <= addr)) {
+			if (!shndx || s->shndx == shndx)
+				best = s;
+			lo = mid + 1;
+		} else {
+			hi = mid;
+		}
+	}
+
+	return best;
+}
+
+static const char *mode_name(enum lr_unwind_mode mode)
+{
+	switch (mode) {
+	case LR_UNWIND_LIVE:
+		return "LIVE";
+	case LR_UNWIND_FP:
+		return "FP";
+	case LR_UNWIND_SP:
+		return "SP";
+	default:
+		return "UNKNOWN";
+	}
+}
+
+static int dump_lr_unwind(struct lr_unwind_ctx *ctx, const Elf64_Shdr *shdrs,
+			  unsigned int shnum, const char *shstrtab)
+{
+	const struct lr_fde *fdes;
+	const Elf64_Rela *relas = NULL;
+	const lr_range_t *ranges = NULL;
+	size_t nr_fdes, nr_ranges = 0;
+	const Elf64_Sym *elf_syms = NULL;
+	size_t nr_elf_syms = 0;
+	const char *strtab = NULL;
+	struct dump_sym *dsyms = NULL;
+	size_t nr_dsyms = 0;
+	bool is_rel = (((const Elf64_Ehdr *)ctx->map)->e_type == ET_REL);
+
+	if (ctx->fde_sec) {
+		fdes = ctx->map + ctx->fde_sec->sh_offset;
+		nr_fdes = ctx->fde_sec->sh_size / sizeof(struct lr_fde);
+		if (ctx->rela_fde_sec)
+			relas = ctx->map + ctx->rela_fde_sec->sh_offset;
+		if (ctx->ranges_sec) {
+			ranges = ctx->map + ctx->ranges_sec->sh_offset;
+			nr_ranges = ctx->ranges_sec->sh_size / sizeof(lr_range_t);
+		}
+	} else {
+		if (!ctx->eh_sec || !ctx->rela_eh_sec || !ctx->symtab_shndx) {
+			fprintf(stderr, "No .lr_unwind_fde or .eh_frame section found\n");
+			return -1;
+		}
+		if (parse_eh_frame(ctx) < 0)
+			return -1;
+		fdes = ctx->fdes;
+		nr_fdes = ctx->nr_fdes;
+		relas = ctx->fde_relas;
+		ranges = ctx->ranges;
+		nr_ranges = ctx->nr_ranges;
+	}
+
+	if (ctx->symtab_shndx) {
+		const Elf64_Shdr *sym_sec = &shdrs[ctx->symtab_shndx];
+
+		elf_syms = ctx->map + sym_sec->sh_offset;
+		nr_elf_syms = sym_sec->sh_size / sizeof(Elf64_Sym);
+		strtab = ctx->map + shdrs[sym_sec->sh_link].sh_offset;
+
+		dsyms = malloc(nr_elf_syms * sizeof(*dsyms));
+		if (!dsyms)
+			return -1;
+
+		for (size_t i = 0; i < nr_elf_syms; i++) {
+			const Elf64_Sym *s = &elf_syms[i];
+			unsigned char type = ELF64_ST_TYPE(s->st_info);
+			const char *name = strtab + s->st_name;
+
+			if (s->st_shndx == SHN_UNDEF ||
+			    s->st_shndx >= SHN_LORESERVE)
+				continue;
+			if (type != STT_FUNC && type != STT_NOTYPE)
+				continue;
+			if (!name[0] || name[0] == '$' ||
+			    !strncmp(name, ".L", 2))
+				continue;
+
+			dsyms[nr_dsyms++] = (struct dump_sym){
+				.addr = s->st_value,
+				.shndx = is_rel ? s->st_shndx : 0,
+				.is_func = (type == STT_FUNC),
+				.name = name,
+			};
+		}
+
+		qsort(dsyms, nr_dsyms, sizeof(*dsyms), cmp_dump_sym);
+	}
+
+	for (size_t i = 0; i < nr_fdes; i++) {
+		const struct lr_fde *fde = &fdes[i];
+		u32 desc = fde->range_desc;
+		u32 sp_off = LR_FDE_SP_OFF_BYTES(desc);
+		lr_range_t inline_ranges[4];
+		const lr_range_t *r_list;
+		u32 r_cnt, total_insns = 0, insn_off = 0;
+
+		if (desc & LR_FDE_INLINE) {
+			lr_fde_unpack_inline(desc, inline_ranges);
+			r_list = inline_ranges;
+			r_cnt = 4;
+		} else {
+			u32 idx = LR_OOL_IDX(desc);
+
+			r_cnt = LR_OOL_COUNT(desc);
+			if (idx + r_cnt > nr_ranges) {
+				free(dsyms);
+				return -1;
+			}
+			r_list = &ranges[idx];
+		}
+
+		for (u32 j = 0; j < r_cnt; j++)
+			total_insns += LR_RANGE_LEN(r_list[j]);
+
+		printf("FDE[%zu] ", i);
+		if (relas && elf_syms) {
+			u32 sym_idx = ELF64_R_SYM(relas[i].r_info);
+
+			if (sym_idx < nr_elf_syms) {
+				const Elf64_Sym *sym = &elf_syms[sym_idx];
+				unsigned int shndx = sym->st_shndx;
+				u64 off = sym->st_value + relas[i].r_addend;
+
+				if (shndx < shnum && shndx < SHN_LORESERVE) {
+					const char *sec_name =
+						shstrtab + shdrs[shndx].sh_name;
+					const struct dump_sym *ds =
+						find_dump_sym(dsyms, nr_dsyms,
+							      shndx, off);
+
+					if (ds)
+						printf("%s+0x%lx <%s+0x%lx>",
+						       sec_name,
+						       (unsigned long)off,
+						       ds->name,
+						       (unsigned long)(off - ds->addr));
+					else
+						printf("%s+0x%lx", sec_name,
+						       (unsigned long)off);
+				}
+			}
+		} else if (ctx->fde_sec) {
+			u64 field_addr = ctx->fde_sec->sh_addr +
+					 i * sizeof(struct lr_fde);
+			u64 pc = field_addr + (s64)fde->start_offset;
+			const struct dump_sym *ds =
+				find_dump_sym(dsyms, nr_dsyms, 0, pc);
+
+			if (ds)
+				printf("0x%016lx <%s+0x%lx>", (unsigned long)pc,
+				       ds->name, (unsigned long)(pc - ds->addr));
+			else
+				printf("0x%016lx", (unsigned long)pc);
+		}
+
+		if (desc & LR_FDE_INLINE)
+			printf(" (len=0x%x, inline, sp_off=0x%x):\n",
+			       total_insns << 2, sp_off);
+		else
+			printf(" (len=0x%x, ool[%u..%u), sp_off=0x%x):\n",
+			       total_insns << 2, LR_OOL_IDX(desc),
+			       LR_OOL_IDX(desc) + r_cnt, sp_off);
+
+		for (u32 j = 0; j < r_cnt; j++) {
+			u32 len = LR_RANGE_LEN(r_list[j]);
+			enum lr_unwind_mode mode = LR_RANGE_MODE(r_list[j]);
+
+			if (!len)
+				continue;
+			printf("  [+0x%04x..+0x%04x) %4u insns  %s",
+			       insn_off << 2, (insn_off + len) << 2,
+			       len, mode_name(mode));
+			if (mode == LR_UNWIND_SP)
+				printf(" (sp+0x%x)", sp_off);
+			printf("\n");
+			insn_off += len;
+		}
+	}
+
+	free(dsyms);
+	return 0;
+}
+
 int main(int argc, char *argv[])
 {
 	struct lr_unwind_ctx ctx = { 0 };
@@ -485,22 +713,29 @@ int main(int argc, char *argv[])
 	const Elf64_Ehdr *ehdr;
 	const Elf64_Shdr *shdrs;
 	const char *shstrtab;
+	const char *fname;
+	bool dump = false;
 	struct stat sb;
 	int fd, ret = 1;
 
-	if (argc != 2) {
-		fprintf(stderr, "Usage: %s <file>\n", argv[0]);
+	if (argc == 3 && (!strcmp(argv[1], "--dump") || !strcmp(argv[1], "-d"))) {
+		dump = true;
+		fname = argv[2];
+	} else if (argc == 2 && argv[1][0] != '-') {
+		fname = argv[1];
+	} else {
+		fprintf(stderr, "Usage: %s [--dump] <file>\n", argv[0]);
 		return 1;
 	}
 
-	fd = open(argv[1], O_RDONLY);
+	fd = open(fname, O_RDONLY);
 	if (fd < 0) {
 		fprintf(stderr, "gen_lr_unwind: cannot open %s: %s\n",
-			argv[1], strerror(errno));
+			fname, strerror(errno));
 		return 1;
 	}
 	if (fstat(fd, &sb) < 0 || (size_t)sb.st_size < sizeof(Elf64_Ehdr)) {
-		fprintf(stderr, "gen_lr_unwind: invalid file %s\n", argv[1]);
+		fprintf(stderr, "gen_lr_unwind: invalid file %s\n", fname);
 		close(fd);
 		return 1;
 	}
@@ -510,7 +745,7 @@ int main(int argc, char *argv[])
 	close(fd);
 	if (ctx.map == MAP_FAILED) {
 		fprintf(stderr, "gen_lr_unwind: mmap failed on %s: %s\n",
-			argv[1], strerror(errno));
+			fname, strerror(errno));
 		return 1;
 	}
 
@@ -518,10 +753,12 @@ int main(int argc, char *argv[])
 	if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) ||
 	    ehdr->e_ident[EI_CLASS] != ELFCLASS64 ||
 	    ehdr->e_ident[EI_DATA] != ELFDATA2LSB ||
-	    ehdr->e_type != ET_REL ||
+	    (!dump && ehdr->e_type != ET_REL) ||
+	    (dump && ehdr->e_type != ET_REL && ehdr->e_type != ET_EXEC &&
+	     ehdr->e_type != ET_DYN) ||
 	    ehdr->e_machine != EM_AARCH64) {
-		fprintf(stderr, "gen_lr_unwind: %s is not an AArch64 ET_REL ELF\n",
-			argv[1]);
+		fprintf(stderr, "gen_lr_unwind: unsupported ELF file %s\n",
+			fname);
 		goto out;
 	}
 
@@ -536,6 +773,10 @@ int main(int argc, char *argv[])
 
 		if (!strcmp(name, ".lr_unwind_fde"))
 			ctx.fde_sec = &shdrs[i];
+		else if (!strcmp(name, ".rela.lr_unwind_fde"))
+			ctx.rela_fde_sec = &shdrs[i];
+		else if (!strcmp(name, ".lr_unwind_ranges"))
+			ctx.ranges_sec = &shdrs[i];
 		else if (!strcmp(name, ".eh_frame") ||
 			 !strcmp(name, ".init.eh_frame"))
 			ctx.eh_sec = &shdrs[i];
@@ -546,6 +787,11 @@ int main(int argc, char *argv[])
 			ctx.symtab_shndx = i;
 	}
 
+	if (dump) {
+		ret = dump_lr_unwind(&ctx, shdrs, shnum, shstrtab) ? 1 : 0;
+		goto out;
+	}
+
 	if (ctx.fde_sec) {
 		ret = 0;
 		goto out;
@@ -558,11 +804,11 @@ int main(int argc, char *argv[])
 
 	if (!parse_eh_frame(&ctx) &&
 	    (!ctx.nr_fdes ||
-	     !emit_lr_unwind_sections(argv[1], &ctx, shnum, shstrndx)))
+	     !emit_lr_unwind_sections(fname, &ctx, shnum, shstrndx)))
 		ret = 0;
 	else
 		fprintf(stderr, "gen_lr_unwind: failed to generate .lr_unwind for %s\n",
-			argv[1]);
+			fname);
 
 out:
 	if (ctx.map)
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 07/12] scripts/sorttable: sort .lr_unwind_fde by start_offset
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (5 preceding siblings ...)
  2026-11-17  4:34 ` [RFC PATCH 06/12] scripts/gen_lr_unwind: add --dump option to inspect .lr_unwind tables Dylan Hatch
@ 2026-11-17  4:34 ` Dylan Hatch
  2026-11-17  4:35 ` [RFC PATCH 08/12] arm64, unwind: add in-kernel .lr_unwind lookup and validation Dylan Hatch
                   ` (4 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:34 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Normalize the PC-relative start_offset word of each 8-byte struct lr_fde
entry in .lr_unwind_fde, qsort by virtual address, and denormalize back
to PC-relative offsets without modifying range_desc.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 scripts/sorttable.c | 50 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 38 insertions(+), 12 deletions(-)

diff --git a/scripts/sorttable.c b/scripts/sorttable.c
index 88e49a5c42512..38495c39e404b 100644
--- a/scripts/sorttable.c
+++ b/scripts/sorttable.c
@@ -600,6 +600,37 @@ static void get_mcount_loc(struct elf_mcount_loc *emloc, Elf_Shdr *symtab_sec,
 static inline int parse_symbols(const char *fname) { return 0; }
 #endif
 
+static int compare_relative_table(const void *a, const void *b)
+{
+	int32_t av = (int32_t)elf_parser.r(a);
+	int32_t bv = (int32_t)elf_parser.r(b);
+
+	if (av < bv)
+		return -1;
+	if (av > bv)
+		return 1;
+	return 0;
+}
+
+static void sort_lr_unwind_fde(char *fde_image, int image_size)
+{
+	int i;
+
+	for (i = 0; i < image_size; i += 8) {
+		uint32_t *loc = (uint32_t *)(fde_image + i);
+
+		elf_parser.w(elf_parser.r(loc) + i, loc);
+	}
+
+	qsort(fde_image, image_size / 8, 8, compare_relative_table);
+
+	for (i = 0; i < image_size; i += 8) {
+		uint32_t *loc = (uint32_t *)(fde_image + i);
+
+		elf_parser.w(elf_parser.r(loc) - i, loc);
+	}
+}
+
 static int do_sort(Elf_Ehdr *ehdr,
 		   char const *const fname,
 		   table_sort_t custom_sort)
@@ -609,6 +640,7 @@ static int do_sort(Elf_Ehdr *ehdr,
 	Elf_Shdr *strtab_sec = NULL;
 	Elf_Shdr *symtab_sec = NULL;
 	Elf_Shdr *extab_sec = NULL;
+	Elf_Shdr *lr_fde_sec = NULL;
 	Elf_Shdr *string_sec;
 	Elf_Sym *sym;
 	const Elf_Sym *symtab;
@@ -656,6 +688,8 @@ static int do_sort(Elf_Ehdr *ehdr,
 		idx = shdr_name(shdr);
 		if (!strcmp(secstrings + idx, "__ex_table"))
 			extab_sec = shdr;
+		if (!strcmp(secstrings + idx, ".lr_unwind_fde"))
+			lr_fde_sec = shdr;
 		if (!strcmp(secstrings + idx, ".symtab"))
 			symtab_sec = shdr;
 		if (!strcmp(secstrings + idx, ".strtab"))
@@ -759,6 +793,10 @@ static int do_sort(Elf_Ehdr *ehdr,
 		      extable_ent_size, compare_extable);
 	}
 
+	if (lr_fde_sec)
+		sort_lr_unwind_fde((char *)ehdr + shdr_offset(lr_fde_sec),
+				   shdr_size(lr_fde_sec));
+
 	/* find the flag main_extable_sort_needed */
 	sym_start = (void *)ehdr + shdr_offset(symtab_sec);
 	sym_end = sym_start + shdr_size(symtab_sec);
@@ -833,18 +871,6 @@ static int do_sort(Elf_Ehdr *ehdr,
 	return rc;
 }
 
-static int compare_relative_table(const void *a, const void *b)
-{
-	int32_t av = (int32_t)elf_parser.r(a);
-	int32_t bv = (int32_t)elf_parser.r(b);
-
-	if (av < bv)
-		return -1;
-	if (av > bv)
-		return 1;
-	return 0;
-}
-
 static void sort_relative_table(char *extab_image, int image_size)
 {
 	int i = 0;
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 08/12] arm64, unwind: add in-kernel .lr_unwind lookup and validation
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (6 preceding siblings ...)
  2026-11-17  4:34 ` [RFC PATCH 07/12] scripts/sorttable: sort .lr_unwind_fde by start_offset Dylan Hatch
@ 2026-11-17  4:35 ` Dylan Hatch
  2026-11-17  4:35 ` [RFC PATCH 09/12] arm64/module, unwind: add .lr_unwind support for modules Dylan Hatch
                   ` (3 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:35 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Add runtime lookup and optional boot-time validation for the vmlinux
.lr_unwind_fde and .lr_unwind_ranges tables.

lr_unwind_find() binary searches the sorted .lr_unwind_fde table with
monotonic bounds checking, validates start IPs against arm64 kernel text
sections (.text, .init.text, .exit.text, and .rodata.text), and scans
the inline or out-of-line instruction ranges to determine the exception
boundary unwind mode (LR_UNWIND_LIVE, LR_UNWIND_FP, or LR_UNWIND_SP) and
any SP-relative frame-record offset.

Suggested-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/Kconfig                 |   9 +
 arch/arm64/include/asm/lr_unwind.h |  31 ++++
 arch/arm64/include/asm/sections.h  |   3 +
 arch/arm64/kernel/Makefile         |   5 +
 arch/arm64/kernel/lr_unwind.c      | 274 +++++++++++++++++++++++++++++
 arch/arm64/kernel/setup.c          |   3 +
 arch/arm64/kernel/vmlinux.lds.S    |   2 +
 7 files changed, 327 insertions(+)
 create mode 100644 arch/arm64/kernel/lr_unwind.c

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index aa09c7ea61be3..1503b8a0b7021 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2517,6 +2517,15 @@ config ARM64_LR_UNWIND
 	  live in x30, saved in the frame record at [x29], or saved in the
 	  frame record at an SP-relative offset).
 
+config ARM64_LR_UNWIND_VALIDATION
+	bool "Validate .lr_unwind sections at boot and module load"
+	depends on ARM64_LR_UNWIND
+	help
+	  Validate every FDE and range entry in .lr_unwind_fde and
+	  .lr_unwind_ranges when initializing vmlinux and module unwind
+	  tables. This is a debugging option that adds boot and module load
+	  overhead and should normally be disabled in production.
+
 config ARM64_CONTPTE
 	bool "Contiguous PTE mappings for user memory" if EXPERT
 	depends on TRANSPARENT_HUGEPAGE
diff --git a/arch/arm64/include/asm/lr_unwind.h b/arch/arm64/include/asm/lr_unwind.h
index 81c743994c4d5..bfb6abed05f76 100644
--- a/arch/arm64/include/asm/lr_unwind.h
+++ b/arch/arm64/include/asm/lr_unwind.h
@@ -108,4 +108,35 @@ static __always_inline void lr_fde_unpack_inline(u32 desc, lr_range_t ranges[4])
 #define LR_OOL_IDX(d)		(((d) >> 6) & LR_OOL_IDX_MAX)
 #define LR_OOL_COUNT(d)		((d) & LR_OOL_COUNT_MAX)
 
+#ifdef __KERNEL__
+
+#include <linux/errno.h>
+#include <linux/init.h>
+
+struct lr_unwind_section {
+	const struct lr_fde	*fdes;
+	unsigned int		num_fdes;
+	const lr_range_t	*ranges;
+	unsigned int		num_ranges;
+};
+
+#ifdef CONFIG_ARM64_LR_UNWIND
+
+void __init init_lr_unwind_table(void);
+int lr_unwind_find(unsigned long ip, enum lr_unwind_mode *mode,
+		   unsigned int *sp_off);
+
+#else /* !CONFIG_ARM64_LR_UNWIND */
+
+static inline void __init init_lr_unwind_table(void) {}
+static inline int lr_unwind_find(unsigned long ip, enum lr_unwind_mode *mode,
+				 unsigned int *sp_off)
+{
+	return -EINVAL;
+}
+
+#endif /* CONFIG_ARM64_LR_UNWIND */
+
+#endif /* __KERNEL__ */
+
 #endif /* __ASM_LR_UNWIND_H */
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 51b0d594239eb..2f950aa28225f 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -23,6 +23,9 @@ extern char __irqentry_text_start[], __irqentry_text_end[];
 extern char __mmuoff_data_start[], __mmuoff_data_end[];
 extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
 extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];
+extern char _srodatatext[], _erodatatext[];
+extern char __start_lr_unwind_fde[], __stop_lr_unwind_fde[];
+extern char __start_lr_unwind_ranges[], __stop_lr_unwind_ranges[];
 
 static inline size_t entry_tramp_text_size(void)
 {
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index fc5d1a0c38b3c..f07a0cead3ee3 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -18,12 +18,16 @@ CFLAGS_syscall.o	+= -fno-stack-protector
 # can significantly impact performance. Avoid instrumenting the stack trace
 # collection code to minimize this impact.
 KASAN_SANITIZE_stacktrace.o := n
+KASAN_SANITIZE_lr_unwind.o := n
+KCSAN_SANITIZE_lr_unwind.o := n
+UBSAN_SANITIZE_lr_unwind.o := n
 
 # It's not safe to invoke KCOV when portions of the kernel environment aren't
 # available or are out-of-sync with HW state. Since `noinstr` doesn't always
 # inhibit KCOV instrumentation, disable it for the entire compilation unit.
 KCOV_INSTRUMENT_entry-common.o := n
 KCOV_INSTRUMENT_idle.o := n
+KCOV_INSTRUMENT_lr_unwind.o := n
 
 # Softirq stack switching can run outside interrupt context.
 KCOV_INSTRUMENT_irq.o := n
@@ -71,6 +75,7 @@ obj-$(CONFIG_CRASH_DUMP)		+= crash_dump.o
 obj-$(CONFIG_VMCORE_INFO)		+= vmcore_info.o
 obj-$(CONFIG_ARM_SDE_INTERFACE)		+= sdei.o
 obj-$(CONFIG_ARM64_PTR_AUTH)		+= pointer_auth.o
+obj-$(CONFIG_ARM64_LR_UNWIND)		+= lr_unwind.o
 obj-$(CONFIG_ARM64_MPAM)		+= mpam.o
 obj-$(CONFIG_ARM64_MTE)			+= mte.o
 obj-y					+= vdso-wrap.o
diff --git a/arch/arm64/kernel/lr_unwind.c b/arch/arm64/kernel/lr_unwind.c
new file mode 100644
index 0000000000000..53e2c78a8bdaf
--- /dev/null
+++ b/arch/arm64/kernel/lr_unwind.c
@@ -0,0 +1,274 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Kernel .lr_unwind lookup and validation
+ */
+
+#define pr_fmt(fmt)	"lr_unwind: " fmt
+
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/kprobes.h>
+#include <linux/limits.h>
+#include <linux/printk.h>
+
+#include <asm/lr_unwind.h>
+#include <asm/sections.h>
+
+static bool lr_unwind_init __ro_after_init;
+static struct lr_unwind_section kernel_lr_sec __ro_after_init;
+
+static const char *objname(const struct lr_unwind_section *sec)
+{
+	return "vmlinux";
+}
+
+#define dbg_sec(sec, fmt, ...) \
+	pr_debug("(%s): " fmt, objname(sec), ##__VA_ARGS__)
+
+static __always_inline unsigned long lr_fde_start_addr(const struct lr_fde *fde)
+{
+	return (unsigned long)&fde->start_offset + fde->start_offset;
+}
+
+static __always_inline bool is_kernel_ip_valid(unsigned long ip)
+{
+	if (ip >= (unsigned long)_stext && ip < (unsigned long)_etext)
+		return true;
+
+	if (system_state < SYSTEM_FREEING_INITMEM &&
+	    ip >= (unsigned long)_sinittext && ip < (unsigned long)_einittext)
+		return true;
+
+	/* .exit.text is retained in vmlinux on arm64. */
+	if (ip >= (unsigned long)__exittext_begin &&
+	    ip < (unsigned long)__exittext_end)
+		return true;
+
+	/*
+	 * .rodata.text is never executed from the kernel mapping, but it still
+	 * has unwind data.
+	 */
+	if (ip >= (unsigned long)_srodatatext &&
+	    ip < (unsigned long)_erodatatext)
+		return true;
+
+	return false;
+}
+
+static __always_inline bool is_sec_ip_valid(const struct lr_unwind_section *sec,
+					    unsigned long ip)
+{
+	return is_kernel_ip_valid(ip);
+}
+
+static __always_inline int __find_fde(const struct lr_unwind_section *sec,
+				      unsigned long ip,
+				      const struct lr_fde **fde_out)
+{
+	unsigned long func_addr_low = 0, func_addr_high = ULONG_MAX;
+	unsigned int low = 0, high = sec->num_fdes;
+	const struct lr_fde *found = NULL;
+
+	while (low < high) {
+		unsigned int mid = low + ((high - low) / 2);
+		const struct lr_fde *fde = &sec->fdes[mid];
+		unsigned long func_addr = lr_fde_start_addr(fde);
+
+		if (ip >= func_addr) {
+			if (func_addr < func_addr_low)
+				return -EFAULT;
+
+			func_addr_low = func_addr;
+			found = fde;
+			low = mid + 1;
+		} else {
+			if (func_addr > func_addr_high)
+				return -EFAULT;
+
+			func_addr_high = func_addr;
+			high = mid;
+		}
+	}
+
+	if (!found || !is_sec_ip_valid(sec, lr_fde_start_addr(found)))
+		return -EINVAL;
+
+	*fde_out = found;
+	return 0;
+}
+
+static __always_inline int __lr_unwind_find(const struct lr_unwind_section *sec,
+					    unsigned long ip,
+					    enum lr_unwind_mode *mode,
+					    unsigned int *sp_off)
+{
+	lr_range_t inline_ranges[4];
+	const lr_range_t *ranges;
+	const struct lr_fde *fde;
+	unsigned long insn_off;
+	u32 desc, cnt;
+	int ret;
+
+	if (ip & 0x3)
+		return -EINVAL;
+
+	ret = __find_fde(sec, ip, &fde);
+	if (ret)
+		return ret;
+
+	insn_off = (ip - lr_fde_start_addr(fde)) >> 2;
+	desc = fde->range_desc;
+
+	if (likely(desc & LR_FDE_INLINE)) {
+		lr_fde_unpack_inline(desc, inline_ranges);
+		ranges = inline_ranges;
+		cnt = 4;
+	} else {
+		u32 idx = LR_OOL_IDX(desc);
+
+		cnt = LR_OOL_COUNT(desc);
+		if (!cnt || idx + cnt > sec->num_ranges)
+			return -EFAULT;
+		ranges = &sec->ranges[idx];
+	}
+
+	for (u32 i = 0; i < cnt; i++) {
+		u32 len = LR_RANGE_LEN(ranges[i]);
+
+		if (insn_off < len) {
+			enum lr_unwind_mode m = LR_RANGE_MODE(ranges[i]);
+
+			if (unlikely(m > LR_UNWIND_SP))
+				return -EFAULT;
+			*mode = m;
+			*sp_off = LR_FDE_SP_OFF_BYTES(desc);
+			return 0;
+		}
+		insn_off -= len;
+	}
+
+	/* Make sure IP is not in an unannotated gap past this FDE */
+	return -EINVAL;
+}
+
+noinstr int lr_unwind_find(unsigned long ip, enum lr_unwind_mode *mode,
+			   unsigned int *sp_off)
+{
+	if (!mode || !sp_off)
+		return -EINVAL;
+
+	if (is_kernel_ip_valid(ip)) {
+		if (!lr_unwind_init)
+			return -EINVAL;
+
+		return __lr_unwind_find(&kernel_lr_sec, ip, mode, sp_off);
+	}
+
+	return -EINVAL;
+}
+NOKPROBE_SYMBOL(lr_unwind_find);
+
+#ifdef CONFIG_ARM64_LR_UNWIND_VALIDATION
+
+static int lr_unwind_validate_section(const struct lr_unwind_section *sec)
+{
+	unsigned long prev_end = 0;
+
+	for (unsigned int i = 0; i < sec->num_fdes; i++) {
+		const struct lr_fde *fde = &sec->fdes[i];
+		unsigned long ip = lr_fde_start_addr(fde);
+		u32 desc = fde->range_desc;
+		u32 total_insns = 0;
+
+		if ((ip & 0x3) || !is_sec_ip_valid(sec, ip)) {
+			dbg_sec(sec, "FDE %u invalid start IP 0x%lx\n", i, ip);
+			return -EINVAL;
+		}
+
+		if (ip < prev_end) {
+			dbg_sec(sec, "FDE %u unsorted or overlapping\n", i);
+			return -EFAULT;
+		}
+
+		if (desc & LR_FDE_INLINE) {
+			total_insns = LR_INLINE_R0(desc) +
+				      LR_INLINE_R1(desc) +
+				      LR_INLINE_R2(desc) +
+				      LR_INLINE_R3(desc);
+		} else {
+			u32 idx = LR_OOL_IDX(desc);
+			u32 cnt = LR_OOL_COUNT(desc);
+
+			if (!cnt || idx + cnt > sec->num_ranges) {
+				dbg_sec(sec, "FDE %u OOL range out of bounds\n", i);
+				return -EFAULT;
+			}
+			for (u32 j = 0; j < cnt; j++) {
+				lr_range_t r = sec->ranges[idx + j];
+				u32 len = LR_RANGE_LEN(r);
+
+				if (!len || LR_RANGE_MODE(r) > LR_UNWIND_SP) {
+					dbg_sec(sec, "FDE %u invalid OOL entry\n", i);
+					return -EINVAL;
+				}
+				total_insns += len;
+			}
+		}
+
+		if (!total_insns) {
+			dbg_sec(sec, "FDE %u has empty range\n", i);
+			return -EINVAL;
+		}
+
+		prev_end = ip + ((unsigned long)total_insns << 2);
+		if (!is_sec_ip_valid(sec, prev_end - 4)) {
+			dbg_sec(sec, "FDE %u end IP out of bounds\n", i);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+#else /* !CONFIG_ARM64_LR_UNWIND_VALIDATION */
+
+static int lr_unwind_validate_section(const struct lr_unwind_section *sec)
+{
+	return 0;
+}
+
+#endif /* CONFIG_ARM64_LR_UNWIND_VALIDATION */
+
+static void dbg_print_section(const struct lr_unwind_section *sec)
+{
+	dbg_sec(sec, "fdes:0x%lx (%u) ranges:0x%lx (%u)\n",
+		(unsigned long)sec->fdes, sec->num_fdes,
+		(unsigned long)sec->ranges, sec->num_ranges);
+}
+
+void __init init_lr_unwind_table(void)
+{
+	size_t fde_bytes = (unsigned long)__stop_lr_unwind_fde -
+			   (unsigned long)__start_lr_unwind_fde;
+	size_t range_bytes = (unsigned long)__stop_lr_unwind_ranges -
+			     (unsigned long)__start_lr_unwind_ranges;
+
+	if (!fde_bytes || (fde_bytes % sizeof(struct lr_fde)) ||
+	    (range_bytes % sizeof(lr_range_t))) {
+		pr_warn("invalid vmlinux .lr_unwind section sizes\n");
+		return;
+	}
+
+	kernel_lr_sec.fdes = (const struct lr_fde *)__start_lr_unwind_fde;
+	kernel_lr_sec.num_fdes = fde_bytes / sizeof(struct lr_fde);
+	kernel_lr_sec.ranges = (const lr_range_t *)__start_lr_unwind_ranges;
+	kernel_lr_sec.num_ranges = range_bytes / sizeof(lr_range_t);
+
+	if (lr_unwind_validate_section(&kernel_lr_sec)) {
+		pr_warn("invalid vmlinux .lr_unwind section\n");
+		return;
+	}
+
+	lr_unwind_init = true;
+	dbg_print_section(&kernel_lr_sec);
+}
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index a707deba6ae70..a19563e382ebd 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -43,6 +43,7 @@
 #include <asm/cpufeature.h>
 #include <asm/cpu_ops.h>
 #include <asm/kasan.h>
+#include <asm/lr_unwind.h>
 #include <asm/numa.h>
 #include <asm/scs.h>
 #include <asm/sections.h>
@@ -373,6 +374,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 			"This indicates a broken bootloader or old kernel\n",
 			boot_args[1], boot_args[2], boot_args[3]);
 	}
+
+	init_lr_unwind_table();
 }
 
 static inline bool cpu_can_disable(unsigned int cpu)
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index b8f2963d10030..dc6c4a041f691 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -256,12 +256,14 @@ SECTIONS
 
 	/* code sections that are never executed via the kernel mapping */
 	.rodata.text : {
+		_srodatatext = .;
 		TRAMP_TEXT
 		HIBERNATE_TEXT
 		KEXEC_TEXT
 		IDMAP_TEXT
 		. = ALIGN(PAGE_SIZE);
 	}
+	_erodatatext = .;
 
 	idmap_pg_dir = .;
 	. += PAGE_SIZE;
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 09/12] arm64/module, unwind: add .lr_unwind support for modules
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (7 preceding siblings ...)
  2026-11-17  4:35 ` [RFC PATCH 08/12] arm64, unwind: add in-kernel .lr_unwind lookup and validation Dylan Hatch
@ 2026-11-17  4:35 ` Dylan Hatch
  2026-11-17  4:35 ` [RFC PATCH 10/12] arm64: entry: add unwind info for call_on_irq_stack() Dylan Hatch
                   ` (2 subsequent siblings)
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:35 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Generate .lr_unwind_fde and .lr_unwind_ranges sections for loadable
kernel modules at modfinal link time, sort each module's FDE table in
module_finalize(), and support module IP lookups in lr_unwind_find().

Co-developed-by: Weinan Liu <wnliu@google.com>
Signed-off-by: Weinan Liu <wnliu@google.com>
Suggested-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/include/asm/lr_unwind.h |   7 ++
 arch/arm64/include/asm/module.h    |   6 ++
 arch/arm64/kernel/lr_unwind.c      | 125 ++++++++++++++++++++++++++++-
 arch/arm64/kernel/module.c         |  11 +++
 scripts/Makefile.modfinal          |  10 ++-
 scripts/gen_lr_unwind.c            |   4 +-
 6 files changed, 157 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/lr_unwind.h b/arch/arm64/include/asm/lr_unwind.h
index bfb6abed05f76..649009b49becf 100644
--- a/arch/arm64/include/asm/lr_unwind.h
+++ b/arch/arm64/include/asm/lr_unwind.h
@@ -113,6 +113,8 @@ static __always_inline void lr_fde_unpack_inline(u32 desc, lr_range_t ranges[4])
 #include <linux/errno.h>
 #include <linux/init.h>
 
+struct module;
+
 struct lr_unwind_section {
 	const struct lr_fde	*fdes;
 	unsigned int		num_fdes;
@@ -123,12 +125,17 @@ struct lr_unwind_section {
 #ifdef CONFIG_ARM64_LR_UNWIND
 
 void __init init_lr_unwind_table(void);
+void lr_unwind_module_init(struct module *mod, void *fdes, size_t fdes_size,
+			   const void *ranges, size_t ranges_size);
 int lr_unwind_find(unsigned long ip, enum lr_unwind_mode *mode,
 		   unsigned int *sp_off);
 
 #else /* !CONFIG_ARM64_LR_UNWIND */
 
 static inline void __init init_lr_unwind_table(void) {}
+static inline void lr_unwind_module_init(struct module *mod, void *fdes,
+					 size_t fdes_size, const void *ranges,
+					 size_t ranges_size) {}
 static inline int lr_unwind_find(unsigned long ip, enum lr_unwind_mode *mode,
 				 unsigned int *sp_off)
 {
diff --git a/arch/arm64/include/asm/module.h b/arch/arm64/include/asm/module.h
index fb9b88eebeb15..3b6457875e453 100644
--- a/arch/arm64/include/asm/module.h
+++ b/arch/arm64/include/asm/module.h
@@ -6,6 +6,7 @@
 #define __ASM_MODULE_H
 
 #include <asm-generic/module.h>
+#include <asm/lr_unwind.h>
 
 struct mod_plt_sec {
 	int			plt_shndx;
@@ -17,6 +18,11 @@ struct mod_arch_specific {
 	struct mod_plt_sec	core;
 	struct mod_plt_sec	init;
 
+#ifdef CONFIG_ARM64_LR_UNWIND
+	struct lr_unwind_section lr_unwind_sec;
+	bool			lr_unwind_init;
+#endif
+
 	/* for CONFIG_DYNAMIC_FTRACE */
 	struct plt_entry	*ftrace_trampolines;
 	struct plt_entry	*init_ftrace_trampolines;
diff --git a/arch/arm64/kernel/lr_unwind.c b/arch/arm64/kernel/lr_unwind.c
index 53e2c78a8bdaf..d554c2e1e57e2 100644
--- a/arch/arm64/kernel/lr_unwind.c
+++ b/arch/arm64/kernel/lr_unwind.c
@@ -5,11 +5,15 @@
 
 #define pr_fmt(fmt)	"lr_unwind: " fmt
 
+#include <linux/cleanup.h>
 #include <linux/init.h>
 #include <linux/kernel.h>
 #include <linux/kprobes.h>
 #include <linux/limits.h>
+#include <linux/module.h>
 #include <linux/printk.h>
+#include <linux/rcupdate.h>
+#include <linux/sort.h>
 
 #include <asm/lr_unwind.h>
 #include <asm/sections.h>
@@ -17,10 +21,20 @@
 static bool lr_unwind_init __ro_after_init;
 static struct lr_unwind_section kernel_lr_sec __ro_after_init;
 
+#ifdef CONFIG_MODULES
+static const char *objname(const struct lr_unwind_section *sec)
+{
+	if (sec == &kernel_lr_sec)
+		return "vmlinux";
+
+	return container_of(sec, struct module, arch.lr_unwind_sec)->name;
+}
+#else
 static const char *objname(const struct lr_unwind_section *sec)
 {
 	return "vmlinux";
 }
+#endif
 
 #define dbg_sec(sec, fmt, ...) \
 	pr_debug("(%s): " fmt, objname(sec), ##__VA_ARGS__)
@@ -55,10 +69,36 @@ static __always_inline bool is_kernel_ip_valid(unsigned long ip)
 	return false;
 }
 
+#ifdef CONFIG_MODULES
+static __always_inline bool within_mod_mem(unsigned long ip,
+					   const struct module_memory *mem)
+{
+	return ip - (unsigned long)mem->base < mem->size;
+}
+
+static __always_inline bool
+is_sec_module_ip_valid(const struct lr_unwind_section *sec, unsigned long ip)
+{
+	struct module *mod = container_of(sec, struct module, arch.lr_unwind_sec);
+
+	return within_mod_mem(ip, &mod->mem[MOD_TEXT]) ||
+	       within_mod_mem(ip, &mod->mem[MOD_INIT_TEXT]);
+}
+#else
+static __always_inline bool
+is_sec_module_ip_valid(const struct lr_unwind_section *sec, unsigned long ip)
+{
+	return false;
+}
+#endif
+
 static __always_inline bool is_sec_ip_valid(const struct lr_unwind_section *sec,
 					    unsigned long ip)
 {
-	return is_kernel_ip_valid(ip);
+	if (sec == &kernel_lr_sec)
+		return is_kernel_ip_valid(ip);
+
+	return is_sec_module_ip_valid(sec, ip);
 }
 
 static __always_inline int __find_fde(const struct lr_unwind_section *sec,
@@ -151,9 +191,31 @@ static __always_inline int __lr_unwind_find(const struct lr_unwind_section *sec,
 	return -EINVAL;
 }
 
+#ifdef CONFIG_MODULES
+static noinline int __lr_unwind_find_module(unsigned long ip,
+					    enum lr_unwind_mode *mode,
+					    unsigned int *sp_off)
+{
+	struct module *mod;
+
+	guard(preempt_notrace)();
+
+	mod = __module_text_address(ip);
+	/* Pairs with smp_store_release() in lr_unwind_module_init() */
+	if (mod && smp_load_acquire(&mod->arch.lr_unwind_init))
+		return __lr_unwind_find(&mod->arch.lr_unwind_sec,
+					ip, mode, sp_off);
+
+	return -EINVAL;
+}
+NOKPROBE_SYMBOL(__lr_unwind_find_module);
+#endif
+
 noinstr int lr_unwind_find(unsigned long ip, enum lr_unwind_mode *mode,
 			   unsigned int *sp_off)
 {
+	int ret = -EINVAL;
+
 	if (!mode || !sp_off)
 		return -EINVAL;
 
@@ -164,7 +226,16 @@ noinstr int lr_unwind_find(unsigned long ip, enum lr_unwind_mode *mode,
 		return __lr_unwind_find(&kernel_lr_sec, ip, mode, sp_off);
 	}
 
-	return -EINVAL;
+#ifdef CONFIG_MODULES
+	if (!rcu_is_watching())
+		return -EINVAL;
+
+	instrumentation_begin();
+	ret = __lr_unwind_find_module(ip, mode, sp_off);
+	instrumentation_end();
+#endif
+
+	return ret;
 }
 NOKPROBE_SYMBOL(lr_unwind_find);
 
@@ -272,3 +343,53 @@ void __init init_lr_unwind_table(void)
 	lr_unwind_init = true;
 	dbg_print_section(&kernel_lr_sec);
 }
+
+#ifdef CONFIG_MODULES
+
+static int lr_unwind_sort_cmp_fde(const void *a, const void *b)
+{
+	return cmp_int(lr_fde_start_addr(a), lr_fde_start_addr(b));
+}
+
+static void lr_unwind_sort_swap_fde(void *a, void *b, int size)
+{
+	struct lr_fde *x = a, *y = b, tmp = *x;
+	int delta = b - a;
+
+	x->start_offset = y->start_offset + delta;
+	x->range_desc = y->range_desc;
+	y->start_offset = tmp.start_offset - delta;
+	y->range_desc = tmp.range_desc;
+}
+
+void lr_unwind_module_init(struct module *mod, void *fdes, size_t fdes_size,
+			   const void *ranges, size_t ranges_size)
+{
+	struct lr_unwind_section *sec = &mod->arch.lr_unwind_sec;
+
+	if (!fdes_size || (fdes_size % sizeof(struct lr_fde)) ||
+	    (ranges_size % sizeof(lr_range_t))) {
+		pr_warn("invalid .lr_unwind section sizes in module %s\n",
+			mod->name);
+		return;
+	}
+
+	sec->fdes = fdes;
+	sec->num_fdes = fdes_size / sizeof(struct lr_fde);
+	sec->ranges = ranges;
+	sec->num_ranges = ranges_size / sizeof(lr_range_t);
+
+	sort(fdes, sec->num_fdes, sizeof(struct lr_fde),
+	     lr_unwind_sort_cmp_fde, lr_unwind_sort_swap_fde);
+
+	if (lr_unwind_validate_section(sec)) {
+		pr_warn("invalid .lr_unwind section in module %s\n", mod->name);
+		return;
+	}
+
+	/* Ensure table is initialized before lr_unwind_find() observes it */
+	smp_store_release(&mod->arch.lr_unwind_init, true);
+	dbg_print_section(sec);
+}
+
+#endif /* CONFIG_MODULES */
diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c
index 24adb581af0eb..685b4301df96e 100644
--- a/arch/arm64/kernel/module.c
+++ b/arch/arm64/kernel/module.c
@@ -21,6 +21,7 @@
 
 #include <asm/alternative.h>
 #include <asm/insn.h>
+#include <asm/lr_unwind.h>
 #include <asm/scs.h>
 #include <asm/sections.h>
 #include <asm/text-patching.h>
@@ -515,5 +516,15 @@ int module_finalize(const Elf_Ehdr *hdr,
 		}
 	}
 
+	s = find_section(hdr, sechdrs, ".lr_unwind_fde");
+	if (s) {
+		const Elf_Shdr *r = find_section(hdr, sechdrs,
+						 ".lr_unwind_ranges");
+
+		lr_unwind_module_init(me, (void *)s->sh_addr, s->sh_size,
+				      r ? (const void *)r->sh_addr : NULL,
+				      r ? r->sh_size : 0);
+	}
+
 	return module_init_ftrace_plt(hdr, sechdrs, me);
 }
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 01a37ec872b90..301393dbb9dd3 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -32,11 +32,17 @@ ifneq ($(WARN_ON_UNUSED_TRACEPOINTS),)
 cmd_check_tracepoint = $(objtree)/scripts/tracepoint-update --module $<;
 endif
 
+ifdef CONFIG_ARM64_LR_UNWIND
+cmd_gen_lr_unwind = ; $(objtree)/scripts/gen_lr_unwind $@ && \
+	$(OBJCOPY) --remove-section=.eh_frame --remove-section=.rela.eh_frame $@
+endif
+
 quiet_cmd_ld_ko_o = LD [M]  $@
       cmd_ld_ko_o =							\
 	$(LD) -r $(KBUILD_LDFLAGS)					\
 		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
-		-T $(objtree)/scripts/module.lds -o $@ $(filter %.o, $^)
+		-T $(objtree)/scripts/module.lds -o $@ $(filter %.o, $^) \
+	$(cmd_gen_lr_unwind)
 
 quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
@@ -47,7 +53,7 @@ quiet_cmd_btf_ko = BTF [M] $@
 	fi;
 
 # Re-generate module BTFs if either module's .ko or vmlinux changed
-%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
+%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(if $(CONFIG_ARM64_LR_UNWIND),$(objtree)/scripts/gen_lr_unwind) $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
 	+$(call if_changed,ld_ko_o)
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	+$(if $(newer-prereqs),$(call cmd,btf_ko))
diff --git a/scripts/gen_lr_unwind.c b/scripts/gen_lr_unwind.c
index 2058601ec6135..cc6b8790e00d8 100644
--- a/scripts/gen_lr_unwind.c
+++ b/scripts/gen_lr_unwind.c
@@ -797,12 +797,12 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
-	if (!ctx.eh_sec || !ctx.rela_eh_sec || !ctx.symtab_shndx) {
+	if (!ctx.eh_sec || !ctx.rela_eh_sec) {
 		ret = 0;
 		goto out;
 	}
 
-	if (!parse_eh_frame(&ctx) &&
+	if (ctx.symtab_shndx && !parse_eh_frame(&ctx) &&
 	    (!ctx.nr_fdes ||
 	     !emit_lr_unwind_sections(fname, &ctx, shnum, shstrndx)))
 		ret = 0;
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 10/12] arm64: entry: add unwind info for call_on_irq_stack()
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (8 preceding siblings ...)
  2026-11-17  4:35 ` [RFC PATCH 09/12] arm64/module, unwind: add .lr_unwind support for modules Dylan Hatch
@ 2026-11-17  4:35 ` Dylan Hatch
  2026-11-17  4:35 ` [RFC PATCH 11/12] arm64, crypto/lib: annotate leaf functions with CFI info Dylan Hatch
  2026-11-17  4:35 ` [RFC PATCH 12/12] unwind: arm64: use .lr_unwind to unwind interrupt frames Dylan Hatch
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:35 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

From: Weinan Liu <wnliu@google.com>

DWARF CFI (Call Frame Information) specifies how to recover the return
address and callee-saved registers at each PC in a given function.
Compilers are able to generate the CFI annotations when they compile
the code to assembly language. For handcrafted assembly, we need to
annotate them by hand.

Frame pointers alone are usually sufficient to recover stack frames
(without CFI), except at the exception boundary, where more information
is needed to determine if the LR is live.

Since an exception can be taken from call_on_irq_stack(), annotate it
with CFI. The actual entry assembly functions are left untouched, since
they are not expected to take exceptions themselves.

Signed-off-by: Weinan Liu <wnliu@google.com>
Suggested-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/kernel/entry.S | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 8d17f61d1b5b2..aae27ccc0f2c1 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -30,6 +30,13 @@
 #include <asm/asm-uaccess.h>
 #include <asm/unistd.h>
 
+#ifndef CONFIG_UNWIND_TABLES
+/*
+ * Do not generate .eh_frame unless CONFIG_UNWIND_TABLES is enabled.
+ */
+	.cfi_sections .debug_frame
+#endif
+
 	.macro	clear_gp_regs
 	.irp	n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29
 	mov	x\n, xzr
@@ -880,8 +887,11 @@ NOKPROBE(ret_from_fork)
  * 		          void (*func)(struct pt_regs *));
  *
  * Calls func(regs) using this CPU's irq stack and shadow irq stack.
+ * Open-code SYM_FUNC_START() so .cfi_startproc precedes the BTI landing pad.
  */
-SYM_FUNC_START(call_on_irq_stack)
+SYM_START(call_on_irq_stack, SYM_L_GLOBAL, SYM_A_ALIGN)
+	.cfi_startproc
+	bti	c
 	save_and_disable_daif x9
 #ifdef CONFIG_SHADOW_CALL_STACK
 	get_current_task x16
@@ -892,6 +902,9 @@ SYM_FUNC_START(call_on_irq_stack)
 	/* Create a frame record to save our LR and SP (implicit in FP) */
 	stp	x29, x30, [sp, #-16]!
 	mov	x29, sp
+	.cfi_def_cfa 29, 16
+	.cfi_offset 29, -16
+	.cfi_offset 30, -8
 
 	ldr_this_cpu x16, irq_stack_ptr, x17
 
@@ -907,9 +920,13 @@ SYM_FUNC_START(call_on_irq_stack)
 	 */
 	mov	sp, x29
 	ldp	x29, x30, [sp], #16
+	.cfi_restore 29
+	.cfi_restore 30
+	.cfi_def_cfa 31, 0
 	scs_load_current
 	restore_irq x9
 	ret
+	.cfi_endproc
 SYM_FUNC_END(call_on_irq_stack)
 NOKPROBE(call_on_irq_stack)
 
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 11/12] arm64, crypto/lib: annotate leaf functions with CFI info
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (9 preceding siblings ...)
  2026-11-17  4:35 ` [RFC PATCH 10/12] arm64: entry: add unwind info for call_on_irq_stack() Dylan Hatch
@ 2026-11-17  4:35 ` Dylan Hatch
  2026-11-17  4:35 ` [RFC PATCH 12/12] unwind: arm64: use .lr_unwind to unwind interrupt frames Dylan Hatch
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:35 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Several arm64 assembly functions are leaf functions (they do not call
other functions and do not modify x30). Add SYM_[TYPED_]LEAF_FUNC_*
macros in <asm/linkage.h> that emit .cfi_startproc / .cfi_endproc so
that .eh_frame entries are generated for these leaf functions and LR can
be recovered by the unwinder when an exception is taken from one of
them.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/crypto/aes-ce-ccm-core.S    | 12 +++----
 arch/arm64/crypto/aes-neonbs-core.S    | 16 +++++-----
 arch/arm64/crypto/ghash-ce-core.S      | 20 ++++++++----
 arch/arm64/crypto/sm4-ce-ccm-core.S    | 16 +++++-----
 arch/arm64/crypto/sm4-ce-cipher-core.S |  4 +--
 arch/arm64/crypto/sm4-ce-core.S        | 44 +++++++++++++-------------
 arch/arm64/crypto/sm4-ce-gcm-core.S    | 16 +++++-----
 arch/arm64/crypto/sm4-neon-core.S      | 12 +++----
 arch/arm64/include/asm/linkage.h       | 33 +++++++++++++++++++
 arch/arm64/kernel/entry.S              |  7 ----
 arch/arm64/lib/clear_user.S            |  4 +--
 arch/arm64/lib/copy_from_user.S        |  4 +--
 arch/arm64/lib/copy_page.S             |  4 +--
 arch/arm64/lib/copy_to_user.S          |  4 +--
 arch/arm64/lib/memchr.S                |  4 +--
 arch/arm64/lib/memcmp.S                |  4 +--
 arch/arm64/lib/memcpy.S                |  8 ++---
 arch/arm64/lib/memset.S                |  8 ++---
 arch/arm64/lib/mte.S                   | 28 ++++++++--------
 arch/arm64/lib/strchr.S                |  4 +--
 arch/arm64/lib/strcmp.S                |  4 +--
 arch/arm64/lib/strlen.S                |  4 +--
 arch/arm64/lib/strncmp.S               |  4 +--
 arch/arm64/lib/strnlen.S               |  4 +--
 arch/arm64/lib/strrchr.S               |  4 +--
 arch/arm64/lib/tishift.S               | 12 +++----
 26 files changed, 159 insertions(+), 125 deletions(-)

diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S
index c2e8f500da144..b6a074a7fe128 100644
--- a/arch/arm64/crypto/aes-ce-ccm-core.S
+++ b/arch/arm64/crypto/aes-ce-ccm-core.S
@@ -80,7 +80,7 @@
 	ret
 	.endm
 
-SYM_FUNC_START_LOCAL(ce_aes_ccm_crypt_tail)
+SYM_LEAF_FUNC_START_LOCAL(ce_aes_ccm_crypt_tail)
 	eor	v0.16b, v0.16b, v5.16b		/* final round mac */
 	eor	v1.16b, v1.16b, v5.16b		/* final round enc */
 
@@ -113,7 +113,7 @@ SYM_INNER_LABEL(ce_aes_ccm_final, SYM_L_LOCAL)
 	eor	v0.16b, v0.16b, v1.16b		/* en-/decrypt the mac */
 0:	st1	{v0.16b}, [x5]			/* store result */
 	ret
-SYM_FUNC_END(ce_aes_ccm_crypt_tail)
+SYM_LEAF_FUNC_END(ce_aes_ccm_crypt_tail)
 
 	/*
 	 * void ce_aes_ccm_encrypt(u8 out[], u8 const in[], u32 cbytes,
@@ -123,15 +123,15 @@ SYM_FUNC_END(ce_aes_ccm_crypt_tail)
 	 * 			   u8 const rk[], u32 rounds, u8 mac[],
 	 * 			   u8 ctr[], u8 const final_iv[]);
 	 */
-SYM_FUNC_START(ce_aes_ccm_encrypt)
+SYM_LEAF_FUNC_START(ce_aes_ccm_encrypt)
 	movi	v22.16b, #255
 	aes_ccm_do_crypt	1
-SYM_FUNC_END(ce_aes_ccm_encrypt)
+SYM_LEAF_FUNC_END(ce_aes_ccm_encrypt)
 
-SYM_FUNC_START(ce_aes_ccm_decrypt)
+SYM_LEAF_FUNC_START(ce_aes_ccm_decrypt)
 	movi	v22.16b, #0
 	aes_ccm_do_crypt	0
-SYM_FUNC_END(ce_aes_ccm_decrypt)
+SYM_LEAF_FUNC_END(ce_aes_ccm_decrypt)
 
 	.section ".rodata", "a"
 	.align	6
diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S
index ba1f981a5464d..5c5ec34d8f94f 100644
--- a/arch/arm64/crypto/aes-neonbs-core.S
+++ b/arch/arm64/crypto/aes-neonbs-core.S
@@ -381,7 +381,7 @@ ISRM0:	.octa		0x0306090c00070a0d01040b0e0205080f
 	/*
 	 * void aesbs_convert_key(u8 out[], u32 const rk[], int rounds)
 	 */
-SYM_FUNC_START(aesbs_convert_key)
+SYM_LEAF_FUNC_START(aesbs_convert_key)
 	ld1		{v7.4s}, [x1], #16		// load round 0 key
 	ld1		{v17.4s}, [x1], #16		// load round 1 key
 
@@ -426,10 +426,10 @@ SYM_FUNC_START(aesbs_convert_key)
 	eor		v17.16b, v17.16b, v7.16b
 	str		q17, [x0]
 	ret
-SYM_FUNC_END(aesbs_convert_key)
+SYM_LEAF_FUNC_END(aesbs_convert_key)
 
 	.align		4
-SYM_FUNC_START_LOCAL(aesbs_encrypt8)
+SYM_LEAF_FUNC_START_LOCAL(aesbs_encrypt8)
 	ldr		q9, [bskey], #16		// round 0 key
 	ldr		q8, M0SR
 	ldr		q24, SR
@@ -489,10 +489,10 @@ SYM_FUNC_START_LOCAL(aesbs_encrypt8)
 	eor		v2.16b, v2.16b, v12.16b
 	eor		v5.16b, v5.16b, v12.16b
 	ret
-SYM_FUNC_END(aesbs_encrypt8)
+SYM_LEAF_FUNC_END(aesbs_encrypt8)
 
 	.align		4
-SYM_FUNC_START_LOCAL(aesbs_decrypt8)
+SYM_LEAF_FUNC_START_LOCAL(aesbs_decrypt8)
 	lsl		x9, rounds, #7
 	add		bskey, bskey, x9
 
@@ -554,7 +554,7 @@ SYM_FUNC_START_LOCAL(aesbs_decrypt8)
 	eor		v3.16b, v3.16b, v12.16b
 	eor		v5.16b, v5.16b, v12.16b
 	ret
-SYM_FUNC_END(aesbs_decrypt8)
+SYM_LEAF_FUNC_END(aesbs_decrypt8)
 
 	/*
 	 * aesbs_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
@@ -735,7 +735,7 @@ SYM_FUNC_END(aesbs_cbc_decrypt)
 	 * aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
 	 *		     int blocks, u8 iv[])
 	 */
-SYM_FUNC_START_LOCAL(__xts_crypt8)
+SYM_LEAF_FUNC_START_LOCAL(__xts_crypt8)
 	movi		v18.2s, #0x1
 	movi		v19.2s, #0x87
 	uzp1		v18.4s, v18.4s, v19.4s
@@ -766,7 +766,7 @@ SYM_FUNC_START_LOCAL(__xts_crypt8)
 	mov		bskey, x2
 	mov		rounds, x3
 	br		x16
-SYM_FUNC_END(__xts_crypt8)
+SYM_LEAF_FUNC_END(__xts_crypt8)
 
 	.macro		__xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7
 	frame_push	0, 32
diff --git a/arch/arm64/crypto/ghash-ce-core.S b/arch/arm64/crypto/ghash-ce-core.S
index ba350d57fc766..3627d68ed1607 100644
--- a/arch/arm64/crypto/ghash-ce-core.S
+++ b/arch/arm64/crypto/ghash-ce-core.S
@@ -66,7 +66,7 @@
 	 * void pmull_ghash_update_p64(int blocks, u64 dg[], const char *src,
 	 *			       u64 const h[4][2], const char *head)
 	 */
-SYM_FUNC_START(pmull_ghash_update_p64)
+SYM_LEAF_FUNC_START(pmull_ghash_update_p64)
 	ld1		{SHASH.2d}, [x3]
 	ld1		{XL.2d}, [x1]
 
@@ -173,7 +173,7 @@ SYM_FUNC_START(pmull_ghash_update_p64)
 
 5:	st1		{XL.2d}, [x1]
 	ret
-SYM_FUNC_END(pmull_ghash_update_p64)
+SYM_LEAF_FUNC_END(pmull_ghash_update_p64)
 
 	KS0		.req	v8
 	KS1		.req	v9
@@ -431,7 +431,7 @@ SYM_FUNC_START(pmull_gcm_decrypt)
 	pmull_gcm_do_crypt	0
 SYM_FUNC_END(pmull_gcm_decrypt)
 
-SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
+SYM_LEAF_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
 	movi		MASK.16b, #0xe1
 	shl		MASK.2d, MASK.2d, #57
 
@@ -443,7 +443,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
 	ext		XL.16b, XL.16b, XL.16b, #8
 
 	tbz		w9, #2, 0f			// <4 blocks?
+	.cfi_endproc
 	.subsection	1
+	.cfi_startproc
 0:	movi		XH2.16b, #0
 	movi		XM2.16b, #0
 	movi		XL2.16b, #0
@@ -462,7 +464,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
 2:	eor		TT4.16b, TT4.16b, XL.16b
 	ext		IN1.16b, TT4.16b, TT4.16b, #8
 	b		.Lgh1
+	.cfi_endproc
 	.previous
+	.cfi_startproc
 
 	eor		T1.16b, T1.16b, XL.16b
 	ext		IN1.16b, T1.16b, T1.16b, #8
@@ -512,9 +516,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
 	eor		XL.16b, XL.16b, T2.16b
 
 	ret
-SYM_FUNC_END(pmull_gcm_ghash_4x)
+SYM_LEAF_FUNC_END(pmull_gcm_ghash_4x)
 
-SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
+SYM_LEAF_FUNC_START_LOCAL(pmull_gcm_enc_4x)
 	ld1		{KS0.16b}, [x5]			// load upper counter
 	sub		w10, w8, #4
 	sub		w11, w8, #3
@@ -539,7 +543,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
 	.endr
 
 	tbnz		x7, #2, .Lnot128
+	.cfi_endproc
 	.subsection	1
+	.cfi_startproc
 .Lnot128:
 	ld1		{K8.4s-K9.4s}, [x10], #32
 	.irp		key, K6, K7
@@ -551,7 +557,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
 	.endr
 	tbz		x7, #1, .Lout192
 	b		.Lout256
+	.cfi_endproc
 	.previous
+	.cfi_startproc
 
 .Lout256:
 	.irp		key, K6, K7
@@ -577,7 +585,7 @@ SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
 	eor		INP3.16b, INP3.16b, KS3.16b
 
 	ret
-SYM_FUNC_END(pmull_gcm_enc_4x)
+SYM_LEAF_FUNC_END(pmull_gcm_enc_4x)
 
 	.section	".rodata", "a"
 	.align		6
diff --git a/arch/arm64/crypto/sm4-ce-ccm-core.S b/arch/arm64/crypto/sm4-ce-ccm-core.S
index fa85856f33cee..20a8853609e04 100644
--- a/arch/arm64/crypto/sm4-ce-ccm-core.S
+++ b/arch/arm64/crypto/sm4-ce-ccm-core.S
@@ -37,7 +37,7 @@
 
 
 .align 3
-SYM_FUNC_START(sm4_ce_cbcmac_update)
+SYM_LEAF_FUNC_START(sm4_ce_cbcmac_update)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: mac
@@ -81,10 +81,10 @@ SYM_FUNC_START(sm4_ce_cbcmac_update)
 .Lcbcmac_end:
 	st1		{RMAC.16b}, [x1]
 	ret
-SYM_FUNC_END(sm4_ce_cbcmac_update)
+SYM_LEAF_FUNC_END(sm4_ce_cbcmac_update)
 
 .align 3
-SYM_FUNC_START(sm4_ce_ccm_final)
+SYM_LEAF_FUNC_START(sm4_ce_ccm_final)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: ctr0 (big endian, 128 bit)
@@ -102,10 +102,10 @@ SYM_FUNC_START(sm4_ce_ccm_final)
 	st1		{RMAC.16b}, [x2]
 
 	ret
-SYM_FUNC_END(sm4_ce_ccm_final)
+SYM_LEAF_FUNC_END(sm4_ce_ccm_final)
 
 .align 3
-SYM_TYPED_FUNC_START(sm4_ce_ccm_enc)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_ccm_enc)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -214,10 +214,10 @@ SYM_TYPED_FUNC_START(sm4_ce_ccm_enc)
 
 .Lccm_enc_ret:
 	ret
-SYM_FUNC_END(sm4_ce_ccm_enc)
+SYM_LEAF_FUNC_END(sm4_ce_ccm_enc)
 
 .align 3
-SYM_TYPED_FUNC_START(sm4_ce_ccm_dec)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_ccm_dec)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -326,4 +326,4 @@ SYM_TYPED_FUNC_START(sm4_ce_ccm_dec)
 
 .Lccm_dec_ret:
 	ret
-SYM_FUNC_END(sm4_ce_ccm_dec)
+SYM_LEAF_FUNC_END(sm4_ce_ccm_dec)
diff --git a/arch/arm64/crypto/sm4-ce-cipher-core.S b/arch/arm64/crypto/sm4-ce-cipher-core.S
index 1682d727e0970..e1c90f879a831 100644
--- a/arch/arm64/crypto/sm4-ce-cipher-core.S
+++ b/arch/arm64/crypto/sm4-ce-cipher-core.S
@@ -15,7 +15,7 @@
 	 * void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in);
 	 */
 	.text
-SYM_FUNC_START(sm4_ce_do_crypt)
+SYM_LEAF_FUNC_START(sm4_ce_do_crypt)
 	ld1		{v8.4s}, [x2]
 	ld1		{v0.4s-v3.4s}, [x0], #64
 	rev32		v8.16b, v8.16b
@@ -33,4 +33,4 @@ SYM_FUNC_START(sm4_ce_do_crypt)
 	rev32		v8.16b, v8.16b
 	st1		{v8.4s}, [x1]
 	ret
-SYM_FUNC_END(sm4_ce_do_crypt)
+SYM_LEAF_FUNC_END(sm4_ce_do_crypt)
diff --git a/arch/arm64/crypto/sm4-ce-core.S b/arch/arm64/crypto/sm4-ce-core.S
index 1f3625c2c67e4..6af5b10859b88 100644
--- a/arch/arm64/crypto/sm4-ce-core.S
+++ b/arch/arm64/crypto/sm4-ce-core.S
@@ -40,7 +40,7 @@
 
 
 .align 3
-SYM_FUNC_START(sm4_ce_expand_key)
+SYM_LEAF_FUNC_START(sm4_ce_expand_key)
 	/* input:
 	 *   x0: 128-bit key
 	 *   x1: rkey_enc
@@ -86,10 +86,10 @@ SYM_FUNC_START(sm4_ce_expand_key)
 	st1		{v20.16b-v23.16b}, [x2]
 
 	ret;
-SYM_FUNC_END(sm4_ce_expand_key)
+SYM_LEAF_FUNC_END(sm4_ce_expand_key)
 
 .align 3
-SYM_FUNC_START(sm4_ce_crypt_block)
+SYM_LEAF_FUNC_START(sm4_ce_crypt_block)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -102,10 +102,10 @@ SYM_FUNC_START(sm4_ce_crypt_block)
 	st1		{v0.16b}, [x1];
 
 	ret;
-SYM_FUNC_END(sm4_ce_crypt_block)
+SYM_LEAF_FUNC_END(sm4_ce_crypt_block)
 
 .align 3
-SYM_FUNC_START(sm4_ce_crypt)
+SYM_LEAF_FUNC_START(sm4_ce_crypt)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -153,10 +153,10 @@ SYM_FUNC_START(sm4_ce_crypt)
 
 .Lcrypt_end:
 	ret;
-SYM_FUNC_END(sm4_ce_crypt)
+SYM_LEAF_FUNC_END(sm4_ce_crypt)
 
 .align 3
-SYM_FUNC_START(sm4_ce_cbc_enc)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_enc)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -208,10 +208,10 @@ SYM_FUNC_START(sm4_ce_cbc_enc)
 	st1		{RIV.16b}, [x3]
 
 	ret
-SYM_FUNC_END(sm4_ce_cbc_enc)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_enc)
 
 .align 3
-SYM_FUNC_START(sm4_ce_cbc_dec)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_dec)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -306,10 +306,10 @@ SYM_FUNC_START(sm4_ce_cbc_dec)
 	st1		{RIV.16b}, [x3]
 
 	ret
-SYM_FUNC_END(sm4_ce_cbc_dec)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_dec)
 
 .align 3
-SYM_FUNC_START(sm4_ce_cbc_cts_enc)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_cts_enc)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -354,10 +354,10 @@ SYM_FUNC_START(sm4_ce_cbc_cts_enc)
 	st1		{v1.16b}, [x1]
 
 	ret
-SYM_FUNC_END(sm4_ce_cbc_cts_enc)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_cts_enc)
 
 .align 3
-SYM_FUNC_START(sm4_ce_cbc_cts_dec)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_cts_dec)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -400,10 +400,10 @@ SYM_FUNC_START(sm4_ce_cbc_cts_dec)
 	st1		{v0.16b}, [x1]
 
 	ret
-SYM_FUNC_END(sm4_ce_cbc_cts_dec)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_cts_dec)
 
 .align 3
-SYM_FUNC_START(sm4_ce_ctr_enc)
+SYM_LEAF_FUNC_START(sm4_ce_ctr_enc)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -506,7 +506,7 @@ SYM_FUNC_START(sm4_ce_ctr_enc)
 	stp		x7, x8, [x3]
 
 	ret
-SYM_FUNC_END(sm4_ce_ctr_enc)
+SYM_LEAF_FUNC_END(sm4_ce_ctr_enc)
 
 
 #define tweak_next(vt, vin, RTMP)					\
@@ -517,7 +517,7 @@ SYM_FUNC_END(sm4_ce_ctr_enc)
 		eor		vt.16b, vt.16b, RTMP.16b;
 
 .align 3
-SYM_FUNC_START(sm4_ce_xts_enc)
+SYM_LEAF_FUNC_START(sm4_ce_xts_enc)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -681,10 +681,10 @@ SYM_FUNC_START(sm4_ce_xts_enc)
 
 .Lxts_enc_ret:
 	ret
-SYM_FUNC_END(sm4_ce_xts_enc)
+SYM_LEAF_FUNC_END(sm4_ce_xts_enc)
 
 .align 3
-SYM_FUNC_START(sm4_ce_xts_dec)
+SYM_LEAF_FUNC_START(sm4_ce_xts_dec)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -848,10 +848,10 @@ SYM_FUNC_START(sm4_ce_xts_dec)
 
 .Lxts_dec_ret:
 	ret
-SYM_FUNC_END(sm4_ce_xts_dec)
+SYM_LEAF_FUNC_END(sm4_ce_xts_dec)
 
 .align 3
-SYM_FUNC_START(sm4_ce_mac_update)
+SYM_LEAF_FUNC_START(sm4_ce_mac_update)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: digest
@@ -917,7 +917,7 @@ SYM_FUNC_START(sm4_ce_mac_update)
 .Lmac_ret:
 	st1		{RMAC.16b}, [x1]
 	ret
-SYM_FUNC_END(sm4_ce_mac_update)
+SYM_LEAF_FUNC_END(sm4_ce_mac_update)
 
 
 	.section	".rodata", "a"
diff --git a/arch/arm64/crypto/sm4-ce-gcm-core.S b/arch/arm64/crypto/sm4-ce-gcm-core.S
index 347f25d757279..dac6db8160f28 100644
--- a/arch/arm64/crypto/sm4-ce-gcm-core.S
+++ b/arch/arm64/crypto/sm4-ce-gcm-core.S
@@ -259,7 +259,7 @@
 #define	RH4	v19
 
 .align 3
-SYM_FUNC_START(sm4_ce_pmull_ghash_setup)
+SYM_LEAF_FUNC_START(sm4_ce_pmull_ghash_setup)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: ghash table
@@ -293,10 +293,10 @@ SYM_FUNC_START(sm4_ce_pmull_ghash_setup)
 	st1		{RH1.16b-RH4.16b}, [x1]
 
 	ret
-SYM_FUNC_END(sm4_ce_pmull_ghash_setup)
+SYM_LEAF_FUNC_END(sm4_ce_pmull_ghash_setup)
 
 .align 3
-SYM_FUNC_START(pmull_ghash_update)
+SYM_LEAF_FUNC_START(pmull_ghash_update)
 	/* input:
 	 *   x0: ghash table
 	 *   x1: ghash result
@@ -368,10 +368,10 @@ SYM_FUNC_START(pmull_ghash_update)
 	st1		{RHASH.2d}, [x1]
 
 	ret
-SYM_FUNC_END(pmull_ghash_update)
+SYM_LEAF_FUNC_END(pmull_ghash_update)
 
 .align 3
-SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_enc)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_pmull_gcm_enc)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -534,7 +534,7 @@ SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_enc)
 	st1		{RHASH.2d}, [x5]
 
 	ret
-SYM_FUNC_END(sm4_ce_pmull_gcm_enc)
+SYM_LEAF_FUNC_END(sm4_ce_pmull_gcm_enc)
 
 #undef	RR1
 #undef	RR3
@@ -582,7 +582,7 @@ SYM_FUNC_END(sm4_ce_pmull_gcm_enc)
 #define	RH3	v20
 
 .align 3
-SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_dec)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_pmull_gcm_dec)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -726,7 +726,7 @@ SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_dec)
 	st1		{RHASH.2d}, [x5]
 
 	ret
-SYM_FUNC_END(sm4_ce_pmull_gcm_dec)
+SYM_LEAF_FUNC_END(sm4_ce_pmull_gcm_dec)
 
 	.section	".rodata", "a"
 	.align 4
diff --git a/arch/arm64/crypto/sm4-neon-core.S b/arch/arm64/crypto/sm4-neon-core.S
index 734dc7193610c..d1fe37fce13ad 100644
--- a/arch/arm64/crypto/sm4-neon-core.S
+++ b/arch/arm64/crypto/sm4-neon-core.S
@@ -257,7 +257,7 @@
 
 
 .align 3
-SYM_FUNC_START(sm4_neon_crypt)
+SYM_LEAF_FUNC_START(sm4_neon_crypt)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -318,10 +318,10 @@ SYM_FUNC_START(sm4_neon_crypt)
 
 .Lcrypt_end:
 	ret
-SYM_FUNC_END(sm4_neon_crypt)
+SYM_LEAF_FUNC_END(sm4_neon_crypt)
 
 .align 3
-SYM_FUNC_START(sm4_neon_cbc_dec)
+SYM_LEAF_FUNC_START(sm4_neon_cbc_dec)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -435,10 +435,10 @@ SYM_FUNC_START(sm4_neon_cbc_dec)
 	st1		{RIV.16b}, [x3]
 
 	ret
-SYM_FUNC_END(sm4_neon_cbc_dec)
+SYM_LEAF_FUNC_END(sm4_neon_cbc_dec)
 
 .align 3
-SYM_FUNC_START(sm4_neon_ctr_crypt)
+SYM_LEAF_FUNC_START(sm4_neon_ctr_crypt)
 	/* input:
 	 *   x0: round key array, CTX
 	 *   x1: dst
@@ -563,4 +563,4 @@ SYM_FUNC_START(sm4_neon_ctr_crypt)
 	stp		x7, x8, [x3]
 
 	ret
-SYM_FUNC_END(sm4_neon_ctr_crypt)
+SYM_LEAF_FUNC_END(sm4_neon_ctr_crypt)
diff --git a/arch/arm64/include/asm/linkage.h b/arch/arm64/include/asm/linkage.h
index d1f7a16729d25..a29daf3b1ffc2 100644
--- a/arch/arm64/include/asm/linkage.h
+++ b/arch/arm64/include/asm/linkage.h
@@ -3,6 +3,13 @@
 
 #ifdef __ASSEMBLER__
 #include <asm/assembler.h>
+
+#if !defined(CONFIG_UNWIND_TABLES) && !defined(BUILD_VDSO)
+/*
+ * Do not generate .eh_frame unless CONFIG_UNWIND_TABLES is enabled.
+ */
+	.cfi_sections .debug_frame
+#endif
 #endif
 
 #define __ALIGN		.balign CONFIG_FUNCTION_ALIGNMENT
@@ -43,6 +50,32 @@
 	SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)	\
 	bti c ;
 
+/*
+ * SYM_[TYPED_]LEAF_FUNC_[START|END] macros add minimal CFI directives
+ * allowing unwind data to be generated for functions which do not modify the
+ * LR (x30). Unwind data will not be correct if these macros are used on
+ * non-leaf functions, as additional CFI directives would be necessary in such
+ * cases.
+ */
+#define SYM_LEAF_FUNC_START(name)			\
+	SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)	\
+	.cfi_startproc ;				\
+	bti c ;
+
+#define SYM_LEAF_FUNC_END(name)				\
+	.cfi_endproc ;					\
+	SYM_FUNC_END(name)
+
+#define SYM_LEAF_FUNC_START_LOCAL(name)			\
+	SYM_START(name, SYM_L_LOCAL, SYM_A_ALIGN)	\
+	.cfi_startproc ;				\
+	bti c ;
+
+#define SYM_TYPED_LEAF_FUNC_START(name)				\
+	SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)	\
+	.cfi_startproc ;					\
+	bti c ;
+
 #define _THIS_IP_ ({ unsigned long __ip; asm volatile("adr %0, ." : "=r" (__ip)); __ip; })
 
 #define __bss_pgtbl __section(".bss..pgtbl") __aligned(PAGE_SIZE)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index aae27ccc0f2c1..aec6363ba9b95 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -30,13 +30,6 @@
 #include <asm/asm-uaccess.h>
 #include <asm/unistd.h>
 
-#ifndef CONFIG_UNWIND_TABLES
-/*
- * Do not generate .eh_frame unless CONFIG_UNWIND_TABLES is enabled.
- */
-	.cfi_sections .debug_frame
-#endif
-
 	.macro	clear_gp_regs
 	.irp	n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29
 	mov	x\n, xzr
diff --git a/arch/arm64/lib/clear_user.S b/arch/arm64/lib/clear_user.S
index de9a303b6ad0e..cf07c010ca927 100644
--- a/arch/arm64/lib/clear_user.S
+++ b/arch/arm64/lib/clear_user.S
@@ -17,7 +17,7 @@
  * Alignment fixed up by hardware.
  */
 
-SYM_FUNC_START(__arch_clear_user)
+SYM_LEAF_FUNC_START(__arch_clear_user)
 	add	x2, x0, x1
 
 #ifdef CONFIG_AS_HAS_MOPS
@@ -68,5 +68,5 @@ USER(7f, sttrb	wzr, [x2, #-1])
 8:	add	x0, x0, #4	// ...or the second word of the 4-7 byte case
 9:	sub	x0, x2, x0
 	ret
-SYM_FUNC_END(__arch_clear_user)
+SYM_LEAF_FUNC_END(__arch_clear_user)
 EXPORT_SYMBOL(__arch_clear_user)
diff --git a/arch/arm64/lib/copy_from_user.S b/arch/arm64/lib/copy_from_user.S
index 400057d607ecd..e9bb9c2dd8e12 100644
--- a/arch/arm64/lib/copy_from_user.S
+++ b/arch/arm64/lib/copy_from_user.S
@@ -61,7 +61,7 @@
 
 end	.req	x5
 srcin	.req	x15
-SYM_FUNC_START(__arch_copy_from_user)
+SYM_LEAF_FUNC_START(__arch_copy_from_user)
 	add	end, x0, x2
 	mov	srcin, x1
 #include "copy_template.S"
@@ -79,5 +79,5 @@ USER(9998f, ldtrb tmp1w, [srcin])
 	strb	tmp1w, [dst], #1
 9998:	sub	x0, end, dst			// bytes not copied
 	ret
-SYM_FUNC_END(__arch_copy_from_user)
+SYM_LEAF_FUNC_END(__arch_copy_from_user)
 EXPORT_SYMBOL(__arch_copy_from_user)
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
index e6374e7e55110..b6048d6483064 100644
--- a/arch/arm64/lib/copy_page.S
+++ b/arch/arm64/lib/copy_page.S
@@ -17,7 +17,7 @@
  *	x0 - dest
  *	x1 - src
  */
-SYM_FUNC_START(__pi_copy_page)
+SYM_LEAF_FUNC_START(__pi_copy_page)
 #ifdef CONFIG_AS_HAS_MOPS
 	.arch_extension mops
 alternative_if_not ARM64_HAS_MOPS
@@ -77,6 +77,6 @@ alternative_else_nop_endif
 	stnp	x16, x17, [x0, #112 - 256]
 
 	ret
-SYM_FUNC_END(__pi_copy_page)
+SYM_LEAF_FUNC_END(__pi_copy_page)
 SYM_FUNC_ALIAS(copy_page, __pi_copy_page)
 EXPORT_SYMBOL(copy_page)
diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/copy_to_user.S
index 819f2e3fc7a93..aea10a5b3cbac 100644
--- a/arch/arm64/lib/copy_to_user.S
+++ b/arch/arm64/lib/copy_to_user.S
@@ -60,7 +60,7 @@
 
 end	.req	x5
 srcin	.req	x15
-SYM_FUNC_START(__arch_copy_to_user)
+SYM_LEAF_FUNC_START(__arch_copy_to_user)
 	add	end, x0, x2
 	mov	srcin, x1
 #include "copy_template.S"
@@ -79,5 +79,5 @@ USER(9998f, sttrb tmp1w, [dst])
 	add	dst, dst, #1
 9998:	sub	x0, end, dst			// bytes not copied
 	ret
-SYM_FUNC_END(__arch_copy_to_user)
+SYM_LEAF_FUNC_END(__arch_copy_to_user)
 EXPORT_SYMBOL(__arch_copy_to_user)
diff --git a/arch/arm64/lib/memchr.S b/arch/arm64/lib/memchr.S
index b14b27cb84195..fd3760b688b19 100644
--- a/arch/arm64/lib/memchr.S
+++ b/arch/arm64/lib/memchr.S
@@ -38,7 +38,7 @@
 
 	.p2align 4
 	nop
-SYM_FUNC_START(__pi_memchr)
+SYM_LEAF_FUNC_START(__pi_memchr)
 	and	chrin, chrin, #0xff
 	lsr	wordcnt, cntin, #3
 	cbz	wordcnt, L(byte_loop)
@@ -71,6 +71,6 @@ L(found_word):
 L(not_found):
 	mov	result, #0
 	ret
-SYM_FUNC_END(__pi_memchr)
+SYM_LEAF_FUNC_END(__pi_memchr)
 SYM_FUNC_ALIAS_WEAK(memchr, __pi_memchr)
 EXPORT_SYMBOL_NOKASAN(memchr)
diff --git a/arch/arm64/lib/memcmp.S b/arch/arm64/lib/memcmp.S
index 1f528ce046e49..29bfd2dba22d2 100644
--- a/arch/arm64/lib/memcmp.S
+++ b/arch/arm64/lib/memcmp.S
@@ -32,7 +32,7 @@
 #define tmp1		x7
 #define tmp2		x8
 
-SYM_FUNC_START(__pi_memcmp)
+SYM_LEAF_FUNC_START(__pi_memcmp)
 	subs	limit, limit, 8
 	b.lo	L(less8)
 
@@ -132,6 +132,6 @@ L(byte_loop):
 	b.eq	L(byte_loop)
 	sub	result, data1w, data2w
 	ret
-SYM_FUNC_END(__pi_memcmp)
+SYM_LEAF_FUNC_END(__pi_memcmp)
 SYM_FUNC_ALIAS_WEAK(memcmp, __pi_memcmp)
 EXPORT_SYMBOL_NOKASAN(memcmp)
diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S
index 9b99106fb95f1..90caf402ea7d9 100644
--- a/arch/arm64/lib/memcpy.S
+++ b/arch/arm64/lib/memcpy.S
@@ -57,7 +57,7 @@
    The loop tail is handled by always copying 64 bytes from the end.
 */
 
-SYM_FUNC_START_LOCAL(__pi_memcpy_generic)
+SYM_LEAF_FUNC_START_LOCAL(__pi_memcpy_generic)
 	add	srcend, src, count
 	add	dstend, dstin, count
 	cmp	count, 128
@@ -238,11 +238,11 @@ L(copy64_from_start):
 	stp	B_l, B_h, [dstin, 16]
 	stp	C_l, C_h, [dstin]
 	ret
-SYM_FUNC_END(__pi_memcpy_generic)
+SYM_LEAF_FUNC_END(__pi_memcpy_generic)
 
 #ifdef CONFIG_AS_HAS_MOPS
 	.arch_extension mops
-SYM_FUNC_START(__pi_memcpy)
+SYM_LEAF_FUNC_START(__pi_memcpy)
 alternative_if_not ARM64_HAS_MOPS
 	b	__pi_memcpy_generic
 alternative_else_nop_endif
@@ -252,7 +252,7 @@ alternative_else_nop_endif
 	cpym	[dst]!, [src]!, count!
 	cpye	[dst]!, [src]!, count!
 	ret
-SYM_FUNC_END(__pi_memcpy)
+SYM_LEAF_FUNC_END(__pi_memcpy)
 #else
 SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic)
 #endif
diff --git a/arch/arm64/lib/memset.S b/arch/arm64/lib/memset.S
index 97157da65ec6b..8ee307f5891b4 100644
--- a/arch/arm64/lib/memset.S
+++ b/arch/arm64/lib/memset.S
@@ -43,7 +43,7 @@ dst		.req	x8
 tmp3w		.req	w9
 tmp3		.req	x9
 
-SYM_FUNC_START_LOCAL(__pi_memset_generic)
+SYM_LEAF_FUNC_START_LOCAL(__pi_memset_generic)
 	mov	dst, dstin	/* Preserve return value.  */
 	and	A_lw, val, #255
 	orr	A_lw, A_lw, A_lw, lsl #8
@@ -202,11 +202,11 @@ SYM_FUNC_START_LOCAL(__pi_memset_generic)
 	ands	count, count, zva_bits_x
 	b.ne	.Ltail_maybe_long
 	ret
-SYM_FUNC_END(__pi_memset_generic)
+SYM_LEAF_FUNC_END(__pi_memset_generic)
 
 #ifdef CONFIG_AS_HAS_MOPS
 	.arch_extension mops
-SYM_FUNC_START(__pi_memset)
+SYM_LEAF_FUNC_START(__pi_memset)
 alternative_if_not ARM64_HAS_MOPS
 	b	__pi_memset_generic
 alternative_else_nop_endif
@@ -216,7 +216,7 @@ alternative_else_nop_endif
 	setm	[dst]!, count!, val_x
 	sete	[dst]!, count!, val_x
 	ret
-SYM_FUNC_END(__pi_memset)
+SYM_LEAF_FUNC_END(__pi_memset)
 #else
 SYM_FUNC_ALIAS(__pi_memset, __pi_memset_generic)
 #endif
diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
index 5018ac03b6bf3..442202cd02a32 100644
--- a/arch/arm64/lib/mte.S
+++ b/arch/arm64/lib/mte.S
@@ -27,14 +27,14 @@
  * Clear the tags in a page
  *   x0 - address of the page to be cleared
  */
-SYM_FUNC_START(mte_clear_page_tags)
+SYM_LEAF_FUNC_START(mte_clear_page_tags)
 	multitag_transfer_size x1, x2
 1:	stgm	xzr, [x0]
 	add	x0, x0, x1
 	tst	x0, #(PAGE_SIZE - 1)
 	b.ne	1b
 	ret
-SYM_FUNC_END(mte_clear_page_tags)
+SYM_LEAF_FUNC_END(mte_clear_page_tags)
 
 /*
  * Zero the page and tags at the same time
@@ -42,7 +42,7 @@ SYM_FUNC_END(mte_clear_page_tags)
  * Parameters:
  *	x0 - address to the beginning of the page
  */
-SYM_FUNC_START(mte_zero_clear_page_tags)
+SYM_LEAF_FUNC_START(mte_zero_clear_page_tags)
 	and	x0, x0, #(1 << MTE_TAG_SHIFT) - 1	// clear the tag
 	mrs	x1, dczid_el0
 	tbnz	x1, #4, 2f	// Branch if DC GZVA is prohibited
@@ -60,14 +60,14 @@ SYM_FUNC_START(mte_zero_clear_page_tags)
 	tst	x0, #(PAGE_SIZE - 1)
 	b.ne	2b
 	ret
-SYM_FUNC_END(mte_zero_clear_page_tags)
+SYM_LEAF_FUNC_END(mte_zero_clear_page_tags)
 
 /*
  * Copy the tags from the source page to the destination one
  *   x0 - address of the destination page
  *   x1 - address of the source page
  */
-SYM_FUNC_START(mte_copy_page_tags)
+SYM_LEAF_FUNC_START(mte_copy_page_tags)
 	mov	x2, x0
 	mov	x3, x1
 	multitag_transfer_size x5, x6
@@ -78,7 +78,7 @@ SYM_FUNC_START(mte_copy_page_tags)
 	tst	x2, #(PAGE_SIZE - 1)
 	b.ne	1b
 	ret
-SYM_FUNC_END(mte_copy_page_tags)
+SYM_LEAF_FUNC_END(mte_copy_page_tags)
 
 /*
  * Read tags from a user buffer (one tag per byte) and set the corresponding
@@ -89,7 +89,7 @@ SYM_FUNC_END(mte_copy_page_tags)
  * Returns:
  *   x0 - number of tags read/set
  */
-SYM_FUNC_START(mte_copy_tags_from_user)
+SYM_LEAF_FUNC_START(mte_copy_tags_from_user)
 	mov	x3, x1
 	cbz	x2, 2f
 1:
@@ -103,7 +103,7 @@ USER(2f, ldtrb	w4, [x1])
 	// exception handling and function return
 2:	sub	x0, x1, x3		// update the number of tags set
 	ret
-SYM_FUNC_END(mte_copy_tags_from_user)
+SYM_LEAF_FUNC_END(mte_copy_tags_from_user)
 
 /*
  * Get the tags from a kernel address range and write the tag values to the
@@ -114,7 +114,7 @@ SYM_FUNC_END(mte_copy_tags_from_user)
  * Returns:
  *   x0 - number of tags read/set
  */
-SYM_FUNC_START(mte_copy_tags_to_user)
+SYM_LEAF_FUNC_START(mte_copy_tags_to_user)
 	mov	x3, x0
 	cbz	x2, 2f
 1:
@@ -129,14 +129,14 @@ USER(2f, sttrb	w4, [x0])
 	// exception handling and function return
 2:	sub	x0, x0, x3		// update the number of tags copied
 	ret
-SYM_FUNC_END(mte_copy_tags_to_user)
+SYM_LEAF_FUNC_END(mte_copy_tags_to_user)
 
 /*
  * Save the tags in a page
  *   x0 - page address
  *   x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes
  */
-SYM_FUNC_START(mte_save_page_tags)
+SYM_LEAF_FUNC_START(mte_save_page_tags)
 	multitag_transfer_size x7, x5
 1:
 	mov	x2, #0
@@ -153,14 +153,14 @@ SYM_FUNC_START(mte_save_page_tags)
 	b.ne	1b
 
 	ret
-SYM_FUNC_END(mte_save_page_tags)
+SYM_LEAF_FUNC_END(mte_save_page_tags)
 
 /*
  * Restore the tags in a page
  *   x0 - page address
  *   x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes
  */
-SYM_FUNC_START(mte_restore_page_tags)
+SYM_LEAF_FUNC_START(mte_restore_page_tags)
 	multitag_transfer_size x7, x5
 1:
 	ldr	x2, [x1], #8
@@ -174,4 +174,4 @@ SYM_FUNC_START(mte_restore_page_tags)
 	b.ne	1b
 
 	ret
-SYM_FUNC_END(mte_restore_page_tags)
+SYM_LEAF_FUNC_END(mte_restore_page_tags)
diff --git a/arch/arm64/lib/strchr.S b/arch/arm64/lib/strchr.S
index 94ee67a6b212c..455582efd07ae 100644
--- a/arch/arm64/lib/strchr.S
+++ b/arch/arm64/lib/strchr.S
@@ -18,7 +18,7 @@
  * Returns:
  *	x0 - address of first occurrence of 'c' or 0
  */
-SYM_FUNC_START(__pi_strchr)
+SYM_LEAF_FUNC_START(__pi_strchr)
 	and	w1, w1, #0xff
 1:	ldrb	w2, [x0], #1
 	cmp	w2, w1
@@ -28,7 +28,7 @@ SYM_FUNC_START(__pi_strchr)
 	cmp	w2, w1
 	csel	x0, x0, xzr, eq
 	ret
-SYM_FUNC_END(__pi_strchr)
+SYM_LEAF_FUNC_END(__pi_strchr)
 
 SYM_FUNC_ALIAS_WEAK(strchr, __pi_strchr)
 EXPORT_SYMBOL_NOKASAN(strchr)
diff --git a/arch/arm64/lib/strcmp.S b/arch/arm64/lib/strcmp.S
index 2389b47dca227..93463ecdc0980 100644
--- a/arch/arm64/lib/strcmp.S
+++ b/arch/arm64/lib/strcmp.S
@@ -45,7 +45,7 @@
    NUL too in big-endian, byte-reverse the data before the NUL check.  */
 
 
-SYM_FUNC_START(__pi_strcmp)
+SYM_LEAF_FUNC_START(__pi_strcmp)
 	sub	off2, src2, src1
 	mov	zeroones, REP8_01
 	and	tmp, src1, 7
@@ -153,6 +153,6 @@ L(tail):
 L(done):
 	sub	result, data1, data2
 	ret
-SYM_FUNC_END(__pi_strcmp)
+SYM_LEAF_FUNC_END(__pi_strcmp)
 SYM_FUNC_ALIAS_WEAK(strcmp, __pi_strcmp)
 EXPORT_SYMBOL_NOKASAN(strcmp)
diff --git a/arch/arm64/lib/strlen.S b/arch/arm64/lib/strlen.S
index a8c49a3886f63..20997fccc02ac 100644
--- a/arch/arm64/lib/strlen.S
+++ b/arch/arm64/lib/strlen.S
@@ -79,7 +79,7 @@
 	   whether the first fetch, which may be misaligned, crosses a page
 	   boundary.  */
 
-SYM_FUNC_START(__pi_strlen)
+SYM_LEAF_FUNC_START(__pi_strlen)
 	and	tmp1, srcin, MIN_PAGE_SIZE - 1
 	mov	zeroones, REP8_01
 	cmp	tmp1, MIN_PAGE_SIZE - 16
@@ -182,6 +182,6 @@ L(page_cross):
 	csel	data1, data1, tmp4, eq
 	csel	data2, data2, tmp2, eq
 	b	L(page_cross_entry)
-SYM_FUNC_END(__pi_strlen)
+SYM_LEAF_FUNC_END(__pi_strlen)
 SYM_FUNC_ALIAS_WEAK(strlen, __pi_strlen)
 EXPORT_SYMBOL_NOKASAN(strlen)
diff --git a/arch/arm64/lib/strncmp.S b/arch/arm64/lib/strncmp.S
index e4fde8586089a..0689bd8e9c4e4 100644
--- a/arch/arm64/lib/strncmp.S
+++ b/arch/arm64/lib/strncmp.S
@@ -45,7 +45,7 @@
 #define offset		pos
 #define neg_offset	x15
 
-SYM_FUNC_START(__pi_strncmp)
+SYM_LEAF_FUNC_START(__pi_strncmp)
 	cbz	limit, L(ret0)
 	eor	tmp1, src1, src2
 	mov	zeroones, #REP8_01
@@ -224,6 +224,6 @@ L(misaligned_mid_loop):
 L(ret0):
 	mov	result, #0
 	ret
-SYM_FUNC_END(__pi_strncmp)
+SYM_LEAF_FUNC_END(__pi_strncmp)
 SYM_FUNC_ALIAS_WEAK(strncmp, __pi_strncmp)
 EXPORT_SYMBOL_NOKASAN(strncmp)
diff --git a/arch/arm64/lib/strnlen.S b/arch/arm64/lib/strnlen.S
index b20011d559f19..2a0171d35d1f5 100644
--- a/arch/arm64/lib/strnlen.S
+++ b/arch/arm64/lib/strnlen.S
@@ -47,7 +47,7 @@ limit_wd	.req	x14
 #define REP8_7f 0x7f7f7f7f7f7f7f7f
 #define REP8_80 0x8080808080808080
 
-SYM_FUNC_START(__pi_strnlen)
+SYM_LEAF_FUNC_START(__pi_strnlen)
 	cbz	limit, .Lhit_limit
 	mov	zeroones, #REP8_01
 	bic	src, srcin, #15
@@ -142,7 +142,7 @@ SYM_FUNC_START(__pi_strnlen)
 .Lhit_limit:
 	mov	len, limit
 	ret
-SYM_FUNC_END(__pi_strnlen)
+SYM_LEAF_FUNC_END(__pi_strnlen)
 
 SYM_FUNC_ALIAS_WEAK(strnlen, __pi_strnlen)
 EXPORT_SYMBOL_NOKASAN(strnlen)
diff --git a/arch/arm64/lib/strrchr.S b/arch/arm64/lib/strrchr.S
index a5123cf0ce125..7f4d6d6a698a9 100644
--- a/arch/arm64/lib/strrchr.S
+++ b/arch/arm64/lib/strrchr.S
@@ -18,7 +18,7 @@
  * Returns:
  *	x0 - address of last occurrence of 'c' or 0
  */
-SYM_FUNC_START(__pi_strrchr)
+SYM_LEAF_FUNC_START(__pi_strrchr)
 	mov	x3, #0
 	and	w1, w1, #0xff
 1:	ldrb	w2, [x0], #1
@@ -29,6 +29,6 @@ SYM_FUNC_START(__pi_strrchr)
 	b	1b
 2:	mov	x0, x3
 	ret
-SYM_FUNC_END(__pi_strrchr)
+SYM_LEAF_FUNC_END(__pi_strrchr)
 SYM_FUNC_ALIAS_WEAK(strrchr, __pi_strrchr)
 EXPORT_SYMBOL_NOKASAN(strrchr)
diff --git a/arch/arm64/lib/tishift.S b/arch/arm64/lib/tishift.S
index a88613834fb07..b12d7f6a60031 100644
--- a/arch/arm64/lib/tishift.S
+++ b/arch/arm64/lib/tishift.S
@@ -7,7 +7,7 @@
 
 #include <asm/assembler.h>
 
-SYM_FUNC_START(__ashlti3)
+SYM_LEAF_FUNC_START(__ashlti3)
 	cbz	x2, 1f
 	mov	x3, #64
 	sub	x3, x3, x2
@@ -26,10 +26,10 @@ SYM_FUNC_START(__ashlti3)
 	lsl	x1, x0, x1
 	mov	x0, x2
 	ret
-SYM_FUNC_END(__ashlti3)
+SYM_LEAF_FUNC_END(__ashlti3)
 EXPORT_SYMBOL(__ashlti3)
 
-SYM_FUNC_START(__ashrti3)
+SYM_LEAF_FUNC_START(__ashrti3)
 	cbz	x2, 1f
 	mov	x3, #64
 	sub	x3, x3, x2
@@ -48,10 +48,10 @@ SYM_FUNC_START(__ashrti3)
 	asr	x0, x1, x0
 	mov	x1, x2
 	ret
-SYM_FUNC_END(__ashrti3)
+SYM_LEAF_FUNC_END(__ashrti3)
 EXPORT_SYMBOL(__ashrti3)
 
-SYM_FUNC_START(__lshrti3)
+SYM_LEAF_FUNC_START(__lshrti3)
 	cbz	x2, 1f
 	mov	x3, #64
 	sub	x3, x3, x2
@@ -70,5 +70,5 @@ SYM_FUNC_START(__lshrti3)
 	lsr	x0, x1, x0
 	mov	x1, x2
 	ret
-SYM_FUNC_END(__lshrti3)
+SYM_LEAF_FUNC_END(__lshrti3)
 EXPORT_SYMBOL(__lshrti3)
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [RFC PATCH 12/12] unwind: arm64: use .lr_unwind to unwind interrupt frames
  2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
                   ` (10 preceding siblings ...)
  2026-11-17  4:35 ` [RFC PATCH 11/12] arm64, crypto/lib: annotate leaf functions with CFI info Dylan Hatch
@ 2026-11-17  4:35 ` Dylan Hatch
  11 siblings, 0 replies; 13+ messages in thread
From: Dylan Hatch @ 2026-11-17  4:35 UTC (permalink / raw)
  To: Catalin Marinas, Will Deacon, Mark Rutland
  Cc: Herbert Xu, David S . Miller, Nathan Chancellor, Nicolas Schier,
	Nick Desaulniers, Bill Wendling, Justin Stitt, Weinan Liu,
	Jens Remus, linux-arm-kernel, linux-kernel, linux-kbuild,
	linux-crypto, llvm, Josh Poimboeuf, Steven Rostedt,
	Ard Biesheuvel, Indu Bhagat, Prasanna Kumar T S M, live-patching,
	Dylan Hatch

Add kunwind_next_regs_lr_unwind() to unwind across pt_regs exception
boundaries using .lr_unwind mode and SP-offset metadata (live x30, [x29]
frame record, or [sp + sp_off] frame record), falling back to
frame-pointer unwinding and marking the stacktrace unreliable on
failure.

Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
 arch/arm64/kernel/stacktrace.c | 85 ++++++++++++++++++++++++++++++----
 1 file changed, 75 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
index 3ebcf8c53fb04..842555a4a3fe2 100644
--- a/arch/arm64/kernel/stacktrace.c
+++ b/arch/arm64/kernel/stacktrace.c
@@ -10,6 +10,7 @@
 #include <linux/filter.h>
 #include <linux/ftrace.h>
 #include <linux/kprobes.h>
+#include <linux/overflow.h>
 #include <linux/sched.h>
 #include <linux/sched/debug.h>
 #include <linux/sched/task_stack.h>
@@ -17,6 +18,7 @@
 
 #include <asm/efi.h>
 #include <asm/irq.h>
+#include <asm/lr_unwind.h>
 #include <asm/stack_pointer.h>
 #include <asm/stacktrace.h>
 
@@ -26,6 +28,7 @@ enum kunwind_source {
 	KUNWIND_SOURCE_CALLER,
 	KUNWIND_SOURCE_TASK,
 	KUNWIND_SOURCE_REGS_PC,
+	KUNWIND_SOURCE_REGS_LR,
 };
 
 union unwind_flags {
@@ -45,6 +48,7 @@ union unwind_flags {
  * @kr_cur:      When KRETPROBES is selected, holds the kretprobe instance
  *               associated with the most recently encountered replacement lr
  *               value.
+ * @unreliable:  Stacktrace is unreliable.
  */
 struct kunwind_state {
 	struct unwind_state common;
@@ -56,6 +60,7 @@ struct kunwind_state {
 	enum kunwind_source source;
 	union unwind_flags flags;
 	struct pt_regs *regs;
+	bool unreliable;
 };
 
 static __always_inline void
@@ -67,6 +72,7 @@ kunwind_init(struct kunwind_state *state,
 	state->source = KUNWIND_SOURCE_UNKNOWN;
 	state->flags.all = 0;
 	state->regs = NULL;
+	state->unreliable = false;
 }
 
 /*
@@ -181,7 +187,6 @@ int kunwind_next_regs_pc(struct kunwind_state *state)
 	state->regs = regs;
 	state->common.pc = regs->pc;
 	state->common.fp = regs->regs[29];
-	state->regs = NULL;
 	state->source = KUNWIND_SOURCE_REGS_PC;
 	return 0;
 }
@@ -244,6 +249,65 @@ kunwind_next_frame_record(struct kunwind_state *state)
 	return 0;
 }
 
+#ifdef CONFIG_ARM64_LR_UNWIND
+
+/*
+ * Unwind from a pt_regs according to .lr_unwind info.
+ */
+static __always_inline int
+kunwind_next_regs_lr_unwind(struct kunwind_state *state)
+{
+	struct pt_regs *regs = state->regs;
+	enum lr_unwind_mode mode;
+	unsigned int sp_off;
+	unsigned long fp;
+	int err;
+
+	if (WARN_ON_ONCE(state->source != KUNWIND_SOURCE_REGS_PC))
+		return -EINVAL;
+	if (WARN_ON_ONCE(!regs))
+		return -EINVAL;
+
+	state->regs = NULL;
+
+	err = lr_unwind_find(state->common.pc, &mode, &sp_off);
+	if (err) {
+		/* Fallback to FP based unwinder */
+		state->unreliable = true;
+		return kunwind_next_frame_record(state);
+	}
+
+	switch (mode) {
+	case LR_UNWIND_LIVE:
+		if (state->common.fp & 0x7)
+			return -EINVAL;
+		state->common.pc = regs->regs[30];
+		state->source = KUNWIND_SOURCE_REGS_LR;
+		return 0;
+	case LR_UNWIND_FP:
+		return kunwind_next_frame_record(state);
+	case LR_UNWIND_SP:
+		if (check_add_overflow(regs->sp, (unsigned long)sp_off, &fp))
+			return -EINVAL;
+		state->common.fp = fp;
+		return kunwind_next_frame_record(state);
+	default:
+		return -EINVAL;
+	}
+}
+
+#else /* !CONFIG_ARM64_LR_UNWIND */
+
+static __always_inline int
+kunwind_next_regs_lr_unwind(struct kunwind_state *state)
+{
+	state->regs = NULL;
+	state->unreliable = true;
+	return kunwind_next_frame_record(state);
+}
+
+#endif /* CONFIG_ARM64_LR_UNWIND */
+
 /*
  * Unwind from one frame record (A) to the next frame record (B).
  *
@@ -259,10 +323,13 @@ kunwind_next(struct kunwind_state *state)
 	state->flags.all = 0;
 
 	switch (state->source) {
+	case KUNWIND_SOURCE_REGS_PC:
+		err = kunwind_next_regs_lr_unwind(state);
+		break;
 	case KUNWIND_SOURCE_FRAME:
 	case KUNWIND_SOURCE_CALLER:
 	case KUNWIND_SOURCE_TASK:
-	case KUNWIND_SOURCE_REGS_PC:
+	case KUNWIND_SOURCE_REGS_LR:
 		err = kunwind_next_frame_record(state);
 		break;
 	default:
@@ -294,7 +361,7 @@ do_kunwind(struct kunwind_state *state, kunwind_consume_fn consume_state,
 			return -EINVAL;
 		ret = kunwind_next(state);
 		if (ret == -ENOENT)
-			return 0;
+			return state->unreliable ? -EINVAL : 0;
 		if (ret < 0)
 			return ret;
 	}
@@ -394,14 +461,11 @@ static __always_inline bool
 arch_reliable_kunwind_consume_entry(const struct kunwind_state *state, void *cookie)
 {
 	/*
-	 * At an exception boundary we can reliably consume the saved PC. We do
-	 * not know whether the LR was live when the exception was taken, and
-	 * so we cannot perform the next unwind step reliably.
-	 *
-	 * All that matters is whether the *entire* unwind is reliable, so give
-	 * up as soon as we hit an exception boundary.
+	 * If unwinding across an exception boundary fell back to frame-pointer
+	 * unwinding without a valid .lr_unwind entry, the caller frame may
+	 * have been skipped or duplicated. Abort reliable stack walks.
 	 */
-	if (state->source == KUNWIND_SOURCE_REGS_PC)
+	if (state->unreliable)
 		return false;
 
 	return arch_kunwind_consume_entry(state, cookie);
@@ -452,6 +516,7 @@ static const char *state_source_string(const struct kunwind_state *state)
 	case KUNWIND_SOURCE_CALLER:	return "C";
 	case KUNWIND_SOURCE_TASK:	return "T";
 	case KUNWIND_SOURCE_REGS_PC:	return "P";
+	case KUNWIND_SOURCE_REGS_LR:	return "L";
 	default:			return "U";
 	}
 }
-- 
2.56.0.679.gc527c65306-goog


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2026-11-17  4:35 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17  4:34 [RFC PATCH 00/12] arm64, unwind: add .lr_unwind unwinder for reliable stacktrace Dylan Hatch
2026-11-17  4:34 ` [RFC PATCH 01/12] arm64, unwind: build with .eh_frame and add gen_lr_unwind skeleton Dylan Hatch
2026-11-17  4:34 ` [RFC PATCH 02/12] arm64: factor .eh_frame parser out of patch-scs.c into <asm/eh_frame.h> Dylan Hatch
2026-11-17  4:34 ` [RFC PATCH 03/12] arm64: eh_frame: track CFA and x29/x30 register state in CFI parser Dylan Hatch
2026-11-17  4:34 ` [RFC PATCH 04/12] arm64, unwind: parse .eh_frame into .lr_unwind entries in gen_lr_unwind Dylan Hatch
2026-11-17  4:34 ` [RFC PATCH 05/12] scripts/gen_lr_unwind: emit .lr_unwind_fde and .lr_unwind_ranges Dylan Hatch
2026-11-17  4:34 ` [RFC PATCH 06/12] scripts/gen_lr_unwind: add --dump option to inspect .lr_unwind tables Dylan Hatch
2026-11-17  4:34 ` [RFC PATCH 07/12] scripts/sorttable: sort .lr_unwind_fde by start_offset Dylan Hatch
2026-11-17  4:35 ` [RFC PATCH 08/12] arm64, unwind: add in-kernel .lr_unwind lookup and validation Dylan Hatch
2026-11-17  4:35 ` [RFC PATCH 09/12] arm64/module, unwind: add .lr_unwind support for modules Dylan Hatch
2026-11-17  4:35 ` [RFC PATCH 10/12] arm64: entry: add unwind info for call_on_irq_stack() Dylan Hatch
2026-11-17  4:35 ` [RFC PATCH 11/12] arm64, crypto/lib: annotate leaf functions with CFI info Dylan Hatch
2026-11-17  4:35 ` [RFC PATCH 12/12] unwind: arm64: use .lr_unwind to unwind interrupt frames Dylan Hatch

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®