[PATCH 00/21] arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0

[PATCH 00/21] arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0

[Will Deacon]

Hi everyone,

This series is a bit of a complicated juggling act that, on its own,
doesn't achieve an awful lot. However, it lays the ground work for
sizing the kernel stack at runtime, e.g. via a cmdline option or even
potentially on a per-task basis and so I would like to work towards
getting it merged independently.

The series is based on v7.3-rc1 and structured as follows:

  * The first 9 patches move the 'current' task pointer from SP_EL0
    to TPIDRRO_EL0.

  * The following 8 patches point the newly-freed SP_EL0 at the overflow
    stack and switch to it explicitly when we detect a kernel stack
    overflow.

  * The final 4 patches turn everything on its head, so that the
    overflow stack and kernel stack are swapped, with the former now
    residing in SP_EL1 and the latter in SP_EL0.

At the end of all that, when we take an exception from EL1, we are
immediately transitioned to the overflow stack (now renamed "exception
stack") and can push registers right away. This also means that using
SPINTMASK to control NMI masking becomes a possibility, although
speaking to Mark, Vladimir and Ada, they all seem to prefer ALLINT.

Mostafa will soon post a follow-up series that allows the kernel stack
size to be specified on the kernel cmdline, which we are hoping to use
to configure an 8k stack size in Android. I will be talking more about
all of this at LPC in the Memory Management MC:

  https://lpc.events/event/20/contributions/2419/

Cheers,

Will

Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Mostafa Saleh <smostafa@google.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>
Cc: Linus Walleij <linusw@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>

--->8

Mostafa Saleh (1):
  KVM: arm64: Protect TPIDRRO_EL0 across guest entry/exit

Will Deacon (20):
  arm64: entry: Defer setting of TPIDRRO_EL0 until exit to userspace
  arm64: entry: Only check for stack overflow on exceptions from EL1
  arm64: stackprotector: Temporarily disable per-task stackprotector
  arm64: bpf: Add support for generating reads of TPIDRRO_EL0
  arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0
  selftests/bpf: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'
  scripts/gdb: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'
  arm64: stackprotector: Re-enable per-task stackprotector
  arm64: percpu: Specialise set_my_cpu_offset() for the primary CPU
  arm64: percpu: Annotate __kern_my_cpu_offset() as '__always_inline'
  KVM: arm64: Preserve handler/thread bit of EL1 mode in
    __finalise_el2()
  arm64: sdei: Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE
  arm64: sdei: Support SDEI events from kernel handler and thread modes
  arm64: entry: Point SP_EL0 at the overflow stack
  arm64: entry: Implement EL1t exception handlers for overflow stack
  arm64: entry: Use SPSel to switch to overflow stack
  arm64: entry: Split up kernel_ventry macro into separate helper macros
  arm64: entry: The great stack switcheroo
  arm64: tracing: Advertise a mode of EL1t in synthetic kernel regs
  arm64: Rename 'overflow_stack' and OVERFLOW_STACK_SIZE

 arch/arm64/Kconfig                            |   2 +-
 arch/arm64/Makefile                           |   2 +-
 arch/arm64/include/asm/assembler.h            |   2 +-
 arch/arm64/include/asm/current.h              |   6 +-
 arch/arm64/include/asm/ftrace.h               |   4 +-
 arch/arm64/include/asm/insn.h                 |   1 +
 arch/arm64/include/asm/kvm_asm.h              |  26 +++++
 arch/arm64/include/asm/memory.h               |  10 +-
 arch/arm64/include/asm/percpu.h               |  10 +-
 arch/arm64/include/asm/perf_event.h           |   2 +-
 arch/arm64/include/asm/sdei.h                 |   2 +
 arch/arm64/include/asm/stacktrace.h           |   9 +-
 arch/arm64/include/asm/stacktrace/nvhe.h      |   3 +-
 arch/arm64/include/uapi/asm/ptrace.h          |   1 +
 arch/arm64/kernel/asm-offsets.c               |   2 +
 arch/arm64/kernel/entry-common.c              |  51 +++++++--
 arch/arm64/kernel/entry.S                     | 108 +++++++++---------
 arch/arm64/kernel/head.S                      |   8 +-
 arch/arm64/kernel/hyp-stub.S                  |   7 +-
 arch/arm64/kernel/process.c                   |  21 +---
 arch/arm64/kernel/sdei.c                      |   6 +-
 arch/arm64/kernel/smp.c                       |  19 ++-
 arch/arm64/kernel/stacktrace.c                |   2 +-
 arch/arm64/kernel/sys_compat.c                |   7 --
 arch/arm64/kernel/traps.c                     |  14 +--
 arch/arm64/kvm/hyp/entry.S                    |  15 ++-
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h    |   2 -
 arch/arm64/kvm/hyp/nvhe/host.S                |   2 +-
 arch/arm64/kvm/hyp/nvhe/stacktrace.c          |   4 +-
 arch/arm64/kvm/stacktrace.c                   |   4 +-
 arch/arm64/mm/proc.S                          |   3 +
 arch/arm64/net/bpf_jit.h                      |   2 +
 arch/arm64/net/bpf_jit_comp.c                 |   4 +-
 scripts/gdb/linux/cpus.py                     |   2 +-
 .../selftests/bpf/progs/verifier_jit_inline.c |   2 +-
 35 files changed, 218 insertions(+), 147 deletions(-)

-- 
2.55.0.979.g7e5102b832-goog

[PATCH 01/21] arm64: entry: Defer setting of TPIDRRO_EL0 until exit to userspace

[Will Deacon]

In preparation for using TPIDRRO_EL0 to point at 'current' while running
inside the kernel, defer its userspace initialisation from the
context-switch patch to exception return. This has the added benefit of
not having to worry about keeping the in-memory value and the register
value in-sync during preemptible sections.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/asm-offsets.c |  1 +
 arch/arm64/kernel/entry.S       |  4 +++-
 arch/arm64/kernel/process.c     | 16 +---------------
 arch/arm64/kernel/sys_compat.c  |  7 -------
 4 files changed, 5 insertions(+), 23 deletions(-)

diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 9c853ed3ceab..6038ab3beb25 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -42,6 +42,7 @@ int main(void)
 #endif
   BLANK();
   DEFINE(THREAD_CPU_CONTEXT,	offsetof(struct task_struct, thread.cpu_context));
+  DEFINE(THREAD_TP_VALUE,	offsetof(struct task_struct, thread.uw.tp_value));
   DEFINE(THREAD_SCTLR_USER,	offsetof(struct task_struct, thread.sctlr_user));
 #ifdef CONFIG_ARM64_PTR_AUTH
   DEFINE(THREAD_KEYS_USER,	offsetof(struct task_struct, thread.keys_user));
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index f63049ac32dc..59f045e496ec 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -47,7 +47,6 @@
 	b	.Lskip_tramp_vectors_cleanup\@
 	.if	\regsize == 64
 	mrs	x30, tpidrro_el0
-	msr	tpidrro_el0, xzr
 	.else
 	mov	x30, xzr
 	.endif
@@ -360,9 +359,12 @@ alternative_else_nop_endif
 	.if	\el == 0
 	ldr	x23, [sp, #S_SP]		// load return stack pointer
 	msr	sp_el0, x23
+	msr	tpidrro_el0, xzr
 	tst	x22, #PSR_MODE32_BIT		// native task?
 	b.eq	3f
 
+	ldr	x0, [tsk, #THREAD_TP_VALUE]
+	msr	tpidrro_el0, x0
 #ifdef CONFIG_ARM64_ERRATUM_845719
 alternative_if ARM64_WORKAROUND_845719
 #ifdef CONFIG_PID_IN_CONTEXTIDR
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 581f80e9b9b7..bfdc12166895 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -256,17 +256,8 @@ static void tls_thread_flush(void)
 	if (system_supports_tpidr2())
 		write_sysreg_s(0, SYS_TPIDR2_EL0);
 
-	if (is_compat_task()) {
+	if (is_compat_task())
 		current->thread.uw.tp_value = 0;
-
-		/*
-		 * We need to ensure ordering between the shadow state and the
-		 * hardware state, so that we don't corrupt the hardware state
-		 * with a stale shadow state during context switch.
-		 */
-		barrier();
-		write_sysreg(0, tpidrro_el0);
-	}
 }
 
 static void flush_tagged_addr_state(void)
@@ -531,11 +522,6 @@ static void tls_thread_switch(struct task_struct *next)
 {
 	tls_preserve_current_state();
 
-	if (is_compat_thread(task_thread_info(next)))
-		write_sysreg(next->thread.uw.tp_value, tpidrro_el0);
-	else
-		write_sysreg(0, tpidrro_el0);
-
 	write_sysreg(*task_user_tls(next), tpidr_el0);
 	if (system_supports_tpidr2())
 		write_sysreg_s(next->thread.tpidr2_el0, SYS_TPIDR2_EL0);
diff --git a/arch/arm64/kernel/sys_compat.c b/arch/arm64/kernel/sys_compat.c
index 0451f96c2c3f..59eee301c6f4 100644
--- a/arch/arm64/kernel/sys_compat.c
+++ b/arch/arm64/kernel/sys_compat.c
@@ -89,13 +89,6 @@ long compat_arm_syscall(struct pt_regs *regs, int scno)
 
 	case __ARM_NR_compat_set_tls:
 		current->thread.uw.tp_value = regs->regs[0];
-
-		/*
-		 * Protect against register corruption from context switch.
-		 * See comment in tls_thread_flush.
-		 */
-		barrier();
-		write_sysreg(regs->regs[0], tpidrro_el0);
 		return 0;
 
 	default:
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 02/21] arm64: entry: Only check for stack overflow on exceptions from EL1

[Will Deacon]

Checking for kernel stack overflow on entry from EL0 is largely
pointless and gets in the way of using SPSel to switch to the overflow
stack. If the kernel stack overflows on entry from userspace, that
implies that it was unbalanced on the last exception return which means
we can't sensibly reason about the state of the system given that we
could've returned to an arbitrary virtual address in an arbitrary mode.

Remove the stack overflow check from the EL0 exception entry path.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/entry.S | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 59f045e496ec..b5d8277f608a 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -54,6 +54,8 @@
 	.endif
 
 	sub	sp, sp, #PT_REGS_SIZE
+
+	.if	\el == 1
 	/*
 	 * Test whether the SP has overflowed, without corrupting a GPR.
 	 * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT)
@@ -64,8 +66,11 @@
 	tbnz	x0, #THREAD_SHIFT, 0f
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
+	.endif
+
 	b	el\el\ht\()_\regsize\()_\label
 
+	.if	\el == 1
 0:
 	/*
 	 * Either we've just detected an overflow, or we've taken an exception
@@ -96,6 +101,7 @@
 	sub	sp, sp, x0
 	mrs	x0, tpidrro_el0
 	b	el\el\ht\()_\regsize\()_\label
+	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
 	.endm
 
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 03/21] arm64: stackprotector: Temporarily disable per-task stackprotector

[Will Deacon]

Per-task stackprotector relies on telling the compiler about the system
register used to hold a pointer to the current task. As we're about to
change that around, temporarily disable per-task stackprotector for now.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef944..b8270b301809 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2455,6 +2455,7 @@ config CC_HAVE_STACKPROTECTOR_SYSREG
 config STACKPROTECTOR_PER_TASK
 	def_bool y
 	depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG
+	depends on BROKEN
 
 config UNWIND_PATCH_PAC_INTO_SCS
 	bool "Enable shadow call stack dynamically using code patching"
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 04/21] arm64: bpf: Add support for generating reads of TPIDRRO_EL0

[Will Deacon]

In preparation for storing 'current' in TPIDRRO_EL0, add support for the
instruction encoding of "MRS Rt, TPIDRRO_EL0" to the BPF JIT.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/insn.h | 1 +
 arch/arm64/net/bpf_jit.h      | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 1ce75a442638..0e739740513d 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -136,6 +136,7 @@ enum aarch64_insn_special_register {
 };
 
 enum aarch64_insn_system_register {
+	AARCH64_INSN_SYSREG_TPIDRRO_EL0	= 0x5e83,
 	AARCH64_INSN_SYSREG_TPIDR_EL1	= 0x4684,
 	AARCH64_INSN_SYSREG_TPIDR_EL2	= 0x6682,
 	AARCH64_INSN_SYSREG_SP_EL0	= 0x4208,
diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h
index b2fe6e6dcf44..df469a005373 100644
--- a/arch/arm64/net/bpf_jit.h
+++ b/arch/arm64/net/bpf_jit.h
@@ -331,6 +331,8 @@
 	aarch64_insn_gen_adr(0, offset, Rd, AARCH64_INSN_ADR_TYPE_ADR)
 
 /* MRS */
+#define A64_MRS_TPIDRRO_EL0(Rt) \
+	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDRRO_EL0)
 #define A64_MRS_TPIDR_EL1(Rt) \
 	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL1)
 #define A64_MRS_TPIDR_EL2(Rt) \
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 05/21] KVM: arm64: Protect TPIDRRO_EL0 across guest entry/exit

[Will Deacon]

From: Mostafa Saleh <smostafa@google.com>

To prepare for using TPIDRRO_EL0 to hold 'current' and repurposing
SP_EL0 as an overflow stack pointer, eagerly switch both registers when
entering and exiting the guest.

Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/kvm_asm.h           | 10 ++++++++++
 arch/arm64/kernel/asm-offsets.c            |  1 +
 arch/arm64/kvm/hyp/entry.S                 | 12 ++++++++----
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h |  2 --
 4 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index e5b92ac09e69..1cfe9216a2ff 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -408,6 +408,16 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt,
 	msr	sp_el0, \tmp
 .endm
 
+.macro save_tpidrro_el0 ctxt, tmp
+	mrs	\tmp,	tpidrro_el0
+	str	\tmp,	[\ctxt, #CPU_TPIDRRO_EL0_OFFSET]
+.endm
+
+.macro restore_tpidrro_el0 ctxt, tmp
+	ldr	\tmp,	  [\ctxt, #CPU_TPIDRRO_EL0_OFFSET]
+	msr	tpidrro_el0, \tmp
+.endm
+
 #endif
 
 #endif /* __ARM_KVM_ASM_H__ */
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 6038ab3beb25..cb8448cc5f51 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -108,6 +108,7 @@ int main(void)
   DEFINE(VCPU_HCR_EL2,		offsetof(struct kvm_vcpu, arch.hcr_el2));
   DEFINE(CPU_USER_PT_REGS,	offsetof(struct kvm_cpu_context, regs));
   DEFINE(CPU_ELR_EL2,		offsetof(struct kvm_cpu_context, sys_regs[ELR_EL2]));
+  DEFINE(CPU_TPIDRRO_EL0_OFFSET, offsetof(struct kvm_cpu_context, sys_regs[TPIDRRO_EL0]));
   DEFINE(CPU_RGSR_EL1,		offsetof(struct kvm_cpu_context, sys_regs[RGSR_EL1]));
   DEFINE(CPU_GCR_EL1,		offsetof(struct kvm_cpu_context, sys_regs[GCR_EL1]));
   DEFINE(CPU_APIAKEYLO_EL1,	offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1]));
diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
index 308100ed25de..4c89931a6a92 100644
--- a/arch/arm64/kvm/hyp/entry.S
+++ b/arch/arm64/kvm/hyp/entry.S
@@ -30,8 +30,9 @@ SYM_FUNC_START(__guest_enter)
 	// Store the hyp regs
 	save_callee_saved_regs x1
 
-	// Save hyp's sp_el0
+	// Save hyp's sp_el0 and tpidrro_el0
 	save_sp_el0	x1, x2
+	save_tpidrro_el0 x1, x2
 
 	// Now the hyp state is stored if we have a pending RAS SError it must
 	// affect the host or hyp. If any asynchronous exception is pending we
@@ -66,8 +67,9 @@ alternative_else_nop_endif
 	// when this feature is enabled for kernel code.
 	ptrauth_switch_to_guest x29, x0, x1, x2
 
-	// Restore the guest's sp_el0
+	// Restore the guest's sp_el0 and tpidrro_el0
 	restore_sp_el0 x29, x0
+	restore_tpidrro_el0 x29, x0
 
 	// Restore guest regs x0-x17
 	ldp	x0, x1,   [x29, #CPU_XREG_OFFSET(0)]
@@ -146,8 +148,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL)
 	// Store the guest regs x18-x29, lr
 	save_callee_saved_regs x1
 
-	// Store the guest's sp_el0
+	// Store the guest's sp_el0 and tpidrro_el0
 	save_sp_el0	x1, x2
+	save_tpidrro_el0 x1, x2
 
 	adr_this_cpu x2, kvm_hyp_ctxt, x3
 
@@ -161,8 +164,9 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL)
 	// mte_switch_to_hyp(g_ctxt, h_ctxt, reg1)
 	mte_switch_to_hyp x1, x2, x3
 
-	// Restore hyp's sp_el0
+	// Restore hyp's sp_el0 and tpidrro_el0
 	restore_sp_el0 x2, x3
+	restore_tpidrro_el0 x2, x3
 
 	// Now restore the hyp regs
 	restore_callee_saved_regs x2
diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
index c382848d3194..b5f7ef8e6371 100644
--- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
+++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
@@ -66,7 +66,6 @@ static inline void __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
 static inline void __sysreg_save_user_state(struct kvm_cpu_context *ctxt)
 {
 	ctxt_sys_reg(ctxt, TPIDR_EL0)	= read_sysreg(tpidr_el0);
-	ctxt_sys_reg(ctxt, TPIDRRO_EL0)	= read_sysreg(tpidrro_el0);
 }
 
 static inline bool ctxt_has_mte(struct kvm_cpu_context *ctxt)
@@ -209,7 +208,6 @@ static inline void __sysreg_restore_common_state(struct kvm_cpu_context *ctxt)
 static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
 {
 	write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL0),	tpidr_el0);
-	write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0),	tpidrro_el0);
 }
 
 static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt,
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 06/21] arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0

[Will Deacon]

To make SP_EL0 available to point at the per-cpu kernel overflow stack,
we must relocate 'current' elsewhere. With TPIDRRO_EL0 now restored late
in the return-to-user path, we can otherwise use it to hold 'current'
and update the various accessors accordingly.

Move 'current' from SP_EL0 to TPIDRRO_EL0.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/assembler.h |  2 +-
 arch/arm64/include/asm/current.h   |  6 +++---
 arch/arm64/kernel/entry.S          | 20 ++++++++++----------
 arch/arm64/kernel/head.S           |  2 +-
 arch/arm64/kernel/process.c        |  5 +++--
 arch/arm64/net/bpf_jit_comp.c      |  4 ++--
 6 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 0b58b550e8dc..fd1ae5337935 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -587,7 +587,7 @@ alternative_else_nop_endif
  * Return the current task_struct.
  */
 	.macro	get_current_task, rd
-	mrs	\rd, sp_el0
+	mrs	\rd, tpidrro_el0
 	.endm
 
 /*
diff --git a/arch/arm64/include/asm/current.h b/arch/arm64/include/asm/current.h
index c92912eaf186..8c7efac05394 100644
--- a/arch/arm64/include/asm/current.h
+++ b/arch/arm64/include/asm/current.h
@@ -14,11 +14,11 @@ struct task_struct;
  */
 static __always_inline struct task_struct *get_current(void)
 {
-	unsigned long sp_el0;
+	unsigned long tpidrro_el0;
 
-	asm ("mrs %0, sp_el0" : "=r" (sp_el0));
+	asm ("mrs %0, tpidrro_el0" : "=r" (tpidrro_el0));
 
-	return (struct task_struct *)sp_el0;
+	return (struct task_struct *)tpidrro_el0;
 }
 
 #define current get_current()
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index b5d8277f608a..a45be0a837c8 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -81,9 +81,9 @@
 	/* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */
 	msr	tpidr_el0, x0
 
-	/* Recover the original x0 value and stash it in tpidrro_el0 */
+	/* Recover the original x0 value and stash it in sp_el0 */
 	sub	x0, sp, x0
-	msr	tpidrro_el0, x0
+	msr	sp_el0, x0
 
 	/* Switch to the overflow stack */
 	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0
@@ -99,7 +99,7 @@
 
 	/* We were already on the overflow stack. Restore sp/x0 and carry on. */
 	sub	sp, sp, x0
-	mrs	x0, tpidrro_el0
+	mrs	x0, sp_el0
 	b	el\el\ht\()_\regsize\()_\label
 	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
@@ -226,7 +226,7 @@ alternative_cb_end
 	clear_gp_regs
 	mrs	x21, sp_el0
 	ldr_this_cpu	tsk, __entry_task, x20
-	msr	sp_el0, tsk
+	msr	tpidrro_el0, tsk
 
 	/*
 	 * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions
@@ -548,7 +548,7 @@ SYM_CODE_START_LOCAL(__bad_stack)
 	 */
 
 	/* Restore the original x0 value */
-	mrs	x0, tpidrro_el0
+	mrs	x0, sp_el0
 
 	/*
 	 * Store the original GPRs to the new stack. The orginal SP (minus
@@ -843,7 +843,7 @@ SYM_FUNC_START(cpu_switch_to)
 	ldp	x29, x9, [x8], #16
 	ldr	lr, [x8]
 	mov	sp, x9
-	msr	sp_el0, x1
+	msr	tpidrro_el0, x1
 	ptrauth_keys_install_kernel x1, x8, x9, x10
 	scs_save x0
 	scs_load_current
@@ -1031,11 +1031,11 @@ SYM_CODE_START(__sdei_asm_handler)
 
 	/*
 	 * We may have interrupted userspace, or a guest, or exit-from or
-	 * return-to either of these. We can't trust sp_el0, restore it.
+	 * return-to either of these. We can't trust tpidrro_el0, restore it.
 	 */
-	mrs	x28, sp_el0
+	mrs	x28, tpidrro_el0
 	ldr_this_cpu	dst=x0, sym=__entry_task, tmp=x1
-	msr	sp_el0, x0
+	msr	tpidrro_el0, x0
 
 	/* If we interrupted the kernel point to the previous stack/frame. */
 	and     x0, x3, #0xc
@@ -1051,7 +1051,7 @@ SYM_CODE_START(__sdei_asm_handler)
 	mov	x1, x19
 	bl	__sdei_handler
 
-	msr	sp_el0, x28
+	msr	tpidrro_el0, x28
 	/* restore regs >x17 that we clobbered */
 	mov	x4, x19         // keep x4 for __sdei_asm_exit_trampoline
 	ldp	x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14]
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 87a822e5c4ca..c33551ee6572 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -193,7 +193,7 @@ SYM_CODE_END(preserve_boot_args)
 	 * for consistency with user tasks and kthreads.
 	 */
 	.macro	init_cpu_task tsk, tmp1, tmp2
-	msr	sp_el0, \tsk
+	msr	tpidrro_el0, \tsk
 
 	ldr	\tmp1, [\tsk, #TSK_STACK]
 	add	sp, \tmp1, #THREAD_SIZE
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index bfdc12166895..79a19be88ced 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -551,8 +551,9 @@ static void ssbs_thread_switch(struct task_struct *next)
 }
 
 /*
- * We store our current task in sp_el0, which is clobbered by userspace. Keep a
- * shadow copy so that we can restore this upon entry from userspace.
+ * We store our current task in tpidrro_el0, which is clobbered when returning
+ * to userspace. Keep a shadow copy so that we can restore this upon entry from
+ * userspace.
  *
  * This is *only* for exception entry from EL0, and is not valid until we
  * __switch_to() a user task.
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c18e005a41db..51d853b92443 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1694,7 +1694,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
 			cpu_offset = offsetof(struct thread_info, cpu);
 
-			emit(A64_MRS_SP_EL0(tmp), ctx);
+			emit(A64_MRS_TPIDRRO_EL0(tmp), ctx);
 			if (is_lsi_offset(cpu_offset, 2)) {
 				emit(A64_LDR32I(r0, tmp, cpu_offset), ctx);
 			} else {
@@ -1707,7 +1707,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 		/* Implement helper call to bpf_get_current_task/_btf() inline */
 		if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
 					   insn->imm == BPF_FUNC_get_current_task_btf)) {
-			emit(A64_MRS_SP_EL0(r0), ctx);
+			emit(A64_MRS_TPIDRRO_EL0(r0), ctx);
 			break;
 		}
 
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 07/21] selftests/bpf: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'

[Will Deacon]

The current task pointer now lives in TPIDRRO_EL0 instead of SP_EL0, so
update the bpf selftests with the new register allocation.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 tools/testing/selftests/bpf/progs/verifier_jit_inline.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/bpf/progs/verifier_jit_inline.c b/tools/testing/selftests/bpf/progs/verifier_jit_inline.c
index 02e562f56f9d..a3d6c095acc3 100644
--- a/tools/testing/selftests/bpf/progs/verifier_jit_inline.c
+++ b/tools/testing/selftests/bpf/progs/verifier_jit_inline.c
@@ -9,7 +9,7 @@ __success __retval(0)
 __arch_x86_64
 __jited("	addq	%gs:{{.*}}, %rax")
 __arch_arm64
-__jited("	mrs	x8, SP_EL0")
+__jited("	mrs	x8, TPIDRRO_EL0")
 __arch_riscv64
 __jited("	mv	a5, tp")
 __arch_loongarch
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 08/21] scripts/gdb: arm64: Use TPIDRRO_EL0 instead of SP_EL0 for 'current'

[Will Deacon]

The current task pointer now lives in TPIDRRO_EL0 instead of SP_EL0, so
update the gdb 'cpus.py' helper script with the new register allocation.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 scripts/gdb/linux/cpus.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/gdb/linux/cpus.py b/scripts/gdb/linux/cpus.py
index 6edf4ef61636..547a623280ae 100644
--- a/scripts/gdb/linux/cpus.py
+++ b/scripts/gdb/linux/cpus.py
@@ -185,7 +185,7 @@ def get_current_task(cpu):
             var_ptr = gdb.parse_and_eval("&current_task")
             return per_cpu(var_ptr, cpu).dereference()
     elif utils.is_target_arch("aarch64"):
-        current_task_addr = gdb.parse_and_eval("(unsigned long)$SP_EL0")
+        current_task_addr = gdb.parse_and_eval("(unsigned long)$TPIDRRO_EL0")
         if (current_task_addr >> 63) != 0:
             current_task = current_task_addr.cast(task_ptr_type)
             return current_task.dereference()
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 09/21] arm64: stackprotector: Re-enable per-task stackprotector

[Will Deacon]

Now that 'current' resides in TPIDRRO_EL0, update the per-task
stackprotector compiler invocation to use the new register.

Note that this requires a very recent build of clang [1], as (unlike
GCC), it historically rejected all registers other than SP_EL0.

Cc: Mark Rutland <mark.rutland@arm.com>
Link: https://github.com/llvm/llvm-project/pull/188054 [1]
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/Kconfig  | 3 +--
 arch/arm64/Makefile | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b8270b301809..020fcaf36b1b 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2450,12 +2450,11 @@ config RANDOMIZE_MODULE_REGION_FULL
 	  exhaustion, modules might be able to fall back to a larger 2GB area.
 
 config CC_HAVE_STACKPROTECTOR_SYSREG
-	def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=0)
+	def_bool $(cc-option,-mstack-protector-guard=sysreg -mstack-protector-guard-reg=tpidrro_el0 -mstack-protector-guard-offset=0)
 
 config STACKPROTECTOR_PER_TASK
 	def_bool y
 	depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG
-	depends on BROKEN
 
 config UNWIND_PATCH_PAC_INTO_SCS
 	bool "Enable shadow call stack dynamically using code patching"
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6b005c8fef70..a293ab67818f 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -72,7 +72,7 @@ ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y)
 prepare: stack_protector_prepare
 stack_protector_prepare: prepare0
 	$(eval KBUILD_CFLAGS += -mstack-protector-guard=sysreg		  \
-				-mstack-protector-guard-reg=sp_el0	  \
+				-mstack-protector-guard-reg=tpidrro_el0	  \
 				-mstack-protector-guard-offset=$(shell	  \
 			awk '{if ($$2 == "TSK_STACK_CANARY") print $$3;}' \
 				$(objtree)/include/generated/asm-offsets.h))
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 10/21] arm64: percpu: Specialise set_my_cpu_offset() for the primary CPU

[Will Deacon]

set_my_cpu_offset() is only called from smp_prepare_boot_cpu() which
runs once on the primary CPU and before alternatives have been applied.

Rework the function so that it operates only on the boot CPU offset,
drop the unused alternative code sequence and internalise it into smp.c
alongside its only caller.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/percpu.h | 8 --------
 arch/arm64/kernel/smp.c         | 8 +++++++-
 2 files changed, 7 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index b57b2bb00967..3dc7291b16e9 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -12,14 +12,6 @@
 #include <asm/stack_pointer.h>
 #include <asm/sysreg.h>
 
-static inline void set_my_cpu_offset(unsigned long off)
-{
-	asm volatile(ALTERNATIVE("msr tpidr_el1, %0",
-				 "msr tpidr_el2, %0",
-				 ARM64_HAS_VIRT_HOST_EXTN)
-			:: "r" (off) : "memory");
-}
-
 static inline unsigned long __hyp_my_cpu_offset(void)
 {
 	/*
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index a61dc3016a11..702dee8d0db5 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -446,6 +446,12 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	mark_linear_text_alias_ro();
 }
 
+static void __init set_boot_cpu_offset(void)
+{
+	asm volatile("msr tpidr_el1, %0"
+			:: "r" (per_cpu_offset(0)) : "memory");
+}
+
 void __init smp_prepare_boot_cpu(void)
 {
 	/*
@@ -453,7 +459,7 @@ void __init smp_prepare_boot_cpu(void)
 	 * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
 	 * freed shortly, so we must move over to the runtime per-cpu area.
 	 */
-	set_my_cpu_offset(per_cpu_offset(smp_processor_id()));
+	set_boot_cpu_offset();
 
 	cpuinfo_store_boot_cpu();
 	setup_boot_cpu_features();
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 11/21] arm64: percpu: Annotate __kern_my_cpu_offset() as '__always_inline'

[Will Deacon]

Allow the use of per-cpu variables from noinstr functions by marking
__kern_my_cpu_offset() as '__always_inline'. This will later be relied
upon to detect stack overflow of the overflow stack from noinstr C code.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/percpu.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 3dc7291b16e9..cb9ae8a1ae87 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -21,7 +21,7 @@ static inline unsigned long __hyp_my_cpu_offset(void)
 	return read_sysreg(tpidr_el2);
 }
 
-static inline unsigned long __kern_my_cpu_offset(void)
+static __always_inline unsigned long __kern_my_cpu_offset(void)
 {
 	unsigned long off;
 
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 12/21] KVM: arm64: Preserve handler/thread bit of EL1 mode in __finalise_el2()

[Will Deacon]

__finalise_el2() unconditionally returns to EL2h when VHE is being used.
In preparation for handling callers from both EL1h and EL1t, preserve
the handler/thread bit of the calling mode when rewriting the exception
level in the SPSR.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/uapi/asm/ptrace.h | 1 +
 arch/arm64/kernel/hyp-stub.S         | 7 ++++---
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h
index 15649a253a57..25d9d9a0a6f7 100644
--- a/arch/arm64/include/uapi/asm/ptrace.h
+++ b/arch/arm64/include/uapi/asm/ptrace.h
@@ -37,6 +37,7 @@
 #define PSR_MODE_EL3t	0x0000000c
 #define PSR_MODE_EL3h	0x0000000d
 #define PSR_MODE_MASK	0x0000000f
+#define PSR_MODE_h_BIT	0x00000001
 
 /* AArch32 CPSR bits */
 #define PSR_MODE32_BIT		0x00000010
diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S
index 37c6976e44a4..f61778cccd3c 100644
--- a/arch/arm64/kernel/hyp-stub.S
+++ b/arch/arm64/kernel/hyp-stub.S
@@ -149,10 +149,11 @@ SYM_CODE_START_LOCAL(__finalise_el2)
 
 	isb
 
-	// Hack the exception return to stay at EL2
+	// Hack the exception return to stay at EL2 ...
 	mrs	x0, spsr_el1
-	and	x0, x0, #~PSR_MODE_MASK
-	mov	x1, #PSR_MODE_EL2h
+	// ... preserving the 'h' bit so we don't swizzle SP.
+	and	x0, x0, #(~PSR_MODE_MASK | PSR_MODE_h_BIT)
+	mov	x1, #PSR_MODE_EL2t
 	orr	x0, x0, x1
 	msr	spsr_el1, x0
 
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 13/21] arm64: sdei: Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE

[Will Deacon]

The SDEI header includes are a bit of a mess: linux/arm_sdei.h only
includes asm/sdei.h if CONFIG_ARM_SDE_INTERFACE=y and then goes on to
provide a definition of the sdei_arch_get_entry_point() macro if it
hasn't already been defined. asm/sdei.h, however, defines the same macro
unconditionally and so this breaks the build if CONFIG_ARM_SDE_INTERFACE
is not set and asm/sdei.h is included after linux/arm_sdei.h.

The kbuild robot ran into this situation with randconfig testing, thanks
to the transitive inclusion of asm/sdei.h via asm/stacktrace.h.

Guard most of asm/sdei.h with CONFIG_ARM_SDE_INTERFACE to avoid
redefining macros provided by linux/arm_sdei.h.

Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/sdei.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm64/include/asm/sdei.h b/arch/arm64/include/asm/sdei.h
index b2248bd3cb58..b96eb9e22d4b 100644
--- a/arch/arm64/include/asm/sdei.h
+++ b/arch/arm64/include/asm/sdei.h
@@ -9,6 +9,7 @@
 
 #define SDEI_STACK_SIZE		IRQ_STACK_SIZE
 
+#ifdef CONFIG_ARM_SDE_INTERFACE
 #ifndef __ASSEMBLER__
 
 #include <linux/linkage.h>
@@ -50,4 +51,5 @@ unsigned long sdei_arch_get_entry_point(int conduit);
 #define sdei_arch_get_entry_point(x)	sdei_arch_get_entry_point(x)
 
 #endif /* __ASSEMBLER__ */
+#endif /* CONFIG_ARM_SDE_INTERFACE */
 #endif	/* __ASM_SDEI_H */
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 14/21] arm64: sdei: Support SDEI events from kernel handler and thread modes

[Will Deacon]

Section 5.2.2 ("Event resume context") of the SDEI specification states
that, as a result of an SDEI_EVENT_COMPLETE_AND_RESUME call:

 | SPSR_ELc is set to the PSTATE when the event was taken

In preparation for making use of both thread and handler modes in the
kernel, adjust the SDEI event handler so that we return to the correct
kernel vector for the interrupted mode.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/sdei.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 778f2a1faac8..6337ce905593 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -208,7 +208,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	int i, err = 0;
 	int clobbered_registers = 4;
 	u64 elr = read_sysreg(elr_el1);
-	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
+	u32 kernel_mode = read_sysreg(CurrentEL);
 	unsigned long vbar = read_sysreg(vbar_el1);
 
 	if (arm64_kernel_unmapped_at_el0())
@@ -233,7 +233,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 		pr_warn("unsafe: exception during handler\n");
 	}
 
-	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
+	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK) & ~PSR_MODE_h_BIT;
 
 	/*
 	 * If we interrupted the kernel with interrupts masked, we always go
@@ -251,7 +251,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	 * address'.
 	 */
 	if (mode == kernel_mode)
-		return vbar + 0x280;
+		return vbar + (regs->pstate & PSR_MODE_h_BIT ? 0x280 : 0x80);
 	else if (mode & PSR_MODE32_BIT)
 		return vbar + 0x680;
 
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 15/21] arm64: entry: Point SP_EL0 at the overflow stack

[Will Deacon]

With SP_EL0 now available for use in the kernel, point it at the
overflow stack so that exception entry from EL1h will later be able to
switch to the overflow stack by writing to SPSel.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/entry.S |  2 ++
 arch/arm64/kernel/head.S  |  3 +++
 arch/arm64/kernel/smp.c   | 12 ++++++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index a45be0a837c8..6958ee238649 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -227,6 +227,8 @@ alternative_cb_end
 	mrs	x21, sp_el0
 	ldr_this_cpu	tsk, __entry_task, x20
 	msr	tpidrro_el0, tsk
+	adr_this_cpu	x19, overflow_stack + OVERFLOW_STACK_SIZE, x20
+	msr	sp_el0, x19
 
 	/*
 	 * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index c33551ee6572..27a19e1a0ee6 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -210,6 +210,9 @@ SYM_CODE_END(preserve_boot_args)
 	ldr	w\tmp2, [\tsk, #TSK_TI_CPU]
 	ldr	\tmp1, [\tmp1, \tmp2, lsl #3]
 	set_this_cpu_offset \tmp1
+
+	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	msr	sp_el0, \tmp1
 	.endm
 
 /*
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 702dee8d0db5..4c122d6598de 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -51,6 +51,7 @@
 #include <asm/processor.h>
 #include <asm/smp_plat.h>
 #include <asm/sections.h>
+#include <asm/stacktrace.h>
 #include <asm/tlbflush.h>
 #include <asm/ptrace.h>
 #include <asm/virt.h>
@@ -448,8 +449,15 @@ void __init smp_cpus_done(unsigned int max_cpus)
 
 static void __init set_boot_cpu_offset(void)
 {
-	asm volatile("msr tpidr_el1, %0"
-			:: "r" (per_cpu_offset(0)) : "memory");
+	u64 ovf_sp = (u64)raw_cpu_ptr(overflow_stack) + OVERFLOW_STACK_SIZE;
+
+	asm volatile(
+	"	msr	tpidr_el1, %1\n"
+	"	add	%0, %0, %1\n"
+	"	msr	sp_el0, %0" /* Update the overflow stack pointer */
+	: "+r" (ovf_sp)
+	: "r" (per_cpu_offset(0))
+	: "memory");
 }
 
 void __init smp_prepare_boot_cpu(void)
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 16/21] arm64: entry: Implement EL1t exception handlers for overflow stack

[Will Deacon]

In preparation for using EL1t to handle exceptions taken whilst running
on the overflow stack, implement some simple wrappers around the EL1h
handlers which perform a best-effort stack overflow check before
proceeding.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/entry-common.c | 43 +++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 72c03ccea59f..9738142780df 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -327,10 +327,45 @@ static void debug_exception_exit(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(debug_exception_exit);
 
-UNHANDLED(el1t, 64, sync)
-UNHANDLED(el1t, 64, irq)
-UNHANDLED(el1t, 64, fiq)
-UNHANDLED(el1t, 64, error)
+static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs)
+{
+	unsigned long sp = kernel_stack_pointer(regs) - sizeof(*regs);
+	unsigned long ovf_stack = (unsigned long)this_cpu_ptr(overflow_stack);
+
+	/*
+	 * We're in big trouble if we've overflowed the overflow stack
+	 * so perform a best-effort check before we proceed. If our SP
+	 * is outside of the overflow stack for this CPU then presumably
+	 * we're already corrupting memory, so park ourselves here in an
+	 * attempt to contain the damage.
+	 */
+	if (sp < ovf_stack || sp > ovf_stack + OVERFLOW_STACK_SIZE)
+		cpu_park_loop();
+}
+
+asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_sync_handler(regs);
+}
+
+asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_irq_handler(regs);
+}
+
+asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_fiq_handler(regs);
+}
+
+asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs)
+{
+	el1t_64_check_overflow_stack(regs);
+	el1h_64_error_handler(regs);
+}
 
 static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
 {
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 17/21] arm64: entry: Use SPSel to switch to overflow stack

[Will Deacon]

When detecting a stack overflow on exception entry from EL1, use SPSel
to switch to the overflow stack without corrupting any GPRs. Not only is
this simpler than the previous logic, but it also opens the door to
more complicated overflow checks (for example, based on per-task stack
sizes or stacks which are not aligned to a specific page order) as well
as the possibility of returning from a stack fault if we were able to
resolve it.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/entry.S | 69 ++++++++++++---------------------------
 1 file changed, 21 insertions(+), 48 deletions(-)

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 6958ee238649..b4df2f23ebe7 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -51,11 +51,11 @@
 	mov	x30, xzr
 	.endif
 .Lskip_tramp_vectors_cleanup\@:
-	.endif
-
+	sub	sp, sp, #PT_REGS_SIZE
+	.else	/* \el == 1 */
+	.ifc	\ht, h
 	sub	sp, sp, #PT_REGS_SIZE
 
-	.if	\el == 1
 	/*
 	 * Test whether the SP has overflowed, without corrupting a GPR.
 	 * Task and IRQ stacks are aligned so that SP & (1 << THREAD_SHIFT)
@@ -63,45 +63,16 @@
 	 */
 	add	sp, sp, x0			// sp' = sp + x0
 	sub	x0, sp, x0			// x0' = sp' - x0 = (sp + x0) - x0 = sp
-	tbnz	x0, #THREAD_SHIFT, 0f
+	tbnz	x0, #THREAD_SHIFT, __bad_stack
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
+	.else	/* EL1t */
+	msr	spsel, #0			// Stay on the overflow stack
+	sub	sp, sp, #PT_REGS_SIZE
+	.endif
 	.endif
 
 	b	el\el\ht\()_\regsize\()_\label
-
-	.if	\el == 1
-0:
-	/*
-	 * Either we've just detected an overflow, or we've taken an exception
-	 * while on the overflow stack. Either way, we won't return to
-	 * userspace, and can clobber EL0 registers to free up GPRs.
-	 */
-
-	/* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */
-	msr	tpidr_el0, x0
-
-	/* Recover the original x0 value and stash it in sp_el0 */
-	sub	x0, sp, x0
-	msr	sp_el0, x0
-
-	/* Switch to the overflow stack */
-	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0
-
-	/*
-	 * Check whether we were already on the overflow stack. This may happen
-	 * after panic() re-enables interrupts.
-	 */
-	mrs	x0, tpidr_el0			// sp of interrupted context
-	sub	x0, sp, x0			// delta with top of overflow stack
-	tst	x0, #~(OVERFLOW_STACK_SIZE - 1)	// within range?
-	b.ne	__bad_stack			// no? -> bad stack pointer
-
-	/* We were already on the overflow stack. Restore sp/x0 and carry on. */
-	sub	sp, sp, x0
-	mrs	x0, sp_el0
-	b	el\el\ht\()_\regsize\()_\label
-	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
 	.endm
 
@@ -544,22 +515,24 @@ SYM_CODE_END(vectors)
 
 SYM_CODE_START_LOCAL(__bad_stack)
 	/*
-	 * We detected an overflow in kernel_ventry, which switched to the
-	 * overflow stack. Stash the exception regs, and head to our overflow
-	 * handler.
+	 * We detected an overflow in kernel_ventry.
+	 * Restore SP and X0.
 	 */
+	sub	x0, sp, x0
+	sub	sp, sp, x0
+	add	sp, sp, #PT_REGS_SIZE
 
-	/* Restore the original x0 value */
-	mrs	x0, sp_el0
+	/* Switch to the overflow stack */
+	msr	spsel, #0
 
-	/*
-	 * Store the original GPRs to the new stack. The orginal SP (minus
-	 * PT_REGS_SIZE) was stashed in tpidr_el0 by kernel_ventry.
-	 */
+	/* Stash the exception regs */
 	sub	sp, sp, #PT_REGS_SIZE
 	kernel_entry 1
-	mrs	x0, tpidr_el0
-	add	x0, x0, #PT_REGS_SIZE
+
+	/* Fix-up the saved SP */
+	msr	spsel, #1
+	mov	x0, sp
+	msr	spsel, #0
 	str	x0, [sp, #S_SP]
 
 	/* Stash the regs for handle_bad_stack */
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 18/21] arm64: entry: Split up kernel_ventry macro into separate helper macros

[Will Deacon]

Handling entry from EL0, EL1h and EL1t in the 'kernel_ventry' assembly
macro is getting a bit unwieldy as the control flow becomes increasingly
dependent on the macro arguments.

Split 'kernel_ventry' into three leaf helper macros for EL0, EL1h and
EL1t, which are then dispatched by simple wrappers.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/kernel/entry.S | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index b4df2f23ebe7..38f9327e6a0a 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -36,10 +36,7 @@
 	.endr
 	.endm
 
-	.macro kernel_ventry, el:req, ht:req, regsize:req, label:req
-	.align 7
-.Lventry_start\@:
-	.if	\el == 0
+	.macro kernel_ventry_el0, ht:req, regsize:req, label:req
 	/*
 	 * This must be the first instruction of the EL0 vector entries. It is
 	 * skipped by the trampoline vectors, to trigger the cleanup.
@@ -52,8 +49,10 @@
 	.endif
 .Lskip_tramp_vectors_cleanup\@:
 	sub	sp, sp, #PT_REGS_SIZE
-	.else	/* \el == 1 */
-	.ifc	\ht, h
+	b	el0\ht\()_\regsize\()_\label
+	.endm
+
+	.macro kernel_ventry_el1h, regsize:req, label:req
 	sub	sp, sp, #PT_REGS_SIZE
 
 	/*
@@ -66,13 +65,23 @@
 	tbnz	x0, #THREAD_SHIFT, __bad_stack
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
-	.else	/* EL1t */
+	b	el1h_\regsize\()_\label
+	.endm
+
+	.macro kernel_ventry_el1t, regsize:req, label:req
 	msr	spsel, #0			// Stay on the overflow stack
 	sub	sp, sp, #PT_REGS_SIZE
-	.endif
-	.endif
+	b	el1t_\regsize\()_\label
+	.endm
 
-	b	el\el\ht\()_\regsize\()_\label
+	.macro kernel_ventry, el:req, ht:req, regsize:req, label:req
+	.align 7
+.Lventry_start\@:
+	.if	\el == 0
+	kernel_ventry_el0	\ht, \regsize, \label
+	.else
+	kernel_ventry_el1\ht	\regsize, \label
+	.endif
 .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
 	.endm
 
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 19/21] arm64: entry: The great stack switcheroo

[Will Deacon]

With the kernel stack pointer in SP_EL1 and the overflow stack pointer
in SP_EL0, it is now straightforward to switch between the two on
exception entry from EL1 by writing to SPSel. However, since exception
entry sets PSTATE.SP to 1 (selecting SP_EL1 as the stack pointer),
repurposing the overflow stack as a more general kernel exception stack
would require writing to SPSel on every exception entry from the kernel.

Switch things around so that the overflow stack resides in SP_EL1, with
the kernel stack residing in SP_EL0.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/kvm_asm.h | 16 +++++++++++++++
 arch/arm64/kernel/entry-common.c | 34 ++++++++++++++++----------------
 arch/arm64/kernel/entry.S        | 26 +++++++++++++-----------
 arch/arm64/kernel/head.S         |  5 ++++-
 arch/arm64/kernel/smp.c          |  8 +++++---
 arch/arm64/kvm/hyp/entry.S       |  3 +++
 arch/arm64/mm/proc.S             |  3 +++
 7 files changed, 63 insertions(+), 32 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 1cfe9216a2ff..bde046c3ca79 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -398,6 +398,22 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt,
 	ldp	x29, lr,  [\ctxt, #CPU_XREG_OFFSET(29)]
 .endm
 
+#ifdef __KVM_VHE_HYPERVISOR__
+.macro activate_exception_stack
+msr	spsel, #1
+.endm
+
+.macro deactivate_exception_stack
+msr	spsel, #0
+.endm
+#else
+.macro activate_exception_stack
+.endm
+
+.macro deactivate_exception_stack
+.endm
+#endif
+
 .macro save_sp_el0 ctxt, tmp
 	mrs	\tmp,	sp_el0
 	str	\tmp,	[\ctxt, #CPU_SP_EL0_OFFSET]
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 9738142780df..9d734cd09f62 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -327,7 +327,7 @@ static void debug_exception_exit(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(debug_exception_exit);
 
-static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs)
+static void noinstr el1h_64_check_overflow_stack(struct pt_regs *regs)
 {
 	unsigned long sp = kernel_stack_pointer(regs) - sizeof(*regs);
 	unsigned long ovf_stack = (unsigned long)this_cpu_ptr(overflow_stack);
@@ -343,28 +343,28 @@ static void noinstr el1t_64_check_overflow_stack(struct pt_regs *regs)
 		cpu_park_loop();
 }
 
-asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_sync_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_sync_handler(regs);
 }
 
-asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_irq_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_irq_handler(regs);
 }
 
-asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_fiq_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_fiq_handler(regs);
 }
 
-asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 {
-	el1t_64_check_overflow_stack(regs);
-	el1h_64_error_handler(regs);
+	el1h_64_check_overflow_stack(regs);
+	el1t_64_error_handler(regs);
 }
 
 static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
@@ -494,7 +494,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
 	arm64_exit_to_kernel_mode(regs, state);
 }
 
-asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_sync_handler(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
 
@@ -578,17 +578,17 @@ static void noinstr el1_interrupt(struct pt_regs *regs,
 		__el1_irq(regs, handler);
 }
 
-asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_irq_handler(struct pt_regs *regs)
 {
 	el1_interrupt(regs, handle_arch_irq);
 }
 
-asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_fiq_handler(struct pt_regs *regs)
 {
 	el1_interrupt(regs, handle_arch_fiq);
 }
 
-asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
+asmlinkage void noinstr el1t_64_error_handler(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
 	irqentry_state_t state;
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 38f9327e6a0a..afcd84510daf 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -54,6 +54,12 @@
 
 	.macro kernel_ventry_el1h, regsize:req, label:req
 	sub	sp, sp, #PT_REGS_SIZE
+	b	el1h_\regsize\()_\label
+	.endm
+
+	.macro kernel_ventry_el1t, regsize:req, label:req
+	msr	spsel, #0			// Stay on the kernel stack
+	sub	sp, sp, #PT_REGS_SIZE
 
 	/*
 	 * Test whether the SP has overflowed, without corrupting a GPR.
@@ -65,12 +71,6 @@
 	tbnz	x0, #THREAD_SHIFT, __bad_stack
 	sub	x0, sp, x0			// x0'' = sp' - x0' = (sp + x0) - sp = x0
 	sub	sp, sp, x0			// sp'' = sp' - x0 = (sp + x0) - x0 = sp
-	b	el1h_\regsize\()_\label
-	.endm
-
-	.macro kernel_ventry_el1t, regsize:req, label:req
-	msr	spsel, #0			// Stay on the overflow stack
-	sub	sp, sp, #PT_REGS_SIZE
 	b	el1t_\regsize\()_\label
 	.endm
 
@@ -208,7 +208,10 @@ alternative_cb_end
 	ldr_this_cpu	tsk, __entry_task, x20
 	msr	tpidrro_el0, tsk
 	adr_this_cpu	x19, overflow_stack + OVERFLOW_STACK_SIZE, x20
-	msr	sp_el0, x19
+	mov	x20, sp
+	msr	sp_el0, x20
+	mov	sp, x19
+	msr	spsel, #0
 
 	/*
 	 * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions
@@ -346,6 +349,9 @@ alternative_else_nop_endif
 
 	.if	\el == 0
 	ldr	x23, [sp, #S_SP]		// load return stack pointer
+	mov	x0, sp
+	msr	spsel, #1
+	mov	sp, x0
 	msr	sp_el0, x23
 	msr	tpidrro_el0, xzr
 	tst	x22, #PSR_MODE32_BIT		// native task?
@@ -532,16 +538,14 @@ SYM_CODE_START_LOCAL(__bad_stack)
 	add	sp, sp, #PT_REGS_SIZE
 
 	/* Switch to the overflow stack */
-	msr	spsel, #0
+	msr	spsel, #1
 
 	/* Stash the exception regs */
 	sub	sp, sp, #PT_REGS_SIZE
 	kernel_entry 1
 
 	/* Fix-up the saved SP */
-	msr	spsel, #1
-	mov	x0, sp
-	msr	spsel, #0
+	mrs	x0, sp_el0
 	str	x0, [sp, #S_SP]
 
 	/* Stash the regs for handle_bad_stack */
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 27a19e1a0ee6..7419e1d8b7ec 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -211,8 +211,11 @@ SYM_CODE_END(preserve_boot_args)
 	ldr	\tmp1, [\tmp1, \tmp2, lsl #3]
 	set_this_cpu_offset \tmp1
 
-	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	mov	\tmp1, sp
 	msr	sp_el0, \tmp1
+	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	mov	sp, \tmp1
+	msr	spsel, #0
 	.endm
 
 /*
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 4c122d6598de..b2e778aae881 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -447,14 +447,16 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	mark_linear_text_alias_ro();
 }
 
-static void __init set_boot_cpu_offset(void)
+static void __init update_boot_cpu_offset_and_overflow_stack(void)
 {
 	u64 ovf_sp = (u64)raw_cpu_ptr(overflow_stack) + OVERFLOW_STACK_SIZE;
 
 	asm volatile(
 	"	msr	tpidr_el1, %1\n"
 	"	add	%0, %0, %1\n"
-	"	msr	sp_el0, %0" /* Update the overflow stack pointer */
+	"	msr	spsel, #1\n"
+	"	mov	sp, %0\n" /* Update the overflow stack pointer */
+	"	msr	spsel, #0"
 	: "+r" (ovf_sp)
 	: "r" (per_cpu_offset(0))
 	: "memory");
@@ -467,7 +469,7 @@ void __init smp_prepare_boot_cpu(void)
 	 * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
 	 * freed shortly, so we must move over to the runtime per-cpu area.
 	 */
-	set_boot_cpu_offset();
+	update_boot_cpu_offset_and_overflow_stack();
 
 	cpuinfo_store_boot_cpu();
 	setup_boot_cpu_features();
diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
index 4c89931a6a92..4d1205d04383 100644
--- a/arch/arm64/kvm/hyp/entry.S
+++ b/arch/arm64/kvm/hyp/entry.S
@@ -31,6 +31,7 @@ SYM_FUNC_START(__guest_enter)
 	save_callee_saved_regs x1
 
 	// Save hyp's sp_el0 and tpidrro_el0
+	activate_exception_stack
 	save_sp_el0	x1, x2
 	save_tpidrro_el0 x1, x2
 
@@ -50,6 +51,7 @@ alternative_else_nop_endif
 	// that would usually be synchonized by the ERET.
 	isb
 	mov	x0, #ARM_EXCEPTION_IRQ
+	deactivate_exception_stack
 	ret
 
 1:
@@ -167,6 +169,7 @@ SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL)
 	// Restore hyp's sp_el0 and tpidrro_el0
 	restore_sp_el0 x2, x3
 	restore_tpidrro_el0 x2, x3
+	deactivate_exception_stack
 
 	// Now restore the hyp regs
 	restore_callee_saved_regs x2
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 12aacc74f764..0811fa569100 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -98,6 +98,7 @@ SYM_FUNC_START(cpu_do_suspend)
 	mrs	x10, oslsr_el1
 	mrs	x11, sctlr_el1
 	get_this_cpu_offset x12
+	msr	spsel, #1
 	mrs	x13, sp_el0
 	stp	x2, x3, [x0]
 	stp	x4, x5, [x0, #16]
@@ -156,6 +157,8 @@ alternative_else_nop_endif
 	msr	sctlr_el1, x12
 	set_this_cpu_offset x13
 	msr	sp_el0, x14
+	msr	spsel, #0
+
 	/*
 	 * Restore oslsr_el1 by writing oslar_el1
 	 */
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 20/21] arm64: tracing: Advertise a mode of EL1t in synthetic kernel regs

[Will Deacon]

When synthesising a kernel regs structure for ftrace or perf, advertise
a mode of EL1t to reflect the mode in which the kernel runs for the vast
majority of the time.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/ftrace.h     | 4 ++--
 arch/arm64/include/asm/perf_event.h | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h
index 1621c84f44b3..d55a019cdf0a 100644
--- a/arch/arm64/include/asm/ftrace.h
+++ b/arch/arm64/include/asm/ftrace.h
@@ -153,7 +153,7 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs)
 	regs->pc = afregs->pc;
 	regs->regs[29] = afregs->fp;
 	regs->regs[30] = afregs->lr;
-	regs->pstate = PSR_MODE_EL1h;
+	regs->pstate = PSR_MODE_EL1t;
 	return regs;
 }
 
@@ -161,7 +161,7 @@ ftrace_partial_regs(const struct ftrace_regs *fregs, struct pt_regs *regs)
 		(_regs)->pc = arch_ftrace_regs(fregs)->pc;			\
 		(_regs)->regs[29] = arch_ftrace_regs(fregs)->fp;		\
 		(_regs)->sp = arch_ftrace_regs(fregs)->sp;			\
-		(_regs)->pstate = PSR_MODE_EL1h;		\
+		(_regs)->pstate = PSR_MODE_EL1t;		\
 	} while (0)
 
 int ftrace_regs_query_register_offset(const char *name);
diff --git a/arch/arm64/include/asm/perf_event.h b/arch/arm64/include/asm/perf_event.h
index ee45b4e77347..b3e7c24a5aab 100644
--- a/arch/arm64/include/asm/perf_event.h
+++ b/arch/arm64/include/asm/perf_event.h
@@ -17,7 +17,7 @@
 	(regs)->pc = (__ip);    \
 	(regs)->regs[29] = (unsigned long) __builtin_frame_address(0); \
 	(regs)->sp = current_stack_pointer; \
-	(regs)->pstate = PSR_MODE_EL1h;	\
+	(regs)->pstate = PSR_MODE_EL1t;	\
 }
 
 #endif
-- 
2.55.0.979.g7e5102b832-goog

[PATCH 21/21] arm64: Rename 'overflow_stack' and OVERFLOW_STACK_SIZE

[Will Deacon]

Now that the kernel "overflow" stack resides in SP_EL1, it is the active
stack for any exception taken from the kernel. Even though we currently
switch immediately to the kernel stack in the EL1t entry code, rename
the "overflow_stack" to "kernel_exception_stack" to better reflect its
usage. Introduce NVHE_OVERFLOW_STACK_SIZE to keep the EL2 overflow stack
size distinct from the EL1 kernel exception stack size.

Signed-off-by: Will Deacon <will@kernel.org>
---
 arch/arm64/include/asm/memory.h          | 10 ++++++----
 arch/arm64/include/asm/stacktrace.h      |  9 +++++----
 arch/arm64/include/asm/stacktrace/nvhe.h |  3 +--
 arch/arm64/kernel/entry-common.c         | 18 +++++++++---------
 arch/arm64/kernel/entry.S                |  2 +-
 arch/arm64/kernel/head.S                 |  2 +-
 arch/arm64/kernel/smp.c                  |  9 +++++----
 arch/arm64/kernel/stacktrace.c           |  2 +-
 arch/arm64/kernel/traps.c                | 14 +++++++-------
 arch/arm64/kvm/hyp/nvhe/host.S           |  2 +-
 arch/arm64/kvm/hyp/nvhe/stacktrace.c     |  4 ++--
 arch/arm64/kvm/stacktrace.c              |  4 ++--
 12 files changed, 41 insertions(+), 38 deletions(-)

diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index a2b7a33966ff..93ce6ef65573 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -139,10 +139,11 @@
 
 #define IRQ_STACK_SIZE		THREAD_SIZE
 
-#define OVERFLOW_STACK_SIZE	SZ_4K
+#define KERNEL_EXC_STACK_SIZE	SZ_4K
 
-#define NVHE_STACK_SHIFT       PAGE_SHIFT
-#define NVHE_STACK_SIZE        (UL(1) << NVHE_STACK_SHIFT)
+#define NVHE_OVERFLOW_STACK_SIZE	SZ_4K
+#define NVHE_STACK_SHIFT		PAGE_SHIFT
+#define NVHE_STACK_SIZE			(UL(1) << NVHE_STACK_SHIFT)
 
 /*
  * With the minimum frame size of [x29, x30], exactly half the combined
@@ -150,7 +151,8 @@
  * save the unwinded stacktrace; plus an additional entry to delimit the
  * end.
  */
-#define NVHE_STACKTRACE_SIZE	((OVERFLOW_STACK_SIZE + NVHE_STACK_SIZE) / 2 + sizeof(long))
+#define NVHE_STACKTRACE_SIZE	((NVHE_OVERFLOW_STACK_SIZE + NVHE_STACK_SIZE) \
+				 / 2 + sizeof(long))
 
 /*
  * Alignment of kernel segments (e.g. .text, .data).
diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h
index 6d3280932bf5..eb18d7cee62f 100644
--- a/arch/arm64/include/asm/stacktrace.h
+++ b/arch/arm64/include/asm/stacktrace.h
@@ -59,12 +59,13 @@ static inline bool on_task_stack(const struct task_struct *tsk,
 
 #define on_thread_stack()	(on_task_stack(current, current_stack_pointer, 1))
 
-DECLARE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);
+DECLARE_PER_CPU(unsigned long [KERNEL_EXC_STACK_SIZE/sizeof(long)],
+		kernel_exception_stack);
 
-static inline struct stack_info stackinfo_get_overflow(void)
+static inline struct stack_info stackinfo_get_kernel_exception(void)
 {
-	unsigned long low = (unsigned long)raw_cpu_ptr(overflow_stack);
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long low = (unsigned long)raw_cpu_ptr(kernel_exception_stack);
+	unsigned long high = low + KERNEL_EXC_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,
diff --git a/arch/arm64/include/asm/stacktrace/nvhe.h b/arch/arm64/include/asm/stacktrace/nvhe.h
index 171f9edef49f..629dd2d2caa3 100644
--- a/arch/arm64/include/asm/stacktrace/nvhe.h
+++ b/arch/arm64/include/asm/stacktrace/nvhe.h
@@ -44,8 +44,7 @@ static inline void kvm_nvhe_unwind_init(struct unwind_state *state,
  * In non-protected mode, the unwinding is done from kernel proper context
  * (by the host in EL1).
  */
-
-DECLARE_KVM_NVHE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);
+DECLARE_KVM_NVHE_PER_CPU(unsigned long [NVHE_OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack);
 DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info);
 DECLARE_PER_CPU(unsigned long, kvm_arm_hyp_stack_base);
 
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 9d734cd09f62..7e9b26cf1bdc 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -327,43 +327,43 @@ static void debug_exception_exit(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(debug_exception_exit);
 
-static void noinstr el1h_64_check_overflow_stack(struct pt_regs *regs)
+static void noinstr el1h_64_check_exception_stack(struct pt_regs *regs)
 {
 	unsigned long sp = kernel_stack_pointer(regs) - sizeof(*regs);
-	unsigned long ovf_stack = (unsigned long)this_cpu_ptr(overflow_stack);
+	unsigned long exc_stack = (unsigned long)this_cpu_ptr(kernel_exception_stack);
 
 	/*
-	 * We're in big trouble if we've overflowed the overflow stack
+	 * We're in big trouble if we've overflowed the exception stack
 	 * so perform a best-effort check before we proceed. If our SP
-	 * is outside of the overflow stack for this CPU then presumably
+	 * is outside of the exceptoin stack for this CPU then presumably
 	 * we're already corrupting memory, so park ourselves here in an
 	 * attempt to contain the damage.
 	 */
-	if (sp < ovf_stack || sp > ovf_stack + OVERFLOW_STACK_SIZE)
+	if (sp < exc_stack || sp > exc_stack + KERNEL_EXC_STACK_SIZE)
 		cpu_park_loop();
 }
 
 asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_sync_handler(regs);
 }
 
 asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_irq_handler(regs);
 }
 
 asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_fiq_handler(regs);
 }
 
 asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 {
-	el1h_64_check_overflow_stack(regs);
+	el1h_64_check_exception_stack(regs);
 	el1t_64_error_handler(regs);
 }
 
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index afcd84510daf..a31ef890a2ee 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -207,7 +207,7 @@ alternative_cb_end
 	mrs	x21, sp_el0
 	ldr_this_cpu	tsk, __entry_task, x20
 	msr	tpidrro_el0, tsk
-	adr_this_cpu	x19, overflow_stack + OVERFLOW_STACK_SIZE, x20
+	adr_this_cpu	x19, kernel_exception_stack + KERNEL_EXC_STACK_SIZE, x20
 	mov	x20, sp
 	msr	sp_el0, x20
 	mov	sp, x19
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 7419e1d8b7ec..794cefaf8482 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -213,7 +213,7 @@ SYM_CODE_END(preserve_boot_args)
 
 	mov	\tmp1, sp
 	msr	sp_el0, \tmp1
-	adr_this_cpu	\tmp1, overflow_stack + OVERFLOW_STACK_SIZE, \tmp2
+	adr_this_cpu	\tmp1, kernel_exception_stack + KERNEL_EXC_STACK_SIZE, \tmp2
 	mov	sp, \tmp1
 	msr	spsel, #0
 	.endm
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index b2e778aae881..5aaeda5362f7 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -447,9 +447,10 @@ void __init smp_cpus_done(unsigned int max_cpus)
 	mark_linear_text_alias_ro();
 }
 
-static void __init update_boot_cpu_offset_and_overflow_stack(void)
+static void __init update_boot_cpu_offset_and_exception_stack(void)
 {
-	u64 ovf_sp = (u64)raw_cpu_ptr(overflow_stack) + OVERFLOW_STACK_SIZE;
+	u64 exc_sp = (u64)raw_cpu_ptr(kernel_exception_stack) +
+		     KERNEL_EXC_STACK_SIZE;
 
 	asm volatile(
 	"	msr	tpidr_el1, %1\n"
@@ -457,7 +458,7 @@ static void __init update_boot_cpu_offset_and_overflow_stack(void)
 	"	msr	spsel, #1\n"
 	"	mov	sp, %0\n" /* Update the overflow stack pointer */
 	"	msr	spsel, #0"
-	: "+r" (ovf_sp)
+	: "+r" (exc_sp)
 	: "r" (per_cpu_offset(0))
 	: "memory");
 }
@@ -469,7 +470,7 @@ void __init smp_prepare_boot_cpu(void)
 	 * setup_per_cpu_areas(), and CPU0's boot time per-cpu area will be
 	 * freed shortly, so we must move over to the runtime per-cpu area.
 	 */
-	update_boot_cpu_offset_and_overflow_stack();
+	update_boot_cpu_offset_and_exception_stack();
 
 	cpuinfo_store_boot_cpu();
 	setup_boot_cpu_features();
diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
index 3ebcf8c53fb0..8e65b81c7502 100644
--- a/arch/arm64/kernel/stacktrace.c
+++ b/arch/arm64/kernel/stacktrace.c
@@ -337,7 +337,7 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,
 	struct stack_info stacks[] = {
 		stackinfo_get_task(task),
 		STACKINFO_CPU(irq),
-		STACKINFO_CPU(overflow),
+		STACKINFO_CPU(kernel_exception),
 #if defined(CONFIG_ARM_SDE_INTERFACE)
 		STACKINFO_SDEI(normal),
 		STACKINFO_SDEI(critical),
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 914282016069..ce0e213d772d 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -897,14 +897,14 @@ void bad_el0_sync(struct pt_regs *regs, int reason, unsigned long esr)
 			      "Bad EL0 synchronous exception");
 }
 
-DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)
-	__aligned(16);
+DEFINE_PER_CPU(unsigned long [KERNEL_EXC_STACK_SIZE/sizeof(long)],
+	       kernel_exception_stack) __aligned(16);
 
 void __noreturn panic_bad_stack(struct pt_regs *regs, unsigned long esr, unsigned long far)
 {
 	unsigned long tsk_stk = (unsigned long)current->stack;
 	unsigned long irq_stk = (unsigned long)this_cpu_read(irq_stack_ptr);
-	unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack);
+	unsigned long exc_stk = (unsigned long)this_cpu_ptr(kernel_exception_stack);
 
 	console_verbose();
 	pr_emerg("Insufficient stack space to handle exception!");
@@ -912,12 +912,12 @@ void __noreturn panic_bad_stack(struct pt_regs *regs, unsigned long esr, unsigne
 	pr_emerg("ESR: 0x%016lx -- %s\n", esr, esr_get_class_string(esr));
 	pr_emerg("FAR: 0x%016lx\n", far);
 
-	pr_emerg("Task stack:     [0x%016lx..0x%016lx]\n",
+	pr_emerg("Task stack:      [0x%016lx..0x%016lx]\n",
 		 tsk_stk, tsk_stk + THREAD_SIZE);
-	pr_emerg("IRQ stack:      [0x%016lx..0x%016lx]\n",
+	pr_emerg("IRQ stack:       [0x%016lx..0x%016lx]\n",
 		 irq_stk, irq_stk + IRQ_STACK_SIZE);
-	pr_emerg("Overflow stack: [0x%016lx..0x%016lx]\n",
-		 ovf_stk, ovf_stk + OVERFLOW_STACK_SIZE);
+	pr_emerg("Exception stack: [0x%016lx..0x%016lx]\n",
+		 exc_stk, exc_stk + KERNEL_EXC_STACK_SIZE);
 
 	__show_regs(regs);
 
diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S
index 9393fe3ea6a1..b05211a55074 100644
--- a/arch/arm64/kvm/hyp/nvhe/host.S
+++ b/arch/arm64/kvm/hyp/nvhe/host.S
@@ -205,7 +205,7 @@ SYM_FUNC_END(__host_hvc)
 
 .L__hyp_sp_overflow\@:
 	/* Switch to the overflow stack */
-	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0
+	adr_this_cpu sp, overflow_stack + NVHE_OVERFLOW_STACK_SIZE, x0
 
 	b	hyp_panic_bad_stack
 	ASM_BUG()
diff --git a/arch/arm64/kvm/hyp/nvhe/stacktrace.c b/arch/arm64/kvm/hyp/nvhe/stacktrace.c
index 7c832d60d22b..92fe8b6d4de0 100644
--- a/arch/arm64/kvm/hyp/nvhe/stacktrace.c
+++ b/arch/arm64/kvm/hyp/nvhe/stacktrace.c
@@ -9,7 +9,7 @@
 #include <asm/memory.h>
 #include <asm/percpu.h>
 
-DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)
+DEFINE_PER_CPU(unsigned long [NVHE_OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)
 	__aligned(16);
 
 DEFINE_PER_CPU(struct kvm_nvhe_stacktrace_info, kvm_stacktrace_info);
@@ -42,7 +42,7 @@ DEFINE_PER_CPU(unsigned long [NVHE_STACKTRACE_SIZE/sizeof(long)], pkvm_stacktrac
 static struct stack_info stackinfo_get_overflow(void)
 {
 	unsigned long low = (unsigned long)this_cpu_ptr(overflow_stack);
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long high = low + NVHE_OVERFLOW_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,
diff --git a/arch/arm64/kvm/stacktrace.c b/arch/arm64/kvm/stacktrace.c
index 9724c320126b..e5b878f959fc 100644
--- a/arch/arm64/kvm/stacktrace.c
+++ b/arch/arm64/kvm/stacktrace.c
@@ -27,7 +27,7 @@ static struct stack_info stackinfo_get_overflow(void)
 	struct kvm_nvhe_stacktrace_info *stacktrace_info
 				= this_cpu_ptr_nvhe_sym(kvm_stacktrace_info);
 	unsigned long low = (unsigned long)stacktrace_info->overflow_stack_base;
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long high = low + NVHE_OVERFLOW_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,
@@ -38,7 +38,7 @@ static struct stack_info stackinfo_get_overflow(void)
 static struct stack_info stackinfo_get_overflow_kern_va(void)
 {
 	unsigned long low = (unsigned long)this_cpu_ptr_nvhe_sym(overflow_stack);
-	unsigned long high = low + OVERFLOW_STACK_SIZE;
+	unsigned long high = low + NVHE_OVERFLOW_STACK_SIZE;
 
 	return (struct stack_info) {
 		.low = low,
-- 
2.55.0.979.g7e5102b832-goog

Re: [PATCH 19/21] arm64: entry: The great stack switcheroo

[Will Deacon]

On Mon, Sep 07, 2026 at 05:42:44PM +0100, Will Deacon wrote:
> With the kernel stack pointer in SP_EL1 and the overflow stack pointer
> in SP_EL0, it is now straightforward to switch between the two on
> exception entry from EL1 by writing to SPSel. However, since exception
> entry sets PSTATE.SP to 1 (selecting SP_EL1 as the stack pointer),
> repurposing the overflow stack as a more general kernel exception stack
> would require writing to SPSel on every exception entry from the kernel.
> 
> Switch things around so that the overflow stack resides in SP_EL1, with
> the kernel stack residing in SP_EL0.
> 
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
>  arch/arm64/include/asm/kvm_asm.h | 16 +++++++++++++++
>  arch/arm64/kernel/entry-common.c | 34 ++++++++++++++++----------------
>  arch/arm64/kernel/entry.S        | 26 +++++++++++++-----------
>  arch/arm64/kernel/head.S         |  5 ++++-
>  arch/arm64/kernel/smp.c          |  8 +++++---
>  arch/arm64/kvm/hyp/entry.S       |  3 +++
>  arch/arm64/mm/proc.S             |  3 +++
>  7 files changed, 63 insertions(+), 32 deletions(-)

[...]

> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 12aacc74f764..0811fa569100 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -98,6 +98,7 @@ SYM_FUNC_START(cpu_do_suspend)
>  	mrs	x10, oslsr_el1
>  	mrs	x11, sctlr_el1
>  	get_this_cpu_offset x12
> +	msr	spsel, #1
>  	mrs	x13, sp_el0
>  	stp	x2, x3, [x0]
>  	stp	x4, x5, [x0, #16]
> @@ -156,6 +157,8 @@ alternative_else_nop_endif
>  	msr	sctlr_el1, x12
>  	set_this_cpu_offset x13
>  	msr	sp_el0, x14
> +	msr	spsel, #0

Sashiko rightfully points out that I've got this slightly wrong because
I've pushed the spsel toggle too deep in the callchain for the
suspend/resume paths. My original rationale was that the current code
already saves/restores SP_EL{0,1}, so we shouldn't need to change
anything. However, that's not quite right because it's the caller (e.g.
__cpu_suspend_enter()) which saves SP_EL1 as 'sp', so the stack toggle
should happen there.

I'll fix that in v2.

Will

Re: [PATCH 06/21] arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0

[David Laight]

On Mon,  7 Sep 2026 17:42:31 +0100
Will Deacon <will@kernel.org> wrote:

> To make SP_EL0 available to point at the per-cpu kernel overflow stack,
> we must relocate 'current' elsewhere. With TPIDRRO_EL0 now restored late
> in the return-to-user path, we can otherwise use it to hold 'current'
> and update the various accessors accordingly.
> 
> Move 'current' from SP_EL0 to TPIDRRO_EL0.

Is it worth having a #define for the name of the register that stores 'current'?
(And probably the other similar registers.)
That would avoid the bisection problems with the following two patches
and might make other code a bit more readable.

David

> 
> Cc: Mark Rutland <mark.rutland@arm.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
>  arch/arm64/include/asm/assembler.h |  2 +-
>  arch/arm64/include/asm/current.h   |  6 +++---
>  arch/arm64/kernel/entry.S          | 20 ++++++++++----------
>  arch/arm64/kernel/head.S           |  2 +-
>  arch/arm64/kernel/process.c        |  5 +++--
>  arch/arm64/net/bpf_jit_comp.c      |  4 ++--
>  6 files changed, 20 insertions(+), 19 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 0b58b550e8dc..fd1ae5337935 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -587,7 +587,7 @@ alternative_else_nop_endif
>   * Return the current task_struct.
>   */
>  	.macro	get_current_task, rd
> -	mrs	\rd, sp_el0
> +	mrs	\rd, tpidrro_el0
>  	.endm
>  
>  /*
> diff --git a/arch/arm64/include/asm/current.h b/arch/arm64/include/asm/current.h
> index c92912eaf186..8c7efac05394 100644
> --- a/arch/arm64/include/asm/current.h
> +++ b/arch/arm64/include/asm/current.h
> @@ -14,11 +14,11 @@ struct task_struct;
>   */
>  static __always_inline struct task_struct *get_current(void)
>  {
> -	unsigned long sp_el0;
> +	unsigned long tpidrro_el0;
>  
> -	asm ("mrs %0, sp_el0" : "=r" (sp_el0));
> +	asm ("mrs %0, tpidrro_el0" : "=r" (tpidrro_el0));
>  
> -	return (struct task_struct *)sp_el0;
> +	return (struct task_struct *)tpidrro_el0;
>  }
>  
>  #define current get_current()
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index b5d8277f608a..a45be0a837c8 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -81,9 +81,9 @@
>  	/* Stash the original SP (minus PT_REGS_SIZE) in tpidr_el0. */
>  	msr	tpidr_el0, x0
>  
> -	/* Recover the original x0 value and stash it in tpidrro_el0 */
> +	/* Recover the original x0 value and stash it in sp_el0 */
>  	sub	x0, sp, x0
> -	msr	tpidrro_el0, x0
> +	msr	sp_el0, x0
>  
>  	/* Switch to the overflow stack */
>  	adr_this_cpu sp, overflow_stack + OVERFLOW_STACK_SIZE, x0
> @@ -99,7 +99,7 @@
>  
>  	/* We were already on the overflow stack. Restore sp/x0 and carry on. */
>  	sub	sp, sp, x0
> -	mrs	x0, tpidrro_el0
> +	mrs	x0, sp_el0
>  	b	el\el\ht\()_\regsize\()_\label
>  	.endif
>  .org .Lventry_start\@ + 128	// Did we overflow the ventry slot?
> @@ -226,7 +226,7 @@ alternative_cb_end
>  	clear_gp_regs
>  	mrs	x21, sp_el0
>  	ldr_this_cpu	tsk, __entry_task, x20
> -	msr	sp_el0, tsk
> +	msr	tpidrro_el0, tsk
>  
>  	/*
>  	 * Ensure MDSCR_EL1.SS is clear, since we can unmask debug exceptions
> @@ -548,7 +548,7 @@ SYM_CODE_START_LOCAL(__bad_stack)
>  	 */
>  
>  	/* Restore the original x0 value */
> -	mrs	x0, tpidrro_el0
> +	mrs	x0, sp_el0
>  
>  	/*
>  	 * Store the original GPRs to the new stack. The orginal SP (minus
> @@ -843,7 +843,7 @@ SYM_FUNC_START(cpu_switch_to)
>  	ldp	x29, x9, [x8], #16
>  	ldr	lr, [x8]
>  	mov	sp, x9
> -	msr	sp_el0, x1
> +	msr	tpidrro_el0, x1
>  	ptrauth_keys_install_kernel x1, x8, x9, x10
>  	scs_save x0
>  	scs_load_current
> @@ -1031,11 +1031,11 @@ SYM_CODE_START(__sdei_asm_handler)
>  
>  	/*
>  	 * We may have interrupted userspace, or a guest, or exit-from or
> -	 * return-to either of these. We can't trust sp_el0, restore it.
> +	 * return-to either of these. We can't trust tpidrro_el0, restore it.
>  	 */
> -	mrs	x28, sp_el0
> +	mrs	x28, tpidrro_el0
>  	ldr_this_cpu	dst=x0, sym=__entry_task, tmp=x1
> -	msr	sp_el0, x0
> +	msr	tpidrro_el0, x0
>  
>  	/* If we interrupted the kernel point to the previous stack/frame. */
>  	and     x0, x3, #0xc
> @@ -1051,7 +1051,7 @@ SYM_CODE_START(__sdei_asm_handler)
>  	mov	x1, x19
>  	bl	__sdei_handler
>  
> -	msr	sp_el0, x28
> +	msr	tpidrro_el0, x28
>  	/* restore regs >x17 that we clobbered */
>  	mov	x4, x19         // keep x4 for __sdei_asm_exit_trampoline
>  	ldp	x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14]
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 87a822e5c4ca..c33551ee6572 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -193,7 +193,7 @@ SYM_CODE_END(preserve_boot_args)
>  	 * for consistency with user tasks and kthreads.
>  	 */
>  	.macro	init_cpu_task tsk, tmp1, tmp2
> -	msr	sp_el0, \tsk
> +	msr	tpidrro_el0, \tsk
>  
>  	ldr	\tmp1, [\tsk, #TSK_STACK]
>  	add	sp, \tmp1, #THREAD_SIZE
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index bfdc12166895..79a19be88ced 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -551,8 +551,9 @@ static void ssbs_thread_switch(struct task_struct *next)
>  }
>  
>  /*
> - * We store our current task in sp_el0, which is clobbered by userspace. Keep a
> - * shadow copy so that we can restore this upon entry from userspace.
> + * We store our current task in tpidrro_el0, which is clobbered when returning
> + * to userspace. Keep a shadow copy so that we can restore this upon entry from
> + * userspace.
>   *
>   * This is *only* for exception entry from EL0, and is not valid until we
>   * __switch_to() a user task.
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index c18e005a41db..51d853b92443 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
> @@ -1694,7 +1694,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
>  		if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
>  			cpu_offset = offsetof(struct thread_info, cpu);
>  
> -			emit(A64_MRS_SP_EL0(tmp), ctx);
> +			emit(A64_MRS_TPIDRRO_EL0(tmp), ctx);
>  			if (is_lsi_offset(cpu_offset, 2)) {
>  				emit(A64_LDR32I(r0, tmp, cpu_offset), ctx);
>  			} else {
> @@ -1707,7 +1707,7 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
>  		/* Implement helper call to bpf_get_current_task/_btf() inline */
>  		if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
>  					   insn->imm == BPF_FUNC_get_current_task_btf)) {
> -			emit(A64_MRS_SP_EL0(r0), ctx);
> +			emit(A64_MRS_TPIDRRO_EL0(r0), ctx);
>  			break;
>  		}
>  

Re: [PATCH 00/21] arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0

[Vladimir Murzin]

Hi Will,

On 9/7/26 17:42, Will Deacon wrote:
> Hi everyone,
> 
> This series is a bit of a complicated juggling act that, on its own,
> doesn't achieve an awful lot. However, it lays the ground work for
> sizing the kernel stack at runtime, e.g. via a cmdline option or even
> potentially on a per-task basis and so I would like to work towards
> getting it merged independently.
> 
> The series is based on v7.3-rc1 and structured as follows:
> 
>   * The first 9 patches move the 'current' task pointer from SP_EL0
>     to TPIDRRO_EL0.
> 
>   * The following 8 patches point the newly-freed SP_EL0 at the overflow
>     stack and switch to it explicitly when we detect a kernel stack
>     overflow.
> 
>   * The final 4 patches turn everything on its head, so that the
>     overflow stack and kernel stack are swapped, with the former now
>     residing in SP_EL1 and the latter in SP_EL0.
> 
> At the end of all that, when we take an exception from EL1, we are
> immediately transitioned to the overflow stack (now renamed "exception
> stack") and can push registers right away. This also means that using
> SPINTMASK to control NMI masking becomes a possibility, although
> speaking to Mark, Vladimir and Ada, they all seem to prefer ALLINT.
> 
> Mostafa will soon post a follow-up series that allows the kernel stack
> size to be specified on the kernel cmdline, which we are hoping to use
> to configure an 8k stack size in Android. I will be talking more about
> all of this at LPC in the Memory Management MC:
> 
>   https://lpc.events/event/20/contributions/2419/
> 
> Cheers,
> 
> Will
> 
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Ard Biesheuvel <ardb@kernel.org>
> Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Cc: David Hildenbrand <david@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Vladimir Murzin <vladimir.murzin@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Mostafa Saleh <smostafa@google.com>
> Cc: Lorenzo Stoakes <ljs@kernel.org>
> Cc: Oliver Upton <oupton@kernel.org>
> Cc: Linus Walleij <linusw@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> 
I gave it a try and I observe splat:

Unable to handle kernel execute from non-executable memory at virtual address ffff000970e81148
Mem abort info:
  ESR = 0x000000008600000f
  EC = 0x21: IABT (current EL), IL = 32 bits
  SET = 0, FnV = 0
  EA = 0, S1PTW = 0
  FSC = 0x0f: level 3 permission fault
swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008129f000
[ffff000970e81148] pgd=0000000000000000, p4d=18000009f1dff403, pud=18000009f1079403, pmd=18000009f0ef1403, pte=00e80009f0e81707
Internal error: Oops: 000000008600000f [#1]  SMP
Modules linked in:
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.3.0-rc2-7fb054f87+ #3627 PREEMPT(lazy) 
Hardware name: Generated (DT)
pstate: 1634023c9 (nZCv DAIF +ALLINT +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
pc : 0xffff000970e81148
lr : 0xffff000970e81148
sp : ffff000970e81150
x29: ffff800080f85fc0 x28: 0000000000000001 x27: 0000000000002000
x26: 0000000000000000 x25: 00000000000000c0 x24: 0000000040000023
x23: ffff800080c0e1d8 x22: 00000000000003c0 x21: ffff800080b58300
x20: cfa2800080b58300 x19: 00000000200003c0 x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000028
x14: 0000000000000000 x13: ffff8000814f3ac0 x12: ffff8008efcac000
x11: b2000c3eb474d99d x10: 0000000000000008 x9 : 0000000000001000
x8 : ffff800080010800 x7 : 055001f2b5503510 x6 : 0000000001310000
x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff800081502d80
x2 : 0000000000000802 x1 : ffff000800106a70 x0 : 0000000000000001
Call trace:
 0xffff000970e81148 (P)
Code: 00000000 00000000 00000000 00000000 (00000002) 
---[ end trace 0000000000000000 ]---
Kernel panic - not syncing: Oops: Fatal exception
SMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x0,00000000,034bfc7d,ff728b42,7ffce667
Memory Limit: none
---[ end Kernel panic - not syncing: Oops: Fatal exception ]---

I suspect it is related to power management, since it can be triggered
with the sleep command, though I haven't debugged it. I noticed that
Sashiko has reported issues related to suspend/resume, so if you
provide fixups for the relevant commits, I can give them another
try. Otherwise, I'll wait for v2 :)

Cheers
Vladimir

Re: [PATCH 00/21] arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0

[Will Deacon]

Hi Vladimir,

On Wed, Sep 09, 2026 at 11:39:24AM +0100, Vladimir Murzin wrote:
> On 9/7/26 17:42, Will Deacon wrote:
> > This series is a bit of a complicated juggling act that, on its own,
> > doesn't achieve an awful lot. However, it lays the ground work for
> > sizing the kernel stack at runtime, e.g. via a cmdline option or even
> > potentially on a per-task basis and so I would like to work towards
> > getting it merged independently.

[...]

> I gave it a try and I observe splat:

Thanks for taking it for a spin!

> Unable to handle kernel execute from non-executable memory at virtual address ffff000970e81148
> Mem abort info:
>   ESR = 0x000000008600000f
>   EC = 0x21: IABT (current EL), IL = 32 bits
>   SET = 0, FnV = 0
>   EA = 0, S1PTW = 0
>   FSC = 0x0f: level 3 permission fault
> swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008129f000
> [ffff000970e81148] pgd=0000000000000000, p4d=18000009f1dff403, pud=18000009f1079403, pmd=18000009f0ef1403, pte=00e80009f0e81707
> Internal error: Oops: 000000008600000f [#1]  SMP
> Modules linked in:
> CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.3.0-rc2-7fb054f87+ #3627 PREEMPT(lazy) 
> Hardware name: Generated (DT)
> pstate: 1634023c9 (nZCv DAIF +ALLINT +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> pc : 0xffff000970e81148
> lr : 0xffff000970e81148
> sp : ffff000970e81150
> x29: ffff800080f85fc0 x28: 0000000000000001 x27: 0000000000002000
> x26: 0000000000000000 x25: 00000000000000c0 x24: 0000000040000023
> x23: ffff800080c0e1d8 x22: 00000000000003c0 x21: ffff800080b58300
> x20: cfa2800080b58300 x19: 00000000200003c0 x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000028
> x14: 0000000000000000 x13: ffff8000814f3ac0 x12: ffff8008efcac000
> x11: b2000c3eb474d99d x10: 0000000000000008 x9 : 0000000000001000
> x8 : ffff800080010800 x7 : 055001f2b5503510 x6 : 0000000001310000
> x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff800081502d80
> x2 : 0000000000000802 x1 : ffff000800106a70 x0 : 0000000000000001
> Call trace:
>  0xffff000970e81148 (P)
> Code: 00000000 00000000 00000000 00000000 (00000002) 
> ---[ end trace 0000000000000000 ]---
> Kernel panic - not syncing: Oops: Fatal exception
> SMP: stopping secondary CPUs
> Kernel Offset: disabled
> CPU features: 0x0,00000000,034bfc7d,ff728b42,7ffce667
> Memory Limit: none
> ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
> 
> I suspect it is related to power management, since it can be triggered
> with the sleep command, though I haven't debugged it. I noticed that
> Sashiko has reported issues related to suspend/resume, so if you
> provide fixups for the relevant commits, I can give them another
> try. Otherwise, I'll wait for v2 :)

It's fiddly to envisage how we end up trying to execute from non-executable
memory, but there are two bugs in the suspend/resume code:

1. I don't save/restore the stack pointers correctly (I suppose this could
   explain almost any crash, tbh)

2. I don't restore the pauth keys properly

I've hacked up an untested diff below, please can you take it for a spin?

Cheers,

Will

--->8

diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
index f093cdf71be1..facf3f1cc3b1 100644
--- a/arch/arm64/kernel/sleep.S
+++ b/arch/arm64/kernel/sleep.S
@@ -133,7 +133,8 @@ SYM_FUNC_START(_cpu_resume)
 	add	x0, x0, #SLEEP_STACK_DATA_SYSTEM_REGS
 	/* load sp from context */
 	ldr	x2, [x0, #CPU_CTX_SP]
-	mov	sp, x2
+	msr	sp_el0, x2
+
 	/*
 	 * cpu_do_resume expects x0 to contain context address pointer
 	 */
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 0811fa569100..bec7f858fae9 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -87,6 +87,7 @@
  * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>.
  */
 SYM_FUNC_START(cpu_do_suspend)
+	msr	spsel, #1
 	mrs	x2, tpidr_el0
 	mrs	x3, tpidrro_el0
 	mrs	x4, contextidr_el1
@@ -98,8 +99,7 @@ SYM_FUNC_START(cpu_do_suspend)
 	mrs	x10, oslsr_el1
 	mrs	x11, sctlr_el1
 	get_this_cpu_offset x12
-	msr	spsel, #1
-	mrs	x13, sp_el0
+	mov	x13, sp				// SP_EL1
 	stp	x2, x3, [x0]
 	stp	x4, x5, [x0, #16]
 	stp	x6, x7, [x0, #32]
@@ -115,6 +115,7 @@ alternative_if ARM64_HAS_TCR2
 	mrs	x2, REG_TCR2_EL1
 	str	x2, [x0, #104]
 alternative_else_nop_endif
+	msr	spsel, #0
 	ret
 SYM_FUNC_END(cpu_do_suspend)
 
@@ -122,6 +123,8 @@ SYM_FUNC_END(cpu_do_suspend)
  * cpu_do_resume - restore CPU register context
  *
  * x0: Address of context pointer
+ *
+ * Entered with SPSel == 1, returns with SPSel == 0.
  */
 SYM_FUNC_START(cpu_do_resume)
 	ldp	x2, x3, [x0]
@@ -130,6 +133,10 @@ SYM_FUNC_START(cpu_do_resume)
 	ldp	x9, x10, [x0, #48]
 	ldp	x11, x12, [x0, #64]
 	ldp	x13, x14, [x0, #80]
+
+	/* Move 'current' somewhere safe */
+	mov	x15, x3
+
 	/*
 	 * Restore x18, as it may be used as a platform register, and clear
 	 * the buffer to minimize the risk of exposure when used for shadow
@@ -156,8 +163,7 @@ alternative_else_nop_endif
 
 	msr	sctlr_el1, x12
 	set_this_cpu_offset x13
-	msr	sp_el0, x14
-	msr	spsel, #0
+	mov	sp, x14				// SP_EL1
 
 	/*
 	 * Restore oslsr_el1 by writing oslar_el1
@@ -179,8 +185,9 @@ alternative_if ARM64_HAS_GIC_PRIO_MASKING
 alternative_else_nop_endif
 #endif
 
-	ptrauth_keys_install_kernel_nosync x14, x1, x2, x3
+	ptrauth_keys_install_kernel_nosync x15, x1, x2, x3
 	isb
+	msr	spsel, #0
 	ret
 SYM_FUNC_END(cpu_do_resume)
 #endif

Re: [PATCH 00/21] arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0

[Vladimir Murzin]

Hi Will,

On 9/9/26 12:29, Will Deacon wrote:
> Hi Vladimir,
> 
> On Wed, Sep 09, 2026 at 11:39:24AM +0100, Vladimir Murzin wrote:
>> On 9/7/26 17:42, Will Deacon wrote:
>>> This series is a bit of a complicated juggling act that, on its own,
>>> doesn't achieve an awful lot. However, it lays the ground work for
>>> sizing the kernel stack at runtime, e.g. via a cmdline option or even
>>> potentially on a per-task basis and so I would like to work towards
>>> getting it merged independently.
> [...]
> 
>> I gave it a try and I observe splat:
> Thanks for taking it for a spin!
> 
>> Unable to handle kernel execute from non-executable memory at virtual address ffff000970e81148
>> Mem abort info:
>>   ESR = 0x000000008600000f
>>   EC = 0x21: IABT (current EL), IL = 32 bits
>>   SET = 0, FnV = 0
>>   EA = 0, S1PTW = 0
>>   FSC = 0x0f: level 3 permission fault
>> swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008129f000
>> [ffff000970e81148] pgd=0000000000000000, p4d=18000009f1dff403, pud=18000009f1079403, pmd=18000009f0ef1403, pte=00e80009f0e81707
>> Internal error: Oops: 000000008600000f [#1]  SMP
>> Modules linked in:
>> CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.3.0-rc2-7fb054f87+ #3627 PREEMPT(lazy) 
>> Hardware name: Generated (DT)
>> pstate: 1634023c9 (nZCv DAIF +ALLINT +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
>> pc : 0xffff000970e81148
>> lr : 0xffff000970e81148
>> sp : ffff000970e81150
>> x29: ffff800080f85fc0 x28: 0000000000000001 x27: 0000000000002000
>> x26: 0000000000000000 x25: 00000000000000c0 x24: 0000000040000023
>> x23: ffff800080c0e1d8 x22: 00000000000003c0 x21: ffff800080b58300
>> x20: cfa2800080b58300 x19: 00000000200003c0 x18: 0000000000000000
>> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000028
>> x14: 0000000000000000 x13: ffff8000814f3ac0 x12: ffff8008efcac000
>> x11: b2000c3eb474d99d x10: 0000000000000008 x9 : 0000000000001000
>> x8 : ffff800080010800 x7 : 055001f2b5503510 x6 : 0000000001310000
>> x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff800081502d80
>> x2 : 0000000000000802 x1 : ffff000800106a70 x0 : 0000000000000001
>> Call trace:
>>  0xffff000970e81148 (P)
>> Code: 00000000 00000000 00000000 00000000 (00000002) 
>> ---[ end trace 0000000000000000 ]---
>> Kernel panic - not syncing: Oops: Fatal exception
>> SMP: stopping secondary CPUs
>> Kernel Offset: disabled
>> CPU features: 0x0,00000000,034bfc7d,ff728b42,7ffce667
>> Memory Limit: none
>> ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
>>
>> I suspect it is related to power management, since it can be triggered
>> with the sleep command, though I haven't debugged it. I noticed that
>> Sashiko has reported issues related to suspend/resume, so if you
>> provide fixups for the relevant commits, I can give them another
>> try. Otherwise, I'll wait for v2 :)
> It's fiddly to envisage how we end up trying to execute from non-executable
> memory, but there are two bugs in the suspend/resume code:
> 
> 1. I don't save/restore the stack pointers correctly (I suppose this could
>    explain almost any crash, tbh)
> 
> 2. I don't restore the pauth keys properly
> 
> I've hacked up an untested diff below, please can you take it for a spin?
> 

With fixup applied I do not see splat anymore :) 

Thanks
Vladimir

> Cheers,
> 
> Will
> 
> --->8
> 
> diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
> index f093cdf71be1..facf3f1cc3b1 100644
> --- a/arch/arm64/kernel/sleep.S
> +++ b/arch/arm64/kernel/sleep.S
> @@ -133,7 +133,8 @@ SYM_FUNC_START(_cpu_resume)
>  	add	x0, x0, #SLEEP_STACK_DATA_SYSTEM_REGS
>  	/* load sp from context */
>  	ldr	x2, [x0, #CPU_CTX_SP]
> -	mov	sp, x2
> +	msr	sp_el0, x2
> +
>  	/*
>  	 * cpu_do_resume expects x0 to contain context address pointer
>  	 */
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 0811fa569100..bec7f858fae9 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -87,6 +87,7 @@
>   * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>.
>   */
>  SYM_FUNC_START(cpu_do_suspend)
> +	msr	spsel, #1
>  	mrs	x2, tpidr_el0
>  	mrs	x3, tpidrro_el0
>  	mrs	x4, contextidr_el1
> @@ -98,8 +99,7 @@ SYM_FUNC_START(cpu_do_suspend)
>  	mrs	x10, oslsr_el1
>  	mrs	x11, sctlr_el1
>  	get_this_cpu_offset x12
> -	msr	spsel, #1
> -	mrs	x13, sp_el0
> +	mov	x13, sp				// SP_EL1
>  	stp	x2, x3, [x0]
>  	stp	x4, x5, [x0, #16]
>  	stp	x6, x7, [x0, #32]
> @@ -115,6 +115,7 @@ alternative_if ARM64_HAS_TCR2
>  	mrs	x2, REG_TCR2_EL1
>  	str	x2, [x0, #104]
>  alternative_else_nop_endif
> +	msr	spsel, #0
>  	ret
>  SYM_FUNC_END(cpu_do_suspend)
>  
> @@ -122,6 +123,8 @@ SYM_FUNC_END(cpu_do_suspend)
>   * cpu_do_resume - restore CPU register context
>   *
>   * x0: Address of context pointer
> + *
> + * Entered with SPSel == 1, returns with SPSel == 0.
>   */
>  SYM_FUNC_START(cpu_do_resume)
>  	ldp	x2, x3, [x0]
> @@ -130,6 +133,10 @@ SYM_FUNC_START(cpu_do_resume)
>  	ldp	x9, x10, [x0, #48]
>  	ldp	x11, x12, [x0, #64]
>  	ldp	x13, x14, [x0, #80]
> +
> +	/* Move 'current' somewhere safe */
> +	mov	x15, x3
> +
>  	/*
>  	 * Restore x18, as it may be used as a platform register, and clear
>  	 * the buffer to minimize the risk of exposure when used for shadow
> @@ -156,8 +163,7 @@ alternative_else_nop_endif
>  
>  	msr	sctlr_el1, x12
>  	set_this_cpu_offset x13
> -	msr	sp_el0, x14
> -	msr	spsel, #0
> +	mov	sp, x14				// SP_EL1
>  
>  	/*
>  	 * Restore oslsr_el1 by writing oslar_el1
> @@ -179,8 +185,9 @@ alternative_if ARM64_HAS_GIC_PRIO_MASKING
>  alternative_else_nop_endif
>  #endif
>  
> -	ptrauth_keys_install_kernel_nosync x14, x1, x2, x3
> +	ptrauth_keys_install_kernel_nosync x15, x1, x2, x3
>  	isb
> +	msr	spsel, #0
>  	ret
>  SYM_FUNC_END(cpu_do_resume)
>  #endif
> 

Re: [PATCH 01/21] arm64: entry: Defer setting of TPIDRRO_EL0 until exit to userspace

[Jinjie Ruan]

在 2026/9/8 0:42, Will Deacon 写道:
> In preparation for using TPIDRRO_EL0 to point at 'current' while running
> inside the kernel, defer its userspace initialisation from the
> context-switch patch to exception return. This has the added benefit of
> not having to worry about keeping the in-memory value and the register
> value in-sync during preemptible sections.
> 
> Cc: Mark Rutland <mark.rutland@arm.com>
> Signed-off-by: Will Deacon <will@kernel.org>
> ---
>  arch/arm64/kernel/asm-offsets.c |  1 +
>  arch/arm64/kernel/entry.S       |  4 +++-
>  arch/arm64/kernel/process.c     | 16 +---------------
>  arch/arm64/kernel/sys_compat.c  |  7 -------
>  4 files changed, 5 insertions(+), 23 deletions(-)
> 
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 9c853ed3ceab..6038ab3beb25 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -42,6 +42,7 @@ int main(void)
>  #endif
>    BLANK();
>    DEFINE(THREAD_CPU_CONTEXT,	offsetof(struct task_struct, thread.cpu_context));
> +  DEFINE(THREAD_TP_VALUE,	offsetof(struct task_struct, thread.uw.tp_value));
>    DEFINE(THREAD_SCTLR_USER,	offsetof(struct task_struct, thread.sctlr_user));
>  #ifdef CONFIG_ARM64_PTR_AUTH
>    DEFINE(THREAD_KEYS_USER,	offsetof(struct task_struct, thread.keys_user));
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index f63049ac32dc..59f045e496ec 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -47,7 +47,6 @@
>  	b	.Lskip_tramp_vectors_cleanup\@
>  	.if	\regsize == 64
>  	mrs	x30, tpidrro_el0
> -	msr	tpidrro_el0, xzr
>  	.else
>  	mov	x30, xzr
>  	.endif
> @@ -360,9 +359,12 @@ alternative_else_nop_endif
>  	.if	\el == 0
>  	ldr	x23, [sp, #S_SP]		// load return stack pointer
>  	msr	sp_el0, x23
> +	msr	tpidrro_el0, xzr
>  	tst	x22, #PSR_MODE32_BIT		// native task?
>  	b.eq	3f
>  
> +	ldr	x0, [tsk, #THREAD_TP_VALUE]
> +	msr	tpidrro_el0, x0

Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  #ifdef CONFIG_ARM64_ERRATUM_845719
>  alternative_if ARM64_WORKAROUND_845719
>  #ifdef CONFIG_PID_IN_CONTEXTIDR
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index 581f80e9b9b7..bfdc12166895 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -256,17 +256,8 @@ static void tls_thread_flush(void)
>  	if (system_supports_tpidr2())
>  		write_sysreg_s(0, SYS_TPIDR2_EL0);
>  
> -	if (is_compat_task()) {
> +	if (is_compat_task())
>  		current->thread.uw.tp_value = 0;
> -
> -		/*
> -		 * We need to ensure ordering between the shadow state and the
> -		 * hardware state, so that we don't corrupt the hardware state
> -		 * with a stale shadow state during context switch.
> -		 */
> -		barrier();
> -		write_sysreg(0, tpidrro_el0);
> -	}
>  }
>  
>  static void flush_tagged_addr_state(void)
> @@ -531,11 +522,6 @@ static void tls_thread_switch(struct task_struct *next)
>  {
>  	tls_preserve_current_state();
>  
> -	if (is_compat_thread(task_thread_info(next)))
> -		write_sysreg(next->thread.uw.tp_value, tpidrro_el0);
> -	else
> -		write_sysreg(0, tpidrro_el0);
> -
>  	write_sysreg(*task_user_tls(next), tpidr_el0);
>  	if (system_supports_tpidr2())
>  		write_sysreg_s(next->thread.tpidr2_el0, SYS_TPIDR2_EL0);
> diff --git a/arch/arm64/kernel/sys_compat.c b/arch/arm64/kernel/sys_compat.c
> index 0451f96c2c3f..59eee301c6f4 100644
> --- a/arch/arm64/kernel/sys_compat.c
> +++ b/arch/arm64/kernel/sys_compat.c
> @@ -89,13 +89,6 @@ long compat_arm_syscall(struct pt_regs *regs, int scno)
>  
>  	case __ARM_NR_compat_set_tls:
>  		current->thread.uw.tp_value = regs->regs[0];
> -
> -		/*
> -		 * Protect against register corruption from context switch.
> -		 * See comment in tls_thread_flush.
> -		 */
> -		barrier();
> -		write_sysreg(regs->regs[0], tpidrro_el0);
>  		return 0;
>  
>  	default:

Re: [PATCH 00/21] arm64: Move overflow sp into SP_EL1 and kernel sp into SP_EL0

[Will Deacon]

On Thu, Sep 10, 2026 at 02:49:37PM +0100, Vladimir Murzin wrote:
> On 9/9/26 12:29, Will Deacon wrote:
> > On Wed, Sep 09, 2026 at 11:39:24AM +0100, Vladimir Murzin wrote:
> > [...]
> > 
> >> I gave it a try and I observe splat:
>
> > I've hacked up an untested diff below, please can you take it for a spin?
> > 
> 
> With fixup applied I do not see splat anymore :) 

Excellent, thanks for testing it!

Will

Re: [PATCH 06/21] arm64: Store 'current' in TPIDRRO_EL0 instead of SP_EL0

[Will Deacon]

On Tue, Sep 08, 2026 at 02:19:50PM +0100, David Laight wrote:
> On Mon,  7 Sep 2026 17:42:31 +0100
> Will Deacon <will@kernel.org> wrote:
> 
> > To make SP_EL0 available to point at the per-cpu kernel overflow stack,
> > we must relocate 'current' elsewhere. With TPIDRRO_EL0 now restored late
> > in the return-to-user path, we can otherwise use it to hold 'current'
> > and update the various accessors accordingly.
> > 
> > Move 'current' from SP_EL0 to TPIDRRO_EL0.
> 
> Is it worth having a #define for the name of the register that stores 'current'?
> (And probably the other similar registers.)
> That would avoid the bisection problems with the following two patches
> and might make other code a bit more readable.

We don't tend to reallocate 'current' all that often, so I don't think it's
worth the indirection tbh. The following patches are to a GDB script and
a bpf selftest, so I'm not going to lose sleep about bisection.

Will