* [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix
@ 2026-11-19 21:31 Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
0 siblings, 2 replies; 3+ messages in thread
From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, kees
Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo,
justinstitt, linux-hardening, llvm, linux-kernel, riel,
shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid,
Usama Arif
With KSTACK_ERASE, every return to user space poisons the part of the
task stack used since the previous return, one 8-byte store at a time.
On Meta hosts running many small VMs, that loop costs about 1.5% of
non-idle host kernel cycles. Most of it is on the way out of futex,
epoll, procfs and socket syscalls that go 1-2 KiB deep.
Patch 2 makes x86-64 use REP STOSQ for spans above 512 bytes, through
the arch hook that s390 already uses. Shorter spans keep the loop, and
the range and value written do not change.
Patch 1 is an independent build fix I found while testing patch 2 with a
small KSTACK_ERASE_TRACK_MIN_SIZE. The startup code still uses the
pre-rename DISABLE_STACKLEAK_PLUGIN, so with Clang, AMD memory
encryption and a tracking minimum below about 90 bytes, vmlinux fails
to link. It can be applied on its own.
Zen 5c KVM guest, one vCPU, before -> after:
stackleak_erase() per return to user 71 ns -> 46 ns
futex wake/wait ping-pong 5-7% faster
Unix socket sendmsg()/recvmsg() ping-pong 5-8% faster
open+read+close of /proc/self/stat 6-8% faster
getppid() (spans below 512 bytes) within 2%
Usama Arif (2):
x86/boot/startup: Disable kernel stack depth tracking
x86/kstack_erase: Use REP STOSQ for large stack fills
arch/x86/boot/startup/Makefile | 2 +-
arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
2 files changed, 26 insertions(+), 1 deletion(-)
--
2.53.1-Meta
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
@ 2026-11-19 21:31 ` Usama Arif
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
1 sibling, 0 replies; 3+ messages in thread
From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, kees
Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo,
justinstitt, linux-hardening, llvm, linux-kernel, riel,
shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid,
Usama Arif
Commit 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE")
renamed DISABLE_STACKLEAK_PLUGIN to DISABLE_KSTACK_ERASE everywhere
except in arch/x86/boot/startup/Makefile. The old name now expands to
nothing, so the startup code is built with stack depth tracking enabled.
The startup code is linked with every symbol prefixed by __pi_, so that
it cannot call into the rest of the kernel by accident. It has no alias
for the tracking callback, and tracking stack depth there is of no use.
At the default KSTACK_ERASE_TRACK_MIN_SIZE of 100 no startup function is
large enough to be tracked, so this went unnoticed. With Clang,
CONFIG_AMD_MEM_ENCRYPT=y and a minimum below about 90 bytes,
pvalidate_4k_page(), snp_cpuid() and do_vc_no_ghcb() get a call to
__sanitizer_cov_stack_depth() and vmlinux fails to link:
ld.lld: error: undefined symbol: __pi___sanitizer_cov_stack_depth
Use DISABLE_KSTACK_ERASE, which disables tracking with either compiler.
Fixes: 57fbad15c2ee ("stackleak: Rename STACKLEAK to KSTACK_ERASE")
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/x86/boot/startup/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/boot/startup/Makefile b/arch/x86/boot/startup/Makefile
index 5e499cfb29b5c0..0815105e7205c7 100644
--- a/arch/x86/boot/startup/Makefile
+++ b/arch/x86/boot/startup/Makefile
@@ -3,7 +3,7 @@
KBUILD_AFLAGS += -D__DISABLE_EXPORTS
KBUILD_CFLAGS += -D__DISABLE_EXPORTS -mcmodel=small -fPIC \
-Os -DDISABLE_BRANCH_PROFILING \
- $(DISABLE_STACKLEAK_PLUGIN) \
+ $(DISABLE_KSTACK_ERASE) \
$(DISABLE_LATENT_ENTROPY_PLUGIN) \
-fno-stack-protector -D__NO_FORTIFY \
-fno-jump-tables \
--
2.53.1-Meta
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
@ 2026-11-19 21:31 ` Usama Arif
1 sibling, 0 replies; 3+ messages in thread
From: Usama Arif @ 2026-11-19 21:31 UTC (permalink / raw)
To: tglx, mingo, bp, dave.hansen, x86, kees
Cc: hpa, gustavoars, ardb, hca, nathan, ndesaulniers, morbo,
justinstitt, linux-hardening, llvm, linux-kernel, riel,
shakeel.butt, kernel-team, rkeilty, atish.patra, dgreid,
Usama Arif
With KSTACK_ERASE, every return to user space poisons the part of the
task stack that was used since the previous return. The generic
__stackleak_poison() does that one word per loop iteration, so a syscall
that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
running many small VMs, stackleak_erase() accounts for about 1.5% of
non-idle host kernel cycles.
Override __stackleak_poison() on x86-64, as s390 already does, through
the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
specific stackleak poison function"). Fill spans larger than 512 bytes
with REP STOSQ and keep the word loop for shorter ones, where the fixed
startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
slower.
The fill is inline asm, so the noinstr function still makes no calls.
memset64() would be the same instruction, but it is only static inline,
and with KMSAN it is an out-of-line function.
In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c7c..049231d484ad0e 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
current_stack_pointer) < THREAD_SIZE;
}
+#ifdef CONFIG_X86_64
+#define __stackleak_poison __stackleak_poison
+static __always_inline void __stackleak_poison(unsigned long erase_low,
+ unsigned long erase_high,
+ unsigned long poison)
+{
+ unsigned long count;
+
+ /* Short spans do not amortize the startup cost of REP STOSQ. */
+ if (erase_high <= erase_low + 512) {
+ while (erase_low < erase_high) {
+ *(unsigned long *)erase_low = poison;
+ erase_low += sizeof(unsigned long);
+ }
+ return;
+ }
+
+ count = (erase_high - erase_low) / sizeof(unsigned long);
+ asm volatile("rep stosq"
+ : "+D" (erase_low), "+c" (count)
+ : "a" (poison)
+ : "memory");
+}
+#endif
+
#ifdef CONFIG_PARAVIRT_XXL
#include <asm/paravirt.h>
#else
--
2.53.1-Meta
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-19 21:32 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®