From: David Laight <david.laight.linux@gmail.com>
To: Usama Arif <usama.arif@linux.dev>
Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de,
dave.hansen@linux.intel.com, x86@kernel.org, kees@kernel.org,
hpa@zytor.com, gustavoars@kernel.org, ardb@kernel.org,
hca@linux.ibm.com, nathan@kernel.org, ndesaulniers@google.com,
morbo@google.com, justinstitt@google.com,
linux-hardening@vger.kernel.org, llvm@lists.linux.dev,
linux-kernel@vger.kernel.org, riel@surriel.com,
shakeel.butt@linux.dev, kernel-team@meta.com, rkeilty@meta.com,
atish.patra@linux.dev, dgreid@meta.com
Subject: Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
Date: Sat, 21 Nov 2026 18:22:08 +0000 [thread overview]
Message-ID: <20261121182208.19594445@pumpkin> (raw)
In-Reply-To: <ee4847f9-53ba-4f0a-9297-7ca8b6408f7f@linux.dev>
On Sat, 21 Nov 2026 13:57:39 +0000
Usama Arif <usama.arif@linux.dev> wrote:
> On 20/11/2026 10:55, David Laight wrote:
> > On Thu, 19 Nov 2026 13:31:08 -0800
> > Usama Arif <usama.arif@linux.dev> wrote:
> >
> >> With KSTACK_ERASE, every return to user space poisons the part of the
> >> task stack that was used since the previous return. The generic
> >> __stackleak_poison() does that one word per loop iteration, so a syscall
> >> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> >> running many small VMs, stackleak_erase() accounts for about 1.5% of
> >> non-idle host kernel cycles.
> >>
> >> Override __stackleak_poison() on x86-64, as s390 already does, through
> >> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> >> specific stackleak poison function"). Fill spans larger than 512 bytes
> >> with REP STOSQ and keep the word loop for shorter ones, where the fixed
> >> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> >> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> >> slower.
> >
> > As a matter of interest (and separately from the exact code) how did you
> > measure the performance to get the 300-400 byte gain point?
> >
>
> With a userspace benchmark pinned to one CPU. Each sample is one fill of
> a stack buffer straight after a getppid() syscall, with
> lfence;rdtsc;lfence around the fill only, an empty call subtracted, and
> the mean of 15 times taken. The size stayed fixed for a whole run,
> so the branches were well predicted. The plain loop and REP STOSQ
> cost the same at about 430 bytes on Zen 5c and 320 bytes on
> Cooper Lake.
The lfence (or mfence - which is the one you really need) take long
enough that they don't give meaningful timings for anything short.
I think they manage to overlap with the setup times for 'rep xxx'.
You also get the problem that the tsc doesn't measure clocks and a lot
of benchmarks run long enough that the cpu speed changes.
I've updated my benchmark code and run some tests on zen5.
This uses the PERF_COUNT_CPU_CYCLES counter but directly executing rdpmc.
I then use data dependencies from the original count to the test code,
and from the test result to the second rdpmc (which nicely has a register
number) to get a time that includes the test function with little overhead.
The clock count for an empty test function is 25, 32 with a memory read.
Each test is run 10 times and the clock count for each shown separately.
The first column includes the cost of the I-cache loads and any branches
that get mispredicted because (in these cases) the length changes.
This is most obvious for the libc calls.
memset() to 4k aligned buffer
0: len 16 ok 445 32 32 32 32 32 32 32 32 32 no_memset
5: len 16 ok 10538 107 52 52 52 52 52 52 52 52 libc
6: len 256 ok 193 101 48 48 51 44 51 44 44 44 libc
7: len 512 ok 120 83 51 51 51 51 51 51 51 51 libc
8: len 1024 ok 111 56 56 55 56 55 56 55 56 56 libc
9: len 2048 ok 117 71 72 72 72 72 72 72 72 72 libc
10: len 16 ok 90 78 74 55 54 55 54 55 54 55 stosb
11: len 256 ok 112 78 78 78 78 78 78 78 78 78 stosb
12: len 512 ok 80 80 80 80 80 80 80 87 80 80 stosb
13: len 1024 ok 90 95 89 95 89 88 89 95 89 95 stosb
14: len 2048 ok 105 111 104 104 104 111 104 111 104 111 stosb
15: len 16 ok 104 88 84 58 58 58 58 58 58 58 stosq
16: len 256 ok 125 80 80 80 80 80 80 80 80 80 stosq
17: len 512 ok 82 83 83 90 82 82 83 83 82 82 stosq
18: len 1024 ok 90 91 91 98 91 98 91 98 91 91 stosq
19: len 2048 ok 106 113 106 113 106 113 106 106 106 113 stosq
20: len 16 ok 133 65 57 33 33 26 26 26 26 26 longs
21: len 256 ok 159 152 80 42 42 42 42 42 42 42 longs
22: len 512 ok 187 115 88 58 56 57 57 57 57 57 longs
23: len 1024 ok 146 211 167 89 88 88 88 88 88 90 longs
24: len 2048 ok 213 213 209 167 153 155 153 153 155 152 longs
memcpy() between 4k aligned buffers
0: len 16 ok 119 63 59 33 33 33 33 33 33 33 no_copy
5: len 16 ok 7333 103 51 51 51 51 51 51 51 51 libc
6: len 256 ok 447 105 51 51 51 51 51 51 51 51 libc
7: len 512 ok 405 56 56 56 56 56 56 56 56 56 libc
8: len 1024 ok 421 96 66 65 64 66 64 70 64 70 libc
9: len 2048 ok 406 139 81 83 81 85 81 83 83 81 libc
10: len 16 ok 91 81 76 56 56 56 56 56 56 56 movsb
11: len 256 ok 115 77 76 77 76 77 76 77 76 77 movsb
12: len 512 ok 80 81 80 81 80 81 81 81 80 81 movsb
13: len 1024 ok 90 89 90 89 90 89 90 89 90 89 movsb
14: len 2048 ok 112 106 105 106 105 106 105 106 105 106 movsb
15: len 16 ok 108 87 85 65 57 65 57 65 57 65 movsq
16: len 256 ok 119 78 78 78 79 78 79 78 79 78 movsq
17: len 512 ok 81 83 82 83 82 82 82 83 82 82 movsq
18: len 1024 ok 91 91 91 91 91 91 91 91 91 91 movsq
19: len 2048 ok 107 107 107 107 107 107 107 107 107 107 movsq
20: len 16 ok 110 62 56 32 32 32 32 32 32 32 longs
21: len 256 ok 203 128 58 57 57 57 57 47 47 47 longs
22: len 512 ok 97 152 190 95 65 63 65 63 65 63 longs
23: len 1024 ok 226 185 155 100 99 98 100 98 99 98 longs
24: len 2048 ok 201 262 260 179 171 166 170 169 171 166 longs
I added enough barrier() and OPTIMIZER_HIDE_VAR() to get the 'longs'
loop to be sane - it sets/copies two 64bit locations each iteration.
Both tests manage a loop iteration every clock (on zen5).
The cutoff for 'rep stos' is pretty much 1k.
For memcpy() it is a bit lower.
'rep movsq' is slightly slower than 'rep movsb' because of the 'len /= 8'.
But it is much slower for counts of 1 (included above) and 0.
I need to boot up one of my other systems to see how older Intel cpu fair.
David
>
> > If the count is variable - likely in real life, you'll get a branch
> > misprediction penalty (about 20 clocks on zen5) every time you change
> > between the two choices.
> > The copy loop will have another mispredict penalty for the loop itself.
> >
> > So you are likely to get an overall improvement if you use 'rep stos'
> > for all lengths - even the ones for with it is slightly slower.
> > Unless, that is, you know at compile time that most of the length
> > will be short.
>
>
> I timed every stackleak_erase() in a Zen 5c guest running a
> futex, epoll, procfs and unix socket mix, with the variants switchable
> at runtime and alternated over 15 rounds:
>
> Unpatched 68.3 ns per erase
> REP STOSQ for all lengths 47.3 ns
> v1 / v2.1 (loop up to 512 B) 46.6 ns
> v2.2 (4 stores/iteration) 43.7 ns
>
> The spans aren't random. Unix socket and /proc/self/stat exits were over
> 1 KiB every time, while getppid(), epoll_wait(), read() and write() exits
> were under 128 bytes almost everytime.
>
> My preference still is v2.1 because of the simplicity.
>
> >
> > David
> >
> >>
> >> The fill is inline asm, so the noinstr function still makes no calls.
> >> memset64() would be the same instruction, but it is only static inline,
> >> and with KMSAN it is an out-of-line function.
> >>
> >> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> >> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> >> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> >> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> >>
> >> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> >> ---
> >> arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
> >> 1 file changed, 25 insertions(+)
> >>
> >> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> >> index 77488ff5eb8c7c..049231d484ad0e 100644
> >> --- a/arch/x86/include/asm/processor.h
> >> +++ b/arch/x86/include/asm/processor.h
> >> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
> >> current_stack_pointer) < THREAD_SIZE;
> >> }
> >>
> >> +#ifdef CONFIG_X86_64
> >> +#define __stackleak_poison __stackleak_poison
> >> +static __always_inline void __stackleak_poison(unsigned long erase_low,
> >> + unsigned long erase_high,
> >> + unsigned long poison)
> >> +{
> >> + unsigned long count;
> >> +
> >> + /* Short spans do not amortize the startup cost of REP STOSQ. */
> >> + if (erase_high <= erase_low + 512) {
> >> + while (erase_low < erase_high) {
> >> + *(unsigned long *)erase_low = poison;
> >> + erase_low += sizeof(unsigned long);
> >> + }
> >> + return;
> >> + }
> >> +
> >> + count = (erase_high - erase_low) / sizeof(unsigned long);
> >> + asm volatile("rep stosq"
> >> + : "+D" (erase_low), "+c" (count)
> >> + : "a" (poison)
> >> + : "memory");
> >> +}
> >> +#endif
> >> +
> >> #ifdef CONFIG_PARAVIRT_XXL
> >> #include <asm/paravirt.h>
> >> #else
> >
>
prev parent reply other threads:[~2026-11-21 18:22 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 22:30 ` Nick Desaulniers
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
2026-11-19 22:28 ` Nick Desaulniers
2026-11-20 11:38 ` David Laight
2026-11-20 12:42 ` David Laight
2026-11-20 18:46 ` Usama Arif
2026-11-20 21:44 ` David Laight
2026-11-20 10:45 ` Peter Zijlstra
2026-11-20 12:26 ` Peter Zijlstra
2026-11-20 18:43 ` Usama Arif
2026-11-20 22:22 ` David Laight
2026-11-21 14:38 ` Usama Arif
2026-11-20 14:02 ` David Laight
2026-11-20 15:35 ` Peter Zijlstra
2026-11-20 16:18 ` David Laight
2026-11-20 18:55 ` David Laight
2026-11-21 13:57 ` Usama Arif
2026-11-21 18:22 ` David Laight [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261121182208.19594445@pumpkin \
--to=david.laight.linux@gmail.com \
--cc=ardb@kernel.org \
--cc=atish.patra@linux.dev \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=dgreid@meta.com \
--cc=gustavoars@kernel.org \
--cc=hca@linux.ibm.com \
--cc=hpa@zytor.com \
--cc=justinstitt@google.com \
--cc=kees@kernel.org \
--cc=kernel-team@meta.com \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=llvm@lists.linux.dev \
--cc=mingo@redhat.com \
--cc=morbo@google.com \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=riel@surriel.com \
--cc=rkeilty@meta.com \
--cc=shakeel.butt@linux.dev \
--cc=tglx@kernel.org \
--cc=usama.arif@linux.dev \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®