mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Usama Arif <usama.arif@linux.dev>
To: David Laight <david.laight.linux@gmail.com>,
	Nick Desaulniers <ndesaulniers@google.com>
Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de,
	dave.hansen@linux.intel.com, x86@kernel.org, kees@kernel.org,
	hpa@zytor.com, gustavoars@kernel.org, ardb@kernel.org,
	hca@linux.ibm.com, nathan@kernel.org, morbo@google.com,
	justinstitt@google.com, linux-hardening@vger.kernel.org,
	llvm@lists.linux.dev, linux-kernel@vger.kernel.org,
	riel@surriel.com, shakeel.butt@linux.dev, kernel-team@meta.com,
	rkeilty@meta.com, atish.patra@linux.dev, dgreid@meta.com
Subject: Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
Date: Fri, 20 Nov 2026 18:46:01 +0000	[thread overview]
Message-ID: <97e94491-ce19-45a1-8737-b30b49e997e8@linux.dev> (raw)
In-Reply-To: <20261120124252.14ff8a74@pumpkin>



On 20/11/2026 04:42, David Laight wrote:
> On Fri, 20 Nov 2026 11:38:25 +0000
> David Laight <david.laight.linux@gmail.com> wrote:
> 
>> On Thu, 19 Nov 2026 14:28:04 -0800
>> Nick Desaulniers <ndesaulniers@google.com> wrote:
>>
>>> On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@linux.dev> wrote:  
>>>>
>>>> With KSTACK_ERASE, every return to user space poisons the part of the
>>>> task stack that was used since the previous return. The generic
>>>> __stackleak_poison() does that one word per loop iteration, so a syscall
>>>> that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
>>>> running many small VMs, stackleak_erase() accounts for about 1.5% of
>>>> non-idle host kernel cycles.
>>>>
>>>> Override __stackleak_poison() on x86-64, as s390 already does, through
>>>> the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
>>>> specific stackleak poison function"). Fill spans larger than 512 bytes
>>>> with REP STOSQ and keep the word loop for shorter ones, where the fixed
>>>> startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
>>>> about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
>>>> slower.
>>>>
>>>> The fill is inline asm, so the noinstr function still makes no calls.
>>>> memset64() would be the same instruction, but it is only static inline,
>>>> and with KMSAN it is an out-of-line function.
>>>>
>>>> In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
>>>> to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
>>>> and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
>>>> sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
>>>>
>>>> Signed-off-by: Usama Arif <usama.arif@linux.dev>
>>>> ---
>>>>  arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
>>>>  1 file changed, 25 insertions(+)
>>>>
>>>> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
>>>> index 77488ff5eb8c7c..049231d484ad0e 100644
>>>> --- a/arch/x86/include/asm/processor.h
>>>> +++ b/arch/x86/include/asm/processor.h
>>>> @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
>>>>                                current_stack_pointer) < THREAD_SIZE;
>>>>  }
>>>>
>>>> +#ifdef CONFIG_X86_64
>>>> +#define __stackleak_poison __stackleak_poison
>>>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>>>> +                                              unsigned long erase_high,
>>>> +                                              unsigned long poison)
>>>> +{
>>>> +       unsigned long count;
>>>> +
>>>> +       /* Short spans do not amortize the startup cost of REP STOSQ. */
>>>> +       if (erase_high <= erase_low + 512) {
>>>> +               while (erase_low < erase_high) {
>>>> +                       *(unsigned long *)erase_low = poison;
>>>> +                       erase_low += sizeof(unsigned long);
>>>> +               }
>>>> +               return;  
>>
>> gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6
>> It does manage to unroll the loop once - which is good because the intel
>> cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the
>> setup code is horrid.
>> But I also think some of the more recent cpu can execute more than
>> one write per clock.
>> If you unroll the loop 4 times it may run a lot faster.
>> Perhaps:
>> 	long count = (erase_high - erase_low) / sizeof(unsigned long);
>> 	if (count <= 64) {
>> 		while ((count -= 4) >= 0) {
>> 			*(unsigned long *)erase_low = poison;
>> 			*(unsigned long *)(erase_low + 8) = poison;
> 
> The add (probably) needs to be in the middle...
>  			erase_low += 32;
>> 			*(unsigned long *)(erase_low - 32 + 16) = poison;
>> 			*(unsigned long *)(erase_low - 32 + 24) = poison;
>> 		}
>> 		if (count & 2) {
>> 			*(unsigned long *)erase_low = poison;
>> 			*(unsigned long *)(erase_low + 8) = poison;
>> 			erase_low += 16;
>> 		}
>> 		if (count & 1)
>> 			*(unsigned long *)erase_low = poison;
>> 	} else {
>> 		...
>> 	}
>>
>> Except that gcc makes a mess of it again.
>> I don't think the 'erase_low += 32' is going to execute at the
>> same time as the last (two probably) writes so costs a clock.
>> So it may need to be asm.
> 
> I threw in enough OPTIMIZER_HIDE_VAR() to stop the optimiser breaking
> the code. https://godbolt.org/z/nx477T4M1
> gcc then generates:
>         movq    %rdx, (%rdi)
>         movq    %rdx, 8(%rdi)
>         addq    $32, %rdi
>         movq    %rdx, -16(%rdi)
>         movq    %rdx, -8(%rdi)
> .L32:
>         subq    $4, %rcx
>         jns     .L22
> for the code loop (clang is a bit worse - but should be ok).
> I think that will be a 'two clock loop' on cpu that can do
> two writes/clock; zen3 and ice lake).
> For 'stack splatting' rounding up to a multiple of 32 bytes
> will save all the 'remainder' code.


The range can't be rounded: erase_low is wherever the poison search
stopped, and erase_high is the stack pointer or the bottom of pt_regs
(168 bytes, so only 8-byte aligned). Rounding up writes live stack,
rounding down can write STACK_END_MAGIC when the stack was used to the
end.


> 
> David
> 
>>
>>>> +       }
>>>> +
>>>> +       count = (erase_high - erase_low) / sizeof(unsigned long);
>>>> +       asm volatile("rep stosq"
>>>> +                    : "+D" (erase_low), "+c" (count)
>>>> +                    : "a" (poison)
>>>> +                    : "memory");    
>>>
>>> Surely this isn't the only place using `rep stosq` in the tree, right?
>>> It's not, see memset64 in arch/x86/include/asm/string_64.h.  
>>
>> Which seems to have very few users.
>> memset_p() and memset_l() have almost no users.
>>
>> David
>>
>>>   
>>>> +}
>>>> +#endif
>>>> +
>>>>  #ifdef CONFIG_PARAVIRT_XXL
>>>>  #include <asm/paravirt.h>
>>>>  #else
>>>> --
>>>> 2.53.1-Meta
>>>>    
>>>
>>>   
>>
> 


  reply	other threads:[~2026-11-20 18:46 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19 21:31 [PATCH 0/2] x86/kstack_erase: Faster stack poisoning, plus a startup build fix Usama Arif
2026-11-19 21:31 ` [PATCH 1/2] x86/boot/startup: Disable kernel stack depth tracking Usama Arif
2026-11-19 22:30   ` Nick Desaulniers
2026-11-19 21:31 ` [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills Usama Arif
2026-11-19 22:28   ` Nick Desaulniers
2026-11-20 11:38     ` David Laight
2026-11-20 12:42       ` David Laight
2026-11-20 18:46         ` Usama Arif [this message]
2026-11-20 10:45   ` Peter Zijlstra
2026-11-20 12:26     ` Peter Zijlstra
2026-11-20 18:43       ` Usama Arif
2026-11-20 14:02     ` David Laight
2026-11-20 15:35       ` Peter Zijlstra
2026-11-20 16:18         ` David Laight
2026-11-20 18:55   ` David Laight

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=97e94491-ce19-45a1-8737-b30b49e997e8@linux.dev \
    --to=usama.arif@linux.dev \
    --cc=ardb@kernel.org \
    --cc=atish.patra@linux.dev \
    --cc=bp@alien8.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=david.laight.linux@gmail.com \
    --cc=dgreid@meta.com \
    --cc=gustavoars@kernel.org \
    --cc=hca@linux.ibm.com \
    --cc=hpa@zytor.com \
    --cc=justinstitt@google.com \
    --cc=kees@kernel.org \
    --cc=kernel-team@meta.com \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=llvm@lists.linux.dev \
    --cc=mingo@redhat.com \
    --cc=morbo@google.com \
    --cc=nathan@kernel.org \
    --cc=ndesaulniers@google.com \
    --cc=riel@surriel.com \
    --cc=rkeilty@meta.com \
    --cc=shakeel.butt@linux.dev \
    --cc=tglx@kernel.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®