Re: [PATCH v2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif <usama.arif@linux.dev>]

On Tue, 10 Nov 2026 16:25:54 +0000 Kiryl Shutsemau <kas@kernel.org> wrote:

> On Wed, Nov 04, 2026 at 02:28:06PM -0800, Usama Arif wrote:
> > +#ifdef CONFIG_MITIGATION_ITS
> > +	/*
> > +	 * Returns are patched before alternatives. Align the replacement
> > +	 * source so its return is in the first half of a cache line and ITS
> > +	 * retains its thunk. The copied return can then execute in either
> > +	 * half of a cache line.
> > +	 */
> > +	.pushsection .altinstr_replacement, "ax"
> > +	.balign 64
> > +	.popsection
> > +#endif
> > +	ALTERNATIVE __stringify(cmpq $0x20, %rcx; jb .Lhandle_tail), \
> > +		    __stringify(rep movsb; RET), X86_FEATURE_FSRM
> 
> Does this mean any RET in a replacement needs the same alignment? Would
> anything catch it if the next one doesn't have it?

The issue is specific to address-dependent return handling, such as
ITS. apply_returns() runs before alternatives, so its decision uses
the replacement source address. Copying an unthunked return from the
safe half to the vulnerable half can invalidate that decision. The
patch had no check of the copied return's final ITS placement.

I have kept RET in ordinary text, so the decision now uses the address
where it executes. There is no return in either replacement and no
replacement alignment to maintain.


> 
> Linus' layout keeps the RET out of the replacement, with no padding:
> 
> 	movq %rdi, %rax
> 	movq %rdx, %rcx
> 	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
> 	RET
> 
> Non-FSRM takes the same jump it takes today. Does that avoid the ITS
> problem? Why not this layout?

Yes. I have adopted that layout and kept memcpy_orig() separate.
The entry prepares RAX and RCX; the scalar function no longer repeats
that preparation. FSRM replaces only the jump with REP MOVSB, leaving
RET in ordinary text. This resolves the ITS source-address issue.


> 
> > +	/*
> > +	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
> > +	 * Shift RDX to check length >= 2048 without destroying the REP count.
> > +	 * The 10-byte alternative fits a NOP without ERMS.
> > +	 */
> > +	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS
> 
> The RCX/RDX swap is most of the diff, and all it saves is one MOV on
> copies of 2K and up.

Agreed that the register change obscured the functional change. I have
split it into a preparatory patch. It consistently uses RCX for scalar
counts and indexes, and RDX for small-copy data. The subsequent ERMS
patch can shift RDX while preserving RCX for REP MOVSB.


> 
> What about something like this? Untested.

The entry layout is now the one you suggested. I retained the RCX
count in a separate patch, so the ERMS arm needs no count reload.
The gate remains after the 32-byte check and keeps the 2048 threshold.


> 
> diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> --- a/arch/x86/lib/memcpy_64.S
> +++ b/arch/x86/lib/memcpy_64.S
> @@ -31,11 +31,9 @@
>   * which the compiler could/should do much better anyway.
>   */
>  SYM_TYPED_FUNC_START(__memcpy)
> -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> -
>  	movq %rdi, %rax
>  	movq %rdx, %rcx
> -	rep movsb
> +	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
>  	RET
>  SYM_FUNC_END(__memcpy)
>  EXPORT_SYMBOL(__memcpy)
> @@ -45,11 +43,11 @@ SYM_PIC_ALIAS(memcpy)
>  EXPORT_SYMBOL(memcpy)
> 
>  SYM_FUNC_START_LOCAL(memcpy_orig)
> -	movq %rdi, %rax
> -
>  	cmpq $0x20, %rdx
>  	jb .Lhandle_tail
> 
> +	ALTERNATIVE "", __stringify(shrq $11, %rcx; jnz .Lerms), X86_FEATURE_ERMS
> +
>  	/*
>  	 * We check whether memory false dependence could occur,
>  	 * then jump to corresponding copy mode.
> @@ -169,5 +167,10 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
> 
>  .Lend:
>  	RET
> +
> +.Lerms:
> +	movq %rdx, %rcx
> +	rep movsb
> +	RET
>  SYM_FUNC_END(memcpy_orig)
> 
> If you keep the swap, please make it a separate patch with no functional
> change.

Done. The first patch changes the scalar register allocation and
initializes RCX in memcpy_orig(), with no copy-policy change. The second
patch hoists that preparation and adds the ERMS path and FSRM layout.

> 
> -- 
>   Kiryl Shutsemau / Kirill A. Shutemov
>