[PATCH v3 0/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[PATCH v3 0/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

Enable REP MOVSB for copies of 2048 bytes and larger on ERMS CPUs without
FSRM, keeping scalar copies below that threshold. FSRM CPUs continue to use
REP MOVSB for every size.

This series splits v2 into 2 patches.
The first changes only the scalar register allocation: RCX holds the count
and tail indexes, and RDX is the small-copy data temporary. The second
adds the ERMS size check and adopts Linus's FSRM entry layout. REP MOVSB
replaces the existing jump to memcpy_orig(), with RET remaining in
ordinary text. ITS therefore selects return handling using the return's
actual execution address, without aligning replacement storage. Keeping
the count in RCX also addresses David's request to avoid reloading it on
the ERMS path.

v2 -> v3: https://lore.kernel.org/r/20261104222806.1826508-1-usama.arif@linux.dev
- Split the scalar RCX/RDX register swap into a preparatory patch with no
  functional change (Kiryl Shutsemau).
- Use REP MOVSB to replace the jump to memcpy_orig() on FSRM CPUs, keeping
  RET in ordinary text (Kiryl Shutsemau, Linus Torvalds). Remove the ITS
  replacement alignment, since no return is copied by alternatives now.

v1 -> v2: https://lore.kernel.org/r/20261013100442.2658311-1-usama.arif@linux.dev
- Raise the threshold from 512 to 2048 bytes. REP MOVSB was slower than
  the scalar loop for some misaligned copies of 512 to 640 bytes.
- Fold memcpy_orig() into __memcpy(). With FSRM, "rep movsb; RET" now
  replaces the 32-byte length check (David Laight, Linus Torvalds). In
  v1, the 13-byte empty FSRM replacement became a jump that FSRM CPUs
  took on every call (Kiryl Shutsemau).
- Do the ERMS check after the 32-byte check, so copies below 32 bytes
  skip it (Kiryl Shutsemau), and keep the count in RCX so REP MOVSB
  needs no reload (David Laight).
- Align the FSRM replacement so that its RET keeps the ITS return thunk.
- Fix the comment that called the return value the source input.
 
Usama Arif (2):
  x86/lib: Keep the scalar memcpy() count in RCX
  x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

 arch/x86/lib/memcpy_64.S | 78 +++++++++++++++++++++-------------------
 1 file changed, 42 insertions(+), 36 deletions(-)

-- 
2.53.0-Meta

[PATCH v3 1/2] x86/lib: Keep the scalar memcpy() count in RCX

[Usama Arif]

Use RCX for the scalar loop count and tail indexing, and RDX for the
small-copy data temporary. Initialize RCX from the incoming RDX count in
memcpy_orig().

This prepares the ERMS size check to use RDX as scratch while preserving
the REP MOVSB count in RCX. The FSRM path and copy policy stay the same.

No functional change.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/lib/memcpy_64.S | 59 ++++++++++++++++++++--------------------
 1 file changed, 30 insertions(+), 29 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 12a23fa7c44c5..7bb146c811029 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -46,8 +46,9 @@ EXPORT_SYMBOL(memcpy)
 
 SYM_FUNC_START_LOCAL(memcpy_orig)
 	movq %rdi, %rax
+	movq %rdx, %rcx
 
-	cmpq $0x20, %rdx
+	cmpq $0x20, %rcx
 	jb .Lhandle_tail
 
 	/*
@@ -56,9 +57,9 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	 */
 	cmp  %dil, %sil
 	jl .Lcopy_backward
-	subq $0x20, %rdx
+	subq $0x20, %rcx
 .Lcopy_forward_loop:
-	subq $0x20,	%rdx
+	subq $0x20,	%rcx
 
 	/*
 	 * Move in blocks of 4x8 bytes:
@@ -75,23 +76,23 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	movq %r11,	3*8(%rdi)
 	leaq 4*8(%rdi),	%rdi
 	jae  .Lcopy_forward_loop
-	addl $0x20,	%edx
+	addl $0x20,	%ecx
 	jmp  .Lhandle_tail
 
 .Lcopy_backward:
 	/*
 	 * Calculate copy position to tail.
 	 */
-	addq %rdx,	%rsi
-	addq %rdx,	%rdi
-	subq $0x20,	%rdx
+	addq %rcx,	%rsi
+	addq %rcx,	%rdi
+	subq $0x20,	%rcx
 	/*
 	 * At most 3 ALU operations in one cycle,
 	 * so append NOPS in the same 16 bytes trunk.
 	 */
 	.p2align 4
 .Lcopy_backward_loop:
-	subq $0x20,	%rdx
+	subq $0x20,	%rcx
 	movq -1*8(%rsi),	%r8
 	movq -2*8(%rsi),	%r9
 	movq -3*8(%rsi),	%r10
@@ -107,11 +108,11 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	/*
 	 * Calculate copy position to head.
 	 */
-	addl $0x20,	%edx
-	subq %rdx,	%rsi
-	subq %rdx,	%rdi
+	addl $0x20,	%ecx
+	subq %rcx,	%rsi
+	subq %rcx,	%rdi
 .Lhandle_tail:
-	cmpl $16,	%edx
+	cmpl $16,	%ecx
 	jb   .Lless_16bytes
 
 	/*
@@ -119,53 +120,53 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	 */
 	movq 0*8(%rsi), %r8
 	movq 1*8(%rsi),	%r9
-	movq -2*8(%rsi, %rdx),	%r10
-	movq -1*8(%rsi, %rdx),	%r11
+	movq -2*8(%rsi, %rcx),	%r10
+	movq -1*8(%rsi, %rcx),	%r11
 	movq %r8,	0*8(%rdi)
 	movq %r9,	1*8(%rdi)
-	movq %r10,	-2*8(%rdi, %rdx)
-	movq %r11,	-1*8(%rdi, %rdx)
+	movq %r10,	-2*8(%rdi, %rcx)
+	movq %r11,	-1*8(%rdi, %rcx)
 	RET
 	.p2align 4
 .Lless_16bytes:
-	cmpl $8,	%edx
+	cmpl $8,	%ecx
 	jb   .Lless_8bytes
 	/*
 	 * Move data from 8 bytes to 15 bytes.
 	 */
 	movq 0*8(%rsi),	%r8
-	movq -1*8(%rsi, %rdx),	%r9
+	movq -1*8(%rsi, %rcx),	%r9
 	movq %r8,	0*8(%rdi)
-	movq %r9,	-1*8(%rdi, %rdx)
+	movq %r9,	-1*8(%rdi, %rcx)
 	RET
 	.p2align 4
 .Lless_8bytes:
-	cmpl $4,	%edx
+	cmpl $4,	%ecx
 	jb   .Lless_3bytes
 
 	/*
 	 * Move data from 4 bytes to 7 bytes.
 	 */
-	movl (%rsi), %ecx
-	movl -4(%rsi, %rdx), %r8d
-	movl %ecx, (%rdi)
-	movl %r8d, -4(%rdi, %rdx)
+	movl (%rsi), %edx
+	movl -4(%rsi, %rcx), %r8d
+	movl %edx, (%rdi)
+	movl %r8d, -4(%rdi, %rcx)
 	RET
 	.p2align 4
 .Lless_3bytes:
-	subl $1, %edx
+	subl $1, %ecx
 	jb .Lend
 	/*
 	 * Move data from 1 bytes to 3 bytes.
 	 */
-	movzbl (%rsi), %ecx
+	movzbl (%rsi), %edx
 	jz .Lstore_1byte
 	movzbq 1(%rsi), %r8
-	movzbq (%rsi, %rdx), %r9
+	movzbq (%rsi, %rcx), %r9
 	movb %r8b, 1(%rdi)
-	movb %r9b, (%rdi, %rdx)
+	movb %r9b, (%rdi, %rcx)
 .Lstore_1byte:
-	movb %cl, (%rdi)
+	movb %dl, (%rdi)
 
 .Lend:
 	RET
-- 
2.53.0-Meta

[PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

Since commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies"), memcpy() uses REP MOVSB only on FSRM CPUs, so ERMS CPUs
without FSRM, such as Skylake and Cooper Lake, copy every size with
the scalar loop. Large copies matter there: in Meta's x86 fleet, about a
third of the time in memcpy_orig() is btrfs zstd decompression copying
pages into and out of zstd's buffers.

Use REP MOVSB from 2048 bytes on ERMS CPUs. On a 1.4 GHz Cooper Lake, an
aligned, cache-hot 4 KiB memcpy() then takes 66.8% less time (123.3 ns
instead of 371.6 ns). In a single-vCPU guest, reading a synthetic
512 MiB btrfs zstd file takes 14.8% less time (median of five kernel
pairs). Smaller copies keep the scalar loop: using REP MOVSB from
32 bytes made 32/64/128/256-byte copies
111.8%/68.0%/63.2%/15.6% slower.

Prepare the return value and RCX count before the FSRM alternative. On
FSRM CPUs, REP MOVSB replaces the jump to memcpy_orig(), followed by a
RET in ordinary text. ITS then selects return handling using the RET's
execution address.

For ERMS CPUs, shift the scratch copy of the length in RDX after the
32-byte check. RCX already holds the count, so large copies need no
reload. Copies below 32 bytes skip the check, which is a single NOP
without ERMS.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/lib/memcpy_64.S | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 7bb146c811029..5320862e0274e 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -27,15 +27,13 @@
  *
  * The 'rep movsb' itself is small enough to replace the call, but the
  * two register moves blow up the code. And one of them is "needed"
- * only for the return value that is the same as the source input,
+ * only for the return value that is the same as the destination input,
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
-
 	movq %rdi, %rax
 	movq %rdx, %rcx
-	rep movsb
+	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
 	RET
 SYM_FUNC_END(__memcpy)
 EXPORT_SYMBOL(__memcpy)
@@ -45,12 +43,16 @@ SYM_PIC_ALIAS(memcpy)
 EXPORT_SYMBOL(memcpy)
 
 SYM_FUNC_START_LOCAL(memcpy_orig)
-	movq %rdi, %rax
-	movq %rdx, %rcx
-
 	cmpq $0x20, %rcx
 	jb .Lhandle_tail
 
+	/*
+	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
+	 * Shift RDX to check length >= 2048 without destroying the REP count.
+	 * The 10-byte alternative fits a NOP without ERMS.
+	 */
+	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS
+
 	/*
 	 * We check whether memory false dependence could occur,
 	 * then jump to corresponding copy mode.
@@ -170,5 +172,8 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 
 .Lend:
 	RET
-SYM_FUNC_END(memcpy_orig)
 
+.Lerms:
+	rep movsb
+	RET
+SYM_FUNC_END(memcpy_orig)
-- 
2.53.0-Meta

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Peter Zijlstra]

On Wed, Nov 11, 2026 at 05:20:53AM -0800, Usama Arif wrote:
> Since commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
> memory copies"), memcpy() uses REP MOVSB only on FSRM CPUs, so ERMS CPUs
> without FSRM, such as Skylake and Cooper Lake, copy every size with
> the scalar loop. Large copies matter there: in Meta's x86 fleet, about a
> third of the time in memcpy_orig() is btrfs zstd decompression copying
> pages into and out of zstd's buffers.
> 
> Use REP MOVSB from 2048 bytes on ERMS CPUs. On a 1.4 GHz Cooper Lake, an
> aligned, cache-hot 4 KiB memcpy() then takes 66.8% less time (123.3 ns
> instead of 371.6 ns). In a single-vCPU guest, reading a synthetic
> 512 MiB btrfs zstd file takes 14.8% less time (median of five kernel
> pairs). Smaller copies keep the scalar loop: using REP MOVSB from
> 32 bytes made 32/64/128/256-byte copies
> 111.8%/68.0%/63.2%/15.6% slower.
> 
> Prepare the return value and RCX count before the FSRM alternative. On
> FSRM CPUs, REP MOVSB replaces the jump to memcpy_orig(), followed by a
> RET in ordinary text. ITS then selects return handling using the RET's
> execution address.
> 
> For ERMS CPUs, shift the scratch copy of the length in RDX after the
> 32-byte check. RCX already holds the count, so large copies need no
> reload. Copies below 32 bytes skip the check, which is a single NOP
> without ERMS.

So no real objection to these patches from me.

However, it did remind me of something I have on the TODO list for a
rainy afternoon, namely merge memcpy and rep_movs_alternative.

It makes very little sense to me that we have two (divergent) copies of
what is basically the same thing.

Re: [PATCH v3 0/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Kiryl Shutsemau]

On Wed, Nov 11, 2026 at 05:20:51AM -0800, Usama Arif wrote:
> Enable REP MOVSB for copies of 2048 bytes and larger on ERMS CPUs without
> FSRM, keeping scalar copies below that threshold. FSRM CPUs continue to use
> REP MOVSB for every size.
> 
> This series splits v2 into 2 patches.
> The first changes only the scalar register allocation: RCX holds the count
> and tail indexes, and RDX is the small-copy data temporary. The second
> adds the ERMS size check and adopts Linus's FSRM entry layout. REP MOVSB
> replaces the existing jump to memcpy_orig(), with RET remaining in
> ordinary text. ITS therefore selects return handling using the return's
> actual execution address, without aligning replacement storage. Keeping
> the count in RCX also addresses David's request to avoid reloading it on
> the ERMS path.
> 
> v2 -> v3: https://lore.kernel.org/r/20261104222806.1826508-1-usama.arif@linux.dev
> - Split the scalar RCX/RDX register swap into a preparatory patch with no
>   functional change (Kiryl Shutsemau).
> - Use REP MOVSB to replace the jump to memcpy_orig() on FSRM CPUs, keeping
>   RET in ordinary text (Kiryl Shutsemau, Linus Torvalds). Remove the ITS
>   replacement alignment, since no return is copied by alternatives now.
> 
> v1 -> v2: https://lore.kernel.org/r/20261013100442.2658311-1-usama.arif@linux.dev
> - Raise the threshold from 512 to 2048 bytes. REP MOVSB was slower than
>   the scalar loop for some misaligned copies of 512 to 640 bytes.
> - Fold memcpy_orig() into __memcpy(). With FSRM, "rep movsb; RET" now
>   replaces the 32-byte length check (David Laight, Linus Torvalds). In
>   v1, the 13-byte empty FSRM replacement became a jump that FSRM CPUs
>   took on every call (Kiryl Shutsemau).
> - Do the ERMS check after the 32-byte check, so copies below 32 bytes
>   skip it (Kiryl Shutsemau), and keep the count in RCX so REP MOVSB
>   needs no reload (David Laight).
> - Align the FSRM replacement so that its RET keeps the ITS return thunk.
> - Fix the comment that called the return value the source input.
>  
> Usama Arif (2):
>   x86/lib: Keep the scalar memcpy() count in RCX
>   x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM
> 
>  arch/x86/lib/memcpy_64.S | 78 +++++++++++++++++++++-------------------
>  1 file changed, 42 insertions(+), 36 deletions(-)

Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Borislav Petkov]

On Wed, Nov 11, 2026 at 05:20:53AM -0800, Usama Arif wrote:
> diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> index 7bb146c811029..5320862e0274e 100644
> --- a/arch/x86/lib/memcpy_64.S
> +++ b/arch/x86/lib/memcpy_64.S
> @@ -27,15 +27,13 @@
>   *
>   * The 'rep movsb' itself is small enough to replace the call, but the
>   * two register moves blow up the code. And one of them is "needed"
> - * only for the return value that is the same as the source input,
> + * only for the return value that is the same as the destination input,
>   * which the compiler could/should do much better anyway.
>   */
>  SYM_TYPED_FUNC_START(__memcpy)
> -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> -
>  	movq %rdi, %rax
>  	movq %rdx, %rcx
> -	rep movsb
> +	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
>  	RET
>  SYM_FUNC_END(__memcpy)
>  EXPORT_SYMBOL(__memcpy)
> @@ -45,12 +43,16 @@ SYM_PIC_ALIAS(memcpy)
>  EXPORT_SYMBOL(memcpy)
>  
>  SYM_FUNC_START_LOCAL(memcpy_orig)
> -	movq %rdi, %rax
> -	movq %rdx, %rcx
> -
>  	cmpq $0x20, %rcx
>  	jb .Lhandle_tail
>  
> +	/*
> +	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
> +	 * Shift RDX to check length >= 2048 without destroying the REP count.
> +	 * The 10-byte alternative fits a NOP without ERMS.
> +	 */
> +	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS

Sorry, this is adding more JMPs in "conditional" alternatives code and thus
turns this into a bigger and unmaintainable mess than it already is.

The 2K limit looks like something conjured out of a thin air as it seems it
work for your particular case. I'm sure we will have to whack it in the future
because someone else will complain of the opposite.

This needs to be done with alternative_call*, even at the price of some
duplicated code but without turning what already is a mess, into a bigger one.

Or a static key or even with stop_machine_nmi() (currently in review) which
patches the kernel even later, when it is convenient to do so.

Whatever it takes to keep this code readable and sane.

But not this.

Sorry.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[David Laight]

On Wed, 11 Nov 2026 15:46:08 -0800
Borislav Petkov <bp@alien8.de> wrote:

> On Wed, Nov 11, 2026 at 05:20:53AM -0800, Usama Arif wrote:
> > diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> > index 7bb146c811029..5320862e0274e 100644
> > --- a/arch/x86/lib/memcpy_64.S
> > +++ b/arch/x86/lib/memcpy_64.S
> > @@ -27,15 +27,13 @@
> >   *
> >   * The 'rep movsb' itself is small enough to replace the call, but the
> >   * two register moves blow up the code. And one of them is "needed"
> > - * only for the return value that is the same as the source input,
> > + * only for the return value that is the same as the destination input,
> >   * which the compiler could/should do much better anyway.
> >   */
> >  SYM_TYPED_FUNC_START(__memcpy)
> > -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> > -
> >  	movq %rdi, %rax
> >  	movq %rdx, %rcx
> > -	rep movsb
> > +	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
> >  	RET
> >  SYM_FUNC_END(__memcpy)
> >  EXPORT_SYMBOL(__memcpy)
> > @@ -45,12 +43,16 @@ SYM_PIC_ALIAS(memcpy)
> >  EXPORT_SYMBOL(memcpy)
> >  
> >  SYM_FUNC_START_LOCAL(memcpy_orig)
> > -	movq %rdi, %rax
> > -	movq %rdx, %rcx
> > -
> >  	cmpq $0x20, %rcx
> >  	jb .Lhandle_tail
> >  
> > +	/*
> > +	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
> > +	 * Shift RDX to check length >= 2048 without destroying the REP count.
> > +	 * The 10-byte alternative fits a NOP without ERMS.
> > +	 */
> > +	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS  
> 
> Sorry, this is adding more JMPs in "conditional" alternatives code and thus
> turns this into a bigger and unmaintainable mess than it already is.
> 
> The 2K limit looks like something conjured out of a thin air as it seems it
> work for your particular case. I'm sure we will have to whack it in the future
> because someone else will complain of the opposite.
> 
> This needs to be done with alternative_call*, even at the price of some
> duplicated code but without turning what already is a mess, into a bigger one.

Multiple functions seems like a good idea.

In some cases you can patch the constant in the instruction.
That is safe here even if a multi-byte constant crosses a cache line
because it is only an optimisation.

Patching the call sites would certainly be cleaner.
More interesting would be having a function with a non-standard calling
convention (%rdi, %rsi, %rcx, everything else preserved, void result)
so that the 5 byte call can be patched with "rep movsb; nop3".
I can't see anywhere that uses the result.
Maybe start with "rep movsb; nop3" and patch with the call?
You'd still need a normal memcpy() for the calls gcc makes and if the
address is taken.

A special function to do the patching would also make sense.
All Intel non-atom cpu since sandy bridge run 'rep movsb' as fast
as 'rep movsq' - so you only need to worry about short copies.
Whereas some of the older Zen cpu run misaligned copies slowly.
So you could pick/patch the most appropriate code using more complex
logic than just whether one of the X86_FEATURE_xxx bits is set.

The same calling convention might make sense for memmove().
Although it probably wants a scratch register for the (dst - src < len)
check (it might want dst in the scratch register!).

David

> 
> Or a static key or even with stop_machine_nmi() (currently in review) which
> patches the kernel even later, when it is convenient to do so.
> 
> Whatever it takes to keep this code readable and sane.
> 
> But not this.
> 
> Sorry.
> 

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

On Wed, 11 Nov 2026 15:46:08 -0800 Borislav Petkov <bp@alien8.de> wrote:

> On Wed, Nov 11, 2026 at 05:20:53AM -0800, Usama Arif wrote:
> > diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> > index 7bb146c811029..5320862e0274e 100644
> > --- a/arch/x86/lib/memcpy_64.S
> > +++ b/arch/x86/lib/memcpy_64.S
> > @@ -27,15 +27,13 @@
> >   *
> >   * The 'rep movsb' itself is small enough to replace the call, but the
> >   * two register moves blow up the code. And one of them is "needed"
> > - * only for the return value that is the same as the source input,
> > + * only for the return value that is the same as the destination input,
> >   * which the compiler could/should do much better anyway.
> >   */
> >  SYM_TYPED_FUNC_START(__memcpy)
> > -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> > -
> >  	movq %rdi, %rax
> >  	movq %rdx, %rcx
> > -	rep movsb
> > +	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
> >  	RET
> >  SYM_FUNC_END(__memcpy)
> >  EXPORT_SYMBOL(__memcpy)
> > @@ -45,12 +43,16 @@ SYM_PIC_ALIAS(memcpy)
> >  EXPORT_SYMBOL(memcpy)
> >  
> >  SYM_FUNC_START_LOCAL(memcpy_orig)
> > -	movq %rdi, %rax
> > -	movq %rdx, %rcx
> > -
> >  	cmpq $0x20, %rcx
> >  	jb .Lhandle_tail
> >  
> > +	/*
> > +	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
> > +	 * Shift RDX to check length >= 2048 without destroying the REP count.
> > +	 * The 10-byte alternative fits a NOP without ERMS.
> > +	 */
> > +	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS
> 
> Sorry, this is adding more JMPs in "conditional" alternatives code and thus
> turns this into a bigger and unmaintainable mess than it already is.
>

How about something like what I have added at the end, would that be more
acceptable in terms of maintainability?
 
> The 2K limit looks like something conjured out of a thin air as it seems it
> work for your particular case. I'm sure we will have to whack it in the future
> because someone else will complain of the opposite.
> 

As mentioned in the commit message as well as [1], 2K came from experiments.

Changing the cmp immediate from 2048 to 32 bytes made aligned warmed
32/64/128/256-byte copies 101.2%/69.5%/53.5%/16.5% slower.

This is usecase driven, as Linus said in [2]: "let's see if somebody finds a
case where the kernel memory copies also care".

If someone finds a case where less than 2K is used and it doesnt cause a
regression for CPUs, they can try lowering it.

> This needs to be done with alternative_call*, even at the price of some
> duplicated code but without turning what already is a mess, into a bigger one.
> 

I tried alternative_call_2() from a C entry with separate helpers. It
adds a helper CALL/RET to every path, including FSRM. The attached
version uses tail jumps, so each helper returns directly to the original
caller and FSRM keeps REP inline. I am hoping this is acceptable?

> Or a static key or even with stop_machine_nmi() (currently in review) which
> patches the kernel even later, when it is convenient to do so.
> 
> Whatever it takes to keep this code readable and sane.
> 
> But not this.
> 
> Sorry.
> 

No worries! Hopefully the version below is more maintainable and acceptable
for everyone?

Thanks for the review!
Usama

[1] https://lore.kernel.org/all/20261017192727.2760919-1-usama.arif@linux.dev/
[2] https://git.kernel.org/linus/47ee3f1dd93bcbe031539b1ecdaafb44b661c772

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 7bb146c811029..8a398edf6e160 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -27,15 +27,15 @@
  *
  * The 'rep movsb' itself is small enough to replace the call, but the
  * two register moves blow up the code. And one of them is "needed"
- * only for the return value that is the same as the source input,
+ * only for the return value that is the same as the destination input,
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-       ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
-
        movq %rdi, %rax
        movq %rdx, %rcx
-       rep movsb
+       /* FSRM takes precedence. Keep RET outside replacements for ITS. */
+       ALTERNATIVE_2 "jmp memcpy_orig", "jmp memcpy_erms", \
+                     X86_FEATURE_ERMS, "rep movsb", X86_FEATURE_FSRM
        RET
 SYM_FUNC_END(__memcpy)
 EXPORT_SYMBOL(__memcpy)
@@ -44,10 +44,15 @@ SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
 SYM_PIC_ALIAS(memcpy)
 EXPORT_SYMBOL(memcpy)

-SYM_FUNC_START_LOCAL(memcpy_orig)
-       movq %rdi, %rax
-       movq %rdx, %rcx
+/* The entry prepares RAX as the return value and RCX as the count. */
+SYM_FUNC_START_LOCAL(memcpy_erms)
+       cmpq $2048, %rcx
+       jb memcpy_orig
+       rep movsb
+       RET
+SYM_FUNC_END(memcpy_erms)

+SYM_FUNC_START_LOCAL(memcpy_orig)
        cmpq $0x20, %rcx
        jb .Lhandle_tail
 

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

On Thu, 12 Nov 2026 09:37:29 +0000 David Laight <david.laight.linux@gmail.com> wrote:

> On Wed, 11 Nov 2026 15:46:08 -0800
> Borislav Petkov <bp@alien8.de> wrote:
> 
> > On Wed, Nov 11, 2026 at 05:20:53AM -0800, Usama Arif wrote:
> > > diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> > > index 7bb146c811029..5320862e0274e 100644
> > > --- a/arch/x86/lib/memcpy_64.S
> > > +++ b/arch/x86/lib/memcpy_64.S
> > > @@ -27,15 +27,13 @@
> > >   *
> > >   * The 'rep movsb' itself is small enough to replace the call, but the
> > >   * two register moves blow up the code. And one of them is "needed"
> > > - * only for the return value that is the same as the source input,
> > > + * only for the return value that is the same as the destination input,
> > >   * which the compiler could/should do much better anyway.
> > >   */
> > >  SYM_TYPED_FUNC_START(__memcpy)
> > > -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> > > -
> > >  	movq %rdi, %rax
> > >  	movq %rdx, %rcx
> > > -	rep movsb
> > > +	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
> > >  	RET
> > >  SYM_FUNC_END(__memcpy)
> > >  EXPORT_SYMBOL(__memcpy)
> > > @@ -45,12 +43,16 @@ SYM_PIC_ALIAS(memcpy)
> > >  EXPORT_SYMBOL(memcpy)
> > >  
> > >  SYM_FUNC_START_LOCAL(memcpy_orig)
> > > -	movq %rdi, %rax
> > > -	movq %rdx, %rcx
> > > -
> > >  	cmpq $0x20, %rcx
> > >  	jb .Lhandle_tail
> > >  
> > > +	/*
> > > +	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
> > > +	 * Shift RDX to check length >= 2048 without destroying the REP count.
> > > +	 * The 10-byte alternative fits a NOP without ERMS.
> > > +	 */
> > > +	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS  
> > 
> > Sorry, this is adding more JMPs in "conditional" alternatives code and thus
> > turns this into a bigger and unmaintainable mess than it already is.
> > 
> > The 2K limit looks like something conjured out of a thin air as it seems it
> > work for your particular case. I'm sure we will have to whack it in the future
> > because someone else will complain of the opposite.
> > 
> > This needs to be done with alternative_call*, even at the price of some
> > duplicated code but without turning what already is a mess, into a bigger one.
> 
> Multiple functions seems like a good idea.
>

Hopefully the version in [1] is acceptable!
 
> In some cases you can patch the constant in the instruction.
> That is safe here even if a multi-byte constant crosses a cache line
> because it is only an optimisation.
> 

I think it might be a good idea to start with just a constant and if
someone can comeup with a usecase that needs it patched, we can then
look at this? Basically prefer to keep it usecase driven.

> Patching the call sites would certainly be cleaner.
> More interesting would be having a function with a non-standard calling
> convention (%rdi, %rsi, %rcx, everything else preserved, void result)
> so that the 5 byte call can be patched with "rep movsb; nop3".
> I can't see anywhere that uses the result.
> Maybe start with "rep movsb; nop3" and patch with the call?
> You'd still need a normal memcpy() for the calls gcc makes and if the
> address is taken.
> 

Replacing the caller's CALL with REP MOVSB could remove actual call
overhead. The C wrapper I tried patches a CALL inside __memcpy() and
adds a helper call and return. The attached tail dispatcher handles the
existing public ABI and leaves caller-side patching for a follow-up.

The scalar helper currently uses RDX and R8-R11 as scratch. Your private
void interface would need an implementation that preserves those
registers, or register saves. RCX already holds the count, and the
scalar helper preserves RAX and the callee-saved registers.

There are return-value users: memdup_buf() in
lib/crypto/tests/test-utils.h returns memcpy()'s pointer. The normal
entry must still return the original destination in RAX and support
compiler-generated and address-taken calls, as you point out. The
current revision preserves that ABI and its aliases.


> A special function to do the patching would also make sense.
> All Intel non-atom cpu since sandy bridge run 'rep movsb' as fast
> as 'rep movsq' - so you only need to worry about short copies.
> Whereas some of the older Zen cpu run misaligned copies slowly.
> So you could pick/patch the most appropriate code using more complex
> logic than just whether one of the X86_FEATURE_xxx bits is set.
>

I agree that CPU model and alignment can inform a better policy.

I think to do this, we need access to many different CPUs to test.
I am confident that the 2K cmp will not regress other CPUs and is a good
start. Would it make sense to go ahead with the apprach in the current
patch and develop later on it if someone comes up with a usecase?
 
> The same calling convention might make sense for memmove().
> Although it probably wants a scratch register for the (dst - src < len)
> check (it might want dst in the scratch register!).
> 
> David
> 

[1] https://lore.kernel.org/all/20261113141137.1113517-1-usama.arif@linux.dev/ 

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[David Laight]

On Fri, 13 Nov 2026 06:11:36 -0800
Usama Arif <usama.arif@linux.dev> wrote:

> How about something like what I have added at the end, would that be more
> acceptable in terms of maintainability?

I was thinking more of something like:
(written in the email so untested)

static always_inline void *memcpy(void *dst, const void *src, size_t len)
{
	void *orig_dst = dst;

	asm volatile(	".pushsection ".memcpy_calls\n"
			"\t.long . - 1f\n"
			".popsection\n"
			"1:\trep movsb\n"
			"\tnop; nop"
			: "+c (len), "+D" (dst), "+S" (src)
			:: ASM_CALL_CONSTRAINT);

	return orig_dst;
}

An unused return value will be optimised away at all the call sites.

That will work (maybe slowly) on all cpu.
Then some code reads all the addresses from .memcpy_calls and, if necessary,
patches the 5 bytes to a call to the appropriate function.

It might be worth giving the code one or two scratch registers so that
a short copy loop doesn't need to spill to stack (perhaps %rax and %rdx).
It'll matter less for a longer copy loop.

So you might have:
_memcpy_erms:
	cmp $nnnn, %rcx
	jb  _memcpy_words
	rep movsb
	RET

_memcpy_words:
	cmpb $nnnn %rcx
	jb  _memcpy_bytes
	mov %rax,%rcx
	or  %rcx,%rdi
	or  %rcx,%rsi
	test $7,%rcx
	jnz _memcpy_misaligned
	sub $8,%rax
1:	mov %rcx,[%rsi]
	mov %rdx,8[%rsi]
	mov [%rdi],%rcx
	mov 8[%rdi],%rdx
	add $16,%rdi
	add $16,%rsi
	sub $16,%rax
	jnc 1b:
	jz  2f
	mov %rax,[%si]
	mov [%rdi]%rax]
2:	RET

That isn't the best 2-word loop by any means.
And you may not care about misaligned pointers.
	
etc

David

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Linus Torvalds]

On Fri, 13 Nov 2026 at 08:03, David Laight <david.laight.linux@gmail.com> wrote:
>
> I was thinking more of something like:
> (written in the email so untested)
>
> static always_inline void *memcpy(void *dst, const void *src, size_t len)

The problem with this approach two-fold:

 - many memcpy() calls aren't actually memcpy() calls on a C level

 - many memcpy() that *are* calls on a C level should never be calls
in the generated code

That first case is for things like structure copies etc. And it's
common, and it's important, although for the kernel it's usually a
non-issue because it's *typicay* a small constant-sized copy, which
brings us to the second case...

The second case is hugely important. The compiler needs to turn small
memcpy() calls into just loads and stores. Calling "memcpy()" for a
known 64-byte copy is hugely inefficient.

We used to - long ago - do that "constant small sizes" thing by hand.
It was a huge pain. We now rely on the compiler to do it.

End result: we must *not* hide 'memcpy()' from the compiler for
performance reasons, and we cannot catch all cases anyway because the
compiler generates implicit memcpy calls.

This is why I really wish we had the compiler just generate a
different sequence for 'call memcpy'. One that used a different
callign convention - preferably just the 'rep movs' one so that we
could just rewrite the call with 'rep movsb' in place.

Without compiler support, we're kind of screwed.

              Linus

              Linus

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[H. Peter Anvin]

On November 13, 2026 9:34:26 AM PST, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>On Fri, 13 Nov 2026 at 08:03, David Laight <david.laight.linux@gmail.com> wrote:
>>
>> I was thinking more of something like:
>> (written in the email so untested)
>>
>> static always_inline void *memcpy(void *dst, const void *src, size_t len)
>
>The problem with this approach two-fold:
>
> - many memcpy() calls aren't actually memcpy() calls on a C level
>
> - many memcpy() that *are* calls on a C level should never be calls
>in the generated code
>
>That first case is for things like structure copies etc. And it's
>common, and it's important, although for the kernel it's usually a
>non-issue because it's *typicay* a small constant-sized copy, which
>brings us to the second case...
>
>The second case is hugely important. The compiler needs to turn small
>memcpy() calls into just loads and stores. Calling "memcpy()" for a
>known 64-byte copy is hugely inefficient.
>
>We used to - long ago - do that "constant small sizes" thing by hand.
>It was a huge pain. We now rely on the compiler to do it.
>
>End result: we must *not* hide 'memcpy()' from the compiler for
>performance reasons, and we cannot catch all cases anyway because the
>compiler generates implicit memcpy calls.
>
>This is why I really wish we had the compiler just generate a
>different sequence for 'call memcpy'. One that used a different
>callign convention - preferably just the 'rep movs' one so that we
>could just rewrite the call with 'rep movsb' in place.
>
>Without compiler support, we're kind of screwed.
>
>              Linus
>
>              Linus

Well, we can put a request in for compiler support. This equally applies to other compiler runtime functions, for that matter...

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[David Laight]

On Fri, 13 Nov 2026 09:34:26 -0800
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Fri, 13 Nov 2026 at 08:03, David Laight <david.laight.linux@gmail.com> wrote:
> >
> > I was thinking more of something like:
> > (written in the email so untested)
> >
> > static always_inline void *memcpy(void *dst, const void *src, size_t len)  
> 
> The problem with this approach two-fold:
> 
>  - many memcpy() calls aren't actually memcpy() calls on a C level
> 
>  - many memcpy() that *are* calls on a C level should never be calls
> in the generated code
> 
> That first case is for things like structure copies etc. And it's
> common, and it's important, although for the kernel it's usually a
> non-issue because it's *typicay* a small constant-sized copy, which
> brings us to the second case...
> 
> The second case is hugely important. The compiler needs to turn small
> memcpy() calls into just loads and stores. Calling "memcpy()" for a
> known 64-byte copy is hugely inefficient.
> 
> We used to - long ago - do that "constant small sizes" thing by hand.
> It was a huge pain. We now rely on the compiler to do it.

Actually you could detect the 'small constant sizes' and call the real
memcpy for them.
The compiler would then (hopefully) inline them as a series of moves.

> End result: we must *not* hide 'memcpy()' from the compiler for
> performance reasons, and we cannot catch all cases anyway because the
> compiler generates implicit memcpy calls.

I had remembered about those, even if I didn't mention them.

> This is why I really wish we had the compiler just generate a
> different sequence for 'call memcpy'. One that used a different
> callign convention - preferably just the 'rep movs' one so that we
> could just rewrite the call with 'rep movsb' in place.

I knew you had wished that in the past.
It isn't as though it doesn't have other calls with (annoying)
non-standard calling conventions.
For one arch we optimise 'shift by count' with a jump table, and fall
foul of needing a scratch register - so have to spill to stack.

David

> 
> Without compiler support, we're kind of screwed.
> 
>               Linus
> 
>               Linus