[PATCH v3 0/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[PATCH v3 0/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

Enable REP MOVSB for copies of 2048 bytes and larger on ERMS CPUs without
FSRM, keeping scalar copies below that threshold. FSRM CPUs continue to use
REP MOVSB for every size.

This series splits v2 into 2 patches.
The first changes only the scalar register allocation: RCX holds the count
and tail indexes, and RDX is the small-copy data temporary. The second
adds the ERMS size check and adopts Linus's FSRM entry layout. REP MOVSB
replaces the existing jump to memcpy_orig(), with RET remaining in
ordinary text. ITS therefore selects return handling using the return's
actual execution address, without aligning replacement storage. Keeping
the count in RCX also addresses David's request to avoid reloading it on
the ERMS path.

v2 -> v3: https://lore.kernel.org/r/20261104222806.1826508-1-usama.arif@linux.dev
- Split the scalar RCX/RDX register swap into a preparatory patch with no
  functional change (Kiryl Shutsemau).
- Use REP MOVSB to replace the jump to memcpy_orig() on FSRM CPUs, keeping
  RET in ordinary text (Kiryl Shutsemau, Linus Torvalds). Remove the ITS
  replacement alignment, since no return is copied by alternatives now.

v1 -> v2: https://lore.kernel.org/r/20261013100442.2658311-1-usama.arif@linux.dev
- Raise the threshold from 512 to 2048 bytes. REP MOVSB was slower than
  the scalar loop for some misaligned copies of 512 to 640 bytes.
- Fold memcpy_orig() into __memcpy(). With FSRM, "rep movsb; RET" now
  replaces the 32-byte length check (David Laight, Linus Torvalds). In
  v1, the 13-byte empty FSRM replacement became a jump that FSRM CPUs
  took on every call (Kiryl Shutsemau).
- Do the ERMS check after the 32-byte check, so copies below 32 bytes
  skip it (Kiryl Shutsemau), and keep the count in RCX so REP MOVSB
  needs no reload (David Laight).
- Align the FSRM replacement so that its RET keeps the ITS return thunk.
- Fix the comment that called the return value the source input.
 
Usama Arif (2):
  x86/lib: Keep the scalar memcpy() count in RCX
  x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

 arch/x86/lib/memcpy_64.S | 78 +++++++++++++++++++++-------------------
 1 file changed, 42 insertions(+), 36 deletions(-)

-- 
2.53.0-Meta

[PATCH v3 1/2] x86/lib: Keep the scalar memcpy() count in RCX

[Usama Arif]

Use RCX for the scalar loop count and tail indexing, and RDX for the
small-copy data temporary. Initialize RCX from the incoming RDX count in
memcpy_orig().

This prepares the ERMS size check to use RDX as scratch while preserving
the REP MOVSB count in RCX. The FSRM path and copy policy stay the same.

No functional change.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/lib/memcpy_64.S | 59 ++++++++++++++++++++--------------------
 1 file changed, 30 insertions(+), 29 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 12a23fa7c44c5..7bb146c811029 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -46,8 +46,9 @@ EXPORT_SYMBOL(memcpy)
 
 SYM_FUNC_START_LOCAL(memcpy_orig)
 	movq %rdi, %rax
+	movq %rdx, %rcx
 
-	cmpq $0x20, %rdx
+	cmpq $0x20, %rcx
 	jb .Lhandle_tail
 
 	/*
@@ -56,9 +57,9 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	 */
 	cmp  %dil, %sil
 	jl .Lcopy_backward
-	subq $0x20, %rdx
+	subq $0x20, %rcx
 .Lcopy_forward_loop:
-	subq $0x20,	%rdx
+	subq $0x20,	%rcx
 
 	/*
 	 * Move in blocks of 4x8 bytes:
@@ -75,23 +76,23 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	movq %r11,	3*8(%rdi)
 	leaq 4*8(%rdi),	%rdi
 	jae  .Lcopy_forward_loop
-	addl $0x20,	%edx
+	addl $0x20,	%ecx
 	jmp  .Lhandle_tail
 
 .Lcopy_backward:
 	/*
 	 * Calculate copy position to tail.
 	 */
-	addq %rdx,	%rsi
-	addq %rdx,	%rdi
-	subq $0x20,	%rdx
+	addq %rcx,	%rsi
+	addq %rcx,	%rdi
+	subq $0x20,	%rcx
 	/*
 	 * At most 3 ALU operations in one cycle,
 	 * so append NOPS in the same 16 bytes trunk.
 	 */
 	.p2align 4
 .Lcopy_backward_loop:
-	subq $0x20,	%rdx
+	subq $0x20,	%rcx
 	movq -1*8(%rsi),	%r8
 	movq -2*8(%rsi),	%r9
 	movq -3*8(%rsi),	%r10
@@ -107,11 +108,11 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	/*
 	 * Calculate copy position to head.
 	 */
-	addl $0x20,	%edx
-	subq %rdx,	%rsi
-	subq %rdx,	%rdi
+	addl $0x20,	%ecx
+	subq %rcx,	%rsi
+	subq %rcx,	%rdi
 .Lhandle_tail:
-	cmpl $16,	%edx
+	cmpl $16,	%ecx
 	jb   .Lless_16bytes
 
 	/*
@@ -119,53 +120,53 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	 */
 	movq 0*8(%rsi), %r8
 	movq 1*8(%rsi),	%r9
-	movq -2*8(%rsi, %rdx),	%r10
-	movq -1*8(%rsi, %rdx),	%r11
+	movq -2*8(%rsi, %rcx),	%r10
+	movq -1*8(%rsi, %rcx),	%r11
 	movq %r8,	0*8(%rdi)
 	movq %r9,	1*8(%rdi)
-	movq %r10,	-2*8(%rdi, %rdx)
-	movq %r11,	-1*8(%rdi, %rdx)
+	movq %r10,	-2*8(%rdi, %rcx)
+	movq %r11,	-1*8(%rdi, %rcx)
 	RET
 	.p2align 4
 .Lless_16bytes:
-	cmpl $8,	%edx
+	cmpl $8,	%ecx
 	jb   .Lless_8bytes
 	/*
 	 * Move data from 8 bytes to 15 bytes.
 	 */
 	movq 0*8(%rsi),	%r8
-	movq -1*8(%rsi, %rdx),	%r9
+	movq -1*8(%rsi, %rcx),	%r9
 	movq %r8,	0*8(%rdi)
-	movq %r9,	-1*8(%rdi, %rdx)
+	movq %r9,	-1*8(%rdi, %rcx)
 	RET
 	.p2align 4
 .Lless_8bytes:
-	cmpl $4,	%edx
+	cmpl $4,	%ecx
 	jb   .Lless_3bytes
 
 	/*
 	 * Move data from 4 bytes to 7 bytes.
 	 */
-	movl (%rsi), %ecx
-	movl -4(%rsi, %rdx), %r8d
-	movl %ecx, (%rdi)
-	movl %r8d, -4(%rdi, %rdx)
+	movl (%rsi), %edx
+	movl -4(%rsi, %rcx), %r8d
+	movl %edx, (%rdi)
+	movl %r8d, -4(%rdi, %rcx)
 	RET
 	.p2align 4
 .Lless_3bytes:
-	subl $1, %edx
+	subl $1, %ecx
 	jb .Lend
 	/*
 	 * Move data from 1 bytes to 3 bytes.
 	 */
-	movzbl (%rsi), %ecx
+	movzbl (%rsi), %edx
 	jz .Lstore_1byte
 	movzbq 1(%rsi), %r8
-	movzbq (%rsi, %rdx), %r9
+	movzbq (%rsi, %rcx), %r9
 	movb %r8b, 1(%rdi)
-	movb %r9b, (%rdi, %rdx)
+	movb %r9b, (%rdi, %rcx)
 .Lstore_1byte:
-	movb %cl, (%rdi)
+	movb %dl, (%rdi)
 
 .Lend:
 	RET
-- 
2.53.0-Meta

[PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

Since commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies"), memcpy() uses REP MOVSB only on FSRM CPUs, so ERMS CPUs
without FSRM, such as Skylake and Cooper Lake, copy every size with
the scalar loop. Large copies matter there: in Meta's x86 fleet, about a
third of the time in memcpy_orig() is btrfs zstd decompression copying
pages into and out of zstd's buffers.

Use REP MOVSB from 2048 bytes on ERMS CPUs. On a 1.4 GHz Cooper Lake, an
aligned, cache-hot 4 KiB memcpy() then takes 66.8% less time (123.3 ns
instead of 371.6 ns). In a single-vCPU guest, reading a synthetic
512 MiB btrfs zstd file takes 14.8% less time (median of five kernel
pairs). Smaller copies keep the scalar loop: using REP MOVSB from
32 bytes made 32/64/128/256-byte copies
111.8%/68.0%/63.2%/15.6% slower.

Prepare the return value and RCX count before the FSRM alternative. On
FSRM CPUs, REP MOVSB replaces the jump to memcpy_orig(), followed by a
RET in ordinary text. ITS then selects return handling using the RET's
execution address.

For ERMS CPUs, shift the scratch copy of the length in RDX after the
32-byte check. RCX already holds the count, so large copies need no
reload. Copies below 32 bytes skip the check, which is a single NOP
without ERMS.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/lib/memcpy_64.S | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 7bb146c811029..5320862e0274e 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -27,15 +27,13 @@
  *
  * The 'rep movsb' itself is small enough to replace the call, but the
  * two register moves blow up the code. And one of them is "needed"
- * only for the return value that is the same as the source input,
+ * only for the return value that is the same as the destination input,
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
-
 	movq %rdi, %rax
 	movq %rdx, %rcx
-	rep movsb
+	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
 	RET
 SYM_FUNC_END(__memcpy)
 EXPORT_SYMBOL(__memcpy)
@@ -45,12 +43,16 @@ SYM_PIC_ALIAS(memcpy)
 EXPORT_SYMBOL(memcpy)
 
 SYM_FUNC_START_LOCAL(memcpy_orig)
-	movq %rdi, %rax
-	movq %rdx, %rcx
-
 	cmpq $0x20, %rcx
 	jb .Lhandle_tail
 
+	/*
+	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
+	 * Shift RDX to check length >= 2048 without destroying the REP count.
+	 * The 10-byte alternative fits a NOP without ERMS.
+	 */
+	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS
+
 	/*
 	 * We check whether memory false dependence could occur,
 	 * then jump to corresponding copy mode.
@@ -170,5 +172,8 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 
 .Lend:
 	RET
-SYM_FUNC_END(memcpy_orig)
 
+.Lerms:
+	rep movsb
+	RET
+SYM_FUNC_END(memcpy_orig)
-- 
2.53.0-Meta

Re: [PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Peter Zijlstra]

On Wed, Nov 11, 2026 at 05:20:53AM -0800, Usama Arif wrote:
> Since commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
> memory copies"), memcpy() uses REP MOVSB only on FSRM CPUs, so ERMS CPUs
> without FSRM, such as Skylake and Cooper Lake, copy every size with
> the scalar loop. Large copies matter there: in Meta's x86 fleet, about a
> third of the time in memcpy_orig() is btrfs zstd decompression copying
> pages into and out of zstd's buffers.
> 
> Use REP MOVSB from 2048 bytes on ERMS CPUs. On a 1.4 GHz Cooper Lake, an
> aligned, cache-hot 4 KiB memcpy() then takes 66.8% less time (123.3 ns
> instead of 371.6 ns). In a single-vCPU guest, reading a synthetic
> 512 MiB btrfs zstd file takes 14.8% less time (median of five kernel
> pairs). Smaller copies keep the scalar loop: using REP MOVSB from
> 32 bytes made 32/64/128/256-byte copies
> 111.8%/68.0%/63.2%/15.6% slower.
> 
> Prepare the return value and RCX count before the FSRM alternative. On
> FSRM CPUs, REP MOVSB replaces the jump to memcpy_orig(), followed by a
> RET in ordinary text. ITS then selects return handling using the RET's
> execution address.
> 
> For ERMS CPUs, shift the scratch copy of the length in RDX after the
> 32-byte check. RCX already holds the count, so large copies need no
> reload. Copies below 32 bytes skip the check, which is a single NOP
> without ERMS.

So no real objection to these patches from me.

However, it did remind me of something I have on the TODO list for a
rainy afternoon, namely merge memcpy and rep_movs_alternative.

It makes very little sense to me that we have two (divergent) copies of
what is basically the same thing.

Re: [PATCH v3 0/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Kiryl Shutsemau]

On Wed, Nov 11, 2026 at 05:20:51AM -0800, Usama Arif wrote:
> Enable REP MOVSB for copies of 2048 bytes and larger on ERMS CPUs without
> FSRM, keeping scalar copies below that threshold. FSRM CPUs continue to use
> REP MOVSB for every size.
> 
> This series splits v2 into 2 patches.
> The first changes only the scalar register allocation: RCX holds the count
> and tail indexes, and RDX is the small-copy data temporary. The second
> adds the ERMS size check and adopts Linus's FSRM entry layout. REP MOVSB
> replaces the existing jump to memcpy_orig(), with RET remaining in
> ordinary text. ITS therefore selects return handling using the return's
> actual execution address, without aligning replacement storage. Keeping
> the count in RCX also addresses David's request to avoid reloading it on
> the ERMS path.
> 
> v2 -> v3: https://lore.kernel.org/r/20261104222806.1826508-1-usama.arif@linux.dev
> - Split the scalar RCX/RDX register swap into a preparatory patch with no
>   functional change (Kiryl Shutsemau).
> - Use REP MOVSB to replace the jump to memcpy_orig() on FSRM CPUs, keeping
>   RET in ordinary text (Kiryl Shutsemau, Linus Torvalds). Remove the ITS
>   replacement alignment, since no return is copied by alternatives now.
> 
> v1 -> v2: https://lore.kernel.org/r/20261013100442.2658311-1-usama.arif@linux.dev
> - Raise the threshold from 512 to 2048 bytes. REP MOVSB was slower than
>   the scalar loop for some misaligned copies of 512 to 640 bytes.
> - Fold memcpy_orig() into __memcpy(). With FSRM, "rep movsb; RET" now
>   replaces the 32-byte length check (David Laight, Linus Torvalds). In
>   v1, the 13-byte empty FSRM replacement became a jump that FSRM CPUs
>   took on every call (Kiryl Shutsemau).
> - Do the ERMS check after the 32-byte check, so copies below 32 bytes
>   skip it (Kiryl Shutsemau), and keep the count in RCX so REP MOVSB
>   needs no reload (David Laight).
> - Align the FSRM replacement so that its RET keeps the ITS return thunk.
> - Fix the comment that called the return value the source input.
>  
> Usama Arif (2):
>   x86/lib: Keep the scalar memcpy() count in RCX
>   x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM
> 
>  arch/x86/lib/memcpy_64.S | 78 +++++++++++++++++++++-------------------
>  1 file changed, 42 insertions(+), 36 deletions(-)

Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>

-- 
  Kiryl Shutsemau / Kirill A. Shutemov