[PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies") stopped selecting REP MOVSB from ERMS alone because it
is not suitable for short copies. Consequently, ordinary out-of-line
__memcpy() calls on CPUs with ERMS but without FSRM use memcpy_orig()
for every length. Its main loop copies 32 bytes per iteration and can be
slower than REP MOVSB for large copies, as measured below on Cooper Lake.

This matters for btrfs zstd decompression. The btrfs-tree commit
e27a6e4823f3 ("btrfs: zstd: avoid a copy in zstd_decompress_bio()")
removes btrfs' copy from a scratch buffer into the page cache by letting
zstd write there directly. It does not remove the copies inside
streaming zstd. ZSTD_decompressStream() still uses memcpy() to flush
decompressed data from its internal output buffer to the supplied page
(zdss_flush), and to stage compressed input when the next block crosses
an input folio supplied by btrfs (zdss_load). In Meta's fleet profile
of x86 kernels, these two btrfs/zstd call sites accounted for about 34%
of the sampled profile weight attributed to memcpy_orig().

On CPUs without FSRM, select REP MOVSB for copies of 512 bytes and above
when ERMS is present. Keep memcpy_orig() below 512 bytes. The later FSRM
alternative continues to select REP MOVSB for all sizes, including when
ERMS is not exposed.

Userspace replicas of the original and patched entry paths were run on
a turbo-disabled Xeon Platinum 8321HC (Cooper Lake), capped at
1.401 GHz. Results average two runs and scale the per-call time to
65,536 4 KB calls, or 256 MB in aggregate.

Here "hot" means cache-resident. The rows model zdss_flush() from
zstd's L2-resident output buffer, zdss_load() from a cold compressed
folio into zstd's hot input buffer, and a page copy into a cold
destination.

                                    orig  REP MOVSB  reduction
  zdss_flush: L2 -> hot page     28.0 ms   11.9 ms      57.5%
  zdss_load: cold -> hot buffer  46.1 ms   28.9 ms      37.3%
  page copy: hot -> cold page    29.8 ms   22.5 ms      24.5%

REP MOVSB was faster in every tested scenario at 512 bytes and above.
Copies below 512 bytes showed no consistent performance change across the
possible function alignments.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
The above benchmarks to model copies in zstd were created using LLM:
https://gist.github.com/uarif1/8409ccd34284f128dbea5822ec628ef3
---
 arch/x86/lib/memcpy_64.S | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 12a23fa7c44c5..a7b3ce4fae0b4 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -31,7 +31,9 @@
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
+	ALTERNATIVE_2 "jmp memcpy_orig", \
+		      "cmpq $512, %rdx; jb memcpy_orig", X86_FEATURE_ERMS, \
+		      "", X86_FEATURE_FSRM
 
 	movq %rdi, %rax
 	movq %rdx, %rcx
-- 
2.53.0-Meta

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Kiryl Shutsemau]

On Tue, Oct 13, 2026 at 03:04:42AM -0700, Usama Arif wrote:
>  SYM_TYPED_FUNC_START(__memcpy)
> -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> +	ALTERNATIVE_2 "jmp memcpy_orig", \
> +		      "cmpq $512, %rdx; jb memcpy_orig", X86_FEATURE_ERMS, \
> +		      "", X86_FEATURE_FSRM

This grows the patch site from 2 bytes to 13. On FSRM the replacement is
empty, and 13 bytes is more than ASM_NOP_MAX, so add_nop() puts an
unconditional jump here, executed on every memcpy() call on FSRM
machines. :/

Maybe it is worth moving the patch site into memcpy_orig()? It already
has a 32-byte check. Which brings me to the other question:

Why 512? memmove() uses REP MOVSB from 32 bytes on ERMS. Can we do the
same here? Then we don't need a new branch at all.

--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -50,6 +50,8 @@
 	cmpq $0x20, %rdx
 	jb .Lhandle_tail
 
+	ALTERNATIVE "", __stringify(movq %rdx, %rcx; rep movsb; RET), X86_FEATURE_ERMS
+
 	/*
 	 * We check whether memory false dependence could occur,
 	 * then jump to corresponding copy mode.
-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

On Thu, 15 Oct 2026 11:39:10 +0100 Kiryl Shutsemau <kas@kernel.org> wrote:

> On Tue, Oct 13, 2026 at 03:04:42AM -0700, Usama Arif wrote:
> >  SYM_TYPED_FUNC_START(__memcpy)
> > -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> > +	ALTERNATIVE_2 "jmp memcpy_orig", \
> > +		      "cmpq $512, %rdx; jb memcpy_orig", X86_FEATURE_ERMS, \
> > +		      "", X86_FEATURE_FSRM
> 
> This grows the patch site from 2 bytes to 13. On FSRM the replacement is
> empty, and 13 bytes is more than ASM_NOP_MAX, so add_nop() puts an
> unconditional jump here, executed on every memcpy() call on FSRM
> machines. :/
> 

Thanks for the review and catching the FSRM padding jump! You're right,
the 13-byte empty replacement introduces an unconditional jump on every call.

> Maybe it is worth moving the patch site into memcpy_orig()? It already
> has a 32-byte check. Which brings me to the other question:

I tested the 32-byte cutoff on Cooper Lake, which has ERMS without FSRM.
With warmed, aligned buffers, 32-byte copies took 10.906 ns with the
parent and 23.148 ns with the proposal (+112%). The increases at 64, 128 and
256 bytes were about 66%, 65% and 22%. These regressions reproduced across
five process runs and all four tested entry placements.
 
I found 2K to be the spot where REP MOVSB is better. For the entry, I moved
the destination save into the FSRM alternative:

ALTERNATIVE_2 "jmp memcpy_orig", "jmp memcpy_erms", X86_FEATURE_ERMS, \
              "movq %rdi, %rax", X86_FEATURE_FSRM

The ordinary entry then only needs MOV to RCX, REP MOVSB and RET. This gives
FSRM a destination move plus one two-byte NOP, without a padding jump, and
keeps the entry size and scalar layout. A separate memcpy_erms helper checks
2048 and falls back to memcpy_orig below that size. Returns stay in normal
text.

Below is the current state of my patch. What do you think?

From cff8b5af45ec9ed10a6884e9c251f0e58147b40e Mon Sep 17 00:00:00 2001
From: Usama Arif <usama.arif@linux.dev>
Date: Thu, 1 Oct 2026 14:15:12 -0700
Subject: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies") stopped selecting REP MOVSB from ERMS alone because it
is not suitable for short copies. Consequently, ordinary out-of-line
__memcpy() calls on CPUs with ERMS but without FSRM such as Cooper Lakes
and Skylakes use memcpy_orig() for every length. Its main loop copies
32 bytes per iteration and can be slower than REP MOVSB for large copies,
as measured below on Cooper Lake.

This matters for btrfs zstd decompression. The btrfs-tree commit
e27a6e4823f3 ("btrfs: zstd: avoid a copy in zstd_decompress_bio()")
removes btrfs' copy from a scratch buffer into the page cache by letting
zstd write there directly. It does not remove the copies inside
streaming zstd. ZSTD_decompressStream() still uses memcpy() to flush
decompressed data from its internal output buffer to the supplied page
(zdss_flush), and to stage compressed input when the next block crosses
an input folio supplied by btrfs (zdss_load). In Meta's fleet profile
of x86 kernels, these two btrfs/zstd call sites accounted for about 34%
of the sampled profile weight attributed to memcpy_orig().

On CPUs without FSRM, select REP MOVSB for copies of 2048 bytes and above
when ERMS is present. Keep memcpy_orig() below 2048 bytes. The later FSRM
alternative continues to select REP MOVSB for all sizes, including when
ERMS is not exposed. Copies below 2048 bytes were showing regression
with REP MOVSB as seen below (medians from five runs):

  bytes          orig    REP MOVSB   increase
     32       10.9 ns      23.1 ns     112.3%
     64       13.9 ns      23.1 ns      66.3%
    128       17.5 ns      28.9 ns      65.2%
    256       26.0 ns      31.7 ns      22.1%

Dispatch ERMS through a separate helper and put the destination save in
the FSRM replacement. This keeps the entry alternative at five bytes and
preserves memcpy_orig()'s instructions and placement. FSRM executes the
destination move and one two-byte NOP before its direct REP MOVSB path.

Userspace replicas of the original and patched entry paths were run on
a turbo-disabled Xeon Platinum 8321HC (Cooper Lake, ERMS without FSRM),
at 1.4 GHz. Results below are median times per call from five process
runs of aligned 4 KiB copies at entry offset zero. The replicas use
plain RET; these are assembly microbenchmark results.

Here "warmed" means repeatedly using both buffers. The other rows flush
every cache line intersecting the source or destination before timing.

                                    orig    patched  reduction
  warmed                        371.6 ns   123.4 ns      66.8%
  source flushed                706.5 ns   400.8 ns      43.3%
  destination flushed           457.7 ns   350.0 ns      23.5%

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/lib/memcpy_64.S | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 12a23fa7c44c5..6b3e203b6087c 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -31,9 +31,10 @@
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
+	/* The FSRM destination move shares the space used by the ERMS jump. */
+	ALTERNATIVE_2 "jmp memcpy_orig", "jmp memcpy_erms", X86_FEATURE_ERMS, \
+		      "movq %rdi, %rax", X86_FEATURE_FSRM
 
-	movq %rdi, %rax
 	movq %rdx, %rcx
 	rep movsb
 	RET
@@ -171,3 +172,11 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 	RET
 SYM_FUNC_END(memcpy_orig)
 
+SYM_FUNC_START_LOCAL(memcpy_erms)
+	cmpq $2048, %rdx
+	jb memcpy_orig
+	movq %rdi, %rax
+	movq %rdx, %rcx
+	rep movsb
+	RET
+SYM_FUNC_END(memcpy_erms)
-- 
2.53.0-Meta


> 
> Why 512? memmove() uses REP MOVSB from 32 bytes on ERMS. Can we do the
> same here? Then we don't need a new branch at all.
> 
> --- a/arch/x86/lib/memcpy_64.S
> +++ b/arch/x86/lib/memcpy_64.S
> @@ -50,6 +50,8 @@
>  	cmpq $0x20, %rdx
>  	jb .Lhandle_tail
>  
> +	ALTERNATIVE "", __stringify(movq %rdx, %rcx; rep movsb; RET), X86_FEATURE_ERMS
> +
>  	/*
>  	 * We check whether memory false dependence could occur,
>  	 * then jump to corresponding copy mode.
> -- 
>   Kiryl Shutsemau / Kirill A. Shutemov
> 

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Kiryl Shutsemau]

On Sat, Oct 17, 2026 at 12:27:25PM -0700, Usama Arif wrote:
> diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> index 12a23fa7c44c5..6b3e203b6087c 100644
> --- a/arch/x86/lib/memcpy_64.S
> +++ b/arch/x86/lib/memcpy_64.S
> @@ -31,9 +31,10 @@
>   * which the compiler could/should do much better anyway.
>   */
>  SYM_TYPED_FUNC_START(__memcpy)
> -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> +	/* The FSRM destination move shares the space used by the ERMS jump. */
> +	ALTERNATIVE_2 "jmp memcpy_orig", "jmp memcpy_erms", X86_FEATURE_ERMS, \
> +		      "movq %rdi, %rax", X86_FEATURE_FSRM
>  
> -	movq %rdi, %rax

Moving MOVQ inside ALTERNATIVE_2 is ugly...

And it only turns a 5-byte NOP into a 2-byte one on FSRM.
What is the point?

>  	movq %rdx, %rcx
>  	rep movsb
>  	RET
> @@ -171,3 +172,11 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
>  	RET
>  SYM_FUNC_END(memcpy_orig)
>  
> +SYM_FUNC_START_LOCAL(memcpy_erms)
> +	cmpq $2048, %rdx
> +	jb memcpy_orig
> +	movq %rdi, %rax
> +	movq %rdx, %rcx
> +	rep movsb
> +	RET
> +SYM_FUNC_END(memcpy_erms)

Copies below 2048 now take two jumps to reach memcpy_orig() instead of
one, and David's point is that nearly all kernel copies are that small.

Can the check go in memcpy_orig() after the existing 32-byte compare?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Linus Torvalds]

On Tue, 20 Oct 2026 at 23:55, Kiryl Shutsemau <kas@kernel.org> wrote:
>
> Moving MOVQ inside ALTERNATIVE_2 is ugly...

Yeah, maybe it would make sense to make that function be

        movq %rdi, %rax
        movq %rdx, %rcx
        ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
        RET

instead? No nop instructions at all, and both alternatives are two bytes.

Then do any ERMS logic at the top of memcpy_orig, which now has the
registers ready for string instructions anyway.

Hmm?

                      Linus

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[David Laight]

On Wed, 21 Oct 2026 08:18:25 -1000
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Tue, 20 Oct 2026 at 23:55, Kiryl Shutsemau <kas@kernel.org> wrote:
> >
> > Moving MOVQ inside ALTERNATIVE_2 is ugly...  
> 
> Yeah, maybe it would make sense to make that function be
> 
>         movq %rdi, %rax
>         movq %rdx, %rcx
>         ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
>         RET
> 
> instead? No nop instructions at all, and both alternatives are two bytes.
> 
> Then do any ERMS logic at the top of memcpy_orig, which now has the
> registers ready for string instructions anyway.

Is it possibly to just patch "rep movsb; RET" over whatever gets generated
for memcpy_orig (after the two register moves) ?
Then there are neither nops nor a jmp.

David

> 
> Hmm?
> 
>                       Linus
> 

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

On 22/10/2026 10:49, David Laight wrote:
> On Wed, 21 Oct 2026 08:18:25 -1000
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
>> On Tue, 20 Oct 2026 at 23:55, Kiryl Shutsemau <kas@kernel.org> wrote:
>>>
>>> Moving MOVQ inside ALTERNATIVE_2 is ugly...  
>>
>> Yeah, maybe it would make sense to make that function be
>>
>>         movq %rdi, %rax
>>         movq %rdx, %rcx
>>         ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
>>         RET
>>
>> instead? No nop instructions at all, and both alternatives are two bytes.
>>
>> Then do any ERMS logic at the top of memcpy_orig, which now has the
>> registers ready for string instructions anyway.
> 
> Is it possibly to just patch "rep movsb; RET" over whatever gets generated
> for memcpy_orig (after the two register moves) ?
> Then there are neither nops nor a jmp.
> 
> David
> 
Hi,

Sorry for the late reply. I had taken time off for the past couple of weeks
and didnt have access to my work station.

I implemented Davids suggestion and I think it looks good. The scalar body is
now part of __memcpy(), and FSRM replaces the existing 32-byte check with
"rep movsb; RET". Non-FSRM calls fall straight into scalar code, and FSRM
returns before any padding.

I have retained the 2048-byte ERMS cutoff, since selecting REP from 32 bytes
regressed short copies in the Cooper Lake tests.

I asked LLM to review it and it indicated a problem with ITS: returns are
patched before alternatives are copied, so the return decision uses the
replacement's source address. I align replacement storage outside the
copied range so ITS retains the return thunk and relocation preserves its
target. This adds no executed padding.

Any thoughts on below? I will send it as a proper patch for review
if it looks acceptable.


From 9baffefb71c7223702c7bcd96d43b9a2828f3939 Mon Sep 17 00:00:00 2001
From: Usama Arif <usama.arif@linux.dev>
Date: Thu, 1 Oct 2026 14:15:12 -0700
Subject: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs
 without FSRM

Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies") left ERMS CPUs without FSRM, including Cooper Lake and
Skylake, using scalar code for all out-of-line copies. Btrfs/zstd input
staging and output flushing accounted for about 34% of memcpy_orig()'s
sampled weight in Meta's x86 fleet profile.

Use REP MOVSB from 2048 bytes on ERMS CPUs without FSRM, keeping scalar
copies below that threshold. FSRM continues to use REP for all sizes.

Merge scalar code into __memcpy() and overlay its existing 32-byte check
with REP MOVSB and RET on FSRM, removing the non-FSRM entry jump. Copies
below 32 bytes bypass the ERMS gate. SHR11/JNZ fits one ten-byte NOP
without ERMS; the REP arm reloads the original count from RDX.

With ITS configured, align replacement storage before its start label so
the raw FSRM return retains its thunk before relocation. The alignment
padding is outside the copied range.

On Cooper Lake (boost disabled, capped at 1.401 GHz), plain-RET userspace
replicas with aligned warmed buffers and entry offset zero modulo 64
measured 4 KiB copies at 371.7 ns originally and 123.4 ns with this patch:
a 66.8% paired reduction across five process runs. A separate sweep using
REP from 32 bytes increased 32/64/128/256-byte times by
114.9%/54.4%/66.1%/23.2%, respectively.

Small-copy costs remain: a warmed aligned zero-byte copy at entry offset
48 modulo 64 took a paired 0.75 ns more (+7.9%) than the original, in all
five runs. These microbenchmarks do not establish a kernel workload gain
or the best threshold across CPUs.

Direct copy checks and memcpy KUnit passed with all four ERMS/FSRM
combinations and with ITS forced on; the copied return target was verified.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/lib/memcpy_64.S | 42 ++++++++++++++++++++++++++--------------
 1 file changed, 27 insertions(+), 15 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 12a23fa7c44c5..407b226d4de2f 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -31,24 +31,27 @@
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
-
 	movq %rdi, %rax
 	movq %rdx, %rcx
-	rep movsb
-	RET
-SYM_FUNC_END(__memcpy)
-EXPORT_SYMBOL(__memcpy)
-
-SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
-SYM_PIC_ALIAS(memcpy)
-EXPORT_SYMBOL(memcpy)
 
-SYM_FUNC_START_LOCAL(memcpy_orig)
-	movq %rdi, %rax
+#ifdef CONFIG_MITIGATION_ITS
+	/*
+	 * Returns are patched before alternatives. Keep the replacement
+	 * return in the first half of a cache line so ITS retains its thunk
+	 * when the instruction is copied to its final address.
+	 */
+	.pushsection .altinstr_replacement, "ax"
+	.balign 64
+	.popsection
+#endif
+	ALTERNATIVE __stringify(cmpq $0x20, %rdx; jb .Lhandle_tail), \
+		    __stringify(rep movsb; RET), X86_FEATURE_FSRM
 
-	cmpq $0x20, %rdx
-	jb .Lhandle_tail
+	/*
+	 * Prepared RCX is scratch for checking length >= 2048.
+	 * The 10-byte alternative fits a NOP without ERMS.
+	 */
+	ALTERNATIVE "", __stringify(shrq $11, %rcx; jnz .Lerms), X86_FEATURE_ERMS
 
 	/*
 	 * We check whether memory false dependence could occur,
@@ -169,5 +172,14 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 
 .Lend:
 	RET
-SYM_FUNC_END(memcpy_orig)
 
+.Lerms:
+	movq %rdx, %rcx
+	rep movsb
+	RET
+SYM_FUNC_END(__memcpy)
+EXPORT_SYMBOL(__memcpy)
+
+SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
+SYM_PIC_ALIAS(memcpy)
+EXPORT_SYMBOL(memcpy)
-- 
2.53.0-Meta

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[David Laight]

On Mon, 2 Nov 2026 23:03:39 +0000
Usama Arif <usama.arif@linux.dev> wrote:

> On 22/10/2026 10:49, David Laight wrote:
> > On Wed, 21 Oct 2026 08:18:25 -1000
> > Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >   
> >> On Tue, 20 Oct 2026 at 23:55, Kiryl Shutsemau <kas@kernel.org> wrote:  
> >>>
> >>> Moving MOVQ inside ALTERNATIVE_2 is ugly...    
> >>
> >> Yeah, maybe it would make sense to make that function be
> >>
> >>         movq %rdi, %rax
> >>         movq %rdx, %rcx
> >>         ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
> >>         RET
> >>
> >> instead? No nop instructions at all, and both alternatives are two bytes.
> >>
> >> Then do any ERMS logic at the top of memcpy_orig, which now has the
> >> registers ready for string instructions anyway.  
> > 
> > Is it possibly to just patch "rep movsb; RET" over whatever gets generated
> > for memcpy_orig (after the two register moves) ?
> > Then there are neither nops nor a jmp.
> > 
> > David
> >   
> Hi,
> 
> Sorry for the late reply. I had taken time off for the past couple of weeks
> and didnt have access to my work station.
> 
> I implemented Davids suggestion and I think it looks good. The scalar body is
> now part of __memcpy(), and FSRM replaces the existing 32-byte check with
> "rep movsb; RET". Non-FSRM calls fall straight into scalar code, and FSRM
> returns before any padding.
> 
> I have retained the 2048-byte ERMS cutoff, since selecting REP from 32 bytes
> regressed short copies in the Cooper Lake tests.
> 
> I asked LLM to review it and it indicated a problem with ITS: returns are
> patched before alternatives are copied, so the return decision uses the
> replacement's source address. I align replacement storage outside the
> copied range so ITS retains the return thunk and relocation preserves its
> target. This adds no executed padding.
> 
> Any thoughts on below? I will send it as a proper patch for review
> if it looks acceptable.
> 
> 
> From 9baffefb71c7223702c7bcd96d43b9a2828f3939 Mon Sep 17 00:00:00 2001
> From: Usama Arif <usama.arif@linux.dev>
> Date: Thu, 1 Oct 2026 14:15:12 -0700
> Subject: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs
>  without FSRM
> 
> Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
> memory copies") left ERMS CPUs without FSRM, including Cooper Lake and
> Skylake, using scalar code for all out-of-line copies. Btrfs/zstd input
> staging and output flushing accounted for about 34% of memcpy_orig()'s
> sampled weight in Meta's x86 fleet profile.
> 
> Use REP MOVSB from 2048 bytes on ERMS CPUs without FSRM, keeping scalar
> copies below that threshold. FSRM continues to use REP for all sizes.
> 
> Merge scalar code into __memcpy() and overlay its existing 32-byte check
> with REP MOVSB and RET on FSRM, removing the non-FSRM entry jump. Copies
> below 32 bytes bypass the ERMS gate. SHR11/JNZ fits one ten-byte NOP
> without ERMS; the REP arm reloads the original count from RDX.
> 
> With ITS configured, align replacement storage before its start label so
> the raw FSRM return retains its thunk before relocation. The alignment
> padding is outside the copied range.
> 
> On Cooper Lake (boost disabled, capped at 1.401 GHz), plain-RET userspace
> replicas with aligned warmed buffers and entry offset zero modulo 64
> measured 4 KiB copies at 371.7 ns originally and 123.4 ns with this patch:
> a 66.8% paired reduction across five process runs. A separate sweep using
> REP from 32 bytes increased 32/64/128/256-byte times by
> 114.9%/54.4%/66.1%/23.2%, respectively.
> 
> Small-copy costs remain: a warmed aligned zero-byte copy at entry offset
> 48 modulo 64 took a paired 0.75 ns more (+7.9%) than the original, in all
> five runs. These microbenchmarks do not establish a kernel workload gain
> or the best threshold across CPUs.
> 
> Direct copy checks and memcpy KUnit passed with all four ERMS/FSRM
> combinations and with ITS forced on; the copied return target was verified.
> 
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
>  arch/x86/lib/memcpy_64.S | 42 ++++++++++++++++++++++++++--------------
>  1 file changed, 27 insertions(+), 15 deletions(-)
> 
> diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> index 12a23fa7c44c5..407b226d4de2f 100644
> --- a/arch/x86/lib/memcpy_64.S
> +++ b/arch/x86/lib/memcpy_64.S
> @@ -31,24 +31,27 @@
>   * which the compiler could/should do much better anyway.
>   */
>  SYM_TYPED_FUNC_START(__memcpy)
> -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> -
>  	movq %rdi, %rax
>  	movq %rdx, %rcx
> -	rep movsb
> -	RET
> -SYM_FUNC_END(__memcpy)
> -EXPORT_SYMBOL(__memcpy)
> -
> -SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
> -SYM_PIC_ALIAS(memcpy)
> -EXPORT_SYMBOL(memcpy)
>  
> -SYM_FUNC_START_LOCAL(memcpy_orig)
> -	movq %rdi, %rax
> +#ifdef CONFIG_MITIGATION_ITS
> +	/*
> +	 * Returns are patched before alternatives. Keep the replacement
> +	 * return in the first half of a cache line so ITS retains its thunk
> +	 * when the instruction is copied to its final address.

I'm not 100% sure what that is about, but does it mean that the cache line
alignment of the code here also matters?

> +	 */
> +	.pushsection .altinstr_replacement, "ax"
> +	.balign 64
> +	.popsection
> +#endif
> +	ALTERNATIVE __stringify(cmpq $0x20, %rdx; jb .Lhandle_tail), \
> +		    __stringify(rep movsb; RET), X86_FEATURE_FSRM
>  
> -	cmpq $0x20, %rdx
> -	jb .Lhandle_tail
> +	/*
> +	 * Prepared RCX is scratch for checking length >= 2048.
> +	 * The 10-byte alternative fits a NOP without ERMS.
> +	 */
> +	ALTERNATIVE "", __stringify(shrq $11, %rcx; jnz .Lerms), X86_FEATURE_ERMS

Can you use %rdx here, preserving %rcx for the 'rep movsb'?
You might need to swap cx<=>dx in the rest of the function.
(I'm sure we could use #defines to give names to registers....)

David

>  	/*
>  	 * We check whether memory false dependence could occur,
> @@ -169,5 +172,14 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
>  
>  .Lend:
>  	RET
> -SYM_FUNC_END(memcpy_orig)
>  
> +.Lerms:
> +	movq %rdx, %rcx
> +	rep movsb
> +	RET
> +SYM_FUNC_END(__memcpy)
> +EXPORT_SYMBOL(__memcpy)
> +
> +SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
> +SYM_PIC_ALIAS(memcpy)
> +EXPORT_SYMBOL(memcpy)

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif]

On 03/11/2026 09:32, David Laight wrote:
> On Mon, 2 Nov 2026 23:03:39 +0000
> Usama Arif <usama.arif@linux.dev> wrote:
> 
>> On 22/10/2026 10:49, David Laight wrote:
>>> On Wed, 21 Oct 2026 08:18:25 -1000
>>> Linus Torvalds <torvalds@linux-foundation.org> wrote:
>>>   
>>>> On Tue, 20 Oct 2026 at 23:55, Kiryl Shutsemau <kas@kernel.org> wrote:  
>>>>>
>>>>> Moving MOVQ inside ALTERNATIVE_2 is ugly...    
>>>>
>>>> Yeah, maybe it would make sense to make that function be
>>>>
>>>>         movq %rdi, %rax
>>>>         movq %rdx, %rcx
>>>>         ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
>>>>         RET
>>>>
>>>> instead? No nop instructions at all, and both alternatives are two bytes.
>>>>
>>>> Then do any ERMS logic at the top of memcpy_orig, which now has the
>>>> registers ready for string instructions anyway.  
>>>
>>> Is it possibly to just patch "rep movsb; RET" over whatever gets generated
>>> for memcpy_orig (after the two register moves) ?
>>> Then there are neither nops nor a jmp.
>>>
>>> David
>>>   
>> Hi,
>>
>> Sorry for the late reply. I had taken time off for the past couple of weeks
>> and didnt have access to my work station.
>>
>> I implemented Davids suggestion and I think it looks good. The scalar body is
>> now part of __memcpy(), and FSRM replaces the existing 32-byte check with
>> "rep movsb; RET". Non-FSRM calls fall straight into scalar code, and FSRM
>> returns before any padding.
>>
>> I have retained the 2048-byte ERMS cutoff, since selecting REP from 32 bytes
>> regressed short copies in the Cooper Lake tests.
>>
>> I asked LLM to review it and it indicated a problem with ITS: returns are
>> patched before alternatives are copied, so the return decision uses the
>> replacement's source address. I align replacement storage outside the
>> copied range so ITS retains the return thunk and relocation preserves its
>> target. This adds no executed padding.
>>
>> Any thoughts on below? I will send it as a proper patch for review
>> if it looks acceptable.
>>
>>
>> From 9baffefb71c7223702c7bcd96d43b9a2828f3939 Mon Sep 17 00:00:00 2001
>> From: Usama Arif <usama.arif@linux.dev>
>> Date: Thu, 1 Oct 2026 14:15:12 -0700
>> Subject: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs
>>  without FSRM
>>
>> Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
>> memory copies") left ERMS CPUs without FSRM, including Cooper Lake and
>> Skylake, using scalar code for all out-of-line copies. Btrfs/zstd input
>> staging and output flushing accounted for about 34% of memcpy_orig()'s
>> sampled weight in Meta's x86 fleet profile.
>>
>> Use REP MOVSB from 2048 bytes on ERMS CPUs without FSRM, keeping scalar
>> copies below that threshold. FSRM continues to use REP for all sizes.
>>
>> Merge scalar code into __memcpy() and overlay its existing 32-byte check
>> with REP MOVSB and RET on FSRM, removing the non-FSRM entry jump. Copies
>> below 32 bytes bypass the ERMS gate. SHR11/JNZ fits one ten-byte NOP
>> without ERMS; the REP arm reloads the original count from RDX.
>>
>> With ITS configured, align replacement storage before its start label so
>> the raw FSRM return retains its thunk before relocation. The alignment
>> padding is outside the copied range.
>>
>> On Cooper Lake (boost disabled, capped at 1.401 GHz), plain-RET userspace
>> replicas with aligned warmed buffers and entry offset zero modulo 64
>> measured 4 KiB copies at 371.7 ns originally and 123.4 ns with this patch:
>> a 66.8% paired reduction across five process runs. A separate sweep using
>> REP from 32 bytes increased 32/64/128/256-byte times by
>> 114.9%/54.4%/66.1%/23.2%, respectively.
>>
>> Small-copy costs remain: a warmed aligned zero-byte copy at entry offset
>> 48 modulo 64 took a paired 0.75 ns more (+7.9%) than the original, in all
>> five runs. These microbenchmarks do not establish a kernel workload gain
>> or the best threshold across CPUs.
>>
>> Direct copy checks and memcpy KUnit passed with all four ERMS/FSRM
>> combinations and with ITS forced on; the copied return target was verified.
>>
>> Signed-off-by: Usama Arif <usama.arif@linux.dev>
>> ---
>>  arch/x86/lib/memcpy_64.S | 42 ++++++++++++++++++++++++++--------------
>>  1 file changed, 27 insertions(+), 15 deletions(-)
>>
>> diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
>> index 12a23fa7c44c5..407b226d4de2f 100644
>> --- a/arch/x86/lib/memcpy_64.S
>> +++ b/arch/x86/lib/memcpy_64.S
>> @@ -31,24 +31,27 @@
>>   * which the compiler could/should do much better anyway.
>>   */
>>  SYM_TYPED_FUNC_START(__memcpy)
>> -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
>> -
>>  	movq %rdi, %rax
>>  	movq %rdx, %rcx
>> -	rep movsb
>> -	RET
>> -SYM_FUNC_END(__memcpy)
>> -EXPORT_SYMBOL(__memcpy)
>> -
>> -SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
>> -SYM_PIC_ALIAS(memcpy)
>> -EXPORT_SYMBOL(memcpy)
>>  
>> -SYM_FUNC_START_LOCAL(memcpy_orig)
>> -	movq %rdi, %rax
>> +#ifdef CONFIG_MITIGATION_ITS
>> +	/*
>> +	 * Returns are patched before alternatives. Keep the replacement
>> +	 * return in the first half of a cache line so ITS retains its thunk
>> +	 * when the instruction is copied to its final address.
> 
> I'm not 100% sure what that is about, but does it mean that the cache line
> alignment of the code here also matters?

Not the alignment of __memcpy() itself, only that of the replacement in
.altinstr_replacement.

On ITS affected CPUs, a RET in the lower 32 bytes of a 64-byte cache
line needs mitigation, so apply_returns() leaves such sites as a jump to
its_return_thunk, whose own RET is in the upper half, and turns the rest
into a plain RET. It runs before apply_alternatives(), so for the RET in
the FSRM replacement it decides from the replacement's address, not from
where the code is copied to. A plain RET could then end up in the lower
half of a line in __memcpy().


> 
>> +	 */
>> +	.pushsection .altinstr_replacement, "ax"
>> +	.balign 64
>> +	.popsection
>> +#endif
>> +	ALTERNATIVE __stringify(cmpq $0x20, %rdx; jb .Lhandle_tail), \
>> +		    __stringify(rep movsb; RET), X86_FEATURE_FSRM
>>  
>> -	cmpq $0x20, %rdx
>> -	jb .Lhandle_tail
>> +	/*
>> +	 * Prepared RCX is scratch for checking length >= 2048.
>> +	 * The 10-byte alternative fits a NOP without ERMS.
>> +	 */
>> +	ALTERNATIVE "", __stringify(shrq $11, %rcx; jnz .Lerms), X86_FEATURE_ERMS
> 
> Can you use %rdx here, preserving %rcx for the 'rep movsb'?
> You might need to swap cx<=>dx in the rest of the function.
> (I'm sure we could use #defines to give names to registers....)


Yeah thats a good idea! It does make the code change a lot bigger, but its
easy to review as its just swapping rdx to rcx. If there arent any major objections
to this version, I will send it as an official patch tomorrow with the swap you
suggested plus a bit more cleanup on the commit message and comments.


Thanks for the review!
Usama

> 
> David
> 
>>  	/*
>>  	 * We check whether memory false dependence could occur,
>> @@ -169,5 +172,14 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
>>  
>>  .Lend:
>>  	RET
>> -SYM_FUNC_END(memcpy_orig)
>>  
>> +.Lerms:
>> +	movq %rdx, %rcx
>> +	rep movsb
>> +	RET
>> +SYM_FUNC_END(__memcpy)
>> +EXPORT_SYMBOL(__memcpy)
>> +
>> +SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy)
>> +SYM_PIC_ALIAS(memcpy)
>> +EXPORT_SYMBOL(memcpy)
> 

Re: [PATCH] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[David Laight]

On Tue, 13 Oct 2026 03:04:42 -0700
Usama Arif <usama.arif@linux.dev> wrote:

> Commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
> memory copies") stopped selecting REP MOVSB from ERMS alone because it
> is not suitable for short copies. Consequently, ordinary out-of-line
> __memcpy() calls on CPUs with ERMS but without FSRM use memcpy_orig()
> for every length. Its main loop copies 32 bytes per iteration and can be
> slower than REP MOVSB for large copies, as measured below on Cooper Lake.

But pretty much all the memcpy() in the kernel are small.
You are adding a conditional branch to all to of them.

If you have somewhere doing long copies then the code should probably
be calling a different function.

You also need to test on a variety of cpu, especially including older zen.
IIRC some of those don't behave as the cpu features might indicate.

Intel cpu going back to ivy bridge (and maybe sandy bridge) treat 'rep movsb'
and 'rep movsq' exactly the same, the only alignment that ever matter is
64 byte alignment (possibly less on ivy/sandy bridge) of the destination
buffer - the copies run twice as fast if 64 byte aligned.
After the initial overhead (which my tests didn't measure) the bytes/clock
was moderately constant across all buffer sizes.
Amd cpu are different, even zen-5 has a big performance hit if (IIRC)
the page offset of source in less than 128 bytes below that of the
destination (the byte alignment doesn't matter).

I need to redo those measurement having worked out hot to time the
instruction setup costs.

David

> 
> This matters for btrfs zstd decompression. The btrfs-tree commit
> e27a6e4823f3 ("btrfs: zstd: avoid a copy in zstd_decompress_bio()")
> removes btrfs' copy from a scratch buffer into the page cache by letting
> zstd write there directly. It does not remove the copies inside
> streaming zstd. ZSTD_decompressStream() still uses memcpy() to flush
> decompressed data from its internal output buffer to the supplied page
> (zdss_flush), and to stage compressed input when the next block crosses
> an input folio supplied by btrfs (zdss_load). In Meta's fleet profile
> of x86 kernels, these two btrfs/zstd call sites accounted for about 34%
> of the sampled profile weight attributed to memcpy_orig().
> 
> On CPUs without FSRM, select REP MOVSB for copies of 512 bytes and above
> when ERMS is present. Keep memcpy_orig() below 512 bytes. The later FSRM
> alternative continues to select REP MOVSB for all sizes, including when
> ERMS is not exposed.
> 
> Userspace replicas of the original and patched entry paths were run on
> a turbo-disabled Xeon Platinum 8321HC (Cooper Lake), capped at
> 1.401 GHz. Results average two runs and scale the per-call time to
> 65,536 4 KB calls, or 256 MB in aggregate.
> 
> Here "hot" means cache-resident. The rows model zdss_flush() from
> zstd's L2-resident output buffer, zdss_load() from a cold compressed
> folio into zstd's hot input buffer, and a page copy into a cold
> destination.
> 
>                                     orig  REP MOVSB  reduction
>   zdss_flush: L2 -> hot page     28.0 ms   11.9 ms      57.5%
>   zdss_load: cold -> hot buffer  46.1 ms   28.9 ms      37.3%
>   page copy: hot -> cold page    29.8 ms   22.5 ms      24.5%
> 
> REP MOVSB was faster in every tested scenario at 512 bytes and above.
> Copies below 512 bytes showed no consistent performance change across the
> possible function alignments.
> 
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> ---
> The above benchmarks to model copies in zstd were created using LLM:
> https://gist.github.com/uarif1/8409ccd34284f128dbea5822ec628ef3
> ---
>  arch/x86/lib/memcpy_64.S | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
> index 12a23fa7c44c5..a7b3ce4fae0b4 100644
> --- a/arch/x86/lib/memcpy_64.S
> +++ b/arch/x86/lib/memcpy_64.S
> @@ -31,7 +31,9 @@
>   * which the compiler could/should do much better anyway.
>   */
>  SYM_TYPED_FUNC_START(__memcpy)
> -	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
> +	ALTERNATIVE_2 "jmp memcpy_orig", \
> +		      "cmpq $512, %rdx; jb memcpy_orig", X86_FEATURE_ERMS, \
> +		      "", X86_FEATURE_FSRM
>  
>  	movq %rdi, %rax
>  	movq %rdx, %rcx