[PATCH v3 2/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif <usama.arif@linux.dev>]

Since commit 68674f94ffc9 ("x86: don't use REP_GOOD or ERMS for small
memory copies"), memcpy() uses REP MOVSB only on FSRM CPUs, so ERMS CPUs
without FSRM, such as Skylake and Cooper Lake, copy every size with
the scalar loop. Large copies matter there: in Meta's x86 fleet, about a
third of the time in memcpy_orig() is btrfs zstd decompression copying
pages into and out of zstd's buffers.

Use REP MOVSB from 2048 bytes on ERMS CPUs. On a 1.4 GHz Cooper Lake, an
aligned, cache-hot 4 KiB memcpy() then takes 66.8% less time (123.3 ns
instead of 371.6 ns). In a single-vCPU guest, reading a synthetic
512 MiB btrfs zstd file takes 14.8% less time (median of five kernel
pairs). Smaller copies keep the scalar loop: using REP MOVSB from
32 bytes made 32/64/128/256-byte copies
111.8%/68.0%/63.2%/15.6% slower.

Prepare the return value and RCX count before the FSRM alternative. On
FSRM CPUs, REP MOVSB replaces the jump to memcpy_orig(), followed by a
RET in ordinary text. ITS then selects return handling using the RET's
execution address.

For ERMS CPUs, shift the scratch copy of the length in RDX after the
32-byte check. RCX already holds the count, so large copies need no
reload. Copies below 32 bytes skip the check, which is a single NOP
without ERMS.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 arch/x86/lib/memcpy_64.S | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/arch/x86/lib/memcpy_64.S b/arch/x86/lib/memcpy_64.S
index 7bb146c811029..5320862e0274e 100644
--- a/arch/x86/lib/memcpy_64.S
+++ b/arch/x86/lib/memcpy_64.S
@@ -27,15 +27,13 @@
  *
  * The 'rep movsb' itself is small enough to replace the call, but the
  * two register moves blow up the code. And one of them is "needed"
- * only for the return value that is the same as the source input,
+ * only for the return value that is the same as the destination input,
  * which the compiler could/should do much better anyway.
  */
 SYM_TYPED_FUNC_START(__memcpy)
-	ALTERNATIVE "jmp memcpy_orig", "", X86_FEATURE_FSRM
-
 	movq %rdi, %rax
 	movq %rdx, %rcx
-	rep movsb
+	ALTERNATIVE "jmp memcpy_orig", "rep movsb", X86_FEATURE_FSRM
 	RET
 SYM_FUNC_END(__memcpy)
 EXPORT_SYMBOL(__memcpy)
@@ -45,12 +43,16 @@ SYM_PIC_ALIAS(memcpy)
 EXPORT_SYMBOL(memcpy)
 
 SYM_FUNC_START_LOCAL(memcpy_orig)
-	movq %rdi, %rax
-	movq %rdx, %rcx
-
 	cmpq $0x20, %rcx
 	jb .Lhandle_tail
 
+	/*
+	 * RCX holds the count for REP MOVSB and scalar copying; RDX is scratch.
+	 * Shift RDX to check length >= 2048 without destroying the REP count.
+	 * The 10-byte alternative fits a NOP without ERMS.
+	 */
+	ALTERNATIVE "", __stringify(shrq $11, %rdx; jnz .Lerms), X86_FEATURE_ERMS
+
 	/*
 	 * We check whether memory false dependence could occur,
 	 * then jump to corresponding copy mode.
@@ -170,5 +172,8 @@ SYM_FUNC_START_LOCAL(memcpy_orig)
 
 .Lend:
 	RET
-SYM_FUNC_END(memcpy_orig)
 
+.Lerms:
+	rep movsb
+	RET
+SYM_FUNC_END(memcpy_orig)
-- 
2.53.0-Meta