[PATCH v3 0/2] x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

[Usama Arif <usama.arif@linux.dev>]

Enable REP MOVSB for copies of 2048 bytes and larger on ERMS CPUs without
FSRM, keeping scalar copies below that threshold. FSRM CPUs continue to use
REP MOVSB for every size.

This series splits v2 into 2 patches.
The first changes only the scalar register allocation: RCX holds the count
and tail indexes, and RDX is the small-copy data temporary. The second
adds the ERMS size check and adopts Linus's FSRM entry layout. REP MOVSB
replaces the existing jump to memcpy_orig(), with RET remaining in
ordinary text. ITS therefore selects return handling using the return's
actual execution address, without aligning replacement storage. Keeping
the count in RCX also addresses David's request to avoid reloading it on
the ERMS path.

v2 -> v3: https://lore.kernel.org/r/20261104222806.1826508-1-usama.arif@linux.dev
- Split the scalar RCX/RDX register swap into a preparatory patch with no
  functional change (Kiryl Shutsemau).
- Use REP MOVSB to replace the jump to memcpy_orig() on FSRM CPUs, keeping
  RET in ordinary text (Kiryl Shutsemau, Linus Torvalds). Remove the ITS
  replacement alignment, since no return is copied by alternatives now.

v1 -> v2: https://lore.kernel.org/r/20261013100442.2658311-1-usama.arif@linux.dev
- Raise the threshold from 512 to 2048 bytes. REP MOVSB was slower than
  the scalar loop for some misaligned copies of 512 to 640 bytes.
- Fold memcpy_orig() into __memcpy(). With FSRM, "rep movsb; RET" now
  replaces the 32-byte length check (David Laight, Linus Torvalds). In
  v1, the 13-byte empty FSRM replacement became a jump that FSRM CPUs
  took on every call (Kiryl Shutsemau).
- Do the ERMS check after the 32-byte check, so copies below 32 bytes
  skip it (Kiryl Shutsemau), and keep the count in RCX so REP MOVSB
  needs no reload (David Laight).
- Align the FSRM replacement so that its RET keeps the ITS return thunk.
- Fix the comment that called the return value the source input.
 
Usama Arif (2):
  x86/lib: Keep the scalar memcpy() count in RCX
  x86/lib: Use REP MOVSB for large memcpy() on ERMS CPUs without FSRM

 arch/x86/lib/memcpy_64.S | 78 +++++++++++++++++++++-------------------
 1 file changed, 42 insertions(+), 36 deletions(-)

-- 
2.53.0-Meta