[PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Martin Leitner-Ankerl]

x86-64 does not define ARCH_HAS_PREFETCH, so prefetch() from
<linux/prefetch.h> is just __builtin_prefetch(). The kernel builds with
-mno-sse, and clang then drops __builtin_prefetch() without a warning,
while gcc emits prefetcht0. An x86-64 defconfig vmlinux has 8
prefetcht0 with clang 22.1.8, all from inline assembly, and 62 with gcc
16.2.1. Also prefetch_range() is empty on x86-64 with any compiler, it
needs ARCH_HAS_PREFETCH.

Give x86-64 its own prefetch(), a prefetcht0 in inline assembly, which
every x86-64 CPU has, and define ARCH_HAS_PREFETCH for 64-bit too. Not
the 32-bit alternative: it becomes prefetchnta on CPUs with SSE, a
non-temporal prefetch, which would change what gcc builds emit today.

Afterwards clang's vmlinux has 57 prefetcht0, the new ones in 34
functions, e.g. __blk_mq_alloc_requests(), prepend_path(),
__alloc_skb(), tcp_ack() and the receive paths of e1000, tg3, sky2,
r8169 and forcedeth. gcc's stay at 62. With gcc 16.2.1, prefetch_range()
adds 16 to hfi1's receive functions and 8 to scsi_debug, which a
defconfig does not build. Youling Tang proposed to define
ARCH_HAS_PREFETCH in <linux/prefetch.h> instead [1].

KVM guest with one CPU on a Ryzen 9 7950X at a fixed 4.5 GHz, clang
22.1.8, 6 boots per kernel in turns, ns per call, median (range):

                    before                 after
  io_uring IOPOLL   680.1 (679.6..684.6)   664.9 (663.1..668.4)   -2.3%
  getcwd()          912.9 (912.0..1019.4)  901.4 (900.1..907.3)   -1.3%

IOPOLL reads 4 KiB from null_blk, 32 per submit; getcwd() is 40
directories deep. Built with gcc, where both kernels have the same
prefetches, the two differ by +0.4% and 0.0%. pread() with O_DIRECT, UDP
and TCP over loopback, times() and clock_gettime() change no more than
that. Unfortunately the NIC receive paths need real hardware, so they
are not measured.

Fixes: ab483570a13b ("x86 & generic: change to __builtin_prefetch()")
Link: https://lore.kernel.org/20240529032059.899347-2-youling.tang@linux.dev/ # [1]
Cc: Youling Tang <tangyouling@kylinos.cn>
Cc: Nikolay Borisov <nik.borisov@suse.com>
Signed-off-by: Martin Leitner-Ankerl <martin.ankerl@gmail.com>
---

Notes:
    Code generation, testing, compiling, analysis, measurements, etc. were
    all driven by an AI agent (Claude Code). I have reviewed the results and
    supervised the process.

    A few minutes ago this went to the x86 maintainers only, without the
    lists, by mistake. Sorry for the duplicate.

 arch/x86/include/asm/processor.h | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c..64748b8705e3 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -624,15 +624,25 @@ extern int			bootloader_version;
 extern char			ignore_fpu_irq;
 
 #define HAVE_ARCH_PICK_MMAP_LAYOUT 1
+#define ARCH_HAS_PREFETCH
 #define ARCH_HAS_PREFETCHW
 
 #ifdef CONFIG_X86_32
 # define BASE_PREFETCH		""
-# define ARCH_HAS_PREFETCH
 #else
 # define BASE_PREFETCH		"prefetcht0 %1"
 #endif
 
+#ifdef CONFIG_X86_64
+/*
+ * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
+ * builds with -mno-sse, and clang then drops the builtin without a warning.
+ */
+static inline void prefetch(const void *x)
+{
+	asm("prefetcht0 %0" : : "m" (*(const char *)x));
+}
+#else
 /*
  * Prefetch instructions for Pentium III (+) and AMD Athlon (+)
  *
@@ -645,6 +655,7 @@ static inline void prefetch(const void *x)
 			  X86_FEATURE_XMM,
 			  "m" (*(const char *)x));
 }
+#endif /* CONFIG_X86_64 */
 
 /*
  * 3dnow prefetch to get an exclusive cache line.

base-commit: 2305edc99ef7bbc11a8dcebf78853bc1b6a761bc
-- 
2.56.0