[PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Martin Leitner-Ankerl]

x86-64 does not define ARCH_HAS_PREFETCH, so prefetch() from
<linux/prefetch.h> is just __builtin_prefetch(). The kernel builds with
-mno-sse, and clang then drops __builtin_prefetch() without a warning,
while gcc emits prefetcht0. An x86-64 defconfig vmlinux has 8
prefetcht0 with clang 22.1.8, all from inline assembly, and 62 with gcc
16.2.1. Also prefetch_range() is empty on x86-64 with any compiler, it
needs ARCH_HAS_PREFETCH.

Give x86-64 its own prefetch(), a prefetcht0 in inline assembly, which
every x86-64 CPU has, and define ARCH_HAS_PREFETCH for 64-bit too. Not
the 32-bit alternative: it becomes prefetchnta on CPUs with SSE, a
non-temporal prefetch, which would change what gcc builds emit today.

Afterwards clang's vmlinux has 57 prefetcht0, the new ones in 34
functions, e.g. __blk_mq_alloc_requests(), prepend_path(),
__alloc_skb(), tcp_ack() and the receive paths of e1000, tg3, sky2,
r8169 and forcedeth. gcc's stay at 62. With gcc 16.2.1, prefetch_range()
adds 16 to hfi1's receive functions and 8 to scsi_debug, which a
defconfig does not build. Youling Tang proposed to define
ARCH_HAS_PREFETCH in <linux/prefetch.h> instead [1].

KVM guest with one CPU on a Ryzen 9 7950X at a fixed 4.5 GHz, clang
22.1.8, 6 boots per kernel in turns, ns per call, median (range):

                    before                 after
  io_uring IOPOLL   680.1 (679.6..684.6)   664.9 (663.1..668.4)   -2.3%
  getcwd()          912.9 (912.0..1019.4)  901.4 (900.1..907.3)   -1.3%

IOPOLL reads 4 KiB from null_blk, 32 per submit; getcwd() is 40
directories deep. Built with gcc, where both kernels have the same
prefetches, the two differ by +0.4% and 0.0%. pread() with O_DIRECT, UDP
and TCP over loopback, times() and clock_gettime() change no more than
that. Unfortunately the NIC receive paths need real hardware, so they
are not measured.

Fixes: ab483570a13b ("x86 & generic: change to __builtin_prefetch()")
Link: https://lore.kernel.org/20240529032059.899347-2-youling.tang@linux.dev/ # [1]
Cc: Youling Tang <tangyouling@kylinos.cn>
Cc: Nikolay Borisov <nik.borisov@suse.com>
Signed-off-by: Martin Leitner-Ankerl <martin.ankerl@gmail.com>
---

Notes:
    Code generation, testing, compiling, analysis, measurements, etc. were
    all driven by an AI agent (Claude Code). I have reviewed the results and
    supervised the process.

    A few minutes ago this went to the x86 maintainers only, without the
    lists, by mistake. Sorry for the duplicate.

 arch/x86/include/asm/processor.h | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 77488ff5eb8c..64748b8705e3 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -624,15 +624,25 @@ extern int			bootloader_version;
 extern char			ignore_fpu_irq;
 
 #define HAVE_ARCH_PICK_MMAP_LAYOUT 1
+#define ARCH_HAS_PREFETCH
 #define ARCH_HAS_PREFETCHW
 
 #ifdef CONFIG_X86_32
 # define BASE_PREFETCH		""
-# define ARCH_HAS_PREFETCH
 #else
 # define BASE_PREFETCH		"prefetcht0 %1"
 #endif
 
+#ifdef CONFIG_X86_64
+/*
+ * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
+ * builds with -mno-sse, and clang then drops the builtin without a warning.
+ */
+static inline void prefetch(const void *x)
+{
+	asm("prefetcht0 %0" : : "m" (*(const char *)x));
+}
+#else
 /*
  * Prefetch instructions for Pentium III (+) and AMD Athlon (+)
  *
@@ -645,6 +655,7 @@ static inline void prefetch(const void *x)
 			  X86_FEATURE_XMM,
 			  "m" (*(const char *)x));
 }
+#endif /* CONFIG_X86_64 */
 
 /*
  * 3dnow prefetch to get an exclusive cache line.

base-commit: 2305edc99ef7bbc11a8dcebf78853bc1b6a761bc
-- 
2.56.0

Re: [PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Nick Desaulniers]

On Mon, Nov 2, 2026 at 10:08 PM Martin Leitner-Ankerl
<martin.ankerl@gmail.com> wrote:
>
> x86-64 does not define ARCH_HAS_PREFETCH, so prefetch() from
> <linux/prefetch.h> is just __builtin_prefetch(). The kernel builds with
> -mno-sse, and clang then drops __builtin_prefetch() without a warning,
> while gcc emits prefetcht0. An x86-64 defconfig vmlinux has 8
> prefetcht0 with clang 22.1.8, all from inline assembly, and 62 with gcc
> 16.2.1. Also prefetch_range() is empty on x86-64 with any compiler, it
> needs ARCH_HAS_PREFETCH.
>
> Give x86-64 its own prefetch(), a prefetcht0 in inline assembly, which
> every x86-64 CPU has, and define ARCH_HAS_PREFETCH for 64-bit too. Not
> the 32-bit alternative: it becomes prefetchnta on CPUs with SSE, a
> non-temporal prefetch, which would change what gcc builds emit today.
>
> Afterwards clang's vmlinux has 57 prefetcht0, the new ones in 34
> functions, e.g. __blk_mq_alloc_requests(), prepend_path(),
> __alloc_skb(), tcp_ack() and the receive paths of e1000, tg3, sky2,
> r8169 and forcedeth. gcc's stay at 62. With gcc 16.2.1, prefetch_range()
> adds 16 to hfi1's receive functions and 8 to scsi_debug, which a
> defconfig does not build. Youling Tang proposed to define
> ARCH_HAS_PREFETCH in <linux/prefetch.h> instead [1].
>
> KVM guest with one CPU on a Ryzen 9 7950X at a fixed 4.5 GHz, clang
> 22.1.8, 6 boots per kernel in turns, ns per call, median (range):
>
>                     before                 after
>   io_uring IOPOLL   680.1 (679.6..684.6)   664.9 (663.1..668.4)   -2.3%
>   getcwd()          912.9 (912.0..1019.4)  901.4 (900.1..907.3)   -1.3%
>
> IOPOLL reads 4 KiB from null_blk, 32 per submit; getcwd() is 40
> directories deep. Built with gcc, where both kernels have the same
> prefetches, the two differ by +0.4% and 0.0%. pread() with O_DIRECT, UDP
> and TCP over loopback, times() and clock_gettime() change no more than
> that. Unfortunately the NIC receive paths need real hardware, so they
> are not measured.
>
> Fixes: ab483570a13b ("x86 & generic: change to __builtin_prefetch()")
> Link: https://lore.kernel.org/20240529032059.899347-2-youling.tang@linux.dev/ # [1]
> Cc: Youling Tang <tangyouling@kylinos.cn>
> Cc: Nikolay Borisov <nik.borisov@suse.com>
> Signed-off-by: Martin Leitner-Ankerl <martin.ankerl@gmail.com>
> ---
>
> Notes:
>     Code generation, testing, compiling, analysis, measurements, etc. were
>     all driven by an AI agent (Claude Code). I have reviewed the results and
>     supervised the process.
>
>     A few minutes ago this went to the x86 maintainers only, without the
>     lists, by mistake. Sorry for the duplicate.
>
>  arch/x86/include/asm/processor.h | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index 77488ff5eb8c..64748b8705e3 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -624,15 +624,25 @@ extern int                        bootloader_version;
>  extern char                    ignore_fpu_irq;
>
>  #define HAVE_ARCH_PICK_MMAP_LAYOUT 1
> +#define ARCH_HAS_PREFETCH
>  #define ARCH_HAS_PREFETCHW
>
>  #ifdef CONFIG_X86_32
>  # define BASE_PREFETCH         ""
> -# define ARCH_HAS_PREFETCH
>  #else
>  # define BASE_PREFETCH         "prefetcht0 %1"
>  #endif
>
> +#ifdef CONFIG_X86_64
> +/*
> + * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
> + * builds with -mno-sse, and clang then drops the builtin without a warning.
> + */
> +static inline void prefetch(const void *x)
> +{
> +       asm("prefetcht0 %0" : : "m" (*(const char *)x));

Thanks for the patch!
Is there a reason to have the cast vs simply use "r" constraint and
deref in the inline asm?
asm("prefetcht0 (%0)" : : "r" (x));
https://godbolt.org/z/xezh15oMe

Either way:
Acked-by: Nick Desaulniers <ndesaulniers@google.com>

> +}
> +#else
>  /*
>   * Prefetch instructions for Pentium III (+) and AMD Athlon (+)
>   *
> @@ -645,6 +655,7 @@ static inline void prefetch(const void *x)
>                           X86_FEATURE_XMM,
>                           "m" (*(const char *)x));
>  }
> +#endif /* CONFIG_X86_64 */
>
>  /*
>   * 3dnow prefetch to get an exclusive cache line.
>
> base-commit: 2305edc99ef7bbc11a8dcebf78853bc1b6a761bc
> --
> 2.56.0
>


-- 
Thanks,
~Nick Desaulniers

Re: [PATCH] x86/cpu: Use prefetcht0 for prefetch() on 64-bit

[Andi Kleen]

> +#ifdef CONFIG_X86_64
> +/*
> + * Every x86-64 CPU has prefetcht0. Not __builtin_prefetch(): the kernel
> + * builds with -mno-sse, and clang then drops the builtin without a warning.
> + */
> +static inline void prefetch(const void *x)
> +{
> +	asm("prefetcht0 %0" : : "m" (*(const char *)x));
> +}
> +#else

One thing you should check if the inline assembly pessimizes the code on
gcc. Normally builtins can be better optimized. Or keep using the builtin
on gcc to be safe.

-Andi