From: "Li Zhe" <lizhe.67@bytedance.com>
To: <akpm@linux-foundation.org>, <apopple@nvidia.com>,
<arnd@arndb.de>, <balbirs@nvidia.com>, <bp@alien8.de>,
<dave.hansen@linux.intel.com>, <david@kernel.org>,
<kees@kernel.org>, <mingo@redhat.com>, <muchun.song@linux.dev>,
<rppt@kernel.org>, <tglx@kernel.org>
Cc: <linux-arch@vger.kernel.org>, <linux-hardening@vger.kernel.org>,
<linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
<x86@kernel.org>, <lizhe.67@bytedance.com>
Subject: [PATCH v8 7/9] x86/string: extend memcpy_flushcache() fixed-size fastpaths
Date: Mon, 27 Jul 2026 20:34:27 +0800 [thread overview]
Message-ID: <20260727123429.5673-8-lizhe.67@bytedance.com> (raw)
In-Reply-To: <20260727123429.5673-1-lizhe.67@bytedance.com>
The new x86 memcpy_nontemporal() helper in this series maps to
memcpy_flushcache(), and the ZONE_DEVICE fast path uses that primitive
for constant-sized struct page template copies.
The immediately relevant x86_64 sizeof(struct page) values here are 64,
80, and 96 bytes. memcpy_flushcache() currently inlines only the 4, 8,
and 16-byte cases, so even those constant-sized struct page copies fall
through to __memcpy_flushcache().
Add 32, 48, 64, 80, and 96-byte MOVNTI sequences to the main fixed-size
switch so those constant-sized copies can stay on the inline path.
Factor the larger sequences into movnti_8()/16()/32()/64() helpers so
the switch can reuse them without duplicating the inline assembly.
While the ZONE_DEVICE template-copy path only needs 64/80/96-byte
copies, keep 32-byte and 48-byte copies inline as well rather than
sending those nearby fixed-size cases back to __memcpy_flushcache().
These new fixed-size cases follow the existing memcpy_flushcache()
fixed-size MOVNTI cases. The existing 4/8/16-byte cases already use
MOVNTI without a separate destination alignment check.
The movnti_8()/16()/32()/64() helpers keep the "memory" clobber
intentionally. The destination memory operand describes the MOVNTI
destination, but it does not express the broader compiler-scheduling
constraint that this path wants: keep the fixed-size copy as a compact
sequence of streaming stores and avoid moving unrelated memory accesses
into the middle of that sequence.
A microbenchmark compared two memcpy_flushcache()-style helpers that
shared the same generic body and differed only in whether
32/48/64/80/96-byte copies stayed in the fixed-size switch or were
redirected to __memcpy_flushcache() when the destination was not 8-byte
aligned. The timed interval covered the copy loop only; wmb() was used
only to separate rounds.
In pseudo-code, the two variants were:
variant A:
switch (len) {
case 32:
case 48:
case 64:
case 80:
case 96:
do fixed-size MOVNTI stores;
return;
default:
generic __memcpy_flushcache()-style body;
}
variant B:
switch (len) {
case 32:
case 48:
case 64:
case 80:
case 96:
if (!IS_ALIGNED(dst, 8))
goto generic_path;
do fixed-size MOVNTI stores;
return;
default:
generic_path:
generic __memcpy_flushcache()-style body;
}
For offsets 1..7 averaged, keeping those sizes in the fixed-size
switch took about 0.410/0.411/0.426/0.426/0.428 us per
32/48/64/80/96-byte copy, while redirecting the same cases to
__memcpy_flushcache() took about 0.962/0.956/0.923/0.998/1.001 us.
The correctness rationale for not adding a separate destination
alignment check is that these cases follow the existing 4/8/16-byte
memcpy_flushcache() MOVNTI cases, and I did not find Intel SDM text
requiring an 8-byte aligned destination for MOVNTI.
The microbenchmark is only the performance motivation for keeping
these small constant-size copies in the inline path.
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
arch/x86/include/asm/string_64.h | 56 +++++++++++++++++++++++++++++++-
1 file changed, 55 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
index 21ae515ae35a..bc6a9f34b346 100644
--- a/arch/x86/include/asm/string_64.h
+++ b/arch/x86/include/asm/string_64.h
@@ -82,7 +82,35 @@ int strcmp(const char *cs, const char *ct);
#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE
#define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1
void __memcpy_flushcache(void *dst, const void *src, size_t cnt);
-static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t cnt)
+
+static __always_inline void movnti_8(void *dst, const void *src)
+{
+ asm volatile("movntiq %1, %0"
+ : "=m"(*(u64 *)dst)
+ : "r"(*(const u64 *)src)
+ : "memory");
+}
+
+static __always_inline void movnti_16(void *dst, const void *src)
+{
+ movnti_8(dst, src);
+ movnti_8(dst + 8, src + 8);
+}
+
+static __always_inline void movnti_32(void *dst, const void *src)
+{
+ movnti_16(dst, src);
+ movnti_16(dst + 16, src + 16);
+}
+
+static __always_inline void movnti_64(void *dst, const void *src)
+{
+ movnti_32(dst, src);
+ movnti_32(dst + 32, src + 32);
+}
+
+static __always_inline void memcpy_flushcache(void *dst, const void *src,
+ size_t cnt)
{
if (__builtin_constant_p(cnt)) {
switch (cnt) {
@@ -96,8 +124,34 @@ static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t
asm ("movntiq %1, %0" : "=m"(*(u64 *)dst) : "r"(*(u64 *)src));
asm ("movntiq %1, %0" : "=m"(*(u64 *)(dst + 8)) : "r"(*(u64 *)(src + 8)));
return;
+ /*
+ * The relevant fixed-size copies here are the
+ * x86_64 struct page sizes: 64, 80, and 96 bytes.
+ * Keep 32-byte and 48-byte copies inline as well
+ * instead of sending those nearby fixed-size
+ * cases back to __memcpy_flushcache().
+ */
+ case 32:
+ movnti_32(dst, src);
+ return;
+ case 48:
+ movnti_32(dst, src);
+ movnti_16(dst + 32, src + 32);
+ return;
+ case 64:
+ movnti_64(dst, src);
+ return;
+ case 80:
+ movnti_64(dst, src);
+ movnti_16(dst + 64, src + 64);
+ return;
+ case 96:
+ movnti_64(dst, src);
+ movnti_32(dst + 64, src + 64);
+ return;
}
}
+
__memcpy_flushcache(dst, src, cnt);
}
--
2.20.1
next prev parent reply other threads:[~2026-07-27 12:37 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-27 12:34 [PATCH v8 0/9] mm: optimize zone-device memmap initialization Li Zhe
2026-07-27 12:34 ` [PATCH v8 1/9] mm: fix stale ZONE_DEVICE refcount comment Li Zhe
2026-07-27 12:34 ` [PATCH v8 2/9] mm: factor zone-device page init helpers out of __init_zone_device_page Li Zhe
2026-07-27 12:34 ` [PATCH v8 3/9] mm: add a set_page_section_from_pfn() helper Li Zhe
2026-07-27 12:34 ` [PATCH v8 4/9] mm: add a template-based fast path for zone-device page init Li Zhe
2026-07-27 12:34 ` [PATCH v8 5/9] mm: extend the template fast path to zone-device compound tails Li Zhe
2026-07-27 12:34 ` [PATCH v8 6/9] string: introduce memcpy_nontemporal() Li Zhe
2026-07-27 12:34 ` Li Zhe [this message]
2026-07-29 23:48 ` [PATCH v8 7/9] x86/string: extend memcpy_flushcache() fixed-size fastpaths Borislav Petkov
2026-07-30 8:05 ` Li Zhe
2026-07-30 8:14 ` David Laight
2026-07-27 12:34 ` [PATCH v8 8/9] mm: use memcpy_nontemporal() in zone-device template copies Li Zhe
2026-07-27 12:34 ` [PATCH v8 9/9] mm: always use the zone-device template init path Li Zhe
2026-07-27 20:57 ` [PATCH v8 0/9] mm: optimize zone-device memmap initialization Andrew Morton
2026-07-28 6:18 ` Li Zhe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260727123429.5673-8-lizhe.67@bytedance.com \
--to=lizhe.67@bytedance.com \
--cc=akpm@linux-foundation.org \
--cc=apopple@nvidia.com \
--cc=arnd@arndb.de \
--cc=balbirs@nvidia.com \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=david@kernel.org \
--cc=kees@kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mingo@redhat.com \
--cc=muchun.song@linux.dev \
--cc=rppt@kernel.org \
--cc=tglx@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®