From: "Li Zhe" <lizhe.67@bytedance.com>
To: "Muchun Song" <muchun.song@linux.dev>
Cc: <linux-arch@vger.kernel.org>, <linux-hardening@vger.kernel.org>,
<linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
<x86@kernel.org>, <akpm@linux-foundation.org>,
<apopple@nvidia.com>, <arnd@arndb.de>, <balbirs@nvidia.com>,
<bp@alien8.de>, <dave.hansen@linux.intel.com>,
<david@kernel.org>, <kees@kernel.org>, <mingo@redhat.com>,
<rppt@kernel.org>, <tglx@kernel.org>
Subject: Re: [PATCH v6 6/8] string: introduce memcpy_nt() helpers
Date: Wed, 15 Jul 2026 14:18:50 +0800 [thread overview]
Message-ID: <abd1f6da-a6f4-40ed-923f-94201bba4690@bytedance.com> (raw)
In-Reply-To: <7bfb13a0-8f4a-4f4b-9df4-da842169f8f1@linux.dev>
On 7/14/26 5:24 PM, Muchun Song wrote:
>
>
> On 2026/7/9 19:25, Li Zhe wrote:
>> Introduce memcpy_nt() and memcpy_nt_drain() for write-once copy sites
>> that want a named non-temporal copy primitive plus an explicit drain
>> step.
>>
>> On x86_64, override both helpers in arch/x86/include/asm/string_64.h
>> using the usual self-macro pattern, next to the existing
>> memcpy_flushcache() backend that memcpy_nt() wraps. The x86_64
>> implementation maps memcpy_nt() to memcpy_flushcache() and uses wmb()
>> for memcpy_nt_drain(), because that backend issues MOVNTI stores and
>> callers need an ordering point before later normal stores that depend
>> on those writes becoming visible.
>>
>> include/linux/string.h provides the generic fallback under
>> memcpy_nt() as plain memcpy() and leaves memcpy_nt_drain() empty, so
>> architectures that do not override memcpy_nt() do not pay an
>> unconditional barrier. Architectures that later grow a specialized
>> memcpy_nt() backend can override memcpy_nt_drain() with whatever
>> drain primitive their memory-ordering rules require.
>>
>> The immediate user is the ZONE_DEVICE template-copy path. It populates
>> struct page descriptors in a write-once pattern, so a regular cached
>> memcpy() can incur avoidable write-allocate traffic and cache
>> pollution for data with little near-term reuse.
>>
>> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
>> ---
>> arch/x86/include/asm/string_64.h | 22 ++++++++++++++++++++++
>> include/linux/string.h | 23 +++++++++++++++++++++++
>> 2 files changed, 45 insertions(+)
>>
>> diff --git a/arch/x86/include/asm/string_64.h
>> b/arch/x86/include/asm/string_64.h
>> index 4635616863f5..6cb9e0ac7fa0 100644
>> --- a/arch/x86/include/asm/string_64.h
>> +++ b/arch/x86/include/asm/string_64.h
>> @@ -4,6 +4,7 @@
>>
>> #ifdef __KERNEL__
>> #include <linux/jump_label.h>
>> +#include <asm/barrier.h>
>>
>> /* Written 2002 by Andi Kleen */
>>
>> @@ -100,6 +101,27 @@ static __always_inline void
>> memcpy_flushcache(void *dst, const void *src, size_t
>> }
>> __memcpy_flushcache(dst, src, cnt);
>> }
>> +
>> +#define memcpy_nt memcpy_nt
>> +/*
>> + * Reuse the existing x86 flushcache backend as the nt copy primitive.
>> + * Callers pair it with memcpy_nt_drain() when later stores must be
>> + * ordered after the copy.
>> + */
>> +static __always_inline void memcpy_nt(void *dst, const void *src,
>> size_t cnt)
>> +{
>> + memcpy_flushcache(dst, src, cnt);
>
> Why not use memcpy_flushcache() directly in device dax path? I don't
> understand the necessity of introducing memcpy_nt here.
>
The reason for introducing memcpy_nt() is to give generic MM code a
named non-temporal copy primitive, instead of hardwiring the x86
memcpy_flushcache() backend into a generic caller.
On x86, memcpy_nt() maps to memcpy_flushcache() today. On other
architectures, memcpy_flushcache() may have different semantics, and we
also do not know whether its implementation would provide the same
optimization opportunity as on x86. Using memcpy_nt() lets the generic
caller express the intent while leaving the backend choice to each
architecture.
>> +}
>> +
>> +#define memcpy_nt_drain memcpy_nt_drain
>> +static __always_inline void memcpy_nt_drain(void)
>> +{
>> + /*
>> + * Order the prior MOVNTI stores issued by memcpy_flushcache()
>> + * before later normal stores.
>> + */
>
> I also have a question here: why are we using wmb to guarantee visibility
> at this stage?
>
> Since we are still in the very early phases of memory initialization
> (specifically,
> struct page initialization), since we are still in an intermediate
> initialization
> state, this shouldn't be visible to other CPUs anyway.
>
> Thanks.
The drain is not about exposing the intermediate initialization state to
other CPUs.
It is there to order the earlier non-temporal stores before the later
normal stores on the same control path, for example before
memmap_init_compound() / prep_compound_head() update overlapping
compound metadata.
On x86, memcpy_nt() maps to MOVNTI-based memcpy_flushcache(), so
memcpy_nt_drain() uses wmb(), which maps to the required sfence there.
Thanks,
Zhe
>
>> + wmb();
>> +}
>> #endif
>>
>> #endif /* __KERNEL__ */
>> diff --git a/include/linux/string.h b/include/linux/string.h
>> index 5702daca4326..a109b2f86ca6 100644
>> --- a/include/linux/string.h
>> +++ b/include/linux/string.h
>> @@ -278,6 +278,29 @@ static inline void memcpy_flushcache(void *dst,
>> const void *src, size_t cnt)
>> }
>> #endif
>>
>> +#ifndef memcpy_nt
>> +/*
>> + * memcpy_nt() requests a non-temporal copy when the architecture has a
>> + * suitable backend. Architectures that do not override it fall back to
>> + * memcpy().
>> + */
>> +static inline void memcpy_nt(void *dst, const void *src, size_t cnt)
>> +{
>> + memcpy(dst, src, cnt);
>> +}
>> +#endif
>> +
>> +#ifndef memcpy_nt_drain
>> +/*
>> + * Callers use memcpy_nt_drain() before later normal stores that
>> need to
>> + * be ordered after memcpy_nt(). Architectures without a specialized
>> + * backend can leave it empty.
>> + */
>> +static inline void memcpy_nt_drain(void)
>> +{
>> +}
>> +#endif
>> +
>> void *memchr_inv(const void *s, int c, size_t n);
>> char *strreplace(char *str, char old, char new);
>>
>> --
>> 2.20.1
>>
>
next prev parent reply other threads:[~2026-07-15 6:19 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-09 11:25 [PATCH v6 0/8] mm: optimize zone-device memmap initialization Li Zhe
2026-07-09 11:25 ` [PATCH v6 1/8] mm: fix stale ZONE_DEVICE refcount comment Li Zhe
2026-07-14 2:25 ` Muchun Song
2026-07-09 11:25 ` [PATCH v6 2/8] mm: factor zone-device page init helpers out of __init_zone_device_page Li Zhe
2026-07-14 2:44 ` Muchun Song
2026-07-15 4:12 ` Li Zhe
2026-07-14 11:35 ` Balbir Singh
2026-07-15 4:17 ` Li Zhe
2026-07-09 11:25 ` [PATCH v6 3/8] mm: add a set_page_section_from_pfn() helper Li Zhe
2026-07-14 2:45 ` Muchun Song
2026-07-14 12:09 ` Balbir Singh
2026-07-09 11:25 ` [PATCH v6 4/8] mm: add a template-based fast path for zone-device page init Li Zhe
2026-07-13 13:28 ` Muchun Song
2026-07-15 4:21 ` Li Zhe
2026-07-14 8:38 ` Muchun Song
2026-07-15 6:11 ` Li Zhe
2026-07-15 9:19 ` Muchun Song
2026-07-15 5:21 ` Balbir Singh
2026-07-15 6:49 ` Li Zhe
2026-07-09 11:25 ` [PATCH v6 5/8] mm: extend the template fast path to zone-device compound tails Li Zhe
2026-07-09 11:25 ` [PATCH v6 6/8] string: introduce memcpy_nt() helpers Li Zhe
2026-07-14 9:24 ` Muchun Song
2026-07-15 6:18 ` Li Zhe [this message]
2026-07-15 9:33 ` Muchun Song
2026-07-16 11:54 ` Li Zhe
2026-07-09 11:25 ` [PATCH v6 7/8] x86/string: extend memcpy_flushcache() fixed-size fastpaths Li Zhe
2026-07-09 11:25 ` [PATCH v6 8/8] mm: use memcpy_nt() in zone-device template copies Li Zhe
2026-07-14 9:45 ` Muchun Song
2026-07-14 11:22 ` Muchun Song
2026-07-15 6:30 ` Li Zhe
2026-07-15 9:45 ` Muchun Song
2026-07-16 11:56 ` Li Zhe
2026-07-13 1:44 ` [PATCH v6 0/8] mm: optimize zone-device memmap initialization Balbir Singh
2026-07-13 3:18 ` ByteDance
2026-07-13 13:15 ` Muchun Song
2026-07-15 6:37 ` Li Zhe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=abd1f6da-a6f4-40ed-923f-94201bba4690@bytedance.com \
--to=lizhe.67@bytedance.com \
--cc=akpm@linux-foundation.org \
--cc=apopple@nvidia.com \
--cc=arnd@arndb.de \
--cc=balbirs@nvidia.com \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=david@kernel.org \
--cc=kees@kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mingo@redhat.com \
--cc=muchun.song@linux.dev \
--cc=rppt@kernel.org \
--cc=tglx@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®