Re: [PATCH RFC v2] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

[Pedro Falcato <pfalcato@suse.de>]

On Mon, Oct 19, 2026 at 10:06:49AM +0000, Shivank Garg wrote:
> folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
> so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
> pays the startup cost of the string instruction ("REP; MOVSB" on x86
> CPUs with ERMS) plus a cond_resched().
> 
> Introduce copy_mc_highpages(), the page range variant of
> copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
> are in the linear map, copy the whole range with a single
> copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
> Otherwise, fall back to copy_mc_highpage() one page at a time.
> 
> On a machine check, queue the poisoned source page for memory_failure()
> and return -EHWPOISON, as copy_mc_highpage() does.
> 
> copy_mc_to_kernel() cannot reschedule, so copying a large folio with one
> call can run for a long time under cooperative preemption models. Add

Do we care, though? I think most architectures are full preempt/preempt lazy
at this point? Is PREEMPT_DYNAMIC the only showstopper?

> COPY_MC_PAGES_NON_PREEMPT_BATCH to bound how much is copied in one go:
> 32MB, the same bound clear_pages() gets from
> PROCESS_PAGES_NON_PREEMPT_BATCH, or one page on the fallback path.
> 
> Switch folio_mc_copy() to copy_mc_highpages(): the whole folio at once
> under preemptible models, COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time
> otherwise.
> 
> Assisted-by: LLM
> Signed-off-by: Shivank Garg <shivankg@amd.com>
> ---
> On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
> 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
> When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
> 
> Changes in v2:
> - Drop the folio_copy() changes as there is little to gain here.
>   On architectures without copy_mc_to_kernel(), copy_mc_highpages()
>   falls back to the per-page copy_mc_highpage(), which ends up in
>   copy_highpage(). A batched copy_highpages() could help there too,
>   but I haven't tested it on those architectures, so that is left for
>   a follow-up.
> - Add COPY_MC_PAGES_NON_PREEMPT_BATCH to limit how much is copied
>   between cond_resched() calls on non-preemptible kernels.
> - Rebase onto mm-new (2887e63d2e8e).
> - v1: https://lore.kernel.org/all/20260427142036.111940-2-shivankg@amd.com
> 
> Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are
> DRAM, node 2 is CXL), with and without this patch.
> 
> move_pages() throughput
> -----------------------
> A single move_pages() call migrates 1GB of anonymous memory, as 512
> 2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each),
> in GiB/s:
> 
> 2MB THP:
>   direction         base            patched         speedup
>   DRAM0 -> DRAM1    19.57 ± 0.17    27.51 ± 0.78    1.41x
>   DRAM1 -> DRAM0    19.44 ± 0.12    27.33 ± 0.27    1.41x
>   DRAM0 -> CXL      18.66 ± 0.18    26.90 ± 0.51    1.44x
>   CXL   -> DRAM0    15.38 ± 0.59    16.60 ± 0.24    1.08x
> 
> 1GB hugetlb:
>   direction         base            patched         speedup
>   DRAM0 -> DRAM1    20.34 ± 0.13    53.88 ± 1.48    2.65x
>   DRAM1 -> DRAM0    19.10 ± 2.65    53.78 ± 0.55    2.82x
>   DRAM0 -> CXL      19.53 ± 0.08    44.10 ± 0.19    2.26x
>   CXL   -> DRAM0    16.41 ± 0.07    20.89 ± 0.06    1.27x
> 
> kswapd demotion
> ---------------
> Node 0 is limited to 16GB of free memory and demotion is enabled. A
> single thread bound to node 0 writes 96GB of anonymous THP memory, so
> ~80GB has to be demoted to CXL. Median of 10 runs:
> 
>                                base        patched
>   kswapd CPU time              4.25 s      3.44 s     (-19%)
>   kswapd CPU per GB demoted    63.6 ms     44.1 ms    (-31%)
>   demoted by direct reclaim    14.1 GB     0.46 GB
>   allocstall                   7427        370
> 
> With the patch, kswapd keeps up with the writer, so the writer rarely
> falls back to direct reclaim.
> 
> Other workloads
> ---------------
> - Memory tiering (numa_balancing=2) with Redis, GAPBS PR/BC, a B-tree
>   benchmark and Graph500: copying 2MB folios during demotion is
>   1.18-1.49x faster.
> - A userspace daemon moving 1GB hugetlb pages between DRAM and CXL
>   with move_pages() while GUPS runs on them: demotion is 1.43x faster
>   and promotion 1.28x faster.
> ---
>  include/linux/highmem.h | 59 +++++++++++++++++++++++++++++++++++++++++++++++++
>  mm/util.c               | 18 +++++++++++----
>  2 files changed, 73 insertions(+), 4 deletions(-)
> 
> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> index d7aac9de1c8a..e43459dc58b0 100644
> --- a/include/linux/highmem.h
> +++ b/include/linux/highmem.h
> @@ -484,6 +484,65 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
>  }
>  #endif
>  
> +#if defined(copy_mc_to_kernel) && !defined(CONFIG_HIGHMEM) && \
> +	!defined(__HAVE_ARCH_COPY_HIGHPAGE)
> +/*
> + * copy_mc_highpages() copies the whole range with a single
> + * copy_mc_to_kernel() call, which cannot reschedule. That is fine under
> + * preemptible models, but under cooperatively scheduled models callers
> + * should copy at most COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time.
> + *
> + * With a value of 32MB and assuming a memory bandwidth of ~10GBps, this
> + * should result in a worst case preemption latency of around 3ms when
> + * copying pages.
> + */

This worst-case is absolutely terrible :)

> +#define COPY_MC_PAGES_NON_PREEMPT_BATCH		(SZ_32M >> PAGE_SHIFT)
> +
> +/**
> + * copy_mc_highpages() - copy a page range with machine check handling
> + * @to: first destination page
> + * @from: first source page
> + * @nr_pages: number of pages
> + *
> + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
> + * source page is queued for memory_failure().
> + *
> + * Does not call cond_resched(). Under cooperatively scheduled models the
> + * caller is expected to limit @nr_pages to COPY_MC_PAGES_NON_PREEMPT_BATCH.

This is very backwards, callers shouldn't need to check this.

> + *
> + * Return: 0 on success, -EHWPOISON if a machine check was handled.
> + */
> +static inline int copy_mc_highpages(struct page *to, struct page *from,
> +		unsigned long nr_pages)
> +{
> +	unsigned long len = nr_pages << PAGE_SHIFT;
> +	unsigned long ret, i;
> +
> +	ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
> +	if (!ret) {
> +		for (i = 0; i < nr_pages; i++)
> +			kmsan_copy_page_meta(to + i, from + i);
> +		return 0;
> +	}
> +	memory_failure_queue(page_to_pfn(from) + ((len - ret) >> PAGE_SHIFT), 0);
> +	return -EHWPOISON;

These codepaths look strangely inverted, it would be more natural if you did:

	if (ret) {
		memory_failure_queue(page_to_pfn(from) + ((len - ret) >> PAGE_SHIFT), 0);
		return -EWPOISON;
	}
	for (i = 0; i < nr_pages; i++)
 		kmsan_copy_page_meta(to + i, from + i);
	/* Or, ideally, kmsan_copy_pages_meta(to, from, nr_pages); */
	return 0;
}

in my opinion.

> +}
> +#else
> +
> +#define COPY_MC_PAGES_NON_PREEMPT_BATCH		1

I would suggest that perhaps this should be the fallback for non-preemptible
models. But it gets a little weird with PREEMPT_DYNAMIC I think?

> +
> +static inline int copy_mc_highpages(struct page *to, struct page *from,
> +		unsigned long nr_pages)
> +{
> +	unsigned long i;
> +
> +	for (i = 0; i < nr_pages; i++)
> +		if (copy_mc_highpage(to + i, from + i))
> +			return -EHWPOISON;
> +	return 0;
> +}
> +#endif
> +
>  static inline void memcpy_page(struct page *dst_page, size_t dst_off,
>  			       struct page *src_page, size_t src_off,
>  			       size_t len)
> diff --git a/mm/util.c b/mm/util.c
> index 7f374bb8bf36..5015cfdfeaa5 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);
>  
>  int folio_mc_copy(struct folio *dst, struct folio *src)
>  {
> -	long nr = folio_nr_pages(src);
> -	long i = 0;
> +	const unsigned long nr_pages = folio_nr_pages(src);
> +	/*
> +	 * Copy the largest extent possible to allow for architecture specific
> +	 * extent based optimizations. copy_mc_highpages() does not call
> +	 * cond_resched(), so limit the unit size when running under
> +	 * non-preemptible scheduling models.
> +	 */
> +	const unsigned long unit = preempt_model_preemptible() ?
> +				   nr_pages : COPY_MC_PAGES_NON_PREEMPT_BATCH;

See? now mm is aware of preemption models! eww!

-- 
Pedro