Re: [PATCH RFC v2] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

["Garg, Shivank" <shivankg@amd.com>]

On Mon, 2026-10-19 at 12:09 +0100, Pedro Falcato wrote:
> On Mon, Oct 19, 2026 at 10:06:49AM +0000, Shivank Garg wrote:
> > folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
> > so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
> > pays the startup cost of the string instruction ("REP; MOVSB" on x86
> > CPUs with ERMS) plus a cond_resched().
> > 
> > Introduce copy_mc_highpages(), the page range variant of
> > copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
> > are in the linear map, copy the whole range with a single
> > copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
> > Otherwise, fall back to copy_mc_highpage() one page at a time.
> > 
> > On a machine check, queue the poisoned source page for memory_failure()
> > and return -EHWPOISON, as copy_mc_highpage() does.
> > 
> > copy_mc_to_kernel() cannot reschedule, so copying a large folio with one
> > call can run for a long time under cooperative preemption models. Add
> 
> Do we care, though? I think most architectures are full preempt/preempt lazy
> at this point? Is PREEMPT_DYNAMIC the only showstopper?

yes, and Since 7dadeaa6e851 ("sched: Further restrict the preemption
modes"),
PREEMPT_DYNAMIC is not problem either.

> 
> > COPY_MC_PAGES_NON_PREEMPT_BATCH to bound how much is copied in one go:
> > 32MB, the same bound clear_pages() gets from
> > PROCESS_PAGES_NON_PREEMPT_BATCH, or one page on the fallback path.
> > 
> > Switch folio_mc_copy() to copy_mc_highpages(): the whole folio at once
> > under preemptible models, COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time
> > otherwise.
> > 
> > Assisted-by: LLM
> > Signed-off-by: Shivank Garg <shivankg@amd.com>
> > ---
> > On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
> > 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
> > When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
> > 
> > Changes in v2:
> > - Drop the folio_copy() changes as there is little to gain here.
> >   On architectures without copy_mc_to_kernel(), copy_mc_highpages()
> >   falls back to the per-page copy_mc_highpage(), which ends up in
> >   copy_highpage(). A batched copy_highpages() could help there too,
> >   but I haven't tested it on those architectures, so that is left for
> >   a follow-up.
> > - Add COPY_MC_PAGES_NON_PREEMPT_BATCH to limit how much is copied
> >   between cond_resched() calls on non-preemptible kernels.
> > - Rebase onto mm-new (2887e63d2e8e).
> > - v1: https://lore.kernel.org/all/20260427142036.111940-2-shivankg@amd.com
> > 
> > Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are
> > DRAM, node 2 is CXL), with and without this patch.
> > 
> > move_pages() throughput
> > -----------------------
> > A single move_pages() call migrates 1GB of anonymous memory, as 512
> > 2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each),
> > in GiB/s:
> > 
> > 2MB THP:
> >   direction         base            patched         speedup
> >   DRAM0 -> DRAM1    19.57 ± 0.17    27.51 ± 0.78    1.41x
> >   DRAM1 -> DRAM0    19.44 ± 0.12    27.33 ± 0.27    1.41x
> >   DRAM0 -> CXL      18.66 ± 0.18    26.90 ± 0.51    1.44x
> >   CXL   -> DRAM0    15.38 ± 0.59    16.60 ± 0.24    1.08x
> > 
> > 1GB hugetlb:
> >   direction         base            patched         speedup
> >   DRAM0 -> DRAM1    20.34 ± 0.13    53.88 ± 1.48    2.65x
> >   DRAM1 -> DRAM0    19.10 ± 2.65    53.78 ± 0.55    2.82x
> >   DRAM0 -> CXL      19.53 ± 0.08    44.10 ± 0.19    2.26x
> >   CXL   -> DRAM0    16.41 ± 0.07    20.89 ± 0.06    1.27x
> > 
> > kswapd demotion
> > ---------------
> > Node 0 is limited to 16GB of free memory and demotion is enabled. A
> > single thread bound to node 0 writes 96GB of anonymous THP memory, so
> > ~80GB has to be demoted to CXL. Median of 10 runs:
> > 
> >                                base        patched
> >   kswapd CPU time              4.25 s      3.44 s     (-19%)
> >   kswapd CPU per GB demoted    63.6 ms     44.1 ms    (-31%)
> >   demoted by direct reclaim    14.1 GB     0.46 GB
> >   allocstall                   7427        370
> > 
> > With the patch, kswapd keeps up with the writer, so the writer rarely
> > falls back to direct reclaim.
> > 
> > Other workloads
> > ---------------
> > - Memory tiering (numa_balancing=2) with Redis, GAPBS PR/BC, a B-tree
> >   benchmark and Graph500: copying 2MB folios during demotion is
> >   1.18-1.49x faster.
> > - A userspace daemon moving 1GB hugetlb pages between DRAM and CXL
> >   with move_pages() while GUPS runs on them: demotion is 1.43x faster
> >   and promotion 1.28x faster.
> > ---
> >  include/linux/highmem.h | 59 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  mm/util.c               | 18 +++++++++++----
> >  2 files changed, 73 insertions(+), 4 deletions(-)
> > 
> > diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> > index d7aac9de1c8a..e43459dc58b0 100644
> > --- a/include/linux/highmem.h
> > +++ b/include/linux/highmem.h
> > @@ -484,6 +484,65 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
> >  }
> >  #endif
> > 
> > +#if defined(copy_mc_to_kernel) && !defined(CONFIG_HIGHMEM) && \
> > +     !defined(__HAVE_ARCH_COPY_HIGHPAGE)
> > +/*
> > + * copy_mc_highpages() copies the whole range with a single
> > + * copy_mc_to_kernel() call, which cannot reschedule. That is fine under
> > + * preemptible models, but under cooperatively scheduled models callers
> > + * should copy at most COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time.
> > + *
> > + * With a value of 32MB and assuming a memory bandwidth of ~10GBps, this
> > + * should result in a worst case preemption latency of around 3ms when
> > + * copying pages.
> > + */
> 
> This worst-case is absolutely terrible :)
> 
> > +#define COPY_MC_PAGES_NON_PREEMPT_BATCH              (SZ_32M >> PAGE_SHIFT)
> > +
> > +/**
> > + * copy_mc_highpages() - copy a page range with machine check handling
> > + * @to: first destination page
> > + * @from: first source page
> > + * @nr_pages: number of pages
> > + *
> > + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
> > + * source page is queued for memory_failure().
> > + *
> > + * Does not call cond_resched(). Under cooperatively scheduled models the
> > + * caller is expected to limit @nr_pages to COPY_MC_PAGES_NON_PREEMPT_BATCH.
> 
> This is very backwards, callers shouldn't need to check this.
> 
> > + *
> > + * Return: 0 on success, -EHWPOISON if a machine check was handled.
> > + */
> > +static inline int copy_mc_highpages(struct page *to, struct page *from,
> > +             unsigned long nr_pages)
> > +{
> > +     unsigned long len = nr_pages << PAGE_SHIFT;
> > +     unsigned long ret, i;
> > +
> > +     ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
> > +     if (!ret) {
> > +             for (i = 0; i < nr_pages; i++)
> > +                     kmsan_copy_page_meta(to + i, from + i);
> > +             return 0;
> > +     }
> > +     memory_failure_queue(page_to_pfn(from) + ((len - ret) >> PAGE_SHIFT), 0);
> > +     return -EHWPOISON;
> 
> These codepaths look strangely inverted, it would be more natural if you did:
> 
>         if (ret) {
>                 memory_failure_queue(page_to_pfn(from) + ((len - ret) >> PAGE_SHIFT), 0);
>                 return -EWPOISON;
>         }
>         for (i = 0; i < nr_pages; i++)
>                 kmsan_copy_page_meta(to + i, from + i);

Will do this way.

>         /* Or, ideally, kmsan_copy_pages_meta(to, from, nr_pages); */

There is no kmsan_copy_pages_meta, I prefer to keep the loop instead of
introducing new wrapper for this single instance.


>         return 0;
> }
> 
> in my opinion.
> 
> > +}
> > +#else
> > +
> > +#define COPY_MC_PAGES_NON_PREEMPT_BATCH              1
> 
> I would suggest that perhaps this should be the fallback for non-preemptible
> models. But it gets a little weird with PREEMPT_DYNAMIC I think?
> 
> > +
> > +static inline int copy_mc_highpages(struct page *to, struct page *from,
> > +             unsigned long nr_pages)
> > +{
> > +     unsigned long i;
> > +
> > +     for (i = 0; i < nr_pages; i++)
> > +             if (copy_mc_highpage(to + i, from + i))
> > +                     return -EHWPOISON;
> > +     return 0;
> > +}
> > +#endif
> > +
> >  static inline void memcpy_page(struct page *dst_page, size_t dst_off,
> >                              struct page *src_page, size_t src_off,
> >                              size_t len)
> > diff --git a/mm/util.c b/mm/util.c
> > index 7f374bb8bf36..5015cfdfeaa5 100644
> > --- a/mm/util.c
> > +++ b/mm/util.c
> > @@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);
> > 
> >  int folio_mc_copy(struct folio *dst, struct folio *src)
> >  {
> > -     long nr = folio_nr_pages(src);
> > -     long i = 0;
> > +     const unsigned long nr_pages = folio_nr_pages(src);
> > +     /*
> > +      * Copy the largest extent possible to allow for architecture specific
> > +      * extent based optimizations. copy_mc_highpages() does not call
> > +      * cond_resched(), so limit the unit size when running under
> > +      * non-preemptible scheduling models.
> > +      */
> > +     const unsigned long unit = preempt_model_preemptible() ?
> > +                                nr_pages : COPY_MC_PAGES_NON_PREEMPT_BATCH;
> 
> See? now mm is aware of preemption models! eww!
> 
> --
> Pedro