[PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

[PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

[Shivank Garg]

folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
pays the startup cost of the string instruction ("REP; MOVSB" on x86
CPUs with ERMS) plus a cond_resched().

Introduce copy_mc_highpages(), the page range variant of
copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
are in the linear map, copy the whole range with a single
copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
Otherwise, fall back to copy_mc_highpage() one page at a time.

On a machine check, queue the poisoned source page for memory_failure(),
as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
does today.

copy_mc_to_kernel() cannot reschedule, so only take the single-call path
when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
they always take this path. The per-page fallback calls cond_resched()
before each page, as clear_contig_highpages() does. Other architectures
keep copying one page at a time.

Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.

Changes in v3:
- Copy the whole range with one copy_mc_to_kernel() call only when
  CONFIG_PREEMPTION is set, otherwise copy page by page with
  cond_resched(), following clear_contig_highpages() [1] (Pedro, David).
- Handle the machine check first in copy_mc_highpages() (Pedro).
- Rebase onto mm/core.git for-next.
- Link to v2: https://lore.kernel.org/r/20261019-batpagecopy-v2-1-d75e7d0dfaae@amd.com

[1] https://lore.kernel.org/r/20261109-clear-pages-batch-v3-1-6693efe18dff@amd.com

Changes in v2:
- Drop the folio_copy() changes as there is little to gain here.
  On architectures without copy_mc_to_kernel(), copy_mc_highpages()
  falls back to the per-page copy_mc_highpage(), which ends up in
  copy_highpage(). A batched copy_highpages() could help there too,
  but I haven't tested it on those architectures, so that is left for
  a follow-up.
- Add COPY_MC_PAGES_NON_PREEMPT_BATCH to limit how much is copied
  between cond_resched() calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/20260427142036.111940-2-shivankg@amd.com

Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are
DRAM, node 2 is CXL), with and without this patch.

move_pages() throughput
-----------------------
A single move_pages() call migrates 1GB of anonymous memory, as 512
2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each),
in GiB/s:

2MB THP:
  direction         base            patched         speedup
  DRAM0 -> DRAM1    19.57 ± 0.17    27.51 ± 0.78    1.41x
  DRAM1 -> DRAM0    19.44 ± 0.12    27.33 ± 0.27    1.41x
  DRAM0 -> CXL      18.66 ± 0.18    26.90 ± 0.51    1.44x
  CXL   -> DRAM0    15.38 ± 0.59    16.60 ± 0.24    1.08x

1GB hugetlb:
  direction         base            patched         speedup
  DRAM0 -> DRAM1    20.34 ± 0.13    53.88 ± 1.48    2.65x
  DRAM1 -> DRAM0    19.10 ± 2.65    53.78 ± 0.55    2.82x
  DRAM0 -> CXL      19.53 ± 0.08    44.10 ± 0.19    2.26x
  CXL   -> DRAM0    16.41 ± 0.07    20.89 ± 0.06    1.27x

kswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A
single thread bound to node 0 writes 96GB of anonymous THP memory, so
~80GB has to be demoted to CXL. Median of 10 runs:

                               base        patched
  kswapd CPU time              4.25 s      3.44 s     (-19%)
  kswapd CPU per GB demoted    63.6 ms     44.1 ms    (-31%)
  demoted by direct reclaim    14.1 GB     0.46 GB
  allocstall                   7427        370

With the patch, kswapd keeps up with the writer, so the writer rarely
falls back to direct reclaim.
---
 include/linux/highmem.h | 51 +++++++++++++++++++++++++++++++++++++++++++++++++
 mm/util.c               | 14 ++------------
 2 files changed, 53 insertions(+), 12 deletions(-)

diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index d7aac9de1c8a..a79e9d68359c 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 }
 #endif
 
+/**
+ * copy_mc_highpages() - copy a page range with machine check handling
+ * @to: first destination page
+ * @from: first source page
+ * @nr_pages: number of pages
+ *
+ * Range variant of copy_mc_highpage(). On a machine check, the poisoned
+ * source page is queued for memory_failure().
+ *
+ * Context: Process context. May sleep.
+ * Return: 0 on success, -EHWPOISON if a machine check was handled.
+ */
+static inline int copy_mc_highpages(struct page *to, struct page *from,
+		unsigned long nr_pages)
+{
+	unsigned long len = nr_pages << PAGE_SHIFT;
+	unsigned long i, ret;
+
+	might_sleep();
+
+	/*
+	 * When copying we want to operate on the largest extent possible to
+	 * allow for architecture specific extent based optimizations.
+	 *
+	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
+	 * preemptible kernels, where the scheduler can preempt the copy at
+	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
+	 * available and the pages are in the linear map. Otherwise, copy one
+	 * page at a time and reschedule in between.
+	 */
+	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
+	    !IS_ENABLED(CONFIG_HIGHMEM)) {
+		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
+		if (ret) {
+			memory_failure_queue(page_to_pfn(from) +
+					     ((len - ret) >> PAGE_SHIFT), 0);
+			return -EHWPOISON;
+		}
+		for (i = 0; i < nr_pages; i++)
+			kmsan_copy_page_meta(to + i, from + i);
+		return 0;
+	}
+
+	for (i = 0; i < nr_pages; i++) {
+		cond_resched();
+		if (copy_mc_highpage(to + i, from + i))
+			return -EHWPOISON;
+	}
+	return 0;
+}
+
 static inline void memcpy_page(struct page *dst_page, size_t dst_off,
 			       struct page *src_page, size_t src_off,
 			       size_t len)
diff --git a/mm/util.c b/mm/util.c
index c9c3dc4b490f..1277daa3ccd6 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -776,18 +776,8 @@ EXPORT_SYMBOL(folio_copy);
 
 int folio_mc_copy(struct folio *dst, struct folio *src)
 {
-	long nr = folio_nr_pages(src);
-	long i = 0;
-
-	for (;;) {
-		if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i)))
-			return -EHWPOISON;
-		if (++i == nr)
-			break;
-		cond_resched();
-	}
-
-	return 0;
+	return copy_mc_highpages(folio_page(dst, 0), folio_page(src, 0),
+				 folio_nr_pages(src));
 }
 EXPORT_SYMBOL(folio_mc_copy);
 

---
base-commit: c6e0cad6c9b4180bb229db06e5b062c5ce0723bd
change-id: 20260714-batpagecopy-5a2b45910e5d

Best regards,
-- 
Shivank Garg <shivankg@amd.com>