mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths
@ 2026-11-19  7:11 Shivank Garg
  2026-11-19  7:11 ` [PATCH v2 1/8] mm/migrate: extract folio unmap phase Shivank Garg
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:11 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

migrate_pages() handles hugetlb folios, movable_ops pages and LRU folios,
although their locking, mapping and retry requirements differ. It already
dispatches hugetlb folios to a dedicated engine, but that engine rescans
the whole source list on every retry pass, movable_ops pages still pass
through the regular folio unmap and move machinery, and the LRU batching,
synchronous fallback and split folio retry are open-coded in
migrate_pages() itself.

This series sorts the source list by migration class once, gives each
class its own entry point, and splits the LRU batch engine into explicit
unmap, TLB flush, move and undo phases.

Thanks to Zi Yan and David Hildenbrand for the discussions and suggestions
that helped shape this series. I also used LLM tools to refine my ideas,
identify the edge-cases/bugs.

Old flow
========

migrate_pages(mode, reason)
  |
  +-- migrate_hugetlbs()
  |     `-- scan the mixed source list (from) on each retry
  |
  `-- split the remaining list into batches
        |
        +-- MIGRATE_ASYNC
        |     `-- migrate_pages_batch()
        |           inline unmap/retry -> flush -> move/retry
        |
        `-- MIGRATE_SYNC[_LIGHT]
              `-- migrate_pages_sync()
                   +-- try async batch pre-pass
                   `-- retry failures one at a time through
                       migrate_pages_batch()

movable_ops pages are identified by branches inside the batch unmap and
move paths.

New flow
========

migrate_pages(mode, reason)
  |
  +-- sort the source list into hugetlb, movable_ops and LRU lists
  |
  +-- migrate_hugetlbs()
  |     `-- retry each hugetlb folio
  |
  +-- migrate_movable_ops_pages()
  |     `-- allocate, lock, invoke the movable_ops callback and retry
  |
  `-- migrate_lru_folios()
        `-- form bounded LRU batches
              `-- __migrate_lru_folios()
                    +-- async: migrate_folios_batch()
                    +-- sync:
                    |   +-- async batch pre-pass
                    |   `-- retry failures one at a time
                    `-- one async attempt for folios split on the way

migrate_folios_batch()
  |
  +-- migrate_folios_unmap()
  +-- try_to_unmap_flush()
  +-- migrate_folios_move()
  `-- migrate_folios_undo()

migrate_pages() now only sorts and dispatches classes. Each entry point
owns its retries; the LRU entry point also owns batching, the synchronous
fallback and the split folio retry. Folios that are not migrated are still
returned through the original source list.

Motivation
==========

This refactor makes the migration control flow easier to follow and gives
new optimizations clear boundaries and insertion points.

Separating movable_ops also prepares for memdescs, where these pages are
expected to lose their folio representation and folio->lru handoff. The
current list interface remains, but its folio dependencies are now isolated.

As a side benefit, a folio's class is tested once when the source
list is sorted, instead of in every unmap, move and retry pass, and
retrying a hugetlb folio no longer rescans the whole source list.

Behavior Changes
================

No functional change is intended for the LRU and hugetlb migration paths.

The movable_ops pass has three intentional differences:

  - movable_ops pages are attempted before LRU folios;
  - movable_ops pages no longer use the LRU asynchronous pre-pass;
    synchronous callers use their requested mode from the first attempt;
  - a movable_ops callback returning -EAGAIN drops both page locks and
    releases the destination, so a retry allocates a new one.

---
Changes in v2:
- Rebase onto mm-core/for-next.
- Drop the migrate_control patches, will post as a separate
  series; and this series is limited to mm/migrate.c.
- migrate_pages() now sorts hugetlb folios and movable_ops pages onto
  their own lists in a single pass, instead of migrate_hugetlbs() and
  migrate_movable_ops_pages() each scanning the source list.
- movable_ops: count permanent failures in nr_failed_pages as they happen,
  mirroring migrate_hugetlbs(); the totals are unchanged.
- Avoid the migrate_lru_folios() -> __migrate_lru_folios() rename
  by introducing the per-batch function under its final name.
- Some rewording of comments and commit text.
- Link to v1: https://lore.kernel.org/r/20260902-migrate-refactor-shivank-v1-0-9dcca87669c4@amd.com

---
Shivank Garg (8):
      mm/migrate: extract folio unmap phase
      mm/migrate: handle retries in migrate_folios_move()
      mm/migrate: factor out folio splitting on allocation failure
      mm/migrate: use a dedicated list for hugetlb folios
      mm/migrate: add a dedicated movable_ops migration pass
      mm/migrate: rename migrate_pages_batch() to migrate_folios_batch()
      mm/migrate: fold the LRU batch paths into __migrate_lru_folios()
      mm/migrate: add migrate_lru_folios() entry point

 mm/migrate.c | 561 ++++++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 378 insertions(+), 183 deletions(-)
---
base-commit: f7d95bf74091373fad7273db4557df9b5a902335
change-id: 20260824-migrate-refactor-shivank-4ee3949fcdcb

Best regards,
-- 
Shivank Garg <shivankg@amd.com>


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 1/8] mm/migrate: extract folio unmap phase
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
@ 2026-11-19  7:11 ` Shivank Garg
  2026-11-19  7:11 ` [PATCH v2 2/8] mm/migrate: handle retries in migrate_folios_move() Shivank Garg
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:11 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

migrate_pages_batch() was hard to follow with the unmap retry handling
inline alongside TLB flush, move, and cleanup. Extract the unmap phase
into migrate_folios_unmap(), which fills @unmap_folios and @dst_folios
for the steps that follow. This makes the unmap phase boundaries clear
and the code more readable.

No functional change intended.

Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 88 ++++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 59 insertions(+), 29 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index a7c94c5fdfd2..3f1639516623 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1833,36 +1833,31 @@ static void migrate_folios_undo(struct list_head *src_folios,
 }
 
 /*
- * migrate_pages_batch() first unmaps folios in the from list as many as
- * possible, then move the unmapped folios.
+ * Unmap the folios in @from, queuing successfully unmapped sources on
+ * @unmap_folios and their destinations on @dst_folios.  Increment @nr_failed
+ * for each failure.
  *
- * We only batch migration if mode == MIGRATE_ASYNC to avoid to wait a
- * lock or bit when we have locked more than one folio.  Which may cause
- * deadlock (e.g., for loop device).  So, if mode != MIGRATE_ASYNC, the
- * length of the from list must be <= 1.
+ * Return: -ENOMEM if a destination allocation failure stopped the unmap
+ * phase, otherwise 0.  Folios already unmapped remain queued for the move
+ * phase.
  */
-static int migrate_pages_batch(struct list_head *from,
+static int migrate_folios_unmap(struct list_head *from,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
 		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
 		struct list_head *ret_folios, struct list_head *split_folios,
-		struct migrate_pages_stats *stats, int nr_pass)
+		struct list_head *unmap_folios, struct list_head *dst_folios,
+		struct migrate_pages_stats *stats, int nr_pass, int *nr_failed)
 {
 	int retry = 1;
 	int thp_retry = 1;
-	int nr_failed = 0;
 	int nr_retry_pages = 0;
 	int pass = 0;
 	bool is_thp = false;
 	bool is_large = false;
 	struct folio *folio, *folio2, *dst = NULL;
-	int rc, rc_saved = 0, nr_pages;
-	LIST_HEAD(unmap_folios);
-	LIST_HEAD(dst_folios);
+	int rc, nr_pages;
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
 
-	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
-			!list_empty(from) && !list_is_singular(from));
-
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
 		thp_retry = 0;
@@ -1902,7 +1897,7 @@ static int migrate_pages_batch(struct list_head *from,
 			   !list_empty(&folio->_deferred_list) &&
 			   folio_test_partially_mapped(folio)) {
 				if (!try_split_folio(folio, split_folios, mode)) {
-					nr_failed++;
+					*nr_failed += 1;
 					stats->nr_thp_failed += is_thp;
 					stats->nr_thp_split += is_thp;
 					stats->nr_split++;
@@ -1921,7 +1916,7 @@ static int migrate_pages_batch(struct list_head *from,
 			 * list is processed.
 			 */
 			if (!thp_migration_supported() && is_thp) {
-				nr_failed++;
+				*nr_failed += 1;
 				stats->nr_thp_failed++;
 				if (!try_split_folio(folio, split_folios, mode)) {
 					stats->nr_thp_split++;
@@ -1964,7 +1959,7 @@ static int migrate_pages_batch(struct list_head *from,
 				 * When memory is low, don't bother to try to migrate
 				 * other folios, move unmapped folios, then exit.
 				 */
-				nr_failed++;
+				*nr_failed += 1;
 				stats->nr_thp_failed += is_thp;
 				/* Large folio NUMA faulting doesn't split to retry. */
 				if (is_large && !nosplit) {
@@ -1984,7 +1979,7 @@ static int migrate_pages_batch(struct list_head *from,
 						thp_retry += is_thp;
 						nr_retry_pages += nr_pages;
 						/* Undo duplicated failure counting. */
-						nr_failed--;
+						*nr_failed -= 1;
 						stats->nr_thp_failed -= is_thp;
 						break;
 					}
@@ -1993,19 +1988,15 @@ static int migrate_pages_batch(struct list_head *from,
 				stats->nr_failed_pages += nr_pages + nr_retry_pages;
 				/* nr_failed isn't updated for not used */
 				stats->nr_thp_failed += thp_retry;
-				rc_saved = rc;
-				if (list_empty(&unmap_folios))
-					goto out;
-				else
-					goto move;
+				return -ENOMEM;
 			case -EAGAIN:
 				retry++;
 				thp_retry += is_thp;
 				nr_retry_pages += nr_pages;
 				break;
 			case 0:
-				list_move_tail(&folio->lru, &unmap_folios);
-				list_add_tail(&dst->lru, &dst_folios);
+				list_move_tail(&folio->lru, unmap_folios);
+				list_add_tail(&dst->lru, dst_folios);
 				break;
 			default:
 				/*
@@ -2014,17 +2005,56 @@ static int migrate_pages_batch(struct list_head *from,
 				 * removed from migration folio list and not
 				 * retried in the next outer loop.
 				 */
-				nr_failed++;
+				*nr_failed += 1;
 				stats->nr_thp_failed += is_thp;
 				stats->nr_failed_pages += nr_pages;
 				break;
 			}
 		}
 	}
-	nr_failed += retry;
+	*nr_failed += retry;
 	stats->nr_thp_failed += thp_retry;
 	stats->nr_failed_pages += nr_retry_pages;
-move:
+
+	return 0;
+}
+
+/*
+ * migrate_pages_batch() first unmaps as many folios in the source list as
+ * possible, flushes the TLBs, then moves the unmapped folios.
+ *
+ * Only MIGRATE_ASYNC may batch multiple folios.  Waiting for a lock or bit
+ * while multiple folios are locked may deadlock (e.g. with a loop device).
+ * Therefore, if mode != MIGRATE_ASYNC, the source list must contain at most
+ * one folio.
+ */
+static int migrate_pages_batch(struct list_head *from,
+		new_folio_t get_new_folio, free_folio_t put_new_folio,
+		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
+		struct list_head *ret_folios, struct list_head *split_folios,
+		struct migrate_pages_stats *stats, int nr_pass)
+{
+	int retry = 1;
+	int thp_retry = 1;
+	int nr_failed = 0;
+	int nr_retry_pages = 0;
+	int pass = 0;
+	int rc, rc_saved;
+	LIST_HEAD(unmap_folios);
+	LIST_HEAD(dst_folios);
+
+	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
+			!list_empty(from) && !list_is_singular(from));
+
+	rc_saved = migrate_folios_unmap(from, get_new_folio, put_new_folio,
+			private, mode, reason, ret_folios, split_folios,
+			&unmap_folios, &dst_folios, stats, nr_pass,
+			&nr_failed);
+	if (rc_saved && list_empty(&unmap_folios)) {
+		rc = rc_saved;
+		goto out;
+	}
+
 	/* Flush TLBs for all unmapped folios */
 	try_to_unmap_flush();
 

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 2/8] mm/migrate: handle retries in migrate_folios_move()
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
  2026-11-19  7:11 ` [PATCH v2 1/8] mm/migrate: extract folio unmap phase Shivank Garg
@ 2026-11-19  7:11 ` Shivank Garg
  2026-11-19  7:11 ` [PATCH v2 3/8] mm/migrate: factor out folio splitting on allocation failure Shivank Garg
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:11 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

After extracting the unmap phase, migrate_pages_batch() still drives the
retry loop for the move phase and passes its retry bookkeeping to
migrate_folios_move(). This splits the move phase across the two
functions and requires several output parameters.

Move the retry loop and its bookkeeping into migrate_folios_move().
The function now takes the number of passes and reports failures through
@nr_failed, matching migrate_folios_unmap().

No functional change is intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 109 +++++++++++++++++++++++++++++------------------------------
 1 file changed, 54 insertions(+), 55 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 3f1639516623..1f4ddd23cea1 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1757,56 +1757,71 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+/*
+ * Move each folio on @src_folios to its destination on @dst_folios, retrying
+ * folios that return -EAGAIN for up to @nr_pass passes.  Pending folios stay on
+ * @src_folios for migrate_folios_undo().
+ */
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
 		enum migrate_mode mode, enum migrate_reason reason,
 		struct list_head *ret_folios,
-		struct migrate_pages_stats *stats,
-		int *retry, int *thp_retry, int *nr_failed,
-		int *nr_retry_pages)
+		struct migrate_pages_stats *stats, int nr_pass,
+		int *nr_failed)
 {
 	struct folio *folio, *folio2, *dst, *dst2;
+	int retry = 1;
+	int thp_retry = 0;
+	int nr_retry_pages = 0;
 	bool is_thp;
 	int nr_pages;
-	int rc;
+	int pass, rc;
 
-	dst = list_first_entry(dst_folios, struct folio, lru);
-	dst2 = list_next_entry(dst, lru);
-	list_for_each_entry_safe(folio, folio2, src_folios, lru) {
-		is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
-		nr_pages = folio_nr_pages(folio);
+	for (pass = 0; pass < nr_pass && retry; pass++) {
+		retry = 0;
+		thp_retry = 0;
+		nr_retry_pages = 0;
 
-		cond_resched();
+		dst = list_first_entry(dst_folios, struct folio, lru);
+		dst2 = list_next_entry(dst, lru);
+		list_for_each_entry_safe(folio, folio2, src_folios, lru) {
+			is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
+			nr_pages = folio_nr_pages(folio);
 
-		rc = migrate_folio_move(put_new_folio, private,
-				folio, dst, mode,
-				reason, ret_folios);
-		/*
-		 * The rules are:
-		 *	0: folio will be freed
-		 *	-EAGAIN: stay on the src_folios list
-		 *	Other errno: put on ret_folios list
-		 */
-		switch (rc) {
-		case -EAGAIN:
-			*retry += 1;
-			*thp_retry += is_thp;
-			*nr_retry_pages += nr_pages;
-			break;
-		case 0:
-			stats->nr_succeeded += nr_pages;
-			stats->nr_thp_succeeded += is_thp;
-			break;
-		default:
-			*nr_failed += 1;
-			stats->nr_thp_failed += is_thp;
-			stats->nr_failed_pages += nr_pages;
-			break;
+			cond_resched();
+
+			rc = migrate_folio_move(put_new_folio, private,
+					folio, dst, mode, reason, ret_folios);
+			/*
+			 * The rules are:
+			 *	0: folio will be freed
+			 *	-EAGAIN: stay on the src_folios list
+			 *	Other errno: put on ret_folios list
+			 */
+			switch (rc) {
+			case -EAGAIN:
+				retry++;
+				thp_retry += is_thp;
+				nr_retry_pages += nr_pages;
+				break;
+			case 0:
+				stats->nr_succeeded += nr_pages;
+				stats->nr_thp_succeeded += is_thp;
+				break;
+			default:
+				*nr_failed += 1;
+				stats->nr_thp_failed += is_thp;
+				stats->nr_failed_pages += nr_pages;
+				break;
+			}
+			dst = dst2;
+			dst2 = list_next_entry(dst, lru);
 		}
-		dst = dst2;
-		dst2 = list_next_entry(dst, lru);
 	}
+	*nr_failed += retry;
+	stats->nr_thp_failed += thp_retry;
+	stats->nr_failed_pages += nr_retry_pages;
 }
 
 static void migrate_folios_undo(struct list_head *src_folios,
@@ -2034,11 +2049,7 @@ static int migrate_pages_batch(struct list_head *from,
 		struct list_head *ret_folios, struct list_head *split_folios,
 		struct migrate_pages_stats *stats, int nr_pass)
 {
-	int retry = 1;
-	int thp_retry = 1;
 	int nr_failed = 0;
-	int nr_retry_pages = 0;
-	int pass = 0;
 	int rc, rc_saved;
 	LIST_HEAD(unmap_folios);
 	LIST_HEAD(dst_folios);
@@ -2058,21 +2069,9 @@ static int migrate_pages_batch(struct list_head *from,
 	/* Flush TLBs for all unmapped folios */
 	try_to_unmap_flush();
 
-	retry = 1;
-	for (pass = 0; pass < nr_pass && retry; pass++) {
-		retry = 0;
-		thp_retry = 0;
-		nr_retry_pages = 0;
-
-		/* Move the unmapped folios */
-		migrate_folios_move(&unmap_folios, &dst_folios,
-				put_new_folio, private, mode, reason,
-				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages);
-	}
-	nr_failed += retry;
-	stats->nr_thp_failed += thp_retry;
-	stats->nr_failed_pages += nr_retry_pages;
+	/* Move the unmapped folios */
+	migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private,
+			mode, reason, ret_folios, stats, nr_pass, &nr_failed);
 
 	rc = rc_saved ? : nr_failed;
 out:

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 3/8] mm/migrate: factor out folio splitting on allocation failure
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
  2026-11-19  7:11 ` [PATCH v2 1/8] mm/migrate: extract folio unmap phase Shivank Garg
  2026-11-19  7:11 ` [PATCH v2 2/8] mm/migrate: handle retries in migrate_folios_move() Shivank Garg
@ 2026-11-19  7:11 ` Shivank Garg
  2026-11-19  7:11 ` [PATCH v2 4/8] mm/migrate: use a dedicated list for hugetlb folios Shivank Garg
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:11 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

If destination folio allocation fails, migrate_folios_unmap() may split
a large folio and retry migration with smaller folios. Factor this logic
out into migrate_folio_split_on_alloc_fail() to simplify the allocation
failure path.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 69 +++++++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 43 insertions(+), 26 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 1f4ddd23cea1..e4e3d1683d1e 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1847,6 +1847,34 @@ static void migrate_folios_undo(struct list_head *src_folios,
 	}
 }
 
+static int migrate_folio_split_on_alloc_fail(struct folio *folio,
+		struct list_head *split_folios, enum migrate_mode mode,
+		enum migrate_reason reason, struct migrate_pages_stats *stats)
+{
+	bool is_thp = folio_test_pmd_mappable(folio);
+	int rc;
+
+	/* Large folio NUMA faulting doesn't split to retry. */
+	if (!folio_test_large(folio) || reason == MR_NUMA_MISPLACED)
+		return -ENOMEM;
+
+	rc = try_split_folio(folio, split_folios, mode);
+	if (!rc) {
+		stats->nr_thp_split += is_thp;
+		stats->nr_split++;
+		return 0;
+	}
+
+	/*
+	 * Try again to split large folio to mitigate the failure of longterm
+	 * pinning.
+	 */
+	if (reason == MR_LONGTERM_PIN && rc == -EAGAIN)
+		return -EAGAIN;
+
+	return -ENOMEM;
+}
+
 /*
  * Unmap the folios in @from, queuing successfully unmapped sources on
  * @unmap_folios and their destinations on @dst_folios.  Increment @nr_failed
@@ -1868,10 +1896,8 @@ static int migrate_folios_unmap(struct list_head *from,
 	int nr_retry_pages = 0;
 	int pass = 0;
 	bool is_thp = false;
-	bool is_large = false;
 	struct folio *folio, *folio2, *dst = NULL;
-	int rc, nr_pages;
-	bool nosplit = (reason == MR_NUMA_MISPLACED);
+	int rc, split_rc, nr_pages;
 
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
@@ -1879,7 +1905,6 @@ static int migrate_folios_unmap(struct list_head *from,
 		nr_retry_pages = 0;
 
 		list_for_each_entry_safe(folio, folio2, from, lru) {
-			is_large = folio_test_large(folio);
 			is_thp = folio_test_pmd_mappable(folio);
 			nr_pages = folio_nr_pages(folio);
 
@@ -1976,28 +2001,20 @@ static int migrate_folios_unmap(struct list_head *from,
 				 */
 				*nr_failed += 1;
 				stats->nr_thp_failed += is_thp;
-				/* Large folio NUMA faulting doesn't split to retry. */
-				if (is_large && !nosplit) {
-					int ret = try_split_folio(folio, split_folios, mode);
-
-					if (!ret) {
-						stats->nr_thp_split += is_thp;
-						stats->nr_split++;
-						break;
-					} else if (reason == MR_LONGTERM_PIN &&
-						   ret == -EAGAIN) {
-						/*
-						 * Try again to split large folio to
-						 * mitigate the failure of longterm pinning.
-						 */
-						retry++;
-						thp_retry += is_thp;
-						nr_retry_pages += nr_pages;
-						/* Undo duplicated failure counting. */
-						*nr_failed -= 1;
-						stats->nr_thp_failed -= is_thp;
-						break;
-					}
+
+				split_rc = migrate_folio_split_on_alloc_fail(folio,
+						split_folios, mode, reason, stats);
+				if (!split_rc)
+					break;
+				/* Retry this folio in a later pass. */
+				if (split_rc == -EAGAIN) {
+					retry++;
+					thp_retry += is_thp;
+					nr_retry_pages += nr_pages;
+					/* Undo duplicated failure counting. */
+					*nr_failed -= 1;
+					stats->nr_thp_failed -= is_thp;
+					break;
 				}
 
 				stats->nr_failed_pages += nr_pages + nr_retry_pages;

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 4/8] mm/migrate: use a dedicated list for hugetlb folios
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
                   ` (2 preceding siblings ...)
  2026-11-19  7:11 ` [PATCH v2 3/8] mm/migrate: factor out folio splitting on allocation failure Shivank Garg
@ 2026-11-19  7:11 ` Shivank Garg
  2026-11-19  7:12 ` [PATCH v2 5/8] mm/migrate: add a dedicated movable_ops migration pass Shivank Garg
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:11 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

migrate_hugetlbs() scans the full source list on every retry pass, even
though it only handles hugetlb folios, and migrate_pages() then scans the
list again to set aside hugetlb folios that exhausted their retries.

Move hugetlb folios to a local list in migrate_pages() before migration
and let migrate_hugetlbs() work on that list alone. Whatever was not
migrated is put back through @ret_folios on the way out, which removes the
later hugetlb check from the batching loop.

While at it, fix the function comment which was copied from migrate_pages()
and drop what is migrate_pages()'s contract with outside callers.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 30 ++++++++++++++----------------
 1 file changed, 14 insertions(+), 16 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index e4e3d1683d1e..3dfdf46681e3 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1662,11 +1662,11 @@ struct migrate_pages_stats {
 };
 
 /*
- * Returns the number of hugetlb folios that were not migrated, or an error code
- * after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no hugetlb folios are movable
- * any more because the list has become empty or no retryable hugetlb folios
- * exist any more. It is caller's responsibility to call putback_movable_pages()
- * only if ret != 0.
+ * Migrate the hugetlb folios on @from, trying each folio up to
+ * NR_MAX_MIGRATE_PAGES_RETRY times.  Permanent failures are moved to
+ * @ret_folios; any other folio that was not migrated stays on @from.
+ *
+ * Return the number of failed folios, or a negative errno.
  */
 static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 			    free_folio_t put_new_folio, unsigned long private,
@@ -1686,9 +1686,6 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 		nr_retry_pages = 0;
 
 		list_for_each_entry_safe(folio, folio2, from, lru) {
-			if (!folio_test_hugetlb(folio))
-				continue;
-
 			nr_pages = folio_nr_pages(folio);
 
 			cond_resched();
@@ -2184,6 +2181,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	int nr_pages;
 	struct folio *folio, *folio2;
 	LIST_HEAD(folios);
+	LIST_HEAD(hugetlb_folios);
 	LIST_HEAD(ret_folios);
 	LIST_HEAD(split_folios);
 	struct migrate_pages_stats stats;
@@ -2192,20 +2190,19 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 
 	memset(&stats, 0, sizeof(stats));
 
-	rc_gather = migrate_hugetlbs(from, get_new_folio, put_new_folio, private,
-				     mode, reason, &stats, &ret_folios);
+	list_for_each_entry_safe(folio, folio2, from, lru)
+		if (folio_test_hugetlb(folio))
+			list_move_tail(&folio->lru, &hugetlb_folios);
+
+	rc_gather = migrate_hugetlbs(&hugetlb_folios, get_new_folio,
+				     put_new_folio, private, mode, reason,
+				     &stats, &ret_folios);
 	if (rc_gather < 0)
 		goto out;
 
 again:
 	nr_pages = 0;
 	list_for_each_entry_safe(folio, folio2, from, lru) {
-		/* Retried hugetlb folios will be kept in list  */
-		if (folio_test_hugetlb(folio)) {
-			list_move_tail(&folio->lru, &ret_folios);
-			continue;
-		}
-
 		nr_pages += folio_nr_pages(folio);
 		if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
 			break;
@@ -2248,6 +2245,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	 * Put the permanent failure folio back to migration list, they
 	 * will be put back to the right list by the caller.
 	 */
+	list_splice_tail(&hugetlb_folios, &ret_folios);
 	list_splice(&ret_folios, from);
 
 	/*

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 5/8] mm/migrate: add a dedicated movable_ops migration pass
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
                   ` (3 preceding siblings ...)
  2026-11-19  7:11 ` [PATCH v2 4/8] mm/migrate: use a dedicated list for hugetlb folios Shivank Garg
@ 2026-11-19  7:12 ` Shivank Garg
  2026-11-19  7:12 ` [PATCH v2 6/8] mm/migrate: rename migrate_pages_batch() to migrate_folios_batch() Shivank Garg
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:12 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

Pages with movable_ops transfer ownership through a driver callback and do
not need the unmap, TLB flush, copy, or LRU handling used for LRU folios.
These pages are expected to stop being represented as folios. Separating
their migration from the LRU folio path prepares for that conversion.

Move movable_ops pages to their own list in migrate_pages(), next to
hugetlb folios, and migrate them in migrate_movable_ops_pages() before the
LRU folios. The new pass handles allocation, locking, retries and cleanup
itself; pages that are not migrated are returned through the source list
as before. Remove the now-unreachable movable_ops branches from the LRU
path.

This changes three details of movable_ops migration:
 - Pages are processed before LRU folios rather than in source-list order.
 - Synchronous callers use the requested mode from the first attempt
   instead of the LRU asynchronous pre-pass.
 - A callback returning -EAGAIN ends the current attempt: both page locks
   are dropped and the destination is released, so a later retry
   allocates a fresh one.

The MIGRATE_SYNC_LIGHT !folio_test_uptodate() check is not carried over:
PG_movable_ops shares its bit with PG_uptodate, so it never fired for
these pages.

Suggested-by: Zi Yan <ziy@nvidia.com>
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 135 insertions(+), 17 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 3dfdf46681e3..bfd38e6339cf 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1233,7 +1233,7 @@ static void migrate_folio_undo_dst(struct folio *dst, bool locked,
 static void migrate_folio_done(struct folio *src,
 			       enum migrate_reason reason)
 {
-	if (likely(!page_has_movable_ops(&src->page)) && reason != MR_DEMOTION)
+	if (reason != MR_DEMOTION)
 		mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON +
 				    folio_is_file_lru(src), -folio_nr_pages(src));
 
@@ -1342,11 +1342,6 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		goto out;
 	dst_locked = true;
 
-	if (unlikely(page_has_movable_ops(&src->page))) {
-		__migrate_folio_record(dst, old_folio_state, anon_vma);
-		return 0;
-	}
-
 	/*
 	 * Corner case handling:
 	 * 1. When a new swap-cache page is read into, it is added to the LRU
@@ -1409,13 +1404,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	prev = dst->lru.prev;
 	list_del(&dst->lru);
 
-	if (unlikely(page_has_movable_ops(&src->page))) {
-		rc = migrate_movable_ops_page(&dst->page, &src->page, mode);
-		if (rc)
-			goto out;
-		goto out_unlock_both;
-	}
-
 	if (folio_order(src) > 1 &&
 	    !data_race(list_empty(&src->_deferred_list))) {
 		src_deferred_split = true;
@@ -1451,7 +1439,6 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	if (old_folio_state & FOLIO_WAS_MAPPED)
 		remove_migration_ptes(src, dst, 0);
 
-out_unlock_both:
 	folio_unlock(dst);
 	folio_set_owner_migrate_reason(dst, reason);
 	/*
@@ -1844,6 +1831,124 @@ static void migrate_folios_undo(struct list_head *src_folios,
 	}
 }
 
+/*
+ * Migrate one isolated movable_ops page. Keep @src on its list for retry and
+ * move it to @ret_folios on permanent failure.
+ *
+ * Return 0 on success or a negative error.
+ */
+static int move_movable_ops_page(struct folio *src,
+		new_folio_t get_new_folio, free_folio_t put_new_folio,
+		unsigned long private, enum migrate_mode mode,
+		enum migrate_reason reason, struct list_head *ret_folios)
+{
+	struct folio *dst;
+	int rc = -EAGAIN;
+
+	dst = get_new_folio(src, private);
+	if (!dst)
+		return -ENOMEM;
+
+	if (!folio_trylock(src)) {
+		if (mode == MIGRATE_ASYNC)
+			goto out_put_dst;
+		/*
+		 * As in migrate_folio_unmap(), direct compaction must not
+		 * sleep on the page lock.
+		 */
+		if (current->flags & PF_MEMALLOC)
+			goto out_put_dst;
+		folio_lock(src);
+	}
+
+	if (unlikely(!folio_trylock(dst)))
+		goto out_unlock_src;
+
+	rc = migrate_movable_ops_page(&dst->page, &src->page, mode);
+	folio_unlock(dst);
+	if (rc)
+		goto out_unlock_src;
+
+	folio_set_owner_migrate_reason(dst, reason);
+	/* Drop migration's reference after transferring ownership to dst. */
+	folio_put(dst);
+
+	list_del(&src->lru);
+	folio_unlock(src);
+
+	if (reason != MR_MEMORY_FAILURE)
+		folio_put(src);
+
+	return 0;
+
+out_unlock_src:
+	folio_unlock(src);
+out_put_dst:
+	if (put_new_folio)
+		put_new_folio(dst, private);
+	else
+		folio_put(dst);
+
+	if (rc != -EAGAIN)
+		list_move_tail(&src->lru, ret_folios);
+
+	return rc;
+}
+
+/*
+ * Migrate the movable_ops pages on @from, trying each page up to
+ * NR_MAX_MIGRATE_PAGES_RETRY times.  Permanent failures are moved to
+ * @ret_folios; any other page that was not migrated stays on @from.
+ * movable_ops pages are order-0, so each failure is one failed page.
+ *
+ * Return the number of failed pages, or a negative errno.
+ */
+static int migrate_movable_ops_pages(struct list_head *from,
+		new_folio_t get_new_folio, free_folio_t put_new_folio,
+		unsigned long private, enum migrate_mode mode,
+		enum migrate_reason reason, struct migrate_pages_stats *stats,
+		struct list_head *ret_folios)
+{
+	int retry = 1;
+	int nr_failed = 0;
+	int pass;
+	struct folio *folio, *folio2;
+	int rc;
+
+	for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && retry; pass++) {
+		retry = 0;
+
+		list_for_each_entry_safe(folio, folio2, from, lru) {
+			cond_resched();
+
+			rc = move_movable_ops_page(folio, get_new_folio,
+						   put_new_folio, private,
+						   mode, reason, ret_folios);
+			switch (rc) {
+			case -ENOMEM:
+				/* Count this page and those awaiting retry. */
+				stats->nr_failed_pages += 1 + retry;
+				return -ENOMEM;
+			case -EAGAIN:
+				retry++;
+				break;
+			case 0:
+				stats->nr_succeeded++;
+				break;
+			default:
+				nr_failed++;
+				stats->nr_failed_pages++;
+				break;
+			}
+		}
+	}
+	/* Count pages that exhausted the retry limit. */
+	nr_failed += retry;
+	stats->nr_failed_pages += retry;
+
+	return nr_failed;
+}
+
 static int migrate_folio_split_on_alloc_fail(struct folio *folio,
 		struct list_head *split_folios, enum migrate_mode mode,
 		enum migrate_reason reason, struct migrate_pages_stats *stats)
@@ -1969,8 +2074,7 @@ static int migrate_folios_unmap(struct list_head *from,
 			 * If we are holding the last folio reference, the folio
 			 * was freed from under us, so just drop our reference.
 			 */
-			if (likely(!page_has_movable_ops(&folio->page)) &&
-			    folio_ref_count(folio) == 1) {
+			if (folio_ref_count(folio) == 1) {
 				folio_clear_active(folio);
 				folio_clear_unevictable(folio);
 				list_del(&folio->lru);
@@ -2182,6 +2286,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	struct folio *folio, *folio2;
 	LIST_HEAD(folios);
 	LIST_HEAD(hugetlb_folios);
+	LIST_HEAD(movable_ops_pages);
 	LIST_HEAD(ret_folios);
 	LIST_HEAD(split_folios);
 	struct migrate_pages_stats stats;
@@ -2190,9 +2295,12 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 
 	memset(&stats, 0, sizeof(stats));
 
-	list_for_each_entry_safe(folio, folio2, from, lru)
+	list_for_each_entry_safe(folio, folio2, from, lru) {
 		if (folio_test_hugetlb(folio))
 			list_move_tail(&folio->lru, &hugetlb_folios);
+		else if (page_has_movable_ops(&folio->page))
+			list_move_tail(&folio->lru, &movable_ops_pages);
+	}
 
 	rc_gather = migrate_hugetlbs(&hugetlb_folios, get_new_folio,
 				     put_new_folio, private, mode, reason,
@@ -2200,6 +2308,15 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	if (rc_gather < 0)
 		goto out;
 
+	rc = migrate_movable_ops_pages(&movable_ops_pages, get_new_folio,
+				       put_new_folio, private, mode, reason,
+				       &stats, &ret_folios);
+	if (rc < 0) {
+		rc_gather = rc;
+		goto out;
+	}
+	rc_gather += rc;
+
 again:
 	nr_pages = 0;
 	list_for_each_entry_safe(folio, folio2, from, lru) {
@@ -2246,6 +2363,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	 * will be put back to the right list by the caller.
 	 */
 	list_splice_tail(&hugetlb_folios, &ret_folios);
+	list_splice_tail(&movable_ops_pages, &ret_folios);
 	list_splice(&ret_folios, from);
 
 	/*

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 6/8] mm/migrate: rename migrate_pages_batch() to migrate_folios_batch()
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
                   ` (4 preceding siblings ...)
  2026-11-19  7:12 ` [PATCH v2 5/8] mm/migrate: add a dedicated movable_ops migration pass Shivank Garg
@ 2026-11-19  7:12 ` Shivank Garg
  2026-11-19  7:12 ` [PATCH v2 7/8] mm/migrate: fold the LRU batch paths into __migrate_lru_folios() Shivank Garg
  2026-11-19  7:12 ` [PATCH v2 8/8] mm/migrate: add migrate_lru_folios() entry point Shivank Garg
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:12 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

migrate_pages_batch() operates on folios and calls the folio-based unmap,
move and undo helpers. Now that movable_ops-based page migration has been
separated out, rename it to migrate_folios_batch().

No functional changes intended.

Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index bfd38e6339cf..1d44c6ce48cd 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2021,7 +2021,7 @@ static int migrate_folios_unmap(struct list_head *from,
 			 * always reports success when its fromlist is empty.
 			 * stats->nr_thp_failed should be increased too,
 			 * otherwise stats inconsistency will happen when
-			 * migrate_pages_batch is called via migrate_pages()
+			 * migrate_folios_batch is called via migrate_pages()
 			 * with MIGRATE_SYNC and MIGRATE_ASYNC.
 			 *
 			 * Only check it without removing it from the list.
@@ -2153,7 +2153,7 @@ static int migrate_folios_unmap(struct list_head *from,
 }
 
 /*
- * migrate_pages_batch() first unmaps as many folios in the source list as
+ * migrate_folios_batch() first unmaps as many folios in the source list as
  * possible, flushes the TLBs, then moves the unmapped folios.
  *
  * Only MIGRATE_ASYNC may batch multiple folios.  Waiting for a lock or bit
@@ -2161,7 +2161,7 @@ static int migrate_folios_unmap(struct list_head *from,
  * Therefore, if mode != MIGRATE_ASYNC, the source list must contain at most
  * one folio.
  */
-static int migrate_pages_batch(struct list_head *from,
+static int migrate_folios_batch(struct list_head *from,
 		new_folio_t get_new_folio, free_folio_t put_new_folio,
 		unsigned long private, enum migrate_mode mode, enum migrate_reason reason,
 		struct list_head *ret_folios, struct list_head *split_folios,
@@ -2212,9 +2212,9 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 
 	memset(&astats, 0, sizeof(astats));
 	/* Try to migrate in batch with MIGRATE_ASYNC mode firstly */
-	rc = migrate_pages_batch(from, get_new_folio, put_new_folio, private, MIGRATE_ASYNC,
-				 reason, &folios, split_folios, &astats,
-				 NR_MAX_MIGRATE_ASYNC_RETRY);
+	rc = migrate_folios_batch(from, get_new_folio, put_new_folio, private,
+			MIGRATE_ASYNC, reason, &folios, split_folios, &astats,
+			NR_MAX_MIGRATE_ASYNC_RETRY);
 	stats->nr_succeeded += astats.nr_succeeded;
 	stats->nr_thp_succeeded += astats.nr_thp_succeeded;
 	stats->nr_thp_split += astats.nr_thp_split;
@@ -2239,9 +2239,9 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 	list_splice_tail_init(&folios, from);
 	while (!list_empty(from)) {
 		list_move(from->next, &folios);
-		rc = migrate_pages_batch(&folios, get_new_folio, put_new_folio,
-					 private, mode, reason, ret_folios,
-					 split_folios, stats, NR_MAX_MIGRATE_SYNC_RETRY);
+		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
+				private, mode, reason, ret_folios, split_folios,
+				stats, NR_MAX_MIGRATE_SYNC_RETRY);
 		list_splice_tail_init(&folios, ret_folios);
 		if (rc < 0)
 			return rc;
@@ -2329,7 +2329,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	else
 		list_splice_init(from, &folios);
 	if (mode == MIGRATE_ASYNC)
-		rc = migrate_pages_batch(&folios, get_new_folio, put_new_folio,
+		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
 				private, mode, reason, &ret_folios,
 				&split_folios, &stats,
 				NR_MAX_MIGRATE_PAGES_RETRY);
@@ -2349,7 +2349,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		 * is counted as 1 failure already.  And, we only try to migrate
 		 * with minimal effort, force MIGRATE_ASYNC mode and retry once.
 		 */
-		migrate_pages_batch(&split_folios, get_new_folio,
+		migrate_folios_batch(&split_folios, get_new_folio,
 				put_new_folio, private, MIGRATE_ASYNC, reason,
 				&ret_folios, NULL, &stats, 1);
 		list_splice_tail_init(&split_folios, &ret_folios);

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 7/8] mm/migrate: fold the LRU batch paths into __migrate_lru_folios()
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
                   ` (5 preceding siblings ...)
  2026-11-19  7:12 ` [PATCH v2 6/8] mm/migrate: rename migrate_pages_batch() to migrate_folios_batch() Shivank Garg
@ 2026-11-19  7:12 ` Shivank Garg
  2026-11-19  7:12 ` [PATCH v2 8/8] mm/migrate: add migrate_lru_folios() entry point Shivank Garg
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:12 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

migrate_pages() calls migrate_folios_batch() directly for MIGRATE_ASYNC
and migrate_pages_sync() otherwise, and then retries the folios split
along the way itself.

Turn migrate_pages_sync() into __migrate_lru_folios(), which migrates one
batch of LRU folios in any mode and owns the split folio retry. This
leaves migrate_pages() with a single call per LRU batch.

Keep the existing retry and accounting behavior unchanged.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 70 ++++++++++++++++++++++++++++++++++--------------------------
 1 file changed, 40 insertions(+), 30 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 1d44c6ce48cd..bad69e1e2ac4 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2200,20 +2200,34 @@ static int migrate_folios_batch(struct list_head *from,
 	return rc;
 }
 
-static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
+/*
+ * Migrate one batch of LRU folios.  MIGRATE_ASYNC processes it directly.  The
+ * other modes first make a short asynchronous pass, then retry each remaining
+ * folio separately in the requested mode.
+ */
+static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
 		free_folio_t put_new_folio, unsigned long private,
 		enum migrate_mode mode, enum migrate_reason reason,
-		struct list_head *ret_folios, struct list_head *split_folios,
+		struct list_head *ret_folios,
 		struct migrate_pages_stats *stats)
 {
 	int rc, nr_failed = 0;
 	LIST_HEAD(folios);
+	LIST_HEAD(split_folios);
 	struct migrate_pages_stats astats;
 
+	if (mode == MIGRATE_ASYNC) {
+		rc = migrate_folios_batch(from, get_new_folio, put_new_folio,
+				private, mode, reason, ret_folios,
+				&split_folios, stats,
+				NR_MAX_MIGRATE_PAGES_RETRY);
+		goto out;
+	}
+
 	memset(&astats, 0, sizeof(astats));
 	/* Try to migrate in batch with MIGRATE_ASYNC mode firstly */
 	rc = migrate_folios_batch(from, get_new_folio, put_new_folio, private,
-			MIGRATE_ASYNC, reason, &folios, split_folios, &astats,
+			MIGRATE_ASYNC, reason, &folios, &split_folios, &astats,
 			NR_MAX_MIGRATE_ASYNC_RETRY);
 	stats->nr_succeeded += astats.nr_succeeded;
 	stats->nr_thp_succeeded += astats.nr_thp_succeeded;
@@ -2223,7 +2237,7 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 		stats->nr_failed_pages += astats.nr_failed_pages;
 		stats->nr_thp_failed += astats.nr_thp_failed;
 		list_splice_tail(&folios, ret_folios);
-		return rc;
+		goto out;
 	}
 	stats->nr_thp_failed += astats.nr_thp_split;
 	/*
@@ -2240,15 +2254,31 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio,
 	while (!list_empty(from)) {
 		list_move(from->next, &folios);
 		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, ret_folios, split_folios,
-				stats, NR_MAX_MIGRATE_SYNC_RETRY);
+				private, mode, reason, ret_folios,
+				&split_folios, stats,
+				NR_MAX_MIGRATE_SYNC_RETRY);
 		list_splice_tail_init(&folios, ret_folios);
 		if (rc < 0)
-			return rc;
+			goto out;
 		nr_failed += rc;
 	}
+	rc = nr_failed;
+out:
+	if (rc < 0) {
+		list_splice_tail(&split_folios, ret_folios);
+	} else if (!list_empty(&split_folios)) {
+		/*
+		 * Folios split along the way get one asynchronous attempt at
+		 * their new order.  Their failure is not counted: the large
+		 * folio they came from was already counted as one failure.
+		 */
+		migrate_folios_batch(&split_folios, get_new_folio, put_new_folio,
+				private, MIGRATE_ASYNC, reason, ret_folios,
+				NULL, stats, 1);
+		list_splice_tail_init(&split_folios, ret_folios);
+	}
 
-	return nr_failed;
+	return rc;
 }
 
 /*
@@ -2288,7 +2318,6 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	LIST_HEAD(hugetlb_folios);
 	LIST_HEAD(movable_ops_pages);
 	LIST_HEAD(ret_folios);
-	LIST_HEAD(split_folios);
 	struct migrate_pages_stats stats;
 
 	trace_mm_migrate_pages_start(mode, reason);
@@ -2328,32 +2357,13 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		list_cut_before(&folios, from, &folio2->lru);
 	else
 		list_splice_init(from, &folios);
-	if (mode == MIGRATE_ASYNC)
-		rc = migrate_folios_batch(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, &ret_folios,
-				&split_folios, &stats,
-				NR_MAX_MIGRATE_PAGES_RETRY);
-	else
-		rc = migrate_pages_sync(&folios, get_new_folio, put_new_folio,
-				private, mode, reason, &ret_folios,
-				&split_folios, &stats);
+	rc = __migrate_lru_folios(&folios, get_new_folio, put_new_folio,
+			private, mode, reason, &ret_folios, &stats);
 	list_splice_tail_init(&folios, &ret_folios);
 	if (rc < 0) {
 		rc_gather = rc;
-		list_splice_tail(&split_folios, &ret_folios);
 		goto out;
 	}
-	if (!list_empty(&split_folios)) {
-		/*
-		 * Failure isn't counted since all split folios of a large folio
-		 * is counted as 1 failure already.  And, we only try to migrate
-		 * with minimal effort, force MIGRATE_ASYNC mode and retry once.
-		 */
-		migrate_folios_batch(&split_folios, get_new_folio,
-				put_new_folio, private, MIGRATE_ASYNC, reason,
-				&ret_folios, NULL, &stats, 1);
-		list_splice_tail_init(&split_folios, &ret_folios);
-	}
 	rc_gather += rc;
 	if (!list_empty(from))
 		goto again;

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 8/8] mm/migrate: add migrate_lru_folios() entry point
  2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
                   ` (6 preceding siblings ...)
  2026-11-19  7:12 ` [PATCH v2 7/8] mm/migrate: fold the LRU batch paths into __migrate_lru_folios() Shivank Garg
@ 2026-11-19  7:12 ` Shivank Garg
  7 siblings, 0 replies; 9+ messages in thread
From: Shivank Garg @ 2026-11-19  7:12 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Brost,
	Joshua Hahn, Byungchul Park, Gregory Price, Ying Huang,
	Alistair Popple, Lorenzo Stoakes, Muchun Song, Oscar Salvador,
	Yiannis Nikolakopoulos, Bharata B Rao, David Rientjes,
	Huan Nguyen, Baolin Wang, Matthew Wilcox, Hugh Dickins,
	Vlastimil Babka
  Cc: linux-mm, linux-kernel, Shivank Garg

The NR_MAX_BATCHED_MIGRATION limit is specific to LRU folio migration,
where the batch engine can keep multiple folios locked at once.

Move the batching loop out of migrate_pages() into a new
migrate_lru_folios(), which feeds __migrate_lru_folios() one batch at a
time. This leaves migrate_pages() responsible only for sorting folios by
migration class and dispatching each class to its entry point.

No functional change intended.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 57 ++++++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 40 insertions(+), 17 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index bad69e1e2ac4..9a39468797d1 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2281,6 +2281,45 @@ static int __migrate_lru_folios(struct list_head *from, new_folio_t get_new_foli
 	return rc;
 }
 
+/*
+ * Migrate LRU folios.  The batch engine keeps every folio of a batch locked
+ * at once, so feed it at most NR_MAX_BATCHED_MIGRATION pages at a time
+ * rather than however many the caller supplied.
+ */
+static int migrate_lru_folios(struct list_head *from, new_folio_t get_new_folio,
+		free_folio_t put_new_folio, unsigned long private,
+		enum migrate_mode mode, enum migrate_reason reason,
+		struct list_head *ret_folios,
+		struct migrate_pages_stats *stats)
+{
+	struct folio *folio, *folio2;
+	int rc, nr_failed = 0;
+	int nr_pages;
+	LIST_HEAD(folios);
+
+	while (!list_empty(from)) {
+		nr_pages = 0;
+		list_for_each_entry_safe(folio, folio2, from, lru) {
+			nr_pages += folio_nr_pages(folio);
+			if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
+				break;
+		}
+		if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
+			list_cut_before(&folios, from, &folio2->lru);
+		else
+			list_splice_init(from, &folios);
+
+		rc = __migrate_lru_folios(&folios, get_new_folio, put_new_folio,
+				private, mode, reason, ret_folios, stats);
+		list_splice_tail_init(&folios, ret_folios);
+		if (rc < 0)
+			return rc;
+		nr_failed += rc;
+	}
+
+	return nr_failed;
+}
+
 /*
  * migrate_pages - migrate the folios specified in a list, to the free folios
  *		   supplied as the target for the page migration
@@ -2312,9 +2351,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		enum migrate_mode mode, enum migrate_reason reason, unsigned int *ret_succeeded)
 {
 	int rc, rc_gather;
-	int nr_pages;
 	struct folio *folio, *folio2;
-	LIST_HEAD(folios);
 	LIST_HEAD(hugetlb_folios);
 	LIST_HEAD(movable_ops_pages);
 	LIST_HEAD(ret_folios);
@@ -2346,27 +2383,13 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	}
 	rc_gather += rc;
 
-again:
-	nr_pages = 0;
-	list_for_each_entry_safe(folio, folio2, from, lru) {
-		nr_pages += folio_nr_pages(folio);
-		if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
-			break;
-	}
-	if (nr_pages >= NR_MAX_BATCHED_MIGRATION)
-		list_cut_before(&folios, from, &folio2->lru);
-	else
-		list_splice_init(from, &folios);
-	rc = __migrate_lru_folios(&folios, get_new_folio, put_new_folio,
+	rc = migrate_lru_folios(from, get_new_folio, put_new_folio,
 			private, mode, reason, &ret_folios, &stats);
-	list_splice_tail_init(&folios, &ret_folios);
 	if (rc < 0) {
 		rc_gather = rc;
 		goto out;
 	}
 	rc_gather += rc;
-	if (!list_empty(from))
-		goto again;
 out:
 	/*
 	 * Put the permanent failure folio back to migration list, they

-- 
2.53.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-11-19  7:13 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19  7:11 [PATCH v2 0/8] mm/migrate: separate hugetlb, movable_ops and LRU migration paths Shivank Garg
2026-11-19  7:11 ` [PATCH v2 1/8] mm/migrate: extract folio unmap phase Shivank Garg
2026-11-19  7:11 ` [PATCH v2 2/8] mm/migrate: handle retries in migrate_folios_move() Shivank Garg
2026-11-19  7:11 ` [PATCH v2 3/8] mm/migrate: factor out folio splitting on allocation failure Shivank Garg
2026-11-19  7:11 ` [PATCH v2 4/8] mm/migrate: use a dedicated list for hugetlb folios Shivank Garg
2026-11-19  7:12 ` [PATCH v2 5/8] mm/migrate: add a dedicated movable_ops migration pass Shivank Garg
2026-11-19  7:12 ` [PATCH v2 6/8] mm/migrate: rename migrate_pages_batch() to migrate_folios_batch() Shivank Garg
2026-11-19  7:12 ` [PATCH v2 7/8] mm/migrate: fold the LRU batch paths into __migrate_lru_folios() Shivank Garg
2026-11-19  7:12 ` [PATCH v2 8/8] mm/migrate: add migrate_lru_folios() entry point Shivank Garg

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®