[PATCH RFC 0/8] mm/huge_memory: clean up and speed up large folio splitting

[PATCH RFC 0/8] mm/huge_memory: clean up and speed up large folio splitting

[Kairui Song via B4 Relay]

Hi all,

This series slghtly cleans up the large folio split path and reduces
its overhead: uniformly splitting PMD sized file folios to order 0
is over 20% faster.

I realize this might not be the best time for pushing more optimizations,
but I am posting this as an early RFC as it adds two helpers, folio_prev()
and xas_prev_canonical(), and uses them to walk the after-split folios
in reverse, saving a few local variables, avoiding a for-loop pitfall [1],
and publishing the folios with a single Xarray walk.

Walking backwards also changes the order the pieces are put on the LRU or
the reclaim list: they are now in ascending order instead of descending,
which is the actually intended order. The effect was not very
measurable though.

New code is mostly the two helpers and an Xarray selftest. The change to
mm/huge_memory.c is LOC-neutral.

Worth noting (not part of the series): I found these useful while
preparation for supporting all kinds of folio splitting folios in the
swap cache. I also tried to get rid of lruvec lock for the splitting,
or at least decouple the lock and reduce lock scope of it. It turns out,
not cleanly doable at this moment, and the main blocker is: MGLRU.
MGLRU will read a folio's size without holding a ref, solely depending
on the LRU lock to stabilize. To improve this, another rework of MGLRU's
reclaim progress is needed (which will also cover the problem fount by
the MGLRU-FG series that MGLRU sometime have a higher contention than
CLRU on reclaim isolation).

Testing
=======

Split overhead on a 32 vCPU x86-64 KVM guest, 31 GiB RAM, 4K base
pages, 2M THP.  Split a 1 GiB shmem file (512 PMD-sized folios)
uniformly to order 0 through /sys/kernel/debug/split_huge_pages and
time that single write with bash's `time`.  Medians over 12
interleaved rounds per kernel

Test script:
   mount -t tmpfs -o size=2G,huge=always tmpfs /mnt
   fallocate -l 1G /mnt/big
   time echo /mnt/big,0x0,0x40000,0 > /sys/kernel/debug/split_huge_pages
   rm -f /mnt/big
   umount /mnt

Result:
                        Before        After
real                    17 ms        13 ms   (-24%)
sys                     17 ms        13 ms   (-24%)

End to end test showed no regression.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
Kairui Song (8):
      mm: add folio_prev()
      xarray: add xas_prev_canonical()
      mm/huge_memory: avoid an unnecessary folio size reading
      mm/huge_memory: walk the after-split file folios in reverse order
      mm/huge_memory: publish split folios by walking backwards
      mm/huge_memory: rearrange split routine handling for better optimization
      mm/huge_memory: walk the after-split anon folios in reverse order
      mm/huge_memory: walk the folios reversely for the release loop

 include/linux/mm.h     |  23 +++++++++++
 include/linux/xarray.h |  49 +++++++++++++++++++++++
 lib/test_xarray.c      |  97 +++++++++++++++++++++++++++++++++++++++++++++
 lib/xarray.c           |   8 ----
 mm/huge_memory.c       | 105 +++++++++++++++++++++++++------------------------
 5 files changed, 222 insertions(+), 60 deletions(-)
---
base-commit: fae273e48c398c56b819151a607a6c8792dadef0
change-id: 20260915-swap-thp-split-0f86b554b42e

Best regards,
--  
Kairui Song <kasong@tencent.com>

[PATCH RFC 1/8] mm: add folio_prev()

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

This is similar to folio_next(). But to get the head of the previous
folio, the compound info of the current folio isn't helpful. So here just
retrieve the previous page frame's compound info instead.

This creates a nice side effect that we don't require the current folio
to be still valid as we don't need any info from it and @folio is never
dereferenced, just pointer arithmetic.

The caller has to ensure the previous folio is legal though, as there
is no reliable way to tell from @folio whether one exists.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/mm.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 77b5d450fe45..5b529bc6624f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3076,6 +3076,29 @@ static inline struct folio *folio_next(struct folio *folio)
 	return (struct folio *)folio_page(folio, folio_nr_pages(folio));
 }
 
+/**
+ * folio_prev - Move to the previous physical folio.
+ * @folio: The folio we're currently operating on.
+ *
+ * The inverse of folio_next().  Use it to move from one folio to the
+ * previous one in physically contiguous memory which may span more than
+ * one folio.  Do not use it if the memory is only virtually contiguous.
+ * This is the folio equivalent to writing ``page--``.
+ *
+ * The caller has to ensure the previous folio is legal; there is no way
+ * to tell from @folio whether one exists.  @folio itself is never
+ * dereferenced, only the page preceding it, so it may be a sentinel
+ * pointer instead of a folio.
+ *
+ * Context: We assume that the folios are refcounted and/or locked at a
+ * higher level and do not adjust the reference counts.
+ * Return: The previous struct folio.
+ */
+static inline struct folio *folio_prev(struct folio *folio)
+{
+	return page_folio(&folio->page - 1);
+}
+
 /**
  * folio_shift - The size of the memory described by this folio.
  * @folio: The folio.

-- 
2.56.1

[PATCH RFC 2/8] xarray: add xas_prev_canonical()

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

xas_prev() moves one index at a time, so stepping backwards into a
multi-index entry can stop on a sibling slot. The new xas_prev_canonical()
steps back over an entry as a whole: it leaves the state on the slot
holding the entry and on the first index that entry covers, so
xas_get_order() also reports its order.

A cursor already inside an entry reports it instead of stepping off.

Cover it in the test suite too.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 include/linux/xarray.h | 49 +++++++++++++++++++++++++
 lib/test_xarray.c      | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/xarray.c           |  8 -----
 3 files changed, 146 insertions(+), 8 deletions(-)

diff --git a/include/linux/xarray.h b/include/linux/xarray.h
index be850174e802..b76d24e7a46b 100644
--- a/include/linux/xarray.h
+++ b/include/linux/xarray.h
@@ -1502,6 +1502,15 @@ static inline bool xas_top(struct xa_node *node)
 	return node <= XAS_RESTART;
 }
 
+/* Move the index either forwards (find) or backwards (sibling slot) */
+static inline void xas_move_index(struct xa_state *xas, unsigned long offset)
+{
+	unsigned int shift = xas->xa_node->shift;
+
+	xas->xa_index &= ~XA_CHUNK_MASK << shift;
+	xas->xa_index += offset << shift;
+}
+
 /**
  * xas_reset() - Reset an XArray operation state.
  * @xas: XArray operation state.
@@ -1883,6 +1892,46 @@ static inline void *xas_prev(struct xa_state *xas)
 	return xa_entry(xas->xa, node, xas->xa_offset);
 }
 
+/**
+ * xas_prev_canonical() - Move iterator to the previous entry, onto the slot
+ * which stores it.
+ * @xas: XArray operation state.
+ *
+ * Same as xas_prev(), but it steps back over a multi-index entry as a
+ * whole: the state is left on the slot which stores the entry and on its
+ * first index, so xas_get_order() reports its order.  A cursor already
+ * inside an entry reports that entry instead of stepping off it.
+ *
+ * Context: Any context.  Hold the xa_lock or the RCU lock across the
+ * whole walk; xas_pause() is forward-only.
+ * Return: The entry at the new position, or %NULL (empty slot, error
+ * state, or the end of the walk).  The result may be an internal entry,
+ * which needs xas_retry().
+ */
+static inline void *xas_prev_canonical(struct xa_state *xas)
+{
+	void *entry = NULL;
+
+	/* step back unless @xas is already inside an entry */
+	if (!xas_is_node(xas) ||
+	    xas->xa_offset >= XA_CHUNK_SIZE ||
+	    xas->xa_offset == ((xas->xa_index >> xas->xa_node->shift) &
+			       XA_CHUNK_MASK))
+		entry = xas_prev(xas);
+	if (!xas_is_node(xas))
+		return entry;
+	if (xa_is_sibling(entry))
+		xas->xa_offset = xa_to_sibling(entry);
+
+	entry = xa_entry(xas->xa, xas->xa_node, xas->xa_offset);
+	/* a concurrent store or split can leave a sibling or a node here */
+	if (xa_is_sibling(entry) ||
+	    (xas->xa_node->shift && xa_is_node(entry)))
+		return XA_RETRY_ENTRY;
+	xas_move_index(xas, xas->xa_offset);
+	return entry;
+}
+
 /**
  * xas_next() - Move state to next index.
  * @xas: XArray operation state.
diff --git a/lib/test_xarray.c b/lib/test_xarray.c
index 615f7730a5bd..ad4507982b5b 100644
--- a/lib/test_xarray.c
+++ b/lib/test_xarray.c
@@ -2011,6 +2011,101 @@ static noinline void check_split(struct xarray *xa)
 static void check_split(struct xarray *xa) { }
 #endif
 
+/* Only the head holds an entry, and there is no node to walk */
+static noinline void check_prev_canonical_no_walk(struct xarray *xa)
+{
+	XA_STATE(xas, xa, 0);
+
+	xa_store(xa, 0, xa_mk_value(0x123), GFP_KERNEL);
+	xas_lock(&xas);
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != xa_mk_value(0x123));
+	XA_BUG_ON(xa, xas.xa_index != 0);
+	xas_unlock(&xas);
+	xa_destroy(xa);
+}
+
+static noinline void check_prev_canonical(struct xarray *xa)
+{
+	XA_STATE(xas, xa, 0);
+	void *entry;
+	unsigned long index;
+
+	/* Order 0 entries at [XA_CHUNK_SIZE / 2, XA_CHUNK_SIZE * 3 / 2 - 1] */
+	for (index = XA_CHUNK_SIZE / 2; index < XA_CHUNK_SIZE * 3 / 2; index++)
+		xa_store(xa, index, xa_mk_value(index), GFP_KERNEL);
+	/* A single order 0 entry at the very end */
+	xa_store(xa, XA_CHUNK_SIZE * 17 / 4, xa_mk_value(0xef), GFP_KERNEL);
+
+#ifdef CONFIG_XARRAY_MULTI
+	/* Entry A at [XA_CHUNK_SIZE * 3 / 2, XA_CHUNK_SIZE * 2 - 1] */
+	xa_store_order(xa, XA_CHUNK_SIZE * 3 / 2, XA_CHUNK_SHIFT - 1,
+		       xa_mk_value(0x123), GFP_KERNEL);
+	/* Entry B at [XA_CHUNK_SIZE * 2, XA_CHUNK_SIZE * 4 - 1] */
+	xa_store_order(xa, 2 * XA_CHUNK_SIZE, XA_CHUNK_SHIFT + 1,
+		       xa_mk_value(0x456), GFP_KERNEL);
+	/* Entry C at [XA_CHUNK_SIZE * 4, XA_CHUNK_SIZE * 17 / 4 - 1] */
+	xa_store_order(xa, 4 * XA_CHUNK_SIZE, XA_CHUNK_SHIFT - 2,
+		       xa_mk_value(0x789), GFP_KERNEL);
+#endif
+
+	xas_lock(&xas);
+#ifdef CONFIG_XARRAY_MULTI
+	/* A cursor inside the entry A reports it */
+	xas_set_order(&xas, 2 * XA_CHUNK_SIZE - 1, 0);
+	xas_load(&xas);
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != xa_mk_value(0x123));
+	XA_BUG_ON(xa, xas.xa_index != XA_CHUNK_SIZE * 3 / 2);
+	XA_BUG_ON(xa, xas_get_order(&xas) != XA_CHUNK_SHIFT - 1);
+	/* Replacing it to 0xABC: an order 0 store keeps its siblings */
+	XA_BUG_ON(xa, xas_store(&xas, xa_mk_value(0xABC)) !=
+		       xa_mk_value(0x123));
+#endif
+
+	/* A bounded walk over the last leaf ends past its last slot */
+	xas_set(&xas, 4 * XA_CHUNK_SIZE);
+	xas_for_each(&xas, entry, 5 * XA_CHUNK_SIZE - 1) { }
+	XA_BUG_ON(xa, xas.xa_offset != XA_CHUNK_SIZE);
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != NULL);
+	XA_BUG_ON(xa, xas.xa_index != 5 * XA_CHUNK_SIZE - 1);
+
+	/* Walk, and every step lands one entry back */
+	xas_set_order(&xas, XA_CHUNK_SIZE * 17 / 4, 0);
+
+	/* Read back the ending order 0 entry */
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != xa_mk_value(0xef));
+	XA_BUG_ON(xa, xas.xa_index != XA_CHUNK_SIZE * 17 / 4);
+	XA_BUG_ON(xa, xas_get_order(&xas) != 0);
+
+#ifdef CONFIG_XARRAY_MULTI
+	/* Read entry C */
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != xa_mk_value(0x789));
+	XA_BUG_ON(xa, xas.xa_index != 4 * XA_CHUNK_SIZE);
+	XA_BUG_ON(xa, xas_get_order(&xas) != XA_CHUNK_SHIFT - 2);
+	/* Read entry B */
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != xa_mk_value(0x456));
+	XA_BUG_ON(xa, xas.xa_index != 2 * XA_CHUNK_SIZE);
+	XA_BUG_ON(xa, xas_get_order(&xas) != XA_CHUNK_SHIFT + 1);
+	/* Read entry A, holding the value stored above */
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != xa_mk_value(0xABC));
+	XA_BUG_ON(xa, xas.xa_index != XA_CHUNK_SIZE * 3 / 2);
+	XA_BUG_ON(xa, xas_get_order(&xas) != XA_CHUNK_SHIFT - 1);
+#endif
+
+	/* The order 0 entries, in the leaf below entry A */
+	xas_set_order(&xas, XA_CHUNK_SIZE * 3 / 2 - 1, 0);
+	for (index = XA_CHUNK_SIZE * 3 / 2; index > XA_CHUNK_SIZE / 2; index--) {
+		XA_BUG_ON(xa, xas_prev_canonical(&xas) !=
+			       xa_mk_value(index - 1));
+		XA_BUG_ON(xa, xas_store(&xas, xa_mk_value(2)) !=
+			       xa_mk_value(index - 1));
+	}
+
+	/* The hole ends the walk */
+	XA_BUG_ON(xa, xas_prev_canonical(&xas) != NULL);
+	xas_unlock(&xas);
+	xa_destroy(xa);
+}
+
 static void check_align_1(struct xarray *xa, char *name)
 {
 	int i;
@@ -2328,6 +2423,8 @@ static int xarray_checks(void)
 	check_store_iter(&array);
 	check_align(&xa0);
 	check_split(&array);
+	check_prev_canonical_no_walk(&array);
+	check_prev_canonical(&array);
 
 	check_workingset(&array, 0);
 	check_workingset(&array, 64);
diff --git a/lib/xarray.c b/lib/xarray.c
index 48f440d114e1..1c295b9404dc 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -152,14 +152,6 @@ static void xas_set_offset(struct xa_state *xas)
 	xas->xa_offset = get_offset(xas->xa_index, xas->xa_node);
 }
 
-/* move the index either forwards (find) or backwards (sibling slot) */
-static void xas_move_index(struct xa_state *xas, unsigned long offset)
-{
-	unsigned int shift = xas->xa_node->shift;
-	xas->xa_index &= ~XA_CHUNK_MASK << shift;
-	xas->xa_index += offset << shift;
-}
-
 static void xas_next_offset(struct xa_state *xas)
 {
 	xas->xa_offset++;

-- 
2.56.1

[PATCH RFC 3/8] mm/huge_memory: avoid an unnecessary folio size reading

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

We already have that in the local variable, a trivial cleanup.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/huge_memory.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 0d589be136d2..b60fd73fe35c 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4287,9 +4287,7 @@ static int __folio_freeze_split_file(struct folio *folio,
 		/* compute next before the folio can be freed below */
 		next = folio_next(new_folio);
 
-		folio_ref_unfreeze(new_folio,
-				   folio_nr_pages(new_folio) + 1);
-
+		folio_ref_unfreeze(new_folio, nr_pages + 1);
 		lru_add_split_folio(folio, new_folio, lruvec, list);
 
 		/* Add the new folio to the page cache. */

-- 
2.56.1

[PATCH RFC 4/8] mm/huge_memory: walk the after-split file folios in reverse order

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Use the new folio_prev() to walk the after-split range backwards. The
@folio is still unfrozen after the loop, so it remains the last folio to
be unfrozen and stays frozen until every page cache entry that used
to point at it has been repointed to an after-split folio.

This improves the code in two ways: A local variable is saved, and this
is also the order the LRU wants.  lru_add_split_folio() inserts each
sub-folio with list_add_tail(), just before @folio on the LRU, or at
the tail of @list. The LRU is drained from the tail, so with the
reverse walk, @folio goes first, then next to it is the
first sub folio, and the pieces are reclaimed in ascending order, where
the forward walk left them descending which is not the correct order.

And even if the folio is isolated, almost all split callers drain the
list from the tail (lru_to_folio() is head->prev). With s1
.. s4 the four sub-folios in ascending address order and @folio
holding the first pages of the range, we have:

  For a folio on the LRU (list == NULL), reading head to tail:

  Before (forward): [ .. s1 s2 s3 s4 folio ]  reclaimed: folio s4 s3 s2 s1
  After  (reverse): [ .. s4 s3 s2 s1 folio ]  reclaimed: folio s1 s2 s3 s4

  For a folio the caller isolated off the LRU, reading head to tail:

  Before (forward): [ .. s1 s2 s3 s4 ]        consumed: s4 s3 s2 s1
  After  (reverse): [ .. s4 s3 s2 s1 ]        consumed: s1 s2 s3 s4

Migration is the one @list user that walks its list from the head; that
list is only a work queue and the order carries no meaning there. We may
unify that part too if needed.

The effect is not very measurable in real workloads but we can get it
right while reducing stack usage, so let's do that.

This also helps avoid a for loop pitfall mentioned by Zi Yan [1]:
folio_next() needs the folio it is given to stay alive until the
increment, folio_prev() does not.  So the cached next pointer and its
comment can go.

The anon helper still walks forward, and will be converted separately.

Link: https://lore.kernel.org/r/DKJSFCLP967N.YBR4DNK1NM2N@nvidia.com [1]
Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/huge_memory.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index b60fd73fe35c..3d3a82a03149 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4187,7 +4187,7 @@ static int __folio_freeze_split_file(struct folio *folio,
 	XA_STATE(xas, &mapping->i_pages, folio->index);
 	struct folio *end_folio = folio_next(folio);
 	struct mem_cgroup *memcg, *old_memcg;
-	struct folio *new_folio, *next;
+	struct folio *new_folio;
 	int nr_shmem_dropped = 0;
 	unsigned int min_order;
 	struct lruvec *lruvec;
@@ -4273,20 +4273,19 @@ static int __folio_freeze_split_file(struct folio *folio,
 	ret = __split_frozen_folio(folio, new_order, split_at, &xas, split_type);
 
 	/*
-	 * Unfreeze after-split folios and put them back to the right
-	 * list. @folio should be kept frozen until page cache
-	 * entries are updated with all the other after-split folios
-	 * to prevent others seeing stale page cache entries.
-	 * As a result, new_folio starts from the next folio of
-	 * @folio.
+	 * Unfreeze the after-split folios and put them back on the right
+	 * list, walking the range in reverse so @folio is unfrozen last.
+	 * It stays frozen until then, so a parallel lookup cannot
+	 * folio_try_get() @folio and see a stale page cache entry.
+	 *
+	 * Reverse is also the reclaim order: sub-folios are inserted right
+	 * before @folio, or at the end of @list, and the LRU is drained
+	 * from the tail, so the last one added goes first.
 	 */
-	for (new_folio = folio_next(folio); new_folio != end_folio;
-	     new_folio = next) {
+	for (new_folio = folio_prev(end_folio); new_folio != folio;
+	     new_folio = folio_prev(new_folio)) {
 		unsigned long nr_pages = folio_nr_pages(new_folio);
 
-		/* compute next before the folio can be freed below */
-		next = folio_next(new_folio);
-
 		folio_ref_unfreeze(new_folio, nr_pages + 1);
 		lru_add_split_folio(folio, new_folio, lruvec, list);
 

-- 
2.56.1

[PATCH RFC 5/8] mm/huge_memory: publish split folios by walking backwards

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

__xa_store() builds a fresh XA_STATE per after-split folio, so a uniform
order-9 to order-0 split does 511 root walks, which is slow and on the
hot path for file folio splitting.

So instead, just use one XA_STATE iterator, and step back folio by folio
using the new xas_prev_canonical() helper: each iteration is much
cheaper, and publishing costs only one full descent walk.

Also simplify the code by unifying the post-processing of past EOF
folios.  We may store a few more folios than necessary, but the store is
much cheaper now, the truncate path is unlikely, and it makes the folio
removal convention in a mapping more consistent: the folio being removed
always matches the content in the mapping now.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/huge_memory.c | 35 +++++++++++++++++++----------------
 1 file changed, 19 insertions(+), 16 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 3d3a82a03149..7270baa20d80 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4187,7 +4187,7 @@ static int __folio_freeze_split_file(struct folio *folio,
 	XA_STATE(xas, &mapping->i_pages, folio->index);
 	struct folio *end_folio = folio_next(folio);
 	struct mem_cgroup *memcg, *old_memcg;
-	struct folio *new_folio;
+	struct folio *new_folio, *old_folio;
 	int nr_shmem_dropped = 0;
 	unsigned int min_order;
 	struct lruvec *lruvec;
@@ -4282,28 +4282,31 @@ static int __folio_freeze_split_file(struct folio *folio,
 	 * before @folio, or at the end of @list, and the LRU is drained
 	 * from the tail, so the last one added goes first.
 	 */
+	xas_set_order(&xas, folio->index + old_nr_pages - 1, 0);
+	old_folio = xas_prev_canonical(&xas);
 	for (new_folio = folio_prev(end_folio); new_folio != folio;
 	     new_folio = folio_prev(new_folio)) {
 		unsigned long nr_pages = folio_nr_pages(new_folio);
 
 		folio_ref_unfreeze(new_folio, nr_pages + 1);
 		lru_add_split_folio(folio, new_folio, lruvec, list);
-
-		/* Add the new folio to the page cache. */
-		if (new_folio->index < end) {
-			__xa_store(&mapping->i_pages, new_folio->index,
-				   new_folio, 0);
-			continue;
+		/* Mapping is split, but content is still old folio */
+		WARN_ON_ONCE(xas_get_order(&xas) != folio_order(new_folio));
+		WARN_ON_ONCE(old_folio != folio);
+		/* order 0 store: siblings keep resolving to this slot */
+		xas_store(&xas, new_folio);
+		/* Step off now as the unlink below can free the xa_node */
+		old_folio = xas_prev_canonical(&xas);
+		if (new_folio->index >= end) {
+			/* Drop folio beyond EOF: ->index >= end */
+			if (shmem_mapping(mapping))
+				nr_shmem_dropped += nr_pages;
+			else if (folio_test_clear_dirty(new_folio))
+				folio_account_cleaned(new_folio,
+						      inode_to_wb(mapping->host));
+			__filemap_remove_folio(new_folio, NULL);
+			folio_put_refs(new_folio, nr_pages);
 		}
-
-		/* Drop folio beyond EOF: ->index >= end */
-		if (shmem_mapping(mapping))
-			nr_shmem_dropped += nr_pages;
-		else if (folio_test_clear_dirty(new_folio))
-			folio_account_cleaned(
-				new_folio, inode_to_wb(mapping->host));
-		__filemap_remove_folio(new_folio, NULL);
-		folio_put_refs(new_folio, nr_pages);
 	}
 
 	/*

-- 
2.56.1

[PATCH RFC 6/8] mm/huge_memory: rearrange split routine handling for better optimization

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Anonymous folios never use a mapping or an xarray, so combine the is_anon
and xas conditions into a single if/else-if chain.  The compiler cannot do
this on its own because the xarray split calls sit between the two
conditions and the anon mTHP stat update comes after the split helpers.

./scripts/bloat-o-meter on mm/huge_memory.o (x86-64 KVM test config) shows
__split_frozen_folio() shrinking from 1077 to 996 bytes.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/huge_memory.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7270baa20d80..e16b93318613 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3929,13 +3929,17 @@ static int __split_frozen_folio(struct folio *folio, int new_order,
 	for (split_order = start_order;
 	     split_order >= new_order;
 	     split_order--) {
-		int nr_new_folios = 1UL << (old_order - split_order);
+		if (is_anon) {
+			int nr_new_folios = 1UL << (old_order - split_order);
 
-		/* order-1 anonymous folio is not supported */
-		if (is_anon && split_order == 1)
-			continue;
+			/* order-1 anonymous folio is not supported */
+			if (split_order == 1)
+				continue;
 
-		if (xas) {
+			/* Split can't fail at this point, update counters now */
+			mod_mthp_stat(old_order, MTHP_STAT_NR_ANON, -1);
+			mod_mthp_stat(split_order, MTHP_STAT_NR_ANON, nr_new_folios);
+		} else if (xas) {
 			/*
 			 * uniform split has xas_split_alloc() called before
 			 * irq is disabled to allocate enough memory, whereas
@@ -3960,10 +3964,6 @@ static int __split_frozen_folio(struct folio *folio, int new_order,
 		pgalloc_tag_split(folio, old_order, split_order);
 		__split_folio_to_order(folio, old_order, split_order);
 
-		if (is_anon) {
-			mod_mthp_stat(old_order, MTHP_STAT_NR_ANON, -1);
-			mod_mthp_stat(split_order, MTHP_STAT_NR_ANON, nr_new_folios);
-		}
 		/*
 		 * If uniform split, the process is complete.
 		 * If non-uniform, continue splitting the folio at @split_at

-- 
2.56.1

[PATCH RFC 7/8] mm/huge_memory: walk the after-split anon folios in reverse order

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

Just like the file folio, walk the after-split range with folio_prev()
so the pieces are queued in ascending reclaim order where the forward
walk left them descending, and the cached "next" goes away. All of its
users either expects an ascending or do not care.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/huge_memory.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index e16b93318613..2ff201a5e5f3 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4076,7 +4076,7 @@ static int __folio_freeze_split_anon(struct folio *folio,
 	struct folio *end_folio = folio_next(folio);
 	struct swap_cluster_info *ci = NULL;
 	const int old_order = folio_order(folio);
-	struct folio *new_folio, *next;
+	struct folio *new_folio;
 	struct anon_vma *anon_vma = NULL;
 	enum ttu_flags ttu_flags = 0;
 	struct lruvec *lruvec;
@@ -4137,15 +4137,15 @@ static int __folio_freeze_split_anon(struct folio *folio,
 	ret = __split_frozen_folio(folio, new_order, split_at, NULL, split_type);
 
 	/*
-	 * Unfreeze the after-split folios and put them back to the right
-	 * place. Keep the head @folio frozen until the end: sub entries
-	 * in swap cache must be updated first, so a concurrent
-	 * swap_cache_get_folio() cannot return the head folio for a sub
-	 * entry (folio_try_get() will fail on the head @folio until unfreeze).
+	 * Unfreeze the after-split folios and put them back, keeping @folio
+	 * frozen until every sub entry in the swap cache is repointed, so
+	 * that a racing swap_cache_get_folio() cannot folio_try_get() the
+	 * head.  Backward is also the reclaim order: the pieces queue just
+	 * before @folio on the LRU, or at the tail of @list, both drained
+	 * from the tail.
 	 */
-	for (new_folio = folio_next(folio); new_folio != end_folio;
-	     new_folio = next) {
-		next = folio_next(new_folio);
+	for (new_folio = folio_prev(end_folio); new_folio != folio;
+	     new_folio = folio_prev(new_folio)) {
 		zone_device_private_split_cb(folio, new_folio);
 		folio_ref_unfreeze(new_folio,
 				   folio_swapcache_ref_count(new_folio) + 1);

-- 
2.56.1

[PATCH RFC 8/8] mm/huge_memory: walk the folios reversely for the release loop

[Kairui Song via B4 Relay]

From: Kairui Song <kasong@tencent.com>

The release loop unlocks and puts every after-split folio except the one
holding @lock_at.  Walk it backwards with folio_prev() instead of
forwards with folio_next().

This also helps avoid a for loop pitfall mentioned by Zi Yan [1]:
folio_next() needs the folio it is given to stay alive until the
increment, folio_prev() does not.  So the cached next pointer can go, and
the folio that was just put is never read again.

Same folios visited, same work on each, only the direction changes.

Link: https://lore.kernel.org/r/DKJSFCLP967N.YBR4DNK1NM2N@nvidia.com [1]
Signed-off-by: Kairui Song <kasong@tencent.com>
---
 mm/huge_memory.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 2ff201a5e5f3..bb09465a3948 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4366,7 +4366,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
 	const bool is_anon = folio_test_anon(folio);
 	const bool is_swapcache = folio_test_swapcache(folio);
 	int old_order = folio_order(folio);
-	struct folio *new_folio, *next;
+	struct folio *new_folio, *lock_folio;
 	int ret;
 
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
@@ -4399,9 +4399,10 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
 	 * Unlock all after-split folios except the one containing
 	 * @lock_at page. If @folio is not split, it will be kept locked.
 	 */
-	for (new_folio = folio; new_folio != end_folio; new_folio = next) {
-		next = folio_next(new_folio);
-		if (new_folio == page_folio(lock_at))
+	lock_folio = page_folio(lock_at);
+	for (new_folio = folio_prev(end_folio); new_folio != folio;
+	     new_folio = folio_prev(new_folio)) {
+		if (new_folio == lock_folio)
 			continue;
 
 		/*

-- 
2.56.1

Re: [PATCH RFC 0/8] mm/huge_memory: clean up and speed up large folio splitting

[David Hildenbrand (Arm)]

On 11/13/26 22:26, Kairui Song via B4 Relay wrote:
> Hi all,
> 
> This series slghtly cleans up the large folio split path and reduces
> its overhead: uniformly splitting PMD sized file folios to order 0
> is over 20% faster.

I expect (well, and I have) some fixes in that area, so we better hold this off
for the next week.

-- 
Cheers,

David

Re: [PATCH RFC 0/8] mm/huge_memory: clean up and speed up large folio splitting

[Kairui Song]

On Sat, Nov 14, 2026 at 5:41 AM David Hildenbrand (Arm)
<david@kernel.org> wrote:
>
> On 11/13/26 22:26, Kairui Song via B4 Relay wrote:
> > Hi all,
> >
> > This series slghtly cleans up the large folio split path and reduces
> > its overhead: uniformly splitting PMD sized file folios to order 0
> > is over 20% faster.
>
> I expect (well, and I have) some fixes in that area, so we better hold this off
> for the next week.

Yeah, that is totally understandable. I mainly wanted to gather
feedback on the two xarray / folio helpers at the moment, in case
anyone has concerns about that approach. I can also help review or
test the upcoming fixes.