[RFC PATCH 0/5] mm/page_alloc: keep non-movable pages out of movable pageblocks for THP

[RFC PATCH 0/5] mm/page_alloc: keep non-movable pages out of movable pageblocks for THP

[Rik van Riel]

Last week I posted the first prototype of the reworked gigablock
page allocation code, which uses the per-migrate_type free lists
in the zone as its fast path:

https://lore.kernel.org/lkml/20261004013657.03a63c4c@fangorn/

Zi Yan took a look, and pointed out that the first patches in
the series might help with THP allocation. Testing showed he
is right.

Having this code be useful by itself changes the gigablock
merge plan. It can now be done in stages:
1) Pageblock claiming fixes & improvements, which show a
   modest improvement in THP allocation. This is the only
   code in the fast path.
2) Page compaction, evacuation, and free list threshold changes,
   which can really help concentrate non-movable allocations in
   non-movable typed pageblocks, hopefully making compaction and
   THP allocation noticeably more reliable.
3) The actual gigablock targeting. This is also a slow path,
   like compaction and evacuation.

This first series addresses pageblock claiming and stealing
policies. Current page allocation policies cause the THP allocation
success rate to collapse under sustained non-movable demand.

A single non-movable page in a movable-type page block prevents
that page block from being used for a THP, but it does not
prevent kcompactd from spending time on that block.

Three paths mix unmovable pages into movable-typed blocks.
try_to_claim_block() converts a movable block for a non-movable
allocation only when half its pages are free or compatible.
__rmqueue_steal() takes pages without converting at all, and a
bulk batch that steals once never reaches the claim path again.

compaction_capture() hands a merged movable block to a
non-movable request with no type change.

Patches 1-2 are standalone accounting fixes: guard pages return
to the counts when their buddy merges, and a merged buddy's
pages move to the merged type in the free page accounting.

Patches 3-5 make the pageblock migrate type follow the content:
convert any movable block on a non-movable claim, skip movable
blocks on non-movable steals, and claim movable blocks fully taken
by non-movable compaction capture.

Pageblocks that straddle zones cannot safely change type,
since the allocator may have locked the "wrong" zone. Those
blocks are left alone.

The series is preventive: it keeps new placements to their own
type and stops polluting movable block. The next step, for
another series, will be to evacuate movable content from
pageblocks stolen by non-movable allocations, in order to
concentrate kernel allocations in fewer pageblocks.

Measured on 64GB 8-vCPU KVM guests, THP=always, defrag=always,
khugepaged collapse off, 8GB virtio swap. The guest boots into
a fragmented base: a 10GB mlocked block flood, 36GB of aged
anonymous memory, and background unmovable allocations.

Then 8 rounds of type-pressure buildup. Each round drains
high-order free memory with a large NOHUGEPAGE touch, adds 550
pagetable-heavy processes plus 200000 slab files as unmovable
demand, and ends with a fixed 2GB 8-thread THP probe that stays
mapped while success is measured. The bursts persist across
rounds while the drain shrinks by ~2.75GB each round, so each
round's demand faces less free memory.

Measured is the thp_fault_alloc success rate, across
4 runs of 8 rounds each.

                                   patched        base
rounds with <50% success            7/31         16/31
THP alloc success rate @r4       74% (47-98)  42% (25-67)
blocks with unmovable pages @r8     43-154     1674-2131

Two rounds died due to swap exhaustion, and are not counted.

Pageblocks with unmovable pages were counted by checking
/proc/kpageflags after each round. The total amount of
unmovable pages is similar with and without the series,
they just get packed into fewer pageblocks.

Converting taken blocks and refusing fallback steals packs
unmovable pages into converted blocks, leaving clean movable
victims for the scanners: a round-2 probe on the series
kernel compacts 690 of 697 attempts at 92.6% THP where the
base compacts 47 of 376 at 4.7%.

This series is broken out of the 1GB gigablock prototype work:

 include/linux/mmzone.h |  13 +++++
 mm/page_alloc.c        | 117 +++++++++++++++++++++++++++++++------------------
 2 files changed, 88 insertions(+), 42 deletions(-)

base-commit: 67f0943b394d9

[RFC PATCH 1/5] mm/page_alloc: count guard pages as free again when their buddy merges

[Rik van Riel]

With debug_guardpage_minorder set, expand() turns the unused halves of a
split into guard pages, which page_del_and_expand() leaves out of the free
counts. Freeing the allocated half credits only that half, and
__free_one_page() then merges the guard buddy with clear_page_guard(),
which no longer touches the counters.

The merged block goes on a free list with the guard pages never counted,
so NR_FREE_PAGES falls behind the free lists by every guard that merges.
Before commit e0932b6c1f94 ("mm: page_alloc: consolidate free page
accounting"), __set_page_guard() and __clear_page_guard() adjusted the
counts themselves; that commit dropped both adjustments but kept the
subtraction implicit in expand().

Count the guard pages under the merged block's migratetype when the
buddy is cleared. account_freepages() skips an isolated block as it does
for every other free page, and moving the block off the isolated list
counts it then.

Booting a 16GB VM with debug_pagealloc=on debug_guardpage_minorder=1,
check the difference in free pages reported between /proc/vmstat
and /proc/buddyinfo.

At three points (idle, after a read and file-creation load, after
a second load), check the difference between nr_free_pages and the
buddyinfo numbers in the normal zone, as a number of pages:

                idle   after load   after second load
    unpatched   -130        79205               99121
    patched      -16           45                   0

The small remaining difference seems to be due to the numbers
not being read at exactly the same time, and is also seen
without guard pages.

Fixes: e0932b6c1f94 ("mm: page_alloc: consolidate free page accounting")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/page_alloc.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c483..4658af97be010 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -988,10 +988,12 @@ static inline void __free_one_page(struct page *page,
 		 * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page,
 		 * merge with it and move up one order.
 		 */
-		if (page_is_guard(buddy))
+		if (page_is_guard(buddy)) {
 			clear_page_guard(zone, buddy, order);
-		else
+			account_freepages(zone, 1 << order, migratetype);
+		} else {
 			__del_page_from_free_list(buddy, zone, order, buddy_mt);
+		}
 
 		if (unlikely(buddy_mt != migratetype)) {
 			/*
-- 
2.53.0-Meta

[RFC PATCH 2/5] mm/page_alloc: count a merged buddy's pages under the merged type

[Rik van Riel]

__free_one_page() merges a freed page with a free buddy of another
mergeable type at pageblock order and above, and retypes the buddy to
match. The buddy's pages stay counted under the old type: nothing moves
them to the merged type in the free counts.

That is unobservable today. NR_FREE_PAGES does not separate types, so the
missing move nets to zero there.  Any future per-type free count would
undercount on a merge from the other side, and allocating the merged
run later could wrap it.

Move the buddy's pages to the merged type in the count too.

Assisted-by: LLM
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/page_alloc.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4658af97be010..ae38e25971559 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -993,6 +993,11 @@ static inline void __free_one_page(struct page *page,
 			account_freepages(zone, 1 << order, migratetype);
 		} else {
 			__del_page_from_free_list(buddy, zone, order, buddy_mt);
+			/* The buddy's free pages join the merged block's type. */
+			if (unlikely(buddy_mt != migratetype)) {
+				account_freepages(zone, -(1 << order), buddy_mt);
+				account_freepages(zone, 1 << order, migratetype);
+			}
 		}
 
 		if (unlikely(buddy_mt != migratetype)) {
-- 
2.53.0-Meta

[RFC PATCH 3/5] mm/page_alloc: convert any movable pageblock on a non-movable allocation

[Rik van Riel]

try_to_claim_block() converts a movable pageblock for a non-movable
allocation only when half its pages are free or compatible.
Otherwise the steal path takes pages without converting, and
non-movable content sits in a block still typed movable.

Convert a movable block on any non-movable claim, however few of
its pages are free, so the block's type matches the allocation it
serves.

When other pages in the block are freed later, they end up on the
non-movable free lists, directing more non-movable allocations to
the already non-movable page blocks.

Converting a non-movable pageblock to movable is made stricter:
a movable claim takes a non-movable block only when every page
is free or movable.

Blocks straddling a zone edge are never claimed;
zone_spans_pageblock() tests for a block wholly inside its zone.

Assisted-by: LLM
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 include/linux/mmzone.h | 13 ++++++++
 mm/page_alloc.c        | 71 ++++++++++++++++++++----------------------
 2 files changed, 46 insertions(+), 38 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff54160..81ad83ff69073 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1227,6 +1227,19 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn)
 	return zone->zone_start_pfn <= pfn && pfn < zone_end_pfn(zone);
 }
 
+/*
+ * Whether the pageblock holding @pfn lies wholly inside @zone. Zone
+ * spans are not pageblock-aligned, so the edge pageblocks of a zone
+ * can straddle into the next zone; those never change type.
+ */
+static inline bool zone_spans_pageblock(const struct zone *zone, unsigned long pfn)
+{
+	unsigned long start = pageblock_start_pfn(pfn);
+
+	return zone->zone_start_pfn <= start &&
+	       start + pageblock_nr_pages <= zone_end_pfn(zone);
+}
+
 static inline bool zone_is_initialized(const struct zone *zone)
 {
 	return zone->initialized;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index ae38e25971559..9f1520506ae1d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2295,18 +2295,39 @@ find_suitable_fallback(struct free_area *area, unsigned int order,
 }
 
 /*
- * This function implements actual block claiming behaviour. If order is large
- * enough, we can claim the whole pageblock for the requested migratetype. If
- * not, we check the pageblock for constituent pages; if at least half of the
- * pages are free or compatible, we can still claim the whole block, so pages
- * freed in the future will be put on the correct free list.
+ * Whether to retype a partly used block. A block wholly inside its zone
+ * takes the allocation's type, so no non-movable page sits in a
+ * movable-typed block; a straddling block is never retyped.
  */
+static bool should_claim_used_block(struct zone *zone, unsigned long start_pfn,
+				    int start_type, int block_type,
+				    int free_pages, int movable_pages)
+{
+	if (!zone_spans_pageblock(zone, start_pfn))
+		return false;
+
+	if (page_group_by_mobility_disabled)
+		return true;
+
+	/* Convert to movable only if no non-movable pages are present. */
+	if (is_migrate_movable(start_type))
+		return free_pages + movable_pages == pageblock_nr_pages;
+
+	/* Non-movable pages break compaction; claim as non-movable */
+	if (is_migrate_movable(block_type))
+		return true;
+
+	/* Unmovable or reclaimable claiming from the other. */
+	return free_pages >= (1 << (pageblock_order - 1));
+}
+
+/* Convert the block so later frees use the allocation's migratetype. */
 static struct page *
 try_to_claim_block(struct zone *zone, struct page *page,
 		   int current_order, int order, int start_type,
 		   int block_type, unsigned int alloc_flags)
 {
-	int free_pages, movable_pages, alike_pages;
+	int free_pages, movable_pages;
 	unsigned long start_pfn;
 
 	/* Take ownership for orders >= pageblock_order */
@@ -2333,39 +2354,13 @@ try_to_claim_block(struct zone *zone, struct page *page,
 				       &movable_pages))
 		return NULL;
 
-	/*
-	 * Determine how many pages are compatible with our allocation.
-	 * For movable allocation, it's the number of movable pages which
-	 * we just obtained. For other types it's a bit more tricky.
-	 */
-	if (start_type == MIGRATE_MOVABLE) {
-		alike_pages = movable_pages;
-	} else {
-		/*
-		 * If we are falling back a RECLAIMABLE or UNMOVABLE allocation
-		 * to MOVABLE pageblock, consider all non-movable pages as
-		 * compatible. If it's UNMOVABLE falling back to RECLAIMABLE or
-		 * vice versa, be conservative since we can't distinguish the
-		 * exact migratetype of non-movable pages.
-		 */
-		if (block_type == MIGRATE_MOVABLE)
-			alike_pages = pageblock_nr_pages
-						- (free_pages + movable_pages);
-		else
-			alike_pages = 0;
-	}
-	/*
-	 * If a sufficient number of pages in the block are either free or of
-	 * compatible migratability as our allocation, claim the whole block.
-	 */
-	if (free_pages + alike_pages >= (1 << (pageblock_order-1)) ||
-			page_group_by_mobility_disabled) {
-		__move_freepages_block(zone, start_pfn, block_type, start_type);
-		set_pageblock_migratetype(pfn_to_page(start_pfn), start_type);
-		return __rmqueue_smallest(zone, order, start_type);
-	}
+	if (!should_claim_used_block(zone, start_pfn, start_type, block_type,
+				     free_pages, movable_pages))
+		return NULL;
 
-	return NULL;
+	__move_freepages_block(zone, start_pfn, block_type, start_type);
+	set_pageblock_migratetype(pfn_to_page(start_pfn), start_type);
+	return __rmqueue_smallest(zone, order, start_type);
 }
 
 /*
-- 
2.53.0-Meta

[RFC PATCH 4/5] mm/page_alloc: skip movable blocks during non-movable steals

[Rik van Riel]

__rmqueue_steal() hands a non-movable allocation pages from a
movable pageblock and leaves the block's type alone. Those pages
can never migrate, so the block reads movable to compaction while
holding content compaction cannot move. On a 3 GB guest running
ls -lR / and a 2.4 GB block device read against a fragmenter, a
temporary counter found 2654, 5341 and 5272 such allocations in
three boots.

__rmqueue_claim() runs first and would convert the block, but
rmqueue_bulk() remembers the mode across its batch. Prevent
those allocations from placing non-movable pages inside
movable pageblocks by refusing __rmqueue_steal() for non-movable
allocations in movable pageblocks.

Movable is last in fallbacks[] for both non-movable types, so
only a larger order is left to try; failing that, the steal
returns NULL and the allocation will loop around to a claim,
another zone, or reclaim.

Keep the steal where mobility grouping is disabled, or where
the block straddles a zone edge. A block that straddles zones
cannot be claimed, because the allocator only holds the lock
for one zone, which leaves stealing as the alloc path there.

Movable allocations can steal, because kcompactd can always
move those pages out of non-movable blocks later.

Over three boots allocstall_normal runs 48 to 129 against 32 to
134 on the base, and compact_stall 324 to 415 against 288 to 387.
Neither direct reclaim nor compaction stalls rise beyond
run-to-run spread.

Assisted-by: LLM
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/page_alloc.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9f1520506ae1d..16d3594d913a0 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2423,8 +2423,10 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype,
 }
 
 /*
- * Try to steal a single page from some fallback migratetype. Leave the rest of
- * the block as its current migratetype, potentially causing fragmentation.
+ * Try to steal one page from a fallback type, leaving the rest of the block
+ * unchanged and possibly fragmented. A non-movable allocation must not steal
+ * from a movable block: without converting its type, the steal would leave
+ * non-movable content under a movable type.
  */
 static __always_inline struct page *
 __rmqueue_steal(struct zone *zone, int order, int start_migratetype)
@@ -2444,6 +2446,19 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype)
 			continue;
 
 		page = get_page_from_free_area(area, fallback_mt);
+
+		/*
+		 * Do not allow non-movable allocations in movable
+		 * pageblocks; that could break compaction.
+		 * Non-movable allocations should claim pageblocks, instead.
+		 */
+		if (!is_migrate_movable(start_migratetype) &&
+		    is_migrate_movable(fallback_mt) &&
+		    !page_group_by_mobility_disabled &&
+		    zone_spans_pageblock(zone, page_to_pfn(page))) {
+			continue;
+		}
+
 		page_del_and_expand(zone, page, order, current_order, fallback_mt);
 		trace_mm_page_alloc_extfrag(page, order, current_order,
 					    start_migratetype, fallback_mt);
-- 
2.53.0-Meta

[RFC PATCH 5/5] mm/page_alloc: claim blocks captured for non-movable use

[Rik van Riel]

compaction_capture() hands any merged block to the compacting task
with no type change, except it refuses a movable block to a
non-movable request below pageblock order. A non-movable request
capturing a whole movable block leaves non-movable content in a
movable-typed block.

Make a non-movable capture claim a movable block: convert it to
the request. A partial block cannot be retyped, so the refusal
below pageblock order stays; reaching the conversion means a
whole block. A movable capture steals: it takes the pages and
leaves the type alone.

Assisted-by: LLM
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/page_alloc.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 16d3594d913a0..edb69d8a74c73 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -728,6 +728,9 @@ static inline struct capture_control *task_capc(struct zone *zone)
 		capc->zone == zone ? capc : NULL;
 }
 
+static void change_pageblock_range(struct page *pageblock_page,
+				   int start_order, int migratetype);
+
 static inline bool
 compaction_capture(struct capture_control *capc, struct page *page,
 		   int order, int migratetype)
@@ -751,6 +754,19 @@ compaction_capture(struct capture_control *capc, struct page *page,
 	    capc->migratetype != MIGRATE_MOVABLE)
 		return false;
 
+	/*
+	 * A non-movable capture claims a movable block: convert it to
+	 * the request. A movable capture steals: take the pages and
+	 * leave the type alone. Partial movable blocks returned above,
+	 * so reaching here means a whole block.
+	 */
+	if (capc->migratetype != MIGRATE_MOVABLE &&
+	    migratetype == MIGRATE_MOVABLE) {
+		change_pageblock_range(page, order, capc->migratetype);
+		/* Converted whole, so no fragmentation to report below. */
+		migratetype = capc->migratetype;
+	}
+
 	if (migratetype != capc->migratetype)
 		trace_mm_page_alloc_extfrag(page, capc->order, order,
 					    capc->migratetype, migratetype);
-- 
2.53.0-Meta