[RFC PATCH 4/5] mm/page_alloc: skip movable blocks during non-movable steals

[Rik van Riel <riel@surriel.com>]

__rmqueue_steal() hands a non-movable allocation pages from a
movable pageblock and leaves the block's type alone. Those pages
can never migrate, so the block reads movable to compaction while
holding content compaction cannot move. On a 3 GB guest running
ls -lR / and a 2.4 GB block device read against a fragmenter, a
temporary counter found 2654, 5341 and 5272 such allocations in
three boots.

__rmqueue_claim() runs first and would convert the block, but
rmqueue_bulk() remembers the mode across its batch. Prevent
those allocations from placing non-movable pages inside
movable pageblocks by refusing __rmqueue_steal() for non-movable
allocations in movable pageblocks.

Movable is last in fallbacks[] for both non-movable types, so
only a larger order is left to try; failing that, the steal
returns NULL and the allocation will loop around to a claim,
another zone, or reclaim.

Keep the steal where mobility grouping is disabled, or where
the block straddles a zone edge. A block that straddles zones
cannot be claimed, because the allocator only holds the lock
for one zone, which leaves stealing as the alloc path there.

Movable allocations can steal, because kcompactd can always
move those pages out of non-movable blocks later.

Over three boots allocstall_normal runs 48 to 129 against 32 to
134 on the base, and compact_stall 324 to 415 against 288 to 387.
Neither direct reclaim nor compaction stalls rise beyond
run-to-run spread.

Assisted-by: LLM
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/page_alloc.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9f1520506ae1d..16d3594d913a0 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2423,8 +2423,10 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype,
 }
 
 /*
- * Try to steal a single page from some fallback migratetype. Leave the rest of
- * the block as its current migratetype, potentially causing fragmentation.
+ * Try to steal one page from a fallback type, leaving the rest of the block
+ * unchanged and possibly fragmented. A non-movable allocation must not steal
+ * from a movable block: without converting its type, the steal would leave
+ * non-movable content under a movable type.
  */
 static __always_inline struct page *
 __rmqueue_steal(struct zone *zone, int order, int start_migratetype)
@@ -2444,6 +2446,19 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype)
 			continue;
 
 		page = get_page_from_free_area(area, fallback_mt);
+
+		/*
+		 * Do not allow non-movable allocations in movable
+		 * pageblocks; that could break compaction.
+		 * Non-movable allocations should claim pageblocks, instead.
+		 */
+		if (!is_migrate_movable(start_migratetype) &&
+		    is_migrate_movable(fallback_mt) &&
+		    !page_group_by_mobility_disabled &&
+		    zone_spans_pageblock(zone, page_to_pfn(page))) {
+			continue;
+		}
+
 		page_del_and_expand(zone, page, order, current_order, fallback_mt);
 		trace_mm_page_alloc_extfrag(page, order, current_order,
 					    start_migratetype, fallback_mt);
-- 
2.53.0-Meta