* [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT
@ 2026-11-16 12:50 Kiryl Shutsemau (Meta)
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Kiryl Shutsemau (Meta) @ 2026-11-16 12:50 UTC (permalink / raw)
To: Andrew Morton, Vlastimil Babka, Johannes Weiner,
David Hildenbrand, Harry Yoo
Cc: Kiryl Shutsemau (Meta),
Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
Lorenzo Stoakes, Liam R. Howlett, Mike Rapoport, Qi Zheng,
Shakeel Butt, Kairui Song, Barry Song, Axel Rasmussen,
Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, Dev Jain,
Usama Arif, linux-mm, linux-kernel, kernel-team
With defrag_mode, 64K mTHP faults under memory pressure stall in
compaction 18 times as often as without it, and get about a third of the
folios. Dev Jain reported it [1].
The high-order watermark check counts free pages of every migratetype.
With defrag_mode, every slowpath allocation is ALLOC_NOFRAGMENT, and a
request below pageblock_order cannot take free pages from another
migratetype's pageblock. Under pressure, that is where the large free
pages end up. Two callers act on the wrong answer:
- direct compaction's up-front check returns COMPACT_SUCCESS without
compacting, so compaction is never deferred;
- direct reclaim for a costly order stops early, having freed little.
Patches 1-2 give the check the request's migratetype and make it skip
the free lists the request cannot take. Patch 3 tells direct reclaim
that the allocation is ALLOC_NOFRAGMENT.
Dev's benchmark in a 7G VM with only 64K mTHP enabled, medians of 15
runs:
defrag_mode=0 defrag_mode=1 defrag_mode=1
+ series
compact_stall 1690 30669 7366
64K faults allocated 20750 7391 20129
64K faults falling back to 4K 725 13832 1339
defrag_mode=1 still compacts more than defrag_mode=0. That is by design,
as it keeps 64K faults out of unmovable and reclaimable pageblocks [2].
With the series, defrag_mode=0 is unchanged at 20698 64K faults
allocated.
Based on page_alloc/for-7.5/misc. Two trivial conflicts with other trees:
- v7.4-rc2 changed the trace_reclaim_retry_zone() call next to patch
2's change in should_reclaim_retry();
- mm-new moved struct scan_control to mm/vmscan.h, where patch 3's
new field goes.
v3:
- Patch 1: declare the watermark functions in mm/page_alloc.h; rewrite
the comment above __zone_watermark_ok() and the changelog
(Vlastimil, Harry)
- Patch 2: plainer comments in should_reclaim_retry() and
__compact_finished()
- Patch 3: pass alloc_flags to try_to_free_pages() (Vlastimil)
- Rebased onto page_alloc/for-7.5/misc
v2:
- Patch 2 changelog: describe what the fast path change does to pages
on the per-cpu lists (Andrew); drop the should_reclaim_retry()
paragraph
- Patch 3 changelog: say that the benchmark baseline already has the
high-order check honouring ALLOC_NOFRAGMENT (Andrew)
- Rebased onto v7.4-rc1
v2: https://lore.kernel.org/all/20261102093939.3340540-1-kas@kernel.org/
v1: https://lore.kernel.org/all/20261028132638.2560361-1-kirill@shutemov.name/
[1] https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@arm.com/
[2] https://lore.kernel.org/all/auCDnqJc7fsBpUW2@thinkstation/
Kiryl Shutsemau (Meta) (3):
mm: page_alloc: add a watermark check that knows the migratetype
mm: page_alloc: make the high-order watermark check honour
ALLOC_NOFRAGMENT
mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
include/linux/mmzone.h | 3 --
mm/compaction.c | 16 +++++++----
mm/internal.h | 3 +-
mm/page_alloc.c | 62 ++++++++++++++++++++++++++++++++----------
mm/page_alloc.h | 8 ++++++
mm/vmscan.c | 24 ++++++++++++----
6 files changed, 88 insertions(+), 28 deletions(-)
Range-diff against v2:
1: ed3ef9f9fd13 ! 1: e3c2bded052a mm: page_alloc: add a watermark check that knows the migratetype
@@ Metadata
## Commit message ##
mm: page_alloc: add a watermark check that knows the migratetype
- The high-order check in __zone_watermark_ok() accepts a free page of any
- migratetype. Under ALLOC_NOFRAGMENT, a request below pageblock_order can
- only take a page of its own migratetype or claim a whole free block, so
- the check can pass when the allocation cannot succeed.
+ For a high-order request, __zone_watermark_ok() accepts a free page of
+ any migratetype. An ALLOC_NOFRAGMENT request below pageblock_order can
+ take a page of its own migratetype or claim a whole free pageblock, but
+ not steal from another migratetype's pageblock. So the check can pass
+ for an allocation that is going to fail.
- Telling those apart needs the request's migratetype, but not every check
- wants them told apart. A check whether an allocation can be served now
- has to count only the pages it can take.
+ To tell these apart, the check needs the migratetype of the request.
+ Add __zone_watermark_ok_mt(), which takes it, and use it where the
+ caller asks whether this allocation can succeed:
- A check whether there is enough free memory counts every migratetype:
- kswapd and kcompactd balancing a zone, order-0 thresholds, the
- slowpath's forecast whether reclaim and compaction can still make
- progress. Reclaim and compaction work across migratetypes to free whole
- pageblocks.
+ - zone_watermark_fast();
+ - the recheck after node reclaim in get_page_from_freelist();
+ - compaction_suit_allocation_order().
- Move the body of __zone_watermark_ok() into __zone_watermark_ok_mt(),
- which takes the migratetype as well, and make __zone_watermark_ok() a
- wrapper around it for the second kind of check. These use the new
- function:
+ Other callers keep __zone_watermark_ok(), which counts every
+ migratetype. kswapd and kcompactd balance the zone as a whole.
+ should_reclaim_retry() asks whether the allocation could succeed if all
+ reclaimable pages were freed, and compaction can then free a whole
+ pageblock for it to claim.
- - the allocator fast path, zone_watermark_fast();
- - its recheck after node reclaim in get_page_from_freelist();
- - compaction's up-front check, compaction_suit_allocation_order().
+ kcompactd passes MIGRATE_MOVABLE to compaction_suit_allocation_order().
+ It never sets ALLOC_NOFRAGMENT, so the value is not used.
- Declare both in mm/internal.h: nothing outside mm uses them.
+ Nothing outside mm/ uses __zone_watermark_ok(). Move its declaration
+ to mm/page_alloc.h, next to the new one.
- Nothing reads the migratetype yet: no functional change.
+ Nothing reads the migratetype yet, so there is no functional change.
+ Preparation for making the high-order check honour ALLOC_NOFRAGMENT.
+ Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
@@ mm/compaction.c: static void kcompactd_do_work(pg_data_t *pgdat)
continue;
- ## mm/internal.h ##
-@@ mm/internal.h: extern int min_free_kbytes;
- extern int defrag_mode;
-
- void setup_per_zone_wmarks(void);
-+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
-+ int highest_zoneidx, unsigned int alloc_flags,
-+ long free_pages);
-+bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
-+ unsigned long mark, int highest_zoneidx,
-+ unsigned int alloc_flags, long free_pages,
-+ int migratetype);
- void calculate_min_free_kbytes(void);
- int __meminit init_per_zone_wmark_min(void);
-
-
## mm/page_alloc.c ##
@@ mm/page_alloc.c: static inline long __zone_watermark_unusable_free(struct zone *z,
* one free page of a suitable size. Checking now avoids taking the zone lock
@@ mm/page_alloc.c: bool __zone_watermark_ok(struct zone *z, unsigned int order, un
}
+/*
-+ * For checks that are not about serving one allocation now: kswapd and
-+ * kcompactd balancing a zone, order-0 thresholds, forecasts whether
-+ * reclaim and compaction can still make progress. They count free pages
-+ * of every migratetype: reclaim and compaction work across migratetypes
-+ * to free whole pageblocks.
++ * Return true if free base pages in the zone are above 'mark' regardless
++ * of migratetype. Use it to check whether the zone as a whole has enough
++ * free memory. Use __zone_watermark_ok_mt() instead to check whether a
++ * particular allocation can be served.
+ */
+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
+ int highest_zoneidx, unsigned int alloc_flags,
@@ mm/page_alloc.c: get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int
continue;
}
+
+ ## mm/page_alloc.h ##
+@@ mm/page_alloc.h: extern bool free_pages_prepare(struct page *page, unsigned int order);
+
+ extern int user_min_free_kbytes;
+
++bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
++ int highest_zoneidx, unsigned int alloc_flags,
++ long free_pages);
++bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
++ unsigned long mark, int highest_zoneidx,
++ unsigned int alloc_flags, long free_pages,
++ int migratetype);
++
+ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
+ nodemask_t *nodemask, unsigned int alloc_flags);
+ #define __alloc_frozen_pages(...) \
2: 3347b0a24b0e ! 2: c991756a139a mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT
@@ Commit message
With defrag_mode, every slowpath allocation carries ALLOC_NOFRAGMENT. A
request below pageblock_order may take a page of its own migratetype or
claim a whole free block, but not steal from another migratetype's block.
- The high-order part of the watermark check still accepts a free page of
- any migratetype.
+ For a high-order request, the watermark check still accepts a free page
+ of any migratetype.
Under memory pressure, movable allocations take free movable pages as
soon as reclaim frees them. Free space in unmovable and reclaimable
@@ mm/compaction.c: static enum compact_result __compact_finished(struct compact_co
!free_area_empty(area, MIGRATE_CMA))
return COMPACT_SUCCESS;
#endif
-+ /* Under ALLOC_NOFRAGMENT it may only claim a whole block */
++ /* An ALLOC_NOFRAGMENT request may only claim a whole block */
+ if ((cc->alloc_flags & ALLOC_NOFRAGMENT) &&
+ order < pageblock_order)
+ continue;
@@ mm/page_alloc.c: should_reclaim_retry(gfp_t gfp_mask, unsigned order,
/*
* Would the allocation succeed if we reclaimed all
- * reclaimable pages?
-+ * reclaimable pages? Count free pages of every migratetype:
-+ * reclaim and compaction work across migratetypes to free a
-+ * whole pageblock, which the allocation can then claim.
++ * reclaimable pages? Count free pages of all migratetypes.
++ * Compaction can then free a whole pageblock for the
++ * allocation to claim.
*/
wmark = __zone_watermark_ok(zone, order, min_wmark,
- ac->highest_zoneidx, alloc_flags, available);
3: 0ecbcc5f5fb0 ! 3: fd8bb9835186 mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
@@ Commit message
64K faults allocated 20750 11538 19759
64K faults falling back to 4K 725 9945 1664
- Pass ALLOC_NOFRAGMENT from the allocation down to try_to_free_pages().
- Both checks then ask __zone_watermark_ok_mt() for a free page the
- allocation can take, and an order-4 direct reclaim frees 40 pages on
- average.
+ Pass the allocation's alloc_flags down to try_to_free_pages(), and
+ record ALLOC_NOFRAGMENT in scan_control. Both checks then ask
+ __zone_watermark_ok_mt() for a free page the allocation can take, and an
+ order-4 direct reclaim frees 40 pages on average.
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Reported-by: Dev Jain <dev.jain@arm.com>
@@ mm/internal.h: void folio_mark_lazyfree(struct folio *folio);
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *mask);
+ gfp_t gfp_mask, const nodemask_t *mask,
-+ bool nofragment);
++ unsigned int alloc_flags);
unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
int zone_idx);
@@ mm/page_alloc.c: __perform_reclaim(gfp_t gfp_mask, unsigned int order,
progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
- ac->nodemask);
-+ ac->nodemask,
-+ alloc_flags & ALLOC_NOFRAGMENT);
++ ac->nodemask, alloc_flags);
memalloc_noreclaim_restore(noreclaim_flag);
fs_reclaim_release(gfp_mask);
@@ mm/vmscan.c: struct scan_control {
/* One of the zones is ready for compaction */
unsigned int compaction_ready:1;
-+ /* The allocation cannot fall back to other migratetypes' blocks */
++ /* The allocation may not steal from other migratetypes' blocks */
+ unsigned int nofragment:1;
+
/* There is easily reclaimable cold cache in the current node */
@@ mm/vmscan.c: static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *nodemask)
+ gfp_t gfp_mask, const nodemask_t *nodemask,
-+ bool nofragment)
++ unsigned int alloc_flags)
{
unsigned long nr_reclaimed;
struct scan_control sc = {
@@ mm/vmscan.c: unsigned long try_to_free_pages(struct zonelist *zonelist, int orde
.may_writepage = 1,
.may_unmap = 1,
.may_swap = 1,
-+ .nofragment = nofragment,
++ .nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
};
/*
base-commit: 3a99594e62fce8c646e8aac01d38af16d6833a40
--
2.54.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype
2026-11-16 12:50 [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
@ 2026-11-16 12:50 ` Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2 siblings, 0 replies; 4+ messages in thread
From: Kiryl Shutsemau (Meta) @ 2026-11-16 12:50 UTC (permalink / raw)
To: Andrew Morton, Vlastimil Babka, Johannes Weiner,
David Hildenbrand, Harry Yoo
Cc: Kiryl Shutsemau (Meta),
Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
Lorenzo Stoakes, Liam R. Howlett, Mike Rapoport, Qi Zheng,
Shakeel Butt, Kairui Song, Barry Song, Axel Rasmussen,
Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, Dev Jain,
Usama Arif, linux-mm, linux-kernel, kernel-team
For a high-order request, __zone_watermark_ok() accepts a free page of
any migratetype. An ALLOC_NOFRAGMENT request below pageblock_order can
take a page of its own migratetype or claim a whole free pageblock, but
not steal from another migratetype's pageblock. So the check can pass
for an allocation that is going to fail.
To tell these apart, the check needs the migratetype of the request.
Add __zone_watermark_ok_mt(), which takes it, and use it where the
caller asks whether this allocation can succeed:
- zone_watermark_fast();
- the recheck after node reclaim in get_page_from_freelist();
- compaction_suit_allocation_order().
Other callers keep __zone_watermark_ok(), which counts every
migratetype. kswapd and kcompactd balance the zone as a whole.
should_reclaim_retry() asks whether the allocation could succeed if all
reclaimable pages were freed, and compaction can then free a whole
pageblock for it to claim.
kcompactd passes MIGRATE_MOVABLE to compaction_suit_allocation_order().
It never sets ALLOC_NOFRAGMENT, so the value is not used.
Nothing outside mm/ uses __zone_watermark_ok(). Move its declaration
to mm/page_alloc.h, next to the new one.
Nothing reads the migratetype yet, so there is no functional change.
Preparation for making the high-order check honour ALLOC_NOFRAGMENT.
Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
include/linux/mmzone.h | 3 ---
mm/compaction.c | 11 ++++++-----
mm/page_alloc.c | 36 +++++++++++++++++++++++++++---------
mm/page_alloc.h | 8 ++++++++
4 files changed, 41 insertions(+), 17 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecd1ebd49756..4d9347a399fa 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1637,9 +1637,6 @@ static inline unsigned long pgdat_end_pfn(pg_data_t *pgdat)
#include <linux/memory_hotplug.h>
void build_all_zonelists(pg_data_t *pgdat);
-bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
- int highest_zoneidx, unsigned int alloc_flags,
- long free_pages);
bool zone_watermark_ok(struct zone *z, unsigned int order,
unsigned long mark, int highest_zoneidx,
unsigned int alloc_flags);
diff --git a/mm/compaction.c b/mm/compaction.c
index f1b2060eb201..6c553d0e90f8 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2520,7 +2520,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
static enum compact_result
compaction_suit_allocation_order(struct zone *zone, unsigned int order,
int highest_zoneidx, unsigned int alloc_flags,
- bool async, bool kcompactd)
+ int migratetype, bool async, bool kcompactd)
{
unsigned long free_pages;
unsigned long watermark;
@@ -2531,8 +2531,8 @@ compaction_suit_allocation_order(struct zone *zone, unsigned int order,
free_pages = zone_page_state(zone, NR_FREE_PAGES);
watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK);
- if (__zone_watermark_ok(zone, order, watermark, highest_zoneidx,
- alloc_flags, free_pages))
+ if (__zone_watermark_ok_mt(zone, order, watermark, highest_zoneidx,
+ alloc_flags, free_pages, migratetype))
return COMPACT_SUCCESS;
/*
@@ -2588,6 +2588,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
ret = compaction_suit_allocation_order(cc->zone, cc->order,
cc->highest_zoneidx,
cc->alloc_flags,
+ cc->migratetype,
cc->mode == MIGRATE_ASYNC,
!cc->direct_compaction);
if (ret != COMPACT_CONTINUE)
@@ -3073,7 +3074,7 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat)
ret = compaction_suit_allocation_order(zone,
pgdat->kcompactd_max_order,
highest_zoneidx, alloc_flags,
- false, true);
+ MIGRATE_MOVABLE, false, true);
if (ret == COMPACT_CONTINUE)
return true;
}
@@ -3116,7 +3117,7 @@ static void kcompactd_do_work(pg_data_t *pgdat)
ret = compaction_suit_allocation_order(zone,
cc.order, zoneid, cc.alloc_flags,
- false, true);
+ MIGRATE_MOVABLE, false, true);
if (ret != COMPACT_CONTINUE)
continue;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9e1261521b6a..0ebfbbb5d7c5 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3588,9 +3588,10 @@ static inline long __zone_watermark_unusable_free(struct zone *z,
* one free page of a suitable size. Checking now avoids taking the zone lock
* to check in the allocation paths if no pages are free.
*/
-bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
- int highest_zoneidx, unsigned int alloc_flags,
- long free_pages)
+bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
+ unsigned long mark, int highest_zoneidx,
+ unsigned int alloc_flags, long free_pages,
+ int migratetype)
{
long min = mark;
int o;
@@ -3666,6 +3667,20 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
return false;
}
+/*
+ * Return true if free base pages in the zone are above 'mark' regardless
+ * of migratetype. Use it to check whether the zone as a whole has enough
+ * free memory. Use __zone_watermark_ok_mt() instead to check whether a
+ * particular allocation can be served.
+ */
+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
+ int highest_zoneidx, unsigned int alloc_flags,
+ long free_pages)
+{
+ return __zone_watermark_ok_mt(z, order, mark, highest_zoneidx,
+ alloc_flags, free_pages, MIGRATE_MOVABLE);
+}
+
bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
int highest_zoneidx, unsigned int alloc_flags)
{
@@ -3698,8 +3713,8 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order,
return true;
}
- if (__zone_watermark_ok(z, order, mark, highest_zoneidx, alloc_flags,
- free_pages))
+ if (__zone_watermark_ok_mt(z, order, mark, highest_zoneidx, alloc_flags,
+ free_pages, gfp_migratetype(gfp_mask)))
return true;
/*
@@ -3711,8 +3726,9 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order,
if (unlikely(!order && (alloc_flags & ALLOC_MIN_RESERVE) && z->watermark_boost
&& ((alloc_flags & ALLOC_WMARK_MASK) == WMARK_MIN))) {
mark = z->_watermark[WMARK_MIN];
- return __zone_watermark_ok(z, order, mark, highest_zoneidx,
- alloc_flags, free_pages);
+ return __zone_watermark_ok_mt(z, order, mark, highest_zoneidx,
+ alloc_flags, free_pages,
+ gfp_migratetype(gfp_mask));
}
return false;
@@ -3932,8 +3948,10 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
continue;
/* did we reclaim enough */
- if (!zone_watermark_ok(zone, order, mark,
- ac->highest_zoneidx, alloc_flags))
+ if (!__zone_watermark_ok_mt(zone, order, mark,
+ ac->highest_zoneidx, alloc_flags,
+ zone_page_state(zone, NR_FREE_PAGES),
+ gfp_migratetype(gfp_mask)))
continue;
}
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index e238ee91ed24..d95420093b15 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -231,6 +231,14 @@ extern bool free_pages_prepare(struct page *page, unsigned int order);
extern int user_min_free_kbytes;
+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
+ int highest_zoneidx, unsigned int alloc_flags,
+ long free_pages);
+bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
+ unsigned long mark, int highest_zoneidx,
+ unsigned int alloc_flags, long free_pages,
+ int migratetype);
+
struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
nodemask_t *nodemask, unsigned int alloc_flags);
#define __alloc_frozen_pages(...) \
--
2.54.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT
2026-11-16 12:50 [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
@ 2026-11-16 12:51 ` Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2 siblings, 0 replies; 4+ messages in thread
From: Kiryl Shutsemau (Meta) @ 2026-11-16 12:51 UTC (permalink / raw)
To: Andrew Morton, Vlastimil Babka, Johannes Weiner,
David Hildenbrand, Harry Yoo
Cc: Kiryl Shutsemau (Meta),
Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
Lorenzo Stoakes, Liam R. Howlett, Mike Rapoport, Qi Zheng,
Shakeel Butt, Kairui Song, Barry Song, Axel Rasmussen,
Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, Dev Jain,
Usama Arif, linux-mm, linux-kernel, kernel-team
With defrag_mode, every slowpath allocation carries ALLOC_NOFRAGMENT. A
request below pageblock_order may take a page of its own migratetype or
claim a whole free block, but not steal from another migratetype's block.
For a high-order request, the watermark check still accepts a free page
of any migratetype.
Under memory pressure, movable allocations take free movable pages as
soon as reclaim frees them. Free space in unmovable and reclaimable
pageblocks is off-limits to them under ALLOC_NOFRAGMENT, and builds up.
The free pages of order 4 and up end up mostly in those pageblocks.
A 64K mTHP fault cannot take them, but direct compaction's up-front check
counts them and returns COMPACT_SUCCESS without compacting anything. The
allocation then fails. Because compaction "succeeded", it is never
deferred, and every fault stalls in it again before falling back to small
pages.
Dev Jain's benchmark faults 64K mTHPs under swap pressure, in a 7G VM
with only the 64K size enabled. With defrag_mode, those faults stall in
compaction 18 times as often and get about a third of the folios
(medians of 15 runs):
defrag_mode=0 defrag_mode=1 defrag_mode=1
+ this patch
compact_stall 1690 30669 2607
compact_fail 621 30531 906
64K faults allocated 20750 7391 11538
64K faults falling back to 4K 725 13832 9945
Make __zone_watermark_ok_mt() skip other migratetypes' free lists below
pageblock_order for an ALLOC_NOFRAGMENT request. Give
__compact_finished() the same rule for its fallback test, so that
compaction, once it runs, stops on a page the request can take.
zone_watermark_fast() calls __zone_watermark_ok_mt() too. With
defrag_mode, an allocation now wakes kswapd and starts reclaim or
compaction as soon as the free lists hold nothing it can take. For the
orders the per-cpu lists serve, that is earlier than before: the check
does not count their pages, so the allocation no longer uses up the local
list first. A request that may not wake kswapd or reclaim, such as an
mTHP fault with THP defrag=never, falls back to a smaller folio instead.
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Reported-by: Dev Jain <dev.jain@arm.com>
Closes: https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@arm.com/
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
mm/compaction.c | 5 +++++
mm/page_alloc.c | 21 ++++++++++++++++++---
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 6c553d0e90f8..7312f7515c53 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2365,6 +2365,11 @@ static enum compact_result __compact_finished(struct compact_control *cc)
!free_area_empty(area, MIGRATE_CMA))
return COMPACT_SUCCESS;
#endif
+ /* An ALLOC_NOFRAGMENT request may only claim a whole block */
+ if ((cc->alloc_flags & ALLOC_NOFRAGMENT) &&
+ order < pageblock_order)
+ continue;
+
/*
* Job done if allocation would steal freepages from
* other migratetype buddy lists.
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 0ebfbbb5d7c5..6e75c03b303b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3587,6 +3587,11 @@ static inline long __zone_watermark_unusable_free(struct zone *z,
* will return true of the order-0 watermark is reached and there is at least
* one free page of a suitable size. Checking now avoids taking the zone lock
* to check in the allocation paths if no pages are free.
+ *
+ * Under ALLOC_NOFRAGMENT, a request below pageblock_order can take another
+ * migratetype's page only by claiming the whole block, so only free pages of
+ * @migratetype count below pageblock_order. Without ALLOC_NOFRAGMENT,
+ * @migratetype is not used.
*/
bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
unsigned long mark, int highest_zoneidx,
@@ -3649,6 +3654,9 @@ bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
continue;
for (mt = 0; mt < MIGRATE_PCPTYPES; mt++) {
+ if ((alloc_flags & ALLOC_NOFRAGMENT) &&
+ mt != migratetype && o < pageblock_order)
+ continue;
if (!free_area_empty(area, mt))
return true;
}
@@ -3677,8 +3685,12 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
int highest_zoneidx, unsigned int alloc_flags,
long free_pages)
{
+ /* Serving an ALLOC_NOFRAGMENT request needs __zone_watermark_ok_mt() */
+ VM_WARN_ON_ONCE(order && order < pageblock_order &&
+ (alloc_flags & ALLOC_NOFRAGMENT));
return __zone_watermark_ok_mt(z, order, mark, highest_zoneidx,
- alloc_flags, free_pages, MIGRATE_MOVABLE);
+ alloc_flags & ~ALLOC_NOFRAGMENT,
+ free_pages, MIGRATE_MOVABLE);
}
bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
@@ -4693,10 +4705,13 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
/*
* Would the allocation succeed if we reclaimed all
- * reclaimable pages?
+ * reclaimable pages? Count free pages of all migratetypes.
+ * Compaction can then free a whole pageblock for the
+ * allocation to claim.
*/
wmark = __zone_watermark_ok(zone, order, min_wmark,
- ac->highest_zoneidx, alloc_flags, available);
+ ac->highest_zoneidx,
+ alloc_flags & ~ALLOC_NOFRAGMENT, available);
trace_reclaim_retry_zone(z, order, reclaimable,
available, min_wmark, *no_progress_loops, wmark);
if (wmark) {
--
2.54.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
2026-11-16 12:50 [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
@ 2026-11-16 12:51 ` Kiryl Shutsemau (Meta)
2 siblings, 0 replies; 4+ messages in thread
From: Kiryl Shutsemau (Meta) @ 2026-11-16 12:51 UTC (permalink / raw)
To: Andrew Morton, Vlastimil Babka, Johannes Weiner,
David Hildenbrand, Harry Yoo
Cc: Kiryl Shutsemau (Meta),
Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
Lorenzo Stoakes, Liam R. Howlett, Mike Rapoport, Qi Zheng,
Shakeel Butt, Kairui Song, Barry Song, Axel Rasmussen,
Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, Dev Jain,
Usama Arif, linux-mm, linux-kernel, kernel-team
Direct reclaim for a costly order stops as soon as the allocation could
succeed. compaction_ready() skips a zone and should_continue_reclaim()
stops reclaiming when zone_watermark_ok() passes, and zone_watermark_ok()
accepts a free page of any migratetype.
With defrag_mode, the allocation behind the reclaim is ALLOC_NOFRAGMENT
and cannot take the free pages left in unmovable and reclaimable
pageblocks. Reclaim stops on them anyway, having freed little.
In Dev Jain's 64K mTHP benchmark, with the high-order watermark check
already honouring ALLOC_NOFRAGMENT, an order-4 direct reclaim frees 16
pages on average with defrag_mode and 45 without. 64K faults get a bit
over half as many folios as without defrag_mode (medians of 15 runs):
defrag_mode=0 defrag_mode=1 defrag_mode=1
+ this patch
64K faults allocated 20750 11538 19759
64K faults falling back to 4K 725 9945 1664
Pass the allocation's alloc_flags down to try_to_free_pages(), and
record ALLOC_NOFRAGMENT in scan_control. Both checks then ask
__zone_watermark_ok_mt() for a free page the allocation can take, and an
order-4 direct reclaim frees 40 pages on average.
Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Reported-by: Dev Jain <dev.jain@arm.com>
Closes: https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@arm.com/
Tested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
try_to_free_pages() takes the whole alloc_flags, but reclaim uses only
ALLOC_NOFRAGMENT. Using all of them would change when costly reclaim
stops for every allocation, not only with defrag_mode. ALLOC_CMA would
count free CMA pages for movable requests, and the reserve flags would
lower the min.
Storing the whole word in scan_control and masking it would work the
same, but kswapd and memcg reclaim have no alloc_flags to put there.
So the patch stores ALLOC_NOFRAGMENT as a bit in scan_control and
reconstructs it in allocation_can_succeed().
mm/internal.h | 3 ++-
mm/page_alloc.c | 7 ++++---
mm/vmscan.c | 24 +++++++++++++++++++-----
3 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index 566cbd2dce48..3ef9b7036afd 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -65,7 +65,8 @@ void folio_mark_lazyfree(struct folio *folio);
/* mm/vmscan.c */
unsigned long zone_reclaimable_pages(struct zone *zone);
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *mask);
+ gfp_t gfp_mask, const nodemask_t *mask,
+ unsigned int alloc_flags);
unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
int zone_idx);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6e75c03b303b..93362e0f699b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4459,7 +4459,7 @@ static unsigned int check_retry_zonelist(unsigned int seq)
/* Perform direct synchronous page reclaim */
static unsigned long
-__perform_reclaim(gfp_t gfp_mask, unsigned int order,
+__perform_reclaim(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags,
const struct alloc_context *ac)
{
unsigned int noreclaim_flag;
@@ -4473,7 +4473,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
noreclaim_flag = memalloc_noreclaim_save();
progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
- ac->nodemask);
+ ac->nodemask, alloc_flags);
memalloc_noreclaim_restore(noreclaim_flag);
fs_reclaim_release(gfp_mask);
@@ -4495,7 +4495,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
int reclaim_order = ac->reclaim_compact_order;
psi_memstall_enter(&pflags);
- *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order, ac);
+ *did_some_progress = __perform_reclaim(gfp_mask, reclaim_order,
+ alloc_flags, ac);
if (unlikely(!(*did_some_progress)))
goto out;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..3fbcdcc87a00 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -142,6 +142,9 @@ struct scan_control {
/* One of the zones is ready for compaction */
unsigned int compaction_ready:1;
+ /* The allocation may not steal from other migratetypes' blocks */
+ unsigned int nofragment:1;
+
/* There is easily reclaimable cold cache in the current node */
unsigned int cache_trim_mode:1;
@@ -6316,6 +6319,17 @@ static bool in_reclaim_compaction(struct scan_control *sc)
return false;
}
+static bool allocation_can_succeed(struct zone *zone, struct scan_control *sc,
+ unsigned long mark)
+{
+ unsigned int alloc_flags = sc->nofragment ? ALLOC_NOFRAGMENT : 0;
+ long free_pages = zone_page_state(zone, NR_FREE_PAGES);
+ int migratetype = gfp_migratetype(sc->gfp_mask);
+
+ return __zone_watermark_ok_mt(zone, sc->order, mark, sc->reclaim_idx,
+ alloc_flags, free_pages, migratetype);
+}
+
/*
* Reclaim/compaction is used for high-order allocation requests. It reclaims
* order-0 pages before compacting the zone. should_continue_reclaim() returns
@@ -6354,8 +6368,7 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat,
unsigned long watermark = min_wmark_pages(zone);
/* Allocation can already succeed, nothing to do */
- if (zone_watermark_ok(zone, sc->order, watermark,
- sc->reclaim_idx, 0))
+ if (allocation_can_succeed(zone, sc, watermark))
return false;
if (compaction_suitable(zone, sc->order, watermark,
@@ -6580,8 +6593,7 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)
return false;
/* Allocation can already succeed, nothing to do */
- if (zone_watermark_ok(zone, sc->order, min_wmark_pages(zone),
- sc->reclaim_idx, 0))
+ if (allocation_can_succeed(zone, sc, min_wmark_pages(zone)))
return true;
/*
@@ -6965,7 +6977,8 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
}
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *nodemask)
+ gfp_t gfp_mask, const nodemask_t *nodemask,
+ unsigned int alloc_flags)
{
unsigned long nr_reclaimed;
struct scan_control sc = {
@@ -6978,6 +6991,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
.may_writepage = 1,
.may_unmap = 1,
.may_swap = 1,
+ .nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
};
/*
--
2.54.0
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-16 12:51 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 12:50 [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®