mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT
@ 2026-11-16 12:50 Kiryl Shutsemau (Meta)
  2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Kiryl Shutsemau (Meta) @ 2026-11-16 12:50 UTC (permalink / raw)
  To: Andrew Morton, Vlastimil Babka, Johannes Weiner,
	David Hildenbrand, Harry Yoo
  Cc: Kiryl Shutsemau (Meta),
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Lorenzo Stoakes, Liam R. Howlett, Mike Rapoport, Qi Zheng,
	Shakeel Butt, Kairui Song, Barry Song, Axel Rasmussen,
	Yuanchu Xie, Wei Xu, Baoquan He, Baolin Wang, Dev Jain,
	Usama Arif, linux-mm, linux-kernel, kernel-team

With defrag_mode, 64K mTHP faults under memory pressure stall in
compaction 18 times as often as without it, and get about a third of the
folios.  Dev Jain reported it [1].

The high-order watermark check counts free pages of every migratetype.
With defrag_mode, every slowpath allocation is ALLOC_NOFRAGMENT, and a
request below pageblock_order cannot take free pages from another
migratetype's pageblock.  Under pressure, that is where the large free
pages end up.  Two callers act on the wrong answer:

  - direct compaction's up-front check returns COMPACT_SUCCESS without
    compacting, so compaction is never deferred;
  - direct reclaim for a costly order stops early, having freed little.

Patches 1-2 give the check the request's migratetype and make it skip
the free lists the request cannot take.  Patch 3 tells direct reclaim
that the allocation is ALLOC_NOFRAGMENT.

Dev's benchmark in a 7G VM with only 64K mTHP enabled, medians of 15
runs:

                              defrag_mode=0  defrag_mode=1  defrag_mode=1
                                                            + series
    compact_stall                      1690          30669          7366
    64K faults allocated              20750           7391         20129
    64K faults falling back to 4K       725          13832          1339

defrag_mode=1 still compacts more than defrag_mode=0.  That is by design,
as it keeps 64K faults out of unmovable and reclaimable pageblocks [2].
With the series, defrag_mode=0 is unchanged at 20698 64K faults
allocated.

Based on page_alloc/for-7.5/misc.  Two trivial conflicts with other trees:

  - v7.4-rc2 changed the trace_reclaim_retry_zone() call next to patch
    2's change in should_reclaim_retry();
  - mm-new moved struct scan_control to mm/vmscan.h, where patch 3's
    new field goes.

v3:
 - Patch 1: declare the watermark functions in mm/page_alloc.h; rewrite
   the comment above __zone_watermark_ok() and the changelog
   (Vlastimil, Harry)
 - Patch 2: plainer comments in should_reclaim_retry() and
   __compact_finished()
 - Patch 3: pass alloc_flags to try_to_free_pages() (Vlastimil)
 - Rebased onto page_alloc/for-7.5/misc

v2:
 - Patch 2 changelog: describe what the fast path change does to pages
   on the per-cpu lists (Andrew); drop the should_reclaim_retry()
   paragraph
 - Patch 3 changelog: say that the benchmark baseline already has the
   high-order check honouring ALLOC_NOFRAGMENT (Andrew)
 - Rebased onto v7.4-rc1

v2: https://lore.kernel.org/all/20261102093939.3340540-1-kas@kernel.org/
v1: https://lore.kernel.org/all/20261028132638.2560361-1-kirill@shutemov.name/

[1] https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@arm.com/
[2] https://lore.kernel.org/all/auCDnqJc7fsBpUW2@thinkstation/

Kiryl Shutsemau (Meta) (3):
  mm: page_alloc: add a watermark check that knows the migratetype
  mm: page_alloc: make the high-order watermark check honour
    ALLOC_NOFRAGMENT
  mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT

 include/linux/mmzone.h |  3 --
 mm/compaction.c        | 16 +++++++----
 mm/internal.h          |  3 +-
 mm/page_alloc.c        | 62 ++++++++++++++++++++++++++++++++----------
 mm/page_alloc.h        |  8 ++++++
 mm/vmscan.c            | 24 ++++++++++++----
 6 files changed, 88 insertions(+), 28 deletions(-)

Range-diff against v2:
1:  ed3ef9f9fd13 ! 1:  e3c2bded052a mm: page_alloc: add a watermark check that knows the migratetype
    @@ Metadata
      ## Commit message ##
         mm: page_alloc: add a watermark check that knows the migratetype
     
    -    The high-order check in __zone_watermark_ok() accepts a free page of any
    -    migratetype.  Under ALLOC_NOFRAGMENT, a request below pageblock_order can
    -    only take a page of its own migratetype or claim a whole free block, so
    -    the check can pass when the allocation cannot succeed.
    +    For a high-order request, __zone_watermark_ok() accepts a free page of
    +    any migratetype.  An ALLOC_NOFRAGMENT request below pageblock_order can
    +    take a page of its own migratetype or claim a whole free pageblock, but
    +    not steal from another migratetype's pageblock.  So the check can pass
    +    for an allocation that is going to fail.
     
    -    Telling those apart needs the request's migratetype, but not every check
    -    wants them told apart.  A check whether an allocation can be served now
    -    has to count only the pages it can take.
    +    To tell these apart, the check needs the migratetype of the request.
    +    Add __zone_watermark_ok_mt(), which takes it, and use it where the
    +    caller asks whether this allocation can succeed:
     
    -    A check whether there is enough free memory counts every migratetype:
    -    kswapd and kcompactd balancing a zone, order-0 thresholds, the
    -    slowpath's forecast whether reclaim and compaction can still make
    -    progress.  Reclaim and compaction work across migratetypes to free whole
    -    pageblocks.
    +      - zone_watermark_fast();
    +      - the recheck after node reclaim in get_page_from_freelist();
    +      - compaction_suit_allocation_order().
     
    -    Move the body of __zone_watermark_ok() into __zone_watermark_ok_mt(),
    -    which takes the migratetype as well, and make __zone_watermark_ok() a
    -    wrapper around it for the second kind of check.  These use the new
    -    function:
    +    Other callers keep __zone_watermark_ok(), which counts every
    +    migratetype.  kswapd and kcompactd balance the zone as a whole.
    +    should_reclaim_retry() asks whether the allocation could succeed if all
    +    reclaimable pages were freed, and compaction can then free a whole
    +    pageblock for it to claim.
     
    -      - the allocator fast path, zone_watermark_fast();
    -      - its recheck after node reclaim in get_page_from_freelist();
    -      - compaction's up-front check, compaction_suit_allocation_order().
    +    kcompactd passes MIGRATE_MOVABLE to compaction_suit_allocation_order().
    +    It never sets ALLOC_NOFRAGMENT, so the value is not used.
     
    -    Declare both in mm/internal.h: nothing outside mm uses them.
    +    Nothing outside mm/ uses __zone_watermark_ok().  Move its declaration
    +    to mm/page_alloc.h, next to the new one.
     
    -    Nothing reads the migratetype yet: no functional change.
    +    Nothing reads the migratetype yet, so there is no functional change.
    +    Preparation for making the high-order check honour ALLOC_NOFRAGMENT.
     
    +    Reviewed-by: Harry Yoo (Meta) <harry@kernel.org>
         Tested-by: Dev Jain <dev.jain@arm.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
    @@ mm/compaction.c: static void kcompactd_do_work(pg_data_t *pgdat)
      			continue;
      
     
    - ## mm/internal.h ##
    -@@ mm/internal.h: extern int min_free_kbytes;
    - extern int defrag_mode;
    - 
    - void setup_per_zone_wmarks(void);
    -+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
    -+			 int highest_zoneidx, unsigned int alloc_flags,
    -+			 long free_pages);
    -+bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
    -+			    unsigned long mark, int highest_zoneidx,
    -+			    unsigned int alloc_flags, long free_pages,
    -+			    int migratetype);
    - void calculate_min_free_kbytes(void);
    - int __meminit init_per_zone_wmark_min(void);
    - 
    -
      ## mm/page_alloc.c ##
     @@ mm/page_alloc.c: static inline long __zone_watermark_unusable_free(struct zone *z,
       * one free page of a suitable size. Checking now avoids taking the zone lock
    @@ mm/page_alloc.c: bool __zone_watermark_ok(struct zone *z, unsigned int order, un
      }
      
     +/*
    -+ * For checks that are not about serving one allocation now: kswapd and
    -+ * kcompactd balancing a zone, order-0 thresholds, forecasts whether
    -+ * reclaim and compaction can still make progress.  They count free pages
    -+ * of every migratetype: reclaim and compaction work across migratetypes
    -+ * to free whole pageblocks.
    ++ * Return true if free base pages in the zone are above 'mark' regardless
    ++ * of migratetype. Use it to check whether the zone as a whole has enough
    ++ * free memory. Use __zone_watermark_ok_mt() instead to check whether a
    ++ * particular allocation can be served.
     + */
     +bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
     +			 int highest_zoneidx, unsigned int alloc_flags,
    @@ mm/page_alloc.c: get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int
      				continue;
      		}
      
    +
    + ## mm/page_alloc.h ##
    +@@ mm/page_alloc.h: extern bool free_pages_prepare(struct page *page, unsigned int order);
    + 
    + extern int user_min_free_kbytes;
    + 
    ++bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
    ++			 int highest_zoneidx, unsigned int alloc_flags,
    ++			 long free_pages);
    ++bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
    ++			    unsigned long mark, int highest_zoneidx,
    ++			    unsigned int alloc_flags, long free_pages,
    ++			    int migratetype);
    ++
    + struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
    + 		nodemask_t *nodemask, unsigned int alloc_flags);
    + #define __alloc_frozen_pages(...) \
2:  3347b0a24b0e ! 2:  c991756a139a mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT
    @@ Commit message
         With defrag_mode, every slowpath allocation carries ALLOC_NOFRAGMENT.  A
         request below pageblock_order may take a page of its own migratetype or
         claim a whole free block, but not steal from another migratetype's block.
    -    The high-order part of the watermark check still accepts a free page of
    -    any migratetype.
    +    For a high-order request, the watermark check still accepts a free page
    +    of any migratetype.
     
         Under memory pressure, movable allocations take free movable pages as
         soon as reclaim frees them.  Free space in unmovable and reclaimable
    @@ mm/compaction.c: static enum compact_result __compact_finished(struct compact_co
      			!free_area_empty(area, MIGRATE_CMA))
      			return COMPACT_SUCCESS;
      #endif
    -+		/* Under ALLOC_NOFRAGMENT it may only claim a whole block */
    ++		/* An ALLOC_NOFRAGMENT request may only claim a whole block */
     +		if ((cc->alloc_flags & ALLOC_NOFRAGMENT) &&
     +		    order < pageblock_order)
     +			continue;
    @@ mm/page_alloc.c: should_reclaim_retry(gfp_t gfp_mask, unsigned order,
      		/*
      		 * Would the allocation succeed if we reclaimed all
     -		 * reclaimable pages?
    -+		 * reclaimable pages?  Count free pages of every migratetype:
    -+		 * reclaim and compaction work across migratetypes to free a
    -+		 * whole pageblock, which the allocation can then claim.
    ++		 * reclaimable pages?  Count free pages of all migratetypes.
    ++		 * Compaction can then free a whole pageblock for the
    ++		 * allocation to claim.
      		 */
      		wmark = __zone_watermark_ok(zone, order, min_wmark,
     -				ac->highest_zoneidx, alloc_flags, available);
3:  0ecbcc5f5fb0 ! 3:  fd8bb9835186 mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
    @@ Commit message
             64K faults allocated              20750          11538         19759
             64K faults falling back to 4K       725           9945          1664
     
    -    Pass ALLOC_NOFRAGMENT from the allocation down to try_to_free_pages().
    -    Both checks then ask __zone_watermark_ok_mt() for a free page the
    -    allocation can take, and an order-4 direct reclaim frees 40 pages on
    -    average.
    +    Pass the allocation's alloc_flags down to try_to_free_pages(), and
    +    record ALLOC_NOFRAGMENT in scan_control.  Both checks then ask
    +    __zone_watermark_ok_mt() for a free page the allocation can take, and an
    +    order-4 direct reclaim frees 40 pages on average.
     
         Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
         Reported-by: Dev Jain <dev.jain@arm.com>
    @@ mm/internal.h: void folio_mark_lazyfree(struct folio *folio);
      unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
     -				gfp_t gfp_mask, const nodemask_t *mask);
     +				gfp_t gfp_mask, const nodemask_t *mask,
    -+				bool nofragment);
    ++				unsigned int alloc_flags);
      unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
      			      int zone_idx);
      
    @@ mm/page_alloc.c: __perform_reclaim(gfp_t gfp_mask, unsigned int order,
      
      	progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
     -								ac->nodemask);
    -+				     ac->nodemask,
    -+				     alloc_flags & ALLOC_NOFRAGMENT);
    ++				     ac->nodemask, alloc_flags);
      
      	memalloc_noreclaim_restore(noreclaim_flag);
      	fs_reclaim_release(gfp_mask);
    @@ mm/vmscan.c: struct scan_control {
      	/* One of the zones is ready for compaction */
      	unsigned int compaction_ready:1;
      
    -+	/* The allocation cannot fall back to other migratetypes' blocks */
    ++	/* The allocation may not steal from other migratetypes' blocks */
     +	unsigned int nofragment:1;
     +
      	/* There is easily reclaimable cold cache in the current node */
    @@ mm/vmscan.c: static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist
      unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
     -				gfp_t gfp_mask, const nodemask_t *nodemask)
     +				gfp_t gfp_mask, const nodemask_t *nodemask,
    -+				bool nofragment)
    ++				unsigned int alloc_flags)
      {
      	unsigned long nr_reclaimed;
      	struct scan_control sc = {
    @@ mm/vmscan.c: unsigned long try_to_free_pages(struct zonelist *zonelist, int orde
      		.may_writepage = 1,
      		.may_unmap = 1,
      		.may_swap = 1,
    -+		.nofragment = nofragment,
    ++		.nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
      	};
      
      	/*

base-commit: 3a99594e62fce8c646e8aac01d38af16d6833a40
-- 
2.54.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-16 12:51 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 12:50 [PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:50 ` [PATCH v3 1/3] mm: page_alloc: add a watermark check that knows the migratetype Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 2/3] mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)
2026-11-16 12:51 ` [PATCH v3 3/3] mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT Kiryl Shutsemau (Meta)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®