[PATCH 1/2] mm/vmscan: consolidate promo vs. high watermark checks

[PATCH 1/2] mm/vmscan: consolidate promo vs. high watermark checks

[Joshua Hahn]

should_abort_scan() and pgdat_balanced() both check the watermark that
kswapd reclaims zones up to, which is the promo watermark when NUMA
balancing memory tiering is enabled, and the high watermark otherwise.

Consolidate the pattern into a helper function.

No functional changes intended.

Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/internal.h |  1 +
 mm/vmscan.c   | 22 +++++++++++++---------
 2 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 78a7f17b9934f..66495845ec058 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -68,6 +68,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 				gfp_t gfp_mask, const nodemask_t *mask);
 unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
 			      int zone_idx);
+unsigned long kswapd_wmark_pages(const struct zone *zone);
 
 #define MEMCG_RECLAIM_MAY_SWAP (1 << 1)
 #define MEMCG_RECLAIM_PROACTIVE (1 << 2)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 071a6eaef1a1a..97c5843e14f15 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -371,6 +371,17 @@ unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone
 	return size;
 }
 
+/*
+ * The watermark kswapd reclaims zones up to. With NUMA balancing memory
+ * tiering, kswapd keeps room above the high watermark for promotions.
+ */
+unsigned long kswapd_wmark_pages(const struct zone *zone)
+{
+	if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
+		return promo_wmark_pages(zone);
+	return high_wmark_pages(zone);
+}
+
 static unsigned long drop_slab_node(int nid)
 {
 	unsigned long freed = 0;
@@ -5049,7 +5060,6 @@ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,
 static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)
 {
 	int i;
-	enum zone_watermarks mark;
 
 	if (unlikely(sc->proactive && signal_pending(current)))
 		return true;
@@ -5061,12 +5071,9 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc)
 	if (!current_is_kswapd() || sc->order)
 		return false;
 
-	mark = sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING ?
-	       WMARK_PROMO : WMARK_HIGH;
-
 	for (i = 0; i <= sc->reclaim_idx; i++) {
 		struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i;
-		unsigned long size = wmark_pages(zone, mark) + MIN_LRU_BATCH;
+		unsigned long size = kswapd_wmark_pages(zone) + MIN_LRU_BATCH;
 
 		if (managed_zone(zone) && !zone_watermark_ok(zone, 0, size, sc->reclaim_idx, 0))
 			return false;
@@ -6966,10 +6973,7 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)
 		enum zone_stat_item item;
 		unsigned long free_pages;
 
-		if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
-			mark = promo_wmark_pages(zone);
-		else
-			mark = high_wmark_pages(zone);
+		mark = kswapd_wmark_pages(zone);
 
 		/*
 		 * In defrag_mode, watermarks must be met in whole
-- 
2.53.0-Meta

[PATCH 2/2] mm/page_alloc: only skip kswapd nodes below their kswapd watermark

[Joshua Hahn]

Commit 19c5fb83f2a4 ("mm: page_alloc: avoid kswapd thrashing due to
NUMA restrictions") makes get_page_from_freelist() skip any node whose
kswapd is active to avoid piling on an already-pressured node that is
constantly churning to make free space that is immediately consumed.

But kswapd can also be woken up for order 9 requests with THP defrag
set to defer. This means that a fragmented node with plenty of free
memory will have a kswapd thread continuously spinning and preventing
order 0 pages from allocating on the node. This causes the opposite
problem of piling on pressure on all other nodes.

This behavior was observed on some Meta production workloads running on
2-NUMA systems, where processes with a preference for node0 allocations
continuously allocated order 0 pages on node1, despite node0 having
hundreds of GB of free memory.

To prevent piling pressure on any one node, only skip a node during
page allocation if the zone also fails the watermark kswapd is
reclaiming it up to. This is the high watermark, or the promo watermark
if NUMA balancing memory tiering is enabled. Make sure to mask the
reserve flags so that __GFP_HIGH / atomic requests in the slowpath
do not get a relaxed version of the check.

This preserves the existing fix to prevent piling memory on the local
node, while also preventing memory pileups on non-local nodes when
the local node has memory.

On a 2-NUMA VM with a 60G node0 and a 20G node1, with THP enabled=always
and defrag=defer, I ran a process that prefers allocating on node0.
With a fragmented node0:
- Before the patch, 66% of faults were served from node0
- After the patch, 100% of faults were served from node0

Suggested-by: Daniel Hodges <hodgesd@meta.com>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
 mm/page_alloc.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5423ff4805f84..8e8a2c8d6e6a4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3882,14 +3882,18 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 		}
 
 		/*
-		 * If kswapd is already active on a node, keep looking
-		 * for other nodes that might be idle. This can happen
-		 * if another process has NUMA bindings and is causing
-		 * kswapd wakeups on only some nodes. Avoid accidental
-		 * "node_reclaim_mode"-like behavior in this case.
+		 * If kswapd is already active on a node and the zone is below
+		 * its kswapd watermark for this request, keep looking for other
+		 * nodes that might be idle. This can happen if another process
+		 * has NUMA bindings and is causing kswapd wakeups on only some
+		 * nodes. Avoid accidental "node_reclaim_mode" behavior.
 		 */
 		if (skip_kswapd_nodes &&
-		    !waitqueue_active(&zone->zone_pgdat->kswapd_wait)) {
+		    !waitqueue_active(&zone->zone_pgdat->kswapd_wait) &&
+		    !zone_watermark_fast(zone, order, kswapd_wmark_pages(zone),
+					 ac->highest_zoneidx,
+					 alloc_flags & ~ALLOC_RESERVES,
+					 gfp_mask)) {
 			skipped_kswapd_nodes = true;
 			continue;
 		}
-- 
2.53.0-Meta

Re: [PATCH 1/2] mm/vmscan: consolidate promo vs. high watermark checks

[Lorenzo Stoakes (ARM)]

Hi Joshua.

Please send patch series of two patches or greater with a cover letter,
with each email sent in-reply-to the cover letter.

The easiest way of accomplishing this (+ our preference) is to use
b4 [0], otherwise format patches like this:

$ git format-patch --cover-letter --thread ...

(You've got 22 patches landed in mm, I shouldn't need to say this :)

[0]: https://b4.docs.kernel.org/en/latest/contributor/send.html

--
Cheers, Lorenzo

Re: [PATCH 2/2] mm/page_alloc: only skip kswapd nodes below their kswapd watermark

[Lorenzo Stoakes (ARM)]

On Fri, Nov 06, 2026 at 09:26:19AM -0800, Joshua Hahn wrote:
> Commit 19c5fb83f2a4 ("mm: page_alloc: avoid kswapd thrashing due to
> NUMA restrictions") makes get_page_from_freelist() skip any node whose
> kswapd is active to avoid piling on an already-pressured node that is
> constantly churning to make free space that is immediately consumed.
>
> But kswapd can also be woken up for order 9 requests with THP defrag
> set to defer. This means that a fragmented node with plenty of free
> memory will have a kswapd thread continuously spinning and preventing
> order 0 pages from allocating on the node. This causes the opposite
> problem of piling on pressure on all other nodes.
>
> This behavior was observed on some Meta production workloads running on
> 2-NUMA systems, where processes with a preference for node0 allocations
> continuously allocated order 0 pages on node1, despite node0 having
> hundreds of GB of free memory.
>
> To prevent piling pressure on any one node, only skip a node during
> page allocation if the zone also fails the watermark kswapd is
> reclaiming it up to. This is the high watermark, or the promo watermark
> if NUMA balancing memory tiering is enabled. Make sure to mask the
> reserve flags so that __GFP_HIGH / atomic requests in the slowpath
> do not get a relaxed version of the check.
>
> This preserves the existing fix to prevent piling memory on the local
> node, while also preventing memory pileups on non-local nodes when
> the local node has memory.
>
> On a 2-NUMA VM with a 60G node0 and a 20G node1, with THP enabled=always
> and defrag=defer, I ran a process that prefers allocating on node0.
> With a fragmented node0:
> - Before the patch, 66% of faults were served from node0
> - After the patch, 100% of faults were served from node0
>
> Suggested-by: Daniel Hodges <hodgesd@meta.com>
> Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> ---
>  mm/page_alloc.c | 16 ++++++++++------
>  1 file changed, 10 insertions(+), 6 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 5423ff4805f84..8e8a2c8d6e6a4 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3882,14 +3882,18 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
>  		}
>
>  		/*
> -		 * If kswapd is already active on a node, keep looking
> -		 * for other nodes that might be idle. This can happen
> -		 * if another process has NUMA bindings and is causing
> -		 * kswapd wakeups on only some nodes. Avoid accidental
> -		 * "node_reclaim_mode"-like behavior in this case.
> +		 * If kswapd is already active on a node and the zone is below
> +		 * its kswapd watermark for this request, keep looking for other
> +		 * nodes that might be idle. This can happen if another process
> +		 * has NUMA bindings and is causing kswapd wakeups on only some
> +		 * nodes. Avoid accidental "node_reclaim_mode" behavior.
>  		 */
>  		if (skip_kswapd_nodes &&
> -		    !waitqueue_active(&zone->zone_pgdat->kswapd_wait)) {
> +		    !waitqueue_active(&zone->zone_pgdat->kswapd_wait) &&
> +		    !zone_watermark_fast(zone, order, kswapd_wmark_pages(zone),
> +					 ac->highest_zoneidx,
> +					 alloc_flags & ~ALLOC_RESERVES,
> +					 gfp_mask)) {

I know this code is hardly the cleanest, but you're now adding a 3rd condition
to a branch that was already complicated enough to have a massive comment and
duplicating a block of code that already dupliclated twice here.

You're also then going against the convention of

	mark = ...
	if (... !zone_watermark_fast(zone, order, mark,
					 ac->highest_zoneidx, alloc_flags,
					 gfp_mask)) {

(Obviously yours varies on alloc_flags too)

And also inlining the mark value.

Could you clean this up a little given this isn't a fix?

In general extending functionality should pay down some technical debt if
possible, it's ok to add a preceding patch in the series to do something like
that.

>  			skipped_kswapd_nodes = true;
>  			continue;
>  		}
> --
> 2.53.0-Meta
>

--
Cheers, Lorenzo

Re: [PATCH 1/2] mm/vmscan: consolidate promo vs. high watermark checks

[Joshua Hahn]

On Fri, Nov 6, 2026 at 12:42 PM Lorenzo Stoakes (ARM) <ljs@kernel.org> wrote:
>
> Hi Joshua.
>
> Please send patch series of two patches or greater with a cover letter,
> with each email sent in-reply-to the cover letter.
>
> The easiest way of accomplishing this (+ our preference) is to use
> b4 [0], otherwise format patches like this:
>
> $ git format-patch --cover-letter --thread ...
>
> (You've got 22 patches landed in mm, I shouldn't need to say this :)

Hi Lorenzo,

You're right ; -) If this series gets a v2, I'll be sure to add a
cover letter. I just wasn't sure what I would include in it that
wasn't covered in the commit messages themselves, but I think I could
offer some more explanation of how this was discovered and why there's
no CC: stable in the patches.

Thanks! Have a great day Lorenzo,
Joshua

> [0]: https://b4.docs.kernel.org/en/latest/contributor/send.html
>
> --
> Cheers, Lorenzo

Re: [PATCH 2/2] mm/page_alloc: only skip kswapd nodes below their kswapd watermark

[Joshua Hahn]

On Fri, Nov 6, 2026 at 12:50 PM Lorenzo Stoakes (ARM) <ljs@kernel.org> wrote:
>
> On Fri, Nov 06, 2026 at 09:26:19AM -0800, Joshua Hahn wrote:
> > Commit 19c5fb83f2a4 ("mm: page_alloc: avoid kswapd thrashing due to
> > NUMA restrictions") makes get_page_from_freelist() skip any node whose
> > kswapd is active to avoid piling on an already-pressured node that is
> > constantly churning to make free space that is immediately consumed.
> >
> > But kswapd can also be woken up for order 9 requests with THP defrag
> > set to defer. This means that a fragmented node with plenty of free
> > memory will have a kswapd thread continuously spinning and preventing
> > order 0 pages from allocating on the node. This causes the opposite
> > problem of piling on pressure on all other nodes.
> >
> > This behavior was observed on some Meta production workloads running on
> > 2-NUMA systems, where processes with a preference for node0 allocations
> > continuously allocated order 0 pages on node1, despite node0 having
> > hundreds of GB of free memory.
> >
> > To prevent piling pressure on any one node, only skip a node during
> > page allocation if the zone also fails the watermark kswapd is
> > reclaiming it up to. This is the high watermark, or the promo watermark
> > if NUMA balancing memory tiering is enabled. Make sure to mask the
> > reserve flags so that __GFP_HIGH / atomic requests in the slowpath
> > do not get a relaxed version of the check.
> >
> > This preserves the existing fix to prevent piling memory on the local
> > node, while also preventing memory pileups on non-local nodes when
> > the local node has memory.
> >
> > On a 2-NUMA VM with a 60G node0 and a 20G node1, with THP enabled=always
> > and defrag=defer, I ran a process that prefers allocating on node0.
> > With a fragmented node0:
> > - Before the patch, 66% of faults were served from node0
> > - After the patch, 100% of faults were served from node0
> >
> > Suggested-by: Daniel Hodges <hodgesd@meta.com>
> > Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> > ---
> >  mm/page_alloc.c | 16 ++++++++++------
> >  1 file changed, 10 insertions(+), 6 deletions(-)
> >
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index 5423ff4805f84..8e8a2c8d6e6a4 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -3882,14 +3882,18 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
> >               }
> >
> >               /*
> > -              * If kswapd is already active on a node, keep looking
> > -              * for other nodes that might be idle. This can happen
> > -              * if another process has NUMA bindings and is causing
> > -              * kswapd wakeups on only some nodes. Avoid accidental
> > -              * "node_reclaim_mode"-like behavior in this case.
> > +              * If kswapd is already active on a node and the zone is below
> > +              * its kswapd watermark for this request, keep looking for other
> > +              * nodes that might be idle. This can happen if another process
> > +              * has NUMA bindings and is causing kswapd wakeups on only some
> > +              * nodes. Avoid accidental "node_reclaim_mode" behavior.
> >                */
> >               if (skip_kswapd_nodes &&
> > -                 !waitqueue_active(&zone->zone_pgdat->kswapd_wait)) {
> > +                 !waitqueue_active(&zone->zone_pgdat->kswapd_wait) &&
> > +                 !zone_watermark_fast(zone, order, kswapd_wmark_pages(zone),
> > +                                      ac->highest_zoneidx,
> > +                                      alloc_flags & ~ALLOC_RESERVES,
> > +                                      gfp_mask)) {
>
> I know this code is hardly the cleanest, but you're now adding a 3rd condition
> to a branch that was already complicated enough to have a massive comment and
> duplicating a block of code that already dupliclated twice here.
>
> You're also then going against the convention of
>
>         mark = ...
>         if (... !zone_watermark_fast(zone, order, mark,
>                                          ac->highest_zoneidx, alloc_flags,
>                                          gfp_mask)) {
>
> (Obviously yours varies on alloc_flags too)
>
> And also inlining the mark value.
>
> Could you clean this up a little given this isn't a fix?

Hi Lorenzo,

Thanks for taking a look at the patches! I think your requests are
valid here, let me fold that whole check into a helper function
including the comment block and hopefully that will make
get_page_from_freelist() a bit easier to read @_@. I also found an
unused gfp argument anyways, so I'll fold in some related cleanups and
send a new series.

I'll wait until tomorrow to send it out to see if other folks have
concerns with this. Thanks again, have a great day!
Joshua

> In general extending functionality should pay down some technical debt if
> possible, it's ok to add a preceding patch in the series to do something like
> that.
>
> >                       skipped_kswapd_nodes = true;
> >                       continue;
> >               }
> > --
> > 2.53.0-Meta
> >
>
> --
> Cheers, Lorenzo