[PATCH v2] mm/page_alloc: only skip kswapd nodes below their kswapd watermark

[Joshua Hahn <joshua.hahnjy@gmail.com>]

Commit 19c5fb83f2a4 ("mm: page_alloc: avoid kswapd thrashing due to
NUMA restrictions") makes get_page_from_freelist() skip any node with an
active kswapd on the first pass through the zonelist, so unrestricted
allocations don't immediately consume the memory kswapd is freeing.

But kswapd is also woken for high-order requests. With THP defrag set to
defer, each THP fault that fails on a fragmented node wakes kswapd at
PMD order. kswapd reclaims a batch, wakes kcompactd, and sleeps until
the next failed fault, so it stays awake much of the time even on a node
with plenty of free order-0 memory. This means those order-0 allocations
get directed elsewhere.

This was observed on some Meta production workloads running on 2-NUMA
systems, where processes with a preference for node0 allocations
continuously allocated order-0 pages on node1, despite node0 having
hundreds of GB of free memory.

Only skip a zone with an active kswapd if it is also below kswapd's
target watermark.

On a 2-NUMA VM with a 60G node0 and a 20G node1, with THP enabled=always
and defrag=defer, I ran a process that prefers allocating on node0.
With a fragmented node0:
- Before the patch, 68% of pages were served from node0
- After the patch, 100% of pages were served from node0

One remaining gap is that since this uses kswapd's own threshold,
unrestricted allocations can take what kswapd frees above the target
watermark. The current code already lets such allocations drain from the
high to the low watermark while kswapd sleeps.

One potential solution, as hinted by Johannes in the discussion from the
original fix [1] is to add some kind of fudge factor (for instance,
wmark_pages + high_wmark_pages - low_wmark_pages).

But in tests that I ran in a VM where the allocation demand outpaces
kswapd, about 4% more pages landed on the busy node and kswapd reclaimed
about 15% more, but kswapd still finished and slept. So let's hold off
on the fudge factor until it becomes an observable issue in workloads.

[1] https://lore.kernel.org/linux-mm/20251001164131.GB1597553@cmpxchg.org/

Fixes: 19c5fb83f2a4 ("mm: page_alloc: avoid kswapd thrashing due to NUMA restrictions")
Suggested-by: Daniel Hodges <hodgesd@meta.com>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
v1 --> v2:
- Added a Fixes tag and separated this from the other cleanup patches
- Check zone_watermark_fast with no flag to match the call in
  pgdat_balanced().
- Calculates mark outside the zone_watermark_fast check to match the
  style for the other zone checks.
- Wordsmithing commit message to explain the unaddressed gap.
- Yes, this is ugly. I have another series to clean this code up but
  I wanted this to be a feature-minimal patch so that it could be
  easily backported. Things will be much prettier once those are landed.

 mm/page_alloc.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8532af052d74b..56121c3f9c4a7 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -45,6 +45,7 @@
 #include <linux/mmu_notifier.h>
 #include <linux/migrate.h>
 #include <linux/sched/mm.h>
+#include <linux/sched/sysctl.h>
 #include <linux/page_owner.h>
 #include <linux/page_table_check.h>
 #include <linux/memcontrol.h>
@@ -3799,6 +3800,9 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 	struct zoneref *z;
 	struct zone *zone;
 	struct pglist_data *last_pgdat = NULL;
+	enum zone_watermarks kswapd_watermark =
+		sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING ?
+		WMARK_PROMO : WMARK_HIGH;
 	bool last_pgdat_dirty_ok = false;
 	bool no_fallback;
 	bool skip_kswapd_nodes = nr_online_nodes > 1;
@@ -3866,16 +3870,21 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 		}
 
 		/*
-		 * If kswapd is already active on a node, keep looking
-		 * for other nodes that might be idle. This can happen
-		 * if another process has NUMA bindings and is causing
-		 * kswapd wakeups on only some nodes. Avoid accidental
-		 * "node_reclaim_mode"-like behavior in this case.
+		 * If kswapd is already active on a node and the zone is below
+		 * its kswapd watermark for this request, keep looking for
+		 * other nodes that might be idle. This can happen if another
+		 * process has NUMA bindings and is causing kswapd wakeups on
+		 * only some nodes. Avoid accidental "node_reclaim_mode"-like
+		 * behavior in this case.
 		 */
 		if (skip_kswapd_nodes &&
 		    !waitqueue_active(&zone->zone_pgdat->kswapd_wait)) {
-			skipped_kswapd_nodes = true;
-			continue;
+			mark = wmark_pages(zone, kswapd_watermark);
+			if (!zone_watermark_fast(zone, order, mark,
+					ac->highest_zoneidx, 0, gfp_mask)) {
+				skipped_kswapd_nodes = true;
+				continue;
+			}
 		}
 
 		cond_accept_memory(zone, order, alloc_flags);
-- 
2.53.0-Meta