[PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Vlastimil Babka (SUSE)]

A customer has reported excessive reclaim activity despite memory above
high watermarks. Analyzing support data (including sar log) revealed the
following scenario:

- the system has almost all memory reserved as 1G hugepages for kvm
  guests, and operates normally
- due to a spike of memory pressure from userspace, highatomic
  reservations occur (probably due to e.g. concurrent network receive
  activity)
- on one of the 8 NUMA nodes, this creates a situation where free pages
  are above high watermark, but when __zone_watermark_unusable_free()
  considers highatomic pageblocks, the result appears to be below the
  low watermark
- pgdat_balanced() is false, kswapd hopelessly keeps trying to balance,
  spending cpu time and disrupting the system. It has nothing to reclaim
  because the node is filled with hugetlbfs hugepages.
- since the overal memory pressure is under control again and there are
  no allocations restricted to the affected node, they are satisfied
  from other nodes without going through direct reclaim
- since direct reclaim is the only code that calls
  unreserve_highatomic_pageblock(), this never happens and nothing fixes
  the situation

Note this was observed on a 6.4-based kernel and there have been recent
fixes upstream limiting the highatomic reservation side, but I don't
think they can prevent the same happening in mainline, only perhaps
lower the chances. Nothing seems to prevent the issue completely.

For example, while we limit the amount of highatomic pageblocks to 1% of
the zone, this calculation does not subtract the amount of hugepages or
otherwise permanent allocations, so the highatomic pageblocks can end up
taking much higher share of actually reclaimable memory.

The most targeted fix thus appears to be to allow kswapd perform the
highatomic pageblock unreserve to match direct reclaim. It's also
somewhat similar in principle to how kswapd reduces watermark boosting.

Hook this highatomic unreserve to the moment we are increasing kswapd
reclaim priority, as that is when kswapd is struggling to make progress
and it should roughly match the case of direct reclaim failure.

The only caveat is that direct reclaim knows which migratetype it wants
to allocate and thus changes the highatomic pageblock to. Kswapd doesn't
track that, so use MIGRATE_UNMOVABLE. Highatomic allocations usually are
not movable, and this will avoid creating nominally movable pageblocks
polluted with unmovable pages, so it should be the least problematic
default.

Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
We interrupt the regular stream of AI fixes by something a bit
different. RFC to get agreement on the approach. Haven't tried
reproducing the issue in mainline, but it seems quite clear to me from
the data I've seen.
---
 mm/page_alloc.c | 126 +++++++++++++++++++++++++++++---------------------------
 mm/page_alloc.h |   3 ++
 mm/vmscan.c     |  16 ++++++-
 3 files changed, 83 insertions(+), 62 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a8cac5e01066..785a13570788 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3476,6 +3476,68 @@ static void reserve_highatomic_pageblock(struct page *page, int order,
 	}
 }
 
+bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
+					 bool force)
+{
+	/*
+	 * Preserve at least one pageblock unless memory pressure
+	 * is really high.
+	 */
+	if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages)
+		return false;
+
+	guard(spinlock_irqsave)(&zone->lock);
+	for (int order = 0; order < NR_PAGE_ORDERS; order++) {
+		struct free_area *area = &(zone->free_area[order]);
+		unsigned long size;
+		struct page *page;
+		int ret;
+
+		page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
+		if (!page)
+			continue;
+
+		size = max(pageblock_nr_pages, 1UL << order);
+		/*
+		 * It should never happen but changes to locking could
+		 * inadvertently allow a per-cpu drain to add pages to
+		 * MIGRATE_HIGHATOMIC while unreserving so be safe and watch for
+		 * underflows.
+		 */
+		if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
+			size = zone->nr_reserved_highatomic;
+		zone->nr_reserved_highatomic -= size;
+
+		/*
+		 * Convert to migratetype and avoid the normal pageblock
+		 * stealing heuristics. Minimally, the caller is doing the work
+		 * and needs the pages. More importantly, if the block was
+		 * always converted to MIGRATE_UNMOVABLE or another type then
+		 * the number of pageblocks that cannot be completely freed may
+		 * increase.
+		 */
+		if (order < pageblock_order)
+			ret = move_freepages_block(zone, page,
+					MIGRATE_HIGHATOMIC, migratetype);
+		else {
+			move_to_free_list(page, zone, order,
+					  MIGRATE_HIGHATOMIC,
+					  migratetype);
+			change_pageblock_range(page, order, migratetype);
+			ret = 1;
+		}
+		/*
+		 * Reserving the block(s) already succeeded,
+		 * so this should not fail on zone boundaries.
+		 */
+		WARN_ON_ONCE(ret == -1);
+		if (ret > 0)
+			return true;
+	}
+
+	return false;
+}
+
 /*
  * Used when an allocation is about to fail under memory pressure. This
  * potentially hurts the reliability of high-order allocations when under
@@ -3491,70 +3553,12 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,
 	struct zonelist *zonelist = ac->zonelist;
 	struct zoneref *z;
 	struct zone *zone;
-	struct page *page;
-	int order;
-	int ret;
 
 	for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx,
 								ac->nodemask) {
-		/*
-		 * Preserve at least one pageblock unless memory pressure
-		 * is really high.
-		 */
-		if (!force && zone->nr_reserved_highatomic <=
-					pageblock_nr_pages)
-			continue;
-
-		guard(spinlock_irqsave)(&zone->lock);
-		for (order = 0; order < NR_PAGE_ORDERS; order++) {
-			struct free_area *area = &(zone->free_area[order]);
-			unsigned long size;
-
-			page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
-			if (!page)
-				continue;
-
-			size = max(pageblock_nr_pages, 1UL << order);
-			/*
-			 * It should never happen but changes to
-			 * locking could inadvertently allow a per-cpu
-			 * drain to add pages to MIGRATE_HIGHATOMIC
-			 * while unreserving so be safe and watch for
-			 * underflows.
-			 */
-			if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
-				size = zone->nr_reserved_highatomic;
-			zone->nr_reserved_highatomic -= size;
-
-			/*
-			 * Convert to ac->migratetype and avoid the normal
-			 * pageblock stealing heuristics. Minimally, the caller
-			 * is doing the work and needs the pages. More
-			 * importantly, if the block was always converted to
-			 * MIGRATE_UNMOVABLE or another type then the number
-			 * of pageblocks that cannot be completely freed
-			 * may increase.
-			 */
-			if (order < pageblock_order)
-				ret = move_freepages_block(zone, page,
-							   MIGRATE_HIGHATOMIC,
-							   ac->migratetype);
-			else {
-				move_to_free_list(page, zone, order,
-						  MIGRATE_HIGHATOMIC,
-						  ac->migratetype);
-				change_pageblock_range(page, order,
-						       ac->migratetype);
-				ret = 1;
-			}
-			/*
-			 * Reserving the block(s) already succeeded,
-			 * so this should not fail on zone boundaries.
-			 */
-			WARN_ON_ONCE(ret == -1);
-			if (ret > 0)
-				return ret;
-		}
+		if (unreserve_highatomic_pageblock_zone(zone,
+							ac->migratetype, force))
+			return true;
 	}
 
 	return false;
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index e238ee91ed24..9fba6480b18d 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -282,6 +282,9 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)
 	return list_empty(&area->free_list[migratetype]);
 }
 
+bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
+					 bool force);
+
 /* Convert GFP flags to their corresponding migrate type */
 #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
 #define GFP_MOVABLE_SHIFT 3
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..91c622068d99 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
 		if (nr_boost_reclaim && !nr_reclaimed)
 			break;
 
-		if (raise_priority || !nr_reclaimed)
+		if (raise_priority || !nr_reclaimed) {
+			/*
+			 * Similarly to when direct reclaim is retrying, try to
+			 * unreserve highatomic pageblocks on the node. Convert
+			 * them to movable ones as kswapd doesn't have a target
+			 * migratetype, and passing unmovable should lead to
+			 * least permanent fragmentation.
+			 */
+			for (int i = 0; i <= highest_zoneidx; i++) {
+				struct zone *zone = pgdat->node_zones + i;
+
+				unreserve_highatomic_pageblock_zone(zone,
+						MIGRATE_UNMOVABLE, false);
+			}
 			sc.priority--;
+		}
 	} while (sc.priority >= 1);
 
 	/*

---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261030-kswapd-highatomic-unreserve-7e6629ec65d0

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Thadeu Lima de Souza Cascardo]

On Wed, Nov 04, 2026 at 07:38:44PM +0100, Vlastimil Babka (SUSE) wrote:
> A customer has reported excessive reclaim activity despite memory above
> high watermarks. Analyzing support data (including sar log) revealed the
> following scenario:
> 
> - the system has almost all memory reserved as 1G hugepages for kvm
>   guests, and operates normally
> - due to a spike of memory pressure from userspace, highatomic
>   reservations occur (probably due to e.g. concurrent network receive
>   activity)

I think you might mean 6a204d4b14c9 ("mm/page_alloc: only set
ALLOC_HIGHATOMIC for __GPF_HIGH allocations") as one of the fixes you
mention below.

In that case, GFP_NOWAIT (which is typical for network activity, AFAIR)
would be one candidate that would produce such reservations. But I agree
that if any other real highatomic allocation burst would happen, it might
produce such reservations.

> - on one of the 8 NUMA nodes, this creates a situation where free pages
>   are above high watermark, but when __zone_watermark_unusable_free()
>   considers highatomic pageblocks, the result appears to be below the
>   low watermark
> - pgdat_balanced() is false, kswapd hopelessly keeps trying to balance,
>   spending cpu time and disrupting the system. It has nothing to reclaim
>   because the node is filled with hugetlbfs hugepages.
> - since the overal memory pressure is under control again and there are
>   no allocations restricted to the affected node, they are satisfied
>   from other nodes without going through direct reclaim
> - since direct reclaim is the only code that calls
>   unreserve_highatomic_pageblock(), this never happens and nothing fixes
>   the situation
> 

I am trying to understand what is going on here: so allocations on that
node are only !__GFP_DIRECT_RECLAIM? That would explain the highatomic
reservations in the presence of the bug just mentioned.

> Note this was observed on a 6.4-based kernel and there have been recent
> fixes upstream limiting the highatomic reservation side, but I don't
> think they can prevent the same happening in mainline, only perhaps
> lower the chances. Nothing seems to prevent the issue completely.
> 
> For example, while we limit the amount of highatomic pageblocks to 1% of
> the zone, this calculation does not subtract the amount of hugepages or
> otherwise permanent allocations, so the highatomic pageblocks can end up
> taking much higher share of actually reclaimable memory.
> 
> The most targeted fix thus appears to be to allow kswapd perform the
> highatomic pageblock unreserve to match direct reclaim. It's also
> somewhat similar in principle to how kswapd reduces watermark boosting.
> 
> Hook this highatomic unreserve to the moment we are increasing kswapd
> reclaim priority, as that is when kswapd is struggling to make progress
> and it should roughly match the case of direct reclaim failure.
> 

I tried considering a different approach, making the allocation do the
unreserve in the absence of __GFP_DIRECT_RECLAIM, but I don't see why not
let kswapd do it.

Regards.
Cascardo.

> The only caveat is that direct reclaim knows which migratetype it wants
> to allocate and thus changes the highatomic pageblock to. Kswapd doesn't
> track that, so use MIGRATE_UNMOVABLE. Highatomic allocations usually are
> not movable, and this will avoid creating nominally movable pageblocks
> polluted with unmovable pages, so it should be the least problematic
> default.
> 
> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> ---
> We interrupt the regular stream of AI fixes by something a bit
> different. RFC to get agreement on the approach. Haven't tried
> reproducing the issue in mainline, but it seems quite clear to me from
> the data I've seen.
> ---
>  mm/page_alloc.c | 126 +++++++++++++++++++++++++++++---------------------------
>  mm/page_alloc.h |   3 ++
>  mm/vmscan.c     |  16 ++++++-
>  3 files changed, 83 insertions(+), 62 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a8cac5e01066..785a13570788 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3476,6 +3476,68 @@ static void reserve_highatomic_pageblock(struct page *page, int order,
>  	}
>  }
>  
> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
> +					 bool force)
> +{
> +	/*
> +	 * Preserve at least one pageblock unless memory pressure
> +	 * is really high.
> +	 */
> +	if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages)
> +		return false;
> +
> +	guard(spinlock_irqsave)(&zone->lock);
> +	for (int order = 0; order < NR_PAGE_ORDERS; order++) {
> +		struct free_area *area = &(zone->free_area[order]);
> +		unsigned long size;
> +		struct page *page;
> +		int ret;
> +
> +		page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
> +		if (!page)
> +			continue;
> +
> +		size = max(pageblock_nr_pages, 1UL << order);
> +		/*
> +		 * It should never happen but changes to locking could
> +		 * inadvertently allow a per-cpu drain to add pages to
> +		 * MIGRATE_HIGHATOMIC while unreserving so be safe and watch for
> +		 * underflows.
> +		 */
> +		if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
> +			size = zone->nr_reserved_highatomic;
> +		zone->nr_reserved_highatomic -= size;
> +
> +		/*
> +		 * Convert to migratetype and avoid the normal pageblock
> +		 * stealing heuristics. Minimally, the caller is doing the work
> +		 * and needs the pages. More importantly, if the block was
> +		 * always converted to MIGRATE_UNMOVABLE or another type then
> +		 * the number of pageblocks that cannot be completely freed may
> +		 * increase.
> +		 */
> +		if (order < pageblock_order)
> +			ret = move_freepages_block(zone, page,
> +					MIGRATE_HIGHATOMIC, migratetype);
> +		else {
> +			move_to_free_list(page, zone, order,
> +					  MIGRATE_HIGHATOMIC,
> +					  migratetype);
> +			change_pageblock_range(page, order, migratetype);
> +			ret = 1;
> +		}
> +		/*
> +		 * Reserving the block(s) already succeeded,
> +		 * so this should not fail on zone boundaries.
> +		 */
> +		WARN_ON_ONCE(ret == -1);
> +		if (ret > 0)
> +			return true;
> +	}
> +
> +	return false;
> +}
> +
>  /*
>   * Used when an allocation is about to fail under memory pressure. This
>   * potentially hurts the reliability of high-order allocations when under
> @@ -3491,70 +3553,12 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,
>  	struct zonelist *zonelist = ac->zonelist;
>  	struct zoneref *z;
>  	struct zone *zone;
> -	struct page *page;
> -	int order;
> -	int ret;
>  
>  	for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx,
>  								ac->nodemask) {
> -		/*
> -		 * Preserve at least one pageblock unless memory pressure
> -		 * is really high.
> -		 */
> -		if (!force && zone->nr_reserved_highatomic <=
> -					pageblock_nr_pages)
> -			continue;
> -
> -		guard(spinlock_irqsave)(&zone->lock);
> -		for (order = 0; order < NR_PAGE_ORDERS; order++) {
> -			struct free_area *area = &(zone->free_area[order]);
> -			unsigned long size;
> -
> -			page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
> -			if (!page)
> -				continue;
> -
> -			size = max(pageblock_nr_pages, 1UL << order);
> -			/*
> -			 * It should never happen but changes to
> -			 * locking could inadvertently allow a per-cpu
> -			 * drain to add pages to MIGRATE_HIGHATOMIC
> -			 * while unreserving so be safe and watch for
> -			 * underflows.
> -			 */
> -			if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
> -				size = zone->nr_reserved_highatomic;
> -			zone->nr_reserved_highatomic -= size;
> -
> -			/*
> -			 * Convert to ac->migratetype and avoid the normal
> -			 * pageblock stealing heuristics. Minimally, the caller
> -			 * is doing the work and needs the pages. More
> -			 * importantly, if the block was always converted to
> -			 * MIGRATE_UNMOVABLE or another type then the number
> -			 * of pageblocks that cannot be completely freed
> -			 * may increase.
> -			 */
> -			if (order < pageblock_order)
> -				ret = move_freepages_block(zone, page,
> -							   MIGRATE_HIGHATOMIC,
> -							   ac->migratetype);
> -			else {
> -				move_to_free_list(page, zone, order,
> -						  MIGRATE_HIGHATOMIC,
> -						  ac->migratetype);
> -				change_pageblock_range(page, order,
> -						       ac->migratetype);
> -				ret = 1;
> -			}
> -			/*
> -			 * Reserving the block(s) already succeeded,
> -			 * so this should not fail on zone boundaries.
> -			 */
> -			WARN_ON_ONCE(ret == -1);
> -			if (ret > 0)
> -				return ret;
> -		}
> +		if (unreserve_highatomic_pageblock_zone(zone,
> +							ac->migratetype, force))
> +			return true;
>  	}
>  
>  	return false;
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index e238ee91ed24..9fba6480b18d 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -282,6 +282,9 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)
>  	return list_empty(&area->free_list[migratetype]);
>  }
>  
> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
> +					 bool force);
> +
>  /* Convert GFP flags to their corresponding migrate type */
>  #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
>  #define GFP_MOVABLE_SHIFT 3
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 9959ca158ce8..91c622068d99 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
>  		if (nr_boost_reclaim && !nr_reclaimed)
>  			break;
>  
> -		if (raise_priority || !nr_reclaimed)
> +		if (raise_priority || !nr_reclaimed) {
> +			/*
> +			 * Similarly to when direct reclaim is retrying, try to
> +			 * unreserve highatomic pageblocks on the node. Convert
> +			 * them to movable ones as kswapd doesn't have a target
> +			 * migratetype, and passing unmovable should lead to
> +			 * least permanent fragmentation.
> +			 */
> +			for (int i = 0; i <= highest_zoneidx; i++) {
> +				struct zone *zone = pgdat->node_zones + i;
> +
> +				unreserve_highatomic_pageblock_zone(zone,
> +						MIGRATE_UNMOVABLE, false);
> +			}
>  			sc.priority--;
> +		}
>  	} while (sc.priority >= 1);
>  
>  	/*
> 
> ---
> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
> change-id: 20261030-kswapd-highatomic-unreserve-7e6629ec65d0
> 

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Vlastimil Babka (SUSE)]

On 11/4/26 21:03, Thadeu Lima de Souza Cascardo wrote:
> On Wed, Nov 04, 2026 at 07:38:44PM +0100, Vlastimil Babka (SUSE) wrote:
>> A customer has reported excessive reclaim activity despite memory above
>> high watermarks. Analyzing support data (including sar log) revealed the
>> following scenario:
>> 
>> - the system has almost all memory reserved as 1G hugepages for kvm
>>   guests, and operates normally
>> - due to a spike of memory pressure from userspace, highatomic
>>   reservations occur (probably due to e.g. concurrent network receive
>>   activity)
> 
> I think you might mean 6a204d4b14c9 ("mm/page_alloc: only set
> ALLOC_HIGHATOMIC for __GPF_HIGH allocations") as one of the fixes you
> mention below.

Indeed.

> In that case, GFP_NOWAIT (which is typical for network activity, AFAIR)
> would be one candidate that would produce such reservations. But I agree
> that if any other real highatomic allocation burst would happen, it might
> produce such reservations.

Ack.

>> - on one of the 8 NUMA nodes, this creates a situation where free pages
>>   are above high watermark, but when __zone_watermark_unusable_free()
>>   considers highatomic pageblocks, the result appears to be below the
>>   low watermark
>> - pgdat_balanced() is false, kswapd hopelessly keeps trying to balance,
>>   spending cpu time and disrupting the system. It has nothing to reclaim
>>   because the node is filled with hugetlbfs hugepages.
>> - since the overal memory pressure is under control again and there are
>>   no allocations restricted to the affected node, they are satisfied
>>   from other nodes without going through direct reclaim
>> - since direct reclaim is the only code that calls
>>   unreserve_highatomic_pageblock(), this never happens and nothing fixes
>>   the situation
>> 
> 
> I am trying to understand what is going on here: so allocations on that
> node are only !__GFP_DIRECT_RECLAIM? That would explain the highatomic
> reservations in the presence of the bug just mentioned.

They are probably __GFP_DIRECT_RECLAIM, but if they only origin on a cpu
local to the affected node but can fallback to any other (which is the usual
case) and zone_reclaim_mode is disabled (by default it is), they will just
succeed on another node and not attempt the direct reclaim.

>> Note this was observed on a 6.4-based kernel and there have been recent
>> fixes upstream limiting the highatomic reservation side, but I don't
>> think they can prevent the same happening in mainline, only perhaps
>> lower the chances. Nothing seems to prevent the issue completely.
>> 
>> For example, while we limit the amount of highatomic pageblocks to 1% of
>> the zone, this calculation does not subtract the amount of hugepages or
>> otherwise permanent allocations, so the highatomic pageblocks can end up
>> taking much higher share of actually reclaimable memory.
>> 
>> The most targeted fix thus appears to be to allow kswapd perform the
>> highatomic pageblock unreserve to match direct reclaim. It's also
>> somewhat similar in principle to how kswapd reduces watermark boosting.
>> 
>> Hook this highatomic unreserve to the moment we are increasing kswapd
>> reclaim priority, as that is when kswapd is struggling to make progress
>> and it should roughly match the case of direct reclaim failure.
>> 
> 
> I tried considering a different approach, making the allocation do the
> unreserve in the absence of __GFP_DIRECT_RECLAIM, but I don't see why not
> let kswapd do it.
> 
> Regards.
> Cascardo.
> 
>> The only caveat is that direct reclaim knows which migratetype it wants
>> to allocate and thus changes the highatomic pageblock to. Kswapd doesn't
>> track that, so use MIGRATE_UNMOVABLE. Highatomic allocations usually are
>> not movable, and this will avoid creating nominally movable pageblocks
>> polluted with unmovable pages, so it should be the least problematic
>> default.
>> 
>> Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>> ---
>> We interrupt the regular stream of AI fixes by something a bit
>> different. RFC to get agreement on the approach. Haven't tried
>> reproducing the issue in mainline, but it seems quite clear to me from
>> the data I've seen.
>> ---
>>  mm/page_alloc.c | 126 +++++++++++++++++++++++++++++---------------------------
>>  mm/page_alloc.h |   3 ++
>>  mm/vmscan.c     |  16 ++++++-
>>  3 files changed, 83 insertions(+), 62 deletions(-)
>> 
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index a8cac5e01066..785a13570788 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -3476,6 +3476,68 @@ static void reserve_highatomic_pageblock(struct page *page, int order,
>>  	}
>>  }
>>  
>> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
>> +					 bool force)
>> +{
>> +	/*
>> +	 * Preserve at least one pageblock unless memory pressure
>> +	 * is really high.
>> +	 */
>> +	if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages)
>> +		return false;
>> +
>> +	guard(spinlock_irqsave)(&zone->lock);
>> +	for (int order = 0; order < NR_PAGE_ORDERS; order++) {
>> +		struct free_area *area = &(zone->free_area[order]);
>> +		unsigned long size;
>> +		struct page *page;
>> +		int ret;
>> +
>> +		page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
>> +		if (!page)
>> +			continue;
>> +
>> +		size = max(pageblock_nr_pages, 1UL << order);
>> +		/*
>> +		 * It should never happen but changes to locking could
>> +		 * inadvertently allow a per-cpu drain to add pages to
>> +		 * MIGRATE_HIGHATOMIC while unreserving so be safe and watch for
>> +		 * underflows.
>> +		 */
>> +		if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
>> +			size = zone->nr_reserved_highatomic;
>> +		zone->nr_reserved_highatomic -= size;
>> +
>> +		/*
>> +		 * Convert to migratetype and avoid the normal pageblock
>> +		 * stealing heuristics. Minimally, the caller is doing the work
>> +		 * and needs the pages. More importantly, if the block was
>> +		 * always converted to MIGRATE_UNMOVABLE or another type then
>> +		 * the number of pageblocks that cannot be completely freed may
>> +		 * increase.
>> +		 */
>> +		if (order < pageblock_order)
>> +			ret = move_freepages_block(zone, page,
>> +					MIGRATE_HIGHATOMIC, migratetype);
>> +		else {
>> +			move_to_free_list(page, zone, order,
>> +					  MIGRATE_HIGHATOMIC,
>> +					  migratetype);
>> +			change_pageblock_range(page, order, migratetype);
>> +			ret = 1;
>> +		}
>> +		/*
>> +		 * Reserving the block(s) already succeeded,
>> +		 * so this should not fail on zone boundaries.
>> +		 */
>> +		WARN_ON_ONCE(ret == -1);
>> +		if (ret > 0)
>> +			return true;
>> +	}
>> +
>> +	return false;
>> +}
>> +
>>  /*
>>   * Used when an allocation is about to fail under memory pressure. This
>>   * potentially hurts the reliability of high-order allocations when under
>> @@ -3491,70 +3553,12 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,
>>  	struct zonelist *zonelist = ac->zonelist;
>>  	struct zoneref *z;
>>  	struct zone *zone;
>> -	struct page *page;
>> -	int order;
>> -	int ret;
>>  
>>  	for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx,
>>  								ac->nodemask) {
>> -		/*
>> -		 * Preserve at least one pageblock unless memory pressure
>> -		 * is really high.
>> -		 */
>> -		if (!force && zone->nr_reserved_highatomic <=
>> -					pageblock_nr_pages)
>> -			continue;
>> -
>> -		guard(spinlock_irqsave)(&zone->lock);
>> -		for (order = 0; order < NR_PAGE_ORDERS; order++) {
>> -			struct free_area *area = &(zone->free_area[order]);
>> -			unsigned long size;
>> -
>> -			page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
>> -			if (!page)
>> -				continue;
>> -
>> -			size = max(pageblock_nr_pages, 1UL << order);
>> -			/*
>> -			 * It should never happen but changes to
>> -			 * locking could inadvertently allow a per-cpu
>> -			 * drain to add pages to MIGRATE_HIGHATOMIC
>> -			 * while unreserving so be safe and watch for
>> -			 * underflows.
>> -			 */
>> -			if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic))
>> -				size = zone->nr_reserved_highatomic;
>> -			zone->nr_reserved_highatomic -= size;
>> -
>> -			/*
>> -			 * Convert to ac->migratetype and avoid the normal
>> -			 * pageblock stealing heuristics. Minimally, the caller
>> -			 * is doing the work and needs the pages. More
>> -			 * importantly, if the block was always converted to
>> -			 * MIGRATE_UNMOVABLE or another type then the number
>> -			 * of pageblocks that cannot be completely freed
>> -			 * may increase.
>> -			 */
>> -			if (order < pageblock_order)
>> -				ret = move_freepages_block(zone, page,
>> -							   MIGRATE_HIGHATOMIC,
>> -							   ac->migratetype);
>> -			else {
>> -				move_to_free_list(page, zone, order,
>> -						  MIGRATE_HIGHATOMIC,
>> -						  ac->migratetype);
>> -				change_pageblock_range(page, order,
>> -						       ac->migratetype);
>> -				ret = 1;
>> -			}
>> -			/*
>> -			 * Reserving the block(s) already succeeded,
>> -			 * so this should not fail on zone boundaries.
>> -			 */
>> -			WARN_ON_ONCE(ret == -1);
>> -			if (ret > 0)
>> -				return ret;
>> -		}
>> +		if (unreserve_highatomic_pageblock_zone(zone,
>> +							ac->migratetype, force))
>> +			return true;
>>  	}
>>  
>>  	return false;
>> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
>> index e238ee91ed24..9fba6480b18d 100644
>> --- a/mm/page_alloc.h
>> +++ b/mm/page_alloc.h
>> @@ -282,6 +282,9 @@ static inline bool free_area_empty(struct free_area *area, int migratetype)
>>  	return list_empty(&area->free_list[migratetype]);
>>  }
>>  
>> +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype,
>> +					 bool force);
>> +
>>  /* Convert GFP flags to their corresponding migrate type */
>>  #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
>>  #define GFP_MOVABLE_SHIFT 3
>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index 9959ca158ce8..91c622068d99 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
>>  		if (nr_boost_reclaim && !nr_reclaimed)
>>  			break;
>>  
>> -		if (raise_priority || !nr_reclaimed)
>> +		if (raise_priority || !nr_reclaimed) {
>> +			/*
>> +			 * Similarly to when direct reclaim is retrying, try to
>> +			 * unreserve highatomic pageblocks on the node. Convert
>> +			 * them to movable ones as kswapd doesn't have a target
>> +			 * migratetype, and passing unmovable should lead to
>> +			 * least permanent fragmentation.
>> +			 */
>> +			for (int i = 0; i <= highest_zoneidx; i++) {
>> +				struct zone *zone = pgdat->node_zones + i;
>> +
>> +				unreserve_highatomic_pageblock_zone(zone,
>> +						MIGRATE_UNMOVABLE, false);
>> +			}
>>  			sc.priority--;
>> +		}
>>  	} while (sc.priority >= 1);
>>  
>>  	/*
>> 
>> ---
>> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
>> change-id: 20261030-kswapd-highatomic-unreserve-7e6629ec65d0
>> 

Re: [PATCH] mm, vmscan: make kswapd unreserve highatomic pageblocks

[Jose A. Perez de Azpillaga]

On 11/4/26 7:38 PM, Vlastimil Babka (SUSE) wrote:
> Hook this highatomic unreserve to the moment we are increasing kswapd
> reclaim priority, as that is when kswapd is struggling to make progress
> and it should roughly match the case of direct reclaim failure.
> 

isn't raise_priority true after almost every pass at DEF_PRIORITY, stuck
or not?

[snip]

> +		if (raise_priority || !nr_reclaimed) {
> +			/*
> +			 * Similarly to when direct reclaim is retrying, try to
> +			 * unreserve highatomic pageblocks on the node. Convert
> +			 * them to movable ones as kswapd doesn't have a target
> +			 * migratetype, and passing unmovable should lead to
> +			 * least permanent fragmentation.
> +			 */
> +			for (int i = 0; i <= highest_zoneidx; i++) {
> +				struct zone *zone = pgdat->node_zones + i;
> +
> +				unreserve_highatomic_pageblock_zone(zone,
> +						MIGRATE_UNMOVABLE, false);
> +			}
s/movable/unmovable/ in the comment?

-- 
cheers,
jose a. p-a