[PATCH v6] mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

[PATCH v6] mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

[Salvatore Dipietro]

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
introduced high-order folio allocations in the iomap buffered write
path.  When memory is fragmented, each failed costly-order allocation
enters __alloc_pages_slowpath() which runs direct reclaim, direct
compaction and drain_all_pages(), causing a 0.38x throughput drop on
PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64
system.

The root issue is that direct reclaim and direct compaction are too
expensive for hot allocation paths that have fallbacks to smaller
allocations.  __filemap_get_folio_mpol() already marks higher-order
allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the
caller can handle failure.  However, the page allocator still enters the
full blocking slowpath for costly orders with __GFP_NORETRY, which is
unnecessarily aggressive when the caller will simply retry at a lower
order.

For costly-order allocations with __GFP_NORETRY, clear
__GFP_DIRECT_RECLAIM at the very start of the slowpath, before
can_direct_reclaim, can_compact and the nofail checks are evaluated.
This makes the entire slowpath treat the request as non-blocking: no
direct reclaim, no direct compaction and no drain_all_pages() IPI
across every CPU.  Reclaim and compaction move to the background
instead: kswapd is still woken for reclaim, and it wakes kcompactd for
defragmentation.  The system keeps its long-term health while the
latency-critical direct allocation path no longer stalls.

Allocations that also request __GFP_THISNODE are exempted.  That flag
pairing identifies the local-node-first THP attempt issued by
alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
form transparent huge pages.

Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the
!can_direct_reclaim bailout, which since commit 4c0ed883e051
("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops
ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on.
With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed
by fragmenting another migratetype's pageblock.  That is the wrong trade
for a caller that has already communicated it can handle failure and has
a cheap lower-order fallback, so exclude __GFP_NORETRY from the
relaxation.  The exclusion is not restricted to costly orders: any
__GFP_NORETRY caller has said it can cope with failure, so failing is
preferable to fragmenting for all of them.

Test environment:
  Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

  Config                                       Avg TPS   % vs baseline
  v7.3-rc1 baseline (vm.defrag_mode=0)          58,602               -
  v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
  v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
  v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%

Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
Link: https://lore.kernel.org/all/20260904115629.3993331-1-dipiets@amazon.it/T/#u [v4]
Link: https://lore.kernel.org/all/20260911142102.2294202-1-dipiets@amazon.it/T/#u [v5]
Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Cc: stable@vger.kernel.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: David Hildenbrand <david@redhat.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Dave Chinner <dgc@kernel.org>
Cc: Ritesh Harjani <ritesh.list@gmail.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Dmitry Ilvokhin <d@ilvokhin.com>
Cc: linux-mm@kvack.org
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-xfs@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
---
 mm/page_alloc.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c48..64d09964ca77 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
@@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;

base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH v6] mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

[Andrew Morton]

On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <dipiets@amazon.it> wrote:

> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.  When memory is fragmented, each failed costly-order allocation
> enters __alloc_pages_slowpath() which runs direct reclaim, direct
> compaction and drain_all_pages(), causing a 0.38x throughput drop on
> PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64
> system.
> 
> The root issue is that direct reclaim and direct compaction are too
> expensive for hot allocation paths that have fallbacks to smaller
> allocations.  __filemap_get_folio_mpol() already marks higher-order
> allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the
> caller can handle failure.  However, the page allocator still enters the
> full blocking slowpath for costly orders with __GFP_NORETRY, which is
> unnecessarily aggressive when the caller will simply retry at a lower
> order.
> 

Thanks, I'll update mm-hotfixes to this version.

Here's what changed:

--- a/mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations
+++ a/mm/page_alloc.c
@@ -4930,8 +4930,11 @@ retry:
 		 * Reclaim/compaction cannot run, so defrag_mode's strategy
 		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
 		 * fallbacks rather than failing the allocation outright.
+		 * Not for __GFP_NORETRY: those have a cheap lower order
+		 * fallback, so failing beats fragmenting.
 		 */
 		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
+		    !(gfp_mask & __GFP_NORETRY) &&
 		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
 			alloc_flags &= ~ALLOC_NOFRAGMENT;
 			goto retry;
_

Re: [PATCH v6] mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 10/2/26 03:29, Andrew Morton wrote:
> On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <dipiets@amazon.it> wrote:
> 
>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>> introduced high-order folio allocations in the iomap buffered write
>> path.  When memory is fragmented, each failed costly-order allocation
>> enters __alloc_pages_slowpath() which runs direct reclaim, direct
>> compaction and drain_all_pages(), causing a 0.38x throughput drop on
>> PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64
>> system.
>> 
>> The root issue is that direct reclaim and direct compaction are too
>> expensive for hot allocation paths that have fallbacks to smaller
>> allocations.  __filemap_get_folio_mpol() already marks higher-order
>> allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the
>> caller can handle failure.  However, the page allocator still enters the
>> full blocking slowpath for costly orders with __GFP_NORETRY, which is
>> unnecessarily aggressive when the caller will simply retry at a lower
>> order.
>> 

Since it expands the impact of a pre-existing bug of nonblocking allocation
eating reserves [1] it shouldn't proceed to mainline without a fix for that
(Johannes posted it there but the review discussion got stalled).

> Thanks, I'll update mm-hotfixes to this version.

I've also argued for not handling it as a hotfix [2]

[1] https://lore.kernel.org/all/9f415dc7-adad-4161-b20d-7c3173f50ff3@kernel.org/

[2] https://lore.kernel.org/all/c6e2c1e1-a285-406e-a2b6-c09e05df91d8@kernel.org/

> Here's what changed:
> 
> --- a/mm/page_alloc.c~mm-page_alloc-avoid-direct-reclaim-and-compaction-for-costly-__gfp_noretry-allocations
> +++ a/mm/page_alloc.c
> @@ -4930,8 +4930,11 @@ retry:
>  		 * Reclaim/compaction cannot run, so defrag_mode's strategy
>  		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
>  		 * fallbacks rather than failing the allocation outright.
> +		 * Not for __GFP_NORETRY: those have a cheap lower order
> +		 * fallback, so failing beats fragmenting.
>  		 */
>  		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
> +		    !(gfp_mask & __GFP_NORETRY) &&
>  		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
>  			alloc_flags &= ~ALLOC_NOFRAGMENT;
>  			goto retry;
> _
> 

Re: [PATCH v6] mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

[Andrew Morton]

On Fri, 2 Oct 2026 10:23:47 +0200 "Vlastimil Babka (SUSE)" <vbabka@kernel.org> wrote:

> On 10/2/26 03:29, Andrew Morton wrote:
> > On Thu, 1 Oct 2026 08:21:51 +0000 Salvatore Dipietro <dipiets@amazon.it> wrote:
> > 
> >> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> >> introduced high-order folio allocations in the iomap buffered write
> >> path.  When memory is fragmented, each failed costly-order allocation
> >> enters __alloc_pages_slowpath() which runs direct reclaim, direct
> >> compaction and drain_all_pages(), causing a 0.38x throughput drop on
> >> PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64
> >> system.
> >> 
> >> The root issue is that direct reclaim and direct compaction are too
> >> expensive for hot allocation paths that have fallbacks to smaller
> >> allocations.  __filemap_get_folio_mpol() already marks higher-order
> >> allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the
> >> caller can handle failure.  However, the page allocator still enters the
> >> full blocking slowpath for costly orders with __GFP_NORETRY, which is
> >> unnecessarily aggressive when the caller will simply retry at a lower
> >> order.
> >> 
> 
> Since it expands the impact of a pre-existing bug of nonblocking allocation
> eating reserves [1] it shouldn't proceed to mainline without a fix for that
> (Johannes posted it there but the review discussion got stalled).
> 
> > Thanks, I'll update mm-hotfixes to this version.
> 
> I've also argued for not handling it as a hotfix [2]
> 
> [1] https://lore.kernel.org/all/9f415dc7-adad-4161-b20d-7c3173f50ff3@kernel.org/
> 
> [2] https://lore.kernel.org/all/c6e2c1e1-a285-406e-a2b6-c09e05df91d8@kernel.org/
> 

Thanks.  I moved this to tail of mm-unstable, removed its cc:stable and
added a note indicating that changes are expected.

Re: [PATCH v6] mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

[Thadeu Lima de Souza Cascardo]

On Thu, Oct 01, 2026 at 08:21:51AM +0000, Salvatore Dipietro wrote:
> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.  When memory is fragmented, each failed costly-order allocation
> enters __alloc_pages_slowpath() which runs direct reclaim, direct
> compaction and drain_all_pages(), causing a 0.38x throughput drop on
> PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64
> system.
> 
> The root issue is that direct reclaim and direct compaction are too
> expensive for hot allocation paths that have fallbacks to smaller
> allocations.  __filemap_get_folio_mpol() already marks higher-order
> allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the
> caller can handle failure.  However, the page allocator still enters the
> full blocking slowpath for costly orders with __GFP_NORETRY, which is
> unnecessarily aggressive when the caller will simply retry at a lower
> order.
> 
> For costly-order allocations with __GFP_NORETRY, clear
> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
> can_direct_reclaim, can_compact and the nofail checks are evaluated.
> This makes the entire slowpath treat the request as non-blocking: no
> direct reclaim, no direct compaction and no drain_all_pages() IPI
> across every CPU.  Reclaim and compaction move to the background
> instead: kswapd is still woken for reclaim, and it wakes kcompactd for
> defragmentation.  The system keeps its long-term health while the
> latency-critical direct allocation path no longer stalls.
> 
> Allocations that also request __GFP_THISNODE are exempted.  That flag
> pairing identifies the local-node-first THP attempt issued by
> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
> form transparent huge pages.
> 
> Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the
> !can_direct_reclaim bailout, which since commit 4c0ed883e051
> ("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops
> ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on.
> With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed
> by fragmenting another migratetype's pageblock.  That is the wrong trade
> for a caller that has already communicated it can handle failure and has
> a cheap lower-order fallback, so exclude __GFP_NORETRY from the
> relaxation.  The exclusion is not restricted to costly orders: any
> __GFP_NORETRY caller has said it can cope with failure, so failing is
> preferable to fragmenting for all of them.
> 
> Test environment:
>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>              12x 1TB IO2 32000 IOPS RAID0 XFS
>   OS:        AL2023
>   Kernel:    v7.3-rc1
>   Database:  PostgreSQL 18.4
>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
> 
> Results (average of 3 runs, TPS):
> 
>   Config                                       Avg TPS   % vs baseline
>   v7.3-rc1 baseline (vm.defrag_mode=0)          58,602               -
>   v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
>   v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
>   v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%
> 
> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
> Link: https://lore.kernel.org/all/20260904115629.3993331-1-dipiets@amazon.it/T/#u [v4]
> Link: https://lore.kernel.org/all/20260911142102.2294202-1-dipiets@amazon.it/T/#u [v5]
> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> Cc: stable@vger.kernel.org
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Vlastimil Babka <vbabka@kernel.org>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Matthew Wilcox <willy@infradead.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Dave Chinner <dgc@kernel.org>
> Cc: Ritesh Harjani <ritesh.list@gmail.com>
> Cc: Zi Yan <ziy@nvidia.com>
> Cc: Dmitry Ilvokhin <d@ilvokhin.com>
> Cc: linux-mm@kvack.org
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-xfs@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
> Acked-by: Zi Yan <ziy@nvidia.com>
> Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
> ---
>  mm/page_alloc.c | 21 ++++++++++++++++++---
>  1 file changed, 18 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 12fac9084c48..64d09964ca77 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4784,10 +4784,10 @@ static inline struct page *
>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  						struct alloc_context *ac)
>  {
> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
> -	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
> -	bool nofail = gfp_mask & __GFP_NOFAIL;
>  	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
> +	bool can_direct_reclaim;
> +	bool can_compact;
> +	bool nofail;
>  	struct page *page = NULL;
>  	unsigned int alloc_flags;
>  	unsigned long did_some_progress;
> @@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  	bool can_retry_reserves = true;
>  	unsigned long alloc_start_time = jiffies;
>  
> +	/*
> +	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
> +	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
> +	 */
> +	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
> +	    !(gfp_mask & __GFP_THISNODE))
> +		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
> +
> +	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
> +	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
> +	nofail = gfp_mask & __GFP_NOFAIL;
> +
>  	if (unlikely(nofail)) {
>  		/*
>  		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,

This would trigger a warning if __GFP_NOFAIL and __GFP_NORETRY were ever
used together, though that does not make sense and I haven't found a case
where it is used in practice.

Aside from either warning for the specific case of these flags being used
together or adding a comment right here to that effect, I don't think any
other change is needed, though.

Another comment about the general approach: TTM pool page allocator does
use __GFP_NORETRY and __GFP_DIRECT_RECLAIM for high orders and when we
previously suggested to drop __GFP_DIRECT_RECLAIM to avoid compaction, it
was frowned upon. In the end, a lot of other changes were made and for the
relevant cases, this does not seem to change its current behavior, amongst
other things, because it uses __GFP_THISNODE.

I did run some TTM GTT allocations with some fragmentation workload and I
haven't seen much change in the latency for these high order allocations,
so:

Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>

> @@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  		 * Reclaim/compaction cannot run, so defrag_mode's strategy
>  		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
>  		 * fallbacks rather than failing the allocation outright.
> +		 * Not for __GFP_NORETRY: those have a cheap lower order
> +		 * fallback, so failing beats fragmenting.
>  		 */
>  		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
> +		    !(gfp_mask & __GFP_NORETRY) &&
>  		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
>  			alloc_flags &= ~ALLOC_NOFRAGMENT;
>  			goto retry;
> 
> base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
> -- 
> 2.50.1
> 
> 
> 
> 
> AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
> 
> 
> 

Re: [PATCH v6] mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 11/4/26 20:11, Thadeu Lima de Souza Cascardo wrote:
> On Thu, Oct 01, 2026 at 08:21:51AM +0000, Salvatore Dipietro wrote:
>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>> introduced high-order folio allocations in the iomap buffered write
>> path.  When memory is fragmented, each failed costly-order allocation
>> enters __alloc_pages_slowpath() which runs direct reclaim, direct
>> compaction and drain_all_pages(), causing a 0.38x throughput drop on
>> PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64
>> system.
>> 
>> The root issue is that direct reclaim and direct compaction are too
>> expensive for hot allocation paths that have fallbacks to smaller
>> allocations.  __filemap_get_folio_mpol() already marks higher-order
>> allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the
>> caller can handle failure.  However, the page allocator still enters the
>> full blocking slowpath for costly orders with __GFP_NORETRY, which is
>> unnecessarily aggressive when the caller will simply retry at a lower
>> order.
>> 
>> For costly-order allocations with __GFP_NORETRY, clear
>> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
>> can_direct_reclaim, can_compact and the nofail checks are evaluated.
>> This makes the entire slowpath treat the request as non-blocking: no
>> direct reclaim, no direct compaction and no drain_all_pages() IPI
>> across every CPU.  Reclaim and compaction move to the background
>> instead: kswapd is still woken for reclaim, and it wakes kcompactd for
>> defragmentation.  The system keeps its long-term health while the
>> latency-critical direct allocation path no longer stalls.
>> 
>> Allocations that also request __GFP_THISNODE are exempted.  That flag
>> pairing identifies the local-node-first THP attempt issued by
>> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
>> form transparent huge pages.
>> 
>> Clearing __GFP_DIRECT_RECLAIM makes these allocations reach the
>> !can_direct_reclaim bailout, which since commit 4c0ed883e051
>> ("mm/page_alloc: fix defrag_mode for non-reclaimable allocations") drops
>> ALLOC_NOFRAGMENT and retries instead of failing when defrag_mode is on.
>> With vm.defrag_mode=1 a __GFP_NORETRY allocation could therefore succeed
>> by fragmenting another migratetype's pageblock.  That is the wrong trade
>> for a caller that has already communicated it can handle failure and has
>> a cheap lower-order fallback, so exclude __GFP_NORETRY from the
>> relaxation.  The exclusion is not restricted to costly orders: any
>> __GFP_NORETRY caller has said it can cope with failure, so failing is
>> preferable to fragmenting for all of them.
>> 
>> Test environment:
>>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>>              12x 1TB IO2 32000 IOPS RAID0 XFS
>>   OS:        AL2023
>>   Kernel:    v7.3-rc1
>>   Database:  PostgreSQL 18.4
>>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
>> 
>> Results (average of 3 runs, TPS):
>> 
>>   Config                                       Avg TPS   % vs baseline
>>   v7.3-rc1 baseline (vm.defrag_mode=0)          58,602               -
>>   v7.3-rc1 + this patch (vm.defrag_mode=0)     157,246         +168.3%
>>   v7.3-rc1 baseline (vm.defrag_mode=1)         109,012               -
>>   v7.3-rc1 + this patch (vm.defrag_mode=1)     157,778          +44.7%
>> 
>> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
>> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
>> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
>> Link: https://lore.kernel.org/all/20260904115629.3993331-1-dipiets@amazon.it/T/#u [v4]
>> Link: https://lore.kernel.org/all/20260911142102.2294202-1-dipiets@amazon.it/T/#u [v5]
>> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>> Cc: stable@vger.kernel.org
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: Vlastimil Babka <vbabka@kernel.org>
>> Cc: David Hildenbrand <david@redhat.com>
>> Cc: Michal Hocko <mhocko@suse.com>
>> Cc: Johannes Weiner <hannes@cmpxchg.org>
>> Cc: Matthew Wilcox <willy@infradead.org>
>> Cc: Christoph Hellwig <hch@lst.de>
>> Cc: Dave Chinner <dgc@kernel.org>
>> Cc: Ritesh Harjani <ritesh.list@gmail.com>
>> Cc: Zi Yan <ziy@nvidia.com>
>> Cc: Dmitry Ilvokhin <d@ilvokhin.com>
>> Cc: linux-mm@kvack.org
>> Cc: linux-fsdevel@vger.kernel.org
>> Cc: linux-xfs@vger.kernel.org
>> Cc: linux-kernel@vger.kernel.org
>> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>> Acked-by: Zi Yan <ziy@nvidia.com>
>> Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
>> ---
>>  mm/page_alloc.c | 21 ++++++++++++++++++---
>>  1 file changed, 18 insertions(+), 3 deletions(-)
>> 
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index 12fac9084c48..64d09964ca77 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -4784,10 +4784,10 @@ static inline struct page *
>>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>  						struct alloc_context *ac)
>>  {
>> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>> -	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>> -	bool nofail = gfp_mask & __GFP_NOFAIL;
>>  	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
>> +	bool can_direct_reclaim;
>> +	bool can_compact;
>> +	bool nofail;
>>  	struct page *page = NULL;
>>  	unsigned int alloc_flags;
>>  	unsigned long did_some_progress;
>> @@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>  	bool can_retry_reserves = true;
>>  	unsigned long alloc_start_time = jiffies;
>>  
>> +	/*
>> +	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
>> +	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
>> +	 */
>> +	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
>> +	    !(gfp_mask & __GFP_THISNODE))
>> +		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
>> +
>> +	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>> +	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>> +	nofail = gfp_mask & __GFP_NOFAIL;
>> +
>>  	if (unlikely(nofail)) {
>>  		/*
>>  		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
> 
> This would trigger a warning if __GFP_NOFAIL and __GFP_NORETRY were ever
> used together, though that does not make sense and I haven't found a case
> where it is used in practice.

Right. But always there is some risk someone does that unintentionally by
trying to turn an allocation with incoming gfp flags (including unexpected
__GFP_NOFAIL) to an opportunistic one first by appending __GFP_NORETRY, etc.

> Aside from either warning for the specific case of these flags being used
> together or adding a comment right here to that effect, I don't think any
> other change is needed, though.

Indeed. I thought we did have such warnings. Also a combination with
__GFP_RETRY_MAYFAIL would make no sense. So we should perhaps add them.

> Another comment about the general approach: TTM pool page allocator does
> use __GFP_NORETRY and __GFP_DIRECT_RECLAIM for high orders and when we
> previously suggested to drop __GFP_DIRECT_RECLAIM to avoid compaction, it
> was frowned upon. In the end, a lot of other changes were made and for the
> relevant cases, this does not seem to change its current behavior, amongst
> other things, because it uses __GFP_THISNODE.
> 
> I did run some TTM GTT allocations with some fragmentation workload and I
> haven't seen much change in the latency for these high order allocations,
> so:
> 
> Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com>

Thanks!

>> @@ -4918,8 +4930,11 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>  		 * Reclaim/compaction cannot run, so defrag_mode's strategy
>>  		 * of enforcing ALLOC_NOFRAGMENT cannot be fulfilled. Allow
>>  		 * fallbacks rather than failing the allocation outright.
>> +		 * Not for __GFP_NORETRY: those have a cheap lower order
>> +		 * fallback, so failing beats fragmenting.
>>  		 */
>>  		if (defrag_mode && (alloc_flags & ALLOC_NOFRAGMENT) &&
>> +		    !(gfp_mask & __GFP_NORETRY) &&
>>  		    (gfp_mask & __GFP_KSWAPD_RECLAIM)) {
>>  			alloc_flags &= ~ALLOC_NOFRAGMENT;
>>  			goto retry;
>> 
>> base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
>> -- 
>> 2.50.1
>> 
>> 
>> 
>> 
>> AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico
>> 
>> 
>>