[PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Salvatore Dipietro]

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
introduced high-order folio allocations in the iomap buffered write
path.  When memory is fragmented, each failed costly-order allocation
enters __alloc_pages_slowpath() which runs direct compaction and
drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

The root issue is that direct compaction is too expensive for hot
allocation paths that have fallbacks to smaller allocations.
__filemap_get_folio_mpol() already marks higher-order allocations with
__GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
failure.  However, the page allocator still attempts full direct
compaction for costly orders with __GFP_NORETRY, which is unnecessarily
aggressive when the caller will simply retry at a lower order.

For costly-order allocations with __GFP_NORETRY, clear
__GFP_DIRECT_RECLAIM at the very start of the slowpath, before
can_direct_reclaim, can_compact and the nofail checks are evaluated.
This makes the entire slowpath treat the request as non-blocking: no
direct reclaim, no direct compaction and no drain_all_pages() IPI
across every CPU. kswapd (and in turn kcompactd) is still woken further
down for background defragmentation, so compaction keeps working for
long-term system health while being removed from the latency-critical
direct allocation path.

Allocations that also request __GFP_THISNODE are exempted.  That flag
pairing identifies the local-node-first THP attempt issued by
alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
form transparent huge pages.

Test environment:
  Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
             12x 1TB IO2 32000 IOPS RAID0 XFS
  OS:        AL2023
  Kernel:    v7.3-rc1
  Database:  PostgreSQL 18.4
  Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s

Results (average of 3 runs, TPS):

  Config                   Avg TPS      % vs Baseline
  baseline (no patch)       59,408       -
  With this patch          155,409      +161.6%

Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Cc: stable@vger.kernel.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: David Hildenbrand <david@redhat.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Dave Chinner <dgc@kernel.org>
Cc: Ritesh Harjani <ritesh.list@gmail.com>
Cc: linux-mm@kvack.org
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-xfs@vger.kernel.org
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
---
v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
    __GFP_THISNODE so THP attempt keeps using direct compaction
v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
v1: Avoid compaction in iomap folio allocation

 mm/page_alloc.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c48..542c2ec31061 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4784,10 +4784,10 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
-	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
-	bool nofail = gfp_mask & __GFP_NOFAIL;
 	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	bool can_direct_reclaim;
+	bool can_compact;
+	bool nofail;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	bool can_retry_reserves = true;
 	unsigned long alloc_start_time = jiffies;
 
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
+	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
+	 */
+	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
+	    !(gfp_mask & __GFP_THISNODE))
+		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+
+	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
+	nofail = gfp_mask & __GFP_NOFAIL;
+
 	if (unlikely(nofail)) {
 		/*
 		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 9/4/26 13:56, Salvatore Dipietro wrote:
> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.  When memory is fragmented, each failed costly-order allocation
> enters __alloc_pages_slowpath() which runs direct compaction and
> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
> 
> The root issue is that direct compaction is too expensive for hot
> allocation paths that have fallbacks to smaller allocations.
> __filemap_get_folio_mpol() already marks higher-order allocations with
> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
> failure.  However, the page allocator still attempts full direct
> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
> aggressive when the caller will simply retry at a lower order.
> 
> For costly-order allocations with __GFP_NORETRY, clear
> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
> can_direct_reclaim, can_compact and the nofail checks are evaluated.
> This makes the entire slowpath treat the request as non-blocking: no
> direct reclaim, no direct compaction and no drain_all_pages() IPI
> across every CPU. kswapd (and in turn kcompactd) is still woken further
> down for background defragmentation, so compaction keeps working for
> long-term system health while being removed from the latency-critical
> direct allocation path.
> 
> Allocations that also request __GFP_THISNODE are exempted.  That flag
> pairing identifies the local-node-first THP attempt issued by
> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
> form transparent huge pages.
> 
> Test environment:
>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>              12x 1TB IO2 32000 IOPS RAID0 XFS
>   OS:        AL2023
>   Kernel:    v7.3-rc1
>   Database:  PostgreSQL 18.4
>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
> 
> Results (average of 3 runs, TPS):
> 
>   Config                   Avg TPS      % vs Baseline
>   baseline (no patch)       59,408       -
>   With this patch          155,409      +161.6%
> 
> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> Cc: stable@vger.kernel.org
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Vlastimil Babka <vbabka@suse.cz>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Matthew Wilcox <willy@infradead.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Dave Chinner <dgc@kernel.org>
> Cc: Ritesh Harjani <ritesh.list@gmail.com>
> Cc: linux-mm@kvack.org
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-xfs@vger.kernel.org
> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>

I guess this will have to do until unlikely(we figure out a better API)...

Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>

> ---
> v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
>     __GFP_THISNODE so THP attempt keeps using direct compaction
> v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
> v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
> v1: Avoid compaction in iomap folio allocation
> 
>  mm/page_alloc.c | 18 +++++++++++++++---
>  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 12fac9084c48..542c2ec31061 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4784,10 +4784,10 @@ static inline struct page *
>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  						struct alloc_context *ac)
>  {
> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
> -	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
> -	bool nofail = gfp_mask & __GFP_NOFAIL;
>  	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
> +	bool can_direct_reclaim;
> +	bool can_compact;
> +	bool nofail;
>  	struct page *page = NULL;
>  	unsigned int alloc_flags;
>  	unsigned long did_some_progress;
> @@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  	bool can_retry_reserves = true;
>  	unsigned long alloc_start_time = jiffies;
>  
> +	/*
> +	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
> +	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
> +	 */
> +	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
> +	    !(gfp_mask & __GFP_THISNODE))
> +		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
> +
> +	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
> +	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
> +	nofail = gfp_mask & __GFP_NOFAIL;
> +
>  	if (unlikely(nofail)) {
>  		/*
>  		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Zi Yan]

On Fri Sep 4, 2026 at 10:11 AM EDT, Vlastimil Babka (SUSE) wrote:
> On 9/4/26 13:56, Salvatore Dipietro wrote:
>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>> introduced high-order folio allocations in the iomap buffered write
>> path.  When memory is fragmented, each failed costly-order allocation
>> enters __alloc_pages_slowpath() which runs direct compaction and
>> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
>> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
>> 
>> The root issue is that direct compaction is too expensive for hot
>> allocation paths that have fallbacks to smaller allocations.
>> __filemap_get_folio_mpol() already marks higher-order allocations with
>> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
>> failure.  However, the page allocator still attempts full direct
>> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
>> aggressive when the caller will simply retry at a lower order.
>> 
>> For costly-order allocations with __GFP_NORETRY, clear
>> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
>> can_direct_reclaim, can_compact and the nofail checks are evaluated.
>> This makes the entire slowpath treat the request as non-blocking: no
>> direct reclaim, no direct compaction and no drain_all_pages() IPI
>> across every CPU. kswapd (and in turn kcompactd) is still woken further
>> down for background defragmentation, so compaction keeps working for
>> long-term system health while being removed from the latency-critical
>> direct allocation path.
>> 
>> Allocations that also request __GFP_THISNODE are exempted.  That flag
>> pairing identifies the local-node-first THP attempt issued by
>> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
>> form transparent huge pages.
>> 
>> Test environment:
>>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>>              12x 1TB IO2 32000 IOPS RAID0 XFS
>>   OS:        AL2023
>>   Kernel:    v7.3-rc1
>>   Database:  PostgreSQL 18.4
>>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
>> 
>> Results (average of 3 runs, TPS):
>> 
>>   Config                   Avg TPS      % vs Baseline
>>   baseline (no patch)       59,408       -
>>   With this patch          155,409      +161.6%
>> 
>> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
>> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
>> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
>> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>> Cc: stable@vger.kernel.org
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: Vlastimil Babka <vbabka@suse.cz>
>> Cc: David Hildenbrand <david@redhat.com>
>> Cc: Michal Hocko <mhocko@suse.com>
>> Cc: Johannes Weiner <hannes@cmpxchg.org>
>> Cc: Matthew Wilcox <willy@infradead.org>
>> Cc: Christoph Hellwig <hch@lst.de>
>> Cc: Dave Chinner <dgc@kernel.org>
>> Cc: Ritesh Harjani <ritesh.list@gmail.com>
>> Cc: linux-mm@kvack.org
>> Cc: linux-fsdevel@vger.kernel.org
>> Cc: linux-xfs@vger.kernel.org
>> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
>
> I guess this will have to do until unlikely(we figure out a better API)...

We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on
various policies we currently have. Even better if callers can do that
instead.

>
> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>
>> ---
>> v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
>>     __GFP_THISNODE so THP attempt keeps using direct compaction
>> v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
>> v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
>> v1: Avoid compaction in iomap folio allocation
>> 
>>  mm/page_alloc.c | 18 +++++++++++++++---
>>  1 file changed, 15 insertions(+), 3 deletions(-)
>> 
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index 12fac9084c48..542c2ec31061 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -4784,10 +4784,10 @@ static inline struct page *
>>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>  						struct alloc_context *ac)
>>  {
>> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>> -	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>> -	bool nofail = gfp_mask & __GFP_NOFAIL;
>>  	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
>> +	bool can_direct_reclaim;
>> +	bool can_compact;
>> +	bool nofail;
>>  	struct page *page = NULL;
>>  	unsigned int alloc_flags;
>>  	unsigned long did_some_progress;
>> @@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>  	bool can_retry_reserves = true;
>>  	unsigned long alloc_start_time = jiffies;
>>  
>> +	/*
>> +	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
>> +	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
>> +	 */

Should this also be documented in gfp_types.h? It currently only says,

"__GFP_NORETRY: The VM implementation will try only very lightweight
memory direct reclaim to get some memory under memory pressure (thus it
can sleep)."

Otherwise,

Acked-by: Zi Yan <ziy@nvidia.com>

>> +	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
>> +	    !(gfp_mask & __GFP_THISNODE))
>> +		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
>> +
>> +	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>> +	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>> +	nofail = gfp_mask & __GFP_NOFAIL;
>> +
>>  	if (unlikely(nofail)) {
>>  		/*
>>  		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,




-- 
Best Regards,
Yan, Zi

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Johannes Weiner]

On Fri, Sep 04, 2026 at 11:56:28AM +0000, Salvatore Dipietro wrote:
> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.  When memory is fragmented, each failed costly-order allocation
> enters __alloc_pages_slowpath() which runs direct compaction and
> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
> 
> The root issue is that direct compaction is too expensive for hot
> allocation paths that have fallbacks to smaller allocations.
> __filemap_get_folio_mpol() already marks higher-order allocations with
> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
> failure.  However, the page allocator still attempts full direct
> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
> aggressive when the caller will simply retry at a lower order.
> 
> For costly-order allocations with __GFP_NORETRY, clear
> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
> can_direct_reclaim, can_compact and the nofail checks are evaluated.
> This makes the entire slowpath treat the request as non-blocking: no
> direct reclaim, no direct compaction and no drain_all_pages() IPI
> across every CPU. kswapd (and in turn kcompactd) is still woken further
> down for background defragmentation, so compaction keeps working for
> long-term system health while being removed from the latency-critical
> direct allocation path.
> 
> Allocations that also request __GFP_THISNODE are exempted.  That flag
> pairing identifies the local-node-first THP attempt issued by
> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
> form transparent huge pages.
> 
> Test environment:
>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>              12x 1TB IO2 32000 IOPS RAID0 XFS
>   OS:        AL2023
>   Kernel:    v7.3-rc1
>   Database:  PostgreSQL 18.4
>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
> 
> Results (average of 3 runs, TPS):
> 
>   Config                   Avg TPS      % vs Baseline
>   baseline (no patch)       59,408       -
>   With this patch          155,409      +161.6%
> 
> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> Cc: stable@vger.kernel.org
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Vlastimil Babka <vbabka@suse.cz>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Matthew Wilcox <willy@infradead.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Dave Chinner <dgc@kernel.org>
> Cc: Ritesh Harjani <ritesh.list@gmail.com>
> Cc: linux-mm@kvack.org
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-xfs@vger.kernel.org
> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>

Looks good to me, thanks for keeping at it.

Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Andrew Morton]

On Fri, 4 Sep 2026 11:56:28 +0000 Salvatore Dipietro <dipiets@amazon.it> wrote:

> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> introduced high-order folio allocations in the iomap buffered write
> path.  When memory is fragmented, each failed costly-order allocation
> enters __alloc_pages_slowpath() which runs direct compaction and
> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.

Is there anything particularly unusual about this test case?

> Results (average of 3 runs, TPS):
> 
>   Config                   Avg TPS      % vs Baseline
>   baseline (no patch)       59,408       -
>   With this patch          155,409      +161.6%

Is this back to pre-5d8edfb900d5 performance?

> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")

That's three years old.  v6.10.  

This is not good.  Can you think of an exceptional reason why this took
so long to surface, or is it simply that our testing isn't good?

And thanks for fixing it.

AI review asked a few serious-looking questions:
	https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Dave Chinner]

On Sat, Sep 05, 2026 at 05:42:39PM -0700, Andrew Morton wrote:
> On Fri, 4 Sep 2026 11:56:28 +0000 Salvatore Dipietro <dipiets@amazon.it> wrote:
> 
> > Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> > introduced high-order folio allocations in the iomap buffered write
> > path.  When memory is fragmented, each failed costly-order allocation
> > enters __alloc_pages_slowpath() which runs direct compaction and
> > drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
> > pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
> 
> Is there anything particularly unusual about this test case?
> 
> > Results (average of 3 runs, TPS):
> > 
> >   Config                   Avg TPS      % vs Baseline
> >   baseline (no patch)       59,408       -
> >   With this patch          155,409      +161.6%
> 
> Is this back to pre-5d8edfb900d5 performance?
> 
> > Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
> > Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
> > Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
> > Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
> 
> That's three years old.  v6.10.  
> 
> This is not good.  Can you think of an exceptional reason why this took
> so long to surface, or is it simply that our testing isn't good?

Neither.  We've know about these excessive high-order allocation->
direct reclaim compaction costs for a lot longer than this, and have
quite a number of filesystem based workloads that have been easily
able to reproduce it on demand.  XFS first worked around direct
compaction overhead back around ~5.17/18.

So why has it taken so long for this to be (kinda) fixed in the
memory reclaim code?

It is not for lack of awareness or trying.

I've responded to about half a dozen "direct compaction is
expensive" bug reports in the past couple of years where I've said
"this needs to be fixed in the reclaim code, not at the individual
call sites". Not just fs stuff, but graphics and other subsystems,
too.

However, convincing the MM manintainers to do anything w.r.t.
changing memory reclaim takes forever, and requires excessive
amounts of persistence before anything happens despite clear
evidence of the issue and the performance numbers showing how costly
direct compaction actually is.

A directly relatable example of this is kvmalloc() semantics. We
needed kvmalloc() use fail fast semantics for the high order
kmalloc() because it is much less costly to fall back to vmalloc()
than do direct compaction....

This was what the XFS changes in 5.17 worked around, and that fix
didn't get pulled into the kvmalloc() implementation until 6.15 with
commit 46459154f997 ("mm: kvmalloc: make kmalloc fast path real fast
path").

IOWs, it often takes years from "bad behaviour known" to "fixed in
MM infrastructure", and most of that time is spent trying to
convince the MM maintainers that the (relatively simple) fix needs
to be done in the MM code itself...

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 9/4/26 17:08, Zi Yan wrote:
> On Fri Sep 4, 2026 at 10:11 AM EDT, Vlastimil Babka (SUSE) wrote:
>> On 9/4/26 13:56, Salvatore Dipietro wrote:
>>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>> introduced high-order folio allocations in the iomap buffered write
>>> path.  When memory is fragmented, each failed costly-order allocation
>>> enters __alloc_pages_slowpath() which runs direct compaction and
>>> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
>>> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
>>> 
>>> The root issue is that direct compaction is too expensive for hot
>>> allocation paths that have fallbacks to smaller allocations.
>>> __filemap_get_folio_mpol() already marks higher-order allocations with
>>> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
>>> failure.  However, the page allocator still attempts full direct
>>> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
>>> aggressive when the caller will simply retry at a lower order.
>>> 
>>> For costly-order allocations with __GFP_NORETRY, clear
>>> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
>>> can_direct_reclaim, can_compact and the nofail checks are evaluated.
>>> This makes the entire slowpath treat the request as non-blocking: no
>>> direct reclaim, no direct compaction and no drain_all_pages() IPI
>>> across every CPU. kswapd (and in turn kcompactd) is still woken further
>>> down for background defragmentation, so compaction keeps working for
>>> long-term system health while being removed from the latency-critical
>>> direct allocation path.
>>> 
>>> Allocations that also request __GFP_THISNODE are exempted.  That flag
>>> pairing identifies the local-node-first THP attempt issued by
>>> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
>>> form transparent huge pages.
>>> 
>>> Test environment:
>>>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>>>              12x 1TB IO2 32000 IOPS RAID0 XFS
>>>   OS:        AL2023
>>>   Kernel:    v7.3-rc1
>>>   Database:  PostgreSQL 18.4
>>>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
>>> 
>>> Results (average of 3 runs, TPS):
>>> 
>>>   Config                   Avg TPS      % vs Baseline
>>>   baseline (no patch)       59,408       -
>>>   With this patch          155,409      +161.6%
>>> 
>>> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
>>> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
>>> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
>>> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>> Cc: stable@vger.kernel.org
>>> Cc: Andrew Morton <akpm@linux-foundation.org>
>>> Cc: Vlastimil Babka <vbabka@suse.cz>
>>> Cc: David Hildenbrand <david@redhat.com>
>>> Cc: Michal Hocko <mhocko@suse.com>
>>> Cc: Johannes Weiner <hannes@cmpxchg.org>
>>> Cc: Matthew Wilcox <willy@infradead.org>
>>> Cc: Christoph Hellwig <hch@lst.de>
>>> Cc: Dave Chinner <dgc@kernel.org>
>>> Cc: Ritesh Harjani <ritesh.list@gmail.com>
>>> Cc: linux-mm@kvack.org
>>> Cc: linux-fsdevel@vger.kernel.org
>>> Cc: linux-xfs@vger.kernel.org
>>> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
>>
>> I guess this will have to do until unlikely(we figure out a better API)...
> 
> We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on
> various policies we currently have.

Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.

> Even better if callers can do that
> instead.

Not sure about burdening the callers (e.g. "every filesystem should do X"
etc), unless they are some intermediate wrappers within mm itself.
For example, THP (but only anonymous?) is now handled in such a special way,
it could be a candidate. But I also have a vague feeling this was already
done in the past. Worth investigating perhaps.

>>
>> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
>>
>>> ---
>>> v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
>>>     __GFP_THISNODE so THP attempt keeps using direct compaction
>>> v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
>>> v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
>>> v1: Avoid compaction in iomap folio allocation
>>> 
>>>  mm/page_alloc.c | 18 +++++++++++++++---
>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>> 
>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>> index 12fac9084c48..542c2ec31061 100644
>>> --- a/mm/page_alloc.c
>>> +++ b/mm/page_alloc.c
>>> @@ -4784,10 +4784,10 @@ static inline struct page *
>>>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>>  						struct alloc_context *ac)
>>>  {
>>> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>>> -	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>>> -	bool nofail = gfp_mask & __GFP_NOFAIL;
>>>  	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
>>> +	bool can_direct_reclaim;
>>> +	bool can_compact;
>>> +	bool nofail;
>>>  	struct page *page = NULL;
>>>  	unsigned int alloc_flags;
>>>  	unsigned long did_some_progress;
>>> @@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>>  	bool can_retry_reserves = true;
>>>  	unsigned long alloc_start_time = jiffies;
>>>  
>>> +	/*
>>> +	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
>>> +	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
>>> +	 */
> 
> Should this also be documented in gfp_types.h? It currently only says,
> 
> "__GFP_NORETRY: The VM implementation will try only very lightweight
> memory direct reclaim to get some memory under memory pressure (thus it
> can sleep)."
> 
> Otherwise,
> 
> Acked-by: Zi Yan <ziy@nvidia.com>
> 
>>> +	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
>>> +	    !(gfp_mask & __GFP_THISNODE))
>>> +		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
>>> +
>>> +	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>>> +	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>>> +	nofail = gfp_mask & __GFP_NOFAIL;
>>> +
>>>  	if (unlikely(nofail)) {
>>>  		/*
>>>  		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
> 
> 
> 
> 

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Zi Yan]

On Mon Sep 7, 2026 at 3:30 AM EDT, Vlastimil Babka (SUSE) wrote:
> On 9/4/26 17:08, Zi Yan wrote:
>> On Fri Sep 4, 2026 at 10:11 AM EDT, Vlastimil Babka (SUSE) wrote:
>>> On 9/4/26 13:56, Salvatore Dipietro wrote:
>>>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>>> introduced high-order folio allocations in the iomap buffered write
>>>> path.  When memory is fragmented, each failed costly-order allocation
>>>> enters __alloc_pages_slowpath() which runs direct compaction and
>>>> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
>>>> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
>>>> 
>>>> The root issue is that direct compaction is too expensive for hot
>>>> allocation paths that have fallbacks to smaller allocations.
>>>> __filemap_get_folio_mpol() already marks higher-order allocations with
>>>> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
>>>> failure.  However, the page allocator still attempts full direct
>>>> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
>>>> aggressive when the caller will simply retry at a lower order.
>>>> 
>>>> For costly-order allocations with __GFP_NORETRY, clear
>>>> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
>>>> can_direct_reclaim, can_compact and the nofail checks are evaluated.
>>>> This makes the entire slowpath treat the request as non-blocking: no
>>>> direct reclaim, no direct compaction and no drain_all_pages() IPI
>>>> across every CPU. kswapd (and in turn kcompactd) is still woken further
>>>> down for background defragmentation, so compaction keeps working for
>>>> long-term system health while being removed from the latency-critical
>>>> direct allocation path.
>>>> 
>>>> Allocations that also request __GFP_THISNODE are exempted.  That flag
>>>> pairing identifies the local-node-first THP attempt issued by
>>>> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
>>>> form transparent huge pages.
>>>> 
>>>> Test environment:
>>>>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>>>>              12x 1TB IO2 32000 IOPS RAID0 XFS
>>>>   OS:        AL2023
>>>>   Kernel:    v7.3-rc1
>>>>   Database:  PostgreSQL 18.4
>>>>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
>>>> 
>>>> Results (average of 3 runs, TPS):
>>>> 
>>>>   Config                   Avg TPS      % vs Baseline
>>>>   baseline (no patch)       59,408       -
>>>>   With this patch          155,409      +161.6%
>>>> 
>>>> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
>>>> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
>>>> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
>>>> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>>> Cc: stable@vger.kernel.org
>>>> Cc: Andrew Morton <akpm@linux-foundation.org>
>>>> Cc: Vlastimil Babka <vbabka@suse.cz>
>>>> Cc: David Hildenbrand <david@redhat.com>
>>>> Cc: Michal Hocko <mhocko@suse.com>
>>>> Cc: Johannes Weiner <hannes@cmpxchg.org>
>>>> Cc: Matthew Wilcox <willy@infradead.org>
>>>> Cc: Christoph Hellwig <hch@lst.de>
>>>> Cc: Dave Chinner <dgc@kernel.org>
>>>> Cc: Ritesh Harjani <ritesh.list@gmail.com>
>>>> Cc: linux-mm@kvack.org
>>>> Cc: linux-fsdevel@vger.kernel.org
>>>> Cc: linux-xfs@vger.kernel.org
>>>> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
>>>
>>> I guess this will have to do until unlikely(we figure out a better API)...
>> 
>> We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on
>> various policies we currently have.
>
> Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.

But alloc_flags are not exposed outside of MM, so alloc_page*() users
are still limited by the existing GFP_* flags. Are you suggesting
alloc_flags could be an addtional input for alloc_page*()?

>
>> Even better if callers can do that
>> instead.
>
> Not sure about burdening the callers (e.g. "every filesystem should do X"
> etc), unless they are some intermediate wrappers within mm itself.

Not all callers. I assume gfp_policy() will only be used by advance
users who want specific page allocation behavior without knowing the
details of the page allocator or changing how the page allocator behave.

> For example, THP (but only anonymous?) is now handled in such a special way,
> it could be a candidate. But I also have a vague feeling this was already
> done in the past. Worth investigating perhaps.
>

The number of wrappers can grow if different users have different needs.
Some might not want direct reclaim, some might not want direct
compaction, some might not want kswapd, and so on. I am not sure we want
to add wrappers for each.

For this patch, the caller can use gfp_policy() to strip out
__GFP_DIRECT_RECLAIM based on the conditions and pass the adjusted gfp
to the page allocator.


-- 
Best Regards,
Yan, Zi

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 9/9/26 04:33, Zi Yan wrote:
> On Mon Sep 7, 2026 at 3:30 AM EDT, Vlastimil Babka (SUSE) wrote:
>> On 9/4/26 17:08, Zi Yan wrote:
>>> On Fri Sep 4, 2026 at 10:11 AM EDT, Vlastimil Babka (SUSE) wrote:
>>>> On 9/4/26 13:56, Salvatore Dipietro wrote:
>>>>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>>>> introduced high-order folio allocations in the iomap buffered write
>>>>> path.  When memory is fragmented, each failed costly-order allocation
>>>>> enters __alloc_pages_slowpath() which runs direct compaction and
>>>>> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
>>>>> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
>>>>> 
>>>>> The root issue is that direct compaction is too expensive for hot
>>>>> allocation paths that have fallbacks to smaller allocations.
>>>>> __filemap_get_folio_mpol() already marks higher-order allocations with
>>>>> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
>>>>> failure.  However, the page allocator still attempts full direct
>>>>> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
>>>>> aggressive when the caller will simply retry at a lower order.
>>>>> 
>>>>> For costly-order allocations with __GFP_NORETRY, clear
>>>>> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
>>>>> can_direct_reclaim, can_compact and the nofail checks are evaluated.
>>>>> This makes the entire slowpath treat the request as non-blocking: no
>>>>> direct reclaim, no direct compaction and no drain_all_pages() IPI
>>>>> across every CPU. kswapd (and in turn kcompactd) is still woken further
>>>>> down for background defragmentation, so compaction keeps working for
>>>>> long-term system health while being removed from the latency-critical
>>>>> direct allocation path.
>>>>> 
>>>>> Allocations that also request __GFP_THISNODE are exempted.  That flag
>>>>> pairing identifies the local-node-first THP attempt issued by
>>>>> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
>>>>> form transparent huge pages.
>>>>> 
>>>>> Test environment:
>>>>>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>>>>>              12x 1TB IO2 32000 IOPS RAID0 XFS
>>>>>   OS:        AL2023
>>>>>   Kernel:    v7.3-rc1
>>>>>   Database:  PostgreSQL 18.4
>>>>>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
>>>>> 
>>>>> Results (average of 3 runs, TPS):
>>>>> 
>>>>>   Config                   Avg TPS      % vs Baseline
>>>>>   baseline (no patch)       59,408       -
>>>>>   With this patch          155,409      +161.6%
>>>>> 
>>>>> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1]
>>>>> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2]
>>>>> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3]
>>>>> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>>>> Cc: stable@vger.kernel.org
>>>>> Cc: Andrew Morton <akpm@linux-foundation.org>
>>>>> Cc: Vlastimil Babka <vbabka@suse.cz>
>>>>> Cc: David Hildenbrand <david@redhat.com>
>>>>> Cc: Michal Hocko <mhocko@suse.com>
>>>>> Cc: Johannes Weiner <hannes@cmpxchg.org>
>>>>> Cc: Matthew Wilcox <willy@infradead.org>
>>>>> Cc: Christoph Hellwig <hch@lst.de>
>>>>> Cc: Dave Chinner <dgc@kernel.org>
>>>>> Cc: Ritesh Harjani <ritesh.list@gmail.com>
>>>>> Cc: linux-mm@kvack.org
>>>>> Cc: linux-fsdevel@vger.kernel.org
>>>>> Cc: linux-xfs@vger.kernel.org
>>>>> Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
>>>>
>>>> I guess this will have to do until unlikely(we figure out a better API)...
>>> 
>>> We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on
>>> various policies we currently have.
>>
>> Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.
> 
> But alloc_flags are not exposed outside of MM, so alloc_page*() users
> are still limited by the existing GFP_* flags. Are you suggesting
> alloc_flags could be an addtional input for alloc_page*()?
> 
>>
>>> Even better if callers can do that
>>> instead.
>>
>> Not sure about burdening the callers (e.g. "every filesystem should do X"
>> etc), unless they are some intermediate wrappers within mm itself.
> 
> Not all callers. I assume gfp_policy() will only be used by advance
> users who want specific page allocation behavior without knowing the
> details of the page allocator or changing how the page allocator behave.

"without knowing the details"

>> For example, THP (but only anonymous?) is now handled in such a special way,
>> it could be a candidate. But I also have a vague feeling this was already
>> done in the past. Worth investigating perhaps.
>>
> 
> The number of wrappers can grow if different users have different needs.
> Some might not want direct reclaim, some might not want direct
> compaction, some might not want kswapd, and so on. I am not sure we want
> to add wrappers for each.

Yet all these are the details?

> For this patch, the caller can use gfp_policy() to strip out
> __GFP_DIRECT_RECLAIM based on the conditions and pass the adjusted gfp
> to the page allocator.

I guess I'd need to see a concrete example to grasp this fully.

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Salvatore Dipietro]

On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:

> Is there anything particularly unusual about this test case?

It is a stock pgbench simple-update PostgreSQL workload on a large
instance (96 vCPUs), using standard PostgreSQL settings and with no huge
pages assigned to the database. We deliberately overprovision the
pgbench clients: 1024 clients over 96 threads. That keeps enough writers
in the buffered write path concurrently to hit the costly-order
allocation failure path continuously. The memory fragmentation comes from 
page tables: PostgreSQL spawns a new process per client, and those page
tables consume ~40% of memory, which significantly limits the page cache
and the free memory available.


> > Results (average of 3 runs, TPS):
> >
> >   Config                   Avg TPS      % vs Baseline
> >   baseline (no patch)       59,408       -
> >   With this patch          155,409      +161.6%
>
> Is this back to pre-5d8edfb900d5 performance?

Yes - fully recovered. Here is the summary, same host and workload
throughout, average of 3 runs:

  Config                                        Avg TPS   % vs baseline
  AL2023 stock 6.1 kernel                       136,942            n/a
  v7.3-rc1 baseline (no patch)                   59,408             -
  v7.3-rc1, 5d8edfb900d5 behaviour reverted     151,184        +154.5%
  v7.3-rc1 + v4                                 155,409        +161.6%

The 6.1 row predates 5d8edfb900d5 entirely. A different kernel version,
so not directly comparable, but it shows the performance this workload
used to get on this host.

A literal "git revert 5d8edfb900d5" does not apply to v7.3-rc1 -
iomap_write_iter() has been rewritten since - so the reverted row is a
one-line behavioural revert, forcing the write loop back to copying at
most PAGE_SIZE per iteration:

  -	size_t chunk = mapping_max_folio_size(mapping);
  +	size_t chunk = PAGE_SIZE;

which is what the pre-5d8edfb900d5 loop computed. That makes
iomap_get_folio() pass no order hint, so the path issues only order-0
allocations.


> AI review asked a few serious-looking questions:
> 	https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it

Thanks for pointing that out. To address them, we can have something
like the patch below. Performance results are still similar to v4. Happy
to submit a formal v5 patch with it if you would like.

  Config                                        Avg TPS   % vs baseline
  v7.3-rc1 baseline (no patch)                   59,408             -
  v7.3-rc1 + v4                                 155,409        +161.6%
  v7.3-rc1 + proposed patch                     161,994        +172.7%

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c48..be8b0d72a2db 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4784,10 +4784,22 @@ static inline struct page *
 __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
-	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
+	/*
+	 * Costly __GFP_NORETRY callers have a cheap fallback to a lower order,
+	 * so don't stall them in direct reclaim or direct compaction.  Exempt
+	 * __GFP_THISNODE (the THP attempt from alloc_pages_mpol() needs direct
+	 * compaction) and __GFP_NOFAIL (must not be made to fail).  Don't
+	 * clear __GFP_DIRECT_RECLAIM from gfp_mask instead: that would also
+	 * change the alloc_flags derived by alloc_flags_slowpath().
+	 */
+	const bool costly_noretry = costly_order &&
+		(gfp_mask & __GFP_NORETRY) &&
+		!(gfp_mask & (__GFP_THISNODE | __GFP_NOFAIL));
+	bool can_direct_reclaim = !costly_noretry &&
+		(gfp_mask & __GFP_DIRECT_RECLAIM);
 	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
 	bool nofail = gfp_mask & __GFP_NOFAIL;
-	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;


Thanks,
Salvatore



AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Andrew Morton]

On Thu, 10 Sep 2026 11:46:02 +0000 Salvatore Dipietro <dipiets@amazon.it> wrote:

> 
> On Sat, 05 Sep 2026 17:42:39 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > Is there anything particularly unusual about this test case?
> 
> It is a stock pgbench simple-update PostgreSQL workload on a large
> instance (96 vCPUs), using standard PostgreSQL settings and with no huge
> pages assigned to the database. We deliberately overprovision the
> pgbench clients: 1024 clients over 96 threads. That keeps enough writers
> in the buffered write path concurrently to hit the costly-order
> allocation failure path continuously. The memory fragmentation comes from 
> page tables: PostgreSQL spawns a new process per client, and those page
> tables consume ~40% of memory, which significantly limits the page cache
> and the free memory available.

OK, thanks.

> 
> > > Results (average of 3 runs, TPS):
> > >
> > >   Config                   Avg TPS      % vs Baseline
> > >   baseline (no patch)       59,408       -
> > >   With this patch          155,409      +161.6%
> >
> > Is this back to pre-5d8edfb900d5 performance?
> 
> Yes - fully recovered.

Great.  That's worth mentioning in the changelog.

> 
> > AI review asked a few serious-looking questions:
> > 	https://sashiko.dev/#/patchset/20260904115629.3993331-1-dipiets@amazon.it
> 
> Thanks for pointing that out. To address them, we can have something
> like the patch below. Performance results are still similar to v4. Happy
> to submit a formal v5 patch with it if you would like.

Yes please, a v5 would be good.