From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9E294CC61A; Wed, 16 Sep 2026 22:35:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789598157; cv=none; b=iUUtlV6GywnYW4Dd1X8dORG4QplZLShVEv8xb07vSza9rdw7oEpJ9Ck8uaDNE2JSTY9Bwgb8pUi+hoVZECmjZPL4mg1dAJwPT+4UZsyaMIo33tZCsyquOjONywZkDv4qjy34LrEZhTvofejk60LwexGuxQGxPV8jbgHzn4zFAck= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789598157; c=relaxed/simple; bh=hTpQv1DiUrsiOgQA0Hk+QbE8a2iCKtBmD0nHiF9SP7Y=; h=Date:From:To:Subject:Message-Id:In-Reply-To:References: Mime-Version:Content-Type; b=W62VI/d0bf2isdlUEOhXNnC2U0eygsUkr9cRU0h8PMXPAojypDVq/KHDNCha7Ow9S2i9uQ5BLLRs7xlFePqjuow8ovr86og/5y9VXZv+sMYFChCBB2ROPJe41d7mh1PW3NC8lNnAMXNL2JkHD3MV8xa7BPWgqD/XSAgb+EkyrbQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=S//aGnBP; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="S//aGnBP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7EBD51F000FF; Wed, 16 Sep 2026 22:35:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1789598141; bh=HGLJYjgnhN5BomAgi3vT5WzHyjvAyhNIqlfJD73iIU0=; h=Date:From:To:Subject:In-Reply-To:References; b=S//aGnBPPGeBvhxSLtAXQy6/qOyf3BT9C340KFYopN6oDtBd14Tli6B4DgxKq7cBM YZV79cvnVzC0ZvMTpy+c2DlqK9OCO1pz/YigbZ7uSxpbddpyvIMMNlFv9rSuQYv+Py Q7DiEaE1wLjJ0f1IvOvNV7f0DFOEViM7CIQYiW74= Date: Wed, 16 Sep 2026 15:35:40 -0700 From: Andrew Morton To: Johannes Weiner , "Vlastimil Babka (SUSE)" , Salvatore Dipietro , abuehaze@amazon.com, alisaidi@amazon.com, blakgeof@amazon.com, brauner@kernel.org, brendan.jackman@linux.dev, david@redhat.com, dgc@kernel.org, dipietro.salvatore@gmail.com, djwong@kernel.org, hch@infradead.org, hch@lst.de, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-xfs@vger.kernel.org, mhocko@suse.com, ritesh.list@gmail.com, rvvandan@amazon.com, stable@vger.kernel.org, surenb@google.com, willy@infradead.org, ziy@nvidia.com Subject: Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations Message-Id: <20260916153540.9a3fb449f5b3fd481ca1b544@linux-foundation.org> In-Reply-To: <20260916153426.3265e2e152cebb6a8d66373d@linux-foundation.org> References: <20260905174239.99e31515fabe220aa7d8e6fa@linux-foundation.org> <20260910114602.926944-1-dipiets@amazon.it> <8d6a8a63-4adc-458a-b548-a47bf5ff8eb7@kernel.org> <20260916153426.3265e2e152cebb6a8d66373d@linux-foundation.org> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Wed, 16 Sep 2026 15:34:26 -0700 Andrew Morton wrote: > On Wed, 16 Sep 2026 11:58:29 -0400 Johannes Weiner wrote: > > > > Unless I'm mistaken about that MIGRATE_HIGHATOMIC part, it seems all sashiko > > > concerns can be dismissed and then indeed v4 is the better version. > > > > It looks like a real issue to me, but one that already exists > > independent of Salvatore's change. > > So I'm hearing that I should drop v5 and revert to v4? > v4: From: Salvatore Dipietro Subject: mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations Date: Fri, 4 Sep 2026 11:56:28 +0000 Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") introduced high-order folio allocations in the iomap buffered write path. When memory is fragmented, each failed costly-order allocation enters __alloc_pages_slowpath() which runs direct compaction and drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system. The root issue is that direct compaction is too expensive for hot allocation paths that have fallbacks to smaller allocations. __filemap_get_folio_mpol() already marks higher-order allocations with __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle failure. However, the page allocator still attempts full direct compaction for costly orders with __GFP_NORETRY, which is unnecessarily aggressive when the caller will simply retry at a lower order. For costly-order allocations with __GFP_NORETRY, clear __GFP_DIRECT_RECLAIM at the very start of the slowpath, before can_direct_reclaim, can_compact and the nofail checks are evaluated. This makes the entire slowpath treat the request as non-blocking: no direct reclaim, no direct compaction and no drain_all_pages() IPI across every CPU. kswapd (and in turn kcompactd) is still woken further down for background defragmentation, so compaction keeps working for long-term system health while being removed from the latency-critical direct allocation path. Allocations that also request __GFP_THISNODE are exempted. That flag pairing identifies the local-node-first THP attempt issued by alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to form transparent huge pages. Test environment: Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64) 12x 1TB IO2 32000 IOPS RAID0 XFS OS: AL2023 Kernel: v7.3-rc1 Database: PostgreSQL 18.4 Workload: pgbench simple-update, 1024 clients, 96 threads, 1200s Results (average of 3 runs, TPS): Config Avg TPS % vs Baseline baseline (no patch) 59,408 - With this patch 155,409 +161.6% Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1] Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2] Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3] Link: https://lore.kernel.org/20260904115629.3993331-1-dipiets@amazon.it Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") Signed-off-by: Salvatore Dipietro Signed-off-by: Andrew Morton Acked-by: Vlastimil Babka (SUSE) Acked-by: Zi Yan Reviewed-by: Johannes Weiner Reviewed-by: Christoph Hellwig Cc: David Hildenbrand Cc: Michal Hocko Cc: Matthew Wilcox Cc: Dave Chinner Cc: Ritesh Harjani Cc: --- mm/page_alloc.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) --- a/mm/page_alloc.c~mm-page_alloc-avoid-direct-compaction-for-costly-__gfp_noretry-allocations +++ a/mm/page_alloc.c @@ -4784,10 +4784,10 @@ static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct alloc_context *ac) { - bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; - bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask); - bool nofail = gfp_mask & __GFP_NOFAIL; const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER; + bool can_direct_reclaim; + bool can_compact; + bool nofail; struct page *page = NULL; unsigned int alloc_flags; unsigned long did_some_progress; @@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, u bool can_retry_reserves = true; unsigned long alloc_start_time = jiffies; + /* + * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall + * them in reclaim or compaction. __GFP_THISNODE callers are exempt. + */ + if (costly_order && (gfp_mask & __GFP_NORETRY) && + !(gfp_mask & __GFP_THISNODE)) + gfp_mask &= ~__GFP_DIRECT_RECLAIM; + + can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; + can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask); + nofail = gfp_mask & __GFP_NOFAIL; + if (unlikely(nofail)) { /* * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM, _