From: "Gregory Price (Meta)" <gourry@gourry.net>
To: linux-mm@kvack.org
Cc: linux-trace-kernel@vger.kernel.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com, akpm@linux-foundation.org,
david@kernel.org, ljs@kernel.org, liam@infradead.org,
vbabka@kernel.org, rppt@kernel.org, surenb@google.com,
mhocko@suse.com, kasong@tencent.com, qi.zheng@linux.dev,
shakeel.butt@linux.dev, baohua@kernel.org,
axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com,
baoquan.he@linux.dev, baolin.wang@linux.alibaba.com,
rientjes@google.com, rostedt@goodmis.org, mhiramat@kernel.org,
mathieu.desnoyers@efficios.com, muchun.song@linux.dev,
osalvador@suse.de, gourry@gourry.net, ziy@nvidia.com,
joshua.hahnjy@gmail.com, rakie.kim@sk.com,
ying.huang@linux.alibaba.com, brendan.jackman@linux.dev,
hannes@cmpxchg.org, bigeasy@linutronix.de, clrkwllms@kernel.org,
nathan@kernel.org, ndesaulniers@google.com, morbo@google.com,
justinstitt@google.com, Bessel.Wang@huawei.com,
jiang.liu@huawei.com, mgorman@techsingularity.net,
anshuman.khandual@arm.com, stable@vger.kernel.org
Subject: [PATCH v5 08/12] mm/page_alloc: retry allocations that raced with a zonelist rebuild
Date: Sat, 31 Oct 2026 22:13:36 -0400 [thread overview]
Message-ID: <20261101021340.2251928-9-gourry@gourry.net> (raw)
In-Reply-To: <20261101021340.2251928-1-gourry@gourry.net>
Onlining the first block of a zone rebuilds every zonelist in place,
shifting the slots after the new zone. The iterator approves a slot,
then the allocator loads the zone from it, so a GFP_KERNEL request can
land in ZONE_MOVABLE and a nodemask request on a node outside its mask:
CPU 0 CPU 1
prepare_alloc_pages()
first_zones_zonelist(nodemask {1})
slot 0 (NORMAL1) approved
online_pages()
build_all_zonelists()
[NORMAL1, NORMAL2, ...]
-> [MOVABLE1, NORMAL1, ...]
get_page_from_freelist()
zone = zonelist_zone(slot 0) /* MOVABLE1 */
allocates from MOVABLE1
zonelist_update_seq detects the rebuild, but only the slowpath's failure
exits check it.
Check it whenever an allocation succeeds too, and if it changed, free
the page and retry, as the memcg charge failure path does. Eligible
pages are dropped as well: zonelists only change when memory comes or
goes, and a retry is simpler than rechecking the page.
Freeing the page does not undo a block the allocation claimed. A
GFP_KERNEL request that lands in a fresh ZONE_MOVABLE converts a free
block to MIGRATE_UNMOVABLE, which stays until the zone's movable free
list runs out and a movable allocation takes it back.
The fast and bulk paths read the seqlock's raw seqcount without waiting:
an ALLOC_NOLOCK caller can be an NMI that interrupted the writer, and on
PREEMPT_RT the seqlock accessor takes the spinlock. A sample taken
mid-write costs a retry, or NULL under ALLOC_NOLOCK.
check_retry_zonelist() returns 0 when compiled out, so a backport
without the hot-add change stays harmless.
Fixes: 11cd8638c37f ("mm, page_alloc: remove stop_machine from build_all_zonelists")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/page_alloc.c | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 28d46f987685..f2f7e96bdef2 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4419,12 +4419,25 @@ static unsigned int zonelist_iter_begin(void)
return 0;
}
+/*
+ * Never wait for the writer, which an ALLOC_NOLOCK caller in NMI may have
+ * interrupted. Read the raw seqcount: on PREEMPT_RT the seqlock accessor takes
+ * the spinlock while a write is in progress.
+ */
+static unsigned int zonelist_iter_sample(void)
+{
+ if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
+ return raw_seqcount_begin(&zonelist_update_seq.seqcount.seqcount);
+
+ return 0;
+}
+
static unsigned int check_retry_zonelist(unsigned int seq)
{
if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
return read_seqretry(&zonelist_update_seq, seq);
- return seq;
+ return 0;
}
/* Perform direct synchronous page reclaim */
@@ -5100,6 +5113,12 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
warn_alloc(gfp_mask, ac->nodemask,
"page allocation failure: order:%u", order);
got_pg:
+ /* A rebuild during the walk may have handed out an ineligible zone. */
+ if (page && check_retry_zonelist(zonelist_iter_cookie)) {
+ __free_frozen_pages(page, order, FPI_NONE);
+ page = NULL;
+ goto restart;
+ }
return page;
}
@@ -5184,6 +5203,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
struct list_head *pcp_list;
struct alloc_context ac;
unsigned int alloc_flags = ALLOC_WMARK_LOW;
+ unsigned int zonelist_iter_cookie;
int nr_populated = 0, nr_account = 0;
/*
@@ -5224,6 +5244,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
/* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
gfp &= gfp_allowed_mask;
gfp = current_gfp_context(gfp);
+ zonelist_iter_cookie = zonelist_iter_sample();
if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
goto out;
@@ -5273,6 +5294,10 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
if (unlikely(!zone))
goto failed;
+ /* The zone was chosen from a zonelist that a rebuild may have changed. */
+ if (unlikely(check_retry_zonelist(zonelist_iter_cookie)))
+ goto failed;
+
/* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */
pcp = pcp_spin_trylock(zone->per_cpu_pageset);
if (!pcp)
@@ -5413,6 +5438,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
.alloc_flags = alloc_flags,
};
unsigned int fastpath_alloc_flags = alloc_flags;
+ unsigned int zonelist_iter_cookie;
/* Other flags could be supported later if needed. */
if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
@@ -5442,6 +5468,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
*/
gfp = current_gfp_context(gfp);
alloc_gfp = gfp;
+ zonelist_iter_cookie = zonelist_iter_sample();
if (!prepare_alloc_pages(gfp, order, preferred_nid, nodemask, &ac,
&alloc_gfp, &fastpath_alloc_flags))
return NULL;
@@ -5458,6 +5485,11 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
/* First allocation attempt (or, for nolock, only attempt) */
page = get_page_from_freelist(alloc_gfp, order, fastpath_alloc_flags, &ac);
+ if (likely(page) && unlikely(check_retry_zonelist(zonelist_iter_cookie))) {
+ __free_frozen_pages(page, order,
+ alloc_flags & ALLOC_NOLOCK ? FPI_NOLOCK : 0);
+ page = NULL;
+ }
if (likely(page) || (alloc_flags & ALLOC_NOLOCK))
goto out;
--
2.56.0
next prev parent reply other threads:[~2026-11-01 2:14 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 01/12] mm/page_alloc: reset can_retry_reserves when the slowpath restarts Gregory Price (Meta)
2026-11-01 19:52 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 02/12] mm/page_alloc: keep the pagesets of an empty zone Gregory Price (Meta)
2026-11-01 20:07 ` Zi Yan
2026-11-03 14:57 ` Vlastimil Babka (SUSE)
2026-11-03 16:04 ` Zi Yan
2026-11-05 9:58 ` Vlastimil Babka (SUSE)
2026-11-05 16:10 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 03/12] mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone Gregory Price (Meta)
2026-11-01 20:28 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 04/12] mm/page_alloc: detect zonelist rebuilds from memory hot-add Gregory Price (Meta)
2026-11-01 20:31 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 05/12] mm/mmzone: read the zoneref slot once in zonelist_zone() Gregory Price (Meta)
2026-11-03 2:54 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 06/12] mm/page_alloc: snapshot the preferred zone against shrink rebuilds Gregory Price (Meta)
2026-11-03 2:56 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 07/12] mm/mmzone: read each zoneref slot once when filtering by nodemask Gregory Price (Meta)
2026-11-03 3:01 ` Zi Yan
2026-11-03 15:00 ` Vlastimil Babka (SUSE)
2026-11-04 6:25 ` Gregory Price
2026-11-01 2:13 ` Gregory Price (Meta) [this message]
2026-11-01 2:13 ` [PATCH v5 09/12] mm/hugetlb: retry the pool walk when the zonelist is rebuilt Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 10/12] mm/page_alloc: retry alloc_contig_pages() " Gregory Price (Meta)
2026-11-03 3:13 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 11/12] mm/mempolicy: read the zoneref slot once in mempolicy_slab_node() Gregory Price (Meta)
2026-11-03 3:14 ` Zi Yan
2026-11-01 2:13 ` [PATCH v5 12/12] mm/mmzone: remove zonelist_node_idx() Gregory Price (Meta)
2026-11-03 3:15 ` Zi Yan
2026-11-01 11:00 ` [PATCH v5 0/12] mm: fix allocator races with memory hotplug Vlastimil Babka (SUSE)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261101021340.2251928-9-gourry@gourry.net \
--to=gourry@gourry.net \
--cc=Bessel.Wang@huawei.com \
--cc=akpm@linux-foundation.org \
--cc=anshuman.khandual@arm.com \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=bigeasy@linutronix.de \
--cc=brendan.jackman@linux.dev \
--cc=clrkwllms@kernel.org \
--cc=david@kernel.org \
--cc=hannes@cmpxchg.org \
--cc=jiang.liu@huawei.com \
--cc=joshua.hahnjy@gmail.com \
--cc=justinstitt@google.com \
--cc=kasong@tencent.com \
--cc=kernel-team@meta.com \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=ljs@kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mgorman@techsingularity.net \
--cc=mhiramat@kernel.org \
--cc=mhocko@suse.com \
--cc=morbo@google.com \
--cc=muchun.song@linux.dev \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=osalvador@suse.de \
--cc=qi.zheng@linux.dev \
--cc=rakie.kim@sk.com \
--cc=rientjes@google.com \
--cc=rostedt@goodmis.org \
--cc=rppt@kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=stable@vger.kernel.org \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
--cc=weixugc@google.com \
--cc=ying.huang@linux.alibaba.com \
--cc=yuanchu@google.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®