* [PATCH v5 0/12] mm: fix allocator races with memory hotplug
@ 2026-11-01 2:13 Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 01/12] mm/page_alloc: reset can_retry_reserves when the slowpath restarts Gregory Price (Meta)
` (12 more replies)
0 siblings, 13 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual
Memory hotplug rebuilds zonelists in place while allocations walk them,
and offlining an emptied zone frees pagesets the vmstat worker can still
hold.
These eleven fixes cover the walkers that crash, fail, or pick the wrong
zone or node as a result, and a twelfth patch removes a helper that
invites the same bug.
1. a restarted slowpath loses its retry with reserves
2. offlining an emptied zone frees pagesets still in use
3. the reclaim_retry_zone tracepoint rereads a zoneref slot
4. the zonelist retry is compiled out without MEMORY_HOTREMOVE
5. zonelist_zone() may load its slot more than once
6. NULL preferred zone after a shrink rebuild
7. __next_zones_zonelist() rereads a slot
8. allocations land outside their zone or nodemask
9. the hugetlb pool walk takes another node's page
10. alloc_contig_pages() takes a range outside its filters
11. mempolicy_slab_node() rereads a slot
12. zonelist_node_idx() invites the same double read
Patches 1-11 are for stable. Patch 12 is a cleanup and is not.
Patches 8-10 need patch 4, without which their retry is compiled
out on HOTREMOVE=n kernels.
Patch 8 needs patch 1.
Patches 6, 7 and 11 need patch 5, which makes zonelist_zone() read
the slot once. Patch 12 needs patches 7 and 11, which remove the
other callers of zonelist_node_idx().
Patches 2 and 3 are patches from anthropic, pulled in from Vlastimil's
series to keep the backport issues self-contained on the series.
Testing
=======
Each race was reproduced on unmodified source in a QEMU guest with three
NUMA nodes and a 512M pc-dimm on node 1, left offline at boot.
A test module widens each window with a kprobe that stalls one CPU inside
it, as an interrupt or a vCPU preemption would.
base this series
patch outcome runs outcome runs
===== ======================= ===== ======================= =====
1 NULL returned 3/3 page from reserves 3/3
2 UAF 6/6 no UAF 3/3
3 NULL dereference 3/3 allocation succeeds 3/3
4 allocation fails 3/3 waits for the rebuild 3/3
6 NULL dereference 3/3 ZONE_NORMAL page 3/3
7 NULL dereference 3/3 ZONE_NORMAL page 3/3
8 wrong zone or node 18/18 right zone and node 18/18
9 page from node 1 3/3 page from node 2 3/3
10 1G page in ZONE_MOVABLE 3/3 1G page in ZONE_NORMAL 3/3
11 NULL dereference 4/4 keeps allocating 3/3
Cost (these tests were run on v4, v5 hasn't substantially changed)
====
Kernel instructions per order-0 allocation and free in the QEMU guest,
counted with a task-bound hardware instruction counter around 1M
iterations on a pinned kthread, non-debug kernel, mean of 15 rounds.
instructions per alloc+free
base this series
alloc_pages() 890.5 900.5
bulk, nr=1, nodemask 865.5 875.5
bulk, nr=16, per page 477.2 478.1
Bare metal, two-socket Xeon Gold 6138 (Skylake-SP, 80 CPUs).
will-it-scale page_fault1-3: each iteration touches one page, so
the unit is millions of faults per second, summed over all tasks.
Base, millions of faults per second:
tasks 1 10 40 80
page_fault1, processes 0.46 4.16 10.51 9.23
page_fault1, threads 0.46 3.51 4.35 4.19
page_fault2, processes 0.43 3.66 9.02 8.33
page_fault2, threads 0.43 3.48 4.37 4.19
page_fault3, processes 3.89 14.78 8.10 8.00
page_fault3, threads 3.87 14.29 6.88 7.96
Change in faults per second with this series, + is faster:
tasks 1 10 40 80
page_fault1, processes -0.3% +0.1% -3.0% -1.7%
page_fault1, threads -0.5% -1.6% -0.9% -0.9%
page_fault2, processes +0.0% +0.9% -1.4% -0.5%
page_fault2, threads +0.3% +1.5% -1.4% -1.0%
page_fault3, processes -1.0% -3.8% -0.7% +1.2%
page_fault3, threads -0.9% -0.7% +0.8% -0.2%
The boot-to-boot stddev is 1.0-1.8% at 1 task and up to
15% in 10-80 tasks, so basically this is all within noise.
Changes in v5
=============
- Pulled in patches from Vlastimil's series (patches 2 and 3 here)
- inline functions (Vlastimil)
- Added patch 5: zonelist_zone() reads the slot with READ_ONCE(), as
recommended by sashiko on v4's mempolicy patch.
Patches 6, 7 and 11 call it instead of open-coding the load.
- Added patch 12, removing zonelist_node_idx(). Not for stable.
- Patch 1: reworded the changelog (Vlastimil).
- Collected Reviewed-by tags.
Link: https://lore.kernel.org/linux-mm/20261028211930.3504634-1-gourry@gourry.net/
Link: https://lore.kernel.org/linux-mm/20261027-pgalloc-ai-hotfixes-v1-0-9c7d1aca801e@kernel.org/
Link: https://lore.kernel.org/linux-mm/20261014222456.2318284-1-gourry@gourry.net/
Link: https://lore.kernel.org/linux-mm/20261015210539.3243563-1-gourry@gourry.net/
Link: https://lore.kernel.org/linux-mm/20261019172346.3344005-1-gourry@gourry.net/
Gregory Price (Meta) (10):
mm/page_alloc: reset can_retry_reserves when the slowpath restarts
mm/page_alloc: detect zonelist rebuilds from memory hot-add
mm/mmzone: read the zoneref slot once in zonelist_zone()
mm/page_alloc: snapshot the preferred zone against shrink rebuilds
mm/mmzone: read each zoneref slot once when filtering by nodemask
mm/page_alloc: retry allocations that raced with a zonelist rebuild
mm/hugetlb: retry the pool walk when the zonelist is rebuilt
mm/page_alloc: retry alloc_contig_pages() when the zonelist is rebuilt
mm/mempolicy: read the zoneref slot once in mempolicy_slab_node()
mm/mmzone: remove zonelist_node_idx()
Josef Bacik (2):
mm/page_alloc: keep the pagesets of an empty zone
mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone
include/linux/mmzone.h | 9 +---
include/trace/events/oom.h | 8 +--
mm/hugetlb.c | 13 +++--
mm/mempolicy.c | 6 ++-
mm/mmzone.c | 11 ++--
mm/page_alloc.c | 107 +++++++++++++++++++++++++++----------
mm/page_alloc.h | 19 +++++++
7 files changed, 123 insertions(+), 50 deletions(-)
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 01/12] mm/page_alloc: reset can_retry_reserves when the slowpath restarts
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 02/12] mm/page_alloc: keep the pagesets of an empty zone Gregory Price (Meta)
` (11 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
Commit 2c4c3e29897d ("mm/page_alloc: simplify __alloc_pages_slowpath()
flow") retries an allocation once with reserve_flags when the caller
becomes eligible for reserves, guarded by can_retry_reserves.
A restart after a cpuset update or a zonelist rebuild retries with
can_retry_reserves already set to false, so a caller that cannot
reclaim (GFP_NOWAIT | __GFP_MEMALLOC), fails without trying reserves.
Reset can_retry_reserves on restart. Only a cpuset or zonelist change
restarts the slowpath, so the retry still runs at most once per change.
With a kprobe helping to make the race deterministic, an allocation
can be made to return NULL. With this patch the allocation will succeed.
Fixes: 2c4c3e29897d ("mm/page_alloc: simplify __alloc_pages_slowpath() flow")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
mm/page_alloc.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a8cac5e01066..f1048168c90c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4783,7 +4783,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
unsigned int zonelist_iter_cookie;
int reserve_flags;
bool compact_first = false;
- bool can_retry_reserves = true;
+ bool can_retry_reserves;
unsigned long alloc_start_time = jiffies;
if (unlikely(nofail)) {
@@ -4805,6 +4805,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
no_progress_loops = 0;
compact_result = COMPACT_SKIPPED;
compact_priority = DEF_COMPACT_PRIORITY;
+ can_retry_reserves = true;
cpuset_mems_cookie = read_mems_allowed_begin();
zonelist_iter_cookie = zonelist_iter_begin();
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 02/12] mm/page_alloc: keep the pagesets of an empty zone
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 01/12] mm/page_alloc: reset can_retry_reserves when the slowpath restarts Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 03/12] mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone Gregory Price (Meta)
` (10 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, Josef Bacik,
stable
From: Josef Bacik <josef@toxicpanda.com>
When the last memory of a zone goes offline, zone_pcp_reset() frees
zone->per_cpu_pageset and zone->per_cpu_zonestats. Nothing waits for those
who found the zone populated and still use the two pointers. They take
neither the memory hotplug lock nor any other lock which offline_pages()
holds, and some of them can sleep:
CPU0 vmstat_update() CPU1 offline_pages()
refresh_cpu_vm_stats()
for_each_populated_zone(zone)
pcp = zone->per_cpu_pageset
cond_resched()
adjust_present_page_count()
zone_pcp_reset(zone)
free_percpu(zone->per_cpu_pageset)
decay_pcp_high(zone, pcp)
__this_cpu_write(pcp->expire, 0)
The same goes for drain_all_pages(NULL), which takes pcp->lock, for
zone_pcp_update(), show_free_areas() and the readers and writers of the
zonestats. If the per-cpu memory has got a new owner in the meantime, it
is corrupted.
The pagesets are only freed because setup_zone_pageset() would otherwise
leak them when the zone gets memory again. Keep them instead and let
setup_zone_pageset() and zone_pcp_init() use them again, like it is done
with pgdat->per_cpu_nodestats. Their lists are empty, and their limits
have been calculated for a zone without memory by the
init_per_zone_wmark_min() of offline_pages(), which is what
setup_zone_pageset() would come up with as well.
Fixes: 340175b7d14d ("mm/hotplug: free zone->pageset when a zone becomes empty")
Cc: <stable@vger.kernel.org>
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
mm/page_alloc.c | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f1048168c90c..1850a41f24f7 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6308,6 +6308,10 @@ void __meminit setup_zone_pageset(struct zone *zone)
{
int cpu;
+ /* The zone has been populated before, see zone_pcp_reset(). */
+ if (zone->per_cpu_pageset != &boot_pageset)
+ return;
+
/* Size may be 0 on !SMP && !NUMA */
if (sizeof(struct per_cpu_zonestat) > 0)
zone->per_cpu_zonestats = alloc_percpu(struct per_cpu_zonestat);
@@ -6400,6 +6404,10 @@ void __init setup_per_cpu_pageset(void)
__meminit void zone_pcp_init(struct zone *zone)
{
+ /* A node which is added again, see zone_pcp_reset(). */
+ if (zone->per_cpu_pageset && zone->per_cpu_pageset != &boot_pageset)
+ return;
+
/*
* per cpu subsystem is not up at this point. The following code
* relies on the ability of the linker to provide the
@@ -7680,6 +7688,12 @@ void zone_pcp_enable(struct zone *zone)
mutex_unlock(&pcp_batch_high_lock);
}
+/*
+ * The zone has become empty. Its pagesets and zonestats are not freed: nothing
+ * keeps somebody who found the zone populated, such as refresh_cpu_vm_stats()
+ * or drain_all_pages(), from still using them. They are used again when the
+ * zone gets memory again.
+ */
void zone_pcp_reset(struct zone *zone)
{
int cpu;
@@ -7690,12 +7704,6 @@ void zone_pcp_reset(struct zone *zone)
pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu);
drain_zonestat(zone, pzstats);
}
- free_percpu(zone->per_cpu_pageset);
- zone->per_cpu_pageset = &boot_pageset;
- if (zone->per_cpu_zonestats != &boot_zonestats) {
- free_percpu(zone->per_cpu_zonestats);
- zone->per_cpu_zonestats = &boot_zonestats;
- }
}
}
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 03/12] mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 01/12] mm/page_alloc: reset can_retry_reserves when the slowpath restarts Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 02/12] mm/page_alloc: keep the pagesets of an empty zone Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 04/12] mm/page_alloc: detect zonelist rebuilds from memory hot-add Gregory Price (Meta)
` (9 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, Josef Bacik,
stable
From: Josef Bacik <josef@toxicpanda.com>
should_reclaim_retry() walks a zonelist and passes the zoneref pointer
to the reclaim_retry_zone tracepoint, which loads the zone from the
entry again to find the node.
Memory hotplug rebuilds the zonelists in place while allocations are
under way. When a zone in front of the zoneref entry has become empty,
the entries move up, and this one can be the end marker by the time the
tracepoint looks at it. zonelist_node_idx() then dereferences a NULL
zone. Short of that, the event can name another zone than the one its
numbers belong to.
Pass the zone pointer instead of the zoneref pointer.
Fixes: d379f01de095 ("oom, trace: add oom detection tracepoints")
Cc: <stable@vger.kernel.org>
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
include/trace/events/oom.h | 8 ++++----
mm/page_alloc.c | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/include/trace/events/oom.h b/include/trace/events/oom.h
index 9f0a5d1482c4..cbcf6fe4cc61 100644
--- a/include/trace/events/oom.h
+++ b/include/trace/events/oom.h
@@ -33,7 +33,7 @@ TRACE_EVENT(oom_score_adj_update,
TRACE_EVENT(reclaim_retry_zone,
- TP_PROTO(struct zoneref *zoneref,
+ TP_PROTO(struct zone *zone,
int order,
unsigned long reclaimable,
unsigned long available,
@@ -41,7 +41,7 @@ TRACE_EVENT(reclaim_retry_zone,
int no_progress_loops,
bool wmark_check),
- TP_ARGS(zoneref, order, reclaimable, available, min_wmark, no_progress_loops, wmark_check),
+ TP_ARGS(zone, order, reclaimable, available, min_wmark, no_progress_loops, wmark_check),
TP_STRUCT__entry(
__field( int, node)
@@ -55,8 +55,8 @@ TRACE_EVENT(reclaim_retry_zone,
),
TP_fast_assign(
- __entry->node = zonelist_node_idx(zoneref);
- __entry->zone_idx = zonelist_zone_idx(zoneref);
+ __entry->node = zone_to_nid(zone);
+ __entry->zone_idx = zone_idx(zone);
__entry->order = order;
__entry->reclaimable = reclaimable;
__entry->available = available;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1850a41f24f7..e14d550d4128 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4670,7 +4670,7 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
*/
wmark = __zone_watermark_ok(zone, order, min_wmark,
ac->highest_zoneidx, alloc_flags, available);
- trace_reclaim_retry_zone(z, order, reclaimable,
+ trace_reclaim_retry_zone(zone, order, reclaimable,
available, min_wmark, *no_progress_loops, wmark);
if (wmark) {
ret = true;
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 04/12] mm/page_alloc: detect zonelist rebuilds from memory hot-add
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (2 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 03/12] mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 05/12] mm/mmzone: read the zoneref slot once in zonelist_zone() Gregory Price (Meta)
` (8 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
The zonelist_update_seq retry was limited to CONFIG_MEMORY_HOTREMOVE,
presumably because only hot-remove takes zones out of a zonelist.
However, build_zonerefs_node() rewrites the zonelists in place, so
onlining the first block of a zone shifts the entries that follow it
in every zonelist containing that zone. This can cause a spurious
allocation failure for a __GFP_THISNODE caller.
For a node with only ZONE_NORMAL, onlining its first ZONE_MOVABLE block
rewrites the node's ZONELIST_NOFALLBACK list as:
[NORMAL, NULL] -> [MOVABLE, NULL] -> [MOVABLE, NORMAL, NULL]
A concurrent GFP_KERNEL | __GFP_THISNODE allocation can observe the
intermediate list, skip ZONE_MOVABLE, and find no eligible zone. With
CONFIG_MEMORY_HOTREMOVE=n, check_retry_zonelist() returns 0 and the
allocation fails even though the node has free ZONE_NORMAL memory.
Gate the sequence checks on CONFIG_MEMORY_HOTPLUG, which covers every
configuration that rebuilds zonelists after boot.
Fixes: 3d36424b3b58 ("mm/page_alloc: fix race condition between build_all_zonelists and page allocation")
Cc: stable@vger.kernel.org
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
mm/page_alloc.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index e14d550d4128..11be12b443ad 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4404,7 +4404,7 @@ static DEFINE_SEQLOCK(zonelist_update_seq);
static unsigned int zonelist_iter_begin(void)
{
- if (IS_ENABLED(CONFIG_MEMORY_HOTREMOVE))
+ if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
return read_seqbegin(&zonelist_update_seq);
return 0;
@@ -4412,7 +4412,7 @@ static unsigned int zonelist_iter_begin(void)
static unsigned int check_retry_zonelist(unsigned int seq)
{
- if (IS_ENABLED(CONFIG_MEMORY_HOTREMOVE))
+ if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
return read_seqretry(&zonelist_update_seq, seq);
return seq;
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 05/12] mm/mmzone: read the zoneref slot once in zonelist_zone()
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (3 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 04/12] mm/page_alloc: detect zonelist rebuilds from memory hot-add Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 06/12] mm/page_alloc: snapshot the preferred zone against shrink rebuilds Gregory Price (Meta)
` (7 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
Memory hotplug rebuilds zonelists in place while allocations walk them,
so a zoneref slot can change between two loads of it.
zonelist_zone() loads the slot with a plain access. The compiler may
load it again later instead of keeping the zone it already read, so a
caller can check one zone and then use another, or the NULL terminator.
Use READ_ONCE(). Callers that take a zone from a slot and keep using
it, such as get_page_from_freelist() with ac->preferred_zoneref, need
that single load.
Fixes: 11cd8638c37f ("mm, page_alloc: remove stop_machine from build_all_zonelists")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
include/linux/mmzone.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecd1ebd49756..4bc2318ac37f 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1855,7 +1855,8 @@ extern struct zone *next_zone(struct zone *zone);
static inline struct zone *zonelist_zone(struct zoneref *zoneref)
{
- return zoneref->zone;
+ /* Memory hotplug rewrites zonelists in place. */
+ return READ_ONCE(zoneref->zone);
}
static inline int zonelist_zone_idx(const struct zoneref *zoneref)
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 06/12] mm/page_alloc: snapshot the preferred zone against shrink rebuilds
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (4 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 05/12] mm/mmzone: read the zoneref slot once in zonelist_zone() Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 07/12] mm/mmzone: read each zoneref slot once when filtering by nodemask Gregory Price (Meta)
` (6 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
get_page_from_freelist() and alloc_pages_bulk_noprof() reread
ac->preferred_zoneref->zone at every use. Offlining a zone's last block
rebuilds the zonelists in place, and the shrunk list's NULL terminator
can land on the preferred slot mid-walk:
CPU 0 CPU 1
get_page_from_freelist(GFP_KERNEL | __GFP_THISNODE)
zone = NORMAL (slot 1)
offline_pages()
build_all_zonelists()
[MOVABLE, NORMAL, NULL]
-> [NORMAL, NULL]
rmqueue(zonelist_zone(slot 1), zone)
zone_statistics(NULL, zone)
BUG: kernel NULL pointer dereference, address: 0000000000000060
RIP: 0010:get_page_from_freelist+0x6a5/0x15f0
Call Trace:
__alloc_frozen_pages_noprof+0x125/0x330
Read the slot once. Zones are never freed, so a stale zone is safe, and
a NULL slot fails the same way as walking off the end.
Fixes: 11cd8638c37f ("mm, page_alloc: remove stop_machine from build_all_zonelists")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/page_alloc.c | 31 +++++++++++++++++++++++--------
1 file changed, 23 insertions(+), 8 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 11be12b443ad..28d46f987685 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3794,12 +3794,21 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
{
struct zoneref *z;
struct zone *zone;
+ struct zone *preferred_zone;
struct pglist_data *last_pgdat = NULL;
bool last_pgdat_dirty_ok = false;
bool no_fallback;
bool skip_kswapd_nodes = nr_online_nodes > 1;
bool skipped_kswapd_nodes = false;
+ /*
+ * A concurrent zonelist rebuild can NULL or replace this slot. Zones
+ * are never freed, so reading it once keeps a usable zone.
+ */
+ preferred_zone = zonelist_zone(ac->preferred_zoneref);
+ if (!preferred_zone)
+ return NULL;
+
retry:
/*
* Scan zonelist, looking for a zone with enough free.
@@ -3846,7 +3855,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
}
if (no_fallback && !defrag_mode && nr_online_nodes > 1 &&
- zone != zonelist_zone(ac->preferred_zoneref)) {
+ zone != preferred_zone) {
int local_nid;
/*
@@ -3854,7 +3863,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
* fragmenting fallbacks. Locality is more important
* than fragmentation avoidance.
*/
- local_nid = zonelist_node_idx(ac->preferred_zoneref);
+ local_nid = zone_to_nid(preferred_zone);
if (zone_to_nid(zone) != local_nid) {
alloc_flags &= ~ALLOC_NOFRAGMENT;
goto retry;
@@ -3916,7 +3925,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
goto try_this_zone;
if (!node_reclaim_enabled() ||
- !zone_allows_reclaim(zonelist_zone(ac->preferred_zoneref), zone))
+ !zone_allows_reclaim(preferred_zone, zone))
continue;
if (!node_reclaim(zone->zone_pgdat, gfp_mask, order))
@@ -3929,7 +3938,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
}
try_this_zone:
- page = rmqueue(zonelist_zone(ac->preferred_zoneref), zone, order,
+ page = rmqueue(preferred_zone, zone, order,
gfp_mask, alloc_flags, ac->migratetype);
if (page) {
prep_new_page(page, order, gfp_mask, alloc_flags);
@@ -5169,6 +5178,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
{
struct page *page;
struct zone *zone;
+ struct zone *preferred_zone;
struct zoneref *z;
struct per_cpu_pages *pcp;
struct list_head *pcp_list;
@@ -5217,6 +5227,11 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
goto out;
+ /* Read once; see get_page_from_freelist(). */
+ preferred_zone = zonelist_zone(ac.preferred_zoneref);
+ if (!preferred_zone)
+ goto failed;
+
/* Find an allowed local zone that meets the low watermark. */
z = ac.preferred_zoneref;
for_next_zone_zonelist_nodemask(zone, z, ac.highest_zoneidx, ac.nodemask) {
@@ -5227,8 +5242,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
continue;
}
- if (nr_online_nodes > 1 && zone != zonelist_zone(ac.preferred_zoneref) &&
- zone_to_nid(zone) != zonelist_node_idx(ac.preferred_zoneref)) {
+ if (nr_online_nodes > 1 && zone != preferred_zone &&
+ zone_to_nid(zone) != zone_to_nid(preferred_zone)) {
goto failed;
}
@@ -5236,7 +5251,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
retry_this_zone:
mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages - nr_populated;
if (zone_watermark_fast(zone, 0, mark,
- zonelist_zone_idx(ac.preferred_zoneref),
+ zone_idx(preferred_zone),
alloc_flags, gfp)) {
break;
}
@@ -5295,7 +5310,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
pcp_spin_unlock(pcp);
__count_zid_vm_events(PGALLOC, zone_idx(zone), nr_account);
- zone_statistics(zonelist_zone(ac.preferred_zoneref), zone, nr_account);
+ zone_statistics(preferred_zone, zone, nr_account);
out:
return nr_populated;
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 07/12] mm/mmzone: read each zoneref slot once when filtering by nodemask
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (5 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 06/12] mm/page_alloc: snapshot the preferred zone against shrink rebuilds Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 08/12] mm/page_alloc: retry allocations that raced with a zonelist rebuild Gregory Price (Meta)
` (5 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
__next_zones_zonelist() tests a slot's zone for NULL, then reads the
slot again to look up the zone's node. Offlining a zone's last block
rebuilds the zonelists in place, so the second read can return the
terminator:
CPU 0 CPU 1
__next_zones_zonelist(nodemask)
zonelist_zone(slot 1) = NORMAL
offline_pages()
build_all_zonelists()
[MOVABLE, NORMAL, NULL]
-> [NORMAL, NULL]
zref_in_nodemask(slot 1)
zone_to_nid(NULL)
Current GCC and clang merge the two reads, but nothing requires it.
With mm/mmzone.o built with -fno-inline -fno-ipa-sra, holding a
nodemask allocation between the reads while the node's last movable
block goes offline oopses every time:
BUG: kernel NULL pointer dereference, address: 0000000000000060
RIP: 0010:zone_to_nid+0x0/0x10
Call Trace:
zref_in_nodemask+0x9/0x20
__next_zones_zonelist+0x23/0x70
prepare_alloc_pages.constprop.0+0x16c/0x190
__alloc_frozen_pages_noprof+0xaf/0x330
Read the slot once and test that zone. A NULL zone still ends the walk.
Fixes: 11cd8638c37f ("mm, page_alloc: remove stop_machine from build_all_zonelists")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/mmzone.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/mm/mmzone.c b/mm/mmzone.c
index 9cc9ef588580..65880396f104 100644
--- a/mm/mmzone.c
+++ b/mm/mmzone.c
@@ -43,13 +43,14 @@ struct zone *next_zone(struct zone *zone)
return zone;
}
-static inline int zref_in_nodemask(struct zoneref *zref,
- const nodemask_t *nodes)
+/* A NULL zone terminates the zonelist, so it ends the search. */
+static inline bool zone_in_nodemask(const struct zone *zone,
+ const nodemask_t *nodes)
{
#ifdef CONFIG_NUMA
- return node_isset(zonelist_node_idx(zref), *nodes);
+ return !zone || node_isset(zone_to_nid(zone), *nodes);
#else
- return 1;
+ return true;
#endif /* CONFIG_NUMA */
}
@@ -67,7 +68,7 @@ struct zoneref *__next_zones_zonelist(struct zoneref *z,
z++;
else
while (zonelist_zone_idx(z) > highest_zoneidx ||
- (zonelist_zone(z) && !zref_in_nodemask(z, nodes)))
+ !zone_in_nodemask(zonelist_zone(z), nodes))
z++;
return z;
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 08/12] mm/page_alloc: retry allocations that raced with a zonelist rebuild
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (6 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 07/12] mm/mmzone: read each zoneref slot once when filtering by nodemask Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 09/12] mm/hugetlb: retry the pool walk when the zonelist is rebuilt Gregory Price (Meta)
` (4 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
Onlining the first block of a zone rebuilds every zonelist in place,
shifting the slots after the new zone. The iterator approves a slot,
then the allocator loads the zone from it, so a GFP_KERNEL request can
land in ZONE_MOVABLE and a nodemask request on a node outside its mask:
CPU 0 CPU 1
prepare_alloc_pages()
first_zones_zonelist(nodemask {1})
slot 0 (NORMAL1) approved
online_pages()
build_all_zonelists()
[NORMAL1, NORMAL2, ...]
-> [MOVABLE1, NORMAL1, ...]
get_page_from_freelist()
zone = zonelist_zone(slot 0) /* MOVABLE1 */
allocates from MOVABLE1
zonelist_update_seq detects the rebuild, but only the slowpath's failure
exits check it.
Check it whenever an allocation succeeds too, and if it changed, free
the page and retry, as the memcg charge failure path does. Eligible
pages are dropped as well: zonelists only change when memory comes or
goes, and a retry is simpler than rechecking the page.
Freeing the page does not undo a block the allocation claimed. A
GFP_KERNEL request that lands in a fresh ZONE_MOVABLE converts a free
block to MIGRATE_UNMOVABLE, which stays until the zone's movable free
list runs out and a movable allocation takes it back.
The fast and bulk paths read the seqlock's raw seqcount without waiting:
an ALLOC_NOLOCK caller can be an NMI that interrupted the writer, and on
PREEMPT_RT the seqlock accessor takes the spinlock. A sample taken
mid-write costs a retry, or NULL under ALLOC_NOLOCK.
check_retry_zonelist() returns 0 when compiled out, so a backport
without the hot-add change stays harmless.
Fixes: 11cd8638c37f ("mm, page_alloc: remove stop_machine from build_all_zonelists")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/page_alloc.c | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 28d46f987685..f2f7e96bdef2 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4419,12 +4419,25 @@ static unsigned int zonelist_iter_begin(void)
return 0;
}
+/*
+ * Never wait for the writer, which an ALLOC_NOLOCK caller in NMI may have
+ * interrupted. Read the raw seqcount: on PREEMPT_RT the seqlock accessor takes
+ * the spinlock while a write is in progress.
+ */
+static unsigned int zonelist_iter_sample(void)
+{
+ if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
+ return raw_seqcount_begin(&zonelist_update_seq.seqcount.seqcount);
+
+ return 0;
+}
+
static unsigned int check_retry_zonelist(unsigned int seq)
{
if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
return read_seqretry(&zonelist_update_seq, seq);
- return seq;
+ return 0;
}
/* Perform direct synchronous page reclaim */
@@ -5100,6 +5113,12 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
warn_alloc(gfp_mask, ac->nodemask,
"page allocation failure: order:%u", order);
got_pg:
+ /* A rebuild during the walk may have handed out an ineligible zone. */
+ if (page && check_retry_zonelist(zonelist_iter_cookie)) {
+ __free_frozen_pages(page, order, FPI_NONE);
+ page = NULL;
+ goto restart;
+ }
return page;
}
@@ -5184,6 +5203,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
struct list_head *pcp_list;
struct alloc_context ac;
unsigned int alloc_flags = ALLOC_WMARK_LOW;
+ unsigned int zonelist_iter_cookie;
int nr_populated = 0, nr_account = 0;
/*
@@ -5224,6 +5244,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
/* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
gfp &= gfp_allowed_mask;
gfp = current_gfp_context(gfp);
+ zonelist_iter_cookie = zonelist_iter_sample();
if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
goto out;
@@ -5273,6 +5294,10 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
if (unlikely(!zone))
goto failed;
+ /* The zone was chosen from a zonelist that a rebuild may have changed. */
+ if (unlikely(check_retry_zonelist(zonelist_iter_cookie)))
+ goto failed;
+
/* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */
pcp = pcp_spin_trylock(zone->per_cpu_pageset);
if (!pcp)
@@ -5413,6 +5438,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
.alloc_flags = alloc_flags,
};
unsigned int fastpath_alloc_flags = alloc_flags;
+ unsigned int zonelist_iter_cookie;
/* Other flags could be supported later if needed. */
if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
@@ -5442,6 +5468,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
*/
gfp = current_gfp_context(gfp);
alloc_gfp = gfp;
+ zonelist_iter_cookie = zonelist_iter_sample();
if (!prepare_alloc_pages(gfp, order, preferred_nid, nodemask, &ac,
&alloc_gfp, &fastpath_alloc_flags))
return NULL;
@@ -5458,6 +5485,11 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
/* First allocation attempt (or, for nolock, only attempt) */
page = get_page_from_freelist(alloc_gfp, order, fastpath_alloc_flags, &ac);
+ if (likely(page) && unlikely(check_retry_zonelist(zonelist_iter_cookie))) {
+ __free_frozen_pages(page, order,
+ alloc_flags & ALLOC_NOLOCK ? FPI_NOLOCK : 0);
+ page = NULL;
+ }
if (likely(page) || (alloc_flags & ALLOC_NOLOCK))
goto out;
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 09/12] mm/hugetlb: retry the pool walk when the zonelist is rebuilt
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (7 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 08/12] mm/page_alloc: retry allocations that raced with a zonelist rebuild Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 10/12] mm/page_alloc: retry alloc_contig_pages() " Gregory Price (Meta)
` (3 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
Onlining the first block of a zone rebuilds every zonelist in place,
shifting the slots after the new zone. dequeue_hugetlb_folio_nodemask()
approves a slot by its node, then loads the zone from it, so a hugetlb
fault bound to one node can take a pool page from another:
CPU 0 CPU 1
dequeue_hugetlb_folio_nodemask(nodemask {2})
slot 3 (NORMAL2) approved
online_pages()
build_all_zonelists()
[.., NORMAL1, NORMAL2]
-> [.., MOVABLE1, NORMAL1, ...]
zone = zonelist_zone(slot 3) /* NORMAL1 */
dequeue_hugetlb_folio_node_exact(h, 1)
Check zonelist_update_seq before using each zone, and on the way out
with nothing found, and restart the walk if it changed. Checking before
the dequeue leaves nothing to put back. Reset the last-tried node on
restart, so it is not skipped.
With kprobes holding the walk across the online, a fault bound to node 2
took a node 1 page in 3 of 3 runs, and a node 2 page in 3 of 3 with this
change.
Fixes: 11cd8638c37f ("mm, page_alloc: remove stop_machine from build_all_zonelists")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
mm/hugetlb.c | 13 ++++++++++---
mm/page_alloc.c | 18 +-----------------
mm/page_alloc.h | 19 +++++++++++++++++++
3 files changed, 30 insertions(+), 20 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index da5d04edf7c1..b8a4fc2a996c 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1293,11 +1293,11 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
static struct folio *dequeue_hugetlb_folio_nodemask(struct hstate *h, gfp_t gfp_mask,
int nid, nodemask_t *nmask)
{
- unsigned int cpuset_mems_cookie;
+ unsigned int cpuset_mems_cookie, zonelist_cookie;
struct zonelist *zonelist;
struct zone *zone;
struct zoneref *z;
- int node = NUMA_NO_NODE;
+ int node;
/* 'nid' should not be NUMA_NO_NODE. Try to catch any misuse of it and rectifiy. */
if (nid == NUMA_NO_NODE)
@@ -1306,10 +1306,16 @@ static struct folio *dequeue_hugetlb_folio_nodemask(struct hstate *h, gfp_t gfp_
zonelist = node_zonelist(nid, gfp_mask);
retry_cpuset:
+ node = NUMA_NO_NODE;
cpuset_mems_cookie = read_mems_allowed_begin();
+ zonelist_cookie = zonelist_iter_begin();
for_each_zone_zonelist_nodemask(zone, z, zonelist, gfp_zone(gfp_mask), nmask) {
struct folio *folio;
+ /* A rebuild may have shifted a node outside @nmask into this slot. */
+ if (unlikely(check_retry_zonelist(zonelist_cookie)))
+ goto retry_cpuset;
+
if (!cpuset_zone_allowed(zone, gfp_mask))
continue;
/*
@@ -1324,7 +1330,8 @@ static struct folio *dequeue_hugetlb_folio_nodemask(struct hstate *h, gfp_t gfp_
if (folio)
return folio;
}
- if (unlikely(read_mems_allowed_retry(cpuset_mems_cookie)))
+ if (unlikely(read_mems_allowed_retry(cpuset_mems_cookie) ||
+ check_retry_zonelist(zonelist_cookie)))
goto retry_cpuset;
return NULL;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f2f7e96bdef2..242bef2fe400 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4409,15 +4409,7 @@ EXPORT_SYMBOL_GPL(fs_reclaim_release);
* retries the allocation if zonelist changes. Writer side is protected by the
* embedded spin_lock.
*/
-static DEFINE_SEQLOCK(zonelist_update_seq);
-
-static unsigned int zonelist_iter_begin(void)
-{
- if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
- return read_seqbegin(&zonelist_update_seq);
-
- return 0;
-}
+DEFINE_SEQLOCK(zonelist_update_seq);
/*
* Never wait for the writer, which an ALLOC_NOLOCK caller in NMI may have
@@ -4432,14 +4424,6 @@ static unsigned int zonelist_iter_sample(void)
return 0;
}
-static unsigned int check_retry_zonelist(unsigned int seq)
-{
- if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
- return read_seqretry(&zonelist_update_seq, seq);
-
- return 0;
-}
-
/* Perform direct synchronous page reclaim */
static unsigned long
__perform_reclaim(gfp_t gfp_mask, unsigned int order,
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index e238ee91ed24..10a40ead6c00 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -9,6 +9,7 @@
#include <linux/mm.h>
#include <linux/mmzone.h>
#include <linux/nodemask.h>
+#include <linux/seqlock.h>
#include <linux/types.h>
#define ALLOC_DEFAULT 0
@@ -265,6 +266,24 @@ extern void zone_pcp_disable(struct zone *zone);
extern void zone_pcp_enable(struct zone *zone);
extern void zone_pcp_init(struct zone *zone);
+extern seqlock_t zonelist_update_seq;
+
+static inline unsigned int zonelist_iter_begin(void)
+{
+ if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
+ return read_seqbegin(&zonelist_update_seq);
+
+ return 0;
+}
+
+static inline unsigned int check_retry_zonelist(unsigned int seq)
+{
+ if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG))
+ return read_seqretry(&zonelist_update_seq, seq);
+
+ return 0;
+}
+
enum fallback_result {
/* Found suitable migratetype, *mt_out is valid. */
FALLBACK_FOUND,
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 10/12] mm/page_alloc: retry alloc_contig_pages() when the zonelist is rebuilt
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (8 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 09/12] mm/hugetlb: retry the pool walk when the zonelist is rebuilt Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 11/12] mm/mempolicy: read the zoneref slot once in mempolicy_slab_node() Gregory Price (Meta)
` (2 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
Onlining the first block of a zone rebuilds every zonelist in place,
shifting the slots after the new zone.
alloc_contig_frozen_pages_noprof() approves a slot by the caller's gfp
and nodemask, then loads the zone from it:
CPU 0 CPU 1
alloc_contig_frozen_pages_noprof(GFP_HIGHUSER | __GFP_THISNODE)
slot 0 (NORMAL1) approved
online_pages()
build_all_zonelists()
[NORMAL1]
-> [MOVABLE1, NORMAL1]
zone = zonelist_zone(slot 0) /* MOVABLE1 */
alloc_contig_frozen_range(MOVABLE1 pfns)
A gigantic hugetlb page can then land in ZONE_MOVABLE, and a nodemask
caller can get a range from another node. Gigantic pages are kept out
of ZONE_MOVABLE since commit 79b63f12abcb ("mm, hugetlb: do not allocate
non-migrateable gigantic pages from movable zones").
Check zonelist_update_seq before using each zone, and restart the walk
if it changed, as the hugetlb pool walk does. Check again before
escalating to the pass that may migrate hugetlb pages, so a pass a
rebuild cut short is redone rather than escalated.
With kprobes holding the walk across the online, the base kernel put the
1G page in ZONE_MOVABLE in 3 of 3 runs, and this change put it in
ZONE_NORMAL in 3 of 3. Direct GFP_KERNEL calls with nodemask {1} and
{2} got a ZONE_MOVABLE range and a node 1 range on base, 3 of 3 each,
and neither with this change.
Fixes: 5e27a2df03b8 ("mm/page_alloc: add alloc_contig_pages()")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/page_alloc.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 242bef2fe400..91ad979409e5 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7570,6 +7570,7 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,
gfp_t gfp_mask, int nid, nodemask_t *nodemask)
{
unsigned long ret, pfn, flags;
+ unsigned int zonelist_cookie;
struct zonelist *zonelist;
struct zone *zone;
struct zoneref *z;
@@ -7578,8 +7579,13 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,
retry:
zonelist = node_zonelist(nid, gfp_mask);
+ zonelist_cookie = zonelist_iter_begin();
for_each_zone_zonelist_nodemask(zone, z, zonelist,
gfp_zone(gfp_mask), nodemask) {
+ /* A rebuild may have shifted an ineligible zone into this slot. */
+ if (unlikely(check_retry_zonelist(zonelist_cookie)))
+ goto retry;
+
spin_lock_irqsave(&zone->lock, flags);
pfn = ALIGN(zone->zone_start_pfn, nr_pages);
@@ -7614,6 +7620,9 @@ struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,
* second pass, we will try moving HugeTLB pages when those pages are
* smaller than the requested contiguous region size.
*/
+ /* Redo a pass that a rebuild cut short before escalating. */
+ if (unlikely(check_retry_zonelist(zonelist_cookie)))
+ goto retry;
if (skip_hugetlb && skipped_hugetlb) {
skip_hugetlb = false;
goto retry;
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 11/12] mm/mempolicy: read the zoneref slot once in mempolicy_slab_node()
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (9 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 10/12] mm/page_alloc: retry alloc_contig_pages() " Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 12/12] mm/mmzone: remove zonelist_node_idx() Gregory Price (Meta)
2026-11-01 11:00 ` [PATCH v5 0/12] mm: fix allocator races with memory hotplug Vlastimil Babka (SUSE)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual, stable
For a bind or preferred-many policy, mempolicy_slab_node() tests the
first matching slot's zone for NULL, then reads the slot again to look
up the zone's node. Offlining a zone's last block rebuilds the
zonelists in place, so the second read can return the terminator:
CPU 0 CPU 1
mempolicy_slab_node()
zonelist_zone(slot 4)
= node 2 NORMAL
offline_pages()
build_all_zonelists()
slot 4 = NULL
zonelist_node_idx(slot 4)
zone_to_nid(NULL)
Current GCC merges the two reads, but nothing requires it. With
mm/mempolicy.o built with -fno-inline -fno-ipa-sra, holding a bound
task's slab_strict_numa allocation between the reads while a movable
block earlier in the zonelist goes offline oopses every time:
BUG: kernel NULL pointer dereference, address: 0000000000000060
RIP: 0010:zone_to_nid+0x0/0x10
Call Trace:
kmem_cache_alloc_noprof+0x294/0x610
do_getname+0x34/0x230
mempolicy_slab_node() and zonelist_node_idx() are tail calls here, so
neither appears in the trace.
Read the slot once and use that zone.
Fixes: 11cd8638c37f ("mm, page_alloc: remove stop_machine from build_all_zonelists")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
---
mm/mempolicy.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 0884c629670d..cd63c3e2c029 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2176,6 +2176,7 @@ unsigned int mempolicy_slab_node(void)
case MPOL_PREFERRED_MANY:
{
struct zoneref *z;
+ struct zone *zone;
/*
* Follow bind policy behavior and start allocation at the
@@ -2186,7 +2187,8 @@ unsigned int mempolicy_slab_node(void)
zonelist = &NODE_DATA(node)->node_zonelists[ZONELIST_FALLBACK];
z = first_zones_zonelist(zonelist, highest_zoneidx,
&policy->nodes);
- return zonelist_zone(z) ? zonelist_node_idx(z) : node;
+ zone = zonelist_zone(z);
+ return zone ? zone_to_nid(zone) : node;
}
case MPOL_LOCAL:
return node;
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v5 12/12] mm/mmzone: remove zonelist_node_idx()
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (10 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 11/12] mm/mempolicy: read the zoneref slot once in mempolicy_slab_node() Gregory Price (Meta)
@ 2026-11-01 2:13 ` Gregory Price (Meta)
2026-11-01 11:00 ` [PATCH v5 0/12] mm: fix allocator races with memory hotplug Vlastimil Babka (SUSE)
12 siblings, 0 replies; 14+ messages in thread
From: Gregory Price (Meta) @ 2026-11-01 2:13 UTC (permalink / raw)
To: linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
baoquan.he, baolin.wang, rientjes, rostedt, mhiramat,
mathieu.desnoyers, muchun.song, osalvador, gourry, ziy,
joshua.hahnjy, rakie.kim, ying.huang, brendan.jackman, hannes,
bigeasy, clrkwllms, nathan, ndesaulniers, morbo, justinstitt,
Bessel.Wang, jiang.liu, mgorman, anshuman.khandual
zonelist_node_idx() looks like it returns a field of the zoneref, as
zonelist_zone_idx() does, but it loads the zone from the slot again.
Memory hotplug rewrites zonelists in place, so a caller that has already
checked the slot can get a different zone, or the NULL terminator, from
that second load.
__next_zones_zonelist() and mempolicy_slab_node() had this bug.
Remove the helper and open-code it in its two remaining callers, neither
of which can race with a rebuild. local_memory_node() runs either inside
__build_all_zonelists() or during CPU bring-up, which holds
cpu_hotplug_lock for write and so excludes mem_hotplug_begin().
The lookup in mpol_misplaced() is unreachable: it needs a policy with
MPOL_F_MOF but not MPOL_F_MORON, and nothing sets one without the other.
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
include/linux/mmzone.h | 6 ------
mm/mempolicy.c | 2 +-
mm/page_alloc.c | 2 +-
3 files changed, 2 insertions(+), 8 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 4bc2318ac37f..72ffb2959c4f 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1439,7 +1439,6 @@ struct zoneref {
*
* zonelist_zone() - Return the struct zone * for an entry in _zonerefs
* zonelist_zone_idx() - Return the index of the zone for an entry
- * zonelist_node_idx() - Return the index of the node for an entry
*/
struct zonelist {
struct zoneref _zonerefs[MAX_ZONES_PER_ZONELIST + 1];
@@ -1864,11 +1863,6 @@ static inline int zonelist_zone_idx(const struct zoneref *zoneref)
return zoneref->zone_idx;
}
-static inline int zonelist_node_idx(const struct zoneref *zoneref)
-{
- return zone_to_nid(zoneref->zone);
-}
-
struct zoneref *__next_zones_zonelist(struct zoneref *z,
enum zone_type highest_zoneidx,
const nodemask_t *nodes);
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index cd63c3e2c029..267576054980 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -3087,7 +3087,7 @@ int mpol_misplaced(struct folio *folio, struct vm_fault *vmf,
node_zonelist(thisnid, GFP_HIGHUSER),
gfp_zone(GFP_HIGHUSER),
&pol->nodes);
- polnid = zonelist_node_idx(z);
+ polnid = zone_to_nid(zonelist_zone(z));
break;
default:
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 91ad979409e5..81c1de714658 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5973,7 +5973,7 @@ int local_memory_node(int node)
z = first_zones_zonelist(node_zonelist(node, GFP_KERNEL),
gfp_zone(GFP_KERNEL),
NULL);
- return zonelist_node_idx(z);
+ return zone_to_nid(zonelist_zone(z));
}
#endif
--
2.56.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v5 0/12] mm: fix allocator races with memory hotplug
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
` (11 preceding siblings ...)
2026-11-01 2:13 ` [PATCH v5 12/12] mm/mmzone: remove zonelist_node_idx() Gregory Price (Meta)
@ 2026-11-01 11:00 ` Vlastimil Babka (SUSE)
12 siblings, 0 replies; 14+ messages in thread
From: Vlastimil Babka (SUSE) @ 2026-11-01 11:00 UTC (permalink / raw)
To: Gregory Price (Meta), linux-mm
Cc: linux-trace-kernel, linux-kernel, kernel-team, akpm, david, ljs,
liam, rppt, surenb, mhocko, kasong, qi.zheng, shakeel.butt,
baohua, axelrasmussen, yuanchu, weixugc, baoquan.he, baolin.wang,
rientjes, rostedt, mhiramat, mathieu.desnoyers, muchun.song,
osalvador, ziy, joshua.hahnjy, rakie.kim, ying.huang,
brendan.jackman, hannes, bigeasy, clrkwllms, nathan,
ndesaulniers, morbo, justinstitt, Bessel.Wang, jiang.liu,
mgorman, anshuman.khandual
On 11/1/26 03:13, Gregory Price (Meta) wrote:
> Memory hotplug rebuilds zonelists in place while allocations walk them,
> and offlining an emptied zone frees pagesets the vmstat worker can still
> hold.
>
> These eleven fixes cover the walkers that crash, fail, or pick the wrong
> zone or node as a result, and a twelfth patch removes a helper that
> invites the same bug.
>
> 1. a restarted slowpath loses its retry with reserves
> 2. offlining an emptied zone frees pagesets still in use
> 3. the reclaim_retry_zone tracepoint rereads a zoneref slot
> 4. the zonelist retry is compiled out without MEMORY_HOTREMOVE
> 5. zonelist_zone() may load its slot more than once
> 6. NULL preferred zone after a shrink rebuild
> 7. __next_zones_zonelist() rereads a slot
> 8. allocations land outside their zone or nodemask
> 9. the hugetlb pool walk takes another node's page
> 10. alloc_contig_pages() takes a range outside its filters
> 11. mempolicy_slab_node() rereads a slot
> 12. zonelist_node_idx() invites the same double read
>
> Patches 1-11 are for stable. Patch 12 is a cleanup and is not.
>
> Patches 8-10 need patch 4, without which their retry is compiled
> out on HOTREMOVE=n kernels.
>
> Patch 8 needs patch 1.
>
> Patches 6, 7 and 11 need patch 5, which makes zonelist_zone() read
> the slot once. Patch 12 needs patches 7 and 11, which remove the
> other callers of zonelist_node_idx().
>
> Patches 2 and 3 are patches from anthropic, pulled in from Vlastimil's
> series to keep the backport issues self-contained on the series.
>
> Testing
> =======
>
> Each race was reproduced on unmodified source in a QEMU guest with three
> NUMA nodes and a 512M pc-dimm on node 1, left offline at boot.
>
> A test module widens each window with a kprobe that stalls one CPU inside
> it, as an interrupt or a vCPU preemption would.
>
> base this series
> patch outcome runs outcome runs
> ===== ======================= ===== ======================= =====
> 1 NULL returned 3/3 page from reserves 3/3
> 2 UAF 6/6 no UAF 3/3
> 3 NULL dereference 3/3 allocation succeeds 3/3
> 4 allocation fails 3/3 waits for the rebuild 3/3
> 6 NULL dereference 3/3 ZONE_NORMAL page 3/3
> 7 NULL dereference 3/3 ZONE_NORMAL page 3/3
> 8 wrong zone or node 18/18 right zone and node 18/18
> 9 page from node 1 3/3 page from node 2 3/3
> 10 1G page in ZONE_MOVABLE 3/3 1G page in ZONE_NORMAL 3/3
> 11 NULL dereference 4/4 keeps allocating 3/3
>
>
> Cost (these tests were run on v4, v5 hasn't substantially changed)
> ====
> Kernel instructions per order-0 allocation and free in the QEMU guest,
> counted with a task-bound hardware instruction counter around 1M
> iterations on a pinned kthread, non-debug kernel, mean of 15 rounds.
>
> instructions per alloc+free
> base this series
> alloc_pages() 890.5 900.5
> bulk, nr=1, nodemask 865.5 875.5
> bulk, nr=16, per page 477.2 478.1
>
>
> Bare metal, two-socket Xeon Gold 6138 (Skylake-SP, 80 CPUs).
>
> will-it-scale page_fault1-3: each iteration touches one page, so
> the unit is millions of faults per second, summed over all tasks.
>
> Base, millions of faults per second:
>
> tasks 1 10 40 80
> page_fault1, processes 0.46 4.16 10.51 9.23
> page_fault1, threads 0.46 3.51 4.35 4.19
> page_fault2, processes 0.43 3.66 9.02 8.33
> page_fault2, threads 0.43 3.48 4.37 4.19
> page_fault3, processes 3.89 14.78 8.10 8.00
> page_fault3, threads 3.87 14.29 6.88 7.96
>
> Change in faults per second with this series, + is faster:
>
> tasks 1 10 40 80
> page_fault1, processes -0.3% +0.1% -3.0% -1.7%
> page_fault1, threads -0.5% -1.6% -0.9% -0.9%
> page_fault2, processes +0.0% +0.9% -1.4% -0.5%
> page_fault2, threads +0.3% +1.5% -1.4% -1.0%
> page_fault3, processes -1.0% -3.8% -0.7% +1.2%
> page_fault3, threads -0.9% -0.7% +0.8% -0.2%
>
> The boot-to-boot stddev is 1.0-1.8% at 1 task and up to
> 15% in 10-80 tasks, so basically this is all within noise.
>
> Changes in v5
> =============
> - Pulled in patches from Vlastimil's series (patches 2 and 3 here)
> - inline functions (Vlastimil)
> - Added patch 5: zonelist_zone() reads the slot with READ_ONCE(), as
> recommended by sashiko on v4's mempolicy patch.
> Patches 6, 7 and 11 call it instead of open-coding the load.
> - Added patch 12, removing zonelist_node_idx(). Not for stable.
> - Patch 1: reworded the changelog (Vlastimil).
> - Collected Reviewed-by tags.
>
> Link: https://lore.kernel.org/linux-mm/20261028211930.3504634-1-gourry@gourry.net/
> Link: https://lore.kernel.org/linux-mm/20261027-pgalloc-ai-hotfixes-v1-0-9c7d1aca801e@kernel.org/
> Link: https://lore.kernel.org/linux-mm/20261014222456.2318284-1-gourry@gourry.net/
> Link: https://lore.kernel.org/linux-mm/20261015210539.3243563-1-gourry@gourry.net/
> Link: https://lore.kernel.org/linux-mm/20261019172346.3344005-1-gourry@gourry.net/
>
> Gregory Price (Meta) (10):
> mm/page_alloc: reset can_retry_reserves when the slowpath restarts
> mm/page_alloc: detect zonelist rebuilds from memory hot-add
> mm/mmzone: read the zoneref slot once in zonelist_zone()
> mm/page_alloc: snapshot the preferred zone against shrink rebuilds
> mm/mmzone: read each zoneref slot once when filtering by nodemask
> mm/page_alloc: retry allocations that raced with a zonelist rebuild
> mm/hugetlb: retry the pool walk when the zonelist is rebuilt
> mm/page_alloc: retry alloc_contig_pages() when the zonelist is rebuilt
> mm/mempolicy: read the zoneref slot once in mempolicy_slab_node()
> mm/mmzone: remove zonelist_node_idx()
>
> Josef Bacik (2):
> mm/page_alloc: keep the pagesets of an empty zone
> mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone
Thanks, applied to mm/page_alloc [1] page_alloc/for-next-fixes
It should result in no conflicts with other trees AFAICS.
Following David's MAINTAINERS updates [2] and mm-next call discussions,
we'll move ahead with the page_alloc.git tree (integrated to mm-next) now to
spread the handling of the endless fixes stream. I'll send the MAINTAINERS
update soon.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mm/page_alloc.git
[2]
https://lore.kernel.org/all/20261030-maintainers-v1-0-97278f8a5f54@kernel.org/
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2026-11-01 11:00 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-01 2:13 [PATCH v5 0/12] mm: fix allocator races with memory hotplug Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 01/12] mm/page_alloc: reset can_retry_reserves when the slowpath restarts Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 02/12] mm/page_alloc: keep the pagesets of an empty zone Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 03/12] mm, page_alloc: pass zone pointer to trace_reclaim_retry_zone Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 04/12] mm/page_alloc: detect zonelist rebuilds from memory hot-add Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 05/12] mm/mmzone: read the zoneref slot once in zonelist_zone() Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 06/12] mm/page_alloc: snapshot the preferred zone against shrink rebuilds Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 07/12] mm/mmzone: read each zoneref slot once when filtering by nodemask Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 08/12] mm/page_alloc: retry allocations that raced with a zonelist rebuild Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 09/12] mm/hugetlb: retry the pool walk when the zonelist is rebuilt Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 10/12] mm/page_alloc: retry alloc_contig_pages() " Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 11/12] mm/mempolicy: read the zoneref slot once in mempolicy_slab_node() Gregory Price (Meta)
2026-11-01 2:13 ` [PATCH v5 12/12] mm/mmzone: remove zonelist_node_idx() Gregory Price (Meta)
2026-11-01 11:00 ` [PATCH v5 0/12] mm: fix allocator races with memory hotplug Vlastimil Babka (SUSE)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®