[PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[Gregory Price]

A per-node vmstat counter is pgdat->vm_stat[] plus per-cpu deltas.
A balanced counter can sit split as global=+N / per-cpu=-N.

The folds reconciling the split only walk online nodes, so when
try_offline_node() marks a node offline - per-cpu deltas are stranded.

A subsequent online zeroes the per-cpu area but not pgdat->vm_stat[],
orphaning the +N permanently. All NR_VM_NODE_STAT_ITEMS are affected.

Flush the deltas before the node leaves the online set.  A remote
fold races the periodic per-cpu fold, so do it as per-cpu work.

Discovered when a node/compact call hung for a nearly empty node, as
the math to determine throttling broke. Reproduced by repeated memory
hotplug/unplug cycles on a node under pressure. NR_ISOLATED_ANON
ratchets up and never returns to zero.

Fixes: 75ef71840539 ("mm, vmstat: add infrastructure for per-node vmstats")
Cc: stable@vger.kernel.org
Signed-off-by: Gregory Price <gourry@gourry.net>
---
 include/linux/vmstat.h |  2 ++
 mm/memory_hotplug.c    |  5 ++++-
 mm/vmstat.c            | 10 ++++++++++
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 3c9c266cf782..ea1017427811 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -293,6 +293,7 @@ extern void __dec_node_state(struct pglist_data *, enum node_stat_item);
 
 void quiet_vmstat(void);
 void cpu_vm_stats_fold(int cpu);
+void sync_vm_stats(void);
 void refresh_zone_stat_thresholds(void);
 
 void drain_zonestat(struct zone *zone, struct per_cpu_zonestat *);
@@ -397,6 +398,7 @@ static inline void __dec_node_page_state(struct page *page,
 
 static inline void refresh_zone_stat_thresholds(void) { }
 static inline void cpu_vm_stats_fold(int cpu) { }
+static inline void sync_vm_stats(void) { }
 static inline void quiet_vmstat(void) { }
 static inline void vmstat_flush_workqueue(void) { }
 
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 7d60a7dd1e7b..10f676566f56 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -2338,8 +2338,11 @@ void try_offline_node(int nid)
 
 	/*
 	 * all memory/cpu of this node are removed, we can offline this
-	 * node now.
+	 * node now.  Fold any pending per-cpu vmstat diffs into the global
+	 * counters first: once the node leaves the online set the periodic
+	 * fold skips it, orphaning the residual on a later online.
 	 */
+	sync_vm_stats();
 	node_set_offline(nid);
 	unregister_node(nid);
 }
diff --git a/mm/vmstat.c b/mm/vmstat.c
index f534972f517d..ad77343212d3 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -941,6 +941,16 @@ void cpu_vm_stats_fold(int cpu)
 	fold_diff(global_zone_diff, global_node_diff);
 }
 
+static void vmstat_fold_work(struct work_struct *w)
+{
+	refresh_cpu_vm_stats(false);
+}
+
+void sync_vm_stats(void)
+{
+	schedule_on_each_cpu(vmstat_fold_work);
+}
+
 /*
  * this is only called if !populated_zone(zone), which implies no other users of
  * pset->vm_stat_diff[] exist.
-- 
2.53.0-Meta

Re: [PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[Gregory Price]

On Sat, Jun 27, 2026 at 03:31:07AM -0400, Gregory Price wrote:
>  
>  	/*
>  	 * all memory/cpu of this node are removed, we can offline this
> -	 * node now.
> +	 * node now.  Fold any pending per-cpu vmstat diffs into the global
> +	 * counters first: once the node leaves the online set the periodic
> +	 * fold skips it, orphaning the residual on a later online.
>  	 */
> +	sync_vm_stats();

Sashiko points out this can deadlock on concurrent cpu hotplug, which
after a quick look seems accurate.

Has also made me realize the sync code is also racy with cpu hotplug
as well as it iterates per-online-cpu.

There are probably more races like this out there.

Will need to give this a little more thought.

>  	node_set_offline(nid);
>  	unregister_node(nid);
>  }

Re: [PATCH] mm/vmstat: flush per-cpu node stats when a node goes offline

[Gregory Price]

On Sat, Jun 27, 2026 at 03:31:07AM -0400, Gregory Price wrote:
> A per-node vmstat counter is pgdat->vm_stat[] plus per-cpu deltas.
> A balanced counter can sit split as global=+N / per-cpu=-N.
> 
> The folds reconciling the split only walk online nodes, so when
> try_offline_node() marks a node offline - per-cpu deltas are stranded.
> 
> A subsequent online zeroes the per-cpu area but not pgdat->vm_stat[],
> orphaning the +N permanently. All NR_VM_NODE_STAT_ITEMS are affected.
> 
> Flush the deltas before the node leaves the online set.  A remote
> fold races the periodic per-cpu fold, so do it as per-cpu work.
> 
> Discovered when a node/compact call hung for a nearly empty node, as
> the math to determine throttling broke. Reproduced by repeated memory
> hotplug/unplug cycles on a node under pressure. NR_ISOLATED_ANON
> ratchets up and never returns to zero.
> 
> Fixes: 75ef71840539 ("mm, vmstat: add infrastructure for per-node vmstats")
> Cc: stable@vger.kernel.org
> Signed-off-by: Gregory Price <gourry@gourry.net>

Realized I changed the title on v2:
https://lore.kernel.org/linux-mm/20260627073107.523499-1-gourry@gourry.net/

Core issue was the zeroing at online time causing the skew, just have to
do the fold there instead.  disregard this version please