[PATCH v2 0/3] mm/hugetlb: Fix CMA parsing, flags and usage accounting

[PATCH v2 0/3] mm/hugetlb: Fix CMA parsing, flags and usage accounting

[Kaitao Cheng]

This series fixes three independent HugeTLB issues reported by Sashiko,
covering CMA parameter parsing, persistent folio flags, and NR_HUGETLB
accounting on allocation rollback.

Patch 1 rejects a global hugetlb_cma size following node-specific entries.
This prevents malformed input from overwriting valid per-node reservations
with an incorrectly parsed global size.

Patch 2 preserves persistent folio flags when a failed vmemmap restore
returns a folio to the pool. This retains CMA allocation tracking and
raw hardware poison information, while ensuring transient flags are
cleared before returning the folio.

Patch 3 moves NR_HUGETLB node accounting to the underlying folio state
transitions and separates it from memcg statistics. This keeps the
counter balanced when surplus folios are freed during reservation
gathering or after a failed overcommit recheck.


Changes in v2:
- Patch 2: Preserve persistent flags, clear transient flags, and warn
  about unexpected flags.
- Patch 3: Account at folio state transitions and separate node and
  memcg statistics.

Link to v1:
https://lore.kernel.org/all/20261028113319.38544-1-kaitao.cheng@linux.dev/

Kaitao Cheng (3):
  mm/hugetlb_cma: fix incorrect hugetlb_cma_size assignment on malformed
    input
  mm/hugetlb: Preserve persistent flags when returning folios to the
    pool
  mm/hugetlb: Fix NR_HUGETLB accounting for surplus folios

 mm/hugetlb.c     | 52 +++++++++++++++++++++++++++++++-----------------
 mm/hugetlb_cma.c |  3 +++
 mm/memcontrol.c  |  6 +-----
 3 files changed, 38 insertions(+), 23 deletions(-)

-- 
2.54.0 (Apple Git-157)

[PATCH v2 1/3] mm/hugetlb_cma: fix incorrect hugetlb_cma_size assignment on malformed input

[Kaitao Cheng]

From: Kaitao Cheng <chengkaitao@kylinos.cn>

When hugetlb_cma contains a global size after node-specific entries,
cmdline_parse_hugetlb_cma() enters the global branch and reparses the
original parameter string. For hugetlb_cma=1:2G,4G, this interprets the
initial node ID as a one-byte global size and clears the 2 GiB reservation
recorded for node 1.

Global and node-specific formats cannot be mixed. Return -EINVAL when a
global entry follows a node-specific entry, before updating any settings.
This preserves the valid prefix, so the example retains node 1's 2 GiB
reservation and reports the malformed option.

Fixes: 38e719ab2673 ("hugetlb: support node specified when using cma for gigantic hugepages")
Closes: https://sashiko.dev/#/bug/linux-10f40d3e-3feb-4622-9da8-1fedecb2d0e2
Acked-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
---
 mm/hugetlb_cma.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index 95fd2d190f0d..8e6b20341fc4 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -167,6 +167,9 @@ static int __init cmdline_parse_hugetlb_cma(char *p)
 		} else {
 			char *next;
 
+			if (s != p)
+				return -EINVAL;
+
 			tmp = memparse(p, &next);
 			if (*next == '%') {
 				if (tmp > 100) {
-- 
2.54.0 (Apple Git-157)

[PATCH v2 2/3] mm/hugetlb: Preserve persistent flags when returning folios to the pool

[Kaitao Cheng]

From: Kaitao Cheng <chengkaitao@kylinos.cn>

add_hugetlb_folio() returns a folio to the HugeTLB pool when vmemmap
restoration fails. It clears folio->private and restores only
HPG_vmemmap_optimized, losing other flags that must survive the rollback.

Clearing HPG_cma causes a later free to bypass the CMA release path,
leaving the CMA allocation bitmap set. Clearing HPG_raw_hwp_unreliable
loses the indication that the locations of poisoned subpages are unknown,
undermining the checks that prevent freeing or unpoisoning such folios.

Preserve the existing flags instead of resetting folio->private. Callers
already clear HPG_restore_reserve, HPG_migratable and HPG_freed before
returning a folio to the pool. Clear HPG_temporary in free_huge_folio()
before attempting to free a temporary folio, so a failed vmemmap restore
does not leave that flag set on a folio returned as surplus.

Add a VM_WARN_ON_ONCE_FOLIO() check for flags other than
HPG_vmemmap_optimized, HPG_raw_hwp_unreliable and HPG_cma to detect
incomplete transient flag cleanup.

Fixes: 161df60e9e89 ("mm, hwpoison, hugetlb: support saving mechanism of raw error pages")
Fixes: d2d786714080 ("mm/hugetlb: enable bootmem allocation from CMA areas")
Closes: https://sashiko.dev/#/bug/linux-5363eb09-9c3c-4f8b-9f3c-a998a8c4b258
Suggested-by: David Hildenbrand <david@kernel.org>
Suggested-by: Muchun Song <muchun.song@linux.dev>
Suggested-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
---
 mm/hugetlb.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4a008afaedf9..931ca3a30773 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1443,6 +1443,12 @@ void add_hugetlb_folio(struct hstate *h, struct folio *folio,
 
 	lockdep_assert_held(&hugetlb_lock);
 
+	/* Callers must clear transient flags before returning to the pool. */
+	VM_WARN_ON_ONCE_FOLIO((unsigned long)folio->private &
+			     ~(BIT(HPG_vmemmap_optimized) |
+			       BIT(HPG_raw_hwp_unreliable) |
+			       BIT(HPG_cma)), folio);
+
 	INIT_LIST_HEAD(&folio->lru);
 	h->nr_huge_pages++;
 	h->nr_huge_pages_node[nid]++;
@@ -1453,9 +1459,6 @@ void add_hugetlb_folio(struct hstate *h, struct folio *folio,
 	}
 
 	__folio_set_hugetlb(folio);
-	/* Clear all folio->private flags except hugetlb_vmemmap_optimized. */
-	folio->private = NULL;
-	folio_set_hugetlb_vmemmap_optimized(folio);
 
 	arch_clear_hugetlb_flags(folio);
 	enqueue_hugetlb_folio(h, folio);
@@ -1743,6 +1746,7 @@ void free_huge_folio(struct folio *folio)
 		h->resv_huge_pages++;
 
 	if (folio_test_hugetlb_temporary(folio)) {
+		folio_clear_hugetlb_temporary(folio);
 		remove_hugetlb_folio(h, folio, false);
 		spin_unlock_irqrestore(&hugetlb_lock, flags);
 		update_and_free_hugetlb_folio(h, folio, true);
-- 
2.54.0 (Apple Git-157)

Re: [PATCH v2 2/3] mm/hugetlb: Preserve persistent flags when returning folios to the pool

[Muchun Song]

> On Nov 2, 2026, at 10:49, Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
> 
> From: Kaitao Cheng <chengkaitao@kylinos.cn>
> 
> add_hugetlb_folio() returns a folio to the HugeTLB pool when vmemmap
> restoration fails. It clears folio->private and restores only
> HPG_vmemmap_optimized, losing other flags that must survive the rollback.
> 
> Clearing HPG_cma causes a later free to bypass the CMA release path,
> leaving the CMA allocation bitmap set. Clearing HPG_raw_hwp_unreliable
> loses the indication that the locations of poisoned subpages are unknown,
> undermining the checks that prevent freeing or unpoisoning such folios.
> 
> Preserve the existing flags instead of resetting folio->private. Callers
> already clear HPG_restore_reserve, HPG_migratable and HPG_freed before
> returning a folio to the pool. Clear HPG_temporary in free_huge_folio()
> before attempting to free a temporary folio, so a failed vmemmap restore
> does not leave that flag set on a folio returned as surplus.
> 
> Add a VM_WARN_ON_ONCE_FOLIO() check for flags other than
> HPG_vmemmap_optimized, HPG_raw_hwp_unreliable and HPG_cma to detect
> incomplete transient flag cleanup.
> 
> Fixes: 161df60e9e89 ("mm, hwpoison, hugetlb: support saving mechanism of raw error pages")
> Fixes: d2d786714080 ("mm/hugetlb: enable bootmem allocation from CMA areas")
> Closes: https://sashiko.dev/#/bug/linux-5363eb09-9c3c-4f8b-9f3c-a998a8c4b258
> Suggested-by: David Hildenbrand <david@kernel.org>
> Suggested-by: Muchun Song <muchun.song@linux.dev>
> Suggested-by: Zi Yan <ziy@nvidia.com>
> Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>

Acked-by: Muchun Song <muchun.song@linux.dev>

But,

> ---
> mm/hugetlb.c | 10 +++++++---
> 1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 4a008afaedf9..931ca3a30773 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -1443,6 +1443,12 @@ void add_hugetlb_folio(struct hstate *h, struct folio *folio,
> 
> 	lockdep_assert_held(&hugetlb_lock);
> 
> +	/* Callers must clear transient flags before returning to the pool. */
> +	VM_WARN_ON_ONCE_FOLIO((unsigned long)folio->private &
> +			     ~(BIT(HPG_vmemmap_optimized) |

It looks like the parameters aren't aligned. A space missing before "~"?

Thanks.

> +			       BIT(HPG_raw_hwp_unreliable) |
> +			       BIT(HPG_cma)), folio);
> +
> 	INIT_LIST_HEAD(&folio->lru);
> 	h->nr_huge_pages++;
> 	h->nr_huge_pages_node[nid]++;
> @@ -1453,9 +1459,6 @@ void add_hugetlb_folio(struct hstate *h, struct folio *folio,
> 	}
> 
> 	__folio_set_hugetlb(folio);
> -	/* Clear all folio->private flags except hugetlb_vmemmap_optimized. */
> -	folio->private = NULL;
> -	folio_set_hugetlb_vmemmap_optimized(folio);
> 
> 	arch_clear_hugetlb_flags(folio);
> 	enqueue_hugetlb_folio(h, folio);
> @@ -1743,6 +1746,7 @@ void free_huge_folio(struct folio *folio)
> 		h->resv_huge_pages++;
> 
> 	if (folio_test_hugetlb_temporary(folio)) {
> +		folio_clear_hugetlb_temporary(folio);
> 		remove_hugetlb_folio(h, folio, false);
> 		spin_unlock_irqrestore(&hugetlb_lock, flags);
> 		update_and_free_hugetlb_folio(h, folio, true);
> -- 
> 2.54.0 (Apple Git-157)
> 

Re: [PATCH v2 2/3] mm/hugetlb: Preserve persistent flags when returning folios to the pool

[Zi Yan]

On Sun Nov 1, 2026 at 9:49 PM EST, Kaitao Cheng wrote:
> From: Kaitao Cheng <chengkaitao@kylinos.cn>
>
> add_hugetlb_folio() returns a folio to the HugeTLB pool when vmemmap
> restoration fails. It clears folio->private and restores only
> HPG_vmemmap_optimized, losing other flags that must survive the rollback.
>
> Clearing HPG_cma causes a later free to bypass the CMA release path,
> leaving the CMA allocation bitmap set. Clearing HPG_raw_hwp_unreliable
> loses the indication that the locations of poisoned subpages are unknown,
> undermining the checks that prevent freeing or unpoisoning such folios.
>
> Preserve the existing flags instead of resetting folio->private. Callers
> already clear HPG_restore_reserve, HPG_migratable and HPG_freed before
> returning a folio to the pool. Clear HPG_temporary in free_huge_folio()
> before attempting to free a temporary folio, so a failed vmemmap restore
> does not leave that flag set on a folio returned as surplus.
>
> Add a VM_WARN_ON_ONCE_FOLIO() check for flags other than
> HPG_vmemmap_optimized, HPG_raw_hwp_unreliable and HPG_cma to detect
> incomplete transient flag cleanup.
>
> Fixes: 161df60e9e89 ("mm, hwpoison, hugetlb: support saving mechanism of raw error pages")
> Fixes: d2d786714080 ("mm/hugetlb: enable bootmem allocation from CMA areas")
> Closes: https://sashiko.dev/#/bug/linux-5363eb09-9c3c-4f8b-9f3c-a998a8c4b258
> Suggested-by: David Hildenbrand <david@kernel.org>
> Suggested-by: Muchun Song <muchun.song@linux.dev>
> Suggested-by: Zi Yan <ziy@nvidia.com>
> Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
> ---
>  mm/hugetlb.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
>

Acked-by: Zi Yan <ziy@nvidia.com>


-- 
Best Regards,
Yan, Zi

Re: [PATCH v2 2/3] mm/hugetlb: Preserve persistent flags when returning folios to the pool

[Qi Zheng]

On 11/2/26 10:49 AM, Kaitao Cheng wrote:
> From: Kaitao Cheng <chengkaitao@kylinos.cn>
> 
> add_hugetlb_folio() returns a folio to the HugeTLB pool when vmemmap
> restoration fails. It clears folio->private and restores only
> HPG_vmemmap_optimized, losing other flags that must survive the rollback.
> 
> Clearing HPG_cma causes a later free to bypass the CMA release path,
> leaving the CMA allocation bitmap set. Clearing HPG_raw_hwp_unreliable
> loses the indication that the locations of poisoned subpages are unknown,
> undermining the checks that prevent freeing or unpoisoning such folios.
> 
> Preserve the existing flags instead of resetting folio->private. Callers
> already clear HPG_restore_reserve, HPG_migratable and HPG_freed before
> returning a folio to the pool. Clear HPG_temporary in free_huge_folio()
> before attempting to free a temporary folio, so a failed vmemmap restore
> does not leave that flag set on a folio returned as surplus.
> 
> Add a VM_WARN_ON_ONCE_FOLIO() check for flags other than
> HPG_vmemmap_optimized, HPG_raw_hwp_unreliable and HPG_cma to detect
> incomplete transient flag cleanup.
> 
> Fixes: 161df60e9e89 ("mm, hwpoison, hugetlb: support saving mechanism of raw error pages")
> Fixes: d2d786714080 ("mm/hugetlb: enable bootmem allocation from CMA areas")
> Closes: https://sashiko.dev/#/bug/linux-5363eb09-9c3c-4f8b-9f3c-a998a8c4b258
> Suggested-by: David Hildenbrand <david@kernel.org>
> Suggested-by: Muchun Song <muchun.song@linux.dev>
> Suggested-by: Zi Yan <ziy@nvidia.com>
> Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
> ---
>   mm/hugetlb.c | 10 +++++++---
>   1 file changed, 7 insertions(+), 3 deletions(-)

Acked-by: Qi Zheng <qi.zheng@linux.dev>

Thanks,
Qi

[PATCH v2 3/3] mm/hugetlb: Fix NR_HUGETLB accounting for surplus folios

[Kaitao Cheng]

From: Kaitao Cheng <chengkaitao@kylinos.cn>

NR_HUGETLB is incremented in the allocation entry points, but decremented
unconditionally in free_huge_folio(). Surplus folios allocated while
gathering reservations can be freed without passing through those entry
points, leaving an unmatched decrement in the per-node counter.

This happens when reservation gathering fails or allocates more folios
than needed, or when the overcommit recheck rejects a newly allocated
surplus folio. These paths cause nr_hugetlb in /proc/vmstat to underreport
usage.

Move node accounting to the underlying folio state transitions. Increment
NR_HUGETLB when dequeuing a free folio or allocating a fresh surplus or
migration folio, and decrement it when returning the folio to the pool or
removing it from active use. Balance the temporary increment when surplus
folios gathered for reservations are added to the free pool.

Separate memcg statistics from node accounting, updating them after a
successful charge and before uncharging. This keeps node accounting
balanced independently of whether the folio is charged to a memcg.

Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
Closes: https://sashiko.dev/#/bug/linux-860a3b8f-0b07-4cd5-a122-aa2288463bee
Suggested-by: Muchun Song <muchun.song@linux.dev>
Suggested-by: Oscar Salvador <osalvador@suse.de>
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
---
 mm/hugetlb.c    | 42 +++++++++++++++++++++++++++---------------
 mm/memcontrol.c |  6 +-----
 2 files changed, 28 insertions(+), 20 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 931ca3a30773..d1d7e14858d3 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1288,6 +1288,8 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
 		folio_clear_hugetlb_freed(folio);
 		h->free_huge_pages--;
 		h->free_huge_pages_node[nid]--;
+		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
+				    pages_per_huge_page(h));
 		return folio;
 	}
 
@@ -1416,6 +1418,9 @@ void remove_hugetlb_folio(struct hstate *h, struct folio *folio,
 		folio_clear_hugetlb_freed(folio);
 		h->free_huge_pages--;
 		h->free_huge_pages_node[nid]--;
+	} else {
+		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
+				    -(long)pages_per_huge_page(h));
 	}
 	if (adjust_surplus) {
 		h->surplus_huge_pages--;
@@ -1693,6 +1698,17 @@ struct hstate *size_to_hstate(unsigned long size)
 	return NULL;
 }
 
+static void mod_hugetlb_memcg_state(struct folio *folio, int nr_pages)
+{
+	struct mem_cgroup *memcg;
+
+	guard(rcu)();
+	memcg = folio_memcg(folio);
+	if (memcg)
+		mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, folio_pgdat(folio)),
+				       NR_HUGETLB, nr_pages);
+}
+
 void free_huge_folio(struct folio *folio)
 {
 	/*
@@ -1740,7 +1756,7 @@ void free_huge_folio(struct folio *folio)
 				     pages_per_huge_page(h), folio);
 	hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),
 					  pages_per_huge_page(h), folio);
-	lruvec_stat_mod_folio(folio, NR_HUGETLB, -pages_per_huge_page(h));
+	mod_hugetlb_memcg_state(folio, -pages_per_huge_page(h));
 	mem_cgroup_uncharge(folio);
 	if (restore_reserve)
 		h->resv_huge_pages++;
@@ -1756,6 +1772,8 @@ void free_huge_folio(struct folio *folio)
 		spin_unlock_irqrestore(&hugetlb_lock, flags);
 		update_and_free_hugetlb_folio(h, folio, true);
 	} else {
+		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
+				    -(long)pages_per_huge_page(h));
 		arch_clear_hugetlb_flags(folio);
 		enqueue_hugetlb_folio(h, folio);
 		spin_unlock_irqrestore(&hugetlb_lock, flags);
@@ -2123,6 +2141,9 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
 	 */
 	account_new_hugetlb_folio(h, folio);
 
+	/* The overcommit recheck below can free this folio. */
+	mod_node_page_state(folio_pgdat(folio), NR_HUGETLB, pages_per_huge_page(h));
+
 	/*
 	 * We could have raced with the pool size change.
 	 * Double check that and simply deallocate the new page
@@ -2160,6 +2181,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mas
 
 	spin_lock_irq(&hugetlb_lock);
 	account_new_hugetlb_folio(h, folio);
+	mod_node_page_state(folio_pgdat(folio), NR_HUGETLB, pages_per_huge_page(h));
 	spin_unlock_irq(&hugetlb_lock);
 
 	/* fresh huge pages are frozen */
@@ -2218,10 +2240,6 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 
 	spin_unlock_irq(&hugetlb_lock);
 
-	if (folio)
-		lruvec_stat_mod_folio(folio, NR_HUGETLB,
-				      folio_nr_pages(folio));
-
 	return folio;
 }
 
@@ -2249,9 +2267,6 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
 						    nmask);
 	}
 
-	if (folio)
-		lruvec_stat_mod_folio(folio, NR_HUGETLB, folio_nr_pages(folio));
-
 	return folio;
 }
 
@@ -2369,6 +2384,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)
 		if ((--needed) < 0)
 			break;
 		/* Add the page to the hugetlb allocator */
+		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
+				    -(long)pages_per_huge_page(h));
 		enqueue_hugetlb_folio(h, folio);
 	}
 free:
@@ -2944,13 +2961,6 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
 
 	ret = mem_cgroup_charge_hugetlb(folio, mm,
 					gfp | __GFP_RETRY_MAYFAIL);
-	/*
-	 * Unconditionally increment NR_HUGETLB here because if
-	 * mem_cgroup_charge_hugetlb failed, freeing the page will
-	 * decrement NR_HUGETLB.
-	 */
-	lruvec_stat_mod_folio(folio, NR_HUGETLB, nr_pages);
-
 	if (ret == -ENOMEM) {
 		folio_put(folio);
 		/*
@@ -2965,6 +2975,8 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
 		return ERR_PTR(-ENOSPC);
 	}
 
+	mod_hugetlb_memcg_state(folio, nr_pages);
+
 	return folio;
 
  err_uncharge_hugetlb_cgroup:
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 84f4a8fd4d16..3bfe4b108170 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5417,11 +5417,7 @@ int mem_cgroup_charge_hugetlb(struct folio *folio, struct mm_struct *mm,
 	struct mem_cgroup *memcg = get_mem_cgroup_from_mm(mm);
 	int ret = 0;
 
-	/*
-	 * Even memcg does not account for hugetlb, we still want to update
-	 * system-level stats via lruvec_stat_mod_folio. Return 0, and skip
-	 * charging the memcg.
-	 */
+	/* Skip charging unless hugetlb accounting is enabled for cgroup v2. */
 	if (mem_cgroup_disabled() || !memcg_accounts_hugetlb() ||
 		!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
 		goto out;
-- 
2.54.0 (Apple Git-157)

Re: [PATCH v2 3/3] mm/hugetlb: Fix NR_HUGETLB accounting for surplus folios

[Muchun Song]

> On Nov 2, 2026, at 10:49, Kaitao Cheng <kaitao.cheng@linux.dev> wrote:
> 
> From: Kaitao Cheng <chengkaitao@kylinos.cn>
> 
> NR_HUGETLB is incremented in the allocation entry points, but decremented
> unconditionally in free_huge_folio(). Surplus folios allocated while
> gathering reservations can be freed without passing through those entry
> points, leaving an unmatched decrement in the per-node counter.
> 
> This happens when reservation gathering fails or allocates more folios
> than needed, or when the overcommit recheck rejects a newly allocated
> surplus folio. These paths cause nr_hugetlb in /proc/vmstat to underreport
> usage.
> 
> Move node accounting to the underlying folio state transitions. Increment
> NR_HUGETLB when dequeuing a free folio or allocating a fresh surplus or
> migration folio, and decrement it when returning the folio to the pool or
> removing it from active use. Balance the temporary increment when surplus
> folios gathered for reservations are added to the free pool.
> 
> Separate memcg statistics from node accounting, updating them after a
> successful charge and before uncharging. This keeps node accounting
> balanced independently of whether the folio is charged to a memcg.
> 
> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> Closes: https://sashiko.dev/#/bug/linux-860a3b8f-0b07-4cd5-a122-aa2288463bee
> Suggested-by: Muchun Song <muchun.song@linux.dev>
> Suggested-by: Oscar Salvador <osalvador@suse.de>
> Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
> ---
> mm/hugetlb.c    | 42 +++++++++++++++++++++++++++---------------
> mm/memcontrol.c |  6 +-----
> 2 files changed, 28 insertions(+), 20 deletions(-)

I want to keep it simple for a backport fix, so your last version is good
to me, and this one should be a cleanup based on the top of that.

Thanks.

> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 931ca3a30773..d1d7e14858d3 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -1288,6 +1288,8 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
> 		folio_clear_hugetlb_freed(folio);
> 		h->free_huge_pages--;
> 		h->free_huge_pages_node[nid]--;
> +		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
> +				    pages_per_huge_page(h));
> 		return folio;
> 	}
> 
> @@ -1416,6 +1418,9 @@ void remove_hugetlb_folio(struct hstate *h, struct folio *folio,
> 		folio_clear_hugetlb_freed(folio);
> 		h->free_huge_pages--;
> 		h->free_huge_pages_node[nid]--;
> +	} else {
> +		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
> +				    -(long)pages_per_huge_page(h));
> 	}
> 	if (adjust_surplus) {
> 		h->surplus_huge_pages--;
> @@ -1693,6 +1698,17 @@ struct hstate *size_to_hstate(unsigned long size)
> 	return NULL;
> }
> 
> +static void mod_hugetlb_memcg_state(struct folio *folio, int nr_pages)
> +{
> +	struct mem_cgroup *memcg;
> +
> +	guard(rcu)();
> +	memcg = folio_memcg(folio);
> +	if (memcg)
> +		mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, folio_pgdat(folio)),
> +				       NR_HUGETLB, nr_pages);
> +}
> +
> void free_huge_folio(struct folio *folio)
> {
> 	/*
> @@ -1740,7 +1756,7 @@ void free_huge_folio(struct folio *folio)
> 				     pages_per_huge_page(h), folio);
> 	hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h),
> 					  pages_per_huge_page(h), folio);
> -	lruvec_stat_mod_folio(folio, NR_HUGETLB, -pages_per_huge_page(h));
> +	mod_hugetlb_memcg_state(folio, -pages_per_huge_page(h));
> 	mem_cgroup_uncharge(folio);
> 	if (restore_reserve)
> 		h->resv_huge_pages++;
> @@ -1756,6 +1772,8 @@ void free_huge_folio(struct folio *folio)
> 		spin_unlock_irqrestore(&hugetlb_lock, flags);
> 		update_and_free_hugetlb_folio(h, folio, true);
> 	} else {
> +		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
> +				    -(long)pages_per_huge_page(h));
> 		arch_clear_hugetlb_flags(folio);
> 		enqueue_hugetlb_folio(h, folio);
> 		spin_unlock_irqrestore(&hugetlb_lock, flags);
> @@ -2123,6 +2141,9 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h,
> 	 */
> 	account_new_hugetlb_folio(h, folio);
> 
> +	/* The overcommit recheck below can free this folio. */
> +	mod_node_page_state(folio_pgdat(folio), NR_HUGETLB, pages_per_huge_page(h));
> +
> 	/*
> 	 * We could have raced with the pool size change.
> 	 * Double check that and simply deallocate the new page
> @@ -2160,6 +2181,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mas
> 
> 	spin_lock_irq(&hugetlb_lock);
> 	account_new_hugetlb_folio(h, folio);
> +	mod_node_page_state(folio_pgdat(folio), NR_HUGETLB, pages_per_huge_page(h));
> 	spin_unlock_irq(&hugetlb_lock);
> 
> 	/* fresh huge pages are frozen */
> @@ -2218,10 +2240,6 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
> 
> 	spin_unlock_irq(&hugetlb_lock);
> 
> -	if (folio)
> -		lruvec_stat_mod_folio(folio, NR_HUGETLB,
> -				      folio_nr_pages(folio));
> -
> 	return folio;
> }
> 
> @@ -2249,9 +2267,6 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
> 						    nmask);
> 	}
> 
> -	if (folio)
> -		lruvec_stat_mod_folio(folio, NR_HUGETLB, folio_nr_pages(folio));
> -
> 	return folio;
> }
> 
> @@ -2369,6 +2384,8 @@ static int gather_surplus_pages(struct hstate *h, long delta)
> 		if ((--needed) < 0)
> 			break;
> 		/* Add the page to the hugetlb allocator */
> +		mod_node_page_state(folio_pgdat(folio), NR_HUGETLB,
> +				    -(long)pages_per_huge_page(h));
> 		enqueue_hugetlb_folio(h, folio);
> 	}
> free:
> @@ -2944,13 +2961,6 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
> 
> 	ret = mem_cgroup_charge_hugetlb(folio, mm,
> 					gfp | __GFP_RETRY_MAYFAIL);
> -	/*
> -	 * Unconditionally increment NR_HUGETLB here because if
> -	 * mem_cgroup_charge_hugetlb failed, freeing the page will
> -	 * decrement NR_HUGETLB.
> -	 */
> -	lruvec_stat_mod_folio(folio, NR_HUGETLB, nr_pages);
> -
> 	if (ret == -ENOMEM) {
> 		folio_put(folio);
> 		/*
> @@ -2965,6 +2975,8 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
> 		return ERR_PTR(-ENOSPC);
> 	}
> 
> +	mod_hugetlb_memcg_state(folio, nr_pages);
> +
> 	return folio;
> 
>  err_uncharge_hugetlb_cgroup:
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 84f4a8fd4d16..3bfe4b108170 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -5417,11 +5417,7 @@ int mem_cgroup_charge_hugetlb(struct folio *folio, struct mm_struct *mm,
> 	struct mem_cgroup *memcg = get_mem_cgroup_from_mm(mm);
> 	int ret = 0;
> 
> -	/*
> -	 * Even memcg does not account for hugetlb, we still want to update
> -	 * system-level stats via lruvec_stat_mod_folio. Return 0, and skip
> -	 * charging the memcg.
> -	 */
> +	/* Skip charging unless hugetlb accounting is enabled for cgroup v2. */
> 	if (mem_cgroup_disabled() || !memcg_accounts_hugetlb() ||
> 		!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
> 		goto out;
> -- 
> 2.54.0 (Apple Git-157)
>