[PATCH 0/2] mm: fix hugetlb NR_HUGETLB accounting on folio migration

[PATCH 0/2] mm: fix hugetlb NR_HUGETLB accounting on folio migration

[Hongfu Li]

The hugeTLB counters added by 05d4532b60e3 ("memcg/hugetlb: add hugeTLB
counters to memcg") are maintained in two per-node places:
  - the per-node vmstat counter NR_HUGETLB, exposed as nr_hugetlb in
    /proc/vmstat;
  - the per-node memcg lruvec stat, exposed via memory.numa_stat.

Both are accounted against the folio's node, and both drift when a
hugetlb folio is migrated, though in different ways.

A migration target folio is allocated by alloc_hugetlb_folio_nodemask()
and inherits the old folio's state without ever being accounted, while
the old folio is freed right after and its free is accounted. That
alone loses vmstat accounting: the target node has no matching increment
for the decrement on the old node, so /proc/vmstat's nr_hugetlb shrinks
by nr_pages per migration. Patch 1 accounts the folio where it is
obtained, so the increment pairs with the free in free_huge_folio() on
the successful as well as the failed migration path. The memfd page
cache preallocation helper has the same asymmetry and is fixed in the
same patch.

The per-node lruvec stat breaks differently. mem_cgroup_migrate()
moves the charge to the new folio and drops the old folio's memcg data,
so the old folio's free right after migration skips the memcg per-node
decrement; the count stays attributed to the old node for the rest of
the charge's life, and the target folio never gets an increment on its
new node. Patch 2 moves that per-node accounting alongside the charge
in mem_cgroup_migrate().

---
Hongfu Li (2):
      mm/hugetlb: account migration target folio in per-node NR_HUGETLB vmstat
      mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

 mm/hugetlb.c    | 35 +++++++++++++++++++++++------------
 mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 12 deletions(-)
---
base-commit: b08a65b93426d86e3f354d655d6225397b591877
change-id: 20260916-for-hugetlb_state-e671831c1bb8

Best regards,
--  
Hongfu Li <lihongfu@kylinos.cn>

[PATCH 1/2] mm/hugetlb: account migration target folio in per-node NR_HUGETLB vmstat

[Hongfu Li]

From: Hongfu Li <lihongfu@kylinos.cn>

The NR_HUGETLB vmstat counter is maintained per folio's node: incremented
when a huge page is handed to a user via hugetlb_alloc_folio() and
decremented when it is returned to the pool via free_huge_folio().

A folio obtained by alloc_hugetlb_folio_nodemask() never goes through
hugetlb_alloc_folio(), so it is never accounted, while its free always
is. For a migration target this means the target node gets no matching
increment for the decrement on the old node, so the global nr_hugetlb in
/proc/vmstat drops by nr_pages for each migration. The same asymmetry
affects the failed migration path, which frees the target again right
away, and the temporary folio hugetlb_mfill_atomic_pte() takes from the
same helper.

alloc_hugetlb_folio_reserve(), used to preallocate the memfd page cache
folios, has the same asymmetry: the folio is handed to a user without
being accounted, while its free is accounted through free_huge_folio().

Account the folio where it is obtained, in alloc_hugetlb_folio_nodemask()
and alloc_hugetlb_folio_reserve(), so that the increment pairs with the
decrement in free_huge_folio(): a successful migration hands the folio
to a user, a failed one frees it again.

Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
Cc: stable@vger.kernel.org
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
---
 mm/hugetlb.c | 35 +++++++++++++++++++++++------------
 1 file changed, 23 insertions(+), 12 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 1b53ba991d36..8ccc769bf48c 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2206,6 +2206,11 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 	}
 
 	spin_unlock_irq(&hugetlb_lock);
+
+	if (folio)
+		lruvec_stat_mod_folio(folio, NR_HUGETLB,
+				      folio_nr_pages(folio));
+
 	return folio;
 }
 
@@ -2213,24 +2218,30 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
 		nodemask_t *nmask, gfp_t gfp_mask, bool allow_alloc_fallback)
 {
-	spin_lock_irq(&hugetlb_lock);
-	if (available_huge_pages(h)) {
-		struct folio *folio;
+	struct folio *folio = NULL;
 
+	spin_lock_irq(&hugetlb_lock);
+	if (available_huge_pages(h))
 		folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask,
 						preferred_nid, nmask);
-		if (folio) {
-			spin_unlock_irq(&hugetlb_lock);
-			return folio;
-		}
-	}
 	spin_unlock_irq(&hugetlb_lock);
 
-	/* We cannot fallback to other nodes, as we could break the per-node pool. */
-	if (!allow_alloc_fallback)
-		gfp_mask |= __GFP_THISNODE;
+	if (!folio) {
+		/*
+		 * We cannot fallback to other nodes, as we could break the
+		 * per-node pool.
+		 */
+		if (!allow_alloc_fallback)
+			gfp_mask |= __GFP_THISNODE;
 
-	return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask);
+		folio = alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid,
+						    nmask);
+	}
+
+	if (folio)
+		lruvec_stat_mod_folio(folio, NR_HUGETLB, folio_nr_pages(folio));
+
+	return folio;
 }
 
 static nodemask_t *policy_mbind_nodemask(gfp_t gfp)

-- 
2.54.0

Re: [PATCH 1/2] mm/hugetlb: account migration target folio in per-node NR_HUGETLB vmstat

[Joshua Hahn]

On Mon, 21 Sep 2026 17:12:34 +0800 Hongfu Li <hongfu.li@linux.dev> wrote:

Hi Hongfu,

Thanks for this fix. From the cover letter I was frowning because I was
thinking to myself "surely there's no way hugetlb accounting is this
broken..." but it seems like indeed it is. 

> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> The NR_HUGETLB vmstat counter is maintained per folio's node: incremented
> when a huge page is handed to a user via hugetlb_alloc_folio() and
> decremented when it is returned to the pool via free_huge_folio().
> 
> A folio obtained by alloc_hugetlb_folio_nodemask() never goes through
> hugetlb_alloc_folio(), so it is never accounted, while its free always
> is.

That's pretty scary! Glad that you caught it here.

> For a migration target this means the target node gets no matching
> increment for the decrement on the old node, so the global nr_hugetlb in
> /proc/vmstat drops by nr_pages for each migration. The same asymmetry
> affects the failed migration path, which frees the target again right
> away, and the temporary folio hugetlb_mfill_atomic_pte() takes from the
> same helper.
> 
> alloc_hugetlb_folio_reserve(), used to preallocate the memfd page cache
> folios, has the same asymmetry: the folio is handed to a user without
> being accounted, while its free is accounted through free_huge_folio().
> 
> Account the folio where it is obtained, in alloc_hugetlb_folio_nodemask()
> and alloc_hugetlb_folio_reserve(), so that the increment pairs with the
> decrement in free_huge_folio(): a successful migration hands the folio
> to a user, a failed one frees it again.

The changes look good to me, and I was able to reproduce the issue on
my host and confirm that after this change, the missing charge is
fully accounted for.
Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

Again, I'm super surprised that a problem this big has gone unnoticed
for so long. Thanks again for working on this fix!

I hope you have a great day : -)
Joshua

> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
> ---

Re: [PATCH 1/2] mm/hugetlb: account migration target folio in per-node NR_HUGETLB vmstat

[Muchun Song]

> On Sep 22, 2026, at 06:35, Joshua Hahn <joshua.hahnjy@gmail.com> wrote:
> 
> On Mon, 21 Sep 2026 17:12:34 +0800 Hongfu Li <hongfu.li@linux.dev> wrote:
> 
> Hi Hongfu,
> 
> Thanks for this fix. From the cover letter I was frowning because I was
> thinking to myself "surely there's no way hugetlb accounting is this
> broken..." but it seems like indeed it is. 
> 
>> From: Hongfu Li <lihongfu@kylinos.cn>
>> 
>> The NR_HUGETLB vmstat counter is maintained per folio's node: incremented
>> when a huge page is handed to a user via hugetlb_alloc_folio() and
>> decremented when it is returned to the pool via free_huge_folio().
>> 
>> A folio obtained by alloc_hugetlb_folio_nodemask() never goes through
>> hugetlb_alloc_folio(), so it is never accounted, while its free always
>> is.
> 
> That's pretty scary! Glad that you caught it here.
> 
>> For a migration target this means the target node gets no matching
>> increment for the decrement on the old node, so the global nr_hugetlb in
>> /proc/vmstat drops by nr_pages for each migration. The same asymmetry
>> affects the failed migration path, which frees the target again right
>> away, and the temporary folio hugetlb_mfill_atomic_pte() takes from the
>> same helper.
>> 
>> alloc_hugetlb_folio_reserve(), used to preallocate the memfd page cache
>> folios, has the same asymmetry: the folio is handed to a user without
>> being accounted, while its free is accounted through free_huge_folio().
>> 
>> Account the folio where it is obtained, in alloc_hugetlb_folio_nodemask()
>> and alloc_hugetlb_folio_reserve(), so that the increment pairs with the
>> decrement in free_huge_folio(): a successful migration hands the folio
>> to a user, a failed one frees it again.
> 
> The changes look good to me, and I was able to reproduce the issue on
> my host and confirm that after this change, the missing charge is
> fully accounted for.
> Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> 
> Again, I'm super surprised that a problem this big has gone unnoticed
> for so long. Thanks again for working on this fix!

I think a migration for HugeTLB is very rare. That's probably why it
took us so long to find this issue.

Thanks.

> 
> I hope you have a great day : -)
> Joshua
> 
>> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
>> ---

Re: [PATCH 1/2] mm/hugetlb: account migration target folio in per-node NR_HUGETLB vmstat

[Muchun Song]

> On Sep 21, 2026, at 17:12, Hongfu Li <hongfu.li@linux.dev> wrote:
> 
> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> The NR_HUGETLB vmstat counter is maintained per folio's node: incremented
> when a huge page is handed to a user via hugetlb_alloc_folio() and
> decremented when it is returned to the pool via free_huge_folio().
> 
> A folio obtained by alloc_hugetlb_folio_nodemask() never goes through
> hugetlb_alloc_folio(), so it is never accounted, while its free always
> is. For a migration target this means the target node gets no matching
> increment for the decrement on the old node, so the global nr_hugetlb in
> /proc/vmstat drops by nr_pages for each migration. The same asymmetry
> affects the failed migration path, which frees the target again right
> away, and the temporary folio hugetlb_mfill_atomic_pte() takes from the
> same helper.
> 
> alloc_hugetlb_folio_reserve(), used to preallocate the memfd page cache
> folios, has the same asymmetry: the folio is handed to a user without
> being accounted, while its free is accounted through free_huge_folio().
> 
> Account the folio where it is obtained, in alloc_hugetlb_folio_nodemask()
> and alloc_hugetlb_folio_reserve(), so that the increment pairs with the
> decrement in free_huge_folio(): a successful migration hands the folio
> to a user, a failed one frees it again.
> 
> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>

Acked-by: Muchun Song <muchun.song@linux.dev>

Thanks.

Re: [PATCH 1/2] mm/hugetlb: account migration target folio in per-node NR_HUGETLB vmstat

[Oscar Salvador (SUSE)]

On Mon, Sep 21, 2026 at 05:12:34PM +0800, Hongfu Li wrote:
> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> The NR_HUGETLB vmstat counter is maintained per folio's node: incremented
> when a huge page is handed to a user via hugetlb_alloc_folio() and
> decremented when it is returned to the pool via free_huge_folio().
> 
> A folio obtained by alloc_hugetlb_folio_nodemask() never goes through
> hugetlb_alloc_folio(), so it is never accounted, while its free always
> is. For a migration target this means the target node gets no matching
> increment for the decrement on the old node, so the global nr_hugetlb in
> /proc/vmstat drops by nr_pages for each migration. The same asymmetry
> affects the failed migration path, which frees the target again right
> away, and the temporary folio hugetlb_mfill_atomic_pte() takes from the
> same helper.
> 
> alloc_hugetlb_folio_reserve(), used to preallocate the memfd page cache
> folios, has the same asymmetry: the folio is handed to a user without
> being accounted, while its free is accounted through free_huge_folio().
> 
> Account the folio where it is obtained, in alloc_hugetlb_folio_nodemask()
> and alloc_hugetlb_folio_reserve(), so that the increment pairs with the
> decrement in free_huge_folio(): a successful migration hands the folio
> to a user, a failed one frees it again.
> 
> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>

Acked-by: Oscar Salvador <osalvador@suse.de>

 

-- 
Oscar Salvador
SUSE Labs

[PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Hongfu Li]

From: Hongfu Li <lihongfu@kylinos.cn>

memory.numa_stat exposes per-node hugetlb counters from per-node lruvec
stats. These stats are accounted against folio_nid(): incremented on
the folio's node when handed to a user, decremented when the folio is
returned to the pool.

During hugetlb folio migration, mem_cgroup_migrate() moves the charge
to the new folio and drops the memcg data of the old one, so the free
of the old folio right after the migration skips the memcg per-node
lruvec decrement. The hugetlb count stays attributed to the old node
for the rest of the life of the charge, while the target folio gets no
increment on the new node; its later free decrements a counter that
was never incremented.

Migrate the per-node lruvec accounting alongside migration. Global
memcg totals remain balanced because they track resource consumption,
not node placement.

Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
Cc: stable@vger.kernel.org
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
---
 mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1460cba53588..9c96ebd5436f 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5598,6 +5598,34 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
 	rcu_read_unlock();
 }
 
+#ifdef CONFIG_HUGETLB_PAGE
+static void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,
+				     struct folio *old, struct folio *new)
+{
+	long nr_pages = folio_nr_pages(old);
+	struct mem_cgroup *memcg;
+	int old_nid = folio_nid(old);
+	int new_nid = folio_nid(new);
+
+	if (old_nid == new_nid)
+		return;
+
+	rcu_read_lock();
+	memcg = obj_cgroup_memcg(objcg);
+	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)),
+			       NR_HUGETLB, -nr_pages);
+	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)),
+			       NR_HUGETLB, nr_pages);
+	rcu_read_unlock();
+}
+#else /* CONFIG_HUGETLB_PAGE */
+static inline void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,
+					    struct folio *old,
+					    struct folio *new)
+{
+}
+#endif /* CONFIG_HUGETLB_PAGE */
+
 /**
  * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio.
  * @old: Currently circulating folio.
@@ -5635,6 +5663,9 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new)
 
 	new_objcg = get_migration_objcg(old, new);
 
+	if (folio_test_hugetlb(old))
+		move_hugetlb_lruvec_stat(new_objcg, old, new);
+
 	/*
 	 * @old was charged through a non-root objcg, so its charge is in the
 	 * page counters. If the re-derivation walked up to the root objcg -

-- 
2.54.0

Re: [PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Joshua Hahn]

On Mon, 21 Sep 2026 17:12:35 +0800 Hongfu Li <hongfu.li@linux.dev> wrote:

> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> memory.numa_stat exposes per-node hugetlb counters from per-node lruvec
> stats. These stats are accounted against folio_nid(): incremented on
> the folio's node when handed to a user, decremented when the folio is
> returned to the pool.
> 
> During hugetlb folio migration, mem_cgroup_migrate() moves the charge
> to the new folio and drops the memcg data of the old one, so the free
> of the old folio right after the migration skips the memcg per-node
> lruvec decrement. The hugetlb count stays attributed to the old node
> for the rest of the life of the charge, while the target folio gets no
> increment on the new node; its later free decrements a counter that
> was never incremented.
> 
> Migrate the per-node lruvec accounting alongside migration. Global
> memcg totals remain balanced because they track resource consumption,
> not node placement.

Hi Hongfu,

This is another great catch, thank you for working on the fix!
I was able to confirm that the issue exists and was fixed by this patch.

Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>

Re: [PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Oscar Salvador (SUSE)]

On Mon, Sep 21, 2026 at 05:12:35PM +0800, Hongfu Li wrote:
> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> memory.numa_stat exposes per-node hugetlb counters from per-node lruvec
> stats. These stats are accounted against folio_nid(): incremented on
> the folio's node when handed to a user, decremented when the folio is
> returned to the pool.
> 
> During hugetlb folio migration, mem_cgroup_migrate() moves the charge
> to the new folio and drops the memcg data of the old one, so the free
> of the old folio right after the migration skips the memcg per-node
> lruvec decrement. The hugetlb count stays attributed to the old node
> for the rest of the life of the charge, while the target folio gets no
> increment on the new node; its later free decrements a counter that
> was never incremented.
> 
> Migrate the per-node lruvec accounting alongside migration. Global
> memcg totals remain balanced because they track resource consumption,
> not node placement.
> 
> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>

For the fix itself:

Reviewed-by: Oscar Salvador <osalvador@suse.de>

question below:

> ---
>  mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 1460cba53588..9c96ebd5436f 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -5598,6 +5598,34 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
>  	rcu_read_unlock();
>  }
>  
> +#ifdef CONFIG_HUGETLB_PAGE
> +static void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,
> +				     struct folio *old, struct folio *new)
> +{
> +	long nr_pages = folio_nr_pages(old);
> +	struct mem_cgroup *memcg;
> +	int old_nid = folio_nid(old);
> +	int new_nid = folio_nid(new);
> +
> +	if (old_nid == new_nid)
> +		return;
> +
> +	rcu_read_lock();
> +	memcg = obj_cgroup_memcg(objcg);
> +	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)),
> +			       NR_HUGETLB, -nr_pages);
> +	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)),
> +			       NR_HUGETLB, nr_pages);
> +	rcu_read_unlock();

Why do we need the whole thing to be embraced by rcu?

 

-- 
Oscar Salvador
SUSE Labs

Re: [PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Joshua Hahn]

On Tue, 22 Sep 2026 06:15:07 +0200 "Oscar Salvador (SUSE)" <osalvador@kernel.org> wrote:

> On Mon, Sep 21, 2026 at 05:12:35PM +0800, Hongfu Li wrote:
> > From: Hongfu Li <lihongfu@kylinos.cn>
> > 
> > memory.numa_stat exposes per-node hugetlb counters from per-node lruvec
> > stats. These stats are accounted against folio_nid(): incremented on
> > the folio's node when handed to a user, decremented when the folio is
> > returned to the pool.
> > 
> > During hugetlb folio migration, mem_cgroup_migrate() moves the charge
> > to the new folio and drops the memcg data of the old one, so the free
> > of the old folio right after the migration skips the memcg per-node
> > lruvec decrement. The hugetlb count stays attributed to the old node
> > for the rest of the life of the charge, while the target folio gets no
> > increment on the new node; its later free decrements a counter that
> > was never incremented.
> > 
> > Migrate the per-node lruvec accounting alongside migration. Global
> > memcg totals remain balanced because they track resource consumption,
> > not node placement.
> > 
> > Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
> 
> For the fix itself:
> 
> Reviewed-by: Oscar Salvador <osalvador@suse.de>
> 
> question below:
> 
> > ---
> >  mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++
> >  1 file changed, 31 insertions(+)
> > 
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index 1460cba53588..9c96ebd5436f 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -5598,6 +5598,34 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
> >  	rcu_read_unlock();
> >  }
> >  
> > +#ifdef CONFIG_HUGETLB_PAGE
> > +static void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,
> > +				     struct folio *old, struct folio *new)
> > +{
> > +	long nr_pages = folio_nr_pages(old);
> > +	struct mem_cgroup *memcg;
> > +	int old_nid = folio_nid(old);
> > +	int new_nid = folio_nid(new);
> > +
> > +	if (old_nid == new_nid)
> > +		return;
> > +
> > +	rcu_read_lock();
> > +	memcg = obj_cgroup_memcg(objcg);
> > +	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)),
> > +			       NR_HUGETLB, -nr_pages);
> > +	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)),
> > +			       NR_HUGETLB, nr_pages);
> > +	rcu_read_unlock();
> 
> Why do we need the whole thing to be embraced by rcu?

Hi Oscar,

I believe it's because now getting the memcg from the objcg requires
an RCU lock to make sure it doesn't get removed while we work on the
memcg. I think this is since Qi Zheng's "Eliminate Dying Memory
Cgroup" series.

Joshua

Re: [PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Oscar Salvador (SUSE)]

On Mon, Sep 21, 2026 at 09:20:53PM -0700, Joshua Hahn wrote:
> On Tue, 22 Sep 2026 06:15:07 +0200 "Oscar Salvador (SUSE)" <osalvador@kernel.org> wrote:

> > Why do we need the whole thing to be embraced by rcu?
> 
> Hi Oscar,

Hi Joshua,

> 
> I believe it's because now getting the memcg from the objcg requires
> an RCU lock to make sure it doesn't get removed while we work on the
> memcg. I think this is since Qi Zheng's "Eliminate Dying Memory
> Cgroup" series.

Yes, I understood that obj_cgroup_memcg might need the rcu-dance, but I was
unsure about the mod_memcg_lruvec_state() calls.


-- 
Oscar Salvador
SUSE Labs

Re: [PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Muchun Song]

> On Sep 22, 2026, at 12:30, Oscar Salvador (SUSE) <osalvador@kernel.org> wrote:
> 
> On Mon, Sep 21, 2026 at 09:20:53PM -0700, Joshua Hahn wrote:
>> On Tue, 22 Sep 2026 06:15:07 +0200 "Oscar Salvador (SUSE)" <osalvador@kernel.org> wrote:
> 
>>> Why do we need the whole thing to be embraced by rcu?
>> 
>> Hi Oscar,
> 
> Hi Joshua,
> 
>> 
>> I believe it's because now getting the memcg from the objcg requires
>> an RCU lock to make sure it doesn't get removed while we work on the
>> memcg. I think this is since Qi Zheng's "Eliminate Dying Memory
>> Cgroup" series.
> 
> Yes, I understood that obj_cgroup_memcg might need the rcu-dance, but I was
> unsure about the mod_memcg_lruvec_state() calls.

The caller needs to make sure the liveness of lruvec, in which case,
lruvec is from memcg, so rcu lock is needed here.

Thanks,
Muchun

> 
> 
> -- 
> Oscar Salvador
> SUSE Labs

Re: [PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Muchun Song]

> On Sep 21, 2026, at 17:12, Hongfu Li <hongfu.li@linux.dev> wrote:
> 
> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> memory.numa_stat exposes per-node hugetlb counters from per-node lruvec
> stats. These stats are accounted against folio_nid(): incremented on
> the folio's node when handed to a user, decremented when the folio is
> returned to the pool.
> 
> During hugetlb folio migration, mem_cgroup_migrate() moves the charge
> to the new folio and drops the memcg data of the old one, so the free
> of the old folio right after the migration skips the memcg per-node
> lruvec decrement. The hugetlb count stays attributed to the old node
> for the rest of the life of the charge, while the target folio gets no
> increment on the new node; its later free decrements a counter that
> was never incremented.
> 
> Migrate the per-node lruvec accounting alongside migration. Global
> memcg totals remain balanced because they track resource consumption,
> not node placement.
> 
> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
> ---
> mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++
> 1 file changed, 31 insertions(+)
> 
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 1460cba53588..9c96ebd5436f 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -5598,6 +5598,34 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
> rcu_read_unlock();
> }
> 
> +#ifdef CONFIG_HUGETLB_PAGE
> +static void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,

Actually, we don't need this objcg parameter since we could get it from folio.
And I'd like to move this function to hugetlb.c.

> +     				struct folio *old, struct folio *new)
> +{
> + 	long nr_pages = folio_nr_pages(old);
> + 	struct mem_cgroup *memcg;
> + 	int old_nid = folio_nid(old);
> + 	int new_nid = folio_nid(new);
> +
> + 	if (old_nid == new_nid)
> + 		return;
> +
> + 	rcu_read_lock();

Please use guard(rcu)() to simplify the code a little.

Thanks.

> + 	memcg = obj_cgroup_memcg(objcg);
> + 	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)),
> +       		NR_HUGETLB, -nr_pages);
> + 	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)),
> +       		NR_HUGETLB, nr_pages);
> + 	rcu_read_unlock();
> +}
> +#else /* CONFIG_HUGETLB_PAGE */
> +static inline void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,
> +    					struct folio *old,
> +    					struct folio *new)
> +{
> +}
> +#endif /* CONFIG_HUGETLB_PAGE */
> +
> /**
>  * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio.
>  * @old: Currently circulating folio.
> @@ -5635,6 +5663,9 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new)
> 
> 	new_objcg = get_migration_objcg(old, new);
> 
> + 	if (folio_test_hugetlb(old))
> + 		move_hugetlb_lruvec_stat(new_objcg, old, new);
> +
> /*
> * @old was charged through a non-root objcg, so its charge is in the
> * page counters. If the re-derivation walked up to the root objcg -
> 
> -- 
> 2.54.0
> 

Re: [PATCH 2/2] mm/memcg: migrate per-node hugetlb lruvec stat together with hugetlb folio

[Hongfu Li]

On 9/22/26 2:18 PM, Muchun Song wrote:
>
>> On Sep 21, 2026, at 17:12, Hongfu Li <hongfu.li@linux.dev> wrote:
>>
>> From: Hongfu Li <lihongfu@kylinos.cn>
>>
>> memory.numa_stat exposes per-node hugetlb counters from per-node lruvec
>> stats. These stats are accounted against folio_nid(): incremented on
>> the folio's node when handed to a user, decremented when the folio is
>> returned to the pool.
>>
>> During hugetlb folio migration, mem_cgroup_migrate() moves the charge
>> to the new folio and drops the memcg data of the old one, so the free
>> of the old folio right after the migration skips the memcg per-node
>> lruvec decrement. The hugetlb count stays attributed to the old node
>> for the rest of the life of the charge, while the target folio gets no
>> increment on the new node; its later free decrements a counter that
>> was never incremented.
>>
>> Migrate the per-node lruvec accounting alongside migration. Global
>> memcg totals remain balanced because they track resource consumption,
>> not node placement.
>>
>> Fixes: 05d4532b60e3 ("memcg/hugetlb: add hugeTLB counters to memcg")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
>> ---
>> mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++
>> 1 file changed, 31 insertions(+)
>>
>> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
>> index 1460cba53588..9c96ebd5436f 100644
>> --- a/mm/memcontrol.c
>> +++ b/mm/memcontrol.c
>> @@ -5598,6 +5598,34 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
>> rcu_read_unlock();
>> }
>>
>> +#ifdef CONFIG_HUGETLB_PAGE
>> +static void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,
> Actually, we don't need this objcg parameter since we could get it from folio.
> And I'd like to move this function to hugetlb.c.
>
>> +     				struct folio *old, struct folio *new)
>> +{
>> + 	long nr_pages = folio_nr_pages(old);
>> + 	struct mem_cgroup *memcg;
>> + 	int old_nid = folio_nid(old);
>> + 	int new_nid = folio_nid(new);
>> +
>> + 	if (old_nid == new_nid)
>> + 		return;
>> +
>> + 	rcu_read_lock();
> Please use guard(rcu)() to simplify the code a little.

Hi Muchun,

Thanks a lot for your suggestions. I will drop the objcg parameter of
move_hugetlb_lruvec_stat(), move the function to mm/hugetlb.c, and use
guard(rcu)() in it.  I will post a v2 for review shortly.

>> + 	memcg = obj_cgroup_memcg(objcg);
>> + 	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)),
>> +       		NR_HUGETLB, -nr_pages);
>> + 	mod_memcg_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)),
>> +       		NR_HUGETLB, nr_pages);
>> + 	rcu_read_unlock();
>> +}
>> +#else /* CONFIG_HUGETLB_PAGE */
>> +static inline void move_hugetlb_lruvec_stat(struct obj_cgroup *objcg,
>> +    					struct folio *old,
>> +    					struct folio *new)
>> +{
>> +}
>> +#endif /* CONFIG_HUGETLB_PAGE */
>> +
>> /**
>>   * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio.
>>   * @old: Currently circulating folio.
>> @@ -5635,6 +5663,9 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new)
>>
>> 	new_objcg = get_migration_objcg(old, new);
>>
>> + 	if (folio_test_hugetlb(old))
>> + 		move_hugetlb_lruvec_stat(new_objcg, old, new);
>> +
>> /*
>> * @old was charged through a non-root objcg, so its charge is in the
>> * page counters. If the re-derivation walked up to the root objcg -
>>
>> -- 
>> 2.54.0
>>
-- 
Best regards,
Hongfu