[PATCH 0/2] mm/hugetlb_cgroup: move the per-node usage along with the folio

[PATCH 0/2] mm/hugetlb_cgroup: move the per-node usage along with the folio

[Hongfu Li]

The per-node usage reported by hugetlb.<size>.numa_stat is accounted
against folio_nid() in __hugetlb_cgroup_commit_charge() and
__hugetlb_cgroup_uncharge_folio(), so it is only correct while a folio
stays charged on the same node and in the same hugetlb_cgroup.

Two paths move a folio which stays charged, and neither moves the usage
with it. hugetlb_cgroup_migrate() only moves the hugetlb_cgroup pointers
of a folio migrated to another node, and hugetlb_cgroup_move_parent()
only moves the page_counter charges and the hugetlb_cgroup pointer of
the folios of a dying cgroup. In both cases the node (or cgroup) which
was charged keeps a usage which never goes away, while the node
(or cgroup) which ends up uncharging the folio underflows as soon as
the folio is freed.

Patch 1/2 moves the usage along with the folio on cross node migration,
patch 2/2 does the same for the folios a dying cgroup reparents.

Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
---
Hongfu Li (2):
      mm/hugetlb_cgroup: move per-node usage on cross node migration
      mm/hugetlb_cgroup: move per-node usage on cgroup reparenting

 mm/hugetlb_cgroup.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
---
base-commit: baa8de2f3448d1466a888a805c18d01c998fe052
change-id: 20260915-for-hugetlb-charge-2cfada8ed44e

Best regards,
--  
Hongfu Li <lihongfu@kylinos.cn>

[PATCH 1/2] mm/hugetlb_cgroup: move per-node usage on cross node migration

[Hongfu Li]

From: Hongfu Li <lihongfu@kylinos.cn>

hugetlb.<size>.numa_stat uses folio_nid() to account usage in
__hugetlb_cgroup_commit_charge() and __hugetlb_cgroup_uncharge_folio().
hugetlb_cgroup_migrate() only moves hugetlb_cgroup pointers, leaving
per-node usage behind on the source node during cross-node migration.

When the migrated folio gets uncharged, we subtract usage from the
destination node counter. This creates stale usage on the source node
and unsigned long counter underflow on the destination node.
The hugetlb.<size>.numa_stat interface exposes these incorrect per-node
usage values to userspace.

Add a hugetlb_cgroup_move_usage() helper which moves the usage from the
old node to the new node, and call it from hugetlb_cgroup_migrate().

Fixes: f47761999052 ("hugetlb: add hugetlb.*.numa_stat file")
Cc: stable@vger.kernel.org
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
---
 mm/hugetlb_cgroup.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c
index ecb6e0b7819a..1040406e7e1d 100644
--- a/mm/hugetlb_cgroup.c
+++ b/mm/hugetlb_cgroup.c
@@ -179,6 +179,34 @@ static void hugetlb_cgroup_css_free(struct cgroup_subsys_state *css)
 	hugetlb_cgroup_free(hugetlb_cgroup_from_css(css));
 }
 
+static void hugetlb_cgroup_move_usage(struct hugetlb_cgroup *from,
+				      struct hugetlb_cgroup *to,
+				      struct folio *from_folio,
+				      struct folio *to_folio)
+{
+	int idx = hstate_index(folio_hstate(from_folio));
+	unsigned long nr_pages = folio_nr_pages(from_folio);
+	int from_nid = folio_nid(from_folio);
+	int to_nid = folio_nid(to_folio);
+	unsigned long usage;
+
+	lockdep_assert_held(&hugetlb_lock);
+
+	if (!from || !to)
+		return;
+
+	if (from == to && from_nid == to_nid)
+		return;
+
+	usage = READ_ONCE(from->nodeinfo[from_nid]->usage[idx]);
+	if (WARN_ON_ONCE(usage < nr_pages))
+		return;
+	WRITE_ONCE(from->nodeinfo[from_nid]->usage[idx], usage - nr_pages);
+
+	usage = READ_ONCE(to->nodeinfo[to_nid]->usage[idx]);
+	WRITE_ONCE(to->nodeinfo[to_nid]->usage[idx], usage + nr_pages);
+}
+
 /*
  * Should be called with hugetlb_lock held.
  * Since we are holding hugetlb_lock, pages cannot get moved from
@@ -906,6 +934,9 @@ void hugetlb_cgroup_migrate(struct folio *old_folio, struct folio *new_folio)
 	/* move the h_cg details to new cgroup */
 	set_hugetlb_cgroup(new_folio, h_cg);
 	set_hugetlb_cgroup_rsvd(new_folio, h_cg_rsvd);
+
+	hugetlb_cgroup_move_usage(h_cg, h_cg, old_folio, new_folio);
+
 	list_move(&new_folio->lru, &h->hugepage_activelist);
 	spin_unlock_irq(&hugetlb_lock);
 }

-- 
2.54.0

Re: [PATCH 1/2] mm/hugetlb_cgroup: move per-node usage on cross node migration

[Muchun Song]

> On Sep 16, 2026, at 16:30, Hongfu Li <hongfu.li@linux.dev> wrote:
> 
> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> hugetlb.<size>.numa_stat uses folio_nid() to account usage in
> __hugetlb_cgroup_commit_charge() and __hugetlb_cgroup_uncharge_folio().
> hugetlb_cgroup_migrate() only moves hugetlb_cgroup pointers, leaving
> per-node usage behind on the source node during cross-node migration.
> 
> When the migrated folio gets uncharged, we subtract usage from the
> destination node counter. This creates stale usage on the source node
> and unsigned long counter underflow on the destination node.
> The hugetlb.<size>.numa_stat interface exposes these incorrect per-node
> usage values to userspace.
> 
> Add a hugetlb_cgroup_move_usage() helper which moves the usage from the
> old node to the new node, and call it from hugetlb_cgroup_migrate().
> 
> Fixes: f47761999052 ("hugetlb: add hugetlb.*.numa_stat file")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
> ---
> mm/hugetlb_cgroup.c | 31 +++++++++++++++++++++++++++++++
> 1 file changed, 31 insertions(+)
> 
> diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c
> index ecb6e0b7819a..1040406e7e1d 100644
> --- a/mm/hugetlb_cgroup.c
> +++ b/mm/hugetlb_cgroup.c
> @@ -179,6 +179,34 @@ static void hugetlb_cgroup_css_free(struct cgroup_subsys_state *css)
> hugetlb_cgroup_free(hugetlb_cgroup_from_css(css));
> }
> 
> +static void hugetlb_cgroup_move_usage(struct hugetlb_cgroup *from,
> + 				     struct hugetlb_cgroup *to,
> + 				     struct folio *from_folio,
> +  			             struct folio *to_folio)
> +{
> + 	int idx = hstate_index(folio_hstate(from_folio));
> + 	unsigned long nr_pages = folio_nr_pages(from_folio);
> + 	int from_nid = folio_nid(from_folio);
> + 	int to_nid = folio_nid(to_folio);
> + 	unsigned long usage;
> +
> + 	lockdep_assert_held(&hugetlb_lock);
> +
> + 	if (!from || !to)
> + 		return;
> +
> + 	if (from == to && from_nid == to_nid)
> + 		return;
> +
> + 	usage = READ_ONCE(from->nodeinfo[from_nid]->usage[idx]);

READ_ONCE is unnecessary because there is no concurrent writer.

> + 	if (WARN_ON_ONCE(usage < nr_pages))
> + 		return;
> + 	WRITE_ONCE(from->nodeinfo[from_nid]->usage[idx], usage - nr_pages);
> +
> + 	usage = READ_ONCE(to->nodeinfo[to_nid]->usage[idx]);

Same here.

> + 	WRITE_ONCE(to->nodeinfo[to_nid]->usage[idx], usage + nr_pages);
> +}
> +
> /*
>  * Should be called with hugetlb_lock held.
>  * Since we are holding hugetlb_lock, pages cannot get moved from
> @@ -906,6 +934,9 @@ void hugetlb_cgroup_migrate(struct folio *old_folio, struct folio *new_folio)
> 	/* move the h_cg details to new cgroup */
> 	set_hugetlb_cgroup(new_folio, h_cg);
> 	set_hugetlb_cgroup_rsvd(new_folio, h_cg_rsvd);
> +
> + 	hugetlb_cgroup_move_usage(h_cg, h_cg, old_folio, new_folio);
> +
> 	list_move(&new_folio->lru, &h->hugepage_activelist);
> 	spin_unlock_irq(&hugetlb_lock);
> }
> 
> -- 
> 2.54.0
> 

Re: [PATCH 1/2] mm/hugetlb_cgroup: move per-node usage on cross node migration

[Hongfu Li]

On 9/17/26 1:58 PM, Muchun Song wrote:
>
>> On Sep 16, 2026, at 16:30, Hongfu Li <hongfu.li@linux.dev> wrote:
>>
>> From: Hongfu Li <lihongfu@kylinos.cn>
>>
>> hugetlb.<size>.numa_stat uses folio_nid() to account usage in
>> __hugetlb_cgroup_commit_charge() and __hugetlb_cgroup_uncharge_folio().
>> hugetlb_cgroup_migrate() only moves hugetlb_cgroup pointers, leaving
>> per-node usage behind on the source node during cross-node migration.
>>
>> When the migrated folio gets uncharged, we subtract usage from the
>> destination node counter. This creates stale usage on the source node
>> and unsigned long counter underflow on the destination node.
>> The hugetlb.<size>.numa_stat interface exposes these incorrect per-node
>> usage values to userspace.
>>
>> Add a hugetlb_cgroup_move_usage() helper which moves the usage from the
>> old node to the new node, and call it from hugetlb_cgroup_migrate().
>>
>> Fixes: f47761999052 ("hugetlb: add hugetlb.*.numa_stat file")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
>> ---
>> mm/hugetlb_cgroup.c | 31 +++++++++++++++++++++++++++++++
>> 1 file changed, 31 insertions(+)
>>
>> diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c
>> index ecb6e0b7819a..1040406e7e1d 100644
>> --- a/mm/hugetlb_cgroup.c
>> +++ b/mm/hugetlb_cgroup.c
>> @@ -179,6 +179,34 @@ static void hugetlb_cgroup_css_free(struct cgroup_subsys_state *css)
>> hugetlb_cgroup_free(hugetlb_cgroup_from_css(css));
>> }
>>
>> +static void hugetlb_cgroup_move_usage(struct hugetlb_cgroup *from,
>> + 				     struct hugetlb_cgroup *to,
>> + 				     struct folio *from_folio,
>> +  			             struct folio *to_folio)
>> +{
>> + 	int idx = hstate_index(folio_hstate(from_folio));
>> + 	unsigned long nr_pages = folio_nr_pages(from_folio);
>> + 	int from_nid = folio_nid(from_folio);
>> + 	int to_nid = folio_nid(to_folio);
>> + 	unsigned long usage;
>> +
>> + 	lockdep_assert_held(&hugetlb_lock);
>> +
>> + 	if (!from || !to)
>> + 		return;
>> +
>> + 	if (from == to && from_nid == to_nid)
>> + 		return;
>> +
>> + 	usage = READ_ONCE(from->nodeinfo[from_nid]->usage[idx]);
> READ_ONCE is unnecessary because there is no concurrent writer.

Thanks for the review.

You're right, the READ_ONCE is indeed unnecessary here. I'll drop it in v2.

>> + 	if (WARN_ON_ONCE(usage < nr_pages))
>> + 		return;
>> + 	WRITE_ONCE(from->nodeinfo[from_nid]->usage[idx], usage - nr_pages);
>> +
>> + 	usage = READ_ONCE(to->nodeinfo[to_nid]->usage[idx]);
> Same here.
>
>> + 	WRITE_ONCE(to->nodeinfo[to_nid]->usage[idx], usage + nr_pages);
>> +}
>> +
>> /*
>>   * Should be called with hugetlb_lock held.
>>   * Since we are holding hugetlb_lock, pages cannot get moved from
>> @@ -906,6 +934,9 @@ void hugetlb_cgroup_migrate(struct folio *old_folio, struct folio *new_folio)
>> 	/* move the h_cg details to new cgroup */
>> 	set_hugetlb_cgroup(new_folio, h_cg);
>> 	set_hugetlb_cgroup_rsvd(new_folio, h_cg_rsvd);
>> +
>> + 	hugetlb_cgroup_move_usage(h_cg, h_cg, old_folio, new_folio);
>> +
>> 	list_move(&new_folio->lru, &h->hugepage_activelist);
>> 	spin_unlock_irq(&hugetlb_lock);
>> }
>>
>> -- 
>> 2.54.0
>>
-- 
Best regards,
Hongfu

[PATCH 2/2] mm/hugetlb_cgroup: move per-node usage on cgroup reparenting

[Hongfu Li]

From: Hongfu Li <lihongfu@kylinos.cn>

hugetlb_cgroup_css_offline() hands the folios of a dying cgroup over to
its parent with hugetlb_cgroup_move_parent(), which moves the page_counter
charges and the hugetlb_cgroup pointer of the folio but not its per-node
usage.  The parent's hugetlb.<size>.numa_stat is short by that usage while
they are charged, and underflows once they are freed, exposing incorrect
per-node usage values to userspace.

Move the per-node usage to the parent as well.  The folios keep their node
here, so only the cgroup which holds the usage changes.

Fixes: f47761999052 ("hugetlb: add hugetlb.*.numa_stat file")
Cc: stable@vger.kernel.org
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
---
 mm/hugetlb_cgroup.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c
index 1040406e7e1d..3f5ec4a06b25 100644
--- a/mm/hugetlb_cgroup.c
+++ b/mm/hugetlb_cgroup.c
@@ -241,6 +241,8 @@ static void hugetlb_cgroup_move_parent(int idx, struct hugetlb_cgroup *h_cg,
 	/* Take the pages off the local counter */
 	page_counter_cancel(counter, nr_pages);
 
+	hugetlb_cgroup_move_usage(h_cg, parent, folio, folio);
+
 	set_hugetlb_cgroup(folio, parent);
 out:
 	return;

-- 
2.54.0

Re: [PATCH 2/2] mm/hugetlb_cgroup: move per-node usage on cgroup reparenting

[Muchun Song]

> On Sep 16, 2026, at 16:30, Hongfu Li <hongfu.li@linux.dev> wrote:
> 
> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> hugetlb_cgroup_css_offline() hands the folios of a dying cgroup over to
> its parent with hugetlb_cgroup_move_parent(), which moves the page_counter
> charges and the hugetlb_cgroup pointer of the folio but not its per-node
> usage.  The parent's hugetlb.<size>.numa_stat is short by that usage while
> they are charged, and underflows once they are freed, exposing incorrect
> per-node usage values to userspace.
> 
> Move the per-node usage to the parent as well.  The folios keep their node
> here, so only the cgroup which holds the usage changes.
> 
> Fixes: f47761999052 ("hugetlb: add hugetlb.*.numa_stat file")
> Cc: stable@vger.kernel.org
> Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>

Acked-by: Muchun Song <muchun.song@linux.dev>

Re: [PATCH 0/2] mm/hugetlb_cgroup: move the per-node usage along with the folio

[Andrew Morton]

On Wed, 16 Sep 2026 16:30:26 +0800 Hongfu Li <hongfu.li@linux.dev> wrote:

> The per-node usage reported by hugetlb.<size>.numa_stat is accounted
> against folio_nid() in __hugetlb_cgroup_commit_charge() and
> __hugetlb_cgroup_uncharge_folio(), so it is only correct while a folio
> stays charged on the same node and in the same hugetlb_cgroup.
> 
> Two paths move a folio which stays charged, and neither moves the usage
> with it. hugetlb_cgroup_migrate() only moves the hugetlb_cgroup pointers
> of a folio migrated to another node, and hugetlb_cgroup_move_parent()
> only moves the page_counter charges and the hugetlb_cgroup pointer of
> the folios of a dying cgroup. In both cases the node (or cgroup) which
> was charged keeps a usage which never goes away, while the node
> (or cgroup) which ends up uncharging the folio underflows as soon as
> the folio is freed.
> 
> Patch 1/2 moves the usage along with the folio on cross node migration,
> patch 2/2 does the same for the folios a dying cgroup reparents.

As I understand it, this series affects numa_stat output but has no
actual runtime effects?

Do you (and maintainers) think that we should fix this in earlier
kernels?

(I do - a lot of userspace infrastructure makes expensive decisions
based on this sort of info).

Re: [PATCH 0/2] mm/hugetlb_cgroup: move the per-node usage along with the folio

[Hongfu Li]

On 9/17/26 8:53 AM, Andrew Morton wrote:
> On Wed, 16 Sep 2026 16:30:26 +0800 Hongfu Li <hongfu.li@linux.dev> wrote:
>
>> The per-node usage reported by hugetlb.<size>.numa_stat is accounted
>> against folio_nid() in __hugetlb_cgroup_commit_charge() and
>> __hugetlb_cgroup_uncharge_folio(), so it is only correct while a folio
>> stays charged on the same node and in the same hugetlb_cgroup.
>>
>> Two paths move a folio which stays charged, and neither moves the usage
>> with it. hugetlb_cgroup_migrate() only moves the hugetlb_cgroup pointers
>> of a folio migrated to another node, and hugetlb_cgroup_move_parent()
>> only moves the page_counter charges and the hugetlb_cgroup pointer of
>> the folios of a dying cgroup. In both cases the node (or cgroup) which
>> was charged keeps a usage which never goes away, while the node
>> (or cgroup) which ends up uncharging the folio underflows as soon as
>> the folio is freed.
>>
>> Patch 1/2 moves the usage along with the folio on cross node migration,
>> patch 2/2 does the same for the folios a dying cgroup reparents.
> As I understand it, this series affects numa_stat output but has no
> actual runtime effects?

Yes, that's right. The per-node usage[] is only read by
hugetlb_cgroup_read_numa_stat(), so this series only fixes the values
reported in hugetlb.<size>.numa_stat. It has no effect on charging or
limit enforcement, which go through page_counter and are already
handled correctly by hugetlb_cgroup_migrate() and
hugetlb_cgroup_move_parent().

>
> Do you (and maintainers) think that we should fix this in earlier
> kernels?
>
> (I do - a lot of userspace infrastructure makes expensive decisions
> based on this sort of info).

I agree with you that this should go to stable.

Patch 1/2 and Patch 2/2 already carry Fixes: and Cc: stable tags.

-- 
Best regards,
Hongfu