[PATCH] mm/hugetlb: charge hugetlb cgroup and memcg in alloc_hugetlb_folio_reserve()

[PATCH] mm/hugetlb: charge hugetlb cgroup and memcg in alloc_hugetlb_folio_reserve()

[Hongfu Li]

From: Hongfu Li <lihongfu@kylinos.cn>

alloc_hugetlb_folio_reserve() hands out a folio from the hugetlb pool and
consumes a global reservation, but unlike the fault path it never charged
the folio to the hugetlb cgroup or to the memcg.  Folios allocated for
memfd_pin_folios() through memfd_alloc_folio() were therefore silently
bypassing the accounting: they are never charged to the hugetlb cgroup
usage counter (hugetlb.<size>.current) nor to the memcg (memory.current),
so both limits (hugetlb.<size>.max, memory.max) are bypassed.

Charge the folios to both the hugetlb cgroup and the memcg, just like
hugetlb_alloc_folio() does.

Test with a process that creates a hugetlb memfd and pins pages through
UDMABUF_CREATE. The counters were read while the folio was still pinned:

Before this patch:
$ echo $$ > /sys/fs/cgroup/test/cgroup.procs
$ ./memfd-alloc-folio 1 &
$ cat /sys/fs/cgroup/test/hugetlb.2MB.current
0
$ cat /sys/fs/cgroup/test/memory.current
1351680
$

After this patch:
$ echo $$ > /sys/fs/cgroup/test/cgroup.procs
$ ./memfd-alloc-folio 1 &
$ cat /sys/fs/cgroup/test/hugetlb.2MB.current
2097152
$ cat /sys/fs/cgroup/test/memory.current
3313664
$

Fixes: 26a8ea80929c ("mm/hugetlb: fix memfd_pin_folios resv_huge_pages leak")
Cc: stable@vger.kernel.org
Signed-off-by: Hongfu Li <lihongfu@kylinos.cn>
---
 mm/hugetlb.c | 33 +++++++++++++++++++++++++++++----
 1 file changed, 29 insertions(+), 4 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index b37d7d0db7fc..62b819b84faa 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2193,12 +2193,19 @@ struct folio *alloc_buddy_hugetlb_folio(struct hstate *h,
 struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 		nodemask_t *nmask, gfp_t gfp_mask)
 {
+	struct hugetlb_cgroup *h_cg = NULL;
+	size_t nr_pages = pages_per_huge_page(h);
 	struct folio *folio;
+	int idx = hstate_index(h);
+	int ret;
+
+	if (hugetlb_cgroup_charge_cgroup(idx, nr_pages, &h_cg))
+		return NULL;
 
 	spin_lock_irq(&hugetlb_lock);
 	if (!h->resv_huge_pages) {
 		spin_unlock_irq(&hugetlb_lock);
-		return NULL;
+		goto err_uncharge_hugetlb_cgroup;
 	}
 
 	folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, preferred_nid,
@@ -2206,15 +2213,33 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid,
 	if (folio) {
 		folio_set_hugetlb_restore_reserve(folio);
 		h->resv_huge_pages--;
+		hugetlb_cgroup_commit_charge(idx, nr_pages, h_cg, folio);
 	}
 
 	spin_unlock_irq(&hugetlb_lock);
 
-	if (folio)
-		lruvec_stat_mod_folio(folio, NR_HUGETLB,
-				      folio_nr_pages(folio));
+	if (!folio)
+		goto err_uncharge_hugetlb_cgroup;
+
+	ret = mem_cgroup_charge_hugetlb(folio, current->mm,
+					gfp_mask | __GFP_RETRY_MAYFAIL);
+	/*
+	 * Unconditionally increment NR_HUGETLB here because if
+	 * mem_cgroup_charge_hugetlb failed, freeing the page will
+	 * decrement NR_HUGETLB.
+	 */
+	lruvec_stat_mod_folio(folio, NR_HUGETLB, nr_pages);
+
+	if (ret) {
+		folio_put(folio);
+		return NULL;
+	}
 
 	return folio;
+
+err_uncharge_hugetlb_cgroup:
+	hugetlb_cgroup_uncharge_cgroup(idx, nr_pages, h_cg);
+	return NULL;
 }
 
 /* folio migration callback function */

---
base-commit: 2887e63d2e8e0cf55402612d8996ce955a2c3e55
change-id: 20261020-for-alloc_hugetlb-1020-00e2dd7a56d1

Best regards,
--  
Hongfu Li <lihongfu@kylinos.cn>

Re: [PATCH] mm/hugetlb: charge hugetlb cgroup and memcg in alloc_hugetlb_folio_reserve()

[Andrew Morton]

On Tue, 20 Oct 2026 16:46:30 +0800 Hongfu Li <hongfu.li@linux.dev> wrote:

> From: Hongfu Li <lihongfu@kylinos.cn>
> 
> alloc_hugetlb_folio_reserve() hands out a folio from the hugetlb pool and
> consumes a global reservation, but unlike the fault path it never charged
> the folio to the hugetlb cgroup or to the memcg.  Folios allocated for
> memfd_pin_folios() through memfd_alloc_folio() were therefore silently
> bypassing the accounting: they are never charged to the hugetlb cgroup
> usage counter (hugetlb.<size>.current) nor to the memcg (memory.current),
> so both limits (hugetlb.<size>.max, memory.max) are bypassed.
> 
> Charge the folios to both the hugetlb cgroup and the memcg, just like
> hugetlb_alloc_folio() does.
> 
> Test with a process that creates a hugetlb memfd and pins pages through
> UDMABUF_CREATE. The counters were read while the folio was still pinned:
> 
> Before this patch:
> $ echo $$ > /sys/fs/cgroup/test/cgroup.procs
> $ ./memfd-alloc-folio 1 &
> $ cat /sys/fs/cgroup/test/hugetlb.2MB.current
> 0
> $ cat /sys/fs/cgroup/test/memory.current
> 1351680
> $
> 
> After this patch:
> $ echo $$ > /sys/fs/cgroup/test/cgroup.procs
> $ ./memfd-alloc-folio 1 &
> $ cat /sys/fs/cgroup/test/hugetlb.2MB.current
> 2097152
> $ cat /sys/fs/cgroup/test/memory.current
> 3313664
> $

Thanks.  The before-and-after output is useful.

However there are potential interactions with Daehyeon Ko's
"mm/hugetlb_cgroup: prevent use-after-free during cgroup offlining". 
https://lore.kernel.org/20261020100345.1677643-1-4ncienth@gmail.com

Your patch adds another caller of hugetlb_cgroup_charge_cgroup(), which
has the cgroup-offlining CSS lifetime race addressed by Daehyeon Ko's
fix.

An existing memfd reservation can belong to cgroup A while the task
pinning the folio belongs to cgroup B. A's reservation CSS reference
does not protect B's usage CSS across charge and commit.

Also, the accounting bypass predates 26a8ea80929c.  The original
memfd_pin_folios() implementation already used an uncharged allocator,
so the Fixes tag should reference 89c1905d9c14 instead.

The memory.max claim should be qualified: HugeTLB accounting in memcg
is conditional on the cgroup v2 hierarchy's accounting configuration.

Daehyeon's fix is in the pipeline at present, awaiting a minor fixup
and maintainer review.  Everything is rather swamped at present.

So please assume that Daehyeon's fix is present and rethink your fix on
top of that?

Re: [PATCH] mm/hugetlb: charge hugetlb cgroup and memcg in alloc_hugetlb_folio_reserve()

[Hongfu Li]

On 10/22/26 7:59 AM, Andrew Morton wrote:
> On Tue, 20 Oct 2026 16:46:30 +0800 Hongfu Li <hongfu.li@linux.dev> wrote:
>
>> From: Hongfu Li <lihongfu@kylinos.cn>
>>
>> alloc_hugetlb_folio_reserve() hands out a folio from the hugetlb pool and
>> consumes a global reservation, but unlike the fault path it never charged
>> the folio to the hugetlb cgroup or to the memcg.  Folios allocated for
>> memfd_pin_folios() through memfd_alloc_folio() were therefore silently
>> bypassing the accounting: they are never charged to the hugetlb cgroup
>> usage counter (hugetlb.<size>.current) nor to the memcg (memory.current),
>> so both limits (hugetlb.<size>.max, memory.max) are bypassed.
>>
>> Charge the folios to both the hugetlb cgroup and the memcg, just like
>> hugetlb_alloc_folio() does.
>>
>> Test with a process that creates a hugetlb memfd and pins pages through
>> UDMABUF_CREATE. The counters were read while the folio was still pinned:
>>
>> Before this patch:
>> $ echo $$ > /sys/fs/cgroup/test/cgroup.procs
>> $ ./memfd-alloc-folio 1 &
>> $ cat /sys/fs/cgroup/test/hugetlb.2MB.current
>> 0
>> $ cat /sys/fs/cgroup/test/memory.current
>> 1351680
>> $
>>
>> After this patch:
>> $ echo $$ > /sys/fs/cgroup/test/cgroup.procs
>> $ ./memfd-alloc-folio 1 &
>> $ cat /sys/fs/cgroup/test/hugetlb.2MB.current
>> 2097152
>> $ cat /sys/fs/cgroup/test/memory.current
>> 3313664
>> $
> Thanks.  The before-and-after output is useful.
>
> However there are potential interactions with Daehyeon Ko's
> "mm/hugetlb_cgroup: prevent use-after-free during cgroup offlining".
> https://lore.kernel.org/20261020100345.1677643-1-4ncienth@gmail.com
>
> Your patch adds another caller of hugetlb_cgroup_charge_cgroup(), which
> has the cgroup-offlining CSS lifetime race addressed by Daehyeon Ko's
> fix.
>
> An existing memfd reservation can belong to cgroup A while the task
> pinning the folio belongs to cgroup B. A's reservation CSS reference
> does not protect B's usage CSS across charge and commit.
>
> Also, the accounting bypass predates 26a8ea80929c.  The original
> memfd_pin_folios() implementation already used an uncharged allocator,
> so the Fixes tag should reference 89c1905d9c14 instead.

Agreed, the bypass has been there since memfd_pin_folios() was
introduced. I'll update the Fixes tag.

> The memory.max claim should be qualified: HugeTLB accounting in memcg
> is conditional on the cgroup v2 hierarchy's accounting configuration.

Right, I'll qualify that in the commit message.

> Daehyeon's fix is in the pipeline at present, awaiting a minor fixup
> and maintainer review.  Everything is rather swamped at present.
>
> So please assume that Daehyeon's fix is present and rethink your fix on
> top of that?

Sure, I'll rework the patch on top of his fix and resend as v2 after his
series has been reviewed.  I'll re-run the test with both applied before
sending.

-- 
Best regards,
Hongfu