From: Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org>
To: Muchun Song <muchun.song@linux.dev>,
Oscar Salvador <osalvador@suse.de>,
David Hildenbrand <david@kernel.org>,
Joshua Hahn <joshua.hahnjy@gmail.com>,
Shakeel Butt <shakeel.butt@linux.dev>,
Nhat Pham <nphamcs@gmail.com>,
Andrew Morton <akpm@linux-foundation.org>,
Peter Xu <peterx@redhat.com>, Wupeng Ma <mawupeng1@huawei.com>,
fvdl@google.com, rientjes@google.com, jthoughton@google.com,
Mike Kravetz <mike.kravetz@oracle.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Michal Hocko <mhocko@kernel.org>,
Roman Gushchin <roman.gushchin@linux.dev>,
Lorenzo Stoakes <ljs@kernel.org>,
"Liam R. Howlett" <liam@infradead.org>,
Vlastimil Babka <vbabka@kernel.org>,
Mike Rapoport <rppt@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>, Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
Alex Shi <alexs@kernel.org>, Yanteng Si <si.yanteng@linux.dev>,
Dongliang Mu <dzm91@hust.edu.cn>,
Hongxiang Lou <louhongxiang@huawei.com>,
Miaohe Lin <linmiaohe@huawei.com>
Cc: vannapurve@google.com, erdemaktas@google.com, linux-mm@kvack.org,
linux-kernel@vger.kernel.org, cgroups@vger.kernel.org,
linux-doc@vger.kernel.org, Ackerley Tng <ackerleytng@google.com>,
stable@vger.kernel.org
Subject: [PATCH v4 00/16] Fix bugs on HugeTLB folio allocation failure paths
Date: Wed, 22 Jul 2026 16:41:08 -0700 [thread overview]
Message-ID: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com> (raw)
This series fixes a few bugs on HugeTLB folio allocation failure paths.
Some of these issues were pointed out by Sashiko while I was working on
[1], and others were revealed while fixing the original issues.
1. subpool usage leak on allocation failure (Sashiko pointed this out
in [2])
When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips
restoring the page to the subpool, leaking the subpool's used_hpages
counter. Fix this by calling hugepage_subpool_put_pages()
unconditionally if map_chg is true.
2. Folio refcount mismatch on memcg charge failure leading to
VM_BUG_ON_FOLIO() (Sashiko pointed this out in [3])
The error path in alloc_hugetlb_folio() calls free_huge_folio() directly
on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier).
This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if
CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise.
Fix this by using folio_put() instead of free_huge_folio() to correctly
drop the refcount before freeing.
3. Returning -ENOMEM on memcg charge failure causes infinite loop
alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps
to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and
use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning
VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and
retries the faulting instruction indefinitely. Fix this by returning
-ENOSPC instead of -ENOMEM on charge failure, which maps to
VM_FAULT_SIGBUS, terminating the process cleanly.
4. vma reservation leak on mem_cgroup_charge_hugetlb() failure
When mem_cgroup_charge_hugetlb() fails, the error path historically
bypassed vma_end_reservation(). Since the reservation had already been
committed via vma_commit_reservation(), this left the reservation map in
an inconsistent state, leaking resv_huge_pages when the process
exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in
alloc_hugetlb_folio(), before vma_commit_reservation() is called.
Cleaned up patches, updated documentation, and fixed Sashiko's comments on
v3:
1. subpool cleanup in alloc_hugetlb_folio() was incomplete - I folded the
fix in.
2. More issues (UAF, and another accounting issue) reported [5] - I added
two more patches.
Keeping this section since I think people didn't have much time to review
v3.
1. subpool page tracking
The subpool now tracks every page allocated via
the subpool through used_hpages, regardless of whether min_hpages or
max_hpages is requested by the user.
When min_hpages is set, rsv_hpages is initialized to be equal to
min_hpages. As reserved pages are used, the count is transferred from
rsv_hpages to used_hpages. When reserved pages run out, as long as
max_hpages is not requested or not yet met, used_hpages continues to be
incremented. The following invariant holds true while used_hpages <=
min_hpages:
used_hpages + rsv_hpages = min_hpages
when used_hpages exceeds min_hpages, rsv_hpages must always be 0.
With this, pages can and must always be returned to the subpool if there's
some reservation or allocation failure after getting a page from the
subpool. One thing I missed in v2 was in processing the returned page. In
v2, the number of pages returned was compared to min_hpages, ignoring
used_hpages if it exists. v3 limits the number of pages returned to
min_hpages - used_hpages, upholding the invariant above.
2. (Re-)introduction of the try-commit-cancel protocol for memcg charging.
In v2, I took the shortcut where I moved mem_cgroup_charge_hugetlb() up
above committing vma reservations, and jumped to subpool cleanup when the
charging failed. With that, Sashiko pointed out that there are still issues
with subpool tracking.
In [4] I thought the issue was that gbl_chg == 0 (I flipped the condition
in [4], omg) doesn't always mean a reservation was actually used, but after
digging more, that doesn't seem to be true, since if gbl_chg == 0,
restore_reserve is set on the folio, and free_huge_folio() would have done
a h->resv_huge_pages++.
In my opinion, calling free_huge_folio() in the middle of the cleanup does
too much, and I think the cleaner approach would be to do all the charging
symmetrically, so that the subpool cleanup will also be symmetric
and (hopefully) easier to understand.
+ I would like reviews on all the non-WIP patches. Those are the ones that
I'd like merged first.
+ There are reproducers (not meant to be merged) to help prove
effectiveness of fixes.
+ There are WIP patches for subpool-related refactoring. Those helped me
check that the subpool stuff is working and is posted to show how it
works. I think the subpool-related refactoring and tests would be good to
merge, but perhaps in another patch series? The refactoring was heavily
AI-driven and should be considered RFC. Would like to have your feedback
on that refactoring too.
+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes
+ Reproducers all pass
v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com
v2: https://lore.kernel.org/r/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b@google.com
v3: https://lore.kernel.org/r/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee@google.com
The series [1] has some changes that are dependent on these fixes, so I'll wait
for your reviews before continuing on [1].
Thank you!
[1] https://lore.kernel.org/all/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6
[4] https://lore.kernel.org/all/CAEvNRgGN0HSJ2iLSDD2haSKOxifa-uhkO9Hwossh0+Q_d9fzOw@mail.gmail.com/
[5] https://sashiko.dev/#/patchset/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee%40google.com?part=8
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (16):
mm: hugetlb: Track used_hpages when getting/putting pages from subpool
mm: hugetlb: Return -ENOSPC on memcg charge failure
mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios
mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function
mm: hugetlb: Fix subpool usage leak on allocation failure
mm: hugetlb: Rename local variables for clarity in hugetlb_reserve_pages()
mm: hugetlb: Fix Use-After-Free in unlock_or_release_subpool()
fs: hugetlbfs: Fix global reservation leak in hugetlbfs_fill_super()
WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c
WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API
WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c
WIP: tools: testing: Add unit tests for HugeTLB subpool functions
WIP: Reproducer for allocation failure due to cgroup v2 memory limits
WIP: Reproducer for subpool usage leak
WIP: Reproducer for false restoration on shared HugeTLB mappings
WIP: Reproducer for out_put_pages subpool reserve leakage
Documentation/mm/hugetlbfs_reserv.rst | 17 +-
.../translations/zh_CN/mm/hugetlbfs_reserv.rst | 11 +-
cgroup_v2_allocation_failure.c | 169 +++++++++
fs/hugetlbfs/inode.c | 31 +-
hugetlb_reserve_pages_out_put_pages.c | 49 +++
hugetlb_reserve_pages_out_put_pages.sh | 153 ++++++++
include/linux/hugetlb.h | 17 +-
include/linux/memcontrol.h | 30 +-
mm/Makefile | 2 +-
mm/hugetlb.c | 260 +++-----------
mm/hugetlb_subpool.c | 230 ++++++++++++
mm/hugetlb_subpool.h | 21 ++
mm/memcontrol.c | 116 +++++-
subpool_leak_max_size.sh | 72 ++++
subpool_shared_leak.c | 43 +++
subpool_shared_leak.sh | 87 +++++
tools/testing/hugetlb_subpool/.gitignore | 1 +
tools/testing/hugetlb_subpool/Makefile | 18 +
tools/testing/hugetlb_subpool/test_subpool.c | 400 +++++++++++++++++++++
19 files changed, 1445 insertions(+), 282 deletions(-)
---
base-commit: 4539944e515183668109bdf4d0c3d7d228383d88
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f
Best regards,
--
Ackerley Tng <ackerleytng@google.com>
next reply other threads:[~2026-07-22 23:41 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-22 23:41 Ackerley Tng via B4 Relay [this message]
2026-07-22 23:41 ` [PATCH v4 01/16] mm: hugetlb: Track used_hpages when getting/putting pages from subpool Ackerley Tng via B4 Relay
2026-07-29 17:14 ` Ackerley Tng
2026-07-22 23:41 ` [PATCH v4 02/16] mm: hugetlb: Return -ENOSPC on memcg charge failure Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 03/16] mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 04/16] mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 05/16] mm: hugetlb: Fix subpool usage leak on allocation failure Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 06/16] mm: hugetlb: Rename local variables for clarity in hugetlb_reserve_pages() Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 07/16] mm: hugetlb: Fix Use-After-Free in unlock_or_release_subpool() Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 08/16] fs: hugetlbfs: Fix global reservation leak in hugetlbfs_fill_super() Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 09/16] WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 10/16] WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 11/16] WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 12/16] WIP: tools: testing: Add unit tests for HugeTLB subpool functions Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 13/16] WIP: Reproducer for allocation failure due to cgroup v2 memory limits Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 14/16] WIP: Reproducer for subpool usage leak Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 15/16] WIP: Reproducer for false restoration on shared HugeTLB mappings Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 16/16] WIP: Reproducer for out_put_pages subpool reserve leakage Ackerley Tng via B4 Relay
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com \
--to=devnull+ackerleytng.google.com@kernel.org \
--cc=ackerleytng@google.com \
--cc=akpm@linux-foundation.org \
--cc=alexs@kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=corbet@lwn.net \
--cc=david@kernel.org \
--cc=dzm91@hust.edu.cn \
--cc=erdemaktas@google.com \
--cc=fvdl@google.com \
--cc=hannes@cmpxchg.org \
--cc=joshua.hahnjy@gmail.com \
--cc=jthoughton@google.com \
--cc=liam@infradead.org \
--cc=linmiaohe@huawei.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=louhongxiang@huawei.com \
--cc=mawupeng1@huawei.com \
--cc=mhocko@kernel.org \
--cc=mhocko@suse.com \
--cc=mike.kravetz@oracle.com \
--cc=muchun.song@linux.dev \
--cc=nphamcs@gmail.com \
--cc=osalvador@suse.de \
--cc=peterx@redhat.com \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=rppt@kernel.org \
--cc=shakeel.butt@linux.dev \
--cc=si.yanteng@linux.dev \
--cc=skhan@linuxfoundation.org \
--cc=stable@vger.kernel.org \
--cc=surenb@google.com \
--cc=vannapurve@google.com \
--cc=vbabka@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®