mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v4 00/16] Fix bugs on HugeTLB folio allocation failure paths
@ 2026-07-22 23:41 Ackerley Tng via B4 Relay
  2026-07-22 23:41 ` [PATCH v4 01/16] mm: hugetlb: Track used_hpages when getting/putting pages from subpool Ackerley Tng via B4 Relay
                   ` (15 more replies)
  0 siblings, 16 replies; 18+ messages in thread
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
  To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
	Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma,
	fvdl, rientjes, jthoughton, Mike Kravetz, Johannes Weiner,
	Michal Hocko, Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett,
	Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
	Jonathan Corbet, Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu,
	Hongxiang Lou, Miaohe Lin
  Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
	linux-doc, Ackerley Tng, stable

This series fixes a few bugs on HugeTLB folio allocation failure paths.

Some of these issues were pointed out by Sashiko while I was working on
[1], and others were revealed while fixing the original issues.

1. subpool usage leak on allocation failure (Sashiko pointed this out
   in [2])

   When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
   hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips
   restoring the page to the subpool, leaking the subpool's used_hpages
   counter. Fix this by calling hugepage_subpool_put_pages()
   unconditionally if map_chg is true.

2. Folio refcount mismatch on memcg charge failure leading to
   VM_BUG_ON_FOLIO() (Sashiko pointed this out in [3])

   The error path in alloc_hugetlb_folio() calls free_huge_folio() directly
   on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier).
   This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if
   CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise.
   Fix this by using folio_put() instead of free_huge_folio() to correctly
   drop the refcount before freeing.

3. Returning -ENOMEM on memcg charge failure causes infinite loop

   alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps
   to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and
   use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning
   VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and
   retries the faulting instruction indefinitely. Fix this by returning
   -ENOSPC instead of -ENOMEM on charge failure, which maps to
   VM_FAULT_SIGBUS, terminating the process cleanly.

4. vma reservation leak on mem_cgroup_charge_hugetlb() failure

   When mem_cgroup_charge_hugetlb() fails, the error path historically
   bypassed vma_end_reservation(). Since the reservation had already been
   committed via vma_commit_reservation(), this left the reservation map in
   an inconsistent state, leaking resv_huge_pages when the process
   exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in
   alloc_hugetlb_folio(), before vma_commit_reservation() is called.

Cleaned up patches, updated documentation, and fixed Sashiko's comments on
v3:

1. subpool cleanup in alloc_hugetlb_folio() was incomplete - I folded the
   fix in.
2. More issues (UAF, and another accounting issue) reported [5] - I added
   two more patches.

Keeping this section since I think people didn't have much time to review
v3.

1. subpool page tracking

The subpool now tracks every page allocated via
the subpool through used_hpages, regardless of whether min_hpages or
max_hpages is requested by the user.

When min_hpages is set, rsv_hpages is initialized to be equal to
min_hpages. As reserved pages are used, the count is transferred from
rsv_hpages to used_hpages. When reserved pages run out, as long as
max_hpages is not requested or not yet met, used_hpages continues to be
incremented. The following invariant holds true while used_hpages <=
min_hpages:

  used_hpages + rsv_hpages = min_hpages

when used_hpages exceeds min_hpages, rsv_hpages must always be 0.

With this, pages can and must always be returned to the subpool if there's
some reservation or allocation failure after getting a page from the
subpool. One thing I missed in v2 was in processing the returned page. In
v2, the number of pages returned was compared to min_hpages, ignoring
used_hpages if it exists. v3 limits the number of pages returned to
min_hpages - used_hpages, upholding the invariant above.

2. (Re-)introduction of the try-commit-cancel protocol for memcg charging.

In v2, I took the shortcut where I moved mem_cgroup_charge_hugetlb() up
above committing vma reservations, and jumped to subpool cleanup when the
charging failed. With that, Sashiko pointed out that there are still issues
with subpool tracking.

In [4] I thought the issue was that gbl_chg == 0 (I flipped the condition
in [4], omg) doesn't always mean a reservation was actually used, but after
digging more, that doesn't seem to be true, since if gbl_chg == 0,
restore_reserve is set on the folio, and free_huge_folio() would have done
a h->resv_huge_pages++.

In my opinion, calling free_huge_folio() in the middle of the cleanup does
too much, and I think the cleaner approach would be to do all the charging
symmetrically, so that the subpool cleanup will also be symmetric
and (hopefully) easier to understand.

+ I would like reviews on all the non-WIP patches. Those are the ones that
  I'd like merged first.
+ There are reproducers (not meant to be merged) to help prove
  effectiveness of fixes.
+ There are WIP patches for subpool-related refactoring. Those helped me
  check that the subpool stuff is working and is posted to show how it
  works. I think the subpool-related refactoring and tests would be good to
  merge, but perhaps in another patch series? The refactoring was heavily
  AI-driven and should be considered RFC. Would like to have your feedback
  on that refactoring too.

+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes
+ Reproducers all pass

v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com
v2: https://lore.kernel.org/r/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b@google.com
v3: https://lore.kernel.org/r/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee@google.com

The series [1] has some changes that are dependent on these fixes, so I'll wait
for your reviews before continuing on [1].

Thank you!

[1] https://lore.kernel.org/all/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6
[4] https://lore.kernel.org/all/CAEvNRgGN0HSJ2iLSDD2haSKOxifa-uhkO9Hwossh0+Q_d9fzOw@mail.gmail.com/
[5] https://sashiko.dev/#/patchset/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee%40google.com?part=8

Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (16):
      mm: hugetlb: Track used_hpages when getting/putting pages from subpool
      mm: hugetlb: Return -ENOSPC on memcg charge failure
      mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios
      mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function
      mm: hugetlb: Fix subpool usage leak on allocation failure
      mm: hugetlb: Rename local variables for clarity in hugetlb_reserve_pages()
      mm: hugetlb: Fix Use-After-Free in unlock_or_release_subpool()
      fs: hugetlbfs: Fix global reservation leak in hugetlbfs_fill_super()
      WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c
      WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API
      WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c
      WIP: tools: testing: Add unit tests for HugeTLB subpool functions
      WIP: Reproducer for allocation failure due to cgroup v2 memory limits
      WIP: Reproducer for subpool usage leak
      WIP: Reproducer for false restoration on shared HugeTLB mappings
      WIP: Reproducer for out_put_pages subpool reserve leakage

 Documentation/mm/hugetlbfs_reserv.rst              |  17 +-
 .../translations/zh_CN/mm/hugetlbfs_reserv.rst     |  11 +-
 cgroup_v2_allocation_failure.c                     | 169 +++++++++
 fs/hugetlbfs/inode.c                               |  31 +-
 hugetlb_reserve_pages_out_put_pages.c              |  49 +++
 hugetlb_reserve_pages_out_put_pages.sh             | 153 ++++++++
 include/linux/hugetlb.h                            |  17 +-
 include/linux/memcontrol.h                         |  30 +-
 mm/Makefile                                        |   2 +-
 mm/hugetlb.c                                       | 260 +++-----------
 mm/hugetlb_subpool.c                               | 230 ++++++++++++
 mm/hugetlb_subpool.h                               |  21 ++
 mm/memcontrol.c                                    | 116 +++++-
 subpool_leak_max_size.sh                           |  72 ++++
 subpool_shared_leak.c                              |  43 +++
 subpool_shared_leak.sh                             |  87 +++++
 tools/testing/hugetlb_subpool/.gitignore           |   1 +
 tools/testing/hugetlb_subpool/Makefile             |  18 +
 tools/testing/hugetlb_subpool/test_subpool.c       | 400 +++++++++++++++++++++
 19 files changed, 1445 insertions(+), 282 deletions(-)
---
base-commit: 4539944e515183668109bdf4d0c3d7d228383d88
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f

Best regards,
--
Ackerley Tng <ackerleytng@google.com>



^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2026-07-29 17:15 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-07-22 23:41 [PATCH v4 00/16] Fix bugs on HugeTLB folio allocation failure paths Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 01/16] mm: hugetlb: Track used_hpages when getting/putting pages from subpool Ackerley Tng via B4 Relay
2026-07-29 17:14   ` Ackerley Tng
2026-07-22 23:41 ` [PATCH v4 02/16] mm: hugetlb: Return -ENOSPC on memcg charge failure Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 03/16] mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 04/16] mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 05/16] mm: hugetlb: Fix subpool usage leak on allocation failure Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 06/16] mm: hugetlb: Rename local variables for clarity in hugetlb_reserve_pages() Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 07/16] mm: hugetlb: Fix Use-After-Free in unlock_or_release_subpool() Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 08/16] fs: hugetlbfs: Fix global reservation leak in hugetlbfs_fill_super() Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 09/16] WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 10/16] WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 11/16] WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 12/16] WIP: tools: testing: Add unit tests for HugeTLB subpool functions Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 13/16] WIP: Reproducer for allocation failure due to cgroup v2 memory limits Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 14/16] WIP: Reproducer for subpool usage leak Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 15/16] WIP: Reproducer for false restoration on shared HugeTLB mappings Ackerley Tng via B4 Relay
2026-07-22 23:41 ` [PATCH v4 16/16] WIP: Reproducer for out_put_pages subpool reserve leakage Ackerley Tng via B4 Relay

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®