[PATCH v4 01/13] mm: hugetlb_vmemmap: fix OOM during head page allocation

[Muchun Song <songmuchun@bytedance.com>]

From: Yili Zhang <zhangyili01@baidu.com>

The head page used during HugeTLB vmemmap optimization is allocated with
a bare GFP_KERNEL mask. When boot-time hugepage allocation drains free
memory, parallel optimization workers can enter OOM handling before
userspace exists, leaving the kernel with no killable process and causing
a panic.

hugetlb_pages_alloc_boot_node() invokes HVO on the folios accumulated so
far when available memory reaches zero, allowing the released vmemmap pages
to support further hugepage allocations. If allocating the HVO head page
returns -ENOMEM, the bulk HVO path frees any vmemmap pages it has already
accumulated and retries, while callers can continue with an unoptimized
folio if the retry still fails. Therefore, neither an allocation warning
nor OOM handling helps this best-effort allocation. Use GFP_KERNEL |
__GFP_NOWARN | __GFP_RETRY_MAYFAIL to allow reclaim retries while avoiding
both.

Fixes: c0b495b91a47 ("mm/hugetlb: refactor code around vmemmap_walk")
Cc: stable@vger.kernel.org
Signed-off-by: Yili Zhang <zhangyili01@baidu.com>
Co-developed-by: Muchun Song <songmuchun@bytedance.com>
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v4:
- Split this patch out from patch 4.
- After splitting it out, the resulting change was almost identical to the
  patch at:
  https://lore.kernel.org/r/20261104133938.50974-1-zhangyili01@baidu.com/
  Retain the author of that patch and only rework the subject and commit
  message.
---
 mm/hugetlb_vmemmap.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index b8ce014ba6e6..7fcc97b12773 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -501,6 +501,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 {
 	unsigned long vmemmap_start, vmemmap_end;
 	struct page *vmemmap_head, *vmemmap_tail;
+	gfp_t gfp_mask = GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
 	int nid, ret = 0;
 
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(folio), folio);
@@ -527,7 +528,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 	 */
 	folio_set_hugetlb_vmemmap_optimized(folio);
 
-	vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);
+	vmemmap_head = alloc_pages_node(nid, gfp_mask, 0);
 	if (!vmemmap_head) {
 		ret = -ENOMEM;
 		goto out;
-- 
2.54.0