[PATCH] mm/hugetlb: do not invoke the OOM killer when optimizing vmemmap

[PATCH] mm/hugetlb: do not invoke the OOM killer when optimizing vmemmap

[Yili Zhang]

Boot-time hugepage allocation is designed to drain all free memory and
let the vmemmap optimization reclaim vmemmap pages, so that more
hugepages can be allocated afterwards:
hugetlb_pages_alloc_boot_node() flushes the folios accumulated so far
once si_mem_available() drops to zero, and hugetlb_pages_alloc_boot()
keeps retrying as long as progress is being made.  The allocation is
parallelized with padata and runs on many threads at once.

The page that will back the new head struct page of a folio being
optimized is allocated with a bare GFP_KERNEL mask in
__hugetlb_vmemmap_optimize_folio():

	vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);

Once the workers have consumed all free memory, this allocation enters
the OOM path.  During boot-time allocation, at the do_one_initcall()
stage, there are no user processes yet, so out_of_memory() finds
nothing to kill and panics:

  Out of memory and no killable processes...
  Kernel panic - not syncing: System is deadlocked on memory
  CPU: 74 UID: 0 PID: 1 Comm: swapper/0
  Call Trace:
    vpanic+0x32f/0x360
    panic+0x57/0x60
    out_of_memory+0x581/0x590
    __alloc_pages_slowpath.constprop.115+0x97d/0xed0
    __alloc_frozen_pages_noprof+0x303/0x330
    __alloc_pages_noprof+0xa/0x20
    __hugetlb_vmemmap_optimize_folio+0xa3/0x210
    __hugetlb_vmemmap_optimize_folios+0x190/0x230
    prep_and_add_allocated_folios+0x16/0x90
    hugetlb_pages_alloc_boot_node+0x152/0x180
    padata_mt_helper+0x5b/0xb0
    padata_do_multithreaded+0x294/0x3a0
    hugetlb_hstate_alloc_pages+0x2e1/0x3b0
    hugetlb_init+0x3b5/0x640
    do_one_initcall+0x49/0x200

Optimizing the vmemmap is best-effort by design: on allocation failure,
__hugetlb_vmemmap_optimize_folios() frees the vmemmap pages reclaimed
from the folios it has processed so far and retries once, and folios
that still cannot be optimized are put into the pool unoptimized by
prep_and_add_allocated_folios().  The worst outcome of a failure is a
higher vmemmap overhead and, as a consequence, fewer hugepages being
allocated (along with a warning) - waking up the OOM killer is never
the right thing to do here.  Note that the same code also runs for
runtime hugepage allocations, where an OOM invocation may kill an
unrelated process instead of failing the optimization gracefully.

The head page is a vmemmap page, and every other vmemmap page
allocation involved is prepared to fail gracefully:
vmemmap_alloc_block() - which vmemmap_shared_tail_page() also uses
to allocate the shared tail vmemmap pages - uses
GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN, and
alloc_vmemmap_page_list() uses GFP_KERNEL|__GFP_RETRY_MAYFAIL.

Align it with the other vmemmap page allocations by using the same mask
that vmemmap_alloc_block() uses for vmemmap pages, so that a failure
returns -ENOMEM to the caller instead of going through the OOM path.

This was observed on a 384-CPU machine with ~1.5TiB of RAM booting
with a hugepages= reservation close to the total memory size and
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y, where the boot
panicked intermittently, depending on how the parallel allocation
workers interleaved.

Fixes: c0b495b91a47b ("mm/hugetlb: refactor code around vmemmap_walk")
Signed-off-by: Yili Zhang <zhangyili01@baidu.com>
---
 mm/hugetlb_vmemmap.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 0057fa2a16a8..3a759b930ed9 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -499,6 +499,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 {
 	unsigned long vmemmap_start, vmemmap_end;
 	struct page *vmemmap_head, *vmemmap_tail;
+	gfp_t gfp_mask = GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;
 	int nid, ret = 0;
 
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(folio), folio);
@@ -525,7 +526,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 	 */
 	folio_set_hugetlb_vmemmap_optimized(folio);
 
-	vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);
+	vmemmap_head = alloc_pages_node(nid, gfp_mask, 0);
 	if (!vmemmap_head) {
 		ret = -ENOMEM;
 		goto out;
-- 
2.27.0

Re: [PATCH] mm/hugetlb: do not invoke the OOM killer when optimizing vmemmap

[Muchun Song]

> On Nov 4, 2026, at 21:39, Yili Zhang <zhangyili01@baidu.com> wrote:
> 
> Boot-time hugepage allocation is designed to drain all free memory and
> let the vmemmap optimization reclaim vmemmap pages, so that more
> hugepages can be allocated afterwards:
> hugetlb_pages_alloc_boot_node() flushes the folios accumulated so far
> once si_mem_available() drops to zero, and hugetlb_pages_alloc_boot()
> keeps retrying as long as progress is being made.  The allocation is
> parallelized with padata and runs on many threads at once.
> 
> The page that will back the new head struct page of a folio being
> optimized is allocated with a bare GFP_KERNEL mask in
> __hugetlb_vmemmap_optimize_folio():
> 
> vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);
> 
> Once the workers have consumed all free memory, this allocation enters
> the OOM path.  During boot-time allocation, at the do_one_initcall()
> stage, there are no user processes yet, so out_of_memory() finds
> nothing to kill and panics:
> 
>  Out of memory and no killable processes...
>  Kernel panic - not syncing: System is deadlocked on memory
>  CPU: 74 UID: 0 PID: 1 Comm: swapper/0
>  Call Trace:
>    vpanic+0x32f/0x360
>    panic+0x57/0x60
>    out_of_memory+0x581/0x590
>    __alloc_pages_slowpath.constprop.115+0x97d/0xed0
>    __alloc_frozen_pages_noprof+0x303/0x330
>    __alloc_pages_noprof+0xa/0x20
>    __hugetlb_vmemmap_optimize_folio+0xa3/0x210
>    __hugetlb_vmemmap_optimize_folios+0x190/0x230
>    prep_and_add_allocated_folios+0x16/0x90
>    hugetlb_pages_alloc_boot_node+0x152/0x180
>    padata_mt_helper+0x5b/0xb0
>    padata_do_multithreaded+0x294/0x3a0
>    hugetlb_hstate_alloc_pages+0x2e1/0x3b0
>    hugetlb_init+0x3b5/0x640
>    do_one_initcall+0x49/0x200
> 
> Optimizing the vmemmap is best-effort by design: on allocation failure,
> __hugetlb_vmemmap_optimize_folios() frees the vmemmap pages reclaimed
> from the folios it has processed so far and retries once, and folios
> that still cannot be optimized are put into the pool unoptimized by
> prep_and_add_allocated_folios().  The worst outcome of a failure is a
> higher vmemmap overhead and, as a consequence, fewer hugepages being
> allocated (along with a warning) - waking up the OOM killer is never
> the right thing to do here.  Note that the same code also runs for
> runtime hugepage allocations, where an OOM invocation may kill an
> unrelated process instead of failing the optimization gracefully.
> 
> The head page is a vmemmap page, and every other vmemmap page
> allocation involved is prepared to fail gracefully:
> vmemmap_alloc_block() - which vmemmap_shared_tail_page() also uses
> to allocate the shared tail vmemmap pages - uses
> GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN, and
> alloc_vmemmap_page_list() uses GFP_KERNEL|__GFP_RETRY_MAYFAIL.
> 
> Align it with the other vmemmap page allocations by using the same mask
> that vmemmap_alloc_block() uses for vmemmap pages, so that a failure
> returns -ENOMEM to the caller instead of going through the OOM path.
> 
> This was observed on a 384-CPU machine with ~1.5TiB of RAM booting
> with a hugepages= reservation close to the total memory size and
> CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y, where the boot
> panicked intermittently, depending on how the parallel allocation
> workers interleaved.

A really detailed analysis, but also lengthy (I think it's generated from
AI, It really should be trimmed before sending because the issue is very
obvious and easy to understand).

> 
> Fixes: c0b495b91a47b ("mm/hugetlb: refactor code around vmemmap_walk")
> Signed-off-by: Yili Zhang <zhangyili01@baidu.com>
> ---
> mm/hugetlb_vmemmap.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
> index 0057fa2a16a8..3a759b930ed9 100644
> --- a/mm/hugetlb_vmemmap.c
> +++ b/mm/hugetlb_vmemmap.c
> @@ -499,6 +499,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
> {
> 	unsigned long vmemmap_start, vmemmap_end;
> 	struct page *vmemmap_head, *vmemmap_tail;
> +	gfp_t gfp_mask = GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;

Thanks for fixing, but I realized that I've already fix it here [1], and I didn't
realize it's a real production issue. If you don't mind, I could add your Reported-by,
and make it a very small fix patch so that it can be backported more easily.

[1] https://lore.kernel.org/all/20261104023510.260766-5-songmuchun@bytedance.com/

Thanks,
Muchun

> 	int nid, ret = 0;
> 
> 	VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(folio), folio);
> @@ -525,7 +526,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
> */
> 	folio_set_hugetlb_vmemmap_optimized(folio);
> 
> -	vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);
> +	vmemmap_head = alloc_pages_node(nid, gfp_mask, 0);
> 	if (!vmemmap_head) {
> 		ret = -ENOMEM;
> 		goto out;
> -- 
> 2.27.0
> 

Re: [PATCH] mm/hugetlb: do not invoke the OOM killer when optimizing vmemmap

[Yili Zhang]

On 2026-11-05 10:38, Muchun Song wrote:
> 
> Thanks for fixing, but I realized that I've already fix it here [1], and I didn't
> realize it's a real production issue. If you don't mind, I could add your Reported-by,
> and make it a very small fix patch so that it can be backported more easily.
> 
> [1] https://lore.kernel.org/all/20261104023510.260766-5-songmuchun@bytedance.com/
> 
> Thanks,
> Muchun
> 
 
I checked [1] and it fixes the issue. Please add:

  Reported-by: Yili Zhang <zhangyili01@baidu.com>

Thanks,
Yili