[PATCH v2 08/20] hugetlb: Fully initialize tail struct pages of non-pre-HVOed bootmem folios

[James Houghton <jthoughton@google.com>]

alloc_bootmem() marks all but the head struct page of a bootmem gigantic
folio as noinit, and gather_bootmem_prealloc_node() then only
initializes the first HUGETLB_VMEMMAP_RESERVE_PAGES struct pages. The
remaining tail struct pages are initialized only if the folio ends up
not being HVOed.

This assumes that a bootmem folio is either pre-HVOed, or will not be
HVOed at all. However, hugetlb_vmemmap_optimize_bootmem_page() may skip
pre-HVO because arch_hugetlb_vmemmap_optimization_supported() does not
yet return true that early in boot (e.g. on arm64, where support depends
on a system-wide CPU capability), while it does return true by the time
hugetlb_vmemmap_optimize_bootmem_folios() runs. Such folios are then
HVOed through the regular remap path with uninitialized tail struct
pages, which trips the PageTail() WARN in vmemmap_remap_pte().

Avoid this by initializing all tail struct pages up front in
gather_bootmem_prealloc_node() for folios that were not pre-HVOed. This
makes the HVO-failure fallback in prep_and_add_bootmem_folios()
unnecessary, as pre-HVOed folios are never passed through the regular
remap path, and all other folios now already have initialized tail
struct pages. A failed optimization either leaves the original vmemmap
in place or restores the tail struct pages from the shared tail page.
Remove the fallback.

For folios that are not HVOed at all, this does not change the amount
of initialization work, only where it is done.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4dac7ed1df57..e971e2362412 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3303,17 +3303,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
 	hugetlb_vmemmap_optimize_bootmem_folios(h, folio_list);
 
 	list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
-		if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
-			/*
-			 * If HVO fails, initialize all tail struct pages
-			 * We do not worry about potential long lock hold
-			 * time as this is early in boot and there should
-			 * be no contention.
-			 */
-			hugetlb_folio_init_tail_vmemmap(folio, h,
-					HUGETLB_VMEMMAP_RESERVE_PAGES,
-					pages_per_huge_page(h));
-		}
 		hugetlb_bootmem_init_migratetype(folio, h);
 		/* Subdivide locks to achieve better parallel performance */
 		spin_lock_irqsave(&hugetlb_lock, flags);
@@ -3337,6 +3326,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
 		struct page *page = virt_to_page(m);
 		struct folio *folio = (void *)page;
 		const unsigned long pfn = folio_pfn(folio);
+		bool pre_hvo;
 
 		h = m->hstate;
 		/*
@@ -3350,11 +3340,23 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
 		VM_BUG_ON(!hstate_is_gigantic(h));
 		WARN_ON(folio_ref_count(folio) != 1);
 
+		pre_hvo = vmemmap_optimizable_order(pfn_to_section_compound_order(pfn));
+
+		/*
+		 * Pre-HVOed folios have their tail struct pages mirrored from
+		 * the shared tail page, so only the first vmemmap page needs
+		 * initializing. Otherwise, the tail struct pages (marked noinit
+		 * in alloc_bootmem()) must all be initialized now: the folio
+		 * may still be HVOed via the regular remap path (e.g. if the
+		 * architecture could not determine HVO support at bootmem
+		 * allocation time), which expects valid tail pages.
+		 */
 		hugetlb_folio_init_vmemmap(folio, h,
-					   HUGETLB_VMEMMAP_RESERVE_PAGES);
+					   pre_hvo ? HUGETLB_VMEMMAP_RESERVE_PAGES :
+						     pages_per_huge_page(h));
 		init_new_hugetlb_folio(folio);
 
-		if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
+		if (pre_hvo)
 			folio_set_hugetlb_vmemmap_optimized(folio);
 		section_set_compound_order_range(pfn, folio_nr_pages(folio), 0);
 
-- 
2.56.0.rc1.315.gc6ed9934b7-goog