* [PATCH v2 01/14] mm/mm_init: factor out memmap_init_page_from_template()
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-16 4:40 ` Lance Yang
2026-11-13 3:34 ` [PATCH v2 02/14] mm/hugetlb: optimize bootmem hugepage tail vmemmap init Muchun Song
` (14 subsequent siblings)
15 siblings, 1 reply; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
From: Li Zhe <lizhe.67@bytedance.com>
ZONE_DEVICE memmap initialization already uses a reusable struct page
template when initializing compound tail pages. The template mostly
contains invariant state, but a few fields still depend on the target PFN
and must be refreshed before copying the template into each destination
struct page.
Factor that operation into memmap_init_page_from_template() so other
boot-time memmap initialization paths can reuse the same helper.
No functional change intended.
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Reviewed-by: Muchun Song (ByteDance) <muchun.song@linux.dev>
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/mm_init.c | 33 +++++++++++++++++++--------------
mm/mm_init.h | 2 ++
2 files changed, 21 insertions(+), 14 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 2367af05570b..3988c19699bc 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -707,6 +707,22 @@ static __meminit void pageblock_migratetype_init_range(unsigned long pfn,
}
}
+void __meminit memmap_init_page_from_template(struct page *page,
+ unsigned long pfn, const struct zone *zone, struct page *template)
+{
+ /*
+ * Refresh PFN-dependent fields in the template before the
+ * non-temporal copy, so the destination page does not need cached
+ * stores after memcpy_nontemporal().
+ */
+ set_page_section_from_pfn(template, pfn);
+#ifdef WANT_PAGE_VIRTUAL
+ if (!is_highmem_idx(zone_idx(zone)))
+ set_page_address(template, __va(pfn << PAGE_SHIFT));
+#endif
+ memcpy_nontemporal(page, template, sizeof(*page));
+}
+
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
{
pg_data_t *pgdat = NODE_DATA(nid);
@@ -1074,17 +1090,6 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
}
}
-static void zone_device_page_init_from_template(struct page *page,
- unsigned long pfn, struct page *template)
-{
- set_page_section_from_pfn(template, pfn);
-#ifdef WANT_PAGE_VIRTUAL
- if (!is_highmem_idx(ZONE_DEVICE))
- set_page_address(template, __va(pfn << PAGE_SHIFT));
-#endif
- memcpy_nontemporal(page, template, sizeof(*page));
-}
-
/*
* With compound page geometry and when struct pages are stored in ram most
* tail pages are reused. Consequently, the amount of unique struct pages to
@@ -1137,8 +1142,8 @@ static void __ref memmap_init_compound(struct page *head,
/* Initialize the remaining tail pages from template. */
for (pfn = head_pfn + 2; pfn < end_pfn; pfn++)
- zone_device_page_init_from_template(pfn_to_page(pfn), pfn,
- &template);
+ memmap_init_page_from_template(pfn_to_page(pfn), pfn,
+ device_zone(nid), &template);
prep_compound_head(head, order);
}
@@ -1193,7 +1198,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
for (; pfn < end_pfn; pfn += pfns_per_compound) {
page = pfn_to_page(pfn);
- zone_device_page_init_from_template(page, pfn, &template);
+ memmap_init_page_from_template(page, pfn, zone, &template);
if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
cond_resched();
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 0034e37dbabd..4478419f0bb9 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -111,5 +111,7 @@ void memblock_free_all(void);
void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid);
+void __meminit memmap_init_page_from_template(struct page *page,
+ unsigned long pfn, const struct zone *zone, struct page *template);
#endif /* __MM_MM_INIT_H */
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 01/14] mm/mm_init: factor out memmap_init_page_from_template()
2026-11-13 3:34 ` [PATCH v2 01/14] mm/mm_init: factor out memmap_init_page_from_template() Muchun Song
@ 2026-11-16 4:40 ` Lance Yang
0 siblings, 0 replies; 22+ messages in thread
From: Lance Yang @ 2026-11-16 4:40 UTC (permalink / raw)
To: songmuchun
Cc: akpm, david, osalvador, rppt, lizhe.67, apopple, balbirs, ljs,
liam, vbabka, surenb, mhocko, muchun.song, jthoughton,
lance.yang, qi.zheng, linux-mm, linux-kernel
On Fri, Nov 13, 2026 at 11:34:29AM +0800, Muchun Song wrote:
>From: Li Zhe <lizhe.67@bytedance.com>
>
>ZONE_DEVICE memmap initialization already uses a reusable struct page
>template when initializing compound tail pages. The template mostly
>contains invariant state, but a few fields still depend on the target PFN
>and must be refreshed before copying the template into each destination
>struct page.
>
>Factor that operation into memmap_init_page_from_template() so other
>boot-time memmap initialization paths can reuse the same helper.
>
>No functional change intended.
>
>Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
>Reviewed-by: Muchun Song (ByteDance) <muchun.song@linux.dev>
>Signed-off-by: Muchun Song <songmuchun@bytedance.com>
>---
Reviewed-by: Lance Yang <lance.yang@linux.dev>
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v2 02/14] mm/hugetlb: optimize bootmem hugepage tail vmemmap init
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
2026-11-13 3:34 ` [PATCH v2 01/14] mm/mm_init: factor out memmap_init_page_from_template() Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-16 10:42 ` Mike Rapoport
2026-11-13 3:34 ` [PATCH v2 03/14] mm/mm_init: verify zones for template-initialized pages Muchun Song
` (13 subsequent siblings)
15 siblings, 1 reply; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
From: Li Zhe <lizhe.67@bytedance.com>
Boot-time HugeTLB allocation has to initialize tail struct pages itself
when HugeTLB vmemmap optimization (HVO) is unavailable or fails. For
gigantic pages, this can cover a large number of tail pages. Most state
installed by __init_single_page() and prep_compound_tail() is identical
for every tail page in the same huge page; only PFN-dependent fields need
refreshing.
Initialize the first tail page normally, then initialize the remaining
tails with hugetlb_init_tail_page(). This copies the already
initialized first tail page with memcpy() and refreshes only the
PFN-dependent fields in the destination page, while avoiding the full
__init_single_page() and prep_compound_tail() sequence for state that is
identical across tail pages in the same huge page.
The fallback path was measured on a v7.4-rc2 base with a temporary
jiffies-based printk modeled after the existing timing in
memmap_init_zone_device(), using 4K base pages and 64 boot-time 1G
hugepages. HVO is unavailable on arm64; on x86-64,
hugetlb_free_vmemmap=off was used to exercise the fallback path.
arch base with this patch reduction
arm64 360.8 ms 256.0 ms 29.05%
x86-64 171.0 ms 165.0 ms 3.51%
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/hugetlb.c | 23 ++++++++++++++++++-----
mm/mm_init.c | 5 ++++-
2 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 88c9aa700533..ed3af959f615 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3266,6 +3266,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
enum zone_type zone = folio_zonenum(folio);
int nid = folio_nid(folio);
struct page *page = folio_page(folio, start_page_number);
+ struct page template;
unsigned long head_pfn = folio_pfn(folio);
unsigned long pfn, end_pfn = head_pfn + end_page_number;
unsigned int order = huge_page_order(h);
@@ -3274,11 +3275,23 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
* As we marked all tail pages with memblock_reserved_mark_noinit(),
* we must initialize them ourselves here.
*/
- for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
- __init_single_page(page, pfn, zone, nid);
- prep_compound_tail(page, &folio->page, order);
- set_page_count(page, 0);
- }
+ if (start_page_number >= end_page_number)
+ return;
+
+ pfn = head_pfn + start_page_number;
+
+ /*
+ * Seed a template from the first real tail page and refresh only
+ * PFN-dependent fields before copying it.
+ */
+ __init_single_page(page, pfn, zone, nid);
+ prep_compound_tail(page, &folio->page, order);
+ set_page_count(page, 0);
+ memcpy(&template, page, sizeof(template));
+
+ for (pfn++; pfn < end_pfn; pfn++)
+ memmap_init_page_from_template(pfn_to_page(pfn), pfn,
+ folio_zone(folio), &template);
}
static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 3988c19699bc..639880afe0ee 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -720,7 +720,10 @@ void __meminit memmap_init_page_from_template(struct page *page,
if (!is_highmem_idx(zone_idx(zone)))
set_page_address(template, __va(pfn << PAGE_SHIFT));
#endif
- memcpy_nontemporal(page, template, sizeof(*page));
+ if(zone_is_zone_device(zone))
+ memcpy_nontemporal(page, template, sizeof(*page));
+ else
+ memcpy(page, template, sizeof(*page));
}
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 02/14] mm/hugetlb: optimize bootmem hugepage tail vmemmap init
2026-11-13 3:34 ` [PATCH v2 02/14] mm/hugetlb: optimize bootmem hugepage tail vmemmap init Muchun Song
@ 2026-11-16 10:42 ` Mike Rapoport
0 siblings, 0 replies; 22+ messages in thread
From: Mike Rapoport @ 2026-11-16 10:42 UTC (permalink / raw)
To: Muchun Song
Cc: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel
Hi,
> Boot-time HugeTLB allocation has to initialize tail struct pages itself
> when HugeTLB vmemmap optimization (HVO) is unavailable or fails. For
> gigantic pages, this can cover a large number of tail pages. Most state
> installed by __init_single_page() and prep_compound_tail() is identical
> for every tail page in the same huge page; only PFN-dependent fields need
> refreshing.
>
> Initialize the first tail page normally, then initialize the remaining
> tails with hugetlb_init_tail_page(). This copies the already
> initialized first tail page with memcpy() and refreshes only the
> PFN-dependent fields in the destination page, while avoiding the full
> __init_single_page() and prep_compound_tail() sequence for state that is
> identical across tail pages in the same huge page.
>
> The fallback path was measured on a v7.4-rc2 base with a temporary
> jiffies-based printk modeled after the existing timing in
> memmap_init_zone_device(), using 4K base pages and 64 boot-time 1G
> hugepages. HVO is unavailable on arm64; on x86-64,
> hugetlb_free_vmemmap=off was used to exercise the fallback path.
>
> arch base with this patch reduction
> arm64 360.8 ms 256.0 ms 29.05%
> x86-64 171.0 ms 165.0 ms 3.51%
>
> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> ---
> mm/hugetlb.c | 23 ++++++++++++++++++-----
> mm/mm_init.c | 5 ++++-
> 2 files changed, 22 insertions(+), 6 deletions(-)
>
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index cb2de24affdc4..d70ed37d2553b 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -3239,6 +3239,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
> enum zone_type zone = folio_zonenum(folio);
> int nid = folio_nid(folio);
> struct page *page = folio_page(folio, start_page_number);
> + struct page template;
> unsigned long head_pfn = folio_pfn(folio);
> unsigned long pfn, end_pfn = head_pfn + end_page_number;
> unsigned int order = huge_page_order(h);
> @@ -3247,11 +3248,23 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
> * As we marked all tail pages with memblock_reserved_mark_noinit(),
> * we must initialize them ourselves here.
> */
> - for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
> - __init_single_page(page, pfn, zone, nid);
> - prep_compound_tail(page, &folio->page, order);
> - set_page_count(page, 0);
> - }
> + if (start_page_number >= end_page_number)
> + return;
> +
> + pfn = head_pfn + start_page_number;
> +
> + /*
> + * Seed a template from the first real tail page and refresh only
> + * PFN-dependent fields before copying it.
> + */
> + __init_single_page(page, pfn, zone, nid);
> + prep_compound_tail(page, &folio->page, order);
> + set_page_count(page, 0);
> + memcpy(&template, page, sizeof(template));
> +
> + for (pfn++; pfn < end_pfn; pfn++)
> + memmap_init_page_from_template(pfn_to_page(pfn), pfn,
> + folio_zone(folio), &template);
> }
>
> static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
> diff --git a/mm/mm_init.c b/mm/mm_init.c
> index f16a047796c88..934512bceda39 100644
> --- a/mm/mm_init.c
> +++ b/mm/mm_init.c
> @@ -719,7 +719,10 @@ void __meminit memmap_init_page_from_template(struct page *page,
> if (!is_highmem_idx(zone_idx(zone)))
> set_page_address(template, __va(pfn << PAGE_SHIFT));
> #endif
> - memcpy_nontemporal(page, template, sizeof(*page));
> + if(zone_is_zone_device(zone))
Nit: missing space after "if".
The comment at the beginning of memmap_init_page_from_template() still
talks about "the non-temporal copy", which is now true only for
ZONE_DEVICE.
--
Sincerely yours,
Mike.
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v2 03/14] mm/mm_init: verify zones for template-initialized pages
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
2026-11-13 3:34 ` [PATCH v2 01/14] mm/mm_init: factor out memmap_init_page_from_template() Muchun Song
2026-11-13 3:34 ` [PATCH v2 02/14] mm/hugetlb: optimize bootmem hugepage tail vmemmap init Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-16 8:41 ` Lance Yang
2026-11-13 3:34 ` [PATCH v2 04/14] mm/mm_init: factor out common struct page initialization Muchun Song
` (12 subsequent siblings)
15 siblings, 1 reply; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Template-based ZONE_DEVICE initialization bypasses
__init_single_page(), which means copied struct pages do not run the
vmemmap zone consistency check.
Verify each destination after copying the template so the fast path gets
the same validation as regular page initialization.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
---
v2:
- Apply the zone verification in memmap_init_page_from_template().
- Collect Reviewed-by from Mike Rapoport.
---
mm/mm_init.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 639880afe0ee..5ce5a325021c 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -724,6 +724,7 @@ void __meminit memmap_init_page_from_template(struct page *page,
memcpy_nontemporal(page, template, sizeof(*page));
else
memcpy(page, template, sizeof(*page));
+ vmemmap_optimization_verify_zone(page);
}
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 03/14] mm/mm_init: verify zones for template-initialized pages
2026-11-13 3:34 ` [PATCH v2 03/14] mm/mm_init: verify zones for template-initialized pages Muchun Song
@ 2026-11-16 8:41 ` Lance Yang
0 siblings, 0 replies; 22+ messages in thread
From: Lance Yang @ 2026-11-16 8:41 UTC (permalink / raw)
To: songmuchun
Cc: akpm, david, osalvador, rppt, lizhe.67, apopple, balbirs, ljs,
liam, vbabka, surenb, mhocko, muchun.song, jthoughton,
lance.yang, qi.zheng, linux-mm, linux-kernel
On Fri, Nov 13, 2026 at 11:34:31AM +0800, Muchun Song wrote:
>Template-based ZONE_DEVICE initialization bypasses
>__init_single_page(), which means copied struct pages do not run the
>vmemmap zone consistency check.
>
>Verify each destination after copying the template so the fast path gets
>the same validation as regular page initialization.
>
>Signed-off-by: Muchun Song <songmuchun@bytedance.com>
>Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
>---
Reviewed-by: Lance Yang <lance.yang@linux.dev>
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v2 04/14] mm/mm_init: factor out common struct page initialization
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (2 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 03/14] mm/mm_init: verify zones for template-initialized pages Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-16 10:42 ` Mike Rapoport
2026-11-13 3:34 ` [PATCH v2 05/14] mm/mm_init: simplify zone device " Muchun Song
` (11 subsequent siblings)
15 siblings, 1 reply; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Several struct page fields have the same initial values for every page in
a zone. __init_single_page() initializes this state as part of its
per-page setup, while the shared vmemmap tail path open-codes only a
subset.
Factor the PFN-independent state into memmap_init_page_common() and use
it for both regular and shared tail struct pages. Keep section-, PFN-
and instance-specific initialization in __init_single_page().
This also initializes the last cpupid and KASAN tag for shared tail
struct pages consistently with regular struct pages.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v2:
- Remove __meminit from memmap_init_page_common() so HVO shared tail
pages can call it after init text is freed (reported by Sashiko).
---
include/linux/mm.h | 8 --------
mm/mm_init.c | 22 +++++++++++++++++-----
mm/mm_init.h | 1 +
mm/sparse-vmemmap.c | 7 +------
4 files changed, 19 insertions(+), 19 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 77b5d450fe45..2e182498898e 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3022,14 +3022,6 @@ static inline void set_page_node(struct page *page, unsigned long node)
page->flags.f |= (node & NODES_MASK) << NODES_PGSHIFT;
}
-static inline void set_page_links(struct page *page, enum zone_type zone,
- unsigned long node, unsigned long pfn)
-{
- set_page_zone(page, zone);
- set_page_node(page, node);
- set_page_section_from_pfn(page, pfn);
-}
-
/**
* folio_nr_pages - The number of pages in the folio.
* @folio: The folio.
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 5ce5a325021c..8d0b8faf3938 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -603,15 +603,27 @@ static inline void vmemmap_optimization_verify_zone(struct page *page)
page_zone_id(page));
}
-void __meminit __init_single_page(struct page *page, unsigned long pfn,
- unsigned long zone, int nid)
+/* Initialize PFN-independent fields common to pages in the same zone. */
+void memmap_init_page_common(struct page *page, enum zone_type zone, int nid)
{
mm_zero_struct_page(page);
- set_page_links(page, zone, nid, pfn);
- init_page_count(page);
- atomic_set(&page->_mapcount, -1);
+ set_page_zone(page, zone);
+ set_page_node(page, nid);
page_cpupid_reset_last(page);
page_kasan_tag_reset(page);
+ atomic_set(&page->_mapcount, -1);
+#ifdef CONFIG_ZONE_DEVICE
+ if (zone == ZONE_DEVICE)
+ __SetPageReserved(page);
+#endif
+}
+
+void __meminit __init_single_page(struct page *page, unsigned long pfn,
+ unsigned long zone, int nid)
+{
+ memmap_init_page_common(page, zone, nid);
+ set_page_section_from_pfn(page, pfn);
+ init_page_count(page);
INIT_LIST_HEAD(&page->lru);
#ifdef WANT_PAGE_VIRTUAL
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 4478419f0bb9..9de9db31320d 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -109,6 +109,7 @@ extern bool mirrored_kernelcore;
bool memblock_has_mirror(void);
void memblock_free_all(void);
+void memmap_init_page_common(struct page *page, enum zone_type zone, int nid);
void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid);
void __meminit memmap_init_page_from_template(struct page *page,
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index 9b80dd7326f7..fd869617423a 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -212,12 +212,7 @@ struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zon
for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) {
page = (struct page *)addr + i;
- mm_zero_struct_page(page);
- atomic_set(&page->_mapcount, -1);
- set_page_node(page, zone_to_nid(zone));
- set_page_zone(page, zone_idx(zone));
- if (zone_is_zone_device(zone))
- __SetPageReserved(page);
+ memmap_init_page_common(page, zone_idx(zone), zone_to_nid(zone));
prep_compound_tail(page, NULL, order);
}
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 04/14] mm/mm_init: factor out common struct page initialization
2026-11-13 3:34 ` [PATCH v2 04/14] mm/mm_init: factor out common struct page initialization Muchun Song
@ 2026-11-16 10:42 ` Mike Rapoport
0 siblings, 0 replies; 22+ messages in thread
From: Mike Rapoport @ 2026-11-16 10:42 UTC (permalink / raw)
To: Muchun Song
Cc: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel
Hi,
> Several struct page fields have the same initial values for every page in
> a zone. __init_single_page() initializes this state as part of its
> per-page setup, while the shared vmemmap tail path open-codes only a
> subset.
>
> Factor the PFN-independent state into memmap_init_page_common() and use
> it for both regular and shared tail struct pages. Keep section-, PFN-
> and instance-specific initialization in __init_single_page().
>
> This also initializes the last cpupid and KASAN tag for shared tail
> struct pages consistently with regular struct pages.
>
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> ---
> v2:
> - Remove __meminit from memmap_init_page_common() so HVO shared tail
> pages can call it after init text is freed (reported by Sashiko).
> ---
> include/linux/mm.h | 8 --------
> mm/mm_init.c | 22 +++++++++++++++++-----
> mm/mm_init.h | 1 +
> mm/sparse-vmemmap.c | 7 +------
> 4 files changed, 19 insertions(+), 19 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 77b5d450fe45b..2e182498898ec 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -3022,14 +3022,6 @@ static inline void set_page_node(struct page *page, unsigned long node)
> page->flags.f |= (node & NODES_MASK) << NODES_PGSHIFT;
> }
>
> -static inline void set_page_links(struct page *page, enum zone_type zone,
> - unsigned long node, unsigned long pfn)
> -{
> - set_page_zone(page, zone);
> - set_page_node(page, node);
> - set_page_section_from_pfn(page, pfn);
> -}
> -
> /**
> * folio_nr_pages - The number of pages in the folio.
> * @folio: The folio.
> diff --git a/mm/mm_init.c b/mm/mm_init.c
> index af3b81733193b..d0a11989bd249 100644
> --- a/mm/mm_init.c
> +++ b/mm/mm_init.c
> @@ -602,15 +602,27 @@ static inline void vmemmap_optimization_verify_zone(struct page *page)
> page_zone_id(page));
> }
>
> -void __meminit __init_single_page(struct page *page, unsigned long pfn,
> - unsigned long zone, int nid)
> +/* Initialize PFN-independent fields common to pages in the same zone. */
> +void memmap_init_page_common(struct page *page, enum zone_type zone, int nid)
I don't think _common adds much here, memmap_init_page() would do.
> {
> mm_zero_struct_page(page);
> - set_page_links(page, zone, nid, pfn);
> - init_page_count(page);
> - atomic_set(&page->_mapcount, -1);
> + set_page_zone(page, zone);
> + set_page_node(page, nid);
> page_cpupid_reset_last(page);
> page_kasan_tag_reset(page);
> + atomic_set(&page->_mapcount, -1);
> +#ifdef CONFIG_ZONE_DEVICE
> + if (zone == ZONE_DEVICE)
> + __SetPageReserved(page);
> +#endif
Can't we keep this in ZONE_DEVICE part?
If we can't, __init_single_page() sets PG_reserved for ZONE_DEVICE pages
and the
if (zone == ZONE_DEVICE)
__SetPageReserved(page);
in the MEMINIT_HOTPLUG part of memmap_init_range() becomes redundant.
--
Sincerely yours,
Mike.
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v2 05/14] mm/mm_init: simplify zone device page initialization
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (3 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 04/14] mm/mm_init: factor out common struct page initialization Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 06/14] mm/mm_init: factor out frozen compound " Muchun Song
` (10 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
memmap_init_zone_device() currently seeds its reusable head and tail
templates by fully initializing the first real struct page and copying
it back into a stack object. This keeps the first page outside the
common loops and leaves __init_zone_device_page() only for template
construction.
Build both templates directly from the PFN-independent state. Add the
head-specific pgmap and refcount state to the head template and the
compound metadata to the tail template, then initialize every page
through memmap_init_page_from_template().
This removes the special handling for the first head and tail pages
without changing their final state.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v2:
- Rebase on memmap_init_page_from_template().
---
mm/mm_init.c | 94 ++++++++--------------------------------------------
1 file changed, 14 insertions(+), 80 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 8d0b8faf3938..69bbaf75f935 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1064,48 +1064,6 @@ static void __init memmap_init(void)
}
#ifdef CONFIG_ZONE_DEVICE
-static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
- unsigned long zone_idx, int nid,
- struct dev_pagemap *pgmap)
-{
-
- __init_single_page(page, pfn, zone_idx, nid);
-
- /*
- * Mark page reserved as it will need to wait for onlining
- * phase for it to be fully associated with a zone.
- *
- * We can use the non-atomic __set_bit operation for setting
- * the flag as we are still initializing the pages.
- */
- __SetPageReserved(page);
-
- /*
- * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer
- * and zone_device_data. It is a bug if a ZONE_DEVICE page is
- * ever freed or placed on a driver-private list.
- */
- page_folio(page)->pgmap = pgmap;
- page->zone_device_data = NULL;
-
- /*
- * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free
- * path. The remaining ZONE_DEVICE types start from 0 here and raise
- * the count again when the allocator or driver hands the page out.
- */
- switch (pgmap->type) {
- case MEMORY_DEVICE_FS_DAX:
- case MEMORY_DEVICE_PRIVATE:
- case MEMORY_DEVICE_COHERENT:
- case MEMORY_DEVICE_PCI_P2PDMA:
- set_page_count(page, 0);
- break;
-
- case MEMORY_DEVICE_GENERIC:
- break;
- }
-}
-
/*
* With compound page geometry and when struct pages are stored in ram most
* tail pages are reused. Consequently, the amount of unique struct pages to
@@ -1134,30 +1092,13 @@ static void __ref memmap_init_compound(struct page *head,
unsigned long pfn, end_pfn = head_pfn + nr_pages;
unsigned int order = pgmap->vmemmap_shift;
struct page template;
- struct page *page;
- /*
- * We have to initialize the pages, including setting up page links.
- * prep_compound_page() does not take care of that, so instead we
- * open-code prep_compound_page() so we can take care of initializing
- * the pages in the same go.
- */
- __SetPageHead(head);
-
- /*
- * All tails of the same compound page share the state established by
- * prep_compound_tail(). Reuse one tail template for the whole range and
- * refresh only the PFN-dependent fields in that template before each copy.
- */
- pfn = head_pfn + 1;
- page = pfn_to_page(pfn);
- __init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
- prep_compound_tail(page, head, order);
- set_page_count(page, 0);
- memcpy(&template, page, sizeof(*page));
+ memmap_init_page_common(&template, zone_idx, nid);
+ prep_compound_tail(&template, head, order);
- /* Initialize the remaining tail pages from template. */
- for (pfn = head_pfn + 2; pfn < end_pfn; pfn++)
+ __SetPageHead(head);
+ /* Initialize the tail pages from template. */
+ for (pfn = head_pfn + 1; pfn < end_pfn; pfn++)
memmap_init_page_from_template(pfn_to_page(pfn), pfn,
device_zone(nid), &template);
prep_compound_head(head, order);
@@ -1191,27 +1132,20 @@ void __ref memmap_init_zone_device(struct zone *zone,
nr_pages = end_pfn - start_pfn;
}
- if (!nr_pages)
- return;
-
VM_WARN_ON(!IS_ALIGNED(start_pfn | nr_pages, pfns_per_compound));
+ memmap_init_page_common(&template, zone_idx, nid);
+ page_folio(&template)->pgmap = pgmap;
/*
- * Seed the reusable head-page template from the first real struct
- * page. The normal page-init and refcount helpers must operate on
- * a real memmap entry rather than a stack object.
+ * MEMORY_DEVICE_GENERIC pages regain a refcount of one in the free
+ * path. The remaining ZONE_DEVICE types start from zero here and raise
+ * the count again when the allocator or driver hands the page out.
*/
- pfn = start_pfn;
- page = pfn_to_page(pfn);
- __init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
- memcpy(&template, page, sizeof(*page));
- if (pfns_per_compound != 1)
- memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
- compound_nr_pages(pfn, pgmap));
- pfn += pfns_per_compound;
+ if (pgmap->type == MEMORY_DEVICE_GENERIC)
+ atomic_set(&template._refcount, 1);
- /* Initialize the remaining head pages from template. */
- for (; pfn < end_pfn; pfn += pfns_per_compound) {
+ /* Initialize the head pages from template. */
+ for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {
page = pfn_to_page(pfn);
memmap_init_page_from_template(page, pfn, zone, &template);
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 06/14] mm/mm_init: factor out frozen compound page initialization
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (4 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 05/14] mm/mm_init: simplify zone device " Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-16 10:42 ` Mike Rapoport
2026-11-13 3:34 ` [PATCH v2 07/14] mm/hugetlb: drop boot-time HVO handling for gigantic folios Muchun Song
` (9 subsequent siblings)
15 siblings, 1 reply; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
ZONE_DEVICE initializes compound struct pages while their refcounts remain
frozen. The setup is open-coded in memmap_init_compound(), including the
per-page initialization and compound tail preparation.
Factor the common setup into memmap_init_compound_page_frozen(). Keep the
pgmap association and MEMORY_DEVICE_GENERIC refcount handling in the
ZONE_DEVICE-specific path.
This provides an MM-internal helper for other early memmap users to
share.
No functional change is intended.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v2:
- Rebase on memmap_init_page_from_template().
- Rename memmap_init_zone_device_head_page to memmap_init_zone_device_head_fields.
---
mm/mm_init.c | 114 +++++++++++++++++++++++++--------------------------
mm/mm_init.h | 2 +
2 files changed, 59 insertions(+), 57 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 69bbaf75f935..ea9ff690d4e0 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -739,6 +739,28 @@ void __meminit memmap_init_page_from_template(struct page *page,
vmemmap_optimization_verify_zone(page);
}
+void __meminit memmap_init_compound_page_frozen(struct page *head,
+ unsigned long pfn, const struct zone *zone, unsigned int order)
+{
+ unsigned long nr_pages = 1UL << order;
+ struct page template;
+
+ memmap_init_page_common(&template, zone_idx(zone), zone_to_nid(zone));
+ prep_compound_tail(&template, head, order);
+
+ /* Initialize the head page. */
+ __init_single_page(head, pfn, zone_idx(zone), zone_to_nid(zone));
+ set_page_count(head, 0);
+ __SetPageHead(head);
+
+ if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
+ nr_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
+ /* Initialize the tail pages from PFN-independent template page. */
+ for (unsigned long i = 1; i < nr_pages; i++)
+ memmap_init_page_from_template(head + i, pfn + i, zone, &template);
+ prep_compound_head(head, order);
+}
+
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
{
pg_data_t *pgdat = NODE_DATA(nid);
@@ -1064,44 +1086,39 @@ static void __init memmap_init(void)
}
#ifdef CONFIG_ZONE_DEVICE
-/*
- * With compound page geometry and when struct pages are stored in ram most
- * tail pages are reused. Consequently, the amount of unique struct pages to
- * initialize is a lot smaller that the total amount of struct pages being
- * mapped. This is a paired / mild layering violation with explicit knowledge
- * of how the sparse_vmemmap internals handle compound pages in the lack
- * of an altmap.
- */
-static inline unsigned long compound_nr_pages(unsigned long pfn,
- struct dev_pagemap *pgmap)
+static void memmap_init_zone_device_head_fields(struct page *head,
+ struct dev_pagemap *pgmap)
{
- const struct mem_section *ms = __pfn_to_section(pfn);
-
- if (!section_vmemmap_optimizable(ms))
- return pgmap_vmemmap_nr(pgmap);
-
- return VMEMMAP_OPTIMIZATION_PAGES * (PAGE_SIZE / sizeof(struct page));
+ /*
+ * MEMORY_DEVICE_GENERIC pages start with a refcount of one; other
+ * ZONE_DEVICE pages raise it when handed out. Use atomic_set() because
+ * @head may be a stack template, while init_page_count() expects a
+ * memmap-backed page.
+ */
+ if (pgmap->type == MEMORY_DEVICE_GENERIC)
+ atomic_set(&head->_refcount, 1);
+ /*
+ * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer and
+ * zone_device_data. It is a bug if a ZONE_DEVICE page is ever freed
+ * or placed on a driver-private list.
+ */
+ ((struct folio *)head)->pgmap = pgmap;
+ head->zone_device_data = NULL;
}
-static void __ref memmap_init_compound(struct page *head,
- unsigned long head_pfn,
- unsigned long zone_idx, int nid,
- struct dev_pagemap *pgmap,
- unsigned long nr_pages)
+static void memmap_init_zone_device_basepages(unsigned long start, unsigned long end,
+ const struct zone *zone, struct dev_pagemap *pgmap)
{
- unsigned long pfn, end_pfn = head_pfn + nr_pages;
- unsigned int order = pgmap->vmemmap_shift;
struct page template;
- memmap_init_page_common(&template, zone_idx, nid);
- prep_compound_tail(&template, head, order);
+ memmap_init_page_common(&template, zone_idx(zone), zone_to_nid(zone));
+ memmap_init_zone_device_head_fields(&template, pgmap);
- __SetPageHead(head);
- /* Initialize the tail pages from template. */
- for (pfn = head_pfn + 1; pfn < end_pfn; pfn++)
- memmap_init_page_from_template(pfn_to_page(pfn), pfn,
- device_zone(nid), &template);
- prep_compound_head(head, order);
+ for (unsigned long pfn = start; pfn < end; pfn++) {
+ memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone, &template);
+ if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
+ cond_resched();
+ }
}
void __ref memmap_init_zone_device(struct zone *zone,
@@ -1110,16 +1127,11 @@ void __ref memmap_init_zone_device(struct zone *zone,
struct dev_pagemap *pgmap)
{
unsigned long pfn, end_pfn = start_pfn + nr_pages;
- struct pglist_data *pgdat = zone->zone_pgdat;
struct vmem_altmap *altmap = pgmap_altmap(pgmap);
unsigned int pfns_per_compound = pgmap_vmemmap_nr(pgmap);
- unsigned long zone_idx = zone_idx(zone);
unsigned long start = jiffies;
- int nid = pgdat->node_id;
- struct page template;
- struct page *page;
- if (WARN_ON_ONCE(!pgmap || zone_idx != ZONE_DEVICE))
+ if (WARN_ON_ONCE(!pgmap || !zone_is_zone_device(zone)))
return;
/*
@@ -1134,32 +1146,20 @@ void __ref memmap_init_zone_device(struct zone *zone,
VM_WARN_ON(!IS_ALIGNED(start_pfn | nr_pages, pfns_per_compound));
- memmap_init_page_common(&template, zone_idx, nid);
- page_folio(&template)->pgmap = pgmap;
- /*
- * MEMORY_DEVICE_GENERIC pages regain a refcount of one in the free
- * path. The remaining ZONE_DEVICE types start from zero here and raise
- * the count again when the allocator or driver hands the page out.
- */
- if (pgmap->type == MEMORY_DEVICE_GENERIC)
- atomic_set(&template._refcount, 1);
+ if (pfns_per_compound == 1) {
+ memmap_init_zone_device_basepages(start_pfn, end_pfn, zone, pgmap);
+ goto out;
+ }
- /* Initialize the head pages from template. */
for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {
- page = pfn_to_page(pfn);
-
- memmap_init_page_from_template(page, pfn, zone, &template);
+ struct page *head = pfn_to_page(pfn);
+ memmap_init_compound_page_frozen(head, pfn, zone, pgmap->vmemmap_shift);
+ memmap_init_zone_device_head_fields(head, pgmap);
if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
cond_resched();
-
- if (pfns_per_compound == 1)
- continue;
-
- memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
- compound_nr_pages(pfn, pgmap));
}
-
+out:
pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE,
/* isolate */ false, /* atomic */ false);
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 9de9db31320d..721221d2e4c0 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -114,5 +114,7 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid);
void __meminit memmap_init_page_from_template(struct page *page,
unsigned long pfn, const struct zone *zone, struct page *template);
+void __meminit memmap_init_compound_page_frozen(struct page *head,
+ unsigned long pfn, const struct zone *zone, unsigned int order);
#endif /* __MM_MM_INIT_H */
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 06/14] mm/mm_init: factor out frozen compound page initialization
2026-11-13 3:34 ` [PATCH v2 06/14] mm/mm_init: factor out frozen compound " Muchun Song
@ 2026-11-16 10:42 ` Mike Rapoport
0 siblings, 0 replies; 22+ messages in thread
From: Mike Rapoport @ 2026-11-16 10:42 UTC (permalink / raw)
To: Muchun Song
Cc: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel
Hi,
> ZONE_DEVICE initializes compound struct pages while their refcounts remain
> frozen. The setup is open-coded in memmap_init_compound(), including the
> per-page initialization and compound tail preparation.
>
> Factor the common setup into memmap_init_compound_page_frozen(). Keep the
> pgmap association and MEMORY_DEVICE_GENERIC refcount handling in the
> ZONE_DEVICE-specific path.
>
> This provides an MM-internal helper for other early memmap users to
> share.
>
> No functional change is intended.
>
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> ---
> v2:
> - Rebase on memmap_init_page_from_template().
> - Rename memmap_init_zone_device_head_page to memmap_init_zone_device_head_fields.
> ---
> mm/mm_init.c | 114 +++++++++++++++++++++++++--------------------------
> mm/mm_init.h | 2 +
> 2 files changed, 59 insertions(+), 57 deletions(-)
>
> diff --git a/mm/mm_init.c b/mm/mm_init.c
> index bf943ae72a3e5..f1e539aec3d91 100644
> --- a/mm/mm_init.c
> +++ b/mm/mm_init.c
> @@ -738,6 +738,28 @@ void __meminit memmap_init_page_from_template(struct page *page,
> vmemmap_optimization_verify_zone(page);
> }
>
> +void __meminit memmap_init_compound_page_frozen(struct page *head,
> + unsigned long pfn, const struct zone *zone, unsigned int order)
Do we really need the _frozen suffix here?
> +{
> + unsigned long nr_pages = 1UL << order;
> + struct page template;
> +
> + memmap_init_page_common(&template, zone_idx(zone), zone_to_nid(zone));
> + prep_compound_tail(&template, head, order);
> +
> + /* Initialize the head page. */
> + __init_single_page(head, pfn, zone_idx(zone), zone_to_nid(zone));
> + set_page_count(head, 0);
> + __SetPageHead(head);
> +
> + if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
> + nr_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
> + /* Initialize the tail pages from PFN-independent template page. */
> + for (unsigned long i = 1; i < nr_pages; i++)
> + memmap_init_page_from_template(head + i, pfn + i, zone, &template);
> + prep_compound_head(head, order);
> +}
> +
> struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
> {
> pg_data_t *pgdat = NODE_DATA(nid);
> @@ -1063,44 +1085,39 @@ static void __init memmap_init(void)
> }
>
> #ifdef CONFIG_ZONE_DEVICE
> -/*
> - * With compound page geometry and when struct pages are stored in ram most
> - * tail pages are reused. Consequently, the amount of unique struct pages to
> - * initialize is a lot smaller that the total amount of struct pages being
> - * mapped. This is a paired / mild layering violation with explicit knowledge
> - * of how the sparse_vmemmap internals handle compound pages in the lack
> - * of an altmap.
> - */
> -static inline unsigned long compound_nr_pages(unsigned long pfn,
> - struct dev_pagemap *pgmap)
> +static void memmap_init_zone_device_head_fields(struct page *head,
> + struct dev_pagemap *pgmap)
I can't say I like this name. Essentailly we initialize ZONE_DEVICE
folio here, so maybe it can be
memmap_init_zone_device_folio(struct folio *folio)
> {
> - const struct mem_section *ms = __pfn_to_section(pfn);
> -
> - if (!section_vmemmap_optimizable(ms))
> - return pgmap_vmemmap_nr(pgmap);
> -
> - return VMEMMAP_OPTIMIZATION_PAGES * (PAGE_SIZE / sizeof(struct page));
> + /*
> + * MEMORY_DEVICE_GENERIC pages start with a refcount of one; other
> + * ZONE_DEVICE pages raise it when handed out. Use atomic_set() because
> + * @head may be a stack template, while init_page_count() expects a
> + * memmap-backed page.
> + */
> + if (pgmap->type == MEMORY_DEVICE_GENERIC)
> + atomic_set(&head->_refcount, 1);
> + /*
> + * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer and
> + * zone_device_data. It is a bug if a ZONE_DEVICE page is ever freed
> + * or placed on a driver-private list.
> + */
> + ((struct folio *)head)->pgmap = pgmap;
... and then we won't need this cast and the caller will just pass
page_folio() here.
> + head->zone_device_data = NULL;
> }
>
> -static void __ref memmap_init_compound(struct page *head,
> - unsigned long head_pfn,
> - unsigned long zone_idx, int nid,
> - struct dev_pagemap *pgmap,
> - unsigned long nr_pages)
> +static void memmap_init_zone_device_basepages(unsigned long start, unsigned long end,
> + const struct zone *zone, struct dev_pagemap *pgmap)
_basepages sounds odd for me, maybe _pages will do?
--
Sincerely yours,
Mike.
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v2 07/14] mm/hugetlb: drop boot-time HVO handling for gigantic folios
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (5 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 06/14] mm/mm_init: factor out frozen compound " Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 08/14] mm/hugetlb: initialize all struct pages for bootmem hugepages Muchun Song
` (8 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
HugeTLB HVO is currently supported on x86-64, riscv64, and LoongArch.
On x86-64 and riscv64, gigantic HugeTLB pages are larger than the
section size, so the existing section-based vmemmap optimization
infrastructure is already sufficient to cover the whole folio. On
LoongArch, HugeTLB HVO is supported without gigantic HugeTLB pages.
Therefore, boot-time HugeTLB HVO folios can rely on the section-based
vmemmap optimization infrastructure directly, without the extra bulk
optimization and fallback handling.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/hugetlb.c | 25 +++++++------------------
mm/hugetlb_vmemmap.c | 17 +----------------
2 files changed, 8 insertions(+), 34 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index ed3af959f615..3a307bc6db17 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3345,21 +3345,7 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
unsigned long flags;
struct folio *folio, *tmp_f;
- /* Send list for bulk vmemmap optimization processing */
- hugetlb_vmemmap_optimize_folios(h, folio_list);
-
list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
- if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
- /*
- * If HVO fails, initialize all tail struct pages
- * We do not worry about potential long lock hold
- * time as this is early in boot and there should
- * be no contention.
- */
- hugetlb_folio_init_tail_vmemmap(folio, h,
- VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES,
- pages_per_huge_page(h));
- }
hugetlb_bootmem_init_migratetype(folio, h);
/* Subdivide locks to achieve better parallel performance */
spin_lock_irqsave(&hugetlb_lock, flags);
@@ -3383,6 +3369,8 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
struct page *page = virt_to_page(m);
struct folio *folio = (void *)page;
const unsigned long pfn = folio_pfn(folio);
+ const unsigned long nr_pages = pages_per_huge_page(m->hstate);
+ unsigned long nr_struct_pages = nr_pages;
h = m->hstate;
/*
@@ -3396,13 +3384,14 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
VM_BUG_ON(!hstate_is_gigantic(h));
WARN_ON(folio_ref_count(folio) != 1);
- hugetlb_folio_init_vmemmap(folio, h,
- VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES);
+ if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
+ nr_struct_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
+ hugetlb_folio_init_vmemmap(folio, h, nr_struct_pages);
init_new_hugetlb_folio(folio);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
folio_set_hugetlb_vmemmap_optimized(folio);
- section_set_compound_order_range(pfn, folio_nr_pages(folio), 0);
+ section_set_compound_order_range(pfn, nr_pages, 0);
list_add(&folio->lru, &folio_list);
@@ -3417,7 +3406,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
if (hugetlb_early_cma(h))
folio_set_hugetlb_cma(folio);
else
- adjust_managed_page_count(page, pages_per_huge_page(h));
+ adjust_managed_page_count(page, nr_pages);
cond_resched();
}
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index ba8e11e399b2..d73ceeef392e 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -552,7 +552,6 @@ static int hugetlb_vmemmap_pteify_folio(const struct hstate *h, struct folio *fo
void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
{
struct folio *folio;
- unsigned long nr_to_optimize = 0;
LIST_HEAD(vmemmap_pages);
unsigned long flags = VMEMMAP_DEFER_TLB_FLUSH;
@@ -560,18 +559,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
return;
list_for_each_entry(folio, folio_list, lru) {
- int ret;
-
- /*
- * Bootmem gigantic folios may already be marked optimized when
- * their vmemmap layout was prepared earlier, so skip them here.
- */
- if (folio_test_hugetlb_vmemmap_optimized(folio))
- continue;
-
- nr_to_optimize++;
-
- ret = hugetlb_vmemmap_pteify_folio(h, folio, flags);
+ int ret = hugetlb_vmemmap_pteify_folio(h, folio, flags);
/*
* Splitting the PMD requires allocating a page, thus let's fail
@@ -583,9 +571,6 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
break;
}
- if (!nr_to_optimize)
- return;
-
flush_tlb_all();
list_for_each_entry(folio, folio_list, lru) {
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 08/14] mm/hugetlb: initialize all struct pages for bootmem hugepages
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (6 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 07/14] mm/hugetlb: drop boot-time HVO handling for gigantic folios Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 09/14] mm/hugetlb: use common compound page initialization helper Muchun Song
` (7 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Boot-time gigantic hugepages currently leave the head struct page to
the generic memmap initialization path while the HugeTLB code
initializes the remaining struct pages itself.
Mark the full hugepage noinit and initialize the head struct page in
hugetlb_folio_init_vmemmap() as well, so the whole compound-page setup
is handled in one place.
This can also reduce memblock metadata overhead when many boot-time
HugeTLB pages are reserved, because physically contiguous hugepages
can be covered by fewer noinit regions.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/hugetlb.c | 27 ++++++++++-----------------
1 file changed, 10 insertions(+), 17 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 3a307bc6db17..2716a83a7386 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3243,15 +3243,7 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid)
list_add_tail(&m->list, &huge_boot_pages[nid]);
m->flags |= HUGE_BOOTMEM_ZONES_VALID;
hugetlb_vmemmap_optimize_bootmem_page(pfn, huge_page_order(h));
- /*
- * Only initialize the head struct page in memmap_init_reserved_pages,
- * rest of the struct pages will be initialized by the HugeTLB
- * subsystem itself.
- * The head struct page is used to get folio information by the HugeTLB
- * subsystem like zone id and node id.
- */
- memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE),
- huge_page_size(h) - PAGE_SIZE);
+ memblock_reserved_mark_noinit(__pa(m), huge_page_size(h));
}
return true;
@@ -3295,22 +3287,23 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
}
static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
+ unsigned long pfn,
struct hstate *h,
+ enum zone_type zone, int nid,
unsigned long nr_pages)
{
- int ret;
+ struct page *head = &folio->page;
/*
* This is an open-coded prep_compound_page() whereby we avoid
* walking pages twice by initializing/preparing+freezing them in the
* same go.
*/
- __folio_clear_reserved(folio);
+ __init_single_page(head, pfn, zone, nid);
__folio_set_head(folio);
- ret = folio_ref_freeze(folio, 1);
- VM_BUG_ON(!ret);
+ set_page_count(head, 0);
hugetlb_folio_init_tail_vmemmap(folio, h, 1, nr_pages);
- prep_compound_head(&folio->page, huge_page_order(h));
+ prep_compound_head(head, huge_page_order(h));
}
/*
@@ -3368,8 +3361,9 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) {
struct page *page = virt_to_page(m);
struct folio *folio = (void *)page;
- const unsigned long pfn = folio_pfn(folio);
+ const unsigned long pfn = PHYS_PFN(__pa(m));
const unsigned long nr_pages = pages_per_huge_page(m->hstate);
+ const enum zone_type zone = zone_idx(pfn_to_zone(pfn, nid));
unsigned long nr_struct_pages = nr_pages;
h = m->hstate;
@@ -3382,11 +3376,10 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
prev_h = h;
VM_BUG_ON(!hstate_is_gigantic(h));
- WARN_ON(folio_ref_count(folio) != 1);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
nr_struct_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
- hugetlb_folio_init_vmemmap(folio, h, nr_struct_pages);
+ hugetlb_folio_init_vmemmap(folio, pfn, h, zone, nid, nr_struct_pages);
init_new_hugetlb_folio(folio);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 09/14] mm/hugetlb: use common compound page initialization helper
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (7 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 08/14] mm/hugetlb: initialize all struct pages for bootmem hugepages Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 10/14] mm/mm_init: make __init_single_page() static Muchun Song
` (6 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Boot-time gigantic hugetlb folios open-code the frozen compound
struct page initialization across two helpers.
Use memmap_init_compound_page_frozen() and remove the open-coded
helpers. This keeps the initialization rules shared with ZONE_DEVICE
instead of maintaining two implementations.
No functional change is intended.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/hugetlb.c | 65 ++--------------------------------------------------
1 file changed, 2 insertions(+), 63 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 2716a83a7386..6ee36ca5f435 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3249,63 +3249,6 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid)
return true;
}
-/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */
-static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
- struct hstate *h,
- unsigned long start_page_number,
- unsigned long end_page_number)
-{
- enum zone_type zone = folio_zonenum(folio);
- int nid = folio_nid(folio);
- struct page *page = folio_page(folio, start_page_number);
- struct page template;
- unsigned long head_pfn = folio_pfn(folio);
- unsigned long pfn, end_pfn = head_pfn + end_page_number;
- unsigned int order = huge_page_order(h);
-
- /*
- * As we marked all tail pages with memblock_reserved_mark_noinit(),
- * we must initialize them ourselves here.
- */
- if (start_page_number >= end_page_number)
- return;
-
- pfn = head_pfn + start_page_number;
-
- /*
- * Seed a template from the first real tail page and refresh only
- * PFN-dependent fields before copying it.
- */
- __init_single_page(page, pfn, zone, nid);
- prep_compound_tail(page, &folio->page, order);
- set_page_count(page, 0);
- memcpy(&template, page, sizeof(template));
-
- for (pfn++; pfn < end_pfn; pfn++)
- memmap_init_page_from_template(pfn_to_page(pfn), pfn,
- folio_zone(folio), &template);
-}
-
-static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
- unsigned long pfn,
- struct hstate *h,
- enum zone_type zone, int nid,
- unsigned long nr_pages)
-{
- struct page *head = &folio->page;
-
- /*
- * This is an open-coded prep_compound_page() whereby we avoid
- * walking pages twice by initializing/preparing+freezing them in the
- * same go.
- */
- __init_single_page(head, pfn, zone, nid);
- __folio_set_head(folio);
- set_page_count(head, 0);
- hugetlb_folio_init_tail_vmemmap(folio, h, 1, nr_pages);
- prep_compound_head(head, huge_page_order(h));
-}
-
/*
* memblock-allocated pageblocks might not have the migrate type set
* if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE)
@@ -3363,8 +3306,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
struct folio *folio = (void *)page;
const unsigned long pfn = PHYS_PFN(__pa(m));
const unsigned long nr_pages = pages_per_huge_page(m->hstate);
- const enum zone_type zone = zone_idx(pfn_to_zone(pfn, nid));
- unsigned long nr_struct_pages = nr_pages;
+ const struct zone *zone = pfn_to_zone(pfn, nid);
h = m->hstate;
/*
@@ -3376,10 +3318,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
prev_h = h;
VM_BUG_ON(!hstate_is_gigantic(h));
-
- if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
- nr_struct_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
- hugetlb_folio_init_vmemmap(folio, pfn, h, zone, nid, nr_struct_pages);
+ memmap_init_compound_page_frozen(page, pfn, zone, huge_page_order(h));
init_new_hugetlb_folio(folio);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 10/14] mm/mm_init: make __init_single_page() static
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (8 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 09/14] mm/hugetlb: use common compound page initialization helper Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 11/14] mm/mm_init: make memmap_init_page_from_template() static Muchun Song
` (5 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
__init_single_page() is only used within mm/mm_init.c, so make it
static.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/mm_init.c | 2 +-
mm/mm_init.h | 2 --
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index ea9ff690d4e0..84de0870f78a 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -618,7 +618,7 @@ void memmap_init_page_common(struct page *page, enum zone_type zone, int nid)
#endif
}
-void __meminit __init_single_page(struct page *page, unsigned long pfn,
+static __meminit void __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid)
{
memmap_init_page_common(page, zone, nid);
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 721221d2e4c0..c821aee774d3 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -110,8 +110,6 @@ bool memblock_has_mirror(void);
void memblock_free_all(void);
void memmap_init_page_common(struct page *page, enum zone_type zone, int nid);
-void __meminit __init_single_page(struct page *page, unsigned long pfn,
- unsigned long zone, int nid);
void __meminit memmap_init_page_from_template(struct page *page,
unsigned long pfn, const struct zone *zone, struct page *template);
void __meminit memmap_init_compound_page_frozen(struct page *head,
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 11/14] mm/mm_init: make memmap_init_page_from_template() static
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (9 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 10/14] mm/mm_init: make __init_single_page() static Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 12/14] mm/cma: centralize CMA pageblock initialization Muchun Song
` (4 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
memmap_init_page_from_template() is only used within mm/mm_init.c, so make
it static.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/mm_init.c | 2 +-
mm/mm_init.h | 2 --
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 84de0870f78a..6bf25d03902d 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -719,7 +719,7 @@ static __meminit void pageblock_migratetype_init_range(unsigned long pfn,
}
}
-void __meminit memmap_init_page_from_template(struct page *page,
+static __meminit void memmap_init_page_from_template(struct page *page,
unsigned long pfn, const struct zone *zone, struct page *template)
{
/*
diff --git a/mm/mm_init.h b/mm/mm_init.h
index c821aee774d3..7cde357d6324 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -110,8 +110,6 @@ bool memblock_has_mirror(void);
void memblock_free_all(void);
void memmap_init_page_common(struct page *page, enum zone_type zone, int nid);
-void __meminit memmap_init_page_from_template(struct page *page,
- unsigned long pfn, const struct zone *zone, struct page *template);
void __meminit memmap_init_compound_page_frozen(struct page *head,
unsigned long pfn, const struct zone *zone, unsigned int order);
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 12/14] mm/cma: centralize CMA pageblock initialization
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (10 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 11/14] mm/mm_init: make memmap_init_page_from_template() static Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 13/14] mm/cma: localize init_cma_pageblock() to cma.c Muchun Song
` (3 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Early reservations made with cma_reserve_early() currently require callers
to set the pageblock migratetype and update CMA statistics. These
operations maintain CMA state and should be handled by the CMA core.
Move initialization of early-reserved pageblocks into cma_activate_area(),
alongside the remaining pageblocks. If activation fails, restore the
managed-page accounting for early-reserved pages because they remain owned
by the caller and are not released by the CMA cleanup path.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/cma.c | 16 ++++++++++------
mm/hugetlb.c | 8 +++-----
2 files changed, 13 insertions(+), 11 deletions(-)
diff --git a/mm/cma.c b/mm/cma.c
index 9ec1d1fd5dff..46e72734ba72 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -163,6 +163,10 @@ static void __init cma_activate_area(struct cma *cma)
count = early_pfn[r] - cmr->base_pfn;
bitmap_count = cma_bitmap_pages_to_bits(cma, count);
bitmap_set(cmr->bitmap, 0, bitmap_count);
+
+ for (pfn = cmr->base_pfn; pfn < early_pfn[r];
+ pfn += pageblock_nr_pages)
+ init_cma_pageblock(pfn_to_page(pfn));
}
WARN_ON_ONCE(!pfn_valid(cmr->base_pfn));
@@ -191,12 +195,13 @@ static void __init cma_activate_area(struct cma *cma)
/* Expose all pages to the buddy, they are useless for CMA. */
if (!test_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags)) {
for (r = 0; r < cma->nranges; r++) {
- unsigned long start_pfn;
-
cmr = &cma->ranges[r];
- start_pfn = r <= allocrange ? early_pfn[r] : cmr->early_pfn;
+ end_pfn = r <= allocrange ? early_pfn[r] : cmr->early_pfn;
+ for (pfn = cmr->base_pfn; pfn < end_pfn; pfn++)
+ adjust_managed_page_count(pfn_to_page(pfn), 1);
+
end_pfn = cmr->base_pfn + cmr->count;
- for (pfn = start_pfn; pfn < end_pfn; pfn++)
+ for (; pfn < end_pfn; pfn++)
free_reserved_page(pfn_to_page(pfn));
}
}
@@ -1110,8 +1115,7 @@ bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end)
*
* The caller is responsible for initializing the page structures
* in the area properly, since this just points to memblock-allocated
- * memory. The caller should subsequently use init_cma_pageblock to
- * set the migrate type and CMA stats the pageblocks that were reserved.
+ * memory.
*
* If the CMA area fails to activate later, memory obtained through
* this interface is not handed to the page allocator, this is
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 6ee36ca5f435..f70c1373235f 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3267,9 +3267,7 @@ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio,
WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio)));
for (i = 0; i < nr_pages; i += pageblock_nr_pages) {
- if (folio_test_hugetlb_cma(folio))
- init_cma_pageblock(folio_page(folio, i));
- else
+ if (!folio_test_hugetlb_cma(folio))
init_pageblock_migratetype(folio_page(folio, i),
MIGRATE_MOVABLE, false);
}
@@ -3332,8 +3330,8 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
* in order to fix confusing memory reports from free(1) and
* other side-effects, like CommitLimit going negative.
*
- * For CMA pages, this is done in init_cma_pageblock
- * (via hugetlb_bootmem_init_migratetype), so skip it here.
+ * For CMA pages, this is done in cma_activate_area(), so skip
+ * it here.
*/
if (hugetlb_early_cma(h))
folio_set_hugetlb_cma(folio);
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 13/14] mm/cma: localize init_cma_pageblock() to cma.c
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (11 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 12/14] mm/cma: centralize CMA pageblock initialization Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-13 3:34 ` [PATCH v2 14/14] mm/mm_init: move pageblock migratetype init to memmap helpers Muchun Song
` (2 subsequent siblings)
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Move init_cma_pageblock() from mm_init.c into cma.c so it lives
alongside the CMA code that uses it.
No functional change intended.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/cma.c | 8 ++++++++
mm/mm_init.c | 9 ---------
mm/mm_init.h | 8 --------
3 files changed, 8 insertions(+), 17 deletions(-)
diff --git a/mm/cma.c b/mm/cma.c
index 46e72734ba72..7ca0e100504d 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -30,6 +30,7 @@
#include <linux/io.h>
#include <linux/kmemleak.h>
#include <trace/events/cma.h>
+#include <linux/page-isolation.h>
#include "internal.h"
#include "cma.h"
@@ -138,6 +139,13 @@ bool cma_validate_zones(struct cma *cma)
return true;
}
+static void __init init_cma_pageblock(struct page *page)
+{
+ init_pageblock_migratetype(page, MIGRATE_CMA, false);
+ adjust_managed_page_count(page, pageblock_nr_pages);
+ page_zone(page)->cma_pages += pageblock_nr_pages;
+}
+
static void __init cma_activate_area(struct cma *cma)
{
unsigned long pfn, end_pfn, early_pfn[CMA_MAX_RANGES];
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 6bf25d03902d..9a51cecee547 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2211,15 +2211,6 @@ void __init init_cma_reserved_pageblock(struct page *page)
adjust_managed_page_count(page, pageblock_nr_pages);
page_zone(page)->cma_pages += pageblock_nr_pages;
}
-/*
- * Similar to above, but only set the migrate type and stats.
- */
-void __init init_cma_pageblock(struct page *page)
-{
- init_pageblock_migratetype(page, MIGRATE_CMA, false);
- adjust_managed_page_count(page, pageblock_nr_pages);
- page_zone(page)->cma_pages += pageblock_nr_pages;
-}
#endif
/*
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 7cde357d6324..1723db7e54cc 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -40,14 +40,6 @@ struct zone *pfn_to_zone(unsigned long pfn, int nid);
void init_cma_reserved_pageblock(struct page *page);
#endif
-#ifdef CONFIG_CMA
-void init_cma_pageblock(struct page *page);
-#else
-static inline void init_cma_pageblock(struct page *page)
-{
-}
-#endif
-
/* Memory initialisation debug and verification */
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
DECLARE_STATIC_KEY_TRUE(deferred_pages);
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v2 14/14] mm/mm_init: move pageblock migratetype init to memmap helpers
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (12 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 13/14] mm/cma: localize init_cma_pageblock() to cma.c Muchun Song
@ 2026-11-13 3:34 ` Muchun Song
2026-11-16 4:20 ` [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Lance Yang
2026-11-16 10:42 ` Mike Rapoport
15 siblings, 0 replies; 22+ messages in thread
From: Muchun Song @ 2026-11-13 3:34 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
ZONE_DEVICE memmap initialization already initializes pageblock
migratetype after all struct pages have been initialized, while hugetlb
bootmem folios have their own duplicate bootmem-specific initialization.
Move the MIGRATE_MOVABLE pageblock initialization into the frozen memmap
initialization helpers instead, so the ZONE_DEVICE and hugetlb bootmem
paths can share the same initialization.
The old hugetlb_bootmem_init_migratetype() skipped CMA folios via
folio_test_hugetlb_cma(), but the shared helper always initializes
pageblocks as MIGRATE_MOVABLE. This is safe because cma_activate_area()
later overrides the migratetype for CMA pageblocks.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v2:
- Fix the helper name in the commit message (reported by Sashiko).
---
mm/hugetlb.c | 25 -------------------------
mm/mm_init.c | 11 ++++++-----
2 files changed, 6 insertions(+), 30 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index f70c1373235f..a2b0912290f4 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3249,30 +3249,6 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid)
return true;
}
-/*
- * memblock-allocated pageblocks might not have the migrate type set
- * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE)
- * here, or MIGRATE_CMA if this was a page allocated through an early CMA
- * reservation.
- *
- * In case of vmemmap optimized folios, the tail vmemmap pages are mapped
- * read-only, but that's ok - for sparse vmemmap this does not write to
- * the page structure.
- */
-static void __init hugetlb_bootmem_init_migratetype(struct folio *folio,
- struct hstate *h)
-{
- unsigned long nr_pages = pages_per_huge_page(h), i;
-
- WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio)));
-
- for (i = 0; i < nr_pages; i += pageblock_nr_pages) {
- if (!folio_test_hugetlb_cma(folio))
- init_pageblock_migratetype(folio_page(folio, i),
- MIGRATE_MOVABLE, false);
- }
-}
-
static void __init prep_and_add_bootmem_folios(struct hstate *h,
struct list_head *folio_list)
{
@@ -3280,7 +3256,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
struct folio *folio, *tmp_f;
list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
- hugetlb_bootmem_init_migratetype(folio, h);
/* Subdivide locks to achieve better parallel performance */
spin_lock_irqsave(&hugetlb_lock, flags);
account_new_hugetlb_folio(h, folio);
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 9a51cecee547..e402c5ed3152 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -759,6 +759,9 @@ void __meminit memmap_init_compound_page_frozen(struct page *head,
for (unsigned long i = 1; i < nr_pages; i++)
memmap_init_page_from_template(head + i, pfn + i, zone, &template);
prep_compound_head(head, order);
+
+ pageblock_migratetype_init_range(pfn, 1UL << order, MIGRATE_MOVABLE,
+ /* isolate */ false, /* atomic */ false);
}
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
@@ -1119,6 +1122,9 @@ static void memmap_init_zone_device_basepages(unsigned long start, unsigned long
if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
cond_resched();
}
+
+ pageblock_migratetype_init_range(start, end - start, MIGRATE_MOVABLE,
+ /* isolate */ false, /* atomic */ false);
}
void __ref memmap_init_zone_device(struct zone *zone,
@@ -1156,13 +1162,8 @@ void __ref memmap_init_zone_device(struct zone *zone,
memmap_init_compound_page_frozen(head, pfn, zone, pgmap->vmemmap_shift);
memmap_init_zone_device_head_fields(head, pgmap);
- if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
- cond_resched();
}
out:
- pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE,
- /* isolate */ false, /* atomic */ false);
-
pr_debug("%s initialised %lu pages in %ums\n", __func__,
nr_pages, jiffies_to_msecs(jiffies - start));
}
--
2.54.0
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 00/14] mm: Unify memmap initialization for compound pages
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (13 preceding siblings ...)
2026-11-13 3:34 ` [PATCH v2 14/14] mm/mm_init: move pageblock migratetype init to memmap helpers Muchun Song
@ 2026-11-16 4:20 ` Lance Yang
2026-11-16 10:42 ` Mike Rapoport
15 siblings, 0 replies; 22+ messages in thread
From: Lance Yang @ 2026-11-16 4:20 UTC (permalink / raw)
To: Muchun Song
Cc: Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, David Hildenbrand,
Suren Baghdasaryan, Oscar Salvador, Andrew Morton, Michal Hocko,
Muchun Song, James Houghton, Mike Rapoport, Qi Zheng, linux-mm,
linux-kernel
On 2026/11/13 11:34, Muchun Song wrote:
> Hi,
>
> This v2 is based on the mm-new branch at commit daed51c91051.
>
> This series is split out from the earlier, larger series
> "mm: Generalize HVO for HugeTLB and device DAX" [1]. That series
> generalized vmemmap optimization across HugeTLB and device DAX. One
> remaining piece is the separate compound struct page initialization used
> by boot-time gigantic HugeTLB pages and compound ZONE_DEVICE ranges.
>
> This v2 also folds in Li Zhe's two-patch series, using that series' v2
> as the integration point [2]. Those patches add
> memmap_init_page_from_template() and use it to reduce bootmem HugeTLB
> tail struct page initialization overhead. The surrounding patches in
> this queue are adjusted to build on that helper directly, so the merged
> series keeps one template-copy helper and then shares frozen compound
> page initialization between ZONE_DEVICE and HugeTLB.
>
> The first two patches are the merged prerequisite series. The next four
> patches make the generic memmap initialization code suitable for
> sharing. They add zone-consistency verification to template copies,
> factor the PFN-independent page state, simplify the ZONE_DEVICE template
> setup, and introduce a helper for initializing frozen compound pages.
>
> The HugeTLB patches then make boot-time gigantic HugeTLB initialization
> rely on the common frozen compound-page helper. The following small
> cleanups make now-local helpers static. The final three patches move
> early-reserved CMA pageblock setup into cma_activate_area() and then move
> MIGRATE_MOVABLE pageblock initialization into the shared memmap helpers.
>
> After this series, boot-time HugeTLB and compound ZONE_DEVICE ranges,
> including device DAX, use the same helpers for frozen compound-page and
> pageblock initialization.
>
> [1] https://lore.kernel.org/20260513130542.35604-1-songmuchun@bytedance.com/
> [2] https://lore.kernel.org/20261112112816.55095-1-lizhe.67@bytedance.com/
>
> v2:
> - Fold in Li Zhe's memmap template helper series, based on its v2 [2].
Nit: actually, the link [2] points to Li Zhe's v3, while this says v2 :)
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH v2 00/14] mm: Unify memmap initialization for compound pages
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
` (14 preceding siblings ...)
2026-11-16 4:20 ` [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Lance Yang
@ 2026-11-16 10:42 ` Mike Rapoport
15 siblings, 0 replies; 22+ messages in thread
From: Mike Rapoport @ 2026-11-16 10:42 UTC (permalink / raw)
To: Muchun Song
Cc: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan,
Michal Hocko, Muchun Song, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel
Hi,
> Hi,
>
> This v2 is based on the mm-new branch at commit daed51c91051.
>
> This series is split out from the earlier, larger series
> "mm: Generalize HVO for HugeTLB and device DAX" [1]. That series
> generalized vmemmap optimization across HugeTLB and device DAX. One
> remaining piece is the separate compound struct page initialization used
> by boot-time gigantic HugeTLB pages and compound ZONE_DEVICE ranges.
>
> This v2 also folds in Li Zhe's two-patch series, using that series' v2
> as the integration point [2].
>
> Li Zhe (2):
> mm/mm_init: factor out memmap_init_page_from_template()
> mm/hugetlb: optimize bootmem hugepage tail vmemmap init
>
> Muchun Song (12):
> mm/mm_init: verify zones for template-initialized pages
> mm/mm_init: factor out common struct page initialization
> mm/mm_init: simplify zone device page initialization
> mm/mm_init: factor out frozen compound page initialization
> mm/hugetlb: drop boot-time HVO handling for gigantic folios
> mm/hugetlb: initialize all struct pages for bootmem hugepages
> mm/hugetlb: use common compound page initialization helper
> mm/mm_init: make __init_single_page() static
> mm/mm_init: make memmap_init_page_from_template() static
> mm/cma: centralize CMA pageblock initialization
> mm/cma: localize init_cma_pageblock() to cma.c
> mm/mm_init: move pageblock migratetype init to memmap helpers
>
> include/linux/mm.h | 8 --
> mm/cma.c | 24 +++--
> mm/hugetlb.c | 111 ++-----------------
> mm/hugetlb_vmemmap.c | 17 +--
> mm/mm_init.c | 247 +++++++++++++++++--------------------------
> mm/mm_init.h | 13 +--
> mm/sparse-vmemmap.c | 7 +-
> 7 files changed, 129 insertions(+), 298 deletions(-)
That's nice :)
The series would be easier to review if Li Zhe's patches were integrated
rather than stacked in. I hoped that the changes for template
initialization would fit into the patches that rework other parts.
The way it's done now several patches in the beginning of the series
change code that the later patches remove or turn some of new functions
into static.
Another thing is that the numbers Li Zhe measured were for a different
implementation, so they would be stale for the state after these
changes.
I have some nits and naming comments in a few patches, other than that
the series LGTM.
--
Sincerely yours,
Mike.
^ permalink raw reply [flat|nested] 22+ messages in thread