* [PATCH v3 00/11] mm: Unify memmap initialization for compound pages
@ 2026-11-20 10:08 Muchun Song
2026-11-20 10:08 ` [PATCH v3 01/11] mm/mm_init: verify zones for template-initialized pages Muchun Song
` (10 more replies)
0 siblings, 11 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Hi,
This v3 is based on the mm-new branch at commit 15f114dac9a7.
This series is split out from the earlier, larger series
"mm: Generalize HVO for HugeTLB and device DAX" [1]. That series
generalized vmemmap optimization across HugeTLB and device DAX. One
remaining piece is the separate compound struct page initialization used
by boot-time gigantic HugeTLB pages and compound ZONE_DEVICE ranges.
In v3, Li Zhe's template-copy work is integrated directly into the
generic compound-page initialization changes instead of being stacked as
two prerequisite patches. The shared memmap_init_compound_page() helper
now covers both compound ZONE_DEVICE ranges and boot-time gigantic
HugeTLB folios. ZONE_DEVICE retains memcpy_nontemporal(), while normal
memory uses memcpy().
The first four patches make the generic memmap initialization code
suitable for sharing. They add zone-consistency verification, factor the
PFN-independent page state, simplify ZONE_DEVICE template setup, and
introduce the common compound-page initialization helper.
The next four patches remove the boot-time HugeTLB HVO special handling,
initialize all bootmem hugepage struct pages in one place, switch HugeTLB
to the common helper, and make the now-local page initializer static.
The final three patches centralize early CMA pageblock setup and move
MIGRATE_MOVABLE pageblock initialization into the shared memmap helpers.
After this series, boot-time HugeTLB and compound ZONE_DEVICE ranges,
including device DAX, use the same helpers for compound-page and
pageblock initialization.
[1] https://lore.kernel.org/20260513130542.35604-1-songmuchun@bytedance.com/
[2] https://lore.kernel.org/20261112112816.55095-1-lizhe.67@bytedance.com/
v3:
- Integrate Li Zhe's template-copy work into the common compound-page
helper and drop the standalone prerequisite patches (suggested by Mike
Rapoport)
- Apply the helper naming cleanups and remove duplicate ZONE_DEVICE
reserved-page initialization (suggested by Mike Rapoport)
- Update the HugeTLB performance results for the integrated
implementation (measured by Li Zhe)
- Restore managed-page accounting after CMA activation failure
(reported by Sashiko)
- Correct [2] to refer to Li Zhe's v3 (reported by Lance Yang)
- Collect Reviewed-by from Lance Yang
v2: https://lore.kernel.org/20261113033442.62970-1-songmuchun@bytedance.com/
- Fold in Li Zhe's memmap template helper series, based on its v2 [2].
- Remove __meminit from memmap_init_page_common().
- Add a cleanup to make memmap_init_page_from_template() static.
- Collect Reviewed-by from Mike Rapoport.
- Fix the helper name in the final commit message.
v1: https://lore.kernel.org/20261109095807.41547-1-songmuchun@bytedance.com/
Muchun Song (11):
mm/mm_init: verify zones for template-initialized pages
mm/mm_init: factor out common struct page initialization
mm/mm_init: simplify zone device page initialization
mm/mm_init: factor out compound page initialization
mm/hugetlb: drop boot-time HVO handling for gigantic folios
mm/hugetlb: initialize all struct pages for bootmem hugepages
mm/hugetlb: use common compound page initialization helper
mm/mm_init: make __init_single_page() static
mm/cma: centralize CMA pageblock initialization
mm/cma: localize init_cma_pageblock() to cma.c
mm/mm_init: move pageblock migratetype init to memmap helpers
include/linux/mm.h | 8 --
mm/cma.c | 38 +++++--
mm/hugetlb.c | 111 ++----------------
mm/hugetlb_vmemmap.c | 19 +---
mm/mm_init.c | 261 +++++++++++++++++--------------------------
mm/mm_init.h | 13 +--
mm/sparse-vmemmap.c | 7 +-
7 files changed, 140 insertions(+), 317 deletions(-)
base-commit: 15f114dac9a7bde1796f0e54e22726fd31cfa8a5
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 01/11] mm/mm_init: verify zones for template-initialized pages
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 02/11] mm/mm_init: factor out common struct page initialization Muchun Song
` (9 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Template-based ZONE_DEVICE initialization bypasses
__init_single_page(), which means copied struct pages do not run the
vmemmap zone consistency check.
Verify each destination after copying the template so the fast path gets
the same validation as regular page initialization.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
---
v3:
- Retarget zone verification to
zone_device_page_init_from_template() after integrating the helper
series (suggested by Mike Rapoport)
- Collect Reviewed-by from Lance Yang
v2:
- Apply the zone verification in memmap_init_page_from_template().
- Collect Reviewed-by from Mike Rapoport.
---
mm/mm_init.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index a9d848c92678..0129152b74d8 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1082,6 +1082,7 @@ static void zone_device_page_init_from_template(struct page *page,
set_page_address(template, __va(pfn << PAGE_SHIFT));
#endif
memcpy_nontemporal(page, template, sizeof(*page));
+ vmemmap_optimization_verify_zone(page);
}
/*
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 02/11] mm/mm_init: factor out common struct page initialization
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
2026-11-20 10:08 ` [PATCH v3 01/11] mm/mm_init: verify zones for template-initialized pages Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 03/11] mm/mm_init: simplify zone device " Muchun Song
` (8 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Several struct page fields have the same initial values for every page in
a zone. __init_single_page() initializes this state as part of its
per-page setup, while the shared vmemmap tail path open-codes only a
subset.
Factor the PFN-independent state into memmap_init_page() and use it for
both regular and shared tail struct pages. Keep section-, PFN- and
instance-specific initialization in __init_single_page().
This also initializes the last cpupid and KASAN tag for shared tail
struct pages consistently with regular struct pages. Since
memmap_init_page() marks ZONE_DEVICE pages reserved, remove the redundant
update from the memory hotplug path.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v3:
- Rename memmap_init_page_common() to memmap_init_page() (suggested by
Mike Rapoport)
- Remove redundant ZONE_DEVICE reserved-page initialization (suggested by
Mike Rapoport)
v2:
- Remove __meminit from memmap_init_page_common() so HVO shared tail
pages can call it after init text is freed (reported by Sashiko).
---
include/linux/mm.h | 8 --------
mm/mm_init.c | 37 +++++++++++++++++++++----------------
mm/mm_init.h | 1 +
mm/sparse-vmemmap.c | 7 +------
4 files changed, 23 insertions(+), 30 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 8883b780b76d..d8bafcd92791 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3096,14 +3096,6 @@ static inline void set_page_node(struct page *page, unsigned long node)
page->flags.f |= (node & NODES_MASK) << NODES_PGSHIFT;
}
-static inline void set_page_links(struct page *page, enum zone_type zone,
- unsigned long node, unsigned long pfn)
-{
- set_page_zone(page, zone);
- set_page_node(page, node);
- set_page_section_from_pfn(page, pfn);
-}
-
/**
* folio_nr_pages - The number of pages in the folio.
* @folio: The folio.
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 0129152b74d8..08d6e47bde22 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -602,15 +602,27 @@ static inline void vmemmap_optimization_verify_zone(struct page *page)
page_zone_id(page));
}
-void __meminit __init_single_page(struct page *page, unsigned long pfn,
- unsigned long zone, int nid)
+/* Initialize PFN-independent fields common to pages in the same zone. */
+void memmap_init_page(struct page *page, enum zone_type zone, int nid)
{
mm_zero_struct_page(page);
- set_page_links(page, zone, nid, pfn);
- init_page_count(page);
- atomic_set(&page->_mapcount, -1);
+ set_page_zone(page, zone);
+ set_page_node(page, nid);
page_cpupid_reset_last(page);
page_kasan_tag_reset(page);
+ atomic_set(&page->_mapcount, -1);
+#ifdef CONFIG_ZONE_DEVICE
+ if (zone == ZONE_DEVICE)
+ __SetPageReserved(page);
+#endif
+}
+
+void __meminit __init_single_page(struct page *page, unsigned long pfn,
+ unsigned long zone, int nid)
+{
+ memmap_init_page(page, zone, nid);
+ set_page_section_from_pfn(page, pfn);
+ init_page_count(page);
INIT_LIST_HEAD(&page->lru);
#ifdef WANT_PAGE_VIRTUAL
@@ -891,11 +903,11 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone
{
unsigned long pfn, end_pfn = start_pfn + size;
struct page *page;
+ const struct zone *z = &NODE_DATA(nid)->node_zones[zone];
if (highest_memmap_pfn < end_pfn - 1)
highest_memmap_pfn = end_pfn - 1;
-#ifdef CONFIG_ZONE_DEVICE
/*
* Honor reservation requested by the driver for this ZONE_DEVICE
* memory. We limit the total number of pages to initialize to just
@@ -903,7 +915,7 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone
* ZONE_DEVICE page initialization until after we have released
* the hotplug lock.
*/
- if (zone == ZONE_DEVICE) {
+ if (zone_is_zone_device(z)) {
if (!altmap)
return;
@@ -911,7 +923,6 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone
start_pfn += altmap->reserve;
end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap);
}
-#endif
for (pfn = start_pfn; pfn < end_pfn; ) {
const unsigned int order = pfn_to_section_compound_order(pfn);
@@ -938,14 +949,8 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone
page = pfn_to_page(pfn);
__init_single_page(page, pfn, zone, nid);
- if (context == MEMINIT_HOTPLUG) {
-#ifdef CONFIG_ZONE_DEVICE
- if (zone == ZONE_DEVICE)
- __SetPageReserved(page);
- else
-#endif
- __SetPageOffline(page);
- }
+ if (context == MEMINIT_HOTPLUG && !zone_is_zone_device(z))
+ __SetPageOffline(page);
if (pageblock_aligned(pfn))
cond_resched();
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 0034e37dbabd..dec22844f41e 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -109,6 +109,7 @@ extern bool mirrored_kernelcore;
bool memblock_has_mirror(void);
void memblock_free_all(void);
+void memmap_init_page(struct page *page, enum zone_type zone, int nid);
void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid);
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index 9b80dd7326f7..51fa77982ca5 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -212,12 +212,7 @@ struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zon
for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) {
page = (struct page *)addr + i;
- mm_zero_struct_page(page);
- atomic_set(&page->_mapcount, -1);
- set_page_node(page, zone_to_nid(zone));
- set_page_zone(page, zone_idx(zone));
- if (zone_is_zone_device(zone))
- __SetPageReserved(page);
+ memmap_init_page(page, zone_idx(zone), zone_to_nid(zone));
prep_compound_tail(page, NULL, order);
}
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 03/11] mm/mm_init: simplify zone device page initialization
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
2026-11-20 10:08 ` [PATCH v3 01/11] mm/mm_init: verify zones for template-initialized pages Muchun Song
2026-11-20 10:08 ` [PATCH v3 02/11] mm/mm_init: factor out common struct page initialization Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 04/11] mm/mm_init: factor out compound " Muchun Song
` (7 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
memmap_init_zone_device() currently seeds its reusable head and tail
templates by fully initializing the first real struct page and copying
it back into a stack object. This keeps the first page outside the
common loops and leaves __init_zone_device_page() only for template
construction.
Build both templates directly from the PFN-independent state. Add the
head-specific pgmap and refcount state to the head template and the
compound metadata to the tail template, then initialize every page
through zone_device_page_init_from_template().
This removes the special handling for the first head and tail pages
without changing their final state.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v3:
- Rebase on zone_device_page_init_from_template() after integrating the
template helper series (suggested by Mike Rapoport)
v2:
- Rebase on memmap_init_page_from_template().
---
mm/mm_init.c | 94 ++++++++--------------------------------------------
1 file changed, 14 insertions(+), 80 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 08d6e47bde22..75d8d17601ed 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1036,48 +1036,6 @@ static void __init memmap_init(void)
}
#ifdef CONFIG_ZONE_DEVICE
-static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
- unsigned long zone_idx, int nid,
- struct dev_pagemap *pgmap)
-{
-
- __init_single_page(page, pfn, zone_idx, nid);
-
- /*
- * Mark page reserved as it will need to wait for onlining
- * phase for it to be fully associated with a zone.
- *
- * We can use the non-atomic __set_bit operation for setting
- * the flag as we are still initializing the pages.
- */
- __SetPageReserved(page);
-
- /*
- * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer
- * and zone_device_data. It is a bug if a ZONE_DEVICE page is
- * ever freed or placed on a driver-private list.
- */
- page_folio(page)->pgmap = pgmap;
- page->zone_device_data = NULL;
-
- /*
- * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free
- * path. The remaining ZONE_DEVICE types start from 0 here and raise
- * the count again when the allocator or driver hands the page out.
- */
- switch (pgmap->type) {
- case MEMORY_DEVICE_FS_DAX:
- case MEMORY_DEVICE_PRIVATE:
- case MEMORY_DEVICE_COHERENT:
- case MEMORY_DEVICE_PCI_P2PDMA:
- set_page_count(page, 0);
- break;
-
- case MEMORY_DEVICE_GENERIC:
- break;
- }
-}
-
static void zone_device_page_init_from_template(struct page *page,
unsigned long pfn, struct page *template)
{
@@ -1118,30 +1076,13 @@ static void __ref memmap_init_compound(struct page *head,
unsigned long pfn, end_pfn = head_pfn + nr_pages;
unsigned int order = pgmap->vmemmap_shift;
struct page template;
- struct page *page;
- /*
- * We have to initialize the pages, including setting up page links.
- * prep_compound_page() does not take care of that, so instead we
- * open-code prep_compound_page() so we can take care of initializing
- * the pages in the same go.
- */
- __SetPageHead(head);
-
- /*
- * All tails of the same compound page share the state established by
- * prep_compound_tail(). Reuse one tail template for the whole range and
- * refresh only the PFN-dependent fields in that template before each copy.
- */
- pfn = head_pfn + 1;
- page = pfn_to_page(pfn);
- __init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
- prep_compound_tail(page, head, order);
- set_page_count(page, 0);
- memcpy(&template, page, sizeof(*page));
+ memmap_init_page(&template, zone_idx, nid);
+ prep_compound_tail(&template, head, order);
- /* Initialize the remaining tail pages from template. */
- for (pfn = head_pfn + 2; pfn < end_pfn; pfn++)
+ __SetPageHead(head);
+ /* Initialize the tail pages from template. */
+ for (pfn = head_pfn + 1; pfn < end_pfn; pfn++)
zone_device_page_init_from_template(pfn_to_page(pfn), pfn,
&template);
prep_compound_head(head, order);
@@ -1175,27 +1116,20 @@ void __ref memmap_init_zone_device(struct zone *zone,
nr_pages = end_pfn - start_pfn;
}
- if (!nr_pages)
- return;
-
VM_WARN_ON(!IS_ALIGNED(start_pfn | nr_pages, pfns_per_compound));
+ memmap_init_page(&template, zone_idx, nid);
+ page_folio(&template)->pgmap = pgmap;
/*
- * Seed the reusable head-page template from the first real struct
- * page. The normal page-init and refcount helpers must operate on
- * a real memmap entry rather than a stack object.
+ * MEMORY_DEVICE_GENERIC pages regain a refcount of one in the free
+ * path. The remaining ZONE_DEVICE types start from zero here and raise
+ * the count again when the allocator or driver hands the page out.
*/
- pfn = start_pfn;
- page = pfn_to_page(pfn);
- __init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
- memcpy(&template, page, sizeof(*page));
- if (pfns_per_compound != 1)
- memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
- compound_nr_pages(pfn, pgmap));
- pfn += pfns_per_compound;
+ if (pgmap->type == MEMORY_DEVICE_GENERIC)
+ atomic_set(&template._refcount, 1);
- /* Initialize the remaining head pages from template. */
- for (; pfn < end_pfn; pfn += pfns_per_compound) {
+ /* Initialize the head pages from template. */
+ for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {
page = pfn_to_page(pfn);
zone_device_page_init_from_template(page, pfn, &template);
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 04/11] mm/mm_init: factor out compound page initialization
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (2 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 03/11] mm/mm_init: simplify zone device " Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 05/11] mm/hugetlb: drop boot-time HVO handling for gigantic folios Muchun Song
` (6 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
ZONE_DEVICE initializes compound struct pages while their refcounts remain
frozen. The setup is open-coded in memmap_init_compound(), including the
per-page initialization and compound tail preparation.
Factor the common setup into memmap_init_compound_page(). Keep the pgmap
association and MEMORY_DEVICE_GENERIC refcount handling in the
ZONE_DEVICE-specific path.
This provides an MM-internal helper for other early memmap users to
share.
No functional change is intended.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v3:
- Fold memmap_init_page_from_template() into this patch and make it
static (suggested by Mike Rapoport)
- Remove the _frozen suffix from memmap_init_compound_page() (suggested
by Mike Rapoport)
- Rename the ZONE_DEVICE head helper to
memmap_init_zone_device_folio() and pass a folio (suggested by Mike
Rapoport)
- Rename memmap_init_zone_device_basepages() to
memmap_init_zone_device_pages() (suggested by Mike Rapoport)
v2:
- Rebase on memmap_init_page_from_template().
- Rename memmap_init_zone_device_head_page to memmap_init_zone_device_head_fields.
---
mm/mm_init.c | 143 ++++++++++++++++++++++++++-------------------------
mm/mm_init.h | 2 +
2 files changed, 76 insertions(+), 69 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 75d8d17601ed..3e645302377e 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -718,6 +718,45 @@ static __meminit void pageblock_migratetype_init_range(unsigned long pfn,
}
}
+static void __meminit memmap_init_page_from_template(struct page *page,
+ unsigned long pfn, const struct zone *zone, struct page *template)
+{
+ /*
+ * Refresh PFN-dependent fields in the template before the non-temporal
+ * copy, so the destination page does not need cached stores after
+ * memcpy_nontemporal().
+ */
+ set_page_section_from_pfn(template, pfn);
+#ifdef WANT_PAGE_VIRTUAL
+ if (!is_highmem_idx(zone_idx(zone)))
+ set_page_address(template, __va(pfn << PAGE_SHIFT));
+#endif
+ memcpy_nontemporal(page, template, sizeof(*page));
+ vmemmap_optimization_verify_zone(page);
+}
+
+void __meminit memmap_init_compound_page(struct page *head, unsigned long pfn,
+ const struct zone *zone, unsigned int order)
+{
+ unsigned long nr_pages = 1UL << order;
+ struct page template;
+
+ memmap_init_page(&template, zone_idx(zone), zone_to_nid(zone));
+ prep_compound_tail(&template, head, order);
+
+ /* Initialize the head page. */
+ __init_single_page(head, pfn, zone_idx(zone), zone_to_nid(zone));
+ set_page_count(head, 0);
+ __SetPageHead(head);
+
+ if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
+ nr_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
+ /* Initialize the tail pages from PFN-independent template page. */
+ for (unsigned long i = 1; i < nr_pages; i++)
+ memmap_init_page_from_template(head + i, pfn + i, zone, &template);
+ prep_compound_head(head, order);
+}
+
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
{
pg_data_t *pgdat = NODE_DATA(nid);
@@ -1036,56 +1075,39 @@ static void __init memmap_init(void)
}
#ifdef CONFIG_ZONE_DEVICE
-static void zone_device_page_init_from_template(struct page *page,
- unsigned long pfn, struct page *template)
+static void memmap_init_zone_device_folio(struct folio *folio,
+ struct dev_pagemap *pgmap)
{
- set_page_section_from_pfn(template, pfn);
-#ifdef WANT_PAGE_VIRTUAL
- if (!is_highmem_idx(ZONE_DEVICE))
- set_page_address(template, __va(pfn << PAGE_SHIFT));
-#endif
- memcpy_nontemporal(page, template, sizeof(*page));
- vmemmap_optimization_verify_zone(page);
-}
-
-/*
- * With compound page geometry and when struct pages are stored in ram most
- * tail pages are reused. Consequently, the amount of unique struct pages to
- * initialize is a lot smaller that the total amount of struct pages being
- * mapped. This is a paired / mild layering violation with explicit knowledge
- * of how the sparse_vmemmap internals handle compound pages in the lack
- * of an altmap.
- */
-static inline unsigned long compound_nr_pages(unsigned long pfn,
- struct dev_pagemap *pgmap)
-{
- const struct mem_section *ms = __pfn_to_section(pfn);
-
- if (!section_vmemmap_optimizable(ms))
- return pgmap_vmemmap_nr(pgmap);
-
- return VMEMMAP_OPTIMIZATION_PAGES * (PAGE_SIZE / sizeof(struct page));
+ /*
+ * MEMORY_DEVICE_GENERIC pages start with a refcount of one; other
+ * ZONE_DEVICE pages raise it when handed out. Use atomic_set() because
+ * @folio may be a stack template, while init_page_count() expects a
+ * memmap-backed page.
+ */
+ if (pgmap->type == MEMORY_DEVICE_GENERIC)
+ atomic_set(&folio->_refcount, 1);
+ /*
+ * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer and
+ * zone_device_data. It is a bug if a ZONE_DEVICE page is ever freed
+ * or placed on a driver-private list.
+ */
+ folio->pgmap = pgmap;
+ folio->page.zone_device_data = NULL;
}
-static void __ref memmap_init_compound(struct page *head,
- unsigned long head_pfn,
- unsigned long zone_idx, int nid,
- struct dev_pagemap *pgmap,
- unsigned long nr_pages)
+static void memmap_init_zone_device_pages(unsigned long start, unsigned long end,
+ const struct zone *zone, struct dev_pagemap *pgmap)
{
- unsigned long pfn, end_pfn = head_pfn + nr_pages;
- unsigned int order = pgmap->vmemmap_shift;
struct page template;
- memmap_init_page(&template, zone_idx, nid);
- prep_compound_tail(&template, head, order);
+ memmap_init_page(&template, zone_idx(zone), zone_to_nid(zone));
+ memmap_init_zone_device_folio(page_folio(&template), pgmap);
- __SetPageHead(head);
- /* Initialize the tail pages from template. */
- for (pfn = head_pfn + 1; pfn < end_pfn; pfn++)
- zone_device_page_init_from_template(pfn_to_page(pfn), pfn,
- &template);
- prep_compound_head(head, order);
+ for (unsigned long pfn = start; pfn < end; pfn++) {
+ memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone, &template);
+ if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
+ cond_resched();
+ }
}
void __ref memmap_init_zone_device(struct zone *zone,
@@ -1094,16 +1116,11 @@ void __ref memmap_init_zone_device(struct zone *zone,
struct dev_pagemap *pgmap)
{
unsigned long pfn, end_pfn = start_pfn + nr_pages;
- struct pglist_data *pgdat = zone->zone_pgdat;
struct vmem_altmap *altmap = pgmap_altmap(pgmap);
unsigned int pfns_per_compound = pgmap_vmemmap_nr(pgmap);
- unsigned long zone_idx = zone_idx(zone);
unsigned long start = jiffies;
- int nid = pgdat->node_id;
- struct page template;
- struct page *page;
- if (WARN_ON_ONCE(!pgmap || zone_idx != ZONE_DEVICE))
+ if (WARN_ON_ONCE(!pgmap || !zone_is_zone_device(zone)))
return;
/*
@@ -1118,32 +1135,20 @@ void __ref memmap_init_zone_device(struct zone *zone,
VM_WARN_ON(!IS_ALIGNED(start_pfn | nr_pages, pfns_per_compound));
- memmap_init_page(&template, zone_idx, nid);
- page_folio(&template)->pgmap = pgmap;
- /*
- * MEMORY_DEVICE_GENERIC pages regain a refcount of one in the free
- * path. The remaining ZONE_DEVICE types start from zero here and raise
- * the count again when the allocator or driver hands the page out.
- */
- if (pgmap->type == MEMORY_DEVICE_GENERIC)
- atomic_set(&template._refcount, 1);
+ if (pfns_per_compound == 1) {
+ memmap_init_zone_device_pages(start_pfn, end_pfn, zone, pgmap);
+ goto out;
+ }
- /* Initialize the head pages from template. */
for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {
- page = pfn_to_page(pfn);
-
- zone_device_page_init_from_template(page, pfn, &template);
+ struct page *head = pfn_to_page(pfn);
+ memmap_init_compound_page(head, pfn, zone, pgmap->vmemmap_shift);
+ memmap_init_zone_device_folio(page_folio(head), pgmap);
if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
cond_resched();
-
- if (pfns_per_compound == 1)
- continue;
-
- memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
- compound_nr_pages(pfn, pgmap));
}
-
+out:
pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE,
/* isolate */ false, /* atomic */ false);
diff --git a/mm/mm_init.h b/mm/mm_init.h
index dec22844f41e..1e6d2f78afdd 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -112,5 +112,7 @@ void memblock_free_all(void);
void memmap_init_page(struct page *page, enum zone_type zone, int nid);
void __meminit __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid);
+void __meminit memmap_init_compound_page(struct page *head, unsigned long pfn,
+ const struct zone *zone, unsigned int order);
#endif /* __MM_MM_INIT_H */
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 05/11] mm/hugetlb: drop boot-time HVO handling for gigantic folios
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (3 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 04/11] mm/mm_init: factor out compound " Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 06/11] mm/hugetlb: initialize all struct pages for bootmem hugepages Muchun Song
` (5 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
HugeTLB HVO is currently supported on x86-64, riscv64, and LoongArch.
On x86-64 and riscv64, gigantic HugeTLB pages are larger than the
section size, so the existing section-based vmemmap optimization
infrastructure is already sufficient to cover the whole folio. On
LoongArch, HugeTLB HVO is supported without gigantic HugeTLB pages.
Therefore, boot-time HugeTLB HVO folios can rely on the section-based
vmemmap optimization infrastructure directly, without the extra bulk
optimization and fallback handling.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/hugetlb.c | 25 +++++++------------------
mm/hugetlb_vmemmap.c | 19 +------------------
2 files changed, 8 insertions(+), 36 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index c01ae4067e64..8b2bf571e781 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3402,21 +3402,7 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
unsigned long flags;
struct folio *folio, *tmp_f;
- /* Send list for bulk vmemmap optimization processing */
- hugetlb_vmemmap_optimize_folios(h, folio_list);
-
list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
- if (!folio_test_hugetlb_vmemmap_optimized(folio)) {
- /*
- * If HVO fails, initialize all tail struct pages
- * We do not worry about potential long lock hold
- * time as this is early in boot and there should
- * be no contention.
- */
- hugetlb_folio_init_tail_vmemmap(folio, h,
- VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES,
- pages_per_huge_page(h));
- }
hugetlb_bootmem_init_migratetype(folio, h);
/* Subdivide locks to achieve better parallel performance */
spin_lock_irqsave(&hugetlb_lock, flags);
@@ -3440,6 +3426,8 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
struct page *page = virt_to_page(m);
struct folio *folio = (void *)page;
const unsigned long pfn = folio_pfn(folio);
+ const unsigned long nr_pages = pages_per_huge_page(m->hstate);
+ unsigned long nr_struct_pages = nr_pages;
h = m->hstate;
/*
@@ -3453,13 +3441,14 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
VM_BUG_ON(!hstate_is_gigantic(h));
WARN_ON(folio_ref_count(folio) != 1);
- hugetlb_folio_init_vmemmap(folio, h,
- VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES);
+ if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
+ nr_struct_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
+ hugetlb_folio_init_vmemmap(folio, h, nr_struct_pages);
init_new_hugetlb_folio(folio);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
folio_set_hugetlb_vmemmap_optimized(folio);
- section_set_compound_order_range(pfn, folio_nr_pages(folio), 0);
+ section_set_compound_order_range(pfn, nr_pages, 0);
list_add(&folio->lru, &folio_list);
@@ -3474,7 +3463,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
if (hugetlb_early_cma(h))
folio_set_hugetlb_cma(folio);
else
- adjust_managed_page_count(page, pages_per_huge_page(h));
+ adjust_managed_page_count(page, nr_pages);
cond_resched();
}
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 5852b0a1c479..09b9fd83c198 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -565,7 +565,6 @@ static int hugetlb_vmemmap_pteify_folio(const struct hstate *h, struct folio *fo
void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
{
struct folio *folio;
- unsigned long nr_to_optimize = 0;
LIST_HEAD(vmemmap_pages);
unsigned long flags = VMEMMAP_DEFER_TLB_FLUSH;
@@ -574,18 +573,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
down_write(&vmemmap_remap_rwsem);
list_for_each_entry(folio, folio_list, lru) {
- int ret;
-
- /*
- * Bootmem gigantic folios may already be marked optimized when
- * their vmemmap layout was prepared earlier, so skip them here.
- */
- if (folio_test_hugetlb_vmemmap_optimized(folio))
- continue;
-
- nr_to_optimize++;
-
- ret = hugetlb_vmemmap_pteify_folio(h, folio, flags);
+ int ret = hugetlb_vmemmap_pteify_folio(h, folio, flags);
/*
* Splitting the PMD requires allocating a page, thus let's fail
@@ -597,11 +585,6 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
break;
}
- if (!nr_to_optimize) {
- up_write(&vmemmap_remap_rwsem);
- return;
- }
-
flush_tlb_all();
downgrade_write(&vmemmap_remap_rwsem);
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 06/11] mm/hugetlb: initialize all struct pages for bootmem hugepages
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (4 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 05/11] mm/hugetlb: drop boot-time HVO handling for gigantic folios Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 07/11] mm/hugetlb: use common compound page initialization helper Muchun Song
` (4 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Boot-time gigantic hugepages currently leave the head struct page to
the generic memmap initialization path while the HugeTLB code
initializes the remaining struct pages itself.
Mark the full hugepage noinit and initialize the head struct page in
hugetlb_folio_init_vmemmap() as well, so the whole compound-page setup
is handled in one place.
This can also reduce memblock metadata overhead when many boot-time
HugeTLB pages are reserved, because physically contiguous hugepages
can be covered by fewer noinit regions.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v3:
- Defer the HugeTLB template-copy optimization to the common helper
switch after integrating the series (suggested by Mike Rapoport)
---
mm/hugetlb.c | 27 ++++++++++-----------------
1 file changed, 10 insertions(+), 17 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 8b2bf571e781..2a94f8f60870 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3313,15 +3313,7 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid)
list_add_tail(&m->list, &huge_boot_pages[nid]);
m->flags |= HUGE_BOOTMEM_ZONES_VALID;
hugetlb_vmemmap_optimize_bootmem_page(pfn, huge_page_order(h));
- /*
- * Only initialize the head struct page in memmap_init_reserved_pages,
- * rest of the struct pages will be initialized by the HugeTLB
- * subsystem itself.
- * The head struct page is used to get folio information by the HugeTLB
- * subsystem like zone id and node id.
- */
- memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE),
- huge_page_size(h) - PAGE_SIZE);
+ memblock_reserved_mark_noinit(__pa(m), huge_page_size(h));
}
return true;
@@ -3352,22 +3344,23 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
}
static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
+ unsigned long pfn,
struct hstate *h,
+ enum zone_type zone, int nid,
unsigned long nr_pages)
{
- int ret;
+ struct page *head = &folio->page;
/*
* This is an open-coded prep_compound_page() whereby we avoid
* walking pages twice by initializing/preparing+freezing them in the
* same go.
*/
- __folio_clear_reserved(folio);
+ __init_single_page(head, pfn, zone, nid);
__folio_set_head(folio);
- ret = folio_ref_freeze(folio, 1);
- VM_BUG_ON(!ret);
+ set_page_count(head, 0);
hugetlb_folio_init_tail_vmemmap(folio, h, 1, nr_pages);
- prep_compound_head(&folio->page, huge_page_order(h));
+ prep_compound_head(head, huge_page_order(h));
}
/*
@@ -3425,8 +3418,9 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) {
struct page *page = virt_to_page(m);
struct folio *folio = (void *)page;
- const unsigned long pfn = folio_pfn(folio);
+ const unsigned long pfn = PHYS_PFN(__pa(m));
const unsigned long nr_pages = pages_per_huge_page(m->hstate);
+ const enum zone_type zone = zone_idx(pfn_to_zone(pfn, nid));
unsigned long nr_struct_pages = nr_pages;
h = m->hstate;
@@ -3439,11 +3433,10 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
prev_h = h;
VM_BUG_ON(!hstate_is_gigantic(h));
- WARN_ON(folio_ref_count(folio) != 1);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
nr_struct_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
- hugetlb_folio_init_vmemmap(folio, h, nr_struct_pages);
+ hugetlb_folio_init_vmemmap(folio, pfn, h, zone, nid, nr_struct_pages);
init_new_hugetlb_folio(folio);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 07/11] mm/hugetlb: use common compound page initialization helper
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (5 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 06/11] mm/hugetlb: initialize all struct pages for bootmem hugepages Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 08/11] mm/mm_init: make __init_single_page() static Muchun Song
` (3 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Boot-time gigantic hugetlb folios currently use a separate path to
initialize their compound struct pages.
Use memmap_init_compound_page() for these folios so that HugeTLB and
ZONE_DEVICE share the common compound page initialization path. This
removes the HugeTLB-specific implementation and improves initialization
performance.
ZONE_DEVICE initializes struct pages by walking one contiguous memory
range. Boot-time HugeTLB instead initializes multiple disjoint ranges,
one for each contiguous hugepage. Measurements showed that memcpy()
performed better than memcpy_nontemporal() for the latter access pattern.
Therefore, use memcpy() for normal memory while retaining
memcpy_nontemporal() for ZONE_DEVICE.
The initialization time for 64 boot-time 1G hugepages with 4K base
pages was measured as follows:
arch without this patch with this patch reduction
x86-64 195.60 ms 170.40 ms 12.88%
arm64 374.40 ms 237.60 ms 36.54%
Co-developed-by: Li Zhe <lizhe.67@bytedance.com>
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v3:
- Integrate the template-copy optimization into the common helper switch
(suggested by Mike Rapoport)
- Use memcpy() for normal memory and keep memcpy_nontemporal() for
ZONE_DEVICE
- Update performance results for the integrated implementation (measured
by Li Zhe)
- Add Co-developed-by for Li Zhe
---
mm/hugetlb.c | 52 ++--------------------------------------------------
mm/mm_init.c | 10 ++++++----
2 files changed, 8 insertions(+), 54 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 2a94f8f60870..6a613d9c9332 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3319,50 +3319,6 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid)
return true;
}
-/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */
-static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
- struct hstate *h,
- unsigned long start_page_number,
- unsigned long end_page_number)
-{
- enum zone_type zone = folio_zonenum(folio);
- int nid = folio_nid(folio);
- struct page *page = folio_page(folio, start_page_number);
- unsigned long head_pfn = folio_pfn(folio);
- unsigned long pfn, end_pfn = head_pfn + end_page_number;
- unsigned int order = huge_page_order(h);
-
- /*
- * As we marked all tail pages with memblock_reserved_mark_noinit(),
- * we must initialize them ourselves here.
- */
- for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
- __init_single_page(page, pfn, zone, nid);
- prep_compound_tail(page, &folio->page, order);
- set_page_count(page, 0);
- }
-}
-
-static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
- unsigned long pfn,
- struct hstate *h,
- enum zone_type zone, int nid,
- unsigned long nr_pages)
-{
- struct page *head = &folio->page;
-
- /*
- * This is an open-coded prep_compound_page() whereby we avoid
- * walking pages twice by initializing/preparing+freezing them in the
- * same go.
- */
- __init_single_page(head, pfn, zone, nid);
- __folio_set_head(folio);
- set_page_count(head, 0);
- hugetlb_folio_init_tail_vmemmap(folio, h, 1, nr_pages);
- prep_compound_head(head, huge_page_order(h));
-}
-
/*
* memblock-allocated pageblocks might not have the migrate type set
* if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE)
@@ -3420,8 +3376,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
struct folio *folio = (void *)page;
const unsigned long pfn = PHYS_PFN(__pa(m));
const unsigned long nr_pages = pages_per_huge_page(m->hstate);
- const enum zone_type zone = zone_idx(pfn_to_zone(pfn, nid));
- unsigned long nr_struct_pages = nr_pages;
+ const struct zone *zone = pfn_to_zone(pfn, nid);
h = m->hstate;
/*
@@ -3433,10 +3388,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
prev_h = h;
VM_BUG_ON(!hstate_is_gigantic(h));
-
- if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
- nr_struct_pages = VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES;
- hugetlb_folio_init_vmemmap(folio, pfn, h, zone, nid, nr_struct_pages);
+ memmap_init_compound_page(page, pfn, zone, huge_page_order(h));
init_new_hugetlb_folio(folio);
if (vmemmap_optimizable_order(pfn_to_section_compound_order(pfn)))
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 3e645302377e..8bf350d5a695 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -722,16 +722,18 @@ static void __meminit memmap_init_page_from_template(struct page *page,
unsigned long pfn, const struct zone *zone, struct page *template)
{
/*
- * Refresh PFN-dependent fields in the template before the non-temporal
- * copy, so the destination page does not need cached stores after
- * memcpy_nontemporal().
+ * Refresh PFN-dependent fields in the template before copying it, so
+ * the destination page does not need follow-up stores.
*/
set_page_section_from_pfn(template, pfn);
#ifdef WANT_PAGE_VIRTUAL
if (!is_highmem_idx(zone_idx(zone)))
set_page_address(template, __va(pfn << PAGE_SHIFT));
#endif
- memcpy_nontemporal(page, template, sizeof(*page));
+ if (zone_is_zone_device(zone))
+ memcpy_nontemporal(page, template, sizeof(*page));
+ else
+ memcpy(page, template, sizeof(*page));
vmemmap_optimization_verify_zone(page);
}
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 08/11] mm/mm_init: make __init_single_page() static
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (6 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 07/11] mm/hugetlb: use common compound page initialization helper Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 09/11] mm/cma: centralize CMA pageblock initialization Muchun Song
` (2 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
__init_single_page() is only used within mm/mm_init.c, so make it
static.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/mm_init.c | 2 +-
mm/mm_init.h | 2 --
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 8bf350d5a695..a427477483d0 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -617,7 +617,7 @@ void memmap_init_page(struct page *page, enum zone_type zone, int nid)
#endif
}
-void __meminit __init_single_page(struct page *page, unsigned long pfn,
+static __meminit void __init_single_page(struct page *page, unsigned long pfn,
unsigned long zone, int nid)
{
memmap_init_page(page, zone, nid);
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 1e6d2f78afdd..c03413a090eb 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -110,8 +110,6 @@ bool memblock_has_mirror(void);
void memblock_free_all(void);
void memmap_init_page(struct page *page, enum zone_type zone, int nid);
-void __meminit __init_single_page(struct page *page, unsigned long pfn,
- unsigned long zone, int nid);
void __meminit memmap_init_compound_page(struct page *head, unsigned long pfn,
const struct zone *zone, unsigned int order);
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 09/11] mm/cma: centralize CMA pageblock initialization
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (7 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 08/11] mm/mm_init: make __init_single_page() static Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 10/11] mm/cma: localize init_cma_pageblock() to cma.c Muchun Song
2026-11-20 10:08 ` [PATCH v3 11/11] mm/mm_init: move pageblock migratetype init to memmap helpers Muchun Song
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Early CMA reservations currently require callers to maintain CMA state.
This exposes internal bookkeeping requirements and splits responsibility
for the same area between CMA and its users.
Make CMA fully responsible for the state of early reservations. Pages
already handed to a caller remain owned by that caller if activation
fails, so keep their managed-page accounting intact regardless of whether
CMA retains the rest of the area.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v3:
- Restore managed-page accounting for early-reserved pages regardless of
CMA_RESERVE_PAGES_ON_ERROR (reported by Sashiko)
---
mm/cma.c | 30 ++++++++++++++++++------------
mm/hugetlb.c | 8 +++-----
2 files changed, 21 insertions(+), 17 deletions(-)
diff --git a/mm/cma.c b/mm/cma.c
index 61fc59fa70f8..3e4366213b49 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -174,6 +174,10 @@ static void __init cma_activate_area(struct cma *cma)
count = early_pfn[r] - cmr->base_pfn;
bitmap_count = cma_bitmap_pages_to_bits(cma, count);
bitmap_set(cmr->bitmap, 0, bitmap_count);
+
+ for (pfn = cmr->base_pfn; pfn < early_pfn[r];
+ pfn += pageblock_nr_pages)
+ init_cma_pageblock(pfn_to_page(pfn));
}
WARN_ON_ONCE(!pfn_valid(cmr->base_pfn));
@@ -199,17 +203,20 @@ static void __init cma_activate_area(struct cma *cma)
for (r = 0; r < allocrange; r++)
bitmap_free(cma->ranges[r].bitmap);
- /* Expose all pages to the buddy, they are useless for CMA. */
- if (!test_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags)) {
- for (r = 0; r < cma->nranges; r++) {
- unsigned long start_pfn;
+ for (r = 0; r < cma->nranges; r++) {
+ cmr = &cma->ranges[r];
+ end_pfn = r <= allocrange ? early_pfn[r] : cmr->early_pfn;
+ /* Restore managed page accounting for early-reserved pages. */
+ for (pfn = cmr->base_pfn; pfn < end_pfn; pfn++)
+ adjust_managed_page_count(pfn_to_page(pfn), 1);
- cmr = &cma->ranges[r];
- start_pfn = r <= allocrange ? early_pfn[r] : cmr->early_pfn;
- end_pfn = cmr->base_pfn + cmr->count;
- for (pfn = start_pfn; pfn < end_pfn; pfn++)
- free_reserved_page(pfn_to_page(pfn));
- }
+ if (test_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags))
+ continue;
+
+ /* Expose the remaining pages to the buddy. */
+ end_pfn = cmr->base_pfn + cmr->count;
+ for (; pfn < end_pfn; pfn++)
+ free_reserved_page(pfn_to_page(pfn));
}
totalcma_pages -= cma->count;
cma->available_count = cma->count = 0;
@@ -1020,8 +1027,7 @@ bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end)
*
* The caller is responsible for initializing the page structures
* in the area properly, since this just points to memblock-allocated
- * memory. The caller should subsequently use init_cma_pageblock to
- * set the migrate type and CMA stats the pageblocks that were reserved.
+ * memory.
*
* If the CMA area fails to activate later, memory obtained through
* this interface is not handed to the page allocator, this is
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 6a613d9c9332..7de65b540438 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3337,9 +3337,7 @@ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio,
WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio)));
for (i = 0; i < nr_pages; i += pageblock_nr_pages) {
- if (folio_test_hugetlb_cma(folio))
- init_cma_pageblock(folio_page(folio, i));
- else
+ if (!folio_test_hugetlb_cma(folio))
init_pageblock_migratetype(folio_page(folio, i),
MIGRATE_MOVABLE, false);
}
@@ -3402,8 +3400,8 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
* in order to fix confusing memory reports from free(1) and
* other side-effects, like CommitLimit going negative.
*
- * For CMA pages, this is done in init_cma_pageblock
- * (via hugetlb_bootmem_init_migratetype), so skip it here.
+ * For CMA pages, this is done in cma_activate_area(), so skip
+ * it here.
*/
if (hugetlb_early_cma(h))
folio_set_hugetlb_cma(folio);
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 10/11] mm/cma: localize init_cma_pageblock() to cma.c
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (8 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 09/11] mm/cma: centralize CMA pageblock initialization Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
2026-11-20 10:08 ` [PATCH v3 11/11] mm/mm_init: move pageblock migratetype init to memmap helpers Muchun Song
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
Move init_cma_pageblock() from mm_init.c into cma.c so it lives
alongside the CMA code that uses it.
No functional change intended.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/cma.c | 8 ++++++++
mm/mm_init.c | 9 ---------
mm/mm_init.h | 8 --------
3 files changed, 8 insertions(+), 17 deletions(-)
diff --git a/mm/cma.c b/mm/cma.c
index 3e4366213b49..0c97aadcfea4 100644
--- a/mm/cma.c
+++ b/mm/cma.c
@@ -31,6 +31,7 @@
#include <linux/kmemleak.h>
#include <linux/hugetlb.h>
#include <trace/events/cma.h>
+#include <linux/page-isolation.h>
#include "internal.h"
#include "cma.h"
@@ -149,6 +150,13 @@ bool cma_validate_zones(struct cma *cma)
return true;
}
+static void __init init_cma_pageblock(struct page *page)
+{
+ init_pageblock_migratetype(page, MIGRATE_CMA, false);
+ adjust_managed_page_count(page, pageblock_nr_pages);
+ page_zone(page)->cma_pages += pageblock_nr_pages;
+}
+
static void __init cma_activate_area(struct cma *cma)
{
unsigned long pfn, end_pfn, early_pfn[CMA_MAX_RANGES];
diff --git a/mm/mm_init.c b/mm/mm_init.c
index a427477483d0..e0686b87fdb1 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2202,15 +2202,6 @@ void __init init_cma_reserved_pageblock(struct page *page)
adjust_managed_page_count(page, pageblock_nr_pages);
page_zone(page)->cma_pages += pageblock_nr_pages;
}
-/*
- * Similar to above, but only set the migrate type and stats.
- */
-void __init init_cma_pageblock(struct page *page)
-{
- init_pageblock_migratetype(page, MIGRATE_CMA, false);
- adjust_managed_page_count(page, pageblock_nr_pages);
- page_zone(page)->cma_pages += pageblock_nr_pages;
-}
#endif
/*
diff --git a/mm/mm_init.h b/mm/mm_init.h
index c03413a090eb..9ee9bcb8e672 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -40,14 +40,6 @@ struct zone *pfn_to_zone(unsigned long pfn, int nid);
void init_cma_reserved_pageblock(struct page *page);
#endif
-#ifdef CONFIG_CMA
-void init_cma_pageblock(struct page *page);
-#else
-static inline void init_cma_pageblock(struct page *page)
-{
-}
-#endif
-
/* Memory initialisation debug and verification */
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
DECLARE_STATIC_KEY_TRUE(deferred_pages);
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v3 11/11] mm/mm_init: move pageblock migratetype init to memmap helpers
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
` (9 preceding siblings ...)
2026-11-20 10:08 ` [PATCH v3 10/11] mm/cma: localize init_cma_pageblock() to cma.c Muchun Song
@ 2026-11-20 10:08 ` Muchun Song
10 siblings, 0 replies; 12+ messages in thread
From: Muchun Song @ 2026-11-20 10:08 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Oscar Salvador, Mike Rapoport
Cc: Liam R . Howlett, Vlastimil Babka, Michal Hocko, Muchun Song,
Li Zhe, Andrew Popple, Balbir Singh, Lorenzo Stoakes,
Suren Baghdasaryan, James Houghton, Lance Yang, Qi Zheng,
linux-mm, linux-kernel, Muchun Song
ZONE_DEVICE memmap initialization already initializes pageblock
migratetype after all struct pages have been initialized, while hugetlb
bootmem folios have their own duplicate bootmem-specific initialization.
Move the MIGRATE_MOVABLE pageblock initialization into the frozen memmap
initialization helpers instead, so the ZONE_DEVICE and hugetlb bootmem
paths can share the same initialization.
The old hugetlb_bootmem_init_migratetype() skipped CMA folios via
folio_test_hugetlb_cma(), but the shared helper always initializes
pageblocks as MIGRATE_MOVABLE. This is safe because cma_activate_area()
later overrides the migratetype for CMA pageblocks.
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
v2:
- Fix the helper name in the commit message (reported by Sashiko).
---
mm/hugetlb.c | 25 -------------------------
mm/mm_init.c | 11 ++++++-----
2 files changed, 6 insertions(+), 30 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 7de65b540438..72839e98f998 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3319,30 +3319,6 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid)
return true;
}
-/*
- * memblock-allocated pageblocks might not have the migrate type set
- * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE)
- * here, or MIGRATE_CMA if this was a page allocated through an early CMA
- * reservation.
- *
- * In case of vmemmap optimized folios, the tail vmemmap pages are mapped
- * read-only, but that's ok - for sparse vmemmap this does not write to
- * the page structure.
- */
-static void __init hugetlb_bootmem_init_migratetype(struct folio *folio,
- struct hstate *h)
-{
- unsigned long nr_pages = pages_per_huge_page(h), i;
-
- WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio)));
-
- for (i = 0; i < nr_pages; i += pageblock_nr_pages) {
- if (!folio_test_hugetlb_cma(folio))
- init_pageblock_migratetype(folio_page(folio, i),
- MIGRATE_MOVABLE, false);
- }
-}
-
static void __init prep_and_add_bootmem_folios(struct hstate *h,
struct list_head *folio_list)
{
@@ -3350,7 +3326,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
struct folio *folio, *tmp_f;
list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
- hugetlb_bootmem_init_migratetype(folio, h);
/* Subdivide locks to achieve better parallel performance */
spin_lock_irqsave(&hugetlb_lock, flags);
account_new_hugetlb_folio(h, folio);
diff --git a/mm/mm_init.c b/mm/mm_init.c
index e0686b87fdb1..a340ba7f0a74 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -757,6 +757,9 @@ void __meminit memmap_init_compound_page(struct page *head, unsigned long pfn,
for (unsigned long i = 1; i < nr_pages; i++)
memmap_init_page_from_template(head + i, pfn + i, zone, &template);
prep_compound_head(head, order);
+
+ pageblock_migratetype_init_range(pfn, 1UL << order, MIGRATE_MOVABLE,
+ /* isolate */ false, /* atomic */ false);
}
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
@@ -1110,6 +1113,9 @@ static void memmap_init_zone_device_pages(unsigned long start, unsigned long end
if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
cond_resched();
}
+
+ pageblock_migratetype_init_range(start, end - start, MIGRATE_MOVABLE,
+ /* isolate */ false, /* atomic */ false);
}
void __ref memmap_init_zone_device(struct zone *zone,
@@ -1147,13 +1153,8 @@ void __ref memmap_init_zone_device(struct zone *zone,
memmap_init_compound_page(head, pfn, zone, pgmap->vmemmap_shift);
memmap_init_zone_device_folio(page_folio(head), pgmap);
- if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
- cond_resched();
}
out:
- pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE,
- /* isolate */ false, /* atomic */ false);
-
pr_debug("%s initialised %lu pages in %ums\n", __func__,
nr_pages, jiffies_to_msecs(jiffies - start));
}
--
2.54.0
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-11-20 10:10 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20 10:08 [PATCH v3 00/11] mm: Unify memmap initialization for compound pages Muchun Song
2026-11-20 10:08 ` [PATCH v3 01/11] mm/mm_init: verify zones for template-initialized pages Muchun Song
2026-11-20 10:08 ` [PATCH v3 02/11] mm/mm_init: factor out common struct page initialization Muchun Song
2026-11-20 10:08 ` [PATCH v3 03/11] mm/mm_init: simplify zone device " Muchun Song
2026-11-20 10:08 ` [PATCH v3 04/11] mm/mm_init: factor out compound " Muchun Song
2026-11-20 10:08 ` [PATCH v3 05/11] mm/hugetlb: drop boot-time HVO handling for gigantic folios Muchun Song
2026-11-20 10:08 ` [PATCH v3 06/11] mm/hugetlb: initialize all struct pages for bootmem hugepages Muchun Song
2026-11-20 10:08 ` [PATCH v3 07/11] mm/hugetlb: use common compound page initialization helper Muchun Song
2026-11-20 10:08 ` [PATCH v3 08/11] mm/mm_init: make __init_single_page() static Muchun Song
2026-11-20 10:08 ` [PATCH v3 09/11] mm/cma: centralize CMA pageblock initialization Muchun Song
2026-11-20 10:08 ` [PATCH v3 10/11] mm/cma: localize init_cma_pageblock() to cma.c Muchun Song
2026-11-20 10:08 ` [PATCH v3 11/11] mm/mm_init: move pageblock migratetype init to memmap helpers Muchun Song
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®