From: Muchun Song <songmuchun@bytedance.com>
To: Andrew Morton <akpm@linux-foundation.org>,
David Hildenbrand <david@kernel.org>,
Oscar Salvador <osalvador@suse.de>,
Mike Rapoport <rppt@kernel.org>
Cc: Li Zhe <lizhe.67@bytedance.com>,
Andrew Popple <apopple@nvidia.com>,
Balbir Singh <balbirs@nvidia.com>,
Lorenzo Stoakes <ljs@kernel.org>,
"Liam R . Howlett" <liam@infradead.org>,
Vlastimil Babka <vbabka@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Muchun Song <muchun.song@linux.dev>,
James Houghton <jthoughton@google.com>,
Lance Yang <lance.yang@linux.dev>, Qi Zheng <qi.zheng@linux.dev>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Muchun Song <songmuchun@bytedance.com>
Subject: [PATCH v2 02/14] mm/hugetlb: optimize bootmem hugepage tail vmemmap init
Date: Fri, 13 Nov 2026 11:34:30 +0800 [thread overview]
Message-ID: <20261113033442.62970-3-songmuchun@bytedance.com> (raw)
In-Reply-To: <20261113033442.62970-1-songmuchun@bytedance.com>
From: Li Zhe <lizhe.67@bytedance.com>
Boot-time HugeTLB allocation has to initialize tail struct pages itself
when HugeTLB vmemmap optimization (HVO) is unavailable or fails. For
gigantic pages, this can cover a large number of tail pages. Most state
installed by __init_single_page() and prep_compound_tail() is identical
for every tail page in the same huge page; only PFN-dependent fields need
refreshing.
Initialize the first tail page normally, then initialize the remaining
tails with hugetlb_init_tail_page(). This copies the already
initialized first tail page with memcpy() and refreshes only the
PFN-dependent fields in the destination page, while avoiding the full
__init_single_page() and prep_compound_tail() sequence for state that is
identical across tail pages in the same huge page.
The fallback path was measured on a v7.4-rc2 base with a temporary
jiffies-based printk modeled after the existing timing in
memmap_init_zone_device(), using 4K base pages and 64 boot-time 1G
hugepages. HVO is unavailable on arm64; on x86-64,
hugetlb_free_vmemmap=off was used to exercise the fallback path.
arch base with this patch reduction
arm64 360.8 ms 256.0 ms 29.05%
x86-64 171.0 ms 165.0 ms 3.51%
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
mm/hugetlb.c | 23 ++++++++++++++++++-----
mm/mm_init.c | 5 ++++-
2 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 88c9aa700533..ed3af959f615 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3266,6 +3266,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
enum zone_type zone = folio_zonenum(folio);
int nid = folio_nid(folio);
struct page *page = folio_page(folio, start_page_number);
+ struct page template;
unsigned long head_pfn = folio_pfn(folio);
unsigned long pfn, end_pfn = head_pfn + end_page_number;
unsigned int order = huge_page_order(h);
@@ -3274,11 +3275,23 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
* As we marked all tail pages with memblock_reserved_mark_noinit(),
* we must initialize them ourselves here.
*/
- for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
- __init_single_page(page, pfn, zone, nid);
- prep_compound_tail(page, &folio->page, order);
- set_page_count(page, 0);
- }
+ if (start_page_number >= end_page_number)
+ return;
+
+ pfn = head_pfn + start_page_number;
+
+ /*
+ * Seed a template from the first real tail page and refresh only
+ * PFN-dependent fields before copying it.
+ */
+ __init_single_page(page, pfn, zone, nid);
+ prep_compound_tail(page, &folio->page, order);
+ set_page_count(page, 0);
+ memcpy(&template, page, sizeof(template));
+
+ for (pfn++; pfn < end_pfn; pfn++)
+ memmap_init_page_from_template(pfn_to_page(pfn), pfn,
+ folio_zone(folio), &template);
}
static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 3988c19699bc..639880afe0ee 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -720,7 +720,10 @@ void __meminit memmap_init_page_from_template(struct page *page,
if (!is_highmem_idx(zone_idx(zone)))
set_page_address(template, __va(pfn << PAGE_SHIFT));
#endif
- memcpy_nontemporal(page, template, sizeof(*page));
+ if(zone_is_zone_device(zone))
+ memcpy_nontemporal(page, template, sizeof(*page));
+ else
+ memcpy(page, template, sizeof(*page));
}
struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid)
--
2.54.0
next prev parent reply other threads:[~2026-11-13 3:35 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-13 3:34 [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Muchun Song
2026-11-13 3:34 ` [PATCH v2 01/14] mm/mm_init: factor out memmap_init_page_from_template() Muchun Song
2026-11-16 4:40 ` Lance Yang
2026-11-13 3:34 ` Muchun Song [this message]
2026-11-13 3:34 ` [PATCH v2 03/14] mm/mm_init: verify zones for template-initialized pages Muchun Song
2026-11-16 8:41 ` Lance Yang
2026-11-13 3:34 ` [PATCH v2 04/14] mm/mm_init: factor out common struct page initialization Muchun Song
2026-11-13 3:34 ` [PATCH v2 05/14] mm/mm_init: simplify zone device " Muchun Song
2026-11-13 3:34 ` [PATCH v2 06/14] mm/mm_init: factor out frozen compound " Muchun Song
2026-11-13 3:34 ` [PATCH v2 07/14] mm/hugetlb: drop boot-time HVO handling for gigantic folios Muchun Song
2026-11-13 3:34 ` [PATCH v2 08/14] mm/hugetlb: initialize all struct pages for bootmem hugepages Muchun Song
2026-11-13 3:34 ` [PATCH v2 09/14] mm/hugetlb: use common compound page initialization helper Muchun Song
2026-11-13 3:34 ` [PATCH v2 10/14] mm/mm_init: make __init_single_page() static Muchun Song
2026-11-13 3:34 ` [PATCH v2 11/14] mm/mm_init: make memmap_init_page_from_template() static Muchun Song
2026-11-13 3:34 ` [PATCH v2 12/14] mm/cma: centralize CMA pageblock initialization Muchun Song
2026-11-13 3:34 ` [PATCH v2 13/14] mm/cma: localize init_cma_pageblock() to cma.c Muchun Song
2026-11-13 3:34 ` [PATCH v2 14/14] mm/mm_init: move pageblock migratetype init to memmap helpers Muchun Song
2026-11-16 4:20 ` [PATCH v2 00/14] mm: Unify memmap initialization for compound pages Lance Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261113033442.62970-3-songmuchun@bytedance.com \
--to=songmuchun@bytedance.com \
--cc=akpm@linux-foundation.org \
--cc=apopple@nvidia.com \
--cc=balbirs@nvidia.com \
--cc=david@kernel.org \
--cc=jthoughton@google.com \
--cc=lance.yang@linux.dev \
--cc=liam@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lizhe.67@bytedance.com \
--cc=ljs@kernel.org \
--cc=mhocko@suse.com \
--cc=muchun.song@linux.dev \
--cc=osalvador@suse.de \
--cc=qi.zheng@linux.dev \
--cc=rppt@kernel.org \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®