From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D3593D7A18 for ; Mon, 20 Jul 2026 09:32:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784539925; cv=none; b=qnuvUnElEB62weQJWCmYMi0TcH7Gha6qAXRNJgElMk5wGu1PLs5wyvR42wS9+df6xhVzjiM/k9nnUtlTbAMQ+rQQMbnxCcZZhhcP9G/bAQlykEVcz6D+62lbFACKHjO9zJaabw9Ne9G6OFsmknXfaYIcNp8MhikkS3JAkIauqFg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784539925; c=relaxed/simple; bh=UlVatc8sakduVZEP4OOQmq1leBTxdEvteeXD1rofskI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XIaeyvhZ6EteNHZnV5QDGP40gUj13Rm2j/jA4OiX34HBNtkxAzgHZd/GjQof3l2s/Fyz8165hPeC9+JtPOuZf7ii765NfXg/41vdf9vShz0E3KP209Yoy16j9EgabhK/A52Pn2/VEg5fjl1UVRqa6P0IFaL0WLBMIRKtE7SSSms= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=NhzUcQfk; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="NhzUcQfk" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2cf6d65d8a7so7781375ad.0 for ; Mon, 20 Jul 2026 02:32:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784539923; x=1785144723; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=T/mMY7wFDptG7mxg7a2Zuzc5cMIttYTPNMUojd8S+fU=; b=NhzUcQfklXJlSf+NWPBID41peYOD9S12DCg6uxdhsYQ6RAfPtbC1cO2VWD/a+/DYYD EQeYbgVpyxka10hbJnl+2o2+VI+YsfGRwXjSQV37wy/KNSlqHxxwwkba4sLJ7Eat7+HL tXie9/bROW3MuUOrIymDAMalHF1SBmcHfBuv/rdctr52AdEcTQqqeT9jicrhj2EUbRBv 8YpZrsI9l3HEGM70HIFapgK6G7khbgnj0gXwWh/yC5L9ReouUMj3MNPYuiTAe7swp0J7 FfPCxGiz+eqOjbeGCEJYRux9RQzAf7m7TXojN+4l0kd/CpOSB54sC8k8Tv/gEIM9Awca k+kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784539923; x=1785144723; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=T/mMY7wFDptG7mxg7a2Zuzc5cMIttYTPNMUojd8S+fU=; b=dlI6mNiLfTdNjqKaUo4Ra4xXURMimz010sq5Njdn53cC/GouW5bcg+b6ugZAK2sB6Z IQCX44LLVhydy4rzV5cbPmZjfS5vYSVebCOmZd7npjASzduWnhjTSFli2/KInjTnhT2l 0/2PDyy4nshwIN9EltHWXdvq4Ro8p+c1PKGE55jZrCrHlpIunTqblBVa+KVmsvou7J26 8vkMz0GgxOhQGKBvy2ttCIBkgdi+b8h0A//aTFH6GsLSV401lIbN6a2tOcdqK6tna/cs mW7svrjrGcGT92ooJZsG3FihjEddo1neAR4HfGSizI+1DVo0fIOxfT22Mlf6BumhL4JU ql7w== X-Forwarded-Encrypted: i=1; AHgh+RrnUQpYWCk6xih93z3Sz3eOBdPez17zoCcN2koyCR0JBKruOd2jUuGLk89g4FBBDjP++UjqzZVPLjqpbzg=@vger.kernel.org X-Gm-Message-State: AOJu0YweKPog4GvBmaLnC6z3ITCwrMfu70Bnh3E1p94RgEf1rIUeP49X jv7pkKJM2M2ruU6gwPhEEZjmbjWFWHC+rcXz9WZbzsvH+dj8nYlOb/P/o5vWXyrk/YE= X-Gm-Gg: AfdE7cnkr+DYXZDve7vdLXPr6cnaqgemX7cEYbA221HDj6+D2HjIw8hkbCRq+qmvUmj WTErn1v5w36oZ+dpMz5ha+QN+JMydIc5IsKPuDbQG9vFs8zAPCy9ajqivAmDHJhCFg9nTmPht6K UgjW+GCrvnClelM1RP2JTQV7aY7Oz0DjQJhyBLvrNIRfUY8BK30nCWsMdti0RY2NIogDNCBUFkS q3OPYWr8rn83EuxTpKzevN4mOSRZoYTlPtKi+Wv91Y7Lq69LKs5tCu3FKbJYATi/HF4GmRRZygC FYoIhOIgQheZ1ypvfCmLtFmspUinTFniPCKrDlBpZnEtMZC1Eug9O2mAeXlWBbcnQ/Nqp8pIagf UkjH7AvwqUcsZ/mARmZXfsu3UK3d18JoZaWjVinDJ6zU9F6S3G/uixLqcKzzXPe4PSUmiD4fqwn YkpvOUmB91B52rPmhWSDtyon0rr6DfQGeqbg== X-Received: by 2002:a17:903:41c3:b0:2c7:1200:bd7e with SMTP id d9443c01a7336-2cf3486b0b1mr144824815ad.18.1784539922678; Mon, 20 Jul 2026 02:32:02 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.101]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf347a7c9fsm53136455ad.84.2026.07.20.02.31.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2026 02:32:01 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v2 03/17] mm/mm_init: skip initializing shared vmemmap tail pages Date: Mon, 20 Jul 2026 17:31:13 +0800 Message-ID: <20260720093127.540540-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260720093127.540540-1-songmuchun@bytedance.com> References: <20260720093127.540540-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit memmap_init_range() initializes every struct page in the target range. For compound pages with vmemmap optimization, the tail struct pages are backed by a shared vmemmap page. Initializing those tail struct pages would overwrite the shared vmemmap page contents, requiring users such as HugeTLB to restore the metadata afterwards. Track the compound order for HVO-backed sections and use that metadata to detect struct pages that fall into the shared tail vmemmap range. Skip those shared tail pages in memmap_init_range(), then initialize pageblock migratetypes for the processed range with a helper after the per-page initialization loop. The !SPARSEMEM __pfn_to_section() stub is needed only for the build: memmap_init_range() references __pfn_to_section() after checking pfn_vmemmap_optimizable(), and !SPARSEMEM builds still have to compile that code even though pfn_vmemmap_optimizable() folds to false. This is a preparatory change for consolidating handling across users of vmemmap optimization, and it also avoids redundant initialization of shared tail vmemmap pages during early boot. Signed-off-by: Muchun Song --- v2: - Fold section order tracking into the first user instead of keeping a standalone API-only patch (suggested by Mike Rapoport) - Rename page_vmemmap_optimizable() to pfn_vmemmap_optimizable() and pass a PFN directly (suggested by Mike Rapoport) - Initialize pageblock migratetypes from a helper after the per-page loop (suggested by Mike Rapoport) - Use a 1G PFN chunk for cond_resched() in the pageblock helper (suggested by Mike Rapoport) - Guard section_order() with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP so it returns 0 when HVO is disabled and lets the compiler optimize the code as much as possible (suggested by Mike Rapoport) - Explain why the !SPARSEMEM __pfn_to_section() stub belongs here (suggested by Mike Rapoport) --- include/linux/mmzone.h | 14 ++++++++++++++ mm/mm_init.c | 33 +++++++++++++++++---------------- mm/sparse.h | 23 +++++++++++++++++++++++ 3 files changed, 54 insertions(+), 16 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 82b0155d886f..2a32101d55e6 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2011,6 +2011,14 @@ struct mem_section { unsigned long section_mem_map; struct mem_section_usage *usage; +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + /* + * Normally, sections hold regular (order-0) pages. However, for + * sections with HVO enabled, this tracks the compound page order + * to enable deduplication of redundant vmemmap pages. + */ + unsigned int order; +#endif #ifdef CONFIG_PAGE_EXTENSION /* * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use @@ -2365,8 +2373,14 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) #endif #else +struct mem_section; + #define sparse_vmemmap_init_nid_early(_nid) do {} while (0) #define pfn_in_present_section pfn_valid +static inline struct mem_section *__pfn_to_section(unsigned long pfn) +{ + return NULL; +} #endif /* CONFIG_SPARSEMEM */ /* diff --git a/mm/mm_init.c b/mm/mm_init.c index ddda9d6837f3..dcb757b36902 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -677,19 +678,17 @@ static inline void fixup_hashdist(void) static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ -#if defined(CONFIG_ZONE_DEVICE) || defined(CONFIG_DEFERRED_STRUCT_PAGE_INIT) static __meminit void pageblock_migratetype_init_range(unsigned long pfn, - unsigned long nr_pages, int migratetype, bool atomic) + unsigned long nr_pages, int migratetype, bool isolate, bool atomic) { const unsigned long end = pfn + nr_pages; for (pfn = pageblock_align(pfn); pfn < end; pfn += pageblock_nr_pages) { - init_pageblock_migratetype(pfn_to_page(pfn), migratetype, false); - if (!atomic && IS_ALIGNED(pfn, PAGES_PER_SECTION)) + init_pageblock_migratetype(pfn_to_page(pfn), migratetype, isolate); + if (!atomic && IS_ALIGNED(pfn, PFN_DOWN(SZ_1G))) cond_resched(); } } -#endif #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) @@ -884,6 +883,13 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone } } + if (pfn_vmemmap_optimizable(pfn)) { + unsigned int order = section_order(__pfn_to_section(pfn)); + + pfn = min(ALIGN(pfn, 1UL << order), end_pfn); + continue; + } + page = pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); if (context == MEMINIT_HOTPLUG) { @@ -895,18 +901,13 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone __SetPageOffline(page); } - /* - * Usually, we want to mark the pageblock MIGRATE_MOVABLE, - * such that unmovable allocations won't be scattered all - * over the place during system boot. - */ - if (pageblock_aligned(pfn)) { - init_pageblock_migratetype(page, migratetype, - isolate_pageblock); + if (pageblock_aligned(pfn)) cond_resched(); - } pfn++; } + + pageblock_migratetype_init_range(start_pfn, pfn - start_pfn, migratetype, + isolate_pageblock, false); } static void __init memmap_init_zone_range(struct zone *zone, @@ -1111,7 +1112,7 @@ void __ref memmap_init_zone_device(struct zone *zone, compound_nr_pages(pfn, altmap, pgmap)); } - pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false); + pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false, false); pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); @@ -1923,7 +1924,7 @@ static void __init deferred_free_pages(unsigned long pfn, if (!nr_pages) return; - pageblock_migratetype_init_range(pfn, nr_pages, mt, true); + pageblock_migratetype_init_range(pfn, nr_pages, mt, false, true); page = pfn_to_page(pfn); diff --git a/mm/sparse.h b/mm/sparse.h index 95aa031213f2..030248030dc7 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -51,6 +51,29 @@ static inline void __section_mark_present(struct mem_section *ms, static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static inline unsigned int section_order(const struct mem_section *section) +{ + return section->order; +} +#else +static inline unsigned int section_order(const struct mem_section *section) +{ + return 0; +} +#endif + +static inline bool pfn_vmemmap_optimizable(unsigned long pfn) +{ + const unsigned int order = section_order(__pfn_to_section(pfn)); + const unsigned long nr_pages = 1UL << order; + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >= VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES; +} + /* * mm/sparse-vmemmap.c */ -- 2.54.0