From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3788E449B01 for ; Wed, 19 Aug 2026 09:53:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133221; cv=none; b=HIcXkWWo98xGog48N/WyGfQ63Tg0mydiTbdqPrW9UcH60kVVlChjwBk864FOm3soStajqEHHxMvH7hYccdzeTp0f3tWj8OT7N07iV3v+uLyGvkMzsGLjYMTob2hIfPEQipCZtOck49CMKrYhtGVZHplk9zILa6HbIyrQ77SvMj0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787133221; c=relaxed/simple; bh=Z7TQ1SBJopSg2m/uDqzifmTewvEx3Vp99MkMw8QQORE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vnx57UJcuufpVJvtww1zDQuJ5DhgAfcQ8VN8TyJbGFfvN4x+tkkycVCsQMTaBp5yZL8BLtuCXtpveCmYdpTdj3FUPLGlaaEKYJUXAWW0buqF+wn399sNz4u+fXABhC8tu7Y+efiRQMXIbkDPXWrw19sW8WXeGYq2M6EAf3AYTgM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=cYa4WKUo; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="cYa4WKUo" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2d5655cc850so9450375ad.3 for ; Wed, 19 Aug 2026 02:53:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1787133210; x=1787738010; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=671AwO9cxknmM2yc6/2dzWMjTOAP7N0nlaeCKAVGNAc=; b=cYa4WKUorAufLaPimnzekSMXp7vMGVNiEQvAiw087kp1UIlXwimwJLpdPnVYbk5jBL O97wFMM+ctyjhtQg8Ib+cBMmC+6zNnv/oBP2xAOyFTYGwG+WrdFG9DCKu7/QxWlVI3wt exqV8NoLTs5Brpady2ICfELKvhSNlm8S9ktk3DBtGg/Nu9h7AzfXa3Qw4fKahguyb//c QGckeo6Pz4NxYCgwTo0x5RxCqak96PLikEbgJa4qeULT8rWslPVHX1ZGjQRAaNfuJOCB A2MJTmqKBdFIDrbCBpcyypDOHNECFY/bR7vDZtme0/0r7UTevFa9WO394tWfylhzom4J BqAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787133210; x=1787738010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=671AwO9cxknmM2yc6/2dzWMjTOAP7N0nlaeCKAVGNAc=; b=nwe7cTU8cB1IzF50TuZqrbfhwBy1kGh2STG35/izOPo8tAEemsANA/65bL9FjhuNAq qYYsEwa1/3VmzoPByQrmaefddcda2prtjZOrKuo7IJ35X0d/mVvPWUlQMig7pA4HTUkE BrWWIf2JUCufO69aUYO1OEMDqph/F2m0xRlDOaUsdKNima+gxwEsjzbZRrY8Ymm45+6p WLx7tg6ZXoE7O+fmLtGfNy4ayKMM9bJw5HoCMMAaOMiDtgWhVHN+/bt2im+7t/H2OD97 Jv2mQw6rDL23iros3IN5vIdnDfdUJtbpG/FWea5ggbxu9olArJm2QFu9eALdHTR8OQDI IYEQ== X-Forwarded-Encrypted: i=1; AHgh+RqRrx1dD5r2lVqeYWmAnjnPgFKRgmTbuvJgG5CgVGJkYAryup77+WfpUVmuWSEwPH7jlBeFg7azm2DdHqQ=@vger.kernel.org X-Gm-Message-State: AFuF++lNAC4AvW92KZnfzZpU9nRinf4watUxhDqI0bJDo3AUC0iRC9Ty GnGLAZy3YzXMrnzjM9Pmrl5Y+dYgG7GI7N0HfkcBeh3uuiVquOuQP8amvAtU6j3v9us= X-Gm-Gg: AR+sD12E/12XTqA76UlaPuADXmk+qkwRdkx4ZfTnMzYU2F2vr3shZWeqK5LzLCOOz/Q gq7Cy21fu7Z/OptHojef/PnjT5HMuzlZU8F9F7RLH0PPRfmJ2QUk5LCvMUo2e1sPIBQqQ+lFUSX 21HKp1UUjfoDOiT0KR7/n6KCA/LON+TTKCD6m/gawabzeh3huNHTxaAKV0zcO57QvkO+hcl1HyR lKRxvVi9FJ40lQcFT/+GWHFST0YB+kW3nzPATiZyfEpa5ugprYTKAdrkUZNS157EFqPPIPXzAH/ 3czcq//9kQiPkxRdHMaJbxrV3ZF4ITt0a8PUAP7F0Fb4wcooWUYga+gLX7b92trdAW8CZsgMJ1Z n+/lwOsya618p4g245eAIVwL/BOrxaE6+n29McZYxiwqxoPD5mPA/AA20AEzt4QUqU5BaDuGhKs P5UCEygZFCiUJlLyA5EHK+RR/KBPSwZHDgN7Rae0J+679x3nw6baGvFwHprzj2p6gOW0YQ1P2l4 ImwIiSQEp1jthOLDVVK1Rjqug== X-Received: by 2002:a17:902:cf08:b0:2d0:cc92:f7a3 with SMTP id d9443c01a7336-2d5fd5ec972mr66156725ad.2.1787133209699; Wed, 19 Aug 2026 02:53:29 -0700 (PDT) Received: from G6L4RL2QG9.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d5c1e5434esm22504235ad.51.2026.08.19.02.53.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 19 Aug 2026 02:53:28 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , "Liam R . Howlett" , Suren Baghdasaryan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v4 08/17] mm/sparse-vmemmap: support section-based vmemmap optimization Date: Wed, 19 Aug 2026 17:51:30 +0800 Message-ID: <20260819095140.17252-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260819095140.17252-1-songmuchun@bytedance.com> References: <20260819095140.17252-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Teach sparse-vmemmap population code to use the compound page order when deciding whether a vmemmap page can be optimized. With this information, the common sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages directly instead of relying on HugeTLB-specific handling. This centralizes vmemmap optimization logic in the sparse-vmemmap code, based on section metadata, and prepares for sharing the same mechanism across different users of vmemmap optimization, including HugeTLB and DAX. Signed-off-by: Muchun Song --- v4: - Move section_nr_vmemmap_pages() outside CONFIG_MEMORY_HOTPLUG to fix CONFIG_MEMORY_HOTPLUG=n builds (reported by kernel test robot) v2: - Keep vmemmap accounting and population logic in sparse-vmemmap.c (suggested by Mike Rapoport) - Move vmemmap_get_tail() before its first use instead of adding only a forward declaration in the previous patch (suggested by Mike Rapoport) - Simplify the PMD path handling for HVO-covered sections --- mm/sparse-vmemmap.c | 88 ++++++++++++++++++++++++++++----------------- mm/sparse.c | 4 +-- mm/sparse.h | 7 ++++ 3 files changed, 65 insertions(+), 34 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b770fe2428fd..737d50bdd3ef 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -186,6 +186,11 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone * return tail; } +#else +static inline struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +{ + return NULL; +} #endif static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, @@ -193,12 +198,24 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in unsigned long ptpfn, unsigned long flags) { pte_t *pte = pte_offset_kernel(pmd, addr); + unsigned long pfn = page_to_pfn((struct page *)addr); + if (pte_none(ptep_get(pte))) { pte_t entry; - void *p; + + if (vmemmap_optimizable_pfn(pfn) && ptpfn == (unsigned long)-1) { + unsigned int order = pfn_to_section_order(pfn); + struct zone *zone = pfn_to_zone(pfn, node); + struct page *page = vmemmap_get_tail(order, zone); + + if (!page) + return NULL; + ptpfn = page_to_pfn(page); + } if (ptpfn == (unsigned long)-1) { - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + void *p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) return NULL; ptpfn = PHYS_PFN(__pa(p)); @@ -217,7 +234,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in } entry = pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); - } + } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) + return NULL; return pte; } @@ -406,6 +424,9 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, pmd_t *pmd; for (addr = start; addr < end; addr = next) { + unsigned long pfn = page_to_pfn((struct page *)addr); + const struct mem_section *ms = __pfn_to_section(pfn); + next = pmd_addr_end(addr, end); pgd = vmemmap_pgd_populate(addr, node); @@ -421,7 +442,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return -ENOMEM; pmd = pmd_offset(pud, addr); - if (pmd_none(pmdp_get(pmd))) { + if (pmd_none(pmdp_get(pmd)) && !section_vmemmap_optimizable(ms)) { void *p; p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); @@ -439,8 +460,11 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, */ return -ENOMEM; } - } else if (vmemmap_check_pmd(pmd, node, addr, next)) + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + if (WARN_ON_ONCE(section_vmemmap_optimizable(ms))) + return -EOPNOTSUPP; continue; + } if (vmemmap_populate_basepages(addr, next, node, altmap)) return -ENOMEM; } @@ -620,6 +644,33 @@ void __init sparse_init_subsection_map(void) sparse_init_subsection_map_range(start, end - start); } +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + const struct mem_section *ms = __pfn_to_section(pfn); + const int order = pgmap ? pgmap->vmemmap_shift : section_order(ms); + const int vmemmap_pages = pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZATION_PAGES; + const unsigned long pages_per_compound = 1UL << order; + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); + VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); + + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(ms)) + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); + + if (order < PFN_SECTION_SHIFT) { + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); + return vmemmap_pages * nr_pages / pages_per_compound; + } + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); + + if (IS_ALIGNED(pfn, pages_per_compound)) + return vmemmap_pages; + + return 0; +} + #ifdef CONFIG_MEMORY_HOTPLUG /* Mark all memory sections within the pfn range as online */ @@ -648,33 +699,6 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) } } -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) -{ - const struct mem_section *ms = __pfn_to_section(pfn); - const int order = pgmap ? pgmap->vmemmap_shift : section_order(ms); - const int vmemmap_pages = pgmap ? VMEMMAP_RESERVE_NR : VMEMMAP_OPTIMIZATION_PAGES; - const unsigned long pages_per_compound = 1UL << order; - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); - VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(ms)) - return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); - - if (order < PFN_SECTION_SHIFT) { - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; - } - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); - - if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; - - return 0; -} - static struct page * __meminit populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) diff --git a/mm/sparse.c b/mm/sparse.c index c84b4c7b8c70..e6cb67ca9c8d 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -305,8 +305,8 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page), - PAGE_SIZE)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); sparse_init_early_section(nid, map, pnum, 0); } } diff --git a/mm/sparse.h b/mm/sparse.h index 02ed0f34eac8..1569774270fb 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -111,8 +111,15 @@ static inline void sparse_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else static inline void sparse_init_subsection_map(void) {} +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); +} #endif /* CONFIG_SPARSEMEM_VMEMMAP */ #endif /* __MM_SPARSE_H */ -- 2.54.0