From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 265433D75D3 for ; Tue, 4 Aug 2026 03:56:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785815781; cv=none; b=hDmC1aVk6GvecrP809BXTEg37qzd1uQEdlrfCCDxVrj7ZTkBPAdNQsBfFmKNUdupHqZ16cOabxDAwXzP3R/+e67uweDvDxdSXQx5dC5OaQb2fKH1F8DNd0XnWy5NGL/lS774m2ZldRYwCylfc7WEe7BI/IUlGgma1hiRumSBaLU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785815781; c=relaxed/simple; bh=LlrwP37Q9dNw/uNASIwodbu8P2owhkU3Wtc0xAgHy54=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EvzE5ZpRQ6NLLl+Q/xLcAlR3Wnu8A6wbCq1k1RkqPNNSnjYlDT8q9PBbHaVc7ynWP2+aq2Y7aHIcK/2ykb54UCNb1vjK6rOEenka4rbpsDgDngRZZVM61hwkW4UwIeyIABbY32bc/ZZMf7WYJAEOz4c1Kb9tAQp1TG7GyOXsw+s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=bkryH+7C; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="bkryH+7C" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2ce98cb8165so7067405ad.1 for ; Mon, 03 Aug 2026 20:56:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1785815779; x=1786420579; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uZCX0uRt3bq26YcZIyKc0yaDQHyGPM1xGVsxvUVCVRw=; b=bkryH+7ClhxRkymQz6ZIWtWI/GCGRBtvcxCk/rbWYdA57rxxasXPeuX499FwAuhn/2 oN/oCgikFmqu1JWB6Xw8L6GYn60ob34KqCVLn3JQZdC8FStPAgHatf12BCuMpoiqeUqt 1m49mdiDw0iNfm/GWdmPmpoO0X/uZK1yoxoliz8ISyYysgq+wnuBbD/iv3YGdf5acJJz k88QrlW66kB9p7OU/E0+3Gk4GSn/YIxf002HhJjxN2x+PJcUq2KH6lIHrsFUz/MC4gjR wrlROYuliq0CoP6t1nh64kLPdtM5z/9KgZ+1luwimlXl1TrryVnjWXG4eqT5/73pfVaQ Gn/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785815779; x=1786420579; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=uZCX0uRt3bq26YcZIyKc0yaDQHyGPM1xGVsxvUVCVRw=; b=cOxBAzdBYo3hVsDbBPdd90KeiZrx80F1HZZykcyxTmEJ5r6R4LYnk2XlRcerwIJR2w 9BMARm6FvtDZZuWeVbtFad2l8M8jSNCHo952ME3hD8j4SVIZAKDSPmDcwsgghgCeQIVW P5q/sr31t6m2jjbfsjqF7AzORu5QUvZxgd09eJWW0n26YFfER0dnnby2UhiSqZoB++rA oR+4MEpbLicXHM2eL1gf60wAXCStB539tJmuP8f3sWdc4M/jubPdPIFNn7FI3cVfY8M0 FRndMpDz1F4b3N4Mji5LZvSWWtLlxshrBW0bloPIUfgpPOtnivCcgyPFZIeuWYSXJxG9 xaBg== X-Forwarded-Encrypted: i=1; AHgh+RqHuhBWKc7AVPM5von9No/9C8WURoFcmAZGwxQwOu+kt08U3ODZZTjGr086/B9/OQ0mHKcG2NSMUtIbjGU=@vger.kernel.org X-Gm-Message-State: AOJu0Yzx2sMtih4EOGEaIEVF4yJfZ4LfWqHKA7MvQC/mNwqz+W0vS6WA WvchysHB63TxLfmfp0eOCMUEL/7ohAHHFO6C6m7cAB23EvoiTwzdMSDOeddaN+YUMgQ= X-Gm-Gg: AR+sD12odOPoCwVlL/U0+pCp2vprx2rioUewQP7RMd+/Cdb4FgUlowXmM88Ydi+Vh1s av6nNSruE7WIv8+OizZLXEgEWQE7vUh1mQSC4wd5Ev+j33Bw9nkshuHof1VEL1AdSfsRCOb0AbU abMVAOuh/1SHTDPs8cYxu9m0CBlYwSrqZ0mmrd5DG64Me3F+xqm7AsqMl1cS7S0cY/4uoujJc+7 IW+0u6TQVL+wZ24Jo5Fvn/csIHysqGhYVLggZEzDBrZAie4mlJMHGDD4Iw1hgqL8pdItq1cszEP cDC1yRGUSqBCRv2oNnZepmPMsRPW8t/oyL9UN0sKdHNuc/UVsYNPj/QyihS3pWe5tn00rRpsJQc XpdMmVod78B3x6jLlFbk3/bGYndBmfVNxuM4KjGAeu7yHHvvYGcb6B3A2TkRzUgbCsnoV7luqFD Bwe3N4v9zoSvQPKoeN8EFyhnLLRTNFWcvQiEe/1oRvmEyKnRYnfLY9+NZChI9Pcku+wxGLRpIxz 7UzFGBI X-Received: by 2002:a17:903:98f:b0:2cf:461a:3863 with SMTP id d9443c01a7336-2d08ab8bec3mr18723735ad.22.1785815779325; Mon, 03 Aug 2026 20:56:19 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.99]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d04ae99af1sm46313695ad.33.2026.08.03.20.56.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 20:56:18 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v3 08/17] mm/sparse-vmemmap: support section-based vmemmap optimization Date: Tue, 4 Aug 2026 11:55:26 +0800 Message-ID: <20260804035535.2846016-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260804035535.2846016-1-songmuchun@bytedance.com> References: <20260804035535.2846016-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Teach sparse-vmemmap population code to use the compound page order when deciding whether a vmemmap page can be optimized. With this information, the common sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages directly instead of relying on HugeTLB-specific handling. This centralizes vmemmap optimization logic in the sparse-vmemmap code, based on section metadata, and prepares for sharing the same mechanism across different users of vmemmap optimization, including HugeTLB and DAX. Signed-off-by: Muchun Song --- v2: - Keep vmemmap accounting and population logic in sparse-vmemmap.c (suggested by Mike Rapoport) - Move vmemmap_get_tail() before its first use instead of adding only a forward declaration in the previous patch (suggested by Mike Rapoport) - Simplify the PMD path handling for HVO-covered sections --- mm/sparse-vmemmap.c | 36 ++++++++++++++++++++++++++++++------ mm/sparse.c | 4 ++-- mm/sparse.h | 7 +++++++ 3 files changed, 39 insertions(+), 8 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b770fe2428fd..b69a7af76858 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -186,6 +186,11 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone * return tail; } +#else +static inline struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +{ + return NULL; +} #endif static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, @@ -193,12 +198,24 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in unsigned long ptpfn, unsigned long flags) { pte_t *pte = pte_offset_kernel(pmd, addr); + unsigned long pfn = page_to_pfn((struct page *)addr); + if (pte_none(ptep_get(pte))) { pte_t entry; - void *p; + + if (pfn_vmemmap_optimizable(pfn) && ptpfn == (unsigned long)-1) { + unsigned int order = pfn_to_section_order(pfn); + struct zone *zone = pfn_to_zone(pfn, node); + struct page *page = vmemmap_get_tail(order, zone); + + if (!page) + return NULL; + ptpfn = page_to_pfn(page); + } if (ptpfn == (unsigned long)-1) { - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + void *p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) return NULL; ptpfn = PHYS_PFN(__pa(p)); @@ -217,7 +234,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in } entry = pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); - } + } else if (WARN_ON_ONCE(pfn_vmemmap_optimizable(pfn))) + return NULL; return pte; } @@ -406,6 +424,9 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, pmd_t *pmd; for (addr = start; addr < end; addr = next) { + unsigned long pfn = page_to_pfn((struct page *)addr); + const struct mem_section *ms = __pfn_to_section(pfn); + next = pmd_addr_end(addr, end); pgd = vmemmap_pgd_populate(addr, node); @@ -421,7 +442,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return -ENOMEM; pmd = pmd_offset(pud, addr); - if (pmd_none(pmdp_get(pmd))) { + if (pmd_none(pmdp_get(pmd)) && !section_vmemmap_optimizable(ms)) { void *p; p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); @@ -439,8 +460,11 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, */ return -ENOMEM; } - } else if (vmemmap_check_pmd(pmd, node, addr, next)) + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + if (WARN_ON_ONCE(section_vmemmap_optimizable(ms))) + return -ENOTSUPP; continue; + } if (vmemmap_populate_basepages(addr, next, node, altmap)) return -ENOMEM; } @@ -648,7 +672,7 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) } } -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms = __pfn_to_section(pfn); diff --git a/mm/sparse.c b/mm/sparse.c index ca9875f568d3..24555a32a5d9 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -315,8 +315,8 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page), - PAGE_SIZE)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); sparse_init_early_section(nid, map, pnum, 0); } } diff --git a/mm/sparse.h b/mm/sparse.h index 6ad190ec48cf..f8f852f9f8a2 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -105,8 +105,15 @@ static inline void sparse_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else static inline void sparse_init_subsection_map(void) {} +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); +} #endif /* CONFIG_SPARSEMEM_VMEMMAP */ #endif /* __MM_SPARSE_H */ -- 2.54.0