From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E22A33D5222 for ; Mon, 20 Jul 2026 09:32:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784539948; cv=none; b=uXQFP36I26PK6rQeH3Aze8kZM4njn9xBSuCe3vRJDJRIIuyv2aA21ldMCqMoCs20YrBcz2oY00pE7gObP3N8xsF6Pi/h5HrSK6NZg3EzTxy85rp2DexFGRmD+74T6kZqenT0x59tlGxfAF6jA+w6Ks2n5VxeYsHGMnDW1YElufQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784539948; c=relaxed/simple; bh=LzdMqlEotTeBoZebm0qCJLpqHk7PFGo/ia/2dpI8Xqo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jm4vMF+nNvo3tMbX1feaTLjn9M1QufxvuWo3VmE0AkfKXIjYf9LOO5kCkMksc6utvqN+GK33IlTMpV501LREV62soatTuygKYLO1klkIUAVpdIVCslE7ix5IWXZ781gzBAZfu2D7DY1SE+SLRrNNlRm5g/4ZMRokKd9N1rAeci0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=bztreUtP; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="bztreUtP" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2cf452def93so12080015ad.1 for ; Mon, 20 Jul 2026 02:32:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1784539946; x=1785144746; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XIl1erdJ2bKlvmuHeltJ/d9wztXmxG8GJiFqIKkszIE=; b=bztreUtPMDqSwEXvpOGrxwR4UjELESWoqgCb5rHOtrSl4lrtvlSrJsw3i0x0nLUU8Z h44BUG9B0BO2kCTaYFOaW5aodCwoHND/ucarvhOOYGoi2ppfe2c7QDMpnLHylbsJkivg lwNltSJNXeVbva7TSvMoH1K/wWTYOyN0GJz+05OwrUHFRAWJPZc7+EPjxRZDbLL1uzAi LjrDBVWfnnitlsBSVgUSBU+Y7ar70ztZKetB8UyQzlXzLG4hd0aewti97obzTEpSJI5z pEBHn+IHszfAiii6HeRc9SvKKgDfa9paE7SJVYIcYMQZ5AE7KaZX9iHJ6ADBQITMNcpe jD3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784539946; x=1785144746; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=XIl1erdJ2bKlvmuHeltJ/d9wztXmxG8GJiFqIKkszIE=; b=VMTTiBGDhUnUw/OwUWaHrbhG79W+YuFjfNlxLLOhal3nyXcJh+0sol5/hbA8koeFde BEMw03WAYSS6c8+vn5XmpBpw1A3uwzOiG+Hmu+ckBij1tw+ONQ9w4q8wRc+Qn05+1AiX QGTHVnClJ2G3HRgwyVs7Tq8zKIcfiyHo9mrDHfJRiML90F1cive2fccd2qBd0vxV1fsK g8m063JYjB2JswlVhiyHGTUFA8004ChCGXq6YyktIe/fF6z5navDPFZU6qy56+OJEnYq hjPHuE+Z6jJoddYFu28DlPDE07w24y6j9R4jJWMHd3ePUbpIdZDK3Sbmz+5RkQIjWRNp K0EA== X-Forwarded-Encrypted: i=1; AHgh+RrpEjluNB9RSLjFxWijSWGIt7xIH1jWAo2W1l1jgHMYHstfdjnaxW5meYl1wgOkjpmMh+1nwU4602D+3Cs=@vger.kernel.org X-Gm-Message-State: AOJu0YxeL936/swPFgxaf8jrVj38McMPfenqWh4grbNh9EaVsaJIJJAS ttR4YIsFTKk7i3bVqE0SCuF7zd3KM+pO/yBMjQ2gpFsHPceRvKIgWNg9eoZYETObPmY= X-Gm-Gg: AfdE7cm19IKA5fYy9CzlRBRX1Scw5XakM9J3LGmzVEcUNUdbSGOV3fcp8M1vYDUH6Ld qrSjYH6eGwuarHr2NErU7YCBCs4TrujqFu/ApjsQI9zDEBIQ+1aQlksR6a3vgnhtgJwzZ9ogZ8t pQ7HhhQFIo1Q5ufkoHqVagtkKUJ9LV8PgPPgXwFy9uxf0dv3B7X8P6whmBZwEWWgQe3CuhxxSqX RTUZqd1dUStJq9KKQWNcH2avlNlGvHYpBSOidQThPWVqXKBfCuFNr7n6u/T0MpC99ksQq2kpEn7 yh4pzd1wOQHaqC+4af/0f/Ueq/CaYeUyUTPXceYcoNZyi+Vd3VCxupw744gnDbga3qnGfebX03N +rtqXXPM3mnhn48bJykrJaSeeGpAdQsEq/PePet8XYGY5mJfzBhoq0xlsYDPiLn1txOe0wzRGkC ctTSJfbuUR7y2lpZUWxEHvWWo= X-Received: by 2002:a17:903:198c:b0:2c9:c517:d08b with SMTP id d9443c01a7336-2cf34a50497mr134993955ad.22.1784539946139; Mon, 20 Jul 2026 02:32:26 -0700 (PDT) Received: from n232-176-004.byted.org ([36.110.163.101]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2cf347a7c9fsm53136455ad.84.2026.07.20.02.32.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2026 02:32:24 -0700 (PDT) From: Muchun Song To: Andrew Morton , Oscar Salvador , David Hildenbrand Cc: Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Michal Hocko , David Laight , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Muchun Song Subject: [PATCH v2 08/17] mm/sparse-vmemmap: support section-based vmemmap optimization Date: Mon, 20 Jul 2026 17:31:18 +0800 Message-ID: <20260720093127.540540-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260720093127.540540-1-songmuchun@bytedance.com> References: <20260720093127.540540-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Teach sparse-vmemmap population code to use the compound page order when deciding whether a vmemmap page can be optimized. With this information, the common sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages directly instead of relying on HugeTLB-specific handling. This centralizes vmemmap optimization logic in the sparse-vmemmap code, based on section metadata, and prepares for sharing the same mechanism across different users of vmemmap optimization, including HugeTLB and DAX. Signed-off-by: Muchun Song --- v2: - Keep vmemmap accounting and population logic in sparse-vmemmap.c (suggested by Mike Rapoport) - Move vmemmap_get_tail() before its first use instead of adding only a forward declaration in the previous patch (suggested by Mike Rapoport) - Simplify the PMD path handling for HVO-covered sections --- mm/sparse-vmemmap.c | 36 ++++++++++++++++++++++++++++++------ mm/sparse.c | 4 ++-- mm/sparse.h | 7 +++++++ 3 files changed, 39 insertions(+), 8 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b770fe2428fd..e818b4109777 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -186,6 +186,11 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone * return tail; } +#else +static inline struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +{ + return NULL; +} #endif static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, @@ -193,12 +198,24 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in unsigned long ptpfn, unsigned long flags) { pte_t *pte = pte_offset_kernel(pmd, addr); + unsigned long pfn = page_to_pfn((struct page *)addr); + if (pte_none(ptep_get(pte))) { pte_t entry; - void *p; + + if (pfn_vmemmap_optimizable(pfn) && ptpfn == (unsigned long)-1) { + unsigned int order = section_order(__pfn_to_section(pfn)); + struct zone *zone = pfn_to_zone(pfn, node); + struct page *page = vmemmap_get_tail(order, zone); + + if (!page) + return NULL; + ptpfn = page_to_pfn(page); + } if (ptpfn == (unsigned long)-1) { - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + void *p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) return NULL; ptpfn = PHYS_PFN(__pa(p)); @@ -217,7 +234,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in } entry = pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); - } + } else if (WARN_ON_ONCE(pfn_vmemmap_optimizable(pfn))) + return NULL; return pte; } @@ -406,6 +424,9 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, pmd_t *pmd; for (addr = start; addr < end; addr = next) { + unsigned long pfn = page_to_pfn((struct page *)addr); + const struct mem_section *ms = __pfn_to_section(pfn); + next = pmd_addr_end(addr, end); pgd = vmemmap_pgd_populate(addr, node); @@ -421,7 +442,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return -ENOMEM; pmd = pmd_offset(pud, addr); - if (pmd_none(pmdp_get(pmd))) { + if (pmd_none(pmdp_get(pmd)) && !section_vmemmap_optimizable(ms)) { void *p; p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); @@ -439,8 +460,11 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, */ return -ENOMEM; } - } else if (vmemmap_check_pmd(pmd, node, addr, next)) + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + if (WARN_ON_ONCE(section_vmemmap_optimizable(ms))) + return -ENOTSUPP; continue; + } if (vmemmap_populate_basepages(addr, next, node, altmap)) return -ENOMEM; } @@ -648,7 +672,7 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) } } -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { const struct mem_section *ms = __pfn_to_section(pfn); diff --git a/mm/sparse.c b/mm/sparse.c index ca9875f568d3..24555a32a5d9 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -315,8 +315,8 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page), - PAGE_SIZE)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); sparse_init_early_section(nid, map, pnum, 0); } } diff --git a/mm/sparse.h b/mm/sparse.h index 1e9fcea10181..4377b097dc8d 100644 --- a/mm/sparse.h +++ b/mm/sparse.h @@ -87,8 +87,15 @@ static inline bool section_vmemmap_optimizable(const struct mem_section *section */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else static inline void sparse_init_subsection_map(void) {} +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); +} #endif /* CONFIG_SPARSEMEM_VMEMMAP */ #endif /* __MM_SPARSE_H */ -- 2.54.0