From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.179 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794018598; b=LNJ7wrrSwD9zfcj1tgOtjnhiXlxbj/6bO7wsRGTLIc11VBShEayA7ddhI2JYrSoJ4pWG h93JJK6/vFMAe8RrrJuUv7TAqSW0oY2z+lnDY8HWr+M/XJlXXDQlVZVpKuMbITiAAkgNI neSV3AR0SOGNLw7yYfj5WwrWhgcxig6P1pP3Hl0l30fcbvVS6d8rNY1jWxI5N+6oLPqX8 aP04Piz12mBd+djtduDpGV1sYpWRe+o8azlJ/SbTisInnbYC1FplcwpdhArmeKXFhyhnC f5YltqSViZoVaUjnAMEeedQugUMVoR0DM3Z0wqAxeSLtrbs+zdN4Sma5hXLgqxosXTw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794018598; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; bh=OqG48vU9uBN7xTzzzeTd+L9N3Q3z6wcaov7tkp2pAS4=; b=qV93iT/68uzoayStY+ld9mWaB6AM6kn7f2coqvzJXal+omkliRhxWp1A4OKjF6cX9AK7 9UFIj+c9KNYJfg82NXFAw2OVQp5T7Ceu34ma2rwjWkDojjtVzr72Z9oTY4wZx32Spv3Au x6PgFE4zYQoTQxl0YTjw1S00tjfYYp3ZoPk553CFoaxgtaYBaxX/IgTb7ZgzKfP1VAgkn 4OHIQKUMDTHhHSyGr7NNEqDtzkEI07ZxeccTak5uj3D2pUH/yuUZRiELDQYMzSHqtOiy6 F1l+Q/X+/6cY53NjcC/qc2PykNuhBg9gfcFrK7EV16WXG5lemlL3N3+xBwNeldxwd4g== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=T+kLBsZ6; arc=none smtp.remote-ip=74.125.82.179 Received: from mail-dy1-f179.google.com (mail-dy1-f179.google.com [74.125.82.179]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF972606A326 for ; Sat, 7 Nov 2026 02:29:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org CF972606A326 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=T+kLBsZ6 Received: by mail-dy1-f179.google.com with SMTP id 5a478bee46e88-36fb88ea38cso283511eec.1 for ; Fri, 06 Nov 2026 18:29:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1794018597; x=1794623397; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=OqG48vU9uBN7xTzzzeTd+L9N3Q3z6wcaov7tkp2pAS4=; b=T+kLBsZ6CjJBjEKgpi33GMp0EGhSc1ipTGNJsoiQY8/iespQLCSJsiyV7bF9U9qAu1 YYcCnbkwqxkwyMRih6J8eE2SoAtVB/ebbtioYKEOyG73FznoeKfZA6wak38Tp7pg5M0D I16KgCDVka7qtEG1pe8M8BaBcdMwUlSlF/hRMnvQh9Gh9udz3FbfFa718njm/KPKDjmG ZiC2e4C5G8+ydCQzr3xeRcboyBG4pO3yHdk3wFLVLOeA66Qk6ZRbSzpCcOSdJ2EI1sDE 9/d9XmZ0tQqV/cAi+J09RKbO5dopmq20yE8cYFIrs4uX333jMzkVWCaE7rjygZLI4Ijj AzUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794018597; x=1794623397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=OqG48vU9uBN7xTzzzeTd+L9N3Q3z6wcaov7tkp2pAS4=; b=jgtcEmmrI8Hgc9CfvlEJzzlSVQ6Z9hZAsduOUxs6xeNcrOEa9JiS79wxNoeFuMEebE MsOGV1hBROPzVVGhooRRkPN+iOf3+8GsZjXp/g4I1lN76NDpmEqUuvPwdAn5xqHHqjeX 1JqhjoQm+81NW3R0SpkB8sO9Ebpn8XGch/9uYKCxIdSFOkG/U7loeQlROb3RspirNnoT K8u3HPVMzaeVtWFD/J0MLvooBRphL/bvepDa2s+1+JIw+Juv8L7fG2wlMVk12IPriAso Q0A0pVZa+mgSg4L/Eg7yWKA3Q5M8CRUY3bsG76ODDzKBNKmmUH80NYace7xvJPZsAeV3 DbBQ== X-Forwarded-Encrypted: i=1; AL+hErF146oQKcGebITdOq+YwjAOrlFCdBxTHSLBAydWsmi5xlXiVfGpRU5236Y+ODXSZZ4QLAhV6f0AHZXDz70=@vger.kernel.org X-Gm-Message-State: APuU4Z9gd1yglyRa/pxuENsN5+MtxYptJEkdzX/Q0Hqqvin8DvDsmjIf WjGME+KVbFEI9lMnTjoEU0uw97qasUxUPQYHU+hPU+ZUjxWt5z/qKBgQuDZu5XlFfzSRuk+Wy8M Nja2OIiijXA== X-Gm-Gg: AYBFou0SV5UX3OhYBRhD95QwiirqxTo4KrifAM2sSejPh2rwkHfAK0OwmyJuGYu6s0B t70pG+0fb3FosoMo9NysYm4Y4dKcIPvr8VrN6GT2zdF8JBnLF93cX7+hREWm7r4Qtmm2vVScZ8s h4Lp6fslQndDipF7Md91qBp3W3Ez+OjncaX2Uxyxv8cnHWujDVw3615cb2trmDPtFOKTUU4YnuN l/J2ml+gLqW7AIwMK0LPfK2fDKe1TkkjOWTb6WoEnwGZMKuPaXaqlMYbw06wvpo+64K9N+rqsqD 4aupGdL7Lyfn5BRbcJ3IEzKPSaODAXonTzAoVPMBVF8rZzrg/QhkC/hJ7mnzUplogbHZtvWuQkf i2s4dpskSLxubve4tZIXNp/fW7jOQ6iuEM3OptIw1KSeRDZAaAfpFMrn7elsuRPe9+WFFgr202+ KXXXGf+B+cW+QmaulIJoBzjVV3uM44rb/3DmXErObSP+aVE4D1pOlbEt7jzJntRXrJ4NgRGrSbc GO/A8IUQyEgaUmEQeoyiCF1ATUxRqvNCVVB0+haNXCS0e/HtaKe1czsT5aJVVZ9uT5nsm942jHb 4+cfzHUYp0l/Dj+pUmufbUiFw8NY2CzV0MVcuDxi7rhbRCcqU9AyK7rXUWow X-Received: by 2002:a05:693c:69c8:b0:35e:af84:bf1e with SMTP id 5a478bee46e88-370f4c6234bmr1072334eec.7.1794018596714; Fri, 06 Nov 2026 18:29:56 -0800 (PST) Received: from G6L4RL2QG9 ([2409:8a00:1832:6900:c8de:13da:804b:a0fa]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-36faa4b8aacsm10457077eec.23.2026.11.06.18.29.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 06 Nov 2026 18:29:56 -0800 (PST) From: Muchun Song To: Oscar Salvador , Andrew Morton , Mike Rapoport Cc: Qi Zheng , Lance Yang , James Houghton , David Hildenbrand , Muchun Song , Yili Zhang , Kiryl Shutsemau , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v4 08/13] mm: hugetlb_vmemmap: simplify shared tail page handling Date: Sat, 7 Nov 2026 10:27:51 +0800 Message-ID: <20261107022756.77731-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261107022756.77731-1-songmuchun@bytedance.com> References: <20261107022756.77731-1-songmuchun@bytedance.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit The remap walker currently carries a shared tail page selected by its caller, even though the optimize and restore callbacks can derive the same information from the folio being processed. This adds redundant state to the walker and couples its interface to caller-side preparation. Derive the shared mapping state in the callbacks instead, keeping the mapping policy with the PTE being processed and simplifying the remap interface. No functional change intended. Signed-off-by: Muchun Song Reviewed-by: Lance Yang Reviewed-by: Qi Zheng Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Oscar Salvador Tested-by: Andrew Morton --- v4: - Keep the caller-selected allocation mask after shared-tail cleanup - Collect Reviewed-by from Oscar Salvador - Collect Tested-by from Andrew Morton v3: - Use two tabs for function parameter continuations (suggested by Mike Rapoport) - Collect Reviewed-by from Mike Rapoport v2: - Update the vmemmap page-list contract for shared tail derivation (follow-up to review by Lance Yang and Qi Zheng) - Collect Reviewed-by from Lance Yang and Qi Zheng --- include/linux/vmemmap-optimization.h | 6 +- mm/hugetlb_vmemmap.c | 93 ++++++++++++---------------- mm/mm_init.c | 6 +- mm/sparse-vmemmap.c | 7 ++- 4 files changed, 52 insertions(+), 60 deletions(-) diff --git a/include/linux/vmemmap-optimization.h b/include/linux/vmemmap-optimization.h index 065840f3c241..858311a71b0c 100644 --- a/include/linux/vmemmap-optimization.h +++ b/include/linux/vmemmap-optimization.h @@ -91,11 +91,13 @@ static inline struct page *vmemmap_shared_tail_page(unsigned int order, } #endif /* CONFIG_VMEMMAP_OPTIMIZATION */ -static inline bool vmemmap_optimizable_pfn(unsigned long pfn) +static inline bool vmemmap_optimizable_pfn(unsigned long pfn, unsigned int order) { - const unsigned int order = pfn_to_section_compound_order(pfn); const unsigned long nr_pages = 1UL << order; + if (!IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION)) + return false; + if (!is_power_of_2(sizeof(struct page))) return false; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 7a3af4fd285b..3968dfe7f5a0 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -25,21 +25,17 @@ * * @remap_pte: called for each lowest-level entry (PTE). * @nr_walked: the number of walked pte. - * @vmemmap_tail: the page to be installed as non-first in the vmemmap range * @vmemmap_pages: list supplying replacement pages for the remap and * receiving displaced backing pages for the caller to free. * @flags: used to modify behavior in vmemmap page table walking * operations. */ struct vmemmap_remap_walk { - void (*remap_pte)(pte_t *pte, unsigned long addr, + int (*remap_pte)(pte_t *pte, unsigned long addr, struct vmemmap_remap_walk *walk); - unsigned long nr_walked; - struct page *vmemmap_tail; struct list_head *vmemmap_pages; - /* Defer safely batchable TLB flushes to the caller. */ #define VMEMMAP_DEFER_TLB_FLUSH BIT(0) unsigned long flags; @@ -140,12 +136,14 @@ static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr, static int vmemmap_pte_entry(pte_t *pte, unsigned long addr, unsigned long next, struct mm_walk *walk) { + int ret; struct vmemmap_remap_walk *vmemmap_walk = walk->private; - vmemmap_walk->remap_pte(pte, addr, vmemmap_walk); - vmemmap_walk->nr_walked++; + ret = vmemmap_walk->remap_pte(pte, addr, vmemmap_walk); + if (!ret) + vmemmap_walk->nr_walked++; - return 0; + return ret; } static const struct mm_walk_ops vmemmap_remap_ops = { @@ -213,53 +211,53 @@ static struct page *vmemmap_dup_page_from_list(const struct page *src, return dst; } -static void vmemmap_optimize_pte(pte_t *pte, unsigned long addr, - struct vmemmap_remap_walk *walk) +static inline struct page *folio_vmemmap_shared_tail_page(const struct folio *folio) { - struct page *page = pte_page(ptep_get(pte)); - pte_t entry; - - /* Remapping the head page requires r/w */ - if (unlikely(walk->nr_walked == 0)) { - struct page *head; + return vmemmap_shared_tail_page(folio_order(folio), folio_zone(folio)); +} - VM_WARN_ON_ONCE(!PageHead((const struct page *)addr)); +static int vmemmap_optimize_pte(pte_t *pte, unsigned long addr, + struct vmemmap_remap_walk *walk) +{ + const struct page *page = (const struct page *)addr; + const struct folio *folio = page_folio(page); + struct page *src = pte_page(ptep_get(pte)), *dst; - head = vmemmap_dup_page_from_list(page, walk->vmemmap_pages); - entry = mk_pte(head, PAGE_KERNEL); + if (likely(vmemmap_optimizable_pfn(page_to_pfn(page), folio_order(folio)))) { + dst = folio_vmemmap_shared_tail_page(folio); + if (!dst) + return -ENOMEM; } else { - VM_WARN_ON_ONCE(!PageTail((const struct page *)addr)); - - /* - * Map the shared tail page writable. A lockless get that read - * a non-zero refcount before the remap can still do its - * cmpxchg after it. The refcount is zero, so the cmpxchg - * fails and leaves the page unchanged. - */ - entry = mk_pte(walk->vmemmap_tail, PAGE_KERNEL); + dst = vmemmap_dup_page_from_list(src, walk->vmemmap_pages); } - list_add(&page->lru, walk->vmemmap_pages); - set_pte_at(&init_mm, addr, pte, entry); + list_add(&src->lru, walk->vmemmap_pages); + /* + * Map the shared tail page writable intentionally. A lockless get that + * read a non-zero refcount before the remap can still do its cmpxchg + * after it. The refcount is zero, so the cmpxchg fails and leaves the + * page unchanged. + */ + set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL)); + + return 0; } -static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, - struct vmemmap_remap_walk *walk) +static int vmemmap_restore_pte(pte_t *pte, unsigned long addr, + struct vmemmap_remap_walk *walk) { + const struct page *page = (const struct page *)addr; + const struct folio *folio = page_folio(page); struct page *src = pte_page(ptep_get(pte)), *dst; - /* - * When rolling back vmemmap_remap_free(), keep the copied head page - * mapping and restore only PTEs currently pointing at the shared tail - * page. - */ - if (walk->vmemmap_tail && walk->vmemmap_tail != src) - return; - - VM_WARN_ON_ONCE(PageHead((const struct page *)addr)); + if (unlikely(!vmemmap_optimizable_pfn(page_to_pfn(page), folio_order(folio)))) + return 0; + VM_WARN_ON_ONCE(folio_vmemmap_shared_tail_page(folio) != src); dst = vmemmap_dup_page_from_list(src, walk->vmemmap_pages); set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL)); + + return 0; } /** @@ -287,7 +285,6 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end) * to remap. * @end: end address of the vmemmap virtual address range that we want to * remap. - * @vmemmap_tail: the page to be installed as non-first in the vmemmap range * @vmemmap_pages: input/output list whose first entry supplies the new backing * page for @start; replaced backing pages are added to the list for * the caller to free. @@ -296,14 +293,11 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end) * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - struct page *vmemmap_tail, - struct list_head *vmemmap_pages, - unsigned long flags) + struct list_head *vmemmap_pages, unsigned long flags) { int ret; struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_optimize_pte, - .vmemmap_tail = vmemmap_tail, .vmemmap_pages = vmemmap_pages, .flags = flags, }; @@ -321,7 +315,6 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, */ walk = (struct vmemmap_remap_walk) { .remap_pte = vmemmap_restore_pte, - .vmemmap_tail = vmemmap_tail, .vmemmap_pages = vmemmap_pages, .flags = 0, }; @@ -498,7 +491,6 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, unsigned long flags) { unsigned long vmemmap_start, vmemmap_end; - struct page *vmemmap_tail; gfp_t gfp_mask = GFP_KERNEL | __GFP_NOWARN; int nid, ret = 0; @@ -509,9 +501,6 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, return ret; nid = folio_nid(folio); - vmemmap_tail = vmemmap_shared_tail_page(h->order, folio_zone(folio)); - if (!vmemmap_tail) - return -ENOMEM; /* * Very Subtle @@ -540,7 +529,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, * they can be freed by the caller. */ ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, - vmemmap_tail, vmemmap_pages, flags); + vmemmap_pages, flags); out: if (ret) folio_clear_hugetlb_vmemmap_optimized(folio); diff --git a/mm/mm_init.c b/mm/mm_init.c index d99af80ec752..a9d848c92678 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -914,6 +914,8 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone #endif for (pfn = start_pfn; pfn < end_pfn; ) { + const unsigned int order = pfn_to_section_compound_order(pfn); + /* * There can be holes in boot-time mem_map[]s handed to this * function. They do not exist on hotplugged memory. @@ -929,9 +931,7 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone * Vmemmap-optimizable PFNs are backed by shared tail struct pages, * which have already been initialized during vmemmap population. */ - if (vmemmap_optimizable_pfn(pfn)) { - const unsigned int order = pfn_to_section_compound_order(pfn); - + if (vmemmap_optimizable_pfn(pfn, order)) { pfn = min(ALIGN(pfn, 1UL << order), end_pfn); continue; } diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 3e9d6617c4bb..b8b55e12f5ba 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -241,7 +241,7 @@ static __meminit void *vmemmap_alloc_pte(unsigned long pfn, int node, struct page *page; const unsigned int order = pfn_to_section_compound_order(pfn); - if (!vmemmap_optimizable_pfn(pfn)) + if (!vmemmap_optimizable_pfn(pfn, order)) return vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); /* @@ -286,6 +286,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in { pte_t *pte = pte_offset_kernel(pmd, addr); unsigned long pfn = page_to_pfn((struct page *)addr); + const unsigned int order = pfn_to_section_compound_order(pfn); if (pte_none(ptep_get(pte))) { pte_t entry; @@ -296,10 +297,10 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in entry = pfn_pte(PHYS_PFN(__pa(p)), PAGE_KERNEL); /* Map the shared tail page as read-only to catch illegal writes. */ - if (vmemmap_optimizable_pfn(pfn)) + if (vmemmap_optimizable_pfn(pfn, order)) entry = pte_wrprotect(entry); set_pte_at(&init_mm, addr, pte, entry); - } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn))) + } else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn, order))) return NULL; return pte; } -- 2.54.0