From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C4D8472786 for ; Wed, 13 May 2026 13:10:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778677858; cv=none; b=ki2EnfZTkw3vxFql0No6oj5/UDk6HpKOb1e/JWHIlsrRau2oLrcK6MiE2LFgY5MFEknCxntB9zjO1v9j30i/HGxOn/xqFPaykjfHNxHsWVdgVl0XV+f17aZHsksevSkrne+iAHjfrUovx96LalLoKcfTYweg8JIQdEF2Rc+4Gw4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778677858; c=relaxed/simple; bh=SgrshUvAW2N2bcDVajsSpokJidezKatlfTdxZF84bx4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Bs3JAfCiAsd9Ffw1pIEGr3IoRwzz7AeQgnl5dxHmQnftJdDK2ZQJTTTWw72RcUseQwVQP122cZWVQlNi0a3C230UsG74+aMUf2tGHSkuc2Se3tcAlonSsIaG1YguF4wMwd0533ITC8WHM8KVTCxg+f8ZAL4LpHysVP/hNV9diB8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IwJQZ82e; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IwJQZ82e" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2ba856db1c0so45838665ad.3 for ; Wed, 13 May 2026 06:10:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677856; x=1779282656; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dToyrDGV72K3gHJcuzQsCSmu29LS3Njm8M96u525pzE=; b=IwJQZ82eVGBsO3p48RZAyCu2XdLW9BOqYL+peC/Db5mFZ7/MNNOZUu3FlIzhMO2pUw bKbIihhjVRbt/XyguAYG/x1r6pr8o9fyUTlrPRU+MXloDz5q5D4aNfqGrZNf+FXKtiz4 Os/5cdFNXomj7XF/ZoJE5E1YeJdmDlpbWpdvWgZGjv/2SChiJ62yJ+uCD4PZvlBMTk0V 0aNqxorI4fu+CyOUcidGHk/bz6gPiEXhXS+93jWEQ2Qjvjq2D4Hhlf+N0TVZSp9/hwKE NRK9sqOGeHFQ1dX8ROO+abln37hBOkRtSegrI5D2BIZnORRYZFQHOWHM8SBbEV6Z6mkz VjkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677856; x=1779282656; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=dToyrDGV72K3gHJcuzQsCSmu29LS3Njm8M96u525pzE=; b=Bw8HJ3fCO7wJRqsTBFv21k+hZ1YPglxr/CR4NXqkKkMdf/4y3TrmJyAZ0NeTCV/cFP 3cE+4cET1i2RrW8UQFeg586dF5cJRwsspJqQI1fgZDz7aPcyquaVrSURIQS7QjePbJD6 x6SDUWvu47YJj0SjxnPWFSLP3HATkcZh7brDhNUV7zgdQyAmzhomdaMh3GMkSW3d4OfT CLWj9mQuWR1Cp7kulJqoOdZdX4gTRBfiSEO/sq1rU2uYOKtuE8PtrWY+qsAvQTikQDqU TBBb1nmdzuGHNKjxCKQHcrgy5yHpSD2ppcY7Xv/FcwPCeuQYvChV/b/s8jKbylC4jfPq JPow== X-Forwarded-Encrypted: i=1; AFNElJ82wDM/cCiWY0NhO9/NLhEwMOPYMeUx3q3n0pNJsMGgbAMzM6S84rOMmTqCkyWwdXCNn+sbo4SnqwVw0xk=@vger.kernel.org X-Gm-Message-State: AOJu0YwP73/x5T2b6xbFp1qFL9sTRWRGTPkiE/uirRHt0UkrwiWR6eFO n+DeABILC4a+H9XjQIuT3jVfmrHWWeCkdHwMD+/rmUX5tI8uQ0ypUkM2uh5aQ19Y/kA= X-Gm-Gg: Acq92OF8ETCGgALAN76dYOPpF4ogm6AYnDBUHp5pm6PXpnEP+OtTmpFUsaWuVS8a9oE g3nowAIXEXk2xoL5knbnUiaTOKcMpX+Ym1RczJyFYRph+ZtexSrjqvMa+T6mMqDitqVOK0PVLOm rJAPOf5epvcwejr+5JnxMZZv4+GGGExskfEuwo/zMmv3tNnjtYENRBv3BNWR1xHz76ImyAv/O5A eQhRMl4e9AkHbgWNhhI6TM8OOVWrNg92x73SllIQnmGe/R+UBfhm0tLYtqLNJCyCY+E5FkxR6hB P38zuG4S7aGOyxgyb12AKP66FcWK5X+78ZI4ZxFcHlHtQRDq90JVgSAGYTFk7ZBbhmr4utX8OQe KBNyniPTtVOM60JVm8T9O7IMDeQ8WRxahimcGjc8Rse2qGfVxZFjInLEFMaHsPbNM7JVRMldrDH sdlCtrWl9Sw+lnc2FS5ys7mBUBROYezKi/uJry3iLP7HnJrc3tQuLe6/XXodY= X-Received: by 2002:a17:903:1b28:b0:2bc:d026:eeff with SMTP id d9443c01a7336-2bd27188006mr39565445ad.8.1778677856477; Wed, 13 May 2026 06:10:56 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:56 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 37/69] mm/sparse-vmemmap: Factor out shared vmemmap page allocation Date: Wed, 13 May 2026 21:05:05 +0800 Message-ID: <20260513130542.35604-38-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit HugeTLB and sparse-vmemmap each have their own helper to allocate the shared tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper in sparse-vmemmap.c. It allocates the page through vmemmap_alloc_block_zero(), initializes the tail struct pages, and uses cmpxchg() to install the per-zone shared page. This removes duplicate allocation logic while still handling both the early boot and runtime paths through the same helper. Signed-off-by: Muchun Song --- include/linux/mm.h | 1 + mm/hugetlb_vmemmap.c | 28 +----------------- mm/sparse-vmemmap.c | 67 ++++++++++++++++++-------------------------- 3 files changed, 29 insertions(+), 67 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fef39be8acd2..5281f073230c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4866,6 +4866,7 @@ int vmemmap_populate(unsigned long start, unsigned long end, int node, void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int node, unsigned long headsize); void vmemmap_populate_print_last(void); +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zone); #ifdef CONFIG_MEMORY_HOTPLUG void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 66362e553870..d24143dd6051 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -499,32 +499,6 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio * return vmemmap_should_optimize(h); } -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; - struct page *tail, *p; - int node = zone_to_nid(zone); - - tail = READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p = page_to_virt(tail); - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail = READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -541,7 +515,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, return ret; nid = folio_nid(folio); - vmemmap_tail = vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail = vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index dde4486195ad..53a341fcde74 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -34,27 +34,13 @@ #include "internal.h" -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask = GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order = get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; page = alloc_pages_node(node, gfp_mask, order); @@ -68,8 +54,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node) } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -138,8 +123,6 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone); - static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, unsigned long ptpfn) @@ -158,7 +141,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in if (WARN_ON_ONCE(!zone)) return NULL; - page = vmemmap_get_tail(section_order(ms), zone); + page = vmemmap_shared_tail_page(section_order(ms), zone); if (!page) return NULL; ptpfn = page_to_pfn(page); @@ -190,7 +173,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in return pte; } -static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) +static void *vmemmap_alloc_block_zero(unsigned long size, int node) { void *p = vmemmap_alloc_block(size, node); @@ -329,32 +312,36 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, } } -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone) { - struct page *p, *tail; - unsigned int idx; - int node = zone_to_nid(zone); + void *addr; + struct page *page; + const unsigned int idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; - if (WARN_ON_ONCE(order < OPTIMIZABLE_FOLIO_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >= ARRAY_SIZE(zone->vmemmap_tails))) return NULL; - idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; - tail = zone->vmemmap_tails[idx]; - if (tail) - return tail; + page = READ_ONCE(zone->vmemmap_tails[idx]); + if (likely(page)) + return page; - p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + addr = vmemmap_alloc_block_zero(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - tail = virt_to_page(p); - zone->vmemmap_tails[idx] = tail; + for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail((struct page *)addr + i, NULL, order, zone); + + page = virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(page_to_virt(page), PAGE_SIZE); + page = READ_ONCE(zone->vmemmap_tails[idx]); + } - return tail; + return page; } void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, -- 2.54.0