From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33C6F480DCA for ; Wed, 13 May 2026 13:11:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778677870; cv=none; b=EJBV4LrINqn+5LljKf43s5iSefwr7DZ9yOfmpcP8LhsiHFC7LBQ4A8nQL1wyxzx+JtWjbupPF13WSuKd+YmRpWMu7axxl8mtKsqX+fsy4CTSY4zl0NCMQtPcSjDXOip/tvyAtagtFsjPfmLGbIjxwCZbLbJccWMxS7Z8EseZn5c= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778677870; c=relaxed/simple; bh=8Pau3UbM8P//6UdXy6VFjgPAqcyUWR3RxacnSIB9gEo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RrBDs/xRIvXX0oDTL9HqtzZW/YD4FvHS8qhGgFUNy4OcBizYpOKc2Bzf27RXHnyJR313xQCRxA520daaW3lXROcVafjBaXMqfqaztpL1sBA/mHzeaJupxOFN0ChbH4DGM+3U7P7UYSssK0ZgnDXI/v/JLcDLvHzx8n99sJqipME= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=j3Rx9eiG; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="j3Rx9eiG" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2b458ca2296so45055265ad.0 for ; Wed, 13 May 2026 06:11:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677867; x=1779282667; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lVy27NFdLuqlZKA5DziPKCwbPsRWePBCMeJ/pOD9y2M=; b=j3Rx9eiGguAMTQQWai35S1WVA5nwlMo1jS7u/qYR5cwPS6d4hYAlCBJDkO0d5tXfyW oGBUtbb/GV1XKNWTLv6Md/Wkb7hnb9A0bhx9T84Wj5DxEHM7TXiZ4pnAdKE6oYr9pyJD H8FexWZRZy0gmhgvwkaZJ1v6/VKvhUi5tvbWuZYDCV32ZdZbEjkOMTlmY9Xg8D4RMaLu AlUzrOvZ+oLPrCmRG4dsETC/Sg3T1UmYzpt9EirKhO600pG3xzEYGiA/GCEefzN22iTV Bm0xuVY2aFbyjNEyWtdq5EEck9Q+Jzz2Z1kRU0s0ov1tOcMJ4/HJIpovihJcXFDOH3aR d6wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677867; x=1779282667; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=lVy27NFdLuqlZKA5DziPKCwbPsRWePBCMeJ/pOD9y2M=; b=rHrUmTzFVV/Z18triT7dwbO+F5uxkl470t06GTLTqL/p/MH+IFjNU6zORgdb49BwGh 1JWsKSJMw8uDsVJbLsfM/394wR1eWodRM6B0U+URab4ZTP905R3YOCd97YQ3K6snVXya qEDXdNrNZ6Ctiv0cveFYCFkRENUsn/BTx0pLCd97ZkEDYnb1+jjSRtUMmQbOooBAYOTL 52tvYDoTJvnH33W1oUJOWZd11uaS9aMPGyuoK24X4qZBobcamEr4M2ImLy5dh7J/hbIy waZJ3czg2d7V/RHK2QokN63GItZ08YvTmZN9fQPx4EkWsbUljdMb91i1bvp6XZk/oJPR It3g== X-Forwarded-Encrypted: i=1; AFNElJ9dV8BdwYsImETDSJCCFhGLpViDyMgBpdgKU/VhcZ0V2NFaG5PvgSSGQTT8h1Hf3Lceytyk7ijXL9CyHQ4=@vger.kernel.org X-Gm-Message-State: AOJu0Yz/2oCkYdhG87Uw9FmTVdlwVZr1houlxpnO4ZKOsDq0bvdJCCxC lpEJQs4C+2BXzRgH8dfaT9V3aSa4ESguQSz56JvbNsP9kQQpp7dps0CnuT16NsUO/1o= X-Gm-Gg: Acq92OG6xt3nMl4wN1oQ0otGeXVMbwRfMheTpwEcNMuOXLRgFHWF4e8ZKMWwunjukZ+ Bp9nOGpJhRZVs5Z+rAqAUM6Hk/K+mOtXLKcBNEvOj5dzqP+jsDzOhxjsVsUixBvXHYHDTyschTP Wci6fT94efDwiMyN2hgbMF8fNJGjSWnV3CO9FZGPGTyS/bKO6cYQahI5k+NDhbuOrpLpE/rcJVP FCF0U6dSsni+R0K88bixbicQdhe8mPpYLVFilqpvc0MGmiSDJwTyaXECAxWlzUIkEkm1Tft988a 0qsxJ9eX80YWlWga6NwdAwTcEo3tYY9vto0ZAVlOIfI1wZoicNz2OSS23qI0RJSmBX1/sQjKIjs ZXyvYFSp6P0pbYCEMdXnGAKcUnO+wOkUqbQCLHk8A80HOMTLTcTKInegeSeaVLsxJit4t4jpY+E 7IH+DTv0Xr7IIBFePDDSE7kU3RVtjtHq9Df52kd/BsNqEXJgBgJSVTwv565Ps= X-Received: by 2002:a17:903:8c8:b0:2b4:65ab:57cd with SMTP id d9443c01a7336-2bd275c0c48mr35313305ad.36.1778677867293; Wed, 13 May 2026 06:11:07 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:06 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 39/69] mm/sparse-vmemmap: Switch DAX to vmemmap_shared_tail_page() Date: Wed, 13 May 2026 21:05:07 +0800 Message-ID: <20260513130542.35604-40-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit DAX compound vmemmap population still has its own way to find a reusable tail page by walking the previous section's PTEs. Switch it to the common vmemmap_shared_tail_page() helper instead, so DAX uses the same per-zone shared tail page as the other vmemmap optimization users. This removes the PTE walk and lets both the section reuse path and the populate path use the same shared page directly. When the target zone is ZONE_DEVICE, mark the shared tail page entries PG_reserved as well, so they match the initialization requirements for device pages. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 10 +++++++++ mm/memory_hotplug.c | 9 ++++++-- mm/sparse-vmemmap.c | 48 ++++++++++++++---------------------------- 3 files changed, 33 insertions(+), 34 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5285d53b0c53..7484e7be7b6d 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1693,11 +1693,21 @@ static inline bool zone_is_zone_device(const struct zone *zone) { return zone_idx(zone) == ZONE_DEVICE; } + +static inline struct zone *device_zone(int nid) +{ + return &NODE_DATA(nid)->node_zones[ZONE_DEVICE]; +} #else static inline bool zone_is_zone_device(const struct zone *zone) { return false; } + +static inline struct zone *device_zone(int nid) +{ + return NULL; +} #endif /* diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 462d8dcd636d..9ff830703785 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -551,8 +551,13 @@ void remove_pfn_range_from_zone(struct zone *zone, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - page_init_poison(pfn_to_page(pfn), - sizeof(struct page) * cur_nr_pages); + /* + * This is a temporary workaround to prevent the shared vmemmap + * page from being overwritten; it will be removed later. + */ + if (!zone_is_zone_device(zone)) + page_init_poison(pfn_to_page(pfn), + sizeof(struct page) * cur_nr_pages); } /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 53a341fcde74..0c0b54e94c07 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -329,8 +329,12 @@ struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zon if (!addr) return NULL; - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail((struct page *)addr + i, NULL, order, zone); + for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page = (struct page *)addr + i; + if (zone_is_zone_device(zone)) + __SetPageReserved(page); + init_compound_tail(page, NULL, order, zone); + } page = virt_to_page(addr); if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) { @@ -442,23 +446,6 @@ static bool __meminit reuse_compound_section(unsigned long start_pfn, return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; } -static pte_t * __meminit compound_section_tail_page(unsigned long addr) -{ - pte_t *pte; - - addr -= PAGE_SIZE; - - /* - * Assuming sections are populated sequentially, the previous section's - * page data can be reused. - */ - pte = pte_offset_kernel(pmd_off_k(addr), addr); - if (!pte) - return NULL; - - return pte; -} - static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -467,19 +454,15 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long size, addr; pte_t *pte; int rc; + struct page *page; - if (reuse_compound_section(start_pfn, pgmap)) { - pte = compound_section_tail_page(start); - if (!pte) - return -ENOMEM; + page = vmemmap_shared_tail_page(pgmap->vmemmap_shift, device_zone(node)); + if (!page) + return -ENOMEM; - /* - * Reuse the page that was populated in the prior iteration - * with just tail struct pages. - */ + if (reuse_compound_section(start_pfn, pgmap)) return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte))); - } + page_to_pfn(page)); size = min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); for (addr = start; addr < end; addr += size) { @@ -497,12 +480,12 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return -ENOMEM; /* - * Reuse the previous page for the rest of tail pages + * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ next += PAGE_SIZE; rc = vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte))); + page_to_pfn(page)); if (rc) return -ENOMEM; } @@ -828,7 +811,8 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - page_init_poison(memmap, sizeof(struct page) * nr_pages); + if (!vmemmap_can_optimize(altmap, pgmap)) + page_init_poison(memmap, sizeof(struct page) * nr_pages); ms = __nr_to_section(section_nr); __section_mark_present(ms, section_nr); -- 2.54.0