From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793123508; b=BwcUVdZ9+2uoBlonfjVxFQItje4doFn657CD7abEtveFQQKx0K5Sf70UsfPv7HRVYYXP v9wDk7V38GWgx4p0bmtgrV2Rvbg7PzAeCNNC5o/mVELhSyFawdpcvKLHstub3oJLGs4v4 i4jf/uh5ZTC5YV6T8ZGdajopvuwuiRGSao6Cy4JQ+aU8eD5cJfvgO4cxdhcw7aVcExrg+ jRXUEDojPY0l/oaQXg8PbOoBc13BADxakN/9NxGijFO3oHQ9qSTyYKL3lkSkzKe/ac+cM 8H9W7g2siy/v2fsHTH18bnG2Jvr3tGdl+eb4G3HJmo3Slu6ckHH3dcJX8olnBkrPxAA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793123508; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:Date:From:To:Cc: Subject:Message-ID:References:MIME-Version:Content-Type: Content-Disposition:In-Reply-To; bh=7cvs2Xtx0PtgOTTn5QWjiXGseQMOAfWpyXw4zUPemHU=; b=zKhyp8pAOSm82aBc0RQLc+iyei846H8b8WyX17jYECwmf5WZmHeN3onTfyGzZVOuT5tD I0FwmJHDoO8xE4sgfX8gtIpcVsref3hL1fyISsI4+AJ+gyVi7XcrbN6mR7czQ+bsxjyku 7HuJsyzKRWxCEHYrMHRpvyn+yZ7JExslKNIUKsXHNA/gv+yAOy3xpb8Dyi534ixhYvCQu WVvEdV81dxHsLzWnY5sLo8g9rmhTfAKljRD1+ZRcdbztaGNh+0brMNnwzA8lwC07lHp+W mQIxQ3mEV88WqVgbD+JXJCQjCvBg83+P5u+MuiMK0uXLvQIhfndhmN/5P9fQzHX8gZA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=cWvbx7Yj; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DE2E606A31D for ; Tue, 27 Oct 2026 17:51:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=cWvbx7Yj Received: by smtp.kernel.org (Postfix) with ESMTPSA id A68461F01825; Tue, 27 Oct 2026 17:51:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1793123507; bh=7cvs2Xtx0PtgOTTn5QWjiXGseQMOAfWpyXw4zUPemHU=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=cWvbx7YjveH6CSYMJOvh61G4XDIsBxwYy1hNAAz7nE6mTdn4YuWe4pTbt53LJaYrN zgB2TiZi0QjI+WcAZYWxbLS2dS2GsyLajprsuWpV8ET5L7dy8P0pPF7I4UqlGJPuRq RR9KuLKDAJkfUHRuc/3BFvnLrB9fx8bj0AyUmRLUkjWaPJvWjv4t6NHZkYvASIWUgz ghb9nT8dZKmnObXCdQ8qBRFKLFGs8+Spt00oVv8ez+3jT1YzF54+aiR1GXwsAiqLQf 9ettFJJHIGMVRMDJjN7CoUgDjVZ9GaBkDteQvB88mmO91MogYCzJFeuqelYqNJsKgL WR+IK4Xm2eqVg== Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfauth.ams.internal (Postfix) with ESMTP id 878B8198003A; Tue, 27 Oct 2026 13:51:40 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Tue, 27 Oct 2026 13:51:45 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEJ54V2o5886gXoRzsAs4tIMx+vUoNXgQPnAayI3VlBJBmvGOiJUBSG0i7ubLHch7 0JSiGiwy5lgoftBWGJG0zy4C9NkZdA2vOF9mUL3B7EeOORJ4ronuYngSgrFlXXDC06cx92 eyEuj6JdAFG/fpBg7FnzXxtPnTMwsB81ldWm5qd5aOYSuDIxHWT2yQ27eM3IYLGVXRK66/ KCn01kjg9WmPw2c6ZxaF6Hw0yqQDSf3HmXpVZIX2yK/z7Z9NchnGg4ACPo3bhA9cQvFPi6 JUtXh/FKR5/BoV7TysiI9pnpYh2ldG7/qRJuLit9m1Vao2ILByUDezNgEXKBj6oOQucyr4 7shXctyr5vOxZrHGDfPPI5M7l1CmLcdabzUyw/CnzHo3ezuHnUxHwIjm1UA1oJostydYVM soSXIKaOrdP0FiPJ/nZBWnwFHv5LsLRNN8YhJswcQFses4jUpndRZKSh+HWHVpD6HiX9/j uPrwMmrjwbkiPrnFNhT3J7QTVtkJQQckkB07ncoHO1iJSXfx+Snc9CHXv0YBCxpDD518SG B03H37qKJmQs0Q7s62Sf0cLjXPa281Zzcto2Be3eHBxkwKBNVfFEqI6PVpKWN3/vwYpbWQ R+0GOwGehvrhAkzvFwzCDWQO0UVuE9jBr/CC+ySqi5Zuy8vF3QPkfG20LqLg X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 27 Oct 2026 13:51:39 -0400 (EDT) Date: Tue, 27 Oct 2026 17:51:38 +0000 From: Kiryl Shutsemau To: Muchun Song Cc: "David Hildenbrand (Arm)" , Peter Zijlstra , Andrew Morton , Matthew Wilcox , Usama Arif , Frank van der Linden , Oscar Salvador , Mike Rapoport , Vlastimil Babka , Lorenzo Stoakes , Zi Yan , Baoquan He , Michal Hocko , Johannes Weiner , Jonathan Corbet , Huacai Chen , WANG Xuerui , Palmer Dabbelt , Paul Walmsley , Albert Ou , Alexandre Ghiti , kernel-team@meta.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, loongarch@lists.linux.dev, linux-riscv@lists.infradead.org, Dave Hansen , Andy Lutomirski Subject: Re: [PATCHv7 13/18] mm: Drop fake head checks Message-ID: References: <20261027110028.GH2668564@noisy.programming.kicks-ass.net> <070DDE5C-2183-4A8B-AC2D-891CAE15E581@linux.dev> <4A32F5E6-977D-4720-8723-B19AFBC08FCF@linux.dev> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4A32F5E6-977D-4720-8723-B19AFBC08FCF@linux.dev> On Tue, Oct 27, 2026 at 10:03:54PM +0800, Muchun Song wrote: > Therefore, I am kind of leaning towards removing the RO protection > and restoring RW permissions to solve the race. Okay, what about this: >From e23ab850837b53cc79d57d58f1cfb4266e745c3d Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Tue, 27 Oct 2026 17:23:47 +0000 Subject: [PATCH] mm/hugetlb_vmemmap: map the shared tail page writable page_ref_add_unless_zero() reads the refcount and then updates it with a cmpxchg. A lockless get, such as a page cache lookup or a PFN scanner, can read a non-zero refcount from a page that is then freed and reused as a hugetlb tail page. If HVO remaps its struct page to the shared tail page in between, the cmpxchg writes to read-only vmemmap: CPU0 (lockless get) CPU1 c = atomic_read(&page->_refcount) page freed page allocated as a hugetlb tail HVO remaps the struct page RO try_cmpxchg(&page->_refcount, &c, c + 1) -> write fault on read-only vmemmap The refcount is zero by then and the compare fails, but LOCK CMPXCHG on x86 writes anyway. The kernel oopses when hugetlb pages are allocated at runtime with HVO enabled while a lockless get races with it. Commit bd225530a4c7 ("mm/hugetlb_vmemmap: fix race with speculative PFN walkers") closed the window with RCU on both sides. The RCU was removed together with fake heads, on the wrong belief that it was only needed for them. Map the shared tail page writable instead of restoring the RCU. Every struct page on it is a tail with a zero refcount, so a late cmpxchg fails and leaves the page unchanged. RCU would put rcu_read_lock() back around every get and synchronize_rcu() back into hugetlb allocation and freeing. The cost is the protection against stray writes to tail struct pages. The shared page is shared by all hugetlb folios of the same order in the zone, so such a write now corrupts all of them instead of oopsing. Bootmem folios optimized in early boot get their tails mapped writable and write protected later. Drop the write protect step. Fixes: 32c440d67e6c ("mm: drop fake head checks") Fixes: 01b1d0ffb6f7 ("hugetlb: remove VMEMMAP_SYNCHRONIZE_RCU") Reported-by: David Hildenbrand (Arm) Closes: https://lore.kernel.org/r/ebc7bde1-f52e-4da8-b16f-822d6c0a4d74@kernel.org Cc: stable@vger.kernel.org Signed-off-by: Kiryl Shutsemau (Meta) --- Documentation/mm/vmemmap_dedup.rst | 13 +++++++++---- include/linux/mm.h | 2 -- mm/hugetlb.c | 4 ---- mm/hugetlb_vmemmap.c | 21 +++++++-------------- mm/sparse-vmemmap.c | 23 ----------------------- 5 files changed, 16 insertions(+), 47 deletions(-) diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_dedup.rst index 9fa8642ded48..6cd85aca4b96 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -131,8 +131,15 @@ pages of ``struct page`` (page 1 to page 7) are tail pages. The optimization is only applied when the size of the struct page is a power of 2. In this case, all tail pages of the same order are identical. See compound_head(). This allows us to remap the tail pages of the vmemmap to a -shared, read-only page. The head page is also remapped to a new page. This -allows the original vmemmap pages to be freed. +shared page. The head page is also remapped to a new page. This allows the +original vmemmap pages to be freed. + +The shared page is mapped writable. A lockless get of a page reference can +read a non-zero refcount before the remap and attempt its atomic update +after it. The refcount of a tail page is zero, so the update fails without +changing the page. Nothing else may write to the shared page: it is shared +by all HugeTLB pages of the same size in the zone, and a stray write would +corrupt all of them. Here is how things look after remapping:: @@ -196,8 +203,6 @@ happens at a later stage when we populate the sections. HugeTLB reuses the the head vmemmap page representing, whereas device-dax reuses the tail vmemmap page. This results in only half of the savings compared to HugeTLB. -Deduplicated tail pages are not mapped read-only. - Here's how things look like on device-dax after the sections are populated:: +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ diff --git a/include/linux/mm.h b/include/linux/mm.h index dd09c438fa23..bfaf4ce6d831 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5162,8 +5162,6 @@ int vmemmap_populate(unsigned long start, unsigned long end, int node, int vmemmap_populate_hvo(unsigned long start, unsigned long end, unsigned int order, struct zone *zone, unsigned long headsize); -void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int node, - unsigned long headsize); void vmemmap_populate_print_last(void); #ifdef CONFIG_MEMORY_HOTPLUG void vmemmap_free(unsigned long start, unsigned long end, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index cea25773a6c9..7596704d94b5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3220,10 +3220,6 @@ static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page *m) * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) * here, or MIGRATE_CMA if this was a page allocated through an early CMA * reservation. - * - * In case of vmemmap optimized folios, the tail vmemmap pages are mapped - * read-only, but that's ok - for sparse vmemmap this does not write to - * the page structure. */ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, struct hstate *h) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 917db0984143..77f9d035d04c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -222,10 +222,12 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, VM_WARN_ON_ONCE(!PageTail((const struct page *)addr)); /* - * Remap the tail pages as read-only to catch illegal write - * operation to the tail pages. + * Map the shared tail page writable. A lockless get that read + * a non-zero refcount before the remap can still do its + * cmpxchg after it. The refcount is zero, so the cmpxchg + * fails and leaves the page unchanged. */ - entry = mk_pte(walk->vmemmap_tail, PAGE_KERNEL_RO); + entry = mk_pte(walk->vmemmap_tail, PAGE_KERNEL); } list_add(&page->lru, walk->vmemmap_pages); @@ -629,19 +631,10 @@ static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, nr_to_optimize = 0; list_for_each_entry(folio, folio_list, lru) { int ret; - unsigned long spfn, epfn; - if (boot && folio_test_hugetlb_vmemmap_optimized(folio)) { - /* - * Already optimized by pre-HVO, just map the - * mirrored tail page structs RO. - */ - spfn = (unsigned long)&folio->page; - epfn = spfn + hugetlb_vmemmap_size(h); - vmemmap_wrprotect_hvo(spfn, epfn, folio_nid(folio), - HUGETLB_VMEMMAP_RESERVE_SIZE); + /* Already optimized by pre-HVO */ + if (boot && folio_test_hugetlb_vmemmap_optimized(folio)) continue; - } nr_to_optimize++; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 5a2469fb1838..3f25d4a0bf47 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -299,29 +299,6 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, return vmemmap_populate_range(start, end, node, altmap, -1, 0); } -/* - * Write protect the mirrored tail page structs for HVO. This will be - * called from the hugetlb code when gathering and initializing the - * memblock allocated gigantic pages. The write protect can't be - * done earlier, since it can't be guaranteed that the reserved - * page structures will not be written to during initialization, - * even if CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled. - * - * The PTEs are known to exist, and nothing else should be touching - * these pages. The caller is responsible for any TLB flushing. - */ -void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, - int node, unsigned long headsize) -{ - unsigned long maddr; - pte_t *pte; - - for (maddr = addr + headsize; maddr < end; maddr += PAGE_SIZE) { - pte = virt_to_kpte(maddr); - ptep_set_wrprotect(&init_mm, maddr, pte); - } -} - #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) { -- Kiryl Shutsemau / Kirill A. Shutemov