From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793057013; b=IvVNRC5wtp5hQRNMMW5B5vO1ATtN6WaJHQ+xUcuBNCA5MJi301HqwuvyhU6AJhRFQCyT Bh6HlND6ARXx928Zxvl974SR86up8/kX5eY1+8fI4fcpcyETTw0C/Usau+vVUiHEs1N3D j7T1vcHdsJJDACPX29T5B0NIAjPw81WjY+pQGddPxfZg5E1D9EVG7zVX3JPtWbTZIAf2h cGuIyF2vDSHbJg3r2SEcCSocBvHMUyjwvdaZMd210UnlHqVXYElQkxdm1R40CUEQ/ray4 CJ4ra9Lf5boM1IdHMDm2jAyrNqjcbkqUSdORwY3s6x3PYmj1odj4PWi1mqPLb5x3RYw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793057013; h=Received:DKIM-Signature:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Mailer:X-Developer-Signature:X-Developer-Key; bh=9ghLBkppx9o1pzNOIkyHYAlxKpBzprLGe09MAo7VESI=; b=RZmj2M1sRiOBiv03iYlh8LeueDetEu2BAJxzFKBXxfwiCjoCtXizC1A6mEAMfTA4296R 9aJfMaOpTY7KsRqx9vSQLYHQPf5x6bbdYtelnj2go7xzbN8748Xt+YTjGNZydraYtylyd yI6Z3HtAW1plzkq3r7gVnjaT4Mmp0L4zHBhmUP3lgPV49OFyiA6HjjKAQcP/b4NzoY3t6 xXx2KJGxx1Kgwvu/XhyV6Xpdj3/t+VTjutnYqFR7DWYw6Png9w8Uvbc44Ey7oeCXp0TpG QuD5YWAeH9rpffF7B9TChFhCxQjG/VAdQ5usjZ3IUx/c3++uSgQ4fApbrYLIC2ZDIUQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=PbgM5nER; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EF54606A31B; Mon, 26 Oct 2026 23:23:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=PbgM5nER Received: by smtp.kernel.org (Postfix) with ESMTPSA id 541741F01825; Mon, 26 Oct 2026 23:23:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1793057013; bh=9ghLBkppx9o1pzNOIkyHYAlxKpBzprLGe09MAo7VESI=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=PbgM5nERw3dhz4H5Dx9e87PKvv2YyQhWlLTY9OxNjtJYEM7nn/ZuhTcjU1MrcHIWN pBgbrO56mec/pIE1fyAZvH7HrvAsbuNqo5qh6M5tzowYd7p3y1G8xKRoQtb8GfBVgM ByI1GHKBu5k+ZDRBHK1UHKKIxpwPSGQjhlfkyBo34U9jhKg8+vy2oMjz62tFqrACPf 3wMlXWWaPRJ3wV0jHp9+Dhd/GGIWNfjqPUIJonptqBAEZBWNkQcK5j8U7qWgxJI6xW FgzdNPHCIsrQz00H2YKv8v6WJQt9/0htHvZYhLqRX3YAQDIHHu71V0h2hBW7HuorAG jELF3w+ixIuxQ== From: "David Hildenbrand (Arm)" Date: Tue, 27 Oct 2026 00:17:38 +0100 Subject: [PATCH 18/46] mm/khugepaged: check if PTE table is empty in try_collapse_pte_mapped_thp() X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261027-fixes_core-v1-18-6f74bec9a5bb@kernel.org> References: <20261027-fixes_core-v1-0-6f74bec9a5bb@kernel.org> In-Reply-To: <20261027-fixes_core-v1-0-6f74bec9a5bb@kernel.org> To: Andrew Morton , Johannes Weiner , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Baolin Wang , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Peter Zijlstra , Jann Horn , Pedro Falcato , Hugh Dickins , Pasha Tatashin , Pratyush Yadav , "Matthew Wilcox (Oracle)" , Jan Kara , Zi Yan , Lance Yang , Rik van Riel , Harry Yoo , Alistair Popple , Nico Pache , Ryan Roberts , Dev Jain , Usama Arif , Kiryl Shutsemau , Matthew Brost , John Hubbard , Peter Xu , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Oscar Salvador , Minchan Kim , Muchun Song , Xu Xin , Chengming Zhou , Chris Li , Kemeng Shi , Nhat Pham , Youngjun Park , Gregory Price , Joshua Hahn , Rakie Kim , Ying Huang , Jason Gunthorpe Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kexec@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-cxl@vger.kernel.org, "David Hildenbrand (Arm)" X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5530; i=david@kernel.org; h=from:subject:message-id; bh=EsD8ZlRQvAEJtfiiEkh+iJ41o7HnIWnYdXqeUX1m2uE=; b=owGbwMvMwCF2/OVpCdfQ3ZWMp9WSGLLu35/68IKvRf+nzZv6FtVNl2nVTRT+UcG1Jp3v/B73r C+ZhcWpHaUsDGIcDLJiiixBwfcNXr5cV1vtxscMM4eVCWQIAxenAEzE/Tkjw+tLZfx3W3S8Hvkq 7b+tY61bdOaqpYfRYo//2zaYBpoo+zEyHHsUVaK369iafwXRBzUMmhcGfHm9LM3JXrly+/vCwrh 6fgA= X-Developer-Key: i=david@kernel.org; a=openpgp; fpr=1BD9CAAD735C4C3A460DFCCA4DDE10F700FF835A The existing code is rather horrible and already resulted in bugs in the past: we special-case UFFD inserting anonymous folios by grabbing the PMD lock early, but ignore any other markers (guard, poison through uffd) that could have been inserted into the page table in the meantime, effectively dropping them. Let's just unify and simplify the locking: try to grab the PMD table lock and if that fails, flush the TLB, drop the PTE lock, take PT locks in the right order and check if the page table is still empty. We still perform the heavy lifting (zapping the folio ptes) under PTE table lock only. Include for now, as x86's flush_tlb_range() macro depends on hugetlb helpers. This dependency needs a separate cleanup. Both issues this fixes are rather corner cases. Let's not CC stable for now. Fixes: f807123d578d ("mm: allow guard regions in file-backed and read-only mappings") Fixes: 1043173eb5eb ("mm/khugepaged: collapse_pte_mapped_thp() with mmap_read_lock()") Signed-off-by: David Hildenbrand (Arm) --- mm/khugepaged.c | 64 ++++++++++++++++++++++++++++----------------------------- 1 file changed, 31 insertions(+), 33 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index dd17c3dae8006..4313abf9cfcb9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -1832,6 +1833,17 @@ static enum scan_result set_huge_pmd(struct vm_area_struct *vma, unsigned long a return SCAN_SUCCEED; } +static bool pte_table_empty(pte_t *pte) +{ + int i; + + for (i = 0; i < HPAGE_PMD_NR; i++, pte++) { + if (!pte_none(ptep_get(pte))) + return false; + } + return true; +} + static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, bool install_pmd) { @@ -1935,33 +1947,14 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign } pte_unmap_unlock(start_pte, ptl); - ptl = NULL; mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddr, haddr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); notified = true; - /* - * pmd_lock covers a wider range than ptl, and (if split from mm's - * page_table_lock) ptl nests inside pml. The less time we hold pml, - * the better; but userfaultfd's mfill_atomic_pte() on a private VMA - * inserts a valid as-if-COWed PTE without even looking up page cache. - * So page lock of folio does not protect from it, so we must not drop - * ptl before pgt_pmd is removed, so uffd private needs pml taken now. - */ - if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) - pml = pmd_lock(mm, pmd); - - start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); + start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); if (!start_pte) goto abort; - if (!pml) - spin_lock(ptl); - else if (ptl != pml) - spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); - - if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) - goto abort; /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; @@ -1992,10 +1985,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign nr_batch_ptes = folio_pte_batch(folio, pte, ptent, max_nr_batch_ptes); - /* - * Must clear entry, or a racing truncate may re-remove it. - * TLB flush can be left until pmdp_collapse_flush() does it. - */ + /* Must clear entry, or a racing truncate may re-remove it. */ ptent = get_and_clear_ptes(mm, addr, pte, nr_batch_ptes); if (pte_dirty(ptent)) folio_mark_dirty(folio); @@ -2003,25 +1993,33 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign nr_mapped_ptes += nr_batch_ptes; } - if (!pml) - spin_unlock(ptl); - /* step 3: set proper refcount and mm_counters. */ if (nr_mapped_ptes) { folio_ref_sub(folio, nr_mapped_ptes); add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes); } - /* step 4: remove empty page table */ - if (!pml) { + /* step 4: acquire the PMD lock and remove the empty page table */ + pml = pmd_lockptr(mm, pmd); + if (ptl != pml && !spin_trylock(pml)) { + /* We must flush before dropping the PTL. */ + if (nr_mapped_ptes) + flush_tlb_range(vma, haddr, end); + pte_unmap_unlock(start_pte, ptl); + ptl = NULL; + pml = pmd_lock(mm, pmd); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); + if (!start_pte) + goto unlock; if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); - if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { - flush_tlb_mm(mm); + + if (!pte_table_empty(start_pte)) goto unlock; - } } + + /* This also flushes the zapped folio ptes properly. */ pgt_pmd = pmdp_collapse_flush(vma, haddr, pmd); pmdp_get_lockless_sync(); pte_unmap_unlock(start_pte, ptl); @@ -2042,7 +2040,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign goto drop_folio; abort: if (nr_mapped_ptes) { - flush_tlb_mm(mm); + flush_tlb_range(vma, haddr, end); folio_ref_sub(folio, nr_mapped_ptes); add_mm_counter(mm, mm_counter_file(folio), -nr_mapped_ptes); } -- 2.43.0