[PATCH 0/3] mm: fix PTE table overruns with folded PMDs

[PATCH 0/3] mm: fix PTE table overruns with folded PMDs

[David Hildenbrand (Arm)]

Fix 3 pmd_addr_end() abusers that can result in PTE table overruns with
folded PMDs. Let's factor out a helper from a place where we
recently stumbled over something related, and use it to fix these 3
cases.

I'd call these more urgent because they can corrupt memory, although
folded PMDs are uncommon.

I'm planning on picking all 3 up.

Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
David Hildenbrand (Arm) (3):
      mm/userfaultfd: fix PTE table overruns in UFFDIO_MOVE
      mm/rmap: fix PTE table overruns during batched unmap
      mm/rmap: fix PTE table overruns during batched reference checks

 include/linux/pgtable.h | 24 ++++++++++++++++++++++++
 mm/memory.c             |  7 +------
 mm/rmap.c               |  4 ++--
 mm/userfaultfd.c        |  4 ++--
 4 files changed, 29 insertions(+), 10 deletions(-)

---

base-commit: d369ba5a46c65c54b94c94713ca1fb4ad6a87ebf

change-id: 20261111-pte_table_clamped_end-69bbcaf44fe7

--

Cheers,

David

[PATCH 1/3] mm/userfaultfd: fix PTE table overruns in UFFDIO_MOVE

[David Hildenbrand (Arm)]

With a folded PMD, pmd_addr_end() simply returns the supplied end.
move_present_ptes() can therefore walk past the source or destination
PTE table during UFFDIO_MOVE.

pmd_addr_end() is only intended for page table traversal, in patterns
like "next = pmd_addr_end(addr, end);". With folded PMDs, a higher level
has already clamped the range to the PTE table.

Folded PMDs are uncommon, but can occur even on arm64, for example
with 64K pages and a 42-bit virtual address space.

Extract the calculation from folio_pte_range() into
pte_table_clamped_end() and use it to clamp both ranges. This preserves
the wraparound handling from commit f681d4580d5c ("mm: fix address
wraparound when scanning large folio PTEs").

Fixes: 50944692052b ("userfaultfd: opportunistic TLB-flush batching for present pages in MOVE")
Cc: stable@vger.kernel.org
Reported-by: Chris Mason <mason@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
 include/linux/pgtable.h | 24 ++++++++++++++++++++++++
 mm/memory.c             |  7 +------
 mm/userfaultfd.c        |  4 ++--
 3 files changed, 27 insertions(+), 8 deletions(-)

diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 780fe849ff8b7..66d369870a740 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -12,6 +12,7 @@
 #ifdef CONFIG_MMU
 
 #include <linux/mm_types.h>
+#include <linux/minmax.h>
 #include <linux/bug.h>
 #include <linux/errno.h>
 #include <asm-generic/pgtable_uffd.h>
@@ -1518,6 +1519,29 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)
 })
 #endif
 
+/**
+ * pte_table_clamped_end() - Limit a range to a single PTE table.
+ * @addr: Start address.
+ * @end: Exclusive end address.
+ *
+ * Only use pmd_addr_end() when traversing page tables, in a pattern like::
+ *
+ *     next = pmd_addr_end(addr, end);
+ *
+ * Higher levels already bound the range when PMDs are folded. Use this
+ * helper elsewhere to clamp the range regardless of PMD folding.
+ *
+ * Return: The exclusive end of the PTE table containing @addr, capped at @end.
+ */
+static inline unsigned long pte_table_clamped_end(unsigned long addr,
+		unsigned long end)
+{
+	const unsigned long pmd_end = (addr & PMD_MASK) + PMD_SIZE;
+
+	/* Compare inclusive ends to avoid wraparounds. */
+	return min(pmd_end - 1, end - 1) + 1;
+}
+
 /*
  * When walking page tables, we usually want to skip any p?d_none entries;
  * and any p?d_bad entries - reporting the error before resetting to none.
diff --git a/mm/memory.c b/mm/memory.c
index ec3d9d99ccee2..ee792a2c0ccea 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4959,13 +4959,8 @@ static void folio_pte_range(const struct folio *folio,
 {
 	/* Stay within the page table and VMA boundaries. */
 	const unsigned long pmd_start = addr & PMD_MASK;
-	const unsigned long pmd_end = pmd_start + PMD_SIZE;
 	const unsigned long lower = max(pmd_start, vma->vm_start);
-	/*
-	 * We cannot use pmd_addr_end(), as it does not clamp reliably. To
-	 * avoid wraparounds, compute using the inclusive end.
-	 */
-	const unsigned long upper = min(pmd_end - 1, vma->vm_end - 1) + 1;
+	const unsigned long upper = pte_table_clamped_end(addr, vma->vm_end);
 	const unsigned long offset = idx << PAGE_SHIFT;
 
 	/*
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 169f681f9849b..57650e6f040d1 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -1266,8 +1266,8 @@ static long move_present_ptes(struct mm_struct *mm,
 	unsigned long src_start = src_addr;
 	unsigned long src_end;
 
-	len = pmd_addr_end(dst_addr, dst_addr + len) - dst_addr;
-	src_end = pmd_addr_end(src_addr, src_addr + len);
+	len = pte_table_clamped_end(dst_addr, dst_addr + len) - dst_addr;
+	src_end = pte_table_clamped_end(src_addr, src_addr + len);
 	flush_cache_range(src_vma, src_addr, src_end);
 	double_pt_lock(dst_ptl, src_ptl);
 

-- 
2.43.0

[PATCH 2/3] mm/rmap: fix PTE table overruns during batched unmap

[David Hildenbrand (Arm)]

With a folded PMD, pmd_addr_end() simply returns the supplied end.
folio_unmap_pte_batch() can therefore walk past the PTE table when
unmapping a large folio.

Folded PMDs are uncommon, but can occur even on arm64, for example
with 64K pages and a 42-bit virtual address space.

Use pte_table_clamped_end() to clamp the range to the PTE table and
VMA boundaries.

Fixes: 354dffd29575 ("mm: support batched unmap for lazyfree large folios during reclamation")
Cc: stable@vger.kernel.org
Reported-by: Chris Mason <mason@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
 mm/rmap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/rmap.c b/mm/rmap.c
index 0faef46806e5c..491a954b7fa44 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1978,7 +1978,7 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
 		return 1;
 
 	/* We may only batch within a single VMA and a single page table. */
-	end_addr = pmd_addr_end(addr, vma->vm_end);
+	end_addr = pte_table_clamped_end(addr, vma->vm_end);
 	max_nr = (end_addr - addr) >> PAGE_SHIFT;
 
 	/* We only support lazyfree or file folios batching for now ... */

-- 
2.43.0

[PATCH 3/3] mm/rmap: fix PTE table overruns during batched reference checks

[David Hildenbrand (Arm)]

With a folded PMD, pmd_addr_end() simply returns the supplied end.
folio_referenced_one() can therefore walk past the PTE table when
checking references to a large folio.

Folded PMDs are uncommon, but can occur even on arm64, for example
with 64K pages and a 42-bit virtual address space.

Use pte_table_clamped_end() to clamp the range to the PTE table and
VMA boundaries.

Fixes: 52e054f71840 ("mm: rmap: support batched checks of the references for large folios")
Cc: stable@vger.kernel.org
Reported-by: Chris Mason <mason@kernel.org>
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
 mm/rmap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/rmap.c b/mm/rmap.c
index 491a954b7fa44..65646ed6fc453 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -973,7 +973,7 @@ static bool folio_referenced_one(struct folio *folio,
 			pteval = ptep_get(pvmw.pte);
 
 		if (pvmw.pte && pte_present(pteval) && folio_test_large(folio)) {
-			const unsigned long end_addr = pmd_addr_end(address, vma->vm_end);
+			const unsigned long end_addr = pte_table_clamped_end(address, vma->vm_end);
 			const unsigned int max_nr = (end_addr - address) >> PAGE_SHIFT;
 
 			nr = folio_pte_batch(folio, pvmw.pte, pteval, max_nr);

-- 
2.43.0