mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling
@ 2026-11-16  7:13 Hao Ge
  2026-11-16  7:13 ` [PATCH v5 01/11] mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic Hao Ge
                   ` (10 more replies)
  0 siblings, 11 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

__vmap_pages_range_noflush() and friends can install some PTEs before
failing and leave them mapped, and the kernel callers do not agree on
who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range()
roll back what they mapped before the failure, while
vm_module_tags_populate() and the __GFP_NOFAIL retry loop in
__vmalloc_area_node() rely on the mapping functions cleaning up and
do not call anything like vunmap_range() themselves. When the same
range is mapped again, the attempt hits the leftovers and fails with
a warning in vmap_pages_pte_range(). A leftover huge pmd is worse.
During the retry, vmap_try_huge_pmd() calls pmd_free_pte_page() on
it, and the x86 implementation cannot tell a leaf entry from a page
table and frees the mapped page.

After discussing with Suren and Ulad [1], we move the rollback into
the mapping functions. A failed vmap leaves nothing mapped, and the
callers no longer clean up after it.

Patch 1 gives __vunmap_range_noflush() a can_resched argument, the
failure paths can be reached with a non-blocking gfp mask and the
undo must not schedule there. Patch 2 is the KMSAN part, the shadow
and the origin of a range are mapped by two separate calls, so the
rollback there has to cover both. Patches 3 and 4 reject an unaligned
range at the entries of vmap_page_range() and __vunmap_range_noflush().
Patch 5 only flushes the cache once the mapping succeeded. Patches 6
to 9 add the rollback to the mapping functions, one entry per patch,
vmap_page_range(), __vmap_pages_range_noflush(), the KMSAN metadata
and the batched vmap() path. Patches 10 and 11 drop the powerpc and
percpu cleanups that are not needed anymore.

Changes since v4:
- new patch 1, the unmap paths can run under a non-blocking gfp mask
  and must skip cond_resched(); it carries a Fixes: tag so it travels
  to stable together with patch 2
- drop v4 patch 2, the KMSAN-disabled vunmap skip. !kmsan_enabled
  does not mean no metadata was installed, KMSAN_WARN_ON() can clear
  the flag at runtime, on Suren's review
- v4 patch 3 now warns and returns -EINVAL at vmap_page_range(),
  instead of BUG_ON()ing inside vmap_range_noflush()
- drop v4 patch 4, the walk check. No in-tree caller passes an
  unaligned range to vmap_pages_range_noflush_walk(), the vmap
  allocator guarantees the alignment for vmalloc()/vmap()/
  vm_map_ram()
- v4 patch 5 warns and bails out instead of BUG_ON(), following
  Ulad's review of v4
- v4 patch 7 is split into patches 6 to 9, one mapping entry per
  patch. The batched vmap() path is covered now too, on Suren's
  request

[1] https://lore.kernel.org/all/20260915070001.113559-3-hao.ge@linux.dev/

v4 link: https://lore.kernel.org/all/20261110112546.261415-1-hao.ge@linux.dev/
v3 link: https://lore.kernel.org/all/20261026080329.140244-1-hao.ge@linux.dev/
v2 link: https://lore.kernel.org/all/20261019054531.87652-1-hao.ge@linux.dev/
v1 link: https://lore.kernel.org/all/20261009063619.112313-1-hao.ge@linux.dev/

Hao Ge (11):
  mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic
  mm/kmsan: undo the shadow mapping when the origin mapping fails
  mm/vmalloc: reject an unaligned range in vmap_page_range()
  mm/vmalloc: reject an unaligned range in __vunmap_range_noflush()
  mm/vmalloc: only flush the cache after the vmap succeeded
  mm/vmalloc: undo what vmap_page_range() mapped on failure
  mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure
  mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() on
    failure
  mm/vmalloc: undo what vmap_pages_range_batched() mapped on failure
  powerpc: drop redundant unmaps of failed vmap mappings
  mm/percpu: stop unmapping the CPU that failed to map

 arch/powerpc/kernel/isa-bridge.c |   5 +-
 arch/powerpc/kernel/pci_64.c     |   4 +-
 include/linux/kmsan.h            |   8 +-
 mm/kmsan/hooks.c                 |  18 ++--
 mm/kmsan/shadow.c                |   9 +-
 mm/percpu-vm.c                   |   5 +-
 mm/vmalloc.c                     | 154 +++++++++++++++++++++++--------
 mm/vmalloc.h                     |   6 +-
 8 files changed, 151 insertions(+), 58 deletions(-)

-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 01/11] mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 02/11] mm/kmsan: undo the shadow mapping when the origin mapping fails Hao Ge
                   ` (9 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge, stable, Sashiko

The vmap failure paths can be reached with GFP_ATOMIC or GFP_NOWAIT
since commit b186a94227b7 ("kmsan: remove hard-coded GFP_KERNEL
flags") and commit 8da89ba18ed4 ("mm/vmalloc: handle non-blocking
GFP in __vmalloc_area_node()"), so an undo of a partially installed
mapping may have to run in atomic context, where the cond_resched()
in vunmap_pmd_range() cannot schedule.

Give __vunmap_range_noflush() and kmsan_vunmap_range_noflush() a
can_resched argument and skip the cond_resched() when it is false.
The existing callers are on sleepable paths and pass true, so nothing
changes for them.

Fixes: 47ebd0310e89 ("mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()")
Cc: stable@vger.kernel.org
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-178439@kernel.org/
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 include/linux/kmsan.h |  8 ++++++--
 mm/kmsan/hooks.c      | 14 +++++++++-----
 mm/vmalloc.c          | 24 ++++++++++++++----------
 mm/vmalloc.h          |  3 ++-
 4 files changed, 31 insertions(+), 18 deletions(-)

diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h
index 7da9fd506b39..0016c032d1e7 100644
--- a/include/linux/kmsan.h
+++ b/include/linux/kmsan.h
@@ -150,11 +150,14 @@ int __must_check kmsan_vmap_pages_range_noflush(unsigned long start,
  * kmsan_vunmap_kernel_range_noflush() - Notify KMSAN about a vunmap.
  * @start: start of vunmapped range.
  * @end:   end of vunmapped range.
+ * @can_resched: whether the unmap walk may call cond_resched(); pass
+ * false from atomic context.
  *
  * KMSAN unmaps the contiguous metadata ranges created by
  * kmsan_map_kernel_range_noflush().
  */
-void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end);
+void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end,
+				bool can_resched);
 
 /**
  * kmsan_ioremap_page_range() - Notify KMSAN about a ioremap_page_range() call.
@@ -355,7 +358,8 @@ static inline int __must_check kmsan_vmap_pages_range_noflush(
 }
 
 static inline void kmsan_vunmap_range_noflush(unsigned long start,
-					      unsigned long end)
+					      unsigned long end,
+					      bool can_resched)
 {
 }
 
diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c
index a706666db097..69008b085658 100644
--- a/mm/kmsan/hooks.c
+++ b/mm/kmsan/hooks.c
@@ -136,10 +136,13 @@ static unsigned long vmalloc_origin(unsigned long addr)
 						 KMSAN_META_ORIGIN);
 }
 
-void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)
+void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end,
+				bool can_resched)
 {
-	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end));
-	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end));
+	__vunmap_range_noflush(vmalloc_shadow(start), vmalloc_shadow(end),
+			       can_resched);
+	__vunmap_range_noflush(vmalloc_origin(start), vmalloc_origin(end),
+			       can_resched);
 	flush_cache_vmap(vmalloc_shadow(start), vmalloc_shadow(end));
 	flush_cache_vmap(vmalloc_origin(start), vmalloc_origin(end));
 }
@@ -168,8 +171,9 @@ static void kmsan_iounmap_pages(unsigned long start, unsigned long end)
 		if (origin)
 			__free_pages(origin, KMSAN_IOREMAP_META_ORDER);
 	}
-	__vunmap_range_noflush(shadow_start, shadow_end);
-	__vunmap_range_noflush(origin_start, origin_end);
+	/* ioremap metadata mappings are always GFP_KERNEL, so resched is safe. */
+	__vunmap_range_noflush(shadow_start, shadow_end, true);
+	__vunmap_range_noflush(origin_start, origin_end, true);
 	flush_tlb_kernel_range(shadow_start, shadow_end);
 	flush_tlb_kernel_range(origin_start, origin_end);
 }
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index ccaf158c5b7f..3c0d9e69b886 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -429,7 +429,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 }
 
 static void vunmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
-			     pgtbl_mod_mask *mask)
+			     pgtbl_mod_mask *mask, bool can_resched)
 {
 	pmd_t *pmd;
 	unsigned long next;
@@ -451,12 +451,14 @@ static void vunmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
 			continue;
 		vunmap_pte_range(pmd, addr, next, mask);
 
-		cond_resched();
+		/* The undo path may run in atomic context. */
+		if (can_resched)
+			cond_resched();
 	} while (pmd++, addr = next, addr != end);
 }
 
 static void vunmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
-			     pgtbl_mod_mask *mask)
+			     pgtbl_mod_mask *mask, bool can_resched)
 {
 	pud_t *pud;
 	unsigned long next;
@@ -476,12 +478,12 @@ static void vunmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
 		}
 		if (pud_none_or_clear_bad(pud))
 			continue;
-		vunmap_pmd_range(pud, addr, next, mask);
+		vunmap_pmd_range(pud, addr, next, mask, can_resched);
 	} while (pud++, addr = next, addr != end);
 }
 
 static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
-			     pgtbl_mod_mask *mask)
+			     pgtbl_mod_mask *mask, bool can_resched)
 {
 	p4d_t *p4d;
 	unsigned long next;
@@ -496,7 +498,7 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 
 		if (p4d_none_or_clear_bad(p4d))
 			continue;
-		vunmap_pud_range(p4d, addr, next, mask);
+		vunmap_pud_range(p4d, addr, next, mask, can_resched);
 	} while (p4d++, addr = next, addr != end);
 }
 
@@ -512,7 +514,8 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
  *
  * This is an internal function only. Do not use outside mm/.
  */
-void __vunmap_range_noflush(unsigned long start, unsigned long end)
+void __vunmap_range_noflush(unsigned long start, unsigned long end,
+			    bool can_resched)
 {
 	unsigned long next;
 	pgd_t *pgd;
@@ -527,7 +530,7 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end)
 			mask |= PGTBL_PGD_MODIFIED;
 		if (pgd_none_or_clear_bad(pgd))
 			continue;
-		vunmap_p4d_range(pgd, addr, next, &mask);
+		vunmap_p4d_range(pgd, addr, next, &mask, can_resched);
 	} while (pgd++, addr = next, addr != end);
 
 	if (mask & ARCH_PAGE_TABLE_SYNC_MASK)
@@ -536,8 +539,9 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end)
 
 void vunmap_range_noflush(unsigned long start, unsigned long end)
 {
-	kmsan_vunmap_range_noflush(start, end);
-	__vunmap_range_noflush(start, end);
+	/* The unmap paths are sleepable, keep the resched point. */
+	kmsan_vunmap_range_noflush(start, end, true);
+	__vunmap_range_noflush(start, end, true);
 }
 
 /**
diff --git a/mm/vmalloc.h b/mm/vmalloc.h
index d8372a4f65c4..4ec114a75f04 100644
--- a/mm/vmalloc.h
+++ b/mm/vmalloc.h
@@ -44,6 +44,7 @@ int __must_check __vmap_pages_range_noflush(unsigned long addr,
 
 void vunmap_range_noflush(unsigned long start, unsigned long end);
 
-void __vunmap_range_noflush(unsigned long start, unsigned long end);
+void __vunmap_range_noflush(unsigned long start, unsigned long end,
+			    bool can_resched);
 
 #endif /* __MM_VMALLOC_H */
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 02/11] mm/kmsan: undo the shadow mapping when the origin mapping fails
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
  2026-11-16  7:13 ` [PATCH v5 01/11] mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 03/11] mm/vmalloc: reject an unaligned range in vmap_page_range() Hao Ge
                   ` (8 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge, stable

kmsan_vmap_pages_range_noflush() first maps the shadow pages and then
the origin pages. If the second mapping fails, the first one is left
mapped, and the callers do not roll it back. The next vmap of the
same metadata range hits the stale PTEs again and fails with a
WARN_ON() and -EBUSY in vmap_pages_pte_range().

Undo the shadow mapping on that error path, the same way
kmsan_ioremap_page_range() cleans up after a partial failure.
__vunmap_range_noflush() only clears the PTEs; nothing has touched
the shadow mapping, so no TLB flush is needed.

Fixes: 47ebd0310e89 ("mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()")
Cc: stable@vger.kernel.org
Suggested-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/kmsan/shadow.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 0c88d89bf0d6..212449db34a1 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -258,6 +258,11 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 					    o_pages, page_shift);
 	kmsan_leave_runtime();
 	if (mapped) {
+		/* Undo the shadow mapping set up above. */
+		kmsan_enter_runtime();
+		__vunmap_range_noflush(shadow_start, shadow_end,
+				       gfpflags_allow_blocking(gfp_mask));
+		kmsan_leave_runtime();
 		err = mapped;
 		goto ret;
 	}
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 03/11] mm/vmalloc: reject an unaligned range in vmap_page_range()
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
  2026-11-16  7:13 ` [PATCH v5 01/11] mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic Hao Ge
  2026-11-16  7:13 ` [PATCH v5 02/11] mm/kmsan: undo the shadow mapping when the origin mapping fails Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 04/11] mm/vmalloc: reject an unaligned range in __vunmap_range_noflush() Hao Ge
                   ` (7 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

commit a739e6b557af ("mm: vmalloc: WARN_ON if mapping size is not
PAGE_SIZE aligned") put the check in vmap_pte_range(), which is too
late, because the upper levels may already have installed page tables
for a range that is rejected anyway.

The pte loop maps whole pages and stops only when addr lands exactly
on end. An unaligned end lets addr jump over it and the loop keeps
mapping past the range. An unaligned start maps the page below the
range and usually misses end too. Such a range is never a legitimate
request.

vmap_range_noflush() has a single caller, vmap_page_range(). Warn on
addr and end there and bail out with -EINVAL before any page table is
installed, and drop the check from vmap_pte_range().

Suggested-by: Suren Baghdasaryan <surenb@google.com>
Suggested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/vmalloc.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 3c0d9e69b886..a168f6f20f4f 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -125,9 +125,6 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 	unsigned long size;
 	unsigned int nr_pages;
 
-	if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
-		return -EINVAL;
-
 	pfn = phys_addr >> PAGE_SHIFT;
 	pte = pte_alloc_kernel_track(pmd, addr, mask);
 	if (!pte)
@@ -369,6 +366,9 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 {
 	int err;
 
+	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
+		return -EINVAL;
+
 	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
 				 ioremap_max_page_shift);
 	flush_cache_vmap(addr, end);
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 04/11] mm/vmalloc: reject an unaligned range in __vunmap_range_noflush()
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (2 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 03/11] mm/vmalloc: reject an unaligned range in vmap_page_range() Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 05/11] mm/vmalloc: only flush the cache after the vmap succeeded Hao Ge
                   ` (6 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge, Sashiko

The PTE loop stops only at addr == end. An unaligned end is never
reached, so the loop keeps clearing past the range. An unaligned
start clears the page below the range first. Both are caller bugs,
and the damage is hard to attribute afterwards.

vmap only installs page aligned mappings, so an unaligned unmap
request is never a legitimate teardown. Warn and bail out at the
entry of __vunmap_range_noflush(), before any PTE is cleared.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-178439@kernel.org/
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/vmalloc.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index a168f6f20f4f..e2b63cb1592b 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -523,6 +523,9 @@ void __vunmap_range_noflush(unsigned long start, unsigned long end,
 	pgtbl_mod_mask mask = 0;
 
 	BUG_ON(addr >= end);
+	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
+		return;
+
 	pgd = pgd_offset_k(addr);
 	do {
 		next = pgd_addr_end(addr, end);
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 05/11] mm/vmalloc: only flush the cache after the vmap succeeded
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (3 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 04/11] mm/vmalloc: reject an unaligned range in __vunmap_range_noflush() Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 06/11] mm/vmalloc: undo what vmap_page_range() mapped on failure Hao Ge
                   ` (5 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge, Sashiko

vmap_page_range(), __vmap_pages_range() and vmap_pages_range_batched()
call flush_cache_vmap() before they even look at the error code. A
failed mapping is never used, so the flush is pointless at best.

According to the Sashiko review it is worse than pointless on UML:
flush_cache_vmap() there is flush_tlb_kernel_range(), which syncs the
PTEs into the host OS. Flushing a half-installed mapping therefore
leaves host mappings behind for a range that was just freed again.

Only flush when the mapping, and for vmap_page_range() also the KMSAN
metadata, is fully set up.

Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-178439@kernel.org/
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/vmalloc.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index e2b63cb1592b..d99d7710e1ba 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -371,10 +371,12 @@ int vmap_page_range(unsigned long addr, unsigned long end,
 
 	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
 				 ioremap_max_page_shift);
-	flush_cache_vmap(addr, end);
 	if (!err)
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
+	if (!err)
+		flush_cache_vmap(addr, end);
+
 	return err;
 }
 
@@ -763,7 +765,9 @@ static int __vmap_pages_range(unsigned long addr, unsigned long end,
 	int err;
 
 	err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift, gfp_mask);
-	flush_cache_vmap(addr, end);
+	if (!err)
+		flush_cache_vmap(addr, end);
+
 	return err;
 }
 
@@ -3729,7 +3733,8 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 				pages + batch_idx, prev_shift);
 
 out:
-	flush_cache_vmap(addr, end);
+	if (!err)
+		flush_cache_vmap(addr, end);
 	return err;
 }
 
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 06/11] mm/vmalloc: undo what vmap_page_range() mapped on failure
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (4 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 05/11] mm/vmalloc: only flush the cache after the vmap succeeded Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 07/11] mm/vmalloc: undo what __vmap_pages_range_noflush() " Hao Ge
                   ` (4 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

ioremap and arch setup code map their ranges with vmap_page_range().
When the walk fails half way, or the KMSAN metadata fails after the
walk, the ptes installed so far stay behind. The caller frees the
vm_area on the error path, so the next mapping that lands on the same
range trips the BUG() in vmap_pte_range().

vmap_range_noflush() now takes a mapped_end argument that the walk
levels advance as they install, and vmap_pte_range() BUG()s before
it installs over a pte that is already set. The ptes below mapped_end
are therefore all ours, while a pte above it can only have been put
there by somebody else, for example when the caller's range overlaps
an existing mapping. On a walk failure the undo is limited to
[addr, mapped_end), going further would clear ptes that are not
ours. If the walk completed and only the KMSAN metadata failed, the
whole range was installed by this call and all of it is undone.

All callers are sleepable, the teardown passes can_resched = true.

Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/vmalloc.c | 52 +++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 39 insertions(+), 13 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index d99d7710e1ba..5c1eacc2a0ac 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -117,7 +117,8 @@ static __always_inline unsigned long vmap_set_ptes(pte_t *ptep,
 
 static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	pte_t *pte;
 	u64 pfn;
@@ -143,6 +144,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 
 		size = vmap_set_ptes(pte, addr, end, pfn, prot, max_page_shift);
 		nr_pages = PFN_DOWN(size);
+		*mapped_end = addr + size;
 	} while (pte += nr_pages, pfn += nr_pages, addr += size, addr != end);
 
 	lazy_mmu_mode_disable();
@@ -188,7 +190,8 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end,
 
 static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	pmd_t *pmd;
 	unsigned long next;
@@ -203,10 +206,12 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,
 		if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot,
 					max_page_shift)) {
 			*mask |= PGTBL_PMD_MODIFIED;
+			*mapped_end = next;
 			continue;
 		}
 
-		err = vmap_pte_range(pmd, addr, next, phys_addr, prot, max_page_shift, mask);
+		err = vmap_pte_range(pmd, addr, next, phys_addr, prot,
+				     max_page_shift, mask, mapped_end);
 		if (err)
 			break;
 	} while (pmd++, phys_addr += (next - addr), addr = next, addr != end);
@@ -245,7 +250,8 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end,
 
 static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	pud_t *pud;
 	unsigned long next;
@@ -260,10 +266,12 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,
 		if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot,
 					max_page_shift)) {
 			*mask |= PGTBL_PUD_MODIFIED;
+			*mapped_end = next;
 			continue;
 		}
 
-		err = vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask);
+		err = vmap_pmd_range(pud, addr, next, phys_addr, prot,
+				     max_page_shift, mask, mapped_end);
 		if (err)
 			break;
 	} while (pud++, phys_addr += (next - addr), addr = next, addr != end);
@@ -302,7 +310,8 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end,
 
 static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift, pgtbl_mod_mask *mask)
+			unsigned int max_page_shift, pgtbl_mod_mask *mask,
+			unsigned long *mapped_end)
 {
 	p4d_t *p4d;
 	unsigned long next;
@@ -317,10 +326,12 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 		if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot,
 					max_page_shift)) {
 			*mask |= PGTBL_P4D_MODIFIED;
+			*mapped_end = next;
 			continue;
 		}
 
-		err = vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask);
+		err = vmap_pud_range(p4d, addr, next, phys_addr, prot,
+				     max_page_shift, mask, mapped_end);
 		if (err)
 			break;
 	} while (p4d++, phys_addr += (next - addr), addr = next, addr != end);
@@ -329,7 +340,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,
 
 static int vmap_range_noflush(unsigned long addr, unsigned long end,
 			phys_addr_t phys_addr, pgprot_t prot,
-			unsigned int max_page_shift)
+			unsigned int max_page_shift, unsigned long *mapped_end)
 {
 	pgd_t *pgd;
 	unsigned long start;
@@ -346,11 +357,12 @@ static int vmap_range_noflush(unsigned long addr, unsigned long end,
 	BUG_ON(addr >= end);
 
 	start = addr;
+	*mapped_end = addr;
 	pgd = pgd_offset_k(addr);
 	do {
 		next = pgd_addr_end(addr, end);
 		err = vmap_p4d_range(pgd, addr, next, phys_addr, prot,
-					max_page_shift, &mask);
+					max_page_shift, &mask, mapped_end);
 		if (err)
 			break;
 	} while (pgd++, phys_addr += (next - addr), addr = next, addr != end);
@@ -364,18 +376,32 @@ static int vmap_range_noflush(unsigned long addr, unsigned long end,
 int vmap_page_range(unsigned long addr, unsigned long end,
 		    phys_addr_t phys_addr, pgprot_t prot)
 {
+	unsigned long mapped_end = addr;
 	int err;
 
 	if (WARN_ON_ONCE(!PAGE_ALIGNED(addr) || !PAGE_ALIGNED(end)))
 		return -EINVAL;
 
 	err = vmap_range_noflush(addr, end, phys_addr, pgprot_nx(prot),
-				 ioremap_max_page_shift);
-	if (!err)
+				 ioremap_max_page_shift, &mapped_end);
+	if (err) {
+		/* Undo the part the walk installed before it failed. */
+		if (mapped_end > addr)
+			__vunmap_range_noflush(addr, mapped_end, true);
+	} else {
 		err = kmsan_ioremap_page_range(addr, end, phys_addr, prot,
 					       ioremap_max_page_shift);
-	if (!err)
-		flush_cache_vmap(addr, end);
+		if (err) {
+			/*
+			 * Only the KMSAN metadata failed, the walk mapped
+			 * the whole range. Undo it all. The callers are
+			 * ioremap and arch setup code, always sleepable.
+			 */
+			__vunmap_range_noflush(addr, end, true);
+		} else {
+			flush_cache_vmap(addr, end);
+		}
+	}
 
 	return err;
 }
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 07/11] mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (5 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 06/11] mm/vmalloc: undo what vmap_page_range() mapped on failure Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 08/11] mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() " Hao Ge
                   ` (3 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

__vmap_pages_range_noflush() maps a struct page array and may fail
half way. The ptes installed so far stay behind, so a retry of the
same range, the __GFP_NOFAIL loop in __vmalloc_area_node() or the
module tags populate, trips the WARN_ON(!pte_none()) in
vmap_pages_pte_range() and fails with -EBUSY.

vmap_pages_range_noflush_walk() now reports how many pages it
installed through its running index into the pages array, and
__vmap_pages_range_noflush() undoes [addr, addr + mapped_pages *
PAGE_SIZE) on failure. vmap_pages_pte_range() fails with -EBUSY
instead of installing over a pte that is already set, so the ptes
below that point are all ours, while a pte above it can only have
been put there by somebody else, for example when the caller's range
overlaps an existing mapping. Going further would clear ptes that
are not ours.

This path can run with a non-blocking gfp mask, the undo passes
gfpflags_allow_blocking() as can_resched.

Suggested-by: Suren Baghdasaryan <surenb@google.com>
Suggested-by: Uladzislau Rezki <urezki@gmail.com>
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/kmsan/hooks.c  |  4 ++--
 mm/kmsan/shadow.c |  4 ++--
 mm/vmalloc.c      | 36 +++++++++++++++++++++++++++++-------
 mm/vmalloc.h      |  3 ++-
 4 files changed, 35 insertions(+), 12 deletions(-)

diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c
index 69008b085658..d830a4e649f1 100644
--- a/mm/kmsan/hooks.c
+++ b/mm/kmsan/hooks.c
@@ -208,7 +208,7 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end,
 		mapped = __vmap_pages_range_noflush(
 			vmalloc_shadow(start + off),
 			vmalloc_shadow(start + off + PAGE_SIZE), prot, &shadow,
-			PAGE_SHIFT);
+			PAGE_SHIFT, gfp_mask);
 		if (mapped) {
 			err = mapped;
 			goto ret;
@@ -218,7 +218,7 @@ int kmsan_ioremap_page_range(unsigned long start, unsigned long end,
 		mapped = __vmap_pages_range_noflush(
 			vmalloc_origin(start + off),
 			vmalloc_origin(start + off + PAGE_SIZE), prot, &origin,
-			PAGE_SHIFT);
+			PAGE_SHIFT, gfp_mask);
 		if (mapped) {
 			err = mapped;
 			goto ret;
diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 212449db34a1..452760a333e4 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -247,7 +247,7 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 	origin_end = vmalloc_meta((void *)end, KMSAN_META_ORIGIN);
 	kmsan_enter_runtime();
 	mapped = __vmap_pages_range_noflush(shadow_start, shadow_end, prot,
-					    s_pages, page_shift);
+					    s_pages, page_shift, gfp_mask);
 	kmsan_leave_runtime();
 	if (mapped) {
 		err = mapped;
@@ -255,7 +255,7 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
 	}
 	kmsan_enter_runtime();
 	mapped = __vmap_pages_range_noflush(origin_start, origin_end, prot,
-					    o_pages, page_shift);
+					    o_pages, page_shift, gfp_mask);
 	kmsan_leave_runtime();
 	if (mapped) {
 		/* Undo the shadow mapping set up above. */
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 5c1eacc2a0ac..4f3c320a4849 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -725,7 +725,8 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr,
  * may have contiguous chunks, as hinted by @shift.
  */
 static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int shift)
+		pgprot_t prot, struct page **pages, unsigned int shift,
+		unsigned int *mapped_pages)
 {
 	unsigned long start = addr;
 	pgd_t *pgd;
@@ -748,6 +749,7 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
 	if (mask & ARCH_PAGE_TABLE_SYNC_MASK)
 		arch_sync_kernel_mappings(start, end);
 
+	*mapped_pages = nr;
 	return err;
 }
 
@@ -758,18 +760,36 @@ static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
  * The caller is responsible for calling flush_cache_vmap() after this
  * function returns successfully and before the addresses are accessed.
  *
+ * On failure the PTEs installed by this call are torn down before
+ * returning, so a retry sees a clean range; PTEs that were already
+ * there when the walk failed are left alone. When @gfp_mask cannot
+ * block, the teardown skips cond_resched(), it may run in atomic
+ * context.
+ *
  * This is an internal function only. Do not use outside mm/.
  */
 int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
-		pgprot_t prot, struct page **pages, unsigned int page_shift)
+		pgprot_t prot, struct page **pages, unsigned int page_shift,
+		gfp_t gfp_mask)
 {
+	int err;
+	unsigned int mapped_pages = 0;
+
 	if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
 		return -EINVAL;
 
 	if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC))
 		page_shift = PAGE_SHIFT;
 
-	return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift);
+	err = vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift,
+					    &mapped_pages);
+
+	/* Undo the PTEs installed before the failure. */
+	if (err && mapped_pages)
+		__vunmap_range_noflush(addr, addr + (mapped_pages * PAGE_SIZE),
+				       gfpflags_allow_blocking(gfp_mask));
+
+	return err;
 }
 
 int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
@@ -781,7 +801,8 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift);
+	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift,
+					  gfp_mask);
 }
 
 static int __vmap_pages_range(unsigned long addr, unsigned long end,
@@ -3715,7 +3736,7 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 		pgprot_t prot, struct page **pages)
 {
 	const unsigned int nr_pages = (end - addr) >> PAGE_SHIFT;
-	unsigned int prev_shift = 0, batch_idx = 0;
+	unsigned int prev_shift = 0, batch_idx = 0, nr_mapped;
 	unsigned long batch_start = addr, batch_end = addr;
 	int err;
 
@@ -3734,7 +3755,8 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 
 		if (shift != prev_shift) {
 			err = vmap_pages_range_noflush_walk(batch_start, batch_end,
-					prot, pages + batch_idx, prev_shift);
+					prot, pages + batch_idx, prev_shift,
+					&nr_mapped);
 			if (err)
 				goto out;
 			prev_shift = shift;
@@ -3756,7 +3778,7 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 	/* Remaining */
 	if (batch_start < end)
 		err = vmap_pages_range_noflush_walk(batch_start, end, prot,
-				pages + batch_idx, prev_shift);
+				pages + batch_idx, prev_shift, &nr_mapped);
 
 out:
 	if (!err)
diff --git a/mm/vmalloc.h b/mm/vmalloc.h
index 4ec114a75f04..9ea28845f713 100644
--- a/mm/vmalloc.h
+++ b/mm/vmalloc.h
@@ -40,7 +40,8 @@ void clear_vm_uninitialized_flag(struct vm_struct *vm);
 
 int __must_check __vmap_pages_range_noflush(unsigned long addr,
 		unsigned long end, pgprot_t prot,
-		struct page **pages, unsigned int page_shift);
+		struct page **pages, unsigned int page_shift,
+		gfp_t gfp_mask);
 
 void vunmap_range_noflush(unsigned long start, unsigned long end);
 
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 08/11] mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() on failure
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (6 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 07/11] mm/vmalloc: undo what __vmap_pages_range_noflush() " Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 09/11] mm/vmalloc: undo what vmap_pages_range_batched() mapped " Hao Ge
                   ` (2 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

vmap_pages_range_noflush() sets up the KMSAN metadata for the whole
range before it starts mapping. If the mapping fails, the page
tables undo themselves now, but the metadata stays behind, and the
next attempt to map the same range hits the leftover shadow PTEs.

Tear the metadata down on the error path, it was fully set up before
the mapping failed. The teardown follows the page tables and skips
cond_resched() when the caller's gfp mask cannot block.

Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/vmalloc.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 4f3c320a4849..28e65073624a 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -801,8 +801,18 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
 
 	if (ret)
 		return ret;
-	return __vmap_pages_range_noflush(addr, end, prot, pages, page_shift,
-					  gfp_mask);
+
+	ret = __vmap_pages_range_noflush(addr, end, prot, pages, page_shift,
+					 gfp_mask);
+	/*
+	 * The page tables undo themselves on failure. Tear down the
+	 * metadata that was fully set up before the mapping failed.
+	 */
+	if (ret)
+		kmsan_vunmap_range_noflush(addr, end,
+					   gfpflags_allow_blocking(gfp_mask));
+
+	return ret;
 }
 
 static int __vmap_pages_range(unsigned long addr, unsigned long end,
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 09/11] mm/vmalloc: undo what vmap_pages_range_batched() mapped on failure
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (7 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 08/11] mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() " Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 10/11] powerpc: drop redundant unmaps of failed vmap mappings Hao Ge
  2026-11-16  7:13 ` [PATCH v5 11/11] mm/percpu: stop unmapping the CPU that failed to map Hao Ge
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

vmap_pages_range_batched() maps the range batch by batch. When a
batch fails, the batches before it are installed, and the failing
one may have installed some of its pages. mapped_end follows the
end of the last installed page, and the error path undoes
[addr, mapped_end), never past the point where the walk stopped.

The KMSAN metadata is set up for the whole range up front, so it is
torn down as a whole when the mapping fails, like in
vmap_pages_range_noflush(). The only caller, vmap(), also calls
vunmap() on its error path; the double teardown only walks over PTEs
that are already cleared.

Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/vmalloc.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 28e65073624a..aa80bc273ec0 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -3748,13 +3748,14 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 	const unsigned int nr_pages = (end - addr) >> PAGE_SHIFT;
 	unsigned int prev_shift = 0, batch_idx = 0, nr_mapped;
 	unsigned long batch_start = addr, batch_end = addr;
+	unsigned long mapped_end = addr;
 	int err;
 
 	err = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,
 					     PAGE_SHIFT, GFP_KERNEL);
 
 	if (err)
-		goto out;
+		return err;
 
 	for (unsigned int i = 0; i < nr_pages; ) {
 		unsigned int shift = PAGE_SHIFT +
@@ -3764,9 +3765,10 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 			prev_shift = shift;
 
 		if (shift != prev_shift) {
-			err = vmap_pages_range_noflush_walk(batch_start, batch_end,
-					prot, pages + batch_idx, prev_shift,
-					&nr_mapped);
+			err = vmap_pages_range_noflush_walk(batch_start,
+					batch_end, prot, pages + batch_idx,
+					prev_shift, &nr_mapped);
+			mapped_end = batch_start + (nr_mapped * PAGE_SIZE);
 			if (err)
 				goto out;
 			prev_shift = shift;
@@ -3786,13 +3788,21 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
 	}
 
 	/* Remaining */
-	if (batch_start < end)
+	if (batch_start < end) {
 		err = vmap_pages_range_noflush_walk(batch_start, end, prot,
 				pages + batch_idx, prev_shift, &nr_mapped);
+		mapped_end = batch_start + (nr_mapped * PAGE_SIZE);
+	}
 
 out:
-	if (!err)
+	if (err) {
+		/* Undo the part the batches installed before they failed. */
+		if (mapped_end > addr)
+			__vunmap_range_noflush(addr, mapped_end, true);
+		kmsan_vunmap_range_noflush(addr, end, true);
+	} else {
 		flush_cache_vmap(addr, end);
+	}
 	return err;
 }
 
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 10/11] powerpc: drop redundant unmaps of failed vmap mappings
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (8 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 09/11] mm/vmalloc: undo what vmap_pages_range_batched() mapped " Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  2026-11-16  7:13 ` [PATCH v5 11/11] mm/percpu: stop unmapping the CPU that failed to map Hao Ge
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

vmap_page_range() and ioremap_page_range() clear the PTEs they
installed when they fail, so the vunmap_range() in remap_isa_base()
and ioremap_phb() has nothing left to remove.

Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 arch/powerpc/kernel/isa-bridge.c | 5 ++---
 arch/powerpc/kernel/pci_64.c     | 4 +---
 2 files changed, 3 insertions(+), 6 deletions(-)

diff --git a/arch/powerpc/kernel/isa-bridge.c b/arch/powerpc/kernel/isa-bridge.c
index 5c064485197a..93029d85e6ea 100644
--- a/arch/powerpc/kernel/isa-bridge.c
+++ b/arch/powerpc/kernel/isa-bridge.c
@@ -46,9 +46,8 @@ static void remap_isa_base(phys_addr_t pa, unsigned long size)
 	WARN_ON_ONCE(size & ~PAGE_MASK);
 
 	if (slab_is_available()) {
-		if (vmap_page_range(ISA_IO_BASE, ISA_IO_BASE + size, pa,
-				    pgprot_noncached(PAGE_KERNEL)))
-			vunmap_range(ISA_IO_BASE, ISA_IO_BASE + size);
+		vmap_page_range(ISA_IO_BASE, ISA_IO_BASE + size, pa,
+				pgprot_noncached(PAGE_KERNEL));
 	} else {
 		early_ioremap_range(ISA_IO_BASE, pa, size,
 				pgprot_noncached(PAGE_KERNEL));
diff --git a/arch/powerpc/kernel/pci_64.c b/arch/powerpc/kernel/pci_64.c
index e27342ef128b..f74269636d5e 100644
--- a/arch/powerpc/kernel/pci_64.c
+++ b/arch/powerpc/kernel/pci_64.c
@@ -139,10 +139,8 @@ void __iomem *ioremap_phb(phys_addr_t paddr, unsigned long size)
 
 	addr = (unsigned long)area->addr;
 	if (ioremap_page_range(addr, addr + size, paddr,
-			pgprot_noncached(PAGE_KERNEL))) {
-		vunmap_range(addr, addr + size);
+			pgprot_noncached(PAGE_KERNEL)))
 		return NULL;
-	}
 
 	return (void __iomem *)addr;
 }
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH v5 11/11] mm/percpu: stop unmapping the CPU that failed to map
  2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
                   ` (9 preceding siblings ...)
  2026-11-16  7:13 ` [PATCH v5 10/11] powerpc: drop redundant unmaps of failed vmap mappings Hao Ge
@ 2026-11-16  7:13 ` Hao Ge
  10 siblings, 0 replies; 12+ messages in thread
From: Hao Ge @ 2026-11-16  7:13 UTC (permalink / raw)
  To: Suren Baghdasaryan, Uladzislau Rezki, Dev Jain, Andrew Morton,
	Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
	Christophe Leroy (CS GROUP), Ritesh Harjani (IBM),
	Shrikanth Hegde, Alexander Potapenko, Marco Elver, Dmitry Vyukov,
	Dennis Zhou, Tejun Heo, Christoph Lameter
  Cc: linuxppc-dev, linux-kernel, kasan-dev, linux-mm, Hao Ge

__pcpu_map_pages() clears the PTEs it installed when it fails, so
pcpu_map_pages() only has to tear down the CPUs it mapped before
the failing one.

Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Hao Ge <hao.ge@linux.dev>
---
 mm/percpu-vm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c
index 509d8901835c..59d3a336a6bd 100644
--- a/mm/percpu-vm.c
+++ b/mm/percpu-vm.c
@@ -252,10 +252,11 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, struct page **pages,
 	return 0;
 err:
 	for_each_possible_cpu(tcpu) {
-		__pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
-				   page_end - page_start);
+		/* The failing CPU's partial mapping undoes itself */
 		if (tcpu == cpu)
 			break;
+		__pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start),
+				   page_end - page_start);
 	}
 	pcpu_post_unmap_tlb_flush(chunk, page_start, page_end);
 	return err;
-- 
2.25.1


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2026-11-16  7:14 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16  7:13 [PATCH v5 00/11] mm/vmalloc: fix up vmap error handling Hao Ge
2026-11-16  7:13 ` [PATCH v5 01/11] mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic Hao Ge
2026-11-16  7:13 ` [PATCH v5 02/11] mm/kmsan: undo the shadow mapping when the origin mapping fails Hao Ge
2026-11-16  7:13 ` [PATCH v5 03/11] mm/vmalloc: reject an unaligned range in vmap_page_range() Hao Ge
2026-11-16  7:13 ` [PATCH v5 04/11] mm/vmalloc: reject an unaligned range in __vunmap_range_noflush() Hao Ge
2026-11-16  7:13 ` [PATCH v5 05/11] mm/vmalloc: only flush the cache after the vmap succeeded Hao Ge
2026-11-16  7:13 ` [PATCH v5 06/11] mm/vmalloc: undo what vmap_page_range() mapped on failure Hao Ge
2026-11-16  7:13 ` [PATCH v5 07/11] mm/vmalloc: undo what __vmap_pages_range_noflush() " Hao Ge
2026-11-16  7:13 ` [PATCH v5 08/11] mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() " Hao Ge
2026-11-16  7:13 ` [PATCH v5 09/11] mm/vmalloc: undo what vmap_pages_range_batched() mapped " Hao Ge
2026-11-16  7:13 ` [PATCH v5 10/11] powerpc: drop redundant unmaps of failed vmap mappings Hao Ge
2026-11-16  7:13 ` [PATCH v5 11/11] mm/percpu: stop unmapping the CPU that failed to map Hao Ge

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®