mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes
@ 2026-11-18  9:20 David Hildenbrand (Arm)
  2026-11-18  9:20 ` [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio David Hildenbrand (Arm)
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-18  9:20 UTC (permalink / raw)
  To: Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
	Borislav Petkov (AMD),
	Cenk Yılmaz, Zhang Peng, Mike Rapoport, Suren Baghdasaryan,
	Michal Hocko, Andrew Morton, Zi Yan, Matthew Brost, Joshua Hahn,
	Byungchul Park, Gregory Price, Ying Huang, Alistair Popple,
	Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
	Barry Song, Youngjun Park, Mel Gorman, Sasha Levin, Rik van Riel,
	Ingo Molnar, Peter Xu, Hugh Dickins, Axel Rasmussen,
	Johannes Weiner, Michal Hocko, Qi Zheng, Shakeel Butt,
	Yuanchu Xie, Wei Xu, Baolin Wang, Andrea Arcangeli,
	Thomas Gleixner
  Cc: linux-mm, linux-kernel, stable, David Hildenbrand (Arm)

David, what keeps you up at night? Well, the last 3 nights, TLB flushes.

We have two mechanisms that can delay/batch TLB flushes and cause
problems:

 (A) mmu_gather zap: we can delay TLB flushes until after dropping the
     PT lock, after having zapped PTEs and adjusted the mapcounts.

 (B) TTU_BATCH_FLUSH when used by reclaim: we can defer (batch) TLB
     flushes, after having unmapped folios and adjusted the mapcounts,
     and after having discarded lazyfree folios, removed folios from the
     page-cache/swap-cache or just dropped the folio lock.

Code that installs new folios (and doesn't just refault the old ones)
must be aware of that, otherwise we can end up with stale TLB entries
while new folios are already mapped into page tables.

So in general, whenever we install a new folio into the page tables,
we must make sure that no such stale TLB entries exist.

Even worse, whenever we replace a folio in the page-cache that might differ
content-wise from the old folio, or we end up modifying a new folio through
write(), we must make sure that no such stale TLB entries exist.

There are some exceptions (e.g., only once MADV_DONTNEED finished is when
a new read from the same address gives any guarantees), so the primary
focus is on folios that got unmapped transparently by the system, without
involving MADV_DONTNEED.

This series handles the following scenarios:

Patch #1: Zapping: flush the TLB before installing a CoW'ed folio.
          Handles CoW of a folio that is still in the page-cache.

Patch #2: Reclaim: flush deferred writable TLB entries before unlocking
          the folio. Handles writes through stale TLB entries getting
          lost when reclaim keeps a dirty folio and somebody else
          starts writeback.

Patch #3: Reclaim: don't defer TLB flushes when unmapping lazyfree
          folios. Handles mapping a new anon folio after reclaim
          discarded a lazyfree folio (do_anonymous_page(),
          migrate_vma_insert_page(), UFFDIO_COPY, UFFDIO_MOVE).

Patch #4: Reclaim: unlock unmapped folios only after the deferred TLB
          flush. Handles refaulting a new page-cache folio, CoW of a
          folio that is still in the page-cache, and swapping in a new
          folio (do_swap_page(), unuse_pte(), UFFDIO_CONTINUE). Also
          handles invalidation replacing a folio that reclaim kept, and
          write() modifying a new page-cache folio.

Patches #1 and #2 are small fixes that should be merged as soon as
possible. Patch #4 makes the flush added by patch #2 unnecessary by
keeping folios locked until the deferred TLB flush, and removes it
again.

Patches #3 and #4 are RFC, because #4 is rather intrusive and #3 comes
with a performance impact (see below). But there isn't really a good
alternative to fix these things for good that I see.

In theory, UFFDIO_COPY installing a new anon folio into a MAP_PRIVATE
shmem mapping is affected by concurrent zapping just like CoW faults in
patch #1. But the required setups (e.g., userfaultfd on a MAP_PRIVATE
mapping of a tmpfs file that backs a loop device) are so unusual that we
will leave it alone for now. Same applies to the hugetlb counterpart.

Remaining Problem
-----------------

There is one remaining known problem that is not handled by this
series:

Zapping (e.g., unmap_mapping_range()) removes the rmap of clean
page-cache folios before performing the deferred TLB flush. If the
mapcount drops to 0, a concurrent invalidation (e.g.,
invalidate_inode_pages2_range()) will skip the unmapped folio and can
remove it from the page-cache before the pending TLB flush.

A page fault (except a CoW fault, see patch #1) can then map a new folio,
or write() can modify it, while other CPUs still read the old folio through
stale TLB entries.

I think that this is low priority: it requires two concurrent invalidations
of the same file range while another process has the file mapped, and CPUs
can only read stale data until the zapper flushes the TLB; writes are never
lost.

It could maybe be fixed in the invalidation code? unmap_mapping_*() holds
the i_mmap_rwsem in read mode until it flushed the TLB, so
folio_unmap_invalidate() could briefly take it in write mode iff the folio
has unexpected references. But dropbehind (RWF_DONTCACHE) invalidates
folios on writeback completion while holding an additional reference, where
we must not sleep on the i_mmap_rwsem, so that would need some care.

Something for the future :)

Testing
-------

I threw a bunch of selftests at it and had AI generate some stress
tests, in particular for the benchmarks below. I also ran stress-ng to
force reclaim.

I managed to reproduce the issues in patch #3 and #4, as described in
the patch descriptions.

Stale reads out of 1024000 checked pages in 2000 iterations:

                               base    series
  lazyfree,  MGLRU            16242         0
  lazyfree,  classic LRU      16274         0
  swapcache, MGLRU            24076         0
  swapcache, classic LRU      23028         0

Performance Impact
------------------

I ran some benchmarks in an x86-64 VM with limited CPUs, RAM and zram
enabled. Running inside a VM might make TLB flushes more expensive, so
the results below might be a bit exaggerated. AI results below.

Classic LRU, 4 threads randomly touching a working set twice the size of
their memcg for 30 seconds. Pages touched per second:

                 base     series
  anon-rw        681k      1055k
  anon-ro       1013k      1027k
  file-ro        417k       419k
  file-rw        173k       183k
  lazyfree      1512k       333k

Swapping out dirty anon folios ("anon-rw") is about 55% faster, because
patch #4 now also batches TLB flushes of stale writable TLB entries.
That is what Zhang Peng's series [1] tried to achieve, and his work also
inspired how patch #4 locks folios again, but the used mechanism is
different.

Reclaiming clean anon folios ("anon-ro"), clean page-cache folios
("file-ro") and dirty page-cache folios ("file-rw") is not noticeably
affected.

Reclaiming lazyfree folios ("lazyfree") is about 4.5x slower, because
we stop batching TLB flushes when discarding them entirely, which seems
to be the right thing to do for now. We can look into optimizing this
separately later.

[1] https://lore.kernel.org/linux-mm/20261031-batch-tlb-flush-v6-0-190539839eec@tencent.com/

---
David Hildenbrand (Arm) (4):
      mm: flush stale TLB entries before installing a CoW'ed folio
      mm/vmscan: flush deferred writable TLB entries before unlocking the folio
      RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios
      RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush

 mm/internal.h |   5 ++
 mm/memory.c   |   6 +++
 mm/rmap.c     |  20 +++++++-
 mm/shmem.c    |   2 +-
 mm/swap.h     |   5 +-
 mm/swapfile.c |   7 ++-
 mm/vmscan.c   | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 7 files changed, 184 insertions(+), 16 deletions(-)
---
base-commit: 0c524518413094607a6178c53a65815762bc9c45
change-id: 20261114-file_folio_flush-7d1749fab2d8

Best regards,
--  
David Hildenbrand (Arm) <david@kernel.org>


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio
  2026-11-18  9:20 [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes David Hildenbrand (Arm)
@ 2026-11-18  9:20 ` David Hildenbrand (Arm)
  2026-11-18 13:55   ` Cenk Yılmaz
  2026-11-18  9:20 ` [PATCH 2/4] mm/vmscan: flush deferred writable TLB entries before unlocking the folio David Hildenbrand (Arm)
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 6+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-18  9:20 UTC (permalink / raw)
  To: Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
	Borislav Petkov (AMD),
	Cenk Yılmaz, Zhang Peng, Mike Rapoport, Suren Baghdasaryan,
	Michal Hocko, Andrew Morton, Zi Yan, Matthew Brost, Joshua Hahn,
	Byungchul Park, Gregory Price, Ying Huang, Alistair Popple,
	Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
	Barry Song, Youngjun Park, Mel Gorman, Sasha Levin, Rik van Riel,
	Ingo Molnar, Peter Xu, Hugh Dickins, Axel Rasmussen,
	Johannes Weiner, Michal Hocko, Qi Zheng, Shakeel Butt,
	Yuanchu Xie, Wei Xu, Baolin Wang, Andrea Arcangeli,
	Thomas Gleixner
  Cc: linux-mm, linux-kernel, stable, David Hildenbrand (Arm)

Zapping PTEs in zap_pte_range() defers the TLB flush until after dropping
the PTL for clean page-cache folios.

A concurrent write fault in a MAP_PRIVATE mapping can then CoW the
page-cache folio and install a new anon folio without a prior TLB
flush:

	CPU0				CPU1
	LOAD(ADDR)
	 -> #PF
	    installs R/O PTE -> A
					fuse_open()
					 -> invalidate_inode_pages2()
					    -> zap_pte_range()
					       clears R/O PTE
					       // delayed flush
	// Other CPUs (incl CPU0) still cache ADDR -> A
	STORE(ADDR)
	 -> #PF
	    do_cow_fault()
	    copies A to B, installs R/W PTE -> B
	    // no TLB flush: PTE was none

CPUs with stale TLB entries will still read from the old folio while writes
go to the new folio, until the TLB is flushed.

We really must process pending TLB flushes if we are not faulting the
same folio back in.

Fix it by flushing the TLB before installing an anon (CoW'ed) folio in
set_pte_range() if a TLB flush is pending. Zapping itself stays as fast
as before.

Other faults are not affected: unmap_mapping_*() users only remove
folios from the page-cache after their own TLB flush, so until then, a
refault maps the same folio again. The exception is a concurrent
invalidation that removes the already-unmapped folio from the
page-cache before the zapper flushed the TLB; that is an unlikely corner
case that we'll not handle for now.

Note that we don't care about MADV_DONTNEED followed by a new read
fault, as MADV_DONTNEED only guarantees its semantics once the operation
completed. CoW faults will flush the TLB either way.

For example, imagine that MADV_DONTNEED zaps a page-cache folio and
doesn't flush the TLB (but still holds a reference). If the page-cache
replaces the folio, a new read fault might map the new folio without
flushing the TLB. That's okay, as MADV_DONTNEED did not complete yet.
Same applies to MADV_DONTNEED of CoW'ed folios when re-faulting the
page-cache folio.

Memory reclaim, which removes folios from the page-cache before performing
its own batched TLB flush, will be handled separately.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by: Cenk Yılmaz <cenk@deepfinresearch.com>
Debugged-by: Cenk Yılmaz <cenk@deepfinresearch.com>
Cc: stable@vger.kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
 mm/memory.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/memory.c b/mm/memory.c
index d2f48d78b0d7b..e1d9759e6fde1 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5888,6 +5888,12 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio,
 	/* copy-on-write page */
 	if (write && !(vma->vm_flags & VM_SHARED)) {
 		VM_BUG_ON_FOLIO(nr != 1, folio);
+		/*
+		 * Concurrent zapping might have dropped the PTL but not flushed
+		 * the TLB yet. Flush before installing a new folio.
+		 */
+		if (mm_tlb_flush_pending(vma->vm_mm))
+			flush_tlb_page(vma, addr);
 		folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
 		folio_add_lru_vma(folio, vma);
 	} else {

-- 
2.56.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 2/4] mm/vmscan: flush deferred writable TLB entries before unlocking the folio
  2026-11-18  9:20 [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes David Hildenbrand (Arm)
  2026-11-18  9:20 ` [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio David Hildenbrand (Arm)
@ 2026-11-18  9:20 ` David Hildenbrand (Arm)
  2026-11-18  9:20 ` [PATCH 3/4] RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios David Hildenbrand (Arm)
  2026-11-18  9:20 ` [PATCH 4/4] RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush David Hildenbrand (Arm)
  3 siblings, 0 replies; 6+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-18  9:20 UTC (permalink / raw)
  To: Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
	Borislav Petkov (AMD),
	Cenk Yılmaz, Zhang Peng, Mike Rapoport, Suren Baghdasaryan,
	Michal Hocko, Andrew Morton, Zi Yan, Matthew Brost, Joshua Hahn,
	Byungchul Park, Gregory Price, Ying Huang, Alistair Popple,
	Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
	Barry Song, Youngjun Park, Mel Gorman, Sasha Levin, Rik van Riel,
	Ingo Molnar, Peter Xu, Hugh Dickins, Axel Rasmussen,
	Johannes Weiner, Michal Hocko, Qi Zheng, Shakeel Butt,
	Yuanchu Xie, Wei Xu, Baolin Wang, Andrea Arcangeli,
	Thomas Gleixner
  Cc: linux-mm, linux-kernel, stable, David Hildenbrand (Arm)

try_to_unmap() with TTU_BATCH_FLUSH transfers the PTE dirty bit to the
folio and defers the TLB flush.

Reclaim must flush the TLB before unlocking the folio, otherwise stale
writable TLB entries might be left behind temporarily: writeback started
by other mechanisms (e.g., fsync) will not find any mappings in
folio_mkclean(), so it will neither write-protect nor flush anything.

A write through a stale TLB entry after the folio was cleaned gets lost
once the folio is reclaimed.

However, reclaim currently does not flush the TLB if the now-dirty folio
does not get written back (e.g., gets activated or kept instead). Fix it
by flushing right after try_to_unmap() if the folio is dirty, which makes
the existing flush before pageout() unnecessary.

For dirty page-cache folios that reclaim activates, we will now flush the
TLB immediately after unmapping them instead of batching the flush, which
is the right thing to do. Performance-wise, it shouldn't really matter
much, because folios with dirty PTEs are usually young and reclaim won't
even try to unmap them.

Fixes: d950c9477d51 ("mm: defer flush of writable TLB entries")
Cc: stable@vger.kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
 mm/vmscan.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 5495d9be90e88..af7831643ecf5 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1455,6 +1455,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 				flags |= TTU_SYNC;
 
 			try_to_unmap(folio, flags);
+			/*
+			 * Flush deferred writable TLB entries before we could
+			 * drop the folio lock and others could start writeback.
+			 */
+			if (folio_test_dirty(folio))
+				try_to_unmap_flush_dirty();
 			if (folio_mapped(folio)) {
 				stat->nr_unmap_fail += nr_pages;
 				if (!was_swapbacked &&
@@ -1496,12 +1502,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 			if (!sc->may_writepage)
 				goto keep_locked;
 
-			/*
-			 * Folio is dirty. Flush the TLB if a writable entry
-			 * potentially exists to avoid CPU writes after I/O
-			 * starts and then write it out here.
-			 */
-			try_to_unmap_flush_dirty();
 			switch (pageout(&ctx, mapping, folio, folio_list)) {
 			case PAGE_KEEP:
 				goto keep_locked;

-- 
2.56.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 3/4] RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios
  2026-11-18  9:20 [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes David Hildenbrand (Arm)
  2026-11-18  9:20 ` [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio David Hildenbrand (Arm)
  2026-11-18  9:20 ` [PATCH 2/4] mm/vmscan: flush deferred writable TLB entries before unlocking the folio David Hildenbrand (Arm)
@ 2026-11-18  9:20 ` David Hildenbrand (Arm)
  2026-11-18  9:20 ` [PATCH 4/4] RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush David Hildenbrand (Arm)
  3 siblings, 0 replies; 6+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-18  9:20 UTC (permalink / raw)
  To: Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
	Borislav Petkov (AMD),
	Cenk Yılmaz, Zhang Peng, Mike Rapoport, Suren Baghdasaryan,
	Michal Hocko, Andrew Morton, Zi Yan, Matthew Brost, Joshua Hahn,
	Byungchul Park, Gregory Price, Ying Huang, Alistair Popple,
	Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
	Barry Song, Youngjun Park, Mel Gorman, Sasha Levin, Rik van Riel,
	Ingo Molnar, Peter Xu, Hugh Dickins, Axel Rasmussen,
	Johannes Weiner, Michal Hocko, Qi Zheng, Shakeel Butt,
	Yuanchu Xie, Wei Xu, Baolin Wang, Andrea Arcangeli,
	Thomas Gleixner
  Cc: linux-mm, linux-kernel, stable, David Hildenbrand (Arm)

When reclaim discards a lazyfree folio, it clears the PTE but delays the
TLB flush. Until flushed, mapping a new folio into the now-none PTE
can leave some CPUs still reading from the old folio while others
already write to the new folio.

This is hard to trigger in practice and would mostly look like a weird
use-after-free bug in an allocator that uses MADV_FREE, but it can be
reproduced reliably with three threads pinned to different CPUs:

(1) Thread A populates the pages, MADV_FREE's them and then
    MADV_PAGEOUT's them.
(2) Thread B reads all pages to keep TLB entries cached and verifies the
    content of pages published by thread C.
(3) Thread C waits until mincore() indicates that a page is no longer
    present, writes a new value to it, and publishes it to thread B.

Thread B can then read the old value after the new folio was mapped and
the content updated.

One way to handle it would be adding conditional flushes to
do_anonymous_page(), migrate_vma_insert_page(), UFFDIO_COPY and
UFFDIO_MOVE ... essentially anywhere we install new anonymous folios,
which would mean that these places flush quite frequently on concurrent
reclaim.

To perform targeted flushes, we'd have to keep some PTE markers behind,
which is also rather nasty and not an easy fix.

So let's not defer the TLB flush for lazyfree folios, which includes
folios in VM_DROPPABLE mappings.

To avoid flushing if the folio was redirtied or has additional
references, perform all non-deferred TLB flushes only once we know that
we won't restore the PTEs, right before removing the rmap.

The downside is that we no longer batch TLB flushes when discarding
lazyfree folios, which harms performance, but is the right thing to do
for now.

Fixes: 854e9ed09ded ("mm: support madvise(MADV_FREE)")
Cc: stable@vger.kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
 mm/rmap.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/mm/rmap.c b/mm/rmap.c
index 09c33b9ec5850..3935eed23a361 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2228,6 +2228,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 	struct mmu_notifier_range range;
 	enum ttu_flags flags = (enum ttu_flags)(long)arg;
 	unsigned long nr_pages = 1, end_addr;
+	bool flush;
 	unsigned long pfn;
 	int ptes = 0;
 
@@ -2253,6 +2254,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 
 	while (page_vma_mapped_walk(&pvmw)) {
 		nr_pages = 1;
+		flush = false;
 
 		/*
 		 * If the folio is in an mlock()d vma, we must not swap it out.
@@ -2339,11 +2341,18 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			 * architecture must guarantee that a clear->dirty
 			 * transition on a cached TLB entry is written through
 			 * and traps if the PTE is unmapped.
+			 *
+			 * If we don't defer the flush, we flush only once we
+			 * know that we won't restore the PTEs below. We never
+			 * defer the flush for lazyfree folios: if we discard
+			 * them, the PTE will be none, and a page fault could
+			 * map a new folio while stale TLB entries still exist.
 			 */
-			if (should_defer_flush(mm, flags))
+			if (!folio_test_lazyfree(folio) &&
+			    should_defer_flush(mm, flags))
 				set_tlb_ubc_flush_pending(mm, pteval, address, end_addr);
 			else
-				flush_tlb_range(vma, address, end_addr);
+				flush = true;
 			if (pte_dirty(pteval))
 				folio_mark_dirty(folio);
 		} else {
@@ -2402,6 +2411,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 			add_mm_counter(mm, mm_counter_file(folio), -nr_pages);
 		}
 finish_unmap:
+		if (flush)
+			flush_tlb_range(vma, address, end_addr);
 		folio_remove_rmap_ptes(folio, page, nr_pages, vma);
 		if (vma_test_any_mask(vma, VMA_LOCKED_MASK))
 			mlock_drain_local();

-- 
2.56.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH 4/4] RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush
  2026-11-18  9:20 [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes David Hildenbrand (Arm)
                   ` (2 preceding siblings ...)
  2026-11-18  9:20 ` [PATCH 3/4] RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios David Hildenbrand (Arm)
@ 2026-11-18  9:20 ` David Hildenbrand (Arm)
  3 siblings, 0 replies; 6+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-18  9:20 UTC (permalink / raw)
  To: Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
	Borislav Petkov (AMD),
	Cenk Yılmaz, Zhang Peng, Mike Rapoport, Suren Baghdasaryan,
	Michal Hocko, Andrew Morton, Zi Yan, Matthew Brost, Joshua Hahn,
	Byungchul Park, Gregory Price, Ying Huang, Alistair Popple,
	Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
	Barry Song, Youngjun Park, Mel Gorman, Sasha Levin, Rik van Riel,
	Ingo Molnar, Peter Xu, Hugh Dickins, Axel Rasmussen,
	Johannes Weiner, Michal Hocko, Qi Zheng, Shakeel Butt,
	Yuanchu Xie, Wei Xu, Baolin Wang, Andrea Arcangeli,
	Thomas Gleixner
  Cc: linux-mm, linux-kernel, stable, David Hildenbrand (Arm)

try_to_unmap() with TTU_BATCH_FLUSH zaps the PTEs, transfers the PTE
dirty bit to the folio and defers the TLB flush. Until the TLB is
flushed, other CPUs might still access the folio through stale TLB
entries.

However, reclaim unlocks folios it keeps and removes folios it frees
from the page-cache / swap-cache before performing the deferred TLB
flush.

Stale writable TLB entries are currently handled by flushing them right
after try_to_unmap() if the folio is dirty, so writes through stale TLB
entries cannot get lost, but they are also not batched.

Stale read-only TLB entries, on the other hand, are problematic:

 (1) A page fault can map a new page-cache folio, or swap in a new folio
     once reclaim removed the old one from the swap-cache (e.g., a clean
     swap-cache folio, or a dirty folio mapped by clean PTEs after
     synchronous swap-out to zram or zswap). Modifications of the new
     folio (e.g., through write() or page table mappings) are then not
     visible to CPUs that still read the old folio.

 (2) A CoW fault in a MAP_PRIVATE mapping can copy a page-cache folio
     that reclaim kept. Writes go to the new anon folio, while other CPUs
     still read the old folio.

 (3) Invalidation (e.g., invalidate_inode_pages2()) can remove a folio
     that reclaim kept from the page-cache without flushing the TLB, as
     the folio is no longer mapped. A page fault can then map a new
     folio while other CPUs still read the old folio.

This can be reproduced easily by swapping out an anon memory range with
MADV_PAGEOUT, reading it back in, and swapping it out again
(MADV_PAGEOUT) while one thread keeps reading the range and another
thread writes to pages as soon as reclaim removed the folios from the
swap-cache. The first thread will occasionally read the old content.

Fix it by reclaiming folios in batches of up to 64 folios:

 (a) Lock each folio, check whether we want to reclaim it, and unmap it.
     Keep the folio locked if any deferred TLB flush is pending after
     unmapping it, otherwise unlock it right away.

 (b) Flush the TLB once, and unlock all folios that are still locked.

 (c) Lock the folios again, one at a time, to try reclaiming them.
     Activate folios that are still or again mapped, and skip folios
     that are under writeback or poisoned by now, or that somebody else
     holds locked.

As others might map or unmap folios while they are unlocked, we might
now activate or reclaim some folios that we would previously have kept
or activated, which is fine.

Keeping folios locked until the TLB flush makes sure that nobody can
remove them from the page-cache / swap-cache, copy them or start
writeback. Page faults will either wait on the folio lock, or find the
old folio and map it again, which is fine.

Unlocking folios before trying to reclaim them makes sure that we never
hold more than one folio lock while performing I/O or waiting on the
filesystem.

The idea of unlocking folios and locking them again using a trylock
before trying to reclaim them was inspired by [1] from Zhang Peng.

As nobody can start writeback before the TLB flush, we no longer have to
flush stale writable TLB entries right after try_to_unmap(), so remove
the try_to_unmap_flush_dirty() call and batch these flushes as well.
This effectively enables batching TLB flushes of stale writable TLB
entries, which is what [1] tried to achieve.

We must not wait for a folio lock, a folio flag or I/O while keeping
other folios locked, as that could deadlock. So flush the TLB and unlock
the batch early before waiting for writeback (legacy memcg), and before
allocating swap space would perform synchronous discard I/O
(folio_alloc_swap() now returns -EBUSY if allocation failed but retrying
with synchronous discard might help).

Allocating swap space can otherwise only sleep when allocating memory
for the swap table, which cannot enter direct reclaim or compaction,
as reclaim runs with PF_MEMALLOC.

We'll limit batching to 64 folios for now; the limit mostly matters for,
e.g., MADV_PAGEOUT and DAMON that can pass more, or when reclaim splits
large folios.

Note that khugepaged and batched (async) folio migration already keep up
to HPAGE_PMD_NR (e.g., 512 with 4 KiB pages) small folios locked after
unmapping them with a deferred TLB flush; migration even allocates new
folios, possibly entering direct reclaim, before flushing the TLB and
unlocking them. Both also call MMU notifiers while keeping other folios
locked, just like we do now.

Keep code churn to a minimum. Some follow-up cleanups and optimizations
make sense, in particular:

 (1) Cleanly separate the stages of shrink_folio_list() (see the TODO).

 (2) Avoid unlocking and locking folios again that do not require a TLB
     flush. But only if it's worth it and doesn't make the code too
     ugly.

 (3) Remove the remaining try_to_unmap_flush() calls before freeing
     folios; we only free folios that we flushed already or that we
     never unmapped.

 (4) Remove the now-unused try_to_unmap_flush_dirty() infrastructure.

[1] https://lore.kernel.org/linux-mm/20261031-batch-tlb-flush-v6-0-190539839eec@tencent.com/

Fixes: 72b252aed506 ("mm: send one IPI per CPU to TLB flush all entries after unmapping pages")
Cc: stable@vger.kernel.org
Signed-off-by: David Hildenbrand (Arm) <david@kernel.org>
---
 mm/internal.h |   5 ++
 mm/rmap.c     |   5 ++
 mm/shmem.c    |   2 +-
 mm/swap.h     |   5 +-
 mm/swapfile.c |   7 ++-
 mm/vmscan.c   | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 6 files changed, 165 insertions(+), 14 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 8a204f7352364..0d89968648b37 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1253,6 +1253,7 @@ extern struct workqueue_struct *mm_percpu_wq;
 #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
 void try_to_unmap_flush(void);
 void try_to_unmap_flush_dirty(void);
+bool try_to_unmap_flush_pending(void);
 void flush_tlb_batched_pending(struct mm_struct *mm);
 #else
 static inline void try_to_unmap_flush(void)
@@ -1261,6 +1262,10 @@ static inline void try_to_unmap_flush(void)
 static inline void try_to_unmap_flush_dirty(void)
 {
 }
+static inline bool try_to_unmap_flush_pending(void)
+{
+	return false;
+}
 static inline void flush_tlb_batched_pending(struct mm_struct *mm)
 {
 }
diff --git a/mm/rmap.c b/mm/rmap.c
index 3935eed23a361..399cba0395742 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -731,6 +731,11 @@ void try_to_unmap_flush_dirty(void)
 		try_to_unmap_flush();
 }
 
+bool try_to_unmap_flush_pending(void)
+{
+	return current->tlb_ubc.flush_required;
+}
+
 /*
  * Bits 0-14 of mm->tlb_flush_batched record pending generations.
  * Bits 16-30 of mm->tlb_flush_batched bit record flushed generations.
diff --git a/mm/shmem.c b/mm/shmem.c
index 5cb628cc9c6bb..3757bc1208068 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1936,7 +1936,7 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
 		folio_mark_uptodate(folio);
 	}
 
-	ret = folio_alloc_swap(folio);
+	ret = folio_alloc_swap(folio, /* may_sync_discard= */ true);
 	if (!ret) {
 		bool first_swapped = shmem_recalc_inode(inode, 0, nr_pages);
 		int error;
diff --git a/mm/swap.h b/mm/swap.h
index d5bf21f517dce..b77c8a071a0a8 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -243,7 +243,7 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp);
  *
  * folio_put_swap(): does the opposite thing of folio_dup_swap().
  */
-int folio_alloc_swap(struct folio *folio);
+int folio_alloc_swap(struct folio *folio, bool may_sync_discard);
 int folio_dup_swap(struct folio *folio, struct page *page);
 void folio_put_swap(struct folio *folio, struct page *page);
 
@@ -363,7 +363,8 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
 	return NULL;
 }
 
-static inline int folio_alloc_swap(struct folio *folio)
+static inline int folio_alloc_swap(struct folio *folio,
+		bool may_sync_discard)
 {
 	return -EINVAL;
 }
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 2d2df53fc51a9..242a361513301 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1732,16 +1732,19 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
 /**
  * folio_alloc_swap - allocate swap space for a folio
  * @folio: folio we want to move to swap
+ * @may_sync_discard: whether we may wait for synchronous discard I/O
  *
  * Allocate swap space for the folio and add the folio to the
  * swap cache.
  *
  * Context: Caller needs to hold the folio lock.
  * Return: %0 on success, %-E2BIG if splitting the folio might allow swapout,
+ * %-EBUSY if allocation failed but retrying with @may_sync_discard might
+ * succeed,
  * %-ENOSPC if no global swap space is available, or %-ENOMEM if splitting
  * would not help.
  */
-int folio_alloc_swap(struct folio *folio)
+int folio_alloc_swap(struct folio *folio, bool may_sync_discard)
 {
 	unsigned int order = folio_order(folio);
 	unsigned int size = 1 << order;
@@ -1774,6 +1777,8 @@ int folio_alloc_swap(struct folio *folio)
 	local_unlock(&percpu_swap_cluster.lock);
 
 	if (!order && unlikely(!folio_test_swapcache(folio))) {
+		if (!may_sync_discard)
+			return -EBUSY;
 		if (swap_sync_discard())
 			goto again;
 	}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index af7831643ecf5..11dbe19b170da 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1155,6 +1155,91 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask)
 	return false;
 }
 
+/*
+ * Maximum number of staged folios that shrink_folio_list() collects before
+ * it flushes the TLB.
+ */
+#define MAX_STAGED_FOLIOS	64
+
+struct staged_folios {
+	/* Locked folios, waiting for the TLB flush. */
+	struct list_head unflushed;
+	/* Unlocked folios, after the TLB flush (if required). */
+	struct list_head flushed;
+	/* Number of folios in the current batch. */
+	unsigned int nr_folios;
+};
+
+static bool staged_folios_empty(struct staged_folios *staged)
+{
+	return list_empty(&staged->unflushed) &&
+	       list_empty(&staged->flushed);
+}
+
+static void staged_folios_add(struct staged_folios *staged,
+		struct folio *folio, bool unmapped)
+{
+	if (unmapped && try_to_unmap_flush_pending()) {
+		/* Keep the folio locked until we flushed the TLB. */
+		list_add_tail(&folio->lru, &staged->unflushed);
+	} else {
+		folio_unlock(folio);
+		list_add_tail(&folio->lru, &staged->flushed);
+	}
+	staged->nr_folios++;
+}
+
+static void staged_folios_flush(struct staged_folios *staged)
+{
+	struct folio *folio;
+
+	try_to_unmap_flush();
+	list_for_each_entry(folio, &staged->unflushed, lru)
+		folio_unlock(folio);
+	list_splice_tail_init(&staged->unflushed, &staged->flushed);
+}
+
+static struct folio *staged_folios_next(struct staged_folios *staged,
+		struct list_head *folio_list)
+{
+	struct folio *folio;
+
+	if (staged_folios_empty(staged))
+		return NULL;
+	/* Collect more folios until we have enough or there are no more. */
+	if (staged->nr_folios < MAX_STAGED_FOLIOS && !list_empty(folio_list))
+		return NULL;
+
+	/* Flush the TLB and unlock the folios that waited for the flush. */
+	staged_folios_flush(staged);
+
+	folio = list_first_entry(&staged->flushed, struct folio, lru);
+	list_del(&folio->lru);
+	/* We handed out all folios: start collecting again. */
+	if (list_empty(&staged->flushed))
+		staged->nr_folios = 0;
+	return folio;
+}
+
+static int staged_folios_alloc_swap(struct staged_folios *staged,
+		struct folio *folio)
+{
+	bool may_sync_discard = list_empty(&staged->unflushed);
+	int ret;
+
+	ret = folio_alloc_swap(folio, may_sync_discard);
+	if (ret != -EBUSY)
+		return ret;
+
+	/*
+	 * Don't wait for synchronous discard I/O while keeping other folios
+	 * locked: there is potential for deadlocks and long delays.
+	 */
+	staged_folios_flush(staged);
+	may_sync_discard = true;
+	return folio_alloc_swap(folio, may_sync_discard);
+}
+
 /*
  * shrink_folio_list() returns the number of reclaimed pages
  */
@@ -1166,6 +1251,10 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 	struct folio_batch free_folios;
 	LIST_HEAD(ret_folios);
 	LIST_HEAD(demote_folios);
+	struct staged_folios staged = {
+		.unflushed = LIST_HEAD_INIT(staged.unflushed),
+		.flushed = LIST_HEAD_INIT(staged.flushed),
+	};
 	unsigned int nr_reclaimed = 0, nr_demoted = 0;
 	unsigned int pgactivate = 0;
 	bool do_demote_pass;
@@ -1177,15 +1266,41 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 	do_demote_pass = can_demote(pgdat->node_id, sc, memcg);
 
 retry:
-	while (!list_empty(folio_list)) {
+	/*
+	 * We take folios from folio_list to pre-process them (lock, check,
+	 * unmap). Then, they move to the staged(unflushed) list, from where
+	 * they move to the staged(flushed) list after a batched TLB flush
+	 * and after unlocking them. If no TLB flush is pending for a folio,
+	 * it moves to the staged(flushed) list directly.
+	 *
+	 * We then take folios from the staged(flushed) list to actually
+	 * reclaim them, or to activate them if they are still mapped.
+	 *
+	 * So we pre-process folios from folio_list until we collected a batch
+	 * of staged folios, or until there are no more folios to
+	 * pre-process. Then, we perform a single TLB flush and reclaim all
+	 * folios of that batch, before we pre-process more folios.
+	 *
+	 * Once all lists are empty, we are done.
+	 *
+	 * TODO: the stages of this algorithm should be cleanly separated.
+	 */
+	while (!list_empty(folio_list) || !staged_folios_empty(&staged)) {
 		struct address_space *mapping;
 		struct folio *folio;
 		enum folio_references references = FOLIOREF_RECLAIM;
 		bool dirty, writeback;
+		bool unmapped = false;
 		unsigned int nr_pages;
 
 		cond_resched();
 
+		folio = staged_folios_next(&staged, folio_list);
+		if (folio) {
+			nr_pages = folio_nr_pages(folio);
+			goto reclaim_staged;
+		}
+
 		folio = lru_to_folio(folio_list);
 		list_del(&folio->lru);
 
@@ -1324,6 +1439,11 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 			/* Case 3 above */
 			} else {
 				folio_unlock(folio);
+				/*
+				 * Don't wait for writeback while keeping
+				 * staged folios locked.
+				 */
+				staged_folios_flush(&staged);
 				folio_wait_writeback(folio);
 				/* then go back and try same folio again */
 				list_add_tail(&folio->lru, folio_list);
@@ -1382,7 +1502,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 				    split_folio_to_list(folio, folio_list))
 					goto activate_locked;
 			}
-			ret = folio_alloc_swap(folio);
+			ret = staged_folios_alloc_swap(&staged, folio);
 			if (ret) {
 				int __maybe_unused order = folio_order(folio);
 
@@ -1401,7 +1521,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 				}
 #endif
 				count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK);
-				if (folio_alloc_swap(folio))
+				if (staged_folios_alloc_swap(&staged, folio))
 					goto activate_locked_split;
 			}
 			/*
@@ -1455,21 +1575,36 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 				flags |= TTU_SYNC;
 
 			try_to_unmap(folio, flags);
-			/*
-			 * Flush deferred writable TLB entries before we could
-			 * drop the folio lock and others could start writeback.
-			 */
-			if (folio_test_dirty(folio))
-				try_to_unmap_flush_dirty();
+			unmapped = true;
 			if (folio_mapped(folio)) {
 				stat->nr_unmap_fail += nr_pages;
 				if (!was_swapbacked &&
 				    folio_test_swapbacked(folio))
 					stat->nr_lazyfree_fail += nr_pages;
-				goto activate_locked;
 			}
 		}
 
+		staged_folios_add(&staged, folio, unmapped);
+		continue;
+
+reclaim_staged:
+		if (!folio_trylock(folio))
+			goto keep;
+
+		/*
+		 * We failed to unmap the folio, or it was mapped again while it
+		 * was unlocked.
+		 */
+		if (folio_mapped(folio))
+			goto activate_locked;
+		/*
+		 * While the folio was unlocked, writeback might have started, or
+		 * it might have been poisoned.
+		 */
+		if (folio_test_writeback(folio) ||
+		    folio_contain_hwpoisoned_page(folio))
+			goto keep_locked;
+
 		/*
 		 * Folio is unmapped now so it cannot be newly pinned anymore.
 		 * No point in trying to reclaim folio if it is pinned.

-- 
2.56.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio
  2026-11-18  9:20 ` [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio David Hildenbrand (Arm)
@ 2026-11-18 13:55   ` Cenk Yılmaz
  0 siblings, 0 replies; 6+ messages in thread
From: Cenk Yılmaz @ 2026-11-18 13:55 UTC (permalink / raw)
  To: David Hildenbrand (Arm),
	Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
	Borislav Petkov (AMD),
	Zhang Peng, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
	Andrew Morton, Zi Yan, Matthew Brost, Joshua Hahn,
	Byungchul Park, Gregory Price, Ying Huang, Alistair Popple,
	Chris Li, Kairui Song, Kemeng Shi, Nhat Pham, Baoquan He,
	Barry Song, Youngjun Park, Mel Gorman, Sasha Levin, Rik van Riel,
	Ingo Molnar, Peter Xu, Hugh Dickins, Axel Rasmussen,
	Johannes Weiner, Michal Hocko, Qi Zheng, Shakeel Butt,
	Yuanchu Xie, Wei Xu, Baolin Wang, Andrea Arcangeli,
	Thomas Gleixner
  Cc: linux-mm, linux-kernel, stable

Tested-by: Cenk Yılmaz <cenk@deepfinresearch.com>

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-18 13:55 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-18  9:20 [PATCH 0/4] mm: fixes for stale TLB entries due to delayed/batched TLB flushes David Hildenbrand (Arm)
2026-11-18  9:20 ` [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio David Hildenbrand (Arm)
2026-11-18 13:55   ` Cenk Yılmaz
2026-11-18  9:20 ` [PATCH 2/4] mm/vmscan: flush deferred writable TLB entries before unlocking the folio David Hildenbrand (Arm)
2026-11-18  9:20 ` [PATCH 3/4] RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios David Hildenbrand (Arm)
2026-11-18  9:20 ` [PATCH 4/4] RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush David Hildenbrand (Arm)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®