[PATCH v6 0/5] mm: batch TLB flushing for dirty folios in vmscan

[PATCH v6 0/5] mm: batch TLB flushing for dirty folios in vmscan

[Zhang Peng]

This series introduces batch TLB flushing optimization for dirty folios
during memory reclaim, aiming to reduce IPI overhead on multi-core systems.

Background
----------
Reclaim calls try_to_unmap_flush_dirty() before writing out each dirty
folio. When writable TLB entries are pending, this triggers a flush.
On systems that use IPIs for TLB shootdowns, frequent flushes can add
significant overhead.

Approach
--------
Collect dirty folios into a folio_batch, releasing their locks while
collecting the batch so swap faults can progress. Before writeout, try
locking each folio again and recheck writeback, mappings and DMA pins.
Keep eligible folios locked, flush pending writable TLB entries once,
then page them out. Process a final partial batch at the end of the scan.

Changes
-------
Patch 1: Extract the folio activation block at activate_locked into
         folio_activate_locked().
Patch 2: Extract the folio-freeing path (buffer release, lazyfree,
         __remove_mapping, folio_batch drain) into folio_try_reclaim_free().
         Group retained folios, deferred frees and the reclaimed count
         in reclaim_folio_state, local to each shrink_folio_list() call.
Patch 3: Extract the pageout() dispatch state machine into folio_try_pageout().
Patch 4: Extract the TTU setup and try_to_unmap() block into folio_try_unmap().
Patch 5: Implement batch TLB flushing logic. Dirty folios are accumulated in
         batches and a single TLB flush is performed for each batch before
         calling pageout.

Testing
-------
The benchmark script uses stress-ng to compare TLB shootdown behavior before and
after this patch. It constrains a stress-ng workload via memcg to force reclaim
through shrink_folio_list(), reporting TLB shootdowns and IPIs.

Core benchmark command: stress-ng --vm 16 --vm-bytes 2G --vm-keep --timeout 60

==========================================================================
                 batch_dirty_tlb_flush Benchmark Results
==========================================================================
  Kernel: 7.0.0-rc1+   CPUs: 16
  MemTotal: 31834M   SwapTotal: 8191M
  memcg limit: 512M   alloc: 2G   workers: 16   duration: 60s
--------------------------------------------------------------------------
Metric                 Before        After             Delta (abs / %)
--------------------------------------------------------------------------
bogo ops/s             28238.63      35833.97          +7595.34 (+26.9%)
TLB shootdowns         55428953      17621697          -37807256 (-68.2%)
Function call IPIs     34073695      14498768          -19574927 (-57.4%)
pgscan_anon (pages)    52856224      60252894          7396670 (+14.0%)
pgsteal_anon (pages)   29004962      34054753          5049791 (+17.4%)
--------------------------------------------------------------------------

Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Zhang Peng <bruzzhang@tencent.com>
---
Changes in v6:
- Keep the four preparatory patches and the batching patch in one series,
  as requested by Barry.
- Patch 1: Clarify the folio_activate_locked() comment and retain Barry's
  Reviewed-by. Document the BUG-to-WARN change in the commit message.
- Patch 2: Return KEEP, ACTIVATE or DONE explicitly, leaving activation
  visible in the caller as requested by Barry. Group retained folios,
  deferred frees and the reclaimed count in reclaim_folio_state.
- Patch 3: Return explicit pageout outcomes, including separate locked
  and unlocked keep results. Leave activation and the subsequent reclaim
  attempt in the caller. Document lock and ownership contracts and make
  the mapping pointer and saved page count const.
- Patch 4: Retain Barry's Reviewed-by.
- Patch 5: Use the explicit helper results and reclaim_folio_state,
  reducing the batch helper from eight arguments to six in response to
  David's suggestion.
- Patch 5: Replace v5's temporary pointer array with batch reuse. Save
  the original count, reinitialize the batch, and use folio_batch_add()
  to collect folios that pass validation. Document that reinitialization
  preserves the array and additions cannot overwrite unread entries.
- Patch 5: Explain the writeback, mapping and DMA-pin rechecks, including
  why the folio lock alone does not exclude GUP. Rename the batch and
  helper around pageout and remove the redundant next label.
- Link to v5:
  https://patch.msgid.link/20260720-batch-tlb-flush-v5-0-db943a0d0d6b@icloud.com

Changes in v5 (addressing David Hildenbrand's review on v4):
- Patch 1: Drop the nr_pages parameter; use folio_nr_pages() inside
  folio_activate_locked(). Add VM_WARN_ON_ONCE_FOLIO(!folio_test_locked)
  at the top of the function. Turn the VM_BUG_ON_FOLIO(folio_test_active)
  into VM_WARN_ON_ONCE_FOLIO and move it to the top.
- Patch 2: Rename folio_free() to folio_try_reclaim_free() to make the
  reclaim-specific role clear and justify the bool return. Make nr_pages
  const. Add VM_WARN_ON_ONCE_FOLIO(folio_ref_count, folio) at the free_it:
  label. Drop the now-redundant "swapped out as a whole" comment and the
  "else continue" after goto keep_locked.
- Patch 3: Rename pageout_one() to folio_try_pageout() for consistency
  with the other helpers. Fix continuation-line alignment.
- Patch 4: Fix continuation-line alignment.
- Patch 5: Replace the in-place folio_batch_reinit() + walk pattern
  with a plain local array; @fbatch is now only read during the walk
  and cleared once afterwards, so its invariant is never temporarily
  broken. Add a comment documenting why each recheck (writeback, mapped,
  dma_pinned) is required.
- Link to v4: https://lore.kernel.org/r/20260525-batch-tlb-flush-v4-0-83789d6abc00@icloud.com

Changes in v4 (addressing Barry Song's review on v3):
- Drop the "track reclaimed pages in reclaim_stat" patch; keep
  shrink_folio_list() returning nr_reclaimed directly. Avoids touching
  the function signature and its MGLRU evict_folios() and
  reclaim_clean_pages_from_list() callers in this series.
- Rename folio_active_bounce() to folio_activate_locked(). The new name
  reflects the precondition (the folio is locked) that callers care about.
- Split the folio_free()/pageout_one() extraction into two patches;
  make pageout_one() return bool so shrink_folio_list() can see whether
  the folio was reclaimed or kept.
- Move the !folio_mapped() check out of folio_try_unmap() into the
  caller, so folio_try_unmap() is only invoked for mapped folios.
- Link to v3: https://lore.kernel.org/r/20260410-batch-tlb-flush-v3-0-ff0b9d3a351a@icloud.com

Changes in v3:
- Patch 5: Replace folio_test_lru() condition check with
  VM_WARN_ON_FOLIO assertion, as PG_lru should never be set for
  isolated folios
- Patch 5: Add comment explaining folio_batch reuse-in-place
  technique in pageout_batch()
- Patch 5: Rewrite comment above folio_unlock() to explain why the
  folio is unlocked while batching
- Link to v2: https://lore.kernel.org/r/20260326-batch-tlb-flush-v2-0-403e523325c4@icloud.com

Changes in v2:
- Fix incorrect comment about page_ref_freeze
- Add folio_maybe_dma_pinned() check in pageout_batch()
- Link to v1: https://lore.kernel.org/r/20260309-batch-tlb-flush-v1-0-eb8fed7d1a9e@icloud.com

---
Zhang Peng (5):
      mm/vmscan: introduce folio_activate_locked() helper
      mm/vmscan: extract folio reclaim freeing from shrink_folio_list()
      mm/vmscan: extract folio pageout from shrink_folio_list()
      mm/vmscan: extract folio unmap logic into folio_try_unmap()
      mm/vmscan: batch dirty TLB flushes before pageout

 mm/vmscan.c | 510 +++++++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 335 insertions(+), 175 deletions(-)
---
base-commit: 9c449f885765cf8d80b2c60b1971e774bba51a3f
change-id: 20260309-batch-tlb-flush-893f0e56b496

Best regards,
-- 
Zhang Peng <bruzzhang@tencent.com>

[PATCH v6 1/5] mm/vmscan: introduce folio_activate_locked() helper

[Zhang Peng]

From: Zhang Peng <bruzzhang@tencent.com>

The activate_locked label in shrink_folio_list() reclaims swap cache
when needed, marks the folio active, and updates activation statistics.
Extract this block into folio_activate_locked() so it can be reused.

Replace the active-folio BUG with a warning so a bad caller is reported
without taking down the system. Otherwise, there is no functional change.

Reviewed-by: Barry Song <baohua@kernel.org>
Signed-off-by: Zhang Peng <bruzzhang@tencent.com>
---
 mm/vmscan.c | 40 +++++++++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 15 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index fb9f8f020f63..54f6a9e62715 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1155,6 +1155,28 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask)
 	return false;
 }
 
+/* Activate an isolated, locked folio and account the activation. */
+static void folio_activate_locked(struct folio *folio,
+		struct reclaim_stat *stat)
+{
+	unsigned int nr_pages = folio_nr_pages(folio);
+
+	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+	VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
+
+	/* Not a candidate for swapping, so reclaim swap space. */
+	if (folio_test_swapcache(folio) &&
+	    (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio)))
+		folio_free_swap(folio);
+	if (!folio_test_mlocked(folio)) {
+		int type = folio_is_file_lru(folio);
+
+		folio_set_active(folio);
+		stat->nr_activate[type] += nr_pages;
+		count_memcg_folio_events(folio, PGACTIVATE, nr_pages);
+	}
+}
+
 /*
  * shrink_folio_list() returns the number of reclaimed pages
  */
@@ -1623,24 +1645,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 activate_locked_split:
 		/*
 		 * The tail pages that are failed to add into swap cache
-		 * reach here.  Fixup nr_scanned and nr_pages.
+		 * reach here.  Fixup nr_scanned.
 		 */
-		if (nr_pages > 1) {
+		if (nr_pages > 1)
 			sc->nr_scanned -= (nr_pages - 1);
-			nr_pages = 1;
-		}
 activate_locked:
-		/* Not a candidate for swapping, so reclaim swap space. */
-		if (folio_test_swapcache(folio) &&
-		    (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio)))
-			folio_free_swap(folio);
-		VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
-		if (!folio_test_mlocked(folio)) {
-			int type = folio_is_file_lru(folio);
-			folio_set_active(folio);
-			stat->nr_activate[type] += nr_pages;
-			count_memcg_folio_events(folio, PGACTIVATE, nr_pages);
-		}
+		folio_activate_locked(folio, stat);
 keep_locked:
 		folio_unlock(folio);
 keep:

-- 
2.56.0

[PATCH v6 2/5] mm/vmscan: extract folio reclaim freeing from shrink_folio_list()

[Zhang Peng]

From: Zhang Peng <bruzzhang@tencent.com>

shrink_folio_list() contains a self-contained folio-freeing section:
buffer release, lazyfree, __remove_mapping(), and folio_batch draining.
Extract it into folio_try_reclaim_free() to reduce the size of
shrink_folio_list() and make the freeing step independently readable.

Return an explicit result so the caller retains the distinction between
activating a folio, keeping it on the inactive list, and reclaiming it.
The helper leaves the folio locked when it returns ACTIVATE or KEEP and
consumes it when it returns DONE.

Group retained folios, deferred frees and the reclaimed count in a
reclaim_folio_state shared by the reclaim paths. Initialize it once
per shrink_folio_list() invocation, including demotion retries.

No functional change.

Signed-off-by: Zhang Peng <bruzzhang@tencent.com>
---
 mm/vmscan.c | 201 +++++++++++++++++++++++++++++++++++-------------------------
 1 file changed, 116 insertions(+), 85 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 54f6a9e62715..9ac125f2e317 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1177,6 +1177,106 @@ static void folio_activate_locked(struct folio *folio,
 	}
 }
 
+/* Results accumulated across a shrink_folio_list() invocation. */
+struct reclaim_folio_state {
+	struct list_head ret_folios;
+	struct folio_batch free_folios;
+	unsigned int nr_reclaimed;
+};
+
+enum folio_reclaim_result {
+	FOLIO_RECLAIM_KEEP,
+	FOLIO_RECLAIM_ACTIVATE,
+	FOLIO_RECLAIM_DONE,
+};
+
+/*
+ * Called with the folio locked. KEEP and ACTIVATE leave it locked and
+ * owned by the caller. DONE consumes the caller's reference; the caller
+ * must not access the folio again, even if freeing is deferred.
+ */
+static enum folio_reclaim_result folio_try_reclaim_free(struct folio *folio,
+		struct reclaim_folio_state *state,
+		struct scan_control *sc)
+{
+	const unsigned int nr_pages = folio_nr_pages(folio);
+	struct address_space *mapping = folio_mapping(folio);
+
+	/*
+	 * If the folio has buffers, try to free the buffer mappings
+	 * associated with this folio. If we succeed we try to free
+	 * the folio as well.
+	 *
+	 * We do this even if the folio is dirty.
+	 * filemap_release_folio() does not perform I/O, but it is
+	 * possible for a folio to have the dirty flag set, but it
+	 * is actually clean (all its buffers are clean).  This
+	 * happens if the buffers were written out directly, with
+	 * bh_submit(). ext3 will do this, as well as the blockdev
+	 * mapping.  filemap_release_folio() will discover that
+	 * cleanness and will drop the buffers and mark the folio
+	 * clean - it can be freed.
+	 *
+	 * Rarely, folios can have buffers and no ->mapping.  These
+	 * are the folios which were not successfully invalidated in
+	 * truncate_cleanup_folio().  We try to drop those buffers
+	 * here and if that worked, and the folio is no longer
+	 * mapped into process address space (refcount == 1) it can
+	 * be freed.  Otherwise, leave the folio on the LRU so it is
+	 * swappable.
+	 */
+	if (folio_needs_release(folio)) {
+		if (!filemap_release_folio(folio, sc->gfp_mask))
+			return FOLIO_RECLAIM_ACTIVATE;
+
+		if (!mapping && folio_ref_count(folio) == 1) {
+			folio_unlock(folio);
+			if (folio_put_testzero(folio))
+				goto free_it;
+
+			/*
+			 * Rare race with speculative reference. The
+			 * speculative reference will free this folio
+			 * shortly, so we may increment nr_reclaimed here
+			 * and leave it off the LRU.
+			 */
+			state->nr_reclaimed += nr_pages;
+			return FOLIO_RECLAIM_DONE;
+		}
+	}
+
+	if (folio_test_lazyfree(folio)) {
+		/* follow __remove_mapping for reference */
+		if (!folio_ref_freeze(folio, 1))
+			return FOLIO_RECLAIM_KEEP;
+		/*
+		 * The folio has only one reference left, which is
+		 * from the isolation. After the caller puts the
+		 * folio back on the lru and drops the reference, the
+		 * folio will be freed anyway. It doesn't matter
+		 * which lru it goes on. So we don't bother checking
+		 * the dirty flag here.
+		 */
+		count_vm_events(PGLAZYFREED, nr_pages);
+		count_memcg_folio_events(folio, PGLAZYFREED, nr_pages);
+	} else if (!mapping || !__remove_mapping(mapping, folio, true,
+							sc->target_mem_cgroup))
+		return FOLIO_RECLAIM_KEEP;
+
+	folio_unlock(folio);
+free_it:
+	VM_WARN_ON_ONCE_FOLIO(folio_ref_count(folio), folio);
+	state->nr_reclaimed += nr_pages;
+
+	folio_unqueue_deferred_split(folio);
+	if (folio_batch_add(&state->free_folios, folio) == 0) {
+		mem_cgroup_uncharge_folios(&state->free_folios);
+		try_to_unmap_flush();
+		free_unref_folios(&state->free_folios);
+	}
+	return FOLIO_RECLAIM_DONE;
+}
+
 /*
  * shrink_folio_list() returns the number of reclaimed pages
  */
@@ -1185,15 +1285,15 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		struct reclaim_stat *stat, bool ignore_references,
 		struct mem_cgroup *memcg)
 {
-	struct folio_batch free_folios;
-	LIST_HEAD(ret_folios);
+	struct reclaim_folio_state state = {};
 	LIST_HEAD(demote_folios);
-	unsigned int nr_reclaimed = 0, nr_demoted = 0;
+	unsigned int nr_demoted = 0;
 	unsigned int pgactivate = 0;
 	bool do_demote_pass;
 	struct swap_io_ctx ctx = {};
 
-	folio_batch_init(&free_folios);
+	INIT_LIST_HEAD(&state.ret_folios);
+	folio_batch_init(&state.free_folios);
 	memset(stat, 0, sizeof(*stat));
 	cond_resched();
 	do_demote_pass = can_demote(pgdat->node_id, sc, memcg);
@@ -1564,83 +1664,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 			}
 		}
 
-		/*
-		 * If the folio has buffers, try to free the buffer
-		 * mappings associated with this folio. If we succeed
-		 * we try to free the folio as well.
-		 *
-		 * We do this even if the folio is dirty.
-		 * filemap_release_folio() does not perform I/O, but it
-		 * is possible for a folio to have the dirty flag set,
-		 * but it is actually clean (all its buffers are clean).
-		 * This happens if the buffers were written out directly,
-		 * with bh_submit(). ext3 will do this, as well as
-		 * the blockdev mapping.  filemap_release_folio() will
-		 * discover that cleanness and will drop the buffers
-		 * and mark the folio clean - it can be freed.
-		 *
-		 * Rarely, folios can have buffers and no ->mapping.
-		 * These are the folios which were not successfully
-		 * invalidated in truncate_cleanup_folio().  We try to
-		 * drop those buffers here and if that worked, and the
-		 * folio is no longer mapped into process address space
-		 * (refcount == 1) it can be freed.  Otherwise, leave
-		 * the folio on the LRU so it is swappable.
-		 */
-		if (folio_needs_release(folio)) {
-			if (!filemap_release_folio(folio, sc->gfp_mask))
-				goto activate_locked;
-			if (!mapping && folio_ref_count(folio) == 1) {
-				folio_unlock(folio);
-				if (folio_put_testzero(folio))
-					goto free_it;
-				else {
-					/*
-					 * rare race with speculative reference.
-					 * the speculative reference will free
-					 * this folio shortly, so we may
-					 * increment nr_reclaimed here (and
-					 * leave it off the LRU).
-					 */
-					nr_reclaimed += nr_pages;
-					continue;
-				}
-			}
-		}
-
-		if (folio_test_lazyfree(folio)) {
-			/* follow __remove_mapping for reference */
-			if (!folio_ref_freeze(folio, 1))
-				goto keep_locked;
-			/*
-			 * The folio has only one reference left, which is
-			 * from the isolation. After the caller puts the
-			 * folio back on the lru and drops the reference, the
-			 * folio will be freed anyway. It doesn't matter
-			 * which lru it goes on. So we don't bother checking
-			 * the dirty flag here.
-			 */
-			count_vm_events(PGLAZYFREED, nr_pages);
-			count_memcg_folio_events(folio, PGLAZYFREED, nr_pages);
-		} else if (!mapping || !__remove_mapping(mapping, folio, true,
-							 sc->target_mem_cgroup))
+		switch (folio_try_reclaim_free(folio, &state, sc)) {
+		case FOLIO_RECLAIM_ACTIVATE:
+			goto activate_locked;
+		case FOLIO_RECLAIM_KEEP:
 			goto keep_locked;
-
-		folio_unlock(folio);
-free_it:
-		/*
-		 * Folio may get swapped out as a whole, need to account
-		 * all pages in it.
-		 */
-		nr_reclaimed += nr_pages;
-
-		folio_unqueue_deferred_split(folio);
-		if (folio_batch_add(&free_folios, folio) == 0) {
-			mem_cgroup_uncharge_folios(&free_folios);
-			try_to_unmap_flush();
-			free_unref_folios(&free_folios);
+		case FOLIO_RECLAIM_DONE:
+			continue;
 		}
-		continue;
 
 activate_locked_split:
 		/*
@@ -1654,7 +1685,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 keep_locked:
 		folio_unlock(folio);
 keep:
-		list_add(&folio->lru, &ret_folios);
+		list_add(&folio->lru, &state.ret_folios);
 		VM_BUG_ON_FOLIO(folio_test_lru(folio) ||
 				folio_test_unevictable(folio), folio);
 	}
@@ -1662,7 +1693,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 	/* Migrate folios selected for demotion */
 	nr_demoted = demote_folio_list(&demote_folios, pgdat, memcg);
-	nr_reclaimed += nr_demoted;
+	state.nr_reclaimed += nr_demoted;
 	stat->nr_demoted += nr_demoted;
 	/* Folios that could not be demoted are still in @demote_folios */
 	if (!list_empty(&demote_folios)) {
@@ -1693,15 +1724,15 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 	pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 
-	mem_cgroup_uncharge_folios(&free_folios);
+	mem_cgroup_uncharge_folios(&state.free_folios);
 	try_to_unmap_flush();
-	free_unref_folios(&free_folios);
+	free_unref_folios(&state.free_folios);
 
-	list_splice(&ret_folios, folio_list);
+	list_splice(&state.ret_folios, folio_list);
 	count_vm_events(PGACTIVATE, pgactivate);
 
 	swap_write_submit(&ctx);
-	return nr_reclaimed;
+	return state.nr_reclaimed;
 }
 
 unsigned int reclaim_clean_pages_from_list(struct zone *zone,

-- 
2.56.0

[PATCH v6 3/5] mm/vmscan: extract folio pageout from shrink_folio_list()

[Zhang Peng]

From: Zhang Peng <bruzzhang@tencent.com>

shrink_folio_list() contains a self-contained pageout() dispatch state
machine. Extract it into folio_try_pageout() so it can be reused by the
batched pageout path.

Return an explicit pageout result instead of a boolean. This keeps the
important distinction between activating and merely keeping a folio
visible to the caller, and also records whether a kept folio remains
locked after pageout().

The clean and synchronous-write cases return FOLIO_PAGEOUT_RECLAIM: the
folio is still locked there, and shrink_folio_list() then runs its
single folio_try_reclaim_free() call, so the reclaim-free step keeps
exactly one caller.

The PAGE_ACTIVATE arm no longer normalises nr_pages to 1 after a split:
folio_activate_locked() derives the count from folio_nr_pages() itself,
which is equal by construction once the folio is order-0.  Note this
makes the patch depend on patch 1; applying it alone would over-account
nr_activate for a split shmem folio.

No functional change.

Signed-off-by: Zhang Peng <bruzzhang@tencent.com>
---
 mm/vmscan.c | 105 ++++++++++++++++++++++++++++++++++++++----------------------
 1 file changed, 66 insertions(+), 39 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9ac125f2e317..36e3f0829f08 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1277,8 +1277,65 @@ static enum folio_reclaim_result folio_try_reclaim_free(struct folio *folio,
 	return FOLIO_RECLAIM_DONE;
 }
 
+enum folio_pageout_result {
+	FOLIO_PAGEOUT_KEEP_LOCKED,
+	FOLIO_PAGEOUT_KEEP_UNLOCKED,
+	FOLIO_PAGEOUT_ACTIVATE,
+	FOLIO_PAGEOUT_RECLAIM,	/* folio is locked, hand it to folio_try_reclaim_free() */
+};
+
+/*
+ * Called with the folio locked. All results leave it locked except
+ * KEEP_UNLOCKED. The caller retains ownership in every case; RECLAIM
+ * only allows a subsequent reclaim attempt, it does not free the folio.
+ */
+static enum folio_pageout_result folio_try_pageout(struct folio *folio,
+		struct scan_control *sc, struct swap_io_ctx *ctx,
+		struct list_head *folio_list)
+{
+	struct address_space * const mapping = folio_mapping(folio);
+	const unsigned int nr_pages = folio_nr_pages(folio);
+
+	switch (pageout(ctx, mapping, folio, folio_list)) {
+	case PAGE_ACTIVATE:
+		/*
+		 * If shmem folio is split when writeback to swap, the
+		 * tail pages will make their own pass through this
+		 * function and be accounted then.  There is no need to
+		 * clamp nr_pages here: folio_activate_locked() derives
+		 * the count from folio_nr_pages() itself.
+		 */
+		if (nr_pages > 1 && !folio_test_large(folio))
+			sc->nr_scanned -= (nr_pages - 1);
+		return FOLIO_PAGEOUT_ACTIVATE;
+	case PAGE_KEEP:
+		return FOLIO_PAGEOUT_KEEP_LOCKED;
+	case PAGE_SUCCESS:
+		if (nr_pages > 1 && !folio_test_large(folio))
+			sc->nr_scanned -= (nr_pages - 1);
+
+		if (folio_test_writeback(folio))
+			return FOLIO_PAGEOUT_KEEP_UNLOCKED;
+		if (folio_test_dirty(folio))
+			return FOLIO_PAGEOUT_KEEP_UNLOCKED;
+
+		/*
+		 * A synchronous write - probably a ramdisk.  Go ahead
+		 * and try to reclaim the folio.
+		 */
+		if (!folio_trylock(folio))
+			return FOLIO_PAGEOUT_KEEP_UNLOCKED;
+		if (folio_test_dirty(folio) || folio_test_writeback(folio))
+			return FOLIO_PAGEOUT_KEEP_LOCKED;
+		fallthrough;
+	case PAGE_CLEAN:
+		return FOLIO_PAGEOUT_RECLAIM;
+	}
+	unreachable();
+}
+
 /*
- * shrink_folio_list() returns the number of reclaimed pages
+ * Reclaimed folios are counted in the return value.
  */
 static unsigned int shrink_folio_list(struct list_head *folio_list,
 		struct pglist_data *pgdat, struct scan_control *sc,
@@ -1596,7 +1653,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		if (folio_maybe_dma_pinned(folio))
 			goto activate_locked;
 
-		mapping = folio_mapping(folio);
 		if (folio_test_dirty(folio)) {
 			if (folio_is_file_lru(folio)) {
 				/*
@@ -1617,50 +1673,21 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 				goto keep_locked;
 			if (!sc->may_writepage)
 				goto keep_locked;
-
 			/*
 			 * Folio is dirty. Flush the TLB if a writable entry
 			 * potentially exists to avoid CPU writes after I/O
 			 * starts and then write it out here.
 			 */
 			try_to_unmap_flush_dirty();
-			switch (pageout(&ctx, mapping, folio, folio_list)) {
-			case PAGE_KEEP:
-				goto keep_locked;
-			case PAGE_ACTIVATE:
-				/*
-				 * If shmem folio is split when writeback to swap,
-				 * the tail pages will make their own pass through
-				 * this function and be accounted then.
-				 */
-				if (nr_pages > 1 && !folio_test_large(folio)) {
-					sc->nr_scanned -= (nr_pages - 1);
-					nr_pages = 1;
-				}
+			switch (folio_try_pageout(folio, sc, &ctx, folio_list)) {
+			case FOLIO_PAGEOUT_ACTIVATE:
 				goto activate_locked;
-			case PAGE_SUCCESS:
-				if (nr_pages > 1 && !folio_test_large(folio)) {
-					sc->nr_scanned -= (nr_pages - 1);
-					nr_pages = 1;
-				}
-				if (folio_test_writeback(folio))
-					goto keep;
-				if (folio_test_dirty(folio))
-					goto keep;
-
-				/*
-				 * A synchronous write - probably a ramdisk.  Go
-				 * ahead and try to reclaim the folio.
-				 */
-				if (!folio_trylock(folio))
-					goto keep;
-				if (folio_test_dirty(folio) ||
-				    folio_test_writeback(folio))
-					goto keep_locked;
-				mapping = folio_mapping(folio);
-				fallthrough;
-			case PAGE_CLEAN:
-				; /* try to free the folio below */
+			case FOLIO_PAGEOUT_KEEP_LOCKED:
+				goto keep_locked;
+			case FOLIO_PAGEOUT_KEEP_UNLOCKED:
+				goto keep;
+			case FOLIO_PAGEOUT_RECLAIM:
+				break;	/* folio is locked; try to free it below */
 			}
 		}
 

-- 
2.56.0

[PATCH v6 4/5] mm/vmscan: extract folio unmap logic into folio_try_unmap()

[Zhang Peng]

From: Zhang Peng <bruzzhang@tencent.com>

shrink_folio_list() contains a self-contained block that sets up
TTU flags and calls try_to_unmap(), accounting for failures via
reclaim_stat. Extract it into folio_try_unmap() to reduce the size
of shrink_folio_list() and make the unmap step independently readable.

folio_try_unmap() is only called when the folio is actually mapped;
the !folio_mapped() check stays in the caller, keeping the function's
semantics clear: it tries to unmap a mapped folio and returns whether
the unmap succeeded.

No functional change.

Reviewed-by: Barry Song <baohua@kernel.org>
Signed-off-by: Zhang Peng <bruzzhang@tencent.com>
---
 mm/vmscan.c | 65 +++++++++++++++++++++++++++++++++----------------------------
 1 file changed, 35 insertions(+), 30 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 36e3f0829f08..061414ff45bb 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1334,6 +1334,38 @@ static enum folio_pageout_result folio_try_pageout(struct folio *folio,
 	unreachable();
 }
 
+static bool folio_try_unmap(struct folio *folio, struct reclaim_stat *stat,
+		unsigned int nr_pages)
+{
+	enum ttu_flags flags = TTU_BATCH_FLUSH;
+	const bool was_swapbacked = folio_test_swapbacked(folio);
+
+	if (folio_test_pmd_mappable(folio))
+		flags |= TTU_SPLIT_HUGE_PMD;
+	/*
+	 * Without TTU_SYNC, try_to_unmap will only begin to hold PTL
+	 * from the first present PTE within a large folio. Some
+	 * initial PTEs might be skipped due to races with parallel
+	 * PTE writes in which PTEs can be cleared temporarily before
+	 * being written new present values. This will lead to a large
+	 * folio is still mapped while some subpages have been
+	 * partially unmapped after try_to_unmap; TTU_SYNC helps
+	 * try_to_unmap acquire PTL from the first PTE, eliminating the
+	 * influence of temporary PTE values.
+	 */
+	if (folio_test_large(folio))
+		flags |= TTU_SYNC;
+
+	try_to_unmap(folio, flags);
+	if (folio_mapped(folio)) {
+		stat->nr_unmap_fail += nr_pages;
+		if (!was_swapbacked && folio_test_swapbacked(folio))
+			stat->nr_lazyfree_fail += nr_pages;
+		return false;
+	}
+	return true;
+}
+
 /*
  * Reclaimed folios are counted in the return value.
  */
@@ -1612,36 +1644,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		 * The folio is mapped into the page tables of one or more
 		 * processes. Try to unmap it here.
 		 */
-		if (folio_mapped(folio)) {
-			enum ttu_flags flags = TTU_BATCH_FLUSH;
-			bool was_swapbacked = folio_test_swapbacked(folio);
-
-			if (folio_test_pmd_mappable(folio))
-				flags |= TTU_SPLIT_HUGE_PMD;
-			/*
-			 * Without TTU_SYNC, try_to_unmap will only begin to
-			 * hold PTL from the first present PTE within a large
-			 * folio. Some initial PTEs might be skipped due to
-			 * races with parallel PTE writes in which PTEs can be
-			 * cleared temporarily before being written new present
-			 * values. This will lead to a large folio is still
-			 * mapped while some subpages have been partially
-			 * unmapped after try_to_unmap; TTU_SYNC helps
-			 * try_to_unmap acquire PTL from the first PTE,
-			 * eliminating the influence of temporary PTE values.
-			 */
-			if (folio_test_large(folio))
-				flags |= TTU_SYNC;
-
-			try_to_unmap(folio, flags);
-			if (folio_mapped(folio)) {
-				stat->nr_unmap_fail += nr_pages;
-				if (!was_swapbacked &&
-				    folio_test_swapbacked(folio))
-					stat->nr_lazyfree_fail += nr_pages;
-				goto activate_locked;
-			}
-		}
+		if (folio_mapped(folio) &&
+		    !folio_try_unmap(folio, stat, nr_pages))
+			goto activate_locked;
 
 		/*
 		 * Folio is unmapped now so it cannot be newly pinned anymore.

-- 
2.56.0

[PATCH v6 5/5] mm/vmscan: batch dirty TLB flushes before pageout

[Zhang Peng]

Currently reclaim flushes the TLB for every dirty folio before pageout.
On systems with many CPUs, the resulting IPIs can be a significant
bottleneck.

Collect dirty folios in a folio_batch instead. When the batch is full,
lock and revalidate each folio, flush pending writable TLB entries once,
and page out the folios that remain eligible. Flush a final partial batch
at the end of the scan.

Folios are unlocked while waiting in the batch so swap faults can make
progress. After reacquiring the folio lock, recheck writeback, mappings,
and DMA pins; a concurrent swapin may have installed a PTE and allowed a
parallel GUP pin while the folio was unlocked. Keep each unmapped folio
locked across the TLB flush and pageout to prevent swap faults from
remapping it before I/O starts. The folio lock alone does not exclude GUP.

Save the original batch count and reinitialize the batch, then add back
folios that pass validation. Reinitialization leaves the array intact,
and each add writes at or before the current read index. This reuses
the batch without overwriting unread entries or needing a second array.

Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Zhang Peng <bruzzhang@tencent.com>
---
 mm/vmscan.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 101 insertions(+), 14 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 061414ff45bb..1150361e7d52 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1334,6 +1334,92 @@ static enum folio_pageout_result folio_try_pageout(struct folio *folio,
 	unreachable();
 }
 
+static void folio_batch_pageout(struct folio_batch *batch,
+		struct reclaim_folio_state *state, struct scan_control *sc,
+		struct reclaim_stat *stat, struct swap_io_ctx *ctx,
+		struct list_head *folio_list)
+{
+	unsigned int nr = folio_batch_count(batch);
+	unsigned int i;
+
+	/*
+	 * Reuse this private batch to collect the locked folios. Reinit
+	 * leaves the array intact; nr preserves the original input count.
+	 * Adding each accepted folio writes at or before the current read
+	 * index, so entries we have not read yet cannot be overwritten.
+	 */
+	folio_batch_reinit(batch);
+
+	/*
+	 * Revalidate after taking the lock: writeback may have started, or
+	 * a swap fault may have remapped the folio and allowed GUP to pin it.
+	 * Keeping an unmapped folio locked prevents swap faults from mapping
+	 * it again before the TLB flush and pageout. The lock alone does not
+	 * exclude GUP, so existing mappings and pins must both be checked.
+	 */
+	for (i = 0; i < nr; i++) {
+		struct folio *folio = batch->folios[i];
+
+		if (!folio_trylock(folio)) {
+			list_add(&folio->lru, &state->ret_folios);
+			continue;
+		}
+
+		VM_WARN_ON_FOLIO(folio_test_lru(folio), folio);
+
+		/* The folio may have changed while it was unlocked. */
+		if (folio_test_writeback(folio) || folio_mapped(folio) ||
+		    folio_maybe_dma_pinned(folio)) {
+			folio_unlock(folio);
+			list_add(&folio->lru, &state->ret_folios);
+			continue;
+		}
+
+		folio_batch_add(batch, folio);
+	}
+	nr = folio_batch_count(batch);
+
+	if (!nr)
+		return;
+
+	/*
+	 * Flush writable TLB entries before any folio in the batch is
+	 * submitted for I/O.
+	 */
+	try_to_unmap_flush_dirty();
+
+	for (i = 0; i < nr; i++) {
+		struct folio *folio = batch->folios[i];
+
+		switch (folio_try_pageout(folio, sc, ctx, folio_list)) {
+		case FOLIO_PAGEOUT_ACTIVATE:
+			goto activate_locked;
+		case FOLIO_PAGEOUT_KEEP_LOCKED:
+			goto keep_locked;
+		case FOLIO_PAGEOUT_KEEP_UNLOCKED:
+			goto keep;
+		case FOLIO_PAGEOUT_RECLAIM:
+			break;	/* folio is locked; try to free it below */
+		}
+
+		switch (folio_try_reclaim_free(folio, state, sc)) {
+		case FOLIO_RECLAIM_ACTIVATE:
+			goto activate_locked;
+		case FOLIO_RECLAIM_KEEP:
+			goto keep_locked;
+		case FOLIO_RECLAIM_DONE:
+			continue;
+		}
+activate_locked:
+		folio_activate_locked(folio, stat);
+keep_locked:
+		folio_unlock(folio);
+keep:
+		list_add(&folio->lru, &state->ret_folios);
+	}
+	folio_batch_reinit(batch);
+}
+
 static bool folio_try_unmap(struct folio *folio, struct reclaim_stat *stat,
 		unsigned int nr_pages)
 {
@@ -1375,6 +1461,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		struct mem_cgroup *memcg)
 {
 	struct reclaim_folio_state state = {};
+	struct folio_batch pageout_folios;
 	LIST_HEAD(demote_folios);
 	unsigned int nr_demoted = 0;
 	unsigned int pgactivate = 0;
@@ -1383,6 +1470,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 	INIT_LIST_HEAD(&state.ret_folios);
 	folio_batch_init(&state.free_folios);
+	folio_batch_init(&pageout_folios);
 	memset(stat, 0, sizeof(*stat));
 	cond_resched();
 	do_demote_pass = can_demote(pgdat->node_id, sc, memcg);
@@ -1679,21 +1767,16 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 			if (!sc->may_writepage)
 				goto keep_locked;
 			/*
-			 * Folio is dirty. Flush the TLB if a writable entry
-			 * potentially exists to avoid CPU writes after I/O
-			 * starts and then write it out here.
+			 * Drop the lock so swap faults finding this folio
+			 * via swap cache lookup can make progress; the
+			 * recheck that this necessitates is documented in
+			 * folio_batch_pageout().
 			 */
-			try_to_unmap_flush_dirty();
-			switch (folio_try_pageout(folio, sc, &ctx, folio_list)) {
-			case FOLIO_PAGEOUT_ACTIVATE:
-				goto activate_locked;
-			case FOLIO_PAGEOUT_KEEP_LOCKED:
-				goto keep_locked;
-			case FOLIO_PAGEOUT_KEEP_UNLOCKED:
-				goto keep;
-			case FOLIO_PAGEOUT_RECLAIM:
-				break;	/* folio is locked; try to free it below */
-			}
+			folio_unlock(folio);
+			if (!folio_batch_add(&pageout_folios, folio))
+				folio_batch_pageout(&pageout_folios, &state,
+						    sc, stat, &ctx, folio_list);
+			continue;
 		}
 
 		switch (folio_try_reclaim_free(folio, &state, sc)) {
@@ -1721,6 +1804,10 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		VM_BUG_ON_FOLIO(folio_test_lru(folio) ||
 				folio_test_unevictable(folio), folio);
 	}
+	if (folio_batch_count(&pageout_folios)) {
+		folio_batch_pageout(&pageout_folios, &state, sc, stat,
+				    &ctx, folio_list);
+	}
 	/* 'folio_list' is always empty here */
 
 	/* Migrate folios selected for demotion */

-- 
2.56.0