[PATCH v3 0/3] mm: fix workingset refaults in the zswap writeback path

[PATCH v3 0/3] mm: fix workingset refaults in the zswap writeback path

[Alexandre Ghiti]

Note: patch 1 also appears as patch 1 of the zswap dropbehind series [1].
It is the same patch, byte for byte. Both series need it and both are
meant to apply on their own, so it is posted in each; whichever lands
first, the other should drop it.

When an anonymous folio is reclaimed, workingset_eviction() stores a
"shadow" (the eviction cookie) in the swap slot so that a later swap-in
can be recognised as a refault and, if the refault distance is short
enough, the page can be re-activated. This is how anon workingset/refault
detection has worked since commit aae466b0052e ("mm/swap: implement
workingset detection for anonymous LRU").

zswap writeback breaks this in two independent ways:

  - Over-count at writeback: the shrinker allocates a buffer folio in the
    swap cache, and the allocation path counts that folio as a refault.

  - Lost eviction cookie at reclaim: adding the buffer to the swap cache
    overwrites the slot's shadow, so the original cookie is lost; when the
    buffer folio is finally reclaimed a fresh, inaccurate cookie is minted
    in its place.

This series preserves the shadow within zswap itself, without any new
swap-table or swap-slot state. At writeback, instead of erasing the freed
zswap entry, the captured shadow is parked in the zswap tree in its place,
so it outlives the writeback buffer folio.

Finally, the refault evaluation is moved out of the swap-cache allocator
into the swap-in callers, so allocating the writeback buffer is no longer
miscounted as a refault.

Results
-------

Measured with a sysbench OLTP (MariaDB) workload in a memory cgroup sized
so the dataset and InnoDB buffer pool both overcommit it, with the zswap
shrinker on so entries are continuously written back to an NVMe swap
device (classic LRU; MGLRU off). Anon workingset counters over the
measured window, baseline vs this series, mean +/- stddev over 10 runs:

  workingset_refault_anon    383,242 +/- 59,345  ->  199,355 +/- 27,042   -48%
  workingset_activate_anon    54,890 +/- 11,742  ->   22,607 +/-  3,118   -59%
  workingset_restore_anon     16,212 +/-  4,590  ->    7,599 +/-  1,190   -53%

Writeback volume is comparable (zswpwb 183k +/- 16k -> 178k +/- 15k), so
the reduction is not from doing less work. Normalised per transaction the
reduction holds (-53%/-49%/-39%) while the swap work per transaction is
unchanged. A kernel build under the same pressure moves all three
counters in the same direction.

The run-to-run variance of these counters drops as well.

Throughput is unaffected: over the same 10 runs, transactions/s is
18.50 +/- 1.08 -> 18.65 +/- 0.94, i.e. +0.8% with a 95% confidence
interval of +/- 7.7%.

Changes in v3:
- David pointed out that the boolean added to workingset_refault() in v2
  makes the calling code hard to read. The boolean is now internal to
  mm/workingset.c and the callers use workingset_refault() or
  workingset_refault_lru_managed(), the same way remove_mapping() and
  remove_mapping_reclaim() do. The three existing callers are left
  untouched.
- zswap_writeback_entry() now restores the shadow into the slot on the
  error paths taken after the buffer was allocated: the allocation had
  already overwritten it and nothing was parked yet, so it was lost.
- Dropped the "if (!shadow) shadow = ZSWAP_WRITEBACK_NO_SHADOW" fallback,
  which can never be taken: the swap table already marks a swapped out
  slot with xa_mk_value(0), so swap_cache_get_shadow() never returns NULL
  for one.
- Rebased on mm-new.

Changes in v2:
- Sashiko pointed out that v1 evaluated the refault after folio_add_lru(),
  which picks the MGLRU generation before PG_workingset is set. Patch 1
  now moves the LRU insertion out of the swap cache allocator so the
  refault is evaluated before it, as it was originally.
- Sashiko also pointed out that a failed writeback redirties the buffer
  and leaves it in the swap cache with its shadow still parked, so a
  later zswap_store() on it would hand the parked value to
  zswap_entry_free(). zswap_store() now bails out for such a folio:
  writeback already decided that data belongs on disk, so it is written
  there instead of being compressed again, which also keeps the parked
  shadow intact until the folio leaves the swap cache.
- The buffer folio a swap-in consumes is already on the LRU, so the
  refault there cannot activate it by setting PG_active: that leaves the
  flag disagreeing with the list the folio is on, which shows up as an
  mm/memcontrol.c lru_size underflow when it is freed. Such a folio is now
  activated with folio_activate() instead. One still sitting in a per-CPU
  batch cannot be moved safely and just misses the activation; a counter
  on that path measured 0.0007% of the activations over a 10 run test.
  This is only needed because zswap writeback still puts its buffer on the
  LRU: once it stops doing so, workingset_refault_lru_managed() has no
  caller left and can go away.

[1] https://lore.kernel.org/linux-mm/20260818163221.589352-2-alex@ghiti.fr/

v1: https://lore.kernel.org/all/20260817144622.137133-1-alex@ghiti.fr/
v2: https://lore.kernel.org/linux-mm/20260821093606.2231216-1-alex@ghiti.fr/

Alexandre Ghiti (3):
  mm/swap: move LRU insertion out of the swap cache allocator
  mm/swap: refault on swap-in, not in the swap cache allocator
  mm/zswap: preserve the workingset shadow across writeback

 include/linux/zswap.h |  12 +++++
 mm/internal.h         |   1 +
 mm/memory.c           |   5 ++
 mm/shmem.c            |   5 ++
 mm/swap.h             |   6 +--
 mm/swap_state.c       |  50 +++++++++++++++----
 mm/vmscan.c           |   4 +-
 mm/workingset.c       |  60 ++++++++++++++++++-----
 mm/zswap.c            | 110 ++++++++++++++++++++++++++++++++++++++++--
 9 files changed, 224 insertions(+), 29 deletions(-)


base-commit: 1a46b1e97bde62afa7d925bb0dcd9f9748a1d7c3
-- 
2.53.0-Meta

[PATCH v3 1/3] mm/swap: move LRU insertion out of the swap cache allocator

[Alexandre Ghiti]

This is a preparatory patch.

__swap_cache_alloc_folio() adds the new folio to the LRU itself, which
leaves its callers no way to act on the folio before it becomes visible
to reclaim.  Two users need exactly that:

 - the next patch moves the refault evaluation out of the swap cache
   folio allocation, and it has to happen before folio_add_lru(): that
   consumes PG_active to file the folio on the inactive or the active
   list, and under MGLRU it also reads PG_workingset to pick the
   generation.  Setting either flag afterwards does not move the folio;

 - the upcoming zswap writeback dropbehind implementation needs the
   buffer folio to stay off the LRU entirely, as the per-CPU LRU batch
   would hold a reference on it and keep remove_mapping() from freeing
   it once writeback completes.

Defer the LRU insertion to the callers of __swap_cache_alloc_folio():
each of them adds the folio right after the allocation, so there is no
functional change intended.

Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
---
 mm/swap.h       |  6 +++---
 mm/swap_state.c | 20 ++++++++++++--------
 mm/zswap.c      |  5 +++--
 3 files changed, 18 insertions(+), 13 deletions(-)

diff --git a/mm/swap.h b/mm/swap.h
index 90a551a88df6..8679cb61268e 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -312,9 +312,9 @@ bool swap_cache_has_folio(swp_entry_t entry);
 struct folio *swap_cache_get_folio(swp_entry_t entry);
 void *swap_cache_get_shadow(swp_entry_t entry);
 void swap_cache_del_folio(struct folio *folio);
-struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
-				     unsigned long orders, struct vm_fault *vmf,
-				     struct mempolicy *mpol, pgoff_t ilx);
+struct folio *__swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
+				       unsigned long orders, struct vm_fault *vmf,
+				       struct mempolicy *mpol, pgoff_t ilx);
 /* Below helpers require the caller to lock and pass in the swap cluster. */
 void __swap_cache_add_folio(struct swap_cluster_info *ci,
 			    struct folio *folio, swp_entry_t entry);
diff --git a/mm/swap_state.c b/mm/swap_state.c
index b76eb3d876fd..bf8ff2d2dbf1 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -489,13 +489,11 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
 	node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
 	lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
 
-	/* Caller will initiate read into locked new_folio */
-	folio_add_lru(folio);
 	return folio;
 }
 
 /**
- * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
+ * __swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
  * @targ_entry: swap entry indicating the target slot
  * @gfp: memory allocation flags
  * @orders: allocation orders, must be non zero
@@ -507,13 +505,17 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
  * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
  * @targ_entry must have a non-zero swap count (swapped out).
  *
+ * The returned folio is locked and is NOT on the LRU. The caller must either
+ * add it to the LRU with folio_add_lru() so page reclaim can find it, or free
+ * it directly once done; a folio left off the LRU is unreclaimable and leaks.
+ *
  * Context: Caller must protect the swap device with reference count or locks.
  * Return: Returns the folio if allocation succeeded and folio is in the swap
  * cache. Returns error code if failed due to race, OOM or invalid arguments.
  */
-struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
-				     unsigned long orders, struct vm_fault *vmf,
-				     struct mempolicy *mpol, pgoff_t ilx)
+struct folio *__swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
+				       unsigned long orders, struct vm_fault *vmf,
+				       struct mempolicy *mpol, pgoff_t ilx)
 {
 	int order, err;
 	struct folio *ret;
@@ -649,12 +651,13 @@ static struct folio *swap_cache_read_folio(struct swap_io_ctx *ctx,
 		folio = swap_cache_get_folio(entry);
 		if (folio)
 			return folio;
-		folio = swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
+		folio = __swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
 	} while (PTR_ERR(folio) == -EEXIST);
 
 	if (IS_ERR_OR_NULL(folio))
 		return NULL;
 
+	folio_add_lru(folio);
 	swap_read_folio(ctx, folio);
 	if (readahead) {
 		folio_set_readahead(folio);
@@ -690,12 +693,13 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
 		folio = swap_cache_get_folio(entry);
 		if (folio)
 			return folio;
-		folio = swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
+		folio = __swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
 	} while (PTR_ERR(folio) == -EEXIST);
 
 	if (IS_ERR(folio))
 		return folio;
 
+	folio_add_lru(folio);
 	swap_read_folio(&ctx, folio);
 	swap_read_submit(&ctx);
 	return folio;
diff --git a/mm/zswap.c b/mm/zswap.c
index 37f34e406c8e..0d2efe21f18a 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1001,8 +1001,8 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 		return -EEXIST;
 
 	mpol = get_task_policy(current);
-	folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
-				       NO_INTERLEAVE_INDEX);
+	folio = __swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
+					 NO_INTERLEAVE_INDEX);
 	put_swap_device(si);
 
 	/*
@@ -1014,6 +1014,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 	 */
 	if (IS_ERR(folio))
 		return PTR_ERR(folio);
+	folio_add_lru(folio);
 
 	/*
 	 * folio is locked, and the swapcache is now secured against
-- 
2.53.0-Meta

[PATCH v3 2/3] mm/swap: refault on swap-in, not in the swap cache allocator

[Alexandre Ghiti]

The swap cache allocator evaluates the refault of every folio it
allocates.  zswap writeback also allocates through it: the shrinker puts
a buffer folio in the swap cache to write the compressed data out, and
that allocation is then counted as an anon refault (and, if the eviction
looks recent, as an activation) even though nothing faulted the page
back in.  On a workload that writes back continuously this inflates
workingset_refault_anon and workingset_activate_anon substantially.

Move the refault evaluation out of the allocator and into the two
swap-in callers, which read the slot's shadow before the allocation
overwrites it.  zswap writeback keeps allocating the buffer, but no
longer reports a refault for it.

The refault is evaluated before folio_add_lru(), as it was before this
patch, so workingset_refault() still sets PG_workingset/PG_active while
the folio is off the LRU: folio_add_lru() consumes both when it picks
the LRU list, and under MGLRU when it picks the generation.

Fixes: aae466b0052e ("mm/swap: implement workingset detection for anonymous LRU")
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
---
 mm/swap_state.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/mm/swap_state.c b/mm/swap_state.c
index bf8ff2d2dbf1..8046fea015c9 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -483,8 +483,6 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
 
 	/* memsw uncharges swap when folio is added to swap cache */
 	memcg1_swapin(folio);
-	if (shadow)
-		workingset_refault(folio, shadow);
 
 	node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
 	lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
@@ -646,17 +644,26 @@ static struct folio *swap_cache_read_folio(struct swap_io_ctx *ctx,
 		pgoff_t ilx, bool readahead)
 {
 	struct folio *folio;
+	void *shadow = NULL;
 
 	do {
 		folio = swap_cache_get_folio(entry);
 		if (folio)
 			return folio;
+		/*
+		 * Capture the slot's shadow before the allocation overwrites it,
+		 * so a fresh swap-in can be evaluated as a refault below.
+		 */
+		shadow = swap_cache_get_shadow(entry);
 		folio = __swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
 	} while (PTR_ERR(folio) == -EEXIST);
 
 	if (IS_ERR_OR_NULL(folio))
 		return NULL;
 
+	if (shadow)
+		workingset_refault(folio, shadow);
+
 	folio_add_lru(folio);
 	swap_read_folio(ctx, folio);
 	if (readahead) {
@@ -688,17 +695,26 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
 {
 	struct swap_io_ctx ctx = {};
 	struct folio *folio;
+	void *shadow = NULL;
 
 	do {
 		folio = swap_cache_get_folio(entry);
 		if (folio)
 			return folio;
+		/*
+		 * Capture the slot's shadow before the allocation overwrites it,
+		 * so a fresh swap-in can be evaluated as a refault below.
+		 */
+		shadow = swap_cache_get_shadow(entry);
 		folio = __swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
 	} while (PTR_ERR(folio) == -EEXIST);
 
 	if (IS_ERR(folio))
 		return folio;
 
+	if (shadow)
+		workingset_refault(folio, shadow);
+
 	folio_add_lru(folio);
 	swap_read_folio(&ctx, folio);
 	swap_read_submit(&ctx);
-- 
2.53.0-Meta

[PATCH v3 3/3] mm/zswap: preserve the workingset shadow across writeback

[Alexandre Ghiti]

When zswap writes an entry back, allocating the buffer folio in the swap
cache overwrites the swap slot's workingset shadow. It is lost and later
re-minted inaccurately, which corrupts anon refault/workingset accounting
for the written-back data.

Preserve it within zswap itself: at writeback, park the shadow in the
zswap tree in place of the freed entry, so it outlives the writeback
buffer folio. The buffer then meets one of three fates, and the parked
shadow is handled at whichever happens first:

 - swap-in: do_swap_page()/shmem_swapin_folio() find the buffer in the swap
   cache, take the shadow and feed it to workingset_refault() to account
   the refault of the now-resident folio, then drop it.
 - reclaim: __remove_mapping() skips minting a fresh shadow for the buffer
   (minting would double-count the eviction) and
   __swap_cache_do_del_folio() restores the parked shadow into the slot, so
   a later disk swap-in still refaults against the original eviction.
 - slot free (e.g. process exit): the shadow is not needed and is cleared.

The last two both go through __swap_cache_do_del_folio(), the single point
every swap-cache removal passes through, which is why the shadow is read
and cleared there.

Fixes: aae466b0052e ("mm/swap: implement workingset detection for anonymous LRU")
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
---
 include/linux/zswap.h |  12 +++++
 mm/internal.h         |   1 +
 mm/memory.c           |   5 ++
 mm/shmem.c            |   5 ++
 mm/swap_state.c       |  10 ++++
 mm/vmscan.c           |   4 +-
 mm/workingset.c       |  60 +++++++++++++++++++-----
 mm/zswap.c            | 105 +++++++++++++++++++++++++++++++++++++++++-
 8 files changed, 188 insertions(+), 14 deletions(-)

diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index 30c193a1207e..dfb7153236f5 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -35,6 +35,8 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);
 void zswap_folio_swapin(struct folio *folio);
 bool zswap_is_enabled(void);
 bool zswap_never_enabled(void);
+bool zswap_folio_is_writeback_buffer(struct folio *folio);
+void *zswap_lookup_and_clear_shadows(struct folio *folio);
 #else
 
 struct zswap_lruvec_state {};
@@ -69,6 +71,16 @@ static inline bool zswap_never_enabled(void)
 	return true;
 }
 
+static inline bool zswap_folio_is_writeback_buffer(struct folio *folio)
+{
+	return false;
+}
+
+static inline void *zswap_lookup_and_clear_shadows(struct folio *folio)
+{
+	return NULL;
+}
+
 #endif
 
 #endif /* _LINUX_ZSWAP_H */
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c9..c191887c1fed 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -38,6 +38,7 @@ void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages);
 void *workingset_eviction(struct folio *folio,
 			  struct mem_cgroup *target_memcg);
 void workingset_refault(struct folio *folio, void *shadow);
+void workingset_refault_lru_managed(struct folio *folio, void *shadow);
 void workingset_activation(struct folio *folio);
 
 /* mm/folio.c */
diff --git a/mm/memory.c b/mm/memory.c
index c6a9242d1b5b..771197b3fcc5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4894,6 +4894,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 	unsigned long page_idx;
 	unsigned long address;
 	pte_t *ptep;
+	void *shadow;
 
 	if (!pte_unmap_same(vmf))
 		goto out;
@@ -5018,6 +5019,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		goto out_page;
 	}
 
+	shadow = zswap_lookup_and_clear_shadows(folio);
+	if (shadow)
+		workingset_refault_lru_managed(folio, shadow);
+
 	/*
 	 * KSM sometimes has to copy on read faults, for example, if
 	 * folio->index of non-ksm folios would be nonlinear inside the
diff --git a/mm/shmem.c b/mm/shmem.c
index 599665a3d6e7..1ef6cc76605e 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2266,6 +2266,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
 	struct folio *folio = NULL;
 	int error, nr_pages, order;
 	pgoff_t offset;
+	void *shadow;
 
 	VM_BUG_ON(!*foliop || !xa_is_value(*foliop));
 	index_entry = radix_to_swp_entry(*foliop);
@@ -2377,6 +2378,10 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
 	 */
 	arch_swap_restore(folio_swap(swap, folio), folio);
 
+	shadow = zswap_lookup_and_clear_shadows(folio);
+	if (shadow)
+		workingset_refault_lru_managed(folio, shadow);
+
 	if (shmem_should_replace_folio(folio, gfp)) {
 		error = shmem_replace_folio(&folio, gfp, info, index, vma);
 		if (error)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 8046fea015c9..df9e0452f19b 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -24,6 +24,7 @@
 #include <linux/shmem_fs.h>
 #include <linux/sysctl.h>
 #include <linux/swap_ops.h>
+#include <linux/zswap.h>
 #include "internal.h"
 #include "swap_table.h"
 #include "swap.h"
@@ -263,12 +264,21 @@ static void __swap_cache_do_del_folio(struct swap_cluster_info *ci,
 	unsigned int ci_start, ci_off, ci_end;
 	bool folio_swapped = false, need_free = false;
 	unsigned long nr_pages = folio_nr_pages(folio);
+	void *shadow_parked;
 
 	VM_WARN_ON_ONCE(__swap_entry_to_cluster(entry) != ci);
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
 	VM_WARN_ON_ONCE_FOLIO(folio_test_writeback(folio), folio);
 
+	/*
+	 * A zswap writeback buffer parked the slot's original shadow in the
+	 * zswap tree: restore it into the slot for later swap-in.
+	 */
+	shadow_parked = zswap_lookup_and_clear_shadows(folio);
+	if (shadow_parked)
+		shadow = shadow_parked;
+
 	si = __swap_entry_to_info(entry);
 	ci_start = swp_cluster_offset(entry);
 	ci_end = ci_start + nr_pages;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 73a81b4a3e16..9905753b7a10 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -48,6 +48,7 @@
 #include <linux/prefetch.h>
 #include <linux/printk.h>
 #include <linux/dax.h>
+#include <linux/zswap.h>
 #include <linux/psi.h>
 #include <linux/pagewalk.h>
 #include <linux/shmem_fs.h>
@@ -733,7 +734,8 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,
 	if (folio_test_swapcache(folio)) {
 		swp_entry_t swap = folio->swap;
 
-		if (reclaimed && !mapping_exiting(mapping))
+		if (reclaimed && !mapping_exiting(mapping) &&
+		    !zswap_folio_is_writeback_buffer(folio))
 			shadow = workingset_eviction(folio, target_memcg);
 		__memcg1_swapout(folio, ci);
 		__swap_cache_del_folio(ci, folio, swap, shadow);
diff --git a/mm/workingset.c b/mm/workingset.c
index 7ac2b88c80ae..e203d631f61b 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -536,16 +536,8 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,
 	return refault_distance <= workingset_size;
 }
 
-/**
- * workingset_refault - Evaluate the refault of a previously evicted folio.
- * @folio: The freshly allocated replacement folio.
- * @shadow: Shadow entry of the evicted folio.
- *
- * Calculates and evaluates the refault distance of the previously
- * evicted folio in the context of the node and the memcg whose memory
- * pressure caused the eviction.
- */
-void workingset_refault(struct folio *folio, void *shadow)
+static void __workingset_refault(struct folio *folio, void *shadow,
+				 bool lru_managed)
 {
 	bool file = folio_is_file_lru(folio);
 	struct mem_cgroup *memcg;
@@ -577,7 +569,16 @@ void workingset_refault(struct folio *folio, void *shadow)
 	if (!workingset_test_recent(shadow, file, &workingset, true))
 		goto out;
 
-	folio_set_active(folio);
+	/*
+	 * An LRU-managed folio may sit in a per-CPU batch, which cannot be
+	 * determined here: setting the flag would race the drain and leave it
+	 * disagreeing with the list. folio_activate() is safe, but misses the
+	 * activation for such a folio.
+	 */
+	if (lru_managed)
+		folio_activate(folio);
+	else
+		folio_set_active(folio);
 	workingset_age_nonresident(lruvec, nr);
 	mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + file, nr);
 
@@ -590,6 +591,43 @@ void workingset_refault(struct folio *folio, void *shadow)
 	mem_cgroup_put(memcg);
 }
 
+/**
+ * workingset_refault - Evaluate the refault of a previously evicted folio.
+ * @folio: The freshly allocated replacement folio.
+ * @shadow: Shadow entry of the evicted folio.
+ *
+ * Calculates and evaluates the refault distance of the previously
+ * evicted folio in the context of the node and the memcg whose memory
+ * pressure caused the eviction.
+ *
+ * Context: @folio must be locked and not on the LRU yet, the caller adds
+ * it with folio_add_lru() afterwards.
+ */
+void workingset_refault(struct folio *folio, void *shadow)
+{
+	__workingset_refault(folio, shadow, false);
+}
+
+/**
+ * workingset_refault_lru_managed - Evaluate the refault of a previously
+ * evicted folio that is already on the LRU.
+ * @folio: The folio the eviction is refaulted into.
+ * @shadow: Shadow entry of the evicted folio.
+ *
+ * Like workingset_refault(), but for a folio the caller has already added
+ * to the LRU, which has to be activated by moving it between the lists
+ * rather than by setting PG_active.
+ *
+ * The activation is best effort: a folio still sitting in a per-CPU LRU
+ * batch cannot be moved yet and is left inactive.
+ *
+ * Context: @folio must be locked and already added to the LRU.
+ */
+void workingset_refault_lru_managed(struct folio *folio, void *shadow)
+{
+	__workingset_refault(folio, shadow, true);
+}
+
 /**
  * workingset_activation - note a page activation
  * @folio: Folio that is being activated.
diff --git a/mm/zswap.c b/mm/zswap.c
index 0d2efe21f18a..c1a78edfa0b8 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -972,6 +972,79 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
 /*********************************
 * writeback code
 **********************************/
+
+#define ZSWAP_WRITEBACK_NO_SHADOW xa_mk_value(0)
+
+/**
+ * zswap_folio_is_writeback_buffer - is @folio a zswap writeback buffer?
+ * @folio: the folio being examined (typically a swap cache folio under reclaim)
+ *
+ * A folio is a zswap writeback buffer when every one of its swap offsets holds
+ * a parked writeback shadow (a real shadow or the ZSWAP_WRITEBACK_NO_SHADOW
+ * sentinel) in the zswap tree rather than a live zswap entry.
+ *
+ * Return: true if @folio is a writeback buffer, in which case the reclaim path
+ * must not mint a fresh workingset shadow for it.
+ */
+bool zswap_folio_is_writeback_buffer(struct folio *folio)
+{
+	swp_entry_t swp = folio->swap;
+	unsigned long nr_pages = folio_nr_pages(folio);
+	pgoff_t offset = swp_offset(swp);
+	unsigned long i;
+
+	if (zswap_never_enabled())
+		return false;
+
+	for (i = 0; i < nr_pages; i++) {
+		swp_entry_t e = swp_entry(swp_type(swp), offset + i);
+
+		if (!xa_is_value(xa_load(swap_zswap_tree(e), offset + i)))
+			return false;
+	}
+
+	return true;
+}
+
+/**
+ * zswap_lookup_and_clear_shadows - retrieve and clear @folio's parked shadow(s)
+ * @folio: the writeback buffer folio (or the swapin folio that consumed it)
+ *
+ * Remove any parked writeback shadows for @folio's swap offset(s) from the
+ * zswap tree.
+ *
+ * Return: the preserved workingset shadow, or NULL if the slot(s) had no shadow
+ * (sentinel only) or nothing parked. The caller either restores the returned
+ * shadow into the swap slot (buffer dropped) or feeds it to
+ * workingset_refault_lru_managed() (buffer consumed by a swapin); clearing
+ * here ensures the two paths never double-count.
+ */
+void *zswap_lookup_and_clear_shadows(struct folio *folio)
+{
+	swp_entry_t swp = folio->swap;
+	unsigned long nr_pages = folio_nr_pages(folio);
+	pgoff_t offset = swp_offset(swp);
+	void *shadow = NULL;
+	unsigned long i;
+
+	if (zswap_never_enabled())
+		return NULL;
+
+	for (i = 0; i < nr_pages; i++) {
+		swp_entry_t e = swp_entry(swp_type(swp), offset + i);
+		struct xarray *tree = swap_zswap_tree(e);
+		void *parked = xa_load(tree, offset + i);
+
+		if (!xa_is_value(parked))
+			continue;
+		xa_erase(tree, offset + i);
+		if (parked != ZSWAP_WRITEBACK_NO_SHADOW)
+			shadow = parked;
+	}
+
+	return shadow;
+}
+
 /*
  * Attempts to free an entry by adding a folio to the swap cache,
  * decompressing the entry data into the folio, and issuing a
@@ -987,12 +1060,14 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
 static int zswap_writeback_entry(struct zswap_entry *entry,
 				 swp_entry_t swpentry)
 {
+	struct swap_cluster_info *ci;
 	struct xarray *tree;
 	pgoff_t offset = swp_offset(swpentry);
 	struct folio *folio;
 	struct mempolicy *mpol;
 	struct swap_info_struct *si;
 	struct swap_io_ctx ctx = {};
+	void *shadow;
 	int ret = 0;
 
 	/* try to allocate swap cache folio */
@@ -1000,6 +1075,8 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 	if (!si)
 		return -EEXIST;
 
+	shadow = swap_cache_get_shadow(swpentry);
+
 	mpol = get_task_policy(current);
 	folio = __swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
 					 NO_INTERLEAVE_INDEX);
@@ -1036,7 +1113,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 		goto out;
 	}
 
-	xa_erase(tree, offset);
+	/*
+	 * A slot with no shadow already reads back as ZSWAP_WRITEBACK_NO_SHADOW:
+	 * the swap table marks a swapped out slot with xa_mk_value(0) and holds
+	 * the shadow, when there is one, in that same value.
+	 */
+	xa_store(tree, offset, shadow, GFP_KERNEL);
 
 	count_vm_event(ZSWPWB);
 	if (entry->objcg)
@@ -1056,7 +1138,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
 
 out:
 	if (ret) {
-		swap_cache_del_folio(folio);
+		/*
+		 * The allocation above overwrote the slot's shadow and nothing
+		 * has been parked for it yet, so restore it as the folio leaves
+		 * the swap cache, or the eviction it records is lost.
+		 */
+		ci = swap_cluster_lock(__swap_entry_to_info(swpentry), offset);
+		__swap_cache_del_folio(ci, folio, swpentry, shadow);
+		swap_cluster_unlock(ci);
+		folio_ref_sub(folio, folio_nr_pages(folio));
 		folio_unlock(folio);
 	}
 	folio_put(folio);
@@ -1499,6 +1589,17 @@ bool zswap_store(struct folio *folio)
 	VM_WARN_ON_ONCE(!folio_test_locked(folio));
 	VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
 
+	/*
+	 * A writeback buffer whose IO failed is redirtied and left in the
+	 * swap cache, so reclaim writes it out again. Writeback already
+	 * decided this data belongs on disk, so send it there instead of
+	 * compressing it back into zswap. This also leaves the shadow
+	 * parked for the buffer in place, so it is still restored into
+	 * the slot once the folio leaves the swap cache.
+	 */
+	if (zswap_folio_is_writeback_buffer(folio))
+		return false;
+
 	if (!zswap_enabled)
 		goto check_old;
 
-- 
2.53.0-Meta

Re: [PATCH v3 1/3] mm/swap: move LRU insertion out of the swap cache allocator

[Usama Arif]

On Tue, 25 Aug 2026 19:24:51 +0200 Alexandre Ghiti <alex@ghiti.fr> wrote:

> This is a preparatory patch.
> 
> __swap_cache_alloc_folio() adds the new folio to the LRU itself, which
> leaves its callers no way to act on the folio before it becomes visible
> to reclaim.  Two users need exactly that:
> 
>  - the next patch moves the refault evaluation out of the swap cache
>    folio allocation, and it has to happen before folio_add_lru(): that
>    consumes PG_active to file the folio on the inactive or the active
>    list, and under MGLRU it also reads PG_workingset to pick the
>    generation.  Setting either flag afterwards does not move the folio;
> 
>  - the upcoming zswap writeback dropbehind implementation needs the
>    buffer folio to stay off the LRU entirely, as the per-CPU LRU batch
>    would hold a reference on it and keep remove_mapping() from freeing
>    it once writeback completes.
> 

I think the kernel convention is to not say "next patch" and "upcoming.."

Especially as this patch is in 2 series, so you don't know where it will land.

> Defer the LRU insertion to the callers of __swap_cache_alloc_folio():
> each of them adds the folio right after the allocation, so there is no
> functional change intended.
> 
> Suggested-by: Kairui Song <kasong@tencent.com>
> Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>


The code change itself looks good to me, so once the commit message is fixed:
Acked-by: Usama Arif <usama.arif@linux.dev>

Re: [PATCH v3 2/3] mm/swap: refault on swap-in, not in the swap cache allocator

[Usama Arif]

On Tue, 25 Aug 2026 19:24:52 +0200 Alexandre Ghiti <alex@ghiti.fr> wrote:

> The swap cache allocator evaluates the refault of every folio it
> allocates.  zswap writeback also allocates through it: the shrinker puts
> a buffer folio in the swap cache to write the compressed data out, and
> that allocation is then counted as an anon refault (and, if the eviction
> looks recent, as an activation) even though nothing faulted the page
> back in.  On a workload that writes back continuously this inflates
> workingset_refault_anon and workingset_activate_anon substantially.
> 
> Move the refault evaluation out of the allocator and into the two
> swap-in callers, which read the slot's shadow before the allocation
> overwrites it.  zswap writeback keeps allocating the buffer, but no
> longer reports a refault for it.
> 
> The refault is evaluated before folio_add_lru(), as it was before this
> patch, so workingset_refault() still sets PG_workingset/PG_active while
> the folio is off the LRU: folio_add_lru() consumes both when it picks
> the LRU list, and under MGLRU when it picks the generation.
> 
> Fixes: aae466b0052e ("mm/swap: implement workingset detection for anonymous LRU")
> Signed-off-by: Nhat Pham <nphamcs@gmail.com>
> Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
> ---
>  mm/swap_state.c | 20 ++++++++++++++++++--
>  1 file changed, 18 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/swap_state.c b/mm/swap_state.c
> index bf8ff2d2dbf1..8046fea015c9 100644
> --- a/mm/swap_state.c
> +++ b/mm/swap_state.c
> @@ -483,8 +483,6 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
>  
>  	/* memsw uncharges swap when folio is added to swap cache */
>  	memcg1_swapin(folio);
> -	if (shadow)
> -		workingset_refault(folio, shadow);
>  
>  	node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
>  	lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
> @@ -646,17 +644,26 @@ static struct folio *swap_cache_read_folio(struct swap_io_ctx *ctx,
>  		pgoff_t ilx, bool readahead)
>  {
>  	struct folio *folio;
> +	void *shadow = NULL;
>  
>  	do {
>  		folio = swap_cache_get_folio(entry);
>  		if (folio)
>  			return folio;
> +		/*
> +		 * Capture the slot's shadow before the allocation overwrites it,
> +		 * so a fresh swap-in can be evaluated as a refault below.
> +		 */
> +		shadow = swap_cache_get_shadow(entry);
>  		folio = __swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);

Can this refault against a shadow from a different cache generation?
Another swap-in can replace this value, reclaim its folio, and install
a newer shadow while this caller is allocating.  This caller can then
win the locked insertion but still use the older snapshot.

What I think should be done here is have an optional shadow output to
__swap_cache_alloc_folio() and return the value captured by the successful
__swap_cache_add_check() under ci->lock and se that output in the swap-in
path?

>  	} while (PTR_ERR(folio) == -EEXIST);
>  
>  	if (IS_ERR_OR_NULL(folio))
>  		return NULL;
>  
> +	if (shadow)
> +		workingset_refault(folio, shadow);
> +
>  	folio_add_lru(folio);
>  	swap_read_folio(ctx, folio);
>  	if (readahead) {
> @@ -688,17 +695,26 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
>  {
>  	struct swap_io_ctx ctx = {};
>  	struct folio *folio;
> +	void *shadow = NULL;
>  
>  	do {
>  		folio = swap_cache_get_folio(entry);
>  		if (folio)
>  			return folio;
> +		/*
> +		 * Capture the slot's shadow before the allocation overwrites it,
> +		 * so a fresh swap-in can be evaluated as a refault below.
> +		 */
> +		shadow = swap_cache_get_shadow(entry);
>  		folio = __swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
>  	} while (PTR_ERR(folio) == -EEXIST);
>  
>  	if (IS_ERR(folio))
>  		return folio;
>  
> +	if (shadow)
> +		workingset_refault(folio, shadow);
> +
>  	folio_add_lru(folio);
>  	swap_read_folio(&ctx, folio);
>  	swap_read_submit(&ctx);
> -- 
> 2.53.0-Meta
> 
> 

Re: [PATCH v3 2/3] mm/swap: refault on swap-in, not in the swap cache allocator

[Alexandre Ghiti]

Hi Usama,

On Wed, Aug 26, 2026 at 3:31 PM Usama Arif <usama.arif@linux.dev> wrote:
>
> >
> On Tue, 25 Aug 2026 19:24:52 +0200 Alexandre Ghiti <alex@ghiti.fr> wrote:
>
> > The swap cache allocator evaluates the refault of every folio it
> > allocates.  zswap writeback also allocates through it: the shrinker puts
> > a buffer folio in the swap cache to write the compressed data out, and
> > that allocation is then counted as an anon refault (and, if the eviction
> > looks recent, as an activation) even though nothing faulted the page
> > back in.  On a workload that writes back continuously this inflates
> > workingset_refault_anon and workingset_activate_anon substantially.
> >
> > Move the refault evaluation out of the allocator and into the two
> > swap-in callers, which read the slot's shadow before the allocation
> > overwrites it.  zswap writeback keeps allocating the buffer, but no
> > longer reports a refault for it.
> >
> > The refault is evaluated before folio_add_lru(), as it was before this
> > patch, so workingset_refault() still sets PG_workingset/PG_active while
> > the folio is off the LRU: folio_add_lru() consumes both when it picks
> > the LRU list, and under MGLRU when it picks the generation.
> >
> > Fixes: aae466b0052e ("mm/swap: implement workingset detection for anonymous LRU")
> > Signed-off-by: Nhat Pham <nphamcs@gmail.com>
> > Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
> > ---
> >  mm/swap_state.c | 20 ++++++++++++++++++--
> >  1 file changed, 18 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/swap_state.c b/mm/swap_state.c
> > index bf8ff2d2dbf1..8046fea015c9 100644
> > --- a/mm/swap_state.c
> > +++ b/mm/swap_state.c
> > @@ -483,8 +483,6 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
> >
> >       /* memsw uncharges swap when folio is added to swap cache */
> >       memcg1_swapin(folio);
> > -     if (shadow)
> > -             workingset_refault(folio, shadow);
> >
> >       node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
> >       lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
> > @@ -646,17 +644,26 @@ static struct folio *swap_cache_read_folio(struct swap_io_ctx *ctx,
> >               pgoff_t ilx, bool readahead)
> >  {
> >       struct folio *folio;
> > +     void *shadow = NULL;
> >
> >       do {
> >               folio = swap_cache_get_folio(entry);
> >               if (folio)
> >                       return folio;
> > +             /*
> > +              * Capture the slot's shadow before the allocation overwrites it,
> > +              * so a fresh swap-in can be evaluated as a refault below.
> > +              */
> > +             shadow = swap_cache_get_shadow(entry);
> >               folio = __swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
>
> Can this refault against a shadow from a different cache generation?
> Another swap-in can replace this value, reclaim its folio, and install
> a newer shadow while this caller is allocating.  This caller can then
> win the locked insertion but still use the older snapshot.

I agree, you're right, I missed that. Thanks.

>
> What I think should be done here is have an optional shadow output to
> __swap_cache_alloc_folio() and return the value captured by the successful
> __swap_cache_add_check() under ci->lock and se that output in the swap-in
> path?

That seems right, I'll give it a try and report back.

Thanks,

Alex


>
> >       } while (PTR_ERR(folio) == -EEXIST);
> >
> >       if (IS_ERR_OR_NULL(folio))
> >               return NULL;
> >
> > +     if (shadow)
> > +             workingset_refault(folio, shadow);
> > +
> >       folio_add_lru(folio);
> >       swap_read_folio(ctx, folio);
> >       if (readahead) {
> > @@ -688,17 +695,26 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
> >  {
> >       struct swap_io_ctx ctx = {};
> >       struct folio *folio;
> > +     void *shadow = NULL;
> >
> >       do {
> >               folio = swap_cache_get_folio(entry);
> >               if (folio)
> >                       return folio;
> > +             /*
> > +              * Capture the slot's shadow before the allocation overwrites it,
> > +              * so a fresh swap-in can be evaluated as a refault below.
> > +              */
> > +             shadow = swap_cache_get_shadow(entry);
> >               folio = __swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
> >       } while (PTR_ERR(folio) == -EEXIST);
> >
> >       if (IS_ERR(folio))
> >               return folio;
> >
> > +     if (shadow)
> > +             workingset_refault(folio, shadow);
> > +
> >       folio_add_lru(folio);
> >       swap_read_folio(&ctx, folio);
> >       swap_read_submit(&ctx);
> > --
> > 2.53.0-Meta
> >
> >
>

Re: [PATCH v3 1/3] mm/swap: move LRU insertion out of the swap cache allocator

[Alexandre Ghiti]

Hi,

On Wed, Aug 26, 2026 at 3:22 PM Usama Arif <usama.arif@linux.dev> wrote:
>
> >
> On Tue, 25 Aug 2026 19:24:51 +0200 Alexandre Ghiti <alex@ghiti.fr> wrote:
>
> > This is a preparatory patch.
> >
> > __swap_cache_alloc_folio() adds the new folio to the LRU itself, which
> > leaves its callers no way to act on the folio before it becomes visible
> > to reclaim.  Two users need exactly that:
> >
> >  - the next patch moves the refault evaluation out of the swap cache
> >    folio allocation, and it has to happen before folio_add_lru(): that
> >    consumes PG_active to file the folio on the inactive or the active
> >    list, and under MGLRU it also reads PG_workingset to pick the
> >    generation.  Setting either flag afterwards does not move the folio;
> >
> >  - the upcoming zswap writeback dropbehind implementation needs the
> >    buffer folio to stay off the LRU entirely, as the per-CPU LRU batch
> >    would hold a reference on it and keep remove_mapping() from freeing
> >    it once writeback completes.
> >
>
> I think the kernel convention is to not say "next patch" and "upcoming.."
>
> Especially as this patch is in 2 series, so you don't know where it will land.

I have to admit that I don't know how to deal with this shared
patch...I'll drop the references as you suggest and we'll see!

>
> > Defer the LRU insertion to the callers of __swap_cache_alloc_folio():
> > each of them adds the folio right after the allocation, so there is no
> > functional change intended.
> >
> > Suggested-by: Kairui Song <kasong@tencent.com>
> > Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
>
>
> The code change itself looks good to me, so once the commit message is fixed:
> Acked-by: Usama Arif <usama.arif@linux.dev>
>

Thanks for that.

Alex