From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2F3B4746C8 for ; Fri, 11 Sep 2026 09:20:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.70.183.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789118430; cv=none; b=c+GowogGyKVas+syGWyX9pChwldXwkSTy7UrLAMcudk6SHz/IW9/nCL47BXHbnVqjqMVDPycHaq3jmoHt3Kh1Lvr05uhKaXSFAWvmARaN75/y/gPE59CVX7ziO4ZUDY3txTyrs25+3CJhy++YcXOu7UD6MmH/8JhullNoHexTwQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789118430; c=relaxed/simple; bh=Ehj5xWPYbfkJCWn5y1oYXNkCMkBh10aGorpyYyX9W8A=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=tnXXucwXJEAvKida7BjXlY5477lRa00rLk1WNj9YQBQExOEnkZZKFEp/gnEKr8NXnpMGS21NIAR8janoHUaJYYAhCBHLEnxRyP74z66ScPe2Z5tgPXvqh3+0oVS+An4NTz0WocNr/zOLKDqclFp6oAN0gIfaGFQMqvUPEw3MjFM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ghiti.fr; spf=pass smtp.mailfrom=ghiti.fr; arc=none smtp.client-ip=217.70.183.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ghiti.fr Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ghiti.fr Received: by mail.gandi.net (Postfix) with ESMTPSA id C948D3E972; Fri, 11 Sep 2026 09:20:13 +0000 (UTC) From: Alexandre Ghiti To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Hugh Dickins , Baolin Wang , Chris Li , Kairui Song , Kemeng Shi , Baoquan He , Barry Song , Youngjun Park , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Joonsoo Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Alexandre Ghiti Subject: [PATCH v4 0/3] mm: fix workingset refaults in the zswap writeback path Date: Fri, 11 Sep 2026 11:20:04 +0200 Message-ID: <20260911092012.92399-1-alex@ghiti.fr> X-Mailer: git-send-email 2.55.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-GND-Sasl: alex@ghiti.fr X-GND-Cause: dmFkZTGzWbzMA5QZanrT8MgC66CDhbKxneLy9Ay/yxMlFVji2OLfhXWQPDvA1tdKqdI4htUQ8rGaY98avqsZNnswh6aMbdwtiezvc7z1tC/Owq0KcV4QllV1SC7O7vY4OsviYvKSYizj54HeiO5NtcHBPpIsNQ0cLcmY0VU+o2O9iLuLxN7Y58cOu9EHyl2cmQQLPtCZYwGCRKbz/GfbYlc6SmLlTs6HausImgU1yP0ti3L7sYEczYM5t5PzUTriKDLL3/Z1DbMXKeHgViDpSRIzpVfSFaFCNhfltK/w6Mqub6XWyQi7gjVWslDKhbP14+qYeF75e07YklW7BqLi4wijPvvvQ723VilEeyZd7lSAC1kK7tTvIXyAMQSpfqkIMBNwjlYc3WYIsgcdMqQOnYRWoqsmjBVxxDQeKRx3CpID/k0CjMNF4JNxGWS/rq//O2xLW3X5zrhPArR822rbcvyjhlCYjv+VqRXWZ8jcO3pwiAi3IiYre4qWQzkVwJ/hyasSRSWPlMgCkG7ViBCiMUB4sNXQkgw8Cbw1ZZg0+sBDAMa8Fu1Cx+fRQ/3ft4K+XjCIUj+AbuYv+z7v6avh8T+2h3k4ls7WoYMLFIQnJeHljZT+jUcQm98qltouk18n50Cp4vJU0ts3FwUH8YjQRnWq0ldVApoFk9Mf3kwZg7He33ORXA X-GND-State: clean X-GND-Score: -100 Note: patch 1 also appears as patch 1 of the zswap dropbehind series [1]. It is the same change. Both series need it and both are meant to apply on their own, so it is posted in each; whichever lands first, the other should drop it. When an anonymous folio is reclaimed, workingset_eviction() stores a "shadow" (the eviction cookie) in the swap slot so that a later swap-in can be recognised as a refault and, if the refault distance is short enough, the page can be re-activated. This is how anon workingset/refault detection has worked since commit aae466b0052e ("mm/swap: implement workingset detection for anonymous LRU"). zswap writeback breaks this in two independent ways: - Over-count at writeback: the shrinker allocates a buffer folio in the swap cache, and the allocation path counts that folio as a refault. - Lost eviction cookie at reclaim: adding the buffer to the swap cache overwrites the slot's shadow, so the original cookie is lost; when the buffer folio is finally reclaimed a fresh, inaccurate cookie is minted in its place. This series preserves the shadow within zswap itself, without any new swap-table or swap-slot state. At writeback, instead of erasing the freed zswap entry, the captured shadow is parked in the zswap tree in its place, so it outlives the writeback buffer folio. Finally, the refault evaluation is moved out of the swap-cache allocator into the swap-in callers, so allocating the writeback buffer is no longer miscounted as a refault. Results ------- Measured with a sysbench OLTP (MariaDB) workload in a memory cgroup sized so the dataset and InnoDB buffer pool both overcommit it, with the zswap shrinker on so entries are continuously written back to an NVMe swap device (classic LRU; MGLRU off). Anon workingset counters over the measured window, baseline vs this series, mean +/- stddev over 10 runs: workingset_refault_anon 383,242 +/- 59,345 -> 199,355 +/- 27,042 -48% workingset_activate_anon 54,890 +/- 11,742 -> 22,607 +/- 3,118 -59% workingset_restore_anon 16,212 +/- 4,590 -> 7,599 +/- 1,190 -53% Writeback volume is comparable (zswpwb 183k +/- 16k -> 178k +/- 15k), so the reduction is not from doing less work. Normalised per transaction the reduction holds (-53%/-49%/-39%) while the swap work per transaction is unchanged. A kernel build under the same pressure moves all three counters in the same direction. The run-to-run variance of these counters drops as well. Throughput is unaffected: over the same 10 runs, transactions/s is 18.50 +/- 1.08 -> 18.65 +/- 0.94, i.e. +0.8% with a 95% confidence interval of +/- 7.7%. Changes in v4: - Usama pointed out that patch 1's changelog referred to "the next patch" and to the "upcoming" dropbehind series, which does not work for a patch posted in two series. Both references are gone. - Kunwu Chan pointed out that mm/swapfile.c still refers to the function by its old name in a comment; patch 1 renames it there too. - Usama pointed out that reading the slot's shadow in the swap-in path before allocating races: the allocation can sleep, so another swap-in can install a folio, have it reclaimed and leave a newer shadow behind, and this caller would then win the insertion but refault against the stale snapshot. __swap_cache_alloc_folio() now hands the shadow back from __swap_cache_add_check(), which captures it under ci->lock at the point the displacing insertion succeeds. zswap writeback took the same racy snapshot and now uses the same output. Changes in v3: - David pointed out that the boolean added to workingset_refault() in v2 makes the calling code hard to read. The boolean is now internal to mm/workingset.c and the callers use workingset_refault() or workingset_refault_lru_managed(), the same way remove_mapping() and remove_mapping_reclaim() do. The three existing callers are left untouched. - zswap_writeback_entry() now restores the shadow into the slot on the error paths taken after the buffer was allocated: the allocation had already overwritten it and nothing was parked yet, so it was lost. - Dropped the "if (!shadow) shadow = ZSWAP_WRITEBACK_NO_SHADOW" fallback, which can never be taken: the swap table already marks a swapped out slot with xa_mk_value(0), so swap_cache_get_shadow() never returns NULL for one. - Rebased on mm-new. Changes in v2: - Sashiko pointed out that v1 evaluated the refault after folio_add_lru(), which picks the MGLRU generation before PG_workingset is set. Patch 1 now moves the LRU insertion out of the swap cache allocator so the refault is evaluated before it, as it was originally. - Sashiko also pointed out that a failed writeback redirties the buffer and leaves it in the swap cache with its shadow still parked, so a later zswap_store() on it would hand the parked value to zswap_entry_free(). zswap_store() now bails out for such a folio: writeback already decided that data belongs on disk, so it is written there instead of being compressed again, which also keeps the parked shadow intact until the folio leaves the swap cache. - The buffer folio a swap-in consumes is already on the LRU, so the refault there cannot activate it by setting PG_active: that leaves the flag disagreeing with the list the folio is on, which shows up as an mm/memcontrol.c lru_size underflow when it is freed. Such a folio is now activated with folio_activate() instead. One still sitting in a per-CPU batch cannot be moved safely and just misses the activation; a counter on that path measured 0.0007% of the activations over a 10 run test. This is only needed because zswap writeback still puts its buffer on the LRU: once it stops doing so, workingset_refault_lru_managed() has no caller left and can go away. [1] https://lore.kernel.org/linux-mm/20260825135209.3135169-2-alex@ghiti.fr/ v1: https://lore.kernel.org/all/20260817144622.137133-1-alex@ghiti.fr/ v2: https://lore.kernel.org/linux-mm/20260821093606.2231216-1-alex@ghiti.fr/ v3: https://lore.kernel.org/linux-mm/20260825172604.3243589-1-alex@ghiti.fr/ Alexandre Ghiti (3): mm: swap: move LRU insertion out of the swap cache allocator mm: swap: refault on swap-in, not in the swap cache allocator mm: zswap: preserve the workingset shadow across writeback include/linux/zswap.h | 12 +++++ mm/internal.h | 1 + mm/memory.c | 5 ++ mm/shmem.c | 5 ++ mm/swap.h | 7 +-- mm/swap_state.c | 52 +++++++++++++++----- mm/swapfile.c | 2 +- mm/vmscan.c | 4 +- mm/workingset.c | 60 ++++++++++++++++++----- mm/zswap.c | 108 ++++++++++++++++++++++++++++++++++++++++-- 10 files changed, 224 insertions(+), 32 deletions(-) base-commit: 1a46b1e97bde62afa7d925bb0dcd9f9748a1d7c3 -- 2.53.0-Meta