* [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition)
@ 2026-09-18 18:02 Nhat Pham
2026-09-18 18:02 ` [PATCH v5 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
` (11 more replies)
0 siblings, 12 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Changelog:
* v4 [v4] -> v5
* OVERCOMMIT_GUESS now allows for 3xRAM margin when vswap is enabled,
to take into account swapfile-less zswap usage (proposed by
Johannes Weiner).
* Limit vswap swapfile size to 8TB and drop the last patch, to avoid
memcg private id refcnt saturation.
* More assorted cleanups and fixes: anon swappability check, etc.
* RFC: Replace the xarray with a new data structure (vmalloc array)
(new patch 11).
* Rebased onto mm-unstable.
* v3 [v3] -> v4
* Replaced the runtime sysctl with a cmdline param, and remove
CONFIG_VSWAP (suggested by Johannes Weiner). CONFIG_VSWAP_DEFAULT_ON
now only gives the default value of the vswap cmdline parameter.
* Refactor swap-related memcg operations into composable building
blocks: reference acquisitions, charging, etc. (patch 8,
suggested by Johannes Weiner).
* Fixed vtable UAF bug reported by syzbot and Kunwu.
* Rebased onto mm-unstable (minimal merge conflicts).
* Re-run benchmarks (no signal change).
* v2 [v2] -> v3:
* Rebased onto current mm-unstable.
* Add a runtime vm.vswap_enabled sysctl and CONFIG_VSWAP_DEFAULT_ON
to gate vswap allocation.
* More cleanups and small bug fixes.
* Split THP swapin enablement into its own patch (patch 5).
* Add production workload benchmark results, and drop RFC tag.
* v1 [v1] -> v2:
* Rebased to a newer mm-unstable tip.
* Fix a bunch of assorted issues (incorrect zswap store failure
rollback, vswap_init() failure handling, rmap-encoding collision,
etc.) and clean up the code (rename a bunch of functions to
more closely follow existing patterns, etc.).
* Some more code clean up and simplification: some renamings to more
closely follow existing patterns, move vswap backing check to
__swap_cache_add_check, store zero state in the swap_table for
vswap entries, etc.. Many of these are proposed by Kairui Song
in [1].
* Defer memcg_table allocation on physical clusters until the first
vswap-backing slot installs. Saves ~512 bytes per physical cluster
that only serves vswap-backing slots (this is the new patch 8).
* Widen swap_info_struct->max and ->pages (and the swapoff unuse-path
index) so vswap supports ~8 PB of swap space (this is the new
patch 9).
* Split the physical-swap-backend patch into three for reviewability:
the core backend (patch 3), zswap writeback to physical swap
(patch 4), and reclaim of cache-only physical slots (patch 5). No
functional change.
* Add kerneldoc for the vswap API.
* Add some benchmark numbers for zswap case.
Patch 11 is an RFC. It swaps vswap's cluster xarray for the VM_SPARSE
vmalloc array Baoquan He designed for xswap, to show that the data
structure and the device model are separable: moving to his is one
self-contained patch that adds no userspace interface. Note that
per Baoquan's commentary (see [5]), I have skipped shrink for now, only
freeing vtable when the cluster becomes free to minimize metadata
overhead while keeping the skeleton in the free list. I have not done
performance testing on this patch yet (the number is from the old design),
but I have run a suite of simple stress tests.
It is adapted almost entirely from Baoquan's code (see [6]), so I have
kept Baoquan's Co-developed-by and Signed-off-by tag.
I. Context and Motivation
=========================
Currently, when an anon page is swapped out, a slot in a backing swap
device is allocated and stored in the page table entries that refer to
the original page. This slot is also used as the "key" to find the
swapped out content, as well as the index to swap data structures, such
as the swap cache, or the swap cgroup mapping. Tying a swap entry to its
backing slot in this way is performant and efficient when swap is purely
just disk space, and swapoff is rare.
However, the advent of many swap optimizations has exposed major
drawbacks of this design. The first problem is that we occupy a physical
slot in the swap space, even for pages that are NEVER expected to hit
the disk: pages compressed and stored in the zswap pool, zero-filled
pages, or pages rejected by both of these optimizations when zswap
writeback is disabled. This is arguably the central shortcoming of
zswap:
* Resource-wise, it is hugely wasteful in terms of disk usage. At Meta,
we size swapfile in the order of 25-50% of host RAM, depending on flash
availability. This is a lot of flash for a fleet of our size, and
with universal zswap enablement, most of this is wasted for zswap
entries.
* In deployments when no disk space can be afforded for swap (such as
mobile and embedded devices), users cannot adopt zswap, and are forced
to use zram. This is confusing for users, and creates extra burdens
for developers, having to develop and maintain similar features for
two separate swap backends (writeback, cgroup charging, THP support,
etc.). For instance, see the discussion in [2].
* Tying zswap (and more generally, other in-memory swap backends) to
the current physical swapfile infrastructure makes zswap implicitly
statically sized. This does not make sense, as unlike disk swap, in
which we consume a limited resource (disk space or swapfile space) to
save another resource (memory), zswap consumes the same resource it is
saving (memory). The more we zswap, the more memory we have available,
not less. We are not rationing a limited resource when we limit
the size of the zswap pool, but rather we are capping the resource
(memory) saving potential of zswap. Under memory pressure, using
more zswap is almost always better than the alternative (disk IOs, or
even worse, OOMs), and dynamically sizing the zswap pool on demand
allows the system to flexibly respond to these precarious scenarios.
* Operationally, static provisioning the swapfile for zswap poses
significant challenges, because the sysadmin has to prescribe how
much swap is needed a priori, for each combination of
(memory size x disk space x workload usage). It is even more
complicated when we take into account the variance of memory
compression, which changes the reclaim dynamics (and as a result,
swap space size requirement). The problem is further exacerbated for
users who rely on swap utilization (and exhaustion) as an OOM signal.
All of these factors make it very difficult to configure the swapfile
for zswap: too small of a swapfile and we risk preventable OOMs and
limit the memory saving potentials of zswap; too big of a swapfile
and we waste disk space and memory due to swap metadata overhead.
This dilemma becomes more drastic in high memory systems, which can
have up to TBs worth of memory.
Swap virtualization is the answer to these issues, with three properties:
1. Decoupled backends. For zswap in particular, this means we eliminate
the unused storage space, and allows zswap to be used in systems that
do not have enough storage capacity for physical swap (without having
to resort to silly hacks). Zero-filled swap pages and swap-cache-only
folios also benefit here.
2. Dynamic swap space. Since virtual swap is not tied to any physical
resource, we can make it effectively infinite and dynamically grow it
on demand.
This massively simplifies operational provisioning, and increases the
utilization of compressed swap backends (zswap). Dynamicity also
reduces overhead on unused swap capacity.
3. Efficient backend transfer. The virtualization scheme should not
introduce PTE/rmap walking overhead for backend transfer. This
is crucial for systems that want to support multiple swap backends
in a tiering fashion (for e.g zswap -> disk swap).
For more historical contexts and references, please take a look at
the cover letter of the older vswap submissions ([3] and [v2]).
II. Design
==========
When vswap is enabled (via vswap=on cmdline parameter), a special vswap
device is allocated at boot time. Anon pages that can be zswapped will
obtain a vswap slot at swap allocation time.
These swap entries can subsequently acquire backend on-demand, such as
a zswap entry, or a slot on a physical swap device (as a fallback option
or at zswap writeback time).
We repurpose much of the existing swap_table infrastructure and
swapfile allocator for this new vswap device, with two notable
differences:
* Clusters are dynamically allocated on demand and managed through
an xarray. This in turn allows us to avoid static provisioning and
let swap space grow dynamically.
* Each cluster of this new vswap device has a virtual_table that stores
the backend information of the entries in the cluster (see below).
Diagrams:
Case 1: vswap entry (virtualized)
PTE swap_cluster_info_dynamic
vswap_entry +---------------------------------+
(swp_entry_t) ------>| swap_cluster_info (ci) |
| +----------------------------+ |
| | swap_table | |
| | PFN / Shadow | |
| | memcg_table | |
| | count,flags,order | |
| | lock, list | |
| +----------------------------+ |
| |
| virtual_table |
| +----------------------------+ |
| | NONE | |
| | SWAPFILE(swp_entry_t) | |
| | ZSWAP(struct zswap_entry*) | |
| +----------------------------+ |
+---------------------------------+
|
| SWAPFILE resolves to
v
PHYSICAL CLUSTER (swap_cluster_info)
+--------------------------+
| swap_table per-slot: |
| NULL - free |
| PFN - cached folio |
| Shadow - swapped out |
| Pointer- vswap rmap |
| Bad - unusable |
| |
| Vswap-backing slot: |
| Pointer(C|swp_entry_t) |
| rmap back to vswap |
+--------------------------+
Case 2: direct-mapped physical entry (no vswap)
PTE PHYSICAL CLUSTER (swap_cluster_info)
phys_entry +--------------------------+
(swp_entry_t) ------>| swap_table per-slot: |
| NULL - free |
| PFN - cached folio |
| Shadow - swapped out |
| Bad - unusable |
+--------------------------+
struct swap_cluster_info_dynamic {
struct swap_cluster_info ci; /* swap_table, lock, etc. */
unsigned int index; /* position in xarray */
struct rcu_head rcu; /* kfree_rcu deferred free */
atomic_long_t *virtual_table; /* backend info, 8 B/slot */
};
Each vswap cluster (swap_cluster_info_dynamic) extends the classic
swap_cluster_info struct with a virtual_table array that stores the
backend information for each virtual swap entry in the cluster. Each
entry is tag-encoded in the low 3 bits to indicate the backend type:
NONE: |----- 0000 ------|000| free / unbacked
ZSWAP: |--- zswap_entry* |001| compressed in zswap
SWAPFILE: |- type:5,off:56 -|010| on a physical swapfile
Other design highlights:
* Note that for the vswap device, we have merged the zswap xarray tree
with the swapfile-level clusters. This means that for zswap only users,
we have negligible extra space overhead.
* Both vswap entries (Case 1) and directly-mapped physical entries
(Case 2) coexist as first-class citizens.
* Backend transitions in the virtual_table are synchronized through the
swap cache and the folio lock - the same mechanism that already
serializes ordinary swap operations (swapin, swapout, zswap
writeback, swap cache reclaim). IOW, we can only assume that the
backend of a vswap entry is stable through swap cache/folio lock.
Looking at the backend without this should be done at best for
optimization purposes, as there is no guarantee that the backend
will not change under the observer.
* Pointer-tagged swap_table entries on physical clusters provide the
rmap (physical -> virtual) lookup.
* Virtual swap slots not backed by physical swap are not charged to
memcg swap counters - only physical backing is charged (I made the
case for this in [4]).
III. Benchmarks
===============
Note that the goal is not to match vswap performance with baseline on
every single case yet - running with vswap off is still supported. We
can optimize further once we have landed this new feature.
A. Production Workload: Instagram
=================================
To test vswap's stability and performance, I ran an A/B experiment on
Instagram (django) workload, with zswap as the swap backend. On these
hosts, the swapfiles' size is 50% of RAM.
Compared to baseline, vswap gives:
* On par request throughput.
* Lower request serving latency (by about 1-3%).
* Lower memory pressure in the system service cgroups running alongside
the workload. PSI-based proactive reclaimer can therefore recover more
from them, lowering their overall memory footprint, allowing the main
workload to expand.
* Elimination of swapfile footprint for all zswap users in the host.
B. Semi-synthetic Workloads (memhog, usemem, kernel build)
==========================================================
All values are mean +/- standard deviation across rounds.
Test system: x86_64, 52 cores, 64 GB swapfile for all 3 benchmarks.
Swap backend: zswap (zstd) with the traditional active/inactive LRU. We
focus on zswap here because it is the motivating use case for vswap.
For each benchmark, we test 3 kernels:
* Baseline: mm-unstable, no vswap patches.
* VSS off: vswap series applied, vswap=off, to verify that there is no
regression to existing swap paths when we disable vswap.
* VSS on: vswap series applied, vswap=on.
1. Memhog: single-threaded, 48GB allocation on a host with 16GB RAM,
20 rounds.
Baseline VSS off VSS on
real (s) 124.05 +/- 11.64 122.31 +/- 10.29 118.57 +/- 15.68
sys (s) 106.75 +/- 10.86 105.01 +/- 9.64 101.34 +/- 13.97
user (s) 10.81 +/- 0.11 10.85 +/- 0.09 10.79 +/- 0.09
delta real - -1.4% -4.4%
delta sys - -1.6% -5.1%
Dropping the best and the worst round to reduce variance:
memhog Baseline VSS off VSS on
real (s) 123.75 +/- 10.39 122.04 +/- 9.10 116.06 +/- 8.22
sys (s) 106.80 +/- 10.21 104.99 +/- 8.86 99.29 +/- 8.27
user (s) 10.82 +/- 0.11 10.85 +/- 0.08 10.79 +/- 0.10
delta real - -1.4% -6.2%
delta sys - -1.7% -7.0%
2. Usemem single-threaded: 56GB allocation on a host with 32GB RAM,
16 rounds.
Baseline VSS off VSS on
real (s) 178.75 +/- 6.47 178.95 +/- 6.56 175.97 +/- 7.74
sys (s) 127.03 +/- 6.56 128.08 +/- 6.59 124.71 +/- 7.90
tput (KB/s) 386662 +/- 14648 386264 +/- 15150 390443 +/- 17532
free (ms) 7669 +/- 146 7678 +/- 136 6439 +/- 111
delta real - +0.1% -1.6%
delta sys - +0.8% -1.8%
delta tput - -0.1% +1.0%
delta free - +0.1% -16.0%
3. Kernel build: 52 workers (one per processor), memory.max=3GB, 10 rounds.
Baseline VSS off VSS on
real (s) 165.58 +/- 0.45 165.83 +/- 0.49 166.01 +/- 0.58
sys (s) 694.24 +/- 26.13 710.76 +/- 19.83 705.06 +/- 21.40
user (s) 5132.62 +/- 1.12 5133.69 +/- 1.57 5134.68 +/- 1.68
delta real - +0.2% +0.3%
delta sys - +2.4% +1.6%
delta user - +0.0% +0.0%
For zswap backend, vswap outperforms baseline on usemem freeing, and is
on par with baseline on the rest.
IV. References
==============
[v1]: https://lore.kernel.org/all/20260528212955.1912856-1-nphamcs@gmail.com/
[v2]: https://lore.kernel.org/all/20260612193738.2183968-1-nphamcs@gmail.com/
[v3]: https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@gmail.com/
[v4]: https://lore.kernel.org/all/20260825153238.2695446-1-nphamcs@gmail.com/
[1]: https://lore.kernel.org/all/CAMgjq7BhOn48xEyC=2j837R7qddfjeBVHMiRqdx8no4ZEBpBLg@mail.gmail.com/
[2]: https://lore.kernel.org/all/Zqe_Nab-Df1CN7iW@infradead.org/
[3]: https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@gmail.com/
[4]: https://lore.kernel.org/linux-mm/CAKEwX=P4syV38jAVCWq198r2OHXXc=xA-fx1dk6+qYef6yzxWQ@mail.gmail.com/
[5]: https://lore.kernel.org/all/aqjpYHbZ14A8xRtK@fedora/
[6]: https://lore.kernel.org/all/20260916101929.149106-1-hebaoquan@kylinos.cn/
Nhat Pham (11):
mm, swap: add virtual swap device infrastructure
mm, swap: support zswap and zero-filled swap pages as vswap backends
mm, swap: prepare the swap IO path for vswap
mm, swap: support physical swap as a vswap backend
mm, swap: enable THP swapin for vswap entries
mm, swap: write back vswap zswap entries to physical swap
mm, swap: reclaim physical slots backing cache-only vswap entries
mm, swap: only charge physical swap entries
mm, swap: add debugfs counters for vswap
mm, swap: defer memcg_table allocation for physical swap clusters
mm, swap: back vswap clusters with a VM_SPARSE array
.../admin-guide/cgroup-v1/memcg_test.rst | 2 +-
Documentation/admin-guide/cgroup-v2.rst | 46 +-
.../admin-guide/kernel-parameters.txt | 7 +
MAINTAINERS | 1 +
include/linux/memcontrol.h | 6 +
include/linux/swap.h | 88 +-
include/linux/swap_ops.h | 9 +-
include/linux/zswap.h | 4 +
mm/Kconfig | 20 +
mm/memcontrol-v1.c | 10 +-
mm/memcontrol.c | 147 +-
mm/memory.c | 21 +-
mm/page_io.c | 101 +-
mm/shmem.c | 4 +-
mm/swap.h | 28 +-
mm/swap_state.c | 50 +-
mm/swap_table.h | 64 +-
mm/swapfile.c | 1254 +++++++++++++++--
mm/util.c | 13 +-
mm/vmscan.c | 18 +-
mm/vswap.h | 422 ++++++
mm/workingset.c | 2 +-
mm/zswap.c | 132 +-
23 files changed, 2180 insertions(+), 269 deletions(-)
create mode 100644 mm/vswap.h
base-commit: 27e4e1835109ef599d72abe6c09711e0b1916033
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 01/11] mm, swap: add virtual swap device infrastructure
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 02/11] mm, swap: support zswap and zero-filled swap pages as vswap backends Nhat Pham
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Create a virtual swap device (just under 8 TiB with 4 KiB pages), along
with the dynamic cluster infrastructure that the rest of the vswap layer is
built on. swap_cluster_info_dynamic keeps per-cluster info in an xarray, so
a device can be sized without a static cluster_info[] array. For now, vswap
requires a 64-bit architecture.
swap_table_lookup() resolves a swap entry's cluster and reads its slot,
both under RCU. A vswap cluster is allocated on demand and freed by
kfree_rcu(), so an unpinned entry cannot have its cluster resolved
beforehand, and the lookup can return NULL. Callers holding a pinned
entry keep resolving it themselves: the pin holds ci->count above zero,
so the cluster cannot be destroyed. mm/workingset.c includes swap_table.h
with CONFIG_SWAP off, so the helpers swap_table_lookup() calls gain
!CONFIG_SWAP stubs.
The dynamic-cluster allocator is wired in, but nothing reaches it yet.
vswap_si is kept off the swap device lists, and no allocation path can
select it. Backends (zswap, zero, physical disk) and the vswap-aware
swap-out / swap-in / writeback paths arrive in subsequent patches.
Routing is controlled by the "vswap=" kernel parameter, defaulting to
CONFIG_VSWAP_DEFAULT_ON. When off, no device exists, every vswap path is
skipped, and swap behavior is unchanged. When on, vswap_init() creates
the device and enables a static key only after it is fully published, so
callers never observe a half-built device. The device lives for the
lifetime of the kernel and cannot be swapon'd or swapoff'd.
The device is capped at REFCOUNT_MAX pages, just under 8 TiB with 4 KiB
pages. Every swapped page takes a reference on its cgroup's
memcg->private_id_ref, a 32-bit refcount_t, so a cgroup that swaps past
that saturates it and leaks the memcg until reboot. The limit is not
specific to vswap, a physical swapfile that large hits it too, but vswap
is the first device that can reach it without provisioning the storage.
Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
.../admin-guide/kernel-parameters.txt | 7 +
MAINTAINERS | 1 +
include/linux/swap.h | 9 +
mm/Kconfig | 20 ++
mm/page_io.c | 14 +
mm/swap.h | 77 ++++-
mm/swap_state.c | 40 +--
mm/swap_table.h | 27 ++
mm/swapfile.c | 275 ++++++++++++++++--
mm/vswap.h | 34 +++
mm/zswap.c | 6 +
11 files changed, 468 insertions(+), 42 deletions(-)
create mode 100644 mm/vswap.h
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 68647ff4bdd2..646e689d9e80 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -8431,6 +8431,13 @@ Kernel parameters
force - force vulnerability detection even on
unaffected processors
+ vswap= [MM,EARLY]
+ Route swapouts through the virtual swap layer, which
+ allows zswap and zero-filled pages to be used without
+ a physical swap device. 64-bit only.
+ Format: { on | off }
+ Default: on if CONFIG_VSWAP_DEFAULT_ON=y, else off.
+
vsyscall= [X86-64,EARLY]
Controls the behavior of vsyscalls (i.e. calls to
fixed addresses of 0xffffffffff600x00 from legacy
diff --git a/MAINTAINERS b/MAINTAINERS
index e4412c3d8d45..e49a3d4e332b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17411,6 +17411,7 @@ F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
F: mm/swapfile.c
+F: mm/vswap.h
MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
M: Andrew Morton <akpm@linux-foundation.org>
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 61005501888c..0b341301d8fa 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -201,6 +201,7 @@ enum {
SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
+ SWP_VSWAP = (1 << 14), /* virtual swap device */
/* add others here before... */
};
@@ -270,8 +271,14 @@ struct swap_info_struct {
struct list_head discard_clusters; /* discard clusters list */
struct plist_node avail_list; /* entry in swap_avail_head */
const struct swap_ops *ops;
+ struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */
};
+static inline bool swap_is_vswap(struct swap_info_struct *si)
+{
+ return si->flags & SWP_VSWAP;
+}
+
/**
* folio_swap_entry - Return the swap entry at a page index within a folio.
* @folio: The folio.
@@ -427,6 +434,8 @@ void swap_free_hibernation_slot(swp_entry_t entry);
static inline void put_swap_device(struct swap_info_struct *si)
{
+ if (swap_is_vswap(si))
+ return;
percpu_ref_put(&si->users);
}
diff --git a/mm/Kconfig b/mm/Kconfig
index 30170a936f1f..beef56e870ce 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -19,6 +19,26 @@ menuconfig SWAP
used to provide more virtual memory than the actual RAM present
in your computer. If unsure say Y.
+config VSWAP_DEFAULT_ON
+ bool "Route swapouts through virtual swap by default"
+ depends on SWAP && 64BIT
+ default n
+ help
+ Virtual swap allows zswap and zero-filled pages to be used
+ without swapping on a physical device first, and lets a page
+ move between zswap and a swapfile without invalidating the page
+ table entries that refer to it.
+
+ Swap entries are handed out by a virtual swap device instead of
+ naming a slot on a real one, so the backing can be chosen and
+ changed after the entry exists.
+
+ Say Y to make "vswap=on" the default, routing swapouts through
+ the virtual swap layer from boot.
+
+ Say N (default) to leave vswap off unless "vswap=on" is passed
+ on the kernel command line.
+
config ZSWAP
bool "Compressed cache for swap pages"
depends on SWAP
diff --git a/mm/page_io.c b/mm/page_io.c
index 1da4ff484f09..5cd77bd1b12c 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -28,6 +28,7 @@
#include <linux/swap_ops.h>
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
int generic_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file,
@@ -248,6 +249,14 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
}
rcu_read_unlock();
+ /*
+ * A vswap folio has no physical slot to write to, so keep it dirty.
+ */
+ if (is_vswap_entry(folio->swap)) {
+ folio_mark_dirty(folio);
+ return AOP_WRITEPAGE_ACTIVATE;
+ }
+
__swap_writeout(ctx, folio);
return 0;
out_unlock:
@@ -482,6 +491,11 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
if (zswap_load(folio) != -ENOENT)
goto finish;
+ if (unlikely(swap_is_vswap(sis))) {
+ folio_unlock(folio);
+ goto finish;
+ }
+
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
swap_add_folio(ctx, folio, READ);
diff --git a/mm/swap.h b/mm/swap.h
index b3b54c28929a..528a27afb335 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -67,6 +67,12 @@ struct swap_cluster_info {
struct list_head list;
};
+struct swap_cluster_info_dynamic {
+ struct swap_cluster_info ci;
+ unsigned int index; /* for cluster_index() */
+ struct rcu_head rcu;
+};
+
/* All on-list cluster must have a non-zero flag. */
enum swap_cluster_flags {
CLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */
@@ -77,6 +83,7 @@ enum swap_cluster_flags {
CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
CLUSTER_FLAG_FULL,
CLUSTER_FLAG_DISCARD,
+ CLUSTER_FLAG_DEAD, /* Vswap dynamic cluster pending kfree_rcu */
CLUSTER_FLAG_MAX,
};
@@ -119,12 +126,33 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
return __swap_type_to_info(swp_type(entry));
}
+/**
+ * __swap_offset_to_cluster - look up the cluster holding a swap offset
+ * @si: the swap device
+ * @offset: the swap entry offset
+ *
+ * Context: A vswap cluster is freed by kfree_rcu(). Callers must hold the
+ * RCU read lock, or know the cluster is pinned by an in-use entry.
+ *
+ * Return: the cluster, or NULL if @si is a vswap device with no cluster
+ * allocated at @offset.
+ */
static inline struct swap_cluster_info *__swap_offset_to_cluster(
struct swap_info_struct *si, pgoff_t offset)
{
+ unsigned int cluster_idx = offset / SWAPFILE_CLUSTER;
+
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
VM_WARN_ON_ONCE(offset >= roundup(si->max, SWAPFILE_CLUSTER));
- return &si->cluster_info[offset / SWAPFILE_CLUSTER];
+
+ if (swap_is_vswap(si)) {
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = xa_load(&si->cluster_info_pool, cluster_idx);
+ return ci_dyn ? &ci_dyn->ci : NULL;
+ }
+
+ return &si->cluster_info[cluster_idx];
}
static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entry)
@@ -133,10 +161,36 @@ static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entr
swp_offset(entry));
}
+static inline struct swap_cluster_info *__vswap_cluster_lock(
+ struct swap_info_struct *si, unsigned long offset, bool irq)
+{
+ struct swap_cluster_info *ci;
+
+ rcu_read_lock();
+ ci = __swap_offset_to_cluster(si, offset);
+ if (ci) {
+ if (irq)
+ spin_lock_irq(&ci->lock);
+ else
+ spin_lock(&ci->lock);
+
+ /* The cluster can be torn down while we wait for the lock. */
+ if (ci->flags == CLUSTER_FLAG_DEAD) {
+ if (irq)
+ spin_unlock_irq(&ci->lock);
+ else
+ spin_unlock(&ci->lock);
+ ci = NULL;
+ }
+ }
+ rcu_read_unlock();
+ return ci;
+}
+
static __always_inline struct swap_cluster_info *__swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset, bool irq)
{
- struct swap_cluster_info *ci = __swap_offset_to_cluster(si, offset);
+ struct swap_cluster_info *ci;
/*
* Nothing modifies swap cache in an IRQ context. All access to
@@ -149,6 +203,11 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
*/
VM_WARN_ON_ONCE(!in_task());
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
+
+ if (swap_is_vswap(si))
+ return __vswap_cluster_lock(si, offset, irq);
+
+ ci = __swap_offset_to_cluster(si, offset);
if (irq)
spin_lock_irq(&ci->lock);
else
@@ -159,10 +218,12 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
/**
* swap_cluster_lock - Lock and return the swap cluster of given offset.
* @si: swap device the cluster belongs to.
- * @offset: the swap entry offset, pointing to a valid slot.
+ * @offset: the swap entry offset.
*
* Context: The caller must ensure the offset is in the valid range and
* protect the swap device with reference count or locks.
+ * Return: the locked cluster, or NULL if it is gone. Only a vswap device
+ * can return NULL, as its clusters are allocated and freed on demand.
*/
static inline struct swap_cluster_info *swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset)
@@ -363,6 +424,16 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
return NULL;
}
+static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entry)
+{
+ return NULL;
+}
+
+static inline unsigned int swp_cluster_offset(swp_entry_t entry)
+{
+ return 0;
+}
+
static inline int folio_alloc_swap(struct folio *folio)
{
return -EINVAL;
diff --git a/mm/swap_state.c b/mm/swap_state.c
index cef44aadee61..424a0040b4b8 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -96,8 +96,7 @@ struct folio *swap_cache_get_folio(swp_entry_t entry)
struct folio *folio;
for (;;) {
- swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
- swp_cluster_offset(entry));
+ swp_tb = swap_table_lookup(entry);
if (!swp_tb_is_folio(swp_tb))
return NULL;
folio = swp_tb_to_folio(swp_tb);
@@ -119,8 +118,7 @@ bool swap_cache_has_folio(swp_entry_t entry)
{
unsigned long swp_tb;
- swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
- swp_cluster_offset(entry));
+ swp_tb = swap_table_lookup(entry);
return swp_tb_is_folio(swp_tb);
}
@@ -136,8 +134,7 @@ void *swap_cache_get_shadow(swp_entry_t entry)
{
unsigned long swp_tb;
- swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
- swp_cluster_offset(entry));
+ swp_tb = swap_table_lookup(entry);
if (swp_tb_is_shadow(swp_tb))
return swp_tb_to_shadow(swp_tb);
return NULL;
@@ -414,14 +411,16 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
* -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
* should abort or try to use the cached folio instead
*/
-static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
- swp_entry_t targ_entry, gfp_t gfp,
+static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
unsigned int order, struct vm_fault *vmf,
struct mempolicy *mpol, pgoff_t ilx)
{
int err;
swp_entry_t entry;
struct folio *folio;
+ struct swap_cluster_info *ci;
+ struct swap_info_struct *si = __swap_entry_to_info(targ_entry);
+ unsigned long offset = swp_offset(targ_entry);
void *shadow = NULL;
unsigned short memcg_id;
unsigned long address, nr_pages = 1UL << order;
@@ -431,9 +430,12 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
entry.val = round_down(targ_entry.val, nr_pages);
/* Check if the slot and range are available, skip allocation if not */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
- spin_unlock(&ci->lock);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci) {
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
+ swap_cluster_unlock(ci);
+ }
if (unlikely(err))
return ERR_PTR(err);
@@ -454,10 +456,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
return ERR_PTR(-ENOMEM);
/* Double check the range is still not in conflict */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci)
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
if (unlikely(err)) {
- spin_unlock(&ci->lock);
+ if (ci)
+ swap_cluster_unlock(ci);
folio_put(folio);
return ERR_PTR(err);
}
@@ -465,10 +470,11 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
__folio_set_locked(folio);
__folio_set_swapbacked(folio);
__swap_cache_do_add_folio(ci, folio, entry);
- spin_unlock(&ci->lock);
+ swap_cluster_unlock(ci);
if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
vmf ? vmf->vma->vm_mm : NULL, gfp)) {
+ /* The folio pins the cluster */
spin_lock(&ci->lock);
__swap_cache_do_del_folio(ci, folio, entry, shadow);
spin_unlock(&ci->lock);
@@ -525,9 +531,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
{
int order, err;
struct folio *ret;
- struct swap_cluster_info *ci;
- ci = __swap_entry_to_cluster(targ_entry);
order = highest_order(orders);
/* orders must be non-zero, and must not exceed cluster size. */
@@ -535,7 +539,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
return ERR_PTR(-EINVAL);
do {
- ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
+ ret = __swap_cache_alloc(targ_entry, gfp, order,
vmf, mpol, ilx);
if (!IS_ERR(ret))
break;
diff --git a/mm/swap_table.h b/mm/swap_table.h
index e6613e62f8d0..3d64d1629882 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -6,6 +6,8 @@
#include <linux/atomic.h>
#include "swap.h"
+extern struct swap_info_struct *vswap_si;
+
/* A typical flat array in each cluster as swap table */
struct swap_table {
atomic_long_t entries[SWAPFILE_CLUSTER];
@@ -264,6 +266,31 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
return swp_tb;
}
+/*
+ * Resolve @entry's cluster and read its slot, both under RCU. A vswap
+ * cluster is allocated on demand and freed by kfree_rcu(), so a caller
+ * starting from an entry cannot resolve it beforehand.
+ */
+static inline unsigned long swap_table_lookup(swp_entry_t entry)
+{
+ struct swap_cluster_info *ci;
+ atomic_long_t *table;
+ unsigned long swp_tb;
+
+ rcu_read_lock();
+ ci = __swap_entry_to_cluster(entry);
+ if (!ci) {
+ rcu_read_unlock();
+ return null_to_swp_tb();
+ }
+ table = rcu_dereference(ci->table);
+ swp_tb = table ? atomic_long_read(&table[swp_cluster_offset(entry)])
+ : null_to_swp_tb();
+ rcu_read_unlock();
+
+ return swp_tb;
+}
+
static inline void __swap_table_set_zero(struct swap_cluster_info *ci,
unsigned int ci_off)
{
diff --git a/mm/swapfile.c b/mm/swapfile.c
index c1c5fbb3c909..edd7bddf7ff9 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -36,6 +36,7 @@
#include <linux/poll.h>
#include <linux/oom.h>
#include <linux/swapfile.h>
+#include <linux/swap_ops.h>
#include <linux/export.h>
#include <linux/sort.h>
#include <linux/completion.h>
@@ -46,6 +47,7 @@
#include <asm/tlbflush.h>
#include <linux/leafops.h>
#include "swap_table.h"
+#include "vswap.h"
#include "internal.h"
#include "swap.h"
@@ -399,6 +401,8 @@ static inline bool cluster_is_usable(struct swap_cluster_info *ci, int order)
static inline unsigned int cluster_index(struct swap_info_struct *si,
struct swap_cluster_info *ci)
{
+ if (swap_is_vswap(si))
+ return container_of(ci, struct swap_cluster_info_dynamic, ci)->index;
return ci - si->cluster_info;
}
@@ -594,10 +598,15 @@ static void move_cluster(struct swap_info_struct *si,
lockdep_assert_held(&ci->lock);
spin_lock(&si->lock);
- if (ci->flags == CLUSTER_FLAG_NONE)
+ if (!list) {
+ /* Going away. An isolated cluster is already off its list. */
+ if (ci->flags != CLUSTER_FLAG_NONE)
+ list_del(&ci->list);
+ } else if (ci->flags == CLUSTER_FLAG_NONE) {
list_add_tail(&ci->list, list);
- else
+ } else {
list_move_tail(&ci->list, list);
+ }
spin_unlock(&si->lock);
ci->flags = new_flags;
}
@@ -615,6 +624,18 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info
{
swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, false);
swap_cluster_free_table(ci);
+
+ if (swap_is_vswap(si)) {
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ /* vswap clusters are destroyed, not returned to free_clusters. */
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ xa_erase(&si->cluster_info_pool, ci_dyn->index);
+ move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD);
+ kfree_rcu(ci_dyn, rcu);
+ return;
+ }
+
move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
ci->order = 0;
}
@@ -851,6 +872,8 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
unsigned long offset = start, end = start + nr_pages;
unsigned long swp_tb;
+ VM_WARN_ON_ONCE(swap_is_vswap(si));
+
spin_unlock(&ci->lock);
do {
swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
@@ -1042,6 +1065,44 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
return found;
}
+static unsigned int vswap_alloc_cluster(struct swap_info_struct *si,
+ struct folio *folio)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
+ unsigned long offset;
+
+ VM_WARN_ON(!swap_is_vswap(si));
+
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_ATOMIC | __GFP_NOWARN);
+ if (!ci_dyn)
+ return SWAP_ENTRY_INVALID;
+
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ if (swap_cluster_alloc_table(&ci_dyn->ci, GFP_ATOMIC | __GFP_NOWARN)) {
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ /* Lock before publishing: xa_alloc makes the cluster findable by offset. */
+ ci = &ci_dyn->ci;
+ spin_lock(&ci->lock);
+
+ if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn,
+ XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1),
+ GFP_ATOMIC | __GFP_NOWARN)) {
+ spin_unlock(&ci->lock);
+ swap_cluster_free_table(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ offset = cluster_offset(si, ci);
+ return alloc_swap_scan_cluster(si, ci, folio, offset);
+}
+
static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
{
long to_scan = 1;
@@ -1064,7 +1125,9 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
spin_unlock(&ci->lock);
nr_reclaim = __try_to_reclaim_swap(si, offset,
TTRS_ANYWAY);
- spin_lock(&ci->lock);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
if (nr_reclaim) {
offset += abs(nr_reclaim);
continue;
@@ -1078,6 +1141,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
relocate_cluster(si, ci);
swap_cluster_unlock(ci);
+next:
if (to_scan <= 0)
break;
@@ -1154,6 +1218,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
goto done;
}
+ if (swap_is_vswap(si)) {
+ found = vswap_alloc_cluster(si, folio);
+ if (found)
+ goto done;
+ }
+
if (!(si->flags & SWP_PAGE_DISCARD)) {
found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
if (found)
@@ -1290,8 +1360,10 @@ static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)
/*
* If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
* remove it from the plist.
+ *
+ * Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val == si->pages)) {
+ if (unlikely(val == si->pages) && !swap_is_vswap(si)) {
del_from_avail_list(si, false);
return true;
}
@@ -1306,8 +1378,10 @@ static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)
/*
* If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
* add it to the plist.
+ *
+ * Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val & SWAP_USAGE_OFFLIST_BIT))
+ if (unlikely(val & SWAP_USAGE_OFFLIST_BIT) && !swap_is_vswap(si))
add_to_avail_list(si, false);
}
@@ -1352,6 +1426,10 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
static bool get_swap_device_info(struct swap_info_struct *si)
{
+ /* The vswap device is always alive, so it needs no refcount. */
+ if (swap_is_vswap(si))
+ return true;
+
if (!percpu_ref_tryget_live(&si->users))
return false;
/*
@@ -1387,11 +1465,11 @@ static bool swap_alloc_fast(struct folio *folio)
return false;
ci = swap_cluster_lock(si, offset);
- if (cluster_is_usable(ci, order)) {
+ if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
alloc_swap_scan_cluster(si, ci, folio, offset);
- } else {
+ } else if (ci) {
swap_cluster_unlock(ci);
}
@@ -1513,6 +1591,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
if (IS_ERR_OR_NULL(si))
return 0;
+ /* The source PTE pins the entry, so its cluster is alive. */
ci = __swap_offset_to_cluster(si, offset);
ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
@@ -1923,7 +2002,7 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)
return NULL;
put_out:
pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_offset, entry.val);
- percpu_ref_put(&si->users);
+ put_swap_device(si);
return ERR_PTR(-EIO);
}
@@ -2002,7 +2081,7 @@ bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry)
unsigned long swp_tb;
ci = swap_cluster_lock(si, offset);
- swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
+ swp_tb = __swap_table_get(ci, swp_cluster_offset(entry));
swap_cluster_unlock(ci);
return swp_tb_get_count(swp_tb) > 0;
@@ -2055,6 +2134,7 @@ static bool folio_maybe_swapped(struct folio *folio)
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ /* Folio is locked and in swap cache, so ci->count > 0: cluster is alive. */
ci = __swap_entry_to_cluster(entry);
ci_off = swp_cluster_offset(entry);
ci_end = ci_off + folio_nr_pages(folio);
@@ -2249,6 +2329,9 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev, so it is never a hibernation target. */
+ if (swap_is_vswap(sis))
+ continue;
if (device == sis->bdev->bd_dev) {
struct swap_extent *se = first_se(sis);
@@ -2375,6 +2458,9 @@ int find_first_swap(dev_t *device)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev, so it is never a hibernation target. */
+ if (swap_is_vswap(sis))
+ continue;
*device = sis->bdev->bd_dev;
spin_unlock(&swap_lock);
return type;
@@ -2591,8 +2677,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
&vmf);
}
if (!folio) {
- swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
- swp_cluster_offset(entry));
+ swp_tb = swap_table_lookup(entry);
if (swp_tb_get_count(swp_tb) <= 0)
continue;
return -ENOMEM;
@@ -3022,15 +3107,24 @@ static int setup_swap_extents(struct swap_info_struct *sis,
static void _enable_swap_info(struct swap_info_struct *si)
{
- atomic_long_add(si->pages, &nr_swap_pages);
- total_swap_pages += si->pages;
+ if (!swap_is_vswap(si)) {
+ atomic_long_add(si->pages, &nr_swap_pages);
+ total_swap_pages += si->pages;
+ }
assert_spin_locked(&swap_lock);
- plist_add(&si->list, &swap_active_head);
+ /*
+ * Vswap has no backing file and no swapoff support, so keep it
+ * off swap_active_head (used by swapoff filename lookup and
+ * swap_sync_discard) and swap_avail_head (physical allocator).
+ */
+ if (!swap_is_vswap(si)) {
+ plist_add(&si->list, &swap_active_head);
- /* Add back to available list */
- add_to_avail_list(si, true);
+ /* Add back to available list */
+ add_to_avail_list(si, true);
+ }
}
/*
@@ -3074,12 +3168,31 @@ static void wait_for_allocation(struct swap_info_struct *si)
}
}
-static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
+static void free_swap_cluster_info(struct swap_info_struct *si,
+ struct swap_cluster_info *cluster_info,
unsigned long maxpages)
{
+ struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *ci;
+ unsigned long idx;
int i, nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
+ if (swap_is_vswap(si)) {
+ xa_for_each(&si->cluster_info_pool, idx, ci_dyn) {
+ ci = &ci_dyn->ci;
+ spin_lock(&ci->lock);
+ if (cluster_table_is_alloced(ci)) {
+ swap_cluster_assert_empty(ci, 0,
+ SWAPFILE_CLUSTER, true);
+ swap_cluster_free_table(ci);
+ }
+ spin_unlock(&ci->lock);
+ kfree(ci_dyn);
+ }
+ xa_destroy(&si->cluster_info_pool);
+ return;
+ }
+
if (!cluster_info)
return;
for (i = 0; i < nr_clusters; i++) {
@@ -3226,7 +3339,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
mutex_unlock(&swapon_mutex);
kfree(p->global_cluster);
p->global_cluster = NULL;
- free_swap_cluster_info(cluster_info, maxpages);
+ free_swap_cluster_info(p, cluster_info, maxpages);
inode = mapping->host;
@@ -3573,10 +3686,39 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
unsigned long maxpages)
{
unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
- struct swap_cluster_info *cluster_info;
+ struct swap_cluster_info *cluster_info = NULL;
+ struct swap_cluster_info_dynamic *ci_dyn = NULL;
int err = -ENOMEM;
unsigned long i;
+ /* A vswap device uses an xarray pool instead of a static array. */
+ if (swap_is_vswap(si)) {
+ nr_clusters = 0;
+ xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC);
+
+ /*
+ * Pre-allocate cluster 0 and mark slot 0 (header page)
+ * as bad so the allocator never hands out page offset 0.
+ */
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);
+ if (!ci_dyn)
+ goto err;
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ err = xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL);
+ if (err) {
+ kfree(ci_dyn);
+ goto err;
+ }
+
+ err = swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false);
+ if (err)
+ goto err;
+
+ goto setup_cluster_info;
+ }
+
cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
if (!cluster_info)
goto err;
@@ -3620,6 +3762,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
goto err;
}
+setup_cluster_info:
INIT_LIST_HEAD(&si->free_clusters);
INIT_LIST_HEAD(&si->full_clusters);
INIT_LIST_HEAD(&si->discard_clusters);
@@ -3641,10 +3784,16 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
}
}
+ /* Slot 0 is bad, so cluster 0 never empties. The rest of it is usable. */
+ if (swap_is_vswap(si)) {
+ ci_dyn->ci.flags = CLUSTER_FLAG_NONFULL;
+ list_add_tail(&ci_dyn->ci.list, &si->nonfull_clusters[0]);
+ }
+
si->cluster_info = cluster_info;
return 0;
err:
- free_swap_cluster_info(cluster_info, maxpages);
+ free_swap_cluster_info(si, cluster_info, maxpages);
return err;
}
@@ -3866,7 +4015,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
si->global_cluster = NULL;
inode = NULL;
destroy_swap_extents(si, swap_file);
- free_swap_cluster_info(si->cluster_info, si->max);
+ free_swap_cluster_info(si, si->cluster_info, si->max);
si->cluster_info = NULL;
/*
* Clear the SWP_USED flag after all resources are freed so
@@ -3997,3 +4146,87 @@ static int __init swapfile_init(void)
return 0;
}
subsys_initcall(swapfile_init);
+
+struct swap_info_struct *vswap_si;
+DEFINE_STATIC_KEY_FALSE(vswap_key);
+
+static bool vswap_enabled_early __initdata = IS_ENABLED(CONFIG_VSWAP_DEFAULT_ON);
+
+static int __init early_vswap(char *buf)
+{
+ return kstrtobool(buf, &vswap_enabled_early);
+}
+early_param("vswap", early_vswap);
+
+/* vswap does no IO on its own. */
+static const struct swap_ops vswap_ops = { };
+
+static int __init vswap_init(void)
+{
+ struct swap_info_struct *si;
+ unsigned long maxpages;
+ int err;
+
+ if (!IS_ENABLED(CONFIG_64BIT)) {
+ if (vswap_enabled_early)
+ pr_warn("vswap: requires 64-bit architecture; vswap disabled, swapout falls back to direct physical swap\n");
+ return 0;
+ }
+
+ if (!vswap_enabled_early)
+ return 0;
+
+ si = alloc_swap_info();
+ if (IS_ERR(si)) {
+ pr_warn("vswap: alloc_swap_info failed (%ld); vswap disabled, swapout falls back to direct physical swap\n",
+ PTR_ERR(si));
+ return 0;
+ }
+
+ /*
+ * Each swapped page holds a reference on its cgroup's
+ * memcg->private_id_ref, a 32-bit refcount_t, so a cgroup that swaps
+ * REFCOUNT_MAX pages saturates it and leaks the memcg until reboot.
+ * Cap the device there; that also keeps page counts inside si->max's
+ * unsigned int. Lifting it needs the memcg refcount widened, which a
+ * physical swapfile of this size needs too.
+ */
+ maxpages = min(swapfile_maximum_size,
+ ALIGN_DOWN((unsigned long)REFCOUNT_MAX, SWAPFILE_CLUSTER));
+ /*
+ * SWP_WRITEOK enables slot allocation. SWP_SOLIDSTATE selects
+ * per-CPU cluster allocation; vswap has no si->global_cluster.
+ */
+ si->flags |= SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK;
+ si->ops = &vswap_ops;
+ si->bdev = NULL;
+ si->max = maxpages;
+ si->pages = maxpages - 1;
+
+ INIT_WORK(&si->discard_work, swap_discard_work);
+ INIT_WORK(&si->reclaim_work, swap_reclaim_work);
+
+ err = setup_swap_clusters_info(si, NULL, maxpages);
+ if (err)
+ goto fail;
+
+ mutex_lock(&swapon_mutex);
+ enable_swap_info(si);
+ mutex_unlock(&swapon_mutex);
+
+ vswap_si = si;
+ pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages);
+
+ /* Last: everything above must be visible before routing starts. */
+ static_branch_enable(&vswap_key);
+ return 0;
+
+fail:
+ pr_warn("vswap: setup_swap_clusters_info failed (%d); vswap disabled, swapout falls back to direct physical swap\n",
+ err);
+ spin_lock(&swap_lock);
+ si->flags = 0;
+ spin_unlock(&swap_lock);
+ return 0;
+}
+late_initcall(vswap_init);
diff --git a/mm/vswap.h b/mm/vswap.h
new file mode 100644
index 000000000000..16395f357955
--- /dev/null
+++ b/mm/vswap.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Virtual swap space
+ *
+ * Copyright (C) 2026 Nhat Pham
+ */
+#ifndef _MM_VSWAP_H
+#define _MM_VSWAP_H
+
+#include <linux/jump_label.h>
+#include <linux/swap.h>
+#include "swap.h"
+
+#ifdef CONFIG_SWAP
+
+DECLARE_STATIC_KEY_FALSE(vswap_key);
+
+/*
+ * Only true once vswap_init() has published vswap_si, so callers never
+ * see the device half built.
+ */
+static inline bool vswap_is_enabled(void)
+{
+ return static_branch_unlikely(&vswap_key);
+}
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return swap_is_vswap(__swap_entry_to_info(entry));
+}
+
+#endif /* CONFIG_SWAP */
+
+#endif /* _MM_VSWAP_H */
diff --git a/mm/zswap.c b/mm/zswap.c
index 507f2d19fd2a..73728b510ff6 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1018,6 +1018,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (IS_ERR_OR_NULL(si))
return -ENOENT;
+ /* Vswap entries have no physical backing to write to. */
+ if (swap_is_vswap(si)) {
+ put_swap_device(si);
+ return -EINVAL;
+ }
+
mpol = get_task_policy(current);
folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 02/11] mm, swap: support zswap and zero-filled swap pages as vswap backends
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
2026-09-18 18:02 ` [PATCH v5 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 03/11] mm, swap: prepare the swap IO path for vswap Nhat Pham
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Build the virtual swap layer on top of the swap-table infrastructure.
Virtual swap entries decouple PTE swap entries from physical backing,
allowing pages to be compressed by zswap (or detected as zero-filled)
without pre-allocating a physical swap slot.
This patch only supports zswap and zero-page backends. If zswap_store
fails, the page stays dirty in the swap cache. Physical disk backing
arrives in later patches.
Zswap writeback of vswap-backed entries is also disabled: they have no
physical slot to write back to yet, so the zswap shrinker (both the
dynamic count path and the pool-full worker path) is skipped while
vswap is enabled. Physical backing and real writeback come in later
patches.
THP swapin is disabled for vswap entries for now.
vswap_alloc() only routes a swapout through vswap when vswap is
enabled, either by CONFIG_VSWAP_DEFAULT_ON or by "vswap=on" on the
kernel command line. It also declines the folio when zswap is off, or
when the folio's cgroup is already over its zswap limit. In both cases
writeout would have to find a physical slot anyway, so the indirection
would buy nothing.
Anon reclaim gating needs a matching change: a vswap zswap-backed
swapout consumes no physical slot, so the physical free count no longer
tells reclaim whether anon is reclaimable. Add mem_cgroup_can_swap(),
which answers that question and gates on the swap.max headroom instead
when vswap and zswap are both on. Its callers only ever compared
mem_cgroup_get_nr_swap_pages() against a threshold, so that function
keeps its meaning and its physical free count.
The OVERCOMMIT_GUESS heuristic gates a single allocation on
totalram_pages() + total_swap_pages. Vswap deliberately contributes to
neither, so on a host with no swapfile the bound collapses to RAM alone
and allocations that vswap could absorb are refused. Compression swap is
backed by memory rather than by a device, so reference the bound to RAM:
add 2 * totalram_pages() when vswap and zswap are both on, for a 3x RAM
ceiling plus whatever physical swap exists.
mem_cgroup_can_vswap() is added alongside, so the !CONFIG_MEMCG stub of
mem_cgroup_can_swap() gets the same carve-out as the real one. Without
it, get_nr_swap_pages() reads 0 under vswap and MGLRU concludes anon can
never be swapped.
Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 13 +++
include/linux/zswap.h | 4 +
mm/memcontrol.c | 29 +++++++
mm/memory.c | 11 ++-
mm/page_io.c | 12 ++-
mm/shmem.c | 4 +-
mm/swap.h | 1 +
mm/swap_state.c | 8 ++
mm/swapfile.c | 187 ++++++++++++++++++++++++++++++++++++++++--
mm/util.c | 13 ++-
mm/vmscan.c | 18 ++--
mm/vswap.h | 185 +++++++++++++++++++++++++++++++++++++++++
mm/workingset.c | 2 +-
mm/zswap.c | 68 ++++++++++++---
14 files changed, 521 insertions(+), 34 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 0b341301d8fa..abf658f7861f 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -367,6 +367,8 @@ extern void __meminit kswapd_stop(int nid);
bool current_is_kswapd(void);
#ifdef CONFIG_SWAP
+bool mem_cgroup_can_vswap(struct mem_cgroup *memcg);
+
int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page,
unsigned long nr_pages, sector_t start_block);
int generic_swapfile_activate(struct swap_info_struct *, struct file *,
@@ -440,6 +442,11 @@ static inline void put_swap_device(struct swap_info_struct *si)
}
#else /* CONFIG_SWAP */
+static inline bool mem_cgroup_can_vswap(struct mem_cgroup *memcg)
+{
+ return false;
+}
+
static inline struct swap_info_struct *get_swap_device(swp_entry_t entry)
{
return NULL;
@@ -538,6 +545,7 @@ static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_p
long mem_cgroup_get_folio_swap_margin(struct folio *folio);
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
+bool mem_cgroup_can_swap(struct mem_cgroup *memcg, long nr_pages);
extern bool mem_cgroup_swap_full(struct folio *folio);
#else
static inline int mem_cgroup_try_charge_swap(struct folio *folio)
@@ -560,6 +568,11 @@ static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
return get_nr_swap_pages();
}
+static inline bool mem_cgroup_can_swap(struct mem_cgroup *memcg, long nr_pages)
+{
+ return mem_cgroup_can_vswap(memcg) || get_nr_swap_pages() >= nr_pages;
+}
+
static inline bool mem_cgroup_swap_full(struct folio *folio)
{
return vm_swap_full();
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index df6cafbe95dc..0cc2cc878566 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -6,6 +6,7 @@
#include <linux/mm_types.h>
struct lruvec;
+struct zswap_entry;
extern atomic_long_t zswap_stored_pages;
@@ -28,6 +29,7 @@ unsigned long zswap_total_pages(void);
bool zswap_store(struct folio *folio);
int zswap_load(struct folio *folio);
void zswap_invalidate(int type, pgoff_t offset, unsigned long nr_entries);
+void zswap_entry_free(struct zswap_entry *entry);
int zswap_swapon(int type, unsigned long nr_pages);
void zswap_swapoff(int type);
void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
@@ -54,6 +56,8 @@ static inline void zswap_invalidate(int type, pgoff_t offset,
{
}
+static inline void zswap_entry_free(struct zswap_entry *entry) {}
+
static inline int zswap_swapon(int type, unsigned long nr_pages)
{
return 0;
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 791e536efaeb..63d2c9e3dbe1 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -66,6 +66,7 @@
#include "internal.h"
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
#include <net/sock.h>
#include <net/ip.h>
#include "slab.h"
@@ -6049,6 +6050,34 @@ long mem_cgroup_get_folio_swap_margin(struct folio *folio)
return margin;
}
+/**
+ * mem_cgroup_can_swap - can @memcg swap out at least @nr_pages more pages?
+ * @memcg: the memcg to query
+ * @nr_pages: the number of pages the caller wants to swap out
+ *
+ * A vswap zswap-backed swapout needs no physical slot, so gate on the
+ * swap.max headroom rather than the physical free count.
+ *
+ * Return: true if @memcg can swap out at least @nr_pages more pages.
+ */
+bool mem_cgroup_can_swap(struct mem_cgroup *memcg, long nr_pages)
+{
+ long avail;
+
+ if (mem_cgroup_can_vswap(memcg))
+ return true;
+
+ if (!vswap_is_enabled() || !zswap_is_enabled())
+ return mem_cgroup_get_nr_swap_pages(memcg) >= nr_pages;
+
+ avail = PAGE_COUNTER_MAX;
+ for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
+ avail = min_t(long, avail,
+ READ_ONCE(memcg->swap.max) -
+ page_counter_read(&memcg->swap));
+ return avail >= nr_pages;
+}
+
bool mem_cgroup_swap_full(struct folio *folio)
{
struct mem_cgroup *memcg;
diff --git a/mm/memory.c b/mm/memory.c
index 338fce99e711..73ebc59d2b03 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -89,6 +89,7 @@
#include "pgalloc-track.h"
#include "internal.h"
#include "swap.h"
+#include "vswap.h"
#if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) && !defined(CONFIG_COMPILE_TEST)
#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.
@@ -4712,6 +4713,9 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
{
if (!folio_test_swapcache(folio))
return false;
+ /* A vswap entry holds no physical slot, so keeping it saves no IO. */
+ if (is_vswap_entry(folio->swap))
+ return true;
/*
* Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap
* cache can help save some IO or memory overhead, but these devices
@@ -4868,15 +4872,18 @@ static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
if (unlikely(userfaultfd_armed(vma)))
return 0;
+ entry = softleaf_from_pte(vmf->orig_pte);
+
/*
* A large swapped out folio could be partially or fully in zswap. We
* lack handling for such cases, so fallback to swapping in order-0
* folio.
+ *
+ * THP swapin for vswap is not supported yet either.
*/
- if (!zswap_never_enabled())
+ if (is_vswap_entry(entry) || !zswap_never_enabled())
return 0;
- entry = softleaf_from_pte(vmf->orig_pte);
/*
* Get a list of all the (large) orders below PMD_ORDER that are enabled
* and suitable for swapping THP.
diff --git a/mm/page_io.c b/mm/page_io.c
index 5cd77bd1b12c..2e7fb335eb01 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -162,13 +162,18 @@ static void swap_zeromap_folio_set(struct folio *folio)
int nr_pages = folio_nr_pages(folio);
struct swap_cluster_info *ci;
swp_entry_t entry = folio->swap;
- unsigned int i;
+ unsigned int voff, i;
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
ci = swap_cluster_get_and_lock(folio);
- for (i = 0; i < folio_nr_pages(folio); i++) {
+ if (is_vswap_entry(folio->swap)) {
+ /* Free any prior backing (e.g. ZSWAP entry from earlier swapout) */
+ voff = swp_cluster_offset(folio->swap);
+ __vswap_release_backing(ci, voff, nr_pages);
+ }
+ for (i = 0; i < nr_pages; i++) {
__swap_table_set_zero(ci, swp_cluster_offset(entry));
entry.val++;
}
@@ -236,6 +241,9 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
*/
swap_zeromap_folio_clear(folio);
+ if (is_vswap_entry(folio->swap))
+ folio_release_vswap_backing(folio);
+
if (zswap_store(folio)) {
count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
goto out_unlock;
diff --git a/mm/shmem.c b/mm/shmem.c
index b572c60f2af8..2871a1f618ec 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -86,6 +86,7 @@ static struct vfsmount *shm_mnt __ro_after_init;
#include <linux/uaccess.h>
#include "internal.h"
+#include "vswap.h"
#define VM_ACCT(size) (PAGE_ALIGN(size) >> PAGE_SHIFT)
@@ -1819,7 +1820,8 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap)
goto redirty;
- if (!total_swap_pages)
+ /* vswap doesn't contribute to total_swap_pages */
+ if (!total_swap_pages && !(vswap_is_enabled() && zswap_is_enabled()))
goto redirty;
/*
diff --git a/mm/swap.h b/mm/swap.h
index 528a27afb335..81e47dc36a02 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -71,6 +71,7 @@ struct swap_cluster_info_dynamic {
struct swap_cluster_info ci;
unsigned int index; /* for cluster_index() */
struct rcu_head rcu;
+ atomic_long_t *virtual_table; /* Backing pointers for vswap slots */
};
/* All on-list cluster must have a non-zero flag. */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 424a0040b4b8..54e4fefdab95 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -27,6 +27,7 @@
#include "internal.h"
#include "swap_table.h"
#include "swap.h"
+#include "vswap.h"
/* Swap readahead cluster size, as a power of 2 pages. */
static int page_cluster;
@@ -188,6 +189,13 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
if (nr == 1)
return 0;
+ /*
+ * Reject a vswap batch so swap_cache_alloc_folio falls back to
+ * order 0.
+ */
+ if (is_vswap_entry(targ_entry))
+ return -EBUSY;
+
is_zero = __swap_table_test_zero(ci, ci_off);
ci_off = round_down(ci_off, nr);
ci_end = ci_off + nr;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index edd7bddf7ff9..67a2399cf1ae 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -46,6 +46,7 @@
#include <asm/tlbflush.h>
#include <linux/leafops.h>
+#include "memcontrol-v1.h"
#include "swap_table.h"
#include "vswap.h"
#include "internal.h"
@@ -131,6 +132,16 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+struct percpu_vswap_cluster {
+ unsigned long offset[SWAP_NR_ORDERS];
+ local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
+ .offset = { [0 ... SWAP_NR_ORDERS - 1] = SWAP_ENTRY_INVALID },
+ .lock = INIT_LOCAL_LOCK(),
+};
+
/* May return NULL on invalid type, caller must check for NULL return */
static struct swap_info_struct *swap_type_to_info(int type)
{
@@ -236,7 +247,8 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
need_reclaim = ((flags & TTRS_ANYWAY) ||
((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) ||
- ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio)));
+ ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) &&
+ !is_vswap_entry(folio->swap)));
if (!need_reclaim || !folio_swapcache_freeable(folio))
goto out_unlock;
@@ -544,7 +556,9 @@ swap_cluster_populate(struct swap_info_struct *si,
/*
* Only cluster isolation from the allocator does table allocation.
* Swap allocator uses percpu clusters and holds the local lock.
+ * vswap clusters are destroyed rather than freed to si->free_clusters.
*/
+ VM_WARN_ON_ONCE(swap_is_vswap(si));
lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
if (!(si->flags & SWP_SOLIDSTATE))
lockdep_assert_held(&si->global_cluster_lock);
@@ -632,6 +646,7 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info
ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
xa_erase(&si->cluster_info_pool, ci_dyn->index);
move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD);
+ vswap_cluster_free_vtable(ci);
kfree_rcu(ci_dyn, rcu);
return;
}
@@ -926,7 +941,8 @@ static bool cluster_scan_range(struct swap_info_struct *si,
if (swp_tb_is_null(swp_tb))
continue;
if (swp_tb_is_folio(swp_tb) && !__swp_tb_get_count(swp_tb)) {
- if (!vm_swap_full())
+ /* vswap slots are abundant; never reclaim to reuse one */
+ if (swap_is_vswap(si) || !vm_swap_full())
return false;
*need_reclaim = true;
continue;
@@ -1034,6 +1050,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
out:
relocate_cluster(si, ci);
swap_cluster_unlock(ci);
+ if (swap_is_vswap(si)) {
+ this_cpu_write(percpu_vswap_cluster.offset[order], next);
+ return found;
+ }
if (si->flags & SWP_SOLIDSTATE) {
this_cpu_write(percpu_swap_cluster.offset[order], next);
this_cpu_write(percpu_swap_cluster.si[order], si);
@@ -1086,6 +1106,12 @@ static unsigned int vswap_alloc_cluster(struct swap_info_struct *si,
return SWAP_ENTRY_INVALID;
}
+ if (vswap_cluster_alloc_vtable(ci_dyn, GFP_ATOMIC | __GFP_NOWARN)) {
+ swap_cluster_free_table(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
/* Lock before publishing: xa_alloc makes the cluster findable by offset. */
ci = &ci_dyn->ci;
spin_lock(&ci->lock);
@@ -1095,6 +1121,7 @@ static unsigned int vswap_alloc_cluster(struct swap_info_struct *si,
GFP_ATOMIC | __GFP_NOWARN)) {
spin_unlock(&ci->lock);
swap_cluster_free_table(&ci_dyn->ci);
+ vswap_cluster_free_vtable(&ci_dyn->ci);
kfree(ci_dyn);
return SWAP_ENTRY_INVALID;
}
@@ -1178,7 +1205,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
* Swapfile is not block device so unable
* to allocate large entries.
*/
- if (order && !(si->flags & SWP_BLKDEV))
+ if (order && !(si->flags & SWP_BLKDEV) && !swap_is_vswap(si))
return 0;
if (!(si->flags & SWP_SOLIDSTATE)) {
@@ -1231,7 +1258,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
}
/* Try reclaim full clusters if free and nonfull lists are drained */
- if (vm_swap_full())
+ if (!swap_is_vswap(si) && vm_swap_full())
swap_reclaim_full_clusters(si, false);
if (order < PMD_ORDER) {
@@ -1392,7 +1419,8 @@ static void swap_range_alloc(struct swap_info_struct *si,
if (vm_swap_full())
schedule_work(&si->reclaim_work);
}
- atomic_long_sub(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_sub(nr_entries, &nr_swap_pages);
}
static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
@@ -1401,7 +1429,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
unsigned long end = offset + nr_entries - 1;
void (*swap_slot_free_notify)(struct block_device *, unsigned long);
- zswap_invalidate(si->type, offset, nr_entries);
+ if (!swap_is_vswap(si))
+ zswap_invalidate(si->type, offset, nr_entries);
if (si->flags & SWP_BLKDEV)
swap_slot_free_notify =
@@ -1420,7 +1449,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
* only after the above cleanups are done.
*/
smp_wmb();
- atomic_long_add(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_add(nr_entries, &nr_swap_pages);
swap_usage_sub(si, nr_entries);
}
@@ -1808,6 +1838,68 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
return err;
}
+/*
+ * Virtual swap is unbounded for a zswap-capable memcg: vswap charges the
+ * physical backing, not the allocation, so the swap.max walk would starve
+ * anon reclaim. swap.max is still enforced when the backing is charged.
+ */
+bool mem_cgroup_can_vswap(struct mem_cgroup *memcg)
+{
+ return vswap_is_enabled() && zswap_is_enabled() &&
+ (mem_cgroup_disabled() || do_memsw_account());
+}
+
+static bool vswap_alloc(struct folio *folio)
+{
+ unsigned int order = folio_order(folio);
+ struct swap_cluster_info *ci;
+ struct obj_cgroup *objcg;
+ unsigned long offset;
+ bool may_zswap;
+
+ if (!vswap_is_enabled() || !zswap_is_enabled())
+ return false;
+
+ /*
+ * If zswap will not take the folio, writeout has to find a physical
+ * slot anyway. We are just incurring indirection overhead
+ * unnecessarily.
+ */
+ objcg = get_obj_cgroup_from_folio(folio);
+ may_zswap = !objcg || obj_cgroup_may_zswap(objcg);
+ if (objcg)
+ obj_cgroup_put(objcg);
+ if (!may_zswap)
+ return false;
+
+ local_lock(&percpu_vswap_cluster.lock);
+ offset = this_cpu_read(percpu_vswap_cluster.offset[order]);
+
+ if (offset != SWAP_ENTRY_INVALID) {
+ ci = swap_cluster_lock(vswap_si, offset);
+ if (ci && cluster_is_usable(ci, order)) {
+ if (cluster_is_empty(ci))
+ offset = cluster_offset(vswap_si, ci);
+ alloc_swap_scan_cluster(vswap_si, ci, folio, offset);
+ } else if (ci) {
+ swap_cluster_unlock(ci);
+ }
+ }
+
+ if (!folio_test_swapcache(folio))
+ cluster_alloc_swap_entry(vswap_si, folio);
+
+ if (folio_test_swapcache(folio)) {
+ /* alloc_swap_scan_cluster updated percpu offset already */
+ local_unlock(&percpu_vswap_cluster.lock);
+ return true;
+ }
+
+ this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID);
+ local_unlock(&percpu_vswap_cluster.lock);
+ return false;
+}
+
/**
* folio_alloc_swap - allocate swap space for a folio
* @folio: folio we want to move to swap
@@ -1846,12 +1938,16 @@ int folio_alloc_swap(struct folio *folio)
}
}
+ if (vswap_alloc(folio))
+ goto done;
+
again:
local_lock(&percpu_swap_cluster.lock);
if (!swap_alloc_fast(folio))
swap_alloc_slow(folio);
local_unlock(&percpu_swap_cluster.lock);
+done:
if (!order && unlikely(!folio_test_swapcache(folio))) {
if (swap_sync_discard())
goto again;
@@ -1869,7 +1965,7 @@ int folio_alloc_swap(struct folio *folio)
return 0;
failed:
- if (get_nr_swap_pages() <= 0)
+ if (get_nr_swap_pages() <= 0 && !vswap_is_enabled())
return -ENOSPC;
if (mem_cgroup_get_folio_swap_margin(folio) <= 0)
return -ENOMEM;
@@ -1877,6 +1973,73 @@ int folio_alloc_swap(struct folio *folio)
return order ? -E2BIG : -ENOMEM;
}
+/**
+ * __vswap_release_backing - release the backing of a range of vtable slots
+ * @ci: the locked vswap cluster
+ * @ci_start: first slot offset within @ci
+ * @nr: number of slots
+ *
+ * Releases the backing of each slot in [@ci_start, @ci_start + @nr).
+ * Clears the zero marks if set.
+ *
+ * Context: caller must hold @ci->lock.
+ */
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int ci_off;
+ unsigned long vt;
+
+ lockdep_assert_held(&ci->lock);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+
+ for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
+ vt = __vtable_get(ci_dyn, ci_off);
+
+ switch (vtable_type(vt)) {
+ case VSWAP_ZSWAP:
+ zswap_entry_free(vtable_to_zswap(vt));
+ break;
+ case VSWAP_NONE:
+ break;
+ default:
+ /* VSWAP_ZERO/VSWAP_FOLIO are return-only, not vtable tags */
+ break;
+ }
+
+ __vtable_set(ci_dyn, ci_off, VSWAP_NONE);
+ /* Zero-backed state lives in swap_table; clear it too. */
+ if (__swap_table_test_zero(ci, ci_off))
+ __swap_table_clear_zero(ci, ci_off);
+ }
+}
+
+/**
+ * folio_release_vswap_backing() - Drop all backing for a folio's vswap entry.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release whatever backing the folio's virtual swap slots currently hold and
+ * reset them to empty, so a fresh backing can be installed. Used when a
+ * folio's swap backend is replaced.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_vswap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
/**
* folio_dup_swap() - Increase swap count of swap entries of a folio.
* @folio: folio with swap entries bounded.
@@ -2022,6 +2185,9 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
VM_WARN_ON(ci->count < nr_pages);
+ if (swap_is_vswap(si))
+ __vswap_release_backing(ci, ci_start, nr_pages);
+
ci->count -= nr_pages;
do {
old_tb = __swap_table_get(ci, ci_off);
@@ -3187,6 +3353,7 @@ static void free_swap_cluster_info(struct swap_info_struct *si,
swap_cluster_free_table(ci);
}
spin_unlock(&ci->lock);
+ vswap_cluster_free_vtable(ci);
kfree(ci_dyn);
}
xa_destroy(&si->cluster_info_pool);
@@ -3716,6 +3883,10 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
if (err)
goto err;
+ err = vswap_cluster_alloc_vtable(ci_dyn, GFP_KERNEL);
+ if (err)
+ goto err;
+
goto setup_cluster_info;
}
diff --git a/mm/util.c b/mm/util.c
index c5ee52aede1e..bde985f242df 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -33,6 +33,7 @@
#include "internal.h"
#include "swap.h"
+#include "vswap.h"
/**
* kfree_const - conditionally free memory
@@ -970,7 +971,17 @@ int __vm_enough_memory(const struct mm_struct *mm, long pages, int cap_sys_admin
return 0;
if (sysctl_overcommit_memory == OVERCOMMIT_GUESS) {
- if (pages > totalram_pages() + total_swap_pages)
+ allowed = totalram_pages() + total_swap_pages;
+ /*
+ * Vswap is in neither term, so with no swapfile the bound
+ * would collapse to RAM alone even though zswap can absorb
+ * more. Compression swap is backed by memory, so budget a 3x
+ * compression ratio, typical of most workloads. Anything that
+ * compresses better can use OVERCOMMIT_ALWAYS.
+ */
+ if (vswap_is_enabled() && zswap_is_enabled())
+ allowed += 2 * totalram_pages();
+ if (pages > allowed)
goto error;
return 0;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 836f50814ffa..6cf689817f2e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -69,6 +69,7 @@
#include "internal.h"
#include "page_alloc.h"
#include "swap.h"
+#include "vswap.h"
#define CREATE_TRACE_POINTS
#include <trace/events/vmscan.h>
@@ -415,10 +416,12 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
if (memcg == NULL) {
/*
* For non-memcg reclaim, is there space in any swap device?
- * And under GFP_NOIO, is there enough swapcached anon to make
- * scanning anon worthwhile?
+ * vswap does not contribute to nr_swap_pages. And under
+ * GFP_NOIO, is there enough swapcached anon to make scanning
+ * anon worthwhile?
*/
- if (get_nr_swap_pages() > 0 &&
+ if ((get_nr_swap_pages() > 0 ||
+ (vswap_is_enabled() && zswap_is_enabled())) &&
!reclaimable_anon_is_low(memcg, nid, sc))
return true;
} else {
@@ -426,7 +429,7 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
* Is the memcg below its swap limit, and under GFP_NOIO does
* it have enough swapcached anon to make scanning worthwhile?
*/
- if (mem_cgroup_get_nr_swap_pages(memcg) > 0 &&
+ if (mem_cgroup_can_swap(memcg, 1) &&
!reclaimable_anon_is_low(memcg, nid, sc))
return true;
}
@@ -1603,7 +1606,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
activate_locked:
/* Not a candidate for swapping, so reclaim swap space. */
if (folio_test_swapcache(folio) &&
- (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio)))
+ ((mem_cgroup_swap_full(folio) && !is_vswap_entry(folio->swap)) ||
+ folio_test_mlocked(folio)))
folio_free_swap(folio);
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
if (!folio_test_mlocked(folio)) {
@@ -2760,7 +2764,7 @@ static bool can_age_anon_pages(struct lruvec *lruvec,
struct scan_control *sc)
{
/* Aging the anon LRU is valuable if swap is present: */
- if (total_swap_pages > 0)
+ if (total_swap_pages > 0 || (vswap_is_enabled() && zswap_is_enabled()))
return true;
/* Also valuable if anon pages can be demoted: */
@@ -2853,7 +2857,7 @@ static int get_swappiness(struct lruvec *lruvec, struct scan_control *sc)
return 0;
if (!can_demote(pgdat->node_id, sc, memcg) &&
- mem_cgroup_get_nr_swap_pages(memcg) < MIN_LRU_BATCH)
+ !mem_cgroup_can_swap(memcg, MIN_LRU_BATCH))
return 0;
return swappiness;
diff --git a/mm/vswap.h b/mm/vswap.h
index 16395f357955..5334c77b6b84 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -11,8 +11,22 @@
#include <linux/swap.h>
#include "swap.h"
+struct zswap_entry;
+
+/*
+ * VSWAP_ZERO and VSWAP_FOLIO are return-only values synthesized from
+ * swap_table state; the rest are stored in the vtable per slot.
+ */
+enum vswap_backing_type {
+ VSWAP_NONE = 0,
+ VSWAP_ZSWAP = 1,
+ VSWAP_ZERO,
+ VSWAP_FOLIO,
+};
+
#ifdef CONFIG_SWAP
+#include "swap_table.h"
DECLARE_STATIC_KEY_FALSE(vswap_key);
/*
@@ -29,6 +43,177 @@ static inline bool is_vswap_entry(swp_entry_t entry)
return swap_is_vswap(__swap_entry_to_info(entry));
}
+/*
+ * Virtual table entry encoding for vswap clusters.
+ *
+ * Each entry in ci_dyn->virtual_table stores the backing type and
+ * pointer for a virtual swap slot. Tag in low 3 bits, payload in
+ * upper 61 bits.
+ *
+ * NONE: |----- 0000 ------|000| - no separate backend pointer
+ * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ *
+ * Pointer payloads (ZSWAP) are stored directly with the tag OR'd into the
+ * low bits (kernel pointers are >= 8-byte aligned, same approach as xarray).
+ *
+ * vtable[i] = NONE does not by itself mean "free". The swap_table entry
+ * and the per-slot zero flag carry the rest of the state. The full
+ * per-slot state table is:
+ *
+ * vtable[i] | swap_table[i] | zero | meaning
+ * ----------+---------------+-------+--------------------------------
+ * NONE | NULL | clear | truly free / unbacked
+ * NONE | PFN | clear | folio cached, no backing
+ * NONE | shadow | clear | evicted, no backing: data lost
+ * NONE | * | set | zero-backed; cached if PFN set
+ * ZSWAP | PFN | clear | folio cached + zswap entry
+ * ZSWAP | shadow / NULL | clear | evicted, only in zswap
+ *
+ * Locking: a slot's vtable entry (the vswap entry's backend) is only
+ * stable while the caller owns and holds the lock on that entry's swap
+ * cache folio. The cluster lock (ci_dyn->ci.lock) only makes an individual
+ * vtable read atomic, and by itself does not give the caller the right to
+ * change the backend. A backend read without the folio lock is
+ * best-effort and must be re-validated under the folio lock before
+ * being acted on.
+ *
+ * Zero-backed slots use the swap_table per-slot zero flag (same as
+ * direct-mapped physical swap), via __swap_table_test_zero() and friends,
+ * which fall back to ci->zero_bitmap where the flag does not fit. Cached
+ * folios are read out of the swap_table PFN entry; there is no separate FOLIO
+ * vtable type because the folio pointer would duplicate that PFN and
+ * would go stale on folio migration / split.
+ */
+
+#define VTABLE_TAG_BITS 3
+#define VTABLE_TAG_MASK ((1UL << VTABLE_TAG_BITS) - 1)
+
+static inline enum vswap_backing_type vtable_type(unsigned long vt)
+{
+ return vt & VTABLE_TAG_MASK;
+}
+
+static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_ZSWAP);
+ return (struct zswap_entry *)(vt & ~VTABLE_TAG_MASK);
+}
+
+/* Virtual table accessors */
+
+static inline unsigned long __vtable_get(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ return atomic_long_read(&ci_dyn->virtual_table[off]);
+}
+
+static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off, unsigned long vt)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ atomic_long_set(&ci_dyn->virtual_table[off], vt);
+}
+
+/**
+ * vswap_lock_cluster - look up and lock the vswap cluster for an entry
+ * @entry: the virtual swap entry
+ * @voff: out param, receives @entry's slot offset within the cluster
+ *
+ * Return: the locked vswap cluster, or NULL if @entry has no live cluster.
+ */
+static inline struct swap_cluster_info_dynamic *
+vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
+{
+ struct swap_cluster_info *ci;
+
+ ci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));
+ if (!ci)
+ return NULL;
+ *voff = swp_cluster_offset(entry);
+ return container_of(ci, struct swap_cluster_info_dynamic, ci);
+}
+
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr);
+
+/**
+ * vswap_zswap_store - record a zswap entry as the backing for a vswap entry.
+ * @entry: the vswap entry
+ * @ze: the zswap entry now holding @entry's compressed data
+ *
+ * Releases @entry's previous backing, and sets the zswap entry @ze as the new
+ * backing.
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ */
+static inline void vswap_zswap_store(swp_entry_t entry,
+ struct zswap_entry *ze)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ __vswap_release_backing(&ci_dyn->ci, voff, 1);
+ __vtable_set(ci_dyn, voff, (unsigned long)ze | VSWAP_ZSWAP);
+ swap_cluster_unlock(&ci_dyn->ci);
+}
+
+/**
+ * vswap_zswap_load - return the zswap entry backing a vswap entry
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing zswap entry, or NULL if @entry is not zswap-backed.
+ */
+static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return NULL;
+ vt = __vtable_get(ci_dyn, voff);
+ swap_cluster_unlock(&ci_dyn->ci);
+
+ if (vtable_type(vt) != VSWAP_ZSWAP)
+ return NULL;
+ return vtable_to_zswap(vt);
+}
+
+void folio_release_vswap_backing(struct folio *folio);
+
+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn,
+ gfp_t gfp)
+{
+ ci_dyn->virtual_table = kcalloc(SWAPFILE_CLUSTER,
+ sizeof(*ci_dyn->virtual_table), gfp);
+ return ci_dyn->virtual_table ? 0 : -ENOMEM;
+}
+
+static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ kfree(ci_dyn->virtual_table);
+ ci_dyn->virtual_table = NULL;
+}
+
+#else /* !CONFIG_SWAP */
+
+static inline bool vswap_is_enabled(void)
+{
+ return false;
+}
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return false;
+}
+
#endif /* CONFIG_SWAP */
#endif /* _MM_VSWAP_H */
diff --git a/mm/workingset.c b/mm/workingset.c
index 8412f4840ae3..fe80cd50f3e9 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -523,7 +523,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset,
workingset_size += lruvec_page_state(eviction_lruvec,
NR_INACTIVE_FILE);
}
- if (mem_cgroup_get_nr_swap_pages(eviction_memcg) > 0) {
+ if (mem_cgroup_can_swap(eviction_memcg, 1)) {
workingset_size += lruvec_page_state(eviction_lruvec,
NR_ACTIVE_ANON);
if (file) {
diff --git a/mm/zswap.c b/mm/zswap.c
index 73728b510ff6..3466c80ac188 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -39,6 +39,7 @@
#include <linux/zsmalloc.h>
#include "swap.h"
+#include "vswap.h"
#include "internal.h"
/*********************************
@@ -257,6 +258,25 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
return zswap_tree(swp_type(swp), swp_offset(swp));
}
+static struct zswap_entry *zswap_entry_load(swp_entry_t swp)
+{
+ if (is_vswap_entry(swp))
+ return vswap_zswap_load(swp);
+ return xa_load(swap_zswap_tree(swp), swp_offset(swp));
+}
+
+static struct zswap_entry *zswap_entry_store(swp_entry_t swp,
+ struct zswap_entry *entry)
+{
+ if (is_vswap_entry(swp)) {
+ vswap_zswap_store(swp, entry);
+ return NULL;
+ }
+
+ return xa_store(swap_zswap_tree(swp), swp_offset(swp), entry,
+ GFP_KERNEL);
+}
+
#define zswap_pool_debug(msg, p) \
pr_debug("%s pool %s\n", msg, (p)->tfm_name)
@@ -774,7 +794,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
* Carries out the common pattern of freeing an entry's zsmalloc allocation,
* freeing the entry itself, and decrementing the number of stored pages.
*/
-static void zswap_entry_free(struct zswap_entry *entry)
+void zswap_entry_free(struct zswap_entry *entry)
{
struct zswap_pool *pool = zswap_entry_pool(entry);
@@ -1226,6 +1246,9 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
+ if (vswap_is_enabled())
+ return 0;
+
/*
* The shrinker resumes swap writeback, which will enter block
* and may enter fs. XXX: Harmonize with vmscan.c __GFP_FS
@@ -1308,6 +1331,8 @@ static struct shrinker *zswap_alloc_shrinker(void)
* Return: 0 if at least one entry was written back, -EAGAIN if entries
* were scanned but none could be written back, or -ENOENT if @memcg has
* writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
+ *
+ * Also returns -ENOENT when vswap is enabled.
*/
static int shrink_memcg(struct mem_cgroup *memcg)
{
@@ -1316,6 +1341,9 @@ static int shrink_memcg(struct mem_cgroup *memcg)
if (!mem_cgroup_zswap_writeback_enabled(memcg))
return -ENOENT;
+ if (vswap_is_enabled())
+ return -ENOENT;
+
/*
* Skip zombies because their LRUs are reparented and we would be
* reclaiming from the parent instead of the dead memcg.
@@ -1344,6 +1372,13 @@ static void shrink_worker(struct work_struct *w)
int ret, failures = 0, attempts = 0;
unsigned long thr;
+ /*
+ * When vswap is enabled, zswap entries are almost all vswap backed,
+ * with no slot to write back to.
+ */
+ if (vswap_is_enabled())
+ return;
+
/* Reclaim down to the accept threshold */
thr = zswap_accept_thr_pages();
@@ -1447,15 +1482,13 @@ static bool zswap_store_page(struct folio *folio, long index,
goto compress_failed;
/*
- * Set pool_idx before the xa_store() below publishes the entry, or a
+ * Set pool_idx before the store below publishes the entry, or a
* concurrent reader could resolve a stale pool_idx left by slab reuse
* to an unrelated live pool.
*/
entry->pool_idx = pool->idx;
- old = xa_store(swap_zswap_tree(page_swpentry),
- swp_offset(page_swpentry),
- entry, GFP_KERNEL);
+ old = zswap_entry_store(page_swpentry, entry);
if (xa_is_err(old)) {
int err = xa_err(old);
@@ -1523,7 +1556,7 @@ bool zswap_store(struct folio *folio)
struct mem_cgroup *memcg = NULL;
struct zswap_pool *pool;
bool ret = false;
- long index;
+ long index = 0;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1576,14 +1609,21 @@ bool zswap_store(struct folio *folio)
if (!ret && zswap_pool_reached_full)
queue_work(shrink_wq, &zswap_shrink_work);
check_old:
+ if (ret)
+ return ret;
+
/*
* If the zswap store fails or zswap is disabled, we must invalidate
* the possibly stale entries which were previously stored at the
* offsets corresponding to each page of the folio. Otherwise,
* writeback could overwrite the new data in the swapfile.
*/
- if (!ret)
+ if (is_vswap_entry(swp)) {
+ if (index > 0)
+ folio_release_vswap_backing(folio);
+ } else {
zswap_invalidate(swp_type(swp), swp_offset(swp), nr_pages);
+ }
return ret;
}
@@ -1634,8 +1674,7 @@ static bool zswap_is_present(swp_entry_t entry, unsigned int nr)
int zswap_load(struct folio *folio)
{
swp_entry_t swp = folio->swap;
- pgoff_t offset = swp_offset(swp);
- struct xarray *tree = swap_zswap_tree(swp);
+ struct swap_info_struct *si = __swap_entry_to_info(swp);
struct zswap_entry *entry;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
@@ -1659,7 +1698,7 @@ int zswap_load(struct folio *folio)
return -ENOENT;
}
- entry = xa_load(tree, offset);
+ entry = zswap_entry_load(swp);
if (!entry)
return -ENOENT;
@@ -1682,8 +1721,13 @@ int zswap_load(struct folio *folio)
* compression work.
*/
folio_mark_dirty(folio);
- xa_erase(tree, offset);
- zswap_entry_free(entry);
+
+ if (swap_is_vswap(si)) {
+ folio_release_vswap_backing(folio);
+ } else {
+ xa_erase(swap_zswap_tree(swp), swp_offset(swp));
+ zswap_entry_free(entry);
+ }
folio_unlock(folio);
return 0;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 03/11] mm, swap: prepare the swap IO path for vswap
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
2026-09-18 18:02 ` [PATCH v5 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
2026-09-18 18:02 ` [PATCH v5 02/11] mm, swap: support zswap and zero-filled swap pages as vswap backends Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 04/11] mm, swap: support physical swap as a vswap backend Nhat Pham
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
In preparation for adding a physical swap backend for vswap, make the
swap IO path able to submit IO for a swap entry other than folio->swap.
The swap IO path derives the target device and sector from folio->swap.
For a vswap folio backed by a physical slot that entry is virtual, so it
identifies neither the backing device nor the sector to submit IO
against.
Compute the sector from an explicit entry (swap_folio_sector becomes
swap_entry_sector), thread that entry through swap_add_folio,
__swap_writeout and ops->can_merge, and stash it in swap_iocb so the
submit and completion paths address the IO from it rather than from
folio->swap.
This lets the batching path serve both vswap entries (backed by a
physical slot) and physical entries mapped directly into PTEs.
ops->can_merge changes anchor with it: swap_bdev_can_merge() compares
sectors and swap_fs_can_merge() byte offsets, and both now measure from the
batch head plus the accumulated length instead of from the previous folio
plus its size. The two agree at every step, since a batch only grows
through merges that passed the same test. The blkg comparison is unchanged
and still anchors on the last folio in the batch.
All callers pass folio->swap, so there is no functional change.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
include/linux/swap.h | 2 +-
include/linux/swap_ops.h | 9 ++++---
mm/page_io.c | 54 +++++++++++++++++++---------------------
mm/swap.h | 3 ++-
mm/swapfile.c | 6 ++---
mm/zswap.c | 2 +-
6 files changed, 38 insertions(+), 38 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index abf658f7861f..5ab050b2457c 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -409,7 +409,7 @@ extern int __swap_count(swp_entry_t entry);
extern bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry);
extern int swp_swapcount(swp_entry_t entry);
extern struct swap_info_struct *get_swap_device(swp_entry_t entry);
-sector_t swap_folio_sector(struct folio *folio);
+sector_t swap_entry_sector(swp_entry_t entry);
/*
* If there is an existing swap slot reference (swap entry) and the caller
diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h
index 57ac6c703f68..223c84548bde 100644
--- a/include/linux/swap_ops.h
+++ b/include/linux/swap_ops.h
@@ -12,6 +12,7 @@ struct swap_iocb {
struct bio_vec bvecs[SWAP_CLUSTER_MAX];
int nr_bvecs;
int len;
+ swp_entry_t entry; /* first slot in the batch; addresses the IO */
};
struct swap_io_ctx {
@@ -30,15 +31,15 @@ struct swap_io_ctx {
struct swap_ops {
unsigned int flags;
- bool (*can_merge)(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw);
+ bool (*can_merge)(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw);
void (*submit_write)(struct swap_io_ctx *ctx);
void (*submit_read)(struct swap_io_ctx *ctx);
};
void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter);
-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw);
+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw);
int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops);
#endif /* _MM_SWAP_OPS_H */
diff --git a/mm/page_io.c b/mm/page_io.c
index 2e7fb335eb01..632683232622 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -265,7 +265,7 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
return AOP_WRITEPAGE_ACTIVATE;
}
- __swap_writeout(ctx, folio);
+ __swap_writeout(ctx, folio, folio->swap);
return 0;
out_unlock:
folio_unlock(folio);
@@ -336,24 +336,22 @@ int sio_pool_init(void)
}
static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,
- int rw)
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
- struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1];
- struct folio *prev_folio = bvec_folio(last_bv);
- size_t prev_folio_size = folio_size(prev_folio);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
if (ctx->sis != sis)
return false;
- return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw);
+ return sis->ops->can_merge(folio, phys, ctx->sio, rw);
}
-static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
+static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
struct swap_iocb *sio = ctx->sio;
- if (sio && !swap_can_merge(ctx, folio, rw)) {
+ if (sio && !swap_can_merge(ctx, folio, phys, rw)) {
if (rw == WRITE)
swap_write_submit(ctx);
else
@@ -366,6 +364,7 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
ctx->sio = sio = mempool_alloc(sio_pool, GFP_NOIO);
sio->nr_bvecs = 0;
sio->len = 0;
+ sio->entry = phys;
}
bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
sio->len += folio_size(folio);
@@ -386,7 +385,8 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
}
}
-void __swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
+void __swap_writeout(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys)
{
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
@@ -402,7 +402,7 @@ void __swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
folio_start_writeback(folio);
folio_unlock(folio);
- swap_add_folio(ctx, folio, WRITE);
+ swap_add_folio(ctx, folio, phys, WRITE);
}
/*
@@ -506,7 +506,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
- swap_add_folio(ctx, folio, READ);
+ swap_add_folio(ctx, folio, folio->swap, READ);
finish:
if (workingset) {
@@ -538,8 +538,6 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)
bool failed = ret != sio->len;
if (failed) {
- struct folio *folio = bvec_folio(&sio->bvecs[0]);
-
/*
* In the case of swap-over-nfs, this can be a temporary failure
* if the system has limited memory for allocating transmit
@@ -547,7 +545,7 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)
* folio_rotate_reclaimable but rate-limit the messages.
*/
pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
- ret, swap_dev_pos(folio->swap));
+ ret, swap_dev_pos(sio->entry));
}
swap_write_end(sio, failed);
@@ -619,7 +617,7 @@ static void swap_bdev_submit_write(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_WRITE | REQ_SWAP);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
bio_associate_blkg_from_folio(bio, bio_first_folio_all(bio));
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
@@ -639,7 +637,7 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_READ);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
/*
@@ -657,13 +655,14 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
}
}
-static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+static bool swap_bdev_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- if (swap_folio_sector(folio) !=
- swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT))
+ if (swap_entry_sector(phys) !=
+ swap_entry_sector(sio->entry) + (sio->len >> SECTOR_SHIFT))
return false;
- if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
+ if (rw == WRITE && !folio_blkg_can_merge(folio,
+ bvec_folio(&sio->bvecs[sio->nr_bvecs - 1])))
return false;
return true;
}
@@ -679,7 +678,7 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)
struct swap_iocb *sio = ctx->sio;
init_sync_kiocb(&sio->iocb, ctx->sis->swap_file);
- sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap);
+ sio->iocb.ki_pos = swap_dev_pos(sio->entry);
if (rw == WRITE)
sio->iocb.ki_complete = swap_fs_write_complete;
else
@@ -690,11 +689,10 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)
}
EXPORT_SYMBOL_GPL(swap_fs_prepare_rw);
-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- return swap_dev_pos(folio->swap) ==
- swap_dev_pos(prev_folio->swap) + prev_folio_size;
+ return swap_dev_pos(phys) == swap_dev_pos(sio->entry) + sio->len;
}
EXPORT_SYMBOL_GPL(swap_fs_can_merge);
diff --git a/mm/swap.h b/mm/swap.h
index 81e47dc36a02..df323d5e8da8 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -320,7 +320,8 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio);
void swap_read_submit(struct swap_io_ctx *ctx);
void swap_write_submit(struct swap_io_ctx *ctx);
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
-void __swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
+void __swap_writeout(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys);
/* linux/mm/swap_state.c */
extern struct address_space swap_space __read_mostly;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 67a2399cf1ae..af6a162c4450 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -342,14 +342,14 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)
BUG();
}
-sector_t swap_folio_sector(struct folio *folio)
+sector_t swap_entry_sector(swp_entry_t entry)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(entry);
struct swap_extent *se;
sector_t sector;
pgoff_t offset;
- offset = swp_offset(folio->swap);
+ offset = swp_offset(entry);
se = offset_to_swap_extent(sis, offset);
sector = se->start_block + (offset - se->start_page);
return sector << (PAGE_SHIFT - 9);
diff --git a/mm/zswap.c b/mm/zswap.c
index 3466c80ac188..c2430dbfc653 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1094,7 +1094,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
folio_set_reclaim(folio);
/* start writeback */
- __swap_writeout(&ctx, folio);
+ __swap_writeout(&ctx, folio, folio->swap);
swap_write_submit(&ctx);
out:
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 04/11] mm, swap: support physical swap as a vswap backend
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (2 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 03/11] mm, swap: prepare the swap IO path for vswap Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 05/11] mm, swap: enable THP swapin for vswap entries Nhat Pham
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Add physical swap as a backend for the virtual swap layer. When zswap
declines a page, the swapout path allocates a physical slot on demand
for swap out.
Each vswap entry's physical slot is tracked via a pointer-tagged
swap_table entry on the physical cluster (an rmap back to the vswap
entry).
Physical readahead scans a whole offset window and would trip over
these rmap slots, so __swap_cache_add_check() now skips
swp_tb_is_pointer() entries. Nothing is lost: a backing slot is
faulted through its owning vswap entry, never through the physical
offset.
swapoff reads each vswap entry back through its rmap slot before freeing
the physical slot. A failed read leaves the folio not uptodate, so drop
it instead of marking it dirty: dirtying would write uninitialised memory
out to swap, and the loss is now reported as SIGBUS on the next fault
rather than silently returning stale data.
If zswap is disabled at the host level or for the folio's cgroup, the
folio still gets a physical swap slot, bypassing vswap and mapping the
slot directly into the PTEs. In practice the swapfile backend is only
reached when zswap declines the folio at swap_writeout() time, most often
because the pool is full.
The machinery is in place, but its main consumer is not. Zswap writeback
to physical swap is added in a following patch. Reclaim of physical slots
backing cache-only vswap entries follows it.
Suggested-by: Kairui Song <kasong@tencent.com>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/memory.c | 13 +-
mm/page_io.c | 41 ++++--
mm/swap_state.c | 6 +-
mm/swap_table.h | 42 +++++-
mm/swapfile.c | 352 +++++++++++++++++++++++++++++++++++++++++++-----
mm/vmscan.c | 2 +-
mm/vswap.h | 198 ++++++++++++++++++++++++++-
mm/zswap.c | 2 +-
8 files changed, 593 insertions(+), 63 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 73ebc59d2b03..e9e05e31c4f8 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4713,8 +4713,11 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
{
if (!folio_test_swapcache(folio))
return false;
- /* A vswap entry holds no physical slot, so keeping it saves no IO. */
- if (is_vswap_entry(folio->swap))
+ /*
+ * Non-swapfile backends cannot be reused for future swapouts.
+ * Free the swap slot unless backed by contiguous physical swap.
+ */
+ if (!folio_phys_swap_backed(folio))
return true;
/*
* Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap
@@ -4722,7 +4725,7 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
* are fast, and meanwhile, swap cache pinning the slot deferring the
* release of metadata or fragmentation is a more critical issue.
*/
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, folio->swap, SWP_SYNCHRONOUS_IO))
return true;
if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
folio_test_mlocked(folio))
@@ -5035,7 +5038,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
swap_update_readahead(folio, vma, vmf->address);
if (!folio) {
/* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, entry, SWP_SYNCHRONOUS_IO))
folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
thp_swapin_suitable_orders(vmf) | BIT(0),
vmf, NULL, 0);
@@ -5200,7 +5203,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
*/
exclusive = true;
} else if (exclusive && folio_test_writeback(folio) &&
- data_race(si->flags & SWP_STABLE_WRITES)) {
+ swap_entry_backend_has_flag(si, entry, SWP_STABLE_WRITES)) {
/*
* This is tricky: not all swap backends support
* concurrent page modifications while under writeback.
diff --git a/mm/page_io.c b/mm/page_io.c
index 632683232622..4858953fc522 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -209,6 +209,7 @@ static void swap_zeromap_folio_clear(struct folio *folio)
*/
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
{
+ swp_entry_t phys;
int ret = 0;
if (folio_free_swap(folio))
@@ -241,8 +242,14 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
*/
swap_zeromap_folio_clear(folio);
+ /*
+ * For vswap: release stale non-swapfile backings (e.g. ZSWAP from a
+ * previous swapout cycle) so zswap_store or folio_realloc_swap
+ * starts on clean slots. Contiguous PHYS backing is preserved for
+ * reuse by folio_realloc_swap.
+ */
if (is_vswap_entry(folio->swap))
- folio_release_vswap_backing(folio);
+ folio_release_non_phys_swap_backing(folio);
if (zswap_store(folio)) {
count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
@@ -257,12 +264,15 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
}
rcu_read_unlock();
- /*
- * A vswap folio has no physical slot to write to, so keep it dirty.
- */
+ /* zswap declined it, so it needs a physical slot. */
if (is_vswap_entry(folio->swap)) {
- folio_mark_dirty(folio);
- return AOP_WRITEPAGE_ACTIVATE;
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ folio_mark_dirty(folio);
+ return AOP_WRITEPAGE_ACTIVATE;
+ }
+ __swap_writeout(ctx, folio, phys);
+ return 0;
}
__swap_writeout(ctx, folio, folio->swap);
@@ -475,6 +485,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
bool workingset = folio_test_workingset(folio);
unsigned long pflags;
bool in_thrashing;
+ swp_entry_t phys;
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -499,14 +510,24 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
if (zswap_load(folio) != -ENOENT)
goto finish;
- if (unlikely(swap_is_vswap(sis))) {
- folio_unlock(folio);
- goto finish;
+ /*
+ * Resolve the physical slot to read from. A vswap entry keeps
+ * folio->swap virtual, so map it to its physical backing; a folio with
+ * no backing has nothing to read.
+ */
+ if (swap_is_vswap(sis)) {
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val) {
+ folio_unlock(folio);
+ goto finish;
+ }
+ } else {
+ phys = folio->swap;
}
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
- swap_add_folio(ctx, folio, folio->swap, READ);
+ swap_add_folio(ctx, folio, phys, READ);
finish:
if (workingset) {
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 54e4fefdab95..657622cfd7f1 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -177,6 +177,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
return -ENOENT;
ci_off = swp_cluster_offset(targ_entry);
old_tb = __swap_table_get(ci, ci_off);
+ /* Physical readahead can hit a vswap-backing rmap slot; skip it. */
+ if (swp_tb_is_pointer(old_tb))
+ return -ENOENT;
if (swp_tb_is_folio(old_tb))
return -EEXIST;
if (!__swp_tb_get_count(old_tb))
@@ -201,7 +204,8 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
- if (unlikely(swp_tb_is_folio(old_tb) ||
+ if (unlikely(swp_tb_is_pointer(old_tb) ||
+ swp_tb_is_folio(old_tb) ||
!__swp_tb_get_count(old_tb) ||
is_zero != __swap_table_test_zero(ci, ci_off) ||
(memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
diff --git a/mm/swap_table.h b/mm/swap_table.h
index 3d64d1629882..7d094694ec37 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -4,6 +4,7 @@
#include <linux/rcupdate.h>
#include <linux/atomic.h>
+#include <linux/swapops.h>
#include "swap.h"
extern struct swap_info_struct *vswap_si;
@@ -30,7 +31,7 @@ struct swap_memcg_table {
* NULL: |---------------- 0 ---------------| - Free slot
* Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot
* PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot
- * Pointer: |----------- Pointer ----------|100| - (Unused)
+ * Pointer: |-------- vswap offset --------|100| - vswap rmap
* Bad: |------------- 1 -------------|1000| - Bad slot
*
* COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,
@@ -51,9 +52,8 @@ struct swap_memcg_table {
* - PFN: Swap slot is in use, and cached. Memcg info is recorded on the page
* struct.
*
- * - Pointer: Unused yet. `0b100` is reserved for potential pointer usage
- * because only the lower three bits can be used as a marker for 8 bytes
- * aligned pointers.
+ * - Pointer: Reverse map from a physical slot to the vswap entry that owns
+ * it. See the layout below.
*
* - Bad: Swap slot is reserved, protects swap header or holes on swap devices.
*/
@@ -393,4 +393,38 @@ static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
}
#endif
+/*
+ * Pointer-tagged swap table entry: rmap for vswap-backing physical slots.
+ *
+ * On physical clusters, a Pointer-tagged entry stores the offset of the
+ * vswap entry that owns this physical slot (the reverse map). Only the
+ * offset is stored; the swap type is implicit (always vswap_si->type,
+ * since there is exactly one vswap device).
+ *
+ * Pointer: |---- vswap offset ----|100|
+ */
+#define SWP_TB_PTR_MARK_BITS 3
+#define SWP_TB_PTR_MARK 0b100UL
+#define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1)
+#define SWP_RMAP_ENTRY_MASK (~SWP_TB_PTR_MARK_MASK)
+
+static inline bool swp_tb_is_pointer(unsigned long swp_tb)
+{
+ return (swp_tb & SWP_TB_PTR_MARK_MASK) == SWP_TB_PTR_MARK;
+}
+
+static inline unsigned long swp_entry_to_swp_tb_ptr(swp_entry_t entry)
+{
+ return (swp_offset(entry) << SWP_TB_PTR_MARK_BITS) | SWP_TB_PTR_MARK;
+}
+
+static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb)
+{
+ unsigned long offset;
+
+ VM_WARN_ON(!swp_tb_is_pointer(swp_tb));
+ offset = (swp_tb & SWP_RMAP_ENTRY_MASK) >> SWP_TB_PTR_MARK_BITS;
+ return swp_entry(vswap_si->type, offset);
+}
+
#endif
diff --git a/mm/swapfile.c b/mm/swapfile.c
index af6a162c4450..13d2ae60fe4c 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -248,7 +248,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
need_reclaim = ((flags & TTRS_ANYWAY) ||
((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) ||
((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) &&
- !is_vswap_entry(folio->swap)));
+ folio_phys_swap_backed(folio)));
if (!need_reclaim || !folio_swapcache_freeable(folio))
goto out_unlock;
@@ -962,6 +962,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
{
unsigned int order;
unsigned long nr_pages;
+ swp_entry_t vswap_entry, v;
+ unsigned int i;
lockdep_assert_held(&ci->lock);
@@ -981,8 +983,26 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
order = folio_order(folio);
nr_pages = 1 << order;
swap_cluster_assert_empty(ci, ci_off, nr_pages, false);
- __swap_cache_add_folio(ci, folio, swp_entry(si->type,
- ci_off + cluster_offset(si, ci)));
+ if (folio_test_swapcache(folio)) {
+ /*
+ * Folio already in the swap cache: we are allocating
+ * physical backing for its vswap entry. Point each
+ * physical slot back at its own vswap entry
+ * (Pointer-tagged rmap).
+ */
+ VM_WARN_ON(!is_vswap_entry(folio->swap));
+ vswap_entry = folio->swap;
+ for (i = 0; i < nr_pages; i++) {
+ v = vswap_entry;
+ v.val += i;
+ __swap_table_set(ci, ci_off + i,
+ swp_entry_to_swp_tb_ptr(v));
+ }
+ } else {
+ __swap_cache_add_folio(ci, folio,
+ swp_entry(si->type,
+ ci_off + cluster_offset(si, ci)));
+ }
} else if (IS_ENABLED(CONFIG_HIBERNATION)) {
order = 0;
nr_pages = 1;
@@ -1478,12 +1498,14 @@ static bool get_swap_device_info(struct swap_info_struct *si)
* Fast path try to get swap entries with specified order from current
* CPU's swap entry pool (a cluster).
*/
-static bool swap_alloc_fast(struct folio *folio)
+static swp_entry_t swap_alloc_fast(struct folio *folio)
{
unsigned int order = folio_order(folio);
struct swap_cluster_info *ci;
struct swap_info_struct *si;
- unsigned int offset;
+ unsigned long offset, found = 0;
+
+ lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
/*
* Once allocated, swap_info_struct will never be completely freed,
@@ -1492,25 +1514,28 @@ static bool swap_alloc_fast(struct folio *folio)
si = this_cpu_read(percpu_swap_cluster.si[order]);
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
if (!si || !offset || !get_swap_device_info(si))
- return false;
+ return (swp_entry_t){};
ci = swap_cluster_lock(si, offset);
if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset);
} else if (ci) {
swap_cluster_unlock(ci);
}
put_swap_device(si);
- return folio_test_swapcache(folio);
+ if (found)
+ return swp_entry(si->type, found);
+ return (swp_entry_t){};
}
/* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static swp_entry_t swap_alloc_slow(struct folio *folio)
{
struct swap_info_struct *si, *next;
+ unsigned long found;
spin_lock(&swap_avail_lock);
start_over:
@@ -1519,12 +1544,12 @@ static void swap_alloc_slow(struct folio *folio)
plist_requeue(&si->avail_list, &swap_avail_head);
spin_unlock(&swap_avail_lock);
if (get_swap_device_info(si)) {
- cluster_alloc_swap_entry(si, folio);
+ found = cluster_alloc_swap_entry(si, folio);
put_swap_device(si);
- if (folio_test_swapcache(folio))
- return;
+ if (found)
+ return swp_entry(si->type, found);
if (folio_test_large(folio))
- return;
+ return (swp_entry_t){};
}
spin_lock(&swap_avail_lock);
@@ -1542,6 +1567,7 @@ static void swap_alloc_slow(struct folio *folio)
goto start_over;
}
spin_unlock(&swap_avail_lock);
+ return (swp_entry_t){};
}
/*
@@ -1900,6 +1926,23 @@ static bool vswap_alloc(struct folio *folio)
return false;
}
+static swp_entry_t folio_alloc_phys_swap(struct folio *folio)
+{
+ swp_entry_t entry;
+
+again:
+ local_lock(&percpu_swap_cluster.lock);
+ entry = swap_alloc_fast(folio);
+ if (!entry.val)
+ entry = swap_alloc_slow(folio);
+ local_unlock(&percpu_swap_cluster.lock);
+
+ if (!entry.val && !folio_order(folio) && swap_sync_discard())
+ goto again;
+
+ return entry;
+}
+
/**
* folio_alloc_swap - allocate swap space for a folio
* @folio: folio we want to move to swap
@@ -1938,20 +1981,8 @@ int folio_alloc_swap(struct folio *folio)
}
}
- if (vswap_alloc(folio))
- goto done;
-
-again:
- local_lock(&percpu_swap_cluster.lock);
- if (!swap_alloc_fast(folio))
- swap_alloc_slow(folio);
- local_unlock(&percpu_swap_cluster.lock);
-
-done:
- if (!order && unlikely(!folio_test_swapcache(folio))) {
- if (swap_sync_discard())
- goto again;
- }
+ if (!vswap_alloc(folio))
+ folio_alloc_phys_swap(folio);
/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
if (unlikely(mem_cgroup_try_charge_swap(folio))) {
@@ -1973,6 +2004,11 @@ int folio_alloc_swap(struct folio *folio)
return order ? -E2BIG : -ENOMEM;
}
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages);
+
/**
* __vswap_release_backing - release the backing of a range of vtable slots
* @ci: the locked vswap cluster
@@ -1988,8 +2024,11 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_start, unsigned int nr)
{
struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_info_struct *psi;
+ unsigned long phys_off_start = 0, phys_off_end = 0;
unsigned int ci_off;
unsigned long vt;
+ swp_entry_t phys_first = {};
lockdep_assert_held(&ci->lock);
ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
@@ -1997,7 +2036,30 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
vt = __vtable_get(ci_dyn, ci_off);
+ /* The free helper takes one contiguous run within one cluster. */
+ if (phys_off_start != phys_off_end &&
+ (vtable_type(vt) != VSWAP_SWAPFILE ||
+ swp_type(vtable_to_phys(vt)) != swp_type(phys_first) ||
+ swp_offset(vtable_to_phys(vt)) != phys_off_end ||
+ phys_off_end % SWAPFILE_CLUSTER == 0)) {
+ psi = __swap_entry_to_info(phys_first);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(phys_first),
+ phys_off_start % SWAPFILE_CLUSTER,
+ phys_off_end - phys_off_start);
+ phys_off_start = phys_off_end = 0;
+ }
+
switch (vtable_type(vt)) {
+ case VSWAP_SWAPFILE:
+ if (phys_off_start == phys_off_end) {
+ phys_first = vtable_to_phys(vt);
+ phys_off_start = swp_offset(phys_first);
+ phys_off_end = phys_off_start + 1;
+ } else {
+ phys_off_end++;
+ }
+ break;
case VSWAP_ZSWAP:
zswap_entry_free(vtable_to_zswap(vt));
break;
@@ -2013,6 +2075,14 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
if (__swap_table_test_zero(ci, ci_off))
__swap_table_clear_zero(ci, ci_off);
}
+
+ if (phys_off_start != phys_off_end) {
+ psi = __swap_entry_to_info(phys_first);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(phys_first),
+ phys_off_start % SWAPFILE_CLUSTER,
+ phys_off_end - phys_off_start);
+ }
}
/**
@@ -2040,6 +2110,103 @@ void folio_release_vswap_backing(struct folio *folio)
spin_unlock(&ci->lock);
}
+/**
+ * folio_release_non_phys_swap_backing() - Drop a folio's non-physical vswap backing.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release the zswap backing recorded for @folio's virtual swap entry,
+ * leaving the slots empty so the writeout path can install fresh physical
+ * backing. Does nothing when the entry is already backed by physical
+ * swapfile slots, which are kept for reuse, or when it has no backing
+ * beyond the swap cache folio itself.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_non_phys_swap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+ unsigned long vt;
+ enum vswap_backing_type type;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ /*
+ * A folio's slots cannot mix swapfile with other backends, except
+ * mid-backend-change, which always starts from slot 0.
+ */
+ vt = __vtable_get(ci_dyn, voff);
+ type = vtable_type(vt);
+
+ if (type == VSWAP_SWAPFILE || type == VSWAP_NONE) {
+ spin_unlock(&ci->lock);
+ return;
+ }
+
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
+/**
+ * folio_realloc_swap() - Back a virtual swap folio with a physical swap slot.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Ensure @folio's virtual swap entry has physical (swapfile) backing,
+ * allocating a physical slot on demand if it has none. If @folio is
+ * already physically backed, the existing physical entry is returned
+ * unchanged.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ * Return: The physical swap entry now backing @folio, or an empty entry
+ * (.val == 0) on failure.
+ */
+swp_entry_t folio_realloc_swap(struct folio *folio)
+{
+ swp_entry_t vswap_entry = folio->swap;
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ swp_entry_t phys_entry = {};
+ swp_entry_t pe;
+ int i, nr = folio_nr_pages(folio);
+
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON(!is_vswap_entry(vswap_entry));
+
+ phys_entry = vswap_to_phys(vswap_entry);
+ if (phys_entry.val)
+ return phys_entry;
+
+ phys_entry = folio_alloc_phys_swap(folio);
+ if (!phys_entry.val)
+ return (swp_entry_t){};
+
+ voff = swp_cluster_offset(vswap_entry);
+
+ ci = __swap_entry_to_cluster(vswap_entry);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ spin_lock(&ci->lock);
+ /*
+ * Install PHYS backing without freeing any prior contents of the
+ * vtable. Releasing the old backing is the caller's job.
+ */
+ for (i = 0; i < nr; i++) {
+ pe.val = phys_entry.val + i;
+ __vtable_set(ci_dyn, voff + i, vtable_mk_phys(pe));
+ }
+ spin_unlock(&ci->lock);
+
+ return phys_entry;
+}
+
/**
* folio_dup_swap() - Increase swap count of swap entries of a folio.
* @folio: folio with swap entries bounded.
@@ -2169,6 +2336,47 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)
return ERR_PTR(-EIO);
}
+/*
+ * Common tail for freeing swap slots: device-level accounting
+ * and cluster list management.
+ */
+static void __swap_cluster_finish_free(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ lockdep_assert_held(&ci->lock);
+ swap_range_free(si, cluster_offset(si, ci) + ci_start, nr_pages);
+ swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
+
+ if (!ci->count)
+ free_cluster(si, ci);
+ else
+ partial_free_cluster(si, ci);
+}
+
+/*
+ * Free physical swap slots that were backing vswap entries (Pointer-tagged).
+ */
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ unsigned int ci_off;
+
+ spin_lock_nested(&pci->lock, SINGLE_DEPTH_NESTING);
+ VM_WARN_ON(pci->count < nr_pages);
+ pci->count -= nr_pages;
+ for (ci_off = ci_start; ci_off < ci_start + nr_pages; ci_off++) {
+ __swap_table_set(pci, ci_off, null_to_swp_tb());
+ if (!SWAP_TABLE_HAS_ZEROFLAG)
+ __swap_table_clear_zero(pci, ci_off);
+ }
+ __swap_cluster_finish_free(psi, pci, ci_start, nr_pages);
+ swap_cluster_unlock(pci);
+}
+
/*
* Free a set of swap slots after their swap count dropped to zero, or will be
* zero after putting the last ref (saves one __swap_cluster_put_entry call).
@@ -2180,7 +2388,6 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned long old_tb;
unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
- unsigned long ci_head = cluster_offset(si, ci);
unsigned int batch_off = ci_off;
VM_WARN_ON(ci->count < nr_pages);
@@ -2218,13 +2425,7 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
if (batch_id)
mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
- swap_range_free(si, ci_head + ci_start, nr_pages);
- swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
-
- if (!ci->count)
- free_cluster(si, ci);
- else
- partial_free_cluster(si, ci);
+ __swap_cluster_finish_free(si, ci, ci_start, nr_pages);
}
int __swap_count(swp_entry_t entry)
@@ -3024,19 +3225,94 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
static int try_to_unuse(unsigned int type)
{
+ struct mempolicy *mpol = get_task_policy(current);
struct mm_struct *prev_mm;
struct mm_struct *mm;
struct list_head *p;
int retval = 0;
struct swap_info_struct *si = swap_info[type];
struct folio *folio;
- swp_entry_t entry;
- unsigned int i;
+ struct swap_io_ctx ctx;
+ swp_entry_t entry, vswap_entry, phys;
+ unsigned long swp_tb;
+ unsigned int i, j;
if (!swap_usage_in_pages(si))
goto success;
retry:
+ /*
+ * Free vswap-backing slots (Pointer-tagged) first. Walk physical
+ * clusters, read the vswap entry from the rmap, ensure the data
+ * is in the swap cache, and transition PHYS to FOLIO. Freeing the
+ * physical backing is enough, so no page table walk is needed.
+ */
+ i = 0;
+ while (vswap_is_enabled() &&
+ swap_usage_in_pages(si) &&
+ !signal_pending(current) &&
+ (i = find_next_to_unuse(si, i)) != 0) {
+ swp_tb = swap_table_lookup(swp_entry(si->type, i));
+ if (!swp_tb_is_pointer(swp_tb))
+ continue;
+
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio) {
+ folio = swap_cache_alloc_folio(vswap_entry,
+ GFP_HIGHUSER_MOVABLE,
+ BIT(0), NULL, mpol,
+ NO_INTERLEAVE_INDEX);
+ if (IS_ERR(folio)) {
+ if (PTR_ERR(folio) == -ENOMEM)
+ return -ENOMEM;
+ continue;
+ }
+ ctx = (struct swap_io_ctx){};
+ swap_read_folio(&ctx, folio);
+ swap_read_submit(&ctx);
+ }
+ folio_lock(folio);
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ /*
+ * Re-validate under folio lock: rmap holds folio->swap + j
+ * for some j in [0, nr_pages). Check folio->swap still maps
+ * to the contiguous physical run that includes our slot i.
+ */
+ j = vswap_entry.val - folio->swap.val;
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val || swp_type(phys) != type ||
+ swp_offset(phys) + j != i) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ folio_wait_writeback(folio);
+ folio_release_vswap_backing(folio);
+ /*
+ * Drop a folio whose read failed rather than dirtying
+ * uninitialised memory; the next fault finds no backing and
+ * gets SIGBUS.
+ */
+ if (unlikely(!folio_test_uptodate(folio)))
+ swap_cache_del_folio(folio);
+ else
+ folio_mark_dirty(folio);
+ folio_unlock(folio);
+ folio_put(folio);
+ }
+
+ if (!swap_usage_in_pages(si))
+ goto success;
+
retval = shmem_unuse(type);
if (retval)
return retval;
@@ -3079,6 +3355,8 @@ static int try_to_unuse(unsigned int type)
(i = find_next_to_unuse(si, i)) != 0) {
entry = swp_entry(type, i);
+
+ /* Pointer-tagged rmap slots have no folio; the pre-pass took them. */
folio = swap_cache_get_folio(entry);
if (!folio)
continue;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6cf689817f2e..2c9cc5c20ea9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1606,7 +1606,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
activate_locked:
/* Not a candidate for swapping, so reclaim swap space. */
if (folio_test_swapcache(folio) &&
- ((mem_cgroup_swap_full(folio) && !is_vswap_entry(folio->swap)) ||
+ ((mem_cgroup_swap_full(folio) && folio_phys_swap_backed(folio)) ||
folio_test_mlocked(folio)))
folio_free_swap(folio);
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
diff --git a/mm/vswap.h b/mm/vswap.h
index 5334c77b6b84..6f952b2591d1 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -20,6 +20,7 @@ struct zswap_entry;
enum vswap_backing_type {
VSWAP_NONE = 0,
VSWAP_ZSWAP = 1,
+ VSWAP_SWAPFILE = 2,
VSWAP_ZERO,
VSWAP_FOLIO,
};
@@ -50,11 +51,15 @@ static inline bool is_vswap_entry(swp_entry_t entry)
* pointer for a virtual swap slot. Tag in low 3 bits, payload in
* upper 61 bits.
*
- * NONE: |----- 0000 ------|000| - no separate backend pointer
- * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ * NONE: |----- 0000 ------|000| - no separate backend pointer
+ * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ * SWAPFILE: |- type:5,off:56 -|010| - on a physical swapfile
*
- * Pointer payloads (ZSWAP) are stored directly with the tag OR'd into the
- * low bits (kernel pointers are >= 8-byte aligned, same approach as xarray).
+ * SWAPFILE packs swp_type in the top MAX_SWAPFILES_SHIFT bits and swp_offset in
+ * the middle VTABLE_PHYS_OFF_BITS bits, both above the tag, so the type is
+ * not shifted off the word. Pointer payloads (ZSWAP) are stored directly with
+ * the tag OR'd into the low bits (kernel pointers are >= 8-byte aligned, same
+ * approach as xarray).
*
* vtable[i] = NONE does not by itself mean "free". The swap_table entry
* and the per-slot zero flag carry the rest of the state. The full
@@ -68,6 +73,8 @@ static inline bool is_vswap_entry(swp_entry_t entry)
* NONE | * | set | zero-backed; cached if PFN set
* ZSWAP | PFN | clear | folio cached + zswap entry
* ZSWAP | shadow / NULL | clear | evicted, only in zswap
+ * SWAPFILE | PFN | clear | folio cached + physical slot
+ * SWAPFILE | shadow / NULL | clear | evicted, only on the swapfile
*
* Locking: a slot's vtable entry (the vswap entry's backend) is only
* stable while the caller owns and holds the lock on that entry's swap
@@ -93,6 +100,23 @@ static inline enum vswap_backing_type vtable_type(unsigned long vt)
return vt & VTABLE_TAG_MASK;
}
+/* swp_offset field width in a physical backend slot; layout described above. */
+#define VTABLE_PHYS_OFF_BITS (BITS_PER_LONG - VTABLE_TAG_BITS - MAX_SWAPFILES_SHIFT)
+
+static inline unsigned long vtable_mk_phys(swp_entry_t entry)
+{
+ VM_WARN_ON_ONCE(swp_offset(entry) >> VTABLE_PHYS_OFF_BITS);
+ return ((unsigned long)swp_type(entry) << (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS)) |
+ (swp_offset(entry) << VTABLE_TAG_BITS) | VSWAP_SWAPFILE;
+}
+
+static inline swp_entry_t vtable_to_phys(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_SWAPFILE);
+ return swp_entry(vt >> (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS),
+ (vt >> VTABLE_TAG_BITS) & ((1UL << VTABLE_PHYS_OFF_BITS) - 1));
+}
+
static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)
{
VM_WARN_ON(vtable_type(vt) != VSWAP_ZSWAP);
@@ -134,6 +158,33 @@ vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
return container_of(ci, struct swap_cluster_info_dynamic, ci);
}
+/**
+ * vswap_to_phys - resolve a vswap entry's physical swap backing
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing physical swp_entry_t, or the null entry (.val == 0)
+ * when @entry has no physical backing (NONE/ZSWAP/ZERO).
+ */
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return (swp_entry_t){};
+
+ vt = __vtable_get(ci_dyn, voff);
+ swap_cluster_unlock(&ci_dyn->ci);
+
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ return (swp_entry_t){};
+
+ return vtable_to_phys(vt);
+}
+
void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_start, unsigned int nr);
@@ -184,6 +235,104 @@ static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
}
void folio_release_vswap_backing(struct folio *folio);
+swp_entry_t folio_realloc_swap(struct folio *folio);
+void folio_release_non_phys_swap_backing(struct folio *folio);
+
+/*
+ * Walk nr vtable slots starting at voff in ci_dyn. Returns the prefix
+ * length of slots sharing one effective backing type. For SWAPFILE,
+ * the prefix is also restricted to contiguous offsets in the same
+ * swapfile.
+ *
+ * Effective type per slot:
+ * vtable=NONE + zero flag set -> VSWAP_ZERO
+ * vtable=NONE + swap_table PFN tag -> VSWAP_FOLIO
+ * vtable=NONE + neither -> VSWAP_NONE
+ * vtable=SWAPFILE -> VSWAP_SWAPFILE
+ * vtable=ZSWAP -> VSWAP_ZSWAP
+ *
+ * *typep returns the effective type of slot 0. Caller holds
+ * ci_dyn->ci.lock.
+ */
+static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int voff, int nr,
+ enum vswap_backing_type *typep)
+{
+ enum vswap_backing_type first_type = VSWAP_NONE;
+ enum vswap_backing_type slot_type;
+ swp_entry_t first_phys = {};
+ unsigned long vt, swap_tb;
+ int i;
+
+ lockdep_assert_held(&ci_dyn->ci.lock);
+
+ for (i = 0; i < nr; i++) {
+ vt = __vtable_get(ci_dyn, voff + i);
+ if (vtable_type(vt) == VSWAP_NONE) {
+ swap_tb = __swap_table_get(&ci_dyn->ci, voff + i);
+ if (__swap_table_test_zero(&ci_dyn->ci, voff + i))
+ slot_type = VSWAP_ZERO;
+ else if (swp_tb_is_folio(swap_tb))
+ slot_type = VSWAP_FOLIO;
+ else
+ slot_type = VSWAP_NONE;
+ } else {
+ slot_type = vtable_type(vt);
+ }
+
+ if (!i) {
+ first_type = slot_type;
+ if (first_type == VSWAP_SWAPFILE)
+ first_phys = vtable_to_phys(vt);
+ } else if (slot_type != first_type) {
+ break;
+ } else if (first_type == VSWAP_SWAPFILE &&
+ vtable_to_phys(vt).val != first_phys.val + i) {
+ break;
+ }
+ }
+
+ if (typep)
+ *typep = first_type;
+ return i;
+}
+
+static inline int vswap_check_backing(swp_entry_t entry, int nr,
+ enum vswap_backing_type *typep)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ int ret;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn) {
+ if (typep)
+ *typep = VSWAP_NONE;
+ return 0;
+ }
+ ret = __vswap_check_backing(ci_dyn, voff, nr, typep);
+ swap_cluster_unlock(&ci_dyn->ci);
+ return ret;
+}
+
+/**
+ * folio_phys_swap_backed - test whether a folio is backed by a contiguous
+ * range of physical swap slots.
+ * @folio: a swap-cache resident folio
+ *
+ * Return: %true if @folio->swap is not a vswap entry, or if these vswap
+ * entries are backed by a contiguous range of physical slots.
+ */
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ swp_entry_t entry = folio->swap;
+ int nr = folio_nr_pages(folio);
+ enum vswap_backing_type type;
+
+ return !is_vswap_entry(entry) ||
+ (vswap_check_backing(entry, nr, &type) == nr &&
+ type == VSWAP_SWAPFILE);
+}
static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn,
gfp_t gfp)
@@ -214,6 +363,47 @@ static inline bool is_vswap_entry(swp_entry_t entry)
return false;
}
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ return (swp_entry_t){};
+}
+
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ return true;
+}
+
#endif /* CONFIG_SWAP */
+/*
+ * Test a per-backend swap flag (SWP_SYNCHRONOUS_IO, SWP_STABLE_WRITES, ...)
+ * for @entry. For a vswap entry the property belongs to the current
+ * physical backing rather than vswap_si itself; resolve to the backing
+ * and test there. Returns false for zswap/zero/unbacked vswap entries
+ * as they don't have a backing bdev.
+ */
+static inline bool swap_entry_backend_has_flag(struct swap_info_struct *si,
+ swp_entry_t entry,
+ unsigned long flag)
+{
+ struct swap_info_struct *phys_si;
+ swp_entry_t phys;
+ bool has_flag;
+
+ if (!swap_is_vswap(si))
+ return data_race(si->flags & flag);
+
+ phys = vswap_to_phys(entry);
+ if (!phys.val)
+ return false;
+
+ phys_si = get_swap_device(phys);
+ if (IS_ERR_OR_NULL(phys_si))
+ return false;
+
+ has_flag = data_race(phys_si->flags & flag);
+ put_swap_device(phys_si);
+ return has_flag;
+}
+
#endif /* _MM_VSWAP_H */
diff --git a/mm/zswap.c b/mm/zswap.c
index c2430dbfc653..bbfaeac00355 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1620,7 +1620,7 @@ bool zswap_store(struct folio *folio)
*/
if (is_vswap_entry(swp)) {
if (index > 0)
- folio_release_vswap_backing(folio);
+ folio_release_non_phys_swap_backing(folio);
} else {
zswap_invalidate(swp_type(swp), swp_offset(swp), nr_pages);
}
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 05/11] mm, swap: enable THP swapin for vswap entries
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (3 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 04/11] mm, swap: support physical swap as a vswap backend Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 06/11] mm, swap: write back vswap zswap entries to physical swap Nhat Pham
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Swap a large anon folio back in as a unit when its vswap entries share a
contiguous run of physical swap slots on a synchronous IO device,
instead of always falling back to order-0 faults.
A zswap-backed or mixed-backing batch is still refused, and the fault
retries at a smaller order.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/memory.c | 5 +++--
mm/swap_state.c | 17 +++++++++++++----
mm/zswap.c | 6 +++++-
3 files changed, 21 insertions(+), 7 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index e9e05e31c4f8..e052de3b4461 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4882,9 +4882,10 @@ static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
* lack handling for such cases, so fallback to swapping in order-0
* folio.
*
- * THP swapin for vswap is not supported yet either.
+ * Vswap entries are checked later, under the cluster lock in
+ * __swap_cache_add_check().
*/
- if (is_vswap_entry(entry) || !zswap_never_enabled())
+ if (!is_vswap_entry(entry) && !zswap_never_enabled())
return 0;
/*
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 657622cfd7f1..2107d05ae8d5 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -165,6 +165,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
unsigned int ci_off, ci_end;
unsigned long old_tb;
bool is_zero;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ enum vswap_backing_type type;
+ int ret;
lockdep_assert_held(&ci->lock);
@@ -193,11 +196,17 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
return 0;
/*
- * Reject a vswap batch so swap_cache_alloc_folio falls back to
- * order 0.
+ * For a vswap entry batch, reject if the backing is not THP-amenable
+ * (e.g. uniformly ZSWAP, or mixed). The order-fallback loop in
+ * swap_cache_alloc_folio will retry with a smaller order on -EBUSY.
*/
- if (is_vswap_entry(targ_entry))
- return -EBUSY;
+ if (is_vswap_entry(targ_entry)) {
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ ret = __vswap_check_backing(ci_dyn, round_down(ci_off, nr),
+ nr, &type);
+ if (ret != nr || type == VSWAP_ZSWAP)
+ return -EBUSY;
+ }
is_zero = __swap_table_test_zero(ci, ci_off);
ci_off = round_down(ci_off, nr);
diff --git a/mm/zswap.c b/mm/zswap.c
index bbfaeac00355..3e1aa295f9dd 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1688,9 +1688,13 @@ int zswap_load(struct folio *folio)
* range on the backing device, so scan the range rather than rejecting
* it outright. The caller has pinned every slot, so zswap cannot start
* a store or a writeback into the range while we look.
+ *
+ * A vswap batch is checked when the folio enters the swap cache, and
+ * its backing cannot change after that.
*/
if (folio_test_large(folio)) {
- if (WARN_ON_ONCE(zswap_is_present(swp,
+ if (WARN_ON_ONCE(!swap_is_vswap(si) &&
+ zswap_is_present(swp,
folio_nr_pages(folio)))) {
folio_unlock(folio);
return -EIO;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 06/11] mm, swap: write back vswap zswap entries to physical swap
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (4 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 05/11] mm, swap: enable THP swapin for vswap entries Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries Nhat Pham
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Add support for writing back zswap-backed vswap entries to physical
swap. The mechanism mirrors the existing zswap writeback path, except
the backing physical slot is allocated on demand at writeback time
rather than already being pinned by the PTE.
The zswap shrinker no longer skips vswap entries, unless we are out of
physical swap space.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/zswap.c | 67 +++++++++++++++++++++++++++++++++++++-----------------
1 file changed, 46 insertions(+), 21 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 3e1aa295f9dd..56315298c291 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1025,12 +1025,13 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
static int zswap_writeback_entry(struct zswap_entry *entry,
swp_entry_t swpentry)
{
- struct xarray *tree;
pgoff_t offset = swp_offset(swpentry);
struct folio *folio;
struct mempolicy *mpol;
struct swap_info_struct *si;
struct swap_io_ctx ctx = {};
+ swp_entry_t phys = {};
+ bool is_vswap;
int ret = 0;
/* try to allocate swap cache folio */
@@ -1038,12 +1039,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (IS_ERR_OR_NULL(si))
return -ENOENT;
- /* Vswap entries have no physical backing to write to. */
- if (swap_is_vswap(si)) {
- put_swap_device(si);
- return -EINVAL;
- }
-
+ is_vswap = swap_is_vswap(si);
mpol = get_task_policy(current);
folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
@@ -1062,24 +1058,44 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*
* folio is locked, and the swapcache is now secured against
* concurrent swapping to and from the slot, and concurrent
- * swapoff so we can safely dereference the zswap tree here.
+ * swapoff so we can safely dereference the zswap tree (or vswap
+ * vtable) here.
* Verify that the swap entry hasn't been invalidated and recycled
* behind our backs, to avoid overwriting a new swap folio with
* old compressed data. Only when this is successful can the entry
* be dereferenced.
*/
- tree = swap_zswap_tree(swpentry);
- if (entry != xa_load(tree, offset)) {
+ if (entry != zswap_entry_load(swpentry)) {
ret = -ENOMEM;
goto out;
}
+ if (is_vswap) {
+ /*
+ * Allocate physical backing before decompress so a failure
+ * wastes no work.
+ */
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+
if (!zswap_decompress(entry, folio)) {
ret = -EIO;
+ /*
+ * The phys allocation above took the entry out of the vtable.
+ * Restore the zswap entry to the vtable, which also frees the
+ * allocated physical swap space.
+ */
+ if (is_vswap)
+ vswap_zswap_store(swpentry, entry);
goto out;
}
- xa_erase(tree, offset);
+ if (!is_vswap)
+ xa_erase(swap_zswap_tree(swpentry), offset);
count_vm_event(ZSWPWB);
if (entry->objcg)
@@ -1094,7 +1110,10 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
folio_set_reclaim(folio);
/* start writeback */
- __swap_writeout(&ctx, folio, folio->swap);
+ if (is_vswap)
+ __swap_writeout(&ctx, folio, phys);
+ else
+ __swap_writeout(&ctx, folio, folio->swap);
swap_write_submit(&ctx);
out:
@@ -1109,6 +1128,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*********************************
* shrinker functions
**********************************/
+/*
+ * vswap zswap entries get a physical slot allocated on demand at writeback
+ * time. Skip the shrinker when none is available.
+ */
+static bool zswap_writeback_possible(void)
+{
+ return !vswap_is_enabled() || get_nr_swap_pages() > 0;
+}
+
/*
* The dynamic shrinker is modulated by the following factors:
*
@@ -1246,7 +1274,7 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
- if (vswap_is_enabled())
+ if (!zswap_writeback_possible())
return 0;
/*
@@ -1332,7 +1360,8 @@ static struct shrinker *zswap_alloc_shrinker(void)
* were scanned but none could be written back, or -ENOENT if @memcg has
* writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
*
- * Also returns -ENOENT when vswap is enabled.
+ * Also returns -ENOENT when vswap is enabled and there is no physical
+ * swap to write back to.
*/
static int shrink_memcg(struct mem_cgroup *memcg)
{
@@ -1341,7 +1370,7 @@ static int shrink_memcg(struct mem_cgroup *memcg)
if (!mem_cgroup_zswap_writeback_enabled(memcg))
return -ENOENT;
- if (vswap_is_enabled())
+ if (!zswap_writeback_possible())
return -ENOENT;
/*
@@ -1372,11 +1401,7 @@ static void shrink_worker(struct work_struct *w)
int ret, failures = 0, attempts = 0;
unsigned long thr;
- /*
- * When vswap is enabled, zswap entries are almost all vswap backed,
- * with no slot to write back to.
- */
- if (vswap_is_enabled())
+ if (!zswap_writeback_possible())
return;
/* Reclaim down to the accept threshold */
@@ -1457,7 +1482,7 @@ static void shrink_worker(struct work_struct *w)
break;
resched:
cond_resched();
- } while (zswap_total_pages() > thr);
+ } while (zswap_total_pages() > thr && zswap_writeback_possible());
}
/*********************************
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (5 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 06/11] mm, swap: write back vswap zswap entries to physical swap Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 08/11] mm, swap: only charge physical swap entries Nhat Pham
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
When a vswap entry's swap_count drops to 0 while its folio is still in
the swap cache, the entry is cache-only and its physical slot is
redundant. swap_put_entries_direct() tries to reclaim it at put time, but
folio_put_swap() does not, and even the direct route can come back empty:
the folio may be under the writeback that allocated the slot, its trylock
may be lost, it may still be mapped, or swap_only_has_cache() may be false
on a partial run. Nothing retried afterwards, so the slot stayed pinned.
Reclaim such slots from the physical reclaim scanner, once swap is more
than half used (vm_swap_full()), to free physical capacity for new
allocations. Reclaim goes through folio_free_swap(), which ends in
folio_set_dirty(), so a reclaimed slot costs a re-swapout if the folio is
dropped again.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/swap_table.h | 12 +++--
mm/swapfile.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++
mm/vswap.h | 31 +++++++++++++
3 files changed, 156 insertions(+), 4 deletions(-)
diff --git a/mm/swap_table.h b/mm/swap_table.h
index 7d094694ec37..79f06642a553 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -31,7 +31,7 @@ struct swap_memcg_table {
* NULL: |---------------- 0 ---------------| - Free slot
* Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot
* PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot
- * Pointer: |-------- vswap offset --------|100| - vswap rmap
+ * Pointer: |C|------- vswap offset -------|100| - vswap rmap
* Bad: |------------- 1 -------------|1000| - Bad slot
*
* COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,
@@ -399,14 +399,18 @@ static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
* On physical clusters, a Pointer-tagged entry stores the offset of the
* vswap entry that owns this physical slot (the reverse map). Only the
* offset is stored; the swap type is implicit (always vswap_si->type,
- * since there is exactly one vswap device).
+ * since there is exactly one vswap device). The top bit is reserved as
+ * a cache-only flag, set when vswap swap_count drops to 0 but the folio
+ * is still in swap cache.
*
- * Pointer: |---- vswap offset ----|100|
+ * Pointer: |C|---- vswap offset ----|100|
+ * C = SWP_RMAP_CACHE_ONLY (the top bit)
*/
#define SWP_TB_PTR_MARK_BITS 3
#define SWP_TB_PTR_MARK 0b100UL
#define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1)
-#define SWP_RMAP_ENTRY_MASK (~SWP_TB_PTR_MARK_MASK)
+#define SWP_RMAP_CACHE_ONLY (1UL << (BITS_PER_LONG - 1))
+#define SWP_RMAP_ENTRY_MASK (~(SWP_RMAP_CACHE_ONLY | SWP_TB_PTR_MARK_MASK))
static inline bool swp_tb_is_pointer(unsigned long swp_tb)
{
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 13d2ae60fe4c..2efb47b4cc4f 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -142,6 +142,11 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+static void vswap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_off);
+static void vswap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr);
+
/* May return NULL on invalid type, caller must check for NULL return */
static struct swap_info_struct *swap_type_to_info(int type)
{
@@ -874,6 +879,55 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
return ret;
}
+/*
+ * Try to reclaim a Pointer-tagged physical slot backing a vswap entry.
+ * The physical cluster lock must NOT be held. Returns the backing folio's
+ * page count, negated if the slots could not be reclaimed, or 0 if the
+ * folio could not be shown to own @offset (i.e. there is a race).
+ */
+static int try_to_reclaim_vswap_backing(struct swap_info_struct *si,
+ unsigned long offset,
+ swp_entry_t vswap_entry)
+{
+ swp_entry_t phys_base;
+ struct folio *folio;
+ unsigned int i;
+ int ret;
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio)
+ return 0;
+
+ if (!folio_trylock(folio)) {
+ /* Unvalidated: the folio may not even own @offset. */
+ folio_put(folio);
+ return 0;
+ }
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return 0;
+ }
+
+ i = vswap_entry.val - folio->swap.val;
+ phys_base = vswap_to_phys(folio->swap);
+ if (!phys_base.val || swp_type(phys_base) != si->type ||
+ swp_offset(phys_base) + i != offset) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return 0;
+ }
+
+ /* The run is ours: skip it all, whether or not the free succeeds. */
+ ret = folio_nr_pages(folio);
+ if (!folio_free_swap(folio))
+ ret = -ret;
+ folio_unlock(folio);
+ folio_put(folio);
+ return ret;
+}
+
/*
* Reclaim drops the ci lock, so the cluster may become unusable (freed or
* stolen by a lower order). @usable will be set to false if that happens.
@@ -1155,6 +1209,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
long to_scan = 1;
unsigned long offset, end;
struct swap_cluster_info *ci;
+ swp_entry_t vswap_entry;
unsigned long swp_tb;
int nr_reclaim;
@@ -1179,6 +1234,19 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
offset += abs(nr_reclaim);
continue;
}
+ } else if (swp_tb_is_pointer(swp_tb) &&
+ (swp_tb & SWP_RMAP_CACHE_ONLY)) {
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+ spin_unlock(&ci->lock);
+ nr_reclaim = try_to_reclaim_vswap_backing(si, offset,
+ vswap_entry);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
+ if (nr_reclaim) {
+ offset += abs(nr_reclaim);
+ continue;
+ }
}
offset++;
}
@@ -1749,6 +1817,8 @@ static void swap_put_entries_cluster(struct swap_info_struct *si,
}
/* count will be 0 after put, slot can be reclaimed */
need_reclaim = true;
+ if (swap_is_vswap(si))
+ vswap_mark_cache_only(ci, ci_off);
}
/*
* A count != 1 or cached slot can't be freed. Put its swap
@@ -1855,6 +1925,8 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
goto failed;
}
} while (++ci_off < ci_end);
+ if (swap_is_vswap(si))
+ vswap_clear_cache_only(ci, ci_start, nr);
swap_cluster_unlock(ci);
return 0;
failed:
@@ -2004,6 +2076,51 @@ int folio_alloc_swap(struct folio *folio)
return order ? -E2BIG : -ENOMEM;
}
+static void vswap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ swp_entry_t phys;
+ unsigned long vt;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ vt = __vtable_get(ci_dyn, ci_off);
+
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_mark_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
+
+/*
+ * Clear the cache-only rmap hint for entries re-referenced from count 0 to 1
+ * (no longer reclaimable), so the physical reclaim scanner skips them.
+ */
+static void vswap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ unsigned long swp_tb, vt;
+ swp_entry_t phys;
+ unsigned int off;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ for (off = ci_start; off < ci_start + nr; off++) {
+ swp_tb = __swap_table_get(ci, off);
+ if (!swp_tb_is_folio(swp_tb) || swp_tb_get_count(swp_tb) != 1)
+ continue;
+ vt = __vtable_get(ci_dyn, off);
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ continue;
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_clear_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
+
static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
struct swap_cluster_info *pci,
unsigned int ci_start,
diff --git a/mm/vswap.h b/mm/vswap.h
index 6f952b2591d1..b79866d5999c 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -44,6 +44,37 @@ static inline bool is_vswap_entry(swp_entry_t entry)
return swap_is_vswap(__swap_entry_to_info(entry));
}
+/*
+ * Rmap cache-only helpers for physical cluster Pointer-tagged entries.
+ * SWP_RMAP_CACHE_ONLY records, inline on the physical swap_table entry,
+ * that the backing vswap entry has swap_count == 0 (swap-cache-only, so
+ * reclaimable). The physical reclaim scanner reads it directly instead of
+ * chasing the rmap into the vswap layer and paying the cluster-lookup
+ * indirection.
+ *
+ * Callers hold the vswap cluster lock, not the physical one. The rmap is
+ * only touched while the vtable holds the slot as SWAPFILE, and that
+ * window is opened and closed under the vswap cluster lock, so the
+ * allocator has finished writing the entry by then.
+ */
+static inline void swap_rmap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_protected(ci->table, true);
+ atomic_long_or(SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
+static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_protected(ci->table, true);
+ atomic_long_and(~SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
/*
* Virtual table entry encoding for vswap clusters.
*
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 08/11] mm, swap: only charge physical swap entries
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (6 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 09/11] mm, swap: add debugfs counters for vswap Nhat Pham
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Zswap-backed and zero-filled pages occupy no swap space, but were charged
against memcg->swap as though they did.
Charge memcg->swap when a vswap entry acquires physical backing rather
than when it is allocated.
This changes what the counter means and when the charge can fail:
* memory.swap.current counts only on-disk swap usage, not zswap-backed or
zero-filled pages.
* A cgroup can reclaim its anon memory even with memory.swap.max set to 0,
provided zswap is allowed for it.
* The charge can fail at writeback rather than at allocation.
swap_writeout() returns AOP_WRITEPAGE_ACTIVATE and
zswap_writeback_entry() returns -ENOMEM for a cgroup at its limit.
* The zswap shrinker skips such a cgroup rather than walking its LRUs.
Also refactor the swap memcg operations into separate get, record, charge,
uncharge and put helpers, since recording the owner and charging it no
longer happen at the same time.
Direct-mapped physical swap charging is unchanged. So is cgroup v1 memsw
accounting: the folio's memsw charge is retained across swapout
regardless of backing, and released when the entry is freed.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
.../admin-guide/cgroup-v1/memcg_test.rst | 2 +-
Documentation/admin-guide/cgroup-v2.rst | 46 +++---
include/linux/memcontrol.h | 6 +
include/linux/swap.h | 61 ++++++-
mm/memcontrol-v1.c | 10 +-
mm/memcontrol.c | 152 ++++++++++--------
mm/swapfile.c | 128 +++++++++++++--
mm/zswap.c | 29 ++--
8 files changed, 312 insertions(+), 122 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v1/memcg_test.rst b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
index d9951c319ef5..cd565626c435 100644
--- a/Documentation/admin-guide/cgroup-v1/memcg_test.rst
+++ b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
@@ -43,7 +43,7 @@ Please note that implementation details can be changed.
mem_cgroup_uncharge()
Called when a page's refcount goes down to 0.
- mem_cgroup_uncharge_swap()
+ mem_cgroup_swap_uncharge()
Called when swp_entry's refcnt goes down to 0. A charge against swap
disappears.
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 8d2603751c51..2a1cba7f01ff 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1845,16 +1845,17 @@ The following nested keys are defined.
A read-only single value file which exists on non-root
cgroups.
- The total amount of swap currently being used by the cgroup
- and its descendants.
+ The total amount of physical swap currently being used by the
+ cgroup and its descendants.
memory.swap.high
A read-write single value file which exists on non-root
cgroups. The default is "max".
- Swap usage throttle limit. If a cgroup's swap usage exceeds
- this limit, all its further allocations will be throttled to
- allow userspace to implement custom out-of-memory procedures.
+ Physical swap usage throttle limit. If a cgroup's physical
+ swap usage exceeds this limit, all its further allocations will
+ be throttled to allow userspace to implement custom
+ out-of-memory procedures.
This limit marks a point of no return for the cgroup. It is NOT
designed to manage the amount of swapping a workload does
@@ -1867,8 +1868,9 @@ The following nested keys are defined.
memory.swap.peak
A read-write single value file which exists on non-root cgroups.
- The max swap usage recorded for the cgroup and its descendants since
- the creation of the cgroup or the most recent reset for that FD.
+ The max physical swap usage recorded for the cgroup and its
+ descendants since the creation of the cgroup or the most recent
+ reset for that FD.
A write of any non-empty string to this file resets it to the
current memory usage for subsequent reads through the same
@@ -1878,8 +1880,9 @@ The following nested keys are defined.
A read-write single value file which exists on non-root
cgroups. The default is "max".
- Swap usage hard limit. If a cgroup's swap usage reaches this
- limit, anonymous memory of the cgroup will not be swapped out.
+ Physical swap usage hard limit. If a cgroup's physical swap
+ usage reaches this limit, anonymous memory of the cgroup will
+ not be swapped out to a physical swap device.
memory.swap.events
A read-only flat-keyed file which exists on non-root cgroups.
@@ -1888,22 +1891,23 @@ The following nested keys are defined.
modified event.
high
- The number of times the cgroup's swap usage was over
- the high threshold.
+ The number of times the cgroup's physical swap usage
+ was over the high threshold.
max
- The number of times the cgroup's swap usage was about
- to go over the max boundary and swap allocation
- failed.
+ The number of times the cgroup's physical swap usage
+ was about to go over the max boundary and physical
+ swap allocation failed.
fail
- The number of times swap allocation failed either
- because of running out of swap system-wide or max
- limit.
-
- When reduced under the current usage, the existing swap
- entries are reclaimed gradually and the swap usage may stay
- higher than the limit for an extended period of time. This
+ The number of times physical swap allocation failed
+ either because of running out of physical swap
+ system-wide or max limit.
+
+ When reduced under the current usage, the existing physical
+ swap entries are reclaimed gradually and the physical swap
+ usage may stay higher than the limit for an extended period of
+ time. This
reduces the impact on the workload and memory management.
memory.zswap.current
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 46bf724cae7a..4add06affefa 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1933,6 +1933,7 @@ static inline void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,
#if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush);
void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg);
@@ -1941,6 +1942,11 @@ static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
{
return true;
}
+
+static inline bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)
+{
+ return true;
+}
static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,
size_t size)
{
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 5ab050b2457c..cd22db50b44c 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -527,20 +527,49 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
#endif
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
-int __mem_cgroup_try_charge_swap(struct folio *folio);
-static inline int mem_cgroup_try_charge_swap(struct folio *folio)
+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio);
+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)
+{
+ if (mem_cgroup_disabled())
+ return NULL;
+ return __mem_cgroup_swap_get(folio);
+}
+
+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages);
+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
if (mem_cgroup_disabled())
return 0;
- return __mem_cgroup_try_charge_swap(folio);
+ return __mem_cgroup_swap_charge(memcg, nr_pages);
}
-extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
-static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg);
+static inline void mem_cgroup_swap_record(struct folio *folio,
+ struct mem_cgroup *memcg)
{
if (mem_cgroup_disabled())
return;
- __mem_cgroup_uncharge_swap(id, nr_pages);
+ __mem_cgroup_swap_record(folio, memcg);
+}
+
+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_swap_uncharge(memcg, nr_pages);
+}
+
+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages);
+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_swap_put(memcg, nr_pages);
}
long mem_cgroup_get_folio_swap_margin(struct folio *folio);
@@ -548,16 +577,32 @@ extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
bool mem_cgroup_can_swap(struct mem_cgroup *memcg, long nr_pages);
extern bool mem_cgroup_swap_full(struct folio *folio);
#else
-static inline int mem_cgroup_try_charge_swap(struct folio *folio)
+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)
+{
+ return NULL;
+}
+
+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
return 0;
}
-static inline void mem_cgroup_uncharge_swap(unsigned short id,
+static inline void mem_cgroup_swap_record(struct folio *folio,
+ struct mem_cgroup *memcg)
+{
+}
+
+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
unsigned int nr_pages)
{
}
+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+}
+
static inline long mem_cgroup_get_folio_swap_margin(struct folio *folio)
{
return PAGE_COUNTER_MAX;
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index bf2c7d53b01b..3e06a8bdf46e 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -341,6 +341,7 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci)
void memcg1_swapin(struct folio *folio)
{
struct swap_cluster_info *ci;
+ struct mem_cgroup *memcg;
unsigned long nr_pages;
unsigned short id;
@@ -372,7 +373,14 @@ void memcg1_swapin(struct folio *folio)
id = __swap_cgroup_clear(ci, swp_cluster_offset(folio->swap),
nr_pages);
swap_cluster_unlock(ci);
- mem_cgroup_uncharge_swap(id, nr_pages);
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_private_id(id);
+ if (memcg) {
+ mem_cgroup_swap_uncharge(memcg, nr_pages);
+ mem_cgroup_swap_put(memcg, nr_pages);
+ }
+ rcu_read_unlock();
}
#endif
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 63d2c9e3dbe1..bba9148b745d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5941,80 +5941,111 @@ int __init mem_cgroup_init(void)
#ifdef CONFIG_SWAP
/**
- * __mem_cgroup_try_charge_swap - try charging swap space for a folio
+ * __mem_cgroup_swap_get - pin the memcg to account a folio's swap slots to
* @folio: folio being added to swap
*
- * Try to charge @folio's memcg for the swap space at folio->swap.
+ * Pins one private ID ref per page of @folio on its memcg, or on its closest
+ * online ancestor if it has been offlined. The caller charges and records
+ * against whichever memcg is returned, so both land on the same one.
*
- * Returns 0 on success, -ENOMEM on failure.
+ * Return: the pinned memcg, or NULL if there is nothing to account. Drop the
+ * pins with __mem_cgroup_swap_put().
*/
-int __mem_cgroup_try_charge_swap(struct folio *folio)
+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio)
{
unsigned int nr_pages = folio_nr_pages(folio);
- struct swap_cluster_info *ci;
- struct page_counter *counter;
struct mem_cgroup *memcg;
struct obj_cgroup *objcg;
if (do_memsw_account())
- return 0;
+ return NULL;
objcg = folio_objcg(folio);
VM_WARN_ON_ONCE_FOLIO(!objcg, folio);
if (!objcg)
- return 0;
+ return NULL;
rcu_read_lock();
memcg = obj_cgroup_memcg(objcg);
if (!folio_test_swapcache(folio)) {
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
rcu_read_unlock();
- return 0;
+ return NULL;
}
memcg = mem_cgroup_private_id_get_online(memcg, nr_pages);
/* memcg is pined by memcg ID. */
rcu_read_unlock();
+ return memcg;
+}
+
+/**
+ * __mem_cgroup_swap_charge - charge physical swap space
+ * @memcg: the mem_cgroup to charge (may be NULL)
+ * @nr_pages: the amount of swap space to charge
+ *
+ * Return: 0 on success, -ENOMEM if memory.swap.max is exceeded.
+ */
+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ struct page_counter *counter;
+
+ if (do_memsw_account() || !memcg)
+ return 0;
+
if (!mem_cgroup_is_root(memcg) &&
!page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
memcg_memory_event(memcg, MEMCG_SWAP_MAX);
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
- mem_cgroup_private_id_put(memcg, nr_pages);
return -ENOMEM;
}
mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
+ return 0;
+}
+
+/**
+ * __mem_cgroup_swap_record - record the owner of a folio's swap slots
+ * @folio: folio being added to swap
+ * @memcg: the memcg pinned by __mem_cgroup_swap_get()
+ */
+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg)
+{
+ struct swap_cluster_info *ci;
ci = swap_cluster_get_and_lock(folio);
- __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
- mem_cgroup_private_id(memcg));
+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap),
+ folio_nr_pages(folio), mem_cgroup_private_id(memcg));
swap_cluster_unlock(ci);
-
- return 0;
}
/**
- * __mem_cgroup_uncharge_swap - uncharge swap space
- * @id: cgroup id to uncharge
+ * __mem_cgroup_swap_uncharge - uncharge physical swap space
+ * @memcg: the mem_cgroup to uncharge (may be NULL)
* @nr_pages: the amount of swap space to uncharge
*/
-void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
- struct mem_cgroup *memcg;
+ if (!memcg)
+ return;
- rcu_read_lock();
- memcg = mem_cgroup_from_private_id(id);
- if (memcg) {
- if (!mem_cgroup_is_root(memcg)) {
- if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, nr_pages);
- else
- page_counter_uncharge(&memcg->swap, nr_pages);
- }
- mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
- mem_cgroup_private_id_put(memcg, nr_pages);
+ if (!mem_cgroup_is_root(memcg)) {
+ if (do_memsw_account())
+ page_counter_uncharge(&memcg->memsw, nr_pages);
+ else
+ page_counter_uncharge(&memcg->swap, nr_pages);
}
- rcu_read_unlock();
+ mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
+}
+
+/**
+ * __mem_cgroup_swap_put - drop the private ID refs taken for swap slots
+ * @memcg: the pinned mem_cgroup
+ * @nr_pages: number of refs to drop
+ */
+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ mem_cgroup_private_id_put(memcg, nr_pages);
}
long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
@@ -6055,27 +6086,12 @@ long mem_cgroup_get_folio_swap_margin(struct folio *folio)
* @memcg: the memcg to query
* @nr_pages: the number of pages the caller wants to swap out
*
- * A vswap zswap-backed swapout needs no physical slot, so gate on the
- * swap.max headroom rather than the physical free count.
- *
* Return: true if @memcg can swap out at least @nr_pages more pages.
*/
bool mem_cgroup_can_swap(struct mem_cgroup *memcg, long nr_pages)
{
- long avail;
-
- if (mem_cgroup_can_vswap(memcg))
- return true;
-
- if (!vswap_is_enabled() || !zswap_is_enabled())
- return mem_cgroup_get_nr_swap_pages(memcg) >= nr_pages;
-
- avail = PAGE_COUNTER_MAX;
- for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
- avail = min_t(long, avail,
- READ_ONCE(memcg->swap.max) -
- page_counter_read(&memcg->swap));
- return avail >= nr_pages;
+ return mem_cgroup_can_vswap(memcg) ||
+ mem_cgroup_get_nr_swap_pages(memcg) >= nr_pages;
}
bool mem_cgroup_swap_full(struct folio *folio)
@@ -6241,8 +6257,10 @@ static struct cftype swap_files[] = {
#ifdef CONFIG_ZSWAP
/**
- * obj_cgroup_may_zswap - check if this cgroup can zswap
- * @objcg: the object cgroup
+ * mem_cgroup_may_zswap - check if this cgroup can zswap
+ * @memcg: the memcg to query
+ * @may_flush: force-flush stats for an accurate check (sleeps). Pass false
+ * from atomic contexts; the check is then best-effort.
*
* Check if the hierarchical zswap limit has been reached.
*
@@ -6252,36 +6270,38 @@ static struct cftype swap_files[] = {
* spending cycles on compression when there is already no room left
* or zswap is disabled altogether somewhere in the hierarchy.
*/
-bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)
{
- struct mem_cgroup *memcg, *original_memcg;
- bool ret = true;
-
if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
return true;
- original_memcg = get_mem_cgroup_from_objcg(objcg);
- for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
- memcg = parent_mem_cgroup(memcg)) {
+ for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
unsigned long max = READ_ONCE(memcg->zswap_max);
unsigned long pages;
if (max == PAGE_COUNTER_MAX)
continue;
- if (max == 0) {
- ret = false;
- break;
- }
+ if (max == 0)
+ return false;
/* Force flush to get accurate stats for charging */
- __mem_cgroup_flush_stats(memcg, true);
+ if (may_flush)
+ __mem_cgroup_flush_stats(memcg, true);
pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
- if (pages < max)
- continue;
- ret = false;
- break;
+ if (pages >= max)
+ return false;
}
- mem_cgroup_put(original_memcg);
+ return true;
+}
+
+bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+{
+ struct mem_cgroup *memcg;
+ bool ret;
+
+ memcg = get_mem_cgroup_from_objcg(objcg);
+ ret = mem_cgroup_may_zswap(memcg, true);
+ mem_cgroup_put(memcg);
return ret;
}
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 2efb47b4cc4f..3c3fc3b87b9b 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1944,7 +1944,8 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
bool mem_cgroup_can_vswap(struct mem_cgroup *memcg)
{
return vswap_is_enabled() && zswap_is_enabled() &&
- (mem_cgroup_disabled() || do_memsw_account());
+ (mem_cgroup_disabled() || do_memsw_account() ||
+ mem_cgroup_may_zswap(memcg, false));
}
static bool vswap_alloc(struct folio *folio)
@@ -2030,6 +2031,7 @@ static swp_entry_t folio_alloc_phys_swap(struct folio *folio)
int folio_alloc_swap(struct folio *folio)
{
unsigned int order = folio_order(folio);
+ struct mem_cgroup *memcg;
unsigned int size = 1 << order;
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -2056,10 +2058,20 @@ int folio_alloc_swap(struct folio *folio)
if (!vswap_alloc(folio))
folio_alloc_phys_swap(folio);
- /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
- if (unlikely(mem_cgroup_try_charge_swap(folio))) {
- swap_cache_del_folio(folio);
- goto failed;
+ /*
+ * Need to call this even if allocation failed, for MEMCG_SWAP_FAIL.
+ * A vswap entry has no physical swap yet, so only record the memcg.
+ * folio_realloc_swap() charges it once backing is allocated.
+ */
+ memcg = mem_cgroup_swap_get(folio);
+ if (memcg) {
+ if (!is_vswap_entry(folio->swap) &&
+ unlikely(mem_cgroup_swap_charge(memcg, size))) {
+ mem_cgroup_swap_put(memcg, size);
+ swap_cache_del_folio(folio);
+ } else {
+ mem_cgroup_swap_record(folio, memcg);
+ }
}
if (unlikely(!folio_test_swapcache(folio)))
@@ -2126,6 +2138,36 @@ static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
unsigned int ci_start,
unsigned int nr_pages);
+static void vswap_uncharge_cgroup_batch(unsigned short memcg_id,
+ unsigned int batch_nr,
+ unsigned int batch_nr_swapfile)
+{
+ struct mem_cgroup *memcg;
+ unsigned int n;
+
+ /*
+ * v1 (memsw): entries keep their memsw charge across swapout
+ * regardless of backing, so uncharge all of them. v2: only
+ * swapfile-backed entries are charged, so uncharge just those.
+ *
+ * On v1 the id is written by __memcg1_swapout() as the folio leaves the
+ * swap cache and cleared by memcg1_swapin() when it comes back, both
+ * under the cluster lock. Callers still holding a cached folio are
+ * outside that window and see @memcg_id == 0, so only the free path
+ * uncharges. On v2 the id is set when swap is allocated, so those
+ * callers do uncharge, which balances the charge folio_realloc_swap()
+ * took.
+ */
+ n = do_memsw_account() ? batch_nr : batch_nr_swapfile;
+ if (!n)
+ return;
+
+ rcu_read_lock();
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+ mem_cgroup_swap_uncharge(memcg, n);
+}
+
/**
* __vswap_release_backing - release the backing of a range of vtable slots
* @ci: the locked vswap cluster
@@ -2146,12 +2188,25 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_off;
unsigned long vt;
swp_entry_t phys_first = {};
+ unsigned short batch_id, cur_id;
+ unsigned int batch_nr = 0, batch_nr_swapfile = 0;
lockdep_assert_held(&ci->lock);
ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ batch_id = __swap_cgroup_get(ci, ci_start);
for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
vt = __vtable_get(ci_dyn, ci_off);
+ cur_id = __swap_cgroup_get(ci, ci_off);
+
+ if (cur_id != batch_id) {
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr,
+ batch_nr_swapfile);
+ batch_id = cur_id;
+ batch_nr = 0;
+ batch_nr_swapfile = 0;
+ }
+ batch_nr++;
/* The free helper takes one contiguous run within one cluster. */
if (phys_off_start != phys_off_end &&
@@ -2169,6 +2224,7 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
switch (vtable_type(vt)) {
case VSWAP_SWAPFILE:
+ batch_nr_swapfile++;
if (phys_off_start == phys_off_end) {
phys_first = vtable_to_phys(vt);
phys_off_start = swp_offset(phys_first);
@@ -2200,6 +2256,8 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
phys_off_start % SWAPFILE_CLUSTER,
phys_off_end - phys_off_start);
}
+
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr, batch_nr_swapfile);
}
/**
@@ -2289,7 +2347,10 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
swp_entry_t vswap_entry = folio->swap;
struct swap_cluster_info *ci;
struct swap_cluster_info_dynamic *ci_dyn;
+ struct mem_cgroup *memcg;
unsigned int voff;
+ unsigned long vt;
+ unsigned short memcg_id;
swp_entry_t phys_entry = {};
swp_entry_t pe;
int i, nr = folio_nr_pages(folio);
@@ -2298,18 +2359,37 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
VM_WARN_ON(!is_vswap_entry(vswap_entry));
- phys_entry = vswap_to_phys(vswap_entry);
- if (phys_entry.val)
- return phys_entry;
+ voff = swp_cluster_offset(vswap_entry);
+ ci = __swap_entry_to_cluster(vswap_entry);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+
+ spin_lock(&ci->lock);
+ vt = __vtable_get(ci_dyn, voff);
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ spin_unlock(&ci->lock);
+ return vtable_to_phys(vt);
+ }
+ memcg_id = __swap_cgroup_get(ci, voff);
+ spin_unlock(&ci->lock);
phys_entry = folio_alloc_phys_swap(folio);
if (!phys_entry.val)
return (swp_entry_t){};
- voff = swp_cluster_offset(vswap_entry);
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ if (!memcg || mem_cgroup_private_id(memcg) != memcg_id)
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+
+ if (mem_cgroup_swap_charge(memcg, nr)) {
+ __swap_cluster_free_phys_backing(__swap_entry_to_info(phys_entry),
+ __swap_entry_to_cluster(phys_entry),
+ swp_cluster_offset(phys_entry),
+ nr);
+ return (swp_entry_t){};
+ }
- ci = __swap_entry_to_cluster(vswap_entry);
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
spin_lock(&ci->lock);
/*
* Install PHYS backing without freeing any prior contents of the
@@ -2494,6 +2574,25 @@ static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
swap_cluster_unlock(pci);
}
+/*
+ * Release the cgroup accounting of a batch of freed slots. For vswap the
+ * physical swap was already uncharged by __vswap_release_backing(), so only
+ * the ID ref is left to drop.
+ */
+static void memcg_swap_free(unsigned short id, unsigned int nr, bool is_vswap)
+{
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_private_id(id);
+ if (memcg) {
+ if (!is_vswap)
+ mem_cgroup_swap_uncharge(memcg, nr);
+ mem_cgroup_swap_put(memcg, nr);
+ }
+ rcu_read_unlock();
+}
+
/*
* Free a set of swap slots after their swap count dropped to zero, or will be
* zero after putting the last ref (saves one __swap_cluster_put_entry call).
@@ -2506,10 +2605,11 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
unsigned int batch_off = ci_off;
+ bool is_vswap = swap_is_vswap(si);
VM_WARN_ON(ci->count < nr_pages);
- if (swap_is_vswap(si))
+ if (is_vswap)
__vswap_release_backing(ci, ci_start, nr_pages);
ci->count -= nr_pages;
@@ -2533,14 +2633,14 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
id_cur = __swap_cgroup_clear(ci, ci_off, 1);
if (batch_id != id_cur) {
if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ memcg_swap_free(batch_id, ci_off - batch_off, is_vswap);
batch_id = id_cur;
batch_off = ci_off;
}
} while (++ci_off < ci_end);
if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ memcg_swap_free(batch_id, ci_off - batch_off, is_vswap);
__swap_cluster_finish_free(si, ci, ci_start, nr_pages);
}
diff --git a/mm/zswap.c b/mm/zswap.c
index 56315298c291..aac09970c4d1 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1130,11 +1130,17 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
**********************************/
/*
* vswap zswap entries get a physical slot allocated on demand at writeback
- * time. Skip the shrinker when none is available.
+ * time, and that slot is charged to @memcg. Skip the shrinker when either
+ * the device or the cgroup has no room left. @memcg may be NULL to check
+ * the device alone.
*/
-static bool zswap_writeback_possible(void)
+static bool zswap_writeback_possible(struct mem_cgroup *memcg)
{
- return !vswap_is_enabled() || get_nr_swap_pages() > 0;
+ if (!vswap_is_enabled())
+ return true;
+ if (!memcg)
+ return get_nr_swap_pages() > 0;
+ return mem_cgroup_get_nr_swap_pages(memcg) > 0;
}
/*
@@ -1199,9 +1205,10 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o
*
* Temporary failures, where the same entry should be tried
* again immediately, almost never happen for this shrinker.
- * We don't do any trylocking; -ENOMEM comes closest,
- * but that's extremely rare and doesn't happen spuriously
- * either. Don't bother distinguishing this case.
+ * We don't do any trylocking; -ENOMEM comes closest, but
+ * zswap_writeback_possible() keeps the shrinker off cgroups
+ * with no physical swap headroom, and it doesn't happen
+ * spuriously either. Don't bother distinguishing this case.
*/
list_move_tail(item, &l->list);
@@ -1274,7 +1281,7 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
- if (!zswap_writeback_possible())
+ if (!zswap_writeback_possible(memcg))
return 0;
/*
@@ -1361,7 +1368,7 @@ static struct shrinker *zswap_alloc_shrinker(void)
* writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
*
* Also returns -ENOENT when vswap is enabled and there is no physical
- * swap to write back to.
+ * swap for @memcg to write back to.
*/
static int shrink_memcg(struct mem_cgroup *memcg)
{
@@ -1370,7 +1377,7 @@ static int shrink_memcg(struct mem_cgroup *memcg)
if (!mem_cgroup_zswap_writeback_enabled(memcg))
return -ENOENT;
- if (!zswap_writeback_possible())
+ if (!zswap_writeback_possible(memcg))
return -ENOENT;
/*
@@ -1401,7 +1408,7 @@ static void shrink_worker(struct work_struct *w)
int ret, failures = 0, attempts = 0;
unsigned long thr;
- if (!zswap_writeback_possible())
+ if (!zswap_writeback_possible(NULL))
return;
/* Reclaim down to the accept threshold */
@@ -1482,7 +1489,7 @@ static void shrink_worker(struct work_struct *w)
break;
resched:
cond_resched();
- } while (zswap_total_pages() > thr && zswap_writeback_possible());
+ } while (zswap_total_pages() > thr && zswap_writeback_possible(NULL));
}
/*********************************
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 09/11] mm, swap: add debugfs counters for vswap
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (7 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 08/11] mm, swap: only charge physical swap entries Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [PATCH v5 10/11] mm, swap: defer memcg_table allocation for physical swap clusters Nhat Pham
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Add /sys/kernel/debug/vswap/ with two counters:
* used: virtual swap slots (pages) currently allocated
* alloc_reject: cumulative pages that failed to get a vswap slot
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/swapfile.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 3c3fc3b87b9b..cf07d87d2301 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -7,6 +7,7 @@
*/
#include <linux/blkdev.h>
+#include <linux/debugfs.h>
#include <linux/mm.h>
#include <linux/sched/mm.h>
#include <linux/sched/task.h>
@@ -142,6 +143,7 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+static atomic_long_t vswap_alloc_reject = ATOMIC_LONG_INIT(0);
static void vswap_mark_cache_only(struct swap_cluster_info *ci,
unsigned int ci_off);
static void vswap_clear_cache_only(struct swap_cluster_info *ci,
@@ -1996,6 +1998,7 @@ static bool vswap_alloc(struct folio *folio)
this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID);
local_unlock(&percpu_vswap_cluster.lock);
+ atomic_long_add(folio_nr_pages(folio), &vswap_alloc_reject);
return false;
}
@@ -4827,9 +4830,25 @@ early_param("vswap", early_vswap);
/* vswap does no IO on its own. */
static const struct swap_ops vswap_ops = { };
+static int vswap_used_get(void *data, u64 *val)
+{
+ *val = swap_usage_in_pages(vswap_si);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(vswap_used_fops, vswap_used_get, NULL, "%llu\n");
+
+static int vswap_alloc_reject_get(void *data, u64 *val)
+{
+ *val = atomic_long_read(&vswap_alloc_reject);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(vswap_alloc_reject_fops, vswap_alloc_reject_get, NULL,
+ "%llu\n");
+
static int __init vswap_init(void)
{
struct swap_info_struct *si;
+ struct dentry *root;
unsigned long maxpages;
int err;
@@ -4881,6 +4900,12 @@ static int __init vswap_init(void)
mutex_unlock(&swapon_mutex);
vswap_si = si;
+
+ root = debugfs_create_dir("vswap", NULL);
+ debugfs_create_file("used", 0444, root, NULL, &vswap_used_fops);
+ debugfs_create_file("alloc_reject", 0444, root, NULL,
+ &vswap_alloc_reject_fops);
+
pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages);
/* Last: everything above must be visible before routing starts. */
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 10/11] mm, swap: defer memcg_table allocation for physical swap clusters
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (8 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 09/11] mm, swap: add debugfs counters for vswap Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:02 ` [RFC PATCH v5 11/11] mm, swap: back vswap clusters with a VM_SPARSE array Nhat Pham
2026-09-18 18:38 ` [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
Stop allocating a memcg table for every physical swap cluster that only
ever holds vswap backings. The table costs SWAPFILE_CLUSTER *
sizeof(unsigned short) per cluster, 1 KB per 2 MB of swap on a 64-bit
kernel with 4 KB pages. On a vswap-heavy workload, where zswap writeback
is the only consumer of physical swap, that is the common case.
Such clusters never have their memcg_table read or written: vswap-layer
charging records on the vswap cluster's table, not the physical one.
Allocate eagerly only where the table is known to be needed: every vswap
cluster, and, when vswap is off, every physical cluster, since none of its
slots is then a vswap backing. A physical cluster otherwise allocates on
its first direct-use slot, and skips entirely if it only holds vswap
backings. Hibernation slots have no folio and record no cgroup, so they do
not trigger it. That deferred allocation is on the allocator's fast path
and can fail; the allocation it serves then fails too, and the caller falls
back to another cluster.
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
---
mm/swapfile.c | 83 +++++++++++++++++++++++++++++++++++++++------------
1 file changed, 64 insertions(+), 19 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index cf07d87d2301..39d1840b0d36 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -472,7 +472,8 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci)
swap_cluster_free_count_table(table);
}
-static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
+static int swap_cluster_alloc_table(struct swap_info_struct *si,
+ struct swap_cluster_info *ci, gfp_t gfp)
{
struct swap_table *table = NULL;
struct folio *folio;
@@ -493,7 +494,14 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
return -ENOMEM;
#ifdef CONFIG_MEMCG
- if (!mem_cgroup_disabled()) {
+ /*
+ * A physical cluster under vswap may hold only vswap backings, which
+ * record their memcg on the vswap cluster's table, not this one. Such
+ * clusters defer memcg_table allocation until they hand out a slot
+ * that maps directly into the PTEs.
+ */
+ if ((!vswap_is_enabled() || swap_is_vswap(si)) &&
+ !mem_cgroup_disabled()) {
VM_WARN_ON_ONCE(ci->memcg_table);
ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
if (!ci->memcg_table) {
@@ -571,8 +579,8 @@ swap_cluster_populate(struct swap_info_struct *si,
lockdep_assert_held(&si->global_cluster_lock);
lockdep_assert_held(&ci->lock);
- if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- __GFP_NOWARN))
+ if (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ __GFP_NOWARN))
return ci;
/*
@@ -585,8 +593,8 @@ swap_cluster_populate(struct swap_info_struct *si,
spin_unlock(&si->global_cluster_lock);
local_unlock(&percpu_swap_cluster.lock);
- ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- GFP_KERNEL);
+ ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ GFP_KERNEL);
/*
* Back to atomic context. We might have migrated to a new CPU with a
@@ -863,7 +871,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
ci = cluster_info + idx;
/* Need to allocate swap table first for initial bad slot marking. */
- if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL))
+ if (!ci->count && swap_cluster_alloc_table(si, ci, GFP_KERNEL))
return -ENOMEM;
spin_lock(&ci->lock);
/* Check for duplicated bad swap slots. */
@@ -1086,7 +1094,9 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
/* Try use a new cluster for current CPU and allocate from it. */
static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
struct swap_cluster_info *ci,
- struct folio *folio, unsigned long offset)
+ struct folio *folio,
+ unsigned long offset,
+ bool *nomem)
{
unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);
@@ -1115,6 +1125,23 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
if (!ret)
continue;
}
+#ifdef CONFIG_MEMCG
+ /*
+ * Lazy-allocate memcg_table on the first direct-use slot of a
+ * physical cluster.
+ */
+ if (vswap_is_enabled() && folio &&
+ !folio_test_swapcache(folio) && !mem_cgroup_disabled() &&
+ !ci->memcg_table) {
+ ci->memcg_table = kzalloc_obj(*ci->memcg_table,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (!ci->memcg_table) {
+ if (nomem)
+ *nomem = true;
+ goto out;
+ }
+ }
+#endif
if (!__swap_cluster_alloc_entries(si, ci, folio, offset % SWAPFILE_CLUSTER))
break;
found = offset;
@@ -1124,7 +1151,15 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
break;
}
out:
- relocate_cluster(si, ci);
+ /*
+ * On a discard-capable device, relocating a cluster whose memcg_table
+ * allocation failed queues a discard for slots that were never used,
+ * which folio_alloc_phys_swap() reads as progress and retries on.
+ */
+ if (nomem && *nomem && !ci->count)
+ __free_cluster(si, ci);
+ else
+ relocate_cluster(si, ci);
swap_cluster_unlock(ci);
if (swap_is_vswap(si)) {
this_cpu_write(percpu_vswap_cluster.offset[order], next);
@@ -1145,7 +1180,13 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
bool scan_all)
{
unsigned int found = SWAP_ENTRY_INVALID;
+ bool nomem = false;
+ /*
+ * In rare cases alloc_swap_scan_cluster() can fail due to
+ * memcg_table allocation failure. Short-circuit to avoid looping
+ * over the list indefinitely.
+ */
do {
struct swap_cluster_info *ci = isolate_lock_cluster(si, list);
unsigned long offset;
@@ -1153,10 +1194,10 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
if (!ci)
break;
offset = cluster_offset(si, ci);
- found = alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset, &nomem);
if (found)
break;
- } while (scan_all);
+ } while (scan_all && !nomem);
return found;
}
@@ -1177,7 +1218,8 @@ static unsigned int vswap_alloc_cluster(struct swap_info_struct *si,
spin_lock_init(&ci_dyn->ci.lock);
INIT_LIST_HEAD(&ci_dyn->ci.list);
- if (swap_cluster_alloc_table(&ci_dyn->ci, GFP_ATOMIC | __GFP_NOWARN)) {
+ if (swap_cluster_alloc_table(si, &ci_dyn->ci,
+ GFP_ATOMIC | __GFP_NOWARN)) {
kfree(ci_dyn);
return SWAP_ENTRY_INVALID;
}
@@ -1203,7 +1245,7 @@ static unsigned int vswap_alloc_cluster(struct swap_info_struct *si,
}
offset = cluster_offset(si, ci);
- return alloc_swap_scan_cluster(si, ci, folio, offset);
+ return alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
}
static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
@@ -1310,7 +1352,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
if (cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- found = alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset,
+ NULL);
} else {
swap_cluster_unlock(ci);
}
@@ -1354,7 +1397,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
if (order < PMD_ORDER) {
/*
* Scan only one fragment cluster is good enough. Order 0
- * allocation will surely success, and large allocation
+ * allocation rarely fails, and large allocation
* failure is not critical. Scanning one cluster still
* keeps the list rotated and reclaimed (for clean swap cache).
*/
@@ -1369,7 +1412,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
/* Order 0 stealing from higher order */
for (int o = 1; o < SWAP_NR_ORDERS; o++) {
/*
- * Clusters here have at least one usable slots and can't fail order 0
+ * Clusters here have at least one usable slots and rarely fail order 0
* allocation, but reclaim may drop si->lock and race with another user.
*/
found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true);
@@ -1590,7 +1633,7 @@ static swp_entry_t swap_alloc_fast(struct folio *folio)
if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- found = alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
} else if (ci) {
swap_cluster_unlock(ci);
}
@@ -1981,7 +2024,8 @@ static bool vswap_alloc(struct folio *folio)
if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(vswap_si, ci);
- alloc_swap_scan_cluster(vswap_si, ci, folio, offset);
+ alloc_swap_scan_cluster(vswap_si, ci, folio, offset,
+ NULL);
} else if (ci) {
swap_cluster_unlock(ci);
}
@@ -2860,7 +2904,8 @@ swp_entry_t swap_alloc_hibernation_slot(int type)
if (pcp_si == si && pcp_offset) {
ci = swap_cluster_lock(si, pcp_offset);
if (cluster_is_usable(ci, 0))
- offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
+ offset = alloc_swap_scan_cluster(si, ci, NULL,
+ pcp_offset, NULL);
else
swap_cluster_unlock(ci);
}
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC PATCH v5 11/11] mm, swap: back vswap clusters with a VM_SPARSE array
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (9 preceding siblings ...)
2026-09-18 18:02 ` [PATCH v5 10/11] mm, swap: defer memcg_table allocation for physical swap clusters Nhat Pham
@ 2026-09-18 18:02 ` Nhat Pham
2026-09-18 18:38 ` [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:02 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, nphamcs, linux-mm, linux-kernel,
linux-doc, cgroups
vswap keeps its cluster_info in an xarray of individually allocated
clusters. Replace it with the VM_SPARSE vmalloc array Baoquan He
designed for xswap: one reservation at init, mapped a page of clusters
at a time as the device grows. That drops the per-cluster allocation and
the xarray nodes, and it simplifies access, because the index gives the
address. A lookup becomes arithmetic instead of an xa_load() that can
return NULL, and no cluster needs an RCU grace period to be freed, so
the NULL arm goes away in every caller along with the index and rcu_head
fields, the kfree_rcu(), CLUSTER_FLAG_DEAD and __vswap_cluster_lock().
Only the grow side is ported; there is no shrink.
That leaves swap_cluster_info_dynamic wrapping swap_cluster_info for a
single pointer, so move the virtual table into swap_cluster_info and
delete the wrapper. Both cluster arrays then have the same element type
and merge into si->cluster_info, which restores cluster_index() to
upstream's subtraction and leaves __swap_offset_to_cluster() a plain
array index. A vswap cluster ends up smaller, having lost the index and
rcu_head; a physical cluster grows by the one pointer it never uses.
A cluster is no longer destroyed when it empties. There is nothing left
to free, since it is now an element of a fixed array, so it goes onto
si->free_clusters like a physical device's cluster and waits to be
reused.
Its virtual table is still freed, and that is the bulk of it:
SWAPFILE_CLUSTER pointers, a full page at the usual layout, against a
few dozen bytes for the cluster itself. An emptied cluster gives back
almost all of what it held; only the mapping stays.
Most of this is Baoquan's code, adapted to vswap's existing cluster
layer rather than to a new device type, so I am keeping his attributions
from the original posting.
Co-developed-by: Baoquan He <hebaoquan@kylinos.cn>
Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
Signed-off-by: Nhat Pham <nphamcs@gmail.com>
Link: https://lore.kernel.org/all/20260916101929.149106-1-hebaoquan@kylinos.cn/
---
include/linux/swap.h | 5 +-
mm/swap.h | 61 +-------
mm/swap_state.c | 19 +--
mm/swap_table.h | 9 --
mm/swapfile.c | 353 ++++++++++++++++++++++++++++---------------
mm/vswap.h | 96 +++++-------
6 files changed, 280 insertions(+), 263 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index cd22db50b44c..dffdec14c407 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -164,6 +164,7 @@ static inline void mm_account_reclaimed_pages(unsigned long pages)
struct address_space;
struct sysinfo;
+struct vm_struct;
struct zone;
/*
@@ -271,7 +272,9 @@ struct swap_info_struct {
struct list_head discard_clusters; /* discard clusters list */
struct plist_node avail_list; /* entry in swap_avail_head */
const struct swap_ops *ops;
- struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */
+ struct vm_struct *cluster_info_area; /* Vswap cluster array reservation */
+ unsigned int nr_mapped_clusters; /* Mapped prefix of cluster_info */
+ struct mutex cluster_grow_lock; /* Serialize growth of the array */
};
static inline bool swap_is_vswap(struct swap_info_struct *si)
diff --git a/mm/swap.h b/mm/swap.h
index df323d5e8da8..83015ff5f390 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -64,16 +64,10 @@ struct swap_cluster_info {
#if !SWAP_TABLE_HAS_ZEROFLAG
unsigned long *zero_bitmap;
#endif
+ atomic_long_t *virtual_table; /* Backing pointers, vswap clusters only */
struct list_head list;
};
-struct swap_cluster_info_dynamic {
- struct swap_cluster_info ci;
- unsigned int index; /* for cluster_index() */
- struct rcu_head rcu;
- atomic_long_t *virtual_table; /* Backing pointers for vswap slots */
-};
-
/* All on-list cluster must have a non-zero flag. */
enum swap_cluster_flags {
CLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */
@@ -84,7 +78,6 @@ enum swap_cluster_flags {
CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
CLUSTER_FLAG_FULL,
CLUSTER_FLAG_DISCARD,
- CLUSTER_FLAG_DEAD, /* Vswap dynamic cluster pending kfree_rcu */
CLUSTER_FLAG_MAX,
};
@@ -127,17 +120,6 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
return __swap_type_to_info(swp_type(entry));
}
-/**
- * __swap_offset_to_cluster - look up the cluster holding a swap offset
- * @si: the swap device
- * @offset: the swap entry offset
- *
- * Context: A vswap cluster is freed by kfree_rcu(). Callers must hold the
- * RCU read lock, or know the cluster is pinned by an in-use entry.
- *
- * Return: the cluster, or NULL if @si is a vswap device with no cluster
- * allocated at @offset.
- */
static inline struct swap_cluster_info *__swap_offset_to_cluster(
struct swap_info_struct *si, pgoff_t offset)
{
@@ -145,13 +127,8 @@ static inline struct swap_cluster_info *__swap_offset_to_cluster(
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
VM_WARN_ON_ONCE(offset >= roundup(si->max, SWAPFILE_CLUSTER));
-
- if (swap_is_vswap(si)) {
- struct swap_cluster_info_dynamic *ci_dyn;
-
- ci_dyn = xa_load(&si->cluster_info_pool, cluster_idx);
- return ci_dyn ? &ci_dyn->ci : NULL;
- }
+ VM_WARN_ON_ONCE(swap_is_vswap(si) &&
+ cluster_idx >= READ_ONCE(si->nr_mapped_clusters));
return &si->cluster_info[cluster_idx];
}
@@ -162,32 +139,6 @@ static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entr
swp_offset(entry));
}
-static inline struct swap_cluster_info *__vswap_cluster_lock(
- struct swap_info_struct *si, unsigned long offset, bool irq)
-{
- struct swap_cluster_info *ci;
-
- rcu_read_lock();
- ci = __swap_offset_to_cluster(si, offset);
- if (ci) {
- if (irq)
- spin_lock_irq(&ci->lock);
- else
- spin_lock(&ci->lock);
-
- /* The cluster can be torn down while we wait for the lock. */
- if (ci->flags == CLUSTER_FLAG_DEAD) {
- if (irq)
- spin_unlock_irq(&ci->lock);
- else
- spin_unlock(&ci->lock);
- ci = NULL;
- }
- }
- rcu_read_unlock();
- return ci;
-}
-
static __always_inline struct swap_cluster_info *__swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset, bool irq)
{
@@ -205,9 +156,6 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
VM_WARN_ON_ONCE(!in_task());
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
- if (swap_is_vswap(si))
- return __vswap_cluster_lock(si, offset, irq);
-
ci = __swap_offset_to_cluster(si, offset);
if (irq)
spin_lock_irq(&ci->lock);
@@ -223,8 +171,7 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
*
* Context: The caller must ensure the offset is in the valid range and
* protect the swap device with reference count or locks.
- * Return: the locked cluster, or NULL if it is gone. Only a vswap device
- * can return NULL, as its clusters are allocated and freed on demand.
+ * Return: The locked cluster.
*/
static inline struct swap_cluster_info *swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 2107d05ae8d5..627fee08593c 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -165,7 +165,6 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
unsigned int ci_off, ci_end;
unsigned long old_tb;
bool is_zero;
- struct swap_cluster_info_dynamic *ci_dyn;
enum vswap_backing_type type;
int ret;
@@ -201,8 +200,7 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
* swap_cache_alloc_folio will retry with a smaller order on -EBUSY.
*/
if (is_vswap_entry(targ_entry)) {
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
- ret = __vswap_check_backing(ci_dyn, round_down(ci_off, nr),
+ ret = __vswap_check_backing(ci, round_down(ci_off, nr),
nr, &type);
if (ret != nr || type == VSWAP_ZSWAP)
return -EBUSY;
@@ -451,12 +449,9 @@ static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
entry.val = round_down(targ_entry.val, nr_pages);
/* Check if the slot and range are available, skip allocation if not */
- err = -ENOENT;
ci = swap_cluster_lock(si, offset);
- if (ci) {
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
- swap_cluster_unlock(ci);
- }
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
+ swap_cluster_unlock(ci);
if (unlikely(err))
return ERR_PTR(err);
@@ -477,13 +472,10 @@ static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
return ERR_PTR(-ENOMEM);
/* Double check the range is still not in conflict */
- err = -ENOENT;
ci = swap_cluster_lock(si, offset);
- if (ci)
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
if (unlikely(err)) {
- if (ci)
- swap_cluster_unlock(ci);
+ swap_cluster_unlock(ci);
folio_put(folio);
return ERR_PTR(err);
}
@@ -495,7 +487,6 @@ static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
vmf ? vmf->vma->vm_mm : NULL, gfp)) {
- /* The folio pins the cluster */
spin_lock(&ci->lock);
__swap_cache_do_del_folio(ci, folio, entry, shadow);
spin_unlock(&ci->lock);
diff --git a/mm/swap_table.h b/mm/swap_table.h
index 79f06642a553..7d005a943881 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -266,11 +266,6 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
return swp_tb;
}
-/*
- * Resolve @entry's cluster and read its slot, both under RCU. A vswap
- * cluster is allocated on demand and freed by kfree_rcu(), so a caller
- * starting from an entry cannot resolve it beforehand.
- */
static inline unsigned long swap_table_lookup(swp_entry_t entry)
{
struct swap_cluster_info *ci;
@@ -279,10 +274,6 @@ static inline unsigned long swap_table_lookup(swp_entry_t entry)
rcu_read_lock();
ci = __swap_entry_to_cluster(entry);
- if (!ci) {
- rcu_read_unlock();
- return null_to_swp_tb();
- }
table = rcu_dereference(ci->table);
swp_tb = table ? atomic_long_read(&table[swp_cluster_offset(entry)])
: null_to_swp_tb();
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 39d1840b0d36..d529a27fdd89 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -144,6 +144,35 @@ static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
};
static atomic_long_t vswap_alloc_reject = ATOMIC_LONG_INIT(0);
+
+/*
+ * Vswap allocates from its own device with a separate percpu cluster cache,
+ * so the allocator has two local locks to pick from.
+ */
+static void swap_percpu_cluster_lock(struct swap_info_struct *si)
+{
+ if (swap_is_vswap(si))
+ local_lock(&percpu_vswap_cluster.lock);
+ else
+ local_lock(&percpu_swap_cluster.lock);
+}
+
+static void swap_percpu_cluster_unlock(struct swap_info_struct *si)
+{
+ if (swap_is_vswap(si))
+ local_unlock(&percpu_vswap_cluster.lock);
+ else
+ local_unlock(&percpu_swap_cluster.lock);
+}
+
+static void swap_percpu_cluster_assert_held(struct swap_info_struct *si)
+{
+ if (swap_is_vswap(si))
+ lockdep_assert_held(&this_cpu_ptr(&percpu_vswap_cluster)->lock);
+ else
+ lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+}
+
static void vswap_mark_cache_only(struct swap_cluster_info *ci,
unsigned int ci_off);
static void vswap_clear_cache_only(struct swap_cluster_info *ci,
@@ -420,8 +449,6 @@ static inline bool cluster_is_usable(struct swap_cluster_info *ci, int order)
static inline unsigned int cluster_index(struct swap_info_struct *si,
struct swap_cluster_info *ci)
{
- if (swap_is_vswap(si))
- return container_of(ci, struct swap_cluster_info_dynamic, ci)->index;
return ci - si->cluster_info;
}
@@ -450,10 +477,14 @@ static void swap_cluster_free_count_table(struct swap_table *table)
swap_cluster_free_table_folio_rcu_cb);
}
-static void swap_cluster_free_table(struct swap_cluster_info *ci)
+static void swap_cluster_free_table(struct swap_info_struct *si,
+ struct swap_cluster_info *ci)
{
struct swap_table *table;
+ if (swap_is_vswap(si))
+ vswap_cluster_free_vtable(ci);
+
#ifdef CONFIG_MEMCG
kfree(ci->memcg_table);
ci->memcg_table = NULL;
@@ -515,12 +546,19 @@ static int swap_cluster_alloc_table(struct swap_info_struct *si,
VM_WARN_ON_ONCE(ci->zero_bitmap);
ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
if (!ci->zero_bitmap) {
- swap_cluster_free_table(ci);
+ swap_cluster_free_table(si, ci);
swap_cluster_free_count_table(table);
return -ENOMEM;
}
#endif
+ /* The virtual table shares the swap table's lifetime. */
+ if (swap_is_vswap(si) && vswap_cluster_alloc_vtable(ci, gfp)) {
+ swap_cluster_free_table(si, ci);
+ swap_cluster_free_count_table(table);
+ return -ENOMEM;
+ }
+
/*
* Make tables visible to cluster_is_usable() after everything is
* ready.
@@ -571,10 +609,8 @@ swap_cluster_populate(struct swap_info_struct *si,
/*
* Only cluster isolation from the allocator does table allocation.
* Swap allocator uses percpu clusters and holds the local lock.
- * vswap clusters are destroyed rather than freed to si->free_clusters.
*/
- VM_WARN_ON_ONCE(swap_is_vswap(si));
- lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+ swap_percpu_cluster_assert_held(si);
if (!(si->flags & SWP_SOLIDSTATE))
lockdep_assert_held(&si->global_cluster_lock);
lockdep_assert_held(&ci->lock);
@@ -591,7 +627,7 @@ swap_cluster_populate(struct swap_info_struct *si,
spin_unlock(&ci->lock);
if (!(si->flags & SWP_SOLIDSTATE))
spin_unlock(&si->global_cluster_lock);
- local_unlock(&percpu_swap_cluster.lock);
+ swap_percpu_cluster_unlock(si);
ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
GFP_KERNEL);
@@ -604,7 +640,7 @@ swap_cluster_populate(struct swap_info_struct *si,
* could happen with ignoring the percpu cluster is fragmentation,
* which is acceptable since this fallback and race is rare.
*/
- local_lock(&percpu_swap_cluster.lock);
+ swap_percpu_cluster_lock(si);
if (!(si->flags & SWP_SOLIDSTATE))
spin_lock(&si->global_cluster_lock);
spin_lock(&ci->lock);
@@ -652,20 +688,7 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si,
static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci)
{
swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, false);
- swap_cluster_free_table(ci);
-
- if (swap_is_vswap(si)) {
- struct swap_cluster_info_dynamic *ci_dyn;
-
- /* vswap clusters are destroyed, not returned to free_clusters. */
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
- xa_erase(&si->cluster_info_pool, ci_dyn->index);
- move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD);
- vswap_cluster_free_vtable(ci);
- kfree_rcu(ci_dyn, rcu);
- return;
- }
-
+ swap_cluster_free_table(si, ci);
move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
ci->order = 0;
}
@@ -1202,50 +1225,147 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
return found;
}
-static unsigned int vswap_alloc_cluster(struct swap_info_struct *si,
- struct folio *folio)
+/*
+ * Reserve address space for the vswap cluster array. Nothing is mapped yet,
+ * so this costs address space only, plus an eighth of it in shadow under
+ * CONFIG_KASAN_VMALLOC.
+ */
+static int vswap_reserve_cluster_array(struct swap_info_struct *si,
+ unsigned long maxpages)
+{
+ unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
+
+ mutex_init(&si->cluster_grow_lock);
+ si->cluster_info_area = get_vm_area(nr_clusters *
+ sizeof(*si->cluster_info),
+ VM_SPARSE);
+ if (!si->cluster_info_area)
+ return -ENOMEM;
+
+ si->cluster_info = si->cluster_info_area->addr;
+ return 0;
+}
+
+static void vswap_free_cluster_array(struct swap_info_struct *si)
+{
+ unsigned long addr, end;
+ struct page *page;
+
+ if (!si->cluster_info_area)
+ return;
+
+ end = round_up((unsigned long)&si->cluster_info[si->nr_mapped_clusters],
+ PAGE_SIZE);
+ for (addr = (unsigned long)si->cluster_info; addr < end;
+ addr += PAGE_SIZE) {
+ page = vmalloc_to_page((void *)addr);
+ vm_area_unmap_pages(si->cluster_info_area, addr,
+ addr + PAGE_SIZE);
+ __free_page(page);
+ }
+
+ free_vm_area(si->cluster_info_area);
+ si->cluster_info_area = NULL;
+ si->cluster_info = NULL;
+ si->nr_mapped_clusters = 0;
+}
+
+static bool vswap_can_grow(struct swap_info_struct *si)
+{
+ return READ_ONCE(si->nr_mapped_clusters) <
+ DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER);
+}
+
+/* Clusters added per growth of the vswap cluster array, one page worth. */
+#define VSWAP_GROW_CLUSTERS \
+ max_t(unsigned long, \
+ PAGE_SIZE / sizeof(struct swap_cluster_info), 16)
+
+/*
+ * Map one more page of the vswap cluster array and hand the clusters it
+ * covers to the allocator. The caller must not hold the percpu cluster
+ * lock: vm_area_map_pages() might sleep.
+ *
+ * The mapped prefix only ever grows, so the pages already backing clusters
+ * [0, si->nr_mapped_clusters) are exactly those below the page boundary
+ * above the last one. A grow whose clusters all fall inside an already
+ * mapped page maps nothing.
+ */
+static int vswap_grow_clusters(struct swap_info_struct *si)
{
- struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *ci;
- unsigned long offset;
+ unsigned int noreclaim_flags;
+ unsigned long start, end;
+ struct page *page;
+ unsigned int i, first, nr;
+ int err = -ENOSPC;
+ BUILD_BUG_ON(VSWAP_GROW_CLUSTERS *
+ sizeof(struct swap_cluster_info) > PAGE_SIZE);
VM_WARN_ON(!swap_is_vswap(si));
- ci_dyn = kzalloc_obj(*ci_dyn, GFP_ATOMIC | __GFP_NOWARN);
- if (!ci_dyn)
- return SWAP_ENTRY_INVALID;
+ /* Rechecked under the mutex, this only keeps a full device cheap. */
+ if (!vswap_can_grow(si))
+ return -ENOSPC;
- spin_lock_init(&ci_dyn->ci.lock);
- INIT_LIST_HEAD(&ci_dyn->ci.list);
+ /* Outside the mutex, so this one may still reclaim. */
+ page = alloc_page(__GFP_HIGH | __GFP_NOMEMALLOC | GFP_KERNEL |
+ __GFP_ZERO);
- if (swap_cluster_alloc_table(si, &ci_dyn->ci,
- GFP_ATOMIC | __GFP_NOWARN)) {
- kfree(ci_dyn);
- return SWAP_ENTRY_INVALID;
- }
+ mutex_lock(&si->cluster_grow_lock);
+ first = si->nr_mapped_clusters;
+ nr = min_t(unsigned int, VSWAP_GROW_CLUSTERS,
+ DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - first);
+ if (!nr)
+ goto out;
- if (vswap_cluster_alloc_vtable(ci_dyn, GFP_ATOMIC | __GFP_NOWARN)) {
- swap_cluster_free_table(&ci_dyn->ci);
- kfree(ci_dyn);
- return SWAP_ENTRY_INVALID;
- }
+ start = round_up((unsigned long)&si->cluster_info[first],
+ PAGE_SIZE);
+ end = round_up((unsigned long)&si->cluster_info[first + nr],
+ PAGE_SIZE);
- /* Lock before publishing: xa_alloc makes the cluster findable by offset. */
- ci = &ci_dyn->ci;
- spin_lock(&ci->lock);
+ if (start != end) {
+ err = -ENOMEM;
+ if (!page)
+ goto out;
+ /*
+ * vm_area_map_pages() allocates page tables with
+ * GFP_PGTABLE_KERNEL, so they carry __GFP_DIRECT_RECLAIM.
+ * A non-reclaim caller of folio_alloc_swap() would otherwise
+ * recurse back here and deadlock on the mutex it already
+ * holds. Callers already under PF_MEMALLOC do not need this,
+ * swapon does. It grants the page tables reserve access, at
+ * most three pages per grow.
+ */
+ noreclaim_flags = memalloc_noreclaim_save();
+ err = vm_area_map_pages(si->cluster_info_area, start, end,
+ &page);
+ memalloc_noreclaim_restore(noreclaim_flags);
+ if (err)
+ goto out;
+ page = NULL;
+ }
- if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn,
- XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1),
- GFP_ATOMIC | __GFP_NOWARN)) {
+ /*
+ * Publish the new clusters before they become reachable by offset.
+ * A zeroed page leaves them off-list with CLUSTER_FLAG_NONE, which
+ * is what move_cluster() expects.
+ */
+ WRITE_ONCE(si->nr_mapped_clusters, first + nr);
+ for (i = first; i < first + nr; i++) {
+ ci = &si->cluster_info[i];
+ spin_lock_init(&ci->lock);
+ INIT_LIST_HEAD(&ci->list);
+ spin_lock(&ci->lock);
+ move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
spin_unlock(&ci->lock);
- swap_cluster_free_table(&ci_dyn->ci);
- vswap_cluster_free_vtable(&ci_dyn->ci);
- kfree(ci_dyn);
- return SWAP_ENTRY_INVALID;
}
-
- offset = cluster_offset(si, ci);
- return alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
+ err = 0;
+out:
+ mutex_unlock(&si->cluster_grow_lock);
+ if (page)
+ __free_page(page);
+ return err;
}
static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
@@ -1272,8 +1392,6 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
nr_reclaim = __try_to_reclaim_swap(si, offset,
TTRS_ANYWAY);
ci = swap_cluster_lock(si, offset);
- if (!ci)
- goto next;
if (nr_reclaim) {
offset += abs(nr_reclaim);
continue;
@@ -1285,8 +1403,6 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
nr_reclaim = try_to_reclaim_vswap_backing(si, offset,
vswap_entry);
ci = swap_cluster_lock(si, offset);
- if (!ci)
- goto next;
if (nr_reclaim) {
offset += abs(nr_reclaim);
continue;
@@ -1300,7 +1416,6 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
relocate_cluster(si, ci);
swap_cluster_unlock(ci);
-next:
if (to_scan <= 0)
break;
@@ -1378,10 +1493,19 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
goto done;
}
- if (swap_is_vswap(si)) {
- found = vswap_alloc_cluster(si, folio);
- if (found)
- goto done;
+ /*
+ * Grow the vswap cluster array and let the free list scan below pick
+ * up the new clusters. Growth sleeps, so drop the percpu cluster lock
+ * across it; the scan does not care which CPU it lands back on. The
+ * list_empty() test is racy either way: a stale empty costs one page,
+ * a stale non-empty skips the grow and leaves the caller to the
+ * fragment and stealing scans below.
+ */
+ if (swap_is_vswap(si) && list_empty(&si->free_clusters) &&
+ vswap_can_grow(si)) {
+ local_unlock(&percpu_vswap_cluster.lock);
+ vswap_grow_clusters(si);
+ local_lock(&percpu_vswap_cluster.lock);
}
if (!(si->flags & SWP_PAGE_DISCARD)) {
@@ -1630,11 +1754,11 @@ static swp_entry_t swap_alloc_fast(struct folio *folio)
return (swp_entry_t){};
ci = swap_cluster_lock(si, offset);
- if (ci && cluster_is_usable(ci, order)) {
+ if (cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
found = alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
- } else if (ci) {
+ } else {
swap_cluster_unlock(ci);
}
@@ -1760,7 +1884,6 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
if (IS_ERR_OR_NULL(si))
return 0;
- /* The source PTE pins the entry, so its cluster is alive. */
ci = __swap_offset_to_cluster(si, offset);
ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
@@ -2021,12 +2144,12 @@ static bool vswap_alloc(struct folio *folio)
if (offset != SWAP_ENTRY_INVALID) {
ci = swap_cluster_lock(vswap_si, offset);
- if (ci && cluster_is_usable(ci, order)) {
+ if (cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(vswap_si, ci);
alloc_swap_scan_cluster(vswap_si, ci, folio, offset,
NULL);
- } else if (ci) {
+ } else {
swap_cluster_unlock(ci);
}
}
@@ -2138,13 +2261,11 @@ int folio_alloc_swap(struct folio *folio)
static void vswap_mark_cache_only(struct swap_cluster_info *ci,
unsigned int ci_off)
{
- struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *pci;
swp_entry_t phys;
unsigned long vt;
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
- vt = __vtable_get(ci_dyn, ci_off);
+ vt = __vtable_get(ci, ci_off);
if (vtable_type(vt) == VSWAP_SWAPFILE) {
phys = vtable_to_phys(vt);
@@ -2160,18 +2281,16 @@ static void vswap_mark_cache_only(struct swap_cluster_info *ci,
static void vswap_clear_cache_only(struct swap_cluster_info *ci,
unsigned int ci_start, int nr)
{
- struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *pci;
unsigned long swp_tb, vt;
swp_entry_t phys;
unsigned int off;
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
for (off = ci_start; off < ci_start + nr; off++) {
swp_tb = __swap_table_get(ci, off);
if (!swp_tb_is_folio(swp_tb) || swp_tb_get_count(swp_tb) != 1)
continue;
- vt = __vtable_get(ci_dyn, off);
+ vt = __vtable_get(ci, off);
if (vtable_type(vt) != VSWAP_SWAPFILE)
continue;
phys = vtable_to_phys(vt);
@@ -2229,7 +2348,6 @@ static void vswap_uncharge_cgroup_batch(unsigned short memcg_id,
void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int ci_start, unsigned int nr)
{
- struct swap_cluster_info_dynamic *ci_dyn;
struct swap_info_struct *psi;
unsigned long phys_off_start = 0, phys_off_end = 0;
unsigned int ci_off;
@@ -2239,11 +2357,10 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
unsigned int batch_nr = 0, batch_nr_swapfile = 0;
lockdep_assert_held(&ci->lock);
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
batch_id = __swap_cgroup_get(ci, ci_start);
for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
- vt = __vtable_get(ci_dyn, ci_off);
+ vt = __vtable_get(ci, ci_off);
cur_id = __swap_cgroup_get(ci, ci_off);
if (cur_id != batch_id) {
@@ -2290,7 +2407,7 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
break;
}
- __vtable_set(ci_dyn, ci_off, VSWAP_NONE);
+ __vtable_set(ci, ci_off, VSWAP_NONE);
/* Zero-backed state lives in swap_table; clear it too. */
if (__swap_table_test_zero(ci, ci_off))
__swap_table_clear_zero(ci, ci_off);
@@ -2348,14 +2465,12 @@ void folio_release_vswap_backing(struct folio *folio)
void folio_release_non_phys_swap_backing(struct folio *folio)
{
struct swap_cluster_info *ci;
- struct swap_cluster_info_dynamic *ci_dyn;
int nr = folio_nr_pages(folio);
unsigned int voff;
unsigned long vt;
enum vswap_backing_type type;
ci = __swap_entry_to_cluster(folio->swap);
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
voff = swp_cluster_offset(folio->swap);
spin_lock(&ci->lock);
@@ -2363,7 +2478,7 @@ void folio_release_non_phys_swap_backing(struct folio *folio)
* A folio's slots cannot mix swapfile with other backends, except
* mid-backend-change, which always starts from slot 0.
*/
- vt = __vtable_get(ci_dyn, voff);
+ vt = __vtable_get(ci, voff);
type = vtable_type(vt);
if (type == VSWAP_SWAPFILE || type == VSWAP_NONE) {
@@ -2393,7 +2508,6 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
{
swp_entry_t vswap_entry = folio->swap;
struct swap_cluster_info *ci;
- struct swap_cluster_info_dynamic *ci_dyn;
struct mem_cgroup *memcg;
unsigned int voff;
unsigned long vt;
@@ -2408,10 +2522,9 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
voff = swp_cluster_offset(vswap_entry);
ci = __swap_entry_to_cluster(vswap_entry);
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
spin_lock(&ci->lock);
- vt = __vtable_get(ci_dyn, voff);
+ vt = __vtable_get(ci, voff);
if (vtable_type(vt) == VSWAP_SWAPFILE) {
spin_unlock(&ci->lock);
return vtable_to_phys(vt);
@@ -2444,7 +2557,7 @@ swp_entry_t folio_realloc_swap(struct folio *folio)
*/
for (i = 0; i < nr; i++) {
pe.val = phys_entry.val + i;
- __vtable_set(ci_dyn, voff + i, vtable_mk_phys(pe));
+ __vtable_set(ci, voff + i, vtable_mk_phys(pe));
}
spin_unlock(&ci->lock);
@@ -2765,7 +2878,6 @@ static bool folio_maybe_swapped(struct folio *folio)
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
- /* Folio is locked and in swap cache, so ci->count > 0: cluster is alive. */
ci = __swap_entry_to_cluster(entry);
ci_off = swp_cluster_offset(entry);
ci_end = ci_off + folio_nr_pages(folio);
@@ -3881,25 +3993,22 @@ static void free_swap_cluster_info(struct swap_info_struct *si,
struct swap_cluster_info *cluster_info,
unsigned long maxpages)
{
- struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *ci;
- unsigned long idx;
int i, nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
if (swap_is_vswap(si)) {
- xa_for_each(&si->cluster_info_pool, idx, ci_dyn) {
- ci = &ci_dyn->ci;
+ nr_clusters = si->nr_mapped_clusters;
+ for (i = 0; i < nr_clusters; i++) {
+ ci = &si->cluster_info[i];
spin_lock(&ci->lock);
if (cluster_table_is_alloced(ci)) {
swap_cluster_assert_empty(ci, 0,
SWAPFILE_CLUSTER, true);
- swap_cluster_free_table(ci);
+ swap_cluster_free_table(si, ci);
}
spin_unlock(&ci->lock);
- vswap_cluster_free_vtable(ci);
- kfree(ci_dyn);
}
- xa_destroy(&si->cluster_info_pool);
+ vswap_free_cluster_array(si);
return;
}
@@ -3911,7 +4020,7 @@ static void free_swap_cluster_info(struct swap_info_struct *si,
spin_lock(&ci->lock);
if (cluster_table_is_alloced(ci)) {
swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, true);
- swap_cluster_free_table(ci);
+ swap_cluster_free_table(si, ci);
}
spin_unlock(&ci->lock);
}
@@ -4397,39 +4506,18 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
{
unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
struct swap_cluster_info *cluster_info = NULL;
- struct swap_cluster_info_dynamic *ci_dyn = NULL;
+ struct swap_cluster_info *ci;
int err = -ENOMEM;
unsigned long i;
- /* A vswap device uses an xarray pool instead of a static array. */
+ /* A vswap device grows its cluster array on demand. */
if (swap_is_vswap(si)) {
nr_clusters = 0;
- xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC);
-
- /*
- * Pre-allocate cluster 0 and mark slot 0 (header page)
- * as bad so the allocator never hands out page offset 0.
- */
- ci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);
- if (!ci_dyn)
- goto err;
- spin_lock_init(&ci_dyn->ci.lock);
- INIT_LIST_HEAD(&ci_dyn->ci.list);
-
- err = xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL);
- if (err) {
- kfree(ci_dyn);
- goto err;
- }
-
- err = swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false);
+ err = vswap_reserve_cluster_array(si, maxpages);
if (err)
goto err;
-
- err = vswap_cluster_alloc_vtable(ci_dyn, GFP_KERNEL);
- if (err)
- goto err;
-
+ /* Reservation owns the array; keep the tail store idempotent. */
+ cluster_info = si->cluster_info;
goto setup_cluster_info;
}
@@ -4487,7 +4575,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
}
for (i = 0; i < nr_clusters; i++) {
- struct swap_cluster_info *ci = &cluster_info[i];
+ ci = &cluster_info[i];
if (ci->count) {
ci->flags = CLUSTER_FLAG_NONFULL;
@@ -4500,8 +4588,23 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
/* Slot 0 is bad, so cluster 0 never empties. The rest of it is usable. */
if (swap_is_vswap(si)) {
- ci_dyn->ci.flags = CLUSTER_FLAG_NONFULL;
- list_add_tail(&ci_dyn->ci.list, &si->nonfull_clusters[0]);
+ err = vswap_grow_clusters(si);
+ if (err)
+ goto err;
+
+ ci = si->cluster_info;
+ spin_lock(&ci->lock);
+ move_cluster(si, ci, NULL, CLUSTER_FLAG_NONE);
+ spin_unlock(&ci->lock);
+
+ err = swap_cluster_setup_bad_slot(si, ci, 0, false);
+ if (err)
+ goto err;
+
+ spin_lock(&ci->lock);
+ move_cluster(si, ci, &si->nonfull_clusters[0],
+ CLUSTER_FLAG_NONFULL);
+ spin_unlock(&ci->lock);
}
si->cluster_info = cluster_info;
diff --git a/mm/vswap.h b/mm/vswap.h
index b79866d5999c..f8235882f3f0 100644
--- a/mm/vswap.h
+++ b/mm/vswap.h
@@ -78,7 +78,7 @@ static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci,
/*
* Virtual table entry encoding for vswap clusters.
*
- * Each entry in ci_dyn->virtual_table stores the backing type and
+ * Each entry in ci->virtual_table stores the backing type and
* pointer for a virtual swap slot. Tag in low 3 bits, payload in
* upper 61 bits.
*
@@ -109,7 +109,7 @@ static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci,
*
* Locking: a slot's vtable entry (the vswap entry's backend) is only
* stable while the caller owns and holds the lock on that entry's swap
- * cache folio. The cluster lock (ci_dyn->ci.lock) only makes an individual
+ * cache folio. The cluster lock (ci->lock) only makes an individual
* vtable read atomic, and by itself does not give the caller the right to
* change the backend. A backend read without the folio lock is
* best-effort and must be re-validated under the folio lock before
@@ -156,18 +156,18 @@ static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)
/* Virtual table accessors */
-static inline unsigned long __vtable_get(struct swap_cluster_info_dynamic *ci_dyn,
+static inline unsigned long __vtable_get(struct swap_cluster_info *ci,
unsigned int off)
{
VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
- return atomic_long_read(&ci_dyn->virtual_table[off]);
+ return atomic_long_read(&ci->virtual_table[off]);
}
-static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,
+static inline void __vtable_set(struct swap_cluster_info *ci,
unsigned int off, unsigned long vt)
{
VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
- atomic_long_set(&ci_dyn->virtual_table[off], vt);
+ atomic_long_set(&ci->virtual_table[off], vt);
}
/**
@@ -175,18 +175,13 @@ static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,
* @entry: the virtual swap entry
* @voff: out param, receives @entry's slot offset within the cluster
*
- * Return: the locked vswap cluster, or NULL if @entry has no live cluster.
+ * Return: the locked vswap cluster.
*/
-static inline struct swap_cluster_info_dynamic *
+static inline struct swap_cluster_info *
vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
{
- struct swap_cluster_info *ci;
-
- ci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));
- if (!ci)
- return NULL;
*voff = swp_cluster_offset(entry);
- return container_of(ci, struct swap_cluster_info_dynamic, ci);
+ return swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));
}
/**
@@ -199,16 +194,13 @@ vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
*/
static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
{
- struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
unsigned int voff;
unsigned long vt;
- ci_dyn = vswap_lock_cluster(entry, &voff);
- if (!ci_dyn)
- return (swp_entry_t){};
-
- vt = __vtable_get(ci_dyn, voff);
- swap_cluster_unlock(&ci_dyn->ci);
+ ci = vswap_lock_cluster(entry, &voff);
+ vt = __vtable_get(ci, voff);
+ swap_cluster_unlock(ci);
if (vtable_type(vt) != VSWAP_SWAPFILE)
return (swp_entry_t){};
@@ -232,13 +224,13 @@ void __vswap_release_backing(struct swap_cluster_info *ci,
static inline void vswap_zswap_store(swp_entry_t entry,
struct zswap_entry *ze)
{
- struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
unsigned int voff;
- ci_dyn = vswap_lock_cluster(entry, &voff);
- __vswap_release_backing(&ci_dyn->ci, voff, 1);
- __vtable_set(ci_dyn, voff, (unsigned long)ze | VSWAP_ZSWAP);
- swap_cluster_unlock(&ci_dyn->ci);
+ ci = vswap_lock_cluster(entry, &voff);
+ __vswap_release_backing(ci, voff, 1);
+ __vtable_set(ci, voff, (unsigned long)ze | VSWAP_ZSWAP);
+ swap_cluster_unlock(ci);
}
/**
@@ -250,15 +242,13 @@ static inline void vswap_zswap_store(swp_entry_t entry,
*/
static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
{
- struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
unsigned int voff;
unsigned long vt;
- ci_dyn = vswap_lock_cluster(entry, &voff);
- if (!ci_dyn)
- return NULL;
- vt = __vtable_get(ci_dyn, voff);
- swap_cluster_unlock(&ci_dyn->ci);
+ ci = vswap_lock_cluster(entry, &voff);
+ vt = __vtable_get(ci, voff);
+ swap_cluster_unlock(ci);
if (vtable_type(vt) != VSWAP_ZSWAP)
return NULL;
@@ -270,7 +260,7 @@ swp_entry_t folio_realloc_swap(struct folio *folio);
void folio_release_non_phys_swap_backing(struct folio *folio);
/*
- * Walk nr vtable slots starting at voff in ci_dyn. Returns the prefix
+ * Walk nr vtable slots starting at voff in ci. Returns the prefix
* length of slots sharing one effective backing type. For SWAPFILE,
* the prefix is also restricted to contiguous offsets in the same
* swapfile.
@@ -283,9 +273,9 @@ void folio_release_non_phys_swap_backing(struct folio *folio);
* vtable=ZSWAP -> VSWAP_ZSWAP
*
* *typep returns the effective type of slot 0. Caller holds
- * ci_dyn->ci.lock.
+ * ci->lock.
*/
-static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn,
+static inline int __vswap_check_backing(struct swap_cluster_info *ci,
unsigned int voff, int nr,
enum vswap_backing_type *typep)
{
@@ -295,13 +285,13 @@ static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn
unsigned long vt, swap_tb;
int i;
- lockdep_assert_held(&ci_dyn->ci.lock);
+ lockdep_assert_held(&ci->lock);
for (i = 0; i < nr; i++) {
- vt = __vtable_get(ci_dyn, voff + i);
+ vt = __vtable_get(ci, voff + i);
if (vtable_type(vt) == VSWAP_NONE) {
- swap_tb = __swap_table_get(&ci_dyn->ci, voff + i);
- if (__swap_table_test_zero(&ci_dyn->ci, voff + i))
+ swap_tb = __swap_table_get(ci, voff + i);
+ if (__swap_table_test_zero(ci, voff + i))
slot_type = VSWAP_ZERO;
else if (swp_tb_is_folio(swap_tb))
slot_type = VSWAP_FOLIO;
@@ -331,18 +321,13 @@ static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn
static inline int vswap_check_backing(swp_entry_t entry, int nr,
enum vswap_backing_type *typep)
{
- struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
unsigned int voff;
int ret;
- ci_dyn = vswap_lock_cluster(entry, &voff);
- if (!ci_dyn) {
- if (typep)
- *typep = VSWAP_NONE;
- return 0;
- }
- ret = __vswap_check_backing(ci_dyn, voff, nr, typep);
- swap_cluster_unlock(&ci_dyn->ci);
+ ci = vswap_lock_cluster(entry, &voff);
+ ret = __vswap_check_backing(ci, voff, nr, typep);
+ swap_cluster_unlock(ci);
return ret;
}
@@ -365,21 +350,18 @@ static inline bool folio_phys_swap_backed(struct folio *folio)
type == VSWAP_SWAPFILE);
}
-static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn,
+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info *ci,
gfp_t gfp)
{
- ci_dyn->virtual_table = kcalloc(SWAPFILE_CLUSTER,
- sizeof(*ci_dyn->virtual_table), gfp);
- return ci_dyn->virtual_table ? 0 : -ENOMEM;
+ ci->virtual_table = kcalloc(SWAPFILE_CLUSTER,
+ sizeof(*ci->virtual_table), gfp);
+ return ci->virtual_table ? 0 : -ENOMEM;
}
static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci)
{
- struct swap_cluster_info_dynamic *ci_dyn;
-
- ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
- kfree(ci_dyn->virtual_table);
- ci_dyn->virtual_table = NULL;
+ kfree(ci->virtual_table);
+ ci->virtual_table = NULL;
}
#else /* !CONFIG_SWAP */
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition)
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
` (10 preceding siblings ...)
2026-09-18 18:02 ` [RFC PATCH v5 11/11] mm, swap: back vswap clusters with a VM_SPARSE array Nhat Pham
@ 2026-09-18 18:38 ` Nhat Pham
11 siblings, 0 replies; 13+ messages in thread
From: Nhat Pham @ 2026-09-18 18:38 UTC (permalink / raw)
To: akpm
Cc: chrisl, kasong, hannes, mhocko, roman.gushchin, shakeel.butt,
yosry, david, muchun.song, shikemeng, baoquan.he, baohua,
youngjun.park, chengming.zhou, ljs, liam, vbabka, rppt, surenb,
qi.zheng, axelrasmussen, yuanchu, weixugc, riel, gourry,
haowenchao22, corbet, hughd, baolin.wang, tj, mkoutny, skhan,
kunwu.chan, kernel-team, linux-mm, linux-kernel, linux-doc,
cgroups
On Fri, Sep 18, 2026 at 11:02 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> Patch 11 is an RFC. It swaps vswap's cluster xarray for the VM_SPARSE
> vmalloc array Baoquan He designed for xswap, to show that the data
> structure and the device model are separable: moving to his is one
> self-contained patch that adds no userspace interface. Note that
> per Baoquan's commentary (see [5]), I have skipped shrink for now, only
> freeing vtable when the cluster becomes free to minimize metadata
> overhead while keeping the skeleton in the free list. I have not done
> performance testing on this patch yet (the number is from the old design),
> but I have run a suite of simple stress tests.
>
> It is adapted almost entirely from Baoquan's code (see [6]), so I have
> kept Baoquan's Co-developed-by and Signed-off-by tag.
To clarify a bit on the motivation of this exercise: Kairui and
Baoquan raised the concerns (for e.g in [1]) that if we were to go
with the xarray as a data structure first, switching it to the
VM_SPARSE vmalloc array would be difficult. I want to check this claim
by prototyping such a conversion. It is fairly straightforward and
simple to me. So it is not a valid concern to block the landing of
vswap.
If we are to actually switch to such a data structure, Baoquan should
be the one driving this, by showing that there are performance wins to
the switch :) It is his proposal, so the onus is on him to drive it to
completion. I'm happy to assist with the reviewing and testing!
[1]: https://lore.kernel.org/all/CAMgjq7AHRzBsbmGOx0s=TQFBwcn=DEU1naR443sO6x+pJd6hAg@mail.gmail.com/
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2026-09-18 18:39 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-18 18:02 [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
2026-09-18 18:02 ` [PATCH v5 01/11] mm, swap: add virtual swap device infrastructure Nhat Pham
2026-09-18 18:02 ` [PATCH v5 02/11] mm, swap: support zswap and zero-filled swap pages as vswap backends Nhat Pham
2026-09-18 18:02 ` [PATCH v5 03/11] mm, swap: prepare the swap IO path for vswap Nhat Pham
2026-09-18 18:02 ` [PATCH v5 04/11] mm, swap: support physical swap as a vswap backend Nhat Pham
2026-09-18 18:02 ` [PATCH v5 05/11] mm, swap: enable THP swapin for vswap entries Nhat Pham
2026-09-18 18:02 ` [PATCH v5 06/11] mm, swap: write back vswap zswap entries to physical swap Nhat Pham
2026-09-18 18:02 ` [PATCH v5 07/11] mm, swap: reclaim physical slots backing cache-only vswap entries Nhat Pham
2026-09-18 18:02 ` [PATCH v5 08/11] mm, swap: only charge physical swap entries Nhat Pham
2026-09-18 18:02 ` [PATCH v5 09/11] mm, swap: add debugfs counters for vswap Nhat Pham
2026-09-18 18:02 ` [PATCH v5 10/11] mm, swap: defer memcg_table allocation for physical swap clusters Nhat Pham
2026-09-18 18:02 ` [RFC PATCH v5 11/11] mm, swap: back vswap clusters with a VM_SPARSE array Nhat Pham
2026-09-18 18:38 ` [PATCH v5 00/11] Virtual Swap Space (Swap Table Edition) Nhat Pham
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®