[PATCH v5 00/19] mm, swap: extendable swap devices (xswap phase I)

[PATCH v5 00/19] mm, swap: extendable swap devices (xswap phase I)

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

A normal swap device has a fixed size, set when it is enabled. This does
not fit zswap well. With zswap the swapped data stays in RAM in
compressed form, so how much swap a machine can really use depends on how
well the workload compresses, not on a number we pick before we know the
workload. Make the device large enough for the worst case and most of it
is never used. Make it just large enough for the average case and the
machine runs out of swap while zswap still has free room.

xswap is a swap device with no backing file. In this phase its swap
entries hold pages in zswap, and its cluster_info array lives in a
VM_SPARSE area that is mapped a chunk at a time, so the device covers a
large swap address space and costs only the chunks it has actually
mapped. The device grows when the workload needs it and gives the tail
back when it does not.

This series is the device itself: the sparse cluster array, growth and
shrink, and the interfaces that create and destroy one. It is phase I of
four; the physical backend, the swap accounting, and the swap-table
backend records (zswap w/o xarray) build on it.

Design notes
------------

In this phase the device is only an swap address space. si->max is set
when the device is created and does not change for its lifetime; what
changes is how much of it is mapped. swap_info_struct carries
nr_clusters_max (the swap address space) and nr_clusters_mapped (the
mapped prefix), and the two walkers that touch cluster_info are bounded
by the latter.

Growth starts when 85% of the mapped range is in use and maps until 65%.
The call comes from the cluster allocation path. Growing maps pages into
the VM_SPARSE area, and mapping can sleep, so it cannot be done inside
the allocation that would otherwise fail: the swap-out path would then
have nothing to fall back on. A grow maps at most XSWAP_GROW_CHUNKS
chunks at a time.

Shrink gives the free tail back when the mapped range is at most 50% in
use and stops at 70%. It is triggered when a cluster becomes completely
free, from a workqueue, because it takes xswap_lock and sleeps in
synchronize_rcu() and vm_area_unmap_pages(). Shrinking on a smaller drop
is not useful: growth happens on demand, so the same clusters would just
be mapped again, and every unmap also costs an RCU grace period. The two
target points sit inside each other's threshold, so a grow cannot leave
the range in a state the shrink would undo, and the other way around.

Configuration
-------------

The device is behind CONFIG_XSWAP, which depends on SWAP, 64BIT, ZSWAP
and SYSFS. CONFIG_XSWAP_DEFAULT_MAX sets the default size of a new
device; it takes a percentage of RAM, a size, or "max", and defaults to
"200%":

  CONFIG_XSWAP_DEFAULT_MAX="200%"

xswap.max= on the kernel command line overrides it:

  xswap.max=8G          8 GiB
  xswap.max=300%        three times RAM
  xswap.max=max         the largest size the device can address

"max" stops at the swap offset limit, capped so that a cluster index fits
in an unsigned int. A slot is still a swap entry, so that limit applies
even though the data never goes to a physical address.

The swap address space cannot move once the device exists, so the size
is settled at creation. That is why this is a boot parameter and not a
runtime knob.

Identity
--------

A device needs a name that survives while it exists. The swap_info type
is a slot in swap_info[] and is reused after swapoff, so it cannot be
one. Each device instead takes an id from a per-boot counter, assigned
at creation and never reused, and that id is what /proc/swaps shows:

  Filename                Type        Size        Used        Priority
  xswap1                  xswap       10485756    0           32767

The same id names the debugfs file and is what destroy takes:

  # cat /sys/kernel/debug/xswap/xswap1
  type 0
  clusters_max 5120
  clusters_mapped 73
  usage_pages 0
  tail_free 72
  grows 0
  shrinks 0
  # echo 1 > /sys/kernel/mm/xswap/destroy

A file-less device has no swapon, so /sys/kernel/mm/xswap/create makes
one. The only argument it takes is a priority; the size comes from
xswap.max=. A priority can be given per device while a size cannot:

  # echo > /sys/kernel/mm/xswap/create          default priority
  # echo 100 > /sys/kernel/mm/xswap/create      priority 100

The default is the highest priority, SWAP_FLAG_PRIO_MASK, so an xswap
device is tried first, ahead of any ordinary swap device. Setting a
lower priority is allowed.

The teardown of a device is shared with sys_swapoff() rather than
duplicated.

swapoff
-------

sys_swapoff() opens the path first. An xswap device has no file, so the
name shown in /proc/swaps cannot be opened. Only a -ENOENT from that
open is taken for an xswap device, and the device with that name is torn
down. A real file or directory keeps precedence: a path that fails for
another reason is never matched. This is what makes both of these work:

  # swapoff xswap1
  # swapoff -a

The name is matched exactly, and /proc/swaps prints the one to use: a
path such as ./xswap1 is not the bare name and is not recognized.

The device needs zswap. Without it swapout takes a swap entry and frees
no memory, so creating one fails.

Limitations
-----------

This phase has no physical backend. If zswap refuses a page, the folio
stays in memory, so an xswap-only system can run out of memory once the
zswap pool is full. Keep a real swap device on such systems; phase II
adds the physical backend.

xswap does not allocate large folios in this phase. The allocation walk
stops at the first device for a large folio, and a new xswap device
defaults to the highest priority, so the folio is offered to xswap,
rejected there, and split into base pages; a lower-priority block device
never gets it whole. Creating the xswap device below a block device in
priority avoids this, at the cost of offering every page to the block
device first. mTHP and THP swapout are both affected; the data is still
swapped, in base pages. A bypass in the walk would avoid the split, but
the next phase teaches xswap to take large folios itself, on a
contiguous physical run, and would remove the bypass again; this phase
leaves the walk unchanged.

An xswap entry stored while no real swap device exists is not put on the
zswap LRU, so adding a real device later does not make it reclaimable.
Phase II puts every xswap entry on the LRU.

Testing
-------

qemu KVM guest, 8G RAM, booted with xswap.max=10G. memhog is a small
local helper: it faults <total_gb> of anon, fills it with a fixed
pattern, and holds it.

Set up zswap and create a device:

  # echo 1 > /sys/module/zswap/parameters/enabled
  # echo > /sys/kernel/mm/xswap/create
  # awk 'NR == 1 || /xswap/' /proc/swaps
  Filename                Type        Size        Used        Priority
  xswap1                  xswap       10485756    0           32767

An idle device maps one chunk of cluster_info: 73 clusters with the
current struct size.

Swap more anon than RAM through a small cgroup:

  # mkdir -p /sys/fs/cgroup/xswap_limit
  # echo 2G > /sys/fs/cgroup/xswap_limit/memory.max
  # echo max > /sys/fs/cgroup/xswap_limit/memory.swap.max
  # ( echo $BASHPID > /sys/fs/cgroup/xswap_limit/cgroup.procs
  #   exec env MEMHOG_FILL=pattern ./memhog 11 600 ) &

clusters_mapped grows to cover the pages that go to zswap (2,414,630
pages against 2,038,301 of RAM in one run), while SwapTotal stays at
10485756. Throughout, SwapTotal never moves and SwapFree stays in
[0, SwapTotal]. Pushing past the size leaves the cgroup out of room and
the OOM killer takes the workload, which is the pass signal there.

Growth and shrink, measured with two cgroups so that one keeps its pages
while the other goes away:

  idle                           mapped=73    occ=0.000  grows=0   shrinks=0
  2G in a 1G cgroup              mapped=657   occ=0.799  grows=7   shrinks=0
  3G more in a second 1G cgroup  mapped=1898  occ=0.829  grows=12  shrinks=0
  kill the second cgroup         mapped=803   occ=0.654  grows=12  shrinks=1

The second cgroup is started later, so its clusters sit at the tail, and
killing it leaves a free tail to take back. The first stays alive, so
usage falls but not to zero: that is the state the shrink has to land in.
occ (occupancy) is usage over the mapped range, which is what the grow and
shrink thresholds are expressed in.

With the first cgroup still alive, destroy the device:

  # echo 1 > /sys/kernel/mm/xswap/destroy

Also tested these, and nothing crashed or warned:

  - a shrink racing a swapoff of the same device, 20 rounds
  - a cgroup that cannot zswap: it gets no xswap slot, and the
    allocator walk does not spin
  - 2000 create/destroy cycles: no per-cluster state is leaked, and the
    ids keep increasing
  - destroy with the workload alive, and a recreate at the same size
  - swapoff -a and swapoff by name both tear the device down

The whole series has also been through the later phases' tests, which
need the physical backend and the swap accounting; those results will be
reported with the phases that own them. Nothing crashed or warned there
either:

  - a destroy returns the physical slots it handed out to their block
    device, and a THP swapout leaves the xswap device and its backend
    with the same usage
  - a swapped THP reads back as one unit from the physical backend, and
    readback through zswap and through the backend both pass a
    byte-for-byte pattern check
  - the memcg swap accounting, memory.swap.current against the backend
    usage, holds across charge, uncharge and destroy

Note on performance
-------------------

This phase is the device and leaves the swap store path alone: an xswap
slot still goes through zswap's xarray. What it adds is covered by the
growth and shrink behaviour above, not by throughput.

When an xswap device exists, the allocation-time zswap gate runs on
every swapout, including the pages that end up on a real device, and in
this phase its cgroup check can flush the cgroup's zswap statistics. A
later phase replaces it with a lockless check.

The series as a whole (phase I+II+III+IV) has been measured on a 288-CPU
server with 1.1 TiB of RAM. The numbers are repeated here only to show
where this leads; each later phase will report the measurements that
belong to it.

  - Release path. usemem -O -y -x -n 1 20G, ten rounds, in a cgroup
    limited to 14G of memory and 12G of swap. The swap-out volumes are
    the same in both arms, 18,370,771 and 18,370,836 pages:

      baseline kernel, zram device    1,067,082 us
      this series,     xswap device     922,396 us    -13.6%

  - Steady-state throughput. The same usemem runs. Per-run medians:

      baseline kernel, zram device     338,260 - 338,760 KB/s
      this series,     xswap device    340,997 - 341,885 KB/s

    The two ranges do not overlap: every pairing of a baseline run with
    a series run is 0.7% to 1.1% ahead, about 1% on average.

  - General workload. A full kernel build, x86_64 defconfig and -j144,
    five rounds per kernel, best of:

      baseline kernel    2,572.6 s CPU (user+sys)
      this series        2,577.7 s CPU (user+sys)

    The 0.2% difference is smaller than the run-to-run spread of either
    kernel.

The large gain is in the release path. How it splits between the swap-table
backend records, the device itself and the accounting is what the later
phases report, each with the measurement that belongs to it.

Overcommit
----------

An xswap device advertises its whole address space as swap: si->pages is
what /proc/swaps and SwapTotal report, and what nr_swap_pages uses to
decide that anonymous memory can be reclaimed. The backing is the zswap
pool in this phase, and real swap devices in the later phases; the
address space is not backing store in either case, yet it raises the
commit limit that __vm_enough_memory() enforces.

Which semantics this should follow is not settled here. Counting the
address space lets a workload commit 2xRAM it can never back; not
counting it changes what SwapTotal means. swap_info_remove() skips its
commit-limit probe for xswap, so destroying a device does not fail on
headroom the device never provided, and real swap devices keep the probe.
The limit can then fall below Committed_AS; new allocations fail until
Committed_AS drops. This is left to a follow-up, together with the rest
of the swap accounting.

Changelog
=========
v4 -> v5:
- Rebased onto the latest mm-new (2c5dafa7f289). 

- A device now has a stable id instead of exposing the swap type:
  /proc/swaps shows xswap<id>, the debugfs file is named by the id and
  prints the type as well, and destroy takes the id. The id is assigned
  per boot and never reused, so a value read earlier cannot name a
  different device.

- swapoff accepts the name: swapoff xswap<id> and swapoff -a tear the
  device down, while a real file or directory keeps precedence. A name
  that does not exist reports -ENOENT; the sysfs destroy keeps -ENODEV
  for an id it does not know. The interfaces are now documented in
  Documentation/ABI/testing/sysfs-kernel-mm-xswap and zswap.rst.

- CONFIG_XSWAP_DEFAULT_MAX sets the default size of a new device. It
  takes a percentage of RAM, a size, or "max", and defaults to "200%";
  xswap.max= still overrides it.

- The cluster_info accesses are RCU-safe: the allocation fast path bounds
  a cached cluster against the mapped range, and the shrink drains the
  per-cpu cache and waits an RCU grace period before unmapping.

- xswap_grow() now links the clusters it mapped even when swapoff has
  already cleared SWP_WRITEOK, so a failed swapoff cannot leave them
  mapped and on no list.

- swap_info_remove() skips the commit limit probe for xswap; real swap
  devices keep it. See the overcommit note above.

- The cgroup zswap gate is checked again after the allocation when an
  xswap device appeared after the initial lookup, so a slot is not kept
  for a cgroup that cannot zswap.

- The patches are reorganized: the VM_SPARSE work is split into helpers
  and the xswap use, the max/pages widening moves before the sysfs
  interface, the shrink trigger now carries the workqueue deferral, and
  the unmap helper is prepared for range shrinking in its own patch. The
  series is now phase I of four, with the swap-table backend records as
  a phase of their own.

Baoquan He (18):
  mm, swap: refactor free_swap_cluster_info to take swap_info_struct
  mm, swap: add VM_SPARSE cluster-array helpers for xswap
  mm, swap: use a VM_SPARSE cluster array for xswap devices
  mm, swap: widen swap_info_struct max/pages to unsigned long
  mm, swap: add a sysfs interface to create xswap devices
  mm, swap: refactor swapoff and add xswap_destroy
  mm, swap: require zswap for xswap devices
  mm, swap: use a stable id for xswap devices
  mm, swap: let the command line set the xswap device size
  mm, swap: add CONFIG_XSWAP_DEFAULT_MAX
  mm, swap: let swapoff destroy xswap devices by name
  mm, swap: add xswap grow trigger on cluster allocation
  mm, swap: add debugfs counters for xswap
  mm, swap: make xswap cluster_info accesses RCU-safe
  mm, swap: prepare xswap_unmap_clusters for range shrinking
  mm, swap: add xswap shrink trigger and defer it to a workqueue
  mm, swap: do not give an xswap slot to a cgroup that cannot zswap
  mm, swap: skip the overcommit probe when removing xswap

Chris Li (1):
  mm: xswap support for zswap

 .../ABI/testing/sysfs-kernel-mm-xswap         |   43 +
 .../admin-guide/kernel-parameters.txt         |    8 +
 Documentation/admin-guide/mm/zswap.rst        |    9 +
 MAINTAINERS                                   |    1 +
 include/linux/swap.h                          |   17 +-
 mm/Kconfig                                    |   20 +
 mm/page_io.c                                  |   19 +
 mm/swap_state.c                               |    9 +
 mm/swapfile.c                                 | 1569 +++++++++++++++--
 mm/zswap.c                                    |   11 +-
 10 files changed, 1593 insertions(+), 113 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-xswap


base-commit: 2c5dafa7f289d760f4492260b0988029aab478b0
-- 
2.54.0

[PATCH v5 01/19] mm: xswap support for zswap

[Baoquan He]

From: Chris Li <chrisl@kernel.org>

Introduce extendable swap device support - xswap.

An xswap device has no backing storage and no swap data section, so
it wastes no disk space. Creation is via a sysfs interface added in a
later patch.

Zswap writeback is gated on whether a real (non-xswap) swap device is
active. nr_real_swapfiles counts such devices and is maintained at
swapon/swapoff only, so the gate reflects "a device exists to write
back to" rather than "a device currently has free slots". This keeps
writeback working even when the real swap device is full, and avoids
a double decrement when a full device is swapped off.

An xswap entry is not added to the zswap writeback LRU: with no backing
store there is nothing to write it back to. zswap_lru_del() tolerates
that, since an entry that was never added is on no list.

Reviewed-by: Kunwu Chan <chentao@kylinos.cn>
Co-developed-by: Baoquan He <hebaoquan@kylinos.cn>
Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
Signed-off-by: Chris Li <chrisl@kernel.org>
---
 include/linux/swap.h |  2 ++
 mm/page_io.c         | 19 +++++++++++++++++++
 mm/swap_state.c      |  9 +++++++++
 mm/swapfile.c        | 37 +++++++++++++++++++++++++++++++------
 mm/zswap.c           | 11 +++++++++--
 5 files changed, 70 insertions(+), 8 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4f686709d71f..9073d29377d8 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -201,6 +201,7 @@ enum {
 	SWP_STABLE_WRITES = (1 << 11),	/* no overwrite PG_writeback pages */
 	SWP_SYNCHRONOUS_IO = (1 << 12),	/* synchronous IO is efficient */
 	SWP_HIBERNATION = (1 << 13),	/* pinned for hibernation */
+	SWP_XSWAP	= (1 << 14),	/* extendable swap device */
 					/* add others here before... */
 };
 
@@ -378,6 +379,7 @@ void free_folio_and_swap_cache(struct folio *folio);
 void free_pages_and_swap_cache(struct encoded_page **, int);
 /* linux/mm/swapfile.c */
 extern atomic_long_t nr_swap_pages;
+extern atomic_t nr_real_swapfiles;
 extern long total_swap_pages;
 extern atomic_t nr_rotate_swap;
 
diff --git a/mm/page_io.c b/mm/page_io.c
index a57cdcf2892d..2285fb1735b3 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -258,6 +258,15 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
 	}
 	rcu_read_unlock();
 
+	/*
+	 * xswap has no backing store: keep the folio.  ctx->sis is not set
+	 * yet, so look the device up from the entry.
+	 */
+	if (unlikely(__swap_entry_to_info(folio->swap)->flags & SWP_XSWAP)) {
+		folio_mark_dirty(folio);
+		return AOP_WRITEPAGE_ACTIVATE;
+	}
+
 	__swap_writeout(ctx, folio);
 	return 0;
 out_unlock:
@@ -515,6 +524,16 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
 	if (zswap_load(folio) != -ENOENT)
 		return;
 
+	if (unlikely(sis->flags & SWP_XSWAP)) {
+		/*
+		 * An xswap entry only ever lives in zswap, so zswap_load()
+		 * must have found it.  Unlock and let the caller retry.
+		 */
+		WARN_ON_ONCE(1);
+		folio_unlock(folio);
+		return;
+	}
+
 	/* We have to read from slower devices. Increase zswap protection. */
 	zswap_folio_swapin(folio);
 	swap_add_folio(ctx, folio, READ);
diff --git a/mm/swap_state.c b/mm/swap_state.c
index da65d75ceead..4c965cf99809 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -899,6 +899,10 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
 	struct blk_plug plug;
 	swp_entry_t ra_entry;
 
+	/* xswap entries live only in zswap; readahead does not help. */
+	if (si->flags & SWP_XSWAP)
+		goto skip;
+
 	mask = swapin_nr_pages(offset) - 1;
 	if (!mask)
 		goto skip;
@@ -984,6 +988,7 @@ static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,
 static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
 		struct mempolicy *mpol, pgoff_t targ_ilx, struct vm_fault *vmf)
 {
+	struct swap_info_struct *si = __swap_entry_to_info(targ_entry);
 	struct swap_io_ctx ctx = {};
 	struct blk_plug plug;
 	struct folio *folio;
@@ -992,6 +997,10 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
 	unsigned long start, end, addr;
 	pgoff_t ilx = targ_ilx;
 
+	/* xswap entries live only in zswap; readahead does not help. */
+	if (si->flags & SWP_XSWAP)
+		goto skip;
+
 	win = swap_vma_ra_win(vmf, &start, &end);
 	if (win == 1)
 		goto skip;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 038d2efe26ad..0b1fef65b6a0 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -66,6 +66,7 @@ static void move_cluster(struct swap_info_struct *si,
 static DEFINE_SPINLOCK(swap_lock);
 static unsigned int nr_swapfiles;
 atomic_long_t nr_swap_pages;
+atomic_t nr_real_swapfiles;
 /*
  * Some modules use swappable objects and may try to swap them out under
  * memory pressure (via the shrinker). Before doing so, they may wish to
@@ -738,7 +739,8 @@ static void free_cluster(struct swap_info_struct *si, struct swap_cluster_info *
 	/*
 	 * If the swap is discardable, prepare discard the cluster
 	 * instead of free it immediately. The cluster will be freed
-	 * after discard.
+	 * after discard.  xswap has no bdev and never sets
+	 * SWP_PAGE_DISCARD, so it always takes the free path below.
 	 */
 	if ((si->flags & (SWP_WRITEOK | SWP_PAGE_DISCARD)) ==
 	    (SWP_WRITEOK | SWP_PAGE_DISCARD)) {
@@ -1221,6 +1223,9 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 		 */
 		lockdep_assert_held(&si->lock);
 		si->flags &= ~SWP_WRITEOK;
+		/* Count active devices, not merely those on the avail list. */
+		if (!(si->flags & SWP_XSWAP))
+			atomic_sub(1, &nr_real_swapfiles);
 		atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 	} else {
 		/*
@@ -1254,6 +1259,12 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
 	if (swapon) {
 		lockdep_assert_held(&si->lock);
 		si->flags |= SWP_WRITEOK;
+		/*
+		 * Count the device here, not after adding it to the list: a
+		 * full device is still active and skips the add below.
+		 */
+		if (!(si->flags & SWP_XSWAP))
+			atomic_add(1, &nr_real_swapfiles);
 	} else {
 		if (!(READ_ONCE(si->flags) & SWP_WRITEOK))
 			goto skip;
@@ -1267,10 +1278,11 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
 	/*
 	 * When device is full and device is on the plist, only one updater will
 	 * see (inuse_pages == si->pages) and will call del_from_avail_list. If
-	 * that updater happen to be here, just skip adding.
+	 * that updater happen to be here, just skip adding.  @val still has
+	 * the off-list bit, so compare its counter part.
 	 */
 	pages = si->pages;
-	if (val == pages) {
+	if ((val & SWAP_USAGE_COUNTER_MASK) == pages) {
 		/* Just like the cmpxchg in del_from_avail_list */
 		if (atomic_long_try_cmpxchg(&si->inuse_pages, &pages,
 					    pages | SWAP_USAGE_OFFLIST_BIT))
@@ -3274,7 +3286,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 
 	destroy_swap_extents(p, p->swap_file);
 
-	if (!(p->flags & SWP_SOLIDSTATE))
+	if (!(p->flags & SWP_XSWAP) &&
+	    !(p->flags & SWP_SOLIDSTATE))
 		atomic_dec(&nr_rotate_swap);
 
 	mutex_lock(&swapon_mutex);
@@ -3384,6 +3397,19 @@ static void swap_stop(struct seq_file *swap, void *v)
 	mutex_unlock(&swapon_mutex);
 }
 
+static const char *swap_type_str(struct swap_info_struct *si)
+{
+	struct file *file = si->swap_file;
+
+	if (si->flags & SWP_XSWAP)
+		return "xswap\t";
+
+	if (S_ISBLK(file_inode(file)->i_mode))
+		return "partition";
+
+	return "file\t";
+}
+
 static int swap_show(struct seq_file *swap, void *v)
 {
 	struct swap_info_struct *si = v;
@@ -3403,8 +3429,7 @@ static int swap_show(struct seq_file *swap, void *v)
 	len = seq_file_path(swap, file, " \t\n\\");
 	seq_printf(swap, "%*s%s\t%lu\t%s%lu\t%s%d\n",
 			len < 40 ? 40 - len : 1, " ",
-			S_ISBLK(file_inode(file)->i_mode) ?
-				"partition" : "file\t",
+			swap_type_str(si),
 			bytes, bytes < 10000000 ? "\t" : "",
 			inuse, inuse < 10000000 ? "\t" : "",
 			si->prio);
diff --git a/mm/zswap.c b/mm/zswap.c
index 37d217860a6b..cef142d2ff85 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1095,6 +1095,11 @@ static int zswap_writeback_entry(struct swap_io_ctx *ctx,
 	if (IS_ERR_OR_NULL(si))
 		return -ENOENT;
 
+	if (si->flags & SWP_XSWAP) {
+		put_swap_device(si);
+		return -EINVAL;
+	}
+
 	mpol = get_task_policy(current);
 	folio = __swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
 					 NO_INTERLEAVE_INDEX);
@@ -1601,7 +1606,9 @@ static bool zswap_store_page(struct folio *folio, long index,
 	entry->referenced = true;
 	if (entry->length) {
 		INIT_LIST_HEAD(&entry->lru);
-		zswap_lru_add(entry);
+		/* No backing store: nothing to write these back to. */
+		if (!(__swap_entry_to_info(page_swpentry)->flags & SWP_XSWAP))
+			zswap_lru_add(entry);
 	}
 
 	return true;
@@ -1671,7 +1678,7 @@ bool zswap_store(struct folio *folio)
 	zswap_pool_put(pool);
 put_objcg:
 	obj_cgroup_put(objcg);
-	if (!ret && zswap_pool_reached_full)
+	if (!ret && zswap_pool_reached_full && atomic_read(&nr_real_swapfiles))
 		queue_work(shrink_wq, &zswap_shrink_work);
 check_old:
 	/*
-- 
2.54.0

[PATCH v5 02/19] mm, swap: refactor free_swap_cluster_info to take swap_info_struct

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

Change free_swap_cluster_info() to take struct swap_info_struct* instead
of (cluster_info, maxpages). It now extracts the fields from si and
clears si->cluster_info after freeing to avoid a double free on the
swapon() error path.

The new parameter also lets the xswap path access si->flags in the
function.

Reviewed-by: Chris Li <chrisl@kernel.org>
Reviewed-by: Kunwu Chan <chentao@kylinos.cn>
Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 0b1fef65b6a0..619bd5305120 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -3155,14 +3155,17 @@ static void wait_for_allocation(struct swap_info_struct *si)
 	}
 }
 
-static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
-				   unsigned long maxpages)
+static void free_swap_cluster_info(struct swap_info_struct *si)
 {
+	struct swap_cluster_info *cluster_info = si->cluster_info;
+	unsigned long maxpages = si->max;
 	struct swap_cluster_info *ci;
-	int i, nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
+	int i, nr_clusters;
 
 	if (!cluster_info)
 		return;
+
+	nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
 	for (i = 0; i < nr_clusters; i++) {
 		ci = cluster_info + i;
 		/* Cluster with bad marks count will have a remaining table */
@@ -3174,6 +3177,7 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
 		spin_unlock(&ci->lock);
 	}
 	kvfree(cluster_info);
+	si->cluster_info = NULL;
 }
 
 /*
@@ -3201,11 +3205,9 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si)
 SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 {
 	struct swap_info_struct *p = NULL;
-	struct swap_cluster_info *cluster_info;
 	struct file *swap_file, *victim;
 	struct address_space *mapping;
 	struct inode *inode;
-	unsigned int maxpages;
 	int err, found = 0;
 
 	if (!capable(CAP_SYS_ADMIN))
@@ -3297,10 +3299,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 
 	swap_file = p->swap_file;
 	p->swap_file = NULL;
-	maxpages = p->max;
-	cluster_info = p->cluster_info;
-	p->max = 0;
-	p->cluster_info = NULL;
 	spin_unlock(&p->lock);
 	spin_unlock(&swap_lock);
 	arch_swap_invalidate_area(p->type);
@@ -3308,7 +3306,9 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	mutex_unlock(&swapon_mutex);
 	kfree(p->global_cluster);
 	p->global_cluster = NULL;
-	free_swap_cluster_info(cluster_info, maxpages);
+	free_swap_cluster_info(p);
+	p->max = 0;
+	p->cluster_info = NULL;
 
 	inode = mapping->host;
 
@@ -3675,6 +3675,8 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 	if (!cluster_info)
 		goto err;
 
+	si->cluster_info = cluster_info;
+
 	for (i = 0; i < nr_clusters; i++)
 		spin_lock_init(&cluster_info[i].lock);
 
@@ -3738,7 +3740,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 	si->cluster_info = cluster_info;
 	return 0;
 err:
-	free_swap_cluster_info(cluster_info, maxpages);
+	free_swap_cluster_info(si);
 	return err;
 }
 
@@ -3960,7 +3962,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	si->global_cluster = NULL;
 	inode = NULL;
 	destroy_swap_extents(si, swap_file);
-	free_swap_cluster_info(si->cluster_info, si->max);
+	free_swap_cluster_info(si);
 	si->cluster_info = NULL;
 	/*
 	 * Clear the SWP_USED flag after all resources are freed so
-- 
2.54.0

[PATCH v5 03/19] mm, swap: add VM_SPARSE cluster-array helpers for xswap

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

Add CONFIG_XSWAP and the state needed for a sparse cluster_info array:
cluster_vm, nr_clusters_max, nr_clusters_mapped and xswap_lock.

Also add the helpers that map and unmap chunks of the VM_SPARSE area,
plus the callback that finds the already-mapped prefix. The helpers
have no callers yet; the next patch adds them into setup/free and the
walkers.

Unmap collects the backing pages before the PTEs go away and frees them
after, so it cannot fail and its callers need no retry loop. If the
collection array cannot be allocated, it falls back to a bounded batch
with a stack array.

They are marked __maybe_unused for this intermediate step so a
CONFIG_WERROR=y build does not warn about the missing callers.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/swap.h |   6 ++
 mm/Kconfig           |   9 ++
 mm/swapfile.c        | 231 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 246 insertions(+)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 9073d29377d8..382a578140b5 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -242,6 +242,12 @@ struct swap_info_struct {
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
 	struct swap_cluster_info *cluster_info; /* array, one entry per cluster */
+#ifdef CONFIG_XSWAP
+	struct vm_struct	*cluster_vm;	/* VM_SPARSE area for cluster_info */
+	unsigned long		nr_clusters_max;/* total clusters in the xswap address space */
+	unsigned long		nr_clusters_mapped; /* currently mapped cluster count */
+	struct mutex		xswap_lock;	/* serialize map/unmap operations */
+#endif
 	struct list_head free_clusters; /* free clusters list */
 	struct list_head full_clusters; /* full clusters list */
 	struct list_head nonfull_clusters[SWAP_NR_ORDERS];
diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b543..9bdb905a66ef 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -122,6 +122,15 @@ config ZSWAP_COMPRESSOR_DEFAULT
        default "zstd" if ZSWAP_COMPRESSOR_DEFAULT_ZSTD
        default ""
 
+config XSWAP
+	bool "Extendable (virtual) swap device"
+	depends on SWAP && 64BIT && ZSWAP && SYSFS
+	help
+	  Adds support for extendable swap devices (xswap) that decouple
+	  PTE swap entries from physical backing storage. The cluster_info
+	  array is backed by a sparse vmalloc area that is mapped lazily
+	  in chunks, avoiding static pre-allocation overhead.
+
 config ZSMALLOC
 	tristate
 
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 619bd5305120..afd0e8718544 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -49,6 +49,21 @@
 #include "internal.h"
 #include "swap.h"
 
+#ifdef CONFIG_XSWAP
+/* Batch size for the fallback unmap; keeps the stack array small. */
+#define XSWAP_UNMAP_BATCH_PAGES		16
+#define XSWAP_UNMAP_BATCH_CLUSTERS					\
+	max_t(unsigned long,						\
+	      ((XSWAP_UNMAP_BATCH_PAGES * PAGE_SIZE) /		\
+	       sizeof(struct swap_cluster_info)), 16)
+
+static int xswap_map_clusters(struct swap_info_struct *si,
+			      unsigned long start_idx, unsigned long nr);
+static void xswap_unmap_clusters(struct swap_info_struct *si,
+				 unsigned long start_idx, unsigned long nr);
+static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data);
+#endif
+
 static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
@@ -3662,6 +3677,222 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
 	return maxpages;
 }
 
+#ifdef CONFIG_XSWAP
+static int __maybe_unused xswap_map_clusters(struct swap_info_struct *si,
+			      unsigned long start_idx, unsigned long nr)
+{
+	unsigned long start_addr = (unsigned long)si->cluster_info +
+				   (size_t)start_idx * sizeof(struct swap_cluster_info);
+	unsigned long end_addr = start_addr + (size_t)nr * sizeof(struct swap_cluster_info);
+	unsigned long vm_start = PAGE_ALIGN(start_addr);
+	unsigned long vm_end = PAGE_ALIGN(end_addr);
+	unsigned int noreclaim_flags;
+	unsigned long mapped_end;
+	unsigned long npages;
+	struct page **pages;
+	unsigned long i;
+	int err;
+
+	mutex_lock(&si->xswap_lock);
+
+	/* Refuse a stale range: the boundary moved since the caller read it. */
+	if (start_idx != READ_ONCE(si->nr_clusters_mapped)) {
+		mutex_unlock(&si->xswap_lock);
+		return -EAGAIN;
+	}
+	if (start_idx + nr > si->nr_clusters_max) {
+		mutex_unlock(&si->xswap_lock);
+		return -EAGAIN;
+	}
+
+	/*
+	 * Page-granular mapping can cover clusters past the previous chunk.
+	 * Find the already-mapped prefix and map only the rest.
+	 */
+	mapped_end = vm_start;
+	if (vm_start < vm_end)
+		apply_to_existing_page_range(&init_mm, vm_start,
+					     vm_end - vm_start,
+					     xswap_mapped_end, &mapped_end);
+	if (vm_start >= vm_end || mapped_end == vm_end)
+		goto mapped;
+	vm_start = mapped_end;
+
+	npages = (vm_end - vm_start) >> PAGE_SHIFT;
+
+	/* Prevent recursive reclaim during vmap page table allocation. */
+	noreclaim_flags = memalloc_noreclaim_save();
+
+	pages = kmalloc_array(npages, sizeof(*pages),
+			      __GFP_HIGH | __GFP_NOMEMALLOC | GFP_KERNEL);
+	if (!pages) {
+		memalloc_noreclaim_restore(noreclaim_flags);
+		mutex_unlock(&si->xswap_lock);
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < npages; i++) {
+		/* __GFP_ZERO: cluster_info pointer fields must start NULL. */
+		pages[i] = alloc_page(__GFP_HIGH | __GFP_NOMEMALLOC |
+				      GFP_KERNEL | __GFP_ZERO);
+		if (!pages[i])
+			goto fail;
+	}
+
+	err = vm_area_map_pages(si->cluster_vm, vm_start, vm_end, pages);
+	if (err) {
+		i = npages;
+		goto fail;
+	}
+
+	for (i = start_idx; i < start_idx + nr; i++)
+		spin_lock_init(&si->cluster_info[i].lock);
+
+	kfree(pages);
+	memalloc_noreclaim_restore(noreclaim_flags);
+
+	/*
+	 * Publish the new mappings and cluster lock initialization before
+	 * the count; walkers without xswap_lock use smp_load_acquire().
+	 */
+	smp_store_release(&si->nr_clusters_mapped, start_idx + nr);
+	mutex_unlock(&si->xswap_lock);
+	return 0;
+
+mapped:
+	for (i = start_idx; i < start_idx + nr; i++)
+		spin_lock_init(&si->cluster_info[i].lock);
+
+	/* Publish the advanced count. */
+	smp_store_release(&si->nr_clusters_mapped, start_idx + nr);
+	mutex_unlock(&si->xswap_lock);
+	return 0;
+
+fail:
+	/* Clear PTEs vm_area_map_pages() may have left before freeing pages. */
+	vm_area_unmap_pages(si->cluster_vm, vm_start, vm_end);
+	while (i > 0) {
+		i--;
+		if (pages[i])
+			__free_page(pages[i]);
+	}
+	memalloc_noreclaim_restore(noreclaim_flags);
+	kfree(pages);
+	mutex_unlock(&si->xswap_lock);
+	return -ENOMEM;
+}
+
+struct xswap_page_data {
+	struct page **pages;
+	unsigned long nr;
+	unsigned long max;
+};
+
+static int xswap_collect_page(pte_t *pte, unsigned long addr, void *data)
+{
+	struct xswap_page_data *xpd = data;
+	pte_t pteval = ptep_get(pte);
+
+	if (!pte_present(pteval))
+		return 0;
+	if (WARN_ON_ONCE(xpd->nr >= xpd->max))
+		return 0;
+	xpd->pages[xpd->nr++] = pte_page(pteval);
+	return 0;
+}
+
+/*
+ * Free the backing pages mapped in [vm_start, vm_end) and clear the PTEs.
+ * @pages must have room for @max pointers.
+ */
+static void xswap_unmap_range(struct swap_info_struct *si,
+			      unsigned long vm_start, unsigned long vm_end,
+			      struct page **pages, unsigned long max)
+{
+	struct xswap_page_data xpd = {
+		.pages	= pages,
+		.max	= max,
+	};
+	unsigned long i;
+
+	apply_to_existing_page_range(&init_mm, vm_start, vm_end - vm_start,
+				     xswap_collect_page, &xpd);
+	vm_area_unmap_pages(si->cluster_vm, vm_start, vm_end);
+
+	for (i = 0; i < xpd.nr; i++)
+		__free_page(pages[i]);
+}
+
+/* Caller must hold si->xswap_lock.  Cannot fail. */
+static void __maybe_unused xswap_unmap_clusters(struct swap_info_struct *si,
+				 unsigned long start_idx, unsigned long nr)
+{
+	unsigned long start_addr = (unsigned long)si->cluster_info +
+				   (size_t)start_idx * sizeof(struct swap_cluster_info);
+	unsigned long end_addr = start_addr +
+				 (size_t)nr * sizeof(struct swap_cluster_info);
+	unsigned long vm_start = PAGE_ALIGN(start_addr);
+	unsigned long vm_end = PAGE_ALIGN(end_addr);
+	struct page **pages;
+	unsigned int noreclaim_flags;
+	unsigned long npages, idx;
+
+	mutex_lock(&si->xswap_lock);
+
+	if (vm_start >= vm_end) {
+		WRITE_ONCE(si->nr_clusters_mapped, start_idx);
+		mutex_unlock(&si->xswap_lock);
+		return;
+	}
+
+	npages = (vm_end - vm_start) >> PAGE_SHIFT;
+
+	if (npages <= totalram_pages()) {
+		/* Reclaim here would re-enter xswap_map_clusters() and its lock. */
+		noreclaim_flags = memalloc_noreclaim_save();
+		pages = kvmalloc_array(npages, sizeof(*pages),
+				       GFP_KERNEL | __GFP_NOMEMALLOC);
+		memalloc_noreclaim_restore(noreclaim_flags);
+		if (pages) {
+			xswap_unmap_range(si, vm_start, vm_end, pages, npages);
+			kvfree(pages);
+			goto out_store;
+		}
+	}
+
+	/* No memory for the array, or too large: unmap in bounded batches. */
+	for (idx = start_idx; idx < start_idx + nr; idx += XSWAP_UNMAP_BATCH_CLUSTERS) {
+		unsigned long batch = min(start_idx + nr - idx,
+					  XSWAP_UNMAP_BATCH_CLUSTERS);
+		unsigned long addr = (unsigned long)si->cluster_info +
+				     (size_t)idx * sizeof(struct swap_cluster_info);
+		unsigned long bstart = PAGE_ALIGN(addr);
+		unsigned long bend = PAGE_ALIGN(addr + (size_t)batch *
+					       sizeof(struct swap_cluster_info));
+		struct page *batch_pages[XSWAP_UNMAP_BATCH_PAGES + 1];
+
+		if (bstart < bend)
+			xswap_unmap_range(si, bstart, bend, batch_pages,
+					  ARRAY_SIZE(batch_pages));
+	}
+out_store:
+	WRITE_ONCE(si->nr_clusters_mapped, start_idx);
+	mutex_unlock(&si->xswap_lock);
+}
+
+/* Track the end of the run of pages that is already mapped. */
+static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data)
+{
+	unsigned long *mapped_end = data;
+
+	/* The mapped range is a prefix, so the first hole ends it. */
+	if (!pte_present(ptep_get(pte)))
+		return 1;
+	*mapped_end = addr + PAGE_SIZE;
+	return 0;
+}
+#endif /* CONFIG_XSWAP */
+
 static int setup_swap_clusters_info(struct swap_info_struct *si,
 				    union swap_header *swap_header,
 				    unsigned long maxpages)
-- 
2.54.0

[PATCH v5 04/19] mm, swap: use a VM_SPARSE cluster array for xswap devices

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

An xswap device does not have to allocate cluster_info for its whole
address space at once. Back it with a VM_SPARSE area and map it in
chunks. At setup, map only the first chunk. The grow path maps more
chunks when they are needed.

Setup, free, and the two cluster_info scans must change together. If
only setup changes, free would free the wrong object, and the scans
would read cluster_info entries that are not mapped yet.

The map helper rejects a stale start_idx and cleans up a partial mapping
if it fails. The grow path added later depends on this check.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 152 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 141 insertions(+), 11 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index afd0e8718544..00c7b398bafe 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -50,6 +50,17 @@
 #include "swap.h"
 
 #ifdef CONFIG_XSWAP
+/*
+ * xswap: dynamically grow the cluster_info array via a VM_SPARSE area.
+ *
+ * XSWAP_GROW_CLUSTERS is the number of clusters to map in one grow
+ * operation.  It is set to the number of cluster_info structs that
+ * fit in a single page (at least 16), so that the vmalloc page table
+ * overhead is proportional to the number of clusters mapped.
+ */
+#define XSWAP_GROW_CLUSTERS \
+	max_t(unsigned long, PAGE_SIZE / sizeof(struct swap_cluster_info), 16)
+
 /* Batch size for the fallback unmap; keeps the stack array small. */
 #define XSWAP_UNMAP_BATCH_PAGES		16
 #define XSWAP_UNMAP_BATCH_CLUSTERS					\
@@ -2825,10 +2836,27 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
 					unsigned int prev)
 {
 	struct swap_cluster_info *ci;
-	unsigned long i, end;
+	unsigned long i, cluster_end, end;
 	unsigned int ci_off;
 	unsigned long swp_tb;
 
+	end = si->max;
+#ifdef CONFIG_XSWAP
+	/*
+	 * An xswap device maps its cluster_info in chunks as it grows, so
+	 * the tail past nr_clusters_mapped has nothing behind it.
+	 */
+	if (si->flags & SWP_XSWAP) {
+		unsigned long mapped_end;
+
+		/* Pairs with the smp_store_release() in xswap_map_clusters(). */
+		mapped_end = smp_load_acquire(&si->nr_clusters_mapped) *
+			     SWAPFILE_CLUSTER;
+		if (mapped_end < end)
+			end = mapped_end;
+	}
+#endif
+
 	/*
 	 * No need for swap_lock here: we're just looking
 	 * for whether an entry is in use, not modifying it; false
@@ -2836,11 +2864,11 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
 	 * allocations from this area (while holding swap_lock).
 	 */
 	i = prev + 1;
-	while (i < si->max) {
+	while (i < end) {
 		ci = __swap_offset_to_cluster(si, i);
-		end = min_t(unsigned long,
-			    ALIGN_DOWN(i, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER,
-			    si->max);
+		cluster_end = min_t(unsigned long,
+				    ALIGN_DOWN(i, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER,
+				    end);
 
 		/*
 		 * An empty cluster has no slot in use, so skip it whole.
@@ -2850,13 +2878,13 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
 		 * enough, unlike in every other cluster_is_empty() caller.
 		 */
 		if (!READ_ONCE(ci->count)) {
-			i = end;
+			i = cluster_end;
 			cond_resched();
 			continue;
 		}
 
 		ci_off = i % SWAPFILE_CLUSTER;
-		for (; i < end; ci_off++, i++) {
+		for (; i < cluster_end; ci_off++, i++) {
 			swp_tb = swap_table_get(ci, ci_off);
 			if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb))
 				return i;
@@ -2866,7 +2894,6 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
 
 	return 0;
 }
-
 static int try_to_unuse(unsigned int type)
 {
 	struct mm_struct *prev_mm;
@@ -3164,6 +3191,14 @@ static void wait_for_allocation(struct swap_info_struct *si)
 
 	BUG_ON(si->flags & SWP_WRITEOK);
 
+#ifdef CONFIG_XSWAP
+	if (si->flags & SWP_XSWAP) {
+		/* Pairs with the smp_store_release() in xswap_map_clusters(). */
+		end = min(end, smp_load_acquire(&si->nr_clusters_mapped) *
+			  SWAPFILE_CLUSTER);
+	}
+#endif
+
 	for (offset = 0; offset < end; offset += SWAPFILE_CLUSTER) {
 		ci = swap_cluster_lock(si, offset);
 		swap_cluster_unlock(ci);
@@ -3175,11 +3210,43 @@ static void free_swap_cluster_info(struct swap_info_struct *si)
 	struct swap_cluster_info *cluster_info = si->cluster_info;
 	unsigned long maxpages = si->max;
 	struct swap_cluster_info *ci;
-	int i, nr_clusters;
+	unsigned long i, nr_clusters;
 
 	if (!cluster_info)
 		return;
 
+#ifdef CONFIG_XSWAP
+	if (si->flags & SWP_XSWAP) {
+		unsigned long nr_mapped;
+
+		/*
+		 * Cluster 0 keeps the bad header slot, so it never empties
+		 * and __free_cluster() never frees its table.
+		 */
+		/* Pairs with the smp_store_release() in xswap_map_clusters(). */
+		nr_mapped = smp_load_acquire(&si->nr_clusters_mapped);
+		for (i = 0; i < nr_mapped; i++) {
+			ci = &cluster_info[i];
+			spin_lock(&ci->lock);
+			if (cluster_table_is_alloced(ci)) {
+				swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, true);
+				swap_cluster_free_table(ci);
+			}
+			spin_unlock(&ci->lock);
+		}
+		/*
+		 * free_vm_area() drops the mapping without freeing the
+		 * pages, and the unmap helper handles an empty range.
+		 */
+		xswap_unmap_clusters(si, 0, si->nr_clusters_mapped);
+		free_vm_area(si->cluster_vm);
+		si->cluster_vm = NULL;
+		si->cluster_info = NULL;
+		si->nr_clusters_mapped = 0;
+		return;
+	}
+#endif
+
 	nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
 	for (i = 0; i < nr_clusters; i++) {
 		ci = cluster_info + i;
@@ -3678,7 +3745,7 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
 }
 
 #ifdef CONFIG_XSWAP
-static int __maybe_unused xswap_map_clusters(struct swap_info_struct *si,
+static int xswap_map_clusters(struct swap_info_struct *si,
 			      unsigned long start_idx, unsigned long nr)
 {
 	unsigned long start_addr = (unsigned long)si->cluster_info +
@@ -3824,7 +3891,7 @@ static void xswap_unmap_range(struct swap_info_struct *si,
 }
 
 /* Caller must hold si->xswap_lock.  Cannot fail. */
-static void __maybe_unused xswap_unmap_clusters(struct swap_info_struct *si,
+static void xswap_unmap_clusters(struct swap_info_struct *si,
 				 unsigned long start_idx, unsigned long nr)
 {
 	unsigned long start_addr = (unsigned long)si->cluster_info +
@@ -3902,6 +3969,69 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 	int err = -ENOMEM;
 	unsigned long i;
 
+#ifdef CONFIG_XSWAP
+	if (si->flags & SWP_XSWAP) {
+		unsigned long size = PAGE_ALIGN(nr_clusters * sizeof(*cluster_info));
+		struct vm_struct *vm;
+
+		vm = get_vm_area(size, VM_SPARSE);
+		if (!vm)
+			goto err;
+
+		cluster_info = vm->addr;
+		si->cluster_vm = vm;
+		si->nr_clusters_max = nr_clusters;
+		si->cluster_info = cluster_info;
+
+		/* Must be initialized before xswap_map_clusters() locks it. */
+		mutex_init(&si->xswap_lock);
+
+		if (xswap_map_clusters(si, 0, min_t(unsigned long,
+					XSWAP_GROW_CLUSTERS, nr_clusters)))
+			goto err_free_vm;
+
+		/* xswap: only cluster 0 slot 0 is bad */
+		err = swap_cluster_setup_bad_slot(si, cluster_info, 0, false);
+		if (err)
+			goto err_unmap;
+
+		INIT_LIST_HEAD(&si->free_clusters);
+		INIT_LIST_HEAD(&si->full_clusters);
+		INIT_LIST_HEAD(&si->discard_clusters);
+		for (i = 0; i < SWAP_NR_ORDERS; i++) {
+			INIT_LIST_HEAD(&si->nonfull_clusters[i]);
+			INIT_LIST_HEAD(&si->frag_clusters[i]);
+		}
+
+		/*
+		 * Cluster 0 holds the header slot, which is marked bad, so it
+		 * is not entirely free.  si->max is cluster aligned, so no
+		 * cluster holds a slot past the end of the device.
+		 */
+		for (i = 0; i < si->nr_clusters_mapped; i++) {
+			struct swap_cluster_info *ci = &cluster_info[i];
+
+			if (ci->count) {
+				ci->flags = CLUSTER_FLAG_NONFULL;
+				list_add_tail(&ci->list, &si->nonfull_clusters[0]);
+			} else {
+				ci->flags = CLUSTER_FLAG_FREE;
+				list_add_tail(&ci->list, &si->free_clusters);
+			}
+		}
+
+		return 0;
+
+err_unmap:
+		xswap_unmap_clusters(si, 0, si->nr_clusters_mapped);
+err_free_vm:
+		free_vm_area(si->cluster_vm);
+		si->cluster_vm = NULL;
+		si->cluster_info = NULL;
+		return err;
+	}
+#endif /* CONFIG_XSWAP */
+
 	cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
 	if (!cluster_info)
 		goto err;
-- 
2.54.0

[PATCH v5 05/19] mm, swap: widen swap_info_struct max/pages to unsigned long

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

si->max and si->pages are unsigned int. With 4 KiB pages, this limits
an xswap device to 16 TB. xswap may need a larger address space, so
change both fields to unsigned long.

Several places still use unsigned int for offsets and indexes. Widen
them as well:

- the return value of cluster_offset() and the cluster scan helpers;
- the allocation offsets in cluster_alloc_swap_entry() and
  swap_alloc_fast();
- the indexes in find_next_to_unuse() and try_to_unuse().

Also change the related print formats from %u to %lu.

A real swap device is still limited by swap_header.info.last_page,
which is __u32. So this only widens the xswap address space. The old
overflow clamp in read_swap_header() is no longer needed; the header
check still limits real devices.

count_swap_pages() returns unsigned int and is only used by
hibernation. Hibernation never selects an xswap device. If that ever
changes, WARN and return 0 instead of silently truncating the value.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/swap.h |  4 +--
 mm/swapfile.c        | 59 +++++++++++++++++++++++---------------------
 2 files changed, 33 insertions(+), 30 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 382a578140b5..569989856c25 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -240,7 +240,7 @@ struct swap_info_struct {
 	signed short	prio;		/* swap priority of this type */
 	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
-	unsigned int	max;		/* size of this swap device */
+	unsigned long	max;		/* size of this swap device */
 	struct swap_cluster_info *cluster_info; /* array, one entry per cluster */
 #ifdef CONFIG_XSWAP
 	struct vm_struct	*cluster_vm;	/* VM_SPARSE area for cluster_info */
@@ -254,7 +254,7 @@ struct swap_info_struct {
 					/* list of cluster that contains at least one free slot */
 	struct list_head frag_clusters[SWAP_NR_ORDERS];
 					/* list of cluster that are fragmented or contented */
-	unsigned int pages;		/* total of usable pages of swap */
+	unsigned long pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
 	struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
 	spinlock_t global_cluster_lock;	/* Serialize usage of global cluster */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 00c7b398bafe..83aaf00b75d3 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -429,10 +429,10 @@ static inline unsigned int cluster_index(struct swap_info_struct *si,
 	return ci - si->cluster_info;
 }
 
-static inline unsigned int cluster_offset(struct swap_info_struct *si,
-					  struct swap_cluster_info *ci)
+static inline unsigned long cluster_offset(struct swap_info_struct *si,
+					   struct swap_cluster_info *ci)
 {
-	return cluster_index(si, ci) * SWAPFILE_CLUSTER;
+	return (unsigned long)cluster_index(si, ci) * SWAPFILE_CLUSTER;
 }
 
 static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
@@ -836,7 +836,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
 
 	/* si->max may got shrunk by swap swap_activate() */
 	if (offset >= si->max && !mask) {
-		pr_debug("Ignoring bad slot %u (max: %u)\n", offset, si->max);
+		pr_debug("Ignoring bad slot %u (max: %lu)\n", offset, si->max);
 		return 0;
 	}
 	/*
@@ -1002,11 +1002,12 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
 }
 
 /* Try use a new cluster for current CPU and allocate from it. */
-static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
-					    struct swap_cluster_info *ci,
-					    struct folio *folio, unsigned long offset)
+static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,
+					     struct swap_cluster_info *ci,
+					     struct folio *folio,
+					     unsigned long offset)
 {
-	unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+	unsigned long next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
 	unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);
 	unsigned int order = likely(folio) ? folio_order(folio) : 0;
 	unsigned long end = start + SWAPFILE_CLUSTER;
@@ -1053,12 +1054,12 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
 	return found;
 }
 
-static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
-					 struct list_head *list,
-					 struct folio *folio,
-					 bool scan_all)
+static unsigned long alloc_swap_scan_list(struct swap_info_struct *si,
+					  struct list_head *list,
+					  struct folio *folio,
+					  bool scan_all)
 {
-	unsigned int found = SWAP_ENTRY_INVALID;
+	unsigned long found = SWAP_ENTRY_INVALID;
 
 	do {
 		struct swap_cluster_info *ci = isolate_lock_cluster(si, list);
@@ -1141,7 +1142,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 {
 	struct swap_cluster_info *ci;
 	unsigned int order = likely(folio) ? folio_order(folio) : 0;
-	unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+	unsigned long offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
 
 	/*
 	 * Swapfile is not block device so unable
@@ -1418,7 +1419,7 @@ static bool swap_alloc_fast(struct folio *folio)
 	unsigned int order = folio_order(folio);
 	struct swap_cluster_info *ci;
 	struct swap_info_struct *si;
-	unsigned int offset;
+	unsigned long offset;
 
 	/*
 	 * Once allocated, swap_info_struct will never be completely freed,
@@ -2509,9 +2510,15 @@ unsigned int count_swap_pages(int type, int free)
 
 		spin_lock(&sis->lock);
 		if (sis->flags & SWP_WRITEOK) {
-			n = sis->pages;
-			if (free)
-				n -= swap_usage_in_pages(sis);
+			/* Hibernation only; it never selects an xswap device. */
+			if (unlikely(sis->flags & SWP_XSWAP)) {
+				WARN_ON_ONCE(1);
+				n = 0;
+			} else {
+				n = sis->pages;
+				if (free)
+					n -= swap_usage_in_pages(sis);
+			}
 		}
 		spin_unlock(&sis->lock);
 	}
@@ -2832,8 +2839,8 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)
  * Return 0 if there are no inuse entries after prev till end of
  * the map.
  */
-static unsigned int find_next_to_unuse(struct swap_info_struct *si,
-					unsigned int prev)
+static unsigned long find_next_to_unuse(struct swap_info_struct *si,
+					unsigned long prev)
 {
 	struct swap_cluster_info *ci;
 	unsigned long i, cluster_end, end;
@@ -2903,7 +2910,7 @@ static int try_to_unuse(unsigned int type)
 	struct swap_info_struct *si = swap_info[type];
 	struct folio *folio;
 	swp_entry_t entry;
-	unsigned int i;
+	unsigned long i;
 
 	if (!swap_usage_in_pages(si))
 		goto success;
@@ -3722,12 +3729,8 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
 		pr_warn("Truncating oversized swap area, only using %luk out of %luk\n",
 			K(maxpages), K(last_page));
 	}
-	if (maxpages > last_page) {
+	if (maxpages > last_page)
 		maxpages = last_page + 1;
-		/* p->max is an unsigned int: don't overflow it */
-		if ((unsigned int)maxpages == 0)
-			maxpages = UINT_MAX;
-	}
 
 	if (!maxpages)
 		return 0;
@@ -4209,7 +4212,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		goto bad_swap_unlock_inode;
 	}
 	if (si->pages != si->max - 1) {
-		pr_err("swap:%u != (max:%u - 1)\n", si->pages, si->max);
+		pr_err("swap:%lu != (max:%lu - 1)\n", si->pages, si->max);
 		error = -EINVAL;
 		goto bad_swap_unlock_inode;
 	}
@@ -4300,7 +4303,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
 	enable_swap_info(si);
 
-	pr_info("Adding %uk swap on %s.  Priority:%d extents:%d across:%lluk %s%s%s%s\n",
+	pr_info("Adding %luk swap on %s.  Priority:%d extents:%d across:%lluk %s%s%s%s\n",
 		K(si->pages), name->name, si->prio, nr_extents,
 		K((unsigned long long)span),
 		(si->flags & SWP_SOLIDSTATE) ? "SS" : "",
-- 
2.54.0

[PATCH v5 06/19] mm, swap: add a sysfs interface to create xswap devices

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

An xswap device has no backing file, so it cannot be created with
swapon.  Add a sysfs interface to create one directly:

  /sys/kernel/mm/xswap/create

Write an optional priority to this file. Write nothing to use the
default priority. By default the device gets the highest priority.

A new device gets si->max and nr_clusters_max equal to twice RAM.
The cluster_info array is a VM_SPARSE area and is mapped lazily.  At
this point only the first XSWAP_GROW_CLUSTERS chunk is mapped; the
grow trigger comes in a later patch.  The device is shown in
/proc/swaps as "xswap<N>".

Hibernation device discovery skips xswap devices because they have no
bdev to store a resume image.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 170 insertions(+), 5 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 83aaf00b75d3..616adbb298e7 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -16,6 +16,8 @@
 #include <linux/kernel_stat.h>
 #include <linux/swap.h>
 #include <linux/vmalloc.h>
+#include <linux/kobject.h>
+#include <linux/sysfs.h>
 #include <linux/pagemap.h>
 #include <linux/namei.h>
 #include <linux/shmem_fs.h>
@@ -48,6 +50,15 @@
 #include "swap_table.h"
 #include "internal.h"
 #include "swap.h"
+#define DEF_SWAP_PRIO  -1
+
+/*
+ * The highest priority a real swap device can use: one ahead of xswap
+ * would take the page directly, and taking that slot is what xswap
+ * exists to avoid.  At equal priority the list is FIFO, so a real device
+ * that asks for this priority too and is added first wins.
+ */
+#define DEF_XSWAP_PRIO	SWAP_FLAG_PRIO_MASK
 
 #ifdef CONFIG_XSWAP
 /*
@@ -73,7 +84,67 @@ static int xswap_map_clusters(struct swap_info_struct *si,
 static void xswap_unmap_clusters(struct swap_info_struct *si,
 				 unsigned long start_idx, unsigned long nr);
 static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data);
-#endif
+
+static int xswap_create(int prio);
+
+static ssize_t xswap_create_store(struct kobject *kobj,
+				  struct kobj_attribute *attr,
+				  const char *buf, size_t count)
+{
+	int prio = DEF_XSWAP_PRIO;
+	const char *p;
+	int err;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	/* "[<prio>]" is optional; a lower one is allowed, not a higher. */
+	p = skip_spaces(buf);
+	if (*p && *p != '\n') {
+		err = kstrtoint(p, 10, &prio);
+		if (err)
+			return err;
+	}
+
+	err = xswap_create(prio);
+	if (err < 0)
+		return err;
+
+	return count;
+}
+
+static struct kobj_attribute xswap_create_attr = __ATTR(create, 0200, NULL,
+							xswap_create_store);
+
+static struct attribute *xswap_attrs[] = {
+	&xswap_create_attr.attr,
+	NULL,
+};
+
+static const struct attribute_group xswap_attr_group = {
+	.attrs = xswap_attrs,
+};
+
+static struct kobject *xswap_kobj;
+
+static void xswap_sysfs_init(void)
+{
+	xswap_kobj = kobject_create_and_add("xswap", mm_kobj);
+	if (!xswap_kobj) {
+		pr_err("xswap: failed to create sysfs kobject\n");
+		return;
+	}
+	if (sysfs_create_group(xswap_kobj, &xswap_attr_group)) {
+		pr_err("xswap: failed to create sysfs group\n");
+		kobject_put(xswap_kobj);
+		xswap_kobj = NULL;
+	}
+}
+#else /* !CONFIG_XSWAP */
+static inline void xswap_sysfs_init(void)
+{
+}
+#endif /* CONFIG_XSWAP */
 
 static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries);
@@ -101,7 +172,6 @@ atomic_t nr_real_swapfiles;
 EXPORT_SYMBOL_GPL(nr_swap_pages);
 /* protected with swap_lock. reading in vm_swap_full() doesn't need lock */
 long total_swap_pages;
-#define DEF_SWAP_PRIO  -1
 unsigned long swapfile_maximum_size;
 #ifdef CONFIG_MIGRATION
 bool swap_migration_ad_supported;
@@ -2284,6 +2354,9 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
 
 		if (!(sis->flags & SWP_WRITEOK))
 			continue;
+		/* xswap has no bdev to match a resume device */
+		if (sis->flags & SWP_XSWAP)
+			continue;
 
 		if (device == sis->bdev->bd_dev) {
 			struct swap_extent *se = first_se(sis);
@@ -2471,6 +2544,8 @@ int find_first_swap(dev_t *device)
 
 		if (!(sis->flags & SWP_WRITEOK))
 			continue;
+		if (sis->flags & SWP_XSWAP)
+			continue;
 		*device = sis->bdev->bd_dev;
 		spin_unlock(&swap_lock);
 		return type;
@@ -3452,7 +3527,8 @@ static void *swap_start(struct seq_file *swap, loff_t *pos)
 		return SEQ_START_TOKEN;
 
 	for (type = 0; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+		if (!si->swap_file &&
+		    !((si->flags & SWP_XSWAP) && (si->flags & SWP_WRITEOK)))
 			continue;
 		if (!--l)
 			return si;
@@ -3473,7 +3549,8 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)
 
 	++(*pos);
 	for (; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+		if (!si->swap_file &&
+		    !((si->flags & SWP_XSWAP) && (si->flags & SWP_WRITEOK)))
 			continue;
 		return si;
 	}
@@ -3515,7 +3592,14 @@ static int swap_show(struct seq_file *swap, void *v)
 	inuse = K(swap_usage_in_pages(si));
 
 	file = si->swap_file;
-	len = seq_file_path(swap, file, " \t\n\\");
+	if (file)
+		len = seq_file_path(swap, file, " \t\n\\");
+	else {
+		char name[16];
+
+		len = scnprintf(name, sizeof(name), "xswap%d", si->type);
+		seq_puts(swap, name);
+	}
 	seq_printf(swap, "%*s%s\t%lu\t%s%lu\t%s%d\n",
 			len < 40 ? 40 - len : 1, " ",
 			swap_type_str(si),
@@ -4026,6 +4110,8 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 		return 0;
 
 err_unmap:
+		/* swap_cluster_setup_bad_slot() allocates cluster 0's table. */
+		swap_cluster_free_table(&si->cluster_info[0]);
 		xswap_unmap_clusters(si, 0, si->nr_clusters_mapped);
 err_free_vm:
 		free_vm_area(si->cluster_vm);
@@ -4108,6 +4194,83 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 	return err;
 }
 
+#ifdef CONFIG_XSWAP
+/* Create a file-less xswap device, sized at twice RAM. */
+static int xswap_create(int prio)
+{
+	struct swap_info_struct *si;
+	unsigned long ram, maxpages;
+	int error;
+
+	if (prio != DEF_XSWAP_PRIO &&
+	    (prio < 0 || prio > SWAP_FLAG_PRIO_MASK))
+		return -EINVAL;
+
+	si = alloc_swap_info();
+	if (IS_ERR(si))
+		return PTR_ERR(si);
+
+	INIT_WORK(&si->discard_work, swap_discard_work);
+	INIT_WORK(&si->reclaim_work, swap_reclaim_work);
+
+	ram = totalram_pages();
+	maxpages = min_t(unsigned long, ram * 2, swapfile_maximum_size);
+	/* A cluster index is an unsigned int, so cap the cluster count. */
+	if (maxpages / SWAPFILE_CLUSTER > UINT_MAX)
+		maxpages = (unsigned long)UINT_MAX * SWAPFILE_CLUSTER;
+	/* Cluster-aligned, so no cluster holds a slot past si->max. */
+	if (maxpages > SWAPFILE_CLUSTER)
+		maxpages = rounddown(maxpages, SWAPFILE_CLUSTER);
+	if (maxpages < 2)
+		maxpages = 2;
+
+	si->bdev = NULL;
+	si->swap_file = NULL;
+	si->flags |= SWP_XSWAP | SWP_SOLIDSTATE;
+	si->max = maxpages;
+	si->pages = maxpages - 1;
+	/*
+	 * No backing file: setup_swap_extents() is only reachable from the
+	 * file-backed swapon() path, so set ops here.  Only ops->flags is
+	 * used, by may_enter_fs(); the IO methods are never called because
+	 * swap_writeout()/swap_read_folio() short circuit xswap.
+	 */
+	si->ops = &swap_bdev_ops;
+
+	error = setup_swap_clusters_info(si, NULL, maxpages);
+	if (error)
+		goto bad_swap;
+
+	error = zswap_swapon(si->type, si->max);
+	if (error)
+		goto bad_swap;
+
+	mutex_lock(&swapon_mutex);
+	si->prio = prio;
+	si->list.prio = -si->prio;
+	si->avail_list.prio = -si->prio;
+	enable_swap_info(si);
+	pr_info("xswap: adding extendable swap type %d (prio %d, %lu pages)\n",
+		si->type, prio, si->pages);
+	mutex_unlock(&swapon_mutex);
+	atomic_inc(&proc_poll_event);
+	wake_up_interruptible(&proc_poll_wait);
+
+	return si->type;
+
+bad_swap:
+	kfree(si->global_cluster);
+	si->global_cluster = NULL;
+	destroy_swap_extents(si, NULL);	/* safe: xswap never sets SWP_ACTIVATED */
+	free_swap_cluster_info(si);
+	si->cluster_info = NULL;
+	spin_lock(&swap_lock);
+	si->flags = 0;
+	spin_unlock(&swap_lock);
+	return error;
+}
+#endif /* CONFIG_XSWAP */
+
 SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 {
 	struct swap_info_struct *si;
@@ -4454,6 +4617,8 @@ static int __init swapfile_init(void)
 		swap_migration_ad_supported = true;
 #endif	/* CONFIG_MIGRATION */
 
+	xswap_sysfs_init();
+
 	return 0;
 }
 subsys_initcall(swapfile_init);
-- 
2.54.0

[PATCH v5 07/19] mm, swap: refactor swapoff and add xswap_destroy

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

Extract __swapoff() from sys_swapoff() so the teardown logic can be
shared, and make it work for file-less devices.

Add xswap_destroy() to tear down a file-less xswap device by swap type,
exposed via /sys/kernel/mm/xswap/destroy. Writing a swap type tears down
that device; it requires CAP_SYS_ADMIN.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 127 +++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 109 insertions(+), 18 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 616adbb298e7..3f2b863f7dd7 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -86,6 +86,7 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data);
 
 static int xswap_create(int prio);
+static int xswap_destroy(int type);
 
 static ssize_t xswap_create_store(struct kobject *kobj,
 				  struct kobj_attribute *attr,
@@ -116,8 +117,35 @@ static ssize_t xswap_create_store(struct kobject *kobj,
 static struct kobj_attribute xswap_create_attr = __ATTR(create, 0200, NULL,
 							xswap_create_store);
 
+static ssize_t xswap_destroy_store(struct kobject *kobj,
+				   struct kobj_attribute *attr,
+				   const char *buf, size_t count)
+{
+	unsigned long type;
+	int err;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	err = kstrtoul(buf, 0, &type);
+	if (err)
+		return err;
+	if (type >= MAX_SWAPFILES)
+		return -EINVAL;
+
+	err = xswap_destroy(type);
+	if (err)
+		return err;
+
+	return count;
+}
+
+static struct kobj_attribute xswap_destroy_attr = __ATTR(destroy, 0200, NULL,
+							 xswap_destroy_store);
+
 static struct attribute *xswap_attrs[] = {
 	&xswap_create_attr.attr,
+	&xswap_destroy_attr.attr,
 	NULL,
 };
 
@@ -3365,13 +3393,14 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si)
 	}
 }
 
+static int swap_info_remove(struct swap_info_struct *p);
+static int __swapoff(struct swap_info_struct *p);
 
 SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 {
 	struct swap_info_struct *p = NULL;
-	struct file *swap_file, *victim;
+	struct file *victim;
 	struct address_space *mapping;
-	struct inode *inode;
 	int err, found = 0;
 
 	if (!capable(CAP_SYS_ADMIN))
@@ -3388,7 +3417,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	spin_lock(&swap_lock);
 	plist_for_each_entry(p, &swap_active_head, list) {
 		if (p->flags & SWP_WRITEOK) {
-			if (p->swap_file->f_mapping == mapping) {
+			if (p->swap_file && p->swap_file->f_mapping == mapping) {
 				found = 1;
 				break;
 			}
@@ -3407,20 +3436,50 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 		goto out_dput;
 	}
 
-	if (!security_vm_enough_memory_mm(current->mm, p->pages))
-		vm_unacct_memory(p->pages);
-	else {
-		err = -ENOMEM;
+	err = swap_info_remove(p);
+	if (err) {
 		spin_unlock(&swap_lock);
 		goto out_dput;
 	}
+	spin_unlock(&swap_lock);
+
+	err = __swapoff(p);
+
+out_dput:
+	filp_close(victim, NULL);
+	return err;
+}
+
+/*
+ * Drop @p from the avail and active lists and undo its accounting.  The
+ * caller must hold swap_lock and have checked that @p is WRITEOK and not
+ * pinned for hibernation.
+ *
+ * Returns 0, or -ENOMEM with swap_lock still held.
+ */
+static int swap_info_remove(struct swap_info_struct *p)
+{
+	if (!security_vm_enough_memory_mm(current->mm, p->pages))
+		vm_unacct_memory(p->pages);
+	else
+		return -ENOMEM;
+
 	spin_lock(&p->lock);
 	del_from_avail_list(p, true);
 	plist_del(&p->list, &swap_active_head);
 	atomic_long_sub(p->pages, &nr_swap_pages);
 	total_swap_pages -= p->pages;
 	spin_unlock(&p->lock);
-	spin_unlock(&swap_lock);
+	return 0;
+}
+
+/* Common swap teardown after list removal; shared by sys_swapoff() and
+ * xswap_destroy().
+ */
+static int __swapoff(struct swap_info_struct *p)
+{
+	struct file *swap_file = NULL;
+	int err;
 
 	wait_for_allocation(p);
 
@@ -3431,7 +3490,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	if (err) {
 		/* re-insert swap space back into swap_list */
 		reinsert_swap_info(p);
-		goto out_dput;
+		return err;
 	}
 
 	/*
@@ -3471,15 +3530,21 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	kfree(p->global_cluster);
 	p->global_cluster = NULL;
 	free_swap_cluster_info(p);
+	/*
+	 * The device is off swap_active_head and no longer WRITEOK, so no
+	 * reader can observe these; clearing them here needs no lock.
+	 */
 	p->max = 0;
 	p->cluster_info = NULL;
 
-	inode = mapping->host;
+	if (swap_file) {
+		struct inode *inode = swap_file->f_mapping->host;
 
-	inode_lock(inode);
-	inode->i_flags &= ~S_SWAPFILE;
-	inode_unlock(inode);
-	filp_close(swap_file, NULL);
+		inode_lock(inode);
+		inode->i_flags &= ~S_SWAPFILE;
+		inode_unlock(inode);
+		filp_close(swap_file, NULL);
+	}
 
 	/*
 	 * Clear the SWP_USED flag after all resources are freed so that swapon
@@ -3490,13 +3555,10 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	p->flags = 0;
 	spin_unlock(&swap_lock);
 
-	err = 0;
 	atomic_inc(&proc_poll_event);
 	wake_up_interruptible(&proc_poll_wait);
 
-out_dput:
-	filp_close(victim, NULL);
-	return err;
+	return 0;
 }
 
 #ifdef CONFIG_PROC_FS
@@ -4269,6 +4331,35 @@ static int xswap_create(int prio)
 	spin_unlock(&swap_lock);
 	return error;
 }
+
+/* Tear down a file-less xswap device by its swap type. */
+static int xswap_destroy(int type)
+{
+	struct swap_info_struct *p;
+	int err;
+
+	p = swap_type_to_info(type);
+	if (!p)
+		return -EINVAL;
+
+	spin_lock(&swap_lock);
+	if (!(p->flags & SWP_WRITEOK) || !(p->flags & SWP_XSWAP)) {
+		spin_unlock(&swap_lock);
+		return -EINVAL;
+	}
+	/* Refuse swapoff while the device is pinned for hibernation */
+	if (p->flags & SWP_HIBERNATION) {
+		spin_unlock(&swap_lock);
+		return -EBUSY;
+	}
+
+	err = swap_info_remove(p);
+	spin_unlock(&swap_lock);
+	if (err)
+		return err;
+
+	return __swapoff(p);
+}
 #endif /* CONFIG_XSWAP */
 
 SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
-- 
2.54.0

[PATCH v5 08/19] mm, swap: require zswap for xswap devices

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

xswap pages live only in zswap.  Without zswap, swapout cannot
free the page, but still takes a swap entry.  So the device consumes
swap entries without freeing memory.  Fail to create a device when
zswap is unavailable.

Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 3f2b863f7dd7..35c8aac1e25d 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -4268,6 +4268,10 @@ static int xswap_create(int prio)
 	    (prio < 0 || prio > SWAP_FLAG_PRIO_MASK))
 		return -EINVAL;
 
+	/* xswap has no backing store, it relies on zswap. */
+	if (!zswap_is_enabled())
+		return -EOPNOTSUPP;
+
 	si = alloc_swap_info();
 	if (IS_ERR(si))
 		return PTR_ERR(si);
-- 
2.54.0

[PATCH v5 09/19] mm, swap: use a stable id for xswap devices

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

Give each xswap device a 64-bit id from a counter. The id is never
reused. Show "xswap<id>" in /proc/swaps and use the same id in
debugfs. Destroy matches the id, so an old name fails with -ENODEV
instead of hitting a new device.

This avoids names based on the swap type, which can be reused after
swapoff. The swap type is still used inside the kernel and in dmesg.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/swap.h |  1 +
 mm/swapfile.c        | 59 ++++++++++++++++++++++++++++++--------------
 2 files changed, 42 insertions(+), 18 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 569989856c25..f4eb1912d1e6 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -244,6 +244,7 @@ struct swap_info_struct {
 	struct swap_cluster_info *cluster_info; /* array, one entry per cluster */
 #ifdef CONFIG_XSWAP
 	struct vm_struct	*cluster_vm;	/* VM_SPARSE area for cluster_info */
+	u64			xswap_id;	/* stable id, shown to userspace */
 	unsigned long		nr_clusters_max;/* total clusters in the xswap address space */
 	unsigned long		nr_clusters_mapped; /* currently mapped cluster count */
 	struct mutex		xswap_lock;	/* serialize map/unmap operations */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 35c8aac1e25d..44b54118d10a 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -86,7 +86,10 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data);
 
 static int xswap_create(int prio);
-static int xswap_destroy(int type);
+static int xswap_destroy(u64 id);
+
+/* Stable device id, assigned at create and never reused. */
+static atomic64_t xswap_id_counter = ATOMIC64_INIT(0);
 
 static ssize_t xswap_create_store(struct kobject *kobj,
 				  struct kobj_attribute *attr,
@@ -121,19 +124,17 @@ static ssize_t xswap_destroy_store(struct kobject *kobj,
 				   struct kobj_attribute *attr,
 				   const char *buf, size_t count)
 {
-	unsigned long type;
+	u64 id;
 	int err;
 
 	if (!capable(CAP_SYS_ADMIN))
 		return -EPERM;
 
-	err = kstrtoul(buf, 0, &type);
+	err = kstrtoull(buf, 0, &id);
 	if (err)
 		return err;
-	if (type >= MAX_SWAPFILES)
-		return -EINVAL;
 
-	err = xswap_destroy(type);
+	err = xswap_destroy(id);
 	if (err)
 		return err;
 
@@ -3657,9 +3658,15 @@ static int swap_show(struct seq_file *swap, void *v)
 	if (file)
 		len = seq_file_path(swap, file, " \t\n\\");
 	else {
-		char name[16];
+		char name[32];
 
+#ifdef CONFIG_XSWAP
+		len = scnprintf(name, sizeof(name), "xswap%llu",
+				(unsigned long long)si->xswap_id);
+#else
+		/* CONFIG_XSWAP=n has no file-less devices; keep the type. */
 		len = scnprintf(name, sizeof(name), "xswap%d", si->type);
+#endif
 		seq_puts(swap, name);
 	}
 	seq_printf(swap, "%*s%s\t%lu\t%s%lu\t%s%d\n",
@@ -4295,6 +4302,11 @@ static int xswap_create(int prio)
 	si->flags |= SWP_XSWAP | SWP_SOLIDSTATE;
 	si->max = maxpages;
 	si->pages = maxpages - 1;
+	/*
+	 * Set before the device is enabled, for debugfs; destroy only matches
+	 * a WRITEOK device, so the create window is not reachable.
+	 */
+	si->xswap_id = atomic64_inc_return(&xswap_id_counter);
 	/*
 	 * No backing file: setup_swap_extents() is only reachable from the
 	 * file-backed swapon() path, so set ops here.  Only ops->flags is
@@ -4316,8 +4328,8 @@ static int xswap_create(int prio)
 	si->list.prio = -si->prio;
 	si->avail_list.prio = -si->prio;
 	enable_swap_info(si);
-	pr_info("xswap: adding extendable swap type %d (prio %d, %lu pages)\n",
-		si->type, prio, si->pages);
+	pr_info("xswap: adding extendable swap device xswap%llu (prio %d, %lu pages)\n",
+		(unsigned long long)si->xswap_id, prio, si->pages);
 	mutex_unlock(&swapon_mutex);
 	atomic_inc(&proc_poll_event);
 	wake_up_interruptible(&proc_poll_wait);
@@ -4336,20 +4348,31 @@ static int xswap_create(int prio)
 	return error;
 }
 
-/* Tear down a file-less xswap device by its swap type. */
-static int xswap_destroy(int type)
+/* Tear down a file-less xswap device by its stable id. */
+static int xswap_destroy(u64 id)
 {
-	struct swap_info_struct *p;
+	struct swap_info_struct *p = NULL;
+	unsigned int type;
 	int err;
 
-	p = swap_type_to_info(type);
-	if (!p)
-		return -EINVAL;
-
 	spin_lock(&swap_lock);
-	if (!(p->flags & SWP_WRITEOK) || !(p->flags & SWP_XSWAP)) {
+	/*
+	 * Match only a device that is fully enabled: one still being created
+	 * is not WRITEOK yet, and one being destroyed has lost it. Neither
+	 * is a valid target, so both read as "no such device".
+	 */
+	for (type = 0; type < nr_swapfiles; type++) {
+		struct swap_info_struct *q = swap_info[type];
+
+		if (q && (q->flags & (SWP_XSWAP | SWP_WRITEOK)) ==
+			 (SWP_XSWAP | SWP_WRITEOK) && q->xswap_id == id) {
+			p = q;
+			break;
+		}
+	}
+	if (!p) {
 		spin_unlock(&swap_lock);
-		return -EINVAL;
+		return -ENODEV;
 	}
 	/* Refuse swapoff while the device is pinned for hibernation */
 	if (p->flags & SWP_HIBERNATION) {
-- 
2.54.0

[PATCH v5 10/19] mm, swap: let the command line set the xswap device size

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

By default, a new xswap device is 200% of RAM, which is twice RAM.
Add xswap.max= to set the size of devices created after it:

  xswap.max=300%      three times RAM
  xswap.max=8G        eight GiB
  xswap.max=max       the largest size the device can address

The default is fine for most users. But a machine with a lot of RAM
and a small working set can get a SwapTotal much larger than it needs.

The size is resolved at device creation, not at parameter parsing
because a percentage depends on RAM, and totalram_pages() is not ready
when parameters are parsed.

An explicit size that is too large or overflows is refused. The default
size is clamped to the limit instead. A size below one cluster is also
refused. "max" means the largest addressable size, still bounded by the
unsigned int cluster index.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 .../admin-guide/kernel-parameters.txt         |   7 +
 mm/swapfile.c                                 | 130 ++++++++++++++++--
 2 files changed, 127 insertions(+), 10 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd9..e92a1b9062a6 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -8738,3 +8738,10 @@ Kernel parameters
 				memory, and other data can't be written using
 				xmon commands.
 			off	xmon is disabled.
+
+	xswap.max=	[KNL,SWAP]
+			Maximum size of a new xswap device. The value is a
+			size in bytes with an optional KMGTPE suffix, a
+			percentage of RAM, or "max" for the largest size
+			the device can address.
+			Format: <size>[KMGTPE] | <percent>% | max
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 44b54118d10a..498fbf6df1cd 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -14,6 +14,7 @@
 #include <linux/mman.h>
 #include <linux/slab.h>
 #include <linux/kernel_stat.h>
+#include <linux/overflow.h>
 #include <linux/swap.h>
 #include <linux/vmalloc.h>
 #include <linux/kobject.h>
@@ -150,6 +151,100 @@ static struct attribute *xswap_attrs[] = {
 	NULL,
 };
 
+/*
+ * Size of a new device, from xswap.max=.  A percentage unless bytes is
+ * set; "max" asks for the largest size the device can address.
+ */
+static unsigned int xswap_size_pct = 200;
+static unsigned long xswap_size_bytes;
+static bool xswap_size_max;
+static bool xswap_size_set;	/* xswap.max= was given */
+
+/*
+ * Parse "200%", "2T" or "max".  @str is not modified.
+ */
+static int __init xswap_parse_size(const char *str, const char *what)
+{
+	const char *pct;
+	char buf[16];
+	unsigned int pct_val;
+	unsigned long bytes;
+	char *end;
+
+	pct = strchr(str, '%');
+	if (!strcmp(str, "max")) {
+		xswap_size_max = true;
+		xswap_size_pct = 0;
+		xswap_size_bytes = 0;
+	} else if (pct) {
+		size_t len = pct - str;
+
+		if (pct[1] || !len || len >= sizeof(buf))
+			goto bad;
+		memcpy(buf, str, len);
+		buf[len] = '\0';
+		if (kstrtouint(buf, 0, &pct_val) || !pct_val)
+			goto bad;
+		xswap_size_pct = pct_val;
+		xswap_size_bytes = 0;
+		xswap_size_max = false;
+	} else {
+		bytes = memparse(str, &end);
+		if (*end || bytes < PAGE_SIZE)
+			goto bad;
+		xswap_size_bytes = bytes;
+		xswap_size_pct = 0;
+		xswap_size_max = false;
+	}
+
+	xswap_size_set = true;
+	return 0;
+bad:
+	pr_err("xswap: bad %s=%s\n", what, str);
+	return -EINVAL;
+}
+
+static int __init xswap_max_setup(char *str)
+{
+	xswap_parse_size(str, "xswap.max");
+	return 1;
+}
+__setup("xswap.max=", xswap_max_setup);
+
+/*
+ * The most a device can address, whatever xswap.max= asked for.  An xswap
+ * slot is still a swap entry, so the swap offset limit applies even though
+ * the data never goes through a physical address.
+ */
+static unsigned long xswap_size_limit(void)
+{
+	unsigned long limit = swapfile_maximum_size;
+
+	/* A cluster index is an unsigned int. */
+	if (limit / SWAPFILE_CLUSTER > UINT_MAX)
+		limit = (unsigned long)UINT_MAX * SWAPFILE_CLUSTER;
+
+	return limit;
+}
+
+static unsigned long xswap_size_pages(void)
+{
+	unsigned long long pages;
+
+	if (xswap_size_max)
+		return xswap_size_limit();
+
+	if (xswap_size_bytes)
+		return xswap_size_bytes >> PAGE_SHIFT;
+
+	/* Let xswap_create() refuse a size that does not fit in the type. */
+	if (check_mul_overflow((unsigned long long)totalram_pages(),
+			       (unsigned long long)xswap_size_pct, &pages))
+		return ULONG_MAX;
+
+	return div_u64(pages, 100);
+}
+
 static const struct attribute_group xswap_attr_group = {
 	.attrs = xswap_attrs,
 };
@@ -4268,7 +4363,8 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 static int xswap_create(int prio)
 {
 	struct swap_info_struct *si;
-	unsigned long ram, maxpages;
+	unsigned long maxpages;
+	unsigned long limit;
 	int error;
 
 	if (prio != DEF_XSWAP_PRIO &&
@@ -4279,6 +4375,28 @@ static int xswap_create(int prio)
 	if (!zswap_is_enabled())
 		return -EOPNOTSUPP;
 
+	/*
+	 * Refuse an explicitly requested size the device cannot address;
+	 * clamp the default size, which nobody asked for.
+	 */
+	limit = xswap_size_limit();
+	maxpages = xswap_size_pages();
+	if (xswap_size_set && maxpages > limit) {
+		pr_err("xswap: xswap.max is larger than the device can address\n");
+		return -EINVAL;
+	}
+	maxpages = min(maxpages, limit);
+	/*
+	 * Below one cluster the device is not cluster-aligned, so its only
+	 * cluster would hold slots past si->max.  Nothing masks those, and a
+	 * slot allocated there is past the end of the device: it can never be
+	 * read back or given up again.
+	 */
+	if (maxpages < SWAPFILE_CLUSTER) {
+		pr_err("xswap: xswap.max is smaller than one cluster\n");
+		return -EINVAL;
+	}
+
 	si = alloc_swap_info();
 	if (IS_ERR(si))
 		return PTR_ERR(si);
@@ -4286,16 +4404,8 @@ static int xswap_create(int prio)
 	INIT_WORK(&si->discard_work, swap_discard_work);
 	INIT_WORK(&si->reclaim_work, swap_reclaim_work);
 
-	ram = totalram_pages();
-	maxpages = min_t(unsigned long, ram * 2, swapfile_maximum_size);
-	/* A cluster index is an unsigned int, so cap the cluster count. */
-	if (maxpages / SWAPFILE_CLUSTER > UINT_MAX)
-		maxpages = (unsigned long)UINT_MAX * SWAPFILE_CLUSTER;
 	/* Cluster-aligned, so no cluster holds a slot past si->max. */
-	if (maxpages > SWAPFILE_CLUSTER)
-		maxpages = rounddown(maxpages, SWAPFILE_CLUSTER);
-	if (maxpages < 2)
-		maxpages = 2;
+	maxpages = rounddown(maxpages, SWAPFILE_CLUSTER);
 
 	si->bdev = NULL;
 	si->swap_file = NULL;
-- 
2.54.0

[PATCH v5 11/19] mm, swap: add CONFIG_XSWAP_DEFAULT_MAX

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

Add a build-time default for the size of a new xswap device. The value
can be a percentage of RAM, a size in bytes, or "max". "max" means the
largest size the device can address. xswap.max= on the command line
overrides this default.

The build-time default is clamped to the device limit. An explicit
xswap.max= that is too large is refused.

If the build-time default cannot be parsed, or is a byte size below one
cluster, use 200% instead. A percentage default can still be below one
cluster after RAM is known. In that case, device creation fails.

Document the sysfs ABI, the stable id, and the synthetic /proc/swaps
name.  Also document that zswap must stay enabled while xswap devices
exist.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 .../ABI/testing/sysfs-kernel-mm-xswap         | 38 +++++++++++++++++++
 .../admin-guide/kernel-parameters.txt         |  3 +-
 Documentation/admin-guide/mm/zswap.rst        |  9 +++++
 MAINTAINERS                                   |  1 +
 mm/Kconfig                                    | 11 ++++++
 mm/swapfile.c                                 | 22 +++++++++--
 6 files changed, 79 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-xswap

diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-xswap b/Documentation/ABI/testing/sysfs-kernel-mm-xswap
new file mode 100644
index 000000000000..38eead0b1b31
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-xswap
@@ -0,0 +1,38 @@
+What:		/sys/kernel/mm/xswap/
+Date:		October 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	File-less swap devices backed by zswap.
+
+		An xswap device has no backing file.  Its slots are backed
+		by the zswap pool, and the device size is set at boot with
+		xswap.max= or CONFIG_XSWAP_DEFAULT_MAX.  The size is fixed
+		for the lifetime of the device.
+
+		The name shown in /proc/swaps, xswap<id>, is synthetic and
+		is not a path.  It cannot be passed to swapoff(2), so
+		swapoff -a reports an error for these entries.  Use the
+		destroy attribute instead.
+
+		An xswap device requires zswap.  Disabling zswap at runtime
+		(enabled=0, or max_pool_percent=0) while xswap devices
+		exist leaves them without backing: existing entries can
+		still be read, but new pages cannot be stored.  Destroy the
+		xswap devices before disabling zswap.
+
+What:		/sys/kernel/mm/xswap/create
+Date:		October 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	Create an xswap device.
+
+		Write an optional priority (0..32767), or write nothing for
+		the default priority.  The write fails if zswap is disabled
+		or if the configured size is smaller than one cluster.
+
+What:		/sys/kernel/mm/xswap/destroy
+Date:		October 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	Destroy an xswap device.
+
+		Write the device's stable id.  The id is the number after
+		"xswap" in /proc/swaps, for example 1 for xswap1.  It is
+		not a swap type and is never reused.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index e92a1b9062a6..6e2d2747a26f 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -8743,5 +8743,6 @@ Kernel parameters
 			Maximum size of a new xswap device. The value is a
 			size in bytes with an optional KMGTPE suffix, a
 			percentage of RAM, or "max" for the largest size
-			the device can address.
+			the device can address. The default is set by
+			CONFIG_XSWAP_DEFAULT_MAX.
 			Format: <size>[KMGTPE] | <percent>% | max
diff --git a/Documentation/admin-guide/mm/zswap.rst b/Documentation/admin-guide/mm/zswap.rst
index 2464425c783d..886b50ca90b1 100644
--- a/Documentation/admin-guide/mm/zswap.rst
+++ b/Documentation/admin-guide/mm/zswap.rst
@@ -136,3 +136,12 @@ selected.
 A debugfs interface is provided for various statistic about pool size, number
 of pages stored, same-value filled pages and various counters for the reasons
 pages are rejected.
+
+xswap devices
+-------------
+
+An xswap device is a file-less swap device backed by zswap.  While such a
+device exists, do not disable zswap at runtime (``enabled=0``) or set
+``max_pool_percent`` to 0.  The device would have no backing: pages already
+in zswap can still be read, but new pages cannot be stored.  Destroy the
+xswap devices first.
diff --git a/MAINTAINERS b/MAINTAINERS
index 0cf24eee20bb..66078761b746 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17910,6 +17910,7 @@ R:	Youngjun Park <youngjun.park@lge.com>
 L:	linux-mm@kvack.org
 S:	Maintained
 F:	Documentation/ABI/testing/sysfs-kernel-mm-swap
+F:	Documentation/ABI/testing/sysfs-kernel-mm-xswap
 F:	Documentation/mm/swap-table.rst
 F:	include/linux/swap.h
 F:	include/linux/swap_ops.h
diff --git a/mm/Kconfig b/mm/Kconfig
index 9bdb905a66ef..902e3b1a6e35 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -131,6 +131,17 @@ config XSWAP
 	  array is backed by a sparse vmalloc area that is mapped lazily
 	  in chunks, avoiding static pre-allocation overhead.
 
+config XSWAP_DEFAULT_MAX
+	string "Default maximum size of a new xswap device"
+	depends on XSWAP
+	default "200%"
+	help
+	  The maximum size of a newly created xswap device. It accepts a
+	  percentage of RAM (for example "200%"), a size (for example "2T"),
+	  or "max" for the largest size the device can address.
+
+	  xswap.max= on the kernel command line overrides this value.
+
 config ZSMALLOC
 	tristate
 
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 498fbf6df1cd..cb53f9415de8 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -161,9 +161,12 @@ static bool xswap_size_max;
 static bool xswap_size_set;	/* xswap.max= was given */
 
 /*
- * Parse "200%", "2T" or "max".  @str is not modified.
+ * Parse "200%", "2T" or "max". @str is not modified. @cmdline tells an
+ * explicit xswap.max= from the build-time default, which is clamped
+ * instead of refused when it is too large.
  */
-static int __init xswap_parse_size(const char *str, const char *what)
+static int __init xswap_parse_size(const char *str, const char *what,
+				   bool cmdline)
 {
 	const char *pct;
 	char buf[16];
@@ -192,12 +195,16 @@ static int __init xswap_parse_size(const char *str, const char *what)
 		bytes = memparse(str, &end);
 		if (*end || bytes < PAGE_SIZE)
 			goto bad;
+		/* A default below one cluster would fail every create. */
+		if (!cmdline && bytes < (unsigned long)SWAPFILE_CLUSTER * PAGE_SIZE)
+			goto bad;
 		xswap_size_bytes = bytes;
 		xswap_size_pct = 0;
 		xswap_size_max = false;
 	}
 
-	xswap_size_set = true;
+	if (cmdline)
+		xswap_size_set = true;
 	return 0;
 bad:
 	pr_err("xswap: bad %s=%s\n", what, str);
@@ -206,7 +213,7 @@ static int __init xswap_parse_size(const char *str, const char *what)
 
 static int __init xswap_max_setup(char *str)
 {
-	xswap_parse_size(str, "xswap.max");
+	xswap_parse_size(str, "xswap.max", true);
 	return 1;
 }
 __setup("xswap.max=", xswap_max_setup);
@@ -4830,6 +4837,13 @@ static int __init swapfile_init(void)
 {
 	swapfile_maximum_size = arch_max_swapfile_size();
 
+#ifdef CONFIG_XSWAP
+	/* xswap.max= on the command line overrides the build-time default. */
+	if (!xswap_size_set)
+		xswap_parse_size(CONFIG_XSWAP_DEFAULT_MAX,
+				 "CONFIG_XSWAP_DEFAULT_MAX", false);
+#endif
+
 	/*
 	 * Once a cluster is freed, it's swap table content is read
 	 * only, and all swap cache readers (swap_cache_*) verifies
-- 
2.54.0

[PATCH v5 12/19] mm, swap: let swapoff destroy xswap devices by name

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

An xswap device has no backing file, so /proc/swaps shows a synthetic
name, xswap<id>, that file_open_name() cannot open. Teach swapoff to
fall back to xswap_destroy() when the name is exactly "xswap<digits>"
and the lookup failed with -ENOENT.

A real file or directory keeps precedence; other open errors are
returned unchanged, so a file that exists but cannot be opened is not
mistaken for an xswap device. The ABI documentation is updated, and
the destroy attribute remains the unambiguous teardown interface.

A well-formed name for an id that does not exist asks xswap_destroy()
for a device that is not there. It returns -ENODEV; map that to
-ENOENT so swapoff reports a missing name like it does for a missing
file.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 .../ABI/testing/sysfs-kernel-mm-xswap         | 11 +++-
 mm/swapfile.c                                 | 56 ++++++++++++++++++-
 2 files changed, 62 insertions(+), 5 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-xswap b/Documentation/ABI/testing/sysfs-kernel-mm-xswap
index 38eead0b1b31..f7e7f3e4d216 100644
--- a/Documentation/ABI/testing/sysfs-kernel-mm-xswap
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-xswap
@@ -9,9 +9,14 @@ Description:	File-less swap devices backed by zswap.
 		for the lifetime of the device.
 
 		The name shown in /proc/swaps, xswap<id>, is synthetic and
-		is not a path.  It cannot be passed to swapoff(2), so
-		swapoff -a reports an error for these entries.  Use the
-		destroy attribute instead.
+		is not a path. swapoff(2) accepts it, so both "swapoff
+		xswap<id>" and swapoff -a tear the device down. A real
+		file or directory with that name takes precedence: if it
+		opens, it is looked up as a swap file and the synthetic
+		device is not used. Only a name that cannot be opened
+		because it does not exist, including a dangling symlink,
+		is treated as an xswap device. Use the destroy attribute
+		for an unambiguous teardown.
 
 		An xswap device requires zswap.  Disabling zswap at runtime
 		(enabled=0, or max_pool_percent=0) while xswap devices
diff --git a/mm/swapfile.c b/mm/swapfile.c
index cb53f9415de8..7b1d16226b7b 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -34,6 +34,7 @@
 #include <linux/backing-dev.h>
 #include <linux/mutex.h>
 #include <linux/capability.h>
+#include <linux/ctype.h>
 #include <linux/syscalls.h>
 #include <linux/memcontrol.h>
 #include <linux/poll.h>
@@ -88,6 +89,7 @@ static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data);
 
 static int xswap_create(int prio);
 static int xswap_destroy(u64 id);
+static int xswap_destroy_by_name(const char *name);
 
 /* Stable device id, assigned at create and never reused. */
 static atomic64_t xswap_id_counter = ATOMIC64_INIT(0);
@@ -275,6 +277,11 @@ static void xswap_sysfs_init(void)
 static inline void xswap_sysfs_init(void)
 {
 }
+
+static inline int xswap_destroy_by_name(const char *name)
+{
+	return -ENOENT;
+}
 #endif /* CONFIG_XSWAP */
 
 static void swap_range_alloc(struct swap_info_struct *si,
@@ -3513,8 +3520,21 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 
 	CLASS(filename, pathname)(specialfile);
 	victim = file_open_name(pathname, O_RDWR|O_LARGEFILE, 0);
-	if (IS_ERR(victim))
-		return PTR_ERR(victim);
+	if (IS_ERR(victim)) {
+		/*
+		 * An xswap device has no file, so /proc/swaps shows a name
+		 * that cannot be opened. Accept it here so swapoff -a works.
+		 * Only a missing name is taken for a device; a real file or
+		 * directory that failed for another reason must not match.
+		 */
+		if (PTR_ERR(victim) != -ENOENT)
+			return PTR_ERR(victim);
+
+		err = xswap_destroy_by_name(pathname->name);
+		if (err == -ENOENT)
+			return PTR_ERR(victim);
+		return err;
+	}
 
 	mapping = victim->f_mapping;
 	spin_lock(&swap_lock);
@@ -4504,6 +4524,38 @@ static int xswap_destroy(u64 id)
 
 	return __swapoff(p);
 }
+
+static int xswap_destroy_by_name(const char *name)
+{
+	const char *p;
+	char buf[24];
+	u64 id;
+	int err;
+
+	/* Strictly "xswap<digits>": no sign, no trailing newline. */
+	if (strncmp(name, "xswap", 5))
+		return -ENOENT;
+
+	p = name + 5;
+	if (!isdigit((unsigned char)*p))
+		return -ENOENT;
+	while (isdigit((unsigned char)*p))
+		p++;
+	if (*p)
+		return -ENOENT;
+
+	if (strscpy(buf, name + 5, sizeof(buf)) < 0)
+		return -ENOENT;
+
+	err = kstrtou64(buf, 10, &id);
+	if (err)
+		return -ENOENT;
+
+	err = xswap_destroy(id);
+	if (err == -ENODEV)
+		return -ENOENT;
+	return err;
+}
 #endif /* CONFIG_XSWAP */
 
 SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
-- 
2.54.0

[PATCH v5 13/19] mm, swap: add xswap grow trigger on cluster allocation

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

Grow the mapped xswap range before it runs out. When usage reaches 85%
of the mapped range, map more clusters from cluster_alloc_swap_entry().

Grow to about 65% usage. This leaves free space in the mapped range
before the next grow at 85%, so one grow is not immediately followed by
another.

Mapping pages can sleep. Grow before the allocation that needs them,
not after it fails. xswap_grow() drops the per-CPU cluster lock around
xswap_map_clusters(), because mapping can sleep and the lock only
protects the per-CPU cache. xswap is always SWP_SOLIDSTATE, so this path
does not use global_cluster_lock.

Link the new clusters even if swapoff cleared SWP_WRITEOK while mapping.
The caller keeps the device alive, and the allocation path checks the
flag before taking a cluster. If try_to_unuse() then fails, the
reinserted device needs these clusters; otherwise they go away with the
device.

After a failed scan, retry the free list once. A concurrent free or
grow may have added clusters.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 7b1d16226b7b..ba730fa19194 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -80,7 +80,18 @@
 	max_t(unsigned long,						\
 	      ((XSWAP_UNMAP_BATCH_PAGES * PAGE_SIZE) /		\
 	       sizeof(struct swap_cluster_info)), 16)
+/*
+ * Grow thresholds, as a percentage of the mapped range in use.  The target
+ * sits above the point where the shrink starts looking (50%) and below the
+ * next grow trigger (85%), so a grow cannot leave the range in a state the
+ * shrink would undo.
+ */
+#define XSWAP_GROW_UNTIL	65
+#define XSWAP_GROW_WHEN		85
+#define XSWAP_GROW_CHUNKS	4
 
+static bool xswap_should_grow(struct swap_info_struct *si);
+static unsigned long xswap_grow(struct swap_info_struct *si);
 static int xswap_map_clusters(struct swap_info_struct *si,
 			      unsigned long start_idx, unsigned long nr);
 static void xswap_unmap_clusters(struct swap_info_struct *si,
@@ -1359,6 +1370,18 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 	if (order && !(si->flags & SWP_BLKDEV))
 		return 0;
 
+#ifdef CONFIG_XSWAP
+	/*
+	 * Top the range up early.  Every path below leaves through `done`,
+	 * so this has to come first; mapping pages can sleep, and doing it
+	 * from the allocation that would otherwise fail leaves nothing to
+	 * fall back on.
+	 */
+	if ((si->flags & SWP_XSWAP) && (si->flags & SWP_WRITEOK) &&
+	    xswap_should_grow(si))
+		xswap_grow(si);
+#endif
+
 	if (!(si->flags & SWP_SOLIDSTATE)) {
 		/* Serialize HDD SWAP allocation for each device. */
 		spin_lock(&si->global_cluster_lock);
@@ -1435,6 +1458,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 		if (found)
 			goto done;
 	}
+
+#ifdef CONFIG_XSWAP
+	/* A concurrent free or grow may have added clusters; retry once. */
+	if (!found && (si->flags & SWP_XSWAP))
+		found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
+#endif
 done:
 	if (!(si->flags & SWP_SOLIDSTATE))
 		spin_unlock(&si->global_cluster_lock);
@@ -4168,6 +4197,77 @@ static void xswap_unmap_range(struct swap_info_struct *si,
 		__free_page(pages[i]);
 }
 
+static bool xswap_should_grow(struct swap_info_struct *si)
+{
+	unsigned long mapped = READ_ONCE(si->nr_clusters_mapped);
+
+	if (mapped >= READ_ONCE(si->nr_clusters_max))
+		return false;
+
+	return swap_usage_in_pages(si) * 100 >=
+	       mapped * SWAPFILE_CLUSTER * XSWAP_GROW_WHEN;
+}
+
+/*
+ * Map more of the cluster_info array, up to GROW_UNTIL in use.  The caller
+ * holds percpu_swap_cluster.lock, which this drops while it sleeps.
+ */
+static unsigned long xswap_grow(struct swap_info_struct *si)
+{
+	unsigned long mapped = READ_ONCE(si->nr_clusters_mapped);
+	unsigned long nr_new, want, i;
+	int ret;
+
+	/* The unmap below is paired with the caller's lock. */
+	lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
+
+	want = DIV_ROUND_UP(swap_usage_in_pages(si) * 100,
+			    XSWAP_GROW_UNTIL * SWAPFILE_CLUSTER);
+	if (want <= mapped)
+		return 0;
+
+	nr_new = rounddown(want - mapped, XSWAP_GROW_CLUSTERS);
+	if (!nr_new)
+		nr_new = XSWAP_GROW_CLUSTERS;
+	if (nr_new > XSWAP_GROW_CHUNKS * XSWAP_GROW_CLUSTERS)
+		nr_new = XSWAP_GROW_CHUNKS * XSWAP_GROW_CLUSTERS;
+	nr_new = min(nr_new, READ_ONCE(si->nr_clusters_max) - mapped);
+	if (!nr_new)
+		return 0;
+
+	/*
+	 * Mapping pages can sleep.  The lock only guards the per-cpu
+	 * cluster cache, which this path does not touch.
+	 */
+	local_unlock(&percpu_swap_cluster.lock);
+	ret = xswap_map_clusters(si, mapped, nr_new);
+	local_lock(&percpu_swap_cluster.lock);
+	if (ret)
+		return 0;
+
+	/*
+	 * Link the clusters even if swapoff cleared SWP_WRITEOK while we
+	 * mapped. The caller keeps the device alive, and nothing takes a
+	 * cluster while WRITEOK is clear. If try_to_unuse() fails, the
+	 * reinserted device needs them.
+	 */
+	for (i = mapped; i < mapped + nr_new; i++) {
+		struct swap_cluster_info *ci = &si->cluster_info[i];
+
+		/*
+		 * A concurrent grower may have taken these already;
+		 * only add the off-list ones.
+		 */
+		spin_lock(&ci->lock);
+		if (ci->flags == CLUSTER_FLAG_NONE)
+			move_cluster(si, ci, &si->free_clusters,
+				     CLUSTER_FLAG_FREE);
+		spin_unlock(&ci->lock);
+	}
+
+	return nr_new;
+}
+
 /* Caller must hold si->xswap_lock.  Cannot fail. */
 static void xswap_unmap_clusters(struct swap_info_struct *si,
 				 unsigned long start_idx, unsigned long nr)
-- 
2.54.0

[PATCH v5 14/19] mm, swap: add debugfs counters for xswap

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

The mapped range and the grow count are not visible from userspace.
This makes it hard to check if the grow thresholds work: we cannot see
how large the mapped range is, how much it grows, or how much free tail
it keeps.

Put this information in debugfs. It is a debugging aid, not a stable
interface, so it does not belong in sysfs or the ABI:

  # cat /sys/kernel/debug/xswap/xswap1
  type 0
  clusters_max 2038783
  clusters_mapped 292
  usage_pages 91456
  tail_free 16
  grows 3
  shrinks 0

The shrink counter is initialized here. It will start to increase when
the shrink path is added.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/swap.h |  3 ++
 mm/swapfile.c        | 78 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 81 insertions(+)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index f4eb1912d1e6..29623c5138ab 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -248,6 +248,9 @@ struct swap_info_struct {
 	unsigned long		nr_clusters_max;/* total clusters in the xswap address space */
 	unsigned long		nr_clusters_mapped; /* currently mapped cluster count */
 	struct mutex		xswap_lock;	/* serialize map/unmap operations */
+	struct dentry		*xswap_debugfs;	/* mapped range and counters */
+	atomic_long_t		xswap_grows;	/* grow operations so far */
+	atomic_long_t		xswap_shrinks;	/* shrink operations so far */
 #endif
 	struct list_head free_clusters; /* free clusters list */
 	struct list_head full_clusters; /* full clusters list */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index ba730fa19194..2624233b4fc2 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -46,12 +46,14 @@
 #include <linux/suspend.h>
 #include <linux/zswap.h>
 #include <linux/plist.h>
+#include <linux/debugfs.h>
 
 #include <asm/tlbflush.h>
 #include <linux/leafops.h>
 #include "swap_table.h"
 #include "internal.h"
 #include "swap.h"
+
 #define DEF_SWAP_PRIO  -1
 
 /*
@@ -92,6 +94,7 @@
 
 static bool xswap_should_grow(struct swap_info_struct *si);
 static unsigned long xswap_grow(struct swap_info_struct *si);
+static void xswap_debugfs_del(struct swap_info_struct *si);
 static int xswap_map_clusters(struct swap_info_struct *si,
 			      unsigned long start_idx, unsigned long nr);
 static void xswap_unmap_clusters(struct swap_info_struct *si,
@@ -3488,6 +3491,7 @@ static void free_swap_cluster_info(struct swap_info_struct *si)
 		 * pages, and the unmap helper handles an empty range.
 		 */
 		xswap_unmap_clusters(si, 0, si->nr_clusters_mapped);
+		xswap_debugfs_del(si);
 		free_vm_area(si->cluster_vm);
 		si->cluster_vm = NULL;
 		si->cluster_info = NULL;
@@ -4265,6 +4269,7 @@ static unsigned long xswap_grow(struct swap_info_struct *si)
 		spin_unlock(&ci->lock);
 	}
 
+	atomic_long_inc(&si->xswap_grows);
 	return nr_new;
 }
 
@@ -4336,6 +4341,73 @@ static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data)
 	*mapped_end = addr + PAGE_SIZE;
 	return 0;
 }
+static ssize_t xswap_debugfs_read(struct file *file, char __user *buf,
+				  size_t count, loff_t *ppos)
+{
+	struct swap_info_struct *si = file->private_data;
+	struct swap_cluster_info *ci;
+	unsigned long mapped, tail = 0;
+	char tmp[192];
+	int len;
+
+	/*
+	 * The same scan the shrink does.  xswap_lock is what keeps
+	 * cluster_info from being unmapped under it.
+	 */
+	mutex_lock(&si->xswap_lock);
+	mapped = READ_ONCE(si->nr_clusters_mapped);
+	if (si->cluster_info) {
+		while (mapped - tail > 1) {
+			ci = &si->cluster_info[mapped - tail - 1];
+			if (READ_ONCE(ci->count) ||
+			    READ_ONCE(ci->flags) != CLUSTER_FLAG_FREE)
+				break;
+			tail++;
+		}
+	}
+	len = scnprintf(tmp, sizeof(tmp),
+			"type %d\nclusters_max %lu\nclusters_mapped %lu\nusage_pages %lu\ntail_free %lu\ngrows %lu\nshrinks %lu\n",
+			si->type, READ_ONCE(si->nr_clusters_max), mapped,
+			swap_usage_in_pages(si), tail,
+			atomic_long_read(&si->xswap_grows),
+			atomic_long_read(&si->xswap_shrinks));
+	mutex_unlock(&si->xswap_lock);
+
+	return simple_read_from_buffer(buf, count, ppos, tmp, len);
+}
+
+static const struct file_operations xswap_debugfs_fops = {
+	.read	= xswap_debugfs_read,
+	.open	= simple_open,
+	.llseek	= default_llseek,
+};
+
+static struct dentry *xswap_debugfs_root;
+
+static void xswap_debugfs_add(struct swap_info_struct *si)
+{
+	char name[32];
+
+	if (!xswap_debugfs_root)
+		xswap_debugfs_root = debugfs_create_dir("xswap", NULL);
+	if (IS_ERR_OR_NULL(xswap_debugfs_root))
+		return;
+
+	/* debugfs_create_file() takes the name as it is; no formatting. */
+	snprintf(name, sizeof(name), "xswap%llu",
+		 (unsigned long long)si->xswap_id);
+	si->xswap_debugfs = debugfs_create_file(name, 0444,
+						xswap_debugfs_root, si,
+						&xswap_debugfs_fops);
+	if (IS_ERR(si->xswap_debugfs))
+		si->xswap_debugfs = NULL;
+}
+
+static void xswap_debugfs_del(struct swap_info_struct *si)
+{
+	debugfs_remove(si->xswap_debugfs);
+	si->xswap_debugfs = NULL;
+}
 #endif /* CONFIG_XSWAP */
 
 static int setup_swap_clusters_info(struct swap_info_struct *si,
@@ -4552,6 +4624,10 @@ static int xswap_create(int prio)
 	 */
 	si->ops = &swap_bdev_ops;
 
+	si->xswap_debugfs = NULL;
+	atomic_long_set(&si->xswap_grows, 0);
+	atomic_long_set(&si->xswap_shrinks, 0);
+
 	error = setup_swap_clusters_info(si, NULL, maxpages);
 	if (error)
 		goto bad_swap;
@@ -4561,6 +4637,8 @@ static int xswap_create(int prio)
 		goto bad_swap;
 
 	mutex_lock(&swapon_mutex);
+	/* Two creates would race to make the root directory. */
+	xswap_debugfs_add(si);
 	si->prio = prio;
 	si->list.prio = -si->prio;
 	si->avail_list.prio = -si->prio;
-- 
2.54.0

[PATCH v5 15/19] mm, swap: make xswap cluster_info accesses RCU-safe

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

An xswap cluster_info page can be unmapped at runtime.  The shrink path
that does this is added by a later patch; this patch establishes the
contract it relies on.  The allocation and reclaim paths drop ci->lock
to reclaim swap cache, then take it again; in that window the cluster
can be freed and its cluster_info page can go away.  Cover the whole
read/drop/reacquire window with RCU, so the cluster pointer stays valid
until the lock is taken again.

Also take RCU in swap_alloc_fast() and the hibernation slot allocation
path, which read cluster_info through a cached offset without holding
ci->lock.  A cached offset can outlive a shrink that did not see it, so
swap_alloc_fast() now loads nr_clusters_mapped with smp_load_acquire()
and fails the fast path when the offset is beyond it.

On the writer side, xswap_unmap_clusters() publishes the smaller range
with smp_store_release() before the memory goes away, invalidates the
per-cpu cluster cache, and waits one RCU grace period.  A reader that
already loaded a cached offset is then either seen by the cache flush or
waited out before the PTEs are cleared.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 82 +++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 73 insertions(+), 9 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 2624233b4fc2..bc30ab42203c 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1239,6 +1239,9 @@ static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,
 	lockdep_assert_held(&ci->lock);
 	VM_WARN_ON(!cluster_is_usable(ci, order));
 
+	/* ci is used without ci->lock; an xswap unmap waits for this. */
+	rcu_read_lock();
+
 	if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER)
 		goto out;
 
@@ -1267,6 +1270,7 @@ static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,
 out:
 	relocate_cluster(si, ci);
 	swap_cluster_unlock(ci);
+	rcu_read_unlock();
 	if (si->flags & SWP_SOLIDSTATE) {
 		this_cpu_write(percpu_swap_cluster.offset[order], next);
 		this_cpu_write(percpu_swap_cluster.si[order], si);
@@ -1310,6 +1314,9 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
 		to_scan = swap_usage_in_pages(si) / SWAPFILE_CLUSTER;
 
 	while ((ci = isolate_lock_cluster(si, &si->full_clusters))) {
+		/* As in alloc_swap_scan_cluster(). */
+		rcu_read_lock();
+
 		offset = cluster_offset(si, ci);
 		end = min(si->max, offset + SWAPFILE_CLUSTER);
 		to_scan--;
@@ -1334,6 +1341,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
 			relocate_cluster(si, ci);
 
 		swap_cluster_unlock(ci);
+		rcu_read_unlock();
 		if (to_scan <= 0)
 			break;
 
@@ -1664,11 +1672,36 @@ static bool swap_alloc_fast(struct folio *folio)
 	/*
 	 * Once allocated, swap_info_struct will never be completely freed,
 	 * so checking it's liveness by get_swap_device_info is enough.
+	 *
+	 * The cached offset indexes si->cluster_info, which xswap can
+	 * unmap; cover both the read and the use with RCU.
 	 */
+	rcu_read_lock();
 	si = this_cpu_read(percpu_swap_cluster.si[order]);
 	offset = this_cpu_read(percpu_swap_cluster.offset[order]);
-	if (!si || !offset || !get_swap_device_info(si))
+	if (!si || !offset || !get_swap_device_info(si)) {
+		rcu_read_unlock();
 		return false;
+	}
+#ifdef CONFIG_XSWAP
+	if (si->flags & SWP_XSWAP) {
+		/*
+		 * The cached offset can outlive a shrink that did not see
+		 * it, and the cluster_info page it indexes may be unmapped.
+		 * Fail the fast path; the slow path picks a live cluster.
+		 */
+		unsigned long mapped;
+
+		/* Pairs with the release store in xswap_unmap_clusters(). */
+		mapped = smp_load_acquire(&si->nr_clusters_mapped);
+
+		if (offset / SWAPFILE_CLUSTER >= mapped) {
+			put_swap_device(si);
+			rcu_read_unlock();
+			return false;
+		}
+	}
+#endif
 
 	ci = swap_cluster_lock(si, offset);
 	if (cluster_is_usable(ci, order)) {
@@ -1680,6 +1713,7 @@ static bool swap_alloc_fast(struct folio *folio)
 	}
 
 	put_swap_device(si);
+	rcu_read_unlock();
 	return folio_test_swapcache(folio);
 }
 
@@ -2472,8 +2506,11 @@ swp_entry_t swap_alloc_hibernation_slot(int type)
 	/*
 	 * Try the local cluster first if it matches the device. If
 	 * not, try grab a new cluster and override local cluster.
+	 *
+	 * Same RCU requirement as swap_alloc_fast().
 	 */
 	local_lock(&percpu_swap_cluster.lock);
+	rcu_read_lock();
 	pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);
 	pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);
 	if (pcp_si == si && pcp_offset) {
@@ -2483,6 +2520,7 @@ swp_entry_t swap_alloc_hibernation_slot(int type)
 		else
 			swap_cluster_unlock(ci);
 	}
+	rcu_read_unlock();
 	if (!offset)
 		offset = cluster_alloc_swap_entry(si, NULL);
 	local_unlock(&percpu_swap_cluster.lock);
@@ -3445,7 +3483,13 @@ static void wait_for_allocation(struct swap_info_struct *si)
 
 #ifdef CONFIG_XSWAP
 	if (si->flags & SWP_XSWAP) {
-		/* Pairs with the smp_store_release() in xswap_map_clusters(). */
+		/*
+		 * Pairs with the smp_store_release() in xswap_map_clusters().
+		 * A grow may publish after this read, but no slot comes from
+		 * those clusters once WRITEOK is clear.  Cached clusters are
+		 * inside the range; try_to_unuse()'s rescan catches any slot
+		 * taken after the walk.
+		 */
 		end = min(end, smp_load_acquire(&si->nr_clusters_mapped) *
 			  SWAPFILE_CLUSTER);
 	}
@@ -3527,9 +3571,9 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si)
 	for_each_possible_cpu(cpu) {
 		pcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu);
 		/*
-		 * Invalidate the percpu swap cluster cache, si->users
-		 * is dead, so no new user will point to it, just flush
-		 * any existing user.
+		 * Invalidate the per-cpu swap cluster cache.  Any reader that
+		 * has already loaded an entry is waited out by the caller's
+		 * synchronize_rcu().
 		 */
 		for (i = 0; i < SWAP_NR_ORDERS; i++)
 			cmpxchg(&pcp_si[i], si, NULL);
@@ -4289,12 +4333,33 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 
 	mutex_lock(&si->xswap_lock);
 
+	/*
+	 * Publish the smaller range before its memory goes away.  Readers
+	 * that use nr_clusters_mapped without xswap_lock pair with this
+	 * store using smp_load_acquire().
+	 */
+	smp_store_release(&si->nr_clusters_mapped, start_idx);
+
 	if (vm_start >= vm_end) {
-		WRITE_ONCE(si->nr_clusters_mapped, start_idx);
+		/*
+		 * The clusters sit inside one page that the preceding
+		 * cluster shares, so there is nothing to unmap.  The page
+		 * goes away with that cluster; publishing the smaller
+		 * cluster count is still correct.
+		 */
 		mutex_unlock(&si->xswap_lock);
 		return;
 	}
 
+	/*
+	 * A per-cpu cluster cache can still hold an offset in this range.
+	 * Invalidate those references, then wait out the readers that have
+	 * already loaded one, so that nobody can dereference cluster_info
+	 * past this point.  swapoff() needs the same before it releases.
+	 */
+	flush_percpu_swap_cluster(si);
+	synchronize_rcu();
+
 	npages = (vm_end - vm_start) >> PAGE_SHIFT;
 
 	if (npages <= totalram_pages()) {
@@ -4306,7 +4371,7 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 		if (pages) {
 			xswap_unmap_range(si, vm_start, vm_end, pages, npages);
 			kvfree(pages);
-			goto out_store;
+			goto out_unlock;
 		}
 	}
 
@@ -4325,8 +4390,7 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 			xswap_unmap_range(si, bstart, bend, batch_pages,
 					  ARRAY_SIZE(batch_pages));
 	}
-out_store:
-	WRITE_ONCE(si->nr_clusters_mapped, start_idx);
+out_unlock:
 	mutex_unlock(&si->xswap_lock);
 }
 
-- 
2.54.0

[PATCH v5 16/19] mm, swap: prepare xswap_unmap_clusters for range shrinking

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

Split xswap_unmap_clusters() into a locked helper and a wrapper that
takes xswap_lock. The shrink path already holds xswap_lock and must
call the helper directly.

This is a pure refactor. The existing path already falls back to
bounded batches when the range is too large for vm_area_unmap_pages();
the comments are expanded to explain why, and the cross-reference in
swap_alloc_fast() follows the new name.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index bc30ab42203c..097ed288b1ca 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1692,7 +1692,7 @@ static bool swap_alloc_fast(struct folio *folio)
 		 */
 		unsigned long mapped;
 
-		/* Pairs with the release store in xswap_unmap_clusters(). */
+		/* Pairs with the release store in xswap_unmap_clusters_locked(). */
 		mapped = smp_load_acquire(&si->nr_clusters_mapped);
 
 		if (offset / SWAPFILE_CLUSTER >= mapped) {
@@ -4318,8 +4318,8 @@ static unsigned long xswap_grow(struct swap_info_struct *si)
 }
 
 /* Caller must hold si->xswap_lock.  Cannot fail. */
-static void xswap_unmap_clusters(struct swap_info_struct *si,
-				 unsigned long start_idx, unsigned long nr)
+static void xswap_unmap_clusters_locked(struct swap_info_struct *si,
+					unsigned long start_idx, unsigned long nr)
 {
 	unsigned long start_addr = (unsigned long)si->cluster_info +
 				   (size_t)start_idx * sizeof(struct swap_cluster_info);
@@ -4331,8 +4331,6 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 	unsigned int noreclaim_flags;
 	unsigned long npages, idx;
 
-	mutex_lock(&si->xswap_lock);
-
 	/*
 	 * Publish the smaller range before its memory goes away.  Readers
 	 * that use nr_clusters_mapped without xswap_lock pair with this
@@ -4347,7 +4345,6 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 		 * goes away with that cluster; publishing the smaller
 		 * cluster count is still correct.
 		 */
-		mutex_unlock(&si->xswap_lock);
 		return;
 	}
 
@@ -4362,6 +4359,11 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 
 	npages = (vm_end - vm_start) >> PAGE_SHIFT;
 
+	/*
+	 * vm_area_unmap_pages() refuses a range above totalram_pages() and
+	 * returns without clearing the PTEs.  Freeing those pages would
+	 * leave dangling mappings, so use the batch path for that case.
+	 */
 	if (npages <= totalram_pages()) {
 		/* Reclaim here would re-enter xswap_map_clusters() and its lock. */
 		noreclaim_flags = memalloc_noreclaim_save();
@@ -4371,11 +4373,11 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 		if (pages) {
 			xswap_unmap_range(si, vm_start, vm_end, pages, npages);
 			kvfree(pages);
-			goto out_unlock;
+			return;
 		}
 	}
 
-	/* No memory for the array, or too large: unmap in bounded batches. */
+	/* No memory for the array, or too large a range: unmap in batches. */
 	for (idx = start_idx; idx < start_idx + nr; idx += XSWAP_UNMAP_BATCH_CLUSTERS) {
 		unsigned long batch = min(start_idx + nr - idx,
 					  XSWAP_UNMAP_BATCH_CLUSTERS);
@@ -4390,7 +4392,13 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 			xswap_unmap_range(si, bstart, bend, batch_pages,
 					  ARRAY_SIZE(batch_pages));
 	}
-out_unlock:
+}
+
+static void xswap_unmap_clusters(struct swap_info_struct *si,
+				 unsigned long start_idx, unsigned long nr)
+{
+	mutex_lock(&si->xswap_lock);
+	xswap_unmap_clusters_locked(si, start_idx, nr);
 	mutex_unlock(&si->xswap_lock);
 }
 
-- 
2.54.0

[PATCH v5 17/19] mm, swap: add xswap shrink trigger and defer it to a workqueue

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

The shrink cannot run directly from __free_cluster(), because that path
holds ci->lock.  xswap_try_shrink() takes xswap_lock and can sleep in
synchronize_rcu() and vm_area_unmap_pages().  __free_cluster() schedules
xswap_shrink_work instead, and teardown disables the work.

The shrink unmaps the free tail of the cluster_info mapping.  It costs
two RCU grace periods: one after draining the per-CPU cache and before
the tail is rechecked, and one inside the unmap helper.  The second one
is needed because a cluster can be allocated and freed between the
recheck and the unmap, and go back into the per-CPU cache.  xswap devices
are SWP_SOLIDSTATE, so the per-CPU cache is the only one that needs
draining.

The unmap helper takes a wait flag.  The shrink passes true.  Teardown
passes false because __swapoff() has already drained and waited.  The
create failure path also passes false because it has no readers.

Count shrink operations in xswap_shrinks.  Initialize the work in
xswap_create() so free_swap_cluster_info() can always call
disable_work_sync().  Using disable_work_sync() instead of only
cancel_work_sync() also rejects a schedule that read SWP_WRITEOK just
before swapoff.  Then no work stays pending when the swap_info is reused
and INIT_WORK() runs again.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 include/linux/swap.h |   1 +
 mm/swapfile.c        | 233 ++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 211 insertions(+), 23 deletions(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 29623c5138ab..1e552a8b421b 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -247,6 +247,7 @@ struct swap_info_struct {
 	u64			xswap_id;	/* stable id, shown to userspace */
 	unsigned long		nr_clusters_max;/* total clusters in the xswap address space */
 	unsigned long		nr_clusters_mapped; /* currently mapped cluster count */
+	struct work_struct	xswap_shrink_work; /* deferred shrink trigger */
 	struct mutex		xswap_lock;	/* serialize map/unmap operations */
 	struct dentry		*xswap_debugfs;	/* mapped range and counters */
 	atomic_long_t		xswap_grows;	/* grow operations so far */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 097ed288b1ca..2e5383036fdc 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -66,12 +66,13 @@
 
 #ifdef CONFIG_XSWAP
 /*
- * xswap: dynamically grow the cluster_info array via a VM_SPARSE area.
+ * xswap: dynamically grow and shrink the cluster_info array via a
+ * VM_SPARSE area.
  *
- * XSWAP_GROW_CLUSTERS is the number of clusters to map in one grow
- * operation.  It is set to the number of cluster_info structs that
- * fit in a single page (at least 16), so that the vmalloc page table
- * overhead is proportional to the number of clusters mapped.
+ * XSWAP_GROW_CLUSTERS is the number of clusters to map/unmap in one
+ * grow/shrink operation: the number of cluster_info structs that fit in
+ * a single page (at least 16), so that the vmalloc page table overhead
+ * is proportional to the number of clusters mapped.
  */
 #define XSWAP_GROW_CLUSTERS \
 	max_t(unsigned long, PAGE_SIZE / sizeof(struct swap_cluster_info), 16)
@@ -82,6 +83,14 @@
 	max_t(unsigned long,						\
 	      ((XSWAP_UNMAP_BATCH_PAGES * PAGE_SIZE) /		\
 	       sizeof(struct swap_cluster_info)), 16)
+/*
+ * Shrink thresholds, as a percentage of the mapped range in use.  The stop
+ * point sits inside (SHRINK_WHEN, GROW_WHEN), so shrinking cannot leave the
+ * range in a state that wakes the grow up.
+ */
+#define XSWAP_SHRINK_WHEN	50
+#define XSWAP_SHRINK_UNTIL	70
+
 /*
  * Grow thresholds, as a percentage of the mapped range in use.  The target
  * sits above the point where the shrink starts looking (50%) and below the
@@ -100,6 +109,7 @@ static int xswap_map_clusters(struct swap_info_struct *si,
 static void xswap_unmap_clusters(struct swap_info_struct *si,
 				 unsigned long start_idx, unsigned long nr);
 static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data);
+static void xswap_try_shrink(struct swap_info_struct *si);
 
 static int xswap_create(int prio);
 static int xswap_destroy(u64 id);
@@ -866,6 +876,16 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info
 	swap_cluster_free_table(ci);
 	move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
 	ci->order = 0;
+#ifdef CONFIG_XSWAP
+	/*
+	 * Only xswap devices, and not while the device is being torn down.
+	 * A racing swapoff can still clear SWP_WRITEOK after this check;
+	 * the work function checks it again, so a late schedule is benign.
+	 */
+	if ((READ_ONCE(si->flags) & (SWP_XSWAP | SWP_WRITEOK)) ==
+	    (SWP_XSWAP | SWP_WRITEOK))
+		schedule_work(&si->xswap_shrink_work);
+#endif
 }
 
 /*
@@ -3515,6 +3535,8 @@ static void free_swap_cluster_info(struct swap_info_struct *si)
 	if (si->flags & SWP_XSWAP) {
 		unsigned long nr_mapped;
 
+		/* A late schedule must not be pending when the work is reinitialized. */
+		disable_work_sync(&si->xswap_shrink_work);
 		/*
 		 * Cluster 0 keeps the bad header slot, so it never empties
 		 * and __free_cluster() never frees its table.
@@ -3560,8 +3582,10 @@ static void free_swap_cluster_info(struct swap_info_struct *si)
 }
 
 /*
- * Called after swap device's reference count is dead, so
- * neither scan nor allocation will use it.
+ * Invalidate the per-cpu cluster cache for @si.  A caller that has not
+ * already killed @si->users and waited must follow with synchronize_rcu()
+ * before unmapping cluster_info, since a reader may already have loaded
+ * a cached offset.
  */
 static void flush_percpu_swap_cluster(struct swap_info_struct *si)
 {
@@ -3570,11 +3594,7 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si)
 
 	for_each_possible_cpu(cpu) {
 		pcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu);
-		/*
-		 * Invalidate the per-cpu swap cluster cache.  Any reader that
-		 * has already loaded an entry is waited out by the caller's
-		 * synchronize_rcu().
-		 */
+		/* Invalidate the per-cpu swap cluster cache for @si. */
 		for (i = 0; i < SWAP_NR_ORDERS; i++)
 			cmpxchg(&pcp_si[i], si, NULL);
 	}
@@ -3681,6 +3701,15 @@ static int __swapoff(struct swap_info_struct *p)
 	struct file *swap_file = NULL;
 	int err;
 
+#ifdef CONFIG_XSWAP
+	if (p->flags & SWP_XSWAP) {
+		cancel_work_sync(&p->xswap_shrink_work);
+		/* Wait out a shrink racing us from the sysfs write path. */
+		mutex_lock(&p->xswap_lock);
+		mutex_unlock(&p->xswap_lock);
+	}
+#endif
+
 	wait_for_allocation(p);
 
 	set_current_oom_origin();
@@ -4317,9 +4346,13 @@ static unsigned long xswap_grow(struct swap_info_struct *si)
 	return nr_new;
 }
 
-/* Caller must hold si->xswap_lock.  Cannot fail. */
+/*
+ * Caller must hold si->xswap_lock.  Cannot fail.  @wait drains the per-cpu
+ * cluster cache and waits one RCU grace period before unmapping.
+ */
 static void xswap_unmap_clusters_locked(struct swap_info_struct *si,
-					unsigned long start_idx, unsigned long nr)
+					unsigned long start_idx, unsigned long nr,
+					bool wait)
 {
 	unsigned long start_addr = (unsigned long)si->cluster_info +
 				   (size_t)start_idx * sizeof(struct swap_cluster_info);
@@ -4348,14 +4381,16 @@ static void xswap_unmap_clusters_locked(struct swap_info_struct *si,
 		return;
 	}
 
-	/*
-	 * A per-cpu cluster cache can still hold an offset in this range.
-	 * Invalidate those references, then wait out the readers that have
-	 * already loaded one, so that nobody can dereference cluster_info
-	 * past this point.  swapoff() needs the same before it releases.
-	 */
-	flush_percpu_swap_cluster(si);
-	synchronize_rcu();
+	if (wait) {
+		/*
+		 * A per-cpu cluster cache can still hold an offset in this
+		 * range.  Invalidate those references, then wait out the
+		 * readers that have already loaded one, so that nobody can
+		 * dereference cluster_info past this point.
+		 */
+		flush_percpu_swap_cluster(si);
+		synchronize_rcu();
+	}
 
 	npages = (vm_end - vm_start) >> PAGE_SHIFT;
 
@@ -4398,7 +4433,11 @@ static void xswap_unmap_clusters(struct swap_info_struct *si,
 				 unsigned long start_idx, unsigned long nr)
 {
 	mutex_lock(&si->xswap_lock);
-	xswap_unmap_clusters_locked(si, start_idx, nr);
+	/*
+	 * The teardown caller has already drained and waited in __swapoff();
+	 * the create failure path has no readers.
+	 */
+	xswap_unmap_clusters_locked(si, start_idx, nr, false);
 	mutex_unlock(&si->xswap_lock);
 }
 
@@ -4413,6 +4452,153 @@ static int xswap_mapped_end(pte_t *pte, unsigned long addr, void *data)
 	*mapped_end = addr + PAGE_SIZE;
 	return 0;
 }
+
+/*
+ * Automatic reclaim: leave one chunk of the free tail mapped as slack, so
+ * that the next allocation has somewhere to land, and only unmap once
+ * several chunks can go, so the unmap is worth the two RCU grace periods
+ * it costs.
+ */
+#define XSWAP_SHRINK_SLACK	XSWAP_GROW_CLUSTERS
+#define XSWAP_SHRINK_MIN	(XSWAP_GROW_CLUSTERS * 8)
+
+static void xswap_shrink_work_fn(struct work_struct *work)
+{
+	struct swap_info_struct *si = container_of(work,
+			struct swap_info_struct, xswap_shrink_work);
+
+	if (!(READ_ONCE(si->flags) & SWP_WRITEOK))
+		return;
+	xswap_try_shrink(si);
+}
+
+/*
+ * Try to shrink the cluster_info tail: unmap contiguous free clusters
+ * at the end of the mapped range.
+ */
+static void xswap_try_shrink(struct swap_info_struct *si)
+{
+	struct swap_cluster_info *ci;
+	unsigned long nr_mapped, nr_tail, keep, nr_unmap, start_idx, i;
+
+	if (!(si->flags & SWP_XSWAP))
+		return;
+
+	mutex_lock(&si->xswap_lock);
+
+	/* A swapoff raced us and is about to walk this mapping. */
+	if (!(READ_ONCE(si->flags) & SWP_WRITEOK))
+		goto out_unlock;
+
+	nr_mapped = READ_ONCE(si->nr_clusters_mapped);
+	if (nr_mapped <= 1)	/* keep cluster 0 */
+		goto out_unlock;
+
+	/*
+	 * Reclaim on our own, but only once the mapped range is at most
+	 * SHRINK_WHEN in use: growth is demand driven, so reclaiming on a
+	 * smaller dip would only map the same clusters again, and every
+	 * unmap costs two RCU grace periods.
+	 */
+	if (swap_usage_in_pages(si) * 100 >
+	    nr_mapped * SWAPFILE_CLUSTER * XSWAP_SHRINK_WHEN)
+		goto out_unlock;
+
+	/*
+	 * Count the free clusters at the tail of the mapped range.  Scanned,
+	 * not tracked: the count must be exact to size the unmap, and an
+	 * incremental count falls behind on out-of-order frees.
+	 */
+	nr_tail = 0;
+	while (nr_mapped - nr_tail > 1) {
+		ci = &si->cluster_info[nr_mapped - nr_tail - 1];
+		if (READ_ONCE(ci->count) ||
+		    READ_ONCE(ci->flags) != CLUSTER_FLAG_FREE)
+			break;
+		nr_tail++;
+	}
+	if (nr_tail < XSWAP_SHRINK_SLACK + XSWAP_SHRINK_MIN)
+		goto out_unlock;
+
+	/*
+	 * Stop at SHRINK_UNTIL rather than at the end of the tail, or the
+	 * range comes out full enough for the grow to be woken.  Below the
+	 * tail everything stays mapped, and one chunk of tail with it.
+	 */
+	keep = DIV_ROUND_UP(swap_usage_in_pages(si) * 100,
+			    XSWAP_SHRINK_UNTIL * SWAPFILE_CLUSTER);
+	if (keep < nr_mapped - nr_tail + XSWAP_SHRINK_SLACK)
+		keep = nr_mapped - nr_tail + XSWAP_SHRINK_SLACK;
+
+	if (nr_mapped < keep + XSWAP_SHRINK_MIN)
+		goto out_unlock;
+
+	nr_unmap = rounddown(nr_mapped - keep, XSWAP_GROW_CLUSTERS);
+	if (!nr_unmap)
+		goto out_unlock;
+	start_idx = nr_mapped - nr_unmap;
+
+	/*
+	 * An allocation can enter a cluster from the per-cpu cache without
+	 * going through the lists, so drain the cache first.  xswap devices
+	 * are SWP_SOLIDSTATE, so the per-cpu cache is the only cache.  Only
+	 * then do the checks below hold: a cluster taken from a list takes
+	 * si->lock, which is what the check holds.
+	 *
+	 * The unmap helper below drains the cache and waits again: a cluster
+	 * can be taken and freed between this recheck and the unmap, putting
+	 * it back in the per-cpu cache.
+	 */
+	flush_percpu_swap_cluster(si);
+	synchronize_rcu();
+
+	/*
+	 * Only shrink a run that reaches the mapped end; otherwise
+	 * truncating nr_clusters_mapped would orphan the active tail.
+	 * move_cluster() takes si->lock under ci->lock, so ci->lock is
+	 * taken with trylock here to avoid the reverse order.
+	 *
+	 * With si->lock held, the writes below need no ci->lock: a FREE
+	 * cluster leaves the free list only in isolate_lock_cluster(),
+	 * every move_cluster() caller holds ci->lock (the trylock above
+	 * catches one), a FREE cluster's count is 0 and only falls, and
+	 * the drained cache holds no FREE cluster.
+	 */
+	spin_lock(&si->lock);
+	for (i = start_idx; i < nr_mapped; i++) {
+		ci = &si->cluster_info[i];
+		if (READ_ONCE(ci->flags) != CLUSTER_FLAG_FREE)
+			break;
+		if (!spin_trylock(&ci->lock)) {
+			spin_unlock(&si->lock);
+			goto out_unlock;
+		}
+		spin_unlock(&ci->lock);
+	}
+	if (i != nr_mapped) {
+		spin_unlock(&si->lock);
+		goto out_unlock;
+	}
+
+	for (i = start_idx; i < nr_mapped; i++) {
+		ci = &si->cluster_info[i];
+		list_del_init(&ci->list);
+		WRITE_ONCE(ci->flags, CLUSTER_FLAG_NONE);
+	}
+	spin_unlock(&si->lock);
+
+	/*
+	 * A cluster can be taken and freed between the recheck above and the
+	 * unmap, putting it back in the per-cpu cache.  Drain and wait again.
+	 */
+	xswap_unmap_clusters_locked(si, start_idx, nr_unmap, true);
+	atomic_long_inc(&si->xswap_shrinks);
+
+out_unlock:
+	mutex_unlock(&si->xswap_lock);
+}
+
+/* Not an interface: how far the mapped range has moved, for testing. */
 static ssize_t xswap_debugfs_read(struct file *file, char __user *buf,
 				  size_t count, loff_t *ppos)
 {
@@ -4674,6 +4860,7 @@ static int xswap_create(int prio)
 
 	INIT_WORK(&si->discard_work, swap_discard_work);
 	INIT_WORK(&si->reclaim_work, swap_reclaim_work);
+	INIT_WORK(&si->xswap_shrink_work, xswap_shrink_work_fn);
 
 	/* Cluster-aligned, so no cluster holds a slot past si->max. */
 	maxpages = rounddown(maxpages, SWAPFILE_CLUSTER);
-- 
2.54.0

[PATCH v5 18/19] mm, swap: do not give an xswap slot to a cgroup that cannot zswap

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

At this point, an xswap slot holds only a zswap-backed page. It has no
storage of its own. If the folio's cgroup cannot take another zswap
page, giving it an xswap slot is wasted: zswap refuses the page and the
slot cannot hold it.

Skip xswap devices when the cgroup cannot zswap, and try a physical swap
device instead.

The check is obj_cgroup_may_zswap(). It is false when memory.zswap.max
is 0 anywhere in the hierarchy, or when the hierarchy has already
reached that limit. So the routing follows how much the cgroup has in
zswap, not only how it is configured. zswap_store() uses the same check,
but it first tries to shrink the memcg. The allocation-time check does
not try to shrink, so it is stricter.

Count active xswap devices, and do the cgroup lookup only when one
exists. Update the counter when the device is enabled, before a full
device can skip the avail-list insert, so a full device is still counted.

A device can appear between that read and the allocation. So if the
slot the allocator picked is on an xswap device, check the cgroup
again; when it cannot zswap, drop the slot and retry without xswap
devices.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 63 insertions(+), 6 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 2e5383036fdc..efb48ef71611 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -326,6 +326,8 @@ static DEFINE_SPINLOCK(swap_lock);
 static unsigned int nr_swapfiles;
 atomic_long_t nr_swap_pages;
 atomic_t nr_real_swapfiles;
+/* Active xswap devices, as in nr_real_swapfiles. */
+static atomic_t nr_xswap_files;
 /*
  * Some modules use swappable objects and may try to swap them out under
  * memory pressure (via the shrinker). Before doing so, they may wish to
@@ -1519,7 +1521,9 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 		lockdep_assert_held(&si->lock);
 		si->flags &= ~SWP_WRITEOK;
 		/* Count active devices, not merely those on the avail list. */
-		if (!(si->flags & SWP_XSWAP))
+		if (si->flags & SWP_XSWAP)
+			atomic_sub(1, &nr_xswap_files);
+		else
 			atomic_sub(1, &nr_real_swapfiles);
 		atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 	} else {
@@ -1558,7 +1562,9 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
 		 * Count the device here, not after adding it to the list: a
 		 * full device is still active and skips the add below.
 		 */
-		if (!(si->flags & SWP_XSWAP))
+		if (si->flags & SWP_XSWAP)
+			atomic_add(1, &nr_xswap_files);
+		else
 			atomic_add(1, &nr_real_swapfiles);
 	} else {
 		if (!(READ_ONCE(si->flags) & SWP_WRITEOK))
@@ -1682,7 +1688,7 @@ static bool get_swap_device_info(struct swap_info_struct *si)
  * Fast path try to get swap entries with specified order from current
  * CPU's swap entry pool (a cluster).
  */
-static bool swap_alloc_fast(struct folio *folio)
+static bool swap_alloc_fast(struct folio *folio, bool may_zswap)
 {
 	unsigned int order = folio_order(folio);
 	struct swap_cluster_info *ci;
@@ -1722,6 +1728,11 @@ static bool swap_alloc_fast(struct folio *folio)
 		}
 	}
 #endif
+	if (!may_zswap && (si->flags & SWP_XSWAP)) {
+		put_swap_device(si);
+		rcu_read_unlock();
+		return false;
+	}
 
 	ci = swap_cluster_lock(si, offset);
 	if (cluster_is_usable(ci, order)) {
@@ -1738,13 +1749,19 @@ static bool swap_alloc_fast(struct folio *folio)
 }
 
 /* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static void swap_alloc_slow(struct folio *folio, bool may_zswap)
 {
 	struct swap_info_struct *si, *next;
 
 	spin_lock(&swap_avail_lock);
 start_over:
 	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
+		/*
+		 * Do not rotate a skipped device: the walk follows the
+		 * rotation and would spin forever.
+		 */
+		if (!may_zswap && (si->flags & SWP_XSWAP))
+			continue;
 		/* Rotate the device and switch to a new cluster */
 		plist_requeue(&si->avail_list, &swap_avail_head);
 		spin_unlock(&swap_avail_lock);
@@ -2067,6 +2084,17 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
 	return err;
 }
 
+/* Can the folio's cgroup take another zswap page? */
+static bool folio_may_zswap(struct folio *folio)
+{
+	struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
+	bool may_zswap = !objcg || obj_cgroup_may_zswap(objcg);
+
+	if (objcg)
+		obj_cgroup_put(objcg);
+	return may_zswap;
+}
+
 /**
  * folio_alloc_swap - allocate swap space for a folio
  * @folio: folio we want to move to swap
@@ -2083,10 +2111,23 @@ int folio_alloc_swap(struct folio *folio)
 {
 	unsigned int order = folio_order(folio);
 	unsigned int size = 1 << order;
+	/* True unless the gate below finds an xswap device to route away from. */
+	bool may_zswap = true;
+	bool zswap_checked = false;
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
 
+	/*
+	 * A folio that zswap will not take must not get an xswap slot. With
+	 * no xswap device to route away from, skip the lookup; a device that
+	 * appears before we pick one is handled after the allocation.
+	 */
+	if (atomic_read(&nr_xswap_files)) {
+		may_zswap = folio_may_zswap(folio);
+		zswap_checked = true;
+	}
+
 	if (order) {
 		/*
 		 * Reject large allocation when THP_SWAP is disabled. Check below
@@ -2107,8 +2148,8 @@ int folio_alloc_swap(struct folio *folio)
 
 again:
 	local_lock(&percpu_swap_cluster.lock);
-	if (!swap_alloc_fast(folio))
-		swap_alloc_slow(folio);
+	if (!swap_alloc_fast(folio, may_zswap))
+		swap_alloc_slow(folio, may_zswap);
 	local_unlock(&percpu_swap_cluster.lock);
 
 	if (!order && unlikely(!folio_test_swapcache(folio))) {
@@ -2116,6 +2157,22 @@ int folio_alloc_swap(struct folio *folio)
 			goto again;
 	}
 
+	/*
+	 * The xswap gate was skipped because there was no device when we
+	 * looked. One may have appeared since; check before keeping the
+	 * xswap slot the allocator may have picked.
+	 */
+	if (unlikely(!zswap_checked && atomic_read(&nr_xswap_files) &&
+		     folio_test_swapcache(folio) &&
+		     (__swap_entry_to_info(folio->swap)->flags & SWP_XSWAP))) {
+		zswap_checked = true;
+		may_zswap = folio_may_zswap(folio);
+		if (!may_zswap) {
+			swap_cache_del_folio(folio);
+			goto again;
+		}
+	}
+
 	/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
 	if (unlikely(mem_cgroup_try_charge_swap(folio))) {
 		swap_cache_del_folio(folio);
-- 
2.54.0

[PATCH v5 19/19] mm, swap: skip the overcommit probe when removing xswap

[Baoquan He]

From: Baoquan He <hebaoquan@kylinos.cn>

swap_info_remove() probes whether Committed_AS still fits under the
commit limit left after the device's pages are subtracted. An xswap
device's pages are its address space, not backing storage, but they
raised the commit limit at creation. A workload can commit into that
headroom, after which the probe refuses to destroy the device.

Skip the probe for xswap until the overcommit accounting for it is
settled. Real swap devices keep the existing protection.

Signed-off-by: Baoquan He <hebaoquan@kylinos.cn>
---
 mm/swapfile.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index efb48ef71611..84b9a8dddff2 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -3736,10 +3736,17 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
  */
 static int swap_info_remove(struct swap_info_struct *p)
 {
-	if (!security_vm_enough_memory_mm(current->mm, p->pages))
-		vm_unacct_memory(p->pages);
-	else
-		return -ENOMEM;
+	/*
+	 * The device's pages are its address space, not backing storage.
+	 * The commit limit it added may have been committed against, so
+	 * skip the probe until the limit stops counting that space.
+	 */
+	if (!(p->flags & SWP_XSWAP)) {
+		if (!security_vm_enough_memory_mm(current->mm, p->pages))
+			vm_unacct_memory(p->pages);
+		else
+			return -ENOMEM;
+	}
 
 	spin_lock(&p->lock);
 	del_from_avail_list(p, true);
-- 
2.54.0

Re: [PATCH v5 00/19] mm, swap: extendable swap devices (xswap phase I)

[Baoquan He]

On 11/09/26 at 01:54pm, Baoquan He wrote:

Forgot adding change history versions before v4, please refer to below
v4 patchset cover letter if you want to check.

[PATCH v4 00/14] mm, swap: extendable swap devices (xswap phase I)
https://lore.kernel.org/all/20261003003136.750633-1-hebaoquan@kylinos.cn/T/#u

> Changelog
> =========
> v4 -> v5:
> - Rebased onto the latest mm-new (2c5dafa7f289). 
> 
> - A device now has a stable id instead of exposing the swap type:
>   /proc/swaps shows xswap<id>, the debugfs file is named by the id and
>   prints the type as well, and destroy takes the id. The id is assigned
>   per boot and never reused, so a value read earlier cannot name a
>   different device.
> 
> - swapoff accepts the name: swapoff xswap<id> and swapoff -a tear the
>   device down, while a real file or directory keeps precedence. A name
>   that does not exist reports -ENOENT; the sysfs destroy keeps -ENODEV
>   for an id it does not know. The interfaces are now documented in
>   Documentation/ABI/testing/sysfs-kernel-mm-xswap and zswap.rst.
> 
> - CONFIG_XSWAP_DEFAULT_MAX sets the default size of a new device. It
>   takes a percentage of RAM, a size, or "max", and defaults to "200%";
>   xswap.max= still overrides it.
> 
> - The cluster_info accesses are RCU-safe: the allocation fast path bounds
>   a cached cluster against the mapped range, and the shrink drains the
>   per-cpu cache and waits an RCU grace period before unmapping.
> 
> - xswap_grow() now links the clusters it mapped even when swapoff has
>   already cleared SWP_WRITEOK, so a failed swapoff cannot leave them
>   mapped and on no list.
> 
> - swap_info_remove() skips the commit limit probe for xswap; real swap
>   devices keep it. See the overcommit note above.
> 
> - The cgroup zswap gate is checked again after the allocation when an
>   xswap device appeared after the initial lookup, so a slot is not kept
>   for a cgroup that cannot zswap.
> 
> - The patches are reorganized: the VM_SPARSE work is split into helpers
>   and the xswap use, the max/pages widening moves before the sysfs
>   interface, the shrink trigger now carries the workqueue deferral, and
>   the unmap helper is prepared for range shrinking in its own patch. The
>   series is now phase I of four, with the swap-table backend records as
>   a phase of their own.
> 
> Baoquan He (18):
>   mm, swap: refactor free_swap_cluster_info to take swap_info_struct
>   mm, swap: add VM_SPARSE cluster-array helpers for xswap
>   mm, swap: use a VM_SPARSE cluster array for xswap devices
>   mm, swap: widen swap_info_struct max/pages to unsigned long
>   mm, swap: add a sysfs interface to create xswap devices
>   mm, swap: refactor swapoff and add xswap_destroy
>   mm, swap: require zswap for xswap devices
>   mm, swap: use a stable id for xswap devices
>   mm, swap: let the command line set the xswap device size
>   mm, swap: add CONFIG_XSWAP_DEFAULT_MAX
>   mm, swap: let swapoff destroy xswap devices by name
>   mm, swap: add xswap grow trigger on cluster allocation
>   mm, swap: add debugfs counters for xswap
>   mm, swap: make xswap cluster_info accesses RCU-safe
>   mm, swap: prepare xswap_unmap_clusters for range shrinking
>   mm, swap: add xswap shrink trigger and defer it to a workqueue
>   mm, swap: do not give an xswap slot to a cgroup that cannot zswap
>   mm, swap: skip the overcommit probe when removing xswap
> 
> Chris Li (1):
>   mm: xswap support for zswap
> 
>  .../ABI/testing/sysfs-kernel-mm-xswap         |   43 +
>  .../admin-guide/kernel-parameters.txt         |    8 +
>  Documentation/admin-guide/mm/zswap.rst        |    9 +
>  MAINTAINERS                                   |    1 +
>  include/linux/swap.h                          |   17 +-
>  mm/Kconfig                                    |   20 +
>  mm/page_io.c                                  |   19 +
>  mm/swap_state.c                               |    9 +
>  mm/swapfile.c                                 | 1569 +++++++++++++++--
>  mm/zswap.c                                    |   11 +-
>  10 files changed, 1593 insertions(+), 113 deletions(-)
>  create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-xswap
> 
> 
> base-commit: 2c5dafa7f289d760f4492260b0988029aab478b0
> -- 
> 2.54.0
> 
>