Re: [PATCH v3 00/14] mm, swap: extendable swap devices (xswap)

[Klara Modin <klarasmodin@gmail.com>]

On 2026-09-16 18:19:07 +0800, Baoquan He wrote:
> xswap is a swap device with no backing storage. Swapped-out pages live                                
> in zswap. Its cluster_info[] array lives in a VM_SPARSE vmalloc area,                                 
> and the area is grown and shrunk on demand as swap usage changes.                                     
>                                                                                                       
> The problem being solved is the static size of compressed swap. Both   
> zram and zswap need the size fixed in advance, and neither gives memory 
> back when the workload shrinks. The solution should be a device whose
> size can scale up/down as per usage. xswap does that by mapping the
> metadata lazily instead of reserving it for the whole range.
> 
> Design   
> ------                           
> - si->cluster_info[] stays a plain array. Access is still
>   &si->cluster_info[offset / SWAPFILE_CLUSTER]: no per-access branch, no 
>   RCU discipline, no tear-down state machine, no NULL return.              
> - Only an initial chunk is mapped at creation. The rest of the address     
>   space is reserved, not allocated, so an idle device costs nothing.
> - Growth is driven by allocation. When no free cluster is left and the       
>   address space has room, the next chunk is mapped and added to the free  
>   list. No userspace involvement.
> - Shrink is driven by frees. The free tail is scanned, and whole chunks
>   are unmapped once the mapped range is at most half in use and several
>   chunks can go. One chunk is left mapped as slack, so the next                                       
>   allocation does not map it straight back. A ceiling lowered below the                               
>   mapped range skips the half-in-use rule and is enforced at once.
> 

> Size
> ----
> A device starts at 1xRAM, rounded down to the cluster. That costs
> nothing, because the mapping is lazy. The underlying address space is
> 2xRAM. An optional per-device cap,
> /sys/kernel/mm/xswap/type<N>/limit, lets an admin lower the ceiling;
> the excess is unmapped right away. Grow and shrink both work without
> it. Creating a device requires zswap.

So I can't set an xswap device to more than twice the RAM? I suppose I
could create multiple xswap devices, but it would get tedious fast on
systems which have a different amount of memory. Is there a particular
reason for this limit? I think I could create an arbitrarily large xswap
device with your previous version which needed the specially crafted
swapfile (with only the header).

As I wrote in the other thread, I would rather not have to set a limit
at all, or at least have a limit I'm sure I won't reach.

> 
> Interface
> ---------
>   /sys/kernel/mm/xswap/create           write an optional priority
>   /sys/kernel/mm/xswap/destroy          write a swap type
>   /sys/kernel/mm/xswap/type<N>/limit    read/write, in pages
> The device shows up in /proc/swaps as xswap<N>.
> 
> Note
> ----
> Writeback, rmap lookup, etc. are consumers of this base. I have a
> writeback prototype on top of this base and will post it as a reference.
> 
> Testing
> -------
> qemu KVM guest, 8G RAM.
> 
> Tested create/destroy, raising and lowering the limit (including clamping
> when it is written below the pages in use), shrink with live entries, and
> 2000 create/destroy cycles for leaks; all passed.
> 
> The workload is memhog: it faults in N GB of anonymous memory inside a
> cgroup with a much smaller memory.max, forcing the pages to swap.
> Set MEMHOG_FILL=pattern: the default fill is all-zero pages that zswap
> compresses to almost nothing, so the device never fills.
> 
>   # echo 1 > /sys/module/zswap/parameters/enabled
>   # mkdir -p /sys/fs/cgroup/xswap_limit
>   # echo max > /sys/fs/cgroup/xswap_limit/memory.swap.max
>   # MEM="MEMHOG_FILL=pattern numactl --cpunodebind=0 --membind=0 ./memhog"
> 
> 1. Create and destroy
> 
>    # echo > /sys/kernel/mm/xswap/create
>    # swapon
>    NAME    TYPE  SIZE USED PRIO
>    xswap0  xswap 7.8G   0B   -1
>    # cat /sys/kernel/mm/xswap/type0/limit
>    2035199
>    # echo 0 > /sys/kernel/mm/xswap/destroy
>    # swapon
>    (nothing)
> 
>    limit is in 4 KiB pages; 2035199 is RAM (2034976 pages) rounded up to a
>    whole number of clusters. The device starts at RAM, not twice RAM.
> 
> 2. The cap holds
> 
>    # echo 2147483648 > /sys/fs/cgroup/xswap_limit/memory.max
>    # ( echo $$ > /sys/fs/cgroup/xswap_limit/cgroup.procs; eval $MEM 11 300 ) &
>    # awk '/SwapTotal|SwapFree/' /proc/meminfo
>    SwapTotal:       8140796 kB
>    SwapFree:         354012 kB
> 
>    The cgroup runs out of room before the device does and the OOM killer
>    takes the workload ??? that is the pass signal. SwapFree never exceeds
>    SwapTotal, so nr_swap_pages never goes negative.
> 
> 3. Raising the cap
> 
>    # echo 3052543 > /sys/kernel/mm/xswap/type0/limit
>    # awk '/SwapTotal/' /proc/meminfo
>    SwapTotal:      12210172 kB
>    # ( echo $$ > /sys/fs/cgroup/xswap_limit/cgroup.procs; eval $MEM 11 300 ) &
> 
>    No OOM this time: 2473705 pages in use against 2034976 pages of RAM, so
>    usage goes past RAM.
> 
> 4. Lowering the cap below the pages in use
> 
>    # echo 1000000 > /sys/kernel/mm/xswap/type0/limit
>    # cat /sys/kernel/mm/xswap/type0/limit
>    2426879
>    # awk '/SwapTotal|SwapFree/' /proc/meminfo
>    SwapTotal:       9707516 kB
>    SwapFree:             860 kB
> 
>    The write is clamped up to the clusters covering the pages in use, so
>    the free slots in the partially used top cluster stay accounted for.
> 
> 5. Shrink with live entries, then destroy
> 
>    # echo 4069887 > /sys/kernel/mm/xswap/type0/limit
>    # sleep 60
>    # awk '/SwapFree/' /proc/meminfo
>    SwapFree:       16279548 kB
> 
>    The shrink unmapped the tail ??? the state find_next_to_unuse() must
>    survive. Put live entries back and destroy:
> 
>    # ( echo $$ > /sys/fs/cgroup/xswap_limit/cgroup.procs; eval $MEM 3 300 ) &
>    # echo max > /sys/fs/cgroup/xswap_limit/memory.max
>    # echo 0 > /sys/kernel/mm/xswap/destroy
>    # swapon
>    (nothing)
> 
>    dmesg stays clean across create, shrink, swapoff and destroy.
> 
> 6. 2000 create/destroy cycles, diffing /proc/slabinfo before and after:
>    the largest growth is 142 objects. One object leaked per cycle would
>    be 2000.
> 
> Performance
> -----------
> (qemu KVM guest, 8G RAM, zram as the swap device)                                                         
> This series should not slow down a kernel that never creates an xswap
> device. I measured that overhead by comparing the base tree with this
> series. Both were built with the same .config and CONFIG_XSWAP=y, and no
> xswap device was created. I ran three 3G MADV_PAGEOUT workloads, three
> rounds each, alternating between the two kernels across reboots. I
> counted retired instructions per page swapped out with perf stat:
> 
>   workload                      base       series     delta
>   swapout                       50824.8    50866.2    +0.08%
>   swapout and swapin            63770.4    63796.8    +0.04%
>   swapout into a full device    67729.9    67707.8    -0.03%
> 
> Two runs of the same kernel differ by less than 0.1%, so the differences
> above are real, not measurement noise. I cannot use wall clock time for
> this comparison, because two runs of the same kernel differ by more than
> the two kernels do.
> 
> Changelog
> =========
> v2 -> v3:
> - Rebased onto the latest mm-new.
> 
> - The grow path now honors the user-set ceiling (si->nr_clusters) instead
>   of growing up to nr_clusters_max, and a ceiling below the mapped range
>   is unmapped exactly instead of rounded to a chunk (patches 12 and 14).
> 
> - The limit write clamps the ceiling up to the clusters covering the pages
>   in use, replacing the earlier WARN_ONCE; si->pages becomes mutable at
>   runtime (patch 13).
> 
> - Minor comment and cleanup changes.
> 
> v1->v2:
> - Patch 1 (mm: zswap: return -ENOENT when the swap device is gone) is not
>   part of this series; it was posted separately.
> 
> - There is only one size knob now. The runtime ceiling and the debugfs
>   per-device limit are gone. All that is left is the optional per-device
>   cap, /sys/kernel/mm/xswap/type<N>/limit. Grow and shrink work without
>   it.
> 
> - The shrink no longer keeps its own count of the free tail. It scans the
>   tail instead, and dropping the counter also removes a call from the
>   cluster allocation path.
> 
> - The priority is no longer a patch of its own. The create attribute
>   takes it:
>     echo 100 > /sys/kernel/mm/xswap/create
> 
> RFC v3 -> v1
> - Add patch 16 to support setting xswap device priority at creation.
>   The create sysfs interface (/sys/kernel/mm/xswap/create) previously
>   hardcoded every new device's priority to DEF_SWAP_PRIO, it now
>   accepts an optional priority:
> 
>     echo "<percent> [<prio>]" > /sys/kernel/mm/xswap/create
> 
> - Bug fix: xswap_lock init ordering. mutex_init(&si->xswap_lock) was called
>   after xswap_map_clusters() (which locks it), i.e. locking an uninitialized
>   mutex. Init now before the first xswap_map_clusters() call. Thanks to Klara.
> 
> - Bug fix: Fixes a compile error in !CONFIG_XSWAP builds. xswap_debugfs_root
>   is declared inside CONFIG_XSWAP ifdeffery scope, so the ungarded use
>   caused error when CONFIG_XSWAP is off.
> 
> RFC v2-> RFC v3:
> - Replace the "header-only swap file + swapon" creation hack with a
>   proper file-less device created and destroyed via sysfs
>   (/sys/kernel/mm/xswap/{create,destroy}).  This required the
>   __swapoff() refactor and the free_swap_cluster_info() signature
>   change (patches 4, 6, 14).
> 
> - Require zswap: refuse to create an xswap device when zswap is
>   unavailable (patch 15).
> 
> - Split the unrelated zswap -ENOENT fix out of the series into a
>   standalone patch (patch 1).
> 
> - Fix nr_free_tail over-counting on concurrent grow, shrink leaking
>   detached clusters on early bail-out, a re-init race on cluster
>   spinlocks in xswap_map_clusters(), the nr_clusters_mapped update
>   ordering, and swapoff accessing the shrinker-unmapped cluster tail.
> 
> - Minor cleanups (checkpatch, /proc/swaps alignment, commit messages).
> 
> RFC v1-> RFC v2:
> - Added __GFP_HIGH | __GFP_NOMEMALLOC to alloc_page() and kmalloc_array()
>   in the grow path, plus memalloc_noreclaim_save/restore() wrapping,
>   to prevent the grow path from consuming emergency memory reserves
>   or recursing into swap under PF_MEMALLOC. This is folded into patch 3.
>   This was pointed out by Nhat.
> 
> - Folded the mutex serialization fix into the cluster grow patch (patch
>   3). This is suggested by Nhat.
> 
> - Fixed coding style issues: corrected indentation of declarations in
>   xswap_unmap_clusters(), removed unnecessary block scope around the
>   err variable in xswap_map_clusters().
> 
> - Rebased onto mm-unstable
> 
> Baoquan He (13):
>   mm, swap: add CONFIG_XSWAP and xswap fields to swap_info_struct
>   mm, swap: refactor free_swap_cluster_info to take swap_info_struct
>   mm, swap: add xswap cluster grow via VM_SPARSE vmalloc
>   mm, swap: add sysfs create interface for xswap
>   mm, swap: add xswap grow trigger on cluster allocation
>   mm, swap: add xswap_try_shrink and shrink trigger on cluster free
>   mm, swap: free backing pages in xswap_unmap_clusters
>   mm, swap: defer xswap shrink to workqueue to avoid lock recursion
>   mm, swap: refactor swapoff and add xswap_destroy
>   mm, swap: require zswap for xswap devices
>   mm, swap: cap xswap growth at nr_clusters
>   mm, swap: add sysfs per-device size limit for xswap
>   mm, swap: shrink xswap to the ceiling when it drops
> 
> Chris Li (1):
>   mm: xswap support for zswap
> 
>  include/linux/swap.h |   26 +-
>  mm/Kconfig           |    9 +
>  mm/page_io.c         |   19 +
>  mm/swap_state.c      |    4 +
>  mm/swapfile.c        | 1240 +++++++++++++++++++++++++++++++++++++-----
>  mm/zswap.c           |    7 +-
>  6 files changed, 1174 insertions(+), 131 deletions(-)
> 
> 
> base-commit: baa8de2f3448d1466a888a805c18d01c998fe052
> -- 
> 2.54.0
>