[RFC PATCH v3 00/18] mm/swap: introduce priority queues for swap device allocation

[Lian Wang <lianux.mm@gmail.com>]

Hi all,

This is v3 of Kairui's swap priority queue RFC [1, 2], retaining the
original authorship. It separates device selection from device-local
cluster allocation to support integration with the ongoing swap tier
work, while preserving existing swap priorities, the swapon interface,
and memcg swap accounting and limits.

The global per-CPU cluster cache couples device choice to cluster
locality. Moving that cache below device selection gives each device its
own allocation hints. The available plist's shared rotation is then
replaced by a priority-ordered array of same-priority device rings, with
per-CPU starting offsets and quotas. A task-local cursor keeps each retry
walk stable. Rotational devices retain per-device sequential hints shared
across CPUs.

This remains an RFC. I would appreciate feedback on the allocation model
and its integration with the ongoing swap tier work.

Following v1, the changes address related allocation problems:

1) A cache above device selection couples the chosen device to the cached
   cluster. Device-local ownership lets selection determine the eligible
   device before consulting its hints, which is useful for tiering.

2) Rotating the available plist changes shared state under
   swap_avail_lock. Per-CPU readers keep rotation state off that shared
   selection path, with a quota measured in base pages.

3) Device rotation was tied to cluster draining and the fast/slow split.
   Selecting a device first also lets allocation handle its pending
   discard directly, with a bounded amount of synchronous work.

The series is organized as follows:

  01-04: Cleanup, device-state boundaries and iteration helpers.
  05-07: Locking, writer-serialized publication and accounting.
  08:    Restore per-device per-CPU cluster locality.
  09:    Keep cluster metadata population out of direct reclaim.
  10-11: Introduce the safely published queue and remove the avail plist.
  12:    Bound synchronous discard to one cluster per device attempt.
  13-14: Preserve priority-aware throttle selection while removing the
         active plist and unused plist_requeue API.
  15-18: Reuse reader capacity, advance cursors directly, bypass singleton
         rotation and redistribute reader phases on membership changes.

Test results
============
The kernel-build and BRD tests follow the v1 workloads on a newer
baseline. The results below compare the unpatched baseline 4dbe24720eb3
with queue revision db66c58fac8f. The additional usemem and lock-call
results compare the same baseline with 18ec4b962755. These are
measurements of the named revisions; this posting contains revision
fdfe5fac0d5b.

Eight-ZRAM kernel build
----------------------
The workload used eight 8-GiB ZRAM devices and make -j96 in 2-GiB and
3-GiB memcgs. Each case had 12 fresh-output builds per revision, with no
warm-up. The workload source was fixed at 4dbe, using x86 defconfig and
the targets bzImage modules. The revisions ran on separate boots, Before
first; the 12 samples are within-boot repeats.

Mean system CPU time in seconds (lower is better):

                         Before       Queue (db66)       delta
  2-GiB memcg          4469.0100          4457.1575       -0.27%
  3-GiB memcg          2352.7325          2338.9325       -0.59%

Mean elapsed time changed from 170.1283 to 170.3675 s (+0.14%) at 2 GiB
and from 136.6125 to 136.6775 s (+0.05%) at 3 GiB. No clear regression
was observed in these kernel-build runs.

Twelve-BRD usemem
----------------
The same Before/db66 pair used twelve 8-GiB BRD devices and a private
memcg with 24 GiB of memory and 64 GiB of swap allowance.

  usemem --init-time -O -y -x -n 32 1536M

There was one complete invocation per revision. Mapped throughput below
is the requested 48 GiB divided by elapsed time, in MiB/s.

                         Before       Queue (db66)       delta
  elapsed time (s)          7.00                7.11       +1.57%
  mapped MiB/s           7021.71             6913.08       -1.55%

All 50 kernel-build and BRD invocations exited successfully, reported no
memcg OOM events and performed swap I/O.

Additional eight-ZRAM usemem and lock-call measurements
-----------------------------------------------------
A separate experiment used eight 8-GiB equal-priority ZRAM devices and
the same usemem command, with memory.max=24 GiB, memory.swap.max=64 GiB
and memory.zswap.max=0 in the workload's memcg. Both kernels ran on the
same x86-64 host with configs differing only in LOCALVERSION. There was
one workload timing run per revision, with no warm-up, on separate boots,
18ec first.

                         Before       Queue (18ec)       delta
  elapsed time (s)          7.97                8.59       +7.78%
  system CPU time (s)     175.13              192.39       +9.86%
  pswpout (pages)      6,317,548           6,317,684         +136

Both runs exited successfully and completed 32 workers, with no memcg
OOM events. The times increased in this single-sample comparison; the
cause has not been established.

Separate sampled runs recorded 12,451 swap_avail_lock acquisitions at
the selected allocation callsites in Before. Source and binary inspection
confirmed that these calls are absent from the 18ec allocation reader
path. Availability-update callsites recorded zero acquisitions during
both sampled workload windows. This shows removal of the shared
selection-lock calls, rather than an end-to-end speedup.

Current revision checks
-----------------------
All 18 patches in fdfe replay to their expected intermediate trees and
pass checkpatch. Static review and userspace checks of the actual helper
code with ASan/UBSan cover publication, ownership, rollback, selector
progression and throttle priority selection. These checks use userspace
shims and do not constitute kernel runtime testing.

Cooperation with swap tiers
==========================
Device-local cluster hints remain below device selection, so a future
tier policy can determine the eligible candidates before those hints are
used.

V1 proposed taking tiering first and rebasing this queue onto it as an
optimization [1]. I am happy to follow that route. Another option, if
reviewers agree, is to take the refactoring under the existing swap
priority model first, then work with Youngjun on adapting the queue and
reader scope to the tier infrastructure [3].

Youngjun, which ordering would fit your current work better? Are there
constraints we should preserve to make a later rebase straightforward?

The integration should have one authority for device membership,
swapon/swapoff publication and lifetime. Tier policy would determine the
eligible candidates before any device-local hint is used. Within-tier
priority semantics need agreement with that work; this series adds no
tier ABI or per-memcg device-selection control.

Link: https://lore.kernel.org/all/20260714-swap-pcp-priq-v1-0-de9b164ed419@tencent.com/ [1]
Link: https://lore.kernel.org/all/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@gmail.com/ [2]
Link: https://lore.kernel.org/all/20260916183437.2946306-1-youngjun.park@lge.com/ [3]

---
Changes in v3:
- Serialize file/ref/writable publication when removing swapon_mutex.
- Establish metadata and writer allocation safety at their first use.
- Preserve priority-aware blkcg throttling when removing the active list.
- Add reader-capacity reuse, direct cursor advancement, singleton
  selection and membership redistribution.
- Present the completed ZRAM/BRD results with their tested revisions and
  sample counts, including the observed time increases.
- Add measurements of shared selection-lock calls.

---
Kairui Song (12):
  mm/swap: remove unused parameter for reading swap header
  mm/swap: slightly cleanup the code for hibernation error handling
  mm/swap: cleanup and document swap device availability flag usage
  mm/swap: introduce swap device iteration helper
  mm/swap: change the swapon lock into a percpu rwsem
  mm/swap: serialize swap publication before removing swapon mutex
  mm/swap: consolidate swap inuse accounting helpers
  mm/swap: add priority queue for swap device allocation
  mm/swap: remove available list
  mm/swap: bound synchronous discard during allocation
  mm/swap: drop swap active plist
  lib/plist.c: remove requeue function

Lian Wang (5):
  mm/swap: avoid direct reclaim when populating swap cluster metadata
  mm/swap: reuse per-CPU queue readers across priority changes
  mm/swap: advance the global queue cursor directly between priority
    rings
  mm/swap: skip reader rotation for single-device priority rings
  mm/swap: redistribute queue readers after membership changes

Youngjun Park (1):
  mm/swap: change back to use each swap device's percpu cluster

 include/linux/plist.h |    2 -
 include/linux/swap.h  |   43 +-
 lib/plist.c           |   64 --
 mm/swap.h             |   12 +-
 mm/swapfile.c         | 1421 ++++++++++++++++++++++++++---------------
 5 files changed, 926 insertions(+), 616 deletions(-)
---
base-commit: 4dbe24720eb3811563cd56b35d87ddf3f9beed2a

Best regards,
-- 
Lian Wang <lianux.mm@gmail.com>