* [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection
@ 2026-09-16 18:34 Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 1/4] mm: swap: introduce swap tier infrastructure Youngjun Park
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Youngjun Park @ 2026-09-16 18:34 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, baoquan.he, baohua, yosry, joshua.hahnjy,
taejoon.song, her0gyugyu, lianux.mm
This is RFC v11 of the swap tier series [1], reworked after the v10
discussion [2][3].
Motivation
==========
After the v10 discussion, I would like to first land the parts that do
not affect users, before anything touches memcg, with two goals in mind.
- Serve as a bridge that causes no trouble when the tier interface and
memcg support are introduced later.
- Make per-cgroup swap device selection possible.
While thinking this through, I came across some insights that I'd like
to discuss with the community. Hence this RFC v11.
As a starting point, let's look at what swap priority already gives us.
- Devices with different priorities are used in priority order.
- Devices with the same priority are used round-robin.
From this point of view, a priority itself can be seen as a tier rather
than a property of a device. Each distinct priority is a tier, the
devices with that priority hang off it, and allocation walks the tiers.
This is one of the insights that led me here. In the v10 design, a tier
was a priority range, so one tier could contain multiple priorities.
That raised two questions.
- If devices in a tier are allocated in priority order, aren't they
effectively different tiers?
- Is there a real use case for several same-priority round-robin
groups inside one tier?
If each distinct priority is its own tier, both questions go away.
A tier is one round-robin group, and ordering exists only between
tiers. (Anyone who wants priority-based allocation inside a tier can
still get it through a separate tier interface later.
mix allocation policy? also possible. we have interface.)
Either way, no explicit tier interface like the one in v10 is needed
yet.
Even when real tiers work are introduced, the view stays the same.
- We want an ordering between tier A and tier B.
- Devices in one tier form one service speed group. How they share
allocations may become flexible, and will normally stay round-robin
as it is today.
- On top of that, we will add backend transfer between tiers after
swap virtualization.
This series changes the current swap code to follow that view, which
gives us the following.
- Per-cgroup swap can later fit into a swap tier interface without
trouble.
- A base for the per-priority allocation queue series [4].
- It is close to a refactoring of the existing structure, so userspace
is not tied to an implementation.
Kairui, Lian, would it be okay to use this as the groundwork for [4]?
What do you think?
Per-cgroup swap in debugfs
==========================
Patches 3 and 4 let a memory cgroup choose its tiers through debugfs.
# swapon -p 100 /dev/nvme0n1p2
# swapon -p 50 /dev/sdb2
# cat /sys/kernel/debug/swap/tiers
Idx Prio
0 100
1 50
# echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
Bit i of the mask is tier i, so /batch swaps only to sdb2. A tier keeps
its index for its lifetime, so the mask keeps selecting the same tier
across swapon and swapoff.
Masks are kept in a list keyed by cgroup ID rather than in struct
mem_cgroup, so the series does not touch memcg code and needs no Kconfig
option. A mask applies only to its own cgroup and is not inherited by
child cgroups.
This debugfs interface is a stepping stone. Once the tier model
settles, /sys/kernel/mm/swap/tiers would list and name the tiers, and a
memcg knob such as memory.swap.tiers.max would take tier names and turn
them into this mask.
Future direction
================
This is not done in this series. If the tiers introduced here settle,
which differs from giving tiers priority ranges, work can continue
along these lines depending on the discussion.
The following are common to both.
- The per-priority allocation queue series from Kairui and Lian [4].
- Add /sys/kernel/mm/swap/tiers.
- A memcg memory.swap.tiers.max interface.
- Per-tier allocation policy (an extension that needs discussion).
If tiers are assigned by priority ranges as before, the tiers here have
to be related to those ranges. When a tier is assigned at runtime, the
devices that fall into its range are merged into that tier.
If tiers are extended the way they work here, a tier stays a priority
and the interface only adds names. A tier starts with a default name,
such as its priority, and can be renamed through the tier interface.
(I think there are a lot of possible way which we can discuss.)
Change log
==========
v11
- Reworked after the v10 discussion [2][3], with no new user ABI.
- A swap tier is now a single swap priority, created and removed by
swapon and swapoff. /sys/kernel/mm/swap/tiers and CONFIG_NR_SWAP_TIERS
are dropped.
- The active and available swap device lists move into the tiers.
(#1, #2 patches)
- Per-cgroup tier selection moves from memory.swap.tiers.max to debugfs.
(#3 patch)
- Masks are kept in a cgroup ID list instead of struct mem_cgroup, and
are no longer inherited by child cgroups. (#3 patch)
- Dropped the selftests, which tested the dropped interfaces.
- Rebased on recent mm-new.
- v10 link: https://lore.kernel.org/linux-mm/20260713025644.170839-1-youngjun.park@lge.com/
Changes up to v10 are in the v10 cover letter [1].
[1] https://lore.kernel.org/linux-mm/20260713025644.170839-1-youngjun.park@lge.com/
[2] https://lore.kernel.org/linux-mm/amDCIl51NoNPL7Op@cmpxchg.org/
[3] https://lore.kernel.org/linux-mm/amIlqHQ40baRoz3O@cmpxchg.org/
[4] https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-0-68d3d925578c@gmail.com/
Youngjun Park (4):
mm: swap: introduce swap tier infrastructure
mm: swap: allocate swap slots from swap tiers
mm: swap: add a debugfs interface for memcg tier selection
mm: swap: filter swap allocation by memcg tier mask
Documentation/mm/index.rst | 1 +
Documentation/mm/swap-tier.rst | 55 +++++
MAINTAINERS | 3 +
include/linux/swap.h | 5 +-
mm/Makefile | 2 +-
mm/swap.h | 2 +
mm/swap_tier.c | 421 +++++++++++++++++++++++++++++++++
mm/swap_tier.h | 71 ++++++
mm/swapfile.c | 173 ++++++++------
9 files changed, 658 insertions(+), 75 deletions(-)
create mode 100644 Documentation/mm/swap-tier.rst
create mode 100644 mm/swap_tier.c
create mode 100644 mm/swap_tier.h
base-commit: b08a65b93426d86e3f354d655d6225397b591877
--
2.48.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC PATCH v11 1/4] mm: swap: introduce swap tier infrastructure
2026-09-16 18:34 [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Youngjun Park
@ 2026-09-16 18:34 ` Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 2/4] mm: swap: allocate swap slots from swap tiers Youngjun Park
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Youngjun Park @ 2026-09-16 18:34 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, baoquan.he, baohua, yosry, joshua.hahnjy,
taejoon.song, her0gyugyu, lianux.mm
Introduce the "swap tier" as an abstraction over swap devices, so that
swap allocation is organized on a tier basis rather than on a flat list
of devices.
Swap priority already points this way. Devices with different priorities
are used in priority order, and devices with the same priority are used
round-robin. A priority therefore already behaves like a tier, a group
of devices that share one service speed. This patch gives that group an
explicit structure, without adding any user interface.
A swap tier is the set of swap devices that share a priority. A tier is
created when the first device with its priority is swapped on, and
removed when the last one is swapped off. The active tiers are kept
sorted by priority for allocation.
Making the tier own its devices gives a same-priority group the data
structures it needs and lays the groundwork for tier-based allocation,
onto which per-cgroup swap device selection can later be fit.
No tier feature and no user-visible change are introduced here. This
only prepares the ground for them.
Suggested-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
MAINTAINERS | 2 +
include/linux/swap.h | 2 +-
mm/Makefile | 2 +-
mm/swap.h | 1 +
mm/swap_tier.c | 114 +++++++++++++++++++++++++++++++++++++++++++
mm/swap_tier.h | 37 ++++++++++++++
mm/swapfile.c | 66 +++++++++++++------------
7 files changed, 192 insertions(+), 32 deletions(-)
create mode 100644 mm/swap_tier.c
create mode 100644 mm/swap_tier.h
diff --git a/MAINTAINERS b/MAINTAINERS
index e4412c3d8d45..37f353015cae 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17410,6 +17410,8 @@ F: mm/page_io.c
F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
+F: mm/swap_tier.c
+F: mm/swap_tier.h
F: mm/swapfile.c
MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 43155e122b5c..22ccb4b5801e 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -237,7 +237,7 @@ struct swap_info_struct {
struct percpu_ref users; /* indicate and keep swap device valid. */
unsigned long flags; /* SWP_USED etc: see above */
signed short prio; /* swap priority of this type */
- struct plist_node list; /* entry in swap_active_head */
+ struct plist_node list; /* entry in its swap tier */
signed char type; /* strange name for an index */
unsigned int max; /* size of this swap device */
struct swap_cluster_info *cluster_info; /* array, one entry per cluster */
diff --git a/mm/Makefile b/mm/Makefile
index 2a3ec53d62ee..d89a7abadc46 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -76,7 +76,7 @@ ifdef CONFIG_MMU
obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
endif
-obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o
+obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o swap_tier.o
obj-$(CONFIG_ZSWAP) += zswap.o
obj-$(CONFIG_HAS_DMA) += dmapool.o
obj-$(CONFIG_HUGETLBFS) += hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
diff --git a/mm/swap.h b/mm/swap.h
index b3b54c28929a..4de6b9b0f261 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -36,6 +36,7 @@ struct swap_io_ctx;
#define swap_entry_order(order) 0
#endif
+extern spinlock_t swap_lock;
extern struct swap_info_struct *swap_info[];
/*
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
new file mode 100644
index 000000000000..8ed1427cee09
--- /dev/null
+++ b/mm/swap_tier.c
@@ -0,0 +1,114 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/swap.h>
+
+#include "swap.h"
+#include "swap_tier.h"
+
+#define MAX_SWAPTIER MAX_SWAPFILES
+
+static struct swap_tier swap_tiers[MAX_SWAPTIER];
+
+/* active swap priority list, sorted in descending order */
+LIST_HEAD(swap_tier_active_list);
+/* unused swap_tier object */
+static LIST_HEAD(swap_tier_inactive_list);
+
+#define for_each_tier(tier, idx) \
+ for (idx = 0, tier = &swap_tiers[0]; idx < MAX_SWAPTIER; \
+ idx++, tier = &swap_tiers[idx])
+
+/*
+ * Naming Convention:
+ * swap_tiers_*() - Public/exported functions
+ * swap_tier_*() - Private/internal functions
+ */
+
+static struct swap_tier *swap_tier_lookup(short prio)
+{
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (tier->prio == prio)
+ return tier;
+ }
+
+ return NULL;
+}
+
+/* Insert new tier into the active list sorted by priority. */
+static void swap_tier_activate(struct swap_tier *new)
+{
+ struct list_head *pos = &swap_tier_active_list;
+ struct swap_tier *tier;
+
+ for_each_active_tier(tier) {
+ if (tier->prio <= new->prio) {
+ pos = &tier->list;
+ break;
+ }
+ }
+
+ list_add_tail(&new->list, pos);
+}
+
+static void swap_tier_inactivate(struct swap_tier *tier)
+{
+ list_move_tail(&tier->list, &swap_tier_inactive_list);
+}
+
+void swap_tiers_init(void)
+{
+ struct swap_tier *tier;
+ int idx;
+
+ BUILD_BUG_ON(BITS_PER_TYPE(int) < MAX_SWAPTIER);
+
+ for_each_tier(tier, idx) {
+ plist_head_init(&tier->active_head);
+ INIT_LIST_HEAD(&tier->list);
+ swap_tier_inactivate(tier);
+ }
+}
+
+static struct swap_tier *swap_tier_prepare(short prio)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+
+ /* A tier holds at least one device, so one is always unused. */
+ tier = list_first_entry(&swap_tier_inactive_list,
+ struct swap_tier, list);
+
+ list_del_init(&tier->list);
+ tier->prio = prio;
+
+ return tier;
+}
+
+void swap_tiers_assign_dev(struct swap_info_struct *swp)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+
+ tier = swap_tier_lookup(swp->prio);
+ if (!tier) {
+ tier = swap_tier_prepare(swp->prio);
+ swap_tier_activate(tier);
+ }
+
+ plist_add(&swp->list, &tier->active_head);
+}
+
+void swap_tiers_remove_dev(struct swap_info_struct *swp)
+{
+ struct swap_tier *tier;
+
+ lockdep_assert_held(&swap_lock);
+
+ tier = swap_tier_lookup(swp->prio);
+ plist_del(&swp->list, &tier->active_head);
+ if (plist_head_empty(&tier->active_head))
+ swap_tier_inactivate(tier);
+}
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
new file mode 100644
index 000000000000..3dce716d23f6
--- /dev/null
+++ b/mm/swap_tier.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SWAP_TIER_H
+#define _SWAP_TIER_H
+
+#include <linux/list.h>
+#include <linux/plist.h>
+#include <linux/types.h>
+
+/* Forward declarations */
+struct swap_info_struct;
+
+/*
+ * struct swap_tier - structure representing a swap tier.
+ *
+ * @prio: priority of the swap devices in the tier.
+ * @active_head: swap devices in the tier.
+ * @list: linkage into swap_tier_active_list or swap_tier_inactive_list.
+ */
+struct swap_tier {
+ short prio;
+ struct plist_head active_head;
+ struct list_head list;
+};
+
+extern struct list_head swap_tier_active_list;
+
+#define for_each_active_tier(tier) \
+ list_for_each_entry(tier, &swap_tier_active_list, list)
+
+/* Initialization and application */
+void swap_tiers_init(void);
+
+/* Tier assignment */
+void swap_tiers_assign_dev(struct swap_info_struct *swp);
+void swap_tiers_remove_dev(struct swap_info_struct *swp);
+
+#endif /* _SWAP_TIER_H */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index c0eddccfaca2..8201ae779833 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -48,6 +48,7 @@
#include "swap_table.h"
#include "internal.h"
#include "swap.h"
+#include "swap_tier.h"
static void swap_range_alloc(struct swap_info_struct *si,
unsigned int nr_entries);
@@ -61,9 +62,9 @@ static void move_cluster(struct swap_info_struct *si,
* lazily allocated & freed swap device info struts, and SWP_USED indicates
* which device is used, ~SWP_USED devices and can be reused.
*
- * Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.
+ * Also protects the swap tiers, total_swap_pages, and the SWP_WRITEOK flag.
*/
-static DEFINE_SPINLOCK(swap_lock);
+DEFINE_SPINLOCK(swap_lock);
static unsigned int nr_swapfiles;
atomic_long_t nr_swap_pages;
/*
@@ -83,17 +84,11 @@ bool swap_migration_ad_supported;
static const char Bad_file[] = "Bad swap file entry ";
static const char Bad_offset[] = "Bad swap offset entry ";
-/*
- * all active swap_info_structs
- * protected with swap_lock, and ordered by priority.
- */
-static PLIST_HEAD(swap_active_head);
-
/*
* all available (active, not full) swap_info_structs
* protected with swap_avail_lock, ordered by priority.
- * This is used by folio_alloc_swap() instead of swap_active_head
- * because swap_active_head includes all swap_info_structs,
+ * This is used by folio_alloc_swap() instead of the active lists of
+ * the swap tiers because those include all swap_info_structs,
* but folio_alloc_swap() doesn't need to look at full ones.
* This uses its own lock instead of swap_lock because when a
* swap_info_struct changes between not-full/full, it needs to
@@ -1444,22 +1439,27 @@ static bool swap_sync_discard(void)
{
bool ret = false;
struct swap_info_struct *si, *next;
+ struct swap_tier *tier;
+ short prio;
spin_lock(&swap_lock);
start_over:
- plist_for_each_entry_safe(si, next, &swap_active_head, list) {
- spin_unlock(&swap_lock);
- if (get_swap_device_info(si)) {
- if (si->flags & SWP_PAGE_DISCARD)
- ret = swap_do_scheduled_discard(si);
- put_swap_device(si);
- }
- if (ret)
- return true;
+ for_each_active_tier(tier) {
+ prio = tier->prio;
+ plist_for_each_entry_safe(si, next, &tier->active_head, list) {
+ spin_unlock(&swap_lock);
+ if (get_swap_device_info(si)) {
+ if (si->flags & SWP_PAGE_DISCARD)
+ ret = swap_do_scheduled_discard(si);
+ put_swap_device(si);
+ }
+ if (ret)
+ return true;
- spin_lock(&swap_lock);
- if (plist_node_empty(&next->list))
- goto start_over;
+ spin_lock(&swap_lock);
+ if (plist_node_empty(&next->list) || tier->prio != prio)
+ goto start_over;
+ }
}
spin_unlock(&swap_lock);
@@ -3088,7 +3088,7 @@ static void _enable_swap_info(struct swap_info_struct *si)
assert_spin_locked(&swap_lock);
- plist_add(&si->list, &swap_active_head);
+ swap_tiers_assign_dev(si);
/* Add back to available list */
add_to_avail_list(si, true);
@@ -3182,6 +3182,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
{
struct swap_info_struct *p = NULL;
struct swap_cluster_info *cluster_info;
+ struct swap_tier *tier;
struct file *swap_file, *victim;
struct address_space *mapping;
struct inode *inode;
@@ -3200,13 +3201,17 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
mapping = victim->f_mapping;
spin_lock(&swap_lock);
- plist_for_each_entry(p, &swap_active_head, list) {
- if (p->flags & SWP_WRITEOK) {
- if (p->swap_file->f_mapping == mapping) {
- found = 1;
- break;
+ for_each_active_tier(tier) {
+ plist_for_each_entry(p, &tier->active_head, list) {
+ if (p->flags & SWP_WRITEOK) {
+ if (p->swap_file->f_mapping == mapping) {
+ found = 1;
+ break;
+ }
}
}
+ if (found)
+ break;
}
if (!found) {
err = -EINVAL;
@@ -3230,7 +3235,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
}
spin_lock(&p->lock);
del_from_avail_list(p, true);
- plist_del(&p->list, &swap_active_head);
+ swap_tiers_remove_dev(p);
atomic_long_sub(p->pages, &nr_swap_pages);
total_swap_pages -= p->pages;
spin_unlock(&p->lock);
@@ -4002,7 +4007,7 @@ int swap_dup_entry_direct(swp_entry_t entry)
#if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
static bool __has_usable_swap(void)
{
- return !plist_head_empty(&swap_active_head);
+ return !list_empty(&swap_tier_active_list);
}
void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
@@ -4055,6 +4060,7 @@ static int __init swapfile_init(void)
swap_migration_ad_supported = true;
#endif /* CONFIG_MIGRATION */
+ swap_tiers_init();
return 0;
}
subsys_initcall(swapfile_init);
--
2.48.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC PATCH v11 2/4] mm: swap: allocate swap slots from swap tiers
2026-09-16 18:34 [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 1/4] mm: swap: introduce swap tier infrastructure Youngjun Park
@ 2026-09-16 18:34 ` Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 3/4] mm: swap: add a debugfs interface for memcg tier selection Youngjun Park
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Youngjun Park @ 2026-09-16 18:34 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, baoquan.he, baohua, yosry, joshua.hahnjy,
taejoon.song, her0gyugyu, lianux.mm
The slow allocation path walks swap_avail_head, one plist of every
available swap device, ordered by priority.
Give each tier its own list of available devices instead. The slow path
now walks the tiers in order, and walks each tier's plist as before.
__folio_throttle_swaprate() walks them the same way.
swap_avail_lock still protects these lists. The tier list changes under
it too, so the walk can go on after it drops the lock, as long as its
tier is still there.
This is the groundwork for the per-priority allocation queue series from
Kairui and Lian [1]. That series can move onto the per-tier device lists
one tier at a time.
[1] https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-0-68d3d925578c@gmail.com/
Assisted-by: Claude:claude-opus-5
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
include/linux/swap.h | 2 +-
mm/swap.h | 1 +
mm/swap_tier.c | 17 +++++++++-
mm/swap_tier.h | 3 ++
mm/swapfile.c | 78 ++++++++++++++++++++++++--------------------
5 files changed, 64 insertions(+), 37 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 22ccb4b5801e..df69c2dd434a 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -268,7 +268,7 @@ struct swap_info_struct {
struct work_struct discard_work; /* discard worker */
struct work_struct reclaim_work; /* reclaim worker */
struct list_head discard_clusters; /* discard clusters list */
- struct plist_node avail_list; /* entry in swap_avail_head */
+ struct plist_node avail_list; /* entry in its tier's avail list */
const struct swap_ops *ops;
};
diff --git a/mm/swap.h b/mm/swap.h
index 4de6b9b0f261..916c0ad128f7 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -37,6 +37,7 @@ struct swap_io_ctx;
#endif
extern spinlock_t swap_lock;
+extern spinlock_t swap_avail_lock;
extern struct swap_info_struct *swap_info[];
/*
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
index 8ed1427cee09..286f319fb125 100644
--- a/mm/swap_tier.c
+++ b/mm/swap_tier.c
@@ -65,6 +65,7 @@ void swap_tiers_init(void)
for_each_tier(tier, idx) {
plist_head_init(&tier->active_head);
+ plist_head_init(&tier->avail_head);
INIT_LIST_HEAD(&tier->list);
swap_tier_inactivate(tier);
}
@@ -92,11 +93,14 @@ void swap_tiers_assign_dev(struct swap_info_struct *swp)
lockdep_assert_held(&swap_lock);
+ /* The allocator walks the tiers under swap_avail_lock. */
+ spin_lock(&swap_avail_lock);
tier = swap_tier_lookup(swp->prio);
if (!tier) {
tier = swap_tier_prepare(swp->prio);
swap_tier_activate(tier);
}
+ spin_unlock(&swap_avail_lock);
plist_add(&swp->list, &tier->active_head);
}
@@ -109,6 +113,17 @@ void swap_tiers_remove_dev(struct swap_info_struct *swp)
tier = swap_tier_lookup(swp->prio);
plist_del(&swp->list, &tier->active_head);
- if (plist_head_empty(&tier->active_head))
+ if (plist_head_empty(&tier->active_head)) {
+ spin_lock(&swap_avail_lock);
swap_tier_inactivate(tier);
+ spin_unlock(&swap_avail_lock);
+ }
+}
+
+/* The avail list of the tier @swp belongs to. */
+struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp)
+{
+ lockdep_assert_held(&swap_avail_lock);
+
+ return &swap_tier_lookup(swp->prio)->avail_head;
}
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
index 3dce716d23f6..c4347c28d8f1 100644
--- a/mm/swap_tier.h
+++ b/mm/swap_tier.h
@@ -14,11 +14,13 @@ struct swap_info_struct;
*
* @prio: priority of the swap devices in the tier.
* @active_head: swap devices in the tier.
+ * @avail_head: available swap devices in the tier.
* @list: linkage into swap_tier_active_list or swap_tier_inactive_list.
*/
struct swap_tier {
short prio;
struct plist_head active_head;
+ struct plist_head avail_head;
struct list_head list;
};
@@ -33,5 +35,6 @@ void swap_tiers_init(void);
/* Tier assignment */
void swap_tiers_assign_dev(struct swap_info_struct *swp);
void swap_tiers_remove_dev(struct swap_info_struct *swp);
+struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp);
#endif /* _SWAP_TIER_H */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 8201ae779833..e9d142c0655b 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -85,8 +85,8 @@ static const char Bad_file[] = "Bad swap file entry ";
static const char Bad_offset[] = "Bad swap offset entry ";
/*
- * all available (active, not full) swap_info_structs
- * protected with swap_avail_lock, ordered by priority.
+ * all available (active, not full) swap_info_structs are on the
+ * avail list of their swap tier, protected with swap_avail_lock.
* This is used by folio_alloc_swap() instead of the active lists of
* the swap tiers because those include all swap_info_structs,
* but folio_alloc_swap() doesn't need to look at full ones.
@@ -96,8 +96,7 @@ static const char Bad_offset[] = "Bad swap offset entry ";
* is held and the locking order requires swap_lock to be taken
* before any swap_info_struct->lock.
*/
-static PLIST_HEAD(swap_avail_head);
-static DEFINE_SPINLOCK(swap_avail_lock);
+DEFINE_SPINLOCK(swap_avail_lock);
struct swap_info_struct *swap_info[MAX_SWAPFILES];
@@ -1226,7 +1225,7 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
goto skip;
}
- plist_del(&si->avail_list, &swap_avail_head);
+ plist_del(&si->avail_list, swap_tiers_avail_head(si));
skip:
spin_unlock(&swap_avail_lock);
@@ -1267,7 +1266,7 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
goto skip;
}
- plist_add(&si->avail_list, &swap_avail_head);
+ plist_add(&si->avail_list, swap_tiers_avail_head(si));
skip:
spin_unlock(&swap_avail_lock);
@@ -1398,35 +1397,40 @@ static bool swap_alloc_fast(struct folio *folio)
static void swap_alloc_slow(struct folio *folio)
{
struct swap_info_struct *si, *next;
+ struct swap_tier *tier;
+ short prio;
spin_lock(&swap_avail_lock);
start_over:
- plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
- /* Rotate the device and switch to a new cluster */
- plist_requeue(&si->avail_list, &swap_avail_head);
- spin_unlock(&swap_avail_lock);
- if (get_swap_device_info(si)) {
- cluster_alloc_swap_entry(si, folio);
- put_swap_device(si);
- if (folio_test_swapcache(folio))
- return;
- if (folio_test_large(folio))
- return;
- }
+ for_each_active_tier(tier) {
+ prio = tier->prio;
+ plist_for_each_entry_safe(si, next, &tier->avail_head, avail_list) {
+ /* Rotate the device and switch to a new cluster */
+ plist_requeue(&si->avail_list, &tier->avail_head);
+ spin_unlock(&swap_avail_lock);
+ if (get_swap_device_info(si)) {
+ cluster_alloc_swap_entry(si, folio);
+ put_swap_device(si);
+ if (folio_test_swapcache(folio))
+ return;
+ if (folio_test_large(folio))
+ return;
+ }
- spin_lock(&swap_avail_lock);
- /*
- * if we got here, it's likely that si was almost full before,
- * multiple callers probably all tried to get a page from the
- * same si and it filled up before we could get one; or, the si
- * filled up between us dropping swap_avail_lock.
- * Since we dropped the swap_avail_lock, the swap_avail_list
- * may have been modified; so if next is still in the
- * swap_avail_head list then try it, otherwise start over if we
- * have not gotten any slots.
- */
- if (plist_node_empty(&next->avail_list))
- goto start_over;
+ spin_lock(&swap_avail_lock);
+ /*
+ * if we got here, it's likely that si was almost full before,
+ * multiple callers probably all tried to get a page from the
+ * same si and it filled up before we could get one; or, the si
+ * filled up between us dropping swap_avail_lock.
+ * Since we dropped the swap_avail_lock, the swap_avail_list
+ * may have been modified; so if next is still in the
+ * tier's avail list and the tier is still there then try it,
+ * otherwise start over if we have not gotten any slots.
+ */
+ if (plist_node_empty(&next->avail_list) || tier->prio != prio)
+ goto start_over;
+ }
}
spin_unlock(&swap_avail_lock);
}
@@ -4013,6 +4017,7 @@ static bool __has_usable_swap(void)
void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
{
struct swap_info_struct *si;
+ struct swap_tier *tier;
if (!(gfp & __GFP_IO))
return;
@@ -4031,12 +4036,15 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
return;
spin_lock(&swap_avail_lock);
- plist_for_each_entry(si, &swap_avail_head, avail_list) {
- if (si->bdev) {
- blkcg_schedule_throttle(si->bdev->bd_disk, true);
- break;
+ for_each_active_tier(tier) {
+ plist_for_each_entry(si, &tier->avail_head, avail_list) {
+ if (si->bdev) {
+ blkcg_schedule_throttle(si->bdev->bd_disk, true);
+ goto out;
+ }
}
}
+out:
spin_unlock(&swap_avail_lock);
}
#endif
--
2.48.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC PATCH v11 3/4] mm: swap: add a debugfs interface for memcg tier selection
2026-09-16 18:34 [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 1/4] mm: swap: introduce swap tier infrastructure Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 2/4] mm: swap: allocate swap slots from swap tiers Youngjun Park
@ 2026-09-16 18:34 ` Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 4/4] mm: swap: filter swap allocation by memcg tier mask Youngjun Park
2026-09-16 20:04 ` [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Johannes Weiner
4 siblings, 0 replies; 6+ messages in thread
From: Youngjun Park @ 2026-09-16 18:34 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, baoquan.he, baohua, yosry, joshua.hahnjy,
taejoon.song, her0gyugyu, lianux.mm
Swap tiers do nothing until something chooses between them. Let a memory
cgroup do that.
Each cgroup carries a mask of the tiers it may swap to. Write a cgroup
path and a hex mask to /sys/kernel/debug/swap/memcg_tiers to set it. Bit
i is the tier at index i. /sys/kernel/debug/swap/tiers lists each tier's
index and priority.
# echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
This lives in debugfs on purpose. A cgroup file is a permanent ABI, and
what a swap tier should look like is not settled yet. It is built only
with CONFIG_MEMCG and CONFIG_DEBUG_FS.
The masks are kept in a list keyed by cgroup ID rather than in struct
mem_cgroup, so memcg itself is not changed. A mask applies to the memory
charged to its cgroup and is not inherited by child cgroups. The entry of
a removed cgroup is dropped on the next write.
Each swap device is stamped with its tier's bit at swapon. A device's
tier never changes, so the stamp is written once and read locklessly by
the allocator.
When the last device of a tier is swapped off, the tier frees its index
for reuse. It waits until swapoff can no longer fail, so a device whose
swapoff fails goes back to the same index. The freed bit is set back in
every cgroup mask, so a cgroup that had disabled that tier must disable
it again once a new tier reuses the index.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
Documentation/mm/index.rst | 1 +
Documentation/mm/swap-tier.rst | 55 ++++++
MAINTAINERS | 1 +
include/linux/swap.h | 1 +
mm/swap_tier.c | 294 ++++++++++++++++++++++++++++++++-
mm/swap_tier.h | 31 ++++
mm/swapfile.c | 5 +
7 files changed, 387 insertions(+), 1 deletion(-)
create mode 100644 Documentation/mm/swap-tier.rst
diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst
index 13a79f5d092c..6afc45cd4b3d 100644
--- a/Documentation/mm/index.rst
+++ b/Documentation/mm/index.rst
@@ -34,6 +34,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
page_reclaim
swap
swap-table
+ swap-tier
page_cache
shmfs
oom
diff --git a/Documentation/mm/swap-tier.rst b/Documentation/mm/swap-tier.rst
new file mode 100644
index 000000000000..4007c23f83a6
--- /dev/null
+++ b/Documentation/mm/swap-tier.rst
@@ -0,0 +1,55 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+:Author: Chris Li <chrisl@kernel.org>,
+ Youngjun Park <youngjun.park@lge.com>
+
+==========
+Swap Tier
+==========
+
+Swap tier is a group of swap devices that share a priority. It acts as a
+facilitation layer, allowing users to manage swap devices based on their
+speeds.
+
+Users are encouraged to assign swap device priorities according to device
+speed to fully utilize this feature.
+
+Tier Index
+----------
+
+A tier is created when the first swap device with its priority is swapped on,
+and removed when the last one is swapped off. Each tier is given an index when
+it is created and keeps it until it is removed, so a tier's index does not
+change when another priority is swapped on or off.
+
+Per-cgroup Tier Selection
+-------------------------
+
+A memory cgroup can be limited to some tiers through debugfs. This is for
+evaluation, not a stable ABI.
+
+``/sys/kernel/debug/swap/tiers`` lists the index and priority of each tier.
+``/sys/kernel/debug/swap/memcg_tiers`` takes a cgroup path and a mask in hex,
+where bit ``i`` allows the tier at index ``i``::
+
+ # cat /sys/kernel/debug/swap/tiers
+ Idx Prio
+ 0 100
+ 1 50
+ # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
+
+There is no separate delete operation. Writing a mask that allows every tier
+clears the restriction, so the cgroup drops out of the file::
+
+ # echo "/batch 0xffffffff" > /sys/kernel/debug/swap/memcg_tiers
+
+A cgroup's mask is also dropped when the cgroup is removed.
+
+A mask applies to the memory charged to its own cgroup and is not inherited by
+child cgroups. A tier keeps its index for its lifetime, so the same mask keeps
+selecting the same tier across a swapon or swapoff.
+
+When a tier's last device is swapped off, its index is freed and can be reused
+by a later tier. The freed index is re-allowed in every cgroup mask, so a
+cgroup that had disabled it must disable it again once a new tier reuses the
+index.
diff --git a/MAINTAINERS b/MAINTAINERS
index 37f353015cae..af17b804f31b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17402,6 +17402,7 @@ L: linux-mm@kvack.org
S: Maintained
F: Documentation/ABI/testing/sysfs-kernel-mm-swap
F: Documentation/mm/swap-table.rst
+F: Documentation/mm/swap-tier.rst
F: include/linux/swap.h
F: include/linux/swap_ops.h
F: include/linux/swapfile.h
diff --git a/include/linux/swap.h b/include/linux/swap.h
index df69c2dd434a..30d3c37ca530 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -237,6 +237,7 @@ struct swap_info_struct {
struct percpu_ref users; /* indicate and keep swap device valid. */
unsigned long flags; /* SWP_USED etc: see above */
signed short prio; /* swap priority of this type */
+ unsigned int tier_mask; /* swap tier mask */
struct plist_node list; /* entry in its swap tier */
signed char type; /* strange name for an index */
unsigned int max; /* size of this swap device */
diff --git a/mm/swap_tier.c b/mm/swap_tier.c
index 286f319fb125..de183c2678e0 100644
--- a/mm/swap_tier.c
+++ b/mm/swap_tier.c
@@ -1,5 +1,10 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/swap.h>
+#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS)
+#include <linux/debugfs.h>
+#include <linux/memcontrol.h>
+#include <linux/seq_file.h>
+#endif
#include "swap.h"
#include "swap_tier.h"
@@ -17,6 +22,10 @@ static LIST_HEAD(swap_tier_inactive_list);
for (idx = 0, tier = &swap_tiers[0]; idx < MAX_SWAPTIER; \
idx++, tier = &swap_tiers[idx])
+/* A tier's index is its slot in the array, stable for its lifetime. */
+#define TIER_IDX(tier) ((tier) - swap_tiers)
+#define TIER_MASK(tier) (1U << TIER_IDX(tier))
+
/*
* Naming Convention:
* swap_tiers_*() - Public/exported functions
@@ -56,6 +65,12 @@ static void swap_tier_inactivate(struct swap_tier *tier)
list_move_tail(&tier->list, &swap_tier_inactive_list);
}
+#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS)
+static void swap_tiers_debugfs_init(void);
+#else
+static inline void swap_tiers_debugfs_init(void) {}
+#endif
+
void swap_tiers_init(void)
{
struct swap_tier *tier;
@@ -69,6 +84,8 @@ void swap_tiers_init(void)
INIT_LIST_HEAD(&tier->list);
swap_tier_inactivate(tier);
}
+
+ swap_tiers_debugfs_init();
}
static struct swap_tier *swap_tier_prepare(short prio)
@@ -103,6 +120,17 @@ void swap_tiers_assign_dev(struct swap_info_struct *swp)
spin_unlock(&swap_avail_lock);
plist_add(&swp->list, &tier->active_head);
+
+ /* Put back by a failed swapoff, so already stamped and counted. */
+ if (swp->tier_mask)
+ return;
+
+ /*
+ * A device's tier never changes, so stamp it once here. Paired with
+ * the READ_ONCE() in the allocator, which reads this without swap_lock.
+ */
+ tier->nr_devs++;
+ WRITE_ONCE(swp->tier_mask, TIER_MASK(tier));
}
void swap_tiers_remove_dev(struct swap_info_struct *swp)
@@ -113,11 +141,31 @@ void swap_tiers_remove_dev(struct swap_info_struct *swp)
tier = swap_tier_lookup(swp->prio);
plist_del(&swp->list, &tier->active_head);
- if (plist_head_empty(&tier->active_head)) {
+}
+
+/*
+ * A failed swapoff puts the device back into its tier, so the tier is given
+ * up only here, once swapoff can no longer fail. Returns the tier's mask if
+ * @swp was its last device, 0 otherwise.
+ */
+unsigned int swap_tiers_release_dev(struct swap_info_struct *swp)
+{
+ struct swap_tier *tier;
+ unsigned int freed = 0;
+
+ lockdep_assert_held(&swap_lock);
+
+ tier = swap_tier_lookup(swp->prio);
+ if (!--tier->nr_devs) {
spin_lock(&swap_avail_lock);
swap_tier_inactivate(tier);
spin_unlock(&swap_avail_lock);
+ freed = TIER_MASK(tier);
}
+
+ WRITE_ONCE(swp->tier_mask, 0);
+
+ return freed;
}
/* The avail list of the tier @swp belongs to. */
@@ -127,3 +175,247 @@ struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp)
return &swap_tier_lookup(swp->prio)->avail_head;
}
+
+#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS)
+static DEFINE_MUTEX(swap_tier_lock);
+
+/*
+ * struct swap_tier_cgroup - tier mask of a cgroup.
+ *
+ * @id: cgroup ID of the cgroup.
+ * @mask: tiers the cgroup may swap to.
+ * @list: linkage into swap_tier_cgroup_list.
+ * @rcu: frees the entry after a grace period.
+ */
+struct swap_tier_cgroup {
+ u64 id;
+ unsigned int mask;
+ struct list_head list;
+ struct rcu_head rcu;
+};
+
+/*
+ * Cgroups written to memcg_tiers. Changed under swap_tier_lock, walked by
+ * the allocator under RCU. A cgroup not on the list may use every tier.
+ */
+static LIST_HEAD(swap_tier_cgroup_list);
+
+static struct swap_tier_cgroup *swap_tier_cgroup_lookup(u64 id)
+{
+ struct swap_tier_cgroup *stc;
+
+ list_for_each_entry_rcu(stc, &swap_tier_cgroup_list, list,
+ lockdep_is_held(&swap_tier_lock)) {
+ if (stc->id == id)
+ return stc;
+ }
+
+ return NULL;
+}
+
+/* Drop the entries that allow every tier or whose cgroup is removed. */
+static void swap_tier_cgroup_prune(void)
+{
+ struct swap_tier_cgroup *stc, *tmp;
+ struct cgroup *cgrp;
+
+ lockdep_assert_held(&swap_tier_lock);
+
+ list_for_each_entry_safe(stc, tmp, &swap_tier_cgroup_list, list) {
+ if (stc->mask != TIER_ALL_MASK) {
+ cgrp = __cgroup_get_from_id(stc->id);
+ if (!IS_ERR(cgrp)) {
+ cgroup_put(cgrp);
+ continue;
+ }
+ }
+
+ list_del_rcu(&stc->list);
+ kfree_rcu(stc, rcu);
+ }
+}
+
+/* One line per cgroup that dropped a tier, in the syntax a write takes. */
+static int swap_tiers_memcg_show(struct seq_file *m, void *v)
+{
+ struct swap_tier_cgroup *stc;
+ struct cgroup *cgrp;
+ char *path;
+
+ path = kmalloc(PATH_MAX, GFP_KERNEL);
+ if (!path)
+ return -ENOMEM;
+
+ mutex_lock(&swap_tier_lock);
+ list_for_each_entry(stc, &swap_tier_cgroup_list, list) {
+ if (stc->mask == TIER_ALL_MASK)
+ continue;
+
+ /* A removed cgroup's entry stays until the next write. */
+ cgrp = __cgroup_get_from_id(stc->id);
+ if (IS_ERR(cgrp))
+ continue;
+
+ cgroup_path(cgrp, path, PATH_MAX);
+ cgroup_put(cgrp);
+ seq_printf(m, "%s 0x%x\n", path, stc->mask);
+ }
+ mutex_unlock(&swap_tier_lock);
+
+ kfree(path);
+ return 0;
+}
+
+/*
+ * Keep only the tiers set in @mask. @cgpath must name a cgroup that has
+ * the memory controller enabled.
+ */
+static int swap_tiers_memcg_set(const char *cgpath, unsigned int mask)
+{
+ struct swap_tier_cgroup *stc;
+ struct cgroup *cgrp;
+ bool enabled;
+ int ret = 0;
+ u64 id;
+
+ cgrp = cgroup_get_from_path(cgpath);
+ if (IS_ERR(cgrp))
+ return PTR_ERR(cgrp);
+
+ /*
+ * Not cgroup_get_e_css(), which falls back to an ancestor when the
+ * memory controller is not enabled here.
+ */
+ rcu_read_lock();
+ enabled = cgroup_css(cgrp, &memory_cgrp_subsys);
+ rcu_read_unlock();
+ id = cgroup_id(cgrp);
+ cgroup_put(cgrp);
+
+ if (!enabled)
+ return -ENOENT;
+
+ mutex_lock(&swap_tier_lock);
+
+ stc = swap_tier_cgroup_lookup(id);
+ if (stc) {
+ WRITE_ONCE(stc->mask, mask);
+ } else if (mask != TIER_ALL_MASK) {
+ stc = kmalloc_obj(*stc, GFP_KERNEL);
+ if (stc) {
+ stc->id = id;
+ stc->mask = mask;
+ list_add_rcu(&stc->list, &swap_tier_cgroup_list);
+ } else {
+ ret = -ENOMEM;
+ }
+ }
+ swap_tier_cgroup_prune();
+
+ mutex_unlock(&swap_tier_lock);
+ return ret;
+}
+
+/* The tiers that the cgroup @folio is charged to may swap to. */
+unsigned int folio_tier_mask(struct folio *folio)
+{
+ struct swap_tier_cgroup *stc;
+ struct mem_cgroup *memcg;
+ unsigned int mask = TIER_ALL_MASK;
+
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ if (memcg) {
+ stc = swap_tier_cgroup_lookup(cgroup_id(memcg->css.cgroup));
+ if (stc)
+ mask = READ_ONCE(stc->mask);
+ }
+ rcu_read_unlock();
+
+ return mask;
+}
+
+/*
+ * When a tier is removed, its index (bit position in the mask) becomes
+ * free for reassignment to a future tier. If a cgroup had previously
+ * disabled this tier (cleared the bit in its memcg_tiers entry), its mask
+ * would keep that bit clear, meaning the new tier at the same index would
+ * be silently unavailable, an invisible cgroup constraint left behind by a
+ * tier that no longer exists.
+ *
+ * To prevent this, OR the removed tier's mask bit into every cgroup's
+ * mask. This resets the bit so the new tier is accessible by default.
+ * Users who want to restrict it must explicitly disable it after the tier
+ * is re-created.
+ */
+void swap_tiers_memcg_propagate(unsigned int mask)
+{
+ struct swap_tier_cgroup *stc;
+
+ mutex_lock(&swap_tier_lock);
+ list_for_each_entry(stc, &swap_tier_cgroup_list, list)
+ WRITE_ONCE(stc->mask, stc->mask | mask);
+ mutex_unlock(&swap_tier_lock);
+}
+
+static int swap_tiers_memcg_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, swap_tiers_memcg_show, NULL);
+}
+
+static ssize_t swap_tiers_memcg_write(struct file *file,
+ const char __user *ubuf,
+ size_t count, loff_t *ppos)
+{
+ char *pos, *tmp, *cgpath;
+ unsigned int mask;
+ int ret;
+
+ tmp = memdup_user_nul(ubuf, count);
+ if (IS_ERR(tmp))
+ return PTR_ERR(tmp);
+
+ pos = strstrip(tmp);
+ cgpath = strsep(&pos, " \t\n");
+ if (!cgpath || !*cgpath || !pos ||
+ kstrtouint(skip_spaces(pos), 16, &mask))
+ ret = -EINVAL;
+ else
+ ret = swap_tiers_memcg_set(cgpath, mask);
+
+ kfree(tmp);
+ return ret ? ret : count;
+}
+
+static const struct file_operations swap_tiers_memcg_fops = {
+ .open = swap_tiers_memcg_open,
+ .read = seq_read,
+ .write = swap_tiers_memcg_write,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static int swap_tiers_show(struct seq_file *m, void *v)
+{
+ struct swap_tier *tier;
+
+ seq_printf(m, "%-5s %s\n", "Idx", "Prio");
+
+ spin_lock(&swap_lock);
+ for_each_active_tier(tier)
+ seq_printf(m, "%-5td %d\n", TIER_IDX(tier), tier->prio);
+ spin_unlock(&swap_lock);
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(swap_tiers);
+
+static void swap_tiers_debugfs_init(void)
+{
+ struct dentry *dir = debugfs_create_dir("swap", NULL);
+
+ debugfs_create_file("tiers", 0400, dir, NULL, &swap_tiers_fops);
+ debugfs_create_file("memcg_tiers", 0600, dir, NULL,
+ &swap_tiers_memcg_fops);
+}
+#endif
diff --git a/mm/swap_tier.h b/mm/swap_tier.h
index c4347c28d8f1..9967cfbcc439 100644
--- a/mm/swap_tier.h
+++ b/mm/swap_tier.h
@@ -9,16 +9,20 @@
/* Forward declarations */
struct swap_info_struct;
+#define TIER_ALL_MASK (~0U)
+
/*
* struct swap_tier - structure representing a swap tier.
*
* @prio: priority of the swap devices in the tier.
+ * @nr_devs: swap devices in the tier, including ones being swapped off.
* @active_head: swap devices in the tier.
* @avail_head: available swap devices in the tier.
* @list: linkage into swap_tier_active_list or swap_tier_inactive_list.
*/
struct swap_tier {
short prio;
+ int nr_devs;
struct plist_head active_head;
struct plist_head avail_head;
struct list_head list;
@@ -35,6 +39,33 @@ void swap_tiers_init(void);
/* Tier assignment */
void swap_tiers_assign_dev(struct swap_info_struct *swp);
void swap_tiers_remove_dev(struct swap_info_struct *swp);
+unsigned int swap_tiers_release_dev(struct swap_info_struct *swp);
struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp);
+/**
+ * swap_tiers_mask_test - test whether two tier masks overlap
+ * @tier_mask: mask to test, e.g. a swap device's tier bit
+ * @mask: mask to test against, e.g. a cgroup's mask
+ *
+ * Return: true if @tier_mask and @mask share at least one tier bit.
+ */
+static inline bool swap_tiers_mask_test(unsigned int tier_mask,
+ unsigned int mask)
+{
+ return tier_mask & mask;
+}
+
+#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS)
+/* Memcg related functions */
+void swap_tiers_memcg_propagate(unsigned int mask);
+unsigned int folio_tier_mask(struct folio *folio);
+#else
+static inline void swap_tiers_memcg_propagate(unsigned int mask) {}
+
+static inline unsigned int folio_tier_mask(struct folio *folio)
+{
+ return TIER_ALL_MASK;
+}
+#endif
+
#endif /* _SWAP_TIER_H */
diff --git a/mm/swapfile.c b/mm/swapfile.c
index e9d142c0655b..bb953dd33ca0 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -3191,6 +3191,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
struct address_space *mapping;
struct inode *inode;
unsigned int maxpages;
+ unsigned int freed_tier;
int err, found = 0;
if (!capable(CAP_SYS_ADMIN))
@@ -3290,7 +3291,11 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
p->max = 0;
p->cluster_info = NULL;
spin_unlock(&p->lock);
+ freed_tier = swap_tiers_release_dev(p);
spin_unlock(&swap_lock);
+ /* Under swapon_mutex, so a swapon cannot reuse the index before this. */
+ if (freed_tier)
+ swap_tiers_memcg_propagate(freed_tier);
arch_swap_invalidate_area(p->type);
zswap_swapoff(p->type);
mutex_unlock(&swapon_mutex);
--
2.48.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC PATCH v11 4/4] mm: swap: filter swap allocation by memcg tier mask
2026-09-16 18:34 [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Youngjun Park
` (2 preceding siblings ...)
2026-09-16 18:34 ` [RFC PATCH v11 3/4] mm: swap: add a debugfs interface for memcg tier selection Youngjun Park
@ 2026-09-16 18:34 ` Youngjun Park
2026-09-16 20:04 ` [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Johannes Weiner
4 siblings, 0 replies; 6+ messages in thread
From: Youngjun Park @ 2026-09-16 18:34 UTC (permalink / raw)
To: akpm
Cc: chrisl, youngjun.park, linux-mm, cgroups, linux-kernel, kasong,
hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
shikemeng, baoquan.he, baohua, yosry, joshua.hahnjy,
taejoon.song, her0gyugyu, lianux.mm
Apply the cgroup tier mask during swap slot allocation to enforce
per-cgroup swap tier restrictions.
The folio's mask is looked up once and passed to the fast, slow and
discard paths as a parameter, so all of them act on the same mask even
if the cgroup's mask changes concurrently.
The device tier_mask is read with READ_ONCE() in the two paths that
do not hold swap_lock, matching how si->flags is read in the same
allocator.
In the fast path, check the percpu cached swap_info's tier_mask
against the folio's mask. If it does not match, fall through to the
slow path. In the slow path, skip swap devices whose tier_mask is not
covered by the folio's mask. The discard fallback honors the mask too.
Without it, a discard on a device outside the folio's tiers still
returns true and drives the retry, so the allocation spins through the
loop consuming another device's discard queue while it cannot succeed.
This works correctly when there is only one non-rotational
device in the system and no devices share the same priority.
However, there are known limitations.
- When non-rotational devices are distributed across multiple
tiers, and different memcgs are configured to use those
distinct tiers, they may constantly overwrite the shared
percpu swap cache. This cache thrashing leads to frequent
fast path misses.
- Combined with the above issue, if same-priority devices exist
among them, a percpu cache miss (overwritten by another memcg)
forces the allocator to round-robin to the next device
prematurely, even if the current cluster is not fully
exhausted.
These edge cases do not affect the primary use case of
directing swap traffic per cgroup. Further optimization is
planned for future work.
Signed-off-by: Youngjun Park <youngjun.park@lge.com>
---
mm/swapfile.c | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index bb953dd33ca0..b246eff25c96 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1364,7 +1364,7 @@ static bool get_swap_device_info(struct swap_info_struct *si)
* Fast path try to get swap entries with specified order from current
* CPU's swap entry pool (a cluster).
*/
-static bool swap_alloc_fast(struct folio *folio)
+static bool swap_alloc_fast(struct folio *folio, unsigned int mask)
{
unsigned int order = folio_order(folio);
struct swap_cluster_info *ci;
@@ -1376,8 +1376,11 @@ static bool swap_alloc_fast(struct folio *folio)
* so checking it's liveness by get_swap_device_info is enough.
*/
si = this_cpu_read(percpu_swap_cluster.si[order]);
+ if (!si || !swap_tiers_mask_test(READ_ONCE(si->tier_mask), mask))
+ return false;
+
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
- if (!si || !offset || !get_swap_device_info(si))
+ if (!offset || !get_swap_device_info(si))
return false;
ci = swap_cluster_lock(si, offset);
@@ -1394,7 +1397,7 @@ static bool swap_alloc_fast(struct folio *folio)
}
/* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static void swap_alloc_slow(struct folio *folio, unsigned int mask)
{
struct swap_info_struct *si, *next;
struct swap_tier *tier;
@@ -1405,6 +1408,9 @@ static void swap_alloc_slow(struct folio *folio)
for_each_active_tier(tier) {
prio = tier->prio;
plist_for_each_entry_safe(si, next, &tier->avail_head, avail_list) {
+ if (!swap_tiers_mask_test(READ_ONCE(si->tier_mask), mask))
+ continue;
+
/* Rotate the device and switch to a new cluster */
plist_requeue(&si->avail_list, &tier->avail_head);
spin_unlock(&swap_avail_lock);
@@ -1439,7 +1445,7 @@ static void swap_alloc_slow(struct folio *folio)
* Discard pending clusters in a synchronized way when under high pressure.
* Return: true if any cluster is discarded.
*/
-static bool swap_sync_discard(void)
+static bool swap_sync_discard(unsigned int mask)
{
bool ret = false;
struct swap_info_struct *si, *next;
@@ -1451,6 +1457,8 @@ static bool swap_sync_discard(void)
for_each_active_tier(tier) {
prio = tier->prio;
plist_for_each_entry_safe(si, next, &tier->active_head, list) {
+ if (!swap_tiers_mask_test(si->tier_mask, mask))
+ continue;
spin_unlock(&swap_lock);
if (get_swap_device_info(si)) {
if (si->flags & SWP_PAGE_DISCARD)
@@ -1749,6 +1757,7 @@ int folio_alloc_swap(struct folio *folio)
{
unsigned int order = folio_order(folio);
unsigned int size = 1 << order;
+ unsigned int mask;
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1772,13 +1781,14 @@ int folio_alloc_swap(struct folio *folio)
}
again:
+ mask = folio_tier_mask(folio);
local_lock(&percpu_swap_cluster.lock);
- if (!swap_alloc_fast(folio))
- swap_alloc_slow(folio);
+ if (!swap_alloc_fast(folio, mask))
+ swap_alloc_slow(folio, mask);
local_unlock(&percpu_swap_cluster.lock);
if (!order && unlikely(!folio_test_swapcache(folio))) {
- if (swap_sync_discard())
+ if (swap_sync_discard(mask))
goto again;
}
--
2.48.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection
2026-09-16 18:34 [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Youngjun Park
` (3 preceding siblings ...)
2026-09-16 18:34 ` [RFC PATCH v11 4/4] mm: swap: filter swap allocation by memcg tier mask Youngjun Park
@ 2026-09-16 20:04 ` Johannes Weiner
4 siblings, 0 replies; 6+ messages in thread
From: Johannes Weiner @ 2026-09-16 20:04 UTC (permalink / raw)
To: Youngjun Park
Cc: akpm, chrisl, linux-mm, cgroups, linux-kernel, kasong, mhocko,
roman.gushchin, shakeel.butt, muchun.song, shikemeng, baoquan.he,
baohua, yosry, joshua.hahnjy, taejoon.song, her0gyugyu,
lianux.mm
On Thu, Sep 17, 2026 at 03:34:33AM +0900, Youngjun Park wrote:
> Per-cgroup swap in debugfs
> ==========================
>
> Patches 3 and 4 let a memory cgroup choose its tiers through debugfs.
>
> # swapon -p 100 /dev/nvme0n1p2
> # swapon -p 50 /dev/sdb2
> # cat /sys/kernel/debug/swap/tiers
> Idx Prio
> 0 100
> 1 50
> # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
>
> Bit i of the mask is tier i, so /batch swaps only to sdb2. A tier keeps
> its index for its lifetime, so the mask keeps selecting the same tier
> across swapon and swapoff.
Can the cgroup be given a priority limit? That would have pretty
obvious inheritance semantics:
root
`- batch (memory.swap.prio.max = 20)
`- task (memory.swap.prio.max = max)
`- logs (memory.swap.prio.max = 10)
`- interactive (memory.swap.prio.max = max)
`- task (memory.swap.prio.max)
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-09-16 20:04 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-16 18:34 [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 1/4] mm: swap: introduce swap tier infrastructure Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 2/4] mm: swap: allocate swap slots from swap tiers Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 3/4] mm: swap: add a debugfs interface for memcg tier selection Youngjun Park
2026-09-16 18:34 ` [RFC PATCH v11 4/4] mm: swap: filter swap allocation by memcg tier mask Youngjun Park
2026-09-16 20:04 ` [RFC PATCH v11 0/4] mm/swap: priority-based swap tiers with per-cgroup selection Johannes Weiner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®