[PATCH 0/6] mm: memcontrol: implement the memsw limit on cgroup v2

[PATCH 0/6] mm: memcontrol: implement the memsw limit on cgroup v2

[Jingxiang Zeng via B4 Relay]

cgroup v1 caps the sum of memory and swap through memsw.limit_in_bytes.
v2 only offers separate memory.max and memory.swap.max, so a workload
that has to be capped on the total of the two has no equivalent knob:
memory.max alone can be met by swapping, and capping both separately
reserves swap the workload may never use.

A container capped at N pages should stay capped at N once swapping is
possible, but with memory.max alone the pages move to swap and it
allocates N more; reserving swap per cgroup instead means guessing how
much of the footprint will be cold at any moment.  The combined limit
states the intent directly - N pages however they are divided between RAM
and swap - and leaves the host free to offload cold pages.

This has been discussed before [1], and the direction agreed on was to
track and control the combined counter as a proper v2 feature rather than
fold it into the existing swap knobs:

  My suggestion is to factor out from struct page_counter all the stuff
  that is not necessary for all users, and then have separate counters
  for swap and memsw.

  The protection stuff is long overdue for this.  It makes up nearly half
  of the struct's members, but is only used by the memory counter.  Even
  before your patches this is unnecessary bloat in the swap/memsw, kmem
  and tcpmem counters.

  Fix that and having separate counters is a non-issue.

and, on the interface:

  This should be new knobs, e.g. memory.memsw.current, memory.memsw.max.

Patches 1-3 are that refactoring.  Hierarchical protection is only used
by the memory counter and by dmem pools, yet every page_counter carried
it; moving it into a separate struct page_counter_protection, embedded
only where it is used, takes struct page_counter from 192 to 128 bytes on
x86_64.  Both embedders put the context on a cache line boundary (384 in
struct mem_cgroup, 192 in the dmem pool state), so the fields the charge
path touches - min, low and the four usage counters - share a cache line
with emin, which only reclaim recomputes; only elow crosses into the
next one.

Patch 4 splits the counters.  swap and memsw share a union today because
v1 only ever charges memsw and v2 only ever charges swap, so charging
both on both hierarchies is what the combined limit needs.  v2 hands the
combined charge over to the swap slot in __mem_cgroup_try_charge_swap()
and drops it again in __mem_cgroup_uncharge_swap().  memsw.max still
defaults to "max" and v2 cannot set it, so behaviour does not change.

Patch 5 exposes memory.memsw.current and memory.memsw.max on non-root
cgroups.  A charge counted there is held for as long as the memory
occupies either RAM or a swap slot, so reclaim cannot get back under the
limit by swapping; try_to_free_mem_cgroup_pages() is called without
MEMCG_RECLAIM_MAY_SWAP when the limit is written, and the cgroup OOM
killer is the last resort, mirroring memory.max.  Both writers keep
memory.max <= memory.memsw.max under a mutex, so the two limits cannot be
configured into a state reclaim could never satisfy - the invariant v1
keeps in mem_cgroup_resize_max(), including its serialization.

Patch 6 clamps mem_cgroup_get_max() to the new limit.  That value becomes
oc->totalpages for memcg OOM and oom_badness() scales oom_score_adj by
it, so while the v2 branch derived the ceiling from memory.max plus
memory.swap.max it overstated the reachable total once a combined limit
was set, and oom_score_adj weighed correspondingly more than intended.

Size effect, measured with pahole (x86_64, 64-byte cache lines).  The
split costs a cgroup one more page counter, and without patches 1-3 that
counter would be 192 bytes:

  struct mem_cgroup (CONFIG_MEMCG_V1=y)  2240 -> 2432 -> 2240
  struct mem_cgroup (CONFIG_MEMCG_V1=n)  1664 -> 1856 -> 1792

The middle figure is the same tree with the union split but the counters
left at 192 bytes, so it is what the series would have cost without the
refactoring.  On a v1 kernel that is the whole 192 bytes paid back; on a
v2-only one the three counters involved shed 192 bytes between them, but
the protection context takes 72 back and alignment a further 56, so 64
are paid back.  Counters that gain nothing keep their share:

  struct page_counter             192 -> 128 bytes
  struct hugetlb_cgroup          1344 -> 1088 bytes
  dmem pool state                 320 ->  320 bytes

Tested on x86_64 with CONFIG_MEMCG_V1=y and =n, each patch building on
its own:

- memory.memsw.current equals memory.current plus memory.swap.current
  across swapout, swapin and swapoff.
- Squeezing memory.max drops memory.current while memory.memsw.current
  holds, and a combined limit stops a 128M working set that otherwise
  escapes a 32M memory.max by swapping.
- Writing memory.memsw.max reclaims without MEMCG_RECLAIM_MAY_SWAP: swap
  usage stays flat and the cgroup goes to OOM, where writing memory.max
  swaps instead.
- Concurrent writers to the two limits could not leave the invariant
  violated in 600 rounds.
- A kretprobe on mem_cgroup_get_max() returns 48M for memory.max=32M and
  memory.memsw.max=48M with 2G of swap, and is unchanged with
  memory.memsw.max left at "max".
- v1 memsw.limit_in_bytes, and the v1 paths reaching the new NULL ->prot
  guards (css_offline, css_reset, lru_gen_age_node), behave as before.

[1] https://lore.kernel.org/all/20250320144722.GH1876369@cmpxchg.org/

Signed-off-by: Jingxiang Zeng <linuszeng@tencent.com>
---
Jingxiang Zeng (6):
      mm: page_counter: add page_counter_protection struct and init API
      mm: page_counter: track protection state in page_counter_protection
      mm: page_counter: drop protection fields from struct page_counter
      mm: memcontrol: give swap and memsw their own page counters
      mm: memcontrol: add memory.memsw.max to the default hierarchy
      mm: memcontrol: clamp mem_cgroup_get_max() to the combined limit

 Documentation/admin-guide/cgroup-v2.rst |  32 +++++
 include/linux/memcontrol.h              |  27 ++--
 include/linux/page_counter.h            |  88 +++++++++---
 kernel/cgroup/dmem.c                    |  21 +--
 mm/hugetlb_cgroup.c                     |   4 +-
 mm/memcontrol.c                         | 241 ++++++++++++++++++++++++++------
 mm/page_counter.c                       |  61 +++++---
 7 files changed, 371 insertions(+), 103 deletions(-)
---
base-commit: 1ed9cdd724d46119dd9adf0ffba2f2daaa3335ef
change-id: 20260916-descriptive-name-0cf0d5346bcb

Best regards,
-- 
Jingxiang Zeng <linuszeng@tencent.com>

[PATCH 1/6] mm: page_counter: add page_counter_protection struct and init API

[Jingxiang Zeng via B4 Relay]

From: Jingxiang Zeng <linuszeng@tencent.com>

Hierarchical protection (memory.min/memory.low) is only used by the
memory page counter and by dmem pools; swap/memsw, kmem, tcpmem and
hugetlb counters never participate in it, yet every struct page_counter
carries the full protection state.

Add struct page_counter_protection to hold that state and link it to
struct page_counter through a ->prot pointer, NULL when protection is
not supported.  page_counter_init() loses its protection_support
argument and the new page_counter_init_protection() attaches the
context; track_protection() tests ->prot instead of the flag.
Protection stays enabled only on the cgroup v2 hierarchy, matching the
previous page_counter_init(..., memcg_on_dfl) behaviour, and the root
memcg keeps it unconditionally.

Order the new structure by access frequency: parent, min, low and the
four usage counters are what propagate_protected_usage() touches on
every charge and uncharge, once per level, so they come first and share
a cache line - both embedders place the structure at a cache line
boundary, at offset 384 in struct mem_cgroup and 192 in the dmem pool
state.  emin and elow are only recomputed during reclaim and read by
the protection checks, so they go last.

struct page_counter still carries the old protection fields at this
point, so nothing shrinks yet: they are migrated onto the new structure
in the next commit and removed in the one after that.

The dmem pool allocator points its counter at the embedded protection
context, and the pool fix-up path in get_cg_pool_locked() links
prot->parent the same way it links cnt.parent, so pools created
bottom-up do not lose hierarchical protection.

No functional change.

Signed-off-by: Jingxiang Zeng <linuszeng@tencent.com>
---
 include/linux/memcontrol.h   |  7 +++++
 include/linux/page_counter.h | 72 ++++++++++++++++++++++++++++++++++++++++----
 kernel/cgroup/dmem.c         |  9 ++++--
 mm/hugetlb_cgroup.c          |  4 +--
 mm/memcontrol.c              | 21 ++++++++-----
 mm/page_counter.c            |  2 +-
 6 files changed, 95 insertions(+), 20 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 46bf724cae7a..5936f497aea6 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -199,6 +199,13 @@ struct mem_cgroup {
 	/* Accounted resources */
 	struct page_counter memory;		/* Both v1 & v2 */
 
+	/*
+	 * Hierarchical memory.min/memory.low protection tracking for the
+	 * memory page counter. swap/memsw, kmem and tcpmem counters do not
+	 * support protection and have no such context.
+	 */
+	struct page_counter_protection memory_prot;
+
 	union {
 		struct page_counter swap;	/* v2 only */
 		struct page_counter memsw;	/* v1 only */
diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index 07b7cb12249c..5020e62aacb4 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -7,6 +7,45 @@
 #include <linux/limits.h>
 #include <asm/page.h>
 
+/*
+ * Hierarchical protection (memory.min / memory.low) tracking.
+ *
+ * Only the memory page counter (and dmem pools) participate in protection.
+ * swap/memsw, kmem and tcpmem page counters never do, so the protection
+ * fields are kept out of struct page_counter in this separate structure to
+ * save space in the common case. struct page_counter links to it via ->prot,
+ * which is NULL for counters without protection support.
+ */
+struct page_counter_protection {
+	/*
+	 * Fields up to and including children_low_usage are read and
+	 * updated by propagate_protected_usage() on every charge and
+	 * uncharge, once per level of the hierarchy.  Keep them together
+	 * so that they share a cache line: both embedders (struct
+	 * mem_cgroup and the dmem pool state) place this structure at a
+	 * cache line boundary.
+	 */
+	struct page_counter_protection *parent;
+
+	unsigned long min;
+	unsigned long low;
+
+	/* memory.min and memory.low usage tracking */
+	atomic_long_t min_usage;
+	atomic_long_t low_usage;
+	atomic_long_t children_min_usage;
+	atomic_long_t children_low_usage;
+
+	/*
+	 * Effective values, recomputed by
+	 * page_counter_calculate_protection() during reclaim and read by
+	 * the mem_cgroup and dmem protection checks.  Not touched by the
+	 * charge path.
+	 */
+	unsigned long emin;
+	unsigned long elow;
+};
+
 struct page_counter {
 	/*
 	 * Make sure 'usage' does not share cacheline with any other field in
@@ -41,6 +80,12 @@ struct page_counter {
 	unsigned long high;
 	unsigned long max;
 	struct page_counter *parent;
+
+	/*
+	 * Hierarchical protection context, NULL for counters that do not
+	 * support memory.min/memory.low (swap, memsw, kmem, tcpmem, ...).
+	 */
+	struct page_counter_protection *prot;
 } ____cacheline_internodealigned_in_smp;
 
 #if BITS_PER_LONG == 32
@@ -49,18 +94,33 @@ struct page_counter {
 #define PAGE_COUNTER_MAX (LONG_MAX / PAGE_SIZE)
 #endif
 
-/*
- * Protection is supported only for the first counter (with id 0).
- */
 static inline void page_counter_init(struct page_counter *counter,
-				     struct page_counter *parent,
-				     bool protection_support)
+				     struct page_counter *parent)
 {
 	counter->usage = (atomic_long_t)ATOMIC_LONG_INIT(0);
 	counter->max = PAGE_COUNTER_MAX;
 	counter->parent = parent;
-	counter->protection_support = protection_support;
 	counter->track_failcnt = false;
+	counter->prot = NULL;
+}
+
+/*
+ * Enable hierarchical protection (memory.min/memory.low) on @counter.
+ * @prot and @parent are the protection contexts of @counter and its
+ * parent page counter respectively. Only the memory page counter (and
+ * dmem pools) call this.
+ *
+ * The remaining members of @prot (emin, elow and the usage counters) are
+ * expected to be zero already, so @prot must come from zeroed memory.
+ */
+static inline void page_counter_init_protection(struct page_counter *counter,
+						struct page_counter_protection *prot,
+						struct page_counter_protection *parent)
+{
+	counter->prot = prot;
+	prot->parent = parent;
+	prot->min = 0;
+	prot->low = 0;
 }
 
 static inline unsigned long page_counter_read(struct page_counter *counter)
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index 4683f3d68022..a4bac0d5ac3b 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -88,6 +88,7 @@ struct dmem_cgroup_pool_state {
 	struct rcu_head rcu;
 
 	struct page_counter cnt;
+	struct page_counter_protection prot;
 	struct dmem_cgroup_pool_state *parent;
 
 	refcount_t ref;
@@ -426,8 +427,9 @@ alloc_pool_single(struct dmemcg_state *dmemcs, struct dmem_cgroup_region *region
 	if (parent)
 		ppool = find_cg_pool_locked(parent, region);
 
-	page_counter_init(&pool->cnt,
-			  ppool ? &ppool->cnt : NULL, true);
+	page_counter_init(&pool->cnt, ppool ? &ppool->cnt : NULL);
+	page_counter_init_protection(&pool->cnt, &pool->prot,
+				     ppool ? &ppool->prot : NULL);
 	reset_all_resource_limits(pool);
 	refcount_set(&pool->ref, 1);
 	kref_get(&region->ref);
@@ -480,8 +482,9 @@ get_cg_pool_locked(struct dmemcg_state *dmemcs, struct dmem_cgroup_region *regio
 		/* ppool was created if it didn't exist by above loop. */
 		ppool = find_cg_pool_locked(pp, region);
 
-		/* Fix up parent links, mark as inited. */
+		/* Fix up parent links (counter and protection), mark as inited. */
 		pool->cnt.parent = &ppool->cnt;
+		pool->prot.parent = &ppool->prot;
 		if (ppool && !pool->parent) {
 			pool->parent = ppool;
 			dmemcg_pool_get(ppool);
diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c
index ecb6e0b7819a..7fdae504cfc6 100644
--- a/mm/hugetlb_cgroup.c
+++ b/mm/hugetlb_cgroup.c
@@ -108,8 +108,8 @@ static void hugetlb_cgroup_init(struct hugetlb_cgroup *h_cgroup,
 		fault = hugetlb_cgroup_counter_from_cgroup(h_cgroup, idx);
 		rsvd = hugetlb_cgroup_counter_from_cgroup_rsvd(h_cgroup, idx);
 
-		page_counter_init(fault, fault_parent, false);
-		page_counter_init(rsvd, rsvd_parent, false);
+		page_counter_init(fault, fault_parent);
+		page_counter_init(rsvd, rsvd_parent);
 
 		if (!cgroup_subsys_on_dfl(hugetlb_cgrp_subsys)) {
 			fault->track_failcnt = true;
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 791e536efaeb..94c538ea3cb5 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4296,25 +4296,30 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 #endif
 	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
 	if (parent) {
-		page_counter_init(&memcg->memory, &parent->memory, memcg_on_dfl);
-		page_counter_init(&memcg->swap, &parent->swap, false);
+		page_counter_init(&memcg->memory, &parent->memory);
+		if (memcg_on_dfl)
+			page_counter_init_protection(&memcg->memory, &memcg->memory_prot,
+						     &parent->memory_prot);
+		page_counter_init(&memcg->swap, &parent->swap);
 #ifdef CONFIG_MEMCG_V1
 		WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent));
 		memcg->memory.track_failcnt = !memcg_on_dfl;
 		memcg->memsw.track_failcnt = !memcg_on_dfl;
 		WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable));
-		page_counter_init(&memcg->kmem, &parent->kmem, false);
-		page_counter_init(&memcg->tcpmem, &parent->tcpmem, false);
+		page_counter_init(&memcg->kmem, &parent->kmem);
+		page_counter_init(&memcg->tcpmem, &parent->tcpmem);
 		memcg->tcpmem.track_failcnt = !memcg_on_dfl;
 #endif
 	} else {
 		init_memcg_stats();
 		init_memcg_events();
-		page_counter_init(&memcg->memory, NULL, true);
-		page_counter_init(&memcg->swap, NULL, false);
+		page_counter_init(&memcg->memory, NULL);
+		page_counter_init_protection(&memcg->memory, &memcg->memory_prot,
+					     NULL);
+		page_counter_init(&memcg->swap, NULL);
 #ifdef CONFIG_MEMCG_V1
-		page_counter_init(&memcg->kmem, NULL, false);
-		page_counter_init(&memcg->tcpmem, NULL, false);
+		page_counter_init(&memcg->kmem, NULL);
+		page_counter_init(&memcg->tcpmem, NULL);
 #endif
 		root_mem_cgroup = memcg;
 		return &memcg->css;
diff --git a/mm/page_counter.c b/mm/page_counter.c
index 98322803941a..aa1f9a9a314f 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -16,7 +16,7 @@
 
 static bool track_protection(struct page_counter *c)
 {
-	return c->protection_support;
+	return c->prot != NULL;
 }
 
 static void propagate_protected_usage(struct page_counter *c,

-- 
2.43.7

[PATCH 2/6] mm: page_counter: track protection state in page_counter_protection

[Jingxiang Zeng via B4 Relay]

From: Jingxiang Zeng <linuszeng@tencent.com>

Move the read/write side of hierarchical protection from struct
page_counter to struct page_counter_protection: propagate_protected_usage()
updates the protection context of the parent, page_counter_set_min()/low()
and page_counter_calculate_protection() operate on it, and memcg and dmem
accessors (including dmem_cgroup_below_min()/below_low()) read
min/low/emin/elow and children_*_usage from it.

struct page_counter keeps its now-unused protection fields for now; they
are removed in a follow-up commit.

No functional change.

Signed-off-by: Jingxiang Zeng <linuszeng@tencent.com>
---
 include/linux/memcontrol.h |  8 +++----
 kernel/cgroup/dmem.c       | 12 +++++-----
 mm/memcontrol.c            |  8 +++----
 mm/page_counter.c          | 59 +++++++++++++++++++++++++++++-----------------
 4 files changed, 52 insertions(+), 35 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 5936f497aea6..ff69a390a809 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -625,8 +625,8 @@ static inline void mem_cgroup_protection(struct mem_cgroup *root,
 	if (root == memcg)
 		return;
 
-	*min = READ_ONCE(memcg->memory.emin);
-	*low = READ_ONCE(memcg->memory.elow);
+	*min = READ_ONCE(memcg->memory_prot.emin);
+	*low = READ_ONCE(memcg->memory_prot.elow);
 }
 
 void mem_cgroup_calculate_protection(struct mem_cgroup *root,
@@ -650,7 +650,7 @@ static inline bool mem_cgroup_below_low(struct mem_cgroup *target,
 	if (mem_cgroup_unprotected(target, memcg))
 		return false;
 
-	return READ_ONCE(memcg->memory.elow) >=
+	return READ_ONCE(memcg->memory_prot.elow) >=
 		page_counter_read(&memcg->memory);
 }
 
@@ -660,7 +660,7 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
 	if (mem_cgroup_unprotected(target, memcg))
 		return false;
 
-	return READ_ONCE(memcg->memory.emin) >=
+	return READ_ONCE(memcg->memory_prot.emin) >=
 		page_counter_read(&memcg->memory);
 }
 
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index a4bac0d5ac3b..4027d3d309c8 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -212,12 +212,12 @@ set_resource_max(struct dmem_cgroup_pool_state *pool, u64 val, bool nonblock)
 
 static u64 get_resource_low(struct dmem_cgroup_pool_state *pool)
 {
-	return pool ? READ_ONCE(pool->cnt.low) : 0;
+	return pool ? READ_ONCE(pool->cnt.prot->low) : 0;
 }
 
 static u64 get_resource_min(struct dmem_cgroup_pool_state *pool)
 {
-	return pool ? READ_ONCE(pool->cnt.min) : 0;
+	return pool ? READ_ONCE(pool->cnt.prot->min) : 0;
 }
 
 static u64 get_resource_max(struct dmem_cgroup_pool_state *pool)
@@ -388,13 +388,13 @@ bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
 	dmem_cgroup_calculate_protection(limit_pool, test_pool);
 
 	used = page_counter_read(ctest);
-	min = READ_ONCE(ctest->emin);
+	min = READ_ONCE(ctest->prot->emin);
 
 	if (used <= min)
 		return false;
 
 	if (!ignore_low) {
-		low = READ_ONCE(ctest->elow);
+		low = READ_ONCE(ctest->prot->elow);
 		if (used > low)
 			return true;
 
@@ -787,7 +787,7 @@ bool dmem_cgroup_below_min(struct dmem_cgroup_pool_state *root,
 	 * here.
 	 */
 	dmem_cgroup_calculate_protection(root, test);
-	return page_counter_read(&test->cnt) <= READ_ONCE(test->cnt.emin);
+	return page_counter_read(&test->cnt) <= READ_ONCE(test->cnt.prot->emin);
 }
 EXPORT_SYMBOL_GPL(dmem_cgroup_below_min);
 
@@ -818,7 +818,7 @@ bool dmem_cgroup_below_low(struct dmem_cgroup_pool_state *root,
 	 * here.
 	 */
 	dmem_cgroup_calculate_protection(root, test);
-	return page_counter_read(&test->cnt) <= READ_ONCE(test->cnt.elow);
+	return page_counter_read(&test->cnt) <= READ_ONCE(test->cnt.prot->elow);
 }
 EXPORT_SYMBOL_GPL(dmem_cgroup_below_low);
 
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 94c538ea3cb5..aaf23e849408 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4860,7 +4860,7 @@ static ssize_t memory_peak_write(struct kernfs_open_file *of, char *buf,
 static int memory_min_show(struct seq_file *m, void *v)
 {
 	return seq_puts_memcg_tunable(m,
-		READ_ONCE(mem_cgroup_from_seq(m)->memory.min));
+		READ_ONCE(mem_cgroup_from_seq(m)->memory_prot.min));
 }
 
 static ssize_t memory_min_write(struct kernfs_open_file *of,
@@ -4883,7 +4883,7 @@ static ssize_t memory_min_write(struct kernfs_open_file *of,
 static int memory_low_show(struct seq_file *m, void *v)
 {
 	return seq_puts_memcg_tunable(m,
-		READ_ONCE(mem_cgroup_from_seq(m)->memory.low));
+		READ_ONCE(mem_cgroup_from_seq(m)->memory_prot.low));
 }
 
 static ssize_t memory_low_write(struct kernfs_open_file *of,
@@ -6457,6 +6457,6 @@ void mem_cgroup_show_protected_memory(struct mem_cgroup *memcg)
 		memcg = root_mem_cgroup;
 
 	pr_warn("Memory cgroup min protection %lukB -- low protection %lukB",
-		K(atomic_long_read(&memcg->memory.children_min_usage)),
-		K(atomic_long_read(&memcg->memory.children_low_usage)));
+		K(atomic_long_read(&memcg->memory_prot.children_min_usage)),
+		K(atomic_long_read(&memcg->memory_prot.children_low_usage)));
 }
diff --git a/mm/page_counter.c b/mm/page_counter.c
index aa1f9a9a314f..d33f40ca2c78 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -22,28 +22,29 @@ static bool track_protection(struct page_counter *c)
 static void propagate_protected_usage(struct page_counter *c,
 				      unsigned long usage)
 {
+	struct page_counter_protection *prot = c->prot;
 	unsigned long protected, old_protected;
 	long delta;
 
-	if (!c->parent)
+	if (!prot || !prot->parent)
 		return;
 
-	protected = min(usage, READ_ONCE(c->min));
-	old_protected = atomic_long_read(&c->min_usage);
+	protected = min(usage, READ_ONCE(prot->min));
+	old_protected = atomic_long_read(&prot->min_usage);
 	if (protected != old_protected) {
-		old_protected = atomic_long_xchg(&c->min_usage, protected);
+		old_protected = atomic_long_xchg(&prot->min_usage, protected);
 		delta = protected - old_protected;
 		if (delta)
-			atomic_long_add(delta, &c->parent->children_min_usage);
+			atomic_long_add(delta, &prot->parent->children_min_usage);
 	}
 
-	protected = min(usage, READ_ONCE(c->low));
-	old_protected = atomic_long_read(&c->low_usage);
+	protected = min(usage, READ_ONCE(prot->low));
+	old_protected = atomic_long_read(&prot->low_usage);
 	if (protected != old_protected) {
-		old_protected = atomic_long_xchg(&c->low_usage, protected);
+		old_protected = atomic_long_xchg(&prot->low_usage, protected);
 		delta = protected - old_protected;
 		if (delta)
-			atomic_long_add(delta, &c->parent->children_low_usage);
+			atomic_long_add(delta, &prot->parent->children_low_usage);
 	}
 }
 
@@ -258,7 +259,10 @@ void page_counter_set_min(struct page_counter *counter, unsigned long nr_pages)
 {
 	struct page_counter *c;
 
-	WRITE_ONCE(counter->min, nr_pages);
+	if (!counter->prot)
+		return;
+
+	WRITE_ONCE(counter->prot->min, nr_pages);
 
 	for (c = counter; c; c = c->parent)
 		propagate_protected_usage(c, atomic_long_read(&c->usage));
@@ -275,7 +279,10 @@ void page_counter_set_low(struct page_counter *counter, unsigned long nr_pages)
 {
 	struct page_counter *c;
 
-	WRITE_ONCE(counter->low, nr_pages);
+	if (!counter->prot)
+		return;
+
+	WRITE_ONCE(counter->prot->low, nr_pages);
 
 	for (c = counter; c; c = c->parent)
 		propagate_protected_usage(c, atomic_long_read(&c->usage));
@@ -454,9 +461,18 @@ void page_counter_calculate_protection(struct page_counter *root,
 				       struct page_counter *counter,
 				       bool recursive_protection)
 {
+	struct page_counter_protection *prot = counter->prot;
+	struct page_counter_protection *parent_prot;
 	unsigned long usage, parent_usage;
 	struct page_counter *parent = counter->parent;
 
+	/*
+	 * Only counters with protection support (memory, dmem pools) are
+	 * ever passed here, but guard anyway.
+	 */
+	if (!prot)
+		return;
+
 	/*
 	 * Effective values of the reclaim targets are ignored so they
 	 * can be stale. Have a look at mem_cgroup_protection for more
@@ -472,23 +488,24 @@ void page_counter_calculate_protection(struct page_counter *root,
 		return;
 
 	if (parent == root) {
-		counter->emin = READ_ONCE(counter->min);
-		counter->elow = READ_ONCE(counter->low);
+		prot->emin = READ_ONCE(prot->min);
+		prot->elow = READ_ONCE(prot->low);
 		return;
 	}
 
+	parent_prot = parent->prot;
 	parent_usage = page_counter_read(parent);
 
-	WRITE_ONCE(counter->emin, effective_protection(usage, parent_usage,
-			READ_ONCE(counter->min),
-			READ_ONCE(parent->emin),
-			atomic_long_read(&parent->children_min_usage),
+	WRITE_ONCE(prot->emin, effective_protection(usage, parent_usage,
+			READ_ONCE(prot->min),
+			READ_ONCE(parent_prot->emin),
+			atomic_long_read(&parent_prot->children_min_usage),
 			recursive_protection));
 
-	WRITE_ONCE(counter->elow, effective_protection(usage, parent_usage,
-			READ_ONCE(counter->low),
-			READ_ONCE(parent->elow),
-			atomic_long_read(&parent->children_low_usage),
+	WRITE_ONCE(prot->elow, effective_protection(usage, parent_usage,
+			READ_ONCE(prot->low),
+			READ_ONCE(parent_prot->elow),
+			atomic_long_read(&parent_prot->children_low_usage),
 			recursive_protection));
 }
 #endif /* CONFIG_MEMCG || CONFIG_CGROUP_DMEM */

-- 
2.43.7

[PATCH 3/6] mm: page_counter: drop protection fields from struct page_counter

[Jingxiang Zeng via B4 Relay]

From: Jingxiang Zeng <linuszeng@tencent.com>

The protection state now lives in struct page_counter_protection, so
remove the emin/min_usage/children_min_usage, elow/low_usage/
children_low_usage, min, low and protection_support fields from struct
page_counter.

Also drop the now-orphaned _pad2_ padding and its comment: with the
protection fields gone it no longer separates the read-mostly fields
from anything, and the structure's cacheline alignment already pads the
tail out.

swap/memsw, kmem, tcpmem and hugetlb counters no longer carry this
unused state: on 64-bit the structure shrinks from three cache lines to
two, saving one cache line.

Signed-off-by: Jingxiang Zeng <linuszeng@tencent.com>
---
 include/linux/page_counter.h | 16 ----------------
 1 file changed, 16 deletions(-)

diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index 5020e62aacb4..27467bf3b8e6 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -56,27 +56,11 @@ struct page_counter {
 
 	CACHELINE_PADDING(_pad1_);
 
-	/* effective memory.min and memory.min usage tracking */
-	unsigned long emin;
-	atomic_long_t min_usage;
-	atomic_long_t children_min_usage;
-
-	/* effective memory.low and memory.low usage tracking */
-	unsigned long elow;
-	atomic_long_t low_usage;
-	atomic_long_t children_low_usage;
-
 	unsigned long watermark;
 	/* Latest cg2 reset watermark */
 	unsigned long local_watermark;
 
-	/* Keep all the read most fields in a separete cacheline. */
-	CACHELINE_PADDING(_pad2_);
-
-	bool protection_support;
 	bool track_failcnt;
-	unsigned long min;
-	unsigned long low;
 	unsigned long high;
 	unsigned long max;
 	struct page_counter *parent;

-- 
2.43.7

[PATCH 4/6] mm: memcontrol: give swap and memsw their own page counters

[Jingxiang Zeng via B4 Relay]

From: Jingxiang Zeng <linuszeng@tencent.com>

struct mem_cgroup keeps the swap and memsw page counters in a union: v1
only ever charges memsw, the combined memory+swap counter, and v2 only
ever charges swap, the standalone swap limit, so the two could share
storage.

Split them into separate counters and maintain memsw on both
hierarchies.  do_memsw_account() no longer gates the combined charge in
try_charge_memcg(), memcg_uncharge(), the force-charge path,
mem_cgroup_replace_folio() and mem_cgroup_margin(); v2 also hands the
combined charge over to the swap slot in __mem_cgroup_try_charge_swap()
and drops it again in __mem_cgroup_uncharge_swap().  The counter is
initialised for both hierarchies in mem_cgroup_css_alloc() and reset in
mem_cgroup_css_reset().

memsw.max defaults to PAGE_COUNTER_MAX and v2 has no interface to
change it yet, so the combined charge cannot fail there and behaviour
on the default hierarchy is unchanged.  The default hierarchy does pay
for one extra page_counter operation per charge and uncharge, and
struct mem_cgroup grows by one page counter (128 bytes on x86_64).

Note that v2 briefly counts a page twice against the combined counter
while it is being swapped out: the folio keeps its memsw charge until
it leaves memory, and the swap slot takes one as soon as it is
allocated.  v1 avoids this by clearing folio->memcg_data under the swap
cluster lock, which the v2 swapout path does not do.  The skew only
ever makes the combined limit stricter, never looser.

No functional change on either hierarchy.

Signed-off-by: Jingxiang Zeng <linuszeng@tencent.com>
---
 include/linux/memcontrol.h | 12 ++++---
 mm/memcontrol.c            | 78 ++++++++++++++++++++++++++++++----------------
 2 files changed, 59 insertions(+), 31 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index ff69a390a809..bd69caee282c 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -206,10 +206,14 @@ struct mem_cgroup {
 	 */
 	struct page_counter_protection memory_prot;
 
-	union {
-		struct page_counter swap;	/* v2 only */
-		struct page_counter memsw;	/* v1 only */
-	};
+	/*
+	 * Swap accounting.  These used to share storage because v1 only ever
+	 * used memsw (the combined memory+swap limit) and v2 only ever used
+	 * swap (a standalone swap limit).  They are separate now so that the
+	 * combined counter can be maintained on both hierarchies.
+	 */
+	struct page_counter swap;	/* Standalone swap limit, v2 only */
+	struct page_counter memsw;	/* Combined memory+swap, v1 & v2 */
 
 	/* Written on the charge, reclaim and socket paths. */
 	__cacheline_group_begin_aligned(memcg_write_hot);
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index aaf23e849408..1daa55f2e99d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1591,14 +1591,17 @@ static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg)
 	if (count < limit)
 		margin = limit - count;
 
-	if (do_memsw_account()) {
-		count = page_counter_read(&memcg->memsw);
-		limit = READ_ONCE(memcg->memsw.max);
-		if (count < limit)
-			margin = min(margin, limit - count);
-		else
-			margin = 0;
-	}
+	/*
+	 * The combined memory+swap limit applies on both hierarchies and
+	 * caps what can still be charged.  It defaults to "max", so this
+	 * only narrows the margin once a combined limit is configured.
+	 */
+	count = page_counter_read(&memcg->memsw);
+	limit = READ_ONCE(memcg->memsw.max);
+	if (count < limit)
+		margin = min(margin, limit - count);
+	else
+		margin = 0;
 
 	return margin;
 }
@@ -2188,8 +2191,7 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
 static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
 {
 	page_counter_uncharge(&memcg->memory, nr_pages);
-	if (do_memsw_account())
-		page_counter_uncharge(&memcg->memsw, nr_pages);
+	page_counter_uncharge(&memcg->memsw, nr_pages);
 }
 
 /*
@@ -2721,12 +2723,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		batch = nr_pages;
 
 	reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
-	if (!do_memsw_account() ||
-	    page_counter_try_charge(&memcg->memsw, batch, &counter)) {
+	/*
+	 * The combined memory+swap counter is charged on both hierarchies.
+	 * Its limit is only configurable through v1's memsw.limit_in_bytes
+	 * for now and defaults to "max", so unless the user configures a
+	 * combined limit this never fails.
+	 *
+	 * Swapping does not reduce the combined charge, so when the combined
+	 * limit is what we hit, reclaim must not count on swap.
+	 */
+	if (page_counter_try_charge(&memcg->memsw, batch, &counter)) {
 		if (page_counter_try_charge(&memcg->memory, batch, &counter))
 			goto done_restock;
-		if (do_memsw_account())
-			page_counter_uncharge(&memcg->memsw, batch);
+		page_counter_uncharge(&memcg->memsw, batch);
 		mem_over_limit = mem_cgroup_from_counter(counter, memory);
 	} else {
 		mem_over_limit = mem_cgroup_from_counter(counter, memsw);
@@ -2836,8 +2845,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	 * temporarily by force charging it.
 	 */
 	page_counter_charge(&memcg->memory, nr_pages);
-	if (do_memsw_account())
-		page_counter_charge(&memcg->memsw, nr_pages);
+	page_counter_charge(&memcg->memsw, nr_pages);
 
 out:
 	/*
@@ -4295,12 +4303,14 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 	WRITE_ONCE(memcg->zswap_writeback, true);
 #endif
 	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
+	page_counter_set_high(&memcg->memsw, PAGE_COUNTER_MAX);
 	if (parent) {
 		page_counter_init(&memcg->memory, &parent->memory);
 		if (memcg_on_dfl)
 			page_counter_init_protection(&memcg->memory, &memcg->memory_prot,
 						     &parent->memory_prot);
 		page_counter_init(&memcg->swap, &parent->swap);
+		page_counter_init(&memcg->memsw, &parent->memsw);
 #ifdef CONFIG_MEMCG_V1
 		WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent));
 		memcg->memory.track_failcnt = !memcg_on_dfl;
@@ -4317,6 +4327,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
 		page_counter_init_protection(&memcg->memory, &memcg->memory_prot,
 					     NULL);
 		page_counter_init(&memcg->swap, NULL);
+		page_counter_init(&memcg->memsw, NULL);
 #ifdef CONFIG_MEMCG_V1
 		page_counter_init(&memcg->kmem, NULL);
 		page_counter_init(&memcg->tcpmem, NULL);
@@ -4490,6 +4501,7 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css)
 
 	page_counter_set_max(&memcg->memory, PAGE_COUNTER_MAX);
 	page_counter_set_max(&memcg->swap, PAGE_COUNTER_MAX);
+	page_counter_set_max(&memcg->memsw, PAGE_COUNTER_MAX);
 	WRITE_ONCE(memcg->oom_group, false);
 #ifdef CONFIG_ZSWAP
 	WRITE_ONCE(memcg->zswap_max, PAGE_COUNTER_MAX);
@@ -4503,6 +4515,7 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css)
 	page_counter_set_low(&memcg->memory, 0);
 	page_counter_set_high(&memcg->memory, PAGE_COUNTER_MAX);
 	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
+	page_counter_set_high(&memcg->memsw, PAGE_COUNTER_MAX);
 	memcg_wb_domain_size_changed(memcg);
 }
 
@@ -5608,8 +5621,7 @@ void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
 	 */
 	if (!obj_cgroup_is_root(objcg)) {
 		page_counter_charge(&memcg->memory, nr_pages);
-		if (do_memsw_account())
-			page_counter_charge(&memcg->memsw, nr_pages);
+		page_counter_charge(&memcg->memsw, nr_pages);
 	}
 
 	commit_charge(new, objcg);
@@ -5980,12 +5992,24 @@ int __mem_cgroup_try_charge_swap(struct folio *folio)
 	/* memcg is pined by memcg ID. */
 	rcu_read_unlock();
 
-	if (!mem_cgroup_is_root(memcg) &&
-	    !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
-		memcg_memory_event(memcg, MEMCG_SWAP_MAX);
-		memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
-		mem_cgroup_private_id_put(memcg, nr_pages);
-		return -ENOMEM;
+	if (!mem_cgroup_is_root(memcg)) {
+		if (!page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
+			memcg_memory_event(memcg, MEMCG_SWAP_MAX);
+			memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
+			mem_cgroup_private_id_put(memcg, nr_pages);
+			return -ENOMEM;
+		}
+		/*
+		 * Hand the combined memory+swap charge over to the swap slot.
+		 * The folio still holds a memsw charge and drops it when it
+		 * leaves memory, so the combined charge stays constant across
+		 * the swapout; it is briefly counted twice in between, which
+		 * only ever makes the combined limit stricter.
+		 *
+		 * Force-charge here: this is a transfer of an existing charge,
+		 * not a new allocation, so it must not fail.
+		 */
+		page_counter_charge(&memcg->memsw, nr_pages);
 	}
 	mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
 
@@ -6010,10 +6034,10 @@ void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
 	memcg = mem_cgroup_from_private_id(id);
 	if (memcg) {
 		if (!mem_cgroup_is_root(memcg)) {
-			if (do_memsw_account())
-				page_counter_uncharge(&memcg->memsw, nr_pages);
-			else
+			/* v1 tracks swap only through the combined counter */
+			if (!do_memsw_account())
 				page_counter_uncharge(&memcg->swap, nr_pages);
+			page_counter_uncharge(&memcg->memsw, nr_pages);
 		}
 		mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
 		mem_cgroup_private_id_put(memcg, nr_pages);

-- 
2.43.7

[PATCH 5/6] mm: memcontrol: add memory.memsw.max to the default hierarchy

[Jingxiang Zeng via B4 Relay]

From: Jingxiang Zeng <linuszeng@tencent.com>

cgroup v1 can cap the sum of memory and swap through
memsw.limit_in_bytes.  v2 only offers separate memory.max and
memory.swap.max, so a workload that must be capped on the total of the
two has no equivalent knob: memory.max alone can be met by swapping,
and capping both separately reserves swap that the workload may never
use.

Now that the combined counter is maintained on both hierarchies, expose
it on the default hierarchy:

  memory.memsw.current  combined memory+swap usage
  memory.memsw.max      combined memory+swap hard limit, default "max"

A charge counted here is held for as long as the memory occupies either
RAM or a swap slot, so reclaim cannot bring a cgroup back under the
limit by swapping; try_to_free_mem_cgroup_pages() is called without
MEMCG_RECLAIM_MAY_SWAP when the limit is written, and the cgroup OOM
killer is the last resort, mirroring memory.max.

Both writers enforce memory.max <= memory.memsw.max and return -EINVAL
otherwise, so the two limits cannot be configured into a state reclaim
could never satisfy.  This matches the invariant v1 keeps in
mem_cgroup_resize_max(), including the serialization: the test and the
store are done under a mutex, as two concurrent writers could otherwise
both pass the check and leave memory.max above the combined limit.  The
mutex is not held across reclaim.

Since memory.max defaults to "max", a combined limit can only be
installed once memory.max has been lowered, and updating both limits has
to start with memory.memsw.max when raising them and with memory.max
when lowering them.  Document that ordering next to the invariant.

Print the combined counter in the OOM dump on the default hierarchy too,
so a kill caused by this limit can be told apart from one caused by
memory.max.  No failcnt is printed there: it is only tracked on v1, and
a breach is already counted as MEMCG_MAX.

The files follow the default-hierarchy naming convention
(current/max) rather than v1's limit_in_bytes, and live in swap_files[]
since a combined limit is only meaningful with swap configured.

Signed-off-by: Jingxiang Zeng <linuszeng@tencent.com>
---
 Documentation/admin-guide/cgroup-v2.rst |  32 ++++++++
 mm/memcontrol.c                         | 136 ++++++++++++++++++++++++++++++--
 2 files changed, 163 insertions(+), 5 deletions(-)

diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 8d2603751c51..dd0cf28d39d9 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1881,6 +1881,38 @@ The following nested keys are defined.
 	Swap usage hard limit.  If a cgroup's swap usage reaches this
 	limit, anonymous memory of the cgroup will not be swapped out.
 
+  memory.memsw.current
+	A read-only single value file which exists on non-root cgroups.
+
+	The total amount of memory and swap space currently charged to
+	the cgroup and its descendants.
+
+  memory.memsw.max
+	A read-write single value file which exists on non-root
+	cgroups.  The default is "max".
+
+	Combined memory and swap usage hard limit.  Unlike memory.max,
+	which can be met by swapping anonymous memory out, a charge
+	counted here is kept for as long as the memory occupies either
+	RAM or a swap slot.  Reclaim therefore cannot bring a cgroup
+	back under this limit by swapping; only dropping pages, or
+	freeing swap slots, does.
+
+	This is the default-hierarchy counterpart of cgroup v1's
+	memory.memsw.limit_in_bytes and is useful for workloads that
+	must be capped on the sum of the two resources rather than on
+	each of them separately.
+
+	The limit must not be lower than memory.max: writes that would
+	violate memory.max <= memory.memsw.max are rejected with
+	EINVAL, in either file.  Since memory.max defaults to "max", a
+	combined limit can only be installed after memory.max has been
+	lowered.  To raise both limits, write memory.memsw.max first;
+	to lower both, write memory.max first.
+
+	If the limit is exceeded and reclaim cannot bring usage back
+	down, the cgroup OOM killer is invoked.
+
   memory.swap.events
 	A read-only flat-keyed file which exists on non-root cgroups.
 	The following entries are defined.  Unless specified
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1daa55f2e99d..9e8a176e7afb 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1873,11 +1873,15 @@ void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg)
 	pr_info("memory: usage %llukB, limit %llukB, failcnt %lu\n",
 		K((u64)page_counter_read(&memcg->memory)),
 		K((u64)READ_ONCE(memcg->memory.max)), memory_failcnt);
-	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
 		pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n",
 			K((u64)page_counter_read(&memcg->swap)),
 			K((u64)READ_ONCE(memcg->swap.max)),
 			atomic_long_read(&memcg->memory_events[MEMCG_SWAP_MAX]));
+		pr_info("memory+swap: usage %llukB, limit %llukB\n",
+			K((u64)page_counter_read(&memcg->memsw)),
+			K((u64)READ_ONCE(memcg->memsw.max)));
+	}
 #ifdef CONFIG_MEMCG_V1
 	else {
 		pr_info("memory+swap: usage %llukB, limit %llukB, failcnt %lu\n",
@@ -2724,10 +2728,10 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 
 	reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
 	/*
-	 * The combined memory+swap counter is charged on both hierarchies.
-	 * Its limit is only configurable through v1's memsw.limit_in_bytes
-	 * for now and defaults to "max", so unless the user configures a
-	 * combined limit this never fails.
+	 * The combined memory+swap counter is charged on both hierarchies:
+	 * v1 exposes it as memsw.limit_in_bytes, v2 as memory.memsw.max.
+	 * It defaults to "max", so unless the user configures a combined
+	 * limit this never fails.
 	 *
 	 * Swapping does not reduce the combined charge, so when the combined
 	 * limit is what we hit, reclaim must not count on swap.
@@ -4981,6 +4985,14 @@ static int memory_max_show(struct seq_file *m, void *v)
 		READ_ONCE(mem_cgroup_from_seq(m)->memory.max));
 }
 
+/*
+ * Serializes memory.max against memory.memsw.max so that the two cannot be
+ * tested and installed concurrently, which would let a pair of writers land
+ * in a state where memory.max exceeds the combined limit.  Only held across
+ * the check and the store, never across reclaim.
+ */
+static DEFINE_MUTEX(dfl_max_mutex);
+
 static ssize_t memory_max_write(struct kernfs_open_file *of,
 				char *buf, size_t nbytes, loff_t off)
 {
@@ -4995,7 +5007,20 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,
 	if (err)
 		return err;
 
+	/*
+	 * Keep the basic invariant memory.max <= memory.memsw.max, so a
+	 * combined memory+swap limit cannot be exceeded through the memory
+	 * limit.  memory.memsw.max defaults to "max", so this only rejects
+	 * writes once a combined limit has been configured.
+	 */
+	mutex_lock(&dfl_max_mutex);
+	if (max > READ_ONCE(memcg->memsw.max)) {
+		mutex_unlock(&dfl_max_mutex);
+		return -EINVAL;
+	}
+
 	xchg(&memcg->memory.max, max);
+	mutex_unlock(&dfl_max_mutex);
 
 	if (of->file->f_flags & O_NONBLOCK)
 		goto out;
@@ -6204,7 +6229,108 @@ static int swap_events_show(struct seq_file *m, void *v)
 	return 0;
 }
 
+static u64 memsw_current_read(struct cgroup_subsys_state *css,
+			      struct cftype *cft)
+{
+	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
+
+	return (u64)page_counter_read(&memcg->memsw) * PAGE_SIZE;
+}
+
+static int memsw_max_show(struct seq_file *m, void *v)
+{
+	return seq_puts_memcg_tunable(m,
+		READ_ONCE(mem_cgroup_from_seq(m)->memsw.max));
+}
+
+/*
+ * The combined memory+swap limit.  Swapping a page out does not release a
+ * combined charge, so reclaim cannot use swap to get back under this limit;
+ * only dropping pages, or freeing swap slots, helps.
+ */
+static ssize_t memsw_max_write(struct kernfs_open_file *of,
+			       char *buf, size_t nbytes, loff_t off)
+{
+	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
+	unsigned int nr_reclaims = MAX_RECLAIM_RETRIES;
+	bool drained = false;
+	unsigned long max;
+	int err;
+
+	buf = strstrip(buf);
+	err = page_counter_memparse(buf, "max", &max);
+	if (err)
+		return err;
+
+	/*
+	 * Keep the basic invariant memory.max <= memory.memsw.max: a combined
+	 * limit below the memory limit could never be met by reclaim.  Lower
+	 * memory.max first to install a combined limit on a fresh cgroup,
+	 * where memory.max still defaults to "max".
+	 */
+	mutex_lock(&dfl_max_mutex);
+	if (max < READ_ONCE(memcg->memory.max)) {
+		mutex_unlock(&dfl_max_mutex);
+		return -EINVAL;
+	}
+
+	xchg(&memcg->memsw.max, max);
+	mutex_unlock(&dfl_max_mutex);
+
+	if (of->file->f_flags & O_NONBLOCK)
+		goto out;
+
+	for (;;) {
+		unsigned long nr_pages = page_counter_read(&memcg->memsw);
+
+		if (max != READ_ONCE(memcg->memsw.max))
+			break;
+
+		if (nr_pages <= max)
+			break;
+
+		if (signal_pending(current))
+			break;
+
+		/* cgroup_rmdir() waits for us with cgroup_mutex held. */
+		if (memcg_is_dying(memcg))
+			break;
+
+		if (!drained) {
+			drain_all_stock(memcg);
+			drained = true;
+			continue;
+		}
+
+		if (nr_reclaims) {
+			if (!try_to_free_mem_cgroup_pages(memcg, nr_pages - max,
+					GFP_KERNEL, 0, NULL))
+				nr_reclaims--;
+			continue;
+		}
+
+		memcg_memory_event(memcg, MEMCG_OOM);
+		if (!mem_cgroup_out_of_memory(memcg, GFP_KERNEL, 0))
+			break;
+		cond_resched();
+	}
+out:
+	memcg_wb_domain_size_changed(memcg);
+	return nbytes;
+}
+
 static struct cftype swap_files[] = {
+	{
+		.name = "memsw.current",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_u64 = memsw_current_read,
+	},
+	{
+		.name = "memsw.max",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.seq_show = memsw_max_show,
+		.write = memsw_max_write,
+	},
 	{
 		.name = "swap.current",
 		.flags = CFTYPE_NOT_ON_ROOT,

-- 
2.43.7

[PATCH 6/6] mm: memcontrol: clamp mem_cgroup_get_max() to the combined limit

[Jingxiang Zeng via B4 Relay]

From: Jingxiang Zeng <linuszeng@tencent.com>

mem_cgroup_get_max() reports the memory ceiling of a cgroup.  Its
default-hierarchy branch derives that ceiling from memory.max plus
memory.swap.max, which overstates the reachable total once a combined
memory+swap limit is configured: with memory.max at 32M,
memory.memsw.max at 48M and 2G of swap online it reports about 2080M.

For memcg OOM, constrained_alloc() stores that value in oc->totalpages,
and oom_badness() scales the task's oom_score_adj by
totalpages / OOM_SCORE_ADJ_MAX, so an overstated ceiling weighs
oom_score_adj far more than intended inside such a cgroup: in the
example above about forty times, enough that a task with a negative
adjustment stops being selectable at all while a positive one is picked
long before its rss would justify it.

Clamp the result to memory.memsw.max, which is what the v1 branch
already derives its ceiling from.  The limit defaults to "max", so this
changes nothing until a combined limit is configured.

Signed-off-by: Jingxiang Zeng <linuszeng@tencent.com>
---
 mm/memcontrol.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 9e8a176e7afb..e229de0d35e0 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1919,6 +1919,12 @@ unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg)
 		if (mem_cgroup_swappiness(memcg))
 			max += min(READ_ONCE(memcg->swap.max),
 				   (unsigned long)total_swap_pages);
+		/*
+		 * A combined memory+swap limit caps the sum of the two, so it
+		 * is the real ceiling once it is configured.  It defaults to
+		 * "max", which leaves the value above unchanged.
+		 */
+		max = min(max, READ_ONCE(memcg->memsw.max));
 	}
 	return max;
 }

-- 
2.43.7