[PATCH v2 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[PATCH v2 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[Konstantin Khorenko]

Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
per 64 memcg-aware shrinker ids up to shrinker_nr_max, whether or not it
ever charges an object to those shrinkers. Every superblock registers
such a shrinker, so the cost is

  memory cgroups x NUMA nodes x shrinker_nr_max / 64 x 1 KiB

because the unit is 520 bytes and comes from kmalloc-1k.

Page cache reclaim does not make this go away:

- Live cgroups keep their units until they are removed.
- The units are not charged to any memcg, so memory.max does not see
  them and a delegated subtree creates cgroups at the host's expense.
- A dying cgroup keeps its units until the last reference to it is
  dropped, for instance by cgroup writeback, a swap entry or a socket.
  Nothing reclaims it for the sake of the units, and MemAvailable
  counts them as unreclaimable.
- shrinker_nr_max never comes down: after a burst of mounts every new
  cgroup pays for the peak until reboot.
- Both factors grow with the number of workloads on the host.

Patch 1 lets a unit cover BITS_PER_LONG - 1 ids so it fills a 512-byte
slot exactly: half the memory, nothing else changes.

Patch 2 allocates a unit only when the cgroup gets a list on that
shrinker's list_lru, which is where map bits are set: a cgroup pays for
the shrinkers it holds objects of, not for every superblock on the host.
The units stay uncharged, but how many a cgroup makes the host allocate
is now bounded by the shrinkers it uses. The unit pointer array in
shrinker_info still grows with shrinker_nr_max, at 8 bytes per 63 ids
per node. Patch 2 relies on commit fafaeceb89a5 ("mm: switch deferred
split shrinker to list_lru") and is not meant for stable.

On a host running a few hundred containers (45000 memory cgroups, most
of them dying, shrinker_nr_max 52480, 4 nodes) the units took 145 GB of
377 GB.

Reproducer without containers: 48 CPUs, 4 nodes, v7.3-rc6, fresh boot
per kernel. 8000 tmpfs mounts raise shrinker_nr_max to 8064, then 2000
empty memory cgroups are created. Active objects added to the 512-byte
and 1-KiB kmalloc caches, and slab per cgroup:

                  objects added       slab per cgroup
  base            1011518 x 1024 B    548 KB
  + patch 1       1023787 x  512 B    289 KB
  + patches 1-2       136 x  512 B     42 KB

The 42 KB left is the same on all three: the memcg's own structures and
8 KB of shrinker_info pointer arrays at this shrinker_nr_max. A cgroup
that looks up names on two filesystems gets 8 units (4 KB) with both
patches, and so does each ancestor that had none. Writing a page of a
file in each cgroup and removing the cgroups keeps the same amounts
pinned by the 2000 dying cgroups until the cache is dropped.

Tested with selftests/cgroup test_kmem and test_memcontrol, a check that
memory.reclaim reaches 140 ext4 superblocks with consecutive shrinker
ids, and unit installs from 64 cgroups racing with 3000 mounts that keep
growing shrinker_nr_max. On a kernel with lockdep, KASAN and DEBUG_VM the
same runs leave nothing in dmesg. test_memcg_sock fails there now and
then, with and without the series: memory.stat's sock counter is not
flushed yet when the test compares it with memory.current.

---
Changes in v2:
- Patch 1: check the unit size with BUILD_BUG_ON(!is_power_of_2())
  in shrinker_unit_alloc() instead of an open-coded static_assert()
  in the header (Muchun Song).
- Link to v1: https://patch.msgid.link/20261012-b4-prep-vstor-144715-size-shrinker_info_unit-to-a-power-of-two-v1-0-5a7608393b7a@virtuozzo.com

---
Konstantin Khorenko (2):
      mm/shrinker: size shrinker_info_unit to a power of two
      mm/shrinker: allocate the per-memcg shrinker units on demand

 include/linux/memcontrol.h |   2 +
 include/linux/shrinker.h   |   6 +-
 mm/list_lru.c              |   6 ++
 mm/shrinker.c              | 163 ++++++++++++++++++++++++++++++++++++---------
 4 files changed, 146 insertions(+), 31 deletions(-)
---
base-commit: a5ebb76233b79db01e82e061173cb73c6d2b5c6b
change-id: 20261011-b4-prep-vstor-144715-size-shrinker_info_unit-to-a-power-of-two-9f2e043163af

Best regards,
--  
Konstantin Khorenko <khorenko@virtuozzo.com>

[PATCH v2 1/2] mm/shrinker: size shrinker_info_unit to a power of two

[Konstantin Khorenko]

Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
for every SHRINKER_UNIT_BITS shrinker ids up to shrinker_nr_max. The
unit holds BITS_PER_LONG atomic_long_t of deferred work, a power of
two, plus the one-word bitmap of shrinkers with objects charged to the
cgroup. That word takes the unit just past the power of two, to 520
bytes on 64-bit, so every unit comes from kmalloc-1k and almost half of
it is padding.

The units are allocated when the cgroup goes online and freed when it is
freed, so a dying cgroup keeps them until the last reference to it is
gone. Their number is the number of memory cgroups, times the NUMA
nodes, times shrinker_nr_max / SHRINKER_UNIT_BITS. shrinker_nr_max is
the peak number of memcg-aware shrinkers registered at once, and every
superblock registers one, so a host that mounts many filesystems and
runs many cgroups carries the padding as unreclaimable slab, and keeps
paying it for new cgroups after the mounts are gone.

Let a unit cover one id less than a word holds. The bitmap still fits
one word, and nr_deferred[] plus that word add up to BITS_PER_LONG
longs, so the unit fills a power-of-two kmalloc slot exactly and, on
64-bit, the memory taken by units halves, for about one unit more in
BITS_PER_LONG.
round_up() in expand_shrinker_info() needs a power-of-two step and
becomes roundup(); the other uses of SHRINKER_UNIT_BITS are divisions,
modulos and multiplications that do not care. Assert the unit size at
build time, since the layout depends on it.

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 include/linux/shrinker.h | 6 +++++-
 mm/shrinker.c            | 4 +++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h
index 1a00be90d93a9..f1cf6096a7a06 100644
--- a/include/linux/shrinker.h
+++ b/include/linux/shrinker.h
@@ -7,7 +7,11 @@
 #include <linux/refcount.h>
 #include <linux/completion.h>
 
-#define SHRINKER_UNIT_BITS	BITS_PER_LONG
+/*
+ * One id less than a word holds, so that nr_deferred[] and the map word add
+ * up to BITS_PER_LONG longs: a power of two, which fills a kmalloc slot.
+ */
+#define SHRINKER_UNIT_BITS	(BITS_PER_LONG - 1)
 
 /*
  * Bitmap and deferred work of shrinker::id corresponding to memcg-aware
diff --git a/mm/shrinker.c b/mm/shrinker.c
index 7ec2a9704f6f2..5e4a5ce231868 100644
--- a/mm/shrinker.c
+++ b/mm/shrinker.c
@@ -46,6 +46,8 @@ static inline int shrinker_unit_alloc(struct shrinker_info *new,
 	int start = old ? DIV_ROUND_UP(old->map_nr_max, SHRINKER_UNIT_BITS) : 0;
 	int i;
 
+	BUILD_BUG_ON(!is_power_of_2(sizeof(struct shrinker_info_unit)));
+
 	for (i = start; i < nr; i++) {
 		unit = kzalloc_node(sizeof(*unit), GFP_KERNEL, nid);
 		if (!unit) {
@@ -159,7 +161,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
 static int expand_shrinker_info(int new_id)
 {
 	int ret = 0;
-	int new_nr_max = round_up(new_id + 1, SHRINKER_UNIT_BITS);
+	int new_nr_max = roundup(new_id + 1, SHRINKER_UNIT_BITS);
 	int new_size, old_size = 0;
 	struct mem_cgroup *memcg;
 

-- 
2.43.0

[PATCH v2 2/2] mm/shrinker: allocate the per-memcg shrinker units on demand

[Konstantin Khorenko]

A memory cgroup allocates a shrinker_info_unit for every
SHRINKER_UNIT_BITS ids up to shrinker_nr_max on every NUMA node when it
comes online, whether or not it will ever hold objects of those
shrinkers. Every superblock registers a memcg-aware shrinker, so on a
host with many mounts each new cgroup pays for all of them while its
tasks touch a few.

A map bit is only ever set on a memcg that holds a list_lru list of
that shrinker. So the point where a memcg gets such a list is the one
place a unit can be installed ahead of its first use, and that is where
it is installed.

The rule the code now relies on:

  A non-root memcg that has a list on a list_lru has the unit of that
  list's shrinker, on every node.

It holds because:

  - the unit is installed before the list is published;
  - a unit lives until the memcg is freed.

So set_shrinker_bit() always finds the unit. A reader that finds none
knows the memcg never held objects of that shrinker: there is nothing
to walk and no deferred work.

There is one case where a memcg has no list: with cgroup.memory=nokmem
a list_lru is never per-memcg, but a SHRINKER_MEMCG_AWARE shrinker that
also has SHRINKER_NONSLAB still gets an id. If such a shrinker keeps
lists of its own and sets map bits itself, it has to call
memcg_shrinker_unit_alloc() first. No shrinker in the tree does.

The install runs with the gfp of the list allocation, possibly atomic,
so it cannot take shrinker_mutex. A spinlock orders it against the info
replacement in expand_one_shrinker_info() instead, so a unit cannot be
installed into an info that is being copied out of.

The root memcg keeps all its units: global reclaim keeps the deferred
work of memcg-aware shrinkers there, and root has no list allocation to
install them from.

The units of a memcg now scale with the number of shrinkers it and its
descendants hold objects of, not with shrinker_nr_max.

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 include/linux/memcontrol.h |   2 +
 mm/list_lru.c              |   6 ++
 mm/shrinker.c              | 159 ++++++++++++++++++++++++++++++++++++---------
 3 files changed, 138 insertions(+), 29 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 7d1c0ce189a88..a14f43c3a1b01 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1676,6 +1676,8 @@ static inline u64 mem_cgroup_get_socket_pressure(struct mem_cgroup *memcg)
 #endif
 
 int alloc_shrinker_info(struct mem_cgroup *memcg);
+int memcg_shrinker_unit_alloc(struct mem_cgroup *memcg, int shrinker_id,
+			      gfp_t gfp);
 void free_shrinker_info(struct mem_cgroup *memcg);
 void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id);
 void reparent_shrinker_deferred(struct mem_cgroup *memcg);
diff --git a/mm/list_lru.c b/mm/list_lru.c
index 36662d02ff963..7247b7c081b4c 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -597,6 +597,12 @@ static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,
 			parent = parent_mem_cgroup(pos);
 		}
 
+		if (!css_is_dying(&pos->css) &&
+		    memcg_shrinker_unit_alloc(pos, lru_shrinker_id(lru), gfp)) {
+			kfree(mlru);
+			return -ENOMEM;
+		}
+
 		if (!mlru) {
 			mlru = memcg_init_list_lru_one(lru, gfp);
 			if (!mlru)
diff --git a/mm/shrinker.c b/mm/shrinker.c
index 5e4a5ce231868..fbe20e105f293 100644
--- a/mm/shrinker.c
+++ b/mm/shrinker.c
@@ -13,6 +13,14 @@ DEFINE_MUTEX(shrinker_mutex);
 #ifdef CONFIG_MEMCG
 static int shrinker_nr_max;
 
+/*
+ * Orders installing a unit against the pointer copy in
+ * expand_one_shrinker_info(). Installs run from the list_lru allocation
+ * path with the caller's gfp and cannot wait for shrinker_mutex; they can
+ * nest in the i_pages lock, which is taken from interrupts.
+ */
+static DEFINE_SPINLOCK(shrinker_unit_lock);
+
 static inline int shrinker_unit_size(int nr_items)
 {
 	return (DIV_ROUND_UP(nr_items, SHRINKER_UNIT_BITS) * sizeof(struct shrinker_info_unit *));
@@ -30,9 +38,6 @@ static inline void shrinker_unit_free(struct shrinker_info *info, int start)
 	nr = DIV_ROUND_UP(info->map_nr_max, SHRINKER_UNIT_BITS);
 
 	for (i = start; i < nr; i++) {
-		if (!unit[i])
-			break;
-
 		kfree(unit[i]);
 		unit[i] = NULL;
 	}
@@ -98,7 +103,13 @@ int alloc_shrinker_info(struct mem_cgroup *memcg)
 		if (!info)
 			goto err;
 		info->map_nr_max = shrinker_nr_max;
-		if (shrinker_unit_alloc(info, NULL, nid)) {
+		/*
+		 * Global reclaim keeps the deferred work of memcg-aware
+		 * shrinkers in the root's units, and root has no list_lru
+		 * allocation to install them from.
+		 */
+		if (mem_cgroup_is_root(memcg) &&
+		    shrinker_unit_alloc(info, NULL, nid)) {
 			kvfree(info);
 			goto err;
 		}
@@ -126,6 +137,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
 {
 	struct shrinker_info *new, *old;
 	struct mem_cgroup_per_node *pn;
+	unsigned long flags;
 	int nid;
 
 	for_each_node(nid) {
@@ -145,13 +157,20 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size,
 
 		new->map_nr_max = new_nr_max;
 
-		memcpy(new->unit, old->unit, old_size);
-		if (shrinker_unit_alloc(new, old, nid)) {
+		if (mem_cgroup_is_root(memcg) &&
+		    shrinker_unit_alloc(new, old, nid)) {
 			kvfree(new);
 			return -ENOMEM;
 		}
 
+		/*
+		 * A unit installed meanwhile lands in old before the copy or
+		 * in new after the publish.
+		 */
+		spin_lock_irqsave(&shrinker_unit_lock, flags);
+		memcpy(new->unit, old->unit, old_size);
 		rcu_assign_pointer(pn->shrinker_info, new);
+		spin_unlock_irqrestore(&shrinker_unit_lock, flags);
 		kvfree_rcu(old, rcu);
 	}
 
@@ -206,21 +225,26 @@ static inline int calc_shrinker_id(int index, int offset)
 
 void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id)
 {
-	if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) {
-		struct shrinker_info *info;
+	struct shrinker_info_unit *unit;
+	struct shrinker_info *info;
 
-		rcu_read_lock();
-		info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
-		if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) {
-			struct shrinker_info_unit *unit;
-
-			unit = info->unit[shrinker_id_to_index(shrinker_id)];
-			/* Pairs with smp mb in shrink_slab() */
-			smp_mb__before_atomic();
-			set_bit(shrinker_id_to_offset(shrinker_id), unit->map);
-		}
-		rcu_read_unlock();
-	}
+	if (shrinker_id < 0 || !memcg || mem_cgroup_is_root(memcg))
+		return;
+
+	rcu_read_lock();
+	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
+	if (WARN_ON_ONCE(shrinker_id >= info->map_nr_max))
+		goto unlock;
+
+	unit = READ_ONCE(info->unit[shrinker_id_to_index(shrinker_id)]);
+	if (WARN_ON_ONCE(!unit))
+		goto unlock;
+
+	/* Pairs with smp mb in shrink_slab() */
+	smp_mb__before_atomic();
+	set_bit(shrinker_id_to_offset(shrinker_id), unit->map);
+unlock:
+	rcu_read_unlock();
 }
 
 static DEFINE_IDR(shrinker_idr);
@@ -265,14 +289,16 @@ static void shrinker_memcg_remove(struct shrinker *shrinker)
 static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker,
 				   struct mem_cgroup *memcg)
 {
+	int offset = shrinker_id_to_offset(shrinker->id);
 	struct shrinker_info *info;
 	struct shrinker_info_unit *unit;
 	long nr_deferred;
 
 	rcu_read_lock();
 	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
-	unit = info->unit[shrinker_id_to_index(shrinker->id)];
-	nr_deferred = atomic_long_xchg(&unit->nr_deferred[shrinker_id_to_offset(shrinker->id)], 0);
+	unit = READ_ONCE(info->unit[shrinker_id_to_index(shrinker->id)]);
+	/* No unit: this memcg never held objects of this shrinker. */
+	nr_deferred = unit ? atomic_long_xchg(&unit->nr_deferred[offset], 0) : 0;
 	rcu_read_unlock();
 
 	return nr_deferred;
@@ -281,20 +307,83 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker,
 static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker,
 				  struct mem_cgroup *memcg)
 {
+	int offset = shrinker_id_to_offset(shrinker->id);
 	struct shrinker_info *info;
 	struct shrinker_info_unit *unit;
 	long nr_deferred;
 
 	rcu_read_lock();
 	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
-	unit = info->unit[shrinker_id_to_index(shrinker->id)];
-	nr_deferred =
-		atomic_long_add_return(nr, &unit->nr_deferred[shrinker_id_to_offset(shrinker->id)]);
+	unit = READ_ONCE(info->unit[shrinker_id_to_index(shrinker->id)]);
+	nr_deferred = unit ? atomic_long_add_return(nr, &unit->nr_deferred[offset]) : 0;
 	rcu_read_unlock();
 
 	return nr_deferred;
 }
 
+static int shrinker_unit_install(struct mem_cgroup *memcg, int nid, int index,
+				 gfp_t gfp)
+{
+	struct shrinker_info_unit *unit;
+	struct shrinker_info *info;
+	unsigned long flags;
+
+	rcu_read_lock();
+	info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info);
+	/* Online memcgs have an info, and ids are expanded into it first. */
+	if (WARN_ON_ONCE(!info ||
+			 index >= shrinker_id_to_index(info->map_nr_max)) ||
+	    READ_ONCE(info->unit[index])) {
+		rcu_read_unlock();
+		return 0;
+	}
+	rcu_read_unlock();
+
+	unit = kzalloc_node(sizeof(*unit), gfp, nid);
+	if (!unit)
+		return -ENOMEM;
+
+	spin_lock_irqsave(&shrinker_unit_lock, flags);
+	/* memcg is pinned: only expand replaces info, and it holds the lock */
+	info = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info,
+					 lockdep_is_held(&shrinker_unit_lock));
+	if (!info->unit[index]) {
+		/* Walkers read the pointer locklessly: publish a zeroed unit. */
+		smp_store_release(&info->unit[index], unit);
+		unit = NULL;
+	}
+	spin_unlock_irqrestore(&shrinker_unit_lock, flags);
+
+	kfree(unit);
+	return 0;
+}
+
+/*
+ * Called before @memcg gets a list on a list_lru of shrinker @shrinker_id,
+ * from the path that allocates the list. A map bit is only ever set on a
+ * memcg that has the list, so set_shrinker_bit() always finds the unit.
+ * A shrinker that sets map bits without a list_lru, such as a NONSLAB
+ * one under cgroup.memory=nokmem, must call this first.
+ */
+int memcg_shrinker_unit_alloc(struct mem_cgroup *memcg, int shrinker_id,
+			      gfp_t gfp)
+{
+	int nid, ret;
+
+	if (shrinker_id < 0 || mem_cgroup_is_root(memcg))
+		return 0;
+
+	for_each_node(nid) {
+		ret = shrinker_unit_install(memcg, nid,
+					    shrinker_id_to_index(shrinker_id),
+					    gfp);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 void reparent_shrinker_deferred(struct mem_cgroup *memcg)
 {
 	int nid, index, offset;
@@ -309,8 +398,17 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg)
 		child_info = shrinker_info_protected(memcg, nid);
 		parent_info = shrinker_info_protected(parent, nid);
 		for (index = 0; index < shrinker_id_to_index(child_info->map_nr_max); index++) {
-			child_unit = child_info->unit[index];
-			parent_unit = parent_info->unit[index];
+			child_unit = READ_ONCE(child_info->unit[index]);
+			if (!child_unit)
+				continue;
+
+			if (!READ_ONCE(parent_info->unit[index]))
+				shrinker_unit_install(parent, nid, index, GFP_KERNEL);
+			parent_unit = READ_ONCE(parent_info->unit[index]);
+			/* Drop the debt rather than fail the cgroup teardown. */
+			if (!parent_unit)
+				continue;
+
 			for (offset = 0; offset < SHRINKER_UNIT_BITS; offset++) {
 				nr = atomic_long_read(&child_unit->nr_deferred[offset]);
 				atomic_long_add(nr, &parent_unit->nr_deferred[offset]);
@@ -531,10 +629,13 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,
 	if (index < shrinker_id_to_index(info->map_nr_max)) {
 		struct shrinker_info_unit *unit;
 
-		unit = info->unit[index];
+		unit = READ_ONCE(info->unit[index]);
 
 		rcu_read_unlock();
 
+		if (!unit)
+			goto next;
+
 		for_each_set_bit(offset, unit->map, SHRINKER_UNIT_BITS) {
 			struct shrink_control sc = {
 				.gfp_mask = gfp_mask,
@@ -589,7 +690,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,
 			freed += ret;
 			shrinker_put(shrinker);
 		}
-
+next:
 		index++;
 		goto again;
 	}

-- 
2.43.0

Re: [PATCH v2 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[Andrew Morton]

On Wed, 14 Oct 2026 16:54:36 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:

> Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
> per 64 memcg-aware shrinker ids up to shrinker_nr_max, whether or not it
> ever charges an object to those shrinkers. Every superblock registers
> such a shrinker, so the cost is
> 
>   memory cgroups x NUMA nodes x shrinker_nr_max / 64 x 1 KiB
> 
> because the unit is 520 bytes and comes from kmalloc-1k.

So we could almost halve this just by using a dedicated kmem_cache?

Re: [PATCH v2 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[Konstantin Khorenko]

On 10/14/26 22:50, Andrew Morton wrote:
> On Wed, 14 Oct 2026 16:54:36 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
> 
>> Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
>> per 64 memcg-aware shrinker ids up to shrinker_nr_max, whether or not it
>> ever charges an object to those shrinkers. Every superblock registers
>> such a shrinker, so the cost is
>>
>>   memory cgroups x NUMA nodes x shrinker_nr_max / 64 x 1 KiB
>>
>> because the unit is 520 bytes and comes from kmalloc-1k.
> 
> So we could almost halve this just by using a dedicated kmem_cache?

Yeap, a dedicated cache gets approximately the same cost per shrinker id.
On a 48-CPU box SLUB puts 31 units in an order-2 slab, about 528 bytes each:

                                      ids/unit  bytes/unit  bytes/id
    today, kmalloc-1k                    64        1024        16
    patch 1, kmalloc-512                 63         512       ~8.1
    dedicated cache                      64        ~528       ~8.3
    dedicated cache, HWCACHE_ALIGN       64         576        9

So the two are ~equal on memory, and the choice is about the code.

I went with 63 ids per unit because the unit then simply fills the
kmalloc slot it already comes from.

The price is an odd SHRINKER_UNIT_BITS and roundup() instead of round_up(),
plus the static_assert.

If you prefer the cache, I'm happy to redo patch 1 that way.
Patch 2 does not care about the unit size.

--
Best regards,
Konstantin

Re: [PATCH v2 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[Muchun Song]

> On Oct 15, 2026, at 19:42, Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
> 
> On 10/14/26 22:50, Andrew Morton wrote:
>>> On Wed, 14 Oct 2026 16:54:36 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
>>> 
>>> Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
>>> per 64 memcg-aware shrinker ids up to shrinker_nr_max, whether or not it
>>> ever charges an object to those shrinkers. Every superblock registers
>>> such a shrinker, so the cost is
>>> 
>>>  memory cgroups x NUMA nodes x shrinker_nr_max / 64 x 1 KiB
>>> 
>>> because the unit is 520 bytes and comes from kmalloc-1k.
>> 
>> So we could almost halve this just by using a dedicated kmem_cache?
> 
> Yeap, a dedicated cache gets approximately the same cost per shrinker id.
> On a 48-CPU box SLUB puts 31 units in an order-2 slab, about 528 bytes each:
> 
>                                      ids/unit  bytes/unit  bytes/id
>    today, kmalloc-1k                    64        1024        16
>    patch 1, kmalloc-512                 63         512       ~8.1
>    dedicated cache                      64        ~528       ~8.3
>    dedicated cache, HWCACHE_ALIGN       64         576        9
> 
> So the two are ~equal on memory, and the choice is about the code.
> 
> I went with 63 ids per unit because the unit then simply fills the
> kmalloc slot it already comes from.
> 
> The price is an odd SHRINKER_UNIT_BITS and roundup() instead of round_up(),
> plus the static_assert.
> 
> If you prefer the cache, I'm happy to redo patch 1 that way.
> Patch 2 does not care about the unit size.

Please use a dedicated cache, more straightforward for me. But also give some time to maintainers for reviewing the patch 2 before your update.

Thanks.

> 
> --
> Best regards,
> Konstantin
> 

Re: [PATCH v2 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[Andrew Morton]

On Thu, 15 Oct 2026 13:42:41 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:

> On 10/14/26 22:50, Andrew Morton wrote:
> > On Wed, 14 Oct 2026 16:54:36 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
> > 
> >> Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
> >> per 64 memcg-aware shrinker ids up to shrinker_nr_max, whether or not it
> >> ever charges an object to those shrinkers. Every superblock registers
> >> such a shrinker, so the cost is
> >>
> >>   memory cgroups x NUMA nodes x shrinker_nr_max / 64 x 1 KiB
> >>
> >> because the unit is 520 bytes and comes from kmalloc-1k.
> > 
> > So we could almost halve this just by using a dedicated kmem_cache?
> 
> Yeap, a dedicated cache gets approximately the same cost per shrinker id.
> On a 48-CPU box SLUB puts 31 units in an order-2 slab, about 528 bytes each:
> 
>                                       ids/unit  bytes/unit  bytes/id
>     today, kmalloc-1k                    64        1024        16
>     patch 1, kmalloc-512                 63         512       ~8.1
>     dedicated cache                      64        ~528       ~8.3
>     dedicated cache, HWCACHE_ALIGN       64         576        9
> 
> So the two are ~equal on memory, and the choice is about the code.
> 
> I went with 63 ids per unit because the unit then simply fills the
> kmalloc slot it already comes from.
> 
> The price is an odd SHRINKER_UNIT_BITS and roundup() instead of round_up(),
> plus the static_assert.
> 
> If you prefer the cache, I'm happy to redo patch 1 that way.
> Patch 2 does not care about the unit size.

OK, thanks.  Yes, using a kmem_cache is a simple no-brainer.  Let's get
that done first and then let's look at additional optimizations on top
as a separate activity.

But a tweak which reduces 145GB of bloat to half of that is lipsticking
the pig, and what a pig it is.

Why the heck is memcg allocating so many of these things?  What sort of
rip-out/rethink/start-over can we do to reduce this by 99.x%?

Re: [PATCH v2 0/2] mm/shrinker: make per-memcg shrinker units cheaper

[Konstantin Khorenko]

On 10/16/26 02:58, Andrew Morton wrote:
> On Thu, 15 Oct 2026 13:42:41 +0200 Konstantin Khorenko <khorenko@virtuozzo.com> wrote:
...
>> If you prefer the cache, I'm happy to redo patch 1 that way.
>> Patch 2 does not care about the unit size.
> 
> OK, thanks.  Yes, using a kmem_cache is a simple no-brainer.  Let's get
> that done first and then let's look at additional optimizations on top
> as a separate activity.

OK, no problem. I'll redo patch 1 with a dedicated kmem_cache and send
it separately.


> But a tweak which reduces 145GB of bloat to half of that is lipsticking
> the pig, and what a pig it is.
> 
> Why the heck is memcg allocating so many of these things?  What sort of
> rip-out/rethink/start-over can we do to reduce this by 99.x%?

That is what patch 2 does:
  [PATCH v2 2/2] mm/shrinker: allocate the per-memcg shrinker units on demand

Today every memcg allocates, on every node, a unit for every 63 shrinker
ids up to shrinker_nr_max, whether or not it ever has an object on those
shrinkers. Patch 2 allocates a unit only when the memcg gets a list on that
shrinker's list_lru, so a memcg pays for the shrinkers it uses, not for
every superblock on the host.

In the synthetic test from the cover letter (8000 tmpfs mounts,
shrinker_nr_max 8064, 2000 empty memcgs, 4 nodes):

                     units               slab per memcg
  base               1011518 x 1024 B    548 KB
  + patch 1          1023787 x  512 B    289 KB
  + patches 1-2          136 x  512 B     42 KB

So patch 2 removes 99.99% of the units, not half. These are the numbers
of a synthetic test, of course. The 42 KB left is the memcg's own
structures plus 8 KB of shrinker_info pointer arrays. A memcg that uses
two filesystems gets 8 units (4 KB) on top of that.

On the host from the cover letter I'd expect the 145 GB to drop to the
pointer arrays plus the units the memcgs really use. The arrays are
about 1.4 GB there (45000 memcgs x 4 nodes x 8 KB at shrinker_nr_max
52480). That is an estimate, not a measurement.

What remains is the pointer array: it is still dense in
shrinker_nr_max, per memcg and per node. We can finish that path, or
do something different.

One option is to replace the array with an xarray of units per memcg
and node, indexed the same way.

Pros:

- An empty memcg pays nothing, and a memcg that uses a few units pays
  one or two 576-byte xa_nodes per node. The worst case is about what
  the array costs today: a leaf covers 64 units, i.e. 4032 ids, so at
  shrinker_nr_max 52480 a memcg that touches every leaf needs 15 nodes
  (8.6 KB) against an 8 KB array.

- expand_shrinker_info() goes away. Today a mount that raises
  shrinker_nr_max past a multiple of 63 reallocates and copies the
  array of every memcg on every node under shrinker_mutex: 180000
  allocations with 45000 memcgs on 4 nodes.

- shrink_slab_memcg() walks only the units present, with xa_find(),
  instead of every index up to shrinker_nr_max / 63.

- The shrinker_unit_lock that patch 2 adds to order an install against
  the copy in expand is no longer needed, the xa_lock does that.

Cons:

- The install can run under the i_pages lock with interrupts off
  (__filemap_add_folio() -> xas_alloc() -> memcg_list_lru_alloc()), so
  the xarray needs XA_FLAGS_LOCK_IRQ and its nodes are allocated with
  the caller's gfp. That is one more allocation on that path that can
  fail. A failure fails the list allocation, as a failed unit
  allocation does today.

- set_shrinker_bit() and the nr_deferred accessors do an xa_load(), a
  two-level walk at these sizes, instead of an array index.
  set_shrinker_bit() runs when a list goes from empty to non-empty,
  not per object, so I don't expect it to show.

- Root still needs its units up front for the deferred work of global
  reclaim. They would be installed when a shrinker registers instead
  of in expand.

Size: it would replace patch 2 with two patches. The first turns the
array into an xarray, with the units still allocated up front, and
removes struct shrinker_info and the expand path. The second allocates
the units on demand, without the lock and the expand handling patch 2
needs now. Together I expect about the size of patch 2 today, around
170 changed lines, mostly in mm/shrinker.c, with fewer lines in total
than now.

Which way would you prefer: patch 2 as it is, or the xarray version in
its place? Or something else?


Cheers,
Konstantin