[RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches

[RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches

[Hongfeng Chen]

Freeing swap entries is expensive when unmapping a VMA that holds many of
them. `zap_pte_range()` eventually calls `zram_slot_free_notify()`, which
frees one compressed object per page. As a result, the unmapping task
has to pay for every `zs_free()` itself. On Android, this slows down the
exit of a task killed to reclaim memory, precisely when that memory is
needed.

Two earlier attempts by Lei and Zhiguo introduced a new thread in the MM
core to asynchronously collect and free swap entries [1][2], which
seems overcomplicated. Wenchao's RFC v3 [3] moved the deferral into
zsmalloc, but introduced push/drain callbacks that zram and zswap had
to implement, potentially violating software layer boundaries. This v4
RFC defers freeing inside `zs_free()` itself, leaving callers unchanged.

Patch 1 makes `zs_free()` append the handle to a per-CPU batch of 64
handles instead of freeing it immediately. Once the batch is full, it
is handed to an unbound drain worker through a lockless list, and the
worker frees the objects. Releasing empty zspages is also deferred to
the worker. If the per-CPU state cannot be allocated, `zs_free()` falls
back to its original path.

With the zsmalloc work moved off the unmapping path by patch 1,
contention on atomic operations in zram statistics becomes significant.
Patch 2 moves the contended counters to per-CPU storage.


Using the following microbenchmark:

// SPDX-License-Identifier: GPL-2.0
/* Times munmap() on a region pushed out to zram: MB. */
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <time.h>
#include <unistd.h>

#ifndef MADV_PAGEOUT
#define MADV_PAGEOUT 21
#endif

static void die(const char *what)
{
	perror(what);
	exit(1);
}

static double now_ms(void)
{
	struct timespec ts;

	clock_gettime(CLOCK_MONOTONIC, &ts);
	return ts.tv_sec * 1e3 + ts.tv_nsec / 1e6;
}

/* About 2.8:1 under lzo-rle, objects in 16 size classes. */
static void fill_data(char *p, size_t bytes)
{
	size_t i, k, n;

	memset(p, 0, bytes);
	for (i = 0; i < bytes; i += 4096) {
		long *q = (long *)(p + i);

		n = 106 + i / 4096 % 16 * 8;
		for (k = 0; k < n; k++)
			q[k] = k * 8 * 2654435761UL;
	}
}

static char *prepare(size_t bytes)
{
	char *p = mmap(NULL, bytes, PROT_READ | PROT_WRITE,
		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

	if (p == MAP_FAILED)
		die("mmap");
	fill_data(p, bytes);
	if (madvise(p, bytes, MADV_PAGEOUT))
		die("madvise");
	return p;
}

int main(int argc, char **argv)
{
	size_t bytes = (argc > 1 ? strtoul(argv[1], NULL, 10) : 256) << 20;
	char *p = prepare(bytes);
	double t0, t1;

	raise(SIGSTOP);	/* ready */
	t0 = now_ms();
	munmap(p, bytes);
	t1 = now_ms();
	printf("%.6f %.6f\n", t0, t1);
	return 0;
}

Performance results:

ROCK 5B+ (RK3588: 4x Cortex-A55 at 1.8GHz, 4x Cortex-A76 at 2.4GHz,
performance governor), single process munmap on A55:

  times in ms, speedups over base
              Base       Patch1       Patch1+2
  size        time    time speedup   time speedup
  64MB       13.81    3.62   3.81x   3.28   4.21x
  128MB      28.50    7.18   3.97x   6.49   4.39x
  192MB      42.19   10.71   3.94x   9.69   4.36x
  256MB      56.06   14.26   3.93x   12.89  4.35x
  512MB     109.50   28.43   3.85x   25.66  4.27x
  1024MB    217.10   56.70   3.83x   51.15  4.24x

Same machine, single process munmap on A76:

  times in ms, speedups over base
              Base       Patch1        Patch1+2
  size        time    time speedup    time speedup
  64MB        5.59    1.58   3.54x    1.49   3.76x
  128MB      11.86    3.16   3.75x    2.95   4.02x
  192MB      17.18    4.72   3.64x    4.45   3.86x
  256MB      22.89    6.29   3.64x    5.87   3.90x
  512MB      45.15   12.52   3.61x   11.64   3.88x
  1024MB     89.16   24.75   3.60x   23.12   3.86x

x86 (Intel Core i7-14700HX, P-cores locked at 2.1GHz with turbo off),
single process munmap:

  times in ms, speedups over base
              Base       Patch1         Patch1+2
  size        time    time speedup    time speedup
  64MB        2.87    1.50   1.92x    1.14   2.53x
  128MB       6.00    2.93   2.05x    2.21   2.71x
  192MB       9.33    4.35   2.15x    3.33   2.80x
  256MB      12.51    5.78   2.17x    4.38   2.85x
  512MB      24.96   11.39   2.19x    8.62   2.90x
  1024MB     50.19   22.78   2.20x   17.05   2.94x

1-4 process munmap of 256MB each:

  times in ms, speedups over base
              Base       Patch1        Patch1+2
  mode        time    time speedup   time speedup
  1p         13.18    5.78   2.28x   4.35   3.03x
  2p         25.83   14.41   1.79x   4.57   5.65x
  3p         35.70   22.14   1.61x   4.68   7.63x
  4p         44.25   28.71   1.54x   4.75   9.31x

[1] https://lore.kernel.org/all/20240805153639.1057-1-justinjiang@vivo.com/
[2] https://lore.kernel.org/all/20250909065349.574894-1-liulei.rjpt@vivo.com/
[3] https://lore.kernel.org/linux-mm/20260508060724.3810904-1-haowenchao@xiaomi.com/

Changes since v3:
- Defer inside zs_free() only and drop the push/drain callbacks, so zram
  and zswap do not change. Yosry and Nhat objected to the layering in v3;
  Nhat also pointed out that v3 leaves torn-down entries on the zswap LRU.
- Buffer handles in per-cpu batches of 64 allocated on demand, instead of
  per-cpu pages rotated through a preallocated pool.
- New patch 2: keep the hot zram statistics per cpu.
- Link to v3: https://lore.kernel.org/linux-mm/20260508060724.3810904-1-haowenchao@xiaomi.com/

Changes since v2:
- Use per-cpu single-page buffers instead of a global list; the hot path
  only writes into the local CPU's buffer with preemption disabled
- Add a page pool for buffer rotation: when the current buffer is full,
  swap it with a free page from the pool and queue the full page for drain
- Introduce push/drain callback ops so that zram and zswap can each define
  their own element size and drain logic (zram stores u32 slot indices,
  zswap stores unsigned long handles)
- Drop the lock optimization patches it will be submitted separately as
  part of a dedicated zsmalloc lock contention series
- Link to v2: https://lore.kernel.org/all/20260421121616.3298845-1-haowenchao@xiaomi.com/

Hongfeng Chen (2):
  mm/zsmalloc: defer object free to per-cpu batches and a drain worker
  zram: keep the hot statistics per cpu

 drivers/block/zram/zram_drv.c | 171 +++++++++++++++++++++++++++-----
 drivers/block/zram/zram_drv.h |  16 ++-
 mm/zsmalloc.c                 | 182 ++++++++++++++++++++++++++++------
 3 files changed, 309 insertions(+), 60 deletions(-)


base-commit: f4e9810097537d862c128935c36216bcf0c31cb2
-- 
2.54.0

[RFC PATCH v4 1/2] mm/zsmalloc: defer object free to per-cpu batches and a drain worker

[Hongfeng Chen]

Unmapping a VMA full of swap entries makes the unmapping task free every
compressed object itself, one zs_free() call per page. On Android this
slows down the exit of a task killed to reclaim memory.

Instead of freeing the object right away, zs_free() appends its handle to
a per-cpu batch of 64 handles. A full batch goes to an unbound drain worker
through a lockless list, and the worker frees the objects. This batches the
release and moves it off the unmapping path, and the change stays within
zsmalloc.

Freeing empty zspages is left to the worker too: it does that directly when
it can take the page locks, and reuses kick_deferred_free() to free them
asynchronously when it cannot.

Failing to allocate the per-cpu state is not fatal: pool->defer stays NULL
and zs_free() keeps its old path.

Signed-off-by: Hongfeng Chen <hongfengchen@xiaomi.com>
---
 mm/zsmalloc.c | 182 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 154 insertions(+), 28 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 7ef80e0da626..eacd771bd542 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -43,6 +43,7 @@
 #include <linux/zsmalloc.h>
 #include <linux/fs.h>
 #include <linux/workqueue.h>
+#include <linux/llist.h>
 #include "zpdesc.h"
 
 #define ZSPAGE_MAGIC	0x58
@@ -267,6 +268,21 @@ struct link_free {
 static struct kmem_cache *handle_cachep;
 static struct kmem_cache *zspage_cachep;
 
+#define ZS_DEFER_BATCH	64
+
+struct zs_defer_batch {
+	struct llist_node node;
+	unsigned int count;
+	unsigned long handles[ZS_DEFER_BATCH];
+};
+
+struct zs_defer_percpu {
+	struct zs_defer_batch *active;
+	struct llist_head pending;
+	struct work_struct drain_work;
+	struct zs_pool *pool;
+};
+
 struct zs_pool {
 	const char *name;
 
@@ -282,12 +298,12 @@ struct zs_pool {
 #ifdef CONFIG_ZSMALLOC_STAT
 	struct dentry *stat_dentry;
 #endif
-#ifdef CONFIG_COMPACTION
 	struct work_struct free_work;
-#endif
 	/* protect zspage migration/compaction */
 	rwlock_t lock;
 	atomic_t compaction_in_progress;
+
+	struct zs_defer_percpu __percpu *defer;
 };
 
 static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc)
@@ -428,13 +444,11 @@ static bool ZsHugePage(struct zspage *zspage)
 	return zspage->huge;
 }
 
-#ifdef CONFIG_COMPACTION
 static void kick_deferred_free(struct zs_pool *pool);
 static void init_deferred_free(struct zs_pool *pool);
+#ifdef CONFIG_COMPACTION
 static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage);
 #else
-static void kick_deferred_free(struct zs_pool *pool) {}
-static void init_deferred_free(struct zs_pool *pool) {}
 static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage) {}
 #endif
 
@@ -684,7 +698,6 @@ static inline void zs_pool_stat_destroy(struct zs_pool *pool)
 }
 #endif
 
-
 /*
  * For each size class, zspages are divided into different groups
  * depending on their usage ratio. This function returns fullness
@@ -1293,7 +1306,6 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	return obj;
 }
 
-
 /**
  * zs_malloc - Allocate block of given size from pool.
  * @pool: pool to allocate from
@@ -1371,7 +1383,6 @@ static void obj_free(int class_size, unsigned long obj)
 	unsigned int f_objidx;
 	void *vaddr;
 
-
 	obj_to_location(obj, &f_zpdesc, &f_objidx);
 	f_offset = offset_in_page(class_size * f_objidx);
 	zspage = get_zspage(f_zpdesc);
@@ -1447,7 +1458,7 @@ static inline void obj_class_get_and_lock(struct zs_pool *pool, unsigned long ha
 }
 #endif
 
-void zs_free(struct zs_pool *pool, unsigned long handle)
+static void __zs_free_one(struct zs_pool *pool, unsigned long handle)
 {
 	struct zspage *zspage;
 	unsigned long obj;
@@ -1455,9 +1466,6 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	int fullness;
 	struct zspage *zspage_to_free = NULL;
 
-	if (IS_ERR_OR_NULL((void *)handle))
-		return;
-
 	obj_class_get_and_lock(pool, handle, &obj, &zspage, &class);
 
 	class_stat_sub(class, ZS_OBJS_INUSE, 1);
@@ -1483,6 +1491,78 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	}
 	cache_free_handle(handle);
 }
+
+static void zs_defer_drain(struct zs_defer_percpu *defer)
+{
+	struct zs_pool *pool = defer->pool;
+	struct zs_defer_batch *batch, *n;
+	struct llist_node *first;
+
+	first = llist_del_all(&defer->pending);
+	llist_for_each_entry_safe(batch, n, first, node) {
+		unsigned int i;
+
+		for (i = 0; i < batch->count; i++)
+			__zs_free_one(pool, batch->handles[i]);
+		kfree(batch);
+		cond_resched();
+	}
+}
+
+static void zs_defer_drain_work(struct work_struct *work)
+{
+	zs_defer_drain(container_of(work, struct zs_defer_percpu, drain_work));
+}
+
+static bool zs_defer_push(struct zs_pool *pool, unsigned long handle)
+{
+	struct zs_defer_batch *new_batch = NULL, *batch;
+	struct zs_defer_percpu *defer;
+	bool queue = false;
+
+	if (!pool->defer)
+		return false;
+
+	defer = get_cpu_ptr(pool->defer);
+	if (!defer->active) {
+		put_cpu_ptr(pool->defer);
+		new_batch = kmalloc_obj(*new_batch, GFP_NOWAIT | __GFP_NOWARN);
+		if (!new_batch)
+			return false;
+		new_batch->count = 0;
+
+		defer = get_cpu_ptr(pool->defer);
+		if (!defer->active) {
+			defer->active = new_batch;
+			new_batch = NULL;
+		}
+	}
+
+	batch = defer->active;
+	batch->handles[batch->count++] = handle;
+	if (batch->count == ZS_DEFER_BATCH) {
+		llist_add(&batch->node, &defer->pending);
+		defer->active = NULL;
+		queue = true;
+	}
+	put_cpu_ptr(pool->defer);
+
+	if (unlikely(new_batch))
+		kfree(new_batch);
+
+	if (queue)
+		queue_work(system_dfl_wq, &defer->drain_work);
+	return true;
+}
+
+void zs_free(struct zs_pool *pool, unsigned long handle)
+{
+	if (IS_ERR_OR_NULL((void *)handle))
+		return;
+
+	if (!zs_defer_push(pool, handle))
+		__zs_free_one(pool, handle);
+}
 EXPORT_SYMBOL_GPL(zs_free);
 
 static void zs_object_copy(struct size_class *class, unsigned long dst,
@@ -1671,7 +1751,6 @@ static int putback_zspage(struct size_class *class, struct zspage *zspage)
 	return fullness;
 }
 
-#ifdef CONFIG_COMPACTION
 /*
  * To prevent zspage destroy during migration, zspage freeing should
  * hold locks of all pages in the zspage.
@@ -1714,6 +1793,8 @@ static void lock_zspage(struct zspage *zspage)
 	zspage_read_unlock(zspage);
 }
 
+#ifdef CONFIG_COMPACTION
+
 static void replace_sub_page(struct size_class *class, struct zspage *zspage,
 				struct zpdesc *newzpdesc, struct zpdesc *oldzpdesc)
 {
@@ -1866,6 +1947,18 @@ const struct movable_operations zsmalloc_mops = {
 	.putback_page = zs_page_putback,
 };
 
+static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage)
+{
+	struct zpdesc *zpdesc = get_first_zpdesc(zspage);
+
+	do {
+		WARN_ON(!zpdesc_trylock(zpdesc));
+		__zpdesc_set_movable(zpdesc);
+		zpdesc_unlock(zpdesc);
+	} while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL);
+}
+#endif /* CONFIG_COMPACTION */
+
 /*
  * Caller should hold page_lock of all pages in the zspage
  * In here, we cannot use zspage meta data.
@@ -1904,7 +1997,7 @@ static void async_free_zspage(struct work_struct *work)
 
 static void kick_deferred_free(struct zs_pool *pool)
 {
-	schedule_work(&pool->free_work);
+	queue_work(system_dfl_wq, &pool->free_work);
 }
 
 static void zs_flush_migration(struct zs_pool *pool)
@@ -1917,20 +2010,6 @@ static void init_deferred_free(struct zs_pool *pool)
 	INIT_WORK(&pool->free_work, async_free_zspage);
 }
 
-static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage)
-{
-	struct zpdesc *zpdesc = get_first_zpdesc(zspage);
-
-	do {
-		WARN_ON(!zpdesc_trylock(zpdesc));
-		__zpdesc_set_movable(zpdesc);
-		zpdesc_unlock(zpdesc);
-	} while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL);
-}
-#else
-static inline void zs_flush_migration(struct zs_pool *pool) { }
-#endif
-
 /*
  *
  * Based on the number of unused allocated objects calculate
@@ -2127,6 +2206,48 @@ static int calculate_zspage_chain_size(int class_size)
 	return chain_size;
 }
 
+static void zs_defer_destroy(struct zs_pool *pool)
+{
+	struct zs_defer_percpu __percpu *defer_pcp = pool->defer;
+	int cpu;
+
+	if (!defer_pcp)
+		return;
+
+	pool->defer = NULL;
+	for_each_possible_cpu(cpu) {
+		struct zs_defer_percpu *defer = per_cpu_ptr(defer_pcp, cpu);
+
+		cancel_work_sync(&defer->drain_work);
+		if (defer->active) {
+			llist_add(&defer->active->node, &defer->pending);
+			defer->active = NULL;
+		}
+		zs_defer_drain(defer);
+	}
+	free_percpu(defer_pcp);
+}
+
+static int zs_defer_init(struct zs_pool *pool)
+{
+	int cpu;
+
+	pool->defer = alloc_percpu(struct zs_defer_percpu);
+	if (!pool->defer)
+		return -ENOMEM;
+
+	for_each_possible_cpu(cpu) {
+		struct zs_defer_percpu *defer = per_cpu_ptr(pool->defer, cpu);
+
+		defer->active = NULL;
+		init_llist_head(&defer->pending);
+		defer->pool = pool;
+		INIT_WORK(&defer->drain_work, zs_defer_drain_work);
+	}
+
+	return 0;
+}
+
 /**
  * zs_create_pool - Creates an allocation pool to work from.
  * @name: pool name to be created
@@ -2151,6 +2272,10 @@ struct zs_pool *zs_create_pool(const char *name)
 	rwlock_init(&pool->lock);
 	atomic_set(&pool->compaction_in_progress, 0);
 
+	if (zs_defer_init(pool))
+		pr_info("deferred free unavailable for <%s>, freeing synchronously\n",
+			name);
+
 	pool->name = kstrdup(name, GFP_KERNEL);
 	if (!pool->name)
 		goto err;
@@ -2253,6 +2378,7 @@ void zs_destroy_pool(struct zs_pool *pool)
 	int i;
 
 	zs_unregister_shrinker(pool);
+	zs_defer_destroy(pool);
 	zs_flush_migration(pool);
 	zs_pool_stat_destroy(pool);
 
-- 
2.54.0

Re: [RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches

[Nhat Pham]

On Fri, Oct 16, 2026 at 5:10 AM Hongfeng Chen
<chenhongfeng0212@gmail.com> wrote:
>
> Performance results:
>
> ROCK 5B+ (RK3588: 4x Cortex-A55 at 1.8GHz, 4x Cortex-A76 at 2.4GHz,
> performance governor), single process munmap on A55:
>
>   times in ms, speedups over base
>               Base       Patch1       Patch1+2
>   size        time    time speedup   time speedup
>   64MB       13.81    3.62   3.81x   3.28   4.21x
>   128MB      28.50    7.18   3.97x   6.49   4.39x
>   192MB      42.19   10.71   3.94x   9.69   4.36x
>   256MB      56.06   14.26   3.93x   12.89  4.35x
>   512MB     109.50   28.43   3.85x   25.66  4.27x
>   1024MB    217.10   56.70   3.83x   51.15  4.24x
>
> Same machine, single process munmap on A76:
>
>   times in ms, speedups over base
>               Base       Patch1        Patch1+2
>   size        time    time speedup    time speedup
>   64MB        5.59    1.58   3.54x    1.49   3.76x
>   128MB      11.86    3.16   3.75x    2.95   4.02x
>   192MB      17.18    4.72   3.64x    4.45   3.86x
>   256MB      22.89    6.29   3.64x    5.87   3.90x
>   512MB      45.15   12.52   3.61x   11.64   3.88x
>   1024MB     89.16   24.75   3.60x   23.12   3.86x
>
> x86 (Intel Core i7-14700HX, P-cores locked at 2.1GHz with turbo off),
> single process munmap:
>
>   times in ms, speedups over base
>               Base       Patch1         Patch1+2
>   size        time    time speedup    time speedup
>   64MB        2.87    1.50   1.92x    1.14   2.53x
>   128MB       6.00    2.93   2.05x    2.21   2.71x
>   192MB       9.33    4.35   2.15x    3.33   2.80x
>   256MB      12.51    5.78   2.17x    4.38   2.85x
>   512MB      24.96   11.39   2.19x    8.62   2.90x
>   1024MB     50.19   22.78   2.20x   17.05   2.94x
>
> 1-4 process munmap of 256MB each:
>
>   times in ms, speedups over base
>               Base       Patch1        Patch1+2
>   mode        time    time speedup   time speedup
>   1p         13.18    5.78   2.28x   4.35   3.03x
>   2p         25.83   14.41   1.79x   4.57   5.65x
>   3p         35.70   22.14   1.61x   4.68   7.63x
>   4p         44.25   28.71   1.54x   4.75   9.31x

The numbers look nice, but I think it's a given that if we measure the
process whose work is offloaded, we'll see a speedup.

But would this come at a cost?

1. Would memory freeing be slowed down? Could you compare the host
memory usage over this period of time etc.? Hopefully the parallelism
would cancel out the extra overhead of setting up the work offloading.

2. Would this introduce isolation issues, especially when this work is
not properly accounted? Say you have a busy-ish host, with multiple
tenants. One of them is killed. With this new mechanism, would this
affect the performance of other tenants? (for e.g if we need more
cycles for the async overhead, or if memory is released slower overall
as stated above, etc.)

Some more benchmarking to show this would be nice!

Another thing I wonder is, how much of this slowed down is because of
zsmalloc's own inefficiency, and how much is because we just have a
larger logical footprint because compressing memory allows us to
allocate more. If it's the former, we should dedicate more time
towards improving zsmalloc. If it's the latter (or if fixing zsmalloc
is just not possible) then yeah maybe parallelism/work offloading is
the answer. An interesting setup would be, turning of zsmalloc, but
allocate just as much memory (logically-speaking), then kill the
process and sees if you observe the same slow down.

[RFC PATCH v4 2/2] zram: keep the hot statistics per cpu

[Hongfeng Chen]

From: Hongfeng Chen <hongfengchen@xiaomi.com>

pages_stored, compr_data_size, same_pages, huge_pages and notify_free are
updated on every page that zram stores or releases. When several processes
munmap() swapped out memory at the same time, all cpus update these
counters, and the cacheline that holds them bounces between cpus.

Move the five counters into a per cpu structure and update them through
zram_pcpu_add(), which disables preemption and adds to the value of the
current cpu. All writers run in process context, so disabling preemption is
enough. The store is WRITE_ONCE() and the remote reads are READ_ONCE(),
since a reader on another cpu can see the value at any time. A sysfs read
sums the per cpu values without a snapshot across cpus, so a sum can be
briefly negative and is clamped at zero.

slot_free() and the write, recompression and writeback paths update all of
their counters while preemption is disabled once. Disabling preemption and
finding the per cpu data has a cost, and this way it is paid once per slot
instead of once per counter.

u64_stats_sync keeps 64-bit reads from tearing on 32-bit systems and is
empty on 64-bit systems.

Signed-off-by: Hongfeng Chen <hongfengchen@xiaomi.com>
---
 drivers/block/zram/zram_drv.c | 171 ++++++++++++++++++++++++++++------
 drivers/block/zram/zram_drv.h |  16 +++-
 2 files changed, 155 insertions(+), 32 deletions(-)

diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 024402438bd1..affe758a9543 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -488,6 +488,56 @@ static ssize_t idle_store(struct device *dev, struct device_attribute *attr,
 	return len;
 }
 
+struct zram_stats_sum {
+	s64 pages_stored;
+	s64 compr_data_size;
+	s64 same_pages;
+	s64 huge_pages;
+	u64 notify_free;
+};
+
+#define zram_pcpu_add(zram_dev, field, val)					\
+do {									\
+	struct zram *__zram = (zram_dev);				\
+	struct zram_pcpu_stats *__stats;				\
+									\
+	__stats = get_cpu_ptr(__zram->pcpu_stats);			\
+	u64_stats_update_begin(&__stats->syncp);			\
+	WRITE_ONCE(__stats->field, __stats->field + (val));		\
+	u64_stats_update_end(&__stats->syncp);			\
+	put_cpu_ptr(__zram->pcpu_stats);				\
+} while (0)
+
+static void zram_pcpu_sum(struct zram *zram, struct zram_stats_sum *sum)
+{
+	int cpu;
+
+	memset(sum, 0, sizeof(*sum));
+
+	for_each_possible_cpu(cpu) {
+		struct zram_pcpu_stats *stats;
+		unsigned int start;
+		s64 pages, bytes, same, huge;
+		u64 notify;
+
+		stats = per_cpu_ptr(zram->pcpu_stats, cpu);
+		do {
+			start = u64_stats_fetch_begin(&stats->syncp);
+			pages = READ_ONCE(stats->pages_stored);
+			bytes = READ_ONCE(stats->compr_data_size);
+			same = READ_ONCE(stats->same_pages);
+			huge = READ_ONCE(stats->huge_pages);
+			notify = READ_ONCE(stats->notify_free);
+		} while (u64_stats_fetch_retry(&stats->syncp, start));
+
+		sum->pages_stored += pages;
+		sum->compr_data_size += bytes;
+		sum->same_pages += same;
+		sum->huge_pages += huge;
+		sum->notify_free += notify;
+	}
+}
+
 #ifdef CONFIG_ZRAM_WRITEBACK
 #define INVALID_BDEV_BLOCK		(~0UL)
 
@@ -919,6 +969,8 @@ static void zram_account_writeback_submit(struct zram *zram)
 static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req)
 {
 	unsigned long index = req->pps->index;
+	struct zram_pcpu_stats *stats;
+	s64 huge, size;
 	int err;
 
 	err = blk_status_to_errno(req->bio.bi_status);
@@ -947,9 +999,14 @@ static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req)
 	}
 
 	clear_slot_flag(zram, index, ZRAM_IDLE);
-	if (test_slot_flag(zram, index, ZRAM_HUGE))
-		atomic64_dec(&zram->stats.huge_pages);
-	atomic64_sub(get_slot_size(zram, index), &zram->stats.compr_data_size);
+	huge = test_slot_flag(zram, index, ZRAM_HUGE) ? -1 : 0;
+	size = get_slot_size(zram, index);
+	stats = get_cpu_ptr(zram->pcpu_stats);
+	u64_stats_update_begin(&stats->syncp);
+	WRITE_ONCE(stats->huge_pages, stats->huge_pages + huge);
+	WRITE_ONCE(stats->compr_data_size, stats->compr_data_size - size);
+	u64_stats_update_end(&stats->syncp);
+	put_cpu_ptr(zram->pcpu_stats);
 	zs_free(zram->mem_pool, get_slot_handle(zram, index));
 	set_slot_handle(zram, index, req->blk_idx);
 	set_slot_flag(zram, index, ZRAM_WB);
@@ -1919,14 +1976,16 @@ static ssize_t io_stat_show(struct device *dev, struct device_attribute *attr,
 			    char *buf)
 {
 	struct zram *zram = dev_to_zram(dev);
+	struct zram_stats_sum sum;
 	ssize_t ret;
 
 	guard(rwsem_read)(&zram->dev_lock);
+	zram_pcpu_sum(zram, &sum);
 	ret = sysfs_emit(buf,
 			"%8llu %8llu 0 %8llu\n",
 			(u64)atomic64_read(&zram->stats.failed_reads),
 			(u64)atomic64_read(&zram->stats.failed_writes),
-			(u64)atomic64_read(&zram->stats.notify_free));
+			sum.notify_free);
 
 	return ret;
 }
@@ -1936,6 +1995,7 @@ static ssize_t mm_stat_show(struct device *dev, struct device_attribute *attr,
 {
 	struct zram *zram = dev_to_zram(dev);
 	struct zs_pool_stats pool_stats;
+	struct zram_stats_sum sum;
 	u64 orig_size, mem_used = 0;
 	long max_used;
 	ssize_t ret;
@@ -1948,19 +2008,20 @@ static ssize_t mm_stat_show(struct device *dev, struct device_attribute *attr,
 		zs_pool_stats(zram->mem_pool, &pool_stats);
 	}
 
-	orig_size = atomic64_read(&zram->stats.pages_stored);
+	zram_pcpu_sum(zram, &sum);
+	orig_size = max_t(s64, sum.pages_stored, 0);
 	max_used = atomic_long_read(&zram->stats.max_used_pages);
 
 	ret = sysfs_emit(buf,
 			"%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu %8llu\n",
 			orig_size << PAGE_SHIFT,
-			(u64)atomic64_read(&zram->stats.compr_data_size),
+			(u64)max_t(s64, sum.compr_data_size, 0),
 			mem_used << PAGE_SHIFT,
 			zram->limit_pages << PAGE_SHIFT,
 			max_used << PAGE_SHIFT,
-			(u64)atomic64_read(&zram->stats.same_pages),
+			(u64)max_t(s64, sum.same_pages, 0),
 			atomic_long_read(&pool_stats.pages_compacted),
-			(u64)atomic64_read(&zram->stats.huge_pages),
+			(u64)max_t(s64, sum.huge_pages, 0),
 			(u64)atomic64_read(&zram->stats.huge_pages_since));
 
 	return ret;
@@ -2027,6 +2088,8 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize)
 
 static void slot_free(struct zram *zram, unsigned long index)
 {
+	s64 pages = 0, bytes = 0, same = 0, huge = 0;
+	struct zram_pcpu_stats *stats;
 	unsigned long handle;
 
 #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME
@@ -2044,7 +2107,7 @@ static void slot_free(struct zram *zram, unsigned long index)
 		 * ZRAM_HUGE flag for deferred decompression path.
 		 */
 		if (!test_slot_flag(zram, index, ZRAM_WB))
-			atomic64_dec(&zram->stats.huge_pages);
+			huge = -1;
 		clear_slot_flag(zram, index, ZRAM_HUGE);
 	}
 
@@ -2060,22 +2123,32 @@ static void slot_free(struct zram *zram, unsigned long index)
 	 */
 	if (test_slot_flag(zram, index, ZRAM_SAME)) {
 		clear_slot_flag(zram, index, ZRAM_SAME);
-		atomic64_dec(&zram->stats.same_pages);
+		same = -1;
 		goto out;
 	}
 
 	handle = get_slot_handle(zram, index);
 	if (!handle)
-		return;
+		goto stats;
 
 	zs_free(zram->mem_pool, handle);
 
-	atomic64_sub(get_slot_size(zram, index),
-		     &zram->stats.compr_data_size);
+	bytes = -(s64)get_slot_size(zram, index);
 out:
-	atomic64_dec(&zram->stats.pages_stored);
+	pages = -1;
 	set_slot_handle(zram, index, 0);
 	set_slot_size(zram, index, 0);
+stats:
+	if (!(pages | bytes | same | huge))
+		return;
+	stats = get_cpu_ptr(zram->pcpu_stats);
+	u64_stats_update_begin(&stats->syncp);
+	WRITE_ONCE(stats->pages_stored, stats->pages_stored + pages);
+	WRITE_ONCE(stats->compr_data_size, stats->compr_data_size + bytes);
+	WRITE_ONCE(stats->same_pages, stats->same_pages + same);
+	WRITE_ONCE(stats->huge_pages, stats->huge_pages + huge);
+	u64_stats_update_end(&stats->syncp);
+	put_cpu_ptr(zram->pcpu_stats);
 }
 
 static int read_same_filled_page(struct zram *zram, struct page *page,
@@ -2237,14 +2310,20 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
 static int write_same_filled_page(struct zram *zram, unsigned long fill,
 				  unsigned long index)
 {
+	struct zram_pcpu_stats *stats;
+
 	slot_lock(zram, index);
 	slot_free(zram, index);
 	set_slot_flag(zram, index, ZRAM_SAME);
 	set_slot_handle(zram, index, fill);
 	slot_unlock(zram, index);
 
-	atomic64_inc(&zram->stats.same_pages);
-	atomic64_inc(&zram->stats.pages_stored);
+	stats = get_cpu_ptr(zram->pcpu_stats);
+	u64_stats_update_begin(&stats->syncp);
+	WRITE_ONCE(stats->same_pages, stats->same_pages + 1);
+	WRITE_ONCE(stats->pages_stored, stats->pages_stored + 1);
+	u64_stats_update_end(&stats->syncp);
+	put_cpu_ptr(zram->pcpu_stats);
 
 	return 0;
 }
@@ -2252,6 +2331,7 @@ static int write_same_filled_page(struct zram *zram, unsigned long fill,
 static int write_incompressible_page(struct zram *zram, struct page *page,
 				     unsigned long index)
 {
+	struct zram_pcpu_stats *stats;
 	unsigned long handle;
 	void *src;
 
@@ -2282,10 +2362,14 @@ static int write_incompressible_page(struct zram *zram, struct page *page,
 	set_slot_size(zram, index, PAGE_SIZE);
 	slot_unlock(zram, index);
 
-	atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size);
-	atomic64_inc(&zram->stats.huge_pages);
+	stats = get_cpu_ptr(zram->pcpu_stats);
+	u64_stats_update_begin(&stats->syncp);
+	WRITE_ONCE(stats->compr_data_size, stats->compr_data_size + PAGE_SIZE);
+	WRITE_ONCE(stats->huge_pages, stats->huge_pages + 1);
+	WRITE_ONCE(stats->pages_stored, stats->pages_stored + 1);
+	u64_stats_update_end(&stats->syncp);
+	put_cpu_ptr(zram->pcpu_stats);
 	atomic64_inc(&zram->stats.huge_pages_since);
-	atomic64_inc(&zram->stats.pages_stored);
 
 	return 0;
 }
@@ -2293,6 +2377,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page,
 static int zram_write_page(struct zram *zram, struct page *page,
 			   unsigned long index)
 {
+	struct zram_pcpu_stats *stats;
 	int ret = 0;
 	unsigned long handle;
 	unsigned int comp_len;
@@ -2348,8 +2433,12 @@ static int zram_write_page(struct zram *zram, struct page *page,
 	slot_unlock(zram, index);
 
 	/* Update stats */
-	atomic64_inc(&zram->stats.pages_stored);
-	atomic64_add(comp_len, &zram->stats.compr_data_size);
+	stats = get_cpu_ptr(zram->pcpu_stats);
+	u64_stats_update_begin(&stats->syncp);
+	WRITE_ONCE(stats->pages_stored, stats->pages_stored + 1);
+	WRITE_ONCE(stats->compr_data_size, stats->compr_data_size + comp_len);
+	u64_stats_update_end(&stats->syncp);
+	put_cpu_ptr(zram->pcpu_stats);
 
 	return ret;
 }
@@ -2449,6 +2538,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 			   u32 threshold, u32 prio)
 {
 	struct zcomp_strm *zstrm = NULL;
+	struct zram_pcpu_stats *stats;
 	unsigned long handle_old;
 	unsigned long handle_new;
 	unsigned int comp_len_old;
@@ -2545,8 +2635,12 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 	set_slot_size(zram, index, comp_len_new);
 	set_slot_comp_priority(zram, index, prio);
 
-	atomic64_add(comp_len_new, &zram->stats.compr_data_size);
-	atomic64_inc(&zram->stats.pages_stored);
+	stats = get_cpu_ptr(zram->pcpu_stats);
+	u64_stats_update_begin(&stats->syncp);
+	WRITE_ONCE(stats->compr_data_size, stats->compr_data_size + comp_len_new);
+	WRITE_ONCE(stats->pages_stored, stats->pages_stored + 1);
+	u64_stats_update_end(&stats->syncp);
+	put_cpu_ptr(zram->pcpu_stats);
 
 	return 0;
 }
@@ -2724,7 +2818,7 @@ static void zram_bio_discard(struct zram *zram, struct bio *bio)
 		slot_lock(zram, index);
 		slot_free(zram, index);
 		slot_unlock(zram, index);
-		atomic64_inc(&zram->stats.notify_free);
+		zram_pcpu_add(zram, notify_free, 1);
 		index++;
 		n -= PAGE_SIZE;
 	}
@@ -2825,7 +2919,7 @@ static void zram_slot_free_notify(struct block_device *bdev,
 
 	zram = bdev->bd_disk->private_data;
 
-	atomic64_inc(&zram->stats.notify_free);
+	zram_pcpu_add(zram, notify_free, 1);
 	if (!slot_trylock(zram, index)) {
 		atomic64_inc(&zram->stats.miss_free);
 		return;
@@ -2866,6 +2960,8 @@ static void zram_destroy_comps(struct zram *zram)
 
 static void zram_reset_device(struct zram *zram)
 {
+	int cpu;
+
 	guard(rwsem_write)(&zram->dev_lock);
 
 	zram->limit_pages = 0;
@@ -2877,6 +2973,16 @@ static void zram_reset_device(struct zram *zram)
 	zram_meta_free(zram, zram->disksize);
 	zram->disksize = 0;
 	zram_destroy_comps(zram);
+	for_each_possible_cpu(cpu) {
+		struct zram_pcpu_stats *stats;
+
+		stats = per_cpu_ptr(zram->pcpu_stats, cpu);
+		stats->pages_stored = 0;
+		stats->compr_data_size = 0;
+		stats->same_pages = 0;
+		stats->huge_pages = 0;
+		stats->notify_free = 0;
+	}
 	memset(&zram->stats, 0, sizeof(zram->stats));
 	reset_bdev(zram);
 }
@@ -3083,15 +3189,23 @@ static int zram_add(void)
 						  BLK_FEAT_SYNCHRONOUS,
 	};
 	struct zram *zram;
-	int ret, device_id;
+	int cpu, ret, device_id;
 
 	zram = kzalloc_obj(struct zram);
 	if (!zram)
 		return -ENOMEM;
 
+	zram->pcpu_stats = alloc_percpu(struct zram_pcpu_stats);
+	if (!zram->pcpu_stats) {
+		ret = -ENOMEM;
+		goto out_free_dev;
+	}
+	for_each_possible_cpu(cpu)
+		u64_stats_init(&per_cpu_ptr(zram->pcpu_stats, cpu)->syncp);
+
 	ret = idr_alloc(&zram_index_idr, zram, 0, 0, GFP_KERNEL);
 	if (ret < 0)
-		goto out_free_dev;
+		goto out_free_pcpu;
 	device_id = ret;
 
 	init_rwsem(&zram->dev_lock);
@@ -3133,6 +3247,8 @@ static int zram_add(void)
 	put_disk(zram->disk);
 out_free_idr:
 	idr_remove(&zram_index_idr, device_id);
+out_free_pcpu:
+	free_percpu(zram->pcpu_stats);
 out_free_dev:
 	kfree(zram);
 	return ret;
@@ -3182,6 +3298,7 @@ static int zram_remove(struct zram *zram)
 	zram_reset_device(zram);
 
 	put_disk(zram->disk);
+	free_percpu(zram->pcpu_stats);
 	kfree(zram);
 	return 0;
 }
diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h
index 7a55d751417e..482e1c28ca1c 100644
--- a/drivers/block/zram/zram_drv.h
+++ b/drivers/block/zram/zram_drv.h
@@ -17,6 +17,7 @@
 
 #include <asm/byteorder.h>
 #include <linux/rwsem.h>
+#include <linux/u64_stats_sync.h>
 #include <linux/zsmalloc.h>
 
 #include "zcomp.h"
@@ -90,15 +91,19 @@ struct zram_table_entry {
 	};
 };
 
+struct zram_pcpu_stats {
+	struct u64_stats_sync syncp;
+	s64 pages_stored;	/* no. of pages currently stored */
+	s64 compr_data_size;	/* compressed size of pages stored */
+	s64 same_pages;		/* no. of same element filled pages */
+	s64 huge_pages;		/* no. of huge pages */
+	u64 notify_free;	/* no. of swap slot free notifications */
+};
+
 struct zram_stats {
-	atomic64_t compr_data_size;	/* compressed size of pages stored */
 	atomic64_t failed_reads;	/* can happen when memory is too low */
 	atomic64_t failed_writes;	/* can happen when memory is too low */
-	atomic64_t notify_free;	/* no. of swap slot free notifications */
-	atomic64_t same_pages;		/* no. of same element filled pages */
-	atomic64_t huge_pages;		/* no. of huge pages */
 	atomic64_t huge_pages_since;	/* no. of huge pages since zram set up */
-	atomic64_t pages_stored;	/* no. of pages currently stored */
 	atomic_long_t max_used_pages;	/* no. of maximum pages stored */
 	atomic64_t miss_free;		/* no. of missed free */
 #ifdef	CONFIG_ZRAM_WRITEBACK
@@ -134,6 +139,7 @@ struct zram {
 	unsigned long limit_pages;
 
 	struct zram_stats stats;
+	struct zram_pcpu_stats __percpu *pcpu_stats;
 	/*
 	 * This is the limit on amount of *uncompressed* worth of data
 	 * we can store in a disk.
-- 
2.54.0

Re: [RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches

[Sergey Senozhatsky]

On (26/10/16 09:09), Nhat Pham wrote:
> On Fri, Oct 16, 2026 at 5:10 AM Hongfeng Chen
> <chenhongfeng0212@gmail.com> wrote:
> >
> > Performance results:
> >
> > ROCK 5B+ (RK3588: 4x Cortex-A55 at 1.8GHz, 4x Cortex-A76 at 2.4GHz,
> > performance governor), single process munmap on A55:
> >
> >   times in ms, speedups over base
> >               Base       Patch1       Patch1+2
> >   size        time    time speedup   time speedup
> >   64MB       13.81    3.62   3.81x   3.28   4.21x
> >   128MB      28.50    7.18   3.97x   6.49   4.39x
> >   192MB      42.19   10.71   3.94x   9.69   4.36x
> >   256MB      56.06   14.26   3.93x   12.89  4.35x
> >   512MB     109.50   28.43   3.85x   25.66  4.27x
> >   1024MB    217.10   56.70   3.83x   51.15  4.24x
> >
> > Same machine, single process munmap on A76:
> >
> >   times in ms, speedups over base
> >               Base       Patch1        Patch1+2
> >   size        time    time speedup    time speedup
> >   64MB        5.59    1.58   3.54x    1.49   3.76x
> >   128MB      11.86    3.16   3.75x    2.95   4.02x
> >   192MB      17.18    4.72   3.64x    4.45   3.86x
> >   256MB      22.89    6.29   3.64x    5.87   3.90x
> >   512MB      45.15   12.52   3.61x   11.64   3.88x
> >   1024MB     89.16   24.75   3.60x   23.12   3.86x
> >
> > x86 (Intel Core i7-14700HX, P-cores locked at 2.1GHz with turbo off),
> > single process munmap:
> >
> >   times in ms, speedups over base
> >               Base       Patch1         Patch1+2
> >   size        time    time speedup    time speedup
> >   64MB        2.87    1.50   1.92x    1.14   2.53x
> >   128MB       6.00    2.93   2.05x    2.21   2.71x
> >   192MB       9.33    4.35   2.15x    3.33   2.80x
> >   256MB      12.51    5.78   2.17x    4.38   2.85x
> >   512MB      24.96   11.39   2.19x    8.62   2.90x
> >   1024MB     50.19   22.78   2.20x   17.05   2.94x
> >
> > 1-4 process munmap of 256MB each:
> >
> >   times in ms, speedups over base
> >               Base       Patch1        Patch1+2
> >   mode        time    time speedup   time speedup
> >   1p         13.18    5.78   2.28x   4.35   3.03x
> >   2p         25.83   14.41   1.79x   4.57   5.65x
> >   3p         35.70   22.14   1.61x   4.68   7.63x
> >   4p         44.25   28.71   1.54x   4.75   9.31x
> 
> The numbers look nice, but I think it's a given that if we measure the
> process whose work is offloaded, we'll see a speedup.
> 
> But would this come at a cost?
> 
> 1. Would memory freeing be slowed down? Could you compare the host
> memory usage over this period of time etc.? Hopefully the parallelism
> would cancel out the extra overhead of setting up the work offloading.

Right, especially interesting what happens if OOM-kill of an app with
a considerable amount of data stored in zsmalloc pool, is not immediately
easing the memory pressure.

[..]
> If it's the former, we should dedicate more time
					^^^^
					and tokens ;)
> towards improving zsmalloc.