[RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches

[Hongfeng Chen <chenhongfeng0212@gmail.com>]

Freeing swap entries is expensive when unmapping a VMA that holds many of
them. `zap_pte_range()` eventually calls `zram_slot_free_notify()`, which
frees one compressed object per page. As a result, the unmapping task
has to pay for every `zs_free()` itself. On Android, this slows down the
exit of a task killed to reclaim memory, precisely when that memory is
needed.

Two earlier attempts by Lei and Zhiguo introduced a new thread in the MM
core to asynchronously collect and free swap entries [1][2], which
seems overcomplicated. Wenchao's RFC v3 [3] moved the deferral into
zsmalloc, but introduced push/drain callbacks that zram and zswap had
to implement, potentially violating software layer boundaries. This v4
RFC defers freeing inside `zs_free()` itself, leaving callers unchanged.

Patch 1 makes `zs_free()` append the handle to a per-CPU batch of 64
handles instead of freeing it immediately. Once the batch is full, it
is handed to an unbound drain worker through a lockless list, and the
worker frees the objects. Releasing empty zspages is also deferred to
the worker. If the per-CPU state cannot be allocated, `zs_free()` falls
back to its original path.

With the zsmalloc work moved off the unmapping path by patch 1,
contention on atomic operations in zram statistics becomes significant.
Patch 2 moves the contended counters to per-CPU storage.


Using the following microbenchmark:

// SPDX-License-Identifier: GPL-2.0
/* Times munmap() on a region pushed out to zram: MB. */
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <time.h>
#include <unistd.h>

#ifndef MADV_PAGEOUT
#define MADV_PAGEOUT 21
#endif

static void die(const char *what)
{
	perror(what);
	exit(1);
}

static double now_ms(void)
{
	struct timespec ts;

	clock_gettime(CLOCK_MONOTONIC, &ts);
	return ts.tv_sec * 1e3 + ts.tv_nsec / 1e6;
}

/* About 2.8:1 under lzo-rle, objects in 16 size classes. */
static void fill_data(char *p, size_t bytes)
{
	size_t i, k, n;

	memset(p, 0, bytes);
	for (i = 0; i < bytes; i += 4096) {
		long *q = (long *)(p + i);

		n = 106 + i / 4096 % 16 * 8;
		for (k = 0; k < n; k++)
			q[k] = k * 8 * 2654435761UL;
	}
}

static char *prepare(size_t bytes)
{
	char *p = mmap(NULL, bytes, PROT_READ | PROT_WRITE,
		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

	if (p == MAP_FAILED)
		die("mmap");
	fill_data(p, bytes);
	if (madvise(p, bytes, MADV_PAGEOUT))
		die("madvise");
	return p;
}

int main(int argc, char **argv)
{
	size_t bytes = (argc > 1 ? strtoul(argv[1], NULL, 10) : 256) << 20;
	char *p = prepare(bytes);
	double t0, t1;

	raise(SIGSTOP);	/* ready */
	t0 = now_ms();
	munmap(p, bytes);
	t1 = now_ms();
	printf("%.6f %.6f\n", t0, t1);
	return 0;
}

Performance results:

ROCK 5B+ (RK3588: 4x Cortex-A55 at 1.8GHz, 4x Cortex-A76 at 2.4GHz,
performance governor), single process munmap on A55:

  times in ms, speedups over base
              Base       Patch1       Patch1+2
  size        time    time speedup   time speedup
  64MB       13.81    3.62   3.81x   3.28   4.21x
  128MB      28.50    7.18   3.97x   6.49   4.39x
  192MB      42.19   10.71   3.94x   9.69   4.36x
  256MB      56.06   14.26   3.93x   12.89  4.35x
  512MB     109.50   28.43   3.85x   25.66  4.27x
  1024MB    217.10   56.70   3.83x   51.15  4.24x

Same machine, single process munmap on A76:

  times in ms, speedups over base
              Base       Patch1        Patch1+2
  size        time    time speedup    time speedup
  64MB        5.59    1.58   3.54x    1.49   3.76x
  128MB      11.86    3.16   3.75x    2.95   4.02x
  192MB      17.18    4.72   3.64x    4.45   3.86x
  256MB      22.89    6.29   3.64x    5.87   3.90x
  512MB      45.15   12.52   3.61x   11.64   3.88x
  1024MB     89.16   24.75   3.60x   23.12   3.86x

x86 (Intel Core i7-14700HX, P-cores locked at 2.1GHz with turbo off),
single process munmap:

  times in ms, speedups over base
              Base       Patch1         Patch1+2
  size        time    time speedup    time speedup
  64MB        2.87    1.50   1.92x    1.14   2.53x
  128MB       6.00    2.93   2.05x    2.21   2.71x
  192MB       9.33    4.35   2.15x    3.33   2.80x
  256MB      12.51    5.78   2.17x    4.38   2.85x
  512MB      24.96   11.39   2.19x    8.62   2.90x
  1024MB     50.19   22.78   2.20x   17.05   2.94x

1-4 process munmap of 256MB each:

  times in ms, speedups over base
              Base       Patch1        Patch1+2
  mode        time    time speedup   time speedup
  1p         13.18    5.78   2.28x   4.35   3.03x
  2p         25.83   14.41   1.79x   4.57   5.65x
  3p         35.70   22.14   1.61x   4.68   7.63x
  4p         44.25   28.71   1.54x   4.75   9.31x

[1] https://lore.kernel.org/all/20240805153639.1057-1-justinjiang@vivo.com/
[2] https://lore.kernel.org/all/20250909065349.574894-1-liulei.rjpt@vivo.com/
[3] https://lore.kernel.org/linux-mm/20260508060724.3810904-1-haowenchao@xiaomi.com/

Changes since v3:
- Defer inside zs_free() only and drop the push/drain callbacks, so zram
  and zswap do not change. Yosry and Nhat objected to the layering in v3;
  Nhat also pointed out that v3 leaves torn-down entries on the zswap LRU.
- Buffer handles in per-cpu batches of 64 allocated on demand, instead of
  per-cpu pages rotated through a preallocated pool.
- New patch 2: keep the hot zram statistics per cpu.
- Link to v3: https://lore.kernel.org/linux-mm/20260508060724.3810904-1-haowenchao@xiaomi.com/

Changes since v2:
- Use per-cpu single-page buffers instead of a global list; the hot path
  only writes into the local CPU's buffer with preemption disabled
- Add a page pool for buffer rotation: when the current buffer is full,
  swap it with a free page from the pool and queue the full page for drain
- Introduce push/drain callback ops so that zram and zswap can each define
  their own element size and drain logic (zram stores u32 slot indices,
  zswap stores unsigned long handles)
- Drop the lock optimization patches it will be submitted separately as
  part of a dedicated zsmalloc lock contention series
- Link to v2: https://lore.kernel.org/all/20260421121616.3298845-1-haowenchao@xiaomi.com/

Hongfeng Chen (2):
  mm/zsmalloc: defer object free to per-cpu batches and a drain worker
  zram: keep the hot statistics per cpu

 drivers/block/zram/zram_drv.c | 171 +++++++++++++++++++++++++++-----
 drivers/block/zram/zram_drv.h |  16 ++-
 mm/zsmalloc.c                 | 182 ++++++++++++++++++++++++++++------
 3 files changed, 309 insertions(+), 60 deletions(-)


base-commit: f4e9810097537d862c128935c36216bcf0c31cb2
-- 
2.54.0