[PATCH v4 0/4] mm/zsmalloc: reduce lock contention in zs_free()

[PATCH v4 0/4] mm/zsmalloc: reduce lock contention in zs_free()

[Wenchao Hao]

From: Wenchao Hao <haowenchao@xiaomi.com>

This series reduces lock contention in zs_free(), which dominates the
unmap path under memory pressure on Android (LMK kills) and on x86
servers running zswap-heavy workloads.

The current zs_free() takes pool->lock (rwlock, read side) just to
look up the size_class for a handle, then takes class->lock and holds
it across __free_zspage() which can call into the buddy allocator and
acquire zone->lock.  Two costs follow:

  * pool->lock reader-counter cacheline bouncing among concurrent
    zs_free() callers.
  * class->lock held across folio_put(), so any zone->lock wait
    fans out to every other zs_free() on the same class.

The series tackles both:

  Patch 1: encode size_class index into obj alongside PFN and obj_idx,
           so zs_free() can locate the class without pool->lock.
  Patch 2: drop pool->lock from zs_free() on 64-bit; 32-bit unchanged.
  Patch 3: move zspage page-freeing out of class->lock.
  Patch 4: document the three free_zspage helper variants that result
           from the split in patch 3.

Performance results:

Test: each process independently mmap 256MB, write data, madvise
MADV_PAGEOUT to swap out via zram (lzo-rle), then concurrent munmap.

Raspberry Pi 4B (4-core ARM64 Cortex-A72):

  mode        Base       Patched     Speedup
  single      59.0ms     56.0ms      1.05x
  multi 2p    94.6ms     66.7ms      1.42x
  multi 4p    202.9ms    110.6ms     1.83x

x86 (20-core Intel i7-12700, 16 concurrent processes):

  mode        Base       Patched     Speedup
  single      11.7ms     9.8ms       1.19x
  multi 2p    24.1ms     17.2ms      1.40x
  multi 4p    63.0ms     45.3ms      1.39x

[1] https://lore.kernel.org/linux-mm/20260508061910.3882831-1-haowenchao@xiaomi.com/
[2] https://lore.kernel.org/linux-mm/20260527115930.3138213-1-haowenchao@xiaomi.com/
[3] https://lore.kernel.org/linux-mm/20260605084242.1549811-1-haowenchao22@gmail.com/
[4] https://lore.kernel.org/linux-mm/20260605084242.1549811-1-haowenchao22@gmail.com/

Changes since v3:
- 1/4: gate ZS_OBJ_CLASS_BITS on a new spare-bit check so unsuitable
  configs (e.g. UML, no sparsemem) fall back to plain obj layout
  instead of breaking the build.  (sashiko AI review)
- 2/4: annotate handle_to_obj()/record_obj() with READ_ONCE/WRITE_ONCE
  to silence KCSAN; sync the #if predicate with patch 1.  (sashiko AI review)
- 4/4: clarify when async_free_zspage() runs; pick up Reviewed-by.  (Nhat Pham)
- Drop Reviewed-by from Nhat Pham on 1/4 and 2/4: substantial logic
  changes since v3, please re-review.

Changes since v2:
- 3/4: drop likely() hint and tidy up else-branch braces, per review
  nits from Nhat Pham and Joshua Hahn.
- 4/4 (new): document free_zspage helper variants in a single comment
  block, per Nhat Pham.
- Pick up Reviewed-by tags from Nhat Pham (1/4, 2/4, 3/4) and
  Joshua Hahn (3/4).
- Fix patch 1/4 and 2/4 From: to match Signed-off-by, per Barry Song.

Changes since v1:
- Rename obj-encoding macros for clarity.
- Make 32-bit / 64-bit handling transparent at call sites
  (no #ifdef in callers).
- Extract a helper to keep zs_free() unified.

Wenchao Hao (3):
  mm/zsmalloc: encode class index in obj value for lockless class lookup
  mm/zsmalloc: drop pool->lock from zs_free on 64-bit systems
  mm/zsmalloc: document free_zspage helper variants

Xueyuan Chen (1):
  mm/zsmalloc: drop class lock before freeing zspage

 mm/zsmalloc.c | 219 ++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 184 insertions(+), 35 deletions(-)

-- 
2.34.1

[PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Wenchao Hao]

From: Wenchao Hao <haowenchao@xiaomi.com>

Encode the size_class index (class_idx) into the obj value so that
zs_free() can determine the correct size_class without dereferencing
the handle->obj->PFN->zpdesc->zspage->class chain under pool->lock.
class_idx is invariant across page migration (only PFN is rewritten),
so a lockless read of obj always yields a valid class_idx.

Where obj has more bits below the PFN field than obj_idx alone
needs, split that space into class_idx and obj_idx subfields:

 |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
 +-----------------+-------------------------+-----------------------+
 |       PFN       |        class_idx        |        obj_idx        |
 +-----------------+-------------------------+-----------------------+
MSB                ^                                                LSB
                   |
                   +-- ZS_OBJ_PFN_SHIFT

The macro layout changes as follows:

    Before            After               Meaning
    ----------------  ------------------  ----------------------------
    OBJ_INDEX_BITS    ZS_OBJ_IDX_BITS     width of obj_idx subfield
    OBJ_INDEX_MASK    ZS_OBJ_IDX_MASK     mask  of obj_idx subfield
    (n/a)             ZS_OBJ_CLASS_BITS   width of class_idx subfield
    (n/a)             ZS_OBJ_CLASS_MASK   mask  of class_idx subfield
    (n/a)             ZS_OBJ_PFN_SHIFT    bit offset of PFN in obj

ZS_OBJ_CLASS_BITS folds to 0 (and the layout collapses to
[PFN | obj_idx]) when obj has no spare bits, i.e. on 32-bit
or on 64-bit fallback paths where MAX_POSSIBLE_PHYSMEM_BITS ==
BITS_PER_LONG (e.g. UML); zs_free() then falls back to
pool->lock.

Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
---
 mm/zsmalloc.c | 99 +++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 85 insertions(+), 14 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 63128ddb7959..f84258d63917 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -67,8 +67,8 @@
 #define MAX_POSSIBLE_PHYSMEM_BITS MAX_PHYSMEM_BITS
 #else
 /*
- * If this definition of MAX_PHYSMEM_BITS is used, OBJ_INDEX_BITS will just
- * be PAGE_SHIFT
+ * If this definition of MAX_PHYSMEM_BITS is used, ZS_OBJ_PFN_SHIFT will
+ * just be PAGE_SHIFT
  */
 #define MAX_POSSIBLE_PHYSMEM_BITS BITS_PER_LONG
 #endif
@@ -88,8 +88,23 @@
 #define OBJ_TAG_BITS	1
 #define OBJ_TAG_MASK	OBJ_ALLOCATED_TAG
 
-#define OBJ_INDEX_BITS	(BITS_PER_LONG - _PFN_BITS)
-#define OBJ_INDEX_MASK	((_AC(1, UL) << OBJ_INDEX_BITS) - 1)
+/*
+ * obj is encoded as [PFN | class_idx | obj_idx] within an unsigned long:
+ *
+ *   |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
+ *   +-----------------+-------------------------+-----------------------+
+ *   |       PFN       |        class_idx        |        obj_idx        |
+ *   +-----------------+-------------------------+-----------------------+
+ *  MSB                ^                                                LSB
+ *                     |
+ *                     +-- ZS_OBJ_PFN_SHIFT
+ *
+ * Encoding class_idx into obj lets zs_free() locate the size_class
+ * without holding pool->lock; class_idx is invariant across page
+ * migration (only PFN changes), so a lockless read of the obj value
+ * always yields a valid class_idx.
+ */
+#define ZS_OBJ_PFN_SHIFT	(BITS_PER_LONG - _PFN_BITS)
 
 #define HUGE_BITS	1
 #define FULLNESS_BITS	4
@@ -98,9 +113,55 @@
 
 #define ZS_MAX_PAGES_PER_ZSPAGE	(_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL))
 
+/*
+ * ceil(log2(ZS_MAX_PAGES_PER_ZSPAGE)) at preprocessor time, for use
+ * in #if below.  Kconfig restricts ZSMALLOC_CHAIN_SIZE to [4, 16].
+ */
+#if ZS_MAX_PAGES_PER_ZSPAGE <= 4
+#define ZS_CHAIN_LOG2		2
+#elif ZS_MAX_PAGES_PER_ZSPAGE <= 8
+#define ZS_CHAIN_LOG2		3
+#elif ZS_MAX_PAGES_PER_ZSPAGE <= 16
+#define ZS_CHAIN_LOG2		4
+#else
+#error "ZSMALLOC_CHAIN_SIZE out of expected range [4,16]"
+#endif
+
+/* PAGE_SHIFT - 5 = log2(PAGE_SIZE / 32); 32 = ZS_MIN_ALLOC_SIZE floor. */
+#define ZS_MAX_OBJ_PER_PAGE_LOG2	(PAGE_SHIFT - 5)
+
+/*
+ * obj_idx width that keeps ZS_MIN_ALLOC_SIZE at its 32-byte floor.
+ * Below this, ZS_MIN_ALLOC_SIZE is auto-raised by the MAX(32, ...)
+ * formula -- still correct, but objects are coarser.
+ */
+#define ZS_OBJ_IDX_DENSE_BITS	(ZS_CHAIN_LOG2 + ZS_MAX_OBJ_PER_PAGE_LOG2)
+
+/*
+ * Encode class_idx only when obj has spare bits; otherwise
+ * ZS_OBJ_CLASS_BITS folds to 0 (32-bit, or 64-bit UML/fallback).
+ */
+#if BITS_PER_LONG >= 64 && \
+	ZS_OBJ_PFN_SHIFT >= (CLASS_BITS + 1) + ZS_OBJ_IDX_DENSE_BITS
+#define ZS_OBJ_CLASS_BITS	(CLASS_BITS + 1)
+#else
+#define ZS_OBJ_CLASS_BITS	0
+#endif
+#define ZS_OBJ_CLASS_MASK	((_AC(1, UL) << ZS_OBJ_CLASS_BITS) - 1)
+
+#define ZS_OBJ_IDX_BITS		(ZS_OBJ_PFN_SHIFT - ZS_OBJ_CLASS_BITS)
+#define ZS_OBJ_IDX_MASK		((_AC(1, UL) << ZS_OBJ_IDX_BITS) - 1)
+
+/*
+ * Belt-and-suspenders: the #if above already guarantees this when
+ * class_idx is enabled.  Catches future tweaks that bypass it.
+ */
+static_assert(ZS_OBJ_IDX_BITS >= ZS_CHAIN_LOG2,
+	      "zsmalloc: ZS_MIN_ALLOC_SIZE would exceed ZS_MAX_ALLOC_SIZE");
+
 /* ZS_MIN_ALLOC_SIZE must be multiple of ZS_ALIGN */
 #define ZS_MIN_ALLOC_SIZE \
-	MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> OBJ_INDEX_BITS))
+	MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> ZS_OBJ_IDX_BITS))
 /* each chunk includes extra space to keep handle */
 #define ZS_MAX_ALLOC_SIZE	PAGE_SIZE
 
@@ -721,26 +782,35 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
 static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc,
 				unsigned int *obj_idx)
 {
-	*zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
-	*obj_idx = (obj & OBJ_INDEX_MASK);
+	*zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT);
+	*obj_idx = (obj & ZS_OBJ_IDX_MASK);
 }
 
 static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc)
 {
-	*zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
+	*zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT);
+}
+
+/* Folds to 0 when ZS_OBJ_CLASS_BITS == 0; no ifdef needed at callers. */
+static unsigned int obj_to_class_idx(unsigned long obj)
+{
+	return (obj >> ZS_OBJ_IDX_BITS) & ZS_OBJ_CLASS_MASK;
 }
 
 /**
- * location_to_obj - get obj value encoded from (<zpdesc>, <obj_idx>)
+ * location_to_obj - encode (<zpdesc>, <obj_idx>, <class_idx>) into obj value
  * @zpdesc: zpdesc object resides in zspage
  * @obj_idx: object index
+ * @class_idx: size class index; ignored when ZS_OBJ_CLASS_BITS == 0
  */
-static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx)
+static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx,
+				     unsigned int class_idx)
 {
 	unsigned long obj;
 
-	obj = zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS;
-	obj |= obj_idx & OBJ_INDEX_MASK;
+	obj  = zpdesc_pfn(zpdesc) << ZS_OBJ_PFN_SHIFT;
+	obj |= (unsigned long)(class_idx & ZS_OBJ_CLASS_MASK) << ZS_OBJ_IDX_BITS;
+	obj |= obj_idx & ZS_OBJ_IDX_MASK;
 
 	return obj;
 }
@@ -1276,7 +1346,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	kunmap_local(vaddr);
 	mod_zspage_inuse(zspage, 1);
 
-	obj = location_to_obj(m_zpdesc, obj);
+	obj = location_to_obj(m_zpdesc, obj, zspage->class);
 	record_obj(handle, obj);
 
 	return obj;
@@ -1762,7 +1832,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);
-			new_obj = (unsigned long)location_to_obj(newzpdesc, obj_idx);
+			new_obj = location_to_obj(newzpdesc, obj_idx,
+						  obj_to_class_idx(old_obj));
 			record_obj(handle, new_obj);
 		}
 	}
-- 
2.34.1

[PATCH v4 2/4] mm/zsmalloc: drop pool->lock from zs_free on 64-bit systems

[Wenchao Hao]

From: Wenchao Hao <haowenchao@xiaomi.com>

With class_idx encoded in obj, zs_free() can locate the size_class
without holding pool->lock on 64-bit systems.  Page migration also
takes class->lock and only rewrites the PFN field of obj, so:

  1. read obj locklessly,
  2. lock the size_class derived from obj's class_idx,
  3. re-read obj under class->lock to get a stable PFN.

This eliminates the rwlock read-side cacheline bouncing between
zs_free() and migration/compaction on multi-core systems.

Annotate handle_to_obj()/record_obj() with READ_ONCE()/WRITE_ONCE() to
prevent load/store tearing on the lockless read path and silence KCSAN
data race reports.

When ZS_OBJ_CLASS_BITS == 0 (32-bit, or 64-bit with obj too narrow to
hold class_idx), zs_free() keeps pool->lock.

Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
---
 mm/zsmalloc.c | 75 ++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 60 insertions(+), 15 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index f84258d63917..fe20ab297542 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -21,6 +21,10 @@
  *	pool->lock
  *	class->lock
  *	zspage->lock
+ *
+ * When ZS_OBJ_CLASS_BITS > 0, zs_free() skips pool->lock; it picks
+ * the size_class from obj's encoded class_idx and serializes against
+ * page migration via class->lock.
  */
 
 #include <linux/module.h>
@@ -457,10 +461,13 @@ static void cache_free_zspage(struct zspage *zspage)
 	kmem_cache_free(zspage_cachep, zspage);
 }
 
-/* class->lock(which owns the handle) synchronizes races */
+/*
+ * Pairs with READ_ONCE() in handle_to_obj(): zs_free() may read the
+ * handle locklessly, so prevent store tearing here.
+ */
 static void record_obj(unsigned long handle, unsigned long obj)
 {
-	*(unsigned long *)handle = obj;
+	WRITE_ONCE(*(unsigned long *)handle, obj);
 }
 
 static inline bool __maybe_unused is_first_zpdesc(struct zpdesc *zpdesc)
@@ -817,7 +824,7 @@ static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx
 
 static unsigned long handle_to_obj(unsigned long handle)
 {
-	return *(unsigned long *)handle;
+	return READ_ONCE(*(unsigned long *)handle);
 }
 
 static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
@@ -1451,10 +1458,58 @@ static void obj_free(int class_size, unsigned long obj)
 	mod_zspage_inuse(zspage, -1);
 }
 
+/*
+ * Resolve @handle to its zspage / size_class and acquire class->lock.
+ *
+ * When class_idx is encoded in obj (ZS_OBJ_CLASS_BITS > 0), it is
+ * invariant under page migration, so the handle can be read locklessly
+ * to pick the size_class.  Once class->lock is held migration is
+ * blocked and the handle is re-read to obtain a stable PFN.
+ *
+ * Otherwise (32-bit, or 64-bit fallback paths like UML where the
+ * encoding is disabled), fall back to pool->lock for the lookup.
+ */
+#if ZS_OBJ_CLASS_BITS > 0
+static inline void obj_handle_class_lock(struct zs_pool *pool, unsigned long handle,
+					 unsigned long *objp, struct zspage **zspagep,
+					 struct size_class **classp)
+	__acquires(&(*classp)->lock)
+{
+	struct zpdesc *f_zpdesc;
+	unsigned long obj;
+
+	obj = handle_to_obj(handle);
+	*classp = pool->size_class[obj_to_class_idx(obj)];
+	spin_lock(&(*classp)->lock);
+	/* Re-read under class->lock: PFN is now stable vs migration. */
+	obj = handle_to_obj(handle);
+	obj_to_zpdesc(obj, &f_zpdesc);
+	*zspagep = get_zspage(f_zpdesc);
+	*objp = obj;
+}
+#else
+static inline void obj_handle_class_lock(struct zs_pool *pool, unsigned long handle,
+					 unsigned long *objp, struct zspage **zspagep,
+					 struct size_class **classp)
+	__acquires(&(*classp)->lock)
+{
+	struct zpdesc *f_zpdesc;
+	unsigned long obj;
+
+	read_lock(&pool->lock);
+	obj = handle_to_obj(handle);
+	obj_to_zpdesc(obj, &f_zpdesc);
+	*zspagep = get_zspage(f_zpdesc);
+	*classp = zspage_class(pool, *zspagep);
+	spin_lock(&(*classp)->lock);
+	read_unlock(&pool->lock);
+	*objp = obj;
+}
+#endif
+
 void zs_free(struct zs_pool *pool, unsigned long handle)
 {
 	struct zspage *zspage;
-	struct zpdesc *f_zpdesc;
 	unsigned long obj;
 	struct size_class *class;
 	int fullness;
@@ -1462,17 +1517,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	if (IS_ERR_OR_NULL((void *)handle))
 		return;
 
-	/*
-	 * The pool->lock protects the race with zpage's migration
-	 * so it's safe to get the page from handle.
-	 */
-	read_lock(&pool->lock);
-	obj = handle_to_obj(handle);
-	obj_to_zpdesc(obj, &f_zpdesc);
-	zspage = get_zspage(f_zpdesc);
-	class = zspage_class(pool, zspage);
-	spin_lock(&class->lock);
-	read_unlock(&pool->lock);
+	obj_handle_class_lock(pool, handle, &obj, &zspage, &class);
 
 	class_stat_sub(class, ZS_OBJS_INUSE, 1);
 	obj_free(class->size, obj);
-- 
2.34.1

[PATCH v4 3/4] mm/zsmalloc: drop class lock before freeing zspage

[Wenchao Hao]

From: Xueyuan Chen <xueyuan.chen21@gmail.com>

Currently in zs_free(), the class->lock is held until the zspage is
completely freed and the counters are updated. However, freeing pages
back to the buddy allocator requires acquiring the zone lock.

Under heavy memory pressure, zone lock contention can be severe. When
this happens, the CPU holding the class->lock will stall waiting for
the zone lock, thereby blocking all other CPUs attempting to acquire
the same class->lock.

This patch shrinks the critical section of the class->lock to reduce
lock contention. By moving the actual page freeing process outside the
class->lock, we can improve the concurrency performance of zs_free().

Testing on the RADXA O6 platform shows that with 12 CPUs concurrently
performing zs_free() operations, the execution time is reduced by 20%.

Signed-off-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
---
 mm/zsmalloc.c | 29 +++++++++++++++++++++++------
 1 file changed, 23 insertions(+), 6 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index fe20ab297542..1602ccb6d814 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -878,13 +878,10 @@ static int trylock_zspage(struct zspage *zspage)
 	return 0;
 }
 
-static void __free_zspage(struct zs_pool *pool, struct size_class *class,
-				struct zspage *zspage)
+static inline void __free_zspage_lockless(struct zs_pool *pool, struct zspage *zspage)
 {
 	struct zpdesc *zpdesc, *next;
 
-	assert_spin_locked(&class->lock);
-
 	VM_BUG_ON(get_zspage_inuse(zspage));
 	VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0);
 
@@ -900,7 +897,13 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
 	} while (zpdesc != NULL);
 
 	cache_free_zspage(zspage);
+}
 
+static void __free_zspage(struct zs_pool *pool, struct size_class *class,
+			  struct zspage *zspage)
+{
+	assert_spin_locked(&class->lock);
+	__free_zspage_lockless(pool, zspage);
 	class_stat_sub(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage);
 	atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
 }
@@ -1513,6 +1516,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	unsigned long obj;
 	struct size_class *class;
 	int fullness;
+	struct zspage *zspage_to_free = NULL;
 
 	if (IS_ERR_OR_NULL((void *)handle))
 		return;
@@ -1523,10 +1527,23 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	obj_free(class->size, obj);
 
 	fullness = fix_fullness_group(class, zspage);
-	if (fullness == ZS_INUSE_RATIO_0)
-		free_zspage(pool, class, zspage);
+	if (fullness == ZS_INUSE_RATIO_0) {
+		if (trylock_zspage(zspage)) {
+			remove_zspage(class, zspage);
+			class_stat_sub(class, ZS_OBJS_ALLOCATED,
+				       class->objs_per_zspage);
+			zspage_to_free = zspage;
+		} else {
+			kick_deferred_free(pool);
+		}
+	}
 
 	spin_unlock(&class->lock);
+
+	if (zspage_to_free) {
+		__free_zspage_lockless(pool, zspage_to_free);
+		atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
+	}
 	cache_free_handle(handle);
 }
 EXPORT_SYMBOL_GPL(zs_free);
-- 
2.34.1

[PATCH v4 4/4] mm/zsmalloc: document free_zspage helper variants

[Wenchao Hao]

From: Wenchao Hao <haowenchao@xiaomi.com>

After splitting __free_zspage() into a lockless core and a wrapper that
does the class-stat bookkeeping, three similarly-named helpers coexist:
free_zspage / __free_zspage / __free_zspage_lockless.

Add a comment block above them describing what each does and where it
is used, so the names are not easy to confuse.

No functional change.

Suggested-by: Nhat Pham <nphamcs@gmail.com>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
---
 mm/zsmalloc.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 1602ccb6d814..c0edce381d20 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -878,6 +878,22 @@ static int trylock_zspage(struct zspage *zspage)
 	return 0;
 }
 
+/*
+ * Three free helpers, kept apart here:
+ *
+ * __free_zspage_lockless(): bare core; walks zpdescs and returns pages
+ *   to the buddy allocator.  Caller owns all zpdesc locks and has
+ *   removed the zspage from its class list.  Used by zs_free() outside
+ *   class->lock so the buddy-side work does not stall the class.
+ *
+ * __free_zspage(): __free_zspage_lockless() + per-class accounting,
+ *   under class->lock.  Used by async_free_zspage(), the worker for
+ *   zspages whose trylock_zspage() failed.
+ *
+ * free_zspage(): full wrapper - trylock zpdescs, remove from class
+ *   list, call __free_zspage(); kicks deferred free on contention.
+ *   Used by compaction.
+ */
 static inline void __free_zspage_lockless(struct zs_pool *pool, struct zspage *zspage)
 {
 	struct zpdesc *zpdesc, *next;
-- 
2.34.1

Re: [PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Nhat Pham]

On Tue, Jun 9, 2026 at 4:35 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
> From: Wenchao Hao <haowenchao@xiaomi.com>
>
> + * ceil(log2(ZS_MAX_PAGES_PER_ZSPAGE)) at preprocessor time, for use
> + * in #if below.  Kconfig restricts ZSMALLOC_CHAIN_SIZE to [4, 16].
> + */
> +#if ZS_MAX_PAGES_PER_ZSPAGE <= 4
> +#define ZS_CHAIN_LOG2          2
> +#elif ZS_MAX_PAGES_PER_ZSPAGE <= 8
> +#define ZS_CHAIN_LOG2          3
> +#elif ZS_MAX_PAGES_PER_ZSPAGE <= 16
> +#define ZS_CHAIN_LOG2          4

Do we not have log2 macro to use here?

> +#else
> +#error "ZSMALLOC_CHAIN_SIZE out of expected range [4,16]"
> +#endif
> +
> +/* PAGE_SHIFT - 5 = log2(PAGE_SIZE / 32); 32 = ZS_MIN_ALLOC_SIZE floor. */
> +#define ZS_MAX_OBJ_PER_PAGE_LOG2       (PAGE_SHIFT - 5)
> +
> +/*
> + * obj_idx width that keeps ZS_MIN_ALLOC_SIZE at its 32-byte floor.
> + * Below this, ZS_MIN_ALLOC_SIZE is auto-raised by the MAX(32, ...)
> + * formula -- still correct, but objects are coarser.
> + */
> +#define ZS_OBJ_IDX_DENSE_BITS  (ZS_CHAIN_LOG2 + ZS_MAX_OBJ_PER_PAGE_LOG2)

ZS_CHAIN_LOG2 is (roughly) number of bits to represent the zpage index
in the zspage, and ZS_MAX_OBJ_PER_PAGE_LOG2 is roughly the number of
bits to represent the object index within the zpage, correct?

> +
> +/*
> + * Encode class_idx only when obj has spare bits; otherwise
> + * ZS_OBJ_CLASS_BITS folds to 0 (32-bit, or 64-bit UML/fallback).
> + */
> +#if BITS_PER_LONG >= 64 && \
> +       ZS_OBJ_PFN_SHIFT >= (CLASS_BITS + 1) + ZS_OBJ_IDX_DENSE_BITS
> +#define ZS_OBJ_CLASS_BITS      (CLASS_BITS + 1)

Seems good to me, albeit a bit complicated... Maybe it's because of
the naming of these macros?

Re: [PATCH v4 2/4] mm/zsmalloc: drop pool->lock from zs_free on 64-bit systems

[Nhat Pham]

On Tue, Jun 9, 2026 at 4:36 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
> From: Wenchao Hao <haowenchao@xiaomi.com>
>
> With class_idx encoded in obj, zs_free() can locate the size_class
> without holding pool->lock on 64-bit systems.  Page migration also
> takes class->lock and only rewrites the PFN field of obj, so:
>
>   1. read obj locklessly,
>   2. lock the size_class derived from obj's class_idx,
>   3. re-read obj under class->lock to get a stable PFN.
>
> This eliminates the rwlock read-side cacheline bouncing between
> zs_free() and migration/compaction on multi-core systems.
>
> Annotate handle_to_obj()/record_obj() with READ_ONCE()/WRITE_ONCE() to
> prevent load/store tearing on the lockless read path and silence KCSAN
> data race reports.
>
> When ZS_OBJ_CLASS_BITS == 0 (32-bit, or 64-bit with obj too narrow to
> hold class_idx), zs_free() keeps pool->lock.
>
> Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>

LGTM.
Reviewed-by: Nhat Pham <nphamcs@gmail.com>

Re: [PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Wenchao Hao]

On Wed, Jun 10, 2026 at 1:56 AM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Tue, Jun 9, 2026 at 4:35 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
> >
> > From: Wenchao Hao <haowenchao@xiaomi.com>
> >
> > + * ceil(log2(ZS_MAX_PAGES_PER_ZSPAGE)) at preprocessor time, for use
> > + * in #if below.  Kconfig restricts ZSMALLOC_CHAIN_SIZE to [4, 16].
> > + */
> > +#if ZS_MAX_PAGES_PER_ZSPAGE <= 4
> > +#define ZS_CHAIN_LOG2          2
> > +#elif ZS_MAX_PAGES_PER_ZSPAGE <= 8
> > +#define ZS_CHAIN_LOG2          3
> > +#elif ZS_MAX_PAGES_PER_ZSPAGE <= 16
> > +#define ZS_CHAIN_LOG2          4
>
> Do we not have log2 macro to use here?

const_ilog2() in <linux/log2.h> wraps everything in __builtin_constant_p(),
which is not a preprocessor expression and so cannot be used in an
#if condition.  Since we need this value at the preprocessor stage
(to decide ZS_OBJ_CLASS_BITS in the next #if), the manual cascade
seems unavoidable.

>
> > +#else
> > +#error "ZSMALLOC_CHAIN_SIZE out of expected range [4,16]"
> > +#endif
> > +
> > +/* PAGE_SHIFT - 5 = log2(PAGE_SIZE / 32); 32 = ZS_MIN_ALLOC_SIZE floor. */
> > +#define ZS_MAX_OBJ_PER_PAGE_LOG2       (PAGE_SHIFT - 5)
> > +
> > +/*
> > + * obj_idx width that keeps ZS_MIN_ALLOC_SIZE at its 32-byte floor.
> > + * Below this, ZS_MIN_ALLOC_SIZE is auto-raised by the MAX(32, ...)
> > + * formula -- still correct, but objects are coarser.
> > + */
> > +#define ZS_OBJ_IDX_DENSE_BITS  (ZS_CHAIN_LOG2 + ZS_MAX_OBJ_PER_PAGE_LOG2)
>
> ZS_CHAIN_LOG2 is (roughly) number of bits to represent the zpage index
> in the zspage, and ZS_MAX_OBJ_PER_PAGE_LOG2 is roughly the number of
> bits to represent the object index within the zpage, correct?
>

Exactly.  Both are really "how many bits are needed to represent X",
just expressed as log2(X):

  ZS_CHAIN_LOG2
    = ceil(log2(ZS_MAX_PAGES_PER_ZSPAGE))
    = bits to index a page within a zspage.

  ZS_MAX_OBJ_PER_PAGE_LOG2
    = log2(PAGE_SIZE / 32)
    = bits to index an object within a single page, at the smallest
      possible object size (32 bytes).

The "_LOG2" suffix names the math but hides the intent.  How about
renaming them as:

  ZS_CHAIN_LOG2            -> ZS_MAX_BITS_PAGES_PER_ZSPAGE
  ZS_MAX_OBJ_PER_PAGE_LOG2 -> ZS_MAX_BITS_OBJS_PER_PAGE
  ZS_OBJ_IDX_DENSE_BITS    -> ZS_MAX_BITS_OBJS_PER_ZSPAGE

Then the third one reads as the sum of the first two: "bits to index
any object in the densest zspage", and the ZS_OBJ_CLASS_BITS predicate
gets easier to follow.

Does that work for you?

Wenchao

Re: [PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Barry Song]

On Wed, Jun 10, 2026 at 11:18 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
[...]
>
> The "_LOG2" suffix names the math but hides the intent.  How about
> renaming them as:
>
>   ZS_CHAIN_LOG2            -> ZS_MAX_BITS_PAGES_PER_ZSPAGE
>   ZS_MAX_OBJ_PER_PAGE_LOG2 -> ZS_MAX_BITS_OBJS_PER_PAGE
>   ZS_OBJ_IDX_DENSE_BITS    -> ZS_MAX_BITS_OBJS_PER_ZSPAGE
>

Do we need MAX_ while everybody just uses _BITS? e.g.

#if defined(MAX_POSSIBLE_PHYSMEM_BITS)
#define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
#elif defined(MAX_PHYSMEM_BITS)
#define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
#else
#define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
#endif

Best Regards
Barry

Re: [PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Wenchao Hao]

On Thu, Jun 11, 2026 at 5:28 AM Barry Song <baohua@kernel.org> wrote:
>
> On Wed, Jun 10, 2026 at 11:18 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
> [...]
> >
> > The "_LOG2" suffix names the math but hides the intent.  How about
> > renaming them as:
> >
> >   ZS_CHAIN_LOG2            -> ZS_MAX_BITS_PAGES_PER_ZSPAGE
> >   ZS_MAX_OBJ_PER_PAGE_LOG2 -> ZS_MAX_BITS_OBJS_PER_PAGE
> >   ZS_OBJ_IDX_DENSE_BITS    -> ZS_MAX_BITS_OBJS_PER_ZSPAGE
> >
>
> Do we need MAX_ while everybody just uses _BITS? e.g.
>
> #if defined(MAX_POSSIBLE_PHYSMEM_BITS)
> #define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
> #elif defined(MAX_PHYSMEM_BITS)
> #define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
> #else
> #define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
> #endif
>

Agreed, will follow the convention and drop MAX_:

  ZS_CHAIN_LOG2            -> ZS_PAGES_PER_ZSPAGE_BITS
  ZS_MAX_OBJ_PER_PAGE_LOG2 -> ZS_OBJS_PER_PAGE_BITS
  ZS_OBJ_IDX_DENSE_BITS    -> ZS_OBJS_PER_ZSPAGE_BITS

Nhat -- sorry for the churn; this supersedes the names I proposed
earlier.  Let me know if the new naming still reads well to you.

Wenchao

> Best Regards
> Barry

Re: [PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Nhat Pham]

On Wed, Jun 10, 2026 at 8:15 PM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
> On Thu, Jun 11, 2026 at 5:28 AM Barry Song <baohua@kernel.org> wrote:
>
> Agreed, will follow the convention and drop MAX_:
>
>   ZS_CHAIN_LOG2            -> ZS_PAGES_PER_ZSPAGE_BITS
>   ZS_MAX_OBJ_PER_PAGE_LOG2 -> ZS_OBJS_PER_PAGE_BITS
>   ZS_OBJ_IDX_DENSE_BITS    -> ZS_OBJS_PER_ZSPAGE_BITS
>
> Nhat -- sorry for the churn; this supersedes the names I proposed
> earlier.  Let me know if the new naming still reads well to you.

No worries at all. I like this a lot :)

Re: [PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Wenchao Hao]

On Thu, Jun 11, 2026 at 10:31 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Wed, Jun 10, 2026 at 8:15 PM Wenchao Hao <haowenchao22@gmail.com> wrote:
> >
> > On Thu, Jun 11, 2026 at 5:28 AM Barry Song <baohua@kernel.org> wrote:
> >
> > Agreed, will follow the convention and drop MAX_:
> >
> >   ZS_CHAIN_LOG2            -> ZS_PAGES_PER_ZSPAGE_BITS
> >   ZS_MAX_OBJ_PER_PAGE_LOG2 -> ZS_OBJS_PER_PAGE_BITS
> >   ZS_OBJ_IDX_DENSE_BITS    -> ZS_OBJS_PER_ZSPAGE_BITS
> >
> > Nhat -- sorry for the churn; this supersedes the names I proposed
> > earlier.  Let me know if the new naming still reads well to you.
>
> No worries at all. I like this a lot :)

Thanks!  Since the rest of 1/4 is unchanged from v4 apart from the
rename, would you be open to re-applying your Reviewed-by on the
next spin?

Wenchao

Re: [PATCH v4 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Nhat Pham]

On Thu, Jun 11, 2026 at 10:00 PM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
>
> Thanks!  Since the rest of 1/4 is unchanged from v4 apart from the
> rename, would you be open to re-applying your Reviewed-by on the
> next spin?

Yeah feel free to keep my review tag.