[RFC PATCH v2 0/3] mm/zsmalloc: reduce lock contention in zs_free()

[RFC PATCH v2 0/3] mm/zsmalloc: reduce lock contention in zs_free()

[Wenchao Hao]

This series reduces lock contention in zs_free(), which dominates the
unmap path under memory pressure on Android (LMK kills) and on x86
servers running zswap-heavy workloads.

The current zs_free() takes pool->lock (rwlock, read side) just to
look up the size_class for a handle, then takes class->lock and holds
it across __free_zspage() which can call into the buddy allocator and
acquire zone->lock.  Two costs follow:

  * pool->lock reader-counter cacheline bouncing among concurrent
    zs_free() callers.
  * class->lock held across folio_put(), so any zone->lock wait
    fans out to every other zs_free() on the same class.

The series tackles both:

  Patch 1: encode size_class index into obj alongside PFN and obj_idx,
           so zs_free() can locate the class without pool->lock.
  Patch 2: drop pool->lock from zs_free() on 64-bit; 32-bit unchanged.
  Patch 3: move zspage page-freeing out of class->lock.

Performance results:

Test: each process independently mmap 256MB, write data, madvise
MADV_PAGEOUT to swap out via zram (lzo-rle), then concurrent munmap.

Raspberry Pi 4B (4-core ARM64 Cortex-A72):

  mode        Base       Patched     Speedup
  single      59.0ms     56.0ms      1.05x
  multi 2p    94.6ms     66.7ms      1.42x
  multi 4p    202.9ms    110.6ms     1.83x

x86 (20-core Intel i7-12700, 16 concurrent processes):

  mode        Base       Patched     Speedup
  single      11.7ms     9.8ms       1.19x
  multi 2p    24.1ms     17.2ms      1.40x
  multi 4p    63.0ms     45.3ms      1.39x

Changes since v1:
- Rename obj-encoding macros for clarity.
- Make 32-bit / 64-bit handling transparent at call sites
  (no #ifdef in callers).
- Extract a helper to keep zs_free() unified.

Wenchao Hao (2):
  mm/zsmalloc: encode class index in obj value for lockless class lookup
  mm/zsmalloc: drop pool->lock from zs_free on 64-bit systems

Xueyuan Chen (1):
  mm/zsmalloc: drop class lock before freeing zspage

 mm/zsmalloc.c | 175 +++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 143 insertions(+), 32 deletions(-)

--
2.34.1

[RFC PATCH v2 1/3] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Wenchao Hao]

Encode the size_class index (class_idx) into the obj value so that
zs_free() can determine the correct size_class without dereferencing
the handle->obj->PFN->zpdesc->zspage->class chain under pool->lock.
class_idx is invariant across page migration (only PFN is rewritten),
so a lockless read of obj always yields a valid class_idx.

The space below the PFN field in obj is over-provisioned on 64-bit
systems, with more bits than obj_idx needs. Split that space into
class_idx and obj_idx subfields:

 |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
 +-----------------+-------------------------+-----------------------+
 |       PFN       |        class_idx        |        obj_idx        |
 +-----------------+-------------------------+-----------------------+
MSB                ^                                                LSB
                   |
                   +-- ZS_OBJ_PFN_SHIFT

The macro layout changes as follows:

    Before            After               Meaning
    ----------------  ------------------  ----------------------------
    OBJ_INDEX_BITS    ZS_OBJ_IDX_BITS     width of obj_idx subfield
    OBJ_INDEX_MASK    ZS_OBJ_IDX_MASK     mask  of obj_idx subfield
    (n/a)             ZS_OBJ_CLASS_BITS   width of class_idx subfield
    (n/a)             ZS_OBJ_CLASS_MASK   mask  of class_idx subfield
    (n/a)             ZS_OBJ_PFN_SHIFT    bit offset of PFN in obj

On 32-bit systems there is no spare room for class_idx, so the
encoding is disabled (ZS_OBJ_CLASS_BITS = 0) and the obj layout
remains [PFN | obj_idx].

Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
---
 mm/zsmalloc.c | 80 ++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 66 insertions(+), 14 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 63128ddb7959..6b0014b43408 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -67,8 +67,8 @@
 #define MAX_POSSIBLE_PHYSMEM_BITS MAX_PHYSMEM_BITS
 #else
 /*
- * If this definition of MAX_PHYSMEM_BITS is used, OBJ_INDEX_BITS will just
- * be PAGE_SHIFT
+ * If this definition of MAX_PHYSMEM_BITS is used, ZS_OBJ_PFN_SHIFT will
+ * just be PAGE_SHIFT
  */
 #define MAX_POSSIBLE_PHYSMEM_BITS BITS_PER_LONG
 #endif
@@ -88,8 +88,27 @@
 #define OBJ_TAG_BITS	1
 #define OBJ_TAG_MASK	OBJ_ALLOCATED_TAG
 
-#define OBJ_INDEX_BITS	(BITS_PER_LONG - _PFN_BITS)
-#define OBJ_INDEX_MASK	((_AC(1, UL) << OBJ_INDEX_BITS) - 1)
+/*
+ * obj is encoded as [PFN | class_idx | obj_idx] within an unsigned long:
+ *
+ *   |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
+ *   +-----------------+-------------------------+-----------------------+
+ *   |       PFN       |        class_idx        |        obj_idx        |
+ *   +-----------------+-------------------------+-----------------------+
+ *  MSB                ^                                                LSB
+ *                     |
+ *                     +-- ZS_OBJ_PFN_SHIFT
+ *
+ * Encoding class_idx into obj lets zs_free() locate the size_class
+ * without holding pool->lock; class_idx is invariant across page
+ * migration (only PFN changes), so a lockless read of the obj value
+ * always yields a valid class_idx.
+ *
+ * On 32-bit systems there is no spare room for class_idx, so
+ * ZS_OBJ_CLASS_BITS is 0 and the layout collapses to the original
+ * [PFN | obj_idx] without any ifdef in callers.
+ */
+#define ZS_OBJ_PFN_SHIFT	(BITS_PER_LONG - _PFN_BITS)
 
 #define HUGE_BITS	1
 #define FULLNESS_BITS	4
@@ -98,9 +117,29 @@
 
 #define ZS_MAX_PAGES_PER_ZSPAGE	(_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL))
 
+/*
+ * Reuse the width that struct zspage already reserves for its
+ * class field (zspage->class:CLASS_BITS + 1) for the class_idx
+ * field encoded in obj.  On 32-bit there is no spare room, so set
+ * it to 0; the encoded class_idx then folds to a constant 0 and
+ * the layout collapses back to [PFN | obj_idx].
+ */
+#if BITS_PER_LONG >= 64
+#define ZS_OBJ_CLASS_BITS	(CLASS_BITS + 1)
+#else
+#define ZS_OBJ_CLASS_BITS	0
+#endif
+#define ZS_OBJ_CLASS_MASK	((_AC(1, UL) << ZS_OBJ_CLASS_BITS) - 1)
+
+#define ZS_OBJ_IDX_BITS		(ZS_OBJ_PFN_SHIFT - ZS_OBJ_CLASS_BITS)
+#define ZS_OBJ_IDX_MASK		((_AC(1, UL) << ZS_OBJ_IDX_BITS) - 1)
+
+static_assert(ZS_OBJ_IDX_BITS > 0,
+	      "zsmalloc: PFN + class_idx leave no room for obj_idx");
+
 /* ZS_MIN_ALLOC_SIZE must be multiple of ZS_ALIGN */
 #define ZS_MIN_ALLOC_SIZE \
-	MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> OBJ_INDEX_BITS))
+	MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> ZS_OBJ_IDX_BITS))
 /* each chunk includes extra space to keep handle */
 #define ZS_MAX_ALLOC_SIZE	PAGE_SIZE
 
@@ -721,26 +760,38 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
 static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc,
 				unsigned int *obj_idx)
 {
-	*zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
-	*obj_idx = (obj & OBJ_INDEX_MASK);
+	*zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT);
+	*obj_idx = (obj & ZS_OBJ_IDX_MASK);
 }
 
 static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc)
 {
-	*zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
+	*zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT);
+}
+
+/*
+ * On 32-bit systems ZS_OBJ_CLASS_BITS is 0 and ZS_OBJ_CLASS_MASK is 0,
+ * so this collapses to a constant 0.  No ifdef needed at the call site.
+ */
+static unsigned int obj_to_class_idx(unsigned long obj)
+{
+	return (obj >> ZS_OBJ_IDX_BITS) & ZS_OBJ_CLASS_MASK;
 }
 
 /**
- * location_to_obj - get obj value encoded from (<zpdesc>, <obj_idx>)
+ * location_to_obj - encode (<zpdesc>, <obj_idx>, <class_idx>) into obj value
  * @zpdesc: zpdesc object resides in zspage
  * @obj_idx: object index
+ * @class_idx: size class index; ignored on 32-bit (ZS_OBJ_CLASS_BITS == 0)
  */
-static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx)
+static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx,
+				     unsigned int class_idx)
 {
 	unsigned long obj;
 
-	obj = zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS;
-	obj |= obj_idx & OBJ_INDEX_MASK;
+	obj  = zpdesc_pfn(zpdesc) << ZS_OBJ_PFN_SHIFT;
+	obj |= (unsigned long)(class_idx & ZS_OBJ_CLASS_MASK) << ZS_OBJ_IDX_BITS;
+	obj |= obj_idx & ZS_OBJ_IDX_MASK;
 
 	return obj;
 }
@@ -1276,7 +1327,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	kunmap_local(vaddr);
 	mod_zspage_inuse(zspage, 1);
 
-	obj = location_to_obj(m_zpdesc, obj);
+	obj = location_to_obj(m_zpdesc, obj, zspage->class);
 	record_obj(handle, obj);
 
 	return obj;
@@ -1762,7 +1813,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);
-			new_obj = (unsigned long)location_to_obj(newzpdesc, obj_idx);
+			new_obj = location_to_obj(newzpdesc, obj_idx,
+						  obj_to_class_idx(old_obj));
 			record_obj(handle, new_obj);
 		}
 	}
-- 
2.34.1

Re: [RFC PATCH v2 1/3] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Nhat Pham]

On Wed, May 27, 2026 at 5:01 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
> Encode the size_class index (class_idx) into the obj value so that
> zs_free() can determine the correct size_class without dereferencing
> the handle->obj->PFN->zpdesc->zspage->class chain under pool->lock.
> class_idx is invariant across page migration (only PFN is rewritten),
> so a lockless read of obj always yields a valid class_idx.
>
> The space below the PFN field in obj is over-provisioned on 64-bit
> systems, with more bits than obj_idx needs. Split that space into
> class_idx and obj_idx subfields:
>
>  |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
>  +-----------------+-------------------------+-----------------------+
>  |       PFN       |        class_idx        |        obj_idx        |
>  +-----------------+-------------------------+-----------------------+
> MSB                ^                                                LSB
>                    |
>                    +-- ZS_OBJ_PFN_SHIFT
>
> The macro layout changes as follows:
>
>     Before            After               Meaning
>     ----------------  ------------------  ----------------------------
>     OBJ_INDEX_BITS    ZS_OBJ_IDX_BITS     width of obj_idx subfield
>     OBJ_INDEX_MASK    ZS_OBJ_IDX_MASK     mask  of obj_idx subfield
>     (n/a)             ZS_OBJ_CLASS_BITS   width of class_idx subfield
>     (n/a)             ZS_OBJ_CLASS_MASK   mask  of class_idx subfield
>     (n/a)             ZS_OBJ_PFN_SHIFT    bit offset of PFN in obj
>
> On 32-bit systems there is no spare room for class_idx, so the
> encoding is disabled (ZS_OBJ_CLASS_BITS = 0) and the obj layout
> remains [PFN | obj_idx].
>
> Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>

LGTM.

Reviewed-by: Nhat Pham <nphamcs@gmail.com>

> ---
>  mm/zsmalloc.c | 80 ++++++++++++++++++++++++++++++++++++++++++---------
>  1 file changed, 66 insertions(+), 14 deletions(-)
>
> diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
> index 63128ddb7959..6b0014b43408 100644
> --- a/mm/zsmalloc.c
> +++ b/mm/zsmalloc.c
> @@ -67,8 +67,8 @@
>  #define MAX_POSSIBLE_PHYSMEM_BITS MAX_PHYSMEM_BITS
>  #else
>  /*
> - * If this definition of MAX_PHYSMEM_BITS is used, OBJ_INDEX_BITS will just
> - * be PAGE_SHIFT
> + * If this definition of MAX_PHYSMEM_BITS is used, ZS_OBJ_PFN_SHIFT will
> + * just be PAGE_SHIFT
>   */
>  #define MAX_POSSIBLE_PHYSMEM_BITS BITS_PER_LONG
>  #endif
> @@ -88,8 +88,27 @@
>  #define OBJ_TAG_BITS   1
>  #define OBJ_TAG_MASK   OBJ_ALLOCATED_TAG
>
> -#define OBJ_INDEX_BITS (BITS_PER_LONG - _PFN_BITS)
> -#define OBJ_INDEX_MASK ((_AC(1, UL) << OBJ_INDEX_BITS) - 1)
> +/*
> + * obj is encoded as [PFN | class_idx | obj_idx] within an unsigned long:
> + *
> + *   |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
> + *   +-----------------+-------------------------+-----------------------+
> + *   |       PFN       |        class_idx        |        obj_idx        |
> + *   +-----------------+-------------------------+-----------------------+
> + *  MSB                ^                                                LSB
> + *                     |
> + *                     +-- ZS_OBJ_PFN_SHIFT
> + *
> + * Encoding class_idx into obj lets zs_free() locate the size_class
> + * without holding pool->lock; class_idx is invariant across page
> + * migration (only PFN changes), so a lockless read of the obj value
> + * always yields a valid class_idx.
> + *
> + * On 32-bit systems there is no spare room for class_idx, so
> + * ZS_OBJ_CLASS_BITS is 0 and the layout collapses to the original
> + * [PFN | obj_idx] without any ifdef in callers.
> + */
> +#define ZS_OBJ_PFN_SHIFT       (BITS_PER_LONG - _PFN_BITS)
>
>  #define HUGE_BITS      1
>  #define FULLNESS_BITS  4
> @@ -98,9 +117,29 @@
>
>  #define ZS_MAX_PAGES_PER_ZSPAGE        (_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL))
>
> +/*
> + * Reuse the width that struct zspage already reserves for its
> + * class field (zspage->class:CLASS_BITS + 1) for the class_idx
> + * field encoded in obj.  On 32-bit there is no spare room, so set
> + * it to 0; the encoded class_idx then folds to a constant 0 and
> + * the layout collapses back to [PFN | obj_idx].
> + */
> +#if BITS_PER_LONG >= 64
> +#define ZS_OBJ_CLASS_BITS      (CLASS_BITS + 1)
> +#else
> +#define ZS_OBJ_CLASS_BITS      0
> +#endif
> +#define ZS_OBJ_CLASS_MASK      ((_AC(1, UL) << ZS_OBJ_CLASS_BITS) - 1)
> +
> +#define ZS_OBJ_IDX_BITS                (ZS_OBJ_PFN_SHIFT - ZS_OBJ_CLASS_BITS)
> +#define ZS_OBJ_IDX_MASK                ((_AC(1, UL) << ZS_OBJ_IDX_BITS) - 1)
> +
> +static_assert(ZS_OBJ_IDX_BITS > 0,
> +             "zsmalloc: PFN + class_idx leave no room for obj_idx");
> +
>  /* ZS_MIN_ALLOC_SIZE must be multiple of ZS_ALIGN */
>  #define ZS_MIN_ALLOC_SIZE \
> -       MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> OBJ_INDEX_BITS))
> +       MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> ZS_OBJ_IDX_BITS))
>  /* each chunk includes extra space to keep handle */
>  #define ZS_MAX_ALLOC_SIZE      PAGE_SIZE
>
> @@ -721,26 +760,38 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
>  static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc,
>                                 unsigned int *obj_idx)
>  {
> -       *zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
> -       *obj_idx = (obj & OBJ_INDEX_MASK);
> +       *zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT);
> +       *obj_idx = (obj & ZS_OBJ_IDX_MASK);
>  }
>
>  static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc)
>  {
> -       *zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
> +       *zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT);
> +}
> +
> +/*
> + * On 32-bit systems ZS_OBJ_CLASS_BITS is 0 and ZS_OBJ_CLASS_MASK is 0,
> + * so this collapses to a constant 0.  No ifdef needed at the call site.

nit: I think this comment is a bit redundant - you already explain
this at the data layout section above.

> + */
> +static unsigned int obj_to_class_idx(unsigned long obj)
> +{
> +       return (obj >> ZS_OBJ_IDX_BITS) & ZS_OBJ_CLASS_MASK;
>  }
>
>  /**
> - * location_to_obj - get obj value encoded from (<zpdesc>, <obj_idx>)
> + * location_to_obj - encode (<zpdesc>, <obj_idx>, <class_idx>) into obj value
>   * @zpdesc: zpdesc object resides in zspage
>   * @obj_idx: object index
> + * @class_idx: size class index; ignored on 32-bit (ZS_OBJ_CLASS_BITS == 0)
>   */
> -static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx)
> +static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx,
> +                                    unsigned int class_idx)
>  {
>         unsigned long obj;
>
> -       obj = zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS;
> -       obj |= obj_idx & OBJ_INDEX_MASK;
> +       obj  = zpdesc_pfn(zpdesc) << ZS_OBJ_PFN_SHIFT;
> +       obj |= (unsigned long)(class_idx & ZS_OBJ_CLASS_MASK) << ZS_OBJ_IDX_BITS;
> +       obj |= obj_idx & ZS_OBJ_IDX_MASK;
>
>         return obj;
>  }
> @@ -1276,7 +1327,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
>         kunmap_local(vaddr);
>         mod_zspage_inuse(zspage, 1);
>
> -       obj = location_to_obj(m_zpdesc, obj);
> +       obj = location_to_obj(m_zpdesc, obj, zspage->class);
>         record_obj(handle, obj);
>
>         return obj;
> @@ -1762,7 +1813,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
>
>                         old_obj = handle_to_obj(handle);
>                         obj_to_location(old_obj, &dummy, &obj_idx);
> -                       new_obj = (unsigned long)location_to_obj(newzpdesc, obj_idx);
> +                       new_obj = location_to_obj(newzpdesc, obj_idx,
> +                                                 obj_to_class_idx(old_obj));
>                         record_obj(handle, new_obj);
>                 }
>         }
> --
> 2.34.1
>

Re: [RFC PATCH v2 1/3] mm/zsmalloc: encode class index in obj value for lockless class lookup

[Barry Song]

On Wed, May 27, 2026 at 8:02 PM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
> Encode the size_class index (class_idx) into the obj value so that
> zs_free() can determine the correct size_class without dereferencing
> the handle->obj->PFN->zpdesc->zspage->class chain under pool->lock.
> class_idx is invariant across page migration (only PFN is rewritten),
> so a lockless read of obj always yields a valid class_idx.
>
> The space below the PFN field in obj is over-provisioned on 64-bit
> systems, with more bits than obj_idx needs. Split that space into
> class_idx and obj_idx subfields:
>
>  |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->|
>  +-----------------+-------------------------+-----------------------+
>  |       PFN       |        class_idx        |        obj_idx        |
>  +-----------------+-------------------------+-----------------------+
> MSB                ^                                                LSB
>                    |
>                    +-- ZS_OBJ_PFN_SHIFT
>
> The macro layout changes as follows:
>
>     Before            After               Meaning
>     ----------------  ------------------  ----------------------------
>     OBJ_INDEX_BITS    ZS_OBJ_IDX_BITS     width of obj_idx subfield
>     OBJ_INDEX_MASK    ZS_OBJ_IDX_MASK     mask  of obj_idx subfield
>     (n/a)             ZS_OBJ_CLASS_BITS   width of class_idx subfield
>     (n/a)             ZS_OBJ_CLASS_MASK   mask  of class_idx subfield
>     (n/a)             ZS_OBJ_PFN_SHIFT    bit offset of PFN in obj
>
> On 32-bit systems there is no spare room for class_idx, so the
> encoding is disabled (ZS_OBJ_CLASS_BITS = 0) and the obj layout
> remains [PFN | obj_idx].
>
> Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
> ---

Much better than v1.

Please make sure that the From: field matches the Signed-off-by: line next time.

Please commit the patch on your machine using:

Wenchao Hao <haowenchao@xiaomi.com>

as the author.
Then git send-email will automatically generate the correct From: line
in the email.

[RFC PATCH v2 2/3] mm/zsmalloc: drop pool->lock from zs_free on 64-bit systems

[Wenchao Hao]

With class_idx encoded in obj, zs_free() can locate the size_class
without holding pool->lock on 64-bit systems.  Page migration also
takes class->lock and only rewrites the PFN field of obj, so:

  1. read obj locklessly,
  2. lock the size_class derived from obj's class_idx,
  3. re-read obj under class->lock to get a stable PFN.

This eliminates the rwlock read-side cacheline bouncing between
zs_free() and migration/compaction on multi-core systems.

On 32-bit systems pool->lock is preserved.

Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
---
 mm/zsmalloc.c | 67 ++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 55 insertions(+), 12 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 6b0014b43408..88d10f814da9 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -21,6 +21,10 @@
  *	pool->lock
  *	class->lock
  *	zspage->lock
+ *
+ * On 64-bit systems zs_free() does not take pool->lock; it locates
+ * the size_class using class_idx encoded in obj and serializes against
+ * page migration via class->lock.
  */
 
 #include <linux/module.h>
@@ -1432,10 +1436,59 @@ static void obj_free(int class_size, unsigned long obj)
 	mod_zspage_inuse(zspage, -1);
 }
 
+/*
+ * Resolve @handle to its zspage / size_class and acquire class->lock.
+ *
+ * 64-bit: class_idx is encoded in obj and is invariant under page
+ * migration, so the handle can be read locklessly to pick the
+ * size_class.  Once class->lock is held migration is blocked and the
+ * handle is re-read to obtain a stable PFN.
+ *
+ * 32-bit: the unlocked load of *(unsigned long *)handle is not
+ * single-copy atomic and class_idx is not encoded (ZS_OBJ_CLASS_BITS
+ * == 0), so fall back to pool->lock for the lookup.
+ */
+#if BITS_PER_LONG >= 64
+static inline void obj_handle_class_lock(struct zs_pool *pool, unsigned long handle,
+					 unsigned long *objp, struct zspage **zspagep,
+					 struct size_class **classp)
+	__acquires(&(*classp)->lock)
+{
+	struct zpdesc *f_zpdesc;
+	unsigned long obj;
+
+	obj = handle_to_obj(handle);
+	*classp = pool->size_class[obj_to_class_idx(obj)];
+	spin_lock(&(*classp)->lock);
+	/* Re-read under class->lock: PFN is now stable vs migration. */
+	obj = handle_to_obj(handle);
+	obj_to_zpdesc(obj, &f_zpdesc);
+	*zspagep = get_zspage(f_zpdesc);
+	*objp = obj;
+}
+#else
+static inline void obj_handle_class_lock(struct zs_pool *pool, unsigned long handle,
+					 unsigned long *objp, struct zspage **zspagep,
+					 struct size_class **classp)
+	__acquires(&(*classp)->lock)
+{
+	struct zpdesc *f_zpdesc;
+	unsigned long obj;
+
+	read_lock(&pool->lock);
+	obj = handle_to_obj(handle);
+	obj_to_zpdesc(obj, &f_zpdesc);
+	*zspagep = get_zspage(f_zpdesc);
+	*classp = zspage_class(pool, *zspagep);
+	spin_lock(&(*classp)->lock);
+	read_unlock(&pool->lock);
+	*objp = obj;
+}
+#endif
+
 void zs_free(struct zs_pool *pool, unsigned long handle)
 {
 	struct zspage *zspage;
-	struct zpdesc *f_zpdesc;
 	unsigned long obj;
 	struct size_class *class;
 	int fullness;
@@ -1443,17 +1496,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	if (IS_ERR_OR_NULL((void *)handle))
 		return;
 
-	/*
-	 * The pool->lock protects the race with zpage's migration
-	 * so it's safe to get the page from handle.
-	 */
-	read_lock(&pool->lock);
-	obj = handle_to_obj(handle);
-	obj_to_zpdesc(obj, &f_zpdesc);
-	zspage = get_zspage(f_zpdesc);
-	class = zspage_class(pool, zspage);
-	spin_lock(&class->lock);
-	read_unlock(&pool->lock);
+	obj_handle_class_lock(pool, handle, &obj, &zspage, &class);
 
 	class_stat_sub(class, ZS_OBJS_INUSE, 1);
 	obj_free(class->size, obj);
-- 
2.34.1

Re: [RFC PATCH v2 2/3] mm/zsmalloc: drop pool->lock from zs_free on 64-bit systems

[Nhat Pham]

On Wed, May 27, 2026 at 5:01 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
> With class_idx encoded in obj, zs_free() can locate the size_class
> without holding pool->lock on 64-bit systems.  Page migration also
> takes class->lock and only rewrites the PFN field of obj, so:
>
>   1. read obj locklessly,
>   2. lock the size_class derived from obj's class_idx,
>   3. re-read obj under class->lock to get a stable PFN.
>
> This eliminates the rwlock read-side cacheline bouncing between
> zs_free() and migration/compaction on multi-core systems.
>
> On 32-bit systems pool->lock is preserved.
>
> Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>

Thanks for doing this!

Reviewed-by: Nhat Pham <nphamcs@gmail.com>

[RFC PATCH v2 3/3] mm/zsmalloc: drop class lock before freeing zspage

[Wenchao Hao]

From: Xueyuan Chen <xueyuan.chen21@gmail.com>

Currently in zs_free(), the class->lock is held until the zspage is
completely freed and the counters are updated. However, freeing pages
back to the buddy allocator requires acquiring the zone lock.

Under heavy memory pressure, zone lock contention can be severe. When
this happens, the CPU holding the class->lock will stall waiting for
the zone lock, thereby blocking all other CPUs attempting to acquire
the same class->lock.

This patch shrinks the critical section of the class->lock to reduce
lock contention. By moving the actual page freeing process outside the
class->lock, we can improve the concurrency performance of zs_free().

Testing on the RADXA O6 platform shows that with 12 CPUs concurrently
performing zs_free() operations, the execution time is reduced by 20%.

Signed-off-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
---
 mm/zsmalloc.c | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 88d10f814da9..5511c347d00b 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -856,13 +856,10 @@ static int trylock_zspage(struct zspage *zspage)
 	return 0;
 }
 
-static void __free_zspage(struct zs_pool *pool, struct size_class *class,
-				struct zspage *zspage)
+static inline void __free_zspage_lockless(struct zs_pool *pool, struct zspage *zspage)
 {
 	struct zpdesc *zpdesc, *next;
 
-	assert_spin_locked(&class->lock);
-
 	VM_BUG_ON(get_zspage_inuse(zspage));
 	VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0);
 
@@ -878,7 +875,13 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
 	} while (zpdesc != NULL);
 
 	cache_free_zspage(zspage);
+}
 
+static void __free_zspage(struct zs_pool *pool, struct size_class *class,
+				struct zspage *zspage)
+{
+	assert_spin_locked(&class->lock);
+	__free_zspage_lockless(pool, zspage);
 	class_stat_sub(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage);
 	atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
 }
@@ -1492,6 +1495,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	unsigned long obj;
 	struct size_class *class;
 	int fullness;
+	struct zspage *zspage_to_free = NULL;
 
 	if (IS_ERR_OR_NULL((void *)handle))
 		return;
@@ -1502,10 +1506,22 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	obj_free(class->size, obj);
 
 	fullness = fix_fullness_group(class, zspage);
-	if (fullness == ZS_INUSE_RATIO_0)
-		free_zspage(pool, class, zspage);
+	if (fullness == ZS_INUSE_RATIO_0) {
+		if (trylock_zspage(zspage)) {
+			remove_zspage(class, zspage);
+			class_stat_sub(class, ZS_OBJS_ALLOCATED,
+				class->objs_per_zspage);
+			zspage_to_free = zspage;
+		} else
+			kick_deferred_free(pool);
+	}
 
 	spin_unlock(&class->lock);
+
+	if (likely(zspage_to_free)) {
+		__free_zspage_lockless(pool, zspage_to_free);
+		atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
+	}
 	cache_free_handle(handle);
 }
 EXPORT_SYMBOL_GPL(zs_free);
-- 
2.34.1

Re: [RFC PATCH v2 3/3] mm/zsmalloc: drop class lock before freeing zspage

[Joshua Hahn]

On Wed, 27 May 2026 19:59:30 +0800 Wenchao Hao <haowenchao22@gmail.com> wrote:

> From: Xueyuan Chen <xueyuan.chen21@gmail.com>
> 
> Currently in zs_free(), the class->lock is held until the zspage is
> completely freed and the counters are updated. However, freeing pages
> back to the buddy allocator requires acquiring the zone lock.
> 
> Under heavy memory pressure, zone lock contention can be severe. When
> this happens, the CPU holding the class->lock will stall waiting for
> the zone lock, thereby blocking all other CPUs attempting to acquire
> the same class->lock.
> 
> This patch shrinks the critical section of the class->lock to reduce
> lock contention. By moving the actual page freeing process outside the
> class->lock, we can improve the concurrency performance of zs_free().

Hello Wenchao and Xueyuan,

Thank you for the patch. This is a really cool idea and I think it makes
a lot of sense. 

LGTM aside from a few nits, with those resolved please feel free to add
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

>  	fullness = fix_fullness_group(class, zspage);
> -	if (fullness == ZS_INUSE_RATIO_0)
> -		free_zspage(pool, class, zspage);
> +	if (fullness == ZS_INUSE_RATIO_0) {
> +		if (trylock_zspage(zspage)) {
> +			remove_zspage(class, zspage);
> +			class_stat_sub(class, ZS_OBJS_ALLOCATED,
> +				class->objs_per_zspage);
> +			zspage_to_free = zspage;
> +		} else
> +			kick_deferred_free(pool);

NIT: We should probably add opening and closing braces here as well since the
if() case is more than one line. 

> +	}
>  
>  	spin_unlock(&class->lock);

Sashiko caught this in [1] and I was confused why the likely() doesn't make
sense here, but I finally realized that zspage_to_free is likely to be set
when comparing trylock_zspage() success vs. failure, but it's unlikely to
enter the ZS_INUSE_RATIO_0 case to begin with, so an unlikely() seems
correct here. 

Thanks again. I hope you have a great day!
Joshua

> +	if (likely(zspage_to_free)) {
> +		__free_zspage_lockless(pool, zspage_to_free);
> +		atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
> +	}
>  	cache_free_handle(handle);
>  }
>  EXPORT_SYMBOL_GPL(zs_free);

[1] https://sashiko.dev/#/patchset/20260527115930.3138213-1-haowenchao%40xiaomi.com

Re: [RFC PATCH v2 3/3] mm/zsmalloc: drop class lock before freeing zspage

[Nhat Pham]

On Wed, May 27, 2026 at 5:01 AM Wenchao Hao <haowenchao22@gmail.com> wrote:
>
> From: Xueyuan Chen <xueyuan.chen21@gmail.com>
>
> Currently in zs_free(), the class->lock is held until the zspage is
> completely freed and the counters are updated. However, freeing pages
> back to the buddy allocator requires acquiring the zone lock.
>
> Under heavy memory pressure, zone lock contention can be severe. When
> this happens, the CPU holding the class->lock will stall waiting for
> the zone lock, thereby blocking all other CPUs attempting to acquire
> the same class->lock.
>
> This patch shrinks the critical section of the class->lock to reduce
> lock contention. By moving the actual page freeing process outside the
> class->lock, we can improve the concurrency performance of zs_free().
>
> Testing on the RADXA O6 platform shows that with 12 CPUs concurrently
> performing zs_free() operations, the execution time is reduced by 20%.
>
> Signed-off-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
> Signed-off-by: Wenchao Hao <haowenchao@xiaomi.com>
> ---
>  mm/zsmalloc.c | 28 ++++++++++++++++++++++------
>  1 file changed, 22 insertions(+), 6 deletions(-)

LGTM for the most part, with small nit.

Reviewed-by: Nhat Pham <nphamcs@gmail.com>

>
> diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
> index 88d10f814da9..5511c347d00b 100644
> --- a/mm/zsmalloc.c
> +++ b/mm/zsmalloc.c
> @@ -856,13 +856,10 @@ static int trylock_zspage(struct zspage *zspage)
>         return 0;
>  }
>
> -static void __free_zspage(struct zs_pool *pool, struct size_class *class,
> -                               struct zspage *zspage)
> +static inline void __free_zspage_lockless(struct zs_pool *pool, struct zspage *zspage)
>  {
>         struct zpdesc *zpdesc, *next;
>
> -       assert_spin_locked(&class->lock);
> -
>         VM_BUG_ON(get_zspage_inuse(zspage));
>         VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0);
>
> @@ -878,7 +875,13 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
>         } while (zpdesc != NULL);
>
>         cache_free_zspage(zspage);
> +}
>

nit: we have *too many* versions of free_zspage now. Can you add some
small documentation for:

free_zspage

__free_zspage

__free_zspage_lockless

to help reader distinguish between the various cases? What do they do,
what cases are they used, etc.?

Can be a small fixlet or follow-up patch.

> +static void __free_zspage(struct zs_pool *pool, struct size_class *class,
> +                               struct zspage *zspage)
> +{
> +       assert_spin_locked(&class->lock);
> +       __free_zspage_lockless(pool, zspage);
>         class_stat_sub(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage);
>         atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
>  }
> @@ -1492,6 +1495,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
>         unsigned long obj;
>         struct size_class *class;
>         int fullness;
> +       struct zspage *zspage_to_free = NULL;
>
>         if (IS_ERR_OR_NULL((void *)handle))
>                 return;
> @@ -1502,10 +1506,22 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
>         obj_free(class->size, obj);
>
>         fullness = fix_fullness_group(class, zspage);
> -       if (fullness == ZS_INUSE_RATIO_0)
> -               free_zspage(pool, class, zspage);
> +       if (fullness == ZS_INUSE_RATIO_0) {
> +               if (trylock_zspage(zspage)) {
> +                       remove_zspage(class, zspage);
> +                       class_stat_sub(class, ZS_OBJS_ALLOCATED,
> +                               class->objs_per_zspage);
> +                       zspage_to_free = zspage;
> +               } else
> +                       kick_deferred_free(pool);
> +       }
>
>         spin_unlock(&class->lock);
> +
> +       if (likely(zspage_to_free)) {
> +               __free_zspage_lockless(pool, zspage_to_free);
> +               atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
> +       }

I think Joshua has a point here. Just plain "if (zspage_to_free)" for
simplicity?

>         cache_free_handle(handle);
>  }
>  EXPORT_SYMBOL_GPL(zs_free);
> --
> 2.34.1
>

Re: [RFC PATCH v2 3/3] mm/zsmalloc: drop class lock before freeing zspage

[Xueyuan Chen]

On Fri, May 29, 2026 at 03:59:50PM -0700, Nhat Pham wrote:

[...]

Hi Joshua, Nhat,

Sorry for the late reply, and thanks for the review.
You are right. I will update the code according to your suggestions in
the next version.

Thanks,
Xueyuan