* [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
We add a new @unit_size data member to struct acomp_req along with a
helper function acomp_request_set_unit_size() for kernel modules to set
the unit size to use while breaking down the request's src/dst
scatterlists.
An acomp_alg can implement batching by using the @req->unit_size to
break down the SG lists passed in via @req->dst and/or @req->src, to
submit individual @req->slen/@req->unit_size compress jobs or
@req->dlen/@req->unit_size decompress jobs, for batch compression and
batch decompression respectively.
In case of batch compression, the folio's pages for the batch can be
retrieved from the @req->src scatterlist by using a struct sg_page_iter
after determining the number of pages as @req->slen/@req->unit_size.
Please note that acomp_request_set_callback() sets the @req->unit_size
to 0.
In zswap_acomp_req_init(), after the call to
zswap_set_acomp_req_callback(), we set the unit size for zswap to
PAGE_SIZE.
acomp_request_set_unit_size(areq->req, PAGE_SIZE);
Suggested-by: Herbert Xu <herbert@gondor.apana.org.au>
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
include/crypto/acompress.h | 49 ++++++++++++++++++++++++++++++++++++++
mm/zswap.c | 6 +++++
2 files changed, 55 insertions(+)
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 5d5358dfab73..6462fc46f4ec 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -79,6 +79,7 @@ struct acomp_req_chain {
* @dvirt: Destination virtual address
* @slen: Size of the input buffer
* @dlen: Size of the output buffer and number of bytes produced
+ * @unit_size: Unit size for the request for use in batching
* @chain: Private API code data, do not use
* @__ctx: Start of private context data
*/
@@ -94,6 +95,7 @@ struct acomp_req {
};
unsigned int slen;
unsigned int dlen;
+ unsigned int unit_size;
struct acomp_req_chain chain;
@@ -329,9 +331,56 @@ static inline void acomp_request_set_callback(struct acomp_req *req,
{
flgs &= ~CRYPTO_ACOMP_REQ_PRIVATE;
flgs |= req->base.flags & CRYPTO_ACOMP_REQ_PRIVATE;
+ req->unit_size = 0;
crypto_request_set_callback(&req->base, flgs, cmpl, data);
}
+/**
+ * acomp_request_set_unit_size() -- Sets the unit size for the request.
+ *
+ * This is a helper function that enables batching for zswap, IPComp, etc.
+ * It allows multiple independent compression (or decompression) operations to
+ * be submitted in a single request's SG lists, where each SG list ("segment")
+ * is processed independently. The unit size helps derive segments from a
+ * single request. crypto_acomp does not expect the segments to be related in
+ * any way.
+ *
+ * Example usage model:
+ *
+ * A module such as zswap that's configured to use a batching compressor, can
+ * accomplish batch compression of "nr_pages" with crypto_acomp by creating an
+ * output SG table for the batch, initialized to contain "nr_pages" SG
+ * lists. Each scatterlist is mapped to the nth destination buffer for the
+ * batch. Depending on whether the @req is used for batch compress/decompress,
+ * zswap must set the @req's source/destination length to be
+ * "nr_pages * @req->unit_size" respectively.
+ *
+ * An acomp_alg can implement batch compression by using the @req->unit_size
+ * to break down the SG lists passed in via @req->dst to submit individual
+ * "@req->slen/@req->unit_size" compress jobs to be processed as a batch.
+ *
+ * Similarly, zswap can implement batch decompression by passing an
+ * SG table with "nr_pages" SG lists via @req->src to process
+ * "@req->dlen/@req->unit_size" decompress jobs as a batch.
+ *
+ * This API must be called after acomp_request_set_callback(),
+ * which sets @req->unit_size to 0. This makes it easy for users of
+ * crypto_acomp to rely on a default of not opting in to batching.
+ * Users such as zswap opt in to batching by defining @req->unit_size
+ * to a non-zero value for use by acomp_algs supporting batching.
+ *
+ * @du would be PAGE_SIZE for zswap, it could be the MTU for IPsec.
+ *
+ * @req: asynchronous compress/decompress request
+ * @du: data unit size of the input/output buffer scatterlist.
+ */
+static inline void acomp_request_set_unit_size(struct acomp_req *req,
+ unsigned int du)
+{
+ req->unit_size = du;
+}
+
+
/**
* acomp_request_set_params() -- Sets request parameters
*
diff --git a/mm/zswap.c b/mm/zswap.c
index ad01c691a263..e1d57a7d9595 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -826,6 +826,12 @@ static int zswap_acomp_req_init(struct zswap_acomp_req *areq,
crypto_init_wait(&areq->wait);
zswap_set_acomp_req_callback(areq->req, &areq->wait);
+ /*
+ * This should be called after acomp_request_set_callback(),
+ * which initializes req->unit_size to 0.
+ */
+ acomp_request_set_unit_size(areq->req, PAGE_SIZE);
+
mutex_init(&areq->mutex);
return 0;
}
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 2/8] crypto: acomp - Add bit to indicate segmentation support.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This patch adds segmentation support for compression.
Add a bit to the crypto_alg flags to indicate support for segmentation.
Also add a helper for acomp to test whether a given tfm supports
segmentation.
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
include/crypto/algapi.h | 5 +++++
include/crypto/internal/acompress.h | 5 +++++
include/linux/crypto.h | 3 +++
3 files changed, 13 insertions(+)
diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index 05deea9dac5e..7d406cfe5751 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -280,6 +280,11 @@ static inline bool crypto_tfm_req_virt(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_VIRT;
}
+static inline bool crypto_tfm_req_seg(struct crypto_tfm *tfm)
+{
+ return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_SEG;
+}
+
static inline u32 crypto_request_flags(struct crypto_async_request *req)
{
return req->flags & ~CRYPTO_TFM_REQ_ON_STACK;
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 9cd37df32dc4..93ae59623f13 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -189,6 +189,11 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)
return crypto_tfm_req_virt(&tfm->base);
}
+static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
+{
+ return crypto_tfm_req_seg(&tfm->base);
+}
+
void crypto_acomp_free_streams(struct crypto_acomp_streams *s);
int crypto_acomp_alloc_streams(struct crypto_acomp_streams *s);
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index b7c97f1c47c9..78c699ac3f82 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -139,6 +139,9 @@
/* Set if the algorithm cannot have a fallback (e.g., phmac). */
#define CRYPTO_ALG_NO_FALLBACK 0x00080000
+/* Set if the algorithm supports segmentation. */
+#define CRYPTO_ALG_REQ_SEG 0x00100000
+
/* The high bits 0xff000000 are reserved for type-specific flags. */
/*
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This patch provides a wrapper for existing algorithms so that they can
accept a single segment while returning the compressed length or error
through the dst SG list length.
This trivial segmentation wrapper only supports compression with a
segment count of exactly one.
The reason is that the first user zswap will only allocate the
extra memory if the underlying algorithm supports segmentation,
and otherwise only one segment will be given at a time.
Having this wrapper means that the same calling convention can
be used for all algorithms, regardless of segmentation support.
To gain latency in the zswap_store() hot path impacted by this patch:
1) The most common code path is invoked first in
crypto_acomp_compress().
2) The helper functions used in (1) are made __always_inline.
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
crypto/acompress.c | 39 ++++++++++++++++++++++++-----
include/crypto/acompress.h | 3 ++-
include/crypto/internal/acompress.h | 4 +--
include/linux/crypto.h | 2 +-
4 files changed, 38 insertions(+), 10 deletions(-)
diff --git a/crypto/acompress.c b/crypto/acompress.c
index c8b4b8589769..c55b0e8eff3d 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -166,14 +166,20 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt)
state->compl = req->base.complete;
state->data = req->base.data;
+ state->unit_size = req->unit_size;
+ state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
+ CRYPTO_ACOMP_REQ_DST_VIRT);
+
req->base.complete = cplt;
req->base.data = req;
+ req->unit_size = 0;
}
static void acomp_restore_req(struct acomp_req *req)
{
req->base.complete = req->chain.compl;
req->base.data = req->chain.data;
+ req->unit_size = req->chain.unit_size;
}
static void acomp_reqchain_virt(struct acomp_req *req)
@@ -192,9 +198,6 @@ static void acomp_virt_to_sg(struct acomp_req *req)
{
struct acomp_req_chain *state = &req->chain;
- state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
- CRYPTO_ACOMP_REQ_DST_VIRT);
-
if (acomp_request_src_isvirt(req)) {
unsigned int slen = req->slen;
const u8 *svirt = req->svirt;
@@ -242,6 +245,10 @@ static int acomp_reqchain_finish(struct acomp_req *req, int err)
{
acomp_reqchain_virt(req);
acomp_restore_req(req);
+
+ if (req->unit_size)
+ req->dst->length = unlikely(err) ? err : req->dlen;
+
return err;
}
@@ -266,10 +273,13 @@ static int acomp_do_req_chain(struct acomp_req *req, bool comp)
{
int err;
+ if (unlikely(req->unit_size && req->slen > req->unit_size))
+ return -ENOSYS;
+
acomp_save_req(req, acomp_reqchain_done);
err = acomp_do_one_req(req, comp);
- if (err == -EBUSY || err == -EINPROGRESS)
+ if (unlikely(err == -EBUSY || err == -EINPROGRESS))
return err;
return acomp_reqchain_finish(req, err);
@@ -279,10 +289,27 @@ int crypto_acomp_compress(struct acomp_req *req)
{
struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
+ if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
+ !crypto_acomp_req_seg(tfm) && !acomp_req_on_stack(req))) {
+ int err = tfm->compress(req);
+
+ req->dst->length = unlikely(err) ? err : req->dlen;
+ return err;
+ }
+
if (acomp_req_on_stack(req) && acomp_is_async(tfm))
return -EAGAIN;
+
+ if (req->unit_size && acomp_request_issg(req)) {
+ if (!crypto_acomp_req_seg(tfm))
+ return acomp_do_req_chain(req, true);
+
+ return tfm->compress(req);
+ }
+
if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
- return crypto_acomp_reqtfm(req)->compress(req);
+ return tfm->compress(req);
+
return acomp_do_req_chain(req, true);
}
EXPORT_SYMBOL_GPL(crypto_acomp_compress);
@@ -294,7 +321,7 @@ int crypto_acomp_decompress(struct acomp_req *req)
if (acomp_req_on_stack(req) && acomp_is_async(tfm))
return -EAGAIN;
if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
- return crypto_acomp_reqtfm(req)->decompress(req);
+ return tfm->decompress(req);
return acomp_do_req_chain(req, false);
}
EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 6462fc46f4ec..e64cf422510b 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -67,6 +67,7 @@ struct acomp_req_chain {
struct folio *dfolio;
};
u32 flags;
+ u32 unit_size;
};
/**
@@ -294,7 +295,7 @@ static inline void *acomp_request_extra(struct acomp_req *req)
return (void *)((char *)req + len);
}
-static inline bool acomp_req_on_stack(struct acomp_req *req)
+static __always_inline bool acomp_req_on_stack(struct acomp_req *req)
{
return crypto_req_on_stack(&req->base);
}
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 93ae59623f13..0c0d3fc8a9fc 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -146,7 +146,7 @@ void crypto_unregister_acomp(struct acomp_alg *alg);
int crypto_register_acomps(struct acomp_alg *algs, int count);
void crypto_unregister_acomps(struct acomp_alg *algs, int count);
-static inline bool acomp_request_issg(struct acomp_req *req)
+static __always_inline bool acomp_request_issg(struct acomp_req *req)
{
return !(req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
CRYPTO_ACOMP_REQ_DST_VIRT));
@@ -189,7 +189,7 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)
return crypto_tfm_req_virt(&tfm->base);
}
-static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
+static __always_inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
{
return crypto_tfm_req_seg(&tfm->base);
}
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 78c699ac3f82..0628b62e4946 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -492,7 +492,7 @@ static inline bool crypto_tfm_is_async(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_ASYNC;
}
-static inline bool crypto_req_on_stack(struct crypto_async_request *req)
+static __always_inline bool crypto_req_on_stack(struct crypto_async_request *req)
{
return req->flags & CRYPTO_TFM_REQ_ON_STACK;
}
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (2 preceding siblings ...)
2026-10-28 3:04 ` [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-28 3:05 ` [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This commit adds a @batch_size data member to struct acomp_alg.
An acomp_alg compression algorithm that supports batching of
compressions and decompressions must provide a @batch_size greater
than 1, representing the maximum batch-size that the compressor
supports, so that kernel users of crypto_acomp, such as zswap, can
allocate resources for submitting multiple compress/decompress jobs that
can be batched, and invoke batching of [de]compressions.
The new crypto_acomp_batch_size() API queries the crypto_acomp's
acomp_alg for the batch-size. If the acomp_alg has registered a
@batch_size greater than 1, this is returned. If not, a default of "1"
is returned.
zswap can invoke crypto_acomp_batch_size() to query the maximum number
of requests that can be batch [de]compressed. Based on this, zswap
can use the minimum of any zswap-specific upper limits for batch-size
and the compressor's max @batch_size, to allocate batching resources.
This enables zswap to compress/decompress multiple pages as a batch
to improve swapout/swapin performance and memory savings.
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
crypto/acompress.c | 14 ++++++++++++++
include/crypto/acompress.h | 12 ++++++++++++
include/crypto/internal/acompress.h | 3 +++
3 files changed, 29 insertions(+)
diff --git a/crypto/acompress.c b/crypto/acompress.c
index c55b0e8eff3d..8e285cb7477e 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -326,6 +326,20 @@ int crypto_acomp_decompress(struct acomp_req *req)
}
EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm)
+{
+ if (acomp_is_async(tfm) &&
+ (crypto_comp_alg_common(tfm)->base.cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS)) {
+ struct acomp_alg *alg = crypto_acomp_alg(tfm);
+
+ if (alg && alg->batch_size > 1)
+ return alg->batch_size;
+ }
+
+ return 1;
+}
+EXPORT_SYMBOL_GPL(crypto_acomp_batch_size);
+
void comp_prepare_alg(struct comp_alg_common *alg)
{
struct crypto_alg *base = &alg->base;
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index e64cf422510b..a71532ccb8dc 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -593,6 +593,18 @@ int crypto_acomp_compress(struct acomp_req *req);
*/
int crypto_acomp_decompress(struct acomp_req *req);
+/**
+ * crypto_acomp_batch_size() -- Get the algorithm's batch size
+ *
+ * Function returns the algorithm's batch size for batching operations
+ *
+ * @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp()
+ *
+ * Return: @tfm's acomp_alg's @batch_size, if it has defined a
+ * @batch_size greater than 1; else return 1.
+ */
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm);
+
static inline struct acomp_req *acomp_request_on_stack_init(
char *buf, struct crypto_acomp *tfm)
{
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 0c0d3fc8a9fc..2d4d302d1b3f 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -28,6 +28,8 @@
*
* @compress: Function performs a compress operation
* @decompress: Function performs a de-compress operation
+ * @batch_size: Maximum batch-size for batching compress/decompress
+ * operations.
* @init: Initialize the cryptographic transformation object.
* This function is used to initialize the cryptographic
* transformation object. This function is called only once at
@@ -47,6 +49,7 @@
struct acomp_alg {
int (*compress)(struct acomp_req *req);
int (*decompress)(struct acomp_req *req);
+ unsigned int batch_size;
int (*init)(struct crypto_acomp *tfm);
void (*exit)(struct crypto_acomp *tfm);
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (3 preceding siblings ...)
2026-10-28 3:04 ` [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
2026-10-28 3:05 ` [PATCH v2 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
When storing a folio in zswap, save the folio's node id in the zswap
entry instead of relying on the zswap_entry being allocated on the same
node. This enables optimizations such as batch allocation of zswap
entries for large folios using kmem_cache_alloc_bulk() that does not
support specifying a node id.
Accordingly, replace the SLUB allocation in zswap_entry_cache_alloc() to
be node-agnostic.
The size of the zswap_entry remains unchanged as 'referenced' is lumped
in with the 'length' (as it doesn't need a full unsigned int anyway).
A build time assertion checks for the zswap_entry's 'length' bitfield
being at least as large as the PAGE_SHIFT, which can be 18 on some
architectures.
Suggested-by: Nhat Pham <nphamcs@gmail.com>
Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 33 ++++++++++++++++++---------------
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index e1d57a7d9595..19b7d60bc7d2 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -180,6 +180,7 @@ struct zswap_pool {
*/
#define ZSWAP_FIRST_POOL_ID 1
#define ZSWAP_MAX_POOL_ID U8_MAX
+#define ZSWAP_MAX_COMPR_LENGTH_SHIFT 18U
static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH);
static struct zswap_pool __rcu *zswap_current_pool;
@@ -206,15 +207,19 @@ static struct shrinker *zswap_shrinker;
* logic if referenced is unset. See comments in the shrinker
* section for context.
* pool_idx - id of the zswap_pool that the entry's data is in.
+ * nid - NUMA node id of the page for which this is the zswap entry.
* handle - zsmalloc allocation handle that stores the compressed page data
* objcg - the obj_cgroup that the compressed memory is charged to
* lru - handle to the pool's lru used to evict pages.
*/
struct zswap_entry {
swp_entry_t swpentry;
- unsigned int length;
- bool referenced;
- u8 pool_idx;
+ struct {
+ unsigned int length:ZSWAP_MAX_COMPR_LENGTH_SHIFT;
+ bool referenced:1U;
+ u8 pool_idx;
+ };
+ int nid;
unsigned long handle;
struct obj_cgroup *objcg;
struct list_head lru;
@@ -679,14 +684,8 @@ static int zswap_enabled_param_set(const char *val,
* lru functions
**********************************/
-static inline int entry_to_nid(struct zswap_entry *entry)
-{
- return page_to_nid(virt_to_page(entry));
-}
-
static void zswap_lru_add(struct zswap_entry *entry)
{
- int nid = entry_to_nid(entry);
struct mem_cgroup *memcg;
/*
@@ -703,19 +702,18 @@ static void zswap_lru_add(struct zswap_entry *entry)
rcu_read_lock();
memcg = obj_cgroup_memcg(entry->objcg);
/* will always succeed */
- list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
+ list_lru_add(&zswap_list_lru, &entry->lru, entry->nid, memcg);
rcu_read_unlock();
}
static void zswap_lru_del(struct zswap_entry *entry)
{
- int nid = entry_to_nid(entry);
struct mem_cgroup *memcg;
rcu_read_lock();
memcg = obj_cgroup_memcg(entry->objcg);
/* will always succeed */
- list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg);
+ list_lru_del(&zswap_list_lru, &entry->lru, entry->nid, memcg);
rcu_read_unlock();
}
@@ -763,10 +761,11 @@ void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg)
**********************************/
static struct kmem_cache *zswap_entry_cache;
-static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid)
+static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp)
{
struct zswap_entry *entry;
- entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid);
+
+ entry = kmem_cache_alloc(zswap_entry_cache, gfp);
if (!entry)
return NULL;
return entry;
@@ -1507,9 +1506,10 @@ static bool zswap_store_page(struct folio *folio, long index,
{
swp_entry_t page_swpentry = folio_swap_entry(folio, index);
struct zswap_entry *entry, *old;
+ int nid = folio_nid(folio);
/* allocate entry */
- entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
+ entry = zswap_entry_cache_alloc(GFP_KERNEL);
if (!entry) {
zswap_reject_kmemcache_fail++;
return false;
@@ -1524,6 +1524,7 @@ static bool zswap_store_page(struct folio *folio, long index,
* to an unrelated live pool.
*/
entry->pool_idx = pool->idx;
+ entry->nid = nid;
old = xa_store(swap_zswap_tree(page_swpentry),
swp_offset(page_swpentry),
@@ -1897,6 +1898,8 @@ static int zswap_setup(void)
struct zswap_pool *pool;
int ret;
+ BUILD_BUG_ON(ZSWAP_MAX_COMPR_LENGTH_SHIFT < PAGE_SHIFT);
+
zswap_entry_cache = KMEM_CACHE(zswap_entry, 0);
if (!zswap_entry_cache) {
pr_err("entry cache creation failed\n");
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 6/8] mm: zswap: Store large folios in batches.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (4 preceding siblings ...)
2026-10-28 3:05 ` [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
2026-10-28 3:05 ` [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing " Kanchana P. Sridhar
2026-10-28 3:05 ` [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
Use batching when storing large folios in zswap. If the underlying
compressor supports batching (e.g. hardware parallel compression),
allocate multiple compression buffers, otherwise allocate one. The
number of buffers is bounded by a new constant, ZSWAP_MAX_BATCH_SIZE, to
limit the memory overhead. For existing software compressors, the only
extra overhead is the extra 'buffers' pointer, so 8 bytes per-CPU on
x86_64. Only the first buffer is currently used.
Regardless of compression batching, always process large folios in
batches. For hardware compressors, the batch size is the compressor
batch size, otherwise ZSWAP_MAX_BATCH_SIZE is used.
zswap_store_page() is replaced with zswap_store_pages(), which processes
a batch of pages and allows for batching optimizations. For now, only
optimize allocating entries by using batch allocations from the slab
cache.
Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
page and only call it once for the folio, since the entire folio is
charged to a single memcg. Similarly, obtain the nid once to use during
each batch store of a folio. To facilitate this, a
struct zswap_store_context is introduced to keep function parameter
lists concise.
Also, place and align the members of struct zswap_pool such that members
used together in compress/decompress hot paths are able to share the same
cacheline. Specifically, these members are now on the same cacheline:
struct zswap_pool {
struct zs_pool *zs_pool;
struct crypto_acomp_ctx __percpu *acomp_ctx;
struct percpu_ref ref;
u8 idx;
u8 compr_batch_size;
// ...
};
The size of struct zswap_pool is unchanged.
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 323 +++++++++++++++++++++++++++++++++++++----------------
1 file changed, 227 insertions(+), 96 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 19b7d60bc7d2..c8890e8a9b83 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -84,6 +84,9 @@ static bool zswap_pool_reached_full;
#define ZSWAP_PARAM_UNSET ""
+/* Limit the batch size to limit per-CPU memory usage for dst buffers. */
+#define ZSWAP_MAX_BATCH_SIZE 8U
+
static int zswap_setup(void);
/* Enable/disable zswap */
@@ -143,10 +146,10 @@ struct zswap_acomp_req {
struct mutex mutex;
};
-/* The compression mutex also protects the output buffer. */
+/* The compression mutex also protects the output buffer(s). */
struct zswap_comp_ctx {
struct zswap_acomp_req areq;
- u8 *buffer;
+ u8 **buffers;
};
/* Separate requests, so that decompression does not wait for compression. */
@@ -161,14 +164,18 @@ struct crypto_acomp_ctx {
* The only case where lru_lock is not acquired while holding tree.lock is
* when a zswap_entry is taken off the lru for writeback, in that case it
* needs to be verified that it's still valid in the tree.
+ *
+ * @compr_batch_size: The max batch size of the compression algorithm,
+ * bounded by ZSWAP_MAX_BATCH_SIZE.
*/
struct zswap_pool {
struct zs_pool *zs_pool;
struct crypto_acomp_ctx __percpu *acomp_ctx;
struct percpu_ref ref;
- struct rcu_work release_rwork;
- struct hlist_node node;
u8 idx;
+ u8 compr_batch_size;
+ struct hlist_node node;
+ struct rcu_work release_rwork;
char tfm_name[CRYPTO_MAX_ALG_NAME];
};
@@ -225,6 +232,15 @@ struct zswap_entry {
struct list_head lru;
};
+struct zswap_store_context {
+ struct folio *folio;
+ long folio_start;
+ bool wb_enabled;
+ int nid;
+ u8 nr_pages;
+ struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
+};
+
/*
* No RCU section is needed around the returned pointer: a stored entry pins
* its pool via percpu_ref (taken in zswap_store_page()), so the id cannot be
@@ -281,8 +297,10 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
**********************************/
static void __zswap_pool_empty(struct percpu_ref *ref);
-static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
+static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx, u8 nr_buffers)
{
+ u8 i;
+
if (!acomp_ctx)
return;
@@ -301,8 +319,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
acomp_ctx->acomp = NULL;
- kfree(acomp_ctx->comp.buffer);
- acomp_ctx->comp.buffer = NULL;
+ if (acomp_ctx->comp.buffers) {
+ for (i = 0; i < nr_buffers; ++i)
+ kfree(acomp_ctx->comp.buffers[i]);
+ kfree(acomp_ctx->comp.buffers);
+ }
+ acomp_ctx->comp.buffers = NULL;
}
static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -386,7 +408,9 @@ static struct zswap_pool *zswap_pool_create(char *compressor)
cpuhp_add_fail:
for_each_possible_cpu(cpu)
- acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
+ acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu),
+ pool->compr_batch_size);
+
error:
if (pool->acomp_ctx)
free_percpu(pool->acomp_ctx);
@@ -424,7 +448,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool)
cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node);
for_each_possible_cpu(cpu)
- acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
+ acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu),
+ pool->compr_batch_size);
free_percpu(pool->acomp_ctx);
@@ -776,6 +801,39 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
kmem_cache_free(zswap_entry_cache, entry);
}
+static inline void zswap_entries_cache_free_batch(
+ struct zswap_entry **entries,
+ u8 nr_entries)
+{
+ /*
+ * It is okay to use this to free entries allocated separately
+ * by zswap_entry_cache_alloc().
+ */
+ kmem_cache_free_bulk(zswap_entry_cache, nr_entries, (void **)entries);
+}
+
+static bool zswap_entries_cache_alloc_batch(
+ struct zswap_store_context *zc,
+ gfp_t gfp)
+{
+ if (unlikely(!kmem_cache_alloc_bulk(zswap_entry_cache, gfp,
+ zc->nr_pages, (void **)zc->entries))) {
+ u8 i;
+
+ for (i = 0; i < zc->nr_pages; ++i) {
+ zc->entries[i] = zswap_entry_cache_alloc(GFP_KERNEL);
+
+ if (unlikely(!zc->entries[i])) {
+ zswap_reject_kmemcache_fail++;
+ zswap_entries_cache_free_batch(zc->entries, i);
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
/*
* Carries out the common pattern of freeing an entry's zsmalloc allocation,
* freeing the entry itself, and decrementing the number of stored pages.
@@ -839,7 +897,9 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
{
struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
+ int nid = cpu_to_node(cpu);
int ret = -ENOMEM;
+ u8 i;
/*
* To handle cases where the CPU goes through online-offline-online
@@ -850,15 +910,11 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
return 0;
}
- acomp_ctx->comp.buffer = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu));
- if (!acomp_ctx->comp.buffer)
- return ret;
-
/*
* In case of an error, crypto_alloc_acomp_node() returns an
* error pointer, never NULL.
*/
- acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu));
+ acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, nid);
if (IS_ERR(acomp_ctx->acomp)) {
pr_err("could not alloc crypto acomp %s : %pe\n",
pool->tfm_name, acomp_ctx->acomp);
@@ -866,10 +922,28 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
goto fail;
}
+ /*
+ * Allocate up to ZSWAP_MAX_BATCH_SIZE dst buffers if the
+ * compressor supports batching.
+ */
+ pool->compr_batch_size = min(ZSWAP_MAX_BATCH_SIZE,
+ crypto_acomp_batch_size(acomp_ctx->acomp));
+
ret = zswap_acomp_req_init(&acomp_ctx->comp.areq, acomp_ctx->acomp);
if (ret)
goto req_fail;
+ acomp_ctx->comp.buffers = kcalloc_node(pool->compr_batch_size, sizeof(u8 *),
+ GFP_KERNEL, nid);
+ if (!acomp_ctx->comp.buffers)
+ goto req_fail;
+
+ for (i = 0; i < pool->compr_batch_size; ++i) {
+ acomp_ctx->comp.buffers[i] = kmalloc_node(PAGE_SIZE, GFP_KERNEL, nid);
+ if (!acomp_ctx->comp.buffers[i])
+ goto req_fail;
+ }
+
/* Skip the per-CPU request when decompression can use the stack. */
if (acomp_is_async(acomp_ctx->acomp)) {
ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -882,12 +956,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
req_fail:
pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name);
fail:
- acomp_ctx_free(acomp_ctx);
+ acomp_ctx_free(acomp_ctx, pool->compr_batch_size);
return ret;
}
static bool zswap_compress(struct folio *folio, long index,
- struct zswap_entry *entry, struct zswap_pool *pool)
+ struct zswap_entry *entry, struct zswap_pool *pool,
+ bool wb_enabled)
{
struct zswap_comp_ctx *comp_ctx;
struct scatterlist input, output;
@@ -901,7 +976,7 @@ static bool zswap_compress(struct folio *folio, long index,
comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
mutex_lock(&comp_ctx->areq.mutex);
- dst = comp_ctx->buffer;
+ dst = comp_ctx->buffers[0];
sg_init_table(&input, 1);
sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
@@ -940,12 +1015,8 @@ static bool zswap_compress(struct folio *folio, long index,
* to the active LRU list in the case.
*/
if (comp_ret) {
- rcu_read_lock();
- if (!mem_cgroup_zswap_writeback_enabled(folio_memcg(folio))) {
- rcu_read_unlock();
+ if (!wb_enabled)
goto unlock;
- }
- rcu_read_unlock();
comp_ret = 0;
dlen = PAGE_SIZE;
dst = kmap_local_folio(folio, index * PAGE_SIZE);
@@ -1500,91 +1571,124 @@ static void shrink_worker(struct work_struct *w)
* main API
**********************************/
-static bool zswap_store_page(struct folio *folio, long index,
- struct obj_cgroup *objcg,
- struct zswap_pool *pool)
+/*
+ * Store multiple pages in @folio, starting from the page at index @start up to
+ * the page at index @end-1.
+ */
+static bool zswap_store_pages(struct zswap_store_context *zc,
+ long start,
+ long end,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
{
- swp_entry_t page_swpentry = folio_swap_entry(folio, index);
- struct zswap_entry *entry, *old;
- int nid = folio_nid(folio);
+ u8 i, store_fail_idx = 0;
- /* allocate entry */
- entry = zswap_entry_cache_alloc(GFP_KERNEL);
- if (!entry) {
- zswap_reject_kmemcache_fail++;
- return false;
- }
+ zc->nr_pages = end - start;
+ zc->folio_start = start;
- if (!zswap_compress(folio, index, entry, pool))
- goto compress_failed;
+ VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
+
+ if (unlikely(!zswap_entries_cache_alloc_batch(zc, GFP_KERNEL)))
+ return false;
/*
- * Set pool_idx before the xa_store() below publishes the entry, or a
- * concurrent reader could resolve a stale pool_idx left by slab reuse
- * to an unrelated live pool.
+ * We co-locate entry initialization as much as possible here to
+ * minimize potential cache misses.
*/
- entry->pool_idx = pool->idx;
- entry->nid = nid;
-
- old = xa_store(swap_zswap_tree(page_swpentry),
- swp_offset(page_swpentry),
- entry, GFP_KERNEL);
- if (xa_is_err(old)) {
- int err = xa_err(old);
+ for (i = 0; i < zc->nr_pages; ++i) {
+ zc->entries[i]->handle = (unsigned long)ERR_PTR(-EINVAL);
+ /*
+ * Set pool_idx before the xa_store() below publishes the entry, or a
+ * concurrent reader could resolve a stale pool_idx left by slab reuse
+ * to an unrelated live pool.
+ */
+ zc->entries[i]->pool_idx = pool->idx;
+ zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
+ zc->folio_start + i);
+ zc->entries[i]->objcg = objcg;
+ zc->entries[i]->referenced = true;
+ zc->entries[i]->nid = zc->nid;
+ INIT_LIST_HEAD(&zc->entries[i]->lru);
+ }
- WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
- zswap_reject_alloc_fail++;
- goto store_failed;
+ for (i = 0; i < zc->nr_pages; ++i) {
+ if (!zswap_compress(zc->folio, zc->folio_start + i,
+ zc->entries[i], pool, zc->wb_enabled))
+ goto store_pages_failed;
}
- /*
- * We may have had an existing entry that became stale when
- * the folio was redirtied and now the new version is being
- * swapped out. Get rid of the old.
- */
- if (old)
- zswap_entry_free(old);
+ for (i = 0; i < zc->nr_pages; ++i) {
+ struct zswap_entry *old, *entry = zc->entries[i];
+
+ old = xa_store(swap_zswap_tree(entry->swpentry),
+ swp_offset(entry->swpentry),
+ entry, GFP_KERNEL);
+ if (unlikely(xa_is_err(old))) {
+ int err = xa_err(old);
+
+ WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
+ zswap_reject_alloc_fail++;
+ /*
+ * Entries up to this point have been stored in the
+ * xarray. zswap_store() will erase them from the xarray
+ * and call zswap_entry_free(). Local cleanup in
+ * 'store_pages_failed' only needs to happen for
+ * entries from [@i to @nr_pages).
+ */
+ store_fail_idx = i;
+ goto store_pages_failed;
+ }
- /*
- * The entry is successfully compressed and stored in the tree, there is
- * no further possibility of failure. Grab refs to the pool and objcg,
- * charge zswap memory, and increment zswap_stored_pages.
- * The opposite actions will be performed by zswap_entry_free()
- * when the entry is removed from the tree.
- */
- zswap_pool_get(pool);
- if (objcg) {
- obj_cgroup_get(objcg);
- obj_cgroup_charge_zswap(objcg, entry->length);
- }
- atomic_long_inc(&zswap_stored_pages);
- if (entry->length == PAGE_SIZE)
- atomic_long_inc(&zswap_stored_incompressible_pages);
+ /*
+ * We may have had an existing entry that became stale when
+ * the folio was redirtied and now the new version is being
+ * swapped out. Get rid of the old.
+ */
+ if (unlikely(old))
+ zswap_entry_free(old);
- /*
- * We finish initializing the entry while it's already in xarray.
- * This is safe because:
- *
- * 1. Concurrent stores and invalidations are excluded by folio lock.
- *
- * 2. Writeback is excluded by the entry not being on the LRU yet.
- * The publishing order matters to prevent writeback from seeing
- * an incoherent entry.
- */
- entry->swpentry = page_swpentry;
- entry->objcg = objcg;
- entry->referenced = true;
- if (entry->length) {
- INIT_LIST_HEAD(&entry->lru);
- zswap_lru_add(entry);
+ /*
+ * The entry is successfully compressed and stored in the tree,
+ * and further failures will be cleaned up in zswap_store().
+ * Grab refs to the pool and objcg, charge zswap memory, and
+ * increment zswap_stored_pages. The opposite actions will be
+ * performed by zswap_entry_free() when the entry is removed
+ * from the tree.
+ */
+ zswap_pool_get(pool);
+ if (objcg) {
+ obj_cgroup_get(objcg);
+ obj_cgroup_charge_zswap(objcg, entry->length);
+ }
+ atomic_long_inc(&zswap_stored_pages);
+ if (unlikely(entry->length == PAGE_SIZE))
+ atomic_long_inc(&zswap_stored_incompressible_pages);
+
+ /*
+ * We finish by adding the entry to the LRU while it's already
+ * in xarray. This is safe because:
+ *
+ * 1. Concurrent stores and invalidations are excluded by folio lock.
+ *
+ * 2. Writeback is excluded by the entry not being on the LRU yet.
+ * The publishing order matters to prevent writeback from seeing
+ * an incoherent entry.
+ */
+ if (likely(entry->length))
+ zswap_lru_add(entry);
}
return true;
-store_failed:
- zs_free(pool->zs_pool, entry->handle);
-compress_failed:
- zswap_entry_cache_free(entry);
+store_pages_failed:
+ for (i = store_fail_idx; i < zc->nr_pages; ++i) {
+ if (!IS_ERR_VALUE(zc->entries[i]->handle))
+ zs_free(pool->zs_pool, zc->entries[i]->handle);
+ }
+
+ zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
+ zc->nr_pages - store_fail_idx);
+
return false;
}
@@ -1594,9 +1698,15 @@ bool zswap_store(struct folio *folio)
swp_entry_t swp = folio->swap;
struct obj_cgroup *objcg = NULL;
struct mem_cgroup *memcg = NULL;
- struct zswap_pool *pool;
+ int nid = folio_nid(folio);
bool ret = false;
- long index;
+ struct zswap_pool *pool;
+ u8 store_batch_size;
+ long start, end;
+ struct zswap_store_context zc = {
+ .folio = folio,
+ .nid = nid,
+ };
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1630,8 +1740,29 @@ bool zswap_store(struct folio *folio)
mem_cgroup_put(memcg);
}
- for (index = 0; index < nr_pages; ++index) {
- if (!zswap_store_page(folio, index, objcg, pool))
+ rcu_read_lock();
+ zc.wb_enabled = mem_cgroup_zswap_writeback_enabled(folio_memcg(folio));
+ rcu_read_unlock();
+
+ /*
+ * For batching compressors, store the folio in batches of the
+ * compressor's batch_size.
+ *
+ * For non-batching compressors, store the folio in batches
+ * of ZSWAP_MAX_BATCH_SIZE, where each page in the batch is
+ * compressed sequentially. This gives better performance than
+ * invoking zswap_store_pages() per-page, due to cache locality
+ * of working set structures.
+ */
+ store_batch_size = (pool->compr_batch_size > 1) ?
+ pool->compr_batch_size : ZSWAP_MAX_BATCH_SIZE;
+
+ for (start = 0; start < nr_pages; start += store_batch_size) {
+ end = min(start + store_batch_size, nr_pages);
+
+ ret = zswap_store_pages(&zc, start, end, pool, objcg);
+
+ if (unlikely(!ret))
goto put_pool;
}
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (5 preceding siblings ...)
2026-10-28 3:05 ` [PATCH v2 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
2026-10-28 3:05 ` [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
A new batching implementation of zswap_compress() is provided for
compressors that do and do not support batching. This eliminates code
duplication and facilitates code maintainability with the introduction
of compress batching.
The vectorized implementation of calling the earlier zswap_compress()
sequentially, one page at a time in zswap_store_pages(), is replaced
with this new version of zswap_compress() that accepts multiple pages to
compress as a batch.
If the compressor does not support batching, each page in the batch is
compressed and stored sequentially.
If the batch is compressed without errors, the compressed buffers for
the batch are stored in zsmalloc. In case of compression errors, the
current behavior based on whether the folio is enabled for zswap
writeback, is preserved.
The batched zswap_compress() incorporates Herbert's suggestion for
SG lists to represent the batch's inputs/outputs to interface with the
crypto API [1].
The struct zswap_store_context is extended to facilitate passing
batching related constraints and data to zswap_compress(). The code that
allocates zsmalloc memory to store compression results is refactored
into a new routine, zswap_update_entries(). Subsequent patches will be
submitted to further optimize this phase by replacing zsmalloc with
the SLUB allocator to improve zswap_store()/zswap_load() latencies.
[1]: https://lore.kernel.org/all/aJ7Fk6RpNc815Ivd@gondor.apana.org.au/T/#m99aea2ce3d284e6c5a3253061d97b08c4752a798
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 271 +++++++++++++++++++++++++++++++++++++----------------
1 file changed, 189 insertions(+), 82 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index c8890e8a9b83..4476fb4aeb5f 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -150,6 +150,7 @@ struct zswap_acomp_req {
struct zswap_comp_ctx {
struct zswap_acomp_req areq;
u8 **buffers;
+ struct sg_table *sg_table;
};
/* Separate requests, so that decompression does not wait for compression. */
@@ -235,9 +236,13 @@ struct zswap_entry {
struct zswap_store_context {
struct folio *folio;
long folio_start;
+ unsigned int slen;
+ u8 nr_batch_pages;
bool wb_enabled;
int nid;
u8 nr_pages;
+ struct zs_pool *zs_pool;
+ struct zswap_comp_ctx *comp_ctx;
struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
};
@@ -325,6 +330,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx, u8 nr_buffers)
kfree(acomp_ctx->comp.buffers);
}
acomp_ctx->comp.buffers = NULL;
+
+ if (acomp_ctx->comp.sg_table) {
+ sg_free_table(acomp_ctx->comp.sg_table);
+ kfree(acomp_ctx->comp.sg_table);
+ }
+ acomp_ctx->comp.sg_table = NULL;
}
static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -898,6 +909,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
int nid = cpu_to_node(cpu);
+ struct scatterlist *sg;
int ret = -ENOMEM;
u8 i;
@@ -944,6 +956,21 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
goto req_fail;
}
+ acomp_ctx->comp.sg_table = kmalloc(sizeof(*acomp_ctx->comp.sg_table), GFP_KERNEL);
+ if (!acomp_ctx->comp.sg_table)
+ goto req_fail;
+
+ if (sg_alloc_table(acomp_ctx->comp.sg_table, pool->compr_batch_size,
+ GFP_KERNEL))
+ goto req_fail;
+
+ /*
+ * Statically map the per-CPU destination buffers to the per-CPU
+ * SG lists; crucial for performance.
+ */
+ for_each_sg(acomp_ctx->comp.sg_table->sgl, sg, pool->compr_batch_size, i)
+ sg_set_buf(sg, acomp_ctx->comp.buffers[i], PAGE_SIZE);
+
/* Skip the per-CPU request when decompression can use the stack. */
if (acomp_is_async(acomp_ctx->acomp)) {
ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -960,92 +987,161 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
return ret;
}
-static bool zswap_compress(struct folio *folio, long index,
- struct zswap_entry *entry, struct zswap_pool *pool,
- bool wb_enabled)
+static bool zswap_update_entries(struct zswap_store_context *zc,
+ u8 batch_start)
{
- struct zswap_comp_ctx *comp_ctx;
- struct scatterlist input, output;
- int comp_ret = 0, alloc_ret = 0;
- unsigned int dlen = PAGE_SIZE;
+ gfp_t gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
+ u8 i, batch_iter, comp_batch_iter;
unsigned long handle;
- gfp_t gfp;
+ unsigned int dlen;
u8 *dst;
- bool mapped = false;
-
- comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
- mutex_lock(&comp_ctx->areq.mutex);
-
- dst = comp_ctx->buffers[0];
- sg_init_table(&input, 1);
- sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
-
- sg_init_one(&output, dst, PAGE_SIZE);
- acomp_request_set_params(comp_ctx->areq.req, &input, &output,
- PAGE_SIZE, dlen);
/*
- * it maybe looks a little bit silly that we send an asynchronous request,
- * then wait for its completion synchronously. This makes the process look
- * synchronous in fact.
- * Theoretically, acomp supports users send multiple acomp requests in one
- * acomp instance, then get those requests done simultaneously. but in this
- * case, zswap actually does store and load page by page, there is no
- * existing method to send the second page before the first page is done
- * in one thread doing zswap.
- * but in different threads running on different cpu, we have different
- * acomp instance, and compression and decompression use separate
- * requests, so multiple threads can do (de)compression in parallel.
+ * For each output SG list in @acomp_ctx->req->sg_table->sgl,
+ * the @sg->length should be set to either the page's compressed
+ * length (success), or it's negative value compression error.
*/
- comp_ret = crypto_wait_req(crypto_acomp_compress(comp_ctx->areq.req),
- &comp_ctx->areq.wait);
- dlen = comp_ctx->areq.req->dlen;
- if (!comp_ret) {
- if (dlen == PAGE_SIZE)
- comp_ret = -ENOSPC;
- else if (WARN_ON_ONCE(!dlen || dlen > PAGE_SIZE))
- comp_ret = -EINVAL;
+ for (comp_batch_iter = 0; comp_batch_iter < zc->nr_batch_pages;
+ ++comp_batch_iter) {
+ batch_iter = batch_start + comp_batch_iter;
+ dlen = zc->entries[batch_iter]->length;
+ handle = zs_malloc(zc->zs_pool, dlen, gfp, zc->nid);
+
+ if (unlikely(IS_ERR_VALUE(handle))) {
+ if (PTR_ERR((void *)handle) == -ENOSPC)
+ zswap_reject_compress_poor++;
+ else
+ zswap_reject_alloc_fail++;
+
+ for (i = 0; i < comp_batch_iter; ++i)
+ zs_free(zc->zs_pool, zc->entries[batch_start + i]->handle);
+
+ return false;
+ }
+
+ zc->entries[batch_iter]->handle = handle;
+
+ if (likely(dlen < PAGE_SIZE)) {
+ dst = zc->comp_ctx->buffers[comp_batch_iter];
+ zs_obj_write(zc->zs_pool, handle, dst, dlen);
+ } else {
+ dst = kmap_local_folio(zc->folio,
+ (zc->folio_start + batch_iter) * PAGE_SIZE);
+ zs_obj_write(zc->zs_pool, handle, dst, dlen);
+ kunmap_local(dst);
+ }
}
+ return true;
+}
+
+/*
+ * zswap_compress() batching implementation for sequential and batching
+ * compressors.
+ *
+ * Description:
+ * ============
+ * Compress multiple @nr_pages in @folio starting from the @folio_start index
+ * in batches of @nr_batch_pages; the latter representing the lesser of the
+ * @nr_pages to be batch compressed and the compressor's batch size.
+ *
+ * @nr_pages can be in (1, ZSWAP_MAX_BATCH_SIZE] even if the compressor does not
+ * support batching.
+ *
+ * If @nr_batch_pages is 1, each page is processed sequentially.
+ *
+ * If @nr_batch_pages is > 1, the compressor may use batching to optimize
+ * compression.
+ */
+static bool zswap_compress(struct zswap_store_context *zc)
+{
+ u8 batch_start, comp_batch_iter, batch_iter;
+ struct scatterlist *sg;
+ bool ret;
+ int err;
+
/*
- * If a page cannot be compressed into a size smaller than PAGE_SIZE,
- * save the content as is without a compression, to keep the LRU order
- * of writebacks. If writeback is disabled, reject the page since it
- * only adds metadata overhead. swap_writeout() will put the page back
- * to the active LRU list in the case.
+ * Compress the @nr_pages in @folio starting at index @folio_start
+ * in batches of @nr_batch_pages.
*/
- if (comp_ret) {
- if (!wb_enabled)
- goto unlock;
- comp_ret = 0;
- dlen = PAGE_SIZE;
- dst = kmap_local_folio(folio, index * PAGE_SIZE);
- mapped = true;
- }
+ for (batch_start = 0; batch_start < zc->nr_pages;
+ batch_start += zc->nr_batch_pages) {
+ /*
+ * Send @nr_batch_pages to crypto_acomp for compression:
+ *
+ * These pages are in @folio's range of indices in the interval
+ * [@folio_start + @batch_start,
+ * @folio_start + @batch_start + @nr_batch_pages).
+ *
+ * @slen indicates the total source length bytes for
+ * @nr_batch_pages.
+ *
+ * The source folio pages in the batch are directly submitted to
+ * crypto_acomp via acomp_request_set_src_folio().
+ *
+ * The pool's compressor batch size is at least @nr_batch_pages,
+ * hence the acomp_ctx has at least @nr_batch_pages dst @buffers.
+ */
+ acomp_request_set_src_folio(zc->comp_ctx->areq.req, zc->folio,
+ (zc->folio_start + batch_start) * PAGE_SIZE,
+ zc->slen);
+
+ zc->comp_ctx->sg_table->sgl->length = zc->slen;
+
+ /*
+ * The per-CPU @zc->comp_ctx->sg_table scatterlists are statically
+ * mapped to the per-CPU dst @buffers at pool creation time.
+ */
+ acomp_request_set_dst_sg(zc->comp_ctx->areq.req,
+ zc->comp_ctx->sg_table->sgl, zc->slen);
+
+ err = crypto_wait_req(crypto_acomp_compress(zc->comp_ctx->areq.req),
+ &zc->comp_ctx->areq.wait);
- gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
- handle = zs_malloc(pool->zs_pool, dlen, gfp, folio_nid(folio));
- if (IS_ERR_VALUE(handle)) {
- alloc_ret = PTR_ERR((void *)handle);
- goto unlock;
+ /*
+ * If a page cannot be compressed into a size smaller than
+ * PAGE_SIZE, save the content as is without a compression, to
+ * keep the LRU order of writebacks. If writeback is disabled,
+ * reject the page since it only adds metadata overhead.
+ * swap_writeout() will put the page back to the active LRU list
+ * in the case.
+ *
+ * It is assumed that any compressor that sets the output length
+ * to 0 or a value >= PAGE_SIZE will also return a negative
+ * error status in @err; i.e, will not return a successful
+ * compression status in @err in this case.
+ */
+ if (unlikely(err && !zc->wb_enabled))
+ goto compress_error;
+
+ for_each_sg(zc->comp_ctx->sg_table->sgl, sg,
+ zc->nr_batch_pages, comp_batch_iter) {
+ batch_iter = batch_start + comp_batch_iter;
+ if (unlikely(sg->length <= 0 || sg->length >= PAGE_SIZE))
+ sg->length = PAGE_SIZE;
+
+ zc->entries[batch_iter]->length = sg->length;
+ }
+
+ ret = zswap_update_entries(zc, batch_start);
+ if (unlikely(!ret))
+ return false;
}
- zs_obj_write(pool->zs_pool, handle, dst, dlen);
- entry->handle = handle;
- entry->length = dlen;
+ return true;
-unlock:
- if (mapped)
- kunmap_local(dst);
- if (comp_ret == -ENOSPC || alloc_ret == -ENOSPC)
- zswap_reject_compress_poor++;
- else if (comp_ret)
- zswap_reject_compress_fail++;
- else if (alloc_ret)
- zswap_reject_alloc_fail++;
-
- mutex_unlock(&comp_ctx->areq.mutex);
- return comp_ret == 0 && alloc_ret == 0;
+compress_error:
+ for_each_sg(zc->comp_ctx->sg_table->sgl, sg,
+ zc->nr_batch_pages, comp_batch_iter) {
+ if ((int)sg->length < 0) {
+ if ((int)sg->length == -ENOSPC)
+ zswap_reject_compress_poor++;
+ else
+ zswap_reject_compress_fail++;
+ }
+ }
+
+ return false;
}
static bool __zswap_decompress(struct zswap_entry *entry,
@@ -1582,6 +1678,7 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
struct obj_cgroup *objcg)
{
u8 i, store_fail_idx = 0;
+ bool ret;
zc->nr_pages = end - start;
zc->folio_start = start;
@@ -1596,7 +1693,6 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
* minimize potential cache misses.
*/
for (i = 0; i < zc->nr_pages; ++i) {
- zc->entries[i]->handle = (unsigned long)ERR_PTR(-EINVAL);
/*
* Set pool_idx before the xa_store() below publishes the entry, or a
* concurrent reader could resolve a stale pool_idx left by slab reuse
@@ -1611,11 +1707,19 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
INIT_LIST_HEAD(&zc->entries[i]->lru);
}
- for (i = 0; i < zc->nr_pages; ++i) {
- if (!zswap_compress(zc->folio, zc->folio_start + i,
- zc->entries[i], pool, zc->wb_enabled))
- goto store_pages_failed;
- }
+ /*
+ * Locking the comp_ctx mutex once per store batch results in better
+ * performance as compared to locking per compress batch.
+ */
+ zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
+ mutex_lock(&zc->comp_ctx->areq.mutex);
+
+ ret = zswap_compress(zc);
+
+ mutex_unlock(&zc->comp_ctx->areq.mutex);
+
+ if (unlikely(!ret))
+ goto compress_failed;
for (i = 0; i < zc->nr_pages; ++i) {
struct zswap_entry *old, *entry = zc->entries[i];
@@ -1681,11 +1785,10 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
return true;
store_pages_failed:
- for (i = store_fail_idx; i < zc->nr_pages; ++i) {
- if (!IS_ERR_VALUE(zc->entries[i]->handle))
- zs_free(pool->zs_pool, zc->entries[i]->handle);
- }
+ for (i = store_fail_idx; i < zc->nr_pages; ++i)
+ zs_free(pool->zs_pool, zc->entries[i]->handle);
+compress_failed:
zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
zc->nr_pages - store_fail_idx);
@@ -1757,6 +1860,10 @@ bool zswap_store(struct folio *folio)
store_batch_size = (pool->compr_batch_size > 1) ?
pool->compr_batch_size : ZSWAP_MAX_BATCH_SIZE;
+ zc.nr_batch_pages = pool->compr_batch_size;
+ zc.slen = zc.nr_batch_pages * PAGE_SIZE;
+ zc.zs_pool = pool->zs_pool;
+
for (start = 0; start < nr_pages; start += store_batch_size) {
end = min(start + store_batch_size, nr_pages);
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (6 preceding siblings ...)
2026-10-28 3:05 ` [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing " Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
7 siblings, 0 replies; 9+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This various stages of zswap_store_pages() are broken down into routines
that work on a batch of data when a large folio is being processed:
zswap_init_entries()
zswap_compress()
zswap_tree_stores()
zswap_account_stores()
The struct zswap_store_context is used to pass constraints and data
among these batching components. The resulting code is highly simplified
and maintainable, where each of the batching sub-routines of
zswap_store() can be optimized modularly.
For instance, the zswap_account_stores() now does the zswap atomics
counter increments, getting pool references, getting objcg references
and objcg zswap charging at the batch level.
Subsequent patch series will be submitted, that implement ideas building
upon this batching framework:
1) Replacing zsmalloc with SLUB allocator, comprising of kmem_caches
for a finite # of compressed size bins. This enables bulk allocation
of compressed buffers in a batch, that belong to the same size
bin. It gets rid of significant zsmalloc latency overhead (locks,
compressed data spanning zspages, etc.), without compromising memory
efficiency.
2) vmscan reclaim batching of hybrid any-order folios, that will submit
a batch of folios to zswap_store()'s batching framework to optimize
storing of batches of pages from different folios, and also within
large folios.
The batching framework introduced in this series improves zswap_store()
latency by a lot.
Performance data: usemem with 56% over-commit:
==============================================
Validation was done with mm-unstable as of 10-25-2026 (Baseline) and with this
patch series. To minimize variation in the data, I made these changes to my
performance measurement methodology:
1) Reduced the number of usemem processes to 2. Each process tried to allocate
2400M, in a 3G cgroup memory.high budget, effectively accomplishing a 56.25%
over-commit.
2) Increased the runtime of the workload to ~9 minutes by enabling PMD THP,
i.e., 2048kB mTHP folios.
3) The use of PMD THP forces zswap_store()/zswap_compress() latency to be in the
critical path, and thus enables very reliable measurements with bpftrace.
4) I used lhist() instead of hist() in bpftrace to gather latencies at a finer
level of granularity and accuracy.
Testing was done on a Chromebook with a QEMU "vng" VM running in Crostini.
Chromebook Diagnostics reports:
Intel(R) Core(TM) Ultra 5 115U (10 CPUs, 4.200GHz)
RAM: 16GB
vng setup:
Memory: 7G
Swap: 3G
Pinned CPUs: 2 (E-Cores: 2,3)
zswap compressor:
ZSTD
mTHP enabled:
2048kB (PMD THP)
I used the help of an LLM to write the bpftrace data post-processing scripts
that generate latency percentiles.
vm-scalability/usemem setup:
cgroup memory.high: 3G
2 usemem processes, each allocates 2400M anonymous memory, reads the
Silesia dataset [1] from a pre-fork-allocated buffer into the 2400M,
writes to each unsigned long in the 2400M, sleeps for 10 sec, then
frees the memory and exits.
usemem -w --read-silesia -O -b 1 -s 10 -n 2 2400M
usemem metrics are reported from a single run after starting the QEMU VM.
Further, I have tabulated the bpftrace p50/p95/p99 latencies with these labels:
"zswap_store_higher_order":
zswap_store() of PMD folios.
"zswap_store_order_0":
zswap_store() of order-0 folios.
"zswap_batch_compress_per_page":
zswap_compress() per-page latency of store batches generated from PMD
folios.
"zswap_single_compress_per_page":
zswap_compress() per-page latency of order-0 folios.
----------------------------------------------------------------------------
usemem Baseline zswap zswap Change with
PMD THP mm-unstable store compress compress
10-25-2026 batching batching batching
(patch 6) (patch 8) %
----------------------------------------------------------------------------
usemem process 1 6,353 7,240 7,428
Throughput (KB/s)
usemem process 2 6,352 7,236 7,422
Throughput (KB/s)
Average Throughput (KB/s) 6,353 7,238 7,425 17%
----------------------------------------------------------------------------
usemem process 1 484,580 308,903 257,230
Time to free memory (usecs)
usemem process 2 518,928 438,585 446,958
Time to free memory (usecs)
Average Time to 501,754 373,744 352,094 -30%
Free Memory (usecs)
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
usemem Baseline zswap zswap Change
PMD THP mm-unstable store compress with
BPFTRACE 10-25-2026 batching batching compress
(patch 6) (patch 8) batching
(ns)
-------------------------------------------------------------------------------
zswap_batch_compress_per_page (ns):
p50 40,086 1,569
p95 54,689 -64,505
p99 83,248 -88,488
-------------------------------------------------------------------------------
zswap_single_compress_per_page (ns):
p50 38,517 38,352 36,650 -1,867
p95 119,194 121,174 133,678 14,484
p99 171,736 158,601 169,010 -2,726
-------------------------------------------------------------------------------
zswap_store_higher_order (ns):
p50 21,012,222 21,322,885 20,176,667 -835,556
p95 25,643,500 26,204,375 24,485,000 -1,158,500
p99 28,567,400 38,577,750 26,158,000 -2,409,400
-------------------------------------------------------------------------------
zswap_store_order_0 (ns):
p50 37,601 36,994 37,568 -33
p95 123,731 137,034 128,366 4,635
p99 159,965 169,282 175,618 15,653
-------------------------------------------------------------------------------
vmstat summary:
-------------------------------------------------------------------------------
zswpin 1,104,226 993,412 1,091,146
zswpout 1,688,181 1,582,664 1,669,671
2048kB-alloc 2,500 2,505 2,500
2048kB-zswpout 1,535 1,535 1,535
2048kB-swpout-fallback 927 929 875
-------------------------------------------------------------------------------
zswap compress batching improves on both, zswap_store() latency and
zswap_compress() latency wrt the baseline, pretty consistently; except for p50
batch compress latency which can be optimized further in subsequent
patches. The batching framework established in this series allows us to make
locking and bulk-processing improvements in each stage of zswap_store(). One of
the ideas that has been showing very promising results is to replace zsmalloc
with SLUB compressed caches and will be submitted separately. This is expected
to resolve the p50 batch compress latency issue since it will be directly
optimizing zswap_update_entries().
The speedup with zswap store batching of large folios has a propelling effect
on other parts of mm, especially, by enabling large folios to be swapped out
without splitting, as the data shows. This enables other mm components that are
optimized for large folios, such as munmap(), folio refcounts/mapcounts
updates, PTE invalidation upon freeing memory, zswap_invalidate(); which
benefit from xarray offsets/range continuity. As a result, subsequent large
folio swapouts are more likely to find contiguous swap slots, there are fewer
large folio splits during reclaim and the cycle continues. As the data
demonstrates, this directly contributes to more memory over-commit
opportunity.
Architectural considerations for the zswap batching framework:
==============================================================
The zswap batching framework is designed to be hardware-agnostic.
It can be leveraged by any hardware accelerator or software-based
compressor.
Potential future clients of the batching framework:
===================================================
This patch-series demonstrates the performance benefits of compression
batching when used in zswap_store() of large folios. Compression
batching can be used for other use cases such as batching compression in
zram, batch compression of different folios during reclaim, kcompressd,
file systems, etc. Decompression batching can be used to improve
efficiency of zswap writeback (Thanks Nhat for this idea), batching
decompressions in zram, etc.
[1]: https://wanos.co/assets/silesia.tar
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 178 +++++++++++++++++++++++++++++++++--------------------
1 file changed, 112 insertions(+), 66 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 4476fb4aeb5f..2831284cfc90 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -504,12 +504,18 @@ static int __must_check zswap_pool_tryget(struct zswap_pool *pool)
}
/* The caller must already have a reference. */
-static void zswap_pool_get(struct zswap_pool *pool)
+static void __maybe_unused zswap_pool_get(struct zswap_pool *pool)
{
percpu_ref_get(&pool->ref);
}
-static void zswap_pool_put(struct zswap_pool *pool)
+/* The caller must already have a reference. */
+static inline void zswap_pool_get_many(struct zswap_pool *pool, u8 nr)
+{
+ percpu_ref_get_many(&pool->ref, nr);
+}
+
+static inline void zswap_pool_put(struct zswap_pool *pool)
{
percpu_ref_put(&pool->ref);
}
@@ -1668,25 +1674,13 @@ static void shrink_worker(struct work_struct *w)
**********************************/
/*
- * Store multiple pages in @folio, starting from the page at index @start up to
- * the page at index @end-1.
+ * Functions called by zswap_store(), that operate on batches of pages.
*/
-static bool zswap_store_pages(struct zswap_store_context *zc,
- long start,
- long end,
- struct zswap_pool *pool,
- struct obj_cgroup *objcg)
+static inline void zswap_init_entries(struct zswap_store_context *zc,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
{
- u8 i, store_fail_idx = 0;
- bool ret;
-
- zc->nr_pages = end - start;
- zc->folio_start = start;
-
- VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
-
- if (unlikely(!zswap_entries_cache_alloc_batch(zc, GFP_KERNEL)))
- return false;
+ u8 i;
/*
* We co-locate entry initialization as much as possible here to
@@ -1694,9 +1688,9 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
*/
for (i = 0; i < zc->nr_pages; ++i) {
/*
- * Set pool_idx before the xa_store() below publishes the entry, or a
- * concurrent reader could resolve a stale pool_idx left by slab reuse
- * to an unrelated live pool.
+ * Set pool_idx before the xa_store() in zswap_tree_stores()
+ * publishes the entry, or a concurrent reader could resolve a
+ * stale pool_idx left by slab reuse to an unrelated live pool.
*/
zc->entries[i]->pool_idx = pool->idx;
zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
@@ -1706,27 +1700,17 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
zc->entries[i]->nid = zc->nid;
INIT_LIST_HEAD(&zc->entries[i]->lru);
}
+}
- /*
- * Locking the comp_ctx mutex once per store batch results in better
- * performance as compared to locking per compress batch.
- */
- zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
- mutex_lock(&zc->comp_ctx->areq.mutex);
-
- ret = zswap_compress(zc);
-
- mutex_unlock(&zc->comp_ctx->areq.mutex);
-
- if (unlikely(!ret))
- goto compress_failed;
+static bool zswap_tree_stores(struct zswap_store_context *zc)
+{
+ struct zswap_entry *old;
+ u8 i, store_fail_idx = 0;
for (i = 0; i < zc->nr_pages; ++i) {
- struct zswap_entry *old, *entry = zc->entries[i];
-
- old = xa_store(swap_zswap_tree(entry->swpentry),
- swp_offset(entry->swpentry),
- entry, GFP_KERNEL);
+ old = xa_store(swap_zswap_tree(zc->entries[i]->swpentry),
+ swp_offset(zc->entries[i]->swpentry),
+ zc->entries[i], GFP_KERNEL);
if (unlikely(xa_is_err(old))) {
int err = xa_err(old);
@@ -1734,13 +1718,11 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
zswap_reject_alloc_fail++;
/*
* Entries up to this point have been stored in the
- * xarray. zswap_store() will erase them from the xarray
- * and call zswap_entry_free(). Local cleanup in
- * 'store_pages_failed' only needs to happen for
- * entries from [@i to @nr_pages).
+ * xarray. Local cleanup in 'tree_store_failed' needs to
+ * erase entries up until now.
*/
store_fail_idx = i;
- goto store_pages_failed;
+ goto tree_store_failed;
}
/*
@@ -1750,24 +1732,30 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
*/
if (unlikely(old))
zswap_entry_free(old);
+ }
- /*
- * The entry is successfully compressed and stored in the tree,
- * and further failures will be cleaned up in zswap_store().
- * Grab refs to the pool and objcg, charge zswap memory, and
- * increment zswap_stored_pages. The opposite actions will be
- * performed by zswap_entry_free() when the entry is removed
- * from the tree.
- */
- zswap_pool_get(pool);
- if (objcg) {
- obj_cgroup_get(objcg);
- obj_cgroup_charge_zswap(objcg, entry->length);
- }
- atomic_long_inc(&zswap_stored_pages);
- if (unlikely(entry->length == PAGE_SIZE))
+ return true;
+
+tree_store_failed:
+ for (i = 0; i < store_fail_idx; ++i)
+ xa_erase(swap_zswap_tree(zc->entries[i]->swpentry),
+ swp_offset(zc->entries[i]->swpentry));
+ return false;
+}
+
+static inline void zswap_account_stores(struct zswap_store_context *zc,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
+{
+ size_t total_batch_comp_length = 0;
+ u8 i;
+
+ for (i = 0; i < zc->nr_pages; ++i) {
+ if (unlikely(zc->entries[i]->length == PAGE_SIZE))
atomic_long_inc(&zswap_stored_incompressible_pages);
+ total_batch_comp_length += zc->entries[i]->length;
+
/*
* We finish by adding the entry to the LRU while it's already
* in xarray. This is safe because:
@@ -1778,20 +1766,78 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
* The publishing order matters to prevent writeback from seeing
* an incoherent entry.
*/
- if (likely(entry->length))
- zswap_lru_add(entry);
+ if (likely(zc->entries[i]->length))
+ zswap_lru_add(zc->entries[i]);
}
+ /*
+ * The entry is successfully compressed and stored in the tree,
+ * and further failures will be cleaned up in zswap_store().
+ * For the batch, grab refs to the pool and objcg, charge zswap memory,
+ * and increment zswap_stored_pages by @zc->nr_pages. The opposite
+ * actions will be performed by zswap_entry_free() when the entry is
+ * removed from the tree.
+ */
+ zswap_pool_get_many(pool, zc->nr_pages);
+ if (objcg) {
+ obj_cgroup_get_many(objcg, zc->nr_pages);
+ obj_cgroup_charge_zswap(objcg, total_batch_comp_length);
+ }
+ atomic_long_add(zc->nr_pages, &zswap_stored_pages);
+}
+
+/*
+ * Store multiple pages in @folio, starting from the page at index @start up to
+ * the page at index @end-1.
+ */
+static bool zswap_store_pages(struct zswap_store_context *zc,
+ long start,
+ long end,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
+{
+ bool ret;
+ u8 i;
+
+ zc->nr_pages = end - start;
+ zc->folio_start = start;
+
+ VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
+
+ if (unlikely(!zswap_entries_cache_alloc_batch(zc, GFP_KERNEL)))
+ return false;
+
+ zswap_init_entries(zc, pool, objcg);
+
+ /*
+ * Locking the comp_ctx mutex once per store batch results in better
+ * performance as compared to locking per compress batch.
+ */
+ zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
+ mutex_lock(&zc->comp_ctx->areq.mutex);
+
+ ret = zswap_compress(zc);
+
+ mutex_unlock(&zc->comp_ctx->areq.mutex);
+
+ if (unlikely(!ret))
+ goto compress_failed;
+
+ ret = zswap_tree_stores(zc);
+
+ if (unlikely(!ret))
+ goto store_pages_failed;
+
+ zswap_account_stores(zc, pool, objcg);
+
return true;
store_pages_failed:
- for (i = store_fail_idx; i < zc->nr_pages; ++i)
+ for (i = 0; i < zc->nr_pages; ++i)
zs_free(pool->zs_pool, zc->entries[i]->handle);
compress_failed:
- zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
- zc->nr_pages - store_fail_idx);
-
+ zswap_entries_cache_free_batch(zc->entries, zc->nr_pages);
return false;
}
--
2.39.5
^ permalink raw reply [flat|nested] 9+ messages in thread