From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CBAB296BBC for ; Fri, 31 Jul 2026 00:32:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785457985; cv=none; b=YiJnqW3EVT3vl5s6GJCJmaDDJOhGvqvK/gEuQxGPPk+Qvj2qyRkHKZ8u1Goc9bGNlbfsn7ae3HA4/ZEBaXVF//ef+THCYUTl9FKbfON7OPYuWD2mDa8+YE/Z0sJXFiPURrBj4HA7uGg2EC8i6ErmIL2QsH3qk2W1UBK+xoeSwOU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785457985; c=relaxed/simple; bh=4NODa6bKku6UKTwryqy2ttwHVO3xYBcmfcWp9LA8SrQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=cP/3U5lCHVs7GrPz1RcCOO1dtIlOsKWgzJeJefiZ0my2ZATzzWh1Qj9NypDjunFt5CVk3twclSNFghZ88Uz4IMl7/dg5ITv8gEtyD7hEdFVYvbOerGN/oPrrBiLRCoJfc4A4fl/ToGREFVGh6HrW0ILkkBZjgnS/LQNGZyLl2PE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PTv7aNAs; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PTv7aNAs" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2cf52d15d88so3241655ad.2 for ; Thu, 30 Jul 2026 17:32:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785457978; x=1786062778; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Pgl8j6vS3C7bSkoMmKnotkN6vZDR6xO4lGHnrMnaWEQ=; b=PTv7aNAsKkjOugWAwE1Q/MBZ3LGTkrWmVPNUbf6TLWM4Ygo0zbN9Wa97PhuJSMxIIb z0OP3IpbE8ugRVjXOiXpa0DEJ2IIjLlF+CO9K5/A/rs3CCUtYP2EiiQaHqWsIW7AfQ7C zbMvkhktU9mIFzpAeMzgNgRpdrOY8q6rg18ZimAvKb4s7IhGc5HaQieGebj1B6K6JZCY XDNvpv4iJA1WSLF3xxTHfGCqlHHr4z5P7h1GzoslSLBiUKtMjT3jpfmEIOkV0C+fxJGd +g5bWl25ujSYhjSGQz9+Y0eE0+wZp4y+KEX6JXKPaBT0KV8Bfe6zDkcvRAnn5TFOkN11 h/7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785457978; x=1786062778; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Pgl8j6vS3C7bSkoMmKnotkN6vZDR6xO4lGHnrMnaWEQ=; b=Lu3wCTuFz8VTLyBWEA4dEro2bxWe4q73SCFFtexlmbA5u8hgx6Oilga5TJHsrutR3l 9qoE/4BcP0xamxEabuWL8t3tFBna27uLoMAuPi6aHAnBJZxzK2X7ECMLznhuBgpl7iQF gwQBZ795I2zFm14PyiEdJthwhzOmUk/MVaAR+Aw6lfCuBTw0jxtfoOXhP/Sm4b37BWQo Rf9HnYNqaaOJ3WzWQg+ZMzcpt3R5xhhNy8YG4287EtoLLPQvk4JphMBvVbklsbMOBN/A EhL90SGptogRMzDWMepHbbnnvzUigifKJm5FYKdClSayrXplPcaDWhNkIA0uD/ocB3bB hxWQ== X-Gm-Message-State: AOJu0YxS/YgRKSPpHJdlG0GON3O5R0itW1Yv36EKnf0LHNu8/2YXwWR5 6InTxjjnzzEb8qCophCe9LyPYJtBTUfZP+YFtEqKWvpoiV14WZNgAu7k X-Gm-Gg: AR+sD10Zm6SgqzFel2UpCsLy5Wy0jQC0QedxIveXiHQgmBDrAXs0UsW0tRbm5zt5PJj HKGYKazN5UnQnXCc+s6ANChnwA7Wl0cDZCMGMOpQRDBC4JzOU2+GP6RpCLrjzkrMOe9ud4d5Nna hGJJA+QpC5B4wClAYy0FOOjlgmIQyZlpRrHqSVV4iYm2hTIWw5xSRSHCS8KHYrNN+seTEj63+d4 g8zz86397/tPUwG0i0ztn+6CHg/sG6qGLamRZgHxk/Z8utbLZ0QKsYvwFyB8uUV3fgz/HDXHDAc tqovwC9kg4B6MPAvd38XD/ilYLeT/bek3tXuUu5KZVKTqi8wkupGuZfGuOpBBxmAVz5lQV2m5q7 xgfjKyysq8Te7H63GpB0NHqPfQ65yZHlXuj0naUhnMjfRMUdJOta5Gt8fYrm0nGPoUH/B6COmoa 7gpG3JH8IgGVwfwcu54CuD/IvI/qbJoobCXNvJXJhapZRUurXmKchlNSfvmFxMmpUjYN6/3jvHk lmM85lz4Tmyr11CVCHTrwY83hOTesZiObuorLeCkqu8YKBH9pg= X-Received: by 2002:a17:90b:5903:b0:38d:adae:4866 with SMTP id 98e67ed59e1d1-38f9bece6e1mr3994144a91.21.1785457978178; Thu, 30 Jul 2026 17:32:58 -0700 (PDT) Received: from NV-J4GCB44.localdomain ([103.74.125.162]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-38f9b42d6c8sm1636301a91.3.2026.07.30.17.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jul 2026 17:32:57 -0700 (PDT) From: Jianyue Wu Date: Fri, 31 Jul 2026 08:32:47 +0800 Subject: [PATCH RFC v2 1/2] mm/zswap: replace the zswap_pools list with a fixed pools array Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260731-shrink_zswap_entry_v2-0-0-v2-1-e72083aa8734@gmail.com> References: <20260731-shrink_zswap_entry_v2-0-0-v2-0-e72083aa8734@gmail.com> In-Reply-To: <20260731-shrink_zswap_entry_v2-0-0-v2-0-e72083aa8734@gmail.com> To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Andrew Morton , Chris Li , Jianyue Wu X-Mailer: b4 0.13.0 Originally zswap holds its pools on an RCU list whose head also serves as the "current pool". Only a handful of pools are ever live at once, since a new pool is only created when the compressor is (re)set and pools are reused across compressor switches. A later change wants to store a reference to each entry's pool in every zswap_entry, where a pointer would cost 8 bytes but a small pool index only one. To make that index possible, the current change holds the pools in a fixed ZSWAP_MAX_POOLS-element array so each pool has a stable slot number, and tracks the current pool with a separate rcu-protected pointer. The array keeps the same RCU publish/retire discipline the list had, so lookup and teardown stay equivalent. Newly created pools are published into the array before they become current, so zswap_total_pages() can observe an empty pool briefly; that is harmless. This also caps the number of live pools at ZSWAP_MAX_POOLS (16), which is plenty in practice; pool creation warns and fails if the array ever fills, and the limit can be raised. Suggested-by: Nhat Pham Suggested-by: Yosry Ahmed Signed-off-by: Jianyue Wu --- mm/zswap.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 67 insertions(+), 21 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 4e76a4a87cdc..b203934d3be8 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -154,12 +154,20 @@ struct zswap_pool { struct zs_pool *zs_pool; struct crypto_acomp_ctx __percpu *acomp_ctx; struct percpu_ref ref; - struct list_head list; struct work_struct release_work; struct hlist_node node; + u8 idx; char tfm_name[CRYPTO_MAX_ALG_NAME]; }; +#define ZSWAP_MAX_POOLS 16 +static struct zswap_pool __rcu *zswap_pools[ZSWAP_MAX_POOLS]; +/* + * The current pool (NULL if none): an alias of one zswap_pools[] slot. It + * always holds a ref, so it is never retired from under us. + */ +static struct zswap_pool __rcu *zswap_current_pool; + /* Global LRU lists shared by all zswap pools. */ static struct list_lru zswap_list_lru; @@ -200,9 +208,7 @@ struct zswap_entry { static struct xarray *zswap_trees[MAX_SWAPFILES]; static unsigned int nr_zswap_trees[MAX_SWAPFILES]; -/* RCU-protected iteration */ -static LIST_HEAD(zswap_pools); -/* protects zswap_pools list modification */ +/* protects the zswap_pools array and zswap_current_pool */ static DEFINE_SPINLOCK(zswap_pools_lock); /* pool counter to provide unique names to zsmalloc */ static atomic_t zswap_pools_count = ATOMIC_INIT(0); @@ -270,6 +276,25 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx) acomp_ctx->buffer = NULL; } +static int zswap_pool_reserve_slot(struct zswap_pool *pool) +{ + int i, ret = -ENOSPC; + + spin_lock_bh(&zswap_pools_lock); + for (i = 0; i < ZSWAP_MAX_POOLS; i++) { + if (!rcu_access_pointer(zswap_pools[i])) { + /* Set idx before publishing so readers never see it stale. */ + pool->idx = i; + rcu_assign_pointer(zswap_pools[i], pool); + ret = i; + break; + } + } + spin_unlock_bh(&zswap_pools_lock); + + return ret; +} + static struct zswap_pool *zswap_pool_create(char *compressor) { struct zswap_pool *pool; @@ -313,19 +338,27 @@ static struct zswap_pool *zswap_pool_create(char *compressor) if (ret) goto cpuhp_add_fail; - /* being the current pool takes 1 ref; this func expects the - * caller to always add the new pool as the current pool + /* + * After a successful create, the caller makes this the current pool. + * If the caller fails, it kills the ref to free the reserved slot. */ ret = percpu_ref_init(&pool->ref, __zswap_pool_empty, PERCPU_REF_ALLOW_REINIT, GFP_KERNEL); if (ret) goto ref_fail; - INIT_LIST_HEAD(&pool->list); + + ret = zswap_pool_reserve_slot(pool); + if (ret < 0) { + pr_err("cannot create more than %d pools\n", ZSWAP_MAX_POOLS); + goto slot_fail; + } zswap_pool_debug("created", pool); return pool; +slot_fail: + percpu_ref_exit(&pool->ref); ref_fail: cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); @@ -388,7 +421,7 @@ static void __zswap_pool_release(struct work_struct *work) WARN_ON(!percpu_ref_is_zero(&pool->ref)); percpu_ref_exit(&pool->ref); - /* pool is now off zswap_pools list and has no references. */ + /* Slot cleared in __zswap_pool_empty(); synchronize_rcu() drained readers. */ zswap_pool_destroy(pool); } @@ -404,7 +437,11 @@ static void __zswap_pool_empty(struct percpu_ref *ref) WARN_ON(pool == zswap_pool_current()); - list_del_rcu(&pool->list); + /* + * Clear the slot before scheduling the release so new readers cannot + * see it; __zswap_pool_release()'s synchronize_rcu() drains the rest. + */ + rcu_assign_pointer(zswap_pools[pool->idx], NULL); INIT_WORK(&pool->release_work, __zswap_pool_release); schedule_work(&pool->release_work); @@ -435,7 +472,8 @@ static struct zswap_pool *__zswap_pool_current(void) { struct zswap_pool *pool; - pool = list_first_or_null_rcu(&zswap_pools, typeof(*pool), list); + pool = rcu_dereference_check(zswap_current_pool, + lockdep_is_held(&zswap_pools_lock)); WARN_ONCE(!pool && zswap_has_pool, "%s: no page storage pool!\n", __func__); @@ -468,11 +506,14 @@ static struct zswap_pool *zswap_pool_current_get(void) static struct zswap_pool *zswap_pool_find_get(char *compressor) { struct zswap_pool *pool; + int i; assert_spin_locked(&zswap_pools_lock); - list_for_each_entry_rcu(pool, &zswap_pools, list) { - if (strcmp(pool->tfm_name, compressor)) + for (i = 0; i < ZSWAP_MAX_POOLS; i++) { + pool = rcu_dereference_protected(zswap_pools[i], + lockdep_is_held(&zswap_pools_lock)); + if (!pool || strcmp(pool->tfm_name, compressor)) continue; /* if we can't get it, it's about to be destroyed */ if (!zswap_pool_tryget(pool)) @@ -497,10 +538,14 @@ unsigned long zswap_total_pages(void) { struct zswap_pool *pool; unsigned long total = 0; + int i; rcu_read_lock(); - list_for_each_entry_rcu(pool, &zswap_pools, list) - total += zs_get_total_pages(pool->zs_pool); + for (i = 0; i < ZSWAP_MAX_POOLS; i++) { + pool = rcu_dereference(zswap_pools[i]); + if (pool) + total += zs_get_total_pages(pool->zs_pool); + } rcu_read_unlock(); return total; @@ -562,7 +607,6 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_param if (pool) { zswap_pool_debug("using existing", pool); WARN_ON(pool == zswap_pool_current()); - list_del_rcu(&pool->list); } spin_unlock_bh(&zswap_pools_lock); @@ -589,16 +633,15 @@ static int zswap_compressor_param_set(const char *val, const struct kernel_param spin_lock_bh(&zswap_pools_lock); if (!ret) { + /* The new pool becomes current; drop the ref of the old one. */ put_pool = zswap_pool_current(); - list_add_rcu(&pool->list, &zswap_pools); + rcu_assign_pointer(zswap_current_pool, pool); zswap_has_pool = true; } else if (pool) { /* - * Add the possibly pre-existing pool to the end of the pools - * list; if it's new (and empty) then it'll be removed and - * destroyed by the put after we drop the lock + * Not current; drop the ref below. This frees a new pool + * (and its slot), or decommissions a reused one (slot kept). */ - list_add_tail_rcu(&pool->list, &zswap_pools); put_pool = pool; } @@ -1763,6 +1806,9 @@ static int zswap_setup(void) struct zswap_pool *pool; int ret; + /* Slot indices are stored in a u8 (pool->idx). */ + BUILD_BUG_ON(ZSWAP_MAX_POOLS - 1 > U8_MAX); + zswap_entry_cache = KMEM_CACHE(zswap_entry, 0); if (!zswap_entry_cache) { pr_err("entry cache creation failed\n"); @@ -1793,7 +1839,7 @@ static int zswap_setup(void) pool = __zswap_pool_create_fallback(); if (pool) { pr_info("loaded using pool %s\n", pool->tfm_name); - list_add(&pool->list, &zswap_pools); + rcu_assign_pointer(zswap_current_pool, pool); zswap_has_pool = true; static_branch_enable(&zswap_ever_enabled); } else { -- 2.43.0