From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6B3E121ABD7 for ; Fri, 26 Jun 2026 01:54:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782438889; cv=none; b=r8zKho3tWsb9imuZsKm8hTz7SFS7DfGUQbvZLRxv1+pMcJchsE5H03yH8cLpQWiZ1fGCBkWgJJSspzI4JVPtbvk09NZavjzLdDToPH6TaVv8Pmk48m7GWNlzdA6L60KLjQUfOYC4LiVc00+F4htF/Cxvj7SQuFNCgKUsZsr7bbU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782438889; c=relaxed/simple; bh=hZfnmklAal4aicxXbBBDgscgHyHYC+vxlJ0gnY6iEOA=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=kWIz6ep01oQOsn/g9JG2DFizg/sCyqkfFVfLPWyiIXVcXN6lWXOSwvHSQtGx/LNCE1uSmEDLMjb4hLQROIPZ80zZATBZSflez/VIPYn0u1SrCvaZEOy8X8NPIDPmzvJfN7zpnRndO3zHPO7W56lRhXGxCi5eJ/wRWYX8YFrhNok= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dFuHchoc; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dFuHchoc" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2c80d215169so4892155ad.3 for ; Thu, 25 Jun 2026 18:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782438887; x=1783043687; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=CsVLAkg0GTe3mySosYexdwuiWNuNXRLwOOfIA7hMuwI=; b=dFuHchoc773eLsynoKfO6imUEA5PFN0oYhsHw2Nh7RfoTetA//6qfWUtWzVGTFaiIA /KOZz6HnM6+J3eADHx8CFcJvmX4nxhe+1gNQoqgz/JvjSHOVmAXTI74G8Zr05ihGEw+N QumNu6MK55ZuZKmQLGm4dOG2FScBTUpDEa6zIS2X0FG22Jt4Q9upfTIIktHJ0zbcCbRZ oJgcBaut7xqXQHsRQMH4NBY4Uf/tBVbvwSQTBcoT3h3lwBMEysJaqr4gexcZ6ueVW7Dq o9XK49nH66ANkGKP3xX8mC4INUfClQwULMQX4kXBnKTgqmcrFsUUQhie1s08NQN95we0 KNJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782438887; x=1783043687; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=CsVLAkg0GTe3mySosYexdwuiWNuNXRLwOOfIA7hMuwI=; b=Wk59Ag0/BvUFkL3zXy8jnYRQJ7gZj4Y3n45GuGIqEJt2fsr4cj7ylai7M2mpRt2U2c SCgWDs+Y0TPsn5+WAQ0pBq/urcZqFi8YwFvd7YRoM0RqIiWnL/3z7oHWKXmlmD9Pm5n9 GWe0OgvGIwNyEB10cApLPAzTBXpgxNPX7geD41Kl5AmCew7lPzaVrdr2m+CjvzZbvcVR ATkJejfEoAd8Kn+bS1JwXUgQC8Ecs0i8oJjrL9fps5exuTq+IdMRGANnc1n0GpFbXmgH Dq8hpI2n4On4azN8bOWj+3xu9dpCy1XEfn8DZWGC90DSP5bZ8JrauxqxrSYUYEjoi8Xo Kf8w== X-Forwarded-Encrypted: i=1; AHgh+Ro4zockzHh+6aepb0F4QZ9iQuQ3sII3CYjiQD4NK2vMdJ1VX4eltjZCEDM5w8T0Tpc7K6l1Insd8dMtuGs=@vger.kernel.org X-Gm-Message-State: AOJu0YwJ82iXBmUEA9JyAKhLvFjOVmzPiSnDWGU/OjfUurW6JDyUuy/a SQke0TaMhdbUXB7yGDk/DSv0hcnzqXRsxdmFaB6BUJ/B2qX58f3JA5ON X-Gm-Gg: AfdE7cne8qlJWPILrHqNRxkd9+g9zKUU8USElCtplgtmY0RJATcDFOk3+Z36fYCLnZU ZAuz91zbIr/nTudoU4l8XHGkq/qLsTViGS6WUFZlEXWT+HOUZHxg//kK6ENFesTegYhD8UdOZ8C pI03W8nDhFvcXZ1JfUPt30uV035B89OEynD3Y2Yw+ZiYMDXWWh7ZHxMrUwxWT4M4VGvTZeAZwuv MGKzRk/FWe7sQ0k4tNuHan2GQO+ieYz7cWztnFIl+8tz97QOUN7OrKhtfMnwTJ7D9I2t9FwkYPs aOI/9MMpzQQY/OYAlV6V4EHx5LBh3bg80gbmjUbwNcK9vOZrs8WuWetlqAGK9ezslweUW8Gu2Ys 8L/nIA2LYiynrmSL6Bzas1i9M8kYp2uWr4ALAcrWnxdTuyF+Jntf4c7eZrwdV/nv0dW7pCr8Cmy UWPGFpyvouxIyChnwGjQMIl14TvDZU X-Received: by 2002:a17:903:1666:b0:2bd:5ab:af95 with SMTP id d9443c01a7336-2c7fc36e61cmr46992765ad.0.1782438887399; Thu, 25 Jun 2026 18:54:47 -0700 (PDT) Received: from ubuntu22.mioffice.cn ([43.224.245.232]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c7f63b2bd7sm29047005ad.48.2026.06.25.18.54.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jun 2026 18:54:46 -0700 (PDT) From: Wenchao Hao To: Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Minchan Kim , Sergey Senozhatsky Cc: Nhat Pham , Joshua Hahn , Barry Song , Wenchao Hao Subject: [PATCH v6 0/4] mm/zsmalloc: reduce lock contention in zs_free() Date: Fri, 26 Jun 2026 09:49:59 +0800 Message-Id: <20260626015003.2965881-1-haowenchao22@gmail.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Wenchao Hao This series reduces lock contention in zs_free(), which dominates the unmap path under memory pressure on Android (LMK kills) and on x86 servers running zswap-heavy workloads. The current zs_free() takes pool->lock (rwlock, read side) just to look up the size_class for a handle, then takes class->lock and holds it across __free_zspage() which can call into the buddy allocator and acquire zone->lock. Two costs follow: * pool->lock reader-counter cacheline bouncing among concurrent zs_free() callers. * class->lock held across folio_put(), so any zone->lock wait fans out to every other zs_free() on the same class. The series tackles both: Patch 1: encode size_class index into obj alongside PFN and obj_idx, so zs_free() can locate the class without pool->lock. Patch 2: drop pool->lock from zs_free() on 64-bit; 32-bit unchanged. Patch 3: move zspage page-freeing out of class->lock. Patch 4: document the three free_zspage helper variants that result from the split in patch 3. Performance results: Test: each process independently mmap 256MB, write data, madvise MADV_PAGEOUT to swap out via zram (lzo-rle), then concurrent munmap. Raspberry Pi 4B (4-core ARM64 Cortex-A72): mode Base Patched Speedup single 59.0ms 56.0ms 1.05x multi 2p 94.6ms 66.7ms 1.42x multi 4p 202.9ms 110.6ms 1.83x x86 (20-core Intel i7-12700, 16 concurrent processes): mode Base Patched Speedup single 11.7ms 9.8ms 1.19x multi 2p 24.1ms 17.2ms 1.40x multi 4p 63.0ms 45.3ms 1.39x [1] https://lore.kernel.org/linux-mm/20260508061910.3882831-1-haowenchao@xiaomi.com/ [2] https://lore.kernel.org/linux-mm/20260527115930.3138213-1-haowenchao@xiaomi.com/ [3] https://lore.kernel.org/linux-mm/20260605084242.1549811-1-haowenchao22@gmail.com/ [4] https://lore.kernel.org/linux-mm/20260609113520.3507783-1-haowenchao22@gmail.com/ [5] https://lore.kernel.org/all/20260616031309.3036234-1-haowenchao22@gmail.com/ Changes since v5: - 2/4: rename obj_handle_class_lock() to obj_class_get_and_lock(). - Pick up Reviewed-by from Barry Song on 2/4, 3/4, 4/4. Changes since v4: - 1/4: rename macros to make their intent clearer, following the kernel's *_BITS convention (Nhat Pham, Barry Song). - 3/4: drop the unused pool argument from __free_zspage_lockless(). Changes since v3: - 1/4: gate ZS_OBJ_CLASS_BITS on a new spare-bit check so unsuitable configs (e.g. UML, no sparsemem) fall back to plain obj layout instead of breaking the build. (sashiko AI review) - 2/4: annotate handle_to_obj()/record_obj() with READ_ONCE/WRITE_ONCE to silence KCSAN; sync the #if predicate with patch 1. (sashiko AI review) - 4/4: clarify when async_free_zspage() runs; pick up Reviewed-by. (Nhat Pham) - Drop Reviewed-by from Nhat Pham on 1/4 and 2/4: substantial logic changes since v3, please re-review. Changes since v2: - 3/4: drop likely() hint and tidy up else-branch braces, per review nits from Nhat Pham and Joshua Hahn. - 4/4 (new): document free_zspage helper variants in a single comment block, per Nhat Pham. - Pick up Reviewed-by tags from Nhat Pham (1/4, 2/4, 3/4) and Joshua Hahn (3/4). - Fix patch 1/4 and 2/4 From: to match Signed-off-by, per Barry Song. Changes since v1: - Rename obj-encoding macros for clarity. - Make 32-bit / 64-bit handling transparent at call sites (no #ifdef in callers). - Extract a helper to keep zs_free() unified. Wenchao Hao (3): mm/zsmalloc: encode class index in obj value for lockless class lookup mm/zsmalloc: drop pool->lock from zs_free on 64-bit systems mm/zsmalloc: document free_zspage helper variants Xueyuan Chen (1): mm/zsmalloc: drop class lock before freeing zspage mm/zsmalloc.c | 225 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 190 insertions(+), 35 deletions(-) -- 2.34.1