From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4449B2DAFA9 for ; Fri, 26 Jun 2026 01:54:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782438899; cv=none; b=Wz+QfQujw4sUFDnLnjyl1Z2F7CaSuS9FLJqEAD1KiX4fRPkw+W+zBIoYwsydC+EnhWnNeomliBQZ2lNQWNHvGcda/nC7od7k562diExSVWqDDzcp/IT07mK+8MY5TfkS28FjuUR3OWq67u3C4Q2s5HH8ODIZVLCX5YrR4ctzWA4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782438899; c=relaxed/simple; bh=Q/OPIHhACaJCyB/pe9MPEFotrM3YAJRN4IRX9EhjntA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=C3oYkxp4Djcwbrbh+4yhMWtBPtVpyNuxysA2ZN9CcOnsppj7A25v9jmKNkEjhHrbS070lKHYlVHib6uHWba2X+mJ9o7HICuEDSdsZ4dEGhWeDDcCUJjv5AZDej1Uc6XKq+0JCmS1piAoNS4JV1HSi4e/lJsvob6h709eJiDMP6A= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RYccFPCw; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RYccFPCw" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2c7e8eea816so6550265ad.1 for ; Thu, 25 Jun 2026 18:54:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782438897; x=1783043697; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UsegwOQMPUPZkD5oEdLRNUkDENZX/KJI8/fJZMWPicQ=; b=RYccFPCwtrYa/iKuq1wzEDUQPyRSEmgDs3QllXqHQb3LKwU3aB7Ln6iR5G1QrEvAN2 pf3rv2uY9sxhvcwRndG8nox9JoMkNqKyfCXzHONXaIaKNAWL/2K0MO9w/oxKd5paBkfj TkH415GFac2XBzxfG4MD9ko2tOIM0th/Hw3f0/lgl1yVh7ZuUf07Pk5Z+xPlDZw4QMUH QpiQrZVzVoxTXtvWCGfR1R14d5T322Kr2TWzGsZozV77KD6tOfp3pwdYGArSHL+vm5YH zKsraNlhsr/a97fxBFtQasiFhQXGNaSUWY8TSJt8D9NG06pvzEAwWvEY4cK5XSoKA+Cg 5Tqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782438897; x=1783043697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=UsegwOQMPUPZkD5oEdLRNUkDENZX/KJI8/fJZMWPicQ=; b=oUhCjSBBk44mBlTnaqxAiPHs+gGCCIVxmG95MerpedHogrVYx39wPoEByxr+FGy1SA I4EFjZdCWy06b9nJGY5XbZZqbzftsVsW33Fz4mzEkYQAPGph2SdfRPJBR/cIU3OYOOiU Dn62dcXHICDM7lAA9L9HZCJHiptlFcvHnK8GaTU51YxjajkPDeQTLf5Qk7zNeDvCI1p0 KdyIYL8V41PUU31vJAGjhta9ymiNi0QybvJq8W+CJ+MiL76apQrtzaxas1A5JED8zx65 ogzwG2XLoiuHEjS5mDj9IXgu7AiR9eiLi+LS7BR4GMFBfTI+OtRgIVQ6xlPNK/8ftX3j QTog== X-Forwarded-Encrypted: i=1; AHgh+RoGtlPULvAWkQqQR7BpJagR2RR2X8XG+dClQ/a5RjZiqiKEsId/B4S+Dmp/y4Sx97ugFqNmq+bPdEkQNrQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyLrS4/1eFKw6jHjaSBRZ1M+e4YPXshir5fvbr6m9hRemvS0n0u 5JiKmN2+cuMXB9C67qLr4Rozilr7FAmHIEVUlGf+IHbBTdHs2CqwUG7o X-Gm-Gg: AfdE7cnTs1A+3ebgki1ZDNHiXOhoEkxcSSsymdgdnUlcSIjArAkAAxEXDgArdwoqpNt 412I1udQ08Lst0sWQMIS0edTGX7LOb/L2RO0JQJreLkwflueDRkQCueBUEhHTSiZBsJw+Cob48T HHRjKLnpYn0+Zj1xv/uW+dB/+JN0GwhtYFvn/fFy0kXZ7j3RZafg97E3R9/r1mVOVzEM6DnUjSK tQ3ZFO3rOU8na8Y77p2eavNgxXIXxXCgHrywMRx6cEbSMGJBN0q0rBRgimFUL0q2ZSOjNvl+WGm NbQMj3BYQo5XrKlx83Q/dZblP/qxhwhr+GhxQ6f+YOBpGf0r1G8Psj0WWgh43otDpmSLjvF7WA3 Rj7kaEhjcKxUUkTPAgp6oOwX99eWMENi+2/uM/oAMXNGALlv2eKyBOipTF0VH9QzWUF4Le0HOiw Tk3cSGwGi/at7mljQQgZ2548WjomGqMIJavTzAnCc= X-Received: by 2002:a17:902:d549:b0:2c7:c63a:2d69 with SMTP id d9443c01a7336-2c7c63a2ed4mr107061855ad.15.1782438897503; Thu, 25 Jun 2026 18:54:57 -0700 (PDT) Received: from ubuntu22.mioffice.cn ([43.224.245.232]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c7f63b2bd7sm29047005ad.48.2026.06.25.18.54.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jun 2026 18:54:57 -0700 (PDT) From: Wenchao Hao To: Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Minchan Kim , Sergey Senozhatsky Cc: Nhat Pham , Joshua Hahn , Barry Song , Xueyuan Chen , Wenchao Hao Subject: [PATCH v6 3/4] mm/zsmalloc: drop class lock before freeing zspage Date: Fri, 26 Jun 2026 09:50:02 +0800 Message-Id: <20260626015003.2965881-4-haowenchao22@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260626015003.2965881-1-haowenchao22@gmail.com> References: <20260626015003.2965881-1-haowenchao22@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Xueyuan Chen Currently in zs_free(), the class->lock is held until the zspage is completely freed and the counters are updated. However, freeing pages back to the buddy allocator requires acquiring the zone lock. Under heavy memory pressure, zone lock contention can be severe. When this happens, the CPU holding the class->lock will stall waiting for the zone lock, thereby blocking all other CPUs attempting to acquire the same class->lock. This patch shrinks the critical section of the class->lock to reduce lock contention. By moving the actual page freeing process outside the class->lock, we can improve the concurrency performance of zs_free(). Testing on the RADXA O6 platform shows that with 12 CPUs concurrently performing zs_free() operations, the execution time is reduced by 20%. Signed-off-by: Xueyuan Chen Reviewed-by: Nhat Pham Reviewed-by: Joshua Hahn Reviewed-by: Barry Song Signed-off-by: Wenchao Hao --- mm/zsmalloc.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 210a777081b7..f3045345e3a8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -884,13 +884,10 @@ static int trylock_zspage(struct zspage *zspage) return 0; } -static void __free_zspage(struct zs_pool *pool, struct size_class *class, - struct zspage *zspage) +static inline void __free_zspage_lockless(struct zspage *zspage) { struct zpdesc *zpdesc, *next; - assert_spin_locked(&class->lock); - VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0); @@ -906,7 +903,13 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, } while (zpdesc != NULL); cache_free_zspage(zspage); +} +static void __free_zspage(struct zs_pool *pool, struct size_class *class, + struct zspage *zspage) +{ + assert_spin_locked(&class->lock); + __free_zspage_lockless(zspage); class_stat_sub(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated); } @@ -1519,6 +1522,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) unsigned long obj; struct size_class *class; int fullness; + struct zspage *zspage_to_free = NULL; if (IS_ERR_OR_NULL((void *)handle)) return; @@ -1529,10 +1533,23 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_free(class->size, obj); fullness = fix_fullness_group(class, zspage); - if (fullness == ZS_INUSE_RATIO_0) - free_zspage(pool, class, zspage); + if (fullness == ZS_INUSE_RATIO_0) { + if (trylock_zspage(zspage)) { + remove_zspage(class, zspage); + class_stat_sub(class, ZS_OBJS_ALLOCATED, + class->objs_per_zspage); + zspage_to_free = zspage; + } else { + kick_deferred_free(pool); + } + } spin_unlock(&class->lock); + + if (zspage_to_free) { + __free_zspage_lockless(zspage_to_free); + atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated); + } cache_free_handle(handle); } EXPORT_SYMBOL_GPL(zs_free); -- 2.34.1