From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C0A42DF719 for ; Fri, 26 Jun 2026 01:54:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782438893; cv=none; b=pPYrY/PEJXoNrN31YUjvbPw6bcEX4EZ1CatcZDC7CmlRZ54Mb6zU0UrRlIJ9ic5KXCZbfsv9gcRJY/GhcQp/mXbu6fSCx/nId1qB/zgih0NotXpIZlq78sn4++pf1INh+OYOyMd3e4KM7ajmQWycc75W+laLVN2FzsCHFvXRO3s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782438893; c=relaxed/simple; bh=ZtaCZnncQ5ThNUtcwBXnXE/MUa64Fm2RgkT7c98OmwA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BOZmcUD2gGmoaNM/q+vb1iDs0+RtUZ7J4M7QE3aMPQrmz87Vn6e+MPeTHQSVnw8zheomTxBeESRS8RvLt7MfC/Pad5drIaoliabGIAeANFipkslOkuMAOCtQt+fBZ8XvzLQZ2ZxKBBVWbFuTfa++uxZTxlCX6GQV5XVp/n4ZTrE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aqS/JCt3; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aqS/JCt3" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-37df416c44bso265882a91.2 for ; Thu, 25 Jun 2026 18:54:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1782438891; x=1783043691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7CMxZGmSd9rDJg1W64dvSL1ed3tT8uDPMgraGKCi12o=; b=aqS/JCt3Nk+91tqjeLFXRgjo7u9Inf3t2YOjar4PdUOlzGXeclppUYt7mqjWzYwcd1 NRIl8QXBZJIatYLrWC9nN6guN183OqVZXTiU5BE/FUzXOBX19syJ58TKwxoLUqoLK2bO 20ug2RdoiXFGWlvpBiSXYsWsm5qCrzUyQZFfybATXww74cq1iO7XQ6bIjpyxTA64CPES +R34UniDmGvWpX9zGV9maGnIC9gdrD7rICF1z11XpYb3U0iifCZ13eUeRF/i588ufqic t1YoQkXxquBEesE7o7iW7WMzRPLKSZ7o9+NlwaGZ7+w2tAFcMnyo88alCL2rF6S8ukSN G5CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782438891; x=1783043691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=7CMxZGmSd9rDJg1W64dvSL1ed3tT8uDPMgraGKCi12o=; b=niilMaeDZwq2zxbV6OKufWnY8r9uIKq2W7rsY4qZUrDWPOTLSDau8TpX/LY4TqACaM DKc/XQisOwCjMDkyhq7HyaWt8zAMMKfpxUPLrUFd3l23j5JxnP1WqQ02mG34FU+dm78d GMYtp6X+GSgH10e7At9zGbU2xwtDIwOx4gn1rs9G72IuxJvB4yJCxUASVO0Of3/8x4e3 TZXOtnt+Fz/UmQbh5q2kOoOtqd+mtoy8IN0z/gr4kc1o7mSLeaVxQCjtEaWxiwIspaNZ Im4HyqGbYnCfKVEFhUHyIjkMIUjbiCClI4Vx4rMrHLv8cWBc5ZpmU1OGcDb9WADzEfq9 dNpw== X-Forwarded-Encrypted: i=1; AHgh+RqxUDHRMAs2nCwqJ7ZkzhPrE6+24yor7iLYOyFUChKqJGdpOyqrynq3F4c/frhu61KvOcAW9rEpEYk129k=@vger.kernel.org X-Gm-Message-State: AOJu0YylEMVwZU1EZLUpqwghCm0YY02VbaCiAf3NvrUH3zICWMElML2Z bJum/KFo6BneJw35uBCo/Pe8+kZeNgiYAQX9gcNaoOA6/wZwPmuxlZsZ X-Gm-Gg: AfdE7cnPPAX45w7nctEykTzHTkB8JP3TQfBUPHE1RS6Su/0qHVHEuUm2TWk9V8aM3vM RwLCAt6HFuSOKMXzv65gowzpzIkQ+MeLbVncRwhfa4mnbru78Gap4Y+oCsRSzd7HWQ6QmQiPwKL AkW5no4YILCcoAa3+4lmRihNUj+LQ2DnmEv65wpVOvGSaYyzh6DXrIoSwm6JjEV2JUnvMO1vgB5 OhEnKCiChBChjHfYLyGBkC90LPz5HQU3y5E/tr7g70Psc5507kDl9riMQhS8oQXa3Gdo4jULRIW MTZ6IzJp9urQKYC/woXrB+nwq0XAqzS4FZ44RM+KyxA4Kp1Y4OgWroHhog4W7gzesUnIcMCYDJ0 XTFPQEI6uXZVRf30SN4XWTmAgiWyowqYXZABLpoDXaQ/jKWPdzhPlRht0BaatJ/nqFRw0zBkd0B emmGVxhw6e5oGvJpoVqcLcu1AR2ct5gHIUD6PuNxU= X-Received: by 2002:a17:902:e84e:b0:2c6:a99a:52b5 with SMTP id d9443c01a7336-2c7fc890641mr46029165ad.12.1782438890700; Thu, 25 Jun 2026 18:54:50 -0700 (PDT) Received: from ubuntu22.mioffice.cn ([43.224.245.232]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c7f63b2bd7sm29047005ad.48.2026.06.25.18.54.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jun 2026 18:54:50 -0700 (PDT) From: Wenchao Hao To: Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Minchan Kim , Sergey Senozhatsky Cc: Nhat Pham , Joshua Hahn , Barry Song , Wenchao Hao Subject: [PATCH v6 1/4] mm/zsmalloc: encode class index in obj value for lockless class lookup Date: Fri, 26 Jun 2026 09:50:00 +0800 Message-Id: <20260626015003.2965881-2-haowenchao22@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260626015003.2965881-1-haowenchao22@gmail.com> References: <20260626015003.2965881-1-haowenchao22@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Wenchao Hao Encode the size_class index (class_idx) into the obj value so that zs_free() can determine the correct size_class without dereferencing the handle->obj->PFN->zpdesc->zspage->class chain under pool->lock. class_idx is invariant across page migration (only PFN is rewritten), so a lockless read of obj always yields a valid class_idx. Where obj has more bits below the PFN field than obj_idx alone needs, split that space into class_idx and obj_idx subfields: |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->| +-----------------+-------------------------+-----------------------+ | PFN | class_idx | obj_idx | +-----------------+-------------------------+-----------------------+ MSB ^ LSB | +-- ZS_OBJ_PFN_SHIFT The macro layout changes as follows: Before After Meaning ---------------- ------------------ ---------------------------- OBJ_INDEX_BITS ZS_OBJ_IDX_BITS width of obj_idx subfield OBJ_INDEX_MASK ZS_OBJ_IDX_MASK mask of obj_idx subfield (n/a) ZS_OBJ_CLASS_BITS width of class_idx subfield (n/a) ZS_OBJ_CLASS_MASK mask of class_idx subfield (n/a) ZS_OBJ_PFN_SHIFT bit offset of PFN in obj ZS_OBJ_CLASS_BITS folds to 0 (and the layout collapses to [PFN | obj_idx]) when obj has no spare bits, i.e. on 32-bit or on 64-bit fallback paths where MAX_POSSIBLE_PHYSMEM_BITS == BITS_PER_LONG (e.g. UML); zs_free() then falls back to pool->lock. Reviewed-by: Nhat Pham Signed-off-by: Wenchao Hao --- mm/zsmalloc.c | 105 +++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 91 insertions(+), 14 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 63128ddb7959..c4fc06b259af 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -67,8 +67,8 @@ #define MAX_POSSIBLE_PHYSMEM_BITS MAX_PHYSMEM_BITS #else /* - * If this definition of MAX_PHYSMEM_BITS is used, OBJ_INDEX_BITS will just - * be PAGE_SHIFT + * If this definition of MAX_PHYSMEM_BITS is used, ZS_OBJ_PFN_SHIFT will + * just be PAGE_SHIFT */ #define MAX_POSSIBLE_PHYSMEM_BITS BITS_PER_LONG #endif @@ -88,8 +88,23 @@ #define OBJ_TAG_BITS 1 #define OBJ_TAG_MASK OBJ_ALLOCATED_TAG -#define OBJ_INDEX_BITS (BITS_PER_LONG - _PFN_BITS) -#define OBJ_INDEX_MASK ((_AC(1, UL) << OBJ_INDEX_BITS) - 1) +/* + * obj is encoded as [PFN | class_idx | obj_idx] within an unsigned long: + * + * |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->| + * +-----------------+-------------------------+-----------------------+ + * | PFN | class_idx | obj_idx | + * +-----------------+-------------------------+-----------------------+ + * MSB ^ LSB + * | + * +-- ZS_OBJ_PFN_SHIFT + * + * Encoding class_idx into obj lets zs_free() locate the size_class + * without holding pool->lock; class_idx is invariant across page + * migration (only PFN changes), so a lockless read of the obj value + * always yields a valid class_idx. + */ +#define ZS_OBJ_PFN_SHIFT (BITS_PER_LONG - _PFN_BITS) #define HUGE_BITS 1 #define FULLNESS_BITS 4 @@ -98,9 +113,61 @@ #define ZS_MAX_PAGES_PER_ZSPAGE (_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL)) +/* + * Bits to index a page within a zspage = ceil(log2(ZS_MAX_PAGES_PER_ZSPAGE)). + * Computed at preprocessor time, for use in #if below. Kconfig + * restricts ZSMALLOC_CHAIN_SIZE to [4, 16]. + */ +#if ZS_MAX_PAGES_PER_ZSPAGE <= 4 +#define ZS_PAGES_PER_ZSPAGE_BITS 2 +#elif ZS_MAX_PAGES_PER_ZSPAGE <= 8 +#define ZS_PAGES_PER_ZSPAGE_BITS 3 +#elif ZS_MAX_PAGES_PER_ZSPAGE <= 16 +#define ZS_PAGES_PER_ZSPAGE_BITS 4 +#else +#error "ZSMALLOC_CHAIN_SIZE out of expected range [4,16]" +#endif + +/* + * Bits to index an object within a single PAGE_SIZE at the smallest + * possible object size: log2(PAGE_SIZE / 32) = PAGE_SHIFT - 5. + * 32 is the hard floor of ZS_MIN_ALLOC_SIZE. + */ +#define ZS_OBJS_PER_PAGE_BITS (PAGE_SHIFT - 5) + +/* + * Bits to index any object in the densest possible zspage. Below this, + * ZS_MIN_ALLOC_SIZE is auto-raised by the MAX(32, ...) formula -- still + * correct, but objects are coarser. + */ +#define ZS_OBJS_PER_ZSPAGE_BITS \ + (ZS_PAGES_PER_ZSPAGE_BITS + ZS_OBJS_PER_PAGE_BITS) + +/* + * Encode class_idx only when obj has spare bits; otherwise + * ZS_OBJ_CLASS_BITS folds to 0 (32-bit, or 64-bit UML/fallback). + */ +#if BITS_PER_LONG >= 64 && \ + ZS_OBJ_PFN_SHIFT >= (CLASS_BITS + 1) + ZS_OBJS_PER_ZSPAGE_BITS +#define ZS_OBJ_CLASS_BITS (CLASS_BITS + 1) +#else +#define ZS_OBJ_CLASS_BITS 0 +#endif +#define ZS_OBJ_CLASS_MASK ((_AC(1, UL) << ZS_OBJ_CLASS_BITS) - 1) + +#define ZS_OBJ_IDX_BITS (ZS_OBJ_PFN_SHIFT - ZS_OBJ_CLASS_BITS) +#define ZS_OBJ_IDX_MASK ((_AC(1, UL) << ZS_OBJ_IDX_BITS) - 1) + +/* + * Belt-and-suspenders: the #if above already guarantees this when + * class_idx is enabled. Catches future tweaks that bypass it. + */ +static_assert(ZS_OBJ_IDX_BITS >= ZS_PAGES_PER_ZSPAGE_BITS, + "zsmalloc: ZS_MIN_ALLOC_SIZE would exceed ZS_MAX_ALLOC_SIZE"); + /* ZS_MIN_ALLOC_SIZE must be multiple of ZS_ALIGN */ #define ZS_MIN_ALLOC_SIZE \ - MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> OBJ_INDEX_BITS)) + MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> ZS_OBJ_IDX_BITS)) /* each chunk includes extra space to keep handle */ #define ZS_MAX_ALLOC_SIZE PAGE_SIZE @@ -721,26 +788,35 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc, unsigned int *obj_idx) { - *zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS); - *obj_idx = (obj & OBJ_INDEX_MASK); + *zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT); + *obj_idx = (obj & ZS_OBJ_IDX_MASK); } static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc) { - *zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS); + *zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT); +} + +/* Folds to 0 when ZS_OBJ_CLASS_BITS == 0; no ifdef needed at callers. */ +static unsigned int obj_to_class_idx(unsigned long obj) +{ + return (obj >> ZS_OBJ_IDX_BITS) & ZS_OBJ_CLASS_MASK; } /** - * location_to_obj - get obj value encoded from (, ) + * location_to_obj - encode (, , ) into obj value * @zpdesc: zpdesc object resides in zspage * @obj_idx: object index + * @class_idx: size class index; ignored when ZS_OBJ_CLASS_BITS == 0 */ -static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx) +static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx, + unsigned int class_idx) { unsigned long obj; - obj = zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS; - obj |= obj_idx & OBJ_INDEX_MASK; + obj = zpdesc_pfn(zpdesc) << ZS_OBJ_PFN_SHIFT; + obj |= (unsigned long)(class_idx & ZS_OBJ_CLASS_MASK) << ZS_OBJ_IDX_BITS; + obj |= obj_idx & ZS_OBJ_IDX_MASK; return obj; } @@ -1276,7 +1352,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, kunmap_local(vaddr); mod_zspage_inuse(zspage, 1); - obj = location_to_obj(m_zpdesc, obj); + obj = location_to_obj(m_zpdesc, obj, zspage->class); record_obj(handle, obj); return obj; @@ -1762,7 +1838,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page, old_obj = handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj = (unsigned long)location_to_obj(newzpdesc, obj_idx); + new_obj = location_to_obj(newzpdesc, obj_idx, + obj_to_class_idx(old_obj)); record_obj(handle, new_obj); } } -- 2.34.1