From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAA7C2E22B5 for ; Wed, 27 May 2026 12:01:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883300; cv=none; b=DViNdTJLCVb0lAmBwDbZDvBSEbLbg6JNir4SkyA9sOeC9JYJEf1czJkcT8csTaUQp+D/ROIjgPC2sS744jefiGVOCVOdsWQi/UTG2LG2oPxLUhaZhAcS0HPXMsUE5ph/fcvFlt7nRLzHFWsqI141bdA57R8yOU+ib39ZmWMQktA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883300; c=relaxed/simple; bh=WnoFY+zGtT1r+C3lHmExFEvcZlJxTD5AUXF8lDS0ZXc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e1Vd9jW3RR+skVDxEiG5mm7D7rdD0CzLCzFC38xcwQqlp66zwEU6ce9Q4G2jtIVEywCymsXJGqe0mrswZSehgaQzapB9SRAO1SPP7kUVsjVk/nAMqv0FfUDS+U7vGVDOWsFKe3QWqKAPHohr/Ht2c1rGOHY00rQY4RjYghXIBac= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LEwVox8x; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LEwVox8x" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2ba856db1c0so85148175ad.3 for ; Wed, 27 May 2026 05:01:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1779883298; x=1780488098; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uHNxVkzgOEtrCskHNiISj95cLTtaoMXSb01E6jPrlAE=; b=LEwVox8xsUgMznfruzwt9xUQC9cBkOsTfyRUqLqmEbDu1YXp/5y82ESFiooMWssG3k G7REr4+ygFbGEMUtIqRdisH0zfawRx6Ks1AXpY0X9/l/zRFcuOiEzqImtgSF6IKwnhgE AUKqhjkrusLF4QQjz1yNmGjY3V0wr5liRFlAxVLQfDUgJVG6xVDD2RkRO5iBGTJLO2Ox j2rjBGngT9aImcKNwSzsdbLo1GXvSDX/NV8gcVue3l+qUct34J0G8MCSdqtTaQ1ScZoF R6d/jof9ALsL4A2kVkwzYq4mIfM4zC9PMwu5PDKj4KthCFcBNm01+8IVAThqul8lHRnX 2E5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779883298; x=1780488098; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=uHNxVkzgOEtrCskHNiISj95cLTtaoMXSb01E6jPrlAE=; b=YLOoF+LBPovbsXJDf+g4rW0oOSrNkIRegMLrqxRjVPwPmh38Ej1jUhmyXkPrtUB+4t 64oUqsWJo2fLIHbVFJnCkxo87gPyqEBIYnFvKXM85DAMiCuD6WiemRyhH/RepC//4NP6 baxRNaCJK0ie0zfqYuAwjZacIvR8olREbpwRiU7ko05rnkuehAoRt1h56jcY+qq7aCZ4 MUcRXMoU9fqhT1nHxpPXeLnglu0OZnf1zG5Jhr2aAESKOc94chPuIVZfMYf7MohO1w6E MXHPN+l3hGpokk5uM2cDT3i/kdjV0AbzwZQjOIbu/0KU6y3J6SfIQaMZ9+5SxiUJhrOp dq0g== X-Forwarded-Encrypted: i=1; AFNElJ9CSYvccBMwlz65RlwH21hW1z2R9NREp/b8VAhpbU07J73i4FY57IKwVNOeAYpjnry5SQ32ruDURH21/N8=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1rdMRgefxGrrJG3i5D+wohiFr98m2xA5WUZAsJ8efz0ch39nN PmFxewK3GqGsw0HL997Zydy6UcQvyNnKDXo8KurI9ldZInMcYL+YZGpp X-Gm-Gg: Acq92OGjPCOMWg6VpFLCCe/a95Ij8+7L717S4IzrpD6CEgbAhWowqMg8f8CIOQobXCw lMnGgQ9gIY6S5nCTym8Fxk9uh4yJlj9qaKlm8rCfaIFp2VAhIGVjRqWuESysDDUi6kidjz0uTYa LhIUQFFq+6EdC9/+aXzv8raIQS8ZTjfo8Vvaux/57fbEofh+iMqCFmLQYI9SElEHZHlTNPOsoTA zPY2zCmbFOQlTm4NSi0Nc9n5pCRrAwnDDuxCC3wxtCArU4V0IxF+FS9bjyWEj5JZBS37Rq0wR18 VtoER9fRv+a1EAtmH9ATlJkEtXY09X0gufTwUUMZaovCxWBGf8J2WUl3xKyY6zwLraqfO/0B+Sn fn9nGdTrOooL8SHD1DKyeQEUYac/G7utFc8WJgqHIM1jcZozCFmAjJnemyHB1yIxYqV5prFW9pA b77dXNGj22E78LKxsAzHuVCRLrb9jhBIJKwPJaidFZTUp2UiLk X-Received: by 2002:a17:903:4405:b0:2b9:6458:1a2c with SMTP id d9443c01a7336-2beb06733bemr252958595ad.13.1779883298133; Wed, 27 May 2026 05:01:38 -0700 (PDT) Received: from ubuntu22.mioffice.cn ([43.224.245.232]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2beb56b5886sm136250465ad.20.2026.05.27.05.01.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 May 2026 05:01:37 -0700 (PDT) From: Wenchao Hao X-Google-Original-From: Wenchao Hao To: Andrew Morton , Nhat Pham , Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Minchan Kim , Sergey Senozhatsky Cc: Wenchao Hao Subject: [RFC PATCH v2 1/3] mm/zsmalloc: encode class index in obj value for lockless class lookup Date: Wed, 27 May 2026 19:59:28 +0800 Message-Id: <20260527115930.3138213-2-haowenchao@xiaomi.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260527115930.3138213-1-haowenchao@xiaomi.com> References: <20260527115930.3138213-1-haowenchao@xiaomi.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Encode the size_class index (class_idx) into the obj value so that zs_free() can determine the correct size_class without dereferencing the handle->obj->PFN->zpdesc->zspage->class chain under pool->lock. class_idx is invariant across page migration (only PFN is rewritten), so a lockless read of obj always yields a valid class_idx. The space below the PFN field in obj is over-provisioned on 64-bit systems, with more bits than obj_idx needs. Split that space into class_idx and obj_idx subfields: |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->| +-----------------+-------------------------+-----------------------+ | PFN | class_idx | obj_idx | +-----------------+-------------------------+-----------------------+ MSB ^ LSB | +-- ZS_OBJ_PFN_SHIFT The macro layout changes as follows: Before After Meaning ---------------- ------------------ ---------------------------- OBJ_INDEX_BITS ZS_OBJ_IDX_BITS width of obj_idx subfield OBJ_INDEX_MASK ZS_OBJ_IDX_MASK mask of obj_idx subfield (n/a) ZS_OBJ_CLASS_BITS width of class_idx subfield (n/a) ZS_OBJ_CLASS_MASK mask of class_idx subfield (n/a) ZS_OBJ_PFN_SHIFT bit offset of PFN in obj On 32-bit systems there is no spare room for class_idx, so the encoding is disabled (ZS_OBJ_CLASS_BITS = 0) and the obj layout remains [PFN | obj_idx]. Signed-off-by: Wenchao Hao --- mm/zsmalloc.c | 80 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 66 insertions(+), 14 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 63128ddb7959..6b0014b43408 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -67,8 +67,8 @@ #define MAX_POSSIBLE_PHYSMEM_BITS MAX_PHYSMEM_BITS #else /* - * If this definition of MAX_PHYSMEM_BITS is used, OBJ_INDEX_BITS will just - * be PAGE_SHIFT + * If this definition of MAX_PHYSMEM_BITS is used, ZS_OBJ_PFN_SHIFT will + * just be PAGE_SHIFT */ #define MAX_POSSIBLE_PHYSMEM_BITS BITS_PER_LONG #endif @@ -88,8 +88,27 @@ #define OBJ_TAG_BITS 1 #define OBJ_TAG_MASK OBJ_ALLOCATED_TAG -#define OBJ_INDEX_BITS (BITS_PER_LONG - _PFN_BITS) -#define OBJ_INDEX_MASK ((_AC(1, UL) << OBJ_INDEX_BITS) - 1) +/* + * obj is encoded as [PFN | class_idx | obj_idx] within an unsigned long: + * + * |<-- _PFN_BITS -->|<-- ZS_OBJ_CLASS_BITS -->|<-- ZS_OBJ_IDX_BITS -->| + * +-----------------+-------------------------+-----------------------+ + * | PFN | class_idx | obj_idx | + * +-----------------+-------------------------+-----------------------+ + * MSB ^ LSB + * | + * +-- ZS_OBJ_PFN_SHIFT + * + * Encoding class_idx into obj lets zs_free() locate the size_class + * without holding pool->lock; class_idx is invariant across page + * migration (only PFN changes), so a lockless read of the obj value + * always yields a valid class_idx. + * + * On 32-bit systems there is no spare room for class_idx, so + * ZS_OBJ_CLASS_BITS is 0 and the layout collapses to the original + * [PFN | obj_idx] without any ifdef in callers. + */ +#define ZS_OBJ_PFN_SHIFT (BITS_PER_LONG - _PFN_BITS) #define HUGE_BITS 1 #define FULLNESS_BITS 4 @@ -98,9 +117,29 @@ #define ZS_MAX_PAGES_PER_ZSPAGE (_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL)) +/* + * Reuse the width that struct zspage already reserves for its + * class field (zspage->class:CLASS_BITS + 1) for the class_idx + * field encoded in obj. On 32-bit there is no spare room, so set + * it to 0; the encoded class_idx then folds to a constant 0 and + * the layout collapses back to [PFN | obj_idx]. + */ +#if BITS_PER_LONG >= 64 +#define ZS_OBJ_CLASS_BITS (CLASS_BITS + 1) +#else +#define ZS_OBJ_CLASS_BITS 0 +#endif +#define ZS_OBJ_CLASS_MASK ((_AC(1, UL) << ZS_OBJ_CLASS_BITS) - 1) + +#define ZS_OBJ_IDX_BITS (ZS_OBJ_PFN_SHIFT - ZS_OBJ_CLASS_BITS) +#define ZS_OBJ_IDX_MASK ((_AC(1, UL) << ZS_OBJ_IDX_BITS) - 1) + +static_assert(ZS_OBJ_IDX_BITS > 0, + "zsmalloc: PFN + class_idx leave no room for obj_idx"); + /* ZS_MIN_ALLOC_SIZE must be multiple of ZS_ALIGN */ #define ZS_MIN_ALLOC_SIZE \ - MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> OBJ_INDEX_BITS)) + MAX(32, (ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT >> ZS_OBJ_IDX_BITS)) /* each chunk includes extra space to keep handle */ #define ZS_MAX_ALLOC_SIZE PAGE_SIZE @@ -721,26 +760,38 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc, unsigned int *obj_idx) { - *zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS); - *obj_idx = (obj & OBJ_INDEX_MASK); + *zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT); + *obj_idx = (obj & ZS_OBJ_IDX_MASK); } static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc) { - *zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS); + *zpdesc = pfn_zpdesc(obj >> ZS_OBJ_PFN_SHIFT); +} + +/* + * On 32-bit systems ZS_OBJ_CLASS_BITS is 0 and ZS_OBJ_CLASS_MASK is 0, + * so this collapses to a constant 0. No ifdef needed at the call site. + */ +static unsigned int obj_to_class_idx(unsigned long obj) +{ + return (obj >> ZS_OBJ_IDX_BITS) & ZS_OBJ_CLASS_MASK; } /** - * location_to_obj - get obj value encoded from (, ) + * location_to_obj - encode (, , ) into obj value * @zpdesc: zpdesc object resides in zspage * @obj_idx: object index + * @class_idx: size class index; ignored on 32-bit (ZS_OBJ_CLASS_BITS == 0) */ -static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx) +static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx, + unsigned int class_idx) { unsigned long obj; - obj = zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS; - obj |= obj_idx & OBJ_INDEX_MASK; + obj = zpdesc_pfn(zpdesc) << ZS_OBJ_PFN_SHIFT; + obj |= (unsigned long)(class_idx & ZS_OBJ_CLASS_MASK) << ZS_OBJ_IDX_BITS; + obj |= obj_idx & ZS_OBJ_IDX_MASK; return obj; } @@ -1276,7 +1327,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, kunmap_local(vaddr); mod_zspage_inuse(zspage, 1); - obj = location_to_obj(m_zpdesc, obj); + obj = location_to_obj(m_zpdesc, obj, zspage->class); record_obj(handle, obj); return obj; @@ -1762,7 +1813,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page, old_obj = handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj = (unsigned long)location_to_obj(newzpdesc, obj_idx); + new_obj = location_to_obj(newzpdesc, obj_idx, + obj_to_class_idx(old_obj)); record_obj(handle, new_obj); } } -- 2.34.1