From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A4B6175A66 for ; Fri, 17 Apr 2026 10:53:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776423221; cv=none; b=gcrNFu9/Pyjfm/4uohOA/BQAJsEBEdOIDVSnskvhUVQQmKdB/5swmERYrtgfAITLntZolysC/+K55M5GtLtJBCPFaAA2USifaWBBbIzGPqer9C+C8HVwEvW7+IRkozj9P9Pe03Ilms8oQD0wJqaxbnEXm3Uo40O7gkr5WzXE890= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776423221; c=relaxed/simple; bh=GFbjUd1khvq4RnoviERnXv4yABv1GtkT6p3RCxu6mVQ=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=FpoWplc0H6DMSFJKpz/fvMr5RYC5boSheIlHDujXq5eEkADNq6wE4Om0QgcMLuz8OEnsPYCg1WJzQUa/7QJptRp73OLSXt9VC+fG7yPHA0vYjuw0Dx9yIbJk2HVWiwJWmDEACQYR0vJ1odf4K0hINa32cf9eRPBoaT1H22mHA1g= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AwF55n+N; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AwF55n+N" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 72A6FC19425; Fri, 17 Apr 2026 10:53:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776423220; bh=GFbjUd1khvq4RnoviERnXv4yABv1GtkT6p3RCxu6mVQ=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=AwF55n+NYWhqDkXh3qykURtFUF8XHsPX0OL003qZFlT19ogtoQwiywSg2GGfqHB8V Sf48qlgJ/YyOLmXW5Qak3jdeqdOJBQYmWuQnM5Q40NltkVJs9Y3gC60qgbq4RL/oNx cw9xbKTCZNkj/V+lIEhmVy49zIFze2FdTZYJG5/WG6cRCYkepNYM1s453DAnO9+pti 5b5YZmTlAPbaV1TsMOpnRW+8JXz2t9z+KOAzXqzje4jib0HAVgBGmo6gScSN3qw5W/ 3bnLZ7LgE8fEQpoPAWxQI06/d6Yl/4vxY+11nfz53Pi+uys1b3JngJup7MioL9TZ6x qP7ts5Bpbel1A== Message-ID: Date: Fri, 17 Apr 2026 12:53:36 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v7] mm/slub: defer freelist construction until after bulk allocation from a new slab Content-Language: en-US To: hu.shengming@zte.com.cn, harry@kernel.org, akpm@linux-foundation.org Cc: hao.li@linux.dev, cl@gentwo.org, rientjes@google.com, roman.gushchin@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, zhang.run@zte.com.cn, xu.xin16@zte.com.cn, yang.tao172@zte.com.cn, yang.yang29@zte.com.cn References: <20260415165255601BBSSoil3iPkco33RXFtU2@zte.com.cn> From: "Vlastimil Babka (SUSE)" In-Reply-To: <20260415165255601BBSSoil3iPkco33RXFtU2@zte.com.cn> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 4/15/26 10:52, hu.shengming@zte.com.cn wrote: > From: Shengming Hu > > Allocations from a fresh slab can consume all of its objects, and the > freelist built during slab allocation is discarded immediately as a result. > > Instead of special-casing the whole-slab bulk refill case, defer freelist > construction until after objects are emitted from a fresh slab. > new_slab() now only allocates the slab and initializes its metadata. > refill_objects() then obtains a fresh slab and lets alloc_from_new_slab() > emit objects directly, building a freelist only for the objects left > unallocated; the same change is applied to alloc_single_from_new_slab(). > > To keep CONFIG_SLAB_FREELIST_RANDOM=y/n on the same path, introduce a > small iterator abstraction for walking free objects in allocation order. > The iterator is used both for filling the sheaf and for building the > freelist of the remaining objects. > > Also mark setup_object() inline. After this optimization, the compiler no > longer consistently inlines this helper in the hot path, which can hurt > performance. Explicitly marking it inline restores the expected code > generation. > > This reduces per-object overhead when allocating from a fresh slab. > The most direct benefit is in the paths that allocate objects first and > only build a freelist for the remainder afterward: bulk allocation from > a new slab in refill_objects(), single-object allocation from a new slab > in ___slab_alloc(), and the corresponding early-boot paths that now use > the same deferred-freelist scheme. Since refill_objects() is also used to > refill sheaves, the optimization is not limited to the small set of > kmem_cache_alloc_bulk()/kmem_cache_free_bulk() users; regular allocation > workloads may benefit as well when they refill from a fresh slab. > > In slub_bulk_bench, the time per object drops by about 32% to 70% with > CONFIG_SLAB_FREELIST_RANDOM=n, and by about 58% to 70% with > CONFIG_SLAB_FREELIST_RANDOM=y. This benchmark is intended to isolate the > cost removed by this change: each iteration allocates exactly > slab->objects from a fresh slab. That makes it a near best-case scenario > for deferred freelist construction, because the old path still built a > full freelist even when no objects remained, while the new path avoids > that work. Realistic workloads may see smaller end-to-end gains depending > on how often allocations reach this fresh-slab refill path. > > Benchmark results (slub_bulk_bench): > Machine: qemu-system-x86 -m 1024M -smp 8 -enable-kvm -cpu host > Kernel: Linux 7.0.0-rc7-next-20260407 > Config: x86_64_defconfig > Cpu: 0 > Rounds: 20 > Total: 256MB > > - CONFIG_SLAB_FREELIST_RANDOM=n - > > obj_size=16, batch=256: > before: 4.85 +- 0.08 ns/object > after: 3.30 +- 0.20 ns/object > delta: -31.9% > > obj_size=32, batch=128: > before: 6.89 +- 0.07 ns/object > after: 3.74 +- 0.06 ns/object > delta: -45.7% > > obj_size=64, batch=64: > before: 10.70 +- 0.17 ns/object > after: 4.60 +- 0.12 ns/object > delta: -57.0% > > obj_size=128, batch=32: > before: 18.69 +- 0.26 ns/object > after: 6.54 +- 1.30 ns/object > delta: -65.0% > > obj_size=256, batch=32: > before: 22.36 +- 0.24 ns/object > after: 6.61 +- 0.09 ns/object > delta: -70.5% > > obj_size=512, batch=32: > before: 20.59 +- 0.36 ns/object > after: 6.90 +- 0.15 ns/object > delta: -66.5% > > - CONFIG_SLAB_FREELIST_RANDOM=y - > > obj_size=16, batch=256: > before: 8.77 +- 0.11 ns/object > after: 3.63 +- 0.09 ns/object > delta: -58.6% > > obj_size=32, batch=128: > before: 11.59 +- 0.31 ns/object > after: 4.24 +- 0.12 ns/object > delta: -63.4% > > obj_size=64, batch=64: > before: 15.58 +- 0.51 ns/object > after: 5.32 +- 0.11 ns/object > delta: -65.9% > > obj_size=128, batch=32: > before: 22.13 +- 0.63 ns/object > after: 7.39 +- 0.20 ns/object > delta: -66.6% > > obj_size=256, batch=32: > before: 27.12 +- 0.74 ns/object > after: 7.92 +- 0.08 ns/object > delta: -70.8% > > obj_size=512, batch=32: > before: 26.92 +- 0.32 ns/object > after: 8.28 +- 0.26 ns/object > delta: -69.2% > > Link: https://github.com/HSM6236/slub_bulk_test.git > Suggested-by: Harry Yoo (Oracle) > Reviewed-by: Harry Yoo (Oracle) > Reviewed-by: Hao Li > Tested-by: Hao Li > Signed-off-by: Shengming Hu Thanks, LGTM. Will pick up to slab/for-next after 7.1-rc1 is released.