From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B21054A43F0 for ; Fri, 18 Sep 2026 15:35:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789745719; cv=none; b=jO9X+CzuVB0imYd1RV/seQOeXhTurFANJVQA1TFiuwOlSZElQXc+Qp33ntYLtjgTleU8fxo/jNyC6IzoKu8IXuZvpWz9iThUbdaGCJQ5eUSQ26eliEqBkaeuBQIAZK3iuHBpbkCTkYOfI+mN1+Oh4W+hlvrcqaPeYoXaxiYUzx4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789745719; c=relaxed/simple; bh=ZMtc2eaIIcPrDV/2C4IMFPmHENeK/N/h7wI5x33wn7I=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=dSckq872rdCgUvl6oCccCPu/qGsEr/utAIPonLZAqPvwuY60/xIaTDiAe3LcrOJ0gffVVoWtx7WQe94jROscwLJyiLVLr7kDkefEY0dnEqp9OOe+c+afDabeyY/9WO+IgwSe1/WOdflfCkrtmp4lQC2bAM732d0PYuF6rhwIfqw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=i0MCptV/; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="i0MCptV/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D38CB1F000FF; Fri, 18 Sep 2026 15:35:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789745717; bh=os+ZB0D+705FI+w63ragnyl5EpLiV81b9g1IpQiTgrM=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=i0MCptV/YkhLNOf1Pa0kPVwKcjspNX4GrvgozU57nBs7lz9Ycdi59VUD9PiiIzgGI D+/J9j/n30fktDBJBb9Vx1E/SOqiBWMee3doQxtUMmWp578i+3MxKRyUIFRkKO24H7 4LMU67+2Bp2NLu7pvHPYFvBHkd8Yy2RPdppcOl+SVUxeO4JqZTA+8hSLp+QoCq75Jk goPNMpPzKnV2rKSzxaLUDKnGnNGR4Whz5xIsLH32zk0d9/rY9FpRb4ypeoR++/cgTs YmjqlzlwmHQokfqrQPyYIQ3LdRoNSERyP4PLIizs8WKeA7Ai6Swfr3GO0uecDLCvKX p7IWVxfVE002w== Date: Fri, 18 Sep 2026 16:35:15 +0100 From: Harry Yoo To: Hao Li Cc: vbabka@kernel.org, akpm@linux-foundation.org, cl@gentwo.org, rientjes@google.com, roman.gushchin@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] mm/slub: refill prefilled sheaves from the barn Message-ID: References: <20260918114318.124346-1-hao.li@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260918114318.124346-1-hao.li@linux.dev> On Fri, Sep 18, 2026 at 07:41:56PM +0800, Hao Li wrote: > Currently, when the prefill API refills a non-full sheaf, it takes the > objects from partial slabs and never from the full sheaves in the barn, > so once the barn's full list becomes saturated, it stays saturated. > > For objects freed via kfree_rcu(), every RCU sheaf then has to be > flushed to slabs because the barn's full list has no room. > > To fix this, let the sheaf refill from the barn first, and introduce a > partial sheaf in the barn, which holds the leftover objects. > > If no partial sheaf exists, take a full sheaf from the barn and return > it to the caller; the non-full sheaf becomes the partial sheaf, or is > put on the barn's empty list if it holds no objects. > > If a partial sheaf exists and the non-full sheaf and the partial sheaf > together reach capacity, the one holding more objects is filled up from > the other and handed out instead, so no full sheaf needs to be taken > from the barn. > > If a partial sheaf exists but the two together do not reach capacity, > take a full sheaf from the barn for the caller; the objects of the > non-full sheaf are absorbed into the partial sheaf, and the non-full > sheaf, now empty, is put on the barn's empty list. Perhaps let's explain why the slab allocator needs this partial sheaf handling only for prefilled sheaves path? while reviewing it i was wondering "why is this just not part of refill_sheaf()" I assume the reason is: usually we only refill empty sheaves because if it has at least one object, it can serve the allocation. > the gain comes from two sides: every full sheaf taken out makes room on > the barn's full list for a future rcu sheaf, and refilling from the > barn is cheaper than refilling from partial slabs under list_lock. > > note that putting the non-full sheaf on the full list instead would not > work: it would occupy room on the full list, so the list would stay > saturated and rcu_free_sheaf() would still keep flushing. [..] > --- > mm/slub.c | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 134 insertions(+), 8 deletions(-) > > diff --git a/mm/slub.c b/mm/slub.c > index 54ec12503357..0b1de6b42b61 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -3164,6 +3165,89 @@ static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn, > return empty; > } > > +/* > + * Exchange @sheaf, which holds fewer objects than requested, for a full one, > + * keeping the leftover objects in the barn's partial sheaf instead of > + * flushing them. > + * > + * Returns a full sheaf, or NULL if the barn cannot make one. > + * The returned sheaf might be @sheaf itself or a new one. > + */ > +static struct slab_sheaf *barn_replace_partial_sheaf(struct kmem_cache *s, > + struct node_barn *barn, > + struct slab_sheaf *sheaf) > +{ > + struct slab_sheaf *full = NULL, *partial; > + unsigned int to_move; > + unsigned long flags; > + > + if (!data_race(barn->nr_full) && !data_race(barn->sheaf_partial)) > + return NULL; > + > + spin_lock_irqsave(&barn->lock, flags); > + > + partial = barn->sheaf_partial; > + if (partial && partial->size + sheaf->size >= s->sheaf_capacity) { > + /* Fill the larger one to capacity from the smaller */ > + if (partial->size > sheaf->size) > + swap(partial, sheaf); Hmm but why switch sheaves when we don't have to? Sounds like we're losing cache affinity unnecessarily. I think we should try to refill from barn->sheaf_partial, or if that's not available, refill from a full sheaf, and then move the previously-full-sheaf to barn->sheaf_partial or barn->sheaf_empty. Then we'll never replace the sheaf with a new one. With that, the control flow could be simplified quite a bit. Something like this. (Warning: pseudocode, it won't compile) // refill a sheaf from barn. // return true when the sheaf becomes full // return false when the sheaf is not full bool refill_sheaf_from_barn(s, sheaf) { struct node_barn *barn = get_barn(s); struct slab_sheaf *partial; unsigned int to_move; unsigned long flags; spin_lock_irqsave(&barn->lock, flags); partial = barn->sheaf_partial; barn->sheaf_partial = NULL; if (!partial && barn->nr_full) { // grab one from full list partial = [...]; } if (!partial) // cannot refill from the barn. the caller will try // refilling from n->partial list goto done; to_move = min(s->sheaf_capacity - sheaf->size, partial->size); partial->size -= to_move; // copy `to_move` objects from `partial` to `sheaf` memcpy(...); sheaf->size += to_move; if (!partial->size) // move it to empty list else barn->sheaf_partial = partial; done: spin_unlock_irqrestore(&barn->lock, flags); return (sheaf->size == s->sheaf_capacity); } What do you think? -- Cheers, Harry / Hyeonggon