From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EC1F30171C for ; Tue, 7 Jul 2026 12:16:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783426568; cv=none; b=rtdZTTbeDlgag2FYWv+7AcW0jUtDczZV74T6W0qIF/AngSJdLKltFbCUKm/CAkTwspoyScY9iYoqSATeYIcWImo7JVoYfHR74ncab+BrHkQ2kFwdto8AcSWT+SyGoHUSc4LyrM3rgHNranCGGxoUVNQ6m8CjSBISjuKj4PqhZkY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783426568; c=relaxed/simple; bh=1Ct2VqkmkFG/EMqePDVoh4JMU9Gpsxh8xQ3nBhR8dKE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=ooDZFqEbxo/1VrsBmpXSI/XSJMGUgue0o7wK9zrjvIyr8c7pceN0fBw+AKz3yzoRjCEhE/Pbo/EYw9YOxS/mS40M0MRmjgE5+QxcMUt2bPBIUJ5mm3nZ+xfMGJr96srfU0/fI33E42AUAZuCzaZnwXicMCA+/j+UiHzp1HrU8xs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VkNEMYlT; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VkNEMYlT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0CAD21F000E9; Tue, 7 Jul 2026 12:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1783426567; bh=n4sdMnJR41YXbGd+HYc7S2MYj2TSWOvClmJSzjoaf1Q=; h=From:Date:Subject:To:Cc; b=VkNEMYlTzNDX0oRswH6ukx9qm2Cg3vUZCE1y0XEdAqEuPRJkWMD0c7F/qmLvQFJL5 HQtz6tG3jF8x6jQfScgACnd03nxq2ApQ1W4npyfDgbe6APpJtRj2b2lZhLBRZ0+HJc VlCowYKGW9b4QTaMEr4bvbkoQe7rKLhk1C1n4EfEvNOy8c7DYSXbUdgVqvRoEI4SuH mPmwUg1SU3ek1rvRD4kbhoeuHxL5XJ593VBqCgEzqKMSJt3D7YP4YGGCQuZnF91xIX Q+FdfEzT97jbzxpqRS1pZapYosKbEhfsA+hpdf+g17f27YvSLIYtxRnxjxi8hZxLBz /JKTtsTuB56sQ== From: "Vlastimil Babka (SUSE)" Date: Tue, 07 Jul 2026 14:16:00 +0200 Subject: [PATCH] mm/slab: extract __free_to_pcs_batch() from free_to_pcs_bulk() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260707-slab-simplify-bulk-pcs-v1-1-4850dbe0d904@kernel.org> X-B4-Tracking: v=1; b=H4sIAP/tTGoC/yXMQQqDMBBA0avIrDsQRYz0KtJFEid22lRDphZFv LvRLt/i/w2EEpPAvdgg0Y+FpzGjvBXgnmYcCLnPhkpVjdJKowRjUfgTA/sV7RzeGJ2gaXtV69Y T+QZyHBN5Xq5x9/hbZvsi9z1vsO8HG0lownoAAAA= X-Change-ID: 20260707-slab-simplify-bulk-pcs-a8d0478feef6 To: Harry Yoo , Andrew Morton Cc: Hao Li , Shengming Hu , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=4953; i=vbabka@kernel.org; h=from:subject:message-id; bh=1Ct2VqkmkFG/EMqePDVoh4JMU9Gpsxh8xQ3nBhR8dKE=; b=owGbwMvMwMG4+8GG0kuuHbMYT6slMWT5vGPad95ix6HD1z36NxwzL5efn+Nj4FGjJDhlzdVm7 pPiEif/djL6szAwcjBYiimyVO8+4Sg6U9ljmofvR5hBrEwgU6RFGhiAgIWBLzcxr9RIx0jPVNtQ zxDI0DFi4OIUgKle8If9f0najHf5X8+5ZPXuDNjpOlFxkoii+EXrrGBNrqaju0wv3FaW25fjm5L +xnbCoUNlXEt2/9RWXTdPpvrWtgXrFBjaujcyid0Vat/cZCfPxB0jKCdwmVfgxXTBHDfO5SKPt1 y/UHZvH8u3U0dvcvPy1K8/oPyuKDHd+EN21wOeSOdVPvIHlp5fVXkzfKvVJ7XmnT7V16/wXOBW9 c40ePlyQc+qwJdXrwrPnGi/t9wseuL5pW9mv5+5c33bioruPRFSK9/xTHr4ajZvavBh62PzxaP3 5KmZ106QPvFUPrCQTX+roUm2/859VvcNY6cFpO7RLXSYN8WEw00uOeLVxqtNoSXC/L+FZIoZp9j /Xmf1HgA= X-Developer-Key: i=vbabka@kernel.org; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 It has been noted that free_to_pcs_bulk() is difficult to follow, with a number of goto labels, and this has contributed to two memory leak bugs in there. Extract part of the code to __free_to_pcs_batch(), which focuses only on freeing free-hook-processed local objects to a percpu sheaf, and returning how many were freed. Zero means a trylock failure or no empty sheaf available, and thus the caller should fallback to __kmem_cache_free_bulk(). Make free_to_pcs_bulk() call this in a while loop, removing all goto labels from the function. __free_to_pcs_batch() retains two rather straightforward ones. Signed-off-by: Vlastimil Babka (SUSE) --- A followup refactoring to Shengming's fix. --- mm/slub.c | 116 +++++++++++++++++++++++++++++++++----------------------------- 1 file changed, 61 insertions(+), 55 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 65febe957886..3f13f497aab4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -6203,51 +6203,21 @@ static __always_inline bool can_free_to_pcs(struct slab *slab) } /* - * Bulk free objects to the percpu sheaves. - * Unlike free_to_pcs() this includes the calls to all necessary hooks - * and the fallback to freeing to slab pages. + * Try to free as many objects (already processed by free hooks) as possible to + * a single per-cpu sheaf. + * + * Returns how many objects were freed. Zero means failure and the caller should + * fall back to __kmem_cache_free_bulk(). */ -static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) +static unsigned int __free_to_pcs_batch(struct kmem_cache *s, size_t size, void **p) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *main, *empty; - bool init = slab_want_init_on_free(s); - unsigned int batch, i = 0; struct node_barn *barn; - void *remote_objects[PCS_BATCH_MAX]; - unsigned int remote_nr = 0; - - while (i < size) { - struct slab *slab = virt_to_slab(p[i]); - - memcg_slab_free_hook(s, slab, p + i, 1); - alloc_tagging_slab_free_hook(s, slab, p + i, 1); - - if (unlikely(!slab_free_hook(s, p[i], init, false))) { - p[i] = p[--size]; - continue; - } - - if (unlikely(!can_free_to_pcs(slab))) { - remote_objects[remote_nr] = p[i]; - p[i] = p[--size]; - if (++remote_nr >= PCS_BATCH_MAX) { - __kmem_cache_free_bulk(s, remote_nr, &remote_objects[0]); - stat_add(s, FREE_SLOWPATH, remote_nr); - remote_nr = 0; - } - continue; - } - - i++; - } - - if (!size) - goto flush_remote; + unsigned int batch; -next_batch: if (!local_trylock(&s->cpu_sheaves->lock)) - goto fallback; + return 0; pcs = this_cpu_ptr(s->cpu_sheaves); @@ -6293,29 +6263,65 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) stat_add(s, FREE_FASTPATH, batch); - if (batch < size) { - p += batch; - size -= batch; - goto next_batch; + return batch; + +no_empty: + local_unlock(&s->cpu_sheaves->lock); + + return 0; +} + +/* + * Bulk free objects to the percpu sheaves. + * Unlike free_to_pcs() this includes the calls to all necessary hooks + * and the fallback to freeing to slab pages. + */ +static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p) +{ + bool init = slab_want_init_on_free(s); + void *remote_objects[PCS_BATCH_MAX]; + unsigned int remote_nr = 0; + + for (unsigned int i = 0; i < size;) { + struct slab *slab = virt_to_slab(p[i]); + + memcg_slab_free_hook(s, slab, p + i, 1); + alloc_tagging_slab_free_hook(s, slab, p + i, 1); + + if (unlikely(!slab_free_hook(s, p[i], init, false))) { + p[i] = p[--size]; + continue; + } + + if (unlikely(!can_free_to_pcs(slab))) { + remote_objects[remote_nr] = p[i]; + p[i] = p[--size]; + if (++remote_nr >= PCS_BATCH_MAX) { + __kmem_cache_free_bulk(s, remote_nr, &remote_objects[0]); + stat_add(s, FREE_SLOWPATH, remote_nr); + remote_nr = 0; + } + continue; + } + + i++; } - if (remote_nr) - goto flush_remote; + while (size) { + unsigned int batch_freed = __free_to_pcs_batch(s, size, p); - return; + if (!batch_freed) + break; -no_empty: - local_unlock(&s->cpu_sheaves->lock); + p += batch_freed; + size -= batch_freed; + } - /* - * if we depleted all empty sheaves in the barn or there are too - * many full sheaves, free the rest to slab pages - */ -fallback: - __kmem_cache_free_bulk(s, size, p); - stat_add(s, FREE_SLOWPATH, size); + if (size) { + __kmem_cache_free_bulk(s, size, p); + stat_add(s, FREE_SLOWPATH, size); + } -flush_remote: if (remote_nr) { __kmem_cache_free_bulk(s, remote_nr, &remote_objects[0]); stat_add(s, FREE_SLOWPATH, remote_nr); --- base-commit: 72bb229f9161a1efcd5df32141b69fcc6ae81a13 change-id: 20260707-slab-simplify-bulk-pcs-a8d0478feef6 Best regards, -- Vlastimil Babka (SUSE)