From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAD6FC3DA7A for ; Tue, 20 Dec 2022 20:59:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234343AbiLTU7v (ORCPT ); Tue, 20 Dec 2022 15:59:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234201AbiLTU7t (ORCPT ); Tue, 20 Dec 2022 15:59:49 -0500 Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABC565592 for ; Tue, 20 Dec 2022 12:59:46 -0800 (PST) Date: Tue, 20 Dec 2022 12:59:40 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1671569984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Y37zBw2z34AekRtu6QE+ZHXpaydFvXhSYB4OwYr6Iew=; b=k2tEgNCeJh1eRpPJ92swxPAVLxAUrukjoi5xvKpU7tyQKrSgio9+6kQaTDBhZn2wjzF03T laGIVhKyL9PhsRhc8uTgtEG+aRnANgJE0Rdz/aGTKAp3GeL8i+VdmfooHGoTAvO8nIPERh X7XL41kR1leDKMbChd4CqsUnhf5Rtvc= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Roman Gushchin To: Shakeel Butt Cc: Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Johannes Weiner , Michal Hocko , Muchun Song , Andrew Morton , Waiman Long , Sven Luther Subject: Re: [PATCH RFC] ipc/mqueue: introduce msg cache Message-ID: References: <20221220184813.1908318-1-roman.gushchin@linux.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Dec 20, 2022 at 11:53:25AM -0800, Shakeel Butt wrote: > +Vlastimil > > On Tue, Dec 20, 2022 at 10:48 AM Roman Gushchin > wrote: > > > > Sven Luther reported a regression in the posix message queues > > performance caused by switching to the per-object tracking of > > slab objects introduced by patch series ending with the > > commit 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all > > allocations"). > > > > To mitigate the regression cache allocated mqueue messages on a small > > percpu cache instead of releasing and re-allocating them every time. > > > > Seems fine with me but I am wondering what is stopping us to do this > caching in the slab layer for all accounted allocations? Does this > only make sense for specific scenarios/use-cases? It's far from trivial, unfortunately. Here we have an mqueue object to associate a percpu cache with and the hit rate is expected to be high, assuming the mqueue will be used to pass a lot of messages. With a generic slab cache we return to the necessity of managing the per-cgroup x per-slab-cache x per-cpu free list (or some other data structure), which is already far from trivial, based on the previous experience. It can easily lead to a significant memory waste, which will fully compensate all perf wins. So probably we need some heuristics to allocate caches only for really hot slab caches and use some sort of a hash map (keyed by cgroup and slab cache) to access freelists. What I miss to commit more time to this project (aside from not having it), is the lack of real workloads which will noticeably win from this work. Sven provided a good example and benchmark to reproduce the regression, so it was easy to justify the work. Thanks!