From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12261381EA3 for ; Tue, 18 Aug 2026 22:59:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093964; cv=none; b=aFalkc/LEPOgk3zUYfz3Dm2rc5T1yjhwG4dVoKXF+B5mrx+BqOdd5hfbyriRqR/vmz5vbQYaf9w/4F25RYHVNX7CX7zB2UyvbBKSFYAFSSsjQImI+4kTvUN+2PAR49/dc8QiW8uZV5/1rlUpLRvPgXhwai7xI4NQcjiku+AIE+k= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093964; c=relaxed/simple; bh=vxbSwwoLKzRzirA80Nl45YrIWdi8uFuuKfYYc+bCIoA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T9GB8r2A1/PCsHZrt/cpPLVS7u5QogKblEJ1UZ4lUyZMiPrGY2tQcHnU0Fh0/2dSo/CYyJPaCpWHC+wlPdT9YQOlzuavUxmzEOCSnV7uQ8TPy2yp0oXuWjJ5eCHi87qpqDshaBpVSsN1kI5jTl5/xBNaMldh9Y0hH1VjyP9kBwU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TyX8nUis; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TyX8nUis" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DA8E1F00A3D; Tue, 18 Aug 2026 22:59:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787093962; bh=HX1DWIadQ6nIRR1fyXkWC0pb34Vk39QSCu5CRyVrb0A=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=TyX8nUis49ca4jtpbJ98TM2fg/c4aU445zPNr7AmIN7lFk2ZW8SS9+OnUsAYqofR9 Yy9kscqbH5fQFNtNq19IH2IPrSk/zdBlAGEa5PrDfnxFt/yWNSe//1UQ0+rdd2dHnQ USWpKcr5QBR84gMw/mwE9X0cnqHvFO/7p7NQ+77CXfLHm7o6RFTQ3ptqpydBk2d/1i 46CL2IasQhIqDnFrQJoXtI7QrzOF1FttzzVk+yFPTrqTq3uh6vVaKxqbdMJ4feI2wM qJitG39O7M/ngE89xcIjRj5En6Rj4Gc3Ebq/HqBkc7Hkrr6wFwVvFbad8xdXV8PNvO LC5MinjdK4MxA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, david@kernel.org, dev.jain@arm.com, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, ljs@kernel.org, mhocko@suse.com, npache@redhat.com, rppt@kernel.org, ryan.roberts@arm.com, surenb@google.com, vbabka@kernel.org, ziy@nvidia.com, hughd@google.com, ackerleytng@google.com, usama.arif@linux.dev, joannelkoong@gmail.com, hannes@cmpxchg.org, "Barry Song (Xiaomi)" Subject: [RFC PATCH v3 1/4] mm: allow smaller large folios to use lru_cache Date: Wed, 19 Aug 2026 06:59:01 +0800 Message-Id: <20260818225904.55236-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260818225904.55236-1-baohua@kernel.org> References: <20260818225904.55236-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit For systems that primarily use smaller-order large folios, enabling the lru_cache can help reduce lock contention. For higher-order large folios, the number of folios involved is likely to be smaller, making lock contention less significant. This patch enables the lru_cache for large folios whose `nr_pages` is smaller than `FOLIO_BATCH_SIZE`. To avoid holding too many pages in the lru_cache, which could affect accounting and reclamation, we also limit the total number of pages in the cache to `FOLIO_BATCH_SIZE`. To track the number of pages, this patch adds an `unsigned short nr_pages` field to `struct folio_batch`. It cannot overflow because the batch contains at most `FOLIO_BATCH_SIZE` folios, each of which has fewer than `FOLIO_BATCH_SIZE` pages. For non-LRU caches, `folio_batch` only needs to track the number of folios, so `nr_pages` is left at zero. Signed-off-by: Barry Song (Xiaomi) --- include/linux/folio_batch.h | 25 +++++++++++++++++++++++++ mm/folio.c | 10 +++++++++- mm/internal.h | 4 ++-- 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/include/linux/folio_batch.h b/include/linux/folio_batch.h index b45946adc50b..ffc7de091fa3 100644 --- a/include/linux/folio_batch.h +++ b/include/linux/folio_batch.h @@ -10,6 +10,7 @@ #define _LINUX_FOLIO_BATCH_H #include +#include /* 31 pointers + header align the folio_batch structure to a power of two */ #define FOLIO_BATCH_SIZE 31 @@ -28,6 +29,7 @@ struct folio; struct folio_batch { unsigned char nr; unsigned char i; + unsigned short nr_pages; bool percpu_pvec_drained; struct folio *folios[FOLIO_BATCH_SIZE]; }; @@ -42,6 +44,7 @@ static inline void folio_batch_init(struct folio_batch *fbatch) { fbatch->nr = 0; fbatch->i = 0; + fbatch->nr_pages = 0; fbatch->percpu_pvec_drained = false; } @@ -49,6 +52,7 @@ static inline void folio_batch_reinit(struct folio_batch *fbatch) { fbatch->nr = 0; fbatch->i = 0; + fbatch->nr_pages = 0; } static inline unsigned int folio_batch_count(const struct folio_batch *fbatch) @@ -78,6 +82,27 @@ static inline unsigned folio_batch_add(struct folio_batch *fbatch, return folio_batch_space(fbatch); } +/** + * folio_batch_add_lru_cache() - Add a folio to a batch of lru_cache + * @fbatch: The folio batch. + * @folio: The folio to add. + * + * The folio is added to the end of the batch. + * The batch must have previously been initialised using folio_batch_init(). + * + * Return: 0 if the lru_cache is filled with more than FOLIO_BATCH_SIZE + * pages; otherwise, the number of available slots. + */ +static inline unsigned folio_batch_add_lru_cache(struct folio_batch *fbatch, + struct folio *folio) +{ + fbatch->folios[fbatch->nr++] = folio; + fbatch->nr_pages += (unsigned short)folio_nr_pages(folio); + if (fbatch->nr_pages > FOLIO_BATCH_SIZE) + return 0; + return folio_batch_space(fbatch); +} + /** * folio_batch_next - Return the next folio to process. * @fbatch: The folio batch being processed. diff --git a/mm/folio.c b/mm/folio.c index 59c477120b9a..e5820d7263e8 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -219,7 +219,7 @@ static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch, else local_lock(&cpu_fbatches.lock); - if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || + if (!folio_batch_add_lru_cache(this_cpu_ptr(fbatch), folio) || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); @@ -981,6 +981,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) int i, j; struct lruvec *lruvec = NULL; unsigned long flags = 0; + unsigned long nr_pages = 0; for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; @@ -1020,6 +1021,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) if (j != i) folios->folios[j] = folio; + nr_pages += folio_nr_pages(folio); j++; } if (lruvec) @@ -1030,6 +1032,12 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) } folios->nr = j; + /* + * For lru_cache, track the number of pages; for non-LRU caches, + * folio_batch->nr_pages is always 0. + */ + if (folios->nr_pages > 0) + folios->nr_pages = nr_pages; mem_cgroup_uncharge_folios(folios); free_unref_folios(folios); } diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c9..06adf78e13a2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -48,9 +48,9 @@ static inline bool folio_may_be_lru_cached(const struct folio *folio) /* * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. * Holding small numbers of low-order mTHP folios in per-CPU LRU cache - * will be sensible, but nobody has implemented and tested that yet. + * will be sensible. */ - return !folio_test_large(folio); + return folio_nr_pages(folio) < FOLIO_BATCH_SIZE; } static inline void lru_cache_enable(void) -- 2.34.1