From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 881E1572680; Thu, 15 Oct 2026 18:15:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088156; cv=none; b=Dy+WWBxVvkfKXX1ocVdQX8mEyfjGo9XVLvqk32S26XOYer0YtJ84YJ6bn9YaxCSRcI9EbYdvxbUVbr15f/2WdJV4sJd8C1V6lbQi96didtZRwARjnlgdYuJUJn5Ur7P+QBoxXdLzvDHVicwtVsh43k/ER4mQfJSLOrGGCuhLPkU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088156; c=relaxed/simple; bh=8cyXuXR2bejdM1i48YPVGkhrLP7RRhUB+l6kEr3xsS4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=TjdFZ4BZFOd5KE1xbr2qexaoa6DzulmcQoylB+WGG3LNkp0wbGcX2ZOr/Txl/wpmDAigT0Cmeds467e8mSSOZZpO56uIf941Zj+qvQeIHZbrv7RstBTD4nW8D2sUVp1qnVvBPXfCRAWZtjIPj3aVdVZ74fn/ngZoUM38jtiHc/Q= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ht2xkvIU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ht2xkvIU" Received: by smtp.kernel.org (Postfix) with ESMTPS id 64E3DC2BD00; Thu, 15 Oct 2026 18:15:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1792088156; bh=8cyXuXR2bejdM1i48YPVGkhrLP7RRhUB+l6kEr3xsS4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=ht2xkvIU+2pFwdRFiiZT67dWqOOU2NbdtsTsO+mcr6ItXLYKkzOHivvV4uEEy75Wc UGi5DH10Vmf9Y/Afs/L9gQVbTE8gTNbNOnd1Bkkpt1KCwc/vTD/Fze9lf15isUhPkl prNZng0dfw7jqdCaH/oxyB2rYWijJLoB0vZaxRnc5j8eaO4KoVDajiz4bFa3V1F+ek 6OEOTh1cfSXp8RkWl4DUYSFThxsJp0J/tMcac42qQWNGliC8y4ZgCJFLQkuef+tGR7 tMneffwZpHlQ3yWFGnuqGEhOWv3BUdRM/p4ZZPjie1XvTF8+Ed4Xk7epcisS4lmUXL CFteevLki/ZeA== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50D3DCCA465; Thu, 15 Oct 2026 18:15:56 +0000 (UTC) From: Kairui Song via B4 Relay Date: Fri, 16 Oct 2026 02:16:05 +0800 Subject: [PATCH v4 16/17] mm/mglru: make folio_test_workingset() work based on folio LRU refs Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261016-mglru-fg-v4-16-2106750c0bfa@tencent.com> References: <20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com> In-Reply-To: <20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com> To: linux-mm@kvack.org, "Liam R. Howlett" , SJ Park Cc: Andrew Morton , Johannes Weiner , Muchun Song , Qi Zheng , Ying Huang , Chris Li , Baoquan He , Nico Pache , Usama Arif , Michal Hocko , Roman Gushchin , Shakeel Butt , David Hildenbrand , Lorenzo Stoakes , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Vlastimil Babka , Suren Baghdasaryan , Kemeng Shi , Nhat Pham , Youngjun Park , Zi Yan , Gregory Price , "Matthew Wilcox (Oracle)" , Baolin Wang , Ryan Roberts , Dev Jain , Lance Yang , Hugh Dickins , David Rientjes , Yu Zhao , Vernon Yang , Zicheng Wang , Chen Ridong , Tal Zussman , Kairui Song , Xiang Liu , Ehab Ababneh , zhaozhengzhuo , Lian Wang , Kunwu Chan , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, damon@lists.linux.dev, Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1792088152; l=5718; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=NZRuMZ9DHJqarWmmM6IMbKyXzp6luS4trNdMObpwCLI=; b=J/5V4RHF/wLROW7MgcSEu29RNBNNJPH06zH+vQfEGTpFxED4h+RDJibV/v5icelk6NMtSB6tg iCohZpXYNmiD81Dr07ISj0ys5x5+g+FdQy/Z62fe2TRmx53BTT8gB4+ X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song folio_test_workingset() currently tests the PG_workingset bit, which is only the second lowest access bit of the folio LRU refs count now. Folios with a higher refs count carry it in the LRU_REFS_MASK field and no longer have PG_workingset set, so the bit test misses them. folio_set_workingset() currently sets the PG_workingset bit with a plain set_bit(), which also over-promotes folios at a higher refs count: e.g. a folio at refs 4 is bumped to refs 6, advancing it toward LRU_REFS_MAX and an unintended generation promotion on the next access. Move the test into mm_inline.h as an inline helper based on folio_lru_refs(), checking refs >= LRU_REFS_WORKINGSET. The set stays a plain PG_workingset bit operation reserved for the classical LRU: under MGLRU the refs count is maintained by folio_inc_lru_refs(), and now triggers a debug WARN when called while MGLRU is fully on (the switching window is exempt, as the classical paths legitimately run alongside MGLRU then). The one PageWorkingset() user in erofs is converted to the folio helper. Under the classical LRU the refs count is not maintained, so the test falls back to the PG_workingset bit test, which is the old behavior. Tested-by: zhaozhengzhuo Signed-off-by: Kairui Song --- fs/btrfs/compression.c | 1 + fs/erofs/zdata.c | 3 ++- include/linux/mm_inline.h | 29 +++++++++++++++++++++++++++++ include/linux/page-flags.h | 2 -- mm/filemap.c | 1 + mm/page_io.c | 1 + 6 files changed, 34 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index c62b5148d5ac..57d24413265e 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index e981e371d6c2..34ee41e1ccdc 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -6,6 +6,7 @@ */ #include "compress.h" #include +#include #include #include @@ -1713,7 +1714,7 @@ static void z_erofs_submit_queue(struct z_erofs_frontend *f, DBG_BUGON(bvec.bv_len < sb->s_blocksize); } - if (unlikely(PageWorkingset(bvec.bv_page)) && + if (unlikely(folio_test_workingset(page_folio(bvec.bv_page))) && !memstall) { psi_memstall_enter(&pflags); memstall = 1; diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index fd77dbfdaf70..ae50f9880da8 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -288,6 +288,23 @@ static inline void folio_migrate_lru_refs(struct folio *new, const struct folio __folio_set_lru_refs(new, folio_lru_refs(old)); } +/** + * folio_test_workingset - Test if a folio is in the workingset. + * @folio: the folio + * + * A folio is workingset when its LRU refs count reaches + * LRU_REFS_WORKINGSET. Under the classical LRU the extra refs counter + * are always zero so this is just testing the PG_workingset bit. + * NOTE: folio_set_workingset() must not be used under MGLRU, as the + * folio refs are tracked by folio_inc_lru_refs() with rules. + * + * Return: true if the folio is workingset. + */ +static __always_inline bool folio_test_workingset(const struct folio *folio) +{ + return folio_lru_refs(folio) >= LRU_REFS_WORKINGSET; +} + #ifdef CONFIG_LRU_GEN static inline bool lru_refs_is_active(unsigned int refs) { @@ -519,6 +536,18 @@ static inline void folio_clear_lru_refs(struct folio *folio) folio_clear_referenced(folio); } +/* + * For the classical LRU only: with MGLRU the PG_workingset bit is + * part of the folio refs count maintained by folio_inc_lru_refs(), + * and a raw set would corrupt it. The switching window is exempt + * because the classical paths legitimately run alongside MGLRU then. + */ +static __always_inline void folio_set_workingset(struct folio *folio) +{ + VM_WARN_ON_ONCE(lru_gen_enabled() && !lru_gen_switching()); + set_bit(PG_workingset, folio_flags(folio, FOLIO_HEAD_PAGE)); +} + static __always_inline void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) { diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index b0ddc652e76c..f6625dc042da 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -549,8 +549,6 @@ PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD) FOLIO_FLAG(active, FOLIO_HEAD_PAGE) __FOLIO_CLEAR_FLAG(active, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(active, FOLIO_HEAD_PAGE) -PAGEFLAG(Workingset, workingset, PF_HEAD) - TESTCLEARFLAG(Workingset, workingset, PF_HEAD) PAGEFLAG(Checked, checked, PF_NO_COMPOUND) /* Used by some filesystems */ /* Xen */ diff --git a/mm/filemap.c b/mm/filemap.c index b74bc1e5015c..49e827058e14 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include diff --git a/mm/page_io.c b/mm/page_io.c index c6824fcd483e..540ae8caa932 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include -- 2.56.0