From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF01D433050; Sat, 3 Oct 2026 12:55:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791032118; cv=none; b=DDAaKDZiMC+hwWpTZzjx3zuWHxcM7P2GoFKwKSCU5hDC3dlVu76N5w19B+CuCEpg7w9LyrO5RVAMRzvMgk1zkAwWHwI+PJua8b/4cdMYa/430oGWDa4qFZk2OyUq/EwFGn/3+Il9SASaGlpcU/QjiK5UM6kwpW8ivF06WsRakAg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791032118; c=relaxed/simple; bh=IPogQoGTg2Xwqc3n6mc2GqluI33lLudTifJvvpPhlpw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=SUuEKyLs2cJmzoM7J5RMoJ3Obg1IXVvcOjkJ81PjeZDjZ6uZcYJtJYDPHF4OjJIfB3fLQFPmFXGftkZqZuafHjf6MGUZiLX4myoObAI7U0s2A4j45PuBgWlY7zFl3oKtoi9R9lSGhQtQY9pdWUt2L3rJDn2P6GsjMHktHYzy8IM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GpTJpzZY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GpTJpzZY" Received: by smtp.kernel.org (Postfix) with ESMTPS id AC7EAC2BCB9; Sat, 3 Oct 2026 12:55:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1791032118; bh=IPogQoGTg2Xwqc3n6mc2GqluI33lLudTifJvvpPhlpw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=GpTJpzZYnON+/dpmqdiy80NiIEElL8y0VJLPC/McG+lm7PNZRK1TPmlkZS90i7esz P9BfqJEaFxQap5pCAv1Th0wWp61j6GB72CGKpZEdcde7vU8QLINAZNK8l77xbwVxaK wbuax8Pl1d/Y4UcvysSNbGqRgFHU5pPdlpOfyX7vpgwx7mZL2CBhVUvnUEMLOy2qk4 IG6MB1f5WcUsLTBBt3+YghwcfXIfdu54i5QLjixlgqNc5De9UgKkdE7SSmelAuTtaI pqWQzV8ezMVwCHUW6vx4CYyOvEEYzCH/56bUZUxVvT8+fxhNCIdpjrvjd6+iRJ5ZDX /8T8wY7fkfZeQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97073CA5FED; Sat, 3 Oct 2026 12:55:18 +0000 (UTC) From: Kairui Song via B4 Relay Date: Sat, 03 Oct 2026 20:55:27 +0800 Subject: [PATCH RFC v3 16/17] mm/mglru: make folio_test_workingset() work based on folio LRU refs Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261003-mglru-fg-v3-16-cbd4546a5bd9@tencent.com> References: <20261003-mglru-fg-v3-0-cbd4546a5bd9@tencent.com> In-Reply-To: <20261003-mglru-fg-v3-0-cbd4546a5bd9@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Johannes Weiner , Muchun Song , Qi Zheng , Ying Huang , Chris Li , Baoquan He , Nico Pache , Usama Arif , Michal Hocko , Roman Gushchin , Shakeel Butt , David Hildenbrand , Lorenzo Stoakes , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Vlastimil Babka , Suren Baghdasaryan , Kemeng Shi , Nhat Pham , Youngjun Park , Zi Yan , Gregory Price , "Matthew Wilcox (Oracle)" , Baolin Wang , Ryan Roberts , Dev Jain , Lance Yang , Hugh Dickins , SeongJae Park , David Rientjes , Yu Zhao , Vernon Yang , Zicheng Wang , Chen Ridong , Tal Zussman , Kairui Song , Xiang Liu , Ehab Ababneh , zhaozhengzhuo , Lian Wang , Kunwu Chan , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1791032114; l=5993; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=0UKPioOHWojVarJEQFJShU55Y1qVkRI23X5MnhRd4eE=; b=neXes4moJ7Eui1l3feEYxn8CVM5V6KsutIso83WMYFlrcGy8zOg81T0UQ2k5gTQC6RkuwIpgR RubQN6qlf5uD6eAOeivvYyb9cLzlnyz2/z6eakf+mInZhePecTUw+2T X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song folio_test_workingset() currently tests the PG_workingset bit, which is only the second lowest access bit of the folio LRU refs count now. Folios with a higher refs count carry it in the LRU_REFS_MASK field and no longer have PG_workingset set, so the bit test misses them. folio_set_workingset() currently sets the PG_workingset bit with a plain set_bit(), which also over-promotes folios at a higher refs count: e.g. a folio at refs 4 is bumped to refs 6, advancing it toward LRU_REFS_MAX and an unintended generation promotion on the next access. Move the test into mm_inline.h as an inline helper based on folio_lru_refs(), checking refs >= LRU_REFS_WORKINGSET. The set stays a plain PG_workingset bit operation reserved for the classical LRU: under MGLRU the refs count is maintained by folio_inc_lru_refs(), and now triggers a debug WARN when called while MGLRU is fully on (the switching window is exempt, as the classical paths legitimately run alongside MGLRU then). The one PageWorkingset() user in erofs is converted to the folio helper. Under the classical LRU the refs count is not maintained, so the test falls back to the PG_workingset bit test, which is the old behavior. Signed-off-by: Kairui Song --- fs/btrfs/compression.c | 1 + fs/erofs/zdata.c | 3 ++- include/linux/mm_inline.h | 35 +++++++++++++++++++++++++++++++++++ include/linux/page-flags.h | 2 -- mm/filemap.c | 1 + mm/page_io.c | 1 + 6 files changed, 40 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index c62b5148d5ac..57d24413265e 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index e981e371d6c2..34ee41e1ccdc 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -6,6 +6,7 @@ */ #include "compress.h" #include +#include #include #include @@ -1713,7 +1714,7 @@ static void z_erofs_submit_queue(struct z_erofs_frontend *f, DBG_BUGON(bvec.bv_len < sb->s_blocksize); } - if (unlikely(PageWorkingset(bvec.bv_page)) && + if (unlikely(folio_test_workingset(page_folio(bvec.bv_page))) && !memstall) { psi_memstall_enter(&pflags); memstall = 1; diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index b68d68101248..e651f60cb914 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -258,6 +258,24 @@ static inline bool lru_gen_enabled(void) } #endif +/** + * folio_test_workingset - Test if a folio is in the workingset. + * @folio: the folio + * + * A folio is workingset when its LRU refs count reaches + * LRU_REFS_WORKINGSET. Under the classical LRU the refs count never + * goes above it, so this is just testing the PG_workingset bit. + * NOTE: folio_set_workingset() must not be used under MGLRU, as the + * folio refs are tracked by folio_inc_lru_refs(), it triggers a debug + * WARN instead for MGLRU. + * + * Return: true if the folio is workingset. + */ +static __always_inline bool folio_test_workingset(const struct folio *folio) +{ + return folio_lru_refs(folio) >= LRU_REFS_WORKINGSET; +} + static inline bool lru_gen_in_fault(void) { return current->in_lru_fault; @@ -445,6 +463,11 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, { return false; } + +static inline bool folio_test_workingset(const struct folio *folio) +{ + return test_bit(PG_workingset, const_folio_flags(folio, FOLIO_HEAD_PAGE)); +} #endif /* CONFIG_LRU_GEN */ /** @@ -473,6 +496,18 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio) } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); } +/* + * For the classical LRU only: under MGLRU the PG_workingset bit is + * part of the folio refs count maintained by folio_inc_lru_refs(), + * and a raw set would corrupt it. The switching window is exempt + * because the classical paths legitimately run alongside MGLRU then. + */ +static __always_inline void folio_set_workingset(struct folio *folio) +{ + VM_WARN_ON_ONCE(lru_gen_enabled() && !lru_gen_switching()); + set_bit(PG_workingset, folio_flags(folio, FOLIO_HEAD_PAGE)); +} + static __always_inline void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) { diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index b0ddc652e76c..f6625dc042da 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -549,8 +549,6 @@ PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD) FOLIO_FLAG(active, FOLIO_HEAD_PAGE) __FOLIO_CLEAR_FLAG(active, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(active, FOLIO_HEAD_PAGE) -PAGEFLAG(Workingset, workingset, PF_HEAD) - TESTCLEARFLAG(Workingset, workingset, PF_HEAD) PAGEFLAG(Checked, checked, PF_NO_COMPOUND) /* Used by some filesystems */ /* Xen */ diff --git a/mm/filemap.c b/mm/filemap.c index b74bc1e5015c..49e827058e14 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include diff --git a/mm/page_io.c b/mm/page_io.c index c6824fcd483e..540ae8caa932 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include -- 2.55.0