From: Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org>
To: linux-mm@kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Muchun Song <muchun.song@linux.dev>,
Qi Zheng <qi.zheng@linux.dev>,
Ying Huang <ying.huang@linux.alibaba.com>,
Chris Li <chrisl@kernel.org>, Baoquan He <baoquan.he@linux.dev>,
Nico Pache <nico.pache@linux.dev>,
Usama Arif <usama.arif@linux.dev>,
Michal Hocko <mhocko@kernel.org>,
Roman Gushchin <roman.gushchin@linux.dev>,
Shakeel Butt <shakeel.butt@linux.dev>,
David Hildenbrand <david@kernel.org>,
Lorenzo Stoakes <ljs@kernel.org>,
"Liam R. Howlett" <liam@infradead.org>,
Barry Song <baohua@kernel.org>,
Axel Rasmussen <axelrasmussen@google.com>,
Yuanchu Xie <yuanchu@google.com>, Wei Xu <weixugc@google.com>,
Vlastimil Babka <vbabka@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Nhat Pham <nphamcs@gmail.com>,
Youngjun Park <youngjun.park@lge.com>, Zi Yan <ziy@nvidia.com>,
Gregory Price <gourry@gourry.net>,
"Matthew Wilcox (Oracle)" <willy@infradead.org>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Ryan Roberts <ryan.roberts@arm.com>, Dev Jain <dev.jain@arm.com>,
Lance Yang <lance.yang@linux.dev>,
Hugh Dickins <hughd@google.com>, SJ Park <sj@kernel.org>,
David Rientjes <rientjes@google.com>,
Yu Zhao <yuzhao@google.com>, Vernon Yang <vernon2gm@gmail.com>,
Zicheng Wang <wangzicheng@honor.com>,
Chen Ridong <chenridong@xiaomi.com>,
Tal Zussman <tz2294@columbia.edu>,
Kairui Song <ryncsn@gmail.com>,
Xiang Liu <liuxiang.277@bytedance.com>,
Ehab Ababneh <ehab.ababneh@intel.com>,
Zhao Zhenzhuo <zhaozhengzhuo@uniontech.com>,
Lian Wang <lianux.mm@gmail.com>,
Joanne Koong <joannelkoong@gmail.com>,
Kunwu Chan <kunwu.chan@gmail.com>, Hui Zhu <hui.zhu@linux.dev>,
linux-kernel@vger.kernel.org, cgroups@vger.kernel.org,
damon@lists.linux.dev, linux-doc@vger.kernel.org,
Kairui Song <kasong@tencent.com>
Subject: [PATCH v5 10/17] mm/mglru, smaps: convert to LRU refs based operations for MGLRU
Date: Thu, 19 Nov 2026 05:03:42 +0800 [thread overview]
Message-ID: <20261119-mglru-fg-v5-10-9eee2fe97a2d@tencent.com> (raw)
In-Reply-To: <20261119-mglru-fg-v5-0-9eee2fe97a2d@tencent.com>
From: Kairui Song <kasong@tencent.com>
PG_referenced is only the lowest bit of the folio LRU refs count
(LRU_REFS_FLAGS) now, and MGLRU never kept it stable: promotion used to
clear it, and now it toggles as the count grows. But with the LRU
referenced count as a stable API now, we can use two helpers to provide
in-kernel users a stable and sane reading of the referenced status of
an LRU managed folio.
Add folio_check_lru_refs() and folio_clear_lru_refs(), which use the refs
count when MGLRU is enabled and PG_referenced otherwise, and switch the
smaps paths over to them. The lru_gen_enabled() test stays inside the
helpers, so its static key is not splattered over the few call sites.
Convert smaps as the first user of them. MGLRU now reports every folio
with a nonzero count, so the reading no longer flickers and clearing no
longer results in an ambiguous status.
The classical LRU case is unchanged.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
fs/proc/task_mmu.c | 6 +++---
include/linux/mm_inline.h | 31 +++++++++++++++++++++++++++++++
2 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 22361607020d..f76a37d052b1 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -889,7 +889,7 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
mss->resident += size;
/* Accumulate the size in pages that have been accessed. */
- if (young || folio_test_young(folio) || folio_test_referenced(folio))
+ if (young || folio_test_young(folio) || folio_check_lru_refs(folio))
mss->referenced += size;
/*
@@ -1686,7 +1686,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
/* Clear accessed and referenced bits. */
pmdp_test_and_clear_young(vma, addr, pmd);
folio_test_clear_young(folio);
- folio_clear_referenced(folio);
+ folio_clear_lru_refs(folio);
out:
spin_unlock(ptl);
return 0;
@@ -1715,7 +1715,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
/* Clear accessed and referenced bits. */
ptep_test_and_clear_young(vma, addr, pte);
folio_test_clear_young(folio);
- folio_clear_referenced(folio);
+ folio_clear_lru_refs(folio);
}
pte_unmap_unlock(pte - 1, ptl);
cond_resched();
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 3323a3c7336b..14aab2b1ed08 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -508,6 +508,37 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
+/**
+ * folio_check_lru_refs - Test if an LRU managed folio has been referenced.
+ * @folio: The folio to test.
+ *
+ * Returns true if the MGLRU refs count is nonzero, or PG_referenced is set,
+ * which is also correct for the non-MGLRU case. Kept as a separate helper,
+ * to avoid splattering the lru_gen_enabled() static key.
+ */
+static inline bool folio_check_lru_refs(const struct folio *folio)
+{
+ if (lru_gen_enabled())
+ return folio_lru_refs(folio) > 0;
+ return folio_test_referenced(folio);
+}
+
+/**
+ * folio_clear_lru_refs - Clear the referenced state of an LRU managed folio.
+ * @folio: The folio to clear.
+ *
+ * Reset the MGLRU refs count to zero, or clear PG_referenced, leaving
+ * PG_workingset alone for the non-MGLRU case. Kept as a separate helper
+ * just like folio_check_lru_refs().
+ */
+static inline void folio_clear_lru_refs(struct folio *folio)
+{
+ if (lru_gen_enabled())
+ folio_set_lru_refs(folio, 0);
+ else
+ folio_clear_referenced(folio);
+}
+
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
--
2.56.1
next prev parent reply other threads:[~2026-11-18 21:03 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 21:03 [PATCH v5 00/17] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 01/17] mm/mglru: rename the generation offset in lru_gen_folio_seq() Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 02/17] mm/memcontrol: allow update of LRU statistic without holding LRU lock Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 03/17] mm/mglru: make generation page counters atomic Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 04/17] mm/memcg: add folio-based lruvec live helper Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 05/17] mm/mglru: skip the flags cmpxchg when nothing changes Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 06/17] mm/mglru: frequency guided workingset promotion (MGLRU-FG) Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 07/17] mm/mglru: make folio lru referenced times count a generic API Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 08/17] mm/mglru: move add/del LRU size accounting out of lru_gen_update_size() Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 09/17] mm/mglru, gup: mark folios referenced via a fast helper Kairui Song via B4 Relay
2026-11-18 21:03 ` Kairui Song via B4 Relay [this message]
2026-11-18 21:03 ` [PATCH v5 11/17] mm/damon: convert to LRU refs based operations for MGLRU Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 12/17] mm/khugepaged: " Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 13/17] mm/madvise: adapt for " Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 14/17] mm/huge_memory: mark folio as accessed more accurately on split Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 15/17] mm/mglru: folio LRU refs based active/inactive number accounting Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 16/17] mm/mglru: make folio_inc_lru_refs lruvec lockless Kairui Song via B4 Relay
2026-11-18 21:03 ` [PATCH v5 17/17] mm/mglru: make folio_test_workingset() work based on folio LRU refs Kairui Song via B4 Relay
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119-mglru-fg-v5-10-9eee2fe97a2d@tencent.com \
--to=devnull+kasong.tencent.com@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=baohua@kernel.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=baoquan.he@linux.dev \
--cc=cgroups@vger.kernel.org \
--cc=chenridong@xiaomi.com \
--cc=chrisl@kernel.org \
--cc=damon@lists.linux.dev \
--cc=david@kernel.org \
--cc=dev.jain@arm.com \
--cc=ehab.ababneh@intel.com \
--cc=gourry@gourry.net \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=hui.zhu@linux.dev \
--cc=joannelkoong@gmail.com \
--cc=kasong@tencent.com \
--cc=kunwu.chan@gmail.com \
--cc=lance.yang@linux.dev \
--cc=liam@infradead.org \
--cc=lianux.mm@gmail.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=liuxiang.277@bytedance.com \
--cc=ljs@kernel.org \
--cc=mhocko@kernel.org \
--cc=muchun.song@linux.dev \
--cc=nico.pache@linux.dev \
--cc=nphamcs@gmail.com \
--cc=qi.zheng@linux.dev \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=ryan.roberts@arm.com \
--cc=ryncsn@gmail.com \
--cc=shakeel.butt@linux.dev \
--cc=sj@kernel.org \
--cc=surenb@google.com \
--cc=tz2294@columbia.edu \
--cc=usama.arif@linux.dev \
--cc=vbabka@kernel.org \
--cc=vernon2gm@gmail.com \
--cc=wangzicheng@honor.com \
--cc=weixugc@google.com \
--cc=willy@infradead.org \
--cc=ying.huang@linux.alibaba.com \
--cc=youngjun.park@lge.com \
--cc=yuanchu@google.com \
--cc=yuzhao@google.com \
--cc=zhaozhengzhuo@uniontech.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®