[PATCH] mm: workingset: account MGLRU pages in count_shadow_nodes()

[PATCH] mm: workingset: account MGLRU pages in count_shadow_nodes()

[Hui Zhu]

From: Hui Zhu <zhuhui@kylinos.cn>

Commit 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the
number of lru pages") switched count_shadow_nodes() to
lruvec_lru_size().  With CONFIG_MEMCG enabled, lruvec_lru_size() reads
mz->lru_zone_size, which is only maintained through
mem_cgroup_update_lru_size().  MGLRU never goes there: all of its
accounting (lru_gen_update_size(), reset_batch_size(), inc_max_seq()
and __lru_gen_reparent_memcg()) uses __update_lru_size(), which skips
the memcg array, and the classic lruvec_add/del_folio() paths return
early once lru_gen_add/del_folio() succeeded.

The four evictable LRU lists are therefore always seen as empty when
MGLRU is on, and the shadow node budget (pages >> 3) is based on slab
and unevictable pages only.  For a memcg holding 1 GiB of page cache
and 64 MiB of slab the budget drops from ~35k nodes to ~2k, so the
workingset shadow shrinker reclaims eviction tokens almost as fast as
they are created.  Refaults then find live pages instead of shadow
entries, lru_gen_refault() cannot restore the workingset state of hot
pages, and thrashing protection is lost.  This hits every memcg reclaim
and, since the root memcg is iterated as well, global reclaim too.

Fix this by adding the per-generation counters of an enabled multi-gen
LRU to the budget.  lrugen->nr_pages[] is eventually consistent and may
go transiently negative while batched updates are pending, so clamp the
sum at zero.  Keep lruvec_lru_size() as it still accounts for the
unevictable pages.

Fixes: 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the number of lru pages")
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
 mm/workingset.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/mm/workingset.c b/mm/workingset.c
index f351798e723a..da20034d28fc 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -653,6 +653,37 @@ void workingset_update_node(struct xa_node *node)
 	}
 }
 
+#ifdef CONFIG_LRU_GEN
+/*
+ * The per-generation counters are eventually consistent and may go
+ * transiently negative while batched updates are pending, so clamp
+ * the sum at zero.
+ */
+static unsigned long count_lru_gen_pages(struct lruvec *lruvec)
+{
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	long nr_pages = 0;
+	int gen, type, zone;
+
+	if (!lrugen->enabled)
+		return 0;
+
+	for (gen = 0; gen < MAX_NR_GENS; gen++) {
+		for (type = 0; type < ANON_AND_FILE; type++) {
+			long *cnt = lrugen->nr_pages[gen][type];
+
+			for (zone = 0; zone < MAX_NR_ZONES; zone++)
+				nr_pages += READ_ONCE(cnt[zone]);
+		}
+	}
+
+	if (nr_pages <= 0)
+		return 0;
+
+	return nr_pages;
+}
+#endif
+
 static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 					struct shrink_control *sc)
 {
@@ -697,6 +728,15 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 		for (pages = 0, i = 0; i < NR_LRU_LISTS; i++)
 			pages += lruvec_lru_size(lruvec, i, MAX_NR_ZONES - 1);
 
+#ifdef CONFIG_LRU_GEN
+		/*
+		 * MGLRU accounts for the evictable pages in
+		 * lrugen->nr_pages[] instead of mz->lru_zone_size, which
+		 * lruvec_lru_size() reads, so the loop above misses them.
+		 */
+		pages += count_lru_gen_pages(lruvec);
+#endif
+
 		pages += lruvec_page_state_local(
 			lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT;
 		pages += lruvec_page_state_local(
-- 
2.53.0

Re: [PATCH] mm: workingset: account MGLRU pages in count_shadow_nodes()

[Shakeel Butt]

On Mon, Aug 24, 2026 at 05:16:30PM +0800, Hui Zhu wrote:
> From: Hui Zhu <zhuhui@kylinos.cn>
> 
> Commit 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the
> number of lru pages") switched count_shadow_nodes() to
> lruvec_lru_size(). 

Instead of adding more complexity and increasing divergence between classic LRU
and MGLRU, let's fix lruvec_page_state_local for all. Usage of lruvec_lru_size
instead lruvec_page_state_local was mainly due to concern on adding upward tree
traversal on stats update side i.e. get_non_dying_memcg_[start|end].

However offlining should be rate. Let's just use get_non_dying_memcg_[start|end]
everywhere and not just v1 and measure the impact.

Shakeel

Re: [PATCH] mm: workingset: account MGLRU pages in count_shadow_nodes()

[kernel test robot]

Hi Hui,

kernel test robot noticed the following build warnings:

[auto build test WARNING on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Hui-Zhu/mm-workingset-account-MGLRU-pages-in-count_shadow_nodes/20260824-171630
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20260824091631.836473-1-hui.zhu%40linux.dev
patch subject: [PATCH] mm: workingset: account MGLRU pages in count_shadow_nodes()
config: parisc-randconfig-002-20260825 (https://download.01.org/0day-ci/archive/20260825/202608252128.XbcsGfzh-lkp@intel.com/config)
compiler: hppa-linux-gcc (GCC) 14.3.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260825/202608252128.XbcsGfzh-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202608252128.XbcsGfzh-lkp@intel.com/

All warnings (new ones prefixed by >>):

>> mm/workingset.c:657:22: warning: 'count_lru_gen_pages' defined but not used [-Wunused-function]
     657 | static unsigned long count_lru_gen_pages(struct lruvec *lruvec)
         |                      ^~~~~~~~~~~~~~~~~~~


vim +/count_lru_gen_pages +657 mm/workingset.c

   650	
   651	#ifdef CONFIG_LRU_GEN
   652	/*
   653	 * The per-generation counters are eventually consistent and may go
   654	 * transiently negative while batched updates are pending, so clamp
   655	 * the sum at zero.
   656	 */
 > 657	static unsigned long count_lru_gen_pages(struct lruvec *lruvec)
   658	{
   659		struct lru_gen_folio *lrugen = &lruvec->lrugen;
   660		long nr_pages = 0;
   661		int gen, type, zone;
   662	
   663		if (!lrugen->enabled)
   664			return 0;
   665	
   666		for (gen = 0; gen < MAX_NR_GENS; gen++) {
   667			for (type = 0; type < ANON_AND_FILE; type++) {
   668				long *cnt = lrugen->nr_pages[gen][type];
   669	
   670				for (zone = 0; zone < MAX_NR_ZONES; zone++)
   671					nr_pages += READ_ONCE(cnt[zone]);
   672			}
   673		}
   674	
   675		if (nr_pages <= 0)
   676			return 0;
   677	
   678		return nr_pages;
   679	}
   680	#endif
   681	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki