[RFC PATCH 0/3] mm/mglru: cacheline follow-ups on the reclaim hot paths

[RFC PATCH 0/3] mm/mglru: cacheline follow-ups on the reclaim hot paths

[Ridong Chen]

From: Ridong Chen <chenridong@xiaomi.com>

Joanne's series to reduce cacheline contention around struct
mem_cgroup_per_node [1], now in linux-next, made me look at the MGLRU
side of the same structures, and at the stores MGLRU does on hot paths,
to see whether any of them write a line for no reason. This RFC has the
three candidates that showed a measurable difference. They are
independent of each other.

1. lru_gen_use_mm() writes mm->lru_gen.bitmap on every context switch
   to a user task, although the bitmap is almost always already set.
   For threads of one process spread over several CPUs, the line
   bounces between them. Read first and skip the store: +2-3% context
   switches per CPU for a shared mm, no change for private mms.

2. folio_update_gen() and folio_set_lru_refs() always end with a lock
   cmpxchg, even when the flags do not change. During aging, a folio
   mapped by N processes is visited N times and only the first visit
   changes anything. Skip the no-op cmpxchg: an aging pass over a 2GiB
   file shared by 8-64 processes gets 18-22% faster.

3. lrugen->evicted[] and ->refaulted[] are updated locklessly on
   every eviction and refault, and the file type counters share a line
   with lrugen->enabled/gen/seg/list, which are read on every LRU add
   and by shrink_many(). Move the read-mostly fields up and give the
   counters their own lines: +4.3% fio IOPS with 24 jobs refaulting
   file pages in one memcg, no change at 8 jobs or fewer. This one
   costs a cacheline per memcg per node, and its gain needs many CPUs
   refaulting from a fast backing store, so I am not sure it is worth
   it on its own; feedback welcome.

All numbers are from an i7-13700HX (8P+8E, 24 threads, one socket) with
turbo disabled and the performance governor, on next-20261023 with the
same config for base and patched kernels. Details are in each patch.
The series is rebased onto next-20261104, which includes [1], without
code changes; the struct sizes in patch 3 are updated for it.

Limitations of the testing, where input would be welcome:

- Patch 1 does not show up in will-it-scale context_switch1 or
  sched_yield (explained in the patch), and perf bench sched messaging
  and schbench are too noisy here to resolve 2-3%. The numbers come
  from a small test that pins two sched_yield()ing threads per CPU,
  with a one-process-per-CPU control.

- Patch 2 has no standard benchmark either. In fio with many mmap()
  readers under memcg pressure, reclaim time goes to rmap walks on the
  eviction side and aging barely runs. The numbers come from triggering
  aging through debugfs with force_scan, which is the upper bound. On
  real systems the gain scales with how many processes in the same
  memcg share the hot file pages, e.g. preforked servers or Android
  without per-app memcgs.

- Patch 3 depends on the layout of the containing structs, which
  depends on the config (LRU_GEN_STATS changes the array sizes). I
  checked LRU_GEN_STATS=n only.

The test programs are small C files; I can post them if that helps.

[1] https://lore.kernel.org/20261023195417.1391216-1-joannelkoong@gmail.com


Ridong Chen (3):
  mm/mglru: skip redundant bitmap store in lru_gen_use_mm()
  mm/mglru: skip the flags cmpxchg when nothing changes
  mm/mglru: move lrugen atomic stats off the read-mostly lines

 include/linux/mm_inline.h |  2 ++
 include/linux/mm_types.h  |  3 ++-
 include/linux/mmzone.h    | 18 ++++++++++--------
 mm/vmscan.c               |  2 ++
 4 files changed, 16 insertions(+), 9 deletions(-)


base-commit: 8b7179210b22f8e48a82c5ceacb9d56a9c4cfec4
-- 
2.34.1

[RFC PATCH 1/3] mm/mglru: skip redundant bitmap store in lru_gen_use_mm()

[Ridong Chen]

From: Ridong Chen <chenridong@xiaomi.com>

lru_gen_use_mm() is called from context_switch() on every switch to a
user task and unconditionally sets mm->lru_gen.bitmap to -1. The bitmap
is only cleared by the page table walkers in get_next_mm() during aging,
so between aging passes it is already all ones and the store is
redundant.

The store is not free when the mm_struct is shared: every CPU running a
thread of the same process must take the cacheline in exclusive state,
so the line bounces between CPUs on every context switch. On x86,
switch_mm_irqs_off() returns early when switching between threads of
the same mm, which can make this the only shared write on that path.

Read the bitmap first and only write it when it is not already set.
The semantics are unchanged: the plain WRITE_ONCE() had no ordering
guarantees, and a walker clearing a bit concurrently can be missed in
either version.

Tested on an i7-13700HX (8P+8E cores, 24 threads), with turbo disabled
and the performance governor. On each CPU two tasks sched_yield() to
each other, either as threads of one process (shared mm) or as one
process per CPU (private mm, control). Context switches per second per
CPU, median of 5 runs, averaged over 2 boots of each kernel:

  CPUs   shared mm              private mm
         base     patched       base     patched
     1   1114766  1106870 -0.7% 1104512  1109284 +0.4%
     2   1086513  1110204 +2.2% 1110334  1112209 +0.2%
     4   1085872  1112258 +2.4% 1113146  1112467 -0.1%
     8   1085945  1111371 +2.3% 1111662  1111876 +0.0%
    16    660870   666748 +0.9%  665470   667058 +0.2%
    24    579382   596472 +2.9%  596177   598500 +0.4%

The 16 CPU case uses both SMT siblings of each P-core, which share the
L1/L2, so less of the traffic crosses cores. With the patch, the shared
mm case is within noise of the private mm case.

will-it-scale context_switch1 and sched_yield do not show a difference
(within 1% for both threads and processes): sched_yield runs one task
per CPU and never switches, and context_switch1 leaves the pipe peer
unpinned, so most switches stay on one CPU and the line does not move.

Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
---
 include/linux/mm_types.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e..3680cc0fd714 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1553,7 +1553,8 @@ static inline void lru_gen_use_mm(struct mm_struct *mm)
 	 * used since the last time it cleared the bitmap. So it might be worth
 	 * walking the page tables of this mm_struct to clear the accessed bit.
 	 */
-	WRITE_ONCE(mm->lru_gen.bitmap, -1);
+	if (READ_ONCE(mm->lru_gen.bitmap) != -1UL)
+		WRITE_ONCE(mm->lru_gen.bitmap, -1UL);
 }
 
 #else /* !CONFIG_LRU_GEN_WALKS_MMU */
-- 
2.34.1

[RFC PATCH 2/3] mm/mglru: skip the flags cmpxchg when nothing changes

[Ridong Chen]

From: Ridong Chen <chenridong@xiaomi.com>

folio_update_gen() and folio_set_lru_refs() always end with a
try_cmpxchg() on folio->flags, even when the new value equals the old
one. The locked instruction still takes the struct page cacheline in
exclusive state.

That is common in practice: a folio mapped by many processes is visited
once per mm during a page table walk, and after the first visit it is
already in the youngest generation with refs cleared and PG_workingset
set. isolate_folio() and the eviction paths also call
folio_set_lru_refs(folio, 0) on folios whose refs are already zero.

Break out of the loop when the flags would not change. The result is
the same as a successful no-op cmpxchg. The only difference is that the
full barrier implied by the locked instruction is gone on that path.
No caller relies on it: the eviction and isolation callers run under
the LRU lock, and the others (page table walks, folio_mark_accessed(),
lru_gen_clear_refs()) only care about the resulting flags value, which
is already in place. Note that lru_gen_inc_refs() already returns
without any atomic when refs is at LRU_REFS_MAX and the folio is in
the workingset.

Tested on an i7-13700HX with turbo disabled and the performance
governor. N processes map the same 2GiB tmpfs file (huge=never) and
read every page, then one aging pass is triggered on their memcg
through /sys/kernel/debug/lru_gen ("+ <memcg_id> 0 <max_seq> 200 1").
Time per aging pass in ms, median of 5 runs of 10 passes each, averaged
over 2 boots of each kernel:

  processes    base   patched
          1    32.9      31.7   -3.5%
          8   195.1     160.2  -17.9%
         32   755.8     592.2  -21.7%
         64  1597.2    1260.6  -21.1%

With one process, every visit changes the generation, so there is
nothing to skip. Before the patch, perf annotate attributes 53-58% of
the walk_update_folio() samples to the lock cmpxchg in the inlined
folio_update_gen() in this test.

Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
---
 include/linux/mm_inline.h | 2 ++
 mm/vmscan.c               | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index b3d85fe91385..4ba9079f4dc8 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -244,6 +244,8 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
 	do {
 		new_flags = old_flags;
 		lru_set_refs_flags(&new_flags, refs);
+		if (new_flags == old_flags)
+			break;
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 }
 
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 50c352e9ffef..3fa76715382d 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3398,6 +3398,8 @@ static int folio_update_gen(struct folio *folio, int new_gen, int *type,
 		lru_set_gen_flags(&new_flags, new_gen);
 		lru_set_refs_flags(&new_flags, 0);
 		new_flags |= BIT(PG_workingset);
+		if (new_flags == old_flags)
+			break;
 	} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
 
 	*type = folio_flags_is_file_lru(&old_flags);
-- 
2.34.1

[RFC PATCH 3/3] mm/mglru: move lrugen atomic stats off the read-mostly lines

[Ridong Chen]

From: Ridong Chen <chenridong@xiaomi.com>

lru_gen_eviction() and lru_gen_refault() update lrugen->evicted[] and
lrugen->refaulted[] locklessly with atomic_long_add(). The arrays sit at
the tail of struct lru_gen_folio, right before enabled/gen/seg/list:
with LRU_GEN_STATS=n the file type counters share a cacheline with them.

Those fields are read far more often than they are written:
lru_gen_add_folio() reads ->enabled on every LRU add, and shrink_many()
walks ->list and reads ->gen/->seg under RCU. With frequent file refaults
from several CPUs, every refault invalidates that line on the CPUs doing
LRU adds.

Move enabled/gen/seg/list next to max_seq/min_seq/timestamps, which are
also mostly read, and put evicted[]/refaulted[] in their own cacheline
group at the end of the struct. This grows struct lru_gen_folio from
1360 to 1408 bytes and struct mem_cgroup_per_node from 2176 to 2240
bytes on x86_64 (one more cacheline per memcg per node).

Tested on an i7-13700HX (24 threads) with turbo disabled and the
performance governor. 24 files of 128MiB on ext4 on brd, read with
fio (psync, randread, bs=4k, fadvise_hint=random, buffered), all jobs
in one memcg with memory.max=1G, so nearly every read refaults.
IOPS, median of 5 runs:

  jobs      base   patched
     1    270537    274303   +1.4%  (sd 3.7%)
     8   1461025   1466293   +0.4%
    16   2455629   2508001   +2.1%
    24   2707720   2823776   +4.3%

A custom random pread() test of the same setup, with this patch alone
against base over two boots of each kernel, shows +2.4% at 16 threads
and +5.9% at 24 threads, and no change at 8 threads or fewer. With
anonymous memory on zram, the refaults mostly hit the tier 0 counters
on a different line, and there is no measurable change.

The gain grows with the number of CPUs refaulting into the same lruvec
and needs a fast backing store; with real IO it is likely to be lost in
the noise. The patch is mostly meant to keep the lockless counters away
from the read-mostly fields regardless of where they end up.

Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
---
 include/linux/mmzone.h | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 6442e662aa4f..eb84af674e23 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -551,6 +551,14 @@ struct lru_gen_folio {
 	unsigned long min_seq[ANON_AND_FILE];
 	/* the birth time of each generation in jiffies */
 	unsigned long timestamps[MAX_NR_GENS];
+	/* whether the multi-gen LRU is enabled */
+	bool enabled;
+	/* the memcg generation this lru_gen_folio belongs to */
+	u8 gen;
+	/* the list segment this lru_gen_folio belongs to */
+	u8 seg;
+	/* per-node lru_gen_folio list for global reclaim */
+	struct hlist_nulls_node list;
 	/* the multi-gen LRU lists, lazily sorted on eviction */
 	struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
 	/* the multi-gen LRU sizes, eventually consistent */
@@ -562,16 +570,10 @@ struct lru_gen_folio {
 	/* can only be modified under the LRU lock */
 	unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	/* can be modified without holding the LRU lock */
+	__cacheline_group_begin_aligned(lrugen_stats);
 	atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
-	/* whether the multi-gen LRU is enabled */
-	bool enabled;
-	/* the memcg generation this lru_gen_folio belongs to */
-	u8 gen;
-	/* the list segment this lru_gen_folio belongs to */
-	u8 seg;
-	/* per-node lru_gen_folio list for global reclaim */
-	struct hlist_nulls_node list;
+	__cacheline_group_end_aligned(lrugen_stats);
 };
 
 enum {
-- 
2.34.1

Re: [RFC PATCH 2/3] mm/mglru: skip the flags cmpxchg when nothing changes

[Kairui Song]

On Thu, Nov 5, 2026 at 10:11 PM Ridong Chen <ridong.chen@linux.dev> wrote:
>
> From: Ridong Chen <chenridong@xiaomi.com>
>
> folio_update_gen() and folio_set_lru_refs() always end with a
> try_cmpxchg() on folio->flags, even when the new value equals the old
> one. The locked instruction still takes the struct page cacheline in
> exclusive state.
>
> That is common in practice: a folio mapped by many processes is visited
> once per mm during a page table walk, and after the first visit it is
> already in the youngest generation with refs cleared and PG_workingset
> set. isolate_folio() and the eviction paths also call
> folio_set_lru_refs(folio, 0) on folios whose refs are already zero.
>
> Break out of the loop when the flags would not change. The result is
> the same as a successful no-op cmpxchg. The only difference is that the
> full barrier implied by the locked instruction is gone on that path.
> No caller relies on it: the eviction and isolation callers run under
> the LRU lock, and the others (page table walks, folio_mark_accessed(),
> lru_gen_clear_refs()) only care about the resulting flags value, which
> is already in place. Note that lru_gen_inc_refs() already returns
> without any atomic when refs is at LRU_REFS_MAX and the folio is in
> the workingset.
>
> Tested on an i7-13700HX with turbo disabled and the performance
> governor. N processes map the same 2GiB tmpfs file (huge=never) and
> read every page, then one aging pass is triggered on their memcg
> through /sys/kernel/debug/lru_gen ("+ <memcg_id> 0 <max_seq> 200 1").
> Time per aging pass in ms, median of 5 runs of 10 passes each, averaged
> over 2 boots of each kernel:
>
>   processes    base   patched
>           1    32.9      31.7   -3.5%
>           8   195.1     160.2  -17.9%
>          32   755.8     592.2  -21.7%
>          64  1597.2    1260.6  -21.1%
>
> With one process, every visit changes the generation, so there is
> nothing to skip. Before the patch, perf annotate attributes 53-58% of
> the walk_update_folio() samples to the lock cmpxchg in the inlined
> folio_update_gen() in this test.
>
> Signed-off-by: Ridong Chen <chenridong@xiaomi.com>
> ---
>  include/linux/mm_inline.h | 2 ++
>  mm/vmscan.c               | 2 ++
>  2 files changed, 4 insertions(+)
>
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index b3d85fe91385..4ba9079f4dc8 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -244,6 +244,8 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
>         do {
>                 new_flags = old_flags;
>                 lru_set_refs_flags(&new_flags, refs);
> +               if (new_flags == old_flags)
> +                       break;
>         } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>  }
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 50c352e9ffef..3fa76715382d 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -3398,6 +3398,8 @@ static int folio_update_gen(struct folio *folio, int new_gen, int *type,
>                 lru_set_gen_flags(&new_flags, new_gen);
>                 lru_set_refs_flags(&new_flags, 0);
>                 new_flags |= BIT(PG_workingset);
> +               if (new_flags == old_flags)
> +                       break;
>         } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));

Hi Ridong,

I had the same idea in V4 of MGLRU-FG which is posted a few weeks ago,
with the same code:
https://lore.kernel.org/linux-mm/20261016-mglru-fg-v4-4-2106750c0bfa@tencent.com/

+ if (new_flags == old_flags)
+ break;

I didn't have time to benchmark that one function but the e2e test
showed very trivial improvement, it's a good improvement though. Feel
free to add:

Reviewed-by: Kairui Song <kasong@tencent.com>

BTW there are a lot of concurrent activity is happening for MGLRU at
the moment so conflicts occur very frequently...