[PATCH 0/5] mm: multi-gen LRU: reduce memcg rotation overhead

[PATCH 0/5] mm: multi-gen LRU: reduce memcg rotation overhead

[Max Kellermann]

This series addresses excessive time spent rotating memcgs during
MGLRU reclaim on systems with a large number of memcgs.

On some servers with tens of thousands of cgroups, we had repeated
soft lockups and RCU stalls:

 rcu: INFO: rcu_sched self-detected stall on CPU
 rcu:         60-...!: (1118 ticks this GP) idle=ca74/1/0x4000000000000000 softirq=140067260/140067260 fqs=10
 rcu:         (t=2101 jiffies g=180256409 q=94266 ncpus=96)
 rcu: rcu_sched kthread starved for 2073 jiffies! g180256409 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=53
 rcu:         Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
 rcu: RCU grace-period kthread stack dump:
 task:rcu_sched       state:R  running task     stack:0     pid:15    tgid:15    ppid:2      task_flags:0x208040 flags:0x00000010
 Call trace:
  __switch_to+0xc4/0x108 (T)
  __schedule+0x2b4/0x978
  schedule+0x2c/0xc0
  schedule_timeout+0x7c/0x110
  rcu_gp_fqs_loop+0xec/0x4e8
  rcu_gp_kthread+0x12c/0x160
  kthread+0x118/0x1e0
  ret_from_fork+0x10/0x20
 rcu: Stack dump where RCU GP kthread last ran:
 Sending NMI from CPU 60 to CPUs 53:
 NMI backtrace for cpu 53
 CPU: 53 UID: 3758108211 PID: 4052395 Comm: php-cgi8.3 Tainted: G        W    L      6.18.52-i1-ampere #1203 NONE
 Tainted: [W]=WARN, [L]=SOFTLOCKUP
 Hardware name: Supermicro ARS-110M-NR/R12SPD-A, BIOS 1.1b 10/17/2023
 pstate: 40400009 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
 pc : handle_softirqs+0xa4/0x2e8
 lr : __do_softirq+0x18/0x20
 [...]
 Call trace:
  handle_softirqs+0xa4/0x2e8 (P)
  __do_softirq+0x18/0x20
  ____do_softirq+0x14/0x28
  call_on_irq_stack+0x30/0x48
  do_softirq_own_stack+0x20/0x38
  __irq_exit_rcu+0xc0/0x100
  irq_exit_rcu+0x14/0x28
  el1_interrupt+0x3c/0x58
  el1h_64_irq_handler+0x14/0x20
  el1h_64_irq+0x6c/0x70
  lru_gen_rotate_memcg+0x210/0x280 (P)
  shrink_node+0xb14/0xcc8
  do_try_to_free_pages+0xd0/0x508
  try_to_free_pages+0x11c/0x230
  __alloc_frozen_pages_noprof+0x4f8/0xe00
  get_free_pages_noprof+0x20/0x60
  ceph_alloc_readdir_reply_buffer+0xa4/0x160
  ceph_readdir+0x248/0xeb0
  wrap_directory_iterator+0x54/0x88
  shared_ceph_readdir+0x1c/0x28

The problem was lru_gen_rotate_memcg().  Its memcg tail insertion
walked the destination bin while holding the per-node `memcg_lru.lock`
with interrupts disabled.  With so many cgroups, the lock duration
grew too long and thousands of concurrent reclaimers were contending
for the spinlock all the time.

Reclaim can also repeatedly promote memcgs that are already in the
young generation, and scans of empty or protected memcgs can bypass
the rescheduling points in page reclaim.

This patch set contains three parts:

The first 3 patches add a cache for each bin's tail to make tail
insertion constant time.

The fourth patch skips redundant YOUNG rotations when the memcg is
already in the default segment of the current young generation.

The last patch adds rescheduling points between memcg reclaim attempts
and at bin/retry boundaries to avoid potential remaining soft lockups
under extreme pressure.

I ran a micro benchmark on branch 6.18 on a 12-core Xeon 5118: 8
concurrent workers repeatedly write `1024M swappiness=0` to the root
cgroup's `memory.reclaim` for 60 seconds, against a previously
prepared set of 1500 memory cgroups.  Each worker issues its next
request when the previous one returns. The test measures request
throughput and latency under concurrent MGLRU reclaim.  All requests
returned EAGAIN, indicating that the requested reclaim target was not
met; the reported throughput counts completed reclaim attempts, not
reclaimed bytes.

Results:

                  Requests/sec  Mean latency  Maximum latency
 Unpatched               4.56     1739.85 ms       4083.55 ms
 Patches 1-3            43.42      183.53 ms        393.15 ms
 Patches 1-4            59.22      134.03 ms        393.44 ms
 Patches 1-5            57.80      137.63 ms        412.72 ms

Constant-time insertion increases request throughput by 9.5x.
Avoiding redundant YOUNG rotations adds another 36.4%.  The full patch
series improves the unpatched throughput by a factor of 12.7.

All requests returned EAGAIN.  These results measure reclaim attempts
that did not meet their requested target.

I did not measure scheduling-latency improvements of the last patch.

Max Kellermann (5):
  include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()
  mm/vmscan: move code to lru_gen_{add,del}_memcg()
  mm/mglru: cache the last memcg node, make tail insertion constant time
  mm/mglru: avoid redundant rotations of young memcgs
  mm/mglru: reschedule between memcg reclaim attempts

 include/linux/mmzone.h        |  4 ++
 include/linux/rculist_nulls.h | 36 ++++++++++++++---
 mm/vmscan.c                   | 76 ++++++++++++++++++++++++++++++-----
 3 files changed, 101 insertions(+), 15 deletions(-)

-- 
2.47.3

[PATCH 1/5] include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()

[Max Kellermann]

Move some code from hlist_nulls_add_tail_rcu() to the new function
hlist_nulls_add_behind_rcu() for callers that already know the last
node of an `hlist_nulls` list, to allow constant-time appends.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 include/linux/rculist_nulls.h | 36 +++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/include/linux/rculist_nulls.h b/include/linux/rculist_nulls.h
index a97c3bcb1656..054e477dd093 100644
--- a/include/linux/rculist_nulls.h
+++ b/include/linux/rculist_nulls.h
@@ -115,6 +115,33 @@ static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n,
 		WRITE_ONCE(first->pprev, &n->next);
 }
 
+/**
+ * hlist_nulls_add_behind_rcu - append after the last list entry
+ * @n: the new element to add to the hash list.
+ * @last: the last element in the list.
+ *
+ * Adds the specified element after @last while permitting racing RCU
+ * traversals.  @last must be the last list entry, with its next
+ * pointer holding the terminating nulls marker.
+ *
+ * The caller must take whatever precautions are necessary
+ * (such as holding appropriate locks) to avoid racing
+ * with another list-mutation primitive, such as hlist_nulls_add_head_rcu()
+ * or hlist_nulls_del_rcu(), running on this same list.
+ * However, it is perfectly legal to run concurrently with
+ * the _rcu list-traversal primitives, such as
+ * hlist_nulls_for_each_entry_rcu(), used to prevent memory-consistency
+ * problems on Alpha CPUs.  Regardless of the type of CPU, the
+ * list-traversal primitive must be guarded by rcu_read_lock().
+ */
+static inline void hlist_nulls_add_behind_rcu(struct hlist_nulls_node *n,
+					      struct hlist_nulls_node *last)
+{
+	WRITE_ONCE(n->next, last->next);
+	WRITE_ONCE(n->pprev, &last->next);
+	rcu_assign_pointer(hlist_nulls_next_rcu(last), n);
+}
+
 /**
  * hlist_nulls_add_tail_rcu
  * @n: the element to add to the hash list.
@@ -143,13 +170,10 @@ static inline void hlist_nulls_add_tail_rcu(struct hlist_nulls_node *n,
 	for (i = h->first; !is_a_nulls(i); i = i->next)
 		last = i;
 
-	if (last) {
-		WRITE_ONCE(n->next, last->next);
-		WRITE_ONCE(n->pprev, &last->next);
-		rcu_assign_pointer(hlist_nulls_next_rcu(last), n);
-	} else {
+	if (last)
+		hlist_nulls_add_behind_rcu(n, last);
+	else
 		hlist_nulls_add_head_rcu(n, h);
-	}
 }
 
 /* after that hlist_nulls_del will work */
-- 
2.47.3

Re: [PATCH 1/5] include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()

[Paul E. McKenney]

On Mon, Oct 12, 2026 at 10:54:43PM +0200, Max Kellermann wrote:
> Move some code from hlist_nulls_add_tail_rcu() to the new function
> hlist_nulls_add_behind_rcu() for callers that already know the last
> node of an `hlist_nulls` list, to allow constant-time appends.
> 
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

I am assuming that this commit should take the same path as the rest
of them.  So:

Reviewed-by: Paul E. McKenney <paulmck@kernel.org>

> ---
>  include/linux/rculist_nulls.h | 36 +++++++++++++++++++++++++++++------
>  1 file changed, 30 insertions(+), 6 deletions(-)
> 
> diff --git a/include/linux/rculist_nulls.h b/include/linux/rculist_nulls.h
> index a97c3bcb1656..054e477dd093 100644
> --- a/include/linux/rculist_nulls.h
> +++ b/include/linux/rculist_nulls.h
> @@ -115,6 +115,33 @@ static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n,
>  		WRITE_ONCE(first->pprev, &n->next);
>  }
>  
> +/**
> + * hlist_nulls_add_behind_rcu - append after the last list entry
> + * @n: the new element to add to the hash list.
> + * @last: the last element in the list.
> + *
> + * Adds the specified element after @last while permitting racing RCU
> + * traversals.  @last must be the last list entry, with its next
> + * pointer holding the terminating nulls marker.
> + *
> + * The caller must take whatever precautions are necessary
> + * (such as holding appropriate locks) to avoid racing
> + * with another list-mutation primitive, such as hlist_nulls_add_head_rcu()
> + * or hlist_nulls_del_rcu(), running on this same list.
> + * However, it is perfectly legal to run concurrently with
> + * the _rcu list-traversal primitives, such as
> + * hlist_nulls_for_each_entry_rcu(), used to prevent memory-consistency
> + * problems on Alpha CPUs.  Regardless of the type of CPU, the
> + * list-traversal primitive must be guarded by rcu_read_lock().
> + */
> +static inline void hlist_nulls_add_behind_rcu(struct hlist_nulls_node *n,
> +					      struct hlist_nulls_node *last)
> +{
> +	WRITE_ONCE(n->next, last->next);
> +	WRITE_ONCE(n->pprev, &last->next);
> +	rcu_assign_pointer(hlist_nulls_next_rcu(last), n);
> +}
> +
>  /**
>   * hlist_nulls_add_tail_rcu
>   * @n: the element to add to the hash list.
> @@ -143,13 +170,10 @@ static inline void hlist_nulls_add_tail_rcu(struct hlist_nulls_node *n,
>  	for (i = h->first; !is_a_nulls(i); i = i->next)
>  		last = i;
>  
> -	if (last) {
> -		WRITE_ONCE(n->next, last->next);
> -		WRITE_ONCE(n->pprev, &last->next);
> -		rcu_assign_pointer(hlist_nulls_next_rcu(last), n);
> -	} else {
> +	if (last)
> +		hlist_nulls_add_behind_rcu(n, last);
> +	else
>  		hlist_nulls_add_head_rcu(n, h);
> -	}
>  }
>  
>  /* after that hlist_nulls_del will work */
> -- 
> 2.47.3
> 
> 

Re: [PATCH 1/5] include/linux/rculist_nulls: add hlist_nulls_add_behind_rcu()

[Barry Song]

On Thu, Oct 15, 2026 at 7:58 AM Paul E. McKenney <paulmck@kernel.org> wrote:
>
> On Mon, Oct 12, 2026 at 10:54:43PM +0200, Max Kellermann wrote:
> > Move some code from hlist_nulls_add_tail_rcu() to the new function
> > hlist_nulls_add_behind_rcu() for callers that already know the last
> > node of an `hlist_nulls` list, to allow constant-time appends.
> >
> > Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
>
> I am assuming that this commit should take the same path as the rest
> of them.  So:

I guess it would still go through Andrew's mm tree.

>
> Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
>

+1
Reviewed-by: Barry Song <baohua@kernel.org>

[PATCH 2/5] mm/vmscan: move code to lru_gen_{add,del}_memcg()

[Max Kellermann]

Preparation for the following patch that adds tail caching.

For simplicity, hlist_nulls_del_init_rcu() is now always used, which
is also valid during rotation, because the entry is immediately
reinserted under `memcg_lru.lock`.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 mm/vmscan.c | 36 ++++++++++++++++++++++++++++--------
 1 file changed, 28 insertions(+), 8 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index f11491ee9ed5..3d4bc12cf18e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4379,6 +4379,29 @@ enum {
 	MEMCG_LRU_YOUNG,
 };
 
+static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
+			      struct lru_gen_folio *lrugen,
+			      int gen, int bin, bool at_head)
+{
+	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
+
+	lockdep_assert_held(&memcg_lru->lock);
+
+	if (at_head)
+		hlist_nulls_add_head_rcu(&lrugen->list, head);
+	else
+		hlist_nulls_add_tail_rcu(&lrugen->list, head);
+}
+
+static void lru_gen_del_memcg(struct lru_gen_memcg *memcg_lru,
+			      struct lru_gen_folio *lrugen,
+			      int gen)
+{
+	lockdep_assert_held(&memcg_lru->lock);
+
+	hlist_nulls_del_init_rcu(&lrugen->list);
+}
+
 static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 {
 	int seg;
@@ -4409,12 +4432,9 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 	WRITE_ONCE(lruvec->lrugen.seg, seg);
 	WRITE_ONCE(lruvec->lrugen.gen, new);
 
-	hlist_nulls_del_rcu(&lruvec->lrugen.list);
-
-	if (op == MEMCG_LRU_HEAD || op == MEMCG_LRU_OLD)
-		hlist_nulls_add_head_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
-	else
-		hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[new][bin]);
+	lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, old);
+	lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, new, bin,
+			  op == MEMCG_LRU_HEAD || op == MEMCG_LRU_OLD);
 
 	pgdat->memcg_lru.nr_memcgs[old]--;
 	pgdat->memcg_lru.nr_memcgs[new]++;
@@ -4445,7 +4465,7 @@ void lru_gen_online_memcg(struct mem_cgroup *memcg)
 
 		lruvec->lrugen.gen = gen;
 
-		hlist_nulls_add_tail_rcu(&lruvec->lrugen.list, &pgdat->memcg_lru.fifo[gen][bin]);
+		lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, gen, bin, false);
 		pgdat->memcg_lru.nr_memcgs[gen]++;
 
 		spin_unlock_irq(&pgdat->memcg_lru.lock);
@@ -4479,7 +4499,7 @@ void lru_gen_release_memcg(struct mem_cgroup *memcg)
 
 		gen = lruvec->lrugen.gen;
 
-		hlist_nulls_del_init_rcu(&lruvec->lrugen.list);
+		lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, gen);
 		pgdat->memcg_lru.nr_memcgs[gen]--;
 
 		if (!pgdat->memcg_lru.nr_memcgs[gen] && gen == get_memcg_gen(pgdat->memcg_lru.seq))
-- 
2.47.3

Re: [PATCH 2/5] mm/vmscan: move code to lru_gen_{add,del}_memcg()

[Barry Song]

On Tue, Oct 13, 2026 at 4:55 AM Max Kellermann <max.kellermann@ionos.com> wrote:
>
> Preparation for the following patch that adds tail caching.
>
> For simplicity, hlist_nulls_del_init_rcu() is now always used, which
> is also valid during rotation, because the entry is immediately
> reinserted under `memcg_lru.lock`.
>
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
> ---
>  mm/vmscan.c | 36 ++++++++++++++++++++++++++++--------
>  1 file changed, 28 insertions(+), 8 deletions(-)

[...]
> +       lru_gen_del_memcg(&pgdat->memcg_lru, &lruvec->lrugen, old);
> +       lru_gen_add_memcg(&pgdat->memcg_lru, &lruvec->lrugen, new, bin,
> +                         op == MEMCG_LRU_HEAD || op == MEMCG_LRU_OLD);

T.J. series[1] removed MEMCG_LRU_OLD, i guess you will need some rebase.

[1] https://lore.kernel.org/linux-mm/20261028010611.3929937-1-tjmercier@google.com/

Best,
Barry

Re: [PATCH 2/5] mm/vmscan: move code to lru_gen_{add,del}_memcg()

[Max Kellermann]

On Sun, Nov 1, 2026 at 10:03 PM Barry Song <baohua@kernel.org> wrote:
> T.J. series[1] removed MEMCG_LRU_OLD, i guess you will need some rebase.

Aye, will do if you decide the series is worth merging.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group

[PATCH 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Max Kellermann]

Rotating or onlining a memcg currently uses hlist_nulls_add_tail_rcu()
which walks the destination bin to find its tail while holding
`memcg_lru.lock` with interrupts disabled.  With thousands of cgroups,
this takes a long time and causes severe lock contention.

Cache a pointer to each bin's last node, making tail insertion O(1).
This requires remembering the bin number (but the new struct field
occupies a hole, so it does not grow).

In a micro benchmark that creates 1500 cgroups and repeatedly writes
`1024M swappiness=0` to `memory.reclaim` in 8 threads, the request
rate increases by a factor of 9.5 and the average duration of each
write request is reduced by 89%.

All writes return EAGAIN in both runs; the benchmark measures the cost
of reclaim attempts that cannot meet their requested target, not
reclaimed throughput.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 include/linux/mmzone.h |  4 ++++
 mm/vmscan.c            | 24 +++++++++++++++++++++---
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff5416..9c1c1f153d1f 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -586,6 +586,8 @@ struct lru_gen_folio {
 	u8 gen;
 	/* the list segment this lru_gen_folio belongs to */
 	u8 seg;
+	/* the memcg bin, protected by memcg_lru.lock */
+	u8 bin;
 	/* per-node lru_gen_folio list for global reclaim */
 	struct hlist_nulls_node list;
 };
@@ -683,6 +685,8 @@ struct lru_gen_memcg {
 	unsigned long nr_memcgs[MEMCG_NR_GENS];
 	/* per-node lru_gen_folio list for global reclaim */
 	struct hlist_nulls_head	fifo[MEMCG_NR_GENS][MEMCG_NR_BINS];
+	/* last entry in each bin, or NULL if empty */
+	struct hlist_nulls_node *tail[MEMCG_NR_GENS][MEMCG_NR_BINS];
 	/* protects the above */
 	spinlock_t lock;
 };
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 3d4bc12cf18e..69976df5d537 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4384,21 +4384,37 @@ static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
 			      int gen, int bin, bool at_head)
 {
 	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
+	struct hlist_nulls_node *last = memcg_lru->tail[gen][bin];
 
 	lockdep_assert_held(&memcg_lru->lock);
 
-	if (at_head)
+	lrugen->bin = bin;
+
+	if (at_head || !last)
 		hlist_nulls_add_head_rcu(&lrugen->list, head);
 	else
-		hlist_nulls_add_tail_rcu(&lrugen->list, head);
+		hlist_nulls_add_behind_rcu(&lrugen->list, last);
+
+	if (!at_head || !last)
+		memcg_lru->tail[gen][bin] = &lrugen->list;
 }
 
 static void lru_gen_del_memcg(struct lru_gen_memcg *memcg_lru,
 			      struct lru_gen_folio *lrugen,
 			      int gen)
 {
+	const int bin = lrugen->bin;
+	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
+
 	lockdep_assert_held(&memcg_lru->lock);
 
+	/* pprev points to head->first for the only entry, otherwise prev->next. */
+	if (memcg_lru->tail[gen][bin] == &lrugen->list)
+		memcg_lru->tail[gen][bin] = lrugen->list.pprev == &head->first
+			? NULL
+			: container_of(lrugen->list.pprev, struct hlist_nulls_node, next);
+
+	/* Keep next intact for readers that still hold this entry. */
 	hlist_nulls_del_init_rcu(&lrugen->list);
 }
 
@@ -5883,8 +5899,10 @@ void lru_gen_init_pgdat(struct pglist_data *pgdat)
 	spin_lock_init(&pgdat->memcg_lru.lock);
 
 	for (i = 0; i < MEMCG_NR_GENS; i++) {
-		for (j = 0; j < MEMCG_NR_BINS; j++)
+		for (j = 0; j < MEMCG_NR_BINS; j++) {
 			INIT_HLIST_NULLS_HEAD(&pgdat->memcg_lru.fifo[i][j], i);
+			pgdat->memcg_lru.tail[i][j] = NULL;
+		}
 	}
 }
 
-- 
2.47.3

Re: [PATCH 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Ridong Chen]

On 10/13/2026 4:54 AM, Max Kellermann wrote:
> Rotating or onlining a memcg currently uses hlist_nulls_add_tail_rcu()
> which walks the destination bin to find its tail while holding
> `memcg_lru.lock` with interrupts disabled.  With thousands of cgroups,
> this takes a long time and causes severe lock contention.
> 
> Cache a pointer to each bin's last node, making tail insertion O(1).
> This requires remembering the bin number (but the new struct field
> occupies a hole, so it does not grow).
> 
> In a micro benchmark that creates 1500 cgroups and repeatedly writes
> `1024M swappiness=0` to `memory.reclaim` in 8 threads, the request
> rate increases by a factor of 9.5 and the average duration of each
> write request is reduced by 89%.
> 
> All writes return EAGAIN in both runs; the benchmark measures the cost
> of reclaim attempts that cannot meet their requested target, not
> reclaimed throughput.
> 
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
> ---
>   include/linux/mmzone.h |  4 ++++
>   mm/vmscan.c            | 24 +++++++++++++++++++++---
>   2 files changed, 25 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 94f9c3ff5416..9c1c1f153d1f 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -586,6 +586,8 @@ struct lru_gen_folio {
>   	u8 gen;
>   	/* the list segment this lru_gen_folio belongs to */
>   	u8 seg;
> +	/* the memcg bin, protected by memcg_lru.lock */
> +	u8 bin;
>   	/* per-node lru_gen_folio list for global reclaim */
>   	struct hlist_nulls_node list;
>   };
> @@ -683,6 +685,8 @@ struct lru_gen_memcg {
>   	unsigned long nr_memcgs[MEMCG_NR_GENS];
>   	/* per-node lru_gen_folio list for global reclaim */
>   	struct hlist_nulls_head	fifo[MEMCG_NR_GENS][MEMCG_NR_BINS];
> +	/* last entry in each bin, or NULL if empty */
> +	struct hlist_nulls_node *tail[MEMCG_NR_GENS][MEMCG_NR_BINS];
>   	/* protects the above */
>   	spinlock_t lock;
>   };
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 3d4bc12cf18e..69976df5d537 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -4384,21 +4384,37 @@ static void lru_gen_add_memcg(struct lru_gen_memcg *memcg_lru,
>   			      int gen, int bin, bool at_head)
>   {
>   	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
> +	struct hlist_nulls_node *last = memcg_lru->tail[gen][bin];
>   
>   	lockdep_assert_held(&memcg_lru->lock);
>   
> -	if (at_head)
> +	lrugen->bin = bin;
> +
> +	if (at_head || !last)
>   		hlist_nulls_add_head_rcu(&lrugen->list, head);
>   	else
> -		hlist_nulls_add_tail_rcu(&lrugen->list, head);
> +		hlist_nulls_add_behind_rcu(&lrugen->list, last);
> +
> +	if (!at_head || !last)
> +		memcg_lru->tail[gen][bin] = &lrugen->list;
>   }
>   
>   static void lru_gen_del_memcg(struct lru_gen_memcg *memcg_lru,
>   			      struct lru_gen_folio *lrugen,
>   			      int gen)
>   {
> +	const int bin = lrugen->bin;
> +	struct hlist_nulls_head *head = &memcg_lru->fifo[gen][bin];
> +
>   	lockdep_assert_held(&memcg_lru->lock);
>   
> +	/* pprev points to head->first for the only entry, otherwise prev->next. */
> +	if (memcg_lru->tail[gen][bin] == &lrugen->list)
> +		memcg_lru->tail[gen][bin] = lrugen->list.pprev == &head->first
> +			? NULL
> +			: container_of(lrugen->list.pprev, struct hlist_nulls_node, next);
> +
> +	/* Keep next intact for readers that still hold this entry. */
>   	hlist_nulls_del_init_rcu(&lrugen->list);
>   }
>   
> @@ -5883,8 +5899,10 @@ void lru_gen_init_pgdat(struct pglist_data *pgdat)
>   	spin_lock_init(&pgdat->memcg_lru.lock);
>   
>   	for (i = 0; i < MEMCG_NR_GENS; i++) {
> -		for (j = 0; j < MEMCG_NR_BINS; j++)
> +		for (j = 0; j < MEMCG_NR_BINS; j++) {
>   			INIT_HLIST_NULLS_HEAD(&pgdat->memcg_lru.fifo[i][j], i);
> +			pgdat->memcg_lru.tail[i][j] = NULL;
> +		}
>   	}
>   }
>   

Hi Max,

I've also submitted an RFC patch that addresses this issue.

https://lore.kernel.org/all/20251119083722.1365680-1-chenridong@huaweicloud.com/

-- 
Best regards
Ridong

Re: [PATCH 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Max Kellermann]

On Wed, Oct 14, 2026 at 5:31 AM Ridong Chen <ridong.chen@linux.dev> wrote:
> I've also submitted an RFC patch that addresses this issue.

Oh, interesting!  I did not know but your patch, but it looks very
similar to mine (but without the hlist_nulls refactoring so yours
comes with some code duplicating the hlist_nulls header).

Too bad your RFC went nowhere, that would have saved us from a
production incident...  Anyway, even if neither gets merged, I'll keep
my patches on our internal kernel fork, because they do help a lot
with our workload.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group

Re: [PATCH 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Barry Song]

On Wed, Oct 14, 2026 at 11:58 AM Max Kellermann
<max.kellermann@ionos.com> wrote:
>
> On Wed, Oct 14, 2026 at 5:31 AM Ridong Chen <ridong.chen@linux.dev> wrote:
> > I've also submitted an RFC patch that addresses this issue.
>
> Oh, interesting!  I did not know but your patch, but it looks very
> similar to mine (but without the hlist_nulls refactoring so yours
> comes with some code duplicating the hlist_nulls header).
>
> Too bad your RFC went nowhere, that would have saved us from a
> production incident...  Anyway, even if neither gets merged, I'll keep
> my patches on our internal kernel fork, because they do help a lot
> with our workload.

Hi Max, Ridong,

Let me discuss this with you and Ridong, run some tests, and get back
to you. I'd be very interested in getting this optimization upstream.
I'm also happy to discuss it with Ridong privately, as that's easier
for me.

Best Regards
Barry

Re: [PATCH 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Andrew Morton]

On Wed, 14 Oct 2026 05:58:28 +0200 Max Kellermann <max.kellermann@ionos.com> wrote:

> On Wed, Oct 14, 2026 at 5:31 AM Ridong Chen <ridong.chen@linux.dev> wrote:
> > I've also submitted an RFC patch that addresses this issue.
> 
> Oh, interesting!  I did not know but your patch, but it looks very
> similar to mine (but without the hlist_nulls refactoring so yours
> comes with some code duplicating the hlist_nulls header).
> 
> Too bad your RFC went nowhere, that would have saved us from a
> production incident...  Anyway, even if neither gets merged, I'll keep
> my patches on our internal kernel fork, because they do help a lot
> with our workload.

hm, this was a bit of a fail, sorry about that.

We had a fairly modest RFC-stage patch which would have addressed the
issues you've encountered in 6.18.x.  That got waylaid into more
ambitious re-engineering project which wouldn't have been backportable
and now this series which also won't be backportable.

So.  Can someone please propose a nice simple quick-fix which we can
backport into the -stable stream while we work on the larger
rearchitecting projects?

I think that might be Max's [5/5]?  Max, do you know whether that patch
alone will fix up 6.18.x, or do we also need to addressed that lengthy
spinlocked tail insertion?

If you think [5/5] is sufficient for -stable then please propose that as
a standalone cc:stable thing then we can look at the broader changes on
top.

Re: [PATCH 3/5] mm/mglru: cache the last memcg node, make tail insertion constant time

[Max Kellermann]

On Sun, Nov 1, 2026 at 10:50 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> So.  Can someone please propose a nice simple quick-fix which we can
> backport into the -stable stream while we work on the larger
> rearchitecting projects?
>
> I think that might be Max's [5/5]?  Max, do you know whether that patch
> alone will fix up 6.18.x, or do we also need to addressed that lengthy
> spinlocked tail insertion?
>
> If you think [5/5] is sufficient for -stable then please propose that as
> a standalone cc:stable thing then we can look at the broader changes on
> top.

Depends on your definition of "sufficient". I guess [5/5] will make
the symptom go away (the stalls), which is a good thing, but I see
this as a minor incremental improvement that doesn't solve any real
problem.

The real problem is the degenerate performance due to a silly linked
list insertion algorithm (traversing the whole list each time). So
with just [5/5], you'd still have hanging processes and abysmal
performance, no progress anywhere, but nothing would be logged. On the
positive side, not stalling RCU would at least not pull the rest of
the kernel down with it...

I repeated my benchmark, this time with Linux 6.18.60 and an isolated
run with just [5/5]:

                       No patches    Only 5/5    All patches
  Requests/s                 5.28        3.20          50.91
  Mean latency (ms)          1494        2463            154
  Max latency (ms)           3932       10467           1685
  CPU time/request (ms)      1494        2452            153

Patch [5/5] does not improve the underlying performance problem,
there's even a regression in this run which might be noise.
(My micro-benchmark does not reproduce RCU stalls and thus does not
prove the RCU stalls are gone.)

If you believe [5/5] should alone should be a stable bugfix, I'll
resubmit it to stable@.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group

[PATCH 4/5] mm/mglru: avoid redundant rotations of young memcgs

[Max Kellermann]

Concurrent reclaimers can request promotion after another reclaimer
has already moved a memcg into the young generation.  Requeueing it
into a random bin again adds list mutations without advancing the
old-generation scan.

Therefore, skip `MEMCG_LRU_YOUNG` when the memcg is already in the
default segment of the current young generation.

In my micro benchmark, this increases the request rate by 36% and
reduces the average duration of each write request by 27%.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 mm/vmscan.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 69976df5d537..922c59eb9629 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4445,6 +4445,14 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 	else
 		VM_WARN_ON_ONCE(true);
 
+	if (op == MEMCG_LRU_YOUNG && old == new && lruvec->lrugen.seg == seg)
+		/*
+		 * This memcg is already in the young generation and
+		 * reshuffling it again cannot help the scan of the
+		 * old generation.
+		 */
+		goto unlock;
+
 	WRITE_ONCE(lruvec->lrugen.seg, seg);
 	WRITE_ONCE(lruvec->lrugen.gen, new);
 
@@ -4458,6 +4466,7 @@ static void lru_gen_rotate_memcg(struct lruvec *lruvec, int op)
 	if (!pgdat->memcg_lru.nr_memcgs[old] && old == get_memcg_gen(pgdat->memcg_lru.seq))
 		WRITE_ONCE(pgdat->memcg_lru.seq, pgdat->memcg_lru.seq + 1);
 
+unlock:
 	spin_unlock_irqrestore(&pgdat->memcg_lru.lock, flags);
 }
 
-- 
2.47.3

[PATCH 5/5] mm/mglru: reschedule between memcg reclaim attempts

[Max Kellermann]

shrink_one() can return immediately for empty or protected memcgs, and
try_to_shrink_lruvec() can break out before reaching its
cond_resched().  A scan through many memcgs can repeatedly rotate
entries without giving other runnable tasks a scheduling opportunity.

Call cond_resched_tasks_rcu_qs() (as the existing shrink_lruvec() loop
does) after shrink_one(), while the memcg reference still pins the
cursor and before reacquiring the RCU read lock.  Also call it between
bins and nulls-marker retries, covering scans that skip every entry.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 mm/vmscan.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 922c59eb9629..c9d53546dc10 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5194,6 +5194,9 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 	gen = get_memcg_gen(READ_ONCE(pgdat->memcg_lru.seq));
 	bin = first_bin = get_random_u32_below(MEMCG_NR_BINS);
 restart:
+	/* Yield between bins/retries that contain only skipped entries. */
+	cond_resched_tasks_rcu_qs();
+
 	op = 0;
 	memcg = NULL;
 
@@ -5224,6 +5227,14 @@ static void shrink_many(struct pglist_data *pgdat, struct scan_control *sc)
 
 		op = shrink_one(lruvec, sc);
 
+		/*
+		 * Empty or protected memcgs can bypass the
+		 * rescheduling points in page reclaim.  The memcg
+		 * reference pins our cursor while we yield outside
+		 * the RCU read-side critical section.
+		 */
+		cond_resched_tasks_rcu_qs();
+
 		rcu_read_lock();
 
 		if (should_abort_scan(lruvec, sc))
-- 
2.47.3

Re: [PATCH 5/5] mm/mglru: reschedule between memcg reclaim attempts

[Barry Song]

On Tue, Oct 13, 2026 at 4:55 AM Max Kellermann <max.kellermann@ionos.com> wrote:
>
> shrink_one() can return immediately for empty or protected memcgs, and
> try_to_shrink_lruvec() can break out before reaching its
> cond_resched().  A scan through many memcgs can repeatedly rotate
> entries without giving other runnable tasks a scheduling opportunity.
>
> Call cond_resched_tasks_rcu_qs() (as the existing shrink_lruvec() loop
> does) after shrink_one(), while the memcg reference still pins the
> cursor and before reacquiring the RCU read lock.  Also call it between
> bins and nulls-marker retries, covering scans that skip every entry.

I'd appreciate it if you could provide more details in the
changelog. Without this patch, are we seeing an RCU stall,
a soft lockup, or something else, right?

Also, as Andrew asked [1], does this patch alone fix the
problem?

I assume patches 3/5 are needed for the performance
improvement, but patch 5/5 is also needed as a quick fix
for the stalls.

[1] https://lore.kernel.org/linux-mm/20261101135047.61fc3f209b3235ed6fc96183@linux-foundation.org/

>
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

Best,
Barry

Re: [PATCH 5/5] mm/mglru: reschedule between memcg reclaim attempts

[Max Kellermann]

On Mon, Nov 2, 2026 at 2:15 AM Barry Song <baohua@kernel.org> wrote:
> I'd appreciate it if you could provide more details in the
> changelog. Without this patch, are we seeing an RCU stall,
> a soft lockup, or something else, right?

As I said in reply to Andrew already, I havn't proved that [5/5] alone
reliably makes the stalls/lockups go away. I guess they are gone
already by fixing the underlying linked-list performance problem. I
thought [5/5] was useful as an additional scheduling point (similar
calls exist already in mm/), but I'm no MM expert, so view this last
patch as a rather uneducated suggestion by a MM newbie.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group