From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BB6F4398F1 for ; Wed, 12 Aug 2026 12:19:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537145; cv=none; b=ihlSG4j5DiX/XzrszTQkj0vGso4522RMKZdql8+K7bmWTBoRx+lWFEoxJpOMkay620wqb3J17dNZidKSoWddmYzCoTTV1S2FeTsi4fdo8ZhV0CaEu/qbIB5UFeit/n5l9vpaJHGKSfdD6SHbI5RkfW/eTkI9eqlVlsu28nhkuZA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537145; c=relaxed/simple; bh=5Vi0aACqDNHGS+E+qWnp4LuFVGcj4UIZjf0cvj5xagQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BYH2BmpCPemSVfgS2rogH/c/Lx29jzK0f5H4Lkk6bqzdRHLn1DPIFWghz800FOmp2WtX1JYX/gGPhQtN+ouyVBDbgjjAsajetu6w5+eVJslM0eLcwayEBSG/jRrcegEP2fiy7Ij/HLzDxOonn3tN3EXKDFv9Jc4kzi9YCkai3ks= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LBhdAmMP; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LBhdAmMP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B32641F00A3A; Wed, 12 Aug 2026 12:18:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537140; bh=EjEFNnCpD4D3Lv/lQ5ONSKNhGjrzUmBvcQ2L3a1JEFU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LBhdAmMPlplf3Bg92sH4iyezEuXbZq+Aty9njEgnfcU6LnlXUhz4A21zPKV9HLkMl ER86pfSDbHephWBRVfUOxQvwVHdGdvrvYsrFhzgMiiCK61wNOrHku4XGoc7hmcaY/o ZX7/uoYReatWEjiHsybiI12sdGVsB4Ms0nZx6VgVrbriv28ehJ8nwzBX9i5MToUJhb x07/d5PhUaaUT5WGDgYJzmegz+wWP2umpkoX0Hh7gr1a0XU+3a6NmK9/M6lkkkciXo FBsv1wRRSEbgWzyGCbtY/bpL1I4z3uiN07VoDfmKzkjIESjlPG09aV94gLQz9wua8X FykVKjgdS6+cg== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, Barry Song Subject: [RFC PATCH v4 14/16] mm/mglru: run aging when pages are severely imbalanced across gens Date: Wed, 12 Aug 2026 20:16:56 +0800 Message-Id: <20260812121658.69965-15-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: lyugaofei This partially restores the reclaim behavior introduced in Yu Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU: minimal implementation"): /* * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) * of the total number of pages for each generation. A reasonable range * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The * aging cares about the upper bound of hot pages, while the eviction * cares about the lower bound of cold pages. */ if (young * MIN_NR_GENS > total) return true; if (old * (MIN_NR_GENS + 2) < total) return true; But with a stricter condition: the younger generations must contain at least MAX_NR_GENS times as many folios as the older generations. We also consider the cost of inc_min_seq(). If the oldest generation of the other type has fallen significantly behind, pulling those folios from the oldest generation to the second oldest generation can be very expensive. In this case, skip imbalance aging unless extreme swappiness is in use. Signed-off-by: lyugaofei Co-developed-by: Barry Song (Xiaomi) Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a276560bc66b..d6fac5b91ac1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4219,20 +4219,28 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY); } +static inline unsigned long lruvec_gen_size(struct lru_gen_folio *lrugen, + int type, unsigned long seq) +{ + int gen = lru_gen_from_seq(seq); + unsigned long size = 0; + + for (int zone = 0; zone < MAX_NR_ZONES; zone++) + size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); + return size; +} + static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness) { - int gen, type, zone; + int type; unsigned long seq, total = 0; struct lru_gen_folio *lrugen = &lruvec->lrugen; DEFINE_MAX_SEQ(lruvec); DEFINE_MIN_SEQ(lruvec); for_each_evictable_type(type, swappiness) { - for (seq = min_seq[type]; seq <= max_seq; seq++) { - gen = lru_gen_from_seq(seq); - for (zone = 0; zone < MAX_NR_ZONES; zone++) - total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - } + for (seq = min_seq[type]; seq <= max_seq; seq++) + total += lruvec_gen_size(lrugen, type, seq); } return total; @@ -5092,6 +5100,37 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, return scanned; } +static bool lru_gen_imbalanced(struct lruvec *lruvec, unsigned long max_seq, + struct scan_control *sc, int type, int swappiness) +{ + struct lru_gen_folio *lrugen = &lruvec->lrugen; + unsigned long young = 0, old = 0, lag = 0; + DEFINE_MIN_SEQ(lruvec); + + /* we still have enough generations to reclaim */ + if (min_seq[type] + MIN_NR_GENS < max_seq) + return false; + + /* + * Trigger aging if the preferred type is running low on reclaimable + * folios, provided the generation lag of the other type remains small + * enough that inc_min_seq() introduces negligible overhead + */ + for (unsigned long seq = min_seq[type]; seq <= max_seq; seq++) { + unsigned long size = lruvec_gen_size(lrugen, type, seq); + + if (seq + MIN_NR_GENS > max_seq) + young += size; + else + old += size; + } + if (min_seq[!type] + MAX_NR_GENS == max_seq + 1) + lag += lruvec_gen_size(lrugen, !type, min_seq[!type]); + + return young > old * MAX_NR_GENS && (lag < MAX_LRU_BATCH || + (is_extreme_swappiness(swappiness) && sc->priority > 2)); +} + static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, struct scan_control *sc, int swappiness) { @@ -5110,7 +5149,11 @@ static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, return false; /* better to run aging even though eviction is still possible */ - return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq; + if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS == max_seq) + return true; + + /* Run aging if the preferred type is severely imbalanced across gens */ + return lru_gen_imbalanced(lruvec, max_seq, sc, type, swappiness); } static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, -- 2.34.1