From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 745D242E8F0; Sat, 3 Oct 2026 12:55:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791032118; cv=none; b=hzSPU34mt7YsYJDT8S81M6G+5bPKENRT+u1yFP/Pzwn17jBk1559FfriiO/o7PEDsy9HUvv0rjyffq7Cs/F+Rsxka8Uczzd5D8XU4VlAZFQDcJXKEzSQeHSfXQU+iI1zBQjVHzVLvSF5129D0ehQo95f8hNWsEY2ZIt1CDq6QmI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791032118; c=relaxed/simple; bh=fZ9kfTv0pJsuntFogg0gXgySlt6y0dqk3g4xZjKklRY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=JD+nTyEOETEEyya69aQEDRhi0w4tkcRhmtYlFHeBUD0zJNmvARPZpJShNR09KKzpkiY91M3DyOYmTji3RLBHnxas3JFxm3afWz0BOHmdWF1eNJm30E9yHNrgv8I1tJORnpltRow0HBVxfGTUmHCHA7lVOiePdcp1E4dpOEcthng= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=e7p6ck+I; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="e7p6ck+I" Received: by smtp.kernel.org (Postfix) with ESMTPS id 1EA70C4DDF7; Sat, 3 Oct 2026 12:55:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1791032118; bh=fZ9kfTv0pJsuntFogg0gXgySlt6y0dqk3g4xZjKklRY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=e7p6ck+IaTVleml0Ie5kW0c/qxTdXkLL8LOOkrMGEBpstJNsuCeEt+u/XPOAoVqs4 O6XiKivUz7pl6GfJzvhQ7uT+C5EtQsUwR2mLNZLuQ2Yhuv5mCgqCmLfKlO0DJPoi1z DPYFw3s0VRYYdToqJGr3jiw05UoiiwxrE7lG+RP1ysQMvMwOF6O1qK8TbmWNvVcrFd ex9KmxMrHZwZQ/2gsv9s4zZIT7kTLw0y8NYwI2q+Mn6r6uBshPdX1t9tnhcJzWX2t3 GkUauR3NBrxS4XrIHwcOoFVeZWseSX8GMqZE5DU3d0TKs1yPjjkem0UlFJgLxKy3ZA 3PPZf0kbrSftg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C189CA5FE9; Sat, 3 Oct 2026 12:55:18 +0000 (UTC) From: Kairui Song via B4 Relay Date: Sat, 03 Oct 2026 20:55:20 +0800 Subject: [PATCH RFC v3 09/17] mm/madvise: adapt for LRU refs based operations in MGLRU Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261003-mglru-fg-v3-9-cbd4546a5bd9@tencent.com> References: <20261003-mglru-fg-v3-0-cbd4546a5bd9@tencent.com> In-Reply-To: <20261003-mglru-fg-v3-0-cbd4546a5bd9@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Johannes Weiner , Muchun Song , Qi Zheng , Ying Huang , Chris Li , Baoquan He , Nico Pache , Usama Arif , Michal Hocko , Roman Gushchin , Shakeel Butt , David Hildenbrand , Lorenzo Stoakes , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Vlastimil Babka , Suren Baghdasaryan , Kemeng Shi , Nhat Pham , Youngjun Park , Zi Yan , Gregory Price , "Matthew Wilcox (Oracle)" , Baolin Wang , Ryan Roberts , Dev Jain , Lance Yang , Hugh Dickins , SeongJae Park , David Rientjes , Yu Zhao , Vernon Yang , Zicheng Wang , Chen Ridong , Tal Zussman , Kairui Song , Xiang Liu , Ehab Ababneh , zhaozhengzhuo , Lian Wang , Kunwu Chan , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1791032114; l=4236; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=jcqbXlUlGQopwPCGT1rfXLrfpvOPIf07bE3tZxVpK7Q=; b=GVN81BjiDmkmkpqOD5UNMIz5qnRK4kO2qlEtONbttik51Nr0+9Q1lUvRT0QbyAOjf20wYw0H7 jTTQDTWBAY4C8qlMxbWO4chF/sEe2tA20j+nBIFsWaEdZoKuJ95IvDS X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song For the active/inactive LRU, madvise clears PG_referenced so that one more access is not enough to reactivate a folio, and keeps PG_workingset on a folio demoted out of the active list so its refault is still accounted as a workingset refault (PSI). MGLRU keeps that history in the folio's refs count instead, which now lives in PG_referenced, PG_workingset and LRU_REFS_MASK, so writing to those bits directly corrupts the count. The two hints want different things: - MADV_COLD resets the count in folio_deactivate(), which also moves the folio to the oldest generation. - MADV_PAGEOUT caps the count at LRU_REFS_WORKINGSET before handing the folio to reclaim_pages(). The hint marks the folio as cold, so the eviction shadow records the capped count and does not inflate the eviction/refault counters of the hot tiers that drive aging, while folios at or above the workingset threshold keep their PSI classification. PG_young is cleared for both LRUs as before, so the referenced walk does not count stale history as a recent access. Signed-off-by: Kairui Song --- mm/madvise.c | 56 ++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index d83ce6abf8c3..c221234fa199 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -361,6 +361,44 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end, FPB_MERGE_YOUNG_DIRTY); } +static void madvise_cold_prep_folio(struct folio *folio) +{ + /* + * VM couldn't reclaim the folio unless we clear PG_young. + * As a side effect, it makes confuse idle-page tracking + * because they will miss recent referenced history. + */ + folio_test_clear_young(folio); + + /* + * For the active/inactive LRU, a folio demoted out of the active + * list should have PG_workingset so its refault is still accounted + * as a workingset refault, and the referenced bit always needs to + * be cleared even for inactive folios. Nothing to do for MGLRU as + * folio_deactivate() always calls lru_gen_clear_refs(). + */ + if (!lru_gen_enabled()) { + folio_clear_referenced(folio); + if (folio_test_active(folio)) + folio_set_workingset(folio); + } +} + +static void madvise_pageout_prep_folio(struct folio *folio) +{ + int refs = folio_lru_refs(folio); + + /* + * For MGLRU, drop the count below LRU_REFS_WORKINGSET and cap + * the rest at it: MADV is + * suggesting the folio is actually cold, so the eviction shadow + * should not record its hotness in the high tier counters, but + * the workingset classification is preserved for PSI. + */ + if (lru_gen_enabled()) + folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0); +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -438,12 +476,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); + madvise_cold_prep_folio(folio); if (pageout) { if (folio_isolate_lru(folio)) { + madvise_pageout_prep_folio(folio); if (folio_test_unevictable(folio)) folio_putback_lru(folio); else @@ -547,18 +583,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, tlb_remove_tlb_entries(tlb, pte, nr, addr); } - /* - * We are deactivating a folio for accelerating reclaiming. - * VM couldn't reclaim the folio unless we clear PG_young. - * As a side effect, it makes confuse idle-page tracking - * because they will miss recent referenced history. - */ - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); + madvise_cold_prep_folio(folio); if (pageout) { if (folio_isolate_lru(folio)) { + madvise_pageout_prep_folio(folio); if (folio_test_unevictable(folio)) folio_putback_lru(folio); else -- 2.55.0