From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=pass header.oldest-pass=0 smtp.remote-ip=100.103.45.18 ARC-Seal: i=2; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=pass; t=1795035837; b=IIFzIqtklUwjCjFi6woysIdI6VJXNACeShBfZJbwEzbYbohlTnjKJRp5ch9CJ+WbmaAk sZaXG2kQ4rWg6jaKyuZ0mdrQU45cBRTyyrX8iiY6vJB/h5VFdnGMNsXAqdPM28vz2butj W7ASJTItwcAT/cJ+dqRtA+nchULfobnux7i2/M8c8gxuOXB2z9M7uamuy5AgIlDcyq7mo WU1yRG+zrCKCL7901zubdTbNE/XTJ0i1QO3iX26Q52tJ7ZldLP/APQUOaAp0VCrdHKti0 Lc5VtjVB0l3qAy6TIGF9drQg/l16Bcmy3SwugZ1AFJBzmAtHLefCUMi91xF5+fpziNw== ARC-Message-Signature: i=2; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795035837; h=Received:DKIM-Signature:Received:From:Date:Subject:MIME-Version: Content-Type:Content-Transfer-Encoding:Message-Id:References: In-Reply-To:To:Cc:X-Mailer:X-Developer-Signature:X-Developer-Key: X-Endpoint-Received:X-Original-From:Reply-To; bh=41vfnI0ZoD897IEdGdgzDciDJt9MzpCIhSsZSG2H/IA=; b=BMxeVPAPAZgmslZKvjrAGBb8HvbUJgeh2hLb+wDd3ZBeFEUeVO/bU/XHXydmb8gUufyw wpHq77mhjfMLpBTKkWK+zTygW7NVD3dwYowczVglqRu+ADpyVG5aT+y81MeP6VYyqQObd 3Q6QZ36SXSeskYQrXa12m+80BdO7ZsVwP1UlXokYQG7DyQbkr7+C3XI3qNBeVcKMZf1rH jyAJz2Puw42pFmKJe67P1QGE68QqFhRh9te35xp6M8Tk6CUHVuwcerQHAbzGFVuUMQ/P4 DlaB7nXKlwFd+ok4V/8kJr6Soxu8Jq0oqC4nZXr6I4bQGKmQArcM2PnatvHXbAp97Mw== ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=ilHSFqBL; arc=pass header.oldest-pass=0 smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29A5B60A5BC5 for ; Wed, 18 Nov 2026 21:03:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=ilHSFqBL Authentication-Results: smtp.kernel.org; arc=none smtp.remote-ip=100.80.115.77 ARC-Seal: i=1; d=kernel.org; s=arc20260519; a=rsa-sha256; cv=none; t=1795035837; b=rPJxtkCUE/IBzlbeSxsNxvS668MaxImPXIL8hRFnwDg0KxjbeCtTfVLQpB+CF7S5Lrz2 FnXD91CaDV6KphaXd+TIAiIufAsKD4hClKonk5V2keQdmJmpFu8tX5QcaWv9NsZITOHLh AcjAVLmfqJ4ZerM31GA3qmOwVyN9lBr8Kfx9rqtSsW3oyNWkMl6QsvbrppG+2VwWW02Um uL1CncO2RxTfefSr6YuBl3q48nDYOk4x4SxleSpoUxIaMrbtXAYyopVRIK1fX92QthfJz FBmpoRM6dKQ5E8HHFf1C8/ZYz4o1xS2/9HhcT0fhpdh5tEFyRCJYnv4VPdAALng5YZw== ARC-Message-Signature: i=1; d=kernel.org; s=arc20260519; a=rsa-sha256; c=relaxed/relaxed; t=1795035837; h=DKIM-Signature:Received:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Mailer:X-Developer-Signature:X-Developer-Key:X-Endpoint-Received: X-Original-From:Reply-To; bh=41vfnI0ZoD897IEdGdgzDciDJt9MzpCIhSsZSG2H/IA=; b=OzuL2jEotWCg1xq78KfrzT1zAuV+0kZwGoAOdT1NaGiGU31knuyx8fSCUbRh8H7o6CtL 6WRENPXwEiAG2AM8lCoqCj0L7GxgWQG0ylnFZ2fxDBHpxansBxrCA0QdEj/KNGEQ3ouxJ LALbovXMol87SibH28UGVPr7YWD8jzWPBKOZ3mRErMUAN/cq/QtIQpPyaF5gM2Ei1ywC2 54Al/KQTMJ/8D/W3h5EWvdP1LsWRLlCKzPwDBAN+q54cmXbIKyx2Wy+fySI1+J8tBUe3H PpYM5dzPjkouhYvw5RvNrLTT8QZ6sZfbuP8Cxc0A/goHk2MqBK9iELsYZ1jANVfxaUg== ARC-Authentication-Results: i=1; smtp.kernel.org; arc=none smtp.remote-ip=100.80.115.77 Received: from lkml.kernel.org (nexus.subspace.kernel.org [100.80.115.77]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.kernel.org (Postfix) with ESMTPS id 2C8D11F000C5; Wed, 18 Nov 2026 21:03:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1795035837; bh=41vfnI0ZoD897IEdGdgzDciDJt9MzpCIhSsZSG2H/IA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To; b=ilHSFqBLaY7cnuqSJS8EPjBIQ9JaVNWI+TP86avDVtWHLf9FC8InNEMhOZl77aqzn Ai4cuEFanbqni1EG/sUJtRfGaT1ixnG6wAnqMVEN3auZ1+EpsmVllfFcQ/PjbRtx+s lDinY2uTmxIUQB3V6/NksH1xXy4ie7X43JNduNqjC7h4Yxam6yrqBmUYf6+VT8IzNS QsZG4pR8zbSI1/wO9DUel/csRrv0bh0z7grsLwxDvY2A8vLoRMiUcITuUPda2LdvKR 6b0cEBq6U7k6jOi0zKSPRh8NOZgpgt8oZ67ckDExX+vnbQMEGY6/ooGoNm8Fqd5s5k iez/vUxIOYmAA== Received: from nexus.subspace.kernel.org (localhost [IPv6:::1]) by lkml.kernel.org (Postfix) with ESMTP id 998694AF98; Wed, 18 Nov 2026 21:03:43 +0000 (UTC) From: Kairui Song via B4 Relay Date: Thu, 19 Nov 2026 05:03:37 +0800 Subject: [PATCH v5 05/17] mm/mglru: skip the flags cmpxchg when nothing changes X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261119-mglru-fg-v5-5-9eee2fe97a2d@tencent.com> References: <20261119-mglru-fg-v5-0-9eee2fe97a2d@tencent.com> In-Reply-To: <20261119-mglru-fg-v5-0-9eee2fe97a2d@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Johannes Weiner , Muchun Song , Qi Zheng , Ying Huang , Chris Li , Baoquan He , Nico Pache , Usama Arif , Michal Hocko , Roman Gushchin , Shakeel Butt , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Vlastimil Babka , Suren Baghdasaryan , Nhat Pham , Youngjun Park , Zi Yan , Gregory Price , "Matthew Wilcox (Oracle)" , Baolin Wang , Ryan Roberts , Dev Jain , Lance Yang , Hugh Dickins , SJ Park , David Rientjes , Yu Zhao , Vernon Yang , Zicheng Wang , Chen Ridong , Tal Zussman , Kairui Song , Xiang Liu , Ehab Ababneh , Zhao Zhenzhuo , Lian Wang , Joanne Koong , Kunwu Chan , Hui Zhu , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, damon@lists.linux.dev, linux-doc@vger.kernel.org, Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1795035819; l=3975; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=rKjtT9cTVQE3kB+eFzFZdlThrfxXKVi5B+1GEDGHtjw=; b=eXy1GT3doCWx4eA2MrHyusXF+V13z0BHWbZolJceGwJbmMtsa+zI9QRCGuCTBY1Rom264jpeW FKGD+ZvzktJA6h8Yvwm38iMU9RTQym7XK1PfXQXh4ylDSTy6KSdhsZ9 X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Ridong Chen folio_update_gen() and folio_set_lru_refs() always end with a try_cmpxchg() on folio->flags, even when the new value equals the old one. The locked instruction still takes the struct page cacheline in exclusive state. That is common in practice: a folio mapped by many processes is visited once per mm during a page table walk, and after the first visit it is already in the youngest generation with refs cleared and PG_workingset set. isolate_folio() and the eviction paths also call folio_set_lru_refs(folio, 0) on folios whose refs are already zero. Break out of the loop when the flags would not change. The update is complete at the point of that read: the flags already hold the target value, and a concurrent change to them is ordered after it. When the cmpxchg would have succeeded, the value written is the same, so no new state becomes reachable. What the patch drops is the retry. A retry re-reads the flags and recomputes new_flags from the newer value, so it overwrites a concurrent update with what this CPU wanted. For folio_update_gen(), a walker that read an older max_seq can lose the cmpxchg against a promotion done by another CPU, and then write its older generation back on the retry. folio_set_lru_refs() can likewise clear a refs increment that raced with it. Breaking out keeps the newer state. No caller relies on the full barrier that the locked instruction implied on the skipped path: the eviction and isolation callers run under the LRU lock, and the others (page table walks, folio_mark_accessed(), lru_gen_clear_refs()) only care about the resulting flags value, which is already in place. Note that lru_gen_inc_refs() already returns without any atomic when refs is at LRU_REFS_MAX and the folio is in the workingset. Tested on an i7-13700HX with turbo disabled and the performance governor. N processes map the same 2GiB tmpfs file (huge=never) and read every page, then one aging pass is triggered on their memcg through /sys/kernel/debug/lru_gen ("+ 0 200 1"). Time per aging pass in ms, median of 5 runs of 10 passes each, averaged over 2 boots of each kernel: processes base patched 1 32.9 31.7 -3.5% 8 195.1 160.2 -17.9% 32 755.8 592.2 -21.7% 64 1597.2 1260.6 -21.1% With one process, every visit changes the generation, so there is nothing to skip. Before the patch, perf annotate attributes 53-58% of the walk_update_folio() samples to the lock cmpxchg in the inlined folio_update_gen() in this test. [kasong: folio_update_gen() is replaced by folio_inc_lru_refs_walk() later in the series, which keeps the same early exit.] Reviewed-by: Kairui Song Signed-off-by: Ridong Chen Signed-off-by: Kairui Song --- include/linux/mm_inline.h | 2 ++ mm/vmscan.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 46beded51c7d..2571b3f01226 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -233,6 +233,8 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs) do { new_flags = old_flags; lru_set_refs_flags(&new_flags, refs); + if (new_flags == old_flags) + break; } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 8d14831cdf83..0f24b6ebc1eb 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3274,6 +3274,8 @@ static int folio_update_gen(struct folio *folio, int new_gen, int *type, lru_set_gen_flags(&new_flags, new_gen); lru_set_refs_flags(&new_flags, 0); new_flags |= BIT(PG_workingset); + if (new_flags == old_flags) + break; } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); *type = folio_flags_is_file_lru(&old_flags); -- 2.56.1