From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=91.218.175.16 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793884993; b=X023whDrj069+MSdYxWKUHQYC6pHWNi9Ce/IbEq1vnjCIWUnEhfModTOZ0Pz3AkcyC53 pG2Vzai+Rmy3uE6e3LZEoiJbxEvNikKwYs7PckZjf7DiVKq5ZsgH2nlwMgVlfPZ7zeqpA XR1+byrRbmXjY7trOhyf9Fna/Zl17gxuNt0FVcQa1hKohXbTX8aawI48wVk6S3bR1tGvW QxEqZckvQOI07HAx4OlCwibPiHlnfSBKc2kKiAkxA7FJRbwTD0Cx/jbIaQgBEi0zuWFBX Ejd82IJ3AHPx/hOA0LvZaSA4lFnDS0qZwk28sr25aBWlp92VNKbz5r6xish8ohLbHKQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793884993; h=DMARC-Filter:X-Envelope-To:DKIM-Signature:X-Envelope-To:Received: X-Mizu-Trace-ID:X-Migadu-Flow:From:To:Cc:Subject:Date:Message-Id: X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=gYV+yJGTziGTRc1MN/3N3LeRV8A5wid4qIJ9qlqLvKQ=; b=mmC2o7KTk9SKc/6yW9J5zwRCKgt6yY+2WS8YSgpGxgXQtZd+HllK9c0kZE3RWAkuQJtc 7TRXz6sUIzN2sn+mDdf9Lcp1Ycds6mn7SzZD2rvN7qBORCOEwnPfx3kEQX7krVqrDrBXL u8VCwjxSQ2Q6HixYFowDa/7KDvq59JIMkp5OLhE0f0aQsvlV70jYZ8StJAYI7sxCm4tei StjL0eHio5vVBxdggjHIFIFDzXeF2P+9LICvcGgzkJn8hoD+gH7SBG1SPYTgaNSjcW+Du 9zpzOnpiMjpoyPWPgIFDpJrKiomakujQprj2oPHBUkyh7IK73AVPXQwHk7yZzqhFu2A== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=pNEbpqKE; arc=none smtp.remote-ip=91.218.175.16 Received: from mta0.migadu.com (out-16.mta0.migadu.com [91.218.175.16]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3130C606A336 for ; Thu, 5 Nov 2026 13:23:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 3130C606A336 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=pNEbpqKE X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=kHs8Y+dHA0ewLCbf5W9kyEteQjnWZcV5rBSJAmaEPvY=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1793884991; v=1; x=1794489791; b=pNEbpqKEsgclgowUTesDhTY0wunJFrkLzHRD1yKj3xJHYknfKvR6shR0K+toHbtesw5uXexN nDe+QHXipMBmUwoZusfnal6KLUW5a/dyTCA0TghtaQqOOKgjw0iMpH+7GekEi7foH9vseZfj9Ih YaUa+tIQ+T6lgT6vTkQ4802Y= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 878a88923bad5b48; Thu, 05 Nov 2026 13:23:11 +0000 X-Mizu-Trace-ID: 878a88923bad5b48 X-Migadu-Flow: FLOW_OUT From: Ridong Chen To: David Hildenbrand , Andrew Morton , Johannes Weiner Cc: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kairui Song , Qi Zheng , Shakeel Butt , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Baolin Wang , Joanne Koong , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Ridong Chen , Ridong Chen Subject: [RFC PATCH 3/3] mm/mglru: move lrugen atomic stats off the read-mostly lines Date: Thu, 5 Nov 2026 21:22:20 +0800 Message-Id: <20261105132220.2233677-4-ridong.chen@linux.dev> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20261105132220.2233677-1-ridong.chen@linux.dev> References: <20261105132220.2233677-1-ridong.chen@linux.dev> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Ridong Chen lru_gen_eviction() and lru_gen_refault() update lrugen->evicted[] and lrugen->refaulted[] locklessly with atomic_long_add(). The arrays sit at the tail of struct lru_gen_folio, right before enabled/gen/seg/list: with LRU_GEN_STATS=n the file type counters share a cacheline with them. Those fields are read far more often than they are written: lru_gen_add_folio() reads ->enabled on every LRU add, and shrink_many() walks ->list and reads ->gen/->seg under RCU. With frequent file refaults from several CPUs, every refault invalidates that line on the CPUs doing LRU adds. Move enabled/gen/seg/list next to max_seq/min_seq/timestamps, which are also mostly read, and put evicted[]/refaulted[] in their own cacheline group at the end of the struct. This grows struct lru_gen_folio from 1360 to 1408 bytes and struct mem_cgroup_per_node from 2176 to 2240 bytes on x86_64 (one more cacheline per memcg per node). Tested on an i7-13700HX (24 threads) with turbo disabled and the performance governor. 24 files of 128MiB on ext4 on brd, read with fio (psync, randread, bs=4k, fadvise_hint=random, buffered), all jobs in one memcg with memory.max=1G, so nearly every read refaults. IOPS, median of 5 runs: jobs base patched 1 270537 274303 +1.4% (sd 3.7%) 8 1461025 1466293 +0.4% 16 2455629 2508001 +2.1% 24 2707720 2823776 +4.3% A custom random pread() test of the same setup, with this patch alone against base over two boots of each kernel, shows +2.4% at 16 threads and +5.9% at 24 threads, and no change at 8 threads or fewer. With anonymous memory on zram, the refaults mostly hit the tier 0 counters on a different line, and there is no measurable change. The gain grows with the number of CPUs refaulting into the same lruvec and needs a fast backing store; with real IO it is likely to be lost in the noise. The patch is mostly meant to keep the lockless counters away from the read-mostly fields regardless of where they end up. Signed-off-by: Ridong Chen --- include/linux/mmzone.h | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 6442e662aa4f..eb84af674e23 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -551,6 +551,14 @@ struct lru_gen_folio { unsigned long min_seq[ANON_AND_FILE]; /* the birth time of each generation in jiffies */ unsigned long timestamps[MAX_NR_GENS]; + /* whether the multi-gen LRU is enabled */ + bool enabled; + /* the memcg generation this lru_gen_folio belongs to */ + u8 gen; + /* the list segment this lru_gen_folio belongs to */ + u8 seg; + /* per-node lru_gen_folio list for global reclaim */ + struct hlist_nulls_node list; /* the multi-gen LRU lists, lazily sorted on eviction */ struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the multi-gen LRU sizes, eventually consistent */ @@ -562,16 +570,10 @@ struct lru_gen_folio { /* can only be modified under the LRU lock */ unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; /* can be modified without holding the LRU lock */ + __cacheline_group_begin_aligned(lrugen_stats); atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; - /* whether the multi-gen LRU is enabled */ - bool enabled; - /* the memcg generation this lru_gen_folio belongs to */ - u8 gen; - /* the list segment this lru_gen_folio belongs to */ - u8 seg; - /* per-node lru_gen_folio list for global reclaim */ - struct hlist_nulls_node list; + __cacheline_group_end_aligned(lrugen_stats); }; enum { -- 2.34.1