From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADA432EC0A4 for ; Fri, 17 Jul 2026 21:08:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784322498; cv=none; b=U0pumnCQ7tk/8vCBE/E2JqGKw4gZfYtIsQ6MgfMpSC4DBT5gEvenpzSaMRkCo/C41iGYDZI7vC6xb6an2FlrRCtKWIjRUvbsCYY34CthjdoRuvunHpk9LCDg0BPJYZDJTorcYwUXDslRoLT3O0KgOMmaoxrvYteEox7JW8mv5pI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784322498; c=relaxed/simple; bh=1oDqHUAbyufZZVARq+gPgcZMWuLd57VzTeUXeJ4g88Y=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=of8iZuVdwGwD/eTRizH90CuaTWrydInVBpEp+seIItNtcFk0lfRhC+P1j0Y1C7+x8jeo6uWlXUrPuNDohk9Qvb4BPB//jAiEoO+xR5vPzuU85i1dmtYI4gXP8XGe2lDckpeH24iK5Q5Tr3CgfYIQoUpkzYn5r7FuUq8gLsCxcm8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=s+7zbabK; arc=none smtp.client-ip=209.85.221.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="s+7zbabK" Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-47c2b362ee2so7194712f8f.1 for ; Fri, 17 Jul 2026 14:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1784322494; x=1784927294; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=AjYCCbO3PLXnys6k424gkT2HzpdBexgKHv5hZJae8ug=; b=s+7zbabKNhgJAWk7dT/jj6BZOFVhvcHqfmFcdf1rFcVg/d7y8E6gJXAH0tmyktRBKh tc+A2zlWlaHsctGn57cYj21Fx87uakFcYqyQzpMkkIBLR68ET6zxyNr4deBna6UKjByo 92jgLyl8E/0dAUHNl6FfZcuYe1Xdp8L6gRidA4L7Teho/lnlxNEJEAsuXSJIMxLX/3E+ pl3fjgjj6jHf90+fgQIand4H9Al7Z3kbhZYZyp/8q4C+8O3K5lm4e4rie+MkGbhxE/Iq OxiKyrEbI5uA/XCrWnDRrOUFCRkOLQq+s0llF9ZnNwhpq7NTMWBi3d2f7MuNc8n84sK9 BzaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784322494; x=1784927294; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=AjYCCbO3PLXnys6k424gkT2HzpdBexgKHv5hZJae8ug=; b=odUdE58hq0v7qI2em4rZuOta+DBBaIxwVx0Wr+Cu10fOP3v7CzK8Txoh+E7gOaEMzw cCiHv1PdURE3uMuYVySekFljXFkyuVo3cXxrRFRBlTSQaO3dTYiIE+ldm7P42oQ4+3DZ 4bTiE1YMfHufxuwrrureAIy6uNMnXex9398wHvuI1cHeBxgcjdXsrRi2vSRUk+cnP/+S Qjtr3L0LUE4RUBDjX0WysXmPNxhewQgARd7uugCGC8VapM0pLONz/G2pj3Fol7hukOXx 4aa4zsaKPkMyZreCYwS+6sygVQBtb8J25+68eUyfZc6NQziTJct6xZVpwICl/izPWrPt Hivw== X-Forwarded-Encrypted: i=1; AHgh+RqGLDpUyO5g7e0yaT9kbEBAPVCZDPFDaXs5ELCvj9DQQnsAqylp+t7CTHKxz3/VloKEINPSs1pIJbmj558=@vger.kernel.org X-Gm-Message-State: AOJu0YyLjsIjQWxb4tefOTe68F46wy6A00N7i+q/zj1kCFIdlVRCt9Jb wpBR/PnOHmmsDhpuOxLWW/RwQkDG5jA5bEi/jxD1S9vi3HnzAK32KrQWnAGQNG9F8rw= X-Gm-Gg: AfdE7cmlAamX/YPYRPOU2FWtEOcgnKvYoCkwfHn0ahUFyCtkcBdSIxqw9u9On0wlEDX jxq5hjfE+/qwkT3oHGDW5n6yJgwuohqWFdm/bQEFPoWEoFMP+enKL+HvnwZMPnavKrv2u5gMPaF HQ0Y9P4lMQu1WgjAEHHISgqRVjJuR86vPuCIqyoXB6TvXmx9qubUhjE+H5FuDA7zeEO0vaWuVm7 WOdfi7ymmJiws1xWY+WjFgNsFC+FXrC/d6E0Udc5pY3s0VMmGlb5kL4YAYdZbmEUnroZDigezuO HEEf5iMKcM/1Htrb2OfHw/esz3YdGgEoVkr9HVb6pb8kasL7+GoAu2AjqJriToaHq1cgATyNPlt CjfBsmuSVc427GgHnSSTztVCA69UdqpcUppTU5oS3EgSXOh4WqxOji/kZy98giy9Oj9+dVnOQbH c= X-Received: by 2002:a05:600c:4515:b0:495:3af9:4cfe with SMTP id 5b1f17b1804b1-4954a40e322mr48126015e9.20.1784322493273; Fri, 17 Jul 2026 14:08:13 -0700 (PDT) Received: from localhost ([2a02:8071:6401:180:d892:bf43:a0b4:83b]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4954a2aa698sm68755195e9.4.2026.07.17.14.08.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Jul 2026 14:08:12 -0700 (PDT) Date: Fri, 17 Jul 2026 23:08:08 +0200 From: Johannes Weiner To: Usama Arif Cc: Andrew Morton , david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, kasong@tencent.com, qi.zheng@linux.dev, shakeel.butt@linux.dev, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, chrisl@kernel.org, nphamcs@gmail.com, baoquan.he@linux.dev, youngjun.park@lge.com, roman.gushchin@linux.dev, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, rientjes@google.com, kernel-team@meta.com Subject: Re: [PATCH v3 2/2] mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost Message-ID: <20260717210808.GI6843@cmpxchg.org> References: <20260717135807.3476029-1-usama.arif@linux.dev> <20260717135807.3476029-3-usama.arif@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260717135807.3476029-3-usama.arif@linux.dev> On Fri, Jul 17, 2026 at 06:57:32AM -0700, Usama Arif wrote: > @@ -2303,12 +2301,63 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) > mem_cgroup_flush_stats_ratelimited(sc->target_mem_cgroup); > > /* > - * Determine the scan balance between anon and file LRUs. > + * Determine the scan balance between anon and file LRUs from per-LRU > + * vmstat counters. The raw cost per side is: > + * > + * PGROTATE - reclaim-driven rotations, bumped from both > + * shrink_inactive_list and shrink_active_list > + * (CPU work). > + * NR_VMSCAN_WRITE - reclaim-driven anon pageout IO. > + * WORKINGSET_RESTORE - refaults of previously-workingset pages. > + * > + * The two IO terms are weighted by SWAP_CLUSTER_MAX to reflect the > + * higher cost of an IO over a rotation. > + * > + * Reads are lock-free per-cpu sum collations, rstat-aggregated up > + * the memcg hierarchy by mem_cgroup_flush_stats_ratelimited() above. > + * Use lruvec_page_state_monotonic() so the unsigned subtraction > + * `now - prev_cost[f]` yields the correct delta across a signed-long > + * wraparound of the underlying counter (a real hazard on 32-bit that > + * the clamp in lruvec_page_state() would otherwise turn into a huge > + * spurious delta). > + * > + * The delta against prev_cost is folded into cost_accum, which is > + * halved on both sides until their sum is within lrusize/4. > + * cost_lock serialises concurrent reclaimers in the same memcg+node. IMO that's a lot of describing what the code does. Why not stick closer to the original comments? > */ > - spin_lock_irq(&target_lruvec->lru_lock); > - sc->anon_cost = target_lruvec->anon_cost; > - sc->file_cost = target_lruvec->file_cost; > - spin_unlock_irq(&target_lruvec->lru_lock); > + spin_lock(&target_lruvec->cost_lock); > + for (int f = 0; f <= 1; f++) { > + unsigned long now, delta; > + > + now = lruvec_page_state_monotonic(target_lruvec, PGROTATE_ANON + f) + > + lruvec_page_state_monotonic(target_lruvec, > + WORKINGSET_RESTORE_BASE + f) * > + SWAP_CLUSTER_MAX; > + if (f == WORKINGSET_ANON) > + now += lruvec_page_state_monotonic(target_lruvec, > + NR_VMSCAN_WRITE) * > + SWAP_CLUSTER_MAX; It's hard to prove overflow behavior is correct. I would keep the delta extraction dead simple, then do the weight math on the delta. > + delta = now - target_lruvec->prev_cost[f]; > + target_lruvec->prev_cost[f] = now; > + target_lruvec->cost_accum[f] += delta; > + } > + unsigned long lrusize = > + lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) + > + lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) + > + lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) + > + lruvec_page_state(target_lruvec, NR_ACTIVE_FILE); > + unsigned long cost_limit = lrusize / 4; > + > + while (target_lruvec->cost_accum[WORKINGSET_ANON] > cost_limit || > + target_lruvec->cost_accum[WORKINGSET_FILE] > cost_limit || > + target_lruvec->cost_accum[WORKINGSET_ANON] + > + target_lruvec->cost_accum[WORKINGSET_FILE] > cost_limit) { > + target_lruvec->cost_accum[WORKINGSET_ANON] /= 2; > + target_lruvec->cost_accum[WORKINGSET_FILE] /= 2; Why do you need to check them individually? Between reclaim cycles, there are no scans->rotations. And I don't see how you could get refault events several times the size of the LRU, let alone in excess of ULONG_MAX. > + } > + sc->anon_cost = target_lruvec->cost_accum[WORKINGSET_ANON]; > + sc->file_cost = target_lruvec->cost_accum[WORKINGSET_FILE]; > + spin_unlock(&target_lruvec->cost_lock); I realize these long descriptor names make it hard, but I think this can be cleaned up a bit and written in a more idiomatic way. How about this (totally untested): struct lru_cost { unsigned long count; unsigned long last_rotated; unsigned long last_io; }; struct lruvec { struct lru_cost cost[2]; }; struct lru_cost *anon_cost, *file_cost; unsigned long lrusize; /* * Determine the scan balance between anon and file LRUs. * * The cost model is based on rotations, refaults and * reclaim-driven writes (anon only) on each side. * * These event counters are monotonic, so each reclaim cycle * the delta since the last scan is extracted and incorporated * into a decaying average. This ensures currency, as workloads * change over time, and avoids overflow in the calculations. */ spin_lock_irq(&target_lruvec->cost_lock); for (file = 0; file <= 1; file++) { struct lru_cost *cost = &target_lruvec->cost[file]; unsigned long rotated, io, nr_rotated, nr_io; rotated = lruvec_page_state_monotonic(target_lruvec, PGROTATE_ANON + file); io = lruvec_page_state_monotonic(target_lruvec, WORKINGSET_RESTORE_BASE + file); if (!file) io += lruvec_page_state_monotonic(target_lruvec, NR_VMSCAN_WRITE); nr_rotated = rotated - cost->last_rotated; nr_io = io - cost->last_io; /* * Reflect the relative cost of incurring IO and spending CPU * time on rotations. This doesn't attempt to make a precise * comparison, it just says: if reloads are about comparable * between the LRU lists, or rotations are overwhelmingly * different between them, adjust scan balance for CPU work. */ cost->count += nr_io * SWAP_CLUSTER_MAX + nr_rotated; cost->last_rotated = rotated; cost->last_io = io; } anon_cost = &target_lruvec->cost[WORKINGSET_ANON]; file_cost = &target_lruvec->cost[WORKINGSET_FILE]; lrusize = lruvec_page_state(target_lruvec, NR_INACTIVE_ANON) + lruvec_page_state(target_lruvec, NR_ACTIVE_ANON) + lruvec_page_state(target_lruvec, NR_INACTIVE_FILE) + lruvec_page_state(target_lruvec, NR_ACTIVE_FILE); while (anon_cost->count + file_cost->count > lrusize / 4) { anon_cost->count /= 2; file_cost->count /= 2; } sc->anon_cost = anon_cost->count; sc->file_cost = file_cost->count; spin_unlock(&target_lruvec->cost_lock);