From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mta0.migadu.com (out-214.mta0.migadu.com [91.218.175.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10167215F42 for ; Sat, 26 Sep 2026 02:07:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.214 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790388471; cv=none; b=IXKqEIcmfZRFKltFVt7T/7bmw2gHsqaKtD1uHriaBV/URQ05DnAekMRIYxYD6cVYd9ZEB61TARYs9/n9FYwEGW+yIvXDDA+Dk8mDLso0pkIvtUaHvTM2Dk9ODakKBsj155/+KDGVvZpy7lRgrzVTMOHd7kIOapIbN3XvGn9wwBQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790388471; c=relaxed/simple; bh=DdcObnhLvU3ls0cGmHLGHqKMExuQVY9LIWDky3y/b9w=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=ClW5DGJiVYqz6Bcx3iim7a/IcwkaiJeH4Fqkwt/7vUapEkLlo3TM6jWKqiLdLNN0nr8dGieezEHNsNSomDJUNAHU4v6E4FpwHYP6VJCzMsCGxc6/B6i8q84wOx2b9GgDPtke9aKGbfY176fJkZRu4EDfdkHLqdFkhZTZxyiq5gE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=GHnpGus/; arc=none smtp.client-ip=91.218.175.214 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="GHnpGus/" X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=DdcObnhLvU3ls0cGmHLGHqKMExuQVY9LIWDky3y/b9w=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1790388465; v=1; x=1790993265; b=GHnpGus/PQOH3EV3bD8vMcFlal64YEC5GP98HrPUIYS4fxyvaQmVCiPNfAl17980fLFfv6l4 lj3+CCRCYEkxikNdckW+7hH1EwDdIUBCv2U9rzEEGp5sTP+L8CLLlvFdU50SOwZoJ+gH3d3K4ty 1XkyoC9tBWsWQ8lj1Q2IMpsU= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 6f0838c226c85a64; Sat, 26 Sep 2026 02:07:43 +0000 X-Mizu-Trace-ID: 6f0838c226c85a64 X-Migadu-Flow: FLOW_OUT Message-ID: <064255ee-c8be-4a82-8be0-fa209eac8637@linux.dev> Date: Fri, 25 Sep 2026 19:07:35 -0700 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 00/26] mm/fbatch: drain lru_add_drain() and _all() To: Hugh Dickins , Andrew Morton Cc: Ackerley Tng , Alexander Viro , Alexandre Ghiti , Baolin Wang , Barry Song , Binbin Wu , Christian Brauner , Christoph Hellwig , Christoph Lameter , Claudio Imbrenda , David Hildenbrand , Jan Kara , Jens Axboe , Johannes Weiner , Kairui Song , Kiryl Shutsemau , Lance Yang , Leonardo Bras , Lorenzo Stoakes , Marcelo Tosatti , Matthew Wilcox , Mel Gorman , Miaohe Lin , Michal Hocko , Minchan Kim , Muchun Song , Oscar Salvador , Peter Zijlstra , Qi Zheng , Rik van Riel , Sebastian Andrzej Siewior , Shakeel Butt , Suren Baghdasaryan , Vlastimil Babka , Yang Shi , Yu Zhao , Zach O'Keefe , Zi Yan , linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: Content-Language: en-US From: JP Kobryn In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi Hugh, On 9/9/26 2:39 AM, Hugh Dickins wrote: > [PATCH v2 00/26] mm/fbatch: drain lru_add_drain() and _all() > > This series was prompted by lru_add_drain_all() appearing in watchdog > backtraces: not to blame, but blocked on an unresponsive CPU to run > its workqueue. lru_add_drain_all() is a heavyweight operation, which > we often hope to avoid by a much lighter lru_add_drain(); but even > those local drains can aggravate the lruvec lock contention which > per-cpu fbatches are intended to ease. > > Now let the folios on the lru_add and other per-cpu fbatches remain > there, isolatable, with PG_lru set and refcount unraised, just like > when on an actual LRU. Then many calls to lru_add_drain() and _all() > can be removed. > > It's something I've wanted to do for years, tried several times, but > only hit on a good way to do it a few weeks ago: now it's uncomfortable > to watch others wrestling with the drainage, while I'm sitting on this. > > The key idea came from the "if (!folio_test_clear_lru(folio)) continue;". > If we don't mind missing to take an action on rare occasions, then maybe > we won't mind taking an action on the wrong folio on rare occasions, so > long as it is a folio consenting to PG_lru rules. > > No new locking, but relies on folio_try_get() and folio_test_clear_lru() > even on the lru_add fbatch; with use of bits not set in aligned pointers, > and some try_cmpxchg()ing. Speculative references to folios are already > accepted: this adds another source of them. > > Performance? You (and the bots) tell me. I'm considering this as a > cleanup, to make life easier for developers. I expect that some loads > will show improvement, but also expect some disappointments (perhaps > I go too far against lru_cache_disable()? or not far enough). > I found one source of regression but I think it's fixable. Using a test of 4 processes repeatedly write faulting anon pages then discarding with MADV_DONTNEED, the perf A/B below shows that the series (minus patch 27) spends more time in __page_cache_release due to new lru_lock traffic. baseline (7891fbb9512f): 0.03% folio-churn [kernel.kallsyms] [k] queued_spin_lock_slowpath 0.26% folio-churn [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.01% folio-churn [kernel.kallsyms] [k] folio_lruvec_lock_irqsave 0.00% folio-churn [kernel.kallsyms] [k] folio_lruvec_relock_irqsave 0.03% folio-churn [kernel.kallsyms] [k] __page_cache_release series (without patch 27): 6.97% folio-churn [kernel.kallsyms] [k] _raw_spin_lock_irqsave | ---_raw_spin_lock_irqsave | --6.96%--folio_lruvec_lock_irqsave folio_lruvec_relock_irqsave | --6.72%--__page_cache_release folios_put_refs free_pages_and_swap_cache tlb_flush_mmu tlb_finish_mmu do_madvise __x64_sys_madvise do_syscall_64 entry_SYSCALL_64_after_hwframe __madvise 6.62% folio-churn [kernel.kallsyms] [k] queued_spin_lock_slowpath | ---queued_spin_lock_slowpath _raw_spin_lock_irqsave folio_lruvec_lock_irqsave folio_lruvec_relock_irqsave | --6.43%--__page_cache_release folios_put_refs free_pages_and_swap_cache tlb_flush_mmu tlb_finish_mmu do_madvise __x64_sys_madvise do_syscall_64 entry_SYSCALL_64_after_hwframe __madvise There was a pre-existing dead folio optimization [0] which allowed caller-released folios remaining on batches to be cleaned up at drain time without lru_lock acquisition. This series removes it which makes sense since the batch reference concept is gone. But now that callers effectively invoke the cleanup when refcounts go directly to zero, there's a point in which an lru_lock will be taken for pending folios that are dead (refcount zero but still on batch). That's seen in the call graphs above and is unnecessary since these folios haven't been inserted into the LRU. Luckily It looks like the new API call lru_add_del_folio() can be used to avoid this, since it returns true after cleaning up the pending LRU batch state. I applied this diff and re-ran the same benchmark. The new locking overhead was nearly all eliminated. The perf output afterward shows the fractional amount that remains. @@ -64,8 +64,10 @@ static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, unsigned long *flagsp) { if (folio_test_lru(folio)) { - folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); - lruvec_del_folio(*lruvecp, folio); + if (!lru_add_del_folio(folio)) { + folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); + lruvec_del_folio(*lruvecp, folio); + } __folio_clear_lru_flags(folio); } } series modified with patch above (without patch 27): 0.77% folio-churn [kernel.kallsyms] [k] queued_spin_lock_slowpath | ---queued_spin_lock_slowpath _raw_spin_lock_irqsave | --0.77%--folio_lruvec_lock_irqsave folio_lruvec_relock_irqsave | --0.52%--__page_cache_release folios_put_refs free_pages_and_swap_cache tlb_flush_mmu tlb_finish_mmu do_madvise __x64_sys_madvise do_syscall_64 entry_SYSCALL_64_after_hwframe __madvise 0.37% folio-churn [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.04% folio-churn [kernel.kallsyms] [k] folio_lruvec_lock_irqsave 0.01% folio-churn [kernel.kallsyms] [k] folio_lruvec_relock_irqsave [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/mm?id=9669b87065a6fe96198f3df2c3d125c5f5c1f210