From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=57.103.78.175 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793442958; b=FpRLDUU5oRISKHL7tJNd0nKJS7Fo8p3jtNHvg3EgMhKJwrgJZyLOhKbwGfYOZrCSrMa+ nRQ24MuvPDuSondRopA0L22FIzKSavGH0BoIAFvpmxhxfaeUS4txPyKHQbAnf9MmRneB7 oEUpjS5HE4T+Za0D4Z9nlptF7UZpdeN6cRoresyNS+66cY9ylo7lYhOnK7RHDM7NwcHaD NyUhHT2Kp4FEbno1ffO7Wlhoc4Agm9loBAishBbq5GCLN5sKgg/nrOR8S+mZECi5l7BX3 u6rEn6HYbqFh9JLhv6ybTqHxsw4UpEu+Llq4rUHuIzWGhNNdHej4sOXVwVsp4xFeb8w== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793442958; h=DMARC-Filter:Received:X-ICL-RepId:X-ICL-Out-Info:Dkim-Signature: Received:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Mailer:X-Proofpoint-Spam-Details-Enc:X-Proofpoint-ORIG-GUID: X-Proofpoint-GUID:X-Authority-Info-Out:X-JNJ; bh=3KiLbCqh3KbXG4ivr0NTs62tHMSh1wth3efulb21WI4=; b=vv4SuvFuWwrWFU3jJtp6CkLrUvEtdydA2bRMQBZt4676Q7/+Fhu9B8VaT/nnDUxSMaKM EjygN941+FAfd5AXLraneqVRyV3wGHfc7tj1JMwGeeIMapeZwdC6+MEVmCfSBtKX0dARP G/jiAmIk22VFSisXonyj2718TAdAB19/dgDPTsVORccTgYcG4y2KJ+gtdNcb7gs7Ymh/S RRs7guyMPL3R5j3plBkXo+npStLkZIgwWTHPufk6SyLjcycjtPB2FwJIMVZwuVR1isKco X5laqG69Eoc6w7C+KMi4585mumT1Y6a+QXUlyHl9++pnT6il+tcCZ71e5aXmUdzk6WA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=icloud.com; spf=pass smtp.mailfrom=icloud.com; dkim=pass header.d=icloud.com header.i=@icloud.com header.a=rsa-sha256 header.s=1a1hai header.b=goKA296C; arc=none smtp.remote-ip=57.103.78.175 Received: from outbound.st.icloud.com (st-2004k-snip4-4.eps.apple.com [57.103.78.175]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD380606A31F for ; Sat, 31 Oct 2026 10:35:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org BD380606A31F Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=icloud.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=icloud.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=icloud.com header.i=@icloud.com header.a=rsa-sha256 header.s=1a1hai header.b=goKA296C Received: from outbound.st.icloud.com (unknown [127.0.0.2]) by p00-icloudmta-asmtp-us-east-1a-60-percent-7 (Postfix) with ESMTPS id D1E4D18002AD; Sat, 31 Oct 2026 10:35:51 +0000 (UTC) X-ICL-RepId: 01a19187-b226-72d1-834e-d22ae37e9ad0 X-ICL-Out-Info: HUtFAUMEWwJACUgBTUQeDx5WFlZNRAJCTQtJCkMGXwFeCEADQwZbC1BcHA4XXhtCFUsVXANcDkswUBtfAkIPHBNWFRMLU1ZbE1UXRgkZCF0dGQpQUAZcEhhcFFxQWB5GElYNXQkZGEZeUBtfAkIPHBNWFRMdQxkPKwhKBEMHRQJeCyUTCVNWWxNVF0YJGQhdHRkVWgkKVwBAC04CXQRaHxJYDFAFGh9eEhQeXxhEAx8GD1UGQAZSAkUKQQteDg5XE3kMVgNUDlwEFxtcAAlLRglJHQ4EVAddBV0= Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=1a1hai; t=1793442956; x=1824978956; bh=3KiLbCqh3KbXG4ivr0NTs62tHMSh1wth3efulb21WI4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:x-icloud-hme; b=goKA296CZN4xa9ZT6VYN887j+QL5AyD2TZjC5YGhTRvB7DuWcP3F1dbk3kVbAP3nMPZJ/qDR1AegPIFcvg686uyvPynKoJyX8aikMmQNOvvmt5xVpGa5Q1TIqQKWShRxygJ+F5BC/EsrfYtRJ8W50LyvBzsCMRvhcGTr3vLARwSnLMOUZJ+cWjvUFKHPr0Pu14L2pxA9PAmcyLqezog4MwD4GlzInWEuRBehoIfy8Vsk96zPxi00PnorHKEYGntyfYoI5ZqBC0picUjdZbnv4VMFwUtrPqnxG6xqdIOAKMEtsgWVQ4QWvkuxLwFu3S/aaI7mwArc6zrPfMG0gXp3LA== Received: from [192.168.5.15] (unknown [17.156.216.30]) by p00-icloudmta-asmtp-us-east-1a-60-percent-7 (Postfix) with ESMTPSA id 62E7F18002BD; Sat, 31 Oct 2026 10:35:46 +0000 (UTC) From: Zhang Peng Date: Sat, 31 Oct 2026 18:35:12 +0800 Subject: [PATCH v6 5/5] mm/vmscan: batch dirty TLB flushes before pageout X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261031-batch-tlb-flush-v6-5-190539839eec@tencent.com> References: <20261031-batch-tlb-flush-v6-0-190539839eec@tencent.com> In-Reply-To: <20261031-batch-tlb-flush-v6-0-190539839eec@tencent.com> To: Andrew Morton , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Kairui Song , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Baolin Wang Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhang Peng X-Mailer: b4 0.14.3 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYxMDMxMDA0MiBTYWx0ZWRfX7sKGwIC7XZPd PlZTtk40uF8LvLnqACQ8aufD5hXMFxFHfWdWKTbDRKlSrMQ1eBHE4hbnZuFnAaMaydk4pV6vNnw Hap7fwxHgWrD8y/xo1OdKTQL6W2RPGuNxCCZaRetMz7sm4vCxAL6iFNaR37L85G4mxBPpITTCAy XLyIzX2H/KllL+lSojsFvyCu6hIqQh+89vK3uiCsoQIlYnEoKCesOJMZjLwxOOLcjFmQfvLjz2V XTkSLVxfsyBzueG9hRWoSGCSww1u20BGEe4Y/XARuOzeTjCDV8Am7kSAm4KxgS5Cgb11YC/PdKA wD6+LCDy5Soi2J7wniaFucu8GkDiim5i91fkNJ6BGh11jxeZOx+hTwn5Ekwn5k= X-Proofpoint-ORIG-GUID: 2mMLgDoyhFjcu4KCPlvbE47ROxDQhDnr X-Proofpoint-GUID: 2mMLgDoyhFjcu4KCPlvbE47ROxDQhDnr X-Authority-Info-Out: v=2.4 cv=TZqbdBQh c=1 sm=1 tr=0 ts=6ae5c48a cx=c_apl:c_pps:t_out a=oyWFxbOnq+dmhQrAPgaJYA==:117 a=oyWFxbOnq+dmhQrAPgaJYA==:17 a=IkcTkHD0fZMA:10 a=660iZSQnnn4A:10 a=x7bEGLp0ZPQA:10 a=YE32fvk_ji8A:10 a=VkNPw1HP01LnGYTKEx00:22 a=GvQkQWPkAAAA:8 a=bSmHYz4YPk1EoIDwg3UA:9 a=QEXdDO2ut3YA:10 X-JNJ: AAAAAAABkHeuJtQ6+/Yz0kcGtFchbRSQCpFyLuLDBdfBx6VKDWxqbOwMQEmXLmjTvwZLHC CjYk9wluuR7x032VqgXQBFkaqUL6E1KqRgawRTh84mFlnEUdUVpctk4idnb2qdUSEn6RY4 UZekgnDWlB1dieovTC9WTvWSR5u5Z7JlLbhnllwscDKuOJeXaEZcsYALJ8qoND+uJgJUtR nRJ5r6SWHVzWW9CG/8hOlt1yagfcMJ3bI5xv8KhLzsWQpxc5qWT1T4fj4sqfTLBYul7uVd fIaKoTVUr3C62OpZ2XASYURr32t+PSBlJ/cg+z5vOomyKLKXRA85vsqZW+DSl3O+pQzt1q tFvyPqM8HIouPuUt64Acnfn4kwW0U0ckZewLhf1ilILAAKYXdzfR2rKB0MrmoGB7ly/uxn mITyoVgcblLaWUC+WfqRj9riTY++EadCsbGDMf/b+8E+vXPEqinbLAxzSyufLdbvsI+W3M +5UkX6PapxJnWon8ZjHFwTBVaRSrc3XaocTV8irVtdhJ1gaFb8zOQEYCxYuvSNoaV+pwZz 8Ge04pMKi7W1dppdUar5mUqK4wzUkunD547xM7+vEz5mbbWtP+1RXg0Acyvn3+fmMBCW5l TFpT3EcANYIhK/CUKq08HsBRrlVetuRb6U5cdHXgXUlMLwEnOgLmDRrNiBuWkXTmS/LGcJ pCcvlggyV32lGTA5/kNyt45vyblCD5mnEidSA2TaIGrgcGyXJbN0wIHv49Rqg5K04//YfY nUkH6nmXUTgt+wPKnDijtsfp2I/4fTgdxyAPSQ+oJQv8iGAwGDYyzvMAZADHCsvntqqhRR X5veXEIQ3Zcqfjl4R+zfaQuFprTFP3NvbZ4idyDZI6k3xZPy3UgjwBYvWsvOPSNwd5sY4B crgRlNslOt7i0t+qsKw6EOsXuXD1sJb4Ob Currently reclaim flushes the TLB for every dirty folio before pageout. On systems with many CPUs, the resulting IPIs can be a significant bottleneck. Collect dirty folios in a folio_batch instead. When the batch is full, lock and revalidate each folio, flush pending writable TLB entries once, and page out the folios that remain eligible. Flush a final partial batch at the end of the scan. Folios are unlocked while waiting in the batch so swap faults can make progress. After reacquiring the folio lock, recheck writeback, mappings, and DMA pins; a concurrent swapin may have installed a PTE and allowed a parallel GUP pin while the folio was unlocked. Keep each unmapped folio locked across the TLB flush and pageout to prevent swap faults from remapping it before I/O starts. The folio lock alone does not exclude GUP. Save the original batch count and reinitialize the batch, then add back folios that pass validation. Reinitialization leaves the array intact, and each add writes at or before the current read index. This reuses the batch without overwriting unread entries or needing a second array. Suggested-by: Kairui Song Signed-off-by: Zhang Peng --- mm/vmscan.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 101 insertions(+), 14 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 061414ff45bb..1150361e7d52 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1334,6 +1334,92 @@ static enum folio_pageout_result folio_try_pageout(struct folio *folio, unreachable(); } +static void folio_batch_pageout(struct folio_batch *batch, + struct reclaim_folio_state *state, struct scan_control *sc, + struct reclaim_stat *stat, struct swap_io_ctx *ctx, + struct list_head *folio_list) +{ + unsigned int nr = folio_batch_count(batch); + unsigned int i; + + /* + * Reuse this private batch to collect the locked folios. Reinit + * leaves the array intact; nr preserves the original input count. + * Adding each accepted folio writes at or before the current read + * index, so entries we have not read yet cannot be overwritten. + */ + folio_batch_reinit(batch); + + /* + * Revalidate after taking the lock: writeback may have started, or + * a swap fault may have remapped the folio and allowed GUP to pin it. + * Keeping an unmapped folio locked prevents swap faults from mapping + * it again before the TLB flush and pageout. The lock alone does not + * exclude GUP, so existing mappings and pins must both be checked. + */ + for (i = 0; i < nr; i++) { + struct folio *folio = batch->folios[i]; + + if (!folio_trylock(folio)) { + list_add(&folio->lru, &state->ret_folios); + continue; + } + + VM_WARN_ON_FOLIO(folio_test_lru(folio), folio); + + /* The folio may have changed while it was unlocked. */ + if (folio_test_writeback(folio) || folio_mapped(folio) || + folio_maybe_dma_pinned(folio)) { + folio_unlock(folio); + list_add(&folio->lru, &state->ret_folios); + continue; + } + + folio_batch_add(batch, folio); + } + nr = folio_batch_count(batch); + + if (!nr) + return; + + /* + * Flush writable TLB entries before any folio in the batch is + * submitted for I/O. + */ + try_to_unmap_flush_dirty(); + + for (i = 0; i < nr; i++) { + struct folio *folio = batch->folios[i]; + + switch (folio_try_pageout(folio, sc, ctx, folio_list)) { + case FOLIO_PAGEOUT_ACTIVATE: + goto activate_locked; + case FOLIO_PAGEOUT_KEEP_LOCKED: + goto keep_locked; + case FOLIO_PAGEOUT_KEEP_UNLOCKED: + goto keep; + case FOLIO_PAGEOUT_RECLAIM: + break; /* folio is locked; try to free it below */ + } + + switch (folio_try_reclaim_free(folio, state, sc)) { + case FOLIO_RECLAIM_ACTIVATE: + goto activate_locked; + case FOLIO_RECLAIM_KEEP: + goto keep_locked; + case FOLIO_RECLAIM_DONE: + continue; + } +activate_locked: + folio_activate_locked(folio, stat); +keep_locked: + folio_unlock(folio); +keep: + list_add(&folio->lru, &state->ret_folios); + } + folio_batch_reinit(batch); +} + static bool folio_try_unmap(struct folio *folio, struct reclaim_stat *stat, unsigned int nr_pages) { @@ -1375,6 +1461,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct mem_cgroup *memcg) { struct reclaim_folio_state state = {}; + struct folio_batch pageout_folios; LIST_HEAD(demote_folios); unsigned int nr_demoted = 0; unsigned int pgactivate = 0; @@ -1383,6 +1470,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, INIT_LIST_HEAD(&state.ret_folios); folio_batch_init(&state.free_folios); + folio_batch_init(&pageout_folios); memset(stat, 0, sizeof(*stat)); cond_resched(); do_demote_pass = can_demote(pgdat->node_id, sc, memcg); @@ -1679,21 +1767,16 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!sc->may_writepage) goto keep_locked; /* - * Folio is dirty. Flush the TLB if a writable entry - * potentially exists to avoid CPU writes after I/O - * starts and then write it out here. + * Drop the lock so swap faults finding this folio + * via swap cache lookup can make progress; the + * recheck that this necessitates is documented in + * folio_batch_pageout(). */ - try_to_unmap_flush_dirty(); - switch (folio_try_pageout(folio, sc, &ctx, folio_list)) { - case FOLIO_PAGEOUT_ACTIVATE: - goto activate_locked; - case FOLIO_PAGEOUT_KEEP_LOCKED: - goto keep_locked; - case FOLIO_PAGEOUT_KEEP_UNLOCKED: - goto keep; - case FOLIO_PAGEOUT_RECLAIM: - break; /* folio is locked; try to free it below */ - } + folio_unlock(folio); + if (!folio_batch_add(&pageout_folios, folio)) + folio_batch_pageout(&pageout_folios, &state, + sc, stat, &ctx, folio_list); + continue; } switch (folio_try_reclaim_free(folio, &state, sc)) { @@ -1721,6 +1804,10 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, VM_BUG_ON_FOLIO(folio_test_lru(folio) || folio_test_unevictable(folio), folio); } + if (folio_batch_count(&pageout_folios)) { + folio_batch_pageout(&pageout_folios, &state, sc, stat, + &ctx, folio_list); + } /* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ -- 2.56.0