From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794343064; b=k2DmXM+7ApucagLQWYwFrO90wF7f2BK689FQn/hk6OPUY9iFx5AgzDHD/5XKt17qx1di 0Ofw7QwZt1vSRmc17jkW+AkckmqzRGHLBGR2LCjuCAmv6o19mDHfEuH4u6j8G5zzZnen+ pCKXPsf5sysHroEZNV/4MD2LcRl3migAqWBHGT4b1XHofLRRKjgDUnO6DjD0Cl1QOoAeB 7hBFHo1eejQXHnuVx/LorETOqLQF6KXV19hKrGUlS0fTk6zUWY/OE+Y4cmj+iAX5jj6Q3 5Qz7IjsqDEqkzLFI7TNVVJy5jpq89QyP25b6OE95SZSd1cxl4gzqb7TKK5BsznMinZQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794343064; h=Received:DKIM-Signature:From:Date:Subject:MIME-Version:Content-Type: Content-Transfer-Encoding:Message-Id:References:In-Reply-To:To:Cc: X-Mailer:X-Developer-Signature:X-Developer-Key; bh=wrSJEhIQdsnATG/TZUF9OCJMIcBwpL3C7sc2OmMxaig=; b=Uli0rRiu5bMxI5n+Q4E+tR8E3WEIos9PCBZqQ9uf82yFjNITjuppDh6ZnqNdP/d2GW9x oDNcsqLvPmOhxbVktQEdCSKQvrnUcPmWG8VUTRn/UYLH/zjprMenRO091OTddIKI94aAk aNYn65zgvv8KPvbc4kqCX/DzB6r9U7+eaH/422qDfd/FrDp2w5Wc3Zw8aSFzBzAYYXEFb HZWA34RzghhUWFwjsOIprDQCfFRTHla9+bwB2/iUJ1IlTobNJ7orlT3HX9PwWRXxikcej zz+h099epq9dOOmJ7AWPc0rA57IIq5NT7Q9BnfxKhNN7q5E44Ojm66lbHRkmbMj1GkA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=eqWuCZXJ; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89BFB60A260C; Tue, 10 Nov 2026 20:37:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=eqWuCZXJ Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3CF381F008A2; Tue, 10 Nov 2026 20:37:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1794343064; bh=wrSJEhIQdsnATG/TZUF9OCJMIcBwpL3C7sc2OmMxaig=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=eqWuCZXJtOsZOej4Wnl3pn804Je0QoNGmJXam1b8c07SE5FueZ2WUt3/dBp8vqNky 0EYLnUASK32t0OJFcSDdyBROAYBEh+zOCCsr5mViX+9vYXqdfhlIbx58E9Tqjrb1MA iDBSMjKZZGGT8BQ0HtIHj7QUJn3ICZ1siqlGeeqabHGn+/NHoFgwoSwkXLDBDRUf0u hG2ChowSyeANPo6aomnttKJ2p8gnJaeyFVzAovb3NwQ3t/3uv4TDd8pCn/R6pCQY+v 3buU0UC6lwkjAUCS1WCtkW98WZiLxiofDrGsIS6hqy5AQdbZh4jO4WpgjIrGQ9kDCI go4yn+eVGZJYA== From: "David Hildenbrand (Arm)" Date: Tue, 10 Nov 2026 21:35:43 +0100 Subject: [PATCH v3 3/9] mm/mmu_gather: flush pending references early before the folio limit X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261110-refcount_overflow_stable-v3-3-318625d63a98@kernel.org> References: <20261110-refcount_overflow_stable-v3-0-318625d63a98@kernel.org> In-Reply-To: <20261110-refcount_overflow_stable-v3-0-318625d63a98@kernel.org> To: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , "Matthew Wilcox (Oracle)" , Jan Kara , Andrew Morton , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kiryl Shutsemau , Matthew Brost , Joshua Hahn , Byungchul Park , Gregory Price , Ying Huang , Alistair Popple , Peter Xu , Johannes Weiner , Song Liu , Rik van Riel , Miklos Szeredi , Alexander Viro , Christian Brauner , Hugh Dickins , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Arnd Bergmann , Jason Gunthorpe , Kevin Tian , "Joerg Roedel (AMD)" , Robin Murphy , Steve Sistare , John Hubbard , "Jose A. Perez de Azpillaga" , Josef Bacik , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Neal Cardwell , Kuniyuki Iwashima , Linus Torvalds Cc: "David Hildenbrand (Arm)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, linux-arch@vger.kernel.org, iommu@lists.linux.dev, netdev@vger.kernel.org, Chris Mason , stable@vger.kernel.org X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4480; i=david@kernel.org; h=from:subject:message-id; bh=4dJ/m1FVSvgW3CYjdpbAHjLF5sx3MyByqT1mM21gZhc=; b=owGbwMvMwCF2QOXvyb15Z08znlZLYsj63BDOdU52ZZZNSNXSk3m3zzPeZO3/vmzD8q3qWffb7 bdlfVU51lHKwiDGwSArpshybV1OavCaVseyKEM3mDmsTCBDGLg4BWAisrsY/led5pFhDz4ldDDv gcl275B5gj66HusyZh2aPdcwdYe4KDfD/2SFsu6JXro3HCT/8UxmLF/+rv6QXMPiQ80zzzhWV1h +ZQAA X-Developer-Key: i=david@kernel.org; a=openpgp; fpr=1BD9CAAD735C4C3A460DFCCA4DDE10F700FF835A When a task zaps mappings (e.g., MADV_DONTNEED), it can drop the PT lock and defer dropping the folio references. In the meantime, page faults can refault pagecache folios, creating more references for the same folios. These deferred references can build up over multiple VMAs. Let's just request a flush after enqueueing a pagecache folio whose refcount reached a certain threshold. Start flushing well before new references are refused, regardless of the caller's privileges. When returning "true" from __tlb_remove_folio_pages_size(), PTE zapping will flush the TLB before dropping the PT lock, but drop the references afterwards. Refaults can still race with that, but flushing early reduces the risk of letting others hit the limit. This is a temporary measure until we have something better. Reported-by: Chris Mason Fixes: 800d8c63b2e9 ("shmem: add huge pages support") Fixes: 99cb0dbd47a1 ("mm,thp: add read-only THP support for (non-shmem) FS") Fixes: 793917d997df ("mm/readahead: Add large folio readahead") Cc: stable@vger.kernel.org Reviewed-by: Jose A. Perez de Azpillaga Reviewed-by: Lance Yang Signed-off-by: David Hildenbrand (Arm) --- include/asm-generic/tlb.h | 4 ++-- include/linux/mm.h | 1 + mm/memory.c | 4 ++-- mm/mmu_gather.c | 10 ++++++---- 4 files changed, 11 insertions(+), 8 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 3f4a180b48f3c..4bba02fd5d543 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -75,8 +75,8 @@ * - __tlb_remove_folio_pages_size() * * __tlb_remove_folio_pages_size() is the basic primitive that queues pages - * for freeing. It will return a boolean indicating if the queue is (now) - * full and a call to tlb_flush_mmu() is required. + * for freeing. It returns true when a call to tlb_flush_mmu() is required, + * either because the queue is full or a folio is near its refcount limit. * * tlb_remove_page() and tlb_remove_page_size() imply the call to * tlb_flush_mmu() when required and has no return value. diff --git a/include/linux/mm.h b/include/linux/mm.h index 33bc57342590c..a36ff83589c61 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2232,6 +2232,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf); /* Leave separate reserves for administrators and concurrent reference gains. */ #define FOLIO_REF_COUNT_LIMIT (INT_MAX - (1U << 26)) #define FOLIO_REF_COUNT_ADMIN_LIMIT (INT_MAX - (1U << 25)) +#define FOLIO_REF_COUNT_FLUSH_LIMIT (INT_MAX - (1U << 27)) bool __folio_ref_count_over_limit(unsigned int ref_count); diff --git a/mm/memory.c b/mm/memory.c index 977e037388963..a1a9ba2430f83 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2072,8 +2072,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, pte_unmap_unlock(start_pte, ptl); /* - * If we forced a TLB flush (either due to running out of - * batch buffers or because we needed to flush dirty TLB + * If we forced a TLB flush (either to release batched + * references or because we needed to flush dirty TLB * entries before releasing the ptl), free the batched * memory too. Come back again if we didn't do everything. */ diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 9f353f0e2ef4d..e98397ac577c4 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -179,10 +179,7 @@ static bool __tlb_remove_folio_pages_size(struct mmu_gather *tlb, #endif batch = tlb->active; - /* - * Add the page and check if we are full. If so - * force a flush. - */ + /* Add the pages before deciding whether we need to flush. */ if (likely(nr_pages == 1)) { batch->encoded_pages[batch->nr++] = encode_page(page, flags); } else { @@ -190,6 +187,11 @@ static bool __tlb_remove_folio_pages_size(struct mmu_gather *tlb, batch->encoded_pages[batch->nr++] = encode_page(page, flags); batch->encoded_pages[batch->nr++] = encode_nr_pages(nr_pages); } + /* Flush pending references well before new references are refused. */ + if (unlikely((unsigned int)folio_ref_count(page_folio(page)) >= + FOLIO_REF_COUNT_FLUSH_LIMIT)) + return true; + /* * Make sure that we can always add another "page" + "nr_pages", * requiring two entries instead of only a single one. -- 2.56.0