[PATCH v3] mm/shmem: remove truncated folios from the page cache in batches

[PATCH v3] mm/shmem: remove truncated folios from the page cache in batches

[Qiliang Yuan]

truncate_inode_pages_range() takes the folios it finds out of the page
cache with delete_from_page_cache_batch(), which takes the inode's
i_lock and the i_pages lock once for each batch of up to
FOLIO_BATCH_SIZE folios.

shmem_undo_range() calls truncate_inode_folio() on each folio instead,
so filemap_remove_folio() takes and drops both locks once per folio.
Truncating, punching or freeing a tmpfs file or memfd made of small
folios, such as a large /dev/shm segment, pays for that on every page.

Clean each folio up as truncate_inode_folio() does and, once the swap
entries are gone, pass the batch to delete_from_page_cache_batch(),
which needs truncate_cleanup_folio() outside mm/truncate.c. Undoing a
failed fallocate() only removes the folios that are not uptodate, so
keep removing those one at a time.

Dropping an 8 GiB file of 4K folios in a 32 vCPU VM takes (median of
five runs, averaged over two boots of each kernel):

  operation           before (ms)  after (ms)  change
  ftruncate() to 0           1144         901    -21%
  hole punch                 1148         906    -21%
  close() of a memfd         1139         912    -20%
  unlink()                   1142         914    -20%

Files of PMD-sized folios take 18-31 ms with or without this change.

Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
---
V2 -> V3:
- Delete the folios from fbatch directly instead of collecting them in
  a second folio_batch, which pushed the shmem_undo_range() stack frame
  past 1024 bytes on mips with clang (reported by the kernel test robot)
- Rebase onto bf87d3fe8b35 in core.git shmem-7.5.misc and update the
  numbers from a new run
- Drop the Reviewed-by tags from Jan and Baolin, since the loop changed

V1 -> V2:
- Unlock the folios after folio_batch_remove_exceptionals(), as Jan
  suggested
- Rebase onto v7.4-rc1-132-g05441d697f7f and update the numbers from a
  new run

v2: https://lore.kernel.org/r/20261105-bug-mm-shmem-undo-batch-v2-1-ae5f89ced044@gmail.com
v1: https://lore.kernel.org/r/20261029-bug-mm-shmem-undo-batch-v1-1-59b34212fe78@gmail.com

To see it, fill and remove a large /dev/shm file:

  fallocate -l 8G /dev/shm/f && time rm /dev/shm/f
---
 mm/internal.h | 1 +
 mm/shmem.c    | 9 +++++++--
 mm/truncate.c | 2 +-
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 566cbd2dce485..abe64e49a4170 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -628,6 +628,7 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start,
 		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
 unsigned find_get_entries(struct address_space *mapping, pgoff_t *start,
 		pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices);
+void truncate_cleanup_folio(struct folio *folio);
 int truncate_inode_folio(struct address_space *mapping, struct folio *folio);
 bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart,
 		loff_t lend, pgoff_t *pstart, pgoff_t *pend);
diff --git a/mm/shmem.c b/mm/shmem.c
index ce92e74baeb75..5bcbdd6078a73 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1363,11 +1363,16 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend,
 				continue;
 			}
 
-			if (!unfalloc || !folio_test_uptodate(folio))
+			if (!unfalloc)
+				truncate_cleanup_folio(folio);
+			else if (!folio_test_uptodate(folio))
 				truncate_inode_folio(mapping, folio);
-			folio_unlock(folio);
 		}
 		folio_batch_remove_exceptionals(&fbatch);
+		if (!unfalloc)
+			delete_from_page_cache_batch(mapping, &fbatch);
+		for (i = 0; i < folio_batch_count(&fbatch); i++)
+			folio_unlock(fbatch.folios[i]);
 		folio_batch_release(&fbatch);
 		cond_resched_tasks_rcu_qs();
 	}
diff --git a/mm/truncate.c b/mm/truncate.c
index 733d2713a875a..c6bf4aa0b91e8 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -151,7 +151,7 @@ EXPORT_SYMBOL_GPL(folio_invalidate);
  * its lock, b) when a concurrent invalidate_mapping_pages got there first and
  * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
  */
-static void truncate_cleanup_folio(struct folio *folio)
+void truncate_cleanup_folio(struct folio *folio)
 {
 	if (folio_mapped(folio))
 		unmap_mapping_folio(folio);

---
base-commit: bf87d3fe8b35087c2c043642e76e9f100d61328b
change-id: 20261029-bug-mm-shmem-undo-batch-de001d9c6f1e

Best regards,
-- 
Qiliang Yuan <qiliang.yuan@linux.dev>

Re: [PATCH v3] mm/shmem: remove truncated folios from the page cache in batches

[Baolin Wang]

On 11/12/26 5:05 PM, Qiliang Yuan wrote:
> truncate_inode_pages_range() takes the folios it finds out of the page
> cache with delete_from_page_cache_batch(), which takes the inode's
> i_lock and the i_pages lock once for each batch of up to
> FOLIO_BATCH_SIZE folios.
> 
> shmem_undo_range() calls truncate_inode_folio() on each folio instead,
> so filemap_remove_folio() takes and drops both locks once per folio.
> Truncating, punching or freeing a tmpfs file or memfd made of small
> folios, such as a large /dev/shm segment, pays for that on every page.
> 
> Clean each folio up as truncate_inode_folio() does and, once the swap
> entries are gone, pass the batch to delete_from_page_cache_batch(),
> which needs truncate_cleanup_folio() outside mm/truncate.c. Undoing a
> failed fallocate() only removes the folios that are not uptodate, so
> keep removing those one at a time.
> 
> Dropping an 8 GiB file of 4K folios in a 32 vCPU VM takes (median of
> five runs, averaged over two boots of each kernel):
> 
>    operation           before (ms)  after (ms)  change
>    ftruncate() to 0           1144         901    -21%
>    hole punch                 1148         906    -21%
>    close() of a memfd         1139         912    -20%
>    unlink()                   1142         914    -20%
> 
> Files of PMD-sized folios take 18-31 ms with or without this change.
> 
> Signed-off-by: Qiliang Yuan <qiliang.yuan@linux.dev>
> ---

LGTM.
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>