[RFC PATCH 00/18] mm: read the page cache without folio references

["Kiryl Shutsemau (Meta)" <kas@kernel.org>]

Small reads of one file from many CPUs all take and drop a reference on
the same folio, and its refcount bounces between the CPUs.  This series
lets filemap_read() and shmem copy from the page cache without taking a
reference.  It is Linus' RFC [1], reworked: a folio that such a reader
may copy from is freed only after an RCU grace period.

Only the folios that reads keep coming back to get the flag,
PG_rcu_free: those that folio_mark_accessed() finds as accessed as the
LRU records.  Streaming reads and data that never gets hot are not
deferred, so the memory held back stays small and most frees are not
touched.

  1      PG_rcu_free and the deferred free (64-bit only)
  2-3    splice and fuse no longer reuse page cache folios that a
         lockless reader may still copy from
  4-7    NR_FREE_RCU_PAGES; an expedited grace period when a CPU's
         deferred frees pile up and memory runs short; the waiting
         pages are freed before the allocator, alloc_contig_range() or
         memory offline give up
  8      memory_failure() and soft offline deal with such pages
  9-11   filemap_read() marks a folio accessed in one place and sets
         PG_rcu_free on the folios that reads keep coming back to
  12     the folios of an inode being evicted are freed at once
  13     f_ra.prev_pos is written only when it changes
  14     filemap_read() copies small folios without a reference
  15     ... and large folios, with one more lookup per folio
  16-17  shmem does the same
  18     a debugfs switch to turn the reads off

Numbers: 64-vCPU KVM guest on a Threadripper 7975WX (Zen 4), 2G, no
debug options; base is v7.3-rc7.

With the default mitigations, every read on this machine pays ~400 ns
at the user/kernel boundary (TSA ~170 ns, SRSO ~230 ns): a single thread
does ~1.8M reads/s.  With mitigations=off it does 7.1-8.8M, and the
64-thread gain on ext4 grows from 4.5x to 13x.  Linus measured 7.4-7.8M
for one thread and 13.3M -> 304M for 64 threads on a Threadripper 3970X
[1].  Rows without "mit=off" use the default mitigations.

  16-byte pread() at offset 0, each thread with its own fd, M reads/s:
                                 base           this series
    ext4, 64 threads             15.0-15.7      68.9-70.1
    ext4, 64 threads, mit=off    17.2           217-220
    tmpfs, 64 threads            1.6-1.7        71-72
    tmpfs, 64 threads, mit=off   1.8-1.9        230-239
    XFS, 64 threads              11.3-11.4      20.4-20.5
    block dev, 64 threads        14.7           28.8-28.9
    1 thread, any of them        1.6-1.9        1.7-2.0

  The mit=off rows are from an earlier version with the same small-read
  path.  Base tmpfs takes the folio lock on every read: 64 threads read
  as fast as one.  XFS and the block device take i_rwsem shared on every
  read (XFS_IOLOCK_SHARED, inode_lock_shared() in blkdev_read_iter()),
  and its count bounces in place of the refcount.

  64 threads reading 64K at offset 0 of one file, one large folio,
  k MiB/s:
                           base           this series
    huge tmpfs             96-100         1370-1390
    ext4                   756-775        1355-1373

  Large sequential reads of a 256M file: single-thread within ~1% of
  base, 16 threads at base.

  Next to a process that touches 1400M in a loop: no OOM kill.  A load
  that drops 1G of hot data with FADV_DONTNEED every round runs between
  9% below and 1% above base, which itself spans 63-75k MiB/s there.  A
  load that keeps refaulting loses 4-6%, as Linus' RFC does.

Open:

- Patch 12 relies on mapping_exiting() meaning that no file holds the
  inode open.  f2fs called truncate_inode_pages_final() on open inodes
  for atomic writes; a fix is on its way [2].
- Patch 17 drops the folio lock from tmpfs reads.  shmem_get_folio()
  held it only for the lookup, so a read was never atomic against a
  write or a hole punch; now a read also does not wait for one that
  holds the lock, as filemap_read() does not.  I believe this is safe,
  but would like Hugh to double check.
- A deferred folio is uncharged from its memcg before it waits, as every
  page cache folio is.  For a grace period it counts towards no cgroup.
- A migrated flagged folio is freed after a grace period, so a pageblock
  that compaction just emptied is free only then.  defrag_mode leans on
  free pageblocks; not measured yet.
- The flag is a page flag, 64-bit only.  On large configs it costs MGLRU
  one refs bit.
- On writably mapped files, architectures that implement
  flush_dcache_page() take the slow path.
- filemap_mark_hot() still tests uptodate.  Willy pointed out that a
  filesystem clearing it under a reader is a filesystem bug; the test is
  cheap.

The series is also at:
  https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git rcu-free/rfc-v1

[1] https://lore.kernel.org/all/CAHk-=wjLjgfwgSZ1RDk4YdNFcV0ntosMO5AmBoHD76Zb3HFY7Q@mail.gmail.com/
[2] https://lore.kernel.org/all/20261015161714.1586334-1-daeho43@gmail.com/

Kiryl Shutsemau (Meta) (11):
  fuse: don't move a folio with PG_rcu_free into a fuse file
  mm: count the pages waiting for a grace period
  mm: free the pages waiting for a grace period before OOM
  mm: free the pages waiting for a grace period before isolation fails
  mm/memory-failure: handle folios with PG_rcu_free
  mm/filemap: don't defer the free of folios of an inode being evicted
  mm/filemap: store the end of a read in f_ra only when it changes
  mm/filemap: read large folios without folio references
  mm/shmem: count accesses to a folio the way filemap_read() does
  mm/shmem: read without folio references
  mm/filemap: add a debugfs switch for lockless page cache reads

Linus Torvalds (7):
  mm: free PG_rcu_free folios only after an RCU grace period
  splice: don't steal page cache folios
  mm: expedite the grace period when a CPU has many deferred frees
  mm: have folio_mark_accessed() return whether the folio is hot
  mm/filemap: mark a folio accessed in one place in filemap_read()
  mm/filemap: set PG_rcu_free on the folios that reads keep coming back
    to
  mm/filemap: read from the page cache without folio references

 fs/fuse/dev.c                  |   4 +
 fs/splice.c                    |  51 -----
 include/linux/mmzone.h         |   1 +
 include/linux/page-flags.h     |  24 +++
 include/linux/pagemap.h        |  12 ++
 include/linux/swap.h           |   2 +-
 include/linux/vm_event_item.h  |   2 +
 include/trace/events/mmflags.h |   7 +
 mm/Kconfig                     |  15 ++
 mm/filemap.c                   | 337 +++++++++++++++++++++++++++++++--
 mm/folio.c                     |  46 +++--
 mm/huge_memory.c               |   1 +
 mm/internal.h                  |   3 +
 mm/memory-failure.c            |  28 +++
 mm/memory_hotplug.c            |   7 +
 mm/page_alloc.c                | 284 ++++++++++++++++++++++++++-
 mm/page_alloc.h                |   1 +
 mm/shmem.c                     |  22 ++-
 mm/show_mem.c                  |   9 +-
 mm/vmstat.c                    |   6 +
 20 files changed, 770 insertions(+), 92 deletions(-)


base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222
-- 
2.54.0