[PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[Seunghyun An <san55@wisc.edu>]

On the per-VMA-lock fault path, each anonymous fault increments
vma->vm_refcnt in vma_start_read() and decrements it in vma_end_read().
When threads fault on the same VMA, that cacheline moves between CPUs
anyway. On current mainline, extra reads of the same line
happen in between:

 1. __folio_set_anon() reads __vm_anon_pgoff_hi (offset 132, next to
    vm_refcnt) via linear_anon_page_index(), since the anon pgoff series.
 2. __vmf_anon_prepare() -> vma_has_anon_rmap() runs vma_is_attached() and
    vma_assert_stabilised(), which load vm_refcnt (and the mmap_lock
    count) even with CONFIG_DEBUG_VM=n, on every anonymous fault that
    allocates a page or breaks CoW. This came with v7.4-rc1.

These reads can miss, and they also make the following inc/dec more
expensive. Patch 1 computes the anon index from vm_pgoff for anonymous
VMAs. Patch 2 makes the assertion in vma_has_anon_rmap() DEBUG_VM-only.

AMD EPYC 9634 (84 cores, 12 CCDs), base is mainline d48723711bae (after
v7.4-rc1), CONFIG_DEBUG_VM=n. Threads faulting on one shared anonymous
VMA (each its own slice, zapped with MADV_DONTNEED), faults/s, median of
3, all three kernels measured on the same day:

  CPUs                      base      +patch 1  +patch 2   gain
  1 CCD,   7 threads        2.360M    2.369M    2.425M     +2.8%
  7 CCDs,  1 thread each    1.738M    1.982M    2.215M    +27.4%
  12 CCDs, 1 thread each    2.288M    2.548M    2.949M    +28.9%

With one VMA per thread, or with processes, both patches together change
throughput by less than 2%. In every cell of this and the next table the
spread between runs (min to max) is about 5% or less.

pft (https://github.com/gormanm/pft), which was mmtests'
config-workload-pft-threads until it was dropped there in favour of the
will-it-scale malloc/brk tests, does the same thing without the zap: one
anonymous region split between the threads. Same machine and kernels,
faults per wall-clock second (pft's fault/wsec), median of 10 (of 5 for
the last row), "pft -a -m <4*t>G -t <t>" (pft needs a one-line fix to
accept a CPU mask that does not start at CPU 0):

  CPUs                      base      +patch 1  +patch 2   gain
  1 CCD,   7 threads        2.938M    2.955M    3.044M     +3.6%
  7 CCDs,  1 thread each    1.938M    2.215M    2.704M    +39.5%
  12 CCDs, 1 thread each    2.456M    2.673M    3.547M    +44.4%
  12 CCDs, 7 threads each   4.132M    4.863M    5.196M    +25.8%

With processes ("-n 7", 7 CCDs) the change is +0.7%. With all 84 cores,
threaded pft reached 4.132M against 5.243M for the same run with
processes (-n 84), 21% less; with both patches it is 5.196M against
5.282M, 1.6% less.

None of the will-it-scale tests mmtests runs now (page_fault*, malloc*,
brk*) has many threads faulting one anonymous VMA. page_fault1 threads
only shares a VMA when neighbouring mappings merge, and it is then
dominated by munmap write-locking that VMA (the threads were about 45%
idle here), so it does not resolve this difference (it also varies by up
to 10% between boots of the same kernel). malloc* and brk* take almost no
page faults.

A real-world instance of the pattern: OpenJDK with -XX:+AlwaysPreTouch
and G1 has the whole heap in one anonymous VMA and pretouches it with
ordinary write faults from all GC threads (108 here, not pinned). OpenJDK
25, -Xms32g -Xmx32g; time from exec to main() with -XX:+AlwaysPreTouch
minus the same without it, median of 10:

                    base      +patch 1  +patch 2
  G1 pretouch       1866 ms   1653 ms   1658 ms   (-11.1%)
  Parallel pretouch 1872 ms   1644 ms   1658 ms   (-11.4%)

Patch 2 adds nothing there; with patch 1 the JVM already faults at about
5.1M faults/s, close to what this machine sustains for 4K anonymous
faults from all CCDs in the other tests. This is a one-time startup cost;
once the heap is populated there are no such faults.

Tested with CONFIG_DEBUG_VM=y and CONFIG_PROVE_LOCKING=y: the mm selftests
(run_vmtests.sh -t "mmap mremap vma_merge rmap cow userfaultfd madv_guard
ksm migration thp") pass with no warnings, and so does a test that
soft-offlines anonymous pages faulted after mremap(), fork() and on a
MAP_PRIVATE /dev/zero mapping. LOCKDEP=y with DEBUG_VM=n, and RISC-V nommu
with and without DEBUG_VM, build without warnings.

Seunghyun An (2):
  mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()
  mm/vma: only assert VMA stability in vma_has_anon_rmap() with
    CONFIG_DEBUG_VM

 include/linux/pagemap.h         | 19 ++++++++++++++-----
 mm/vma.h                        |  8 +++++++-
 tools/testing/vma/include/dup.h | 13 ++++++++-----
 3 files changed, 29 insertions(+), 11 deletions(-)


base-commit: d48723711bae9b8cdc56719c448ba42fef28cbee
-- 
2.53.0