[PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[Seunghyun An]

On the per-VMA-lock fault path, each anonymous fault increments
vma->vm_refcnt in vma_start_read() and decrements it in vma_end_read().
When threads fault on the same VMA, that cacheline moves between CPUs
anyway. On current mainline, extra reads of the same line
happen in between:

 1. __folio_set_anon() reads __vm_anon_pgoff_hi (offset 132, next to
    vm_refcnt) via linear_anon_page_index(), since the anon pgoff series.
 2. __vmf_anon_prepare() -> vma_has_anon_rmap() runs vma_is_attached() and
    vma_assert_stabilised(), which load vm_refcnt (and the mmap_lock
    count) even with CONFIG_DEBUG_VM=n, on every anonymous fault that
    allocates a page or breaks CoW. This came with v7.4-rc1.

These reads can miss, and they also make the following inc/dec more
expensive. Patch 1 computes the anon index from vm_pgoff for anonymous
VMAs. Patch 2 makes the assertion in vma_has_anon_rmap() DEBUG_VM-only.

AMD EPYC 9634 (84 cores, 12 CCDs), base is mainline d48723711bae (after
v7.4-rc1), CONFIG_DEBUG_VM=n. Threads faulting on one shared anonymous
VMA (each its own slice, zapped with MADV_DONTNEED), faults/s, median of
3, all three kernels measured on the same day:

  CPUs                      base      +patch 1  +patch 2   gain
  1 CCD,   7 threads        2.360M    2.369M    2.425M     +2.8%
  7 CCDs,  1 thread each    1.738M    1.982M    2.215M    +27.4%
  12 CCDs, 1 thread each    2.288M    2.548M    2.949M    +28.9%

With one VMA per thread, or with processes, both patches together change
throughput by less than 2%. In every cell of this and the next table the
spread between runs (min to max) is about 5% or less.

pft (https://github.com/gormanm/pft), which was mmtests'
config-workload-pft-threads until it was dropped there in favour of the
will-it-scale malloc/brk tests, does the same thing without the zap: one
anonymous region split between the threads. Same machine and kernels,
faults per wall-clock second (pft's fault/wsec), median of 10 (of 5 for
the last row), "pft -a -m <4*t>G -t <t>" (pft needs a one-line fix to
accept a CPU mask that does not start at CPU 0):

  CPUs                      base      +patch 1  +patch 2   gain
  1 CCD,   7 threads        2.938M    2.955M    3.044M     +3.6%
  7 CCDs,  1 thread each    1.938M    2.215M    2.704M    +39.5%
  12 CCDs, 1 thread each    2.456M    2.673M    3.547M    +44.4%
  12 CCDs, 7 threads each   4.132M    4.863M    5.196M    +25.8%

With processes ("-n 7", 7 CCDs) the change is +0.7%. With all 84 cores,
threaded pft reached 4.132M against 5.243M for the same run with
processes (-n 84), 21% less; with both patches it is 5.196M against
5.282M, 1.6% less.

None of the will-it-scale tests mmtests runs now (page_fault*, malloc*,
brk*) has many threads faulting one anonymous VMA. page_fault1 threads
only shares a VMA when neighbouring mappings merge, and it is then
dominated by munmap write-locking that VMA (the threads were about 45%
idle here), so it does not resolve this difference (it also varies by up
to 10% between boots of the same kernel). malloc* and brk* take almost no
page faults.

A real-world instance of the pattern: OpenJDK with -XX:+AlwaysPreTouch
and G1 has the whole heap in one anonymous VMA and pretouches it with
ordinary write faults from all GC threads (108 here, not pinned). OpenJDK
25, -Xms32g -Xmx32g; time from exec to main() with -XX:+AlwaysPreTouch
minus the same without it, median of 10:

                    base      +patch 1  +patch 2
  G1 pretouch       1866 ms   1653 ms   1658 ms   (-11.1%)
  Parallel pretouch 1872 ms   1644 ms   1658 ms   (-11.4%)

Patch 2 adds nothing there; with patch 1 the JVM already faults at about
5.1M faults/s, close to what this machine sustains for 4K anonymous
faults from all CCDs in the other tests. This is a one-time startup cost;
once the heap is populated there are no such faults.

Tested with CONFIG_DEBUG_VM=y and CONFIG_PROVE_LOCKING=y: the mm selftests
(run_vmtests.sh -t "mmap mremap vma_merge rmap cow userfaultfd madv_guard
ksm migration thp") pass with no warnings, and so does a test that
soft-offlines anonymous pages faulted after mremap(), fork() and on a
MAP_PRIVATE /dev/zero mapping. LOCKDEP=y with DEBUG_VM=n, and RISC-V nommu
with and without DEBUG_VM, build without warnings.

Seunghyun An (2):
  mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()
  mm/vma: only assert VMA stability in vma_has_anon_rmap() with
    CONFIG_DEBUG_VM

 include/linux/pagemap.h         | 19 ++++++++++++++-----
 mm/vma.h                        |  8 +++++++-
 tools/testing/vma/include/dup.h | 13 ++++++++-----
 3 files changed, 29 insertions(+), 11 deletions(-)


base-commit: d48723711bae9b8cdc56719c448ba42fef28cbee
-- 
2.53.0

[PATCH 1/2] mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()

[Seunghyun An]

__vm_anon_pgoff_hi sits right after vm_refcnt in the vm_refcnt
cacheline. Since commit 93c0c8dc87f6 ("mm/rmap: use anon pgoff to track
MAP_PRIVATE file-backed anon folios"), which made anon rmap use it,
__folio_set_anon() reads it on every new anonymous folio, i.e. once per
fault that maps a new anonymous page, between vma_start_read() and
vma_end_read().

When several threads fault on the same VMA, the other CPUs keep writing
vm_refcnt from vma_start_read()/vma_end_read(), so this read often
misses and pulls the line over. The extra load can force a coherence
transfer and makes the following vm_refcnt updates more expensive.

For anonymous VMAs the anonymous page offset is always equal to vm_pgoff
(linear_anon_page_index() already asserts this under CONFIG_DEBUG_VM;
it holds for MAP_PRIVATE /dev/zero too since commit e59c9a971a9f
("mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous")),
so compute the index from vm_pgoff as before. That brings back the
pre-series behaviour for the hot path; MAP_PRIVATE file-backed VMAs still
read the split field. Keep the assertion, the other way round.

Moving __vm_anon_pgoff_hi out of the line does not work without growing
vm_area_struct. With e.g. CONFIG_ANON_VMA_NAME=y the struct is exactly
192 bytes, cachelines 0 and 1 have no hole, and the only 4-byte fields
that could fill the slot next to vm_refcnt are vm_lock_seq and the lo/hi
halves. vm_lock_seq is read before vm_refcnt is taken in vma_start_read(),
so it would just move the miss.

Measured on an AMD EPYC 9634 (84 cores, 12 CCDs, CONFIG_DEBUG_VM=n), on
mainline after v7.4-rc1 (the base of this series). Threads share one
anonymous VMA, each faults its own slice and zaps it with MADV_DONTNEED;
faults/s, median of 3:

  CPUs                      base      patched
  1 CCD,   7 threads        2.360M    2.369M   +0.4%
  7 CCDs,  1 thread each    1.738M    1.982M  +14.0%
  12 CCDs, 1 thread each    2.288M    2.548M  +11.4%

On 7.3.1 (with the 7.3 variant of this change) the same runs gave +1.9%,
+16.0% and +29.4%. There, perf c2c shows the folio_add_new_anon_rmap()
load at offset 132 disappearing from the vm_refcnt line and the average
load latency of the vm_refcnt inc/dec on that line dropping by about
45%. With one VMA per thread, or with processes, this patch alone
changes throughput by -2.0% to -0.1% on that base (less than 1% on 7.3.1);
the next patch more than recovers that.

will-it-scale page_fault1 threads does not show the difference on this
machine: each thread mmap()s and munmap()s its own region, which merges
VMAs and is dominated by VMA write locking, and its results vary by up
to 10% between boots of the same kernel.

This relies on the /dev/zero commit above, which is not in v7.3, so this
is not tagged for stable: without it, MAP_PRIVATE /dev/zero VMAs are
anonymous but keep the file offset in vm_pgoff, and must keep using the
anon pgoff field.

The tools/testing/vma copy of the helper is updated to match.

Signed-off-by: Seunghyun An <san55@wisc.edu>
---
 include/linux/pagemap.h         | 19 ++++++++++++++-----
 tools/testing/vma/include/dup.h | 13 ++++++++-----
 2 files changed, 22 insertions(+), 10 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 73af18a373..300ee83fc1 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1122,13 +1122,22 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
 static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
 		const unsigned long address)
 {
-	const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
 	VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
-	if (vma_is_anonymous(vma))
-		VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
 
-	return pgoff;
+	/*
+	 * For anonymous VMAs the anonymous page offset always equals vm_pgoff,
+	 * so use it. This keeps the anonymous fault path (__folio_set_anon())
+	 * off __vm_anon_pgoff_hi, which shares a cacheline with vm_refcnt and
+	 * is therefore contended by concurrent faults on the same VMA.
+	 */
+	if (vma_is_anonymous(vma)) {
+		const pgoff_t pgoff = linear_page_index(vma, address);
+
+		VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+		return pgoff;
+	}
+
+	return __linear_anon_page_index(vma, address);
 }
 
 struct wait_page_key {
diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h
index dc4927d262..f3d69b0c74 100644
--- a/tools/testing/vma/include/dup.h
+++ b/tools/testing/vma/include/dup.h
@@ -1655,13 +1655,16 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
 static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
 		const unsigned long address)
 {
-	const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
 	VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
-	if (vma_is_anonymous(vma))
-		VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
 
-	return pgoff;
+	if (vma_is_anonymous(vma)) {
+		const pgoff_t pgoff = linear_page_index(vma, address);
+
+		VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+		return pgoff;
+	}
+
+	return __linear_anon_page_index(vma, address);
 }
 
 extern const struct file_operations zero_fops;
-- 
2.53.0

[PATCH 2/2] mm/vma: only assert VMA stability in vma_has_anon_rmap() with CONFIG_DEBUG_VM

[Seunghyun An]

vma_has_anon_rmap() calls vma_is_attached() and vma_assert_stabilised()
unconditionally. Without lockdep, vma_assert_stabilised() reads
mm->mmap_lock and then vma_assert_locked() reads vm_refcnt. These reads
stay in CONFIG_DEBUG_VM=n builds: refcount_read() and rwsem_is_locked()
are atomic reads (READ_ONCE()), which the compiler has to keep even
though the VM_WARN_ON_ONCE_VMA() checks consuming them compile away.

__vmf_anon_prepare() calls this on every anonymous fault that allocates
a new page or breaks CoW, so every such fault handled under the per-VMA
lock now loads vm_refcnt twice more, plus the mmap_lock count. When
threads fault on the same VMA,
other CPUs keep writing vm_refcnt in vma_start_read() and vma_end_read(),
so these loads can miss.

The assertion has no effect outside debug builds, so only do it with
CONFIG_DEBUG_VM. The fast path is then a plain vma->anon_vma test again.

On top of the previous patch, same setup as there (AMD EPYC 9634, threads
faulting on one shared anonymous VMA, faults/s, median of 3):

  CPUs                      before    after
  1 CCD,   7 threads        2.369M    2.425M   +2.4%
  7 CCDs,  1 thread each    1.982M    2.215M  +11.7%
  12 CCDs, 1 thread each    2.548M    2.949M  +15.7%

With one VMA per thread, or with processes, the change is +0.3% to
+1.9% (the loads are gone from the local path as well). perf c2c no
longer shows __vmf_anon_prepare() on the vm_refcnt line.

Fixes: 38564710f443 ("mm: implement and use vma_has_anon_rmap(), silence KCSAN")
Signed-off-by: Seunghyun An <san55@wisc.edu>
---
 mm/vma.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/mm/vma.h b/mm/vma.h
index 3fc3b47a95..4562bd4881 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -278,7 +278,13 @@ static inline pgoff_t vmg_end_pgoff(const struct vma_merge_struct *vmg)
  */
 static inline bool vma_has_anon_rmap(const struct vm_area_struct *vma)
 {
-	if (vma_is_attached(vma))
+	/*
+	 * Only check the locking under CONFIG_DEBUG_VM. vma_is_attached() and
+	 * vma_assert_locked() read vm_refcnt, which concurrent faults on the
+	 * same VMA keep writing, and __vmf_anon_prepare() calls this on the
+	 * anonymous page allocation and CoW fault paths.
+	 */
+	if (IS_ENABLED(CONFIG_DEBUG_VM) && vma_is_attached(vma))
 		vma_assert_stabilised(vma);
 	/* KCSAN gets confused about the optimistic check. Silence it. */
 	return data_race(vma->anon_vma);
-- 
2.53.0