[PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[Seunghyun An]

On the per-VMA-lock fault path, each anonymous fault increments
vma->vm_refcnt in vma_start_read() and decrements it in vma_end_read().
When threads fault on the same VMA, that cacheline moves between CPUs
anyway. On current mainline, extra reads of the same line
happen in between:

 1. __folio_set_anon() reads __vm_anon_pgoff_hi (offset 132, next to
    vm_refcnt) via linear_anon_page_index(), since the anon pgoff series.
 2. __vmf_anon_prepare() -> vma_has_anon_rmap() runs vma_is_attached() and
    vma_assert_stabilised(), which load vm_refcnt (and the mmap_lock
    count) even with CONFIG_DEBUG_VM=n, on every anonymous fault that
    allocates a page or breaks CoW. This came with v7.4-rc1.

These reads can miss, and they also make the following inc/dec more
expensive. Patch 1 computes the anon index from vm_pgoff for anonymous
VMAs. Patch 2 makes the assertion in vma_has_anon_rmap() DEBUG_VM-only.

AMD EPYC 9634 (84 cores, 12 CCDs), base is mainline d48723711bae (after
v7.4-rc1), CONFIG_DEBUG_VM=n. Threads faulting on one shared anonymous
VMA (each its own slice, zapped with MADV_DONTNEED), faults/s, median of
3, all three kernels measured on the same day:

  CPUs                      base      +patch 1  +patch 2   gain
  1 CCD,   7 threads        2.360M    2.369M    2.425M     +2.8%
  7 CCDs,  1 thread each    1.738M    1.982M    2.215M    +27.4%
  12 CCDs, 1 thread each    2.288M    2.548M    2.949M    +28.9%

With one VMA per thread, or with processes, both patches together change
throughput by less than 2%. In every cell of this and the next table the
spread between runs (min to max) is about 5% or less.

pft (https://github.com/gormanm/pft), which was mmtests'
config-workload-pft-threads until it was dropped there in favour of the
will-it-scale malloc/brk tests, does the same thing without the zap: one
anonymous region split between the threads. Same machine and kernels,
faults per wall-clock second (pft's fault/wsec), median of 10 (of 5 for
the last row), "pft -a -m <4*t>G -t <t>" (pft needs a one-line fix to
accept a CPU mask that does not start at CPU 0):

  CPUs                      base      +patch 1  +patch 2   gain
  1 CCD,   7 threads        2.938M    2.955M    3.044M     +3.6%
  7 CCDs,  1 thread each    1.938M    2.215M    2.704M    +39.5%
  12 CCDs, 1 thread each    2.456M    2.673M    3.547M    +44.4%
  12 CCDs, 7 threads each   4.132M    4.863M    5.196M    +25.8%

With processes ("-n 7", 7 CCDs) the change is +0.7%. With all 84 cores,
threaded pft reached 4.132M against 5.243M for the same run with
processes (-n 84), 21% less; with both patches it is 5.196M against
5.282M, 1.6% less.

None of the will-it-scale tests mmtests runs now (page_fault*, malloc*,
brk*) has many threads faulting one anonymous VMA. page_fault1 threads
only shares a VMA when neighbouring mappings merge, and it is then
dominated by munmap write-locking that VMA (the threads were about 45%
idle here), so it does not resolve this difference (it also varies by up
to 10% between boots of the same kernel). malloc* and brk* take almost no
page faults.

A real-world instance of the pattern: OpenJDK with -XX:+AlwaysPreTouch
and G1 has the whole heap in one anonymous VMA and pretouches it with
ordinary write faults from all GC threads (108 here, not pinned). OpenJDK
25, -Xms32g -Xmx32g; time from exec to main() with -XX:+AlwaysPreTouch
minus the same without it, median of 10:

                    base      +patch 1  +patch 2
  G1 pretouch       1866 ms   1653 ms   1658 ms   (-11.1%)
  Parallel pretouch 1872 ms   1644 ms   1658 ms   (-11.4%)

Patch 2 adds nothing there; with patch 1 the JVM already faults at about
5.1M faults/s, close to what this machine sustains for 4K anonymous
faults from all CCDs in the other tests. This is a one-time startup cost;
once the heap is populated there are no such faults.

Tested with CONFIG_DEBUG_VM=y and CONFIG_PROVE_LOCKING=y: the mm selftests
(run_vmtests.sh -t "mmap mremap vma_merge rmap cow userfaultfd madv_guard
ksm migration thp") pass with no warnings, and so does a test that
soft-offlines anonymous pages faulted after mremap(), fork() and on a
MAP_PRIVATE /dev/zero mapping. LOCKDEP=y with DEBUG_VM=n, and RISC-V nommu
with and without DEBUG_VM, build without warnings.

Seunghyun An (2):
  mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()
  mm/vma: only assert VMA stability in vma_has_anon_rmap() with
    CONFIG_DEBUG_VM

 include/linux/pagemap.h         | 19 ++++++++++++++-----
 mm/vma.h                        |  8 +++++++-
 tools/testing/vma/include/dup.h | 13 ++++++++-----
 3 files changed, 29 insertions(+), 11 deletions(-)


base-commit: d48723711bae9b8cdc56719c448ba42fef28cbee
-- 
2.53.0

[PATCH 1/2] mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()

[Seunghyun An]

__vm_anon_pgoff_hi sits right after vm_refcnt in the vm_refcnt
cacheline. Since commit 93c0c8dc87f6 ("mm/rmap: use anon pgoff to track
MAP_PRIVATE file-backed anon folios"), which made anon rmap use it,
__folio_set_anon() reads it on every new anonymous folio, i.e. once per
fault that maps a new anonymous page, between vma_start_read() and
vma_end_read().

When several threads fault on the same VMA, the other CPUs keep writing
vm_refcnt from vma_start_read()/vma_end_read(), so this read often
misses and pulls the line over. The extra load can force a coherence
transfer and makes the following vm_refcnt updates more expensive.

For anonymous VMAs the anonymous page offset is always equal to vm_pgoff
(linear_anon_page_index() already asserts this under CONFIG_DEBUG_VM;
it holds for MAP_PRIVATE /dev/zero too since commit e59c9a971a9f
("mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous")),
so compute the index from vm_pgoff as before. That brings back the
pre-series behaviour for the hot path; MAP_PRIVATE file-backed VMAs still
read the split field. Keep the assertion, the other way round.

Moving __vm_anon_pgoff_hi out of the line does not work without growing
vm_area_struct. With e.g. CONFIG_ANON_VMA_NAME=y the struct is exactly
192 bytes, cachelines 0 and 1 have no hole, and the only 4-byte fields
that could fill the slot next to vm_refcnt are vm_lock_seq and the lo/hi
halves. vm_lock_seq is read before vm_refcnt is taken in vma_start_read(),
so it would just move the miss.

Measured on an AMD EPYC 9634 (84 cores, 12 CCDs, CONFIG_DEBUG_VM=n), on
mainline after v7.4-rc1 (the base of this series). Threads share one
anonymous VMA, each faults its own slice and zaps it with MADV_DONTNEED;
faults/s, median of 3:

  CPUs                      base      patched
  1 CCD,   7 threads        2.360M    2.369M   +0.4%
  7 CCDs,  1 thread each    1.738M    1.982M  +14.0%
  12 CCDs, 1 thread each    2.288M    2.548M  +11.4%

On 7.3.1 (with the 7.3 variant of this change) the same runs gave +1.9%,
+16.0% and +29.4%. There, perf c2c shows the folio_add_new_anon_rmap()
load at offset 132 disappearing from the vm_refcnt line and the average
load latency of the vm_refcnt inc/dec on that line dropping by about
45%. With one VMA per thread, or with processes, this patch alone
changes throughput by -2.0% to -0.1% on that base (less than 1% on 7.3.1);
the next patch more than recovers that.

will-it-scale page_fault1 threads does not show the difference on this
machine: each thread mmap()s and munmap()s its own region, which merges
VMAs and is dominated by VMA write locking, and its results vary by up
to 10% between boots of the same kernel.

This relies on the /dev/zero commit above, which is not in v7.3, so this
is not tagged for stable: without it, MAP_PRIVATE /dev/zero VMAs are
anonymous but keep the file offset in vm_pgoff, and must keep using the
anon pgoff field.

The tools/testing/vma copy of the helper is updated to match.

Signed-off-by: Seunghyun An <san55@wisc.edu>
---
 include/linux/pagemap.h         | 19 ++++++++++++++-----
 tools/testing/vma/include/dup.h | 13 ++++++++-----
 2 files changed, 22 insertions(+), 10 deletions(-)

diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 73af18a373..300ee83fc1 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1122,13 +1122,22 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
 static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
 		const unsigned long address)
 {
-	const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
 	VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
-	if (vma_is_anonymous(vma))
-		VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
 
-	return pgoff;
+	/*
+	 * For anonymous VMAs the anonymous page offset always equals vm_pgoff,
+	 * so use it. This keeps the anonymous fault path (__folio_set_anon())
+	 * off __vm_anon_pgoff_hi, which shares a cacheline with vm_refcnt and
+	 * is therefore contended by concurrent faults on the same VMA.
+	 */
+	if (vma_is_anonymous(vma)) {
+		const pgoff_t pgoff = linear_page_index(vma, address);
+
+		VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+		return pgoff;
+	}
+
+	return __linear_anon_page_index(vma, address);
 }
 
 struct wait_page_key {
diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h
index dc4927d262..f3d69b0c74 100644
--- a/tools/testing/vma/include/dup.h
+++ b/tools/testing/vma/include/dup.h
@@ -1655,13 +1655,16 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma,
 static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma,
 		const unsigned long address)
 {
-	const pgoff_t pgoff = __linear_anon_page_index(vma, address);
-
 	VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma));
-	if (vma_is_anonymous(vma))
-		VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address));
 
-	return pgoff;
+	if (vma_is_anonymous(vma)) {
+		const pgoff_t pgoff = linear_page_index(vma, address);
+
+		VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address));
+		return pgoff;
+	}
+
+	return __linear_anon_page_index(vma, address);
 }
 
 extern const struct file_operations zero_fops;
-- 
2.53.0

[PATCH 2/2] mm/vma: only assert VMA stability in vma_has_anon_rmap() with CONFIG_DEBUG_VM

[Seunghyun An]

vma_has_anon_rmap() calls vma_is_attached() and vma_assert_stabilised()
unconditionally. Without lockdep, vma_assert_stabilised() reads
mm->mmap_lock and then vma_assert_locked() reads vm_refcnt. These reads
stay in CONFIG_DEBUG_VM=n builds: refcount_read() and rwsem_is_locked()
are atomic reads (READ_ONCE()), which the compiler has to keep even
though the VM_WARN_ON_ONCE_VMA() checks consuming them compile away.

__vmf_anon_prepare() calls this on every anonymous fault that allocates
a new page or breaks CoW, so every such fault handled under the per-VMA
lock now loads vm_refcnt twice more, plus the mmap_lock count. When
threads fault on the same VMA,
other CPUs keep writing vm_refcnt in vma_start_read() and vma_end_read(),
so these loads can miss.

The assertion has no effect outside debug builds, so only do it with
CONFIG_DEBUG_VM. The fast path is then a plain vma->anon_vma test again.

On top of the previous patch, same setup as there (AMD EPYC 9634, threads
faulting on one shared anonymous VMA, faults/s, median of 3):

  CPUs                      before    after
  1 CCD,   7 threads        2.369M    2.425M   +2.4%
  7 CCDs,  1 thread each    1.982M    2.215M  +11.7%
  12 CCDs, 1 thread each    2.548M    2.949M  +15.7%

With one VMA per thread, or with processes, the change is +0.3% to
+1.9% (the loads are gone from the local path as well). perf c2c no
longer shows __vmf_anon_prepare() on the vm_refcnt line.

Fixes: 38564710f443 ("mm: implement and use vma_has_anon_rmap(), silence KCSAN")
Signed-off-by: Seunghyun An <san55@wisc.edu>
---
 mm/vma.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/mm/vma.h b/mm/vma.h
index 3fc3b47a95..4562bd4881 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -278,7 +278,13 @@ static inline pgoff_t vmg_end_pgoff(const struct vma_merge_struct *vmg)
  */
 static inline bool vma_has_anon_rmap(const struct vm_area_struct *vma)
 {
-	if (vma_is_attached(vma))
+	/*
+	 * Only check the locking under CONFIG_DEBUG_VM. vma_is_attached() and
+	 * vma_assert_locked() read vm_refcnt, which concurrent faults on the
+	 * same VMA keep writing, and __vmf_anon_prepare() calls this on the
+	 * anonymous page allocation and CoW fault paths.
+	 */
+	if (IS_ENABLED(CONFIG_DEBUG_VM) && vma_is_attached(vma))
 		vma_assert_stabilised(vma);
 	/* KCSAN gets confused about the optimistic check. Silence it. */
 	return data_race(vma->anon_vma);
-- 
2.53.0

Re: [PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[Lorenzo Stoakes (ARM)]

Thanks for the report.

You are a complete newcomer to mm, and you are sending very subtle changes
to a core part of it, so I am not confident that you have the depth of
understanding the kernel requires for such work. See
https://docs.kernel.org/process/generated-content.html

I'll therefore take over this change and treat this as a bug report.

I will give you due credit for reporting the issue of course.

I'd like to collaborate on the perf measurements however, so if you could
re-run your test results on my series when I send it that'd be appreciated.

In general it seems that the actual issue is the debug assertion, given you
state only vma_has_anon_rmap() as a problem here.

In general in future - while we're fine with you using LLMs, newcomers
should work on small, incremental changes to mm to build up understanding
first.

--
Cheers, Lorenzo

Re: [PATCH 1/2] mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()

[Pedro Falcato]

On Mon, Nov 09, 2026 at 06:00:01PM -0600, Seunghyun An wrote:
> __vm_anon_pgoff_hi sits right after vm_refcnt in the vm_refcnt
> cacheline. Since commit 93c0c8dc87f6 ("mm/rmap: use anon pgoff to track
> MAP_PRIVATE file-backed anon folios"), which made anon rmap use it,
> __folio_set_anon() reads it on every new anonymous folio, i.e. once per
> fault that maps a new anonymous page, between vma_start_read() and
> vma_end_read().
> 
> When several threads fault on the same VMA, the other CPUs keep writing
> vm_refcnt from vma_start_read()/vma_end_read(), so this read often
> misses and pulls the line over. The extra load can force a coherence
> transfer and makes the following vm_refcnt updates more expensive.
> 
> For anonymous VMAs the anonymous page offset is always equal to vm_pgoff
> (linear_anon_page_index() already asserts this under CONFIG_DEBUG_VM;
> it holds for MAP_PRIVATE /dev/zero too since commit e59c9a971a9f
> ("mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous")),
> so compute the index from vm_pgoff as before. That brings back the
> pre-series behaviour for the hot path; MAP_PRIVATE file-backed VMAs still
> read the split field. Keep the assertion, the other way round.
> 
> Moving __vm_anon_pgoff_hi out of the line does not work without growing
> vm_area_struct. With e.g. CONFIG_ANON_VMA_NAME=y the struct is exactly
> 192 bytes, cachelines 0 and 1 have no hole, and the only 4-byte fields
> that could fill the slot next to vm_refcnt are vm_lock_seq and the lo/hi
> halves. vm_lock_seq is read before vm_refcnt is taken in vma_start_read(),
> so it would just move the miss.
> 

I trust Lorenzo knows more about his plans to use anon_pgoff than I do,
so I'll let him reply. A comment though:

> Measured on an AMD EPYC 9634 (84 cores, 12 CCDs, CONFIG_DEBUG_VM=n), on
> mainline after v7.4-rc1 (the base of this series). Threads share one
> anonymous VMA, each faults its own slice and zaps it with MADV_DONTNEED;
> faults/s, median of 3:
> 
>   CPUs                      base      patched
>   1 CCD,   7 threads        2.360M    2.369M   +0.4%
>   7 CCDs,  1 thread each    1.738M    1.982M  +14.0%

This is not particularly surprising. CCDs have their own L3 cache. I'm not
surprised the coherency penalty is much higher cross-CCD.

-- 
Pedro

Re: [PATCH 1/2] mm/rmap: use vm_pgoff for anonymous VMAs in linear_anon_page_index()

[Lorenzo Stoakes (ARM)]

On Tue, Nov 10, 2026 at 11:25:13AM +0000, Pedro Falcato wrote:
> I trust Lorenzo knows more about his plans to use anon_pgoff than I do,
> so I'll let him reply. A comment though:

Thanks.

I'm taking over the series rather than replying further, as is general practice
with this kind of submission.

Will post something in a little while.

>
> > Measured on an AMD EPYC 9634 (84 cores, 12 CCDs, CONFIG_DEBUG_VM=n), on
> > mainline after v7.4-rc1 (the base of this series). Threads share one
> > anonymous VMA, each faults its own slice and zaps it with MADV_DONTNEED;
> > faults/s, median of 3:
> >
> >   CPUs                      base      patched
> >   1 CCD,   7 threads        2.360M    2.369M   +0.4%
> >   7 CCDs,  1 thread each    1.738M    1.982M  +14.0%
>
> This is not particularly surprising. CCDs have their own L3 cache. I'm not
> surprised the coherency penalty is much higher cross-CCD.

Right. And a lot of these workloads seem to work backwards from the problem
'let's find the worst possible scenario for this, then microbenchmark it,
surprise it turns out it performs badly'.

Even the real-world usecase (no citation as to _actually_ encountering it) seems
to follow the same pattern.

I expect we'll continue to see more of this stuff over time, sigh.

In any case, the solution seems to be the same in general - don't do this check
in non-debug.

--
Cheers, Lorenzo

RE: [PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[Seunghyun An]

Thanks for taking this on.

I should have disclosed the LLM involvement up front, and I'm sorry
I didn't. I came across this while researching the performance of the
anonymous fault path and worked with Claude to investigate and quantify
it. Claude also drafted the code and changelogs.

My main aim was to report the findings to the community. In hindsight,
a plain report would have been a better way to raise it. I wanted to share
the measured effects, especially on EPYC given the higher cross-CCD
coherency penalty, and offer a possible way to address them.

I'd be glad to collaborate on measurements for your series and any
follow-up work in this area.

Thanks,
Seunghyun

Re: [PATCH 0/2] mm: avoid redundant reads of the vm_refcnt cacheline in anonymous faults

[Lorenzo Stoakes (ARM)]

On Tue, Nov 10, 2026 at 02:34:31PM +0000, Seunghyun An wrote:
> Thanks for taking this on.
>
> I should have disclosed the LLM involvement up front, and I'm sorry
> I didn't. I came across this while researching the performance of the
> anonymous fault path and worked with Claude to investigate and quantify
> it. Claude also drafted the code and changelogs.
>
> My main aim was to report the findings to the community. In hindsight,
> a plain report would have been a better way to raise it. I wanted to share
> the measured effects, especially on EPYC given the higher cross-CCD
> coherency penalty, and offer a possible way to address them.
>
> I'd be glad to collaborate on measurements for your series and any
> follow-up work in this area.

Thanks! I will cc- you on the patch, and if there is any remaining issue we can
chase that down.

And of course credit you on the patch as reporter! :)

--
Cheers, Lorenzo