[PATCH] mm/memory: reuse the whole exclusive large folio on a write fault

[PATCH] mm/memory: reuse the whole exclusive large folio on a write fault

[Yuan-Hao Hsu]

fork() maps the anonymous pages of the parent read-only and clears
PageAnonExclusive on them.  Once the child has exec'ed or exited, a
write fault of the parent ends up in the reuse path of do_wp_page():
wp_can_reuse_anon_folio() finds that all references to the folio come
from mappings in this MM, the page is marked exclusive again and its PTE
is made writable.

For a large folio that check is about the folio, and what it finds holds
for every page of it.  Still only the page that faulted is marked
exclusive and only its PTE becomes writable, so each of the other pages
takes a write fault of its own, and each of those takes the large
mapcount lock to find out the same thing again: 16 faults for a 64K
folio, 512 for a 2M THP that is mapped by PTEs.  The same THP mapped by
a PMD is reused by one fault in do_huge_pmd_wp_page(), do_swap_page()
maps all PTEs of an exclusive large folio writable at once, and
numa_rebuild_large_mapping() upgrades all PTEs of the folio from one
hinting fault.

Commit 1da190f4d0a6 ("mm: Copy-on-Write (COW) reuse support for
PTE-mapped THP") left this for later because faulting around might
increase the COW latency.  Numbers for that are below.

When a large folio has been found exclusive, walk the part of it that
this page table maps inside the VMA.  PTEs that still map it read-only
are batched with folio_pte_batch_flags(), their pages are marked
exclusive, and where can_change_pte_writable() agrees the batch is made
writable with modify_prot_start_ptes()/modify_prot_commit_ptes(), the
way mprotect() does it.  That leaves NUMA hinting and uffd-wp PTEs
alone, keeps soft-dirty tracking exact, and on arm64 writes a contpte
block back as a whole where ptep_set_access_flags() on a single PTE has
to unfold it.  Pages that cannot be made writable are marked exclusive
all the same, so their own fault skips the folio check.  The PTE that
faulted is in one of the batches and is then completed by
wp_page_reuse() as before.  Small folios, PMD-mapped THPs, unsharing
faults and the copy path are not changed.

x86-64, i7-12700KF, 256 MiB of anonymous memory, fork(), the child
exits, then the parent stores to the memory.  Medians of 15 runs, two
boots of each kernel, taken alternately:

                          v7.3-rc3+           patched
  one byte per page, a clock_gettime() between the stores
    write faults
      4K pages            65,601              65,601
      64K mTHP            65,601               4,161
      1M mTHP             65,601                 321
      2M THP, PTE-mapped  65,601                 193
      2M THP, PMD-mapped     193                 193
    time (ms)
      4K pages            29.7 / 30.1         30.0 / 31.1
      64K mTHP            29.4 / 29.7          5.5 /  5.6
      1M mTHP             29.3 / 29.8          3.9 /  3.9
      2M THP, PTE-mapped  29.9 / 29.2          3.8 /  3.9
      2M THP, PMD-mapped   2.5 /  2.5          2.5 /  2.6
  memset() of all of it (ms)
      4K pages            57.2 / 56.5         56.9 / 57.1
      64K mTHP            56.5 / 56.3         39.0 / 38.7
      2M THP, PTE-mapped  58.6 / 56.5         36.8 / 37.7
      2M THP, PMD-mapped  37.5 / 35.9         35.9 / 36.5
  8 threads, one byte per page, random order (ms)
      64K mTHP             6.1 /  6.2          0.9 /  1.0
      2M THP, PTE-mapped   6.0 /  6.7          0.7 /  0.7

The latency of the one fault that now does the work for the folio,
measured as the time of the store that takes it, against 420 ns for a
reuse fault today (medians, ns):

                          reuse fault   fault that     COW fault that
                          (patched)     allocated it   copies 4K
      64K mTHP                   730          4,600          1,500
      1M mTHP                  5,500         63,000          1,500
      2M THP, PTE-mapped      10,100        126,000          1,500

That is 14 to 20 ns per PTE.  Builds that differ only by NOPs in front
of the new function take either 10,100 or 7,500 ns for the 2M folio,
with a period of 32 bytes: it is the loop of modify_prot_commit_ptes()
that changes speed with its address.

Capping the walk to the 16 PTEs around the fault instead was measured as
well: it takes 5.4 ms where the above takes 3.9 ms on 1M and 2M folios,
it is slower than today when only one page per 64K is written (3.1 ms
against 1.9 ms, the whole folio takes 1.5 ms), and it is only ahead when
no more than one page per 2M is ever written (0.1 ms against 1.3 ms for
the 256 MiB).

Redis 7.0.15 with 1.3 M keys of 512 bytes on 64K mTHP, BGSAVE and then
1,000,000 SETs: the faults of redis-server during the SETs go from
262,100 to 18,500, its CPU time from 1.48-1.53 s to 1.25-1.33 s, and
redis-benchmark reports 742,000 to 794,000 requests per second instead
of 652,000 to 658,000.  With THP off all three stay where they were.

arm64 was only run under QEMU, for the counters and with DEBUG_VM and
PAGE_TABLE_CHECK: the faults are the same as above, and with 64K folios
the first pass after fork() unfolds every contpte block today (512
contpte_convert() calls for 512 blocks, and nothing folds them again)
while none is unfolded with this patch.

What does not get faster on x86 are stores to pages that this CPU still
has a read-only TLB entry for.  The fault makes the PTEs writable but,
like mprotect(), does not flush, so such a page still takes a fault, a
spurious one that costs about the same as the reuse fault it replaces.
That happens to pages that were read since fork(): reading the 16 pages
of every 64K folio before storing to them takes 65,500 faults and 28 ms
before and after.  And it happens in a loop that does nothing but store
one byte to every page in ascending order, which is what the reuse-byte
mode of David's pte-mapped-folio-benchmarks does (120 ms before and
after for 1 GiB of 64K folios; 2M folios: 119 ms to 18 ms; the reuse
mode, a memset(), goes from 232 ms to 157 ms with 64K folios): while the
first store of a folio is faulting, the CPU has already run the next
stores speculatively and has filled the TLB with the read-only
translations of their pages.  Counting with kprobes, such a run enters
handle_mm_fault() 135,687 times and do_wp_page() 8,457 times; with an
LFENCE after every store the faults are 4,100 instead of 65,400 and the
loop takes 4 ms instead of 29 ms.  The PMD-mapped case, which this patch
does not touch, shows the same: 8,800 faults for 128 THPs, 133 with the
LFENCE.

With a flush_tlb_local() in the new function, as an experiment, the
ascending loop takes 4.1 ms instead of 28 ms on 64K folios, the
read-then-store loop 5.3 ms instead of 28 ms and the memset() 21 ms
instead of 38 ms, for a fault of 870 instead of 730 ns and 11% more time
for the pass with the clock_gettime().  Generic code has no way to ask
x86 for a flush that stays on this CPU, so that is left for later.

Assisted-by: LLM sparse
Signed-off-by: Yuan-Hao Hsu <aa9736195201@gmail.com>
---
 mm/memory.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 2 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 8b0c2c735d3d..85c883d1e558 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4355,6 +4355,64 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	return true;
 }
 
+/*
+ * wp_can_reuse_anon_folio() found a large folio to be exclusive to this MM.
+ * That holds for all of its pages and not only for the one that faulted: mark
+ * the ones that this page table maps exclusive as well and map them writable,
+ * like mprotect() would. Each of them would otherwise take a write fault of
+ * its own that repeats the check on the very same folio.
+ *
+ * The PTE that faulted is among them; wp_page_reuse() completes it.
+ */
+static void wp_reuse_large_anon_folio(struct vm_fault *vmf,
+		struct folio *folio)
+{
+	const fpb_t flags = FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY;
+	const unsigned long idx = folio_page_idx(folio, vmf->page);
+	struct vm_area_struct *vma = vmf->vma;
+	unsigned long addr = vmf->address;
+	unsigned long pt_start = ALIGN_DOWN(addr, PMD_SIZE);
+	unsigned long nr_before, nr_after, end;
+	struct page *page;
+	unsigned int nr, i;
+	pte_t *ptep, pte;
+
+	/* Stay within the folio, the VMA and the page table. */
+	nr_before = min3(idx, (addr - pt_start) >> PAGE_SHIFT,
+			 (addr - vma->vm_start) >> PAGE_SHIFT);
+	nr_after = min3(folio_nr_pages(folio) - idx,
+			(pt_start + PMD_SIZE - addr) >> PAGE_SHIFT,
+			(vma->vm_end - addr) >> PAGE_SHIFT);
+	end = addr + (nr_after << PAGE_SHIFT);
+	addr -= nr_before << PAGE_SHIFT;
+	ptep = vmf->pte - nr_before;
+	page = vmf->page - nr_before;
+
+	for (; addr != end; addr += nr * PAGE_SIZE, ptep += nr, page += nr) {
+		pte = ptep_get(ptep);
+		nr = 1;
+
+		/* Unmapped or replaced since, or writable already. */
+		if (!pte_present(pte) || pte_pfn(pte) != page_to_pfn(page) ||
+		    pte_write(pte))
+			continue;
+
+		nr = folio_pte_batch_flags(folio, NULL, ptep, &pte,
+					   (end - addr) >> PAGE_SHIFT, flags);
+		for (i = 0; i < nr; i++)
+			if (!PageAnonExclusive(page + i))
+				SetPageAnonExclusive(page + i);
+
+		/* The PTEs of a batch agree on everything this looks at. */
+		if (!can_change_pte_writable(vma, addr, pte))
+			continue;
+
+		pte = modify_prot_start_ptes(vma, addr, ptep, nr);
+		modify_prot_commit_ptes(vma, addr, ptep, pte,
+					pte_mkwrite(pte, vma), nr);
+	}
+}
+
 /*
  * This routine handles present pages, when
  * * users try to write to a shared page (FAULT_FLAG_WRITE)
@@ -4449,8 +4507,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 	 */
 	if (folio && folio_test_anon(folio) &&
 	    (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) {
-		if (!PageAnonExclusive(vmf->page))
-			SetPageAnonExclusive(vmf->page);
+		if (!PageAnonExclusive(vmf->page)) {
+			if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
+			    folio_test_large(folio) && likely(!unshare) &&
+			    likely(vma->vm_flags & VM_WRITE))
+				wp_reuse_large_anon_folio(vmf, folio);
+			else
+				SetPageAnonExclusive(vmf->page);
+		}
 		if (unlikely(unshare)) {
 			pte_unmap_unlock(vmf->pte, vmf->ptl);
 			return 0;

base-commit: 238650ef6c7c7cca08e032527329424c9fbd70e5
-- 
2.43.0

Re: [PATCH] mm/memory: reuse the whole exclusive large folio on a write fault

[David Hildenbrand (Arm)]

On 9/18/26 08:42, Yuan-Hao Hsu wrote:
> fork() maps the anonymous pages of the parent read-only and clears
> PageAnonExclusive on them.  Once the child has exec'ed or exited, a
> write fault of the parent ends up in the reuse path of do_wp_page():
> wp_can_reuse_anon_folio() finds that all references to the folio come
> from mappings in this MM, the page is marked exclusive again and its PTE
> is made writable.
> 
> For a large folio that check is about the folio, and what it finds holds
> for every page of it.  Still only the page that faulted is marked
> exclusive and only its PTE becomes writable, so each of the other pages
> takes a write fault of its own, and each of those takes the large
> mapcount lock to find out the same thing again: 16 faults for a 64K
> folio, 512 for a 2M THP that is mapped by PTEs.  The same THP mapped by
> a PMD is reused by one fault in do_huge_pmd_wp_page(), do_swap_page()
> maps all PTEs of an exclusive large folio writable at once, and
> numa_rebuild_large_mapping() upgrades all PTEs of the folio from one
> hinting fault.
> 
> Commit 1da190f4d0a6 ("mm: Copy-on-Write (COW) reuse support for
> PTE-mapped THP") left this for later because faulting around might
> increase the COW latency.  Numbers for that are below.
> 
> When a large folio has been found exclusive, walk the part of it that
> this page table maps inside the VMA.  PTEs that still map it read-only
> are batched with folio_pte_batch_flags(), their pages are marked
> exclusive, and where can_change_pte_writable() agrees the batch is made
> writable with modify_prot_start_ptes()/modify_prot_commit_ptes(), the
> way mprotect() does it.  That leaves NUMA hinting and uffd-wp PTEs
> alone, keeps soft-dirty tracking exact, and on arm64 writes a contpte
> block back as a whole where ptep_set_access_flags() on a single PTE has
> to unfold it.  Pages that cannot be made writable are marked exclusive
> all the same, so their own fault skips the folio check.  The PTE that
> faulted is in one of the batches and is then completed by
> wp_page_reuse() as before.  Small folios, PMD-mapped THPs, unsharing
> faults and the copy path are not changed.
> 
> x86-64, i7-12700KF, 256 MiB of anonymous memory, fork(), the child
> exits, then the parent stores to the memory.  Medians of 15 runs, two
> boots of each kernel, taken alternately:
> 
>                           v7.3-rc3+           patched
>   one byte per page, a clock_gettime() between the stores
>     write faults
>       4K pages            65,601              65,601
>       64K mTHP            65,601               4,161
>       1M mTHP             65,601                 321
>       2M THP, PTE-mapped  65,601                 193
>       2M THP, PMD-mapped     193                 193
>     time (ms)
>       4K pages            29.7 / 30.1         30.0 / 31.1
>       64K mTHP            29.4 / 29.7          5.5 /  5.6
>       1M mTHP             29.3 / 29.8          3.9 /  3.9
>       2M THP, PTE-mapped  29.9 / 29.2          3.8 /  3.9
>       2M THP, PMD-mapped   2.5 /  2.5          2.5 /  2.6
>   memset() of all of it (ms)
>       4K pages            57.2 / 56.5         56.9 / 57.1
>       64K mTHP            56.5 / 56.3         39.0 / 38.7
>       2M THP, PTE-mapped  58.6 / 56.5         36.8 / 37.7
>       2M THP, PMD-mapped  37.5 / 35.9         35.9 / 36.5
>   8 threads, one byte per page, random order (ms)
>       64K mTHP             6.1 /  6.2          0.9 /  1.0
>       2M THP, PTE-mapped   6.0 /  6.7          0.7 /  0.7
> 
> The latency of the one fault that now does the work for the folio,
> measured as the time of the store that takes it, against 420 ns for a
> reuse fault today (medians, ns):
> 
>                           reuse fault   fault that     COW fault that
>                           (patched)     allocated it   copies 4K
>       64K mTHP                   730          4,600          1,500
>       1M mTHP                  5,500         63,000          1,500
>       2M THP, PTE-mapped      10,100        126,000          1,500
> 
> That is 14 to 20 ns per PTE.  Builds that differ only by NOPs in front
> of the new function take either 10,100 or 7,500 ns for the 2M folio,
> with a period of 32 bytes: it is the loop of modify_prot_commit_ptes()
> that changes speed with its address.
> 
> Capping the walk to the 16 PTEs around the fault instead was measured as
> well: it takes 5.4 ms where the above takes 3.9 ms on 1M and 2M folios,
> it is slower than today when only one page per 64K is written (3.1 ms
> against 1.9 ms, the whole folio takes 1.5 ms), and it is only ahead when
> no more than one page per 2M is ever written (0.1 ms against 1.3 ms for
> the 256 MiB).
> 
> Redis 7.0.15 with 1.3 M keys of 512 bytes on 64K mTHP, BGSAVE and then
> 1,000,000 SETs: the faults of redis-server during the SETs go from
> 262,100 to 18,500, its CPU time from 1.48-1.53 s to 1.25-1.33 s, and
> redis-benchmark reports 742,000 to 794,000 requests per second instead
> of 652,000 to 658,000.  With THP off all three stay where they were.
> 
> arm64 was only run under QEMU, for the counters and with DEBUG_VM and
> PAGE_TABLE_CHECK: the faults are the same as above, and with 64K folios
> the first pass after fork() unfolds every contpte block today (512
> contpte_convert() calls for 512 blocks, and nothing folds them again)
> while none is unfolded with this patch.
> 
> What does not get faster on x86 are stores to pages that this CPU still
> has a read-only TLB entry for.  The fault makes the PTEs writable but,
> like mprotect(), does not flush, so such a page still takes a fault, a
> spurious one that costs about the same as the reuse fault it replaces.
> That happens to pages that were read since fork(): reading the 16 pages
> of every 64K folio before storing to them takes 65,500 faults and 28 ms
> before and after.  And it happens in a loop that does nothing but store
> one byte to every page in ascending order, which is what the reuse-byte
> mode of David's pte-mapped-folio-benchmarks does (120 ms before and
> after for 1 GiB of 64K folios; 2M folios: 119 ms to 18 ms; the reuse
> mode, a memset(), goes from 232 ms to 157 ms with 64K folios): while the
> first store of a folio is faulting, the CPU has already run the next
> stores speculatively and has filled the TLB with the read-only
> translations of their pages.  Counting with kprobes, such a run enters
> handle_mm_fault() 135,687 times and do_wp_page() 8,457 times; with an
> LFENCE after every store the faults are 4,100 instead of 65,400 and the
> loop takes 4 ms instead of 29 ms.  The PMD-mapped case, which this patch
> does not touch, shows the same: 8,800 faults for 128 THPs, 133 with the
> LFENCE.
> 
> With a flush_tlb_local() in the new function, as an experiment, the
> ascending loop takes 4.1 ms instead of 28 ms on 64K folios, the
> read-then-store loop 5.3 ms instead of 28 ms and the memset() 21 ms
> instead of 38 ms, for a fault of 870 instead of 730 ns and 11% more time
> for the pass with the clock_gettime().  Generic code has no way to ask
> x86 for a flush that stays on this CPU, so that is left for later.
> 
> Assisted-by: LLM sparse


My review backlog is large enough for me to just go through this wall of text.

There were previous discussions on this, in particular around how much we should
actually try operating around the target PTE.

How did you use the LLM for coming up with this patch + description?

-- 
Cheers,

David

Re: [PATCH] mm/memory: reuse the whole exclusive large folio on a write fault

[Lorenzo Stoakes (ARM)]

This is the friendly patch-bot of Lorenzo Stoakes.

You have sent him a patch/series that has triggered this response.

He used to manually respond to these common problems, but in order to save
his sanity (he kept writing the same thing over and over, yet to different
people), I was created.

Hopefully you will not take offence and will fix the problem in your patch
and resubmit it so that it can be accepted into the Linux kernel tree.

When sending emails to mm:

For one of several possible reasons this mail has triggered an AI
detector script.

Thank you for using the Assisted-by tag to acknowledge this.

However, please do note the policy at
https://docs.kernel.org/process/generated-content.html most notably:

        If tools permit you to generate a contribution automatically, expect
        additional scrutiny in proportion to how much of it was generated.

        As with the output of any tooling, the result may be incorrect or
        inappropriate. You are expected to understand and to be able to
        defend everything you submit. If you are unable to do so, then do
        not submit the resulting changes.

        If you do so anyway, maintainers are entitled to reject your series
        without detailed review.

This change is significant, and you are a relative newcomer to mm. As a
result, I must ask that you provide some details as to how you solved this
problem and your overall understanding of these changes and mm as a whole.

In general, if you are a newcomer to mm, we expect you to do smaller work
before moving on to larger changes, so you build understanding of both the
technical aspects of mm and how we do things.

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Lorenzo will reply once he has dug out from the pending patches received
from other developers.

thanks,

Lorenzo's patch email bot

[ Idea shamelessly stolen from greg-kh ]

--
Cheers, Lorenzo

Re: [PATCH] mm/memory: reuse the whole exclusive large folio on a write fault

[Yuan-Hao Hsu]

On Fri, 18 Sep 2026 14:14:11 +0200, David Hildenbrand (Arm) wrote:

> There were previous discussions on this, in particular around how much
> we should actually try operating around the target PTE.

Yes, Barry's RFC from 2024 and your replies to it, where you were fine
with contpte granularity and not convinced about anything bigger. That
is why the description has the numbers for both: capped to 16 PTEs, the
1M and 2M cases take 5.4 ms instead of 3.9 ms, and the only pattern where
the cap wins is one store per 2M. If you would rather start with the
contpte-sized version, that is a two-line change and I can send that
version instead.

> How did you use the LLM for coming up with this patch + description?

I did the initial investigation and made the design decisions myself.
Once I had a clear idea of the approach and the overall structure, I used
an LLM to help with parts of the implementation and the test
programs. I also used it to help with drafting the commit message.

Before sending the patch, I went through it line by line and checked the
test results and measurements to make sure they matched what the code was
actually doing and that there were no obvious issues.

Lorenzo, for your bot's question: the reuse decision in
wp_can_reuse_anon_folio() is per folio. The patch applies it to the other
PTEs of the folio within the same VMA and page table using the mprotect
helpers (can_change_pte_writable() and modify_prot_*_ptes()), so
soft-dirty, uffd-wp and NUMA hinting PTEs are handled the way mprotect()
handles them, and the PTE that faulted is finished by wp_page_reuse() as
before.

The part I spent most of the time on was the measurements, including the
cases where it does not help.

Yuan-Hao Hsu

Re: [PATCH] mm/memory: reuse the whole exclusive large folio on a write fault

[Barry Song]

On Sat, Sep 19, 2026 at 2:29 AM Yuan-Hao Hsu <aa9736195201@gmail.com> wrote:
>
> On Fri, 18 Sep 2026 14:14:11 +0200, David Hildenbrand (Arm) wrote:
>
> > There were previous discussions on this, in particular around how much
> > we should actually try operating around the target PTE.
>
> Yes, Barry's RFC from 2024 and your replies to it, where you were fine
> with contpte granularity and not convinced about anything bigger. That
> is why the description has the numbers for both: capped to 16 PTEs, the
> 1M and 2M cases take 5.4 ms instead of 3.9 ms, and the only pattern where

Hi Yuan-Hao,

I'm fine with your follow-up work, but I'd appreciate it if you could
mention the previous work and include the link in your changelog, along
with an explanation of how you address David's concerns.

> the cap wins is one store per 2M. If you would rather start with the
> contpte-sized version, that is a two-line change and I can send that
> version instead.

I think a major concern is that we may spend too much time scanning
PTEs to determine whether we can batch them. If we don't support
CONT-PTE, we may simply scan fewer PTEs; if we do support CONT-PTE, we
may end up scanning more PTEs. So we need to show that the scanning
cost is controlled and explain how it is bounded.

>
> > How did you use the LLM for coming up with this patch + description?
>
> I did the initial investigation and made the design decisions myself.
> Once I had a clear idea of the approach and the overall structure, I used
> an LLM to help with parts of the implementation and the test
> programs. I also used it to help with drafting the commit message.
>
> Before sending the patch, I went through it line by line and checked the
> test results and measurements to make sure they matched what the code was
> actually doing and that there were no obvious issues.

I also personally feel that both the changelog and the code are too
long to read. Could we somehow split the changes into smaller pieces
and describe how each of those concerns is addressed at a finer
granularity?

>
> Lorenzo, for your bot's question: the reuse decision in
> wp_can_reuse_anon_folio() is per folio. The patch applies it to the other
> PTEs of the folio within the same VMA and page table using the mprotect
> helpers (can_change_pte_writable() and modify_prot_*_ptes()), so
> soft-dirty, uffd-wp and NUMA hinting PTEs are handled the way mprotect()
> handles them, and the PTE that faulted is finished by wp_page_reuse() as
> before.
>
> The part I spent most of the time on was the measurements, including the
> cases where it does not help.
>

Best Regards
Barry