[PATCH v2 2/2] mm/memory: reuse the whole exclusive large folio on a write fault

[Yuan-Hao Hsu <aa9736195201@gmail.com>]

With blocks of 16 PTEs a PTE-mapped 2M THP still takes 32 reuse faults
and a 1M folio 16.  Lift the bound to the page table: the walk then
covers the PTEs that map the folio in this page table, at most
PTRS_PER_PTE of them, and one fault does the work for the folio.

Its cost, measured as the time of the store that takes it, against
420 ns for a reuse fault today (x86-64, i7-12700KF, medians, ns):

                          reuse fault   fault that     COW fault that
                          (patched)     allocated it   copies 4K
      1M mTHP            5,400- 5,700   63,000-68,000          1,500
      2M THP, PTE-mapped  10,000-10,500         126,000          1,500

That is 14-20 ns per PTE, about 2 ns of it the scan.  Builds that
differ only by NOPs in front of the function take either 10,100 or
7,500 ns for the 2M folio, with a period of 32 bytes: the loop of
modify_prot_commit_ptes() changes speed with its address.

What it buys, 256 MiB after fork() and the child's exit, medians of 15
runs, two boots of each kernel:

                                   16 PTEs         whole folio
  one byte per page, seq
      1M mTHP                     5.7 /  5.8 ms    4.1 /  4.0 ms
      2M THP, PTE-mapped          5.7 /  5.8 ms    3.9 /  3.9 ms
  one byte per page, random order
      2M THP, PTE-mapped          7.4 /  7.5 ms    4.9 /  5.1 ms
  memset()
      2M THP, PTE-mapped         40.2 / 39.4 ms   38.1 / 37.2 ms
  8 threads, random order
      2M THP, PTE-mapped          0.9 /  1.0 ms    0.7 /  0.7 ms
  one store per 64K, 2M folios    3.3 /  3.3 ms    1.5 /  1.6 ms
  one store per 2M, 2M folios     0.1 /  0.1 ms    1.4 /  1.4 ms

The last row is the pattern where the whole-folio fault is pure cost:
511 pages made writable that nobody writes, 1.3 ms more per 256 MiB,
paid once per fork().  Anything that stores to more than one page per
64K comes out ahead.

Link: https://lore.kernel.org/all/36933711-ae0f-468c-93bd-d6a67d974c9d@redhat.com/
Assisted-by: LLM sparse
Signed-off-by: Yuan-Hao Hsu <aa9736195201@gmail.com>
---
 mm/memory.c | 21 +++++++--------------
 1 file changed, 7 insertions(+), 14 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 73e5691b3ed8..85c883d1e558 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4355,19 +4355,12 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	return true;
 }
 
-/*
- * The pages of the folio around the one that faulted are handled in aligned
- * blocks of this many PTEs: a 64K folio with 4K pages, and the size of a
- * contpte block on arm64.
- */
-#define WP_REUSE_NR_PTES	16
-
 /*
  * wp_can_reuse_anon_folio() found a large folio to be exclusive to this MM.
  * That holds for all of its pages and not only for the one that faulted: mark
- * the ones in the same block exclusive as well and map them writable, like
- * mprotect() would. Each of them would otherwise take a write fault of its own
- * that repeats the check on the very same folio.
+ * the ones that this page table maps exclusive as well and map them writable,
+ * like mprotect() would. Each of them would otherwise take a write fault of
+ * its own that repeats the check on the very same folio.
  *
  * The PTE that faulted is among them; wp_page_reuse() completes it.
  */
@@ -4378,17 +4371,17 @@ static void wp_reuse_large_anon_folio(struct vm_fault *vmf,
 	const unsigned long idx = folio_page_idx(folio, vmf->page);
 	struct vm_area_struct *vma = vmf->vma;
 	unsigned long addr = vmf->address;
-	unsigned long block = ALIGN_DOWN(addr, WP_REUSE_NR_PTES * PAGE_SIZE);
+	unsigned long pt_start = ALIGN_DOWN(addr, PMD_SIZE);
 	unsigned long nr_before, nr_after, end;
 	struct page *page;
 	unsigned int nr, i;
 	pte_t *ptep, pte;
 
-	/* Stay within the folio, the VMA and the block. */
-	nr_before = min3(idx, (addr - block) >> PAGE_SHIFT,
+	/* Stay within the folio, the VMA and the page table. */
+	nr_before = min3(idx, (addr - pt_start) >> PAGE_SHIFT,
 			 (addr - vma->vm_start) >> PAGE_SHIFT);
 	nr_after = min3(folio_nr_pages(folio) - idx,
-			(block + WP_REUSE_NR_PTES * PAGE_SIZE - addr) >> PAGE_SHIFT,
+			(pt_start + PMD_SIZE - addr) >> PAGE_SHIFT,
 			(vma->vm_end - addr) >> PAGE_SHIFT);
 	end = addr + (nr_after << PAGE_SHIFT);
 	addr -= nr_before << PAGE_SHIFT;
-- 
2.43.0