[PATCH 0/4] mm/ksm: use per-VMA locking for find_mergeable_vma()

[PATCH 0/4] mm/ksm: use per-VMA locking for find_mergeable_vma()

[xu.xin16]

From: Xu Xin (ZTE) <xu.xin@linux.dev>

KSM scans VM_MERGEABLE VMAs and currently protects each scan with
mmap_read_lock(). Per-VMA locking allows KSM to read-lock only the VMA
it is actually interested in, so that unrelated mmap()/munmap() activity
in the same mm no longer blocks ksmd.

This series is organized as follows:

  Patch 1 removes an unused 'vma' member from struct folio_walk.  It has
  never been used since its introduction and is pure cleanup.

  Patch 2 adds a 'walk_lock' member to struct folio_walk and extends
  folio_walk_start() to assert the required locking mode.  Existing
  callers are converted to pass PGWALK_RDLOCK, so there is no functional
  change.  This prepares folio_walk_start() for callers that hold a
  per-VMA read lock instead of mmap_read_lock(), which is needed by the
  Patch 4. No functional change.

  Patch 3 tranforms the boolean 'lock_vma' into the enum 'page_walk_lock'
  without any behavior changed, which is prepared for the Patch 4 to use
  per-VMA locking. No functional change.

  Patch 4 introduces find_mergeable_vma_locked(), which uses the
  universal per-VMA locking helper vma_start_read_unlocked() to look up
  and read-lock a VM_MERGEABLE VMA without taking mmap_read_lock().  All
  KSM call sites that previously used find_mergeable_vma() under
  mmap_read_lock() are converted to the new helper, and the locking in
  get_mergeable_page() is switched to PGWALK_VMA_RDLOCK_VERIFY so that
  folio_walk_start() can verify the per-VMA lock is held.

A microbenchmark was run to measure the time KSM takes to merge a
victim region under mmap_lock contention. Under interference from 4 churner
threads, the merge time of the per-VMA KSM-optimized kernel is
significantly reduced by 50%.

Xu Xin (4):
  mm/pagewalk: delete the unused member
  mm: make folio_walk_start()'s locking asserts scalable
  mm/ksm: make break_ksm() more scalable
  mm/ksm: add find_mergeable_vma_locked() to use per-VMA locking

 arch/s390/mm/fault.c     |  4 +-
 include/linux/pagewalk.h |  2 +-
 kernel/events/uprobes.c  |  4 +-
 mm/huge_memory.c         |  4 +-
 mm/ksm.c                 | 85 ++++++++++++++++++++++++----------------
 mm/migrate.c             |  8 +++-
 mm/pagewalk.c            | 10 ++++-
 mm/rmap.c                |  4 +-
 8 files changed, 79 insertions(+), 42 deletions(-)

-- 
2.25.1

[PATCH 1/4] mm/pagewalk: delete the unused member

[xu.xin16]

From: Xu Xin (ZTE) <xu.xin@linux.dev>

The member vma has never been used, we should delete it

Fixes: aa39ca6940f1a ("mm/pagewalk: introduce folio_walk_start() + folio_walk_end()")
Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>
---
 include/linux/pagewalk.h | 1 -
 1 file changed, 1 deletion(-)

diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h
index b41d7265c01b..1c397be0d092 100644
--- a/include/linux/pagewalk.h
+++ b/include/linux/pagewalk.h
@@ -183,7 +183,6 @@ struct folio_walk {
 		pmd_t pmd;
 	};
 	/* private */
-	struct vm_area_struct *vma;
 	spinlock_t *ptl;
 };

-- 
2.25.1

[PATCH 2/4] mm: make folio_walk_start()'s locking asserts scalable

[xu.xin16]

From: Xu Xin (ZTE) <xu.xin@linux.dev>

Add an additional member 'walk_lock' to folio_walk to indicate
locking requirements for the walk. Similar to commit 49b0638502da0
("mm: enable page walking API to lock vmas during the walk").
But no change is made on any existing locking behavior,
all existing folio_walk_start() callers are currently still
under mmap_read_lock() protection.

This change is prepared for the latter patch to enable VMA locking
asserts. folio_walk_start now operate under write-locked mmap_lock.
With introduction of vma locks at the next patch, the vmas have to
be locked as well during such walks to prevent concurrent page
faults in these areas.

No functional change intended.

Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>
---
 arch/s390/mm/fault.c     |  4 +++-
 include/linux/pagewalk.h |  1 +
 kernel/events/uprobes.c  |  4 +++-
 mm/huge_memory.c         |  4 +++-
 mm/ksm.c                 |  4 +++-
 mm/migrate.c             |  8 ++++++--
 mm/pagewalk.c            | 10 +++++++++-
 mm/rmap.c                |  4 +++-
 8 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index 46d828926009..968f128a5232 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -412,7 +412,9 @@ __context_unsafe(/* folio_walk_end() not instrumented */)
 	unsigned long addr = get_fault_address(regs);
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct *vma;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct folio *folio;
 	int rc;

diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h
index 1c397be0d092..455428eaf1fc 100644
--- a/include/linux/pagewalk.h
+++ b/include/linux/pagewalk.h
@@ -184,6 +184,7 @@ struct folio_walk {
 	};
 	/* private */
 	spinlock_t *ptl;
+	enum page_walk_lock walk_lock;
 };

 struct folio *folio_walk_start(struct folio_walk *fw,
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 7709ea882477..054fdcb52136 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -507,7 +507,9 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
 	int ret, ref_ctr_updated = 0;
 	unsigned int gup_flags = FOLL_FORCE;
 	struct mmu_notifier_range range;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct folio *folio;
 	struct page *page;

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index dd66c6ad5af1..b714677e2f20 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4795,7 +4795,9 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start,
 	 */
 	for (addr = vaddr_start; addr < vaddr_end; addr += PAGE_SIZE) {
 		struct vm_area_struct *vma = vma_lookup(mm, addr);
-		struct folio_walk fw;
+		struct folio_walk fw = {
+			.walk_lock  = PGWALK_RDLOCK,
+		};
 		struct folio *folio;
 		struct address_space *mapping;
 		unsigned int target_order = new_order;
diff --git a/mm/ksm.c b/mm/ksm.c
index 624f37975e12..8df66b4e5de0 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -817,8 +817,10 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 	unsigned long addr = rmap_item->address;
 	struct vm_area_struct *vma;
 	struct page *page = NULL;
-	struct folio_walk fw;
 	struct folio *folio;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};

 	mmap_read_lock(mm);
 	vma = find_mergeable_vma(mm, addr);
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..fa638adfb0de 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2302,7 +2302,9 @@ static int add_folio_for_migration(struct mm_struct *mm, const void __user *p,
 		int node, struct list_head *pagelist, bool migrate_all)
 {
 	struct vm_area_struct *vma;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct folio *folio;
 	unsigned long addr;
 	int err = -EFAULT;
@@ -2464,7 +2466,9 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
 	for (i = 0; i < nr_pages; i++) {
 		unsigned long addr = (unsigned long)(*pages);
 		struct vm_area_struct *vma;
-		struct folio_walk fw;
+		struct folio_walk fw = {
+			.walk_lock  = PGWALK_RDLOCK,
+		};
 		struct folio *folio;
 		int err = -EFAULT;

diff --git a/mm/pagewalk.c b/mm/pagewalk.c
index 7411702a37f5..8eb29fba20ad 100644
--- a/mm/pagewalk.c
+++ b/mm/pagewalk.c
@@ -910,7 +910,15 @@ struct folio *folio_walk_start(struct folio_walk *fw,
 	pgd_t *pgdp;
 	p4d_t *p4dp;

-	mmap_assert_locked(vma->vm_mm);
+	/*
+	 * Other locking modes except for mmap or vma read locking are not
+	 * expected.
+	 */
+	if (fw->walk_lock != PGWALK_RDLOCK && fw->walk_lock != PGWALK_VMA_RDLOCK_VERIFY)
+		WARN_ONCE(1, "walk_lock is not expected!\n");
+	process_mm_walk_lock(vma->vm_mm, fw->walk_lock);
+	process_vma_walk_lock(vma, fw->walk_lock);
+
 	vma_pgtable_walk_begin(vma);

 	if (WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end))
diff --git a/mm/rmap.c b/mm/rmap.c
index 5fefe5b060b1..fd8e1c9e4424 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2871,7 +2871,9 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,
 	struct mmu_notifier_range range;
 	struct folio *folio, *fw_folio;
 	struct vm_area_struct *vma;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct page *page;
 	swp_entry_t entry;
 	pte_t swp_pte;
-- 
2.25.1

[PATCH 3/4] mm/ksm: make break_ksm() more scalable

[xu.xin16]

From: Xu Xin (ZTE) <xu.xin@linux.dev>

Currently the last argument 'walk_lock' of break_ksm() is used to
indicate whether the page_walk is protected by mmap_read_lock or
mmap_write_lock. If 'walk_lock' is true, we suppose its context to
be under mmap_write_lock() protection, then mark it PGWALK_WRLOCK and
make its vma be write-locked during the walk; If 'walk_lock' is
false, we suppose its context to be mmap_read_lock(), then mark it
PGWALK_RDLOCK.

This change is prepared for the latter patch to enable VMA
read-locking where break_ksm() might be under the third new proctecion
way: VMA read-locking, so we have to replace the boolean variable to
the enum 'page_walk_lock', but without any function changed.

No functional change intended.

Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>
---
 mm/ksm.c | 21 ++++++++-------------
 1 file changed, 8 insertions(+), 13 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index 8df66b4e5de0..dda105681d7f 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -660,16 +660,11 @@ static int break_ksm_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned long en
 	return found;
 }

-static const struct mm_walk_ops break_ksm_ops = {
+static struct mm_walk_ops break_ksm_ops = {
 	.pmd_entry = break_ksm_pmd_entry,
 	.walk_lock = PGWALK_RDLOCK,
 };

-static const struct mm_walk_ops break_ksm_lock_vma_ops = {
-	.pmd_entry = break_ksm_pmd_entry,
-	.walk_lock = PGWALK_WRLOCK,
-};
-
 /*
  * Though it's very tempting to unmerge rmap_items from stable tree rather
  * than check every pte of a given vma, the locking doesn't quite work for
@@ -696,11 +691,11 @@ static const struct mm_walk_ops break_ksm_lock_vma_ops = {
  * protection keys here anyway.
  */
 static int break_ksm(struct vm_area_struct *vma, unsigned long addr,
-		unsigned long end, bool lock_vma)
+		unsigned long end, enum page_walk_lock walk_lock)
 {
 	vm_fault_t ret = 0;
-	const struct mm_walk_ops *ops = lock_vma ?
-				&break_ksm_lock_vma_ops : &break_ksm_ops;
+	struct mm_walk_ops *ops = &break_ksm_ops;
+	ops->walk_lock = walk_lock;

 	do {
 		int ksm_page;
@@ -807,7 +802,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item)
 	mmap_read_lock(mm);
 	vma = find_mergeable_vma(mm, addr);
 	if (vma)
-		break_ksm(vma, addr, addr + PAGE_SIZE, false);
+		break_ksm(vma, addr, addr + PAGE_SIZE, PGWALK_RDLOCK);
 	mmap_read_unlock(mm);
 }

@@ -1245,7 +1240,7 @@ static int unmerge_and_remove_all_rmap_items(void)
 		for_each_vma(vmi, vma) {
 			if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
 				continue;
-			err = break_ksm(vma, vma->vm_start, vma->vm_end, false);
+			err = break_ksm(vma, vma->vm_start, vma->vm_end, PGWALK_RDLOCK);
 			if (err)
 				goto error;
 		}
@@ -2885,7 +2880,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma)
 		return 0;

 	if (vma->anon_vma) {
-		err = break_ksm(vma, vma->vm_start, vma->vm_end, true);
+		err = break_ksm(vma, vma->vm_start, vma->vm_end, PGWALK_WRLOCK);
 		if (err)
 			return err;
 	}
@@ -3037,7 +3032,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
 			return 0;		/* just ignore the advice */

 		if (vma->anon_vma) {
-			err = break_ksm(vma, start, end, true);
+			err = break_ksm(vma, start, end, PGWALK_WRLOCK);
 			if (err)
 				return err;
 		}
-- 
2.25.1

[PATCH 4/4] mm/ksm: add find_mergeable_vma_locked() to use per-VMA locking

[xu.xin16]

From: Xu Xin (ZTE) <xu.xin@linux.dev>

Purpose
=======
Let's add find_mergeable_vma_locked(), which is similar to find_tcp_vma(),
using the universal per-VMA locking helper, so that we can avoid
mmap_read_lock() to reduce contention.

To be used in KSM code to replace find_mergeable_vma() with
mmap_read_lock(), the helper find_mergeable_vma_locked() uses the
universal per-VMA locking allowing us to lock a struct vm_area_struct
without taking the process-wide mmap lock in read mode.

Performance
===========
As a test, I construct a testcase which follows the approach: Create
one victim and several churner threads sharing one mm_struct; The
victim registers a 32 MiB anonymous VM_MERGEABLE region containing
8192 pages totally:churners hammer mmap_lock via mmap/munmap
repeatedly; ksmd merges the victim's pages; Compare latency baseline
VS this per-VMA patch.

               Before patched         After Patched
0 churner:     1.627 seconds          1.426 seconds
4 churners:    72.45 seconds          36.61 seconds

In conclusion, when no mmap_lock contention (0 churner), there is little
difference between the baseline kernel and the per-VMA optimized kernel;
But under interference from 4 churner threads, the merge time of the
per-VMA KSM-optimized kernel is significantly reduced by 50%.

Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>
---
 mm/ksm.c | 64 +++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 42 insertions(+), 22 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index dda105681d7f..1d85769ec7db 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -765,15 +765,36 @@ static bool vma_ksm_compatible(struct vm_area_struct *vma)
 	return ksm_compatible(vma->vm_file, vma->flags);
 }

-static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,
-		unsigned long addr)
+/**
+ * find_mergeable_vma_locked() - Find the VMA covering 'address' which is
+ * VM_MERGEABLE and read-lock it by per-VMA locks. Please use vma_end_read()
+ * to unlock vma after finishing reading the VMA (non-NULL).
+ *
+ * Return: If a VMA exists which spans @address, return that VMA, read-locked.
+ * If no VMA is mapped there or, very unlikely, a reference count overflow
+ * occurred, return NULL, and no read-locked.
+ *
+ * IMPORTANT: If a VMA exists but is not VM_MERGEABLE or has no anon_vma,
+ * this function releases the per-VMA read lock before returning NULL.
+ * Callers must NOT call vma_end_read() on a NULL return value.
+ */
+static struct vm_area_struct *find_mergeable_vma_locked(struct mm_struct *mm,
+		unsigned long address)
 {
 	struct vm_area_struct *vma;
+
 	if (ksm_test_exit(mm))
 		return NULL;
-	vma = vma_lookup(mm, addr);
-	if (!vma || !(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
+
+	vma = vma_start_read_unlocked(mm, address);
+	if (!vma)
+		return NULL;
+
+	if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) {
+		vma_end_read(vma);
 		return NULL;
+	}
+
 	return vma;
 }

@@ -799,11 +820,12 @@ static void break_cow(struct ksm_rmap_item *rmap_item)
 	 */
 	rmap_item->linear_page_index = 0;

-	mmap_read_lock(mm);
-	vma = find_mergeable_vma(mm, addr);
-	if (vma)
-		break_ksm(vma, addr, addr + PAGE_SIZE, PGWALK_RDLOCK);
-	mmap_read_unlock(mm);
+	vma = find_mergeable_vma_locked(mm, addr);
+	if (!vma)
+		return;
+
+	break_ksm(vma, addr, addr + PAGE_SIZE, PGWALK_VMA_RDLOCK_VERIFY);
+	vma_end_read(vma);
 }

 static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
@@ -814,13 +836,12 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 	struct page *page = NULL;
 	struct folio *folio;
 	struct folio_walk fw = {
-		.walk_lock  = PGWALK_RDLOCK,
+		.walk_lock  = PGWALK_VMA_RDLOCK_VERIFY,
 	};

-	mmap_read_lock(mm);
-	vma = find_mergeable_vma(mm, addr);
+	vma = find_mergeable_vma_locked(mm, addr);
 	if (!vma)
-		goto out;
+		return NULL;

 	folio = folio_walk_start(&fw, vma, addr, 0);
 	if (folio) {
@@ -831,12 +852,12 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 		}
 		folio_walk_end(&fw, vma);
 	}
-out:
+
 	if (page) {
 		flush_anon_page(vma, page, addr);
 		flush_dcache_page(page);
 	}
-	mmap_read_unlock(mm);
+	vma_end_read(vma);
 	return page;
 }

@@ -1568,14 +1589,14 @@ static int try_to_merge_with_zero_page(struct ksm_rmap_item *rmap_item,
 	if (ksm_use_zero_pages && (rmap_item->oldchecksum == zero_checksum)) {
 		struct vm_area_struct *vma;

-		mmap_read_lock(mm);
-		vma = find_mergeable_vma(mm, rmap_item->address);
+		vma = find_mergeable_vma_locked(mm, rmap_item->address);
 		if (vma) {
 			err = try_to_merge_one_page(vma, page,
 					ZERO_PAGE(rmap_item->address));
 			trace_ksm_merge_one_page(
 				page_to_pfn(ZERO_PAGE(rmap_item->address)),
 				rmap_item, mm, err);
+			vma_end_read(vma);
 		} else {
 			/*
 			 * If the vma is out of date, we do not need to
@@ -1583,7 +1604,6 @@ static int try_to_merge_with_zero_page(struct ksm_rmap_item *rmap_item,
 			 */
 			err = 0;
 		}
-		mmap_read_unlock(mm);
 	}

 	return err;
@@ -1602,10 +1622,9 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item,
 	struct vm_area_struct *vma;
 	int err = -EFAULT;

-	mmap_read_lock(mm);
-	vma = find_mergeable_vma(mm, rmap_item->address);
+	vma = find_mergeable_vma_locked(mm, rmap_item->address);
 	if (!vma)
-		goto out;
+		goto out_trace;

 	err = try_to_merge_one_page(vma, page, kpage);
 	if (err)
@@ -1625,7 +1644,8 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item,
 	rmap_item->linear_page_index = linear_anon_page_index(vma, rmap_item->address);
 	get_anon_vma(vma->anon_vma);
 out:
-	mmap_read_unlock(mm);
+	vma_end_read(vma);
+out_trace:
 	trace_ksm_merge_with_ksm_page(kpage, page_to_pfn(kpage ? kpage : page),
 				rmap_item, mm, err);
 	return err;
-- 
2.25.1

Test Case Code Re: [PATCH 4/4] mm/ksm: add find_mergeable_vma_locked() to use per-VMA locking

[xu.xin16]

/*
 * ksm_merge_latency.c - Measure KSM victim merge latency under
 *                       mmap_lock contention from churner threads
 *                       in the SAME process.
 *
 * Build:  gcc -O2 -pthread -o ksm_merge_latency ksm_merge_latency.c
 * Run:    sudo ./ksm_merge_latency [nr_churners]   (default: 4)
 *
 * The victim region and the churner threads share one mm_struct, hence
 * one mmap_lock.  ksmd must scan this mm to merge the victim; on a
 * baseline kernel that requires mmap_read_lock(mm), which contends with
 * the churners' mmap()/munmap() write-side mmap_lock.  On a kernel with
 * per-VMA locking in mm/ksm.c, ksmd only takes a per-VMA read lock on
 * the victim VMA, which the churners never touch, so the contention
 * disappears.
 */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <time.h>
#include <pthread.h>
#include <sys/mman.h>

#define PAGE_SIZE      4096UL
#define VICTIM_SIZE    (32UL * 1024 * 1024)     /* 32 MiB              */
#define UNIQUE_PAGES   2048                     /* distinct contents    */
#define DUPLICATES     4                        /* copies per content   */
#define CHURN_SIZE     (256UL * 1024 * 1024)    /* 256 MiB churn region */
#define TIMEOUT_SEC    180
#define POLL_INTERVAL  200000                   /* 200 ms               */
#define STABLE_TICKS   5                        /* 1 sec of stability   */
#define MAX_CHURNERS   64

static volatile int stop_churn = 0;

static long sysfs_read_long(const char *path)
{
    char buf[64];
    int fd = open(path, O_RDONLY);
    ssize_t n;

    if (fd < 0)
        return -1;
    n = read(fd, buf, sizeof(buf) - 1);
    close(fd);
    if (n <= 0)
        return -1;
    buf[n] = '\0';
    return atol(buf);
}

static void sysfs_write_str(const char *path, const char *val)
{
    int fd = open(path, O_WRONLY);
    if (fd < 0) {
        fprintf(stderr, "open %s: %s\n", path, strerror(errno));
        exit(1);
    }
    if (write(fd, val, strlen(val)) < 0) {
        fprintf(stderr, "write %s: %s\n", path, strerror(errno));
        close(fd);
        exit(1);
    }
    close(fd);
}

static double monotonic_sec(void)
{
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return ts.tv_sec + ts.tv_nsec / 1e9;
}

/*
 * Churner thread: repeatedly mmap()/munmap() a large anonymous region.
 * Each iteration takes mmap_lock in write mode on the shared mm_struct.
 * This is what creates contention with ksmd's mmap_read_lock on the
 * baseline kernel.
 */
static void *churn_thread(void *arg)
{
    (void)arg;
    while (!stop_churn) {
        void *p = mmap(NULL, CHURN_SIZE, PROT_READ | PROT_WRITE,
                       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
        if (p == MAP_FAILED)
            continue;
        *(volatile char *)p = 1;   /* fault in first page */
        munmap(p, CHURN_SIZE);
    }
    return NULL;
}

static void ksm_prepare(void)
{
    sysfs_write_str("/sys/kernel/mm/ksm/run", "0");
    sysfs_write_str("/sys/kernel/mm/ksm/run", "2");   /* unmerge + reset */
    sysfs_write_str("/sys/kernel/mm/ksm/pages_to_scan", "100000");
    sysfs_write_str("/sys/kernel/mm/ksm/sleep_millisecs", "0");
}

/*
 * Allocate and register the victim region in the CURRENT process.
 * Must be called from the main thread before starting ksmd.
 */
static void *setup_victim(void)
{
    size_t npages = VICTIM_SIZE / PAGE_SIZE;
    unsigned char *p;
    size_t i;
    int j;

    p = mmap(NULL, VICTIM_SIZE, PROT_READ | PROT_WRITE,
             MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (p == MAP_FAILED) {
        perror("victim mmap");
        exit(1);
    }

    /*
     * Fill with UNIQUE_PAGES distinct patterns, each repeated DUPLICATES
     * times.  KSM will eventually collapse each group of DUPLICATES
     * identical pages into a single shared page.
     */
    for (i = 0; i < npages; i++) {
        unsigned int idx = (unsigned int)(i % UNIQUE_PAGES);
        unsigned int *page = (unsigned int *)(p + i * PAGE_SIZE);
        for (j = 0; j < (int)(PAGE_SIZE / sizeof(unsigned int)); j++)
            page[j] = idx * 2654435761u + (unsigned int)j;
    }

    if (madvise(p, VICTIM_SIZE, MADV_MERGEABLE) != 0) {
        perror("madvise(MADV_MERGEABLE)");
        exit(1);
    }
    return p;
}

int main(int argc, char **argv)
{
    int n_churn = (argc > 1) ? atoi(argv[1]) : 4;
    pthread_t threads[MAX_CHURNERS];
    long baseline_shared, baseline_scans;
    long cur_shared, cur_scans;
    double t_start, t_end;
    void *victim;
    long last = -1;
    int  stable = 0;
    double deadline;
    int i;

    if (n_churn < 0) n_churn = 0;
    if (n_churn > MAX_CHURNERS) n_churn = MAX_CHURNERS;

    ksm_prepare();

    /*
     * Set up the victim region while we still hold mmap_lock
     * uncontended.  It will be in the same mm as the churners.
     */
    fprintf(stderr, "setting up victim region (%lu MiB, %d unique pages)...\n",
            VICTIM_SIZE / (1024 * 1024), UNIQUE_PAGES);
    victim = setup_victim();

    fprintf(stderr, "starting %d churner thread(s) in the same process...\n",
            n_churn);
    for (i = 0; i < n_churn; i++) {
        if (pthread_create(&threads[i], NULL, churn_thread, NULL) != 0) {
            perror("pthread_create");
            exit(1);
        }
    }

    /* Give churners time to start hammering mmap_lock. */
    usleep(500 * 1000);

    baseline_shared = sysfs_read_long("/sys/kernel/mm/ksm/pages_shared");
    baseline_scans  = sysfs_read_long("/sys/kernel/mm/ksm/full_scans");
    if (baseline_shared < 0 || baseline_scans < 0) {
        fprintf(stderr, "cannot read KSM counters (run as root?)\n");
        return 1;
    }
    fprintf(stderr, "baseline: pages_shared=%ld full_scans=%ld\n",
            baseline_shared, baseline_scans);

    /* Start ksmd and mark t_start. */
    fprintf(stderr, "starting ksmd...\n");
    t_start = monotonic_sec();
    sysfs_write_str("/sys/kernel/mm/ksm/run", "1");

    /*
     * Wait until the victim's pages are merged: require at least two
     * full ksmd scans (to confirm the pages are stable) and then require
     * pages_shared to be unchanged for STABLE_TICKS * POLL_INTERVAL.
     */
    deadline = t_start + TIMEOUT_SEC;
    while (monotonic_sec() < deadline) {
        usleep(POLL_INTERVAL);
        cur_shared = sysfs_read_long("/sys/kernel/mm/ksm/pages_shared");
        cur_scans  = sysfs_read_long("/sys/kernel/mm/ksm/full_scans");
        if (cur_shared < 0)
            break;
        if (cur_scans - baseline_scans < 2)
            continue;
        if (cur_shared == last) {
            if (++stable >= STABLE_TICKS)
                break;
        } else {
            stable = 0;
            last = cur_shared;
        }
    }

    t_end = monotonic_sec();
    cur_shared = sysfs_read_long("/sys/kernel/mm/ksm/pages_shared");
    cur_scans  = sysfs_read_long("/sys/kernel/mm/ksm/full_scans");

    printf("=== result ===\n");
    printf("merge_time_sec : %.3f\n", t_end - t_start);
    printf("pages_shared   : %ld -> %ld (delta %ld)\n",
           baseline_shared, cur_shared, cur_shared - baseline_shared);
    printf("full_scans     : %ld -> %ld (delta %ld)\n",
           baseline_scans, cur_scans, cur_scans - baseline_scans);

    stop_churn = 1;
    for (i = 0; i < n_churn; i++)
        pthread_join(threads[i], NULL);

    munmap(victim, VICTIM_SIZE);
    return 0;
}

Re: [PATCH 0/4] mm/ksm: use per-VMA locking for find_mergeable_vma()

[Jinjiang Tu]

在 2026/9/11 16:04, xu.xin16@zte.com.cn 写道:
> From: Xu Xin (ZTE) <xu.xin@linux.dev>
>
> KSM scans VM_MERGEABLE VMAs and currently protects each scan with
> mmap_read_lock(). Per-VMA locking allows KSM to read-lock only the VMA
> it is actually interested in, so that unrelated mmap()/munmap() activity
> in the same mm no longer blocks ksmd.
>
> This series is organized as follows:
>
>    Patch 1 removes an unused 'vma' member from struct folio_walk.  It has
>    never been used since its introduction and is pure cleanup.
>
>    Patch 2 adds a 'walk_lock' member to struct folio_walk and extends
>    folio_walk_start() to assert the required locking mode.  Existing
>    callers are converted to pass PGWALK_RDLOCK, so there is no functional
>    change.  This prepares folio_walk_start() for callers that hold a
>    per-VMA read lock instead of mmap_read_lock(), which is needed by the
>    Patch 4. No functional change.
>
>    Patch 3 tranforms the boolean 'lock_vma' into the enum 'page_walk_lock'
>    without any behavior changed, which is prepared for the Patch 4 to use
>    per-VMA locking. No functional change.
>
>    Patch 4 introduces find_mergeable_vma_locked(), which uses the
>    universal per-VMA locking helper vma_start_read_unlocked() to look up
>    and read-lock a VM_MERGEABLE VMA without taking mmap_read_lock().  All
>    KSM call sites that previously used find_mergeable_vma() under
>    mmap_read_lock() are converted to the new helper, and the locking in
>    get_mergeable_page() is switched to PGWALK_VMA_RDLOCK_VERIFY so that
>    folio_walk_start() can verify the per-VMA lock is held.
>
> A microbenchmark was run to measure the time KSM takes to merge a
> victim region under mmap_lock contention. Under interference from 4 churner
> threads, the merge time of the per-VMA KSM-optimized kernel is
> significantly reduced by 50%.

Hi.

During task exiting, __ksm_exit() uses mmap_write_lock() to synchronize with ksmd.
see the comment of ksm_test_exit().

void __ksm_exit(struct mm_struct *mm)
{
   ...

	if (easy_to_free) {
		mm_slot_free(mm_slot_cache, mm_slot);
		mm_flags_clear(MMF_VM_MERGE_ANY, mm);
		mm_flags_clear(MMF_VM_MERGEABLE, mm);
		mmdrop(mm);
	} else if (mm_slot) {
		mmap_write_lock(mm);
		mmap_write_unlock(mm);
	}

}

When ksmd currently is scanning the exiting mm, we should guarantee the mm pagetable
still valid (i.e., mm_users > 0). However, ksm_mm_slot only holds mm_count, which only
guarantees the mm_strcut isn't freed. So, __ksm_exit() uses mmap write lock to synchronize
with ksmd.

IIUC, vma_read_lock cannot be exclusive with mmap_write_lock().

>
> Xu Xin (4):
>    mm/pagewalk: delete the unused member
>    mm: make folio_walk_start()'s locking asserts scalable
>    mm/ksm: make break_ksm() more scalable
>    mm/ksm: add find_mergeable_vma_locked() to use per-VMA locking
>
>   arch/s390/mm/fault.c     |  4 +-
>   include/linux/pagewalk.h |  2 +-
>   kernel/events/uprobes.c  |  4 +-
>   mm/huge_memory.c         |  4 +-
>   mm/ksm.c                 | 85 ++++++++++++++++++++++++----------------
>   mm/migrate.c             |  8 +++-
>   mm/pagewalk.c            | 10 ++++-
>   mm/rmap.c                |  4 +-
>   8 files changed, 79 insertions(+), 42 deletions(-)
>

Re: [PATCH 1/4] mm/pagewalk: delete the unused member

[Lorenzo Stoakes (ARM)]

Please fix your cc'ing.

Cc- everybody on every mail.

You cc me only on this (1/4) and 3/4 but then don't on an mm/rmap.c change in
2/4 when I co-maintain rmap.

And 4/4 is about use of VMA locks which I also co-maintain :)

Just cc- everybody on everything...

On Fri, Sep 11, 2026 at 04:07:25PM +0800, xu.xin16@zte.com.cn wrote:
> From: Xu Xin (ZTE) <xu.xin@linux.dev>
>
> The member vma has never been used, we should delete it

This is very sparse. Can you back this by talking about commits and how the VMA
is actually used? Have you audited it to make sure?

>
> Fixes: aa39ca6940f1a ("mm/pagewalk: introduce folio_walk_start() + folio_walk_end()")

I don't see why a fixes tag is necessary here. The struct folio_walk value is
kept on the stack so what exactly is the point of treating it like a fix?

> Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>

AFAICT this isn't currently used but I think I need to hear a stronger argument
in the commit msg before I go looking myself...!

> ---
>  include/linux/pagewalk.h | 1 -
>  1 file changed, 1 deletion(-)
>
> diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h
> index b41d7265c01b..1c397be0d092 100644
> --- a/include/linux/pagewalk.h
> +++ b/include/linux/pagewalk.h
> @@ -183,7 +183,6 @@ struct folio_walk {
>  		pmd_t pmd;
>  	};
>  	/* private */
> -	struct vm_area_struct *vma;
>  	spinlock_t *ptl;
>  };
>
> --
> 2.25.1

--
Cheers, Lorenzo

Re: [PATCH 3/4] mm/ksm: make break_ksm() more scalable

[Lorenzo Stoakes (ARM)]

On Fri, Sep 11, 2026 at 04:12:10PM +0800, xu.xin16@zte.com.cn wrote:
> From: Xu Xin (ZTE) <xu.xin@linux.dev>
>
> Currently the last argument 'walk_lock' of break_ksm() is used to
> indicate whether the page_walk is protected by mmap_read_lock or
> mmap_write_lock. If 'walk_lock' is true, we suppose its context to
> be under mmap_write_lock() protection, then mark it PGWALK_WRLOCK and
> make its vma be write-locked during the walk; If 'walk_lock' is
> false, we suppose its context to be mmap_read_lock(), then mark it
> PGWALK_RDLOCK.

I thnk this whole block is unnecessary. You're basically writing what the code
does in English

>
> This change is prepared for the latter patch to enable VMA

Latter -> later.

And it's the patch I'm not cc'd on so I don't see unless I go do a bunch of
stuff to try to download it... great :)

> read-locking where break_ksm() might be under the third new proctecion
> way: VMA read-locking, so we have to replace the boolean variable to
> the enum 'page_walk_lock', but without any function changed.

You don't, this is just horrible.

>
> No functional change intended.
>
> Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>
> ---
>  mm/ksm.c | 21 ++++++++-------------
>  1 file changed, 8 insertions(+), 13 deletions(-)
>
> diff --git a/mm/ksm.c b/mm/ksm.c
> index 8df66b4e5de0..dda105681d7f 100644
> --- a/mm/ksm.c
> +++ b/mm/ksm.c
> @@ -660,16 +660,11 @@ static int break_ksm_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned long en
>  	return found;
>  }
>
> -static const struct mm_walk_ops break_ksm_ops = {
> +static struct mm_walk_ops break_ksm_ops = {
>  	.pmd_entry = break_ksm_pmd_entry,
>  	.walk_lock = PGWALK_RDLOCK,
>  };
>
> -static const struct mm_walk_ops break_ksm_lock_vma_ops = {
> -	.pmd_entry = break_ksm_pmd_entry,
> -	.walk_lock = PGWALK_WRLOCK,
> -};
> -
>  /*
>   * Though it's very tempting to unmerge rmap_items from stable tree rather
>   * than check every pte of a given vma, the locking doesn't quite work for
> @@ -696,11 +691,11 @@ static const struct mm_walk_ops break_ksm_lock_vma_ops = {
>   * protection keys here anyway.
>   */
>  static int break_ksm(struct vm_area_struct *vma, unsigned long addr,
> -		unsigned long end, bool lock_vma)
> +		unsigned long end, enum page_walk_lock walk_lock)

Ugh yuck this is horrible, you're exposing internal page walker state here as a
parameter...?

And then this commit makes it possible for any walk_lock to be passed but then
you change none of the code to handle it?

>  {
>  	vm_fault_t ret = 0;
> -	const struct mm_walk_ops *ops = lock_vma ?
> -				&break_ksm_lock_vma_ops : &break_ksm_ops;
> +	struct mm_walk_ops *ops = &break_ksm_ops;
> +	ops->walk_lock = walk_lock;

Are you sure this can't be run concurrently by two walkers?

I didn't see any arguments about that in the commit message. Having a single,
static, struct where you change the walk_lock is gross.

What would be better is to have your own enum that lists ksm lock state or
express it some other way, then if possible have it on the stack otherwise
ensure that state can't be corrupted.

Again, if you'd sent me 4/4 too I could see the overall structure and give
advice but...

>
>  	do {
>  		int ksm_page;
> @@ -807,7 +802,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item)
>  	mmap_read_lock(mm);
>  	vma = find_mergeable_vma(mm, addr);
>  	if (vma)
> -		break_ksm(vma, addr, addr + PAGE_SIZE, false);
> +		break_ksm(vma, addr, addr + PAGE_SIZE, PGWALK_RDLOCK);
>  	mmap_read_unlock(mm);
>  }
>
> @@ -1245,7 +1240,7 @@ static int unmerge_and_remove_all_rmap_items(void)
>  		for_each_vma(vmi, vma) {
>  			if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
>  				continue;
> -			err = break_ksm(vma, vma->vm_start, vma->vm_end, false);
> +			err = break_ksm(vma, vma->vm_start, vma->vm_end, PGWALK_RDLOCK);
>  			if (err)
>  				goto error;
>  		}
> @@ -2885,7 +2880,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma)
>  		return 0;
>
>  	if (vma->anon_vma) {
> -		err = break_ksm(vma, vma->vm_start, vma->vm_end, true);
> +		err = break_ksm(vma, vma->vm_start, vma->vm_end, PGWALK_WRLOCK);
>  		if (err)
>  			return err;
>  	}
> @@ -3037,7 +3032,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start,
>  			return 0;		/* just ignore the advice */
>
>  		if (vma->anon_vma) {
> -			err = break_ksm(vma, start, end, true);
> +			err = break_ksm(vma, start, end, PGWALK_WRLOCK);
>  			if (err)
>  				return err;
>  		}
> --
> 2.25.1

--
Cheers, Lorenzo

Re: [PATCH 1/4] mm/pagewalk: delete the unused member

[xu.xin16]

> Please fix your cc'ing.
> 
> Cc- everybody on every mail.
> 
> You cc me only on this (1/4) and 3/4 but then don't on an mm/rmap.c change in
> 2/4 when I co-maintain rmap.
> 
> And 4/4 is about use of VMA locks which I also co-maintain :)
> 
> Just cc- everybody on everything...
> 

Oh, I’m really sorry—I accidentally missed sending it to you. 

I’ll CC everyone in the follow-ups.

Re: [PATCH 4/4] mm/ksm: add find_mergeable_vma_locked() to use per-VMA locking

[xu.xin16]

CC: Lorenzo Stoakes <ljs@kernel.org>
---
From: Xu Xin (ZTE) <xu.xin@linux.dev>

Purpose
=======
Let's add find_mergeable_vma_locked(), which is similar to find_tcp_vma(),
using the universal per-VMA locking helper, so that we can avoid
mmap_read_lock() to reduce contention.

To be used in KSM code to replace find_mergeable_vma() with
mmap_read_lock(), the helper find_mergeable_vma_locked() uses the
universal per-VMA locking allowing us to lock a struct vm_area_struct
without taking the process-wide mmap lock in read mode.

Performance
===========
As a test, I construct a testcase which follows the approach: Create
one victim and several churner threads sharing one mm_struct; The
victim registers a 32 MiB anonymous VM_MERGEABLE region containing
8192 pages totally:churners hammer mmap_lock via mmap/munmap
repeatedly; ksmd merges the victim's pages; Compare latency baseline
VS this per-VMA patch.

               Before patched         After Patched
0 churner:     1.627 seconds          1.426 seconds
4 churners:    72.45 seconds          36.61 seconds

In conclusion, when no mmap_lock contention (0 churner), there is little
difference between the baseline kernel and the per-VMA optimized kernel;
But under interference from 4 churner threads, the merge time of the
per-VMA KSM-optimized kernel is significantly reduced by 50%.

Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>
---
 mm/ksm.c | 64 +++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 42 insertions(+), 22 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index dda105681d7f..1d85769ec7db 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -765,15 +765,36 @@ static bool vma_ksm_compatible(struct vm_area_struct *vma)
 	return ksm_compatible(vma->vm_file, vma->flags);
 }

-static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,
-		unsigned long addr)
+/**
+ * find_mergeable_vma_locked() - Find the VMA covering 'address' which is
+ * VM_MERGEABLE and read-lock it by per-VMA locks. Please use vma_end_read()
+ * to unlock vma after finishing reading the VMA (non-NULL).
+ *
+ * Return: If a VMA exists which spans @address, return that VMA, read-locked.
+ * If no VMA is mapped there or, very unlikely, a reference count overflow
+ * occurred, return NULL, and no read-locked.
+ *
+ * IMPORTANT: If a VMA exists but is not VM_MERGEABLE or has no anon_vma,
+ * this function releases the per-VMA read lock before returning NULL.
+ * Callers must NOT call vma_end_read() on a NULL return value.
+ */
+static struct vm_area_struct *find_mergeable_vma_locked(struct mm_struct *mm,
+		unsigned long address)
 {
 	struct vm_area_struct *vma;
+
 	if (ksm_test_exit(mm))
 		return NULL;
-	vma = vma_lookup(mm, addr);
-	if (!vma || !(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
+
+	vma = vma_start_read_unlocked(mm, address);
+	if (!vma)
+		return NULL;
+
+	if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) {
+		vma_end_read(vma);
 		return NULL;
+	}
+
 	return vma;
 }

@@ -799,11 +820,12 @@ static void break_cow(struct ksm_rmap_item *rmap_item)
 	 */
 	rmap_item->linear_page_index = 0;

-	mmap_read_lock(mm);
-	vma = find_mergeable_vma(mm, addr);
-	if (vma)
-		break_ksm(vma, addr, addr + PAGE_SIZE, PGWALK_RDLOCK);
-	mmap_read_unlock(mm);
+	vma = find_mergeable_vma_locked(mm, addr);
+	if (!vma)
+		return;
+
+	break_ksm(vma, addr, addr + PAGE_SIZE, PGWALK_VMA_RDLOCK_VERIFY);
+	vma_end_read(vma);
 }

 static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
@@ -814,13 +836,12 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 	struct page *page = NULL;
 	struct folio *folio;
 	struct folio_walk fw = {
-		.walk_lock  = PGWALK_RDLOCK,
+		.walk_lock  = PGWALK_VMA_RDLOCK_VERIFY,
 	};

-	mmap_read_lock(mm);
-	vma = find_mergeable_vma(mm, addr);
+	vma = find_mergeable_vma_locked(mm, addr);
 	if (!vma)
-		goto out;
+		return NULL;

 	folio = folio_walk_start(&fw, vma, addr, 0);
 	if (folio) {
@@ -831,12 +852,12 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 		}
 		folio_walk_end(&fw, vma);
 	}
-out:
+
 	if (page) {
 		flush_anon_page(vma, page, addr);
 		flush_dcache_page(page);
 	}
-	mmap_read_unlock(mm);
+	vma_end_read(vma);
 	return page;
 }

@@ -1568,14 +1589,14 @@ static int try_to_merge_with_zero_page(struct ksm_rmap_item *rmap_item,
 	if (ksm_use_zero_pages && (rmap_item->oldchecksum == zero_checksum)) {
 		struct vm_area_struct *vma;

-		mmap_read_lock(mm);
-		vma = find_mergeable_vma(mm, rmap_item->address);
+		vma = find_mergeable_vma_locked(mm, rmap_item->address);
 		if (vma) {
 			err = try_to_merge_one_page(vma, page,
 					ZERO_PAGE(rmap_item->address));
 			trace_ksm_merge_one_page(
 				page_to_pfn(ZERO_PAGE(rmap_item->address)),
 				rmap_item, mm, err);
+			vma_end_read(vma);
 		} else {
 			/*
 			 * If the vma is out of date, we do not need to
@@ -1583,7 +1604,6 @@ static int try_to_merge_with_zero_page(struct ksm_rmap_item *rmap_item,
 			 */
 			err = 0;
 		}
-		mmap_read_unlock(mm);
 	}

 	return err;
@@ -1602,10 +1622,9 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item,
 	struct vm_area_struct *vma;
 	int err = -EFAULT;

-	mmap_read_lock(mm);
-	vma = find_mergeable_vma(mm, rmap_item->address);
+	vma = find_mergeable_vma_locked(mm, rmap_item->address);
 	if (!vma)
-		goto out;
+		goto out_trace;

 	err = try_to_merge_one_page(vma, page, kpage);
 	if (err)
@@ -1625,7 +1644,8 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item,
 	rmap_item->linear_page_index = linear_anon_page_index(vma, rmap_item->address);
 	get_anon_vma(vma->anon_vma);
 out:
-	mmap_read_unlock(mm);
+	vma_end_read(vma);
+out_trace:
 	trace_ksm_merge_with_ksm_page(kpage, page_to_pfn(kpage ? kpage : page),
 				rmap_item, mm, err);
 	return err;
-- 
2.25.1

Re: [PATCH 2/4] mm: make folio_walk_start()'s locking asserts scalable

[xu.xin16]

CC: Lorenzo Stoakes <ljs@kernel.org>
----
From: Xu Xin (ZTE) <xu.xin@linux.dev>

Add an additional member 'walk_lock' to folio_walk to indicate
locking requirements for the walk. Similar to commit 49b0638502da0
("mm: enable page walking API to lock vmas during the walk").
But no change is made on any existing locking behavior,
all existing folio_walk_start() callers are currently still
under mmap_read_lock() protection.

This change is prepared for the latter patch to enable VMA locking
asserts. folio_walk_start now operate under write-locked mmap_lock.
With introduction of vma locks at the next patch, the vmas have to
be locked as well during such walks to prevent concurrent page
faults in these areas.

No functional change intended.

Signed-off-by: Xu Xin (ZTE) <xu.xin@linux.dev>
---
 arch/s390/mm/fault.c     |  4 +++-
 include/linux/pagewalk.h |  1 +
 kernel/events/uprobes.c  |  4 +++-
 mm/huge_memory.c         |  4 +++-
 mm/ksm.c                 |  4 +++-
 mm/migrate.c             |  8 ++++++--
 mm/pagewalk.c            | 10 +++++++++-
 mm/rmap.c                |  4 +++-
 8 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index 46d828926009..968f128a5232 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -412,7 +412,9 @@ __context_unsafe(/* folio_walk_end() not instrumented */)
 	unsigned long addr = get_fault_address(regs);
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct *vma;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct folio *folio;
 	int rc;

diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h
index 1c397be0d092..455428eaf1fc 100644
--- a/include/linux/pagewalk.h
+++ b/include/linux/pagewalk.h
@@ -184,6 +184,7 @@ struct folio_walk {
 	};
 	/* private */
 	spinlock_t *ptl;
+	enum page_walk_lock walk_lock;
 };

 struct folio *folio_walk_start(struct folio_walk *fw,
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 7709ea882477..054fdcb52136 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -507,7 +507,9 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
 	int ret, ref_ctr_updated = 0;
 	unsigned int gup_flags = FOLL_FORCE;
 	struct mmu_notifier_range range;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct folio *folio;
 	struct page *page;

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index dd66c6ad5af1..b714677e2f20 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4795,7 +4795,9 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start,
 	 */
 	for (addr = vaddr_start; addr < vaddr_end; addr += PAGE_SIZE) {
 		struct vm_area_struct *vma = vma_lookup(mm, addr);
-		struct folio_walk fw;
+		struct folio_walk fw = {
+			.walk_lock  = PGWALK_RDLOCK,
+		};
 		struct folio *folio;
 		struct address_space *mapping;
 		unsigned int target_order = new_order;
diff --git a/mm/ksm.c b/mm/ksm.c
index 624f37975e12..8df66b4e5de0 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -817,8 +817,10 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 	unsigned long addr = rmap_item->address;
 	struct vm_area_struct *vma;
 	struct page *page = NULL;
-	struct folio_walk fw;
 	struct folio *folio;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};

 	mmap_read_lock(mm);
 	vma = find_mergeable_vma(mm, addr);
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..fa638adfb0de 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2302,7 +2302,9 @@ static int add_folio_for_migration(struct mm_struct *mm, const void __user *p,
 		int node, struct list_head *pagelist, bool migrate_all)
 {
 	struct vm_area_struct *vma;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct folio *folio;
 	unsigned long addr;
 	int err = -EFAULT;
@@ -2464,7 +2466,9 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
 	for (i = 0; i < nr_pages; i++) {
 		unsigned long addr = (unsigned long)(*pages);
 		struct vm_area_struct *vma;
-		struct folio_walk fw;
+		struct folio_walk fw = {
+			.walk_lock  = PGWALK_RDLOCK,
+		};
 		struct folio *folio;
 		int err = -EFAULT;

diff --git a/mm/pagewalk.c b/mm/pagewalk.c
index 7411702a37f5..8eb29fba20ad 100644
--- a/mm/pagewalk.c
+++ b/mm/pagewalk.c
@@ -910,7 +910,15 @@ struct folio *folio_walk_start(struct folio_walk *fw,
 	pgd_t *pgdp;
 	p4d_t *p4dp;

-	mmap_assert_locked(vma->vm_mm);
+	/*
+	 * Other locking modes except for mmap or vma read locking are not
+	 * expected.
+	 */
+	if (fw->walk_lock != PGWALK_RDLOCK && fw->walk_lock != PGWALK_VMA_RDLOCK_VERIFY)
+		WARN_ONCE(1, "walk_lock is not expected!\n");
+	process_mm_walk_lock(vma->vm_mm, fw->walk_lock);
+	process_vma_walk_lock(vma, fw->walk_lock);
+
 	vma_pgtable_walk_begin(vma);

 	if (WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end))
diff --git a/mm/rmap.c b/mm/rmap.c
index 5fefe5b060b1..fd8e1c9e4424 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2871,7 +2871,9 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,
 	struct mmu_notifier_range range;
 	struct folio *folio, *fw_folio;
 	struct vm_area_struct *vma;
-	struct folio_walk fw;
+	struct folio_walk fw = {
+		.walk_lock  = PGWALK_RDLOCK,
+	};
 	struct page *page;
 	swp_entry_t entry;
 	pte_t swp_pte;
-- 
2.25.1

[PATCH RFC 0/3] mm/ksm: scan with per-VMA locks

[Longlong Xia]

From: Longlong Xia <xialonglong@kylinos.cn>

This work was developed independently before I saw the above series.
After comparing the two implementations, I found that the approaches
overlap, so I am posting this draft version as an alternative 
implementation and would be happy to combine the useful parts.

  The main differences are:

  - use per-VMA locking in the ksmd VMA scanner;
  - revalidate the mmap-lock sequence before skipping the fallback walk;
  - avoid repeating a completed VMA traversal when no candidate page is
    found.

Benchmark (QEMU q35, TCG multi-thread, 4 vCPUs, 8 GiB, x86_64 guest on
Ubuntu 24.04, 7.3-rc2 base, CONFIG_KSM=y + CONFIG_PER_VMA_LOCK=y;
victim: 32 MiB anon, 2048 unique pages x4 duplicates,
pages_to_scan=100000, sleep_millisecs=0, 3 runs each; churners: 4
threads mmap/munmaping 256 MiB in the same mm. Microbenchmark as
posted in [3]; relative numbers only, TCG is not native hardware):

                       0 churners      4 churners (mean)   
  7.3-rc2              1.21-1.41 s     90.9 s             
  + patch 2/3          1.41 s          48.7 s   (-46%)   
  + patch 2/3 + 3/3    1.21-1.41 s     30.1 s   (-67%)  


 1. mm/pagewalk: allow folio_walk_start() under a vma read lock

 2. mm/ksm: use the VMA lock when looking up mergeable pages

 3. mm/ksm: scan VMAs with per-VMA locks

Link:https://lore.kernel.org/all/5cd47bc3-bc3d-473c-80d0-8be8b7b79881@163.com/

Longlong Xia (3):
  mm/pagewalk: allow folio_walk_start() under a vma read lock
  mm/ksm: use the VMA lock when looking up mergeable pages
  mm/ksm: scan VMAs with per-VMA locks

 include/linux/pagewalk.h |   3 +
 mm/ksm.c                 | 166 ++++++++++++++++++++++++++++++++++++---
 mm/pagewalk.c            |  11 ++-
 3 files changed, 166 insertions(+), 14 deletions(-)

base-commit: df2908090cda368b01ff43709f51890076c56157
-- 
2.43.0

[PATCH RFC 1/3] mm/pagewalk: allow folio_walk_start() under a vma read lock

[Longlong Xia]

From: Longlong Xia <xialonglong@kylinos.cn>

Allow folio_walk_start() to use a vma read lock by replacing
mmap_assert_locked() with vma_assert_locked().  This lets KSM look
up pages without taking the mmap read lock.

The vma lock stabilizes the mapping, while page table locks serialize
entry updates.  Callers walking another mm under the vma lock must
also hold an mm_users reference to prevent exit_mmap() from freeing
the page tables.

Assisted-by: Zcode:GLM-5.3
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
---
 include/linux/pagewalk.h |  3 +++
 mm/pagewalk.c            | 11 +++++++++--
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h
index b41d7265c01b..cafd7b15f480 100644
--- a/include/linux/pagewalk.h
+++ b/include/linux/pagewalk.h
@@ -151,6 +151,9 @@ typedef int __bitwise folio_walk_flags_t;
 /* Walk shared zeropages (small + huge) as well. */
 #define FW_ZEROPAGE			((__force folio_walk_flags_t)BIT(0))
 
+/* The caller holds the VMA read lock instead of the mmap lock. */
+#define FW_VMA_LOCKED			((__force folio_walk_flags_t)BIT(1))
+
 enum folio_walk_level {
 	FW_LEVEL_PTE,
 	FW_LEVEL_PMD,
diff --git a/mm/pagewalk.c b/mm/pagewalk.c
index cc07fcf50e87..57ffaf2a85b8 100644
--- a/mm/pagewalk.c
+++ b/mm/pagewalk.c
@@ -894,7 +894,11 @@ int walk_page_mapping(struct address_space *mapping, pgoff_t first_index,
  * huge_ptep_set_*, ...). Note that the page table entry stored in @fw might
  * not correspond to the first physical entry of a logical hugetlb entry.
  *
- * The mmap lock must be held in read mode.
+ * The mmap lock must be held in read mode.  Alternatively, with
+ * CONFIG_PER_VMA_LOCK and @FW_VMA_LOCKED, the vma lock may be held in read mode: the
+ * page tables of a read-locked vma cannot be torn down while the mm has
+ * users, so a caller that walks an mm other than its own must also hold a
+ * mm_users reference for the duration of the walk.
  *
  * Return: folio pointer on success, otherwise NULL.
  */
@@ -912,7 +916,10 @@ struct folio *folio_walk_start(struct folio_walk *fw,
 	pgd_t *pgdp;
 	p4d_t *p4dp;
 
-	mmap_assert_locked(vma->vm_mm);
+	if (flags & FW_VMA_LOCKED)
+		vma_assert_locked(vma);
+	else
+		mmap_assert_locked(vma->vm_mm);
 	vma_pgtable_walk_begin(vma);
 
 	if (WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end))
-- 
2.43.0

[PATCH RFC 2/3] mm/ksm: use the VMA lock when looking up mergeable pages

[Longlong Xia]

From: Longlong Xia <xialonglong@kylinos.cn>

get_mergeable_page() currently takes mm->mmap_lock for every unstable-tree
lookup, so KSM is serialized with layout changes to unrelated VMAs. Try
lock_vma_under_rcu() while holding an mm_users reference and walk the page
tables under that VMA's read lock. Retain the mmap read-lock path as a
fallback when the VMA cannot be locked or per-VMA locking is disabled.

Extend folio_walk_start() to verify a VMA read lock and keep the mm_users
reference across the walk, since a VMA lock alone does not prevent
exit_mmap() from freeing page tables. Drop the reference before waiting for
mmap_lock on the fallback path so an exiting mm is not delayed.

Assisted-by: Zcode:GLM-5.3
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
---
 mm/ksm.c | 64 +++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 52 insertions(+), 12 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index 49d48d1e0998..aee1a1b49b1b 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -811,21 +811,20 @@ static void break_cow(struct ksm_rmap_item *rmap_item)
 	mmap_read_unlock(mm);
 }
 
-static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
+/*
+ * Get the page that @addr maps in @vma, with an elevated reference, or NULL
+ * when the address no longer maps an anon page.  The caller must hold a lock
+ * that stabilizes @vma: either the mmap read lock, or the vma read lock
+ * together with an mm_users reference.
+ */
+static struct page *__get_mergeable_page(struct vm_area_struct *vma,
+					 unsigned long addr, folio_walk_flags_t flags)
 {
-	struct mm_struct *mm = rmap_item->mm;
-	unsigned long addr = rmap_item->address;
-	struct vm_area_struct *vma;
-	struct page *page = NULL;
 	struct folio_walk fw;
+	struct page *page = NULL;
 	struct folio *folio;
 
-	mmap_read_lock(mm);
-	vma = find_mergeable_vma(mm, addr);
-	if (!vma)
-		goto out;
-
-	folio = folio_walk_start(&fw, vma, addr, 0);
+	folio = folio_walk_start(&fw, vma, addr, flags);
 	if (folio) {
 		if (!folio_is_zone_device(folio) &&
 		    folio_test_anon(folio)) {
@@ -834,11 +833,52 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 		}
 		folio_walk_end(&fw, vma);
 	}
-out:
 	if (page) {
 		flush_anon_page(vma, page, addr);
 		flush_dcache_page(page);
 	}
+	return page;
+}
+
+static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
+{
+	struct mm_struct *mm = rmap_item->mm;
+	unsigned long addr = rmap_item->address;
+	struct vm_area_struct *vma;
+	struct page *page = NULL;
+
+	/*
+	 * Try the vma lock before the mmap lock, so that ksmd does not queue
+	 * behind a writer that changes the address space layout anywhere in
+	 * this mm: the vma lock only contends with modification of this very
+	 * vma.  Pin mm_users for the walk: exit_mmap() frees the page tables
+	 * under the mmap lock alone, so a vma read lock cannot keep it away,
+	 * but an mm_users reference can; the pin also stands in for the
+	 * ksm_test_exit() check of find_mergeable_vma() on this path.  Drop
+	 * it again before waiting for the mmap lock below, so that an exiting
+	 * mm is not delayed by us.
+	 */
+	if (IS_ENABLED(CONFIG_PER_VMA_LOCK) && mmget_not_zero(mm)) {
+		vma = lock_vma_under_rcu(mm, addr);
+		if (vma) {
+			if ((vma->vm_flags & VM_MERGEABLE) && vma->anon_vma)
+				page = __get_mergeable_page(vma, addr, FW_VMA_LOCKED);
+			vma_end_read(vma);
+			mmput_async(mm);
+			return page;
+		}
+		mmput_async(mm);
+	}
+
+	/*
+	 * The vma is being modified, or CONFIG_PER_VMA_LOCK is off: take the
+	 * mmap read lock as before.  We are prepared to wait rather than skip
+	 * this page, so a contended mm only slows down its own merging.
+	 */
+	mmap_read_lock(mm);
+	vma = find_mergeable_vma(mm, addr);
+	if (vma)
+		page = __get_mergeable_page(vma, addr, 0);
 	mmap_read_unlock(mm);
 	return page;
 }
-- 
2.43.0

[PATCH RFC 3/3] mm/ksm: scan VMAs with per-VMA locks

[Longlong Xia]

From: Longlong Xia <xialonglong@kylinos.cn>

KSM currently holds mm->mmap_lock while scanning the VMAs of an mm. Use
lock_next_vma() and walk_page_range_vma() to acquire each VMA's read lock
and release it after finding a candidate page, allowing unrelated VMA
updates to proceed while ksmd scans.

Pin mm_users for the duration of the lockless VMA traversal to keep page
tables alive, and retain the mmap-lock scan as a fallback when a VMA lock
cannot be acquired or per-VMA locking is disabled. Allocation failures
preserve the existing early-stop semantics. This reduces mmap_lock
contention for workloads with concurrent address-space updates while
preserving existing scan semantics.

When the per-VMA scan completes without finding a candidate, remember the
complete traversal and revalidate the mmap-lock sequence before entering
the fallback. If the address space is unchanged, skip the duplicate VMA
walk; if a VMA writer raced with the traversal, restart the fallback walk.

Assisted-by: Zcode:GLM-5.3
Signed-off-by: Longlong Xia <xialonglong@kylinos.cn>
---
 mm/ksm.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 107 insertions(+)

diff --git a/mm/ksm.c b/mm/ksm.c
index aee1a1b49b1b..2c0b2adb83d4 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -2647,6 +2647,98 @@ static struct mm_walk_ops ksm_next_page_ops = {
 	.walk_lock = PGWALK_RDLOCK,
 };
 
+#ifdef CONFIG_PER_VMA_LOCK
+static const struct mm_walk_ops ksm_next_page_vma_ops = {
+	.pmd_entry = ksm_next_page_pmd_entry,
+	.walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
+};
+
+/* Return true when a candidate was processed, including allocation failure. */
+static bool scan_get_next_rmap_item_vma(struct ksm_mm_slot *mm_slot,
+		struct page **page, struct ksm_rmap_item **result,
+		bool *complete, unsigned int *mm_wr_seq)
+{
+	struct mm_struct *mm = mm_slot->slot.mm;
+	unsigned long address = ksm_scan.address;
+	struct vm_area_struct *vma;
+	struct vma_iterator vmi;
+	bool done = false;
+	bool mmap_unlocked;
+
+	if (!mmget_not_zero(mm))
+		return false;
+
+	mmap_unlocked = mmap_lock_speculate_try_begin(mm, mm_wr_seq);
+
+	for (;;) {
+		rcu_read_lock();
+		vma_iter_init(&vmi, mm, address);
+		vma = lock_next_vma(mm, &vmi, address);
+		rcu_read_unlock();
+		if (IS_ERR_OR_NULL(vma))
+			break;
+
+		address = vma->vm_end;
+		if (!(vma->vm_flags & VM_MERGEABLE))
+			goto next_vma;
+		if (ksm_scan.address < vma->vm_start)
+			ksm_scan.address = vma->vm_start;
+		if (!vma->anon_vma)
+			ksm_scan.address = vma->vm_end;
+
+		while (ksm_scan.address < vma->vm_end) {
+			struct ksm_next_page_arg arg;
+			struct ksm_rmap_item *rmap_item;
+			int found;
+
+			found = walk_page_range_vma(vma, ksm_scan.address,
+					vma->vm_end, &ksm_next_page_vma_ops, &arg);
+			if (found <= 0) {
+				VM_WARN_ON_ONCE(found < 0);
+				ksm_scan.address = vma->vm_end;
+				break;
+			}
+
+			ksm_scan.address = arg.addr;
+			flush_anon_page(vma, arg.page, arg.addr);
+			flush_dcache_page(arg.page);
+			rmap_item = get_next_rmap_item(mm_slot,
+					ksm_scan.rmap_list, arg.addr);
+			if (rmap_item) {
+				ksm_scan.rmap_list = &rmap_item->rmap_list;
+				if (should_skip_rmap_item(arg.folio, rmap_item)) {
+					folio_put(arg.folio);
+					ksm_scan.address += PAGE_SIZE;
+					cond_resched();
+					continue;
+				}
+				ksm_scan.address += PAGE_SIZE;
+				*page = arg.page;
+			} else {
+				folio_put(arg.folio);
+			}
+			*result = rmap_item;
+			done = true;
+			vma_end_read(vma);
+			goto out;
+		}
+next_vma:
+		/*
+		 * Don't advance ksm_scan.address for VMAs the mmap-lock loop
+		 * skips with a plain continue: the cursor has to stay 0 when
+		 * this mm holds no VM_MERGEABLE vma, so the fallback walk can
+		 * remove the mm from the scan list at the end of the pass.
+		 */
+		vma_end_read(vma);
+		cond_resched();
+	}
+	*complete = mmap_unlocked && !vma;
+out:
+	mmput_async(mm);
+	return done;
+}
+#endif
+
 static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 {
 	struct mm_struct *mm;
@@ -2655,6 +2747,9 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 	struct vm_area_struct *vma;
 	struct ksm_rmap_item *rmap_item;
 	struct vma_iterator vmi;
+	bool skip_vma_scan = false;
+	bool vma_scan_complete = false;
+	unsigned int mm_wr_seq;
 	int nid;
 
 	if (list_empty(&ksm_mm_head.slot.mm_node))
@@ -2719,12 +2814,23 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 
 	slot = &mm_slot->slot;
 	mm = slot->mm;
+#ifdef CONFIG_PER_VMA_LOCK
+	rmap_item = NULL;
+	if (scan_get_next_rmap_item_vma(mm_slot, page, &rmap_item,
+			&vma_scan_complete, &mm_wr_seq))
+		return rmap_item;
+#endif
+	/* Recheck the end of the scan under mmap_lock before removing the mm. */
 	vma_iter_init(&vmi, mm, ksm_scan.address);
 
 	mmap_read_lock(mm);
+	if (vma_scan_complete && !mmap_lock_speculate_retry(mm, mm_wr_seq))
+		skip_vma_scan = true;
 	if (ksm_test_exit(mm))
 		goto no_vmas;
 
+	if (skip_vma_scan)
+		goto scan_cleanup;
 	for_each_vma(vmi, vma) {
 		if (!(vma->vm_flags & VM_MERGEABLE))
 			continue;
@@ -2785,6 +2891,7 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 		}
 	}
 
+scan_cleanup:
 	if (ksm_test_exit(mm)) {
 no_vmas:
 		ksm_scan.address = 0;
-- 
2.43.0

Re: [PATCH 0/4] mm/ksm: use per-VMA locking for find_mergeable_vma()

[xu.xin16]

> >    Patch 1 removes an unused 'vma' member from struct folio_walk.  It has
> >    never been used since its introduction and is pure cleanup.
> >
> >    Patch 2 adds a 'walk_lock' member to struct folio_walk and extends
> >    folio_walk_start() to assert the required locking mode.  Existing
> >    callers are converted to pass PGWALK_RDLOCK, so there is no functional
> >    change.  This prepares folio_walk_start() for callers that hold a
> >    per-VMA read lock instead of mmap_read_lock(), which is needed by the
> >    Patch 4. No functional change.
> >
> >    Patch 3 tranforms the boolean 'lock_vma' into the enum 'page_walk_lock'
> >    without any behavior changed, which is prepared for the Patch 4 to use
> >    per-VMA locking. No functional change.
> >
> >    Patch 4 introduces find_mergeable_vma_locked(), which uses the
> >    universal per-VMA locking helper vma_start_read_unlocked() to look up
> >    and read-lock a VM_MERGEABLE VMA without taking mmap_read_lock().  All
> >    KSM call sites that previously used find_mergeable_vma() under
> >    mmap_read_lock() are converted to the new helper, and the locking in
> >    get_mergeable_page() is switched to PGWALK_VMA_RDLOCK_VERIFY so that
> >    folio_walk_start() can verify the per-VMA lock is held.
> >
> > A microbenchmark was run to measure the time KSM takes to merge a
> > victim region under mmap_lock contention. Under interference from 4 churner
> > threads, the merge time of the per-VMA KSM-optimized kernel is
> > significantly reduced by 50%.
> 
> Hi.
> 
> During task exiting, __ksm_exit() uses mmap_write_lock() to synchronize with ksmd.
> see the comment of ksm_test_exit().
> 
> void __ksm_exit(struct mm_struct *mm)
> {
>    ...
> 
> 	if (easy_to_free) {
> 		mm_slot_free(mm_slot_cache, mm_slot);
> 		mm_flags_clear(MMF_VM_MERGE_ANY, mm);
> 		mm_flags_clear(MMF_VM_MERGEABLE, mm);
> 		mmdrop(mm);
> 	} else if (mm_slot) {
> 		mmap_write_lock(mm);
> 		mmap_write_unlock(mm);
> 	}
> 
> }
> 
> When ksmd currently is scanning the exiting mm, we should guarantee the mm pagetable
> still valid (i.e., mm_users > 0). However, ksm_mm_slot only holds mm_count, which only
> guarantees the mm_strcut isn't freed. So, __ksm_exit() uses mmap write lock to synchronize
> with ksmd.
> 
> IIUC, vma_read_lock cannot be exclusive with mmap_write_lock().
>

Nice catch. Thanks for pointing this out. Indeed, the original exclusion between
__ksm_exit() and ksmd relied on mmap_write_lock() blocking mmap_read_lock(),
and per-VMA read locks do not provide that exclusion.

A possible approach to restore the necessary guarantee is to pin mm_users while
ksmd is walking the page tables:

Before scanning a given mm, try to take a reference with mmget_not_zero(mm).
If it fails, the mm is exiting, so we skip it.

Hold that reference for the entire duration of scanning that mm (not per-VMA),
and drop it with mmput() when done.

On the fallback path where we need to acquire mmap_read_lock(), drop the mm_users
reference before waiting, to avoid delaying an exiting mm.

This directly guarantees that mm_users > 0 while ksmd is accessing the page tables,
so __mmput() cannot reach exit_mmap() and free them. It is more precise than the
old mmap_write_lock() synchronization and should not introduce noticeable delay,
since the reference is only held for the scan duration.

Re: [PATCH RFC 0/3] mm/ksm: scan with per-VMA locks

[xu.xin16]

> From: Longlong Xia <xialonglong@kylinos.cn>
> 
> This work was developed independently before I saw the above series.
> After comparing the two implementations, I found that the approaches
> overlap, so I am posting this draft version as an alternative 
> implementation and would be happy to combine the useful parts.
> 
>   The main differences are:
> 
>   - use per-VMA locking in the ksmd VMA scanner;
>   - revalidate the mmap-lock sequence before skipping the fallback walk;
>   - avoid repeating a completed VMA traversal when no candidate page is
>     found.
> 
> Benchmark (QEMU q35, TCG multi-thread, 4 vCPUs, 8 GiB, x86_64 guest on
> Ubuntu 24.04, 7.3-rc2 base, CONFIG_KSM=y + CONFIG_PER_VMA_LOCK=y;
> victim: 32 MiB anon, 2048 unique pages x4 duplicates,
> pages_to_scan=100000, sleep_millisecs=0, 3 runs each; churners: 4
> threads mmap/munmaping 256 MiB in the same mm. Microbenchmark as
> posted in [3]; relative numbers only, TCG is not native hardware):
> 
>                        0 churners      4 churners (mean)   
>   7.3-rc2              1.21-1.41 s     90.9 s             
>   + patch 2/3          1.41 s          48.7 s   (-46%)   
>   + patch 2/3 + 3/3    1.21-1.41 s     30.1 s   (-67%)  
> 
> 
>  1. mm/pagewalk: allow folio_walk_start() under a vma read lock
> 
>  2. mm/ksm: use the VMA lock when looking up mergeable pages
> 
>  3. mm/ksm: scan VMAs with per-VMA locks
> 

it seems your patches may not be rebased on the latest mm-unstable or linux-next. 
It might also be worth double-checking any AI-generated code before sending it out.

I also noticed that the content of patches 1 and 2 is already included in my patch series.
Perhaps it would be better to focus on my series for those and I'd appreciate your help
reviewing them. For patch 3, feel free to start a separate email thread if that works better.

Thanks

Re: [PATCH 3/4] mm/ksm: make break_ksm() more scalable

[Matthew Wilcox]

On Fri, Sep 11, 2026 at 04:12:10PM +0800, xu.xin16@zte.com.cn wrote:
> +++ b/mm/ksm.c
> @@ -660,16 +660,11 @@ static int break_ksm_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned long en
>  	return found;
>  }
> 
> -static const struct mm_walk_ops break_ksm_ops = {
> +static struct mm_walk_ops break_ksm_ops = {
>  	.pmd_entry = break_ksm_pmd_entry,
>  	.walk_lock = PGWALK_RDLOCK,
>  };

Absolutely not.  This is a struct full of function pointers, and
you're making it modifiable.  That makes it a big juicy target for
attackers to find and exploit.