[PATCH v2 0/9] KVM: don't zap secondary MMU on permission upgrade

[PATCH v2 0/9] KVM: don't zap secondary MMU on permission upgrade

[Carlos López]

Stop KVM from zapping its mappings on permission upgrades on x86 and
s390, as these keep the PFNs unchanged and revoke no access. KVM faults
in the new permissions on demand, so dropping mappings early gains
nothing.

To do so, add an MMU notifier flag that marks an
MMU_NOTIFY_PROTECTION_VMA invalidation as a strict permission upgrade.
For the flag to be accurate, change_protection() keeps writable entries
writable on such changes. A few cases where it needlessly leaves entries
read-only are fixed along the way.

Background
==========

change_protection() pairs every protection change with an
MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in
response, as it cannot tell whether permissions widened or narrowed.
This is costly, e.g. during a QEMU background snapshot, which uses
uffd-wp to write-protect guest RAM and unprotects each page once it is
saved. Every unprotect then makes the guest refault memory it could
already access.

Secondary MMUs that share the CPU page tables (e.g. ARM SMMU,
IOMMU/ATS) can keep read-only entries in their TLBs that a fault does
not flush, so they are already notified of permission upgrades on the
CPU TLB flush path [1]. KVM keeps its own page tables and faults in
permissions on demand, so it needs no notification, as Sean noted when
that signalling was first proposed [2].

Note: the notification cannot simply be skipped in MM code (see [3]).
Secondary MMUs that keep their own page tables but cannot fault rely on
it to pick up the new permissions. A flag lets each user opt in instead.

Series overview
===============

Patches 1-3 fix cases where change_protection() leaves entries
read-only even though the write fault handler would make them writable
without further checks. They are performance fixes on their own and
can be picked up independently.

Patches 4-6 add MM_CP_PERM_UPGRADE for mprotect() calls that only add
permissions, keep writable entries writable on these and on userfaultfd
unprotects, and add MMU_NOTIFIER_RANGE_PERM_UPGRADE.

Patches 7-9 pass the MMU notifier event and flags to KVM and skip the
zap on x86 and s390.

Considerations
==============

1. This series covers only x86 and s390. Claudio confirmed that
   skipping the zap should be safe on s390 and offered to test it [4],
   so patch 9 is only compile-tested. Other architectures should be
   able to skip it as well, but I have not tested or benchmarked
   them. Moving the code to arch-independent KVM would also avoid the
   SEV cache flush in kvm_arch_guest_memory_reclaimed().

2. struct kvm_gfn_range carries the MMU notifier event and flags, as
   agreed at LPC [5] and proposed in the past for other purposes
   [6] [7]. The fields are only set for MMU notifier invalidations.

3. Writable migration and device-private entries are still made
   read-only on a permission upgrade. This does not affect secondary
   MMUs (see patch 6), but the downgrade could be skipped as well if
   preferred.

4. hugetlb does not benefit, as hugetlb_change_protection() does
   not restore the write bit and may unshare page tables (see patch 6).

5. The notification cannot simply be skipped in MM code (see [3]).
   Secondary MMUs that keep their own page tables but cannot fault
   rely on it to pick up new permissions. A flag lets each user opt
   in instead.

Benchmarks
==========

These were measured on v1.

AMD EPYC 9654 96-Core Processor host running a recent Linux kernel
(6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while
the host takes a QEMU background snapshot. Results are aggregated over
5 runs on a stock kernel and on one with this series, rebooting the
guest between runs, with NUMA balancing disabled.

pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush
debugfs stats):

  guest pattern          baseline                    patched
  -----------------------------------------------------------------------
  read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
  rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
  write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3

Guest read latency was sampled with a paced probe thread in the guest,
reading one byte per page with an idle gap between reads, while
stress-ng ran with read64. Each figure is the median across 5 runs, with
the run-to-run min and max, in ns:

  metric           baseline             patched
  ------------------------------------------------------
  p50 (min max)      340     (270 380)   260   (250 260)
  p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
  p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)

The median improves modestly, while p99 drops ~3.8x.

A separate stress-ng run with write64 and --verify showed no data
corruption and no livelocks.

Changes in v2
=============

- New patch 1: fix setting MM_CP_TRY_CHANGE_WRITABLE for shmem with
  uffd-wp or soft-dirty (Sashiko).
- New patches 2-3: honor MM_CP_TRY_CHANGE_WRITABLE in change_huge_pud(),
  adding pud_soft_dirty() on riscv (Sashiko).
- Patch 4 (was 1): only treat changes to the read, write and execute
  flags as an upgrade, plus VM_ACCOUNT (Sashiko).
- New patch 5: keep writable entries writable on permission upgrades
  (Sashiko).
- Patch 6 (was 2): don't set the flag for hugetlb (Sashiko).
- Patch 7 (was 3): pass the MMU notifier event and flags instead of a
  bool (Claudio).
- Patch 8 (was 4): check the flag in range->flags. The APIC-access page
  reload issue that Sashiko pointed out is pre-existing.
- New patch 9: skip the unmap on s390 as well (Claudio).

v1: https://lore.kernel.org/all/20261029143336.390231-3-clopez@suse.de/

References
==========

[1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
[2] https://lore.kernel.org/linux-mm/ZGu1vsbscdO48V6h@google.com/
[3] https://lore.kernel.org/all/7ba2ca65-1eab-40fd-ae4c-f7cd9e156d8c@suse.de/
[4] https://lore.kernel.org/all/20261030133939.630e29af@p-imbrenda/
[5] https://lpc.events/event/20/contributions/2552/
[6] https://lore.kernel.org/all/20240801183453.57199-1-seanjc@google.com
[7] https://lore.kernel.org/all/20231110003734.1014084-1-jackyli@google.com

Carlos López (9):
  mm: fix eager write-bit restore for write-notified shared mappings
  riscv: mm: add pud_soft_dirty()
  mm/huge_memory: honor MM_CP_TRY_CHANGE_WRITABLE in change_huge_pud()
  mm/mprotect: add MM_CP_PERM_UPGRADE
  mm/mprotect: keep writable entries writable on permission upgrades
  mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
  KVM: pass the mmu_notifier event and flags to kvm_gfn_range
  KVM: x86/mmu: don't zap secondary MMU on permission upgrade
  KVM: s390: don't unmap the gmap on permission upgrade

 arch/riscv/include/asm/pgtable.h |  5 +++++
 arch/s390/kvm/arm64/arm.c        |  7 +++++++
 arch/s390/kvm/s390/s390.c        |  7 +++++++
 arch/x86/kvm/mmu/mmu.c           |  7 +++++++
 include/linux/kvm_host.h         |  3 +++
 include/linux/mm.h               |  2 ++
 include/linux/mmu_notifier.h     |  5 +++++
 include/linux/pgtable.h          |  5 +++++
 mm/huge_memory.c                 | 35 +++++++++++++++++++++++++++++++-
 mm/hugetlb.c                     |  4 ++++
 mm/internal.h                    |  7 +++++++
 mm/mprotect.c                    | 22 ++++++++++++++++++--
 mm/vma.h                         | 23 +++++++++++++--------
 virt/kvm/kvm_main.c              |  8 ++++++++
 14 files changed, 128 insertions(+), 12 deletions(-)

-- 
2.51.0

[PATCH v2 1/9] mm: fix eager write-bit restore for write-notified shared mappings

[Carlos López]

change_protection() applies vma->vm_page_prot, which lacks
the write bit for CoW and write-notified mappings. To avoid
needless write faults, callers set MM_CP_TRY_CHANGE_WRITABLE when
vma_wants_manual_pte_write_upgrade() returns true, so that entries that
can be written without a fault, such as dirty PTEs in a shared mapping,
are made writable right away.

For shared mappings, that function passes vma->vm_page_prot to
vma_wants_writenotify(). Once writenotify is enabled, this protection
lacks the write bit, while the vma_pgprot_modify() result it is compared
against adds it back. The two never match, so vma_wants_writenotify()
returns false before checking for soft-dirty, uffd-wp and writeback.

As a result, write-notified shmem mappings never get
MM_CP_TRY_CHANGE_WRITABLE, and their dirty PTEs stay read-only after a
protection change. For example, commit 931298e103c2 ("mm/userfaultfd:
rely on vma->vm_page_prot in uffd_wp_range()") made uffd-wp unprotect
set MM_CP_TRY_CHANGE_WRITABLE so that dirty shmem PTEs become writable
again, but this never takes effect. Unprotecting N dirty shmem pages
still leads to N faults that only set the write bit.

This dates back to the commit in the Fixes tag, which moved the
equivalent check in mprotect_fixup() before vma_set_page_prot().

Fix this by passing the protection recomputed from the VMA flags, as
vma_set_page_prot() does. Note that this is a performance-only fix, as
the write bit is restored on the next page fault anyway.

Fixes: 64e455079e1b ("mm: softdirty: enable write notifications on VMAs after VM_SOFTDIRTY cleared")
Signed-off-by: Carlos López <clopez@suse.de>
---
 mm/vma.h | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/mm/vma.h b/mm/vma.h
index 3fc3b47a954b..e83700e248e4 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -589,6 +589,14 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	      unsigned long addr, int new_below);
 #endif
 
+#ifdef CONFIG_MMU
+static inline pgprot_t vma_pgprot_modify(pgprot_t oldprot, vma_flags_t vma_flags)
+{
+	const pgprot_t prot = vma_flags_to_page_prot(vma_flags);
+
+	return pgprot_modify(oldprot, prot);
+}
+
 static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma)
 {
 	/*
@@ -596,19 +604,16 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma
 	 * if we can't do that automatically for all PTEs in a mapping. For
 	 * private mappings, that's always the case when we have write
 	 * permissions as we properly have to handle COW.
+	 *
+	 * vma->vm_page_prot lacks the write bit if writenotify is already
+	 * enabled, so pass it with the bits restored, as vma_set_page_prot()
+	 * does.
 	 */
 	if (vma->vm_flags & VM_SHARED)
-		return vma_wants_writenotify(vma, vma->vm_page_prot);
+		return vma_wants_writenotify(vma,
+		       vma_pgprot_modify(vma->vm_page_prot, vma->flags));
 	return !!(vma->vm_flags & VM_WRITE);
 }
-
-#ifdef CONFIG_MMU
-static inline pgprot_t vma_pgprot_modify(pgprot_t oldprot, vma_flags_t vma_flags)
-{
-	const pgprot_t prot = vma_flags_to_page_prot(vma_flags);
-
-	return pgprot_modify(oldprot, prot);
-}
 #endif
 
 static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi,
-- 
2.51.0

[PATCH v2 2/9] riscv: mm: add pud_soft_dirty()

[Carlos López]

riscv supports soft-dirty and PUD-sized transparent huge pages, but only
provides pmd_soft_dirty() and not pud_soft_dirty(). Add it, so generic
code can check the soft-dirty bit of a huge PUD.

Signed-off-by: Carlos López <clopez@suse.de>
---
 arch/riscv/include/asm/pgtable.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 2ebd5fa5adc5..40890f51dec6 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -949,6 +949,11 @@ static inline pmd_t pmd_clear_soft_dirty(pmd_t pmd)
 	return pte_pmd(pte_clear_soft_dirty(pmd_pte(pmd)));
 }
 
+static inline bool pud_soft_dirty(pud_t pud)
+{
+	return pte_soft_dirty(pud_pte(pud));
+}
+
 #ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
 static inline bool pmd_swp_soft_dirty(pmd_t pmd)
 {
-- 
2.51.0

[PATCH v2 3/9] mm/huge_memory: honor MM_CP_TRY_CHANGE_WRITABLE in change_huge_pud()

[Carlos López]

When change_protection() is called on a range whose vma->vm_page_prot
does not contain the write bit (MAP_PRIVATE mappings, which need CoW
handling, or MAP_SHARED mappings that want write tracking), the new
entries are created without the write bit, letting the fault handler
re-introduce it. In such cases, MM_CP_TRY_CHANGE_WRITABLE is set to
eagerly restore the W bit in the page table entries when stripping the
write protection was not needed.

However, change_huge_pud() does not currently respect the flag. The only
user of huge PUDs, device DAX, is never CoW and does not need write
notifications for dirty tracking, but it does get them once soft-dirty
tracking is armed through /proc/pid/clear_refs. vm_page_prot then lacks
the write bit, and a protection change leaves dirty PUDs read-only, so
the next write takes a fault only to set the write bit again.

Align change_huge_pud() with change_pte_range() and change_huge_pmd() by
making it respect MM_CP_TRY_CHANGE_WRITABLE. Besides avoiding those
faults, this keeps huge PUDs consistent with the other levels if PUDs
that need CoW, such as anonymous ones, are ever introduced. They would
otherwise lose the write bit on every protection change, even where the
write fault handler would restore it without further checks.

Do this by introducing can_change_pud_writable(), which mimics
can_change_pmd_writable(). Omit the NUMA and uffd-wp checks, which
change_huge_pud() already rejects.

Signed-off-by: Carlos López <clopez@suse.de>
---
 include/linux/pgtable.h |  5 +++++
 mm/huge_memory.c        | 28 +++++++++++++++++++++++++++-
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 780fe849ff8b..943d7e95f426 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1866,6 +1866,11 @@ static inline int pmd_soft_dirty(pmd_t pmd)
 	return 0;
 }
 
+static inline int pud_soft_dirty(pud_t pud)
+{
+	return 0;
+}
+
 static inline pte_t pte_mksoft_dirty(pte_t pte)
 {
 	return pte;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index bd44d58e8684..506e1e61c0fd 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2824,6 +2824,29 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	return ret;
 }
 
+#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
+static inline bool can_change_pud_writable(struct vm_area_struct *vma,
+					   unsigned long addr, pud_t pud)
+{
+	struct page *page;
+
+	if (WARN_ON_ONCE(!(vma->vm_flags & VM_WRITE)))
+		return false;
+
+	/* Do we need write faults for softdirty tracking? */
+	if (vma_soft_dirty_enabled(vma) && !pud_soft_dirty(pud))
+		return false;
+
+	if (!(vma->vm_flags & VM_SHARED)) {
+		/* See can_change_pte_writable(). */
+		page = vm_normal_page_pud(vma, addr, pud);
+		return page && PageAnon(page) && PageAnonExclusive(page);
+	}
+
+	/* See can_change_pte_writable(). */
+	return pud_dirty(pud);
+}
+
 /*
  * Returns:
  *
@@ -2831,7 +2854,6 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
  * - 1: if pud can be skipped
  * - HPAGE_PUD_NR: if pud was successfully processed
  */
-#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
 int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		    pud_t *pudp, unsigned long addr, pgprot_t newprot,
 		    unsigned long cp_flags)
@@ -2863,6 +2885,10 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	 */
 	oldpud = pudp_invalidate(vma, addr, pudp);
 	entry = pud_modify(oldpud, newprot);
+	/* See change_pte_range(). */
+	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pud_write(entry) &&
+	    can_change_pud_writable(vma, addr, entry))
+		entry = pud_mkwrite(entry);
 	set_pud_at(mm, addr, pudp, entry);
 	tlb_flush_pud_range(tlb, addr, HPAGE_PUD_SIZE);
 
-- 
2.51.0

[PATCH v2 4/9] mm/mprotect: add MM_CP_PERM_UPGRADE

[Carlos López]

Add a change_protection() flag for changes that only add permissions,
keeping the PFN and revoking no access. mprotect() sets it when the new
VMA flags only add read, write or execute permissions. VM_ACCOUNT may be
added as well, as it does not affect the page tables. Any other change,
such as a different protection key, is not considered an upgrade.

This will allow MMU notifier users to perform optimizations, as they
can avoid invalidating PTEs whose permissions can be faulted in.

Signed-off-by: Carlos López <clopez@suse.de>
---
 include/linux/mm.h |  2 ++
 mm/mprotect.c      | 13 +++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9f167950926a..dddbd76a2760 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3581,6 +3581,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 #define  MM_CP_UFFD_RWP_RESOLVE            (1UL << 5) /* resolve rwp */
 #define  MM_CP_UFFD_RWP_ALL                (MM_CP_UFFD_RWP | \
 					    MM_CP_UFFD_RWP_RESOLVE)
+/* Whether this change only upgrades/adds permissions */
+#define  MM_CP_PERM_UPGRADE                (1UL << 6)
 
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 754333698c1e..683131c9c4a0 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -756,6 +756,17 @@ static const struct mm_walk_ops prot_none_walk_ops = {
 	.walk_lock		= PGWALK_WRLOCK,
 };
 
+static bool vma_flags_is_perm_upgrade(vma_flags_t old_vma_flags,
+				      vma_flags_t new_vma_flags)
+{
+	vm_flags_t old = vma_flags_to_legacy(old_vma_flags);
+	vm_flags_t new = vma_flags_to_legacy(new_vma_flags);
+	/* VM_ACCOUNT does not affect page tables */
+	vm_flags_t mask = VM_READ | VM_WRITE | VM_EXEC | VM_ACCOUNT;
+
+	return new == (old | (new & mask));
+}
+
 int
 mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	       struct vm_area_struct *vma, struct vm_area_struct **pprev,
@@ -836,6 +847,8 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	vma_flags_reset_once(vma, &new_vma_flags);
 	if (vma_wants_manual_pte_write_upgrade(vma))
 		mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
+	if (vma_flags_is_perm_upgrade(old_vma_flags, new_vma_flags))
+		mm_cp_flags |= MM_CP_PERM_UPGRADE;
 	vma_set_page_prot(vma);
 
 	change_protection(tlb, vma, start, end, mm_cp_flags);
-- 
2.51.0

[PATCH v2 5/9] mm/mprotect: keep writable entries writable on permission upgrades

[Carlos López]

change_protection() applies vma->vm_page_prot, which lacks the write bit
for CoW and write-notified mappings, so it write-protects every entry
in them. When an entry would simply be made writable on fault without
further checks (e.g. dirty PTEs in shared mappings, exclusive anonymous
pages), MM_CP_TRY_CHANGE_WRITABLE makes the entry writable again to
avoid the future fault.

However, writable entries in shared mappings that are not dirty still
become read-only. For example, a uffd-wp unprotect makes clean shmem
PTEs read-only even if they were never write-protected, e.g. when they
were mapped by a read fault before uffd-wp was registered. The next
write to each of them causes a fault that only restores the write bit.

Now that MM_CP_PERM_UPGRADE indicates whether mprotect() only adds
permissions, change_protection() can tell, together with the
userfaultfd unprotect flags, when a change revokes no access. Keep
writable entries writable in these cases, at the PTE, PMD and PUD
levels. This is consistent with the write fault handler, as an entry
that is already writable needs no write fault.

Signed-off-by: Carlos López <clopez@suse.de>
---
 mm/huge_memory.c | 7 +++++++
 mm/internal.h    | 7 +++++++
 mm/mprotect.c    | 4 ++++
 3 files changed, 18 insertions(+)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 506e1e61c0fd..6e23d0e4ae66 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2809,6 +2809,10 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	if (userfaultfd_rwp(vma) && pmd_uffd(entry))
 		entry = pmd_modify(entry, PAGE_NONE);
 
+	/* See change_present_ptes(). */
+	if (mm_cp_is_perm_upgrade(cp_flags) && pmd_write(oldpmd))
+		entry = pmd_mkwrite(entry, vma);
+
 	/* See change_pte_range(). */
 	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
 	    can_change_pmd_writable(vma, addr, entry))
@@ -2885,6 +2889,9 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	 */
 	oldpud = pudp_invalidate(vma, addr, pudp);
 	entry = pud_modify(oldpud, newprot);
+	/* See change_present_ptes(). */
+	if (mm_cp_is_perm_upgrade(cp_flags) && pud_write(oldpud))
+		entry = pud_mkwrite(entry);
 	/* See change_pte_range(). */
 	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pud_write(entry) &&
 	    can_change_pud_writable(vma, addr, entry))
diff --git a/mm/internal.h b/mm/internal.h
index 566cbd2dce48..a1a66f3b4b9c 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1423,6 +1423,13 @@ static inline bool pte_needs_soft_dirty_wp(struct vm_area_struct *vma, pte_t pte
 	return vma_soft_dirty_enabled(vma) && !pte_soft_dirty(pte);
 }
 
+/* Whether a change_protection() only adds permissions. */
+static inline bool mm_cp_is_perm_upgrade(unsigned long cp_flags)
+{
+	return cp_flags & (MM_CP_PERM_UPGRADE | MM_CP_UFFD_WP_RESOLVE |
+			   MM_CP_UFFD_RWP_RESOLVE);
+}
+
 /* shrinker related functions */
 unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg,
 			  int priority);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 683131c9c4a0..16b8aef755ab 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -306,6 +306,10 @@ static __always_inline void change_present_ptes(struct mmu_gather *tlb,
 	if (userfaultfd_rwp(vma) && pte_uffd(ptent))
 		ptent = pte_modify(ptent, PAGE_NONE);
 
+	/* A permission upgrade keeps writable entries writable. */
+	if (mm_cp_is_perm_upgrade(cp_flags) && pte_write(oldpte))
+		ptent = pte_mkwrite(ptent, vma);
+
 	/*
 	 * In some writable, shared mappings, we might want
 	 * to catch actual write access -- see
-- 
2.51.0

Re: [PATCH v2 5/9] mm/mprotect: keep writable entries writable on permission upgrades

[David Hildenbrand (Arm)]

On 11/5/26 17:50, Carlos López wrote:
> change_protection() applies vma->vm_page_prot, which lacks the write bit
> for CoW and write-notified mappings, so it write-protects every entry
> in them. When an entry would simply be made writable on fault without
> further checks (e.g. dirty PTEs in shared mappings, exclusive anonymous
> pages), MM_CP_TRY_CHANGE_WRITABLE makes the entry writable again to
> avoid the future fault.
> 
> However, writable entries in shared mappings that are not dirty still
> become read-only. For example, a uffd-wp unprotect makes clean shmem
> PTEs read-only even if they were never write-protected, e.g. when they
> were mapped by a read fault before uffd-wp was registered. The next
> write to each of them causes a fault that only restores the write bit.
> 
> Now that MM_CP_PERM_UPGRADE indicates whether mprotect() only adds
> permissions, change_protection() can tell, together with the
> userfaultfd unprotect flags, when a change revokes no access. Keep
> writable entries writable in these cases, at the PTE, PMD and PUD
> levels. This is consistent with the write fault handler, as an entry
> that is already writable needs no write fault.
> 
> Signed-off-by: Carlos López <clopez@suse.de>
> ---
>  mm/huge_memory.c | 7 +++++++
>  mm/internal.h    | 7 +++++++
>  mm/mprotect.c    | 4 ++++
>  3 files changed, 18 insertions(+)
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 506e1e61c0fd..6e23d0e4ae66 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2809,6 +2809,10 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  	if (userfaultfd_rwp(vma) && pmd_uffd(entry))
>  		entry = pmd_modify(entry, PAGE_NONE);
>  
> +	/* See change_present_ptes(). */
> +	if (mm_cp_is_perm_upgrade(cp_flags) && pmd_write(oldpmd))
> +		entry = pmd_mkwrite(entry, vma);
> +
>  	/* See change_pte_range(). */
>  	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
>  	    can_change_pmd_writable(vma, addr, entry))
> @@ -2885,6 +2889,9 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  	 */
>  	oldpud = pudp_invalidate(vma, addr, pudp);
>  	entry = pud_modify(oldpud, newprot);
> +	/* See change_present_ptes(). */
> +	if (mm_cp_is_perm_upgrade(cp_flags) && pud_write(oldpud))
> +		entry = pud_mkwrite(entry);

That looks completely hacked on. We do have can_change_pud_writable() below to
make such decisions.

Whaatever you propose here is either wrong or way too shaky, sorry.

-- 
Cheers,

David

Re: [PATCH v2 5/9] mm/mprotect: keep writable entries writable on permission upgrades

[Carlos López]

On 11/5/26 6:06 PM, David Hildenbrand (Arm) wrote:
> On 11/5/26 17:50, Carlos López wrote:
>> change_protection() applies vma->vm_page_prot, which lacks the write bit
>> for CoW and write-notified mappings, so it write-protects every entry
>> in them. When an entry would simply be made writable on fault without
>> further checks (e.g. dirty PTEs in shared mappings, exclusive anonymous
>> pages), MM_CP_TRY_CHANGE_WRITABLE makes the entry writable again to
>> avoid the future fault.
>>
>> However, writable entries in shared mappings that are not dirty still
>> become read-only. For example, a uffd-wp unprotect makes clean shmem
>> PTEs read-only even if they were never write-protected, e.g. when they
>> were mapped by a read fault before uffd-wp was registered. The next
>> write to each of them causes a fault that only restores the write bit.
>>
>> Now that MM_CP_PERM_UPGRADE indicates whether mprotect() only adds
>> permissions, change_protection() can tell, together with the
>> userfaultfd unprotect flags, when a change revokes no access. Keep
>> writable entries writable in these cases, at the PTE, PMD and PUD
>> levels. This is consistent with the write fault handler, as an entry
>> that is already writable needs no write fault.
>>
>> Signed-off-by: Carlos López <clopez@suse.de>
>> ---
>>  mm/huge_memory.c | 7 +++++++
>>  mm/internal.h    | 7 +++++++
>>  mm/mprotect.c    | 4 ++++
>>  3 files changed, 18 insertions(+)
>>
>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
>> index 506e1e61c0fd..6e23d0e4ae66 100644
>> --- a/mm/huge_memory.c
>> +++ b/mm/huge_memory.c
>> @@ -2809,6 +2809,10 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>>  	if (userfaultfd_rwp(vma) && pmd_uffd(entry))
>>  		entry = pmd_modify(entry, PAGE_NONE);
>>  
>> +	/* See change_present_ptes(). */
>> +	if (mm_cp_is_perm_upgrade(cp_flags) && pmd_write(oldpmd))
>> +		entry = pmd_mkwrite(entry, vma);
>> +
>>  	/* See change_pte_range(). */
>>  	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
>>  	    can_change_pmd_writable(vma, addr, entry))
>> @@ -2885,6 +2889,9 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
>>  	 */
>>  	oldpud = pudp_invalidate(vma, addr, pudp);
>>  	entry = pud_modify(oldpud, newprot);
>> +	/* See change_present_ptes(). */
>> +	if (mm_cp_is_perm_upgrade(cp_flags) && pud_write(oldpud))
>> +		entry = pud_mkwrite(entry);
> 
> That looks completely hacked on. We do have can_change_pud_writable() below to
> make such decisions.

Okay, for v3 I can pass the old entry to can_change_{pmd,pud}_writable()
and return true if it was writable, after the existing write-tracking
checks (uffd-wp/soft-dirty). For PTEs, can_change_pte_writable() is not
used on the change_protection() path, so the same check would go in
set_write_prot_commit_flush_ptes().

Does that sound more reasonable?

> Whaatever you propose here is either wrong or way too shaky, sorry.

The idea is that MM_CP_TRY_CHANGE_WRITABLE relies on the dirty bit to
make entries in shared mappings writable, which misses clean entries
that were already writable (e.g. shmem read-faulted before uffd-wp
registration). Such entries can stay writable, as anything that relies
on write faults write-protects the entry first.

[PATCH v2 6/9] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

[Carlos López]

Add a range flag marking an invalidation that only upgrades permissions,
with no PFN change and no revoked access. change_protection() sets it
for MM_CP_PERM_UPGRADE and for the userfaultfd resolves, which unprotect
a page and thus only restore access.

Now that change_protection() keeps writable entries writable on these
changes, no present entry loses access. Writable migration and
device-private entries are still made read-only, which does not affect
secondary MMUs. Migration entries are only installed after secondary
MMUs are invalidated, and device-private pages are migrated back, with
an invalidation, before the CPU can access them.

hugetlb_change_protection() never sets the flag, as hugetlb does not
restore the write bit and may unshare page tables.

A secondary MMU can use the flag to keep its mappings rather than zap
them. Consumers that ignore it are unaffected.

Signed-off-by: Carlos López <clopez@suse.de>
---
 include/linux/mmu_notifier.h | 5 +++++
 mm/hugetlb.c                 | 4 ++++
 mm/mprotect.c                | 5 +++--
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h
index a11a44eef521..112be28cdf6b 100644
--- a/include/linux/mmu_notifier.h
+++ b/include/linux/mmu_notifier.h
@@ -60,6 +60,11 @@ enum mmu_notifier_event {
 };
 
 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)
+/*
+ * Set on an invalidation that only expands permissions without
+ * revoking any existing access, nor freeing/moving the underlying page.
+ */
+#define MMU_NOTIFIER_RANGE_PERM_UPGRADE (1 << 1)
 
 struct mmu_notifier_ops {
 	/*
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index da5d04edf7c1..1a69d1da527e 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -6606,6 +6606,10 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 	 * In the case of shared PMDs, the area to flush could be beyond
 	 * start/end.  Set range.start/range.end to cover the maximum possible
 	 * range if PMD sharing is possible.
+	 *
+	 * MMU_NOTIFIER_RANGE_PERM_UPGRADE is never set here. hugetlb does not
+	 * restore the write bit, and huge_pmd_unshare() may drop the mapping,
+	 * so this is not a strict permission upgrade.
 	 */
 	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
 				0, mm, start, end);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 16b8aef755ab..fb2023dc6e66 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -601,8 +601,9 @@ static inline long change_pud_range(struct mmu_gather *tlb,
 			continue;
 
 		if (!range.start) {
-			mmu_notifier_range_init(&range,
-						MMU_NOTIFY_PROTECTION_VMA, 0,
+			mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
+						mm_cp_is_perm_upgrade(cp_flags) ?
+						MMU_NOTIFIER_RANGE_PERM_UPGRADE : 0,
 						vma->vm_mm, addr, end);
 			mmu_notifier_invalidate_range_start(&range);
 		}
-- 
2.51.0

[PATCH v2 7/9] KVM: pass the mmu_notifier event and flags to kvm_gfn_range

[Carlos López]

Propagate the event and flags of the mmu_notifier invalidation into
struct kvm_gfn_range, so that architecture handlers can tell what kind
of change they are handling, such as a permission upgrade
(MMU_NOTIFIER_RANGE_PERM_UPGRADE). They are not consumed yet, so no
functional change.

Signed-off-by: Carlos López <clopez@suse.de>
---
 include/linux/kvm_host.h | 3 +++
 virt/kvm/kvm_main.c      | 8 ++++++++
 2 files changed, 11 insertions(+)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e7fc94e2910a..1f5ce4410ddb 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -274,6 +274,9 @@ struct kvm_gfn_range {
 	enum kvm_gfn_range_filter attr_filter;
 	bool may_block;
 	bool lockless;
+	/* From the mmu_notifier invalidation, if any. */
+	enum mmu_notifier_event event;
+	unsigned int flags;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index cf4cb586f140..c517d1deb488 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -532,6 +532,8 @@ struct kvm_mmu_notifier_range {
 	bool flush_on_ret;
 	bool may_block;
 	bool lockless;
+	enum mmu_notifier_event event;
+	unsigned int flags;
 };
 
 /*
@@ -613,6 +615,8 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			 */
 			gfn_range.arg = range->arg;
 			gfn_range.may_block = range->may_block;
+			gfn_range.event = range->event;
+			gfn_range.flags = range->flags;
 			/*
 			 * HVA-based notifications aren't relevant to private
 			 * mappings as they don't have a userspace mapping.
@@ -738,6 +742,8 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 		.on_lock	= kvm_mmu_invalidate_start,
 		.flush_on_ret	= true,
 		.may_block	= mmu_notifier_range_blockable(range),
+		.event		= range->event,
+		.flags		= range->flags,
 	};
 
 	trace_kvm_unmap_hva_range(range->start, range->end);
@@ -2552,6 +2558,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 
 	gfn_range.arg = range->arg;
 	gfn_range.may_block = range->may_block;
+	gfn_range.event = range->event;
+	gfn_range.flags = range->flags;
 
 	/*
 	 * If/when KVM supports more attributes beyond private .vs shared, this
-- 
2.51.0

[PATCH v2 8/9] KVM: x86/mmu: don't zap secondary MMU on permission upgrade

[Carlos López]

When an mmu_notifier invalidation only upgrades permissions
(MMU_NOTIFIER_RANGE_PERM_UPGRADE), keep the existing SPTEs instead of
zapping them. The PFN is unchanged and no access is revoked, so an
existing SPTE is no more permissive than the new host PTE. Permissions
for the SPTE are updated on the next fault, so dropping them early
gains nothing.

The motivating case is a userfaultfd-wp dirty tracker such as QEMU's
background snapshot, which unprotects every guest page as it saves it.
Zapping on each unprotect makes the guest refault its whole working
set. On a 64GB guest under a background snapshot, a read-mostly workload
drops secondary-MMU refaults from ~1M/s to ~2.5K/s and cross-vCPU TLB
shootdowns from ~1400/s to near zero. Uses of mprotect() that only add
access benefit likewise.

Note that this also skips the APIC-access page reload, which is not
needed as the page's PFN is unchanged. KVM does not retry a failed
reload, but that issue is pre-existing and unrelated to this change.

Signed-off-by: Carlos López <clopez@suse.de>
---
 arch/x86/kvm/mmu/mmu.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index f89e9f8a6cc2..3c92379bdbe0 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -1719,6 +1719,13 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
 			    lockdep_is_held(&kvm->slots_lock));
 
+	/*
+	 * Do not bother zapping SPTEs if this is only a protection upgrade.
+	 * Permissions will be faulted-in on demand.
+	 */
+	if (range->flags & MMU_NOTIFIER_RANGE_PERM_UPGRADE)
+		return false;
+
 	if (kvm_memslots_have_rmaps(kvm))
 		flush = __kvm_rmap_zap_gfn_range(kvm, range->slot,
 						 range->start, range->end,
-- 
2.51.0

[PATCH v2 9/9] KVM: s390: don't unmap the gmap on permission upgrade

[Carlos López]

When an mmu_notifier invalidation only upgrades permissions
(MMU_NOTIFIER_RANGE_PERM_UPGRADE), keep the existing gmap entries
instead of unmapping them. The PFN is unchanged and no access is
revoked, so an existing entry is no more permissive than the new host
PTE.

A write to an entry that is still read-only causes a protection
exception, which kvm_s390_faultin_gfn() handles as a write fault,
picking up the new permissions. This applies to both s390 and arm64
guests.

Signed-off-by: Carlos López <clopez@suse.de>
---
 arch/s390/kvm/arm64/arm.c | 7 +++++++
 arch/s390/kvm/s390/s390.c | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/arch/s390/kvm/arm64/arm.c b/arch/s390/kvm/arm64/arm.c
index bd9d85235f43..5aee1817aad8 100644
--- a/arch/s390/kvm/arm64/arm.c
+++ b/arch/s390/kvm/arm64/arm.c
@@ -678,6 +678,13 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old,
 
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
+	/*
+	 * Do not bother unmapping if this is only a protection upgrade.
+	 * Permissions will be faulted-in on demand.
+	 */
+	if (range->flags & MMU_NOTIFIER_RANGE_PERM_UPGRADE)
+		return false;
+
 	return gmap_unmap_gfn_range(kvm->arch.gmap, range->slot, range->start, range->end);
 }
 
diff --git a/arch/s390/kvm/s390/s390.c b/arch/s390/kvm/s390/s390.c
index 18fb200f854a..954bf6b64683 100644
--- a/arch/s390/kvm/s390/s390.c
+++ b/arch/s390/kvm/s390/s390.c
@@ -5871,6 +5871,13 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
  */
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
+	/*
+	 * Do not bother unmapping if this is only a protection upgrade.
+	 * Permissions will be faulted-in on demand.
+	 */
+	if (range->flags & MMU_NOTIFIER_RANGE_PERM_UPGRADE)
+		return false;
+
 	return gmap_unmap_gfn_range(kvm->arch.gmap, range->slot, range->start, range->end);
 }
 
-- 
2.51.0