[PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Wei-Lin Chang]

Hi,

This is v6 of optimizing the shadow s2 mmu unmapping during MMU
notifiers.

This version fixes a few issues, and adds Marc's space optimization
for kvm_guest_s2_mapping [1]. Please see the changelog for the details.

Tested by booting L3, and running in-kernel targetted tests described
in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
carry them over since some bugs fixed in v6 are non-obvious. Some
retest would be much appreciated!

Series based on v7.3-rc3 + Marc's nested mmu lifecycle fixes [3].

* Changes from v5 [4]:

  - Align the addresses down to the mapping size when recording the
    guest s2 mappings. s2fd->fault_ipa isn't necessarily PAGE_SIZE aligned.

  - Record guest s2 mappings even when page table maps return -EAGAIN.
    kvm_pgtable_stage2_map() can create mappings while returning -EAGAIN.
    For example, a 2M block map (A) could race with a 4K page map (B):
    1. (A) maps the 2M block in kvm_pgtable_visitor_cb()
    2. (B) breaks that block into a table and maps 4K
    3. (A) reloads and finds the table after kvm_pgtable_visitor_cb(),
       then descends into it.
    4. (A) maps some 4K, but before it finishes reads entry mapped by (B).
    5. (A) returns -EAGAIN although it had mapped a few pages.
    In this case, we don't know what subrange is mapped, just track the whole
    requested mapping range.

  - Don't remove tracked mappings from the interval trees if they only
    partially overlap the removal range. Because of the previous bullet point
    we can have a 4K shadow mapping tracked as a 2M range in the interval
    trees. It would be wrong to remove the 2M range when a guest TLBI doesn't
    touch the 4K mapped.

  - Check mmu->pgt during mmu unmap notifier, as it could race against
    MMU teardown.

  - Make guest_s2_tracking_destroy() canonical mmu only. We simply don't
    need to detach the nodes from the nested mmus' trees. Freeing them
    during canonical mmu's teardown is enough.

Thanks!

[1]: https://lore.kernel.org/kvmarm/86h5jv7qrd.wl-maz@kernel.org/
[2]: https://lore.kernel.org/kvmarm/gerjpm62a2gszzggc6vuai22bf3prfquvfsp7ueumps7vz2ev3@odlmovaklg2b/
[3]: https://lore.kernel.org/kvmarm/20260911162203.1919330-1-maz@kernel.org/
[4]: https://lore.kernel.org/kvmarm/20260810205038.118843-1-weilin.chang@arm.com/

Marc Zyngier (1):
  KVM: arm64: nv: Drop kvm_s2_mmu pointer from kvm_guest_s2_mapping

Wei-Lin Chang (6):
  KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map()
  KVM: arm64: nv: Introduce guest stage-2 tracking structures
  KVM: arm64: nv: Track guest stage-2 mapping creation
  KVM: arm64: nv: Track guest stage-2 mapping removal
  KVM: arm64: nv: Avoid full shadow stage-2 unmap
  KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables

 arch/arm64/include/asm/kvm_host.h   |  31 +++++-
 arch/arm64/include/asm/kvm_nested.h |   7 ++
 arch/arm64/kvm/mmu.c                | 109 +++++++++++++++++++---
 arch/arm64/kvm/nested.c             | 140 +++++++++++++++++++++++++++-
 4 files changed, 269 insertions(+), 18 deletions(-)

-- 
2.43.0

[PATCH v6 1/7] KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map()

[Wei-Lin Chang]

Create a variable to store the canonical IPA, instead of calculating it
when needed. This will be useful when we need to use the canonical IPA
for guest stage-2 tracking later.

Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
 arch/arm64/kvm/mmu.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 9ba86450fe4a..671e2941d619 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2021,6 +2021,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED;
 	bool writable = prot & KVM_PGTABLE_PROT_W;
 	struct kvm *kvm = s2fd->vcpu->kvm;
+	phys_addr_t canonical_ipa;
 	struct kvm_pgtable *pgt;
 	long perm_fault_granule;
 	long mapping_size;
@@ -2039,6 +2040,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	mapping_size = s2vi->vma_pagesize;
 	pfn = s2vi->pfn;
 	gfn = s2vi->gfn;
+	canonical_ipa = gfn_to_gpa(get_canonical_gfn(s2fd, s2vi));
 
 	/*
 	 * If we are not forced to use page mapping, check if we are
@@ -2057,6 +2059,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 				goto out_unlock;
 			}
 		}
+		canonical_ipa = ALIGN_DOWN(canonical_ipa, mapping_size);
 	}
 
 	if (!perm_fault_granule && !s2vi->map_non_cacheable && kvm_has_mte(kvm))
@@ -2090,11 +2093,9 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	 * making sure we adjust the canonical IPA if the mapping size has
 	 * been updated (via a THP upgrade, for example).
 	 */
-	if (writable && !ret) {
-		phys_addr_t ipa = gfn_to_gpa(get_canonical_gfn(s2fd, s2vi));
-		ipa &= ~(mapping_size - 1);
-		mark_page_dirty_in_slot(kvm, s2fd->memslot, gpa_to_gfn(ipa));
-	}
+	if (writable && !ret)
+		mark_page_dirty_in_slot(kvm, s2fd->memslot,
+					gpa_to_gfn(canonical_ipa));
 
 	if (ret != -EAGAIN)
 		return ret;
-- 
2.43.0

[PATCH v6 2/7] KVM: arm64: nv: Introduce guest stage-2 tracking structures

[Wei-Lin Chang]

In order to avoid unmapping all shadow stage-2 mappings when KVM
receives a MMU notifier unmap call, we have to keep track of the
canonical IPA -> nested IPA relationship of the shadow mappings
created. This essentially means tracking the guest's stage-2.

To do this, represent each mapping by struct kvm_guest_s2_mapping. It
stores the mapping's canonical IPA range and the nested IPA range using
two interval tree nodes. Both nodes will be inserted into their
respective interval trees called guest_s2_mappings. The canonical IPA
ranges will be stored in the tree within the canonical MMU, and the
nested IPA ranges will be stored in the corresponding nested MMU's tree.

For example:

struct kvm_guest_s2_mapping mapping1, mapping2;

    ---------------------> mapping2.canonical
    |                      mapping1.canonical
    |                          ^   (both stored in canonical mmu's tree)
    |                          |
--*****-----------------------*****----------- CIPA
   \\\\\                      |||||                  mapping1.nested_mmu
    \\\\\                     \\\\\                            |
     \\\\\                     \\\\\                           v
------\\\\\---------------------*****--------- NIPA #1 (nested mmu #1)
       \\\\\                      |
        \\\\\                     -> mapping1.nested
         \\\\\                       (stored in nested mmu #1's tree)
          \\\\\
-----------*****------------------------------ NIPA #2 (nested mmu #2)
             |                                                 ^
             -> mapping2.nested                                |
                (stored in nested mmu #2's tree)   mapping2.nested_mmu

Using the trees we can look up nodes in either of the IPA spaces, and
for each node, find the corresponding range in the other IPA space from
the other node in the enclosing kvm_guest_s2_mapping.

Define kvm_guest_s2_mapping and the interval tree here. Guest stage-2
mapping tracking will come in subsequent patches.

Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
 arch/arm64/include/asm/kvm_host.h | 17 +++++++++++++++++
 arch/arm64/kvm/mmu.c              | 24 ++++++++++++++++++++++++
 arch/arm64/kvm/nested.c           |  1 +
 3 files changed, 42 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index cd9b9d2462f9..365ec57d6d7a 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -14,6 +14,7 @@
 #include <linux/arm-smccc.h>
 #include <linux/bitmap.h>
 #include <linux/types.h>
+#include <linux/interval_tree.h>
 #include <linux/jump_label.h>
 #include <linux/kvm_types.h>
 #include <linux/maple_tree.h>
@@ -150,6 +151,16 @@ struct kvm_vmid {
 	atomic64_t id;
 };
 
+/*
+ * Record of a guest stage-2 mapping, storing canonical and nested IPA
+ * ranges. Both ranges have the same size.
+ */
+struct kvm_guest_s2_mapping {
+	struct interval_tree_node canonical;
+	struct interval_tree_node nested;
+	struct kvm_s2_mmu *nested_mmu;
+};
+
 struct kvm_s2_mmu {
 	struct kvm_vmid vmid;
 
@@ -227,6 +238,9 @@ struct kvm_s2_mmu {
 	 */
 	bool	pending_unmap;
 
+	/* Guest s2 mapping records indexed in this MMU's IPA space. */
+	struct rb_root_cached guest_s2_mappings;
+
 	/*
 	 *  0: Nobody is currently using this, check vttbr for validity
 	 * >0: Somebody is actively using this.
@@ -326,6 +340,9 @@ struct kvm_arch {
 	size_t nested_mmus_size;
 	int nested_mmus_next;
 
+	/* Guest s2 tracking trees access serialization. */
+	spinlock_t guest_s2_tracking_lock;
+
 	/* Interrupt controller */
 	struct vgic_dist	vgic;
 
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 671e2941d619..061cd1e09af2 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -7,6 +7,7 @@
 #include <linux/acpi.h>
 #include <linux/mman.h>
 #include <linux/kvm_host.h>
+#include <linux/interval_tree.h>
 #include <linux/io.h>
 #include <linux/hugetlb.h>
 #include <linux/sched/signal.h>
@@ -1033,6 +1034,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
 
 	mmu->pgd_phys = __pa(pgt->pgd);
 
+	mmu->guest_s2_mappings = RB_ROOT_CACHED;
+
 	if (kvm_is_nested_s2_mmu(kvm, mmu))
 		kvm_init_nested_s2_mmu(mmu);
 
@@ -1122,10 +1125,25 @@ void stage2_unmap_vm(struct kvm *kvm)
 	srcu_read_unlock(&kvm->srcu, idx);
 }
 
+static void guest_s2_tracking_destroy(struct rb_root_cached *tree)
+{
+	struct kvm_guest_s2_mapping *mapping;
+	struct interval_tree_node *node;
+
+	while ((node = interval_tree_iter_first(tree, 0, ULONG_MAX))) {
+		interval_tree_remove(node, tree);
+		mapping = container_of(node, struct kvm_guest_s2_mapping,
+				       canonical);
+		kfree(mapping);
+		cond_resched();
+	}
+}
+
 void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
 {
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 	struct kvm_pgtable *pgt = NULL;
+	struct rb_root_cached mappings_tree;
 
 	write_lock(&kvm->mmu_lock);
 	pgt = mmu->pgt;
@@ -1138,12 +1156,18 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu)
 	if (kvm_is_nested_s2_mmu(kvm, mmu))
 		kvm_init_nested_s2_mmu(mmu);
 
+	mappings_tree = mmu->guest_s2_mappings;
+	mmu->guest_s2_mappings = RB_ROOT_CACHED;
+
 	write_unlock(&kvm->mmu_lock);
 
 	if (pgt) {
 		kvm_stage2_destroy(pgt);
 		kfree(pgt);
 	}
+
+	if (!kvm_is_nested_s2_mmu(kvm, mmu))
+		guest_s2_tracking_destroy(&mappings_tree);
 }
 
 static void hyp_mc_free_fn(void *addr, void *mc)
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index d60f6f69e293..b7bed02e38f7 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -52,6 +52,7 @@ int kvm_init_nested(struct kvm *kvm)
 					      GFP_KERNEL_ACCOUNT);
 	kvm->arch.nested_mmus_size = 0;
 	atomic_set(&kvm->arch.vncr_tlb_count, 0);
+	spin_lock_init(&kvm->arch.guest_s2_tracking_lock);
 
 	return kvm->arch.nested_mmus ? 0 : -ENOMEM;
 }
-- 
2.43.0

[PATCH v6 3/7] KVM: arm64: nv: Track guest stage-2 mapping creation

[Wei-Lin Chang]

During shadow stage-2 faults, in addition to creating mappings in the
shadow page tables, also allocate kvm_guest_s2_mapping objects, record
the mapping ranges, and insert them into the canonical and nested mmu's
guest_s2_mappings tree.

Note that because we allow parallel faulting, the interval trees could
store mappings that are not live in the shadow page tables. Storing a
superset of the live mappings is fine because we will only over-unmap
when we use this information later to do the targeted MMU notifier
unmap.

The mapping is also added to the interval trees if
kvm_pgtable_stage2_map() returns -EAGAIN. This is required for example,
when a 2M block map (A) races with a 4K page map (B):

1. (A) maps the 2M block in kvm_pgtable_visitor_cb()
2. (B) breaks that block into a table and maps 4K
3. (A) reloads and finds the table after kvm_pgtable_visitor_cb(),
   then descends into it.
4. (A) maps some 4K, but before it finishes it reads the entry mapped by
   (B).
5. (A) returns -EAGAIN although it had mapped a few pages.

In this case, we don't know what subrange is mapped, just track the
whole requested mapping range.

Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
 arch/arm64/include/asm/kvm_nested.h |  3 +++
 arch/arm64/kvm/mmu.c                | 39 +++++++++++++++++++++++++++++
 arch/arm64/kvm/nested.c             | 25 ++++++++++++++++++
 3 files changed, 67 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index 586026e85903..2d71f686064a 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -78,6 +78,9 @@ extern void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u16 vmid,
 				       const union tlbi_info *info,
 				       void (*)(struct kvm_s2_mmu *,
 						const union tlbi_info *));
+extern void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
+					gpa_t nested_ipa, size_t map_size,
+					struct kvm_guest_s2_mapping *mapping);
 extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu);
 extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu);
 
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 061cd1e09af2..9bc799553ce9 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1636,6 +1636,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
 	enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED;
 	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
 	struct kvm_pgtable *pgt = s2fd->vcpu->arch.hw_mmu->pgt;
+	struct kvm_guest_s2_mapping *mapping = NULL;
 	unsigned long mmu_seq;
 	struct page *page;
 	struct kvm *kvm = s2fd->vcpu->kvm;
@@ -1649,6 +1650,11 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
 		ret = topup_mmu_memcache(s2fd->vcpu, memcache);
 		if (ret)
 			return ret;
+		if (kvm_is_nested_s2_mmu(kvm, pgt->mmu)) {
+			mapping = kmalloc_obj(struct kvm_guest_s2_mapping, GFP_KERNEL_ACCOUNT);
+			if (!mapping)
+				return -ENOMEM;
+		}
 	}
 
 	if (s2fd->nested)
@@ -1669,6 +1675,7 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
 	if (ret) {
 		kvm_prepare_memory_fault_exit(s2fd->vcpu, s2fd->fault_ipa, PAGE_SIZE,
 					      write_fault, exec_fault, false);
+		kfree(mapping);
 		return ret;
 	}
 
@@ -1702,11 +1709,22 @@ static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
 		ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, s2fd->fault_ipa, PAGE_SIZE,
 							 __pfn_to_phys(pfn), prot,
 							 memcache, flags);
+		/*
+		 * -EAGAIN from kvm_pgtable_stage2_map() can install mappings.
+		 * We don't know which subrange is installed, track the whole
+		 * thing.
+		 */
+		if ((ret == 0 || ret == -EAGAIN) && kvm_is_nested_s2_mmu(kvm, pgt->mmu)) {
+			kvm_record_guest_s2_mapping(pgt->mmu, gfn << PAGE_SHIFT,
+						    s2fd->fault_ipa, PAGE_SIZE, mapping);
+			mapping = NULL;
+		}
 	}
 
 out_unlock:
 	kvm_release_faultin_page(kvm, page, !!ret, prot & KVM_PGTABLE_PROT_W);
 	kvm_fault_unlock(kvm);
+	kfree(mapping);
 
 	if ((prot & KVM_PGTABLE_PROT_W) && !ret)
 		mark_page_dirty_in_slot(kvm, s2fd->memslot, gfn);
@@ -2043,6 +2061,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 			    void *memcache)
 {
 	enum kvm_pgtable_walk_flags flags = KVM_PGTABLE_WALK_SHARED;
+	struct kvm_guest_s2_mapping *mapping = NULL;
 	bool writable = prot & KVM_PGTABLE_PROT_W;
 	struct kvm *kvm = s2fd->vcpu->kvm;
 	phys_addr_t canonical_ipa;
@@ -2053,6 +2072,15 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 	gfn_t gfn;
 	int ret;
 
+	if (kvm_is_nested_s2_mmu(kvm, s2fd->vcpu->arch.hw_mmu)) {
+		mapping = kmalloc_obj(struct kvm_guest_s2_mapping,
+				      GFP_KERNEL_ACCOUNT);
+		if (!mapping) {
+			kvm_release_page_unused(s2vi->page);
+			return -ENOMEM;
+		}
+	}
+
 	kvm_fault_lock(kvm);
 	pgt = s2fd->vcpu->arch.hw_mmu->pgt;
 	ret = -EAGAIN;
@@ -2106,11 +2134,22 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 		ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, gfn_to_gpa(gfn), mapping_size,
 							 __pfn_to_phys(pfn), prot,
 							 memcache, flags);
+		/*
+		 * -EAGAIN from kvm_pgtable_stage2_map() can install mappings.
+		 * We don't know which subrange is installed, track the whole
+		 * thing.
+		 */
+		if ((ret == 0 || ret == -EAGAIN) && kvm_is_nested_s2_mmu(kvm, pgt->mmu)) {
+			kvm_record_guest_s2_mapping(pgt->mmu, canonical_ipa,
+						    gfn_to_gpa(gfn), mapping_size, mapping);
+			mapping = NULL;
+		}
 	}
 
 out_unlock:
 	kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable);
 	kvm_fault_unlock(kvm);
+	kfree(mapping);
 
 	/*
 	 * Mark the page dirty only if the fault is handled successfully,
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index b7bed02e38f7..d9acff507fd2 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -5,6 +5,7 @@
  */
 
 #include <linux/bitfield.h>
+#include <linux/interval_tree.h>
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 
@@ -872,6 +873,30 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu)
 	return s2_mmu;
 }
 
+void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
+				 gpa_t nested_ipa, size_t map_size,
+				 struct kvm_guest_s2_mapping *mapping)
+{
+	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+
+	lockdep_assert_held_read(&kvm->mmu_lock);
+
+	canonical_ipa = ALIGN_DOWN(canonical_ipa, map_size);
+	nested_ipa = ALIGN_DOWN(nested_ipa, map_size);
+
+	mapping->canonical.start = canonical_ipa;
+	mapping->canonical.last  = canonical_ipa + map_size - 1;
+
+	mapping->nested.start    = nested_ipa;
+	mapping->nested.last     = nested_ipa + map_size - 1;
+
+	mapping->nested_mmu      = mmu;
+
+	guard(spinlock)(&kvm->arch.guest_s2_tracking_lock);
+	interval_tree_insert(&mapping->nested, &mmu->guest_s2_mappings);
+	interval_tree_insert(&mapping->canonical, &kvm->arch.mmu.guest_s2_mappings);
+}
+
 void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu)
 {
 	/* CnP being set denotes an invalid entry */
-- 
2.43.0

[PATCH v6 4/7] KVM: arm64: nv: Track guest stage-2 mapping removal

[Wei-Lin Chang]

kvm_stage2_unmap_range() is the helper to remove mappings from the
stage-2 page tables. It is called during guest TLBI handling, memslot
removal, nested mmu reuse, etc.

Teach it about the guest stage-2 tracking trees and remove mappings from
there when shadow mappings are removed. This keeps the tracking trees
from having stale mappings pile up.

Don't remove tracked mappings from the interval trees if they only
partially overlap the removal range. For instance we can have a 4K
shadow mapping tracked as a 2M range in the interval trees. It would be
wrong to remove the tracked 2M range when a guest TLBI doesn't touch the
4K mapped.

Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
 arch/arm64/include/asm/kvm_host.h   |  5 ++++-
 arch/arm64/include/asm/kvm_nested.h |  2 ++
 arch/arm64/kvm/mmu.c                | 23 +++++++++++++++++--
 arch/arm64/kvm/nested.c             | 34 +++++++++++++++++++++++++++++
 4 files changed, 61 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 365ec57d6d7a..f12883a42081 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -340,7 +340,10 @@ struct kvm_arch {
 	size_t nested_mmus_size;
 	int nested_mmus_next;
 
-	/* Guest s2 tracking trees access serialization. */
+	/*
+	 * Serializes guest s2 tracking trees access when the mmu_lock
+	 * is only held for read.
+	 */
 	spinlock_t guest_s2_tracking_lock;
 
 	/* Interrupt controller */
diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index 2d71f686064a..954f532bc11b 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -81,6 +81,8 @@ extern void kvm_s2_mmu_iterate_by_vmid(struct kvm *kvm, u16 vmid,
 extern void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
 					gpa_t nested_ipa, size_t map_size,
 					struct kvm_guest_s2_mapping *mapping);
+extern void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu,
+					 gpa_t nipa, size_t size);
 extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu);
 extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu);
 
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 9bc799553ce9..c46e92d67f04 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -314,6 +314,19 @@ static void invalidate_icache_guest_page(void *va, size_t size)
  * we then fully enforce cacheability of RAM, no matter what the guest
  * does.
  */
+
+static int kvm_pgtable_stage2_unmap_tracked(struct kvm_pgtable *pgt, u64 addr, u64 size)
+{
+	int ret;
+
+	ret = kvm_pgtable_stage2_unmap(pgt, addr, size);
+	if (ret)
+		return ret;
+
+	kvm_remove_guest_s2_mappings(pgt->mmu, addr, size);
+	return 0;
+}
+
 /**
  * __unmap_stage2_range -- Clear stage2 page table entries to unmap a range
  * @mmu:   The KVM stage-2 MMU pointer
@@ -331,11 +344,17 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64
 {
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 	phys_addr_t end = start + size;
+	int (*fn)(struct kvm_pgtable *, u64, u64);
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
 	WARN_ON(size & ~PAGE_MASK);
-	WARN_ON(stage2_apply_range(mmu, start, end, KVM_PGT_FN(kvm_pgtable_stage2_unmap),
-				   may_block));
+
+	if (kvm_is_nested_s2_mmu(kvm, mmu))
+		fn = kvm_pgtable_stage2_unmap_tracked;
+	else
+		fn = KVM_PGT_FN(kvm_pgtable_stage2_unmap);
+
+	WARN_ON(stage2_apply_range(mmu, start, end, fn, may_block));
 }
 
 void kvm_stage2_unmap_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index d9acff507fd2..92a9a32f9f2b 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -897,6 +897,40 @@ void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
 	interval_tree_insert(&mapping->canonical, &kvm->arch.mmu.guest_s2_mappings);
 }
 
+void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, gpa_t nipa,
+				  size_t size)
+{
+	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+	struct interval_tree_node *node, *next;
+	struct kvm_guest_s2_mapping *mapping;
+	gpa_t nipa_end = nipa + size - 1;
+
+	/*
+	 * Guest s2 tracking interval trees are only accessed while holding the
+	 * mmu_lock, hence we don't have to take guest_s2_tracking_lock if the
+	 * mmu_lock is held for write.
+	 */
+	lockdep_assert_held_write(&kvm->mmu_lock);
+
+	node = interval_tree_iter_first(&mmu->guest_s2_mappings, nipa, nipa_end);
+	while (node) {
+		next = interval_tree_iter_next(node, nipa, nipa_end);
+		mapping = container_of(node, struct kvm_guest_s2_mapping,
+				       nested);
+		/*
+		 * Tracking must be conservative on removal, only remove
+		 * mappings that are within the unmap range.
+		 */
+		if (nipa <= mapping->nested.start && nipa_end >= mapping->nested.last) {
+			interval_tree_remove(&mapping->nested, &mmu->guest_s2_mappings);
+			interval_tree_remove(&mapping->canonical,
+					     &kvm->arch.mmu.guest_s2_mappings);
+			kfree(mapping);
+		}
+		node = next;
+	}
+}
+
 void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu)
 {
 	/* CnP being set denotes an invalid entry */
-- 
2.43.0

[PATCH v6 5/7] KVM: arm64: nv: Avoid full shadow stage-2 unmap

[Wei-Lin Chang]

With guest stage-2 tracking in place, we can improve MMU notifier unmaps
from unmapping all existing shadow stage-2 mappings to only unmapping
the ones affected by the given canonical IPA range.

Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
 arch/arm64/include/asm/kvm_nested.h |  2 ++
 arch/arm64/kvm/mmu.c                |  7 ++--
 arch/arm64/kvm/nested.c             | 50 +++++++++++++++++++++++++++--
 3 files changed, 53 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index 954f532bc11b..0f899addafa8 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -171,6 +171,8 @@ extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu,
 				    struct kvm_s2_trans *trans);
 extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2);
 extern void kvm_nested_s2_wp(struct kvm *kvm);
+extern void kvm_nested_unmap_cipa_range(struct kvm *kvm, gpa_t cipa,
+					size_t unmap_size, bool may_block);
 extern void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block);
 extern void kvm_nested_s2_flush(struct kvm *kvm);
 
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index c46e92d67f04..b8ba2d09ed96 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2525,8 +2525,9 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	__unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT,
 			     (range->end - range->start) << PAGE_SHIFT,
 			     range->may_block);
-
-	kvm_nested_s2_unmap(kvm, range->may_block);
+	kvm_nested_unmap_cipa_range(kvm, range->start << PAGE_SHIFT,
+				    (range->end - range->start) << PAGE_SHIFT,
+				    range->may_block);
 	return false;
 }
 
@@ -2808,7 +2809,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 
 	write_lock(&kvm->mmu_lock);
 	kvm_stage2_unmap_range(&kvm->arch.mmu, gpa, size, true);
-	kvm_nested_s2_unmap(kvm, true);
+	kvm_nested_unmap_cipa_range(kvm, gpa, size, true);
 	write_unlock(&kvm->mmu_lock);
 }
 
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 92a9a32f9f2b..4cbb4f1b8b35 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -906,9 +906,8 @@ void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, gpa_t nipa,
 	gpa_t nipa_end = nipa + size - 1;
 
 	/*
-	 * Guest s2 tracking interval trees are only accessed while holding the
-	 * mmu_lock, hence we don't have to take guest_s2_tracking_lock if the
-	 * mmu_lock is held for write.
+	 * See kvm_nested_unmap_cipa_range() for why guest_s2_tracking_lock
+	 * isn't taken here.
 	 */
 	lockdep_assert_held_write(&kvm->mmu_lock);
 
@@ -1342,6 +1341,51 @@ void kvm_nested_s2_wp(struct kvm *kvm)
 	kvm_invalidate_vncr_ipa(kvm, 0, BIT(kvm->arch.mmu.pgt->ia_bits));
 }
 
+void kvm_nested_unmap_cipa_range(struct kvm *kvm, gpa_t cipa, size_t unmap_size,
+				 bool may_block)
+{
+	gpa_t cipa_end = cipa + unmap_size - 1;
+	struct kvm_guest_s2_mapping *mapping;
+	struct interval_tree_node *node;
+	size_t mapping_size;
+
+	/*
+	 * Guest s2 tracking interval trees are only accessed while holding the
+	 * mmu_lock, hence we don't have to take guest_s2_tracking_lock if the
+	 * mmu_lock is held for write. This saves us from having to manually
+	 * lock/unlock guest_s2_tracking_lock below around
+	 * cond_resched_rwlock_write().
+	 */
+	lockdep_assert_held_write(&kvm->mmu_lock);
+
+	if (!kvm->arch.nested_mmus_size)
+		return;
+
+	while ((node = interval_tree_iter_first(&kvm->arch.mmu.guest_s2_mappings,
+						cipa, cipa_end))) {
+		mapping = container_of(node, struct kvm_guest_s2_mapping, canonical);
+		mapping_size = mapping->nested.last - mapping->nested.start + 1;
+
+		/* We could race against MMU teardown, which frees mmu->pgt. */
+		if (mapping->nested_mmu->pgt) {
+			if (WARN_ON_ONCE(kvm_pgtable_stage2_unmap(mapping->nested_mmu->pgt,
+								  mapping->nested.start,
+								  mapping_size)))
+				return;
+
+			interval_tree_remove(&mapping->nested,
+					     &mapping->nested_mmu->guest_s2_mappings);
+		}
+		interval_tree_remove(node, &kvm->arch.mmu.guest_s2_mappings);
+		kfree(mapping);
+
+		if (may_block)
+			cond_resched_rwlock_write(&kvm->mmu_lock);
+	}
+
+	kvm_invalidate_vncr_ipa(kvm, cipa, cipa + unmap_size);
+}
+
 void kvm_nested_s2_unmap(struct kvm *kvm, bool may_block)
 {
 	int i;
-- 
2.43.0

[PATCH v6 6/7] KVM: arm64: nv: Drop kvm_s2_mmu pointer from kvm_guest_s2_mapping

[Wei-Lin Chang]

From: Marc Zyngier <maz@kernel.org>

As it appears that the kvm_guest_s2_mapping structure is quite large,
and results in a 128 byte slab allocation, there is some incentive
to shrink a bit.

For this, replace the S2 MMU back-pointer with an index tucked into
the low bits of the nested.start field. This allows us to shrink
the structure by 8 bytes, and therefore to fit in a 96 byte slab.

The index is also stored in the S2 MMU structure itself, which
comes for free as it fits in an existing hole. The structure is
also repacked to avoid other disgracious holes.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
 arch/arm64/include/asm/kvm_host.h | 19 ++++++-----
 arch/arm64/kvm/nested.c           | 54 ++++++++++++++++++++++++-------
 2 files changed, 54 insertions(+), 19 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index f12883a42081..8dfaa33c6cd5 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -153,12 +153,12 @@ struct kvm_vmid {
 
 /*
  * Record of a guest stage-2 mapping, storing canonical and nested IPA
- * ranges. Both ranges have the same size.
+ * ranges. Both ranges have the same size. The lower bits of nested.start
+ * store the index of the nested mmu this mapping belongs to.
  */
 struct kvm_guest_s2_mapping {
 	struct interval_tree_node canonical;
 	struct interval_tree_node nested;
-	struct kvm_s2_mmu *nested_mmu;
 };
 
 struct kvm_s2_mmu {
@@ -222,30 +222,33 @@ struct kvm_s2_mmu {
 	u64	tlb_vttbr;
 	u64	tlb_vtcr;
 
+	/* Guest s2 mapping records indexed in this MMU's IPA space. */
+	struct rb_root_cached guest_s2_mappings;
+
 	/*
 	 * true when this represents a nested context where virtual
 	 * HCR_EL2.VM == 1
 	 */
 	bool	nested_stage2_enabled;
 
-#ifdef CONFIG_PTDUMP_STAGE2_DEBUGFS
-	struct dentry *shadow_pt_debugfs_dentry;
-#endif
-
 	/*
 	 * true when this MMU needs to be unmapped before being used for a new
 	 * purpose.
 	 */
 	bool	pending_unmap;
 
-	/* Guest s2 mapping records indexed in this MMU's IPA space. */
-	struct rb_root_cached guest_s2_mappings;
+	/* Index in the S2 MMU array, only valid for a shadow S2 */
+	u16	s2_mmu_idx;
 
 	/*
 	 *  0: Nobody is currently using this, check vttbr for validity
 	 * >0: Somebody is actively using this.
 	 */
 	atomic_t refcnt;
+
+#ifdef CONFIG_PTDUMP_STAGE2_DEBUGFS
+	struct dentry *shadow_pt_debugfs_dentry;
+#endif
 };
 
 struct kvm_arch_memory_slot {
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 4cbb4f1b8b35..61d4dae6be6a 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -45,11 +45,12 @@ struct vncr_tlb {
  * will invalidate them more often).
  */
 #define S2_MMU_PER_VCPU		2
+#define S2_MMU_PER_VM		(KVM_MAX_VCPUS * S2_MMU_PER_VCPU)
 
 int kvm_init_nested(struct kvm *kvm)
 {
 	kvm->arch.nested_mmus = kvmalloc_objs(struct kvm_s2_mmu *,
-					      KVM_MAX_VCPUS * S2_MMU_PER_VCPU,
+					      S2_MMU_PER_VM,
 					      GFP_KERNEL_ACCOUNT);
 	kvm->arch.nested_mmus_size = 0;
 	atomic_set(&kvm->arch.vncr_tlb_count, 0);
@@ -128,8 +129,10 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
 
 		guard(write_lock)(&kvm->mmu_lock);
 
-		for (i = 0; i < S2_MMU_PER_VCPU; i++)
+		for (i = 0; i < S2_MMU_PER_VCPU; i++) {
+			tmp[i].s2_mmu_idx = i + kvm->arch.nested_mmus_size;
 			kvm->arch.nested_mmus[i + kvm->arch.nested_mmus_size] = &tmp[i];
+		}
 
 		kvm->arch.nested_mmus_size += S2_MMU_PER_VCPU;
 	}
@@ -873,6 +876,27 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu)
 	return s2_mmu;
 }
 
+#define S2_MMU_IDX_MASK	GENMASK_ULL(11, 0)
+
+static void tag_s2_mapping_mmu(struct kvm_guest_s2_mapping *mapping,
+			       struct kvm_s2_mmu *mmu)
+{
+	BUILD_BUG_ON(S2_MMU_PER_VM > SZ_4K);
+	mapping->nested.start &= ~S2_MMU_IDX_MASK;
+	mapping->nested.start |= mmu->s2_mmu_idx;
+}
+
+static struct kvm_s2_mmu *s2_mapping_to_mmu(struct kvm *kvm,
+					    struct kvm_guest_s2_mapping *mapping)
+{
+	return kvm->arch.nested_mmus[mapping->nested.start & S2_MMU_IDX_MASK];
+}
+
+static unsigned long s2_mapping_to_nested_start(struct kvm_guest_s2_mapping *mapping)
+{
+	return mapping->nested.start & ~S2_MMU_IDX_MASK;
+}
+
 void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
 				 gpa_t nested_ipa, size_t map_size,
 				 struct kvm_guest_s2_mapping *mapping)
@@ -890,7 +914,7 @@ void kvm_record_guest_s2_mapping(struct kvm_s2_mmu *mmu, gpa_t canonical_ipa,
 	mapping->nested.start    = nested_ipa;
 	mapping->nested.last     = nested_ipa + map_size - 1;
 
-	mapping->nested_mmu      = mmu;
+	tag_s2_mapping_mmu(mapping, mmu);
 
 	guard(spinlock)(&kvm->arch.guest_s2_tracking_lock);
 	interval_tree_insert(&mapping->nested, &mmu->guest_s2_mappings);
@@ -913,6 +937,8 @@ void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, gpa_t nipa,
 
 	node = interval_tree_iter_first(&mmu->guest_s2_mappings, nipa, nipa_end);
 	while (node) {
+		unsigned long nested_start;
+
 		next = interval_tree_iter_next(node, nipa, nipa_end);
 		mapping = container_of(node, struct kvm_guest_s2_mapping,
 				       nested);
@@ -920,7 +946,8 @@ void kvm_remove_guest_s2_mappings(struct kvm_s2_mmu *mmu, gpa_t nipa,
 		 * Tracking must be conservative on removal, only remove
 		 * mappings that are within the unmap range.
 		 */
-		if (nipa <= mapping->nested.start && nipa_end >= mapping->nested.last) {
+		nested_start = s2_mapping_to_nested_start(mapping);
+		if (nipa <= nested_start && nipa_end >= mapping->nested.last) {
 			interval_tree_remove(&mapping->nested, &mmu->guest_s2_mappings);
 			interval_tree_remove(&mapping->canonical,
 					     &kvm->arch.mmu.guest_s2_mappings);
@@ -1363,18 +1390,23 @@ void kvm_nested_unmap_cipa_range(struct kvm *kvm, gpa_t cipa, size_t unmap_size,
 
 	while ((node = interval_tree_iter_first(&kvm->arch.mmu.guest_s2_mappings,
 						cipa, cipa_end))) {
-		mapping = container_of(node, struct kvm_guest_s2_mapping, canonical);
-		mapping_size = mapping->nested.last - mapping->nested.start + 1;
+		unsigned long nested_start;
+		struct kvm_s2_mmu *mmu;
+
+		mapping = container_of(node, struct kvm_guest_s2_mapping,
+				       canonical);
+		nested_start = s2_mapping_to_nested_start(mapping);
+		mmu = s2_mapping_to_mmu(kvm, mapping);
 
 		/* We could race against MMU teardown, which frees mmu->pgt. */
-		if (mapping->nested_mmu->pgt) {
-			if (WARN_ON_ONCE(kvm_pgtable_stage2_unmap(mapping->nested_mmu->pgt,
-								  mapping->nested.start,
+		if (mmu->pgt) {
+			mapping_size = mapping->nested.last - nested_start + 1;
+
+			if (WARN_ON_ONCE(kvm_pgtable_stage2_unmap(mmu->pgt, nested_start,
 								  mapping_size)))
 				return;
 
-			interval_tree_remove(&mapping->nested,
-					     &mapping->nested_mmu->guest_s2_mappings);
+			interval_tree_remove(&mapping->nested, &mmu->guest_s2_mappings);
 		}
 		interval_tree_remove(node, &kvm->arch.mmu.guest_s2_mappings);
 		kfree(mapping);
-- 
2.43.0

[PATCH v6 7/7] KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables

[Wei-Lin Chang]

__unmap_stage2_range() and kvm_nested_unmap_cipa_range() are using the
same arguments. Clean this up by using local variables.

Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com>
---
 arch/arm64/kvm/mmu.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index b8ba2d09ed96..3c3bc231eab7 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -2519,15 +2519,16 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
 
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
+	gpa_t gpa = range->start << PAGE_SHIFT;
+	size_t size = (range->end - range->start) << PAGE_SHIFT;
+	bool may_block = range->may_block;
+
 	if (!kvm->arch.mmu.pgt || kvm_vm_is_protected(kvm))
 		return false;
 
-	__unmap_stage2_range(&kvm->arch.mmu, range->start << PAGE_SHIFT,
-			     (range->end - range->start) << PAGE_SHIFT,
-			     range->may_block);
-	kvm_nested_unmap_cipa_range(kvm, range->start << PAGE_SHIFT,
-				    (range->end - range->start) << PAGE_SHIFT,
-				    range->may_block);
+	__unmap_stage2_range(&kvm->arch.mmu, gpa, size, may_block);
+	kvm_nested_unmap_cipa_range(kvm, gpa, size, may_block);
+
 	return false;
 }
 
-- 
2.43.0

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Itaru Kitayama]

On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> Hi,
> 
> This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> notifiers.
> 
> This version fixes a few issues, and adds Marc's space optimization
> for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> 
> Tested by booting L3, and running in-kernel targetted tests described
> in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> carry them over since some bugs fixed in v6 are non-obvious. Some
> retest would be much appreciated!

How did you boot into L3, with KVM selftest or recrusively boot on HW?

Thanks,
Itaru.

> 
> Series based on v7.3-rc3 + Marc's nested mmu lifecycle fixes [3].
> 
> * Changes from v5 [4]:
> 
>   - Align the addresses down to the mapping size when recording the
>     guest s2 mappings. s2fd->fault_ipa isn't necessarily PAGE_SIZE aligned.
> 
>   - Record guest s2 mappings even when page table maps return -EAGAIN.
>     kvm_pgtable_stage2_map() can create mappings while returning -EAGAIN.
>     For example, a 2M block map (A) could race with a 4K page map (B):
>     1. (A) maps the 2M block in kvm_pgtable_visitor_cb()
>     2. (B) breaks that block into a table and maps 4K
>     3. (A) reloads and finds the table after kvm_pgtable_visitor_cb(),
>        then descends into it.
>     4. (A) maps some 4K, but before it finishes reads entry mapped by (B).
>     5. (A) returns -EAGAIN although it had mapped a few pages.
>     In this case, we don't know what subrange is mapped, just track the whole
>     requested mapping range.
> 
>   - Don't remove tracked mappings from the interval trees if they only
>     partially overlap the removal range. Because of the previous bullet point
>     we can have a 4K shadow mapping tracked as a 2M range in the interval
>     trees. It would be wrong to remove the 2M range when a guest TLBI doesn't
>     touch the 4K mapped.
> 
>   - Check mmu->pgt during mmu unmap notifier, as it could race against
>     MMU teardown.
> 
>   - Make guest_s2_tracking_destroy() canonical mmu only. We simply don't
>     need to detach the nodes from the nested mmus' trees. Freeing them
>     during canonical mmu's teardown is enough.
> 
> Thanks!
> 
> [1]: https://lore.kernel.org/kvmarm/86h5jv7qrd.wl-maz@kernel.org/
> [2]: https://lore.kernel.org/kvmarm/gerjpm62a2gszzggc6vuai22bf3prfquvfsp7ueumps7vz2ev3@odlmovaklg2b/
> [3]: https://lore.kernel.org/kvmarm/20260911162203.1919330-1-maz@kernel.org/
> [4]: https://lore.kernel.org/kvmarm/20260810205038.118843-1-weilin.chang@arm.com/
> 
> Marc Zyngier (1):
>   KVM: arm64: nv: Drop kvm_s2_mmu pointer from kvm_guest_s2_mapping
> 
> Wei-Lin Chang (6):
>   KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map()
>   KVM: arm64: nv: Introduce guest stage-2 tracking structures
>   KVM: arm64: nv: Track guest stage-2 mapping creation
>   KVM: arm64: nv: Track guest stage-2 mapping removal
>   KVM: arm64: nv: Avoid full shadow stage-2 unmap
>   KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables
> 
>  arch/arm64/include/asm/kvm_host.h   |  31 +++++-
>  arch/arm64/include/asm/kvm_nested.h |   7 ++
>  arch/arm64/kvm/mmu.c                | 109 +++++++++++++++++++---
>  arch/arm64/kvm/nested.c             | 140 +++++++++++++++++++++++++++-
>  4 files changed, 269 insertions(+), 18 deletions(-)
> 
> -- 
> 2.43.0
> 

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Oliver Upton]

On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> Hi,
> 
> This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> notifiers.
> 
> This version fixes a few issues, and adds Marc's space optimization
> for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> 
> Tested by booting L3, and running in-kernel targetted tests described
> in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> carry them over since some bugs fixed in v6 are non-obvious. Some
> retest would be much appreciated!
> 
> Series based on v7.3-rc3 + Marc's nested mmu lifecycle fixes [3].

Looking pretty good to me, thanks Wei-Lin!

Reviewed-by: Oliver Upton <oupton@kernel.org>

Thanks,
Oliver

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Wei-Lin Chang]

On Wed, Sep 16, 2026 at 06:49:57AM +0900, Itaru Kitayama wrote:
> On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > Hi,
> > 
> > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > notifiers.
> > 
> > This version fixes a few issues, and adds Marc's space optimization
> > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > 
> > Tested by booting L3, and running in-kernel targetted tests described
> > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > carry them over since some bugs fixed in v6 are non-obvious. Some
> > retest would be much appreciated!
> 
> How did you boot into L3, with KVM selftest or recrusively boot on HW?

I just recrusively booted VMs on hardware. :)

Thanks,
Wei-Lin Chang

> 
> Thanks,
> Itaru.
> 

[...]

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Itaru Kitayama]

On Wed, Sep 16, 2026 at 12:22:30AM +0100, Wei-Lin Chang wrote:
> On Wed, Sep 16, 2026 at 06:49:57AM +0900, Itaru Kitayama wrote:
> > On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > > Hi,
> > > 
> > > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > > notifiers.
> > > 
> > > This version fixes a few issues, and adds Marc's space optimization
> > > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > > 
> > > Tested by booting L3, and running in-kernel targetted tests described
> > > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > > carry them over since some bugs fixed in v6 are non-obvious. Some
> > > retest would be much appreciated!
> > 
> > How did you boot into L3, with KVM selftest or recrusively boot on HW?
> 
> I just recrusively booted VMs on hardware. :)

Good. Then I actually tried that on Grace CPUs, but letting L1 boot 
with nested mode did not print "VHE+NV2" KVM init log in dmesg, thus not
able to boot L3. Do I need to add other than kvm.arm-mode=nested to command line?

Thanks,
Itaru.

> 
> Thanks,
> Wei-Lin Chang
> 
> > 
> > Thanks,
> > Itaru.
> > 
> 
> [...]
> 

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Itaru Kitayama]

Hi Wei-Lin,

On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> Hi,
> 
> This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> notifiers.
> 
> This version fixes a few issues, and adds Marc's space optimization
> for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> 
> Tested by booting L3, and running in-kernel targetted tests described
> in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> carry them over since some bugs fixed in v6 are non-obvious. Some
> retest would be much appreciated!
> 
> Series based on v7.3-rc3 + Marc's nested mmu lifecycle fixes [3].

Observed on QEMU (accel tcg) only, but I saw:

[ 1812.052752] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1630
[ 1812.052896] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 596, name: shadow_stage2

The kernel is built on top of Marc's fix and your version six series, but
the fix for debugfs will be posted later?

Your hello_nested and shadow_stage2 ran without an issue.

Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com>

Thanks,
Itaru.

> 
> * Changes from v5 [4]:
> 
>   - Align the addresses down to the mapping size when recording the
>     guest s2 mappings. s2fd->fault_ipa isn't necessarily PAGE_SIZE aligned.
> 
>   - Record guest s2 mappings even when page table maps return -EAGAIN.
>     kvm_pgtable_stage2_map() can create mappings while returning -EAGAIN.
>     For example, a 2M block map (A) could race with a 4K page map (B):
>     1. (A) maps the 2M block in kvm_pgtable_visitor_cb()
>     2. (B) breaks that block into a table and maps 4K
>     3. (A) reloads and finds the table after kvm_pgtable_visitor_cb(),
>        then descends into it.
>     4. (A) maps some 4K, but before it finishes reads entry mapped by (B).
>     5. (A) returns -EAGAIN although it had mapped a few pages.
>     In this case, we don't know what subrange is mapped, just track the whole
>     requested mapping range.
> 
>   - Don't remove tracked mappings from the interval trees if they only
>     partially overlap the removal range. Because of the previous bullet point
>     we can have a 4K shadow mapping tracked as a 2M range in the interval
>     trees. It would be wrong to remove the 2M range when a guest TLBI doesn't
>     touch the 4K mapped.
> 
>   - Check mmu->pgt during mmu unmap notifier, as it could race against
>     MMU teardown.
> 
>   - Make guest_s2_tracking_destroy() canonical mmu only. We simply don't
>     need to detach the nodes from the nested mmus' trees. Freeing them
>     during canonical mmu's teardown is enough.
> 
> Thanks!
> 
> [1]: https://lore.kernel.org/kvmarm/86h5jv7qrd.wl-maz@kernel.org/
> [2]: https://lore.kernel.org/kvmarm/gerjpm62a2gszzggc6vuai22bf3prfquvfsp7ueumps7vz2ev3@odlmovaklg2b/
> [3]: https://lore.kernel.org/kvmarm/20260911162203.1919330-1-maz@kernel.org/
> [4]: https://lore.kernel.org/kvmarm/20260810205038.118843-1-weilin.chang@arm.com/
> 
> Marc Zyngier (1):
>   KVM: arm64: nv: Drop kvm_s2_mmu pointer from kvm_guest_s2_mapping
> 
> Wei-Lin Chang (6):
>   KVM: arm64: Use a variable for the canonical IPA in kvm_s2_fault_map()
>   KVM: arm64: nv: Introduce guest stage-2 tracking structures
>   KVM: arm64: nv: Track guest stage-2 mapping creation
>   KVM: arm64: nv: Track guest stage-2 mapping removal
>   KVM: arm64: nv: Avoid full shadow stage-2 unmap
>   KVM: arm64: Refactor kvm_unmap_gfn_range() with common variables
> 
>  arch/arm64/include/asm/kvm_host.h   |  31 +++++-
>  arch/arm64/include/asm/kvm_nested.h |   7 ++
>  arch/arm64/kvm/mmu.c                | 109 +++++++++++++++++++---
>  arch/arm64/kvm/nested.c             | 140 +++++++++++++++++++++++++++-
>  4 files changed, 269 insertions(+), 18 deletions(-)
> 
> -- 
> 2.43.0
> 

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Marc Zyngier]

On Wed, 16 Sep 2026 05:58:03 +0100,
Itaru Kitayama <itaru.kitayama@fujitsu.com> wrote:
> 
> Hi Wei-Lin,
> 
> On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > Hi,
> > 
> > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > notifiers.
> > 
> > This version fixes a few issues, and adds Marc's space optimization
> > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > 
> > Tested by booting L3, and running in-kernel targetted tests described
> > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > carry them over since some bugs fixed in v6 are non-obvious. Some
> > retest would be much appreciated!
> > 
> > Series based on v7.3-rc3 + Marc's nested mmu lifecycle fixes [3].
> 
> Observed on QEMU (accel tcg) only, but I saw:
> 
> [ 1812.052752] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1630
> [ 1812.052896] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 596, name: shadow_stage2
> 
> The kernel is built on top of Marc's fix and your version six series, but
> the fix for debugfs will be posted later?
> 
> Your hello_nested and shadow_stage2 ran without an issue.
> 
> Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com>

So you have established that there is a critical locking bug, make
sure you don't communicate any information to help narrow where that
bug is, and yet give your Tested-by: tag?

What is the point? I'm baffled.

Can you at least provide an indication of what you did to get there as
well as the full backtrace?

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Marc Zyngier]

On Wed, 16 Sep 2026 00:27:22 +0100,
Itaru Kitayama <itaru.kitayama@fujitsu.com> wrote:
> 
> On Wed, Sep 16, 2026 at 12:22:30AM +0100, Wei-Lin Chang wrote:
> > On Wed, Sep 16, 2026 at 06:49:57AM +0900, Itaru Kitayama wrote:
> > > On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > > > Hi,
> > > > 
> > > > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > > > notifiers.
> > > > 
> > > > This version fixes a few issues, and adds Marc's space optimization
> > > > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > > > 
> > > > Tested by booting L3, and running in-kernel targetted tests described
> > > > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > > > carry them over since some bugs fixed in v6 are non-obvious. Some
> > > > retest would be much appreciated!
> > > 
> > > How did you boot into L3, with KVM selftest or recrusively boot on HW?
> > 
> > I just recrusively booted VMs on hardware. :)
> 
> Good. Then I actually tried that on Grace CPUs, but letting L1 boot 
> with nested mode did not print "VHE+NV2" KVM init log in dmesg, thus not
> able to boot L3. Do I need to add other than kvm.arm-mode=nested to command line?

kvm-arm.mode=nested, as documented. That also puts in perspective your
TB tag on the whole series...

	M.


-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Wei-Lin Chang]

On Wed, Sep 16, 2026 at 08:04:53AM +0100, Marc Zyngier wrote:
> On Wed, 16 Sep 2026 05:58:03 +0100,
> Itaru Kitayama <itaru.kitayama@fujitsu.com> wrote:
> > 
> > Hi Wei-Lin,
> > 
> > On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > > Hi,
> > > 
> > > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > > notifiers.
> > > 
> > > This version fixes a few issues, and adds Marc's space optimization
> > > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > > 
> > > Tested by booting L3, and running in-kernel targetted tests described
> > > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > > carry them over since some bugs fixed in v6 are non-obvious. Some
> > > retest would be much appreciated!
> > > 
> > > Series based on v7.3-rc3 + Marc's nested mmu lifecycle fixes [3].
> > 
> > Observed on QEMU (accel tcg) only, but I saw:
> > 
> > [ 1812.052752] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1630
> > [ 1812.052896] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 596, name: shadow_stage2
> > 
> > The kernel is built on top of Marc's fix and your version six series, but
> > the fix for debugfs will be posted later?
> > 
> > Your hello_nested and shadow_stage2 ran without an issue.
> > 
> > Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com>
> 
> So you have established that there is a critical locking bug, make
> sure you don't communicate any information to help narrow where that
> bug is, and yet give your Tested-by: tag?
> 
> What is the point? I'm baffled.
> 
> Can you at least provide an indication of what you did to get there as
> well as the full backtrace?

It is certainly the known shadow ptdump sleeping-in-atomic problem [1]
reported by Itaru, and not a problem of the reverse map series.

I have the fix for that, will send it shortly.

Thanks,
Wei-Lin Chang

[1]: https://lore.kernel.org/kvmarm/aiuF0KSvvv-ZozI1@sm-arm-grace07/

> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Itaru Kitayama]

On Wed, Sep 16, 2026 at 08:08:20AM +0100, Marc Zyngier wrote:
> On Wed, 16 Sep 2026 00:27:22 +0100,
> Itaru Kitayama <itaru.kitayama@fujitsu.com> wrote:
> > 
> > On Wed, Sep 16, 2026 at 12:22:30AM +0100, Wei-Lin Chang wrote:
> > > On Wed, Sep 16, 2026 at 06:49:57AM +0900, Itaru Kitayama wrote:
> > > > On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > > > > Hi,
> > > > > 
> > > > > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > > > > notifiers.
> > > > > 
> > > > > This version fixes a few issues, and adds Marc's space optimization
> > > > > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > > > > 
> > > > > Tested by booting L3, and running in-kernel targetted tests described
> > > > > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > > > > carry them over since some bugs fixed in v6 are non-obvious. Some
> > > > > retest would be much appreciated!
> > > > 
> > > > How did you boot into L3, with KVM selftest or recrusively boot on HW?
> > > 
> > > I just recrusively booted VMs on hardware. :)
> > 
> > Good. Then I actually tried that on Grace CPUs, but letting L1 boot 
> > with nested mode did not print "VHE+NV2" KVM init log in dmesg, thus not
> > able to boot L3. Do I need to add other than kvm.arm-mode=nested to command line?
> 
> kvm-arm.mode=nested, as documented. That also puts in perspective your
> TB tag on the whole series...

Yes. Using kvmtool's --nested option did not bring vCPU with the VHE+NV2 boot
log. The Host kernel is Wei-Lin's version 6 booted on Grace CPUs, and
the guest kernel is today's kvmarm/fixes plus Wei-Lin's shadow stage 2 ptdump fix.

Thanks,
Itaru.

> 
> 	M.
> 
> 
> -- 
> Without deviation from the norm, progress is not possible.

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Marc Zyngier]

On Thu, 17 Sep 2026 07:51:19 +0100,
Itaru Kitayama <itaru.kitayama@fujitsu.com> wrote:
> 
> On Wed, Sep 16, 2026 at 08:08:20AM +0100, Marc Zyngier wrote:
> > On Wed, 16 Sep 2026 00:27:22 +0100,
> > Itaru Kitayama <itaru.kitayama@fujitsu.com> wrote:
> > > 
> > > On Wed, Sep 16, 2026 at 12:22:30AM +0100, Wei-Lin Chang wrote:
> > > > On Wed, Sep 16, 2026 at 06:49:57AM +0900, Itaru Kitayama wrote:
> > > > > On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > > > > > notifiers.
> > > > > > 
> > > > > > This version fixes a few issues, and adds Marc's space optimization
> > > > > > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > > > > > 
> > > > > > Tested by booting L3, and running in-kernel targetted tests described
> > > > > > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > > > > > carry them over since some bugs fixed in v6 are non-obvious. Some
> > > > > > retest would be much appreciated!
> > > > > 
> > > > > How did you boot into L3, with KVM selftest or recrusively boot on HW?
> > > > 
> > > > I just recrusively booted VMs on hardware. :)
> > > 
> > > Good. Then I actually tried that on Grace CPUs, but letting L1 boot 
> > > with nested mode did not print "VHE+NV2" KVM init log in dmesg, thus not
> > > able to boot L3. Do I need to add other than kvm.arm-mode=nested to command line?
> > 
> > kvm-arm.mode=nested, as documented. That also puts in perspective your
> > TB tag on the whole series...
> 
> Yes. Using kvmtool's --nested option did not bring vCPU with the VHE+NV2 boot
> log. The Host kernel is Wei-Lin's version 6 booted on Grace CPUs, and
> the guest kernel is today's kvmarm/fixes plus Wei-Lin's shadow stage 2 ptdump fix.

What have you tested exactly? If you're not able to boot a working L1
guest as a NV2 hypervisor, you're probably doing something wrong.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v6 0/7] KVM: arm64: nv: Implement nested stage-2 reverse map

[Wei-Lin Chang]

On Thu, Sep 17, 2026 at 03:51:19PM +0900, Itaru Kitayama wrote:
> On Wed, Sep 16, 2026 at 08:08:20AM +0100, Marc Zyngier wrote:
> > On Wed, 16 Sep 2026 00:27:22 +0100,
> > Itaru Kitayama <itaru.kitayama@fujitsu.com> wrote:
> > > 
> > > On Wed, Sep 16, 2026 at 12:22:30AM +0100, Wei-Lin Chang wrote:
> > > > On Wed, Sep 16, 2026 at 06:49:57AM +0900, Itaru Kitayama wrote:
> > > > > On Tue, Sep 15, 2026 at 04:42:58PM +0100, Wei-Lin Chang wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > This is v6 of optimizing the shadow s2 mmu unmapping during MMU
> > > > > > notifiers.
> > > > > > 
> > > > > > This version fixes a few issues, and adds Marc's space optimization
> > > > > > for kvm_guest_s2_mapping [1]. Please see the changelog for the details.
> > > > > > 
> > > > > > Tested by booting L3, and running in-kernel targetted tests described
> > > > > > in [2]. v5 got some tested-by's from Itaru and Wang (thanks), I didn't
> > > > > > carry them over since some bugs fixed in v6 are non-obvious. Some
> > > > > > retest would be much appreciated!
> > > > > 
> > > > > How did you boot into L3, with KVM selftest or recrusively boot on HW?
> > > > 
> > > > I just recrusively booted VMs on hardware. :)
> > > 
> > > Good. Then I actually tried that on Grace CPUs, but letting L1 boot 
> > > with nested mode did not print "VHE+NV2" KVM init log in dmesg, thus not
> > > able to boot L3. Do I need to add other than kvm.arm-mode=nested to command line?
> > 
> > kvm-arm.mode=nested, as documented. That also puts in perspective your
> > TB tag on the whole series...
> 
> Yes. Using kvmtool's --nested option did not bring vCPU with the VHE+NV2 boot
> log. The Host kernel is Wei-Lin's version 6 booted on Grace CPUs, and
> the guest kernel is today's kvmarm/fixes plus Wei-Lin's shadow stage 2 ptdump fix.

I'm not sure what you did exactly, but just to be clear, you need to
pass kvm-arm.mode=nested to L1's command line as well, in addition to
kvmtool's --nested option.

./lkvm run --nested -p kvm-arm.mode=nested <...>

Thanks,
Wei-Lin Chang

> 
> Thanks,
> Itaru.
> 
> > 
> > 	M.
> > 
> > 
> > -- 
> > Without deviation from the norm, progress is not possible.