mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs
@ 2026-11-17 21:09 Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE Sean Christopherson
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
  To: Sean Christopherson, Paolo Bonzini
  Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason

Fix a variety of bugs related to KVM's handling of CR3 loads during nested
transitions.  I originally tried to fix a svm_set_nested_state() bug by
forcing the vCPU back into L1 if a late CR3 load fails, but Sashiko rightly
hated on that patch because forcing the vCPU from L2=>L1 would still leave
vCPU state in a half-baked state.

I finally realized that Paolo's patch to always check CR3 validity is the way
to go, svm_set_nested_state() just needs to also pre-check the current CR3
(because userspace could theoretically stuff a then-legal CR3 and then swizzle
CPUID to make CR3 illegal).

Patch 5 fixes another related flaw where KVM doesn't initialize the MMU if
loading PDPTRs on nested VM-Exit fails.  Patches 3-4 are cleanups and prep
work to dedup the nSVM vs. nVMX code to avoid copy+paste and confusing code
for the final fix.

v2:
 - Pre-check the current CR3 in svm_set_nested_state(). [Sashiko]
 - Pull in Paolo's fix. [Sashiko]
 - Add patches 3-5.

v1: https://lore.kernel.org/all/20261113170405.473841-1-seanjc@google.com

Paolo Bonzini (1):
  KVM: nVMX/nSVM: check unconditionally whether CR3 is valid

Sean Christopherson (4):
  KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE
  KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions
  KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested
    xfer
  KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested
    transitions

 arch/x86/kvm/mmu/mmu.c    |  2 --
 arch/x86/kvm/regs.c       | 30 ++++++++++++++++++++++++++++++
 arch/x86/kvm/regs.h       |  2 ++
 arch/x86/kvm/svm/nested.c | 36 ++++++++++--------------------------
 arch/x86/kvm/vmx/nested.c | 31 ++++---------------------------
 5 files changed, 46 insertions(+), 55 deletions(-)


base-commit: d21eef2084a9af87ac2a783bc7c2b175ba114e4e
-- 
2.56.1.703.gf848ed807c-goog


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE
  2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 2/5] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid Sean Christopherson
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
  To: Sean Christopherson, Paolo Bonzini
  Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason

When loading nested state, explicitly check that the current CR3 is valid
before loading any state, as a misbehaving userspace can muck with guest
CPUID to make KVM think the current CR3 is invalid, and thus cause the late
"load" of CR3 to fail when initializing the guest's MMU.  If that happens,
KVM will leave half-baked state in the vCPU, e.g. in L2 with L1's page
walker, and if nested NPT is disabled, L1's PGD as well.

Alternatively, KVM could try and unwind vCPU state if the late CR3 "load"
fails, but that is subtly quite difficult as it would require snapshotting
and restoring the previous state.  Simply leaving nested on failure isn't
an option either, because that too would leave the vCPU with half-baked
state (in L1 with L2 state and a very stale MMU).

Pre-checking CR3 will also allow hoisting the CR3 check out of
nested_svm_load_cr3() entirely.

WARN and bug the VM if the late CR3 load fails, as loading nested state
never loads PDPTRs, i.e. failure is impossible, and KVM doesn't unwind on
failure (see above).  I.e. if the impossible happens, KVM needs to block
all operations on the half-baked vCPU.

Fixes: e1779c2714c3 ("KVM: x86: nSVM: fix potential NULL derefernce on nested migration")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-210535@kernel.org
Closes: https://lore.kernel.org/all/sashiko-outbox-205939@kernel.org
Closes: https://lore.kernel.org/all/sashiko-outbox-193903@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 07554089ec05..1e3842851dae 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -2047,6 +2047,9 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
         if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW))
 		goto out_free;
 
+	if (!kvm_vcpu_is_legal_cr3(vcpu, vcpu->arch.cr3))
+		goto out_free;
+
 	/*
 	 * Validate host state saved from before VMRUN (see
 	 * nested_svm_check_permissions). Note that the g_pat field is not
@@ -2114,7 +2117,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	 */
 	ret = nested_svm_load_cr3(&svm->vcpu, vcpu->arch.cr3,
 				  nested_npt_enabled(svm), false);
-	if (ret)
+	if (KVM_BUG_ON(ret, vcpu->kvm))
 		goto out_free;
 
 	svm->nested.force_msr_bitmap_recalc = true;
-- 
2.56.1.703.gf848ed807c-goog


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2 2/5] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid
  2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 3/5] KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions Sean Christopherson
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
  To: Sean Christopherson, Paolo Bonzini
  Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason

From: Paolo Bonzini <pbonzini@redhat.com>

nested_vmcb_check_save() checks CR3 legality only if long mode is active,
but it really needs to be checked in all cases - the check uses must be
zero bits of the host, and indeed nested_svm_load_cr3() later will
reject illegal CR3 in every mode.  Move the check earlier, and thus out
of nested_svm_load_cr3().

Likewise for nested_vmx_load_cr3() and nested_vmx_check_guest_state().

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Tested-by: syzbot@syzkaller.appspotmail.com
Cc: stable@vger.kernel.org
[sean: tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 9 +++------
 arch/x86/kvm/vmx/nested.c | 8 ++------
 2 files changed, 5 insertions(+), 12 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 1e3842851dae..a743179b172a 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -443,7 +443,8 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 		return false;
 
 	if (CC((save->cr0 & X86_CR0_CD) == 0 && (save->cr0 & X86_CR0_NW)) ||
-	    CC(save->cr0 & ~0xffffffffULL))
+	    CC(save->cr0 & ~0xffffffffULL) ||
+	    CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3)))
 		return false;
 
 	if (CC(!kvm_dr6_valid(save->dr6)) || CC(!kvm_dr7_valid(save->dr7)))
@@ -456,8 +457,7 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 	 */
 	if ((save->efer & EFER_LME) && (save->cr0 & X86_CR0_PG)) {
 		if (CC(!(save->cr4 & X86_CR4_PAE)) ||
-		    CC(!(save->cr0 & X86_CR0_PE)) ||
-		    CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3)))
+		    CC(!(save->cr0 & X86_CR0_PE)))
 			return false;
 
 		if (CC((save->cs.attrib & SVM_SELECTOR_L_MASK) &&
@@ -717,9 +717,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
 static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_npt, bool reload_pdptrs)
 {
-	if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3)))
-		return -EINVAL;
-
 	if (reload_pdptrs && is_pae_paging(vcpu)) {
 		if (nested_npt)
 			kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index f0d75f5a968c..ef4ca1906fb0 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1235,11 +1235,6 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_ept, bool reload_pdptrs,
 			       enum vm_entry_failure_code *entry_failure_code)
 {
-	if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) {
-		*entry_failure_code = ENTRY_FAIL_DEFAULT;
-		return -EINVAL;
-	}
-
 	/*
 	 * If PAE paging and EPT are both on, CR3 is not used by the CPU and
 	 * must not be dereferenced.
@@ -3323,7 +3318,8 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu,
 	*entry_failure_code = ENTRY_FAIL_DEFAULT;
 
 	if (CC(!nested_guest_cr0_valid(vcpu, vmcs12->guest_cr0)) ||
-	    CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4)))
+	    CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4)) ||
+	    CC(!kvm_vcpu_is_legal_cr3(vcpu, vmcs12->guest_cr3)))
 		return -EINVAL;
 
 	if (CC(vmcs12->guest_cr4 & X86_CR4_CET && !(vmcs12->guest_cr0 & X86_CR0_WP)))
-- 
2.56.1.703.gf848ed807c-goog


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2 3/5] KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions
  2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 2/5] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 4/5] KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested xfer Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 5/5] KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested transitions Sean Christopherson
  4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
  To: Sean Christopherson, Paolo Bonzini
  Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason

Mark CR3 as dirty when loading CR3 on nested VMRUN and nested #VMEXIT, even
though svm_load_mmu_pgd() unconditionally updates vmcb.save.CR3, i.e. never
consumes the dirty bit.  This will allow deduplicating nearly-identical
code between nVMX and nSVM with respect to loading CR3.

Note, VMX consumes the dirty flag in kvm_mmu_new_pgd(), i.e. having nVMX
mark CR3 after the fact isn't a viable option.  Marking CR3 dirty in nVMX
code before invoking the to-be-common helper also isn't safe in general, as
it's theoretically possible KVM could observe+consume the dirty flag before
vcpu->arch.cr3 is updated (even though that's extremely unlikely to ever be
the case).

Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index a743179b172a..32f95c9d72be 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -725,6 +725,7 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	}
 
 	vcpu->arch.cr3 = cr3;
+	kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
 
 	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
 	kvm_init_mmu(vcpu);
-- 
2.56.1.703.gf848ed807c-goog


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2 4/5] KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested xfer
  2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
                   ` (2 preceding siblings ...)
  2026-11-17 21:09 ` [PATCH v2 3/5] KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
  2026-11-17 21:09 ` [PATCH v2 5/5] KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested transitions Sean Christopherson
  4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
  To: Sean Christopherson, Paolo Bonzini
  Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason

Dedup nested_{svm,vmx}_load_cr3() into kvm_nested_load_cr3(), and drop
several MMU exports that existed purely to support the related vendor code.
Except for slight differences around PDPTRs, the logic and reasoning for
handling CR3 and PDPTRs on nested transitions is identical between Intel
and AMD.

Add a KVM_BUG_ON() in nested_vmx_load_cr3() to ensure that the new helper
doesn't unexpectedly fail when PDPTRs aren't being reloaded.

No functional change intended (the new KVM_BUG_ON() is unreachable).

Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/mmu/mmu.c    |  2 --
 arch/x86/kvm/regs.c       | 24 ++++++++++++++++++++++++
 arch/x86/kvm/regs.h       |  2 ++
 arch/x86/kvm/svm/nested.c | 23 +++--------------------
 arch/x86/kvm/vmx/nested.c | 23 ++---------------------
 5 files changed, 31 insertions(+), 43 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index f4b12d6aef55..9186e8e2ec10 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5475,7 +5475,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd)
 			__clear_sp_write_flooding_count(sp);
 	}
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_new_pgd);
 
 static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,
 			   unsigned int access)
@@ -6275,7 +6274,6 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu)
 			init_kvm_shadow_mmu(vcpu, cpu_role);
 	}
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_mmu);
 
 void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu)
 {
diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 3621f1a47d37..b43ad11f061c 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -201,6 +201,30 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs);
 
+int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
+			bool nested_tdp, bool reload_pdptrs)
+{
+	/*
+	 * If TDP is enabled, then the PDPTRs either come from the VMCS (EPT)
+	 * or are read from memory on-demand (NPT), i.e. aren't loaded here.
+	 */
+	if (reload_pdptrs && !nested_tdp && is_pae_paging(vcpu) &&
+	    !load_pdptrs(vcpu, cr3))
+		return -EINVAL;
+
+	vcpu->arch.cr3 = cr3;
+	kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
+
+	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
+	kvm_init_mmu(vcpu);
+
+	if (!nested_tdp)
+		kvm_mmu_new_pgd(vcpu, cr3);
+
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nested_load_cr3);
+
 static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
 {
 #ifdef CONFIG_X86_64
diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h
index 97c17b120a57..3c34ca7d8a8e 100644
--- a/arch/x86/kvm/regs.h
+++ b/arch/x86/kvm/regs.h
@@ -64,6 +64,8 @@ unsigned long kvm_get_dr6_fixed_1(struct kvm_vcpu *vcpu);
 unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu);
 void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw);
 int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
+int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
+			bool nested_tdp, bool reload_pdptrs);
 
 static inline bool is_long_mode(struct kvm_vcpu *vcpu)
 {
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 32f95c9d72be..a045cb5ba9ff 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -710,30 +710,13 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
 }
 
-/*
- * Load guest's/host's cr3 on nested vmentry or vmexit. @nested_npt is true
- * if we are emulating VM-Entry into a guest with NPT enabled.
- */
 static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_npt, bool reload_pdptrs)
 {
-	if (reload_pdptrs && is_pae_paging(vcpu)) {
-		if (nested_npt)
-			kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
-		else if (CC(!load_pdptrs(vcpu, cr3)))
-			return -EINVAL;
-	}
+	if (reload_pdptrs && is_pae_paging(vcpu) && nested_npt)
+		kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
 
-	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
-
-	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
-	kvm_init_mmu(vcpu);
-
-	if (!nested_npt)
-		kvm_mmu_new_pgd(vcpu, cr3);
-
-	return 0;
+	return kvm_nested_load_cr3(vcpu, cr3, nested_npt, reload_pdptrs);
 }
 
 static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index ef4ca1906fb0..6adfd1fe7fa6 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1225,35 +1225,16 @@ static bool nested_msr_store_list_has_msr(struct kvm_vcpu *vcpu, u32 msr_index)
 	return false;
 }
 
-/*
- * Load guest's/host's cr3 at nested entry/exit.  @nested_ept is true if we are
- * emulating VM-Entry into a guest with EPT enabled.  On failure, the expected
- * Exit Qualification (for a VM-Entry consistency check VM-Exit) is assigned to
- * @entry_failure_code.
- */
 static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_ept, bool reload_pdptrs,
 			       enum vm_entry_failure_code *entry_failure_code)
 {
-	/*
-	 * If PAE paging and EPT are both on, CR3 is not used by the CPU and
-	 * must not be dereferenced.
-	 */
-	if (reload_pdptrs && !nested_ept && is_pae_paging(vcpu) &&
-	    CC(!load_pdptrs(vcpu, cr3))) {
+	if (CC(kvm_nested_load_cr3(vcpu, cr3, nested_ept, reload_pdptrs))) {
+		KVM_BUG_ON(!reload_pdptrs, vcpu->kvm);
 		*entry_failure_code = ENTRY_FAIL_PDPTE;
 		return -EINVAL;
 	}
 
-	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
-
-	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
-	kvm_init_mmu(vcpu);
-
-	if (!nested_ept)
-		kvm_mmu_new_pgd(vcpu, cr3);
-
 	return 0;
 }
 
-- 
2.56.1.703.gf848ed807c-goog


^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v2 5/5] KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested transitions
  2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
                   ` (3 preceding siblings ...)
  2026-11-17 21:09 ` [PATCH v2 4/5] KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested xfer Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
  4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
  To: Sean Christopherson, Paolo Bonzini
  Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason

Initialize the MMU and PGD (root) before loading PDPTRs when emulating a
nested VM-Enter or VM-Exit to ensure the MMU is up-to-date even if loading
PDPTRs fails.  If a nested VM-Exit fails to load L1's PDPTEs, and L1 ran L2
without EPT/NP, KVM can run L1 on a freed MMU root, i.e. the CPU's page
walk is a use-after-free, if userspace blows past the KVM_EXIT_SHUTDOWN.

                              root_mmu  vmcx01  vmcx02
                              ========  ======  ======
    L1 runs                   R1        R1
    VM-Enter to L2            R2        R1      R2
    memslot deleted, L2 runs  R3        R1      R3      R1, R2 freed
    VM-Exit, bad PDPTEs       R3        R1
    KVM_EXIT_SHUTDOWN         R3        R1
    KVM_RUN enters L1         R3        R1

Unconditionally initializing the MMU if KVM sets CR3 is a-ok, as the nested
VM-Exit paths need exactly that behavior, emulation of VM-Enter generates a
VM-Exit on both SVM and VMX (i.e. KVM is guaranteed to load L1 state and
re-initialize the MMU), and RSM+KVM_SET_NESTED_STATE defer loading PDPTRs
until the next entry, i.e. don't load PDPTRs and thus can't fail.

Note, pre-checking the PDPTRs isn't an option because they come from guest
memory when TDP is disabled, i.e. the PDPTEs in memory can legally change
between the VM-Enter and VM-Exit.

Fixes: 1dc35dacc16b ("KVM: nVMX: check host CR3 on vmentry and vmexit")
Fixes: d82aaef9c88a ("KVM: nSVM: use nested_svm_load_cr3() on guest->host switch")
Reported-by: Chris Mason <mason@kernel.org>
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/regs.c | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index b43ad11f061c..1cbc7b9f6ef4 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -204,23 +204,29 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs);
 int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			bool nested_tdp, bool reload_pdptrs)
 {
+	vcpu->arch.cr3 = cr3;
+	kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
+
+	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
+	kvm_init_mmu(vcpu);
+
+	if (!nested_tdp)
+		kvm_mmu_new_pgd(vcpu, cr3);
+
 	/*
 	 * If TDP is enabled, then the PDPTRs either come from the VMCS (EPT)
 	 * or are read from memory on-demand (NPT), i.e. aren't loaded here.
+	 * Load PDPTRs *after* initializing the MMU and PGD, to ensure the MMU
+	 * is initialized even if the nested transition ultimately fails.  On
+	 * an emulated VM-Exit, failure will put the vCPU in a shutdown state,
+	 * but KVM still needs to keep the MMU up-to-date.  On an emulated
+	 * VM-Entry, failure will generate a nested VM-Exit, i.e. L1's state
+	 * and MMU will be explicitly loaded.
 	 */
 	if (reload_pdptrs && !nested_tdp && is_pae_paging(vcpu) &&
 	    !load_pdptrs(vcpu, cr3))
 		return -EINVAL;
 
-	vcpu->arch.cr3 = cr3;
-	kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
-
-	/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
-	kvm_init_mmu(vcpu);
-
-	if (!nested_tdp)
-		kvm_mmu_new_pgd(vcpu, cr3);
-
 	return 0;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nested_load_cr3);
-- 
2.56.1.703.gf848ed807c-goog


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-17 21:09 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 2/5] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 3/5] KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 4/5] KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested xfer Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 5/5] KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested transitions Sean Christopherson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®