* [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE
2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 2/5] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid Sean Christopherson
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini
Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason
When loading nested state, explicitly check that the current CR3 is valid
before loading any state, as a misbehaving userspace can muck with guest
CPUID to make KVM think the current CR3 is invalid, and thus cause the late
"load" of CR3 to fail when initializing the guest's MMU. If that happens,
KVM will leave half-baked state in the vCPU, e.g. in L2 with L1's page
walker, and if nested NPT is disabled, L1's PGD as well.
Alternatively, KVM could try and unwind vCPU state if the late CR3 "load"
fails, but that is subtly quite difficult as it would require snapshotting
and restoring the previous state. Simply leaving nested on failure isn't
an option either, because that too would leave the vCPU with half-baked
state (in L1 with L2 state and a very stale MMU).
Pre-checking CR3 will also allow hoisting the CR3 check out of
nested_svm_load_cr3() entirely.
WARN and bug the VM if the late CR3 load fails, as loading nested state
never loads PDPTRs, i.e. failure is impossible, and KVM doesn't unwind on
failure (see above). I.e. if the impossible happens, KVM needs to block
all operations on the half-baked vCPU.
Fixes: e1779c2714c3 ("KVM: x86: nSVM: fix potential NULL derefernce on nested migration")
Cc: stable@vger.kernel.org
Reported-by: Stefan Teodorescu <fane@google.com>
Reported-by: Sashiko Bot <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/sashiko-outbox-210535@kernel.org
Closes: https://lore.kernel.org/all/sashiko-outbox-205939@kernel.org
Closes: https://lore.kernel.org/all/sashiko-outbox-193903@kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/nested.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 07554089ec05..1e3842851dae 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -2047,6 +2047,9 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW))
goto out_free;
+ if (!kvm_vcpu_is_legal_cr3(vcpu, vcpu->arch.cr3))
+ goto out_free;
+
/*
* Validate host state saved from before VMRUN (see
* nested_svm_check_permissions). Note that the g_pat field is not
@@ -2114,7 +2117,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
*/
ret = nested_svm_load_cr3(&svm->vcpu, vcpu->arch.cr3,
nested_npt_enabled(svm), false);
- if (ret)
+ if (KVM_BUG_ON(ret, vcpu->kvm))
goto out_free;
svm->nested.force_msr_bitmap_recalc = true;
--
2.56.1.703.gf848ed807c-goog
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH v2 2/5] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid
2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 3/5] KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions Sean Christopherson
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini
Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason
From: Paolo Bonzini <pbonzini@redhat.com>
nested_vmcb_check_save() checks CR3 legality only if long mode is active,
but it really needs to be checked in all cases - the check uses must be
zero bits of the host, and indeed nested_svm_load_cr3() later will
reject illegal CR3 in every mode. Move the check earlier, and thus out
of nested_svm_load_cr3().
Likewise for nested_vmx_load_cr3() and nested_vmx_check_guest_state().
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Tested-by: syzbot@syzkaller.appspotmail.com
Cc: stable@vger.kernel.org
[sean: tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/nested.c | 9 +++------
arch/x86/kvm/vmx/nested.c | 8 ++------
2 files changed, 5 insertions(+), 12 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 1e3842851dae..a743179b172a 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -443,7 +443,8 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
return false;
if (CC((save->cr0 & X86_CR0_CD) == 0 && (save->cr0 & X86_CR0_NW)) ||
- CC(save->cr0 & ~0xffffffffULL))
+ CC(save->cr0 & ~0xffffffffULL) ||
+ CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3)))
return false;
if (CC(!kvm_dr6_valid(save->dr6)) || CC(!kvm_dr7_valid(save->dr7)))
@@ -456,8 +457,7 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
*/
if ((save->efer & EFER_LME) && (save->cr0 & X86_CR0_PG)) {
if (CC(!(save->cr4 & X86_CR4_PAE)) ||
- CC(!(save->cr0 & X86_CR0_PE)) ||
- CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3)))
+ CC(!(save->cr0 & X86_CR0_PE)))
return false;
if (CC((save->cs.attrib & SVM_SELECTOR_L_MASK) &&
@@ -717,9 +717,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
bool nested_npt, bool reload_pdptrs)
{
- if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3)))
- return -EINVAL;
-
if (reload_pdptrs && is_pae_paging(vcpu)) {
if (nested_npt)
kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index f0d75f5a968c..ef4ca1906fb0 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1235,11 +1235,6 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
bool nested_ept, bool reload_pdptrs,
enum vm_entry_failure_code *entry_failure_code)
{
- if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) {
- *entry_failure_code = ENTRY_FAIL_DEFAULT;
- return -EINVAL;
- }
-
/*
* If PAE paging and EPT are both on, CR3 is not used by the CPU and
* must not be dereferenced.
@@ -3323,7 +3318,8 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu,
*entry_failure_code = ENTRY_FAIL_DEFAULT;
if (CC(!nested_guest_cr0_valid(vcpu, vmcs12->guest_cr0)) ||
- CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4)))
+ CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4)) ||
+ CC(!kvm_vcpu_is_legal_cr3(vcpu, vmcs12->guest_cr3)))
return -EINVAL;
if (CC(vmcs12->guest_cr4 & X86_CR4_CET && !(vmcs12->guest_cr0 & X86_CR0_WP)))
--
2.56.1.703.gf848ed807c-goog
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH v2 3/5] KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions
2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 1/5] KVM: nSVM: Pre-check current (L2's) CR3 during KVM_SET_NESTED_STATE Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 2/5] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 4/5] KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested xfer Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 5/5] KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested transitions Sean Christopherson
4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini
Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason
Mark CR3 as dirty when loading CR3 on nested VMRUN and nested #VMEXIT, even
though svm_load_mmu_pgd() unconditionally updates vmcb.save.CR3, i.e. never
consumes the dirty bit. This will allow deduplicating nearly-identical
code between nVMX and nSVM with respect to loading CR3.
Note, VMX consumes the dirty flag in kvm_mmu_new_pgd(), i.e. having nVMX
mark CR3 after the fact isn't a viable option. Marking CR3 dirty in nVMX
code before invoking the to-be-common helper also isn't safe in general, as
it's theoretically possible KVM could observe+consume the dirty flag before
vcpu->arch.cr3 is updated (even though that's extremely unlikely to ever be
the case).
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/nested.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index a743179b172a..32f95c9d72be 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -725,6 +725,7 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
}
vcpu->arch.cr3 = cr3;
+ kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
/* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
kvm_init_mmu(vcpu);
--
2.56.1.703.gf848ed807c-goog
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 4/5] KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested xfer
2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
` (2 preceding siblings ...)
2026-11-17 21:09 ` [PATCH v2 3/5] KVM: nSVM: Mark CR3 as dirty when loading CR3 on nested transitions Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
2026-11-17 21:09 ` [PATCH v2 5/5] KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested transitions Sean Christopherson
4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini
Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason
Dedup nested_{svm,vmx}_load_cr3() into kvm_nested_load_cr3(), and drop
several MMU exports that existed purely to support the related vendor code.
Except for slight differences around PDPTRs, the logic and reasoning for
handling CR3 and PDPTRs on nested transitions is identical between Intel
and AMD.
Add a KVM_BUG_ON() in nested_vmx_load_cr3() to ensure that the new helper
doesn't unexpectedly fail when PDPTRs aren't being reloaded.
No functional change intended (the new KVM_BUG_ON() is unreachable).
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/mmu/mmu.c | 2 --
arch/x86/kvm/regs.c | 24 ++++++++++++++++++++++++
arch/x86/kvm/regs.h | 2 ++
arch/x86/kvm/svm/nested.c | 23 +++--------------------
arch/x86/kvm/vmx/nested.c | 23 ++---------------------
5 files changed, 31 insertions(+), 43 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index f4b12d6aef55..9186e8e2ec10 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5475,7 +5475,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd)
__clear_sp_write_flooding_count(sp);
}
}
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_new_pgd);
static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn,
unsigned int access)
@@ -6275,7 +6274,6 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu)
init_kvm_shadow_mmu(vcpu, cpu_role);
}
}
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_mmu);
void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu)
{
diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 3621f1a47d37..b43ad11f061c 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -201,6 +201,30 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs);
+int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
+ bool nested_tdp, bool reload_pdptrs)
+{
+ /*
+ * If TDP is enabled, then the PDPTRs either come from the VMCS (EPT)
+ * or are read from memory on-demand (NPT), i.e. aren't loaded here.
+ */
+ if (reload_pdptrs && !nested_tdp && is_pae_paging(vcpu) &&
+ !load_pdptrs(vcpu, cr3))
+ return -EINVAL;
+
+ vcpu->arch.cr3 = cr3;
+ kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
+
+ /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
+ kvm_init_mmu(vcpu);
+
+ if (!nested_tdp)
+ kvm_mmu_new_pgd(vcpu, cr3);
+
+ return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nested_load_cr3);
+
static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
{
#ifdef CONFIG_X86_64
diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h
index 97c17b120a57..3c34ca7d8a8e 100644
--- a/arch/x86/kvm/regs.h
+++ b/arch/x86/kvm/regs.h
@@ -64,6 +64,8 @@ unsigned long kvm_get_dr6_fixed_1(struct kvm_vcpu *vcpu);
unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu);
void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw);
int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
+int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
+ bool nested_tdp, bool reload_pdptrs);
static inline bool is_long_mode(struct kvm_vcpu *vcpu)
{
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 32f95c9d72be..a045cb5ba9ff 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -710,30 +710,13 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
}
-/*
- * Load guest's/host's cr3 on nested vmentry or vmexit. @nested_npt is true
- * if we are emulating VM-Entry into a guest with NPT enabled.
- */
static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
bool nested_npt, bool reload_pdptrs)
{
- if (reload_pdptrs && is_pae_paging(vcpu)) {
- if (nested_npt)
- kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
- else if (CC(!load_pdptrs(vcpu, cr3)))
- return -EINVAL;
- }
+ if (reload_pdptrs && is_pae_paging(vcpu) && nested_npt)
+ kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
- vcpu->arch.cr3 = cr3;
- kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
-
- /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
- kvm_init_mmu(vcpu);
-
- if (!nested_npt)
- kvm_mmu_new_pgd(vcpu, cr3);
-
- return 0;
+ return kvm_nested_load_cr3(vcpu, cr3, nested_npt, reload_pdptrs);
}
static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index ef4ca1906fb0..6adfd1fe7fa6 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1225,35 +1225,16 @@ static bool nested_msr_store_list_has_msr(struct kvm_vcpu *vcpu, u32 msr_index)
return false;
}
-/*
- * Load guest's/host's cr3 at nested entry/exit. @nested_ept is true if we are
- * emulating VM-Entry into a guest with EPT enabled. On failure, the expected
- * Exit Qualification (for a VM-Entry consistency check VM-Exit) is assigned to
- * @entry_failure_code.
- */
static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
bool nested_ept, bool reload_pdptrs,
enum vm_entry_failure_code *entry_failure_code)
{
- /*
- * If PAE paging and EPT are both on, CR3 is not used by the CPU and
- * must not be dereferenced.
- */
- if (reload_pdptrs && !nested_ept && is_pae_paging(vcpu) &&
- CC(!load_pdptrs(vcpu, cr3))) {
+ if (CC(kvm_nested_load_cr3(vcpu, cr3, nested_ept, reload_pdptrs))) {
+ KVM_BUG_ON(!reload_pdptrs, vcpu->kvm);
*entry_failure_code = ENTRY_FAIL_PDPTE;
return -EINVAL;
}
- vcpu->arch.cr3 = cr3;
- kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
-
- /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
- kvm_init_mmu(vcpu);
-
- if (!nested_ept)
- kvm_mmu_new_pgd(vcpu, cr3);
-
return 0;
}
--
2.56.1.703.gf848ed807c-goog
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH v2 5/5] KVM: x86: Initialize MMU+PGD before loading PDPTRs on nested transitions
2026-11-17 21:09 [PATCH v2 0/5] KVM: x86: Fix nested CR3 load bugs Sean Christopherson
` (3 preceding siblings ...)
2026-11-17 21:09 ` [PATCH v2 4/5] KVM: x86: Consolidate nVMX vs. nSVM logic for loading CR3 on nested xfer Sean Christopherson
@ 2026-11-17 21:09 ` Sean Christopherson
4 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2026-11-17 21:09 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini
Cc: kvm, linux-kernel, Stefan Teodorescu, Sashiko Bot, syzbot, Chris Mason
Initialize the MMU and PGD (root) before loading PDPTRs when emulating a
nested VM-Enter or VM-Exit to ensure the MMU is up-to-date even if loading
PDPTRs fails. If a nested VM-Exit fails to load L1's PDPTEs, and L1 ran L2
without EPT/NP, KVM can run L1 on a freed MMU root, i.e. the CPU's page
walk is a use-after-free, if userspace blows past the KVM_EXIT_SHUTDOWN.
root_mmu vmcx01 vmcx02
======== ====== ======
L1 runs R1 R1
VM-Enter to L2 R2 R1 R2
memslot deleted, L2 runs R3 R1 R3 R1, R2 freed
VM-Exit, bad PDPTEs R3 R1
KVM_EXIT_SHUTDOWN R3 R1
KVM_RUN enters L1 R3 R1
Unconditionally initializing the MMU if KVM sets CR3 is a-ok, as the nested
VM-Exit paths need exactly that behavior, emulation of VM-Enter generates a
VM-Exit on both SVM and VMX (i.e. KVM is guaranteed to load L1 state and
re-initialize the MMU), and RSM+KVM_SET_NESTED_STATE defer loading PDPTRs
until the next entry, i.e. don't load PDPTRs and thus can't fail.
Note, pre-checking the PDPTRs isn't an option because they come from guest
memory when TDP is disabled, i.e. the PDPTEs in memory can legally change
between the VM-Enter and VM-Exit.
Fixes: 1dc35dacc16b ("KVM: nVMX: check host CR3 on vmentry and vmexit")
Fixes: d82aaef9c88a ("KVM: nSVM: use nested_svm_load_cr3() on guest->host switch")
Reported-by: Chris Mason <mason@kernel.org>
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/regs.c | 24 +++++++++++++++---------
1 file changed, 15 insertions(+), 9 deletions(-)
diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index b43ad11f061c..1cbc7b9f6ef4 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -204,23 +204,29 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs);
int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
bool nested_tdp, bool reload_pdptrs)
{
+ vcpu->arch.cr3 = cr3;
+ kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
+
+ /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
+ kvm_init_mmu(vcpu);
+
+ if (!nested_tdp)
+ kvm_mmu_new_pgd(vcpu, cr3);
+
/*
* If TDP is enabled, then the PDPTRs either come from the VMCS (EPT)
* or are read from memory on-demand (NPT), i.e. aren't loaded here.
+ * Load PDPTRs *after* initializing the MMU and PGD, to ensure the MMU
+ * is initialized even if the nested transition ultimately fails. On
+ * an emulated VM-Exit, failure will put the vCPU in a shutdown state,
+ * but KVM still needs to keep the MMU up-to-date. On an emulated
+ * VM-Entry, failure will generate a nested VM-Exit, i.e. L1's state
+ * and MMU will be explicitly loaded.
*/
if (reload_pdptrs && !nested_tdp && is_pae_paging(vcpu) &&
!load_pdptrs(vcpu, cr3))
return -EINVAL;
- vcpu->arch.cr3 = cr3;
- kvm_register_mark_dirty(vcpu, VCPU_REG_CR3);
-
- /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */
- kvm_init_mmu(vcpu);
-
- if (!nested_tdp)
- kvm_mmu_new_pgd(vcpu, cr3);
-
return 0;
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nested_load_cr3);
--
2.56.1.703.gf848ed807c-goog
^ permalink raw reply [flat|nested] 6+ messages in thread