[PATCH v2 0/8] KVM: SVM: A fix and cleanups for VMCB intercepts

[PATCH v2 0/8] KVM: SVM: A fix and cleanups for VMCB intercepts

[Sean Christopherson]

Fix a likely-benign bug where KVM fails to mark vmcb01 intercepts as dirty
after recalculating intercepts while L2 is active, then do a bunch of related
cleanup, e.g. to split recalc_intercepts() into nested vs. non-nested
functionality.

v2:
 - Fix the aforementioned bug.
 - Split recalc_intercepts() instead of simply renaming it.
 - Move the new WARN in nested_vmcb02_recalc_intercepts() to its own patch.
 - Use less weird local variables even if they aren't consistent with the
   existing code...
 - ... and then change some names in the existing code to provide consistency.

v1: https://lkml.kernel.org/r/20260112182022.771276-1-yosry.ahmed%40linux.dev

Sean Christopherson (6):
  KVM: SVM: Explicitly mark vmcb01 dirty after modifying VMCB intercepts
  KVM: SVM: Separate recalc_intercepts() into nested vs. non-nested
    parts
  KVM: nSVM: Directly (re)calc vmcb02 intercepts from
    nested_vmcb02_prepare_control()
  KVM: nSVM: Use intuitive local variables in
    nested_vmcb02_recalc_intercepts()
  KVM: nSVM: Move vmcb_ctrl_area_cached.bus_lock_rip to svm_nested_state
  KVM: nSVM: Capture svm->nested.ctl as vmcb12_ctrl when preparing
    vmcb02

Yosry Ahmed (2):
  KVM: nSVM: WARN and abort vmcb02 intercepts recalc if vmcb02 isn't
    active
  KVM: nSVM: Use vmcb12_is_intercept() in
    nested_sync_control_from_vmcb02()

 arch/x86/kvm/svm/nested.c | 88 +++++++++++++++++++--------------------
 arch/x86/kvm/svm/sev.c    |  2 +-
 arch/x86/kvm/svm/svm.c    |  6 +--
 arch/x86/kvm/svm/svm.h    | 28 +++++++++----
 4 files changed, 67 insertions(+), 57 deletions(-)


base-commit: 183bb0ce8c77b0fd1fb25874112bc8751a461e49
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 1/8] KVM: SVM: Explicitly mark vmcb01 dirty after modifying VMCB intercepts

[Sean Christopherson]

When reacting to an intercept update, explicitly mark vmcb01's intercepts
dirty, as KVM always initially operates on vmcb01, and nested_svm_vmexit()
isn't guaranteed to mark VMCB_INTERCEPTS as dirty.  I.e. if L2 is active,
KVM will modify the intercepts for L1, but might not mark them as dirty
before the next VMRUN of L1.

Fixes: 116a0a23676e ("KVM: SVM: Add clean-bit for intercetps, tsc-offset and pause filter count")
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index de90b104a0dd..66701106a51b 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -128,11 +128,13 @@ void recalc_intercepts(struct vcpu_svm *svm)
 	struct vmcb_ctrl_area_cached *g;
 	unsigned int i;
 
-	vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS);
+	vmcb_mark_dirty(svm->vmcb01.ptr, VMCB_INTERCEPTS);
 
 	if (!is_guest_mode(&svm->vcpu))
 		return;
 
+	vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS);
+
 	c = &svm->vmcb->control;
 	h = &svm->vmcb01.ptr->control;
 	g = &svm->nested.ctl;
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 2/8] KVM: SVM: Separate recalc_intercepts() into nested vs. non-nested parts

[Sean Christopherson]

Extract the non-nested aspects of recalc_intercepts() into a separate
helper, svm_mark_intercepts_dirty(), to make it clear that the call isn't
*just* recalculating (vmcb02's) intercepts, and to not bury non-nested
code in nested.c.

As suggested by Yosry, opportunistically prepend "nested_vmbc02_" to
recalc_intercepts() so that it's obvious the function specifically deals
with recomputing intercepts for L2.

No functional change intended.

Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c |  9 ++-------
 arch/x86/kvm/svm/sev.c    |  2 +-
 arch/x86/kvm/svm/svm.c    |  4 ++--
 arch/x86/kvm/svm/svm.h    | 26 ++++++++++++++++++++------
 4 files changed, 25 insertions(+), 16 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 66701106a51b..48b60dd6e7a3 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -122,17 +122,12 @@ static bool nested_vmcb_needs_vls_intercept(struct vcpu_svm *svm)
 	return false;
 }
 
-void recalc_intercepts(struct vcpu_svm *svm)
+void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)
 {
 	struct vmcb_control_area *c, *h;
 	struct vmcb_ctrl_area_cached *g;
 	unsigned int i;
 
-	vmcb_mark_dirty(svm->vmcb01.ptr, VMCB_INTERCEPTS);
-
-	if (!is_guest_mode(&svm->vcpu))
-		return;
-
 	vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS);
 
 	c = &svm->vmcb->control;
@@ -918,7 +913,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	 * Merge guest and host intercepts - must be called with vcpu in
 	 * guest-mode to take effect.
 	 */
-	recalc_intercepts(svm);
+	svm_mark_intercepts_dirty(svm);
 }
 
 static void nested_svm_copy_common_state(struct vmcb *from_vmcb, struct vmcb *to_vmcb)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index ea515cf41168..03b6dc75a6e8 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4639,7 +4639,7 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm, bool init_event)
 	if (!sev_vcpu_has_debug_swap(svm)) {
 		vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_READ);
 		vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_WRITE);
-		recalc_intercepts(svm);
+		svm_mark_intercepts_dirty(svm);
 	} else {
 		/*
 		 * Disable #DB intercept iff DebugSwap is enabled.  KVM doesn't
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 8f8bc863e214..9e76bf1671da 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -635,7 +635,7 @@ static void set_dr_intercepts(struct vcpu_svm *svm)
 	vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_READ);
 	vmcb_set_intercept(&vmcb->control, INTERCEPT_DR7_WRITE);
 
-	recalc_intercepts(svm);
+	svm_mark_intercepts_dirty(svm);
 }
 
 static void clr_dr_intercepts(struct vcpu_svm *svm)
@@ -644,7 +644,7 @@ static void clr_dr_intercepts(struct vcpu_svm *svm)
 
 	vmcb->control.intercepts[INTERCEPT_DR] = 0;
 
-	recalc_intercepts(svm);
+	svm_mark_intercepts_dirty(svm);
 }
 
 static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index ebd7b36b1ceb..92a1691dc7be 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -357,8 +357,6 @@ struct svm_cpu_data {
 
 DECLARE_PER_CPU(struct svm_cpu_data, svm_data);
 
-void recalc_intercepts(struct vcpu_svm *svm);
-
 static __always_inline struct kvm_svm *to_kvm_svm(struct kvm *kvm)
 {
 	return container_of(kvm, struct kvm_svm, kvm);
@@ -486,6 +484,22 @@ static inline bool vmcb12_is_intercept(struct vmcb_ctrl_area_cached *control, u3
 	return __vmcb_is_intercept((unsigned long *)&control->intercepts, bit);
 }
 
+void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm);
+
+static inline void svm_mark_intercepts_dirty(struct vcpu_svm *svm)
+{
+	vmcb_mark_dirty(svm->vmcb01.ptr, VMCB_INTERCEPTS);
+
+	/*
+	 * If L2 is active, recalculate the intercepts for vmcb02 to account
+	 * for the changes made to vmcb01.  All intercept configuration is done
+	 * for vmcb01 and then propagated to vmcb02 to combine KVM's intercepts
+	 * with L1's intercepts (from the vmcb12 snapshot).
+	 */
+	if (is_guest_mode(&svm->vcpu))
+		nested_vmcb02_recalc_intercepts(svm);
+}
+
 static inline void set_exception_intercept(struct vcpu_svm *svm, u32 bit)
 {
 	struct vmcb *vmcb = svm->vmcb01.ptr;
@@ -493,7 +507,7 @@ static inline void set_exception_intercept(struct vcpu_svm *svm, u32 bit)
 	WARN_ON_ONCE(bit >= 32);
 	vmcb_set_intercept(&vmcb->control, INTERCEPT_EXCEPTION_OFFSET + bit);
 
-	recalc_intercepts(svm);
+	svm_mark_intercepts_dirty(svm);
 }
 
 static inline void clr_exception_intercept(struct vcpu_svm *svm, u32 bit)
@@ -503,7 +517,7 @@ static inline void clr_exception_intercept(struct vcpu_svm *svm, u32 bit)
 	WARN_ON_ONCE(bit >= 32);
 	vmcb_clr_intercept(&vmcb->control, INTERCEPT_EXCEPTION_OFFSET + bit);
 
-	recalc_intercepts(svm);
+	svm_mark_intercepts_dirty(svm);
 }
 
 static inline void svm_set_intercept(struct vcpu_svm *svm, int bit)
@@ -512,7 +526,7 @@ static inline void svm_set_intercept(struct vcpu_svm *svm, int bit)
 
 	vmcb_set_intercept(&vmcb->control, bit);
 
-	recalc_intercepts(svm);
+	svm_mark_intercepts_dirty(svm);
 }
 
 static inline void svm_clr_intercept(struct vcpu_svm *svm, int bit)
@@ -521,7 +535,7 @@ static inline void svm_clr_intercept(struct vcpu_svm *svm, int bit)
 
 	vmcb_clr_intercept(&vmcb->control, bit);
 
-	recalc_intercepts(svm);
+	svm_mark_intercepts_dirty(svm);
 }
 
 static inline bool svm_is_intercept(struct vcpu_svm *svm, int bit)
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 3/8] KVM: nSVM: WARN and abort vmcb02 intercepts recalc if vmcb02 isn't active

[Sean Christopherson]

From: Yosry Ahmed <yosry.ahmed@linux.dev>

WARN and bail early from nested_vmcb02_recalc_intercepts() if vmcb02 isn't
the active/current VMCB, as recalculating intercepts for vmcb01 using logic
intended for merging vmcb12 and vmcb01 intercepts can yield unexpected and
unwanted results.

In addition to hardening against general bugs, this will provide additional
safeguards "if" nested_vmcb02_recalc_intercepts() is invoked directly from
nested_vmcb02_prepare_control().

Signed-off-by: Yosry Ahmed <yosry.ahmed@linux.dev>
[sean: split to separate patch, bail early on "failure"]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 48b60dd6e7a3..793f5d2eed3a 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -128,6 +128,9 @@ void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)
 	struct vmcb_ctrl_area_cached *g;
 	unsigned int i;
 
+	if (WARN_ON_ONCE(svm->vmcb != svm->nested.vmcb02.ptr))
+		return;
+
 	vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS);
 
 	c = &svm->vmcb->control;
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 4/8] KVM: nSVM: Directly (re)calc vmcb02 intercepts from nested_vmcb02_prepare_control()

[Sean Christopherson]

Now that nested_vmcb02_recalc_intercepts() provides guardrails against it
being incorrectly called without vmcb02 active, invoke it directly from
nested_vmcb02_recalc_intercepts() instead of bouncing through
svm_mark_intercepts_dirty(), which unnecessarily marks vmcb01 as dirty.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 793f5d2eed3a..e8512de5aef7 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -916,7 +916,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	 * Merge guest and host intercepts - must be called with vcpu in
 	 * guest-mode to take effect.
 	 */
-	svm_mark_intercepts_dirty(svm);
+	nested_vmcb02_recalc_intercepts(svm);
 }
 
 static void nested_svm_copy_common_state(struct vmcb *from_vmcb, struct vmcb *to_vmcb)
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 5/8] KVM: nSVM: Use intuitive local variables in nested_vmcb02_recalc_intercepts()

[Sean Christopherson]

Now that nested_vmcb02_recalc_intercepts() is explicitly scoped to deal
with *only* recalculating vmcb02 intercepts, rename its local variables to
use more intuivite names.  The current "c", "h", and "g" local variables,
for the current VMCB, vmcb01, and (cached) vmcb12 respectively, are short
and sweet, but don't do much to help unfamiliar readers understand what
the code is doing.

Use vmcb12_ctrl/vmcb01/vmcb02/vmcb12_ctrl in lieu of c/h/g to make it clear
the function is updating intercepts in vmcb02 based on the intercepts in
vmcb01 and (cached) vmcb12.

Opportunistically change the existing WARN_ON to a WARN_ON_ONCE so that a
KVM bug doesn't unintentionally DoS the host.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry.ahmed@linux.dev>
[sean: use WARN_ON_ONCE, keep local vmcb12 cache as vmcb12_ctrl]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 35 ++++++++++++++++-------------------
 1 file changed, 16 insertions(+), 19 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index e8512de5aef7..bda2d6d613c9 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -124,23 +124,20 @@ static bool nested_vmcb_needs_vls_intercept(struct vcpu_svm *svm)
 
 void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)
 {
-	struct vmcb_control_area *c, *h;
-	struct vmcb_ctrl_area_cached *g;
+	struct vmcb_ctrl_area_cached *vmcb12_ctrl = &svm->nested.ctl;
+	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
+	struct vmcb *vmcb01 = svm->vmcb01.ptr;
 	unsigned int i;
 
-	if (WARN_ON_ONCE(svm->vmcb != svm->nested.vmcb02.ptr))
+	if (WARN_ON_ONCE(svm->vmcb != vmcb02))
 		return;
 
-	vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS);
-
-	c = &svm->vmcb->control;
-	h = &svm->vmcb01.ptr->control;
-	g = &svm->nested.ctl;
+	vmcb_mark_dirty(vmcb02, VMCB_INTERCEPTS);
 
 	for (i = 0; i < MAX_INTERCEPT; i++)
-		c->intercepts[i] = h->intercepts[i];
+		vmcb02->control.intercepts[i] = vmcb01->control.intercepts[i];
 
-	if (g->int_ctl & V_INTR_MASKING_MASK) {
+	if (vmcb12_ctrl->int_ctl & V_INTR_MASKING_MASK) {
 		/*
 		 * If L2 is active and V_INTR_MASKING is enabled in vmcb12,
 		 * disable intercept of CR8 writes as L2's CR8 does not affect
@@ -151,9 +148,9 @@ void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)
 		 * the effective RFLAGS.IF for L1 interrupts will never be set
 		 * while L2 is running (L2's RFLAGS.IF doesn't affect L1 IRQs).
 		 */
-		vmcb_clr_intercept(c, INTERCEPT_CR8_WRITE);
-		if (!(svm->vmcb01.ptr->save.rflags & X86_EFLAGS_IF))
-			vmcb_clr_intercept(c, INTERCEPT_VINTR);
+		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_CR8_WRITE);
+		if (!(vmcb01->save.rflags & X86_EFLAGS_IF))
+			vmcb_clr_intercept(&vmcb02->control, INTERCEPT_VINTR);
 	}
 
 	/*
@@ -161,14 +158,14 @@ void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)
 	 * flush feature is enabled.
 	 */
 	if (!nested_svm_l2_tlb_flush_enabled(&svm->vcpu))
-		vmcb_clr_intercept(c, INTERCEPT_VMMCALL);
+		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_VMMCALL);
 
 	for (i = 0; i < MAX_INTERCEPT; i++)
-		c->intercepts[i] |= g->intercepts[i];
+		vmcb02->control.intercepts[i] |= vmcb12_ctrl->intercepts[i];
 
 	/* If SMI is not intercepted, ignore guest SMI intercept as well  */
 	if (!intercept_smi)
-		vmcb_clr_intercept(c, INTERCEPT_SMI);
+		vmcb_clr_intercept(&vmcb02->control, INTERCEPT_SMI);
 
 	if (nested_vmcb_needs_vls_intercept(svm)) {
 		/*
@@ -176,10 +173,10 @@ void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)
 		 * we must intercept these instructions to correctly
 		 * emulate them in case L1 doesn't intercept them.
 		 */
-		vmcb_set_intercept(c, INTERCEPT_VMLOAD);
-		vmcb_set_intercept(c, INTERCEPT_VMSAVE);
+		vmcb_set_intercept(&vmcb02->control, INTERCEPT_VMLOAD);
+		vmcb_set_intercept(&vmcb02->control, INTERCEPT_VMSAVE);
 	} else {
-		WARN_ON(!(c->virt_ext & VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK));
+		WARN_ON_ONCE(!(vmcb02->control.virt_ext & VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK));
 	}
 }
 
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 6/8] KVM: nSVM: Use vmcb12_is_intercept() in nested_sync_control_from_vmcb02()

[Sean Christopherson]

From: Yosry Ahmed <yosry.ahmed@linux.dev>

Use vmcb12_is_intercept() instead of open-coding the intercept check.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index bda2d6d613c9..bbb8dfc9979b 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -530,7 +530,7 @@ void nested_sync_control_from_vmcb02(struct vcpu_svm *svm)
 	 * int_ctl (because it was never recognized while L2 was running).
 	 */
 	if (svm_is_intercept(svm, INTERCEPT_VINTR) &&
-	    !test_bit(INTERCEPT_VINTR, (unsigned long *)svm->nested.ctl.intercepts))
+	    !vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_VINTR))
 		mask &= ~V_IRQ_MASK;
 
 	if (nested_vgif_enabled(svm))
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 7/8] KVM: nSVM: Move vmcb_ctrl_area_cached.bus_lock_rip to svm_nested_state

[Sean Christopherson]

Move "bus_lock_rip" from "vmcb_ctrl_area_cached" to "svm_nested_state" as
"last_bus_lock_rip" to more accurately reflect what it tracks, and because
it is NOT a cached vmcb12 control field.  The misplaced field isn't all
that apparent in the current code base, as KVM uses "svm->nested.ctl"
broadly, but the bad placement becomes glaringly obvious if
"svm->nested.ctl" is captured as a local "vmcb12_ctrl" variable.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 8 ++++----
 arch/x86/kvm/svm/svm.c    | 2 +-
 arch/x86/kvm/svm/svm.h    | 2 +-
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index bbb8dfc9979b..bcd6304f3c0c 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -806,7 +806,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	 * L1 re-enters L2, the same instruction will trigger a VM-Exit and the
 	 * entire cycle start over.
 	 */
-	if (vmcb02->save.rip && (svm->nested.ctl.bus_lock_rip == vmcb02->save.rip))
+	if (vmcb02->save.rip && (svm->nested.last_bus_lock_rip == vmcb02->save.rip))
 		vmcb02->control.bus_lock_counter = 1;
 	else
 		vmcb02->control.bus_lock_counter = 0;
@@ -1191,11 +1191,11 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 	}
 
 	/*
-	 * Invalidate bus_lock_rip unless KVM is still waiting for the guest
-	 * to make forward progress before re-enabling bus lock detection.
+	 * Invalidate last_bus_lock_rip unless KVM is still waiting for the
+	 * guest to make forward progress before re-enabling bus lock detection.
 	 */
 	if (!vmcb02->control.bus_lock_counter)
-		svm->nested.ctl.bus_lock_rip = INVALID_GPA;
+		svm->nested.last_bus_lock_rip = INVALID_GPA;
 
 	nested_svm_copy_common_state(svm->nested.vmcb02.ptr, svm->vmcb01.ptr);
 
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 9e76bf1671da..7c832a0decc2 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3223,7 +3223,7 @@ static int bus_lock_exit(struct kvm_vcpu *vcpu)
 	vcpu->arch.complete_userspace_io = complete_userspace_buslock;
 
 	if (is_guest_mode(vcpu))
-		svm->nested.ctl.bus_lock_rip = vcpu->arch.cui_linear_rip;
+		svm->nested.last_bus_lock_rip = vcpu->arch.cui_linear_rip;
 
 	return 0;
 }
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 92a1691dc7be..c4ed1be38ceb 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -173,7 +173,6 @@ struct vmcb_ctrl_area_cached {
 	u64 nested_cr3;
 	u64 virt_ext;
 	u32 clean;
-	u64 bus_lock_rip;
 	union {
 #if IS_ENABLED(CONFIG_HYPERV) || IS_ENABLED(CONFIG_KVM_HYPERV)
 		struct hv_vmcb_enlightenments hv_enlightenments;
@@ -188,6 +187,7 @@ struct svm_nested_state {
 	u64 vm_cr_msr;
 	u64 vmcb12_gpa;
 	u64 last_vmcb12_gpa;
+	u64 last_bus_lock_rip;
 
 	/*
 	 * The MSR permissions map used for vmcb02, which is the merge result
-- 
2.53.0.345.g96ddfc5eaa-goog

[PATCH v2 8/8] KVM: nSVM: Capture svm->nested.ctl as vmcb12_ctrl when preparing vmcb02

[Sean Christopherson]

Grab svm->nested.ctl as vmcb12_ctrl when preparing the vmcb02 controls to
make it more obvious that much of the data is coming from vmcb12 (or
rather, a snapshot of vmcb12 at the time of L1's VMRUN).

Opportunistically reorder the variable definitions to create a pretty
reverse fir tree.

No functional change intended.

Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 39 +++++++++++++++++++--------------------
 1 file changed, 19 insertions(+), 20 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index bcd6304f3c0c..1814522db6b4 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -745,11 +745,11 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	u32 int_ctl_vmcb01_bits = V_INTR_MASKING_MASK;
 	u32 int_ctl_vmcb12_bits = V_TPR_MASK | V_IRQ_INJECTION_BITS_MASK;
 
-	struct kvm_vcpu *vcpu = &svm->vcpu;
-	struct vmcb *vmcb01 = svm->vmcb01.ptr;
+	struct vmcb_ctrl_area_cached *vmcb12_ctrl = &svm->nested.ctl;
 	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
-	u32 pause_count12;
-	u32 pause_thresh12;
+	struct vmcb *vmcb01 = svm->vmcb01.ptr;
+	struct kvm_vcpu *vcpu = &svm->vcpu;
+	u32 pause_count12, pause_thresh12;
 
 	nested_svm_transition_tlb_flush(vcpu);
 
@@ -762,7 +762,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	 */
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
-	    (svm->nested.ctl.int_ctl & V_GIF_ENABLE_MASK))
+	    (vmcb12_ctrl->int_ctl & V_GIF_ENABLE_MASK))
 		int_ctl_vmcb12_bits |= (V_GIF_MASK | V_GIF_ENABLE_MASK);
 	else
 		int_ctl_vmcb01_bits |= (V_GIF_MASK | V_GIF_ENABLE_MASK);
@@ -820,10 +820,9 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	if (nested_npt_enabled(svm))
 		nested_svm_init_mmu_context(vcpu);
 
-	vcpu->arch.tsc_offset = kvm_calc_nested_tsc_offset(
-			vcpu->arch.l1_tsc_offset,
-			svm->nested.ctl.tsc_offset,
-			svm->tsc_ratio_msr);
+	vcpu->arch.tsc_offset = kvm_calc_nested_tsc_offset(vcpu->arch.l1_tsc_offset,
+							   vmcb12_ctrl->tsc_offset,
+							   svm->tsc_ratio_msr);
 
 	vmcb02->control.tsc_offset = vcpu->arch.tsc_offset;
 
@@ -832,13 +831,13 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 		nested_svm_update_tsc_ratio_msr(vcpu);
 
 	vmcb02->control.int_ctl             =
-		(svm->nested.ctl.int_ctl & int_ctl_vmcb12_bits) |
+		(vmcb12_ctrl->int_ctl & int_ctl_vmcb12_bits) |
 		(vmcb01->control.int_ctl & int_ctl_vmcb01_bits);
 
-	vmcb02->control.int_vector          = svm->nested.ctl.int_vector;
-	vmcb02->control.int_state           = svm->nested.ctl.int_state;
-	vmcb02->control.event_inj           = svm->nested.ctl.event_inj;
-	vmcb02->control.event_inj_err       = svm->nested.ctl.event_inj_err;
+	vmcb02->control.int_vector          = vmcb12_ctrl->int_vector;
+	vmcb02->control.int_state           = vmcb12_ctrl->int_state;
+	vmcb02->control.event_inj           = vmcb12_ctrl->event_inj;
+	vmcb02->control.event_inj_err       = vmcb12_ctrl->event_inj_err;
 
 	/*
 	 * next_rip is consumed on VMRUN as the return address pushed on the
@@ -849,7 +848,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	 * prior to injecting the event).
 	 */
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
-		vmcb02->control.next_rip    = svm->nested.ctl.next_rip;
+		vmcb02->control.next_rip    = vmcb12_ctrl->next_rip;
 	else if (boot_cpu_has(X86_FEATURE_NRIPS))
 		vmcb02->control.next_rip    = vmcb12_rip;
 
@@ -859,7 +858,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 		svm->soft_int_csbase = vmcb12_csbase;
 		svm->soft_int_old_rip = vmcb12_rip;
 		if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
-			svm->soft_int_next_rip = svm->nested.ctl.next_rip;
+			svm->soft_int_next_rip = vmcb12_ctrl->next_rip;
 		else
 			svm->soft_int_next_rip = vmcb12_rip;
 	}
@@ -870,11 +869,11 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 		vmcb02->control.virt_ext |= VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK;
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_PAUSEFILTER))
-		pause_count12 = svm->nested.ctl.pause_filter_count;
+		pause_count12 = vmcb12_ctrl->pause_filter_count;
 	else
 		pause_count12 = 0;
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_PFTHRESHOLD))
-		pause_thresh12 = svm->nested.ctl.pause_filter_thresh;
+		pause_thresh12 = vmcb12_ctrl->pause_filter_thresh;
 	else
 		pause_thresh12 = 0;
 	if (kvm_pause_in_guest(svm->vcpu.kvm)) {
@@ -888,7 +887,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 		vmcb02->control.pause_filter_thresh = vmcb01->control.pause_filter_thresh;
 
 		/* ... but ensure filtering is disabled if so requested.  */
-		if (vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_PAUSE)) {
+		if (vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_PAUSE)) {
 			if (!pause_count12)
 				vmcb02->control.pause_filter_count = 0;
 			if (!pause_thresh12)
@@ -905,7 +904,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	 * L2 is the "guest").
 	 */
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS))
-		vmcb02->control.erap_ctl = (svm->nested.ctl.erap_ctl &
+		vmcb02->control.erap_ctl = (vmcb12_ctrl->erap_ctl &
 					    ERAP_CONTROL_ALLOW_LARGER_RAP) |
 					   ERAP_CONTROL_CLEAR_RAP;
 
-- 
2.53.0.345.g96ddfc5eaa-goog

Re: [PATCH v2 1/8] KVM: SVM: Explicitly mark vmcb01 dirty after modifying VMCB intercepts

[Yosry Ahmed]

On Wed, Feb 18, 2026 at 03:09:51PM -0800, Sean Christopherson wrote:
> When reacting to an intercept update, explicitly mark vmcb01's intercepts
> dirty, as KVM always initially operates on vmcb01, and nested_svm_vmexit()
> isn't guaranteed to mark VMCB_INTERCEPTS as dirty.  I.e. if L2 is active,
> KVM will modify the intercepts for L1, but might not mark them as dirty
> before the next VMRUN of L1.
> 
> Fixes: 116a0a23676e ("KVM: SVM: Add clean-bit for intercetps, tsc-offset and pause filter count")
> Cc: stable@vger.kernel.org
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/nested.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index de90b104a0dd..66701106a51b 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -128,11 +128,13 @@ void recalc_intercepts(struct vcpu_svm *svm)
>  	struct vmcb_ctrl_area_cached *g;
>  	unsigned int i;
>  
> -	vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS);
> +	vmcb_mark_dirty(svm->vmcb01.ptr, VMCB_INTERCEPTS);
>  
>  	if (!is_guest_mode(&svm->vcpu))
>  		return;
>  
> +	vmcb_mark_dirty(svm->vmcb, VMCB_INTERCEPTS);
> +
>  	c = &svm->vmcb->control;
>  	h = &svm->vmcb01.ptr->control;
>  	g = &svm->nested.ctl;
> -- 
> 2.53.0.345.g96ddfc5eaa-goog
> 

Re: [PATCH v2 2/8] KVM: SVM: Separate recalc_intercepts() into nested vs. non-nested parts

[Yosry Ahmed]

On Wed, Feb 18, 2026 at 03:09:52PM -0800, Sean Christopherson wrote:
> Extract the non-nested aspects of recalc_intercepts() into a separate
> helper, svm_mark_intercepts_dirty(), to make it clear that the call isn't
> *just* recalculating (vmcb02's) intercepts, and to not bury non-nested
> code in nested.c.
> 
> As suggested by Yosry, opportunistically prepend "nested_vmbc02_" to
> recalc_intercepts() so that it's obvious the function specifically deals
> with recomputing intercepts for L2.
> 
> No functional change intended.
> 
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

Re: [PATCH v2 4/8] KVM: nSVM: Directly (re)calc vmcb02 intercepts from nested_vmcb02_prepare_control()

[Yosry Ahmed]

On Wed, Feb 18, 2026 at 03:09:54PM -0800, Sean Christopherson wrote:
> Now that nested_vmcb02_recalc_intercepts() provides guardrails against it
> being incorrectly called without vmcb02 active, invoke it directly from
> nested_vmcb02_recalc_intercepts() instead of bouncing through
> svm_mark_intercepts_dirty(), which unnecessarily marks vmcb01 as dirty.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/nested.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index 793f5d2eed3a..e8512de5aef7 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -916,7 +916,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
>  	 * Merge guest and host intercepts - must be called with vcpu in
>  	 * guest-mode to take effect.
>  	 */
> -	svm_mark_intercepts_dirty(svm);
> +	nested_vmcb02_recalc_intercepts(svm);
>  }
>  
>  static void nested_svm_copy_common_state(struct vmcb *from_vmcb, struct vmcb *to_vmcb)
> -- 
> 2.53.0.345.g96ddfc5eaa-goog
> 

Re: [PATCH v2 7/8] KVM: nSVM: Move vmcb_ctrl_area_cached.bus_lock_rip to svm_nested_state

[Yosry Ahmed]

On Wed, Feb 18, 2026 at 03:09:57PM -0800, Sean Christopherson wrote:
> Move "bus_lock_rip" from "vmcb_ctrl_area_cached" to "svm_nested_state" as
> "last_bus_lock_rip" to more accurately reflect what it tracks, and because
> it is NOT a cached vmcb12 control field.  The misplaced field isn't all
> that apparent in the current code base, as KVM uses "svm->nested.ctl"
> broadly, but the bad placement becomes glaringly obvious if
> "svm->nested.ctl" is captured as a local "vmcb12_ctrl" variable.
> 
> No functional change intended.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

Re: [PATCH v2 8/8] KVM: nSVM: Capture svm->nested.ctl as vmcb12_ctrl when preparing vmcb02

[Yosry Ahmed]

On Wed, Feb 18, 2026 at 03:09:58PM -0800, Sean Christopherson wrote:
> Grab svm->nested.ctl as vmcb12_ctrl when preparing the vmcb02 controls to
> make it more obvious that much of the data is coming from vmcb12 (or
> rather, a snapshot of vmcb12 at the time of L1's VMRUN).
> 
> Opportunistically reorder the variable definitions to create a pretty
> reverse fir tree.
> 
> No functional change intended.
> 
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

Re: [PATCH v2 0/8] KVM: SVM: A fix and cleanups for VMCB intercepts

[Sean Christopherson]

On Wed, 18 Feb 2026 15:09:50 -0800, Sean Christopherson wrote:
> Fix a likely-benign bug where KVM fails to mark vmcb01 intercepts as dirty
> after recalculating intercepts while L2 is active, then do a bunch of related
> cleanup, e.g. to split recalc_intercepts() into nested vs. non-nested
> functionality.
> 
> v2:
>  - Fix the aforementioned bug.
>  - Split recalc_intercepts() instead of simply renaming it.
>  - Move the new WARN in nested_vmcb02_recalc_intercepts() to its own patch.
>  - Use less weird local variables even if they aren't consistent with the
>    existing code...
>  - ... and then change some names in the existing code to provide consistency.
> 
> [...]

Applied to kvm-x86 nested, thanks!

[1/8] KVM: SVM: Explicitly mark vmcb01 dirty after modifying VMCB intercepts
      https://github.com/kvm-x86/linux/commit/d5bde6113aed
[2/8] KVM: SVM: Separate recalc_intercepts() into nested vs. non-nested parts
      https://github.com/kvm-x86/linux/commit/0b97f929831a
[3/8] KVM: nSVM: WARN and abort vmcb02 intercepts recalc if vmcb02 isn't active
      https://github.com/kvm-x86/linux/commit/a367b6e10372
[4/8] KVM: nSVM: Directly (re)calc vmcb02 intercepts from nested_vmcb02_prepare_control()
      https://github.com/kvm-x86/linux/commit/4a80c4bc1f10
[5/8] KVM: nSVM: Use intuitive local variables in nested_vmcb02_recalc_intercepts()
      https://github.com/kvm-x86/linux/commit/586160b75091
[6/8] KVM: nSVM: Use vmcb12_is_intercept() in nested_sync_control_from_vmcb02()
      https://github.com/kvm-x86/linux/commit/ef09eebc5736
[7/8] KVM: nSVM: Move vmcb_ctrl_area_cached.bus_lock_rip to svm_nested_state
      https://github.com/kvm-x86/linux/commit/af75470944f4
[8/8] KVM: nSVM: Capture svm->nested.ctl as vmcb12_ctrl when preparing vmcb02
      https://github.com/kvm-x86/linux/commit/56bfbe68f78e

--
https://github.com/kvm-x86/linux/tree/next