[PATCH v3 0/5] KVM: nVMX: Honor L1-owned CR0 bits for LMSW

[PATCH v3 0/5] KVM: nVMX: Honor L1-owned CR0 bits for LMSW

[Sean Christopherson]

Fix a bug where KVM ignores vmcs12's CR0 guest/host mask when emulating an
LMSW from L2, i.e. an LMSW tha exits to L0 but not to L1.

v3:
 - Handle the read_shadow in handle_set_cr0() (and for CR4, because why not).
 - Kill off kvm_lmsw() entirely.
 - Add a bunch of comments to the selftest to explain the dangers of running
   with CR0.TS=1.
 - Test MOV CR0 as well as LMSW.
 - Add a patch to take full ownership of the must-be-1 CR0 and CR4 bits.

v2:
 - https://lore.kernel.org/all/20261030032937.2813-1-ma.mandourr@gmail.com
 - Patch 2: order the CR0 store before FNOP and check L2's whole CR0.

v1: https://lore.kernel.org/all/20261029235040.12013-1-ma.mandourr@gmail.com

Mahmoud Abumandour (3):
  KVM: nVMX: Honor L1-owned CR0 bits when emulating intercepted LMSW for
    L2
  KVM: nVMX: Synthesize #GP if L2 writes a "bad" CR0 value via LMSW
  KVM: selftests: Verify KVM handling of L1-owned CR0 bits on writes
    from L2

Sean Christopherson (2):
  KVM: nVMX: Preserve L1-owned bits in CR{0,4} read shadows
  KVM: nVMX: Take full ownership of the post-VMXON CR{0,4} must-be-1
    bits

 Documentation/virt/kvm/api.rst                |  9 ++-
 arch/x86/kvm/regs.c                           |  6 --
 arch/x86/kvm/regs.h                           |  1 -
 arch/x86/kvm/vmx/nested.c                     | 30 +-------
 arch/x86/kvm/vmx/vmx.c                        | 40 +++++-----
 .../selftests/kvm/include/x86/processor.h     |  5 ++
 .../selftests/kvm/x86/feature_msrs_test.c     |  2 +-
 .../kvm/x86/nested_exceptions_test.c          | 73 +++++++++++++++++++
 8 files changed, 106 insertions(+), 60 deletions(-)


base-commit: 7e60162b16f9da0d2fa83da5c1723b6853a0ac1d
-- 
2.56.0.385.gac671c819a-goog

[PATCH v3 1/5] KVM: nVMX: Preserve L1-owned bits in CR{0,4} read shadows

[Sean Christopherson]

When emulating an intercepted MOV CR{0,4} for L2, i.e. when L2 is toggling
CR{0,4} bits that L0 intercepts but L1 does not, explicitly preserve
L1-owned bits in the vmcs02 read shadow instead of subtly relying on the
interception logic to preserve the bits.  For straight MOV instructions,
relying on the interception logic works, because KVM would have synthesized
a nested VM-Exit to L1 if L2 were toggling L1-owned bits, i.e. if L2 were
explicitly changing any of the shadowed bits.

However, for LMSW, which preserves bits 31:4, and conditionally preserves
CR0.PE (bit 0), the effective value being set by L2 can diverge from the
read shadow *without* triggering a nested VM-Exit.  E.g. if CR0.PE=1 in
"hardware", CR0_READ_SHADOW.PE=0, and CR0_GUEST_HOST_MASK.PE=1, then a LMSW
with CR0.PE=0 won't VM-Exit, but will preserve/set CR0.PE=1 in "hardware".

Explicitly preserving the shadowed bits will allow using handle_set_cr0()
to honor L1's guest/host mask when emulating LMSW.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/vmx/vmx.c | 30 +++++++++++++-----------------
 1 file changed, 13 insertions(+), 17 deletions(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index b341af4c6057..544b7840897c 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5629,22 +5629,19 @@ static int handle_set_cr0(struct kvm_vcpu *vcpu, unsigned long val)
 {
 	if (is_guest_mode(vcpu)) {
 		struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
-		unsigned long orig_val = val;
+		u64 mask = vmcs12->cr0_guest_host_mask;
 
 		/*
-		 * We get here when L2 changed cr0 in a way that did not change
-		 * any of L1's shadowed bits (see nested_vmx_exit_handled_cr),
-		 * but did change L0 shadowed bits. So we first calculate the
-		 * effective cr0 value that L1 would like to write into the
-		 * hardware. It consists of the L2-owned bits from the new
-		 * value combined with the L1-owned bits from L1's guest_cr0.
+		 * L2 is writing a CR0 value that isn't modifying any of L1's
+		 * shadowed bits (see nested_vmx_exit_handled_cr()), but *is*
+		 * modifying L0 shadowed bits.  Set the "hardware" CR0 and the
+		 * read shadow using the L2-owned bits from the new value,
+		 * combined with the L1-owned bits.
 		 */
-		val = (val & ~vmcs12->cr0_guest_host_mask) |
-			(vmcs12->guest_cr0 & vmcs12->cr0_guest_host_mask);
-
-		if (kvm_set_cr0(vcpu, val))
+		val &= ~mask;
+		if (kvm_set_cr0(vcpu, val | (vmcs12->guest_cr0 & mask)))
 			return 1;
-		vmcs_writel(CR0_READ_SHADOW, orig_val);
+		vmcs_writel(CR0_READ_SHADOW, val | (vmcs12->cr0_read_shadow & mask));
 		return 0;
 	} else {
 		return kvm_set_cr0(vcpu, val);
@@ -5655,14 +5652,13 @@ static int handle_set_cr4(struct kvm_vcpu *vcpu, unsigned long val)
 {
 	if (is_guest_mode(vcpu)) {
 		struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
-		unsigned long orig_val = val;
+		u64 mask = vmcs12->cr4_guest_host_mask;
 
 		/* analogously to handle_set_cr0 */
-		val = (val & ~vmcs12->cr4_guest_host_mask) |
-			(vmcs12->guest_cr4 & vmcs12->cr4_guest_host_mask);
-		if (kvm_set_cr4(vcpu, val))
+		val &= ~mask;
+		if (kvm_set_cr4(vcpu, val | (vmcs12->guest_cr4 & mask)))
 			return 1;
-		vmcs_writel(CR4_READ_SHADOW, orig_val);
+		vmcs_writel(CR4_READ_SHADOW, val | (vmcs12->cr4_read_shadow & mask));
 		return 0;
 	} else
 		return kvm_set_cr4(vcpu, val);
-- 
2.56.0.385.gac671c819a-goog

[PATCH v3 2/5] KVM: nVMX: Honor L1-owned CR0 bits when emulating intercepted LMSW for L2

[Sean Christopherson]

From: Mahmoud Abumandour <ma.mandourr@gmail.com>

When emulating an intercepted LMSW on behalf L2, i.e. when LMSW exits to L0
but not to L1, emulate it via handle_set_cr0(), as is done for MOV to CR0,
so that KVM honor's the guest/host mask and read shadow from vmcs12.  This
fixes a bug where so L2 could change L1-owned CR0 bits, e.g. clear TS, and
read their effective values instead of the read shadow.

Drop kvm_lmsw() as it is no longer used, and opportunistically trace the
effectively CR0 value (previously, KVM traced the raw CR0.PE value), e.g.
so that users don't have to piece together that CR0.PE will be preserved if
it was already '1'.

Fixes: eeadf9e7558c ("KVM: nVMX: Handling of CR0 and CR4 modifying instructions")
Signed-off-by: Mahmoud Abumandour <ma.mandourr@gmail.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/regs.c    | 6 ------
 arch/x86/kvm/regs.h    | 1 -
 arch/x86/kvm/vmx/vmx.c | 6 ++++--
 3 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 3621f1a47d37..44a88c70b77b 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -303,12 +303,6 @@ int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr0);
 
-void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw)
-{
-	(void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f));
-}
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw);
-
 int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
 {
 	bool skip_tlb_flush = false;
diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h
index 97c17b120a57..c30ac8394271 100644
--- a/arch/x86/kvm/regs.h
+++ b/arch/x86/kvm/regs.h
@@ -62,7 +62,6 @@ int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val);
 unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr);
 unsigned long kvm_get_dr6_fixed_1(struct kvm_vcpu *vcpu);
 unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu);
-void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw);
 int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
 
 static inline bool is_long_mode(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 544b7840897c..2d5feeb8a059 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5743,8 +5743,10 @@ static int handle_cr(struct kvm_vcpu *vcpu)
 		break;
 	case 3: /* lmsw */
 		val = (exit_qualification >> LMSW_SOURCE_DATA_SHIFT) & 0x0f;
-		trace_kvm_cr_write(0, (kvm_read_cr0_bits(vcpu, ~0xful) | val));
-		kvm_lmsw(vcpu, val);
+		val |= kvm_read_cr0_bits(vcpu, ~0x0eul);
+
+		trace_kvm_cr_write(cr, val);
+		(void)handle_set_cr0(vcpu, val);
 
 		return kvm_skip_emulated_instruction(vcpu);
 	default:
-- 
2.56.0.385.gac671c819a-goog

[PATCH v3 3/5] KVM: nVMX: Synthesize #GP if L2 writes a "bad" CR0 value via LMSW

[Sean Christopherson]

From: Mahmoud Abumandour <ma.mandourr@gmail.com>

Synthesize a #GP if writing CR0 when emulating LMSW fails, as it's possible
for LMSW to #GP on a bad value when executed post-VMXON, i.e. if LMSW
attempts to set/clear bits in violation of MSR_IA32_VMX_CR0_FIXED{0,1}.

Outside of VMX operation, LMSW can only arbitrarily modify bits 3:1 and can
only set bit 0 (PE).  I.e. LMSW can't modify any CR0 bits that are
unsupported or are subject to additional checks and thus should never #GP
due to a bad value.

Note, KVM has a virtualization hole where it doesn't enforce must-be-1 CR0
or CR4 bits that KVM doesn't intercepted.  E.g. if userspace marked CR0.TS
as a must-be-1 bit (while post-VMXON), then the guest could clear CR0.TS.
That hole will be plugged shortly.

Note #2, with the virtualization hole plugged, it *should* be impossible
for LMSW to #GP on a bad value.  Unfortunately, KVM's save/restore ioctls
allow userspace to load a CR0 value that violates the must-be-1 masks, i.e.
userspace can make the impossible happen (and plugging *those* holes is all
but guaranteed to be a waste of time).  Ideally, KVM would WARN, but since
userspace could trip the WARN, take the easy route and handle failure
appropriately.

Fixes: eeadf9e7558c ("KVM: nVMX: Handling of CR0 and CR4 modifying instructions")
Signed-off-by: Mahmoud Abumandour <ma.mandourr@gmail.com>
[sean: split to a separate patch]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/vmx/vmx.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 2d5feeb8a059..68c1361a92d1 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5746,9 +5746,8 @@ static int handle_cr(struct kvm_vcpu *vcpu)
 		val |= kvm_read_cr0_bits(vcpu, ~0x0eul);
 
 		trace_kvm_cr_write(cr, val);
-		(void)handle_set_cr0(vcpu, val);
-
-		return kvm_skip_emulated_instruction(vcpu);
+		err = handle_set_cr0(vcpu, val);
+		return kvm_complete_insn_gp(vcpu, err);
 	default:
 		break;
 	}
-- 
2.56.0.385.gac671c819a-goog

[PATCH v3 4/5] KVM: nVMX: Take full ownership of the post-VMXON CR{0,4} must-be-1 bits

[Sean Christopherson]

Effectively drop support for restoring/stuffing fixed-0 CR{0,4} MSRs, as
restoring to anything other than KVM's hardcoded value is broken, and is
absolutely not worth fixing.  KVM doesn't adjust the "guest owned" masks
based on the MSR values, and so if userspace were to require any of the
passed-through bits to be '1', KVM would fail to prevent the guest from
clearing a must-be-1 bit.

Even worse, if userspace disables KVM_X86_QUIRK_STUFF_FEATURE_MSRS without
ever setting the CR{0,4} FIXED0 MSRs, then KVM would allow running with an
impossible configuration, as KVM would advertise to L1 that there are *no*
must-be-1 bits, but couldn't honor that because VM-Entry would fail due to
the bits still being must-be-1 in real hardware.  E.g. KVM would tell the
guest that CR4.VMXE can be toggled at will, in complete violation of VMXON.

Don't try to fix KVM, as no known VMM sets values other than KVM's own
hardcoded value.  KVM already requires the incoming value be a superset of
KVM's value, i.e. KVM doesn't allowed reducing the set of must-be-1 bits,
and so it's only possible for userspace to require *extra* bits to always
be 1.  Drop KVM's "support" before anyone tries to utilize it, e.g. to
harden a guest or something, as honoring things like CR0.TS in particular
would be painful, e.g. KVM would need to handle CLTS interception.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 Documentation/virt/kvm/api.rst                |  9 +++---
 arch/x86/kvm/vmx/nested.c                     | 30 ++-----------------
 arch/x86/kvm/vmx/vmx.c                        |  3 ++
 .../selftests/kvm/x86/feature_msrs_test.c     |  2 +-
 4 files changed, 11 insertions(+), 33 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index e95c9cda085d..c9f7284f76b4 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -8764,10 +8764,11 @@ KVM_X86_QUIRK_STUFF_FEATURE_MSRS           By default, at vCPU creation, KVM set
                                            i.e. treats the feature MSRs like CPUID
                                            leaves and gives userspace full control of
                                            the vCPU model definition.  This quirk does
-                                           not affect VMX MSRs CR0/CR4_FIXED1 (0x487
-                                           and 0x489), as KVM does now allow them to
-                                           be set by userspace (KVM sets them based on
-                                           guest CPUID, for safety purposes).
+                                           not affect VMX MSRs CR0/CR4_FIXED{0,1}
+                                           (0x486 - 0x489), as KVM does now allow them
+                                           to be set by userspace (KVM sets them based
+                                           on the virtual CPU model, for simplicity and
+                                           safety purposes).
 
 KVM_X86_QUIRK_IGNORE_GUEST_PAT             By default, on Intel platforms, KVM ignores
                                            guest PAT and forces the effective memory
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 9b0bfa2f854c..a5547ec78846 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1508,33 +1508,6 @@ static int vmx_restore_vmx_ept_vpid_cap(struct vcpu_vmx *vmx, u64 data)
 	return 0;
 }
 
-static u64 *vmx_get_fixed0_msr(struct nested_vmx_msrs *msrs, u32 msr_index)
-{
-	switch (msr_index) {
-	case MSR_IA32_VMX_CR0_FIXED0:
-		return &msrs->cr0_fixed0;
-	case MSR_IA32_VMX_CR4_FIXED0:
-		return &msrs->cr4_fixed0;
-	default:
-		BUG();
-	}
-}
-
-static int vmx_restore_fixed0_msr(struct vcpu_vmx *vmx, u32 msr_index, u64 data)
-{
-	const u64 *msr = vmx_get_fixed0_msr(&vmcs_config.nested, msr_index);
-
-	/*
-	 * 1 bits (which indicates bits which "must-be-1" during VMX operation)
-	 * must be 1 in the restored value.
-	 */
-	if (!is_bitwise_subset(data, *msr, -1ULL))
-		return -EINVAL;
-
-	*vmx_get_fixed0_msr(&vmx->nested.msrs, msr_index) = data;
-	return 0;
-}
-
 /*
  * Called when userspace is restoring VMX MSRs.
  *
@@ -1577,8 +1550,9 @@ int vmx_set_vmx_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data)
 	case MSR_IA32_VMX_MISC:
 		return vmx_restore_vmx_misc(vmx, data);
 	case MSR_IA32_VMX_CR0_FIXED0:
+		return data && data != vmcs_config.nested.cr0_fixed0;
 	case MSR_IA32_VMX_CR4_FIXED0:
-		return vmx_restore_fixed0_msr(vmx, msr_index, data);
+		return data && data != vmcs_config.nested.cr4_fixed0;
 	case MSR_IA32_VMX_CR0_FIXED1:
 	case MSR_IA32_VMX_CR4_FIXED1:
 		/*
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 68c1361a92d1..4afbb3986a4c 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -7712,7 +7712,10 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu)
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct kvm_cpuid_entry2 *entry;
 
+	vmx->nested.msrs.cr0_fixed0 = vmcs_config.nested.cr0_fixed0;
 	vmx->nested.msrs.cr0_fixed1 = 0xffffffff;
+
+	vmx->nested.msrs.cr4_fixed0 = vmcs_config.nested.cr4_fixed0;
 	vmx->nested.msrs.cr4_fixed1 = X86_CR4_PCE;
 
 #define cr4_fixed1_update(_cr4_mask, _reg, _cpuid_mask) do {		\
diff --git a/tools/testing/selftests/kvm/x86/feature_msrs_test.c b/tools/testing/selftests/kvm/x86/feature_msrs_test.c
index d56d0ddb3a57..837fee78835c 100644
--- a/tools/testing/selftests/kvm/x86/feature_msrs_test.c
+++ b/tools/testing/selftests/kvm/x86/feature_msrs_test.c
@@ -14,7 +14,7 @@
 
 static bool is_kvm_controlled_msr(u32 msr)
 {
-	return msr == MSR_IA32_VMX_CR0_FIXED1 || msr == MSR_IA32_VMX_CR4_FIXED1;
+	return msr >= MSR_IA32_VMX_CR0_FIXED0 && msr <= MSR_IA32_VMX_CR4_FIXED1;
 }
 
 /*
-- 
2.56.0.385.gac671c819a-goog

[PATCH v3 5/5] KVM: selftests: Verify KVM handling of L1-owned CR0 bits on writes from L2

[Sean Christopherson]

From: Mahmoud Abumandour <ma.mandourr@gmail.com>

Expand the nested exceptions test to verify KVM's handling of VMX's CR0
guest/host masks, i.e. of VMX's conditional interception of CR0 writes.
LMSW in particular is nasty, as it preserves bits 31:4 as well as CR0.PE
if it was already set.

Verify that MOV CR0 and LMSW in L2 that exits to L0, i.e. isn't intercepted
by L1, doesn't modify an L1-owned bits and reads the correct value from the
read shadow.  Use CR0.TS to do the bulk of validation, as its value in
hardware can be easily inferred by triggering an #NM in L2.  Specifically,
run with CR0.TS=1 in hardware but CR0.TS=0 in the read shadow, and verify
that non-intercepted writes to CR0 from L2 preserve the CR0.TS values in
hardware *and* the shadow.

Note, the downside of using CR0.TS is that any SIMD instructions in L2 will
hit a #NM, which is especially problematic since ucall(), e.g. for guest
prints and asserts, includes a memset(), for which the compiler likes to
generate SSE instructions.  To avoid false passes and false failures, have
L2 snapshot the CR0 it sees, but do the actual validation in L1.  E.g. if
L2 observed the wrong value, then doing GUEST_ASSERT_EQ() could get a false
pass as memset() would trigger the #NM expected by L1 (when L2 is trying to
report the failure).

Include CR0.NE and CR0.PE in the divergent shadow/hardware values as well
for extra coverage, e.g. CR0.NE is a must-be-1 bit, and CR0.PE is preserved
by LMSW, but if and only if it's already '1'.

Signed-off-by: Mahmoud Abumandour <ma.mandourr@gmail.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 .../selftests/kvm/include/x86/processor.h     |  5 ++
 .../kvm/x86/nested_exceptions_test.c          | 73 +++++++++++++++++++
 2 files changed, 78 insertions(+)

diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index 5898a277a2e0..79c4b502b00c 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -617,6 +617,11 @@ static inline void set_cr0(u64 val)
 	__asm__ __volatile__("mov %0, %%cr0" : : "r" (val) : "memory");
 }
 
+static inline void lmsw(u8 val)
+{
+	__asm__ __volatile__("lmsw %w0" : : "r" (val) : "memory");
+}
+
 static inline u64 get_cr2(void)
 {
 	u64 cr2;
diff --git a/tools/testing/selftests/kvm/x86/nested_exceptions_test.c b/tools/testing/selftests/kvm/x86/nested_exceptions_test.c
index f47432526ea1..e83c1f41703d 100644
--- a/tools/testing/selftests/kvm/x86/nested_exceptions_test.c
+++ b/tools/testing/selftests/kvm/x86/nested_exceptions_test.c
@@ -109,6 +109,51 @@ static void l1_svm_code(struct svm_test_data *svm)
 	GUEST_DONE();
 }
 
+static u64 l2_cr0;
+
+static void l2_vmx_cr0_test(bool use_lmsw)
+{
+	/*
+	 * DANGER!!!  DO NOT add anything that might cause the compiler to emit
+	 * a "string" operation, e.g. memset().  Compilers often generate SIMD
+	 * instructions for memset() and friends, and so will hit an #NM due to
+	 * CR0.TS being set in hardware.  Note, all ucalls trigger memset()!
+	 */
+	u64 cr0 = (get_cr0() ^ X86_CR0_MP);
+
+	if (use_lmsw)
+		lmsw(cr0);
+	else
+		set_cr0(cr0);
+
+	/*
+	 * Snapshot CR0 and let L1 check the value; as above, attempting to use
+	 * GUEST_ASSERT_EQ() => ucall will likely trigger a premature #NM and
+	 * result in a false pass (L1 will see the #NM and think all is well).
+	 */
+	l2_cr0 = get_cr0();
+
+	/* FNOP should #NM as CR0.TS is set vmcs12.CR0 and thus vmcs02. */
+	asm volatile("fnop" ::: "memory");
+
+	/*
+	 * Signal failure with a #UD, again to avoid ucall/memset().  If FNOP
+	 * somehow succeeds but then #NM is triggered by a SIMD instruction in
+	 * memset(), the test will get a false pass.
+	 */
+	asm volatile("ud2");
+}
+
+static void l2_mov_cr0_test(void)
+{
+	l2_vmx_cr0_test(false);
+}
+
+static void l2_lmsw_test(void)
+{
+	l2_vmx_cr0_test(true);
+}
+
 static void vmx_run_l2(void *l2_code, int vector, u32 error_code)
 {
 	vmwrite(GUEST_RIP, (u64)l2_code);
@@ -129,6 +174,8 @@ static void vmx_run_l2(void *l2_code, int vector, u32 error_code)
 
 static void l1_vmx_code(struct vmx_pages *vmx)
 {
+	unsigned long cr0;
+
 	prepare_for_vmx_operation(vmx);
 
 	load_vmcs(vmx);
@@ -153,6 +200,32 @@ static void l1_vmx_code(struct vmx_pages *vmx)
 	vmx_run_l2(l2_ss_injected_tf_test, FAKE_TRIPLE_FAULT_VECTOR, 0);
 	GUEST_ASSERT_EQ(vmread(VM_EXIT_REASON), EXIT_REASON_TRIPLE_FAULT);
 
+	/*
+	 * Validate KVM's CR0 mask+shadow emulation, for both MOV CR and LMSW.
+	 * Set TS and NE in the value loaded in hardware, to verify that doing
+	 * e.g. FNOP generates an #NM in L2, but clear the bits in L2's read
+	 * shadow and verify that L2 always sees '0' and can't set those bits.
+	 */
+	cr0 = vmread(GUEST_CR0) | X86_CR0_TS | X86_CR0_NE;
+	vmwrite(CR0_GUEST_HOST_MASK, X86_CR0_TS | X86_CR0_NE | X86_CR0_PE);
+	vmwrite(CR0_READ_SHADOW, cr0 & ~(X86_CR0_TS | X86_CR0_NE));
+	vmwrite(GUEST_CR0, cr0);
+	vmwrite(EXCEPTION_BITMAP, BIT_ULL(NM_VECTOR));
+
+	vmx_run_l2(l2_mov_cr0_test, NM_VECTOR, 0);
+	GUEST_ASSERT_EQ(l2_cr0, (cr0 & ~(X86_CR0_TS | X86_CR0_NE)) ^ X86_CR0_MP);
+	GUEST_ASSERT_EQ(vmread(GUEST_CR0), cr0 ^ X86_CR0_MP);
+
+	/*
+	 * LMSW preserves CR0.PE *if* CR0.PE=1.  Clear the bit in the shadow to
+	 * validate that KVM preserves CR0.PE in "hardware" even if L2 attempts
+	 * to clear CR0.PE via LMSW.
+	 */
+	vmwrite(CR0_READ_SHADOW, cr0 & ~(X86_CR0_TS | X86_CR0_NE | X86_CR0_PE));
+	vmx_run_l2(l2_lmsw_test, NM_VECTOR, 0);
+	GUEST_ASSERT_EQ(l2_cr0, cr0 & ~(X86_CR0_TS | X86_CR0_NE | X86_CR0_PE));
+	GUEST_ASSERT_EQ(vmread(GUEST_CR0), cr0);
+
 	GUEST_DONE();
 }
 
-- 
2.56.0.385.gac671c819a-goog