[PATCH] KVM: x86: Don't reuse the GPA of a fault on the guest's page tables

[PATCH] KVM: x86: Don't reuse the GPA of a fault on the guest's page tables

[Sasha Levin]

When a #NPF or EPT violation leads to emulation, x86_emulate_instruction()
saves the faulting GPA for direct MMUs, and emulator_read_write_onepage()
uses it instead of walking the guest's page tables for an access whose
page offset matches.  If the fault occurred while the CPU was walking the
guest's page tables (PFERR_GUEST_PAGE_MASK), the GPA is that of a guest
paging structure, not of the access being emulated, and the access goes
to the wrong address.

KVM selftest set_memory_region_test runs into this on AMD.
test_delete_memory_region() deletes the memslot holding the guest's page
tables while the guest spins on a read of 0xc0000000 that userspace
handles as MMIO.  If the vCPU is at the read when the memslot goes away,
the CPU takes a #NPF on the guest's root page table at 0x180000.
DecodeAssist provides the instruction bytes, the read of 0xc0000000 has
the same page offset as 0x180000, and the emulator turns it into an MMIO
read of 0x180000:

  ==== Test Assertion Failure ====
    set_memory_region_test.c:83: run->mmio.phys_addr == MEM_REGION_GPA
    ...
    Unexpected exit mmio address = 0x180000

If the vCPU is at the next instruction instead, the #NPF happens on the
code fetch, DecodeAssist provides no bytes, KVM fails to fetch the
instruction itself and injects a #PF, and the guest shuts down as the
test expects.  KernelCI sees the failure in about half of the runs on a
DL345 Gen11 (EPYC).

The CPU walks the guest's page tables with write accesses, so such faults
also set EMULTYPE_PF_WRITE, and a write would be redirected the same way.

Tell the emulator when the fault occurred on the guest's page tables and
don't use the GPA for the access then, so that the access is translated
through the guest's page tables.  KVM can't walk the encrypted page tables
of SEV guests, so resume an SEV guest instead of emulating, as is done
when the #NPF occurred on the instruction fetch.

Fixes: 0f89b207b04a ("kvm: svm: Use the hardware provided GPA instead of page walk")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
Reproduced on an HPE DL345 Gen11 (EPYC 9655P) with KernelCI's x86 build of
mainline d135ab453a74: set_memory_region_test failed in 15 of 30 runs, and
in 6 of 12 runs with tracing, always with the MMIO exit at 0x180000. The
traces show the #NPF at 0x180000 with error code 0x200000006 on the cmpq
that reads 0xc0000000, followed by an MMIO read of 0x180000.

With this patch on the same config, the test passed 50 of 50 runs. When
the #NPF hits the cmpq, KVM now injects a #PF for the read of 0xc0000000
and the guest shuts down. The rest of the KVM selftests give the same
results as without the patch, with and without
kvm.force_emulation_prefix=1. Not tested on SEV or Intel.

 arch/x86/kvm/mmu/mmu.c |  2 ++
 arch/x86/kvm/svm/svm.c |  9 +++++++++
 arch/x86/kvm/x86.c     | 10 ++++++++--
 arch/x86/kvm/x86.h     |  6 ++++++
 4 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index dc2aef72ffddc..66e5365f63f20 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6719,6 +6719,8 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 emulate:
 	if (error_code & PFERR_WRITE_MASK)
 		emulation_type |= EMULTYPE_PF_WRITE;
+	if (error_code & PFERR_GUEST_PAGE_MASK)
+		emulation_type |= EMULTYPE_PF_GUEST_PAGE;
 
 	return x86_emulate_instruction(vcpu, cr2_or_gpa, emulation_type, insn,
 				       insn_len);
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 136069dfcd44f..2240d122ceabc 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5324,6 +5324,15 @@ static int svm_check_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
 		return X86EMUL_PROPAGATE_FAULT;
 	}
 
+	/*
+	 * Don't emulate if the #NPF occurred while walking the guest's page
+	 * tables.  The GPA is that of a guest paging structure, not of the
+	 * access, and KVM can't walk the guest's encrypted page tables to
+	 * translate the access itself.  Resume the guest, see below.
+	 */
+	if (emul_type & EMULTYPE_PF_GUEST_PAGE)
+		goto resume_guest;
+
 	/*
 	 * Emulate for SEV guests if the insn buffer is not empty.  The buffer
 	 * will be empty if the DecodeAssist microcode cannot fetch bytes for
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32563a91fba99..c947c3096838a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6495,8 +6495,14 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		/* Save the faulting GPA (cr2) in the address field */
 		ctxt->exception.address = cr2_or_gpa;
 
-		/* With shadow page tables, cr2 contains a GVA or nGPA. */
-		if (vcpu->arch.mmu->root_role.direct) {
+		/*
+		 * With shadow page tables, cr2 contains a GVA or nGPA.  If the
+		 * fault occurred while walking the guest's page tables, the
+		 * GPA is that of a guest paging structure, not of the access
+		 * that needs to be emulated, and can't be used for the access.
+		 */
+		if (vcpu->arch.mmu->root_role.direct &&
+		    !(emulation_type & EMULTYPE_PF_GUEST_PAGE)) {
 			ctxt->gpa_access = ACC_READ_MASK;
 			/*
 			 * Always allow writes for guests with protected page
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 436e2fb1396c4..4d54e31133668 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -410,6 +410,11 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
  *
  * EMULTYPE_PF_WRITE - Set with EMULTYPE_PF when hardware reports a write access.
  *
+ * EMULTYPE_PF_GUEST_PAGE - Set with EMULTYPE_PF when hardware reports that the
+ *			    fault occurred while walking the guest's page
+ *			    tables, i.e. the GPA is that of a guest paging
+ *			    structure, not of the access being emulated.
+ *
  * EMULTYPE_COMPLETE_USER_EXIT - Set when the emulator should update interruptibility
  *				 state and inject single-step #DBs after skipping
  *				 an instruction (after completing userspace I/O).
@@ -449,6 +454,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 #define EMULTYPE_WRITE_PF_TO_SP	    (1 << 8)
 #define EMULTYPE_SKIP_SOFT_INT	    (1 << 9)
 #define EMULTYPE_PF_WRITE	    (1 << 10)
+#define EMULTYPE_PF_GUEST_PAGE	    (1 << 11)
 
 #define EMULTYPE_SET_SOFT_INT_VECTOR(v)	((u32)((v) & 0xff) << 16)
 #define EMULTYPE_GET_SOFT_INT_VECTOR(e)	(((e) >> 16) & 0xff)
-- 
2.53.0