[PATCH v4 00/45] KVM: x86: Introduce memory protection attributes

[PATCH v4 00/45] KVM: x86: Introduce memory protection attributes

[Paolo Bonzini]

This series introduces a mechanism to let userspace block read,
write or execute access to individual GFNs via KVM's memory
attribute mechanism, and have them reported via KVM_EXIT_MEMORY_FAULT.

KVM needs to check the attributes anytime KVM takes GPAs as input for any
action initiated by the guest; if the memory attributes are incompatible
with such action, it should be stopped.  This means that there are quite
a lot of cases to handle, and while in most cases they occur before
committing to the execution of an instructions (for example, page walks),
there are other cases where a full implementation of attributes would
basically require an implementation of transactional memory accesses:
notably (from inevitable to where I finally drew the line) nested
virtualization, emulation, SMM switch, task switch.  In the worst case,
the processor might have executed the instruction before issuing a
trap-like vmexit - and if the instruction length is not provided and
the instruction is not accessible, there's no practical alternative
to returning -EIO and marking the VM as dead.

For this reason, documentation now clarifies that invalid accesses may not be
recoverable and may be reported in the same way as MMIO.

Along the way, the patches fix some issues in the memory attributes code,
that surfaced due to having more than one attribute.  Still, no matter
how careful I tried to be, I'm ready for the absolute massacre that
Sashiko will make of this stuff.

Paolo

v3->v4:
- New patch "KVM: nVMX/nSVM: check unconditionally whether CR3 is valid"
- New patch "KVM: x86/emulate: Check TSS busy-bit writes during task switches"
- Rebase on top of new VM memory attributes code
- Use kvm_check_gen in Longarch
- New patches to surface memory faults from the emulator and from nested
  GPA translation (15, 29-36, 38-39)
- Clarify in the docs that memory faults in some cases may not be recoverable

Anish Moorthy (1):
  KVM: Define and communicate KVM_EXIT_MEMORY_FAULT RWX flags to
    userspace

Nicolas Saenz Julienne (11):
  KVM: selftests: Take into account mixed memory fault flags
  KVM: x86/mmu: Init memslot hugepage information for non-private_mem
    VMs too
  KVM: Introduce NR/NW/NX memory attributes
  KVM: Include memory protections in result of gfn->hva conversion
  KVM: Take memory protections into account for memory read/write/fetch
  KVM: Encapsulate memattrs array into anonymous struct
  KVM: Introduce a generation number for memory attributes
  KVM: Take memory protections into account for accesses with cached
    gfn->hva
  KVM: pfncache: Fail to refresh if it contains memory protections
  KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory
    attributes
  KVM: x86/mmu: Take memory protection attributes into account during
    faults

Paolo Bonzini (33):
  KVM: nVMX/nSVM: check unconditionally whether CR3 is valid
  KVM: x86/hyperv: do not overwrite hc->ingpa for SIGNAL_EVENT hypercall
  KVM: x86: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING
  KVM: x86/emulate: Check TSS busy-bit writes during task switches
  KVM: loongarch: do full validity check on the gfn-to-hva cache
  KVM: selftests: Test address translation for Hyper-V direct L2
    hypercalls
  selftests: kvm: drop TEST_ASSERT_SUPPORTED_ATTRIBUTES
  KVM: x86: Introduce memory fault on invalid hypercalls reads/writes
  KVM: selftests: test hypercall memory fault exits
  KVM: x86/mmu: intersect writability from __kvm_faultin_pfn with
    fault->map_writable
  KVM: x86/mmu: Extend map_writable to a full ACC_* mask
  KVM: VMX: return error from kvm_set_cr*
  KVM: adjust for presence of more than one attribute
  KVM: Introduce kvm_fetch_guest_page() and use it for x86
  KVM: Take memory protections into account for __kvm_vcpu_map
  KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()
  KVM: x86/emulate: change return value of emulator_io_permitted to int
  KVM: x86/emulate: Propagate memory faults to userspace
  KVM: x86: "Handle" failed decode when skipping instructions
  KVM: vmx/sgx: make return value of sgx_gva_to_gpa consistent with its
    callers
  KVM: x86: Allow page fault fixup to return errors
  KVM: x86/mmu: Adjust gva_to_gpa to allow returning an error
  KVM: x86/mmu: Propagate errors from nested GPA translation
  KVM: x86: Propagate memory faults through load_pdptrs()
  KVM: x86/mmu: Issue memory fault exit if walk failed due to memory
    attribute
  KVM: x86: mark VM as dead after KVM_SET_SREGS fails to load PDPTRs
  KVM: VMX: mark VM dead on shadow VMCS or eVMCS access failures
  KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock
  KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs
  KVM: x86: selftests: Introduce memory protection attributes test
  KVM: x86: selftests: Introduce memory attributes PTE test
  KVM: x86: selftests: Introduce memory attributes side-channel tests
  KVM: x86: split out parts of load_pdptrs

 Documentation/virt/kvm/api.rst                |  57 ++-
 arch/loongarch/kvm/vcpu.c                     |  12 +-
 arch/x86/include/asm/kvm_host.h               |  18 +-
 arch/x86/kvm/Kconfig                          |   2 +-
 arch/x86/kvm/emulate.c                        |  82 +++-
 arch/x86/kvm/hyperv.c                         | 174 +++++--
 arch/x86/kvm/kvm_emulate.h                    |   3 +
 arch/x86/kvm/mmu.h                            |  28 +-
 arch/x86/kvm/mmu/mmu.c                        | 200 +++++++--
 arch/x86/kvm/mmu/mmu_internal.h               |  21 +-
 arch/x86/kvm/mmu/mmutrace.h                   |  36 ++
 arch/x86/kvm/mmu/paging_tmpl.h                |  63 ++-
 arch/x86/kvm/mmu/spte.c                       |  12 +-
 arch/x86/kvm/mmu/spte.h                       |  13 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    |   2 +-
 arch/x86/kvm/regs.c                           | 103 +++--
 arch/x86/kvm/regs.h                           |   9 +
 arch/x86/kvm/svm/nested.c                     |  51 +--
 arch/x86/kvm/svm/svm.c                        |  19 +-
 arch/x86/kvm/vmx/hyperv.c                     |   7 +-
 arch/x86/kvm/vmx/nested.c                     |  53 ++-
 arch/x86/kvm/vmx/sgx.c                        |  35 +-
 arch/x86/kvm/vmx/vmx.c                        |  59 ++-
 arch/x86/kvm/x86.c                            | 204 ++++++---
 arch/x86/kvm/x86.h                            |   2 +-
 include/linux/kvm_host.h                      | 133 +++++-
 include/linux/kvm_types.h                     |   6 +-
 include/trace/events/kvm.h                    |  14 +-
 include/uapi/linux/kvm.h                      |   7 +
 tools/include/uapi/linux/kvm.h                |   3 +
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../testing/selftests/kvm/include/kvm_util.h  |  37 +-
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../testing/selftests/kvm/lib/x86/processor.c |   5 +
 .../testing/selftests/kvm/memory_attributes.c | 424 ++++++++++++++++++
 tools/testing/selftests/kvm/x86/hcall_fault.c | 246 ++++++++++
 .../testing/selftests/kvm/x86/hyperv_evmcs.c  |  16 +-
 .../selftests/kvm/x86/hyperv_svm_test.c       |  15 +-
 .../selftests/kvm/x86/memory_attributes.c     | 415 +++++++++++++++++
 .../kvm/x86/private_mem_kvm_exits_test.c      |   5 +-
 virt/kvm/kvm_main.c                           | 315 +++++++++++--
 virt/kvm/pfncache.c                           |  30 +-
 42 files changed, 2498 insertions(+), 441 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/memory_attributes.c
 create mode 100644 tools/testing/selftests/kvm/x86/hcall_fault.c
 create mode 100644 tools/testing/selftests/kvm/x86/memory_attributes.c

-- 
2.52.0

[PATCH 01/45] KVM: nVMX/nSVM: check unconditionally whether CR3 is valid

[Paolo Bonzini]

nested_vmcb_check_save() checks CR3 legality only if long mode is active,
but it really needs to be checked in all cases - the check uses must be
zero bits of the host, and indeed nested_svm_load_cr3() later will
reject illegal CR3 in every mode.  Move the check earlier, and thus out
of nested_svm_load_cr3().

Likewise for nested_vmx_load_cr3() and nested_vmx_check_guest_state().

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/svm/nested.c | 9 +++------
 arch/x86/kvm/vmx/nested.c | 8 ++------
 2 files changed, 5 insertions(+), 12 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index af0ef678433d..fedddcd1cc96 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -443,7 +443,8 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 		return false;
 
 	if (CC((save->cr0 & X86_CR0_CD) == 0 && (save->cr0 & X86_CR0_NW)) ||
-	    CC(save->cr0 & ~0xffffffffULL))
+	    CC(save->cr0 & ~0xffffffffULL) ||
+	    CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3)))
 		return false;
 
 	if (CC(!kvm_dr6_valid(save->dr6)) || CC(!kvm_dr7_valid(save->dr7)))
@@ -456,8 +457,7 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu,
 	 */
 	if ((save->efer & EFER_LME) && (save->cr0 & X86_CR0_PG)) {
 		if (CC(!(save->cr4 & X86_CR4_PAE)) ||
-		    CC(!(save->cr0 & X86_CR0_PE)) ||
-		    CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3)))
+		    CC(!(save->cr0 & X86_CR0_PE)))
 			return false;
 
 		if (CC((save->cs.attrib & SVM_SELECTOR_L_MASK) &&
@@ -717,9 +717,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
 static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_npt, bool reload_pdptrs)
 {
-	if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3)))
-		return -EINVAL;
-
 	if (reload_pdptrs && is_pae_paging(vcpu)) {
 		if (nested_npt)
 			kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 9b0bfa2f854c..b133e53fde39 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1235,11 +1235,6 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_ept, bool reload_pdptrs,
 			       enum vm_entry_failure_code *entry_failure_code)
 {
-	if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) {
-		*entry_failure_code = ENTRY_FAIL_DEFAULT;
-		return -EINVAL;
-	}
-
 	/*
 	 * If PAE paging and EPT are both on, CR3 is not used by the CPU and
 	 * must not be dereferenced.
@@ -3323,7 +3318,8 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu,
 	*entry_failure_code = ENTRY_FAIL_DEFAULT;
 
 	if (CC(!nested_guest_cr0_valid(vcpu, vmcs12->guest_cr0)) ||
-	    CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4)))
+	    CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4)) ||
+	    CC(!kvm_vcpu_is_legal_cr3(vcpu, vmcs12->guest_cr3)))
 		return -EINVAL;
 
 	if (CC(vmcs12->guest_cr4 & X86_CR4_CET && !(vmcs12->guest_cr0 & X86_CR0_WP)))
-- 
2.52.0

[PATCH 02/45] KVM: x86/hyperv: do not overwrite hc->ingpa for SIGNAL_EVENT hypercall

[Paolo Bonzini]

When a guest makes a HVCALL_SIGNAL_EVENT hypercall that is not fast
(i.e. has arguments in memory), kvm_hvcall_signal_event() reads the
connection ID and overwrites hc->ingpa.

If the connection ID that is valid in userspace but not registered in the
kernel conn_to_evt, KVM exits to userspace; however, hc->param still
signals that the hypercall was not fast, and userspace will then treat
the connection ID as an address.

Cc: stable@vger.kernel.org
Fixes: bd38b32053eb ("KVM: hyper-v: Collect hypercall params into struct")
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-2-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/hyperv.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 5d2e3617c952..798baa9eaafb 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2437,19 +2437,22 @@ static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *h
 {
 	struct kvm_hv *hv = to_kvm_hv(vcpu->kvm);
 	struct eventfd_ctx *eventfd;
+	u64 conn_id;
 
 	if (unlikely(!hc->fast)) {
 		int ret;
 		gpa_t gpa = hc->ingpa;
 
-		if ((gpa & (__alignof__(hc->ingpa) - 1)) ||
-		    offset_in_page(gpa) + sizeof(hc->ingpa) > PAGE_SIZE)
+		if ((gpa & (__alignof__(conn_id) - 1)) ||
+		    offset_in_page(gpa) + sizeof(conn_id) > PAGE_SIZE)
 			return HV_STATUS_INVALID_ALIGNMENT;
 
 		ret = kvm_vcpu_read_guest(vcpu, gpa,
-					  &hc->ingpa, sizeof(hc->ingpa));
+					  &conn_id, sizeof(conn_id));
 		if (ret < 0)
 			return HV_STATUS_INVALID_ALIGNMENT;
+	} else {
+		conn_id = hc->ingpa;
 	}
 
 	/*
@@ -2457,15 +2460,15 @@ static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *h
 	 * have no use for it, and in all known usecases it is zero, so just
 	 * report lookup failure if it isn't.
 	 */
-	if (hc->ingpa & 0xffff00000000ULL)
+	if (conn_id & 0xffff00000000ULL)
 		return HV_STATUS_INVALID_PORT_ID;
 	/* remaining bits are reserved-zero */
-	if (hc->ingpa & ~KVM_HYPERV_CONN_ID_MASK)
+	if (conn_id & ~KVM_HYPERV_CONN_ID_MASK)
 		return HV_STATUS_INVALID_HYPERCALL_INPUT;
 
 	/* the eventfd is protected by vcpu->kvm->srcu, but conn_to_evt isn't */
 	rcu_read_lock();
-	eventfd = idr_find(&hv->conn_to_evt, hc->ingpa);
+	eventfd = idr_find(&hv->conn_to_evt, conn_id);
 	rcu_read_unlock();
 	if (!eventfd)
 		return HV_STATUS_INVALID_PORT_ID;
-- 
2.52.0

[PATCH 03/45] KVM: x86: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING

[Paolo Bonzini]

KVM_HC_CLOCK_PAIRING writes to a guest physical addresses.  Treat it
as an L2 address when running nested, which is consistent with the
Hyper-V TLB flush hypercalls for example.

This is technically backwards incompatible, but it should be extremely
unlikely to happen in the wild and thus we can treat it as a bugfix, I think.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-6-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/x86.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32563a91fba9..ec151374ca9e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7256,8 +7256,8 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 {
 	struct kvm_clock_pairing clock_pairing;
 	struct timespec64 ts;
+	size_t offset = 0;
 	u64 cycle;
-	int ret;
 
 	if (clock_type != KVM_CLOCK_PAIRING_WALLCLOCK)
 		return -KVM_EOPNOTSUPP;
@@ -7278,12 +7278,20 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 	clock_pairing.flags = 0;
 	memset(&clock_pairing.pad, 0, sizeof(clock_pairing.pad));
 
-	ret = 0;
-	if (kvm_write_guest(vcpu->kvm, paddr, &clock_pairing,
-			    sizeof(struct kvm_clock_pairing)))
-		ret = -KVM_EFAULT;
+	while (offset < sizeof(clock_pairing)) {
+		gpa_t gpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, paddr + offset,
+					      PFERR_WRITE_MASK | PFERR_GUEST_FINAL_MASK, NULL, 0);
+		size_t len = min_t(size_t, sizeof(clock_pairing) - offset,
+				   PAGE_SIZE - offset_in_page(gpa));
 
-	return ret;
+		if (gpa == INVALID_GPA)
+			return -KVM_EFAULT;
+		if (kvm_write_guest(vcpu->kvm, gpa, (u8 *)&clock_pairing + offset, len))
+			return -KVM_EFAULT;
+
+		offset += len;
+	}
+	return 0;
 }
 #endif
 
-- 
2.52.0

[PATCH 04/45] KVM: x86/emulate: Check TSS busy-bit writes during task switches

[Paolo Bonzini]

Task switching ignores failures when clearing the old TSS descriptor's
busy bit and setting the new descriptor's busy bit.  A memory attribute
restriction can therefore drop either write while the task switch
continues with inconsistent descriptor state.

Check both writes and stop the task switch on failure.  Memory attribute
faults reach the existing emulation-failure exit in kvm_task_switch().

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/emulate.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index c1b21282187f..711794592c2c 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -2925,7 +2925,9 @@ static int emulator_do_task_switch(struct x86_emulate_ctxt *ctxt,
 
 	if (reason == TASK_SWITCH_IRET || reason == TASK_SWITCH_JMP) {
 		curr_tss_desc.type &= ~(1 << 1); /* clear busy flag */
-		write_segment_descriptor(ctxt, old_tss_sel, &curr_tss_desc);
+		ret = write_segment_descriptor(ctxt, old_tss_sel, &curr_tss_desc);
+		if (ret != X86EMUL_CONTINUE)
+			return ret;
 	}
 
 	if (reason == TASK_SWITCH_IRET)
@@ -2949,7 +2951,9 @@ static int emulator_do_task_switch(struct x86_emulate_ctxt *ctxt,
 
 	if (reason != TASK_SWITCH_IRET) {
 		next_tss_desc.type |= (1 << 1); /* set busy flag */
-		write_segment_descriptor(ctxt, tss_selector, &next_tss_desc);
+		ret = write_segment_descriptor(ctxt, tss_selector, &next_tss_desc);
+		if (ret != X86EMUL_CONTINUE)
+			return ret;
 	}
 
 	ops->set_cr(ctxt, 0,  ops->get_cr(ctxt, 0) | X86_CR0_TS);
-- 
2.52.0

[PATCH 05/45] KVM: loongarch: do full validity check on the gfn-to-hva cache

[Paolo Bonzini]

If the HVA or the memslots are invalid, reinitialize the cache.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-20-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/loongarch/kvm/vcpu.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/loongarch/kvm/vcpu.c b/arch/loongarch/kvm/vcpu.c
index 4026b9bf9f2d..059a6d605de7 100644
--- a/arch/loongarch/kvm/vcpu.c
+++ b/arch/loongarch/kvm/vcpu.c
@@ -165,7 +165,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	slots = kvm_memslots(vcpu->kvm);
-	if (slots->generation != ghc->generation || gpa != ghc->gpa) {
+	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
@@ -1820,7 +1821,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
 	slots = kvm_memslots(vcpu->kvm);
-	if (slots->generation != ghc->generation || gpa != ghc->gpa) {
+	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
-- 
2.52.0

[PATCH 06/45] KVM: selftests: Test address translation for Hyper-V direct L2 hypercalls

[Paolo Bonzini]

Test that the TLB flush hypercalls, when executed by L2, translate their
input addresses through the guest's EPT or NPT.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-5-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tools/testing/selftests/kvm/x86/hyperv_evmcs.c    | 12 +++++++++++-
 tools/testing/selftests/kvm/x86/hyperv_svm_test.c | 13 ++++++++++++-
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/kvm/x86/hyperv_evmcs.c b/tools/testing/selftests/kvm/x86/hyperv_evmcs.c
index c81f2dd4f2a1..2e1150c4bbbf 100644
--- a/tools/testing/selftests/kvm/x86/hyperv_evmcs.c
+++ b/tools/testing/selftests/kvm/x86/hyperv_evmcs.c
@@ -18,6 +18,8 @@
 #include "hyperv.h"
 #include "vmx.h"
 
+#define L2_HCALL_INPUT_GPA	0x100000000ull
+
 static int ud_count;
 
 static void guest_ud_handler(struct ex_regs *regs)
@@ -64,6 +66,8 @@ void l2_guest_code(void)
 	rdmsr_from_l2(MSR_GS_BASE); /* intercepted */
 
 	/* L2 TLB flush tests */
+	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+			 L2_HCALL_INPUT_GPA, 0);
 	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE | HV_HYPERCALL_FAST_BIT, 0x0,
 			 HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES | HV_FLUSH_ALL_PROCESSORS);
 	rdmsr_from_l2(MSR_FS_BASE);
@@ -239,7 +243,7 @@ static struct kvm_vcpu *save_restore_vm(struct kvm_vm *vm,
 int main(int argc, char *argv[])
 {
 	gva_t vmx_pages_gva = 0, hv_pages_gva = 0;
-	gva_t hcall_page;
+	gva_t hcall_input_page, hcall_page;
 
 	struct kvm_vcpu *vcpu;
 	struct kvm_vm *vm;
@@ -250,17 +254,23 @@ int main(int argc, char *argv[])
 	TEST_REQUIRE(kvm_has_cap(KVM_CAP_NESTED_STATE));
 	TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_ENLIGHTENED_VMCS));
 	TEST_REQUIRE(kvm_hv_cpu_has(HV_X64_NESTED_DIRECT_FLUSH));
+	TEST_REQUIRE(kvm_cpu_has_ept());
 
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	vm_enable_ept(vm);
 
 	hcall_page = vm_alloc_pages(vm, 1);
 	memset(addr_gva2hva(vm, hcall_page), 0x0,  getpagesize());
+	hcall_input_page = vm_alloc_page(vm);
+	memset(addr_gva2hva(vm, hcall_input_page), 0, getpagesize());
 
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_enable_evmcs(vcpu);
 
 	vcpu_alloc_vmx(vm, &vmx_pages_gva);
 	vcpu_alloc_hyperv_test_pages(vm, &hv_pages_gva);
+	tdp_identity_map_default_memslots(vm);
+	tdp_map(vm, L2_HCALL_INPUT_GPA, addr_gva2gpa(vm, hcall_input_page), PAGE_SIZE);
 	vcpu_args_set(vcpu, 3, vmx_pages_gva, hv_pages_gva, addr_gva2gpa(vm, hcall_page));
 	vcpu_set_msr(vcpu, HV_X64_MSR_VP_INDEX, vcpu->id);
 
diff --git a/tools/testing/selftests/kvm/x86/hyperv_svm_test.c b/tools/testing/selftests/kvm/x86/hyperv_svm_test.c
index 3a3dc31072e8..1c95b86ce2a5 100644
--- a/tools/testing/selftests/kvm/x86/hyperv_svm_test.c
+++ b/tools/testing/selftests/kvm/x86/hyperv_svm_test.c
@@ -18,6 +18,8 @@
 #include "svm_util.h"
 #include "hyperv.h"
 
+#define L2_HCALL_INPUT_GPA	0x100000000ull
+
 /* Exit to L1 from L2 with RDMSR instruction */
 static inline void rdmsr_from_l2(u32 msr)
 {
@@ -45,6 +47,8 @@ void l2_guest_code(void)
 	GUEST_SYNC(5);
 
 	/* L2 TLB flush tests */
+	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+			 L2_HCALL_INPUT_GPA, 0);
 	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE |
 			 HV_HYPERCALL_FAST_BIT, 0x0,
 			 HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES |
@@ -146,7 +150,7 @@ static void __attribute__((__flatten__)) guest_code(struct svm_test_data *svm,
 int main(int argc, char *argv[])
 {
 	gva_t nested_gva = 0, hv_pages_gva = 0;
-	gva_t hcall_page;
+	gva_t hcall_input_page, hcall_page;
 	struct kvm_vcpu *vcpu;
 	struct kvm_vm *vm;
 	struct ucall uc;
@@ -154,15 +158,22 @@ int main(int argc, char *argv[])
 
 	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM));
 	TEST_REQUIRE(kvm_hv_cpu_has(HV_X64_NESTED_DIRECT_FLUSH));
+	TEST_REQUIRE(kvm_cpu_has_npt());
 
 	/* Create VM */
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	vm_enable_npt(vm);
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_alloc_svm(vm, &nested_gva);
 	vcpu_alloc_hyperv_test_pages(vm, &hv_pages_gva);
 
 	hcall_page = vm_alloc_pages(vm, 1);
 	memset(addr_gva2hva(vm, hcall_page), 0x0,  getpagesize());
+	hcall_input_page = vm_alloc_page(vm);
+	memset(addr_gva2hva(vm, hcall_input_page), 0, getpagesize());
+
+	tdp_identity_map_default_memslots(vm);
+	tdp_map(vm, L2_HCALL_INPUT_GPA, addr_gva2gpa(vm, hcall_input_page), PAGE_SIZE);
 
 	vcpu_args_set(vcpu, 3, nested_gva, hv_pages_gva, addr_gva2gpa(vm, hcall_page));
 	vcpu_set_msr(vcpu, HV_X64_MSR_VP_INDEX, vcpu->id);
-- 
2.52.0

[PATCH 07/45] KVM: selftests: Take into account mixed memory fault flags

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Update x86_64/private_mem_kvm_exits_test to take into account memory
fault flags might contain multiple bits set while remaining valid---for
example KVM might return KVM_MEMORY_EXIT_FLAG_READ in addition to
KVM_MEMORY_EXIT_FLAG_PRIVATE.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-3-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c b/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c
index f7228e0f1ac6..ed8edde9b718 100644
--- a/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c
+++ b/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c
@@ -91,6 +91,8 @@ static void test_private_access_memslot_deleted(void)
 	} else {
 		TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MEMORY_FAULT);
 		TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, KVM_MEMORY_EXIT_FLAG_PRIVATE);
+		TEST_ASSERT(vcpu->run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_PRIVATE,
+			    "Memory fault didn't occur on a private memory access");
 		TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, EXITS_TEST_GPA);
 		TEST_ASSERT_EQ(vcpu->run->memory_fault.size, EXITS_TEST_SIZE);
 	}
@@ -127,7 +129,8 @@ static void test_private_access_memslot_not_private(void)
 	exit_reason = run_vcpu_get_exit_reason(vcpu);
 
 	TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MEMORY_FAULT);
-	TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, KVM_MEMORY_EXIT_FLAG_PRIVATE);
+	TEST_ASSERT(vcpu->run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_PRIVATE,
+		    "Memory fault didn't occur on a private memory access");
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, EXITS_TEST_GPA);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, EXITS_TEST_SIZE);
 
-- 
2.52.0

[PATCH 08/45] selftests: kvm: drop TEST_ASSERT_SUPPORTED_ATTRIBUTES

[Paolo Bonzini]

TEST_ASSERT_SUPPORTED_ATTRIBUTES() is supposedly reminding tha
__vm_set_memory_attributes() does not support multiple attributes, but
that is not really a problem; the tests need to know that multiple
attributes can be overwritten, but that is not a problem for a simple
wrapper around KVM_SET_MEMORY_ATTRIBUTES.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tools/testing/selftests/kvm/include/kvm_util.h | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h
index e2618877619e..7e480c034a87 100644
--- a/tools/testing/selftests/kvm/include/kvm_util.h
+++ b/tools/testing/selftests/kvm/include/kvm_util.h
@@ -473,14 +473,6 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0)
 int kvm_gpa_to_guest_memfd(struct kvm_vm *vm, gpa_t gpa, off_t *fd_offset,
 			   size_t *nr_bytes);
 
-/*
- * KVM_SET_MEMORY_ATTRIBUTES{,2} overwrites _all_ attributes.  These
- * flows need significant enhancements to support multiple attributes.
- */
-#define TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes)				\
-	TEST_ASSERT(!(attributes) || (attributes) == KVM_MEMORY_ATTRIBUTE_PRIVATE,	\
-		    "Update me to support multiple attributes!")
-
 static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
 					    size_t size, u64 attributes)
 {
@@ -491,8 +483,6 @@ static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
 		.flags = 0,
 	};
 
-	TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes);
-
 	vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
 }
 
@@ -543,8 +533,6 @@ static inline void gmem_set_memory_attributes(int fd, u64 offset,
 		.flags = 0,
 	};
 
-	TEST_ASSERT_SUPPORTED_ATTRIBUTES(attributes);
-
 	__TEST_REQUIRE(kvm_check_cap(KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES) > 0,
 		       "No valid attributes for guest_memfd ioctl!");
 
-- 
2.52.0

[PATCH 09/45] KVM: Define and communicate KVM_EXIT_MEMORY_FAULT RWX flags to userspace

[Paolo Bonzini]

From: Anish Moorthy <amoorthy@google.com>

kvm_prepare_memory_fault_exit() already takes parameters describing the
RWX-ness of the relevant access but doesn't actually do anything with
them. Define and use the flags necessary to pass this information on to
userspace.

Suggested-by: Sean Christopherson <seanjc@google.com>
Link: https://lore.kernel.org/kvm/ZR4N8cwzTMDanPUY@google.com/
Signed-off-by: Anish Moorthy <amoorthy@google.com>
Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-4-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst | 5 +++++
 include/linux/kvm_host.h       | 9 ++++++++-
 include/uapi/linux/kvm.h       | 3 +++
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index e95c9cda085d..7123d021bdbb 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7481,6 +7481,9 @@ spec refer, https://github.com/riscv/riscv-sbi-doc.
 
 		/* KVM_EXIT_MEMORY_FAULT */
 		struct {
+  #define KVM_MEMORY_EXIT_FLAG_READ     (1ULL << 0)
+  #define KVM_MEMORY_EXIT_FLAG_WRITE    (1ULL << 1)
+  #define KVM_MEMORY_EXIT_FLAG_EXEC     (1ULL << 2)
   #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
 			__u64 flags;
 			__u64 gpa;
@@ -7492,6 +7495,8 @@ could not be resolved by KVM.  The 'gpa' and 'size' (in bytes) describe the
 guest physical address range [gpa, gpa + size) of the fault.  The 'flags' field
 describes properties of the faulting access that are likely pertinent:
 
+ - KVM_MEMORY_EXIT_FLAG_READ/WRITE/EXEC - When set, indicates that the memory
+   fault occurred on a read/write/exec access respectively.
  - KVM_MEMORY_EXIT_FLAG_PRIVATE - When set, indicates the memory fault occurred
    on a private memory access.  When clear, indicates the fault occurred on a
    shared access.
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e7fc94e2910a..5e386c3d2495 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2556,8 +2556,15 @@ static inline void kvm_prepare_memory_fault_exit(struct kvm_vcpu *vcpu,
 	vcpu->run->memory_fault.gpa = gpa;
 	vcpu->run->memory_fault.size = size;
 
-	/* RWX flags are not (yet) defined or communicated to userspace. */
 	vcpu->run->memory_fault.flags = 0;
+
+	if (is_write)
+		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_WRITE;
+	else if (is_exec)
+		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_EXEC;
+	else
+		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_READ;
+
 	if (is_private)
 		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_PRIVATE;
 }
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 8dff2fc1972e..229b3bdb7aed 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -457,6 +457,9 @@ struct kvm_run {
 		} notify;
 		/* KVM_EXIT_MEMORY_FAULT */
 		struct {
+#define KVM_MEMORY_EXIT_FLAG_READ       (1ULL << 0)
+#define KVM_MEMORY_EXIT_FLAG_WRITE      (1ULL << 1)
+#define KVM_MEMORY_EXIT_FLAG_EXEC       (1ULL << 2)
 #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
 			__u64 flags;
 			__u64 gpa;
-- 
2.52.0

[PATCH 10/45] KVM: x86: Introduce memory fault on invalid hypercalls reads/writes

[Paolo Bonzini]

Allow userspace to perform arbitrary actions when a hypercall refers to an
invalid address, by exiting with KVM_EXIT_MEMORY_FAULT.  This will for
example allow userspace to perform a VTL call.

Co-developed-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-7-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst  |  19 ++++
 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/kvm/hyperv.c           | 157 +++++++++++++++++++++++++-------
 arch/x86/kvm/x86.c              |  34 +++++--
 include/uapi/linux/kvm.h        |   1 +
 5 files changed, 173 insertions(+), 39 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 7123d021bdbb..d0f91687bdb9 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -9186,6 +9186,25 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key
 interpretation are disabled. If cmma has already been enabled or the
 hpage_2g module parameter is not set to 1, -EINVAL is returned.
 
+7.48 KVM_CAP_HCALL_FAULT_EXIT
+------------------------------------
+
+:Architectures: x86
+:Parameters: args[0] is 0 to disable, 1 to enable
+
+When enabled, KVM checks the memory that is read or written by
+hypercalls (including slow Hyper-V hypercalls and KVM_HC_CLOCK_PAIRING).
+An inaccessible input page causes a KVM_EXIT_MEMORY_FAULT with
+KVM_MEMORY_EXIT_FLAG_READ.  An inaccessible or read-only output page causes
+a KVM_EXIT_MEMORY_FAULT with KVM_MEMORY_EXIT_FLAG_WRITE.  The reported
+range identifies the page containing the parameter GPA.
+
+Hypercall parameters that are unused by the selected hypercall are not checked.
+
+For Hyper-V, unknown hypercalls are passed to userspace without checking their
+parameter pages.  The fault GPA is in the physical address space of the
+VM managed by userspace, after nested GPA translation.
+
 8. Other capabilities.
 ======================
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 0378577d9d8b..25256d59cfde 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1178,6 +1178,7 @@ struct kvm_arch {
 	bool has_protected_eoi;
 	bool has_protected_pmu;
 	bool pre_fault_allowed;
+	bool hcall_fault_exit;
 	struct hlist_head *mmu_page_hash;
 	struct list_head active_mmu_pages;
 	struct kvm_possible_nx_huge_pages possible_nx_huge_pages[KVM_NR_MMU_TYPES];
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 798baa9eaafb..ee5189815a06 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2034,7 +2034,68 @@ int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu)
 	return -ENOSPC;
 }
 
-static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+static int kvm_hv_hypercall_check_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
+				      bool write)
+{
+	bool writable = true;
+	gfn_t gfn = gpa_to_gfn(gpa);
+	unsigned long addr;
+
+	addr = kvm_vcpu_gfn_to_hva_prot(vcpu, gfn, &writable);
+	if (!kvm_is_error_hva(addr) && (!write || writable))
+		return 0;
+
+	kvm_prepare_memory_fault_exit(vcpu, gfn_to_gpa(gfn), PAGE_SIZE,
+				      write, false, false);
+	return -EFAULT;
+}
+
+static unsigned int kvm_hv_hypercall_mem_access(u16 code)
+{
+	switch (code) {
+	case HVCALL_SIGNAL_EVENT:
+	case HVCALL_POST_MESSAGE:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX:
+	case HVCALL_SEND_IPI:
+	case HVCALL_SEND_IPI_EX:
+		return KVM_MEMORY_EXIT_FLAG_READ;
+	case HVCALL_POST_DEBUG_DATA:
+	case HVCALL_RETRIEVE_DEBUG_DATA:
+		return KVM_MEMORY_EXIT_FLAG_READ | KVM_MEMORY_EXIT_FLAG_WRITE;
+	case HVCALL_RESET_DEBUG_SESSION:
+	case HV_EXT_CALL_QUERY_CAPABILITIES:
+		return KVM_MEMORY_EXIT_FLAG_WRITE;
+	}
+
+	return 0;
+}
+
+static int kvm_hv_hypercall_check_params(struct kvm_vcpu *vcpu,
+					 struct kvm_hv_hcall *hc)
+{
+	unsigned access;
+	int r;
+
+	if (hc->fast || !vcpu->kvm->arch.hcall_fault_exit)
+		return 0;
+
+	access = kvm_hv_hypercall_mem_access(hc->code);
+	if (access & KVM_MEMORY_EXIT_FLAG_READ) {
+		r = kvm_hv_hypercall_check_gpa(vcpu, hc->ingpa, false);
+		if (r)
+			return r;
+	}
+
+	if (access & KVM_MEMORY_EXIT_FLAG_WRITE)
+		return kvm_hv_hypercall_check_gpa(vcpu, hc->outgpa, true);
+
+	return 0;
+}
+
+static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	unsigned long *vcpu_mask = hv_vcpu->vcpu_mask;
@@ -2054,6 +2115,19 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	struct kvm_vcpu *v;
 	unsigned long i;
 	bool all_cpus;
+	int r;
+
+	/* Slow direct hypercalls from L2 provide a nested GPA. */
+	if (!hc->fast) {
+		hc->ingpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, hc->ingpa,
+					      PFERR_GUEST_FINAL_MASK, NULL, 0);
+		if (unlikely(hc->ingpa == INVALID_GPA))
+			return HV_STATUS_INVALID_HYPERCALL_INPUT;
+	}
+
+	r = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (r)
+		return r;
 
 	/*
 	 * The Hyper-V TLFS doesn't allow more than HV_MAX_SPARSE_VCPU_BANKS
@@ -2062,20 +2136,6 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	 */
 	BUILD_BUG_ON(KVM_HV_MAX_SPARSE_VCPU_SET_BITS > HV_MAX_SPARSE_VCPU_BANKS);
 
-	/*
-	 * 'Slow' hypercall's first parameter is the address in guest's memory
-	 * where hypercall parameters are placed. This is either a GPA or a
-	 * nested GPA when KVM is handling the call from L2 ('direct' TLB
-	 * flush).  Translate the address here so the memory can be uniformly
-	 * read with kvm_read_guest().
-	 */
-	if (!hc->fast) {
-		hc->ingpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, hc->ingpa,
-					      PFERR_GUEST_FINAL_MASK, NULL, 0);
-		if (unlikely(hc->ingpa == INVALID_GPA))
-			return HV_STATUS_INVALID_HYPERCALL_INPUT;
-	}
-
 	if (hc->code == HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST ||
 	    hc->code == HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE) {
 		if (hc->fast) {
@@ -2243,7 +2303,7 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector,
 	}
 }
 
-static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	u64 *sparse_banks = hv_vcpu->sparse_banks;
@@ -2253,6 +2313,11 @@ static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	u64 valid_bank_mask;
 	u32 vector;
 	bool all_cpus;
+	int r;
+
+	r = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (r)
+		return r;
 
 	if (!lapic_in_kernel(vcpu))
 		return HV_STATUS_INVALID_HYPERCALL_INPUT;
@@ -2433,11 +2498,16 @@ static int kvm_hv_hypercall_complete_userspace(struct kvm_vcpu *vcpu)
 	return kvm_hv_hypercall_complete(vcpu, vcpu->run->hyperv.u.hcall.result);
 }
 
-static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+static int kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 {
 	struct kvm_hv *hv = to_kvm_hv(vcpu->kvm);
 	struct eventfd_ctx *eventfd;
 	u64 conn_id;
+	int ret;
+
+	ret = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (ret)
+		return ret;
 
 	if (unlikely(!hc->fast)) {
 		int ret;
@@ -2551,11 +2621,30 @@ static bool hv_check_hypercall_access(struct kvm_vcpu_hv *hv_vcpu, u16 code)
 	return true;
 }
 
+static int kvm_hv_hypercall_userspace_exit(struct kvm_vcpu *vcpu,
+					   struct kvm_hv_hcall *hc)
+{
+	int r;
+
+	r = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (r)
+		return r;
+
+	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
+	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
+	vcpu->run->hyperv.u.hcall.input = hc->param;
+	vcpu->run->hyperv.u.hcall.params[0] = hc->ingpa;
+	vcpu->run->hyperv.u.hcall.params[1] = hc->outgpa;
+	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
+	return 0;
+}
+
 int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	struct kvm_hv_hcall hc;
 	u64 ret = HV_STATUS_SUCCESS;
+	s64 r;
 
 	/*
 	 * hypercall generates UD from non zero cpl and real mode
@@ -2623,7 +2712,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hvcall_signal_event(vcpu, &hc);
+		r = kvm_hvcall_signal_event(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		if (ret != HV_STATUS_INVALID_PORT_ID)
 			break;
 		fallthrough;	/* maybe userspace knows this conn_id */
@@ -2633,7 +2725,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		goto hypercall_userspace_exit;
+		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST:
 		if (unlikely(hc.var_cnt)) {
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
@@ -2645,7 +2737,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hv_flush_tlb(vcpu, &hc);
+		r = kvm_hv_flush_tlb(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		break;
 	case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE:
 		if (unlikely(hc.var_cnt)) {
@@ -2658,7 +2753,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hv_flush_tlb(vcpu, &hc);
+		r = kvm_hv_flush_tlb(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		break;
 	case HVCALL_SEND_IPI:
 		if (unlikely(hc.var_cnt)) {
@@ -2671,7 +2769,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hv_send_ipi(vcpu, &hc);
+		r = kvm_hv_send_ipi(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		break;
 	case HVCALL_POST_DEBUG_DATA:
 	case HVCALL_RETRIEVE_DEBUG_DATA:
@@ -2692,14 +2793,14 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_OPERATION_DENIED;
 			break;
 		}
-		goto hypercall_userspace_exit;
+		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	}
 	case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
 		if (unlikely(hc.fast)) {
 			ret = HV_STATUS_INVALID_PARAMETER;
 			break;
 		}
-		goto hypercall_userspace_exit;
+		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	default:
 		ret = HV_STATUS_INVALID_HYPERCALL_CODE;
 		break;
@@ -2708,14 +2809,6 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 hypercall_complete:
 	return kvm_hv_hypercall_complete(vcpu, ret);
 
-hypercall_userspace_exit:
-	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
-	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
-	vcpu->run->hyperv.u.hcall.input = hc.param;
-	vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
-	vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
-	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
-	return 0;
 }
 
 void kvm_hv_init_vm(struct kvm *kvm)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ec151374ca9e..1fcef4a8ac22 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2253,6 +2253,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_HYPERV_ENFORCE_CPUID:
 	case KVM_CAP_SYS_HYPERV_CPUID:
 #endif
+	case KVM_CAP_HCALL_FAULT_EXIT:
 	case KVM_CAP_PCI_SEGMENT:
 	case KVM_CAP_DEBUGREGS:
 	case KVM_CAP_X86_ROBUST_SINGLESTEP:
@@ -4252,6 +4253,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		mutex_unlock(&kvm->lock);
 		break;
 	}
+	case KVM_CAP_HCALL_FAULT_EXIT:
+		kvm->arch.hcall_fault_exit = cap->args[0];
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -7252,25 +7257,28 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_vendor_exit);
 
 #ifdef CONFIG_X86_64
 static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
-			        unsigned long clock_type)
+			        unsigned long clock_type, unsigned long *ret)
 {
 	struct kvm_clock_pairing clock_pairing;
 	struct timespec64 ts;
 	size_t offset = 0;
+	bool writable;
+	unsigned long hva;
 	u64 cycle;
 
+	*ret = -KVM_EOPNOTSUPP;
 	if (clock_type != KVM_CLOCK_PAIRING_WALLCLOCK)
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	/*
 	 * When tsc is in permanent catchup mode guests won't be able to use
 	 * pvclock_read_retry loop to get consistent view of pvclock
 	 */
 	if (vcpu->arch.tsc_always_catchup)
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	if (!kvm_get_walltime_and_clockread(&ts, &cycle))
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	clock_pairing.sec = ts.tv_sec;
 	clock_pairing.nsec = ts.tv_nsec;
@@ -7278,6 +7286,7 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 	clock_pairing.flags = 0;
 	memset(&clock_pairing.pad, 0, sizeof(clock_pairing.pad));
 
+	*ret = -KVM_EFAULT;
 	while (offset < sizeof(clock_pairing)) {
 		gpa_t gpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, paddr + offset,
 					      PFERR_WRITE_MASK | PFERR_GUEST_FINAL_MASK, NULL, 0);
@@ -7285,12 +7294,22 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 				   PAGE_SIZE - offset_in_page(gpa));
 
 		if (gpa == INVALID_GPA)
-			return -KVM_EFAULT;
+			return 0;
+		if (vcpu->kvm->arch.hcall_fault_exit) {
+			hva = kvm_vcpu_gfn_to_hva_prot(vcpu, gpa_to_gfn(gpa), &writable);
+			if (kvm_is_error_hva(hva) || !writable) {
+				kvm_prepare_memory_fault_exit(vcpu, gpa & PAGE_MASK, PAGE_SIZE,
+							      true, false, false);
+				return -EFAULT;
+			}
+		}
 		if (kvm_write_guest(vcpu->kvm, gpa, (u8 *)&clock_pairing + offset, len))
-			return -KVM_EFAULT;
+			return 0;
 
 		offset += len;
 	}
+
+	*ret = 0;
 	return 0;
 }
 #endif
@@ -7449,7 +7468,8 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl,
 		break;
 #ifdef CONFIG_X86_64
 	case KVM_HC_CLOCK_PAIRING:
-		ret = kvm_pv_clock_pairing(vcpu, a0, a1);
+		if (kvm_pv_clock_pairing(vcpu, a0, a1, &ret))
+			return -EFAULT;
 		break;
 #endif
 	case KVM_HC_SEND_IPI:
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 229b3bdb7aed..4e0bab3db95c 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1003,6 +1003,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
 #define KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES 252
+#define KVM_CAP_HCALL_FAULT_EXIT 253
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
-- 
2.52.0

[PATCH 11/45] KVM: selftests: test hypercall memory fault exits

[Paolo Bonzini]

Cover various scenarios where hypercalls are invoked with invalid or
read-only GPAs, and check that they exit to userspace.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-8-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 tools/testing/selftests/kvm/x86/hcall_fault.c | 246 ++++++++++++++++++
 .../testing/selftests/kvm/x86/hyperv_evmcs.c  |   4 +
 .../selftests/kvm/x86/hyperv_svm_test.c       |   2 +
 4 files changed, 253 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/hcall_fault.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 3cb35a6963d5..478b85ceb7fb 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -83,6 +83,7 @@ TEST_GEN_PROGS_x86 += x86/evmcs_smm_controls_test
 TEST_GEN_PROGS_x86 += x86/exit_on_emulation_failure_test
 TEST_GEN_PROGS_x86 += x86/fastops_test
 TEST_GEN_PROGS_x86 += x86/fix_hypercall_test
+TEST_GEN_PROGS_x86 += x86/hcall_fault
 TEST_GEN_PROGS_x86 += x86/hwcr_msr_test
 TEST_GEN_PROGS_x86 += x86/hyperv_clock
 TEST_GEN_PROGS_x86 += x86/hyperv_cpuid
diff --git a/tools/testing/selftests/kvm/x86/hcall_fault.c b/tools/testing/selftests/kvm/x86/hcall_fault.c
new file mode 100644
index 000000000000..725c59ff750c
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/hcall_fault.c
@@ -0,0 +1,246 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <errno.h>
+
+#include "kvm_util.h"
+#include "processor.h"
+#include "hyperv.h"
+
+#define TEST_MEM_GPA		0x100000000ull
+#define READONLY_GPA		(TEST_MEM_GPA + PAGE_SIZE)
+#define UNMAPPED_IN_GPA		(TEST_MEM_GPA + 2 * PAGE_SIZE)
+#define UNMAPPED_OUT_GPA	(TEST_MEM_GPA + 3 * PAGE_SIZE)
+
+struct hcall_test {
+	/* TEST_MEM_GPA in the guest */
+	u64 control;
+	u64 ingpa;
+	u64 outgpa;
+	u64 expected_status;
+	u8 unused[PAGE_SIZE - 32];
+
+	/* READONLY_GPA in the guest */
+	u8 readonly_data[];
+};
+
+#define HV_FLUSH_ALL_PROCESSORS                 BIT(0)
+#define HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES     BIT(1)
+
+/* HvFlushVirtualAddressSpace, HvFlushVirtualAddressList hypercalls */
+struct hv_tlb_flush {
+        u64 address_space;
+        u64 flags;
+        u64 processor_mask;
+        u64 gva_list[];
+} __packed;
+
+static void guest_code(gpa_t hcall_page, struct hcall_test *test)
+{
+	u64 result;
+	u8 vector;
+
+	wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
+	wrmsr(HV_X64_MSR_HYPERCALL, hcall_page);
+
+	vector = __hyperv_hypercall(test->control, test->ingpa, test->outgpa, &result);
+	GUEST_ASSERT(!vector);
+	GUEST_ASSERT_EQ(result & 0xffff, test->expected_status);
+	GUEST_DONE();
+}
+
+static void clock_pairing_guest(gpa_t gpa)
+{
+	kvm_hypercall(KVM_HC_CLOCK_PAIRING, gpa, KVM_CLOCK_PAIRING_WALLCLOCK, 0, 0);
+	GUEST_DONE();
+}
+
+static gva_t create_test_regions(struct kvm_vm *vm)
+{
+	gva_t test_gva;
+
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, TEST_MEM_GPA,
+				    10, 1, 0);
+	memset(addr_gpa2hva(vm, TEST_MEM_GPA), 0, PAGE_SIZE);
+
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, READONLY_GPA,
+				    11, 1, KVM_MEM_READONLY);
+	memset(addr_gpa2hva(vm, READONLY_GPA), 0, PAGE_SIZE);
+
+	test_gva = vm_unused_gva_gap(vm, PAGE_SIZE * 2, 1 << 24);
+	for (int i = 0; i < 2; i++)
+		virt_pg_map(vm, test_gva + PAGE_SIZE * i, TEST_MEM_GPA + PAGE_SIZE * i);
+
+	return test_gva;
+}
+
+static struct kvm_vm *create_vm(struct kvm_vcpu **vcpu, struct hcall_test **test,
+				gpa_t *hcall_page_gpa)
+{
+	gva_t hcall_page, test_gva;
+	struct kvm_vm *vm;
+
+	vm = vm_create_with_one_vcpu(vcpu, guest_code);
+	vcpu_set_hv_cpuid(*vcpu);
+	vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
+
+	hcall_page = vm_alloc_page(vm);
+	memset(addr_gva2hva(vm, hcall_page), 0, PAGE_SIZE);
+	*hcall_page_gpa = addr_gva2gpa(vm, hcall_page);
+
+	test_gva = create_test_regions(vm);
+	*test = addr_gva2hva(vm, test_gva);
+
+	vcpu_args_set(*vcpu, 2, *hcall_page_gpa, test_gva);
+	return vm;
+}
+
+static void run_vm(struct kvm_vcpu *vcpu)
+{
+	struct ucall uc;
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+	switch (get_ucall(vcpu, &uc)) {
+	case UCALL_ABORT:
+		REPORT_GUEST_ASSERT(uc);
+		break;
+	case UCALL_DONE:
+		break;
+	default:
+		TEST_FAIL("Unexpected ucall: %lu", uc.cmd);
+	}
+}
+
+static void test_unused_gpas(void)
+{
+	struct hcall_test *test;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	gpa_t hcall_page;
+
+	vm = create_vm(&vcpu, &test, &hcall_page);
+	test->control = HVCALL_NOTIFY_LONG_SPIN_WAIT;
+	test->ingpa = UNMAPPED_IN_GPA;
+	test->outgpa = UNMAPPED_OUT_GPA;
+	test->expected_status = HV_STATUS_SUCCESS;
+	run_vm(vcpu);
+	kvm_vm_free(vm);
+
+	vm = create_vm(&vcpu, &test, &hcall_page);
+	test->control = 0xbeef;
+	test->ingpa = UNMAPPED_IN_GPA;
+	test->outgpa = UNMAPPED_OUT_GPA;
+	test->expected_status = HV_STATUS_INVALID_HYPERCALL_CODE;
+	run_vm(vcpu);
+	kvm_vm_free(vm);
+}
+
+static void assert_memory_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 flags)
+{
+	int r;
+
+	r = _vcpu_run(vcpu);
+	TEST_ASSERT(r == -1 && errno == EFAULT, KVM_IOCTL_ERROR(KVM_RUN, r));
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MEMORY_FAULT);
+	TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, flags);
+	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, gpa & PAGE_MASK);
+	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, PAGE_SIZE);
+}
+
+/* Read-only hypercalls, read-only inputs, valid/invalid argument */
+static void test_readonly_input(void)
+{
+	struct hv_tlb_flush *flush;
+	struct hcall_test *test;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	gpa_t hcall_page;
+
+	vm = create_vm(&vcpu, &test, &hcall_page);
+	flush = (struct hv_tlb_flush *)addr_gpa2hva(vm, READONLY_GPA);
+	flush->flags = HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES | HV_FLUSH_ALL_PROCESSORS;
+	flush->processor_mask = 0;
+	test->control = HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE;
+	test->ingpa = READONLY_GPA;
+	test->outgpa = UNMAPPED_OUT_GPA;
+	test->expected_status = HV_STATUS_SUCCESS;
+	run_vm(vcpu);
+	kvm_vm_free(vm);
+
+	/* fails immediately because it requires SynIC */
+	vm = create_vm(&vcpu, &test, &hcall_page);
+	test->control = HVCALL_POST_MESSAGE;
+	test->ingpa = UNMAPPED_IN_GPA;
+	test->outgpa = UNMAPPED_OUT_GPA;
+	test->expected_status = HV_STATUS_INVALID_HYPERCALL_INPUT;
+	run_vm(vcpu);
+	kvm_vm_free(vm);
+}
+
+/* Read-only hypercall, unmapped input */
+static void test_input_fault(void)
+{
+	struct hcall_test *test;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	gpa_t hcall_page;
+
+	vm = create_vm(&vcpu, &test, &hcall_page);
+	test->control = HVCALL_SIGNAL_EVENT;
+	test->ingpa = UNMAPPED_IN_GPA;
+	test->outgpa = UNMAPPED_OUT_GPA;
+	assert_memory_fault(vcpu, test->ingpa, KVM_MEMORY_EXIT_FLAG_READ);
+	kvm_vm_free(vm);
+}
+
+/* Read-write hypercall, unmapped or readonly input and output */
+static void test_output_fault(bool readonly_in, bool readonly_out)
+{
+	struct hcall_test *test;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	gpa_t hcall_page;
+
+	vm = create_vm(&vcpu, &test, &hcall_page);
+	test->control = HV_EXT_CALL_QUERY_CAPABILITIES;
+	test->ingpa = readonly_in ? READONLY_GPA : UNMAPPED_IN_GPA;
+	test->outgpa = readonly_out ? READONLY_GPA : UNMAPPED_OUT_GPA;
+
+	assert_memory_fault(vcpu, test->outgpa, KVM_MEMORY_EXIT_FLAG_WRITE);
+	kvm_vm_free(vm);
+}
+
+static void test_clock_pairing_fault(bool readonly)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	gpa_t gpa;
+
+	/* Unlike create_vm do not enable Hyper-V hypercalls. */
+	vm = vm_create_with_one_vcpu(&vcpu, clock_pairing_guest);
+	create_test_regions(vm);
+	vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
+	gpa = readonly ? READONLY_GPA : UNMAPPED_OUT_GPA;
+	vcpu_args_set(vcpu, 1, gpa);
+
+	assert_memory_fault(vcpu, gpa, KVM_MEMORY_EXIT_FLAG_WRITE);
+	kvm_vm_free(vm);
+}
+
+int main(void)
+{
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_CPUID));
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT));
+	TEST_REQUIRE(kvm_cpuid_has(kvm_get_supported_hv_cpuid(),
+				   HV_ENABLE_EXTENDED_HYPERCALLS));
+
+	test_unused_gpas();
+	test_readonly_input();
+	test_input_fault();
+	test_output_fault(false, false);
+	test_output_fault(true, false);
+	test_output_fault(false, true);
+	test_clock_pairing_fault(false);
+	test_clock_pairing_fault(true);
+	return 0;
+}
diff --git a/tools/testing/selftests/kvm/x86/hyperv_evmcs.c b/tools/testing/selftests/kvm/x86/hyperv_evmcs.c
index 2e1150c4bbbf..6b2fb3fbccd9 100644
--- a/tools/testing/selftests/kvm/x86/hyperv_evmcs.c
+++ b/tools/testing/selftests/kvm/x86/hyperv_evmcs.c
@@ -227,6 +227,8 @@ static struct kvm_vcpu *save_restore_vm(struct kvm_vm *vm,
 
 	/* Restore state in a new VM.  */
 	vcpu = vm_recreate_with_one_vcpu(vm);
+	if (kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT))
+		vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_enable_evmcs(vcpu);
 	vcpu_load_state(vcpu, state);
@@ -257,6 +259,8 @@ int main(int argc, char *argv[])
 	TEST_REQUIRE(kvm_cpu_has_ept());
 
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	if (kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT))
+		vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
 	vm_enable_ept(vm);
 
 	hcall_page = vm_alloc_pages(vm, 1);
diff --git a/tools/testing/selftests/kvm/x86/hyperv_svm_test.c b/tools/testing/selftests/kvm/x86/hyperv_svm_test.c
index 1c95b86ce2a5..ccd45a233b9b 100644
--- a/tools/testing/selftests/kvm/x86/hyperv_svm_test.c
+++ b/tools/testing/selftests/kvm/x86/hyperv_svm_test.c
@@ -162,6 +162,8 @@ int main(int argc, char *argv[])
 
 	/* Create VM */
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	if (kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT))
+		vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
 	vm_enable_npt(vm);
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_alloc_svm(vm, &nested_gva);
-- 
2.52.0

[PATCH 12/45] KVM: x86/mmu: intersect writability from __kvm_faultin_pfn with fault->map_writable

[Paolo Bonzini]

fault->map_writable is currently a pure output of __kvm_faultin_pfn(),
which is the only thing that restricts it.  This will no longer hold once
memory protections derived from memory attributes are applied: those
compute their own access permissions that combine with those from
__kvm_faultin_pfn().

Applying them *before* faulting in the pfn lets a fault that violates
the attributes exit to userspace without the cost of gup and/or an
async #PF; but it means that permissions will then be restricted in two
independent steps, first by memory attributes and then by __kvm_faultin_pfn().
Switch fault->map_writable to that model by letting kvm_mmu_faultin_pfn()
only clear bits rather than assign them.

No functional change intended: nothing writes fault->map_writable between
the initializer and __kvm_mmu_faultin_pfn() yet, so the AND is equivalent
to the assignment it replaces.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-9-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/mmu/mmu.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 5a2cc6a17a9b..38a4583d1ec2 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4688,7 +4688,7 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 		return r;
 	}
 
-	fault->map_writable = !(fault->slot->flags & KVM_MEM_READONLY);
+	fault->map_writable &= !(fault->slot->flags & KVM_MEM_READONLY);
 	fault->max_level = kvm_max_level_for_order(max_order);
 
 	return RET_PF_CONTINUE;
@@ -4698,13 +4698,14 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 				 struct kvm_page_fault *fault)
 {
 	unsigned int foll = fault->write ? FOLL_WRITE : 0;
+	bool writable;
 
 	if (fault->is_private || kvm_memslot_is_gmem_only(fault->slot))
 		return kvm_mmu_faultin_pfn_gmem(vcpu, fault);
 
 	foll |= FOLL_NOWAIT;
 	fault->pfn = __kvm_faultin_pfn(fault->slot, fault->gfn, foll,
-				       &fault->map_writable, &fault->refcounted_page);
+				       &writable, &fault->refcounted_page);
 
 	/*
 	 * If resolving the page failed because I/O is needed to fault-in the
@@ -4713,7 +4714,7 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	 * other failures are terminal, i.e. retrying won't help.
 	 */
 	if (fault->pfn != KVM_PFN_ERR_NEEDS_IO)
-		return RET_PF_CONTINUE;
+		goto out_pf_continue;
 
 	if (!fault->prefetch && kvm_can_do_async_pf(vcpu)) {
 		trace_kvm_try_async_get_page(fault->addr, fault->gfn);
@@ -4734,8 +4735,10 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	foll |= FOLL_INTERRUPTIBLE;
 	foll &= ~FOLL_NOWAIT;
 	fault->pfn = __kvm_faultin_pfn(fault->slot, fault->gfn, foll,
-				       &fault->map_writable, &fault->refcounted_page);
+				       &writable, &fault->refcounted_page);
 
+out_pf_continue:
+	fault->map_writable &= writable;
 	return RET_PF_CONTINUE;
 }
 
@@ -5054,6 +5057,7 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.is_private = err & PFERR_PRIVATE_ACCESS,
 
 		.pfn = KVM_PFN_ERR_FAULT,
+		.map_writable = true,
 	};
 	int r;
 
-- 
2.52.0

[PATCH 13/45] KVM: x86/mmu: Extend map_writable to a full ACC_* mask

[Paolo Bonzini]

Support for memory protection attributes opens the door to installing
non-executable mappings. Instead of introducing yet another member in
struct kvm_page_fault and another argument to make_spte(), make the
existing member map_writable a mask of ACC_* bits.  This also avoids the
need for make_spte() to map a single bool to either the NX bit or
the XS/XU bits together.

Unlike for mappings that are not writable because the fault did
not request write premission, it is not not necessary to track
executability for these SPTEs; the gfn is always available and it will
be possible to access the attributes directly in FNAME(sync_spte).

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-10-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/mmu/mmu.c          | 21 ++++++++++++---------
 arch/x86/kvm/mmu/mmu_internal.h |  2 +-
 arch/x86/kvm/mmu/paging_tmpl.h  |  8 +++++---
 arch/x86/kvm/mmu/spte.c         | 12 ++++++------
 arch/x86/kvm/mmu/spte.h         |  2 +-
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 6 files changed, 26 insertions(+), 21 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 38a4583d1ec2..5be24b67f212 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -3158,7 +3158,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot,
 	u64 spte;
 
 	/* Prefetching always gets a writable pfn.  */
-	bool host_writable = !fault || fault->map_writable;
+	unsigned host_access = fault ? fault->host_access : ACC_ALL;
 	bool prefetch = !fault || fault->prefetch;
 	bool write_fault = fault && fault->write;
 
@@ -3195,7 +3195,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot,
 	}
 
 	wrprot = make_spte(vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch,
-			   false, host_writable, &spte);
+			   false, host_access, &spte);
 
 	if (*sptep == spte) {
 		ret = RET_PF_SPURIOUS;
@@ -3643,7 +3643,7 @@ static int kvm_handle_noslot_fault(struct kvm_vcpu *vcpu,
 
 	fault->slot = NULL;
 	fault->pfn = KVM_PFN_NOSLOT;
-	fault->map_writable = false;
+	fault->host_access = 0;
 
 	/*
 	 * If MMIO caching is disabled, emulate immediately without
@@ -4668,7 +4668,8 @@ static void kvm_mmu_finish_page_fault(struct kvm_vcpu *vcpu,
 				      struct kvm_page_fault *fault, int r)
 {
 	kvm_release_faultin_page(vcpu->kvm, fault->refcounted_page,
-				 r == RET_PF_RETRY, fault->map_writable);
+				 r == RET_PF_RETRY,
+				 !!(fault->host_access & ACC_WRITE_MASK));
 }
 
 static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
@@ -4688,9 +4689,10 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 		return r;
 	}
 
-	fault->map_writable &= !(fault->slot->flags & KVM_MEM_READONLY);
-	fault->max_level = kvm_max_level_for_order(max_order);
+	if (fault->slot->flags & KVM_MEM_READONLY)
+		fault->host_access &= ~ACC_WRITE_MASK;
 
+	fault->max_level = kvm_max_level_for_order(max_order);
 	return RET_PF_CONTINUE;
 }
 
@@ -4738,7 +4740,8 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 				       &writable, &fault->refcounted_page);
 
 out_pf_continue:
-	fault->map_writable &= writable;
+	if (!writable)
+		fault->host_access &= ~ACC_WRITE_MASK;
 	return RET_PF_CONTINUE;
 }
 
@@ -5057,7 +5060,7 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.is_private = err & PFERR_PRIVATE_ACCESS,
 
 		.pfn = KVM_PFN_ERR_FAULT,
-		.map_writable = true,
+		.host_access = ACC_ALL,
 	};
 	int r;
 
@@ -5252,7 +5255,7 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
 		.gfn = gfn,
 		.slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn),
 		.pfn = pfn,
-		.map_writable = true,
+		.host_access = ACC_ALL,
 	};
 	struct kvm *kvm = vcpu->kvm;
 	int r;
diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h
index c29002c60126..00215b9f309f 100644
--- a/arch/x86/kvm/mmu/mmu_internal.h
+++ b/arch/x86/kvm/mmu/mmu_internal.h
@@ -280,7 +280,7 @@ struct kvm_page_fault {
 	unsigned long mmu_seq;
 	kvm_pfn_t pfn;
 	struct page *refcounted_page;
-	bool map_writable;
+	u8 host_access;
 
 	/*
 	 * Indicates the guest is trying to write a gfn that contains one or
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index 8e350095508c..533ad280accc 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -938,7 +938,7 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
  */
 static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int i)
 {
-	bool host_writable;
+	u8 host_access;
 	gpa_t first_pte_gpa;
 	u64 *sptep, spte;
 	struct kvm_memory_slot *slot;
@@ -995,11 +995,13 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int
 
 	sptep = &sp->spt[i];
 	spte = *sptep;
-	host_writable = spte & shadow_host_writable_mask;
+	host_access = ACC_ALL;
+	if (!(spte & shadow_host_writable_mask))
+		host_access &= ~ACC_WRITE_MASK;
 	slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 	make_spte(vcpu, sp, slot, pte_access, gfn,
 		  spte_to_pfn(spte), spte, true, true,
-		  host_writable, &spte);
+		  host_access, &spte);
 
 	/*
 	 * There is no need to mark the pfn dirty, as the new protections must
diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c
index 5fc27e9733b3..1434164fa372 100644
--- a/arch/x86/kvm/mmu/spte.c
+++ b/arch/x86/kvm/mmu/spte.c
@@ -189,7 +189,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	       const struct kvm_memory_slot *slot,
 	       unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn,
 	       u64 old_spte, bool prefetch, bool synchronizing,
-	       bool host_writable, u64 *new_spte)
+	       unsigned int host_access, u64 *new_spte)
 {
 	int level = sp->role.level;
 	u64 spte = SPTE_MMU_PRESENT_MASK;
@@ -207,6 +207,11 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	if (!prefetch || synchronizing)
 		spte |= shadow_accessed_mask;
 
+	if (host_access & ACC_WRITE_MASK)
+		spte |= shadow_host_writable_mask;
+
+	pte_access &= host_access;
+
 	/*
 	 * For simplicity, enforce the NX huge page mitigation even if not
 	 * strictly necessary.  KVM could ignore the mitigation if paging is
@@ -246,11 +251,6 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	if (kvm_x86_ops.get_mt_mask)
 		spte |= kvm_x86_call(get_mt_mask)(vcpu, gfn,
 						  kvm_is_mmio_pfn(pfn, &is_host_mmio));
-	if (host_writable)
-		spte |= shadow_host_writable_mask;
-	else
-		pte_access &= ~ACC_WRITE_MASK;
-
 	if (shadow_me_value && !kvm_is_mmio_pfn(pfn, &is_host_mmio))
 		spte |= shadow_me_value;
 
diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h
index e730717824b3..589f3954633e 100644
--- a/arch/x86/kvm/mmu/spte.h
+++ b/arch/x86/kvm/mmu/spte.h
@@ -563,7 +563,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	       const struct kvm_memory_slot *slot,
 	       unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn,
 	       u64 old_spte, bool prefetch, bool synchronizing,
-	       bool host_writable, u64 *new_spte);
+	       unsigned int host_access, u64 *new_spte);
 u64 make_small_spte(struct kvm *kvm, u64 huge_spte,
 		    union kvm_mmu_page_role role, int index);
 u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level);
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 44dad106fad1..dce44b9ce73a 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1143,7 +1143,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,
 	else
 		wrprot = make_spte(vcpu, sp, fault->slot, sp->role.access, iter->gfn,
 				   fault->pfn, iter->old_spte, fault->prefetch,
-				   false, fault->map_writable, &new_spte);
+				   false, fault->host_access, &new_spte);
 
 	if (new_spte == iter->old_spte)
 		ret = RET_PF_SPURIOUS;
-- 
2.52.0

[PATCH 14/45] KVM: x86/mmu: Init memslot hugepage information for non-private_mem VMs too

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

The list of supported memory attributes is about to grow, and they will
be available on any VM type (as opposed to only ones targeted at
confidential computing). As such, update the check in
kvm_mmu_init_memslot_memory_attributes() to initialize huge page
information if any kind of memory attributes is available for the VM.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-11-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/mmu/mmu.c   | 3 ++-
 include/linux/kvm_host.h | 6 ++++++
 virt/kvm/kvm_main.c      | 2 +-
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 5be24b67f212..3755393de4ca 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8361,7 +8361,8 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
 {
 	int level;
 
-	if (!kvm_arch_has_private_mem(kvm))
+	if (!kvm_arch_has_private_mem(kvm) &&
+	    !kvm_supported_vm_mem_attributes(kvm))
 		return;
 
 	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 5e386c3d2495..3d0293ebc864 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2589,6 +2589,12 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
 					   struct kvm_gfn_range *range);
 bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
 					    struct kvm_gfn_range *range);
+u64 kvm_supported_vm_mem_attributes(struct kvm *kvm);
+#else
+static inline u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
+{
+	return 0;
+}
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 #ifdef CONFIG_KVM_PRIVATE_MEMATTR_VM
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index cf4cb586f140..2fd595a901bd 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2474,7 +2474,7 @@ static u64 kvm_supports_private_mem(struct kvm *kvm)
 #endif
 
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-static u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
+u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
 	if (gmem_in_place_conversion || !kvm_supports_private_mem(kvm))
 		return 0;
-- 
2.52.0

[PATCH 15/45] KVM: VMX: return error from kvm_set_cr*

[Paolo Bonzini]

This is already done by svm.c, copy it over to vmx.c so
that kvm_complete_insn_gp() can return -EFAULT on PDPTR
loads.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/vmx/vmx.c | 12 ++++++++----
 arch/x86/kvm/x86.c     |  2 ++
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index b341af4c6057..fe43718b751b 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5630,6 +5630,7 @@ static int handle_set_cr0(struct kvm_vcpu *vcpu, unsigned long val)
 	if (is_guest_mode(vcpu)) {
 		struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 		unsigned long orig_val = val;
+		int r;
 
 		/*
 		 * We get here when L2 changed cr0 in a way that did not change
@@ -5642,8 +5643,9 @@ static int handle_set_cr0(struct kvm_vcpu *vcpu, unsigned long val)
 		val = (val & ~vmcs12->cr0_guest_host_mask) |
 			(vmcs12->guest_cr0 & vmcs12->cr0_guest_host_mask);
 
-		if (kvm_set_cr0(vcpu, val))
-			return 1;
+		r = kvm_set_cr0(vcpu, val);
+		if (r)
+			return r;
 		vmcs_writel(CR0_READ_SHADOW, orig_val);
 		return 0;
 	} else {
@@ -5656,12 +5658,14 @@ static int handle_set_cr4(struct kvm_vcpu *vcpu, unsigned long val)
 	if (is_guest_mode(vcpu)) {
 		struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 		unsigned long orig_val = val;
+		int r;
 
 		/* analogously to handle_set_cr0 */
 		val = (val & ~vmcs12->cr4_guest_host_mask) |
 			(vmcs12->guest_cr4 & vmcs12->cr4_guest_host_mask);
-		if (kvm_set_cr4(vcpu, val))
-			return 1;
+		r = kvm_set_cr4(vcpu, val);
+		if (r)
+			return r;
 		vmcs_writel(CR4_READ_SHADOW, orig_val);
 		return 0;
 	} else
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 1fcef4a8ac22..c7970bcfdc24 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -580,6 +580,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_requeue_exception);
 
 int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err)
 {
+	if (err < 0)
+		return err;
 	if (err)
 		kvm_inject_gp(vcpu, 0);
 	else
-- 
2.52.0

[PATCH 16/45] KVM: adjust for presence of more than one attribute

[Paolo Bonzini]

Remove or adjust code that assumed that the only valid attribute was
KVM_MEMORY_ATTRIBUTE_PRIVATE, in preparation for the introduction of RWX
memory attributes.

In particular, the optimization in kvm_range_has_memory_attributes
simply is not valid if attrs == 0 but some of the attributes are being
ignored.  The WARNs in arch/x86/kvm/mmu/mmu.c are just a duplicate of
checking kvm_supported_mem_attributes() during the ioctl, so drop it.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-13-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/mmu/mmu.c |  9 +--------
 virt/kvm/kvm_main.c    | 10 +++++++---
 2 files changed, 8 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 3755393de4ca..b89517794034 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8226,9 +8226,6 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
 	 * Zapping SPTEs in this case ensures KVM will reassess whether or not
 	 * a hugepage can be used for affected ranges.
 	 */
-	if (WARN_ON_ONCE(!kvm_arch_has_private_mem(kvm)))
-		return false;
-
 	if (WARN_ON_ONCE(range->end <= range->start))
 		return false;
 
@@ -8304,11 +8301,7 @@ bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
 	 * can't map memory PRIVATE.  KVM mustn't create a SHARED hugepage over
 	 * a range that has PRIVATE GFNs, and conversely converting a range to
 	 * SHARED may now allow hugepages.
-	 */
-	if (WARN_ON_ONCE(!kvm_arch_has_private_mem(kvm)))
-		return false;
-
-	/*
+	 *
 	 * The sequence matters here: upper levels consume the result of lower
 	 * level's scanning.
 	 */
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 2fd595a901bd..17341c689d11 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2490,10 +2490,12 @@ bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 					unsigned long mask, unsigned long attrs)
 {
 	XA_STATE(xas, &kvm->mem_attr_array, start);
+	unsigned long supported_attrs;
 	unsigned long index;
 	void *entry;
 
-	mask &= kvm_supported_vm_mem_attributes(kvm);
+	supported_attrs = kvm_supported_vm_mem_attributes(kvm);
+	mask &= supported_attrs;
 	if (attrs & ~mask)
 		return false;
 
@@ -2522,8 +2524,10 @@ bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	 */
 	for (index = start; index < end; index++) {
 		do {
-			entry = attrs ? xas_next(&xas) :
-					xas_next_entry(&xas, end - 1);
+			if (!attrs && mask == supported_attrs)
+				entry = xas_next_entry(&xas, end - 1);
+			else
+				entry = xas_next(&xas);
 		} while (xas_retry(&xas, entry));
 
 		if (!entry)
-- 
2.52.0

[PATCH 17/45] KVM: Introduce NR/NW/NX memory attributes

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Introduce memory attributes to map guest physical memory regions as
non-readable, non-writable, and/or non-executable. Only a subset of flag
combinations is supported. Notably write and exec permissions require
read permission, and memory protection attributes are incompatible with
private memory.

As mentioned in 5a475554db1e ("KVM: Introduce per-page memory attributes",
2023-11-13), bits 0-2 of the memory attributes were reserved for RWX
protection; they are negated to support current memory attribute users
which use 0 to indicate no special treatment.  Since 0 is not available,
a non-negated version of the flags would need an extra bit to express
no-access (R=0/W=0/X=0) mappings.

Unfortunately this precaution did not age too well; KVM now supports
MBEC/GMET and adding mode-based memory protections will require a
non-contiguous bit.  But that's something left for later.

Different architectures may have different limitations on the set of
valid protections, for example execution-only and XU=0 mappings are
supported by Intel but not AMD processors[1].  So, add an architecture-specific
callback and add a basic implementation for x86.

[1] When adding support for MBEC/GMET, since NX would remain to mean no
    execution at all, it is possible to use either an NXS bit or two
    separate NXS/NXU bits in addition to NX.  The former would only
    support permissions that are available with either MBEC or GMET.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-14-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst  | 29 +++++++++++++--
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/Kconfig            |  2 +-
 arch/x86/kvm/mmu/mmu.c          | 63 ++++++++++++++++++++++++---------
 arch/x86/kvm/x86.c              |  2 --
 include/linux/kvm_host.h        | 28 +++++++++++++++
 include/uapi/linux/kvm.h        |  3 ++
 virt/kvm/kvm_main.c             | 46 ++++++++++++++++++++----
 8 files changed, 145 insertions(+), 29 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index d0f91687bdb9..ac8cd51bdada 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6445,15 +6445,38 @@ of guest physical memory.
 	__u64 flags;
   };
 
+  #define KVM_MEMORY_ATTRIBUTE_NR                (1ULL << 0)
+  #define KVM_MEMORY_ATTRIBUTE_NW                (1ULL << 1)
+  #define KVM_MEMORY_ATTRIBUTE_NX                (1ULL << 2)
   #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
 
 The address and size must be page aligned.  The supported attributes can be
 retrieved via ioctl(KVM_CHECK_EXTENSION) on KVM_CAP_MEMORY_ATTRIBUTES.  If
 executed on a VM, KVM_CAP_MEMORY_ATTRIBUTES precisely returns the attributes
 supported by that VM.  If executed at system scope, KVM_CAP_MEMORY_ATTRIBUTES
-returns all attributes supported by KVM.  The only attribute defined at this
-time is KVM_MEMORY_ATTRIBUTE_PRIVATE, which marks the associated gfn as being
-guest private memory.
+returns all attributes supported by KVM.  The attribute defined at this
+time are:
+
+ - KVM_MEMORY_ATTRIBUTE_NR/NW/NX - Respectively marks the memory region as
+   non-read, non-write and/or non-exec.  Note that write-only, exec-only
+   and write-exec mappings are not supported.  When accesses to page are
+   denied by memory attributes, KVM **can** inform userspace through a
+   KVM_EXIT_MEMORY_FAULT exit, and userspace is allowed to restart the
+   VM after fixing the issue.
+
+   Note however that while KVM tries to report failures as
+   KVM_EXIT_MEMORY_FAULT in as many cases as possible, in some cases such
+   accesses are not recoverable.  KVM can report these accesses in the same
+   way as accesses to unmapped regions, for example as KVM_EXIT_SHUTDOWN,
+   KVM_EXIT_INTERNAL_ERROR, inject exceptions, or even by returning -EIO.
+   For exit reasons other than KVM_EXIT_MEMORY_FAULT, resetting the
+   attribute does not guarantee that execution can safely resume.
+
+   These attributes must therefore not be used as a replacement for
+   userfaultfd.
+
+ - KVM_MEMORY_ATTRIBUTE_PRIVATE - Which marks the associated gfn as being guest
+   private memory.
 
 Note, there is no "get" API.  Userspace is responsible for explicitly tracking
 the state of a gfn/page as needed.
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 25256d59cfde..6a35fd384a74 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1882,6 +1882,7 @@ enum kvm_intr_type {
 #define kvm_arch_has_gmem_convert() (!!kvm_x86_ops.gmem_make_private)
 #endif
 
+#define kvm_arch_has_memory_protection_attr(kvm) (!(kvm)->arch.has_private_mem)
 #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state)
 
 #ifdef CONFIG_X86_64
diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index 1719059570fb..3cde8bc5ff01 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -44,6 +44,7 @@ config KVM_X86
 	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select KVM_WERROR if WERROR
 	select KVM_GUEST_MEMFD if X86_64
+	select KVM_VM_MEMORY_ATTRIBUTES
 
 config KVM
 	tristate "Kernel-based Virtual Machine (KVM) support"
@@ -79,7 +80,6 @@ config KVM_WERROR
 config KVM_PRIVATE_MEMATTR_VM
 	bool "Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)"
 	depends on KVM_SW_PROTECTED_VM || KVM_INTEL_TDX || KVM_AMD_SEV
-	select KVM_VM_MEMORY_ATTRIBUTES
 	default KVM_INTEL_TDX || KVM_AMD_SEV
 	help
 	  Enable support for tracking PRIVATE vs. SHARED memory using per-VM
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b89517794034..6348669bf882 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8190,7 +8190,6 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 		vhost_task_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
 
-#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static bool hugepage_test_mixed(struct kvm_memory_slot *slot, gfn_t gfn,
 				int level)
 {
@@ -8213,19 +8212,27 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
 					   struct kvm_gfn_range *range)
 {
 	struct kvm_memory_slot *slot = range->slot;
+	bool flush = false;
 	int level;
 
 	/*
-	 * Zap SPTEs even if the slot can't be mapped PRIVATE.  KVM x86 only
-	 * supports KVM_MEMORY_ATTRIBUTE_PRIVATE, and so it *seems* like KVM
-	 * can simply ignore such slots.  But if userspace is making memory
-	 * PRIVATE, then KVM must prevent the guest from accessing the memory
-	 * as shared.  And if userspace is making memory SHARED and this point
-	 * is reached, then at least one page within the range was previously
-	 * PRIVATE, i.e. the slot's possible hugepage ranges are changing.
-	 * Zapping SPTEs in this case ensures KVM will reassess whether or not
-	 * a hugepage can be used for affected ranges.
+	 * For KVM_MEMORY_ATTRIBUTE_PRIVATE:
+	 *  Zap SPTEs even if the slot can't be mapped PRIVATE.  KVM x86 only
+	 *  supports KVM_MEMORY_ATTRIBUTE_PRIVATE, and so it *seems* like KVM
+	 *  can simply ignore such slots.  But if userspace is making memory
+	 *  PRIVATE, then KVM must prevent the guest from accessing the memory
+	 *  as shared.  And if userspace is making memory SHARED and this point
+	 *  is reached, then at least one page within the range was previously
+	 *  PRIVATE, i.e. the slot's possible hugepage ranges are changing.
+	 *  Zapping SPTEs in this case ensures KVM will reassess whether or not
+	 *  a hugepage can be used for affected ranges.
+	 *
+	 * For KVM_MEMORY_ATTRIBUTE_NR/NW/NX:
+	 *  Zap even when loosening restrictions R=>RW, which is not strictly
+	 *  necessary, but will allow KVM to reasses whether a hugepage can be
+	 *  used for the affected pages.
 	 */
+
 	if (WARN_ON_ONCE(range->end <= range->start))
 		return false;
 
@@ -8259,12 +8266,17 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
 	}
 
 	/* Unmap the old attribute page. */
-	if (range->arg.attributes & KVM_MEMORY_ATTRIBUTE_PRIVATE)
-		range->attr_filter = KVM_FILTER_SHARED;
-	else
-		range->attr_filter = KVM_FILTER_PRIVATE;
+	range->attr_filter = KVM_FILTER_SHARED;
+	if ((kvm_supported_vm_mem_attributes(kvm) & ~KVM_MEMORY_ATTRIBUTE_PRIVATE) ||
+	    (range->arg.attributes & KVM_MEMORY_ATTRIBUTE_PRIVATE))
+		flush |= kvm_unmap_gfn_range(kvm, range);
 
-	return kvm_unmap_gfn_range(kvm, range);
+	range->attr_filter = KVM_FILTER_PRIVATE;
+	if (kvm->arch.has_private_mem &&
+	    !(range->arg.attributes & KVM_MEMORY_ATTRIBUTE_PRIVATE))
+		flush |= kvm_unmap_gfn_range(kvm, range);
+
+	return flush;
 }
 
 
@@ -8386,4 +8398,23 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
 		}
 	}
 }
-#endif
+
+/* The bits are flipped but remain in the same position.  */
+#define KVM_PROT_READ  KVM_MEMORY_ATTRIBUTE_NR
+#define KVM_PROT_WRITE KVM_MEMORY_ATTRIBUTE_NW
+#define KVM_PROT_EXEC  KVM_MEMORY_ATTRIBUTE_NX
+
+bool kvm_arch_mem_attributes_supported_prot(struct kvm *kvm, unsigned long attrs)
+{
+	unsigned long prot = (attrs & KVM_MEMORY_ATTRIBUTE_PROT) ^ KVM_MEMORY_ATTRIBUTE_PROT;
+
+        /* Private memory and access permissions are incompatible */
+	if (attrs & KVM_MEMORY_ATTRIBUTE_PRIVATE)
+		return false;
+
+	/* For now do now support exec-only, even though EPT can handle it.  */
+	if (prot && !(prot & KVM_PROT_READ))
+		return false;
+
+	return true;
+}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index c7970bcfdc24..63d48ac4e1c6 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10263,9 +10263,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		}
 	}
 
-#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	kvm_mmu_init_memslot_memory_attributes(kvm, slot);
-#endif
 
 	if (kvm_page_track_create_memslot(kvm, slot, npages))
 		goto out_free;
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 3d0293ebc864..8442fb29a1ee 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -726,6 +726,15 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
 }
 #endif
 
+#ifdef kvm_arch_has_memory_protection_attr
+bool kvm_arch_mem_attributes_supported_prot(struct kvm *kvm, unsigned long attrs);
+#else
+static inline bool kvm_arch_has_memory_protection_attr(struct kvm *kvm)
+{
+	return false;
+}
+#endif
+
 #ifndef kvm_arch_has_readonly_mem
 static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm)
 {
@@ -2577,12 +2586,31 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot)
 	return slot->flags & KVM_MEMSLOT_GMEM_ONLY;
 }
 
+#define KVM_MEMORY_ATTRIBUTE_PROT \
+	(KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW | KVM_MEMORY_ATTRIBUTE_NX)
+
+static inline bool kvm_mem_attributes_may_read(u64 attrs)
+{
+	return !(attrs & KVM_MEMORY_ATTRIBUTE_NR);
+}
+
+static inline bool kvm_mem_attributes_may_write(u64 attrs)
+{
+	return !(attrs & KVM_MEMORY_ATTRIBUTE_NW);
+}
+
+static inline bool kvm_mem_attributes_may_exec(u64 attrs)
+{
+	return !(attrs & KVM_MEMORY_ATTRIBUTE_NX);
+}
+
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static inline unsigned long kvm_get_vm_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
 	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
 }
 
+bool kvm_vm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs);
 bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 					unsigned long mask, unsigned long attrs);
 bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 4e0bab3db95c..5cced8e4aa21 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1673,6 +1673,9 @@ struct kvm_memory_attributes2 {
 	__u64 reserved[11];
 };
 
+#define KVM_MEMORY_ATTRIBUTE_NR		       (1ULL << 0)
+#define KVM_MEMORY_ATTRIBUTE_NW		       (1ULL << 1)
+#define KVM_MEMORY_ATTRIBUTE_NX		       (1ULL << 2)
 #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
 
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 17341c689d11..717d7aec9a6c 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2473,13 +2473,28 @@ static u64 kvm_supports_private_mem(struct kvm *kvm)
 #define kvm_supports_private_mem(kvm) false
 #endif
 
+
+#ifdef kvm_arch_has_memory_protection_attr
+static u64 kvm_supports_memory_protection_attr(struct kvm *kvm)
+{
+	return !kvm || kvm_arch_has_memory_protection_attr(kvm);
+}
+#else
+#define kvm_supports_private_mem(kvm) false
+#endif
+
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
-	if (gmem_in_place_conversion || !kvm_supports_private_mem(kvm))
-		return 0;
+	u64 supported_attrs = 0;
 
-	return KVM_MEMORY_ATTRIBUTE_PRIVATE;
+	if (kvm_supports_memory_protection_attr(kvm))
+		supported_attrs |= KVM_MEMORY_ATTRIBUTE_PROT;
+
+	if (!gmem_in_place_conversion && kvm_supports_private_mem(kvm))
+		supported_attrs |= KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	return supported_attrs;
 }
 
 /*
@@ -2600,9 +2615,7 @@ static bool kvm_pre_set_vm_memory_attributes(struct kvm *kvm,
 {
 	/*
 	 * Unconditionally add the range to the invalidation set, regardless of
-	 * whether or not the arch callback actually needs to zap SPTEs.  E.g.
-	 * if KVM supports RWX attributes in the future and the attributes are
-	 * going from R=>RW, zapping isn't strictly necessary.  Unconditionally
+	 * whether or not the arch callback actually needs to zap SPTEs.  Always
 	 * adding the range allows KVM to require that MMU invalidations add at
 	 * least one range between begin() and end(), e.g. allows KVM to detect
 	 * bugs where the add() is missed.  Relaxing the rule *might* be safe,
@@ -2678,6 +2691,25 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 
 	return r;
 }
+
+bool __weak kvm_arch_mem_attributes_supported_prot(struct kvm *kvm, unsigned long attrs)
+{
+	WARN_ONCE(true, "KVM_MEMORY_ATTRIBUTE_PROT requires kvm_arch_mem_attributes_supported_prot()");
+	return false;
+}
+
+bool kvm_vm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs)
+{
+	if (attrs & ~kvm_supported_vm_mem_attributes(kvm))
+		return false;
+
+	if ((attrs & KVM_MEMORY_ATTRIBUTE_PROT) &&
+	    !kvm_arch_mem_attributes_supported_prot(kvm, attrs))
+		return false;
+
+	return true;
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
@@ -2686,7 +2718,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	/* flags is currently not used. */
 	if (attrs->flags)
 		return -EINVAL;
-	if (attrs->attributes & ~kvm_supported_vm_mem_attributes(kvm))
+	if (!kvm_vm_mem_attributes_valid(kvm, attrs->attributes))
 		return -EINVAL;
 	if (attrs->size == 0 || attrs->address + attrs->size < attrs->address)
 		return -EINVAL;
-- 
2.52.0

[PATCH 18/45] KVM: Include memory protections in result of gfn->hva conversion

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

All paths that lead to guest memory accesses now need to check whether
memory attributes allow that access.  Users of gfn_to_hva,
kvm_vcpu_gfn_to_hva and their *_prot variant can get it more or
less for free via erroneous return values, so do this first.

Note however that this is not true of the variants that take a cached
kvm_memslots pointer.  These include caches (gfn-to-hva and gfn-to-pfn)
and page faults, both of which will need specific changes; but the more
optimized functions in virt/kvm/kvm_main.c such as kvm_read_guest()
and kvm_write_guest() also retrieve the memslot high in the call chain,
and therefore they will need changes in __kvm_read/write_guest_page().

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-15-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 include/linux/kvm_host.h | 19 ++++++++++++++
 virt/kvm/kvm_main.c      | 55 +++++++++++++++++++++++++++++++++++++---
 2 files changed, 70 insertions(+), 4 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 8442fb29a1ee..3791e4290742 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2623,6 +2623,11 @@ static inline u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
 	return 0;
 }
+
+static inline unsigned long kvm_get_vm_memory_attributes(struct kvm *kvm, gfn_t gfn)
+{
+	return 0;
+}
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 #ifdef CONFIG_KVM_PRIVATE_MEMATTR_VM
@@ -2653,6 +2658,20 @@ static inline bool kvm_is_private_gfn(struct kvm *kvm, gfn_t gfn)
 }
 #endif /* kvm_arch_has_private_mem */
 
+static inline int kvm_mem_attributes_may_read_gfn(struct kvm *kvm, gfn_t gfn)
+{
+	unsigned long attrs = kvm_get_vm_memory_attributes(kvm, gfn);
+
+	return kvm_mem_attributes_may_read(attrs);
+}
+
+static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
+{
+	unsigned long attrs = kvm_get_vm_memory_attributes(kvm, gfn);
+
+	return kvm_mem_attributes_may_write(attrs);
+}
+
 #ifdef CONFIG_KVM_GUEST_MEMFD
 bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn);
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 717d7aec9a6c..d3f074d7fd6f 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2874,13 +2874,38 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(gfn_to_hva_memslot);
 
 unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn)
 {
-	return gfn_to_hva_many(gfn_to_memslot(kvm, gfn), gfn, NULL);
+	unsigned long addr;
+
+	addr = gfn_to_hva_many(gfn_to_memslot(kvm, gfn), gfn, NULL);
+	if (kvm_is_error_hva(addr))
+		return addr;
+
+	if (!kvm_mem_attributes_may_read_gfn(kvm, gfn))
+		return KVM_HVA_ERR_BAD;
+
+	if (!kvm_mem_attributes_may_write_gfn(kvm, gfn))
+		return KVM_HVA_ERR_RO_BAD;
+
+	return addr;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(gfn_to_hva);
 
 unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn)
 {
-	return gfn_to_hva_many(kvm_vcpu_gfn_to_memslot(vcpu, gfn), gfn, NULL);
+	struct kvm *kvm = vcpu->kvm;
+	unsigned long addr;
+
+	addr = gfn_to_hva_many(kvm_vcpu_gfn_to_memslot(vcpu, gfn), gfn, NULL);
+	if (kvm_is_error_hva(addr))
+		return addr;
+
+	if (!kvm_mem_attributes_may_read_gfn(kvm, gfn))
+		return KVM_HVA_ERR_BAD;
+
+	if (!kvm_mem_attributes_may_write_gfn(kvm, gfn))
+		return KVM_HVA_ERR_RO_BAD;
+
+	return addr;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_gfn_to_hva);
 
@@ -2906,15 +2931,37 @@ unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot,
 unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable)
 {
 	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
+	unsigned long addr;
 
-	return gfn_to_hva_memslot_prot(slot, gfn, writable);
+	addr = gfn_to_hva_memslot_prot(slot, gfn, writable);
+	if (kvm_is_error_hva(addr))
+		return addr;
+
+	if (!kvm_mem_attributes_may_read_gfn(kvm, gfn))
+		return KVM_HVA_ERR_BAD;
+
+	if (writable && !kvm_mem_attributes_may_write_gfn(kvm, gfn))
+		*writable = false;
+
+	return addr;
 }
 
 unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable)
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
+	unsigned long addr;
 
-	return gfn_to_hva_memslot_prot(slot, gfn, writable);
+	addr = gfn_to_hva_memslot_prot(slot, gfn, writable);
+	if (kvm_is_error_hva(addr))
+		return addr;
+
+	if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gfn))
+		return KVM_HVA_ERR_BAD;
+
+	if (writable && !kvm_mem_attributes_may_write_gfn(vcpu->kvm, gfn))
+		*writable = false;
+
+	return addr;
 }
 
 static bool kvm_is_ad_tracked_page(struct page *page)
-- 
2.52.0

[PATCH 19/45] KVM: Introduce kvm_fetch_guest_page() and use it for x86

[Paolo Bonzini]

Introduce a separate function so that the emulator will be able to take
into account execution memory attributes rather than read.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-16-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/x86.c       | 4 ++--
 include/linux/kvm_host.h | 2 ++
 virt/kvm/kvm_main.c      | 9 +++++++++
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 63d48ac4e1c6..ab7a859dda79 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4916,8 +4916,8 @@ static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt,
 	offset = addr & (PAGE_SIZE-1);
 	if (WARN_ON(offset + bytes > PAGE_SIZE))
 		bytes = (unsigned)PAGE_SIZE - offset;
-	ret = kvm_vcpu_read_guest_page(vcpu, gpa >> PAGE_SHIFT, val,
-				       offset, bytes);
+	ret = kvm_vcpu_fetch_guest_page(vcpu, gpa >> PAGE_SHIFT, val,
+					offset, bytes);
 	if (unlikely(ret < 0))
 		return X86EMUL_IO_NEEDED;
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 3791e4290742..1974cd8d361f 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1456,6 +1456,8 @@ DEFINE_VCPU_MAP_CLASS(_readonly);
 
 unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn);
 unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable);
+int kvm_vcpu_fetch_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
+			      int offset, int len);
 int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, int offset,
 			     int len);
 int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa, void *data,
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index d3f074d7fd6f..41818ed5eb3a 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3398,6 +3398,15 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_page);
 
+int kvm_vcpu_fetch_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
+			      int offset, int len)
+{
+	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
+
+	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_fetch_guest_page);
+
 int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long len)
 {
 	gfn_t gfn = gpa >> PAGE_SHIFT;
-- 
2.52.0

[PATCH 20/45] KVM: Take memory protections into account for memory read/write/fetch

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Take into account memory attributes when accessing guest memory through
the kvm_{read,write,fetch}*() family of functions.

All of these pass a struct kvm_memory_slot pointer to the actual
workhorse functions, in order to share code between the VM-wide and
vCPU-specific version of the functions (the latter of which handles the
multi-address-space case).  For this reason they need specific changes
and do not work even though the gfn_to_hva() path has been taught already
about memory protection attributes.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-17-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 virt/kvm/kvm_main.c | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 41818ed5eb3a..de54f52f2158 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3362,11 +3362,13 @@ static int next_segment(unsigned long len, int offset)
 }
 
 /* Copy @len bytes from guest memory at '(@gfn * PAGE_SIZE) + @offset' to @data */
-static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
-				 void *data, int offset, int len)
+static int __kvm_read_guest_page(struct kvm *kvm, struct kvm_memory_slot *slot,
+				 gfn_t gfn, void *data, int offset, int len,
+				 unsigned long attr)
 {
 	int r;
 	unsigned long addr;
+	unsigned long attrs;
 
 	if (WARN_ON_ONCE(offset + len > PAGE_SIZE))
 		return -EFAULT;
@@ -3374,6 +3376,11 @@ static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
 	addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
+
+	attrs = kvm_get_vm_memory_attributes(kvm, gfn);
+	if (attrs & attr)
+		return -EFAULT;
+
 	r = __copy_from_user(data, (void __user *)addr + offset, len);
 	if (r)
 		return -EFAULT;
@@ -3385,7 +3392,8 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 {
 	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(kvm, slot, gfn, data, offset, len,
+				     KVM_MEMORY_ATTRIBUTE_NR);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_read_guest_page);
 
@@ -3394,7 +3402,8 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len,
+				     KVM_MEMORY_ATTRIBUTE_NR);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_page);
 
@@ -3403,7 +3412,8 @@ int kvm_vcpu_fetch_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len,
+				     KVM_MEMORY_ATTRIBUTE_NX);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_fetch_guest_page);
 
@@ -3447,8 +3457,9 @@ int kvm_vcpu_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa, void *data, unsigned l
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest);
 
-static int __kvm_read_guest_atomic(struct kvm_memory_slot *slot, gfn_t gfn,
-			           void *data, int offset, unsigned long len)
+static int __kvm_read_guest_atomic(struct kvm *kvm,
+				   struct kvm_memory_slot *slot, gfn_t gfn,
+				   void *data, int offset, unsigned long len)
 {
 	int r;
 	unsigned long addr;
@@ -3456,6 +3467,9 @@ static int __kvm_read_guest_atomic(struct kvm_memory_slot *slot, gfn_t gfn,
 	if (WARN_ON_ONCE(offset + len > PAGE_SIZE))
 		return -EFAULT;
 
+	if (!kvm_mem_attributes_may_read_gfn(kvm, gfn))
+		return -EFAULT;
+
 	addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
@@ -3474,7 +3488,7 @@ int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa,
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 	int offset = offset_in_page(gpa);
 
-	return __kvm_read_guest_atomic(slot, gfn, data, offset, len);
+	return __kvm_read_guest_atomic(vcpu->kvm, slot, gfn, data, offset, len);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_atomic);
 
@@ -3492,6 +3506,10 @@ static int __kvm_write_guest_page(struct kvm *kvm,
 	addr = gfn_to_hva_memslot(memslot, gfn);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
+
+	if (!kvm_mem_attributes_may_write_gfn(kvm, gfn))
+		return -EFAULT;
+
 	r = __copy_to_user((void __user *)addr + offset, data, len);
 	if (r)
 		return -EFAULT;
-- 
2.52.0

[PATCH 21/45] KVM: Take memory protections into account for __kvm_vcpu_map

[Paolo Bonzini]

Take into account memory attributes when establishing a long term map
through kvm_vcpu_map()/kvm_vcpu_map_readonly().

Unlike the gfn-to-pfn cache, changes to the attributes will not be
propagated, so that it will be possible to place a nested VMCS, VMCB,
or APIC access page on a non-readable/writable page.  This will be
fixed separately by using a gfn-to-pfn cache for nested virtualization
data structures, in the meanwhile apply a stopgap fix.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-18-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 virt/kvm/kvm_main.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index de54f52f2158..c7b8d1bcf893 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3302,6 +3302,11 @@ int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map,
 	if (WARN_ON_ONCE(map->hva))
 		kvm_vcpu_unmap(vcpu, map);
 
+	if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gfn))
+		return -EFAULT;
+	if (writable && !kvm_mem_attributes_may_write_gfn(vcpu->kvm, gfn))
+		return -EFAULT;
+
 	map->pinned_page = NULL;
 	map->page = NULL;
 	map->hva = NULL;
-- 
2.52.0

[PATCH 22/45] KVM: Encapsulate memattrs array into anonymous struct

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

The metadata surrounding memory attributes is about to grow, so
encapsulate the memory attributes array within an anonymous struct to
provide namespacing.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-19-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 include/linux/kvm_host.h |  8 +++++---
 virt/kvm/kvm_main.c      | 10 +++++-----
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 1974cd8d361f..4a1985fe2737 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -876,8 +876,10 @@ struct kvm {
 	struct notifier_block pm_notifier;
 #endif
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-	/* Protected by slots_lock (for writes) and RCU (for reads) */
-	struct xarray mem_attr_array;
+	struct {
+		/* Protected by slots_lock (for writes) and RCU (for reads) */
+		struct xarray array;
+	} mem_attrs;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
 };
@@ -2609,7 +2611,7 @@ static inline bool kvm_mem_attributes_may_exec(u64 attrs)
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static inline unsigned long kvm_get_vm_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
-	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
+	return xa_to_value(xa_load(&kvm->mem_attrs.array, gfn));
 }
 
 bool kvm_vm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index c7b8d1bcf893..635730b5930c 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1135,7 +1135,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 	rcuwait_init(&kvm->mn_memslots_update_rcuwait);
 	xa_init(&kvm->vcpu_array);
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-	xa_init_flags(&kvm->mem_attr_array, XA_FLAGS_ACCOUNT);
+	xa_init_flags(&kvm->mem_attrs.array, XA_FLAGS_ACCOUNT);
 #endif
 
 	INIT_LIST_HEAD(&kvm->gpc_list);
@@ -1320,7 +1320,7 @@ static void kvm_destroy_vm(struct kvm *kvm)
 	srcu_barrier(&kvm->srcu);
 	cleanup_srcu_struct(&kvm->srcu);
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
-	xa_destroy(&kvm->mem_attr_array);
+	xa_destroy(&kvm->mem_attrs.array);
 #endif
 	kvm_arch_free_vm(kvm);
 	preempt_notifier_dec();
@@ -2504,7 +2504,7 @@ u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 					unsigned long mask, unsigned long attrs)
 {
-	XA_STATE(xas, &kvm->mem_attr_array, start);
+	XA_STATE(xas, &kvm->mem_attrs.array, start);
 	unsigned long supported_attrs;
 	unsigned long index;
 	void *entry;
@@ -2668,7 +2668,7 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	 * allocates, so no reservations are needed when clearing.
 	 */
 	for (i = start; entry && i < end; i++) {
-		r = xa_reserve(&kvm->mem_attr_array, i, GFP_KERNEL_ACCOUNT);
+		r = xa_reserve(&kvm->mem_attrs.array, i, GFP_KERNEL_ACCOUNT);
 		if (r)
 			goto out_unlock;
 
@@ -2678,7 +2678,7 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	kvm_handle_gfn_range(kvm, &pre_set_range);
 
 	for (i = start; i < end; i++) {
-		r = xa_err(xa_store(&kvm->mem_attr_array, i, entry,
+		r = xa_err(xa_store(&kvm->mem_attrs.array, i, entry,
 				    GFP_KERNEL_ACCOUNT));
 		KVM_BUG_ON(r, kvm);
 		cond_resched();
-- 
2.52.0

[PATCH 23/45] KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()

[Paolo Bonzini]

In many cases, retrieving kvm_memslots is followed by a check on
the generation of the slots.  Introduce a helper function that
either does the check alone, or compounds it with returning the
struct kvm_memslots* to the caller.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-21-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/loongarch/kvm/vcpu.c | 12 ++++--------
 arch/x86/kvm/x86.c        | 10 ++--------
 include/linux/kvm_host.h  | 13 +++++++++++++
 virt/kvm/kvm_main.c       |  8 ++++----
 virt/kvm/pfncache.c       | 10 ++++------
 5 files changed, 27 insertions(+), 26 deletions(-)

diff --git a/arch/loongarch/kvm/vcpu.c b/arch/loongarch/kvm/vcpu.c
index 059a6d605de7..c3e71ef983bc 100644
--- a/arch/loongarch/kvm/vcpu.c
+++ b/arch/loongarch/kvm/vcpu.c
@@ -154,7 +154,6 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 	u32 version;
 	u64 steal;
 	gpa_t gpa;
-	struct kvm_memslots *slots;
 	struct kvm_steal_time __user *st;
 	struct gfn_to_hva_cache *ghc;
 
@@ -164,9 +163,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 		return;
 
 	gpa &= KVM_STEAL_PHYS_MASK;
-	slots = kvm_memslots(vcpu->kvm);
-	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
-		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
@@ -1803,7 +1801,6 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 {
 	gpa_t gpa;
 	struct gfn_to_hva_cache *ghc;
-	struct kvm_memslots *slots;
 	struct kvm_steal_time __user *st;
 
 	gpa = vcpu->arch.st.guest_addr;
@@ -1820,9 +1817,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
-	slots = kvm_memslots(vcpu->kvm);
-	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
-		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ab7a859dda79..8574d75df4dc 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2062,7 +2062,6 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 {
 	struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
 	struct kvm_steal_time __user *st;
-	struct kvm_memslots *slots;
 	gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
 	u64 steal;
 	u32 version;
@@ -2078,9 +2077,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	if (WARN_ON_ONCE(current->mm != vcpu->kvm->mm))
 		return;
 
-	slots = kvm_memslots(vcpu->kvm);
-
-	if (unlikely(slots->generation != ghc->generation ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		/* We rely on the fact that it fits in a single page. */
@@ -2630,7 +2627,6 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
 {
 	struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
 	struct kvm_steal_time __user *st;
-	struct kvm_memslots *slots;
 	static const u8 preempted = KVM_VCPU_PREEMPTED;
 	gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
 
@@ -2657,9 +2653,7 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
 	if (unlikely(current->mm != vcpu->kvm->mm))
 		return;
 
-	slots = kvm_memslots(vcpu->kvm);
-
-	if (unlikely(slots->generation != ghc->generation ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot))
 		return;
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 4a1985fe2737..5cbb50afdb94 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2676,6 +2676,19 @@ static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
 	return kvm_mem_attributes_may_write(attrs);
 }
 
+static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation, struct kvm_memslots **p_slots)
+{
+	struct kvm_memslots *slots = *p_slots = kvm_memslots(kvm);
+	return slots->generation == slots_generation;
+}
+
+static inline bool kvm_check_gen(struct kvm *kvm, u64 slots_generation)
+{
+	struct kvm_memslots *slots;
+	return kvm_memslots_check_gen(kvm, slots_generation, &slots);
+}
+
+
 #ifdef CONFIG_KVM_GUEST_MEMFD
 bool kvm_gmem_is_private_gfn(struct kvm *kvm, gfn_t gfn);
 bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 635730b5930c..da12f4d43c91 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3635,14 +3635,14 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 				  void *data, unsigned int offset,
 				  unsigned long len)
 {
-	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memslots *slots;
 	int r;
 	gpa_t gpa = ghc->gpa + offset;
 
 	if (WARN_ON_ONCE(len + offset > ghc->len))
 		return -EINVAL;
 
-	if (slots->generation != ghc->generation) {
+	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->generation, &slots))) {
 		if (__kvm_gfn_to_hva_cache_init(slots, ghc, ghc->gpa, ghc->len))
 			return -EFAULT;
 	}
@@ -3673,14 +3673,14 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 				 void *data, unsigned int offset,
 				 unsigned long len)
 {
-	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memslots *slots;
 	int r;
 	gpa_t gpa = ghc->gpa + offset;
 
 	if (WARN_ON_ONCE(len + offset > ghc->len))
 		return -EINVAL;
 
-	if (slots->generation != ghc->generation) {
+	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->generation, &slots))) {
 		if (__kvm_gfn_to_hva_cache_init(slots, ghc, ghc->gpa, ghc->len))
 			return -EFAULT;
 	}
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index 3659686b97c2..ac02f6a065cc 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -72,8 +72,6 @@ static bool kvm_gpc_is_valid_len(gpa_t gpa, unsigned long uhva,
 
 bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
 {
-	struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
-
 	if (!gpc->active)
 		return false;
 
@@ -81,7 +79,7 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
 	 * If the page was cached from a memslot, make sure the memslots have
 	 * not been re-configured.
 	 */
-	if (!kvm_is_error_gpa(gpc->gpa) && gpc->generation != slots->generation)
+	if (!kvm_is_error_gpa(gpc->gpa) && !kvm_check_gen(gpc->kvm, gpc->generation))
 		return false;
 
 	if (kvm_is_error_hva(gpc->uhva))
@@ -290,12 +288,12 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
 		if (gpc->uhva != old_uhva)
 			hva_change = true;
 	} else {
-		struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
+		struct kvm_memslots *slots;
 
 		page_offset = offset_in_page(gpa);
 
-		if (gpc->gpa != gpa || gpc->generation != slots->generation ||
-		    kvm_is_error_hva(gpc->uhva)) {
+		if (!kvm_memslots_check_gen(gpc->kvm, gpc->generation, &slots) ||
+		    gpc->gpa != gpa || kvm_is_error_hva(gpc->uhva)) {
 			gfn_t gfn = gpa_to_gfn(gpa);
 
 			gpc->gpa = gpa;
-- 
2.52.0

[PATCH 24/45] KVM: Introduce a generation number for memory attributes

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Introduce a generation number to track memory attribute modifications.
This will allow KVM components to invalidate any assumptions they might
have about guest's physical addresses and their permissions when memory
attributes change.

Like with memory slot updates, it's mandatory for components that access
guest memory based on cached information to do so within a KVM SRCU
read-side critical section, and that they validate the generation number
before accessing memory. This, in combination with the
synchronize_srcu() call within the memory attributes ioctl handler,
ensures the following:
- A memory attribute modification operation only returns after all
  users of outdated GPA data are done running.
- Any component accessing cached data after the memory attribute
  modification returned will see the updated generation number.

Additionally, loads/stores of the generation number have
acquire/release semantics; which ensures all attribute writes are
visible before updating the generation, and loads from attributes happen
after having read the current generation number.

Ultimately, since synchronize_srcu_expedited() is an expensive
operation, only perform it when absolutely necessary. Do so if the
introduced memory attribute is known to require synchronization or if
the attribute being cleared contained a memory attribute that required
synchronization.

There shouldn't be any performance loss for memory attributes that
don't require synchronization (and in general for any VMM that does not
apply memory protections), because the attributes generation will always
remain unchanged.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-22-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/loongarch/kvm/vcpu.c  |  6 ++-
 arch/x86/kvm/x86.c         |  4 +-
 include/linux/kvm_host.h   | 33 +++++++++++++--
 include/linux/kvm_types.h  |  6 ++-
 include/trace/events/kvm.h | 14 +++++--
 virt/kvm/kvm_main.c        | 82 +++++++++++++++++++++++++++++++++-----
 virt/kvm/pfncache.c        | 13 +++---
 7 files changed, 129 insertions(+), 29 deletions(-)

diff --git a/arch/loongarch/kvm/vcpu.c b/arch/loongarch/kvm/vcpu.c
index c3e71ef983bc..6301e726a536 100644
--- a/arch/loongarch/kvm/vcpu.c
+++ b/arch/loongarch/kvm/vcpu.c
@@ -163,7 +163,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 		return;
 
 	gpa &= KVM_STEAL_PHYS_MASK;
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
@@ -1817,7 +1818,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8574d75df4dc..caff8534267d 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2077,7 +2077,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	if (WARN_ON_ONCE(current->mm != vcpu->kvm->mm))
 		return;
 
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		/* We rely on the fact that it fits in a single page. */
@@ -2653,7 +2653,7 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
 	if (unlikely(current->mm != vcpu->kvm->mm))
 		return;
 
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot))
 		return;
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 5cbb50afdb94..2c854aaab92a 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -879,6 +879,7 @@ struct kvm {
 	struct {
 		/* Protected by slots_lock (for writes) and RCU (for reads) */
 		struct xarray array;
+		u64 generation;
 	} mem_attrs;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
@@ -2621,7 +2622,23 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
 					   struct kvm_gfn_range *range);
 bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
 					    struct kvm_gfn_range *range);
+bool kvm_range_has_any_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
+					    unsigned long mask);
 u64 kvm_supported_vm_mem_attributes(struct kvm *kvm);
+
+static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
+{
+	RCU_LOCKDEP_WARN(!lockdep_is_held(&kvm->slots_lock) &&
+				 !srcu_read_lock_held(&kvm->srcu),
+			 "Suspicious memory attribute generation usage\n");
+
+	/*
+	 * The acquire pairs with the release in kvm_vm_set_mem_attributes().
+	 * Memory attributes should only be queried _after_ storing the
+	 * generation number.
+	 */
+	return smp_load_acquire(&kvm->mem_attrs.generation);
+}
 #else
 static inline u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
@@ -2632,6 +2649,11 @@ static inline unsigned long kvm_get_vm_memory_attributes(struct kvm *kvm, gfn_t
 {
 	return 0;
 }
+
+static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
+{
+	return 0;
+}
 #endif  /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
 #ifdef CONFIG_KVM_PRIVATE_MEMATTR_VM
@@ -2676,16 +2698,19 @@ static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
 	return kvm_mem_attributes_may_write(attrs);
 }
 
-static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation, struct kvm_memslots **p_slots)
+static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation,
+					  u64 attrs_generation, struct kvm_memslots **p_slots)
 {
 	struct kvm_memslots *slots = *p_slots = kvm_memslots(kvm);
-	return slots->generation == slots_generation;
+	return slots->generation == slots_generation &&
+		kvm_mem_attributes_generation(kvm) == attrs_generation;
 }
 
-static inline bool kvm_check_gen(struct kvm *kvm, u64 slots_generation)
+static inline bool kvm_check_gen(struct kvm *kvm, u64 slots_generation,
+				 u64 attrs_generation)
 {
 	struct kvm_memslots *slots;
-	return kvm_memslots_check_gen(kvm, slots_generation, &slots);
+	return kvm_memslots_check_gen(kvm, slots_generation, attrs_generation, &slots);
 }
 
 
diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h
index a568d8e6f4e8..7d911220e00d 100644
--- a/include/linux/kvm_types.h
+++ b/include/linux/kvm_types.h
@@ -74,7 +74,8 @@ typedef u64            hfn_t;
 typedef hfn_t kvm_pfn_t;
 
 struct gfn_to_hva_cache {
-	u64 generation;
+	u64 slots_generation;
+	u64 attrs_generation;
 	gpa_t gpa;
 	unsigned long hva;
 	unsigned long len;
@@ -82,7 +83,8 @@ struct gfn_to_hva_cache {
 };
 
 struct gfn_to_pfn_cache {
-	u64 generation;
+	u64 slots_generation;
+	u64 attrs_generation;
 	gpa_t gpa;
 	unsigned long uhva;
 	struct kvm_memory_slot *memslot;
diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h
index 07013f88e655..0c5cb91ff929 100644
--- a/include/trace/events/kvm.h
+++ b/include/trace/events/kvm.h
@@ -371,23 +371,29 @@ TRACE_EVENT(kvm_dirty_ring_exit,
  * @attr:	The value of the attribute being set.
  */
 TRACE_EVENT(kvm_set_vm_mem_attributes,
-	TP_PROTO(gfn_t start, gfn_t end, unsigned long attr),
-	TP_ARGS(start, end, attr),
+	TP_PROTO(gfn_t start, gfn_t end, unsigned long attr, bool sync, u64 generation),
+	TP_ARGS(start, end, attr, sync, generation),
 
 	TP_STRUCT__entry(
 		__field(gfn_t,		start)
 		__field(gfn_t,		end)
 		__field(unsigned long,	attr)
+		__field(bool,		sync)
+		__field(u64,		generation)
 	),
 
 	TP_fast_assign(
 		__entry->start		= start;
 		__entry->end		= end;
 		__entry->attr		= attr;
+		__entry->sync		= sync;
+		__entry->generation	= generation;
 	),
 
-	TP_printk("%#016llx -- %#016llx [0x%lx]",
-		  __entry->start, __entry->end, __entry->attr)
+	TP_printk("%#016llx -- %#016llx [0x%lx], sync %d gen %llu",
+		  __entry->start, __entry->end, __entry->attr,
+		  __entry->sync, __entry->generation)
+
 );
 #endif /* CONFIG_KVM_VM_MEMORY_ATTRIBUTES */
 
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index da12f4d43c91..6aa6a0a55b07 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1136,6 +1136,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 	xa_init(&kvm->vcpu_array);
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 	xa_init_flags(&kvm->mem_attrs.array, XA_FLAGS_ACCOUNT);
+	kvm->mem_attrs.generation = 0;
 #endif
 
 	INIT_LIST_HEAD(&kvm->gpc_list);
@@ -2558,6 +2559,53 @@ bool kvm_range_has_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	return true;
 }
 
+/*
+ * Returns true if _any_ gfns in the range [@start, @end) have attributes that
+ * match _any_ bit in @mask.
+ */
+bool kvm_range_has_any_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
+					unsigned long mask)
+{
+	XA_STATE(xas, &kvm->mem_attrs.array, start);
+	void *entry;
+
+	mask &= kvm_supported_vm_mem_attributes(kvm);
+	if (!mask)
+		return false;
+
+	if (end == start + 1)
+		return !!(kvm_get_vm_memory_attributes(kvm, start) & mask);
+
+	guard(rcu)();
+	for (;;) {
+		do {
+			entry = xas_next(&xas);
+		} while (xas_retry(&xas, entry));
+
+		if (xas.xa_node == XAS_BOUNDS || xas.xa_index >= end)
+			break;
+
+		if (xa_to_value(entry) & mask)
+			return true;
+	}
+
+	return false;
+}
+
+#define KVM_MEMORY_ATTRIBUTE_NEEDS_SYNC_MASK KVM_MEMORY_ATTRIBUTE_PROT
+
+static bool kvm_range_memory_attributes_need_sync(struct kvm *kvm,
+						  gfn_t start, gfn_t end,
+						  unsigned long attributes)
+{
+	u64 mask = KVM_MEMORY_ATTRIBUTE_NEEDS_SYNC_MASK;
+
+	if (attributes & mask)
+		return true;
+
+	return kvm_range_has_any_vm_memory_attributes(kvm, start, end, mask);
+}
+
 static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 						 struct kvm_mmu_notifier_range *range)
 {
@@ -2648,20 +2696,25 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		.on_lock = kvm_mmu_invalidate_end,
 		.may_block = true,
 	};
+	bool sync = false;
 	unsigned long i;
 	void *entry;
 	int r = 0;
 
 	entry = attributes ? xa_mk_value(attributes) : NULL;
 
-	trace_kvm_set_vm_mem_attributes(start, end, attributes);
-
 	mutex_lock(&kvm->slots_lock);
 
 	/* Nothing to do if the entire range has the desired attributes. */
 	if (kvm_range_has_vm_memory_attributes(kvm, start, end, ~0, attributes))
 		goto out_unlock;
 
+	sync = kvm_range_memory_attributes_need_sync(kvm, start, end,
+						     attributes);
+
+	trace_kvm_set_vm_mem_attributes(start, end, attributes, sync,
+					kvm->mem_attrs.generation + 1);
+
 	/*
 	 * Reserve memory ahead of time to avoid having to deal with failures
 	 * partway through setting the new attributes.  Storing NULL never
@@ -2684,10 +2737,16 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		cond_resched();
 	}
 
+	/* Pairs with acquire in kvm_mem_attributes_generation() */
+	smp_store_release(&kvm->mem_attrs.generation,
+			  kvm->mem_attrs.generation + 1);
+
 	kvm_handle_gfn_range(kvm, &post_set_range);
 
 out_unlock:
 	mutex_unlock(&kvm->slots_lock);
+	if (sync)
+		synchronize_srcu_expedited(&kvm->srcu);
 
 	return r;
 }
@@ -3582,7 +3641,8 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data,
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_write_guest);
 
-static int __kvm_gfn_to_hva_cache_init(struct kvm_memslots *slots,
+static int __kvm_gfn_to_hva_cache_init(struct kvm *kvm,
+				       struct kvm_memslots *slots,
 				       struct gfn_to_hva_cache *ghc,
 				       gpa_t gpa, unsigned long len)
 {
@@ -3592,8 +3652,8 @@ static int __kvm_gfn_to_hva_cache_init(struct kvm_memslots *slots,
 	gfn_t nr_pages_needed = end_gfn - start_gfn + 1;
 	gfn_t nr_pages_avail;
 
-	/* Update ghc->generation before performing any error checks. */
-	ghc->generation = slots->generation;
+	/* Update ghc->slots_generation before performing any error checks. */
+	ghc->slots_generation = slots->generation;
 
 	if (start_gfn > end_gfn) {
 		ghc->hva = KVM_HVA_ERR_BAD;
@@ -3612,6 +3672,8 @@ static int __kvm_gfn_to_hva_cache_init(struct kvm_memslots *slots,
 			return -EFAULT;
 	}
 
+	ghc->attrs_generation = kvm_mem_attributes_generation(kvm);
+
 	/* Use the slow path for cross page reads and writes. */
 	if (nr_pages_needed == 1)
 		ghc->hva += offset;
@@ -3627,7 +3689,7 @@ int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 			      gpa_t gpa, unsigned long len)
 {
 	struct kvm_memslots *slots = kvm_memslots(kvm);
-	return __kvm_gfn_to_hva_cache_init(slots, ghc, gpa, len);
+	return __kvm_gfn_to_hva_cache_init(kvm, slots, ghc, gpa, len);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gfn_to_hva_cache_init);
 
@@ -3642,8 +3704,8 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	if (WARN_ON_ONCE(len + offset > ghc->len))
 		return -EINVAL;
 
-	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->generation, &slots))) {
-		if (__kvm_gfn_to_hva_cache_init(slots, ghc, ghc->gpa, ghc->len))
+	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->slots_generation, ghc->attrs_generation, &slots))) {
+		if (__kvm_gfn_to_hva_cache_init(kvm, slots, ghc, ghc->gpa, ghc->len))
 			return -EFAULT;
 	}
 
@@ -3680,8 +3742,8 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	if (WARN_ON_ONCE(len + offset > ghc->len))
 		return -EINVAL;
 
-	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->generation, &slots))) {
-		if (__kvm_gfn_to_hva_cache_init(slots, ghc, ghc->gpa, ghc->len))
+	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->slots_generation, ghc->attrs_generation, &slots))) {
+		if (__kvm_gfn_to_hva_cache_init(kvm, slots, ghc, ghc->gpa, ghc->len))
 			return -EFAULT;
 	}
 
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index ac02f6a065cc..e82247ca4637 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -76,10 +76,11 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
 		return false;
 
 	/*
-	 * If the page was cached from a memslot, make sure the memslots have
-	 * not been re-configured.
+	 * If the page was cached from a memslot, make sure the memslots nor
+	 * memory attributes have not been re-configured.
 	 */
-	if (!kvm_is_error_gpa(gpc->gpa) && !kvm_check_gen(gpc->kvm, gpc->generation))
+	if (!kvm_is_error_gpa(gpc->gpa) &&
+	    !kvm_check_gen(gpc->kvm, gpc->slots_generation, gpc->attrs_generation))
 		return false;
 
 	if (kvm_is_error_hva(gpc->uhva))
@@ -253,6 +254,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
 
 static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long uhva)
 {
+	struct kvm *kvm = gpc->kvm;
 	unsigned long page_offset;
 	bool unmap_old = false;
 	unsigned long old_uhva;
@@ -292,12 +294,13 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
 
 		page_offset = offset_in_page(gpa);
 
-		if (!kvm_memslots_check_gen(gpc->kvm, gpc->generation, &slots) ||
+		if (!kvm_memslots_check_gen(gpc->kvm, gpc->slots_generation, gpc->attrs_generation, &slots) ||
 		    gpc->gpa != gpa || kvm_is_error_hva(gpc->uhva)) {
 			gfn_t gfn = gpa_to_gfn(gpa);
 
+			gpc->attrs_generation = kvm_mem_attributes_generation(kvm);
 			gpc->gpa = gpa;
-			gpc->generation = slots->generation;
+			gpc->slots_generation = slots->generation;
 			gpc->memslot = __gfn_to_memslot(slots, gfn);
 			gpc->uhva = gfn_to_hva_memslot(gpc->memslot, gfn);
 
-- 
2.52.0

[PATCH 25/45] KVM: Take memory protections into account for accesses with cached gfn->hva

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Account for memory attributes when accessing guest memory through
kvm_get/put_guest().

This requires tracking the memory attributes generation as part of
gfn_to_hva_cache's data, invalidate the cached information if the
generation changes, and failing to refresh the cache if restrictive
memory attributes are found within the GPA range.

Similar to how gfn_to_hva_cache disallows caching gfns mapped within
read-only memory slots, gfns marked as read-only by memory attributes
will also fail to initialize. Unsurprisingly, the same behaviour applies
to gfns mapped as non-accessible (NR/NW), while non-executable mappings
are okay.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-23-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 include/linux/kvm_host.h | 19 +++++++++++++++++--
 virt/kvm/kvm_main.c      | 15 +++++++++++----
 2 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 2c854aaab92a..de540af1fa24 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1371,7 +1371,8 @@ int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	typeof(v) __user *__uaddr = (typeof(__uaddr))(__addr + offset);	\
 	int __ret = -EFAULT;						\
 									\
-	if (!kvm_is_error_hva(__addr))					\
+	if (!kvm_is_error_hva(__addr) &&                                \
+	    kvm_mem_attributes_may_read_gfn(kvm, gfn))		\
 		__ret = get_user(v, __uaddr);				\
 	__ret;								\
 })
@@ -1391,7 +1392,8 @@ int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	typeof(v) __user *__uaddr = (typeof(__uaddr))(__addr + offset);	\
 	int __ret = -EFAULT;						\
 									\
-	if (!kvm_is_error_hva(__addr))					\
+	if (!kvm_is_error_hva(__addr) &&                                \
+	    kvm_mem_attributes_may_write_gfn(kvm, gfn))		\
 		__ret = put_user(v, __uaddr);				\
 	if (!__ret)							\
 		mark_page_dirty(kvm, gfn);				\
@@ -2640,6 +2642,12 @@ static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
 	return smp_load_acquire(&kvm->mem_attrs.generation);
 }
 #else
+static inline bool kvm_range_has_any_vm_memory_attributes(struct kvm *kvm,
+			gfn_t start, gfn_t end, unsigned long mask)
+{
+	return false;
+}
+
 static inline u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
 	return 0;
@@ -2698,6 +2706,13 @@ static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
 	return kvm_mem_attributes_may_write(attrs);
 }
 
+static inline bool kvm_range_has_rw_memory_protections(struct kvm *kvm,
+						       gfn_t start, gfn_t end)
+{
+	return kvm_range_has_any_vm_memory_attributes(kvm, start, end,
+			KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW);
+}
+
 static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation,
 					  u64 attrs_generation, struct kvm_memslots **p_slots)
 {
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 6aa6a0a55b07..8283b2415adc 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3664,15 +3664,22 @@ static int __kvm_gfn_to_hva_cache_init(struct kvm *kvm,
 	 * If the requested region crosses two memslots, we still
 	 * verify that the entire region is valid here.
 	 */
-	for ( ; start_gfn <= end_gfn; start_gfn += nr_pages_avail) {
-		ghc->memslot = __gfn_to_memslot(slots, start_gfn);
-		ghc->hva = gfn_to_hva_many(ghc->memslot, start_gfn,
-					   &nr_pages_avail);
+	for (gfn_t gfn = start_gfn ; gfn <= end_gfn; gfn += nr_pages_avail) {
+		ghc->memslot = __gfn_to_memslot(slots, gfn);
+		ghc->hva = gfn_to_hva_many(ghc->memslot, gfn, &nr_pages_avail);
 		if (kvm_is_error_hva(ghc->hva))
 			return -EFAULT;
 	}
 
+	/*
+	 * RW memory attributes are incompatible with GHC. The RW protection
+	 * check has to happen after storing the generation number.
+	 */
 	ghc->attrs_generation = kvm_mem_attributes_generation(kvm);
+	if (kvm_range_has_rw_memory_protections(kvm, start_gfn, end_gfn + 1)) {
+		ghc->hva = KVM_HVA_ERR_BAD;
+		return -EFAULT;
+	}
 
 	/* Use the slow path for cross page reads and writes. */
 	if (nr_pages_needed == 1)
-- 
2.52.0

[PATCH 26/45] KVM: pfncache: Fail to refresh if it contains memory protections

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Similar to how pfncache disallows caching gfns mapped within read-only
memory slots, gfns marked as read-only by memory attributes must
fail to initialize. Unsurprisingly, the same behaviour applies to gfns
mapped as non-accessible (NR/NW), while non-executable mappings are fine.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-24-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 virt/kvm/pfncache.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index e82247ca4637..ed8884a6d240 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -298,7 +298,18 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
 		    gpc->gpa != gpa || kvm_is_error_hva(gpc->uhva)) {
 			gfn_t gfn = gpa_to_gfn(gpa);
 
+			/*
+			 * RW memory attributes are incompatible with GPC. The
+			 * RW protection check has to happen after storing the
+			 * generation number.
+			 */
 			gpc->attrs_generation = kvm_mem_attributes_generation(kvm);
+			if (kvm_range_has_rw_memory_protections(kvm, gfn, gfn + 1)) {
+				gpc->uhva = KVM_HVA_ERR_BAD;
+				ret = -EFAULT;
+				goto out;
+			}
+
 			gpc->gpa = gpa;
 			gpc->slots_generation = slots->generation;
 			gpc->memslot = __gfn_to_memslot(slots, gfn);
-- 
2.52.0

[PATCH 27/45] KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory attributes

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Prefetched SPTEs are always given full access.  Do not prefetch GFNs
that have memory protections applied.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-25-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/mmu/mmu.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 6348669bf882..f689e4890e42 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -3240,6 +3240,11 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,
 		return false;
 
 	for (i = 0; i < nr_pages; i++, gfn++, sptep++) {
+		if (kvm_get_vm_memory_attributes(vcpu->kvm, gfn)) {
+			kvm_release_page_clean(pages[i]);
+			continue;
+		}
+
 		mmu_set_spte(vcpu, slot, sptep, access, gfn,
 			     page_to_pfn(pages[i]), NULL);
 
-- 
2.52.0

[PATCH 28/45] KVM: x86/mmu: Take memory protection attributes into account during faults

[Paolo Bonzini]

From: Nicolas Saenz Julienne <nsaenz@amazon.com>

Take memory protection attributes when faulting guest memory.  Prohibited
memory accesses will cause a user-space -EFAULT exit just like private
memory accesses.  Userspace will either enable the access or bump it to
the guest as some kind of exception (e.g. a VTL return).

Since the struct kvm_page_fault already has the access type in PFERR_*
format, the check is done via the kvm_page_format permissions table.
This means that it supports naturally all page table format variants,
and it can even handle mode-based memory protection when the host
uses MBEC/GMET.  The only thing that needs some care is to build the
restricted ACC_* mask with the root page's own access mask as a base
(and not ACC_ALL).  Otherwise, supervisor mode execution would be
handled incorrectly on AMD processors with GMET.

To avoid spamming the trace buffer too much, the new trace event only
kicks in if memory protection attributes are present for the faulted gfn.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-26-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/mmu/mmu.c          | 49 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/mmu/mmu_internal.h | 19 +++++++++++++
 arch/x86/kvm/mmu/mmutrace.h     | 36 ++++++++++++++++++++++++
 arch/x86/kvm/mmu/paging_tmpl.h  |  2 +-
 arch/x86/kvm/mmu/spte.h         | 11 ++------
 5 files changed, 107 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index f689e4890e42..787bc31acda3 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4701,6 +4701,50 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 	return RET_PF_CONTINUE;
 }
 
+static inline unsigned kvm_get_gfn_protections(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm *kvm = vcpu->kvm;
+	unsigned int access = vcpu->arch.mmu->root_role.access;
+	unsigned long attrs = kvm_get_vm_memory_attributes(kvm, gfn);
+	if (!attrs)
+		return access;
+
+	WARN_ON_ONCE(!kvm_vm_mem_attributes_valid(kvm, attrs));
+
+	if (!kvm_mem_attributes_may_read(attrs))
+		access &= ~ACC_READ_MASK;
+	if (!kvm_mem_attributes_may_write(attrs))
+		access &= ~ACC_WRITE_MASK;
+	if (!kvm_mem_attributes_may_exec(attrs)) {
+		access &= ~ACC_EXEC_MASK;
+		if (shadow_xu_mask)
+			access &= ~ACC_USER_EXEC_MASK;
+	}
+
+	return access;
+}
+
+static int kvm_faultin_memory_protections(struct kvm_vcpu *vcpu,
+					  struct kvm_page_fault *fault)
+{
+	unsigned access;
+
+	/* Memory attributes don't apply to MMIO regions */
+	if (unlikely(!fault->slot))
+		return RET_PF_CONTINUE;
+
+	access = kvm_get_gfn_protections(vcpu, fault->gfn);
+	if (access == ACC_ALL)
+		return RET_PF_CONTINUE;
+
+	trace_kvm_faultin_memory_protections(vcpu, fault, access);
+	if (__permission_fault(vcpu->arch.mmu, access, fault))
+		return -EFAULT;
+
+	fault->host_access &= access;
+	return RET_PF_CONTINUE;
+}
+
 static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 				 struct kvm_page_fault *fault)
 {
@@ -4781,6 +4825,11 @@ static int kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	if (unlikely(!slot))
 		return kvm_handle_noslot_fault(vcpu, fault, access);
 
+	if (kvm_faultin_memory_protections(vcpu, fault)) {
+		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
+		return -EFAULT;
+	}
+
 	/*
 	 * Retry the page fault if the gfn hit a memslot that is being deleted
 	 * or moved.  This ensures any existing SPTEs for the old memslot will
diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h
index 00215b9f309f..0bad60c5aab6 100644
--- a/arch/x86/kvm/mmu/mmu_internal.h
+++ b/arch/x86/kvm/mmu/mmu_internal.h
@@ -290,6 +290,25 @@ struct kvm_page_fault {
 	bool write_fault_to_shadow_pgtable;
 };
 
+/*
+ * Returns true if the access indicated by @fault is forbidden by the existing
+ * SPTE protections.
+ */
+static inline bool __permission_fault(struct kvm_mmu *mmu, unsigned access,
+				      struct kvm_page_fault *fault)
+{
+	unsigned pfec;
+
+	/*
+	 * RSVD is handled elsewhere, and is used for SMAP in the context
+	 * of accessing fmt.permissions[].  SPTEs never use PK or SS, as
+	 * they are not supported for shadow paging and irrelevant for TDP.
+	 */
+	pfec = fault->error_code & (
+		PFERR_WRITE_MASK | PFERR_USER_MASK | PFERR_FETCH_MASK);
+	return (mmu->fmt.permissions[pfec >> 1] >> access) & 1;
+}
+
 /*
  * Return values of handle_mmio_page_fault(), mmu.page_fault(), fast_page_fault(),
  * and of course kvm_mmu_do_page_fault().
diff --git a/arch/x86/kvm/mmu/mmutrace.h b/arch/x86/kvm/mmu/mmutrace.h
index 8354d9f39777..744f111644cb 100644
--- a/arch/x86/kvm/mmu/mmutrace.h
+++ b/arch/x86/kvm/mmu/mmutrace.h
@@ -8,6 +8,13 @@
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM kvmmmu
 
+#ifdef CREATE_TRACE_POINTS
+#define tracing_kvm_rip_read(vcpu) ({					\
+	typeof(vcpu) __vcpu = vcpu;					\
+	__vcpu->arch.guest_state_protected ? 0 : kvm_rip_read(__vcpu);	\
+	})
+#endif
+
 #define KVM_MMU_PAGE_FIELDS		\
 	__field(__u8, mmu_valid_gen)	\
 	__field(__u64, gfn)		\
@@ -447,6 +454,35 @@ TRACE_EVENT(
 		  __entry->gfn, __entry->spte, __entry->level, __entry->errno)
 );
 
+TRACE_EVENT(kvm_faultin_memory_protections,
+	TP_PROTO(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
+		 unsigned access),
+	TP_ARGS(vcpu, fault, access),
+
+	TP_STRUCT__entry(
+		__field(unsigned int, vcpu_id)
+		__field(unsigned long, guest_rip)
+		__field(u64, fault_address)
+		__field(bool, write)
+		__field(bool, exec)
+		__field(unsigned, access)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_id = vcpu->vcpu_id;
+		__entry->guest_rip = tracing_kvm_rip_read(vcpu);
+		__entry->fault_address = fault->gfn;
+		__entry->write = fault->write;
+		__entry->exec = fault->exec;
+		__entry->access = access;
+	),
+
+	TP_printk("vcpu %d rip 0x%lx gfn 0x%016llx access %s protections 0x%x",
+		  __entry->vcpu_id, __entry->guest_rip, __entry->fault_address,
+		  __entry->exec ? "X" : (__entry->write ? "W" : "R"),
+		  __entry->access)
+);
+
 #endif /* _TRACE_KVMMMU_H */
 
 #undef TRACE_INCLUDE_PATH
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index 533ad280accc..a1212104882c 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -995,7 +995,7 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int
 
 	sptep = &sp->spt[i];
 	spte = *sptep;
-	host_access = ACC_ALL;
+	host_access = kvm_get_gfn_protections(vcpu, gfn);
 	if (!(spte & shadow_host_writable_mask))
 		host_access &= ~ACC_WRITE_MASK;
 	slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h
index 589f3954633e..14f322944671 100644
--- a/arch/x86/kvm/mmu/spte.h
+++ b/arch/x86/kvm/mmu/spte.h
@@ -491,7 +491,7 @@ static inline bool is_mmu_writable_spte(u64 spte)
 static inline bool spte_permission_fault(struct kvm_mmu *mmu, u64 spte,
 					 struct kvm_page_fault *fault)
 {
-	unsigned pfec, pte_access;
+	unsigned pte_access;
 
 	if (!is_shadow_present_pte(spte))
 		return true;
@@ -511,14 +511,7 @@ static inline bool spte_permission_fault(struct kvm_mmu *mmu, u64 spte,
 		pte_access |= spte & shadow_xu_mask ? ACC_USER_EXEC_MASK : 0;
 	}
 
-	/*
-	 * RSVD is handled elsewhere, and is used for SMAP in the context
-	 * of accessing fmt.permissions[].  SPTEs never use PK or SS, as
-	 * they are not supported for shadow paging and irrelevant for TDP.
-	 */
-	pfec = fault->error_code & (
-		PFERR_WRITE_MASK | PFERR_USER_MASK | PFERR_FETCH_MASK);
-	return (mmu->fmt.permissions[pfec >> 1] >> pte_access) & 1;
+	return __permission_fault(mmu, pte_access, fault);
 }
 
 /*
-- 
2.52.0

[PATCH 29/45] KVM: x86/emulate: change return value of emulator_io_permitted to int

[Paolo Bonzini]

Allow emulator_io_permitted to distinguish memory attribute faults
from disallowed I/O ports.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/emulate.c | 50 +++++++++++++++++++++++++-----------------
 1 file changed, 30 insertions(+), 20 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 711794592c2c..8464816349b5 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -2526,8 +2526,8 @@ static bool emulator_bad_iopl(struct x86_emulate_ctxt *ctxt)
 #define VMWARE_PORT_VMPORT	(0x5658)
 #define VMWARE_PORT_VMRPC	(0x5659)
 
-static bool emulator_io_port_access_allowed(struct x86_emulate_ctxt *ctxt,
-					    u16 port, u16 len)
+static int emulator_io_port_access_allowed(struct x86_emulate_ctxt *ctxt,
+					   u16 port, u16 len)
 {
 	const struct x86_emulate_ops *ops = ctxt->ops;
 	struct desc_struct tr_seg;
@@ -2543,43 +2543,47 @@ static bool emulator_io_port_access_allowed(struct x86_emulate_ctxt *ctxt,
 	 */
 	if (enable_vmware_backdoor &&
 	    ((port == VMWARE_PORT_VMPORT) || (port == VMWARE_PORT_VMRPC)))
-		return true;
+		return 0;
 
 	ops->get_segment(ctxt, &tr, &tr_seg, &base3, VCPU_SREG_TR);
 	if (!tr_seg.p)
-		return false;
+		return -EACCES;
 	if (desc_limit_scaled(&tr_seg) < 103)
-		return false;
+		return -EACCES;
 	base = get_desc_base(&tr_seg);
 #ifdef CONFIG_X86_64
 	base |= ((u64)base3) << 32;
 #endif
 	r = ops->read_std(ctxt, base + 102, &io_bitmap_ptr, 2, NULL, true);
 	if (r != X86EMUL_CONTINUE)
-		return false;
+		return -EACCES;
 	if (io_bitmap_ptr + port/8 > desc_limit_scaled(&tr_seg))
-		return false;
+		return -EACCES;
 	r = ops->read_std(ctxt, base + io_bitmap_ptr + port/8, &perm, 2, NULL, true);
 	if (r != X86EMUL_CONTINUE)
-		return false;
+		return -EACCES;
 	if ((perm >> bit_idx) & mask)
-		return false;
-	return true;
+		return -EACCES;
+	return 0;
 }
 
-static bool emulator_io_permitted(struct x86_emulate_ctxt *ctxt,
-				  u16 port, u16 len)
+static int emulator_io_permitted(struct x86_emulate_ctxt *ctxt,
+				 u16 port, u16 len)
 {
-	if (ctxt->perm_ok)
-		return true;
+	int r;
 
-	if (emulator_bad_iopl(ctxt))
-		if (!emulator_io_port_access_allowed(ctxt, port, len))
-			return false;
+	if (ctxt->perm_ok)
+		return 0;
+
+	if (emulator_bad_iopl(ctxt)) {
+		r = emulator_io_port_access_allowed(ctxt, port, len);
+		if (r)
+			return r;
+	}
 
 	ctxt->perm_ok = true;
 
-	return true;
+	return 0;
 }
 
 static void string_registers_quirk(struct x86_emulate_ctxt *ctxt)
@@ -3931,8 +3935,11 @@ static int check_rdpmc(struct x86_emulate_ctxt *ctxt)
 
 static int check_perm_in(struct x86_emulate_ctxt *ctxt)
 {
+	int r;
+
 	ctxt->dst.bytes = min(ctxt->dst.bytes, 4u);
-	if (!emulator_io_permitted(ctxt, ctxt->src.val, ctxt->dst.bytes))
+	r = emulator_io_permitted(ctxt, ctxt->src.val, ctxt->dst.bytes);
+	if (r)
 		return emulate_gp(ctxt, 0);
 
 	return X86EMUL_CONTINUE;
@@ -3940,8 +3947,11 @@ static int check_perm_in(struct x86_emulate_ctxt *ctxt)
 
 static int check_perm_out(struct x86_emulate_ctxt *ctxt)
 {
+	int r;
+
 	ctxt->src.bytes = min(ctxt->src.bytes, 4u);
-	if (!emulator_io_permitted(ctxt, ctxt->dst.val, ctxt->src.bytes))
+	r = emulator_io_permitted(ctxt, ctxt->dst.val, ctxt->src.bytes);
+	if (r)
 		return emulate_gp(ctxt, 0);
 
 	return X86EMUL_CONTINUE;
-- 
2.52.0

[PATCH 30/45] KVM: x86/emulate: Propagate memory faults to userspace

[Paolo Bonzini]

Full handling of memory faults is not really possible with KVM's
architecture, short of adding something like a journal to complex
operations like task switches or SMM enter/leave.  However, KVM
also should at least try to propagate faults to userspace, as an
emulation failure if it is not possible to return them as -EFAULT.

Introduce new return values for both the emulator ops
(X86EMUL_MEMORY_FAULT) and emulate.c (EMULATION_MEMORY_FAULT), and
plumb them through the entire call chain.  Terminate them with
-EFAULT as a best effort.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/emulate.c     | 24 ++++++++++++++--
 arch/x86/kvm/kvm_emulate.h |  3 ++
 arch/x86/kvm/svm/svm.c     |  6 +++-
 arch/x86/kvm/vmx/sgx.c     |  4 ++-
 arch/x86/kvm/vmx/vmx.c     | 22 +++++++++------
 arch/x86/kvm/x86.c         | 56 ++++++++++++++++++++++++++------------
 include/linux/kvm_host.h   |  3 ++
 virt/kvm/kvm_main.c        | 24 ++++++++++++++++
 8 files changed, 111 insertions(+), 31 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 8464816349b5..93596cf6aa3d 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -2555,11 +2555,15 @@ static int emulator_io_port_access_allowed(struct x86_emulate_ctxt *ctxt,
 	base |= ((u64)base3) << 32;
 #endif
 	r = ops->read_std(ctxt, base + 102, &io_bitmap_ptr, 2, NULL, true);
+	if (r == X86EMUL_MEMORY_FAULT)
+		return r;
 	if (r != X86EMUL_CONTINUE)
 		return -EACCES;
 	if (io_bitmap_ptr + port/8 > desc_limit_scaled(&tr_seg))
 		return -EACCES;
 	r = ops->read_std(ctxt, base + io_bitmap_ptr + port/8, &perm, 2, NULL, true);
+	if (r == X86EMUL_MEMORY_FAULT)
+		return r;
 	if (r != X86EMUL_CONTINUE)
 		return -EACCES;
 	if ((perm >> bit_idx) & mask)
@@ -2758,7 +2762,10 @@ static int load_state_from_tss32(struct x86_emulate_ctxt *ctxt,
 	int ret;
 	u8 cpl;
 
-	if (ctxt->ops->set_cr(ctxt, 3, tss->cr3))
+	ret = ctxt->ops->set_cr(ctxt, 3, tss->cr3);
+	if (ret == X86EMUL_MEMORY_FAULT)
+		return ret;
+	if (ret)
 		return emulate_gp(ctxt, 0);
 	ctxt->_eip = tss->eip;
 	ctxt->eflags = tss->eflags | 2;
@@ -2993,6 +3000,8 @@ int emulator_task_switch(struct x86_emulate_ctxt *ctxt,
 		ctxt->eip = ctxt->_eip;
 		writeback_registers(ctxt);
 	}
+	if (rc == X86EMUL_MEMORY_FAULT)
+		return EMULATION_MEMORY_FAULT;
 
 	return (rc == X86EMUL_UNHANDLEABLE) ? EMULATION_FAILED : EMULATION_OK;
 }
@@ -3260,7 +3269,10 @@ static int em_cr_write(struct x86_emulate_ctxt *ctxt)
 	int cr_num = ctxt->modrm_reg;
 	int r;
 
-	if (ctxt->ops->set_cr(ctxt, cr_num, ctxt->src.val))
+	r = ctxt->ops->set_cr(ctxt, cr_num, ctxt->src.val);
+	if (r == X86EMUL_MEMORY_FAULT)
+		return r;
+	if (r)
 		return emulate_gp(ctxt, 0);
 
 	/* Disable writeback. */
@@ -3939,6 +3951,8 @@ static int check_perm_in(struct x86_emulate_ctxt *ctxt)
 
 	ctxt->dst.bytes = min(ctxt->dst.bytes, 4u);
 	r = emulator_io_permitted(ctxt, ctxt->src.val, ctxt->dst.bytes);
+	if (r == X86EMUL_MEMORY_FAULT)
+		return r;
 	if (r)
 		return emulate_gp(ctxt, 0);
 
@@ -3951,6 +3965,8 @@ static int check_perm_out(struct x86_emulate_ctxt *ctxt)
 
 	ctxt->src.bytes = min(ctxt->src.bytes, 4u);
 	r = emulator_io_permitted(ctxt, ctxt->dst.val, ctxt->src.bytes);
+	if (r == X86EMUL_MEMORY_FAULT)
+		return r;
 	if (r)
 		return emulate_gp(ctxt, 0);
 
@@ -5219,6 +5235,8 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
 done:
 	if (rc == X86EMUL_PROPAGATE_FAULT)
 		ctxt->have_exception = true;
+	if (rc == X86EMUL_MEMORY_FAULT)
+		return EMULATION_MEMORY_FAULT;
 	return (rc != X86EMUL_CONTINUE) ? EMULATION_FAILED : EMULATION_OK;
 }
 
@@ -5579,6 +5597,8 @@ int x86_emulate_insn(struct x86_emulate_ctxt *ctxt, bool check_intercepts)
 		ctxt->eip = (u32)ctxt->_eip;
 
 done:
+	if (rc == X86EMUL_MEMORY_FAULT)
+		return EMULATION_MEMORY_FAULT;
 	if (rc == X86EMUL_PROPAGATE_FAULT) {
 		if (KVM_EMULATOR_BUG_ON(ctxt->exception.vector > 0x1f, ctxt))
 			return EMULATION_FAILED;
diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h
index 10886bea82c9..488738794bb6 100644
--- a/arch/x86/kvm/kvm_emulate.h
+++ b/arch/x86/kvm/kvm_emulate.h
@@ -97,6 +97,8 @@ struct x86_instruction_info {
 #define X86EMUL_INTERCEPTED     6 /* Intercepted by nested VMCB/VMCS */
 /* Emulation during event vectoring is unhandleable. */
 #define X86EMUL_UNHANDLEABLE_VECTORING	7
+/* Terminate emulation and return a memory fault to userspace. */
+#define X86EMUL_MEMORY_FAULT	8
 
 /* x86-specific emulation flags */
 #define X86EMUL_F_WRITE			BIT(0)
@@ -532,6 +534,7 @@ bool x86_page_table_writing_insn(struct x86_emulate_ctxt *ctxt);
 #define EMULATION_OK 0
 #define EMULATION_RESTART 1
 #define EMULATION_INTERCEPTED 2
+#define EMULATION_MEMORY_FAULT 3
 void init_decode_cache(struct x86_emulate_ctxt *ctxt);
 int x86_emulate_insn(struct x86_emulate_ctxt *ctxt, bool check_intercepts);
 int emulator_task_switch(struct x86_emulate_ctxt *ctxt,
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 487897ef11dd..205d8783aa13 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2363,13 +2363,17 @@ static int gp_interception(struct kvm_vcpu *vcpu)
 	struct vcpu_svm *svm = to_svm(vcpu);
 	u32 error_code = svm->vmcb->control.exit_info_1;
 	u64 svm_exit_code;
+	int r;
 
 	/* Both #GP cases have zero error_code */
 	if (error_code)
 		goto reinject;
 
 	/* Decode the instruction for usage later */
-	if (x86_decode_emulated_instruction(vcpu, 0, NULL, 0) != EMULATION_OK)
+	r = x86_decode_emulated_instruction(vcpu, 0, NULL, 0);
+	if (r == EMULATION_MEMORY_FAULT)
+		return -EFAULT;
+	if (r != EMULATION_OK)
 		goto reinject;
 
 	/* FIXME: Handle SVM instructions through the emulator */
diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c
index 771c75a58343..162548b97c00 100644
--- a/arch/x86/kvm/vmx/sgx.c
+++ b/arch/x86/kvm/vmx/sgx.c
@@ -236,7 +236,9 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu)
 	 */
 	r = kvm_read_guest_virt(vcpu, pageinfo_gva, &pageinfo,
 				sizeof(pageinfo), &ex);
-	if (r == X86EMUL_PROPAGATE_FAULT) {
+	if (r == X86EMUL_MEMORY_FAULT) {
+		return -EFAULT;
+	} else if (r == X86EMUL_PROPAGATE_FAULT) {
 		kvm_inject_emulated_page_fault(vcpu, &ex);
 		return 1;
 	} else if (r != X86EMUL_CONTINUE) {
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index fe43718b751b..80dfb3c6268b 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5342,19 +5342,21 @@ static bool rmode_exception(struct kvm_vcpu *vcpu, int vec)
 static int handle_rmode_exception(struct kvm_vcpu *vcpu,
 				  int vec, u32 err_code)
 {
+	int r;
+
 	/*
 	 * Instruction with address size override prefix opcode 0x67
 	 * Cause the #SS fault with 0 error code in VM86 mode.
 	 */
 	if (((vec == GP_VECTOR) || (vec == SS_VECTOR)) && err_code == 0) {
-		if (kvm_emulate_instruction(vcpu, 0)) {
-			if (vcpu->arch.halt_request) {
-				vcpu->arch.halt_request = 0;
-				return kvm_emulate_halt_noskip(vcpu);
-			}
-			return 1;
+		r = kvm_emulate_instruction(vcpu, 0);
+		if (r <= 0)
+			return r;
+		if (vcpu->arch.halt_request) {
+			vcpu->arch.halt_request = 0;
+			return kvm_emulate_halt_noskip(vcpu);
 		}
-		return 0;
+		return 1;
 	}
 
 	/*
@@ -6080,6 +6082,7 @@ static int handle_invalid_guest_state(struct kvm_vcpu *vcpu)
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	bool intr_window_requested;
 	unsigned count = 130;
+	int r;
 
 	intr_window_requested = exec_controls_get(vmx) &
 				CPU_BASED_INTR_WINDOW_EXITING;
@@ -6098,8 +6101,9 @@ static int handle_invalid_guest_state(struct kvm_vcpu *vcpu)
 		 */
 		smp_rmb();
 
-		if (!kvm_emulate_instruction(vcpu, 0))
-			return 0;
+		r = kvm_emulate_instruction(vcpu, 0);
+		if (r <= 0)
+			return r;
 
 		if (vmx_unhandleable_emulation_required(vcpu)) {
 			kvm_prepare_emulation_failure_exit(vcpu);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index caff8534267d..5f7f87ff4d17 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4865,7 +4865,6 @@ static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 	void *data = val;
-	int r = X86EMUL_CONTINUE;
 
 	while (bytes) {
 		gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, exception);
@@ -4878,16 +4877,16 @@ static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
 		ret = kvm_vcpu_read_guest_page(vcpu, gpa >> PAGE_SHIFT, data,
 					       offset, toread);
 		if (ret < 0) {
-			r = X86EMUL_IO_NEEDED;
-			goto out;
+			if (kvm_prepare_memory_fault_exit_for_range(vcpu, gpa, toread, false))
+				return X86EMUL_MEMORY_FAULT;
+			return X86EMUL_IO_NEEDED;
 		}
 
 		bytes -= toread;
 		data += toread;
 		addr += toread;
 	}
-out:
-	return r;
+	return X86EMUL_CONTINUE;
 }
 
 /* used for instruction fetching */
@@ -4912,8 +4911,16 @@ static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt,
 		bytes = (unsigned)PAGE_SIZE - offset;
 	ret = kvm_vcpu_fetch_guest_page(vcpu, gpa >> PAGE_SHIFT, val,
 					offset, bytes);
-	if (unlikely(ret < 0))
+	if (unlikely(ret < 0)) {
+		unsigned long attrs = kvm_get_vm_memory_attributes(vcpu->kvm, gpa_to_gfn(gpa));
+
+		if (!kvm_mem_attributes_may_exec(attrs)) {
+			kvm_prepare_memory_fault_exit(vcpu, gpa & PAGE_MASK, PAGE_SIZE,
+						      false, true, false);
+			return X86EMUL_MEMORY_FAULT;
+		}
 		return X86EMUL_IO_NEEDED;
+	}
 
 	return X86EMUL_CONTINUE;
 }
@@ -4957,7 +4964,6 @@ static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 	void *data = val;
-	int r = X86EMUL_CONTINUE;
 
 	while (bytes) {
 		gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, exception);
@@ -4969,16 +4975,16 @@ static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes
 			return X86EMUL_PROPAGATE_FAULT;
 		ret = kvm_vcpu_write_guest(vcpu, gpa, data, towrite);
 		if (ret < 0) {
-			r = X86EMUL_IO_NEEDED;
-			goto out;
+			if (kvm_prepare_memory_fault_exit_for_range(vcpu, gpa, towrite, true))
+				return X86EMUL_MEMORY_FAULT;
+			return X86EMUL_IO_NEEDED;
 		}
 
 		bytes -= towrite;
 		data += towrite;
 		addr += towrite;
 	}
-out:
-	return r;
+	return X86EMUL_CONTINUE;
 }
 
 static int emulator_write_std(struct x86_emulate_ctxt *ctxt, gva_t addr, void *val,
@@ -5098,7 +5104,7 @@ struct read_write_emulator_ops {
 static int emulator_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa,
 			       void *val, int bytes)
 {
-	return !kvm_vcpu_read_guest(vcpu, gpa, val, bytes);
+	return kvm_vcpu_read_guest(vcpu, gpa, val, bytes);
 }
 
 static int emulator_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa,
@@ -5107,10 +5113,10 @@ static int emulator_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa,
 	int ret;
 
 	ret = kvm_vcpu_write_guest(vcpu, gpa, val, bytes);
-	if (ret < 0)
-		return 0;
+	if (ret)
+		return ret;
 	kvm_page_track_write(vcpu, gpa, val, bytes);
-	return 1;
+	return 0;
 }
 
 static int emulator_read_write_onepage(unsigned long addr, void *val,
@@ -5139,6 +5145,8 @@ static int emulator_read_write_onepage(unsigned long addr, void *val,
 		ret = vcpu_is_mmio_gpa(vcpu, addr, gpa, write);
 	} else {
 		ret = vcpu_mmio_gva_to_gpa(vcpu, addr, &gpa, exception, write);
+		if (ret == -EFAULT)
+			return X86EMUL_MEMORY_FAULT;
 		if (ret < 0)
 			return X86EMUL_PROPAGATE_FAULT;
 	}
@@ -5151,8 +5159,14 @@ static int emulator_read_write_onepage(unsigned long addr, void *val,
 	 * i.e. if accessing host user memory failed, but this has been KVM's
 	 * historical ABI for decades.
 	 */
-	if (!ret && ops->read_write_guest(vcpu, gpa, val, bytes))
-		return X86EMUL_CONTINUE;
+	if (!ret) {
+		ret = ops->read_write_guest(vcpu, gpa, val, bytes);
+		if (!ret)
+			return X86EMUL_CONTINUE;
+	}
+
+	if (kvm_prepare_memory_fault_exit_for_range(vcpu, gpa, bytes, write))
+		return X86EMUL_MEMORY_FAULT;
 
 	/*
 	 * Attempt to handle emulated MMIO within the kernel, e.g. for accesses
@@ -5602,6 +5616,8 @@ static int emulator_set_cr(struct x86_emulate_ctxt *ctxt, int cr, ulong val)
 		res = -1;
 	}
 
+	if (res == -EFAULT)
+		return X86EMUL_MEMORY_FAULT;
 	return res;
 }
 
@@ -6416,6 +6432,8 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 
 		r = x86_decode_emulated_instruction(vcpu, emulation_type,
 						    insn, insn_len);
+		if (r == EMULATION_MEMORY_FAULT)
+			return -EFAULT;
 		if (r != EMULATION_OK)  {
 			if ((emulation_type & EMULTYPE_TRAP_UD) ||
 			    (emulation_type & EMULTYPE_TRAP_UD_FORCED)) {
@@ -6528,7 +6546,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	if (r == EMULATION_INTERCEPTED)
 		return 1;
 
-	if (r == EMULATION_FAILED) {
+	if (r == EMULATION_MEMORY_FAULT || r == EMULATION_FAILED) {
 		if (kvm_unprotect_and_retry_on_failure(vcpu, cr2_or_gpa,
 						       emulation_type))
 			return 1;
@@ -9210,6 +9228,8 @@ int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index,
 
 	ret = emulator_task_switch(ctxt, tss_selector, idt_index, reason,
 				   has_error_code, error_code);
+	if (ret == EMULATION_MEMORY_FAULT)
+		goto unhandled_task_switch;
 
 	/*
 	 * Report an error userspace if MMIO is needed, as KVM doesn't support
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index de540af1fa24..4128cdb71869 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2641,6 +2641,9 @@ static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
 	 */
 	return smp_load_acquire(&kvm->mem_attrs.generation);
 }
+
+bool kvm_prepare_memory_fault_exit_for_range(struct kvm_vcpu *vcpu, gpa_t gpa,
+					     size_t size, bool write);
 #else
 static inline bool kvm_range_has_any_vm_memory_attributes(struct kvm *kvm,
 			gfn_t start, gfn_t end, unsigned long mask)
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 8283b2415adc..326bb8e144b7 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2485,6 +2485,30 @@ static u64 kvm_supports_memory_protection_attr(struct kvm *kvm)
 #endif
 
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
+bool kvm_prepare_memory_fault_exit_for_range(struct kvm_vcpu *vcpu, gpa_t gpa,
+					     size_t size, bool write)
+{
+	gfn_t gfn, last;
+
+	last = gpa_to_gfn(gpa + size - 1);
+	for (gfn = gpa_to_gfn(gpa); gfn <= last; gfn++) {
+		if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gfn)) {
+			kvm_prepare_memory_fault_exit(vcpu, gfn_to_gpa(gfn), PAGE_SIZE,
+						      write, false, false);
+			return true;
+		}
+
+		if (write && !kvm_mem_attributes_may_write_gfn(vcpu->kvm, gfn)) {
+			kvm_prepare_memory_fault_exit(vcpu, gfn_to_gpa(gfn), PAGE_SIZE,
+						      true, false, false);
+			return true;
+		}
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_prepare_memory_fault_exit_for_range);
+
 u64 kvm_supported_vm_mem_attributes(struct kvm *kvm)
 {
 	u64 supported_attrs = 0;
-- 
2.52.0

[PATCH 31/45] KVM: x86: "Handle" failed decode when skipping instructions

[Paolo Bonzini]

When calling into the emulator to skip an instruction, the caller has
already performed the side effects of the instruction and cannot
propagate -EFAULT.  Mark the VM dead.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/svm/svm.c |  9 +++++++--
 arch/x86/kvm/vmx/vmx.c | 10 +++++++---
 arch/x86/kvm/x86.c     |  4 ++--
 3 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 205d8783aa13..2601f8359137 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -304,6 +304,7 @@ static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	unsigned long old_rflags;
+	int r;
 
 	/*
 	 * SEV-ES does not expose the next RIP. The RIP update is controlled by
@@ -321,7 +322,10 @@ static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
 		if (unlikely(!commit_side_effects))
 			old_rflags = svm->vmcb->save.rflags;
 
-		if (!kvm_emulate_instruction(vcpu, emul_type))
+		r = kvm_emulate_instruction(vcpu, emul_type);
+		if (r < 0)
+			kvm_vm_dead(vcpu->kvm);
+		if (r <= 0)
 			return 0;
 
 		if (unlikely(!commit_side_effects))
@@ -2128,7 +2132,8 @@ static int icebp_interception(struct kvm_vcpu *vcpu)
 	 * instead of after it (and SVM doesn't provide enough information for
 	 * KVM to detect and manually advance the pre-#DB RIP).
 	 */
-	svm_skip_emulated_instruction(vcpu);
+	if (!svm_skip_emulated_instruction(vcpu))
+		return 0;
 	kvm_queue_exception(vcpu, DB_VECTOR);
 	return 1;
 }
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 80dfb3c6268b..27cb0d64fb5c 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -1809,6 +1809,7 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu)
 	union vmx_exit_reason exit_reason = vt_get_exit_reason(vcpu);
 	unsigned long rip, orig_rip;
 	u32 instr_len;
+	int r;
 
 	/*
 	 * Using VMCS.VM_EXIT_INSTRUCTION_LEN on EPT misconfig depends on
@@ -1857,7 +1858,10 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu)
 #endif
 		kvm_rip_write(vcpu, rip);
 	} else {
-		if (!kvm_emulate_instruction(vcpu, EMULTYPE_SKIP))
+		r = kvm_emulate_instruction(vcpu, EMULTYPE_SKIP);
+		if (r < 0)
+			kvm_vm_dead(vcpu->kvm);
+		if (r <= 0)
 			return 0;
 	}
 
@@ -5538,8 +5542,8 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu)
 			 * higher priority.  Note, skipping ICEBP still clears
 			 * STI and MOVSS blocking.
 			 */
-			if (is_icebp(intr_info))
-				WARN_ON(!skip_emulated_instruction(vcpu));
+			if (is_icebp(intr_info) && !skip_emulated_instruction(vcpu))
+				return 0;
 
 			kvm_queue_exception_p(vcpu, DB_VECTOR, dr6);
 			return 1;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 5f7f87ff4d17..5818101d1d8a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -6235,8 +6235,8 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu)
 	int r;
 
 	r = kvm_x86_call(skip_emulated_instruction)(vcpu);
-	if (unlikely(!r))
-		return 0;
+	if (unlikely(r <= 0))
+		return r;
 
 	kvm_pmu_instruction_retired(vcpu);
 
-- 
2.52.0

[PATCH 32/45] KVM: vmx/sgx: make return value of sgx_gva_to_gpa consistent with its callers

[Paolo Bonzini]

Return 1 instead of -EFAULT, in preparation for using -EFAULT for
memory protection failures.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/vmx/sgx.c | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c
index 162548b97c00..df18adbdd829 100644
--- a/arch/x86/kvm/vmx/sgx.c
+++ b/arch/x86/kvm/vmx/sgx.c
@@ -84,7 +84,7 @@ static int sgx_gva_to_gpa(struct kvm_vcpu *vcpu, gva_t gva, bool write,
 
 	if (*gpa == INVALID_GPA) {
 		kvm_inject_emulated_page_fault(vcpu, &ex);
-		return -EFAULT;
+		return 1;
 	}
 
 	return 0;
@@ -256,10 +256,13 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu)
 	 * Translate the SECINFO, SOURCE and SECS pointers from GVA to GPA.
 	 * Resume the guest on failure to inject a #PF.
 	 */
-	if (sgx_gva_to_gpa(vcpu, metadata_gva, false, &metadata_gpa) ||
-	    sgx_gva_to_gpa(vcpu, contents_gva, false, &contents_gpa) ||
-	    sgx_gva_to_gpa(vcpu, secs_gva, true, &secs_gpa))
-		return 1;
+	r = sgx_gva_to_gpa(vcpu, metadata_gva, false, &metadata_gpa);
+	if (!r)
+		r = sgx_gva_to_gpa(vcpu, contents_gva, false, &contents_gpa);
+	if (!r)
+		r = sgx_gva_to_gpa(vcpu, secs_gva, true, &secs_gpa);
+	if (r)
+		return r < 0 ? r : 1;
 
 	/*
 	 * ...and then to HVA.  The order of accesses isn't architectural, i.e.
@@ -314,10 +317,13 @@ static int handle_encls_einit(struct kvm_vcpu *vcpu)
 	 * Translate the SIGSTRUCT, SECS and TOKEN pointers from GVA to GPA.
 	 * Resume the guest on failure to inject a #PF.
 	 */
-	if (sgx_gva_to_gpa(vcpu, sig_gva, false, &sig_gpa) ||
-	    sgx_gva_to_gpa(vcpu, secs_gva, true, &secs_gpa) ||
-	    sgx_gva_to_gpa(vcpu, token_gva, false, &token_gpa))
-		return 1;
+	ret = sgx_gva_to_gpa(vcpu, sig_gva, false, &sig_gpa);
+	if (!ret)
+		ret = sgx_gva_to_gpa(vcpu, secs_gva, true, &secs_gpa);
+	if (!ret)
+		ret = sgx_gva_to_gpa(vcpu, token_gva, false, &token_gpa);
+	if (ret)
+		return ret < 0 ? ret : 1;
 
 	/*
 	 * ...and then to HVA.  The order of accesses isn't architectural, i.e.
-- 
2.52.0

[PATCH 33/45] KVM: x86: Allow page fault fixup to return errors

[Paolo Bonzini]

kvm_fixup_and_inject_pf_error() reflects intercepted hardware page
faults into the guest, walking the guest page tables when necessary
to obtain an error code consistent with the guest's paging rules.
For example, a GPA that exceeds guest.MAXPHYADDR but fits within
host.MAXPHYADDR must cause a reserved-bit fault, even though hardware
does not report one.

Allow the function's callers to propagate memory attribute failures
from the page table walk; subsequent changes will return from
kvm_fixup_and_inject_pf_error() any error from the page-table walk.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/vmx/vmx.c | 15 +++++++++++----
 arch/x86/kvm/x86.c     |  3 ++-
 arch/x86/kvm/x86.h     |  2 +-
 3 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 27cb0d64fb5c..c4511f83d59b 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5418,10 +5418,14 @@ static int vmx_handle_page_fault(struct kvm_vcpu *vcpu, u32 error_code)
 		WARN_ON_ONCE(!IS_ENABLED(CONFIG_X86_SGX_KVM) ||
 			     !cpu_feature_enabled(X86_FEATURE_SGX2));
 
-		if (guest_cpu_cap_has(vcpu, X86_FEATURE_SGX2))
-			kvm_fixup_and_inject_pf_error(vcpu, cr2, error_code);
-		else
+		if (guest_cpu_cap_has(vcpu, X86_FEATURE_SGX2)) {
+			int r = kvm_fixup_and_inject_pf_error(vcpu, cr2, error_code);
+
+			if (r)
+				return r;
+		} else {
 			kvm_inject_gp(vcpu, 0);
+		}
 		return 1;
 	}
 
@@ -5431,7 +5435,10 @@ static int vmx_handle_page_fault(struct kvm_vcpu *vcpu, u32 error_code)
 	 * to the GPA being legal with respect to host.MAXPHYADDR).
 	 */
 	if (enable_ept) {
-		kvm_fixup_and_inject_pf_error(vcpu, cr2, error_code);
+		int r = kvm_fixup_and_inject_pf_error(vcpu, cr2, error_code);
+
+		if (r)
+			return r;
 		return 1;
 	}
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 5818101d1d8a..bb36ca1106e5 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10811,7 +10811,7 @@ void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range
 #endif
 #endif
 
-void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code)
+int kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code)
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 	struct x86_exception fault;
@@ -10833,6 +10833,7 @@ void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_c
 		fault.async_page_fault = false;
 	}
 	gva_walk->inject_page_fault(vcpu, &fault, true);
+	return 0;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_fixup_and_inject_pf_error);
 
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index cfe96962a4e4..4239765629b0 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -368,7 +368,7 @@ void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu,
 				   struct kvm_queued_exception *ex);
 void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu);
 
-void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code);
+int kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code);
 int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_type,
 				    void *insn, int insn_len);
 int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
-- 
2.52.0

[PATCH 34/45] KVM: x86/mmu: Adjust gva_to_gpa to allow returning an error

[Paolo Bonzini]

In order to allow the emulator to check memory attributes and
return -EFAULT to userspace, INVALID_GPA is not sufficient anymore
to distinguish all possible return values of w->gva_to_gpa.  Move the
guest physical address to an out parameter, and use the return
value for errno.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 Documentation/virt/kvm/api.rst  |  4 +-
 arch/x86/include/asm/kvm_host.h |  6 +--
 arch/x86/kvm/mmu.h              | 10 ++---
 arch/x86/kvm/mmu/mmu.c          | 11 +++---
 arch/x86/kvm/mmu/paging_tmpl.h  | 15 ++++----
 arch/x86/kvm/svm/nested.c       |  4 +-
 arch/x86/kvm/vmx/nested.c       |  4 +-
 arch/x86/kvm/vmx/sgx.c          |  7 +++-
 arch/x86/kvm/x86.c              | 66 ++++++++++++++++++++++-----------
 9 files changed, 80 insertions(+), 47 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ac8cd51bdada..743a738ff8d3 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -521,7 +521,9 @@ data structures.
 :Architectures: x86
 :Type: vcpu ioctl
 :Parameters: struct kvm_translation (in/out)
-:Returns: 0 on success, -1 on error
+:Returns: 0 on success, -1 on error.  errno is EFAULT if
+          the parameter is invalid but also if guest memory
+          could not be accessed
 
 Translates a virtual address according to the vcpu's current address
 translation mode.
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 6a35fd384a74..99f9c9269f2f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -490,9 +490,9 @@ struct kvm_pagewalk {
 	void (*inject_page_fault)(struct kvm_vcpu *vcpu,
 				  struct x86_exception *fault,
 				  bool from_hardware);
-	gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
-			    gpa_t gva_or_gpa, u64 access,
-			    struct x86_exception *exception);
+	int (*gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
+			  gpa_t gva_or_gpa, u64 access, gpa_t *gpa,
+			  struct x86_exception *exception);
 
 	union kvm_cpu_role cpu_role;
 	struct kvm_page_format fmt;
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 2ae7f9ed4cf8..224f4c1a957e 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -185,12 +185,12 @@ static inline bool kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu,
 void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu,
 			ulong roots_to_free);
 void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu);
-gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva,
+int kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, gpa_t *gpa,
+			    struct x86_exception *exception);
+int kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, gpa_t *gpa,
+			     struct x86_exception *exception);
+int kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, gpa_t *gpa,
 			      struct x86_exception *exception);
-gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva,
-			       struct x86_exception *exception);
-gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,
-				struct x86_exception *exception);
 
 static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu)
 {
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 787bc31acda3..c827f2e1f3d1 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -2993,7 +2993,7 @@ bool __kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		goto out;
 
 	if (!vcpu->arch.mmu->root_role.direct) {
-		gpa = kvm_mmu_gva_to_gpa_write(vcpu, cr2_or_gpa, NULL);
+		kvm_mmu_gva_to_gpa_write(vcpu, cr2_or_gpa, &gpa, NULL);
 		if (gpa == INVALID_GPA)
 			goto out;
 	}
@@ -4462,9 +4462,9 @@ void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu)
 	kvm_mmu_free_roots(vcpu->kvm, vcpu->arch.mmu, roots_to_free);
 }
 
-static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
-				  gpa_t vaddr, u64 access,
-				  struct x86_exception *exception)
+static int nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
+				gpa_t vaddr, u64 access, gpa_t *gpa,
+				struct x86_exception *exception)
 {
 	if (exception)
 		exception->error_code = 0;
@@ -4474,8 +4474,9 @@ static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 	 * user-mode address if CR0.PG=0.  Therefore *include* ACC_USER_MASK in
 	 * the last argument to kvm_translate_gpa (which NPT does not use).
 	 */
-	return kvm_translate_gpa(vcpu, w, vaddr, access | PFERR_GUEST_FINAL_MASK,
+	*gpa = kvm_translate_gpa(vcpu, w, vaddr, access | PFERR_GUEST_FINAL_MASK,
 				 exception, ACC_ALL);
+	return 0;
 }
 
 static bool mmio_info_in_cache(struct kvm_vcpu *vcpu, u64 addr, bool direct)
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index a1212104882c..7775ba789258 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -901,14 +901,15 @@ static gpa_t FNAME(get_level1_sp_gpa)(struct kvm_mmu_page *sp)
 }
 
 /* Note, @addr is a GPA when gva_to_gpa() translates an L2 GPA to an L1 GPA. */
-static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
-			       gpa_t addr, u64 access,
-			       struct x86_exception *exception)
+static int FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
+			     gpa_t addr, u64 access, gpa_t *gpa,
+			     struct x86_exception *exception)
 {
 	struct guest_walker walker;
-	gpa_t gpa = INVALID_GPA;
 	int r;
 
+	*gpa = INVALID_GPA;
+
 #ifndef CONFIG_X86_64
 	/* A 64-bit GVA should be impossible on 32-bit KVM. */
 	WARN_ON_ONCE((addr >> 32) && w == &vcpu->arch.gva_walk);
@@ -917,12 +918,12 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 	r = FNAME(walk_addr_generic)(&walker, vcpu, w, addr, access);
 
 	if (r) {
-		gpa = gfn_to_gpa(walker.gfn);
-		gpa |= addr & ~PAGE_MASK;
+		*gpa = gfn_to_gpa(walker.gfn);
+		*gpa |= addr & ~PAGE_MASK;
 	} else if (exception)
 		*exception = walker.fault;
 
-	return gpa;
+	return 0;
 }
 
 /*
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index fedddcd1cc96..5d0a2f658365 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -2155,6 +2155,7 @@ static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_pagewalk *w = &vcpu->arch.ngpa_walk;
+	gpa_t result;
 
 	if (WARN_ON_ONCE(!mmu_is_nested(vcpu)))
 		return gpa;
@@ -2163,7 +2164,8 @@ static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
 	if (!(svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_GMET))
 		access |= PFERR_USER_MASK;
 
-	return w->gva_to_gpa(vcpu, w, gpa, access, exception);
+	w->gva_to_gpa(vcpu, w, gpa, access, &result, exception);
+	return result;
 }
 
 struct kvm_x86_nested_ops svm_nested_ops __initdata = {
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index b133e53fde39..a666c2908889 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -7483,6 +7483,7 @@ static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
 				      u64 pte_access)
 {
 	struct kvm_pagewalk *w = &vcpu->arch.ngpa_walk;
+	gpa_t result;
 
 	if (WARN_ON_ONCE(!mmu_is_nested(vcpu)))
 		return gpa;
@@ -7495,7 +7496,8 @@ static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
 	if ((pte_access & ACC_USER_MASK) && (access & PFERR_GUEST_FINAL_MASK))
 		access |= PFERR_USER_MASK;
 
-	return w->gva_to_gpa(vcpu, w, gpa, access, exception);
+	w->gva_to_gpa(vcpu, w, gpa, access, &result, exception);
+	return result;
 }
 
 struct kvm_x86_nested_ops vmx_nested_ops __initdata = {
diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c
index df18adbdd829..2336aa9f68c5 100644
--- a/arch/x86/kvm/vmx/sgx.c
+++ b/arch/x86/kvm/vmx/sgx.c
@@ -76,11 +76,14 @@ static int sgx_gva_to_gpa(struct kvm_vcpu *vcpu, gva_t gva, bool write,
 			  gpa_t *gpa)
 {
 	struct x86_exception ex;
+	int r;
 
 	if (write)
-		*gpa = kvm_mmu_gva_to_gpa_write(vcpu, gva, &ex);
+		r = kvm_mmu_gva_to_gpa_write(vcpu, gva, gpa, &ex);
 	else
-		*gpa = kvm_mmu_gva_to_gpa_read(vcpu, gva, &ex);
+		r = kvm_mmu_gva_to_gpa_read(vcpu, gva, gpa, &ex);
+	if (r)
+		return r;
 
 	if (*gpa == INVALID_GPA) {
 		kvm_inject_emulated_page_fault(vcpu, &ex);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index bb36ca1106e5..53400fda9f44 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4829,34 +4829,34 @@ static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa_t addr, int len, void *v)
 	return handled;
 }
 
-gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva,
-			      struct x86_exception *exception)
+int kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, gpa_t *gpa,
+			    struct x86_exception *exception)
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 
 	u64 access = (kvm_x86_call(get_cpl)(vcpu) == 3) ? PFERR_USER_MASK : 0;
-	return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, exception);
+	return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, gpa, exception);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_gva_to_gpa_read);
 
-gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva,
-			       struct x86_exception *exception)
+int kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, gpa_t *gpa,
+			     struct x86_exception *exception)
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 
 	u64 access = (kvm_x86_call(get_cpl)(vcpu) == 3) ? PFERR_USER_MASK : 0;
 	access |= PFERR_WRITE_MASK;
-	return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, exception);
+	return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, gpa, exception);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_gva_to_gpa_write);
 
 /* uses this to access any guest's mapped memory without checking CPL */
-gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,
-				struct x86_exception *exception)
+int kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, gpa_t *gpa,
+			      struct x86_exception *exception)
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 
-	return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, 0, exception);
+	return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, 0, gpa, exception);
 }
 
 static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
@@ -4867,11 +4867,14 @@ static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
 	void *data = val;
 
 	while (bytes) {
-		gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, exception);
+		gpa_t gpa;
 		unsigned offset = addr & (PAGE_SIZE-1);
 		unsigned toread = min(bytes, (unsigned)PAGE_SIZE - offset);
 		int ret;
 
+		ret = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, &gpa, exception);
+		if (ret == -EFAULT)
+			return X86EMUL_MEMORY_FAULT;
 		if (gpa == INVALID_GPA)
 			return X86EMUL_PROPAGATE_FAULT;
 		ret = kvm_vcpu_read_guest_page(vcpu, gpa >> PAGE_SHIFT, data,
@@ -4901,8 +4904,12 @@ static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt,
 	int ret;
 
 	/* Inline kvm_read_guest_virt_helper for speed.  */
-	gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access|PFERR_FETCH_MASK,
-					  exception);
+	gpa_t gpa;
+
+	ret = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access | PFERR_FETCH_MASK,
+				   &gpa, exception);
+	if (unlikely(ret == -EFAULT))
+		return X86EMUL_MEMORY_FAULT;
 	if (unlikely(gpa == INVALID_GPA))
 		return X86EMUL_PROPAGATE_FAULT;
 
@@ -4966,11 +4973,14 @@ static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes
 	void *data = val;
 
 	while (bytes) {
-		gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, exception);
+		gpa_t gpa;
 		unsigned offset = addr & (PAGE_SIZE-1);
 		unsigned towrite = min(bytes, (unsigned)PAGE_SIZE - offset);
 		int ret;
 
+		ret = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, &gpa, exception);
+		if (ret == -EFAULT)
+			return X86EMUL_MEMORY_FAULT;
 		if (gpa == INVALID_GPA)
 			return X86EMUL_PROPAGATE_FAULT;
 		ret = kvm_vcpu_write_guest(vcpu, gpa, data, towrite);
@@ -5085,7 +5095,8 @@ static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva,
 		return 1;
 	}
 
-	*gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, exception);
+	if (gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, gpa, exception))
+		return -EFAULT;
 
 	if (*gpa == INVALID_GPA)
 		return -1;
@@ -5332,7 +5343,9 @@ static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt,
 	if (bytes > 8 || (bytes & (bytes - 1)))
 		goto emul_write;
 
-	gpa = kvm_mmu_gva_to_gpa_write(vcpu, addr, NULL);
+	r = kvm_mmu_gva_to_gpa_write(vcpu, addr, &gpa, NULL);
+	if (r == -EFAULT)
+		return X86EMUL_MEMORY_FAULT;
 
 	if (gpa == INVALID_GPA ||
 	    (gpa & PAGE_MASK) == APIC_DEFAULT_PHYS_BASE)
@@ -9341,19 +9354,24 @@ int kvm_arch_vcpu_ioctl_translate(struct kvm_vcpu *vcpu,
 {
 	unsigned long vaddr = tr->linear_address;
 	gpa_t gpa;
-	int idx;
+	int idx, r;
 
 	vcpu_load(vcpu);
-
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
-	gpa = kvm_mmu_gva_to_gpa_system(vcpu, vaddr, NULL);
+	r = kvm_mmu_gva_to_gpa_system(vcpu, vaddr, &gpa, NULL);
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
+	vcpu_put(vcpu);
+
+	if (r) {
+		if (r != -EFAULT)
+			return r;
+		gpa = INVALID_GPA;
+	}
+
 	tr->physical_address = gpa;
 	tr->valid = gpa != INVALID_GPA;
 	tr->writeable = 1;
 	tr->usermode = 0;
-
-	vcpu_put(vcpu);
 	return 0;
 }
 
@@ -10815,11 +10833,15 @@ int kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_co
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 	struct x86_exception fault;
+	gpa_t gpa;
+	int r;
 	u64 access = error_code &
 		(PFERR_WRITE_MASK | PFERR_FETCH_MASK | PFERR_USER_MASK);
 
-	if (!(error_code & PFERR_PRESENT_MASK) ||
-	    gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, &fault) != INVALID_GPA) {
+	r = gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, &gpa, &fault);
+	if (r)
+		return r;
+	if (!(error_code & PFERR_PRESENT_MASK) || gpa != INVALID_GPA) {
 		/*
 		 * If gva_walk->gva_to_gpa succeeded, the page
 		 * tables probably do not match the TLB.  Just proceed
-- 
2.52.0

[PATCH 35/45] KVM: x86/mmu: Propagate errors from nested GPA translation

[Paolo Bonzini]

The nested translation callbacks currently discard errors returned
by gva_to_gpa().  Adjust translate_nested_gpa()'s calling convention
similar to what was done to gva_to_gpa(), so that errors can be
propagated from page walk as well as from users that directly care
about L2 GPAs.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/include/asm/kvm_host.h |  8 ++++----
 arch/x86/kvm/hyperv.c           |  6 ++++--
 arch/x86/kvm/mmu.h              | 18 ++++++++++--------
 arch/x86/kvm/mmu/mmu.c          |  5 ++---
 arch/x86/kvm/mmu/paging_tmpl.h  | 12 ++++++------
 arch/x86/kvm/regs.c             |  8 +++++---
 arch/x86/kvm/svm/nested.c       | 18 +++++++++---------
 arch/x86/kvm/vmx/nested.c       | 18 +++++++++---------
 arch/x86/kvm/x86.c              | 15 +++++++++++----
 9 files changed, 60 insertions(+), 48 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 99f9c9269f2f..7fbcf60e56cf 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1780,10 +1780,10 @@ struct kvm_x86_nested_ops {
 			 struct kvm_nested_state *kvm_state);
 	bool (*get_nested_state_pages)(struct kvm_vcpu *vcpu);
 	int (*write_log_dirty)(struct kvm_vcpu *vcpu, gpa_t l2_gpa);
-	gpa_t (*translate_nested_gpa)(struct kvm_vcpu *vcpu, gpa_t gpa,
-				      u64 access,
-				      struct x86_exception *exception,
-				      u64 pte_access);
+	int (*translate_nested_gpa)(struct kvm_vcpu *vcpu, gpa_t gpa,
+				    u64 access, gpa_t *translated_gpa,
+				    struct x86_exception *exception,
+				    u64 pte_access);
 
 	int (*enable_evmcs)(struct kvm_vcpu *vcpu,
 			    uint16_t *vmcs_version);
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index ee5189815a06..090955541518 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -2119,8 +2119,10 @@ static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 
 	/* Slow direct hypercalls from L2 provide a nested GPA. */
 	if (!hc->fast) {
-		hc->ingpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, hc->ingpa,
-					      PFERR_GUEST_FINAL_MASK, NULL, 0);
+		r = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, hc->ingpa,
+				      PFERR_GUEST_FINAL_MASK, &hc->ingpa, NULL, 0);
+		if (r)
+			return r;
 		if (unlikely(hc->ingpa == INVALID_GPA))
 			return HV_STATUS_INVALID_HYPERCALL_INPUT;
 	}
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 224f4c1a957e..40dcbcbae173 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -377,15 +377,17 @@ static inline bool mmu_is_nested(struct kvm_vcpu *vcpu)
 	return vcpu->arch.mmu == &vcpu->arch.guest_mmu;
 }
 
-static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu,
-				      struct kvm_pagewalk *w,
-				      gpa_t gpa, u64 access,
-				      struct x86_exception *exception,
-				      u64 pte_access)
+static inline int kvm_translate_gpa(struct kvm_vcpu *vcpu,
+				    struct kvm_pagewalk *w,
+				    gpa_t gpa, u64 access, gpa_t *translated_gpa,
+				    struct x86_exception *exception,
+				    u64 pte_access)
 {
-	if (!mmu_is_nested(vcpu) || w == &vcpu->arch.ngpa_walk)
-		return gpa;
-	return kvm_nested_call(translate_nested_gpa)(vcpu, gpa, access,
+	if (!mmu_is_nested(vcpu) || w == &vcpu->arch.ngpa_walk) {
+		*translated_gpa = gpa;
+		return 0;
+	}
+	return kvm_nested_call(translate_nested_gpa)(vcpu, gpa, access, translated_gpa,
 						     exception, pte_access);
 }
 
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c827f2e1f3d1..9567c109306b 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4474,9 +4474,8 @@ static int nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 	 * user-mode address if CR0.PG=0.  Therefore *include* ACC_USER_MASK in
 	 * the last argument to kvm_translate_gpa (which NPT does not use).
 	 */
-	*gpa = kvm_translate_gpa(vcpu, w, vaddr, access | PFERR_GUEST_FINAL_MASK,
-				 exception, ACC_ALL);
-	return 0;
+	return kvm_translate_gpa(vcpu, w, vaddr, access | PFERR_GUEST_FINAL_MASK,
+				 gpa, exception, ACC_ALL);
 }
 
 static bool mmio_info_in_cache(struct kvm_vcpu *vcpu, u64 addr, bool direct)
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index 7775ba789258..a76ec22ce3f2 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -398,9 +398,9 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 		walker->table_gfn[walker->level - 1] = table_gfn;
 		walker->pte_gpa[walker->level - 1] = pte_gpa;
 
-		real_gpa = kvm_translate_gpa(vcpu, w, gfn_to_gpa(table_gfn),
-					     nested_access | PFERR_GUEST_PAGE_MASK,
-					     &walker->fault, 0);
+		kvm_translate_gpa(vcpu, w, gfn_to_gpa(table_gfn),
+				  nested_access | PFERR_GUEST_PAGE_MASK,
+				  &real_gpa, &walker->fault, 0);
 
 		if (unlikely(real_gpa == INVALID_GPA))
 			return 0;
@@ -458,9 +458,9 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 		gfn += pse36_gfn_delta(pte);
 #endif
 
-	real_gpa = kvm_translate_gpa(vcpu, w, gfn_to_gpa(gfn),
-				     access | PFERR_GUEST_FINAL_MASK,
-				     &walker->fault, walker->pte_access);
+	kvm_translate_gpa(vcpu, w, gfn_to_gpa(gfn),
+			  access | PFERR_GUEST_FINAL_MASK, &real_gpa,
+			  &walker->fault, walker->pte_access);
 	if (real_gpa == INVALID_GPA)
 		return 0;
 
diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 3621f1a47d37..3969b769f1c1 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -165,9 +165,11 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
 	 * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated
 	 * to an L1 GPA.
 	 */
-	real_gpa = kvm_translate_gpa(vcpu, w, gfn_to_gpa(pdpt_gfn),
-				     PFERR_USER_MASK | PFERR_WRITE_MASK |
-				     PFERR_GUEST_PAGE_MASK, NULL, 0);
+	ret = kvm_translate_gpa(vcpu, w, gfn_to_gpa(pdpt_gfn),
+				PFERR_USER_MASK | PFERR_WRITE_MASK | PFERR_GUEST_PAGE_MASK,
+				&real_gpa, NULL, 0);
+	if (ret)
+		return ret;
 	if (real_gpa == INVALID_GPA)
 		return 0;
 
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 5d0a2f658365..73f1f40aad47 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -2148,24 +2148,24 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)
 	return true;
 }
 
-static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
-				      u64 access,
-				      struct x86_exception *exception,
-				      u64 pte_access)
+static int svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
+				    u64 access, gpa_t *translated_gpa,
+				    struct x86_exception *exception,
+				    u64 pte_access)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_pagewalk *w = &vcpu->arch.ngpa_walk;
-	gpa_t result;
 
-	if (WARN_ON_ONCE(!mmu_is_nested(vcpu)))
-		return gpa;
+	if (WARN_ON_ONCE(!mmu_is_nested(vcpu))) {
+		*translated_gpa = gpa;
+		return 0;
+	}
 
 	/* Non-GMET walks are always user-walks */
 	if (!(svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_GMET))
 		access |= PFERR_USER_MASK;
 
-	w->gva_to_gpa(vcpu, w, gpa, access, &result, exception);
-	return result;
+	return w->gva_to_gpa(vcpu, w, gpa, access, translated_gpa, exception);
 }
 
 struct kvm_x86_nested_ops svm_nested_ops __initdata = {
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index a666c2908889..f6994a95523a 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -7477,16 +7477,17 @@ __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *))
 }
 
 
-static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
-				      u64 access,
-				      struct x86_exception *exception,
-				      u64 pte_access)
+static int vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
+				    u64 access, gpa_t *translated_gpa,
+				    struct x86_exception *exception,
+				    u64 pte_access)
 {
 	struct kvm_pagewalk *w = &vcpu->arch.ngpa_walk;
-	gpa_t result;
 
-	if (WARN_ON_ONCE(!mmu_is_nested(vcpu)))
-		return gpa;
+	if (WARN_ON_ONCE(!mmu_is_nested(vcpu))) {
+		*translated_gpa = gpa;
+		return 0;
+	}
 
 	/*
 	 * MBEC differentiates based on the effective U/S bit of
@@ -7496,8 +7497,7 @@ static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
 	if ((pte_access & ACC_USER_MASK) && (access & PFERR_GUEST_FINAL_MASK))
 		access |= PFERR_USER_MASK;
 
-	w->gva_to_gpa(vcpu, w, gpa, access, &result, exception);
-	return result;
+	return w->gva_to_gpa(vcpu, w, gpa, access, translated_gpa, exception);
 }
 
 struct kvm_x86_nested_ops vmx_nested_ops __initdata = {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 53400fda9f44..59b38484a0d4 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7315,10 +7315,17 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 
 	*ret = -KVM_EFAULT;
 	while (offset < sizeof(clock_pairing)) {
-		gpa_t gpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, paddr + offset,
-					      PFERR_WRITE_MASK | PFERR_GUEST_FINAL_MASK, NULL, 0);
-		size_t len = min_t(size_t, sizeof(clock_pairing) - offset,
-				   PAGE_SIZE - offset_in_page(gpa));
+		gpa_t gpa;
+		size_t len;
+		int r;
+
+		r = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, paddr + offset,
+				      PFERR_WRITE_MASK | PFERR_GUEST_FINAL_MASK,
+				      &gpa, NULL, 0);
+		if (r)
+			return r;
+		len = min_t(size_t, sizeof(clock_pairing) - offset,
+			    PAGE_SIZE - offset_in_page(gpa));
 
 		if (gpa == INVALID_GPA)
 			return 0;
-- 
2.52.0

[PATCH 36/45] KVM: x86: Propagate memory faults through load_pdptrs()

[Paolo Bonzini]

load_pdptrs() returns a 0/1 value, and therefore its callers do not
distinguish memory attribute errors (including from nested GPA
translation) from other failures.  Change them to propagate
negative errno.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/regs.c       | 43 ++++++++++++++++++++++++++++-----------
 arch/x86/kvm/svm/nested.c | 10 +++++----
 arch/x86/kvm/svm/svm.c    |  4 ++--
 arch/x86/kvm/vmx/nested.c |  6 ++++--
 4 files changed, 43 insertions(+), 20 deletions(-)

diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 3969b769f1c1..b7c25b16ffca 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -176,7 +176,12 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
 	/* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */
 	ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte,
 				       cr3 & GENMASK(11, 5), sizeof(pdpte));
-	if (ret < 0)
+	if (ret == -EFAULT) {
+		kvm_prepare_memory_fault_exit(vcpu, real_gpa & PAGE_MASK, PAGE_SIZE,
+					      false, false, false);
+		return ret;
+	}
+	if (ret)
 		return 0;
 
 	for (i = 0; i < ARRAY_SIZE(pdpte); ++i) {
@@ -264,6 +269,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr0);
 int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
 {
 	unsigned long old_cr0 = kvm_read_cr0(vcpu);
+	int r;
 
 	if (!kvm_is_valid_cr0(vcpu, cr0))
 		return 1;
@@ -286,9 +292,11 @@ int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
 	}
 #endif
 	if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) &&
-	    is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) &&
-	    !load_pdptrs(vcpu, kvm_read_cr3(vcpu)))
-		return 1;
+	    is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS)) {
+		r = load_pdptrs(vcpu, kvm_read_cr3(vcpu));
+		if (r <= 0)
+			return r ?: 1;
+	}
 
 	if (!(cr0 & X86_CR0_PG) &&
 	    (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)))
@@ -315,6 +323,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
 {
 	bool skip_tlb_flush = false;
 	unsigned long pcid = 0;
+	int r;
 #ifdef CONFIG_X86_64
 	if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) {
 		skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH;
@@ -335,8 +344,11 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
 	if (!kvm_vcpu_is_legal_cr3(vcpu, cr3))
 		return 1;
 
-	if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3))
-		return 1;
+	if (is_pae_paging(vcpu)) {
+		r = load_pdptrs(vcpu, cr3);
+		if (r <= 0)
+			return r ?: 1;
+	}
 
 	if (cr3 != kvm_read_cr3(vcpu))
 		kvm_mmu_new_pgd(vcpu, cr3);
@@ -411,6 +423,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr4);
 int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 {
 	unsigned long old_cr4 = kvm_read_cr4(vcpu);
+	int r;
 
 	if (!kvm_is_valid_cr4(vcpu, cr4))
 		return 1;
@@ -420,10 +433,12 @@ int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 			return 1;
 		if ((cr4 ^ old_cr4) & X86_CR4_LA57)
 			return 1;
-	} else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE)
-		   && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS)
-		   && !load_pdptrs(vcpu, kvm_read_cr3(vcpu)))
-		return 1;
+	} else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE) &&
+		   ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS)) {
+		r = load_pdptrs(vcpu, kvm_read_cr3(vcpu));
+		if (r <= 0)
+			return r ?: 1;
+	}
 
 	if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) {
 		/* PCID can not be enabled when cr3[11:0]!=000H or EFER.LMA=0 */
@@ -579,7 +594,7 @@ static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
 static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
 			      int *mmu_reset_needed, bool update_pdptrs)
 {
-	int idx;
+	int idx, r;
 	struct desc_ptr dt;
 
 	if (!kvm_is_valid_sregs(vcpu, sregs))
@@ -616,7 +631,11 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
 	if (update_pdptrs) {
 		idx = srcu_read_lock(&vcpu->kvm->srcu);
 		if (is_pae_paging(vcpu)) {
-			load_pdptrs(vcpu, kvm_read_cr3(vcpu));
+			r = load_pdptrs(vcpu, kvm_read_cr3(vcpu));
+			if (r < 0) {
+				srcu_read_unlock(&vcpu->kvm->srcu, idx);
+				return r;
+			}
 			*mmu_reset_needed = 1;
 		}
 		srcu_read_unlock(&vcpu->kvm->srcu, idx);
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 73f1f40aad47..a512e0e8dd93 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -720,7 +720,7 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	if (reload_pdptrs && is_pae_paging(vcpu)) {
 		if (nested_npt)
 			kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
-		else if (CC(!load_pdptrs(vcpu, cr3)))
+		else if (CC(load_pdptrs(vcpu, cr3) <= 0))
 			return -EINVAL;
 	}
 
@@ -2132,11 +2132,13 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)
 		 * nGPA translation. For !nNPT, reload eagerly unless userspace
 		 * already supplied authoritative PDPTRs via KVM_SET_SREGS2.
 		 */
-		if (nested_npt_enabled(to_svm(vcpu)))
+		if (nested_npt_enabled(to_svm(vcpu))) {
 			kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
-		else if (!vcpu->arch.pdptrs_from_userspace &&
-			 CC(!load_pdptrs(vcpu, vcpu->arch.cr3)))
+		} else if (!vcpu->arch.pdptrs_from_userspace &&
+			 CC(load_pdptrs(vcpu, vcpu->arch.cr3) <= 0)) {
+			kvm_prepare_emulation_failure_exit(vcpu);
 			return false;
+		}
 	}
 
 	if (!nested_svm_merge_msrpm(vcpu))
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 2601f8359137..a8b3d57aabef 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1587,8 +1587,8 @@ static void svm_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg)
 		 * When !npt_enabled, vcpu->pdptrs[] is already available since
 		 * it is always updated per SDM when moving to CRs.
 		 */
-		if (npt_enabled)
-			load_pdptrs(vcpu, kvm_read_cr3(vcpu));
+		if (npt_enabled && load_pdptrs(vcpu, kvm_read_cr3(vcpu)) < 0)
+			kvm_vm_dead(vcpu->kvm);
 		break;
 	default:
 		KVM_BUG_ON(1, vcpu->kvm);
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index f6994a95523a..8c110f3d9cc3 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1240,7 +1240,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 	 * must not be dereferenced.
 	 */
 	if (reload_pdptrs && !nested_ept && is_pae_paging(vcpu) &&
-	    CC(!load_pdptrs(vcpu, cr3))) {
+	    CC(load_pdptrs(vcpu, cr3) <= 0)) {
 		*entry_failure_code = ENTRY_FAIL_PDPTE;
 		return -EINVAL;
 	}
@@ -3454,8 +3454,10 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
 		 * the guest CR3 might be restored prior to setting the nested
 		 * state which can lead to a load of wrong PDPTRs.
 		 */
-		if (CC(!load_pdptrs(vcpu, vcpu->arch.cr3)))
+		if (CC(load_pdptrs(vcpu, vcpu->arch.cr3) <= 0)) {
+			kvm_prepare_emulation_failure_exit(vcpu);
 			return false;
+		}
 	}
 
 
-- 
2.52.0

[PATCH 37/45] KVM: x86/mmu: Issue memory fault exit if walk failed due to memory attribute

[Paolo Bonzini]

During execution, page table entries are subject to memory protection
simply because their GPA is obtained with an EPT walk.  During emulation,
the checks need to be done by hand and result in an -EFAULT userspace
exit.

However, if a guest PTE is read-only, do not go all the way out to
userspace and just stomp out the write.  This is consistent with the way
KVM handles read-only memory slots during emulation, though admittedly
inconsistent with everything else: NPT for example always wants nested
page table entries for the guest page tables to be writable, and will
fault.

Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-27-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/include/asm/kvm_host.h |  2 +-
 arch/x86/kvm/mmu/mmu.c          |  6 +++--
 arch/x86/kvm/mmu/paging_tmpl.h  | 40 ++++++++++++++++++++++++++-------
 arch/x86/kvm/regs.h             |  6 +++++
 arch/x86/kvm/svm/nested.c       | 16 ++++++-------
 arch/x86/kvm/vmx/nested.c       |  2 +-
 6 files changed, 52 insertions(+), 20 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 7fbcf60e56cf..e33ca5ea59bd 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -486,7 +486,7 @@ struct kvm_page_fault;
  */
 struct kvm_pagewalk {
 	unsigned long (*get_guest_pgd)(struct kvm_vcpu *vcpu);
-	u64 (*get_pdptr)(struct kvm_vcpu *vcpu, int index);
+	int (*get_pdptr)(struct kvm_vcpu *vcpu, int index, u64 *pdptr);
 	void (*inject_page_fault)(struct kvm_vcpu *vcpu,
 				  struct x86_exception *fault,
 				  bool from_hardware);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 9567c109306b..c08168595732 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -4204,7 +4204,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)
 	 */
 	if (w->cpu_role.base.level == PT32E_ROOT_LEVEL) {
 		for (i = 0; i < 4; ++i) {
-			pdptrs[i] = w->get_pdptr(vcpu, i);
+			r = w->get_pdptr(vcpu, i, &pdptrs[i]);
+			if (r)
+				return r;
 			if (!(pdptrs[i] & PT_PRESENT_MASK))
 				continue;
 
@@ -6211,7 +6213,7 @@ static void init_kvm_page_walk(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 
 	w->cpu_role.as_u64   = cpu_role.as_u64;
 	w->inject_page_fault = kvm_inject_page_fault;
-	w->get_pdptr         = kvm_pdptr_read;
+	w->get_pdptr         = kvm_pdptr_get;
 	w->get_guest_pgd     = get_guest_cr3;
 
 	if (!is_cr0_pg(w))
diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h
index a76ec22ce3f2..ddab65e5689d 100644
--- a/arch/x86/kvm/mmu/paging_tmpl.h
+++ b/arch/x86/kvm/mmu/paging_tmpl.h
@@ -89,6 +89,7 @@ struct guest_walker {
 	unsigned int pt_access[PT_MAX_FULL_LEVELS];
 	unsigned int pte_access;
 	gfn_t gfn;
+	int error;
 	struct x86_exception fault;
 };
 
@@ -342,6 +343,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 
 	trace_kvm_mmu_pagetable_walk(addr, access);
 retry_walk:
+	walker->error = 0;
 	walker->level = w->cpu_role.base.level;
 	pte           = kvm_mmu_get_guest_pgd(vcpu, w);
 	have_ad       = PT_HAVE_ACCESSED_DIRTY(w);
@@ -349,7 +351,9 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 #if PTTYPE == 64
 	walk_nx_mask = 1ULL << PT64_NX_SHIFT;
 	if (walker->level == PT32E_ROOT_LEVEL) {
-		pte = w->get_pdptr(vcpu, (addr >> 30) & 3);
+		walker->error = w->get_pdptr(vcpu, (addr >> 30) & 3, &pte);
+		if (walker->error)
+			goto error;
 		trace_kvm_mmu_paging_element(pte, walker->level);
 		if (!FNAME(is_present_gpte)(w, pte))
 			goto error;
@@ -398,9 +402,11 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 		walker->table_gfn[walker->level - 1] = table_gfn;
 		walker->pte_gpa[walker->level - 1] = pte_gpa;
 
-		kvm_translate_gpa(vcpu, w, gfn_to_gpa(table_gfn),
-				  nested_access | PFERR_GUEST_PAGE_MASK,
-				  &real_gpa, &walker->fault, 0);
+		walker->error = kvm_translate_gpa(vcpu, w, gfn_to_gpa(table_gfn),
+						  nested_access | PFERR_GUEST_PAGE_MASK,
+						  &real_gpa, &walker->fault, 0);
+		if (walker->error)
+			return 0;
 
 		if (unlikely(real_gpa == INVALID_GPA))
 			return 0;
@@ -414,6 +420,16 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 		if (unlikely(kvm_is_error_hva(host_addr)))
 			goto error;
 
+		if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gpa_to_gfn(real_gpa))) {
+			walker->error = -EFAULT;
+			kvm_prepare_memory_fault_exit(vcpu, real_gpa & PAGE_MASK,
+						      PAGE_SIZE, false, false, false);
+			goto error;
+		}
+
+		if (!kvm_mem_attributes_may_write_gfn(vcpu->kvm, gpa_to_gfn(real_gpa)))
+			walker->pte_writable[walker->level - 1] = false;
+
 		ptep_user = (pt_element_t __user *)((void *)host_addr + offset);
 		if (unlikely(get_user(pte, ptep_user)))
 			goto error;
@@ -458,9 +474,11 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 		gfn += pse36_gfn_delta(pte);
 #endif
 
-	kvm_translate_gpa(vcpu, w, gfn_to_gpa(gfn),
-			  access | PFERR_GUEST_FINAL_MASK, &real_gpa,
-			  &walker->fault, walker->pte_access);
+	walker->error = kvm_translate_gpa(vcpu, w, gfn_to_gpa(gfn),
+					  access | PFERR_GUEST_FINAL_MASK, &real_gpa,
+					  &walker->fault, walker->pte_access);
+	if (walker->error)
+		return 0;
 	if (real_gpa == INVALID_GPA)
 		return 0;
 
@@ -823,6 +841,9 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	 * The page is not mapped by the guest.  Let the guest handle it.
 	 */
 	if (!r) {
+		if (walker.error)
+			return walker.error;
+
 		if (!fault->prefetch)
 			__kvm_inject_emulated_page_fault(vcpu, &walker.fault, true);
 
@@ -920,8 +941,11 @@ static int FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 	if (r) {
 		*gpa = gfn_to_gpa(walker.gfn);
 		*gpa |= addr & ~PAGE_MASK;
-	} else if (exception)
+	} else if (walker.error) {
+		return walker.error;
+	} else if (exception) {
 		*exception = walker.fault;
+	}
 
 	return 0;
 }
diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h
index 97c17b120a57..31f4d3f6ea42 100644
--- a/arch/x86/kvm/regs.h
+++ b/arch/x86/kvm/regs.h
@@ -319,6 +319,12 @@ static inline u64 kvm_pdptr_read(struct kvm_vcpu *vcpu, int index)
 	return vcpu->arch.pdptrs[index];
 }
 
+static inline int kvm_pdptr_get(struct kvm_vcpu *vcpu, int index, u64 *pdptr)
+{
+	*pdptr = kvm_pdptr_read(vcpu, index);
+	return 0;
+}
+
 static inline void kvm_pdptr_write(struct kvm_vcpu *vcpu, int index, u64 value)
 {
 	vcpu->arch.pdptrs[index] = value;
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index a512e0e8dd93..4d76e8880d99 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -71,22 +71,22 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 	nested_svm_vmexit(svm);
 }
 
-static u64 nested_svm_get_tdp_pdptr(struct kvm_vcpu *vcpu, int index)
+static int nested_svm_get_tdp_pdptr(struct kvm_vcpu *vcpu, int index, u64 *pdptr)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	u64 cr3 = svm->nested.ctl.nested_cr3;
-	u64 pdpte;
-	int ret;
+	int r;
 
 	/*
 	 * Note, nCR3 is "assumed" to be 32-byte aligned, i.e. the CPU ignores
 	 * nCR3[4:0] when loading PDPTEs from memory.
 	 */
-	ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(cr3), &pdpte,
-				       (cr3 & GENMASK(11, 5)) + index * 8, 8);
-	if (ret)
-		return 0;
-	return pdpte;
+	r = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(cr3), pdptr,
+				     (cr3 & GENMASK(11, 5)) + index * 8, 8);
+	if (r == -EFAULT)
+		kvm_prepare_memory_fault_exit(vcpu, cr3 & PAGE_MASK, PAGE_SIZE,
+					      false, false, false);
+	return r;
 }
 
 static unsigned long nested_svm_get_tdp_cr3(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 8c110f3d9cc3..2ac4a366ace4 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -520,7 +520,7 @@ static void nested_ept_init_mmu_context(struct kvm_vcpu *vcpu)
 	vcpu->arch.mmu = &vcpu->arch.guest_mmu;
 	nested_ept_new_eptp(vcpu);
 	vcpu->arch.ngpa_walk.get_guest_pgd     = nested_ept_get_eptp;
-	vcpu->arch.ngpa_walk.get_pdptr       = kvm_pdptr_read;
+	vcpu->arch.ngpa_walk.get_pdptr       = kvm_pdptr_get;
 
 	vcpu->arch.ngpa_walk.inject_page_fault = nested_ept_inject_page_fault;
 }
-- 
2.52.0

[PATCH 38/45] KVM: x86: mark VM as dead after KVM_SET_SREGS fails to load PDPTRs

[Paolo Bonzini]

If loading PDPTRs from guest memory fails, the KVM_SET_SREGS ioctl
returns before resetting the MMU context and with stale PDPTRs.  Mark the
VM dead on the error path, so that the vCPU cannot run with inconsistent
state; userspace should anyway use KVM_SET_SREGS2, which provides the
cached PDPTRs and avoids the guest-memory read.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/regs.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index b7c25b16ffca..3a11a6d06889 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -634,6 +634,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
 			r = load_pdptrs(vcpu, kvm_read_cr3(vcpu));
 			if (r < 0) {
 				srcu_read_unlock(&vcpu->kvm->srcu, idx);
+				kvm_vm_dead(vcpu->kvm);
 				return r;
 			}
 			*mmu_reset_needed = 1;
-- 
2.52.0

[PATCH 39/45] KVM: VMX: mark VM dead on shadow VMCS or eVMCS access failures

[Paolo Bonzini]

KVM ignores failures when reading or writing L1's in-memory VMCSs (shadow
VMCS, eVMCS), allowing execution to continue with stale shadow VMCS values
or silently losing updates.  Check both the HVA cache initialization
and the cached memory accesses, and mark the VM dead on failure; these
failures can only occur when guest memory permissions deny the access,
and terminating the VM is safe (though it precludes using memory attributes
as a replacement for e.g. userfaultfd, and only leaves the feature usable
for security purposes).

Likewise, treat a failed read of an enabled VP assist page as fatal too,
otherwise KVM continues by disabling enlightened VMCS.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/vmx/hyperv.c |  7 ++++++-
 arch/x86/kvm/vmx/nested.c | 21 ++++++++++++++-------
 2 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/arch/x86/kvm/vmx/hyperv.c b/arch/x86/kvm/vmx/hyperv.c
index fa41d036acd4..746540e503e3 100644
--- a/arch/x86/kvm/vmx/hyperv.c
+++ b/arch/x86/kvm/vmx/hyperv.c
@@ -18,9 +18,14 @@ u64 nested_get_evmptr(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 
-	if (unlikely(kvm_hv_get_assist_page(vcpu)))
+	if (unlikely(!kvm_hv_assist_page_enabled(vcpu)))
 		return EVMPTR_INVALID;
 
+	if (unlikely(kvm_hv_get_assist_page(vcpu))) {
+		kvm_vm_dead(vcpu->kvm);
+		return EVMPTR_INVALID;
+	}
+
 	if (unlikely(!hv_vcpu->vp_assist_page.enlighten_vmentry))
 		return EVMPTR_INVALID;
 
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 2ac4a366ace4..bde151de0bf7 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -861,10 +861,13 @@ static void nested_cache_shadow_vmcs12(struct kvm_vcpu *vcpu,
 	if (ghc->gpa != vmcs12->vmcs_link_pointer &&
 	    kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc,
 				      vmcs12->vmcs_link_pointer, VMCS12_SIZE))
-		return;
+		goto error;
 
-	kvm_read_guest_cached(vcpu->kvm, ghc, get_shadow_vmcs12(vcpu),
-			      VMCS12_SIZE);
+	if (!kvm_read_guest_cached(vcpu->kvm, ghc, get_shadow_vmcs12(vcpu),
+				   VMCS12_SIZE))
+		return;
+error:
+	kvm_vm_dead(vcpu->kvm);
 }
 
 static void nested_flush_cached_shadow_vmcs12(struct kvm_vcpu *vcpu,
@@ -880,10 +883,13 @@ static void nested_flush_cached_shadow_vmcs12(struct kvm_vcpu *vcpu,
 	if (ghc->gpa != vmcs12->vmcs_link_pointer &&
 	    kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc,
 				      vmcs12->vmcs_link_pointer, VMCS12_SIZE))
-		return;
+		goto error;
 
-	kvm_write_guest_cached(vcpu->kvm, ghc, get_shadow_vmcs12(vcpu),
-			       VMCS12_SIZE);
+	if (!kvm_write_guest_cached(vcpu->kvm, ghc, get_shadow_vmcs12(vcpu),
+				    VMCS12_SIZE))
+		return;
+error:
+	kvm_vm_dead(vcpu->kvm);
 }
 
 /*
@@ -5091,7 +5097,8 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 		 * do that when something is forcing L2->L1 exit prior to
 		 * the first L2 run.
 		 */
-		(void)nested_get_evmcs_page(vcpu);
+		if (!nested_get_evmcs_page(vcpu))
+			kvm_vm_dead(vcpu->kvm);
 #endif
 	}
 
-- 
2.52.0

[PATCH 40/45] KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock

[Paolo Bonzini]

Ensure that mmu_invalidate_in_progress is not cleared until
kvm_arch_post_set_memory_attributes() returns.

kvm_arch_post_set_memory_attributes() runs with the MMU lock taken
exclusively, so a concurrent page fault would be blocked anyway, and
delaying the clearing makes it possible to use cond_resched() within
kvm_arch_post_set_memory_attributes().

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-28-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 virt/kvm/kvm_main.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 326bb8e144b7..2d5d45e4ee71 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2630,7 +2630,7 @@ static bool kvm_range_memory_attributes_need_sync(struct kvm *kvm,
 	return kvm_range_has_any_vm_memory_attributes(kvm, start, end, mask);
 }
 
-static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
+static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm,
 						 struct kvm_mmu_notifier_range *range)
 {
 	struct kvm_gfn_range gfn_range;
@@ -2680,6 +2680,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 
 	if (found_memslot)
 		KVM_MMU_UNLOCK(kvm);
+
+	return found_memslot;
 }
 
 static bool kvm_pre_set_vm_memory_attributes(struct kvm *kvm,
@@ -2717,10 +2719,11 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		.end = end,
 		.arg.attributes = attributes,
 		.handler = kvm_arch_post_set_vm_memory_attributes,
-		.on_lock = kvm_mmu_invalidate_end,
+		.on_lock = (void *)kvm_null_fn,
 		.may_block = true,
 	};
 	bool sync = false;
+	bool found_memslot;
 	unsigned long i;
 	void *entry;
 	int r = 0;
@@ -2752,7 +2755,7 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		cond_resched();
 	}
 
-	kvm_handle_gfn_range(kvm, &pre_set_range);
+	found_memslot = kvm_handle_gfn_range(kvm, &pre_set_range);
 
 	for (i = start; i < end; i++) {
 		r = xa_err(xa_store(&kvm->mem_attrs.array, i, entry,
@@ -2765,7 +2768,12 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	smp_store_release(&kvm->mem_attrs.generation,
 			  kvm->mem_attrs.generation + 1);
 
-	kvm_handle_gfn_range(kvm, &post_set_range);
+	if (found_memslot) {
+		kvm_handle_gfn_range(kvm, &post_set_range);
+		KVM_MMU_LOCK(kvm);
+		kvm_mmu_invalidate_end(kvm);
+		KVM_MMU_UNLOCK(kvm);
+	}
 
 out_unlock:
 	mutex_unlock(&kvm->slots_lock);
-- 
2.52.0

[PATCH 41/45] KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs

[Paolo Bonzini]

Note that the zapping of obsolete roots needs to happen after
mmu_lock is dropped, so introduce a separate kvm_arch callback.
The two map neatly to the "front" and "back" halves of
kvm_mmu_zap_all_fast().

Co-developed-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-29-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/mmu/mmu.c   | 30 ++++++++++++++++++++++++++++++
 include/linux/kvm_host.h |  1 +
 virt/kvm/kvm_main.c      |  1 +
 3 files changed, 32 insertions(+)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c08168595732..1cee8c3f07af 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8359,6 +8359,7 @@ bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
 {
 	unsigned long attrs = range->arg.attributes;
 	struct kvm_memory_slot *slot = range->slot;
+	struct kvm_mmu_page *sp;
 	int level;
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
@@ -8414,9 +8415,38 @@ bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
 				hugepage_set_mixed(slot, gfn, level);
 		}
 	}
+
+	/*
+	 * There are special considerations when applying an memory protection
+	 * attibute against a page containing guest PTEs, because virtual
+	 * addresses that requires traversing that page need to fault.
+	 *
+	 * When using shadow paging, the PTEs that the CPU actually walks
+	 * do not have any information about attributes (unlike TDP where
+	 * the attributes become part of the TDP permission bits).  Since
+	 * this should be a rare occasion, just invalidate all the mmu roots
+	 * if shadow paging is in use and attributes do cover a guest PTE.
+	 */
+	for (gfn_t gfn = range->start; gfn < range->end; gfn++) {
+		for_each_gfn_valid_sp_with_gptes(kvm, sp, gfn) {
+			__kvm_mmu_zap_all_fast_front_half(kvm);
+			return false;
+		}
+	}
 	return false;
 }
 
+void kvm_arch_post_set_vm_memory_attributes_unlocked(struct kvm *kvm)
+{
+	/*
+	 * This may also be called if no zap was done in
+	 * kvm_arch_post_set_memory_attributes().  It is not an issue because
+	 * __kvm_mmu_zap_all_fast_back_half() is cheap when it does not do
+	 * anything.
+	 */
+	 __kvm_mmu_zap_all_fast_back_half(kvm);
+}
+
 void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
 					    struct kvm_memory_slot *slot)
 {
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 4128cdb71869..c5defda4d9d4 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2624,6 +2624,7 @@ bool kvm_arch_pre_set_vm_memory_attributes(struct kvm *kvm,
 					   struct kvm_gfn_range *range);
 bool kvm_arch_post_set_vm_memory_attributes(struct kvm *kvm,
 					    struct kvm_gfn_range *range);
+void kvm_arch_post_set_vm_memory_attributes_unlocked(struct kvm *kvm);
 bool kvm_range_has_any_vm_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 					    unsigned long mask);
 u64 kvm_supported_vm_mem_attributes(struct kvm *kvm);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 2d5d45e4ee71..061e952ea080 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2773,6 +2773,7 @@ static int kvm_set_vm_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		KVM_MMU_LOCK(kvm);
 		kvm_mmu_invalidate_end(kvm);
 		KVM_MMU_UNLOCK(kvm);
+		kvm_arch_post_set_vm_memory_attributes_unlocked(kvm);
 	}
 
 out_unlock:
-- 
2.52.0

[PATCH 42/45] KVM: x86: selftests: Introduce memory protection attributes test

[Paolo Bonzini]

Introduce architecture-independent tests for memory attributes, using
small architecture-specific snippets to handle the execution tests.

Co-developed-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-30-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tools/include/uapi/linux/kvm.h                |   3 +
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../testing/selftests/kvm/include/kvm_util.h  |  18 +-
 .../testing/selftests/kvm/memory_attributes.c | 344 ++++++++++++++++++
 .../selftests/kvm/x86/memory_attributes.c     |  43 +++
 5 files changed, 405 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/memory_attributes.c
 create mode 100644 tools/testing/selftests/kvm/x86/memory_attributes.c

diff --git a/tools/include/uapi/linux/kvm.h b/tools/include/uapi/linux/kvm.h
index 419011097fa8..9bd6488adc62 100644
--- a/tools/include/uapi/linux/kvm.h
+++ b/tools/include/uapi/linux/kvm.h
@@ -1649,6 +1649,9 @@ struct kvm_memory_attributes {
 	__u64 flags;
 };
 
+#define KVM_MEMORY_ATTRIBUTE_NR		       (1ULL << 0)
+#define KVM_MEMORY_ATTRIBUTE_NW		       (1ULL << 1)
+#define KVM_MEMORY_ATTRIBUTE_NX		       (1ULL << 2)
 #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
 
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 478b85ceb7fb..6e123f690b43 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -173,6 +173,7 @@ TEST_GEN_PROGS_x86 += rseq_test
 TEST_GEN_PROGS_x86 += steal_time
 TEST_GEN_PROGS_x86 += system_counter_offset_test
 TEST_GEN_PROGS_x86 += pre_fault_memory_test
+TEST_GEN_PROGS_x86 += memory_attributes
 
 # Compiled outputs used by test targets
 TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h
index 7e480c034a87..51b173105d3c 100644
--- a/tools/testing/selftests/kvm/include/kvm_util.h
+++ b/tools/testing/selftests/kvm/include/kvm_util.h
@@ -473,17 +473,27 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0)
 int kvm_gpa_to_guest_memfd(struct kvm_vm *vm, gpa_t gpa, off_t *fd_offset,
 			   size_t *nr_bytes);
 
-static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
-					    size_t size, u64 attributes)
+static inline int __vm_set_memory_attributes(struct kvm_vm *vm, u64 gpa,
+					     u64 size, u64 attributes,
+					     u64 flags)
 {
 	struct kvm_memory_attributes attr = {
 		.attributes = attributes,
 		.address = gpa,
 		.size = size,
-		.flags = 0,
+		.flags = flags,
 	};
 
-	vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
+	return __vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
+}
+
+static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
+					    u64 size, u64 attributes)
+{
+	int rc;
+
+	rc = __vm_set_memory_attributes(vm, gpa, size, attributes, 0);
+	TEST_ASSERT_VM_VCPU_IOCTL(!rc, KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
 }
 
 static inline int __gmem_set_memory_attributes(int fd, u64 offset,
diff --git a/tools/testing/selftests/kvm/memory_attributes.c b/tools/testing/selftests/kvm/memory_attributes.c
new file mode 100644
index 000000000000..81ec59382f55
--- /dev/null
+++ b/tools/testing/selftests/kvm/memory_attributes.c
@@ -0,0 +1,344 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2024, Amazon.com, Inc. or its affiliates. All Rights Reserved
+ *
+ * Test for KVM_MEMORY_ATTRIBUTES
+ */
+#include <pthread.h>
+#include <signal.h>
+
+#include "test_util.h"
+#include "ucall_common.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "hyperv.h"
+#include "apic.h"
+#include "asm/pvclock-abi.h"
+
+#define KVM_MEMORY_ATTRIBUTE_NO_ACCESS                       \
+	(KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW | \
+	 KVM_MEMORY_ATTRIBUTE_NX)
+
+#define TEST_MEM_SLOT	1
+#define TEST_MEM_GPA	0x80000000
+
+#define MMIO_GPA	0x700000000
+#define MMIO_GVA	MMIO_GPA
+
+enum {
+	TEST_OP_NOP,
+	TEST_OP_READ,
+	TEST_OP_WRITE,
+	TEST_OP_EXEC,
+	TEST_OP_EXIT,
+};
+
+const char *test_op_names[] =
+{
+	[TEST_OP_READ] = "Read",
+	[TEST_OP_WRITE] = "Write",
+	[TEST_OP_EXEC] = "Exec",
+	[TEST_OP_EXIT] = "Exit",
+};
+
+struct test_data {
+	uint8_t op;
+	int stage;
+	gva_t vaddr;
+
+	struct kvm_vcpu *vcpu;
+};
+
+static struct test_data *test_data;
+
+static uint64_t arch_controlled_read(gva_t addr);
+static void arch_controlled_write(gva_t addr, uint64_t val);
+static void arch_controlled_exec(gva_t addr);
+static void arch_write_return_insn(struct kvm_vm *vm, gpa_t vaddr);
+
+static void guest_code(void *data)
+{
+	struct test_data *test_data = data;
+	int stage = 1;
+
+	while (true) {
+		gva_t vaddr = READ_ONCE(test_data->vaddr);
+
+		switch(READ_ONCE(test_data->op)) {
+		case TEST_OP_READ:
+			(void) arch_controlled_read(vaddr);
+			GUEST_SYNC(stage++);
+			break;
+		case TEST_OP_WRITE:
+			arch_controlled_write(vaddr, 1);
+			GUEST_SYNC(stage++);
+			break;
+		case TEST_OP_EXEC:
+			arch_controlled_exec(vaddr);
+			GUEST_SYNC(stage++);
+			break;
+		default:
+			goto exit;
+		};
+	}
+
+exit:
+	GUEST_DONE();
+}
+
+static void vcpu_run_and_inc_stage(struct kvm_vcpu *vcpu)
+{
+	struct ucall uc;
+
+	vcpu_run(vcpu);
+
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	switch (get_ucall(vcpu, &uc)) {
+	case UCALL_SYNC:
+		TEST_ASSERT(uc.args[1] == test_data->stage,
+			    "Unexpected stage: %ld (%d expected)",
+			    uc.args[1], test_data->stage);
+		break;
+	case UCALL_ABORT:
+		REPORT_GUEST_ASSERT(uc);
+		/* NOT REACHED */
+	default:
+		TEST_FAIL("Unknown ucall %lu", uc.cmd);
+	}
+
+	test_data->stage++;
+}
+
+static void test_page_restricted(struct kvm_vcpu *vcpu, int op,
+				 gva_t vaddr, gpa_t fault_paddr,
+				 uint64_t fault_reason)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	int rc;
+
+	test_data->op = op;
+	test_data->vaddr = vaddr;
+
+	rc = _vcpu_run(vcpu);
+	TEST_ASSERT(rc == -1 && errno == EFAULT,
+		    "KVM_RUN IOCTL didn't return EFAULT on %s, rc %d, errno %d",
+		    test_op_names[op], rc, errno);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MEMORY_FAULT);
+	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, fault_paddr);
+	TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, fault_reason);
+	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, vm->page_size);
+}
+
+static void test_page_accessible(struct kvm_vcpu *vcpu, int op, gva_t vaddr)
+{
+	test_data->op = op;
+	test_data->vaddr = vaddr;
+	vcpu_run_and_inc_stage(vcpu);
+}
+
+#include "x86/memory_attributes.c"
+
+/*
+ * We want to test the following cases:
+ * - Sucessful access to GPAs backed by memory attributes (for ex. read access
+ *   on an read-only page).
+ * - First fault after setting memory attributes, with unpopulated SPTEs/EPTS.
+ * - Fault caused by an SPTE/EPT reflecting the memory attributes.
+ *
+ * The list of ops below tests the 3 situations for each memory attribute
+ * combination.
+ */
+const struct memory_access {
+	const char *name;
+	uint64_t attrs;
+	int ops[5];
+} access_array[] = {
+	{ "all allowed", 0, { TEST_OP_READ, TEST_OP_WRITE, TEST_OP_EXEC } },
+	{ "no write (unmapped)", KVM_MEMORY_ATTRIBUTE_NW,
+	  { TEST_OP_WRITE, TEST_OP_READ, TEST_OP_EXEC, TEST_OP_WRITE } },
+	{ "no write (mapped)", KVM_MEMORY_ATTRIBUTE_NW,
+	  { TEST_OP_READ, TEST_OP_WRITE, TEST_OP_READ, TEST_OP_EXEC, TEST_OP_WRITE } },
+	{ "no exec (unmapped)", KVM_MEMORY_ATTRIBUTE_NX,
+	  { TEST_OP_EXEC, TEST_OP_READ, TEST_OP_WRITE, TEST_OP_EXEC } },
+	{ "no exec (mapped)", KVM_MEMORY_ATTRIBUTE_NX,
+	  { TEST_OP_READ, TEST_OP_EXEC, TEST_OP_READ, TEST_OP_WRITE, TEST_OP_EXEC } },
+	{ "read only", KVM_MEMORY_ATTRIBUTE_NW | KVM_MEMORY_ATTRIBUTE_NX,
+	  { TEST_OP_EXEC, TEST_OP_WRITE, TEST_OP_READ, TEST_OP_WRITE,
+	    TEST_OP_EXEC } },
+	{ "no access (map on read)", KVM_MEMORY_ATTRIBUTE_NO_ACCESS,
+	  { TEST_OP_READ, TEST_OP_WRITE, TEST_OP_EXEC } },
+	{ "no access (map on write)", KVM_MEMORY_ATTRIBUTE_NO_ACCESS,
+	  { TEST_OP_WRITE, TEST_OP_READ, TEST_OP_EXEC } },
+	{ "no access (map on exec)", KVM_MEMORY_ATTRIBUTE_NO_ACCESS,
+	  { TEST_OP_EXEC, TEST_OP_READ, TEST_OP_WRITE } },
+	/* Verify everything is back to normal */
+	{ "all allowed (2)", 0, { TEST_OP_READ, TEST_OP_WRITE, TEST_OP_EXEC } },
+};
+
+static void test_page_access(struct kvm_vcpu *vcpu, gva_t vaddr,
+			     uint64_t attrs, const int ops[])
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gpa_t paddr = addr_gva2gpa(vm, vaddr);
+
+	for (int i = 0; i < ARRAY_SIZE(access_array[0].ops); i++) {
+		int op = ops[i];
+
+		if (op == TEST_OP_NOP)
+			continue;
+
+		/*
+		 * We're about to have the guest jump into 'vaddr', make it a
+		 * 'ret' instruction so it returns right away.
+		 */
+		if (op == TEST_OP_EXEC)
+			arch_write_return_insn(vm, paddr);
+
+		vm_set_memory_attributes(vm, paddr, vm->page_size, attrs);
+
+		/*
+		 * Attributes are negated, a match means the operation should
+		 * fail.
+		 */
+		if (attrs & BIT_ULL(op - 1)) {
+			test_page_restricted(vcpu, op, vaddr, paddr, BIT(op - 1));
+			vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+		}
+
+		test_page_accessible(vcpu, op, vaddr);
+	}
+
+}
+
+static void test_memory_access(struct kvm_vcpu *vcpu, gva_t test_vm_vaddr,
+			       size_t size)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gpa_t test_vm_paddr = addr_gva2gpa(vm, test_vm_vaddr);
+
+	printf("gva %lx gpa %lx\n", test_vm_vaddr, test_vm_paddr);
+	for (size_t i = 0; i < ARRAY_SIZE(access_array); i++) {
+		uint64_t attrs = access_array[i].attrs;
+
+		printf("starting %s test...\n", access_array[i].name);
+		vm_set_memory_attributes(vm, test_vm_paddr, size, attrs);
+
+		for (gva_t vaddr = test_vm_vaddr;
+		     vaddr < test_vm_vaddr + size; vaddr += PAGE_SIZE) {
+			test_page_access(vcpu, vaddr, attrs, access_array[i].ops);
+		}
+	}
+}
+
+static void test_input_validation(struct kvm_vm *vm)
+{
+	uint64_t flags, gpa = 0, size = 0, attrs = 0;
+	int rc;
+
+	/* 'flags' is unsupported */
+	flags = BIT(0);
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+
+	/* 'size' can't be 0 */
+	flags = 0;
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+
+	/* 'gpa' shouldn't overflow */
+	gpa = 0ULL - vm->page_size;
+	size = vm->page_size;
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+
+	/* 'gpa' should be page aligned */
+	gpa = 1;
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+
+	/* 'size' should be page aligned */
+	gpa = 0;
+	size = 1;
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+
+	/* exec mappings require read access */
+	size = vm->page_size;
+	attrs = KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW;
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+
+	/* write mappings require read access */
+	size = vm->page_size;
+	attrs = KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NX;
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+
+	/* private mappings are incompatible with access restrictions */
+	attrs = KVM_MEMORY_ATTRIBUTE_NW | KVM_MEMORY_ATTRIBUTE_PRIVATE;
+	rc = __vm_set_memory_attributes(vm, gpa, size, attrs, flags);
+	TEST_ASSERT_VM_VCPU_IOCTL(rc == -1 && errno == EINVAL,
+				  KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
+}
+
+static void test_finalize(struct kvm_vcpu *vcpu)
+{
+	test_data->op = TEST_OP_EXIT;
+	vcpu_run(vcpu);
+	TEST_ASSERT_EQ(get_ucall(vcpu, NULL), UCALL_DONE);
+}
+
+static struct test_data *init_test_data(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gva_t test_data_vm_vaddr;
+
+	test_data_vm_vaddr = vm_alloc_page(vm);
+	vcpu_args_set(vcpu, 1, test_data_vm_vaddr);
+
+	test_data = addr_gva2hva(vm, test_data_vm_vaddr);
+	test_data->stage = 1;
+	test_data->vcpu = vcpu;
+
+	return test_data;
+}
+
+int main(int argc, char *argv[])
+{
+	uint32_t guest_page_size = vm_guest_mode_params[VM_MODE_DEFAULT].page_size;
+	unsigned int ptes_per_page = guest_page_size / 8;
+	size_t pages = ptes_per_page * 2; /* 2 huge-pages */
+	size_t size = guest_page_size * pages;
+	struct kvm_vcpu *vcpu;
+	gva_t test_mem;
+	struct kvm_vm *vm;
+
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_MEMORY_ATTRIBUTES) &
+		     KVM_MEMORY_ATTRIBUTE_NO_ACCESS);
+
+	vm = __vm_create_with_one_vcpu(&vcpu, pages, guest_code);
+
+	/* add a few extra pages for use by tests */
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+				    TEST_MEM_GPA, TEST_MEM_SLOT,
+				    pages + 100, 0);
+	vm->memslots[MEM_REGION_TEST_DATA] = TEST_MEM_SLOT;
+
+	test_mem = vm_alloc_pages(vm, pages);
+	virt_map(vcpu->vm, MMIO_GVA, MMIO_GPA, 1);
+	test_data = init_test_data(vcpu);
+
+	test_input_validation(vm);
+	test_memory_access(vcpu, test_mem, size);
+	test_finalize(vcpu);
+
+	kvm_vm_free(vm);
+	return 0;
+}
diff --git a/tools/testing/selftests/kvm/x86/memory_attributes.c b/tools/testing/selftests/kvm/x86/memory_attributes.c
new file mode 100644
index 000000000000..2e1148f5146d
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/memory_attributes.c
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2024, Amazon.com, Inc. or its affiliates. All Rights Reserved
+ *
+ * Test for KVM_MEMORY_ATTRIBUTES
+ */
+#include "kvm_util.h"
+#include "apic.h"
+
+uint64_t arch_controlled_read(gva_t addr)
+{
+	uint64_t val;
+
+	asm volatile("mov %[addr], %%rax \n\r"
+		     "mov (%%rax), %[val] \n\r"
+		     : [val] "=r" (val)
+		     : [addr] "m"(addr)
+		     : "memory", "rax");
+
+	return val;
+}
+
+void arch_controlled_write(gva_t addr, uint64_t val)
+{
+	asm volatile("mov %[addr], %%rax \n\r"
+		     "mov %[val], %%rbx \n\r"
+		     "mov %%rbx, (%%rax) \n\r"
+		     :: [addr] "m" (addr), [val] "m" (val)
+		     : "memory", "rax", "rbx");
+}
+
+void arch_controlled_exec(gva_t addr)
+{
+	asm volatile("mov %[addr], %%rax \n\r"
+		     "call *%%rax \n\t"
+		     :: [addr] "m"(addr)
+		     : "memory", "rax");
+}
+
+void arch_write_return_insn(struct kvm_vm *vm, gpa_t vaddr)
+{
+	memset(addr_gpa2hva(vm, vaddr), 0xc3, 1);
+}
-- 
2.52.0

[PATCH 43/45] KVM: x86: selftests: Introduce memory attributes PTE test

[Paolo Bonzini]

Add more memory attributes tests to check that memory protection
restricts accesses even when installed on a page that holds guest page
table entries. Notably two cases are taken into account:
- The page is made non-accesible. In such case the next access to a
  virtual memory address translated by that paging structure should
  fault.
- The page is made read-only. In such case the next access to a virtual
  memory address translated by that paging structure should either fault,
  or succeed yet not perform any writes into the PTE (accessed and
  dirty bits).

Co-developed-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-31-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 .../testing/selftests/kvm/memory_attributes.c |  39 ++++-
 .../selftests/kvm/x86/memory_attributes.c     | 144 +++++++++++++++++-
 2 files changed, 177 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/kvm/memory_attributes.c b/tools/testing/selftests/kvm/memory_attributes.c
index 81ec59382f55..c5305b4f3dbe 100644
--- a/tools/testing/selftests/kvm/memory_attributes.c
+++ b/tools/testing/selftests/kvm/memory_attributes.c
@@ -25,11 +25,16 @@
 #define MMIO_GPA	0x700000000
 #define MMIO_GVA	MMIO_GPA
 
+#define PT_WRITABLE_MASK	BIT_ULL(1)
+#define PT_ACCESSED_MASK	BIT_ULL(5)
+#define PTE_VADDR		0x1000000000
+
 enum {
 	TEST_OP_NOP,
 	TEST_OP_READ,
 	TEST_OP_WRITE,
 	TEST_OP_EXEC,
+	TEST_OP_INVPLG,
 	TEST_OP_EXIT,
 };
 
@@ -38,6 +43,7 @@ const char *test_op_names[] =
 	[TEST_OP_READ] = "Read",
 	[TEST_OP_WRITE] = "Write",
 	[TEST_OP_EXEC] = "Exec",
+	[TEST_OP_INVPLG] = "Invplg",
 	[TEST_OP_EXIT] = "Exit",
 };
 
@@ -45,6 +51,7 @@ struct test_data {
 	uint8_t op;
 	int stage;
 	gva_t vaddr;
+	uint64_t expected_val;
 
 	struct kvm_vcpu *vcpu;
 };
@@ -54,7 +61,8 @@ static struct test_data *test_data;
 static uint64_t arch_controlled_read(gva_t addr);
 static void arch_controlled_write(gva_t addr, uint64_t val);
 static void arch_controlled_exec(gva_t addr);
-static void arch_write_return_insn(struct kvm_vm *vm, gpa_t vaddr);
+static void arch_write_return_insn(struct kvm_vm *vm, gpa_t paddr);
+static bool arch_test_op(struct test_data *test_data);
 
 static void guest_code(void *data)
 {
@@ -62,6 +70,7 @@ static void guest_code(void *data)
 	int stage = 1;
 
 	while (true) {
+		uint64_t expected_val = READ_ONCE(test_data->expected_val);
 		gva_t vaddr = READ_ONCE(test_data->vaddr);
 
 		switch(READ_ONCE(test_data->op)) {
@@ -70,7 +79,7 @@ static void guest_code(void *data)
 			GUEST_SYNC(stage++);
 			break;
 		case TEST_OP_WRITE:
-			arch_controlled_write(vaddr, 1);
+			arch_controlled_write(vaddr, expected_val);
 			GUEST_SYNC(stage++);
 			break;
 		case TEST_OP_EXEC:
@@ -78,7 +87,10 @@ static void guest_code(void *data)
 			GUEST_SYNC(stage++);
 			break;
 		default:
-			goto exit;
+			if (!arch_test_op(test_data))
+				goto exit;
+			GUEST_SYNC(stage++);
+			break;
 		};
 	}
 
@@ -109,6 +121,21 @@ static void vcpu_run_and_inc_stage(struct kvm_vcpu *vcpu)
 	test_data->stage++;
 }
 
+static int test_page(struct kvm_vcpu *vcpu, int op, gva_t vaddr)
+{
+	int rc;
+
+	test_data->op = op;
+	test_data->vaddr = vaddr;
+
+	rc = _vcpu_run(vcpu);
+
+	if (rc >= 0)
+		test_data->stage++;
+
+	return rc < 0 ? -errno : rc;
+}
+
 static void test_page_restricted(struct kvm_vcpu *vcpu, int op,
 				 gva_t vaddr, gpa_t fault_paddr,
 				 uint64_t fault_reason)
@@ -125,7 +152,8 @@ static void test_page_restricted(struct kvm_vcpu *vcpu, int op,
 		    test_op_names[op], rc, errno);
 	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MEMORY_FAULT);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, fault_paddr);
-	TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, fault_reason);
+	if (fault_reason)
+		TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, fault_reason);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, vm->page_size);
 }
 
@@ -337,6 +365,9 @@ int main(int argc, char *argv[])
 
 	test_input_validation(vm);
 	test_memory_access(vcpu, test_mem, size);
+#ifdef __x86_64__
+	arch_test_memory_access_pte(vcpu, test_mem);
+#endif
 	test_finalize(vcpu);
 
 	kvm_vm_free(vm);
diff --git a/tools/testing/selftests/kvm/x86/memory_attributes.c b/tools/testing/selftests/kvm/x86/memory_attributes.c
index 2e1148f5146d..12395feb6ac7 100644
--- a/tools/testing/selftests/kvm/x86/memory_attributes.c
+++ b/tools/testing/selftests/kvm/x86/memory_attributes.c
@@ -37,7 +37,147 @@ void arch_controlled_exec(gva_t addr)
 		     : "memory", "rax");
 }
 
-void arch_write_return_insn(struct kvm_vm *vm, gpa_t vaddr)
+void arch_write_return_insn(struct kvm_vm *vm, gpa_t paddr)
 {
-	memset(addr_gpa2hva(vm, vaddr), 0xc3, 1);
+	memset(addr_gpa2hva(vm, paddr), 0xc3, 1);
+}
+
+bool arch_test_op(struct test_data *test_data)
+{
+	gva_t vaddr = READ_ONCE(test_data->vaddr);
+
+	switch(READ_ONCE(test_data->op)) {
+	case TEST_OP_INVPLG:
+		asm volatile("invlpg (%0)"
+			     :: "b" (vaddr): "memory");
+		return true;
+	default:
+		return false;
+	}
+}
+
+/*
+ * This test validates that, during a page walk, if the page a PTE is placed in
+ * is read-only the accesss and dirty bits will not be written. Note There's a
+ * slight variation in behaviour between TDP and non-TDP VMs:
+ *  - With TDP enabled, KVM issues a fault exit upon observing the non-writable
+ *  page.
+ *  - With non-TDP, the access bit is not set, but the walk succeeds.
+ *
+ *  This is aligned with read-only memslots' behaviour.
+ */
+static void test_memory_access_pte_ro(struct kvm_vcpu *vcpu, gva_t vaddr)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gpa_t paddr;
+	u64 *pte;
+	const u64 accessed_mask = PTE_ACCESSED_MASK(&vm->mmu);
+
+	pte = vm_get_pte(vm, vaddr);
+	paddr = addr_hva2gpa(vm, pte) & GENMASK(61, vm->page_shift);
+
+	*pte &= ~accessed_mask;
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NW);
+	if (test_page(vcpu, TEST_OP_READ, vaddr) < 0) {
+		test_page_restricted(vcpu, TEST_OP_READ, vaddr, paddr,
+				     /* write PTE's accessed bit */
+				     KVM_MEMORY_EXIT_FLAG_WRITE);
+
+		vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+		test_page_accessible(vcpu, TEST_OP_READ, vaddr);
+		TEST_ASSERT_EQ(*pte & accessed_mask, accessed_mask);
+
+		/* Re-run the test, now vaddr is backed by an EPT. */
+		*pte &= ~accessed_mask;
+		vm_set_memory_attributes(vm, paddr, vm->page_size,
+					 KVM_MEMORY_ATTRIBUTE_NW);
+		test_page_restricted(vcpu, TEST_OP_READ, vaddr, paddr,
+				     /* write PTE's accessed bit */
+				     KVM_MEMORY_EXIT_FLAG_WRITE);
+		vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+		test_page_accessible(vcpu, TEST_OP_READ, vaddr);
+	} else {
+		TEST_ASSERT_EQ(*pte & accessed_mask, 0);
+		vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	}
+}
+
+/*
+ * This test validates that, during a page walk, if the page a PTE is placed in
+ * is maked as non-accesible, KVM issues a fault exit.
+ */
+static void test_memory_access_pte_nr(struct kvm_vcpu *vcpu, gva_t vaddr)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gpa_t paddr;
+	uint64_t *pte;
+
+	pte = vm_get_pte(vm, vaddr);
+	paddr = addr_hva2gpa(vm, pte) & GENMASK(61, vm->page_shift);
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size,
+				 KVM_MEMORY_ATTRIBUTE_NO_ACCESS);
+
+	test_page_restricted(vcpu, TEST_OP_READ, vaddr, paddr, 0);
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	test_page_accessible(vcpu, TEST_OP_READ, vaddr);
+
+	/* Re-run the test, now vaddr is backed by an SPTE. */
+	vm_set_memory_attributes(vm, paddr, vm->page_size,
+				 KVM_MEMORY_ATTRIBUTE_NO_ACCESS);
+	test_page_restricted(vcpu, TEST_OP_READ, vaddr, paddr, 0);
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	test_page_accessible(vcpu, TEST_OP_READ, vaddr);
+}
+
+static void test_memory_access_sync_spte(struct kvm_vcpu *vcpu, gva_t vaddr)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gpa_t paddr = addr_gva2gpa(vm, vaddr);
+	uint64_t *pte, old_pte, new_pte;
+
+	pte = vm_get_pte(vm, vaddr);
+	gpa_t pte_paddr = addr_hva2gpa(vm, pte);
+	gpa_t pte_page_paddr = pte_paddr & GENMASK(61, vm->page_shift);
+	int pte_offset = pte_paddr - pte_page_paddr;
+	virt_pg_map(vm, PTE_VADDR, pte_page_paddr);
+	old_pte = *pte;
+
+	/* Set vmaddr as non-executable */
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NX);
+
+	/*
+	 * Make sure SPTEs are populated as previous op might have destroyed
+	 * them. We new have a non-executable SPTE.
+	 */
+	test_page_accessible(vcpu, TEST_OP_READ, vaddr);
+
+	/*
+	 * Update PTE, make it non-writable and flush TLBs to make sure we go
+	 * through the sync_spte path. This should update the SPTE and make it
+	 * read-only.
+	 */
+	new_pte = (old_pte & ~PT_WRITABLE_MASK) | PT_ACCESSED_MASK;
+	test_data->expected_val = new_pte;
+	test_page_accessible(vcpu, TEST_OP_WRITE, PTE_VADDR + pte_offset);
+	TEST_ASSERT_EQ(*pte, new_pte);
+	test_page_accessible(vcpu, TEST_OP_INVPLG, vaddr);
+
+	/* The not executable attrs remain valid */
+	arch_write_return_insn(vm, paddr);
+	test_page_restricted(vcpu, TEST_OP_EXEC, vaddr, paddr,
+			     KVM_MEMORY_EXIT_FLAG_EXEC);
+
+	/* Cleanup */
+	*pte = old_pte;
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	test_page_accessible(vcpu, TEST_OP_EXEC, vaddr);
+}
+
+static void arch_test_memory_access_pte(struct kvm_vcpu *vcpu, gva_t vaddr)
+{
+	test_memory_access_pte_nr(vcpu, vaddr);
+	test_memory_access_pte_ro(vcpu, vaddr);
+	test_memory_access_sync_spte(vcpu, vaddr);
 }
-- 
2.52.0

[PATCH 44/45] KVM: x86: selftests: Introduce memory attributes side-channel tests

[Paolo Bonzini]

Introduce memory attributes selftests to catch any vulnerable
side-channels.

Memory attributes access restrictions are vulnerable to side-channel
attacks. This means that any KVM operation initiated by the guest that
requires guest memory access (which is the case for most
para-virtualised interfaces) needs to consider memory attributes.

The tests confirm this requirement is upheld for a variety of
use-cases, exercising various kinds of guest memory access such as
kvm_read/write_guest(), gfn_to_hva_cache and gfn_to_pfn_cache.

Co-developed-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Nicolas Saenz Julienne <nsaenz@amazon.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Link: https://patch.msgid.link/20260918135030.171564-32-pbonzini@redhat.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 .../testing/selftests/kvm/include/kvm_util.h  |   9 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../testing/selftests/kvm/lib/x86/processor.c |   5 +
 .../testing/selftests/kvm/memory_attributes.c |  51 +++-
 .../selftests/kvm/x86/memory_attributes.c     | 232 ++++++++++++++++++
 5 files changed, 297 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h
index 51b173105d3c..e887d3022855 100644
--- a/tools/testing/selftests/kvm/include/kvm_util.h
+++ b/tools/testing/selftests/kvm/include/kvm_util.h
@@ -1058,6 +1058,15 @@ static inline int vcpu_get_stats_fd(struct kvm_vcpu *vcpu)
 	return fd;
 }
 
+static inline struct kvm_translation vcpu_translate(struct kvm_vcpu *vcpu,
+						    u64 gva)
+{
+	struct kvm_translation tr = { .linear_address = gva };
+
+	vcpu_ioctl(vcpu, KVM_TRANSLATE, &tr);
+	return tr;
+}
+
 int __kvm_has_device_attr(int dev_fd, u32 group, u64 attr);
 
 static inline void kvm_has_device_attr(int dev_fd, u32 group, u64 attr)
diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index 5898a277a2e0..e9a997aa7290 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -1493,6 +1493,7 @@ static inline bool kvm_is_nested_virtualization_enabled(void)
 	return get_kvm_amd_param_integer("nested");
 }
 
+u64 *vm_get_pte_level(struct kvm_vm *vm, gva_t gva, int *level);
 u64 *vm_get_pte(struct kvm_vm *vm, gva_t gva);
 
 u64 kvm_hypercall(u64 nr, u64 a0, u64 a1, u64 a2, u64 a3);
diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c
index 8374afd08baf..d4d0b7b472c8 100644
--- a/tools/testing/selftests/kvm/lib/x86/processor.c
+++ b/tools/testing/selftests/kvm/lib/x86/processor.c
@@ -402,6 +402,11 @@ u64 *tdp_get_pte(struct kvm_vm *vm, u64 l2_gpa)
 	return __vm_get_page_table_entry(vm, &vm->stage2_mmu, l2_gpa, &level);
 }
 
+u64 *vm_get_pte_level(struct kvm_vm *vm, gva_t gva, int *level)
+{
+	return __vm_get_page_table_entry(vm, &vm->mmu, gva, level);
+}
+
 u64 *vm_get_pte(struct kvm_vm *vm, gva_t gva)
 {
 	int level = PG_LEVEL_4K;
diff --git a/tools/testing/selftests/kvm/memory_attributes.c b/tools/testing/selftests/kvm/memory_attributes.c
index c5305b4f3dbe..2883f6c772be 100644
--- a/tools/testing/selftests/kvm/memory_attributes.c
+++ b/tools/testing/selftests/kvm/memory_attributes.c
@@ -22,6 +22,8 @@
 #define TEST_MEM_SLOT	1
 #define TEST_MEM_GPA	0x80000000
 
+#define HV_STATUS_INVALID_HYPERCALL_INPUT	3
+
 #define MMIO_GPA	0x700000000
 #define MMIO_GVA	MMIO_GPA
 
@@ -29,12 +31,35 @@
 #define PT_ACCESSED_MASK	BIT_ULL(5)
 #define PTE_VADDR		0x1000000000
 
+static volatile uint64_t ipis_rcvd;
+
+static pthread_t vcpu_thread;
+
+struct hv_vpset {
+	u64 format;
+	u64 valid_bank_mask;
+	u64 bank_contents[2];
+};
+
+enum HV_GENERIC_SET_FORMAT {
+	HV_GENERIC_SET_SPARSE_4K,
+	HV_GENERIC_SET_ALL,
+};
+
+struct hv_send_ipi_ex {
+	u32 vector;
+	u32 reserved;
+	struct hv_vpset vp_set;
+};
+
 enum {
 	TEST_OP_NOP,
 	TEST_OP_READ,
 	TEST_OP_WRITE,
 	TEST_OP_EXEC,
 	TEST_OP_INVPLG,
+	TEST_OP_HYPERV_HYPERCALL_INPUT,
+	TEST_OP_MONITOR_ADDRESS,
 	TEST_OP_EXIT,
 };
 
@@ -44,6 +69,8 @@ const char *test_op_names[] =
 	[TEST_OP_WRITE] = "Write",
 	[TEST_OP_EXEC] = "Exec",
 	[TEST_OP_INVPLG] = "Invplg",
+	[TEST_OP_HYPERV_HYPERCALL_INPUT] = "HvHcall input",
+	[TEST_OP_MONITOR_ADDRESS] = "Monitor address",
 	[TEST_OP_EXIT] = "Exit",
 };
 
@@ -51,6 +78,8 @@ struct test_data {
 	uint8_t op;
 	int stage;
 	gva_t vaddr;
+	gpa_t paddr;
+	uint8_t confirm_read;
 	uint64_t expected_val;
 
 	struct kvm_vcpu *vcpu;
@@ -63,19 +92,26 @@ static void arch_controlled_write(gva_t addr, uint64_t val);
 static void arch_controlled_exec(gva_t addr);
 static void arch_write_return_insn(struct kvm_vm *vm, gpa_t paddr);
 static bool arch_test_op(struct test_data *test_data);
+static void arch_guest_init(void);
 
 static void guest_code(void *data)
 {
 	struct test_data *test_data = data;
 	int stage = 1;
 
+	arch_guest_init();
+
 	while (true) {
 		uint64_t expected_val = READ_ONCE(test_data->expected_val);
+		bool confirm_read = READ_ONCE(test_data->confirm_read);
 		gva_t vaddr = READ_ONCE(test_data->vaddr);
+		u64 val;
 
 		switch(READ_ONCE(test_data->op)) {
 		case TEST_OP_READ:
-			(void) arch_controlled_read(vaddr);
+			val = arch_controlled_read(vaddr);
+			if (confirm_read)
+				GUEST_ASSERT_EQ(expected_val, val);
 			GUEST_SYNC(stage++);
 			break;
 		case TEST_OP_WRITE:
@@ -86,6 +122,14 @@ static void guest_code(void *data)
 			arch_controlled_exec(vaddr);
 			GUEST_SYNC(stage++);
 			break;
+		case TEST_OP_MONITOR_ADDRESS: {
+			if (arch_controlled_read(vaddr) != expected_val)
+				GUEST_SYNC(stage++);
+			break;
+		}
+		case TEST_OP_NOP:
+			GUEST_SYNC(stage++);
+			break;
 		default:
 			if (!arch_test_op(test_data))
 				goto exit;
@@ -145,6 +189,7 @@ static void test_page_restricted(struct kvm_vcpu *vcpu, int op,
 
 	test_data->op = op;
 	test_data->vaddr = vaddr;
+	test_data->paddr = fault_paddr;
 
 	rc = _vcpu_run(vcpu);
 	TEST_ASSERT(rc == -1 && errno == EFAULT,
@@ -362,11 +407,15 @@ int main(int argc, char *argv[])
 	test_mem = vm_alloc_pages(vm, pages);
 	virt_map(vcpu->vm, MMIO_GVA, MMIO_GPA, 1);
 	test_data = init_test_data(vcpu);
+#ifdef __x86_64__
+	vcpu_set_hv_cpuid(vcpu);
+#endif
 
 	test_input_validation(vm);
 	test_memory_access(vcpu, test_mem, size);
 #ifdef __x86_64__
 	arch_test_memory_access_pte(vcpu, test_mem);
+	arch_test_side_channels(vcpu, test_mem, size);
 #endif
 	test_finalize(vcpu);
 
diff --git a/tools/testing/selftests/kvm/x86/memory_attributes.c b/tools/testing/selftests/kvm/x86/memory_attributes.c
index 12395feb6ac7..36e788be0e05 100644
--- a/tools/testing/selftests/kvm/x86/memory_attributes.c
+++ b/tools/testing/selftests/kvm/x86/memory_attributes.c
@@ -45,17 +45,28 @@ void arch_write_return_insn(struct kvm_vm *vm, gpa_t paddr)
 bool arch_test_op(struct test_data *test_data)
 {
 	gva_t vaddr = READ_ONCE(test_data->vaddr);
+	gpa_t paddr = READ_ONCE(test_data->paddr);
 
 	switch(READ_ONCE(test_data->op)) {
 	case TEST_OP_INVPLG:
 		asm volatile("invlpg (%0)"
 			     :: "b" (vaddr): "memory");
 		return true;
+	case TEST_OP_HYPERV_HYPERCALL_INPUT:
+		hyperv_hypercall(HVCALL_SEND_IPI_EX, paddr, 0);
+		asm volatile ("sti; hlt; cli;");
+		GUEST_ASSERT_EQ(ipis_rcvd, 1);
+		return true;
 	default:
 		return false;
 	}
 }
 
+void arch_guest_init(void)
+{
+	x2apic_enable();
+}
+
 /*
  * This test validates that, during a page walk, if the page a PTE is placed in
  * is read-only the accesss and dirty bits will not be written. Note There's a
@@ -181,3 +192,224 @@ static void arch_test_memory_access_pte(struct kvm_vcpu *vcpu, gva_t vaddr)
 	test_memory_access_pte_ro(vcpu, vaddr);
 	test_memory_access_sync_spte(vcpu, vaddr);
 }
+
+#define IPI_VECTOR	 0xfe
+
+static void guest_ipi_handler_hv(struct ex_regs *regs)
+{
+	ipis_rcvd++;
+	wrmsr(HV_X64_MSR_EOI, 1);
+}
+
+/*
+ * This test verifies that the Hyper-V hypercall exit handler takes memory
+ * attributes into account before accessing input data. It coordinates with the
+ * guest through the 'TEST_OP_HYPERV_HYPERCALL_INPUT' operation and instructs
+ * the guest to issue two PV IPIs. The first PV IPI fails because the input
+ * data is held in read-protected memory. Subsequently, the memory protection
+ * is lifted, and the second PV IPI succeeds.
+ */
+static void test_side_channel_hyperv_hypercall_inputs(struct kvm_vcpu *vcpu,
+						      gva_t vaddr,
+						      size_t size)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	struct hv_send_ipi_ex *ipi_ex = addr_gva2hva(vm, vaddr);
+	gpa_t paddr = addr_gva2gpa(vm, vaddr);
+
+	if (!kvm_has_cap(KVM_CAP_HYPERV_SEND_IPI) ||
+	    !kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT))
+		return;
+
+	vm_enable_cap(vcpu->vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
+
+	ipis_rcvd = 0;
+	vm_install_exception_handler(vm, IPI_VECTOR, guest_ipi_handler_hv);
+	vcpu_set_msr(vcpu, HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID);
+
+	*ipi_ex = (struct hv_send_ipi_ex){
+		.vector = IPI_VECTOR,
+		.vp_set.format = HV_GENERIC_SET_ALL,
+	};
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NO_ACCESS);
+	test_page_restricted(vcpu, TEST_OP_HYPERV_HYPERCALL_INPUT, vaddr, paddr,
+			     KVM_MEMORY_EXIT_FLAG_READ);
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	vcpu_run_and_inc_stage(vcpu);
+}
+
+/*
+ * Verifies that the guest page table walker fails the walk if it encounters a
+ * page table entry address read-protected by a memory attribute.
+ */
+static void test_side_channel_emul_page_walks(struct kvm_vcpu *vcpu,
+					      gva_t test_vm_addr,
+					      size_t size)
+{
+	const uint64_t pte_addr_mask = GENMASK(51, 12);
+	struct kvm_vm *vm = vcpu->vm;
+	struct kvm_translation tr;
+	int level = PG_LEVEL_1G;
+	gpa_t paddr;
+	uint64_t *pte;
+
+	pte = vm_get_pte_level(vm, test_vm_addr, &level);
+	TEST_ASSERT_EQ(level, PG_LEVEL_1G);
+	paddr = *pte & pte_addr_mask;
+
+	tr = vcpu_translate(vcpu, test_vm_addr);
+	TEST_ASSERT_EQ(tr.valid, true);
+	TEST_ASSERT_EQ(tr.physical_address, addr_gva2gpa(vm, test_vm_addr));
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NO_ACCESS);
+	tr = vcpu_translate(vcpu, test_vm_addr);
+	TEST_ASSERT_EQ(tr.valid, false);
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+}
+
+static void vm_set_vapic_addr(struct kvm_vcpu *vcpu, uint64_t addr)
+{
+	struct kvm_vapic_addr va;
+
+	va.vapic_addr = addr;
+	vcpu_ioctl(vcpu, KVM_SET_VAPIC_ADDR, &va);
+}
+
+/*
+ * Perform a dummy regs update to issue a KVM_REQ_EVENT. This forces
+ * vapic to be synced before entering the guest.
+ */
+static void vcpu_force_vapic_update(struct kvm_vcpu *vcpu)
+{
+	struct kvm_regs regs;
+
+	vcpu_regs_get(vcpu, &regs);
+	vcpu_regs_set(vcpu, &regs);
+}
+
+/*
+ * Setup the vapic address on a GPA that is write-protected. Force an vapic
+ * update and validate its contents were not changes. Then, lift the write
+ * restriction and validate the page's contents are updated.
+ */
+static void test_side_channel_vapic_addr(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gva_t vaddr = vm_alloc_page(vm);
+	gpa_t paddr = addr_gva2gpa(vm, vaddr);
+
+	vm_set_vapic_addr(vcpu, paddr);
+	test_data->op = TEST_OP_READ;
+	test_data->vaddr = vaddr;
+	test_data->confirm_read = 1;
+	test_data->expected_val = ~0ULL >> 32;
+	memset(addr_gva2hva(vm, vaddr), 0xff, sizeof(uint32_t));
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NW);
+	vcpu_run_and_inc_stage(vcpu);
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	vcpu_force_vapic_update(vcpu);
+	test_data->expected_val = 0ULL;
+	vcpu_run_and_inc_stage(vcpu);
+
+	vm_set_vapic_addr(vcpu, 0);
+	test_data->confirm_read = 0;
+}
+
+static void *vcpu_worker(void *data)
+{
+	struct test_data *test_data = data;
+	struct kvm_vcpu *vcpu = test_data->vcpu;
+
+	vcpu_run_and_inc_stage(vcpu);
+
+	return NULL;
+}
+
+/*
+ * Set up the pvclock page and validate that KVM periodically updates the
+ * 'version' field. Subsequently, make the pvclock page non-writable and
+ * verify that the 'version' field is no longer updated.
+ */
+static void test_side_channel_pvclock(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gva_t vaddr = vm_alloc_page(vm);
+	gpa_t paddr = addr_gva2gpa(vm, vaddr);
+	struct pvclock_vcpu_time_info *pvclock = addr_gpa2hva(vm, paddr);
+
+	pvclock->version = 0;
+
+	test_data->op = TEST_OP_MONITOR_ADDRESS;
+	test_data->vaddr = vaddr + offsetof(struct pvclock_vcpu_time_info, version);
+	test_data->expected_val = pvclock->version;
+
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, paddr | 0x1);
+	pthread_create(&vcpu_thread, NULL, vcpu_worker, test_data);
+	usleep(msecs_to_usecs(1000));
+	TEST_ASSERT_EQ(pthread_tryjoin_np(vcpu_thread, NULL), 0);
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, 0);
+
+	test_data->expected_val = pvclock->version;
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NW);
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, paddr | 0x1);
+	pthread_create(&vcpu_thread, NULL, vcpu_worker, test_data);
+	usleep(msecs_to_usecs(1000));
+	TEST_ASSERT_EQ(pthread_tryjoin_np(vcpu_thread, NULL), EBUSY);
+
+	/* Force the 'monitor_address' guest operation to finish */
+	test_data->op = TEST_OP_NOP;
+	TEST_ASSERT_EQ(pthread_join(vcpu_thread, NULL), 0);
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, 0);
+}
+
+/*
+ * Write to MSR_KVM_WALL_CLOCK_NEW and verify that the struct's 'version' field
+ * is updated. Subsequently, make the target guest physical address
+ * non-writable, and verify the 'version' field isn't updated anymore.
+ */
+static void test_side_channel_wallclock(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gva_t vaddr = vm_alloc_page(vm);
+	gpa_t paddr = addr_gva2gpa(vm, vaddr);
+	struct pvclock_wall_clock *wc = addr_gva2hva(vm, vaddr);
+
+	wc->version = 0x0;
+	vcpu_set_msr(vcpu, MSR_KVM_WALL_CLOCK_NEW, paddr);
+	TEST_ASSERT_EQ(READ_ONCE(wc->version), 2);
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NW);
+	vcpu_set_msr(vcpu, MSR_KVM_WALL_CLOCK_NEW, paddr);
+	TEST_ASSERT_EQ(READ_ONCE(wc->version), 2);
+
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+}
+
+/*
+ * Memory attributes are vulnerable to side-channel attacks. This means that
+ * any KVM operation initiated by the guest that requires guest memory access
+ * (which is the case for most pv-interfaces) needs to consider memory
+ * attributes.
+ *
+ * The following tests validate that this requirement is upheld for a variety
+ * of use-cases. These test cases were selected to exercise specific approaches
+ * to accessing guest memory, including:
+ *
+ *  - kvm_read/write_guest()
+ *  - gfn_to_hva_cache
+ *  - gfn_to_pfn_cache
+ *  - Guest page walker
+ */
+static void arch_test_side_channels(struct kvm_vcpu *vcpu, gva_t test_vm_addr,
+			            size_t size)
+{
+	test_side_channel_hyperv_hypercall_inputs(vcpu, test_vm_addr, size);
+	test_side_channel_emul_page_walks(vcpu, test_vm_addr, size);
+	test_side_channel_vapic_addr(vcpu);
+	test_side_channel_wallclock(vcpu);
+	test_side_channel_pvclock(vcpu);
+}
-- 
2.52.0

[PATCH 45/45] KVM: x86: split out parts of load_pdptrs

[Paolo Bonzini]

Add new functions that can be used to read and validate the PDPTRs
without placing them in the kvm_vcpu struct.  KVM can use them to
read all the information that is needed for a nested vmentry or vmexit,
and fault out to userspace is needed, prior to committing to completing
the vmentry/vmexit.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 arch/x86/kvm/regs.c | 51 ++++++++++++++++++++++++++++++++-------------
 arch/x86/kvm/regs.h |  3 +++
 2 files changed, 40 insertions(+), 14 deletions(-)

diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 3a11a6d06889..17e5aeb746ef 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -149,17 +149,29 @@ static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu)
 	return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2);
 }
 
+int kvm_validate_pdptrs(struct kvm_vcpu *vcpu, const u64 *pdptrs)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(vcpu->arch.pdptrs); ++i) {
+		if ((pdptrs[i] & PT_PRESENT_MASK) &&
+		    (pdptrs[i] & pdptr_rsvd_bits(vcpu)))
+			return 0;
+	}
+	return 1;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_validate_pdptrs);
+
 /*
- * Load the pae pdptrs.  Return 1 if they are all valid, 0 otherwise.
+ * Read the PAE PDPTRs into @pdptrs.  Return 1 if they are valid, 0 if they are
+ * invalid, and a negative value on error.
  */
-int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
+int kvm_read_validate_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3, u64 *pdptrs)
 {
 	struct kvm_pagewalk *w = &vcpu->arch.gva_walk;
 	gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT;
 	gpa_t real_gpa;
-	int i;
 	int ret;
-	u64 pdpte[ARRAY_SIZE(vcpu->arch.pdptrs)];
 
 	/*
 	 * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated
@@ -174,8 +186,8 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
 		return 0;
 
 	/* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */
-	ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte,
-				       cr3 & GENMASK(11, 5), sizeof(pdpte));
+	ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdptrs,
+				       cr3 & GENMASK(11, 5), sizeof(vcpu->arch.pdptrs));
 	if (ret == -EFAULT) {
 		kvm_prepare_memory_fault_exit(vcpu, real_gpa & PAGE_MASK, PAGE_SIZE,
 					      false, false, false);
@@ -184,26 +196,37 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
 	if (ret)
 		return 0;
 
-	for (i = 0; i < ARRAY_SIZE(pdpte); ++i) {
-		if ((pdpte[i] & PT_PRESENT_MASK) &&
-		    (pdpte[i] & pdptr_rsvd_bits(vcpu))) {
-			return 0;
-		}
-	}
+	return kvm_validate_pdptrs(vcpu, pdptrs);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_read_validate_pdptrs);
 
+void kvm_commit_pdptrs(struct kvm_vcpu *vcpu, const u64 *pdptrs)
+{
 	/*
 	 * Marking VCPU_REG_PDPTR dirty doesn't work for !tdp_enabled.
 	 * Shadow page roots need to be reconstructed instead.
 	 */
-	if (!tdp_enabled && memcmp(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs)))
+	if (!tdp_enabled && memcmp(vcpu->arch.pdptrs, pdptrs, sizeof(vcpu->arch.pdptrs)))
 		kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.root_mmu,
 				   KVM_MMU_ROOT_CURRENT);
 
-	memcpy(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs));
+	memcpy(vcpu->arch.pdptrs, pdptrs, sizeof(vcpu->arch.pdptrs));
 	kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR);
 	kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
 	vcpu->arch.pdptrs_from_userspace = false;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_commit_pdptrs);
 
+int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
+{
+	u64 pdptrs[ARRAY_SIZE(vcpu->arch.pdptrs)];
+	int r;
+
+	r = kvm_read_validate_pdptrs(vcpu, cr3, pdptrs);
+	if (r <= 0)
+		return r;
+
+	kvm_commit_pdptrs(vcpu, pdptrs);
 	return 1;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs);
diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h
index 31f4d3f6ea42..86a92a877374 100644
--- a/arch/x86/kvm/regs.h
+++ b/arch/x86/kvm/regs.h
@@ -63,6 +63,9 @@ unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr);
 unsigned long kvm_get_dr6_fixed_1(struct kvm_vcpu *vcpu);
 unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu);
 void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw);
+int kvm_read_validate_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3, u64 *pdptrs);
+int kvm_validate_pdptrs(struct kvm_vcpu *vcpu, const u64 *pdptrs);
+void kvm_commit_pdptrs(struct kvm_vcpu *vcpu, const u64 *pdptrs);
 int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
 
 static inline bool is_long_mode(struct kvm_vcpu *vcpu)
-- 
2.52.0