[PATCH RFC 00/10] KVM: Enable Clang Context Analysis

[PATCH RFC 00/10] KVM: Enable Clang Context Analysis

[Marco Elver]

Enable compiler-based static context analysis [1] for x86 KVM, along with
prerequisite lock annotations in virt/kvm.

During the initial annotation pass, context analysis uncovered a missing
SRCU read-side critical section in x86 PMU filter lookups; patch 1 fixes
this bug.

The remainder of the series is strictly non-functional: it establishes
basic function annotations, guarded_by annotations on core structs, and
straightforward refactorings in hva/gfn range walks and guest_memfd to
eliminate conditional locking patterns that cannot be tracked statically.

Annotating the MMU (mmu/mmu.c, mmu/tdp_mmu.c), i8259, and Xen requires
more invasive changes and is deferred to follow-ups.

[1] https://docs.kernel.org/next/dev-tools/context-analysis.html

Marco Elver (10):
  KVM: x86/pmu: Acquire SRCU in pmc_is_event_allowed() to protect filter
    lookup
  KVM: Allow reading memslots while holding slots_arch_lock
  KVM: guest_memfd: Avoid conditional mmu_lock acquisition
  KVM: Refactor kvm_handle_hva_range() to avoid conditional mmu_lock
  KVM: Refactor kvm_handle_gfn_range() to avoid conditional mmu_lock
  KVM: Add basic lock context annotations
  KVM: x86: Add basic lock context annotations
  KVM: Add guarded_by to members in struct kvm
  KVM: x86: Add guarded_by annotations for kvm_arch, kvm_hv, and ioapic
  KVM: x86: Enable CONTEXT_ANALYSIS with opt-outs

 arch/x86/include/asm/kvm_host.h          |  32 ++--
 arch/x86/kvm/Makefile                    |   8 +
 arch/x86/kvm/debugfs.c                   |   1 +
 arch/x86/kvm/hyperv.c                    |  21 +++
 arch/x86/kvm/hyperv.h                    |  16 +-
 arch/x86/kvm/ioapic.c                    |  28 ++-
 arch/x86/kvm/ioapic.h                    |  16 +-
 arch/x86/kvm/irq.c                       |   2 +
 arch/x86/kvm/lapic.c                     |   8 +-
 arch/x86/kvm/lapic.h                     |  15 +-
 arch/x86/kvm/mmu/page_track.c            |   1 +
 arch/x86/kvm/mmu/page_track.h            |   3 +-
 arch/x86/kvm/mmu/spte.h                  |   3 +-
 arch/x86/kvm/msrs.c                      |   5 +
 arch/x86/kvm/msrs.h                      |   3 +-
 arch/x86/kvm/pmu.c                       |   9 +-
 arch/x86/kvm/regs.h                      |  15 +-
 arch/x86/kvm/smm.c                       |   6 +
 arch/x86/kvm/smm.h                       |   2 +-
 arch/x86/kvm/svm/hyperv.c                |   2 +
 arch/x86/kvm/svm/hyperv.h                |   6 +-
 arch/x86/kvm/svm/nested.c                |  30 +++-
 arch/x86/kvm/svm/sev.c                   |  18 ++
 arch/x86/kvm/svm/svm.c                   |  32 ++++
 arch/x86/kvm/svm/svm.h                   |  28 ++-
 arch/x86/kvm/vmx/hyperv.h                |   6 +-
 arch/x86/kvm/vmx/main.c                  |   5 +
 arch/x86/kvm/vmx/nested.c                |  54 +++++-
 arch/x86/kvm/vmx/nested.h                |  19 +-
 arch/x86/kvm/vmx/sgx.c                   |   3 +
 arch/x86/kvm/vmx/sgx.h                   |   3 +-
 arch/x86/kvm/vmx/tdx.c                   |   9 +
 arch/x86/kvm/vmx/vmx.c                   |  11 ++
 arch/x86/kvm/vmx/x86_ops.h               |  24 ++-
 arch/x86/kvm/x86.c                       |  66 ++++++-
 arch/x86/kvm/x86.h                       |  37 ++--
 include/linux/kvm_host.h                 | 215 ++++++++++++++++-------
 scripts/context-analysis-suppression.txt |   1 +
 virt/kvm/dirty_ring.c                    |   2 +
 virt/kvm/eventfd.c                       |   9 +-
 virt/kvm/guest_memfd.c                   |  21 ++-
 virt/kvm/kvm_main.c                      | 183 +++++++++++++------
 virt/kvm/pfncache.c                      |   3 +
 43 files changed, 745 insertions(+), 236 deletions(-)

-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 01/10] KVM: x86/pmu: Acquire SRCU in pmc_is_event_allowed() to protect filter lookup

[Marco Elver]

Dereferencing kvm->arch.pmu_event_filter via srcu_dereference() requires
holding kvm->srcu to guard against concurrent filter replacement and
freeing by kvm_vm_ioctl_set_pmu_event_filter().

Counter reprogramming can reach pmc_is_event_allowed() without holding
kvm->srcu. Specifically, on AMD SVM, toggling EFER.SVME via KVM_SET_SREGS
or KVM_SET_SREGS2 triggers synchronous counter reprogramming outside of
any SRCU read-side critical section:

  kvm_vcpu_ioctl(KVM_SET_SREGS{,2})
    kvm_vcpu_ioctl_x86_set_sregs{,2}()
      __set_sregs_common()
        kvm_x86_call(set_efer)()
          svm_set_efer()
            svm_pmu_handle_nested_transition()
              __svm_pmu_handle_nested_transition(..., defer=false)
                __kvm_pmu_reprogram_counters()
                  kvm_pmu_handle_event()
                    reprogram_counter()
                      pmc_is_event_allowed()
                        srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu)

If userspace concurrently updates the filter (KVM_SET_PMU_EVENT_FILTER),
a concurrent free and subsequent use-after-free is possible.

Protect filter lookups directly in pmc_is_event_allowed():
1. check rcu_access_pointer() first for the common fast path;
2. acquire guard(srcu)(&kvm->srcu) only when a filter is present;
3. drop redundant outer srcu_read_lock() in kvm_pmu_trigger_event().

Found with Clang context analysis.

Fixes: a02a25a65246 ("KVM: x86/pmu: Reprogram Host/Guest-Only counters on nested transitions")
Signed-off-by: Marco Elver <elver@google.com>
---
 arch/x86/kvm/pmu.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
index a7d60c8785cd..3ad1e696edca 100644
--- a/arch/x86/kvm/pmu.c
+++ b/arch/x86/kvm/pmu.c
@@ -536,6 +536,11 @@ static bool pmc_is_event_allowed(struct kvm_pmc *pmc)
 	struct kvm_x86_pmu_event_filter *filter;
 	struct kvm *kvm = pmc->vcpu->kvm;
 
+	if (!rcu_access_pointer(kvm->arch.pmu_event_filter))
+		return true;
+
+	guard(srcu)(&kvm->srcu);
+
 	filter = srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu);
 	if (!filter)
 		return true;
@@ -1132,7 +1137,7 @@ static void kvm_pmu_trigger_event(struct kvm_vcpu *vcpu,
 	DECLARE_BITMAP(bitmap, X86_PMC_IDX_MAX);
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 	struct kvm_pmc *pmc;
-	int i, idx;
+	int i;
 
 	BUILD_BUG_ON(sizeof(pmu->global_ctrl) * BITS_PER_BYTE != X86_PMC_IDX_MAX);
 
@@ -1145,14 +1150,12 @@ static void kvm_pmu_trigger_event(struct kvm_vcpu *vcpu,
 			     (unsigned long *)&pmu->global_ctrl, X86_PMC_IDX_MAX))
 		return;
 
-	idx = srcu_read_lock(&vcpu->kvm->srcu);
 	kvm_for_each_pmc(pmu, pmc, i, bitmap) {
 		if (!pmc_is_event_allowed(pmc) || !cpl_is_matched(pmc))
 			continue;
 
 		kvm_pmu_incr_counter(pmc);
 	}
-	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 }
 
 void kvm_pmu_instruction_retired(struct kvm_vcpu *vcpu)
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 02/10] KVM: Allow reading memslots while holding slots_arch_lock

[Marco Elver]

kvm_swap_active_memslots() updates kvm->memslots[as_id] while holding both
kvm->slots_lock and kvm->slots_arch_lock. Holding either lock guarantees
that memslots cannot be concurrently modified.

Allow reading memslots in __kvm_memslots() when kvm->slots_arch_lock is
held.

Signed-off-by: Marco Elver <elver@google.com>
---
 include/linux/kvm_host.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 03bfc92864b6..5ed8260ef01f 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1089,6 +1089,7 @@ static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)
 	as_id = array_index_nospec(as_id, KVM_MAX_NR_ADDRESS_SPACES);
 	return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu,
 			lockdep_is_held(&kvm->slots_lock) ||
+			lockdep_is_held(&kvm->slots_arch_lock) ||
 			!refcount_read(&kvm->users_count));
 }
 
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 03/10] KVM: guest_memfd: Avoid conditional mmu_lock acquisition

[Marco Elver]

Refactor __kvm_gmem_invalidate_start() to check for present bindings
upfront via xa_find() instead of tracking found memslots inside the
range iteration with a 'found_memslot' flag and conditionally acquiring
and releasing KVM's mmu_lock.

Eliminating the conditional locking inside the loop simplifies control
flow, ensures lock scoping is straightforward, and subsequently allows
Clang context analysis to verify that mmu_lock is held unconditionally
across kvm_mmu_unmap_gfn_range().

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 virt/kvm/guest_memfd.c | 20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 625e62e1a031..896f3b076562 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -124,10 +124,16 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 					pgoff_t end,
 					enum kvm_gfn_range_filter attr_filter)
 {
-	bool flush = false, found_memslot = false;
+	bool flush = false;
 	struct kvm_memory_slot *slot;
 	struct kvm *kvm = f->kvm;
-	unsigned long index;
+	pgoff_t index = start;
+
+	if (!xa_find(&f->bindings, &index, end - 1, XA_PRESENT))
+		return;
+
+	KVM_MMU_LOCK(kvm);
+	kvm_mmu_invalidate_start(kvm);
 
 	xa_for_each_range(&f->bindings, index, slot, start, end - 1) {
 		pgoff_t pgoff = slot->gmem.pgoff;
@@ -140,13 +146,6 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 			.attr_filter = attr_filter,
 		};
 
-		if (!found_memslot) {
-			found_memslot = true;
-
-			KVM_MMU_LOCK(kvm);
-			kvm_mmu_invalidate_start(kvm);
-		}
-
 		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
@@ -157,8 +156,7 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 	if (flush)
 		kvm_flush_remote_tlbs(kvm);
 
-	if (found_memslot)
-		KVM_MMU_UNLOCK(kvm);
+	KVM_MMU_UNLOCK(kvm);
 }
 
 static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start,
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 04/10] KVM: Refactor kvm_handle_hva_range() to avoid conditional mmu_lock

[Marco Elver]

Refactor kvm_handle_hva_range() to check for overlapping memslots
upfront via interval_tree_iter_first() instead of tracking found
memslots inside the range iteration with a 'found_memslot' flag and
conditionally acquiring and releasing mmu_lock.

This simplifies the control flow by cleanly decoupling the search for
overlapping memslots from the subsequent walk. It also separates the
lockless path from the serialized path into distinct branches,
eliminating the conditional locking, which subsequently enables Clang
context analysis to validate locking in this function.

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 virt/kvm/kvm_main.c | 94 ++++++++++++++++++++++++++++-----------------
 1 file changed, 58 insertions(+), 36 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 65eb26a0520d..f7bfa2d32507 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -558,31 +558,14 @@ static void kvm_null_fn(void)
 	     node;							     \
 	     node = interval_tree_iter_next(node, start, last))	     \
 
-static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
-							 const struct kvm_mmu_notifier_range *range)
+static __always_inline bool __kvm_handle_hva_range_walk(struct kvm *kvm,
+							const struct kvm_mmu_notifier_range *range)
 {
-	struct kvm_mmu_notifier_return r = {
-		.ret = false,
-		.found_memslot = false,
-	};
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
 	struct kvm_memslots *slots;
-	int i, idx;
-
-	if (WARN_ON_ONCE(range->end <= range->start))
-		return r;
-
-	/* A null handler is allowed if and only if on_lock() is provided. */
-	if (WARN_ON_ONCE(IS_KVM_NULL_FN(range->on_lock) &&
-			 IS_KVM_NULL_FN(range->handler)))
-		return r;
-
-	/* on_lock will never be called for lockless walks */
-	if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock)))
-		return r;
-
-	idx = srcu_read_lock(&kvm->srcu);
+	bool ret = false;
+	int i;
 
 	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
 		struct interval_tree_node *node;
@@ -620,28 +603,67 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			gfn_range.slot = slot;
 			gfn_range.lockless = range->lockless;
 
-			if (!r.found_memslot) {
-				r.found_memslot = true;
-				if (!range->lockless) {
-					KVM_MMU_LOCK(kvm);
-					if (!IS_KVM_NULL_FN(range->on_lock))
-						range->on_lock(kvm);
+			ret |= range->handler(kvm, &gfn_range);
+		}
+	}
+
+	return ret;
+}
 
-					if (IS_KVM_NULL_FN(range->handler))
-						goto mmu_unlock;
-				}
-			}
-			r.ret |= range->handler(kvm, &gfn_range);
+static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
+							 const struct kvm_mmu_notifier_range *range)
+{
+	struct kvm_mmu_notifier_return r = {
+		.ret = false,
+		.found_memslot = false,
+	};
+	struct kvm_memslots *slots;
+	int i, idx;
+
+	if (WARN_ON_ONCE(range->end <= range->start))
+		return r;
+
+	/* A null handler is allowed if and only if on_lock() is provided. */
+	if (WARN_ON_ONCE(IS_KVM_NULL_FN(range->on_lock) &&
+			 IS_KVM_NULL_FN(range->handler)))
+		return r;
+
+	/* on_lock will never be called for lockless walks */
+	if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock)))
+		return r;
+
+	idx = srcu_read_lock(&kvm->srcu);
+
+	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
+		slots = __kvm_memslots(kvm, i);
+		if (interval_tree_iter_first(&slots->hva_tree, range->start, range->end - 1)) {
+			r.found_memslot = true;
+			break;
 		}
 	}
 
-	if (range->flush_on_ret && r.ret)
-		kvm_flush_remote_tlbs(kvm);
+	if (!r.found_memslot)
+		goto out;
+
+	if (range->lockless) {
+		r.ret = __kvm_handle_hva_range_walk(kvm, range);
+		if (range->flush_on_ret && r.ret)
+			kvm_flush_remote_tlbs(kvm);
+	} else {
+		KVM_MMU_LOCK(kvm);
+		if (!IS_KVM_NULL_FN(range->on_lock))
+			range->on_lock(kvm);
+
+		if (!IS_KVM_NULL_FN(range->handler))
+			r.ret = __kvm_handle_hva_range_walk(kvm, range);
+
+		if (range->flush_on_ret && r.ret)
+			kvm_flush_remote_tlbs(kvm);
 
-mmu_unlock:
-	if (r.found_memslot && !range->lockless)
 		KVM_MMU_UNLOCK(kvm);
+	}
 
+out:
 	srcu_read_unlock(&kvm->srcu, idx);
 
 	return r;
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 05/10] KVM: Refactor kvm_handle_gfn_range() to avoid conditional mmu_lock

[Marco Elver]

Refactor kvm_handle_gfn_range() to check for overlapping memslots
upfront via kvm_for_each_memslot_in_gfn_range() instead of tracking
found memslots inside the range iteration with a 'found_memslot' flag
and conditionally acquiring and releasing mmu_lock.

This simplifies the control flow by cleanly decoupling the search for
overlapping memslots from the subsequent walk, eliminating the
conditional locking and establishing a clean, unconditional lock scope
for mmu_lock, which subsequently enables Clang context analysis to
validate locking in this function.

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 virt/kvm/kvm_main.c | 46 +++++++++++++++++++++++++++++++++------------
 1 file changed, 34 insertions(+), 12 deletions(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f7bfa2d32507..f86e690a1798 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2485,14 +2485,13 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	return true;
 }
 
-static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
-						 struct kvm_mmu_notifier_range *range)
+static __always_inline bool __kvm_handle_gfn_range_walk(struct kvm *kvm,
+							struct kvm_mmu_notifier_range *range)
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
 	struct kvm_memslots *slots;
 	struct kvm_memslot_iter iter;
-	bool found_memslot = false;
 	bool ret = false;
 	int i;
 
@@ -2519,22 +2518,45 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 			if (gfn_range.start >= gfn_range.end)
 				continue;
 
-			if (!found_memslot) {
-				found_memslot = true;
-				KVM_MMU_LOCK(kvm);
-				if (!IS_KVM_NULL_FN(range->on_lock))
-					range->on_lock(kvm);
-			}
-
 			ret |= range->handler(kvm, &gfn_range);
 		}
 	}
 
+	return ret;
+}
+
+static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
+						 struct kvm_mmu_notifier_range *range)
+{
+	struct kvm_memslot_iter iter;
+	struct kvm_memslots *slots;
+	bool found_memslot = false;
+	bool ret;
+	int i;
+
+	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
+		slots = __kvm_memslots(kvm, i);
+		kvm_for_each_memslot_in_gfn_range(&iter, slots, range->start, range->end) {
+			found_memslot = true;
+			break;
+		}
+		if (found_memslot)
+			break;
+	}
+
+	if (!found_memslot)
+		return;
+
+	KVM_MMU_LOCK(kvm);
+	if (!IS_KVM_NULL_FN(range->on_lock))
+		range->on_lock(kvm);
+
+	ret = __kvm_handle_gfn_range_walk(kvm, range);
+
 	if (range->flush_on_ret && ret)
 		kvm_flush_remote_tlbs(kvm);
 
-	if (found_memslot)
-		KVM_MMU_UNLOCK(kvm);
+	KVM_MMU_UNLOCK(kvm);
 }
 
 static bool kvm_pre_set_memory_attributes(struct kvm *kvm,
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 06/10] KVM: Add basic lock context annotations

[Marco Elver]

Add basic lock context annotations across common KVM sources.

The only opted-out functions are kvm_trylock_all_vcpus(),
kvm_lock_all_vcpus(), and kvm_unlock_all_vcpus(), which dynamically
acquire and release an arbitrary number of vCPU mutexes that cannot be
represented statically.

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 include/linux/kvm_host.h | 202 +++++++++++++++++++++++++++------------
 virt/kvm/dirty_ring.c    |   2 +
 virt/kvm/eventfd.c       |   5 +
 virt/kvm/guest_memfd.c   |   1 +
 virt/kvm/kvm_main.c      |  40 +++++++-
 virt/kvm/pfncache.c      |   3 +
 6 files changed, 189 insertions(+), 64 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 5ed8260ef01f..12f241304228 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -220,18 +220,6 @@ enum kvm_bus {
 	KVM_NR_BUSES
 };
 
-int kvm_io_bus_write(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t addr,
-		     int len, const void *val);
-int kvm_io_bus_write_cookie(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx,
-			    gpa_t addr, int len, const void *val, long cookie);
-int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t addr,
-		    int len, void *val);
-int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
-			    int len, struct kvm_io_device *dev);
-int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx,
-			      struct kvm_io_device *dev);
-struct kvm_io_device *kvm_io_bus_get_dev(struct kvm *kvm, enum kvm_bus bus_idx,
-					 gpa_t addr);
 
 #ifdef CONFIG_KVM_ASYNC_PF
 struct kvm_async_pf {
@@ -245,12 +233,6 @@ struct kvm_async_pf {
 	bool   wakeup_all;
 	bool notpresent_injected;
 };
-
-void kvm_clear_async_pf_completion_queue(struct kvm_vcpu *vcpu);
-void kvm_check_async_pf_completion(struct kvm_vcpu *vcpu);
-bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
-			unsigned long hva, struct kvm_arch_async_pf *arch);
-int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
 #endif
 
 union kvm_mmu_notifier_arg {
@@ -905,6 +887,25 @@ struct kvm {
 #define vcpu_err(vcpu, fmt, ...)					\
 	kvm_err("vcpu%i " fmt, (vcpu)->vcpu_id, ## __VA_ARGS__)
 
+int kvm_io_bus_write(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t addr,
+		     int len, const void *val)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int kvm_io_bus_write_cookie(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx,
+			    gpa_t addr, int len, const void *val, long cookie)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t addr,
+		    int len, void *val)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
+			    int len, struct kvm_io_device *dev)
+	__must_hold(&kvm->slots_lock);
+int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx,
+			      struct kvm_io_device *dev)
+	__must_hold(&kvm->slots_lock);
+struct kvm_io_device *kvm_io_bus_get_dev(struct kvm *kvm, enum kvm_bus bus_idx,
+					 gpa_t addr)
+	__must_hold_shared(&kvm->srcu);
+
 static inline void kvm_vm_dead(struct kvm *kvm)
 {
 	kvm->vm_dead = true;
@@ -956,6 +957,7 @@ static inline void kvm_vm_bugged(struct kvm *kvm)
 })
 
 static inline void kvm_vcpu_srcu_read_lock(struct kvm_vcpu *vcpu)
+	__acquires_shared(&vcpu->kvm->srcu)
 {
 #ifdef CONFIG_PROVE_RCU
 	WARN_ONCE(vcpu->srcu_depth++,
@@ -965,6 +967,7 @@ static inline void kvm_vcpu_srcu_read_lock(struct kvm_vcpu *vcpu)
 }
 
 static inline void kvm_vcpu_srcu_read_unlock(struct kvm_vcpu *vcpu)
+	__releases_shared(&vcpu->kvm->srcu)
 {
 	srcu_read_unlock(&vcpu->kvm->srcu, vcpu->____srcu_idx);
 
@@ -985,6 +988,7 @@ static inline bool kvm_dirty_log_manual_protect_and_init_set(struct kvm *kvm)
  * registrations.
  */
 static inline struct kvm_io_bus *kvm_get_bus(struct kvm *kvm, enum kvm_bus idx)
+	__must_hold(&kvm->slots_lock)
 {
 	return rcu_dereference_protected(kvm->buses[idx],
 					 lockdep_is_held(&kvm->slots_lock));
@@ -1047,9 +1051,9 @@ static inline bool kvm_is_vcpu_creation_in_progress(struct kvm *kvm)
 
 void kvm_destroy_vcpus(struct kvm *kvm);
 
-int kvm_trylock_all_vcpus(struct kvm *kvm);
-int kvm_lock_all_vcpus(struct kvm *kvm);
-void kvm_unlock_all_vcpus(struct kvm *kvm);
+int kvm_trylock_all_vcpus(struct kvm *kvm) __must_hold(&kvm->lock);
+int kvm_lock_all_vcpus(struct kvm *kvm)    __must_hold(&kvm->lock);
+void kvm_unlock_all_vcpus(struct kvm *kvm) __must_hold(&kvm->lock);
 
 void vcpu_load(struct kvm_vcpu *vcpu);
 void vcpu_put(struct kvm_vcpu *vcpu);
@@ -1085,6 +1089,7 @@ bool file_is_kvm(struct file *file);
 void kvm_put_kvm_no_destroy(struct kvm *kvm);
 
 static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)
+	__must_hold_shared(&kvm->srcu)
 {
 	as_id = array_index_nospec(as_id, KVM_MAX_NR_ADDRESS_SPACES);
 	return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu,
@@ -1094,11 +1099,13 @@ static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)
 }
 
 static inline struct kvm_memslots *kvm_memslots(struct kvm *kvm)
+	__must_hold_shared(&kvm->srcu)
 {
 	return __kvm_memslots(kvm, 0);
 }
 
 static inline struct kvm_memslots *kvm_vcpu_memslots(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int as_id = kvm_arch_vcpu_memslots_id(vcpu);
 
@@ -1110,7 +1117,8 @@ static inline bool kvm_memslots_empty(struct kvm_memslots *slots)
 	return RB_EMPTY_ROOT(&slots->gfn_tree);
 }
 
-bool kvm_are_all_memslots_empty(struct kvm *kvm);
+bool kvm_are_all_memslots_empty(struct kvm *kvm)
+	__must_hold(&kvm->slots_lock);
 
 #define kvm_for_each_memslot(memslot, bkt, slots)			      \
 	hash_for_each(slots->id_hash, bkt, memslot, id_node[slots->node_idx]) \
@@ -1225,9 +1233,10 @@ static inline bool kvm_memslot_iter_is_valid(struct kvm_memslot_iter *iter, gfn_
 	     kvm_memslot_iter_is_valid(iter, end);			\
 	     kvm_memslot_iter_next(iter))
 
-struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn);
-struct kvm_memslots *kvm_vcpu_memslots(struct kvm_vcpu *vcpu);
-struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn);
+struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn)
+	__must_hold_shared(&kvm->srcu);
+struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 /*
  * KVM_SET_USER_MEMORY_REGION ioctl allows the following operations:
@@ -1248,7 +1257,8 @@ enum kvm_mr_change {
 };
 
 int kvm_set_internal_memslot(struct kvm *kvm,
-			     const struct kvm_userspace_memory_region2 *mem);
+			     const struct kvm_userspace_memory_region2 *mem)
+	__must_hold(&kvm->slots_lock);
 void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot);
 void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen);
 int kvm_arch_prepare_memory_region(struct kvm *kvm,
@@ -1268,14 +1278,18 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn,
 		       struct page **pages, int nr_pages);
 
-struct page *__gfn_to_page(struct kvm *kvm, gfn_t gfn, bool write);
+struct page *__gfn_to_page(struct kvm *kvm, gfn_t gfn, bool write)
+	__must_hold_shared(&kvm->srcu);
 static inline struct page *gfn_to_page(struct kvm *kvm, gfn_t gfn)
+	__must_hold_shared(&kvm->srcu)
 {
 	return __gfn_to_page(kvm, gfn, true);
 }
 
-unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn);
-unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable);
+unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn)
+	__must_hold_shared(&kvm->srcu);
+unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable)
+	__must_hold_shared(&kvm->srcu);
 unsigned long gfn_to_hva_memslot(struct kvm_memory_slot *slot, gfn_t gfn);
 unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot, gfn_t gfn,
 				      bool *writable);
@@ -1324,30 +1338,40 @@ kvm_pfn_t __kvm_faultin_pfn(const struct kvm_memory_slot *slot, gfn_t gfn,
 static inline kvm_pfn_t kvm_faultin_pfn(struct kvm_vcpu *vcpu, gfn_t gfn,
 					bool write, bool *writable,
 					struct page **refcounted_page)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return __kvm_faultin_pfn(kvm_vcpu_gfn_to_memslot(vcpu, gfn), gfn,
 				 write ? FOLL_WRITE : 0, writable, refcounted_page);
 }
 
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
-			int len);
-int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long len);
+			int len)
+	__must_hold_shared(&kvm->srcu);
+int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long len)
+	__must_hold_shared(&kvm->srcu);
 int kvm_read_guest_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
-			   void *data, unsigned long len);
+			   void *data, unsigned long len)
+	__must_hold_shared(&kvm->srcu);
 int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 				 void *data, unsigned int offset,
-				 unsigned long len);
+				 unsigned long len)
+	__must_hold_shared(&kvm->srcu);
 int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, const void *data,
-			 int offset, int len);
+			 int offset, int len)
+	__must_hold_shared(&kvm->srcu);
 int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data,
-		    unsigned long len);
+		    unsigned long len)
+	__must_hold_shared(&kvm->srcu);
 int kvm_write_guest_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
-			   void *data, unsigned long len);
+			   void *data, unsigned long len)
+	__must_hold_shared(&kvm->srcu);
 int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 				  void *data, unsigned int offset,
-				  unsigned long len);
+				  unsigned long len)
+	__must_hold_shared(&kvm->srcu);
 int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
-			      gpa_t gpa, unsigned long len);
+			      gpa_t gpa, unsigned long len)
+	__must_hold_shared(&kvm->srcu);
 
 #define __kvm_get_guest(kvm, gfn, offset, v)				\
 ({									\
@@ -1391,32 +1415,42 @@ int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 			offset_in_page(__gpa), v);			\
 })
 
-int kvm_clear_guest(struct kvm *kvm, gpa_t gpa, unsigned long len);
-bool kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn);
-bool kvm_vcpu_is_visible_gfn(struct kvm_vcpu *vcpu, gfn_t gfn);
-unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn);
+int kvm_clear_guest(struct kvm *kvm, gpa_t gpa, unsigned long len)
+	__must_hold_shared(&kvm->srcu);
+bool kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn)
+	__must_hold_shared(&kvm->srcu);
+bool kvm_vcpu_is_visible_gfn(struct kvm_vcpu *vcpu, gfn_t gfn)
+	__must_hold_shared(&vcpu->kvm->srcu);
+unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void mark_page_dirty_in_slot(struct kvm *kvm, const struct kvm_memory_slot *memslot, gfn_t gfn);
-void mark_page_dirty(struct kvm *kvm, gfn_t gfn);
-void kvm_vcpu_mark_page_dirty(struct kvm_vcpu *vcpu, gfn_t gfn);
+void mark_page_dirty(struct kvm *kvm, gfn_t gfn)
+	__must_hold_shared(&kvm->srcu);
+void kvm_vcpu_mark_page_dirty(struct kvm_vcpu *vcpu, gfn_t gfn)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map,
-		   bool writable);
+		   bool writable)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map);
 
 static inline int kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn,
 			       struct kvm_host_map *map)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return __kvm_vcpu_map(vcpu, gfn, map, true);
 }
 
 static inline int kvm_vcpu_map_readonly(struct kvm_vcpu *vcpu, gfn_t gfn,
 					struct kvm_host_map *map)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return __kvm_vcpu_map(vcpu, gfn, map, false);
 }
 
 static inline void kvm_vcpu_map_mark_dirty(struct kvm_vcpu *vcpu,
 					   struct kvm_host_map *map)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (kvm_vcpu_mapped(map))
 		kvm_vcpu_mark_page_dirty(vcpu, map->gfn);
@@ -1441,18 +1475,25 @@ DEFINE_CLASS(kvm_vcpu_map_local##ro, kvm_vcpu_local_map_t,		\
 DEFINE_VCPU_MAP_CLASS();
 DEFINE_VCPU_MAP_CLASS(_readonly);
 
-unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn);
-unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable);
+unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn)
+	__must_hold_shared(&vcpu->kvm->srcu);
+unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, int offset,
-			     int len);
+			     int len)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa, void *data,
-			       unsigned long len);
+			       unsigned long len)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_vcpu_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa, void *data,
-			unsigned long len);
+			unsigned long len)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_vcpu_write_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, const void *data,
-			      int offset, int len);
+			      int offset, int len)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data,
-			 unsigned long len);
+			 unsigned long len)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 /**
  * kvm_gpc_init - initialize gfn_to_pfn_cache.
@@ -1482,7 +1523,8 @@ void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm);
  * invalidations to be processed.  Callers are required to use kvm_gpc_check()
  * to ensure that the cache is valid before accessing the target page.
  */
-int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long len);
+int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long len)
+	__must_hold_shared(&gpc->kvm->srcu);
 
 /**
  * kvm_gpc_activate_hva - prepare a cached kernel mapping and HPA for a given HVA.
@@ -1498,7 +1540,8 @@ int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long len)
  * The semantics of this function are the same as those of kvm_gpc_activate(). It
  * merely bypasses a layer of address translation.
  */
-int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long hva, unsigned long len);
+int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long hva, unsigned long len)
+	__must_hold_shared(&gpc->kvm->srcu);
 
 /**
  * kvm_gpc_check - check validity of a gfn_to_pfn_cache.
@@ -1516,7 +1559,8 @@ int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long hva, unsign
  * Callers in IN_GUEST_MODE may do so without locking, although they should
  * still hold a read lock on kvm->scru for the memslot checks.
  */
-bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len);
+bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
+	__must_hold_shared(&gpc->kvm->srcu);
 
 /**
  * kvm_gpc_refresh - update a previously initialized cache.
@@ -1534,7 +1578,8 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len);
  * still lock and check the cache status, as this function does not return
  * with the lock still held to permit access.
  */
-int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len);
+int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len)
+	__must_hold_shared(&gpc->kvm->srcu);
 
 /**
  * kvm_gpc_deactivate - deactivate and unlink a gfn_to_pfn_cache.
@@ -1652,6 +1697,31 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu);
 void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu);
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu);
 
+#ifdef CONFIG_KVM_ASYNC_PF
+void kvm_clear_async_pf_completion_queue(struct kvm_vcpu *vcpu);
+void kvm_check_async_pf_completion(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
+bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
+			unsigned long hva, struct kvm_arch_async_pf *arch)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu);
+
+bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu,
+				     struct kvm_async_pf *work)
+	__must_hold_shared(&vcpu->kvm->srcu);
+void kvm_arch_async_page_present(struct kvm_vcpu *vcpu,
+				 struct kvm_async_pf *work)
+	__must_hold_shared(&vcpu->kvm->srcu);
+void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu,
+			       struct kvm_async_pf *work)
+	__must_hold_shared(&vcpu->kvm->srcu);
+#ifndef CONFIG_KVM_ASYNC_PF_SYNC
+void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu);
+#endif
+bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
+#endif
+
 #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER
 int kvm_arch_pm_notifier(struct kvm *kvm, unsigned long state);
 #endif
@@ -1824,8 +1894,10 @@ struct kvm_irq_ack_notifier {
 };
 
 int kvm_irq_map_gsi(struct kvm *kvm,
-		    struct kvm_kernel_irq_routing_entry *entries, int gsi);
-int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin);
+		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
+	__must_hold_shared(&kvm->irq_srcu);
+int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
+	__must_hold_shared(&kvm->irq_srcu);
 
 int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
 		bool line_status);
@@ -1835,7 +1907,8 @@ int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *e,
 			       struct kvm *kvm, int irq_source_id,
 			       int level, bool line_status);
 bool kvm_irq_has_notifier(struct kvm *kvm, unsigned irqchip, unsigned pin);
-void kvm_notify_acked_gsi(struct kvm *kvm, int gsi);
+void kvm_notify_acked_gsi(struct kvm *kvm, int gsi)
+	__must_hold_shared(&kvm->irq_srcu);
 void kvm_notify_acked_irq(struct kvm *kvm, unsigned irqchip, unsigned pin);
 void kvm_register_irq_ack_notifier(struct kvm *kvm,
 				   struct kvm_irq_ack_notifier *kian);
@@ -1937,6 +2010,7 @@ __gfn_to_hva_memslot(const struct kvm_memory_slot *slot, gfn_t gfn)
 }
 
 static inline int memslot_id(struct kvm *kvm, gfn_t gfn)
+	__must_hold_shared(&kvm->srcu)
 {
 	return gfn_to_memslot(kvm, gfn)->id;
 }
@@ -1965,6 +2039,7 @@ static inline hpa_t pfn_to_hpa(kvm_pfn_t pfn)
 }
 
 static inline bool kvm_is_gpa_in_memslot(struct kvm *kvm, gpa_t gpa)
+	__must_hold_shared(&kvm->srcu)
 {
 	unsigned long hva = gfn_to_hva(kvm, gpa_to_gfn(gpa));
 
@@ -2265,7 +2340,8 @@ void kvm_irqfd_release(struct kvm *kvm);
 bool kvm_notify_irqfd_resampler(struct kvm *kvm,
 				unsigned int irqchip,
 				unsigned int pin);
-void kvm_irq_routing_update(struct kvm *);
+void kvm_irq_routing_update(struct kvm *kvm)
+	__must_hold(&kvm->irq_lock);
 #else
 static inline int kvm_irqfd(struct kvm *kvm, struct kvm_irqfd *args)
 {
@@ -2282,7 +2358,8 @@ static inline bool kvm_notify_irqfd_resampler(struct kvm *kvm,
 }
 #endif /* CONFIG_HAVE_KVM_IRQCHIP */
 
-void kvm_arch_irq_routing_update(struct kvm *kvm);
+void kvm_arch_irq_routing_update(struct kvm *kvm)
+	__must_hold(&kvm->irq_lock);
 
 static inline void __kvm_make_request(int req, struct kvm_vcpu *vcpu)
 {
@@ -2632,7 +2709,8 @@ typedef int (*kvm_gmem_populate_cb)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 
 long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 		       long npages, bool may_writeback_src,
-		       kvm_gmem_populate_cb post_populate, void *opaque);
+		       kvm_gmem_populate_cb post_populate, void *opaque)
+	__must_hold(&kvm->slots_lock);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c
index 572b854edf74..11a6aba98b5e 100644
--- a/virt/kvm/dirty_ring.c
+++ b/virt/kvm/dirty_ring.c
@@ -51,6 +51,7 @@ static bool kvm_dirty_ring_full(struct kvm_dirty_ring *ring)
 }
 
 static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64 mask)
+	__must_hold(&kvm->slots_lock)
 {
 	struct kvm_memory_slot *memslot;
 	int as_id, id;
@@ -61,6 +62,7 @@ static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64 mask)
 	if (as_id >= kvm_arch_nr_memslot_as_ids(kvm) || id >= KVM_USER_MEM_SLOTS)
 		return;
 
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 	memslot = id_to_memslot(__kvm_memslots(kvm, as_id), id);
 
 	if (!memslot || offset >= memslot->npages ||
diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
index 93ad2ebc963f..6701390336f6 100644
--- a/virt/kvm/eventfd.c
+++ b/virt/kvm/eventfd.c
@@ -271,6 +271,7 @@ irqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)
 }
 
 static void irqfd_update(struct kvm *kvm, struct kvm_kernel_irqfd *irqfd)
+	__must_hold_shared(&kvm->irq_srcu)
 {
 	struct kvm_kernel_irq_routing_entry *e;
 	struct kvm_kernel_irq_routing_entry entries[KVM_NR_IRQCHIPS];
@@ -300,6 +301,7 @@ struct kvm_irqfd_pt {
 
 static void kvm_irqfd_register(struct file *file, wait_queue_head_t *wqh,
 			       poll_table *pt)
+	__must_hold_shared(&container_of(pt, struct kvm_irqfd_pt, pt)->kvm->irq_srcu)
 {
 	struct kvm_irqfd_pt *p = container_of(pt, struct kvm_irqfd_pt, pt);
 	struct kvm_kernel_irqfd *irqfd = p->irqfd;
@@ -660,6 +662,9 @@ void kvm_irq_routing_update(struct kvm *kvm)
 {
 	struct kvm_kernel_irqfd *irqfd;
 
+	/* Update-side mutex kvm->irq_lock is held. */
+	__assume_shared_ctx_lock(&kvm->irq_srcu);
+
 	spin_lock_irq(&kvm->irqfds.lock);
 
 	list_for_each_entry(irqfd, &kvm->irqfds.items, list) {
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 896f3b076562..bc97e566559e 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -836,6 +836,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 	long i;
 
 	lockdep_assert_held(&kvm->slots_lock);
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 
 	if (WARN_ON_ONCE(npages <= 0))
 		return -EINVAL;
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f86e690a1798..2f22d5439d39 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -560,6 +560,7 @@ static void kvm_null_fn(void)
 
 static __always_inline bool __kvm_handle_hva_range_walk(struct kvm *kvm,
 							const struct kvm_mmu_notifier_range *range)
+	__must_hold_shared(&kvm->srcu)
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
@@ -1379,6 +1380,7 @@ static int kvm_vm_release(struct inode *inode, struct file *filp)
 }
 
 int kvm_trylock_all_vcpus(struct kvm *kvm)
+	__context_unsafe(/* multi-lock acquisition */)
 {
 	struct kvm_vcpu *vcpu;
 	unsigned long i, j;
@@ -1401,6 +1403,7 @@ int kvm_trylock_all_vcpus(struct kvm *kvm)
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_trylock_all_vcpus);
 
 int kvm_lock_all_vcpus(struct kvm *kvm)
+	__context_unsafe(/* multi-lock acquisition */)
 {
 	struct kvm_vcpu *vcpu;
 	unsigned long i, j;
@@ -1426,6 +1429,7 @@ int kvm_lock_all_vcpus(struct kvm *kvm)
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lock_all_vcpus);
 
 void kvm_unlock_all_vcpus(struct kvm *kvm)
+	__context_unsafe(/* multi-lock release */)
 {
 	struct kvm_vcpu *vcpu;
 	unsigned long i;
@@ -1453,7 +1457,9 @@ static int kvm_alloc_dirty_bitmap(struct kvm_memory_slot *memslot)
 }
 
 static struct kvm_memslots *kvm_get_inactive_memslots(struct kvm *kvm, int as_id)
+	__must_hold(&kvm->slots_lock)
 {
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 	struct kvm_memslots *active = __kvm_memslots(kvm, as_id);
 	int node_idx_inactive = active->node_idx ^ 1;
 
@@ -1535,6 +1541,7 @@ static void kvm_replace_gfn_node(struct kvm_memslots *slots,
 static void kvm_replace_memslot(struct kvm *kvm,
 				struct kvm_memory_slot *old,
 				struct kvm_memory_slot *new)
+	__must_hold(&kvm->slots_lock)
 {
 	int as_id = kvm_memslots_get_as_id(old, new);
 	struct kvm_memslots *slots = kvm_get_inactive_memslots(kvm, as_id);
@@ -1621,9 +1628,11 @@ static int check_memory_region_flags(struct kvm *kvm,
 }
 
 static void kvm_swap_active_memslots(struct kvm *kvm, int as_id)
+	__must_hold(&kvm->slots_lock)
+	__releases(&kvm->slots_arch_lock)
 {
 	struct kvm_memslots *slots = kvm_get_inactive_memslots(kvm, as_id);
-
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side locks are held */
 	/* Grab the generation from the activate memslots. */
 	u64 gen = __kvm_memslots(kvm, as_id)->generation;
 
@@ -1796,6 +1805,8 @@ static void kvm_commit_memory_region(struct kvm *kvm,
 static void kvm_activate_memslot(struct kvm *kvm,
 				 struct kvm_memory_slot *old,
 				 struct kvm_memory_slot *new)
+	__must_hold(&kvm->slots_lock)
+	__releases(&kvm->slots_arch_lock)
 {
 	int as_id = kvm_memslots_get_as_id(old, new);
 
@@ -1821,6 +1832,8 @@ static void kvm_copy_memslot(struct kvm_memory_slot *dest,
 static void kvm_invalidate_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *old,
 				   struct kvm_memory_slot *invalid_slot)
+	__must_hold(&kvm->slots_lock)
+	__must_hold(&kvm->slots_arch_lock)
 {
 	/*
 	 * Mark the current slot INVALID.  As with all memslot modifications,
@@ -1862,6 +1875,8 @@ static void kvm_invalidate_memslot(struct kvm *kvm,
 
 static void kvm_create_memslot(struct kvm *kvm,
 			       struct kvm_memory_slot *new)
+	__must_hold(&kvm->slots_lock)
+	__releases(&kvm->slots_arch_lock)
 {
 	/* Add the new memslot to the inactive set and activate. */
 	kvm_replace_memslot(kvm, NULL, new);
@@ -1871,6 +1886,8 @@ static void kvm_create_memslot(struct kvm *kvm,
 static void kvm_delete_memslot(struct kvm *kvm,
 			       struct kvm_memory_slot *old,
 			       struct kvm_memory_slot *invalid_slot)
+	__must_hold(&kvm->slots_lock)
+	__releases(&kvm->slots_arch_lock)
 {
 	/*
 	 * Remove the old memslot (in the inactive memslots) by passing NULL as
@@ -1884,6 +1901,8 @@ static void kvm_move_memslot(struct kvm *kvm,
 			     struct kvm_memory_slot *old,
 			     struct kvm_memory_slot *new,
 			     struct kvm_memory_slot *invalid_slot)
+	__must_hold(&kvm->slots_lock)
+	__releases(&kvm->slots_arch_lock)
 {
 	/*
 	 * Replace the old memslot in the inactive slots, and then swap slots
@@ -1896,6 +1915,8 @@ static void kvm_move_memslot(struct kvm *kvm,
 static void kvm_update_flags_memslot(struct kvm *kvm,
 				     struct kvm_memory_slot *old,
 				     struct kvm_memory_slot *new)
+	__must_hold(&kvm->slots_lock)
+	__releases(&kvm->slots_arch_lock)
 {
 	/*
 	 * Similar to the MOVE case, but the slot doesn't need to be zapped as
@@ -1910,6 +1931,7 @@ static int kvm_set_memslot(struct kvm *kvm,
 			   struct kvm_memory_slot *old,
 			   struct kvm_memory_slot *new,
 			   enum kvm_mr_change change)
+	__must_hold(&kvm->slots_lock)
 {
 	struct kvm_memory_slot *invalid_slot;
 	int r;
@@ -2016,6 +2038,7 @@ static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id,
 
 static int kvm_set_memory_region(struct kvm *kvm,
 				 const struct kvm_userspace_memory_region2 *mem)
+	__must_hold(&kvm->slots_lock)
 {
 	struct kvm_memory_slot *old, *new;
 	struct kvm_memslots *slots;
@@ -2026,6 +2049,7 @@ static int kvm_set_memory_region(struct kvm *kvm,
 	int r;
 
 	lockdep_assert_held(&kvm->slots_lock);
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 
 	r = check_memory_region_flags(kvm, mem);
 	if (r)
@@ -2242,6 +2266,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dirty_log);
  *
  */
 static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log)
+	__must_hold(&kvm->slots_lock)
 {
 	struct kvm_memslots *slots;
 	struct kvm_memory_slot *memslot;
@@ -2260,6 +2285,7 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log)
 	if (as_id >= kvm_arch_nr_memslot_as_ids(kvm) || id >= KVM_USER_MEM_SLOTS)
 		return -EINVAL;
 
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 	slots = __kvm_memslots(kvm, as_id);
 	memslot = id_to_memslot(slots, id);
 	if (!memslot || !memslot->dirty_bitmap)
@@ -2353,6 +2379,7 @@ static int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
  */
 static int kvm_clear_dirty_log_protect(struct kvm *kvm,
 				       struct kvm_clear_dirty_log *log)
+	__must_hold(&kvm->slots_lock)
 {
 	struct kvm_memslots *slots;
 	struct kvm_memory_slot *memslot;
@@ -2375,6 +2402,7 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm,
 	if (log->first_page & 63)
 		return -EINVAL;
 
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 	slots = __kvm_memslots(kvm, as_id);
 	memslot = id_to_memslot(slots, id);
 	if (!memslot || !memslot->dirty_bitmap)
@@ -2487,6 +2515,7 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 
 static __always_inline bool __kvm_handle_gfn_range_walk(struct kvm *kvm,
 							struct kvm_mmu_notifier_range *range)
+	__must_hold(&kvm->slots_lock)
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
@@ -2506,6 +2535,7 @@ static __always_inline bool __kvm_handle_gfn_range_walk(struct kvm *kvm,
 	 * if the private flag is being toggled, i.e. all mappings are in play.
 	 */
 
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
 		slots = __kvm_memslots(kvm, i);
 
@@ -2527,6 +2557,7 @@ static __always_inline bool __kvm_handle_gfn_range_walk(struct kvm *kvm,
 
 static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 						 struct kvm_mmu_notifier_range *range)
+	__must_hold(&kvm->slots_lock)
 {
 	struct kvm_memslot_iter iter;
 	struct kvm_memslots *slots;
@@ -2534,6 +2565,7 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 	bool ret;
 	int i;
 
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
 		slots = __kvm_memslots(kvm, i);
 		kvm_for_each_memslot_in_gfn_range(&iter, slots, range->start, range->end) {
@@ -3201,8 +3233,10 @@ void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map)
 		memunmap(map->hva);
 #endif
 
-	if (map->writable)
+	if (map->writable) {
+		__assume_shared_ctx_lock(&vcpu->kvm->srcu); /* srcu held or VM being destroyed */
 		kvm_vcpu_mark_page_dirty(vcpu, map->gfn);
+	}
 
 	if (map->pinned_page) {
 		if (map->writable)
@@ -5075,6 +5109,7 @@ bool kvm_are_all_memslots_empty(struct kvm *kvm)
 	int i;
 
 	lockdep_assert_held(&kvm->slots_lock);
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 
 	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
 		if (!kvm_memslots_empty(__kvm_memslots(kvm, i)))
@@ -5930,6 +5965,7 @@ static int __kvm_io_bus_write(struct kvm_vcpu *vcpu, struct kvm_io_bus *bus,
 }
 
 static struct kvm_io_bus *kvm_get_bus_srcu(struct kvm *kvm, enum kvm_bus idx)
+	__must_hold_shared(&kvm->srcu)
 {
 	/*
 	 * Ensure that any updates to kvm_buses[] observed by the previous vCPU
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index 728d2c1b488a..f883fbad3487 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -153,6 +153,7 @@ static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long mmu_s
 }
 
 static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
+	__must_hold(&gpc->lock)
 {
 	/* Note, the new page offset may be different than the old! */
 	void *old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva);
@@ -254,6 +255,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
 }
 
 static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long uhva)
+	__must_hold_shared(&gpc->kvm->srcu)
 {
 	unsigned long page_offset;
 	bool unmap_old = false;
@@ -396,6 +398,7 @@ void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm)
 
 static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long uhva,
 			      unsigned long len)
+	__must_hold_shared(&gpc->kvm->srcu)
 {
 	struct kvm *kvm = gpc->kvm;
 
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 07/10] KVM: x86: Add basic lock context annotations

[Marco Elver]

Add basic lock context annotations across common x86, VMX, SVM, and
Hyper-V sources in preparation for enabling context analysis.

The only opted-out functions are tdx_acquire_vm_state_locks() and
tdx_release_vm_state_locks(), which dynamically acquire and release an
arbitrary number of locks that cannot be represented statically.

Because Clang context analysis evaluates lock expressions syntactically,
it cannot prove the equivalence of heap pointer aliases (e.g. `vcpu->kvm`
vs. `to_svm(vcpu)->vcpu.kvm`). When a callee expects a lock expression
through a different alias path than the caller, bridge the alias using
lockdep_assert_held() on the target expression. This dynamically validates
lock ownership under lockdep while satisfying the compiler's static analysis.

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 arch/x86/kvm/debugfs.c        |  1 +
 arch/x86/kvm/hyperv.c         | 21 +++++++++++++
 arch/x86/kvm/hyperv.h         | 16 +++++++---
 arch/x86/kvm/ioapic.c         | 24 ++++++++++++---
 arch/x86/kvm/irq.c            |  2 ++
 arch/x86/kvm/lapic.c          |  8 ++++-
 arch/x86/kvm/lapic.h          | 15 ++++++---
 arch/x86/kvm/mmu/page_track.c |  1 +
 arch/x86/kvm/mmu/page_track.h |  3 +-
 arch/x86/kvm/mmu/spte.h       |  3 +-
 arch/x86/kvm/msrs.c           |  5 +++
 arch/x86/kvm/msrs.h           |  3 +-
 arch/x86/kvm/regs.h           | 15 ++++++---
 arch/x86/kvm/smm.c            |  6 ++++
 arch/x86/kvm/smm.h            |  2 +-
 arch/x86/kvm/svm/hyperv.c     |  2 ++
 arch/x86/kvm/svm/hyperv.h     |  6 ++--
 arch/x86/kvm/svm/nested.c     | 30 ++++++++++++++++--
 arch/x86/kvm/svm/sev.c        | 18 +++++++++++
 arch/x86/kvm/svm/svm.c        | 32 +++++++++++++++++++
 arch/x86/kvm/svm/svm.h        | 28 +++++++++++------
 arch/x86/kvm/vmx/hyperv.h     |  6 ++--
 arch/x86/kvm/vmx/main.c       |  5 +++
 arch/x86/kvm/vmx/nested.c     | 54 ++++++++++++++++++++++++++++++--
 arch/x86/kvm/vmx/nested.h     | 19 ++++++++----
 arch/x86/kvm/vmx/sgx.c        |  3 ++
 arch/x86/kvm/vmx/sgx.h        |  3 +-
 arch/x86/kvm/vmx/tdx.c        |  9 ++++++
 arch/x86/kvm/vmx/vmx.c        | 11 +++++++
 arch/x86/kvm/vmx/x86_ops.h    | 24 ++++++++++-----
 arch/x86/kvm/x86.c            | 58 ++++++++++++++++++++++++++++++++---
 arch/x86/kvm/x86.h            | 37 +++++++++++++++-------
 32 files changed, 398 insertions(+), 72 deletions(-)

diff --git a/arch/x86/kvm/debugfs.c b/arch/x86/kvm/debugfs.c
index 0074a56e45b4..0bf8c021ee89 100644
--- a/arch/x86/kvm/debugfs.c
+++ b/arch/x86/kvm/debugfs.c
@@ -109,6 +109,7 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v)
 	}
 
 	mutex_lock(&kvm->slots_lock);
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 	write_lock(&kvm->mmu_lock);
 
 	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 604651cb2739..cca7c98a3808 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -268,6 +268,8 @@ static int synic_set_msr(struct kvm_vcpu_hv_synic *synic,
 	struct kvm_vcpu *vcpu = hv_synic_to_vcpu(synic);
 	int ret;
 
+	lockdep_assert_held(&vcpu->kvm->srcu);
+
 	if (!synic->active && (!host || data))
 		return 1;
 
@@ -780,6 +782,8 @@ static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint,
 	struct hv_message_header hv_hdr;
 	int r;
 
+	lockdep_assert_held(&vcpu->kvm->srcu);
+
 	if (!(synic->msg_page & HV_SYNIC_SIMP_ENABLE))
 		return -ENOENT;
 
@@ -1056,6 +1060,7 @@ static bool kvm_hv_msr_partition_wide(u32 msr)
 }
 
 static int kvm_hv_msr_get_crash_data(struct kvm *kvm, u32 index, u64 *pdata)
+	__must_hold(&to_kvm_hv(kvm)->hv_lock)
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 	size_t size = ARRAY_SIZE(hv->hv_crash_param);
@@ -1068,6 +1073,7 @@ static int kvm_hv_msr_get_crash_data(struct kvm *kvm, u32 index, u64 *pdata)
 }
 
 static int kvm_hv_msr_get_crash_ctl(struct kvm *kvm, u64 *pdata)
+	__must_hold(&to_kvm_hv(kvm)->hv_lock)
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 
@@ -1076,6 +1082,7 @@ static int kvm_hv_msr_get_crash_ctl(struct kvm *kvm, u64 *pdata)
 }
 
 static int kvm_hv_msr_set_crash_ctl(struct kvm *kvm, u64 data)
+	__must_hold(&to_kvm_hv(kvm)->hv_lock)
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 
@@ -1085,6 +1092,7 @@ static int kvm_hv_msr_set_crash_ctl(struct kvm *kvm, u64 data)
 }
 
 static int kvm_hv_msr_set_crash_data(struct kvm *kvm, u32 index, u64 data)
+	__must_hold(&to_kvm_hv(kvm)->hv_lock)
 {
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 	size_t size = ARRAY_SIZE(hv->hv_crash_param);
@@ -1175,6 +1183,7 @@ static bool compute_tsc_page_parameters(struct pvclock_vcpu_time_info *hv_clock,
  * TSC scaling is unsupported).
  */
 static inline bool tsc_page_update_unsafe(struct kvm_hv *hv)
+	__must_hold(&hv->hv_lock)
 {
 	return (hv->hv_tsc_page_status != HV_TSC_PAGE_GUEST_CHANGED) &&
 		hv->hv_tsc_emulation_control;
@@ -1397,6 +1406,8 @@ void kvm_hv_xsaves_xsavec_maybe_warn(struct kvm_vcpu *vcpu)
 
 static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data,
 			     bool host)
+	__must_hold(&to_kvm_hv(vcpu->kvm)->hv_lock)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm *kvm = vcpu->kvm;
 	struct kvm_hv *hv = to_kvm_hv(kvm);
@@ -1539,6 +1550,7 @@ static u64 current_task_runtime_100ns(void)
 }
 
 static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 
@@ -1652,6 +1664,7 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host)
 
 static int kvm_hv_get_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata,
 			     bool host)
+	__must_hold(&to_kvm_hv(vcpu->kvm)->hv_lock)
 {
 	u64 data = 0;
 	struct kvm *kvm = vcpu->kvm;
@@ -1910,6 +1923,7 @@ struct kvm_hv_hcall {
 
 static int kvm_hv_get_hc_data(struct kvm *kvm, struct kvm_hv_hcall *hc,
 			      u16 orig_cnt, u16 cnt_cap, u64 *data)
+	__must_hold_shared(&kvm->srcu)
 {
 	/*
 	 * Preserve the original count when ignoring entries via a "cap", KVM
@@ -1943,6 +1957,7 @@ static int kvm_hv_get_hc_data(struct kvm *kvm, struct kvm_hv_hcall *hc,
 
 static u64 kvm_get_sparse_vp_set(struct kvm *kvm, struct kvm_hv_hcall *hc,
 				 u64 *sparse_banks)
+	__must_hold_shared(&kvm->srcu)
 {
 	if (hc->var_cnt > HV_MAX_SPARSE_VCPU_BANKS)
 		return -EINVAL;
@@ -1953,6 +1968,7 @@ static u64 kvm_get_sparse_vp_set(struct kvm *kvm, struct kvm_hv_hcall *hc,
 }
 
 static int kvm_hv_get_tlb_flush_entries(struct kvm *kvm, struct kvm_hv_hcall *hc, u64 entries[])
+	__must_hold_shared(&kvm->srcu)
 {
 	return kvm_hv_get_hc_data(kvm, hc, hc->rep_cnt, hc->rep_cnt, entries);
 }
@@ -2034,6 +2050,7 @@ int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu)
 }
 
 static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	unsigned long *vcpu_mask = hv_vcpu->vcpu_mask;
@@ -2243,6 +2260,7 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector,
 }
 
 static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	u64 *sparse_banks = hv_vcpu->sparse_banks;
@@ -2405,6 +2423,7 @@ static void kvm_hv_hypercall_set_result(struct kvm_vcpu *vcpu, u64 result)
 }
 
 static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 tlb_lock_count = 0;
 	int ret;
@@ -2428,11 +2447,13 @@ static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result)
 }
 
 static int kvm_hv_hypercall_complete_userspace(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return kvm_hv_hypercall_complete(vcpu, vcpu->run->hyperv.u.hcall.result);
 }
 
 static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_hv *hv = to_kvm_hv(vcpu->kvm);
 	struct eventfd_ctx *eventfd;
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 622a6553e9ac..09dbab1fddb1 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -197,7 +197,8 @@ static inline u32 kvm_hv_get_vpindex(struct kvm_vcpu *vcpu)
 	return hv_vcpu ? hv_vcpu->vp_index : vcpu->vcpu_idx;
 }
 
-int kvm_hv_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host);
+int kvm_hv_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_hv_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool host);
 
 static inline bool kvm_hv_hypercall_enabled(struct kvm_vcpu *vcpu)
@@ -205,9 +206,11 @@ static inline bool kvm_hv_hypercall_enabled(struct kvm_vcpu *vcpu)
 	return vcpu->arch.hyperv_enabled && to_kvm_hv(vcpu->kvm)->hv_guest_os_id;
 }
 
-int kvm_hv_hypercall(struct kvm_vcpu *vcpu);
+int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
-void kvm_hv_irq_routing_update(struct kvm *kvm);
+void kvm_hv_irq_routing_update(struct kvm *kvm)
+	__must_hold_shared(&kvm->irq_srcu);
 int kvm_hv_synic_set_irq(struct kvm_kernel_irq_routing_entry *e, struct kvm *kvm,
 			 int irq_source_id, int level, bool line_status);
 void kvm_hv_synic_send_eoi(struct kvm_vcpu *vcpu, int vector);
@@ -227,7 +230,8 @@ static inline bool kvm_hv_synic_auto_eoi_set(struct kvm_vcpu *vcpu, int vector)
 void kvm_hv_vcpu_uninit(struct kvm_vcpu *vcpu);
 
 bool kvm_hv_assist_page_enabled(struct kvm_vcpu *vcpu);
-int kvm_hv_get_assist_page(struct kvm_vcpu *vcpu);
+int kvm_hv_get_assist_page(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 static inline struct kvm_vcpu_hv_stimer *to_hv_stimer(struct kvm_vcpu *vcpu,
 						      int timer_index)
@@ -285,7 +289,8 @@ static inline bool kvm_hv_invtsc_suppressed(struct kvm_vcpu *vcpu)
 void kvm_hv_process_stimers(struct kvm_vcpu *vcpu);
 
 void kvm_hv_setup_tsc_page(struct kvm *kvm,
-			   struct pvclock_vcpu_time_info *hv_clock);
+			   struct pvclock_vcpu_time_info *hv_clock)
+	__must_hold_shared(&kvm->srcu);
 void kvm_hv_request_tsc_page_update(struct kvm *kvm);
 
 void kvm_hv_xsaves_xsavec_maybe_warn(struct kvm_vcpu *vcpu);
@@ -352,6 +357,7 @@ static inline bool kvm_hv_is_tlb_flush_hcall(struct kvm_vcpu *vcpu)
 }
 
 static inline int kvm_hv_verify_vp_assist(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (!to_hv_vcpu(vcpu))
 		return 0;
diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c
index 0d59b9c758c2..d6865e557abe 100644
--- a/arch/x86/kvm/ioapic.c
+++ b/arch/x86/kvm/ioapic.c
@@ -36,9 +36,11 @@
 #include "x86.h"
 
 static int ioapic_service(struct kvm_ioapic *vioapic, int irq,
-		bool line_status);
+		bool line_status)
+	__must_hold(&vioapic->lock);
 
 static unsigned long ioapic_read_indirect(struct kvm_ioapic *ioapic)
+	__must_hold(&ioapic->lock)
 {
 	unsigned long result = 0;
 
@@ -76,20 +78,24 @@ static unsigned long ioapic_read_indirect(struct kvm_ioapic *ioapic)
 }
 
 static void rtc_irq_eoi_tracking_reset(struct kvm_ioapic *ioapic)
+	__must_hold(&ioapic->lock)
 {
 	ioapic->rtc_status.pending_eoi = 0;
 	bitmap_zero(ioapic->rtc_status.map, KVM_MAX_VCPU_IDS);
 }
 
-static void kvm_rtc_eoi_tracking_restore_all(struct kvm_ioapic *ioapic);
+static void kvm_rtc_eoi_tracking_restore_all(struct kvm_ioapic *ioapic)
+	__must_hold(&ioapic->lock);
 
 static void rtc_status_pending_eoi_check_valid(struct kvm_ioapic *ioapic)
+	__must_hold(&ioapic->lock)
 {
 	if (WARN_ON_ONCE(ioapic->rtc_status.pending_eoi < 0))
 		kvm_rtc_eoi_tracking_restore_all(ioapic);
 }
 
 static void __rtc_irq_eoi_tracking_restore_one(struct kvm_vcpu *vcpu)
+	__must_hold(&vcpu->kvm->arch.vioapic->lock)
 {
 	bool new_val, old_val;
 	struct kvm_ioapic *ioapic = vcpu->kvm->arch.vioapic;
@@ -137,12 +143,15 @@ static void kvm_rtc_eoi_tracking_restore_all(struct kvm_ioapic *ioapic)
 		return;
 
 	rtc_irq_eoi_tracking_reset(ioapic);
-	kvm_for_each_vcpu(i, vcpu, ioapic->kvm)
-	    __rtc_irq_eoi_tracking_restore_one(vcpu);
+	kvm_for_each_vcpu(i, vcpu, ioapic->kvm) {
+		lockdep_assert_held(&vcpu->kvm->arch.vioapic->lock); /* vcpu->kvm->arch.vioapic == ioapic */
+		__rtc_irq_eoi_tracking_restore_one(vcpu);
+	}
 }
 
 static void rtc_irq_eoi(struct kvm_ioapic *ioapic, struct kvm_vcpu *vcpu,
 			int vector)
+	__must_hold(&ioapic->lock)
 {
 	struct rtc_status *status = &ioapic->rtc_status;
 
@@ -156,6 +165,7 @@ static void rtc_irq_eoi(struct kvm_ioapic *ioapic, struct kvm_vcpu *vcpu,
 }
 
 static bool rtc_irq_check_coalesced(struct kvm_ioapic *ioapic)
+	__must_hold(&ioapic->lock)
 {
 	if (ioapic->rtc_status.pending_eoi > 0)
 		return true; /* coalesced */
@@ -164,6 +174,7 @@ static bool rtc_irq_check_coalesced(struct kvm_ioapic *ioapic)
 }
 
 static void ioapic_lazy_update_eoi(struct kvm_ioapic *ioapic, int irq)
+	__must_hold(&ioapic->lock)
 {
 	unsigned long i;
 	struct kvm_vcpu *vcpu;
@@ -187,6 +198,7 @@ static void ioapic_lazy_update_eoi(struct kvm_ioapic *ioapic, int irq)
 
 static int ioapic_set_irq(struct kvm_ioapic *ioapic, unsigned int irq,
 		int irq_level, bool line_status)
+	__must_hold(&ioapic->lock)
 {
 	union kvm_ioapic_redirect_entry entry;
 	u32 mask = 1 << irq;
@@ -246,6 +258,7 @@ static int ioapic_set_irq(struct kvm_ioapic *ioapic, unsigned int irq,
 }
 
 static void kvm_ioapic_inject_all(struct kvm_ioapic *ioapic, unsigned long irr)
+	__must_hold(&ioapic->lock)
 {
 	u32 idx;
 
@@ -330,6 +343,7 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin,
 }
 
 static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val)
+	__must_hold(&ioapic->lock)
 {
 	unsigned index;
 	bool mask_before, mask_after;
@@ -541,6 +555,7 @@ static void kvm_ioapic_update_eoi_one(struct kvm_vcpu *vcpu,
 				      struct kvm_ioapic *ioapic,
 				      int trigger_mode,
 				      int pin)
+	__must_hold(&ioapic->lock)
 {
 	struct kvm_lapic *apic = vcpu->arch.apic;
 	union kvm_ioapic_redirect_entry *ent = &ioapic->redirtbl[pin];
@@ -695,6 +710,7 @@ static int ioapic_mmio_write(struct kvm_vcpu *vcpu, struct kvm_io_device *this,
 }
 
 static void kvm_ioapic_reset(struct kvm_ioapic *ioapic)
+	__must_hold(&ioapic->lock)
 {
 	int i;
 
diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c
index 871977288272..5dd3bfe2aeb4 100644
--- a/arch/x86/kvm/irq.c
+++ b/arch/x86/kvm/irq.c
@@ -416,6 +416,8 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 void kvm_arch_irq_routing_update(struct kvm *kvm)
 {
 #ifdef CONFIG_KVM_HYPERV
+	/* Update-side mutex kvm->irq_lock is held. */
+	__assume_shared_ctx_lock(&kvm->irq_srcu);
 	kvm_hv_irq_routing_update(kvm);
 #endif
 
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index e1f3cea14765..f5022de46725 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -901,6 +901,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low,
 }
 
 static int pv_eoi_put_user(struct kvm_vcpu *vcpu, u8 val)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 
 	return kvm_write_guest_cached(vcpu->kvm, &vcpu->arch.pv_eoi.data, &val,
@@ -908,6 +909,7 @@ static int pv_eoi_put_user(struct kvm_vcpu *vcpu, u8 val)
 }
 
 static int pv_eoi_get_user(struct kvm_vcpu *vcpu, u8 *val)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 
 	return kvm_read_guest_cached(vcpu->kvm, &vcpu->arch.pv_eoi.data, val,
@@ -920,6 +922,7 @@ static inline bool pv_eoi_enabled(struct kvm_vcpu *vcpu)
 }
 
 static void pv_eoi_set_pending(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (pv_eoi_put_user(vcpu, KVM_PV_EOI_ENABLED) < 0)
 		return;
@@ -928,6 +931,7 @@ static void pv_eoi_set_pending(struct kvm_vcpu *vcpu)
 }
 
 static bool pv_eoi_test_and_clr_pending(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u8 val;
 
@@ -3378,6 +3382,7 @@ void __kvm_migrate_apic_timer(struct kvm_vcpu *vcpu)
  */
 static void apic_sync_pv_eoi_from_guest(struct kvm_vcpu *vcpu,
 					struct kvm_lapic *apic)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int vector;
 
@@ -3428,6 +3433,7 @@ void kvm_lapic_sync_from_vapic(struct kvm_vcpu *vcpu)
  */
 static void apic_sync_pv_eoi_to_guest(struct kvm_vcpu *vcpu,
 					struct kvm_lapic *apic)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (!pv_eoi_enabled(vcpu) ||
 	    /* IRR set or many bits in ISR: could be nested. */
@@ -3443,7 +3449,7 @@ static void apic_sync_pv_eoi_to_guest(struct kvm_vcpu *vcpu,
 		return;
 	}
 
-	pv_eoi_set_pending(apic->vcpu);
+	pv_eoi_set_pending(vcpu);
 }
 
 void kvm_lapic_sync_to_vapic(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h
index bd1098c89d99..a193f32a5688 100644
--- a/arch/x86/kvm/lapic.h
+++ b/arch/x86/kvm/lapic.h
@@ -148,7 +148,8 @@ int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq,
 int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type);
 void kvm_apic_update_apicv(struct kvm_vcpu *vcpu);
 int kvm_alloc_apic_access_page(struct kvm *kvm);
-void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu);
+void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 bool kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *src,
 				   struct kvm_lapic_irq *irq, int *r);
@@ -179,9 +180,12 @@ void kvm_set_lapic_tscdeadline_msr(struct kvm_vcpu *vcpu, u64 data);
 void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset);
 void kvm_apic_set_eoi_accelerated(struct kvm_vcpu *vcpu, int vector);
 
-int kvm_lapic_set_vapic_addr(struct kvm_vcpu *vcpu, gpa_t vapic_addr);
-void kvm_lapic_sync_from_vapic(struct kvm_vcpu *vcpu);
-void kvm_lapic_sync_to_vapic(struct kvm_vcpu *vcpu);
+int kvm_lapic_set_vapic_addr(struct kvm_vcpu *vcpu, gpa_t vapic_addr)
+	__must_hold_shared(&vcpu->kvm->srcu);
+void kvm_lapic_sync_from_vapic(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
+void kvm_lapic_sync_to_vapic(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 int kvm_x2apic_icr_write_fast(struct kvm_lapic *apic, u64 data);
 int kvm_x2apic_msr_write(struct kvm_vcpu *vcpu, u32 msr, u64 data);
@@ -190,7 +194,8 @@ int kvm_x2apic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data);
 int kvm_hv_vapic_msr_write(struct kvm_vcpu *vcpu, u32 msr, u64 data);
 int kvm_hv_vapic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data);
 
-int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len);
+int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void kvm_lapic_exit(void);
 
 u64 kvm_x2apic_disable_read_intercept_reg_mask(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c
index 7e8195a311bb..058a26d8065a 100644
--- a/arch/x86/kvm/mmu/page_track.c
+++ b/arch/x86/kvm/mmu/page_track.c
@@ -196,6 +196,7 @@ static int kvm_enable_external_write_tracking(struct kvm *kvm)
 		return -EOPNOTSUPP;
 
 	mutex_lock(&kvm->slots_arch_lock);
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 
 	/*
 	 * Check for *any* write tracking user (not just external users) under
diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h
index d4d72ed999b1..8f7e176af858 100644
--- a/arch/x86/kvm/mmu/page_track.h
+++ b/arch/x86/kvm/mmu/page_track.h
@@ -21,7 +21,8 @@ void __kvm_write_track_remove_gfn(struct kvm *kvm,
 				  struct kvm_memory_slot *slot, gfn_t gfn);
 
 bool kvm_gfn_is_write_tracked(struct kvm *kvm,
-			      const struct kvm_memory_slot *slot, gfn_t gfn);
+			      const struct kvm_memory_slot *slot, gfn_t gfn)
+	__must_hold_shared(&kvm->srcu);
 
 #ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
 int kvm_page_track_init(struct kvm *kvm);
diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h
index e730717824b3..cf8fc077ce04 100644
--- a/arch/x86/kvm/mmu/spte.h
+++ b/arch/x86/kvm/mmu/spte.h
@@ -568,7 +568,8 @@ u64 make_small_spte(struct kvm *kvm, u64 huge_spte,
 		    union kvm_mmu_page_role role, int index);
 u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level);
 u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled);
-u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access);
+u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access)
+	__must_hold_shared(&vcpu->kvm->srcu);
 u64 mark_spte_for_access_track(u64 spte);
 
 /* Restore an acc-track PTE back to a regular PTE */
diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index dd3bb04878ca..a90d6576e3cf 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -1222,6 +1222,7 @@ static void kvm_get_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 }
 
 static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int sec_hi_ofs)
+	__must_hold_shared(&kvm->srcu)
 {
 	int version;
 	int r;
@@ -1264,9 +1265,12 @@ static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int sec_hi_o
 
 static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time,
 				  bool old_msr, bool host_initiated)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_arch *ka = &vcpu->kvm->arch;
 
+	lockdep_assert_held(&vcpu->arch.pv_time.kvm->srcu); /* vcpu->arch.pv_time.kvm == vcpu->kvm */
+
 	if (vcpu->vcpu_id == 0 && !host_initiated) {
 		if (ka->boot_vcpu_runs_old_kvmclock != old_msr)
 			kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
@@ -1382,6 +1386,7 @@ static int set_msr_mce(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 }
 
 static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	gpa_t gpa = data & ~0x3f;
 
diff --git a/arch/x86/kvm/msrs.h b/arch/x86/kvm/msrs.h
index 7cc182a15b3b..0736b0f61873 100644
--- a/arch/x86/kvm/msrs.h
+++ b/arch/x86/kvm/msrs.h
@@ -74,7 +74,8 @@ fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu);
 fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg);
 
 int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr);
-int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr);
+int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 int kvm_add_user_return_msr(u32 msr);
 int kvm_find_user_return_msr(u32 msr);
diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h
index 447f0ec3e63e..3f6d4a0ac97f 100644
--- a/arch/x86/kvm/regs.h
+++ b/arch/x86/kvm/regs.h
@@ -56,15 +56,20 @@ static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_PDPTR_BITS));
 
 void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0);
 void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4);
-int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0);
-int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3);
-int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4);
+int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8);
 int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val);
 unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr);
 unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu);
-void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw);
-int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
+void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 static inline bool is_long_mode(struct kvm_vcpu *vcpu)
 {
diff --git a/arch/x86/kvm/smm.c b/arch/x86/kvm/smm.c
index 656a38dad7e7..5f93257961c3 100644
--- a/arch/x86/kvm/smm.c
+++ b/arch/x86/kvm/smm.c
@@ -418,6 +418,7 @@ static int rsm_load_seg_64(struct kvm_vcpu *vcpu,
 
 static int rsm_enter_protected_mode(struct kvm_vcpu *vcpu,
 				    u64 cr0, u64 cr3, u64 cr4)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int bad;
 	u64 pcid;
@@ -463,6 +464,7 @@ static int rsm_enter_protected_mode(struct kvm_vcpu *vcpu,
 
 static int rsm_load_state_32(struct x86_emulate_ctxt *ctxt,
 			     const struct kvm_smram_state_32 *smstate)
+	__must_hold_shared(&((struct kvm_vcpu *)ctxt->vcpu)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = ctxt->vcpu;
 	struct desc_ptr dt;
@@ -515,6 +517,7 @@ static int rsm_load_state_32(struct x86_emulate_ctxt *ctxt,
 #ifdef CONFIG_X86_64
 static int rsm_load_state_64(struct x86_emulate_ctxt *ctxt,
 			     const struct kvm_smram_state_64 *smstate)
+	__must_hold_shared(&((struct kvm_vcpu *)ctxt->vcpu)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = ctxt->vcpu;
 	struct desc_ptr dt;
@@ -578,6 +581,9 @@ int emulator_leave_smm(struct x86_emulate_ctxt *ctxt)
 	u64 smbase;
 	int ret;
 
+	/* Called via struct x86_emulate_ops callback; assert SRCU dynamically. */
+	lockdep_assert_held(&vcpu->kvm->srcu);
+
 	smbase = vcpu->arch.smbase;
 
 	ret = kvm_vcpu_read_guest(vcpu, smbase + 0xfe00, smram.bytes, sizeof(smram));
diff --git a/arch/x86/kvm/smm.h b/arch/x86/kvm/smm.h
index db3c88f16138..6c15db642538 100644
--- a/arch/x86/kvm/smm.h
+++ b/arch/x86/kvm/smm.h
@@ -155,7 +155,7 @@ static inline bool is_smm(struct kvm_vcpu *vcpu)
 }
 
 void kvm_smm_changed(struct kvm_vcpu *vcpu, bool in_smm);
-void enter_smm(struct kvm_vcpu *vcpu);
+void enter_smm(struct kvm_vcpu *vcpu) __must_hold_shared(&vcpu->kvm->srcu);
 int emulator_leave_smm(struct x86_emulate_ctxt *ctxt);
 void process_smi(struct kvm_vcpu *vcpu);
 #else
diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c
index 4f24dcb45116..117c069c5bfa 100644
--- a/arch/x86/kvm/svm/hyperv.c
+++ b/arch/x86/kvm/svm/hyperv.c
@@ -10,6 +10,8 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	/*
 	 * The exit code used by Hyper-V for software-defined exits is reserved
 	 * by AMD specifically for such use cases.
diff --git a/arch/x86/kvm/svm/hyperv.h b/arch/x86/kvm/svm/hyperv.h
index f70d076911a6..e9990cabd332 100644
--- a/arch/x86/kvm/svm/hyperv.h
+++ b/arch/x86/kvm/svm/hyperv.h
@@ -48,14 +48,16 @@ static inline bool nested_svm_is_l2_tlb_flush_hcall(struct kvm_vcpu *vcpu)
 	       kvm_hv_is_tlb_flush_hcall(vcpu);
 }
 
-void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu);
+void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 #else /* CONFIG_KVM_HYPERV */
 static inline void nested_svm_hv_update_vm_vp_ids(struct kvm_vcpu *vcpu) {}
 static inline bool nested_svm_is_l2_tlb_flush_hcall(struct kvm_vcpu *vcpu)
 {
 	return false;
 }
-static inline void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu) {}
+static inline void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu) {}
 #endif /* CONFIG_KVM_HYPERV */
 
 #endif /* __ARCH_X86_KVM_SVM_HYPERV_H__ */
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 73f37b050d0a..d421d6807166 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -38,9 +38,12 @@
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 				       struct x86_exception *fault,
 				       bool from_hardware)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb *vmcb = svm->vmcb;
+
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
 	u64 fault_stage;
 
 	/*
@@ -72,6 +75,7 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 }
 
 static u64 nested_svm_get_tdp_pdptr(struct kvm_vcpu *vcpu, int index)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	u64 cr3 = svm->nested.ctl.nested_cr3;
@@ -293,6 +297,7 @@ int __init nested_svm_init_msrpm_merge_offsets(void)
  * may contain zero bits.
  */
 static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	nsvm_msrpm_merge_t *msrpm02 = svm->nested.msrpm;
@@ -713,6 +718,7 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
  */
 static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_npt, bool reload_pdptrs)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3)))
 		return -EINVAL;
@@ -1065,7 +1071,7 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
 	nested_vmcb02_prepare_control(svm);
 	nested_vmcb02_prepare_save(svm);
 
-	ret = nested_svm_load_cr3(&svm->vcpu, svm->nested.save.cr3,
+	ret = nested_svm_load_cr3(vcpu, svm->nested.save.cr3,
 				  nested_npt_enabled(svm), from_vmrun);
 	if (ret)
 		return ret;
@@ -1117,6 +1123,8 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
 	u64 vmcb12_gpa;
 	struct vmcb *vmcb01 = svm->vmcb01.ptr;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	if (!svm->nested.hsave_msr) {
 		kvm_inject_gp(vcpu, 0);
 		return 1;
@@ -1320,6 +1328,8 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	struct vmcb *vmcb01 = svm->vmcb01.ptr;
 	struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
 
+	lockdep_assert_held(&vcpu->kvm->srcu); /* vcpu == &svm->vcpu */
+
 	if (nested_svm_vmexit_update_vmcb12(vcpu))
 		kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
 
@@ -1467,14 +1477,17 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 }
 
 static void nested_svm_triple_fault(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	if (!vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_SHUTDOWN))
 		return;
 
 	kvm_clear_request(KVM_REQ_TRIPLE_FAULT, vcpu);
-	nested_svm_simple_vmexit(to_svm(vcpu), SVM_EXIT_SHUTDOWN);
+	nested_svm_simple_vmexit(svm, SVM_EXIT_SHUTDOWN);
 }
 
 int svm_allocate_nested(struct vcpu_svm *svm)
@@ -1562,6 +1575,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
 }
 
 static int nested_svm_exit_handled_msr(struct vcpu_svm *svm)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	gpa_t base = svm->nested.ctl.msrpm_base_pa;
 	int write, bit_nr;
@@ -1587,6 +1601,7 @@ static int nested_svm_exit_handled_msr(struct vcpu_svm *svm)
 }
 
 static int nested_svm_intercept_ioio(struct vcpu_svm *svm)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	unsigned port, size, iopm_len;
 	u16 val, mask;
@@ -1612,6 +1627,7 @@ static int nested_svm_intercept_ioio(struct vcpu_svm *svm)
 }
 
 static int nested_svm_intercept(struct vcpu_svm *svm)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	u64 exit_code = svm->vmcb->control.exit_code;
 	int vmexit = NESTED_EXIT_HOST;
@@ -1679,11 +1695,14 @@ static bool nested_svm_is_exception_vmexit(struct kvm_vcpu *vcpu, u8 vector,
 }
 
 static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit;
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb *vmcb = svm->vmcb;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + ex->vector;
 
 	if (ex->has_error_code)
@@ -1719,9 +1738,12 @@ static inline bool nested_exit_on_init(struct vcpu_svm *svm)
 }
 
 static int svm_check_nested_events(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_lapic *apic = vcpu->arch.apic;
 	struct vcpu_svm *svm = to_svm(vcpu);
+
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
 	/*
 	 * Only a pending nested run blocks a pending exception.  If there is a
 	 * previously injected event, the pending exception occurred while said
@@ -1946,6 +1968,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu,
 static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 				struct kvm_nested_state __user *user_kvm_nested_state,
 				struct kvm_nested_state *kvm_state)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb __user *user_vmcb = (struct vmcb __user *)
@@ -2086,7 +2109,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	 * thus MMU might not be initialized correctly.
 	 * Set it again to fix this.
 	 */
-	ret = nested_svm_load_cr3(&svm->vcpu, vcpu->arch.cr3,
+	ret = nested_svm_load_cr3(vcpu, vcpu->arch.cr3,
 				  nested_npt_enabled(svm), false);
 	if (ret)
 		goto out_free;
@@ -2105,6 +2128,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 }
 
 static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (WARN_ON(!is_guest_mode(vcpu)))
 		return true;
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 5705723f1f41..ec9589cd0abe 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -1117,6 +1117,7 @@ static int __sev_launch_update_vmsa(struct kvm *kvm, struct kvm_vcpu *vcpu,
 }
 
 static int sev_launch_update_vmsa(struct kvm *kvm, struct kvm_sev_cmd *argp)
+	__must_hold(&kvm->lock)
 {
 	struct kvm_vcpu *vcpu;
 	unsigned long i;
@@ -1981,6 +1982,8 @@ static bool is_cmd_allowed_from_mirror(u32 cmd_id)
 }
 
 static int sev_lock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
+	__cond_acquires(0, &dst_kvm->lock)
+	__cond_acquires(0, &src_kvm->lock)
 {
 	struct kvm_sev_info *dst_sev = to_kvm_sev_info(dst_kvm);
 	struct kvm_sev_info *src_sev = to_kvm_sev_info(src_kvm);
@@ -2016,6 +2019,8 @@ static int sev_lock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
 }
 
 static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
+	__releases(&dst_kvm->lock)
+	__releases(&src_kvm->lock)
 {
 	struct kvm_sev_info *dst_sev = to_kvm_sev_info(dst_kvm);
 	struct kvm_sev_info *src_sev = to_kvm_sev_info(src_kvm);
@@ -2473,6 +2478,7 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	 * situations.
 	 */
 	guard(mutex)(&kvm->slots_lock);
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
 
 	memslot = gfn_to_memslot(kvm, params.gfn_start);
 	if (!kvm_slot_has_gmem(memslot))
@@ -2503,6 +2509,7 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)
 }
 
 static int snp_launch_update_vmsa(struct kvm *kvm, struct kvm_sev_cmd *argp)
+	__must_hold(&kvm->lock)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	struct sev_data_snp_launch_update data = {};
@@ -2560,6 +2567,7 @@ static int snp_launch_update_vmsa(struct kvm *kvm, struct kvm_sev_cmd *argp)
 }
 
 static int snp_launch_finish(struct kvm *kvm, struct kvm_sev_cmd *argp)
+	__must_hold(&kvm->lock)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	struct kvm_sev_snp_launch_finish params;
@@ -3622,6 +3630,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 
 #define GHCB_SCRATCH_AREA_LIMIT		(16ULL * PAGE_SIZE)
 static int setup_vmgexit_scratch(struct vcpu_svm *svm, bool sync, u64 min_len)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	struct vmcb_control_area *control = &svm->vmcb->control;
 	u64 ghcb_scratch_beg, ghcb_scratch_end;
@@ -4036,6 +4045,7 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	kvm_pfn_t pfn;
 
 	lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex);
+	lockdep_assert_held(&vcpu->kvm->srcu);
 
 	/*
 	 * Clear use of the VMSA.  Ensure snp_guest_vmsa_gpa is written exactly
@@ -4216,6 +4226,7 @@ static int sev_snp_ap_creation(struct vcpu_svm *svm)
 }
 
 static int snp_handle_guest_req(struct vcpu_svm *svm, gpa_t req_gpa, gpa_t resp_gpa)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	struct sev_data_snp_guest_request data = {0};
 	struct kvm *kvm = svm->vcpu.kvm;
@@ -4262,10 +4273,13 @@ static int snp_req_certs_err(struct vcpu_svm *svm, u32 vmm_error)
 }
 
 static int snp_complete_req_certs(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb_control_area *control = &svm->vmcb->control;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	switch (READ_ONCE(vcpu->run->snp_req_certs.ret)) {
 	case 0:
 		return snp_handle_guest_req(svm, control->exit_info_1,
@@ -4285,6 +4299,7 @@ static int snp_complete_req_certs(struct kvm_vcpu *vcpu)
 }
 
 static int snp_handle_ext_guest_req(struct vcpu_svm *svm, gpa_t req_gpa, gpa_t resp_gpa)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 	struct kvm *kvm = vcpu->kvm;
@@ -4347,6 +4362,7 @@ static int snp_handle_ext_guest_req(struct vcpu_svm *svm, gpa_t req_gpa, gpa_t r
 }
 
 static int sev_handle_vmgexit_msr_protocol(struct vcpu_svm *svm)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	struct vmcb_control_area *control = &svm->vmcb->control;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
@@ -4509,6 +4525,8 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 	struct vmcb_control_area *control = &svm->vmcb->control;
 	u64 ghcb_gpa;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	/* Validate the GHCB */
 	ghcb_gpa = control->ghcb_gpa;
 	if (ghcb_gpa & GHCB_MSR_INFO_MASK)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 7d59d301e1e5..c20b6a9fe8dd 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1432,10 +1432,13 @@ static void svm_srso_vm_destroy(void) { }
 #endif
 
 static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	if (is_sev_es_guest(vcpu))
 		sev_es_unmap_ghcb(svm);
 
@@ -1533,6 +1536,7 @@ static bool svm_get_if_flag(struct kvm_vcpu *vcpu)
 }
 
 static void svm_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	kvm_register_mark_available(vcpu, reg);
 
@@ -1972,6 +1976,7 @@ static int svm_check_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
 					 void *insn, int insn_len);
 
 static int npf_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	int rc;
@@ -2094,6 +2099,7 @@ static int icebp_interception(struct kvm_vcpu *vcpu)
 }
 
 static int ud_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return handle_ud(vcpu);
 }
@@ -2193,12 +2199,15 @@ static int shutdown_interception(struct kvm_vcpu *vcpu)
 }
 
 static int io_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	u32 io_info = svm->vmcb->control.exit_info_1; /* address size bug? */
 	int size, in, string;
 	unsigned port;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	++vcpu->stat.io_exits;
 	string = (io_info & SVM_IOIO_STR_MASK) != 0;
 	in = (io_info & SVM_IOIO_TYPE_MASK) != 0;
@@ -2279,6 +2288,7 @@ static int vmsave_interception(struct kvm_vcpu *vcpu)
 }
 
 static int vmrun_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (nested_svm_check_permissions(vcpu))
 		return 1;
@@ -2319,11 +2329,14 @@ static u64 svm_get_decoded_instr_exit_code(struct kvm_vcpu *vcpu)
  *   2) VMware backdoor
  */
 static int gp_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	u32 error_code = svm->vmcb->control.exit_info_1;
 	u64 svm_exit_code;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	/* Both #GP cases have zero error_code */
 	if (error_code)
 		goto reinject;
@@ -2562,11 +2575,14 @@ static int rsm_interception(struct kvm_vcpu *vcpu)
 
 static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
 					    unsigned long val)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	unsigned long cr0 = vcpu->arch.cr0;
 	bool ret = false;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	if (!is_guest_mode(vcpu) ||
 	    (!(vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_SELECTIVE_CR0))))
 		return false;
@@ -2585,6 +2601,7 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
 #define CR_VALID (1ULL << 63)
 
 static int cr_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	int reg, cr;
@@ -2733,6 +2750,7 @@ static int dr_interception(struct kvm_vcpu *vcpu)
 }
 
 static int cr8_write_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u8 cr8_prev = kvm_get_cr8(vcpu);
 	int r;
@@ -2750,6 +2768,7 @@ static int cr8_write_interception(struct kvm_vcpu *vcpu)
 }
 
 static int efer_trap(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct msr_data msr_info;
 	int ret;
@@ -2988,6 +3007,7 @@ static int svm_set_vm_cr(struct kvm_vcpu *vcpu, u64 data)
 }
 
 static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	int ret = 0;
@@ -3265,6 +3285,7 @@ static int pause_interception(struct kvm_vcpu *vcpu)
 }
 
 static int invpcid_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	unsigned long type;
@@ -3329,6 +3350,7 @@ static int bus_lock_exit(struct kvm_vcpu *vcpu)
 }
 
 static int vmmcall_interception(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	/*
 	 * Inject a #UD if L2 is active and the VMMCALL isn't a Hyper-V TLB
@@ -3713,10 +3735,13 @@ static void svm_get_entry_info(struct kvm_vcpu *vcpu, u32 *intr_info,
 }
 
 static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_run *kvm_run = vcpu->run;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE))
 		return 0;
 
@@ -4832,12 +4857,15 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 			       struct x86_instruction_info *info,
 			       enum x86_intercept_stage stage,
 			       struct x86_exception *exception)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	int vmexit, ret = X86EMUL_CONTINUE;
 	struct __x86_intercept icpt_info;
 	struct vmcb *vmcb = svm->vmcb;
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	if (info->intercept >= ARRAY_SIZE(x86_intercept_map))
 		goto out;
 
@@ -5011,9 +5039,12 @@ static int svm_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection)
 }
 
 static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 
+	lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm == vcpu->kvm */
+
 	if (!is_guest_mode(vcpu))
 		return 0;
 
@@ -5057,6 +5088,7 @@ static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram)
 }
 
 static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb *vmcb12;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index e958943b8162..e62700fc678d 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -818,7 +818,8 @@ bool svm_smi_blocked(struct kvm_vcpu *vcpu);
 bool svm_nmi_blocked(struct kvm_vcpu *vcpu);
 bool svm_interrupt_blocked(struct kvm_vcpu *vcpu);
 void svm_set_gif(struct vcpu_svm *svm, bool value);
-int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code);
+int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void set_msr_interception(struct kvm_vcpu *vcpu, u32 *msrpm, u32 msr,
 			  int read, int write);
 void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode,
@@ -870,17 +871,21 @@ static inline bool nested_exit_on_nmi(struct vcpu_svm *svm)
 
 int __init nested_svm_init_msrpm_merge_offsets(void);
 
-int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb_gpa, bool from_vmrun);
+int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb_gpa, bool from_vmrun)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void svm_leave_nested(struct kvm_vcpu *vcpu);
 void svm_free_nested(struct vcpu_svm *svm);
 int svm_allocate_nested(struct vcpu_svm *svm);
-int nested_svm_vmrun(struct kvm_vcpu *vcpu);
+int nested_svm_vmrun(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void svm_copy_vmrun_state(struct vmcb_save_area *to_save,
 			  struct vmcb_save_area *from_save);
 void svm_copy_vmloadsave_state(struct vmcb *to_vmcb, struct vmcb *from_vmcb);
-void nested_svm_vmexit(struct vcpu_svm *svm);
+void nested_svm_vmexit(struct vcpu_svm *svm)
+	__must_hold_shared(&svm->vcpu.kvm->srcu);
 
 static inline void nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
+	__must_hold_shared(&svm->vcpu.kvm->srcu)
 {
 	svm->vmcb->control.exit_code	= exit_code;
 	svm->vmcb->control.exit_info_1	= 0;
@@ -888,7 +893,8 @@ static inline void nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
 	nested_svm_vmexit(svm);
 }
 
-int nested_svm_exit_handled(struct vcpu_svm *svm);
+int nested_svm_exit_handled(struct vcpu_svm *svm)
+	__must_hold_shared(&svm->vcpu.kvm->srcu);
 int nested_svm_check_permissions(struct kvm_vcpu *vcpu);
 int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu);
 int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr,
@@ -973,11 +979,13 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu);
 int pre_sev_run(struct vcpu_svm *svm, int cpu);
 void sev_init_vmcb(struct vcpu_svm *svm, bool init_event);
 void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm);
-int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in);
+int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in)
+	__must_hold_shared(&svm->vcpu.kvm->srcu);
 void sev_es_recalc_msr_intercepts(struct kvm_vcpu *vcpu);
 void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector);
 void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_save_area *hostsa);
-void sev_es_unmap_ghcb(struct vcpu_svm *svm);
+void sev_es_unmap_ghcb(struct vcpu_svm *svm)
+	__must_hold_shared(&svm->vcpu.kvm->srcu);
 
 #ifdef CONFIG_KVM_AMD_SEV
 bool sev_vcpu_needs_initialization(struct kvm_vcpu *vcpu);
@@ -989,7 +997,8 @@ int sev_mem_enc_unregister_region(struct kvm *kvm,
 int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd);
 int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd);
 void sev_guest_memory_reclaimed(struct kvm *kvm);
-int sev_handle_vmgexit(struct kvm_vcpu *vcpu);
+int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 /* These symbols are used in common code and are stubbed below.  */
 
@@ -1010,7 +1019,8 @@ void sev_hardware_unsetup(void);
 int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
-void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
+void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages);
 void sev_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
diff --git a/arch/x86/kvm/vmx/hyperv.h b/arch/x86/kvm/vmx/hyperv.h
index 11a339009781..a0e126d2a5b4 100644
--- a/arch/x86/kvm/vmx/hyperv.h
+++ b/arch/x86/kvm/vmx/hyperv.h
@@ -52,14 +52,16 @@ static inline bool guest_cpu_cap_has_evmcs(struct kvm_vcpu *vcpu)
 	       to_vmx(vcpu)->nested.enlightened_vmcs_enabled;
 }
 
-u64 nested_get_evmptr(struct kvm_vcpu *vcpu);
+u64 nested_get_evmptr(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu);
 int nested_enable_evmcs(struct kvm_vcpu *vcpu,
 			uint16_t *vmcs_version);
 void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata);
 int nested_evmcs_check_controls(struct vmcs12 *vmcs12);
 bool nested_evmcs_l2_tlb_flush_enabled(struct kvm_vcpu *vcpu);
-void vmx_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu);
+void vmx_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 #else
 static inline bool evmptr_is_valid(u64 evmptr)
 {
diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c
index 4c52ab8d0786..c2a0c259ceea 100644
--- a/arch/x86/kvm/vmx/main.c
+++ b/arch/x86/kvm/vmx/main.c
@@ -156,6 +156,7 @@ static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
 static int vt_handle_exit(struct kvm_vcpu *vcpu,
 			  enum exit_fastpath_completion fastpath)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (is_td_vcpu(vcpu))
 		return tdx_handle_exit(vcpu, fastpath);
@@ -174,6 +175,7 @@ static bool vt_unhandleable_emulation_required(struct kvm_vcpu *vcpu)
 }
 
 static int vt_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (unlikely(is_td_vcpu(vcpu)))
 		return tdx_set_msr(vcpu, msr_info);
@@ -233,6 +235,7 @@ static int vt_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection)
 }
 
 static int vt_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm))
 		return 0;
@@ -241,6 +244,7 @@ static int vt_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram)
 }
 
 static int vt_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm))
 		return 0;
@@ -722,6 +726,7 @@ static void vt_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)
 }
 
 static void vt_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (is_td_vcpu(vcpu))
 		return;
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 151873407abd..506e206bad5d 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -250,6 +250,7 @@ static inline void nested_release_evmcs(struct kvm_vcpu *vcpu)
 }
 
 static bool nested_evmcs_handle_vmclear(struct kvm_vcpu *vcpu, gpa_t vmptr)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 #ifdef CONFIG_KVM_HYPERV
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -386,7 +387,8 @@ static void free_nested(struct kvm_vcpu *vcpu)
 void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu)
 {
 	vcpu_load(vcpu);
-	vmx_leave_nested(vcpu);
+	/* VM is being destroyed; vcpu->kvm->srcu is not held. */
+	context_unsafe(vmx_leave_nested(vcpu));
 	vcpu_put(vcpu);
 }
 
@@ -421,6 +423,7 @@ static void nested_ept_invalidate_addr(struct kvm_vcpu *vcpu, gpa_t eptp,
 static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu,
 					 struct x86_exception *fault,
 					 bool from_hardware)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -587,6 +590,7 @@ static int nested_vmx_check_msr_bitmap_controls(struct kvm_vcpu *vcpu,
 
 static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu,
 						struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	gpa_t vtpr_gpa = vmcs12->virtual_apic_page_addr + APIC_TASKPRI;
 	u32 vtpr;
@@ -850,6 +854,7 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
 
 static void nested_cache_shadow_vmcs12(struct kvm_vcpu *vcpu,
 				       struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct gfn_to_hva_cache *ghc = &vmx->nested.shadow_vmcs12_cache;
@@ -869,6 +874,7 @@ static void nested_cache_shadow_vmcs12(struct kvm_vcpu *vcpu,
 
 static void nested_flush_cached_shadow_vmcs12(struct kvm_vcpu *vcpu,
 					      struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct gfn_to_hva_cache *ghc = &vmx->nested.shadow_vmcs12_cache;
@@ -1095,6 +1101,7 @@ static int nested_vmx_store_msr_check(struct kvm_vcpu *vcpu,
  * for a capacity violation.
  */
 static u32 nested_vmx_load_msr(struct kvm_vcpu *vcpu, u64 gpa, u32 count)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 i;
 	struct vmx_msr_entry e;
@@ -1159,6 +1166,7 @@ static bool nested_vmx_get_vmexit_msr_value(struct kvm_vcpu *vcpu,
 
 static bool read_and_check_msr_entry(struct kvm_vcpu *vcpu, u64 gpa, int i,
 				     struct vmx_msr_entry *e)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (kvm_vcpu_read_guest(vcpu,
 				gpa + i * sizeof(*e),
@@ -1178,6 +1186,7 @@ static bool read_and_check_msr_entry(struct kvm_vcpu *vcpu, u64 gpa, int i,
 }
 
 static int nested_vmx_store_msr(struct kvm_vcpu *vcpu, u64 gpa, u32 count)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u64 data;
 	u32 i;
@@ -1208,6 +1217,7 @@ static int nested_vmx_store_msr(struct kvm_vcpu *vcpu, u64 gpa, u32 count)
 }
 
 static bool nested_msr_store_list_has_msr(struct kvm_vcpu *vcpu, u32 msr_index)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 	u32 count = vmcs12->vm_exit_msr_store_count;
@@ -1234,6 +1244,7 @@ static bool nested_msr_store_list_has_msr(struct kvm_vcpu *vcpu, u32 msr_index)
 static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3,
 			       bool nested_ept, bool reload_pdptrs,
 			       enum vm_entry_failure_code *entry_failure_code)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) {
 		*entry_failure_code = ENTRY_FAIL_DEFAULT;
@@ -2183,6 +2194,7 @@ static void copy_vmcs12_to_enlightened(struct vcpu_vmx *vmx)
  */
 static enum nested_evmptrld_status nested_vmx_handle_enlightened_vmptrld(
 	struct kvm_vcpu *vcpu, bool from_launch)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 #ifdef CONFIG_KVM_HYPERV
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -2622,6 +2634,7 @@ static void vmcs_write_cet_state(struct kvm_vcpu *vcpu, u64 s_cet,
 }
 
 static void prepare_vmcs02_rare(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)
+	__must_hold_shared(&vmx->vcpu.kvm->srcu)
 {
 	struct hv_enlightened_vmcs *hv_evmcs = nested_vmx_evmcs(vmx);
 
@@ -2760,11 +2773,14 @@ static void prepare_vmcs02_rare(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)
 static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,
 			  bool from_vmentry,
 			  enum vm_entry_failure_code *entry_failure_code)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct hv_enlightened_vmcs *evmcs = nested_vmx_evmcs(vmx);
 	bool load_guest_pdptrs_vmcs12 = false;
 
+	lockdep_assert_held(&vmx->vcpu.kvm->srcu); /* vmx->vcpu.kvm == vcpu->kvm */
+
 	if (vmx->nested.dirty_vmcs12 || nested_vmx_is_evmptr12_valid(vmx)) {
 		prepare_vmcs02_rare(vmx, vmcs12);
 		vmx->nested.dirty_vmcs12 = false;
@@ -2968,6 +2984,7 @@ static bool nested_vmx_check_eptp(struct kvm_vcpu *vcpu, u64 new_eptp)
  */
 static int nested_check_vm_execution_controls(struct kvm_vcpu *vcpu,
                                               struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 
@@ -3126,6 +3143,7 @@ static int nested_check_vm_entry_controls(struct kvm_vcpu *vcpu,
 
 static int nested_vmx_check_controls(struct kvm_vcpu *vcpu,
 				     struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (nested_check_vm_execution_controls(vcpu, vmcs12) ||
 	    nested_check_vm_exit_controls(vcpu, vmcs12) ||
@@ -3267,6 +3285,7 @@ static int nested_vmx_check_host_state(struct kvm_vcpu *vcpu,
 
 static int nested_vmx_check_vmcs_link_ptr(struct kvm_vcpu *vcpu,
 					  struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct gfn_to_hva_cache *ghc = &vmx->nested.shadow_vmcs12_cache;
@@ -3311,6 +3330,7 @@ static int nested_check_guest_non_reg_state(struct vmcs12 *vmcs12)
 static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu,
 					struct vmcs12 *vmcs12,
 					enum vm_entry_failure_code *entry_failure_code)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	bool ia32e = !!(vmcs12->vm_entry_controls & VM_ENTRY_IA32E_MODE);
 
@@ -3411,6 +3431,7 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu,
 
 #ifdef CONFIG_KVM_HYPERV
 static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 
@@ -3440,6 +3461,7 @@ static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu)
 #endif
 
 static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -3528,6 +3550,7 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
 }
 
 static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 #ifdef CONFIG_KVM_HYPERV
 	/*
@@ -3555,6 +3578,7 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
 }
 
 static int nested_vmx_write_pml_buffer(struct kvm_vcpu *vcpu, gpa_t gpa)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12;
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -3790,6 +3814,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
  * for running an L2 nested guest.
  */
 static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12;
 	enum nvmx_vmentry_status status;
@@ -4027,6 +4052,7 @@ static void vmcs12_save_pending_event(struct kvm_vcpu *vcpu,
 }
 
 static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	int max_irr;
@@ -4070,6 +4096,7 @@ static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
 }
 
 static void nested_vmx_inject_exception_vmexit(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit;
 	u32 intr_info = ex->vector | INTR_INFO_VALID_MASK;
@@ -4303,6 +4330,7 @@ static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injection)
  *     priority over external interrupts and lower priority events.
  */
 static int vmx_check_nested_events(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_lapic *apic = vcpu->arch.apic;
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -4744,6 +4772,7 @@ static void sync_vmcs02_to_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12)
 static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,
 			   u32 vm_exit_reason, u32 exit_intr_info,
 			   unsigned long exit_qualification, u32 exit_insn_len)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	/* update exit information fields: */
 	vmcs12->vm_exit_reason = vm_exit_reason;
@@ -4799,6 +4828,7 @@ static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,
  */
 static void load_vmcs12_host_state(struct kvm_vcpu *vcpu,
 				   struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	enum vm_entry_failure_code ignored;
 	struct kvm_segment seg;
@@ -4958,6 +4988,7 @@ static inline u64 nested_vmx_get_vmcs01_guest_efer(struct vcpu_vmx *vmx)
 }
 
 static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -5230,6 +5261,7 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 }
 
 static void nested_vmx_triple_fault(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	kvm_clear_request(KVM_REQ_TRIPLE_FAULT, vcpu);
 	nested_vmx_vmexit(vcpu, EXIT_REASON_TRIPLE_FAULT, 0, 0);
@@ -5368,6 +5400,7 @@ int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsigned long exit_qualification,
 
 static int nested_vmx_get_vmptr(struct kvm_vcpu *vcpu, gpa_t *vmpointer,
 				int *ret)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	gva_t gva;
 	struct x86_exception e;
@@ -5475,6 +5508,7 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu)
 
 /* Emulate the VMXON instruction. */
 static int handle_vmxon(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int ret;
 	gpa_t vmptr;
@@ -5566,6 +5600,7 @@ static int handle_vmxon(struct kvm_vcpu *vcpu)
 }
 
 static inline void nested_release_vmcs12(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 
@@ -5608,6 +5643,7 @@ static int handle_vmxoff(struct kvm_vcpu *vcpu)
 
 /* Emulate the VMCLEAR instruction */
 static int handle_vmclear(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	u32 zero = 0;
@@ -5650,18 +5686,20 @@ static int handle_vmclear(struct kvm_vcpu *vcpu)
 
 /* Emulate the VMLAUNCH instruction */
 static int handle_vmlaunch(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return nested_vmx_run(vcpu, true);
 }
 
 /* Emulate the VMRESUME instruction */
 static int handle_vmresume(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
-
 	return nested_vmx_run(vcpu, false);
 }
 
 static int handle_vmread(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = is_guest_mode(vcpu) ? get_shadow_vmcs12(vcpu)
 						    : get_vmcs12(vcpu);
@@ -5768,6 +5806,7 @@ static bool is_shadow_field_ro(unsigned long field)
 }
 
 static int handle_vmwrite(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = is_guest_mode(vcpu) ? get_shadow_vmcs12(vcpu)
 						    : get_vmcs12(vcpu);
@@ -5889,6 +5928,7 @@ static void set_current_vmptr(struct vcpu_vmx *vmx, gpa_t vmptr)
 
 /* Emulate the VMPTRLD instruction */
 static int handle_vmptrld(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	gpa_t vmptr;
@@ -5959,6 +5999,7 @@ static int handle_vmptrld(struct kvm_vcpu *vcpu)
 
 /* Emulate the VMPTRST instruction */
 static int handle_vmptrst(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long exit_qual = vmx_get_exit_qual(vcpu);
 	u32 instr_info = vmcs_read32(VMX_INSTRUCTION_INFO);
@@ -5987,6 +6028,7 @@ static int handle_vmptrst(struct kvm_vcpu *vcpu)
 
 /* Emulate the INVEPT instruction */
 static int handle_invept(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	u32 vmx_instruction_info, types;
@@ -6067,6 +6109,7 @@ static int handle_invept(struct kvm_vcpu *vcpu)
 }
 
 static int handle_invvpid(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	u32 vmx_instruction_info;
@@ -6159,6 +6202,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)
 
 static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu,
 				     struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 index = kvm_ecx_read(vcpu);
 	u64 new_eptp;
@@ -6191,6 +6235,7 @@ static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu,
 }
 
 static int handle_vmfunc(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct vmcs12 *vmcs12;
@@ -6280,6 +6325,7 @@ bool nested_vmx_check_io_bitmaps(struct kvm_vcpu *vcpu, unsigned int port,
 
 static bool nested_vmx_exit_handled_io(struct kvm_vcpu *vcpu,
 				       struct vmcs12 *vmcs12)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long exit_qualification;
 	unsigned short port;
@@ -6305,6 +6351,7 @@ static bool nested_vmx_exit_handled_io(struct kvm_vcpu *vcpu,
 static bool nested_vmx_exit_handled_msr(struct kvm_vcpu *vcpu,
 					struct vmcs12 *vmcs12,
 					union vmx_exit_reason exit_reason)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 msr_index;
 	gpa_t bitmap;
@@ -6434,6 +6481,7 @@ static bool nested_vmx_exit_handled_encls(struct kvm_vcpu *vcpu,
 
 static bool nested_vmx_exit_handled_vmcs_access(struct kvm_vcpu *vcpu,
 	struct vmcs12 *vmcs12, gpa_t bitmap)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 vmx_instruction_info;
 	unsigned long field;
@@ -6561,6 +6609,7 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu,
  */
 static bool nested_vmx_l1_wants_exit(struct kvm_vcpu *vcpu,
 				     union vmx_exit_reason exit_reason)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 	u32 intr_info;
@@ -6908,6 +6957,7 @@ int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu)
 static int vmx_set_nested_state(struct kvm_vcpu *vcpu,
 				struct kvm_nested_state __user *user_kvm_nested_state,
 				struct kvm_nested_state *kvm_state)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct vmcs12 *vmcs12;
diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h
index c6de848bd9ce..3b8b6579c104 100644
--- a/arch/x86/kvm/vmx/nested.h
+++ b/arch/x86/kvm/vmx/nested.h
@@ -17,23 +17,29 @@ enum nvmx_vmentry_status {
 	NVMX_VMENTRY_KVM_INTERNAL_ERROR,/* KVM internal error */
 };
 
-void vmx_leave_nested(struct kvm_vcpu *vcpu);
+void vmx_leave_nested(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps);
 void nested_vmx_hardware_unsetup(void);
 __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *));
 void nested_vmx_set_vmcs_shadowing_bitmap(void);
-int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu);
+int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu);
 enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
-						     bool from_vmentry);
-bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu);
+						     bool from_vmentry)
+	__must_hold_shared(&vcpu->kvm->srcu);
+bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 			 u32 exit_intr_info, unsigned long exit_qualification,
-			 u32 exit_insn_len);
+			 u32 exit_insn_len)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 static inline void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 				     u32 exit_intr_info,
 				     unsigned long exit_qualification)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 exit_insn_len;
 
@@ -53,7 +59,8 @@ int vmx_get_vmx_msr(struct nested_vmx_msrs *msrs, u32 msr_index, u64 *pdata);
 int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsigned long exit_qualification,
 			u32 vmx_instruction_info, bool wr, int len, gva_t *ret);
 bool nested_vmx_check_io_bitmaps(struct kvm_vcpu *vcpu, unsigned int port,
-				 int size);
+				 int size)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 static inline struct vmcs12 *get_vmcs12(struct kvm_vcpu *vcpu)
 {
diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c
index 771c75a58343..588cc8ae1f8b 100644
--- a/arch/x86/kvm/vmx/sgx.c
+++ b/arch/x86/kvm/vmx/sgx.c
@@ -91,6 +91,7 @@ static int sgx_gva_to_gpa(struct kvm_vcpu *vcpu, gva_t gva, bool write,
 }
 
 static int sgx_gpa_to_hva(struct kvm_vcpu *vcpu, gpa_t gpa, unsigned long *hva)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	*hva = kvm_vcpu_gfn_to_hva(vcpu, PFN_DOWN(gpa));
 	if (kvm_is_error_hva(*hva)) {
@@ -216,6 +217,7 @@ static int __handle_encls_ecreate(struct kvm_vcpu *vcpu,
 }
 
 static int handle_encls_ecreate(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	gva_t pageinfo_gva, secs_gva;
 	gva_t metadata_gva, contents_gva;
@@ -296,6 +298,7 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu)
 }
 
 static int handle_encls_einit(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long sig_hva, secs_hva, token_hva, rflags;
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
diff --git a/arch/x86/kvm/vmx/sgx.h b/arch/x86/kvm/vmx/sgx.h
index a400888b376d..08b1d5e90aa8 100644
--- a/arch/x86/kvm/vmx/sgx.h
+++ b/arch/x86/kvm/vmx/sgx.h
@@ -10,7 +10,8 @@
 #ifdef CONFIG_X86_SGX_KVM
 extern bool __read_mostly enable_sgx;
 
-int handle_encls(struct kvm_vcpu *vcpu);
+int handle_encls(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 void setup_default_sgx_lepubkeyhash(void);
 void vcpu_setup_sgx_lepubkeyhash(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index b272c20586a7..7cf53c5dc712 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -1155,6 +1155,7 @@ static int complete_hypercall_exit(struct kvm_vcpu *vcpu)
 }
 
 static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	kvm_rax_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r10);
 	kvm_rbx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r11);
@@ -1408,6 +1409,7 @@ static int tdx_complete_mmio_read(struct kvm_vcpu *vcpu)
 
 static inline int tdx_mmio_write(struct kvm_vcpu *vcpu, gpa_t gpa, int size,
 				 unsigned long val)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (!kvm_io_bus_write(vcpu, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) {
 		trace_kvm_fast_mmio(gpa);
@@ -1422,6 +1424,7 @@ static inline int tdx_mmio_write(struct kvm_vcpu *vcpu, gpa_t gpa, int size,
 }
 
 static inline int tdx_mmio_read(struct kvm_vcpu *vcpu, gpa_t gpa, int size)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long val;
 
@@ -1434,6 +1437,7 @@ static inline int tdx_mmio_read(struct kvm_vcpu *vcpu, gpa_t gpa, int size)
 }
 
 static int tdx_emulate_mmio(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_tdx *tdx = to_tdx(vcpu);
 	int size, write, r;
@@ -1912,6 +1916,7 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
 }
 
 static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long exit_qual;
 	gpa_t gpa = to_tdx(vcpu)->exit_gpa;
@@ -2723,6 +2728,7 @@ static int tdx_read_cpuid(struct kvm_vcpu *vcpu, u32 leaf, u32 sub_leaf,
 typedef void *tdx_vm_state_guard_t;
 
 static tdx_vm_state_guard_t tdx_acquire_vm_state_locks(struct kvm *kvm)
+	__context_unsafe(/* multi-lock acquisition */)
 {
 	int r;
 
@@ -2750,6 +2756,7 @@ static tdx_vm_state_guard_t tdx_acquire_vm_state_locks(struct kvm *kvm)
 }
 
 static void tdx_release_vm_state_locks(struct kvm *kvm)
+	__context_unsafe(/* multi-lock release */)
 {
 	mutex_unlock(&kvm->slots_lock);
 	kvm_unlock_all_vcpus(kvm);
@@ -3213,6 +3220,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 }
 
 static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd)
+	__must_hold(&vcpu->kvm->slots_lock)
 {
 	struct vcpu_tdx *tdx = to_tdx(vcpu);
 	struct kvm *kvm = vcpu->kvm;
@@ -3292,6 +3300,7 @@ int tdx_vcpu_unlocked_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
 	CLASS(tdx_vm_state_guard, guard)(kvm);
 	if (IS_ERR(guard))
 		return PTR_ERR(guard);
+	lockdep_assert_held(&kvm->slots_lock);
 
 	if (!is_hkid_assigned(kvm_tdx) || kvm_tdx->state == TD_STATE_RUNNABLE)
 		return -EINVAL;
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 612ab07d4100..ddf5c05955e1 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -5429,6 +5429,7 @@ static int vmx_handle_page_fault(struct kvm_vcpu *vcpu, u32 error_code)
 }
 
 static int handle_exception_nmi(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	struct kvm_run *kvm_run = vcpu->run;
@@ -5589,6 +5590,7 @@ static int handle_triple_fault(struct kvm_vcpu *vcpu)
 }
 
 static int handle_io(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long exit_qualification;
 	int size, in, string;
@@ -5621,6 +5623,7 @@ void vmx_patch_hypercall(struct kvm_vcpu *vcpu, unsigned char *hypercall)
 
 /* called to set cr0 as appropriate for a mov-to-cr0 exit. */
 static int handle_set_cr0(struct kvm_vcpu *vcpu, unsigned long val)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (is_guest_mode(vcpu)) {
 		struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
@@ -5647,6 +5650,7 @@ static int handle_set_cr0(struct kvm_vcpu *vcpu, unsigned long val)
 }
 
 static int handle_set_cr4(struct kvm_vcpu *vcpu, unsigned long val)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (is_guest_mode(vcpu)) {
 		struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
@@ -5676,6 +5680,7 @@ static int handle_desc(struct kvm_vcpu *vcpu)
 }
 
 static int handle_cr(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long exit_qualification, val;
 	int cr;
@@ -6003,6 +6008,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu)
 }
 
 static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	gpa_t gpa;
 
@@ -6140,6 +6146,7 @@ static int handle_monitor_trap(struct kvm_vcpu *vcpu)
 }
 
 static int handle_invpcid(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 vmx_instruction_info;
 	unsigned long type;
@@ -6427,6 +6434,7 @@ static void vmx_destroy_pml_buffer(struct vcpu_vmx *vmx)
 }
 
 static void vmx_flush_pml_buffer(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	u16 pml_idx, pml_tail_index;
@@ -6466,6 +6474,7 @@ static void vmx_flush_pml_buffer(struct kvm_vcpu *vcpu)
 }
 
 static void nested_vmx_mark_all_vmcs12_pages_dirty(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 
@@ -6706,6 +6715,7 @@ void dump_vmcs(struct kvm_vcpu *vcpu)
  * assistance.
  */
 static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 	union vmx_exit_reason exit_reason = vmx_get_exit_reason(vcpu);
@@ -8144,6 +8154,7 @@ static __init void vmx_set_cpu_caps(void)
 static bool vmx_is_io_intercepted(struct kvm_vcpu *vcpu,
 				  struct x86_instruction_info *info,
 				  unsigned long *exit_qualification)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 	unsigned short port;
diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h
index 054fd14bb2e1..f92ae9e60211 100644
--- a/arch/x86/kvm/vmx/x86_ops.h
+++ b/arch/x86/kvm/vmx/x86_ops.h
@@ -26,16 +26,20 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu);
 void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);
 void vmx_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
 void vmx_vcpu_put(struct kvm_vcpu *vcpu);
-int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath);
+int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu);
 int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu);
 void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu);
 bool vmx_unhandleable_emulation_required(struct kvm_vcpu *vcpu);
-int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
+int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
+	__must_hold_shared(&vcpu->kvm->srcu);
 #ifdef CONFIG_KVM_SMM
 int vmx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection);
-int vmx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram);
-int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram);
+int vmx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram)
+	__must_hold_shared(&vcpu->kvm->srcu);
+int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void vmx_enable_smi_window(struct kvm_vcpu *vcpu);
 #endif
 int vmx_check_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
@@ -43,7 +47,8 @@ int vmx_check_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
 int vmx_check_intercept(struct kvm_vcpu *vcpu,
 			struct x86_instruction_info *info,
 			enum x86_intercept_stage stage,
-			struct x86_exception *exception);
+			struct x86_exception *exception)
+	__must_hold_shared(&vcpu->kvm->srcu);
 bool vmx_apic_init_signal_blocked(struct kvm_vcpu *vcpu);
 void vmx_migrate_timers(struct kvm_vcpu *vcpu);
 void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu);
@@ -98,7 +103,8 @@ void vmx_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked);
 void vmx_enable_nmi_window(struct kvm_vcpu *vcpu);
 void vmx_enable_irq_window(struct kvm_vcpu *vcpu);
 void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr);
-void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu);
+void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 void vmx_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu);
 void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap);
 int vmx_set_tss_addr(struct kvm *kvm, unsigned int addr);
@@ -142,7 +148,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags);
 void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);
 void tdx_vcpu_put(struct kvm_vcpu *vcpu);
 int tdx_handle_exit(struct kvm_vcpu *vcpu,
-		enum exit_fastpath_completion fastpath);
+		enum exit_fastpath_completion fastpath)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
 			   int trig_mode, int vector);
@@ -151,7 +158,8 @@ void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,
 		u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code);
 bool tdx_has_emulated_msr(u32 index);
 int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr);
-int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr);
+int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
 int tdx_vcpu_unlocked_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 79468ddfe473..9893705d0dfa 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -321,7 +321,8 @@ static struct kmem_cache *kvm_alloc_emulator_cache(void)
 					  size - useroffset, NULL);
 }
 
-static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt);
+static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu);
 
 /*
  * Handle a fault on a hardware virtualization (VMX or SVM) instruction.
@@ -1076,6 +1077,7 @@ static inline bool gtod_is_based_on_tsc(int mode)
 #endif
 
 static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool new_generation)
+	__must_hold(&vcpu->kvm->arch.tsc_write_lock)
 {
 #ifdef CONFIG_X86_64
 	struct kvm_arch *ka = &vcpu->kvm->arch;
@@ -1604,12 +1606,14 @@ static void kvm_make_mclock_inprogress_request(struct kvm *kvm)
 }
 
 static void __kvm_start_pvclock_update(struct kvm *kvm)
+	__acquires(&kvm->arch.tsc_write_lock)
 {
 	raw_spin_lock_irq(&kvm->arch.tsc_write_lock);
 	write_seqcount_begin(&kvm->arch.pvclock_sc);
 }
 
 static void kvm_start_pvclock_update(struct kvm *kvm)
+	__acquires(&kvm->arch.tsc_write_lock)
 {
 	kvm_make_mclock_inprogress_request(kvm);
 
@@ -1618,6 +1622,7 @@ static void kvm_start_pvclock_update(struct kvm *kvm)
 }
 
 static void kvm_end_pvclock_update(struct kvm *kvm)
+	__releases(&kvm->arch.tsc_write_lock)
 {
 	struct kvm_arch *ka = &kvm->arch;
 	struct kvm_vcpu *vcpu;
@@ -1716,11 +1721,14 @@ static void kvm_setup_guest_pvclock(struct pvclock_vcpu_time_info *ref_hv_clock,
 				    struct kvm_vcpu *vcpu,
 				    struct gfn_to_pfn_cache *gpc,
 				    unsigned int offset)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct pvclock_vcpu_time_info *guest_hv_clock;
 	struct pvclock_vcpu_time_info hv_clock;
 	unsigned long flags;
 
+	lockdep_assert_held(&gpc->kvm->srcu); /* gpc->kvm == vcpu->kvm */
+
 	memcpy(&hv_clock, ref_hv_clock, sizeof(hv_clock));
 
 	read_lock_irqsave(&gpc->lock, flags);
@@ -2046,6 +2054,7 @@ void kvm_service_local_tlb_flush_requests(struct kvm_vcpu *vcpu)
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_service_local_tlb_flush_requests);
 
 static void record_steal_time(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
 	struct kvm_steal_time __user *st;
@@ -2613,6 +2622,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 }
 
 static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
 	struct kvm_steal_time __user *st;
@@ -4739,6 +4749,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg)
 
 static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len,
 			   void *__v)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	const void *v = __v;
 	int handled = 0;
@@ -4762,6 +4773,7 @@ static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len,
 }
 
 static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa_t addr, int len, void *v)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int handled = 0;
 	int n;
@@ -4819,6 +4831,7 @@ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,
 static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
 				      struct kvm_vcpu *vcpu, u64 access,
 				      struct x86_exception *exception)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 	void *data = val;
@@ -4851,6 +4864,7 @@ static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
 static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt,
 				gva_t addr, void *val, unsigned int bytes,
 				struct x86_exception *exception)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
@@ -4896,6 +4910,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_read_guest_virt);
 static int emulator_read_std(struct x86_emulate_ctxt *ctxt,
 			     gva_t addr, void *val, unsigned int bytes,
 			     struct x86_exception *exception, bool system)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	u64 access = 0;
@@ -4911,6 +4926,7 @@ static int emulator_read_std(struct x86_emulate_ctxt *ctxt,
 static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
 				      struct kvm_vcpu *vcpu, u64 access,
 				      struct x86_exception *exception)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 	void *data = val;
@@ -4941,6 +4957,7 @@ static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes
 static int emulator_write_std(struct x86_emulate_ctxt *ctxt, gva_t addr, void *val,
 			      unsigned int bytes, struct x86_exception *exception,
 			      bool system)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	u64 access = PFERR_WRITE_MASK;
@@ -5001,6 +5018,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_ud);
 
 static int vcpu_is_mmio_gpa(struct kvm_vcpu *vcpu, unsigned long gva,
 			    gpa_t gpa, bool write)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	/* For APIC access vmexit */
 	if ((gpa & PAGE_MASK) == APIC_DEFAULT_PHYS_BASE)
@@ -5017,6 +5035,7 @@ static int vcpu_is_mmio_gpa(struct kvm_vcpu *vcpu, unsigned long gva,
 static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva,
 				gpa_t *gpa, struct x86_exception *exception,
 				bool write)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
 	u64 access = ((kvm_x86_call(get_cpl)(vcpu) == 3) ? PFERR_USER_MASK : 0)
@@ -5054,12 +5073,14 @@ struct read_write_emulator_ops {
 
 static int emulator_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa,
 			       void *val, int bytes)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return !kvm_vcpu_read_guest(vcpu, gpa, val, bytes);
 }
 
 static int emulator_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa,
 				void *val, int bytes)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int ret;
 
@@ -5075,6 +5096,7 @@ static int emulator_read_write_onepage(unsigned long addr, void *val,
 				       struct x86_exception *exception,
 				       struct kvm_vcpu *vcpu,
 				       const struct read_write_emulator_ops *ops)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	gpa_t gpa;
 	int handled, ret;
@@ -5150,6 +5172,7 @@ static int emulator_read_write(struct x86_emulate_ctxt *ctxt,
 			void *val, unsigned int bytes,
 			struct x86_exception *exception,
 			const struct read_write_emulator_ops *ops)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	int rc;
@@ -5229,6 +5252,7 @@ static int emulator_read_emulated(struct x86_emulate_ctxt *ctxt,
 				  void *val,
 				  unsigned int bytes,
 				  struct x86_exception *exception)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	static const struct read_write_emulator_ops ops = {
 		.read_write_guest = emulator_read_guest,
@@ -5244,6 +5268,7 @@ static int emulator_write_emulated(struct x86_emulate_ctxt *ctxt,
 			    const void *val,
 			    unsigned int bytes,
 			    struct x86_exception *exception)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	static const struct read_write_emulator_ops ops = {
 		.read_write_guest = emulator_write_guest,
@@ -5263,6 +5288,7 @@ static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt,
 				     const void *new,
 				     unsigned int bytes,
 				     struct x86_exception *exception)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	u64 page_line_mask;
@@ -5343,6 +5369,7 @@ static int emulator_cmpxchg_emulated(struct x86_emulate_ctxt *ctxt,
 static int emulator_pio_in_out(struct kvm_vcpu *vcpu, int size,
 			       unsigned short port, void *data,
 			       unsigned int count, bool in)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned i;
 	int r;
@@ -5393,6 +5420,7 @@ static int emulator_pio_in_out(struct kvm_vcpu *vcpu, int size,
 
 static int emulator_pio_in(struct kvm_vcpu *vcpu, int size,
       			   unsigned short port, void *val, unsigned int count)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int r = emulator_pio_in_out(vcpu, size, port, val, count, true);
 	if (r)
@@ -5413,6 +5441,7 @@ static void complete_emulator_pio_in(struct kvm_vcpu *vcpu, void *val)
 static int emulator_pio_in_emulated(struct x86_emulate_ctxt *ctxt,
 				    int size, unsigned short port, void *val,
 				    unsigned int count)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	if (vcpu->arch.pio.count) {
@@ -5433,6 +5462,7 @@ static int emulator_pio_in_emulated(struct x86_emulate_ctxt *ctxt,
 static int emulator_pio_out(struct kvm_vcpu *vcpu, int size,
 			    unsigned short port, const void *val,
 			    unsigned int count)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	trace_kvm_pio(KVM_PIO_OUT, port, size, count, val);
 	return emulator_pio_in_out(vcpu, size, port, (void *)val, count, false);
@@ -5441,6 +5471,7 @@ static int emulator_pio_out(struct kvm_vcpu *vcpu, int size,
 static int emulator_pio_out_emulated(struct x86_emulate_ctxt *ctxt,
 				     int size, unsigned short port,
 				     const void *val, unsigned int count)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	return emulator_pio_out(emul_to_vcpu(ctxt), size, port, val, count);
 }
@@ -5533,6 +5564,7 @@ static unsigned long emulator_get_cr(struct x86_emulate_ctxt *ctxt, int cr)
 }
 
 static int emulator_set_cr(struct x86_emulate_ctxt *ctxt, int cr, ulong val)
+	__must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu)
 {
 	struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
 	int res = 0;
@@ -6578,6 +6610,7 @@ static int complete_fast_pio_out(struct kvm_vcpu *vcpu)
 
 static int kvm_fast_pio_out(struct kvm_vcpu *vcpu, int size,
 			    unsigned short port)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long val = kvm_rax_read_raw(vcpu);
 	int ret = emulator_pio_out(vcpu, size, port, &val, 1);
@@ -6625,6 +6658,7 @@ static int complete_fast_pio_in(struct kvm_vcpu *vcpu)
 
 static int kvm_fast_pio_in(struct kvm_vcpu *vcpu, int size,
 			   unsigned short port)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned long val;
 	int ret;
@@ -7192,6 +7226,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_vendor_exit);
 #ifdef CONFIG_X86_64
 static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 			        unsigned long clock_type)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	struct kvm_clock_pairing clock_pairing;
 	struct timespec64 ts;
@@ -7571,6 +7606,7 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)
  */
 static int kvm_check_and_inject_events(struct kvm_vcpu *vcpu,
 				       bool *req_immediate_exit)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	bool can_inject;
 	int r;
@@ -7868,6 +7904,7 @@ void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_vcpu_update_apicv);
 
 static void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (!lapic_in_kernel(vcpu))
 		return;
@@ -8049,6 +8086,7 @@ static void kvm_vcpu_reload_apic_access_page(struct kvm_vcpu *vcpu)
  * userspace.
  */
 static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int r;
 	bool req_int_win =
@@ -8561,6 +8599,7 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu)
 
 /* Called within kvm->srcu read side.  */
 static inline int vcpu_block(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	bool hv_timer;
 
@@ -8629,6 +8668,7 @@ static inline int vcpu_block(struct kvm_vcpu *vcpu)
 
 /* Called within kvm->srcu read side.  */
 static int vcpu_run(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int r;
 
@@ -9940,10 +9980,13 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa,
 {
 	int i, r;
 	unsigned long hva, old_npages;
-	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memslots *slots;
 	struct kvm_memory_slot *slot;
 
 	lockdep_assert_held(&kvm->slots_lock);
+	__assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */
+
+	slots = kvm_memslots(kvm);
 
 	if (WARN_ON(id >= KVM_MEM_SLOTS_NUM))
 		return ERR_PTR_USR(-EINVAL);
@@ -10452,6 +10495,7 @@ static void kvm_del_async_pf_gfn(struct kvm_vcpu *vcpu, gfn_t gfn)
 }
 
 static inline int apf_put_user_notpresent(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u32 reason = KVM_PV_REASON_PAGE_NOT_PRESENT;
 
@@ -10460,6 +10504,7 @@ static inline int apf_put_user_notpresent(struct kvm_vcpu *vcpu)
 }
 
 static inline int apf_put_user_ready(struct kvm_vcpu *vcpu, u32 token)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned int offset = offsetof(struct kvm_vcpu_pv_apf_data, token);
 
@@ -10468,6 +10513,7 @@ static inline int apf_put_user_ready(struct kvm_vcpu *vcpu, u32 token)
 }
 
 static inline bool apf_pageready_slot_free(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned int offset = offsetof(struct kvm_vcpu_pv_apf_data, token);
 	u32 val;
@@ -10892,9 +10938,11 @@ static void advance_sev_es_emulated_pio(struct kvm_vcpu *vcpu, unsigned count, i
 }
 
 static int kvm_sev_es_outs(struct kvm_vcpu *vcpu, unsigned int size,
-			   unsigned int port);
+			   unsigned int port)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 static int complete_sev_es_emulated_outs(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	int size = vcpu->arch.pio.size;
 	int port = vcpu->arch.pio.port;
@@ -10928,9 +10976,11 @@ static int kvm_sev_es_outs(struct kvm_vcpu *vcpu, unsigned int size,
 }
 
 static int kvm_sev_es_ins(struct kvm_vcpu *vcpu, unsigned int size,
-			  unsigned int port);
+			  unsigned int port)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 static int complete_sev_es_emulated_ins(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	unsigned count = vcpu->arch.pio.count;
 	int size = vcpu->arch.pio.size;
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 0f5919b092e4..b97b23e2b262 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -249,6 +249,7 @@ static inline bool is_noncanonical_invlpg_address(u64 la, struct kvm_vcpu *vcpu)
 
 static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu,
 					gva_t gva, gfn_t gfn, unsigned access)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	u64 gen = kvm_memslots(vcpu->kvm)->generation;
 
@@ -266,6 +267,7 @@ static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu,
 }
 
 static inline bool vcpu_match_mmio_gen(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	return vcpu->arch.mmio_gen == kvm_memslots(vcpu->kvm)->generation;
 }
@@ -285,6 +287,7 @@ static inline void vcpu_clear_mmio_info(struct kvm_vcpu *vcpu, gva_t gva)
 }
 
 static inline bool vcpu_match_mmio_gva(struct kvm_vcpu *vcpu, unsigned long gva)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (vcpu_match_mmio_gen(vcpu) && vcpu->arch.mmio_gva &&
 	      vcpu->arch.mmio_gva == (gva & PAGE_MASK))
@@ -294,6 +297,7 @@ static inline bool vcpu_match_mmio_gva(struct kvm_vcpu *vcpu, unsigned long gva)
 }
 
 static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu *vcpu, gpa_t gpa)
+	__must_hold_shared(&vcpu->kvm->srcu)
 {
 	if (vcpu_match_mmio_gen(vcpu) && vcpu->arch.mmio_gfn &&
 	      vcpu->arch.mmio_gfn == gpa >> PAGE_SHIFT)
@@ -328,7 +332,8 @@ void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc_eip);
 u64 get_kvmclock_ns(struct kvm *kvm);
 uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm);
 bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp);
-int kvm_guest_time_update(struct kvm_vcpu *v);
+int kvm_guest_time_update(struct kvm_vcpu *v)
+	__must_hold_shared(&v->kvm->srcu);
 
 void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value);
 u64 kvm_scale_tsc(u64 tsc, u64 ratio);
@@ -355,13 +360,16 @@ static inline void adjust_tsc_offset_host(struct kvm_vcpu *vcpu, s64 adjustment)
 
 int kvm_read_guest_virt(struct kvm_vcpu *vcpu,
 	gva_t addr, void *val, unsigned int bytes,
-	struct x86_exception *exception);
+	struct x86_exception *exception)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu,
 	gva_t addr, void *val, unsigned int bytes,
-	struct x86_exception *exception);
+	struct x86_exception *exception)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
-int handle_ud(struct kvm_vcpu *vcpu);
+int handle_ud(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu,
 				   struct kvm_queued_exception *ex);
@@ -473,7 +481,8 @@ int kvm_emulate_mwait(struct kvm_vcpu *vcpu);
 int kvm_handle_invalid_op(struct kvm_vcpu *vcpu);
 int kvm_emulate_monitor(struct kvm_vcpu *vcpu);
 
-int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int in);
+int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int in)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_emulate_cpuid(struct kvm_vcpu *vcpu);
 int kvm_emulate_halt(struct kvm_vcpu *vcpu);
 int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu);
@@ -526,7 +535,8 @@ void kvm_inject_nmi(struct kvm_vcpu *vcpu);
 int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu);
 
 void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa,
-				     u32 size);
+				     u32 size)
+	__must_hold(&kvm->slots_lock);
 int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);
 
 bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu);
@@ -840,13 +850,16 @@ static inline void kvm_machine_check(void)
 int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r,
 			      struct x86_exception *e);
 void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid);
-int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva);
+int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa,
-		    unsigned int bytes, void *data);
+		    unsigned int bytes, void *data)
+	__must_hold_shared(&vcpu->kvm->srcu);
 int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size,
 			 unsigned int port, void *data,  unsigned int count,
-			 int in);
+			 int in)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 static inline void __kvm_prepare_emulated_mmio_exit(struct kvm_vcpu *vcpu,
 						    gpa_t gpa, unsigned int len,
@@ -886,7 +899,8 @@ static inline bool user_exit_on_hypercall(struct kvm *kvm, unsigned long hc_nr)
 }
 
 int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl,
-			      int (*complete_hypercall)(struct kvm_vcpu *));
+			      int (*complete_hypercall)(struct kvm_vcpu *))
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 #define __kvm_emulate_hypercall(_vcpu, cpl, complete_hypercall)			\
 ({										\
@@ -898,6 +912,7 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl,
 	__ret;									\
 })
 
-int kvm_emulate_hypercall(struct kvm_vcpu *vcpu);
+int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
+	__must_hold_shared(&vcpu->kvm->srcu);
 
 #endif
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 08/10] KVM: Add guarded_by to members in struct kvm

[Marco Elver]

Add __guarded_by annotations to a subset of members in struct kvm where
the protecting locks reside in the same struct scope. Members protected
by locks across nested or parent struct boundaries are omitted due to C
lexical scoping limitations.

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 include/linux/kvm_host.h | 12 ++++++------
 virt/kvm/eventfd.c       |  4 ++--
 virt/kvm/kvm_main.c      |  3 +++
 3 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 12f241304228..36edf0a59107 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -771,7 +771,7 @@ struct kvm {
 	/* The two memslot sets - active and inactive (per address space) */
 	struct kvm_memslots __memslots[KVM_MAX_NR_ADDRESS_SPACES][2];
 	/* The current active memslot set for each address space */
-	struct kvm_memslots __rcu *memslots[KVM_MAX_NR_ADDRESS_SPACES];
+	struct kvm_memslots __rcu *memslots[KVM_MAX_NR_ADDRESS_SPACES] __guarded_by(&srcu, &slots_lock, &slots_arch_lock);
 	struct xarray vcpu_array;
 	DECLARE_BITMAP(vcpu_ids, KVM_MAX_VCPU_IDS);
 	/*
@@ -787,7 +787,7 @@ struct kvm {
 
 	/* For management / invalidation of gfn_to_pfn_caches */
 	spinlock_t gpc_lock;
-	struct list_head gpc_list;
+	struct list_head gpc_list __guarded_by(&gpc_lock);
 
 	/*
 	 * created_vcpus is protected by kvm->lock, and is incremented
@@ -801,11 +801,11 @@ struct kvm {
 	int last_boosted_vcpu;
 	struct list_head vm_list;
 	struct mutex lock;
-	struct kvm_io_bus __rcu *buses[KVM_NR_BUSES];
+	struct kvm_io_bus __rcu *buses[KVM_NR_BUSES] __guarded_by(&srcu, &slots_lock);
 #ifdef CONFIG_HAVE_KVM_IRQCHIP
 	struct {
 		spinlock_t        lock;
-		struct list_head  items;
+		struct list_head  items __guarded_by(&lock);
 		/* resampler_list update side is protected by resampler_lock. */
 		struct list_head  resampler_list;
 		struct mutex      resampler_lock;
@@ -826,9 +826,9 @@ struct kvm {
 	/*
 	 * Update side is protected by irq_lock.
 	 */
-	struct kvm_irq_routing_table __rcu *irq_routing;
+	struct kvm_irq_routing_table __rcu *irq_routing __guarded_by(&irq_srcu, &irq_lock);
 
-	struct hlist_head irq_ack_notifier_list;
+	struct hlist_head irq_ack_notifier_list __guarded_by(&irq_srcu, &irq_lock);
 #endif
 
 	struct mmu_notifier mmu_notifier;
diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
index 6701390336f6..6051fec5ec90 100644
--- a/virt/kvm/eventfd.c
+++ b/virt/kvm/eventfd.c
@@ -1046,8 +1046,8 @@ void
 kvm_eventfd_init(struct kvm *kvm)
 {
 #ifdef CONFIG_HAVE_KVM_IRQCHIP
-	spin_lock_init(&kvm->irqfds.lock);
-	INIT_LIST_HEAD(&kvm->irqfds.items);
+	scoped_guard(spinlock_init, &kvm->irqfds.lock)
+		INIT_LIST_HEAD(&kvm->irqfds.items);
 	INIT_LIST_HEAD(&kvm->irqfds.resampler_list);
 	mutex_init(&kvm->irqfds.resampler_lock);
 #endif
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 2f22d5439d39..871acc3aebbd 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1111,6 +1111,7 @@ void __weak kvm_arch_create_vm_debugfs(struct kvm *kvm)
 /* Called only on cleanup and destruction paths when there are no users. */
 static inline struct kvm_io_bus *kvm_get_bus_for_destruction(struct kvm *kvm,
 							     enum kvm_bus idx)
+	__context_unsafe(/* destruction */)
 {
 	return rcu_dereference_protected(kvm->buses[idx],
 					 !refcount_read(&kvm->users_count));
@@ -1120,6 +1121,7 @@ static int kvm_enable_virtualization(void);
 static void kvm_disable_virtualization(void);
 
 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
+	__context_unsafe(/* constructor */)
 {
 	struct kvm *kvm = kvm_arch_alloc_vm();
 	struct kvm_memslots *slots;
@@ -1276,6 +1278,7 @@ static void kvm_destroy_devices(struct kvm *kvm)
 }
 
 static void kvm_destroy_vm(struct kvm *kvm)
+	__context_unsafe(/* destructor */)
 {
 	int i;
 	struct mm_struct *mm = kvm->mm;
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 09/10] KVM: x86: Add guarded_by annotations for kvm_arch, kvm_hv, and ioapic

[Marco Elver]

Add __guarded_by annotations to a subset of fields across x86 state
(struct kvm_hv, struct kvm_arch, and struct kvm_ioapic) where the
protecting locks reside in the same struct scope. Mark deliberate
lockless updates with data_race().

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 arch/x86/include/asm/kvm_host.h | 32 ++++++++++++++++----------------
 arch/x86/kvm/ioapic.c           |  4 ++--
 arch/x86/kvm/ioapic.h           | 16 ++++++++--------
 arch/x86/kvm/x86.c              |  8 ++++----
 4 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 683bb8bf43a9..a0d2d6c08e47 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1073,20 +1073,20 @@ struct kvm_hv {
 	struct mutex hv_lock;
 	u64 hv_guest_os_id;
 	u64 hv_hypercall;
-	u64 hv_tsc_page;
+	u64 hv_tsc_page __guarded_by(&hv_lock);
 	enum hv_tsc_page_status hv_tsc_page_status;
 
 	/* Hyper-v based guest crash (NT kernel bugcheck) parameters */
-	u64 hv_crash_param[HV_X64_MSR_CRASH_PARAMS];
-	u64 hv_crash_ctl;
+	u64 hv_crash_param[HV_X64_MSR_CRASH_PARAMS] __guarded_by(&hv_lock);
+	u64 hv_crash_ctl __guarded_by(&hv_lock);
 
 	struct ms_hyperv_tsc_page tsc_ref;
 
 	struct idr conn_to_evt;
 
-	u64 hv_reenlightenment_control;
-	u64 hv_tsc_emulation_control;
-	u64 hv_tsc_emulation_status;
+	u64 hv_reenlightenment_control __guarded_by(&hv_lock);
+	u64 hv_tsc_emulation_control __guarded_by(&hv_lock);
+	u64 hv_tsc_emulation_status __guarded_by(&hv_lock);
 	u64 hv_invtsc_control;
 
 	/* How many vCPUs have VP index != vCPU index */
@@ -1232,15 +1232,15 @@ struct kvm_arch {
 	 * preemption-disabled region, so it must be a raw spinlock.
 	 */
 	raw_spinlock_t tsc_write_lock;
-	u64 last_tsc_nsec;
-	u64 last_tsc_write;
-	u32 last_tsc_khz;
-	u64 last_tsc_offset;
-	u64 cur_tsc_nsec;
-	u64 cur_tsc_write;
-	u64 cur_tsc_offset;
-	u64 cur_tsc_generation;
-	int nr_vcpus_matched_tsc;
+	u64 last_tsc_nsec __guarded_by(&tsc_write_lock);
+	u64 last_tsc_write __guarded_by(&tsc_write_lock);
+	u32 last_tsc_khz __guarded_by(&tsc_write_lock);
+	u64 last_tsc_offset __guarded_by(&tsc_write_lock);
+	u64 cur_tsc_nsec __guarded_by(&tsc_write_lock);
+	u64 cur_tsc_write __guarded_by(&tsc_write_lock);
+	u64 cur_tsc_offset __guarded_by(&tsc_write_lock);
+	u64 cur_tsc_generation __guarded_by(&tsc_write_lock);
+	int nr_vcpus_matched_tsc __guarded_by(&tsc_write_lock);
 
 	u32 default_tsc_khz;
 	bool user_set_tsc;
@@ -1370,7 +1370,7 @@ struct kvm_arch {
 #endif
 
 #if IS_ENABLED(CONFIG_HYPERV)
-	hpa_t	hv_root_tdp;
+	hpa_t	hv_root_tdp __guarded_by(&hv_root_tdp_lock);
 	spinlock_t hv_root_tdp_lock;
 	struct hv_partition_assist_pg *hv_pa_pg;
 #endif
diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c
index d6865e557abe..7affe2584036 100644
--- a/arch/x86/kvm/ioapic.c
+++ b/arch/x86/kvm/ioapic.c
@@ -739,11 +739,11 @@ int kvm_ioapic_init(struct kvm *kvm)
 	ioapic = kzalloc_obj(struct kvm_ioapic, GFP_KERNEL_ACCOUNT);
 	if (!ioapic)
 		return -ENOMEM;
-	spin_lock_init(&ioapic->lock);
 	INIT_DELAYED_WORK(&ioapic->eoi_inject, kvm_ioapic_eoi_inject_work);
 	INIT_HLIST_HEAD(&ioapic->mask_notifier_list);
 	kvm->arch.vioapic = ioapic;
-	kvm_ioapic_reset(ioapic);
+	scoped_guard(spinlock_init, &ioapic->lock)
+		kvm_ioapic_reset(ioapic);
 	kvm_iodevice_init(&ioapic->dev, &ioapic_mmio_ops);
 	ioapic->kvm = kvm;
 	mutex_lock(&kvm->slots_lock);
diff --git a/arch/x86/kvm/ioapic.h b/arch/x86/kvm/ioapic.h
index 81b576513116..1f87396c0a79 100644
--- a/arch/x86/kvm/ioapic.h
+++ b/arch/x86/kvm/ioapic.h
@@ -70,19 +70,19 @@ union kvm_ioapic_redirect_entry {
 
 struct kvm_ioapic {
 	u64 base_address;
-	u32 ioregsel;
-	u32 id;
-	u32 irr;
+	u32 ioregsel __guarded_by(&lock);
+	u32 id __guarded_by(&lock);
+	u32 irr __guarded_by(&lock);
 	u32 pad;
-	union kvm_ioapic_redirect_entry redirtbl[IOAPIC_NUM_PINS];
-	unsigned long irq_states[IOAPIC_NUM_PINS];
+	union kvm_ioapic_redirect_entry redirtbl[IOAPIC_NUM_PINS] __guarded_by(&lock);
+	unsigned long irq_states[IOAPIC_NUM_PINS] __guarded_by(&lock);
 	struct kvm_io_device dev;
 	struct kvm *kvm;
 	spinlock_t lock;
-	struct rtc_status rtc_status;
+	struct rtc_status rtc_status __guarded_by(&lock);
 	struct delayed_work eoi_inject;
-	u32 irq_eoi[IOAPIC_NUM_PINS];
-	u32 irr_delivered;
+	u32 irq_eoi[IOAPIC_NUM_PINS] __guarded_by(&lock);
+	u32 irr_delivered __guarded_by(&lock);
 
 	/* reads protected by irq_srcu, writes by irq_lock */
 	struct hlist_head mask_notifier_list;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 9893705d0dfa..5d4b2c7aa9b8 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -9813,8 +9813,8 @@ int kvm_arch_enable_virtualization_cpu(void)
 			 * you may have some problem.  Solving this issue is
 			 * left as an exercise to the reader.
 			 */
-			kvm->arch.last_tsc_nsec = 0;
-			kvm->arch.last_tsc_write = 0;
+			data_race(kvm->arch.last_tsc_nsec = 0);
+			data_race(kvm->arch.last_tsc_write = 0);
 		}
 
 	}
@@ -9927,8 +9927,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;
 
 #if IS_ENABLED(CONFIG_HYPERV)
-	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
-	kvm->arch.hv_root_tdp = INVALID_PAGE;
+	scoped_guard(spinlock_init, &kvm->arch.hv_root_tdp_lock)
+		kvm->arch.hv_root_tdp = INVALID_PAGE;
 #endif
 
 	kvm_apicv_init(kvm);
-- 
2.55.0.1003.g10538fe699-goog

[PATCH RFC 10/10] KVM: x86: Enable CONTEXT_ANALYSIS with opt-outs

[Marco Elver]

Enable Clang context analysis (-Wthread-safety) for x86 KVM.

Because virt/kvm/ objects are built directly into the x86 KVM module,
this also enables checking for common KVM sources.

Temporarily opt out sources that require more invasive annotations:
mmu/mmu.c, mmu/tdp_mmu.c, i8259.c, and xen.c.

No functional change intended.

Signed-off-by: Marco Elver <elver@google.com>
---
 arch/x86/kvm/Makefile                    | 8 ++++++++
 scripts/context-analysis-suppression.txt | 1 +
 2 files changed, 9 insertions(+)

diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 0474604ab8a1..a93b54b00ae2 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -1,5 +1,13 @@
 # SPDX-License-Identifier: GPL-2.0
 
+CONTEXT_ANALYSIS := y
+
+# Opt out x86 objects pending context annotations:
+CONTEXT_ANALYSIS_mmu/mmu.o             := n
+CONTEXT_ANALYSIS_mmu/tdp_mmu.o         := n
+CONTEXT_ANALYSIS_i8259.o               := n
+CONTEXT_ANALYSIS_xen.o                 := n
+
 ccflags-y += -I $(srctree)/arch/x86/kvm
 ccflags-$(CONFIG_KVM_WERROR) += -Werror
 
diff --git a/scripts/context-analysis-suppression.txt b/scripts/context-analysis-suppression.txt
index 1c51b6153f08..666c4ca54708 100644
--- a/scripts/context-analysis-suppression.txt
+++ b/scripts/context-analysis-suppression.txt
@@ -17,6 +17,7 @@ src:*include/net/*
 src:*include/linux/bit_spinlock.h=emit
 src:*include/linux/cleanup.h=emit
 src:*include/linux/kref.h=emit
+src:*include/linux/kvm_host.h=emit
 src:*include/linux/list*.h=emit
 src:*include/linux/local_lock*.h=emit
 src:*include/linux/lockdep.h=emit
-- 
2.55.0.1003.g10538fe699-goog

Re: [PATCH RFC 02/10] KVM: Allow reading memslots while holding slots_arch_lock

[Sean Christopherson]

On Thu, Sep 10, 2026, Marco Elver wrote:
> kvm_swap_active_memslots() updates kvm->memslots[as_id] while holding both
> kvm->slots_lock and kvm->slots_arch_lock. Holding either lock guarantees
> that memslots cannot be concurrently modified.

Sure, but that's irrelevant.  The goal of the srcu_dereference_check() is to
ensure that readers see a stable view of the VM's overall memory, not simply that
kvm->memslots can't be written.

> Allow reading memslots in __kvm_memslots() when kvm->slots_arch_lock is
> held.

Why?

> 
> Signed-off-by: Marco Elver <elver@google.com>
> ---
>  include/linux/kvm_host.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index 03bfc92864b6..5ed8260ef01f 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1089,6 +1089,7 @@ static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)
>  	as_id = array_index_nospec(as_id, KVM_MAX_NR_ADDRESS_SPACES);
>  	return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu,
>  			lockdep_is_held(&kvm->slots_lock) ||
> +			lockdep_is_held(&kvm->slots_arch_lock) ||
>  			!refcount_read(&kvm->users_count));
>  }
>  
> -- 
> 2.55.0.1003.g10538fe699-goog
> 

Re: [PATCH RFC 04/10] KVM: Refactor kvm_handle_hva_range() to avoid conditional mmu_lock

[Sean Christopherson]

On Thu, Sep 10, 2026, Marco Elver wrote:
> Refactor kvm_handle_hva_range() to check for overlapping memslots
> upfront via interval_tree_iter_first() instead of tracking found
> memslots inside the range iteration with a 'found_memslot' flag and
> conditionally acquiring and releasing mmu_lock.
> 
> This simplifies the control flow by cleanly decoupling the search for
> overlapping memslots from the subsequent walk. It also separates the
> lockless path from the serialized path into distinct branches,
> eliminating the conditional locking, which subsequently enables Clang
> context analysis to validate locking in this function.

At the cost of duplicating code and adding overhead when there is a relevant
memslot.  I'm not wedded to the exact implementation, but IMO this is not at all
an improvement.  Without a clear explanation of what value is added by enabling
Clang to "validate locking", I don't see any justificatoin for the change.

> No functional change intended.
> 
> Signed-off-by: Marco Elver <elver@google.com>
> ---
>  virt/kvm/kvm_main.c | 94 ++++++++++++++++++++++++++++-----------------
>  1 file changed, 58 insertions(+), 36 deletions(-)
> 
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 65eb26a0520d..f7bfa2d32507 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -558,31 +558,14 @@ static void kvm_null_fn(void)
>  	     node;							     \
>  	     node = interval_tree_iter_next(node, start, last))	     \
>  
> -static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
> -							 const struct kvm_mmu_notifier_range *range)
> +static __always_inline bool __kvm_handle_hva_range_walk(struct kvm *kvm,
> +							const struct kvm_mmu_notifier_range *range)
>  {
> -	struct kvm_mmu_notifier_return r = {
> -		.ret = false,
> -		.found_memslot = false,
> -	};
>  	struct kvm_gfn_range gfn_range;
>  	struct kvm_memory_slot *slot;
>  	struct kvm_memslots *slots;
> -	int i, idx;
> -
> -	if (WARN_ON_ONCE(range->end <= range->start))
> -		return r;
> -
> -	/* A null handler is allowed if and only if on_lock() is provided. */
> -	if (WARN_ON_ONCE(IS_KVM_NULL_FN(range->on_lock) &&
> -			 IS_KVM_NULL_FN(range->handler)))
> -		return r;
> -
> -	/* on_lock will never be called for lockless walks */
> -	if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock)))
> -		return r;
> -
> -	idx = srcu_read_lock(&kvm->srcu);
> +	bool ret = false;
> +	int i;
>  
>  	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
>  		struct interval_tree_node *node;
> @@ -620,28 +603,67 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
>  			gfn_range.slot = slot;
>  			gfn_range.lockless = range->lockless;
>  
> -			if (!r.found_memslot) {
> -				r.found_memslot = true;
> -				if (!range->lockless) {
> -					KVM_MMU_LOCK(kvm);
> -					if (!IS_KVM_NULL_FN(range->on_lock))
> -						range->on_lock(kvm);
> +			ret |= range->handler(kvm, &gfn_range);
> +		}
> +	}
> +
> +	return ret;
> +}
>  
> -					if (IS_KVM_NULL_FN(range->handler))
> -						goto mmu_unlock;
> -				}
> -			}
> -			r.ret |= range->handler(kvm, &gfn_range);
> +static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
> +							 const struct kvm_mmu_notifier_range *range)
> +{
> +	struct kvm_mmu_notifier_return r = {
> +		.ret = false,
> +		.found_memslot = false,
> +	};
> +	struct kvm_memslots *slots;
> +	int i, idx;
> +
> +	if (WARN_ON_ONCE(range->end <= range->start))
> +		return r;
> +
> +	/* A null handler is allowed if and only if on_lock() is provided. */
> +	if (WARN_ON_ONCE(IS_KVM_NULL_FN(range->on_lock) &&
> +			 IS_KVM_NULL_FN(range->handler)))
> +		return r;
> +
> +	/* on_lock will never be called for lockless walks */
> +	if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock)))
> +		return r;
> +
> +	idx = srcu_read_lock(&kvm->srcu);
> +
> +	for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) {
> +		slots = __kvm_memslots(kvm, i);
> +		if (interval_tree_iter_first(&slots->hva_tree, range->start, range->end - 1)) {
> +			r.found_memslot = true;
> +			break;
>  		}
>  	}
>  
> -	if (range->flush_on_ret && r.ret)
> -		kvm_flush_remote_tlbs(kvm);
> +	if (!r.found_memslot)
> +		goto out;
> +
> +	if (range->lockless) {
> +		r.ret = __kvm_handle_hva_range_walk(kvm, range);
> +		if (range->flush_on_ret && r.ret)
> +			kvm_flush_remote_tlbs(kvm);
> +	} else {
> +		KVM_MMU_LOCK(kvm);
> +		if (!IS_KVM_NULL_FN(range->on_lock))
> +			range->on_lock(kvm);
> +
> +		if (!IS_KVM_NULL_FN(range->handler))
> +			r.ret = __kvm_handle_hva_range_walk(kvm, range);
> +
> +		if (range->flush_on_ret && r.ret)
> +			kvm_flush_remote_tlbs(kvm);
>  
> -mmu_unlock:
> -	if (r.found_memslot && !range->lockless)
>  		KVM_MMU_UNLOCK(kvm);
> +	}
>  
> +out:
>  	srcu_read_unlock(&kvm->srcu, idx);
>  
>  	return r;
> -- 
> 2.55.0.1003.g10538fe699-goog
> 

Re: [PATCH RFC 01/10] KVM: x86/pmu: Acquire SRCU in pmc_is_event_allowed() to protect filter lookup

[Sean Christopherson]

On Thu, Sep 10, 2026, Marco Elver wrote:
> Dereferencing kvm->arch.pmu_event_filter via srcu_dereference() requires
> holding kvm->srcu to guard against concurrent filter replacement and
> freeing by kvm_vm_ioctl_set_pmu_event_filter().
> 
> Counter reprogramming can reach pmc_is_event_allowed() without holding
> kvm->srcu. Specifically, on AMD SVM, toggling EFER.SVME via KVM_SET_SREGS
> or KVM_SET_SREGS2 triggers synchronous counter reprogramming outside of
> any SRCU read-side critical section:
> 
>   kvm_vcpu_ioctl(KVM_SET_SREGS{,2})
>     kvm_vcpu_ioctl_x86_set_sregs{,2}()
>       __set_sregs_common()
>         kvm_x86_call(set_efer)()
>           svm_set_efer()
>             svm_pmu_handle_nested_transition()
>               __svm_pmu_handle_nested_transition(..., defer=false)
>                 __kvm_pmu_reprogram_counters()
>                   kvm_pmu_handle_event()
>                     reprogram_counter()
>                       pmc_is_event_allowed()
>                         srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu)
> 
> If userspace concurrently updates the filter (KVM_SET_PMU_EVENT_FILTER),
> a concurrent free and subsequent use-after-free is possible.
> 
> Protect filter lookups directly in pmc_is_event_allowed():
> 1. check rcu_access_pointer() first for the common fast path;
> 2. acquire guard(srcu)(&kvm->srcu) only when a filter is present;
> 3. drop redundant outer srcu_read_lock() in kvm_pmu_trigger_event().
> 
> Found with Clang context analysis.
> 
> Fixes: a02a25a65246 ("KVM: x86/pmu: Reprogram Host/Guest-Only counters on nested transitions")
> Signed-off-by: Marco Elver <elver@google.com>
> ---
>  arch/x86/kvm/pmu.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
> index a7d60c8785cd..3ad1e696edca 100644
> --- a/arch/x86/kvm/pmu.c
> +++ b/arch/x86/kvm/pmu.c
> @@ -536,6 +536,11 @@ static bool pmc_is_event_allowed(struct kvm_pmc *pmc)
>  	struct kvm_x86_pmu_event_filter *filter;
>  	struct kvm *kvm = pmc->vcpu->kvm;
>  
> +	if (!rcu_access_pointer(kvm->arch.pmu_event_filter))
> +		return true;
> +
> +	guard(srcu)(&kvm->srcu);
> +
>  	filter = srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu);
>  	if (!filter)
>  		return true;
> @@ -1132,7 +1137,7 @@ static void kvm_pmu_trigger_event(struct kvm_vcpu *vcpu,
>  	DECLARE_BITMAP(bitmap, X86_PMC_IDX_MAX);
>  	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
>  	struct kvm_pmc *pmc;
> -	int i, idx;
> +	int i;
>  
>  	BUILD_BUG_ON(sizeof(pmu->global_ctrl) * BITS_PER_BYTE != X86_PMC_IDX_MAX);
>  
> @@ -1145,14 +1150,12 @@ static void kvm_pmu_trigger_event(struct kvm_vcpu *vcpu,
>  			     (unsigned long *)&pmu->global_ctrl, X86_PMC_IDX_MAX))
>  		return;
>  
> -	idx = srcu_read_lock(&vcpu->kvm->srcu);
>  	kvm_for_each_pmc(pmu, pmc, i, bitmap) {
>  		if (!pmc_is_event_allowed(pmc) || !cpl_is_matched(pmc))
>  			continue;
>  
>  		kvm_pmu_incr_counter(pmc);
>  	}
> -	srcu_read_unlock(&vcpu->kvm->srcu, idx);
>  }

I would very strongly prefer to fix this in __set_sregs_common():

diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 8f66438989e4..2ce16e96d796 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -571,9 +571,10 @@ static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
 static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
 			      int *mmu_reset_needed, bool update_pdptrs)
 {
-	int idx;
 	struct desc_ptr dt;
 
+	guard(srcu)(&vcpu->kvm->srcu);
+
 	if (!kvm_is_valid_sregs(vcpu, sregs))
 		return -EINVAL;
 
@@ -605,13 +606,9 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
 	*mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4;
 	kvm_x86_call(set_cr4)(vcpu, sregs->cr4);
 
-	if (update_pdptrs) {
-		idx = srcu_read_lock(&vcpu->kvm->srcu);
-		if (is_pae_paging(vcpu)) {
-			load_pdptrs(vcpu, kvm_read_cr3(vcpu));
-			*mmu_reset_needed = 1;
-		}
-		srcu_read_unlock(&vcpu->kvm->srcu, idx);
+	if (update_pdptrs && is_pae_paging(vcpu)) {
+		load_pdptrs(vcpu, kvm_read_cr3(vcpu));
+		*mmu_reset_needed = 1;
 	}
 
 	kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS);

>  
>  void kvm_pmu_instruction_retired(struct kvm_vcpu *vcpu)
> -- 
> 2.55.0.1003.g10538fe699-goog
> 

Re: [PATCH RFC 00/10] KVM: Enable Clang Context Analysis

[Sean Christopherson]

On Thu, Sep 10, 2026, Marco Elver wrote:
> Enable compiler-based static context analysis [1] for x86 KVM, along with
> prerequisite lock annotations in virt/kvm.
> 
> During the initial annotation pass, context analysis uncovered a missing
> SRCU read-side critical section in x86 PMU filter lookups; patch 1 fixes
> this bug.
> 
> The remainder of the series is strictly non-functional: it establishes
> basic function annotations, guarded_by annotations on core structs, and
> straightforward refactorings in hva/gfn range walks and guest_memfd to
> eliminate conditional locking patterns that cannot be tracked statically.
> 
> Annotating the MMU (mmu/mmu.c, mmu/tdp_mmu.c), i8259, and Xen requires
> more invasive changes and is deferred to follow-ups.
> 
> [1] https://docs.kernel.org/next/dev-tools/context-analysis.html

...

>  43 files changed, 745 insertions(+), 236 deletions(-)

For me, there needs to be a _lot_ more explanation of what this buys us to justify
the extra annotations and ongoing maintenance burden.  And to a lesser extent, why
we should rework code to play nice with context analysis (I assume we can simply
opt-out on a per-function basis if we don't want to play nice?).

I'd also like to see what the MMU changes look like before committing to supporting
this.

Re: [PATCH RFC 02/10] KVM: Allow reading memslots while holding slots_arch_lock

[Marco Elver]

On Thu, 10 Sept 2026 at 18:30, Sean Christopherson <seanjc@google.com> wrote:
>
> On Thu, Sep 10, 2026, Marco Elver wrote:
> > kvm_swap_active_memslots() updates kvm->memslots[as_id] while holding both
> > kvm->slots_lock and kvm->slots_arch_lock. Holding either lock guarantees
> > that memslots cannot be concurrently modified.
>
> Sure, but that's irrelevant.  The goal of the srcu_dereference_check() is to
> ensure that readers see a stable view of the VM's overall memory, not simply that
> kvm->memslots can't be written.

Functionally, this is irrelevant for readers. But under lockdep it
isn't for writers: srcu_dereference_check() (with lockdep) asserts
that the srcu reader-lock is held, or the condition 'c' holds, which
here is holding any of the writer locks.

> > Allow reading memslots in __kvm_memslots() when kvm->slots_arch_lock is
> > held.
>
> Why?

Holding any of the writer locks guarantees no concurrent modification;
therefore, if any writer lock is held, it's not required that the srcu
reader-lock is held. There are few places where only either slots_lock
or slots_arch_lock is held, which is sufficient for reading.

Re: [PATCH RFC 02/10] KVM: Allow reading memslots while holding slots_arch_lock

[Sean Christopherson]

On Thu, Sep 10, 2026, Marco Elver wrote:
> On Thu, 10 Sept 2026 at 18:30, Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Thu, Sep 10, 2026, Marco Elver wrote:
> > > kvm_swap_active_memslots() updates kvm->memslots[as_id] while holding both
> > > kvm->slots_lock and kvm->slots_arch_lock. Holding either lock guarantees
> > > that memslots cannot be concurrently modified.
> >
> > Sure, but that's irrelevant.  The goal of the srcu_dereference_check() is to
> > ensure that readers see a stable view of the VM's overall memory, not simply that
> > kvm->memslots can't be written.
> 
> Functionally, this is irrelevant for readers. But under lockdep it
> isn't for writers: srcu_dereference_check() (with lockdep) asserts
> that the srcu reader-lock is held, or the condition 'c' holds, which
> here is holding any of the writer locks.

No, the rules for writing kvm->memslots is that *both* are held.

> > > Allow reading memslots in __kvm_memslots() when kvm->slots_arch_lock is
> > > held.
> >
> > Why?
> 
> Holding any of the writer locks guarantees no concurrent modification;
> therefore, if any writer lock is held, it's not required that the srcu
> reader-lock is held. There are few places where only either slots_lock
> or slots_arch_lock is held, which is sufficient for reading.

Yes, but with caveats.  And more importantly, pure readers really shouldn't be
taking slots_arch_lock, because either it's overkill and will generate unnecessary
lock content, or the alleged reader is doing more than just reading.

Holding just slots_arch_lock *could* be fine, depending on the usage, but those
details matter, which is why I asked "why".  I want to know exactly why we should
relax the locking rules.

Ah, poking around the code, I suspect that the motivation is
kvm_enable_external_write_tracking()?  Which grabs __kvm_memslots() but only
holds slots_arch_lock, i.e. would get a lockdep splat if someone with KVMGT ran
with lockdep enabled.

That thing isn't a pure reader.  It only reads the actual kvm->memslots pointer,
but it writes to each of the slots metadata.

So, allowing __kvm_memslots() to be called with just slot_arch_lock is ok in
this situation, and if my guess is right, necessary to fix a false positive.  But
I'm on the fence as to whether or not we generally want to allow that, versus
taking kvm->srcu in kvm_enable_external_write_tracking() even though strictly
speaking it's unnecessary.

Re: [PATCH RFC 02/10] KVM: Allow reading memslots while holding slots_arch_lock

[Marco Elver]

On Thu, 10 Sept 2026 at 19:52, Sean Christopherson <seanjc@google.com> wrote:
> On Thu, Sep 10, 2026, Marco Elver wrote:
> > On Thu, 10 Sept 2026 at 18:30, Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > On Thu, Sep 10, 2026, Marco Elver wrote:
> > > > kvm_swap_active_memslots() updates kvm->memslots[as_id] while holding both
> > > > kvm->slots_lock and kvm->slots_arch_lock. Holding either lock guarantees
> > > > that memslots cannot be concurrently modified.
> > >
> > > Sure, but that's irrelevant.  The goal of the srcu_dereference_check() is to
> > > ensure that readers see a stable view of the VM's overall memory, not simply that
> > > kvm->memslots can't be written.
> >
> > Functionally, this is irrelevant for readers. But under lockdep it
> > isn't for writers: srcu_dereference_check() (with lockdep) asserts
> > that the srcu reader-lock is held, or the condition 'c' holds, which
> > here is holding any of the writer locks.
>
> No, the rules for writing kvm->memslots is that *both* are held.

Right. But the condition 'c' is only the srcu-less fallback for reads,
which is "any".

> > > > Allow reading memslots in __kvm_memslots() when kvm->slots_arch_lock is
> > > > held.
> > >
> > > Why?
> >
> > Holding any of the writer locks guarantees no concurrent modification;
> > therefore, if any writer lock is held, it's not required that the srcu
> > reader-lock is held. There are few places where only either slots_lock
> > or slots_arch_lock is held, which is sufficient for reading.
>
> Yes, but with caveats.  And more importantly, pure readers really shouldn't be
> taking slots_arch_lock, because either it's overkill and will generate unnecessary
> lock content, or the alleged reader is doing more than just reading.

Pure readers never should, and not what this patch suggested. But
writers (or those about to become writers if they only hold one lock)
can read w/o the srcu reader-lock held.

> Holding just slots_arch_lock *could* be fine, depending on the usage, but those
> details matter, which is why I asked "why".  I want to know exactly why we should
> relax the locking rules.
>
> Ah, poking around the code, I suspect that the motivation is
> kvm_enable_external_write_tracking()?  Which grabs __kvm_memslots() but only
> holds slots_arch_lock, i.e. would get a lockdep splat if someone with KVMGT ran
> with lockdep enabled.

Right.

> That thing isn't a pure reader.  It only reads the actual kvm->memslots pointer,
> but it writes to each of the slots metadata.
>
> So, allowing __kvm_memslots() to be called with just slot_arch_lock is ok in
> this situation, and if my guess is right, necessary to fix a false positive.  But
> I'm on the fence as to whether or not we generally want to allow that, versus
> taking kvm->srcu in kvm_enable_external_write_tracking() even though strictly
> speaking it's unnecessary.

Clarification here would be good. In building up the locking
annotations, this was one inconsistency I found vs. what
srcu_dereference_check() said, so the conclusion was that the fallback
condition 'c' was outdated (or there was a bug).

Re: [PATCH RFC 00/10] KVM: Enable Clang Context Analysis

[Marco Elver]

On Thu, 10 Sept 2026 at 18:55, Sean Christopherson <seanjc@google.com> wrote:
> On Thu, Sep 10, 2026, Marco Elver wrote:
> > Enable compiler-based static context analysis [1] for x86 KVM, along with
> > prerequisite lock annotations in virt/kvm.
> >
> > During the initial annotation pass, context analysis uncovered a missing
> > SRCU read-side critical section in x86 PMU filter lookups; patch 1 fixes
> > this bug.
> >
> > The remainder of the series is strictly non-functional: it establishes
> > basic function annotations, guarded_by annotations on core structs, and
> > straightforward refactorings in hva/gfn range walks and guest_memfd to
> > eliminate conditional locking patterns that cannot be tracked statically.
> >
> > Annotating the MMU (mmu/mmu.c, mmu/tdp_mmu.c), i8259, and Xen requires
> > more invasive changes and is deferred to follow-ups.
> >
> > [1] https://docs.kernel.org/next/dev-tools/context-analysis.html
>
> ...
>
> >  43 files changed, 745 insertions(+), 236 deletions(-)
>
> For me, there needs to be a _lot_ more explanation of what this buys us to justify
> the extra annotations and ongoing maintenance burden.  And to a lesser extent, why
> we should rework code to play nice with context analysis

Patch 1 is an example of what this buys: static checking shifts left
and prevents bugs from actually being committed. Context analysis can
be seen as an extension of kernel C, improving its safety, and the
nice thing is it can be enabled incrementally (no rewrites in other
languages needed, though yielding modest safety gains at a fraction of
the cost). A side-effect is compiler-checked documentation
(annotations), which helps humans and robots reading & writing kernel
code.

The question is simple: is the cost of ongoing maintenance with
context analysis greater than the cost of the bugs it prevents?

> (I assume we can simply
> opt-out on a per-function basis if we don't want to play nice?).

Yes with __context_unsafe(/* explanation */) attribute, or
context_unsafe(...) expressions.

> I'd also like to see what the MMU changes look like before committing to supporting
> this.

If you're not entirely opposed, a v2 can include that.