[PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes

[PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes

[Yosry Ahmed]

This series optimizes TLB flushes for nested SVM by using and maintaining
a separate ASID for L2 for each vCPU (similar to VMX's handling of VPIDs),
rather than sharing the same ASID and flushing everything on every nested
transition.

The series also drops the dynamic ASID allocation scheme and uses a
static ASID per vCPU (or two if using nested), bringing it closer to how
VMX handles VPIDs. Modern CPUs (Rome, Milan, Genoa, Turin) advertise 32K
ASIDs, so there is no need for dynamic allocation.

The Hyper-V bits only got build tested. The SEV bits were only tested
using the SEV selftests.

Otherwise, this series was tested in two ways:
1. A new selftest:
  - Tests most basic scenarios where L1 flushes the TLB for L2, as well
    as when KVM triggers a TLB flush for both L1 and L2.
  - On AMD, also tests that flushing (or changing) L2's ASID flushes
    nested NPT translations for all nCR3 roots, not just the current one.
  - Fails reliably when injecting manual TLB bugs (e.g. commenting out a
    flush).
  - Caught an actual bug when I was rebasing the series on top of
    Paolo's MMU structures rework.
  - Works for both SVM and VMX.
  - Was used to repro a separate nVMX TLB bug [1].

2. An AI generated script (now 1300 lines of bash that I still don't dare
   to read):
  - Runs 20 L1 VMs, split into two groups:
    - 10 L1 VMs with vCPUs pinned 1:1 to dedicated host CPUs, each
      running a single L2 VM with as many vCPUs as L1. This avoids TLB
      flushes on vCPU migration and L2 ASID switching, which would
      otherwise hide missed flushes.
    - 10 L1 VMs with vCPUs confined to half as many host CPUs, each
      running 4 L2 VMs. This overcommits CPU in both L0 and L1 to
      exercise TLB flushes on vCPU migration and L2 ASID switching.
  - Each L2 VM runs a workload that writes to most of its memory and then
    validates its content from a different vCPU.
  - L2 VMs overcommit memory in L1, and are periodically recycled to
    exercise ASID reallocation.
  - L0 periodically reclaims memory from L1 VMs proactively (exercise
    L0-induced TLB flushes).
  - The number of ASIDs is synthetically reduced (using a test-only
    patch) such that some L1 vCPUs use the fallback ASID for L2.
  - Runs with NPT enabled/disabled in L0 and L1.
  - Catches most manually injected TLB bugs, some only semi-reliably.
  - Caught an actual bug when testing the fallback ASID logic with
    synthetically reduced number of ASIDs (in v1).

Overall, I think I am happy with the testing I put this series through,
hence dropping the RFC tag at last.

Breakdown:
- Patch 1 is a bug fix that can be taken separately.
- Patches 2-11 implement the change to use static ASIDs per vCPU for
  SVM (including SEV-specific handling and fallback ASID optimization),
  and can also be taken separately if needed.
- Patches 12-28 add the L2 ASID and proper handling for it, leading to
  dropping the unconditional flushes on nested transitions.
- Patch 29 adds the nested TLB flush selftest.

v1 -> v2:
- Rebased on top of kvm-x86-next-2026.10.02. Drop the fixes to the
  dynamic ASID allocation that landed since v1 when switching to static
  ASIDs, as they are no longer needed (bumping the ASID generation on CPU
  online, resetting the ASID generation of both VMCBs on pCPU switch,
  and tracking pending full flushes per-CPU).
- Dropped "KVM: SVM: Duplicate pre-run ASID check for SEV and non-SEV
  guests", and keep syncing the ASID into the VMCB before VMRUN until
  static ASIDs are used, to avoid leaving vmcb02's ASID uninitialized
  for SEV guests in between [Sashiko].
- Explicitly initialize svm->asid during SEV VM migration in a separate
  patch, before the VMCB is initialized [Sashiko].
- Free any existing non-SEV ASID on the destination vCPUs during SEV VM
  migration to avoid leaking it, and decide whether an ASID needs to be
  freed based on the ASID range rather than the VM type [Sashiko].
- Update the L2 ASID (and vmcb02) during SEV VM migration if nested was
  already initialized on the destination vCPUs [Sashiko].
- Moved "KVM: nSVM: Drop svm->nested.initialized" before introducing the
  L2 ASID, so that the vmcb02 pointer can be reliably used to check if
  nested is initialized.
- Moved the WARN on L1 and L2 unexpectedly sharing an ASID to the patch
  that actually uses different ASIDs [Sashiko].
- Added KVM_REQ_MMU_SYNC_ALL_ROOTS, and sync all nested NPT roots (not
  just the current one) when L1 flushes or changes L2's ASID, as shadow
  NPT mappings are not tagged by ASID.
- Reset the last L2 ASID if nested VMRUN fails, to avoid losing the
  nested NPT sync.
- Extended the selftest to cover flushing with multiple nested NPT roots.
- Typo and changelog fixes.

RFC v2 -> v1:
- Dropped the RFC tag.
- Document KVM's handling of number of ASIDs in CPUID rather than
  increase the number of advertised ASIDs [Jim, Sean].
- Explicitly support reserved TLB tags rather than a minimum TLB tag
  [Sean].
- Set an upper bound on the number of TLB tags [Sean].
- Allow reserving all ASIDs to allow allocating all ASIDs to SEV (if at
  all possible). [Sashiko].
- Actually handle TLB flushes for nested on SEV [Sean, Sashiko].
- Optimize fallback ASID flushes by tracking the last vCPU using it on
  each pCPU instead of flushing it on every VMRUN [Sean].
- Drop svm->nested.initialized and properly NULLify the vmcb02 pointer
  instead (and use it).
- Make sure servicing local TLB flushes is done when both the VMCB and
  guest_mode are consistent [Sashiko].
- Always initialize last_asid to 0 to flush the TLB on the first use of
  L2's ASID [Sashiko].
- Open-code TLB flush handling on nested VM-Enter and VM-Exit instead of
  burying it in svm_switch_vmcb() [Sean].
- Always flush the hardware ASID on nested transitions if L1 and L2
  share it (e.g. when using the fallback ASID, or for SEV guests) to
  prevent TLB entry leakage between L1 and L2.
- Rename __kvm_mmu_invalidate_addr() to kvm_mmu_sync_addr() [Sean].

RFC v1: https://lore.kernel.org/lkml/20250326193619.3714986-1-yosry.ahmed@linux.dev/
RFC v2: https://lore.kernel.org/kvm/20260616004155.1435766-1-yosry@kernel.org/
v1: https://lore.kernel.org/kvm/20260728003557.1136583-1-yosry@kernel.org/

[1] https://lore.kernel.org/kvm/20260722230128.1587363-1-yosry@kernel.org/

Yosry Ahmed (29):
  KVM: nSVM: Flush the TLB after forcefully leaving nested
  KVM: SVM: Document number of ASIDs CPUID setting
  KVM: VMX: Generalize VPID allocation to be vendor-neutral
  KVM: x86/mmu: Support specifying reserved TLB tags
  KVM: SVM: Add helpers to set/clear ASID flush in VMCB
  KVM: SVM: Fallback to flush everything if FLUSHBYASID is not available
  KVM: SEV: Do ASID initialization at VMCB initialization
  KVM: SEV: Expose sev_get_asid() outside of sev.c
  KVM: SEV: Explicitly initialize the per vCPU ASID on SEV VM migration
  KVM: SVM: Use a static ASID per vCPU
  KVM: SVM: Only flush the fallback ASID when used by a different vCPU
  KVM: nSVM: Drop svm->nested.initialized
  KVM: nSVM: Add a placeholder ASID for L2
  KVM: x86: hyper-v: Rename kvm_hv_vcpu_purge_flush_tlb()
  KVM: x86: hyper-v: Allow purging all TLB flush FIFOs
  KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH
  KVM: nSVM: Always switch VMCB before leaving guest mode
  KVM: nSVM: Split nested_svm_transition_tlb_flush() into entry/exit fns
  KVM: nSVM: Service local TLB flushes before nested transitions
  KVM: x86: Add KVM_REQ_MMU_SYNC_ALL_ROOTS
  KVM: nSVM: Handle nested TLB flush requests through TLB_CONTROL
  KVM: nSVM: Flush the TLB if L1 changes L2's ASID in vmcb12
  KVM: nSVM: Do not reset TLB_CONTROL in vmcb02 on nested VM-Enter
  KVM: x86/mmu: Rename __kvm_mmu_invalidate_addr() to
    kvm_mmu_sync_addr()
  KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the GVA
    flush
  KVM: nSVM: Flush L2's ASID when emulating INVLPGA
  KVM: nSVM: Flush the ASID on nested transitions if shared by L1 and L2
  KVM: nSVM: Use different ASIDs for L1 and L2
  KVM: selftests: Add a test for nested TLB flushes

 Documentation/virt/kvm/api.rst                |  20 +
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/cpuid.c                          |   4 +-
 arch/x86/kvm/hyperv.h                         |  33 +-
 arch/x86/kvm/mmu.h                            |  18 +
 arch/x86/kvm/mmu/mmu.c                        | 126 ++++-
 arch/x86/kvm/svm/nested.c                     | 151 +++--
 arch/x86/kvm/svm/sev.c                        |  37 +-
 arch/x86/kvm/svm/svm.c                        | 150 +++--
 arch/x86/kvm/svm/svm.h                        |  55 +-
 arch/x86/kvm/vmx/vmx.c                        |  40 +-
 arch/x86/kvm/vmx/vmx.h                        |  28 +-
 arch/x86/kvm/x86.c                            |  13 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 tools/testing/selftests/kvm/include/x86/svm.h |   5 +
 tools/testing/selftests/kvm/include/x86/vmx.h |  21 +
 .../testing/selftests/kvm/lib/x86/processor.c |  36 ++
 .../selftests/kvm/x86/nested_tlb_flush_test.c | 526 ++++++++++++++++++
 19 files changed, 1093 insertions(+), 173 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c


base-commit: 6bd2905303c58679e303941b7d8ae8c074cd95ce
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 01/29] KVM: nSVM: Flush the TLB after forcefully leaving nested

[Yosry Ahmed]

KVM flushes the TLB on nested VM-Enter and nested VM-Exit, but not when
forcefully leaving nested. In this case, L2 TLB entries can leak into
L1. Flush the TLB after forcefully exiting L2, similar to nested
VM-Exits.

Note that vmx_leave_nested() handles this correctly, as it reuses
nested_vmx_vmexit(), which handles the necessary TLB flushes on an L2 ->
L1 transition.

Cc: stable@vger.kernel.org
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index af0ef678433d7..2bfc9d2c341b1 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1558,6 +1558,8 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
 
 		svm_switch_vmcb(svm, &svm->vmcb01);
 
+		nested_svm_transition_tlb_flush(vcpu);
+
 		nested_svm_uninit_mmu_context(vcpu);
 		vmcb_mark_all_dirty(svm->vmcb);
 
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 02/29] KVM: SVM: Document number of ASIDs CPUID setting

[Yosry Ahmed]

KVM arbitrarily advertises 8 ASIDs in KVM_GET_SUPPORTED_CPUID for
historical reasons. Advertising a small number of ASIDs to L1 is
currently harmless, as KVM (L0) uses the same ASID for both L1 and L2,
and flushes that ASID on nested transitions.

However, following changes will add proper ASID emulation and a separate
ASID for L2, minimizing the TLB flushes on nested transitions. At that
point, a full TLB flush from a KVM guest (L1) would flush both L1 and L2
ASIDs, so should be avoided as much as possible by advertising more
ASIDs to L1.

Increasing the value advertised by KVM is tricky, because a VMM copying
the number of ASIDs from KVM_GET_SUPPORTED_CPUID may perceive a new VM
with a higher number of ASIDs as incompatible with older KVM (which is
not true).

Avoid changing the value advertised by KVM, but document that it is
essentially meaningless and does not represent a limit or a hint. Also,
more-or-less encourage userspace to advertise a higher value to avoid
potential unnecessary TLB flushes from an L1 hypervisor.

Suggested-by: Jim Mattson <jmattson@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 Documentation/virt/kvm/api.rst | 20 ++++++++++++++++++++
 arch/x86/kvm/cpuid.c           |  4 ++--
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ddc61ea36ac59..60f49b3675920 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -9770,6 +9770,26 @@ the APIC ID is found in EDX for all subleaves of 0x0b and 0x1f, and in EAX
 for 0x8000001e; the latter also encodes the core id and node id in bits
 7:0 of EBX and ECX respectively.
 
+Number of ASIDs (nested SVM)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CPUID[0x8000000A]:EBX reports the number of ASIDs supported by the processor
+on AMD CPUs. KVM's ``KVM_GET_SUPPORTED_CPUID`` hardcodes this value to 8 for
+historical reasons. This value is arbitrary and does not represent a limit
+on the supported number of ASIDs, or even a reasonable default value.
+
+KVM allows userspace to set any value (up to 0xFFFFFFFF), and emulates those
+ASIDs by multiplexing them onto hardware ASIDs. AMD CPUs have only ever
+supported 3 values: 0x40, 0x8000, and 0x1000 -- with modern CPUs advertising
+0x8000.
+
+Advertising a small number of ASIDs could lead an L1 hypervisor to perform
+unnecessary TLB flushes (e.g. older KVM would flush the entire TLB when running
+out of ASIDs). Userspace should probably set the number of ASIDs to a
+reasonably high power of 2 value to remain consistent with hardware and avoid
+any misbehavior by L1 hypervisors, ideally one of the values advertised by
+actual hardware.
+
 Obsolete ioctls and capabilities
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index adafd6fa2c542..e45c01d6faeaf 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -1846,8 +1846,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 			break;
 		}
 		entry->eax = 1; /* SVM revision 1 */
-		entry->ebx = 8; /* Lets support 8 ASIDs in case we add proper
-				   ASID emulation to nested SVM */
+		entry->ebx = 8; /* Arbitrary value for historical reasons. KVM
+				   practically allows and ignores all values. */
 		entry->ecx = 0; /* Reserved */
 		cpuid_entry_override(entry, CPUID_8000_000A_EDX);
 		break;
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 03/29] KVM: VMX: Generalize VPID allocation to be vendor-neutral

[Yosry Ahmed]

In preparation for sharing with SVM, generalize the VMX VPID allocation
code and move it to common code as a TLB tags allocator. Parameterize
the TLB tags allocator by the number of tags, and allocate the bitmap
dynamically. Opportunisitcally use guards to acquire the lock instead of
spin_{lock/unlock}().

Keep the number of allowed tags capped at VMX's hardware cap, to avoid
allocating a huge bitmap if L0 advertises a huge number of ASIDs to an
L1 KVM. Realistically, the number of actual hardware ASIDs wouldn't be
that large so there is no benefit.

Initialize the TLB tags allocator during hardware setup/unsetup, and
reserve tag=0 during initialziation, similar to how VPID=0 is currently
reserved in the VMX-specific bitmap during hardware setup.

Allow nr=0 to allow allocating all ASIDs to SEV on AMD without failing
hardware setup, if at all possible, in which case any tag allocation
fails (SEV won't use the tag allocator).  The number of tags includes
tag=0, which is not usable. The interface is a little confusing in that
regard, but this will be changed soon when reserved tags are explicitly
introduced.

Keep allocate_vpid() and free_vpid() as wrapper that check enable_vpid
to avoid checking at all callsites, and add init_vpids() and
destroy_vpids() to wrap init/destroy calls as well.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/mmu.h     |  8 +++++
 arch/x86/kvm/mmu/mmu.c | 78 ++++++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/vmx.c | 40 +++++-----------------
 arch/x86/kvm/vmx/vmx.h | 28 ++++++++++++---
 4 files changed, 119 insertions(+), 35 deletions(-)

diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 2ae7f9ed4cf86..de79e002edf8f 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -410,4 +410,12 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)
 {
 	return gfn & kvm_gfn_direct_bits(kvm);
 }
+
+typedef unsigned int kvm_tlb_tag_t;
+
+int kvm_init_tlb_tags(unsigned int nr);
+void kvm_destroy_tlb_tags(void);
+kvm_tlb_tag_t kvm_alloc_tlb_tag(void);
+void kvm_free_tlb_tag(kvm_tlb_tag_t tag);
+
 #endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index aadd5bdc2c97a..97d49247d9d82 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8178,6 +8178,84 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
 		vhost_task_stop(kvm->arch.nx_huge_page_recovery_thread);
 }
 
+static struct {
+	spinlock_t	lock;
+	unsigned long	*bitmap;
+	unsigned int	nr;
+} tlb_tags;
+
+int kvm_init_tlb_tags(unsigned int nr)
+{
+	/*
+	 * Limit the number of TLB tags to VMX's hardcoded maximum of 0x10000
+	 * to avoid wasting memory for the bitmap in the unlikely scenario the
+	 * CPU supports an inordinate number of ASIDs (on AMD).  If userspace
+	 * wants to concurrently run tens of thousands of vCPUs, they'll likely
+	 * need a solution that works for both Intel and AMD.
+	 */
+	const unsigned int MAX_NR_TLB_TAGS = VMX_NR_VPIDS;
+
+	if (!nr)
+		return 0;
+
+	if (nr > MAX_NR_TLB_TAGS) {
+		pr_warn_once("Number of TLB tags capped (%u instead of %u)\n",
+			     MAX_NR_TLB_TAGS, nr);
+		nr = MAX_NR_TLB_TAGS;
+	}
+
+	tlb_tags.bitmap = bitmap_zalloc(nr, GFP_KERNEL);
+	if (!tlb_tags.bitmap)
+		return -ENOMEM;
+
+	/*
+	 * 0 is the host's TLB tag for both VMX's VPID and SVM's ASID, and is
+	 * returned on failed allocations (e.g. no more tags left).
+	 */
+	__set_bit(0, tlb_tags.bitmap);
+
+	tlb_tags.nr = nr;
+	spin_lock_init(&tlb_tags.lock);
+	return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_tlb_tags);
+
+void kvm_destroy_tlb_tags(void)
+{
+	bitmap_free(tlb_tags.bitmap);
+	tlb_tags.bitmap = NULL;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags);
+
+kvm_tlb_tag_t kvm_alloc_tlb_tag(void)
+{
+	kvm_tlb_tag_t tag;
+
+	if (!tlb_tags.bitmap)
+		return 0;
+
+	guard(spinlock)(&tlb_tags.lock);
+
+	tag = find_first_zero_bit(tlb_tags.bitmap, tlb_tags.nr);
+	if (tag >= tlb_tags.nr)
+		return 0;
+
+	__set_bit(tag, tlb_tags.bitmap);
+	return tag;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_alloc_tlb_tag);
+
+void kvm_free_tlb_tag(kvm_tlb_tag_t tag)
+{
+	if (!tag || WARN_ON_ONCE(tag >= tlb_tags.nr))
+		return;
+
+	guard(spinlock)(&tlb_tags.lock);
+
+	__clear_bit(tag, tlb_tags.bitmap);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_free_tlb_tag);
+
 #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
 static bool hugepage_test_mixed(struct kvm_memory_slot *slot, gfn_t gfn,
 				int level)
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index b341af4c6057a..a1caa2bd04d7a 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -598,9 +598,6 @@ DEFINE_PER_CPU(struct vmcs *, current_vmcs);
  */
 static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
 
-static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);
-static DEFINE_SPINLOCK(vmx_vpid_lock);
-
 struct vmcs_config vmcs_config __ro_after_init;
 struct vmx_capability vmx_capability __ro_after_init;
 
@@ -4092,31 +4089,6 @@ static void seg_setup(int seg)
 	vmcs_write32(sf->ar_bytes, ar);
 }
 
-int allocate_vpid(void)
-{
-	int vpid;
-
-	if (!enable_vpid)
-		return 0;
-	spin_lock(&vmx_vpid_lock);
-	vpid = find_first_zero_bit(vmx_vpid_bitmap, VMX_NR_VPIDS);
-	if (vpid < VMX_NR_VPIDS)
-		__set_bit(vpid, vmx_vpid_bitmap);
-	else
-		vpid = 0;
-	spin_unlock(&vmx_vpid_lock);
-	return vpid;
-}
-
-void free_vpid(int vpid)
-{
-	if (!enable_vpid || vpid == 0)
-		return;
-	spin_lock(&vmx_vpid_lock);
-	__clear_bit(vpid, vmx_vpid_bitmap);
-	spin_unlock(&vmx_vpid_lock);
-}
-
 static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx)
 {
 	/*
@@ -8415,6 +8387,8 @@ void vmx_hardware_unsetup(void)
 
 	if (nested)
 		nested_vmx_hardware_unsetup();
+
+	destroy_vpids();
 }
 
 void vmx_vm_destroy(struct kvm *kvm)
@@ -8640,8 +8614,6 @@ __init int vmx_hardware_setup(void)
 	kvm_caps.has_bus_lock_exit = cpu_has_vmx_bus_lock_detection();
 	kvm_caps.has_notify_vmexit = cpu_has_notify_vmexit();
 
-	set_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */
-
 	if (enable_ept)
 		kvm_mmu_set_ept_masks(enable_ept_ad_bits);
 	else
@@ -8681,6 +8653,10 @@ __init int vmx_hardware_setup(void)
 
 	vmx_set_cpu_caps();
 
+	r = init_vpids();
+	if (r)
+		return r;
+
 	/*
 	 * Configure nested capabilities after core CPU capabilities so that
 	 * nested support can be conditional on base support, e.g. so that KVM
@@ -8688,8 +8664,10 @@ __init int vmx_hardware_setup(void)
 	 */
 	if (nested) {
 		r = nested_vmx_hardware_setup(kvm_vmx_exit_handlers);
-		if (r)
+		if (r) {
+			destroy_vpids();
 			return r;
+		}
 	}
 	vmx_nested_ops.enabled = nested;
 
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index 6c9dd5b2365d5..a8df797163ecc 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -182,7 +182,7 @@ struct nested_vmx {
 	u64 pre_vmenter_ssp;
 	u64 pre_vmenter_ssp_tbl;
 
-	u16 vpid02;
+	kvm_tlb_tag_t vpid02;
 	u16 last_vpid;
 
 	int tsc_autostore_slot;
@@ -256,7 +256,7 @@ struct vcpu_vmx {
 			u32 ar;
 		} seg[8];
 	} segment_cache;
-	int vpid;
+	kvm_tlb_tag_t vpid;
 
 	/* Support for a guest hypervisor (nested VMX) */
 	struct nested_vmx nested;
@@ -297,9 +297,29 @@ struct kvm_vmx {
 	u64 *pid_table;
 };
 
+static __always_inline int init_vpids(void)
+{
+	return enable_vpid ? kvm_init_tlb_tags(VMX_NR_VPIDS) : 0;
+}
+
+static __always_inline void destroy_vpids(void)
+{
+	if (enable_vpid)
+		kvm_destroy_tlb_tags();
+}
+
+static __always_inline kvm_tlb_tag_t allocate_vpid(void)
+{
+	return enable_vpid ? kvm_alloc_tlb_tag() : 0;
+}
+
+static __always_inline void free_vpid(kvm_tlb_tag_t vpid)
+{
+	if (enable_vpid)
+		kvm_free_tlb_tag(vpid);
+}
+
 void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu);
-int allocate_vpid(void);
-void free_vpid(int vpid);
 void vmx_set_constant_host_state(struct vcpu_vmx *vmx);
 void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);
 void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_sel,
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 04/29] KVM: x86/mmu: Support specifying reserved TLB tags

[Yosry Ahmed]

In preparation for using the TLB tags allocator for SVM, which has a
range of ASIDs allocated for SEV/SNP, pass in the number of reserved TLB
tags when initializing the TLB tags allocator. The bitmap is
conceptually shifted such that bit=0 corresponds to tag=nr_reserved.

Specifying the minimum value during initialization also makes the API
clearer, as tag=0 is now explicitly reserved by the caller.

No functional change intended for VMX as VPID=0 was already implicitly
reserved.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/mmu.h     |  2 +-
 arch/x86/kvm/mmu/mmu.c | 42 ++++++++++++++++++++++++++++--------------
 arch/x86/kvm/vmx/vmx.h |  2 +-
 3 files changed, 30 insertions(+), 16 deletions(-)

diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index de79e002edf8f..b443a5083bfb0 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -413,7 +413,7 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)
 
 typedef unsigned int kvm_tlb_tag_t;
 
-int kvm_init_tlb_tags(unsigned int nr);
+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved);
 void kvm_destroy_tlb_tags(void);
 kvm_tlb_tag_t kvm_alloc_tlb_tag(void);
 void kvm_free_tlb_tag(kvm_tlb_tag_t tag);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 97d49247d9d82..c096c1a2b4684 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8182,9 +8182,17 @@ static struct {
 	spinlock_t	lock;
 	unsigned long	*bitmap;
 	unsigned int	nr;
+	kvm_tlb_tag_t	start;
 } tlb_tags;
 
-int kvm_init_tlb_tags(unsigned int nr)
+/*
+ * Initialize @nr_total TLB tags, where @nr_reserved of those tags are reserved.
+ * Reserved tags start at 0 and are contiguous. Hence, usable tags are
+ * [nr_reserved, nr_total). At least one reserved tag is required, as 0 is the
+ * host TLB tag on both VMX and AMD, and is the value returned on a failed tag
+ * allocation.
+ */
+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved)
 {
 	/*
 	 * Limit the number of TLB tags to VMX's hardcoded maximum of 0x10000
@@ -8194,7 +8202,12 @@ int kvm_init_tlb_tags(unsigned int nr)
 	 * need a solution that works for both Intel and AMD.
 	 */
 	const unsigned int MAX_NR_TLB_TAGS = VMX_NR_VPIDS;
+	unsigned int nr;
 
+	if (WARN_ON_ONCE(!nr_reserved || nr_reserved > nr_total))
+		return -EINVAL;
+
+	nr = nr_total - nr_reserved;
 	if (!nr)
 		return 0;
 
@@ -8208,13 +8221,8 @@ int kvm_init_tlb_tags(unsigned int nr)
 	if (!tlb_tags.bitmap)
 		return -ENOMEM;
 
-	/*
-	 * 0 is the host's TLB tag for both VMX's VPID and SVM's ASID, and is
-	 * returned on failed allocations (e.g. no more tags left).
-	 */
-	__set_bit(0, tlb_tags.bitmap);
-
 	tlb_tags.nr = nr;
+	tlb_tags.start = nr_reserved;
 	spin_lock_init(&tlb_tags.lock);
 	return 0;
 }
@@ -8229,30 +8237,36 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags);
 
 kvm_tlb_tag_t kvm_alloc_tlb_tag(void)
 {
-	kvm_tlb_tag_t tag;
+	unsigned int bit;
 
 	if (!tlb_tags.bitmap)
 		return 0;
 
 	guard(spinlock)(&tlb_tags.lock);
 
-	tag = find_first_zero_bit(tlb_tags.bitmap, tlb_tags.nr);
-	if (tag >= tlb_tags.nr)
+	bit = find_first_zero_bit(tlb_tags.bitmap, tlb_tags.nr);
+	if (bit >= tlb_tags.nr)
 		return 0;
 
-	__set_bit(tag, tlb_tags.bitmap);
-	return tag;
+	__set_bit(bit, tlb_tags.bitmap);
+	return tlb_tags.start + bit;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_alloc_tlb_tag);
 
 void kvm_free_tlb_tag(kvm_tlb_tag_t tag)
 {
-	if (!tag || WARN_ON_ONCE(tag >= tlb_tags.nr))
+	unsigned int bit;
+
+	if (!tag || WARN_ON_ONCE(tag < tlb_tags.start))
+		return;
+
+	bit = tag - tlb_tags.start;
+	if (WARN_ON_ONCE(bit >= tlb_tags.nr))
 		return;
 
 	guard(spinlock)(&tlb_tags.lock);
 
-	__clear_bit(tag, tlb_tags.bitmap);
+	__clear_bit(bit, tlb_tags.bitmap);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_free_tlb_tag);
 
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index a8df797163ecc..3e1ecd9148e9c 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -299,7 +299,7 @@ struct kvm_vmx {
 
 static __always_inline int init_vpids(void)
 {
-	return enable_vpid ? kvm_init_tlb_tags(VMX_NR_VPIDS) : 0;
+	return enable_vpid ? kvm_init_tlb_tags(VMX_NR_VPIDS, 1) : 0;
 }
 
 static __always_inline void destroy_vpids(void)
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 05/29] KVM: SVM: Add helpers to set/clear ASID flush in VMCB

[Yosry Ahmed]

Introduce set/clear helpers to set tlb_ctl to
TLB_CONTROL_FLUSH_ASID or TLB_CONTROL_DO_NOTHING. Incoming changes will
eliminate the use of TLB_CONTROL_FLUSH_ALL_ASID except as a fallback
when FLUSHBYASID is not available, so the helpers will create a common
path for this.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c |  2 +-
 arch/x86/kvm/svm/sev.c    |  2 +-
 arch/x86/kvm/svm/svm.c    |  4 ++--
 arch/x86/kvm/svm/svm.h    | 10 ++++++++++
 4 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 2bfc9d2c341b1..c3c12a4d6fe07 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -939,7 +939,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	/* Done at vmrun: asid.  */
 
 	/* Also overwritten later if necessary.  */
-	vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+	vmcb_clr_flush_asid(vmcb02);
 
 	/* Use vmcb01 MMU and format if guest does not use nNPT */
 	if (nested_npt_enabled(svm)) {
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index fad72cab76cf4..74c4555a32650 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3637,7 +3637,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 		return 0;
 
 	sd->sev_vmcbs[asid] = svm->vmcb;
-	svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+	vmcb_set_flush_asid(svm->vmcb);
 	return 0;
 }
 
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 136069dfcd44f..ad4f172f170d9 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4327,7 +4327,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
 	 * VM-Exit (via kvm_mmu_reset_context()).
 	 */
 	if (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))
-		svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+		vmcb_set_flush_asid(svm->vmcb);
 	else
 		svm->current_vmcb->asid_generation--;
 }
@@ -4746,7 +4746,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
 	if (!svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
 		this_cpu_ptr(&svm_data)->flush_all_asids = false;
-		svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+		vmcb_clr_flush_asid(svm->vmcb);
 
 		/*
 		 * Unconditionally mask off the CLEAR_RAP bit, the AND is just
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 6d0242966dc87..ba5077b9838c0 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -480,6 +480,16 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
 	vmcb_mark_dirty(vmcb, VMCB_NPT);
 }
 
+static inline void vmcb_set_flush_asid(struct vmcb *vmcb)
+{
+	vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+}
+
+static inline void vmcb_clr_flush_asid(struct vmcb *vmcb)
+{
+	vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+}
+
 static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
 {
 	return container_of(vcpu, struct vcpu_svm, vcpu);
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 06/29] KVM: SVM: Fallback to flush everything if FLUSHBYASID is not available

[Yosry Ahmed]

Currently, if FLUSHBYASID is not available when performing a TLB flush,
the fallback is decrementing the ASID generation to trigger allocating a
new ASID. In preparation for using a static ASID per vCPU (like VMX),
just fallback to flushing everything if FLUSHBYASID is not available.
This is probably worse from a performance perspective, but FLUSHBYASID
has been around for ~15 years and it's not worth carrying the
complexity.

Move the fallback logic into vmcb_set_flush_asid(), as more callers will
be added and will need the fallback as well. The only other current
caller is from SEV code, and no CPUs support SEV but not FLUSHBYASID (at
least according to AI).

Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/svm.c | 5 +----
 arch/x86/kvm/svm/svm.h | 5 ++++-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index ad4f172f170d9..0c3befd9bc850 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4326,10 +4326,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
 	 * unconditionally does a TLB flush on both nested VM-Enter and nested
 	 * VM-Exit (via kvm_mmu_reset_context()).
 	 */
-	if (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))
-		vmcb_set_flush_asid(svm->vmcb);
-	else
-		svm->current_vmcb->asid_generation--;
+	vmcb_set_flush_asid(svm->vmcb);
 }
 
 static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index ba5077b9838c0..3b59561d6cc2c 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -482,7 +482,10 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
 
 static inline void vmcb_set_flush_asid(struct vmcb *vmcb)
 {
-	vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+	if (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))
+		vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+	else
+		vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
 }
 
 static inline void vmcb_clr_flush_asid(struct vmcb *vmcb)
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 07/29] KVM: SEV: Do ASID initialization at VMCB initialization

[Yosry Ahmed]

The per-vCPU ASID does not change for SEV guests, so move the per-vCPU
(and per-VMCB) ASID initialization to sev_init_vmcb() to do it once,
instead of setting it before every VMRUN.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/sev.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 74c4555a32650..b58e3831b8f05 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3623,9 +3623,6 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 	if (!cpumask_test_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus))
 		cpumask_set_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus);
 
-	/* Assign the asid allocated with this SEV guest */
-	svm->asid = asid;
-
 	/*
 	 * Flush guest TLB:
 	 *
@@ -4889,6 +4886,10 @@ void sev_init_vmcb(struct vcpu_svm *svm, bool init_event)
 	svm->vmcb->control.misc_ctl |= SVM_MISC_ENABLE_SEV;
 	clr_exception_intercept(svm, UD_VECTOR);
 
+	svm->asid = sev_get_asid(vcpu->kvm);
+	svm->vmcb->control.asid = svm->asid;
+	vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
+
 	/*
 	 * Don't intercept #GP for SEV guests, e.g. for the VMware backdoor, as
 	 * KVM can't decrypt guest memory to decode the faulting instruction.
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 08/29] KVM: SEV: Expose sev_get_asid() outside of sev.c

[Yosry Ahmed]

Move the function definition to svm.h in preparation for using it in
more SVM code.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/sev.c |  5 -----
 arch/x86/kvm/svm/svm.h | 10 ++++++++++
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index b58e3831b8f05..706a91db7b44a 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -315,11 +315,6 @@ static int sev_asid_new(struct kvm_sev_info *sev, unsigned long vm_type)
 	return ret;
 }
 
-static unsigned int sev_get_asid(struct kvm *kvm)
-{
-	return to_kvm_sev_info(kvm)->asid;
-}
-
 static void sev_asid_free(struct kvm_sev_info *sev)
 {
 	struct svm_cpu_data *sd;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 3b59561d6cc2c..14b57c5cdf916 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -432,6 +432,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)
 {
 	return ____sev_snp_guest(vcpu->kvm);
 }
+
+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)
+{
+	return to_kvm_sev_info(kvm)->asid;
+}
 #else
 static __always_inline bool is_sev_guest(struct kvm_vcpu *vcpu)
 {
@@ -446,6 +451,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)
 {
 	return false;
 }
+
+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)
+{
+	return 0;
+}
 #endif
 
 static inline bool ghcb_gpa_is_registered(struct vcpu_svm *svm, u64 val)
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 09/29] KVM: SEV: Explicitly initialize the per vCPU ASID on SEV VM migration

[Yosry Ahmed]

Explicitly initialize svm->asid on the destination vCPUs as part of SEV
VM migration, before calling sev_init_vmcb() to make sure the per-vCPU
ASID is initialized before the VMCB. This is currently unnecessary as
sev_init_vmcb() initializes the per-vCPU ASID anyway, but will be needed
when the ASID initialization is moved to vCPU creation.

Add a helper function for the ASID migration as more changes will be
needed with incoming changes.

Note that svm->asid is left as-is on the source vCPUs, even though the
ASID now belongs to the destination VM. Clearing it would be a nice
hardening measure, but it is not as simple as clearing src_svm->asid, as
the source and destination VMs do not necessarily have the same number
of vCPUs (e.g. for non SEV-ES), so NULL-handling would be needed in a
couple of places. Leave it alone for now.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/sev.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 706a91db7b44a..0fbb72d57e6da 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2032,6 +2032,12 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
 	atomic_set_release(&src_sev->migration_in_progress, 0);
 }
 
+static void sev_vcpu_migrate_asid(struct vcpu_svm *dst_svm,
+				  unsigned int asid)
+{
+	dst_svm->asid = asid;
+}
+
 static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
 {
 	struct kvm_sev_info *dst = to_kvm_sev_info(dst_kvm);
@@ -2096,6 +2102,7 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
 	kvm_for_each_vcpu(i, dst_vcpu, dst_kvm) {
 		dst_svm = to_svm(dst_vcpu);
 
+		sev_vcpu_migrate_asid(dst_svm, dst->asid);
 		sev_init_vmcb(dst_svm, false);
 
 		if (!dst->es_active)
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 10/29] KVM: SVM: Use a static ASID per vCPU

[Yosry Ahmed]

Switch from dynamic ASID allocation to a static per-vCPU ASID.  The
dynamic ASID allocation logic is now only effectively used when running
out of ASIDs on a CPU (uncommon on modern hardware), or switching CPUs.

The per-VMCB ASID generation is initialized to 0, which never matches the
per-CPU ASID generation (bumped every time a CPU is enabled for
virtualization). This leads to a TLB flush of the ASID on the first VMRUN,
allocating a new ASID, and bumping the per-VMCB generation to match the
per-CPU generation. The ASID remains static until one of the following:

- The vCPU runs on a new CPU, in which case KVM resets the generation of
  both VMCBs to allocate a new ASID on the new CPU.

- KVM hits the maximum ASID on a CPU and increments the per-CPU ASID
  generation, at which point all vCPUs on this CPU will allocate a new
  ASID.

- The CPU is re-enabled for virtualization (e.g. after CPU hotplug), at
  which point the per-CPU generation is bumped and all vCPUs that
  previously ran on it will allocate a new ASID.

Drop the complexity and make the ASID static for each vCPU.  This makes
SVM's handling of ASIDs closer to VMX's handling of VPIDs, and
simplifies the code. It also completely avoids full TLB flushes (i.e.
TLB_CONTROL_FLUSH_ALL_ASID) on systems with FlushByAsid.  Full flushes
previously happened when updating the generation, so not so common, but
should generally be avoided as they cause a VMRUN to invalidate the TLB
entries for other VMs as well as the host.

When a vCPU is migrated to a new physical CPU, flush the (now static)
ASID instead of allocating a new one. This might cause extra TLB flushes
when switching CPUs (compared to just using a new ASID), but odds are
that the TLB is cold on the new CPU anyway.

Static ASIDs also make the following fixes to the dynamic ASID
allocation scheme unnecessary, so drop them:

- Bumping the per-CPU generation when enabling virtualization on a CPU,
  added by commit 25f744ffa0c8 ("KVM: SVM: Bump asid_generation on CPU
  online to avoid ASID collision after hotplug").  A vCPU can no longer
  run with an ASID that is (re)assigned to another vCPU.

- Resetting the generation of both VMCBs on a CPU switch, added by
  commit bf260dc9e7b8 ("KVM: SVM: Trigger new ASID allocation in both
  VMCBs on pCPU switch").  An ASID is now owned by the same vCPU on all
  CPUs, so a VMCB that misses a CPU switch can no longer end up using an
  ASID owned by another vCPU on that CPU.

- Tracking a pending full flush per-CPU (i.e. flush_all_asids), added by
  commit 60d93c27859f ("KVM: SVM: Preserve TLB control (i.e. pending TLB
  flush) on failed VMRUN").  The full flush on generation wraparound was
  needed on behalf of all vCPUs on the CPU, as their ASIDs were about to
  be recycled, so it had to be tracked per-CPU instead of in the VMCB of
  the vCPU that happened to trigger the wraparound.  With static ASIDs,
  TLB flushes are only needed for the vCPU's own ASID, so tracking them in
  the VMCB is sufficient, and a pending flush in the VMCB is still
  preserved on a failed VMRUN.

As using ASIDs cannot be disabled like VPIDs, allocate a fallback ASID
to be shared by all vCPUs after running out of ASIDs (and flushed on
every vCPU run, for now).  If a fallback ASID cannot be allocated, fail
vCPU creation (for non-SEV guests), but do not fail SVM initialization,
to allow for allocating all ASIDs for SEV (if at all possible).

For SEV, the same ASID continues to be used by all vCPUs in a VM. Move
svm->asid initialization from SEV-specific VMCB initialization to vCPU
creation, such that svm->asid is always initialized on vCPU creation for
both SEV and non-SEV vCPUs. This makes the logic simpler to follow, and
makes it easier to hide it in allocate_asid() and free_asid() helpers.
Decide whether to free an ASID based on the ASID range rather than the
VM type, as an SEV VM's vCPUs may stop being SEV vCPUs (or vice versa)
as a result of SEV VM migration.

Keep the ASID initialization in sev_init_vmcb(), even though it's
already done in init_vmcb(), to handle SEV VM migration.  svm->asid is
already reinitialized on destination SEV VMs during migration before
VMCB initialization, so the ASID ends up correctly in the VMCB.
Additionally, free any existing non-SEV ASID on the destination vCPU
before overwriting it, to avoid leaking ASIDs when creating vCPUs then
migrating an SEV VM onto them.

Note: A nice side-effect is reading the min/max ASIDs once during
initialization, instead of once per-CPU.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c |  4 +-
 arch/x86/kvm/svm/sev.c    |  7 +++-
 arch/x86/kvm/svm/svm.c    | 79 +++++++++++++++------------------------
 arch/x86/kvm/svm/svm.h    | 25 +++++++++----
 4 files changed, 55 insertions(+), 60 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index c3c12a4d6fe07..5d63ae47ebb6c 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -936,9 +936,9 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	else
 		vmcb02->control.bus_lock_counter = 0;
 
-	/* Done at vmrun: asid.  */
+	vmcb02->control.asid = vmcb01->control.asid;
 
-	/* Also overwritten later if necessary.  */
+	/* Overwritten later if necessary.  */
 	vmcb_clr_flush_asid(vmcb02);
 
 	/* Use vmcb01 MMU and format if guest does not use nNPT */
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 0fbb72d57e6da..99577859e1aab 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2035,6 +2035,11 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
 static void sev_vcpu_migrate_asid(struct vcpu_svm *dst_svm,
 				  unsigned int asid)
 {
+	/*
+	 * Free the (potentially non-SEV) ASID on the destination vCPU before
+	 * setting the new SEV ASID.
+	 */
+	free_asid(dst_svm->asid);
 	dst_svm->asid = asid;
 }
 
@@ -4888,7 +4893,7 @@ void sev_init_vmcb(struct vcpu_svm *svm, bool init_event)
 	svm->vmcb->control.misc_ctl |= SVM_MISC_ENABLE_SEV;
 	clr_exception_intercept(svm, UD_VECTOR);
 
-	svm->asid = sev_get_asid(vcpu->kvm);
+	WARN_ON_ONCE(svm->asid != sev_get_asid(vcpu->kvm));
 	svm->vmcb->control.asid = svm->asid;
 	vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
 
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 0c3befd9bc850..8f27722bc4faf 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -192,6 +192,8 @@ DEFINE_PER_CPU(struct svm_cpu_data, svm_data);
 
 static DEFINE_MUTEX(vmcb_dump_mutex);
 
+kvm_tlb_tag_t fallback_asid;
+
 /*
  * Only MSR_TSC_AUX is switched via the user return hook.  EFER is switched via
  * the VMCB, and the SYSCALL/SYSENTER MSRs are handled by VMLOAD/VMSAVE.
@@ -583,15 +585,6 @@ static int svm_enable_virtualization_cpu(void)
 		return r;
 
 	sd = per_cpu_ptr(&svm_data, me);
-	/*
-	 * Bump the current asid_generation value to ensure any vCPU that
-	 * previously ran on this CPU sees a stale generation and is forced
-	 * to acquire a new ASID, preventing a latent ASID collision.
-	 */
-	sd->asid_generation++;
-	sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
-	sd->next_asid = sd->max_asid + 1;
-	sd->min_asid = max_sev_asid + 1;
 
 	wrmsrq(MSR_VM_HSAVE_PA, sd->save_area_pa);
 
@@ -994,6 +987,8 @@ static void svm_hardware_unsetup(void)
 
 	__free_pages(__sme_pa_to_page(iopm_base), get_order(IOPM_SIZE));
 	iopm_base = 0;
+
+	kvm_destroy_tlb_tags();
 }
 
 static void init_seg(struct vmcb_seg *seg)
@@ -1246,8 +1241,8 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event)
 	if (gmet_enabled)
 		control->misc_ctl |= SVM_MISC_ENABLE_GMET;
 
-	svm->current_vmcb->asid_generation = 0;
-	svm->asid = 0;
+	control->asid = svm->asid;
+	vmcb_set_flush_asid(vmcb);
 
 	svm->nested.vmcb12_gpa = INVALID_GPA;
 	svm->nested.last_vmcb12_gpa = INVALID_GPA;
@@ -1375,6 +1370,12 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
 		goto error_free_avic;
 	}
 
+	svm->asid = allocate_asid(vcpu);
+	if (!svm->asid) {
+		err = -EINVAL;
+		goto error_free_msrpm;
+	}
+
 	svm->x2avic_msrs_intercepted = true;
 	svm->lbr_msrs_intercepted = true;
 
@@ -1386,6 +1387,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
 
 	return 0;
 
+error_free_msrpm:
+	svm_vcpu_free_msrpm(svm->msrpm);
 error_free_avic:
 	avic_vcpu_free(vcpu);
 error_free_sev:
@@ -1417,6 +1420,8 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu)
 
 	__free_page(__sme_pa_to_page(svm->vmcb01.pa));
 	svm_vcpu_free_msrpm(svm->msrpm);
+
+	free_asid(svm->asid);
 }
 
 #ifdef CONFIG_CPU_MITIGATIONS
@@ -1942,18 +1947,6 @@ static void svm_update_exception_bitmap(struct kvm_vcpu *vcpu)
 	}
 }
 
-static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd)
-{
-	if (sd->next_asid > sd->max_asid) {
-		++sd->asid_generation;
-		sd->next_asid = sd->min_asid;
-		sd->flush_all_asids = true;
-	}
-
-	svm->current_vmcb->asid_generation = sd->asid_generation;
-	svm->asid = sd->next_asid++;
-}
-
 static void svm_set_dr6(struct kvm_vcpu *vcpu, unsigned long value)
 {
 	struct vmcb *vmcb = to_svm(vcpu)->vmcb;
@@ -3875,37 +3868,24 @@ static void svm_set_nested_run_soft_int_state(struct kvm_vcpu *vcpu)
 
 static int pre_svm_run(struct kvm_vcpu *vcpu)
 {
-	struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
 	struct vcpu_svm *svm = to_svm(vcpu);
 
 	/*
 	 * If the previous VMRUN of the VMCB occurred on a different physical
-	 * cpu, then mark the VMCB dirty as hardware's clean bits are per pCPU.
-	 *
-	 * Reset the ASID generation in both VMCBs. This will lead to assigning
-	 * a new ASID now, and then again when switching to the other VMCB.
-	 * However, this is needed as the ASID is shared between the VMCBs, and
-	 * otherwise it would be possible to use an ASID allocated on one pCPU
-	 * on another, for example:
-	 * - vCPU migrates from pCPU A to pCPU B, allocates a new ASID.
-	 * - vCPU migrates back to pCPU A, and then switches the VMCB.
-	 * - The new VMCB does not detect a pCPU change and runs on pCPU A with
-	 *   the new ASID allocated on pCPU B, which is potentially used by
-	 *   another vCPU/VM.
+	 * cpu, then mark the VMCB dirty and flush the ASID, as hardware's clean
+	 * bits and TLB entries are per pCPU.
 	 */
 	if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) {
 		vmcb_mark_all_dirty(svm->vmcb);
 		svm->current_vmcb->cpu = vcpu->cpu;
-		svm->vmcb01.asid_generation = 0;
-		svm->nested.vmcb02.asid_generation = 0;
+		vmcb_set_flush_asid(svm->vmcb);
         }
 
 	if (is_sev_guest(vcpu))
 		return pre_sev_run(svm, vcpu->cpu);
 
-	/* FIXME: handle wraparound of asid_generation */
-	if (svm->current_vmcb->asid_generation != sd->asid_generation)
-		new_asid(svm, sd);
+	if (unlikely(svm->vmcb->control.asid == fallback_asid))
+		vmcb_set_flush_asid(svm->vmcb);
 
 	return 0;
 }
@@ -4644,13 +4624,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
 	sync_lapic_to_cr8(vcpu);
 
-	if (unlikely(svm->asid != svm->vmcb->control.asid)) {
-		svm->vmcb->control.asid = svm->asid;
-		vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
-	}
-	if (this_cpu_ptr(&svm_data)->flush_all_asids)
-		svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
-
 	svm->vmcb->save.cr2 = vcpu->arch.cr2;
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) &&
@@ -4742,7 +4715,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 	}
 
 	if (!svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
-		this_cpu_ptr(&svm_data)->flush_all_asids = false;
 		vmcb_clr_flush_asid(svm->vmcb);
 
 		/*
@@ -5754,6 +5726,7 @@ static __init void svm_set_cpu_caps(void)
 
 static __init int svm_hardware_setup(void)
 {
+	unsigned long nr_asids;
 	void *iopm_va;
 	int cpu, r;
 
@@ -5911,6 +5884,14 @@ static __init int svm_hardware_setup(void)
 
 	kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED;
 
+	/* Consumes max_sev_asid initialized by sev_hardware_setup() */
+	nr_asids = cpuid_ebx(SVM_CPUID_FUNC);
+	r = kvm_init_tlb_tags(nr_asids, max_sev_asid + 1);
+	if (r)
+		goto err;
+
+	fallback_asid = kvm_alloc_tlb_tag();
+
 	for_each_possible_cpu(cpu) {
 		r = svm_cpu_init(cpu);
 		if (r)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 14b57c5cdf916..53c5ed2d93d91 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -26,6 +26,7 @@
 #include "regs.h"
 #include "x86.h"
 #include "pmu.h"
+#include "mmu.h"
 
 /*
  * Helpers to convert to/from physical addresses for pages whose address is
@@ -145,7 +146,6 @@ struct kvm_vmcb_info {
 	struct vmcb *ptr;
 	unsigned long pa;
 	int cpu;
-	uint64_t asid_generation;
 };
 
 struct vmcb_save_area_cached {
@@ -285,7 +285,7 @@ struct vcpu_svm {
 	struct vmcb *vmcb;
 	struct kvm_vmcb_info vmcb01;
 	struct kvm_vmcb_info *current_vmcb;
-	u32 asid;
+	kvm_tlb_tag_t asid;
 	u32 sysenter_esp_hi;
 	u32 sysenter_eip_hi;
 	uint64_t tsc_aux;
@@ -372,12 +372,6 @@ struct vcpu_svm {
 };
 
 struct svm_cpu_data {
-	u64 asid_generation;
-	u32 max_asid;
-	u32 next_asid;
-	u32 min_asid;
-
-	bool flush_all_asids;
 	bool bp_spec_reduce_set;
 
 	struct vmcb *save_area;
@@ -1080,6 +1074,21 @@ static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)
 static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa) {}
 #endif
 
+extern kvm_tlb_tag_t fallback_asid;
+
+static inline kvm_tlb_tag_t allocate_asid(struct kvm_vcpu *vcpu)
+{
+	if (is_sev_guest(vcpu))
+		return sev_get_asid(vcpu->kvm);
+	return kvm_alloc_tlb_tag() ?: fallback_asid;
+}
+
+static inline void free_asid(kvm_tlb_tag_t asid)
+{
+	if (asid > max_sev_asid && asid != fallback_asid)
+		kvm_free_tlb_tag(asid);
+}
+
 /* vmenter.S */
 
 void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, unsigned int flags,
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 11/29] KVM: SVM: Only flush the fallback ASID when used by a different vCPU

[Yosry Ahmed]

Instead of always flushing the fallback ASID on every vCPU run, only
flush it if the last vCPU that used it on the pCPU changes. This avoids
constant TLB flushing on a vCPU using the fallback ASID if no other
vCPUs that use the same ASID ran on the same pCPU.

Note that checking (and flushing) the current ASID on vCPU load is not
sufficient, as it is possible that a vCPU is using the fallback ASID
only for L1 or only for L2. To do this correctly on vCPU load, both L1
and L2 ASIDs would need to be checked, and the respective VMCB would
need to be flushed. This would not be much simpler, and would perform
unnecessary flushes in some cases (e.g.  loading a vCPU that uses the
fallback ASID for L2, but L2 doesn't actually run until the vCPU is
unloaded).

Another alternative is always flushing on vCPU load if the current ASID
is the fallback ASID, and flushing on nested transitions if the new ASID
is the fallback ASID. Again, this is not much simpler and also incurs
unnecessary flushes in some cases.

Do the logical thing and track the vCPU using the fallback ASID instead.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/svm.c | 13 ++++++++++++-
 arch/x86/kvm/svm/svm.h |  3 +++
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 8f27722bc4faf..0dbb1962d32b6 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1405,6 +1405,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
 static void svm_vcpu_free(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
+	struct svm_cpu_data *sd;
+	int cpu;
 
 	WARN_ON_ONCE(!list_empty(&svm->ir_list));
 
@@ -1422,6 +1424,11 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu)
 	svm_vcpu_free_msrpm(svm->msrpm);
 
 	free_asid(svm->asid);
+
+	for_each_possible_cpu(cpu) {
+		sd = per_cpu_ptr(&svm_data, cpu);
+		cmpxchg(&sd->fallback_asid_vcpu, vcpu, NULL);
+	}
 }
 
 #ifdef CONFIG_CPU_MITIGATIONS
@@ -3868,6 +3875,7 @@ static void svm_set_nested_run_soft_int_state(struct kvm_vcpu *vcpu)
 
 static int pre_svm_run(struct kvm_vcpu *vcpu)
 {
+	struct svm_cpu_data *sd = this_cpu_ptr(&svm_data);
 	struct vcpu_svm *svm = to_svm(vcpu);
 
 	/*
@@ -3884,8 +3892,11 @@ static int pre_svm_run(struct kvm_vcpu *vcpu)
 	if (is_sev_guest(vcpu))
 		return pre_sev_run(svm, vcpu->cpu);
 
-	if (unlikely(svm->vmcb->control.asid == fallback_asid))
+	if (unlikely(svm->vmcb->control.asid == fallback_asid &&
+		     sd->fallback_asid_vcpu != vcpu)) {
 		vmcb_set_flush_asid(svm->vmcb);
+		sd->fallback_asid_vcpu = vcpu;
+	}
 
 	return 0;
 }
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 53c5ed2d93d91..d386f2a4f72d7 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -377,6 +377,9 @@ struct svm_cpu_data {
 	struct vmcb *save_area;
 	unsigned long save_area_pa;
 
+	/* Last vCPU to use fallback_asid on this CPU */
+	struct kvm_vcpu *fallback_asid_vcpu;
+
 	/* index = sev_asid, value = vmcb pointer */
 	struct vmcb **sev_vmcbs;
 };
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 12/29] KVM: nSVM: Drop svm->nested.initialized

[Yosry Ahmed]

Do NULL checks on svm->nested.vmcb02.ptr instead of using a dedicated
proxy 'initialized' field.  Make sure svm->nested.vmcb02.ptr is
correctly set to NULL on initialization failures, as it is currently
left as a dangling pointer.

This makes it possible to directly do a NULL check on
svm->nested.vmcb02.ptr before using it, instead of checking a different
field, which is less error-prone.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 14 ++++++--------
 arch/x86/kvm/svm/svm.h    |  2 --
 2 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 5d63ae47ebb6c..bfb297b7d71ab 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1142,7 +1142,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
 		return ret;
 	}
 
-	if (WARN_ON_ONCE(!svm->nested.initialized))
+	if (WARN_ON_ONCE(!svm->nested.vmcb02.ptr))
 		return -EINVAL;
 
 	vmcb12_gpa = kvm_rax_read(vcpu);
@@ -1489,21 +1489,21 @@ int svm_allocate_nested(struct vcpu_svm *svm)
 {
 	struct page *vmcb02_page;
 
-	if (svm->nested.initialized)
+	if (svm->nested.vmcb02.ptr)
 		return 0;
 
 	vmcb02_page = snp_safe_alloc_page();
 	if (!vmcb02_page)
 		return -ENOMEM;
-	svm->nested.vmcb02.ptr = page_address(vmcb02_page);
-	svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
 
 	svm->nested.msrpm = svm_vcpu_alloc_msrpm();
 	if (!svm->nested.msrpm)
 		goto err_free_vmcb02;
 
 	svm->nested.vmcb02.cpu = -1;
-	svm->nested.initialized = true;
+	svm->nested.vmcb02.ptr = page_address(vmcb02_page);
+	svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
+
 	return 0;
 
 err_free_vmcb02:
@@ -1513,7 +1513,7 @@ int svm_allocate_nested(struct vcpu_svm *svm)
 
 void svm_free_nested(struct vcpu_svm *svm)
 {
-	if (!svm->nested.initialized)
+	if (!svm->nested.vmcb02.ptr)
 		return;
 
 	if (WARN_ON_ONCE(svm->vmcb != svm->vmcb01.ptr))
@@ -1533,8 +1533,6 @@ void svm_free_nested(struct vcpu_svm *svm)
 	 * When the vmcb02 is freed, this optimization becomes invalid.
 	 */
 	svm->nested.last_vmcb12_gpa = INVALID_GPA;
-
-	svm->nested.initialized = false;
 }
 
 void svm_leave_nested(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index d386f2a4f72d7..2e1b5481ddd92 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -234,8 +234,6 @@ struct svm_nested_state {
 	 */
 	struct vmcb_save_area_cached save;
 
-	bool initialized;
-
 	/*
 	 * Indicates whether MSR bitmap for L2 needs to be rebuilt due to
 	 * changes in MSR bitmap for L1 or switching to a different L2. Note,
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 13/29] KVM: nSVM: Add a placeholder ASID for L2

[Yosry Ahmed]

In preparation for introducing a separate ASID for L2, introduce a
'placeholder' ASID that is still the same as L1's ASID. This will
facilitate future changes that need to distinguish L1 and L2's ASIDs,
before actually using a different ASID for L1 and L2.

On SEV VM migration, copy the new SEV ASID into asid02 and vmcb02, in
case nested was already initialized on the destination vCPU before
migration.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 5 +++--
 arch/x86/kvm/svm/sev.c    | 9 +++++++++
 arch/x86/kvm/svm/svm.h    | 2 ++
 3 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index bfb297b7d71ab..780132725bb79 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -701,7 +701,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
 	 *  - Honor L1's request to flush an ASID on nested VMRUN
 	 *  - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*]
 	 *  - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
-	 *  - Flush L1's ASID on KVM_REQ_TLB_FLUSH_GUEST
 	 *
 	 * [*] Unlike nested EPT, SVM's ASID management can invalidate nested
 	 *     NPT guest-physical mappings on VMRUN.
@@ -936,7 +935,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	else
 		vmcb02->control.bus_lock_counter = 0;
 
-	vmcb02->control.asid = vmcb01->control.asid;
+	vmcb02->control.asid = svm->nested.asid02;
 
 	/* Overwritten later if necessary.  */
 	vmcb_clr_flush_asid(vmcb02);
@@ -1504,6 +1503,8 @@ int svm_allocate_nested(struct vcpu_svm *svm)
 	svm->nested.vmcb02.ptr = page_address(vmcb02_page);
 	svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
 
+	svm->nested.asid02 = svm->asid;
+
 	return 0;
 
 err_free_vmcb02:
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 99577859e1aab..b2e6acfbb081f 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2041,6 +2041,15 @@ static void sev_vcpu_migrate_asid(struct vcpu_svm *dst_svm,
 	 */
 	free_asid(dst_svm->asid);
 	dst_svm->asid = asid;
+
+	/*
+	 * If nested is already initialized on the destination vCPU, also update
+	 * the nested ASID to match the new SEV ASID.
+	 */
+	if (dst_svm->nested.vmcb02.ptr) {
+		dst_svm->nested.asid02 = asid;
+		dst_svm->nested.vmcb02.ptr->control.asid = asid;
+	}
 }
 
 static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 2e1b5481ddd92..ed67c18e2b900 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -234,6 +234,8 @@ struct svm_nested_state {
 	 */
 	struct vmcb_save_area_cached save;
 
+	kvm_tlb_tag_t asid02;
+
 	/*
 	 * Indicates whether MSR bitmap for L2 needs to be rebuilt due to
 	 * changes in MSR bitmap for L1 or switching to a different L2. Note,
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 14/29] KVM: x86: hyper-v: Rename kvm_hv_vcpu_purge_flush_tlb()

[Yosry Ahmed]

Rename kvm_hv_vcpu_purge_flush_tlb() to kvm_hv_purge_tlb_flush_fifo() to
clarify that it purges the TLB flush FIFO, not purge *and* flush the
TLB.

No functional change intended.

Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/hyperv.h  | 4 ++--
 arch/x86/kvm/svm/svm.c | 2 +-
 arch/x86/kvm/x86.c     | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 622a6553e9ac2..03cc65f8cc52d 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -312,7 +312,7 @@ static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(struc
 	return &hv_vcpu->tlb_flush_fifo[i];
 }
 
-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu)
+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo;
 
@@ -396,7 +396,7 @@ static inline int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 {
 	return HV_STATUS_ACCESS_DENIED;
 }
-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu) {}
+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu) {}
 static inline bool kvm_hv_synic_has_vector(struct kvm_vcpu *vcpu, int vector)
 {
 	return false;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 0dbb1962d32b6..51b3ea3d5acfe 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4308,7 +4308,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
 	 * A TLB flush for the current ASID flushes both "host" and "guest" TLB
 	 * entries, and thus is a superset of Hyper-V's fine grained flushing.
 	 */
-	kvm_hv_vcpu_purge_flush_tlb(vcpu);
+	kvm_hv_purge_tlb_flush_fifo(vcpu);
 
 	/*
 	 * Flush only the current ASID even if the TLB flush was invoked via
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32563a91fba99..12eaeaea51633 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2030,7 +2030,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
 	 * Flushing all "guest" TLB is always a superset of Hyper-V's fine
 	 * grained flushing.
 	 */
-	kvm_hv_vcpu_purge_flush_tlb(vcpu);
+	kvm_hv_purge_tlb_flush_fifo(vcpu);
 }
 
 
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 15/29] KVM: x86: hyper-v: Allow purging all TLB flush FIFOs

[Yosry Ahmed]

Refactor kvm_hv_purge_tlb_flush_fifo() to introduce an inner helper,
__kvm_hv_purge_tlb_flush_fifo(), parameterized by an optional FIFO. If a
FIFO is not passed in, all FIFOs are purged.

Note that KVM_REQ_HV_TLB_FLUSH is consumed by
__kvm_hv_purge_tlb_flush_fifo(), so a selective purge on one FIFO will
render any subsequent non-selective purge useless. This is not a problem
because purging TLB flush FIFOs is an optimization (and there are
currently no users of the non-selective purges).

No functional change intended as a FIFO is always being passed in now.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/hyperv.h | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 03cc65f8cc52d..a23ef05c1075c 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -312,18 +312,29 @@ static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(struc
 	return &hv_vcpu->tlb_flush_fifo[i];
 }
 
-static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)
+/* Purge pending TLB flushes in @fifo, or in all FIFOs if @fifo is NULL */
+static inline void __kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu,
+						 struct kvm_vcpu_hv_tlb_flush_fifo *fifo)
 {
-	struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo;
+	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+	int i;
 
-	if (!kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))
+	if (!hv_vcpu || !kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))
 		return;
 
-	tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));
-	if (!tlb_flush_fifo)
-		return;
+	for (i = 0; i < ARRAY_SIZE(hv_vcpu->tlb_flush_fifo); i++) {
+		if (!fifo || fifo == &hv_vcpu->tlb_flush_fifo[i])
+			kfifo_reset_out(&hv_vcpu->tlb_flush_fifo[i].entries);
+	}
+}
+
+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_hv_tlb_flush_fifo *fifo;
 
-	kfifo_reset_out(&tlb_flush_fifo->entries);
+	fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));
+	if (fifo)
+		__kvm_hv_purge_tlb_flush_fifo(vcpu, fifo);
 }
 
 static inline bool guest_hv_cpuid_has_l2_tlb_flush(struct kvm_vcpu *vcpu)
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 16/29] KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH

[Yosry Ahmed]

Flush both L1 and L2 ASIDs in svm_flush_tlb_all() to appropriately
handle KVM_REQ_TLB_FLUSH by flushing all TLB entries in all contexts
(e.g. for kvm_flush_remote_tlbs()). Since both L1 and L2 currently share
an ASID, this is effectively a noop, but it won't be once L2 has a
separate ASID.

Purge all Hyper-V TLB FIFOs (for both L1 and L2), since both ASIDs are
flushed, and an ASID flush is a superset of Hyper-V's fine-grained
flushing (see comment in svm_flush_tlb_asid()).

Note that if one TLB flush FIFO is purged (e.g. as a result of
KVM_REQ_TLB_FLUSH_CURRENT), it will consume KVM_REQ_HV_TLB_FLUSH, and a
subsequent KVM_REQ_TLB_FLUSH will not flush the other FIFO. This is
alright as flushing both FIFOs is a (newly introduced) optimization
anyway. The other FIFO will be checked after a nested transition, as
KVM_REQ_HV_TLB_FLUSH is always set on nested transitions.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/hyperv.h     | 6 ++++++
 arch/x86/kvm/svm/nested.c | 1 -
 arch/x86/kvm/svm/svm.c    | 8 +++++++-
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index a23ef05c1075c..b82f70e6fcea1 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -337,6 +337,11 @@ static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)
 		__kvm_hv_purge_tlb_flush_fifo(vcpu, fifo);
 }
 
+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu)
+{
+	__kvm_hv_purge_tlb_flush_fifo(vcpu, NULL);
+}
+
 static inline bool guest_hv_cpuid_has_l2_tlb_flush(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
@@ -408,6 +413,7 @@ static inline int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 	return HV_STATUS_ACCESS_DENIED;
 }
 static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu) {}
+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu) {}
 static inline bool kvm_hv_synic_has_vector(struct kvm_vcpu *vcpu, int vector)
 {
 	return false;
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 780132725bb79..75ada771026b2 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -697,7 +697,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
 	 * TODO: optimize unconditional TLB flush/MMU sync.  A partial list of
 	 * things to fix before this can be conditional:
 	 *
-	 *  - Flush TLBs for both L1 and L2 remote TLB flush
 	 *  - Honor L1's request to flush an ASID on nested VMRUN
 	 *  - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*]
 	 *  - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 51b3ea3d5acfe..a6d14937e7777 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4338,6 +4338,8 @@ static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)
 
 static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)
 {
+	struct vcpu_svm *svm = to_svm(vcpu);
+
 	/*
 	 * When running on Hyper-V with EnlightenedNptTlb enabled, remote TLB
 	 * flushes should be routed to hv_flush_remote_tlbs() without requesting
@@ -4348,7 +4350,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)
 	if (WARN_ON_ONCE(svm_hv_is_enlightened_tlb_enabled(vcpu)))
 		hv_flush_remote_tlbs(vcpu->kvm);
 
-	svm_flush_tlb_asid(vcpu);
+	kvm_hv_purge_all_tlb_flush_fifos(vcpu);
+
+	vmcb_set_flush_asid(svm->vmcb01.ptr);
+	if (svm->nested.vmcb02.ptr)
+		vmcb_set_flush_asid(svm->nested.vmcb02.ptr);
 }
 
 static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu)
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 17/29] KVM: nSVM: Always switch VMCB before leaving guest mode

[Yosry Ahmed]

Move svm_switch_vmcb() calls ahead of leave_guest_mode(), as nothing
between the calls depends on svm->vmcb being set to vmcb01. This makes
the nested VM-Exit path consistent with the nested VM-Enter path, and
having the calls to svm_switch_vmcb() and leave_guest_mode() close makes
it easier to reason about incoming changes that depend on both VMCB and
guest_mode context.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 75ada771026b2..27f78228a42d5 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1329,7 +1329,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	if (nested_svm_vmexit_update_vmcb12(vcpu))
 		kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
 
-	/* Exit Guest-Mode */
+	svm_switch_vmcb(svm, &svm->vmcb01);
 	leave_guest_mode(vcpu);
 	svm_pmu_handle_nested_transition(svm);
 
@@ -1356,8 +1356,6 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS))
 		vmcb01->control.erap_ctl |= ERAP_CONTROL_CLEAR_RAP;
 
-	svm_switch_vmcb(svm, &svm->vmcb01);
-
 	/*
 	 * Rules for synchronizing int_ctl bits from vmcb02 to vmcb01:
 	 *
@@ -1543,6 +1541,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
 		vcpu->arch.nested_run_pending = 0;
 		svm->nested.vmcb12_gpa = INVALID_GPA;
 
+		svm_switch_vmcb(svm, &svm->vmcb01);
 		leave_guest_mode(vcpu);
 
 		/*
@@ -1554,8 +1553,6 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
 		 */
 		__svm_pmu_handle_nested_transition(svm, true);
 
-		svm_switch_vmcb(svm, &svm->vmcb01);
-
 		nested_svm_transition_tlb_flush(vcpu);
 
 		nested_svm_uninit_mmu_context(vcpu);
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 18/29] KVM: nSVM: Split nested_svm_transition_tlb_flush() into entry/exit fns

[Yosry Ahmed]

The handling for the entry and exit TLB flushes will diverge
significantly in the following changes. Instead of adding an 'is_vmenter'
argument like nested_vmx_transition_tlb_flush(), just split the function
into two variants for 'entry' and 'exit'.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 27f78228a42d5..efe7a39b53656 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -688,7 +688,7 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,
 	vmcb12->control.exit_int_info = exit_int_info;
 }
 
-static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
+static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 {
 	/* Handle pending Hyper-V TLB flush requests */
 	kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
@@ -708,6 +708,14 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
 }
 
+static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)
+{
+	kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
+
+	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
+	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+}
+
 /*
  * Load guest's/host's cr3 on nested vmentry or vmexit. @nested_npt is true
  * if we are emulating VM-Entry into a guest with NPT enabled.
@@ -858,7 +866,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	struct vmcb *vmcb01 = svm->vmcb01.ptr;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 
-	nested_svm_transition_tlb_flush(vcpu);
+	nested_svm_entry_tlb_flush(vcpu);
 
 	/* Enter Guest-Mode */
 	enter_guest_mode(vcpu);
@@ -1434,7 +1442,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	svm->vcpu.arch.dr7 = DR7_FIXED_1;
 	kvm_update_dr7(&svm->vcpu);
 
-	nested_svm_transition_tlb_flush(vcpu);
+	nested_svm_exit_tlb_flush(vcpu);
 
 	nested_svm_uninit_mmu_context(vcpu);
 
@@ -1553,7 +1561,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
 		 */
 		__svm_pmu_handle_nested_transition(svm, true);
 
-		nested_svm_transition_tlb_flush(vcpu);
+		nested_svm_exit_tlb_flush(vcpu);
 
 		nested_svm_uninit_mmu_context(vcpu);
 		vmcb_mark_all_dirty(svm->vmcb);
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 19/29] KVM: nSVM: Service local TLB flushes before nested transitions

[Yosry Ahmed]

Service pending TLB flushes for a specific context (L1 vs. L2) before
entering/leaving guest mode, as KVM does not track TLB flush requests
for L1 vs. L2. This is not currently required as KVM always flushes the
TLB on nested transitions, but it is a requirement to drop the
unconditional flushes.

TLB flushes are performed through the VMCB's TLB_CONTROL field, so
service the flushes before every VMCB switch (except the precautionary
switch in svm_free_nested()). Servicing flushes may end up purging the
relevant HV TLB flush FIFO, which is based on guest_mode, so make sure
local TLB flushes are handled before calling {enter/leave}_guest_mode().

This is conceptually similar to how nVMX calls
kvm_service_local_tlb_flush_requests(), except that the VMX calls only
have ordering requirements on {enter/leave}_guest_mode(), and not
switching the VMCS, because VPID used for TLB flushes depends on
guest_mode, and flushes are not VMCS-based.

Note that nested_svm_{entry/exit}_tlb_flush() must be called after
kvm_service_local_tlb_flush_requests(), otherwise the TLB flushes will
be immediately applied on the outgoing VMCB rather than the incoming
one.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index efe7a39b53656..e216818f68372 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1044,6 +1044,20 @@ static void nested_svm_copy_common_state(struct vmcb *from_vmcb, struct vmcb *to
 	to_vmcb->save.spec_ctrl = from_vmcb->save.spec_ctrl;
 }
 
+static void nested_svm_service_local_tlb_flushes(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+
+	/*
+	 * TLB flushes are applied to the VMCB, so apply any pending TLB flushes
+	 * on the outgoing VMCB before switching to a new one. A TLB flush could
+	 * purge the relevant HV TLB flush FIFO depending on guest_mode, so make
+	 * sure the VMCB and guest_mode context is consistent.
+	 */
+	WARN_ON_ONCE(is_guest_mode(vcpu) != (svm->vmcb == svm->nested.vmcb02.ptr));
+	kvm_service_local_tlb_flush_requests(vcpu);
+}
+
 int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -1075,6 +1089,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
 
 	nested_svm_copy_common_state(svm->vmcb01.ptr, svm->nested.vmcb02.ptr);
 
+	nested_svm_service_local_tlb_flushes(vcpu);
+
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
 	nested_vmcb02_prepare_control(svm);
 	nested_vmcb02_prepare_save(svm);
@@ -1337,6 +1353,8 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
 	if (nested_svm_vmexit_update_vmcb12(vcpu))
 		kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
 
+	nested_svm_service_local_tlb_flushes(vcpu);
+
 	svm_switch_vmcb(svm, &svm->vmcb01);
 	leave_guest_mode(vcpu);
 	svm_pmu_handle_nested_transition(svm);
@@ -1549,6 +1567,8 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
 		vcpu->arch.nested_run_pending = 0;
 		svm->nested.vmcb12_gpa = INVALID_GPA;
 
+		nested_svm_service_local_tlb_flushes(vcpu);
+
 		svm_switch_vmcb(svm, &svm->vmcb01);
 		leave_guest_mode(vcpu);
 
@@ -2089,6 +2109,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
 	svm_copy_vmrun_state(vcpu, &svm->vmcb01.ptr->save, save);
 	nested_copy_vmcb_control_to_cache(svm, ctl);
 
+	nested_svm_service_local_tlb_flushes(vcpu);
+
 	svm_switch_vmcb(svm, &svm->nested.vmcb02);
 
 	if (use_separate_l2_pat)
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 20/29] KVM: x86: Add KVM_REQ_MMU_SYNC_ALL_ROOTS

[Yosry Ahmed]

Add a new x86 request to sync all roots, current and previous. This will
be used when syncing the shadow NPTs on nested VMRUN when an ASID flush
(or new ASID) is requested by L1, as all physical translations in all
roots should be sync'd (since shadow mappings are not tagged with an
ASID).

Refactor syncing all roots (currently only done by
kvm_vcpu_flush_tlb_guest()) to a helper, and use it to handle the
request.

Clear KVM_REQ_MMU_SYNC when handling KVM_REQ_MMU_SYNC_ALL_ROOTS as the
latter is a superset. Do not do so inside kvm_mmu_sync_all_roots(), to
avoid any subtle behavioral changes from kvm_vcpu_flush_tlb_guest()
clearing KVM_REQ_MMU_SYNC. It makes sense to always clear it when
syncing the MMU, but there might be weird dependencies on the sync only
happening before the actual vCPU run, and kvm_vcpu_flush_tlb_guest() is
called in other code paths than handling KVM_REQ_TLB_FLUSH_GUEST before
vCPU run.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/mmu.h              |  7 +++++++
 arch/x86/kvm/x86.c              | 11 ++++++++---
 3 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index e2eef7057bd35..1d28bb7e92c7f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -127,6 +127,7 @@
 	KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \
 	KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
+#define KVM_REQ_MMU_SYNC_ALL_ROOTS	KVM_ARCH_REQ(35)
 
 #define INVALID_PAGE (~(hpa_t)0)
 #define VALID_PAGE(x) ((x) != INVALID_PAGE)
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index b443a5083bfb0..b6ad0ff72393e 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -170,6 +170,13 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu);
 void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu);
 void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);
 void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu);
+
+static inline void kvm_mmu_sync_all_roots(struct kvm_vcpu *vcpu)
+{
+	kvm_mmu_sync_roots(vcpu);
+	kvm_mmu_sync_prev_roots(vcpu);
+}
+
 void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
 			 int bytes);
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 12eaeaea51633..a54e9da2ed09e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2020,8 +2020,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
 		 * a forced sync of the shadow page tables.  Ensure all the
 		 * roots are synced and the guest TLB in hardware is clean.
 		 */
-		kvm_mmu_sync_roots(vcpu);
-		kvm_mmu_sync_prev_roots(vcpu);
+		kvm_mmu_sync_all_roots(vcpu);
 	}
 
 	kvm_x86_call(flush_tlb_guest)(vcpu);
@@ -8167,8 +8166,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 			if (unlikely(r))
 				goto out;
 		}
-		if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu))
+
+		if (kvm_check_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu)) {
+			kvm_mmu_sync_all_roots(vcpu);
+			kvm_clear_request(KVM_REQ_MMU_SYNC, vcpu);
+		} else if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu)) {
 			kvm_mmu_sync_roots(vcpu);
+		}
+
 		if (kvm_check_request(KVM_REQ_LOAD_MMU_PGD, vcpu))
 			kvm_mmu_load_pgd(vcpu);
 
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 21/29] KVM: nSVM: Handle nested TLB flush requests through TLB_CONTROL

[Yosry Ahmed]

Handle L1's requests to flush L2's TLB through the TLB_CONTROL field of
VMCB12.

On nested VM-Enter, flush L2's ASID if any flush is specified in
TLB_CONTROL. This handles TLB_CONTROL_FLUSH_ASID and
TLB_CONTROL_FLUSH_ASID_LOCAL equally for simplicity.

On nested VM-Exit, flush L1's own ASID if L1 requested a *full* TLB
flush (i.e. TLB_CONTROL_FLUSH_ALL_ASID).

Essentially, TLB_CONTROL_FLUSH_ASID[_LOCAL] cause a TLB flush on nested
VM-Enter only, while TLB_CONTROL_FLUSH_ALL_ASID causes a TLB flush on
both nested VM-Enter and nested VM-Exit.

Additionally, sync the nested NPTs on nested VM-Enter on an ASID flush,
as ASID flushes also invalidate guest-physical translations on SVM
(unlike VMX, which has separate VPID-based and EPT-based flushing). Sync
all roots, as shadow mappings are not tagged with an ASID, and there
could be translations belonging to the outgoing or flushed ASID in
previous roots.

All TLB_CONTROL values can be handled by KVM regardless of FLUSHBYASID
support on the underlying CPU, so keep advertising FLUSHBYASID to the
guest unconditionally.

Note, TLB_CONTROL_FLUSH_ALL_ASID is never propagated from the vmcb12 to
the vmcb02 (unless FLUSHBYASID is not available), as this gives the
guest the power to flush the entire physical TLB (including translations
for the host and other VMs).

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 31 ++++++++++++++++++++++++++-----
 arch/x86/kvm/svm/svm.c    |  5 ++---
 2 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index e216818f68372..e04cc4f7669ea 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -690,19 +690,34 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,
 
 static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 {
+	struct vcpu_svm *svm = to_svm(vcpu);
+
 	/* Handle pending Hyper-V TLB flush requests */
 	kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
 
+	/*
+	 * If L1 requested a TLB flush for L2, flush L2's TLB on nested entry
+	 * and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT
+	 * guest-physical mappings. Sync all roots, since the TLB flush is not
+	 * specific to the currently loaded nCR3.
+	 *
+	 * If L1 requested a full TLB flush for all ASIDs (including its own),
+	 * L1's own ASID is also flushed on nested VM-Exit, before running L1.
+	 *
+	 * TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL are handled
+	 * equally for simplicity.
+	 */
+	if (svm->nested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING) {
+		if (nested_npt_enabled(svm))
+			kvm_make_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu);
+		kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+	}
+
 	/*
 	 * TODO: optimize unconditional TLB flush/MMU sync.  A partial list of
 	 * things to fix before this can be conditional:
 	 *
-	 *  - Honor L1's request to flush an ASID on nested VMRUN
-	 *  - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*]
 	 *  - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
-	 *
-	 * [*] Unlike nested EPT, SVM's ASID management can invalidate nested
-	 *     NPT guest-physical mappings on VMRUN.
 	 */
 	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
@@ -710,8 +725,14 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 
 static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)
 {
+	struct vcpu_svm *svm = to_svm(vcpu);
+
 	kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
 
+	/* Flush L1's own ASID if it requested a *full* TLB flush on VMRUN */
+	if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID)
+		kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+
 	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
 }
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index a6d14937e7777..f37cba4f34c2b 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5645,9 +5645,8 @@ static __init void svm_set_cpu_caps(void)
 		kvm_cpu_cap_set(X86_FEATURE_VMCBCLEAN);
 
 		/*
-		 * KVM currently flushes TLBs on *every* nested SVM transition,
-		 * and so for all intents and purposes KVM supports flushing by
-		 * ASID, i.e. KVM is guaranteed to honor every L1 ASID flush.
+		 * KVM handles all TLB_CONTROL values set by L1, even if the
+		 * underlying CPU does not.
 		 */
 		kvm_cpu_cap_set(X86_FEATURE_FLUSHBYASID);
 
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 22/29] KVM: nSVM: Flush the TLB if L1 changes L2's ASID in vmcb12

[Yosry Ahmed]

KVM uses a single ASID for L2 guests per-vCPU. Hence, when L1 changes
L2's ASID in vmcb12 (e.g. due to switching to a different L2
vCPU or simply to avoid flushing an existing ASID), KVM needs to flush
the L2 ASID to correctly emulate different ASIDs as different TLB
domains.

Additionally, the nested NPT needs to be resync'd, as the MMU context
is not tagged by ASID, and KVM cannot reuse nested NPT entries across
different L2 ASIDs.

Essentially, L1 switching L2's ASID is treated exactly the same as L1
flushing L2's ASID, which is consistent with the APM:

  Software may effectively flush the guest's TLB entries by allocating a
  new ASID for the guest and not reusing the old ASID until the entire
  TLB has been flushed at least once.

This is similar to nVMX's handling of last_vpid, except that when L1
changes VPID12, KVM does *not* need to resync the nested EPT, because
VMX VPIDs, unlike SVM ASIDs, do not tag guest-physical translations
(i.e. nGPA to GPA translations).

Reset last_asid when nested VMRUN fails to force a flush and sync on the
next nested VMRUN. The TLB flush request survives the failure, but the
request to sync the nested NPT does not, as it ends up being serviced in
L1's MMU context after the synthesized VM-Exit.

Drop the commentary about vmcb12's ASID being copied around only for the
consistency checks, as they no longer apply, and opportunistically fix
whitespace alignment.

This is currently functionally a noop, as a full flush and sync is
triggered on every nested transition, but is a step toward eliminating
that.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 29 +++++++++++++++++++++++------
 arch/x86/kvm/svm/svm.h    |  2 ++
 2 files changed, 25 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index e04cc4f7669ea..8e4686b0d603a 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -556,10 +556,8 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu,
 	to->nested_cr3          = from->nested_cr3;
 	to->pause_filter_count  = from->pause_filter_count;
 	to->pause_filter_thresh = from->pause_filter_thresh;
-
-	/* Copy asid here because nested_vmcb_check_controls() will check it */
-	to->asid           = from->asid;
-	to->clean = from->clean;
+	to->asid		= from->asid;
+	to->clean		= from->clean;
 
 #ifdef CONFIG_KVM_HYPERV
 	/* Hyper-V extensions (Enlightened VMCB) */
@@ -691,23 +689,35 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,
 static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
+	bool new_asid = false;
 
 	/* Handle pending Hyper-V TLB flush requests */
 	kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
 
+	if (svm->nested.ctl.asid != svm->nested.last_asid) {
+		svm->nested.last_asid = svm->nested.ctl.asid;
+		new_asid = true;
+	}
+
 	/*
 	 * If L1 requested a TLB flush for L2, flush L2's TLB on nested entry
 	 * and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT
 	 * guest-physical mappings. Sync all roots, since the TLB flush is not
 	 * specific to the currently loaded nCR3.
 	 *
+	 * Handle L1 changing L2's ASID12 similarly, as KVM only uses one ASID
+	 * for L2 in hardware (per vCPU), so it must start fresh when L1 changes
+	 * ASID12 to emulate different ASIDs correctly.  Additionally, the MMU
+	 * context is not tagged by the ASID, so the shadow NPTs cannot be
+	 * reused across different L2 ASIDs.
+	 *
 	 * If L1 requested a full TLB flush for all ASIDs (including its own),
 	 * L1's own ASID is also flushed on nested VM-Exit, before running L1.
 	 *
 	 * TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL are handled
 	 * equally for simplicity.
 	 */
-	if (svm->nested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING) {
+	if (new_asid || (svm->nested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING)) {
 		if (nested_npt_enabled(svm))
 			kvm_make_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu);
 		kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
@@ -1237,6 +1247,13 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
 		svm->vmcb->control.exit_info_1  = 0;
 		svm->vmcb->control.exit_info_2  = 0;
 
+		/*
+		 * Force a flush+sync on the next nested VMRUN. Otherwise,
+		 * pending MMU syncs due to a changed ASID are lost, as
+		 * last_asid was already updated.
+		 */
+		svm->nested.last_asid = 0;
+
 		nested_svm_vmexit(svm);
 	}
 
@@ -1899,7 +1916,7 @@ void nested_svm_update_tsc_ratio_msr(struct kvm_vcpu *vcpu)
 	svm_write_tsc_multiplier(vcpu);
 }
 
-/* Inverse operation of nested_copy_vmcb_control_to_cache(). asid is copied too. */
+/* Inverse operation of nested_copy_vmcb_control_to_cache() */
 static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,
 					      struct vmcb_ctrl_area_cached *from)
 {
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index ed67c18e2b900..15c4aa3b02c2d 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -244,6 +244,8 @@ struct svm_nested_state {
 	 * on its side.
 	 */
 	bool force_msr_bitmap_recalc;
+
+	u32 last_asid;
 };
 
 struct vcpu_sev_es_state {
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 23/29] KVM: nSVM: Do not reset TLB_CONTROL in vmcb02 on nested VM-Enter

[Yosry Ahmed]

Stop clearing TLB_CONTROL when preparing the control area of vmcb02, as
this potentially undoes pending TLB flushes for L2 (e.g. through
KVM_REQ_TLB_FLUSH while L1 is running), and remove the associated TODO
comment.

This is currently harmless, because nested_svm_entry_tlb_flush() always
requests KVM_REQ_TLB_FLUSH_CURRENT on nested VM-Enter, which sets
TLB_CONTROL again before L2 is actually run. However, always flushing
will soon go away with proper TLB handling for L2, at which point always
clearing TLB_CONTROL would be a bug.

Clearing TLB_CONTROL on nested VM-Enter was probably done because
TLB_CONTROL is not cleared by the CPU on VM-Exit. However, KVM always
clears TLB_CONTROL in the active VMCB after a successful VMRUN. Hence,
at nested VM-Enter, TLB_CONTROL in vmcb02 can only be non-zero if a TLB
flush is queued for L2 while L1 is running (i.e. KVM_REQ_TLB_FLUSH), and
that should never be ignored.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 8e4686b0d603a..738ec1f2ec869 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -723,12 +723,7 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 		kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
 	}
 
-	/*
-	 * TODO: optimize unconditional TLB flush/MMU sync.  A partial list of
-	 * things to fix before this can be conditional:
-	 *
-	 *  - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
-	 */
+	/* TODO: optimize unconditional TLB flush/MMU sync */
 	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
 }
@@ -975,9 +970,6 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 
 	vmcb02->control.asid = svm->nested.asid02;
 
-	/* Overwritten later if necessary.  */
-	vmcb_clr_flush_asid(vmcb02);
-
 	/* Use vmcb01 MMU and format if guest does not use nNPT */
 	if (nested_npt_enabled(svm)) {
 		vmcb02->control.misc_ctl &= ~SVM_MISC_ENABLE_GMET;
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 24/29] KVM: x86/mmu: Rename __kvm_mmu_invalidate_addr() to kvm_mmu_sync_addr()

[Yosry Ahmed]

In preparation for creating another helper for
kvm_mmu_invalidate_addr(), rename __kvm_mmu_invalidate_addr() to
kvm_mmu_sync_addr(), which is arguably a more descriptive name.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/mmu/mmu.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c096c1a2b4684..b3085f409d0de 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6732,8 +6732,8 @@ void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_print_sptes);
 
-static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
-				      u64 addr, hpa_t root_hpa)
+static void kvm_mmu_sync_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
+			      u64 addr, hpa_t root_hpa)
 {
 	struct kvm_shadow_walk_iterator iterator;
 
@@ -6798,11 +6798,11 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 		return;
 
 	if (roots & KVM_MMU_ROOT_CURRENT)
-		__kvm_mmu_invalidate_addr(vcpu, mmu, addr, mmu->root.hpa);
+		kvm_mmu_sync_addr(vcpu, mmu, addr, mmu->root.hpa);
 
 	for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) {
 		if (roots & KVM_MMU_ROOT_PREVIOUS(i))
-			__kvm_mmu_invalidate_addr(vcpu, mmu, addr, mmu->prev_roots[i].hpa);
+			kvm_mmu_sync_addr(vcpu, mmu, addr, mmu->prev_roots[i].hpa);
 	}
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_invalidate_addr);
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 25/29] KVM: x86/mmu: Refactor kvm_mmu_invlpg() to allow skipping the GVA flush

[Yosry Ahmed]

Refactor helpers out of kvm_mmu_invalidate_addr() and kvm_mmu_invlpg()
that take in an extra argument to skip the GVA flush.

This will be used when invalidating GVAs in a different context than the
current one (i.e.  invalidating an L2 GVA from L1), so flushing the
current context would flush the wrong TLB entries.

No functional change intended.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/mmu/mmu.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index b3085f409d0de..433ed69b1e1e4 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6769,8 +6769,8 @@ static void kvm_mmu_sync_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
 	write_unlock(&vcpu->kvm->mmu_lock);
 }
 
-void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
-			     u64 addr, unsigned long roots)
+static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
+				      u64 addr, unsigned long roots, bool flush_gva)
 {
 	struct kvm_mmu *mmu;
 	int i;
@@ -6783,7 +6783,8 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 		if (is_noncanonical_invlpg_address(addr, vcpu))
 			return;
 
-		kvm_x86_call(flush_tlb_gva)(vcpu, addr);
+		if (flush_gva)
+			kvm_x86_call(flush_tlb_gva)(vcpu, addr);
 
 		if (tdp_enabled)
 			return;
@@ -6805,9 +6806,15 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 			kvm_mmu_sync_addr(vcpu, mmu, addr, mmu->prev_roots[i].hpa);
 	}
 }
+
+void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
+			     u64 addr, unsigned long roots)
+{
+	__kvm_mmu_invalidate_addr(vcpu, w, addr, roots, true);
+}
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_invalidate_addr);
 
-void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
+static void __kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva, bool flush_gva)
 {
 	/*
 	 * INVLPG is required to invalidate any global mappings for the VA,
@@ -6819,11 +6826,16 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
 	 * be synced when switching to that new cr3, so nothing needs to be
 	 * done here for them.
 	 */
-	kvm_mmu_invalidate_addr(vcpu, &vcpu->arch.gva_walk, gva, KVM_MMU_ROOTS_ALL);
+	__kvm_mmu_invalidate_addr(vcpu, &vcpu->arch.gva_walk, gva,
+				  KVM_MMU_ROOTS_ALL, flush_gva);
 	++vcpu->stat.invlpg;
 }
-EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_invlpg);
 
+void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
+{
+	__kvm_mmu_invlpg(vcpu, gva, true);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_invlpg);
 
 void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid)
 {
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 26/29] KVM: nSVM: Flush L2's ASID when emulating INVLPGA

[Yosry Ahmed]

KVM currently handles INVLPGA in the same way as INVLPG, flushing L1's
own ASID. This is currently correct because L1 and L2 share an ASID, but
it doesn't work once they have separate ASIDs.

If L1 is flushing a different L2 ASID than the one KVM is tracking, do
nothing, as KVM will flush L2's ASID in hardware (and sync the MMU if
needed) when L1 switches to the target ASID.

If L1 is flushing its own ASID, handle the flush the same as INVLPG.
Otherwise, skip the GVA flush for the current context (L1's ASID), and
flush the L2 ASID in hardware using INVLPGA if running on the same CPU
as L2. If not, fallback to a VMCB-based ASID flush. Note that if L2 then
runs on a different CPU KVM will flush the ASID anyway.

Either way, sync the MMU if NPT is disabled, which is handled by
__kvm_mmu_invlpg(). Note that all MMU roots are sync'd when NPT is
disabled, which can be optimized by keying off guest_mode to only sync
the appropriate context (L1 vs. L2).

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/mmu.h     |  3 +++
 arch/x86/kvm/mmu/mmu.c |  8 +++++---
 arch/x86/kvm/svm/svm.c | 38 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 44 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index b6ad0ff72393e..b5100f31f6a35 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -150,7 +150,10 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly,
 int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
 		       void *insn, int insn_len);
 void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg);
+void __kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva, bool flush_gva);
 void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva);
+void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
+			       u64 addr, unsigned long roots, bool flush_gva);
 void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 			     u64 addr, unsigned long roots);
 void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 433ed69b1e1e4..bfc3dd08bedb0 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6769,8 +6769,8 @@ static void kvm_mmu_sync_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
 	write_unlock(&vcpu->kvm->mmu_lock);
 }
 
-static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
-				      u64 addr, unsigned long roots, bool flush_gva)
+void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
+			       u64 addr, unsigned long roots, bool flush_gva)
 {
 	struct kvm_mmu *mmu;
 	int i;
@@ -6806,6 +6806,7 @@ static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk
 			kvm_mmu_sync_addr(vcpu, mmu, addr, mmu->prev_roots[i].hpa);
 	}
 }
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_mmu_invalidate_addr);
 
 void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 			     u64 addr, unsigned long roots)
@@ -6814,7 +6815,7 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_invalidate_addr);
 
-static void __kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva, bool flush_gva)
+void __kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva, bool flush_gva)
 {
 	/*
 	 * INVLPG is required to invalidate any global mappings for the VA,
@@ -6830,6 +6831,7 @@ static void __kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva, bool flush_gva)
 				  KVM_MMU_ROOTS_ALL, flush_gva);
 	++vcpu->stat.invlpg;
 }
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_mmu_invlpg);
 
 void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
 {
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index f37cba4f34c2b..4b95a05c37f60 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2468,17 +2468,51 @@ static int clgi_interception(struct kvm_vcpu *vcpu)
 
 static int invlpga_interception(struct kvm_vcpu *vcpu)
 {
+	struct vcpu_svm *svm = to_svm(vcpu);
 	/* FIXME: Handle an address size prefix. */
 	gva_t gva = kvm_rax_read(vcpu);
 	u32 asid = kvm_ecx_read(vcpu);
+	int cpu;
 
 	if (nested_svm_check_permissions(vcpu))
 		return 1;
 
 	trace_kvm_invlpga(to_svm(vcpu)->vmcb->save.rip, asid, gva);
 
-	/* Let's treat INVLPGA the same as INVLPG (can be optimized!) */
-	kvm_mmu_invlpg(vcpu, gva);
+	/*
+	 * INVLPG on a non-canonical address is a NOP according to the SDM,
+	 * assume the same behavior from INVLPGA since the APM doesn't specify.
+	 */
+	if (is_noncanonical_invlpg_address(gva, vcpu))
+		return kvm_skip_emulated_instruction(vcpu);
+
+	/*
+	 * Do nothing if L1 is flushing a different L2 ASID than the one KVM is
+	 * currently tracking.  KVM tracks a single L2 ASID, and performs a TLB
+	 * flush (and MMU resync if needed) when L1 switches ASIDs anyway.
+	 */
+	if (asid && asid != svm->nested.last_asid)
+		return kvm_skip_emulated_instruction(vcpu);
+
+	/*
+	 * Handle INVLPGA similar to INVLPG, with one caveat. If the specified
+	 * ASID is non-zero (i.e. L1 is not flushing its own ASID), skip
+	 * flushing the TLB for the current context (L1's), and use INVLPGA to
+	 * flush L2's ASID in hardware if running on the same CPU (otherwise
+	 * fallback to a full ASID flush).
+	 *
+	 * Note, if NPT is disabled, this will sync all the shadow page tables.
+	 * This can be optimized by keying off guest_mode.
+	 */
+	__kvm_mmu_invlpg(vcpu, gva, !asid);
+	if (asid) {
+		cpu = get_cpu();
+		if (cpu == svm->nested.vmcb02.cpu)
+			invlpga(gva, svm->nested.asid02);
+		else
+			vmcb_set_flush_asid(svm->nested.vmcb02.ptr);
+		put_cpu();
+	}
 
 	return kvm_skip_emulated_instruction(vcpu);
 }
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 27/29] KVM: nSVM: Flush the ASID on nested transitions if shared by L1 and L2

[Yosry Ahmed]

If L1 and L2 use the same ASID in hardware, always flush it on nested
transitions to avoid using L1 TLB entries for L2 or vice versa, since
from L1's perspective they are two different TLB domains.

This is currently always the case, but KVM will start using separate
ASIDs for L1 and L2 in most cases soon.

Note that KVM_REQ_TLB_FLUSH_CURRENT is required here, not
KVM_REQ_TLB_FLUSH_GUEST (used for L1 TLB flush requests). This is
because KVM could be switching between different NPT roots on the same
ASID. While this generally requires an ASID flush in the VMCB, which is
done by both KVM_REQ_TLB_FLUSH_CURRENT and KVM_REQ_TLB_FLUSH_GUEST, it
also requires a hypercall to specifically flush NPT mappings when
running on Hyper-V, which is only done with KVM_REQ_TLB_FLUSH_CURRENT
(as KVM_REQ_TLB_FLUSH_GUEST is only meant to flush translations created
by the guest).

Note that this is currently a noop as KVM requests
KVM_REQ_TLB_FLUSH_CURRENT unconditionally on nested transitions, but
this will soon be removed in favor of conditional flushes.

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 738ec1f2ec869..4ffc10bf9232e 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -723,6 +723,13 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 		kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
 	}
 
+	/*
+	 * If L1 and L2 share the same ASID in hardware (when using the fallback
+	 * ASID for both, or for SEV guests), flush it on nested transitions.
+	 */
+	if (svm->asid == svm->nested.asid02)
+		kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+
 	/* TODO: optimize unconditional TLB flush/MMU sync */
 	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
@@ -738,6 +745,9 @@ static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)
 	if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID)
 		kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
 
+	if (svm->asid == svm->nested.asid02)
+		kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+
 	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
 }
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 28/29] KVM: nSVM: Use different ASIDs for L1 and L2

[Yosry Ahmed]

Now that TLB flushes are properly handled and tracked for L1 vs L2
ASIDs, allocate a separate new ASID for L2 for each vCPU, similar to how
VMX handles VPIDs. Drop the unconditional flushes and syncs on nested
transitions.

L1 and L2 can now only share an ASID in two cases:
- KVM runs out of ASIDs and uses the fallback ASID for both L1 and L2.
- SEV VMs always use the same ASID for SEV-specific requirements.

WARN if the ASID is shared otherwise.

Initialize last_asid to 0, such that the L2 ASID is always flushed on
first nested VM-Enter after it's allocated.

On SEV VM migration, free the L2 ASID on the destination vCPU (if nested
was already initialized) before overwriting it with the SEV ASID.

On a Turin CPU, this results in 8-15% performance boost in CPUID rate
microbenchmark [1] and netperf TCP_RR latency/throughput.

[1] https://lore.kernel.org/kvm/20231109180646.2963718-1-khorenko@virtuozzo.com/

Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 arch/x86/kvm/svm/nested.c | 32 +++++++++++++++++++++++---------
 arch/x86/kvm/svm/sev.c    |  6 ++++--
 2 files changed, 27 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 4ffc10bf9232e..6fe68f2ae282d 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -711,6 +711,10 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 	 * context is not tagged by the ASID, so the shadow NPTs cannot be
 	 * reused across different L2 ASIDs.
 	 *
+	 * Note, last_asid is initialized as 0, so the first nested VM-Enter
+	 * after setting EFER.SVME will always flush the TLB to avoid using
+	 * stale entries.
+	 *
 	 * If L1 requested a full TLB flush for all ASIDs (including its own),
 	 * L1's own ASID is also flushed on nested VM-Exit, before running L1.
 	 *
@@ -727,12 +731,10 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
 	 * If L1 and L2 share the same ASID in hardware (when using the fallback
 	 * ASID for both, or for SEV guests), flush it on nested transitions.
 	 */
-	if (svm->asid == svm->nested.asid02)
+	if (svm->asid == svm->nested.asid02) {
+		WARN_ON_ONCE(svm->asid != fallback_asid && !is_sev_guest(vcpu));
 		kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
-
-	/* TODO: optimize unconditional TLB flush/MMU sync */
-	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
-	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+	}
 }
 
 static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)
@@ -747,9 +749,6 @@ static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)
 
 	if (svm->asid == svm->nested.asid02)
 		kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
-
-	kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
-	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
 }
 
 /*
@@ -1566,7 +1565,20 @@ int svm_allocate_nested(struct vcpu_svm *svm)
 	svm->nested.vmcb02.ptr = page_address(vmcb02_page);
 	svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
 
-	svm->nested.asid02 = svm->asid;
+	svm->nested.asid02 = allocate_asid(&svm->vcpu);
+
+	/*
+	 * KVM uses a fallback ASID when out of ASIDs, and fails vCPU creation
+	 * if there's no fallback ASID. ASID allocation must succeed here.
+	 */
+	KVM_BUG_ON(!svm->nested.asid02, svm->vcpu.kvm);
+
+	/*
+	 * Clear last_asid to ensure that the ASID is flushed on the first
+	 * nested VM-Enter. Otherwise, stale TLB entries from a previous life of
+	 * the ASID (e.g. different vCPU or even different VM) could be used.
+	 */
+	svm->nested.last_asid = 0;
 
 	return 0;
 
@@ -1583,6 +1595,8 @@ void svm_free_nested(struct vcpu_svm *svm)
 	if (WARN_ON_ONCE(svm->vmcb != svm->vmcb01.ptr))
 		svm_switch_vmcb(svm, &svm->vmcb01);
 
+	free_asid(svm->nested.asid02);
+
 	svm_vcpu_free_msrpm(svm->nested.msrpm);
 	svm->nested.msrpm = NULL;
 
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index b2e6acfbb081f..646108e70e21a 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -2043,10 +2043,12 @@ static void sev_vcpu_migrate_asid(struct vcpu_svm *dst_svm,
 	dst_svm->asid = asid;
 
 	/*
-	 * If nested is already initialized on the destination vCPU, also update
-	 * the nested ASID to match the new SEV ASID.
+	 * If nested is already initialized on the destination vCPU, also free
+	 * the previous ASID (in case it was non-SEV) and update the nested ASID
+	 * to match the new SEV ASID.
 	 */
 	if (dst_svm->nested.vmcb02.ptr) {
+		free_asid(dst_svm->nested.asid02);
 		dst_svm->nested.asid02 = asid;
 		dst_svm->nested.vmcb02.ptr->control.asid = asid;
 	}
-- 
2.56.0.360.g66cac248cb-goog

[PATCH v2 29/29] KVM: selftests: Add a test for nested TLB flushes

[Yosry Ahmed]

Add a test that exercises most common scenarios for TLB flushes. The
test is split into two testing scenarios:
1. Guest-triggered TLB flushes
2. KVM-triggered TLB flushes

For (1), the test runs an L1 guest that keeps switching an L2 PTE (or
TDP PTE) between two GPAs, flushes the TLB, and runs L2 to check that
accessing that address accesses the correct page. A missed TLB flush
causes L2 to access the wrong GPA through a stale TLB entry. The test
exercises multiple ways for L1 to flush the TLB, including changing L2's
ASID/VPID, flushing L2's entire ASID/VPID/EPTP, or flushing an
individual L2 GVA (using INVLPGA/INVVPID) when L1 does not enable TDP.

On AMD, the test also checks that an ASID flush (or an ASID change)
flushes shadow NPT mappings for all nCR3 roots, not just the current
one, by using multiple nested NPT trees.

For (2), the test creates a memslot with 2 pages, and keeps moving the
base GPA of the memslot between a TEST_GPA and TEST_GPA - PAGE_SIZE,
essentially resulting in the TEST_GPA being switched between two
underlying HPAs. KVM should flush the TLB for both L1 and L2 contexts
when updating a GPA -> HPA mapping.

The test runs an L1 guest that verifies access to page 1, triggers the
memslot move, and checks that both L1 and L2 now (correctly) access
page 2. This verifies that a TLB flush in L1's context flushes the TLB
for both L1 and L2. After that, the test does the opposite and triggers
the memslot move from L2, to double check that a TLB flush in L2's
context flushes the TLB for both L1 and L2.

The test runs various test cases with TDP on/off in L1, and actually
catches multiple injected bugs (in SVM), including:
- Missing the TLB flush on nested VM-Enter if ASID12 changes.
- Missing the TLB flush on nested VM-Enter if L1 uses
  TLB_CONTROL_FLUSH_ASID.
- Missing the nested NPT resync on nested VM-Enter if L1 uses
  TLB_CONTROL_FLUSH_ASID with nested NPT enabled.
- Missing the TLB flush when emulating INVLPGA.
- Missing flushing both L1 and L2 ASIDs when handling KVM_REQ_TLB_FLUSH.

Assisted-by: LLM
Signed-off-by: Yosry Ahmed <yosry@kernel.org>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 tools/testing/selftests/kvm/include/x86/svm.h |   5 +
 tools/testing/selftests/kvm/include/x86/vmx.h |  21 +
 .../testing/selftests/kvm/lib/x86/processor.c |  36 ++
 .../selftests/kvm/x86/nested_tlb_flush_test.c | 526 ++++++++++++++++++
 6 files changed, 590 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index be5eb2dd85aa2..5347c56786b08 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -122,6 +122,7 @@ TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa
 TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test
+TEST_GEN_PROGS_x86 += x86/nested_tlb_flush_test
 TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state
 TEST_GEN_PROGS_x86 += x86/svm_pmu_host_guest_test
 TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync
diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index ffec2291b1e1f..62d255aee5a34 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -1617,6 +1617,7 @@ void tdp_map(struct kvm_vm *vm, gpa_t l2_gpa, gpa_t gpa, u64 size);
 void tdp_identity_map_default_memslots(struct kvm_vm *vm);
 void tdp_identity_map_1g(struct kvm_vm *vm,  u64 addr, u64 size);
 u64 *tdp_get_pte(struct kvm_vm *vm, u64 l2_gpa);
+gpa_t tdp_copy_page_tables(struct kvm_vm *vm);
 
 /*
  * Basic CPU control in CR0
diff --git a/tools/testing/selftests/kvm/include/x86/svm.h b/tools/testing/selftests/kvm/include/x86/svm.h
index c8539166270ea..7644e37c7042c 100644
--- a/tools/testing/selftests/kvm/include/x86/svm.h
+++ b/tools/testing/selftests/kvm/include/x86/svm.h
@@ -316,4 +316,9 @@ struct __attribute__ ((__packed__)) vmcb {
 
 #define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP)
 
+static inline void invlpga(unsigned long addr, u32 asid)
+{
+	asm volatile("invlpga %0, %1" : : "a"(addr), "c"(asid));
+}
+
 #endif /* SELFTEST_KVM_SVM_H */
diff --git a/tools/testing/selftests/kvm/include/x86/vmx.h b/tools/testing/selftests/kvm/include/x86/vmx.h
index 8fd97434eee5c..4f2a0e3ac5ba1 100644
--- a/tools/testing/selftests/kvm/include/x86/vmx.h
+++ b/tools/testing/selftests/kvm/include/x86/vmx.h
@@ -518,4 +518,25 @@ static inline bool kvm_cpu_has_vmx_virtualize_apic_accesses(void)
 void vm_enable_ept(struct kvm_vm *vm);
 void prepare_virtualize_apic_accesses(struct vmx_pages *vmx, struct kvm_vm *vm);
 
+static inline void invvpid(unsigned long ext, u16 vpid, gva_t gva)
+{
+	struct {
+		u64 vpid : 16;
+		u64 rsvd : 48;
+		u64 gva;
+	} operand = { vpid, 0, gva };
+
+	asm volatile("invvpid %0, %1" : : "m"(operand), "r"(ext) : "memory");
+}
+
+static inline void invept(unsigned long ext, u64 eptp)
+{
+	struct {
+		u64 eptp;
+		u64 rsvd;
+	} operand = { eptp, 0 };
+
+	asm volatile("invept %0, %1" : : "m"(operand), "r"(ext) : "memory");
+}
+
 #endif /* SELFTEST_KVM_VMX_H */
diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c
index 8374afd08bafe..57ecf8b381b5a 100644
--- a/tools/testing/selftests/kvm/lib/x86/processor.c
+++ b/tools/testing/selftests/kvm/lib/x86/processor.c
@@ -554,6 +554,42 @@ void tdp_identity_map_1g(struct kvm_vm *vm, u64 addr, u64 size)
 	__tdp_map(vm, addr, addr, size, PG_LEVEL_1G);
 }
 
+static gpa_t __virt_copy_page_tables(struct kvm_vm *vm, struct kvm_mmu *mmu,
+				     gpa_t table_gpa, int level)
+{
+	gpa_t copy_gpa = vm_alloc_page_table(vm);
+	u64 *copy = addr_gpa2hva(vm, copy_gpa);
+	gpa_t pa;
+	int i;
+
+	memcpy(copy, addr_gpa2hva(vm, table_gpa), vm->page_size);
+	if (level == PG_LEVEL_4K)
+		return copy_gpa;
+
+	for (i = 0; i < vm->page_size / sizeof(u64); i++) {
+		if (!is_present_pte(mmu, &copy[i]) || is_huge_pte(mmu, &copy[i]))
+			continue;
+
+		/* Clear only the address bits, preserving C/S bits and flags */
+		pa = vm_untag_gpa(vm, PTE_GET_PA(copy[i]));
+		copy[i] &= ~pa;
+		copy[i] |= __virt_copy_page_tables(vm, mmu, pa, level - 1);
+	}
+	return copy_gpa;
+}
+
+/*
+ * Deep copy all levels of the TDP page tables, such that no page table pages
+ * are shared with the original.  Returns the GPA of the new root.
+ */
+gpa_t tdp_copy_page_tables(struct kvm_vm *vm)
+{
+	struct kvm_mmu *mmu = &vm->stage2_mmu;
+
+	TEST_ASSERT(mmu->pgd_created, "TDP page tables not created");
+	return __virt_copy_page_tables(vm, mmu, mmu->pgd, mmu->pgtable_levels);
+}
+
 /*
  * Set Unusable Segment
  *
diff --git a/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c
new file mode 100644
index 0000000000000..fb32b7bf6b19f
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c
@@ -0,0 +1,526 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026, Google LLC.
+ *
+ * Test TLB flushes with nested virtualization across three cases:
+ * 1. When L1 updates L2's (shadow) page tables
+ * 2. When L1 updates nested TDP
+ * 3. When KVM updates mappings on the host (KVM-induced TLB flushes)
+ */
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "svm_util.h"
+#include "vmx.h"
+
+#define NR_ITERATIONS	100
+
+#define VAL1		0x11111111ULL
+#define VAL2		0x22222222ULL
+
+#define TEST_VADDR	0x10000000ULL
+#define TEST_GPA	0x30000000ULL
+#define TEST_NESTED_GPA	0x40000000ULL
+
+#define TEST_MEMSLOT	10
+
+#define SYNC_MOVE_MEMSLOT 1
+
+enum guest_flush_method {
+	TLB_FLUSH_TAG = 0,
+	TLB_FLUSH_VADDR,
+	TLB_CHANGE_TAG,
+	TLB_FLUSH_NONE,
+};
+
+static u64 *pte_gva;
+static gpa_t test_gpa[2];
+static enum guest_flush_method flush_method;
+static u32 max_asid;
+
+/* TDP roots for the multi-root test, root 1 is a non-aliasing copy of root 0 */
+static gpa_t tdp_root_gpa[2];
+/* Value L2 expects at TEST_VADDR, 0 if L2 shouldn't access TEST_VADDR */
+static u64 l2_expected_val;
+
+static void svm_tlb_flush(struct svm_test_data *svm,
+			  enum guest_flush_method method)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	switch (method) {
+	case TLB_CHANGE_TAG:
+		vmcb->control.asid++;
+		vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+		if (vmcb->control.asid >= max_asid) {
+			vmcb->control.asid = 1;
+			vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
+		}
+		break;
+	case TLB_FLUSH_TAG:
+		vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+		break;
+	case TLB_FLUSH_VADDR:
+		GUEST_ASSERT(!svm->ncr3_gpa);
+		vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+		invlpga(TEST_VADDR, vmcb->control.asid);
+		break;
+	case TLB_FLUSH_NONE:
+		vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+		break;
+	}
+}
+
+static void vmx_tlb_flush(struct vmx_pages *vmx, enum guest_flush_method method)
+{
+	u16 vpid = vmread(VIRTUAL_PROCESSOR_ID);
+
+	switch (method) {
+	case TLB_CHANGE_TAG:
+		GUEST_ASSERT(!vmx->eptp_gpa);
+		vmwrite(VIRTUAL_PROCESSOR_ID, vpid + 1);
+		break;
+	case TLB_FLUSH_TAG:
+		if (vmx->eptp_gpa)
+			invept(1, vmread(EPT_POINTER));
+		else
+			invvpid(1, vpid, 0);
+		break;
+	case TLB_FLUSH_VADDR:
+		GUEST_ASSERT(!vmx->eptp_gpa);
+		invvpid(0, vpid, TEST_VADDR);
+		break;
+	case TLB_FLUSH_NONE:
+		break;
+	}
+}
+
+static void prepare_l2(void *nested_state, void *l2_guest_code)
+{
+	struct vmx_pages *vmx = nested_state;
+
+	if (this_cpu_has(X86_FEATURE_VMX)) {
+		u32 sec_exec_ctl;
+
+		prepare_for_vmx_operation(vmx);
+		load_vmcs(vmx);
+		prepare_vmcs(vmx, l2_guest_code);
+
+		/* Enable VPID */
+		sec_exec_ctl = vmread(SECONDARY_VM_EXEC_CONTROL);
+		sec_exec_ctl |= SECONDARY_EXEC_ENABLE_VPID;
+		vmwrite(SECONDARY_VM_EXEC_CONTROL, sec_exec_ctl);
+		vmwrite(VIRTUAL_PROCESSOR_ID, 1);
+	} else {
+		generic_svm_setup(nested_state, l2_guest_code);
+	}
+}
+
+static void __run_l2(void *nested_state, bool launch,
+		     enum guest_flush_method method)
+{
+	struct svm_test_data *svm;
+
+	if (this_cpu_has(X86_FEATURE_VMX)) {
+		vmx_tlb_flush(nested_state, method);
+		if (launch)
+			vmlaunch();
+		else
+			vmresume();
+		GUEST_ASSERT_EQ(vmread(VM_EXIT_REASON), EXIT_REASON_VMCALL);
+		vmwrite(GUEST_RIP, vmread(GUEST_RIP) + 3); /* skip over VMCALL */
+	} else {
+		svm = nested_state;
+		svm_tlb_flush(svm, method);
+		run_guest(svm->vmcb, svm->vmcb_gpa);
+		GUEST_ASSERT_EQ(svm->vmcb->control.exit_code, SVM_EXIT_VMMCALL);
+		svm->vmcb->save.rip += 3; /* skip over VMMCALL */
+	}
+}
+
+static void run_l2(void *nested_state, bool launch)
+{
+	__run_l2(nested_state, launch, flush_method);
+}
+
+static void l2_guest_code(void)
+{
+	u64 expected_val;
+
+	for (;;) {
+		expected_val = READ_ONCE(l2_expected_val);
+		if (expected_val)
+			GUEST_ASSERT_EQ(READ_ONCE(*(u64 *)TEST_VADDR), expected_val);
+		vmcall();
+	}
+}
+
+static void l1_guest_code(void *data)
+{
+	gpa_t gpa;
+	int i;
+
+	prepare_l2(data, l2_guest_code);
+
+	/*
+	 * Alternately switch the PTE (or TDP PTE) mapping TEST_VADDR between
+	 * two pages containing VAL1 and VAL2, flush TLBs, and verify that L2
+	 * reads the expected value.
+	 */
+	for (i = 0; i < NR_ITERATIONS; i++) {
+		gpa = test_gpa[i % 2];
+
+		*pte_gva &= ~PHYSICAL_PAGE_MASK;
+		*pte_gva |= gpa & PHYSICAL_PAGE_MASK;
+
+		WRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);
+		run_l2(data, i == 0);
+	}
+
+	GUEST_DONE();
+}
+
+/*
+ * Verify that flushing L2's ASID invalidates nested TDP translations for all
+ * TDP roots, not just the one in use at the time of the flush.  L1 updates the
+ * nested TDP PTE for TEST_VADDR in root 0, flushes L2's ASID while running L2
+ * with root 1, then runs L2 with root 0 *without* flushing.  L2 must observe
+ * the new mapping, as the flush invalidated all translations for the ASID.
+ *
+ * Root 1 is a full copy of root 0, such that none of the nested TDP pages are
+ * shared, i.e. KVM's shadow pages for root 0 are not reachable through the
+ * shadow root for root 1.
+ */
+static void l1_guest_code_multi_tdp_root(void *data)
+{
+	struct svm_test_data *svm = data;
+	struct vmcb *vmcb = svm->vmcb;
+	int i;
+
+	GUEST_ASSERT(this_cpu_has(X86_FEATURE_SVM));
+	GUEST_ASSERT_EQ(svm->ncr3_gpa, tdp_root_gpa[0]);
+
+	prepare_l2(data, l2_guest_code);
+
+	for (i = 0; i < NR_ITERATIONS; i++) {
+		*pte_gva &= ~PHYSICAL_PAGE_MASK;
+		*pte_gva |= test_gpa[i % 2] & PHYSICAL_PAGE_MASK;
+
+		/* Flush L2's ASID (per flush_method) while using root 1 */
+		vmcb->control.nested_cr3 = tdp_root_gpa[1];
+		WRITE_ONCE(l2_expected_val, 0);
+		run_l2(data, i == 0);
+
+		/* Switch back to root 0 without flushing */
+		vmcb->control.nested_cr3 = tdp_root_gpa[0];
+		WRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);
+		__run_l2(data, false, TLB_FLUSH_NONE);
+	}
+
+	GUEST_DONE();
+}
+
+static void prep_guest_flush_test(struct kvm_vm *vm, bool tdp_enabled,
+				  bool copy_tdp_root)
+{
+	gva_t page_gva[2], pte_map_gva;
+	gpa_t pte_gpa;
+	u64 *ptep;
+
+	/*
+	 * Allocate two pages in the VM and initialize them with different
+	 * values. L1 will update the mappings to switch between these pages and
+	 * L2 will verify that the accessed value indicates the right page.
+	 */
+	page_gva[0] = vm_alloc_page(vm);
+	page_gva[1] = vm_alloc_page(vm);
+
+	*(u64 *)addr_gva2hva(vm, page_gva[0]) = VAL1;
+	*(u64 *)addr_gva2hva(vm, page_gva[1]) = VAL2;
+
+	test_gpa[0] = addr_gva2gpa(vm, page_gva[0]);
+	test_gpa[1] = addr_gva2gpa(vm, page_gva[1]);
+
+	if (tdp_enabled) {
+		virt_pg_map(vm, TEST_VADDR, TEST_NESTED_GPA);
+		tdp_map(vm, TEST_NESTED_GPA, test_gpa[0], vm->page_size);
+		ptep = tdp_get_pte(vm, TEST_NESTED_GPA);
+	} else {
+		virt_pg_map(vm, TEST_VADDR, test_gpa[0]);
+		ptep = vm_get_pte(vm, TEST_VADDR);
+	}
+
+	/*
+	 * Map the PTE (or TDP PTE) for TEST_VADDR, such that L1 can read and
+	 * update the mapping. Pass the PTE GVA to L1 to directly use it (rather
+	 * than L1 walking the page tables.
+	 */
+	pte_gpa = addr_hva2gpa(vm, ptep);
+	pte_map_gva = vm_unused_gva_gap(vm, vm->page_size, KVM_UTIL_MIN_VADDR);
+	virt_pg_map(vm, pte_map_gva, pte_gpa & PAGE_MASK);
+	pte_gva = (u64 *)(pte_map_gva + (pte_gpa & ~PAGE_MASK));
+
+	sync_global_to_guest(vm, pte_gva);
+	sync_global_to_guest(vm, test_gpa);
+
+	if (!tdp_enabled)
+		return;
+
+	/*
+	 * Must be done after all pages accessed by L2 are allocated, including
+	 * nested state (e.g. L2's stack) and page tables for TEST_VADDR.
+	 */
+	tdp_identity_map_default_memslots(vm);
+
+	/* Must be done after all TDP mappings are created */
+	if (copy_tdp_root) {
+		tdp_root_gpa[0] = vm->stage2_mmu.pgd;
+		tdp_root_gpa[1] = tdp_copy_page_tables(vm);
+		sync_global_to_guest(vm, tdp_root_gpa);
+	}
+}
+
+static void l2_guest_code_memslot_move(void)
+{
+	int i;
+	u64 val;
+
+	for (i = 0; i < NR_ITERATIONS; i++) {
+		val = READ_ONCE(*(u64 *)TEST_VADDR);
+		GUEST_ASSERT_EQ(val, VAL1);
+
+		vmcall();
+
+		val = READ_ONCE(*(u64 *)TEST_VADDR);
+		GUEST_ASSERT_EQ(val, VAL2);
+
+		GUEST_SYNC2(SYNC_MOVE_MEMSLOT, 1);
+
+		val = READ_ONCE(*(u64 *)TEST_VADDR);
+		GUEST_ASSERT_EQ(val, VAL1);
+
+		vmcall();
+	}
+}
+
+static void l1_guest_code_memslot_move(void *data)
+{
+	int i;
+	u64 val;
+
+	prepare_l2(data, l2_guest_code_memslot_move);
+
+	for (i = 0; i < NR_ITERATIONS; i++) {
+		/* Verify that accessing VADDR reads from page 1 in both L1 & L2 */
+		val = READ_ONCE(*(u64 *)TEST_VADDR);
+		GUEST_ASSERT_EQ(val, VAL1);
+
+		run_l2(data, i == 0);
+
+		/*
+		 * Update VADDR to point at page 2. KVM should flush both L1 and
+		 * L2's TLB translations so that further accesses go to page 2.
+		 */
+		GUEST_SYNC2(SYNC_MOVE_MEMSLOT, 2);
+
+		/*
+		 * Verify that accessing VADDR reads from page 2. A stale TLB
+		 * entry would lead to reading from page 1. Enter L2 and check
+		 * access to page 2, to verify that a TLB flush from L1's
+		 * context flushes both L1 and L2.
+		 */
+		val = READ_ONCE(*(u64 *)TEST_VADDR);
+		GUEST_ASSERT_EQ(val, VAL2);
+
+		run_l2(data, false);
+
+		/*
+		 * L2 updated VADDR to point at page 1 again. Verify access to
+		 * page 1 again to verify that a TLB flush from L2's context
+		 * flushes both L1 and L2 TLB translations.
+		 */
+		val = READ_ONCE(*(u64 *)TEST_VADDR);
+		GUEST_ASSERT_EQ(val, VAL1);
+	}
+
+	GUEST_DONE();
+}
+
+static void prep_memslot_move_test(struct kvm_vm *vm, bool tdp_enabled)
+{
+	/*
+	 * Disable KVM_X86_QUIRK_SLOT_ZAP_ALL. By default, moving a memslot
+	 * invalidates all MMU roots, which implicitly flushes the guest TLB
+	 * and masks missing TLB flush bugs. Disabling it forces KVM to keep
+	 * the same roots, relying solely on explicit TLB flushes.
+	 */
+	TEST_REQUIRE(kvm_check_cap(KVM_CAP_DISABLE_QUIRKS2) & KVM_X86_QUIRK_SLOT_ZAP_ALL);
+	vm_enable_cap(vm, KVM_CAP_DISABLE_QUIRKS2, KVM_X86_QUIRK_SLOT_ZAP_ALL);
+
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+				    TEST_GPA, TEST_MEMSLOT,
+				    /*npages=*/2, /*flags=*/0);
+
+	virt_pg_map(vm, TEST_VADDR, TEST_GPA);
+
+	if (tdp_enabled)
+		tdp_map(vm, TEST_GPA, TEST_GPA, vm->page_size);
+
+	*(u64 *)addr_gpa2hva(vm, TEST_GPA) = VAL1;
+	*(u64 *)addr_gpa2hva(vm, TEST_GPA + vm->page_size) = VAL2;
+
+	/*
+	 * Must be done after all pages accessed by L2 are allocated, including
+	 * nested state (e.g. L2's stack) and page tables for TEST_VADDR.
+	 */
+	if (tdp_enabled)
+		tdp_identity_map_default_memslots(vm);
+}
+
+static void calc_max_svm_asid(void)
+{
+	const struct kvm_cpuid_entry2 *entry;
+
+	if (!kvm_cpu_has(X86_FEATURE_SVM))
+		return;
+
+	entry = get_cpuid_entry(kvm_get_supported_cpuid(), 0x8000000A, 0);
+	max_asid = entry ? entry->ebx : 0;
+}
+
+static void run_test(void *guest_code, bool tdp_enabled)
+{
+	struct kvm_vcpu *vcpu;
+	gva_t nested_gva = 0;
+	struct kvm_vm *vm;
+	struct ucall uc;
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	if (tdp_enabled)
+		vm_enable_tdp(vm);
+
+	if (kvm_cpu_has(X86_FEATURE_VMX))
+		vcpu_alloc_vmx(vm, &nested_gva);
+	else
+		vcpu_alloc_svm(vm, &nested_gva);
+
+	if (guest_code == l1_guest_code_memslot_move)
+		prep_memslot_move_test(vm, tdp_enabled);
+	else if (guest_code == l1_guest_code_multi_tdp_root)
+		prep_guest_flush_test(vm, tdp_enabled, true);
+	else
+		prep_guest_flush_test(vm, tdp_enabled, false);
+
+	vcpu_args_set(vcpu, 1, nested_gva);
+
+	sync_global_to_guest(vm, flush_method);
+	sync_global_to_guest(vm, max_asid);
+
+	for (;;) {
+		vcpu_run(vcpu);
+		TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+		switch (get_ucall(vcpu, &uc)) {
+		case UCALL_ABORT:
+			REPORT_GUEST_ASSERT(uc);
+			break;
+		case UCALL_SYNC:
+			/*
+			 * Move the base GPA of the memslot between TEST_GPA and
+			 * TEST_GPA - PAGE_SIZE. This effectively moves
+			 * TEST_VADDR to point at page 1 or page 2 in the
+			 * memslot.
+			 */
+			if (uc.args[0] == SYNC_MOVE_MEMSLOT) {
+				gpa_t new_gpa = (uc.args[1] == 1) ?
+					TEST_GPA : (TEST_GPA - vm->page_size);
+
+				vm_mem_region_move(vm, TEST_MEMSLOT, new_gpa);
+			} else {
+				TEST_FAIL("Unknown sync code: %lu", uc.args[0]);
+			}
+			break;
+		case UCALL_DONE:
+			goto done;
+		default:
+			TEST_FAIL("Unknown ucall %lu", uc.cmd);
+		}
+	}
+
+done:
+	kvm_vm_free(vm);
+}
+
+#define tlb_test(test_name, guest_code, tdp_setting, flush_setting)		\
+do {										\
+	tdp_setting;								\
+	flush_setting;								\
+										\
+	if (tdp_enabled && !kvm_cpu_has_tdp()) {				\
+		pr_info("Skipping: " test_name " (no TDP support)\n");		\
+		break;								\
+	}									\
+										\
+	/*									\
+	 * VPIDs do not tag guest-physical translations, i.e. changing or	\
+	 * flushing VPIDs does not flush EPT translations, and single-context	\
+	 * INVEPT only flushes the target EPTP (not all TDP roots).		\
+	 */									\
+	if (tdp_enabled && kvm_cpu_has_ept() &&					\
+	    (flush_method == TLB_CHANGE_TAG ||					\
+	     guest_code == l1_guest_code_multi_tdp_root)) {			\
+		pr_info("Skipping: " test_name " (not applicable to EPTs)\n");	\
+		break;								\
+	}									\
+										\
+	pr_info("Testing " test_name "...\n");					\
+	run_test(guest_code, tdp_enabled);					\
+} while (0)
+
+int main(int argc, char *argv[])
+{
+	bool tdp_enabled;
+
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM) || kvm_cpu_has(X86_FEATURE_VMX));
+
+	calc_max_svm_asid();
+
+	tlb_test("Guest-triggered TLB flush (flush entire tag), TDP disabled",
+		 l1_guest_code, tdp_enabled = false,
+		 flush_method = TLB_FLUSH_TAG);
+	tlb_test("Guest-triggered TLB flush (flush individual address), TDP disabled",
+		 l1_guest_code, tdp_enabled = false,
+		 flush_method = TLB_FLUSH_VADDR);
+	tlb_test("Guest-triggered TLB flush (change tag), TDP disabled",
+		 l1_guest_code, tdp_enabled = false,
+		 flush_method = TLB_CHANGE_TAG);
+
+	tlb_test("Guest-triggered TLB flush (flush entire tag), TDP enabled",
+		 l1_guest_code, tdp_enabled = true,
+		 flush_method = TLB_FLUSH_TAG);
+	tlb_test("Guest-triggered TLB flush (change tag), TDP enabled",
+		 l1_guest_code, tdp_enabled = true,
+		 flush_method = TLB_CHANGE_TAG);
+
+	tlb_test("Guest-triggered TLB flush with multiple TDP roots (flush entire tag), TDP enabled",
+		 l1_guest_code_multi_tdp_root, tdp_enabled = true,
+		 flush_method = TLB_FLUSH_TAG);
+	tlb_test("Guest-triggered TLB flush with multiple TDP roots (change tag), TDP enabled",
+		 l1_guest_code_multi_tdp_root, tdp_enabled = true,
+		 flush_method = TLB_CHANGE_TAG);
+
+
+	tlb_test("KVM-triggered TLB flush via memslot move, TDP disabled",
+		 l1_guest_code_memslot_move, tdp_enabled = false,
+		 flush_method = TLB_FLUSH_NONE);
+	tlb_test("KVM-triggered TLB flush via memslot move, TDP enabled",
+		 l1_guest_code_memslot_move, tdp_enabled = true,
+		 flush_method = TLB_FLUSH_NONE);
+
+	return 0;
+}
-- 
2.56.0.360.g66cac248cb-goog

Re: [PATCH v2 20/29] KVM: x86: Add KVM_REQ_MMU_SYNC_ALL_ROOTS

[Lai Jiangshan]

On Thu, Oct 08, 2026 at 12:14:16AM +0800, Yosry Ahmed wrote:
> Add a new x86 request to sync all roots, current and previous. This will
> be used when syncing the shadow NPTs on nested VMRUN when an ASID flush
> (or new ASID) is requested by L1, as all physical translations in all
> roots should be sync'd (since shadow mappings are not tagged with an
> ASID).
> 
> Refactor syncing all roots (currently only done by
> kvm_vcpu_flush_tlb_guest()) to a helper, and use it to handle the
> request.
> 
> Clear KVM_REQ_MMU_SYNC when handling KVM_REQ_MMU_SYNC_ALL_ROOTS as the
> latter is a superset. Do not do so inside kvm_mmu_sync_all_roots(), to
> avoid any subtle behavioral changes from kvm_vcpu_flush_tlb_guest()
> clearing KVM_REQ_MMU_SYNC. It makes sense to always clear it when
> syncing the MMU, but there might be weird dependencies on the sync only
> happening before the actual vCPU run, and kvm_vcpu_flush_tlb_guest() is
> called in other code paths than handling KVM_REQ_TLB_FLUSH_GUEST before
> vCPU run.
> 
> No functional change intended.

Reviewed-by: Lai Jiangshan <jiangshanlai@gmail.com>

> 
> Signed-off-by: Yosry Ahmed <yosry@kernel.org>
> ---
>  arch/x86/include/asm/kvm_host.h |  1 +
>  arch/x86/kvm/mmu.h              |  7 +++++++
>  arch/x86/kvm/x86.c              | 11 ++++++++---
>  3 files changed, 16 insertions(+), 3 deletions(-)
>