From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out28-51.mail.aliyun.com (out28-51.mail.aliyun.com [115.124.28.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D95A1F09A5; Sat, 20 Sep 2025 06:52:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.28.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758351168; cv=none; b=BkjrcnsXZMES2RSbjPWy080Bdi6ZKJG12MVmZAu3RYxInh6PlE8bRAb8eImfaxP7Fw/x6NSYfLx01LEBFYvmstGtGf68D9iDw5QFHM5vGqFxSbukmbZq6mOyaJSVurVp6G9wCMBS3hx94nuMMf2xUHG+L6RMgJl9xXSZib6XCeU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758351168; c=relaxed/simple; bh=fnb/zxd71rqI2UFUVI9iXzUfKPPCRzK8bXijM1GASc0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=lwhshRrZvE3jLfb0LihOSm0Vh5ju9dkLbUqo2XbrfMAlL3IT4YCi+mc/l4WYu0MGFLVa1xj3bwofvzetvbwYk3TTlvCvyE5j65qtOyUtlRjJbw2Rxq128KjlTR0xgJy9AIsbG6mgbex7jxg160gHZu3BuvuNd8ceUvlz1HBopUc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=antgroup.com; spf=pass smtp.mailfrom=antgroup.com; dkim=pass (1024-bit key) header.d=antgroup.com header.i=@antgroup.com header.b=HHH5TPQU; arc=none smtp.client-ip=115.124.28.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=antgroup.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=antgroup.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=antgroup.com header.i=@antgroup.com header.b="HHH5TPQU" DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=antgroup.com; s=default; t=1758351154; h=Date:From:To:Subject:Message-ID:MIME-Version:Content-Type; bh=m1yLKs765q27aYv4Da+INz4B+7chhtyioEysvoSOx8E=; b=HHH5TPQUKsl4scUxIDXcKG0mnoGCTtIwDaeg16ZT/tCDAwgi5P+iVCHqjYbKO0ne08qIzOy1nZzJhVEcDn517WXsZg4XsAfuP4mt14eWxO1x2Q1KE5eB/AEk6VLQiWeks+K6QqhvMt9F/DW/bQoJfjnl8Qepdo4Mb142Hi1howk= Received: from localhost(mailfrom:houwenlong.hwl@antgroup.com fp:SMTPD_---.ejMJ8-W_1758351153 cluster:ay29) by smtp.aliyun-inc.com; Sat, 20 Sep 2025 14:52:33 +0800 Date: Sat, 20 Sep 2025 14:52:33 +0800 From: Hou Wenlong To: Sean Christopherson Cc: Tom Lendacky , kvm@vger.kernel.org, Lai Jiangshan , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org Subject: Re: [PATCH 2/2] KVM: SVM: Use cached value as restore value of TSC_AUX for SEV-ES guest Message-ID: <20250920065233.GA48402@k08j02272.eu95sqa> References: <05a018a6997407080b3b7921ba692aa69a720f07.1758166596.git.houwenlong.hwl@antgroup.com> <9da5eb48ccf403e1173484195d3d7d96978125b7.1758166596.git.houwenlong.hwl@antgroup.com> <9991df11-fe7c-41e1-9890-f0c38adc8137@amd.com> <20250919131535.GA73646@k08j02272.eu95sqa> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) On Fri, Sep 19, 2025 at 09:23:26AM -0700, Sean Christopherson wrote: > On Fri, Sep 19, 2025, Hou Wenlong wrote: > > On Thu, Sep 18, 2025 at 01:47:06PM -0500, Tom Lendacky wrote: > > > On 9/17/25 22:38, Hou Wenlong wrote: > > > > The commit 916e3e5f26ab ("KVM: SVM: Do not use user return MSR support > > > > for virtualized TSC_AUX") assumes that TSC_AUX is not changed by Linux > > > > post-boot, so it always restores the initial host value on #VMEXIT. > > > > However, this is not true in KVM, as it can be modified by user return > > > > MSR support for normal guests. If an SEV-ES guest always restores the > > > > initial host value on #VMEXIT, this may result in the cached value in > > > > user return MSR being different from the hardware value if the previous > > > > vCPU was a non-SEV-ES guest that had called kvm_set_user_return_msr(). > > > > Consequently, this may pose a problem when switching back to that vCPU, > > > > as kvm_set_user_return_msr() would not update the hardware value because > > > > the cached value matches the target value. Unlike the TDX case, the > > > > SEV-ES guest has the ability to set the restore value in the host save > > > > area, and the cached value in the user return MSR is always the current > > > > hardware value. Therefore, the cached value could be used directly > > > > without RDMSR in svm_prepare_switch_to_guest(), making this change > > > > minimal. > > > > > > I'm not sure I follow. If Linux never changes the value of TSC_AUX once it > > > has set it, then how can it ever be different? Have you seen this issue? > > > > > > Thanks, > > > Tom > > > > > Hi, Tom. > > > > IIUD, the normal guest still uses the user return MSR to load the guest > > TSC_AUX value into the hardware when TSC_AUX virtualization is > > supported. However, the user return MSR only restores the host value > > when returning to userspace, rather than when the vCPU is scheduled out. > > This may lead to an issue during vCPU switching on a single pCPU, which > > appears as follows: > > > > normal vCPU -> SEV-ES vCPU -> normal vCPU > > > > When the normal vCPU switches to the SEV-ES vCPU, the hardware TSC_AUX > > value remains as the guest value set in kvm_set_user_return_msr() by the > > normal vCPU. After the #VMEXIT from the SEV-ES vCPU, the hardware value > > becomes the host value. However, the cached TSC_AUX value in the user > > return MSR remains the guest value of previous normal vCPU. Therefore, > > when switching back to that normal vCPU, kvm_set_user_return_msr() does > > not perform a WRMSR to load the guest value into the hardware, because > > the cached value matches the target value. As a result, during the > > execution of the normal vCPU, the normal vCPU would get an incorrect > > TSC_AUX value for RDTSCP/RDPID. > > > > I didn't find the available description of TSC_AUX virtualization in > > APM; all my analysis is based on the current KVM code. > > I'm guessing TSC_AUX virtualization works like SEV-ES, where hardware context > switches the MSR on VMRUN/#VMEXIT. > > > Am I missing something? > > Nope, I don't think so. I also found the changelog a bit confusing though. I > would say omit the details about Linux not changing the value, and instead focus > on the need to re-load the current hardware value. That should be intuitive for > all readers, and is correct regradless of what/whose value is currently in hardware. > > I also think we should handle setting hostsa->tsc_aux in > sev_es_prepare_switch_to_guest(). That obviously requires duplicating some of > logic related to SEV-ES and TSC_AUX, but I think I prefer that to splitting the > handling of the host save area. > > How's this look? (compile tested only) > I'm fine with it. Thanks for your fixup; I think I need to improve my changlog next time. :) Thanks! > -- > Subject: [PATCH] KVM: SVM: Re-load current, not host, TSC_AUX on #VMEXIT from > SEV-ES guest > > Prior to running an SEV-ES guest, set TSC_AUX in the host save area to the > current value in hardware, as tracked by the user return infrastructure, > instead of always loading the host's desired value for the CPU. If the > pCPU is also running a non-SEV-ES vCPU, loading the hosts value on #VMEXIT > could clobber the other vCPU's value, e.g. if the SEV-ES vCPU preempted > the non-SEV-ES vCPU. > > Note, unlike TDX, which blindly _zeroes_ TSC_AUX on exit, SEV-ES CPUs > can load an arbitrary value. Stuff the current value in the host save > area instead of refreshing the user return cache so that KVM doesn't need > to track whether or not the vCPU actually enterred the guest and thus > loaded TSC_AUX from the host save area. > > Fixes: 916e3e5f26ab ("KVM: SVM: Do not use user return MSR support for virtualized TSC_AUX") > Cc: stable@vger.kernel.org > Suggested-by: Lai Jiangshan > Signed-off-by: Hou Wenlong > [sean: handle the SEV-ES case in sev_es_prepare_switch_to_guest()] > Signed-off-by: Sean Christopherson > --- > arch/x86/kvm/svm/sev.c | 14 +++++++++++++- > arch/x86/kvm/svm/svm.c | 26 +++++++------------------- > arch/x86/kvm/svm/svm.h | 4 +++- > 3 files changed, 23 insertions(+), 21 deletions(-) > > diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c > index cce48fff2e6c..95767b9d0d55 100644 > --- a/arch/x86/kvm/svm/sev.c > +++ b/arch/x86/kvm/svm/sev.c > @@ -4664,7 +4664,9 @@ int sev_vcpu_create(struct kvm_vcpu *vcpu) > return 0; > } > > -void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_save_area *hostsa) > +void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, > + struct sev_es_save_area *hostsa, > + int tsc_aux_uret_slot) > { > struct kvm *kvm = svm->vcpu.kvm; > > @@ -4712,6 +4714,16 @@ void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_save_are > hostsa->dr2_addr_mask = amd_get_dr_addr_mask(2); > hostsa->dr3_addr_mask = amd_get_dr_addr_mask(3); > } > + > + /* > + * TSC_AUX is always virtualized for SEV-ES guests when the feature is > + * available, i.e. TSC_AUX is loaded on #VMEXIT from the host save area. > + * Set the save area to the current hardware value, i.e. the current > + * user return value, so that the correct value is restored on #VMEXIT. > + */ > + if (cpu_feature_enabled(X86_FEATURE_V_TSC_AUX) && > + !WARN_ON_ONCE(tsc_aux_uret_slot < 0)) > + hostsa->tsc_aux = kvm_get_user_return_msr(tsc_aux_uret_slot); > } > > void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) > diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c > index 67f4eed01526..662cf680faf7 100644 > --- a/arch/x86/kvm/svm/svm.c > +++ b/arch/x86/kvm/svm/svm.c > @@ -577,18 +577,6 @@ static int svm_enable_virtualization_cpu(void) > > amd_pmu_enable_virt(); > > - /* > - * If TSC_AUX virtualization is supported, TSC_AUX becomes a swap type > - * "B" field (see sev_es_prepare_switch_to_guest()) for SEV-ES guests. > - * Since Linux does not change the value of TSC_AUX once set, prime the > - * TSC_AUX field now to avoid a RDMSR on every vCPU run. > - */ > - if (boot_cpu_has(X86_FEATURE_V_TSC_AUX)) { > - u32 __maybe_unused msr_hi; > - > - rdmsr(MSR_TSC_AUX, sev_es_host_save_area(sd)->tsc_aux, msr_hi); > - } > - > return 0; > } > > @@ -1400,16 +1388,17 @@ static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu) > */ > vmsave(sd->save_area_pa); > if (sev_es_guest(vcpu->kvm)) > - sev_es_prepare_switch_to_guest(svm, sev_es_host_save_area(sd)); > + sev_es_prepare_switch_to_guest(svm, sev_es_host_save_area(sd), > + tsc_aux_uret_slot); > > if (tsc_scaling) > __svm_write_tsc_multiplier(vcpu->arch.tsc_scaling_ratio); > > /* > - * TSC_AUX is always virtualized for SEV-ES guests when the feature is > - * available. The user return MSR support is not required in this case > - * because TSC_AUX is restored on #VMEXIT from the host save area > - * (which has been initialized in svm_enable_virtualization_cpu()). > + * TSC_AUX is always virtualized (context switched by hardware) for > + * SEV-ES guests when the feature is available. For non-SEV-ES guests, > + * context switch TSC_AUX via the user_return MSR infrastructure (not > + * all CPUs support TSC_AUX virtualization). > */ > if (likely(tsc_aux_uret_slot >= 0) && > (!boot_cpu_has(X86_FEATURE_V_TSC_AUX) || !sev_es_guest(vcpu->kvm))) > @@ -3004,8 +2993,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) > * TSC_AUX is always virtualized for SEV-ES guests when the > * feature is available. The user return MSR support is not > * required in this case because TSC_AUX is restored on #VMEXIT > - * from the host save area (which has been initialized in > - * svm_enable_virtualization_cpu()). > + * from the host save area. > */ > if (boot_cpu_has(X86_FEATURE_V_TSC_AUX) && sev_es_guest(vcpu->kvm)) > break; > diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h > index 5d39c0b17988..4fda677e8ab3 100644 > --- a/arch/x86/kvm/svm/svm.h > +++ b/arch/x86/kvm/svm/svm.h > @@ -831,7 +831,9 @@ void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm); > int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in); > void sev_es_recalc_msr_intercepts(struct kvm_vcpu *vcpu); > void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); > -void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_save_area *hostsa); > +void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, > + struct sev_es_save_area *hostsa, > + int tsc_aux_uret_slot); > void sev_es_unmap_ghcb(struct vcpu_svm *svm); > > #ifdef CONFIG_KVM_AMD_SEV > > base-commit: 60e396349c19320485a249005256d1fafee60290 > --