From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CFEAC28D13 for ; Mon, 22 Aug 2022 16:22:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236596AbiHVQV6 (ORCPT ); Mon, 22 Aug 2022 12:21:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235896AbiHVQV4 (ORCPT ); Mon, 22 Aug 2022 12:21:56 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FB7A3F313 for ; Mon, 22 Aug 2022 09:21:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1661185314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bj0u0rbGbus5Dq/FJao2yGhu5YXJhcLCBMsadHoSqdo=; b=FSi65sizTpV7Lr056g0sRhUug+GFxj11LWv4/JuJGT69C3+e69tXwJdapDLfqYLve1L1pS VBSK3VX/RXvcYx//tyzvw4uZAVVzFck2nQKw7sploKm3EBulobbHXlAhjjfq4AK9r9LBpN pjAYJCrMG+/OkoatwZ1w9gyFt3OqxQ4= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-114-RXN8aUEOOVCQ6cKnwWJktA-1; Mon, 22 Aug 2022 12:21:53 -0400 X-MC-Unique: RXN8aUEOOVCQ6cKnwWJktA-1 Received: by mail-wr1-f70.google.com with SMTP id l25-20020adfa399000000b002252058bad2so1852096wrb.11 for ; Mon, 22 Aug 2022 09:21:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc; bh=bj0u0rbGbus5Dq/FJao2yGhu5YXJhcLCBMsadHoSqdo=; b=UisqL7LQ0P/X8+2t0Yz8Or27Brb0QkNhd1BRH8wP0u/bxE2z3qUOmmJKzz+sWFk2/I b8+jy9blixpXOj8oryAvZ9jdjAqodhjeBF/k3y6h8aT6+rBfOrEirDPZNGDKDJ8DVj6l T89uHsjCMEQYjiAZs4xbhM1Y/8OuXlclnOlOzcAXlTC6q2MoCf/VhP0Ck9eD9lF6SFP7 W9hfctfjRu3kR33+tfwvOupOigF5q9dw0s2q5hXLL/Ush4vL/a6x1tDDPSuSiaFGL6Y4 PPeGbxdk5PeXEBhTQnQ9Vs/rTZqqaUHmxdPsddD4hp9VvcftshnULzLvi9mDzl8f48Am idig== X-Gm-Message-State: ACgBeo39FIw7yVxvGKqqZ2ZnKD/almTF9OxKKlrtZfBAJUR4lPGXZt37 12ckZLDAt3sxJa2rBdvrhFDroc61/vQ59aPvsRH0IhgW8a/nZC/GgJcOlY/Ige/bux1JAokN8Gh NvWJSXGWXRQzy5AmfMrFeNwWl8htfIqTtu+/8VS5t6uzdpkAkY7I48ZbReYkIVX7sYMAmIQ48Sx 3P X-Received: by 2002:a05:600c:3c9b:b0:3a6:58b2:b80 with SMTP id bg27-20020a05600c3c9b00b003a658b20b80mr5223071wmb.132.1661185312255; Mon, 22 Aug 2022 09:21:52 -0700 (PDT) X-Google-Smtp-Source: AA6agR7Kn2mEJXNUBEvtI+h/P2RDMUSo7Laz1MIfn2LUJ5bhNZYgLeYAwA6bjpOc6ng8CeUauHnPPw== X-Received: by 2002:a05:600c:3c9b:b0:3a6:58b2:b80 with SMTP id bg27-20020a05600c3c9b00b003a658b20b80mr5223047wmb.132.1661185311985; Mon, 22 Aug 2022 09:21:51 -0700 (PDT) Received: from fedora (nat-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id g17-20020a5d46d1000000b0020fff0ea0a3sm11988449wrs.116.2022.08.22.09.21.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Aug 2022 09:21:51 -0700 (PDT) From: Vitaly Kuznetsov To: Sean Christopherson Cc: kvm@vger.kernel.org, Paolo Bonzini , Anirudh Rayabharam , Wanpeng Li , Jim Mattson , Maxim Levitsky , Nathan Chancellor , Michael Kelley , linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 03/26] x86/hyperv: Update 'struct hv_enlightened_vmcs' definition In-Reply-To: References: <20220802160756.339464-1-vkuznets@redhat.com> <20220802160756.339464-4-vkuznets@redhat.com> <875yiptvsc.fsf@redhat.com> <87czcsskkj.fsf@redhat.com> Date: Mon, 22 Aug 2022 18:21:50 +0200 Message-ID: <87edx8xn8h.fsf@redhat.com> MIME-Version: 1.0 Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Sean Christopherson writes: > On Mon, Aug 22, 2022, Vitaly Kuznetsov wrote: >> Sean Christopherson writes: >> >> > On Thu, Aug 18, 2022, Vitaly Kuznetsov wrote: >> >> Sean Christopherson writes: >> >> >> >> > On Tue, Aug 02, 2022, Vitaly Kuznetsov wrote: >> >> >> + * Note: HV_X64_NESTED_EVMCS1_2022_UPDATE is not currently documented in any >> >> >> + * published TLFS version. When the bit is set, nested hypervisor can use >> >> >> + * 'updated' eVMCSv1 specification (perf_global_ctrl, s_cet, ssp, lbr_ctl, >> >> >> + * encls_exiting_bitmap, tsc_multiplier fields which were missing in 2016 >> >> >> + * specification). >> >> >> + */ >> >> >> +#define HV_X64_NESTED_EVMCS1_2022_UPDATE BIT(0) >> >> > >> >> > This bit is now defined[*], but the docs says it's only for perf_global_ctrl. Are >> >> > we expecting an update to the TLFS? >> >> > >> >> > Indicates support for the GuestPerfGlobalCtrl and HostPerfGlobalCtrl fields >> >> > in the enlightened VMCS. >> >> > >> >> > [*] https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/feature-discovery#hypervisor-nested-virtualization-features---0x4000000a >> >> > >> >> >> >> Oh well, better this than nothing. I'll ping the people who told me >> >> about this bit that their description is incomplete. >> > >> > Not that it changes anything, but I'd rather have no documentation. I'd much rather >> > KVM say "this is the undocumented behavior" than "the document behavior is wrong". >> > >> >> So I reached out to Microsoft and their answer was that for all these new >> eVMCS fields (including *PerfGlobalCtrl) observing architectural VMX >> MSRs should be enough. *PerfGlobalCtrl case is special because of Win11 >> bug (if we expose the feature in VMX feature MSRs but don't set >> CPUID.0x4000000A.EBX BIT(0) it just doesn't boot). > > I.e. TSC_SCALING shouldn't be gated on the flag? If so, then the 2-D array approach > is overkill since (a) the CPUID flag only controls PERF_GLOBAL_CTRL and (b) we aren't > expecting any more flags in the future. > Unfortunately, we have to gate the presence of these new features on something, otherwise VMM has no way to specify which particular eVMCS "revision" it wants (TL;DR: we will break migration). My initial implementation was inventing 'eVMCS revision' concept: https://lore.kernel.org/kvm/20220629150625.238286-7-vkuznets@redhat.com/ which is needed if we don't gate all these new fields on CPUID.0x4000000A.EBX BIT(0). Going forward, we will still (likely) need something when new fields show up. > What about this for an implementation? > > static bool evmcs_has_perf_global_ctrl(struct kvm_vcpu *vcpu) > { > struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); > > /* > * Filtering VMX controls for eVMCS compatibility should only be done > * for guest accesses, and all such accesses should be gated on Hyper-V > * being enabled and initialized. > */ > if (WARN_ON_ONCE(!hv_vcpu)) > return false; > > return hv_vcpu->cpuid_cache.nested_ebx & HV_X64_NESTED_EVMCS1_PERF_GLOBAL_CTRL; > } > > static u32 evmcs_get_unsupported_ctls(struct kvm_vcpu *vcpu, u32 msr_index) > { > u32 unsupported_ctrls; > > switch (msr_index) { > case MSR_IA32_VMX_EXIT_CTLS: > case MSR_IA32_VMX_TRUE_EXIT_CTLS: > unsupported_ctrls = EVMCS1_UNSUPPORTED_VMEXIT_CTRL; > if (!evmcs_has_perf_global_ctrl(vcpu)) > unsupported_ctrls |= VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL; > return unsupported_ctrls; > case MSR_IA32_VMX_ENTRY_CTLS: > case MSR_IA32_VMX_TRUE_ENTRY_CTLS: > unsupported_ctrls = EVMCS1_UNSUPPORTED_VMENTRY_CTRL; > if (!evmcs_has_perf_global_ctrl(vcpu)) > unsupported_ctrls |= VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; > return unsupported_ctrls; > case MSR_IA32_VMX_PROCBASED_CTLS2: > return EVMCS1_UNSUPPORTED_2NDEXEC; > case MSR_IA32_VMX_TRUE_PINBASED_CTLS: > case MSR_IA32_VMX_PINBASED_CTLS: > return EVMCS1_UNSUPPORTED_PINCTRL; > case MSR_IA32_VMX_VMFUNC: > return EVMCS1_UNSUPPORTED_VMFUNC; > default: > KVM_BUG_ON(1, vcpu->kvm); > return 0; > } > } > > void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata) > { > u64 unsupported_ctrls = evmcs_get_unsupported_ctls(vcpu, msr_index); > > if (msr_index == MSR_IA32_VMX_VMFUNC) > *pdata &= ~unsupported_ctrls; > else > *pdata &= ~(unsupported_ctrls << 32); > } > It's smaller and I like it but it would only work in conjunction with KVM_CAP_HYPERV_ENLIGHTENED_VMCS2... > >> What I'm still concerned about is future proofing KVM for new >> features. When something is getting added to KVM for which no eVMCS >> field is currently defined, both Hyper-V-on-KVM and KVM-on-Hyper-V cases >> should be taken care of. It would probably be better to reverse our >> filtering, explicitly listing features supported in eVMCS. The lists are >> going to be fairly long but at least we won't have to take care of any >> new architectural feature added to KVM. > > Having the filtering be opt-in crossed my mind as well. Reversing the filtering > can be done after this series though, correct? > Yes, that's my plan, Get this in to fix the immediate issue with 2022 features and probably reverse the filtering before Microsoft releases something else :-) -- Vitaly