From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3757CC433F5 for ; Mon, 13 Sep 2021 11:34:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1D5B360F44 for ; Mon, 13 Sep 2021 11:34:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239671AbhIMLfX (ORCPT ); Mon, 13 Sep 2021 07:35:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:36424 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239504AbhIMLfV (ORCPT ); Mon, 13 Sep 2021 07:35:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631532846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pC86/tdCjsTEO9k3s80UA5iBYjkl8q26l2+GsC2jp0o=; b=QaPT19q6YXE8hfnSqMoKIb5IcPIEg1rH+/5C7rARRnxlh++srCf702bFAk+W64z9KRlgpU 4AtFFmg0t9zRj2hsEwTV4V18gmfLPv+x5tjBJ1Q9VH5fSrBWRkBe7r9KF3wm2hhAaaxBvd MJXDHgrxc8SrAAxTvUAT2IIw8oEyRTU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-150-DC818uQJMWaMnvRzT_rLRA-1; Mon, 13 Sep 2021 07:34:03 -0400 X-MC-Unique: DC818uQJMWaMnvRzT_rLRA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E658280196C; Mon, 13 Sep 2021 11:34:01 +0000 (UTC) Received: from starship (unknown [10.35.206.50]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D4C760BE5; Mon, 13 Sep 2021 11:33:59 +0000 (UTC) Message-ID: <8c9b9040ba27ba0961678cb228199fc41bdcbc74.camel@redhat.com> Subject: Re: [PATCH 1/4] KVM: nVMX: Don't use Enlightened MSR Bitmap for L3 From: Maxim Levitsky To: Vitaly Kuznetsov , kvm@vger.kernel.org, Paolo Bonzini Cc: Sean Christopherson , Wanpeng Li , Jim Mattson , linux-kernel@vger.kernel.org Date: Mon, 13 Sep 2021 14:33:58 +0300 In-Reply-To: <87lf412cgi.fsf@vitty.brq.redhat.com> References: <20210910160633.451250-1-vkuznets@redhat.com> <20210910160633.451250-2-vkuznets@redhat.com> <6424b309216b276e46a66573320b3eed8209a0ed.camel@redhat.com> <87lf412cgi.fsf@vitty.brq.redhat.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2021-09-13 at 08:53 +0200, Vitaly Kuznetsov wrote: > Maxim Levitsky writes: > > > On Fri, 2021-09-10 at 18:06 +0200, Vitaly Kuznetsov wrote: > > > When KVM runs as a nested hypervisor on top of Hyper-V it uses Enlightened > > > VMCS and enables Enlightened MSR Bitmap feature for its L1s and L2s (which > > > are actually L2s and L3s from Hyper-V's perspective). When MSR bitmap is > > > updated, KVM has to reset HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP from > > > clean fields to make Hyper-V aware of the change. For KVM's L1s, this is > > > done in vmx_disable_intercept_for_msr()/vmx_enable_intercept_for_msr(). > > > MSR bitmap for L2 is build in nested_vmx_prepare_msr_bitmap() by blending > > > MSR bitmap for L1 and L1's idea of MSR bitmap for L2. KVM, however, doesn't > > > check if the resulting bitmap is different and never cleans > > > HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP in eVMCS02. This is incorrect and > > > may result in Hyper-V missing the update. > > > > > > The issue could've been solved by calling evmcs_touch_msr_bitmap() for > > > eVMCS02 from nested_vmx_prepare_msr_bitmap() unconditionally but doing so > > > would not give any performance benefits (compared to not using Enlightened > > > MSR Bitmap at all). 3-level nesting is also not a very common setup > > > nowadays. > > > > > > Don't enable 'Enlightened MSR Bitmap' feature for KVM's L2s (real L3s) for > > > now. > > > > > > Signed-off-by: Vitaly Kuznetsov > > > --- > > > arch/x86/kvm/vmx/vmx.c | 22 +++++++++++++--------- > > > 1 file changed, 13 insertions(+), 9 deletions(-) > > > > > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > > > index 0c2c0d5ae873..ae470afcb699 100644 > > > --- a/arch/x86/kvm/vmx/vmx.c > > > +++ b/arch/x86/kvm/vmx/vmx.c > > > @@ -2654,15 +2654,6 @@ int alloc_loaded_vmcs(struct loaded_vmcs *loaded_vmcs) > > > if (!loaded_vmcs->msr_bitmap) > > > goto out_vmcs; > > > memset(loaded_vmcs->msr_bitmap, 0xff, PAGE_SIZE); > > > - > > > - if (IS_ENABLED(CONFIG_HYPERV) && > > > - static_branch_unlikely(&enable_evmcs) && > > > - (ms_hyperv.nested_features & HV_X64_NESTED_MSR_BITMAP)) { > > > - struct hv_enlightened_vmcs *evmcs = > > > - (struct hv_enlightened_vmcs *)loaded_vmcs->vmcs; > > > - > > > - evmcs->hv_enlightenments_control.msr_bitmap = 1; > > > - } > > > } > > > > > > memset(&loaded_vmcs->host_state, 0, sizeof(struct vmcs_host_state)); > > > @@ -6861,6 +6852,19 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu) > > > } > > > > > > vmx->loaded_vmcs = &vmx->vmcs01; > > > + > > > + /* > > > + * Use Hyper-V 'Enlightened MSR Bitmap' feature when KVM runs as a > > > + * nested (L1) hypervisor and Hyper-V in L0 supports it. > > > + */ > > > + if (IS_ENABLED(CONFIG_HYPERV) && static_branch_unlikely(&enable_evmcs) > > > + && (ms_hyperv.nested_features & HV_X64_NESTED_MSR_BITMAP)) { > > > + struct hv_enlightened_vmcs *evmcs = > > > + (struct hv_enlightened_vmcs *)vmx->loaded_vmcs->vmcs; > > > + > > > + evmcs->hv_enlightenments_control.msr_bitmap = 1; > > > + } > > > + > > > cpu = get_cpu(); > > > vmx_vcpu_load(vcpu, cpu); > > > vcpu->cpu = cpu; > > > > Makes sense. > > > > Reviewed-by: Maxim Levitsky > > > > > > However, just a note that it is very very confusing that KVM can use eVMCS in both ways. > > > > > > 'Client': It can both run under HyperV, and thus take advantage of eVMCS when it runs its guests (with > > help of > > HyperV) > > > > 'Server' KVM can emulate some HyperV features, and one of these is eVMCS, thus a windows guest running > > under KVM, can use KVM's eVMCS implementation to run nested guests. > > > > This patch fails under > > 'Client', while the other patches in the series fall under the 'Server' category, > > and even more confusing, the patch 2 moves 'Client' code around, but it is intended for following patches > > 3,4 which are > > for Server. > > > > All this is confusing indeed, KVM-on-Hyper-V and Hyper-V-on-KVM are two > different beasts but it's not always clear from patch subject. I was > thinking about adding this to patch prexes: > > "KVM: VMX: KVM-on-Hyper-V: ... " > "KVM: nVMX: Hyper-V-on-KVM ..." Makes sense! > > or something similar. > > > Thus this patch probably should be a separate patch, just to avoid confusion. > > > > This patch is a weird one. We actually fix > > Hyper-V-on-KVM-on-Hyper-V case. > > Don't get confused! :-) Ah right! This reminds me of these double windows with an air gap in between two glass layers :-) Best regards, Maxim Levitsky > > > > However, since this patch series is already posted, and I figured that out, and hopefully explained it here, > > no need to do anything though! > > > > > > Best regards, > > Maxim Levitsky > > > > > >