From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDD64C001E0 for ; Sat, 29 Jul 2023 14:54:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230309AbjG2Oyn (ORCPT ); Sat, 29 Jul 2023 10:54:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231150AbjG2Oyk (ORCPT ); Sat, 29 Jul 2023 10:54:40 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A7A435B6 for ; Sat, 29 Jul 2023 07:53:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1690642424; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nq//HWsZ0YlNk9f1keqXb51bZ7c40xck1LtIyJE5/ZQ=; b=EIJviEFdoQbxWCskump2U0qJhagZtaiSBxIyAFyZ9VVmIGTWDsPi+t6Gw/NFusgHVYiSQC d2t+OCJQnWHT8JQe4puQadpuZhFI2YPfOiQQYZz6gPQlnKIY9HcVI7cg6PIgzgnSuzC9+q dvmBVttyFAROlw1tqrvfpbunHkZe1G4= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-307-B4LSbax8NyCb8zXg7J2XcQ-1; Sat, 29 Jul 2023 10:53:43 -0400 X-MC-Unique: B4LSbax8NyCb8zXg7J2XcQ-1 Received: by mail-ed1-f69.google.com with SMTP id 4fb4d7f45d1cf-521da4c99d4so2347687a12.1 for ; Sat, 29 Jul 2023 07:53:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690642421; x=1691247221; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nq//HWsZ0YlNk9f1keqXb51bZ7c40xck1LtIyJE5/ZQ=; b=Rw+a2CoWogZzw7z/13pjPAQfEKJj11SgSyf6nnVjiL7NL0Rg0c+YqB2/syzhAgiwLF 2Mw4u8mVh/9ZXTcLMN6foe5dU2sHsjOGFULbGgeDsvtfzCEHgglPemHTQuks8qdFlw1L Yl6Rj9PJAUX/JGRrp4dAP0XNu1L/CIdYZqA9B93EUOgM4gStkS/WC5DZlH8fiY/ZNGwt eh8oEP1CwarPDlQ1kouSeDuPg3gFx1D5nCmIjlMJMVm47f+vd1iP40bdhTFn4HqPeeZf bKDsMsX0+x2DE8KS0vDrzA8Q0Xy9zwP9YE+Fr3TExsDZ8p6Pyh8JCZKs/Op36LcPnEwz zHVg== X-Gm-Message-State: ABy/qLZyxI5+XwH9j0vZ8XE/DT6G6yPWG8RGJglLK3/JsGfemRkZme+/ U3hefkAsdt8UasXQ4T1Y0sS/9dZUPzMjjdUh2hT2WCiSbe/Y1xAqF31vMpkEBkS0pPMatEZXkFZ oxBkfXKiMDZYsaonYIUd4d5Lsy4b+jezU X-Received: by 2002:aa7:c143:0:b0:522:27f1:3c06 with SMTP id r3-20020aa7c143000000b0052227f13c06mr3987608edp.21.1690642421581; Sat, 29 Jul 2023 07:53:41 -0700 (PDT) X-Google-Smtp-Source: APBJJlHcLjwyLS0eEAF2f90zM6l73C9+BegNJtS2/XMqpaqk6VNw0aCeVqXlcfx3eu66X64IzYJ67A== X-Received: by 2002:aa7:c143:0:b0:522:27f1:3c06 with SMTP id r3-20020aa7c143000000b0052227f13c06mr3987596edp.21.1690642421264; Sat, 29 Jul 2023 07:53:41 -0700 (PDT) Received: from ?IPV6:2001:b07:6468:f312:5e2c:eb9a:a8b6:fd3e? ([2001:b07:6468:f312:5e2c:eb9a:a8b6:fd3e]) by smtp.googlemail.com with ESMTPSA id r13-20020aa7d58d000000b005227b065a78sm3015413edq.70.2023.07.29.07.53.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 29 Jul 2023 07:53:40 -0700 (PDT) Message-ID: Date: Sat, 29 Jul 2023 16:53:38 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v2] KVM: x86/tsc: Use calculated tsc_offset before matching the fist vcpu's tsc Content-Language: en-US To: Like Xu , Sean Christopherson Cc: Oliver Upton , kvm@vger.kernel.org, linux-kernel@vger.kernel.org References: <20230724073516.45394-1-likexu@tencent.com> From: Paolo Bonzini In-Reply-To: <20230724073516.45394-1-likexu@tencent.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 7/24/23 09:35, Like Xu wrote: > From: Like Xu > > Avoid using kvm->arch.cur_tsc_offset until tsc sync is actually needed. > > When the vcpu is created for the first time, its tsc is 0, and KVM > calculates its tsc_offset according to kvm_compute_l1_tsc_offset(). > The userspace will then set the first non-zero tsc expected value for the > first guest vcpu, at this time there is no need to play the tsc synchronize > mechanism, the KVM should continue to write tsc_offset based on previously > used kvm_compute_l1_tsc_offset(). > > If the tsc synchronization mechanism is incorrectly applied at this point, > KVM will use the rewritten offset of the kvm->arch.cur_tsc_offset (on > tsc_stable machines) to write tsc_offset, which is not in line with the > expected tsc of the user space. > > Based on the current code, the vcpu's tsc_offset is not configured as > expected, resulting in significant guest service response latency, which > is observed in our production environment. > > Applying the tsc synchronization logic after the vcpu's tsc_generation > and KVM's cur_tsc_generation have completed their first match and started > keeping tracked helps to fix this issue, which also does not break any > existing guest tsc test cases. > > Reported-by: Yong He > Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217423 > Suggested-by: Sean Christopherson > Suggested-by: Oliver Upton > Signed-off-by: Like Xu > --- > V1 -> V2 Changelog: > - Test the 'kvm_vcpu_has_run(vcpu)' proposal; (Sean) > - Test the kvm->arch.user_changed_tsc proposal; (Oliver) > V1: https://lore.kernel.org/kvm/20230629164838.66847-1-likexu@tencent.com/ > > arch/x86/kvm/x86.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c > index a6b9bea62fb8..4724dacea2df 100644 > --- a/arch/x86/kvm/x86.c > +++ b/arch/x86/kvm/x86.c > @@ -2721,7 +2721,7 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 data) > * kvm_clock stable after CPU hotplug > */ > synchronizing = true; > - } else { > + } else if (kvm->arch.nr_vcpus_matched_tsc) { > u64 tsc_exp = kvm->arch.last_tsc_write + > nsec_to_cycles(vcpu, elapsed); > u64 tsc_hz = vcpu->arch.virtual_tsc_khz * 1000LL; > I am not sure, when will kvm->arch.nr_vcpus_matched_tsc ever be nonzero again once a new generation starts? If this patch ever causes synchronizing to be false, matched will also be false when calling __kvm_synchronize_tsc. What was wrong with Oliver's proposed patch? Paolo