From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12EA9C433B4 for ; Thu, 1 Apr 2021 23:38:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E4404610A5 for ; Thu, 1 Apr 2021 23:38:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236604AbhDAXiy (ORCPT ); Thu, 1 Apr 2021 19:38:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236348AbhDAXiZ (ORCPT ); Thu, 1 Apr 2021 19:38:25 -0400 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30DBFC061794 for ; Thu, 1 Apr 2021 16:38:25 -0700 (PDT) Received: by mail-qv1-xf4a.google.com with SMTP id a7so4293641qvx.10 for ; Thu, 01 Apr 2021 16:38:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=YE6UzfKVCMMZ4FmPHxQVQ73AQ/1CyoycUg8d9T2zy44=; b=CwnbxgidzNPuXnwFCcwLc0dnCKpIJlWSx1AvsEh15lhiQbUlAUFFLuyeIx9jEcbIR2 NfOHt06yFwWyOmyqV5T7OW0pBvdkJRe+RlSIHOGgcCPK99OVVJbyApcCrHsIh+kLN7xj TT9s35RzNt+9zAi/s7zCnMZTKKaK7utN/gNVvLImzTxzQG1MV6NlAVpNtPpU7V+Pb8Jh o/lM7PAKtPi9+d9o1Q1r6OOs9kcFJMFXFUqZaJBYRRDWyGjBQS32zdwfHwWn4EHsG61b S7VwuTmol+jhNH0mHHXAabMT1zGiSD/dRAO8MlweJSxGrhZXOStfm9RIf5zjlAqxziuc q8zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=YE6UzfKVCMMZ4FmPHxQVQ73AQ/1CyoycUg8d9T2zy44=; b=f44BZVFR0C0Vvpx2szdSMN0ewj7gagOQ6+iziDalgvrf6IpJQQhNujn8OWWn8+AIpZ 1Soagapq/8XHv5Mt2iBFTbMwlDHUoVIw/QIpPAxCtsZlJzIpvtD69PI2b8chJgT5sFlN DE243gegVJZXc655v7/Yc0jyArcqdZssD0MPUaKird2phhWXNMKjiRELdYwf7Hs6XC2p Wl9MgNAU3un/5Cg8SBB40upiPmOgVjV+Ecqtlz7JOKhyK74pAgvrvb6+oSxdR36FF6oP 4P4tVXOzUNdi9a4Z/13Wx6RtADNGse2ec+t/5ygZ0K2dsWKbMjiiszhoLpTRb20L7uuP o6Sw== X-Gm-Message-State: AOAM530TvE8jsdnhoxQzfZtyrZQxSJ+k23cn/ZJDINjXIqEmimhuHzTU Y+suRPB4RVnu/NhIevf1HM+ODWNMP6zJv+NNs/WW3cnuWBAhY1F1+IlbGWJi/y9cWGnbCAD+hDr tZOZg+Pjm4jG4MCYhjreV7k13ER80j1WQzHTyeaQuSHv+KANkyNvfRO3z/weSxDoQdi1UYH/1 X-Google-Smtp-Source: ABdhPJxCEndlAJ+pZNKpeF2guBvvAQXBNGg4bSKL89RTDPSsU7uP0y6Y5yqnEY78od/usyul99Ss6XBxrcV6 X-Received: from bgardon.sea.corp.google.com ([2620:15c:100:202:e088:88b8:ea4a:22b6]) (user=bgardon job=sendgmr) by 2002:a0c:bd2f:: with SMTP id m47mr10623557qvg.53.1617320304345; Thu, 01 Apr 2021 16:38:24 -0700 (PDT) Date: Thu, 1 Apr 2021 16:37:35 -0700 In-Reply-To: <20210401233736.638171-1-bgardon@google.com> Message-Id: <20210401233736.638171-13-bgardon@google.com> Mime-Version: 1.0 References: <20210401233736.638171-1-bgardon@google.com> X-Mailer: git-send-email 2.31.0.208.g409f899ff0-goog Subject: [PATCH v2 12/13] KVM: x86/mmu: Fast invalidation for TDP MMU From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , Peter Shier , Peter Feiner , Junaid Shahid , Jim Mattson , Yulei Zhang , Wanpeng Li , Vitaly Kuznetsov , Xiao Guangrong , Ben Gardon Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Provide a real mechanism for fast invalidation by marking roots as invalid so that their reference count will quickly fall to zero and they will be torn down. One negative side affect of this approach is that a vCPU thread will likely drop the last reference to a root and be saddled with the work of tearing down an entire paging structure. This issue will be resolved in a later commit. Signed-off-by: Ben Gardon --- Changelog v2: -- open code root invalidation arch/x86/kvm/mmu/mmu.c | 26 +++++++++++++++++++++++--- arch/x86/kvm/mmu/tdp_mmu.h | 3 +++ 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a3837f8ad4ed..ba0c65076200 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5418,6 +5418,8 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm) */ static void kvm_mmu_zap_all_fast(struct kvm *kvm) { + struct kvm_mmu_page *root; + lockdep_assert_held(&kvm->slots_lock); write_lock(&kvm->mmu_lock); @@ -5432,6 +5434,27 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) */ kvm->arch.mmu_valid_gen = kvm->arch.mmu_valid_gen ? 0 : 1; + + if (is_tdp_mmu_enabled(kvm)) { + /* + * Mark each TDP MMU root as invalid so that other threads + * will drop their references and allow the root count to + * go to 0. + * + * This has essentially the same effect for the TDP MMU + * as updating mmu_valid_gen above does for the shadow + * MMU. + * + * In order to ensure all threads see this change when + * handling the MMU reload signal, this must happen in the + * same critical section as kvm_reload_remote_mmus, and + * before kvm_zap_obsolete_pages as kvm_zap_obsolete_pages + * could drop the MMU lock and yield. + */ + list_for_each_entry(root, &kvm->arch.tdp_mmu_roots, link) + root->role.invalid = true; + } + /* * Notify all vcpus to reload its shadow page table and flush TLB. * Then all vcpus will switch to new shadow page table with the new @@ -5444,9 +5467,6 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) kvm_zap_obsolete_pages(kvm); - if (is_tdp_mmu_enabled(kvm)) - kvm_tdp_mmu_zap_all(kvm); - write_unlock(&kvm->mmu_lock); } diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index d703c6d6024a..8fa3e7421a93 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -10,6 +10,9 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu); __must_check static inline bool kvm_tdp_mmu_get_root(struct kvm *kvm, struct kvm_mmu_page *root) { + if (root->role.invalid) + return false; + return refcount_inc_not_zero(&root->tdp_mmu_root_count); } -- 2.31.0.208.g409f899ff0-goog