From: Paolo Bonzini <pbonzini@redhat.com>
To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org
Cc: bgardon@google.com
Subject: [PATCH 22/22] kvm: x86/mmu: NX largepage recovery for TDP MMU
Date: Fri, 23 Oct 2020 12:30:24 -0400 [thread overview]
Message-ID: <20201023163024.2765558-23-pbonzini@redhat.com> (raw)
In-Reply-To: <20201023163024.2765558-1-pbonzini@redhat.com>
From: Ben Gardon <bgardon@google.com>
When KVM maps a largepage backed region at a lower level in order to
make it executable (i.e. NX large page shattering), it reduces the TLB
performance of that region. In order to avoid making this degradation
permanent, KVM must periodically reclaim shattered NX largepages by
zapping them and allowing them to be rebuilt in the page fault handler.
With this patch, the TDP MMU does not respect KVM's rate limiting on
reclaim. It traverses the entire TDP structure every time. This will be
addressed in a future patch.
Tested by running kvm-unit-tests and KVM selftests on an Intel Haswell
machine. This series introduced no new failures.
This series can be viewed in Gerrit at:
https://linux-review.googlesource.com/c/virt/kvm/kvm/+/2538
Signed-off-by: Ben Gardon <bgardon@google.com>
Message-Id: <20201014182700.2888246-21-bgardon@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
arch/x86/kvm/mmu/mmu.c | 13 +++++++++----
arch/x86/kvm/mmu/mmu_internal.h | 3 +++
arch/x86/kvm/mmu/tdp_mmu.c | 6 ++++++
3 files changed, 18 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 7b52fa1f01b0..17587f496ec7 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -776,7 +776,7 @@ static void account_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
kvm_mmu_gfn_disallow_lpage(slot, gfn);
}
-static void account_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp)
+void account_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp)
{
if (sp->lpage_disallowed)
return;
@@ -804,7 +804,7 @@ static void unaccount_shadowed(struct kvm *kvm, struct kvm_mmu_page *sp)
kvm_mmu_gfn_allow_lpage(slot, gfn);
}
-static void unaccount_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp)
+void unaccount_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp)
{
--kvm->stat.nx_lpage_splits;
sp->lpage_disallowed = false;
@@ -5988,8 +5988,13 @@ static void kvm_recover_nx_lpages(struct kvm *kvm)
struct kvm_mmu_page,
lpage_disallowed_link);
WARN_ON_ONCE(!sp->lpage_disallowed);
- kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
- WARN_ON_ONCE(sp->lpage_disallowed);
+ if (sp->tdp_mmu_page)
+ kvm_tdp_mmu_zap_gfn_range(kvm, sp->gfn,
+ sp->gfn + KVM_PAGES_PER_HPAGE(sp->role.level));
+ else {
+ kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
+ WARN_ON_ONCE(sp->lpage_disallowed);
+ }
if (need_resched() || spin_needbreak(&kvm->mmu_lock)) {
kvm_mmu_commit_zap_page(kvm, &invalid_list);
diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h
index 6db40ea85974..bfc6389edc28 100644
--- a/arch/x86/kvm/mmu/mmu_internal.h
+++ b/arch/x86/kvm/mmu/mmu_internal.h
@@ -143,4 +143,7 @@ bool is_nx_huge_page_enabled(void);
void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
+void account_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp);
+void unaccount_huge_nx_page(struct kvm *kvm, struct kvm_mmu_page *sp);
+
#endif /* __KVM_X86_MMU_INTERNAL_H */
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 5158d02b8925..e246d71b8ea2 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -273,6 +273,9 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,
list_del(&sp->link);
+ if (sp->lpage_disallowed)
+ unaccount_huge_nx_page(kvm, sp);
+
for (i = 0; i < PT64_ENT_PER_PAGE; i++) {
old_child_spte = READ_ONCE(*(pt + i));
WRITE_ONCE(*(pt + i), 0);
@@ -571,6 +574,9 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
!shadow_accessed_mask);
trace_kvm_mmu_get_page(sp, true);
+ if (huge_page_disallowed && req_level >= iter.level)
+ account_huge_nx_page(vcpu->kvm, sp);
+
tdp_mmu_set_spte(vcpu->kvm, &iter, new_spte);
}
}
--
2.26.2
prev parent reply other threads:[~2020-10-23 16:30 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-23 16:30 [PATCH 00/22] Introduce the " Paolo Bonzini
2020-10-23 16:30 ` [PATCH 01/22] kvm: mmu: Separate making non-leaf sptes from link_shadow_page Paolo Bonzini
2020-10-23 16:30 ` [PATCH 02/22] kvm: x86/mmu: Separate making SPTEs from set_spte Paolo Bonzini
2020-10-23 16:30 ` [PATCH 03/22] KVM: mmu: Separate updating a PTE from kvm_set_pte_rmapp Paolo Bonzini
2020-10-23 16:30 ` [PATCH 04/22] KVM: mmu: extract spte.h and spte.c Paolo Bonzini
2020-10-27 14:46 ` Valdis Klētnieks
2020-10-28 17:36 ` Paolo Bonzini
2020-10-23 16:30 ` [PATCH 05/22] kvm: x86/mmu: Introduce tdp_iter Paolo Bonzini
2020-10-23 16:30 ` [PATCH 06/22] kvm: x86/mmu: Init / Uninit the TDP MMU Paolo Bonzini
2020-10-23 16:30 ` [PATCH 07/22] kvm: x86/mmu: Allocate and free TDP MMU roots Paolo Bonzini
2020-10-23 16:30 ` [PATCH 08/22] kvm: x86/mmu: Add functions to handle changed TDP SPTEs Paolo Bonzini
2020-10-23 16:30 ` [PATCH 09/22] KVM: Cache as_id in kvm_memory_slot Paolo Bonzini
2020-10-23 16:30 ` [PATCH 10/22] kvm: x86/mmu: Support zapping SPTEs in the TDP MMU Paolo Bonzini
2020-10-23 16:30 ` [PATCH 11/22] kvm: x86/mmu: Remove disallowed_hugepage_adjust shadow_walk_iterator arg Paolo Bonzini
2020-10-23 16:30 ` [PATCH 12/22] kvm: x86/mmu: Add TDP MMU PF handler Paolo Bonzini
2020-10-23 16:30 ` [PATCH 13/22] kvm: x86/mmu: Allocate struct kvm_mmu_pages for all pages in TDP MMU Paolo Bonzini
2020-10-23 16:30 ` [PATCH 14/22] kvm: x86/mmu: Support invalidate range MMU notifier for " Paolo Bonzini
2020-10-23 16:30 ` [PATCH 15/22] kvm: x86/mmu: Add access tracking for tdp_mmu Paolo Bonzini
2020-10-23 16:30 ` [PATCH 16/22] kvm: x86/mmu: Support changed pte notifier in tdp MMU Paolo Bonzini
2020-10-23 16:30 ` [PATCH 17/22] kvm: x86/mmu: Support dirty logging for the TDP MMU Paolo Bonzini
2020-10-23 16:30 ` [PATCH 18/22] kvm: x86/mmu: Support disabling dirty logging for the tdp MMU Paolo Bonzini
2020-10-23 16:30 ` [PATCH 19/22] kvm: x86/mmu: Support write protection for nesting in " Paolo Bonzini
2020-10-23 16:30 ` [PATCH 20/22] kvm: x86/mmu: Support MMIO in the TDP MMU Paolo Bonzini
2020-10-23 16:30 ` [PATCH 21/22] kvm: x86/mmu: Don't clear write flooding count for direct roots Paolo Bonzini
2020-10-23 16:30 ` Paolo Bonzini [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201023163024.2765558-23-pbonzini@redhat.com \
--to=pbonzini@redhat.com \
--cc=bgardon@google.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®