From: David Woodhouse <dwmw2@infradead.org>
To: kvm@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org,
rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev,
Paolo Bonzini <pbonzini@redhat.com>,
Sean Christopherson <seanjc@google.com>,
Paul Durrant <paul@xen.org>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
Fred Griffoul <fgriffo@amazon.co.uk>,
"Paul E . McKenney" <paulmck@kernel.org>,
Kunwu Chan <kunwu.chan@linux.dev>,
Kunwu Chan <kunwu.chan@gmail.com>,
syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com,
David Woodhouse <dwmw@amazon.co.uk>
Subject: [PATCH v2 09/20] KVM: x86: Use gfn_to_pfn_cache for steal time / preempted status
Date: Thu, 5 Nov 2026 02:00:51 +0000 [thread overview]
Message-ID: <20261105020104.2507083-10-dwmw2@infradead.org> (raw)
In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org>
From: Carsten Stollmaier <stollmc@amazon.com>
This largely reverts commit 7e2175ebd695 ("KVM: x86: Fix recording of
guest steal time / preempted status"), which dropped the use of the
gfn_to_pfn_cache because it was not integrated with the MMU notifiers
at the time. That shortcoming has long since been addressed, making
the GPC work correctly for this use case.
Aside from cleaning up the last open-coded assembler access to user
addresses and associated explicit asm exception fixups, moving back
to the now-functional GPC also resolves an issue with contention on
the mmap_lock with userfaultfd. The contention issue is as follows:
On vcpu_run, before entering the guest, the update of the steal time
information causes a page-fault if the page is not present. In our
scenario, this gets handled by do_user_addr_fault() and successively
handle_userfault() because the region is registered to that.
Since handle_userfault() uses TASK_INTERRUPTIBLE, it is interruptible
by signals. But do_user_addr_fault() then busy-retries if the pending
signal is non-fatal, which leads to heavy contention of the mmap_lock.
By restoring the use of GPC for accessing the guest steal time, the
contention is avoided and refreshing the GPC happens when the vCPU is
next scheduled.
Since the gfn_to_pfn_cache gives a kernel mapping rather than a
userspace HVA, accesses are now plain C instead of unsafe_put_user()
et al. READ_ONCE()/WRITE_ONCE() and an smp_wmb() before the final
version increment replace the ordering the old inline assembly
provided implicitly.
kvm_steal_time_set_preempted(), called from the scheduler path, just
takes the atomic SRCU read lock (unconditionally safe there) and
bails if the cache is not currently valid: setting the preempted
flag is best-effort anyway — the old code used
copy_to_user_nofault(), which could equally silently fail — and
kvm_arch_vcpu_load() raises KVM_REQ_STEAL_UPDATE, so the cache is
refreshed from task context before the vCPU next enters the guest.
MSR disable (kvm_steal_time_reset()) leaves last_steal alone: it is
the baseline against the vCPU task's monotonic run_delay, and zeroing
it would dump the task's lifetime run_delay into the guest's steal
counter when the MSR is next enabled (e.g. after INIT-SIPI or kexec).
[dwmw2: Port to the SRCU-protected GPC. The original was written
against a gpc->lock rwlock and needed a read_trylock() dance
to be safe from the scheduler path on PREEMPT_RT; the atomic
SRCU read lock is unconditionally safe there, so all of that
goes away.]
Signed-off-by: Carsten Stollmaier <stollmc@amazon.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/msrs.c | 7 +-
arch/x86/kvm/x86.c | 140 ++++++++++++++++----------------
3 files changed, 76 insertions(+), 73 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1ee4b127004d..4b0c18c77771 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -872,7 +872,7 @@ struct kvm_vcpu_arch {
u8 preempted;
u64 msr_val;
u64 last_steal;
- struct gfn_to_hva_cache cache;
+ struct gfn_to_pfn_cache cache;
} st;
u64 l1_tsc_offset;
diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index e91650af44c2..23407f8c441d 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -1728,8 +1728,13 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
vcpu->arch.st.msr_val = data;
- if (!(data & KVM_MSR_ENABLED))
+ if (!(data & KVM_MSR_ENABLED)) {
+ kvm_gpc_deactivate(&vcpu->arch.st.cache);
break;
+ }
+
+ kvm_gpc_activate(&vcpu->arch.st.cache, data & ~KVM_MSR_ENABLED,
+ sizeof(struct kvm_steal_time));
kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index f5f3459ed4a5..321f29789f9f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2058,12 +2058,11 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_service_local_tlb_flush_requests);
static void record_steal_time(struct kvm_vcpu *vcpu)
{
- struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
- struct kvm_steal_time __user *st;
- struct kvm_memslots *slots;
- gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
+ struct gfn_to_pfn_cache *gpc = &vcpu->arch.st.cache;
+ struct kvm_steal_time *st;
u64 steal;
u32 version;
+ int idx;
if (kvm_xen_msr_enabled(vcpu->kvm)) {
kvm_xen_runstate_set_running(vcpu);
@@ -2076,55 +2075,41 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
if (WARN_ON_ONCE(current->mm != vcpu->kvm->mm))
return;
- slots = kvm_memslots(vcpu->kvm);
+ /* We rely on the fact that it fits in a single page. */
+ BUILD_BUG_ON((sizeof(*st) - 1) & KVM_STEAL_VALID_BITS);
- if (unlikely(slots->generation != ghc->generation ||
- gpa != ghc->gpa ||
- kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
- /* We rely on the fact that it fits in a single page. */
- BUILD_BUG_ON((sizeof(*st) - 1) & KVM_STEAL_VALID_BITS);
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
+ while (!kvm_gpc_check(gpc, sizeof(*st))) {
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
- if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st)) ||
- kvm_is_error_hva(ghc->hva) || !ghc->memslot)
+ if (kvm_gpc_refresh(gpc, sizeof(*st)))
return;
+
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
}
- st = (struct kvm_steal_time __user *)ghc->hva;
+ st = gpc->khva;
+
/*
* Doing a TLB flush here, on the guest's behalf, can avoid
* expensive IPIs.
*/
if (guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) {
- u8 st_preempted = 0;
- int err = -EFAULT;
-
- if (!user_access_begin(st, sizeof(*st)))
- return;
-
- asm volatile("1: xchgb %0, %2\n"
- "xor %1, %1\n"
- "2:\n"
- _ASM_EXTABLE_UA(1b, 2b)
- : "+q" (st_preempted),
- "+&r" (err),
- "+m" (st->preempted));
- if (err)
- goto out;
-
- user_access_end();
+ u8 st_preempted = xchg(&st->preempted, 0);
vcpu->arch.st.preempted = 0;
trace_kvm_pv_tlb_flush(vcpu->vcpu_id,
st_preempted & KVM_VCPU_FLUSH_TLB);
/*
- * Request the flush instead of performing it inline. With
- * shadow paging kvm_vcpu_flush_tlb_guest() can reschedule
- * (via mmu_sync_children()), so it could not be called while
- * the user access above was open. Requesting is just a bit
- * set on the running vCPU, and the request is serviced
- * before the vCPU enters the guest; nothing requires the
- * flush to have completed any earlier than that.
+ * Request the flush instead of performing it inline. This
+ * runs inside an atomic gpc_srcu read-side section, with
+ * preemption disabled, and with shadow paging
+ * kvm_vcpu_flush_tlb_guest() can reschedule (via
+ * mmu_sync_children()). Requesting is just a bit set on the
+ * running vCPU, and the request is serviced before the vCPU
+ * enters the guest; nothing requires the flush to have
+ * completed any earlier than that.
*
* Note the xchg above must remain as-is: it atomically
* consumes any KVM_VCPU_FLUSH_TLB the guest set, as required
@@ -2133,39 +2118,33 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
*/
if (st_preempted & KVM_VCPU_FLUSH_TLB)
kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
-
- if (!user_access_begin(st, sizeof(*st)))
- goto dirty;
} else {
- if (!user_access_begin(st, sizeof(*st)))
- return;
-
- unsafe_put_user(0, &st->preempted, out);
+ WRITE_ONCE(st->preempted, 0);
vcpu->arch.st.preempted = 0;
}
- unsafe_get_user(version, &st->version, out);
+ version = READ_ONCE(st->version);
if (version & 1)
version += 1; /* first time write, random junk */
version += 1;
- unsafe_put_user(version, &st->version, out);
+ WRITE_ONCE(st->version, version);
smp_wmb();
- unsafe_get_user(steal, &st->steal, out);
+ steal = READ_ONCE(st->steal);
steal += current->sched_info.run_delay -
vcpu->arch.st.last_steal;
vcpu->arch.st.last_steal = current->sched_info.run_delay;
- unsafe_put_user(steal, &st->steal, out);
+ WRITE_ONCE(st->steal, steal);
+
+ smp_wmb();
version += 1;
- unsafe_put_user(version, &st->version, out);
+ WRITE_ONCE(st->version, version);
- out:
- user_access_end();
- dirty:
- mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
+ kvm_gpc_mark_dirty_in_slot(gpc);
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
}
static inline bool kvm_can_mwait_in_guest(void)
@@ -2640,11 +2619,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
{
- struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
- struct kvm_steal_time __user *st;
- struct kvm_memslots *slots;
- static const u8 preempted = KVM_VCPU_PREEMPTED;
- gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
+ struct gfn_to_pfn_cache *gpc = &vcpu->arch.st.cache;
+ struct kvm_steal_time *st;
/*
* The vCPU can be marked preempted if and only if the VM-Exit was on
@@ -2669,20 +2645,37 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
if (unlikely(current->mm != vcpu->kvm->mm))
return;
- slots = kvm_memslots(vcpu->kvm);
-
- if (unlikely(slots->generation != ghc->generation ||
- gpa != ghc->gpa ||
- kvm_is_error_hva(ghc->hva) || !ghc->memslot))
- return;
-
- st = (struct kvm_steal_time __user *)ghc->hva;
- BUILD_BUG_ON(sizeof(st->preempted) != sizeof(preempted));
-
- if (!copy_to_user_nofault(&st->preempted, &preempted, sizeof(preempted)))
+ /*
+ * This is called from the scheduler path (via kvm_sched_out), so
+ * the cache cannot be refreshed here. Setting 'preempted' is
+ * best-effort anyway; the old HVA-based code used
+ * copy_to_user_nofault(), which could also silently fail. If the
+ * cache is invalid, it will be refreshed from task context by
+ * record_steal_time() before the vCPU next enters the guest, since
+ * kvm_arch_vcpu_load() unconditionally raises KVM_REQ_STEAL_UPDATE.
+ */
+ guard(srcu_atomic)(&vcpu->kvm->gpc_srcu);
+ if (kvm_gpc_check(gpc, sizeof(*st))) {
+ st = gpc->khva;
+ WRITE_ONCE(st->preempted, KVM_VCPU_PREEMPTED);
vcpu->arch.st.preempted = KVM_VCPU_PREEMPTED;
- mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
+ kvm_gpc_mark_dirty_in_slot(gpc);
+ }
+}
+
+static void kvm_steal_time_reset(struct kvm_vcpu *vcpu)
+{
+ kvm_gpc_deactivate(&vcpu->arch.st.cache);
+ vcpu->arch.st.preempted = 0;
+ vcpu->arch.st.msr_val = 0;
+ /*
+ * Leave last_steal alone: it is the baseline against the vCPU
+ * task's cumulative sched_info.run_delay, which is never reset.
+ * Zeroing it here would cause the first record_steal_time() after
+ * a vCPU reset (INIT-SIPI, kexec) to account the task's entire
+ * lifetime run_delay as steal time in one jump.
+ */
}
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
@@ -9460,6 +9453,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
kvm_gpc_init(&vcpu->arch.pv_time, vcpu->kvm);
+ kvm_gpc_init(&vcpu->arch.st.cache, vcpu->kvm);
+
if (!irqchip_in_kernel(vcpu->kvm) || kvm_vcpu_is_reset_bsp(vcpu))
kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
else
@@ -9569,6 +9564,8 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
kvm_clear_async_pf_completion_queue(vcpu);
kvm_mmu_unload(vcpu);
+ kvm_steal_time_reset(vcpu);
+
kvmclock_reset(vcpu);
for_each_possible_cpu(cpu)
@@ -9689,7 +9686,8 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
kvm_make_request(KVM_REQ_EVENT, vcpu);
vcpu->arch.apf.msr_en_val = 0;
vcpu->arch.apf.msr_int_val = 0;
- vcpu->arch.st.msr_val = 0;
+
+ kvm_steal_time_reset(vcpu);
kvmclock_reset(vcpu);
--
2.43.0
next prev parent reply other threads:[~2026-11-05 2:01 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
2026-11-05 2:00 ` [PATCH v2 01/20] KVM: x86/xen: Fill the hypercall page with a single guest write David Woodhouse
2026-11-05 2:00 ` [PATCH v2 02/20] KVM: x86/xen: Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE David Woodhouse
2026-11-05 2:00 ` [PATCH v2 03/20] KVM: x86/xen: Extract delivery of event to vCPU into a separate helper David Woodhouse
2026-11-05 2:00 ` [PATCH v2 04/20] KVM: x86/xen: Explicitly tag "shared info" page as never being dirty tracked David Woodhouse
2026-11-05 2:00 ` [PATCH v2 05/20] KVM: x86/xen: Don't dirty track "vCPU info" page David Woodhouse
2026-11-05 2:00 ` [PATCH v2 06/20] KVM: x86: Request the guest TLB flush from record_steal_time() David Woodhouse
2026-11-10 20:19 ` Sean Christopherson
2026-11-10 21:14 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 07/20] KVM: selftests: Actually deactivate the HVA-based shared_info cache in xen_shinfo_test David Woodhouse
2026-11-05 2:00 ` [PATCH v2 08/20] KVM: pfncache: Use atomic SRCU for readers instead of a rwlock David Woodhouse
2026-11-05 2:00 ` David Woodhouse [this message]
2026-11-05 2:00 ` [PATCH v2 10/20] KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor) David Woodhouse
2026-11-05 2:29 ` sashiko-bot
2026-11-05 2:00 ` [PATCH v2 11/20] KVM: nVMX: Implement cache for L1 MSR bitmap David Woodhouse
2026-11-05 2:00 ` [PATCH v2 12/20] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages David Woodhouse
2026-11-05 2:33 ` sashiko-bot
2026-11-05 2:00 ` [PATCH v2 13/20] KVM: x86: Post KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates David Woodhouse
2026-11-05 2:00 ` [PATCH v2 14/20] KVM: selftests: Add nested VMX APIC cache invalidation test David Woodhouse
2026-11-05 2:25 ` sashiko-bot
2026-11-05 2:00 ` [PATCH v2 15/20] KVM: nSVM: Use a gfn_to_pfn_cache for the vmcb12 page David Woodhouse
2026-11-05 2:00 ` [PATCH v2 16/20] KVM: nSVM: Cache L1's MSR permissions map pages David Woodhouse
2026-11-05 2:00 ` [PATCH v2 17/20] KVM: nSVM: Cache L1's IO " David Woodhouse
2026-11-05 2:01 ` [PATCH v2 18/20] KVM: selftests: Add nested transition benchmark David Woodhouse
2026-11-05 2:01 ` [PATCH v2 19/20] KVM: selftests: Add Xen evtchn vs memslot-update race test David Woodhouse
2026-11-05 2:01 ` [PATCH v2 20/20] KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches David Woodhouse
2026-11-05 2:28 ` sashiko-bot
2026-11-10 19:52 ` [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
2026-11-10 20:14 ` Sean Christopherson
2026-11-11 18:38 ` David Woodhouse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261105020104.2507083-10-dwmw2@infradead.org \
--to=dwmw2@infradead.org \
--cc=dwmw@amazon.co.uk \
--cc=fgriffo@amazon.co.uk \
--cc=kunwu.chan@gmail.com \
--cc=kunwu.chan@linux.dev \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=paul@xen.org \
--cc=paulmck@kernel.org \
--cc=pbonzini@redhat.com \
--cc=rcu@vger.kernel.org \
--cc=seanjc@google.com \
--cc=syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com \
--cc=vkuznets@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®