mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH] kvm: hyper-v: Delay firing of expired stimers
@ 2026-01-15 14:15 Alexander Graf
  2026-01-23 18:21 ` Sean Christopherson
  2026-01-24  0:37 ` David Woodhouse
  0 siblings, 2 replies; 8+ messages in thread
From: Alexander Graf @ 2026-01-15 14:15 UTC (permalink / raw)
  To: kvm
  Cc: linux-kernel, hpa, x86, Paolo Bonzini, Sean Christopherson,
	Vitaly Kuznetsov, nh-open-source, gurugubs, jalliste,
	Michael Kelley, John Starks

During Windows Server 2025 hibernation, I have seen Windows' calculation
of interrupt target time get skewed over the hypervisor view of the same.
This can cause Windows to emit timer events in the past for events that
do not fire yet according to the real time source. This then leads to
interrupt storms in the guest which slow down execution to a point where
watchdogs trigger. Those manifest as bugchecks 0x9f and 0xa0 during
hibernation, typically in the resume path.

To work around this problem, we can delay timers that get created with a
target time in the past by a tiny bit (10µs) to give the guest CPU time
to process real work and make forward progress, hopefully recovering its
interrupt logic in the process. While this small delay can marginally
reduce accuracy of guest timers, 10µs are within the noise of VM
entry/exit overhead (~1-2 µs) so I do not expect to see real world impact.

To still provide some level of visibility when this happens, add a trace
point that clearly shows the discrepancy between the target time and the
current time.

Signed-off-by: Alexander Graf <graf@amazon.com>
---
 arch/x86/kvm/hyperv.c | 22 ++++++++++++++++++----
 arch/x86/kvm/trace.h  | 26 ++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 72b19a88a776..c41061acbcbc 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -666,13 +666,27 @@ static int stimer_start(struct kvm_vcpu_hv_stimer *stimer)
 	stimer->exp_time = stimer->count;
 	if (time_now >= stimer->count) {
 		/*
-		 * Expire timer according to Hypervisor Top-Level Functional
-		 * specification v4(15.3.1):
+		 * Hypervisor Top-Level Functional specification v4(15.3.1):
 		 * "If a one shot is enabled and the specified count is in
 		 * the past, it will expire immediately."
+		 *
+		 * However, there are cases during hibernation when Windows's
+		 * interrupt count calculation can go out of sync with KVM's
+		 * view of it, causing Windows to emit timer events in the past
+		 * for events that do not fire yet according to the real time
+		 * source. This then leads to interrupt storms in the guest
+		 * which slow down execution to a point where watchdogs trigger.
+		 *
+		 * Instead of taking TLFS literally on what "immediately" means,
+		 * give the guest at least 10µs to process work. While this can
+		 * marginally reduce accuracy of guest timers, 10µs are within
+		 * the noise of VM entry/exit overhead (~1-2 µs).
 		 */
-		stimer_mark_pending(stimer, false);
-		return 0;
+		trace_kvm_hv_stimer_start_expired(
+					hv_stimer_to_vcpu(stimer)->vcpu_id,
+					stimer->index,
+					time_now, stimer->count);
+		stimer->count = time_now + 100;
 	}
 
 	trace_kvm_hv_stimer_start_one_shot(hv_stimer_to_vcpu(stimer)->vcpu_id,
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 57d79fd31df0..f9e69c4d9e9b 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -1401,6 +1401,32 @@ TRACE_EVENT(kvm_hv_stimer_start_one_shot,
 		  __entry->count)
 );
 
+/*
+ * Tracepoint for stimer_start(one-shot timer already expired).
+ */
+TRACE_EVENT(kvm_hv_stimer_start_expired,
+	TP_PROTO(int vcpu_id, int timer_index, u64 time_now, u64 count),
+	TP_ARGS(vcpu_id, timer_index, time_now, count),
+
+	TP_STRUCT__entry(
+		__field(int, vcpu_id)
+		__field(int, timer_index)
+		__field(u64, time_now)
+		__field(u64, count)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_id = vcpu_id;
+		__entry->timer_index = timer_index;
+		__entry->time_now = time_now;
+		__entry->count = count;
+	),
+
+	TP_printk("vcpu_id %d timer %d time_now %llu count %llu (expired)",
+		  __entry->vcpu_id, __entry->timer_index, __entry->time_now,
+		  __entry->count)
+);
+
 /*
  * Tracepoint for stimer_timer_callback.
  */
-- 
2.47.1




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
  2026-01-15 14:15 [PATCH] kvm: hyper-v: Delay firing of expired stimers Alexander Graf
@ 2026-01-23 18:21 ` Sean Christopherson
  2026-01-24 21:26   ` Alexander Graf
  2026-01-24  0:37 ` David Woodhouse
  1 sibling, 1 reply; 8+ messages in thread
From: Sean Christopherson @ 2026-01-23 18:21 UTC (permalink / raw)
  To: Alexander Graf
  Cc: kvm, linux-kernel, hpa, x86, Paolo Bonzini, Vitaly Kuznetsov,
	nh-open-source, gurugubs, jalliste, Michael Kelley, John Starks

On Thu, Jan 15, 2026, Alexander Graf wrote:
> During Windows Server 2025 hibernation, I have seen Windows' calculation
> of interrupt target time get skewed over the hypervisor view of the same.

> This can cause Windows to emit timer events in the past for events that
> do not fire yet according to the real time source. This then leads to
> interrupt storms in the guest which slow down execution to a point where
> watchdogs trigger. Those manifest as bugchecks 0x9f and 0xa0 during
> hibernation, typically in the resume path.
> 
> To work around this problem, we can delay timers that get created with a
> target time in the past by a tiny bit (10µs) to give the guest CPU time
> to process real work and make forward progress, hopefully recovering its
> interrupt logic in the process. While this small delay can marginally
> reduce accuracy of guest timers, 10µs are within the noise of VM
> entry/exit overhead (~1-2 µs) so I do not expect to see real world impact.

There is a lot of hope piled into this.  And *always* padding the count makes me
more than a bit uncomfortable.  If the skew is really due to a guest bug and not
something on the host's side, i.e. if this isn't just a symptom of a real bug that
can be fixed and the _only_ option is to chuck in a workaround, then I would
strongly prefer to be as conservative as possible.  E.g. is it possible to
precisely detect this scenario and only add the delay when the guest appears to
be stuck?

> To still provide some level of visibility when this happens, add a trace
> point that clearly shows the discrepancy between the target time and the
> current time.

This honestly doesn't seem all that useful.  As a debug tool, sure, but once the
workaround is in place, it doesn't seem like it'll add a lot of value since it
would require the end user to be aware of the workaround in the first place.

If we really want something, a stat or a pr_xxx_once() (even though I generally
dislike those) seems like it'd be more helpful.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
  2026-01-15 14:15 [PATCH] kvm: hyper-v: Delay firing of expired stimers Alexander Graf
  2026-01-23 18:21 ` Sean Christopherson
@ 2026-01-24  0:37 ` David Woodhouse
  1 sibling, 0 replies; 8+ messages in thread
From: David Woodhouse @ 2026-01-24  0:37 UTC (permalink / raw)
  To: Alexander Graf, kvm
  Cc: linux-kernel, hpa, x86, Paolo Bonzini, Sean Christopherson,
	Vitaly Kuznetsov, nh-open-source, gurugubs, jalliste,
	Michael Kelley, John Starks

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Thu, 2026-01-15 at 14:15 +0000, Alexander Graf wrote:
> 
> +		 *
> +		 * However, there are cases during hibernation when
> Windows's
> +		 * interrupt count calculation can go out of sync
> with KVM's
> +		 * view of it, causing Windows to emit timer events
> in the past
> +		 * for events that do not fire yet according to the
> real time
> +		 * source. This then leads to interrupt storms in
> the guest
> +		 * which slow down execution to a point where
> watchdogs trigger.

Do these 'cases during hibernation' occur when the TSC page hasn't been
set up yet in the new environment?

I note get_time_ref_counter() falls back to get_kvmclock_ns() in that
case, but get_kvmclock_ns() is known to be hosed. We stopped using it
for Xen timers for precisely that reason; see commit 451a707813aee.

The detail in this case might be different, but we really should
*understand* why stimer_start() has a different idea of the time than
the guest does, and address that properly.

FWIW it's probably in the noise and not the actual cause of this error,
but it looks like stimer_start() might benefit from using
kvm_get_monotonic_and_clockread(), to get an actual paired reading from
the same TSC read instead of calling get_time_ref_counter() and then
*later* calling ktime_get() to read CLOCK_MONOTONIC at a slightly
different time. You can pass a TSC reading into get_time_ref_counter(),
can't you? Let callers who don't care pass rdtsc() in.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
  2026-01-23 18:21 ` Sean Christopherson
@ 2026-01-24 21:26   ` Alexander Graf
  2026-01-26  9:41     ` Vitaly Kuznetsov
  0 siblings, 1 reply; 8+ messages in thread
From: Alexander Graf @ 2026-01-24 21:26 UTC (permalink / raw)
  To: Sean Christopherson
  Cc: kvm, linux-kernel, hpa, x86, Paolo Bonzini, Vitaly Kuznetsov,
	nh-open-source, gurugubs, jalliste, Michael Kelley, John Starks


On 23.01.26 19:21, Sean Christopherson wrote:
> On Thu, Jan 15, 2026, Alexander Graf wrote:
>> During Windows Server 2025 hibernation, I have seen Windows' calculation
>> of interrupt target time get skewed over the hypervisor view of the same.
>> This can cause Windows to emit timer events in the past for events that
>> do not fire yet according to the real time source. This then leads to
>> interrupt storms in the guest which slow down execution to a point where
>> watchdogs trigger. Those manifest as bugchecks 0x9f and 0xa0 during
>> hibernation, typically in the resume path.
>>
>> To work around this problem, we can delay timers that get created with a
>> target time in the past by a tiny bit (10µs) to give the guest CPU time
>> to process real work and make forward progress, hopefully recovering its
>> interrupt logic in the process. While this small delay can marginally
>> reduce accuracy of guest timers, 10µs are within the noise of VM
>> entry/exit overhead (~1-2 µs) so I do not expect to see real world impact.
> There is a lot of hope piled into this.  And *always* padding the count makes me
> more than a bit uncomfortable.  If the skew is really due to a guest bug and not
> something on the host's side, i.e. if this isn't just a symptom of a real bug that
> can be fixed and the _only_ option is to chuck in a workaround, then I would
> strongly prefer to be as conservative as possible.  E.g. is it possible to
> precisely detect this scenario and only add the delay when the guest appears to
> be stuck?


This patch only pads when a timer is in the past, which I have not seen 
happen much on real systems. Usually you're trying to configure a timer 
for the future :).

That said, I have continued digging deeper since I posted this patch and 
I'm still trying to wrap my head around under which exact conditions any 
of this really does happen. Let's put this patch on hold until I have a 
more reliable reproducer.


Alex




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
  2026-01-24 21:26   ` Alexander Graf
@ 2026-01-26  9:41     ` Vitaly Kuznetsov
  2026-11-07 16:22       ` DV
  0 siblings, 1 reply; 8+ messages in thread
From: Vitaly Kuznetsov @ 2026-01-26  9:41 UTC (permalink / raw)
  To: Alexander Graf, Sean Christopherson
  Cc: kvm, linux-kernel, hpa, x86, Paolo Bonzini, nh-open-source,
	gurugubs, jalliste, Michael Kelley, John Starks

Alexander Graf <graf@amazon.com> writes:

> On 23.01.26 19:21, Sean Christopherson wrote:
>> On Thu, Jan 15, 2026, Alexander Graf wrote:
>>> During Windows Server 2025 hibernation, I have seen Windows' calculation
>>> of interrupt target time get skewed over the hypervisor view of the same.
>>> This can cause Windows to emit timer events in the past for events that
>>> do not fire yet according to the real time source. This then leads to
>>> interrupt storms in the guest which slow down execution to a point where
>>> watchdogs trigger. Those manifest as bugchecks 0x9f and 0xa0 during
>>> hibernation, typically in the resume path.
>>>
>>> To work around this problem, we can delay timers that get created with a
>>> target time in the past by a tiny bit (10µs) to give the guest CPU time
>>> to process real work and make forward progress, hopefully recovering its
>>> interrupt logic in the process. While this small delay can marginally
>>> reduce accuracy of guest timers, 10µs are within the noise of VM
>>> entry/exit overhead (~1-2 µs) so I do not expect to see real world impact.
>> There is a lot of hope piled into this.  And *always* padding the count makes me
>> more than a bit uncomfortable.  If the skew is really due to a guest bug and not
>> something on the host's side, i.e. if this isn't just a symptom of a real bug that
>> can be fixed and the _only_ option is to chuck in a workaround, then I would
>> strongly prefer to be as conservative as possible.  E.g. is it possible to
>> precisely detect this scenario and only add the delay when the guest appears to
>> be stuck?
>
>
> This patch only pads when a timer is in the past, which I have not seen 
> happen much on real systems. Usually you're trying to configure a timer 
> for the future :).
>
> That said, I have continued digging deeper since I posted this patch and 
> I'm still trying to wrap my head around under which exact conditions any 
> of this really does happen. Let's put this patch on hold until I have a 
> more reliable reproducer.

My bet goes to the clocksource switch, e.g. the guest disables (or just
stops using, good luck detecting that :-) ) TSC page and uses raw TSC
for some period or something. 

I remember we had to add some fairly ugly hacks where we also "piled a
log of hope", e.g.:

commit 0469f2f7ab4c6a6cae4b74c4f981c4da6d909411
Author: Vitaly Kuznetsov <vkuznets@redhat.com>
Date:   Tue Mar 16 15:37:36 2021 +0100

    KVM: x86: hyper-v: Don't touch TSC page values when guest opted for re-enlightenment

Also, AFAIR we don't currently implement "Synthetic Time-Unhalted Timer"
from TLFS and who knows, maybe Windows' behavior is going to change when
we do...

-- 
Vitaly


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
  2026-01-26  9:41     ` Vitaly Kuznetsov
@ 2026-11-07 16:22       ` DV
  2026-11-16 17:12         ` Vitaly Kuznetsov
  0 siblings, 1 reply; 8+ messages in thread
From: DV @ 2026-11-07 16:22 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: Alexander Graf, Sean Christopherson, kvm, linux-kernel, hpa, x86,
	Paolo Bonzini, nh-open-source, gurugubs, jalliste,
	Michael Kelley, John Starks, David Woodhouse, Marcelo Tosatti,
	qemu-devel

Hi Vitaly, Alex, all,

On Mon, 26 Jan 2026, Vitaly Kuznetsov wrote:
> My bet goes to the clocksource switch, e.g. the guest disables (or
> just stops using, good luck detecting that :-) ) TSC page and uses
> raw TSC for some period or something.

I think I've hit a close relative of this, with a reproducer that
works on every boot. A Windows guest running its own Hyper-V computes
direct-mode stimer deadlines from the raw TSC, while KVM's reference
counter lags the TSC because of how QEMU restores kvmclock. Here the
lag is positive, so the guest's timers fire late instead of storming.

Setup
-----

Host: Intel Core Ultra 9 285K (invariant TSC, 3686.4 MHz), Linux
7.3.0, QEMU 10.2.4 (pc-q35-10.2), libvirt 12.2.0.

Guest: Windows 11 Pro 26H2 (build 26300.9711), 8 vCPUs, with
virtualization-based security on. So Windows runs as the root
partition on its own Hyper-V (L1), nested on KVM.

   -cpu host,migratable=off,hv-time=on,hv-relaxed=on,hv-vapic=on,
        hv-spinlocks=0xfff,hv-vpindex=on,hv-runtime=on,hv-synic=on,
        hv-stimer=on,hv-stimer-direct=on,hv-frequencies=on,
        hv-tlbflush=on,hv-tlbflush-direct=on,hv-tlbflush-ext=on,
        hv-evmcs=on,hv-avic=on,hv-emsr-bitmap=on,hv-xmm-input=on,...

Symptom
-------

Every timer inside Windows fired 0.3-0.7 s late. A 1 ms wait on a
high-resolution waitable timer took up to 466 ms, and an indirect
display driver paced only by a timer refreshed at 6-8 Hz instead of
120. Things only moved when some other interrupt woke a vCPU.

What L1 does
------------

With both hv-stimer-direct and hv-frequencies, L1 arms stimer0 in
direct mode (config 0x1ef8: vector 0xef, direct, auto-enable) and
writes absolute counts. Call X the amount by which the reference
counter lags guest TSC / HV_X64_MSR_TSC_FREQUENCY. L1's counts are
X too far out. A timer that should fire right away looks like this:

   kvm_hv_stimer_start_one_shot: vcpu_id 6 timer 0
       time_now 40095770231 count 40100417164

That count is 464.69 ms ahead of time_now, and X on that boot was
464.7 ms. L1 writes the count again on every L2 timer exit, so such a
timer keeps moving to now + X and only fires once the vCPU is quiet
for that long.

Without hv-stimer-direct, without hv-frequencies, or without
hv-stimer, L1 uses the local APIC timer instead (TMICT/TDCR writes,
a relative countdown), and all timers are on time. I've turned direct
mode off as a workaround: a 1 ms wait is now late by at most 1.24 ms.

Why the reference counter lags
------------------------------

get_time_ref_counter() follows kvmclock on both paths. The fallback
divides get_kvmclock_ns() by 100, and compute_tsc_page_parameters()
takes the TSC page's tsc_offset from the pvclock system_time.

QEMU's kvmclock_realize() calls KVM_GET_CLOCK when the VM is created.
kvmclock_vm_state_change() writes the saved value back with
KVM_SET_CLOCK when the VM starts running, and saves a new value
whenever it stops. So kvmclock, and with it the reference counter,
stands still from creation to first run and during every pause. The
guest TSC keeps counting through both. On six boots X was 297, 465,
570, 495, 669 and 512 ms.

Two measurements tie X to this:

- In a trace from QEMU startup, extrapolated back to zero, the guest
   TSC started at host monotonic 265208.1505 s and the reference
   counter at 265208.8205 s, 0.670 s later. X on that boot was
   669.2 ms.

- virsh suspend for 1.045 s, then resume: X grew from 669.2 to
   1689.3 ms (+1020.1 ms), and L1's far stimer deadline moved out by
   the same amount.

hv-evmcs off (X = 570.1 ms) and hv-reenlightenment on (X = 494.7 ms)
didn't change the behaviour. Host load and power management were
ruled out too.

Connection to your thread
-------------------------

If the reference counter were ahead of TSC time instead, L1's
deadlines would land in the past. stimer_start() expires a one-shot
whose count is in the past at once, L1 arms it again on its next
exit, and that would look like the expired-stimer storm Alex's patch
delays.

I haven't been able to try that here, because libvirt is set up to
start a fresh QEMU whenever the guest reboots. But a reset inside the
same QEMU process sets the guest TSC back to 0 while kvmclock keeps
counting, so I'd expect it to flip the sign, possibly by the whole
previous uptime. A resume path that restores the two clocks
differently could do the same.

Questions
---------

1. Should KVM keep the Hyper-V reference counter equal to guest TSC /
    TSC frequency when hv-frequencies is exposed? L1 seems to assume
    it is.

2. If so, where does the fix belong: in QEMU's kvmclock save/restore,
    or in KVM deriving the reference counter (and TSC page) from the
    guest TSC?

3. If not, should hv-stimer-direct be documented as unsafe for guests
    that run their own Hyper-V?

I can share the full traces and the Windows-side timer and frame
probes if they'd help.

Claude Code, an AI coding assistant, did the tracing and analysis
while I was looking into why the VM's virtual display ran at a few
frames per second.

Thanks,
David

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
  2026-11-07 16:22       ` DV
@ 2026-11-16 17:12         ` Vitaly Kuznetsov
  2026-11-16 17:28           ` DV
  0 siblings, 1 reply; 8+ messages in thread
From: Vitaly Kuznetsov @ 2026-11-16 17:12 UTC (permalink / raw)
  To: DV
  Cc: Alexander Graf, Sean Christopherson, kvm, linux-kernel, hpa, x86,
	Paolo Bonzini, nh-open-source, gurugubs, jalliste,
	Michael Kelley, John Starks, David Woodhouse, Marcelo Tosatti,
	qemu-devel

DV <dv@etik.com> writes:

> Hi Vitaly, Alex, all,
>
> On Mon, 26 Jan 2026, Vitaly Kuznetsov wrote:
>> My bet goes to the clocksource switch, e.g. the guest disables (or
>> just stops using, good luck detecting that :-) ) TSC page and uses
>> raw TSC for some period or something.
>
> I think I've hit a close relative of this, with a reproducer that
> works on every boot. A Windows guest running its own Hyper-V computes
> direct-mode stimer deadlines from the raw TSC, while KVM's reference
> counter lags the TSC because of how QEMU restores kvmclock. Here the
> lag is positive, so the guest's timers fire late instead of storming.
>
> Setup
> -----
>
> Host: Intel Core Ultra 9 285K (invariant TSC, 3686.4 MHz), Linux
> 7.3.0, QEMU 10.2.4 (pc-q35-10.2), libvirt 12.2.0.
>
> Guest: Windows 11 Pro 26H2 (build 26300.9711), 8 vCPUs, with
> virtualization-based security on. So Windows runs as the root
> partition on its own Hyper-V (L1), nested on KVM.
>
>    -cpu host,migratable=off,hv-time=on,hv-relaxed=on,hv-vapic=on,
>         hv-spinlocks=0xfff,hv-vpindex=on,hv-runtime=on,hv-synic=on,
>         hv-stimer=on,hv-stimer-direct=on,hv-frequencies=on,
>         hv-tlbflush=on,hv-tlbflush-direct=on,hv-tlbflush-ext=on,
>         hv-evmcs=on,hv-avic=on,hv-emsr-bitmap=on,hv-xmm-input=on,...
>
> Symptom
> -------
>
> Every timer inside Windows fired 0.3-0.7 s late. A 1 ms wait on a
> high-resolution waitable timer took up to 466 ms, and an indirect
> display driver paced only by a timer refreshed at 6-8 Hz instead of
> 120. Things only moved when some other interrupt woke a vCPU.
>
> What L1 does
> ------------
>
> With both hv-stimer-direct and hv-frequencies, L1 arms stimer0 in
> direct mode (config 0x1ef8: vector 0xef, direct, auto-enable) and
> writes absolute counts. Call X the amount by which the reference
> counter lags guest TSC / HV_X64_MSR_TSC_FREQUENCY. L1's counts are
> X too far out. A timer that should fire right away looks like this:
>
>    kvm_hv_stimer_start_one_shot: vcpu_id 6 timer 0
>        time_now 40095770231 count 40100417164
>
> That count is 464.69 ms ahead of time_now, and X on that boot was
> 464.7 ms. L1 writes the count again on every L2 timer exit, so such a
> timer keeps moving to now + X and only fires once the vCPU is quiet
> for that long.
>
> Without hv-stimer-direct, without hv-frequencies, or without
> hv-stimer, L1 uses the local APIC timer instead (TMICT/TDCR writes,
> a relative countdown), and all timers are on time. I've turned direct
> mode off as a workaround: a 1 ms wait is now late by at most 1.24 ms.
>
> Why the reference counter lags
> ------------------------------
>
> get_time_ref_counter() follows kvmclock on both paths. The fallback
> divides get_kvmclock_ns() by 100, and compute_tsc_page_parameters()
> takes the TSC page's tsc_offset from the pvclock system_time.
>
> QEMU's kvmclock_realize() calls KVM_GET_CLOCK when the VM is created.
> kvmclock_vm_state_change() writes the saved value back with
> KVM_SET_CLOCK when the VM starts running, and saves a new value
> whenever it stops. So kvmclock, and with it the reference counter,
> stands still from creation to first run and during every pause. The
> guest TSC keeps counting through both. On six boots X was 297, 465,
> 570, 495, 669 and 512 ms.
>
> Two measurements tie X to this:
>
> - In a trace from QEMU startup, extrapolated back to zero, the guest
>    TSC started at host monotonic 265208.1505 s and the reference
>    counter at 265208.8205 s, 0.670 s later. X on that boot was
>    669.2 ms.
>
> - virsh suspend for 1.045 s, then resume: X grew from 669.2 to
>    1689.3 ms (+1020.1 ms), and L1's far stimer deadline moved out by
>    the same amount.
>
> hv-evmcs off (X = 570.1 ms) and hv-reenlightenment on (X = 494.7 ms)
> didn't change the behaviour. Host load and power management were
> ruled out too.
>
> Connection to your thread
> -------------------------
>
> If the reference counter were ahead of TSC time instead, L1's
> deadlines would land in the past. stimer_start() expires a one-shot
> whose count is in the past at once, L1 arms it again on its next
> exit, and that would look like the expired-stimer storm Alex's patch
> delays.
>
> I haven't been able to try that here, because libvirt is set up to
> start a fresh QEMU whenever the guest reboots. But a reset inside the
> same QEMU process sets the guest TSC back to 0 while kvmclock keeps
> counting, so I'd expect it to flip the sign, possibly by the whole
> previous uptime. A resume path that restores the two clocks
> differently could do the same.
>
> Questions
> ---------
>
> 1. Should KVM keep the Hyper-V reference counter equal to guest TSC /
>     TSC frequency when hv-frequencies is exposed? L1 seems to assume
>     it is.
>
> 2. If so, where does the fix belong: in QEMU's kvmclock save/restore,
>     or in KVM deriving the reference counter (and TSC page) from the
>     guest TSC?
>
> 3. If not, should hv-stimer-direct be documented as unsafe for guests
>     that run their own Hyper-V?
>

From what I remember, direct mode synthetic timers were added
specifically to aid Hyper-V-on-KVM case as non-nested Windows was
preferring non-direct version anyway. So I believe we need a fix and not
3).

I'm wondering if the issue is somewhat related to the nasty hack we put
in 

commit 0469f2f7ab4c6a6cae4b74c4f981c4da6d909411
Author: Vitaly Kuznetsov <vkuznets@redhat.com>
Date:   Tue Mar 16 15:37:36 2021 +0100

    KVM: x86: hyper-v: Don't touch TSC page values when guest opted for re-enlightenment

but maybe hibernation is somewhat different. Or maybe we misunderstood
the problem in the first place and only fixed one sympthom of it?

> I can share the full traces and the Windows-side timer and frame
> probes if they'd help.
>
> Claude Code, an AI coding assistant, did the tracing and analysis
> while I was looking into why the VM's virtual display ran at a few
> frames per second.
>
> Thanks,
> David
>

-- 
Vitaly


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
  2026-11-16 17:12         ` Vitaly Kuznetsov
@ 2026-11-16 17:28           ` DV
  0 siblings, 0 replies; 8+ messages in thread
From: DV @ 2026-11-16 17:28 UTC (permalink / raw)
  To: Vitaly Kuznetsov
  Cc: Alexander Graf, Sean Christopherson, kvm, linux-kernel, hpa, x86,
	Paolo Bonzini, nh-open-source, gurugubs, jalliste,
	Michael Kelley, John Starks, David Woodhouse, Marcelo Tosatti,
	qemu-devel

On Mon, 16 Nov 2026, Vitaly Kuznetsov wrote:
> From what I remember, direct mode synthetic timers were added
> specifically to aid Hyper-V-on-KVM case as non-nested Windows was
> preferring non-direct version anyway. So I believe we need a fix and
> not 3).

Thanks, that rules out 3).

> I'm wondering if the issue is somewhat related to the nasty hack we
> put in commit 0469f2f7ab4c ("KVM: x86: hyper-v: Don't touch TSC page
> values when guest opted for re-enlightenment") but maybe hibernation
> is somewhat different. Or maybe we misunderstood the problem in the
> first place and only fixed one sympthom of it?

I don't think the hack itself is involved here, but your second guess
fits what I see.

The hack only takes effect once the guest has written
HV_X64_MSR_TSC_EMULATION_CONTROL. In the failing setup
hv-reenlightenment isn't exposed, so KVM recomputes the TSC page on
every masterclock update and every KVM_SET_CLOCK as usual. With
hv-reenlightenment on, X was 494.7 ms and the timers were just as
late. I didn't trace whether L1 wrote the MSR in that run, but it
wouldn't change the lag at boot: L1 enabling the page sets
HV_TSC_PAGE_GUEST_CHANGED, so the first setup is computed normally
either way.

What points to your second guess: X is already in kvmclock before the
guest runs its first instruction, so when L1 enables the TSC page,
KVM writes a tsc_offset of about -X into it. Yet on a boot with no
pause and no migration, L1's counts were still X too far out. If L1
used the values it read from the TSC page, even cached ones, it would
have been right at that point. So L1 seems to take L0's reference
time to be guest TSC / TSC frequency with no offset at all, perhaps
because on a Hyper-V host the two start together.

If so, the 2021 migration storm and this are the same assumption
broken in two ways. The hack keeps tsc_offset from changing after L1
has started, which helps on migration, but it can't help when
tsc_offset isn't zero to begin with. And without re-enlightenment,
nothing keeps it from changing on pause/resume, which matches the
virsh suspend result in my first mail.

If it helps, I can rerun with hv-reenlightenment on, trace whether L1
writes HV_X64_MSR_TSC_EMULATION_CONTROL, and repeat the suspend. If
the hack engages, I'd expect X to stay the same across the pause,
while the timers stay X late from boot.

As before, Claude Code did the code reading for this.

Thanks,
David

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-16 17:28 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-01-15 14:15 [PATCH] kvm: hyper-v: Delay firing of expired stimers Alexander Graf
2026-01-23 18:21 ` Sean Christopherson
2026-01-24 21:26   ` Alexander Graf
2026-01-26  9:41     ` Vitaly Kuznetsov
2026-11-07 16:22       ` DV
2026-11-16 17:12         ` Vitaly Kuznetsov
2026-11-16 17:28           ` DV
2026-01-24  0:37 ` David Woodhouse

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®