mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: DV <dv@etik.com>
To: Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: Alexander Graf <graf@amazon.com>,
	Sean Christopherson <seanjc@google.com>,
	kvm@vger.kernel.org, linux-kernel@vger.kernel.org, hpa@zytor.com,
	x86@kernel.org, Paolo Bonzini <pbonzini@redhat.com>,
	nh-open-source@amazon.com, gurugubs@amazon.com,
	jalliste@amazon.co.uk, Michael Kelley <mhklinux@outlook.com>,
	John Starks <jostarks@microsoft.com>,
	David Woodhouse <dwmw2@infradead.org>,
	Marcelo Tosatti <mtosatti@redhat.com>,
	qemu-devel@nongnu.org
Subject: Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
Date: Sat, 7 Nov 2026 17:22:25 +0100	[thread overview]
Message-ID: <1435d106-d4b0-487e-85d3-4032a9fe9332@etik.com> (raw)
In-Reply-To: <87bjigaeyx.fsf@redhat.com>

Hi Vitaly, Alex, all,

On Mon, 26 Jan 2026, Vitaly Kuznetsov wrote:
> My bet goes to the clocksource switch, e.g. the guest disables (or
> just stops using, good luck detecting that :-) ) TSC page and uses
> raw TSC for some period or something.

I think I've hit a close relative of this, with a reproducer that
works on every boot. A Windows guest running its own Hyper-V computes
direct-mode stimer deadlines from the raw TSC, while KVM's reference
counter lags the TSC because of how QEMU restores kvmclock. Here the
lag is positive, so the guest's timers fire late instead of storming.

Setup
-----

Host: Intel Core Ultra 9 285K (invariant TSC, 3686.4 MHz), Linux
7.3.0, QEMU 10.2.4 (pc-q35-10.2), libvirt 12.2.0.

Guest: Windows 11 Pro 26H2 (build 26300.9711), 8 vCPUs, with
virtualization-based security on. So Windows runs as the root
partition on its own Hyper-V (L1), nested on KVM.

   -cpu host,migratable=off,hv-time=on,hv-relaxed=on,hv-vapic=on,
        hv-spinlocks=0xfff,hv-vpindex=on,hv-runtime=on,hv-synic=on,
        hv-stimer=on,hv-stimer-direct=on,hv-frequencies=on,
        hv-tlbflush=on,hv-tlbflush-direct=on,hv-tlbflush-ext=on,
        hv-evmcs=on,hv-avic=on,hv-emsr-bitmap=on,hv-xmm-input=on,...

Symptom
-------

Every timer inside Windows fired 0.3-0.7 s late. A 1 ms wait on a
high-resolution waitable timer took up to 466 ms, and an indirect
display driver paced only by a timer refreshed at 6-8 Hz instead of
120. Things only moved when some other interrupt woke a vCPU.

What L1 does
------------

With both hv-stimer-direct and hv-frequencies, L1 arms stimer0 in
direct mode (config 0x1ef8: vector 0xef, direct, auto-enable) and
writes absolute counts. Call X the amount by which the reference
counter lags guest TSC / HV_X64_MSR_TSC_FREQUENCY. L1's counts are
X too far out. A timer that should fire right away looks like this:

   kvm_hv_stimer_start_one_shot: vcpu_id 6 timer 0
       time_now 40095770231 count 40100417164

That count is 464.69 ms ahead of time_now, and X on that boot was
464.7 ms. L1 writes the count again on every L2 timer exit, so such a
timer keeps moving to now + X and only fires once the vCPU is quiet
for that long.

Without hv-stimer-direct, without hv-frequencies, or without
hv-stimer, L1 uses the local APIC timer instead (TMICT/TDCR writes,
a relative countdown), and all timers are on time. I've turned direct
mode off as a workaround: a 1 ms wait is now late by at most 1.24 ms.

Why the reference counter lags
------------------------------

get_time_ref_counter() follows kvmclock on both paths. The fallback
divides get_kvmclock_ns() by 100, and compute_tsc_page_parameters()
takes the TSC page's tsc_offset from the pvclock system_time.

QEMU's kvmclock_realize() calls KVM_GET_CLOCK when the VM is created.
kvmclock_vm_state_change() writes the saved value back with
KVM_SET_CLOCK when the VM starts running, and saves a new value
whenever it stops. So kvmclock, and with it the reference counter,
stands still from creation to first run and during every pause. The
guest TSC keeps counting through both. On six boots X was 297, 465,
570, 495, 669 and 512 ms.

Two measurements tie X to this:

- In a trace from QEMU startup, extrapolated back to zero, the guest
   TSC started at host monotonic 265208.1505 s and the reference
   counter at 265208.8205 s, 0.670 s later. X on that boot was
   669.2 ms.

- virsh suspend for 1.045 s, then resume: X grew from 669.2 to
   1689.3 ms (+1020.1 ms), and L1's far stimer deadline moved out by
   the same amount.

hv-evmcs off (X = 570.1 ms) and hv-reenlightenment on (X = 494.7 ms)
didn't change the behaviour. Host load and power management were
ruled out too.

Connection to your thread
-------------------------

If the reference counter were ahead of TSC time instead, L1's
deadlines would land in the past. stimer_start() expires a one-shot
whose count is in the past at once, L1 arms it again on its next
exit, and that would look like the expired-stimer storm Alex's patch
delays.

I haven't been able to try that here, because libvirt is set up to
start a fresh QEMU whenever the guest reboots. But a reset inside the
same QEMU process sets the guest TSC back to 0 while kvmclock keeps
counting, so I'd expect it to flip the sign, possibly by the whole
previous uptime. A resume path that restores the two clocks
differently could do the same.

Questions
---------

1. Should KVM keep the Hyper-V reference counter equal to guest TSC /
    TSC frequency when hv-frequencies is exposed? L1 seems to assume
    it is.

2. If so, where does the fix belong: in QEMU's kvmclock save/restore,
    or in KVM deriving the reference counter (and TSC page) from the
    guest TSC?

3. If not, should hv-stimer-direct be documented as unsafe for guests
    that run their own Hyper-V?

I can share the full traces and the Windows-side timer and frame
probes if they'd help.

Claude Code, an AI coding assistant, did the tracing and analysis
while I was looking into why the VM's virtual display ran at a few
frames per second.

Thanks,
David

  reply	other threads:[~2026-11-07 16:22 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-01-15 14:15 Alexander Graf
2026-01-23 18:21 ` Sean Christopherson
2026-01-24 21:26   ` Alexander Graf
2026-01-26  9:41     ` Vitaly Kuznetsov
2026-11-07 16:22       ` DV [this message]
2026-11-16 17:12         ` Vitaly Kuznetsov
2026-11-16 17:28           ` DV
2026-01-24  0:37 ` David Woodhouse

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1435d106-d4b0-487e-85d3-4032a9fe9332@etik.com \
    --to=dv@etik.com \
    --cc=dwmw2@infradead.org \
    --cc=graf@amazon.com \
    --cc=gurugubs@amazon.com \
    --cc=hpa@zytor.com \
    --cc=jalliste@amazon.co.uk \
    --cc=jostarks@microsoft.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mhklinux@outlook.com \
    --cc=mtosatti@redhat.com \
    --cc=nh-open-source@amazon.com \
    --cc=pbonzini@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=seanjc@google.com \
    --cc=vkuznets@redhat.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®