Re: [PATCH v4 04/30] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration

[David Woodhouse <dwmw2@infradead.org>]

[-- Attachment #1: Type: text/plain, Size: 7240 bytes --]

On Tue, 2026-05-19 at 14:23 -0700, Dongli Zhang wrote:
> I think I now understand why I feel like I am always asking weird questions. I
> have been thinking about how to account for downtime, so I see
> KVM_SET_CLOCK_GUEST as a supplement to KVM_SET_CLOCK.

I do not believe in "downtime". There is no such thing.
There is only "steal time".

A CPU may be off in the weeds — a vCPU suffering steal time, or even a
pCPU in SMM which is effectively the same thing — but time doesn't
stop, and neither does the TSC.

> Suppose we are not going to account for any downtime. With KVM_SET_CLOCK_GUEST:
> 
> 1. The masterclock is active, so gTSC is synchronized across vCPUs. All vCPUs
> share the same kvm_read_l1_tsc(v, ka->master_cycle_now).

Strictly, by the time we get to the end of my series, masterclock is
active *because* all the vCPUs are running at the same TSC rate (even
if the guest set them to different offsets). But OK.

> 2. Migrate the gTSC to the target VM however people want (either ablolute value
> or offset value). (Optional) Account for downtime in gTSC however people want,
> even with KVM_SET_CLOCK/KVM_CLOCK_REALTIME, which you may not like.
>
> 3. Adjust kvm-clock (that is, ka->kvmclock_offset) with KVM_SET_CLOCK_GUEST.
> 
> That is why you think KVM_SET_CLOCK is no longer required if we have
> KVM_SET_CLOCK_GUEST. While I think KVM_SET_CLOCK is required because of
> KVM_CLOCK_REALTIME.

If I recall correctly what we described in
https://lore.kernel.org/all/20240522001817.619072-8-dwmw2@infradead.org/
I don't think we actually needed KVM_SET_CLOCK at all, did we?

We *abuse* KVM_GET_CLOCK to give us a tuple of {realtime, host TSC}
because there's actually no other way for *userspace* to get that. We
don't actually *care* about the KVM clock part.

We use the {realtime, host TSC} pair to reconstitute the guest TSC
values to correctly reflect the passing of time while the guest was in
the ether.

> It it isn't required to account any downtime for gTSC or if there is another way
> to do so, only KVM_SET_CLOCK_GUEST is enough.

Right. If you only want the guest to come back with the *same* values
in its TSC as before the migration, as if the TSC was *paused* during
the migration, then you can just restore those values and use
KVM_SET_CLOCK_GUEST. Assuming you are on modern hardware and have set
all vCPUs to the same rate (and are using this series so the *guest*
can't break masterclock for you, and you can trust the
KVM_SET_CLOCK_GUEST will work).

> > 
> > > Another scenario is when only MASTERCLOCK_UPDATE is pending and there is no
> > > pending CLOCK_UPDATE.
> > > 
> > > In this scenario, is it fine to skip processing MASTERCLOCK_UPDATE before saving
> > > pvclock_vcpu_time_info?
> > > 
> > 
> > I'm not sure I understand that scenario. 
> > 
> > MASTERCLOCK_UPDATE means we have to actually recalculate the master
> > clock (which really *should* be rare, now!). And then any time we do
> > that, we also have to do a CLOCK_UPDATE on every vCPU to disseminate
> > the new information. Which is why kvm_end_pvclock_update() does exactly
> > that.
> > 
> > So your "MASTERCLOCK_UPDATE is pending and there is no pending
> > CLOCK_UPDATE" doesn't make much sense to me. If MASTERCLOCK_UPDATE is
> > pending, then there *will* be a CLOCK_UPDATE pending.
> 
> Suppose the VM is stopped and the master clock is active.

I don't know what it means for a VM to be 'stopped'. Do you mean that
all vCPUs happen to be experiencing steal time at the present moment?

> Suddenly, we change the host clocksource from TSC to HPET. pvclock_gtod_notify()
> may call pvclock_gtod_update_fn() to set a pending KVM_REQ_MASTERCLOCK_UPDATE
> for all vCPUs. Unless the pending KVM_REQ_MASTERCLOCK_UPDATE is processed by
> kvm_update_masterclock(), kvm_end_pvclock_update() will not set a pending
> KVM_REQ_CLOCK_UPDATE.

You say 'Unless'... do you mean 'Until'?

> Therefore, this is a scenario in which only KVM_REQ_MASTERCLOCK_UPDATE is pending.
> 
> I do not think this scenario is important. I am just curious about the expected
> way to implement similar code in the future :)

I think that's working correctly. Until the master clock has *actually*
been updated, there's no point in setting CLOCK_UPDATE for each vCPU to
disseminate the new information to its own pvclock?



> > 
> > 
> > > > > 
> > > > > Would it be helpful to validate that the delta is within a reasonable range,
> > > > > e.g. that the drift can never be more than five minutes (forward or backward)?
> > > > 
> > > > If a guest has been running for months on a previous host and is
> > > > migrated to a new host, don't we expect that the KVM clock of the new
> > > > VM on the new host is tweaked from its default near-zero after
> > > > creation, to some large amount?
> > > > 
> > > 
> > > Regarding live migration, my own investigation does not show a proportional
> > > relationship between VM uptime and the amount of drift.
> > 
> > You're comparing the VM on the source host, with the VM on the
> > destination post-migration.
> 
> Apologies for making it confusing. I was just trying to explain why I think the
> kvm-clock drift will not be large.

Sure, but I don't care. If we have a sane API, the drift should be zero
:)

> We previously discussed the vCPU hotplug and kvm-clock drift issue. The longer
> the time interval between two vCPU hotplug events, the larger the drift.
> 
> For live migration (with QEMU), I provided the equation to show that the drift
> will not be large, because it is determined by something else rather than by how
> long the VM has been running on the source server.
> 
> 
> For the previous vCPU hotplug and kvm-clock bug, if we add more vCPUs to a guest
> that has been running for three months, the drift will be relatively larger.
> 
> For QEMU live migration, migrating a guest VM that has been running on the
> source host for *three months* versus one that has been running for *one day*
> will not cause much difference in kvm-clock drift.

Right.

> For the ideal live update case (on the same host), there may be no need to
> adjust gTSC so that it keeps incrementing. In that case, KVM_SET_CLOCK_GUEST can
> be used to adjust kvm-clock based on gTSC.

Right. You restore the gTSC using its *offset* from the host TSC which
hasn't stopped counting on the same host. Then use KVM_SET_CLOCK_GUEST
to restore the kvmclock in terms of the gTSC. And you have an
absolutely cycle-perfect migration.

> For the live migration scenario, the current QEMU implementation not only fails
> to account for downtime, but also has a drift issue. That is what I would like
> to address in QEMU.

Again, restore the gTSC as accurately as possible. Probably by working
out for *yourself* the relationships of the source and destination host
TSCs to real time, and then reconstituting on the destination using TSC
offset just as for live migration.

And then use KVM_SET_CLOCK_GUEST too.

That's what I attempted to document in
https://lore.kernel.org/all/20240522001817.619072-8-dwmw2@infradead.org/
and should probably revive.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]