[BUG REPORT] KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

[BUG REPORT] KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

[Chenzhendong (alex)]

Hi,

While testing storage snapshots that suspend and resume a VM on the same
host (vm_stop -> snapshot -> vm_start), we observed that the guest
CLOCK_MONOTONIC jumps *backward* by the freeze duration after every
resume.

The problem
-----------

On a same-host pause/resume cycle, QEMU saves the kvmclock value at the
freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via
KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).

kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:

    kvmclock_offset = data.clock - now_raw_ns

with now_raw_ns sampled at T2. Since

    data.clock  = now_raw_ns(T1) + old_kvmclock_offset

this gives

    new_offset  = old_kvmclock_offset - freeze_duration

i.e. the offset is pulled back by the freeze duration.

The guest computes CLOCK_MONOTONIC from the pvclock page as:

    CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift

where system_time = kernel_ns + kvmclock_offset is filled by the host at
each vCPU entry. After resume the host re-samples tsc_timestamp, so the
(rdtsc - tsc_timestamp) term starts growing normally again and never
goes backward. The only term that can shrink is system_time: because
kvmclock_offset was pulled back by freeze_duration, the new system_time
is freeze_duration smaller than the value the guest last observed before
the freeze. The guest's next clock_gettime(CLOCK_MONOTONIC) therefore
reads a value freeze_duration smaller than before the freeze -- a
backward jump, violating the invariant that CLOCK_MONOTONIC never goes
backward.

The fix
-------

Only allow kvmclock_offset to move forward (or stay the same). If the
newly computed offset would be smaller than the current one, keep the
current value, so the guest clock keeps running forward and the freeze
duration is simply skipped from the guest's perspective -- which is the
expected behavior for a same-host pause/resume. This is a pure monotonic
protection with no time threshold; the normal forward path (guest boot,
KVM_CLOCK_REALTIME, cross-host migration toward a later destination host
clock) is unaffected.

    old_off = ka->kvmclock_offset;
    new_off = data.clock - now_raw_ns;
    if (new_off >= old_off)
            ka->kvmclock_offset = new_off;

Test
----

Inside the guest, pin a vCPU and call clock_gettime(CLOCK_MONOTONIC)
twice in a row, flagging any backward step. Over 50 snapshot
suspend/resume cycles the backward-step counter stays at 0 with this
patch, while the unpatched kernel reports a backward jump on every cycle
(measured freeze duration 27-36 ms).

To the best of our knowledge this scenario is not handled upstream. Any
feedback on whether monotonic protection is the right approach (vs. e.g.
teaching QEMU to feed a post-resume clock) would be much appreciated.

Signed-off-by: <Alex Chen> <alex.chen@huawei.com>
---
 arch/x86/kvm/x86.c | 19 ++++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 7a0d0e8e5c1f..000000000000 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -XXXX,XX +XXXX,XX @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
 	struct kvm_arch *ka = &kvm->arch;
 	struct kvm_clock_data data;
 	u64 now_raw_ns;
+	s64 old_off, new_off;
 	...
 	if (ka->use_master_clock)
 		now_raw_ns = ka->master_kernel_ns;
 	else
 		now_raw_ns = get_kvmclock_base_ns();
-	ka->kvmclock_offset = data.clock - now_raw_ns;
+
+	/*
+	 * Monotonic protection: data.clock is the kvmclock value userspace
+	 * saved at vm_stop (freeze instant); now_raw_ns is sampled at resume.
+	 * The naive kvmclock_offset = data.clock - now_raw_ns pulls the
+	 * offset back to the freeze instant, which makes pvti.system_time =
+	 * kernel_ns + kvmclock_offset jump backward by the freeze duration ->
+	 * guest vDSO CLOCK_MONOTONIC goes backward.
+	 *
+	 * Only allow kvmclock_offset to move forward (or stay). If the new
+	 * offset would shrink, keep the old one so the guest clock keeps
+	 * running forward (the freeze duration is simply skipped from the
+	 * guest's perspective, which is the desired behavior for a same-host
+	 * pause/resume). Pure monotonic protection, no time threshold.
+	 */
+	old_off = ka->kvmclock_offset;
+	new_off = data.clock - now_raw_ns;
+	if (new_off >= old_off)
+		ka->kvmclock_offset = new_off;
 	kvm_end_pvclock_update(kvm);

Re: [BUG REPORT] KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

[Dongli Zhang]

+ David.

FYI there were some prior discussions related to this topic.

David has a patch at KVM to enhance the KVM API to resolve the issue. It is
still not in mainline.

[PATCH v7 31/36] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock
migration
https://lore.kernel.org/all/20260728144954.355376-32-dwmw2@infradead.org

I used to try to enhance from QEMU side to avoid clock drift while accounting
for downtime.

[PATCH 1/1] target/i386/kvm: account blackout downtime for kvm-clock and guest TSC
https://lore.kernel.org/qemu-devel/20251009095831.46297-1-dongli.zhang@oracle.com

Here is my RCA about forward drift and backward drift from another perspective.

https://lore.kernel.org/all/935312be-9a86-49fd-8bb4-2c998a68e2df@oracle.com


On Sun, Nov 1, 2026 8:19:14AM -0800, Chenzhendong (alex) wrote:
> Hi,
> 
> While testing storage snapshots that suspend and resume a VM on the same
> host (vm_stop -> snapshot -> vm_start), we observed that the guest
> CLOCK_MONOTONIC jumps *backward* by the freeze duration after every
> resume.
> 
> The problem
> -----------
> 
> On a same-host pause/resume cycle, QEMU saves the kvmclock value at the
> freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via
> KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).
> 
> kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:
> 
>     kvmclock_offset = data.clock - now_raw_ns
> 
> with now_raw_ns sampled at T2. Since
> 
>     data.clock  = now_raw_ns(T1) + old_kvmclock_offset
> 
> this gives
> 
>     new_offset  = old_kvmclock_offset - freeze_duration
> 
> i.e. the offset is pulled back by the freeze duration.
> 
> The guest computes CLOCK_MONOTONIC from the pvclock page as:
> 
>     CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift

CLOCK_MONOTONIC and kvm-clock are different topics.

I would suggest here use kvm-clock or PV clock.

> 
> where system_time = kernel_ns + kvmclock_offset is filled by the host at
> each vCPU entry. After resume the host re-samples tsc_timestamp, so the
> (rdtsc - tsc_timestamp) term starts growing normally again and never
> goes backward. The only term that can shrink is system_time: because
> kvmclock_offset was pulled back by freeze_duration, the new system_time
> is freeze_duration smaller than the value the guest last observed before
> the freeze. The guest's next clock_gettime(CLOCK_MONOTONIC) therefore
> reads a value freeze_duration smaller than before the freeze -- a
> backward jump, violating the invariant that CLOCK_MONOTONIC never goes
> backward.
> 
> The fix
> -------
> 
> Only allow kvmclock_offset to move forward (or stay the same). If the

Ideally, neither forward nor backward drift is appreciated.

Thank you very much!

Dongli Zhang


> newly computed offset would be smaller than the current one, keep the
> current value, so the guest clock keeps running forward and the freeze
> duration is simply skipped from the guest's perspective -- which is the
> expected behavior for a same-host pause/resume. This is a pure monotonic
> protection with no time threshold; the normal forward path (guest boot,
> KVM_CLOCK_REALTIME, cross-host migration toward a later destination host
> clock) is unaffected.
> 
>     old_off = ka->kvmclock_offset;
>     new_off = data.clock - now_raw_ns;
>     if (new_off >= old_off)
>             ka->kvmclock_offset = new_off;
> 
> Test
> ----
> 
> Inside the guest, pin a vCPU and call clock_gettime(CLOCK_MONOTONIC)
> twice in a row, flagging any backward step. Over 50 snapshot
> suspend/resume cycles the backward-step counter stays at 0 with this
> patch, while the unpatched kernel reports a backward jump on every cycle
> (measured freeze duration 27-36 ms).
> 
> To the best of our knowledge this scenario is not handled upstream. Any
> feedback on whether monotonic protection is the right approach (vs. e.g.
> teaching QEMU to feed a post-resume clock) would be much appreciated.
> 
> Signed-off-by: <Alex Chen> <alex.chen@huawei.com>
> ---
>  arch/x86/kvm/x86.c | 19 ++++++++++++++++++++-
>  1 file changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 7a0d0e8e5c1f..000000000000 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -XXXX,XX +XXXX,XX @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
>  	struct kvm_arch *ka = &kvm->arch;
>  	struct kvm_clock_data data;
>  	u64 now_raw_ns;
> +	s64 old_off, new_off;
>  	...
>  	if (ka->use_master_clock)
>  		now_raw_ns = ka->master_kernel_ns;
>  	else
>  		now_raw_ns = get_kvmclock_base_ns();
> -	ka->kvmclock_offset = data.clock - now_raw_ns;
> +
> +	/*
> +	 * Monotonic protection: data.clock is the kvmclock value userspace
> +	 * saved at vm_stop (freeze instant); now_raw_ns is sampled at resume.
> +	 * The naive kvmclock_offset = data.clock - now_raw_ns pulls the
> +	 * offset back to the freeze instant, which makes pvti.system_time =
> +	 * kernel_ns + kvmclock_offset jump backward by the freeze duration ->
> +	 * guest vDSO CLOCK_MONOTONIC goes backward.
> +	 *
> +	 * Only allow kvmclock_offset to move forward (or stay). If the new
> +	 * offset would shrink, keep the old one so the guest clock keeps
> +	 * running forward (the freeze duration is simply skipped from the
> +	 * guest's perspective, which is the desired behavior for a same-host
> +	 * pause/resume). Pure monotonic protection, no time threshold.
> +	 */
> +	old_off = ka->kvmclock_offset;
> +	new_off = data.clock - now_raw_ns;
> +	if (new_off >= old_off)
> +		ka->kvmclock_offset = new_off;
>  	kvm_end_pvclock_update(kvm);
> 
>