* Re: [PATCH v4 17/47] x86/kvm: Mark TSC as reliable when it's constant and nonstop
[not found] <ahnhnjvfIblFxTFX@google.com>
@ 2026-06-01 22:02 ` David Woodhouse
0 siblings, 0 replies; 2+ messages in thread
From: David Woodhouse @ 2026-06-01 22:02 UTC (permalink / raw)
To: seanjc
Cc: pbonzini, tglx, mingo, bp, dave.hansen, x86, kas, kys, haiyangz,
wei.liu, decui, longli, ajay.kaher, alexey.makhalov, jan.kiszka,
luto, peterz, jgross, daniel.lezcano, jstultz, hpa,
rick.p.edgecombe, vkuznets, bcm-kernel-feedback-list,
boris.ostrovsky, sboyd, kvm, linux-kernel, linux-coco,
linux-hyperv, virtualization, xen-devel, dwmw, thomas.lendacky,
nikunj, dwmw2, mhklinux, tglx, sashiko-reviews
[-- Attachment #1: Type: text/plain, Size: 2028 bytes --]
On Fri, 29 May 2026 11:57:34 -0700, Sean Christopherson wrote:
> On Fri, May 29, 2026, sashiko-bot@kernel.org wrote:
> > > diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
> > > index 909d3e5e5bcd5..4fe9c69bf40b3 100644
> > > --- a/arch/x86/kernel/kvm.c
> > > +++ b/arch/x86/kernel/kvm.c
> > [ ... ]
> > > @@ -1040,7 +1041,20 @@ static void __init kvm_init_platform(void)
> > [ ... ]
> > > - kvmclock_init();
> > > + /*
> > > + * If the TSC counts at a constant frequency across P/T states, counts
> > > + * in deep C-states, and the TSC hasn't been marked unstable, treat the
> > > + * TSC reliable, as guaranteed by KVM. Note, the TSC unstable check
> > > + * exists purely to honor the TSC being marked unstable via command
> > > + * line, any runtime detection of an unstable will happen after this.
> > > + */
> > > + tsc_is_reliable = boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
> > > + boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
> > > + !check_tsc_unstable();
> >
> > [Severity: High]
> > Does this evaluate check_tsc_unstable() too early to catch the command line
> > parameter?
>
> Huh, it does indeed.
>
> > It looks like kvm_init_platform() is called from setup_arch(), but the
> > tsc=unstable kernel parameter is parsed via __setup() later during
> > parse_args() in start_kernel().
> >
> > If check_tsc_unstable() evaluates to 0 here because the parameter hasn't
> > been parsed yet, wouldn't it incorrectly force X86_FEATURE_TSC_RELIABLE
> > and set prefer_tsc to true?
>
> Yep, but this is a pre-existing problem that goes all the way back to the original
> commit 7539b174aef4 ("x86: kvmguest: use TSC clocksource if invariant TSC is exposed").
>
> We could try to fix that, but I'm _very_ strongly inclined to add (yet another)
> patch to simply drop the check_tsc_unstable() since it has always been dead code.
Yeah, kill it with fire.
Reviewed-by: David Woodhouse <dwmw@amazon.co.uk>
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread
* [PATCH v4 00/47] x86: Try to wrangle PV clocks vs. TSC
@ 2026-05-29 14:43 Sean Christopherson
2026-05-29 14:44 ` [PATCH v4 17/47] x86/kvm: Mark TSC as reliable when it's constant and nonstop Sean Christopherson
0 siblings, 1 reply; 2+ messages in thread
From: Sean Christopherson @ 2026-05-29 14:43 UTC (permalink / raw)
To: Paolo Bonzini, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Kiryl Shutsemau, Sean Christopherson,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li,
Ajay Kaher, Alexey Makhalov, Jan Kiszka, Andy Lutomirski,
Peter Zijlstra, Juergen Gross, Daniel Lezcano, John Stultz
Cc: H. Peter Anvin, Rick Edgecombe, Vitaly Kuznetsov,
Broadcom internal kernel review list, Boris Ostrovsky,
Stephen Boyd, kvm, linux-kernel, linux-coco, linux-hyperv,
virtualization, xen-devel, David Woodhouse, Tom Lendacky,
Nikunj A Dadhania, David Woodhouse, Michael Kelley,
Thomas Gleixner
Well, the number of patches in the series is going in the wrong direction,
but I'm much happier with this version, which eschews the x86_platform
overrides entirely in favor of a fixed sequence for selecting the TSC/CPU
frequency "routine".
Given that previous versions had fatal NULL pointer deref bugs that affected
VMware and Xen, this series needs testing and acks from those maintainers.
The primary goal of this series to fix flaws with SNP and TDX guests where a
PV clock provided by the untrusted hypervisor is used instead of the secure
TSC that is controlled by trusted firmware.
The secondary goal is modernize running under KVM. Currently, KVM guests will
use TSC for clocksource, but not sched_clock. And Linux-as-a-KVM-guest doesn't
support paravirt enumeration of the TSC/APIC frequencies, even though QEMU
provides that information by default.
The tertiary goal is to clean up the PV clock code to deduplicate logic across
hypervisors, and to hopefully make it all easier to maintain going forward.
v4 also adds a quaternary goal of cleaning up the TSC calibration code, which
was made stupidly hard to follow by hypervisor code mixing in with the native
calibration routines, instead of being implemented as a pure alternative.
Lots more background on the SNP/TDX motiviation:
https://lore.kernel.org/all/20250106124633.1418972-13-nikunj@amd.com
As before, I deliberately omitted jailhouse-dev@googlegroups.com from the To/Cc,
as those emails bounced on v1, AFAICT nothing has changed.
Note, I deliberately didn't collect a few reviews as the patches changed quite
a bit from what was reviewed in v3.
v4:
- Use x86_init_noop() to skip save/restore on VMware and Xen instead of
nullifying x86_platform.{save,restore}_sched_clock_state. [Sashiko]
- Use '0' to indicate "failure" when getting the CPU frequency from CPUID, to
avoid using an out-param and thus make it all but impossible to
unintentionally clobber the global cpu_khz (which v3 did). [Sashiko]
- Rename cpuid_get_cpu_freq() => __cpu_khz_from_cpuid() to capture its
relationship with cpu_khz_from_cpuid().
- Compute lapic_timer_period in units of ticks, not Khz. [Sashiko]
- Kill off x86_platform_ops.calibrate_{cpu,tsc}(), and instead use dedicated
hooks for hypervisor code, and direct calls for TDX and SNP. [David, loosely]
- Drop SNP's secure TSC override of _CPU_ calibration, as there's zero
evidence it's justified or a net positive.
- Collect reviews/acks. [David, Wei]
- Decouple getting TSC/APIC frequencies from KVM PV CPUID from kvmclock. [David]
- Fix an amusing number of Opportunistically misspellings. [David]
- Set kvm_sched_clock_offset _before_ registering kvmclock as sched_clock,
and add a comment to guard against future goofs. [Sashiko]
- Keep "setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE)" in Hyper-V's handling
of HV_ACCESS_TSC_INVARIANT, as it's technically possible to have a VM
with HV_ACCESS_TSC_INVARIANT but not HV_ACCESS_FREQUENCY_MSRS. Though as
a _very_ nice side effect of using dedicated sequencing for selecting the
TSC frequency source, this would have naturally happened anyways. [Sashiko]
v3:
- https://lore.kernel.org/all/20260515191942.1892718-1-seanjc@google.com
- Collect reviews. [Michael, Thomas]
- Use Hyper-V reference counter / refcounter instead of Hyper-V timer. [Michael]
- Use the paravirt CPUID interface first proposed by VMware for KVM's
"official" mechanism for communicating frequency to KVM-aware guests,
instead of abusing Intel's CPUID leafs. [David]
- Deal with paravirt code being moved into asm/timers.h and
arch/x86/kernel/tsc.c.
v2:
- https://lore.kernel.org/all/Z8YWttWDtvkyCtdJ@google.com
- Add struct to hold the TSC CPUID output. [Boris]
- Don't pointlessly inline the TSC CPUID helpers. [Boris]
- Fix a variable goof in a helper, hopefully for real this time. [Dan]
- Collect reviews. [Nikunj]
- Override the sched_clock save/restore hooks if and only if a PV clock
is successfully registered.
- During resome, restore clocksources before reading persistent time.
- Clean up more warts created by kvmclock.
- Fix more bugs in kvmclock's suspend/resume handling.
- Try to harden kvmclock against future bugs.
v1: https://lore.kernel.org/all/20250201021718.699411-1-seanjc@google.com
David Woodhouse (3):
KVM: x86: Officially define CPUID 0x40000010 as PV Timing Info (TSC
and Bus)
x86/kvm: Obtain TSC frequency from PV CPUID if present
x86/xen: Obtain TSC frequency from CPUID if present
Sean Christopherson (44):
x86/tsc: Never re-calibrate TSC frequency if its exact timing is known
x86/tsc: Add a standalone helpers for getting TSC info from CPUID.0x15
x86/sev: Mark TSC as reliable when configuring Secure TSC
x86/sev: Don't override CPU frequency calibration for SNP's Secure TSC
x86/sev: Move check for SNP Secure TSC support to tsc_early_init()
x86/sev: Shove SNP's secure/trusted TSC frequency directly into
"calibration"
x86/tdx: Force TSC frequency with CPUID-based info provided by the
TDX-Module
x86/tsc: Add dedicated hypervisor hooks for getting known TSC/CPU
frequencies
x86/acrn: Mark TSC frequency as known when using ACRN for calibration
x86/tsc: Consolidate forcing of X86_FEATURE_TSC_KNOWN_FREQ for PV code
x86/tsc: Kill off x86_platform_ops.calibrate_{cpu,tsc}() hooks
x86/tsc: Rename pit_hpet_ptimer_calibrate_cpu() =>
native_calibrate_cpu_late()
x86/tsc: Fold native_calibrate_cpu() into recalibrate_cpu_khz()
x86/kvmclock: Rename kvm_get_tsc_khz() to kvmclock_get_tsc_khz()
x86/kvm: Mark TSC as reliable when it's constant and nonstop
x86/kvm: Get local APIC bus frequency from PV CPUID Timing Info
x86/tsc: Add standalone helper for getting CPU frequency from CPUID
x86/kvm: Get CPU base frequency from CPUID when it's available
clocksource: hyper-v: Register sched_clock save/restore iff it's
necessary
clocksource: hyper-v: Drop wrappers to sched_clock save/restore
helpers
clocksource: hyper-v: Don't save/restore TSC offset when using HV
sched_clock
x86/kvmclock: Setup kvmclock for secondary CPUs iff CONFIG_SMP=y
x86/kvm: Don't disable kvmclock on BSP in syscore_suspend()
x86/paravirt: Remove unnecessary PARAVIRT=n stub for
paravirt_set_sched_clock()
x86/paravirt: Move handling of unstable PV clocks into
paravirt_set_sched_clock()
x86/kvmclock: Move sched_clock save/restore helpers up in kvmclock.c
x86/xen/time: NOP-ify x86_platform's sched_clock save/restore hooks
x86/vmware: NOP-ify save/restore hooks when using VMware's sched_clock
x86/tsc: WARN if TSC sched_clock save/restore used with PV sched_clock
x86/paravirt: Pass sched_clock save/restore helpers during
registration
x86/kvmclock: Move kvm_sched_clock_init() down in kvmclock.c
x86/xen/time: Mark xen_setup_vsyscall_time_info() as __init
x86/pvclock: Mark setup helpers and related various as
__init/__ro_after_init
x86/pvclock: WARN if pvclock's valid_flags are overwritten
x86/kvmclock: Refactor handling of PVCLOCK_TSC_STABLE_BIT during
kvmclock_init()
timekeeping: Resume clocksources before reading persistent clock
x86/kvmclock: Hook clocksource.suspend/resume when kvmclock isn't
sched_clock
x86/kvmclock: WARN if wall clock is read while kvmclock is suspended
x86/paravirt: Mark __paravirt_set_sched_clock() as __init
x86/paravirt: Plumb a return code into __paravirt_set_sched_clock()
x86/paravirt: Don't use a PV sched_clock in CoCo guests with trusted
TSC
x86/kvmclock: Use TSC for sched_clock if it's constant and non-stop
x86/kvmclock: Plumb in AP-online and BSP-resume to kvmlock, for
documentation
x86/paravirt: Move using_native_sched_clock() stub into timer.h
Documentation/virt/kvm/x86/cpuid.rst | 12 ++
arch/x86/coco/sev/core.c | 21 +--
arch/x86/coco/tdx/tdx.c | 19 ++-
arch/x86/include/asm/acrn.h | 5 -
arch/x86/include/asm/kvm_para.h | 12 +-
arch/x86/include/asm/sev.h | 4 +-
arch/x86/include/asm/tdx.h | 2 +
arch/x86/include/asm/timer.h | 15 +-
arch/x86/include/asm/tsc.h | 11 +-
arch/x86/include/asm/x86_init.h | 8 +-
arch/x86/include/uapi/asm/kvm_para.h | 11 ++
arch/x86/kernel/cpu/acrn.c | 10 +-
arch/x86/kernel/cpu/mshyperv.c | 65 +-------
arch/x86/kernel/cpu/vmware.c | 13 +-
arch/x86/kernel/jailhouse.c | 7 +-
arch/x86/kernel/kvm.c | 108 +++++++++++--
arch/x86/kernel/kvmclock.c | 208 ++++++++++++++++---------
arch/x86/kernel/pvclock.c | 9 +-
arch/x86/kernel/tsc.c | 218 +++++++++++++++++----------
arch/x86/kernel/x86_init.c | 2 -
arch/x86/mm/mem_encrypt_amd.c | 3 -
arch/x86/xen/time.c | 25 ++-
drivers/clocksource/hyperv_timer.c | 38 +++--
include/clocksource/hyperv_timer.h | 2 -
kernel/time/timekeeping.c | 9 +-
25 files changed, 533 insertions(+), 304 deletions(-)
base-commit: 4678d11f294de0fd295a265e02955b5d1a4a2684
--
2.54.0.823.g6e5bcc1fc9-goog
^ permalink raw reply [flat|nested] 2+ messages in thread* [PATCH v4 17/47] x86/kvm: Mark TSC as reliable when it's constant and nonstop
2026-05-29 14:43 [PATCH v4 00/47] x86: Try to wrangle PV clocks vs. TSC Sean Christopherson
@ 2026-05-29 14:44 ` Sean Christopherson
0 siblings, 0 replies; 2+ messages in thread
From: Sean Christopherson @ 2026-05-29 14:44 UTC (permalink / raw)
To: Paolo Bonzini, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Kiryl Shutsemau, Sean Christopherson,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li,
Ajay Kaher, Alexey Makhalov, Jan Kiszka, Andy Lutomirski,
Peter Zijlstra, Juergen Gross, Daniel Lezcano, John Stultz
Cc: H. Peter Anvin, Rick Edgecombe, Vitaly Kuznetsov,
Broadcom internal kernel review list, Boris Ostrovsky,
Stephen Boyd, kvm, linux-kernel, linux-coco, linux-hyperv,
virtualization, xen-devel, David Woodhouse, Tom Lendacky,
Nikunj A Dadhania, David Woodhouse, Michael Kelley,
Thomas Gleixner
Mark the TSC as reliable if the hypervisor (KVM) has enumerated the TSC
as constant and nonstop, and the admin hasn't explicitly marked the TSC
as unstable. Like most (all?) virtualization setups, any secondary
clocksource that's used as a watchdog is guaranteed to be less reliable
than a constant, nonstop TSC, as all clocksources the kernel uses as a
watchdog are all but guaranteed to be emulated when running as a KVM
guest. I.e. any observed discrepancies between the TSC and watchdog will
be due to jitter in the watchdog.
This is especially true for KVM, as the watchdog clocksource is usually
emulated in host userspace, i.e. reading the clock incurs a roundtrip
cost of thousands of cycles.
Marking the TSC reliable addresses a flaw where the TSC will occasionally
be marked unstable if the host is under moderate/heavy load.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/include/asm/kvm_para.h | 2 +-
arch/x86/kernel/kvm.c | 16 +++++++++++++++-
arch/x86/kernel/kvmclock.c | 15 +++++----------
3 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/kvm_para.h b/arch/x86/include/asm/kvm_para.h
index 4a47c16e2df8..4a49fc286b4c 100644
--- a/arch/x86/include/asm/kvm_para.h
+++ b/arch/x86/include/asm/kvm_para.h
@@ -118,7 +118,7 @@ static inline long kvm_sev_hypercall3(unsigned int nr, unsigned long p1,
}
#ifdef CONFIG_KVM_GUEST
-void kvmclock_init(void);
+void kvmclock_init(bool prefer_tsc);
void kvmclock_disable(void);
bool kvm_para_available(void);
unsigned int kvm_arch_para_features(void);
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 909d3e5e5bcd..4fe9c69bf40b 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -978,6 +978,7 @@ static void __init kvm_init_platform(void)
.mask_hi = (BIT_ULL(boot_cpu_data.x86_phys_bits) - 1) >> 32,
};
u32 timing_info_leaf;
+ bool tsc_is_reliable;
if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL)) {
@@ -1040,7 +1041,20 @@ static void __init kvm_init_platform(void)
}
}
- kvmclock_init();
+ /*
+ * If the TSC counts at a constant frequency across P/T states, counts
+ * in deep C-states, and the TSC hasn't been marked unstable, treat the
+ * TSC reliable, as guaranteed by KVM. Note, the TSC unstable check
+ * exists purely to honor the TSC being marked unstable via command
+ * line, any runtime detection of an unstable will happen after this.
+ */
+ tsc_is_reliable = boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
+ boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
+ !check_tsc_unstable();
+ if (tsc_is_reliable)
+ setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);
+
+ kvmclock_init(tsc_is_reliable);
x86_platform.apic_post_init = kvm_apic_init;
/*
diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
index 404f60741aa8..69a15fbfb779 100644
--- a/arch/x86/kernel/kvmclock.c
+++ b/arch/x86/kernel/kvmclock.c
@@ -285,7 +285,7 @@ static int kvmclock_setup_percpu(unsigned int cpu)
return p ? 0 : -ENOMEM;
}
-void __init kvmclock_init(void)
+void __init kvmclock_init(bool prefer_tsc)
{
u8 flags;
@@ -334,16 +334,11 @@ void __init kvmclock_init(void)
kvm_get_preset_lpj();
/*
- * X86_FEATURE_NONSTOP_TSC is TSC runs at constant rate
- * with P/T states and does not stop in deep C-states.
- *
- * Invariant TSC exposed by host means kvmclock is not necessary:
- * can use TSC as clocksource.
- *
+ * If TSC is preferred over kvmlock, drop kvmclock's rating so that TSC
+ * is chosen as the clocksource (but still register kvmclock in case
+ * the kernel doesn't want to use TSC for whatever reason).
*/
- if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
- boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
- !check_tsc_unstable())
+ if (prefer_tsc)
kvm_clock.rating = 299;
clocksource_register_hz(&kvm_clock, NSEC_PER_SEC);
--
2.54.0.823.g6e5bcc1fc9-goog
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-06-01 22:02 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <ahnhnjvfIblFxTFX@google.com>
2026-06-01 22:02 ` [PATCH v4 17/47] x86/kvm: Mark TSC as reliable when it's constant and nonstop David Woodhouse
2026-05-29 14:43 [PATCH v4 00/47] x86: Try to wrangle PV clocks vs. TSC Sean Christopherson
2026-05-29 14:44 ` [PATCH v4 17/47] x86/kvm: Mark TSC as reliable when it's constant and nonstop Sean Christopherson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®