* Re: x86/KVM: host reads guest IDT gates after VMexit on Cascade Lake
[not found] <CAL1mVc0W0oirnpd-im6r+Vewo6yfsWuXQB7UecEenYnE4BOJ4A@mail.gmail.com>
@ 2026-08-26 23:12 ` Sean Christopherson
2026-08-27 0:08 ` Jim Mattson
0 siblings, 1 reply; 2+ messages in thread
From: Sean Christopherson @ 2026-08-26 23:12 UTC (permalink / raw)
To: Yaohui Hu
Cc: kvm, linux-kernel, x86, Paolo Bonzini, Peter Zijlstra,
Dave Hansen, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Chao Gao
+Chao, who has helped with errata in the past
On Wed, Aug 26, 2026, Yaohui Hu wrote:
> Hi,
>
> We are chasing a host crash on Intel Cascade Lake that we believe is a
> CPU defect, but the exposure changed sharply across a kernel upgrade
> and we would like input on the kernel side.
>
> Symptom: nine hypervisor crashes over ~120 days, all the same shape.
> On a KVM vCPU thread, shortly after a VM-exit, the host faults at an
> address unmapped in the host, then faults again on every subsequent
> exception delivery, recursing until the #DF stack is exhausted.
>
> The phantom handler addresses are not random. In one dump we located
> the IDT they came from: it lives in the tmpfs-backed guest RAM of the
> VM running on that CPU. The host read gate descriptors out of the
> *guest's* IDT and tried to deliver host exceptions to guest handler
> addresses. A second dump shows this in software rather than by
> inference, with the oops registers caught in vmx_do_interrupt_irqoff()
> having read a garbage gate from host_idt_base.
>
> Linux maps the IDT through the CPU-entry-area alias at a fixed
> virtual address (0xfffffe0000000000) that KASLR never relocates, so a
> Linux guest and a Linux host use the same address for their own IDTs.
> With VPID enabled the guest's translation survives the VM-exit, and a
> host access tagged VPID 0000H must not match the guest-tagged entry.
> On these parts it appears to.
>
> Environment:
>
> - Xeon, family 6 model 85 stepping 7 (Cascade Lake-SP), microcode
> 0x05003901 (latest public for this stepping) on all nine hosts.
> - Two server vendors, three datacenters, VPID enabled.
> - 6.1.51 ran this fleet without such a crash; 6.12.51 produces
> them. Microcode and hardware are unchanged across the upgrade.
>
> There is a documented precedent for the failure class on a different
> part: Arrow Lake erratum ARL029, "Incorrect Core TLB Entry May be
> Retrieved Following VM Exit". Cascade Lake has no public equivalent.
>
> We diffed the relevant paths between the two versions. The gate read
> in handle_external_interrupt_irqoff() is functionally identical, and
> the TLB/VPID/EPT invalidation layer is byte-identical
> (vmx_flush_tlb_*, allocate_vpid, vmx_vcpu_load_vmcs, and the
> INVVPID/INVEPT sites), so we do not believe KVM changed its
> invalidation behaviour. The one change we found that plausibly alters
> exposure is 97e3d26b5e5f ("x86/mm: Randomize per-cpu entry area",
> v6.2), which changes the paging topology of exactly this region. We
> can see the effect in our builds, but we have no mechanism that
> predicts its sign.
>
> Questions:
>
> 1. Has anyone seen this signature? A host faulting after VM-exit at
> an address that turns out to belong to a guest seems distinctive
> enough to be memorable.
I don't recall seeing anything like this in our fleet, but I don't think we have
much exposure to v6.7+ kernels running VMs. I.e. we may not be seeing anything
purely because we haven't picked up the "bad" kernel, yet...
> 2. Is CEA randomization a plausible amplifier, or is there a better
> candidate in the 6.1..6.12 window that we have missed? We think
> this is a hardware defect either way; we are trying to explain why
> the same silicon and microcode behaved differently.
>
> 3. Would moving the host IDT off the fixed CEA address, or
> randomizing it per boot, be acceptable upstream as defence in
> depth? It removes the guest/host virtual-address collision the
> failure depends on, and unlike the KVM-side change below it also
> covers hardware event delivery, not just the software read.
Without knowing what's going wrong, hacking around something like this probably
isn't going to be a viable option. :-/
> 4. Suggestions for making this reproducible? We have a rig: a
> kvm-unit-tests guest that maps a recognizable IDT at the CEA
> address and drives ~7e7 VM-exits/s on one host, plus a host module
What types of exits, and what is the guest doing? If this is more or less the
same thing as ARL029, my read of the erratum is that it requires the CPU to be
reading the IDT at the time of exit, so that the TLB fill completes after VM-Exit
(and presumably gets tagged with VPID=0).
And given that at least one splat specifically traces to vmx_do_interrupt_irqoff(),
and that generally speaking KVM will only be forwarding events to the host via the
IDT when handling IRQs, I think it's a safe bet that the VM-Exits need to be IRQ
exits.
So, maybe try having the guest generate a horde of faults that along with INVLPG
opreations (or full TLB flushes?) to purge the IDT translation from the TLB, and
then spam the guest with IRQs?
> that validates the gate KVM reads on every exit. It sustains
> ~4.2e9 validated checks per minute and has not yet produced a
> mismatch. Our own estimate puts the target near one event per
> host-day, so this may only need runtime, but advice on which
> conditions to stress would help.
>
> We are aware of 28d11e4548b7 ("x86/fred: KVM: VMX: Always use FRED for
> IRQs when CONFIG_X86_FRED=y"), which removes the software IDT read for
> external interrupts and which we are backporting. As it was motivated
> by CFI and does not stop hardware from reading the IDT at the same
> address, we treat it as hardening rather than a fix.
>
> Thanks,
> Yaohui Hu
^ permalink raw reply [flat|nested] 2+ messages in thread