Re: [RFC PATCH v2 6/5] KVM: TDX: Explicitly do WBINVD upon reboot notifier

["Huang, Kai" <kai.huang@intel.com>]

On Fri, 2025-06-13 at 13:36 +0200, Paolo Bonzini wrote:
> On 5/10/25 13:25, Kai Huang wrote:
> > On TDX platforms, during kexec, the kernel needs to make sure there's no
> > dirty cachelines of TDX private memory before booting to the new kernel
> > to avoid silent memory corruption to the new kernel.
> > 
> > During kexec, the kexec-ing CPU firstly invokes native_stop_other_cpus()
> > to stop all remote CPUs before booting to the new kernel.  The remote
> > CPUs will then execute stop_this_cpu() to stop themselves.
> > 
> > The kernel has a percpu boolean to indicate whether the cache of a CPU
> > may be in incoherent state.  In stop_this_cpu(), the kernel does WBINVD
> > if that percpu boolean is true.
> > 
> > TDX turns on that percpu boolean on a CPU when the kernel does SEAMCALL.
> > This makes sure the cahces will be flushed during kexec.
> > 
> > However, the native_stop_other_cpus() and stop_this_cpu() have a "race"
> > which is extremely rare to happen but if did could cause system to hang.
> 
> s/if did//
> 
> > Specifically, the native_stop_other_cpus() firstly sends normal reboot
> > IPI to remote CPUs and wait one second for them to stop.  If that times
> > out, native_stop_other_cpus() then sends NMIs to remote CPUs to stop
> > them.
> > 
> > The aforementioned race happens when NMIs are sent.  Doing WBINVD in
> > stop_this_cpu() makes each CPU take longer time to stop and increases
> > the chance of the race to happen.
> > 
> > Register reboot notifier in KVM to explcitly flush caches upon reboot
> > for TDX.  This brings doing WBINVD at earlier stage and aovids the
> > WBINVD in stop_this_cpu(), eliminating the possibility of increasing the
> > chance of the aforementioned race.
> 
> "This moves the WBINVD to an earlier stage than stop_this_cpus(), 
> avoiding a possibly lengthy operation at a time where it could cause 
> this race."
> 
> Acked-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> Waiting for v3. :)

Thanks Paolo, and will do!