[PATCH] KVM: x86: Emulate SYSENTER in 64-bit mode for Intel-compatible vCPUs

[PATCH] KVM: x86: Emulate SYSENTER in 64-bit mode for Intel-compatible vCPUs

[Hao Zhang]

From: Hao Zhang <zhanghao1@kylinos.cn>

Intel allows SYSENTER in both IA-32e submodes, whereas AMD raises #UD
in either submode.  On AMD hosts, KVM intercepts #UD and emulates
SYSENTER to support Intel-compatible vCPUs.

However, em_sysenter() unconditionally rejects execution from 64-bit
mode.  As a result, a valid SYSENTER in an Intel-compatible guest can
fail emulation instead of entering the guest's kernel.  For a
non-nested guest at CPL0, this results in a
KVM_INTERNAL_ERROR_EMULATION exit to userspace even when
KVM_CAP_EXIT_ON_EMULATION_FAILURE is not enabled.

Remove the unconditional rejection.  The existing IA-32e path already
loads the full-width SYSENTER EIP and ESP values and installs the
required CS and SS attributes.

Also reject SYSENTER for AMD-compatible vCPUs whenever EFER.LMA is
set, covering both compatibility and 64-bit modes.  This preserves
AMD's architectural #UD behavior instead of attempting Intel-style
emulation or reporting an emulation failure.

See the SYSENTER instruction descriptions in Intel SDM Volume 2 and
AMD APM Volume 3.

Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
---
 arch/x86/kvm/emulate.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index c1b21282187f..695957ebd871 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -2413,17 +2413,13 @@ static int em_sysenter(struct x86_emulate_ctxt *ctxt)
 		return emulate_gp(ctxt, 0);
 
 	/*
-	 * Intel's architecture allows SYSENTER in compatibility mode, but AMD
-	 * does not.  Note, AMD does allow SYSENTER in legacy protected mode.
+	 * Intel allows SYSENTER in both IA-32e submodes, but AMD does not
+	 * allow SYSENTER in either.  Both allow it in legacy protected mode.
 	 */
-	if ((ctxt->mode != X86EMUL_MODE_PROT64) && (efer & EFER_LMA) &&
+	if ((efer & EFER_LMA) &&
 	    !ctxt->ops->guest_cpuid_is_intel_compatible(ctxt))
 		return emulate_ud(ctxt);
 
-	/* sysenter/sysexit have not been tested in 64bit mode. */
-	if (ctxt->mode == X86EMUL_MODE_PROT64)
-		return X86EMUL_UNHANDLEABLE;
-
 	ops->get_msr(ctxt, MSR_IA32_SYSENTER_CS, &msr_data);
 	if ((msr_data & 0xfffc) == 0x0)
 		return emulate_gp(ctxt, 0);

base-commit: 6c377d19d4a5116d9bec5203aa3c6c11523e7898
-- 
2.15.0

Re: [PATCH] KVM: x86: Emulate SYSENTER in 64-bit mode for Intel-compatible vCPUs

[Sean Christopherson]

On Wed, Oct 21, 2026, Hao Zhang wrote:
> From: Hao Zhang <zhanghao1@kylinos.cn>
> 
> Intel allows SYSENTER in both IA-32e submodes, whereas AMD raises #UD
> in either submode.  On AMD hosts, KVM intercepts #UD and emulates
> SYSENTER to support Intel-compatible vCPUs.
> 
> However, em_sysenter() unconditionally rejects execution from 64-bit
> mode.  As a result, a valid SYSENTER in an Intel-compatible guest can
> fail emulation instead of entering the guest's kernel.  For a
> non-nested guest at CPL0, this results in a
> KVM_INTERNAL_ERROR_EMULATION exit to userspace even when
> KVM_CAP_EXIT_ON_EMULATION_FAILURE is not enabled.
> 
> Remove the unconditional rejection.  

Why?  I'm not inclined to expand the emulator's capabilities for something like
this without a *really* good reason to do so.