Re: [PATCH RFC v1 16/20] KVM: x86: Decode REX2 prefix in the emulator

[Paolo Bonzini <pbonzini@redhat.com>]

On 11/10/25 19:01, Chang S. Bae wrote:
> Extend the instruction emulator to recognize and interpret the REX2
> prefix byte. Also, detect and flag invalid prefix sequences after a REX2
> prefix.
> 
> In the existing prefix-decoding loop,
>    * The loop exits when the first non-prefix byte is encountered.
>    * Any non-REX prefix clears previously recorded REX information.
> 
> For REX2, however, once a REX2 prefix is encountered, most subsequent
> prefixes are invalid. So, each subsequent prefix needs to be validated
> before continuing the loop.
> 
> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
> ---
> RFC note:
> The REX2 decoding itself is straightforward. The additional logic is
> mainly to detect and handle invalid prefix sequences. If this seems
> excessive, there is a chance to cut off this check since VMX would raise
> '#UD' on such cases anyway.
> ---
>   arch/x86/kvm/emulate.c | 24 +++++++++++++++++++++++-
>   1 file changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
> index 9bd61ea496e5..f9381a4055d6 100644
> --- a/arch/x86/kvm/emulate.c
> +++ b/arch/x86/kvm/emulate.c
> @@ -4844,7 +4844,7 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
>   	ctxt->op_bytes = def_op_bytes;
>   	ctxt->ad_bytes = def_ad_bytes;
>   
> -	/* Legacy prefixes. */
> +	/* Legacy and REX/REX2 prefixes. */
>   	for (;;) {
>   		switch (ctxt->b = insn_fetch(u8, ctxt)) {
>   		case 0x66:	/* operand-size override */
> @@ -4887,9 +4887,20 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
>   		case 0x40 ... 0x4f: /* REX */
>   			if (mode != X86EMUL_MODE_PROT64)
>   				goto done_prefixes;
> +			if (ctxt->rex_prefix == REX2_PREFIX)
> +				break;
>   			ctxt->rex_prefix = REX_PREFIX;
>   			ctxt->rex.raw    = 0x0f & ctxt->b;
>   			continue;
> +		case 0xd5: /* REX2 */
> +			if (mode != X86EMUL_MODE_PROT64)
> +				goto done_prefixes;
Here you should also check

	if (ctxt->rex_prefix == REX_PREFIX) {
		ctxt->rex_prefix = REX2_INVALID;
		goto done_prefixes;
	}

> +			if (ctxt->rex_prefix == REX2_PREFIX &&
> +			    ctxt->rex.bits.m0 == 0)
> +				break;
> +			ctxt->rex_prefix = REX2_PREFIX;
> +			ctxt->rex.raw    = insn_fetch(u8, ctxt);
> +			continue;
After REX2 always comes the main opcode byte, so you can "goto 
done_prefixes" here.  Or even jump here already; in pseudocode:

	ctxt->b = insn_fetch(u8, ctxt);
	if (rex2 & REX_M)
		goto decode_twobyte;
	else
		goto decode_onebyte;

...

	if (ctxt->b == 0x0f) {
decode_twobyte:
		...
		if (ctxt->b == 0x38 && ctxt->rex_prefix != REX2_PREFIX)
			...
	} else {
decode_onebyte:
		...
	}



>   		case 0xf0:	/* LOCK */
>   			ctxt->lock_prefix = 1;
>   			break;
> @@ -4901,6 +4912,17 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
>   			goto done_prefixes;
>   		}
>   
> +		if (ctxt->rex_prefix == REX2_PREFIX) {
> +			/*
> +			 * A legacy or REX prefix following a REX2 prefix
> +			 * forms an invalid byte sequences. Likewise,
> +			 * a second REX2 prefix following a REX2 prefix
> +			 * with M0=0 is invalid.
> +			 */
> +			ctxt->rex_prefix = REX2_INVALID;
> +			goto done_prefixes;
> +		}

... and this is not needed.

Paolo

>   		/* Any legacy prefix after a REX prefix nullifies its effect. */
>   		ctxt->rex_prefix = REX_NONE;
>   		ctxt->rex.raw = 0;