Re: [PATCH v1 1/1] x86/fred: Clear the WFE bit in missing-ENDBRANCH #CP

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 12/09/2024 9:53 am, Xin Li wrote:
> On 9/11/2024 5:22 PM, Dave Hansen wrote:
>> On 9/11/24 16:19, Xin Li (Intel) wrote:
>>> +/*
>>> + * The WFE (WAIT_FOR_ENDBRANCH) bit in the augmented CS of FRED
>>> stack frame is
>>> + * set to 1 in missing-ENDBRANCH #CP exceptions.
>>
>> I think there's a bit of relatively irrelevant info in there.  For
>> instance, I don't think it's super important to mention that FRED is
>> involved and where the WFE bit is in memory.
>>
>> FRED's involvement is kinda a no-brainer from the whole X86_FEATURE_FRED
>> thing, and if you're reading exception handler code and don't know that
>> 'regs' is on the stack, this probably isn't the place to explain that.
>
> I often find myself in a dilemma, should I mention some technical
> background which sometimes could also be distracting :(
>
> Based on your feedback, maybe the following is better?
>
> static void ibt_clear_fred_wfe(struct pt_regs *regs)
> {
>     if (regs->fred_cs.wfe)
>         regs->fred_cs.wfe = 0;
> }

static void ibt_clear_fred_wfe(struct pt_regs *regs)
{
        regs->fred_cs.wfe = 0;
}

would be better.  With any luck, the compiler would drop the if() on
your behalf, but it would still be better not to have it to start with.

>
> And we know only FRED hardware will set the WFE bit.
>
>>
>>> + * If the WFE bit is left as 1, the CPU will generate another
>>> missing-ENDBRANCH
>>> + * #CP because the indirect branch tracker will be set in the
>>> WAIT_FOR_ENDBRANCH
>>> + * state upon completion of the following ERETS instruction and the
>>> CPU will
>>> + * restart from the IP that just caused a previous
>>> missing-ENDBRANCH #CP.
>>> + *
>>> + * Clear the WFE bit to avoid dead looping due to the above reason.
>>> + */
>>> +static void ibt_clear_fred_wfe(struct pt_regs *regs)
>>> +{
>>> +    if (cpu_feature_enabled(X86_FEATURE_FRED))
>>> +        regs->fred_cs.wfe = 0;
>>> +}
>>
>> Can I suggest a slightly different comment?
>>
>> /*
>>   * WFE==1 (WAIT_FOR_ENDBRANCH) means that the CPU expects the next
>> ERETS
>>   * to jump to an ENDBR instruction. If the ENDBR is missing, the CPU
>>   * raises a #CP.
>>   *
>>   * Clear WFE to avoid that #CP.
>>   *
>>   * Use this function in a #CP handler to effectively give the next
>>   * ERETS a free pass to ignore IBT for a single instruction.
>>   */
>>
>> I think original comment really needs a "How do I use this?" sentence or
>> two.
>>
>> A comment at the call site also wouldn't hurt:
>>
>>       if (unlikely(regs->ip == (unsigned long)&ibt_selftest_noendbr)){
>>           regs->ax = 0;
>> +        /* Disable IBT enforcement for one exception return: */
>> +        ibt_clear_fred_wfe(regs);
>>           return;
>>       }
>>
>> I'm finding it kinda hard to concisely differentiate between the
>> "disable IBT at one ERETS" and "disable IBT forever", but I hope this
>> sounds good to folks.
>>
>
> My understanding is that a missing-ENDBRANCH #CP is triggered in two
> steps:
>
>     1) Upon completion of an indirect call/jmp, or an event return
>        instruction, the CPU indirect branch tracker is put in the
>        WAIT_FOR_ENDBRANCH state.
>
>     2) As the CPU is in WAIT_FOR_ENDBRANCH state, if the instruction to
>        be executed is ENDBR, the CPU indirect branch tracker exits
>        WAIT_FOR_ENDBRANCH state, otherwise a #CP is generated.
>
> So this is more of "preserve WAIT_FOR_ENDBRANCH state" or not.
>
> IDT is unable to preserve WAIT_FOR_ENDBRANCH state when returning from
> event handling, which as Andrew mentioned is a security hole.

I just said hole, but that was the subtext :)


But yes - mentioning ERETS is unhelpful; it's not relevant, and it
confuses how this works.

In a FRED world, WFE will be restored for the interrupted context.  Then
there's an instruction boundary (normal #RESET/MC/INIT/INTR processing),
and only on decoding the next instruction might a #CP[endbr] be raised.

WFE is a state that crosses an instruction boundary.  It is very similar
to EFLAGS.RF, existing to alter the behaviour of the *next* instruction.

~Andrew