Re: WARNING: can't access registers at asm_common_interrupt

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 11/11/2020 18:13, Josh Poimboeuf wrote:
> On Wed, Nov 11, 2020 at 06:47:36PM +0100, Peter Zijlstra wrote:
>> This is PARAVIRT_XXL only, which is a Xen special. My preference, as
>> always, is to kill it... Sadly the Xen people have a different opinion.
> That would be soooo nice... then we could get rid of paravirt patching
> altogether and replace it with static calls.
>
>>> Objtool doesn't know about the pushf/pop paravirt patch, so ORC gets
>>> confused by the changed stack layout.
>>>
>>> I'm thinking we either need to teach objtool how to deal with
>>> save_fl/restore_fl patches, or we need to just get rid of those nasty
>>> patches somehow.  Peter, any thoughts?
>> Don't use Xen? ;-)
>>
>> So with PARAVIRT_XXL the compiler will emit something like:
>>
>>   "CALL *pvops.save_fl"
>>
>> Which we then overwrite at runtime with "pushf; pop %[re]ax" and a few
>> NOPs.
>>
>> Now, objtool understands alternatives, and ensures they have the same
>> stack layout, it has no chance in hell of understanding this, simply
>> because paravirt_patch.c is magic.
>>
>> I don't have any immediate clever ideas, but let me ponder it a wee bit.

Well...

static_calls are a newer, and more generic, form of pvops.  Most of the
magic is to do with inlining small fragments, but static calls can do
that now too, IIRC?

>> ....
>>
>> Something really disguisting we could do is recognise the indirect call
>> offset and emit an extra ORC entry for RIP+1. So the cases are:
>>
>> 	CALL *pv_ops.save_fl	-- 7 bytes IIRC
>> 	CALL $imm;		-- 5 bytes
>> 	PUSHF; POP %[RE]AX	-- 2 bytes
>>
>> so the RIP+1 (the POP insn) will only ever exist in this case. The
>> indirect and direct call cases would never land on that IP.
> I had a similar idea, and a bit of deja vu - we may have talked about
> this before.  At least I know we talked about doing something similar
> for alternatives which muck with the stack.

The main complexity with pvops is that the

    CALL *pv_ops.save_fl

form needs to be usable from extremely early in the day (pre general
patching), hence the use of function pointers and some non-standard ABIs.

For performance reasons, the end result of this pvop wants to be `pushf;
pop %[re]ax` in then native case, and `call xen_pv_save_fl` in the Xen
case, but this doesn't mean that the compiled instruction needs to be a
function pointer to begin with.

Would objtool have an easier time coping if this were implemented in
terms of a static call?

~Andrew