Re: objtool warning for next-20221118

[Andrew Cooper <Andrew.Cooper3@citrix.com>]

On 23/11/2022 08:55, Peter Zijlstra wrote:
> On Tue, Nov 22, 2022 at 05:23:50PM -0800, Josh Poimboeuf wrote:
>> On Tue, Nov 22, 2022 at 09:35:17AM +0100, Peter Zijlstra wrote:
>>> On Mon, Nov 21, 2022 at 09:16:05PM -0800, Josh Poimboeuf wrote:
>>>
>>>> It's complaining about an unreachable instruction after a call to
>>>> arch_cpu_idle_dead().  In this case objtool detects the fact
>>>> arch_cpu_idle_dead() doesn't return due to its call to the
>>>> non-CONFIG_SMP version of play_dead().  But GCC has no way of detecting
>>>> that because the caller is in another translation unit.
>>>>
>>>> As far as I can tell, that function should never return.  Though it
>>>> seems to have some dubious semantics (see xen_pv_play_dead() for
>>>> example, which *does* seem to return?).  I'm thinking it would be an
>>>> improvement to enforce that noreturn behavior across all arches and
>>>> platforms, sprinkling __noreturn and BUG() on arch_cpu_idle_dead() and
>>>> maybe some of it callees, where needed.
>>>>
>>>> Peter, what do you think?  I could attempt a patch.
>>> I'm thinking the Xen case makes all this really rather difficult :/
>>>
>>> While normally a CPU is brought up through a trampoline, Xen seems to
>>> have implemented it by simply returning from play_dead(), and afaict
>>> that is actually a valid way to go about doing it.
>> o_O
>>
>> How the @#$% is that a valid way of doing it?  Why not just do it the
>> normal way?
> Well, if you return from arch_cpu_idle_dead() you're back in the idle
> loop -- exactly where you would be if you were to bootstrap the whole
> CPU -- provided you have it remember the whole state (easier with a
> vCPU).
>
> But maybe I'm missing something, lets add Xen folks on.

Calling VCPUOP_down on oneself always succeeds, but all it does is
deschedule the vCPU.

It can be undone at a later point by a different vcpu issuing VCPUOP_up
against the previously-downed CPU, at which point the vCPU gets rescheduled.

This is why the VCPUOP_down hypercall returns normally.  All state
really is intact.

As for what Linux does, this is how xen_pv_cpu_up() currently behaves. 
If you want to make Xen behave more everything else, then bug a BUG()
after VCPUOP_down, and adjust xen_pv_cpu_up() to skip its initialised
check and always use VCPUOP_initialise to bring the vCPU back online.

~Andrew