arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Mauro Rossi]

Hi Andy,

I am an hobbyist contributing to android-x86 FOSS project lead by
Chih-Huwei Huang (in Cc: for information/alignement)

I am performing periodic tests to build kernel for Android 11 based iso image
which relies on aosp shipped prebuild clang toolchain (clang version 11.0.2)

When building linux 5.18rc4 and also with linux 5.17 x86_64 64bit kernel targets
there is a building error in arch/x86/entry

  AS      arch/x86/entry/entry_64.o
<instantiation>:2:2: error: unknown use of instruction mnemonic
without a size suffix
 lsl %rax, %rax
 ^
<instantiation>:1:1: note: while in macro instantiation
LOAD_CPU_AND_NODE_SEG_LIMIT %rax
^
<instantiation>:2:2: note: while in macro instantiation
 GET_PERCPU_BASE %rax
 ^
/home/utente/r-x86_kernel/kernel/arch/x86/entry/entry_64.S:890:2:
note: while in macro instantiation
 SAVE_AND_SET_GSBASE scratch_reg=%rax save_reg=%rbx
 ^
make[3]: *** [/home/utente/r-x86_kernel/kernel/scripts/Makefile.build:389:
arch/x86/entry/entry_64.o] Error 1
make[2]: *** [/home/utente/r-x86_kernel/kernel/scripts/Makefile.build:550:
arch/x86/entry] Error 2
make[1]: *** [/home/utente/r-x86_kernel/kernel/Makefile:1887: arch/x86] Error 2
make[1]: *** Waiting for unfinished jobs....

As other interesting info, the building error does not happen when
building x86 32bit kernel target and i can build 86_64 64bit kernel
target only by setting the LLVM_IAS=0 parameter to disable the
internal llvm assembler

I wanted to ask you if you could help us, if there could be a way to
improve arch/x86/entry/entry_64.S code to be able to complete the
build without disabling the llvm internal assembler.

I don't know if this building error may be caused by the clang version
11.0.2, but at some point the aosp and android version may hit this
same issue,
so I wanted to highlight this issue to you to have a competent person feedback,
as I am more a "trial and error" guy than a kernel expert

Thanks in advance for any info

Mauro Rossi
android-x86 team

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Nathan Chancellor]

Hi Mauro,

[+ llvm@lists.linux.dev and Nick]

On Fri, Apr 29, 2022 at 07:40:07PM +0200, Mauro Rossi wrote:
> Hi Andy,
> 
> I am an hobbyist contributing to android-x86 FOSS project lead by
> Chih-Huwei Huang (in Cc: for information/alignement)
> 
> I am performing periodic tests to build kernel for Android 11 based iso image
> which relies on aosp shipped prebuild clang toolchain (clang version 11.0.2)
> 
> When building linux 5.18rc4 and also with linux 5.17 x86_64 64bit kernel targets
> there is a building error in arch/x86/entry
> 
>   AS      arch/x86/entry/entry_64.o
> <instantiation>:2:2: error: unknown use of instruction mnemonic
> without a size suffix
>  lsl %rax, %rax
>  ^
> <instantiation>:1:1: note: while in macro instantiation
> LOAD_CPU_AND_NODE_SEG_LIMIT %rax
> ^
> <instantiation>:2:2: note: while in macro instantiation
>  GET_PERCPU_BASE %rax
>  ^
> /home/utente/r-x86_kernel/kernel/arch/x86/entry/entry_64.S:890:2:
> note: while in macro instantiation
>  SAVE_AND_SET_GSBASE scratch_reg=%rax save_reg=%rbx
>  ^
> make[3]: *** [/home/utente/r-x86_kernel/kernel/scripts/Makefile.build:389:
> arch/x86/entry/entry_64.o] Error 1
> make[2]: *** [/home/utente/r-x86_kernel/kernel/scripts/Makefile.build:550:
> arch/x86/entry] Error 2
> make[1]: *** [/home/utente/r-x86_kernel/kernel/Makefile:1887: arch/x86] Error 2
> make[1]: *** Waiting for unfinished jobs....
> 
> As other interesting info, the building error does not happen when
> building x86 32bit kernel target and i can build 86_64 64bit kernel
> target only by setting the LLVM_IAS=0 parameter to disable the
> internal llvm assembler

This error was fixed in LLVM 11.0.0 final, which was released after
Android's LLVM 11.0.2:

https://github.com/ClangBuiltLinux/linux/issues/1079

> I wanted to ask you if you could help us, if there could be a way to
> improve arch/x86/entry/entry_64.S code to be able to complete the
> build without disabling the llvm internal assembler.
> 
> I don't know if this building error may be caused by the clang version
> 11.0.2, but at some point the aosp and android version may hit this
> same issue,
> so I wanted to highlight this issue to you to have a competent person feedback,
> as I am more a "trial and error" guy than a kernel expert

I am open to other opinions but I am not inclined to suggest working
around this in the kernel for two reasons:

1. This issue was resolved in the toolchain almost two years ago, so it
   is not a recent failure.

2. Android's LLVM 11.0.2 is technically older than the minimum version
   that the kernel supports (11.0.0), which I would argue means it is
   unsupported. 11.0.0 final was released on October 12th, 2020 and
   Android's LLVM 11.0.2 was committed on June 11th, 2020, so you are
   potentially missing four months worth of fixes:

   https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86/+/431c74471920f3f9b0517692fb69515c023bde41

   Unfortunately, due to the way that LLVM versions work, it is not so
   easy to check for this but perhaps we should consider trying to
   handle this, so that others don't continue to trip over old bugs.

Moving to LLVM_IAS=0 is the solution that we went with for clang-10 when
it was supported after the switch to the integrated assembler by
default, which I do not think is an unreasonable solution for this
issue.

Alternatively, you could apply the hack that Nick inserted into Android
for this issue if upgrading your toolchain or turning off the integrated
assembler is not possible:

https://android.googlesource.com/kernel/common/+/e58f084735b8abf744d61083b92172ee23d45aae

I really do not mean to sound dismissive or rude, I apologize if it
comes off that way, but we have worked quite hard to avoid inserting
unnecessary workarounds, as they are ultimately technical debt that can
be hard to manage over the long term.

Cheers,
Nathan

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Mauro Rossi]

On Sat, Apr 30, 2022 at 11:30 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> Hi Mauro,
>
> [+ llvm@lists.linux.dev and Nick]
>
> On Fri, Apr 29, 2022 at 07:40:07PM +0200, Mauro Rossi wrote:
> > Hi Andy,
> >
> > I am an hobbyist contributing to android-x86 FOSS project lead by
> > Chih-Huwei Huang (in Cc: for information/alignement)
> >
> > I am performing periodic tests to build kernel for Android 11 based iso image
> > which relies on aosp shipped prebuild clang toolchain (clang version 11.0.2)
> >
> > When building linux 5.18rc4 and also with linux 5.17 x86_64 64bit kernel targets
> > there is a building error in arch/x86/entry
> >
> >   AS      arch/x86/entry/entry_64.o
> > <instantiation>:2:2: error: unknown use of instruction mnemonic
> > without a size suffix
> >  lsl %rax, %rax
> >  ^
> > <instantiation>:1:1: note: while in macro instantiation
> > LOAD_CPU_AND_NODE_SEG_LIMIT %rax
> > ^
> > <instantiation>:2:2: note: while in macro instantiation
> >  GET_PERCPU_BASE %rax
> >  ^
> > /home/utente/r-x86_kernel/kernel/arch/x86/entry/entry_64.S:890:2:
> > note: while in macro instantiation
> >  SAVE_AND_SET_GSBASE scratch_reg=%rax save_reg=%rbx
> >  ^
> > make[3]: *** [/home/utente/r-x86_kernel/kernel/scripts/Makefile.build:389:
> > arch/x86/entry/entry_64.o] Error 1
> > make[2]: *** [/home/utente/r-x86_kernel/kernel/scripts/Makefile.build:550:
> > arch/x86/entry] Error 2
> > make[1]: *** [/home/utente/r-x86_kernel/kernel/Makefile:1887: arch/x86] Error 2
> > make[1]: *** Waiting for unfinished jobs....
> >
> > As other interesting info, the building error does not happen when
> > building x86 32bit kernel target and i can build 86_64 64bit kernel
> > target only by setting the LLVM_IAS=0 parameter to disable the
> > internal llvm assembler
>
> This error was fixed in LLVM 11.0.0 final, which was released after
> Android's LLVM 11.0.2:
>
> https://github.com/ClangBuiltLinux/linux/issues/1079
>
> > I wanted to ask you if you could help us, if there could be a way to
> > improve arch/x86/entry/entry_64.S code to be able to complete the
> > build without disabling the llvm internal assembler.
> >
> > I don't know if this building error may be caused by the clang version
> > 11.0.2, but at some point the aosp and android version may hit this
> > same issue,
> > so I wanted to highlight this issue to you to have a competent person feedback,
> > as I am more a "trial and error" guy than a kernel expert
>
> I am open to other opinions but I am not inclined to suggest working
> around this in the kernel for two reasons:
>
> 1. This issue was resolved in the toolchain almost two years ago, so it
>    is not a recent failure.
>
> 2. Android's LLVM 11.0.2 is technically older than the minimum version
>    that the kernel supports (11.0.0), which I would argue means it is
>    unsupported. 11.0.0 final was released on October 12th, 2020 and
>    Android's LLVM 11.0.2 was committed on June 11th, 2020, so you are
>    potentially missing four months worth of fixes:
>
>    https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86/+/431c74471920f3f9b0517692fb69515c023bde41
>
>    Unfortunately, due to the way that LLVM versions work, it is not so
>    easy to check for this but perhaps we should consider trying to
>    handle this, so that others don't continue to trip over old bugs.
>
> Moving to LLVM_IAS=0 is the solution that we went with for clang-10 when
> it was supported after the switch to the integrated assembler by
> default, which I do not think is an unreasonable solution for this
> issue.
>
> Alternatively, you could apply the hack that Nick inserted into Android
> for this issue if upgrading your toolchain or turning off the integrated
> assembler is not possible:
>
> https://android.googlesource.com/kernel/common/+/e58f084735b8abf744d61083b92172ee23d45aae
>
> I really do not mean to sound dismissive or rude, I apologize if it
> comes off that way, but we have worked quite hard to avoid inserting
> unnecessary workarounds, as they are ultimately technical debt that can
> be hard to manage over the long term.
>
> Cheers,
> Nathan

Thanks a lot Nathan

It is definitely the clang version 11.0.x which is not updated in aosp
Android 11 production tags

I will use Nick's workaround which works since only lsl %rax, %rax is
currently happening

Many thanks, problem solved

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Mauro Rossi]

> > Alternatively, you could apply the hack that Nick inserted into Android
> > for this issue if upgrading your toolchain or turning off the integrated
> > assembler is not possible:
> >
> > https://android.googlesource.com/kernel/common/+/e58f084735b8abf744d61083b92172ee23d45aae
> >
> > I really do not mean to sound dismissive or rude, I apologize if it
> > comes off that way, but we have worked quite hard to avoid inserting
> > unnecessary workarounds, as they are ultimately technical debt that can
> > be hard to manage over the long term.
> >
> > Cheers,
> > Nathan
>
> Thanks a lot Nathan
>
> It is definitely the clang version 11.0.x which is not updated in aosp
> Android 11 production tags
>
> I will use Nick's workaround which works since only lsl %rax, %rax is
> currently happening
>
> Many thanks, problem solved

Hello,
I'm back again because I was assuming that Nick's workaround was working ok,
but I have found that ARCH=x86_64 i.e. 64bit built kernel is causing
an immediate hard reboot at initrd execution,
just after hitting [ENTER] at grub/efi menu.

ARCh=x86 i.e. 32bit kernel binary is not affected, but is Nick's
workaround targeting 32 bit kernel builds?
Should it be modified to work for 64bit kernel binary?

How do aosp android-mailine kernels avoid this instantaneous hard reboot issue?
Mauro

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Nathan Chancellor]

On Sat, May 07, 2022 at 02:19:00PM +0200, Mauro Rossi wrote:
> > > Alternatively, you could apply the hack that Nick inserted into Android
> > > for this issue if upgrading your toolchain or turning off the integrated
> > > assembler is not possible:
> > >
> > > https://android.googlesource.com/kernel/common/+/e58f084735b8abf744d61083b92172ee23d45aae
> > >
> > > I really do not mean to sound dismissive or rude, I apologize if it
> > > comes off that way, but we have worked quite hard to avoid inserting
> > > unnecessary workarounds, as they are ultimately technical debt that can
> > > be hard to manage over the long term.
> > >
> > > Cheers,
> > > Nathan
> >
> > Thanks a lot Nathan
> >
> > It is definitely the clang version 11.0.x which is not updated in aosp
> > Android 11 production tags
> >
> > I will use Nick's workaround which works since only lsl %rax, %rax is
> > currently happening
> >
> > Many thanks, problem solved
> 
> Hello,
> I'm back again because I was assuming that Nick's workaround was working ok,
> but I have found that ARCH=x86_64 i.e. 64bit built kernel is causing
> an immediate hard reboot at initrd execution,
> just after hitting [ENTER] at grub/efi menu.

It looks like there was a follow up fix for that workaround, maybe that
resolves this issue as well?

https://android.googlesource.com/kernel/common/+/cc7f7a84191f5defc2ea4633eeea4acb4486b549

> ARCh=x86 i.e. 32bit kernel binary is not affected, but is Nick's
> workaround targeting 32 bit kernel builds?

No, the file that the build error originates from (entry_64.S) is only
built on x86_64.

> How do aosp android-mailine kernels avoid this instantaneous hard reboot issue?

That hack was reverted once the toolchain was upgraded:

https://android.googlesource.com/kernel/common/+/ff0216d09fd31802537f2d1702ec2f3e9be73aa3
https://android.googlesource.com/kernel/common/+/3c2c8d8f7f2639e319212d10cb8df5bd13098dae

Cheers,
Nathan

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Mauro Rossi]

On Mon, May 9, 2022 at 2:07 AM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Sat, May 07, 2022 at 02:19:00PM +0200, Mauro Rossi wrote:
> > > > Alternatively, you could apply the hack that Nick inserted into Android
> > > > for this issue if upgrading your toolchain or turning off the integrated
> > > > assembler is not possible:
> > > >
> > > > https://android.googlesource.com/kernel/common/+/e58f084735b8abf744d61083b92172ee23d45aae
> > > >
> > > > I really do not mean to sound dismissive or rude, I apologize if it
> > > > comes off that way, but we have worked quite hard to avoid inserting
> > > > unnecessary workarounds, as they are ultimately technical debt that can
> > > > be hard to manage over the long term.
> > > >
> > > > Cheers,
> > > > Nathan
> > >
> > > Thanks a lot Nathan
> > >
> > > It is definitely the clang version 11.0.x which is not updated in aosp
> > > Android 11 production tags
> > >
> > > I will use Nick's workaround which works since only lsl %rax, %rax is
> > > currently happening
> > >
> > > Many thanks, problem solved
> >
> > Hello,
> > I'm back again because I was assuming that Nick's workaround was working ok,
> > but I have found that ARCH=x86_64 i.e. 64bit built kernel is causing
> > an immediate hard reboot at initrd execution,
> > just after hitting [ENTER] at grub/efi menu.
>
> It looks like there was a follow up fix for that workaround, maybe that
> resolves this issue as well?
>
> https://android.googlesource.com/kernel/common/+/cc7f7a84191f5defc2ea4633eeea4acb4486b549
>
> > ARCh=x86 i.e. 32bit kernel binary is not affected, but is Nick's
> > workaround targeting 32 bit kernel builds?
>
> No, the file that the build error originates from (entry_64.S) is only
> built on x86_64.
>
> > How do aosp android-mailine kernels avoid this instantaneous hard reboot issue?
>
> That hack was reverted once the toolchain was upgraded:
>
> https://android.googlesource.com/kernel/common/+/ff0216d09fd31802537f2d1702ec2f3e9be73aa3
> https://android.googlesource.com/kernel/common/+/3c2c8d8f7f2639e319212d10cb8df5bd13098dae
>
> Cheers,
> Nathan

Hello,

sorry i still have the issue when building with hack "ANDROID: x86:
entry: work around LLVM_IAS=1 bug in LSL"
I'd just like to understand why I get instant reboot in 64bit kernel
built with LLVM_IAS=1

I searched for the places where #include "calling.h" is used

utente@utente-3TB:~/r-x86_kernel/kernel$ grep -r \#include\ \"calling
arch/x86/entry/thunk_64.S:#include "calling.h"
arch/x86/entry/entry_32.S:#include "calling.h"
arch/x86/entry/entry_64_compat.S:#include "calling.h"
arch/x86/entry/entry_64.S:#include "calling.h"

Probably the hardcoding done in the hack is impacting some other asm
target built in 64 bit kernel and causes a wrong X86 instruction , is
there a way to refine the #if defto avoid that problem while still
building with LLVM_IAS=1 ?

Thanks for the info
Mauro

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Nick Desaulniers]

On Sun, May 29, 2022 at 10:52 AM Mauro Rossi <issor.oruam@gmail.com> wrote:
>
>
>
> On Mon, May 9, 2022 at 2:07 AM Nathan Chancellor <nathan@kernel.org> wrote:
>>
>> On Sat, May 07, 2022 at 02:19:00PM +0200, Mauro Rossi wrote:
>> > > > Alternatively, you could apply the hack that Nick inserted into Android
>> > > > for this issue if upgrading your toolchain or turning off the integrated
>> > > > assembler is not possible:
>> > > >
>> > > > https://android.googlesource.com/kernel/common/+/e58f084735b8abf744d61083b92172ee23d45aae
>> > > >
>> > > > I really do not mean to sound dismissive or rude, I apologize if it
>> > > > comes off that way, but we have worked quite hard to avoid inserting
>> > > > unnecessary workarounds, as they are ultimately technical debt that can
>> > > > be hard to manage over the long term.
>> > > >
>> > > > Cheers,
>> > > > Nathan
>> > >
>> > > Thanks a lot Nathan
>> > >
>> > > It is definitely the clang version 11.0.x which is not updated in aosp
>> > > Android 11 production tags
>> > >
>> > > I will use Nick's workaround which works since only lsl %rax, %rax is
>> > > currently happening
>> > >
>> > > Many thanks, problem solved
>> >
>> > Hello,
>> > I'm back again because I was assuming that Nick's workaround was working ok,
>> > but I have found that ARCH=x86_64 i.e. 64bit built kernel is causing
>> > an immediate hard reboot at initrd execution,
>> > just after hitting [ENTER] at grub/efi menu.
>>
>> It looks like there was a follow up fix for that workaround, maybe that
>> resolves this issue as well?
>>
>> https://android.googlesource.com/kernel/common/+/cc7f7a84191f5defc2ea4633eeea4acb4486b549
>>
>> > ARCh=x86 i.e. 32bit kernel binary is not affected, but is Nick's
>> > workaround targeting 32 bit kernel builds?
>>
>> No, the file that the build error originates from (entry_64.S) is only
>> built on x86_64.
>>
>> > How do aosp android-mailine kernels avoid this instantaneous hard reboot issue?
>>
>> That hack was reverted once the toolchain was upgraded:
>>
>> https://android.googlesource.com/kernel/common/+/ff0216d09fd31802537f2d1702ec2f3e9be73aa3
>> https://android.googlesource.com/kernel/common/+/3c2c8d8f7f2639e319212d10cb8df5bd13098dae
>>
>> Cheers,
>> Nathan
>
>
> Hello,
>
> sorry i still have the issue when building with hack "ANDROID: x86: entry: work around LLVM_IAS=1 bug in LSL"
> I'd just like to understand why I get instant reboot in 64bit kernel built with LLVM_IAS=1

As Nathan noted, I messed up the commit "ANDROID: x86: entry: work
around LLVM_IAS=1 bug in LSL". Please see:
https://android-review.googlesource.com/c/kernel/common/+/1521061
https://android-review.googlesource.com/c/kernel/common/+/1560152/

If you're using an older toolchain, you'll need just the first. If
you're using a newer toolchain, you'll need BOTH (or none, including
dropping "ANDROID: x86: entry: work around LLVM_IAS=1 bug in LSL").

>
> I searched for the places where #include "calling.h" is used
>
> utente@utente-3TB:~/r-x86_kernel/kernel$ grep -r \#include\ \"calling
> arch/x86/entry/thunk_64.S:#include "calling.h"
> arch/x86/entry/entry_32.S:#include "calling.h"
> arch/x86/entry/entry_64_compat.S:#include "calling.h"
> arch/x86/entry/entry_64.S:#include "calling.h"
>
> Probably the hardcoding done in the hack is impacting some other asm target built in 64 bit kernel and causes a wrong X86 instruction , is there a way to refine the #if to avoid that problem while still building with LLVM_IAS=1 ?
>
> Thanks for the info
> Mauro
>
>


-- 
Thanks,
~Nick Desaulniers

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Mauro Rossi]

On Wed, Jun 1, 2022 at 12:09 AM Nick Desaulniers
<ndesaulniers@google.com> wrote:
>
> On Sun, May 29, 2022 at 10:52 AM Mauro Rossi <issor.oruam@gmail.com> wrote:
> >
> >
> >
> > On Mon, May 9, 2022 at 2:07 AM Nathan Chancellor <nathan@kernel.org> wrote:
> >>
> >> On Sat, May 07, 2022 at 02:19:00PM +0200, Mauro Rossi wrote:
> >> > > > Alternatively, you could apply the hack that Nick inserted into Android
> >> > > > for this issue if upgrading your toolchain or turning off the integrated
> >> > > > assembler is not possible:
> >> > > >
> >> > > > https://android.googlesource.com/kernel/common/+/e58f084735b8abf744d61083b92172ee23d45aae
> >> > > >
> >> > > > I really do not mean to sound dismissive or rude, I apologize if it
> >> > > > comes off that way, but we have worked quite hard to avoid inserting
> >> > > > unnecessary workarounds, as they are ultimately technical debt that can
> >> > > > be hard to manage over the long term.
> >> > > >
> >> > > > Cheers,
> >> > > > Nathan
> >> > >
> >> > > Thanks a lot Nathan
> >> > >
> >> > > It is definitely the clang version 11.0.x which is not updated in aosp
> >> > > Android 11 production tags
> >> > >
> >> > > I will use Nick's workaround which works since only lsl %rax, %rax is
> >> > > currently happening
> >> > >
> >> > > Many thanks, problem solved
> >> >
> >> > Hello,
> >> > I'm back again because I was assuming that Nick's workaround was working ok,
> >> > but I have found that ARCH=x86_64 i.e. 64bit built kernel is causing
> >> > an immediate hard reboot at initrd execution,
> >> > just after hitting [ENTER] at grub/efi menu.
> >>
> >> It looks like there was a follow up fix for that workaround, maybe that
> >> resolves this issue as well?
> >>
> >> https://android.googlesource.com/kernel/common/+/cc7f7a84191f5defc2ea4633eeea4acb4486b549
> >>
> >> > ARCh=x86 i.e. 32bit kernel binary is not affected, but is Nick's
> >> > workaround targeting 32 bit kernel builds?
> >>
> >> No, the file that the build error originates from (entry_64.S) is only
> >> built on x86_64.
> >>
> >> > How do aosp android-mailine kernels avoid this instantaneous hard reboot issue?
> >>
> >> That hack was reverted once the toolchain was upgraded:
> >>
> >> https://android.googlesource.com/kernel/common/+/ff0216d09fd31802537f2d1702ec2f3e9be73aa3
> >> https://android.googlesource.com/kernel/common/+/3c2c8d8f7f2639e319212d10cb8df5bd13098dae
> >>
> >> Cheers,
> >> Nathan
> >
> >
> > Hello,
> >
> > sorry i still have the issue when building with hack "ANDROID: x86: entry: work around LLVM_IAS=1 bug in LSL"
> > I'd just like to understand why I get instant reboot in 64bit kernel built with LLVM_IAS=1
>
> As Nathan noted, I messed up the commit "ANDROID: x86: entry: work
> around LLVM_IAS=1 bug in LSL". Please see:
> https://android-review.googlesource.com/c/kernel/common/+/1521061
> https://android-review.googlesource.com/c/kernel/common/+/1560152/
>
> If you're using an older toolchain, you'll need just the first. If
> you're using a newer toolchain, you'll need BOTH (or none, including
> dropping "ANDROID: x86: entry: work around LLVM_IAS=1 bug in LSL").

Thanks Nick,

I had already applied the squashed commit composed of  "ANDROID: x86:
entry: work around LLVM_IAS=1 bug in LSL" (the one using .quad) and
"ANDROID: x86: entry: fix LSL open coding", so I have already:

.macro LOAD_CPU_AND_NODE_SEG_LIMIT reg:req
movq $__CPUNODE_SEG, \reg
+#ifdef __clang__
+.long 0xc0030f48
+#else
lsl \reg, \reg
+#endif
.endm


So in principle my kernel image should boot when built with LLVM_IAS=1
but to my surprise all my systems (Sony VAIO i7, Intel NUC DN2820FYKH
with Celeron D2830, Athlon 200GE) are affected by hard reboot when
executing the kernel image

I'm trying to understand how to build (and boot) with LLVM_IAS=1 and
using clang 11.0.2 shipped with AOSP Android 11

>
> >
> > I searched for the places where #include "calling.h" is used
> >
> > utente@utente-3TB:~/r-x86_kernel/kernel$ grep -r \#include\ \"calling
> > arch/x86/entry/thunk_64.S:#include "calling.h"
> > arch/x86/entry/entry_32.S:#include "calling.h"
> > arch/x86/entry/entry_64_compat.S:#include "calling.h"
> > arch/x86/entry/entry_64.S:#include "calling.h"
> >
> > Probably the hardcoding done in the hack is impacting some other asm target built in 64 bit kernel and causes a wrong X86 instruction , is there a way to refine the #if to avoid that problem while still building with LLVM_IAS=1 ?
> >
> > Thanks for the info
> > Mauro
> >
> >
>
>
> --
> Thanks,
> ~Nick Desaulniers

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Nick Desaulniers]

On Wed, Jun 1, 2022 at 5:57 AM Mauro Rossi <issor.oruam@gmail.com> wrote:
>
> On Wed, Jun 1, 2022 at 12:09 AM Nick Desaulniers
> <ndesaulniers@google.com> wrote:
> >
> > As Nathan noted, I messed up the commit "ANDROID: x86: entry: work
> > around LLVM_IAS=1 bug in LSL". Please see:
> > https://android-review.googlesource.com/c/kernel/common/+/1521061
> > https://android-review.googlesource.com/c/kernel/common/+/1560152/
> >
> > If you're using an older toolchain, you'll need just the first. If
> > you're using a newer toolchain, you'll need BOTH (or none, including
> > dropping "ANDROID: x86: entry: work around LLVM_IAS=1 bug in LSL").
>
> Thanks Nick,
>
> I had already applied the squashed commit composed of  "ANDROID: x86:
> entry: work around LLVM_IAS=1 bug in LSL" (the one using .quad) and
> "ANDROID: x86: entry: fix LSL open coding", so I have already:
>
> .macro LOAD_CPU_AND_NODE_SEG_LIMIT reg:req
> movq $__CPUNODE_SEG, \reg
> +#ifdef __clang__
> +.long 0xc0030f48

LGTM

> +#else
> lsl \reg, \reg
> +#endif
> .endm
>
>
> So in principle my kernel image should boot when built with LLVM_IAS=1
> but to my surprise all my systems (Sony VAIO i7, Intel NUC DN2820FYKH
> with Celeron D2830, Athlon 200GE) are affected by hard reboot when
> executing the kernel image

Might need more info.  Do they boot when LLVM_IAS=0 is explicitly set
with your command line invocation of make? i.e. `make LLVM=1
LLVM_IAS=0 ...`?  Can you launch these kernels in qemu?

>
> I'm trying to understand how to build (and boot) with LLVM_IAS=1 and
> using clang 11.0.2 shipped with AOSP Android 11

I think this combo should work; we are testing x86_64 with mainline
https://github.com/ClangBuiltLinux/continuous-integration2/blob/95b9a12cad31675118d61c26d0b541fa4e3c8f09/generator.yml#L1694

Could be something in your .config files though.
-- 
Thanks,
~Nick Desaulniers

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Mauro Rossi]

On Sat, Jun 4, 2022 at 12:13 AM Nick Desaulniers
<ndesaulniers@google.com> wrote:
>
> On Wed, Jun 1, 2022 at 5:57 AM Mauro Rossi <issor.oruam@gmail.com> wrote:
> >
> > On Wed, Jun 1, 2022 at 12:09 AM Nick Desaulniers
> > <ndesaulniers@google.com> wrote:
> > >
> > > As Nathan noted, I messed up the commit "ANDROID: x86: entry: work
> > > around LLVM_IAS=1 bug in LSL". Please see:
> > > https://android-review.googlesource.com/c/kernel/common/+/1521061
> > > https://android-review.googlesource.com/c/kernel/common/+/1560152/
> > >
> > > If you're using an older toolchain, you'll need just the first. If
> > > you're using a newer toolchain, you'll need BOTH (or none, including
> > > dropping "ANDROID: x86: entry: work around LLVM_IAS=1 bug in LSL").
> >
> > Thanks Nick,
> >
> > I had already applied the squashed commit composed of  "ANDROID: x86:
> > entry: work around LLVM_IAS=1 bug in LSL" (the one using .quad) and
> > "ANDROID: x86: entry: fix LSL open coding", so I have already:
> >
> > .macro LOAD_CPU_AND_NODE_SEG_LIMIT reg:req
> > movq $__CPUNODE_SEG, \reg
> > +#ifdef __clang__
> > +.long 0xc0030f48
>
> LGTM
>
> > +#else
> > lsl \reg, \reg
> > +#endif
> > .endm
> >
> >
> > So in principle my kernel image should boot when built with LLVM_IAS=1
> > but to my surprise all my systems (Sony VAIO i7, Intel NUC DN2820FYKH
> > with Celeron D2830, Athlon 200GE) are affected by hard reboot when
> > executing the kernel image
>
> Might need more info.  Do they boot when LLVM_IAS=0 is explicitly set
> with your command line invocation of make? i.e. `make LLVM=1
> LLVM_IAS=0 ...`?  Can you launch these kernels in qemu?

Yes, kernel 5.17 and 5.18 built with LLVM=1 LLVM_IAS=0 do not cause
instantaneus hard reboot and proceed in the boot stages

The complete list of make variables Ihave used is as follows:

LLVM=1 LLVM_IAS=0 \
        CC=$(abspath $(LLVM_PREBUILTS_PATH)/clang) \
        LD=$(abspath $(LLVM_PREBUILTS_PATH)/ld.lld) \
        AR=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-ar) \
        NM=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-nm) \
        OBJCOPY=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-objcopy) \
        OBJDUMP=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-objdump) \
        READELF=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-readelf) \
        OBJSIZE=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-size) \
        STRIP=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-strip) \
        HOSTCC=$(abspath $(LLVM_PREBUILTS_PATH)/clang) \
        HOSTCXX=$(abspath $(LLVM_PREBUILTS_PATH)/clang++) \
        HOSTLD=$(abspath $(LLVM_PREBUILTS_PATH)/ld.lld) \
        HOSTLDFLAGS=-fuse-ld=lld \
        HOSTAR=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-ar)

>
> >
> > I'm trying to understand how to build (and boot) with LLVM_IAS=1 and
> > using clang 11.0.2 shipped with AOSP Android 11
>
> I think this combo should work; we are testing x86_64 with mainline
> https://github.com/ClangBuiltLinux/continuous-integration2/blob/95b9a12cad31675118d61c26d0b541fa4e3c8f09/generator.yml#L1694
>
> Could be something in your .config files though.
> --
> Thanks,
> ~Nick Desaulniers

Re: arch/x86/entry/entry: RFC on recent kernels building error with llvm 11.0.2 internal assembler

[Nick Desaulniers]

On Mon, Jun 6, 2022 at 2:57 PM Mauro Rossi <issor.oruam@gmail.com> wrote:
>
> On Sat, Jun 4, 2022 at 12:13 AM Nick Desaulniers
> <ndesaulniers@google.com> wrote:
> >
> > On Wed, Jun 1, 2022 at 5:57 AM Mauro Rossi <issor.oruam@gmail.com> wrote:
> > >
> > > On Wed, Jun 1, 2022 at 12:09 AM Nick Desaulniers
> > > <ndesaulniers@google.com> wrote:
> > > >
> > > > As Nathan noted, I messed up the commit "ANDROID: x86: entry: work
> > > > around LLVM_IAS=1 bug in LSL". Please see:
> > > > https://android-review.googlesource.com/c/kernel/common/+/1521061
> > > > https://android-review.googlesource.com/c/kernel/common/+/1560152/
> > > >
> > > > If you're using an older toolchain, you'll need just the first. If
> > > > you're using a newer toolchain, you'll need BOTH (or none, including
> > > > dropping "ANDROID: x86: entry: work around LLVM_IAS=1 bug in LSL").
> > >
> > > Thanks Nick,
> > >
> > > I had already applied the squashed commit composed of  "ANDROID: x86:
> > > entry: work around LLVM_IAS=1 bug in LSL" (the one using .quad) and
> > > "ANDROID: x86: entry: fix LSL open coding", so I have already:
> > >
> > > .macro LOAD_CPU_AND_NODE_SEG_LIMIT reg:req
> > > movq $__CPUNODE_SEG, \reg
> > > +#ifdef __clang__
> > > +.long 0xc0030f48
> >
> > LGTM
> >
> > > +#else
> > > lsl \reg, \reg
> > > +#endif
> > > .endm
> > >
> > >
> > > So in principle my kernel image should boot when built with LLVM_IAS=1
> > > but to my surprise all my systems (Sony VAIO i7, Intel NUC DN2820FYKH
> > > with Celeron D2830, Athlon 200GE) are affected by hard reboot when
> > > executing the kernel image
> >
> > Might need more info.  Do they boot when LLVM_IAS=0 is explicitly set
> > with your command line invocation of make? i.e. `make LLVM=1
> > LLVM_IAS=0 ...`?  Can you launch these kernels in qemu?
>
> Yes, kernel 5.17 and 5.18 built with LLVM=1 LLVM_IAS=0 do not cause
> instantaneus hard reboot and proceed in the boot stages
>
> The complete list of make variables Ihave used is as follows:
>
> LLVM=1 LLVM_IAS=0 \
>         CC=$(abspath $(LLVM_PREBUILTS_PATH)/clang) \
>         LD=$(abspath $(LLVM_PREBUILTS_PATH)/ld.lld) \
>         AR=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-ar) \
>         NM=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-nm) \
>         OBJCOPY=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-objcopy) \
>         OBJDUMP=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-objdump) \
>         READELF=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-readelf) \
>         OBJSIZE=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-size) \
>         STRIP=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-strip) \
>         HOSTCC=$(abspath $(LLVM_PREBUILTS_PATH)/clang) \
>         HOSTCXX=$(abspath $(LLVM_PREBUILTS_PATH)/clang++) \
>         HOSTLD=$(abspath $(LLVM_PREBUILTS_PATH)/ld.lld) \
>         HOSTLDFLAGS=-fuse-ld=lld \
>         HOSTAR=$(abspath $(LLVM_PREBUILTS_PATH)/llvm-ar)

You could probably simplify the above to:

$ PATH=$PATH:$LLVM_PREBUILTS_PATH make LLVM=1 LLVM_IAS=0

>
> >
> > >
> > > I'm trying to understand how to build (and boot) with LLVM_IAS=1 and
> > > using clang 11.0.2 shipped with AOSP Android 11

Can you try a newer version of LLVM?  AOSP LLVM 11 is technically a
pre-release version somewhere between the LLVM 10 and 11 releases.  We
no longer support LLVM 10 with mainline, so it's hard to say if AOSP
LLVM 11.0.2 contains or does not contain problematic or necessary
commits.

Also, do you have a kernel tree I can fetch? I'd be happy to try to
boot the result in QEMU. Have you tried that?

> >
> > I think this combo should work; we are testing x86_64 with mainline
> > https://github.com/ClangBuiltLinux/continuous-integration2/blob/95b9a12cad31675118d61c26d0b541fa4e3c8f09/generator.yml#L1694
> >
> > Could be something in your .config files though.
> > --
> > Thanks,
> > ~Nick Desaulniers
>


-- 
Thanks,
~Nick Desaulniers