v5.17-rc3 compile failure on aarch64

v5.17-rc3 compile failure on aarch64

[Qu Wenruo]

Hi,

Recently I found v5.17-rc3 kernel failed to compile on my aarch64 setup,
it failed with the following error messages:

   AR      lib/zstd/built-in.a
   LD [M]  lib/zstd/zstd_compress.o
ld: __patchable_function_entries has both ordered
[`__patchable_function_entries' in lib/zstd/common/entropy_common.o] and
unordered [`__patchable_function_entries' in
lib/zstd/common/error_private.o] sections
ld: final link failed: bad value
make[2]: *** [scripts/Makefile.build:484: lib/zstd/zstd_compress.o] Error 1
make[1]: *** [scripts/Makefile.build:550: lib/zstd] Error 2
make: *** [Makefile:1831: lib] Error 2
make: *** Waiting for unfinished jobs....

But the same code base compiles fine inside my x86_64 VM.

The aarch64 environment is an VM running on RockPro64 or RPI CM4.

With a little older gcc (v10.2.0) and bintuils (2.35).

While for my x86_64 VM it has newer gcc (11.1.0) and binutiles (2.36.1)

I'm super happy to upgrade my tool chain for my aarch64 VM, but I'm a
little concerned if there is something worthy fixed.

Thanks,
Qu

Re: v5.17-rc3 compile failure on aarch64

[Mark Rutland]

On Thu, Feb 17, 2022 at 04:37:49PM +0800, Qu Wenruo wrote:
> Hi,

Hi,

[adding LAKML so other arm/arm64 folk can chime in]

> Recently I found v5.17-rc3 kernel failed to compile on my aarch64 setup,
> it failed with the following error messages:
> 
>   AR      lib/zstd/built-in.a
>   LD [M]  lib/zstd/zstd_compress.o
> ld: __patchable_function_entries has both ordered
> [`__patchable_function_entries' in lib/zstd/common/entropy_common.o] and
> unordered [`__patchable_function_entries' in
> lib/zstd/common/error_private.o] sections
> ld: final link failed: bad value
> make[2]: *** [scripts/Makefile.build:484: lib/zstd/zstd_compress.o] Error 1
> make[1]: *** [scripts/Makefile.build:550: lib/zstd] Error 2
> make: *** [Makefile:1831: lib] Error 2
> make: *** Waiting for unfinished jobs....

Uh-oh; that doesn't look good.

In case this is configuration-dependent, could you share your .config?

> But the same code base compiles fine inside my x86_64 VM.
> 
> The aarch64 environment is an VM running on RockPro64 or RPI CM4.
> 
> With a little older gcc (v10.2.0) and bintuils (2.35).

Is this a custom-built toolchain, or one shipped by a distribution?

If possible, can you see if you can reproduce this with one of the
kernel.org cross toolchains:

  https://mirrors.edge.kernel.org/pub/tools/crosstool/

... because if you can, that will be much easier for someone else to
reproduce and analyse.

The closest builds are 9.3.0 and 10.3.0, for both arm64 and x86_64.

FWIW, I had a go building v5.17-rc3 defconfig with ftrace and zstd bits
eanbled with both GCC 9.3.0 and GCC 10.3.0, and I could not reproduce
the problem.

Thanks,
Mark.

> While for my x86_64 VM it has newer gcc (11.1.0) and binutiles (2.36.1)
> 
> I'm super happy to upgrade my tool chain for my aarch64 VM, but I'm a
> little concerned if there is something worthy fixed.
> 
> Thanks,
> Qu

Re: v5.17-rc3 compile failure on aarch64

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 2700 bytes --]



On 2022/2/17 19:50, Mark Rutland wrote:
> On Thu, Feb 17, 2022 at 04:37:49PM +0800, Qu Wenruo wrote:
>> Hi,
>
> Hi,
>
> [adding LAKML so other arm/arm64 folk can chime in]
>
>> Recently I found v5.17-rc3 kernel failed to compile on my aarch64 setup,
>> it failed with the following error messages:
>>
>>    AR      lib/zstd/built-in.a
>>    LD [M]  lib/zstd/zstd_compress.o
>> ld: __patchable_function_entries has both ordered
>> [`__patchable_function_entries' in lib/zstd/common/entropy_common.o] and
>> unordered [`__patchable_function_entries' in
>> lib/zstd/common/error_private.o] sections
>> ld: final link failed: bad value
>> make[2]: *** [scripts/Makefile.build:484: lib/zstd/zstd_compress.o] Error 1
>> make[1]: *** [scripts/Makefile.build:550: lib/zstd] Error 2
>> make: *** [Makefile:1831: lib] Error 2
>> make: *** Waiting for unfinished jobs....
>
> Uh-oh; that doesn't look good.
>
> In case this is configuration-dependent, could you share your .config?

Sure, attached.

>
>> But the same code base compiles fine inside my x86_64 VM.
>>
>> The aarch64 environment is an VM running on RockPro64 or RPI CM4.
>>
>> With a little older gcc (v10.2.0) and bintuils (2.35).
>
> Is this a custom-built toolchain, or one shipped by a distribution?

Shipped by distro (ArchlinuxARM), native, no cross chain.

And it's mostly upstream, or very lightly backported.

>
> If possible, can you see if you can reproduce this with one of the
> kernel.org cross toolchains:
>
>    https://mirrors.edge.kernel.org/pub/tools/crosstool/
>
> ... because if you can, that will be much easier for someone else to
> reproduce and analyse.
>
> The closest builds are 9.3.0 and 10.3.0, for both arm64 and x86_64.

Sorry, not cross tool chain but native toolchain.

>
> FWIW, I had a go building v5.17-rc3 defconfig with ftrace and zstd bits
> eanbled with both GCC 9.3.0 and GCC 10.3.0, and I could not reproduce
> the problem.

After the incident, I tried the following things, and got it solved:

- Upgrade to gcc 11.1.0 and binutils 2.36.1
   The same kernel base, no "make clean"

   And got the same error.

- "make clean" for the kernel and recompile
   Now the failure is gone.

   Thus I guess it's mostly some old .o is being re-used...

Sorry for the inconvenience, I should just re-compile (even it's super
time-consuming) before reporting...

Thanks,
Qu

>
> Thanks,
> Mark.
>
>> While for my x86_64 VM it has newer gcc (11.1.0) and binutiles (2.36.1)
>>
>> I'm super happy to upgrade my tool chain for my aarch64 VM, but I'm a
>> little concerned if there is something worthy fixed.
>>
>> Thanks,
>> Qu

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 38334 bytes --]

Re: v5.17-rc3 compile failure on aarch64

[Mark Rutland]

On Thu, Feb 17, 2022 at 07:59:10PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/2/17 19:50, Mark Rutland wrote:
> > On Thu, Feb 17, 2022 at 04:37:49PM +0800, Qu Wenruo wrote:
> > > Hi,
> > 
> > Hi,
> > 
> > [adding LAKML so other arm/arm64 folk can chime in]
> > 
> > > Recently I found v5.17-rc3 kernel failed to compile on my aarch64 setup,
> > > it failed with the following error messages:
> > > 
> > >    AR      lib/zstd/built-in.a
> > >    LD [M]  lib/zstd/zstd_compress.o
> > > ld: __patchable_function_entries has both ordered
> > > [`__patchable_function_entries' in lib/zstd/common/entropy_common.o] and
> > > unordered [`__patchable_function_entries' in
> > > lib/zstd/common/error_private.o] sections
> > > ld: final link failed: bad value
> > > make[2]: *** [scripts/Makefile.build:484: lib/zstd/zstd_compress.o] Error 1
> > > make[1]: *** [scripts/Makefile.build:550: lib/zstd] Error 2
> > > make: *** [Makefile:1831: lib] Error 2
> > > make: *** Waiting for unfinished jobs....
> > 
> > Uh-oh; that doesn't look good.
> > 
> > In case this is configuration-dependent, could you share your .config?
> 
> Sure, attached.

Thanks!

> > > But the same code base compiles fine inside my x86_64 VM.
> > > 
> > > The aarch64 environment is an VM running on RockPro64 or RPI CM4.
> > > 
> > > With a little older gcc (v10.2.0) and bintuils (2.35).
> > 
> > Is this a custom-built toolchain, or one shipped by a distribution?
> 
> Shipped by distro (ArchlinuxARM), native, no cross chain.
> 
> And it's mostly upstream, or very lightly backported.
> 
> > If possible, can you see if you can reproduce this with one of the
> > kernel.org cross toolchains:
> > 
> >    https://mirrors.edge.kernel.org/pub/tools/crosstool/
> > 
> > ... because if you can, that will be much easier for someone else to
> > reproduce and analyse.
> > 
> > The closest builds are 9.3.0 and 10.3.0, for both arm64 and x86_64.
> 
> Sorry, not cross tool chain but native toolchain.

I understood that; the page has builds for arm64 hosts. :)

> > FWIW, I had a go building v5.17-rc3 defconfig with ftrace and zstd bits
> > eanbled with both GCC 9.3.0 and GCC 10.3.0, and I could not reproduce
> > the problem.
> 
> After the incident, I tried the following things, and got it solved:
> 
> - Upgrade to gcc 11.1.0 and binutils 2.36.1
>   The same kernel base, no "make clean"
> 
>   And got the same error.
> 
> - "make clean" for the kernel and recompile
>   Now the failure is gone.
> 
>   Thus I guess it's mostly some old .o is being re-used...
> 
> Sorry for the inconvenience, I should just re-compile (even it's super
> time-consuming) before reporting...

No worries; thanks for confirming!

Thanks,
Mark.

Re: v5.17-rc3 compile failure on aarch64

[Nick Terrell]

> On Feb 17, 2022, at 12:37 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> Hi,
> 
> Recently I found v5.17-rc3 kernel failed to compile on my aarch64 setup,
> it failed with the following error messages:
> 
>  AR      lib/zstd/built-in.a
>  LD [M]  lib/zstd/zstd_compress.o
> ld: __patchable_function_entries has both ordered
> [`__patchable_function_entries' in lib/zstd/common/entropy_common.o] and
> unordered [`__patchable_function_entries' in
> lib/zstd/common/error_private.o] sections
> ld: final link failed: bad value
> make[2]: *** [scripts/Makefile.build:484: lib/zstd/zstd_compress.o] Error 1
> make[1]: *** [scripts/Makefile.build:550: lib/zstd] Error 2
> make: *** [Makefile:1831: lib] Error 2
> make: *** Waiting for unfinished jobs....

I'm not sure what's going on, but maybe I can provide a hint to people
more knowledgable about the situation.

These objects are included in both the `zstd_compress` and
`zstd_decompress` modules [0]. That may be relevant. Back in 2018
we used to avoid having duplicate objects in different modules,
but then kbuild added support for it, and it was cleaned up in commit
dc35da16 [1].

It also may be irrelevant. Zstd was updated in the v5.16 kernel, but hasn't
changed in the v5.17 kernel.

Best,
Nick Terrell

[0] https://github.com/torvalds/linux/blob/master/lib/zstd/Makefile
[1] https://github.com/torvalds/linux/commit/dc35da16a2e23db04822f0129cd5b28b7b0e72b4

> But the same code base compiles fine inside my x86_64 VM.
> 
> The aarch64 environment is an VM running on RockPro64 or RPI CM4.
> 
> With a little older gcc (v10.2.0) and bintuils (2.35).
> 
> While for my x86_64 VM it has newer gcc (11.1.0) and binutiles (2.36.1)
> 
> I'm super happy to upgrade my tool chain for my aarch64 VM, but I'm a
> little concerned if there is something worthy fixed.
> 
> Thanks,
> Qu