[PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Josh Poimboeuf]

On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > .tmp_vmlinux.btf
> > btf_encoder__encode: btf__dedup failed!
> > Failed to encode BTF
> >
> > Thanks,
> >
> 
> I encountered the same problem when building a new kernel and I found some
> reasons for the error.
> 
> In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> .notes section. In addition, due to historical problems, the alignment of
> records in the .notes section is not unified, which leads to the inability of
> gelf_getnote() to read the records after the wrong one.

Alexandre, Tianyi, are you still seeing this issue with the latest
dwarves?  If so can you confirm the below patch fixes it?

Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
does anybody know if there a specific dwarves and/or libbpf change for
this?

[1] https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2346#note_1348057786

---8<---

From: Josh Poimboeuf <jpoimboe@kernel.org>
Subject: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

When tooling reads ELF notes, it assumes each note entry is aligned to
the value listed in the .note section header's sh_addralign field.

The kernel-created ELF notes in the .note.Linux and .note.Xen sections
are aligned to 4 bytes.  This causes the toolchain to set those
sections' sh_addralign values to 4.

On the other hand, the GCC-created .note.gnu.property section has an
sh_addralign value of 8 for some reason, despite being based on struct
Elf32_Nhdr which only needs 4-byte alignment.

When the mismatched input sections get linked together into the vmlinux
.notes output section, the higher alignment "wins", resulting in an
sh_addralign of 8, which confuses tooling.  For example:

  $ readelf -n .tmp_vmlinux.btf
  ...
  readelf: .tmp_vmlinux.btf: Warning: note with invalid namesz and/or descsz found at offset 0x170
  readelf: .tmp_vmlinux.btf: Warning:  type: 0x4, namesize: 0x006e6558, descsize: 0x00008801, alignment: 8

In this case readelf thinks there's alignment padding where there is
none, so it starts reading an ELF note in the middle.

With newer toolchains (e.g., latest Fedora Rawhide), a similar mismatch
triggers a build failure when combined with CONFIG_X86_KERNEL_IBT:

  btf_encoder__encode: btf__dedup failed!
  Failed to encode BTF
  libbpf: failed to find '.BTF' ELF section in vmlinux
  FAILED: load BTF from vmlinux: No data available
  make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 255

Fix it by forcing the .notes section input and output alignments to 4 to
match the kernel's note entry alignments.

Note this doesn't break the 8-byte-aligned .note.gnu.property entries
because their internal data representations fill the entire 8-byte
alignment boundary, so there's no padding between entries to be
misinterpreted.  And there's only a single entry in that section anyway.

Reported-by: Daniel Xu <dxu@dxuuu.xyz>
Debugged-by: Tianyi Liu <i.pear@outlook.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 include/asm-generic/vmlinux.lds.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index d1f57e4868ed..1c7c87c9ae71 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -894,7 +894,7 @@
  */
 #define NOTES								\
 	/DISCARD/ : { *(.note.GNU-stack) }				\
-	.notes : AT(ADDR(.notes) - LOAD_OFFSET) {			\
+	.notes ALIGN(4) : AT(ADDR(.notes) - LOAD_OFFSET) SUBALIGN(4) {	\
 		BOUNDED_SECTION_BY(.note.*, _notes)			\
 	} NOTES_HEADERS							\
 	NOTES_HEADERS_RESTORE
-- 
2.39.2

Re: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Tianyi Liu]

On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > .tmp_vmlinux.btf
> > > btf_encoder__encode: btf__dedup failed!
> > > Failed to encode BTF
> > >
> > > Thanks,
> > >
> >
> > I encountered the same problem when building a new kernel and I found some
> > reasons for the error.
> >
> > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > .notes section. In addition, due to historical problems, the alignment of
> > records in the .notes section is not unified, which leads to the inability of
> > gelf_getnote() to read the records after the wrong one.
> 
> Alexandre, Tianyi, are you still seeing this issue with the latest
> dwarves?  If so can you confirm the below patch fixes it?
> 

Josh, first of all, thank you very much for your help. However, this patch
doesn't work in my environment. I am using gcc 12.2.1 20230201.
After compiling, when I use readelf -S to view ELF sections,
the align of .notes section is still 8:

$ readelf -S .tmp_vmlinux.btf
[20] .notes            NOTE             ffffffff8250b570  0170b570
     0000000000000084  0000000000000000   A       0     0     8

> Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
> does anybody know if there a specific dwarves and/or libbpf change for
> this?
> 

It has been fixed in dwarves[1], but it may just be a coincidence.

> [1] https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2346#note_1348057786
> 
> ---8<---
> 
> From: Josh Poimboeuf <jpoimboe@kernel.org>
> Subject: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes
> 
> When tooling reads ELF notes, it assumes each note entry is aligned to
> the value listed in the .note section header's sh_addralign field.
> 
> The kernel-created ELF notes in the .note.Linux and .note.Xen sections
> are aligned to 4 bytes.  This causes the toolchain to set those
> sections' sh_addralign values to 4.
> 
> On the other hand, the GCC-created .note.gnu.property section has an
> sh_addralign value of 8 for some reason, despite being based on struct
> Elf32_Nhdr which only needs 4-byte alignment.
> 
> When the mismatched input sections get linked together into the vmlinux
> .notes output section, the higher alignment "wins", resulting in an
> sh_addralign of 8, which confuses tooling.  For example:
> 
>   $ readelf -n .tmp_vmlinux.btf
>   ...
>   readelf: .tmp_vmlinux.btf: Warning: note with invalid namesz and/or descsz found at offset 0x170
>   readelf: .tmp_vmlinux.btf: Warning:  type: 0x4, namesize: 0x006e6558, descsize: 0x00008801, alignment: 8
> 
> In this case readelf thinks there's alignment padding where there is
> none, so it starts reading an ELF note in the middle.
> 
> With newer toolchains (e.g., latest Fedora Rawhide), a similar mismatch
> triggers a build failure when combined with CONFIG_X86_KERNEL_IBT:
> 
>   btf_encoder__encode: btf__dedup failed!
>   Failed to encode BTF
>   libbpf: failed to find '.BTF' ELF section in vmlinux
>   FAILED: load BTF from vmlinux: No data available
>   make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 255
> 
> Fix it by forcing the .notes section input and output alignments to 4 to
> match the kernel's note entry alignments.
> 
> Note this doesn't break the 8-byte-aligned .note.gnu.property entries
> because their internal data representations fill the entire 8-byte
> alignment boundary, so there's no padding between entries to be
> misinterpreted.  And there's only a single entry in that section anyway.
> 
> Reported-by: Daniel Xu <dxu@dxuuu.xyz>
> Debugged-by: Tianyi Liu <i.pear@outlook.com>
> Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
> ---
>  include/asm-generic/vmlinux.lds.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index d1f57e4868ed..1c7c87c9ae71 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -894,7 +894,7 @@
>   */
>  #define NOTES                                                           \
>          /DISCARD/ : { *(.note.GNU-stack) }                              \
> -       .notes : AT(ADDR(.notes) - LOAD_OFFSET) {                       \
> +       .notes ALIGN(4) : AT(ADDR(.notes) - LOAD_OFFSET) SUBALIGN(4) {  \
>                  BOUNDED_SECTION_BY(.note.*, _notes)                     \
>          } NOTES_HEADERS                                                 \
>          NOTES_HEADERS_RESTORE
> --
> 2.39.2

[1] https://github.com/acmel/dwarves/commit/a9498899109d3be14f17abbc322a8f55a1067bee

Re: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Josh Poimboeuf]

On Wed, Apr 12, 2023 at 03:10:14PM +0800, Tianyi Liu wrote:
> On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> > On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > > .tmp_vmlinux.btf
> > > > btf_encoder__encode: btf__dedup failed!
> > > > Failed to encode BTF
> > > >
> > > > Thanks,
> > > >
> > >
> > > I encountered the same problem when building a new kernel and I found some
> > > reasons for the error.
> > >
> > > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > > .notes section. In addition, due to historical problems, the alignment of
> > > records in the .notes section is not unified, which leads to the inability of
> > > gelf_getnote() to read the records after the wrong one.
> > 
> > Alexandre, Tianyi, are you still seeing this issue with the latest
> > dwarves?  If so can you confirm the below patch fixes it?
> > 
> 
> Josh, first of all, thank you very much for your help. However, this patch
> doesn't work in my environment. I am using gcc 12.2.1 20230201.
> After compiling, when I use readelf -S to view ELF sections,
> the align of .notes section is still 8:
> 
> $ readelf -S .tmp_vmlinux.btf
> [20] .notes            NOTE             ffffffff8250b570  0170b570
>      0000000000000084  0000000000000000   A       0     0     8

Hm, weird.

> > Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
> > does anybody know if there a specific dwarves and/or libbpf change for
> > this?
> > 
> 
> It has been fixed in dwarves[1], but it may just be a coincidence.

So just to confirm, the btf__dedup error is gone for you with the latest
dwarves?

-- 
Josh

Re: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Joan Bruguera Micó]

I have done some tests with Arch Linux which is also affected by this issue:

Base scenario: Running Arch Linux updated as of 2023-04-13.
               Building the linux-mainline 6.3rc6-1 from AUR
               (https://aur.archlinux.org/packages/linux-mainline)
               with CONFIG_X86_KERNEL_IBT changed to 'y'
Result:        Build fails with the "btf__dedup failed!" error

Test 1: Update dwarves (=pahole package on Arch Linux) from the current version
        (1:1.24+r29+g02d67c5-1) to the staging version (1:1.25-1).
Result: The build works correctly.

        However, the notes section is still not parsed correctly, as confirmed
        by `readelf -n` or adding a printf near the code on `cus__merging_cu`
        (on `dwarf_loader.c`) pointed to by Tianyi.

        A bisect shows the commit that fixes the build is
        a9498899109d3be14f17abbc322a8f55a1067bee
        "dwarf_loader: Fix for BTF id drift caused by adding unspecified types"
        I don't know why though.

Test 2: Applying Josh's patch to force-align the ELF notes section to 4 bytes
        (clarification: using the base pahole version again here)
Result: The build works correctly.
        I can read the notes correctly using `readelf -n` as well,
        and they are aligned to 4 bytes instead of 8.

Test 3: Similar to Josh's patch, but instead of force-aligning the sections,
        I added `.note.gnu.property` to the `/DISCARD/` list in the line above.
Result: The build works correctly.
        I can read the notes correctly using `readelf -n` as well
        (of course, the GNU notes which made the alignment be 8 bytes are gone;
         I don't know if this has any negative effect).

So, in summary, either upgrading dwarves/pahole from 1.24 to (not yet fully
released?) 1.25, or applying Josh's patch fixes the issue for me.
Despite the new dwarves/pahole version fixing the build, Josh's patch or
something else to fix the notes alignment is needed.

Re: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Tianyi Liu]

On Thu, Apr 13, 2023 at 02:21:49 +0000, Joan Bruguera Micó wrote:
> I have done some tests with Arch Linux which is also affected by this issue:
> 
> Base scenario: Running Arch Linux updated as of 2023-04-13.
>                Building the linux-mainline 6.3rc6-1 from AUR
>                (https://aur.archlinux.org/packages/linux-mainline)
>                with CONFIG_X86_KERNEL_IBT changed to 'y'
> Result:        Build fails with the "btf__dedup failed!" error

I use MANJARO with kernel version 6.2.10-1. We should be using the same kernel.

> Test 1: Update dwarves (=pahole package on Arch Linux) from the current version
>         (1:1.24+r29+g02d67c5-1) to the staging version (1:1.25-1).
> Result: The build works correctly.
> 
>         However, the notes section is still not parsed correctly, as confirmed
>         by `readelf -n` or adding a printf near the code on `cus__merging_cu`
>         (on `dwarf_loader.c`) pointed to by Tianyi.
> 
>         A bisect shows the commit that fixes the build is
>         a9498899109d3be14f17abbc322a8f55a1067bee
>         "dwarf_loader: Fix for BTF id drift caused by adding unspecified types"
>         I don't know why though.

Pahole reads .notes to look for LINUX_ELFNOTE_BUILD_LTO. When LTO is
enabled, pahole needs to call cus__merge_and_process_cu to merge compile
units, at which point there should only be one unspecified type (used to
represent some compilation information) in the global context.

However, when the kernel is compiled without LTO, if pahole calls
cus__merge_and_process_cu due to alignment issues with notes, multiple
unspecified types may appear after merging the cus, and older versions of
pahole only support up to one. This is why pahole 1.24 crashes, while
newer versions support multiple. However, the latest version of pahole
still does not solve the problem of incorrect LTO recognition, so
compiling the kernel may be slower than normal.

If the only objective is to fix the bug of incorrect LTO recognition,
perhaps this naive patch[1] could be sufficient.

[1]: https://lore.kernel.org/bpf/SY4P282MB1084A0E31D4228DF89FC42639DA29@SY4P282MB1084.AUSP282.PROD.OUTLOOK.COM/

> Test 2: Applying Josh's patch to force-align the ELF notes section to 4 bytes
>         (clarification: using the base pahole version again here)
> Result: The build works correctly.
>         I can read the notes correctly using `readelf -n` as well,
>         and they are aligned to 4 bytes instead of 8.

I'm still currently unable to reproduce this fix, but it may just be an
issue with my environment.

> Test 3: Similar to Josh's patch, but instead of force-aligning the sections,
>         I added `.note.gnu.property` to the `/DISCARD/` list in the line above.
> Result: The build works correctly.
>         I can read the notes correctly using `readelf -n` as well
>         (of course, the GNU notes which made the alignment be 8 bytes are gone;
>          I don't know if this has any negative effect).

I think discarding .note.gnu.property will not have side effects. For some
reason, this note does not exist in my final vmlinux. I also did not find
any usage of this property in the later steps of compiling the kernel.

> So, in summary, either upgrading dwarves/pahole from 1.24 to (not yet fully
> released?) 1.25, or applying Josh's patch fixes the issue for me.
> Despite the new dwarves/pahole version fixing the build, Josh's patch or
> something else to fix the notes alignment is needed.

[PATCH] vmlinux.lds.h: Discard .note.gnu.property section

[Josh Poimboeuf]

On Thu, Apr 13, 2023 at 05:23:08PM +0800, Tianyi Liu wrote:
> > Test 1: Update dwarves (=pahole package on Arch Linux) from the current version
> >         (1:1.24+r29+g02d67c5-1) to the staging version (1:1.25-1).
> > Result: The build works correctly.
> > 
> >         However, the notes section is still not parsed correctly, as confirmed
> >         by `readelf -n` or adding a printf near the code on `cus__merging_cu`
> >         (on `dwarf_loader.c`) pointed to by Tianyi.
> > 
> >         A bisect shows the commit that fixes the build is
> >         a9498899109d3be14f17abbc322a8f55a1067bee
> >         "dwarf_loader: Fix for BTF id drift caused by adding unspecified types"
> >         I don't know why though.
> 
> Pahole reads .notes to look for LINUX_ELFNOTE_BUILD_LTO. When LTO is
> enabled, pahole needs to call cus__merge_and_process_cu to merge compile
> units, at which point there should only be one unspecified type (used to
> represent some compilation information) in the global context.
> 
> However, when the kernel is compiled without LTO, if pahole calls
> cus__merge_and_process_cu due to alignment issues with notes, multiple
> unspecified types may appear after merging the cus, and older versions of
> pahole only support up to one. This is why pahole 1.24 crashes, while
> newer versions support multiple. However, the latest version of pahole
> still does not solve the problem of incorrect LTO recognition, so
> compiling the kernel may be slower than normal.

Thanks for the explanation.  So pahole is still mis-reading the LTO
note, it just doesn't crash now.

> If the only objective is to fix the bug of incorrect LTO recognition,
> perhaps this naive patch[1] could be sufficient.
> 
> [1]: https://lore.kernel.org/bpf/SY4P282MB1084A0E31D4228DF89FC42639DA29@SY4P282MB1084.AUSP282.PROD.OUTLOOK.COM/

I think we still need to fix the underlying issue (corrupt .notes
section).

> > Test 2: Applying Josh's patch to force-align the ELF notes section to 4 bytes
> >         (clarification: using the base pahole version again here)
> > Result: The build works correctly.
> >         I can read the notes correctly using `readelf -n` as well,
> >         and they are aligned to 4 bytes instead of 8.
> 
> I'm still currently unable to reproduce this fix, but it may just be an
> issue with my environment.
> 
> > Test 3: Similar to Josh's patch, but instead of force-aligning the sections,
> >         I added `.note.gnu.property` to the `/DISCARD/` list in the line above.
> > Result: The build works correctly.
> >         I can read the notes correctly using `readelf -n` as well
> >         (of course, the GNU notes which made the alignment be 8 bytes are gone;
> >          I don't know if this has any negative effect).
> 
> I think discarding .note.gnu.property will not have side effects. For some
> reason, this note does not exist in my final vmlinux. I also did not find
> any usage of this property in the later steps of compiling the kernel.

It looks like CONFIG_DEBUG_INFO_BTF is already (inadvertently) stripping
it from vmlinux due to how GNU properties are merged by the linker (see
"How GNU properties are merged" in the ld man page).

The btf data is extracted from vmlinux.o with "objcopy
--only-section=.BTF" into .btf.vmlinux.bin.o.  That file doesn't have
.note.gnu.property, so when it gets modified and linked back into the
main object, the linker strips it.

GNU properties are important for user space but they don't seem to have
a purpose for vmlinux.  So yeah, let's just discard .note.gnu.property.

---8<---

From: Josh Poimboeuf <jpoimboe@kernel.org>
Subject: [PATCH] vmlinux.lds.h: Discard .note.gnu.property section

When tooling reads ELF notes, it assumes each note entry is aligned to
the value listed in the .note section header's sh_addralign field.

The kernel-created ELF notes in the .note.Linux and .note.Xen sections
are aligned to 4 bytes.  This causes the toolchain to set those
sections' sh_addralign values to 4.

On the other hand, the GCC-created .note.gnu.property section has an
sh_addralign value of 8 for some reason, despite being based on struct
Elf32_Nhdr which only needs 4-byte alignment.

When the mismatched input sections get linked together into the vmlinux
.notes output section, the higher alignment "wins", resulting in an
sh_addralign of 8, which confuses tooling.  For example:

  $ readelf -n .tmp_vmlinux.btf
  ...
  readelf: .tmp_vmlinux.btf: Warning: note with invalid namesz and/or descsz found at offset 0x170
  readelf: .tmp_vmlinux.btf: Warning:  type: 0x4, namesize: 0x006e6558, descsize: 0x00008801, alignment: 8

In this case readelf thinks there's alignment padding where there is
none, so it starts reading an ELF note in the middle.

With newer toolchains (e.g., latest Fedora Rawhide), a similar mismatch
triggers a build failure when combined with CONFIG_X86_KERNEL_IBT:

  btf_encoder__encode: btf__dedup failed!
  Failed to encode BTF
  libbpf: failed to find '.BTF' ELF section in vmlinux
  FAILED: load BTF from vmlinux: No data available
  make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 255

This latter error was caused by pahole crashing when it encountered the
corrupt .notes section.  This crash has been fixed in dwarves version
1.25.  As Tianyi Liu describes:

  "Pahole reads .notes to look for LINUX_ELFNOTE_BUILD_LTO. When LTO is
   enabled, pahole needs to call cus__merge_and_process_cu to merge
   compile units, at which point there should only be one unspecified
   type (used to represent some compilation information) in the global
   context.

   However, when the kernel is compiled without LTO, if pahole calls
   cus__merge_and_process_cu due to alignment issues with notes,
   multiple unspecified types may appear after merging the cus, and
   older versions of pahole only support up to one. This is why pahole
   1.24 crashes, while newer versions support multiple. However, the
   latest version of pahole still does not solve the problem of
   incorrect LTO recognition, so compiling the kernel may be slower
   than normal."

Even with the newer pahole, the note section misaligment issue still
exists and pahole is misinterpreting the LTO note.  Fix it by discarding
the .note.gnu.property section.  While GNU properties are important for
user space (and VDSO), they don't seem to have any use for vmlinux.

(In fact, they're already getting (inadvertently) stripped from vmlinux
when CONFIG_DEBUG_INFO_BTF is enabled.  The BTF data is extracted from
vmlinux.o with "objcopy --only-section=.BTF" into .btf.vmlinux.bin.o.
That file doesn't have .note.gnu.property, so when it gets modified and
linked back into the main object, the linker automatically strips it
(see "How GNU properties are merged" in the ld man page).)

Reported-by: Daniel Xu <dxu@dxuuu.xyz>
Link: https://lkml.kernel.org/lkml/57830c30-cd77-40cf-9cd1-3bb608aa602e@app.fastmail.com
Debugged-by: Tianyi Liu <i.pear@outlook.com>
Suggested-by: Joan Bruguera Micó <joanbrugueram@gmail.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 include/asm-generic/vmlinux.lds.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index d1f57e4868ed..1770b7d87a80 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -894,6 +894,7 @@
  */
 #define NOTES								\
 	/DISCARD/ : { *(.note.GNU-stack) }				\
+	/DISCARD/ : { *(.note.gnu.property) }				\
 	.notes : AT(ADDR(.notes) - LOAD_OFFSET) {			\
 		BOUNDED_SECTION_BY(.note.*, _notes)			\
 	} NOTES_HEADERS							\
-- 
2.39.2

[PATCH] vmlinux.lds.h: Discard .note.gnu.property section

[Joan Bruguera Micó]

Two small nitpicks:

> Link: https://lkml.kernel.org/lkml/57830c30-cd77-40cf-9cd1-3bb608aa602e@app.fastmail.com

This link is "semi-broken", it should go to /bpf/ instead of /lkml/.

>  	/DISCARD/ : { *(.note.GNU-stack) }				\
> +	/DISCARD/ : { *(.note.gnu.property) }				\

Both discards can go in the same DISCARD block.
(just style; it's how it's most often done in other linker scripts)

Re: [PATCH] vmlinux.lds.h: Discard .note.gnu.property section

[Josh Poimboeuf]

On Sun, Apr 16, 2023 at 07:02:19PM +0000, Joan Bruguera Micó wrote:
> Two small nitpicks:
> 
> > Link: https://lkml.kernel.org/lkml/57830c30-cd77-40cf-9cd1-3bb608aa602e@app.fastmail.com
> 
> This link is "semi-broken", it should go to /bpf/ instead of /lkml/.
> 
> >  	/DISCARD/ : { *(.note.GNU-stack) }				\
> > +	/DISCARD/ : { *(.note.gnu.property) }				\
> 
> Both discards can go in the same DISCARD block.
> (just style; it's how it's most often done in other linker scripts)

Thanks.  I also noticed the comment above the change needs an update.
Posting v2 shortly.

-- 
Josh

[PATCH v2] vmlinux.lds.h: Discard .note.gnu.property section

[Josh Poimboeuf]

When tooling reads ELF notes, it assumes each note entry is aligned to
the value listed in the .note section header's sh_addralign field.

The kernel-created ELF notes in the .note.Linux and .note.Xen sections
are aligned to 4 bytes.  This causes the toolchain to set those
sections' sh_addralign values to 4.

On the other hand, the GCC-created .note.gnu.property section has an
sh_addralign value of 8 for some reason, despite being based on struct
Elf32_Nhdr which only needs 4-byte alignment.

When the mismatched input sections get linked together into the vmlinux
.notes output section, the higher alignment "wins", resulting in an
sh_addralign of 8, which confuses tooling.  For example:

  $ readelf -n .tmp_vmlinux.btf
  ...
  readelf: .tmp_vmlinux.btf: Warning: note with invalid namesz and/or descsz found at offset 0x170
  readelf: .tmp_vmlinux.btf: Warning:  type: 0x4, namesize: 0x006e6558, descsize: 0x00008801, alignment: 8

In this case readelf thinks there's alignment padding where there is
none, so it starts reading an ELF note in the middle.

With newer toolchains (e.g., latest Fedora Rawhide), a similar mismatch
triggers a build failure when combined with CONFIG_X86_KERNEL_IBT:

  btf_encoder__encode: btf__dedup failed!
  Failed to encode BTF
  libbpf: failed to find '.BTF' ELF section in vmlinux
  FAILED: load BTF from vmlinux: No data available
  make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 255

This latter error was caused by pahole crashing when it encountered the
corrupt .notes section.  This crash has been fixed in dwarves version
1.25.  As Tianyi Liu describes:

  "Pahole reads .notes to look for LINUX_ELFNOTE_BUILD_LTO. When LTO is
   enabled, pahole needs to call cus__merge_and_process_cu to merge
   compile units, at which point there should only be one unspecified
   type (used to represent some compilation information) in the global
   context.

   However, when the kernel is compiled without LTO, if pahole calls
   cus__merge_and_process_cu due to alignment issues with notes,
   multiple unspecified types may appear after merging the cus, and
   older versions of pahole only support up to one. This is why pahole
   1.24 crashes, while newer versions support multiple. However, the
   latest version of pahole still does not solve the problem of
   incorrect LTO recognition, so compiling the kernel may be slower
   than normal."

Even with the newer pahole, the note section misaligment issue still
exists and pahole is misinterpreting the LTO note.  Fix it by discarding
the .note.gnu.property section.  While GNU properties are important for
user space (and VDSO), they don't seem to have any use for vmlinux.

(In fact, they're already getting (inadvertently) stripped from vmlinux
when CONFIG_DEBUG_INFO_BTF is enabled.  The BTF data is extracted from
vmlinux.o with "objcopy --only-section=.BTF" into .btf.vmlinux.bin.o.
That file doesn't have .note.gnu.property, so when it gets modified and
linked back into the main object, the linker automatically strips it
(see "How GNU properties are merged" in the ld man page).)

Reported-by: Daniel Xu <dxu@dxuuu.xyz>
Link: https://lkml.kernel.org/bpf/57830c30-cd77-40cf-9cd1-3bb608aa602e@app.fastmail.com
Debugged-by: Tianyi Liu <i.pear@outlook.com>
Suggested-by: Joan Bruguera Micó <joanbrugueram@gmail.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
v2:
- fixed link
- combined discards
- updated comment

 include/asm-generic/vmlinux.lds.h | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index d1f57e4868ed..cebdf1ca415d 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -891,9 +891,16 @@
 /*
  * Discard .note.GNU-stack, which is emitted as PROGBITS by the compiler.
  * Otherwise, the type of .notes section would become PROGBITS instead of NOTES.
+ *
+ * Also, discard .note.gnu.property, otherwise it forces the notes section to
+ * be 8-byte aligned which causes alignment mismatches with the kernel's custom
+ * 4-byte aligned notes.
  */
 #define NOTES								\
-	/DISCARD/ : { *(.note.GNU-stack) }				\
+	/DISCARD/ : {							\
+		*(.note.GNU-stack)					\
+		*(.note.gnu.property)					\
+	}								\
 	.notes : AT(ADDR(.notes) - LOAD_OFFSET) {			\
 		BOUNDED_SECTION_BY(.note.*, _notes)			\
 	} NOTES_HEADERS							\
-- 
2.39.2

Re: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Tianyi Liu]

On Wed, Apr 12, 2023 at 16:30PM UTC, Josh Poimboeuf wrote:
> On Wed, Apr 12, 2023 at 03:10:14PM +0800, Tianyi Liu wrote:
> > On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> > > On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > > > .tmp_vmlinux.btf
> > > > > btf_encoder__encode: btf__dedup failed!
> > > > > Failed to encode BTF
> > > > >
> > > > > Thanks,
> > > > >
> > > >
> > > > I encountered the same problem when building a new kernel and I found some
> > > > reasons for the error.
> > > >
> > > > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > > > .notes section. In addition, due to historical problems, the alignment of
> > > > records in the .notes section is not unified, which leads to the inability of
> > > > gelf_getnote() to read the records after the wrong one.
> > >
> > > Alexandre, Tianyi, are you still seeing this issue with the latest
> > > dwarves?  If so can you confirm the below patch fixes it?
> > >
> >
> > Josh, first of all, thank you very much for your help. However, this patch
> > doesn't work in my environment. I am using gcc 12.2.1 20230201.
> > After compiling, when I use readelf -S to view ELF sections,
> > the align of .notes section is still 8:
> >
> > $ readelf -S .tmp_vmlinux.btf
> > [20] .notes            NOTE             ffffffff8250b570  0170b570
> >      0000000000000084  0000000000000000   A       0     0     8
> 
> Hm, weird.

I have consulted some materials and found that using ALIGN() in linker
scripts can only "increase" alignment, not decrease it.

Perhaps could you try modifying your patch to use ALIGN(2) and SUBALIGN(2)
and see if the .notes section in the output file is aligned to 2?
In my tests, this had no effect.

[1] https://sourceware.org/binutils/docs/ld/Forced-Output-Alignment.html

> 
> > > Apparently the latest dwarves release fixes it on Fedora Rawhide [1],
> > > does anybody know if there a specific dwarves and/or libbpf change for
> > > this?
> > >
> >
> > It has been fixed in dwarves[1], but it may just be a coincidence.
> 
> So just to confirm, the btf__dedup error is gone for you with the latest
> dwarves?

Yes, this issue was fixed after a9498899109d3be14f17abbc322a8f55a1067bee
("dwarf_loader: Fix for BTF id drift caused by adding unspecified types"),
which will be included in version 1.25.

Re: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Joan Bruguera Micó]

On Thu, Apr 13, 2023 at 12:54:19PM +0800, Tianyi Liu wrote:
> On Wed, Apr 12, 2023 at 16:30PM UTC, Josh Poimboeuf wrote:
> > On Wed, Apr 12, 2023 at 03:10:14PM +0800, Tianyi Liu wrote:
> > > On Tue, Apr 11, 2023 at 17:00 , Josh Poimboeuf wrote:
> > > > On Tue, Feb 14, 2023 at 02:33:02PM +0800, Tianyi Liu wrote:
> > > > > > LLVM_OBJCOPY=objcopy pahole -J --btf_gen_floats -j
> > > > > > --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> > > > > > .tmp_vmlinux.btf
> > > > > > btf_encoder__encode: btf__dedup failed!
> > > > > > Failed to encode BTF
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > >
> > > > > I encountered the same problem when building a new kernel and I found some
> > > > > reasons for the error.
> > > > >
> > > > > In short, enabling CONFIG_X86_KERNEL_IBT will change the order of records in
> > > > > .notes section. In addition, due to historical problems, the alignment of
> > > > > records in the .notes section is not unified, which leads to the inability of
> > > > > gelf_getnote() to read the records after the wrong one.
> > > >
> > > > Alexandre, Tianyi, are you still seeing this issue with the latest
> > > > dwarves?  If so can you confirm the below patch fixes it?
> > > >
> > >
> > > Josh, first of all, thank you very much for your help. However, this patch
> > > doesn't work in my environment. I am using gcc 12.2.1 20230201.
> > > After compiling, when I use readelf -S to view ELF sections,
> > > the align of .notes section is still 8:
> > >
> > > $ readelf -S .tmp_vmlinux.btf
> > > [20] .notes            NOTE             ffffffff8250b570  0170b570
> > >      0000000000000084  0000000000000000   A       0     0     8
> > 
> > Hm, weird.
> 
> I have consulted some materials and found that using ALIGN() in linker
> scripts can only "increase" alignment, not decrease it.
> 
> Perhaps could you try modifying your patch to use ALIGN(2) and SUBALIGN(2)
> and see if the .notes section in the output file is aligned to 2?
> In my tests, this had no effect.
> 
> [1] https://sourceware.org/binutils/docs/ld/Forced-Output-Alignment.html

Not sure about the ld documentation (it may just be ambiguous wording)
but while doing some tests, I have found that Josh's ALIGN/SUBALIGN(4)
patch only works for certain kernel configs and fails for others.
This likely explains why Josh's patch worked for me but not for Tianyi.

I haven't investigated why or when it works though, but here's a config
where Josh's ALIGN(4) patch works and or where it doesn't (for me):
https://zealcharm.com/20230416-btf-dedup-bug-sample-configs/align-4-working
https://zealcharm.com/20230416-btf-dedup-bug-sample-configs/align-4-not-working

OTOH the new patch to discard .note.gnu.property works in both cases.

Re: [PATCH] vmlinux.lds.h: Force-align ELF notes section to four bytes

[Joan Bruguera Micó]

PS: As additional information for posterity, there are various conditions that
explain why this is/was only a problem on some specific distributions:

* In addition to dwarves/pahole 1.24, binutils 2.40 seems to also be required
  for the problem to occur (in Arch, downgrading to binutils 2.39 fixes it).
* Debian (and thus derived distributions) configures binutils with
  `--disable-x86-used-note`, which by default disables emitting the GNU notes
  which ultimately cause the .notes section to be aligned to 8 bytes.

  On Debian Bookworm & Ubuntu 23.04, the problem reproduces when building like:
  ```
  export KCFLAGS="-Xassembler -mx86-used-note=yes"
  export KAFLAGS="-Xassembler -mx86-used-note=yes"
  make
  ```

Finally, here's a smaller .config to reproduce the problem on affected systems:
https://zealcharm.com/20230416-btf-dedup-bug-sample-configs/minimal-repro