aarch64 binaries using nolibc segfault before reaching the entry point

aarch64 binaries using nolibc segfault before reaching the entry point

[Sebastian Ott]

Hi,

the tpidr2 selftest on an arm box segfaults before reaching the entry point.
I have no clue what is to blame for this or how to debug it but for a
statically linked binary there shouldn't be much stuff going on besides the
elf loader?

I can reproduce this with a program using an empty main function. Also checked
for other nolibc users - same result for init.c from rcutorture.

tools/testing/selftests/arm64/fp/za-fork is working though - the only
difference I could spot here is that it is linked together with another object
file. I also looked at the elf headers but didn't find anything obvious (but
I'm a bit out of my comfort zone here..)

After playing around with linker options I found that using -static-pie
lets the binaries run successful.

[root@arm abi]# cat test.c
int main(void)
{
         return 1;
}
[root@arm abi]# gcc -Os -static -Wall -lgcc -nostdlib -ffreestanding  -include ../../../../include/nolibc/nolibc.h  test.c
[root@arm abi]# ./a.out 
Segmentation fault
[root@arm abi]# gcc -Os -static -Wall -lgcc -nostdlib -ffreestanding -static-pie -include ../../../../include/nolibc/nolibc.h  test.c
[root@arm abi]# ./a.out 
[root@arm abi]#

All on aarch64 running fedora37 + upstream kernel. Any hints on what could
be borken here or how to actually fix it?

Sebastian

Re: aarch64 binaries using nolibc segfault before reaching the entry point

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

On Wed, Sep 13, 2023 at 08:44:59PM +0200, Sebastian Ott wrote:

> the tpidr2 selftest on an arm box segfaults before reaching the entry point.
> I have no clue what is to blame for this or how to debug it but for a
> statically linked binary there shouldn't be much stuff going on besides the
> elf loader?

I did notice this on some systems but didn't manage to reproduce it
interactively and hence debug it yet :/

> After playing around with linker options I found that using -static-pie
> lets the binaries run successful.

...

> All on aarch64 running fedora37 + upstream kernel. Any hints on what could
> be borken here or how to actually fix it?

That suggests that there's some randomisation of the executable location
going on which is triggering the issue...  TBH it seems sensible to take
a patch that enables that for now, it's clearly better to get the tests
running more reliably even if we don't fully understand the issue.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: aarch64 binaries using nolibc segfault before reaching the entry point

[Thomas Weißschuh]

On 2023-09-13 20:44:59+0200, Sebastian Ott wrote:
> Hi,
> 
> the tpidr2 selftest on an arm box segfaults before reaching the entry point.
> I have no clue what is to blame for this or how to debug it but for a
> statically linked binary there shouldn't be much stuff going on besides the
> elf loader?
> 
> I can reproduce this with a program using an empty main function. Also checked
> for other nolibc users - same result for init.c from rcutorture.
> 
> tools/testing/selftests/arm64/fp/za-fork is working though - the only
> difference I could spot here is that it is linked together with another object
> file. I also looked at the elf headers but didn't find anything obvious (but
> I'm a bit out of my comfort zone here..)
> 
> After playing around with linker options I found that using -static-pie
> lets the binaries run successful.
> 
> [root@arm abi]# cat test.c
> int main(void)
> {
>         return 1;
> }
> [root@arm abi]# gcc -Os -static -Wall -lgcc -nostdlib -ffreestanding  -include ../../../../include/nolibc/nolibc.h  test.c
> [root@arm abi]# ./a.out Segmentation fault
> [root@arm abi]# gcc -Os -static -Wall -lgcc -nostdlib -ffreestanding -static-pie -include ../../../../include/nolibc/nolibc.h  test.c
> [root@arm abi]# ./a.out [root@arm abi]#
> 
> All on aarch64 running fedora37 + upstream kernel. Any hints on what could
> be borken here or how to actually fix it?

I reduced it to the following reproducer:

    $ cat test.c
    int foo;  /* It works when deleting this variable */
    
    void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) _start(void)
    {
    	__asm__ volatile (
    		"mov x8, 93\n"       /* NR_exit == 93 */
    		"svc #0\n"
    	);
    	__builtin_unreachable();
    }
    
    $ aarch64-linux-gnu-gcc -Os -static -fno-stack-protector -Wall -nostdlib test.c
    $ ./a.out
    Segmentation fault

Also when running under gdb the error message is:

    During startup program terminated with signal SIGSEGV, Segmentation fault.

So it seems the error already happens during loading.

Could be a compiler or kernel bug?

Thomas

Re: aarch64 binaries using nolibc segfault before reaching the entry point

[Willy Tarreau]

On Wed, Sep 13, 2023 at 10:19:00PM +0200, Thomas Weißschuh wrote:
> > All on aarch64 running fedora37 + upstream kernel. Any hints on what could
> > be borken here or how to actually fix it?
> 
> I reduced it to the following reproducer:
> 
>     $ cat test.c
>     int foo;  /* It works when deleting this variable */
>     
>     void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) _start(void)
>     {
>     	__asm__ volatile (
>     		"mov x8, 93\n"       /* NR_exit == 93 */
>     		"svc #0\n"
>     	);
>     	__builtin_unreachable();
>     }
>     
>     $ aarch64-linux-gnu-gcc -Os -static -fno-stack-protector -Wall -nostdlib test.c
>     $ ./a.out
>     Segmentation fault
> 
> Also when running under gdb the error message is:
> 
>     During startup program terminated with signal SIGSEGV, Segmentation fault.
> 
> So it seems the error already happens during loading.
> 
> Could be a compiler or kernel bug?

I tried here with gcc-11.4.0 native on an ubuntu-22.04 and using a
cross gcc-9.5 executed on another box and couldn't reproduce the issue
at all. It could be that the compiler inserts whatever, did someone
try to disassemble de resulting program to see what it looks like ?
Maybe we're even dealing with issues related to random stack alignment
that causes issues past a function call due to some garbage being placed
at the wrong place in the stack. Also, dmesg should generally report
what (and where) the segv happened. Similarly, gdb with "info reg"
and "disassemble $pc" should report some info.

In my case, I just have this:

  $ objdump -d a.out 

  a.out:     file format elf64-littleaarch64


  Disassembly of section .text:

  0000000000400144 <_start>:
    400144:       d2800ba8        mov     x8, #0x5d                       // #93
    400148:       d4000001        svc     #0x0

The kernel is a 6.2:

  $ uname -a
  Linux ampere 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 20:49:15 UTC 2 aarch64 aarch64 aarch64 GNU/Linux

Cheers,
Willy

Re: aarch64 binaries using nolibc segfault before reaching the entry point

[Thomas Weißschuh]

On 2023-09-13 22:58:38+0200, Willy Tarreau wrote:
> On Wed, Sep 13, 2023 at 10:19:00PM +0200, Thomas Weißschuh wrote:
> > > All on aarch64 running fedora37 + upstream kernel. Any hints on what could
> > > be borken here or how to actually fix it?
> > 
> > I reduced it to the following reproducer:
> > 
> >     $ cat test.c
> >     int foo;  /* It works when deleting this variable */
> >     
> >     void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) _start(void)
> >     {
> >     	__asm__ volatile (
> >     		"mov x8, 93\n"       /* NR_exit == 93 */
> >     		"svc #0\n"
> >     	);
> >     	__builtin_unreachable();
> >     }
> >     
> >     $ aarch64-linux-gnu-gcc -Os -static -fno-stack-protector -Wall -nostdlib test.c
> >     $ ./a.out
> >     Segmentation fault
> > 
> > Also when running under gdb the error message is:
> > 
> >     During startup program terminated with signal SIGSEGV, Segmentation fault.
> > 
> > So it seems the error already happens during loading.
> > 
> > Could be a compiler or kernel bug?
> 
> I tried here with gcc-11.4.0 native on an ubuntu-22.04 and using a
> cross gcc-9.5 executed on another box and couldn't reproduce the issue
> at all. It could be that the compiler inserts whatever, did someone
> try to disassemble de resulting program to see what it looks like ?
> Maybe we're even dealing with issues related to random stack alignment
> that causes issues past a function call due to some garbage being placed
> at the wrong place in the stack. Also, dmesg should generally report
> what (and where) the segv happened. Similarly, gdb with "info reg"
> and "disassemble $pc" should report some info.

Im using:

aarch64-linux-gnu-gcc (GCC) 13.2.0

It's reproducible reliably.

No output in dmesg, binary works in qemu-user.
There should be no function calls at all, or?
GDB also doesn't show any registers, it seems to fail before anything is
executed.

> 
> In my case, I just have this:
> 
>   $ objdump -d a.out 
> 
>   a.out:     file format elf64-littleaarch64
> 
> 
>   Disassembly of section .text:
> 
>   0000000000400144 <_start>:
>     400144:       d2800ba8        mov     x8, #0x5d                       // #93
>     400148:       d4000001        svc     #0x0

Looks absolutely identical for me.

> 
> The kernel is a 6.2:
> 
>   $ uname -a
>   Linux ampere 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 20:49:15 UTC 2 aarch64 aarch64 aarch64 GNU/Linux

Linux fedora-4gb-fsn1-1 6.4.11-200.fc38.aarch64 #1 SMP PREEMPT_DYNAMIC Wed Aug 16 18:01:59 UTC 2023 aarch64 GNU/Linux

(Just a default ARM VM on Hetzner with Fedora 38)

Re: aarch64 binaries using nolibc segfault before reaching the entry point

[Sebastian Ott]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

On Wed, 13 Sep 2023, Willy Tarreau wrote:
> On Wed, Sep 13, 2023 at 10:19:00PM +0200, Thomas Weißschuh wrote:
>>> All on aarch64 running fedora37 + upstream kernel. Any hints on what could
>>> be borken here or how to actually fix it?
>>
>> I reduced it to the following reproducer:
>>
>>     $ cat test.c
>>     int foo;  /* It works when deleting this variable */
>>
>>     void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) _start(void)
>>     {
>>     	__asm__ volatile (
>>     		"mov x8, 93\n"       /* NR_exit == 93 */
>>     		"svc #0\n"
>>     	);
>>     	__builtin_unreachable();
>>     }
>>
>>     $ aarch64-linux-gnu-gcc -Os -static -fno-stack-protector -Wall -nostdlib test.c
>>     $ ./a.out
>>     Segmentation fault
>>
>> Also when running under gdb the error message is:
>>
>>     During startup program terminated with signal SIGSEGV, Segmentation fault.
>>
>> So it seems the error already happens during loading.
>>
>> Could be a compiler or kernel bug?
>
> I tried here with gcc-11.4.0 native on an ubuntu-22.04 and using a
> cross gcc-9.5 executed on another box and couldn't reproduce the issue
> at all. It could be that the compiler inserts whatever, did someone
> try to disassemble de resulting program to see what it looks like ?
> Maybe we're even dealing with issues related to random stack alignment
> that causes issues past a function call due to some garbage being placed
> at the wrong place in the stack. Also, dmesg should generally report
> what (and where) the segv happened. Similarly, gdb with "info reg"
> and "disassemble $pc" should report some info.

Sadly there is no kernel output for this I guess this happens just too
early. gdb just reports "The program has no registers now."

Sebastian

Re: aarch64 binaries using nolibc segfault before reaching the entry point

[Thomas Weißschuh]

On 2023-09-13 22:19:00+0200, Thomas Weißschuh wrote:
> On 2023-09-13 20:44:59+0200, Sebastian Ott wrote:
> > the tpidr2 selftest on an arm box segfaults before reaching the entry point.
> > I have no clue what is to blame for this or how to debug it but for a
> > statically linked binary there shouldn't be much stuff going on besides the
> > elf loader?

> [..]

> 
> I reduced it to the following reproducer:
> 
>     $ cat test.c
>     int foo;  /* It works when deleting this variable */
>     
>     void __attribute__((weak, noreturn, optimize("Os", "omit-frame-pointer"))) _start(void)
>     {
>     	__asm__ volatile (
>     		"mov x8, 93\n"       /* NR_exit == 93 */
>     		"svc #0\n"
>     	);
>     	__builtin_unreachable();
>     }
>     
>     $ aarch64-linux-gnu-gcc -Os -static -fno-stack-protector -Wall -nostdlib test.c
>     $ ./a.out
>     Segmentation fault
> 
> Also when running under gdb the error message is:
> 
>     During startup program terminated with signal SIGSEGV, Segmentation fault.
> 
> So it seems the error already happens during loading.
> 
> Could be a compiler or kernel bug?

Callchain for the failure:

load_elf_binary()
  -> if (likely(elf_bss != elf_brk) && unlikely(padzero(elf_bess)))
    -> padzero()
      -> clear_user()
        -> __arch_clear_user()
	  -> failure in arch/arm64/lib/clear_user.S

Resulting in a EFAULT which gets translated to SIGSEGV somewhere.


The following patch, which seems sensible to me, fixes it for me.
But as this is really old, heavily used code I'm a bit hesitant.

diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index 7b3d2d491407..13f71733ba63 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -112,7 +112,7 @@ static struct linux_binfmt elf_format = {
 
 static int set_brk(unsigned long start, unsigned long end, int prot)
 {
-	start = ELF_PAGEALIGN(start);
+	start = ELF_PAGESTART(start);
 	end = ELF_PAGEALIGN(end);
 	if (end > start) {
 		/*