[PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Daniel Palmer]

bsr has limited displacement and cannot reach _start_c if its
more than +/-4KiB away.

This starts to show up if you have big programs (static binary
of the game Quake in my case) and/or use LTO and things get moved
around.

Replace bsr with a calculation of the effective address of
_start_c and jsr.

Tested on QEMU and real SH4 and SH4a machines.

Signed-off-by: Daniel Palmer <daniel@0x0f.com>
---

I'm no SuperH asm expert but this works for me. :)
Maybe there is a way in C to get _start and _start_c to always
be close together that would avoid this entirely?

 tools/include/nolibc/arch-sh.h | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tools/include/nolibc/arch-sh.h b/tools/include/nolibc/arch-sh.h
index 996cfbe1bee3..7318f52dcba9 100644
--- a/tools/include/nolibc/arch-sh.h
+++ b/tools/include/nolibc/arch-sh.h
@@ -154,8 +154,13 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_prote
 		"_start:\n"
 
 		"mov sp, r4\n"               /* save argc pointer to r4, as arg1 of _start_c */
-		"bsr _start_c\n"             /* transfer to c runtime                        */
+		"mova 1f, r0\n"              /* load the effective address of 1: into r0     */
+		"mov.l @r0, r1\n"            /* load the distance to _start_c from 1:        */
+		"add r0, r1\n"               /* calculate address of _start_c                */
+		"jsr @r1\n"                  /* transfer to c runtime                        */
 		"nop\n"                      /* delay slot                                   */
+		".balign 4\n"
+		"1: .long _start_c - 1b\n"   /* Distance from here to _start_c               */
 
 		".size _start, .-_start\n"
 	);
-- 
2.55.0

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Thomas Weißschuh]

Hi Daniel!

On 2026-10-22 19:01:59+0900, Daniel Palmer wrote:
> bsr has limited displacement and cannot reach _start_c if its
> more than +/-4KiB away.
> 
> This starts to show up if you have big programs (static binary
> of the game Quake in my case) and/or use LTO and things get moved
> around.

That is certainly a usecase that justifies making the two other SH4
nolibc users' binaries a bit larger :-)

> Replace bsr with a calculation of the effective address of
> _start_c and jsr.
> 
> Tested on QEMU and real SH4 and SH4a machines.
> 
> Signed-off-by: Daniel Palmer <daniel@0x0f.com>
> ---
> 
> I'm no SuperH asm expert but this works for me. :)
> Maybe there is a way in C to get _start and _start_c to always
> be close together that would avoid this entirely?

That would be my preference, too.

Maybe one of these works:

1) Put both _start() and _start_c() in a dedicated .text subsection.

Apparently the default GNU ld linkerscript contains '*(SORT(.text.sorted.*))'.
So if we use .text.sorted.nolibc_start for both functions they should
end up next to each other.
See https://sourceware.org/binutils/docs/ld/Input-Section-Wildcards.html.

2) Put _start() *into* _start_c:

diff --git a/tools/include/nolibc/crt.h b/tools/include/nolibc/crt.h
index 714256228914..ebcd0ea2b9f2 100644
--- a/tools/include/nolibc/crt.h
+++ b/tools/include/nolibc/crt.h
@@ -115,6 +115,19 @@ void _start_c(long *sp)
                (*--dtor_func)();

        exit(exitcode);
+
+       __asm__ volatile (
+               ".global _start\n"           /* The C function will have a prologue,         */
+               ".type _start, @function\n"  /* corrupting "sp"                              */
+               ".weak _start\n"
+               "_start:\n"
+
+               "mov sp, r4\n"               /* save argc pointer to r4, as arg1 of _start_c */
+               "bsr _start_c\n"             /* transfer to c runtime                        */
+               "nop\n"                      /* delay slot                                   */
+
+               ".size _start, .-_start\n"
+       );
 }

 #endif /* NOLIBC_NO_RUNTIME */

But that for all architectures, using macros etc.

If the .text.sorted. solution works, I'd prefer that.

>  tools/include/nolibc/arch-sh.h | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/include/nolibc/arch-sh.h b/tools/include/nolibc/arch-sh.h
> index 996cfbe1bee3..7318f52dcba9 100644
> --- a/tools/include/nolibc/arch-sh.h
> +++ b/tools/include/nolibc/arch-sh.h
> @@ -154,8 +154,13 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_prote
>  		"_start:\n"
>  
>  		"mov sp, r4\n"               /* save argc pointer to r4, as arg1 of _start_c */
> -		"bsr _start_c\n"             /* transfer to c runtime                        */
> +		"mova 1f, r0\n"              /* load the effective address of 1: into r0     */
> +		"mov.l @r0, r1\n"            /* load the distance to _start_c from 1:        */
> +		"add r0, r1\n"               /* calculate address of _start_c                */
> +		"jsr @r1\n"                  /* transfer to c runtime                        */
>  		"nop\n"                      /* delay slot                                   */
> +		".balign 4\n"
> +		"1: .long _start_c - 1b\n"   /* Distance from here to _start_c               */
>  
>  		".size _start, .-_start\n"
>  	);
> -- 
> 2.55.0
> 

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Willy Tarreau]

Hi guys,

On Thu, Oct 22, 2026 at 06:12:27PM +0200, Thomas Weißschuh wrote:
> Hi Daniel!
> 
> On 2026-10-22 19:01:59+0900, Daniel Palmer wrote:
> > bsr has limited displacement and cannot reach _start_c if its
> > more than +/-4KiB away.
> > 
> > This starts to show up if you have big programs (static binary
> > of the game Quake in my case) and/or use LTO and things get moved
> > around.
> 
> That is certainly a usecase that justifies making the two other SH4
> nolibc users' binaries a bit larger :-)

Indeed!

> > Replace bsr with a calculation of the effective address of
> > _start_c and jsr.
> > 
> > Tested on QEMU and real SH4 and SH4a machines.
> > 
> > Signed-off-by: Daniel Palmer <daniel@0x0f.com>
> > ---
> > 
> > I'm no SuperH asm expert but this works for me. :)
> > Maybe there is a way in C to get _start and _start_c to always
> > be close together that would avoid this entirely?
> 
> That would be my preference, too.
> 
> Maybe one of these works:
> 
> 1) Put both _start() and _start_c() in a dedicated .text subsection.
> 
> Apparently the default GNU ld linkerscript contains '*(SORT(.text.sorted.*))'.
> So if we use .text.sorted.nolibc_start for both functions they should
> end up next to each other.
> See https://sourceware.org/binutils/docs/ld/Input-Section-Wildcards.html.
> 
> 2) Put _start() *into* _start_c:
> 
> diff --git a/tools/include/nolibc/crt.h b/tools/include/nolibc/crt.h
> index 714256228914..ebcd0ea2b9f2 100644
> --- a/tools/include/nolibc/crt.h
> +++ b/tools/include/nolibc/crt.h
> @@ -115,6 +115,19 @@ void _start_c(long *sp)
>                 (*--dtor_func)();
> 
>         exit(exitcode);
> +
> +       __asm__ volatile (
> +               ".global _start\n"           /* The C function will have a prologue,         */
> +               ".type _start, @function\n"  /* corrupting "sp"                              */
> +               ".weak _start\n"
> +               "_start:\n"
> +
> +               "mov sp, r4\n"               /* save argc pointer to r4, as arg1 of _start_c */
> +               "bsr _start_c\n"             /* transfer to c runtime                        */
> +               "nop\n"                      /* delay slot                                   */
> +
> +               ".size _start, .-_start\n"
> +       );
>  }
> 
>  #endif /* NOLIBC_NO_RUNTIME */
> 
> But that for all architectures, using macros etc.
> 
> If the .text.sorted. solution works, I'd prefer that.

Do be honest, I think that Daniel's proposal remains the most maintainable
one. We rarely touch the start code, and having only a few do arch-
specific tricks is likely less of a burden for the long term than
relying on linker's ordering of sections, or starting to play macro
games. And it gets a chance that someone more familiar with that arch
improves it later if needed. But I'm not opiniated on this at all :-)

Thanks,
Willy

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Thomas Weißschuh]

On 2026-10-23 07:09:55+0200, Willy Tarreau wrote:
(...)
> > > Replace bsr with a calculation of the effective address of
> > > _start_c and jsr.
> > > 
> > > Tested on QEMU and real SH4 and SH4a machines.
> > > 
> > > Signed-off-by: Daniel Palmer <daniel@0x0f.com>
> > > ---
> > > 
> > > I'm no SuperH asm expert but this works for me. :)
> > > Maybe there is a way in C to get _start and _start_c to always
> > > be close together that would avoid this entirely?
> > 
> > That would be my preference, too.
> > 
> > Maybe one of these works:
> > 
> > 1) Put both _start() and _start_c() in a dedicated .text subsection.
> > 
> > Apparently the default GNU ld linkerscript contains '*(SORT(.text.sorted.*))'.
> > So if we use .text.sorted.nolibc_start for both functions they should
> > end up next to each other.
> > See https://sourceware.org/binutils/docs/ld/Input-Section-Wildcards.html.
> > 
> > 2) Put _start() *into* _start_c:
> > 
> > diff --git a/tools/include/nolibc/crt.h b/tools/include/nolibc/crt.h
> > index 714256228914..ebcd0ea2b9f2 100644
> > --- a/tools/include/nolibc/crt.h
> > +++ b/tools/include/nolibc/crt.h
> > @@ -115,6 +115,19 @@ void _start_c(long *sp)
> >                 (*--dtor_func)();
> > 
> >         exit(exitcode);
> > +
> > +       __asm__ volatile (
> > +               ".global _start\n"           /* The C function will have a prologue,         */
> > +               ".type _start, @function\n"  /* corrupting "sp"                              */
> > +               ".weak _start\n"
> > +               "_start:\n"
> > +
> > +               "mov sp, r4\n"               /* save argc pointer to r4, as arg1 of _start_c */
> > +               "bsr _start_c\n"             /* transfer to c runtime                        */
> > +               "nop\n"                      /* delay slot                                   */
> > +
> > +               ".size _start, .-_start\n"
> > +       );
> >  }
> > 
> >  #endif /* NOLIBC_NO_RUNTIME */
> > 
> > But that for all architectures, using macros etc.
> > 
> > If the .text.sorted. solution works, I'd prefer that.
> 
> Do be honest, I think that Daniel's proposal remains the most maintainable
> one. We rarely touch the start code, and having only a few do arch-
> specific tricks is likely less of a burden for the long term than
> relying on linker's ordering of sections, or starting to play macro
> games. And it gets a chance that someone more familiar with that arch
> improves it later if needed. But I'm not opiniated on this at all :-)

I fear that we will have the same problems in other architectures.
And for those we'll also need custom assembly. While the mechanisms
above would not need that.

The sorted linker sections should be stable. They have been introduced
as an explicit feature by GNU ld. They are fairly new (2019), but they
should fall back to non-sorted linking just fine for older linkers.

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Willy Tarreau]

On Fri, Oct 23, 2026 at 05:53:51PM +0200, Thomas Weißschuh wrote:
> On 2026-10-23 07:09:55+0200, Willy Tarreau wrote:
> (...)
> > > > Replace bsr with a calculation of the effective address of
> > > > _start_c and jsr.
> > > > 
> > > > Tested on QEMU and real SH4 and SH4a machines.
> > > > 
> > > > Signed-off-by: Daniel Palmer <daniel@0x0f.com>
> > > > ---
> > > > 
> > > > I'm no SuperH asm expert but this works for me. :)
> > > > Maybe there is a way in C to get _start and _start_c to always
> > > > be close together that would avoid this entirely?
> > > 
> > > That would be my preference, too.
> > > 
> > > Maybe one of these works:
> > > 
> > > 1) Put both _start() and _start_c() in a dedicated .text subsection.
> > > 
> > > Apparently the default GNU ld linkerscript contains '*(SORT(.text.sorted.*))'.
> > > So if we use .text.sorted.nolibc_start for both functions they should
> > > end up next to each other.
> > > See https://sourceware.org/binutils/docs/ld/Input-Section-Wildcards.html.
> > > 
> > > 2) Put _start() *into* _start_c:
> > > 
> > > diff --git a/tools/include/nolibc/crt.h b/tools/include/nolibc/crt.h
> > > index 714256228914..ebcd0ea2b9f2 100644
> > > --- a/tools/include/nolibc/crt.h
> > > +++ b/tools/include/nolibc/crt.h
> > > @@ -115,6 +115,19 @@ void _start_c(long *sp)
> > >                 (*--dtor_func)();
> > > 
> > >         exit(exitcode);
> > > +
> > > +       __asm__ volatile (
> > > +               ".global _start\n"           /* The C function will have a prologue,         */
> > > +               ".type _start, @function\n"  /* corrupting "sp"                              */
> > > +               ".weak _start\n"
> > > +               "_start:\n"
> > > +
> > > +               "mov sp, r4\n"               /* save argc pointer to r4, as arg1 of _start_c */
> > > +               "bsr _start_c\n"             /* transfer to c runtime                        */
> > > +               "nop\n"                      /* delay slot                                   */
> > > +
> > > +               ".size _start, .-_start\n"
> > > +       );
> > >  }
> > > 
> > >  #endif /* NOLIBC_NO_RUNTIME */
> > > 
> > > But that for all architectures, using macros etc.
> > > 
> > > If the .text.sorted. solution works, I'd prefer that.
> > 
> > Do be honest, I think that Daniel's proposal remains the most maintainable
> > one. We rarely touch the start code, and having only a few do arch-
> > specific tricks is likely less of a burden for the long term than
> > relying on linker's ordering of sections, or starting to play macro
> > games. And it gets a chance that someone more familiar with that arch
> > improves it later if needed. But I'm not opiniated on this at all :-)
> 
> I fear that we will have the same problems in other architectures.
> And for those we'll also need custom assembly. While the mechanisms
> above would not need that.

But all archs support indirect calls one way or another. Normally we
should be able to do the equivalent of placing an absolute value in a
storage area, load it in a register and branch to it at least. For me,
sorted sections are most likely meant to be used to group code by
locality, not to guarantee that branch offsets will always remain
small enough for all architectures. An indirect call on the other
hand is well defined, standard and can be adopted where needed at the
cost of only 4 or 8 bytes in the startup code.

Willy

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Thomas Weißschuh]

On 2026-10-23 18:20:55+0200, Willy Tarreau wrote:
> On Fri, Oct 23, 2026 at 05:53:51PM +0200, Thomas Weißschuh wrote:
> > On 2026-10-23 07:09:55+0200, Willy Tarreau wrote:
> > (...)
> > > > > Replace bsr with a calculation of the effective address of
> > > > > _start_c and jsr.
> > > > > 
> > > > > Tested on QEMU and real SH4 and SH4a machines.
> > > > > 
> > > > > Signed-off-by: Daniel Palmer <daniel@0x0f.com>
> > > > > ---
> > > > > 
> > > > > I'm no SuperH asm expert but this works for me. :)
> > > > > Maybe there is a way in C to get _start and _start_c to always
> > > > > be close together that would avoid this entirely?
> > > > 
> > > > That would be my preference, too.
> > > > 
> > > > Maybe one of these works:
> > > > 
> > > > 1) Put both _start() and _start_c() in a dedicated .text subsection.
> > > > 
> > > > Apparently the default GNU ld linkerscript contains '*(SORT(.text.sorted.*))'.
> > > > So if we use .text.sorted.nolibc_start for both functions they should
> > > > end up next to each other.
> > > > See https://sourceware.org/binutils/docs/ld/Input-Section-Wildcards.html.
> > > > 
> > > > 2) Put _start() *into* _start_c:
> > > > 
> > > > diff --git a/tools/include/nolibc/crt.h b/tools/include/nolibc/crt.h
> > > > index 714256228914..ebcd0ea2b9f2 100644
> > > > --- a/tools/include/nolibc/crt.h
> > > > +++ b/tools/include/nolibc/crt.h
> > > > @@ -115,6 +115,19 @@ void _start_c(long *sp)
> > > >                 (*--dtor_func)();
> > > > 
> > > >         exit(exitcode);
> > > > +
> > > > +       __asm__ volatile (
> > > > +               ".global _start\n"           /* The C function will have a prologue,         */
> > > > +               ".type _start, @function\n"  /* corrupting "sp"                              */
> > > > +               ".weak _start\n"
> > > > +               "_start:\n"
> > > > +
> > > > +               "mov sp, r4\n"               /* save argc pointer to r4, as arg1 of _start_c */
> > > > +               "bsr _start_c\n"             /* transfer to c runtime                        */
> > > > +               "nop\n"                      /* delay slot                                   */
> > > > +
> > > > +               ".size _start, .-_start\n"
> > > > +       );
> > > >  }
> > > > 
> > > >  #endif /* NOLIBC_NO_RUNTIME */
> > > > 
> > > > But that for all architectures, using macros etc.
> > > > 
> > > > If the .text.sorted. solution works, I'd prefer that.
> > > 
> > > Do be honest, I think that Daniel's proposal remains the most maintainable
> > > one. We rarely touch the start code, and having only a few do arch-
> > > specific tricks is likely less of a burden for the long term than
> > > relying on linker's ordering of sections, or starting to play macro
> > > games. And it gets a chance that someone more familiar with that arch
> > > improves it later if needed. But I'm not opiniated on this at all :-)
> > 
> > I fear that we will have the same problems in other architectures.
> > And for those we'll also need custom assembly. While the mechanisms
> > above would not need that.
> 
> But all archs support indirect calls one way or another. Normally we
> should be able to do the equivalent of placing an absolute value in a
> storage area, load it in a register and branch to it at least.

Of course it is possible to do. But it differs between architectures and
somebody will have to write and maintain that assembly code.

> For me, sorted sections are most likely meant to be used to group code by
> locality, not to guarantee that branch offsets will always remain
> small enough for all architectures.

Exactly this locality should provide small enough branch offsets.
The section ".text.sorted.nolibc_crt" should only ever have two functions:
Our _start() and _start_c().
While there won't be a formal guarantee that the architecture's branch
offsets will be able to handle those, I find it hard to imagine that
they won't.

> An indirect call on the other hand is well defined, standard and can
> be adopted where needed at the cost of only 4 or 8 bytes in the
> startup code.

Its a few more bytes, not that it really matters.
* 3 additional instructions -> 6 bytes
* The embedded offset -> 4 bytes
* Up to 3 bytes of alignment

The adoption will require additional fiddling with assembly.

Maybe we let Daniel break the tie? :-)


Thomas

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Willy Tarreau]

On Fri, Oct 23, 2026 at 07:32:55PM +0200, Thomas Weißschuh wrote:
> > > I fear that we will have the same problems in other architectures.
> > > And for those we'll also need custom assembly. While the mechanisms
> > > above would not need that.
> > 
> > But all archs support indirect calls one way or another. Normally we
> > should be able to do the equivalent of placing an absolute value in a
> > storage area, load it in a register and branch to it at least.
> 
> Of course it is possible to do. But it differs between architectures and
> somebody will have to write and maintain that assembly code.
> 
> > For me, sorted sections are most likely meant to be used to group code by
> > locality, not to guarantee that branch offsets will always remain
> > small enough for all architectures.
> 
> Exactly this locality should provide small enough branch offsets.
> The section ".text.sorted.nolibc_crt" should only ever have two functions:
> Our _start() and _start_c().
> While there won't be a formal guarantee that the architecture's branch
> offsets will be able to handle those, I find it hard to imagine that
> they won't.

It has worked till then at least so it should.

> > An indirect call on the other hand is well defined, standard and can
> > be adopted where needed at the cost of only 4 or 8 bytes in the
> > startup code.
> 
> Its a few more bytes, not that it really matters.
> * 3 additional instructions -> 6 bytes
> * The embedded offset -> 4 bytes
> * Up to 3 bytes of alignment

It depends on the arch in fact but that remains the order of magnitude.

> The adoption will require additional fiddling with assembly.

Yes I agree.

> Maybe we let Daniel break the tie? :-)

The advantage of the ld-based one is that it should be the lowest effort
across all archs at once, so we should probably try with this one first.
If we later discover that it displaces the problem somewhere else, then
you can reconsider it and switch to the asm one.

Willy

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Daniel Palmer]

Hi Willy, Thomas

On Sat, 24 Oct 2026 at 03:35, Willy Tarreau <w@1wt.eu> wrote:
> > Maybe we let Daniel break the tie? :-)

mmm I have the same big binary (Quake using nolibc + a thin DRM/KMS +
ALSA layer, some libm bits from musl) on 32 bit ARM and x86-64 and
they were fine so my gut feeling would be that this is an SH issue and
a few extra bytes is fine for such a niche arch.
On SH this also seems to happen for much more normal binaries if you
enable LTO and get unlucky which points me towards letting the SH
entry code deal with it.

But also I'm still working on the static PIE thing so nolibc can
create binaries that nommu can run and in that case it would probably
be helpful for all of the code that runs before relocation to be close
together.

I'll have a go at the ld solution and see what happens.

Slightly different topic but related: Another thing on my pile of
patches is some stuff to extend memcpy() slightly so it generates
word-by-word copies where possible without passing -O3. This helps
m68k and SH a lot but everyone would end up with the extra bytes for
the word-by-word copy loop and the check if fallback to byte-by-byte
is needed. Would wrapping that in something like "#if
NOLIBC_FAST_MEMCPY" be acceptable?

Cheers,

Daniel

Re: [PATCH] tools/nolibc: sh: enter _start_c with jsr instead of bsr

[Willy Tarreau]

Hi Daniel,

On Sat, Oct 24, 2026 at 02:16:10PM +0900, Daniel Palmer wrote:
> Hi Willy, Thomas
> 
> On Sat, 24 Oct 2026 at 03:35, Willy Tarreau <w@1wt.eu> wrote:
> > > Maybe we let Daniel break the tie? :-)
> 
> mmm I have the same big binary (Quake using nolibc + a thin DRM/KMS +
> ALSA layer, some libm bits from musl) on 32 bit ARM and x86-64 and
> they were fine so my gut feeling would be that this is an SH issue and
> a few extra bytes is fine for such a niche arch.

x86 can call 32-bit displacements, so you have some margin there ;-)

> On SH this also seems to happen for much more normal binaries if you
> enable LTO and get unlucky which points me towards letting the SH
> entry code deal with it.
> 
> But also I'm still working on the static PIE thing so nolibc can
> create binaries that nommu can run and in that case it would probably
> be helpful for all of the code that runs before relocation to be close
> together.
> 
> I'll have a go at the ld solution and see what happens.

OK!

> Slightly different topic but related: Another thing on my pile of
> patches is some stuff to extend memcpy() slightly so it generates
> word-by-word copies where possible without passing -O3. This helps
> m68k and SH a lot but everyone would end up with the extra bytes for
> the word-by-word copy loop and the check if fallback to byte-by-byte
> is needed. Would wrapping that in something like "#if
> NOLIBC_FAST_MEMCPY" be acceptable?

I've been thinking about similar things a few times, and I think that
for such cases where we implementa naive function for size purposes, we
could also rely on #ifdef __OPTIMIZE_SIZE__ to keep the small one and
switch to the faster one otherwise. That way we can respect each user's
expectations.

Willy