Re: #! incompatible -- binfmt_script.c broken?

Re: #! incompatible -- binfmt_script.c broken?

[Chris Adams]

Once upon a time, Matthias Andree <matthias.andree@gmx.de> said:
>Nope. It cannot be correct if it breaks compatibility without giving us
>any advantage.

Compatibility with _what_?  Linux works exactly the same as Tru64 Unix
for example - it supplies everything after the program to execute as the
first argument.  Are you going to make the kernel honor the setting of
the IFS environment variable?  Should it split only on space?  What
about tab?  Or $LANG (maybe space is different in different character
sets)?

The difference is that Tru64's /bin/sh (and /usr/bin/posix/sh and
/usr/bin/ksh) stops processing the argument and continues without error
after it hits a (single or double) dash and a space, while bash (and
pdksh and ash and zsh) don't handle that.

Try the following under your shell.  On Solaris and Tru64 sh and ksh, it
is handled with no error.  Under bash (on Linux, Solaris, and Tru64), it
returns an error:

$ set "-- xyzzy"
$ echo $?

According to SUSv3, bash is not compliant, because for set, under the
section "CONSEQUENCES OF ERRORS" is listed "None." and the "EXIT STATUS"
is "Zero."

Fix the shell(s).
-- 
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: #! incompatible -- binfmt_script.c broken?

[Matthias Andree]

On Wed, 04 Dec 2002, Chris Adams wrote:

> Try the following under your shell.  On Solaris and Tru64 sh and ksh, it
> is handled with no error.  Under bash (on Linux, Solaris, and Tru64), it
> returns an error:
> 
> $ set "-- xyzzy"
> $ echo $?
> 
> According to SUSv3, bash is not compliant, because for set, under the
> section "CONSEQUENCES OF ERRORS" is listed "None." and the "EXIT STATUS"
> is "Zero."

> Fix the shell(s).

That's correct. But how do you derive that the sh command line must
behave the same? The sh command is not the sh special built-in.

However, it would be reasonable if a /bin/sh set $1 to be "-- xyzzy" if
a file "foo" with

#! /bin/sh -- xyzzy

was executed (as path = [/bin/sh] argv = [./foo] [-- xyzzy]);
and although I didn't check, I wonder how shells without the "--" long
options parse that line.

Re: #! incompatible -- binfmt_script.c broken?

[Andreas Schwab]

Matthias Andree <matthias.andree@gmx.de> writes:

|> On Wed, 04 Dec 2002, Chris Adams wrote:
|> 
|> > Try the following under your shell.  On Solaris and Tru64 sh and ksh, it
|> > is handled with no error.  Under bash (on Linux, Solaris, and Tru64), it
|> > returns an error:
|> > 
|> > $ set "-- xyzzy"
|> > $ echo $?
|> > 
|> > According to SUSv3, bash is not compliant, because for set, under the
|> > section "CONSEQUENCES OF ERRORS" is listed "None." and the "EXIT STATUS"
|> > is "Zero."
|> 
|> > Fix the shell(s).
|> 
|> That's correct. But how do you derive that the sh command line must
|> behave the same? The sh command is not the sh special built-in.
|> 
|> However, it would be reasonable if a /bin/sh set $1 to be "-- xyzzy" if
|> a file "foo" with
|> 
|> #! /bin/sh -- xyzzy
|> 
|> was executed (as path = [/bin/sh] argv = [./foo] [-- xyzzy]);
|> and although I didn't check, I wonder how shells without the "--" long
|> options parse that line.

POSIX is quite clear about that: only "--" as a single argument is
defined, other uses are undefined.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: #! incompatible -- binfmt_script.c broken?

[Matthias Andree]

On Wed, 04 Dec 2002, Matthew Garrett wrote:

> See http://www.uni-ulm.de/~s_smasch/various/shebang/ . FreeBSD is the
> *only* OS to pass multiple arguments. SUS says nothing about it, and
> pretty much every single implementation varies. It does not work
> everywhere else.

Now that's a useful resource.

> File a bug against perlrun(1).

Hum, no. In fact, the offending line will work:

- on systems that only pass on the first argument or none at all
- on systems that split on whitespace
- on /bin/sh that silently ignore options starting with "-- "

So it's somewhat compatible already.

Re: #! incompatible -- binfmt_script.c broken?

[Matthias Andree]

On Wed, 04 Dec 2002, Horst von Brand wrote:

> Alex Riesen <Alexander.Riesen@synopsys.com> said:
> 
> [...]
> 
> > looks correct. The interpreter (/bin/sh) has got everything after
> > its name. IOW: "-- # -*- perl -*- -T"
> > It's just solaris' shell (/bin/sh) just ignores options starting with
> > "--". And freebsd's as well.
> 
> And Linux's too. Try it.

Is there the "Linux's /bin/sh"? I believe most distributions use GNU
bash for /bin/sh, and that certainly does NOT ignore these, but parse.

The problem is that binfmt_script.c does not split the remainder of the
line at whitespace.

Assuming your current working directory does not have files that match
-*-:

$ /bin/sh '-- # -*- perl -*- -T'
/bin/sh: -- # -*- perl -*- -T: invalid option
Usage:  /bin/sh [GNU long option] [option] ...
        /bin/sh [GNU long option] [option] script-file ...
GNU long options:
        --debug
...

$ /bin/bash -- # -*- perl -*- -T
$ 

$ /usr/bin/ksh '-- # -*- perl -*- -T'
/usr/bin/ksh: /usr/bin/ksh: --: unknown option

$ /usr/bin/ksh -- # -*- perl -*- -T
$

$ /usr/bin/zsh '-- # -*- perl -*- -T'
/usr/bin/zsh: no such option:  # _*_ perl _*_ _T

$ /usr/bin/zsh -- # -*- perl -*- -T
$

-- 
Matthias Andree

Re: #! incompatible -- binfmt_script.c broken?

[Horst von Brand]

Alex Riesen <Alexander.Riesen@synopsys.com> said:

[...]

> looks correct. The interpreter (/bin/sh) has got everything after
> its name. IOW: "-- # -*- perl -*- -T"
> It's just solaris' shell (/bin/sh) just ignores options starting with
> "--". And freebsd's as well.

And Linux's too. Try it.
--
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: #! incompatible -- binfmt_script.c broken?

[Horst von Brand]

Matthias Andree <matthias.andree@gmx.de> said:
> I tried some of the Perl magic tricks shown in the perlrun man page with
> Linux 2.4.19; consider this Perl one-liner. It works on FreeBSD and
> Solaris, but fails on Linux. Looking at binfmt_script.c, I believe the
> "pass the rest of the line as the first argument to the interpreter" is
> the problem with Linux. Haven't yet figured if the other boxes just use
> the interpreter, ignoring the arguments or if they are doing argument
> splitting.
> 
> ------------------------------------------------------------------------
> #!/bin/sh -- # -*- perl -*- -T
> eval 'exec perl -wTS $0 ${1+"$@"}'
>   if 0; 
> print "Hello there.\n";
> ------------------------------------------------------------------------
> 

[...]

> SuSE Linux 7.0, 7.3, 8.1 (2.4.19 kernel, binfmt_script.c identical to
> 2.4.20 BK):
> $ /tmp/try.pl
> /bin/sh: -- # -*- perl -*- -T: invalid option
> Usage:  /bin/sh [GNU long option] [option] ...
>         /bin/sh [GNU long option] [option] script-file ...
> [...]

RH 8.0 with linux-2.5.50 (some early bk version, last CSET is 1.849) gives
the same.
--
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: #! incompatible -- binfmt_script.c broken?

[H. Peter Anvin]

Followup to:  <E18JgHI-0006Gx-00@chiark.greenend.org.uk>
By author:    Matthew Garrett <mgarrett@chiark.greenend.org.uk>
In newsgroup: linux.dev.kernel
> 
> The *only* thing you can reliably use in #! lines is an interpreter
> followed by a single argument with no trailing space. On NetBSD with
> bash as /bin/sh:
> 
> mjg59@cysteine:/tmp$ cat foo.pl
> #!/bin/sh -- # -*- perl -*- -p
> mjg59@cysteine:/tmp$ ./foo.pl
> /bin/sh: -- # -*- perl -*- -p: unrecognized option
> 
> File a bug against perlrun(1).
> 

I personally think that it would be nice to split it by spaces (but
yes, ' " and \ need to be handled for this to work.)  It allows things
like using env to spawn a binary where the location of the interpreter
is to be extracted from PATH, for example.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: #! incompatible -- binfmt_script.c broken?

[Andries Brouwer]

On Wed, Dec 04, 2002 at 12:34:19PM +0100, Matthias Andree wrote:

> I tried some of the Perl magic tricks shown in the perlrun man page with
> Linux 2.4.19; consider this Perl one-liner. It works on FreeBSD and
> Solaris, but fails on Linux.
> 
> ------------------------------------------------------------------------
> #!/bin/sh -- # -*- perl -*- -T
> eval 'exec perl -wTS $0 ${1+"$@"}'
>   if 0; 
> print "Hello there.\n";
> ------------------------------------------------------------------------
> 
> FreeBSD 4.7 and Solaris 8:
> $ /tmp/try.pl
> Hello there.
> 
> SuSE Linux 7.0, 7.3, 8.1
> $ /tmp/try.pl
> /bin/sh: -- # -*- perl -*- -T: invalid option

See http://www.cwi.nl/~aeb/std/hashexclam-1.html

There I wrote long ago:

...
The interpreter is called with a parameter list consisting of four groups
of arguments: arg0, argi, argn, args.

The first group, arg0, consists of one argument. For SysVR4, SunOS, Solaris,
IRIX, BSDI, DU, AIX, Unixware, Linux this argument is /path/interpreter.
For FreeBSD and HPUX this argument is /scriptpath/script. 

The second group, argi, consists of the 0 or 1 or perhaps more arguments
to the interpreter found in the #! line. Thus, this group is empty if there
is no nonblank text following the interpreter name in the #! line.
If there is such nonblank text then for SysVR4, SunOS, Solaris, IRIX, HPUX,
AIX, Unixware, Linux, FreeBSD this group consists of precisely one argument,
as in the example above where argi consists of the single argument "-a -b".
BSDI splits the text following the interpreter name into zero or more arguments,
and hence has an argi consisting of the two arguments "-a", "-b" in the
above example. (Also Plan9 allows several arguments here.) 
...

Instead of trying this complicated perl example, you might try small scripts
that test one thing at a time. I am interested in the results of such tests
on various operating systems.

But regarding your post: there is no reason to assume that Linux does
something "wrong" here if it disagrees with Solaris and FreeBSD.
There is no standard and no uniformity here.

Andries

Re: #! incompatible -- binfmt_script.c broken?

[Matthew Garrett]

Matthias Andre wrote:
>On Wed, 04 Dec 2002, Alex Riesen wrote:
>> The interpreter (/bin/sh) has got everything after
>> its name. IOW: "-- # -*- perl -*- -T"
>
>Yes, as SINGLE argument. Therefore, Perl programs break if they use this
>procedure recommended by "man perlrun".
>
>I don't care WHY it works everywhere else, I want this incompatibility
>fixed and I'm not going through a flame war as with the 4.4BSD
>SIOCGIFNETMASK issue again. This is not negotiable.

See http://www.uni-ulm.de/~s_smasch/various/shebang/ . FreeBSD is the
*only* OS to pass multiple arguments. SUS says nothing about it, and
pretty much every single implementation varies. It does not work
everywhere else.

>We have enough braindead frivulous incompatibilities in Linux.

The *only* thing you can reliably use in #! lines is an interpreter
followed by a single argument with no trailing space. On NetBSD with
bash as /bin/sh:

mjg59@cysteine:/tmp$ cat foo.pl
#!/bin/sh -- # -*- perl -*- -p
mjg59@cysteine:/tmp$ ./foo.pl
/bin/sh: -- # -*- perl -*- -p: unrecognized option

File a bug against perlrun(1).
-- 
Matthew Garrett | mjg59-chiark.mail.linux-rutgers.kernel@srcf.ucam.org

Re: #! incompatible -- binfmt_script.c broken?

[Matthias Andree]

On Wed, 04 Dec 2002, Alex Riesen wrote:

> > SuSE Linux 7.0, 7.3, 8.1 (2.4.19 kernel, binfmt_script.c identical to
> > 2.4.20 BK):
> > $ /tmp/try.pl
> > /bin/sh: -- # -*- perl -*- -T: invalid option
> 
> looks correct.

Nope. It cannot be correct if it breaks compatibility without giving us
any advantage.

> The interpreter (/bin/sh) has got everything after
> its name. IOW: "-- # -*- perl -*- -T"

Yes, as SINGLE argument. Therefore, Perl programs break if they use this
procedure recommended by "man perlrun".

I don't care WHY it works everywhere else, I want this incompatibility
fixed and I'm not going through a flame war as with the 4.4BSD
SIOCGIFNETMASK issue again. This is not negotiable.

BTW, 2.2 is also affected.

Think of someone using /usr/bin/env -i /path/to/program -- won't work on
Linux, but works on FreeBSD.

I cannot see technical reasons why this should remain unfixed.

We have enough braindead frivulous incompatibilities in Linux.

> It's just solaris' shell (/bin/sh) just ignores options starting with
> "--". And freebsd's as well.
> Anyway - it's bash, not the bin_fmt.

Nope, zsh as /bin/sh complains as well:

/bin/sh: no such option:  # _*_ perl _*_ _T

so does pdksh:

/bin/sh: /bin/sh: --: unknown option

Re: #! incompatible -- binfmt_script.c broken?

[Matthew Garrett]

Alexander Riesen wrote:
>On Wed, Dec 04, 2002 at 12:34:19PM +0100, Matthias Andree wrote:
>> SuSE Linux 7.0, 7.3, 8.1 (2.4.19 kernel, binfmt_script.c identical to
>> 2.4.20 BK):
>> $ /tmp/try.pl
>> /bin/sh: -- # -*- perl -*- -T: invalid option
>
>looks correct. The interpreter (/bin/sh) has got everything after
>its name. IOW: "-- # -*- perl -*- -T"
>It's just solaris' shell (/bin/sh) just ignores options starting with
>"--". And freebsd's as well.

FreeBSD splits #! magic strings on whitespace and passes multiple
arguments. Linux passes everything after the first whitespace as a
single argument but strips trailing whitespace. NetBSD does the same as
Linux but passes trailing whitespace as part of the argument.

>Anyway - it's bash, not the bin_fmt.

Bash is (correctly) complaining that it's been passed an invalid
argument, but the reason for the different behaviour between it and
FreeBSD is because of binfmt_script. There's no clearly defined standard
for how this should behave.

-- 
Matthew Garrett | mjg59-chiark.mail.linux-rutgers.kernel@srcf.ucam.org

Re: #! incompatible -- binfmt_script.c broken?

[Alex Riesen]

On Wed, Dec 04, 2002 at 12:34:19PM +0100, Matthias Andree wrote:
> Hi,
> 
> I tried some of the Perl magic tricks shown in the perlrun man page with
> Linux 2.4.19; consider this Perl one-liner. It works on FreeBSD and
> Solaris, but fails on Linux. Looking at binfmt_script.c, I believe the
> "pass the rest of the line as the first argument to the interpreter" is
> the problem with Linux. Haven't yet figured if the other boxes just use
> the interpreter, ignoring the arguments or if they are doing argument
> splitting.
> 
> ------------------------------------------------------------------------
> #!/bin/sh -- # -*- perl -*- -T
> eval 'exec perl -wTS $0 ${1+"$@"}'
>   if 0; 
> print "Hello there.\n";
> ------------------------------------------------------------------------
> 
> FreeBSD 4.7:
> $ /tmp/try.pl
> Hello there.
> 
> Solaris 8:
> $ /tmp/try.pl
> Hello there.
> 
> SuSE Linux 7.0, 7.3, 8.1 (2.4.19 kernel, binfmt_script.c identical to
> 2.4.20 BK):
> $ /tmp/try.pl
> /bin/sh: -- # -*- perl -*- -T: invalid option

looks correct. The interpreter (/bin/sh) has got everything after
its name. IOW: "-- # -*- perl -*- -T"
It's just solaris' shell (/bin/sh) just ignores options starting with
"--". And freebsd's as well.
Anyway - it's bash, not the bin_fmt.

> Usage:  /bin/sh [GNU long option] [option] ...
>         /bin/sh [GNU long option] [option] script-file ...
> [...]

#! incompatible -- binfmt_script.c broken?

[Matthias Andree]

Hi,

I tried some of the Perl magic tricks shown in the perlrun man page with
Linux 2.4.19; consider this Perl one-liner. It works on FreeBSD and
Solaris, but fails on Linux. Looking at binfmt_script.c, I believe the
"pass the rest of the line as the first argument to the interpreter" is
the problem with Linux. Haven't yet figured if the other boxes just use
the interpreter, ignoring the arguments or if they are doing argument
splitting.

------------------------------------------------------------------------
#!/bin/sh -- # -*- perl -*- -T
eval 'exec perl -wTS $0 ${1+"$@"}'
  if 0; 
print "Hello there.\n";
------------------------------------------------------------------------

FreeBSD 4.7:
$ /tmp/try.pl
Hello there.

Solaris 8:
$ /tmp/try.pl
Hello there.

SuSE Linux 7.0, 7.3, 8.1 (2.4.19 kernel, binfmt_script.c identical to
2.4.20 BK):
$ /tmp/try.pl
/bin/sh: -- # -*- perl -*- -T: invalid option
Usage:  /bin/sh [GNU long option] [option] ...
        /bin/sh [GNU long option] [option] script-file ...
[...]

-- 
Matthias Andree