[PATCH] [kbuild] Possibility to sanely link against off-directory .so

[PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Petr Baudis]

  Hello,

  this patch (against 2.5.46) introduces two special variables which make it
actually possible to have .so as the only product of build in some directory
and to link something against .so being built in another directory. The
variable host-cshlib-extra makes it possible to explicitly mention shared
objects to be built and the variable $(<foo>-linkobjs) allows user to specify
additional objects to link <foo> against, while not creating any dependencies
of <foo> on the objects.

  The changes are minimal while dramatically extending possibilities for
messing with the shared objects and they should have no unwanted side-effects,
and it appears to actually work for me. Please apply.

 scripts/Makefile.build |    6 ++++++
 scripts/Makefile.lib   |    2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

  Kind regards,
                                Petr Baudis

diff -ru linux/scripts/Makefile.build linux+pasky/scripts/Makefile.build
--- linux/scripts/Makefile.build	Tue Nov  5 19:29:20 2002
+++ linux+pasky/scripts/Makefile.build	Wed Nov  6 19:39:52 2002
@@ -197,6 +197,11 @@
 # libkconfig.so as the executable conf.
 # Note: Shared libraries consisting of C++ files are not supported  
 #
+# host-progs     := mconf
+# mconf-objs     := mconf.o
+# mconf-linkobjs := ../lxdialog/liblxdialog.so
+# Will link mconf.o against ../lxdialog/liblxdialog.so, but will not attempt
+# to build ../lxdialog/liblxdialog.so nor will make mconf depend on it.
 
 # Create executable from a single .c file
 # host-csingle -> Executable
@@ -212,6 +217,7 @@
 quiet_cmd_host-cmulti	= HOSTLD  $@
       cmd_host-cmulti	= $(HOSTCC) $(HOSTLDFLAGS) -o $@ \
 			  $(addprefix $(obj)/,$($(@F)-objs)) \
+			  $(addprefix $(obj)/,$($(@F)-linkobjs)) \
 			  $(HOST_LOADLIBES) $(HOSTLOADLIBES_$(@F))
 $(host-cmulti): %: $(host-cobjs) $(host-cshlib) FORCE
 	$(call if_changed,host-cmulti)
diff -ru linux/scripts/Makefile.lib linux+pasky/scripts/Makefile.lib
--- linux/scripts/Makefile.lib	Fri Nov  1 22:22:07 2002
+++ linux+pasky/scripts/Makefile.lib	Wed Nov  6 19:07:47 2002
@@ -78,7 +78,7 @@
 
 # Shared libaries (only .c supported)
 # Shared libraries (.so) - all .so files referenced in "xxx-objs"
-host-cshlib	:= $(sort $(filter %.so, $(host-cobjs)))
+host-cshlib	:= $(host-cshlib-extra) $(sort $(filter %.so, $(host-cobjs)))
 # Remove .so files from "xxx-objs"
 host-cobjs	:= $(filter-out %.so,$(host-cobjs))
 

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Sam Ravnborg]

On Wed, Nov 06, 2002 at 07:52:30PM +0100, Petr Baudis wrote:
>   Hello,
> 
>   this patch (against 2.5.46) introduces two special variables which make it
> actually possible to have .so as the only product of build in some directory
> and to link something against .so being built in another directory. The
> variable host-cshlib-extra makes it possible to explicitly mention shared
> objects to be built and the variable $(<foo>-linkobjs) allows user to specify
> additional objects to link <foo> against, while not creating any dependencies
> of <foo> on the objects.
> 
>   The changes are minimal while dramatically extending possibilities for
> messing with the shared objects and they should have no unwanted side-effects,
> and it appears to actually work for me. Please apply.

There is only one user of shared libaries today, thats Kconfig.
And Kconfig is the only user of C++ as well.

There is quite a lot of added complexity in Makefile.lib + Makefile.build
only to support this. Being the one that introduced it, I would like to
see it go away again.
Rationale behind this is that the current added complexity has an penalty
when compiling a kernel, and I would like to move the complexity to
the only user.

Care to try that approach?

	Sam

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Petr Baudis]

Dear diary, on Wed, Nov 06, 2002 at 10:29:52PM CET, I got a letter,
where Sam Ravnborg <sam@ravnborg.org> told me, that...
> On Wed, Nov 06, 2002 at 07:52:30PM +0100, Petr Baudis wrote:
> >   Hello,
> > 
> >   this patch (against 2.5.46) introduces two special variables which make it
> > actually possible to have .so as the only product of build in some directory
> > and to link something against .so being built in another directory. The
> > variable host-cshlib-extra makes it possible to explicitly mention shared
> > objects to be built and the variable $(<foo>-linkobjs) allows user to specify
> > additional objects to link <foo> against, while not creating any dependencies
> > of <foo> on the objects.
> > 
> >   The changes are minimal while dramatically extending possibilities for
> > messing with the shared objects and they should have no unwanted side-effects,
> > and it appears to actually work for me. Please apply.
> 
> There is only one user of shared libaries today, thats Kconfig.
> And Kconfig is the only user of C++ as well.
> 
> There is quite a lot of added complexity in Makefile.lib + Makefile.build
> only to support this. Being the one that introduced it, I would like to
> see it go away again.
> Rationale behind this is that the current added complexity has an penalty
> when compiling a kernel, and I would like to move the complexity to
> the only user.
> 
> Care to try that approach?

Can't say anything about the C++ stuff, but the second user of shared libraries
is going to be lxdialog - hopefully this evening already, in my patches (it
already works, I'm only cleaning out few details now; lxdialog + mconf is also
user of both these extensions).

I don't think the complexity increase is so dramatical - theoretically, it
almost shouldn't affect the normal build, except one scan for .so extensions,
right? Maybe we could do with some less generic way here, like specifying .so
dependencies in a special variable? On the other side, moving .so processing to
the user entirely would already mean some amount of duplication now (given that
my lxdialog + mconf patch will be accepted ;-).

I personally think that the -linkobjs variable adds practically zero overhead,
while having potential to be generically useful in other places than lxdialog.
About host-cshlib-extra, if we aren't going to entirely remove .so processing,
I believe that it should go in as well, since eventual move of .so processing
to separate set of rules will probably mostly affect one step higher level of
rules / variables than this, and this variable is going to be useful in the
both cases.

Kind regards,

-- 
 
				Petr "Pasky" Baudis
.
This host is a black hole at HTTP wavelengths. GETs go in, and nothing
comes out, not even Hawking radiation.
                -- Graaagh the Mighty on rec.games.roguelike.angband
.
Public PGP key && geekcode && homepage: http://pasky.ji.cz/~pasky/

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Peter Samuelson]

[Petr Baudis]
> Can't say anything about the C++ stuff, but the second user of shared
> libraries is going to be lxdialog - hopefully this evening already,
> in my patches (it already works, I'm only cleaning out few details
> now; lxdialog + mconf is also user of both these extensions).

What is so all-fired great about shared libraries anyway?  It's not
like our burning need is to save memory when two people run 'make
menuconfig' in parallel.  What's wrong with 'ar cq libxxx.a $(OBJS)'
anyway?  It's fast, it's easy, it's portable, and you never have to
worry about things like LD_LIBRARY_PATH or `-rpath'.

Sure, with Linux you can create a shared library with 'gcc -shared' ...
but what about bootstrapping a Linux kernel from a legacy OS?  (Yes,
people do compile Linux on Solaris, for example.)  HOSTCC may or may
not be gcc, and if it is, it may or may not support creating shared
libraries, and if it does, you might need funky flags or variables to
denote the link-time or run-time search path.  Why bother?  `ar' is
basically universal, both in availability and usage.  (Well, *almost*
universal usage: if you want to run 'ranlib', be prepared for it not to
exist.)

Basically, what I'm saying is, I see no need for the existing .so in
the kernel build, much less another one.  Static libraries are ever so
much easier to manage.

Peter

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Roman Zippel]

Hi,

On Thu, 7 Nov 2002, Peter Samuelson wrote:

> Basically, what I'm saying is, I see no need for the existing .so in
> the kernel build, much less another one.  Static libraries are ever so
> much easier to manage.

If you want to limit people to the config tools in the kernel, there is 
indeed no need for a shared library. Note that during the next development 
cycle all graphical front ends are possibly removed.

bye, Roman

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Peter Samuelson]

  [Peter Samuelson]
> > Basically, what I'm saying is, I see no need for the existing .so in
> > the kernel build, much less another one.  Static libraries are ever so
> > much easier to manage.

[Roman Zippel]
> If you want to limit people to the config tools in the kernel, there
> is indeed no need for a shared library. Note that during the next
> development cycle all graphical front ends are possibly removed.

Huh?  I don't get it.  How is a shared library any better than a static
library in this regard?  I'm pondering the traditional advantages of
shared libraries, and I cannot think of a single one that matters here.

Peter

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Roman Zippel]

Hi,

On Thu, 7 Nov 2002, Peter Samuelson wrote:

> > If you want to limit people to the config tools in the kernel, there
> > is indeed no need for a shared library. Note that during the next
> > development cycle all graphical front ends are possibly removed.
> 
> Huh?  I don't get it.  How is a shared library any better than a static
> library in this regard?  I'm pondering the traditional advantages of
> shared libraries, and I cannot think of a single one that matters here.

Shared libraries can be loaded dynamically, this means distribution can 
package the graphical front ends and the user doesn't need to install 
huge development packages.

bye, Roman

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Peter Samuelson]

  [Peter Samuelson]
> > Huh?  I don't get it.  How is a shared library any better than a static
> > library in this regard?  I'm pondering the traditional advantages of
> > shared libraries, and I cannot think of a single one that matters here.

[Roman Zippel]
> Shared libraries can be loaded dynamically, this means distribution can 
> package the graphical front ends and the user doesn't need to install 
> huge development packages.

I still don't get it.  Why can't the distribution vendor just link
/usr/bin/qconf against $(LINUX)/scripts/kconfig/libkconfig.a?

Peter

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Roman Zippel]

Hi,

On Thu, 7 Nov 2002, Peter Samuelson wrote:

> > Shared libraries can be loaded dynamically, this means distribution can 
> > package the graphical front ends and the user doesn't need to install 
> > huge development packages.
> 
> I still don't get it.  Why can't the distribution vendor just link
> /usr/bin/qconf against $(LINUX)/scripts/kconfig/libkconfig.a?

New features will be added and only the parser that comes with the kernel 
will understand them. It's less likley that the library API will change.

bye, Roman

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Peter Samuelson]

[Roman Zippel]
> New features will be added and only the parser that comes with the kernel 
> will understand them. It's less likley that the library API will change.

Even if this is true - I'll grant you that it may be - let the vendor
package /usr/bin/qconf as a shell script that links /usr/lib/qconf.o
with -lqt and $(LINUX)/scripts/kconfig/libkconfig.a .  It's a little
unorthodox, but it removes the hackery of figuring out how HOSTCC is
supposed to build a shared library and whether any magic is needed at
runtime.  Removes the need for the horrible stuff libtool was invented
for, in other words.

Remember, the whole point of HOSTCC is to support a build environment
different from the compile target - arbitrarily different, even.

Peter

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Roman Zippel]

Hi,

On Thu, 7 Nov 2002, Peter Samuelson wrote:

> Even if this is true - I'll grant you that it may be - let the vendor
> package /usr/bin/qconf as a shell script that links /usr/lib/qconf.o
> with -lqt and $(LINUX)/scripts/kconfig/libkconfig.a .  It's a little
> unorthodox, but it removes the hackery of figuring out how HOSTCC is
> supposed to build a shared library and whether any magic is needed at
> runtime.  Removes the need for the horrible stuff libtool was invented
> for, in other words.
> 
> Remember, the whole point of HOSTCC is to support a build environment
> different from the compile target - arbitrarily different, even.

If your build environment doesn't support shared libraries, you can easily 
generate a static library instead and link against it yourself, like you 
described, but it's no reason to deny the convenience to working 
environments.

bye, Roman

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Peter Samuelson]

[Roman Zippel]
> If your build environment doesn't support shared libraries, you can
> easily generate a static library instead and link against it
> yourself, like you described, but it's no reason to deny the
> convenience to working environments.

Yeah, but until I do, I can't even run 'make oldconfig'.

(This isn't about me - I will probably always build on Linux, with gcc
- it's about weird environments like cross-compiling from Solaris,
which I'm told was often done in the earlier stages of SPARC Linux.)

I suggest that *at least* the built-in targets should link libkconfig
statically - either via libkconfig.a or just the list of .o files.

Peter

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Petr Baudis]

Dear diary, on Thu, Nov 07, 2002 at 02:22:45PM CET, I got a letter,
where Peter Samuelson <peter@cadcamlab.org> told me, that...
> Remember, the whole point of HOSTCC is to support a build environment
> different from the compile target - arbitrarily different, even.

I'm a bit lost here - the kernel uses tons of gcc extensions - how is another
compiler supposed to understand them? And if it is specifically extended to
understand them, isn't it likely that it'll understand the -shared switch in
gcc-like way as well?

Or better, what other compiler is known to build a kernel than gcc? At least
anything that doesn't define __GNUC__ should IMHO fail inside of init/main.c.
And how likely is situation when someone want to configure a kernel with
non-gcc compiler and actually build it with gcc?

I thought that the point of HOSTCC is to allow to use a non-standart version
of gcc for kernel build.

-- 
 
				Petr "Pasky" Baudis
.
This host is a black hole at HTTP wavelengths. GETs go in, and nothing
comes out, not even Hawking radiation.
                -- Graaagh the Mighty on rec.games.roguelike.angband
.
Public PGP key && geekcode && homepage: http://pasky.ji.cz/~pasky/

RE: [kbuild-devel] Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Kenn Humborg]

> Dear diary, on Thu, Nov 07, 2002 at 02:22:45PM CET, I got a letter,
> where Peter Samuelson <peter@cadcamlab.org> told me, that...
> > Remember, the whole point of HOSTCC is to support a build environment
> > different from the compile target - arbitrarily different, even.
> 
> I'm a bit lost here - the kernel uses tons of gcc extensions - 
> how is another
> compiler supposed to understand them? And if it is specifically 
> extended to
> understand them, isn't it likely that it'll understand the 
> -shared switch in
> gcc-like way as well?
> 
> Or better, what other compiler is known to build a kernel than 
> gcc? At least
> anything that doesn't define __GNUC__ should IMHO fail inside of 
> init/main.c.
> And how likely is situation when someone want to configure a kernel with
> non-gcc compiler and actually build it with gcc?

When you're cross-compiling a kernel.

> I thought that the point of HOSTCC is to allow to use a 
> non-standart version
> of gcc for kernel build.

Nope.  It's mainly for cross-compilation.  You want to compile the
kernel itself for your targer architecture, but the compilation tools
need to run on the build machine so need a different compiler.

The gcc/kgcc thing is only a convenient side effect.

Later,
Kenn

Re: [PATCH] [kbuild] Possibility to sanely link against off-directory .so

[Peter Samuelson]

[Petr Baudis]
> I'm a bit lost here - the kernel uses tons of gcc extensions - how is
> another compiler supposed to understand them? And if it is
> specifically extended to understand them, isn't it likely that it'll
> understand the -shared switch in gcc-like way as well?

There is a difference between $(CC) and $(HOSTCC).  We have both of
them for a reason.  $(CC) specifies the compiler used to build the
kernel - it may or may not be your regular system compiler.  It can
even be for another architecture - for example, it may produce PowerPC
object files from the comfort (?) of your x86 PC.

$(HOSTCC) is specifically to build executables that are to run *on* the
build system itself - kconfig, for example.  If you're running on x86
and building a PowerPC kernel, you need an x86 version of kconfig, not
a PowerPC version.

So $(CC) *must* be a gcc (or a convincing facsimile) - but $(HOSTCC)
can be *any* C compiler on your system.  I don't think anyone tries to
use any gcc extensions with $(HOSTCC).

> And how likely is situation when someone want to configure a kernel
> with non-gcc compiler and actually build it with gcc?

Not very likely, but you take a lot for granted.  On many combinations
of Unix + gcc versions, `gcc -shared' doesn't work until you spend half
a day hand-tweaking the specs file whilst poring over the obscure flags
section of the `ld' manpage.  Try it for gcc 2.95 on AIX sometime, for
a bit of good clean fun.

My point was and is that while shared libraries are a (relative!)
breeze on Linux, they have always been a horrible portability headache;
furthermore, the kernel build system doesn't need them and ought to do
without.

Peter