Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Petr Vandrovec]

On 31 Jan 02 at 13:47, Jeff Garzik wrote:
> On Thu, Jan 31, 2002 at 02:24:46PM +0100, Petr Vandrovec wrote:
> >     I've got strange idea and tried to build diskless machine around
> > 2.5.3... Besides problem with segfaulting crc32 (it is initialized after 
> > net/ipv4/ipconfig.c due to lib/lib.a being a library... I had to hardcode
> > lib/crc32.o before --start-group in main Makefile, but it is another
> > story)
> 
> Would you be willing to cook up a patch for this problem?
> 
> I ran into this too.  It was solved by setting CONFIG_CRC32=n and
> letting the Makefile rules pull it in...  but lib/lib.a needs to be
> lib/lib.o really.

Unfortunately during conversion I found that there is lib/bust_spinlocks.c,
which is always included in lib.a, is always compiled, even if architecture
provides its own bust_spinlocks function.

As no other module in lib/ uses module_init() initalization, it looks
to me like that we should move crc32.c from lib/ to kernel/, instead of
turning lib.a into lib.o.

But of course if there is consensus that I should convert lib/lib.a
into lib/lib.o, I can either create Config.in symbol 
CONFIG_NEED_GENERIC_BUST_SPINLOCK, or add HAVE_ARCH_BUST_SPINLOCK #define
into some of i386, ia64, mips64, s390 and s390x architecture dependent
headers.
                                        Thanks,
                                            Petr Vandrovec
                                            vandrove@vc.cvut.cz
                                            

Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Jeff Garzik]

On Thu, Jan 31, 2002 at 09:27:35PM +0100, Petr Vandrovec wrote:
> On 31 Jan 02 at 13:47, Jeff Garzik wrote:
> > On Thu, Jan 31, 2002 at 02:24:46PM +0100, Petr Vandrovec wrote:
> > >     I've got strange idea and tried to build diskless machine around
> > > 2.5.3... Besides problem with segfaulting crc32 (it is initialized after 
> > > net/ipv4/ipconfig.c due to lib/lib.a being a library... I had to hardcode
> > > lib/crc32.o before --start-group in main Makefile, but it is another
> > > story)
> > 
> > Would you be willing to cook up a patch for this problem?
> > 
> > I ran into this too.  It was solved by setting CONFIG_CRC32=n and
> > letting the Makefile rules pull it in...  but lib/lib.a needs to be
> > lib/lib.o really.
> 
> Unfortunately during conversion I found that there is lib/bust_spinlocks.c,
> which is always included in lib.a, is always compiled, even if architecture
> provides its own bust_spinlocks function.

Yep


> As no other module in lib/ uses module_init() initalization, it looks
> to me like that we should move crc32.c from lib/ to kernel/, instead of
> turning lib.a into lib.o.

Having lib.a is really a special case, and we can easily fix that by
fixing bust_spinlocks, not by moving what is truly a library routine to
somewhere other than lib/


> But of course if there is consensus that I should convert lib/lib.a
> into lib/lib.o, I can either create Config.in symbol 
> CONFIG_NEED_GENERIC_BUST_SPINLOCK, or add HAVE_ARCH_BUST_SPINLOCK #define
> into some of i386, ia64, mips64, s390 and s390x architecture dependent
> headers.

Implementing HAVE_ARCH_BUST_SPINLOCK would follow kernel convention
quite nicely...

	Jeff

[PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Petr Vandrovec]

On Thu, Jan 31, 2002 at 03:31:15PM -0500, Jeff Garzik wrote:
> > Unfortunately during conversion I found that there is lib/bust_spinlocks.c,
> > which is always included in lib.a, is always compiled, even if architecture
> > provides its own bust_spinlocks function.
> 
> Yep

> > But of course if there is consensus that I should convert lib/lib.a
> > into lib/lib.o, I can either create Config.in symbol 
> > CONFIG_NEED_GENERIC_BUST_SPINLOCK, or add HAVE_ARCH_BUST_SPINLOCK #define
> > into some of i386, ia64, mips64, s390 and s390x architecture dependent
> > headers.
> 
> Implementing HAVE_ARCH_BUST_SPINLOCK would follow kernel convention
> quite nicely...

Hi Linus,
   included patch (for 2.5.3) fixes problems with late initialization
of lib/crc32.o - as it was part of .a library file, it was picked by
link process AFTER at least one .o file required it - for example
when ipv4 autoconfiguration file needed it. So crc32's initalization
function was invoked after ipconfig's one - and it crashed inside
of ipconfig due to this problem.

   Jeff Garzik advised me that I should convert lib.a to lib.o instead
of moving crc32 somewhere else, so I did it. I decided to put 
HAVE_ARCH_BUST_SPINLOCKS macro to asm/system.h headers - if you disagree
with this location, tell me.

   Couple of architectures adds $(TOPDIR)/lib/lib.o into its LIBS
instead of reusing already set $(LIBS) variable. I was not sure
what PPC people meant with this, so I left it as is.

   I'd like to ask architecture maintainers (especially ia64, PPC, Sparc 
and Sparc64 maintainers) for verification that my changes did not break 
anything... I tested only i386, but changes were obvious...

				Best regards,
					Petr Vandrovec
					vandrove@vc.cvut.cz


diff -urdN linux/Makefile linux/Makefile
--- linux/Makefile	Wed Jan 30 02:24:01 2002
+++ linux/Makefile	Thu Jan 31 19:36:31 2002
@@ -121,7 +121,7 @@
 CORE_FILES	=kernel/kernel.o mm/mm.o fs/fs.o ipc/ipc.o
 NETWORKS	=net/network.o
 
-LIBS		=$(TOPDIR)/lib/lib.a
+LIBS		=$(TOPDIR)/lib/lib.o
 SUBDIRS		=kernel lib drivers mm fs net ipc
 
 DRIVERS-n :=
diff -urdN linux/arch/ia64/boot/Makefile linux/arch/ia64/boot/Makefile
--- linux/arch/ia64/boot/Makefile	Thu Jan  4 20:50:17 2001
+++ linux/arch/ia64/boot/Makefile	Thu Jan 31 19:31:04 2002
@@ -23,7 +23,7 @@
 all:	$(targets-y)
 
 bootloader: $(OBJECTS)
-	$(LD) $(LINKFLAGS) $(OBJECTS) $(TOPDIR)/lib/lib.a $(TOPDIR)/arch/$(ARCH)/lib/lib.a \
+	$(LD) $(LINKFLAGS) $(OBJECTS) $(TOPDIR)/lib/lib.o $(TOPDIR)/arch/$(ARCH)/lib/lib.a \
 	-o bootloader
 
 clean:
diff -urdN linux/arch/ppc/boot/chrp/Makefile linux/arch/ppc/boot/chrp/Makefile
--- linux/arch/ppc/boot/chrp/Makefile	Tue Aug 28 13:58:33 2001
+++ linux/arch/ppc/boot/chrp/Makefile	Thu Jan 31 19:29:34 2002
@@ -23,7 +23,7 @@
 
 OBJS = ../common/crt0.o start.o main.o misc.o ../common/string.o image.o \
 	../common/ofcommon.o
-LIBS = $(TOPDIR)/lib/lib.a ../lib/zlib.a
+LIBS = $(TOPDIR)/lib/lib.o ../lib/zlib.a
 ADDNOTE = ../utils/addnote
 PIGGYBACK = ../utils/piggyback
 
diff -urdN linux/arch/ppc/boot/pmac/Makefile linux/arch/ppc/boot/pmac/Makefile
--- linux/arch/ppc/boot/pmac/Makefile	Tue Aug 28 13:58:33 2001
+++ linux/arch/ppc/boot/pmac/Makefile	Thu Jan 31 19:29:14 2002
@@ -15,7 +15,7 @@
 COMMONOBJS = start.o misc.o ../common/string.o image.o ../common/ofcommon.o
 COFFOBJS = ../common/coffcrt0.o $(COMMONOBJS) coffmain.o
 CHRPOBJS = ../common/crt0.o $(COMMONOBJS) chrpmain.o
-LIBS = $(TOPDIR)/lib/lib.a ../lib/zlib.a
+LIBS = $(TOPDIR)/lib/lib.o ../lib/zlib.a
 
 MKNOTE := ../utils/mknote
 SIZE := ../utils/size
diff -urdN linux/arch/sparc/Makefile linux/arch/sparc/Makefile
--- linux/arch/sparc/Makefile	Sat Jul 28 19:12:37 2001
+++ linux/arch/sparc/Makefile	Thu Jan 31 19:26:07 2002
@@ -41,7 +41,7 @@
 CORE_FILES := arch/sparc/kernel/kernel.o arch/sparc/mm/mm.o $(CORE_FILES) \
 	arch/sparc/math-emu/math-emu.o
 
-LIBS := $(TOPDIR)/lib/lib.a $(LIBS) $(TOPDIR)/arch/sparc/prom/promlib.a \
+LIBS := $(LIBS) $(TOPDIR)/arch/sparc/prom/promlib.a \
 	$(TOPDIR)/arch/sparc/lib/lib.a
 
 # This one has to come last
diff -urdN linux/arch/sparc64/Makefile linux/arch/sparc64/Makefile
--- linux/arch/sparc64/Makefile	Mon Jan 14 18:10:44 2002
+++ linux/arch/sparc64/Makefile	Thu Jan 31 19:30:09 2002
@@ -69,7 +69,7 @@
 
 CORE_FILES += arch/sparc64/math-emu/math-emu.o
 
-LIBS := $(TOPDIR)/lib/lib.a $(LIBS) $(TOPDIR)/arch/sparc64/prom/promlib.a \
+LIBS := $(LIBS) $(TOPDIR)/arch/sparc64/prom/promlib.a \
 	$(TOPDIR)/arch/sparc64/lib/lib.a
 
 vmlinux.aout: vmlinux
diff -urdN linux/include/asm-i386/system.h linux/include/asm-i386/system.h
--- linux/include/asm-i386/system.h	Wed Jan 30 05:41:09 2002
+++ linux/include/asm-i386/system.h	Thu Jan 31 20:38:16 2002
@@ -123,6 +123,8 @@
 	__asm__("movl %0,%%cr4": :"r" (x));
 #define stts() write_cr0(8 | read_cr0())
 
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif	/* __KERNEL__ */
 
 #define wbinvd() \
diff -urdN linux/include/asm-ia64/system.h linux/include/asm-ia64/system.h
--- linux/include/asm-ia64/system.h	Fri Nov  9 22:26:17 2001
+++ linux/include/asm-ia64/system.h	Thu Jan 31 20:38:55 2002
@@ -412,6 +412,8 @@
 } while (0)
 #endif
 
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif /* __KERNEL__ */
 
 #endif /* __ASSEMBLY__ */
diff -urdN linux/include/asm-mips64/system.h linux/include/asm-mips64/system.h
--- linux/include/asm-mips64/system.h	Wed Jul  4 18:50:39 2001
+++ linux/include/asm-mips64/system.h	Thu Jan 31 20:39:49 2002
@@ -256,4 +256,6 @@
 
 extern void set_except_vector(int n, void *addr);
 
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif /* _ASM_SYSTEM_H */
diff -urdN linux/include/asm-s390/system.h linux/include/asm-s390/system.h
--- linux/include/asm-s390/system.h	Wed Jul 25 21:12:02 2001
+++ linux/include/asm-s390/system.h	Thu Jan 31 20:40:24 2002
@@ -250,6 +250,9 @@
 extern void save_fp_regs(s390_fp_regs *fpregs);
 extern int restore_fp_regs1(s390_fp_regs *fpregs);
 extern void restore_fp_regs(s390_fp_regs *fpregs);
+
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif
 
 #endif
diff -urdN linux/include/asm-s390x/system.h linux/include/asm-s390x/system.h
--- linux/include/asm-s390x/system.h	Wed Jul 25 21:12:03 2001
+++ linux/include/asm-s390x/system.h	Thu Jan 31 20:40:40 2002
@@ -260,6 +260,9 @@
 extern void save_fp_regs(s390_fp_regs *fpregs);
 extern int restore_fp_regs1(s390_fp_regs *fpregs);
 extern void restore_fp_regs(s390_fp_regs *fpregs);
+
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif
 
 #endif
diff -urdN linux/lib/Makefile linux/lib/Makefile
--- linux/lib/Makefile	Wed Jan 30 05:30:41 2002
+++ linux/lib/Makefile	Thu Jan 31 20:28:16 2002
@@ -6,7 +6,7 @@
 # unless it's something special (ie not a .c file).
 #
 
-L_TARGET := lib.a
+O_TARGET := lib.o
 
 export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o crc32.o
 
diff -urdN linux/lib/bust_spinlocks.c linux/lib/bust_spinlocks.c
--- linux/lib/bust_spinlocks.c	Mon Sep 17 04:22:40 2001
+++ linux/lib/bust_spinlocks.c	Thu Jan 31 20:43:10 2002
@@ -14,6 +14,8 @@
 #include <linux/wait.h>
 #include <linux/vt_kern.h>
 
+#ifndef HAVE_ARCH_BUST_SPINLOCKS
+
 extern spinlock_t timerlist_lock;
 
 void bust_spinlocks(int yes)
@@ -38,4 +40,4 @@
 	}
 }
 
-
+#endif /* HAVE_ARCH_BUST_SPINLOCKS */

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[David S. Miller]

   From: Petr Vandrovec <vandrove@vc.cvut.cz>
   Date: Thu, 31 Jan 2002 23:53:06 +0100
   
      I'd like to ask architecture maintainers (especially ia64, PPC, Sparc 
   and Sparc64 maintainers) for verification that my changes did not break 
   anything... I tested only i386, but changes were obvious...

I'll test this out up after Linus releases a pre-patch with this
included.

As a side note, this thing is so tiny (less than 4K on sparc64!) so
why don't we just include it unconditionally instead of having all
of this "turn it on for these drivers" stuff?

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Jeff Garzik]

On Thu, Jan 31, 2002 at 02:59:04PM -0800, David S. Miller wrote:
> As a side note, this thing is so tiny (less than 4K on sparc64!) so
> why don't we just include it unconditionally instead of having all
> of this "turn it on for these drivers" stuff?

Does that 4K include the BE and LE crc tables?

<shrug>  I don't mind much either way, except that I am general
resistant to "turn this on unconditionally" for bloat reasons.
[ie. its a reflex :)]

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Arnaldo Carvalho de Melo]

Em Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox escreveu:
> > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > why don't we just include it unconditionally instead of having all
> > of this "turn it on for these drivers" stuff?
> 
> Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
> to stuff Linux into embedded devices who if anything want more configuration
> options not people taking stuff out.
> 
> What I'd much rather see if this is an issue is:
> 
> bool	'Do you want to customise for a very small system' 
> 
> which auto enables all the random small stuff if you say no, and goes
> much deeper into options if you say yes.

heh, after I've read that you managed to boot 2.4 + rmap in a machine with
just 4 MB after tweaking some table sizes I thought about devoting some
time to identify those tables and making them options in make *config, with
even a nice CONFIG_TINY, like you said 8)

I'll eventually do this, and I'd appreciate if people send me suggestions
of tables/data structures that can be trimmed/reduced. Yeah, I'll take a
look at the .config files used in the embedded distros.

- Arnaldo

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

> As a side note, this thing is so tiny (less than 4K on sparc64!) so
> why don't we just include it unconditionally instead of having all
> of this "turn it on for these drivers" stuff?

Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
to stuff Linux into embedded devices who if anything want more configuration
options not people taking stuff out.

What I'd much rather see if this is an issue is:

bool	'Do you want to customise for a very small system' 

which auto enables all the random small stuff if you say no, and goes
much deeper into options if you say yes.

Alan

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[David Lang]

remember that CML2 claims to be able to detect that it needs to be on and
turn it on when needed

David Lang

On Thu, 31 Jan 2002, Jeff Garzik wrote:

> Date: Thu, 31 Jan 2002 18:08:42 -0500
> From: Jeff Garzik <garzik@havoc.gtf.org>
> To: David S. Miller <davem@redhat.com>
> Cc: vandrove@vc.cvut.cz, torvalds@transmeta.com, linux-kernel@vger.kernel.org,
>      paulus@samba.org, davidm@hpl.hp.com, ralf@gnu.org
> Subject: Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3
>     does not
>
> On Thu, Jan 31, 2002 at 02:59:04PM -0800, David S. Miller wrote:
> > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > why don't we just include it unconditionally instead of having all
> > of this "turn it on for these drivers" stuff?
>
> Does that 4K include the BE and LE crc tables?
>
> <shrug>  I don't mind much either way, except that I am general
> resistant to "turn this on unconditionally" for bloat reasons.
> [ie. its a reflex :)]
>
> 	Jeff
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox wrote:
> What I'd much rather see if this is an issue is:
> 
> bool	'Do you want to customise for a very small system' 
> 
> which auto enables all the random small stuff if you say no, and goes
> much deeper into options if you say yes.

CONFIG_SMALL or similar would indeed be nice...

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Alan Cox <alan@lxorguk.ukuu.org.uk>
   Date: Thu, 31 Jan 2002 23:24:10 +0000 (GMT)

   > As a side note, this thing is so tiny (less than 4K on sparc64!) so
   > why don't we just include it unconditionally instead of having all
   > of this "turn it on for these drivers" stuff?
   
   Because 100 4K drivers suddenly becomes 0.5Mb.

However this isn't a driver, the crc library stuff is more akin to
"strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
too?  I wish you luck building that kernel :-)

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

>    Because 100 4K drivers suddenly becomes 0.5Mb.
> 
> However this isn't a driver, the crc library stuff is more akin to
> "strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
> too?  I wish you luck building that kernel :-)

For a large number of systems you don't need the CRC library. There are no
systems where you don't need memcpy, so your comparison is stupid to say
the least.

"Gee making this link is hard" is not a good reason to simplify the
config file, its a good reason to simplify the make file system

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

acme@conectiva.com.br said:
>  heh, after I've read that you managed to boot 2.4 + rmap in a machine
> with just 4 MB after tweaking some table sizes I thought about
> devoting some time to identify those tables and making them options in
> make *config, with even a nice CONFIG_TINY, like you said 8)

> I'll eventually do this, and I'd appreciate if people send me
> suggestions of tables/data structures that can be trimmed/reduced.
> Yeah, I'll take a look at the .config files used in the embedded
> distros. 

CONFIG_BLK_DEV=n should allow you to trim a lot of cruft that most of my 
embedded boxen never use.


--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Horst von Brand]

"David S. Miller" <davem@redhat.com> said:

[...]

> However this isn't a driver, the crc library stuff is more akin to
> "strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
> too?  I wish you luck building that kernel :-)

libc.a was invented precisely to handle such stuff automatically when
linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
is a step in the wrong direction IMVHO.
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Fri, 01 Feb 2002 11:07:42 +0100, 
Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
>"David S. Miller" <davem@redhat.com> said:
>> However this isn't a driver, the crc library stuff is more akin to
>> "strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
>> too?  I wish you luck building that kernel :-)
>
>libc.a was invented precisely to handle such stuff automatically when
>linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
>is a step in the wrong direction IMVHO.

There are two contradictory requirements.  crc32.o must only be linked
if anything in the kernel needs it, linker puts crc32.o after the code
that uses it.  crc32.o must be linked before anything that uses it so
the crc32 __init code can initialize first.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Keith Owens <kaos@ocs.com.au>
   Date: Fri, 01 Feb 2002 21:28:52 +1100

   On Fri, 01 Feb 2002 11:07:42 +0100, 
   Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
   >libc.a was invented precisely to handle such stuff automatically when
   >linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
   >is a step in the wrong direction IMVHO.
   
   There are two contradictory requirements.  crc32.o must only be linked
   if anything in the kernel needs it, linker puts crc32.o after the code
   that uses it.  crc32.o must be linked before anything that uses it so
   the crc32 __init code can initialize first.
   
Horst isn't even talking about the initcall issues, he's talking about
how linking with libc works in that it only brings in the routines you
actually reference.

Will you get over the initcall thing already, you must be dreaming
about it at this point. I mean really, just GET OVER IT. :-)

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Fri, 01 Feb 2002 03:03:45 -0800 (PST), 
"David S. Miller" <davem@redhat.com> wrote:
>   From: Keith Owens <kaos@ocs.com.au>
>   Date: Fri, 01 Feb 2002 21:28:52 +1100
>
>   On Fri, 01 Feb 2002 11:07:42 +0100, 
>   Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
>   >libc.a was invented precisely to handle such stuff automatically when
>   >linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
>   >is a step in the wrong direction IMVHO.
>   
>   There are two contradictory requirements.  crc32.o must only be linked
>   if anything in the kernel needs it, linker puts crc32.o after the code
>   that uses it.  crc32.o must be linked before anything that uses it so
>   the crc32 __init code can initialize first.
>   
>Horst isn't even talking about the initcall issues, he's talking about
>how linking with libc works in that it only brings in the routines you
>actually reference.

Go back and read the entire thread Dave.  It started with

http://www.lib.uaa.alaska.edu/linux-kernel/archive/2002-Week-04/1235.html

  "Besides problem with segfaulting crc32 (it is initialized after
  net/ipv4/ipconfig.c due to lib/lib.a being a library... I had to
  hardcode lib/crc32.o before --start-group in main Makefile, but it is
  another story)"

http://www.lib.uaa.alaska.edu/linux-kernel/archive/2002-Week-04/1384.html

  "included patch (for 2.5.3) fixes problems with late initialization
  of lib/crc32.o - as it was part of .a library file, it was picked by
  link process AFTER at least one .o file required it - for example
  when ipv4 autoconfiguration file needed it. So crc32's initalization
  function was invoked after ipconfig's one - and it crashed inside of
  ipconfig due to this problem."

Horst was not talking about initcall but his suggestion of using lib.a
for crc32.o had already been proven not to work, because of the
(surprise) initcall problem.  I was merely pointing out to Horst why
lib.a was not working and why other options had to be considered.

>Will you get over the initcall thing already, you must be dreaming
>about it at this point. I mean really, just GET OVER IT. :-)

Nightmare is more like it.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Fri, Feb 01, 2002 at 10:25:20PM +1100, Keith Owens wrote:
> On Fri, 01 Feb 2002 03:03:45 -0800 (PST), 
> "David S. Miller" <davem@redhat.com> wrote:
> >Will you get over the initcall thing already, you must be dreaming
> >about it at this point. I mean really, just GET OVER IT. :-)
> 
> Nightmare is more like it.

Exaggeration is more like it :)

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox wrote:

    Because 100 4K drivers suddenly becomes 0.5Mb. There are those of
    us trying to stuff Linux into embedded devices who if anything
    want more configuration options not people taking stuff out.

Well, I'm more or less in agreement here, especially when working with
small embedded devices which have a few (say 16 or 32) MB of RAM for
EVERYTHING, kernel, userspace AND filesystems.

However, I wonder if we can't have the linker remove unnecessary and
unreferences objects, functions and variables?

    What I'd much rather see if this is an issue is:

    bool	'Do you want to customise for a very small system'

_IF_ the linker can remove things, it would simplify this too --- we
could if a few important places produce code slightly differently to
favour speed over size and not reference various things.  Also, the
above option would turn-off inlining as that seems to makie quite a
difference at times (BTW, I'm not sure about this, but it seems gcc
and C99 don't agree with static/extern inline semantics?)



  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Andrew Morton]

Chris Wedgwood wrote:
> 
> On Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox wrote:
> 
>     Because 100 4K drivers suddenly becomes 0.5Mb. There are those of
>     us trying to stuff Linux into embedded devices who if anything
>     want more configuration options not people taking stuff out.
> 
> Well, I'm more or less in agreement here, especially when working with
> small embedded devices which have a few (say 16 or 32) MB of RAM for
> EVERYTHING, kernel, userspace AND filesystems.
> 
> However, I wonder if we can't have the linker remove unnecessary and
> unreferences objects, functions and variables?

We can.  Graham Stoney had all this going against 2.2.  See

http://www.google.com/search?q=stoney+ffunction-sections&hl=en&start=10&sa=N
http://www.cs.helsinki.fi/linux/linux-kernel/Year-2000/2000-29/0415.html

>     What I'd much rather see if this is an issue is:
> 
>     bool        'Do you want to customise for a very small system'
> 
> _IF_ the linker can remove things, it would simplify this too --- we
> could if a few important places produce code slightly differently to
> favour speed over size and not reference various things.  Also, the
> above option would turn-off inlining as that seems to makie quite a
> difference at times (BTW, I'm not sure about this, but it seems gcc
> and C99 don't agree with static/extern inline semantics?)

The kernel doesn't link when you compile with -fno-inline because of all
the `extern inline' qualifiers.  These need to be converted to `static
inline'.    Jim Houston has a script which does this.  See

	http://www.uwsg.iu.edu/hypermail/linux/kernel/0201.3/0888.html

It would be rather good if we could get that script run across the
tree - no-inline has its uses at times.

-

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Fri, Feb 01, 2002 at 07:01:56PM -0800, Andrew Morton wrote:

    We can.  Graham Stoney had all this going against 2.2.  See

    http://www.google.com/search?q=stoney+ffunction-sections&hl=en&start=10&sa=N
    http://www.cs.helsinki.fi/linux/linux-kernel/Year-2000/2000-29/0415.html

This puts every function it's own section.  That seems not only
cumbersome to me, but also a little complex.  That said,  it may be a
good way if run every now and then to detect when cruft starts to
accumulate for any given .config and allow people to tweak things for
smaller kernels.

Is there no way to write something like:

--snip-- foo.c --snip--
void
blem()
{
}

void
bar()
{
    blem();
    return 0;
}

int foo()
{
    return 1;
}

int main(...)
{
    return foo();
}
--snip-- foo.c --snip--

compile and link it, have the linker know main or some part of crt0 is
special, build a graph of the final ELF object, see that bar and blem
are not connected to 'main' and discard them?

I'm pretty sure (but not 100% certain) that oher compilers can do
this, maybe someone can test on other platforms?

A really smart linker (if given enough compiler help) could build a
directional graph and still remove this code even if blem called foo.


Perhaps I'm making something that's extremely complex sound simple,
but it doesn't seem to me that this should be that complex...  maybe
someone more farmiliar with binutils and/or gcc can comment and tell
me why I'm being a fool :)

    The kernel doesn't link when you compile with -fno-inline because
    of all the `extern inline' qualifiers.  These need to be converted
    to `static inline'.  Jim Houston has a script which does this.

Semantically, in gcc land, someone explain the difference between:

     inline

     extern inline

     static inline

please?


My tests seem to indicte:

  inline creates an non-inline functions.  Simple tests failed to have
  this function inlined at all

  static inline and extern inline functions can and will be inlined depending
  on optimization level

  extern inline doesn't product a non-inlined function (even if it is
  referenced) and hence barfs if you don't compile with optimisations
  for my simple test

Now, I wonder

  why 'inline' for me, never inlines?

  is there a way to force inlining of a function?

  are non-inlined functions ever called when optimizations are
  enabled?




Thanks,


  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Daniel Jacobowitz]

On Fri, Feb 01, 2002 at 11:30:20PM -0800, Chris Wedgwood wrote:
> Is there no way to write something like:
> 
> --snip-- foo.c --snip--
> void
> blem()
> {
> }
> 
> void
> bar()
> {
>     blem();
>     return 0;
> }
> 
> int foo()
> {
>     return 1;
> }
> 
> int main(...)
> {
>     return foo();
> }
> --snip-- foo.c --snip--
> 
> compile and link it, have the linker know main or some part of crt0 is
> special, build a graph of the final ELF object, see that bar and blem
> are not connected to 'main' and discard them?
> 
> I'm pretty sure (but not 100% certain) that oher compilers can do
> this, maybe someone can test on other platforms?
> 
> A really smart linker (if given enough compiler help) could build a
> directional graph and still remove this code even if blem called foo.

One piece of the necessary compiler help would be -ffunction-sections. 
If they are in the same section in the same object file, they simply
can not be removed safely.  Relocation information for calls to local
functions is not reliably available at link time.


-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Fri, Feb 01, 2002 at 11:30:20PM -0800, Chris Wedgwood wrote:
> A really smart linker (if given enough compiler help) could build a
> directional graph and still remove this code even if blem called foo.

Agreed and this has been my objection to the function-sections patch.
There is no need for it if you make the toolchain smarter.

For example I would love to see a !CONFIG_MODULES build rip out all the
code that was not actively referenced, such as EXPORT_SYMBOL functions
which are never called.

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

[-- Attachment #1: Type: text/plain, Size: 2323 bytes --]

cc: list trimmed.

On Fri, 1 Feb 2002 23:30:20 -0800, 
Chris Wedgwood <cw@f00f.org> wrote:
>compile and link it, have the linker know main or some part of crt0 is
>special, build a graph of the final ELF object, see that bar and blem
>are not connected to 'main' and discard them?
>
>I'm pretty sure (but not 100% certain) that oher compilers can do
>this, maybe someone can test on other platforms?
>
>A really smart linker (if given enough compiler help) could build a
>directional graph and still remove this code even if blem called foo.

AFAICT it has to be done in two stages.  Identify extern symbols that
are not referenced and convert them to static, then compile and let gcc
warn about unused statics and remove them.

The two stage approach is required because ld does not have enough
information to identify all function calls.  ld views an object as
opaque blobs of text and data with relocation entries that tie them
together.  When a function is called from the same source file, gcc/gas
tend to use PC relative branches for the inter-source call.  PC
relative local branches do not usually generate relocation entries.
Without a relocation entry, ld cannot detect the inter-source call.

There is also a problem with exported symbols.  To ld, EXPORT_SYMBOL
looks like a reference to the symbol, but the export entry is
irrelevant, what really matters is if any module refers to those
symbols.  ld cannot distinguish between used and unused exported
symbols.  To make matters worse, if you define foo as static and
EXPORT_SYMBOL(foo) in the same source then gcc thinks that the symbol
is used, because of the EXPORT_SYMBOL() reference.  Again what really
matters is if any module uses the exported symbol.

The script below runs the .o files for the kernel and identifies extern
symbols with no extern references, i.e. the first part of the problem.
It knows about exported symbols and modules and handles them correctly.
I have not run the script for years, the list of conglomerates and the
list starting with mod_use_count_ will need updating.

Beware that you get a lot of false positives because of the kernel
config mechanism.  A symbol may be listed as unused but changing the
config may cause it to be used.  This is especially true for exported
symbols.

<davem>What, no initc**l reference?</davem>

[-- Attachment #2: namespace.pl --]
[-- Type: text/plain , Size: 7714 bytes --]

#!/usr/bin/perl -w
#
#	namespace.pl.  Mon Jan 27 22:25:47 EST 1997
#
#	Perform a name space analysis on the linux kernel.
#
#	Copyright Keith Owens <kaos@ocs.com.au>.  GPL.
#
#	Invoke by changing directory to the top of the kernel source
#	tree then namespace.pl, no parameters.
#
#	Tuned for 2.1.x kernels with the new module handling, it will
#	work with 2.0 kernels as well.  Last change 2.1.81.
#
#	The source must be compiled/assembled first, the object files
#	are the primary input to this script.  Incomplete or missing
#	objects will result in a flawed analysis.
#
#	Even with complete objects, treat the result of the analysis
#	with caution.  Some external references are only used by
#	certain architectures, others with certain combinations of
#	configuration parameters.  Ideally the source should include
#	something like
#
#	#ifndef CONFIG_...
#	static
#	#endif
#	symbol_definition;
#
#	so the symbols are defined as static unless a particular
#	CONFIG_... requires it to be external.
#

require 5;	# at least perl 5
use strict;
use File::Find;

my $nm = "/usr/bin/nm -p";	# in case somebody moves nm

if ($#ARGV != -1) {
	print STDERR "usage: $0 takes no parameters\n";
	die("giving up\n");
}

my %nmdata = ();	# nm data for each object
my %def = ();		# all definitions for each name
my %ksymtab = ();	# names that appear in __ksymtab_
my %ref = ();		# $ref{$name} exists if there is a true external reference to $name
my %export = ();	# $export{$name} exists if there is an EXPORT_... of $name

&find(\&linux_objects, '.');	# find the objects and do_nm on them
list_multiply_defined();
resolve_external_references();
list_extra_externals();

exit(0);

sub linux_objects
{
	# Select objects, ignoring objects which are only created by
	# merging other objects.  Also ignore all of modules, scripts
	# and compressed.
	my $basename = $_;
	$_ = $File::Find::name;
	s:^\./::;
	if (/.*\.o$/ && ! (
	    m:/fs.o$: || m:/isofs.o$: || m:/nfs.o$:
	    || m:/xiafs.o$: || m:/umsdos.o$: || m:/hpfs.o$:
	    || m:/smbfs.o$: || m:/ncpfs.o$: || m:/ufs.o$:
	    || m:/affs.o$: || m:/romfs.o$: || m:/kernel.o$:
	    || m:/mm.o$: || m:/ipc.o$: || m:/ext.o$:
	    || m:/msdos.o$: || m:/proc/proc.o$: || m:/minix.o$:
	    || m:/ext2.o$: || m:/sysv.o$: || m:/fat.o$:
	    || m:/vfat.o$: || m:/unix.o$: || m:/802.o$:
	    || m:/appletalk.o$: || m:/ax25.o$: || m:/core.o$:
	    || m:/ethernet.o$: || m:/ipv4.o$: || m:/ipx.o$:
	    || m:/netrom.o$: || m:/ipv6.o$: || m:/x25.o$:
	    || m:/rose.o$: || m:/bridge.o$: || m:/lapb.o$:
	    || m:/sock_n_syms.o$: || m:/teles.o$: || m:/pcbit.o$:
	    || m:/isdn.o$: || m:/ftape.o$: || m:/scsi_mod.o$:
	    || m:/sd_mod.o$: || m:/sr_mod.o$: || m:/lowlevel.o$:
	    || m:/sound.o$: || m:/piggy.o$: || m:/bootsect.o$:
	    || m:/boot/setup.o$: || m:^modules/: || m:^scripts/:
	    || m:/compressed/:
	    || m:/autofs.o$: || m:/lockd/lockd.o$: || m:/nfsd.o$:
	    || m:/sunrpc.o$: || m:/scsi_n_syms.o$:
	    || m:/boot/bbootsect.o$: || m:/boot/bsetup.o$:
	    || m:/misc/parport.o$: || m:/nls/nls.o$:
	    || m:/debug/debug.o$: || m:/netlink/netlink.o$:
	    || m:/sched/sched.o$: || m:/sound/sb.o$: 
	    || m:/sound/soundcore.o$: || m:/pci/pci_syms.o$: 
	    || m:/devpts/devpts.o$: || m:/video/fbdev.o$: 
	  )
	) {
		do_nm($basename, $_);
	}
	$_ = $basename;		# File::Find expects $_ untouched (undocumented)
}

sub do_nm
{
	my ($basename, $fullname) = @_;
	my ($source, $type, $name);
	if (! -e $basename) {
		printf STDERR "$basename does not exist\n";
		return;
	}
	if ($fullname !~ /\.o$/) {
		printf STDERR "$fullname is not an object file\n";
		return;
	}
	$source = $basename;
	$source =~ s/\.o$//;
	if (! -e "$source.c" && ! -e "$source.S") {
		printf STDERR "No source file found for $fullname\n";
		return;
	}
	if (! open(NMDATA, "$nm $basename|")) {
		printf STDERR "$nm $fullname failed $!\n";
		return;
	}
	my @nmdata;
	while (<NMDATA>) {
		chop;
		($type, $name) = (split(/ +/, $_, 3))[1..2];
		# Expected types
		# B weak external reference to data that has been resolved
		# C global variable, uninitialised
		# D global variable, initialised
		# R global array, initialised
		# T global label/procedure
		# U external reference
		# W weak external reference to text that has been resolved
		# a assembler equate
		# b static variable, uninitialised
		# d static variable, initialised
		# r static array, initialised
		# t static label/procedures
		# ? undefined type, used a lot by modules
		if ($type !~ /^[BCDRTUWabdrt?]$/) {
			printf STDERR "nm output for $fullname contains unknown type '$_'\n";
		}
		elsif ($name =~ /\./) {
			# name with '.' is local static
		}
		else {
			$name =~ s/_R[a-f0-9]{8}$//;	# module versions adds this
			if ($type =~ /[BCDRTW]/ &&
				$name ne 'init_module' &&
				$name ne 'cleanup_module' &&
				$name ne 'Using_Versions' &&
				$name !~ /^Version_[0-9]+$/ &&
				$name ne 'kernel_version') {
				if (!exists($def{$name})) {
					$def{$name} = [];
				}
				push(@{$def{$name}}, $fullname);
			}
			push(@nmdata, "$type $name");
			if ($name =~ /^__ksymtab_/) {
				$name = substr($name, 10);
				if (!exists($ksymtab{$name})) {
					$ksymtab{$name} = [];
				}
				push(@{$ksymtab{$name}}, $fullname);
			}
		}
	}
	close(NMDATA);
	if ($#nmdata < 0) {
		printf "No nm data for $fullname\n";
		return;
	}
	$nmdata{$fullname} = \@nmdata;
}

sub list_multiply_defined
{
	my ($name, $module);
	foreach $name (keys(%def)) {
		if ($#{$def{$name}} > 0) {
			printf "$name is multiply defined in :-\n";
			foreach $module (@{$def{$name}}) {
				printf "\t$module\n";
			}
		}
	}
}

sub resolve_external_references
{
	my ($object, $type, $name, $i, $j, $kstrtab, $ksymtab, $export);
	printf "\n";
	foreach $object (keys(%nmdata)) {
		my $nmdata = $nmdata{$object};
		for ($i = 0; $i <= $#{$nmdata}; ++$i) {
			($type, $name) = split(' ', $nmdata->[$i], 2);
			if ($type eq "U") {
				if (exists($def{$name}) || exists($ksymtab{$name})) {
					# add the owning object to the nmdata
					$nmdata->[$i] = "$type $name $object";
					# only count as a reference if it is not EXPORT_...
					$kstrtab = "? __kstrtab_$name";
					$ksymtab = "? __ksymtab_$name";
					$export = 0;
					for ($j = 0; $j <= $#{$nmdata}; ++$j) {
						if ($nmdata->[$j] eq $kstrtab ||
						    $nmdata->[$j] eq $ksymtab) {
							$export = 1;
							last;
						}
					}
					if ($export) {
						$export{$name} = "";
					}
					else {
						$ref{$name} = ""
					}
				}
				elsif ($name ne "mod_use_count_" &&
				       $name ne "__this_module" &&
				       $name ne "_etext" &&
				       $name ne "_edata" &&
				       $name ne "_end" &&
				       $name ne "__start___ksymtab" &&
				       $name ne "__start___ex_table" &&
				       $name ne "__stop___ksymtab" &&
				       $name ne "__stop___ex_table" &&
				       $name ne "__stop___ex_table" &&
				       $name ne "__bss_start" &&
				       $name ne "_text" &&
				       $name ne "__init_begin" &&
				       $name ne "__init_end") {
					printf "Cannot resolve reference to $name from $object\n";
				}
			}
		}
	}
}

sub list_extra_externals
{
	my %noref = ();
	my ($name, @module, $module, $export);
	foreach $name (keys(%def)) {
		if (! exists($ref{$name})) {
			@module = @{$def{$name}};
			foreach $module (@module) {
				if (! exists($noref{$module})) {
					$noref{$module} = [];
				}
				push(@{$noref{$module}}, $name);
			}
		}
	}
	if (%noref) {
		printf "\nExternally defined symbols with no external references\n";
		foreach $module (sort(keys(%noref))) {
			printf "  $module\n";
			foreach (sort(@{$noref{$module}})) {
				if (exists($export{$_})) {
					$export = " (export only)";
				}
				else {
					$export = "";
				}
				printf "    $_$export\n";
			}
		}
	}
}

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Sat, Feb 02, 2002 at 02:42:36AM -0500, Daniel Jacobowitz wrote:
> One piece of the necessary compiler help would be -ffunction-sections. 
> If they are in the same section in the same object file, they simply
> can not be removed safely.

Such as the patch that was mentioned earlier in this thread :)


> Relocation information for calls to local
> functions is not reliably available at link time.

With a smarter toolchain it could be.

One will need a smarter toolchain in order to re-order functions
anyway, which is an area where benchmarks on other compilers are
showing benefits.  (ie. moving "cold" functions to the end of the
module, given profiling feedback)

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

kaos@ocs.com.au said:
>  There is also a problem with exported symbols.  To ld, EXPORT_SYMBOL
> looks like a reference to the symbol, 

Er, surely that's not a problem at all? This is desired behaviour?

>  but the export entry is irrelevant,  what really matters is if any module
> refers to those symbols.

Absolutely not.  If we mark a symbol EXPORT_SYMBOL, we want it exported. No 
questions asked.

The export entry _is_ relevant; furthermore it's the _only_ thing that's
relevant, and there is no way of knowing if there's a module that isn't in
the tree, or maybe even a module that _is_ in the tree but not compiled
today, that needs the symbol in question.

I sincerely hope it's just too early in the morning here and I'm 
misunderstanding you :)

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Sat, 02 Feb 2002 08:40:11 +0000, 
David Woodhouse <dwmw2@infradead.org> wrote:
>kaos@ocs.com.au said:
>>  There is also a problem with exported symbols.  To ld, EXPORT_SYMBOL
>> looks like a reference to the symbol, 
>
>Er, surely that's not a problem at all? This is desired behaviour?
>
>>  but the export entry is irrelevant,  what really matters is if any module
>> refers to those symbols.
>
>Absolutely not.  If we mark a symbol EXPORT_SYMBOL, we want it exported. No 
>questions asked.

Does anything still use this symbol or is it dead?  Quite valid question.

>The export entry _is_ relevant; furthermore it's the _only_ thing that's
>relevant, and there is no way of knowing if there's a module that isn't in
>the tree, or maybe even a module that _is_ in the tree but not compiled
>today, that needs the symbol in question.

I said that you get false positives because the code that uses the
symbol might not be compiled in your kernel.  The script flags symbols
that are extern or exported and are not used in the current kernel.
Somebody then has to go through the report and decide if those symbols
are required under other config settings or by third party modules,
assuming that they care about third party modules.

The script is NOT an automatic list of what can be cleaned up.  I doubt
that such a list can be generated in the face of the kernel config
system and third party modules.  It is only a starting point for
symbols that need reviewing.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

kaos@ocs.com.au said:
>  Does anything still use this symbol or is it dead?  Quite valid
> question. 

Indeed. 

>  The script is NOT an automatic list of what can be cleaned up.  I
> doubt that such a list can be generated in the face of the kernel
> config system and third party modules.  It is only a starting point
> for symbols that need reviewing.

OK, cool. As it came out of the discussion about -ffunction-sections and
automatically garbage-collecting unused functions during the build, I
thought you had other, less sane intentions. Sorry for the misunderstanding.

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jens Axboe]

> include/linux/blk.h
> 
>         Changed NR_REQUEST from 256 to 16.  This reduces the number of
>         requests that can be queued.  The size of the queue is reduced
>         from 16K to 1K.

You can do even more than this -- I dunno what I/O interface these
embedded system generally uses (the mtd stuff?). Provided you provide a
direct make_request_fn entry into these instead of using the queue, you
can basically cut all of ll_rw_blk.c and elevator.c. ll_rw_block,
submit_bh, and generic_make_request would be all that is left.

That should reclaim quite a lot of space. If there's any interest in
this (has it already been done?), I can help out getting it done.

-- 
Jens Axboe

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[arjan]

In article <20020202135749.B4934@suse.de> you wrote:
>> include/linux/blk.h
>> 
>>         Changed NR_REQUEST from 256 to 16.  This reduces the number of
>>         requests that can be queued.  The size of the queue is reduced
>>         from 16K to 1K.

> You can do even more than this -- I dunno what I/O interface these
> embedded system generally uses (the mtd stuff?)

the MTD stuff wants CONFIG_BLOCKLAYER ;)
flash + jffs2 doesn't need any blocklayer at all ;)

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jens Axboe]

On Sat, Feb 02 2002, arjan@fenrus.demon.nl wrote:
> In article <20020202135749.B4934@suse.de> you wrote:
> >> include/linux/blk.h
> >> 
> >>         Changed NR_REQUEST from 256 to 16.  This reduces the number of
> >>         requests that can be queued.  The size of the queue is reduced
> >>         from 16K to 1K.
> 
> > You can do even more than this -- I dunno what I/O interface these
> > embedded system generally uses (the mtd stuff?)
> 
> the MTD stuff wants CONFIG_BLOCKLAYER ;)
> flash + jffs2 doesn't need any blocklayer at all ;)

Right, that was exactly my point...

-- 
Jens Axboe

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Denis Vlasenko]

On 31 January 2002 21:21, Arnaldo Carvalho de Melo wrote:
> Em Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox escreveu:
> > What I'd much rather see if this is an issue is:
> >
> > bool	'Do you want to customise for a very small system'
> >
> > which auto enables all the random small stuff if you say no, and goes
> > much deeper into options if you say yes.
>
> heh, after I've read that you managed to boot 2.4 + rmap in a machine with
> just 4 MB after tweaking some table sizes I thought about devoting some
> time to identify those tables and making them options in make *config, with
> even a nice CONFIG_TINY, like you said 8)
>
> I'll eventually do this, and I'd appreciate if people send me suggestions
> of tables/data structures that can be trimmed/reduced. Yeah, I'll take a
> look at the .config files used in the embedded distros.

One of inhabitants of my "Don't delete!" mail folder:

Re: Limited RAM - how to save it?
 From: Lars Brinkhoff <lars.spam@nocrew.org>
 To: Jan-Benedict Glaw <jbglaw@microdata-pos.de>
 Cc: linux-kernel@vger.kernel.org
 
Jan-Benedict Glaw <jbglaw@microdata-pos.de> writes:
> I'm working on a 4MB linux system (for a customer) which has quite
> limited resources [...] If you've got further ideas on getting the
> kernel a bit smaller, would be nice to get a mail dropped...

I started a port of Linux 2.3.99 to a MIPS device which usually has 2M
flash and 4M RAM.

To reduce the size of the text and data sections, I sorted the output
of the "size" command and investigated the files with the largest
sections.

These are the memory-saving changes I arrived at.  They are not really
tested, so some of them may break some functionality.  Also, I don't
know whether they apply to the current kernels.

fs/dcache.c

        Changed HASH_BITS from 14 to 8.  This reduces the size of the
        cache from 128K to 2K.

fs/inode.c

        Changed HASH_BITS from 14 to 8.  This reduces the size of the
        cache from 128K to 2K.

include/linux/blk.h

        Changed NR_REQUEST from 256 to 16.  This reduces the number of
        requests that can be queued.  The size of the queue is reduced
        from 16K to 1K.

include/linux/major.h

        Changed MAX_BLKDEV and MAX_CHRDEV from 256 to 10 and 20,
        respectively.  This reduces the number of block and character
        devices and saves about 40K.

kernel/printk.c

        Changed LOG_BUF_LEN from 16384 bytes to 2048 bytes.

include/linux/tty.h

        Changed NR_PTYS and NR_LDISCS from 256 and 16, respectively,
        to 16 and 4, respectively.  Saved about 12K.

        Warning: this change may break the pty driver, in which case
        further modifications will have to be done to
        drivers/char/pty.c.

kernel/panic.c

        Changed a buffer from 1024 bytes to 200 bytes.

include/linux/sched.h

        Changed PIDHASH_SZ from 1024 to 16, which saves 1008 bytes.

include/linux/mmzone.h

        NR_GPFINDEX from 0x100 to 0x10.  Saves 4800 bytes, but I'm not
        sure it doesn't break anything.

net/Makefile, net/socket.c, net/nosocket.c

        Replacing socket.c with nosocket.c, a file containing dummy
        replacement functions for those in socket.c, saves about 24K.

        Warning: this disables the socket API entirely, but it is
        currently not used in the product.

mm/Makefile, mm/swapfile.c, mm/swap_state.c, mm/noswapfile.c, 
mm/noswap_state.c

        Replacing swapfile.c with noswapfile.c, and swap_state.c with
        noswap_state.c saves about 12K.  The no*.c files contains
        empty replacement functions.

        Warning: this disables swapping of anonymous memory, which
        isn't used in the product.  But note that demand paging of
        executables still works.

mm/Makefile, mm/mmap.c

        The functions in mmap.c could probably also be replaced by
        empty functions.  Estimated saving: 9K (not included in the
        grand total below).

*, CONFIG_MESSAGES

        Applying the CONFIG_MESSAGES patch and disabling all kernel
        messages saves about 80K.

        The CONFIG_MESSAGES patch was written by Graham Stoney
        <greyham@research.canon.com.au>.

With all of the above, and only this enabled in .config:
        CONFIG_EXPERIMENTAL=y
        CONFIG_CPU_R3000=y
        CONFIG_CPU_LITTLE_ENDIAN=y
        CONFIG_ELF_KERNEL=y
        CONFIG_BINFMT_ELF=y
        CONFIG_MODULES=y
        CONFIG_MODVERSIONS=y
        CONFIG_KMOD=y
        CONFIG_CROSSCOMPILE=y
, the kernel is down to about 550K.

Here is the output of "size vmlinux".  I think this is without the
CONFIG_MESSAGES patch (it was long since I worked with this).

text    data    bss     dec     hex     filename
572128  41964   15860   629952  99cc0   vmlinux

-- 
Lars Brinkhoff          http://lars.nocrew.org/     Linux, GCC, PDP-10
Brinkhoff Consulting    http://www.brinkhoff.se/    programming
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Daniel Jacobowitz]

On Sat, Feb 02, 2002 at 03:08:47AM -0500, Jeff Garzik wrote:
> On Sat, Feb 02, 2002 at 02:42:36AM -0500, Daniel Jacobowitz wrote:
> > One piece of the necessary compiler help would be -ffunction-sections. 
> > If they are in the same section in the same object file, they simply
> > can not be removed safely.
> 
> Such as the patch that was mentioned earlier in this thread :)

Yes, the patch that he was objecting to :)

> > Relocation information for calls to local
> > functions is not reliably available at link time.
> 
> With a smarter toolchain it could be.
> 
> One will need a smarter toolchain in order to re-order functions
> anyway, which is an area where benchmarks on other compilers are
> showing benefits.  (ie. moving "cold" functions to the end of the
> module, given profiling feedback)

No, the linker simply can not assume that this information is present. 
That's a limitation of ELF.  The right way to do this sort of
optimization is to compile every function into its own section and then
handle the (very small) performance cost of doing so by a relaxation
pass in the linker.  You can't safely remove any part of an input
section, no matter what relocation information you think you have.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > why don't we just include it unconditionally instead of having all
> > of this "turn it on for these drivers" stuff?
> 
> Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
> to stuff Linux into embedded devices who if anything want more configuration
> options not people taking stuff out.
> 
> What I'd much rather see if this is an issue is:
> 
> bool	'Do you want to customise for a very small system' 
> 
> which auto enables all the random small stuff if you say no, and goes
> much deeper into options if you say yes.

I mostly agree.  Except when I have looked at trying to get the kernel
(compiled size down) the biggest bloat was in the core.  Things like
having both a page and a block cache.

Getting code reuse in the core higher would cut down on kernel size a
lot.  But that isn't quick fix territory.

Eric

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Ralf Baechle]

On Sat, Feb 02, 2002 at 09:14:33PM -0700, Eric W. Biederman wrote:

> > > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > > why don't we just include it unconditionally instead of having all
> > > of this "turn it on for these drivers" stuff?
> > 
> > Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
> > to stuff Linux into embedded devices who if anything want more configuration
> > options not people taking stuff out.
> > 
> > What I'd much rather see if this is an issue is:
> > 
> > bool	'Do you want to customise for a very small system' 
> > 
> > which auto enables all the random small stuff if you say no, and goes
> > much deeper into options if you say yes.
> 
> I mostly agree.  Except when I have looked at trying to get the kernel
> (compiled size down) the biggest bloat was in the core.  Things like
> having both a page and a block cache.
> 
> Getting code reuse in the core higher would cut down on kernel size a
> lot.  But that isn't quick fix territory.

Is it really worth the effort?  During the past year the average size of
embedded systems that people want to use for seems to have increased
dramatically.  In case of the MIPS port the core activity is about to
move away from the 32-bit to 64-bit kernel.

  Ralf

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Sun, Feb 03, 2002 at 08:01:35AM +0100, Ralf Baechle wrote:

    Is it really worth the effort?  During the past year the average
    size of embedded systems that people want to use for seems to have
    increased dramatically.  In case of the MIPS port the core
    activity is about to move away from the 32-bit to 64-bit kernel.

For some hand-held devices (eg. iPAQ), we want as much memory free as
possible as the only filesystem available is ramfs... moving to 64-bit
would be unthinkable! :)


  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

axboe@suse.de said:
>  You can do even more than this -- I dunno what I/O interface these
> embedded system generally uses (the mtd stuff?). Provided you provide
> a direct make_request_fn entry into these instead of using the queue,
> you can basically cut all of ll_rw_blk.c and elevator.c. 

We don't even do that - look at jffs2_read_super(). The only reason it even 
_looks_ like we're using a block device is because it was the easiest way 
to arrange mounting. We only allow you to mount the 'mtdblock' device, 
which isn't actually used - all we do is use the minor number to look up 
the real underlying MTD device. The dependency on _any_ of the block code 
isn't real - I can fix it as soon as there's an incentive to do so (like 
CONFIG_BLK_DEV).

> ll_rw_block, submit_bh, and generic_make_request would be all that is left. 

Those can go too. Likewise page->buffers.

>  That should reclaim quite a lot of space. If there's any interest in
> this (has it already been done?), I can help out getting it done.

I had a go once. Long enough ago that it's probably not worth trying to find
it again. I'd suggest a boolean like CONFIG_BUFFER_CACHE which does the
really intrusive stuff like removing page->buffers, and CONFIG_BLK_DEV which
can be modular (if CONFIG_BUFFER_CACHE is on), and which controls
compilation of everything in drivers/block/ and fs/block_dev.c

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

cw@f00f.org said:
>  For some hand-held devices (eg. iPAQ), we want as much memory free as
> possible as the only filesystem available is ramfs...

You took the flash out of your iPAQ? How does it boot?

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Sun, Feb 03, 2002 at 12:16:00PM +0000, David Woodhouse wrote:

    cw@f00f.org said:

    You took the flash out of your iPAQ? How does it boot?

Obviously it still has flash... but much of the filesystem is ramfs
and thus ram is rather precious.

For now anyhow, I think I'll upgrade it to 128M shortly.



  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

cw@f00f.org said:
>  Obviously it still has flash... but much of the filesystem is ramfs
> and thus ram is rather precious.

Ah, I see. What are you doing with it? Mine hardly touches the ramfs.

sh-2.03# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mtdblock/2          32256     26908      5348  84% /
tmpfs                    15428       260     15168   2% /mnt/ramfs


--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

> Is it really worth the effort?  During the past year the average size of
> embedded systems that people want to use for seems to have increased
> dramatically.  In case of the MIPS port the core activity is about to
> move away from the 32-bit to 64-bit kernel.

Embedded system and fancy handheld toys are not really the same thing. One
subset of the market gets cheaper the other gets flashier and more high
powered.

Mips is now dead in the handheld market if windows ce drops it.