Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Petr Vandrovec]

On 31 Jan 02 at 13:47, Jeff Garzik wrote:
> On Thu, Jan 31, 2002 at 02:24:46PM +0100, Petr Vandrovec wrote:
> >     I've got strange idea and tried to build diskless machine around
> > 2.5.3... Besides problem with segfaulting crc32 (it is initialized after 
> > net/ipv4/ipconfig.c due to lib/lib.a being a library... I had to hardcode
> > lib/crc32.o before --start-group in main Makefile, but it is another
> > story)
> 
> Would you be willing to cook up a patch for this problem?
> 
> I ran into this too.  It was solved by setting CONFIG_CRC32=n and
> letting the Makefile rules pull it in...  but lib/lib.a needs to be
> lib/lib.o really.

Unfortunately during conversion I found that there is lib/bust_spinlocks.c,
which is always included in lib.a, is always compiled, even if architecture
provides its own bust_spinlocks function.

As no other module in lib/ uses module_init() initalization, it looks
to me like that we should move crc32.c from lib/ to kernel/, instead of
turning lib.a into lib.o.

But of course if there is consensus that I should convert lib/lib.a
into lib/lib.o, I can either create Config.in symbol 
CONFIG_NEED_GENERIC_BUST_SPINLOCK, or add HAVE_ARCH_BUST_SPINLOCK #define
into some of i386, ia64, mips64, s390 and s390x architecture dependent
headers.
                                        Thanks,
                                            Petr Vandrovec
                                            vandrove@vc.cvut.cz
                                            

Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Jeff Garzik]

On Thu, Jan 31, 2002 at 09:27:35PM +0100, Petr Vandrovec wrote:
> On 31 Jan 02 at 13:47, Jeff Garzik wrote:
> > On Thu, Jan 31, 2002 at 02:24:46PM +0100, Petr Vandrovec wrote:
> > >     I've got strange idea and tried to build diskless machine around
> > > 2.5.3... Besides problem with segfaulting crc32 (it is initialized after 
> > > net/ipv4/ipconfig.c due to lib/lib.a being a library... I had to hardcode
> > > lib/crc32.o before --start-group in main Makefile, but it is another
> > > story)
> > 
> > Would you be willing to cook up a patch for this problem?
> > 
> > I ran into this too.  It was solved by setting CONFIG_CRC32=n and
> > letting the Makefile rules pull it in...  but lib/lib.a needs to be
> > lib/lib.o really.
> 
> Unfortunately during conversion I found that there is lib/bust_spinlocks.c,
> which is always included in lib.a, is always compiled, even if architecture
> provides its own bust_spinlocks function.

Yep


> As no other module in lib/ uses module_init() initalization, it looks
> to me like that we should move crc32.c from lib/ to kernel/, instead of
> turning lib.a into lib.o.

Having lib.a is really a special case, and we can easily fix that by
fixing bust_spinlocks, not by moving what is truly a library routine to
somewhere other than lib/


> But of course if there is consensus that I should convert lib/lib.a
> into lib/lib.o, I can either create Config.in symbol 
> CONFIG_NEED_GENERIC_BUST_SPINLOCK, or add HAVE_ARCH_BUST_SPINLOCK #define
> into some of i386, ia64, mips64, s390 and s390x architecture dependent
> headers.

Implementing HAVE_ARCH_BUST_SPINLOCK would follow kernel convention
quite nicely...

	Jeff

[PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Petr Vandrovec]

On Thu, Jan 31, 2002 at 03:31:15PM -0500, Jeff Garzik wrote:
> > Unfortunately during conversion I found that there is lib/bust_spinlocks.c,
> > which is always included in lib.a, is always compiled, even if architecture
> > provides its own bust_spinlocks function.
> 
> Yep

> > But of course if there is consensus that I should convert lib/lib.a
> > into lib/lib.o, I can either create Config.in symbol 
> > CONFIG_NEED_GENERIC_BUST_SPINLOCK, or add HAVE_ARCH_BUST_SPINLOCK #define
> > into some of i386, ia64, mips64, s390 and s390x architecture dependent
> > headers.
> 
> Implementing HAVE_ARCH_BUST_SPINLOCK would follow kernel convention
> quite nicely...

Hi Linus,
   included patch (for 2.5.3) fixes problems with late initialization
of lib/crc32.o - as it was part of .a library file, it was picked by
link process AFTER at least one .o file required it - for example
when ipv4 autoconfiguration file needed it. So crc32's initalization
function was invoked after ipconfig's one - and it crashed inside
of ipconfig due to this problem.

   Jeff Garzik advised me that I should convert lib.a to lib.o instead
of moving crc32 somewhere else, so I did it. I decided to put 
HAVE_ARCH_BUST_SPINLOCKS macro to asm/system.h headers - if you disagree
with this location, tell me.

   Couple of architectures adds $(TOPDIR)/lib/lib.o into its LIBS
instead of reusing already set $(LIBS) variable. I was not sure
what PPC people meant with this, so I left it as is.

   I'd like to ask architecture maintainers (especially ia64, PPC, Sparc 
and Sparc64 maintainers) for verification that my changes did not break 
anything... I tested only i386, but changes were obvious...

				Best regards,
					Petr Vandrovec
					vandrove@vc.cvut.cz


diff -urdN linux/Makefile linux/Makefile
--- linux/Makefile	Wed Jan 30 02:24:01 2002
+++ linux/Makefile	Thu Jan 31 19:36:31 2002
@@ -121,7 +121,7 @@
 CORE_FILES	=kernel/kernel.o mm/mm.o fs/fs.o ipc/ipc.o
 NETWORKS	=net/network.o
 
-LIBS		=$(TOPDIR)/lib/lib.a
+LIBS		=$(TOPDIR)/lib/lib.o
 SUBDIRS		=kernel lib drivers mm fs net ipc
 
 DRIVERS-n :=
diff -urdN linux/arch/ia64/boot/Makefile linux/arch/ia64/boot/Makefile
--- linux/arch/ia64/boot/Makefile	Thu Jan  4 20:50:17 2001
+++ linux/arch/ia64/boot/Makefile	Thu Jan 31 19:31:04 2002
@@ -23,7 +23,7 @@
 all:	$(targets-y)
 
 bootloader: $(OBJECTS)
-	$(LD) $(LINKFLAGS) $(OBJECTS) $(TOPDIR)/lib/lib.a $(TOPDIR)/arch/$(ARCH)/lib/lib.a \
+	$(LD) $(LINKFLAGS) $(OBJECTS) $(TOPDIR)/lib/lib.o $(TOPDIR)/arch/$(ARCH)/lib/lib.a \
 	-o bootloader
 
 clean:
diff -urdN linux/arch/ppc/boot/chrp/Makefile linux/arch/ppc/boot/chrp/Makefile
--- linux/arch/ppc/boot/chrp/Makefile	Tue Aug 28 13:58:33 2001
+++ linux/arch/ppc/boot/chrp/Makefile	Thu Jan 31 19:29:34 2002
@@ -23,7 +23,7 @@
 
 OBJS = ../common/crt0.o start.o main.o misc.o ../common/string.o image.o \
 	../common/ofcommon.o
-LIBS = $(TOPDIR)/lib/lib.a ../lib/zlib.a
+LIBS = $(TOPDIR)/lib/lib.o ../lib/zlib.a
 ADDNOTE = ../utils/addnote
 PIGGYBACK = ../utils/piggyback
 
diff -urdN linux/arch/ppc/boot/pmac/Makefile linux/arch/ppc/boot/pmac/Makefile
--- linux/arch/ppc/boot/pmac/Makefile	Tue Aug 28 13:58:33 2001
+++ linux/arch/ppc/boot/pmac/Makefile	Thu Jan 31 19:29:14 2002
@@ -15,7 +15,7 @@
 COMMONOBJS = start.o misc.o ../common/string.o image.o ../common/ofcommon.o
 COFFOBJS = ../common/coffcrt0.o $(COMMONOBJS) coffmain.o
 CHRPOBJS = ../common/crt0.o $(COMMONOBJS) chrpmain.o
-LIBS = $(TOPDIR)/lib/lib.a ../lib/zlib.a
+LIBS = $(TOPDIR)/lib/lib.o ../lib/zlib.a
 
 MKNOTE := ../utils/mknote
 SIZE := ../utils/size
diff -urdN linux/arch/sparc/Makefile linux/arch/sparc/Makefile
--- linux/arch/sparc/Makefile	Sat Jul 28 19:12:37 2001
+++ linux/arch/sparc/Makefile	Thu Jan 31 19:26:07 2002
@@ -41,7 +41,7 @@
 CORE_FILES := arch/sparc/kernel/kernel.o arch/sparc/mm/mm.o $(CORE_FILES) \
 	arch/sparc/math-emu/math-emu.o
 
-LIBS := $(TOPDIR)/lib/lib.a $(LIBS) $(TOPDIR)/arch/sparc/prom/promlib.a \
+LIBS := $(LIBS) $(TOPDIR)/arch/sparc/prom/promlib.a \
 	$(TOPDIR)/arch/sparc/lib/lib.a
 
 # This one has to come last
diff -urdN linux/arch/sparc64/Makefile linux/arch/sparc64/Makefile
--- linux/arch/sparc64/Makefile	Mon Jan 14 18:10:44 2002
+++ linux/arch/sparc64/Makefile	Thu Jan 31 19:30:09 2002
@@ -69,7 +69,7 @@
 
 CORE_FILES += arch/sparc64/math-emu/math-emu.o
 
-LIBS := $(TOPDIR)/lib/lib.a $(LIBS) $(TOPDIR)/arch/sparc64/prom/promlib.a \
+LIBS := $(LIBS) $(TOPDIR)/arch/sparc64/prom/promlib.a \
 	$(TOPDIR)/arch/sparc64/lib/lib.a
 
 vmlinux.aout: vmlinux
diff -urdN linux/include/asm-i386/system.h linux/include/asm-i386/system.h
--- linux/include/asm-i386/system.h	Wed Jan 30 05:41:09 2002
+++ linux/include/asm-i386/system.h	Thu Jan 31 20:38:16 2002
@@ -123,6 +123,8 @@
 	__asm__("movl %0,%%cr4": :"r" (x));
 #define stts() write_cr0(8 | read_cr0())
 
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif	/* __KERNEL__ */
 
 #define wbinvd() \
diff -urdN linux/include/asm-ia64/system.h linux/include/asm-ia64/system.h
--- linux/include/asm-ia64/system.h	Fri Nov  9 22:26:17 2001
+++ linux/include/asm-ia64/system.h	Thu Jan 31 20:38:55 2002
@@ -412,6 +412,8 @@
 } while (0)
 #endif
 
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif /* __KERNEL__ */
 
 #endif /* __ASSEMBLY__ */
diff -urdN linux/include/asm-mips64/system.h linux/include/asm-mips64/system.h
--- linux/include/asm-mips64/system.h	Wed Jul  4 18:50:39 2001
+++ linux/include/asm-mips64/system.h	Thu Jan 31 20:39:49 2002
@@ -256,4 +256,6 @@
 
 extern void set_except_vector(int n, void *addr);
 
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif /* _ASM_SYSTEM_H */
diff -urdN linux/include/asm-s390/system.h linux/include/asm-s390/system.h
--- linux/include/asm-s390/system.h	Wed Jul 25 21:12:02 2001
+++ linux/include/asm-s390/system.h	Thu Jan 31 20:40:24 2002
@@ -250,6 +250,9 @@
 extern void save_fp_regs(s390_fp_regs *fpregs);
 extern int restore_fp_regs1(s390_fp_regs *fpregs);
 extern void restore_fp_regs(s390_fp_regs *fpregs);
+
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif
 
 #endif
diff -urdN linux/include/asm-s390x/system.h linux/include/asm-s390x/system.h
--- linux/include/asm-s390x/system.h	Wed Jul 25 21:12:03 2001
+++ linux/include/asm-s390x/system.h	Thu Jan 31 20:40:40 2002
@@ -260,6 +260,9 @@
 extern void save_fp_regs(s390_fp_regs *fpregs);
 extern int restore_fp_regs1(s390_fp_regs *fpregs);
 extern void restore_fp_regs(s390_fp_regs *fpregs);
+
+#define HAVE_ARCH_BUST_SPINLOCKS
+
 #endif
 
 #endif
diff -urdN linux/lib/Makefile linux/lib/Makefile
--- linux/lib/Makefile	Wed Jan 30 05:30:41 2002
+++ linux/lib/Makefile	Thu Jan 31 20:28:16 2002
@@ -6,7 +6,7 @@
 # unless it's something special (ie not a .c file).
 #
 
-L_TARGET := lib.a
+O_TARGET := lib.o
 
 export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o crc32.o
 
diff -urdN linux/lib/bust_spinlocks.c linux/lib/bust_spinlocks.c
--- linux/lib/bust_spinlocks.c	Mon Sep 17 04:22:40 2001
+++ linux/lib/bust_spinlocks.c	Thu Jan 31 20:43:10 2002
@@ -14,6 +14,8 @@
 #include <linux/wait.h>
 #include <linux/vt_kern.h>
 
+#ifndef HAVE_ARCH_BUST_SPINLOCKS
+
 extern spinlock_t timerlist_lock;
 
 void bust_spinlocks(int yes)
@@ -38,4 +40,4 @@
 	}
 }
 
-
+#endif /* HAVE_ARCH_BUST_SPINLOCKS */

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[David S. Miller]

   From: Petr Vandrovec <vandrove@vc.cvut.cz>
   Date: Thu, 31 Jan 2002 23:53:06 +0100
   
      I'd like to ask architecture maintainers (especially ia64, PPC, Sparc 
   and Sparc64 maintainers) for verification that my changes did not break 
   anything... I tested only i386, but changes were obvious...

I'll test this out up after Linus releases a pre-patch with this
included.

As a side note, this thing is so tiny (less than 4K on sparc64!) so
why don't we just include it unconditionally instead of having all
of this "turn it on for these drivers" stuff?

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[Jeff Garzik]

On Thu, Jan 31, 2002 at 02:59:04PM -0800, David S. Miller wrote:
> As a side note, this thing is so tiny (less than 4K on sparc64!) so
> why don't we just include it unconditionally instead of having all
> of this "turn it on for these drivers" stuff?

Does that 4K include the BE and LE crc tables?

<shrug>  I don't mind much either way, except that I am general
resistant to "turn this on unconditionally" for bloat reasons.
[ie. its a reflex :)]

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does not

[David Lang]

remember that CML2 claims to be able to detect that it needs to be on and
turn it on when needed

David Lang

On Thu, 31 Jan 2002, Jeff Garzik wrote:

> Date: Thu, 31 Jan 2002 18:08:42 -0500
> From: Jeff Garzik <garzik@havoc.gtf.org>
> To: David S. Miller <davem@redhat.com>
> Cc: vandrove@vc.cvut.cz, torvalds@transmeta.com, linux-kernel@vger.kernel.org,
>      paulus@samba.org, davidm@hpl.hp.com, ralf@gnu.org
> Subject: Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3
>     does not
>
> On Thu, Jan 31, 2002 at 02:59:04PM -0800, David S. Miller wrote:
> > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > why don't we just include it unconditionally instead of having all
> > of this "turn it on for these drivers" stuff?
>
> Does that 4K include the BE and LE crc tables?
>
> <shrug>  I don't mind much either way, except that I am general
> resistant to "turn this on unconditionally" for bloat reasons.
> [ie. its a reflex :)]
>
> 	Jeff
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

> As a side note, this thing is so tiny (less than 4K on sparc64!) so
> why don't we just include it unconditionally instead of having all
> of this "turn it on for these drivers" stuff?

Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
to stuff Linux into embedded devices who if anything want more configuration
options not people taking stuff out.

What I'd much rather see if this is an issue is:

bool	'Do you want to customise for a very small system' 

which auto enables all the random small stuff if you say no, and goes
much deeper into options if you say yes.

Alan

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Arnaldo Carvalho de Melo]

Em Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox escreveu:
> > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > why don't we just include it unconditionally instead of having all
> > of this "turn it on for these drivers" stuff?
> 
> Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
> to stuff Linux into embedded devices who if anything want more configuration
> options not people taking stuff out.
> 
> What I'd much rather see if this is an issue is:
> 
> bool	'Do you want to customise for a very small system' 
> 
> which auto enables all the random small stuff if you say no, and goes
> much deeper into options if you say yes.

heh, after I've read that you managed to boot 2.4 + rmap in a machine with
just 4 MB after tweaking some table sizes I thought about devoting some
time to identify those tables and making them options in make *config, with
even a nice CONFIG_TINY, like you said 8)

I'll eventually do this, and I'd appreciate if people send me suggestions
of tables/data structures that can be trimmed/reduced. Yeah, I'll take a
look at the .config files used in the embedded distros.

- Arnaldo

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Denis Vlasenko]

On 31 January 2002 21:21, Arnaldo Carvalho de Melo wrote:
> Em Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox escreveu:
> > What I'd much rather see if this is an issue is:
> >
> > bool	'Do you want to customise for a very small system'
> >
> > which auto enables all the random small stuff if you say no, and goes
> > much deeper into options if you say yes.
>
> heh, after I've read that you managed to boot 2.4 + rmap in a machine with
> just 4 MB after tweaking some table sizes I thought about devoting some
> time to identify those tables and making them options in make *config, with
> even a nice CONFIG_TINY, like you said 8)
>
> I'll eventually do this, and I'd appreciate if people send me suggestions
> of tables/data structures that can be trimmed/reduced. Yeah, I'll take a
> look at the .config files used in the embedded distros.

One of inhabitants of my "Don't delete!" mail folder:

Re: Limited RAM - how to save it?
 From: Lars Brinkhoff <lars.spam@nocrew.org>
 To: Jan-Benedict Glaw <jbglaw@microdata-pos.de>
 Cc: linux-kernel@vger.kernel.org
 
Jan-Benedict Glaw <jbglaw@microdata-pos.de> writes:
> I'm working on a 4MB linux system (for a customer) which has quite
> limited resources [...] If you've got further ideas on getting the
> kernel a bit smaller, would be nice to get a mail dropped...

I started a port of Linux 2.3.99 to a MIPS device which usually has 2M
flash and 4M RAM.

To reduce the size of the text and data sections, I sorted the output
of the "size" command and investigated the files with the largest
sections.

These are the memory-saving changes I arrived at.  They are not really
tested, so some of them may break some functionality.  Also, I don't
know whether they apply to the current kernels.

fs/dcache.c

        Changed HASH_BITS from 14 to 8.  This reduces the size of the
        cache from 128K to 2K.

fs/inode.c

        Changed HASH_BITS from 14 to 8.  This reduces the size of the
        cache from 128K to 2K.

include/linux/blk.h

        Changed NR_REQUEST from 256 to 16.  This reduces the number of
        requests that can be queued.  The size of the queue is reduced
        from 16K to 1K.

include/linux/major.h

        Changed MAX_BLKDEV and MAX_CHRDEV from 256 to 10 and 20,
        respectively.  This reduces the number of block and character
        devices and saves about 40K.

kernel/printk.c

        Changed LOG_BUF_LEN from 16384 bytes to 2048 bytes.

include/linux/tty.h

        Changed NR_PTYS and NR_LDISCS from 256 and 16, respectively,
        to 16 and 4, respectively.  Saved about 12K.

        Warning: this change may break the pty driver, in which case
        further modifications will have to be done to
        drivers/char/pty.c.

kernel/panic.c

        Changed a buffer from 1024 bytes to 200 bytes.

include/linux/sched.h

        Changed PIDHASH_SZ from 1024 to 16, which saves 1008 bytes.

include/linux/mmzone.h

        NR_GPFINDEX from 0x100 to 0x10.  Saves 4800 bytes, but I'm not
        sure it doesn't break anything.

net/Makefile, net/socket.c, net/nosocket.c

        Replacing socket.c with nosocket.c, a file containing dummy
        replacement functions for those in socket.c, saves about 24K.

        Warning: this disables the socket API entirely, but it is
        currently not used in the product.

mm/Makefile, mm/swapfile.c, mm/swap_state.c, mm/noswapfile.c, 
mm/noswap_state.c

        Replacing swapfile.c with noswapfile.c, and swap_state.c with
        noswap_state.c saves about 12K.  The no*.c files contains
        empty replacement functions.

        Warning: this disables swapping of anonymous memory, which
        isn't used in the product.  But note that demand paging of
        executables still works.

mm/Makefile, mm/mmap.c

        The functions in mmap.c could probably also be replaced by
        empty functions.  Estimated saving: 9K (not included in the
        grand total below).

*, CONFIG_MESSAGES

        Applying the CONFIG_MESSAGES patch and disabling all kernel
        messages saves about 80K.

        The CONFIG_MESSAGES patch was written by Graham Stoney
        <greyham@research.canon.com.au>.

With all of the above, and only this enabled in .config:
        CONFIG_EXPERIMENTAL=y
        CONFIG_CPU_R3000=y
        CONFIG_CPU_LITTLE_ENDIAN=y
        CONFIG_ELF_KERNEL=y
        CONFIG_BINFMT_ELF=y
        CONFIG_MODULES=y
        CONFIG_MODVERSIONS=y
        CONFIG_KMOD=y
        CONFIG_CROSSCOMPILE=y
, the kernel is down to about 550K.

Here is the output of "size vmlinux".  I think this is without the
CONFIG_MESSAGES patch (it was long since I worked with this).

text    data    bss     dec     hex     filename
572128  41964   15860   629952  99cc0   vmlinux

-- 
Lars Brinkhoff          http://lars.nocrew.org/     Linux, GCC, PDP-10
Brinkhoff Consulting    http://www.brinkhoff.se/    programming
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jens Axboe]

> include/linux/blk.h
> 
>         Changed NR_REQUEST from 256 to 16.  This reduces the number of
>         requests that can be queued.  The size of the queue is reduced
>         from 16K to 1K.

You can do even more than this -- I dunno what I/O interface these
embedded system generally uses (the mtd stuff?). Provided you provide a
direct make_request_fn entry into these instead of using the queue, you
can basically cut all of ll_rw_blk.c and elevator.c. ll_rw_block,
submit_bh, and generic_make_request would be all that is left.

That should reclaim quite a lot of space. If there's any interest in
this (has it already been done?), I can help out getting it done.

-- 
Jens Axboe

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[arjan]

In article <20020202135749.B4934@suse.de> you wrote:
>> include/linux/blk.h
>> 
>>         Changed NR_REQUEST from 256 to 16.  This reduces the number of
>>         requests that can be queued.  The size of the queue is reduced
>>         from 16K to 1K.

> You can do even more than this -- I dunno what I/O interface these
> embedded system generally uses (the mtd stuff?)

the MTD stuff wants CONFIG_BLOCKLAYER ;)
flash + jffs2 doesn't need any blocklayer at all ;)

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jens Axboe]

On Sat, Feb 02 2002, arjan@fenrus.demon.nl wrote:
> In article <20020202135749.B4934@suse.de> you wrote:
> >> include/linux/blk.h
> >> 
> >>         Changed NR_REQUEST from 256 to 16.  This reduces the number of
> >>         requests that can be queued.  The size of the queue is reduced
> >>         from 16K to 1K.
> 
> > You can do even more than this -- I dunno what I/O interface these
> > embedded system generally uses (the mtd stuff?)
> 
> the MTD stuff wants CONFIG_BLOCKLAYER ;)
> flash + jffs2 doesn't need any blocklayer at all ;)

Right, that was exactly my point...

-- 
Jens Axboe

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

axboe@suse.de said:
>  You can do even more than this -- I dunno what I/O interface these
> embedded system generally uses (the mtd stuff?). Provided you provide
> a direct make_request_fn entry into these instead of using the queue,
> you can basically cut all of ll_rw_blk.c and elevator.c. 

We don't even do that - look at jffs2_read_super(). The only reason it even 
_looks_ like we're using a block device is because it was the easiest way 
to arrange mounting. We only allow you to mount the 'mtdblock' device, 
which isn't actually used - all we do is use the minor number to look up 
the real underlying MTD device. The dependency on _any_ of the block code 
isn't real - I can fix it as soon as there's an incentive to do so (like 
CONFIG_BLK_DEV).

> ll_rw_block, submit_bh, and generic_make_request would be all that is left. 

Those can go too. Likewise page->buffers.

>  That should reclaim quite a lot of space. If there's any interest in
> this (has it already been done?), I can help out getting it done.

I had a go once. Long enough ago that it's probably not worth trying to find
it again. I'd suggest a boolean like CONFIG_BUFFER_CACHE which does the
really intrusive stuff like removing page->buffers, and CONFIG_BLK_DEV which
can be modular (if CONFIG_BUFFER_CACHE is on), and which controls
compilation of everything in drivers/block/ and fs/block_dev.c

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox wrote:
> What I'd much rather see if this is an issue is:
> 
> bool	'Do you want to customise for a very small system' 
> 
> which auto enables all the random small stuff if you say no, and goes
> much deeper into options if you say yes.

CONFIG_SMALL or similar would indeed be nice...

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

acme@conectiva.com.br said:
>  heh, after I've read that you managed to boot 2.4 + rmap in a machine
> with just 4 MB after tweaking some table sizes I thought about
> devoting some time to identify those tables and making them options in
> make *config, with even a nice CONFIG_TINY, like you said 8)

> I'll eventually do this, and I'd appreciate if people send me
> suggestions of tables/data structures that can be trimmed/reduced.
> Yeah, I'll take a look at the .config files used in the embedded
> distros. 

CONFIG_BLK_DEV=n should allow you to trim a lot of cruft that most of my 
embedded boxen never use.


--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox wrote:

    Because 100 4K drivers suddenly becomes 0.5Mb. There are those of
    us trying to stuff Linux into embedded devices who if anything
    want more configuration options not people taking stuff out.

Well, I'm more or less in agreement here, especially when working with
small embedded devices which have a few (say 16 or 32) MB of RAM for
EVERYTHING, kernel, userspace AND filesystems.

However, I wonder if we can't have the linker remove unnecessary and
unreferences objects, functions and variables?

    What I'd much rather see if this is an issue is:

    bool	'Do you want to customise for a very small system'

_IF_ the linker can remove things, it would simplify this too --- we
could if a few important places produce code slightly differently to
favour speed over size and not reference various things.  Also, the
above option would turn-off inlining as that seems to makie quite a
difference at times (BTW, I'm not sure about this, but it seems gcc
and C99 don't agree with static/extern inline semantics?)



  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Andrew Morton]

Chris Wedgwood wrote:
> 
> On Thu, Jan 31, 2002 at 11:24:10PM +0000, Alan Cox wrote:
> 
>     Because 100 4K drivers suddenly becomes 0.5Mb. There are those of
>     us trying to stuff Linux into embedded devices who if anything
>     want more configuration options not people taking stuff out.
> 
> Well, I'm more or less in agreement here, especially when working with
> small embedded devices which have a few (say 16 or 32) MB of RAM for
> EVERYTHING, kernel, userspace AND filesystems.
> 
> However, I wonder if we can't have the linker remove unnecessary and
> unreferences objects, functions and variables?

We can.  Graham Stoney had all this going against 2.2.  See

http://www.google.com/search?q=stoney+ffunction-sections&hl=en&start=10&sa=N
http://www.cs.helsinki.fi/linux/linux-kernel/Year-2000/2000-29/0415.html

>     What I'd much rather see if this is an issue is:
> 
>     bool        'Do you want to customise for a very small system'
> 
> _IF_ the linker can remove things, it would simplify this too --- we
> could if a few important places produce code slightly differently to
> favour speed over size and not reference various things.  Also, the
> above option would turn-off inlining as that seems to makie quite a
> difference at times (BTW, I'm not sure about this, but it seems gcc
> and C99 don't agree with static/extern inline semantics?)

The kernel doesn't link when you compile with -fno-inline because of all
the `extern inline' qualifiers.  These need to be converted to `static
inline'.    Jim Houston has a script which does this.  See

	http://www.uwsg.iu.edu/hypermail/linux/kernel/0201.3/0888.html

It would be rather good if we could get that script run across the
tree - no-inline has its uses at times.

-

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Fri, Feb 01, 2002 at 07:01:56PM -0800, Andrew Morton wrote:

    We can.  Graham Stoney had all this going against 2.2.  See

    http://www.google.com/search?q=stoney+ffunction-sections&hl=en&start=10&sa=N
    http://www.cs.helsinki.fi/linux/linux-kernel/Year-2000/2000-29/0415.html

This puts every function it's own section.  That seems not only
cumbersome to me, but also a little complex.  That said,  it may be a
good way if run every now and then to detect when cruft starts to
accumulate for any given .config and allow people to tweak things for
smaller kernels.

Is there no way to write something like:

--snip-- foo.c --snip--
void
blem()
{
}

void
bar()
{
    blem();
    return 0;
}

int foo()
{
    return 1;
}

int main(...)
{
    return foo();
}
--snip-- foo.c --snip--

compile and link it, have the linker know main or some part of crt0 is
special, build a graph of the final ELF object, see that bar and blem
are not connected to 'main' and discard them?

I'm pretty sure (but not 100% certain) that oher compilers can do
this, maybe someone can test on other platforms?

A really smart linker (if given enough compiler help) could build a
directional graph and still remove this code even if blem called foo.


Perhaps I'm making something that's extremely complex sound simple,
but it doesn't seem to me that this should be that complex...  maybe
someone more farmiliar with binutils and/or gcc can comment and tell
me why I'm being a fool :)

    The kernel doesn't link when you compile with -fno-inline because
    of all the `extern inline' qualifiers.  These need to be converted
    to `static inline'.  Jim Houston has a script which does this.

Semantically, in gcc land, someone explain the difference between:

     inline

     extern inline

     static inline

please?


My tests seem to indicte:

  inline creates an non-inline functions.  Simple tests failed to have
  this function inlined at all

  static inline and extern inline functions can and will be inlined depending
  on optimization level

  extern inline doesn't product a non-inlined function (even if it is
  referenced) and hence barfs if you don't compile with optimisations
  for my simple test

Now, I wonder

  why 'inline' for me, never inlines?

  is there a way to force inlining of a function?

  are non-inlined functions ever called when optimizations are
  enabled?




Thanks,


  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Daniel Jacobowitz]

On Fri, Feb 01, 2002 at 11:30:20PM -0800, Chris Wedgwood wrote:
> Is there no way to write something like:
> 
> --snip-- foo.c --snip--
> void
> blem()
> {
> }
> 
> void
> bar()
> {
>     blem();
>     return 0;
> }
> 
> int foo()
> {
>     return 1;
> }
> 
> int main(...)
> {
>     return foo();
> }
> --snip-- foo.c --snip--
> 
> compile and link it, have the linker know main or some part of crt0 is
> special, build a graph of the final ELF object, see that bar and blem
> are not connected to 'main' and discard them?
> 
> I'm pretty sure (but not 100% certain) that oher compilers can do
> this, maybe someone can test on other platforms?
> 
> A really smart linker (if given enough compiler help) could build a
> directional graph and still remove this code even if blem called foo.

One piece of the necessary compiler help would be -ffunction-sections. 
If they are in the same section in the same object file, they simply
can not be removed safely.  Relocation information for calls to local
functions is not reliably available at link time.


-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Sat, Feb 02, 2002 at 02:42:36AM -0500, Daniel Jacobowitz wrote:
> One piece of the necessary compiler help would be -ffunction-sections. 
> If they are in the same section in the same object file, they simply
> can not be removed safely.

Such as the patch that was mentioned earlier in this thread :)


> Relocation information for calls to local
> functions is not reliably available at link time.

With a smarter toolchain it could be.

One will need a smarter toolchain in order to re-order functions
anyway, which is an area where benchmarks on other compilers are
showing benefits.  (ie. moving "cold" functions to the end of the
module, given profiling feedback)

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Daniel Jacobowitz]

On Sat, Feb 02, 2002 at 03:08:47AM -0500, Jeff Garzik wrote:
> On Sat, Feb 02, 2002 at 02:42:36AM -0500, Daniel Jacobowitz wrote:
> > One piece of the necessary compiler help would be -ffunction-sections. 
> > If they are in the same section in the same object file, they simply
> > can not be removed safely.
> 
> Such as the patch that was mentioned earlier in this thread :)

Yes, the patch that he was objecting to :)

> > Relocation information for calls to local
> > functions is not reliably available at link time.
> 
> With a smarter toolchain it could be.
> 
> One will need a smarter toolchain in order to re-order functions
> anyway, which is an area where benchmarks on other compilers are
> showing benefits.  (ie. moving "cold" functions to the end of the
> module, given profiling feedback)

No, the linker simply can not assume that this information is present. 
That's a limitation of ELF.  The right way to do this sort of
optimization is to compile every function into its own section and then
handle the (very small) performance cost of doing so by a relaxation
pass in the linker.  You can't safely remove any part of an input
section, no matter what relocation information you think you have.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Fri, Feb 01, 2002 at 11:30:20PM -0800, Chris Wedgwood wrote:
> A really smart linker (if given enough compiler help) could build a
> directional graph and still remove this code even if blem called foo.

Agreed and this has been my objection to the function-sections patch.
There is no need for it if you make the toolchain smarter.

For example I would love to see a !CONFIG_MODULES build rip out all the
code that was not actively referenced, such as EXPORT_SYMBOL functions
which are never called.

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

[-- Attachment #1: Type: text/plain, Size: 2323 bytes --]

cc: list trimmed.

On Fri, 1 Feb 2002 23:30:20 -0800, 
Chris Wedgwood <cw@f00f.org> wrote:
>compile and link it, have the linker know main or some part of crt0 is
>special, build a graph of the final ELF object, see that bar and blem
>are not connected to 'main' and discard them?
>
>I'm pretty sure (but not 100% certain) that oher compilers can do
>this, maybe someone can test on other platforms?
>
>A really smart linker (if given enough compiler help) could build a
>directional graph and still remove this code even if blem called foo.

AFAICT it has to be done in two stages.  Identify extern symbols that
are not referenced and convert them to static, then compile and let gcc
warn about unused statics and remove them.

The two stage approach is required because ld does not have enough
information to identify all function calls.  ld views an object as
opaque blobs of text and data with relocation entries that tie them
together.  When a function is called from the same source file, gcc/gas
tend to use PC relative branches for the inter-source call.  PC
relative local branches do not usually generate relocation entries.
Without a relocation entry, ld cannot detect the inter-source call.

There is also a problem with exported symbols.  To ld, EXPORT_SYMBOL
looks like a reference to the symbol, but the export entry is
irrelevant, what really matters is if any module refers to those
symbols.  ld cannot distinguish between used and unused exported
symbols.  To make matters worse, if you define foo as static and
EXPORT_SYMBOL(foo) in the same source then gcc thinks that the symbol
is used, because of the EXPORT_SYMBOL() reference.  Again what really
matters is if any module uses the exported symbol.

The script below runs the .o files for the kernel and identifies extern
symbols with no extern references, i.e. the first part of the problem.
It knows about exported symbols and modules and handles them correctly.
I have not run the script for years, the list of conglomerates and the
list starting with mod_use_count_ will need updating.

Beware that you get a lot of false positives because of the kernel
config mechanism.  A symbol may be listed as unused but changing the
config may cause it to be used.  This is especially true for exported
symbols.

<davem>What, no initc**l reference?</davem>

[-- Attachment #2: namespace.pl --]
[-- Type: text/plain , Size: 7714 bytes --]

#!/usr/bin/perl -w
#
#	namespace.pl.  Mon Jan 27 22:25:47 EST 1997
#
#	Perform a name space analysis on the linux kernel.
#
#	Copyright Keith Owens <kaos@ocs.com.au>.  GPL.
#
#	Invoke by changing directory to the top of the kernel source
#	tree then namespace.pl, no parameters.
#
#	Tuned for 2.1.x kernels with the new module handling, it will
#	work with 2.0 kernels as well.  Last change 2.1.81.
#
#	The source must be compiled/assembled first, the object files
#	are the primary input to this script.  Incomplete or missing
#	objects will result in a flawed analysis.
#
#	Even with complete objects, treat the result of the analysis
#	with caution.  Some external references are only used by
#	certain architectures, others with certain combinations of
#	configuration parameters.  Ideally the source should include
#	something like
#
#	#ifndef CONFIG_...
#	static
#	#endif
#	symbol_definition;
#
#	so the symbols are defined as static unless a particular
#	CONFIG_... requires it to be external.
#

require 5;	# at least perl 5
use strict;
use File::Find;

my $nm = "/usr/bin/nm -p";	# in case somebody moves nm

if ($#ARGV != -1) {
	print STDERR "usage: $0 takes no parameters\n";
	die("giving up\n");
}

my %nmdata = ();	# nm data for each object
my %def = ();		# all definitions for each name
my %ksymtab = ();	# names that appear in __ksymtab_
my %ref = ();		# $ref{$name} exists if there is a true external reference to $name
my %export = ();	# $export{$name} exists if there is an EXPORT_... of $name

&find(\&linux_objects, '.');	# find the objects and do_nm on them
list_multiply_defined();
resolve_external_references();
list_extra_externals();

exit(0);

sub linux_objects
{
	# Select objects, ignoring objects which are only created by
	# merging other objects.  Also ignore all of modules, scripts
	# and compressed.
	my $basename = $_;
	$_ = $File::Find::name;
	s:^\./::;
	if (/.*\.o$/ && ! (
	    m:/fs.o$: || m:/isofs.o$: || m:/nfs.o$:
	    || m:/xiafs.o$: || m:/umsdos.o$: || m:/hpfs.o$:
	    || m:/smbfs.o$: || m:/ncpfs.o$: || m:/ufs.o$:
	    || m:/affs.o$: || m:/romfs.o$: || m:/kernel.o$:
	    || m:/mm.o$: || m:/ipc.o$: || m:/ext.o$:
	    || m:/msdos.o$: || m:/proc/proc.o$: || m:/minix.o$:
	    || m:/ext2.o$: || m:/sysv.o$: || m:/fat.o$:
	    || m:/vfat.o$: || m:/unix.o$: || m:/802.o$:
	    || m:/appletalk.o$: || m:/ax25.o$: || m:/core.o$:
	    || m:/ethernet.o$: || m:/ipv4.o$: || m:/ipx.o$:
	    || m:/netrom.o$: || m:/ipv6.o$: || m:/x25.o$:
	    || m:/rose.o$: || m:/bridge.o$: || m:/lapb.o$:
	    || m:/sock_n_syms.o$: || m:/teles.o$: || m:/pcbit.o$:
	    || m:/isdn.o$: || m:/ftape.o$: || m:/scsi_mod.o$:
	    || m:/sd_mod.o$: || m:/sr_mod.o$: || m:/lowlevel.o$:
	    || m:/sound.o$: || m:/piggy.o$: || m:/bootsect.o$:
	    || m:/boot/setup.o$: || m:^modules/: || m:^scripts/:
	    || m:/compressed/:
	    || m:/autofs.o$: || m:/lockd/lockd.o$: || m:/nfsd.o$:
	    || m:/sunrpc.o$: || m:/scsi_n_syms.o$:
	    || m:/boot/bbootsect.o$: || m:/boot/bsetup.o$:
	    || m:/misc/parport.o$: || m:/nls/nls.o$:
	    || m:/debug/debug.o$: || m:/netlink/netlink.o$:
	    || m:/sched/sched.o$: || m:/sound/sb.o$: 
	    || m:/sound/soundcore.o$: || m:/pci/pci_syms.o$: 
	    || m:/devpts/devpts.o$: || m:/video/fbdev.o$: 
	  )
	) {
		do_nm($basename, $_);
	}
	$_ = $basename;		# File::Find expects $_ untouched (undocumented)
}

sub do_nm
{
	my ($basename, $fullname) = @_;
	my ($source, $type, $name);
	if (! -e $basename) {
		printf STDERR "$basename does not exist\n";
		return;
	}
	if ($fullname !~ /\.o$/) {
		printf STDERR "$fullname is not an object file\n";
		return;
	}
	$source = $basename;
	$source =~ s/\.o$//;
	if (! -e "$source.c" && ! -e "$source.S") {
		printf STDERR "No source file found for $fullname\n";
		return;
	}
	if (! open(NMDATA, "$nm $basename|")) {
		printf STDERR "$nm $fullname failed $!\n";
		return;
	}
	my @nmdata;
	while (<NMDATA>) {
		chop;
		($type, $name) = (split(/ +/, $_, 3))[1..2];
		# Expected types
		# B weak external reference to data that has been resolved
		# C global variable, uninitialised
		# D global variable, initialised
		# R global array, initialised
		# T global label/procedure
		# U external reference
		# W weak external reference to text that has been resolved
		# a assembler equate
		# b static variable, uninitialised
		# d static variable, initialised
		# r static array, initialised
		# t static label/procedures
		# ? undefined type, used a lot by modules
		if ($type !~ /^[BCDRTUWabdrt?]$/) {
			printf STDERR "nm output for $fullname contains unknown type '$_'\n";
		}
		elsif ($name =~ /\./) {
			# name with '.' is local static
		}
		else {
			$name =~ s/_R[a-f0-9]{8}$//;	# module versions adds this
			if ($type =~ /[BCDRTW]/ &&
				$name ne 'init_module' &&
				$name ne 'cleanup_module' &&
				$name ne 'Using_Versions' &&
				$name !~ /^Version_[0-9]+$/ &&
				$name ne 'kernel_version') {
				if (!exists($def{$name})) {
					$def{$name} = [];
				}
				push(@{$def{$name}}, $fullname);
			}
			push(@nmdata, "$type $name");
			if ($name =~ /^__ksymtab_/) {
				$name = substr($name, 10);
				if (!exists($ksymtab{$name})) {
					$ksymtab{$name} = [];
				}
				push(@{$ksymtab{$name}}, $fullname);
			}
		}
	}
	close(NMDATA);
	if ($#nmdata < 0) {
		printf "No nm data for $fullname\n";
		return;
	}
	$nmdata{$fullname} = \@nmdata;
}

sub list_multiply_defined
{
	my ($name, $module);
	foreach $name (keys(%def)) {
		if ($#{$def{$name}} > 0) {
			printf "$name is multiply defined in :-\n";
			foreach $module (@{$def{$name}}) {
				printf "\t$module\n";
			}
		}
	}
}

sub resolve_external_references
{
	my ($object, $type, $name, $i, $j, $kstrtab, $ksymtab, $export);
	printf "\n";
	foreach $object (keys(%nmdata)) {
		my $nmdata = $nmdata{$object};
		for ($i = 0; $i <= $#{$nmdata}; ++$i) {
			($type, $name) = split(' ', $nmdata->[$i], 2);
			if ($type eq "U") {
				if (exists($def{$name}) || exists($ksymtab{$name})) {
					# add the owning object to the nmdata
					$nmdata->[$i] = "$type $name $object";
					# only count as a reference if it is not EXPORT_...
					$kstrtab = "? __kstrtab_$name";
					$ksymtab = "? __ksymtab_$name";
					$export = 0;
					for ($j = 0; $j <= $#{$nmdata}; ++$j) {
						if ($nmdata->[$j] eq $kstrtab ||
						    $nmdata->[$j] eq $ksymtab) {
							$export = 1;
							last;
						}
					}
					if ($export) {
						$export{$name} = "";
					}
					else {
						$ref{$name} = ""
					}
				}
				elsif ($name ne "mod_use_count_" &&
				       $name ne "__this_module" &&
				       $name ne "_etext" &&
				       $name ne "_edata" &&
				       $name ne "_end" &&
				       $name ne "__start___ksymtab" &&
				       $name ne "__start___ex_table" &&
				       $name ne "__stop___ksymtab" &&
				       $name ne "__stop___ex_table" &&
				       $name ne "__stop___ex_table" &&
				       $name ne "__bss_start" &&
				       $name ne "_text" &&
				       $name ne "__init_begin" &&
				       $name ne "__init_end") {
					printf "Cannot resolve reference to $name from $object\n";
				}
			}
		}
	}
}

sub list_extra_externals
{
	my %noref = ();
	my ($name, @module, $module, $export);
	foreach $name (keys(%def)) {
		if (! exists($ref{$name})) {
			@module = @{$def{$name}};
			foreach $module (@module) {
				if (! exists($noref{$module})) {
					$noref{$module} = [];
				}
				push(@{$noref{$module}}, $name);
			}
		}
	}
	if (%noref) {
		printf "\nExternally defined symbols with no external references\n";
		foreach $module (sort(keys(%noref))) {
			printf "  $module\n";
			foreach (sort(@{$noref{$module}})) {
				if (exists($export{$_})) {
					$export = " (export only)";
				}
				else {
					$export = "";
				}
				printf "    $_$export\n";
			}
		}
	}
}

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

kaos@ocs.com.au said:
>  There is also a problem with exported symbols.  To ld, EXPORT_SYMBOL
> looks like a reference to the symbol, 

Er, surely that's not a problem at all? This is desired behaviour?

>  but the export entry is irrelevant,  what really matters is if any module
> refers to those symbols.

Absolutely not.  If we mark a symbol EXPORT_SYMBOL, we want it exported. No 
questions asked.

The export entry _is_ relevant; furthermore it's the _only_ thing that's
relevant, and there is no way of knowing if there's a module that isn't in
the tree, or maybe even a module that _is_ in the tree but not compiled
today, that needs the symbol in question.

I sincerely hope it's just too early in the morning here and I'm 
misunderstanding you :)

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Sat, 02 Feb 2002 08:40:11 +0000, 
David Woodhouse <dwmw2@infradead.org> wrote:
>kaos@ocs.com.au said:
>>  There is also a problem with exported symbols.  To ld, EXPORT_SYMBOL
>> looks like a reference to the symbol, 
>
>Er, surely that's not a problem at all? This is desired behaviour?
>
>>  but the export entry is irrelevant,  what really matters is if any module
>> refers to those symbols.
>
>Absolutely not.  If we mark a symbol EXPORT_SYMBOL, we want it exported. No 
>questions asked.

Does anything still use this symbol or is it dead?  Quite valid question.

>The export entry _is_ relevant; furthermore it's the _only_ thing that's
>relevant, and there is no way of knowing if there's a module that isn't in
>the tree, or maybe even a module that _is_ in the tree but not compiled
>today, that needs the symbol in question.

I said that you get false positives because the code that uses the
symbol might not be compiled in your kernel.  The script flags symbols
that are extern or exported and are not used in the current kernel.
Somebody then has to go through the report and decide if those symbols
are required under other config settings or by third party modules,
assuming that they care about third party modules.

The script is NOT an automatic list of what can be cleaned up.  I doubt
that such a list can be generated in the face of the kernel config
system and third party modules.  It is only a starting point for
symbols that need reviewing.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

kaos@ocs.com.au said:
>  Does anything still use this symbol or is it dead?  Quite valid
> question. 

Indeed. 

>  The script is NOT an automatic list of what can be cleaned up.  I
> doubt that such a list can be generated in the face of the kernel
> config system and third party modules.  It is only a starting point
> for symbols that need reviewing.

OK, cool. As it came out of the discussion about -ffunction-sections and
automatically garbage-collecting unused functions during the build, I
thought you had other, less sane intentions. Sorry for the misunderstanding.

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > why don't we just include it unconditionally instead of having all
> > of this "turn it on for these drivers" stuff?
> 
> Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
> to stuff Linux into embedded devices who if anything want more configuration
> options not people taking stuff out.
> 
> What I'd much rather see if this is an issue is:
> 
> bool	'Do you want to customise for a very small system' 
> 
> which auto enables all the random small stuff if you say no, and goes
> much deeper into options if you say yes.

I mostly agree.  Except when I have looked at trying to get the kernel
(compiled size down) the biggest bloat was in the core.  Things like
having both a page and a block cache.

Getting code reuse in the core higher would cut down on kernel size a
lot.  But that isn't quick fix territory.

Eric

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Ralf Baechle]

On Sat, Feb 02, 2002 at 09:14:33PM -0700, Eric W. Biederman wrote:

> > > As a side note, this thing is so tiny (less than 4K on sparc64!) so
> > > why don't we just include it unconditionally instead of having all
> > > of this "turn it on for these drivers" stuff?
> > 
> > Because 100 4K drivers suddenly becomes 0.5Mb. There are those of us trying
> > to stuff Linux into embedded devices who if anything want more configuration
> > options not people taking stuff out.
> > 
> > What I'd much rather see if this is an issue is:
> > 
> > bool	'Do you want to customise for a very small system' 
> > 
> > which auto enables all the random small stuff if you say no, and goes
> > much deeper into options if you say yes.
> 
> I mostly agree.  Except when I have looked at trying to get the kernel
> (compiled size down) the biggest bloat was in the core.  Things like
> having both a page and a block cache.
> 
> Getting code reuse in the core higher would cut down on kernel size a
> lot.  But that isn't quick fix territory.

Is it really worth the effort?  During the past year the average size of
embedded systems that people want to use for seems to have increased
dramatically.  In case of the MIPS port the core activity is about to
move away from the 32-bit to 64-bit kernel.

  Ralf

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Sun, Feb 03, 2002 at 08:01:35AM +0100, Ralf Baechle wrote:

    Is it really worth the effort?  During the past year the average
    size of embedded systems that people want to use for seems to have
    increased dramatically.  In case of the MIPS port the core
    activity is about to move away from the 32-bit to 64-bit kernel.

For some hand-held devices (eg. iPAQ), we want as much memory free as
possible as the only filesystem available is ramfs... moving to 64-bit
would be unthinkable! :)


  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

cw@f00f.org said:
>  For some hand-held devices (eg. iPAQ), we want as much memory free as
> possible as the only filesystem available is ramfs...

You took the flash out of your iPAQ? How does it boot?

--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Chris Wedgwood]

On Sun, Feb 03, 2002 at 12:16:00PM +0000, David Woodhouse wrote:

    cw@f00f.org said:

    You took the flash out of your iPAQ? How does it boot?

Obviously it still has flash... but much of the filesystem is ramfs
and thus ram is rather precious.

For now anyhow, I think I'll upgrade it to 128M shortly.



  --cw

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

cw@f00f.org said:
>  Obviously it still has flash... but much of the filesystem is ramfs
> and thus ram is rather precious.

Ah, I see. What are you doing with it? Mine hardly touches the ramfs.

sh-2.03# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/mtdblock/2          32256     26908      5348  84% /
tmpfs                    15428       260     15168   2% /mnt/ramfs


--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

> Is it really worth the effort?  During the past year the average size of
> embedded systems that people want to use for seems to have increased
> dramatically.  In case of the MIPS port the core activity is about to
> move away from the 32-bit to 64-bit kernel.

Embedded system and fancy handheld toys are not really the same thing. One
subset of the market gets cheaper the other gets flashier and more high
powered.

Mips is now dead in the handheld market if windows ce drops it.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Alan Cox <alan@lxorguk.ukuu.org.uk>
   Date: Thu, 31 Jan 2002 23:24:10 +0000 (GMT)

   > As a side note, this thing is so tiny (less than 4K on sparc64!) so
   > why don't we just include it unconditionally instead of having all
   > of this "turn it on for these drivers" stuff?
   
   Because 100 4K drivers suddenly becomes 0.5Mb.

However this isn't a driver, the crc library stuff is more akin to
"strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
too?  I wish you luck building that kernel :-)

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

>    Because 100 4K drivers suddenly becomes 0.5Mb.
> 
> However this isn't a driver, the crc library stuff is more akin to
> "strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
> too?  I wish you luck building that kernel :-)

For a large number of systems you don't need the CRC library. There are no
systems where you don't need memcpy, so your comparison is stupid to say
the least.

"Gee making this link is hard" is not a good reason to simplify the
config file, its a good reason to simplify the make file system

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Horst von Brand]

"David S. Miller" <davem@redhat.com> said:

[...]

> However this isn't a driver, the crc library stuff is more akin to
> "strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
> too?  I wish you luck building that kernel :-)

libc.a was invented precisely to handle such stuff automatically when
linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
is a step in the wrong direction IMVHO.
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Fri, 01 Feb 2002 11:07:42 +0100, 
Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
>"David S. Miller" <davem@redhat.com> said:
>> However this isn't a driver, the crc library stuff is more akin to
>> "strlen()".  Are you suggesting to provide a CONFIG_STRINGOPS=n
>> too?  I wish you luck building that kernel :-)
>
>libc.a was invented precisely to handle such stuff automatically when
>linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
>is a step in the wrong direction IMVHO.

There are two contradictory requirements.  crc32.o must only be linked
if anything in the kernel needs it, linker puts crc32.o after the code
that uses it.  crc32.o must be linked before anything that uses it so
the crc32 __init code can initialize first.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Keith Owens <kaos@ocs.com.au>
   Date: Fri, 01 Feb 2002 21:28:52 +1100

   On Fri, 01 Feb 2002 11:07:42 +0100, 
   Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
   >libc.a was invented precisely to handle such stuff automatically when
   >linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
   >is a step in the wrong direction IMVHO.
   
   There are two contradictory requirements.  crc32.o must only be linked
   if anything in the kernel needs it, linker puts crc32.o after the code
   that uses it.  crc32.o must be linked before anything that uses it so
   the crc32 __init code can initialize first.
   
Horst isn't even talking about the initcall issues, he's talking about
how linking with libc works in that it only brings in the routines you
actually reference.

Will you get over the initcall thing already, you must be dreaming
about it at this point. I mean really, just GET OVER IT. :-)

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Fri, 01 Feb 2002 03:03:45 -0800 (PST), 
"David S. Miller" <davem@redhat.com> wrote:
>   From: Keith Owens <kaos@ocs.com.au>
>   Date: Fri, 01 Feb 2002 21:28:52 +1100
>
>   On Fri, 01 Feb 2002 11:07:42 +0100, 
>   Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
>   >libc.a was invented precisely to handle such stuff automatically when
>   >linking... if it doesn't work, it should be fixed. I.e., making .a --> .o
>   >is a step in the wrong direction IMVHO.
>   
>   There are two contradictory requirements.  crc32.o must only be linked
>   if anything in the kernel needs it, linker puts crc32.o after the code
>   that uses it.  crc32.o must be linked before anything that uses it so
>   the crc32 __init code can initialize first.
>   
>Horst isn't even talking about the initcall issues, he's talking about
>how linking with libc works in that it only brings in the routines you
>actually reference.

Go back and read the entire thread Dave.  It started with

http://www.lib.uaa.alaska.edu/linux-kernel/archive/2002-Week-04/1235.html

  "Besides problem with segfaulting crc32 (it is initialized after
  net/ipv4/ipconfig.c due to lib/lib.a being a library... I had to
  hardcode lib/crc32.o before --start-group in main Makefile, but it is
  another story)"

http://www.lib.uaa.alaska.edu/linux-kernel/archive/2002-Week-04/1384.html

  "included patch (for 2.5.3) fixes problems with late initialization
  of lib/crc32.o - as it was part of .a library file, it was picked by
  link process AFTER at least one .o file required it - for example
  when ipv4 autoconfiguration file needed it. So crc32's initalization
  function was invoked after ipconfig's one - and it crashed inside of
  ipconfig due to this problem."

Horst was not talking about initcall but his suggestion of using lib.a
for crc32.o had already been proven not to work, because of the
(surprise) initcall problem.  I was merely pointing out to Horst why
lib.a was not working and why other options had to be considered.

>Will you get over the initcall thing already, you must be dreaming
>about it at this point. I mean really, just GET OVER IT. :-)

Nightmare is more like it.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Fri, Feb 01, 2002 at 10:25:20PM +1100, Keith Owens wrote:
> On Fri, 01 Feb 2002 03:03:45 -0800 (PST), 
> "David S. Miller" <davem@redhat.com> wrote:
> >Will you get over the initcall thing already, you must be dreaming
> >about it at this point. I mean really, just GET OVER IT. :-)
> 
> Nightmare is more like it.

Exaggeration is more like it :)

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

> What I did propose was to eliminate the Config.in special casings all
> over the place to turn this stuff on.  It's just a wart when I look at
> it.  But you don't seem to like that idea so I'll drop it.

I'd like to eliminate lots of the magic weird cases in Config.in too - but
by making the language express it. Something like

tristate_orif "blah" CONFIG_FOO $CONFIG_SMALL

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Alan Cox <alan@lxorguk.ukuu.org.uk>
   Date: Fri, 1 Feb 2002 00:42:44 +0000 (GMT)
   
   I'd like to eliminate lots of the magic weird cases in Config.in too - but
   by making the language express it. Something like
   
   tristate_orif "blah" CONFIG_FOO $CONFIG_SMALL
   
This doesn't solve the CRC32 case.  What if you want
CONFIG_SMALL, yet some net driver that needs the crc32
routines?

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Thu, Jan 31, 2002 at 04:30:54PM -0800, David S. Miller wrote:
>    From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>    Date: Fri, 1 Feb 2002 00:42:44 +0000 (GMT)
>    
>    I'd like to eliminate lots of the magic weird cases in Config.in too - but
>    by making the language express it. Something like
>    
>    tristate_orif "blah" CONFIG_FOO $CONFIG_SMALL
>    
> This doesn't solve the CRC32 case.  What if you want
> CONFIG_SMALL, yet some net driver that needs the crc32
> routines?

Maybe not in this hypothetical future situation, but currently makefile
magic was added for crc32 specifically to ensure that it is linked
in when needed... even when CONFIG_CRC32=n.

The Config.in for crc32 only exists for the case where no driver in the
built kernel uses it... but a 3rd party module might want it.

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[H. Peter Anvin]

Followup to:  <20020131.163054.41634626.davem@redhat.com>
By author:    "David S. Miller" <davem@redhat.com>
In newsgroup: linux.dev.kernel
>    
>    tristate_orif "blah" CONFIG_FOO $CONFIG_SMALL
>    
> This doesn't solve the CRC32 case.  What if you want
> CONFIG_SMALL, yet some net driver that needs the crc32
> routines?
> 

We could do it something like what I did with inflate_fs -- build it
as a module if the kernel proper doesn't need it (and modules are
enabled.)

It *does* mean the configure rules need to contain these dependencies,
though.

crc32 is an interesting case... you can create code to make the tables
with a very small amount of code.  This saves space on disk, but not
in memory; in fact, if you can't jettison this code you lose in
memory...

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Thu, Jan 31, 2002 at 08:18:01PM -0800, H. Peter Anvin wrote:
> It *does* mean the configure rules need to contain these dependencies,
> though.

No.  The Makefile not configure needs to mention these dependencies.

Look at how crc32 is -really- included in the kernel.

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Jeff Garzik <garzik@havoc.gtf.org>
   Date: Thu, 31 Jan 2002 22:46:35 -0500
   
   Maybe not in this hypothetical future situation, but currently makefile
   magic was added for crc32 specifically to ensure that it is linked
   in when needed... even when CONFIG_CRC32=n.
   
   The Config.in for crc32 only exists for the case where no driver in the
   built kernel uses it... but a 3rd party module might want it.
   
My point is this: Having to say something like "CONFIG_INEED_CRC32"
for each driver that needs it is just plain stupid and a total eye
sore.

It would be really great if, some day, you just add your source
file(s) to drivers/net and that is the only thing you ever touch.  You
DO NOT touch Makefiles, you DO NOT touch Config.in files, you DO NOT
add Config.help entries.

The Makefile rules are auto-generated from keys in the *.c file(s), as
are the Config.in and help entries.  Ie. cp driver.[ch]
linux/drivers/net and then simply rebuild the tree.

Think of what kind of pains this would save from a maintainership
point of view.  When multiple new drivers are added at once we run
into all kinds of conflict issues today, whereas my scheme would do
away with that for all cases except totally new subsystems. (but I
think that could be automated in a similar fashion as well)

I have this feeling Keith Owens is going to scream "the new build
system DOES EXACTLY THAT!"  If so, that's fscking great. :-)

That is the kind of direction I'd like to see things going in.  The
lib config thing checking "if LANCE or 8130TOO or ... then force CRC32
on", on the other hand, is just garbage.  It is rediculious to expect
J. Random driver author to know to put junk into weird places like
that just to get at the crc32 routines.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Thu, Jan 31, 2002 at 08:25:09PM -0800, David S. Miller wrote:
>    From: Jeff Garzik <garzik@havoc.gtf.org>
>    Date: Thu, 31 Jan 2002 22:46:35 -0500
>    
>    Maybe not in this hypothetical future situation, but currently makefile
>    magic was added for crc32 specifically to ensure that it is linked
>    in when needed... even when CONFIG_CRC32=n.
>    
>    The Config.in for crc32 only exists for the case where no driver in the
>    built kernel uses it... but a 3rd party module might want it.
>    
> My point is this: Having to say something like "CONFIG_INEED_CRC32"
> for each driver that needs it is just plain stupid and a total eye
> sore.

I agree.

This is -not- the situation now, and it will never be the situation.

Did you read drivers/net/Makefile.lib ?

Another example, take a look at drivers/video/Config.in.  We actually do
not need those hideously long if statements "if foofb or barfb or bazfb"...
Makefile rules clean that shit up nicely.


> It would be really great if, some day, you just add your source
> file(s) to drivers/net and that is the only thing you ever touch.  You
> DO NOT touch Makefiles, you DO NOT touch Config.in files, you DO NOT
> add Config.help entries.

You are preaching to the choir.  ;-)


> The Makefile rules are auto-generated from keys in the *.c file(s), as
> are the Config.in and help entries.  Ie. cp driver.[ch]
> linux/drivers/net and then simply rebuild the tree.

Close...  Check out this thread from December.  I agree with Linus
we need metadata files (driver.conf), not yanking all that info out
of the source code.

http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.3/0969.html
	and
http://www.uwsg.iu.edu/hypermail/linux/kernel/0112.3/0801.html

To sum, Makefile rules are better than Config.in garbage because it can
be far more modular, and more resistant to breakage.  But, for the
longer term, we want 8139too.conf and lance.conf and...

Regards,

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Thu, 31 Jan 2002 20:25:09 -0800 (PST), 
"David S. Miller" <davem@redhat.com> wrote:
>It would be really great if, some day, you just add your source
>file(s) to drivers/net and that is the only thing you ever touch.  You
>DO NOT touch Makefiles, you DO NOT touch Config.in files, you DO NOT
>add Config.help entries.
>...
>I have this feeling Keith Owens is going to scream "the new build
>system DOES EXACTLY THAT!"  If so, that's fscking great. :-)

Close but not quite.  I would love to do exactly that and remove all
the monolithic files like Configure.*, Config.in and Makefile.  But I
can't.

The only thing stopping me writing a simple "install this source and
the kernel detects it" model is initialization order, module_init ->
.text.init.  The initialization order is controlled by the order of
entries in Makefiles and subdirs entries between Makefiles.  That
sucks, as the recent problems with crc32.o have shown.

Two years ago I suggested breaking the implicit nexus between Makefile
order and initialization order, by adding new directives to explicitly
define initialization order - where required.  For example, group
memory must initialize before group network, within group network
driver ne2000 must initialize before driver eepro100.

As long as Makefiles control initialization order, you need monolithic
Makefiles.  Adding another layer to say which order the make entries
for each source are to be combined just compounds the problem.  Let me
separate the initialization order from Makefiles and I will happily
give you "add a source and kbuild autobuilds it" model.

Why can't I do it?  Linus wants the current method, where monolithic
Makefiles control initialization order.
http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg10645.html

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Fri, Feb 01, 2002 at 04:10:13PM +1100, Keith Owens wrote:
> Why can't I do it?  Linus wants the current method, where monolithic
> Makefiles control initialization order.

Wrong, we have multi-level initcalls now.

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Fri, 1 Feb 2002 00:12:42 -0500, 
Jeff Garzik <garzik@havoc.gtf.org> wrote:
>On Fri, Feb 01, 2002 at 04:10:13PM +1100, Keith Owens wrote:
>> Why can't I do it?  Linus wants the current method, where monolithic
>> Makefiles control initialization order.
>
>Wrong, we have multi-level initcalls now.

I know, it makes it even harder to see what the initialization order
is.  Some are controlled by the Makefile/subdirs order, some by special
calls in the code.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Horst von Brand]

Keith Owens <kaos@ocs.com.au> said:

[...]

> I know, it makes it even harder to see what the initialization order
> is.  Some are controlled by the Makefile/subdirs order, some by special
> calls in the code.

Just to repeat myself: This is clearly a problem for tsort(1): Give
restrictions of the form "This has to come after that" (perhaps a special
comment at the start of the file containing the init function?), tsort that
and pick the order out of the result. Should be a few lines of script. No
central repository for the dependencies, no messing around with half the
world to fix dependencies. Plus they become explicit, which they aren't
today.
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Fri, 01 Feb 2002 14:42:11 +0100, 
Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
>Keith Owens <kaos@ocs.com.au> said:
>> I know, it makes it even harder to see what the initialization order
>> is.  Some are controlled by the Makefile/subdirs order, some by special
>> calls in the code.
>
>Just to repeat myself: This is clearly a problem for tsort(1): Give
>restrictions of the form "This has to come after that" (perhaps a special
>comment at the start of the file containing the init function?), tsort that
>and pick the order out of the result. Should be a few lines of script. No
>central repository for the dependencies, no messing around with half the
>world to fix dependencies. Plus they become explicit, which they aren't
>today.

Just to repeat myself: That is exactly what I want to do.  Linus vetoed
it in October 2000.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Horst von Brand]

Keith Owens <kaos@ocs.com.au> said:
> On Fri, 01 Feb 2002 14:42:11 +0100, 
> Horst von Brand <brand@jupiter.cs.uni-dortmund.de> wrote:
> >Keith Owens <kaos@ocs.com.au> said:
> >> I know, it makes it even harder to see what the initialization order
> >> is.  Some are controlled by the Makefile/subdirs order, some by special
> >> calls in the code.

> >Just to repeat myself: This is clearly a problem for tsort(1): Give
> >restrictions of the form "This has to come after that" (perhaps a special
> >comment at the start of the file containing the init function?), tsort that
> >and pick the order out of the result. Should be a few lines of script. No
> >central repository for the dependencies, no messing around with half the
> >world to fix dependencies. Plus they become explicit, which they aren't
> >today.

> Just to repeat myself: That is exactly what I want to do.  Linus vetoed
> it in October 2000.

Right. And I still think Linus is dead wrong here. IMVVHO.
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Jeff Garzik <garzik@havoc.gtf.org>
   Date: Thu, 31 Jan 2002 23:48:45 -0500

   I agree with Linus we need metadata files (driver.conf), not
   yanking all that info out of the source code.

I'm perfectly fine with this too.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Keith Owens <kaos@ocs.com.au>
   Date: Fri, 01 Feb 2002 16:10:13 +1100
   
   As long as Makefiles control initialization order, you need monolithic
   Makefiles.

With the current 2.5.x scheme, you can put your init into the
appropriate group.  Makefiles only control init order within
the groups.

I actually like this solution a lot.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Thu, 31 Jan 2002 22:01:21 -0800 (PST), 
"David S. Miller" <davem@redhat.com> wrote:
>   From: Keith Owens <kaos@ocs.com.au>
>   Date: Fri, 01 Feb 2002 16:10:13 +1100
>   
>   As long as Makefiles control initialization order, you need monolithic
>   Makefiles.
>
>With the current 2.5.x scheme, you can put your init into the
>appropriate group.  Makefiles only control init order within
>the groups.

But you still need monolithic makefiles.

They also control order across directories as well, including up one,
down two constructs like this.

obj-$(CONFIG_SCSI_AHA1740)      += aha1740.o
ifeq ($(CONFIG_SCSI_AIC7XXX),y)
obj-$(CONFIG_SCSI_AIC7XXX)      += aic7xxx/aic7xxx_drv.o
endif
obj-$(CONFIG_SCSI_AIC7XXX_OLD)  += aic7xxx_old.o

and the abomination

obj-$(CONFIG_SCSI_LASI700)      += lasi700.o 53c700.o

subdir-$(CONFIG_ARCH_ACORN)     += ../acorn/scsi
obj-$(CONFIG_ARCH_ACORN)        += ../acorn/scsi/acorn-scsi.o

obj-$(CONFIG_CHR_DEV_ST)        += st.o

BTW, the grouped initcalls in the code are exactly what I proposed in
October 2000 and was shouted down.  Except my proposal would have put
all the initialization order information together, instead of putting
some information in the source and some in the Makefiles.  It is now
even harder to see the precise order, you have to check which type of
initcall each source is using in addition to scanning the makefile
order.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David S. Miller]

   From: Keith Owens <kaos@ocs.com.au>
   Date: Fri, 01 Feb 2002 17:11:05 +1100
   
   Except my proposal would have put all the initialization order
   information together, instead of putting some information in the
   source and some in the Makefiles.

NONE OF IT IS IN THE MAKEFILES!  Please change your perspective
to see this!

If you have a dependency concern, you put yourself in the
right initcall group.  You don't depend ever on the order within the
group, thats the whole idea.  You can't depend on that, so you must
group things correctly.

Again, if you have some dependency within the initcall group, then
you're grouping things incorrectly.  If the grouping is selected
properly, you could link in the total opposite order all the objects
and it still would work out fine.

I think your complaints are for problems that don't really exist.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Keith Owens]

On Thu, 31 Jan 2002 22:26:43 -0800 (PST), 
"David S. Miller" <davem@redhat.com> wrote:
>If you have a dependency concern, you put yourself in the
>right initcall group.  You don't depend ever on the order within the
>group, thats the whole idea.  You can't depend on that, so you must
>group things correctly.

Again this is exactly what I argued back in 2000.  I have long held
that the kernel link order is over defined where it should be fuzzy.
Defining an order between groups but not within groups is exactly what
I wanted but I was told that the initialization order must be
explicitly and fully specified for the entire kernel.  Nice to see that
I have been proved right, pity it took this long.  C'est la vie.

The Makefiles still control order within the .text.init section
(__init, module_init).  Many drivers depend on the Makefile getting
that order correct, otherwise probes stuff up.  But which entries are
order sensitive and which ones are from a developer picking a random
place to insert obj-$(CONFIG) is anyone's guess.

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

> If you have a dependency concern, you put yourself in the
> right initcall group.  You don't depend ever on the order within the
> group, thats the whole idea.  You can't depend on that, so you must
> group things correctly.

This was proposed right back at the start. Linus point blank vetoed it.

Alan

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Fri, Feb 01, 2002 at 03:03:13PM +0000, Alan Cox wrote:
> > If you have a dependency concern, you put yourself in the
> > right initcall group.  You don't depend ever on the order within the
> > group, thats the whole idea.  You can't depend on that, so you must
> > group things correctly.
> 
> This was proposed right back at the start. Linus point blank vetoed it.

My ideal would be to express dependencies in driver.conf (when that is
implemented), and that will in turn affect the link order by
autogenerating part of vmlinux.lds.  Until then, initcall groups are
fine with me...

	Jeff

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Petr Vandrovec]

On Fri, Feb 01, 2002 at 09:55:10AM -0500, Jeff Garzik wrote:
> On Fri, Feb 01, 2002 at 03:03:13PM +0000, Alan Cox wrote:
> > > If you have a dependency concern, you put yourself in the
> > > right initcall group.  You don't depend ever on the order within the
> > > group, thats the whole idea.  You can't depend on that, so you must
> > > group things correctly.
> > 
> > This was proposed right back at the start. Linus point blank vetoed it.

Hi Linus,
  I've found that multiple level initcalls went into kernel
behind my back, so you can throw away my yesterday patch
which converted lib.a => lib.o, and apply this one.

  I decided to use fs_initcall level, as nobody is using this
yet. If jffs2 will get initialized by fs_initcall, we'll have
to switch init_crc32 from fs_initcall to arch_initcall or
even sooner - crc32 initialization needs only working
kmalloc, nothing else (and it could use static buffer happilly,
as zeroes compress very good), so subsys_initcall could work
too.

  This patch does NOT solve problem with useless option CRC32 in
Config.in, as we compile crc32.o, but nobody picks it up for kernel
link. It is impossible to have .a for this directory unless we'll
start using --undefined= linker option to bring crc32 into final
image.

  Best is probably applying this over lib.a=>lib.o patch. Then
ordering is ensured by initcall level, and not by Makefiles,
and in addition CRC32 makefile option really does something...
					Best regards,
						Petr Vandrovec
						vandrove@vc.cvut.cz

[Patch tested with both lib.a and lib.o - it boots correctly
in both cases]


diff -urdN linux/lib/crc32.c linux/lib/crc32.c
--- linux/lib/crc32.c	Tue Jan 15 19:08:05 2002
+++ linux/lib/crc32.c	Fri Feb  1 14:54:28 2002
@@ -564,7 +564,7 @@
 	crc32cleanup_be();
 }
 
-module_init(init_crc32);
+fs_initcall(init_crc32);
 module_exit(cleanup_crc32);
 
 EXPORT_SYMBOL(crc32_le);

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[David Woodhouse]

vandrove@vc.cvut.cz said:
>   I decided to use fs_initcall level, as nobody is using this yet. If
> jffs2 will get initialized by fs_initcall, we'll have to switch
> init_crc32 from fs_initcall to arch_initcall or even sooner - crc32
> initialization needs only working kmalloc, nothing else (and it could
> use static buffer happilly, as zeroes compress very good), so
> subsys_initcall could work too. 

JFFS2 won't actually use the crc32 routines till it tries to mount a file 
system, so you shouldn't have a problem unless you try to mount the rootfs 
before init_crc32().


--
dwmw2

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Horst von Brand]

Jeff Garzik <garzik@havoc.gtf.org> said:
> On Fri, Feb 01, 2002 at 03:03:13PM +0000, Alan Cox wrote:
> > > If you have a dependency concern, you put yourself in the
> > > right initcall group.  You don't depend ever on the order within the
> > > group, thats the whole idea.  You can't depend on that, so you must
> > > group things correctly.

> > This was proposed right back at the start. Linus point blank vetoed it.

> My ideal would be to express dependencies in driver.conf (when that is
> implemented), and that will in turn affect the link order by
> autogenerating part of vmlinux.lds.  Until then, initcall groups are
> fine with me...

Not _one_ central file telling everything, please! Let each driver declare
what it needs and provides, and sort it out from there. Even better would
be something like "depmod -a" that doesn't need explicit dependencies
declared, but that looks like too much hair IMVHO.

Why do I have this uneasy feeling it would somehow overlap with CML2's job?
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

Horst von Brand wrote:
> 
> Jeff Garzik <garzik@havoc.gtf.org> said:
> > On Fri, Feb 01, 2002 at 03:03:13PM +0000, Alan Cox wrote:
> > > > If you have a dependency concern, you put yourself in the
> > > > right initcall group.  You don't depend ever on the order within the
> > > > group, thats the whole idea.  You can't depend on that, so you must
> > > > group things correctly.
> 
> > > This was proposed right back at the start. Linus point blank vetoed it.
> 
> > My ideal would be to express dependencies in driver.conf (when that is
> > implemented), and that will in turn affect the link order by
> > autogenerating part of vmlinux.lds.  Until then, initcall groups are
> > fine with me...
> 
> Not _one_ central file telling everything, please! Let each driver declare
> what it needs and provides, and sort it out from there.

"driver.conf" is a reference is a metadata file per driver or
per-subdirectory, not one big file.

Linus suggested this back in December.  See the links I posted.


> Why do I have this uneasy feeling it would somehow overlap with CML2's job?

It could either (a) generate CML2 or (b) make CML2 irrelevant, depending
on the implementation.

	Jeff



-- 
Jeff Garzik      | "I went through my candy like hot oatmeal
Building 1024    |  through an internally-buttered weasel."
MandrakeSoft     |             - goats.com

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Alan Cox]

>    tristate_orif "blah" CONFIG_FOO $CONFIG_SMALL
>    
> This doesn't solve the CRC32 case.  What if you want
> CONFIG_SMALL, yet some net driver that needs the crc32
> routines?

The crc32 dependancy deduction is a seperate problem to asking excessive
questions. That is down to the makefiles. If its hard to express then
presumably Rules.make isnt yet perfect.

Does kbuild make it all simpler ?

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Rob Landley]

On Thursday 31 January 2002 07:30 pm, David S. Miller wrote:
>    From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>    Date: Fri, 1 Feb 2002 00:42:44 +0000 (GMT)
>
>    I'd like to eliminate lots of the magic weird cases in Config.in too -
> but by making the language express it. Something like
>
>    tristate_orif "blah" CONFIG_FOO $CONFIG_SMALL
>
> This doesn't solve the CRC32 case.  What if you want
> CONFIG_SMALL, yet some net driver that needs the crc32
> routines?

I thought this was the sort of reason CML2 was invented?

Rob

Re: [PATCH] Re: crc32 and lib.a (was Re: [PATCH] nbd in 2.5.3 does

[Jeff Garzik]

On Fri, Feb 01, 2002 at 02:37:30PM -0500, Rob Landley wrote:
> On Thursday 31 January 2002 07:30 pm, David S. Miller wrote:
> >    From: Alan Cox <alan@lxorguk.ukuu.org.uk>
> >    Date: Fri, 1 Feb 2002 00:42:44 +0000 (GMT)
> >
> >    I'd like to eliminate lots of the magic weird cases in Config.in too -
> > but by making the language express it. Something like
> >
> >    tristate_orif "blah" CONFIG_FOO $CONFIG_SMALL
> >
> > This doesn't solve the CRC32 case.  What if you want
> > CONFIG_SMALL, yet some net driver that needs the crc32
> > routines?
> 
> I thought this was the sort of reason CML2 was invented?

This case is handled now, CML2 is not needed for this.

RTFS

	Jeff, allowing himself to be trolled