[PATCH] m68k: Add target for lz4 compressed vmlinux

[PATCH] m68k: Add target for lz4 compressed vmlinux

[Daniel Palmer]

Traditionally gzip (/bzip?) has been the compressed image format
but its a pain to decompress because its very hard to load and
decompress it in chunks which you need to do if you don't
have enough memory to load the whole kernel image and decompress
it somewhere else.

With lz4 you can set a block size, read it from the header,
and then you only need memory for a single block and the
decompressed kernel.

I use lz4 compressed images on 68000 with 8MB of ram
and MVME147 with 16MB. I want to use lz4 in my fork of EMILE
to boot m68k macs because streaming decompress of gzip is painful.

Signed-off-by: Daniel Palmer <daniel@0x0f.com>
---
 arch/m68k/Makefile | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/m68k/Makefile b/arch/m68k/Makefile
index 0abcf994ce55..a56e853037c0 100644
--- a/arch/m68k/Makefile
+++ b/arch/m68k/Makefile
@@ -124,6 +124,17 @@ else
 	$(KBZIP2) -1c vmlinux >vmlinux.bz2
 endif
 
+vmlinux.lz4: vmlinux
+
+ifndef CONFIG_KGDB
+	cp vmlinux vmlinux.tmp
+	$(STRIP) vmlinux.tmp
+	$(LZ4) -z9f vmlinux.tmp vmlinux.lz4
+	rm vmlinux.tmp
+else
+	$(LZ4) -z9f vmlinux vmlinux.lz4
+endif
+
 CLEAN_FILES += vmlinux.gz vmlinux.bz2
 
 archheaders:
-- 
2.50.0

Re: [PATCH] m68k: Add target for lz4 compressed vmlinux

[Thomas Weißschuh]

On 2025-08-09 10:55:29+0900, Daniel Palmer wrote:
> Traditionally gzip (/bzip?) has been the compressed image format
> but its a pain to decompress because its very hard to load and
> decompress it in chunks which you need to do if you don't
> have enough memory to load the whole kernel image and decompress
> it somewhere else.
> 
> With lz4 you can set a block size, read it from the header,
> and then you only need memory for a single block and the
> decompressed kernel.
> 
> I use lz4 compressed images on 68000 with 8MB of ram
> and MVME147 with 16MB. I want to use lz4 in my fork of EMILE
> to boot m68k macs because streaming decompress of gzip is painful.
> 
> Signed-off-by: Daniel Palmer <daniel@0x0f.com>
> ---
>  arch/m68k/Makefile | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/arch/m68k/Makefile b/arch/m68k/Makefile
> index 0abcf994ce55..a56e853037c0 100644
> --- a/arch/m68k/Makefile
> +++ b/arch/m68k/Makefile
> @@ -124,6 +124,17 @@ else
>  	$(KBZIP2) -1c vmlinux >vmlinux.bz2
>  endif
>  
> +vmlinux.lz4: vmlinux
> +
> +ifndef CONFIG_KGDB
> +	cp vmlinux vmlinux.tmp
> +	$(STRIP) vmlinux.tmp
> +	$(LZ4) -z9f vmlinux.tmp vmlinux.lz4
> +	rm vmlinux.tmp
> +else
> +	$(LZ4) -z9f vmlinux vmlinux.lz4
> +endif

Splitting the vmlinux.tmp creation into a dedicated target would make
all the compressor targets simpler. It will need a bit more disk space,
but there are a bunch of vmlinux copies already in any case.

> +
>  CLEAN_FILES += vmlinux.gz vmlinux.bz2

CLEAN_FILES also needs to be updated.

>  archheaders:
> -- 
> 2.50.0
> 

Re: [PATCH] m68k: Add target for lz4 compressed vmlinux

[Daniel Palmer]

Hi Thomas,

On Sat, 9 Aug 2025 at 16:50, Thomas Weißschuh <linux@weissschuh.net> wrote:
> Splitting the vmlinux.tmp creation into a dedicated target would make
> all the compressor targets simpler. It will need a bit more disk space,
> but there are a bunch of vmlinux copies already in any case.
>

That's true. I think a target for vmlinux.stripped and then use that
in the compressed image targets.

How about this?:

vmlinux.stripped: vmlinux
       cp $< $@
       $(STRIP) $@

vmlinux.gz: vmlinux.stripped

ifndef CONFIG_KGDB
       $(KGZIP) -9c vmlinux.stripped >vmlinux.gz
else
       $(KGZIP) -9c vmlinux >vmlinux.gz
endif

<snip>

> > +
> >  CLEAN_FILES += vmlinux.gz vmlinux.bz2
>
> CLEAN_FILES also needs to be updated.

Noted. Will fix for v2.

Thanks,

Daniel

Re: [PATCH] m68k: Add target for lz4 compressed vmlinux

[Thomas Weißschuh]

Hi Daniel,

On 2025-08-09 20:35:39+0900, Daniel Palmer wrote:
> On Sat, 9 Aug 2025 at 16:50, Thomas Weißschuh <linux@weissschuh.net> wrote:
> > Splitting the vmlinux.tmp creation into a dedicated target would make
> > all the compressor targets simpler. It will need a bit more disk space,
> > but there are a bunch of vmlinux copies already in any case.
> >
> 
> That's true. I think a target for vmlinux.stripped and then use that
> in the compressed image targets.
> 
> How about this?:
> 
> vmlinux.stripped: vmlinux
>        cp $< $@
>        $(STRIP) $@
> 
> vmlinux.gz: vmlinux.stripped
> 
> ifndef CONFIG_KGDB
>        $(KGZIP) -9c vmlinux.stripped >vmlinux.gz
> else
>        $(KGZIP) -9c vmlinux >vmlinux.gz
> endif
> 
> <snip>

I would continue with vmlinux.tmp. It might not actually be stripped.

quiet_cmd_precompress = PRECOMPRESS $@
ifndef CONFIG_KGDB
      cmd_precompress = cp $< $@
else
      cmd_precompress = $(STRIP) $< -o $@
endif

vmlinux.tmp: vmlinux FORCE
	$(call if_changed,precompress)

targets += vmlinux.tmp

This will also correctly handle CONFIG_KGDB changing.

(Maybe the naming can be improved)


And for the compressor invocations we already have predefined commands:

vmlinux.gz: vmlinux.tmp FORCE
	$(call if_changed,gzip)

targets += vmlinux.gz

> > > +
> > >  CLEAN_FILES += vmlinux.gz vmlinux.bz2
> >
> > CLEAN_FILES also needs to be updated.
> 
> Noted. Will fix for v2.

If you use $(call if_changed) as explained above, the "targets += ..."
will make the CLEAN_FILES unnecessary.

Re: [PATCH] m68k: Add target for lz4 compressed vmlinux

[Daniel Palmer]

Hi Thomas,

On Sat, 9 Aug 2025 at 21:29, Thomas Weißschuh <linux@weissschuh.net> wrote:
>
> Hi Daniel,
>
> On 2025-08-09 20:35:39+0900, Daniel Palmer wrote:
> > On Sat, 9 Aug 2025 at 16:50, Thomas Weißschuh <linux@weissschuh.net> wrote:
> I would continue with vmlinux.tmp. It might not actually be stripped.
>
> quiet_cmd_precompress = PRECOMPRESS $@
> ifndef CONFIG_KGDB
>       cmd_precompress = cp $< $@
> else
>       cmd_precompress = $(STRIP) $< -o $@
> endif
>
> vmlinux.tmp: vmlinux FORCE
>         $(call if_changed,precompress)
>
> targets += vmlinux.tmp
>
> This will also correctly handle CONFIG_KGDB changing.
>
> (Maybe the naming can be improved)

Nice, that's a lot better and I've reworked my patch to do that.

> And for the compressor invocations we already have predefined commands:
>
> vmlinux.gz: vmlinux.tmp FORCE
>         $(call if_changed,gzip)

I also did this for all of the targets and $(call if_changed,lz4)
results in an lz4 file with the old header format[0] and 8MB block
size which is not what I need.
So v2 will have a patch to add a helper to generate that style of lz4
in scripts/Makefile.lib

Thanks!,

Daniel

0 - https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#legacy-frame

Re: [PATCH] m68k: Add target for lz4 compressed vmlinux

[Thomas Weißschuh]

On 2025-08-10 18:08:15+0900, Daniel Palmer wrote:
> On Sat, 9 Aug 2025 at 21:29, Thomas Weißschuh <linux@weissschuh.net> wrote:
> > On 2025-08-09 20:35:39+0900, Daniel Palmer wrote:
> > > On Sat, 9 Aug 2025 at 16:50, Thomas Weißschuh <linux@weissschuh.net> wrote:
> > I would continue with vmlinux.tmp. It might not actually be stripped.
> >
> > quiet_cmd_precompress = PRECOMPRESS $@
> > ifndef CONFIG_KGDB
> >       cmd_precompress = cp $< $@
> > else
> >       cmd_precompress = $(STRIP) $< -o $@
> > endif
> >
> > vmlinux.tmp: vmlinux FORCE
> >         $(call if_changed,precompress)
> >
> > targets += vmlinux.tmp
> >
> > This will also correctly handle CONFIG_KGDB changing.
> >
> > (Maybe the naming can be improved)
> 
> Nice, that's a lot better and I've reworked my patch to do that.

If you also modify the existing code, you should split up the changes
over multiple patches. First split out the rule for vmlinux.tmp, then
switch over the compressors. Each patch should do a single step and
explain why it is done.

And I am still not happy about the naming in my example :-/

> > And for the compressor invocations we already have predefined commands:
> >
> > vmlinux.gz: vmlinux.tmp FORCE
> >         $(call if_changed,gzip)
> 
> I also did this for all of the targets and $(call if_changed,lz4)
> results in an lz4 file with the old header format[0] and 8MB block
> size which is not what I need.
> So v2 will have a patch to add a helper to generate that style of lz4
> in scripts/Makefile.lib

Modifying generic files in scripts/ will be more effort to argue for.
Also that change should be its own patch again.

I am wondering a bit why the legacy header format does not work.
The manpage of lz4 specifically mentions that it is commonly used for
Linux kernel images. Which is exactly what we are doing here.
Does your usecase work not work with the legacy header? If not, please
mention this in the patches.


Thomas