mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Nick Terrell <terrelln@fb.com>
To: Masahiro Yamada <masahiroy@kernel.org>
Cc: Oleksandr Natalenko <oleksandr@natalenko.name>,
	Piotr Gorski <lucjan.lucjanov@gmail.com>,
	LKML <linux-kernel@vger.kernel.org>,
	"Linux Kbuild mailing list" <linux-kbuild@vger.kernel.org>,
	Michal Marek <michal.lkml@markovi.net>,
	Andrew Morton <akpm@linux-foundation.org>,
	"Kees Cook" <keescook@chromium.org>,
	Nathan Chancellor <nathan@kernel.org>,
	"Daniel Borkmann" <daniel@iogearbox.net>,
	Valentin Schneider <valentin.schneider@arm.com>,
	"Jan Alexander Steffens (heftig)" <jan.steffens@gmail.com>,
	David Howells <dhowells@redhat.com>,
	"Johannes Weiner" <hannes@cmpxchg.org>
Subject: Re: [PATCH] init: add support for zstd compressed modules
Date: Wed, 7 Apr 2021 17:45:14 +0000	[thread overview]
Message-ID: <8EC13983-4A78-422B-BCA4-16A250B7B5FC@fb.com> (raw)
In-Reply-To: <CAK7LNARiB5mDJGjg1Fi1iAHbiP99Z=0_uC9069mQhZ86hE4Vtg@mail.gmail.com>



> On Apr 7, 2021, at 6:53 AM, Masahiro Yamada <masahiroy@kernel.org> wrote:
> 
> On Thu, Apr 1, 2021 at 4:21 AM Nick Terrell <terrelln@fb.com> wrote:
>> 
>> 
>> 
>>> On Mar 31, 2021, at 10:48 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote:
>>> 
>>> Hello.
>>> 
>>> On Wed, Mar 31, 2021 at 05:39:25PM +0000, Nick Terrell wrote:
>>>> 
>>>> 
>>>>> On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote:
>>>>> 
>>>>> On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote:
>>>>>> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel.
>>>>>> 
>>>>>> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com>
>>>>>> ---
>>>>>> Makefile     | 7 +++++--
>>>>>> init/Kconfig | 9 ++++++---
>>>>>> 2 files changed, 11 insertions(+), 5 deletions(-)
>>>>>> 
>>>>>> diff --git a/Makefile b/Makefile
>>>>>> index 5160ff8903c1..82f4f4cc2955 100644
>>>>>> --- a/Makefile
>>>>>> +++ b/Makefile
>>>>>> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP
>>>>>> export mod_strip_cmd
>>>>>> 
>>>>>> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
>>>>>> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
>>>>>> -# or CONFIG_MODULE_COMPRESS_XZ.
>>>>>> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP,
>>>>>> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD.
>>>>>> 
>>>>>> mod_compress_cmd = true
>>>>>> ifdef CONFIG_MODULE_COMPRESS
>>>>>> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS
>>>>>> ifdef CONFIG_MODULE_COMPRESS_XZ
>>>>>>   mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f
>>>>>> endif # CONFIG_MODULE_COMPRESS_XZ
>>>>>> +  ifdef CONFIG_MODULE_COMPRESS_ZSTD
>>>>>> +    mod_compress_cmd = $(ZSTD) -T0 --rm -f -q
>>>> 
>>>> This will use the default zstd level, level 3. I think it would make more sense to use a high
>>>> compression level. Level 19 would probably be a good choice. That will choose a window
>>>> size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that
>>>> is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a
>>>> window size of up to 2MB, to match the XZ command. Note that if the file is smaller than
>>>> the window size, it will be shrunk to the smallest power of two at least as large as the file.
>>> 
>>> Please no. We've already done that with initramfs in Arch, and it
>>> increased the time to generate it enormously.
>>> 
>>> I understand that building a kernel is a more rare operation than
>>> regenerating initramfs, but still I'd go against hard-coding the level.
>>> And if it should be specified anyway, I'd opt in for an explicit
>>> configuration option. Remember, not all the kernel are built on
>>> build farms...
>>> 
>>> FWIW, Piotr originally used level 9 which worked okay, but I insisted
>>> on sending the patch initially without specifying level at all like it is
>>> done for other compressors. If this is a wrong approach, then oh meh,
>>> mea culpa ;).
>>> 
>>> Whatever default non-standard compression level you choose, I'm fine
>>> as long as I can change it without editing Makefile.
>> 
>> That makes sense to me. I have a deep seated need to compress files as
>> efficiently as possible for widely distributed packages. But, I understand that
>> slow compression significantly impacts build times for quick iteration. I’d be
>> happy with a compression level parameter that defaults to a happy middle.
>> 
>> I’m also fine with taking this patch as-is if it is easier, and I can put up another
>> patch that adds a compression level parameter, since I don’t want to block
>> merging this.
> 
> 
> I do not want to take such a patch.
> Meeking everyone's requirement
> results in a bad project for everyone.
> 
> 
> Does this work for you?
> 
> make modules_install ZSTD="zstd -19"

Yeah, that’s perfect. Do you think it is worth mentioning in the docs for
`MODULE_COMPRESS_ZSTD`?

Best,
Nick

>> Best,
>> Nick Terrell
>> 
>>> Thanks!
>>> 
>>>> 
>>>> Best,
>>>> Nick Terrell
>>>> 
>>>>>> +  endif # CONFIG_MODULE_COMPRESS_ZSTD
>>>>>> endif # CONFIG_MODULE_COMPRESS
>>>>>> export mod_compress_cmd
>>>>>> 
>>>>>> diff --git a/init/Kconfig b/init/Kconfig
>>>>>> index 8c2cfd88f6ef..86a452bc2747 100644
>>>>>> --- a/init/Kconfig
>>>>>> +++ b/init/Kconfig
>>>>>> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS
>>>>>>   bool "Compress modules on installation"
>>>>>>   help
>>>>>> 
>>>>>> -    Compresses kernel modules when 'make modules_install' is run; gzip or
>>>>>> -    xz depending on "Compression algorithm" below.
>>>>>> +    Compresses kernel modules when 'make modules_install' is run; gzip,
>>>>>> +    xz, or zstd depending on "Compression algorithm" below.
>>>>>> 
>>>>>>     module-init-tools MAY support gzip, and kmod MAY support gzip and xz.
>>>>>> 
>>>>>> @@ -2273,7 +2273,7 @@ choice
>>>>>>     This determines which sort of compression will be used during
>>>>>>     'make modules_install'.
>>>>>> 
>>>>>> -    GZIP (default) and XZ are supported.
>>>>>> +    GZIP (default), XZ, and ZSTD are supported.
>>>>>> 
>>>>>> config MODULE_COMPRESS_GZIP
>>>>>>   bool "GZIP"
>>>>>> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP
>>>>>> config MODULE_COMPRESS_XZ
>>>>>>   bool "XZ"
>>>>>> 
>>>>>> +config MODULE_COMPRESS_ZSTD
>>>>>> +  bool "ZSTD"
>>>>>> +
>>>>>> endchoice
>>>>>> 
>>>>>> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS
>>>>>> --
>>>>>> 2.31.0.97.g1424303384
>>>>>> 
>>>>> 
>>>>> Great!
>>>>> 
>>>>> Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name>
>>>>> 
>>>>> This works perfectly fine in Arch Linux if accompanied by the
>>>>> following mkinitcpio amendment: [1].
>>>>> 
>>>>> I'm also Cc'ing other people from get_maintainers output just
>>>>> to make this submission more visible.
>>>>> 
>>>>> Thanks.
>>>>> 
>>>>> [1] https://github.com/archlinux/mkinitcpio/pull/43
>>>>> 
>>>>> --
>>>>> Oleksandr Natalenko (post-factum)
>>>> 
>>> 
>>> --
>>> Oleksandr Natalenko (post-factum)
>> 
> 
> 
> -- 
> Best Regards
> Masahiro Yamada


  reply	other threads:[~2021-04-07 17:46 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-30 11:32 Piotr Gorski
2021-03-30 11:50 ` Oleksandr Natalenko
2021-03-31 17:39   ` Nick Terrell
2021-03-31 17:48     ` Oleksandr Natalenko
2021-03-31 19:21       ` Nick Terrell
2021-03-31 20:28         ` Oleksandr Natalenko
2021-04-07 13:53         ` Masahiro Yamada
2021-04-07 17:45           ` Nick Terrell [this message]
2021-04-07 14:32 ` Masahiro Yamada
     [not found] <722602000.5829.1617263646814@office.mailbox.org>
2021-04-01  8:01 ` torvic9
2021-04-01 19:18 ` Nick Terrell

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8EC13983-4A78-422B-BCA4-16A250B7B5FC@fb.com \
    --to=terrelln@fb.com \
    --cc=akpm@linux-foundation.org \
    --cc=daniel@iogearbox.net \
    --cc=dhowells@redhat.com \
    --cc=hannes@cmpxchg.org \
    --cc=jan.steffens@gmail.com \
    --cc=keescook@chromium.org \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lucjan.lucjanov@gmail.com \
    --cc=masahiroy@kernel.org \
    --cc=michal.lkml@markovi.net \
    --cc=nathan@kernel.org \
    --cc=oleksandr@natalenko.name \
    --cc=valentin.schneider@arm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®