From: Nick Terrell <terrelln@fb.com>
To: Oleksandr Natalenko <oleksandr@natalenko.name>
Cc: Piotr Gorski <lucjan.lucjanov@gmail.com>,
LKML <linux-kernel@vger.kernel.org>,
Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>,
Masahiro Yamada <masahiroy@kernel.org>,
Michal Marek <michal.lkml@markovi.net>,
Andrew Morton <akpm@linux-foundation.org>,
Kees Cook <keescook@chromium.org>,
"Nathan Chancellor" <nathan@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Valentin Schneider <valentin.schneider@arm.com>,
"Jan Alexander Steffens (heftig)" <jan.steffens@gmail.com>,
David Howells <dhowells@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>
Subject: Re: [PATCH] init: add support for zstd compressed modules
Date: Wed, 31 Mar 2021 19:21:07 +0000 [thread overview]
Message-ID: <5288EB41-5257-4B4C-9D37-3C314FA90573@fb.com> (raw)
In-Reply-To: <20210331174826.uhunfmkhlpyteb3d@spock.localdomain>
> On Mar 31, 2021, at 10:48 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote:
>
> Hello.
>
> On Wed, Mar 31, 2021 at 05:39:25PM +0000, Nick Terrell wrote:
>>
>>
>>> On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote:
>>>
>>> On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote:
>>>> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel.
>>>>
>>>> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com>
>>>> ---
>>>> Makefile | 7 +++++--
>>>> init/Kconfig | 9 ++++++---
>>>> 2 files changed, 11 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/Makefile b/Makefile
>>>> index 5160ff8903c1..82f4f4cc2955 100644
>>>> --- a/Makefile
>>>> +++ b/Makefile
>>>> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP
>>>> export mod_strip_cmd
>>>>
>>>> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed
>>>> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP
>>>> -# or CONFIG_MODULE_COMPRESS_XZ.
>>>> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP,
>>>> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD.
>>>>
>>>> mod_compress_cmd = true
>>>> ifdef CONFIG_MODULE_COMPRESS
>>>> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS
>>>> ifdef CONFIG_MODULE_COMPRESS_XZ
>>>> mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f
>>>> endif # CONFIG_MODULE_COMPRESS_XZ
>>>> + ifdef CONFIG_MODULE_COMPRESS_ZSTD
>>>> + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q
>>
>> This will use the default zstd level, level 3. I think it would make more sense to use a high
>> compression level. Level 19 would probably be a good choice. That will choose a window
>> size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that
>> is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a
>> window size of up to 2MB, to match the XZ command. Note that if the file is smaller than
>> the window size, it will be shrunk to the smallest power of two at least as large as the file.
>
> Please no. We've already done that with initramfs in Arch, and it
> increased the time to generate it enormously.
>
> I understand that building a kernel is a more rare operation than
> regenerating initramfs, but still I'd go against hard-coding the level.
> And if it should be specified anyway, I'd opt in for an explicit
> configuration option. Remember, not all the kernel are built on
> build farms...
>
> FWIW, Piotr originally used level 9 which worked okay, but I insisted
> on sending the patch initially without specifying level at all like it is
> done for other compressors. If this is a wrong approach, then oh meh,
> mea culpa ;).
>
> Whatever default non-standard compression level you choose, I'm fine
> as long as I can change it without editing Makefile.
That makes sense to me. I have a deep seated need to compress files as
efficiently as possible for widely distributed packages. But, I understand that
slow compression significantly impacts build times for quick iteration. I’d be
happy with a compression level parameter that defaults to a happy middle.
I’m also fine with taking this patch as-is if it is easier, and I can put up another
patch that adds a compression level parameter, since I don’t want to block
merging this.
Best,
Nick Terrell
> Thanks!
>
>>
>> Best,
>> Nick Terrell
>>
>>>> + endif # CONFIG_MODULE_COMPRESS_ZSTD
>>>> endif # CONFIG_MODULE_COMPRESS
>>>> export mod_compress_cmd
>>>>
>>>> diff --git a/init/Kconfig b/init/Kconfig
>>>> index 8c2cfd88f6ef..86a452bc2747 100644
>>>> --- a/init/Kconfig
>>>> +++ b/init/Kconfig
>>>> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS
>>>> bool "Compress modules on installation"
>>>> help
>>>>
>>>> - Compresses kernel modules when 'make modules_install' is run; gzip or
>>>> - xz depending on "Compression algorithm" below.
>>>> + Compresses kernel modules when 'make modules_install' is run; gzip,
>>>> + xz, or zstd depending on "Compression algorithm" below.
>>>>
>>>> module-init-tools MAY support gzip, and kmod MAY support gzip and xz.
>>>>
>>>> @@ -2273,7 +2273,7 @@ choice
>>>> This determines which sort of compression will be used during
>>>> 'make modules_install'.
>>>>
>>>> - GZIP (default) and XZ are supported.
>>>> + GZIP (default), XZ, and ZSTD are supported.
>>>>
>>>> config MODULE_COMPRESS_GZIP
>>>> bool "GZIP"
>>>> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP
>>>> config MODULE_COMPRESS_XZ
>>>> bool "XZ"
>>>>
>>>> +config MODULE_COMPRESS_ZSTD
>>>> + bool "ZSTD"
>>>> +
>>>> endchoice
>>>>
>>>> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS
>>>> --
>>>> 2.31.0.97.g1424303384
>>>>
>>>
>>> Great!
>>>
>>> Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name>
>>>
>>> This works perfectly fine in Arch Linux if accompanied by the
>>> following mkinitcpio amendment: [1].
>>>
>>> I'm also Cc'ing other people from get_maintainers output just
>>> to make this submission more visible.
>>>
>>> Thanks.
>>>
>>> [1] https://github.com/archlinux/mkinitcpio/pull/43
>>>
>>> --
>>> Oleksandr Natalenko (post-factum)
>>
>
> --
> Oleksandr Natalenko (post-factum)
next prev parent reply other threads:[~2021-03-31 19:22 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-30 11:32 Piotr Gorski
2021-03-30 11:50 ` Oleksandr Natalenko
2021-03-31 17:39 ` Nick Terrell
2021-03-31 17:48 ` Oleksandr Natalenko
2021-03-31 19:21 ` Nick Terrell [this message]
2021-03-31 20:28 ` Oleksandr Natalenko
2021-04-07 13:53 ` Masahiro Yamada
2021-04-07 17:45 ` Nick Terrell
2021-04-07 14:32 ` Masahiro Yamada
[not found] <722602000.5829.1617263646814@office.mailbox.org>
2021-04-01 8:01 ` torvic9
2021-04-01 19:18 ` Nick Terrell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5288EB41-5257-4B4C-9D37-3C314FA90573@fb.com \
--to=terrelln@fb.com \
--cc=akpm@linux-foundation.org \
--cc=daniel@iogearbox.net \
--cc=dhowells@redhat.com \
--cc=hannes@cmpxchg.org \
--cc=jan.steffens@gmail.com \
--cc=keescook@chromium.org \
--cc=linux-kbuild@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lucjan.lucjanov@gmail.com \
--cc=masahiroy@kernel.org \
--cc=michal.lkml@markovi.net \
--cc=nathan@kernel.org \
--cc=oleksandr@natalenko.name \
--cc=valentin.schneider@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®