[PATCH v3 20/20] kbuild: compress the kernel with pigz if available

["Lorenzo Stoakes (ARM)" <ljs@kernel.org>]

The gzip step of a kernel build is very lengthy, especially for larger
builds such as allmodconfig.

gzip itself cannot be run in parallel, however an alternative tool exists
that can, providing the same feature set as gzip itself - pigz - which
works as a drop-in replacement.

On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
1.6s, and with pigz it takes 0.09s, so the performance increase is
significant.

Therefore introduce the KPGZIP build variable which specifies the
compressor to use for the kernel image and default it to pigz if available,
or KGZIP otherwise (which itself defaults to gzip).

Modules, packaging and the embedded kernel configuration (CONFIG_IKCONFIG)
continue to use KGZIP which defaults to a single-threaded gzip instance.

Module compression is run in parallel by make, so pigz offers no benefit
there, packaging sits outside the build and the embedded configuration is
too small to matter.

pigz is executed using scripts/jobserver-exec, which reserves the job slots
make has free - necessary, as otherwise pigz would just grab unbounded
CPUs.

The kernel image compression step is always performed at the end of the
build so should typically have all job slots available.

pigz always provides byte-for-byte identical output across runs regardless
of thread count, but pigz and gzip are not guaranteed to generate the same
binary output, so reproducible builds must use the same tools consistently.

Also update Documentation/kbuild/kbuild.rst to describe KPGZIP and
Documentation/process/changes.rst to list the optional pigz dependency.

While there, add entries for the other compressor programs.

Every x86 build ends with the compression of vmlinux.bin, 36MB for
defconfig and over 200MB for allmodconfig, no-op builds are unchanged.

Whole build, 128-thread Threadripper 9980X, best of N runs:

                                         before   after     delta
                                         -------------------------------
  x86 defconfig, touch mm/vma.c, gcc        7.7s     6.0s     -1.7s (-22%)
  x86 defconfig, touch mm/vma.c, clang      6.9s     5.4s     -1.5s (-21%)
  x86 defconfig, clean, gcc                28.2s    26.3s     -1.8s (-6%)
  x86 defconfig, clean, clang              28.7s    27.2s     -1.5s (-5%)
  x86 allmodconfig, touch mm/vma.c, gcc    23.7s    15.4s     -8.3s (-35%)
  x86 allmodconfig, touch mm/vma.c, clang  22.5s    15.6s     -7.0s (-31%)
  x86 allmodconfig, clean, gcc            294.1s   278.6s     -15.5s (-5%)
  x86 allmodconfig, clean, clang          283.9s   266.8s     -17.1s (-6%)

Suggested-by: Kees Cook <kees@kernel.org>
Link: https://zlib.net/pigz/
Assisted-by: LLM
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
 Documentation/kbuild/kbuild.rst   | 17 +++++++++++++++++
 Documentation/process/changes.rst |  8 ++++++++
 Makefile                          |  5 ++++-
 kernel/Makefile                   |  6 +++++-
 scripts/Makefile.lib              |  4 ++--
 scripts/jobserver-exec            |  3 ++-
 tools/lib/python/jobserver.py     | 13 +++++++++++++
 7 files changed, 51 insertions(+), 5 deletions(-)

diff --git a/Documentation/kbuild/kbuild.rst b/Documentation/kbuild/kbuild.rst
index 5a9013bacfb7..e71a692a559a 100644
--- a/Documentation/kbuild/kbuild.rst
+++ b/Documentation/kbuild/kbuild.rst
@@ -110,6 +110,23 @@ HOSTLDLIBS
 ----------
 Additional libraries to link against when building host programs.
 
+KGZIP
+-----
+The gzip program used for compressed modules, packaging and the embedded
+kernel configuration. Defaults to gzip.
+
+KPGZIP
+------
+The gzip program used for the compressed kernel image. Defaults to pigz (a
+parallel implementation of gzip), run under scripts/jobserver-exec so that it
+uses the job slots make has free, if pigz is installed, otherwise to
+``KGZIP``. Set ``KPGZIP=`` on the make command line to use another program.
+
+KBZIP2, KLZOP, LZMA, LZ4, XZ, ZSTD
+----------------------------------
+The compressor programs for the other formats. Each defaults to the program
+of the same name.
+
 .. _userkbuildflags:
 
 USERCFLAGS
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 0aa232b117b5..28f2ca63318e 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -55,6 +55,7 @@ nfs-utils              1.0.5            showmount --version
 openssl & libcrypto    1.0.0            openssl version
 pahole                 1.26             pahole --version
 pcmciautils            004              pccardctl -V
+pigz (optional)        2.4              pigz --version
 PPP                    2.4.0            pppd --version
 procps                 3.2.0            ps --version
 Python                 3.9.x            python3 --version
@@ -206,6 +207,13 @@ GNU AWK
 GNU AWK is needed if you want kernel builds to generate address range data for
 builtin modules (CONFIG_BUILTIN_MODULE_RANGES).
 
+pigz (optional)
+---------------
+
+pigz is a parallel implementation of gzip. If it is installed the compressed
+kernel image is produced with it rather than with gzip, see ``KPGZIP`` in
+Documentation/kbuild/kbuild.rst.
+
 System utilities
 ****************
 
diff --git a/Makefile b/Makefile
index ede9cfeffd56..d50bcb480134 100644
--- a/Makefile
+++ b/Makefile
@@ -568,6 +568,9 @@ LZMA		= lzma
 LZ4		= lz4
 XZ		= xz
 ZSTD		= zstd
+# The kernel image is compressed with pigz, on the job slots make has free,
+# if it is installed. Everything else uses KGZIP.
+KPGZIP		:= $(if $(shell command -v pigz 2>/dev/null),$(PYTHON3) $(abs_srctree)/scripts/jobserver-exec pigz -p %PARALLELISM%,$(KGZIP))
 TAR		= tar
 
 CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
@@ -648,7 +651,7 @@ export RUSTC RUSTDOC RUSTFMT RUSTC_OR_CLIPPY_QUIET RUSTC_OR_CLIPPY BINDGEN LLVM_
 export HOSTRUSTC KBUILD_HOSTRUSTFLAGS
 export CPP AR NM STRIP OBJCOPY OBJDUMP READELF PAHOLE RESOLVE_BTFIDS LEX YACC AWK INSTALLKERNEL
 export PERL PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX
-export KGZIP KBZIP2 KLZOP LZMA LZ4 XZ ZSTD TAR
+export KGZIP KPGZIP KBZIP2 KLZOP LZMA LZ4 XZ ZSTD TAR
 export KBUILD_HOSTCXXFLAGS KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS KBUILD_PROCMACROLDFLAGS LDFLAGS_MODULE
 export KBUILD_USERCFLAGS KBUILD_USERLDFLAGS
 
diff --git a/kernel/Makefile b/kernel/Makefile
index 1e1a31673577..c64c82c96b40 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -153,9 +153,13 @@ obj-$(CONFIG_SCF_TORTURE_TEST) += scftorture.o
 
 $(obj)/configs.o: $(obj)/config_data.gz
 
+# .config is small, so it is compressed with gzip rather than the image compressor.
+quiet_cmd_gzip_config = GZIP    $@
+      cmd_gzip_config = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
+
 targets += config_data config_data.gz
 $(obj)/config_data.gz: $(obj)/config_data FORCE
-	$(call if_changed,gzip)
+	$(call if_changed,gzip_config)
 
 filechk_cat = cat $<
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 2f447bc25e7b..8936925d4d50 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -332,11 +332,11 @@ quiet_cmd_ar = AR      $@
 quiet_cmd_objcopy = OBJCOPY $@
 cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
 
-# Gzip
+# Gzip, for the kernel image
 # ---------------------------------------------------------------------------
 
 quiet_cmd_gzip = GZIP    $@
-      cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
+      cmd_gzip = cat $(real-prereqs) | $(KPGZIP) -n -f -9 > $@
 
 # Bzip2
 # ---------------------------------------------------------------------------
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 21b319e6c9a5..89119acb1b9e 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -4,7 +4,8 @@
 """
 Determines how many parallel tasks "make" is expecting, as it is
 not exposed via any special variables, reserves them all, runs a subprocess
-with PARALLELISM environment variable set, and releases the jobs back again.
+with PARALLELISM environment variable set and any %PARALLELISM% in its
+arguments replaced by the count, and releases the jobs back again.
 
 See:
     https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
index 0b1ffdf9f7a3..7e94635bf7d9 100755
--- a/tools/lib/python/jobserver.py
+++ b/tools/lib/python/jobserver.py
@@ -29,6 +29,10 @@ $claim child to do the actual work.
 The end goal here is to keep the total number of build tasks under the
 limit established by the initial ``make -j$n_proc`` call.
 
+A command that takes its thread count on the command line rather than from
+the environment can use the ``%PARALLELISM%`` token in its arguments, which
+is replaced by the count, or by 1 when there is no jobserver.
+
 See:
     https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
 """
@@ -38,6 +42,8 @@ import os
 import subprocess
 import sys
 
+PARALLELISM_TOKEN = "%PARALLELISM%"
+
 def warn(text, *args):
     print(f'WARNING: {text}', *args, file = sys.stderr)
 
@@ -182,6 +188,9 @@ class JobserverExec:
         Run a command setting PARALLELISM env variable to the number of
         available job slots (claim) + 1, e.g. it will reserve claim slots
         to do the actual build work, plus one to monitor its children.
+
+        Any %PARALLELISM% in the command's arguments is replaced by the same
+        number, or by 1 when there is no jobserver.
         """
         self.open()             # Ensure that self.claim is set
 
@@ -192,4 +201,8 @@ class JobserverExec:
         if self.claim:
             os.environ["PARALLELISM"] = str(self.claim)
 
+        if not isinstance(cmd, str):
+            parallelism = str(self.claim or 1)
+            cmd = [arg.replace(PARALLELISM_TOKEN, parallelism) for arg in cmd]
+
         return subprocess.call(cmd, *args, **pwargs)

-- 
2.55.0