Re: [PATCH v2 21/21] kbuild: use pigz for gzip compression if available

["Lorenzo Stoakes (ARM)" <ljs@kernel.org>]

On Mon, Sep 14, 2026 at 09:39:07AM -0700, Kees Cook wrote:
> On Mon, Sep 14, 2026 at 10:22:20AM +0100, Lorenzo Stoakes (ARM) wrote:
> > On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
> > 1.6s, and with pigz it takes 0.09s, so the performance increase is
> > significant.
>
> Neat! I wanna go learn how pigz accomplishes this -- I thought the
> problem with gzip was a common lookup table. Anyway...

Yeah not sure on the details :)

>
> > --- a/Documentation/kbuild/reproducible-builds.rst
> > +++ b/Documentation/kbuild/reproducible-builds.rst
> > @@ -76,6 +76,22 @@ include generated files.  You should ensure the source tree is
> >  pristine by running ``make mrproper`` or ``git clean -d -f -x`` before
> >  building a source package.
> >
> > +Compression tools
> > +-----------------
> > +
> > +The compressed kernel image, compressed modules and packages are produced
> > +using the program named by the make variable ``KGZIP`` (described in
> > +Documentation/kbuild/kbuild.rst).
> > +
> > +It defaults to ``pigz`` if installed (a parallel implementation of gzip),
> > +or ``gzip`` otherwise.
> > +
> > +The generated output between two invocations of identical builds with
> > +either of the default tools will be byte-for-byte equivalent.
> > +
> > +However, for reproducible builds, ensure the same tool is used on all build
> > +hosts, as different tools may generate different output from one another.
> > +
> >  Module signing
> >  --------------
> >
>
> This doc update seems totally unneeded? Having the same build tools for RB
> is already a known requirement. I don't think anything new is added here?

Ack that's fair enough, will drop the doc update.

>
> > diff --git a/Makefile b/Makefile
> > index 790ef23c5e8a..38c0cdc9f591 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -561,7 +561,7 @@ PERL		= perl
> >  PYTHON3		= python3
> >  CHECK		= sparse
> >  BASH		= bash
> > -KGZIP		= gzip
> > +KGZIP		:= $(if $(shell command -v pigz 2>/dev/null),pigz,gzip)
>
> I think the more idiomatic way to do this is:
>
> KGZIP := $(call try-run,command -v pigz,pigz,gzip)

try-run is defined in scripts/Makefile.compiler which is only included ~200
lines after KGZIP is set.

That'll also set up and tear down a temp dir for a probe that doesn't need
that, so I think it's fine as it is.

>
> However, parallelism needs to be set. We can't let it eat all CPUs: it
> needs to respect the -j make option (and make its CPU reservation known
> to "make"), which we already have a solution for in
> scripts/jobserver-exec.

The only place where it's invoked is vmlinux.bin at the end of the serial
tail, where all the tokens would be free anyway.

So I don't think it really buys anything at all?

Using jobserver-exec would also put python3 and two wrapper scripts in
front of every gzip in the build including tar -I "$(KGZIP)" when packaging
and some arm and m68k scripts too.

(For the module zips it's already constrained to a single process.)

Overall I think it's less complexity and really no delta to just invoke it
as normal.

It's designed as drop-in so it makes sense to use it as that.

>
> However, I would actually argue that given such an improvement we should just
> make pigz explicitly required and not optional. It is packaged everywhere:
>
>   │ Debian / Ubuntu              │ ✅          │ pigz (main)
>   │ Fedora                       │ ✅          │ pigz 2.8 (current)
>   │ RHEL / CentOS / Rocky / Alma │ ✅ via EPEL │ pigz — not in base/AppStream
>   │ openSUSE / SLE               │ ✅          │ pigz
>   │ Arch Linux                   │ ✅          │ pigz (extra)
>   │ Alpine                       │ ✅          │ pigz
>   │ Gentoo                       │ ✅          │ app-arch/pigz 2.8
>
> Only RHEL appears a little glitchy, but likely they would trivially move
> it to base since it's already packaged, but off in EPEL.

Definitely not something for this series, the fallback is one invocation of
command -v and I don't really want to break RHEL either :)

If, once this has landed, we want to go that way then it's simple enough
for us to change it.

>
> So, I would say that pigz would be best run as something like:
>
>   KGZIP := $(PYTHON3) $(abs_srctree)/scripts/jobserver-exec $(abs_srctree)/scripts/parallel-pigz
>
> with scripts/parallel-pigz being something like:
>
> 	#!/bin/sh
> 	exec pigz -p ${PARALLELISM:-1} "$@"
>
> > --- a/scripts/Makefile.modinst
> > +++ b/scripts/Makefile.modinst
> > @@ -145,8 +145,10 @@ endif
> >  #
> >  # Compression
> >  #
> > +# Modules are compressed in parallel by make itself, so keep the compressor
> > +# single-threaded when it is pigz.
> >  quiet_cmd_gzip = GZIP    $@
> > -      cmd_gzip = $(KGZIP) -n -f $<
> > +      cmd_gzip = $(KGZIP) $(if $(filter pigz,$(notdir $(firstword $(KGZIP)))),-p 1) -n -f $<
> >  quiet_cmd_xz = XZ      $@
>
> Then this could be:
>
> 	cmd_gzip = SINGLE_THREADED=1 $(KGZIP) -n -f $<

This is already achieved for the modinst case with a one-liner in any case.

>
> and we patch scripts/jobserver-exec:
>
> diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
> index 21b319e6c9a5..8b953148ee9f 100755
> --- a/scripts/jobserver-exec
> +++ b/scripts/jobserver-exec
> @@ -5,6 +5,7 @@
>  Determines how many parallel tasks "make" is expecting, as it is
>  not exposed via any special variables, reserves them all, runs a subprocess
>  with PARALLELISM environment variable set, and releases the jobs back again.
> +If SINGLE_THREADED is set, nothing is reserved and PARALLELISM is 1.
>
>  See:
>      https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
> diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
> index 0b1ffdf9f7a3..fc38c5020b22 100755
> --- a/tools/lib/python/jobserver.py
> +++ b/tools/lib/python/jobserver.py
> @@ -29,6 +29,11 @@ $claim child to do the actual work.
>  The end goal here is to keep the total number of build tasks under the
>  limit established by the initial ``make -j$n_proc`` call.
>
> +Setting the ``SINGLE_THREADED`` environment variable skips the reservation
> +entirely and runs the command with ``PARALLELISM=1``. This is meant for callers
> +that run many short commands in parallel themselves, where each one should use
> +only the job slot it already holds.
> +
>  See:
>      https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
>  """
> @@ -68,6 +73,13 @@ class JobserverExec:
>          self.is_open = True  # We only try once
>          self.claim = None
>          #
> +        # SINGLE_THREADED asks for no reservation at all: the command runs
> +        # with PARALLELISM=1, using only the slot its caller already holds.
> +        #
> +        if os.environ.get('SINGLE_THREADED'):
> +            self.claim = 1
> +            return
> +        #
>          # Check the make flags for "--jobserver=R,W"
>          # Note that GNU Make has used --jobserver-fds and --jobserver-auth
>          # so this handles all of them.
>
> --
> Kees Cook

--
Cheers, Lorenzo