Re: [PATCH v7 11/24] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build

[Akira Yokosawa <akiyks@gmail.com>]

[+CC: Jani, -CC: rust people and list]

Hi, (just got v8, but sending anyway ...)

Now that I could actually apply v7 of the series, here is a
(not-so-much knee-jerk) reaction to this change.
Please see below.

On Wed, 17 Sep 2025 14:15:05 +0200, Mauro Carvalho Chehab wrote:
> There are too much magic inside docs Makefile to properly run
> sphinx-build. Create an ancillary script that contains all
> kernel-related sphinx-build call logic currently at Makefile.
> 
> Such script is designed to work both as an standalone command
> and as part of a Makefile. As such, it properly handles POSIX
> jobserver used by GNU make.
> 
> On a side note, there was a line number increase due to the
> conversion (ignoring comments) is:
> 
>  Documentation/Makefile          |  131 +++----------
>  tools/docs/sphinx-build-wrapper |  293 +++++++++++++++++++++++++++++++
>  2 files changed, 323 insertions(+), 101 deletions(-)
> 
> Comments and descriptions adds:
>  tools/docs/sphinx-build-wrapper | 261 +++++++++++++++++++++++++++++++-
> 
> So, about half of the script are comments/descriptions.
> 
> This is because some things are more verbosed on Python and because
> it requires reading env vars from Makefile. Besides it, this script
> has some extra features that don't exist at the Makefile:
> 
> - It can be called directly from command line;
> - It properly return PDF build errors.
> 
> When running the script alone, it will only take handle sphinx-build
> targets. On other words, it won't runn make rustdoc after building
> htmlfiles, nor it will run the extra check scripts.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
>  Documentation/Makefile          | 129 ++-----
>  tools/docs/sphinx-build-wrapper | 599 ++++++++++++++++++++++++++++++++
>  2 files changed, 627 insertions(+), 101 deletions(-)
>  create mode 100755 tools/docs/sphinx-build-wrapper
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index fd6399c79fab..380284026c13 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
[...]

> +# Common documentation targets
> +infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
> +	$(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> +	+$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> +		--sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> +		--builddir="$(BUILDDIR)" --deny-vf=$(FONTS_CONF_DENY_VF) \
> +		--theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
>  
[...]
> +# Special handling for pdfdocs
> +ifneq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0)
> +pdfdocs:
> +	$(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> +	@echo "  SKIP    Sphinx $@ target."
>  endif
[...]

> +	$(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> +	+$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> +		--sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> +		--builddir="$(BUILDDIR)" \
> +		--theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
>  # If Rust support is available and .config exists, add rustdoc generated contents.
>  # If there are any, the errors from this make rustdoc will be displayed but
>  # won't stop the execution of htmldocs
> @@ -118,49 +85,6 @@ ifeq ($(CONFIG_RUST),y)
>  endif
>  endif
>  
[...]

> -
> -latexdocs:
> -	@$(srctree)/tools/docs/sphinx-pre-install --version-check
> -	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
> -
> -ifeq ($(HAVE_PDFLATEX),0)
> -
> -pdfdocs:
> -	$(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> -	@echo "  SKIP    Sphinx $@ target."
> -
> -else # HAVE_PDFLATEX
> -
> -pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
> -pdfdocs: latexdocs

So, this is removing explicit dependency from pdfdocs to latexdocs.

Although it is rare, there is a small chance where Sphinx latex builder
crashes and can't complete generating all the necessary files (.tex, .sty).

In such cases, I want "make pdfdocs" to give up immediately and not to
try to run $(PDFLATEX) at all.

It looks like "./tools/docs/sphinx-build-wrapper pdfdocs" doesn't give up
at the latexdocs stage in such cases with default SPHINXDIRS="." and
continues to run $(PDFLATEX) anyway.

As a matter of fact, recent linux-next does exhibit such crashes in the
latexdocs stage:

As of next-20250917, running:
    make cleandocs: make pdfdocs" will end up in this way:

--------------------------------------------------------------
[...]
Markup is unsupported in LaTeX!

Versions
========

[...]

Last Messages
=============

        filesystems/xfs/xfs-delayed-logging-design
        filesystems/xfs/xfs-maintainer-entry-profile
        filesystems/xfs/xfs-self-describing-metadata
        filesystems/xfs/xfs-online-fsck-design
       filesystems/zonefs

    resolving references...
    processing filesystems.tex: done
    writing...
    failed

Loaded Extensions
=================

[...]

Traceback
=========

      File "/[...]/sphinx-8.2.3/lib/python3.12/site-packages/sphinx/writers/latex.py", line 1152, in visit_table
        raise UnsupportedError(
    sphinx.writers.latex.UnsupportedError: filesystems/f2fs:: longtable does not support nesting a table.

[...]

make[2]: *** [Documentation/Makefile:138: latexdocs] Error 2
make[1]: *** [/home/akira/git/linux/Makefile:1805: pdfdocs] Error 2
make: *** [Makefile:248: __sub-make] Error 2
--------------------------------------------------------------

If you merge your v7 series into linux-next and resolve confilcts properly,
running the same:

    make cleandocs; make pdfdocs

will end in this way:

Error: Can't build 45 PDF file(s): pdf/RCU.pdf, pdf/admin-guide.pdf, pdf/locking.pdf, pdf/dev-tools.pdf, pdf/process.pdf, pdf/core-api.pdf, pdf/mm.pdf, pdf/virt.pdf, pdf/block.pdf, pdf/sound.pdf, pdf/kernel-hacking.pdf, pdf/networking.pdf, pdf/infiniband.pdf, pdf/leds.pdf, pdf/maintainer.pdf, pdf/crypto.pdf, pdf/fb.pdf, pdf/accounting.pdf, pdf/livepatch.pdf, pdf/nvme.pdf, pdf/hid.pdf, pdf/isdn.pdf, pdf/trace.pdf, pdf/i2c.pdf, pdf/fault-injection.pdf, pdf/netlabel.pdf, pdf/staging.pdf, pdf/timers.pdf, pdf/firmware-guide.pdf, pdf/hwmon.pdf, pdf/scheduler.pdf, pdf/tools.pdf, pdf/watchdog.pdf, pdf/mhi.pdf, pdf/power.pdf, pdf/devicetree.pdf, pdf/arch.pdf, pdf/spi.pdf, pdf/userspace-api.pdf, pdf/translations.pdf, pdf/fpga.pdf, pdf/cpu-freq.pdf, pdf/security.pdf, pdf/bpf.pdf, pdf/pcmcia.pdf

Returning back to next-20250917, running with additional options to GNU Make:

    make clean; make -k -i pdfdocs

behaves mostly the same as the one with sphinx-build-wrapper, which ends up
like this:

-----------------------------------------------------------------------
[...]

make[3]: [Makefile:29: virt.pdf] Error 12 (ignored)
mv: cannot stat 'Documentation/output/./latex/RCU.pdf': No such file or directory
mv: cannot stat 'Documentation/output/./latex/accounting.pdf': No such file or directory
mv: cannot stat 'Documentation/output/./latex/admin-guide.pdf': No such file or directory
[...]

mv: cannot stat 'Documentation/output/./latex/virt.pdf': No such file or directory
mv: cannot stat 'Documentation/output/./latex/watchdog.pdf': No such file or directory
make[2]: [Documentation/Makefile:151: pdfdocs] Error 1 (ignored)
-----------------------------------------------------------------------

As you might have already noticed, GNU Make provides useful and convinenent
command options readily availabe.  Some of those I often use include:

   -i, --ignore-errors
        Ignore all errors in commands executed to remake files.

   -k, --keep-going
        Continue as much as possible after an error.  While the target
        that failed, and those that depend on it, cannot be remade, the
        other dependencies of these targets can be processed all the same.

   -O[type], --output-sync[=type]
        When  running multiple jobs in parallel with -j, ensure the output
        of each job is collected together rather  than  interspersed  with
        output from other jobs. [...]

There are abundunt of other helpful options for various situations.

Your change of pdfdocs recipe would deprive me of those flexible options
of GNU Make.  Because passing those options through MAKEFLAGS is
availabe in sub-make invocation of $(MAKE) [1].

[1]: https://www.gnu.org/software/make/manual/html_node/Options_002fRecursion.html

And it would diverge our scheme of pdfdocs build from that of Sphinx's
latex builder's choice.

I don't think re-implementing GNU Make's (limited range of) behavior in
python can be justified just because Makefile's way of describing recipes
look "magic" to your eyes.

Honestly speaking, I just trunt the history proven tool much more than your
re-implementation.  I'm sure I would miss that make--sub-make chain if your
sphinx-build-wrapper were employed in pdfdocs target.

As I said above, the behavior you wants can be achieved by using a couple
of options to GNU Make. (Give or take the "|| exit;" case.)
I'm more inclined to continue using existing approach.

In summary, this is my suggestion for the development for the v6.19 (not v6.18)
merge window.

1) Using sphinx-build-wrapper in (*}docs targets other than pdfdocs is the way
   to go.

2) pdfdocs should be a different target that depends on latexdocs.

3) Preserve the make--sub-make chain in the pdfdocs recipe. (for easy access
   to GNU Make's options)

4) sphinx-build-wrapper has own its rights to cover pdfdocs.  I don't care
   how it behaves when it is directly called for pdfdocs.

5) Build summary as the final message from "make pdfdocs" is a good idea and
   it should be doable in the Makefile recipe in a different manner.

6) The issue Mauro calls "false positive" (the "|| exit;" pattern in Makefile's
   loop) might be a desired behavior, but it should be possible to add a knob
   to suppress it in the Makefile.

Mauro, how does this "compromize" sound to you?

Regards,
Akira

> -	@$(srctree)/tools/docs/sphinx-pre-install --version-check
> -	$(foreach var,$(SPHINXDIRS), \
> -	   $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || $(PYTHON3) $(srctree)/tools/docs/check-variable-fonts.py || exit; \
> -	   mkdir -p $(BUILDDIR)/$(var)/pdf; \
> -	   mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
> -	)
> -
[...]