[PATCH v3 0/7] kbuild: record cmd-prereqs in .cmd files

[Luis Augenstein <luis.augenstein@tngtech.com>]

KernelSbom (scripts/sbom/sbom.py) reconstructs the kernel build graph
from the .cmd files written by Kbuild. For most build outputs,
KernelSbom must infer the build inputs from the shell command stored in
savedcmd_*. This requires command-specific parsers, which add
unnecessary complexity and are difficult to maintain.

Kbuild already knows these inputs as $(real-prereqs), the non-phony
prerequisites of the Make rule. This series records them in .cmd files
using a new metadata field:

    make_prereqs_<target> := <prerequisites>

KernelSbom is updated to read the new field instead of parsing the
shell command, and the obsolete command-specific parsers are removed.

The issue of manually parsing shell commands was originally mentioned
in https://lore.kernel.org/r/a01233b9-23a2-4666-91ed-f1cf030dcb9f@tngtech.com.
This series is a proposal to address that.

Cmd writers and consumers
=========================

The new field should not affect existing behavior.
Below is a summary of what I checked.

.cmd files are primarily written by cmd_and_savecmd and cmd_and_fixdep.
Both writers add make_prereqs.

The following rules append information to existing .cmd files.
None of these is affected by the new field:

- scripts/Makefile.lib: cmd_gen_objtooldep only appends new rule
  target: $(wildcard ./tools/objtool/objtool)
- scripts/Makefile.build: gen_symversions only appends comments
  #SYMVER <name> <crc>
- scripts/Makefile.lib: cmd_save_c_flags only appends a variable
  saved_c_flags_<target> := <flags>
- rust/Makefile: cmd_gendwarfksyms only appends comments
  #SYMVER <name> <crc>
- kernel/Makefile: kheaders_data_dep appends a deps_* block
  deps_<target> := <headers>.

This series updates the following .cmd file consumers:

- scripts/sbom/sbom/cmd_graph/cmd_file.py: CmdFile.create() is
  rewritten to parse the fields by name, including the new
  make_prereqs field.
- scripts/make_fit.py: process_dtb() previously read and split the
  complete .cmd file to find the fdtoverlay command. It is changed to
  read only the first line, so it keeps working once the new field is
  introduced.

The remaining consumers are not affected:

- Kbuild Makefiles: -include reads .cmd files as Makefile fragments. The
  new field is an ordinary variable assignment not used in any dependency
  rule.
- scripts/Makefile.thinlto: the saved_c_flags_* lookup only reads
  saved_c_flags_<target> := <flags>.
- scripts/mod/modpost.c: extract_crcs_for_object() only reads comments
  #SYMVER <name> <crc>.
- scripts/mod/sumversion.c: parse_source_files() only reads
  source_<target> := <source> and deps_<target> := <dependencies>.
- scripts/clang-tools/gen_compile_commands.py: main() only reads the
  first line containing savedcmd_<target> := <command>.
- scripts/generate_builtin_ranges.awk and
  scripts/verify_builtin_ranges.awk: get_module_info() only reads the
  first line and searches the saved command for DKBUILD_MODFILE or
  RUST_MODFILE.

Alternative: extending the deps_* field
=======================================

An alternative to the new make_prereqs field is to merge $(real-prereqs)
into the existing deps_* field for both cmd_and_savecmd and cmd_and_fixdep.
This would avoid a new field and provide one dependency list. However, it
would broaden the meaning of deps_* from dependencies discovered through
generated .d files to include prerequisites already declared to Make.
This increases the risk of undesired side effects.

The writers listed above would need to ensure that each .cmd file
contains only one deps_* block. In particular, kheaders_data_dep would
need to merge its header dependencies into the block written by
cmd_and_savecmd instead of appending a second deps_* block.
cmd_and_fixdep, or fixdep.c depending on the implementation, would
likewise need to merge $(real-prereqs) with the generated dependencies.

Apart from KernelSbom, scripts/mod/sumversion.c is the only consumer
that explicitly reads deps_*. It uses selected dependency files to
calculate a module's srcversion. Extending deps_* could change this
module metadata. Such an unrelated change should probably be avoided.

Kbuild Makefiles also consume deps_* by including .cmd files. This
should not be an issue though, because $(real-prereqs) are already
present on the original target rule, so repeating them through deps_*
should not change rebuild behavior.

SBOM comparison
===============

I compared the SBOMs before and after the series for the same kernel
builds on x86 and arm64 allmodconfig. They are ~99% the same. The
following files are newly included in the SBOM because the manual
parsers did not cover them.

- Linker scripts: *.lds.S sources and the generated *.lds scripts.
  The old tool never extracted -T arguments, so the whole
  linker-script chain was missing.

- The intermediate vmlinux link steps: vmlinux.o,
  .vmlinux.export.c/.o, the realmode pasyms.h. Previously vmlinux.a
  was hardcoded as an input of scripts/link-vmlinux.sh. Now the real
  chain vmlinux link -> vmlinux.o -> vmlinux.a is included.

- Per-module .mod object lists: the old tool expanded the .mod file
  contents in place and recorded the objects directly as link inputs,
  without explicitly including the .mod file in the SBOM.

- Generated headers: the drm/xe and msm register tables, the aic7xxx
  sequence tables, the x32 syscall header, and module.lds.h. The old
  tool found a header only where a .c file actually includes it via
  the dep file; the new tool records the full make view, including
  headers that are never included at all.

- Config and header archives: kernel/config_data.gz and
  kernel/kheaders_data.tar.xz. The old tool missed them because their
  generation commands had no matching parser.

- Devicetree overlay sources: the .dtso files are now recorded as
  inputs of the dtb/dtbo build steps. The old tool saw only the
  temporary files those steps produce, never the sources themselves.

The following files are removed:

- dtc preprocessing scratch files: .dts.tmp, .d.pre.tmp,
  .d.dtc.tmp. The old tool recorded these temporary build by-products
  as inputs; the new tool records the real .dtso source instead. This
  is a cleanup, not a loss.

- Host tooling: the rust toolchain .md docs, the gen_smb1/2_mapping
  perl generators, and the dtc host tool's libfdt headers.
  Deliberately excluded. The SBOM should track what ends up in the image or
  modules, not the tooling that produces it.

Changes in v3:
- split the changes into individual patches
- include the sbom changes directly in this series instead of deferring it
- add the SBOM comparison section to the cover letter
- Assisted-by trailer follows
  Documentation/process/coding-assistants.rst

Changes in v2:
- filter depfile entries from make_prereqs in fixdep.c to ensure
  make_prereqs is consistent between clean and incremental builds.
  Filtering $(deps_$@) from previous builds is still required to not
  include deps that got removed in the current depfile.
---

Luis Augenstein (7):
  scripts/make_fit: limit .cmd file parsing to the savedcmd_ line
  kbuild: refactor fixdep to collect dependencies before printing
  scripts/sbom: skip make_prereqs lines when parsing .cmd files
  kbuild: record cmd-prereqs in .cmd files
  scripts/sbom: parse make_prereqs from .cmd files
  scripts/sbom: use make_prereqs to determine build inputs
  scripts/sbom: remove the command-specific parsers

 Documentation/tools/sbom/sbom.rst             |  22 +-
 scripts/Kbuild.include                        |  10 +-
 scripts/basic/fixdep.c                        | 163 +++++-
 scripts/make_fit.py                           |   2 +-
 scripts/sbom/sbom/cmd_graph/cmd_file.py       | 152 ++----
 scripts/sbom/sbom/cmd_graph/cmd_graph_node.py |   5 +-
 .../sbom/sbom/cmd_graph/dependency_filter.py  |  59 ++
 .../sbom/cmd_graph/hardcoded_dependencies.py  |   1 -
 .../cmd_graph/savedcmd_parser/__init__.py     |   6 -
 .../command_parser_registry.py                | 516 ------------------
 .../savedcmd_parser/command_splitter.py       | 128 -----
 .../savedcmd_parser/savedcmd_parser.py        |  67 ---
 .../cmd_graph/savedcmd_parser/tokenizer.py    |  92 ----
 scripts/sbom/sbom/config.py                   |  14 -
 scripts/sbom/sbom/spdx_graph/kernel_file.py   |  64 ++-
 .../tests/cmd_graph/test_savedcmd_parser.py   | 468 ----------------
 16 files changed, 296 insertions(+), 1473 deletions(-)
 create mode 100644 scripts/sbom/sbom/cmd_graph/dependency_filter.py
 delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/__init__.py
 delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/command_parser_registry.py
 delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/command_splitter.py
 delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/savedcmd_parser.py
 delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/tokenizer.py
 delete mode 100644 scripts/sbom/tests/cmd_graph/test_savedcmd_parser.py

-- 
2.43.0