From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=34.107.87.41 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795121122; b=g2+eL4U1LlACzeEG6wKlAR7uP83k/2tpPMOjGllR/cpkU1rEL/fwdLrGaNbGw5wYrYz0 CP/xVcwZNfWXS0JuX1pXmKkKLXl11extOQ5+c0xADBSsTrc8wipf1yvb2WI9oMr1L2j1h S122H1RQRsOG/kG9tNcP7zml/o3XW3Xo0WNx9ojLt9Si+yVfaOXlu5oNSPOva18QtDldH XwcfwgGDPr05CFltJFnagcG9bxmyado9H5twMeBRfO3m4tGJDZ1xJJmVyMOTVdfMFOnHI N5eO9y0GsLR9uLXK+fKXKKV17UC9QQ3jNQMR1Vx5kNafF5EESnMeyzsggUzEmftrmOw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795121122; h=DMARC-Filter:Received:Received:Received:Received:DKIM-Filter: DKIM-Signature:X-Virus-Scanned:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:MIME-Version:Content-Transfer-Encoding; bh=/1Grfte5MGcWY78IHhh2XGB835lwhfLoqukaCJMf9Lg=; b=a8WNQxaRJFqAAA1Ds/TwxAg+8iGJToOHeocf3v1hXLRFbBFA1wx8PkJx2Y9ZxIyRBx6g n75zrq4BJiU8ZAvHnMLNiwKfQzhSVykPh3TwVwmeqyggAqSV5DdOdeh7jmqjYjs1DR5hd Q/n+yiww5RGThXXy51c8Un9Fiv0OJg9fjfjS/yVHjzg/ghQVsOSwFPGOQDTxNv1aSgezk ioms1ZAgm/oCQjI/va7sJ8aVuucOQCigHwcVD4Ydgjw8aC7renL5bIaTPGzgkBek6Ocvi +loacOJ+nLFKU48H4t7fcnNQYHHE2urwKpm2p7zMQ1U3ijCA6QDnI9zeM4Exa5HUSig== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=tngtech.com; spf=pass smtp.mailfrom=tngtech.com; dkim=pass header.d=tngtech.com header.i=@tngtech.com header.a=rsa-sha256 header.s=B14491C6-869D-11EB-BB6C-8DD33D883B31 header.b=HLMomBvq; arc=none smtp.remote-ip=34.107.87.41 Received: from mailout.tng.vnc.biz (mailout.tng.vnc.biz [34.107.87.41]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1D27606BBAA; Thu, 19 Nov 2026 20:45:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org C1D27606BBAA Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=tngtech.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tngtech.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=tngtech.com header.i=@tngtech.com header.a=rsa-sha256 header.s=B14491C6-869D-11EB-BB6C-8DD33D883B31 header.b=HLMomBvq Received: from zmproxy.tng.vnc.biz (zmproxy.tng.vnc.biz [172.16.1.23]) by mailout.tng.vnc.biz (Postfix) with ESMTPS id D992E4049B; Thu, 19 Nov 2026 21:45:13 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by zmproxy.tng.vnc.biz (Postfix) with ESMTP id B5C611FB967; Thu, 19 Nov 2026 21:45:13 +0100 (CET) Received: from zmproxy.tng.vnc.biz ([127.0.0.1]) by localhost (zmproxy.tng.vnc.biz [127.0.0.1]) (amavis, port 10032) with ESMTP id zVWT51JsfnWK; Thu, 19 Nov 2026 21:45:12 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by zmproxy.tng.vnc.biz (Postfix) with ESMTP id BCBAC1FBD50; Thu, 19 Nov 2026 21:45:12 +0100 (CET) DKIM-Filter: OpenDKIM Filter v2.10.3 zmproxy.tng.vnc.biz BCBAC1FBD50 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tngtech.com; s=B14491C6-869D-11EB-BB6C-8DD33D883B31; t=1795121112; bh=/1Grfte5MGcWY78IHhh2XGB835lwhfLoqukaCJMf9Lg=; h=From:To:Subject:Date:Message-ID:MIME-Version; b=HLMomBvqlAPjdF1iY9gq7xRZRXB6NsQQBl1ZrVwmVw36czDbXs2ux0iqfYAfsz/gR sMaGOvaEj/q+poaoJ9pkIUGQhZKWBbLJ5P7AyMZJJnEa3ohrF7xztrsmVZF7ZhrjqD 9ju5yH2X3Kk9eGh/tfMEuWn7pF3ElA/OfV+rQhxAyLsPH72jxPSWY4j/+IiRL/0u0B wEzUyZMP6OCDvYug01c1EQc34uNigAvh8VLC6KHnXu8FppArMcIsLi0XNgzNr5K+sM ShLE7tTFYiWlOnjBxmm0KIURj02qutmU5jwgQCpx0CRkS3FvOMkpbiBBQyDHZdmzai d9RRFL/5sHIkA== X-Virus-Scanned: amavis at zmproxy.tng.vnc.biz Received: from zmproxy.tng.vnc.biz ([127.0.0.1]) by localhost (zmproxy.tng.vnc.biz [127.0.0.1]) (amavis, port 10026) with ESMTP id 4xWlmrBG_DDL; Thu, 19 Nov 2026 21:45:12 +0100 (CET) Received: from luis-Precision-5480.. (ipservice-092-209-236-032.092.209.pools.vodafone-ip.de [92.209.236.32]) by zmproxy.tng.vnc.biz (Postfix) with ESMTPSA id 414D81FBD38; Thu, 19 Nov 2026 21:45:12 +0100 (CET) From: Luis Augenstein To: nathan@kernel.org, nsc@kernel.org Cc: linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, gregkh@linuxfoundation.org, kstewart@linuxfoundation.org, maximilian.huber@tngtech.com, Luis Augenstein Subject: [PATCH v6 0/7] kbuild: record cmd-prereqs in .cmd files Date: Thu, 19 Nov 2026 21:44:46 +0100 Message-ID: <20261119204457.9968-1-luis.augenstein@tngtech.com> X-Mailer: git-send-email 2.43.0 X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable KernelSbom (scripts/sbom/sbom.py) reconstructs the kernel build graph from the .cmd files written by Kbuild. For most build outputs, KernelSbom must infer the build inputs from the shell command stored in savedcmd_*. This requires command-specific parsers, which add unnecessary complexity and are difficult to maintain. Kbuild already knows these inputs as $(real-prereqs), the non-phony prerequisites of the Make rule. This series records them in .cmd files using a new metadata field: make_prereqs_ :=3D KernelSbom is updated to read the new field instead of parsing the shell command, and the obsolete command-specific parsers are removed. The issue of manually parsing shell commands was originally mentioned in https://lore.kernel.org/r/a01233b9-23a2-4666-91ed-f1cf030dcb9f@tngtech= .com. This series is a proposal to address that. Cmd writers and consumers =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The new field should not affect existing behavior. Below is a summary of what I checked. .cmd files are primarily written by cmd_and_savecmd and cmd_and_fixdep. Both writers add make_prereqs. The following rules append information to existing .cmd files. None of these is affected by the new field: - scripts/Makefile.lib: cmd_gen_objtooldep only appends new rule target: $(wildcard ./tools/objtool/objtool) - scripts/Makefile.build: gen_symversions only appends comments #SYMVER - scripts/Makefile.lib: cmd_save_c_flags only appends a variable saved_c_flags_ :=3D - rust/Makefile: cmd_gendwarfksyms only appends comments #SYMVER - kernel/Makefile: kheaders_data_dep appends a deps_* block deps_ :=3D . This series updates the following .cmd file consumers: - scripts/sbom/sbom/cmd_graph/cmd_file.py: CmdFile.create() is rewritten to parse the fields by name, including the new make_prereqs field. - scripts/make_fit.py: process_dtb() previously read and split the complete .cmd file to find the fdtoverlay command. It is changed to read only the first line, so it keeps working once the new field is introduced. The remaining consumers are not affected: - Kbuild Makefiles: -include reads .cmd files as Makefile fragments. The new field is an ordinary variable assignment not used in any dependency rule. - scripts/Makefile.thinlto: the saved_c_flags_* lookup only reads saved_c_flags_ :=3D . - scripts/mod/modpost.c: extract_crcs_for_object() only reads comments #SYMVER . - scripts/mod/sumversion.c: parse_source_files() only reads source_ :=3D and deps_ :=3D . - scripts/clang-tools/gen_compile_commands.py: main() only reads the first line containing savedcmd_ :=3D . - scripts/generate_builtin_ranges.awk and scripts/verify_builtin_ranges.awk: get_module_info() only reads the first line and searches the saved command for DKBUILD_MODFILE or RUST_MODFILE. Alternative: extending the deps_* field =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D An alternative to the new make_prereqs field is to merge $(real-prereqs) into the existing deps_* field for both cmd_and_savecmd and cmd_and_fixde= p. This would avoid a new field and provide one dependency list. However, it would broaden the meaning of deps_* from dependencies discovered through generated .d files to include prerequisites already declared to Make. This increases the risk of undesired side effects. The writers listed above would need to ensure that each .cmd file contains only one deps_* block. In particular, kheaders_data_dep would need to merge its header dependencies into the block written by cmd_and_savecmd instead of appending a second deps_* block. cmd_and_fixdep, or fixdep.c depending on the implementation, would likewise need to merge $(real-prereqs) with the generated dependencies. Apart from KernelSbom, scripts/mod/sumversion.c is the only consumer that explicitly reads deps_*. It uses selected dependency files to calculate a module's srcversion. Extending deps_* could change this module metadata. Such an unrelated change should probably be avoided. Kbuild Makefiles also consume deps_* by including .cmd files. This should not be an issue though, because $(real-prereqs) are already present on the original target rule, so repeating them through deps_* should not change rebuild behavior. SBOM comparison =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D I compared the SBOMs before and after the series for the same kernel builds on x86 and arm64 allmodconfig. They are ~99% the same. The following files are newly included in the SBOM because the manual parsers did not cover them. - Linker scripts: *.lds.S sources and the generated *.lds scripts. The old tool never extracted -T arguments, so the whole linker-script chain was missing. - The intermediate vmlinux link steps: vmlinux.o, .vmlinux.export.c/.o, the realmode pasyms.h. Previously vmlinux.a was hardcoded as an input of scripts/link-vmlinux.sh. Now the real chain vmlinux link -> vmlinux.o -> vmlinux.a is included. - Per-module .mod object lists: the old tool expanded the .mod file contents in place and recorded the objects directly as link inputs, without explicitly including the .mod file in the SBOM. - Generated headers: the drm/xe and msm register tables, the aic7xxx sequence tables, the x32 syscall header, and module.lds.h. The old tool found a header only where a .c file actually includes it via the dep file; the new tool records the full make view, including headers that are never included at all. - Config and header archives: kernel/config_data.gz and kernel/kheaders_data.tar.xz. The old tool missed them because their generation commands had no matching parser. - Devicetree overlay sources: the .dtso files are now recorded as inputs of the dtb/dtbo build steps. The old tool saw only the temporary files those steps produce, never the sources themselves. The following files are removed: - dtc preprocessing scratch files: .dts.tmp, .d.pre.tmp, .d.dtc.tmp. The old tool recorded these temporary build by-products as inputs; the new tool records the real .dtso source instead. This is a cleanup, not a loss. - Host tooling: the rust toolchain .md docs, the gen_smb1/2_mapping perl generators, and the dtc host tool's libfdt headers. Deliberately excluded. The SBOM should track what ends up in the image = or modules, not the tooling that produces it. Changes in v6: - strip leading ./ prefixes from deps_* paths in cmd-prereqs. Changes in v5: - exclude .awk files from the SBOM - add an explicit include list for genuine kernel inputs that are covered by the general exclusion rules, for example genuine kernel inpu= t files under scripts/ and tools/ directories. - remove Assisted-by: LLM trailers from the commit messages Changes in v4: - fix a bug where excluding src tree dependencies only worked when the object tree was a direct child of the source tree - fix a bug where non-normalized paths containing '..' segments bypassed the exclude list.=20 Changes in v3: - split the changes into individual patches - include the sbom changes directly in this series instead of deferring i= t - add the SBOM comparison section to the cover letter - Assisted-by trailer follows Documentation/process/coding-assistants.rst Changes in v2: - filter depfile entries from make_prereqs in fixdep.c to ensure make_prereqs is consistent between clean and incremental builds. Filtering $(deps_$@) from previous builds is still required to not include deps that got removed in the current depfile. --- Luis Augenstein (7): scripts/make_fit: limit .cmd file parsing to the savedcmd_ line kbuild: refactor fixdep to collect dependencies before printing scripts/sbom: skip make_prereqs lines when parsing .cmd files kbuild: record cmd-prereqs in .cmd files scripts/sbom: parse make_prereqs from .cmd files scripts/sbom: use make_prereqs to determine build inputs scripts/sbom: remove the command-specific parsers Documentation/tools/sbom/sbom.rst | 23 +- scripts/Kbuild.include | 10 +- scripts/basic/fixdep.c | 163 +++++- scripts/make_fit.py | 2 +- scripts/sbom/sbom/cmd_graph/cmd_file.py | 149 ++--- scripts/sbom/sbom/cmd_graph/cmd_graph.py | 6 +- scripts/sbom/sbom/cmd_graph/cmd_graph_node.py | 57 +- .../sbom/sbom/cmd_graph/dependency_filter.py | 109 ++++ .../sbom/cmd_graph/hardcoded_dependencies.py | 1 - .../cmd_graph/savedcmd_parser/__init__.py | 6 - .../command_parser_registry.py | 516 ------------------ .../savedcmd_parser/command_splitter.py | 128 ----- .../savedcmd_parser/savedcmd_parser.py | 67 --- .../cmd_graph/savedcmd_parser/tokenizer.py | 92 ---- scripts/sbom/sbom/config.py | 14 - scripts/sbom/sbom/spdx_graph/kernel_file.py | 63 ++- .../tests/cmd_graph/test_savedcmd_parser.py | 468 ---------------- 17 files changed, 376 insertions(+), 1498 deletions(-) create mode 100644 scripts/sbom/sbom/cmd_graph/dependency_filter.py delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/__init__.= py delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/command_p= arser_registry.py delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/command_s= plitter.py delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/savedcmd_= parser.py delete mode 100644 scripts/sbom/sbom/cmd_graph/savedcmd_parser/tokenizer= .py delete mode 100644 scripts/sbom/tests/cmd_graph/test_savedcmd_parser.py --=20 2.43.0