[PATCH v2 0/1] kbuild: record real-prereqs in .cmd files

[PATCH v2 0/1] kbuild: record real-prereqs in .cmd files

[Luis Augenstein]

KernelSbom (scripts/sbom/sbom.py) reconstructs the kernel build graph
from the .cmd files written by Kbuild. For most build outputs,
KernelSbom must infer the build inputs from the shell command stored in
savedcmd_*. This requires command-specific parsers, which add
unnecessary complexity and are difficult to maintain.

Kbuild already knows these inputs as $(real-prereqs), the non-phony
prerequisites of the Make rule. This patch records them in .cmd files
using a new metadata field:

    make_prereqs_<target> := <prerequisites>

KernelSbom will be able to read the new field instead of parsing the
shell command. This submission only records the field. Support for using
it in KernelSbom will follow separately.

The issue of manually parsing shell commands was originally mentioned in
https://lore.kernel.org/r/a01233b9-23a2-4666-91ed-f1cf030dcb9f@tngtech.com.
This patch is a proposal to address that.

Cmd writers and consumers
=========================

The new field should not affect existing behavior. 
Below is a summary of what I checked.

.cmd files are primarily written by cmd_and_savecmd and cmd_and_fixdep.
This patch extends both writers to add make_prereqs_*.

The following rules append information to existing .cmd files.
None of these is affected by the new field:

- scripts/Makefile.lib: cmd_gen_objtooldep only appends new rule
  target: $(wildcard ./tools/objtool/objtool)
- scripts/Makefile.build: gen_symversions only appends comments
  #SYMVER <name> <crc>
- scripts/Makefile.lib: cmd_save_c_flags only appends a variable
  saved_c_flags_<target> := <flags>
- rust/Makefile: cmd_gendwarfksyms only appends comments
  #SYMVER <name> <crc>
- kernel/Makefile: kheaders_data_dep appends a deps_* block
  deps_<target> := <headers>.

This patch updates the following .cmd file consumers to account for the
new field:

- scripts/sbom/sbom/cmd_graph/cmd_file.py: CmdFile.create() parses
  savedcmd_<target>, source_<target>, and deps_<target>. It is updated
  to ignore make_prereqs_<target> for now. Parsing the new field will
  follow in a separate patch.
- scripts/make_fit.py: process_dtb() previously read and split the
  complete .cmd file to find the fdtoverlay command. It is changed to
  read only savedcmd_<target>.

The remaining consumers are not affected:

- Kbuild Makefiles: -include reads .cmd files as Makefile fragments. The
  new field is an ordinary variable assignment not used in any dependency
  rule.
- scripts/Makefile.thinlto: the saved_c_flags_* lookup only reads
  saved_c_flags_<target> := <flags>.
- scripts/mod/modpost.c: extract_crcs_for_object() only reads comments
  #SYMVER <name> <crc>.
- scripts/mod/sumversion.c: parse_source_files() only reads
  source_<target> := <source> and deps_<target> := <dependencies>.
- scripts/clang-tools/gen_compile_commands.py: main() only reads the
  first line containing savedcmd_<target> := <command>.
- scripts/generate_builtin_ranges.awk and
  scripts/verify_builtin_ranges.awk: get_module_info() only reads the
  first line and searches the saved command for DKBUILD_MODFILE or
  RUST_MODFILE.

Alternative: extending the deps_* field
=======================================

An alternative to the new make_prereqs_* field is to merge $(real-prereqs)
into the existing deps_* field for both cmd_and_savecmd and cmd_and_fixdep.
This would avoid a new field and provide one dependency list. However, it
would broaden the meaning of deps_* from dependencies discovered through
generated .d files to include prerequisites already declared to Make.
This increases the risk of undesired side effects.

The writers listed above would need to ensure that each .cmd file
contains only one deps_* block. In particular, kheaders_data_dep would
need to merge its header dependencies into the block written by
cmd_and_savecmd instead of appending a second deps_* block.
cmd_and_fixdep, or fixdep.c depending on the implementation, would
likewise need to merge $(real-prereqs) with the generated dependencies.

Apart from KernelSbom, scripts/mod/sumversion.c is the only consumer
that explicitly reads deps_*. It uses selected dependency files to
calculate a module's srcversion. Extending deps_* could change this
module metadata. Such an unrelated change should probably be avoided.

Kbuild Makefiles also consume deps_* by including .cmd files. This
should not be an issue though, because $(real-prereqs) are already
present on the original target rule, so repeating them through deps_*
should not change rebuild behavior.

Changes in v2:
- filter depfile entries from make_prereqs_ in fixdep.c to ensure
  make_prereqs_* is consistent between clean and incremental builds.
  Filtering $(deps_$@) from previous builds is still required to not
  include deps that got removed in the current depfile.
---

Luis Augenstein (1):
  kbuild: record real-prereqs in .cmd files

 scripts/Kbuild.include                  |   9 +-
 scripts/basic/fixdep.c                  | 159 ++++++++++++++++++++----
 scripts/make_fit.py                     |   2 +-
 scripts/sbom/sbom/cmd_graph/cmd_file.py |   4 +
 4 files changed, 148 insertions(+), 26 deletions(-)

-- 
2.43.0

[PATCH v2 1/1] kbuild: record real-prereqs in .cmd files

[Luis Augenstein]

Record $(real-prereqs), the non-phony prerequisites of the target, in a
new metadata field:

    make_prereqs_<target> := <prerequisites>

Write the field from both cmd_and_savecmd and cmd_and_fixdep.

Update scripts/make_fit.py to read only savedcmd_* instead of parsing the
complete .cmd file.

Ignore make_prereqs_* in KernelSbom.

Link: https://lore.kernel.org/r/a01233b9-23a2-4666-91ed-f1cf030dcb9f@tngtech.com
Assisted-by: Cursor:GPT-5.6 Sol
Co-developed-by: Maximilian Huber <maximilian.huber@tngtech.com>
Signed-off-by: Maximilian Huber <maximilian.huber@tngtech.com>
Signed-off-by: Luis Augenstein <luis.augenstein@tngtech.com>
---
 scripts/Kbuild.include                  |   9 +-
 scripts/basic/fixdep.c                  | 159 ++++++++++++++++++++----
 scripts/make_fit.py                     |   2 +-
 scripts/sbom/sbom/cmd_graph/cmd_file.py |   4 +
 4 files changed, 148 insertions(+), 26 deletions(-)

diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 8c311b997e2..6daa244ba0e 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -181,6 +181,9 @@ endif
 # (needed for the shell)
 make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
 
+# prerequisites to record in .cmd files, excluding those covered in deps_*
+cmd-prereqs = $(call escsq,$(filter-out $(deps_$@), $(real-prereqs)))
+
 # Find any prerequisites that are newer than target or that do not exist.
 # PHONY targets skipped in both cases.
 # If there is no prerequisite other than phony targets, $(newer-prereqs) becomes
@@ -198,14 +201,16 @@ if_changed = $(if $(if-changed-cond),$(cmd_and_savecmd),@:)
 
 cmd_and_savecmd =                                                            \
 	$(cmd);                                                              \
-	printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd
+	printf '%s\n\n%s\n' 'savedcmd_$@ := $(make-cmd)'                    \
+		'make_prereqs_$@ := $(cmd-prereqs)' > $(dot-target).cmd
 
 # Execute the command and also postprocess generated .d dependencies file.
 if_changed_dep = $(if $(if-changed-cond),$(cmd_and_fixdep),@:)
 
 cmd_and_fixdep =                                                             \
 	$(cmd);                                                              \
-	$(objtree)/scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).cmd;\
+	$(objtree)/scripts/basic/fixdep $(depfile) $@ '$(make-cmd)'          \
+		'$(cmd-prereqs)' > $(dot-target).cmd;                        \
 	rm -f $(depfile)
 
 # Usage: $(call if_changed_rule,foo)
diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c
index cdd5da7e009..e644c1590a9 100644
--- a/scripts/basic/fixdep.c
+++ b/scripts/basic/fixdep.c
@@ -62,15 +62,17 @@
  *
  * It is invoked as
  *
- *   fixdep <depfile> <target> <cmdline>
+ *   fixdep <depfile> <target> <cmdline> <prereqs>
  *
  * and will read the dependency file <depfile>
  *
  * The transformed dependency snipped is written to stdout.
  *
- * It first generates a line
+ * It first generates the lines
  *
- *   savedcmd_<target> = <cmdline>
+ *   savedcmd_<target> := <cmdline>
+ *
+ *   make_prereqs_<target> := <prereqs_without_deps>
  *
  * and then basically copies the .<target>.d file to stdout, in the
  * process filtering out the dependency on autoconf.h and adding
@@ -103,7 +105,7 @@
 
 static void usage(void)
 {
-	fprintf(stderr, "Usage: fixdep <depfile> <target> <cmdline>\n");
+	fprintf(stderr, "Usage: fixdep <depfile> <target> <cmdline> <prereqs>\n");
 	exit(1);
 }
 
@@ -114,6 +116,17 @@ struct item {
 	char		name[];
 };
 
+struct dep_item {
+	struct dep_item	*next;
+	char		name[];
+};
+
+struct dep_info {
+	char			*source;
+	struct dep_item		*deps;
+	struct dep_item		*last_dep;
+};
+
 #define HASHSZ 256
 static struct item *config_hashtab[HASHSZ], *file_hashtab[HASHSZ];
 
@@ -163,16 +176,42 @@ static bool in_hashtable(const char *name, int len, struct item *hashtab[])
 	return false;
 }
 
+/*
+ * Append a dependency or include/config/<SYM> path to the info linked list.
+ */
+static void dep_info_add(struct dep_info *info, const char *name, int len)
+{
+	struct dep_item *dep = xmalloc(sizeof(*dep) + len + 1);
+
+	memcpy(dep->name, name, len);
+	dep->name[len] = '\0';
+	dep->next = NULL;
+
+	if (info->last_dep)
+		info->last_dep->next = dep;
+	else
+		info->deps = dep;
+	info->last_dep = dep;
+}
+
 /*
  * Record the use of a CONFIG_* word.
  */
-static void use_config(const char *m, int slen)
+static void use_config(const char *m, int slen, struct dep_info *info)
 {
 	if (in_hashtable(m, slen, config_hashtab))
 		return;
 
-	/* Print out a dependency path from a symbol name. */
-	printf("    $(wildcard include/config/%.*s) \\\n", slen, m);
+	/* Build a dependency path from a symbol name. */
+	static const char config_path[] = "include/config/";
+	char *path;
+	int path_len = sizeof(config_path) - 1 + slen;
+
+	path = xmalloc(path_len);
+	memcpy(path, config_path, sizeof(config_path) - 1);
+	memcpy(path + sizeof(config_path) - 1, m, slen);
+	dep_info_add(info, path, path_len);
+	free(path);
 }
 
 /* test if s ends in sub */
@@ -186,7 +225,12 @@ static int str_ends_with(const char *s, int slen, const char *sub)
 	return !memcmp(s + slen - sublen, sub, sublen);
 }
 
-static void parse_config_file(const char *p)
+/*
+ * Scan dependency p for CONFIG_ words, map each to an
+ * include/config/ path and append to the dep_info list if not
+ * already included.
+ */
+static void parse_config_file(const char *p, struct dep_info *info)
 {
 	const char *q, *r;
 	const char *start = p;
@@ -205,7 +249,7 @@ static void parse_config_file(const char *p)
 		else
 			r = q;
 		if (r > p)
-			use_config(p, r - p);
+			use_config(p, r - p, info);
 		p = q;
 	}
 }
@@ -253,13 +297,9 @@ static int is_no_parse_file(const char *s, int len)
 	       str_ends_with(s, len, ".so");
 }
 
-/*
- * Important: The below generated source_foo.o and deps_foo.o variable
- * assignments are parsed not only by make, but also by the rather simple
- * parser in scripts/mod/sumversion.c.
- */
-static void parse_dep_file(char *p, const char *target)
+static struct dep_info parse_dep_file(char *p)
 {
+	struct dep_info info = {};
 	bool saw_any_target = false;
 	bool is_target = true;
 	bool is_source = false;
@@ -375,13 +415,12 @@ static void parse_dep_file(char *p, const char *target)
 			 */
 			if (!saw_any_target) {
 				saw_any_target = true;
-				printf("source_%s := %s\n\n", target, p);
-				printf("deps_%s := \\\n", target);
+				info.source = xstrdup(p);
 				need_parse = true;
 			}
 		} else if (!is_ignored_file(p, q - p) &&
 			   !in_hashtable(p, q - p, file_hashtab)) {
-			printf("  %s \\\n", p);
+			dep_info_add(&info, p, q - p);
 			need_parse = true;
 		}
 
@@ -389,7 +428,7 @@ static void parse_dep_file(char *p, const char *target)
 			void *buf;
 
 			buf = read_file(p);
-			parse_config_file(buf);
+			parse_config_file(buf, &info);
 			free(buf);
 		}
 
@@ -403,26 +442,100 @@ static void parse_dep_file(char *p, const char *target)
 		exit(1);
 	}
 
+	return info;
+}
+
+static void free_dep_info(struct dep_info *info)
+{
+	struct dep_item *dep, *next;
+
+	for (dep = info->deps; dep; dep = next) {
+		next = dep->next;
+		free(dep);
+	}
+	free(info->source);
+}
+
+/*
+ * Important: The below generated source_foo.o and deps_foo.o variable
+ * assignments are parsed not only by make, but also by the rather simple
+ * parser in scripts/mod/sumversion.c.
+ */
+static void print_dep_file(const char *target, const struct dep_info *info)
+{
+	static const char config_path[] = "include/config/";
+	const struct dep_item *dep;
+
+	printf("source_%s := %s\n\n", target, info->source);
+	printf("deps_%s := \\\n", target);
+
+	for (dep = info->deps; dep; dep = dep->next) {
+		if (!strncmp(dep->name, config_path, sizeof(config_path) - 1))
+			printf("    $(wildcard %s) \\\n", dep->name);
+		else
+			printf("  %s \\\n", dep->name);
+	}
+
 	printf("\n%s: $(deps_%s)\n\n", target, target);
 	printf("$(deps_%s):\n", target);
 }
 
+static void print_make_prereqs(const char *target, const char *prereqs,
+			       const struct dep_info *info)
+{
+	struct item *seen_prereqs_hashtab[HASHSZ] = {};
+	const struct dep_item *dep;
+	const char *p = prereqs;
+
+	for (dep = info->deps; dep; dep = dep->next)
+		in_hashtable(dep->name, strlen(dep->name),
+			     seen_prereqs_hashtab);
+
+	printf("make_prereqs_%s :=", target);
+
+	while (*p) {
+		const char *start;
+		int len;
+
+		while (*p == ' ' || *p == '\t')
+			p++;
+		if (!*p)
+			break;
+
+		start = p;
+		while (*p && *p != ' ' && *p != '\t')
+			p++;
+		len = p - start;
+
+		if (!in_hashtable(start, len, seen_prereqs_hashtab))
+			printf(" %.*s", len, start);
+	}
+
+	printf("\n\n");
+}
+
 int main(int argc, char *argv[])
 {
-	const char *depfile, *target, *cmdline;
+	const char *depfile, *target, *cmdline, *prereqs;
+	struct dep_info info;
 	void *buf;
 
-	if (argc != 4)
+	if (argc != 5)
 		usage();
 
 	depfile = argv[1];
 	target = argv[2];
 	cmdline = argv[3];
+	prereqs = argv[4];
+
+	buf = read_file(depfile);
+	info = parse_dep_file(buf);
 
 	printf("savedcmd_%s := %s\n\n", target, cmdline);
+	print_make_prereqs(target, prereqs, &info);
+	print_dep_file(target, &info);
 
-	buf = read_file(depfile);
-	parse_dep_file(buf, target);
+	free_dep_info(&info);
 	free(buf);
 
 	fflush(stdout);
diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 15ba26974fd..346e8a7ec12 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -288,7 +288,7 @@ def process_dtb(fname, args):
         path, basename = os.path.split(fname)
         cmd_fname = os.path.join(path, f'.{basename}.cmd')
         with open(cmd_fname, 'r', encoding='ascii') as inf:
-            cmd = inf.read()
+            cmd = inf.readline()
 
         if 'scripts/dtc/fdtoverlay' in cmd:
             # This depends on the structure of the composite DTB command
diff --git a/scripts/sbom/sbom/cmd_graph/cmd_file.py b/scripts/sbom/sbom/cmd_graph/cmd_file.py
index dcd63e284a3..08819b4d117 100644
--- a/scripts/sbom/sbom/cmd_graph/cmd_file.py
+++ b/scripts/sbom/sbom/cmd_graph/cmd_file.py
@@ -50,6 +50,10 @@ class CmdFile:
         with open(cmd_file_path, "rt", encoding="utf-8") as f:
             lines = [line.strip() for line in f.readlines() if line.strip() != "" and not line.startswith("#")]
 
+        # make_prereqs_* is recorded for future use. Ignore it for now to
+        # preserve the existing parser behavior.
+        lines = [line for line in lines if not line.startswith("make_prereqs_")]
+
         # savedcmd
         match = SAVEDCMD_PATTERN.match(lines[0] if lines else "")
         if match is None:
-- 
2.43.0

Re: [PATCH v2 1/1] kbuild: record real-prereqs in .cmd files

[Nicolas Schier]

First of all, sorry for the very long delay.  I finally had a first view
and tests on the patch, but I haven't tested in depth yet.  Some
first-round feedback:

> Record $(real-prereqs), the non-phony prerequisites of the target, in a
> new metadata field:
> 
>     make_prereqs_<target> := <prerequisites>
> 
> Write the field from both cmd_and_savecmd and cmd_and_fixdep.
>
> Update scripts/make_fit.py to read only savedcmd_* instead of parsing the
> complete .cmd file.

This sounds like a good thing; but I can't find that in the diff.

While at it, you changed the handling of deps in fixdep -- I think in a
good way.  I'd like to see this in three steps:

  * Limit make_fit's parsing to savedcmd_
  * Update fixdeps handling of 'deps_'
  * Add 'make_prereqs_<target>' support to Kbuild.include and fixdep

> Ignore make_prereqs_* in KernelSbom.

I know this makes sense to fit the every-interface-needs-a-user rule,
but I don't like to add a dummy consumer here to match that rule.

Why do you defer the actual implementation in the SBOM scripts?  From my
point of view, the introduction of 'make_prereqs_<target>' looks good
and I don't see a reason for waiting with the corresponding SBOM
implementation.

> 
> Link: https://lore.kernel.org/r/a01233b9-23a2-4666-91ed-f1cf030dcb9f@tngtech.com
> Assisted-by: Cursor:GPT-5.6 Sol

The new rule wants:

    Assisted-by: LLM 

instead, cp. https://git.kernel.org/stable/c/816d9992d9ed434ec52cfbd63080d518e535a41b

> Co-developed-by: Maximilian Huber <maximilian.huber@tngtech.com>
> Signed-off-by: Maximilian Huber <maximilian.huber@tngtech.com>
> Signed-off-by: Luis Augenstein <luis.augenstein@tngtech.com>
> ---
>  scripts/Kbuild.include                  |   9 +-
>  scripts/basic/fixdep.c                  | 159 ++++++++++++++++++++----
>  scripts/make_fit.py                     |   2 +-
>  scripts/sbom/sbom/cmd_graph/cmd_file.py |   4 +
>  4 files changed, 148 insertions(+), 26 deletions(-)
>
> diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
> index 8c311b997e24..6daa244ba0ef 100644
> --- a/scripts/Kbuild.include
> +++ b/scripts/Kbuild.include
> @@ -181,6 +181,9 @@ endif
>  # (needed for the shell)
>  make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
>  
> +# prerequisites to record in .cmd files, excluding those covered in deps_*
> +cmd-prereqs = $(call escsq,$(filter-out $(deps_$@), $(real-prereqs)))
> +
>  # Find any prerequisites that are newer than target or that do not exist.
>  # PHONY targets skipped in both cases.
>  # If there is no prerequisite other than phony targets, $(newer-prereqs) becomes
> @@ -198,14 +201,16 @@ if_changed = $(if $(if-changed-cond),$(cmd_and_savecmd),@:)
>  
>  cmd_and_savecmd =                                                            \
>  	$(cmd);                                                              \
> -	printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd
> +	printf '%s\n\n%s\n' 'savedcmd_$@ := $(make-cmd)'                    \
> +		'make_prereqs_$@ := $(cmd-prereqs)' > $(dot-target).cmd

Just bike-shedding:  I think the main points are tiny bit easier to read
this way:

 cmd_and_savecmd =							\
 	$(cmd);								\
	printf '%s\n'							\
		'savedcmd_$@ := $(make-cmd)' ''				\
		'make_prereqs_$@ := $(cmd-prereqs)' > $(dot-target).cmd

Take it or just keep your version, as you like.




All in all, this looks good to me. Thanks

-- 
Nicolas

Re: [PATCH v2 1/1] kbuild: record real-prereqs in .cmd files

[Luis Augenstein]

[-- Attachment #1.1: Type: text/plain, Size: 1791 bytes --]

On 10/7/26 16:46, Nicolas Schier wrote:
>> Record $(real-prereqs), the non-phony prerequisites of the target, in a
>> new metadata field:
>>
>>      make_prereqs_<target> := <prerequisites>
>>
>> Write the field from both cmd_and_savecmd and cmd_and_fixdep.
>>
>> Update scripts/make_fit.py to read only savedcmd_* instead of parsing the
>> complete .cmd file.
> 
> This sounds like a good thing; but I can't find that in the diff.

It's the first section in the diff in scripts/Kbuild.include.
The original sentence might have been a bit unclear though.
$(cmd-prereqs) is the new variable that is actually used by
cmd_and_savecmd and cmd_and_fixdep. However, since $(cmd-prereqs) is
just $(real-prereqs) with $(deps_) filtered out, I omitted this
intermediate step in the description. I will rephrase it in v3 to make
it more clear.

> Why do you defer the actual implementation in the SBOM scripts?  From my
> point of view, the introduction of 'make_prereqs_<target>' looks good
> and I don't see a reason for waiting with the corresponding SBOM
> implementation.

I thought it would be a sensible boundary to first change Kbuild and
then separately update the SBOM tool to use the new information the
.cmd files offer. Also, when I posted this series, I simply didn't
have the SBOM patches finished yet. By now they are ready for review
as well, so I will include them in v3 of this series.

thanks, Luis

-- 
Luis Augenstein * luis.augenstein@tngtech.com * +4915225275761
TNG Technology Consulting GmbH, Beta-Str. 13, 85774 Unterföhring
Geschäftsführer: Dr. Robert Dahlke, Thomas Endres, Moritz Prinz
Aufsichtsratsvorsitzender: Henrik Klagges
Sitz: Unterföhring * Registergericht: Amtsgericht München * 
Handelsregisternummer: HRB 135082


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]