[PATCH] clang-tools: Decode dollar escaping in compile commands

[PATCH] clang-tools: Decode dollar escaping in compile commands

[houtinghang]

Kbuild doubles dollar signs when saving commands in .cmd files, but
process_line() only decodes $(pound). Consequently, the compilation
database retains doubled dollar signs in compiler arguments. It also
corrupts a literal $(pound), saved as $$(pound), into $#.

Decode $$ and $(pound) in a single pass, matching how Make reads the
saved command. A single pass avoids interpreting a decoded dollar sign
as the start of another escape.

Add CLI regression coverage for ordinary text, dollar signs, hash signs
and adjacent escapes. Four of the six cases fail before this change;
all six pass after it. The expected values were also checked against
GNU Make using scripts/Kbuild.include.

Fixes: b30204640192 ("scripts: add a tool to produce a compile_commands.json file")
Assisted-by: LLM
Signed-off-by: houtinghang <ue081723@gmail.com>
---
Testing on Ubuntu with GCC 13.3.0, Clang/clangd 18.1.3 and Make 4.3:
- x86-64 GCC, x86-64 Clang and ARM64 Clang defconfig builds of the kernel
  image and configured modules passed without compiler warnings.
- All three native make compile_commands.json targets passed.
- 17,084 ordinary compilation database entries match the original tool.
- 432 Kbuild/CLI/shell/compiler cases produced identical objects.
- 259 mixed escape cases agree with Make for each compiler test run.
- 22 clangd macro checks and real x86/ARM64 kernel checks passed.
- The original tool fails the integration and clangd negative controls.
- Six CLI cases pass on Linux/Python 3.12.3 and Windows/Python 3.13.

AI disclosure: Codex identified the issue, wrote the fix and regression
test, verified CLI output and Make semantics, reviewed the diff, and
drafted this patch after a request to find another Linux kernel bug.

 scripts/clang-tools/gen_compile_commands.py   | 10 ++--
 .../clang-tools/gen_compile_commands_test.py  | 47 +++++++++++++++++++
 2 files changed, 52 insertions(+), 5 deletions(-)
 create mode 100644 scripts/clang-tools/gen_compile_commands_test.py

diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
index 8d14b81..d11d15b 100755
--- a/scripts/clang-tools/gen_compile_commands.py
+++ b/scripts/clang-tools/gen_compile_commands.py
@@ -166,11 +166,11 @@ def process_line(root_directory, command_prefix, file_path):
         ValueError: Could not find the extracted file based on file_path and
             root_directory or file_directory.
     """
-    # The .cmd files are intended to be included directly by Make, so they
-    # escape the pound sign '#' as '$(pound)'. The compile_commands.json file
-    # is not interepreted by Make, so this code replaces the escaped version
-    # with '#'.
-    prefix = command_prefix.replace('$(pound)', '#')
+    # Undo the escaping performed by make-cmd in scripts/Kbuild.include.
+    # Decode both escapes in one pass to preserve a literal '$(pound)'.
+    prefix = re.sub(r'\$\$|\$\(pound\)',
+                    lambda match: '$' if match.group() == '$$' else '#',
+                    command_prefix)
 
     # Return the canonical path, eliminating any symbolic links encountered in the path.
     abs_path = os.path.realpath(os.path.join(root_directory, file_path))
diff --git a/scripts/clang-tools/gen_compile_commands_test.py b/scripts/clang-tools/gen_compile_commands_test.py
new file mode 100644
index 0000000..fe259af
--- /dev/null
+++ b/scripts/clang-tools/gen_compile_commands_test.py
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Command-line regression tests for gen_compile_commands.py."""
+
+import json
+from pathlib import Path
+import subprocess
+import sys
+import tempfile
+import unittest
+
+
+SCRIPT = Path(__file__).with_name("gen_compile_commands.py")
+
+
+class TestCompileCommands(unittest.TestCase):
+    def test_make_escaping(self):
+        cases = [
+            ("plain", "plain"),
+            ("$$value", "$value"),
+            ("$$$$", "$$"),
+            ("$(pound)value", "#value"),
+            ("$$(pound)", "$(pound)"),
+            ("$$$(pound)", "$#"),
+        ]
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            (root / "test.c").write_text("int test;\n", encoding="utf-8")
+            for escaped, original in cases:
+                with self.subTest(escaped=escaped):
+                    prefix = "gcc -DVALUE='\"{}\"' -c -o test.o "
+                    (root / ".test.o.cmd").write_text(
+                        "savedcmd_test.o := " + prefix.format(escaped)
+                        + "test.c\n", encoding="utf-8")
+                    output = root / "compile_commands.json"
+                    subprocess.run(
+                        [sys.executable, str(SCRIPT), "-d", str(root),
+                         "-o", str(output)], check=True, capture_output=True)
+                    entries = json.loads(output.read_text(encoding="utf-8"))
+                    self.assertEqual(len(entries), 1)
+                    self.assertEqual(entries[0]["command"],
+                                     prefix.format(original) + "test.c")
+
+
+if __name__ == "__main__":
+    unittest.main()

base-commit: fe2ec83746e501645709761605c2464a44fd2929
-- 
2.52.0.windows.1

Re: [PATCH] clang-tools: Decode dollar escaping in compile commands

[Nathan Chancellor]

On 2026-09-23 23:10 +0800, houtinghang wrote:
> Kbuild doubles dollar signs when saving commands in .cmd files, but
> process_line() only decodes $(pound). Consequently, the compilation
> database retains doubled dollar signs in compiler arguments. It also

While I agree that '$$' should be handled like '$(pound)', I suspect it
is likely not a problem in practice since most commands that appear to
use '$$' are for generated files, which should not matter for
compile_commands.json but I might be mistaken.

> corrupts a literal $(pound), saved as $$(pound), into $#.

How would this happen, aside from the contrived test?

> Decode $$ and $(pound) in a single pass, matching how Make reads the
> saved command. A single pass avoids interpreting a decoded dollar sign
> as the start of another escape.
> 
> Add CLI regression coverage for ordinary text, dollar signs, hash signs
> and adjacent escapes. Four of the six cases fail before this change;
> all six pass after it. The expected values were also checked against
> GNU Make using scripts/Kbuild.include.
> 
> Fixes: b30204640192 ("scripts: add a tool to produce a compile_commands.json file")
> Assisted-by: LLM
> Signed-off-by: houtinghang <ue081723@gmail.com>
> ---
> Testing on Ubuntu with GCC 13.3.0, Clang/clangd 18.1.3 and Make 4.3:
> - x86-64 GCC, x86-64 Clang and ARM64 Clang defconfig builds of the kernel
>   image and configured modules passed without compiler warnings.
> - All three native make compile_commands.json targets passed.
> - 17,084 ordinary compilation database entries match the original tool.
> - 432 Kbuild/CLI/shell/compiler cases produced identical objects.
> - 259 mixed escape cases agree with Make for each compiler test run.
> - 22 clangd macro checks and real x86/ARM64 kernel checks passed.
> - The original tool fails the integration and clangd negative controls.
> - Six CLI cases pass on Linux/Python 3.12.3 and Windows/Python 3.13.
> 
> AI disclosure: Codex identified the issue, wrote the fix and regression
> test, verified CLI output and Make semantics, reviewed the diff, and
> drafted this patch after a request to find another Linux kernel bug.
> 
>  scripts/clang-tools/gen_compile_commands.py   | 10 ++--
>  .../clang-tools/gen_compile_commands_test.py  | 47 +++++++++++++++++++
>  2 files changed, 52 insertions(+), 5 deletions(-)
>  create mode 100644 scripts/clang-tools/gen_compile_commands_test.py
> 
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 8d14b81..d11d15b 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -166,11 +166,11 @@ def process_line(root_directory, command_prefix, file_path):
>          ValueError: Could not find the extracted file based on file_path and
>              root_directory or file_directory.
>      """
> -    # The .cmd files are intended to be included directly by Make, so they
> -    # escape the pound sign '#' as '$(pound)'. The compile_commands.json file
> -    # is not interepreted by Make, so this code replaces the escaped version
> -    # with '#'.
> -    prefix = command_prefix.replace('$(pound)', '#')
> +    # Undo the escaping performed by make-cmd in scripts/Kbuild.include.
> +    # Decode both escapes in one pass to preserve a literal '$(pound)'.

I think the original comment is more informative. Please just extend it.

  # The .cmd files are intended to be included directly by Make, so they
  # escape the literal '#' as '$(pound)' and the literal '$' as '$$'. The
  # compile_commands.json file is not interepreted by Make, so undo this
  # transformation.

> +    prefix = re.sub(r'\$\$|\$\(pound\)',
> +                    lambda match: '$' if match.group() == '$$' else '#',
> +                    command_prefix)

I think it would be more efficient and readable to just use a double
.replace() over re.sub() here, these strings are not that long.

  prefix = command_prefix.replace('$(pound)', '#').replace('$$', '$')

>      # Return the canonical path, eliminating any symbolic links encountered in the path.
>      abs_path = os.path.realpath(os.path.join(root_directory, file_path))
> diff --git a/scripts/clang-tools/gen_compile_commands_test.py b/scripts/clang-tools/gen_compile_commands_test.py
> new file mode 100644
> index 0000000..fe259af
> --- /dev/null
> +++ b/scripts/clang-tools/gen_compile_commands_test.py

Same comment on an earlier change, I don't think this test adds much to
belong in the tree so please drop it.

-- 
Cheers,
Nathan