Re: [PATCH] clang-tools: Decode dollar escaping in compile commands

[Nathan Chancellor <nathan@kernel.org>]

On 2026-09-23 23:10 +0800, houtinghang wrote:
> Kbuild doubles dollar signs when saving commands in .cmd files, but
> process_line() only decodes $(pound). Consequently, the compilation
> database retains doubled dollar signs in compiler arguments. It also

While I agree that '$$' should be handled like '$(pound)', I suspect it
is likely not a problem in practice since most commands that appear to
use '$$' are for generated files, which should not matter for
compile_commands.json but I might be mistaken.

> corrupts a literal $(pound), saved as $$(pound), into $#.

How would this happen, aside from the contrived test?

> Decode $$ and $(pound) in a single pass, matching how Make reads the
> saved command. A single pass avoids interpreting a decoded dollar sign
> as the start of another escape.
> 
> Add CLI regression coverage for ordinary text, dollar signs, hash signs
> and adjacent escapes. Four of the six cases fail before this change;
> all six pass after it. The expected values were also checked against
> GNU Make using scripts/Kbuild.include.
> 
> Fixes: b30204640192 ("scripts: add a tool to produce a compile_commands.json file")
> Assisted-by: LLM
> Signed-off-by: houtinghang <ue081723@gmail.com>
> ---
> Testing on Ubuntu with GCC 13.3.0, Clang/clangd 18.1.3 and Make 4.3:
> - x86-64 GCC, x86-64 Clang and ARM64 Clang defconfig builds of the kernel
>   image and configured modules passed without compiler warnings.
> - All three native make compile_commands.json targets passed.
> - 17,084 ordinary compilation database entries match the original tool.
> - 432 Kbuild/CLI/shell/compiler cases produced identical objects.
> - 259 mixed escape cases agree with Make for each compiler test run.
> - 22 clangd macro checks and real x86/ARM64 kernel checks passed.
> - The original tool fails the integration and clangd negative controls.
> - Six CLI cases pass on Linux/Python 3.12.3 and Windows/Python 3.13.
> 
> AI disclosure: Codex identified the issue, wrote the fix and regression
> test, verified CLI output and Make semantics, reviewed the diff, and
> drafted this patch after a request to find another Linux kernel bug.
> 
>  scripts/clang-tools/gen_compile_commands.py   | 10 ++--
>  .../clang-tools/gen_compile_commands_test.py  | 47 +++++++++++++++++++
>  2 files changed, 52 insertions(+), 5 deletions(-)
>  create mode 100644 scripts/clang-tools/gen_compile_commands_test.py
> 
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 8d14b81..d11d15b 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -166,11 +166,11 @@ def process_line(root_directory, command_prefix, file_path):
>          ValueError: Could not find the extracted file based on file_path and
>              root_directory or file_directory.
>      """
> -    # The .cmd files are intended to be included directly by Make, so they
> -    # escape the pound sign '#' as '$(pound)'. The compile_commands.json file
> -    # is not interepreted by Make, so this code replaces the escaped version
> -    # with '#'.
> -    prefix = command_prefix.replace('$(pound)', '#')
> +    # Undo the escaping performed by make-cmd in scripts/Kbuild.include.
> +    # Decode both escapes in one pass to preserve a literal '$(pound)'.

I think the original comment is more informative. Please just extend it.

  # The .cmd files are intended to be included directly by Make, so they
  # escape the literal '#' as '$(pound)' and the literal '$' as '$$'. The
  # compile_commands.json file is not interepreted by Make, so undo this
  # transformation.

> +    prefix = re.sub(r'\$\$|\$\(pound\)',
> +                    lambda match: '$' if match.group() == '$$' else '#',
> +                    command_prefix)

I think it would be more efficient and readable to just use a double
.replace() over re.sub() here, these strings are not that long.

  prefix = command_prefix.replace('$(pound)', '#').replace('$$', '$')

>      # Return the canonical path, eliminating any symbolic links encountered in the path.
>      abs_path = os.path.realpath(os.path.join(root_directory, file_path))
> diff --git a/scripts/clang-tools/gen_compile_commands_test.py b/scripts/clang-tools/gen_compile_commands_test.py
> new file mode 100644
> index 0000000..fe259af
> --- /dev/null
> +++ b/scripts/clang-tools/gen_compile_commands_test.py

Same comment on an earlier change, I don't think this test adds much to
belong in the tree so please drop it.

-- 
Cheers,
Nathan