[PATCH] gen_compile_commands: prune some directories

[PATCH] gen_compile_commands: prune some directories

[Masahiro Yamada]

If directories are passed to gen_compile_commands.py, os.walk() traverses
all the subdirectories to search for .cmd files, but we know some of them
are not worth traversing.

Use the 'topdown' parameter of os.walk to prune them.

Documentation about the 'topdown' option of os.walk:
  When topdown is True, the caller can modify the dirnames list
  in-place (perhaps using del or slice assignment), and walk() will
  only recurse into the subdirectories whose names remain in dirnames;
  this can be used to prune the search, impose a specific order of
  visiting, or even to inform walk() about directories the caller
  creates or renames before it resumes walk() again. Modifying
  dirnames when topdown is False has no effect on the behavior of
  the walk, because in bottom-up mode the directories in dirnames
  are generated before dirpath itself is generated.

This commit prunes four directories, .git, Documentation, include, and
tools.

The first three do not contain any C files. My main motivation is the
last one, tools/ directory.

Commit 6ca4c6d25949 ("gen_compile_commands: do not support .cmd files
under tools/ directory") stopped supporting the tools/ directory.
The current code no longer picks up .cmd files from the tools/
directory.

If you run:

  ./scripts/clang-tools/gen_compile_commands.py --log_level=INFO

then, you will see several "File ... not found" log messages.

This is expected, and I do not want to support the tools/ directory.
However, without an explicit comment "do not support tools/", somebody
might try to get it back. Clarify this.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 scripts/clang-tools/gen_compile_commands.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
index 19963708bcf8..eb5faefbdf74 100755
--- a/scripts/clang-tools/gen_compile_commands.py
+++ b/scripts/clang-tools/gen_compile_commands.py
@@ -20,7 +20,9 @@ _DEFAULT_LOG_LEVEL = 'WARNING'
 _FILENAME_PATTERN = r'^\..*\.cmd$'
 _LINE_PATTERN = r'^cmd_[^ ]*\.o := (.* )([^ ]*\.c)$'
 _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
-
+# The tools/ directory adopts a different build system, and produces .cmd
+# files in a different format. Do not support it.
+_EXCLUDE_DIRS = ['.git', 'Documentation', 'include', 'tools']
 
 def parse_arguments():
     """Sets up and parses command-line arguments.
@@ -80,8 +82,14 @@ def cmdfiles_in_dir(directory):
     """
 
     filename_matcher = re.compile(_FILENAME_PATTERN)
+    exclude_dirs = [ os.path.join(directory, d) for d in _EXCLUDE_DIRS ]
+
+    for dirpath, dirnames, filenames in os.walk(directory, topdown=True):
+        # Prune unwanted directories.
+        if dirpath in exclude_dirs:
+            dirnames[:] = []
+            continue
 
-    for dirpath, _, filenames in os.walk(directory):
         for filename in filenames:
             if filename_matcher.match(filename):
                 yield os.path.join(dirpath, filename)
-- 
2.27.0

Re: [PATCH] gen_compile_commands: prune some directories

[Nathan Chancellor]

On Fri, Feb 12, 2021 at 01:11:54AM +0900, Masahiro Yamada wrote:
> If directories are passed to gen_compile_commands.py, os.walk() traverses
> all the subdirectories to search for .cmd files, but we know some of them
> are not worth traversing.
> 
> Use the 'topdown' parameter of os.walk to prune them.
> 
> Documentation about the 'topdown' option of os.walk:
>   When topdown is True, the caller can modify the dirnames list
>   in-place (perhaps using del or slice assignment), and walk() will
>   only recurse into the subdirectories whose names remain in dirnames;
>   this can be used to prune the search, impose a specific order of
>   visiting, or even to inform walk() about directories the caller
>   creates or renames before it resumes walk() again. Modifying
>   dirnames when topdown is False has no effect on the behavior of
>   the walk, because in bottom-up mode the directories in dirnames
>   are generated before dirpath itself is generated.
> 
> This commit prunes four directories, .git, Documentation, include, and
> tools.
> 
> The first three do not contain any C files. My main motivation is the
> last one, tools/ directory.
> 
> Commit 6ca4c6d25949 ("gen_compile_commands: do not support .cmd files
> under tools/ directory") stopped supporting the tools/ directory.
> The current code no longer picks up .cmd files from the tools/
> directory.
> 
> If you run:
> 
>   ./scripts/clang-tools/gen_compile_commands.py --log_level=INFO
> 
> then, you will see several "File ... not found" log messages.
> 
> This is expected, and I do not want to support the tools/ directory.
> However, without an explicit comment "do not support tools/", somebody
> might try to get it back. Clarify this.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Sorry, I did not realize that gen_compile_commands.py did not intend to
support. I was only looking at the history for the current location, not
the former one of scripts/gen_compile_commands.py.

Acked-by: Nathan Chancellor <nathan@kernel.org>

> ---
> 
>  scripts/clang-tools/gen_compile_commands.py | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 19963708bcf8..eb5faefbdf74 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -20,7 +20,9 @@ _DEFAULT_LOG_LEVEL = 'WARNING'
>  _FILENAME_PATTERN = r'^\..*\.cmd$'
>  _LINE_PATTERN = r'^cmd_[^ ]*\.o := (.* )([^ ]*\.c)$'
>  _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
> -
> +# The tools/ directory adopts a different build system, and produces .cmd
> +# files in a different format. Do not support it.
> +_EXCLUDE_DIRS = ['.git', 'Documentation', 'include', 'tools']
>  
>  def parse_arguments():
>      """Sets up and parses command-line arguments.
> @@ -80,8 +82,14 @@ def cmdfiles_in_dir(directory):
>      """
>  
>      filename_matcher = re.compile(_FILENAME_PATTERN)
> +    exclude_dirs = [ os.path.join(directory, d) for d in _EXCLUDE_DIRS ]
> +
> +    for dirpath, dirnames, filenames in os.walk(directory, topdown=True):
> +        # Prune unwanted directories.
> +        if dirpath in exclude_dirs:
> +            dirnames[:] = []
> +            continue
>  
> -    for dirpath, _, filenames in os.walk(directory):
>          for filename in filenames:
>              if filename_matcher.match(filename):
>                  yield os.path.join(dirpath, filename)
> -- 
> 2.27.0
> 

Re: [PATCH] gen_compile_commands: prune some directories

[Masahiro Yamada]

On Fri, Feb 12, 2021 at 2:19 AM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Fri, Feb 12, 2021 at 01:11:54AM +0900, Masahiro Yamada wrote:
> > If directories are passed to gen_compile_commands.py, os.walk() traverses
> > all the subdirectories to search for .cmd files, but we know some of them
> > are not worth traversing.
> >
> > Use the 'topdown' parameter of os.walk to prune them.
> >
> > Documentation about the 'topdown' option of os.walk:
> >   When topdown is True, the caller can modify the dirnames list
> >   in-place (perhaps using del or slice assignment), and walk() will
> >   only recurse into the subdirectories whose names remain in dirnames;
> >   this can be used to prune the search, impose a specific order of
> >   visiting, or even to inform walk() about directories the caller
> >   creates or renames before it resumes walk() again. Modifying
> >   dirnames when topdown is False has no effect on the behavior of
> >   the walk, because in bottom-up mode the directories in dirnames
> >   are generated before dirpath itself is generated.
> >
> > This commit prunes four directories, .git, Documentation, include, and
> > tools.
> >
> > The first three do not contain any C files. My main motivation is the
> > last one, tools/ directory.
> >
> > Commit 6ca4c6d25949 ("gen_compile_commands: do not support .cmd files
> > under tools/ directory") stopped supporting the tools/ directory.
> > The current code no longer picks up .cmd files from the tools/
> > directory.
> >
> > If you run:
> >
> >   ./scripts/clang-tools/gen_compile_commands.py --log_level=INFO
> >
> > then, you will see several "File ... not found" log messages.
> >
> > This is expected, and I do not want to support the tools/ directory.
> > However, without an explicit comment "do not support tools/", somebody
> > might try to get it back. Clarify this.
> >
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
>
> Sorry, I did not realize that gen_compile_commands.py did not intend to
> support. I was only looking at the history for the current location, not
> the former one of scripts/gen_compile_commands.py.
>
> Acked-by: Nathan Chancellor <nathan@kernel.org>
>

Applied to linux-kbuild.



Best Regards
Masahiro Yamada