Re: [PATCH v3 1/3] scripts: kconfig: merge_config.sh: refactor from shell/sed/grep to awk

[Andreas Larsson <andreas@gaisler.com>]

On 2026-03-10 11:01, Mikko Rapeli wrote:
> Hi,
> 
> On Tue, Mar 10, 2026 at 10:55:46AM +0100, Andreas Larsson wrote:
>> On 2026-03-10 08:50, Mikko Rapeli wrote:
>>> On Mon, Mar 09, 2026 at 10:09:04AM -0700, Nathan Chancellor wrote:
>>>> On Mon, Mar 09, 2026 at 05:38:58PM +0100, Andreas Larsson wrote:
>>>>> On 2026-01-22 11:57, Mikko Rapeli wrote:
>>>>>> From: Anders Roxell <anders.roxell@linaro.org>
>>>>>>
>>>>>> merge_config.sh shell/sed/grep loop scales poorly and is slow.
>>>>>> With Yocto genericarm64 kernel and around 190 config fragments
>>>>>> the script takes more than 20 minutes to run on a fast build machine.
>>>>>> Re-implementation with awk does the same job in 10 seconds.
>>>>>> Using awk since it is likely available in the build environments
>>>>>> and using perl, python etc would introduce more complex runtime
>>>>>> dependencies. awk is good enough and lot better than shell/sed/grep.
>>>>>>
>>>>>> Output stays the same but changed execution time means that
>>>>>> parallel job output may be ordered differently.
>>>>>>
>>>>>> Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
>>>>>> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
>>>>>> ---
>>>>>>  scripts/kconfig/merge_config.sh | 168 ++++++++++++++++++++++++--------
>>>>>>  1 file changed, 128 insertions(+), 40 deletions(-)
>>>>>
>>>>> Hi,
>>>>>
>>>>> Commit 5fa9b82cbcfc ("scripts: kconfig: merge_config.sh: refactor from
>>>>> shell/sed/grep to awk") breaks merge_config.sh for me:
>>>>>
>>>>> ---------->%----------
>>>>> $ make tiny.config
>>>>>   HOSTCC  scripts/basic/fixdep
>>>>>   HOSTCC  scripts/kconfig/conf.o
>>>>>   HOSTCC  scripts/kconfig/confdata.o
>>>>>   HOSTCC  scripts/kconfig/expr.o
>>>>>   LEX     scripts/kconfig/lexer.lex.c
>>>>>   YACC    scripts/kconfig/parser.tab.[ch]
>>>>>   HOSTCC  scripts/kconfig/lexer.lex.o
>>>>>   HOSTCC  scripts/kconfig/menu.o
>>>>>   HOSTCC  scripts/kconfig/parser.tab.o
>>>>>   HOSTCC  scripts/kconfig/preprocess.o
>>>>>   HOSTCC  scripts/kconfig/symbol.o
>>>>>   HOSTCC  scripts/kconfig/util.o
>>>>>   HOSTLD  scripts/kconfig/conf
>>>>> The base file '.config' does not exist. Creating one...
>>>>> Using .config as base
>>>>> Merging ./kernel/configs/tiny.config
>>>>> awk: cannot open ./.tmp.config.U9SROCKTBj.new (No such file or directory)
>>>>> mv: cannot stat './.tmp.config.U9SROCKTBj.new': No such file or directory
>>>>> make[2]: *** [scripts/kconfig/Makefile:112: tiny.config] Error 1
>>>>> make[1]: *** [<kernel-source-top-dir>/Makefile:744: tiny.config] Error 2
>>>>> make: *** [Makefile:248: __sub-make] Error 2
>>>>> ---------->%----------
>>>>>
>>>>> with this underlying call to merge_config.sh, an empty .config and where
>>>>> my awk is GNU Awk 5.1.0:
>>>>>
>>>>> ---------->%----------
>>>>> $ make tiny.config V=1
>>>>> make --no-print-directory -C <kernel-source-top-dir> \
>>>>> -f <kernel-source-top-dir>/Makefile tiny.config
>>>>> make -f ./scripts/Makefile.build obj=scripts/basic
>>>>> make -f ./scripts/Makefile.build obj=scripts/kconfig tiny.config
>>>>> # cmd_merge_fragments tiny.config
>>>>>   ./scripts/kconfig/merge_config.sh -m .config ./kernel/configs/tiny.config ./arch/x86/configs/tiny.config
>>>>> ...
>>>>>
>>>>> $ awk --version
>>>>> GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)
>>>>> ---------->%----------
>>>>>
>>>>> At 5fa9b82cbcfc~ things works as expected.
>>>
>>> Hmm, all these steps work for me with GNU awk 5.1.0 so something else
>>> is different here. On line 267 awk is given input file args
>>> "$ORIG_MERGE_FILE" "$TMP_FILE" "$TMP_FILE.new" and it fails to
>>> create the output file "$TMP_FILE.new" with >> and plain print statement.
>>>
>>>> Hmmm, not sure how I have not seen this error myself since I test
>>>> tiny.config but I can reproduce with a clean output directory. Something
>>>> like this seems like a simple fix but the only instances of ARGV[3] in
>>>> the awk script that I can see use '>>', which should create the file if
>>>> it does not exist. Mikko, any ideas?
>>>>
>>>> diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
>>>> index 735e1de450c6..070ecae87a1c 100755
>>>> --- a/scripts/kconfig/merge_config.sh
>>>> +++ b/scripts/kconfig/merge_config.sh
>>>> @@ -123,6 +123,7 @@ fi
>>>>  MERGE_LIST=$*
>>>>  
>>>>  TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
>>>> +touch "$TMP_FILE.new"
>>>>  
>>>>  echo "Using $INITFILE as base"
>>>
>>> This could help but I fail to understand why this would be needed. Why is awk
>>> not able to create this file on line 256?
>>>
>>> awk manual says:
>>>
>>> https://www.gnu.org/software/gawk/manual/html_node/Redirection.html
>>>
>>>     print items >> output-file
>>>
>>>     This redirection prints the items into the preexisting output file named
>>>     output-file. The difference between this and the single-‘>’ redirection
>>>     is that the old contents (if any) of output-file are not erased. Instead,
>>>     the awk output is appended to the file. If output-file does not exist,
>>>     then it is created.
>>
>> Awk is opening the file for reading and fails before any prints with >>
>> to it, much like this one does:
>>
>> $ awk 'FILENAME == "non-existent" {nextfile}' non-existent
>> awk: fatal: cannot open file `non-existent' for reading: No such file or directory
>>
>> $ strace awk 'FILENAME == "non-existent" {nextfile}' non-existent |& grep non-existent
>> execve("/usr/bin/awk", ["awk", "FILENAME == \"non-existent\" {next"..., "non-existent"], 0x7ffd32bda2e8 /* 66 vars */) = 0
>> openat(AT_FDCWD, "non-existent", O_RDONLY) = -1 ENOENT (No such file or directory)
>> write(2, "cannot open file `non-existent' "..., 70cannot open file `non-existent' for reading: No such file or directory) = 70
>>
>>
>>> And in all my testing this works. Which distro is this? Is something else like
>>> file system setup in some way unusual?
>> Ubuntu 22.04.5 and a regular ext4 file system.
> 
> Right, so some awk versions behave differently. For the record in my testing on
> Ubuntu 22.04.5 LTS (Jammy Jellyfish) this script works :/
> 
> Can you try if the touch fix from Nathan works?

Yes, with that fix it works fine.

Cheers,
Andreas