From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 825DC340A6A for ; Tue, 10 Mar 2026 10:01:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.50 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773136888; cv=none; b=Gxj2H43QMkvcQ20UphtQAFnQe3ZIzLebUe/IsgzOhvL4NIRg5p+b/CtJtaqDyiHrRIPwLBycSLpV39/Pb0G9V1X6DKXa8GuJfUu33EjUkjScMbNl7IFquhzmzSd2gHq7WlPFXuhG4UhfBznQq+WkaxB42r1dzg5Eg68OJKRy4nQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773136888; c=relaxed/simple; bh=Urwk7x4TITggfD1JqD5XD6o7yzS3Pil0oE2seibVOnU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=rno/UfOfqEzmZMcoG8iM24SioCD8gw8BQj/UgfsB62mQMvQri1wb055dr0WYXntivgFqX75qptFGp9Xez2OrU+GiOCOg8k4byRW75wmm46QJcyD/kbuY53qjscK/lfNyvGQUC3KX82kJAgLt7FoHEB9GgbrQAe8FPTcCfFD522M= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=wnFzpGUT; arc=none smtp.client-ip=209.85.167.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="wnFzpGUT" Received: by mail-lf1-f50.google.com with SMTP id 2adb3069b0e04-5a142464316so3632723e87.1 for ; Tue, 10 Mar 2026 03:01:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1773136885; x=1773741685; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=Ac/UQiG4Lk57r2LyCgOQ4I3DQTnWytHe+MjruKr6HsQ=; b=wnFzpGUTlqBx1hDDs6IR9p4s1jp/mcuKPIKLT22uuj9G54nNB2cjTPDfby72v/LTgP Dd95judhf+c0estbPQvzlTmkES9H4y34yjWV8v5tuRjgvKVv//a4OvGHYmJlNO6jra6E RoE3XrCrv7yf1HlMYPamiCC4PIlaOAel0HBkTUQayBzK9fiemiw5S1wQHzHG5tYHwa0e noiKfM1FmPLQ22wr3D54xkYmNw0DtwcEIdcRtWg51Evxc9bdXlopW7k8+wnPGnefGvrO UYDhONTF4OAGF+IYW6IPn0TCdd1sqFcFzwNM/+lhrNYwYMWUrjyy9Yu3FZxmsYYHbe2E L96Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773136885; x=1773741685; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ac/UQiG4Lk57r2LyCgOQ4I3DQTnWytHe+MjruKr6HsQ=; b=oIe2cAYszqkO/UyrWsFbmc3qyRvWd5ExTW95hfzeVdPTNVR4wD6B5hQ3XggGJmAm1f 6eWjPYWUFWfGouwVh4hdfmulVl/WQtNgQoG7wt98XpUc80jWmh7I8k5fCQ+c3wqFKL/h DQKSGTk0IVRlIQrtwPK+86PPdsUBLpAar2RQ4sWvrWl6o7nyVa0H1HJbfxKsnBC1c7h1 c4pYWqTczC/p21d49MiS5tesuo3rjXF74ZPp4DdyLVMG480mVSbgdL0l5+gxAMKnup3u jfMhaC5hFiBXKRJDYsCh0QkHW9hjnxczTLcujo4j4H43L7JOLHitYq34leNfPRjht3WB iuIg== X-Forwarded-Encrypted: i=1; AJvYcCWQOdXKgCWU++zn7n5n5vIc9PIfRIDHDqwA0AHWwzcisDn41+jv5CoegleYw5aYrHtSd5n5218qovDMYj4=@vger.kernel.org X-Gm-Message-State: AOJu0YyEUNB6z3V7gqmdZMJKrAdnDurY988oYZ4Mn+s2UGYch7usW8PI PgdAt/mnAjtOoU2UWxeRg7BmTsrlOgqT+5sZQuIehkdiDfG9V2eEs9WTQh93oEfnTfM= X-Gm-Gg: ATEYQzzGKwRPBtSxK0J0GWkS00N88654PhTLjY0961xHJ9bDXGB3eRTrsklle+3jbLf mthsL1IVOyZ+SMVAH9h7xcoseNsxBMr6V+sjo85L42LjjUpiukBadVIv4BLTI5RFHokEPZYo76O FhtmgVz9KRP1onMU/ACU68vgHvGUOZHYXtLENUfvtYbG2dtq5VpN6bZUl1bBId5NzFZgAY1cPKY c9Jw9Is+tYjMioW86EffUWknuygXRlFLw1fmXXocxJc5Q1p/vz3heSAzsBPN+pRdnlmISU1+m0l F0vGGONw9lzaEMZq9RYsLYySJbZjoQCFWoONuLD2pZNyvtTu0R5thwHmBJvMI5Kxq1uWyIfGwXX tDYoeMf0LHevl2hodLMugbfTxjM5eUkgQD2qG6gS11XfuXC+7mU0DDpEVO/2pgMAoQdzv/Jyeu/ 4xqNs4Tyds/7AfyXTZY+qnLfg9fbsRjJaGez6CAaWVE+uVkIJsIjqQZEJR X-Received: by 2002:a05:6512:230a:b0:5a1:2eaa:ac83 with SMTP id 2adb3069b0e04-5a13cd4b765mr4645749e87.43.1773136884381; Tue, 10 Mar 2026 03:01:24 -0700 (PDT) Received: from nuoska (87-100-249-247.bb.dnainternet.fi. [87.100.249.247]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5a13d08d737sm2675951e87.88.2026.03.10.03.01.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2026 03:01:24 -0700 (PDT) Date: Tue, 10 Mar 2026 12:01:22 +0200 From: Mikko Rapeli To: Andreas Larsson Cc: Nathan Chancellor , Nicolas Schier , Anders Roxell , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 1/3] scripts: kconfig: merge_config.sh: refactor from shell/sed/grep to awk Message-ID: References: <20260122105751.2186609-1-mikko.rapeli@linaro.org> <20260309170904.GA2779008@ax162> <5e948d9f-dac3-4329-b7fe-e7f9a80108d9@gaisler.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5e948d9f-dac3-4329-b7fe-e7f9a80108d9@gaisler.com> Hi, On Tue, Mar 10, 2026 at 10:55:46AM +0100, Andreas Larsson wrote: > On 2026-03-10 08:50, Mikko Rapeli wrote: > > On Mon, Mar 09, 2026 at 10:09:04AM -0700, Nathan Chancellor wrote: > >> On Mon, Mar 09, 2026 at 05:38:58PM +0100, Andreas Larsson wrote: > >>> On 2026-01-22 11:57, Mikko Rapeli wrote: > >>>> From: Anders Roxell > >>>> > >>>> merge_config.sh shell/sed/grep loop scales poorly and is slow. > >>>> With Yocto genericarm64 kernel and around 190 config fragments > >>>> the script takes more than 20 minutes to run on a fast build machine. > >>>> Re-implementation with awk does the same job in 10 seconds. > >>>> Using awk since it is likely available in the build environments > >>>> and using perl, python etc would introduce more complex runtime > >>>> dependencies. awk is good enough and lot better than shell/sed/grep. > >>>> > >>>> Output stays the same but changed execution time means that > >>>> parallel job output may be ordered differently. > >>>> > >>>> Signed-off-by: Anders Roxell > >>>> Signed-off-by: Mikko Rapeli > >>>> --- > >>>> scripts/kconfig/merge_config.sh | 168 ++++++++++++++++++++++++-------- > >>>> 1 file changed, 128 insertions(+), 40 deletions(-) > >>> > >>> Hi, > >>> > >>> Commit 5fa9b82cbcfc ("scripts: kconfig: merge_config.sh: refactor from > >>> shell/sed/grep to awk") breaks merge_config.sh for me: > >>> > >>> ---------->%---------- > >>> $ make tiny.config > >>> HOSTCC scripts/basic/fixdep > >>> HOSTCC scripts/kconfig/conf.o > >>> HOSTCC scripts/kconfig/confdata.o > >>> HOSTCC scripts/kconfig/expr.o > >>> LEX scripts/kconfig/lexer.lex.c > >>> YACC scripts/kconfig/parser.tab.[ch] > >>> HOSTCC scripts/kconfig/lexer.lex.o > >>> HOSTCC scripts/kconfig/menu.o > >>> HOSTCC scripts/kconfig/parser.tab.o > >>> HOSTCC scripts/kconfig/preprocess.o > >>> HOSTCC scripts/kconfig/symbol.o > >>> HOSTCC scripts/kconfig/util.o > >>> HOSTLD scripts/kconfig/conf > >>> The base file '.config' does not exist. Creating one... > >>> Using .config as base > >>> Merging ./kernel/configs/tiny.config > >>> awk: cannot open ./.tmp.config.U9SROCKTBj.new (No such file or directory) > >>> mv: cannot stat './.tmp.config.U9SROCKTBj.new': No such file or directory > >>> make[2]: *** [scripts/kconfig/Makefile:112: tiny.config] Error 1 > >>> make[1]: *** [/Makefile:744: tiny.config] Error 2 > >>> make: *** [Makefile:248: __sub-make] Error 2 > >>> ---------->%---------- > >>> > >>> with this underlying call to merge_config.sh, an empty .config and where > >>> my awk is GNU Awk 5.1.0: > >>> > >>> ---------->%---------- > >>> $ make tiny.config V=1 > >>> make --no-print-directory -C \ > >>> -f /Makefile tiny.config > >>> make -f ./scripts/Makefile.build obj=scripts/basic > >>> make -f ./scripts/Makefile.build obj=scripts/kconfig tiny.config > >>> # cmd_merge_fragments tiny.config > >>> ./scripts/kconfig/merge_config.sh -m .config ./kernel/configs/tiny.config ./arch/x86/configs/tiny.config > >>> ... > >>> > >>> $ awk --version > >>> GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1) > >>> ---------->%---------- > >>> > >>> At 5fa9b82cbcfc~ things works as expected. > > > > Hmm, all these steps work for me with GNU awk 5.1.0 so something else > > is different here. On line 267 awk is given input file args > > "$ORIG_MERGE_FILE" "$TMP_FILE" "$TMP_FILE.new" and it fails to > > create the output file "$TMP_FILE.new" with >> and plain print statement. > > > >> Hmmm, not sure how I have not seen this error myself since I test > >> tiny.config but I can reproduce with a clean output directory. Something > >> like this seems like a simple fix but the only instances of ARGV[3] in > >> the awk script that I can see use '>>', which should create the file if > >> it does not exist. Mikko, any ideas? > >> > >> diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh > >> index 735e1de450c6..070ecae87a1c 100755 > >> --- a/scripts/kconfig/merge_config.sh > >> +++ b/scripts/kconfig/merge_config.sh > >> @@ -123,6 +123,7 @@ fi > >> MERGE_LIST=$* > >> > >> TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX) > >> +touch "$TMP_FILE.new" > >> > >> echo "Using $INITFILE as base" > > > > This could help but I fail to understand why this would be needed. Why is awk > > not able to create this file on line 256? > > > > awk manual says: > > > > https://www.gnu.org/software/gawk/manual/html_node/Redirection.html > > > > print items >> output-file > > > > This redirection prints the items into the preexisting output file named > > output-file. The difference between this and the single-‘>’ redirection > > is that the old contents (if any) of output-file are not erased. Instead, > > the awk output is appended to the file. If output-file does not exist, > > then it is created. > > Awk is opening the file for reading and fails before any prints with >> > to it, much like this one does: > > $ awk 'FILENAME == "non-existent" {nextfile}' non-existent > awk: fatal: cannot open file `non-existent' for reading: No such file or directory > > $ strace awk 'FILENAME == "non-existent" {nextfile}' non-existent |& grep non-existent > execve("/usr/bin/awk", ["awk", "FILENAME == \"non-existent\" {next"..., "non-existent"], 0x7ffd32bda2e8 /* 66 vars */) = 0 > openat(AT_FDCWD, "non-existent", O_RDONLY) = -1 ENOENT (No such file or directory) > write(2, "cannot open file `non-existent' "..., 70cannot open file `non-existent' for reading: No such file or directory) = 70 > > > > And in all my testing this works. Which distro is this? Is something else like > > file system setup in some way unusual? > Ubuntu 22.04.5 and a regular ext4 file system. Right, so some awk versions behave differently. For the record in my testing on Ubuntu 22.04.5 LTS (Jammy Jellyfish) this script works :/ Can you try if the touch fix from Nathan works? Cheers, -Mikko