From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BC0747D95E for ; Thu, 22 Jan 2026 10:58:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.177 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769079498; cv=none; b=IcXrOZHtLcJHrvFc1AOHLWNF0OU1s/5vvzH59QfmfV4T1PMQ1w2G2L759+qMWqaU3Ffr45ezosfuMXjI4GjYBc4asYIWwm3DwobrqHf6OPML6jDqEiXlb/+5JV8U/ff5Pj92YpcgIAiZV73cFrWiAXY+DkKlZrAYgdT4TxLGz4Y= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769079498; c=relaxed/simple; bh=8inItK2TlZQFCmtq5Ve2zysDmNVCy/TfewYqA8GAnyo=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=PF2uwUhZIzrCR0eTzR7suU74JgfY2NrV+QUI80Ny1t6nDPUC9I7lAYHRlaevIPf8HYipK23Qllec+e0qBld1AV5KHtwgoSYPACkXICXLa9Wbhx91wpxiAjYlujmhmkdzfxY/Ih1rsIu0+tCwjlWcdye7RUnTZYE/JDtso2c3phg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=UU+9JOb4; arc=none smtp.client-ip=209.85.208.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="UU+9JOb4" Received: by mail-lj1-f177.google.com with SMTP id 38308e7fff4ca-382fd8aaa6eso6531801fa.1 for ; Thu, 22 Jan 2026 02:58:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1769079493; x=1769684293; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=siS/LAEKILEgu9VaJuOyrKUxf9FW9n3qL3magrZsc2Y=; b=UU+9JOb4c/WBavtKTTvNz4vMZKVEI63VxvNyL3xmQdllnzEbpLXaSpVCxL0uqcBOwt fQuFGD0VHjhbARSNqzZlKhUoxHfqO5fN/Y23ZDXwolfnT8AwABcdjCYl7QuMtP8GBPZr ULJEvYesWtIlxN0dCGp6CC5u3p5aSODbqp7vU2HhJy/pgg6QYd+slQbXLDVmB3VOF46b SEqa+dVy8Np7edc06xaxiC9JfKOG9c7kZzZTYrHW/OipkNfpRm81QqV5Scs5E5x9EPCP qPtcNwQdYeBOddEhBhviS9qlsxXf/0Zc5+V0VOWxreRe2xC1ODCebbXrg1qUge3JHaBj DJzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769079493; x=1769684293; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=siS/LAEKILEgu9VaJuOyrKUxf9FW9n3qL3magrZsc2Y=; b=P0Ajg/RA9PNedC5TkmsCxgesJJXZfUYZ4NO7OrIDCs6ENsuLruO1y2kdjsCscDXH1s v2ptQ+HsrW0fC0o88zD36ZIAA5Q3F+8tY/T3O6EwC8ogEx4EkmhxcPQiN7r7iWUWVUX6 o40edR90W1TfQUNbl+64uUKG6PxqJ/Omn9HTnnB4iY+IlLiBNVi5SkUe7UDOj5HkQWHT tTgt4it012hTdDO3bVkNrwxQsNjwnrl4x9Nu+HC9JWb2+IxSNdGh6AJLYEXZvbEhNF1i 6/rBlbpTOXhjaGbu32B9O6lztSw2VZRz9DBq6ptfwIV7ttwhZn4E6dF6ey7vTxPdRIHs WbGw== X-Forwarded-Encrypted: i=1; AJvYcCX7P03glEFc8yMmNa/jG19HmV4DK+uRFptfcJqsvkrt0MTM2v9uOjBxqRaWlpcFM8H3J8fbt10JMEJJRK0=@vger.kernel.org X-Gm-Message-State: AOJu0YyTXTdoGRhOOuL58asI6WpxGxhEnILX/s6NC5FTQIiwTIBpEOCK EygD7wDpeq9tixQ+5ffcesmg3w6kHBBEO/wQl2p3XzJnYI/FkpLOVWFXFlra8t/W4OSfEMJsWDB 9gL0v X-Gm-Gg: AZuq6aI09bBKZwgJH7l8AD8EjWJQykICgAJhPnXx/Dc429WmuRZt1kT2J+wG+16dVyp u36l92ytCpJ8BfzxVJtFQzcGABA4oXZcbQ/gp9vzmBNZpxW0+dreBQDwwCeNkayNoII3+m512zd Du85eEagsksLxRUl4OBzDNJ0vkmGL81CwvEI8LjVAHcjrr5726cLQzTgudmrSwwTBdrN55lq6yU xqz1anfaksIWP/09HUcEU3OwVUJjFRHEQv/kJL+GTx/Ne5gSGHO0+e0AivvMkW1iXll6jNNLoCo 1GkdUAlRAe5YM0gGArRdw91ZPe0IXkVvt8bpphuUt9SOVhA3DN0pXGzaA90mXxoumqBm5n7Sz1S D7xIZKwnSXIui5x7uxkMYDFaVT0P1v896JS4t+wS2p+EYBzQ/WB6hd+9c4wDggHbGqAGd73OJnF OHWkP+cY8mKpP2zyXL2HTem8H5jz+ec3q+NKVzZUX5Vn2/ X-Received: by 2002:a2e:bd81:0:b0:37b:aaf7:f007 with SMTP id 38308e7fff4ca-385a54b02a7mr28386301fa.31.1769079493061; Thu, 22 Jan 2026 02:58:13 -0800 (PST) Received: from nuoska (87-100-249-247.bb.dnainternet.fi. [87.100.249.247]) by smtp.gmail.com with ESMTPSA id 38308e7fff4ca-385d5091433sm350841fa.14.2026.01.22.02.58.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jan 2026 02:58:12 -0800 (PST) From: Mikko Rapeli To: Nathan Chancellor , Nicolas Schier , Anders Roxell , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Mikko Rapeli Subject: [PATCH v3 1/3] scripts: kconfig: merge_config.sh: refactor from shell/sed/grep to awk Date: Thu, 22 Jan 2026 12:57:49 +0200 Message-ID: <20260122105751.2186609-1-mikko.rapeli@linaro.org> X-Mailer: git-send-email 2.51.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Anders Roxell merge_config.sh shell/sed/grep loop scales poorly and is slow. With Yocto genericarm64 kernel and around 190 config fragments the script takes more than 20 minutes to run on a fast build machine. Re-implementation with awk does the same job in 10 seconds. Using awk since it is likely available in the build environments and using perl, python etc would introduce more complex runtime dependencies. awk is good enough and lot better than shell/sed/grep. Output stays the same but changed execution time means that parallel job output may be ordered differently. Signed-off-by: Anders Roxell Signed-off-by: Mikko Rapeli --- scripts/kconfig/merge_config.sh | 168 ++++++++++++++++++++++++-------- 1 file changed, 128 insertions(+), 40 deletions(-) v3: use shell variables with plain $VAR syntax, don't convert to =n syntax and use " is not set" instead, reverted warning output to same as before in the shell script implementation, changed warning output from stderr to stdout as in the shell script implementation, tested that output from make ARCH=arm64 CROSS_COMPILE=aarch64-linux- clean defconfig hardening.config and make -s ARCH=arm64 CROSS_COMPILE=aarch64-linux- clean defconfig hardening.config is the same with and without this change v2: remove unused sed variables, awk from ${AWK} variable, curly brace syntax fix after rebase, triple check that correct revision of patches are used in testing with yocto/bitbake https://lore.kernel.org/linux-kbuild/20251229114447.45236-1-mikko.rapeli@linaro.org/T/#me07dcaaa205b608d56003aeb56a240e5eca1ad52 v1: https://lore.kernel.org/linux-kbuild/20251229114447.45236-1-mikko.rapeli@linaro.org/T/#t diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh index 79c09b378be8..de217771b78c 100755 --- a/scripts/kconfig/merge_config.sh +++ b/scripts/kconfig/merge_config.sh @@ -16,8 +16,8 @@ set -e clean_up() { - rm -f $TMP_FILE - rm -f $MERGE_FILE + rm -f "$TMP_FILE" + rm -f "$TMP_FILE.new" } usage() { @@ -43,6 +43,10 @@ STRICT=false CONFIG_PREFIX=${CONFIG_-CONFIG_} WARNOVERRIDE=echo +if [ -z "$AWK" ]; then + AWK=awk +fi + while true; do case $1 in "-n") @@ -117,11 +121,8 @@ if [ ! -r "$INITFILE" ]; then fi MERGE_LIST=$* -SED_CONFIG_EXP1="s/^\(${CONFIG_PREFIX}[a-zA-Z0-9_]*\)=.*/\1/p" -SED_CONFIG_EXP2="s/^# \(${CONFIG_PREFIX}[a-zA-Z0-9_]*\) is not set$/\1/p" TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX) -MERGE_FILE=$(mktemp ./.merge_tmp.config.XXXXXXXXXX) echo "Using $INITFILE as base" @@ -136,42 +137,129 @@ for ORIG_MERGE_FILE in $MERGE_LIST ; do echo "The merge file '$ORIG_MERGE_FILE' does not exist. Exit." >&2 exit 1 fi - cat $ORIG_MERGE_FILE > $MERGE_FILE - CFG_LIST=$(sed -n -e "$SED_CONFIG_EXP1" -e "$SED_CONFIG_EXP2" $MERGE_FILE) - - for CFG in $CFG_LIST ; do - grep -q -w $CFG $TMP_FILE || continue - PREV_VAL=$(grep -w $CFG $TMP_FILE) - NEW_VAL=$(grep -w $CFG $MERGE_FILE) - BUILTIN_FLAG=false - if [ "$BUILTIN" = "true" ] && [ "${NEW_VAL#CONFIG_*=}" = "m" ] && [ "${PREV_VAL#CONFIG_*=}" = "y" ]; then - ${WARNOVERRIDE} Previous value: $PREV_VAL - ${WARNOVERRIDE} New value: $NEW_VAL - ${WARNOVERRIDE} -y passed, will not demote y to m - ${WARNOVERRIDE} - BUILTIN_FLAG=true - elif [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then - ${WARNOVERRIDE} Value of $CFG is redefined by fragment $ORIG_MERGE_FILE: - ${WARNOVERRIDE} Previous value: $PREV_VAL - ${WARNOVERRIDE} New value: $NEW_VAL - ${WARNOVERRIDE} - if [ "$STRICT" = "true" ]; then - STRICT_MODE_VIOLATED=true - fi - elif [ "$WARNREDUN" = "true" ]; then - ${WARNOVERRIDE} Value of $CFG is redundant by fragment $ORIG_MERGE_FILE: - fi - if [ "$BUILTIN_FLAG" = "false" ]; then - sed -i "/$CFG[ =]/d" $TMP_FILE - else - sed -i "/$CFG[ =]/d" $MERGE_FILE - fi - done - # In case the previous file lacks a new line at the end - echo >> $TMP_FILE - cat $MERGE_FILE >> $TMP_FILE -done + # Use awk for single-pass processing instead of per-symbol grep/sed + if ! "$AWK" -v prefix="$CONFIG_PREFIX" \ + -v warnoverride="$WARNOVERRIDE" \ + -v strict="$STRICT" \ + -v builtin="$BUILTIN" \ + -v warnredun="$WARNREDUN" ' + BEGIN { + strict_violated = 0 + cfg_regex = "^" prefix "[a-zA-Z0-9_]+" + notset_regex = "^# " prefix "[a-zA-Z0-9_]+ is not set$" + } + + # Extract config name from a line, returns "" if not a config line + function get_cfg(line) { + if (match(line, cfg_regex)) { + return substr(line, RSTART, RLENGTH) + } else if (match(line, notset_regex)) { + # Extract CONFIG_FOO from "# CONFIG_FOO is not set" + sub(/^# /, "", line) + sub(/ is not set$/, "", line) + return line + } + return "" + } + + function warn_builtin(cfg, prev, new) { + if (warnoverride == "true") return + print cfg ": -y passed, will not demote y to m" + print "Previous value: " prev + print "New value: " new + print "" + } + + function warn_redefined(cfg, prev, new) { + if (warnoverride == "true") return + print "Value of " cfg " is redefined by fragment " mergefile ":" + print "Previous value: " prev + print "New value: " new + print "" + } + + function warn_redundant(cfg) { + if (warnredun != "true" || warnoverride == "true") return + print "Value of " cfg " is redundant by fragment " mergefile ":" + } + + # First pass: read merge file, store all lines and index + FILENAME == ARGV[1] { + mergefile = FILENAME + merge_lines[FNR] = $0 + merge_total = FNR + cfg = get_cfg($0) + if (cfg != "") { + merge_cfg[cfg] = $0 + merge_cfg_line[cfg] = FNR + } + next + } + # Second pass: process base file (TMP_FILE) + FILENAME == ARGV[2] { + cfg = get_cfg($0) + + # Not a config or not in merge file - keep it + if (cfg == "" || !(cfg in merge_cfg)) { + print $0 >> ARGV[3] + next + } + + prev_val = $0 + new_val = merge_cfg[cfg] + + # BUILTIN: do not demote y to m + if (builtin == "true" && new_val ~ /=m$/ && prev_val ~ /=y$/) { + warn_builtin(cfg, prev_val, new_val) + print $0 >> ARGV[3] + skip_merge[merge_cfg_line[cfg]] = 1 + next + } + + # Values equal - redundant + if (prev_val == new_val) { + warn_redundant(cfg) + next + } + + # "=n" is the same as "is not set" + if (prev_val ~ /=n$/ && new_val ~ / is not set$/) { + print $0 >> ARGV[3] + next + } + + # Values differ - redefined + warn_redefined(cfg, prev_val, new_val) + if (strict == "true") { + strict_violated = 1 + } + } + + # output file, skip all lines + FILENAME == ARGV[3] { + nextfile + } + + END { + # Newline in case base file lacks trailing newline + print "" >> ARGV[3] + # Append merge file, skipping lines marked for builtin preservation + for (i = 1; i <= merge_total; i++) { + if (!(i in skip_merge)) { + print merge_lines[i] >> ARGV[3] + } + } + if (strict_violated) { + exit 1 + } + }' \ + "$ORIG_MERGE_FILE" "$TMP_FILE" "$TMP_FILE.new"; then + # awk exited non-zero, strict mode was violated + STRICT_MODE_VIOLATED=true + fi + mv "$TMP_FILE.new" "$TMP_FILE" +done if [ "$STRICT_MODE_VIOLATED" = "true" ]; then echo "The fragment redefined a value and strict mode had been passed." exit 1 -- 2.34.1