From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lf1-f48.google.com (mail-lf1-f48.google.com [209.85.167.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D40E9313E1C for ; Wed, 31 Dec 2025 08:41:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767170490; cv=none; b=TA02Av8POpXJ2jvrsxecsiWjhpOwHlbK5b+GP+m7+pKU19l4OM4n+rUZ8gcu3nNICLVR/ekWq2SKHtFMTc7icw+RvezyZpWIEorQ/appVp+xXqCWXQ9DO4nm9eIH4tMdKdzDV4GKoTw/LP+CyYrFBQwwaaWHPyTluU56Um4naJ4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767170490; c=relaxed/simple; bh=RF9BVH1lrMCCOUkRUOnmjjxgV+zbxqDfBSzLoTTiQ1M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LfD7A9rk97UEh6vsZftNa2YrsIJMaXrpHsk2JuCC2WpBs0trZlumZOj0gAG2GLx6aBmflLdPnvI0ErJfwi3pEC506IJMrvWYn5QCwE8cUOI4GJ3UJWyxyu1Cx62wBgLWdA5sua7LoFgFRqxsCye4RV10k0HY26b4oorf1LIOw3s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=j7DXOGbL; arc=none smtp.client-ip=209.85.167.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="j7DXOGbL" Received: by mail-lf1-f48.google.com with SMTP id 2adb3069b0e04-5945510fd7aso8605658e87.0 for ; Wed, 31 Dec 2025 00:41:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1767170485; x=1767775285; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wFVU9Q/1Cok58g44IhCYQSr/mO6T/jqCKS7T/Pv3sYs=; b=j7DXOGbL7ycxDAwUuoDiyuyAgMTOUoizGhTO4yH6eLRD6FSK5Nt5XBdttR9VJucAwP dDd2SVG1xNnK1TpLzUSE/Aou04ix1oVqFNbWZf+I8ern9j0elIIqqi75LW6zXTA2U39X 3fx6hYTh88cI7/G7IWxkZjRoEKNqq4xGtsusF5vyM/p9ikWNNeyCMczjatJZWbbTmtA5 MseCbANJctZjP6U4SSVXxd4lQRttjwyTv+8QTCO5RT902MyPBJj+ShcCPKjpGHuQLP9A QN/gf4TQVHWzwCjyc/ItGvmXxZc2+eJyfISOoWOYPkw6wEth00nzOrcU9ibTd45NFEoT OKvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767170485; x=1767775285; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=wFVU9Q/1Cok58g44IhCYQSr/mO6T/jqCKS7T/Pv3sYs=; b=kRwiep9PCoSEgHBmsmx1i4jq0TwDBowzzIYGH30pMFLj7hShy8hvP4DtVh1YYwWcVj gFknY/hGgzeqf8zunw2lcpchbA3c0xrlEPwrSs5uIiiIMvE7IVTxhjrLji0Ag3yP0xVB +lLtrj0AmbckEBURKM6d9/rYzZXHcIDeJ55+xyOxxQAmdGAmV9oghkilJfu+FwnDYZl2 kxoUeDFIyF5e6efFbb9KVm9ayJhP+MLRAsywHVPz3zqyDxNutrdZm6HZVl7lcM/d3tRh kGn+zuKmxRk5jBNYp0HVjiC/5huDuXvAqROvEMNS6bZoPWYvRMt5l3DhSKlVzpek8o5o BqPQ== X-Forwarded-Encrypted: i=1; AJvYcCXN6jwGfVDkEfAUJuaVzBJ3+g4hWvbxQ39uYc/lJeZ0I95V+BzZ8Zd8vvBZC/ouEUxNMuZwYNSJHdiq5Gc=@vger.kernel.org X-Gm-Message-State: AOJu0YyzgYSCrWnf4E2AG/YcxFDO/u8slWq4Mu2DjBAFmYVkC/jHz1dS GRfqqRgApwYv5fowfA476nSR3meckGtI4sgQlHAJjcsxAfOmj9yBc9j9pMJY/ff9cA8= X-Gm-Gg: AY/fxX7cKh1Ziqo5fkwR2hrm3h9pIftIWKrmVcWz19EibY9/yrepKiQKn1+RN2Tsrzl rQsiZqX+nKTJ1S6tcyeKDrxbfpPLraJHKKcKgy6IiAzl4g+IrF0bSLY7xlLM/BisA3dPqkOSWUr e0//gYq0+XFCOrK/oAnn0eMdcubjJ+yQIP4F6MdwCJEPIHK7cwGzrRgJ6HlVCnocxTt1BsiVrv2 5YkrP1o1DWrDrPMp0t5139o5dtK0eekhswQFDHsup+Mt9rGKSrW7n5wjyGmyOs2cW8MPeNX7+Cu O1DxECfZVxVb8NtRTmS5p9I/va2oEJaWlY4ut/BGGZgstYDABQ62fRF4VlceGGZZBGpp/0/x8Aj MLtI2jMV4Cdha7xxCyH9T3mqoSbCqxMzPesOwbK3uGprxXBRMadmEYFuWo0Ske+b+YHwCHngDid sLbuYWK+zFLXr47/bQ9cACFZPvINchuei+J+Maf9lbm93p X-Google-Smtp-Source: AGHT+IHwqkiiXgHJUJdc9Gjzn1M9Ef+MQQ8/6gUCZVigT0vTPMHNx5NVj6LjhKlzdxjz9hnB1gyK6Q== X-Received: by 2002:a05:6512:23a7:b0:59a:107a:45a9 with SMTP id 2adb3069b0e04-59a17df40a5mr12498438e87.52.1767170484911; Wed, 31 Dec 2025 00:41:24 -0800 (PST) Received: from nuoska (87-100-249-247.bb.dnainternet.fi. [87.100.249.247]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-59a181beb5csm10666566e87.0.2025.12.31.00.41.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Dec 2025 00:41:24 -0800 (PST) From: Mikko Rapeli To: Nathan Chancellor , Nicolas Schier , Anders Roxell , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Mikko Rapeli Subject: [PATCH v2 1/3] scripts: kconfig: merge_config.sh: refactor from shell/sed/grep to awk Date: Wed, 31 Dec 2025 10:40:48 +0200 Message-ID: <20251231084050.186874-1-mikko.rapeli@linaro.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20251230205549.GE4062669@ax162> References: <20251230205549.GE4062669@ax162> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Anders Roxell merge_config.sh shell/sed/grep loop scales poorly and is slow. With Yocto genericarm64 kernel and around 190 config fragments the script takes more than 20 minutes to run on a fast build machine. Re-implementation with awk does the same job in 10 seconds. Using awk since it is likely available in the build environments and using perl, python etc would introduce more complex runtime dependencies. awk is good enough and lot better than shell/sed/grep. Signed-off-by: Anders Roxell Signed-off-by: Mikko Rapeli --- scripts/kconfig/merge_config.sh | 168 ++++++++++++++++++++++++-------- 1 file changed, 128 insertions(+), 40 deletions(-) v2: remove unused sed variables, awk from ${AWK} variable, curly brace syntax fix after rebase, triple check that correct revision of patches are used in testing with yocto/bitbake v1: https://lore.kernel.org/linux-kbuild/20251229114447.45236-1-mikko.rapeli@linaro.org/T/#t diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh index 79c09b378be8..4cefe3cdfc2f 100755 --- a/scripts/kconfig/merge_config.sh +++ b/scripts/kconfig/merge_config.sh @@ -16,8 +16,8 @@ set -e clean_up() { - rm -f $TMP_FILE - rm -f $MERGE_FILE + rm -f "${TMP_FILE}" + rm -f "${TMP_FILE}.new" } usage() { @@ -43,6 +43,10 @@ STRICT=false CONFIG_PREFIX=${CONFIG_-CONFIG_} WARNOVERRIDE=echo +if [ -z "${AWK}" ]; then + AWK=awk +fi + while true; do case $1 in "-n") @@ -117,11 +121,8 @@ if [ ! -r "$INITFILE" ]; then fi MERGE_LIST=$* -SED_CONFIG_EXP1="s/^\(${CONFIG_PREFIX}[a-zA-Z0-9_]*\)=.*/\1/p" -SED_CONFIG_EXP2="s/^# \(${CONFIG_PREFIX}[a-zA-Z0-9_]*\) is not set$/\1/p" TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX) -MERGE_FILE=$(mktemp ./.merge_tmp.config.XXXXXXXXXX) echo "Using $INITFILE as base" @@ -136,42 +137,129 @@ for ORIG_MERGE_FILE in $MERGE_LIST ; do echo "The merge file '$ORIG_MERGE_FILE' does not exist. Exit." >&2 exit 1 fi - cat $ORIG_MERGE_FILE > $MERGE_FILE - CFG_LIST=$(sed -n -e "$SED_CONFIG_EXP1" -e "$SED_CONFIG_EXP2" $MERGE_FILE) - - for CFG in $CFG_LIST ; do - grep -q -w $CFG $TMP_FILE || continue - PREV_VAL=$(grep -w $CFG $TMP_FILE) - NEW_VAL=$(grep -w $CFG $MERGE_FILE) - BUILTIN_FLAG=false - if [ "$BUILTIN" = "true" ] && [ "${NEW_VAL#CONFIG_*=}" = "m" ] && [ "${PREV_VAL#CONFIG_*=}" = "y" ]; then - ${WARNOVERRIDE} Previous value: $PREV_VAL - ${WARNOVERRIDE} New value: $NEW_VAL - ${WARNOVERRIDE} -y passed, will not demote y to m - ${WARNOVERRIDE} - BUILTIN_FLAG=true - elif [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then - ${WARNOVERRIDE} Value of $CFG is redefined by fragment $ORIG_MERGE_FILE: - ${WARNOVERRIDE} Previous value: $PREV_VAL - ${WARNOVERRIDE} New value: $NEW_VAL - ${WARNOVERRIDE} - if [ "$STRICT" = "true" ]; then - STRICT_MODE_VIOLATED=true - fi - elif [ "$WARNREDUN" = "true" ]; then - ${WARNOVERRIDE} Value of $CFG is redundant by fragment $ORIG_MERGE_FILE: - fi - if [ "$BUILTIN_FLAG" = "false" ]; then - sed -i "/$CFG[ =]/d" $TMP_FILE - else - sed -i "/$CFG[ =]/d" $MERGE_FILE - fi - done - # In case the previous file lacks a new line at the end - echo >> $TMP_FILE - cat $MERGE_FILE >> $TMP_FILE -done + # Use awk for single-pass processing instead of per-symbol grep/sed + if ! "${AWK}" -v prefix="$CONFIG_PREFIX" \ + -v warnoverride="$WARNOVERRIDE" \ + -v strict="$STRICT" \ + -v builtin="$BUILTIN" \ + -v warnredun="$WARNREDUN" ' + BEGIN { + strict_violated = 0 + cfg_regex = "^" prefix "[a-zA-Z0-9_]+" + notset_regex = "^# " prefix "[a-zA-Z0-9_]+ is not set$" + } + + # Extract config name from a line, returns "" if not a config line + function get_cfg(line) { + if (match(line, cfg_regex)) { + return substr(line, RSTART, RLENGTH) + } else if (match(line, notset_regex)) { + # Extract CONFIG_FOO from "# CONFIG_FOO is not set" + sub(/^# /, "", line) + sub(/ is not set$/, "", line) + return line + } + return "" + } + + # Normalize: strip trailing comments, convert "is not set" to "=n" + function normalize(line) { + if (line == "") return "" + sub(/[[:space:]]+#.*/, "", line) + if (line ~ / is not set$/) { + sub(/^# /, "", line) + sub(/ is not set$/, "=n", line) + } + return line + } + + function warn_builtin(cfg, prev, new) { + if (warnoverride == "true") return + print cfg ": -y passed, will not demote y to m" > "/dev/stderr" + print " Previous value: " prev > "/dev/stderr" + print " New value: " new > "/dev/stderr" + print "" > "/dev/stderr" + } + + function warn_redefined(cfg, prev, new) { + if (warnoverride == "true") return + print "Value of " cfg " is redefined by fragment " mergefile ":" > "/dev/stderr" + print " Previous value: " prev > "/dev/stderr" + print " New value: " new > "/dev/stderr" + print "" > "/dev/stderr" + } + + function warn_redundant(cfg) { + if (warnredun != "true" || warnoverride == "true") return + print "Value of " cfg " is redundant by fragment " mergefile ":" > "/dev/stderr" + } + # First pass: read merge file, store all lines and index + FILENAME == ARGV[1] { + mergefile = FILENAME + merge_lines[FNR] = $0 + merge_total = FNR + cfg = get_cfg($0) + if (cfg != "") { + merge_cfg[cfg] = $0 + merge_cfg_line[cfg] = FNR + } + next + } + + # Second pass: process base file (TMP_FILE) + { + cfg = get_cfg($0) + + # Not a config or not in merge file - keep it + if (cfg == "" || !(cfg in merge_cfg)) { + print $0 + next + } + + prev_val = normalize($0) + new_val = normalize(merge_cfg[cfg]) + + # BUILTIN: do not demote y to m + if (builtin == "true" && new_val ~ /=m$/ && prev_val ~ /=y$/) { + warn_builtin(cfg, prev_val, new_val) + print $0 + skip_merge[merge_cfg_line[cfg]] = 1 + next + } + + # Values equal - redundant + if (prev_val == new_val) { + warn_redundant(cfg) + next + } + + # Values differ - redefined + warn_redefined(cfg, prev_val, new_val) + if (strict == "true") { + strict_violated = 1 + } + } + + END { + # Newline in case base file lacks trailing newline + print "" + # Append merge file, skipping lines marked for builtin preservation + for (i = 1; i <= merge_total; i++) { + if (!(i in skip_merge)) { + print merge_lines[i] + } + } + if (strict_violated) { + exit 1 + } + }' \ + "$ORIG_MERGE_FILE" "$TMP_FILE" > "${TMP_FILE}.new"; then + # awk exited non-zero, strict mode was violated + STRICT_MODE_VIOLATED=true + fi + mv "${TMP_FILE}.new" "$TMP_FILE" +done if [ "$STRICT_MODE_VIOLATED" = "true" ]; then echo "The fragment redefined a value and strict mode had been passed." exit 1 -- 2.34.1