From: HONG Yifan <elsk@google.com>
To: enh@google.com, ccross@google.com,
Masahiro Yamada <masahiroy@kernel.org>,
Miguel Ojeda <ojeda@kernel.org>,
Matthias Maennich <maennich@google.com>
Cc: HONG Yifan <elsk@google.com>,
kernel-team@android.com, linux-kernel@vger.kernel.org
Subject: [PATCH v1] kheaders: prevent `find` from seeing perl temp files
Date: Thu, 7 Nov 2024 00:58:22 +0000 [thread overview]
Message-ID: <20241107005831.15434-1-elsk@google.com> (raw)
Symptom:
The command
find ... | xargs ... perl -i
occasionally triggers error messages like the following, with the build
still succeeding:
Can't open <redacted>/kernel/.tmp_cpio_dir/include/dt-bindings/clock/XXNX4nW9: No such file or directory.
Analysis:
With strace, the root cause has been identified to be `perl -i` creating
temporary files inside $cpio_dir, which causes `find` to see the
temporary files and emit the names. `find` is likely implemented with
readdir. POSIX `readdir` says:
If a file is removed from or added to the directory after the most
recent call to opendir() or rewinddir(), whether a subsequent call
to readdir() returns an entry for that file is unspecified.
So if the libc that `find` links against choose to return that entry
in readdir(), a possible sequence of events is the following:
1. find emits foo.h
2. xargs executes `perl -i foo.h`
3. perl (pid=100) creates temporary file `XXXXXXXX`
4. find sees file `XXXXXXXX` and emit it
5. PID 100 exits, cleaning up the temporary file `XXXXXXXX`
6. xargs executes `perl -i XXXXXXXX`
7. perl (pid=200) tries to read the file, but it doesn't exist any more.
... triggering the error message.
One can reproduce the bug with the following command (assuming PWD
contains the list of headers in kheaders.tar.xz)
for i in $(seq 100); do
find -type f -print0 |
xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;';
done
With a `find` linking against musl libc, the error message is emitted
6/100 times.
The fix:
This change modifies the command so `find` only emits .h names, thereby
skipping the temporary files. Another possible fix would be to store
the results of `find` before feeding them into xargs; I didn't take this
approach because the current approach appears marginally more optimized
and involves a smaller change.
Signed-off-by: HONG Yifan <elsk@google.com>
---
| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index 383fd43ac612..271e0145f406 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -84,7 +84,7 @@ for f in $dir_list;
done | cpio --quiet -pdu $cpio_dir >/dev/null 2>&1
# Remove comments except SDPX lines
-find $cpio_dir -type f -print0 |
+find $cpio_dir -type f -name '*.h' -print0 |
xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
# Create archive and try to normalize metadata for reproducibility.
--
2.47.0.199.ga7371fff76-goog
next reply other threads:[~2024-11-07 0:58 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-11-07 0:58 HONG Yifan [this message]
2024-11-10 11:18 ` Masahiro Yamada
2024-11-10 11:21 ` Masahiro Yamada
2024-12-05 6:16 ` Hong, Yifan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241107005831.15434-1-elsk@google.com \
--to=elsk@google.com \
--cc=ccross@google.com \
--cc=enh@google.com \
--cc=kernel-team@android.com \
--cc=linux-kernel@vger.kernel.org \
--cc=maennich@google.com \
--cc=masahiroy@kernel.org \
--cc=ojeda@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®