* [PATCH] Revert "kbuild: create deterministic initramfs directory listings" [not found] <CGME20180816161825eucas1p11be9cea5d622b905154590bae90a47a4@eucas1p1.samsung.com> @ 2018-08-16 16:18 ` Andrzej Pietrasiewicz 2018-08-16 23:34 ` Bjørn Forsman 0 siblings, 1 reply; 6+ messages in thread From: Andrzej Pietrasiewicz @ 2018-08-16 16:18 UTC (permalink / raw) To: linux-kernel Cc: Andrzej Pietrasiewicz, Michael Forney, Masahiro Yamada, Bjørn Forsman This reverts commit 9e6e0d5f2a2713402cf9dce69b9f9b516e4185d2. The reverted commit introduces broken builds. Even though the cpio archive does contain all the specified files, it seems that the kernel, while populating rootfs, scans the cpio buffer linearly and fails to create files whose parent directories are nonexistent at the moment of this failed creation. As a result, such files are not accessible when kernel boots into initramfs. Here is how to reproduce, from inside kernel source directory, with the patch in question not reverted: $ mkdir -p ../test/find/me/if/you/can $ touch ../test/find/me/if/you/can/file.txt $ cat ../initramfs.list file /busybox-x86_64 ../initramfs/busybox-x86_64 0755 0 0 dir /sbin 0755 0 0 dir /proc 0755 0 0 dir /sys 0755 0 0 dir /bin 0755 0 0 dir /usr 0755 0 0 dir /usr/sbin 0755 0 0 dir /usr/bin 0755 0 0 dir /etc 0755 0 0 dir /etc/init.d 0755 0 0 file /etc/inittab ../initramfs/inittab 0755 0 0 file /etc/init.d/rcS ../initramfs/rcS-x86_64 0755 0 0 dir /lib 0755 0 0 dir /lib/modules 0755 0 0 $ ls ../initramfs busybox-x86_64 inittab rcS-x86_64 $ cat ../initramfs/inittab ::sysinit:/etc/init.d/rcS ::askfirst:-/bin/sh ::ctrlaltdel:/sbin/reboot ::shutdown:/sbin/swapoff -a ::shutdown:/sbin/umount -a -r ::restart:/sbin/init $ cat ../initramfs/rcS-x86_64 #!/busybox-x86_64 sh /busybox-x86_64 --install mount -t proc none /proc mount -t sysfs none /sys mdev -s echo /sbin/mdev > /proc/sys/kernel/hotplug $ scripts/gen_initramfs_list.sh ../test/ ../initramfs.list ##################### # ../test/ # Last modified: 1534428527.0818855150 file /find/me/if/you/can/file.txt ../test/find/me/if/you/can/file.txt 664 1000 1000 dir /find/me/if/you/can 775 1000 1000 dir /find/me/if/you 775 1000 1000 dir /find/me/if 775 1000 1000 dir /find/me 775 1000 1000 dir /find 775 1000 1000 ##################### # ../initramfs.list # Last modified: 1534433777.8384497600 file /busybox-x86_64 ../initramfs/busybox-x86_64 0755 0 0 dir /sbin 0755 0 0 dir /proc 0755 0 0 dir /sys 0755 0 0 dir /bin 0755 0 0 dir /usr 0755 0 0 dir /usr/sbin 0755 0 0 dir /usr/bin 0755 0 0 dir /etc 0755 0 0 dir /etc/init.d 0755 0 0 file /etc/inittab ../initramfs/inittab 0755 0 0 file /etc/init.d/rcS ../initramfs/rcS-x86_64 0755 0 0 dir /lib 0755 0 0 dir /lib/modules 0755 0 0 $ cat ../initramfs.cpio | cpio -t find/me/if/you/can/file.txt find/me/if/you/can find/me/if/you find/me/if find/me find busybox-x86_64 sbin proc sys bin usr usr/sbin usr/bin etc etc/init.d etc/inittab etc/init.d/rcS lib lib/modules 1961 bloków BUT: $ qemu-system-x86_64 -nographic -serial mon:stdio \ -kernel arch/x86/boot/bzImage \ -append "root=/dev/ram console=ttyS0 rdinit=/busybox-x86_64 init" \ -M q35 -initrd ../initramfs.cpio # ls bin etc linuxrc sbin busybox-x86_64 find proc sys dev lib root usr / # ls -l find total 0 With the patch reverted the file /find/me/if/you/can/file.txt is accessible: $ scripts/gen_initramfs_list.sh ../test/ ../initramfs.list ##################### # ../test/ # Last modified: 1534428527.0818855150 dir /find 775 1000 1000 dir /find/me 775 1000 1000 dir /find/me/if 775 1000 1000 dir /find/me/if/you 775 1000 1000 dir /find/me/if/you/can 775 1000 1000 file /find/me/if/you/can/file.txt ../test/find/me/if/you/can/file.txt 664 1000 1000 ##################### # ../initramfs.list # Last modified: 1534433777.8384497600 file /busybox-x86_64 ../initramfs/busybox-x86_64 0755 0 0 dir /sbin 0755 0 0 dir /proc 0755 0 0 dir /sys 0755 0 0 dir /bin 0755 0 0 dir /usr 0755 0 0 dir /usr/sbin 0755 0 0 dir /usr/bin 0755 0 0 dir /etc 0755 0 0 dir /etc/init.d 0755 0 0 file /etc/inittab ../initramfs/inittab 0755 0 0 file /etc/init.d/rcS ../initramfs/rcS-x86_64 0755 0 0 dir /lib 0755 0 0 dir /lib/modules 0755 0 0 $ cat ../initramfs.cpio | cpio -t find find/me find/me/if find/me/if/you find/me/if/you/can find/me/if/you/can/file.txt busybox-x86_64 sbin proc sys bin usr usr/sbin usr/bin etc etc/init.d etc/inittab etc/init.d/rcS lib lib/modules 1961 bloków $ qemu-system-x86_64 -nographic -serial mon:stdio \ -kernel arch/x86/boot/bzImage \ -append "root=/dev/ram console=ttyS0 rdinit=/busybox-x86_64 init" \ -M q35 -initrd ../initramfs.cpio # find find/ find/ find/me find/me/if find/me/if/you find/me/if/you/can find/me/if/you/can/file.txt / # Signed-off-by: Andrzej Pietrasiewicz <andrzej.p@samsung.com> --- scripts/gen_initramfs_list.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh index 10e528b..e5d7b03 100755 --- a/scripts/gen_initramfs_list.sh +++ b/scripts/gen_initramfs_list.sh @@ -174,7 +174,7 @@ dir_filelist() { ${dep_list}header "$1" srcdir=$(echo "$1" | sed -e 's://*:/:g') - dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n" | sort) + dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n") # If $dirlist is only one line, then the directory is empty if [ "$(echo "${dirlist}" | wc -l)" -gt 1 ]; then -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] Revert "kbuild: create deterministic initramfs directory listings" 2018-08-16 16:18 ` [PATCH] Revert "kbuild: create deterministic initramfs directory listings" Andrzej Pietrasiewicz @ 2018-08-16 23:34 ` Bjørn Forsman 2018-08-17 7:47 ` Andrzej Pietrasiewicz 0 siblings, 1 reply; 6+ messages in thread From: Bjørn Forsman @ 2018-08-16 23:34 UTC (permalink / raw) To: andrzej.p; +Cc: LKML, forney, Masahiro Yamada On Thu, 16 Aug 2018 at 18:18, Andrzej Pietrasiewicz <andrzej.p@samsung.com> wrote: > > This reverts commit 9e6e0d5f2a2713402cf9dce69b9f9b516e4185d2. > > The reverted commit introduces broken builds. Even though the cpio archive > does contain all the specified files, it seems that the kernel, while > populating rootfs, scans the cpio buffer linearly and fails to create > files whose parent directories are nonexistent at the moment of this failed > creation. As a result, such files are not accessible when kernel boots into > initramfs. > > Here is how to reproduce, from inside kernel source directory, with the > patch in question not reverted: > > $ mkdir -p ../test/find/me/if/you/can > $ touch ../test/find/me/if/you/can/file.txt > $ cat ../initramfs.list > file /busybox-x86_64 ../initramfs/busybox-x86_64 0755 0 0 > dir /sbin 0755 0 0 > dir /proc 0755 0 0 > dir /sys 0755 0 0 > dir /bin 0755 0 0 > dir /usr 0755 0 0 > dir /usr/sbin 0755 0 0 > dir /usr/bin 0755 0 0 > dir /etc 0755 0 0 > dir /etc/init.d 0755 0 0 > file /etc/inittab ../initramfs/inittab 0755 0 0 > file /etc/init.d/rcS ../initramfs/rcS-x86_64 0755 0 0 > dir /lib 0755 0 0 > dir /lib/modules 0755 0 0 > $ ls ../initramfs > busybox-x86_64 inittab rcS-x86_64 > $ cat ../initramfs/inittab > ::sysinit:/etc/init.d/rcS > ::askfirst:-/bin/sh > ::ctrlaltdel:/sbin/reboot > ::shutdown:/sbin/swapoff -a > ::shutdown:/sbin/umount -a -r > ::restart:/sbin/init > $ cat ../initramfs/rcS-x86_64 > #!/busybox-x86_64 sh > > /busybox-x86_64 --install > > mount -t proc none /proc > mount -t sysfs none /sys > > mdev -s > echo /sbin/mdev > /proc/sys/kernel/hotplug > $ scripts/gen_initramfs_list.sh ../test/ ../initramfs.list > > ##################### > # ../test/ > # Last modified: 1534428527.0818855150 > > file /find/me/if/you/can/file.txt ../test/find/me/if/you/can/file.txt 664 1000 1000 > dir /find/me/if/you/can 775 1000 1000 > dir /find/me/if/you 775 1000 1000 > dir /find/me/if 775 1000 1000 > dir /find/me 775 1000 1000 > dir /find 775 1000 1000 I'm unable to reproduce it. On my system the listing is sorted so that it works (parent directories appear before files). I tried to run with LANG=C and it also sorts correctly. What is your LANG=? I think we better add a LANG=C somewhere in the kernel build system, because I think you have a LANG= that makes it sort differently. A quick fix would of course be to insert it right next to sort, but there may be other places that may break due to LANG= settings. Best regards, Bjørn Forsman ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] Revert "kbuild: create deterministic initramfs directory listings" 2018-08-16 23:34 ` Bjørn Forsman @ 2018-08-17 7:47 ` Andrzej Pietrasiewicz 2018-08-17 8:14 ` Masahiro Yamada 0 siblings, 1 reply; 6+ messages in thread From: Andrzej Pietrasiewicz @ 2018-08-17 7:47 UTC (permalink / raw) To: Bjørn Forsman; +Cc: LKML, forney, Masahiro Yamada Hi Bjørn, W dniu 17.08.2018 o 01:34, Bjørn Forsman pisze: > On Thu, 16 Aug 2018 at 18:18, Andrzej Pietrasiewicz > <andrzej.p@samsung.com> wrote: >> >> This reverts commit 9e6e0d5f2a2713402cf9dce69b9f9b516e4185d2. >> >> The reverted commit introduces broken builds. Even though the cpio archive >> does contain all the specified files, it seems that the kernel, while >> populating rootfs, scans the cpio buffer linearly and fails to create >> files whose parent directories are nonexistent at the moment of this failed >> creation. As a result, such files are not accessible when kernel boots into >> initramfs. >> <snip> > > I'm unable to reproduce it. On my system the listing is sorted so that > it works (parent directories appear before files). I tried to run with > LANG=C and it also sorts correctly. What is your LANG=? I think we > better add a LANG=C somewhere in the kernel build system, because I > think you have a LANG= that makes it sort differently. A quick fix > would of course be to insert it right next to sort, but there may be > other places that may break due to LANG= settings. > Thanks for the tip. My LANG is pl_PL.UTF-8. Indeed, prefixing the "sort" invocation in question with LANG=C does solve the problem. I don't feel competent enough to say whether this is the correct place for the fix, or if the fix should be made elsewhere. Another option is that it is the "sort" itself that should be fixed (or the locale, or both). Would you be able (as the author of the patch in question) to check if there are other locales which cause the problem? Thanks, Andrzej ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] Revert "kbuild: create deterministic initramfs directory listings" 2018-08-17 7:47 ` Andrzej Pietrasiewicz @ 2018-08-17 8:14 ` Masahiro Yamada [not found] ` <CGME20180817120447eucas1p1dde0176cf7d824e8a42e2d39ac5531f9@eucas1p1.samsung.com> 0 siblings, 1 reply; 6+ messages in thread From: Masahiro Yamada @ 2018-08-17 8:14 UTC (permalink / raw) To: Andrzej Pietrasiewicz; +Cc: Bjørn Forsman, LKML, forney 2018-08-17 16:47 GMT+09:00 Andrzej Pietrasiewicz <andrzej.p@samsung.com>: > Hi Bjørn, > > W dniu 17.08.2018 o 01:34, Bjørn Forsman pisze: >> On Thu, 16 Aug 2018 at 18:18, Andrzej Pietrasiewicz >> <andrzej.p@samsung.com> wrote: >>> >>> This reverts commit 9e6e0d5f2a2713402cf9dce69b9f9b516e4185d2. >>> >>> The reverted commit introduces broken builds. Even though the cpio archive >>> does contain all the specified files, it seems that the kernel, while >>> populating rootfs, scans the cpio buffer linearly and fails to create >>> files whose parent directories are nonexistent at the moment of this failed >>> creation. As a result, such files are not accessible when kernel boots into >>> initramfs. >>> > > <snip> > >> >> I'm unable to reproduce it. On my system the listing is sorted so that >> it works (parent directories appear before files). I tried to run with >> LANG=C and it also sorts correctly. What is your LANG=? I think we >> better add a LANG=C somewhere in the kernel build system, because I >> think you have a LANG= that makes it sort differently. A quick fix >> would of course be to insert it right next to sort, but there may be >> other places that may break due to LANG= settings. >> > > Thanks for the tip. My LANG is pl_PL.UTF-8. Indeed, prefixing the "sort" > invocation in question with LANG=C does solve the problem. I don't feel > competent enough to say whether this is the correct place for the fix, > or if the fix should be made elsewhere. Another option is that it is the > "sort" itself that should be fixed (or the locale, or both). > > Would you be able (as the author of the patch in question) to check if > there are other locales which cause the problem? > > Thanks, > > Andrzej I cannot add LANG=C to the high-level of the build system at least since it would break the localization. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <CGME20180817120447eucas1p1dde0176cf7d824e8a42e2d39ac5531f9@eucas1p1.samsung.com>]
* [PATCH] kbuild: make sorting initramfs contents independent of locale [not found] ` <CGME20180817120447eucas1p1dde0176cf7d824e8a42e2d39ac5531f9@eucas1p1.samsung.com> @ 2018-08-17 12:03 ` Andrzej Pietrasiewicz 2018-08-20 15:03 ` Masahiro Yamada 0 siblings, 1 reply; 6+ messages in thread From: Andrzej Pietrasiewicz @ 2018-08-17 12:03 UTC (permalink / raw) To: linux-kernel; +Cc: Andrzej Pietrasiewicz, Masahiro Yamada, Bjørn Forsman Some LANG values (e.g. pl_PL.UTF-8) cause the sort command to output files before their parent directories, which makes them inaccessible for the kernel. In other words, when the kernel populates the rootfs, it is unable to create files whose parent directories have not been yet created. This patch makes sorting use the default (LANG=C) locale, which results in correctly laid out initramfs images (parent directories before files). Signed-off-by: Andrzej Pietrasiewicz <andrzej.p@samsung.com> --- scripts/gen_initramfs_list.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh index 10e528b..0aad760 100755 --- a/scripts/gen_initramfs_list.sh +++ b/scripts/gen_initramfs_list.sh @@ -174,7 +174,7 @@ dir_filelist() { ${dep_list}header "$1" srcdir=$(echo "$1" | sed -e 's://*:/:g') - dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n" | sort) + dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n" | LANG=C sort) # If $dirlist is only one line, then the directory is empty if [ "$(echo "${dirlist}" | wc -l)" -gt 1 ]; then -- 2.7.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] kbuild: make sorting initramfs contents independent of locale 2018-08-17 12:03 ` [PATCH] kbuild: make sorting initramfs contents independent of locale Andrzej Pietrasiewicz @ 2018-08-20 15:03 ` Masahiro Yamada 0 siblings, 0 replies; 6+ messages in thread From: Masahiro Yamada @ 2018-08-20 15:03 UTC (permalink / raw) To: Andrzej Pietrasiewicz; +Cc: Linux Kernel Mailing List, Bjørn Forsman 2018-08-17 21:03 GMT+09:00 Andrzej Pietrasiewicz <andrzej.p@samsung.com>: > Some LANG values (e.g. pl_PL.UTF-8) cause the sort command to output > files before their parent directories, which makes them inaccessible for > the kernel. In other words, when the kernel populates the rootfs, it is > unable to create files whose parent directories have not been yet created. > > This patch makes sorting use the default (LANG=C) locale, which results in > correctly laid out initramfs images (parent directories before files). > > Signed-off-by: Andrzej Pietrasiewicz <andrzej.p@samsung.com> > --- Applied to linux-kbuild. Thanks! > scripts/gen_initramfs_list.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh > index 10e528b..0aad760 100755 > --- a/scripts/gen_initramfs_list.sh > +++ b/scripts/gen_initramfs_list.sh > @@ -174,7 +174,7 @@ dir_filelist() { > ${dep_list}header "$1" > > srcdir=$(echo "$1" | sed -e 's://*:/:g') > - dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n" | sort) > + dirlist=$(find "${srcdir}" -printf "%p %m %U %G\n" | LANG=C sort) > > # If $dirlist is only one line, then the directory is empty > if [ "$(echo "${dirlist}" | wc -l)" -gt 1 ]; then > -- > 2.7.4 > -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-08-20 15:03 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CGME20180816161825eucas1p11be9cea5d622b905154590bae90a47a4@eucas1p1.samsung.com>
2018-08-16 16:18 ` [PATCH] Revert "kbuild: create deterministic initramfs directory listings" Andrzej Pietrasiewicz
2018-08-16 23:34 ` Bjørn Forsman
2018-08-17 7:47 ` Andrzej Pietrasiewicz
2018-08-17 8:14 ` Masahiro Yamada
[not found] ` <CGME20180817120447eucas1p1dde0176cf7d824e8a42e2d39ac5531f9@eucas1p1.samsung.com>
2018-08-17 12:03 ` [PATCH] kbuild: make sorting initramfs contents independent of locale Andrzej Pietrasiewicz
2018-08-20 15:03 ` Masahiro Yamada
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®