gen_initramfs_list.sh escaping problem or stale dependency file?

gen_initramfs_list.sh escaping problem or stale dependency file?

[Florian Fainelli]

Hi,

I have a root filesystem embedding filenames that look like these:

/lib/data/<vid>:<pid>

these are essentially files that can be matched against an USB
vendor/product id in an easy way.

Now, the fun part is that this is only a problem when doing the
following (using OpenWrt/LEDE as a build system):

1:
- set CONFIG_INITRAMFS_SOURCE=""
- build kernel modules
- build my user-space tools
- build the kernel image
- reconfigure the kernel to now use an initramfs
- build the kernel w/ initramfs

and then back to step 1 with the kernel build, would I hit this error:

usr/Makefile:64: *** multiple target patterns.  Stop.

which comes from usr/.initramfs_data.cpio.d containing these files
without escaping:

deps_initramfs := ./scripts/gen_initramfs_list.sh \
/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb
\
/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb/lib
\
...

/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb/lib/network/wwan
\
/exp00/fainelli/openwrt/trunk/build_dir/target-arm-linux-gnueabihf/root-brcmstb/lib/network/wwan/19d2:0063
\

Which sorts of make sense here because the file name contains a ":"
which is not escaped, so GNU Make tries to interpret it.

Now the part that does not quite make sense to me is why this file is
even relevant here considering that the first thing we do is set
CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.

Any clues what could be wrong here? I am happy to provide any build
drops you may need to reproduce that.

Thanks!
-- 
Florian

Re: gen_initramfs_list.sh escaping problem or stale dependency file?

[Michal Marek]

On Mon, Sep 12, 2016 at 05:12:15PM -0700, Florian Fainelli wrote:
> Hi,
> 
> I have a root filesystem embedding filenames that look like these:
> 
> /lib/data/<vid>:<pid>
> 
> these are essentially files that can be matched against an USB
> vendor/product id in an easy way.
> 
> Now, the fun part is that this is only a problem when doing the
> following (using OpenWrt/LEDE as a build system):
> 
> 1:
> - set CONFIG_INITRAMFS_SOURCE=""
> - build kernel modules
> - build my user-space tools
> - build the kernel image
> - reconfigure the kernel to now use an initramfs
> - build the kernel w/ initramfs
> 
> and then back to step 1 with the kernel build, would I hit this error:
> 
> usr/Makefile:64: *** multiple target patterns.  Stop.
[...]
> Which sorts of make sense here because the file name contains a ":"
> which is not escaped, so GNU Make tries to interpret it.
> 
> Now the part that does not quite make sense to me is why this file is
> even relevant here considering that the first thing we do is set
> CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.

It is possible that we read usr/Makefile twice for some reason. But the
real problem is the lack of escaping. Can you try the following
(untested) patch?


diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
index 17fa901418ae..5d3188e74101 100755
--- a/scripts/gen_initramfs_list.sh
+++ b/scripts/gen_initramfs_list.sh
@@ -97,7 +97,10 @@ print_mtime() {
 }
 
 list_parse() {
-	[ ! -L "$1" ] && echo "$1 \\" || :
+	if [ -L "$1" ]; then
+		return
+	fi
+	echo "$1" | sed 's/\([:%]\)/\\\1/g; s/$/ \\/'
 }
 
 # for each file print a line in following format

Thanks,
Michal

Re: gen_initramfs_list.sh escaping problem or stale dependency file?

[Florian Fainelli]

On 09/13/2016 12:24 AM, Michal Marek wrote:
> On Mon, Sep 12, 2016 at 05:12:15PM -0700, Florian Fainelli wrote:
>> Hi,
>>
>> I have a root filesystem embedding filenames that look like these:
>>
>> /lib/data/<vid>:<pid>
>>
>> these are essentially files that can be matched against an USB
>> vendor/product id in an easy way.
>>
>> Now, the fun part is that this is only a problem when doing the
>> following (using OpenWrt/LEDE as a build system):
>>
>> 1:
>> - set CONFIG_INITRAMFS_SOURCE=""
>> - build kernel modules
>> - build my user-space tools
>> - build the kernel image
>> - reconfigure the kernel to now use an initramfs
>> - build the kernel w/ initramfs
>>
>> and then back to step 1 with the kernel build, would I hit this error:
>>
>> usr/Makefile:64: *** multiple target patterns.  Stop.
> [...]
>> Which sorts of make sense here because the file name contains a ":"
>> which is not escaped, so GNU Make tries to interpret it.
>>
>> Now the part that does not quite make sense to me is why this file is
>> even relevant here considering that the first thing we do is set
>> CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.
> 
> It is possible that we read usr/Makefile twice for some reason. But the
> real problem is the lack of escaping. Can you try the following
> (untested) patch?

This patch works for me:

Tested-by: Florian Fainelli <f.fainelli@gmail.com>

Kind of surprising that this has not showed up before, but maybe people
don't really go through the same steps while re-configuring/re-building
their kernels

Thanks Michal!

> 
> 
> diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
> index 17fa901418ae..5d3188e74101 100755
> --- a/scripts/gen_initramfs_list.sh
> +++ b/scripts/gen_initramfs_list.sh
> @@ -97,7 +97,10 @@ print_mtime() {
>  }
>  
>  list_parse() {
> -	[ ! -L "$1" ] && echo "$1 \\" || :
> +	if [ -L "$1" ]; then
> +		return
> +	fi
> +	echo "$1" | sed 's/\([:%]\)/\\\1/g; s/$/ \\/'
>  }
>  
>  # for each file print a line in following format
> 
> Thanks,
> Michal
> 


-- 
Florian

Re: gen_initramfs_list.sh escaping problem or stale dependency file?

[Florian Fainelli]

On 09/13/2016 12:24 AM, Michal Marek wrote:
> On Mon, Sep 12, 2016 at 05:12:15PM -0700, Florian Fainelli wrote:
>> Hi,
>>
>> I have a root filesystem embedding filenames that look like these:
>>
>> /lib/data/<vid>:<pid>
>>
>> these are essentially files that can be matched against an USB
>> vendor/product id in an easy way.
>>
>> Now, the fun part is that this is only a problem when doing the
>> following (using OpenWrt/LEDE as a build system):
>>
>> 1:
>> - set CONFIG_INITRAMFS_SOURCE=""
>> - build kernel modules
>> - build my user-space tools
>> - build the kernel image
>> - reconfigure the kernel to now use an initramfs
>> - build the kernel w/ initramfs
>>
>> and then back to step 1 with the kernel build, would I hit this error:
>>
>> usr/Makefile:64: *** multiple target patterns.  Stop.
> [...]
>> Which sorts of make sense here because the file name contains a ":"
>> which is not escaped, so GNU Make tries to interpret it.
>>
>> Now the part that does not quite make sense to me is why this file is
>> even relevant here considering that the first thing we do is set
>> CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.
> 
> It is possible that we read usr/Makefile twice for some reason. But the
> real problem is the lack of escaping. Can you try the following
> (untested) patch?

Can you submit an official patch for this? Thanks a lot!

> 
> 
> diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
> index 17fa901418ae..5d3188e74101 100755
> --- a/scripts/gen_initramfs_list.sh
> +++ b/scripts/gen_initramfs_list.sh
> @@ -97,7 +97,10 @@ print_mtime() {
>  }
>  
>  list_parse() {
> -	[ ! -L "$1" ] && echo "$1 \\" || :
> +	if [ -L "$1" ]; then
> +		return
> +	fi
> +	echo "$1" | sed 's/\([:%]\)/\\\1/g; s/$/ \\/'
>  }
>  
>  # for each file print a line in following format
> 
> Thanks,
> Michal
> 


-- 
Florian

Re: gen_initramfs_list.sh escaping problem or stale dependency file?

[Michal Marek]

On 2016-09-19 22:00, Florian Fainelli wrote:
> On 09/13/2016 12:24 AM, Michal Marek wrote:
>> On Mon, Sep 12, 2016 at 05:12:15PM -0700, Florian Fainelli wrote:
>>> Hi,
>>>
>>> I have a root filesystem embedding filenames that look like these:
>>>
>>> /lib/data/<vid>:<pid>
>>>
>>> these are essentially files that can be matched against an USB
>>> vendor/product id in an easy way.
>>>
>>> Now, the fun part is that this is only a problem when doing the
>>> following (using OpenWrt/LEDE as a build system):
>>>
>>> 1:
>>> - set CONFIG_INITRAMFS_SOURCE=""
>>> - build kernel modules
>>> - build my user-space tools
>>> - build the kernel image
>>> - reconfigure the kernel to now use an initramfs
>>> - build the kernel w/ initramfs
>>>
>>> and then back to step 1 with the kernel build, would I hit this error:
>>>
>>> usr/Makefile:64: *** multiple target patterns.  Stop.
>> [...]
>>> Which sorts of make sense here because the file name contains a ":"
>>> which is not escaped, so GNU Make tries to interpret it.
>>>
>>> Now the part that does not quite make sense to me is why this file is
>>> even relevant here considering that the first thing we do is set
>>> CONFIG_INITRAMFS_SOURCE="" to disable the initramfs basically.
>>
>> It is possible that we read usr/Makefile twice for some reason. But the
>> real problem is the lack of escaping. Can you try the following
>> (untested) patch?
> 
> Can you submit an official patch for this? Thanks a lot!

The % escape is wrong. I'm trying to fix it or drop this escape.

Michal

[PATCH] initramfs: Escape colons in depfile

[Michal Marek]

Special characters are problematic in depfiles, but we can fix colons
easily.

Reported-by: Florian Fainelli <f.fainelli@gmail.com>
Signed-off-by: Michal Marek <mmarek@suse.com>
---
 scripts/gen_initramfs_list.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
index 17fa901418ae..0055b07b03b6 100755
--- a/scripts/gen_initramfs_list.sh
+++ b/scripts/gen_initramfs_list.sh
@@ -97,7 +97,10 @@ print_mtime() {
 }
 
 list_parse() {
-	[ ! -L "$1" ] && echo "$1 \\" || :
+	if [ -L "$1" ]; then
+		return
+	fi
+	echo "$1" | sed 's/:/\\:/g; s/$/ \\/'
 }
 
 # for each file print a line in following format
-- 
2.6.6

Re: [LEDE-DEV] [PATCH] initramfs: Escape colons in depfile

[John Crispin]

Hi,

this seems to be a backport for the kernel tree. please resend it as a
patch adding the actual patch that is being backported to the generic
targets patch folder. this would need to be done for 3.18, 4.1 and 4.4

	John

On 23/09/2016 10:35, Michal Marek wrote:
> Special characters are problematic in depfiles, but we can fix colons
> easily.
> 
> Reported-by: Florian Fainelli <f.fainelli@gmail.com>
> Signed-off-by: Michal Marek <mmarek@suse.com>
> ---
>  scripts/gen_initramfs_list.sh | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/gen_initramfs_list.sh b/scripts/gen_initramfs_list.sh
> index 17fa901418ae..0055b07b03b6 100755
> --- a/scripts/gen_initramfs_list.sh
> +++ b/scripts/gen_initramfs_list.sh
> @@ -97,7 +97,10 @@ print_mtime() {
>  }
>  
>  list_parse() {
> -	[ ! -L "$1" ] && echo "$1 \\" || :
> +	if [ -L "$1" ]; then
> +		return
> +	fi
> +	echo "$1" | sed 's/:/\\:/g; s/$/ \\/'
>  }
>  
>  # for each file print a line in following format
>