Re: Executable loading issues with erofs on arm?

[Gao Xiang <hsiangkao@linux.alibaba.com>]

On 2025/7/8 23:22, Jan Kiszka wrote:
> On 08.07.25 17:12, Gao Xiang wrote:
>> Hi Jan,
>>
>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>> Hi all,
>>>>
>>>> for some days, I'm trying to understand if we have an integration issue
>>>> with erofs or rather some upstream bug. After playing with various
>>>> parameters, it rather looks like the latter:
>>>>
>>>> $ ls -l erofs-dir/
>>>> total 132
>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>> (from Debian bookworm)
>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
>>>> Build completed.
>>>> ------
>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>> Filesystem total inodes: 2
>>>> Filesystem total metadata blocks: 1
>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>
>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>>> target BeagleBone Black. When booting into init=/bin/sh, then running
>>>>
>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>> # /mnt/dash
>>>> Segmentation fault
>>>>
>>>> I once also got this:
>>>>
>>>> Alignment trap: not handling instruction 2b00 at [<004debc0>]
>>>> 8<--- cut here ---
>>>> Unhandled fault: alignment exception (0x001) at 0x000004d9
>>>> [000004d9] *pgd=00000000
>>>> Bus error
>>>>
>>>> All is fine if I
>>>>    - run the command once more
>>>>    - dump the file first (cat /mnt/dash > /dev/null; /mnt/dash)
>>>
>>> Forgot to mention: That first dump when done via md5sum or so actually
>>> gives the right checksum. So pure reading of the binary is also ok, just
>>> trying to load it for execution fails on the first attempt.
>>
>> Thanks for your report.  I rarely take care arm32 platform
>> because I don't have such setup.
>>
>> but could you share a reproducible rootfs image and
>> I wonder if qemu could reproduce this?
> 
> The image can be generated from isar-cip-core
> (https://gitlab.com/cip-project/cip-core/isar-cip-core), bbb image with
> swupdate extension and erofs as immutable rootfs. As I wrote, those will
> be 6.12 or 6.1 based, but I also injected a mainline kernel into that
> with the same result. But all that only helps if you have some
> beaglebone black in reach right now.

Could you check 5.4 lts, 5.15 lts, 5.10 lts if possible?

I wonder if it's a regression or it does not work from
the beginning. Anyway, I have no chance to look after arm32
due to lack of use cases and hardware resource.

> 
> The same configuration, just for qemuarm as target, unfortunately does
> not reproduce the issue.

That is too bad for me honestly because I do think it's much
easier for me to quickly shooting down with a reproducable
environment on my side...

> 
>>
>> Otherwise it's hard for me to debug this issue...
> 
> If you tell me how I could do that, I'm happy to instrument and analyze.
> I just have no understanding of erofs yet, specifically how reading
> files might be different from loading executables.

I have no idea too.  But it seems this issue is quite similar to
https://lore.kernel.org/r/76f0ea6f-d4c2-434a-8ca5-4bd93921209f@linux.alibaba.com/T/#t

I've given several ways to help finding the cause, but it
seems that data checksum is all good, and that issue is
still not resolved now either.

Thanks,
Gao Xiang

> 
> Jan
>