UBIFS errors since Linux 5.15

UBIFS errors since Linux 5.15

[Eddie James]

Hello,

We are updating some BMC (Aspeed AST2600, ARM32) systems that were on 
Linux 5.4 but observe consistent UBIFS errors with Linux 5.15 and later. 
The errors seem to vary; there is a sampling below.


We performed a git bisect and came up with the following commit, which 
of course is completely unrelated to UBIFS. Reverting it does prevent 
the issue... Does anyone have any suggestions?

commit 33fc9e26b7cb39f0d4219c875a2451802249c225
Author: Seth Jenkins <sethjenkins@google.com>
Date:   Thu Oct 27 11:36:52 2022 -0400

     mm: /proc/pid/smaps_rollup: fix no vma's null-deref

     Commit 258f669e7e88 ("mm: /proc/pid/smaps_rollup: convert to single 
value
     seq_file") introduced a null-deref if there are no vma's in the task in
     show_smaps_rollup.

     Fixes: 258f669e7e88 ("mm: /proc/pid/smaps_rollup: convert to single 
value seq_file")
     Signed-off-by: Seth Jenkins <sethjenkins@google.com>
     Reviewed-by: Alexey Dobriyan <adobriyan@gmail.com>
     Tested-by: Alexey Dobriyan <adobriyan@gmail.com>
     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>


Logs:

[    6.450395] UBIFS (ubi0:2): Mounting in unauthenticated mode
[    6.470547] UBIFS (ubi0:2): background thread "ubifs_bgt0_2" started, 
PID 64
[    6.478908] UBIFS (ubi0:2): recovery needed
[    7.299651] UBIFS (ubi0:2): recovery completed
[    7.304454] UBIFS (ubi0:2): UBIFS: mounted UBI device 0, volume 2, 
name "rwfs"
[    7.311869] UBIFS (ubi0:2): LEB size: 65408 bytes (63 KiB), min./max. 
I/O unit sizes: 8 bytes/256 bytes
[    7.321399] UBIFS (ubi0:2): FS size: 5690496 bytes (5 MiB, 87 LEBs), 
max 97 LEBs, journal size 523265 bytes (0 MiB, 6 LEBs)
[    7.332661] UBIFS (ubi0:2): reserved for root: 268775 bytes (262 KiB)
[    7.339141] UBIFS (ubi0:2): media format: w5/r0 (latest is w5/r0), 
UUID 7E9147BE-0964-4405-8FC7-12EF85CF0943, small LPT model
[    8.180486] UBIFS error (ubi0:2 pid 66): ubifs_check_node: bad magic 
0x101831, expected 0x6101831
[    8.189578] UBIFS error (ubi0:2 pid 66): ubifs_check_node: bad node 
at LEB 17:64664
[    8.197309] Not a node, first 24 bytes:
[    8.197336] 00000000: 31 18 10 00 a3 80 84 00 05 00 21 00 00 00 00 00 
80 00 00 00 00 02 00 00 1.........!.............
[    8.214233] CPU: 0 PID: 66 Comm: rm Not tainted 6.1.130-0c9470d #1
[    8.220464] Hardware name: Generic DT based system
[    8.225299]  unwind_backtrace from show_stack+0x18/0x1c
[    8.230611]  show_stack from dump_stack_lvl+0x24/0x2c
[    8.235739]  dump_stack_lvl from ubifs_check_node+0x258/0x280
[    8.241547]  ubifs_check_node from ubifs_read_node+0x18c/0x280
[    8.247438]  ubifs_read_node from ubifs_tnc_read_node+0x7c/0x1b0
[    8.253512]  ubifs_tnc_read_node from ubifs_tnc_locate.part.3+0xec/0x1d8
[    8.260282]  ubifs_tnc_locate.part.3 from ubifs_iget+0xb0/0x4cc
[    8.266263]  ubifs_iget from ubifs_lookup+0x1ec/0x2c4
[    8.271366]  ubifs_lookup from __lookup_slow+0xd4/0x104
[    8.276654]  __lookup_slow from walk_component+0x88/0xd4
[    8.282017]  walk_component from path_lookupat+0x88/0x110
[    8.287469]  path_lookupat from filename_lookup.part.16+0x48/0xac
[    8.293614]  filename_lookup.part.16 from vfs_statx+0x60/0x12c
[    8.299517]  vfs_statx from vfs_fstatat+0x54/0x70
[    8.304282]  vfs_fstatat from __do_sys_lstat64+0x2c/0x60
[    8.309656]  __do_sys_lstat64 from ret_fast_syscall+0x0/0x54
[    8.315370] Exception stack(0xbfbb9fa8 to 0xbfbb9ff0)
[    8.320461] 9fa0:                   7ef33faf 0000000c 7ef33faf 
7ef33cc0 7ef33cc0 00000077
[    8.328670] 9fc0: 7ef33faf 0000000c 00000000 000000c4 00000000 
00000000 4ea70fa4 00000000
[    8.336871] 9fe0: 000b34a0 7ef33cb8 00090268 4eb44178
[    8.342060] UBIFS error (ubi0:2 pid 66): ubifs_read_node: expected 
node type 0
[    8.349459] UBIFS error (ubi0:2 pid 66): ubifs_iget: failed to read 
inode 15929, error -117
[    8.357903] UBIFS error (ubi0:2 pid 66): ubifs_lookup: dead directory 
entry 'work', error -117
[    8.366682] UBIFS warning (ubi0:2 pid 66): ubifs_ro_mode: switched to 
read-only mode, error -11


[ 3739.678968] UBIFS error (ubi0:2 pid 13): ubifs_check_node: bad CRC: 
calculated 0x9179af29, read 0xa760f4f5
[ 3739.688901] UBIFS error (ubi0:2 pid 13): ubifs_check_node: bad node 
at LEB 56:48856
[ 3739.696772]  magic          0x6101831
[ 3739.700480]  crc            0xa760f4f5
[ 3739.704362]  node_type      1 (data node)
[ 3739.708463]  group_type     0 (no node group)
[ 3739.712855]  sqnum          119441
[ 3739.716384]  len            1633
[ 3739.719660]  key            (1338, data, 11)
[ 3739.724062]  size           4096
[ 3739.727329]  compr_typ      3
[ 3739.730386]  data size      1585
[ 3739.733760]  data (length = 1585):
[ 3740.458131] UBIFS error (ubi0:2 pid 13): ubifs_scan: bad node
[ 3740.464115] UBIFS error (ubi0:2 pid 13): ubifs_scanned_corruption: 
corruption at LEB 56:48856
[ 3740.472733] UBIFS error (ubi0:2 pid 13): ubifs_scanned_corruption: 
first 8192 bytes from LEB 56:48856
[ 3740.490666] UBIFS error (ubi0:2 pid 13): ubifs_scan: LEB 56 scanning 
failed


[10412.723111] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad magic 
0x6100830, expected 0x6101831
[10412.732436] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad node 
at LEB 13:60248
[10412.740386] Not a node, first 24 bytes:
[10412.740426] 00000000: 30 08 10 06 10 2c cb 50 22 1c 20 00 00 00 00 00 
80 00 00 00 09 00 00 00                          0....,.P". .............
[10412.757797] CPU: 0 PID: 297 Comm: login Not tainted 6.1.130-0c9470d #1
[10412.764391] Hardware name: Generic DT based system
[10412.769225]  unwind_backtrace from show_stack+0x18/0x1c
[10412.774551]  show_stack from dump_stack_lvl+0x24/0x2c
[10412.779689]  dump_stack_lvl from ubifs_check_node+0x258/0x280
[10412.785502]  ubifs_check_node from ubifs_read_node+0x18c/0x280
[10412.791392]  ubifs_read_node from ubifs_load_znode+0x90/0x47c
[10412.797210]  ubifs_load_znode from ubifs_lookup_level0+0x140/0x2c8
[10412.803460]  ubifs_lookup_level0 from ubifs_tnc_locate.part.3+0x5c/0x1d8
[10412.810227]  ubifs_tnc_locate.part.3 from ubifs_iget+0xb0/0x4cc
[10412.816207]  ubifs_iget from ubifs_lookup+0x1ec/0x2c4
[10412.821312]  ubifs_lookup from __lookup_slow+0xd4/0x104
[10412.826599]  __lookup_slow from walk_component+0x88/0xd4
[10412.831972]  walk_component from path_lookupat+0x88/0x110
[10412.837423]  path_lookupat from filename_lookup.part.16+0x48/0xac
[10412.843576]  filename_lookup.part.16 from vfs_statx+0x60/0x12c
[10412.849490]  vfs_statx from vfs_fstatat+0x54/0x70
[10412.854253]  vfs_fstatat from __do_sys_lstat64+0x2c/0x60
[10412.859628]  __do_sys_lstat64 from ret_fast_syscall+0x0/0x54
[10412.865342] Exception stack(0xd00a5fa8 to 0xd00a5ff0)
[10412.870442] 5fa0:                   7e98d758 ffffffff 76ed8d00 
7e98d5a0 7e98d5a0 7e98d758
[10412.878661] 5fc0: 7e98d758 ffffffff 00000000 000000c4 0026a7e8 
7e98d5a0 00270bc8 76ed85a8
[10412.886871] 5fe0: 76eea084 7e98d590 76ed8250 4eb44178
[10412.893099] UBIFS error (ubi0:2 pid 297): ubifs_read_node: expected 
node type 9
[10412.900645] UBIFS error (ubi0:2 pid 297): ubifs_iget: failed to read 
inode 173, error -117
[10412.909019] UBIFS error (ubi0:2 pid 297): ubifs_lookup: dead 
directory entry 'tallylog', error -117
[10412.929539] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad magic 
0x6100830, expected 0x6101831
[10412.938710] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad node 
at LEB 13:60248
[10412.946635] Not a node, first 24 bytes:
[10412.946674] 00000000: 30 08 10 06 10 2c cb 50 22 1c 20 00 00 00 00 00 
80 00 00 00 09 00 00 00                          0....,.P". .............

Re: UBIFS errors since Linux 5.15

[Richard Weinberger]

Eddie,

----- Ursprüngliche Mail -----
> Von: "Eddie James" <eajames@linux.ibm.com>
> An: "linux-mtd" <linux-mtd@lists.infradead.org>, "richard" <richard@nod.at>, "chengzhihao1" <chengzhihao1@huawei.com>,
> "OpenBMC Maillist" <openbmc@lists.ozlabs.org>, "linux-kernel" <linux-kernel@vger.kernel.org>, sethjenkins@google.com,
> "adobriyan" <adobriyan@gmail.com>
> Gesendet: Freitag, 7. November 2025 18:38:54
> Betreff: UBIFS errors since Linux 5.15

> Hello,
> 
> We are updating some BMC (Aspeed AST2600, ARM32) systems that were on
> Linux 5.4 but observe consistent UBIFS errors with Linux 5.15 and later.
> The errors seem to vary; there is a sampling below.
> 
> 
> We performed a git bisect and came up with the following commit, which
> of course is completely unrelated to UBIFS. Reverting it does prevent
> the issue... Does anyone have any suggestions?
> 
> commit 33fc9e26b7cb39f0d4219c875a2451802249c225
> Author: Seth Jenkins <sethjenkins@google.com>
> Date:   Thu Oct 27 11:36:52 2022 -0400

That's indeed strange.
Are you absolutely sure that this commit triggers the issue?
E.g. if you revert it on top of 5.15 the issue vanishes and when you
re-apply it, the issue is back?

Thanks,
//richard

Re: UBIFS errors since Linux 5.15

[Eddie James]

On 11/7/25 2:03 PM, Richard Weinberger wrote:
> Eddie,
>
> ----- Ursprüngliche Mail -----
>> Von: "Eddie James" <eajames@linux.ibm.com>
>> An: "linux-mtd" <linux-mtd@lists.infradead.org>, "richard" <richard@nod.at>, "chengzhihao1" <chengzhihao1@huawei.com>,
>> "OpenBMC Maillist" <openbmc@lists.ozlabs.org>, "linux-kernel" <linux-kernel@vger.kernel.org>, sethjenkins@google.com,
>> "adobriyan" <adobriyan@gmail.com>
>> Gesendet: Freitag, 7. November 2025 18:38:54
>> Betreff: UBIFS errors since Linux 5.15
>> Hello,
>>
>> We are updating some BMC (Aspeed AST2600, ARM32) systems that were on
>> Linux 5.4 but observe consistent UBIFS errors with Linux 5.15 and later.
>> The errors seem to vary; there is a sampling below.
>>
>>
>> We performed a git bisect and came up with the following commit, which
>> of course is completely unrelated to UBIFS. Reverting it does prevent
>> the issue... Does anyone have any suggestions?
>>
>> commit 33fc9e26b7cb39f0d4219c875a2451802249c225
>> Author: Seth Jenkins <sethjenkins@google.com>
>> Date:   Thu Oct 27 11:36:52 2022 -0400
> That's indeed strange.
> Are you absolutely sure that this commit triggers the issue?
> E.g. if you revert it on top of 5.15 the issue vanishes and when you
> re-apply it, the issue is back?


Thanks for your quick reply. Yes, that is the case; it's not very 
logical. We are currently running more extended tests with 5.15 with 
33fc9e26b reverted. I assume the potential for corruption is there 
without this commit too, but somehow it doesn't happen.


Eddie


>
> Thanks,
> //richard
>

Re: UBIFS errors since Linux 5.15

[Zhihao Cheng]

在 2025/11/8 1:38, Eddie James 写道:
> 
> Hello,
> 
> We are updating some BMC (Aspeed AST2600, ARM32) systems that were on 
> Linux 5.4 but observe consistent UBIFS errors with Linux 5.15 and later. 
> The errors seem to vary; there is a sampling below.
> 
> 
> We performed a git bisect and came up with the following commit, which 
> of course is completely unrelated to UBIFS. Reverting it does prevent 
> the issue... Does anyone have any suggestions?
> 
> commit 33fc9e26b7cb39f0d4219c875a2451802249c225
[...]
> [    8.180486] UBIFS error (ubi0:2 pid 66): ubifs_check_node: bad magic 
> 0x101831, expected 0x6101831

Hi, look here, the magic number 0x101831 vs 0x6101831, the difference is 
the highest bit(0 vs 6[0110])
> [    8.189578] UBIFS error (ubi0:2 pid 66): ubifs_check_node: bad node 
> at LEB 17:64664

[...]
> [ 3739.678968] UBIFS error (ubi0:2 pid 13): ubifs_check_node: bad CRC: 
> calculated 0x9179af29, read 0xa760f4f5

[...]
> [10412.723111] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad magic 
> 0x6100830, expected 0x6101831

and there, magic number 0x6100830 vs 0x6101831, the difference is lowest 
bit(0 vs 1).

So, I guess the crc error is caused by the similar reason, maybe there 
is something wrong with the nand driver or flash? According to the 
version information(5.15, and the unrelated commit 33fc9e26b7cb3), is it 
possible that there is a compatibility issue between your nand(bsp) 
driver with kernel nand common layer(which from different versions)?

> [10412.732436] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad node 
> at LEB 13:60248

Re: UBIFS errors since Linux 5.15

[Zhihao Cheng]

在 2025/11/8 11:03, Zhihao Cheng 写道:
> 在 2025/11/8 1:38, Eddie James 写道:
>>
>> Hello,
>>
>> We are updating some BMC (Aspeed AST2600, ARM32) systems that were on 
>> Linux 5.4 but observe consistent UBIFS errors with Linux 5.15 and 
>> later. The errors seem to vary; there is a sampling below.
>>
>>
>> We performed a git bisect and came up with the following commit, which 
>> of course is completely unrelated to UBIFS. Reverting it does prevent 
>> the issue... Does anyone have any suggestions?
>>
>> commit 33fc9e26b7cb39f0d4219c875a2451802249c225
> [...]
>> [    8.180486] UBIFS error (ubi0:2 pid 66): ubifs_check_node: bad 
>> magic 0x101831, expected 0x6101831
> 
> Hi, look here, the magic number 0x101831 vs 0x6101831, the difference is 
> the highest bit(0 vs 6[0110])
>> [    8.189578] UBIFS error (ubi0:2 pid 66): ubifs_check_node: bad node 
>> at LEB 17:64664
> 
> [...]
>> [ 3739.678968] UBIFS error (ubi0:2 pid 13): ubifs_check_node: bad CRC: 
>> calculated 0x9179af29, read 0xa760f4f5
> 
> [...]
>> [10412.723111] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad 
>> magic 0x6100830, expected 0x6101831
> 
> and there, magic number 0x6100830 vs 0x6101831, the difference is lowest 
> bit(0 vs 1).

Sorry, I need to make a correction, the differences are the lowest bit(0 
vs 1) and the middle bit (0 vs 1). The number of flipped bits is also two.
> 
> So, I guess the crc error is caused by the similar reason, maybe there 
> is something wrong with the nand driver or flash? According to the 
> version information(5.15, and the unrelated commit 33fc9e26b7cb3), is it 
> possible that there is a compatibility issue between your nand(bsp) 
> driver with kernel nand common layer(which from different versions)?
> 
>> [10412.732436] UBIFS error (ubi0:2 pid 297): ubifs_check_node: bad 
>> node at LEB 13:60248
> 
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/