mnt_list corruption triggered during btrfs/326

mnt_list corruption triggered during btrfs/326

[Qu Wenruo]

Hi,

Although I know it's triggered from btrfs, but the mnt_list handling is
out of btrfs' control, so I'm here asking for some help.

[BUG]
With CONFIG_DEBUG_LIST and CONFIG_BUG_ON_DATA_CORRUPTION, and an
upstream 6.13-rc kernel, which has commit 951a3f59d268 ("btrfs: fix
mount failure due to remount races"), I can hit the following crash,
with varied frequency (from 1/4 to hundreds runs no crash):

[  303.356328] BTRFS: device fsid 6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
devid 1 transid 9 /dev/mapper/test-scratch1 (253:2) scanned by mount
(358060)
[  303.358614] BTRFS info (device dm-2): first mount of filesystem
6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
[  303.359475] BTRFS info (device dm-2): using crc32c (crc32c-intel)
checksum algorithm
[  303.360134] BTRFS info (device dm-2): using free-space-tree
[  313.264317] list_del corruption, ffff8fd48a7b2c90->prev is NULL
[  313.264966] ------------[ cut here ]------------
[  313.265402] kernel BUG at lib/list_debug.c:54!
[  313.265847] Oops: invalid opcode: 0000 [#1] PREEMPT SMP
[  313.266335] CPU: 4 UID: 0 PID: 370457 Comm: mount Kdump: loaded Not
tainted 6.13.0-rc4-custom+ #8
[  313.267252] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
Arch Linux 1.16.3-1-1 04/01/2014
[  313.268147] RIP: 0010:__list_del_entry_valid_or_report.cold+0x6d/0x6f
[  313.268777] Code: 05 77 a0 e8 4b 10 fd ff 0f 0b 48 89 fe 48 c7 c7 90
05 77 a0 e8 3a 10 fd ff 0f 0b 48 89 fe 48 c7 c7 60 05 77 a0 e8 29 10 fd
ff <0f> 0b 4c 89 ea be 01 00 00 00 4c 89 44 24 48 48 c7 c7 20 7c 2b a1
[  313.270493] RSP: 0018:ffffa7620d2b3a38 EFLAGS: 00010246
[  313.270960] RAX: 0000000000000033 RBX: ffff8fd48a7b2c00 RCX:
0000000000000000
[  313.271565] RDX: 0000000000000000 RSI: ffff8fd5f7c21900 RDI:
ffff8fd5f7c21900
[  313.272226] RBP: ffff8fd48a7b2c00 R08: 0000000000000000 R09:
0000000000000000
[  313.272895] R10: 74707572726f6320 R11: 6c65645f7473696c R12:
ffffa7620d2b3a58
[  313.273521] R13: ffff8fd48a7b2c00 R14: 0000000000000000 R15:
ffff8fd48a7b2c90
[  313.274138] FS:  00007f04740d4800(0000) GS:ffff8fd5f7c00000(0000)
knlGS:0000000000000000
[  313.274864] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  313.275392] CR2: 00007f0473ff6000 CR3: 000000011a8eb000 CR4:
0000000000750ef0
[  313.276084] PKRU: 55555554
[  313.276327] Call Trace:
[  313.276551]  <TASK>
[  313.276752]  ? __die_body.cold+0x19/0x28
[  313.277102]  ? die+0x2e/0x50
[  313.277699]  ? do_trap+0xc6/0x110
[  313.278033]  ? do_error_trap+0x6a/0x90
[  313.278401]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
[  313.278941]  ? exc_invalid_op+0x50/0x60
[  313.279308]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
[  313.279850]  ? asm_exc_invalid_op+0x1a/0x20
[  313.280241]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
[  313.280777]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
[  313.281285]  umount_tree+0xed/0x3c0
[  313.281589]  put_mnt_ns+0x51/0x90
[  313.281886]  mount_subtree+0x92/0x130
[  313.282205]  btrfs_get_tree+0x343/0x6b0 [btrfs]
[  313.282785]  vfs_get_tree+0x23/0xc0
[  313.283089]  vfs_cmd_create+0x59/0xd0
[  313.283406]  __do_sys_fsconfig+0x4eb/0x6b0
[  313.283764]  do_syscall_64+0x82/0x160
[  313.284085]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
[  313.284598]  ? __fs_parse+0x68/0x1b0
[  313.284929]  ? btrfs_parse_param+0x64/0x870 [btrfs]
[  313.285381]  ? vfs_parse_fs_param_source+0x20/0x90
[  313.285825]  ? __do_sys_fsconfig+0x1b8/0x6b0
[  313.286215]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
[  313.286719]  ? syscall_exit_to_user_mode+0x10/0x200
[  313.287151]  ? do_syscall_64+0x8e/0x160
[  313.287498]  ? vfs_fstatat+0x75/0xa0
[  313.287835]  ? __do_sys_newfstatat+0x56/0x90
[  313.288240]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
[  313.288749]  ? syscall_exit_to_user_mode+0x10/0x200
[  313.289188]  ? do_syscall_64+0x8e/0x160
[  313.289544]  ? do_syscall_64+0x8e/0x160
[  313.289892]  ? do_syscall_64+0x8e/0x160
[  313.290253]  ? syscall_exit_to_user_mode+0x10/0x200
[  313.290692]  ? do_syscall_64+0x8e/0x160
[  313.291034]  ? exc_page_fault+0x7e/0x180
[  313.291380]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  313.291845] RIP: 0033:0x7f04742a919e
[  313.292182] Code: 73 01 c3 48 8b 0d 72 3c 0f 00 f7 d8 64 89 01 48 83
c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f
05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 3c 0f 00 f7 d8 64 89 01 48
[  313.293830] RSP: 002b:00007ffc3df08df8 EFLAGS: 00000246 ORIG_RAX:
00000000000001af
[  313.294529] RAX: ffffffffffffffda RBX: 000056407e37aa00 RCX:
00007f04742a919e
[  313.295201] RDX: 0000000000000000 RSI: 0000000000000006 RDI:
0000000000000003
[  313.295864] RBP: 00007ffc3df08f40 R08: 0000000000000000 R09:
0000000000000001
[  313.296602] R10: 0000000000000000 R11: 0000000000000246 R12:
00007f0474423b00
[  313.297416] R13: 0000000000000000 R14: 000056407e37cbe0 R15:
00007f0474418561
[  313.298242]  </TASK>
[  313.298832] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
nf_tables binfmt_misc btrfs xor raid6_pq zstd_compress iTCO_wdt
intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus virtio_net joydev
net_failover lpc_ich virtio_balloon failover loop dm_multipath nfnetlink
vsock_loopback vmw_vsock_virtio_transport_common vsock zram
crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
polyval_generic ghash_clmulni_intel virtio_console sha512_ssse3
sha256_ssse3 bochs sha1_ssse3 virtio_blk serio_raw scsi_dh_rdac
scsi_dh_emc scsi_dh_alua fuse qemu_fw_cfg
[  313.304504] Dumping ftrace buffer:
[  313.304876]    (ftrace buffer empty)

[EARLY ANALYZE]

The offending line is the list_move() call inside unmount_tree().

With crash core dump, the offending mnt_list is totally corrupted:

crash> struct list_head ffff8fd48a7b2c90
struct list_head {
   next = 0x1,
   prev = 0x0
}

umount_tree() should be protected by @mount_lock seqlock, and
@namespace_sem rwsem.

I also checked other mnt_list users:

- commit_tree()
- do_umount()
- copy_tree()

They all hold write @mount_lock at least.

The only caller doesn't hold @mount_lock is iterate_mounts() but that's
only called from audit, and I'm not sure if audit is even involved in
this case.

So I ran out of ideas why this mnt_list can even happen.

Even if it's some btrfs' abuse, all mnt_list users are properly
protected thus it should not lead to such list corruption.

Any advice would be appreciated.

Thanks,
Qu

Re: mnt_list corruption triggered during btrfs/326

[Qu Wenruo]

在 2024/12/30 19:59, Qu Wenruo 写道:
> Hi,
>
> Although I know it's triggered from btrfs, but the mnt_list handling is
> out of btrfs' control, so I'm here asking for some help.
>
> [BUG]
> With CONFIG_DEBUG_LIST and CONFIG_BUG_ON_DATA_CORRUPTION, and an
> upstream 6.13-rc kernel, which has commit 951a3f59d268 ("btrfs: fix
> mount failure due to remount races"), I can hit the following crash,
> with varied frequency (from 1/4 to hundreds runs no crash):

There is also another WARNING triggered, without btrfs callback involved
at all:

[  192.688671] ------------[ cut here ]------------
[  192.690016] WARNING: CPU: 3 PID: 59747 at fs/mount.h:150
attach_recursive_mnt+0xc58/0x1260
[  192.692051] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
nf_tables binfmt_misc btrfs xor raid6_pq zstd_compress iTCO_wdt
intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus virtio_net
net_failover virtio_balloon lpc_ich failover joydev loop dm_multipath
nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock zram
crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3
virtio_blk virtio_console bochs serio_raw scsi_dh_rdac scsi_dh_emc
scsi_dh_alua fuse qemu_fw_cfg
[  192.707547] CPU: 3 UID: 0 PID: 59747 Comm: mount Kdump: loaded Not
tainted 6.13.0-rc4-custom+ #9
[  192.709485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
Arch Linux 1.16.3-1-1 04/01/2014
[  192.711601] RIP: 0010:attach_recursive_mnt+0xc58/0x1260
[  192.712725] Code: 85 c0 0f 85 79 ff ff ff 48 c7 c7 04 e7 00 8e 83 05
9c 18 28 03 01 e8 97 1d c8 01 31 f6 48 89 ef e8 dd c1 fe ff e9 9c f5 ff
ff <0f> 0b e9 48 f8 ff ff 48 8b 44 24 10 48 8d 78 20 48 b8 00 00 00 00
[  192.716521] RSP: 0018:ffff888105cafb68 EFLAGS: 00010246
[  192.717621] RAX: 0000000000001020 RBX: ffff88811cc24030 RCX:
ffffffff8ca0e8e5
[  192.719078] RDX: ffff888105cafbf0 RSI: ffff888118db0800 RDI:
ffff88811cc240f0
[  192.720313] RBP: ffff88811cc24000 R08: ffff88810f21a840 R09:
ffffed1020b95f62
[  192.721028] R10: 0000000000000003 R11: ffff88810a56e558 R12:
ffff88811cc24000
[  192.721718] R13: dffffc0000000000 R14: ffff88810f21a840 R15:
ffff888105cafbf0
[  192.722426] FS:  00007fdf69887800(0000) GS:ffff888236f80000(0000)
knlGS:0000000000000000
[  192.723229] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  192.723849] CR2: 00007fdf697a9000 CR3: 000000010d925000 CR4:
0000000000750ef0
[  192.724549] PKRU: 55555554
[  192.724860] Call Trace:
[  192.725101]  <TASK>
[  192.725311]  ? __warn.cold+0xb6/0x176
[  192.725672]  ? attach_recursive_mnt+0xc58/0x1260
[  192.726149]  ? report_bug+0x1f0/0x2a0
[  192.726520]  ? handle_bug+0x54/0x90
[  192.726895]  ? exc_invalid_op+0x17/0x40
[  192.727259]  ? asm_exc_invalid_op+0x1a/0x20
[  192.727664]  ? _raw_spin_lock+0x85/0xe0
[  192.728053]  ? attach_recursive_mnt+0xc58/0x1260
[  192.728501]  ? attach_recursive_mnt+0xb82/0x1260
[  192.728954]  ? _raw_spin_unlock+0xe/0x20
[  192.729330]  ? count_mounts+0x1e0/0x1e0
[  192.729703]  ? _raw_spin_lock+0x85/0xe0
[  192.730082]  ? _raw_write_lock_bh+0xe0/0xe0
[  192.730493]  do_move_mount+0x7a8/0x1a20
[  192.730871]  __do_sys_move_mount+0x7e2/0xcf0
[  192.731288]  ? syscall_exit_to_user_mode+0x10/0x200
[  192.731762]  ? do_syscall_64+0x8e/0x160
[  192.732180]  ? do_move_mount+0x1a20/0x1a20
[  192.732587]  do_syscall_64+0x82/0x160
[  192.732950]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
[  192.733500]  ? syscall_exit_to_user_mode+0x10/0x200
[  192.733977]  ? do_syscall_64+0x8e/0x160
[  192.734374]  ? from_kuid_munged+0x86/0x100
[  192.734765]  ? from_kuid+0xc0/0xc0
[  192.735115]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
[  192.735644]  ? syscall_exit_to_user_mode+0x10/0x200
[  192.736124]  ? do_syscall_64+0x8e/0x160
[  192.736487]  ? exc_page_fault+0x76/0xf0
[  192.736861]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  192.737352] RIP: 0033:0x7fdf69a5c3de
[  192.737725] Code: 73 01 c3 48 8b 0d 32 3a 0f 00 f7 d8 64 89 01 48 83
c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 ad 01 00 00 0f
05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 02 3a 0f 00 f7 d8 64 89 01 48
[  192.739455] RSP: 002b:00007ffd2b7c36d8 EFLAGS: 00000246 ORIG_RAX:
00000000000001ad
[  192.740176] RAX: ffffffffffffffda RBX: 0000557ac60e9a00 RCX:
00007fdf69a5c3de
[  192.740895] RDX: 00000000ffffff9c RSI: 00007fdf69bc9902 RDI:
0000000000000004
[  192.741578] RBP: 00007ffd2b7c3700 R08: 0000000000000004 R09:
0000000000000001
[  192.742247] R10: 0000557ac60e9e40 R11: 0000000000000246 R12:
00007fdf69bd6b00
[  192.742911] R13: 0000557ac60e9e40 R14: 0000557ac60ebbe0 R15:
0000000000000066
[  192.743573]  </TASK>
[  192.743803] ---[ end trace 0000000000000000 ]---

Thanks,
Qu

>
> [  303.356328] BTRFS: device fsid 6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
> devid 1 transid 9 /dev/mapper/test-scratch1 (253:2) scanned by mount
> (358060)
> [  303.358614] BTRFS info (device dm-2): first mount of filesystem
> 6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
> [  303.359475] BTRFS info (device dm-2): using crc32c (crc32c-intel)
> checksum algorithm
> [  303.360134] BTRFS info (device dm-2): using free-space-tree
> [  313.264317] list_del corruption, ffff8fd48a7b2c90->prev is NULL
> [  313.264966] ------------[ cut here ]------------
> [  313.265402] kernel BUG at lib/list_debug.c:54!
> [  313.265847] Oops: invalid opcode: 0000 [#1] PREEMPT SMP
> [  313.266335] CPU: 4 UID: 0 PID: 370457 Comm: mount Kdump: loaded Not
> tainted 6.13.0-rc4-custom+ #8
> [  313.267252] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> Arch Linux 1.16.3-1-1 04/01/2014
> [  313.268147] RIP: 0010:__list_del_entry_valid_or_report.cold+0x6d/0x6f
> [  313.268777] Code: 05 77 a0 e8 4b 10 fd ff 0f 0b 48 89 fe 48 c7 c7 90
> 05 77 a0 e8 3a 10 fd ff 0f 0b 48 89 fe 48 c7 c7 60 05 77 a0 e8 29 10 fd
> ff <0f> 0b 4c 89 ea be 01 00 00 00 4c 89 44 24 48 48 c7 c7 20 7c 2b a1
> [  313.270493] RSP: 0018:ffffa7620d2b3a38 EFLAGS: 00010246
> [  313.270960] RAX: 0000000000000033 RBX: ffff8fd48a7b2c00 RCX:
> 0000000000000000
> [  313.271565] RDX: 0000000000000000 RSI: ffff8fd5f7c21900 RDI:
> ffff8fd5f7c21900
> [  313.272226] RBP: ffff8fd48a7b2c00 R08: 0000000000000000 R09:
> 0000000000000000
> [  313.272895] R10: 74707572726f6320 R11: 6c65645f7473696c R12:
> ffffa7620d2b3a58
> [  313.273521] R13: ffff8fd48a7b2c00 R14: 0000000000000000 R15:
> ffff8fd48a7b2c90
> [  313.274138] FS:  00007f04740d4800(0000) GS:ffff8fd5f7c00000(0000)
> knlGS:0000000000000000
> [  313.274864] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  313.275392] CR2: 00007f0473ff6000 CR3: 000000011a8eb000 CR4:
> 0000000000750ef0
> [  313.276084] PKRU: 55555554
> [  313.276327] Call Trace:
> [  313.276551]  <TASK>
> [  313.276752]  ? __die_body.cold+0x19/0x28
> [  313.277102]  ? die+0x2e/0x50
> [  313.277699]  ? do_trap+0xc6/0x110
> [  313.278033]  ? do_error_trap+0x6a/0x90
> [  313.278401]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> [  313.278941]  ? exc_invalid_op+0x50/0x60
> [  313.279308]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> [  313.279850]  ? asm_exc_invalid_op+0x1a/0x20
> [  313.280241]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> [  313.280777]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> [  313.281285]  umount_tree+0xed/0x3c0
> [  313.281589]  put_mnt_ns+0x51/0x90
> [  313.281886]  mount_subtree+0x92/0x130
> [  313.282205]  btrfs_get_tree+0x343/0x6b0 [btrfs]
> [  313.282785]  vfs_get_tree+0x23/0xc0
> [  313.283089]  vfs_cmd_create+0x59/0xd0
> [  313.283406]  __do_sys_fsconfig+0x4eb/0x6b0
> [  313.283764]  do_syscall_64+0x82/0x160
> [  313.284085]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> [  313.284598]  ? __fs_parse+0x68/0x1b0
> [  313.284929]  ? btrfs_parse_param+0x64/0x870 [btrfs]
> [  313.285381]  ? vfs_parse_fs_param_source+0x20/0x90
> [  313.285825]  ? __do_sys_fsconfig+0x1b8/0x6b0
> [  313.286215]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> [  313.286719]  ? syscall_exit_to_user_mode+0x10/0x200
> [  313.287151]  ? do_syscall_64+0x8e/0x160
> [  313.287498]  ? vfs_fstatat+0x75/0xa0
> [  313.287835]  ? __do_sys_newfstatat+0x56/0x90
> [  313.288240]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> [  313.288749]  ? syscall_exit_to_user_mode+0x10/0x200
> [  313.289188]  ? do_syscall_64+0x8e/0x160
> [  313.289544]  ? do_syscall_64+0x8e/0x160
> [  313.289892]  ? do_syscall_64+0x8e/0x160
> [  313.290253]  ? syscall_exit_to_user_mode+0x10/0x200
> [  313.290692]  ? do_syscall_64+0x8e/0x160
> [  313.291034]  ? exc_page_fault+0x7e/0x180
> [  313.291380]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> [  313.291845] RIP: 0033:0x7f04742a919e
> [  313.292182] Code: 73 01 c3 48 8b 0d 72 3c 0f 00 f7 d8 64 89 01 48 83
> c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 3c 0f 00 f7 d8 64 89 01 48
> [  313.293830] RSP: 002b:00007ffc3df08df8 EFLAGS: 00000246 ORIG_RAX:
> 00000000000001af
> [  313.294529] RAX: ffffffffffffffda RBX: 000056407e37aa00 RCX:
> 00007f04742a919e
> [  313.295201] RDX: 0000000000000000 RSI: 0000000000000006 RDI:
> 0000000000000003
> [  313.295864] RBP: 00007ffc3df08f40 R08: 0000000000000000 R09:
> 0000000000000001
> [  313.296602] R10: 0000000000000000 R11: 0000000000000246 R12:
> 00007f0474423b00
> [  313.297416] R13: 0000000000000000 R14: 000056407e37cbe0 R15:
> 00007f0474418561
> [  313.298242]  </TASK>
> [  313.298832] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
> nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
> nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> nf_tables binfmt_misc btrfs xor raid6_pq zstd_compress iTCO_wdt
> intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus virtio_net joydev
> net_failover lpc_ich virtio_balloon failover loop dm_multipath nfnetlink
> vsock_loopback vmw_vsock_virtio_transport_common vsock zram
> crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
> polyval_generic ghash_clmulni_intel virtio_console sha512_ssse3
> sha256_ssse3 bochs sha1_ssse3 virtio_blk serio_raw scsi_dh_rdac
> scsi_dh_emc scsi_dh_alua fuse qemu_fw_cfg
> [  313.304504] Dumping ftrace buffer:
> [  313.304876]    (ftrace buffer empty)
>
> [EARLY ANALYZE]
>
> The offending line is the list_move() call inside unmount_tree().
>
> With crash core dump, the offending mnt_list is totally corrupted:
>
> crash> struct list_head ffff8fd48a7b2c90
> struct list_head {
>    next = 0x1,
>    prev = 0x0
> }
>
> umount_tree() should be protected by @mount_lock seqlock, and
> @namespace_sem rwsem.
>
> I also checked other mnt_list users:
>
> - commit_tree()
> - do_umount()
> - copy_tree()
>
> They all hold write @mount_lock at least.
>
> The only caller doesn't hold @mount_lock is iterate_mounts() but that's
> only called from audit, and I'm not sure if audit is even involved in
> this case.
>
> So I ran out of ideas why this mnt_list can even happen.
>
> Even if it's some btrfs' abuse, all mnt_list users are properly
> protected thus it should not lead to such list corruption.
>
> Any advice would be appreciated.
>
> Thanks,
> Qu
>

Re: mnt_list corruption triggered during btrfs/326

[Christian Brauner]

On Wed, Jan 01, 2025 at 07:05:10AM +1030, Qu Wenruo wrote:
> 
> 
> 在 2024/12/30 19:59, Qu Wenruo 写道:
> > Hi,
> > 
> > Although I know it's triggered from btrfs, but the mnt_list handling is
> > out of btrfs' control, so I'm here asking for some help.

Thanks for the report.

> > 
> > [BUG]
> > With CONFIG_DEBUG_LIST and CONFIG_BUG_ON_DATA_CORRUPTION, and an
> > upstream 6.13-rc kernel, which has commit 951a3f59d268 ("btrfs: fix
> > mount failure due to remount races"), I can hit the following crash,
> > with varied frequency (from 1/4 to hundreds runs no crash):
> 
> There is also another WARNING triggered, without btrfs callback involved
> at all:
> 
> [  192.688671] ------------[ cut here ]------------
> [  192.690016] WARNING: CPU: 3 PID: 59747 at fs/mount.h:150

This would indicate that move_from_ns() was called on a mount that isn't
attached to a mount namespace (anymore or never has).

Here's it's particularly peculiar because it looks like the warning is
caused by calling move_from_ns() when moving a mount from an anonymous
mount namespace in attach_recursive_mnt().

Can you please try and reproduce this with
commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")
from the vfs-6.14.mount branch in
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git ?

> attach_recursive_mnt+0xc58/0x1260
> [  192.692051] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
> nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
> nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> nf_tables binfmt_misc btrfs xor raid6_pq zstd_compress iTCO_wdt
> intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus virtio_net
> net_failover virtio_balloon lpc_ich failover joydev loop dm_multipath
> nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock zram
> crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
> polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3
> virtio_blk virtio_console bochs serio_raw scsi_dh_rdac scsi_dh_emc
> scsi_dh_alua fuse qemu_fw_cfg
> [  192.707547] CPU: 3 UID: 0 PID: 59747 Comm: mount Kdump: loaded Not
> tainted 6.13.0-rc4-custom+ #9
> [  192.709485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> Arch Linux 1.16.3-1-1 04/01/2014
> [  192.711601] RIP: 0010:attach_recursive_mnt+0xc58/0x1260
> [  192.712725] Code: 85 c0 0f 85 79 ff ff ff 48 c7 c7 04 e7 00 8e 83 05
> 9c 18 28 03 01 e8 97 1d c8 01 31 f6 48 89 ef e8 dd c1 fe ff e9 9c f5 ff
> ff <0f> 0b e9 48 f8 ff ff 48 8b 44 24 10 48 8d 78 20 48 b8 00 00 00 00
> [  192.716521] RSP: 0018:ffff888105cafb68 EFLAGS: 00010246
> [  192.717621] RAX: 0000000000001020 RBX: ffff88811cc24030 RCX:
> ffffffff8ca0e8e5
> [  192.719078] RDX: ffff888105cafbf0 RSI: ffff888118db0800 RDI:
> ffff88811cc240f0
> [  192.720313] RBP: ffff88811cc24000 R08: ffff88810f21a840 R09:
> ffffed1020b95f62
> [  192.721028] R10: 0000000000000003 R11: ffff88810a56e558 R12:
> ffff88811cc24000
> [  192.721718] R13: dffffc0000000000 R14: ffff88810f21a840 R15:
> ffff888105cafbf0
> [  192.722426] FS:  00007fdf69887800(0000) GS:ffff888236f80000(0000)
> knlGS:0000000000000000
> [  192.723229] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  192.723849] CR2: 00007fdf697a9000 CR3: 000000010d925000 CR4:
> 0000000000750ef0
> [  192.724549] PKRU: 55555554
> [  192.724860] Call Trace:
> [  192.725101]  <TASK>
> [  192.725311]  ? __warn.cold+0xb6/0x176
> [  192.725672]  ? attach_recursive_mnt+0xc58/0x1260
> [  192.726149]  ? report_bug+0x1f0/0x2a0
> [  192.726520]  ? handle_bug+0x54/0x90
> [  192.726895]  ? exc_invalid_op+0x17/0x40
> [  192.727259]  ? asm_exc_invalid_op+0x1a/0x20
> [  192.727664]  ? _raw_spin_lock+0x85/0xe0
> [  192.728053]  ? attach_recursive_mnt+0xc58/0x1260
> [  192.728501]  ? attach_recursive_mnt+0xb82/0x1260
> [  192.728954]  ? _raw_spin_unlock+0xe/0x20
> [  192.729330]  ? count_mounts+0x1e0/0x1e0
> [  192.729703]  ? _raw_spin_lock+0x85/0xe0
> [  192.730082]  ? _raw_write_lock_bh+0xe0/0xe0
> [  192.730493]  do_move_mount+0x7a8/0x1a20
> [  192.730871]  __do_sys_move_mount+0x7e2/0xcf0
> [  192.731288]  ? syscall_exit_to_user_mode+0x10/0x200
> [  192.731762]  ? do_syscall_64+0x8e/0x160
> [  192.732180]  ? do_move_mount+0x1a20/0x1a20
> [  192.732587]  do_syscall_64+0x82/0x160
> [  192.732950]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> [  192.733500]  ? syscall_exit_to_user_mode+0x10/0x200
> [  192.733977]  ? do_syscall_64+0x8e/0x160
> [  192.734374]  ? from_kuid_munged+0x86/0x100
> [  192.734765]  ? from_kuid+0xc0/0xc0
> [  192.735115]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> [  192.735644]  ? syscall_exit_to_user_mode+0x10/0x200
> [  192.736124]  ? do_syscall_64+0x8e/0x160
> [  192.736487]  ? exc_page_fault+0x76/0xf0
> [  192.736861]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> [  192.737352] RIP: 0033:0x7fdf69a5c3de
> [  192.737725] Code: 73 01 c3 48 8b 0d 32 3a 0f 00 f7 d8 64 89 01 48 83
> c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 ad 01 00 00 0f
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 02 3a 0f 00 f7 d8 64 89 01 48
> [  192.739455] RSP: 002b:00007ffd2b7c36d8 EFLAGS: 00000246 ORIG_RAX:
> 00000000000001ad
> [  192.740176] RAX: ffffffffffffffda RBX: 0000557ac60e9a00 RCX:
> 00007fdf69a5c3de
> [  192.740895] RDX: 00000000ffffff9c RSI: 00007fdf69bc9902 RDI:
> 0000000000000004
> [  192.741578] RBP: 00007ffd2b7c3700 R08: 0000000000000004 R09:
> 0000000000000001
> [  192.742247] R10: 0000557ac60e9e40 R11: 0000000000000246 R12:
> 00007fdf69bd6b00
> [  192.742911] R13: 0000557ac60e9e40 R14: 0000557ac60ebbe0 R15:
> 0000000000000066
> [  192.743573]  </TASK>
> [  192.743803] ---[ end trace 0000000000000000 ]---
> 
> Thanks,
> Qu
> 
> > 
> > [  303.356328] BTRFS: device fsid 6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
> > devid 1 transid 9 /dev/mapper/test-scratch1 (253:2) scanned by mount
> > (358060)
> > [  303.358614] BTRFS info (device dm-2): first mount of filesystem
> > 6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
> > [  303.359475] BTRFS info (device dm-2): using crc32c (crc32c-intel)
> > checksum algorithm
> > [  303.360134] BTRFS info (device dm-2): using free-space-tree
> > [  313.264317] list_del corruption, ffff8fd48a7b2c90->prev is NULL
> > [  313.264966] ------------[ cut here ]------------
> > [  313.265402] kernel BUG at lib/list_debug.c:54!
> > [  313.265847] Oops: invalid opcode: 0000 [#1] PREEMPT SMP
> > [  313.266335] CPU: 4 UID: 0 PID: 370457 Comm: mount Kdump: loaded Not
> > tainted 6.13.0-rc4-custom+ #8
> > [  313.267252] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> > Arch Linux 1.16.3-1-1 04/01/2014
> > [  313.268147] RIP: 0010:__list_del_entry_valid_or_report.cold+0x6d/0x6f
> > [  313.268777] Code: 05 77 a0 e8 4b 10 fd ff 0f 0b 48 89 fe 48 c7 c7 90
> > 05 77 a0 e8 3a 10 fd ff 0f 0b 48 89 fe 48 c7 c7 60 05 77 a0 e8 29 10 fd
> > ff <0f> 0b 4c 89 ea be 01 00 00 00 4c 89 44 24 48 48 c7 c7 20 7c 2b a1
> > [  313.270493] RSP: 0018:ffffa7620d2b3a38 EFLAGS: 00010246
> > [  313.270960] RAX: 0000000000000033 RBX: ffff8fd48a7b2c00 RCX:
> > 0000000000000000
> > [  313.271565] RDX: 0000000000000000 RSI: ffff8fd5f7c21900 RDI:
> > ffff8fd5f7c21900
> > [  313.272226] RBP: ffff8fd48a7b2c00 R08: 0000000000000000 R09:
> > 0000000000000000
> > [  313.272895] R10: 74707572726f6320 R11: 6c65645f7473696c R12:
> > ffffa7620d2b3a58
> > [  313.273521] R13: ffff8fd48a7b2c00 R14: 0000000000000000 R15:
> > ffff8fd48a7b2c90
> > [  313.274138] FS:  00007f04740d4800(0000) GS:ffff8fd5f7c00000(0000)
> > knlGS:0000000000000000
> > [  313.274864] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  313.275392] CR2: 00007f0473ff6000 CR3: 000000011a8eb000 CR4:
> > 0000000000750ef0
> > [  313.276084] PKRU: 55555554
> > [  313.276327] Call Trace:
> > [  313.276551]  <TASK>
> > [  313.276752]  ? __die_body.cold+0x19/0x28
> > [  313.277102]  ? die+0x2e/0x50
> > [  313.277699]  ? do_trap+0xc6/0x110
> > [  313.278033]  ? do_error_trap+0x6a/0x90
> > [  313.278401]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > [  313.278941]  ? exc_invalid_op+0x50/0x60
> > [  313.279308]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > [  313.279850]  ? asm_exc_invalid_op+0x1a/0x20
> > [  313.280241]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > [  313.280777]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > [  313.281285]  umount_tree+0xed/0x3c0
> > [  313.281589]  put_mnt_ns+0x51/0x90
> > [  313.281886]  mount_subtree+0x92/0x130
> > [  313.282205]  btrfs_get_tree+0x343/0x6b0 [btrfs]
> > [  313.282785]  vfs_get_tree+0x23/0xc0
> > [  313.283089]  vfs_cmd_create+0x59/0xd0
> > [  313.283406]  __do_sys_fsconfig+0x4eb/0x6b0
> > [  313.283764]  do_syscall_64+0x82/0x160
> > [  313.284085]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > [  313.284598]  ? __fs_parse+0x68/0x1b0
> > [  313.284929]  ? btrfs_parse_param+0x64/0x870 [btrfs]
> > [  313.285381]  ? vfs_parse_fs_param_source+0x20/0x90
> > [  313.285825]  ? __do_sys_fsconfig+0x1b8/0x6b0
> > [  313.286215]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > [  313.286719]  ? syscall_exit_to_user_mode+0x10/0x200
> > [  313.287151]  ? do_syscall_64+0x8e/0x160
> > [  313.287498]  ? vfs_fstatat+0x75/0xa0
> > [  313.287835]  ? __do_sys_newfstatat+0x56/0x90
> > [  313.288240]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > [  313.288749]  ? syscall_exit_to_user_mode+0x10/0x200
> > [  313.289188]  ? do_syscall_64+0x8e/0x160
> > [  313.289544]  ? do_syscall_64+0x8e/0x160
> > [  313.289892]  ? do_syscall_64+0x8e/0x160
> > [  313.290253]  ? syscall_exit_to_user_mode+0x10/0x200
> > [  313.290692]  ? do_syscall_64+0x8e/0x160
> > [  313.291034]  ? exc_page_fault+0x7e/0x180
> > [  313.291380]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> > [  313.291845] RIP: 0033:0x7f04742a919e
> > [  313.292182] Code: 73 01 c3 48 8b 0d 72 3c 0f 00 f7 d8 64 89 01 48 83
> > c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f
> > 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 3c 0f 00 f7 d8 64 89 01 48
> > [  313.293830] RSP: 002b:00007ffc3df08df8 EFLAGS: 00000246 ORIG_RAX:
> > 00000000000001af
> > [  313.294529] RAX: ffffffffffffffda RBX: 000056407e37aa00 RCX:
> > 00007f04742a919e
> > [  313.295201] RDX: 0000000000000000 RSI: 0000000000000006 RDI:
> > 0000000000000003
> > [  313.295864] RBP: 00007ffc3df08f40 R08: 0000000000000000 R09:
> > 0000000000000001
> > [  313.296602] R10: 0000000000000000 R11: 0000000000000246 R12:
> > 00007f0474423b00
> > [  313.297416] R13: 0000000000000000 R14: 000056407e37cbe0 R15:
> > 00007f0474418561
> > [  313.298242]  </TASK>
> > [  313.298832] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
> > nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
> > nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> > nf_tables binfmt_misc btrfs xor raid6_pq zstd_compress iTCO_wdt
> > intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus virtio_net joydev
> > net_failover lpc_ich virtio_balloon failover loop dm_multipath nfnetlink
> > vsock_loopback vmw_vsock_virtio_transport_common vsock zram
> > crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
> > polyval_generic ghash_clmulni_intel virtio_console sha512_ssse3
> > sha256_ssse3 bochs sha1_ssse3 virtio_blk serio_raw scsi_dh_rdac
> > scsi_dh_emc scsi_dh_alua fuse qemu_fw_cfg
> > [  313.304504] Dumping ftrace buffer:
> > [  313.304876]    (ftrace buffer empty)
> > 
> > [EARLY ANALYZE]
> > 
> > The offending line is the list_move() call inside unmount_tree().
> > 
> > With crash core dump, the offending mnt_list is totally corrupted:
> > 
> > crash> struct list_head ffff8fd48a7b2c90
> > struct list_head {
> >    next = 0x1,
> >    prev = 0x0
> > }
> > 
> > umount_tree() should be protected by @mount_lock seqlock, and
> > @namespace_sem rwsem.
> > 
> > I also checked other mnt_list users:
> > 
> > - commit_tree()
> > - do_umount()
> > - copy_tree()
> > 
> > They all hold write @mount_lock at least.
> > 
> > The only caller doesn't hold @mount_lock is iterate_mounts() but that's
> > only called from audit, and I'm not sure if audit is even involved in
> > this case.

This is fine as audit creates a private copy of the mount tree it is
interested in. The mount tree is not visible to other callers anymore.

> > 
> > So I ran out of ideas why this mnt_list can even happen.
> > 
> > Even if it's some btrfs' abuse, all mnt_list users are properly
> > protected thus it should not lead to such list corruption.
> > 
> > Any advice would be appreciated.
> > 
> > Thanks,
> > Qu
> > 
> 

Re: mnt_list corruption triggered during btrfs/326

[Qu Wenruo]

在 2025/1/4 21:56, Christian Brauner 写道:
> On Wed, Jan 01, 2025 at 07:05:10AM +1030, Qu Wenruo wrote:
>>
>>
>> 在 2024/12/30 19:59, Qu Wenruo 写道:
>>> Hi,
>>>
>>> Although I know it's triggered from btrfs, but the mnt_list handling is
>>> out of btrfs' control, so I'm here asking for some help.
> 
> Thanks for the report.
> 
>>>
>>> [BUG]
>>> With CONFIG_DEBUG_LIST and CONFIG_BUG_ON_DATA_CORRUPTION, and an
>>> upstream 6.13-rc kernel, which has commit 951a3f59d268 ("btrfs: fix
>>> mount failure due to remount races"), I can hit the following crash,
>>> with varied frequency (from 1/4 to hundreds runs no crash):
>>
>> There is also another WARNING triggered, without btrfs callback involved
>> at all:
>>
>> [  192.688671] ------------[ cut here ]------------
>> [  192.690016] WARNING: CPU: 3 PID: 59747 at fs/mount.h:150
> 
> This would indicate that move_from_ns() was called on a mount that isn't
> attached to a mount namespace (anymore or never has).
> 
> Here's it's particularly peculiar because it looks like the warning is
> caused by calling move_from_ns() when moving a mount from an anonymous
> mount namespace in attach_recursive_mnt().
> 
> Can you please try and reproduce this with
> commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")
> from the vfs-6.14.mount branch in
> https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git ?
> 

After the initial 1000 runs (with 951a3f59d268 ("btrfs: fix mount 
failure due to remount races") cherry picked, or it won't pass that test 
case), there is no crash nor warning so far.

It's already the best run so far, but I'll keep it running for another 
day or so just to be extra safe.

So I guess the offending commit is 2eea9ce4310d ("mounts: keep list of 
mounts in an rbtree")?
Putting a list and rb_tree into a union indeed seems a little dangerous, 
sorry I didn't notice that earlier, but my vmcore indeed show a 
seemingly valid mnt_node (color = 1, both left/right are NULL).

Thanks a lot for the fix, and it's really a huge relief that it's not 
something inside btrfs causing the bug.

Thanks,
Qu

[...]
>>>
>>> The only caller doesn't hold @mount_lock is iterate_mounts() but that's
>>> only called from audit, and I'm not sure if audit is even involved in
>>> this case.
> 
> This is fine as audit creates a private copy of the mount tree it is
> interested in. The mount tree is not visible to other callers anymore.
> 
>>>
>>> So I ran out of ideas why this mnt_list can even happen.
>>>
>>> Even if it's some btrfs' abuse, all mnt_list users are properly
>>> protected thus it should not lead to such list corruption.
>>>
>>> Any advice would be appreciated.
>>>
>>> Thanks,
>>> Qu
>>>
>>
> 

Re: mnt_list corruption triggered during btrfs/326

[Daniel Vacek]

On Sat, 4 Jan 2025 at 23:26, Qu Wenruo <wqu@suse.com> wrote:
>
>
>
> 在 2025/1/4 21:56, Christian Brauner 写道:
> > On Wed, Jan 01, 2025 at 07:05:10AM +1030, Qu Wenruo wrote:
> >>
> >>
> >> 在 2024/12/30 19:59, Qu Wenruo 写道:
> >>> Hi,
> >>>
> >>> Although I know it's triggered from btrfs, but the mnt_list handling is
> >>> out of btrfs' control, so I'm here asking for some help.
> >
> > Thanks for the report.
> >
> >>>
> >>> [BUG]
> >>> With CONFIG_DEBUG_LIST and CONFIG_BUG_ON_DATA_CORRUPTION, and an
> >>> upstream 6.13-rc kernel, which has commit 951a3f59d268 ("btrfs: fix
> >>> mount failure due to remount races"), I can hit the following crash,
> >>> with varied frequency (from 1/4 to hundreds runs no crash):
> >>
> >> There is also another WARNING triggered, without btrfs callback involved
> >> at all:
> >>
> >> [  192.688671] ------------[ cut here ]------------
> >> [  192.690016] WARNING: CPU: 3 PID: 59747 at fs/mount.h:150
> >
> > This would indicate that move_from_ns() was called on a mount that isn't
> > attached to a mount namespace (anymore or never has).
> >
> > Here's it's particularly peculiar because it looks like the warning is
> > caused by calling move_from_ns() when moving a mount from an anonymous
> > mount namespace in attach_recursive_mnt().
> >
> > Can you please try and reproduce this with
> > commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")
> > from the vfs-6.14.mount branch in
> > https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git ?
> >
>
> After the initial 1000 runs (with 951a3f59d268 ("btrfs: fix mount
> failure due to remount races") cherry picked, or it won't pass that test
> case), there is no crash nor warning so far.
>
> It's already the best run so far, but I'll keep it running for another
> day or so just to be extra safe.
>
> So I guess the offending commit is 2eea9ce4310d ("mounts: keep list of
> mounts in an rbtree")?

This one was merged in v6.8 - why would it cause crashes only now?

> Putting a list and rb_tree into a union indeed seems a little dangerous,
> sorry I didn't notice that earlier, but my vmcore indeed show a
> seemingly valid mnt_node (color = 1, both left/right are NULL).

The union seems fine to me as long as the `MNT_ONRB` bit stays
consistent. The crashes (nor warnings) are simply caused by the flag
missing where it should have been set.

--nX

> Thanks a lot for the fix, and it's really a huge relief that it's not
> something inside btrfs causing the bug.
>
> Thanks,
> Qu
>
> [...]
> >>>
> >>> The only caller doesn't hold @mount_lock is iterate_mounts() but that's
> >>> only called from audit, and I'm not sure if audit is even involved in
> >>> this case.
> >
> > This is fine as audit creates a private copy of the mount tree it is
> > interested in. The mount tree is not visible to other callers anymore.
> >
> >>>
> >>> So I ran out of ideas why this mnt_list can even happen.
> >>>
> >>> Even if it's some btrfs' abuse, all mnt_list users are properly
> >>> protected thus it should not lead to such list corruption.
> >>>
> >>> Any advice would be appreciated.
> >>>
> >>> Thanks,
> >>> Qu
> >>>
> >>
> >
>
>

Re: mnt_list corruption triggered during btrfs/326

[Qu Wenruo]

在 2025/1/7 08:20, Daniel Vacek 写道:
> On Sat, 4 Jan 2025 at 23:26, Qu Wenruo <wqu@suse.com> wrote:
>>
>>
>>
>> 在 2025/1/4 21:56, Christian Brauner 写道:
>>> On Wed, Jan 01, 2025 at 07:05:10AM +1030, Qu Wenruo wrote:
>>>>
>>>>
>>>> 在 2024/12/30 19:59, Qu Wenruo 写道:
>>>>> Hi,
>>>>>
>>>>> Although I know it's triggered from btrfs, but the mnt_list handling is
>>>>> out of btrfs' control, so I'm here asking for some help.
>>>
>>> Thanks for the report.
>>>
>>>>>
>>>>> [BUG]
>>>>> With CONFIG_DEBUG_LIST and CONFIG_BUG_ON_DATA_CORRUPTION, and an
>>>>> upstream 6.13-rc kernel, which has commit 951a3f59d268 ("btrfs: fix
>>>>> mount failure due to remount races"), I can hit the following crash,
>>>>> with varied frequency (from 1/4 to hundreds runs no crash):
>>>>
>>>> There is also another WARNING triggered, without btrfs callback involved
>>>> at all:
>>>>
>>>> [  192.688671] ------------[ cut here ]------------
>>>> [  192.690016] WARNING: CPU: 3 PID: 59747 at fs/mount.h:150
>>>
>>> This would indicate that move_from_ns() was called on a mount that isn't
>>> attached to a mount namespace (anymore or never has).
>>>
>>> Here's it's particularly peculiar because it looks like the warning is
>>> caused by calling move_from_ns() when moving a mount from an anonymous
>>> mount namespace in attach_recursive_mnt().
>>>
>>> Can you please try and reproduce this with
>>> commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")
>>> from the vfs-6.14.mount branch in
>>> https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git ?
>>>
>>
>> After the initial 1000 runs (with 951a3f59d268 ("btrfs: fix mount
>> failure due to remount races") cherry picked, or it won't pass that test
>> case), there is no crash nor warning so far.
>>
>> It's already the best run so far, but I'll keep it running for another
>> day or so just to be extra safe.
>>
>> So I guess the offending commit is 2eea9ce4310d ("mounts: keep list of
>> mounts in an rbtree")?
> 
> This one was merged in v6.8 - why would it cause crashes only now?

Because in v6.8 btrfs also migrated to the new mount API, which caused 
the ro/rw mount race which can fail the mount.

That's exactly why the test case is introduced.

Before the recent ro/rw mount fix, the test case won't go that far but 
error out early so we don't have enough loops to trigger the bug.

> 
>> Putting a list and rb_tree into a union indeed seems a little dangerous,
>> sorry I didn't notice that earlier, but my vmcore indeed show a
>> seemingly valid mnt_node (color = 1, both left/right are NULL).
> 
> The union seems fine to me as long as the `MNT_ONRB` bit stays
> consistent. The crashes (nor warnings) are simply caused by the flag
> missing where it should have been set.

That also means the mnt_flag needs to be properly protected, at least 
with the same level of mnt_list/mnt_node.

But a lot of time such flag is atomically accessed using 
test/set/clear_bit(), without the same level of lock protection.
So my current uneducated guess is, there is a race window where the flag 
and member got de-synced.

Thus it's not as safe as a non-unioned member.

Thanks,
Qu

> 
> --nX
> 
>> Thanks a lot for the fix, and it's really a huge relief that it's not
>> something inside btrfs causing the bug.
>>
>> Thanks,
>> Qu
>>
>> [...]
>>>>>
>>>>> The only caller doesn't hold @mount_lock is iterate_mounts() but that's
>>>>> only called from audit, and I'm not sure if audit is even involved in
>>>>> this case.
>>>
>>> This is fine as audit creates a private copy of the mount tree it is
>>> interested in. The mount tree is not visible to other callers anymore.
>>>
>>>>>
>>>>> So I ran out of ideas why this mnt_list can even happen.
>>>>>
>>>>> Even if it's some btrfs' abuse, all mnt_list users are properly
>>>>> protected thus it should not lead to such list corruption.
>>>>>
>>>>> Any advice would be appreciated.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>
>>>
>>
>>

Re: mnt_list corruption triggered during btrfs/326

[Daniel Vacek]

On Mon, 6 Jan 2025 at 23:00, Qu Wenruo <wqu@suse.com> wrote:
> 在 2025/1/7 08:20, Daniel Vacek 写道:
> > On Sat, 4 Jan 2025 at 23:26, Qu Wenruo <wqu@suse.com> wrote:
> >> 在 2025/1/4 21:56, Christian Brauner 写道:
> >>> Can you please try and reproduce this with
> >>> commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")
> >>> from the vfs-6.14.mount branch in
> >>> https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git ?
> >>>
> >>
> >> After the initial 1000 runs (with 951a3f59d268 ("btrfs: fix mount
> >> failure due to remount races") cherry picked, or it won't pass that test
> >> case), there is no crash nor warning so far.
> >>
> >> It's already the best run so far, but I'll keep it running for another
> >> day or so just to be extra safe.
> >>
> >> So I guess the offending commit is 2eea9ce4310d ("mounts: keep list of
> >> mounts in an rbtree")?
> >
> > This one was merged in v6.8 - why would it cause crashes only now?
>
> Because in v6.8 btrfs also migrated to the new mount API, which caused
> the ro/rw mount race which can fail the mount.
>
> That's exactly why the test case is introduced.
>
> Before the recent ro/rw mount fix, the test case won't go that far but
> error out early so we don't have enough loops to trigger the bug.

So the bug has been there since v6.8, IIUC. In that case I think
211364bef430 ("fs: kill MNT_ONRB") should add Fixes: 2eea9ce4310d
("mounts: keep list of mounts in an rbtree") and CC stable?

--nX

Re: mnt_list corruption triggered during btrfs/326

[Daniel Vacek]

On Sat, 4 Jan 2025 at 12:26, Christian Brauner <brauner@kernel.org> wrote:
>
> On Wed, Jan 01, 2025 at 07:05:10AM +1030, Qu Wenruo wrote:
> >
> >
> > 在 2024/12/30 19:59, Qu Wenruo 写道:
> > > Hi,
> > >
> > > Although I know it's triggered from btrfs, but the mnt_list handling is
> > > out of btrfs' control, so I'm here asking for some help.
>
> Thanks for the report.
>
> > >
> > > [BUG]
> > > With CONFIG_DEBUG_LIST and CONFIG_BUG_ON_DATA_CORRUPTION, and an
> > > upstream 6.13-rc kernel, which has commit 951a3f59d268 ("btrfs: fix
> > > mount failure due to remount races"), I can hit the following crash,
> > > with varied frequency (from 1/4 to hundreds runs no crash):
> >
> > There is also another WARNING triggered, without btrfs callback involved
> > at all:
> >
> > [  192.688671] ------------[ cut here ]------------
> > [  192.690016] WARNING: CPU: 3 PID: 59747 at fs/mount.h:150
>
> This would indicate that move_from_ns() was called on a mount that isn't
> attached to a mount namespace (anymore or never has).

Hi, I think the warning really means the mount just misses the
`MNT_ONRB` flag which actually should have been set. But for some
reason it is not.

And exactly the same applies to the reported crash.

```
1763         /* Gather the mounts to umount */
1764         for (p = mnt; p; p = next_mnt(p, mnt)) {
1765                 p->mnt.mnt_flags |= MNT_UMOUNT;
1766                 if (p->mnt.mnt_flags & MNT_ONRB)
1767                         move_from_ns(p, &tmp_list);
1768                 else
1769                         list_move(&p->mnt_list, &tmp_list);
1770         }
```

Here the missing flag caused the crash in `list_move()` while
`move_from_ns()` should have been executed instead. All the evidence
supports this scenario. Due to the condition on line 1766 we got a
crash instead of the warning this time.

> Here's it's particularly peculiar because it looks like the warning is
> caused by calling move_from_ns() when moving a mount from an anonymous
> mount namespace in attach_recursive_mnt().

Indeed the crash happened with a detached mount in an anonymous mount namespace:

```
crash> mount.mnt_parent,mnt_ns 0xffff88810b9d0180
  mnt_parent = 0xffff88810b9d0180,    <<< Detached/root mount.
  mnt_ns = 0xffff888103a12600,

crash> mnt_namespace.seq,mnt_ns_tree_node.__rb_parent_color 0xffff888103a12600
  seq = 0x0,    <<< Anonymous NS.
  mnt_ns_tree_node.__rb_parent_color = 0xffff888103a12668,    <<< Not
in the `mnt_ns_tree`.
```

It's also the condition which needs to be met in `do_move_mount()` to
not bail out before the warning in `attach_recursive_mnt()`:

```
3289         attached = mnt_has_parent(old);
--
3304         /* ... and either ours or the root of anon namespace */
3305         if (!(attached ? check_mnt(old) : is_anon_ns(ns)))
3306                 goto out;
```

And the mount indeed is in the namespace mounts tree:

```
crash> mnt_namespace.root,mounts.rb_node ffff888103a12600
  root = 0xffff88810b9d0180,
  mounts.rb_node = 0xffff88810b9d0210

crash> mount.mnt_node 0xffff88810b9d0180
    mnt_node = {
      __rb_parent_color = 0x1,
      rb_right = 0x0,
      rb_left = 0x0
    },
```

The `MNT_ONRB` flag should have been set so that the condition on line
1766 is true and we avoid the crash and `move_from_ns()` does not
complain with warning.

> Can you please try and reproduce this with
> commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")

This patch should indirectly address both errors but it does not
explain why the flag is sometimes missing.

--nX

> from the vfs-6.14.mount branch in
> https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git ?
>
> > attach_recursive_mnt+0xc58/0x1260
> > [  192.692051] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
> > nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
> > nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> > nf_tables binfmt_misc btrfs xor raid6_pq zstd_compress iTCO_wdt
> > intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus virtio_net
> > net_failover virtio_balloon lpc_ich failover joydev loop dm_multipath
> > nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vsock zram
> > crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
> > polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3
> > virtio_blk virtio_console bochs serio_raw scsi_dh_rdac scsi_dh_emc
> > scsi_dh_alua fuse qemu_fw_cfg
> > [  192.707547] CPU: 3 UID: 0 PID: 59747 Comm: mount Kdump: loaded Not
> > tainted 6.13.0-rc4-custom+ #9
> > [  192.709485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> > Arch Linux 1.16.3-1-1 04/01/2014
> > [  192.711601] RIP: 0010:attach_recursive_mnt+0xc58/0x1260
> > [  192.712725] Code: 85 c0 0f 85 79 ff ff ff 48 c7 c7 04 e7 00 8e 83 05
> > 9c 18 28 03 01 e8 97 1d c8 01 31 f6 48 89 ef e8 dd c1 fe ff e9 9c f5 ff
> > ff <0f> 0b e9 48 f8 ff ff 48 8b 44 24 10 48 8d 78 20 48 b8 00 00 00 00
> > [  192.716521] RSP: 0018:ffff888105cafb68 EFLAGS: 00010246
> > [  192.717621] RAX: 0000000000001020 RBX: ffff88811cc24030 RCX:
> > ffffffff8ca0e8e5
> > [  192.719078] RDX: ffff888105cafbf0 RSI: ffff888118db0800 RDI:
> > ffff88811cc240f0
> > [  192.720313] RBP: ffff88811cc24000 R08: ffff88810f21a840 R09:
> > ffffed1020b95f62
> > [  192.721028] R10: 0000000000000003 R11: ffff88810a56e558 R12:
> > ffff88811cc24000
> > [  192.721718] R13: dffffc0000000000 R14: ffff88810f21a840 R15:
> > ffff888105cafbf0
> > [  192.722426] FS:  00007fdf69887800(0000) GS:ffff888236f80000(0000)
> > knlGS:0000000000000000
> > [  192.723229] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  192.723849] CR2: 00007fdf697a9000 CR3: 000000010d925000 CR4:
> > 0000000000750ef0
> > [  192.724549] PKRU: 55555554
> > [  192.724860] Call Trace:
> > [  192.725101]  <TASK>
> > [  192.725311]  ? __warn.cold+0xb6/0x176
> > [  192.725672]  ? attach_recursive_mnt+0xc58/0x1260
> > [  192.726149]  ? report_bug+0x1f0/0x2a0
> > [  192.726520]  ? handle_bug+0x54/0x90
> > [  192.726895]  ? exc_invalid_op+0x17/0x40
> > [  192.727259]  ? asm_exc_invalid_op+0x1a/0x20
> > [  192.727664]  ? _raw_spin_lock+0x85/0xe0
> > [  192.728053]  ? attach_recursive_mnt+0xc58/0x1260
> > [  192.728501]  ? attach_recursive_mnt+0xb82/0x1260
> > [  192.728954]  ? _raw_spin_unlock+0xe/0x20
> > [  192.729330]  ? count_mounts+0x1e0/0x1e0
> > [  192.729703]  ? _raw_spin_lock+0x85/0xe0
> > [  192.730082]  ? _raw_write_lock_bh+0xe0/0xe0
> > [  192.730493]  do_move_mount+0x7a8/0x1a20
> > [  192.730871]  __do_sys_move_mount+0x7e2/0xcf0
> > [  192.731288]  ? syscall_exit_to_user_mode+0x10/0x200
> > [  192.731762]  ? do_syscall_64+0x8e/0x160
> > [  192.732180]  ? do_move_mount+0x1a20/0x1a20
> > [  192.732587]  do_syscall_64+0x82/0x160
> > [  192.732950]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > [  192.733500]  ? syscall_exit_to_user_mode+0x10/0x200
> > [  192.733977]  ? do_syscall_64+0x8e/0x160
> > [  192.734374]  ? from_kuid_munged+0x86/0x100
> > [  192.734765]  ? from_kuid+0xc0/0xc0
> > [  192.735115]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > [  192.735644]  ? syscall_exit_to_user_mode+0x10/0x200
> > [  192.736124]  ? do_syscall_64+0x8e/0x160
> > [  192.736487]  ? exc_page_fault+0x76/0xf0
> > [  192.736861]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> > [  192.737352] RIP: 0033:0x7fdf69a5c3de
> > [  192.737725] Code: 73 01 c3 48 8b 0d 32 3a 0f 00 f7 d8 64 89 01 48 83
> > c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 ad 01 00 00 0f
> > 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 02 3a 0f 00 f7 d8 64 89 01 48
> > [  192.739455] RSP: 002b:00007ffd2b7c36d8 EFLAGS: 00000246 ORIG_RAX:
> > 00000000000001ad
> > [  192.740176] RAX: ffffffffffffffda RBX: 0000557ac60e9a00 RCX:
> > 00007fdf69a5c3de
> > [  192.740895] RDX: 00000000ffffff9c RSI: 00007fdf69bc9902 RDI:
> > 0000000000000004
> > [  192.741578] RBP: 00007ffd2b7c3700 R08: 0000000000000004 R09:
> > 0000000000000001
> > [  192.742247] R10: 0000557ac60e9e40 R11: 0000000000000246 R12:
> > 00007fdf69bd6b00
> > [  192.742911] R13: 0000557ac60e9e40 R14: 0000557ac60ebbe0 R15:
> > 0000000000000066
> > [  192.743573]  </TASK>
> > [  192.743803] ---[ end trace 0000000000000000 ]---
> >
> > Thanks,
> > Qu
> >
> > >
> > > [  303.356328] BTRFS: device fsid 6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
> > > devid 1 transid 9 /dev/mapper/test-scratch1 (253:2) scanned by mount
> > > (358060)
> > > [  303.358614] BTRFS info (device dm-2): first mount of filesystem
> > > 6fd8eb6f-1ea5-40aa-9857-05c64efe6d43
> > > [  303.359475] BTRFS info (device dm-2): using crc32c (crc32c-intel)
> > > checksum algorithm
> > > [  303.360134] BTRFS info (device dm-2): using free-space-tree
> > > [  313.264317] list_del corruption, ffff8fd48a7b2c90->prev is NULL
> > > [  313.264966] ------------[ cut here ]------------
> > > [  313.265402] kernel BUG at lib/list_debug.c:54!
> > > [  313.265847] Oops: invalid opcode: 0000 [#1] PREEMPT SMP
> > > [  313.266335] CPU: 4 UID: 0 PID: 370457 Comm: mount Kdump: loaded Not
> > > tainted 6.13.0-rc4-custom+ #8
> > > [  313.267252] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> > > Arch Linux 1.16.3-1-1 04/01/2014
> > > [  313.268147] RIP: 0010:__list_del_entry_valid_or_report.cold+0x6d/0x6f
> > > [  313.268777] Code: 05 77 a0 e8 4b 10 fd ff 0f 0b 48 89 fe 48 c7 c7 90
> > > 05 77 a0 e8 3a 10 fd ff 0f 0b 48 89 fe 48 c7 c7 60 05 77 a0 e8 29 10 fd
> > > ff <0f> 0b 4c 89 ea be 01 00 00 00 4c 89 44 24 48 48 c7 c7 20 7c 2b a1
> > > [  313.270493] RSP: 0018:ffffa7620d2b3a38 EFLAGS: 00010246
> > > [  313.270960] RAX: 0000000000000033 RBX: ffff8fd48a7b2c00 RCX:
> > > 0000000000000000
> > > [  313.271565] RDX: 0000000000000000 RSI: ffff8fd5f7c21900 RDI:
> > > ffff8fd5f7c21900
> > > [  313.272226] RBP: ffff8fd48a7b2c00 R08: 0000000000000000 R09:
> > > 0000000000000000
> > > [  313.272895] R10: 74707572726f6320 R11: 6c65645f7473696c R12:
> > > ffffa7620d2b3a58
> > > [  313.273521] R13: ffff8fd48a7b2c00 R14: 0000000000000000 R15:
> > > ffff8fd48a7b2c90
> > > [  313.274138] FS:  00007f04740d4800(0000) GS:ffff8fd5f7c00000(0000)
> > > knlGS:0000000000000000
> > > [  313.274864] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [  313.275392] CR2: 00007f0473ff6000 CR3: 000000011a8eb000 CR4:
> > > 0000000000750ef0
> > > [  313.276084] PKRU: 55555554
> > > [  313.276327] Call Trace:
> > > [  313.276551]  <TASK>
> > > [  313.276752]  ? __die_body.cold+0x19/0x28
> > > [  313.277102]  ? die+0x2e/0x50
> > > [  313.277699]  ? do_trap+0xc6/0x110
> > > [  313.278033]  ? do_error_trap+0x6a/0x90
> > > [  313.278401]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > > [  313.278941]  ? exc_invalid_op+0x50/0x60
> > > [  313.279308]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > > [  313.279850]  ? asm_exc_invalid_op+0x1a/0x20
> > > [  313.280241]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > > [  313.280777]  ? __list_del_entry_valid_or_report.cold+0x6d/0x6f
> > > [  313.281285]  umount_tree+0xed/0x3c0
> > > [  313.281589]  put_mnt_ns+0x51/0x90
> > > [  313.281886]  mount_subtree+0x92/0x130
> > > [  313.282205]  btrfs_get_tree+0x343/0x6b0 [btrfs]
> > > [  313.282785]  vfs_get_tree+0x23/0xc0
> > > [  313.283089]  vfs_cmd_create+0x59/0xd0
> > > [  313.283406]  __do_sys_fsconfig+0x4eb/0x6b0
> > > [  313.283764]  do_syscall_64+0x82/0x160
> > > [  313.284085]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > > [  313.284598]  ? __fs_parse+0x68/0x1b0
> > > [  313.284929]  ? btrfs_parse_param+0x64/0x870 [btrfs]
> > > [  313.285381]  ? vfs_parse_fs_param_source+0x20/0x90
> > > [  313.285825]  ? __do_sys_fsconfig+0x1b8/0x6b0
> > > [  313.286215]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > > [  313.286719]  ? syscall_exit_to_user_mode+0x10/0x200
> > > [  313.287151]  ? do_syscall_64+0x8e/0x160
> > > [  313.287498]  ? vfs_fstatat+0x75/0xa0
> > > [  313.287835]  ? __do_sys_newfstatat+0x56/0x90
> > > [  313.288240]  ? syscall_exit_to_user_mode_prepare+0x15a/0x190
> > > [  313.288749]  ? syscall_exit_to_user_mode+0x10/0x200
> > > [  313.289188]  ? do_syscall_64+0x8e/0x160
> > > [  313.289544]  ? do_syscall_64+0x8e/0x160
> > > [  313.289892]  ? do_syscall_64+0x8e/0x160
> > > [  313.290253]  ? syscall_exit_to_user_mode+0x10/0x200
> > > [  313.290692]  ? do_syscall_64+0x8e/0x160
> > > [  313.291034]  ? exc_page_fault+0x7e/0x180
> > > [  313.291380]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> > > [  313.291845] RIP: 0033:0x7f04742a919e
> > > [  313.292182] Code: 73 01 c3 48 8b 0d 72 3c 0f 00 f7 d8 64 89 01 48 83
> > > c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f
> > > 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 42 3c 0f 00 f7 d8 64 89 01 48
> > > [  313.293830] RSP: 002b:00007ffc3df08df8 EFLAGS: 00000246 ORIG_RAX:
> > > 00000000000001af
> > > [  313.294529] RAX: ffffffffffffffda RBX: 000056407e37aa00 RCX:
> > > 00007f04742a919e
> > > [  313.295201] RDX: 0000000000000000 RSI: 0000000000000006 RDI:
> > > 0000000000000003
> > > [  313.295864] RBP: 00007ffc3df08f40 R08: 0000000000000000 R09:
> > > 0000000000000001
> > > [  313.296602] R10: 0000000000000000 R11: 0000000000000246 R12:
> > > 00007f0474423b00
> > > [  313.297416] R13: 0000000000000000 R14: 000056407e37cbe0 R15:
> > > 00007f0474418561
> > > [  313.298242]  </TASK>
> > > [  313.298832] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
> > > nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
> > > nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> > > nf_tables binfmt_misc btrfs xor raid6_pq zstd_compress iTCO_wdt
> > > intel_pmc_bxt iTCO_vendor_support i2c_i801 i2c_smbus virtio_net joydev
> > > net_failover lpc_ich virtio_balloon failover loop dm_multipath nfnetlink
> > > vsock_loopback vmw_vsock_virtio_transport_common vsock zram
> > > crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni
> > > polyval_generic ghash_clmulni_intel virtio_console sha512_ssse3
> > > sha256_ssse3 bochs sha1_ssse3 virtio_blk serio_raw scsi_dh_rdac
> > > scsi_dh_emc scsi_dh_alua fuse qemu_fw_cfg
> > > [  313.304504] Dumping ftrace buffer:
> > > [  313.304876]    (ftrace buffer empty)
> > >
> > > [EARLY ANALYZE]
> > >
> > > The offending line is the list_move() call inside unmount_tree().
> > >
> > > With crash core dump, the offending mnt_list is totally corrupted:
> > >
> > > crash> struct list_head ffff8fd48a7b2c90
> > > struct list_head {
> > >    next = 0x1,
> > >    prev = 0x0
> > > }
> > >
> > > umount_tree() should be protected by @mount_lock seqlock, and
> > > @namespace_sem rwsem.
> > >
> > > I also checked other mnt_list users:
> > >
> > > - commit_tree()
> > > - do_umount()
> > > - copy_tree()
> > >
> > > They all hold write @mount_lock at least.
> > >
> > > The only caller doesn't hold @mount_lock is iterate_mounts() but that's
> > > only called from audit, and I'm not sure if audit is even involved in
> > > this case.
>
> This is fine as audit creates a private copy of the mount tree it is
> interested in. The mount tree is not visible to other callers anymore.
>
> > >
> > > So I ran out of ideas why this mnt_list can even happen.
> > >
> > > Even if it's some btrfs' abuse, all mnt_list users are properly
> > > protected thus it should not lead to such list corruption.
> > >
> > > Any advice would be appreciated.
> > >
> > > Thanks,
> > > Qu
> > >
> >
>

Re: mnt_list corruption triggered during btrfs/326

[Christian Brauner]

> > Can you please try and reproduce this with
> > commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")
> 
> This patch should indirectly address both errors but it does not
> explain why the flag is sometimes missing.

Yeah, I'm well aware that's why I didn't fast-track it.
I just didn't have the time to think about this yet.

Re: mnt_list corruption triggered during btrfs/326

[Christian Brauner]

On Tue, Jan 07, 2025 at 04:00:34PM +0100, Christian Brauner wrote:
> > > Can you please try and reproduce this with
> > > commit 211364bef4301838b2e1 ("fs: kill MNT_ONRB")
> > 
> > This patch should indirectly address both errors but it does not
> > explain why the flag is sometimes missing.
> 
> Yeah, I'm well aware that's why I didn't fast-track it.
> I just didn't have the time to think about this yet.

I think I know how it happens.

btrfs_get_tree_subvol()
{
	mnt = fc_mount()
	// Register the newly allocated mount with sb->mounts:
	lock_mount_hash();
	list_add_tail(&mnt->mnt_instance, &mnt->mnt.mnt_sb->s_mounts);
	unlock_mount_hash();
}

So now it's public on sb->s_mounts.

Concurrently someone does a ro remount:

reconfigure_super()
-> sb_prepare_remount_readonly()
   {
           list_for_each_entry(mnt, &sb->s_mounts, mnt_instance) {
   }

This walks all mounts registered in sb->s_mounts and raises
MNT_WRITE_HOLD, then raise MNT_READONLY, and then removes
MNT_WRITE_HOLD.

This can happen concurrently with mount_subvol() because sb->s_umount
isn't held anymore:

-> mount_subvol()
   -> mount_subtree()
      -> alloc_mnt_ns()
         mnt_add_to_ns()
	 vfs_path_lookup()
	 put_mnt_ns()

The flag modification of mnt_add_to_ns() races the flag modification of
the read-only remount. So MNT_ONRB might be lost...

If that's correct, then a) we know how this happens and b) that killing
MNT_ONRB is the correct fix for this.