[syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    eca631b8fe80 Merge tag 'f2fs-6.12-rc4' of git://git.kernel..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=101f4030580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=cfbd94c114a3d407
dashboard link: https://syzkaller.appspot.com/bug?extid=9ef37ac20608f4836256
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1705f85f980000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17d81887980000

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7feb34a89c2a/non_bootable_disk-eca631b8.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/830e1433408d/vmlinux-eca631b8.xz
kernel image: https://storage.googleapis.com/syzbot-assets/5538dfbaa4ef/bzImage-eca631b8.xz
mounted in repro #1: https://storage.googleapis.com/syzbot-assets/049290da6bbe/mount_0.gz
mounted in repro #2: https://storage.googleapis.com/syzbot-assets/c0f1baa786eb/mount_4.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
6.12.0-rc3-syzkaller-00013-geca631b8fe80 #0 Not tainted
------------------------------------------------------
kswapd0/73 is trying to acquire lock:
ffff888011a34610 (sb_internal#2){.+.+}-{0:0}, at: nilfs_dirty_inode+0x158/0x200 fs/nilfs2/inode.c:1110

but task is already holding lock:
ffffffff8ea37160 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat mm/vmscan.c:6844 [inline]
ffffffff8ea37160 (fs_reclaim){+.+.}-{0:0}, at: kswapd+0xbf1/0x3700 mm/vmscan.c:7226

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (fs_reclaim){+.+.}-{0:0}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5825
       __fs_reclaim_acquire mm/page_alloc.c:3834 [inline]
       fs_reclaim_acquire+0x88/0x130 mm/page_alloc.c:3848
       might_alloc include/linux/sched/mm.h:318 [inline]
       prepare_alloc_pages+0x147/0x5b0 mm/page_alloc.c:4493
       __alloc_pages_noprof+0x16f/0x710 mm/page_alloc.c:4722
       alloc_pages_mpol_noprof+0x3e8/0x680 mm/mempolicy.c:2265
       alloc_pages_noprof mm/mempolicy.c:2345 [inline]
       folio_alloc_noprof+0x128/0x180 mm/mempolicy.c:2352
       filemap_alloc_folio_noprof+0xdf/0x500 mm/filemap.c:1010
       __filemap_get_folio+0x446/0xbd0 mm/filemap.c:1952
       block_write_begin+0x66/0x120 fs/buffer.c:2226
       nilfs_write_begin+0xa0/0x110 fs/nilfs2/inode.c:259
       page_symlink+0x2a6/0x4a0 fs/namei.c:5362
       nilfs_symlink+0x236/0x380 fs/nilfs2/namei.c:153
       vfs_symlink+0x137/0x2e0 fs/namei.c:4615
       do_symlinkat+0x222/0x3a0 fs/namei.c:4641
       __do_sys_symlink fs/namei.c:4662 [inline]
       __se_sys_symlink fs/namei.c:4660 [inline]
       __x64_sys_symlink+0x7a/0x90 fs/namei.c:4660
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #1 (&nilfs->ns_segctor_sem){++++}-{3:3}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5825
       down_read+0xb1/0xa40 kernel/locking/rwsem.c:1524
       nilfs_transaction_begin+0x320/0x6e0 fs/nilfs2/segment.c:223
       nilfs_symlink+0x135/0x380 fs/nilfs2/namei.c:140
       vfs_symlink+0x137/0x2e0 fs/namei.c:4615
       do_symlinkat+0x222/0x3a0 fs/namei.c:4641
       __do_sys_symlink fs/namei.c:4662 [inline]
       __se_sys_symlink fs/namei.c:4660 [inline]
       __x64_sys_symlink+0x7a/0x90 fs/namei.c:4660
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

-> #0 (sb_internal#2){.+.+}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3161 [inline]
       check_prevs_add kernel/locking/lockdep.c:3280 [inline]
       validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
       __lock_acquire+0x1384/0x2050 kernel/locking/lockdep.c:5202
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5825
       percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
       __sb_start_write include/linux/fs.h:1716 [inline]
       sb_start_intwrite include/linux/fs.h:1899 [inline]
       nilfs_transaction_begin+0x216/0x6e0 fs/nilfs2/segment.c:220
       nilfs_dirty_inode+0x158/0x200 fs/nilfs2/inode.c:1110
       __mark_inode_dirty+0x2ee/0xe90 fs/fs-writeback.c:2493
       mark_inode_dirty_sync include/linux/fs.h:2479 [inline]
       iput+0x1f1/0xa50 fs/inode.c:1900
       __dentry_kill+0x20d/0x630 fs/dcache.c:615
       shrink_kill+0xa9/0x2c0 fs/dcache.c:1060
       shrink_dentry_list+0x2c0/0x5b0 fs/dcache.c:1087
       prune_dcache_sb+0x10f/0x180 fs/dcache.c:1168
       super_cache_scan+0x34f/0x4b0 fs/super.c:221
       do_shrink_slab+0x701/0x1160 mm/shrinker.c:435
       shrink_slab+0x1093/0x14d0 mm/shrinker.c:662
       shrink_one+0x43b/0x850 mm/vmscan.c:4818
       shrink_many mm/vmscan.c:4879 [inline]
       lru_gen_shrink_node mm/vmscan.c:4957 [inline]
       shrink_node+0x3799/0x3de0 mm/vmscan.c:5937
       kswapd_shrink_node mm/vmscan.c:6765 [inline]
       balance_pgdat mm/vmscan.c:6957 [inline]
       kswapd+0x1ca3/0x3700 mm/vmscan.c:7226
       kthread+0x2f0/0x390 kernel/kthread.c:389
       ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

other info that might help us debug this:

Chain exists of:
  sb_internal#2 --> &nilfs->ns_segctor_sem --> fs_reclaim

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(fs_reclaim);
                               lock(&nilfs->ns_segctor_sem);
                               lock(fs_reclaim);
  rlock(sb_internal#2);

 *** DEADLOCK ***

2 locks held by kswapd0/73:
 #0: ffffffff8ea37160 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat mm/vmscan.c:6844 [inline]
 #0: ffffffff8ea37160 (fs_reclaim){+.+.}-{0:0}, at: kswapd+0xbf1/0x3700 mm/vmscan.c:7226
 #1: ffff888011a340e0 (&type->s_umount_key#44){++++}-{3:3}, at: super_trylock_shared fs/super.c:562 [inline]
 #1: ffff888011a340e0 (&type->s_umount_key#44){++++}-{3:3}, at: super_cache_scan+0x94/0x4b0 fs/super.c:196

stack backtrace:
CPU: 0 UID: 0 PID: 73 Comm: kswapd0 Not tainted 6.12.0-rc3-syzkaller-00013-geca631b8fe80 #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
 print_circular_bug+0x13a/0x1b0 kernel/locking/lockdep.c:2074
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2206
 check_prev_add kernel/locking/lockdep.c:3161 [inline]
 check_prevs_add kernel/locking/lockdep.c:3280 [inline]
 validate_chain+0x18ef/0x5920 kernel/locking/lockdep.c:3904
 __lock_acquire+0x1384/0x2050 kernel/locking/lockdep.c:5202
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5825
 percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
 __sb_start_write include/linux/fs.h:1716 [inline]
 sb_start_intwrite include/linux/fs.h:1899 [inline]
 nilfs_transaction_begin+0x216/0x6e0 fs/nilfs2/segment.c:220
 nilfs_dirty_inode+0x158/0x200 fs/nilfs2/inode.c:1110
 __mark_inode_dirty+0x2ee/0xe90 fs/fs-writeback.c:2493
 mark_inode_dirty_sync include/linux/fs.h:2479 [inline]
 iput+0x1f1/0xa50 fs/inode.c:1900
 __dentry_kill+0x20d/0x630 fs/dcache.c:615
 shrink_kill+0xa9/0x2c0 fs/dcache.c:1060
 shrink_dentry_list+0x2c0/0x5b0 fs/dcache.c:1087
 prune_dcache_sb+0x10f/0x180 fs/dcache.c:1168
 super_cache_scan+0x34f/0x4b0 fs/super.c:221
 do_shrink_slab+0x701/0x1160 mm/shrinker.c:435
 shrink_slab+0x1093/0x14d0 mm/shrinker.c:662
 shrink_one+0x43b/0x850 mm/vmscan.c:4818
 shrink_many mm/vmscan.c:4879 [inline]
 lru_gen_shrink_node mm/vmscan.c:4957 [inline]
 shrink_node+0x3799/0x3de0 mm/vmscan.c:5937
 kswapd_shrink_node mm/vmscan.c:6765 [inline]
 balance_pgdat mm/vmscan.c:6957 [inline]
 kswapd+0x1ca3/0x3700 mm/vmscan.c:7226
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[Edward Adam Davis]

make sure all allocations with GFP_NOFS

#syz test

diff --git a/fs/nilfs2/inode.c b/fs/nilfs2/inode.c
index be6acf6e2bfc..183c16ad8776 100644
--- a/fs/nilfs2/inode.c
+++ b/fs/nilfs2/inode.c
@@ -252,11 +252,14 @@ static int nilfs_write_begin(struct file *file, struct address_space *mapping,
 {
 	struct inode *inode = mapping->host;
 	int err = nilfs_transaction_begin(inode->i_sb, NULL, 1);
+	unsigned flags;
 
 	if (unlikely(err))
 		return err;
 
+	falgs = memalloc_nofs_save();
 	err = block_write_begin(mapping, pos, len, foliop, nilfs_get_block);
+	memalloc_nofs_restore(flags);
 	if (unlikely(err)) {
 		nilfs_write_failed(mapping, pos + len);
 		nilfs_transaction_abort(inode->i_sb);

Re: [syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[syzbot]

Hello,

syzbot tried to test the proposed patch but the build/boot failed:

fs/nilfs2/inode.c:260:2: error: use of undeclared identifier 'falgs'


Tested on:

commit:         715ca9dd Merge tag 'io_uring-6.12-20241019' of git://g..
git tree:       upstream
kernel config:  https://syzkaller.appspot.com/x/.config?x=cfbd94c114a3d407
dashboard link: https://syzkaller.appspot.com/bug?extid=9ef37ac20608f4836256
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=17260c87980000

Re: [syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[Ryusuke Konishi]

drop __GFP_FS for symlinks

#syz test

diff --git a/fs/nilfs2/namei.c b/fs/nilfs2/namei.c
index c950139db6ef..bcd7d1288e2c 100644
--- a/fs/nilfs2/namei.c
+++ b/fs/nilfs2/namei.c
@@ -149,6 +149,9 @@ static int nilfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
 	/* slow symlink */
 	inode->i_op = &nilfs_symlink_inode_operations;
 	inode_nohighmem(inode);
+	mapping_set_gfp_mask(inode->i_mapping,
+			     mapping_gfp_constraint(inode->i_mapping,
+						    ~__GFP_FS));
 	inode->i_mapping->a_ops = &nilfs_aops;
 	err = page_symlink(inode, symname, l);
 	if (err)

Re: [syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[Edward Adam Davis]

make sure all allocations with GFP_NOFS

#syz test

diff --git a/fs/nilfs2/inode.c b/fs/nilfs2/inode.c
index be6acf6e2bfc..183c16ad8776 100644
--- a/fs/nilfs2/inode.c
+++ b/fs/nilfs2/inode.c
@@ -252,11 +252,14 @@ static int nilfs_write_begin(struct file *file, struct address_space *mapping,
 {
 	struct inode *inode = mapping->host;
 	int err = nilfs_transaction_begin(inode->i_sb, NULL, 1);
+	unsigned flags;
 
 	if (unlikely(err))
 		return err;
 
+	flags = memalloc_nofs_save();
 	err = block_write_begin(mapping, pos, len, foliop, nilfs_get_block);
+	memalloc_nofs_restore(flags);
 	if (unlikely(err)) {
 		nilfs_write_failed(mapping, pos + len);
 		nilfs_transaction_abort(inode->i_sb);

Re: [syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com
Tested-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com

Tested on:

commit:         715ca9dd Merge tag 'io_uring-6.12-20241019' of git://g..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1450825f980000
kernel config:  https://syzkaller.appspot.com/x/.config?x=16e543edc81a3008
dashboard link: https://syzkaller.appspot.com/bug?extid=9ef37ac20608f4836256
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=148e0c87980000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] [nilfs?] possible deadlock in nilfs_dirty_inode (4)

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com
Tested-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com

Tested on:

commit:         715ca9dd Merge tag 'io_uring-6.12-20241019' of git://g..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1759b240580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=16e543edc81a3008
dashboard link: https://syzkaller.appspot.com/bug?extid=9ef37ac20608f4836256
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=12e1b240580000

Note: testing is done by a robot and is best-effort only.

[PATCH] nilfs2: fix potential deadlock with newly created symlinks

[Ryusuke Konishi]

Syzbot reported that page_symlink(), called by nilfs_symlink(),
triggers memory reclamation involving the filesystem layer, which can
result in circular lock dependencies among the reader/writer semaphore
nilfs->ns_segctor_sem, s_writers percpu_rwsem (intwrite) and the
fs_reclaim pseudo lock.

This is because after commit 21fc61c73c39 ("don't put symlink bodies in
pagecache into highmem"), the gfp flags of the page cache for symbolic
links are overwritten to GFP_KERNEL via inode_nohighmem().

This is not a problem for symlinks read from the backing device,
because the __GFP_FS flag is dropped after inode_nohighmem() is called.
However, when a new symlink is created with nilfs_symlink(), the gfp
flags remain overwritten to GFP_KERNEL.  Then, memory allocation called
from page_symlink() etc. triggers memory reclamation including the FS
layer, which may call nilfs_evict_inode() or nilfs_dirty_inode().  And
these can cause a deadlock if they are called while
nilfs->ns_segctor_sem is held:

Fix this issue by dropping the __GFP_FS flag from the page cache GFP
flags of newly created symlinks in the same way that nilfs_new_inode()
and __nilfs_read_inode() do, as a workaround until we adopt nofs
allocation scope consistently or improve the locking constraints.

Signed-off-by: Ryusuke Konishi <konishi.ryusuke@gmail.com>
Reported-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=9ef37ac20608f4836256
Fixes: 21fc61c73c39 ("don't put symlink bodies in pagecache into highmem")
Tested-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com
Cc: stable@vger.kernel.org
---
Andrew, please apply this patch as a bug fix.

This fixes a potential deadlock issue recently reported by syzbot.

Thanks,
Ryusuke Konishi

 fs/nilfs2/namei.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/nilfs2/namei.c b/fs/nilfs2/namei.c
index 4905063790c5..9b108052d9f7 100644
--- a/fs/nilfs2/namei.c
+++ b/fs/nilfs2/namei.c
@@ -157,6 +157,9 @@ static int nilfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
 	/* slow symlink */
 	inode->i_op = &nilfs_symlink_inode_operations;
 	inode_nohighmem(inode);
+	mapping_set_gfp_mask(inode->i_mapping,
+			     mapping_gfp_constraint(inode->i_mapping,
+						    ~__GFP_FS));
 	inode->i_mapping->a_ops = &nilfs_aops;
 	err = page_symlink(inode, symname, l);
 	if (err)
-- 
2.43.0

[PATCH] nilfs2: fix potential deadlock with newly created symlinks

[Ryusuke Konishi]

Syzbot reported that page_symlink(), called by nilfs_symlink(),
triggers memory reclamation involving the filesystem layer, which can
result in circular lock dependencies among the reader/writer semaphore
nilfs->ns_segctor_sem, s_writers percpu_rwsem (intwrite) and the
fs_reclaim pseudo lock.

This is because after commit 21fc61c73c39 ("don't put symlink bodies in
pagecache into highmem"), the gfp flags of the page cache for symbolic
links are overwritten to GFP_KERNEL via inode_nohighmem().

This is not a problem for symlinks read from the backing device,
because the __GFP_FS flag is dropped after inode_nohighmem() is called.
However, when a new symlink is created with nilfs_symlink(), the gfp
flags remain overwritten to GFP_KERNEL.  Then, memory allocation called
from page_symlink() etc. triggers memory reclamation including the FS
layer, which may call nilfs_evict_inode() or nilfs_dirty_inode().  And
these can cause a deadlock if they are called while
nilfs->ns_segctor_sem is held:

Fix this issue by dropping the __GFP_FS flag from the page cache GFP
flags of newly created symlinks in the same way that nilfs_new_inode()
and __nilfs_read_inode() do, as a workaround until we adopt nofs
allocation scope consistently or improve the locking constraints.

Signed-off-by: Ryusuke Konishi <konishi.ryusuke@gmail.com>
Reported-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=9ef37ac20608f4836256
Fixes: 21fc61c73c39 ("don't put symlink bodies in pagecache into highmem")
Tested-by: syzbot+9ef37ac20608f4836256@syzkaller.appspotmail.com
Cc: stable@vger.kernel.org
---
Part of the syzbot address was missing in the recipient's addresses,
so I'll resend this to syzbot.

Please be careful when replying to the previous patch submission.

Ryusuke Konishi

 fs/nilfs2/namei.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/nilfs2/namei.c b/fs/nilfs2/namei.c
index 4905063790c5..9b108052d9f7 100644
--- a/fs/nilfs2/namei.c
+++ b/fs/nilfs2/namei.c
@@ -157,6 +157,9 @@ static int nilfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
 	/* slow symlink */
 	inode->i_op = &nilfs_symlink_inode_operations;
 	inode_nohighmem(inode);
+	mapping_set_gfp_mask(inode->i_mapping,
+			     mapping_gfp_constraint(inode->i_mapping,
+						    ~__GFP_FS));
 	inode->i_mapping->a_ops = &nilfs_aops;
 	err = page_symlink(inode, symname, l);
 	if (err)
-- 
2.43.0