[PATCH] f2fs: fix to don't set SB_RDONLY in f2fs_handle_critical_error()

[PATCH] f2fs: fix to don't set SB_RDONLY in f2fs_handle_critical_error()

[Chao Yu]

syzbot reports a f2fs bug as below:

------------[ cut here ]------------
WARNING: CPU: 1 PID: 58 at kernel/rcu/sync.c:177 rcu_sync_dtor+0xcd/0x180 kernel/rcu/sync.c:177
CPU: 1 UID: 0 PID: 58 Comm: kworker/1:2 Not tainted 6.10.0-syzkaller-12562-g1722389b0d86 #0
Workqueue: events destroy_super_work
RIP: 0010:rcu_sync_dtor+0xcd/0x180 kernel/rcu/sync.c:177
Call Trace:
 percpu_free_rwsem+0x41/0x80 kernel/locking/percpu-rwsem.c:42
 destroy_super_work+0xec/0x130 fs/super.c:282
 process_one_work kernel/workqueue.c:3231 [inline]
 process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312
 worker_thread+0x86d/0xd40 kernel/workqueue.c:3390
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

As Christian Brauner pointed out [1]: the root cause is f2fs sets
SB_RDONLY flag in internal function, rather than setting the flag
covered w/ sb->s_umount semaphore via remount procedure, then below
race condition causes this bug:

- freeze_super()
 - sb_wait_write(sb, SB_FREEZE_WRITE)
 - sb_wait_write(sb, SB_FREEZE_PAGEFAULT)
 - sb_wait_write(sb, SB_FREEZE_FS)
					- f2fs_handle_critical_error
					 - sb->s_flags |= SB_RDONLY
- thaw_super
 - thaw_super_locked
  - sb_rdonly() is true, so it skips
    sb_freeze_unlock(sb, SB_FREEZE_FS)
  - deactivate_locked_super

Since f2fs has almost the same logic as ext4 [2] when handling critical
error in filesystem if it mounts w/ errors=remount-ro option:
- set CP_ERROR_FLAG flag which indicates filesystem is stopped
- record errors to superblock
- set SB_RDONLY falg
Once we set CP_ERROR_FLAG flag, all writable interfaces can detect the
flag and stop any further updates on filesystem. So, it is safe to not
set SB_RDONLY flag, let's remove the logic and keep in line w/ ext4 [3].

[1] https://lore.kernel.org/all/20240729-himbeeren-funknetz-96e62f9c7aee@brauner
[2] https://lore.kernel.org/all/20240729132721.hxih6ehigadqf7wx@quack3
[3] https://lore.kernel.org/linux-ext4/20240805201241.27286-1-jack@suse.cz

Fixes: b62e71be2110 ("f2fs: support errors=remount-ro|continue|panic mountoption")
Reported-by: syzbot+20d7e439f76bbbd863a7@syzkaller.appspotmail.com
Closes: https://lore.kernel.org/all/000000000000b90a8e061e21d12f@google.com/
Cc: Jan Kara <jack@suse.cz>
Cc: Christian Brauner <brauner@kernel.org>
Signed-off-by: Chao Yu <chao@kernel.org>
---
 fs/f2fs/super.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index e774bdf875b2..acfd000c6bb0 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -4207,12 +4207,14 @@ void f2fs_handle_critical_error(struct f2fs_sb_info *sbi, unsigned char reason,
 	}
 
 	f2fs_warn(sbi, "Remounting filesystem read-only");
+
 	/*
-	 * Make sure updated value of ->s_mount_flags will be visible before
-	 * ->s_flags update
+	 * We have already set CP_ERROR_FLAG flag to stop all updates
+	 * to filesystem, so it doesn't need to set SB_RDONLY flag here
+	 * because the flag should be set covered w/ sb->s_umount semaphore
+	 * via remount procedure, otherwise, it will confuse code like
+	 * freeze_super() which will lead to deadlocks and other problems.
 	 */
-	smp_wmb();
-	sb->s_flags |= SB_RDONLY;
 }
 
 static void f2fs_record_error_work(struct work_struct *work)
-- 
2.40.1

Re: [PATCH] f2fs: fix to don't set SB_RDONLY in f2fs_handle_critical_error()

[Christian Brauner]

On Tue, Sep 10, 2024 at 11:07:13AM GMT, Chao Yu wrote:
> syzbot reports a f2fs bug as below:
> 
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 58 at kernel/rcu/sync.c:177 rcu_sync_dtor+0xcd/0x180 kernel/rcu/sync.c:177
> CPU: 1 UID: 0 PID: 58 Comm: kworker/1:2 Not tainted 6.10.0-syzkaller-12562-g1722389b0d86 #0
> Workqueue: events destroy_super_work
> RIP: 0010:rcu_sync_dtor+0xcd/0x180 kernel/rcu/sync.c:177
> Call Trace:
>  percpu_free_rwsem+0x41/0x80 kernel/locking/percpu-rwsem.c:42
>  destroy_super_work+0xec/0x130 fs/super.c:282
>  process_one_work kernel/workqueue.c:3231 [inline]
>  process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312
>  worker_thread+0x86d/0xd40 kernel/workqueue.c:3390
>  kthread+0x2f0/0x390 kernel/kthread.c:389
>  ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
>  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> 
> As Christian Brauner pointed out [1]: the root cause is f2fs sets
> SB_RDONLY flag in internal function, rather than setting the flag
> covered w/ sb->s_umount semaphore via remount procedure, then below
> race condition causes this bug:
> 
> - freeze_super()
>  - sb_wait_write(sb, SB_FREEZE_WRITE)
>  - sb_wait_write(sb, SB_FREEZE_PAGEFAULT)
>  - sb_wait_write(sb, SB_FREEZE_FS)
> 					- f2fs_handle_critical_error
> 					 - sb->s_flags |= SB_RDONLY
> - thaw_super
>  - thaw_super_locked
>   - sb_rdonly() is true, so it skips
>     sb_freeze_unlock(sb, SB_FREEZE_FS)
>   - deactivate_locked_super
> 
> Since f2fs has almost the same logic as ext4 [2] when handling critical
> error in filesystem if it mounts w/ errors=remount-ro option:
> - set CP_ERROR_FLAG flag which indicates filesystem is stopped
> - record errors to superblock
> - set SB_RDONLY falg
> Once we set CP_ERROR_FLAG flag, all writable interfaces can detect the
> flag and stop any further updates on filesystem. So, it is safe to not
> set SB_RDONLY flag, let's remove the logic and keep in line w/ ext4 [3].
> 
> [1] https://lore.kernel.org/all/20240729-himbeeren-funknetz-96e62f9c7aee@brauner
> [2] https://lore.kernel.org/all/20240729132721.hxih6ehigadqf7wx@quack3
> [3] https://lore.kernel.org/linux-ext4/20240805201241.27286-1-jack@suse.cz
> 
> Fixes: b62e71be2110 ("f2fs: support errors=remount-ro|continue|panic mountoption")
> Reported-by: syzbot+20d7e439f76bbbd863a7@syzkaller.appspotmail.com
> Closes: https://lore.kernel.org/all/000000000000b90a8e061e21d12f@google.com/
> Cc: Jan Kara <jack@suse.cz>
> Cc: Christian Brauner <brauner@kernel.org>
> Signed-off-by: Chao Yu <chao@kernel.org>
> ---

Thanks!
Reviewed-by: Christian Brauner <brauner@kernel.org>

Re: [PATCH] f2fs: fix to don't set SB_RDONLY in f2fs_handle_critical_error()

[Christoph Hellwig]

Another problem is that it sets SB_RDONLY instead of an internal
shutdown flag. But that can be solved later.

Re: [f2fs-dev] [PATCH] f2fs: fix to don't set SB_RDONLY in f2fs_handle_critical_error()

[patchwork-bot+f2fs]

Hello:

This patch was applied to jaegeuk/f2fs.git (dev)
by Jaegeuk Kim <jaegeuk@kernel.org>:

On Tue, 10 Sep 2024 11:07:13 +0800 you wrote:
> syzbot reports a f2fs bug as below:
> 
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 58 at kernel/rcu/sync.c:177 rcu_sync_dtor+0xcd/0x180 kernel/rcu/sync.c:177
> CPU: 1 UID: 0 PID: 58 Comm: kworker/1:2 Not tainted 6.10.0-syzkaller-12562-g1722389b0d86 #0
> Workqueue: events destroy_super_work
> RIP: 0010:rcu_sync_dtor+0xcd/0x180 kernel/rcu/sync.c:177
> Call Trace:
>  percpu_free_rwsem+0x41/0x80 kernel/locking/percpu-rwsem.c:42
>  destroy_super_work+0xec/0x130 fs/super.c:282
>  process_one_work kernel/workqueue.c:3231 [inline]
>  process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312
>  worker_thread+0x86d/0xd40 kernel/workqueue.c:3390
>  kthread+0x2f0/0x390 kernel/kthread.c:389
>  ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
>  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
> 
> [...]

Here is the summary with links:
  - [f2fs-dev] f2fs: fix to don't set SB_RDONLY in f2fs_handle_critical_error()
    https://git.kernel.org/jaegeuk/f2fs/c/930c6ab93492

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html