[PATCH] f2fs: fix to avoid potential deadlock

[PATCH] f2fs: fix to avoid potential deadlock

[Chao Yu]

There is a potential deadlock reported by syzbot as below:

F2FS-fs (loop2): invalid crc value
F2FS-fs (loop2): Found nat_bits in checkpoint
F2FS-fs (loop2): Mounted with checkpoint version = 48b305e4
======================================================
WARNING: possible circular locking dependency detected
6.1.0-rc8-syzkaller-33330-ga5541c0811a0 #0 Not tainted
------------------------------------------------------
syz-executor.2/32123 is trying to acquire lock:
ffff0000c0e1a608 (&mm->mmap_lock){++++}-{3:3}, at: __might_fault+0x54/0xb4 mm/memory.c:5644

but task is already holding lock:
ffff0001317c6088 (&sbi->sb_lock){++++}-{3:3}, at: f2fs_down_write fs/f2fs/f2fs.h:2205 [inline]
ffff0001317c6088 (&sbi->sb_lock){++++}-{3:3}, at: f2fs_ioc_get_encryption_pwsalt fs/f2fs/file.c:2334 [inline]
ffff0001317c6088 (&sbi->sb_lock){++++}-{3:3}, at: __f2fs_ioctl+0x1370/0x3318 fs/f2fs/file.c:4151

which lock already depends on the new lock.

Chain exists of:
  &mm->mmap_lock --> &nm_i->nat_tree_lock --> &sbi->sb_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&sbi->sb_lock);
                               lock(&nm_i->nat_tree_lock);
                               lock(&sbi->sb_lock);
  lock(&mm->mmap_lock);

Let's try to avoid above deadlock condition by moving __might_fault()
out of sbi->sb_lock coverage.

Fixes: 95fa90c9e5a7 ("f2fs: support recording errors into superblock")
Link: https://lore.kernel.org/linux-f2fs-devel/000000000000cd5fe305ef617fe2@google.com/T/#u
Reported-by: syzbot+4793f6096d174c90b4f7@syzkaller.appspotmail.com
Signed-off-by: Chao Yu <chao@kernel.org>
---
 fs/f2fs/file.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index cad4bdd6f097..4bc98dbe8292 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -2336,6 +2336,7 @@ static int f2fs_ioc_get_encryption_pwsalt(struct file *filp, unsigned long arg)
 {
 	struct inode *inode = file_inode(filp);
 	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+	u8 encrypt_pw_salt[16];
 	int err;
 
 	if (!f2fs_sb_has_encrypt(sbi))
@@ -2360,12 +2361,14 @@ static int f2fs_ioc_get_encryption_pwsalt(struct file *filp, unsigned long arg)
 		goto out_err;
 	}
 got_it:
-	if (copy_to_user((__u8 __user *)arg, sbi->raw_super->encrypt_pw_salt,
-									16))
-		err = -EFAULT;
+	memcpy(encrypt_pw_salt, sbi->raw_super->encrypt_pw_salt, 16);
 out_err:
 	f2fs_up_write(&sbi->sb_lock);
 	mnt_drop_write_file(filp);
+
+	if (!err && copy_to_user((__u8 __user *)arg, encrypt_pw_salt, 16))
+		err = -EFAULT;
+
 	return err;
 }
 
-- 
2.36.1

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Eric Biggers]

On Fri, Dec 16, 2022 at 11:50:00PM +0800, Chao Yu wrote:
> diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
> index cad4bdd6f097..4bc98dbe8292 100644
> --- a/fs/f2fs/file.c
> +++ b/fs/f2fs/file.c
> @@ -2336,6 +2336,7 @@ static int f2fs_ioc_get_encryption_pwsalt(struct file *filp, unsigned long arg)
>  {
>  	struct inode *inode = file_inode(filp);
>  	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
> +	u8 encrypt_pw_salt[16];
>  	int err;
>  
>  	if (!f2fs_sb_has_encrypt(sbi))
> @@ -2360,12 +2361,14 @@ static int f2fs_ioc_get_encryption_pwsalt(struct file *filp, unsigned long arg)
>  		goto out_err;
>  	}
>  got_it:
> -	if (copy_to_user((__u8 __user *)arg, sbi->raw_super->encrypt_pw_salt,
> -									16))
> -		err = -EFAULT;
> +	memcpy(encrypt_pw_salt, sbi->raw_super->encrypt_pw_salt, 16);
>  out_err:
>  	f2fs_up_write(&sbi->sb_lock);
>  	mnt_drop_write_file(filp);
> +
> +	if (!err && copy_to_user((__u8 __user *)arg, encrypt_pw_salt, 16))
> +		err = -EFAULT;
> +
>  	return err;

Reviewed-by: Eric Biggers <ebiggers@google.com>

- Eric

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[patchwork-bot+f2fs]

Hello:

This patch was applied to jaegeuk/f2fs.git (dev)
by Jaegeuk Kim <jaegeuk@kernel.org>:

On Tue, 14 Oct 2025 19:47:35 +0800 you wrote:
> As Jiaming Zhang and syzbot reported, there is potential deadlock in
> f2fs as below:
> 
> Chain exists of:
>   &sbi->cp_rwsem --> fs_reclaim --> sb_internal#2
> 
>  Possible unsafe locking scenario:
> 
> [...]

Here is the summary with links:
  - [f2fs-dev] f2fs: fix to avoid potential deadlock
    https://git.kernel.org/jaegeuk/f2fs/c/7976ef973cab

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Jaegeuk Kim]

On 03/03, Chao Yu wrote:
> On 2022/3/3 3:45, Jaegeuk Kim wrote:
> > On 03/02, Chao Yu wrote:
> > > On 2022/3/2 13:26, Jaegeuk Kim wrote:
> > > > On 03/02, Chao Yu wrote:
> > > > > ping,
> > > > > 
> > > > > On 2022/2/25 11:02, Chao Yu wrote:
> > > > > > On 2022/2/3 22:57, Chao Yu wrote:
> > > > > > > On 2022/2/3 9:51, Jaegeuk Kim wrote:
> > > > > > > > On 01/29, Chao Yu wrote:
> > > > > > > > > On 2022/1/29 8:37, Jaegeuk Kim wrote:
> > > > > > > > > > On 01/28, Chao Yu wrote:
> > > > > > > > > > > On 2022/1/28 5:59, Jaegeuk Kim wrote:
> > > > > > > > > > > > On 01/27, Chao Yu wrote:
> > > > > > > > > > > > > Quoted from Jing Xia's report, there is a potential deadlock may happen
> > > > > > > > > > > > > between kworker and checkpoint as below:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > [T:writeback]                [T:checkpoint]
> > > > > > > > > > > > > - wb_writeback
> > > > > > > > > > > > >       - blk_start_plug
> > > > > > > > > > > > > bio contains NodeA was plugged in writeback threads
> > > > > > > > > > > > 
> > > > > > > > > > > > I'm still trying to understand more precisely. So, how is it possible to
> > > > > > > > > > > > have bio having node write in this current context?
> > > > > > > > > > > 
> > > > > > > > > > > IMO, after above blk_start_plug(), it may plug some inode's node page in kworker
> > > > > > > > > > > during writebacking node_inode's data page (which should be node page)?
> > > > > > > > > > 
> > > > > > > > > > Wasn't that added into a different task->plug?
> > > > > > > > > 
> > > > > > > > > I'm not sure I've got your concern correctly...
> > > > > > > > > 
> > > > > > > > > Do you mean NodeA and other IOs from do_writepages() were plugged in
> > > > > > > > > different local plug variables?
> > > > > > > > 
> > > > > > > > I think so.
> > > > > > > 
> > > > > > > I guess block plug helper says it doesn't allow to use nested plug, so there
> > > > > > > is only one plug in kworker thread?
> > > > 
> > > > Is there only one kworker thread that flushes node and inode pages?
> > > 
> > > IIRC, =one kworker per block device?
> > 
> > If there's one kworker only, f2fs_write_node_pages() should have flushed its
> > plug?
> 
> No, f2fs_write_node_pages() failed to attach local plug into current->plug due to
> current has attached plug from wb_writeback(), and also, f2fs_write_node_pages()
> will fail to flush current->plug due to its local plug doesn't match current->plug.
> 
> void blk_start_plug_nr_ios()
> {
> 	if (tsk->plug)
> 		return;
> ...
> }
> 
> void blk_finish_plug(struct blk_plug *plug)
> {
> 	if (plug == current->plug) {
> 		__blk_flush_plug(plug, false);
> 		current->plug = NULL;
> 	}
> }

Ah, okay. Now I see. Thanks for the chasing down.

> 
> Thanks,
> 
> > 
> > > 
> > > Thanks,
> > > 
> > > > 
> > > > > > > 
> > > > > > > void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
> > > > > > > {
> > > > > > >        struct task_struct *tsk = current;
> > > > > > > 
> > > > > > >        /*
> > > > > > >         * If this is a nested plug, don't actually assign it.
> > > > > > >         */
> > > > > > >        if (tsk->plug)
> > > > > > >            return;
> > > > > > > ...
> > > > > > > }
> > > > > > 
> > > > > > Any further comments?
> > > > > > 
> > > > > > Thanks,
> > > > > > 
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > 
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Thanks,
> > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > Thanks,
> > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > >                       - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
> > > > > > > > > > > > >                        - f2fs_write_data_pages
> > > > > > > > > > > > >                         - f2fs_write_single_data_page -- write last dirty page
> > > > > > > > > > > > >                          - f2fs_do_write_data_page
> > > > > > > > > > > > >                           - set_page_writeback  -- clear page dirty flag and
> > > > > > > > > > > > >                           PAGECACHE_TAG_DIRTY tag in radix tree
> > > > > > > > > > > > >                           - f2fs_outplace_write_data
> > > > > > > > > > > > >                            - f2fs_update_data_blkaddr
> > > > > > > > > > > > >                             - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
> > > > > > > > > > > > >                          - inode_dec_dirty_pages
> > > > > > > > > > > > >       - writeback_sb_inodes
> > > > > > > > > > > > >        - writeback_single_inode
> > > > > > > > > > > > >         - do_writepages
> > > > > > > > > > > > >          - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
> > > > > > > > > > > > >           - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
> > > > > > > > > > > > >        - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
> > > > > > > > > > > > >       - blk_finish_plug
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Let's try to avoid deadlock condition by forcing unplugging previous bio via
> > > > > > > > > > > > > blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
> > > > > > > > > > > > > due to valid sbi->wb_sync_req[DATA/NODE].
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
> > > > > > > > > > > > > Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
> > > > > > > > > > > > > Signed-off-by: Jing Xia <jing.xia@unisoc.com>
> > > > > > > > > > > > > Signed-off-by: Chao Yu <chao@kernel.org>
> > > > > > > > > > > > > ---
> > > > > > > > > > > > >       fs/f2fs/data.c | 6 +++++-
> > > > > > > > > > > > >       fs/f2fs/node.c | 6 +++++-
> > > > > > > > > > > > >       2 files changed, 10 insertions(+), 2 deletions(-)
> > > > > > > > > > > > > 
> > > > > > > > > > > > > diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
> > > > > > > > > > > > > index 76d6fe7b0c8f..932a4c81acaf 100644
> > > > > > > > > > > > > --- a/fs/f2fs/data.c
> > > > > > > > > > > > > +++ b/fs/f2fs/data.c
> > > > > > > > > > > > > @@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
> > > > > > > > > > > > >           /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
> > > > > > > > > > > > >           if (wbc->sync_mode == WB_SYNC_ALL)
> > > > > > > > > > > > >               atomic_inc(&sbi->wb_sync_req[DATA]);
> > > > > > > > > > > > > -    else if (atomic_read(&sbi->wb_sync_req[DATA]))
> > > > > > > > > > > > > +    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
> > > > > > > > > > > > > +        /* to avoid potential deadlock */
> > > > > > > > > > > > > +        if (current->plug)
> > > > > > > > > > > > > +            blk_finish_plug(current->plug);
> > > > > > > > > > > > >               goto skip_write;
> > > > > > > > > > > > > +    }
> > > > > > > > > > > > >           if (__should_serialize_io(inode, wbc)) {
> > > > > > > > > > > > >               mutex_lock(&sbi->writepages);
> > > > > > > > > > > > > diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
> > > > > > > > > > > > > index 556fcd8457f3..69c6bcaf5aae 100644
> > > > > > > > > > > > > --- a/fs/f2fs/node.c
> > > > > > > > > > > > > +++ b/fs/f2fs/node.c
> > > > > > > > > > > > > @@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
> > > > > > > > > > > > >           if (wbc->sync_mode == WB_SYNC_ALL)
> > > > > > > > > > > > >               atomic_inc(&sbi->wb_sync_req[NODE]);
> > > > > > > > > > > > > -    else if (atomic_read(&sbi->wb_sync_req[NODE]))
> > > > > > > > > > > > > +    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
> > > > > > > > > > > > > +        /* to avoid potential deadlock */
> > > > > > > > > > > > > +        if (current->plug)
> > > > > > > > > > > > > +            blk_finish_plug(current->plug);
> > > > > > > > > > > > >               goto skip_write;
> > > > > > > > > > > > > +    }
> > > > > > > > > > > > >           trace_f2fs_writepages(mapping->host, wbc, NODE);
> > > > > > > > > > > > > -- 
> > > > > > > > > > > > > 2.32.0
> > > > > > > 
> > > > > > > 
> > > > > > > _______________________________________________
> > > > > > > Linux-f2fs-devel mailing list
> > > > > > > Linux-f2fs-devel@lists.sourceforge.net
> > > > > > > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> > > > > > 
> > > > > > 
> > > > > > _______________________________________________
> > > > > > Linux-f2fs-devel mailing list
> > > > > > Linux-f2fs-devel@lists.sourceforge.net
> > > > > > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Chao Yu]

On 2022/3/3 3:45, Jaegeuk Kim wrote:
> On 03/02, Chao Yu wrote:
>> On 2022/3/2 13:26, Jaegeuk Kim wrote:
>>> On 03/02, Chao Yu wrote:
>>>> ping,
>>>>
>>>> On 2022/2/25 11:02, Chao Yu wrote:
>>>>> On 2022/2/3 22:57, Chao Yu wrote:
>>>>>> On 2022/2/3 9:51, Jaegeuk Kim wrote:
>>>>>>> On 01/29, Chao Yu wrote:
>>>>>>>> On 2022/1/29 8:37, Jaegeuk Kim wrote:
>>>>>>>>> On 01/28, Chao Yu wrote:
>>>>>>>>>> On 2022/1/28 5:59, Jaegeuk Kim wrote:
>>>>>>>>>>> On 01/27, Chao Yu wrote:
>>>>>>>>>>>> Quoted from Jing Xia's report, there is a potential deadlock may happen
>>>>>>>>>>>> between kworker and checkpoint as below:
>>>>>>>>>>>>
>>>>>>>>>>>> [T:writeback]                [T:checkpoint]
>>>>>>>>>>>> - wb_writeback
>>>>>>>>>>>>       - blk_start_plug
>>>>>>>>>>>> bio contains NodeA was plugged in writeback threads
>>>>>>>>>>>
>>>>>>>>>>> I'm still trying to understand more precisely. So, how is it possible to
>>>>>>>>>>> have bio having node write in this current context?
>>>>>>>>>>
>>>>>>>>>> IMO, after above blk_start_plug(), it may plug some inode's node page in kworker
>>>>>>>>>> during writebacking node_inode's data page (which should be node page)?
>>>>>>>>>
>>>>>>>>> Wasn't that added into a different task->plug?
>>>>>>>>
>>>>>>>> I'm not sure I've got your concern correctly...
>>>>>>>>
>>>>>>>> Do you mean NodeA and other IOs from do_writepages() were plugged in
>>>>>>>> different local plug variables?
>>>>>>>
>>>>>>> I think so.
>>>>>>
>>>>>> I guess block plug helper says it doesn't allow to use nested plug, so there
>>>>>> is only one plug in kworker thread?
>>>
>>> Is there only one kworker thread that flushes node and inode pages?
>>
>> IIRC, =one kworker per block device?
> 
> If there's one kworker only, f2fs_write_node_pages() should have flushed its
> plug?

No, f2fs_write_node_pages() failed to attach local plug into current->plug due to
current has attached plug from wb_writeback(), and also, f2fs_write_node_pages()
will fail to flush current->plug due to its local plug doesn't match current->plug.

void blk_start_plug_nr_ios()
{
	if (tsk->plug)
		return;
...
}

void blk_finish_plug(struct blk_plug *plug)
{
	if (plug == current->plug) {
		__blk_flush_plug(plug, false);
		current->plug = NULL;
	}
}

Thanks,

> 
>>
>> Thanks,
>>
>>>
>>>>>>
>>>>>> void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
>>>>>> {
>>>>>>        struct task_struct *tsk = current;
>>>>>>
>>>>>>        /*
>>>>>>         * If this is a nested plug, don't actually assign it.
>>>>>>         */
>>>>>>        if (tsk->plug)
>>>>>>            return;
>>>>>> ...
>>>>>> }
>>>>>
>>>>> Any further comments?
>>>>>
>>>>> Thanks,
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>                       - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
>>>>>>>>>>>>                        - f2fs_write_data_pages
>>>>>>>>>>>>                         - f2fs_write_single_data_page -- write last dirty page
>>>>>>>>>>>>                          - f2fs_do_write_data_page
>>>>>>>>>>>>                           - set_page_writeback  -- clear page dirty flag and
>>>>>>>>>>>>                           PAGECACHE_TAG_DIRTY tag in radix tree
>>>>>>>>>>>>                           - f2fs_outplace_write_data
>>>>>>>>>>>>                            - f2fs_update_data_blkaddr
>>>>>>>>>>>>                             - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
>>>>>>>>>>>>                          - inode_dec_dirty_pages
>>>>>>>>>>>>       - writeback_sb_inodes
>>>>>>>>>>>>        - writeback_single_inode
>>>>>>>>>>>>         - do_writepages
>>>>>>>>>>>>          - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
>>>>>>>>>>>>           - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
>>>>>>>>>>>>        - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
>>>>>>>>>>>>       - blk_finish_plug
>>>>>>>>>>>>
>>>>>>>>>>>> Let's try to avoid deadlock condition by forcing unplugging previous bio via
>>>>>>>>>>>> blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
>>>>>>>>>>>> due to valid sbi->wb_sync_req[DATA/NODE].
>>>>>>>>>>>>
>>>>>>>>>>>> Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
>>>>>>>>>>>> Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
>>>>>>>>>>>> Signed-off-by: Jing Xia <jing.xia@unisoc.com>
>>>>>>>>>>>> Signed-off-by: Chao Yu <chao@kernel.org>
>>>>>>>>>>>> ---
>>>>>>>>>>>>       fs/f2fs/data.c | 6 +++++-
>>>>>>>>>>>>       fs/f2fs/node.c | 6 +++++-
>>>>>>>>>>>>       2 files changed, 10 insertions(+), 2 deletions(-)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
>>>>>>>>>>>> index 76d6fe7b0c8f..932a4c81acaf 100644
>>>>>>>>>>>> --- a/fs/f2fs/data.c
>>>>>>>>>>>> +++ b/fs/f2fs/data.c
>>>>>>>>>>>> @@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
>>>>>>>>>>>>           /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
>>>>>>>>>>>>           if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>>>>>>               atomic_inc(&sbi->wb_sync_req[DATA]);
>>>>>>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[DATA]))
>>>>>>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
>>>>>>>>>>>> +        /* to avoid potential deadlock */
>>>>>>>>>>>> +        if (current->plug)
>>>>>>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>>>>>>               goto skip_write;
>>>>>>>>>>>> +    }
>>>>>>>>>>>>           if (__should_serialize_io(inode, wbc)) {
>>>>>>>>>>>>               mutex_lock(&sbi->writepages);
>>>>>>>>>>>> diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
>>>>>>>>>>>> index 556fcd8457f3..69c6bcaf5aae 100644
>>>>>>>>>>>> --- a/fs/f2fs/node.c
>>>>>>>>>>>> +++ b/fs/f2fs/node.c
>>>>>>>>>>>> @@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
>>>>>>>>>>>>           if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>>>>>>               atomic_inc(&sbi->wb_sync_req[NODE]);
>>>>>>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[NODE]))
>>>>>>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
>>>>>>>>>>>> +        /* to avoid potential deadlock */
>>>>>>>>>>>> +        if (current->plug)
>>>>>>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>>>>>>               goto skip_write;
>>>>>>>>>>>> +    }
>>>>>>>>>>>>           trace_f2fs_writepages(mapping->host, wbc, NODE);
>>>>>>>>>>>> -- 
>>>>>>>>>>>> 2.32.0
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Linux-f2fs-devel mailing list
>>>>>> Linux-f2fs-devel@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Linux-f2fs-devel mailing list
>>>>> Linux-f2fs-devel@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Jaegeuk Kim]

On 03/02, Chao Yu wrote:
> On 2022/3/2 13:26, Jaegeuk Kim wrote:
> > On 03/02, Chao Yu wrote:
> > > ping,
> > > 
> > > On 2022/2/25 11:02, Chao Yu wrote:
> > > > On 2022/2/3 22:57, Chao Yu wrote:
> > > > > On 2022/2/3 9:51, Jaegeuk Kim wrote:
> > > > > > On 01/29, Chao Yu wrote:
> > > > > > > On 2022/1/29 8:37, Jaegeuk Kim wrote:
> > > > > > > > On 01/28, Chao Yu wrote:
> > > > > > > > > On 2022/1/28 5:59, Jaegeuk Kim wrote:
> > > > > > > > > > On 01/27, Chao Yu wrote:
> > > > > > > > > > > Quoted from Jing Xia's report, there is a potential deadlock may happen
> > > > > > > > > > > between kworker and checkpoint as below:
> > > > > > > > > > > 
> > > > > > > > > > > [T:writeback]                [T:checkpoint]
> > > > > > > > > > > - wb_writeback
> > > > > > > > > > >      - blk_start_plug
> > > > > > > > > > > bio contains NodeA was plugged in writeback threads
> > > > > > > > > > 
> > > > > > > > > > I'm still trying to understand more precisely. So, how is it possible to
> > > > > > > > > > have bio having node write in this current context?
> > > > > > > > > 
> > > > > > > > > IMO, after above blk_start_plug(), it may plug some inode's node page in kworker
> > > > > > > > > during writebacking node_inode's data page (which should be node page)?
> > > > > > > > 
> > > > > > > > Wasn't that added into a different task->plug?
> > > > > > > 
> > > > > > > I'm not sure I've got your concern correctly...
> > > > > > > 
> > > > > > > Do you mean NodeA and other IOs from do_writepages() were plugged in
> > > > > > > different local plug variables?
> > > > > > 
> > > > > > I think so.
> > > > > 
> > > > > I guess block plug helper says it doesn't allow to use nested plug, so there
> > > > > is only one plug in kworker thread?
> > 
> > Is there only one kworker thread that flushes node and inode pages?
> 
> IIRC, =one kworker per block device?

If there's one kworker only, f2fs_write_node_pages() should have flushed its
plug?

> 
> Thanks,
> 
> > 
> > > > > 
> > > > > void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
> > > > > {
> > > > >       struct task_struct *tsk = current;
> > > > > 
> > > > >       /*
> > > > >        * If this is a nested plug, don't actually assign it.
> > > > >        */
> > > > >       if (tsk->plug)
> > > > >           return;
> > > > > ...
> > > > > }
> > > > 
> > > > Any further comments?
> > > > 
> > > > Thanks,
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > 
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Thanks,
> > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > >                      - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
> > > > > > > > > > >                       - f2fs_write_data_pages
> > > > > > > > > > >                        - f2fs_write_single_data_page -- write last dirty page
> > > > > > > > > > >                         - f2fs_do_write_data_page
> > > > > > > > > > >                          - set_page_writeback  -- clear page dirty flag and
> > > > > > > > > > >                          PAGECACHE_TAG_DIRTY tag in radix tree
> > > > > > > > > > >                          - f2fs_outplace_write_data
> > > > > > > > > > >                           - f2fs_update_data_blkaddr
> > > > > > > > > > >                            - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
> > > > > > > > > > >                         - inode_dec_dirty_pages
> > > > > > > > > > >      - writeback_sb_inodes
> > > > > > > > > > >       - writeback_single_inode
> > > > > > > > > > >        - do_writepages
> > > > > > > > > > >         - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
> > > > > > > > > > >          - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
> > > > > > > > > > >       - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
> > > > > > > > > > >      - blk_finish_plug
> > > > > > > > > > > 
> > > > > > > > > > > Let's try to avoid deadlock condition by forcing unplugging previous bio via
> > > > > > > > > > > blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
> > > > > > > > > > > due to valid sbi->wb_sync_req[DATA/NODE].
> > > > > > > > > > > 
> > > > > > > > > > > Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
> > > > > > > > > > > Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
> > > > > > > > > > > Signed-off-by: Jing Xia <jing.xia@unisoc.com>
> > > > > > > > > > > Signed-off-by: Chao Yu <chao@kernel.org>
> > > > > > > > > > > ---
> > > > > > > > > > >      fs/f2fs/data.c | 6 +++++-
> > > > > > > > > > >      fs/f2fs/node.c | 6 +++++-
> > > > > > > > > > >      2 files changed, 10 insertions(+), 2 deletions(-)
> > > > > > > > > > > 
> > > > > > > > > > > diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
> > > > > > > > > > > index 76d6fe7b0c8f..932a4c81acaf 100644
> > > > > > > > > > > --- a/fs/f2fs/data.c
> > > > > > > > > > > +++ b/fs/f2fs/data.c
> > > > > > > > > > > @@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
> > > > > > > > > > >          /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
> > > > > > > > > > >          if (wbc->sync_mode == WB_SYNC_ALL)
> > > > > > > > > > >              atomic_inc(&sbi->wb_sync_req[DATA]);
> > > > > > > > > > > -    else if (atomic_read(&sbi->wb_sync_req[DATA]))
> > > > > > > > > > > +    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
> > > > > > > > > > > +        /* to avoid potential deadlock */
> > > > > > > > > > > +        if (current->plug)
> > > > > > > > > > > +            blk_finish_plug(current->plug);
> > > > > > > > > > >              goto skip_write;
> > > > > > > > > > > +    }
> > > > > > > > > > >          if (__should_serialize_io(inode, wbc)) {
> > > > > > > > > > >              mutex_lock(&sbi->writepages);
> > > > > > > > > > > diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
> > > > > > > > > > > index 556fcd8457f3..69c6bcaf5aae 100644
> > > > > > > > > > > --- a/fs/f2fs/node.c
> > > > > > > > > > > +++ b/fs/f2fs/node.c
> > > > > > > > > > > @@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
> > > > > > > > > > >          if (wbc->sync_mode == WB_SYNC_ALL)
> > > > > > > > > > >              atomic_inc(&sbi->wb_sync_req[NODE]);
> > > > > > > > > > > -    else if (atomic_read(&sbi->wb_sync_req[NODE]))
> > > > > > > > > > > +    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
> > > > > > > > > > > +        /* to avoid potential deadlock */
> > > > > > > > > > > +        if (current->plug)
> > > > > > > > > > > +            blk_finish_plug(current->plug);
> > > > > > > > > > >              goto skip_write;
> > > > > > > > > > > +    }
> > > > > > > > > > >          trace_f2fs_writepages(mapping->host, wbc, NODE);
> > > > > > > > > > > -- 
> > > > > > > > > > > 2.32.0
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > Linux-f2fs-devel mailing list
> > > > > Linux-f2fs-devel@lists.sourceforge.net
> > > > > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> > > > 
> > > > 
> > > > _______________________________________________
> > > > Linux-f2fs-devel mailing list
> > > > Linux-f2fs-devel@lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Chao Yu]

On 2022/3/2 13:26, Jaegeuk Kim wrote:
> On 03/02, Chao Yu wrote:
>> ping,
>>
>> On 2022/2/25 11:02, Chao Yu wrote:
>>> On 2022/2/3 22:57, Chao Yu wrote:
>>>> On 2022/2/3 9:51, Jaegeuk Kim wrote:
>>>>> On 01/29, Chao Yu wrote:
>>>>>> On 2022/1/29 8:37, Jaegeuk Kim wrote:
>>>>>>> On 01/28, Chao Yu wrote:
>>>>>>>> On 2022/1/28 5:59, Jaegeuk Kim wrote:
>>>>>>>>> On 01/27, Chao Yu wrote:
>>>>>>>>>> Quoted from Jing Xia's report, there is a potential deadlock may happen
>>>>>>>>>> between kworker and checkpoint as below:
>>>>>>>>>>
>>>>>>>>>> [T:writeback]                [T:checkpoint]
>>>>>>>>>> - wb_writeback
>>>>>>>>>>      - blk_start_plug
>>>>>>>>>> bio contains NodeA was plugged in writeback threads
>>>>>>>>>
>>>>>>>>> I'm still trying to understand more precisely. So, how is it possible to
>>>>>>>>> have bio having node write in this current context?
>>>>>>>>
>>>>>>>> IMO, after above blk_start_plug(), it may plug some inode's node page in kworker
>>>>>>>> during writebacking node_inode's data page (which should be node page)?
>>>>>>>
>>>>>>> Wasn't that added into a different task->plug?
>>>>>>
>>>>>> I'm not sure I've got your concern correctly...
>>>>>>
>>>>>> Do you mean NodeA and other IOs from do_writepages() were plugged in
>>>>>> different local plug variables?
>>>>>
>>>>> I think so.
>>>>
>>>> I guess block plug helper says it doesn't allow to use nested plug, so there
>>>> is only one plug in kworker thread?
> 
> Is there only one kworker thread that flushes node and inode pages?

IIRC, =one kworker per block device?

Thanks,

> 
>>>>
>>>> void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
>>>> {
>>>>       struct task_struct *tsk = current;
>>>>
>>>>       /*
>>>>        * If this is a nested plug, don't actually assign it.
>>>>        */
>>>>       if (tsk->plug)
>>>>           return;
>>>> ...
>>>> }
>>>
>>> Any further comments?
>>>
>>> Thanks,
>>>
>>>>
>>>> Thanks,
>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>                      - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
>>>>>>>>>>                       - f2fs_write_data_pages
>>>>>>>>>>                        - f2fs_write_single_data_page -- write last dirty page
>>>>>>>>>>                         - f2fs_do_write_data_page
>>>>>>>>>>                          - set_page_writeback  -- clear page dirty flag and
>>>>>>>>>>                          PAGECACHE_TAG_DIRTY tag in radix tree
>>>>>>>>>>                          - f2fs_outplace_write_data
>>>>>>>>>>                           - f2fs_update_data_blkaddr
>>>>>>>>>>                            - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
>>>>>>>>>>                         - inode_dec_dirty_pages
>>>>>>>>>>      - writeback_sb_inodes
>>>>>>>>>>       - writeback_single_inode
>>>>>>>>>>        - do_writepages
>>>>>>>>>>         - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
>>>>>>>>>>          - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
>>>>>>>>>>       - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
>>>>>>>>>>      - blk_finish_plug
>>>>>>>>>>
>>>>>>>>>> Let's try to avoid deadlock condition by forcing unplugging previous bio via
>>>>>>>>>> blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
>>>>>>>>>> due to valid sbi->wb_sync_req[DATA/NODE].
>>>>>>>>>>
>>>>>>>>>> Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
>>>>>>>>>> Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
>>>>>>>>>> Signed-off-by: Jing Xia <jing.xia@unisoc.com>
>>>>>>>>>> Signed-off-by: Chao Yu <chao@kernel.org>
>>>>>>>>>> ---
>>>>>>>>>>      fs/f2fs/data.c | 6 +++++-
>>>>>>>>>>      fs/f2fs/node.c | 6 +++++-
>>>>>>>>>>      2 files changed, 10 insertions(+), 2 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
>>>>>>>>>> index 76d6fe7b0c8f..932a4c81acaf 100644
>>>>>>>>>> --- a/fs/f2fs/data.c
>>>>>>>>>> +++ b/fs/f2fs/data.c
>>>>>>>>>> @@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
>>>>>>>>>>          /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
>>>>>>>>>>          if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>>>>              atomic_inc(&sbi->wb_sync_req[DATA]);
>>>>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[DATA]))
>>>>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
>>>>>>>>>> +        /* to avoid potential deadlock */
>>>>>>>>>> +        if (current->plug)
>>>>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>>>>              goto skip_write;
>>>>>>>>>> +    }
>>>>>>>>>>          if (__should_serialize_io(inode, wbc)) {
>>>>>>>>>>              mutex_lock(&sbi->writepages);
>>>>>>>>>> diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
>>>>>>>>>> index 556fcd8457f3..69c6bcaf5aae 100644
>>>>>>>>>> --- a/fs/f2fs/node.c
>>>>>>>>>> +++ b/fs/f2fs/node.c
>>>>>>>>>> @@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
>>>>>>>>>>          if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>>>>              atomic_inc(&sbi->wb_sync_req[NODE]);
>>>>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[NODE]))
>>>>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
>>>>>>>>>> +        /* to avoid potential deadlock */
>>>>>>>>>> +        if (current->plug)
>>>>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>>>>              goto skip_write;
>>>>>>>>>> +    }
>>>>>>>>>>          trace_f2fs_writepages(mapping->host, wbc, NODE);
>>>>>>>>>> -- 
>>>>>>>>>> 2.32.0
>>>>
>>>>
>>>> _______________________________________________
>>>> Linux-f2fs-devel mailing list
>>>> Linux-f2fs-devel@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>>
>>>
>>> _______________________________________________
>>> Linux-f2fs-devel mailing list
>>> Linux-f2fs-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Jaegeuk Kim]

On 03/02, Chao Yu wrote:
> ping,
> 
> On 2022/2/25 11:02, Chao Yu wrote:
> > On 2022/2/3 22:57, Chao Yu wrote:
> > > On 2022/2/3 9:51, Jaegeuk Kim wrote:
> > > > On 01/29, Chao Yu wrote:
> > > > > On 2022/1/29 8:37, Jaegeuk Kim wrote:
> > > > > > On 01/28, Chao Yu wrote:
> > > > > > > On 2022/1/28 5:59, Jaegeuk Kim wrote:
> > > > > > > > On 01/27, Chao Yu wrote:
> > > > > > > > > Quoted from Jing Xia's report, there is a potential deadlock may happen
> > > > > > > > > between kworker and checkpoint as below:
> > > > > > > > > 
> > > > > > > > > [T:writeback]                [T:checkpoint]
> > > > > > > > > - wb_writeback
> > > > > > > > >     - blk_start_plug
> > > > > > > > > bio contains NodeA was plugged in writeback threads
> > > > > > > > 
> > > > > > > > I'm still trying to understand more precisely. So, how is it possible to
> > > > > > > > have bio having node write in this current context?
> > > > > > > 
> > > > > > > IMO, after above blk_start_plug(), it may plug some inode's node page in kworker
> > > > > > > during writebacking node_inode's data page (which should be node page)?
> > > > > > 
> > > > > > Wasn't that added into a different task->plug?
> > > > > 
> > > > > I'm not sure I've got your concern correctly...
> > > > > 
> > > > > Do you mean NodeA and other IOs from do_writepages() were plugged in
> > > > > different local plug variables?
> > > > 
> > > > I think so.
> > > 
> > > I guess block plug helper says it doesn't allow to use nested plug, so there
> > > is only one plug in kworker thread?

Is there only one kworker thread that flushes node and inode pages?

> > > 
> > > void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
> > > {
> > >      struct task_struct *tsk = current;
> > > 
> > >      /*
> > >       * If this is a nested plug, don't actually assign it.
> > >       */
> > >      if (tsk->plug)
> > >          return;
> > > ...
> > > }
> > 
> > Any further comments?
> > 
> > Thanks,
> > 
> > > 
> > > Thanks,
> > > 
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > 
> > > > > > > > 
> > > > > > > > >                     - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
> > > > > > > > >                      - f2fs_write_data_pages
> > > > > > > > >                       - f2fs_write_single_data_page -- write last dirty page
> > > > > > > > >                        - f2fs_do_write_data_page
> > > > > > > > >                         - set_page_writeback  -- clear page dirty flag and
> > > > > > > > >                         PAGECACHE_TAG_DIRTY tag in radix tree
> > > > > > > > >                         - f2fs_outplace_write_data
> > > > > > > > >                          - f2fs_update_data_blkaddr
> > > > > > > > >                           - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
> > > > > > > > >                        - inode_dec_dirty_pages
> > > > > > > > >     - writeback_sb_inodes
> > > > > > > > >      - writeback_single_inode
> > > > > > > > >       - do_writepages
> > > > > > > > >        - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
> > > > > > > > >         - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
> > > > > > > > >      - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
> > > > > > > > >     - blk_finish_plug
> > > > > > > > > 
> > > > > > > > > Let's try to avoid deadlock condition by forcing unplugging previous bio via
> > > > > > > > > blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
> > > > > > > > > due to valid sbi->wb_sync_req[DATA/NODE].
> > > > > > > > > 
> > > > > > > > > Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
> > > > > > > > > Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
> > > > > > > > > Signed-off-by: Jing Xia <jing.xia@unisoc.com>
> > > > > > > > > Signed-off-by: Chao Yu <chao@kernel.org>
> > > > > > > > > ---
> > > > > > > > >     fs/f2fs/data.c | 6 +++++-
> > > > > > > > >     fs/f2fs/node.c | 6 +++++-
> > > > > > > > >     2 files changed, 10 insertions(+), 2 deletions(-)
> > > > > > > > > 
> > > > > > > > > diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
> > > > > > > > > index 76d6fe7b0c8f..932a4c81acaf 100644
> > > > > > > > > --- a/fs/f2fs/data.c
> > > > > > > > > +++ b/fs/f2fs/data.c
> > > > > > > > > @@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
> > > > > > > > >         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
> > > > > > > > >         if (wbc->sync_mode == WB_SYNC_ALL)
> > > > > > > > >             atomic_inc(&sbi->wb_sync_req[DATA]);
> > > > > > > > > -    else if (atomic_read(&sbi->wb_sync_req[DATA]))
> > > > > > > > > +    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
> > > > > > > > > +        /* to avoid potential deadlock */
> > > > > > > > > +        if (current->plug)
> > > > > > > > > +            blk_finish_plug(current->plug);
> > > > > > > > >             goto skip_write;
> > > > > > > > > +    }
> > > > > > > > >         if (__should_serialize_io(inode, wbc)) {
> > > > > > > > >             mutex_lock(&sbi->writepages);
> > > > > > > > > diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
> > > > > > > > > index 556fcd8457f3..69c6bcaf5aae 100644
> > > > > > > > > --- a/fs/f2fs/node.c
> > > > > > > > > +++ b/fs/f2fs/node.c
> > > > > > > > > @@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
> > > > > > > > >         if (wbc->sync_mode == WB_SYNC_ALL)
> > > > > > > > >             atomic_inc(&sbi->wb_sync_req[NODE]);
> > > > > > > > > -    else if (atomic_read(&sbi->wb_sync_req[NODE]))
> > > > > > > > > +    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
> > > > > > > > > +        /* to avoid potential deadlock */
> > > > > > > > > +        if (current->plug)
> > > > > > > > > +            blk_finish_plug(current->plug);
> > > > > > > > >             goto skip_write;
> > > > > > > > > +    }
> > > > > > > > >         trace_f2fs_writepages(mapping->host, wbc, NODE);
> > > > > > > > > -- 
> > > > > > > > > 2.32.0
> > > 
> > > 
> > > _______________________________________________
> > > Linux-f2fs-devel mailing list
> > > Linux-f2fs-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> > 
> > 
> > _______________________________________________
> > Linux-f2fs-devel mailing list
> > Linux-f2fs-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Chao Yu]

ping,

On 2022/2/25 11:02, Chao Yu wrote:
> On 2022/2/3 22:57, Chao Yu wrote:
>> On 2022/2/3 9:51, Jaegeuk Kim wrote:
>>> On 01/29, Chao Yu wrote:
>>>> On 2022/1/29 8:37, Jaegeuk Kim wrote:
>>>>> On 01/28, Chao Yu wrote:
>>>>>> On 2022/1/28 5:59, Jaegeuk Kim wrote:
>>>>>>> On 01/27, Chao Yu wrote:
>>>>>>>> Quoted from Jing Xia's report, there is a potential deadlock may happen
>>>>>>>> between kworker and checkpoint as below:
>>>>>>>>
>>>>>>>> [T:writeback]                [T:checkpoint]
>>>>>>>> - wb_writeback
>>>>>>>>     - blk_start_plug
>>>>>>>> bio contains NodeA was plugged in writeback threads
>>>>>>>
>>>>>>> I'm still trying to understand more precisely. So, how is it possible to
>>>>>>> have bio having node write in this current context?
>>>>>>
>>>>>> IMO, after above blk_start_plug(), it may plug some inode's node page in kworker
>>>>>> during writebacking node_inode's data page (which should be node page)?
>>>>>
>>>>> Wasn't that added into a different task->plug?
>>>>
>>>> I'm not sure I've got your concern correctly...
>>>>
>>>> Do you mean NodeA and other IOs from do_writepages() were plugged in
>>>> different local plug variables?
>>>
>>> I think so.
>>
>> I guess block plug helper says it doesn't allow to use nested plug, so there
>> is only one plug in kworker thread?
>>
>> void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
>> {
>>      struct task_struct *tsk = current;
>>
>>      /*
>>       * If this is a nested plug, don't actually assign it.
>>       */
>>      if (tsk->plug)
>>          return;
>> ...
>> }
> 
> Any further comments?
> 
> Thanks,
> 
>>
>> Thanks,
>>
>>>
>>>>
>>>> Thanks,
>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>>
>>>>>>>>                     - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
>>>>>>>>                      - f2fs_write_data_pages
>>>>>>>>                       - f2fs_write_single_data_page -- write last dirty page
>>>>>>>>                        - f2fs_do_write_data_page
>>>>>>>>                         - set_page_writeback  -- clear page dirty flag and
>>>>>>>>                         PAGECACHE_TAG_DIRTY tag in radix tree
>>>>>>>>                         - f2fs_outplace_write_data
>>>>>>>>                          - f2fs_update_data_blkaddr
>>>>>>>>                           - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
>>>>>>>>                        - inode_dec_dirty_pages
>>>>>>>>     - writeback_sb_inodes
>>>>>>>>      - writeback_single_inode
>>>>>>>>       - do_writepages
>>>>>>>>        - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
>>>>>>>>         - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
>>>>>>>>      - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
>>>>>>>>     - blk_finish_plug
>>>>>>>>
>>>>>>>> Let's try to avoid deadlock condition by forcing unplugging previous bio via
>>>>>>>> blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
>>>>>>>> due to valid sbi->wb_sync_req[DATA/NODE].
>>>>>>>>
>>>>>>>> Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
>>>>>>>> Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
>>>>>>>> Signed-off-by: Jing Xia <jing.xia@unisoc.com>
>>>>>>>> Signed-off-by: Chao Yu <chao@kernel.org>
>>>>>>>> ---
>>>>>>>>     fs/f2fs/data.c | 6 +++++-
>>>>>>>>     fs/f2fs/node.c | 6 +++++-
>>>>>>>>     2 files changed, 10 insertions(+), 2 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
>>>>>>>> index 76d6fe7b0c8f..932a4c81acaf 100644
>>>>>>>> --- a/fs/f2fs/data.c
>>>>>>>> +++ b/fs/f2fs/data.c
>>>>>>>> @@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
>>>>>>>>         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
>>>>>>>>         if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>>             atomic_inc(&sbi->wb_sync_req[DATA]);
>>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[DATA]))
>>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
>>>>>>>> +        /* to avoid potential deadlock */
>>>>>>>> +        if (current->plug)
>>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>>             goto skip_write;
>>>>>>>> +    }
>>>>>>>>         if (__should_serialize_io(inode, wbc)) {
>>>>>>>>             mutex_lock(&sbi->writepages);
>>>>>>>> diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
>>>>>>>> index 556fcd8457f3..69c6bcaf5aae 100644
>>>>>>>> --- a/fs/f2fs/node.c
>>>>>>>> +++ b/fs/f2fs/node.c
>>>>>>>> @@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
>>>>>>>>         if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>>             atomic_inc(&sbi->wb_sync_req[NODE]);
>>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[NODE]))
>>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
>>>>>>>> +        /* to avoid potential deadlock */
>>>>>>>> +        if (current->plug)
>>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>>             goto skip_write;
>>>>>>>> +    }
>>>>>>>>         trace_f2fs_writepages(mapping->host, wbc, NODE);
>>>>>>>> -- 
>>>>>>>> 2.32.0
>>
>>
>> _______________________________________________
>> Linux-f2fs-devel mailing list
>> Linux-f2fs-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> 
> 
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] [PATCH] f2fs: fix to avoid potential deadlock

[Chao Yu]

On 2022/2/3 22:57, Chao Yu wrote:
> On 2022/2/3 9:51, Jaegeuk Kim wrote:
>> On 01/29, Chao Yu wrote:
>>> On 2022/1/29 8:37, Jaegeuk Kim wrote:
>>>> On 01/28, Chao Yu wrote:
>>>>> On 2022/1/28 5:59, Jaegeuk Kim wrote:
>>>>>> On 01/27, Chao Yu wrote:
>>>>>>> Quoted from Jing Xia's report, there is a potential deadlock may happen
>>>>>>> between kworker and checkpoint as below:
>>>>>>>
>>>>>>> [T:writeback]                [T:checkpoint]
>>>>>>> - wb_writeback
>>>>>>>     - blk_start_plug
>>>>>>> bio contains NodeA was plugged in writeback threads
>>>>>>
>>>>>> I'm still trying to understand more precisely. So, how is it possible to
>>>>>> have bio having node write in this current context?
>>>>>
>>>>> IMO, after above blk_start_plug(), it may plug some inode's node page in kworker
>>>>> during writebacking node_inode's data page (which should be node page)?
>>>>
>>>> Wasn't that added into a different task->plug?
>>>
>>> I'm not sure I've got your concern correctly...
>>>
>>> Do you mean NodeA and other IOs from do_writepages() were plugged in
>>> different local plug variables?
>>
>> I think so.
> 
> I guess block plug helper says it doesn't allow to use nested plug, so there
> is only one plug in kworker thread?
> 
> void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
> {
>      struct task_struct *tsk = current;
> 
>      /*
>       * If this is a nested plug, don't actually assign it.
>       */
>      if (tsk->plug)
>          return;
> ...
> }

Any further comments?

Thanks,

> 
> Thanks,
> 
>>
>>>
>>> Thanks,
>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>>
>>>>>>>                     - do_writepages  -- sync write inodeB, inc wb_sync_req[DATA]
>>>>>>>                      - f2fs_write_data_pages
>>>>>>>                       - f2fs_write_single_data_page -- write last dirty page
>>>>>>>                        - f2fs_do_write_data_page
>>>>>>>                         - set_page_writeback  -- clear page dirty flag and
>>>>>>>                         PAGECACHE_TAG_DIRTY tag in radix tree
>>>>>>>                         - f2fs_outplace_write_data
>>>>>>>                          - f2fs_update_data_blkaddr
>>>>>>>                           - f2fs_wait_on_page_writeback -- wait NodeA to writeback here
>>>>>>>                        - inode_dec_dirty_pages
>>>>>>>     - writeback_sb_inodes
>>>>>>>      - writeback_single_inode
>>>>>>>       - do_writepages
>>>>>>>        - f2fs_write_data_pages -- skip writepages due to wb_sync_req[DATA]
>>>>>>>         - wbc->pages_skipped += get_dirty_pages() -- PAGECACHE_TAG_DIRTY is not set but get_dirty_pages() returns one
>>>>>>>      - requeue_inode -- requeue inode to wb->b_dirty queue due to non-zero.pages_skipped
>>>>>>>     - blk_finish_plug
>>>>>>>
>>>>>>> Let's try to avoid deadlock condition by forcing unplugging previous bio via
>>>>>>> blk_finish_plug(current->plug) once we'v skipped writeback in writepages()
>>>>>>> due to valid sbi->wb_sync_req[DATA/NODE].
>>>>>>>
>>>>>>> Fixes: 687de7f1010c ("f2fs: avoid IO split due to mixed WB_SYNC_ALL and WB_SYNC_NONE")
>>>>>>> Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
>>>>>>> Signed-off-by: Jing Xia <jing.xia@unisoc.com>
>>>>>>> Signed-off-by: Chao Yu <chao@kernel.org>
>>>>>>> ---
>>>>>>>     fs/f2fs/data.c | 6 +++++-
>>>>>>>     fs/f2fs/node.c | 6 +++++-
>>>>>>>     2 files changed, 10 insertions(+), 2 deletions(-)
>>>>>>>
>>>>>>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
>>>>>>> index 76d6fe7b0c8f..932a4c81acaf 100644
>>>>>>> --- a/fs/f2fs/data.c
>>>>>>> +++ b/fs/f2fs/data.c
>>>>>>> @@ -3174,8 +3174,12 @@ static int __f2fs_write_data_pages(struct address_space *mapping,
>>>>>>>         /* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
>>>>>>>         if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>             atomic_inc(&sbi->wb_sync_req[DATA]);
>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[DATA]))
>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[DATA])) {
>>>>>>> +        /* to avoid potential deadlock */
>>>>>>> +        if (current->plug)
>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>             goto skip_write;
>>>>>>> +    }
>>>>>>>         if (__should_serialize_io(inode, wbc)) {
>>>>>>>             mutex_lock(&sbi->writepages);
>>>>>>> diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
>>>>>>> index 556fcd8457f3..69c6bcaf5aae 100644
>>>>>>> --- a/fs/f2fs/node.c
>>>>>>> +++ b/fs/f2fs/node.c
>>>>>>> @@ -2106,8 +2106,12 @@ static int f2fs_write_node_pages(struct address_space *mapping,
>>>>>>>         if (wbc->sync_mode == WB_SYNC_ALL)
>>>>>>>             atomic_inc(&sbi->wb_sync_req[NODE]);
>>>>>>> -    else if (atomic_read(&sbi->wb_sync_req[NODE]))
>>>>>>> +    else if (atomic_read(&sbi->wb_sync_req[NODE])) {
>>>>>>> +        /* to avoid potential deadlock */
>>>>>>> +        if (current->plug)
>>>>>>> +            blk_finish_plug(current->plug);
>>>>>>>             goto skip_write;
>>>>>>> +    }
>>>>>>>         trace_f2fs_writepages(mapping->host, wbc, NODE);
>>>>>>> -- 
>>>>>>> 2.32.0
> 
> 
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel