[PATCH] fix dead lock caused by ocfs2_defrag_extent

[PATCH] fix dead lock caused by ocfs2_defrag_extent

[Larry Chen]

ocfs2_defrag_extent may fall into dead lock.

ocfs2_ioctl_move_extents
  ocfs2_ioctl_move_extents	
    ocfs2_move_extents
      ocfs2_defrag_extent
        ocfs2_lock_allocators_move_extents  

          ocfs2_reserve_clusters
            inode_lock GLOBAL_BITMAP_SYSTEM_INODE

	  __ocfs2_flush_truncate_log
            inode_lock GLOBAL_BITMAP_SYSTEM_INODE

As back trace shows above, ocfs2_reserve_clusters will call inode_lock
against the global bitmap if local allocator has not sufficient cluters. 
Once global bitmap could meet the demand, ocfs2_reserve_cluster will 
return success with global bitmap locked.

After ocfs2_reserve_cluster, if truncate log is full, 
__ocfs2_flush_truncate_log will definitely fall into dead lock
because it needs to inode_lock global bitmap, which has
already been locked.

To fix this bug, we could remove from ocfs2_lock_allocators_move_extents 
the code which intends to lock global allocator, and put the removed
code after __ocfs2_flush_truncate_log

The ocfs2_lock_allocators_move_extents has been refered by 2 places, one
is here, the other does not need the data allocator context, which means
this patch does not affect the caller so far.



Signed-off-by: Larry Chen <lchen@suse.com>
---
 fs/ocfs2/move_extents.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/fs/ocfs2/move_extents.c b/fs/ocfs2/move_extents.c
index 7eb3b0a6347e..064dedf40d74 100644
--- a/fs/ocfs2/move_extents.c
+++ b/fs/ocfs2/move_extents.c
@@ -192,13 +192,6 @@ static int ocfs2_lock_allocators_move_extents(struct inode *inode,
 		goto out;
 	}
 
-	if (data_ac) {
-		ret = ocfs2_reserve_clusters(osb, clusters_to_move, data_ac);
-		if (ret) {
-			mlog_errno(ret);
-			goto out;
-		}
-	}
 
 	*credits += ocfs2_calc_extend_credits(osb->sb, et->et_root_el);
 
@@ -283,6 +276,12 @@ static int ocfs2_defrag_extent(struct ocfs2_move_extents_context *context,
 		}
 	}
 
+	ret = ocfs2_reserve_clusters(osb, *len, &context->data_ac);
+	if (ret) {
+		mlog_errno(ret);
+		goto out_unlock_mutex;
+	}
+
 	handle = ocfs2_start_trans(osb, credits);
 	if (IS_ERR(handle)) {
 		ret = PTR_ERR(handle);
-- 
2.13.7

Re: [Ocfs2-devel] [PATCH] fix dead lock caused by ocfs2_defrag_extent

[Joseph Qi]

Hi Larry,

On 18/8/27 16:01, Larry Chen wrote:
> ocfs2_defrag_extent may fall into dead lock.
> 
> ocfs2_ioctl_move_extents
>   ocfs2_ioctl_move_extents	
>     ocfs2_move_extents
>       ocfs2_defrag_extent
>         ocfs2_lock_allocators_move_extents  
> 
>           ocfs2_reserve_clusters
>             inode_lock GLOBAL_BITMAP_SYSTEM_INODE
> 
> 	  __ocfs2_flush_truncate_log
>             inode_lock GLOBAL_BITMAP_SYSTEM_INODE
> 
> As back trace shows above, ocfs2_reserve_clusters will call inode_lock
> against the global bitmap if local allocator has not sufficient cluters. 
> Once global bitmap could meet the demand, ocfs2_reserve_cluster will 
> return success with global bitmap locked.
> 
> After ocfs2_reserve_cluster, if truncate log is full, 
> __ocfs2_flush_truncate_log will definitely fall into dead lock
> because it needs to inode_lock global bitmap, which has
> already been locked.
> 
> To fix this bug, we could remove from ocfs2_lock_allocators_move_extents 
> the code which intends to lock global allocator, and put the removed
> code after __ocfs2_flush_truncate_log
> 
> The ocfs2_lock_allocators_move_extents has been refered by 2 places, one
> is here, the other does not need the data allocator context, which means
> this patch does not affect the caller so far.
> 
After this change, data_ac in ocfs2_lock_allocators_move_extents() is
no longer used. So I'd prefer we also clean it up.

Thanks,
Joseph

> 
> 
> Signed-off-by: Larry Chen <lchen@suse.com>
> ---
>  fs/ocfs2/move_extents.c | 13 ++++++-------
>  1 file changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/fs/ocfs2/move_extents.c b/fs/ocfs2/move_extents.c
> index 7eb3b0a6347e..064dedf40d74 100644
> --- a/fs/ocfs2/move_extents.c
> +++ b/fs/ocfs2/move_extents.c
> @@ -192,13 +192,6 @@ static int ocfs2_lock_allocators_move_extents(struct inode *inode,
>  		goto out;
>  	}
>  
> -	if (data_ac) {
> -		ret = ocfs2_reserve_clusters(osb, clusters_to_move, data_ac);
> -		if (ret) {
> -			mlog_errno(ret);
> -			goto out;
> -		}
> -	}
>  
>  	*credits += ocfs2_calc_extend_credits(osb->sb, et->et_root_el);
>  
> @@ -283,6 +276,12 @@ static int ocfs2_defrag_extent(struct ocfs2_move_extents_context *context,
>  		}
>  	}
>  
> +	ret = ocfs2_reserve_clusters(osb, *len, &context->data_ac);
> +	if (ret) {
> +		mlog_errno(ret);
> +		goto out_unlock_mutex;
> +	}
> +
>  	handle = ocfs2_start_trans(osb, credits);
>  	if (IS_ERR(handle)) {
>  		ret = PTR_ERR(handle);
> 

Re: [Ocfs2-devel] [PATCH] fix dead lock caused by ocfs2_defrag_extent

[Changwei Ge]

Hi Larry,

I'd like to propose another way to fix this issue, which might be easier.
Actually, ocfs2_reserve_clusters() inside will flush truncate log if 
there is no enough clusters left.
So how about just remove __ocfs2_flush_truncate_log() in 
ocfs2_defrag_extent()?

Thanks,
Changwei


On 2018/8/27 16:01, Larry Chen wrote:
> ocfs2_defrag_extent may fall into dead lock.
>
> ocfs2_ioctl_move_extents
>    ocfs2_ioctl_move_extents	
>      ocfs2_move_extents
>        ocfs2_defrag_extent
>          ocfs2_lock_allocators_move_extents
>
>            ocfs2_reserve_clusters
>              inode_lock GLOBAL_BITMAP_SYSTEM_INODE
>
> 	  __ocfs2_flush_truncate_log
>              inode_lock GLOBAL_BITMAP_SYSTEM_INODE
>
> As back trace shows above, ocfs2_reserve_clusters will call inode_lock
> against the global bitmap if local allocator has not sufficient cluters.
> Once global bitmap could meet the demand, ocfs2_reserve_cluster will
> return success with global bitmap locked.
>
> After ocfs2_reserve_cluster, if truncate log is full,
> __ocfs2_flush_truncate_log will definitely fall into dead lock
> because it needs to inode_lock global bitmap, which has
> already been locked.
>
> To fix this bug, we could remove from ocfs2_lock_allocators_move_extents
> the code which intends to lock global allocator, and put the removed
> code after __ocfs2_flush_truncate_log
>
> The ocfs2_lock_allocators_move_extents has been refered by 2 places, one
> is here, the other does not need the data allocator context, which means
> this patch does not affect the caller so far.
>
>
>
> Signed-off-by: Larry Chen <lchen@suse.com>
> ---
>   fs/ocfs2/move_extents.c | 13 ++++++-------
>   1 file changed, 6 insertions(+), 7 deletions(-)
>
> diff --git a/fs/ocfs2/move_extents.c b/fs/ocfs2/move_extents.c
> index 7eb3b0a6347e..064dedf40d74 100644
> --- a/fs/ocfs2/move_extents.c
> +++ b/fs/ocfs2/move_extents.c
> @@ -192,13 +192,6 @@ static int ocfs2_lock_allocators_move_extents(struct inode *inode,
>   		goto out;
>   	}
>   
> -	if (data_ac) {
> -		ret = ocfs2_reserve_clusters(osb, clusters_to_move, data_ac);
> -		if (ret) {
> -			mlog_errno(ret);
> -			goto out;
> -		}
> -	}
>   
>   	*credits += ocfs2_calc_extend_credits(osb->sb, et->et_root_el);
>   
> @@ -283,6 +276,12 @@ static int ocfs2_defrag_extent(struct ocfs2_move_extents_context *context,
>   		}
>   	}
>   
> +	ret = ocfs2_reserve_clusters(osb, *len, &context->data_ac);
> +	if (ret) {
> +		mlog_errno(ret);
> +		goto out_unlock_mutex;
> +	}
> +
>   	handle = ocfs2_start_trans(osb, credits);
>   	if (IS_ERR(handle)) {
>   		ret = PTR_ERR(handle);