From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 233FEC433F5 for ; Thu, 14 Apr 2022 01:55:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239557AbiDNB5g (ORCPT ); Wed, 13 Apr 2022 21:57:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239548AbiDNB5e (ORCPT ); Wed, 13 Apr 2022 21:57:34 -0400 Received: from support.corp-email.com (support.corp-email.com [222.73.234.235]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93C052CC8F for ; Wed, 13 Apr 2022 18:55:06 -0700 (PDT) Received: from ([114.119.32.142]) by support.corp-email.com ((D)) with ASMTP (SSL) id IVQ00159; Thu, 14 Apr 2022 09:54:59 +0800 Received: from [172.16.35.4] (172.16.35.4) by GCY-MBS-28.TCL.com (10.136.3.28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Thu, 14 Apr 2022 09:54:59 +0800 Message-ID: <39c4ded0-09c0-3e38-85cb-5535099b177d@tcl.com> Date: Thu, 14 Apr 2022 09:54:58 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: [PATCH] f2fs: avoid deadlock in gc thread under low memory Content-Language: en-US To: Jaegeuk Kim CC: , , References: <660530eb62e71fb6520d3596704162e5@sslemail.net> From: Wu Yan In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [172.16.35.4] X-ClientProxiedBy: GCY-EXS-21.TCL.com (10.74.128.61) To GCY-MBS-28.TCL.com (10.136.3.28) tUid: 2022414095459fee587205794eb62e0e39d5e16303e5f X-Abuse-Reports-To: service@corp-email.com Abuse-Reports-To: service@corp-email.com X-Complaints-To: service@corp-email.com X-Report-Abuse-To: service@corp-email.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 4/14/22 01:00, Jaegeuk Kim wrote: > On 04/13, Rokudo Yan wrote: >> There is a potential deadlock in gc thread may happen >> under low memory as below: >> >> gc_thread_func >> -f2fs_gc >> -do_garbage_collect >> -gc_data_segment >> -move_data_block >> -set_page_writeback(fio.encrypted_page); >> -f2fs_submit_page_write >> as f2fs_submit_page_write try to do io merge when possible, so the >> encrypted_page is marked PG_writeback but may not submit to block >> layer immediately, if system enter low memory when gc thread try >> to move next data block, it may do direct reclaim and enter fs layer >> as below: >> -move_data_block >> -f2fs_grab_cache_page(index=?, for_write=false) >> -grab_cache_page >> -find_or_create_page >> -pagecache_get_page >> -__page_cache_alloc -- __GFP_FS is set >> -alloc_pages_node >> -__alloc_pages >> -__alloc_pages_slowpath >> -__alloc_pages_direct_reclaim >> -__perform_reclaim >> -try_to_free_pages >> -do_try_to_free_pages >> -shrink_zones >> -mem_cgroup_soft_limit_reclaim >> -mem_cgroup_soft_reclaim >> -mem_cgroup_shrink_node >> -shrink_node_memcg >> -shrink_list >> -shrink_inactive_list >> -shrink_page_list >> -wait_on_page_writeback -- the page is marked >> writeback during previous move_data_block call >> >> the gc thread wait for the encrypted_page writeback complete, >> but as gc thread held sbi->gc_lock, the writeback & sync thread >> may blocked waiting for sbi->gc_lock, so the bio contain the >> encrypted_page may nerver submit to block layer and complete the >> writeback, which cause deadlock. To avoid this deadlock condition, >> we mark the gc thread with PF_MEMALLOC_NOFS flag, then it will nerver >> enter fs layer when try to alloc cache page during move_data_block. >> >> Signed-off-by: Rokudo Yan >> --- >> fs/f2fs/gc.c | 6 ++++++ >> 1 file changed, 6 insertions(+) >> >> diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c >> index e020804f7b07..cc71f77b98c8 100644 >> --- a/fs/f2fs/gc.c >> +++ b/fs/f2fs/gc.c >> @@ -38,6 +38,12 @@ static int gc_thread_func(void *data) >> >> wait_ms = gc_th->min_sleep_time; >> >> + /* >> + * Make sure that no allocations from gc thread will ever >> + * recurse to the fs layer to avoid deadlock as it will >> + * hold sbi->gc_lock during garbage collection >> + */ >> + memalloc_nofs_save(); > > I think this cannot cover all the f2fs_gc() call cases. Can we just avoid by: > > --- a/fs/f2fs/gc.c > +++ b/fs/f2fs/gc.c > @@ -1233,7 +1233,7 @@ static int move_data_block(struct inode *inode, block_t bidx, > CURSEG_ALL_DATA_ATGC : CURSEG_COLD_DATA; > > /* do not read out */ > - page = f2fs_grab_cache_page(inode->i_mapping, bidx, false); > + page = f2fs_grab_cache_page(inode->i_mapping, bidx, true); > if (!page) > return -ENOMEM; > > Thanks, > >> set_freezable(); >> do { >> bool sync_mode, foreground = false; >> -- >> 2.25.1 Hi, Jaegeuk I'm not sure if any other case may trigger the issue, but the stack traces I have caught so far are all the same as below: f2fs_gc-253:12 D 226966.808196 572 302561 150976 0x1200840 0x0 572 237207473347056 __switch_to+0x134/0x150 __schedule+0xd5c/0x1100 io_schedule+0x90/0xc0 wait_on_page_bit+0x194/0x208 shrink_page_list+0x62c/0xe74 shrink_inactive_list+0x2c0/0x698 shrink_node_memcg+0x3dc/0x97c mem_cgroup_shrink_node+0x144/0x218 mem_cgroup_soft_limit_reclaim+0x188/0x47c do_try_to_free_pages+0x204/0x3a0 try_to_free_pages+0x35c/0x4d0 __alloc_pages_nodemask+0x7a4/0x10d0 pagecache_get_page+0x184/0x2ec do_garbage_collect+0xfe0/0x2828 f2fs_gc+0x4a0/0x8ec gc_thread_func+0x240/0x4d4 kthread+0x17c/0x18c ret_from_fork+0x10/0x18 Thanks yanwu