From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3AE5934FF5A for ; Sat, 2 May 2026 12:41:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777725719; cv=none; b=kgeaGRpBGvJEzI59CyzSzO0VlLvRNra7sibY83TQn8w1h/TGAEvvc9BEDSYR+HT7jWSeuEMi1MWX1BdbaycUrt5IVZjrivHT43KUx30FZUaR59/xhB2Flu2Gg1m8SNg5vnjGVXr7sxFtqJ/tchPoLt49aMhnO8T5t7uWNQdHPis= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777725719; c=relaxed/simple; bh=z2L/hfDvuz8U0tZ7EyfMt+dCpPC+AB5aj/ElWefWuc0=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=q+xa4OrWO3Au1aCTQ7Vx4t9jjCGrM5B9tTR5kRlNRwAHr1ee7IYDtn2r5rPil/uBKbt6rpeimk5euLouEgtLH3urbUBgowdkgs33UxqZYCQ4ejENEJESc/e5rsUqLG16vFd2vfgzdZPcDZyKbam6+KoBdqftb1IvZWJQfSzU4X4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Iu9xu7T1; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Iu9xu7T1" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-35e576110adso1924969a91.0 for ; Sat, 02 May 2026 05:41:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1777725717; x=1778330517; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=MLJ2RJkBCgsAX/0kSFHltXvZ52HXQ2CM6/lwTXhqGWI=; b=Iu9xu7T134rC7sRSqdxrhWZTGWrLcq6YJ2AS7glq8inz/w1MdlmWEagBI20oHi9xXc 1ZkurlHE63qaIid2atHwkBanYnaxVZPzGXmUntUdYjVjdoNr5oVjUykNylZVezmtJQft CVP7K5HRYskw30snjOyfw8FlkpepXD+cciF+QNyh0cMpdHNSMkC6t1eigBNK0LnB6SFK OppSK7cGCcS6KHDKTZbB24vCs1n6fdswlTuhxB92i/BvBKUDTu7iwCGcII/DPOhF5gm9 2fSgurljvvN9/VA9ZA5qgdtncR2B3AFIq1/EYIUNBMr+pFhCxqc/2QUyhm/pkJS26rhG xZQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777725717; x=1778330517; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MLJ2RJkBCgsAX/0kSFHltXvZ52HXQ2CM6/lwTXhqGWI=; b=M0WtZeo3Eh2rMf6eNi46fGTUIRHUsjRSqYym6BVOupTqDCLrKeiqMBKq434hvucMqk 1kIgVHtk8IWL0SeeCr6e8humyrBkACNqshXL4KgVF3QLyXbyQ8wcdgEZK4pP+TceS+bJ VO5tn0EO9fnuJmiFIB03gJVSUp9zfCNakgIii2rxlA0Haedek4ksLsuZ5YnZDkGF8E29 RMrNsB52BOWi5VNpWTpA29jQ/PncYnuFeocWxCI6xYbBNRCiFMy9V7YWj53CzKEoJrrF KslmXJ6Y9bEYbhDTSaS196eywSo/ypXe2K9aPLoIu9M3X/6wwIXKIr7G/gewqv9MkTie wuQQ== X-Forwarded-Encrypted: i=1; AFNElJ+ThnTRlIhu+xQXbsITrt4NpUGO/tllW10wEkXolbAAoSXK82kmfVmkOp50Seji8MwFOa4whqAU+09lLGA=@vger.kernel.org X-Gm-Message-State: AOJu0YwzIDEjYzi7IAQaPieklSUgKDEfj+QZwn5KNkTnQ5WcBnABs276 a7PKuUaLw2lNu/FxRpxjX9ZbuGADnqo4PKHW5+F1WjfqTs5mGNcaMn+8 X-Gm-Gg: AeBDietlDa/+qa3H+Ka8PvRk/dlkc2lMOIic9hpdKe0ZvpmK8llec+HwFE/S4egBuZN W9sZU0E9yWYyx5rZqepdXTJRJC4nf9cEBBCdpkMprX+IWsqNIsAs9BGakXxlDfB5kJO1nRJd+SI mbzmvAgBjR/NxJ5ZYNC3sxToCyV/Jjk/P64l3I3EJjyZfDRGUk2NeLll3B8tMiKPNxZbS0zGckG dI4NB4UTR1dY0GvJjzJZqqjVPmyaWRuyWHCIBvo6H/Pugbg7hk33R1a05wAmy9cWc91UKQbgRvl VtH0+8BxzTuXlTRck8+BfVfVzv6EgqeCDXenCpEVPMil8032WvBgOmJiEjMoJ4aaEbgq0sBRNFW g/xORF22nwNbqtulC3xCeo16BG1m3EZR2jo3SxyQkUXjsGMOTrx2J4VndCgu60Lgxm3RKEtpe6g Cb9au2az7qNnHO/6SSPj0FbbewYj0TJYHqM4WN4RtSvI3TKSnr X-Received: by 2002:a17:90b:54cc:b0:35f:c6bf:2bba with SMTP id 98e67ed59e1d1-364c49a1ccemr9960980a91.11.1777725717362; Sat, 02 May 2026 05:41:57 -0700 (PDT) Received: from [10.121.80.58] ([210.184.73.204]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2b9caaba933sm49220705ad.28.2026.05.02.05.41.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 02 May 2026 05:41:56 -0700 (PDT) Message-ID: <531732db-4113-4a58-bff1-6a59f3c630b3@gmail.com> Date: Sat, 2 May 2026 20:41:52 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] f2fs: fix potential deadlock in f2fs_balance_fs() To: Chao Yu , jaegeuk@kernel.org Cc: linux-f2fs-devel@lists.sourceforge.net, linux-kernel@vger.kernel.org References: <20260426093239.165767-1-ruipengqi3@gmail.com> <70254f4c-80ce-4c53-ba60-be023d0cd6fc@gmail.com> <90abfbae-f999-4dad-9d71-d452a98d5ad8@kernel.org> Content-Language: en-US From: Ruipeng Qi In-Reply-To: <90abfbae-f999-4dad-9d71-d452a98d5ad8@kernel.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2026/4/29 15:59, Chao Yu wrote: > On 4/29/26 11:39, Ruipeng Qi wrote: >> >> On 2026/4/27 16:38, Chao Yu wrote: >>> On 4/26/26 17:32, ruipengqi wrote: >>>> From: Ruipeng Qi >>>> >>>> When the f2fs filesystem space is nearly exhausted, we encounter >>>> deadlock >>>> issues as below: >>>> >>>> INFO: task A:1890 blocked for more than 120 seconds. >>>>        Tainted: G           O       6.12.41-g3fe07ddf05ab #1 >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>> message. >>>> task:A    state:D stack:0     pid:1890  tgid:1626  ppid:1153 >>>> flags:0x00000204 >>>> Call trace: >>>>   __switch_to+0xf4/0x158 >>>>   __schedule+0x27c/0x908 >>>>   schedule+0x3c/0x118 >>>>   io_schedule+0x44/0x68 >>>>   folio_wait_bit_common+0x174/0x370 >>>>   folio_wait_bit+0x20/0x38 >>>>   folio_wait_writeback+0x54/0xc8 >>>>   truncate_inode_partial_folio+0x70/0x1e0 >>>>   truncate_inode_pages_range+0x1b0/0x450 >>>>   truncate_pagecache+0x54/0x88 >>>>   f2fs_file_write_iter+0x3e8/0xb80 >>>>   do_iter_readv_writev+0xf0/0x1e0 >>>>   vfs_writev+0x138/0x2c8 >>>>   do_writev+0x88/0x130 >>>>   __arm64_sys_writev+0x28/0x40 >>>>   invoke_syscall+0x50/0x120 >>>>   el0_svc_common.constprop.0+0xc8/0xf0 >>>>   do_el0_svc+0x24/0x38 >>>>   el0_svc+0x30/0xf8 >>>>   el0t_64_sync_handler+0x120/0x130 >>>>   el0t_64_sync+0x190/0x198 >>>> >>>> INFO: task kworker/u8:11:2680853 blocked for more than 120 seconds. >>>>        Tainted: G           O       6.12.41-g3fe07ddf05ab #1 >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>> message. >>>> task:kworker/u8:11   state:D stack:0     pid:2680853 tgid:2680853 >>>> ppid:2      flags:0x00000208 >>>> Workqueue: writeback wb_workfn (flush-254:0) >>>> Call trace: >>>>   __switch_to+0xf4/0x158 >>>>   __schedule+0x27c/0x908 >>>>   schedule+0x3c/0x118 >>>>   io_schedule+0x44/0x68 >>>>   folio_wait_bit_common+0x174/0x370 >>>>   __filemap_get_folio+0x214/0x348 >>>>   pagecache_get_page+0x20/0x70 >>>>   f2fs_get_read_data_page+0x150/0x3e8 >>>>   f2fs_get_lock_data_page+0x2c/0x160 >>>>   move_data_page+0x50/0x478 >>>>   do_garbage_collect+0xd38/0x1528 >>>>   f2fs_gc+0x240/0x7e0 >>>>   f2fs_balance_fs+0x1a0/0x208 >>>>   f2fs_write_single_data_page+0x6e4/0x730 //0xfffffe0d6ca08300 >>>>   f2fs_write_cache_pages+0x378/0x9b0 >>>>   f2fs_write_data_pages+0x2e4/0x388 >>>>   do_writepages+0x8c/0x2c8 >>>>   __writeback_single_inode+0x4c/0x498 >>>>   writeback_sb_inodes+0x234/0x4a8 >>>>   __writeback_inodes_wb+0x58/0x118 >>>>   wb_writeback+0x2f8/0x3c0 >>>>   wb_workfn+0x2c4/0x508 >>>>   process_one_work+0x180/0x408 >>>>   worker_thread+0x258/0x368 >>>>   kthread+0x118/0x128 >>>>   ret_from_fork+0x10/0x200 >>>> >>>> INFO: task kworker/u8:8:2641297 blocked for more than 120 seconds. >>>>        Tainted: G           O       6.12.41-g3fe07ddf05ab #1 >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>> message. >>>> task:kworker/u8:8    state:D stack:0     pid:2641297 tgid:2641297 >>>> ppid:2      flags:0x00000208 >>>> Workqueue: writeback wb_workfn (flush-254:0) >>>> Call trace: >>>>   __switch_to+0xf4/0x158 >>>>   __schedule+0x27c/0x908 >>>>   rt_mutex_schedule+0x30/0x60 >>>>   __rt_mutex_slowlock_locked.constprop.0+0x460/0x8a8 >>>>   rwbase_write_lock+0x24c/0x378 >>>>   down_write+0x1c/0x30 >>>>   f2fs_balance_fs+0x184/0x208 >>>>   f2fs_write_inode+0xf4/0x328 >>>>   __writeback_single_inode+0x370/0x498 >>>>   writeback_sb_inodes+0x234/0x4a8 >>>>   __writeback_inodes_wb+0x58/0x118 >>>>   wb_writeback+0x2f8/0x3c0 >>>>   wb_workfn+0x2c4/0x508 >>>>   process_one_work+0x180/0x408 >>>>   worker_thread+0x258/0x368 >>>>   kthread+0x118/0x128 >>>>   ret_from_fork+0x10/0x20 >>>> >>>> INFO: task B:1902 blocked for more than 120 seconds. >>>>        Tainted: G           O       6.12.41-g3fe07ddf05ab #1 >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>> message. >>>> task:B     state:D stack:0     pid:1902  tgid:1626 ppid:1153 >>>> flags:0x0000020c >>>> Call trace: >>>>   __switch_to+0xf4/0x158 >>>>   __schedule+0x27c/0x908 >>>>   rt_mutex_schedule+0x30/0x60 >>>>   __rt_mutex_slowlock_locked.constprop.0+0x460/0x8a8 >>>>   rwbase_write_lock+0x24c/0x378 >>>>   down_write+0x1c/0x30 >>>>   f2fs_balance_fs+0x184/0x208 >>>>   f2fs_map_blocks+0x94c/0x1110 >>>>   f2fs_file_write_iter+0x228/0xb80 >>>>   do_iter_readv_writev+0xf0/0x1e0 >>>>   vfs_writev+0x138/0x2c8 >>>>   do_writev+0x88/0x130 >>>>   __arm64_sys_writev+0x28/0x40 >>>>   invoke_syscall+0x50/0x120 >>>>   el0_svc_common.constprop.0+0xc8/0xf0 >>>>   do_el0_svc+0x24/0x38 >>>>   el0_svc+0x30/0xf8 >>>>   el0t_64_sync_handler+0x120/0x130 >>>>   el0t_64_sync+0x190/0x198 >>>> >>>> INFO: task sync:2769849 blocked for more than 120 seconds. >>>>        Tainted: G           O       6.12.41-g3fe07ddf05ab #1 >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>> message. >>>> task:sync            state:D stack:0     pid:2769849 tgid:2769849 >>>> ppid:736    flags:0x0000020c >>>> Call trace: >>>>   __switch_to+0xf4/0x158 >>>>   __schedule+0x27c/0x908 >>>>   schedule+0x3c/0x118 >>>>   wb_wait_for_completion+0xb0/0xe8 >>>>   sync_inodes_sb+0xc8/0x2b0 >>>>   sync_inodes_one_sb+0x24/0x38 >>>>   iterate_supers+0xa8/0x138 >>>>   ksys_sync+0x54/0xc8 >>>>   __arm64_sys_sync+0x18/0x30 >>>>   invoke_syscall+0x50/0x120 >>>>   el0_svc_common.constprop.0+0xc8/0xf0 >>>>   do_el0_svc+0x24/0x38 >>>>   el0_svc+0x30/0xf8 >>>>   el0t_64_sync_handler+0x120/0x130 >>>>   el0t_64_sync+0x190/0x198 >>>> >>>> The root cause is a potential deadlock between the following tasks: >>>> >>>> kworker/u8:11                Thread A >>>> - f2fs_write_single_data_page >>>>   - f2fs_do_write_data_page >>>>    - folio_start_writeback(X) >>>>    - f2fs_outplace_write_data >>>>     - bio_add_folio(X) >>>>   - folio_unlock(X) >>>>                     - truncate_inode_pages_range >>>>                      - __filemap_get_folio(X, FGP_LOCK) >>>>                      - truncate_inode_partial_folio(X) >>>>                       - folio_wait_writeback(X) >>>>   - f2fs_balance_fs >>>>    - f2fs_gc >>>>     - do_garbage_collect >>>>      - move_data_page >>>>       - f2fs_get_lock_data_page >>>>        - __filemap_get_folio(X, FGP_LOCK) >>>> >>>> Both threads try to access folio X. Thread A holds the lock but waits >>>> for writeback, while kworker waits for the lock. This causes a >>>> deadlock. >>>> >>>> Other threads also enter D state, waiting for locks such as gc_lock >>>> and >>>> writepages. >>>> >>>> OPU/IPU DATA folio are all affected by this issue. To avoid such >>>> potential deadlocks, always commit these cached folios before >>>> triggering f2fs_gc() in f2fs_balance_fs(). >>>> >>>> v2: >>>> - Commit cached OPU/IPU folios, not just OPU folios as in v1. >>>> >>>> Suggested-by: Chao >>>> Signed-off-by: Ruipeng Qi >>>> --- >>>>   fs/f2fs/data.c    | 26 ++++++++++++++++++++++++++ >>>>   fs/f2fs/f2fs.h    |  1 + >>>>   fs/f2fs/segment.c |  9 +++++++++ >>>>   3 files changed, 36 insertions(+) >>>> >>>> diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c >>>> index 338df7a2aea6..fd03366b3228 100644 >>>> --- a/fs/f2fs/data.c >>>> +++ b/fs/f2fs/data.c >>>> @@ -939,6 +939,32 @@ void f2fs_submit_merged_ipu_write(struct >>>> f2fs_sb_info *sbi, >>>>       } >>>>   } >>>>   +void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi) >>>> +{ >>>> +    struct bio_entry *be, *tmp; >>>> +    struct f2fs_bio_info *io; >>>> +    enum temp_type temp; >>>> +    LIST_HEAD(list); >>>> + >>>> +    for (temp = HOT; temp < NR_TEMP_TYPE; temp++) { >>>> +        io = sbi->write_io[DATA] + temp; >>>> + >>>> +        if (list_empty(&io->bio_list)) >>>> +            continue; >>> >>> Needs to be covered w/ bio_list_lock to avoid race condition. >> >> Hi,Chao >> >> The lockless list_empty() here is intentional and acceptable. >> >> >> If list_empty() returns true but the list becomes non-empty >> afterwards (due to race), the newly added bio will be submitted >> by the subsequent write path, so no bio will be lost. > > Ah, right, we only need to submit the folios cached by local thread. > >> >> >> Similar patterns exist in the kernel, e.g.: >>    net/rfkill/core.c: rfkill_fop_read() >>      /* since we re-check and it just compares pointers, >>       * using !list_empty() without locking isn't a problem >>       */ >>    fs/f2fs/data.c: f2fs_submit_merged_ipu_write() >>      list_empty() is also used without holding bio_list_lock >>      as a lockless pre-check >> >> >> If you'd prefer, we can add a comment to make the intent clear: >> >>      /* list_empty() without lock is safe here - READ_ONCE() >>       * ensures pointer read atomicity. A false negative is >>       * acceptable since any bio added concurrently will be >>       * submitted by the next write path. >>       */ >>      if (list_empty(&io->bio_list)) >>          continue; >>> >>>> + >>>> +        f2fs_down_write(&io->bio_list_lock); >>>> +        list_splice_init(&io->bio_list, &list); >>>> +        f2fs_up_write(&io->bio_list_lock); >>>> + >>>> +        list_for_each_entry_safe(be, tmp, &list, list) { >>>> +            f2fs_submit_write_bio(sbi, be->bio, DATA); >>>> +            del_bio_entry(be); >>>> +        } >>>> + >>> >>> Unnecessary blank line. >>> >>> Thanks, >> >> Thanks for your correction. Will fix in v3. >>      v3: >>      - Fixed minor grammatical issues >>      - Add comment on lockless list_empty() to explain why it is safe >>    without holding bio_list_lock > > Seems fine. > >> >> >> Thanks, >> >>> >>>> +    } >>>> + >>>> +} >>>> + >>>>   int f2fs_merge_page_bio(struct f2fs_io_info *fio) >>>>   { >>>>       struct bio *bio = *fio->bio; >>>> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h >>>> index bb34e864d0ef..e9038ab1b2bd 100644 >>>> --- a/fs/f2fs/f2fs.h >>>> +++ b/fs/f2fs/f2fs.h >>>> @@ -4148,6 +4148,7 @@ void f2fs_submit_merged_write_folio(struct >>>> f2fs_sb_info *sbi, >>>>                   struct folio *folio, enum page_type type); >>>>   void f2fs_submit_merged_ipu_write(struct f2fs_sb_info *sbi, >>>>                       struct bio **bio, struct folio *folio); >>>> +void f2fs_submit_all_merged_ipu_writes(struct f2fs_sb_info *sbi); >>>>   void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi); >>>>   int f2fs_submit_page_bio(struct f2fs_io_info *fio); >>>>   int f2fs_merge_page_bio(struct f2fs_io_info *fio); >>>> diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c >>>> index 6a97fe76712b..856ffe91b94f 100644 >>>> --- a/fs/f2fs/segment.c >>>> +++ b/fs/f2fs/segment.c >>>> @@ -454,6 +454,15 @@ void f2fs_balance_fs(struct f2fs_sb_info *sbi, >>>> bool need) >>>>           io_schedule(); >>>>           finish_wait(&sbi->gc_thread->fggc_wq, &wait); >>>>       } else { >>>> + >>>> +        /* >>>> +         * Submit all cached OPU/IPU DATA bios before triggering >>>> +         * foreground GC to avoid potential deadlocks. >>>> +         */ >>>> + >>>> +        f2fs_submit_merged_write(sbi, DATA); >>>> +        f2fs_submit_all_merged_ipu_writes(sbi); > > Can we relocate above code to below the variable definitions? > > Thanks, > Hi, Chao Sure, will fix it in V3. BTW, To avoid potential deadlocks, this patch submits cached OPU/IPU folios before triggering f2fs_gc() in f2fs_balance_fs(), which changes the existing IPU/OPU BIO lifecycle. For OPU, io->io_rwsem provides the necessary synchronization. For IPU, io->bio_list_lock ensures race-free submission. In both cases, new BIOs will be allocated as needed after submission. I may have missed something in the current implementation. Your professional review would be much appreciated. Thanks, >>>> + >>>>           struct f2fs_gc_control gc_control = { >>>>               .victim_segno = NULL_SEGNO, >>>>               .init_gc_type = f2fs_sb_has_blkzoned(sbi) ? >>> >