From: Zhang Yi <yi.zhang@huaweicloud.com>
To: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Cc: "Jose Fernandez (Anthropic)" <jose.fernandez@linux.dev>,
Theodore Ts'o <tytso@mit.edu>, Jan Kara <jack@suse.cz>,
Baokun Li <libaokun@linux.alibaba.com>,
Andreas Dilger <adilger.kernel@dilger.ca>,
"Ritesh Harjani (IBM)" <ritesh.list@gmail.com>,
guolingxing <guolingxing@tju.edu.cn>,
Josef Bacik <josef@toxicpanda.com>,
Chris Mason <mason@kernel.org>,
linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org,
regressions@lists.linux.dev
Subject: Re: [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC
Date: Wed, 18 Nov 2026 18:48:50 +0800 [thread overview]
Message-ID: <1102a630-de07-4494-aea2-207e5b84e2a0@huaweicloud.com> (raw)
In-Reply-To: <av1Z_RLuH4vd0Wyi@li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com>
On 11/18/2026 3:55 PM, Ojaswin Mujoo wrote:
> On Tue, Nov 17, 2026 at 07:37:10PM +0800, Zhang Yi wrote:
>> On 11/16/2026 10:27 AM, Jose Fernandez (Anthropic) wrote:
>>> Ojaswin's cover letter [1] explains how, since commit ea96cb5c4ae3
>>> ("ext4: don't split extent before submitting I/O"), the split of an
>>> unwritten extent at I/O completion can get ENOSPC. [1] stops the zeroout
>>> from hitting other writes in flight, and [2] reports the failed
>>> conversion to fsync(). Neither gives the split its block, so data is
>>> still lost.
>>>
>>> LLM agents working with me found that it happens with default options
>>> on a filesystem that is not full, and ran the tests. write() and fsync()
>>> return success, and after the pages are evicted part of the file reads
>>> back as zeros.
>>>
>>> Under a load built to run the allocator dry (QEMU without KVM), 42 of
>>> 72 files lost data at ea96cb5c4ae3, none of 72 at its parent and none
>>> of 48 on v6.18.63. On mainline at 23c62304dde2 32 of 72 did. With [1]
>>> rebased on it 42 of 72 still did, now as failed conversions in dmesg.
>>> With [1] and [2] fsync() failed with ENOSPC on 47 of 72, and the other
>>> 25 read back intact.
>>>
>>
>> Hi Jose!
>>
>> I'm sorry for causing this regression. The cause of this problem is that
>> the transaction freeing the file's space has not yet been committed and
>> thus cannot be returned to the mb allocator, so the extent cannot be
>> split at endio time to allocate space for metadata. This is easily
>> reproduced in scenarios with frequent file creation and deletion, even
>> though there is still plenty of free space on disk.
>>
>> I think the solution should be based on Ojaswin's counting approach in
>> [1] or the PENDING_CONVERSION solution we discussed, and then, after
>> extent splitting fails, try to flush the journal and retry. My idea is
>> roughly as follows:
>>
>> 1. For the iomap buffered I/O path and the DIO path, this is relatively
>> simple. We just need to add journal flush and retry logic in
>> ext4_iomap_finish_ioend() and ext4_dio_write_end_io().
>>
>> 2. For the traditional bh buffered I/O path, we cannot flush the journal
>> in ext4_convert_unwritten_io_end_vec(), because this may lead to an
>> ABBA deadlock with data=ordered mode. The journal process needs to
>> wait for ordered folio writeback, but this folio may contain some
>> ranges that need to be converted, and if we flush the journal after
>> the conversion fails, it will form a mutual dependency with journal
>> flushing. Therefore, we have to fall back to splitting before issuing
>> I/O, and when splitting fails, flush the journal and retry via
>> jbd2_journal_force_commit_nested() in ext4_do_writepages().
>>
>> 3. Split the workqueues used at the end of iomap and bh writeback to
>> avoid deadlocks. Because the filesystem may simultaneously have
>> different inodes going through the bh path and the iomap path, if the
>> iomap completion worker is flushing the journal, it may need to wait
>> for the folio of another inode going through the bh path to finish
>> conversion, and that conversion worker still needs to be handled on
>> the same wq, thus causing a deadlock.
>>
>> The current bh buffered I/O handling flow cannot be unified with the
>> other two, which is indeed not ideal. However, once data=ordered mode is
>> sunset in the future, we should be able to unify them. What do you all
>> think of this solution? Any better ideas?
>>
>> Thanks,
>> Yi.
>
> Hey Zhang, thanks for the poc, it looks largely okay. I agree that this
> seems to be the right way forward although its unfortunate to have to
> revert back the unwrit to unwrit split. Hopefully we can move completely
> switch to iomap soon enough.
>
> On the extent status cache for zeroout approach, i'm currently fixing a
> few bs < ps bugs I'm seeing in it and would clean it up and post it as
> v2. My idea is to keep the first patch disabling zeroout and rest of
> them implementing the es changes and renabling it.
Hi Ojaswin,
Yeah, this makes sense to me. Looking forward to your v2. I'll re-adapt
this POC based on your patch afterwards. :)
> We can just backport
> the first patch. Along with your poc fixes here also backported to 7.0
> i believe we should be good to go.
>
Sorry, I don't quite understand. As far as I know, 7.0 is already EOL, so
why does this still need to be backported to this version?
Thanks,
Yi.
prev parent reply other threads:[~2026-11-18 10:49 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 2:27 Jose Fernandez (Anthropic)
2026-11-17 11:37 ` Zhang Yi
2026-11-17 14:23 ` Jose Fernandez (Anthropic)
2026-11-18 3:44 ` Zhang Yi
2026-11-18 14:06 ` Jose Fernandez (Anthropic)
2026-11-18 7:55 ` Ojaswin Mujoo
2026-11-18 10:48 ` Zhang Yi [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1102a630-de07-4494-aea2-207e5b84e2a0@huaweicloud.com \
--to=yi.zhang@huaweicloud.com \
--cc=adilger.kernel@dilger.ca \
--cc=guolingxing@tju.edu.cn \
--cc=jack@suse.cz \
--cc=jose.fernandez@linux.dev \
--cc=josef@toxicpanda.com \
--cc=libaokun@linux.alibaba.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mason@kernel.org \
--cc=ojaswin@linux.ibm.com \
--cc=regressions@lists.linux.dev \
--cc=ritesh.list@gmail.com \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®