* [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC
@ 2026-11-16 2:27 Jose Fernandez (Anthropic)
2026-11-17 11:37 ` Zhang Yi
0 siblings, 1 reply; 3+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-11-16 2:27 UTC (permalink / raw)
To: Theodore Ts'o, Zhang Yi
Cc: Jan Kara, Baokun Li, Ojaswin Mujoo, Andreas Dilger,
Ritesh Harjani (IBM),
guolingxing, Josef Bacik, Chris Mason, linux-ext4, linux-kernel,
regressions
Ojaswin's cover letter [1] explains how, since commit ea96cb5c4ae3
("ext4: don't split extent before submitting I/O"), the split of an
unwritten extent at I/O completion can get ENOSPC. [1] stops the zeroout
from hitting other writes in flight, and [2] reports the failed
conversion to fsync(). Neither gives the split its block, so data is
still lost.
LLM agents working with me found that it happens with default options
on a filesystem that is not full, and ran the tests. write() and fsync()
return success, and after the pages are evicted part of the file reads
back as zeros.
Under a load built to run the allocator dry (QEMU without KVM), 42 of
72 files lost data at ea96cb5c4ae3, none of 72 at its parent and none
of 48 on v6.18.63. On mainline at 23c62304dde2 32 of 72 did. With [1]
rebased on it 42 of 72 still did, now as failed conversions in dmesg.
With [1] and [2] fsync() failed with ENOSPC on 47 of 72, and the other
25 read back intact.
Nothing our agents have tried fixes it fully. I'm reporting this so you
are aware of the regression since v7.0. A reproducer is available to the
maintainers off-list.
[1] https://lore.kernel.org/linux-ext4/20261102-ext4-endio-zero-bug-v1-0-266d2bb9e6dc@linux.ibm.com/
[2] https://lore.kernel.org/linux-ext4/20261101102145.647496-1-guolingxing@tju.edu.cn/
#regzbot introduced: ea96cb5c4ae3
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC
2026-11-16 2:27 [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC Jose Fernandez (Anthropic)
@ 2026-11-17 11:37 ` Zhang Yi
2026-11-17 14:23 ` Jose Fernandez (Anthropic)
0 siblings, 1 reply; 3+ messages in thread
From: Zhang Yi @ 2026-11-17 11:37 UTC (permalink / raw)
To: Jose Fernandez (Anthropic), Theodore Ts'o
Cc: Jan Kara, Baokun Li, Ojaswin Mujoo, Andreas Dilger,
Ritesh Harjani (IBM),
guolingxing, Josef Bacik, Chris Mason, linux-ext4, linux-kernel,
regressions
On 11/16/2026 10:27 AM, Jose Fernandez (Anthropic) wrote:
> Ojaswin's cover letter [1] explains how, since commit ea96cb5c4ae3
> ("ext4: don't split extent before submitting I/O"), the split of an
> unwritten extent at I/O completion can get ENOSPC. [1] stops the zeroout
> from hitting other writes in flight, and [2] reports the failed
> conversion to fsync(). Neither gives the split its block, so data is
> still lost.
>
> LLM agents working with me found that it happens with default options
> on a filesystem that is not full, and ran the tests. write() and fsync()
> return success, and after the pages are evicted part of the file reads
> back as zeros.
>
> Under a load built to run the allocator dry (QEMU without KVM), 42 of
> 72 files lost data at ea96cb5c4ae3, none of 72 at its parent and none
> of 48 on v6.18.63. On mainline at 23c62304dde2 32 of 72 did. With [1]
> rebased on it 42 of 72 still did, now as failed conversions in dmesg.
> With [1] and [2] fsync() failed with ENOSPC on 47 of 72, and the other
> 25 read back intact.
>
Hi Jose!
I'm sorry for causing this regression. The cause of this problem is that
the transaction freeing the file's space has not yet been committed and
thus cannot be returned to the mb allocator, so the extent cannot be
split at endio time to allocate space for metadata. This is easily
reproduced in scenarios with frequent file creation and deletion, even
though there is still plenty of free space on disk.
I think the solution should be based on Ojaswin's counting approach in
[1] or the PENDING_CONVERSION solution we discussed, and then, after
extent splitting fails, try to flush the journal and retry. My idea is
roughly as follows:
1. For the iomap buffered I/O path and the DIO path, this is relatively
simple. We just need to add journal flush and retry logic in
ext4_iomap_finish_ioend() and ext4_dio_write_end_io().
2. For the traditional bh buffered I/O path, we cannot flush the journal
in ext4_convert_unwritten_io_end_vec(), because this may lead to an
ABBA deadlock with data=ordered mode. The journal process needs to
wait for ordered folio writeback, but this folio may contain some
ranges that need to be converted, and if we flush the journal after
the conversion fails, it will form a mutual dependency with journal
flushing. Therefore, we have to fall back to splitting before issuing
I/O, and when splitting fails, flush the journal and retry via
jbd2_journal_force_commit_nested() in ext4_do_writepages().
3. Split the workqueues used at the end of iomap and bh writeback to
avoid deadlocks. Because the filesystem may simultaneously have
different inodes going through the bh path and the iomap path, if the
iomap completion worker is flushing the journal, it may need to wait
for the folio of another inode going through the bh path to finish
conversion, and that conversion worker still needs to be handled on
the same wq, thus causing a deadlock.
The current bh buffered I/O handling flow cannot be unified with the
other two, which is indeed not ideal. However, once data=ordered mode is
sunset in the future, we should be able to unify them. What do you all
think of this solution? Any better ideas?
Thanks,
Yi.
The specific POC fix code is as follows. I tested it with the reproducer
and the issue no longer reproduces.
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 8069521442a7..441356999594 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1761,6 +1761,7 @@ struct ext4_sb_info {
/* workqueue for reserved extent conversions (buffered io) */
struct workqueue_struct *rsv_conversion_wq;
+ struct workqueue_struct *iomap_conversion_wq;
/* timer for periodic error stats printing */
struct timer_list s_err_report;
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 91857b46ce24..82fcb71330dc 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -3423,10 +3423,29 @@ static int ext4_split_extent_zeroout(handle_t *handle, struct inode *inode,
if (err)
/* ZEROOUT failed, just return original error */
goto out_unlock;
+ } else if (flags & EXT4_GET_BLOCKS_SPLIT_NOMERGE) {
+ /*
+ * The caller wants to split the extent before submitting
+ * the I/O, but the split failed. Zeroout the whole extent
+ * and mark it initialized as the last resort, the upcoming
+ * I/O will overwrite the map range.
+ */
+ if (!is_unwrit) {
+ /* Shouldn't happen. Just exit */
+ err = -EINVAL;
+ goto out_unlock;
+ }
+
+ lblk = ee_block;
+ len = ee_len;
+ pblk = ext4_ext_pblock(ex);
+ err = ext4_issue_zeroout(inode, lblk, pblk, len);
+ if (err)
+ /* ZEROOUT failed, just return original error */
+ goto out_unlock;
} else {
/*
- * We no longer perform unwritten to unwritten splits in IO paths.
- * Hence this should not happen.
+ * This should not happen.
*/
WARN_ON_ONCE(true);
err = -EINVAL;
@@ -4073,6 +4092,36 @@ ext4_ext_handle_unwritten_extents(handle_t *handle, struct inode *inode,
trace_ext4_ext_handle_unwritten_extents(inode, map, flags,
*allocated, newblock);
+ /* get_block() before submitting IO, split the extent */
+ if (flags & EXT4_GET_BLOCKS_SPLIT_NOMERGE) {
+ int depth;
+
+ path = ext4_split_convert_extents(handle, inode, map, path,
+ flags, allocated);
+ if (IS_ERR(path))
+ return path;
+
+ /*
+ * shouldn't get a 0 allocated when splitting an extent unless
+ * m_len is 0 (bug) or extent has been corrupted
+ */
+ if (unlikely(*allocated == 0)) {
+ EXT4_ERROR_INODE(inode,
+ "unexpected allocated == 0, m_len = %u",
+ map->m_len);
+ err = -EFSCORRUPTED;
+ goto errout;
+ }
+ /* Don't mark unwritten if the extent has been zeroed out. */
+ path = ext4_find_extent(inode, map->m_lblk, path, flags);
+ if (IS_ERR(path))
+ return path;
+ depth = ext_depth(inode);
+ if (ext4_ext_is_unwritten(path[depth].p_ext))
+ map->m_flags |= EXT4_MAP_UNWRITTEN;
+ goto split_out;
+ }
+
/* IO end_io complete, convert the filled extent to written */
if (flags & EXT4_GET_BLOCKS_CONVERT) {
path = ext4_convert_unwritten_extents_endio(handle, inode,
@@ -4126,7 +4175,7 @@ ext4_ext_handle_unwritten_extents(handle_t *handle, struct inode *inode,
err = -EFSCORRUPTED;
goto errout;
}
-
+split_out:
map->m_flags |= EXT4_MAP_NEW;
map_out:
map->m_flags |= EXT4_MAP_MAPPED;
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index 52b1fb99484d..96049dc54963 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -450,14 +450,30 @@ static int ext4_dio_write_end_io(struct kiocb *iocb, ssize_t size,
loff_t pos = iocb->ki_pos;
struct inode *inode = file_inode(iocb->ki_filp);
+ if (!error && size && (flags & IOMAP_DIO_UNWRITTEN)) {
+ int retries = 0;
- if (!error && size && (flags & IOMAP_DIO_UNWRITTEN) &&
- (iocb->ki_flags & IOCB_ATOMIC))
- error = ext4_convert_unwritten_extents_atomic(NULL, inode, pos,
- size);
- else if (!error && size && flags & IOMAP_DIO_UNWRITTEN)
- error = ext4_convert_unwritten_extents(NULL, inode, pos, size,
- NULL, 0);
+retry_convert:
+ if (iocb->ki_flags & IOCB_ATOMIC)
+ error = ext4_convert_unwritten_extents_atomic(NULL,
+ inode, pos, size);
+ else
+ error = ext4_convert_unwritten_extents(NULL, inode,
+ pos, size, NULL, 0);
+
+ /*
+ * The conversion may have failed to allocate metadata for
+ * splitting the extent. Blocks freed by other operations
+ * (e.g. an unlink) may still be waiting for a journal
+ * commit, so commit and retry the conversion.
+ */
+ if (error == -ENOSPC && EXT4_SB(inode->i_sb)->s_journal &&
+ ext4_count_free_clusters(inode->i_sb) &&
+ ++retries <= 3) {
+ if (ext4_force_commit(inode->i_sb) == 0)
+ goto retry_convert;
+ }
+ }
/*
* If this dio covered an unwritten extent, we would have incremented
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 13449e4ebb6a..03d296f54b6d 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -991,7 +991,8 @@ int ext4_map_blocks(handle_t *handle, struct inode *inode,
goto out;
if (map->m_flags & EXT4_MAP_UNWRITTEN &&
(flags & EXT4_GET_BLOCKS_UNWRIT_EXT) &&
- !(flags & EXT4_GET_BLOCKS_CONVERT)) {
+ !(flags & (EXT4_GET_BLOCKS_CONVERT |
+ EXT4_GET_BLOCKS_SPLIT_NOMERGE))) {
/* Contains EXT4_GET_BLOCKS_CREATE - mark mapped. */
ret = check_block_validity(inode, map);
if (!ret)
@@ -2600,8 +2601,17 @@ static int mpage_map_one_extent(handle_t *handle, struct mpage_da_data *mpd)
EXT4_EX_NOCACHE;
dioread_nolock = ext4_should_dioread_nolock(inode);
- if (dioread_nolock)
- get_blocks_flags |= EXT4_GET_BLOCKS_UNWRIT_EXT;
+ if (dioread_nolock) {
+ get_blocks_flags |= EXT4_GET_BLOCKS_SPLIT_NOMERGE |
+ EXT4_GET_BLOCKS_UNWRIT_EXT;
+ /*
+ * The io_end is accounted in i_conv_pending_ios only after
+ * the first unwritten extent has been mapped, so tell the
+ * zeroout fallback whether our own count is already included.
+ */
+ if (!(mpd->io_submit.io_end->flag & EXT4_IO_END_UNWRITTEN))
+ get_blocks_flags |= EXT4_GET_BLOCKS_NO_CONV_ACC;
+ }
err = ext4_map_blocks(handle, inode, map, get_blocks_flags);
if (err < 0)
diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c
index e99816017a18..ceeb15ae9b40 100644
--- a/fs/ext4/page-io.c
+++ b/fs/ext4/page-io.c
@@ -649,7 +649,7 @@ static void ext4_iomap_finish_ioend(struct iomap_ioend *ioend)
unsigned long io_mode = (unsigned long)ioend->io_private;
bool is_disksize_grow = (io_mode == EXT4_IOMAP_IOEND_DISKSIZE_GROW_IO);
handle_t *handle;
- int credits;
+ int credits, retries = 0;
int ret, err;
/*
@@ -684,6 +684,7 @@ static void ext4_iomap_finish_ioend(struct iomap_ioend *ioend)
/* Wait for disksize-pending zeroed data to be written out. */
ext4_iomap_wb_disksize_pending_wait(inode, pos, size);
+retry:
/*
* We may need to convert one extent, update the i_disksize and
* dirty the inode.
@@ -702,6 +703,8 @@ static void ext4_iomap_finish_ioend(struct iomap_ioend *ioend)
is_disksize_grow);
if (ret)
goto out_journal;
+
+ is_disksize_grow = false;
}
if (ioend->io_flags & IOMAP_IOEND_UNWRITTEN)
@@ -714,6 +717,12 @@ static void ext4_iomap_finish_ioend(struct iomap_ioend *ioend)
ret = err;
out_err:
if (ret < 0 && !ext4_emergency_state(sb)) {
+ if (ret == -ENOSPC && EXT4_SB(sb)->s_journal &&
+ ext4_count_free_clusters(sb) && ++retries <= 3) {
+ if (ext4_force_commit(sb) == 0)
+ goto retry;
+ }
+
ext4_msg(sb, KERN_EMERG,
"failed to convert unwritten extents to written extents or update inode size -- potential data loss! (inode %llu, error %d)",
inode->i_ino, ret);
@@ -768,7 +777,7 @@ void ext4_iomap_end_bio(struct bio *bio)
spin_lock_irqsave(&ei->i_completed_io_lock, flags);
if (list_empty(&ei->i_rsv_conversion_list))
- queue_work(EXT4_SB(ioend->io_inode->i_sb)->rsv_conversion_wq,
+ queue_work(EXT4_SB(ioend->io_inode->i_sb)->iomap_conversion_wq,
&ei->i_rsv_conversion_work);
list_add_tail(&ioend->io_list, &ei->i_rsv_conversion_list);
spin_unlock_irqrestore(&ei->i_completed_io_lock, flags);
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 3c53bf27da45..9012fce8dd9a 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -1326,6 +1326,7 @@ static void ext4_put_super(struct super_block *sb)
flush_delayed_work(&sbi->s_ea_inode_work);
ext4_quotas_off(sb, EXT4_MAXQUOTAS);
+ destroy_workqueue(sbi->iomap_conversion_wq);
destroy_workqueue(sbi->rsv_conversion_wq);
ext4_release_orphan_info(sb);
@@ -5668,6 +5669,24 @@ static int __ext4_fill_super(struct fs_context *fc, struct super_block *sb)
goto failed_mount4;
}
+ /*
+ * Buffered iomap conversions may have to force a journal commit and
+ * retry, so they cannot share the single-threaded rsv_conversion_wq
+ * with the buffer_head path. That commit may wait for the writeback
+ * of a buffer_head io_end which is completed on rsv_conversion_wq.
+ * Allow some concurrency here as a retry can block on the commit.
+ */
+ EXT4_SB(sb)->iomap_conversion_wq =
+ alloc_workqueue("ext4-iomap-conversion",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+ if (!EXT4_SB(sb)->iomap_conversion_wq) {
+ printk(KERN_ERR "EXT4-fs: failed to create workqueue\n");
+ err = -ENOMEM;
+ destroy_workqueue(EXT4_SB(sb)->rsv_conversion_wq);
+ EXT4_SB(sb)->rsv_conversion_wq = NULL;
+ goto failed_mount4;
+ }
+
/*
* The jbd2_journal_load will have done any necessary log recovery,
* so we can safely mount the rest of the filesystem now.
@@ -5842,6 +5861,8 @@ failed_mount8: __maybe_unused
sb->s_root = NULL;
failed_mount4:
ext4_msg(sb, KERN_ERR, "mount failed");
+ if (EXT4_SB(sb)->iomap_conversion_wq)
+ destroy_workqueue(EXT4_SB(sb)->iomap_conversion_wq);
if (EXT4_SB(sb)->rsv_conversion_wq)
destroy_workqueue(EXT4_SB(sb)->rsv_conversion_wq);
failed_mount_wq:
@@ -6543,6 +6564,7 @@ static int ext4_sync_fs(struct super_block *sb, int wait)
return ret;
trace_ext4_sync_fs(sb, wait);
+ flush_workqueue(sbi->iomap_conversion_wq);
flush_workqueue(sbi->rsv_conversion_wq);
flush_delayed_work(&sbi->s_ea_inode_work);
/*
> Nothing our agents have tried fixes it fully. I'm reporting this so you
> are aware of the regression since v7.0. A reproducer is available to the
> maintainers off-list.
>
> [1] https://lore.kernel.org/linux-ext4/20261102-ext4-endio-zero-bug-v1-0-266d2bb9e6dc@linux.ibm.com/
> [2] https://lore.kernel.org/linux-ext4/20261101102145.647496-1-guolingxing@tju.edu.cn/
>
> #regzbot introduced: ea96cb5c4ae3
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC
2026-11-17 11:37 ` Zhang Yi
@ 2026-11-17 14:23 ` Jose Fernandez (Anthropic)
0 siblings, 0 replies; 3+ messages in thread
From: Jose Fernandez (Anthropic) @ 2026-11-17 14:23 UTC (permalink / raw)
To: Zhang Yi, Theodore Ts'o
Cc: Jan Kara, Baokun Li, Ojaswin Mujoo, Andreas Dilger,
Ritesh Harjani (IBM),
guolingxing, Josef Bacik, Chris Mason, linux-ext4, linux-kernel,
regressions
On Tue, Nov 17, 2026 Zhang Yi wrote:
> I think the solution should be based on Ojaswin's counting approach in
> [1] or the PENDING_CONVERSION solution we discussed, and then, after
> extent splitting fails, try to flush the journal and retry.
Thanks for the quick POC. Our agents ran it on [1] rebased on
23c62304dde2 with the report's reproducer, in two separate setups (QEMU
without KVM). None of 72 files lost data in either, against 42 and 53 of
72 with [1] alone.
They found two gaps:
- The new branch in ext4_ext_handle_unwritten_extents() skips both
ext4_update_inode_fsync_trans() calls, so fdatasync() and O_DSYNC can
return before the commit. On a full filesystem mounted with
commit=3600, a shutdown test lost the write in all 13 buffered
fdatasync() and O_DSYNC cases that reach the zeroout. They pass with
[1] alone and with that call added.
- Direct writes into preallocated files can still get ENOSPC with the
POC, on a few percent of files in both setups, with no data lost.
The POC tries the zeroout before the commit and retry. Should it be the
other way around? And will you take guolingxing's patch ([2] in my
report) as well?
- Jose
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-17 14:23 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 2:27 [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC Jose Fernandez (Anthropic)
2026-11-17 11:37 ` Zhang Yi
2026-11-17 14:23 ` Jose Fernandez (Anthropic)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®