[PATCH RESEND 0/9] f2fs: fix stale data exposure on failed O_DIRECT writes

[PATCH RESEND 0/9] f2fs: fix stale data exposure on failed O_DIRECT writes

[Jan Prusakowski]

Resending to include Jaegeuk Kim and the mailing lists
(linux-f2fs-devel and linux-kernel), which were accidentally omitted
from the initial submission. No changes to the patches.

When an O_DIRECT write fails with an I/O error, f2fs can leave the file
pointing to blocks that were never written, so reads return whatever the
disk contained before, possibly data of a deleted file. xfstests
generic/250 and generic/252 check this for direct writes to fallocated
blocks under dm-error, and both fail on f2fs. This series fixes them,
along with several error handling issues found on the way.

Patches 1-2 fix issues in compressed file paths hit by these tests:
sleeping in atomic context when reporting a decompression error, and a
read error in prepare_compress_overwrite() stopping the checkpoint
instead of failing the write.

Patch 3 keeps the old block address in the dnode when the block
allocation of an out-of-place write fails, instead of replacing it with
NULL_ADDR.

Patch 4 makes the buffered fallback of O_DIRECT writes return errors,
following direct_write_fallback(); generic/299 hits one of them.

Patch 5 truncates blocks allocated past EOF by a failed direct write.

Patch 6 tracks out-of-place block updates while an O_DIRECT write falls
back to buffered I/O, and rolls back those whose write failed, restoring
the old blocks or resetting the blocks to NEW_ADDR.

Patch 7 records blocks allocated during a native direct write (for holes,
preallocated NEW_ADDR ranges, and out-of-place overwrites in LFS mode)
per kiocb, and rolls back any unwritten blocks if the direct write fails
or stops short. Overlapping reads and writes wait for the in-flight
allocating direct write to complete, while non-overlapping async direct
writes proceed in parallel. This fixes generic/250 and generic/252
without falling back to buffered I/O.

Patches 8-9 fix compressed files (where O_DIRECT writes fall back to
buffered I/O): patch 8 cleans up when a compressed cluster write fails
to be submitted, and patch 9 extends the buffered fallback rollback to
compressed cluster writes.

Tested with full xfstests (-g auto) across default, compress and encrypt
configurations with CONFIG_PROVE_LOCKING and CONFIG_DEBUG_KMEMLEAK
enabled.

Assisted-by: LLM

Jan Prusakowski (9):
  f2fs: compress: avoid sleeping in atomic context in
    f2fs_decompress_cluster()
  f2fs: compress: abort on read error in prepare_compress_overwrite()
  f2fs: keep old block address when out-of-place write fails to allocate
  f2fs: propagate errors from buffered fallback of O_DIRECT writes
  f2fs: truncate blocks allocated past EOF by a failed direct write
  f2fs: roll back failed block updates of O_DIRECT buffered fallback
  f2fs: roll back failed block updates of direct writes
  f2fs: compress: fix cleanup when a cluster write fails to be submitted
  f2fs: compress: roll back failed cluster writes of O_DIRECT buffered
    fallback

 fs/f2fs/compress.c | 179 ++++++++++-
 fs/f2fs/data.c     | 185 ++++++++++-
 fs/f2fs/f2fs.h     |  99 ++++++
 fs/f2fs/file.c     | 745 +++++++++++++++++++++++++++++++++++++++++++--
 fs/f2fs/segment.c  |  49 ++-
 fs/f2fs/super.c    |  11 +
 6 files changed, 1217 insertions(+), 51 deletions(-)


base-commit: 9b12299b437e0cc3ebf402e7b36a89cc69151238
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 1/9] f2fs: compress: avoid sleeping in atomic context in f2fs_decompress_cluster()

[Jan Prusakowski]

f2fs_decompress_cluster() can be invoked directly from bio completion
(softirq/IRQ context) when in_task is false. Passing GFP_NOFS to
fserror_report_file_metadata() in that context can sleep and trigger a
scheduling-while-atomic bug.

Use GFP_NOWAIT when !in_task, matching f2fs_sanity_check_node_footer().

Fixes: ccc6436abb84 ("f2fs: support to report fserror")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/compress.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c
index c7910688c318..9453254ff522 100644
--- a/fs/f2fs/compress.c
+++ b/fs/f2fs/compress.c
@@ -763,7 +763,8 @@ void f2fs_decompress_cluster(struct decompress_io_ctx *dic, bool in_task)
 
 		/* Avoid f2fs_commit_super in irq context */
 		f2fs_handle_error(sbi, ERROR_FAIL_DECOMPRESSION);
-		fserror_report_file_metadata(dic->inode, ret, GFP_NOFS);
+		fserror_report_file_metadata(dic->inode, ret,
+					     in_task ? GFP_NOFS : GFP_NOWAIT);
 		goto out_release;
 	}
 
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 2/9] f2fs: compress: abort on read error in prepare_compress_overwrite()

[Jan Prusakowski]

prepare_compress_overwrite() reads the existing pages of a compressed
cluster before they are partially overwritten. If a folio is still not
uptodate after the read, the function releases everything and restarts
from the beginning.

When the read failed because of a persistent device I/O error (e.g.
dm-error in generic/250 and generic/252), every retry fails the same
way. Commit 3147ee567dd9 ("f2fs: fix potential deadloop in
prepare_compress_overwrite()") bounded this loop with
f2fs_handle_page_eio(), so after MAX_RETRY_PAGE_EIO attempts checkpoint
is stopped with STOP_CP_REASON_READ_DATA. As a result, a read error in a
single data cluster shuts down the whole filesystem instead of failing
the write with -EIO.

Return -EIO as soon as a folio is found not uptodate. Keep calling
f2fs_handle_page_eio(), so that repeated failures on the same page
across calls (e.g. from background writeback) still stop checkpoint as
before. Both callers already handle errors: f2fs_write_begin() fails the
write, and f2fs_write_cache_pages() stops writeback of the inode and
returns the error, leaving the pages dirty.

A folio that disappeared because of truncation (filemap_lock_folio()
fails) still restarts via release_and_retry, which is now a separate
error path after the success return.

While at it, unlock the folios in cc->rpages before dropping our
references to them in both error paths, so that a folio is never
touched after its reference is released, and drop the check of ret
before the success return, which is always true at that point.

Fixes: 4c8ff7095bef ("f2fs: support data compression")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/compress.c | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c
index 9453254ff522..c5758da7c621 100644
--- a/fs/f2fs/compress.c
+++ b/fs/f2fs/compress.c
@@ -1160,23 +1160,27 @@ static int prepare_compress_overwrite(struct compress_ctx *cc,
 
 		if (!folio_test_uptodate(folio)) {
 			f2fs_handle_page_eio(sbi, folio->index, DATA);
-release_and_retry:
-			f2fs_put_rpages(cc);
+			ret = -EIO;
+			/* folio i has been added to cc->rpages as well */
 			f2fs_unlock_rpages(cc, i + 1);
-			f2fs_destroy_compress_ctx(cc, true);
-			goto retry;
+			goto put_pages;
 		}
 	}
 
-	if (likely(!ret)) {
-		*fsdata = cc->rpages;
-		*pagep = cc->rpages[offset_in_cluster(cc, index)];
-		return cc->cluster_size;
-	}
+	*fsdata = cc->rpages;
+	*pagep = cc->rpages[offset_in_cluster(cc, index)];
+	return cc->cluster_size;
 
-unlock_pages:
+release_and_retry:
+	f2fs_unlock_rpages(cc, i);
 	f2fs_put_rpages(cc);
+	f2fs_destroy_compress_ctx(cc, true);
+	goto retry;
+
+unlock_pages:
 	f2fs_unlock_rpages(cc, i);
+put_pages:
+	f2fs_put_rpages(cc);
 	f2fs_destroy_compress_ctx(cc, true);
 out:
 	return ret;
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 3/9] f2fs: keep old block address when out-of-place write fails to allocate

[Jan Prusakowski]

If f2fs_allocate_data_block() fails, do_write_block() ends writeback of
the folio without submitting it and f2fs_allocate_data_block() sets
fio->new_blkaddr to NULL_ADDR. f2fs_outplace_write_data() then still
calls f2fs_update_data_blkaddr() with that address, which replaces the
old (or reserved NEW_ADDR) block address in the dnode with NULL_ADDR.
The block stays accounted in i_blocks and valid_block_count, so the
in-memory metadata becomes inconsistent and the old data becomes
unreachable.

Only update the dnode when the write was actually submitted.

Allocation only fails after the checkpoint has been stopped with
CP_ERROR_FLAG, so this does not reach the disk, but it keeps the
in-memory state consistent for later reads and for code that rolls back
failed writes.

Fixes: 7d009e048d7c ("f2fs: fix to handle segment allocation failure correctly")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/segment.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index a5f9de723a70..a26cde2dc60e 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -4257,7 +4257,12 @@ void f2fs_outplace_write_data(struct dnode_of_data *dn,
 		f2fs_update_age_extent_cache(dn);
 	set_summary(&sum, dn->nid, dn->ofs_in_node, fio->version);
 	do_write_block(&sum, fio);
-	f2fs_update_data_blkaddr(dn, fio->new_blkaddr);
+	/*
+	 * If block allocation failed, fio->new_blkaddr is NULL_ADDR and the
+	 * folio was not submitted, so keep the old block address in the dnode.
+	 */
+	if (likely(fio->submitted))
+		f2fs_update_data_blkaddr(dn, fio->new_blkaddr);
 
 	f2fs_update_iostat(sbi, dn->inode, fio->io_type,
 					F2FS_BLKSIZE(sbi));
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 4/9] f2fs: propagate errors from buffered fallback of O_DIRECT writes

[Jan Prusakowski]

f2fs falls back to buffered I/O for O_DIRECT writes in two places:
f2fs_file_write_iter() when f2fs_should_use_dio() rejects direct I/O
(e.g. compressed files or unaligned requests), and f2fs_dio_write_iter()
for the remainder of a partial direct write. In both cases the data is
then written back and dropped from the page cache by
f2fs_flush_buffered_write() to preserve O_DIRECT semantics, but errors
are lost:

- f2fs_flush_buffered_write() ignores the result of
  filemap_write_and_wait_range(), so a write that fails at writeback
  time (e.g. on compressed files with dm-error in generic/250 and
  generic/252) is reported to the user as fully successful.

- In f2fs_dio_write_iter(), if nothing was written by direct I/O and the
  buffered write fails (e.g. with -ENOSPC in generic/299), the function
  returns 0 instead of the error.

Follow direct_write_fallback(): if the buffered write fails, return the
number of bytes written by direct I/O, or the error if there were none.
If writeback fails, reset ki_pos to the start of the buffered part,
since we don't know how much of it reached the disk, and return the
direct I/O byte count or the error.

For f2fs_file_write_iter(), flush before generic_write_sync() and while
still holding the inode lock, so the error can be returned and the range
can't be re-dirtied by another writer in the meantime.

Writing a part of a compressed cluster marks the whole cluster dirty
(set_cluster_dirty()), so flush and invalidate whole clusters for
compressed files; otherwise dirty pages outside the written range stay
in the page cache.

Also don't fall back to buffered I/O in f2fs_dio_write_iter() for
IOCB_NOWAIT, where f2fs_buffered_write_iter() would just return
-EOPNOTSUPP. Return -EAGAIN instead if nothing was written, so that the
caller retries in a blocking context.

Fixes: a1e09b03e6f5 ("f2fs: use iomap for direct I/O")
Fixes: 92318f20d703 ("f2fs: preserve direct write semantics when buffering is forced")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/file.c | 77 +++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 61 insertions(+), 16 deletions(-)

diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 05061254c3b2..b67782043396 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -5643,17 +5643,32 @@ static const struct iomap_dio_ops f2fs_iomap_dio_write_ops = {
 	.submit_io	= f2fs_dio_write_submit_io,
 };
 
-static void f2fs_flush_buffered_write(struct address_space *mapping,
-				      loff_t start_pos, loff_t end_pos)
+/*
+ * Write back and drop the page cache of a buffered fallback of an O_DIRECT
+ * write.  Writing a part of a compressed cluster dirties the whole cluster,
+ * so extend the range to cluster boundaries for compressed files.
+ */
+static int f2fs_flush_buffered_write(struct address_space *mapping,
+				     loff_t start_pos, loff_t end_pos)
 {
+	struct inode *inode = mapping->host;
 	int ret;
 
+	if (f2fs_compressed_file(inode)) {
+		loff_t cluster_bytes =
+			(loff_t)F2FS_I(inode)->i_cluster_size << PAGE_SHIFT;
+
+		start_pos = round_down(start_pos, cluster_bytes);
+		end_pos = round_up(end_pos + 1, cluster_bytes) - 1;
+	}
+
 	ret = filemap_write_and_wait_range(mapping, start_pos, end_pos);
 	if (ret < 0)
-		return;
+		return ret;
 	invalidate_mapping_pages(mapping,
 				 start_pos >> PAGE_SHIFT,
 				 end_pos >> PAGE_SHIFT);
+	return 0;
 }
 
 static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
@@ -5740,6 +5755,13 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 		ssize_t ret2;
 		loff_t bufio_start_pos = iocb->ki_pos;
 
+		/* A buffered write can block, so don't fall back for NOWAIT. */
+		if (iocb->ki_flags & IOCB_NOWAIT) {
+			if (!ret)
+				ret = -EAGAIN;
+			goto out;
+		}
+
 		/*
 		 * The direct write was partial, so we need to fall back to a
 		 * buffered write for the remainder.
@@ -5748,8 +5770,15 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 		ret2 = f2fs_buffered_write_iter(iocb, from);
 		if (iov_iter_count(from))
 			f2fs_write_failed(inode, iocb->ki_pos);
-		if (ret2 < 0)
+		/*
+		 * Return the number of bytes written by direct I/O, or the
+		 * error if there were none, same as direct_write_fallback().
+		 */
+		if (ret2 < 0) {
+			if (!ret)
+				ret = ret2;
 			goto out;
+		}
 
 		/*
 		 * Ensure that the pagecache pages are written to disk and
@@ -5757,12 +5786,20 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 		 */
 		if (ret2 > 0) {
 			loff_t bufio_end_pos = bufio_start_pos + ret2 - 1;
+			int err;
+
+			err = f2fs_flush_buffered_write(file->f_mapping,
+							bufio_start_pos,
+							bufio_end_pos);
+			if (err) {
+				/* We don't know how much of it was written. */
+				iocb->ki_pos = bufio_start_pos;
+				if (!ret)
+					ret = err;
+				goto out;
+			}
 
 			ret += ret2;
-
-			f2fs_flush_buffered_write(file->f_mapping,
-						  bufio_start_pos,
-						  bufio_end_pos);
 		}
 	} else {
 		/* iomap_dio_rw() already handled the generic_write_sync(). */
@@ -5841,6 +5878,22 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 			f2fs_buffered_write_iter(iocb, from);
 
 		trace_f2fs_datawrite_end(inode, orig_pos, ret);
+
+		/*
+		 * If buffered IO was forced, flush and drop the data from
+		 * the page cache to preserve O_DIRECT semantics.
+		 */
+		if (ret > 0 && !dio && (iocb->ki_flags & IOCB_DIRECT)) {
+			int err;
+
+			err = f2fs_flush_buffered_write(iocb->ki_filp->f_mapping,
+							bufio_start_pos,
+							bufio_start_pos + ret - 1);
+			if (err) {
+				iocb->ki_pos = bufio_start_pos;
+				ret = err;
+			}
+		}
 	}
 
 	/* Don't leave any preallocated blocks around past i_size. */
@@ -5864,14 +5917,6 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (ret > 0 && may_need_sync)
 		ret = generic_write_sync(iocb, ret);
 
-	/* If buffered IO was forced, flush and drop the data from
-	 * the page cache to preserve O_DIRECT semantics
-	 */
-	if (ret > 0 && !dio && (iocb->ki_flags & IOCB_DIRECT))
-		f2fs_flush_buffered_write(iocb->ki_filp->f_mapping,
-					  bufio_start_pos,
-					  bufio_start_pos + ret - 1);
-
 	return ret;
 }
 
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 5/9] f2fs: truncate blocks allocated past EOF by a failed direct write

[Jan Prusakowski]

When a direct write extends a file, f2fs_map_blocks() allocates the
blocks past EOF and maps them into the dnode before the bio is
submitted. If the write then fails, i_size is not updated, but the
blocks stay mapped past EOF with whatever the disk contained before.
A later extending write or truncate can make them visible.

f2fs_preallocate_blocks() already handles this for blocks it
preallocates with F2FS_GET_BLOCK_PRE_DIO: it marks the inode with
file_need_truncate() and f2fs_file_write_iter() truncates everything
past i_size after a short write. But the preallocation is skipped in
LFS mode and when the user buffer can't be faulted in, and then the
blocks are allocated by F2FS_GET_BLOCK_DIO from f2fs_iomap_begin() and
nothing truncates them.

Mark the inode with file_need_truncate() also when F2FS_GET_BLOCK_DIO
allocates a block past EOF, and truncate after the write if the flag is
set and the file is shorter than intended. Skip KEEP_SIZE files, where
the truncation would also drop the blocks reserved by fallocate().

The extending direct write is always synchronous (IOMAP_DIO_FORCE_WAIT),
so the result is known before f2fs_file_write_iter() returns.

Fixes: d4dd19ec1ea0 ("f2fs: do not expose unwritten blocks to user by DIO")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/data.c | 10 +++++++++-
 fs/f2fs/file.c |  8 ++++++--
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 4ae61138a003..e978b6b18c26 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -2171,7 +2171,15 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag)
 			err = __allocate_data_block(&dn, map->m_seg_type);
 			if (err)
 				goto sync_out;
-			if (flag == F2FS_GET_BLOCK_PRE_DIO)
+			/*
+			 * Blocks allocated past EOF must be truncated if the
+			 * write fails. Don't do it for KEEP_SIZE files, where
+			 * truncation would also drop the fallocated blocks.
+			 */
+			if (flag == F2FS_GET_BLOCK_PRE_DIO ||
+			    (flag == F2FS_GET_BLOCK_DIO &&
+			     !file_keep_isize(inode) &&
+			     pgofs >= F2FS_BLK_ALIGN(sbi, i_size_read(inode))))
 				file_need_truncate(inode);
 			set_inode_flag(inode, FI_APPEND_WRITE);
 			break;
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index b67782043396..3fcea85c36a2 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -5896,8 +5896,12 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		}
 	}
 
-	/* Don't leave any preallocated blocks around past i_size. */
-	if (preallocated && i_size_read(inode) < target_size) {
+	/*
+	 * Don't leave any preallocated blocks, or blocks allocated by a failed
+	 * direct write, around past i_size.
+	 */
+	if ((preallocated || file_should_truncate(inode)) &&
+	    i_size_read(inode) < target_size) {
 		f2fs_down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
 		filemap_invalidate_lock(inode->i_mapping);
 		if (!f2fs_truncate(inode))
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 6/9] f2fs: roll back failed block updates of O_DIRECT buffered fallback

[Jan Prusakowski]

When an O_DIRECT write falls back to buffered I/O, the data is written
back by f2fs_flush_buffered_write() before the write returns. Writeback
does out-of-place updates: f2fs_outplace_write_data() allocates a new
block, points the dnode at it and invalidates the old block before the
bio completes. If the bio then fails (e.g. with dm-error when O_DIRECT
is forced to buffered I/O), the error is returned to the user, but the
dnode keeps pointing to a block that was never written, and reads return
whatever the disk contained before, possibly data of a deleted file.

Track out-of-place updates of the written range while the fallback is in
progress, and roll back the ones whose write failed:

- f2fs_start_dio_bufio() records the page range of the write. It is
  called with the inode lock held, before the buffered write.

- f2fs_do_write_data_page() records the old and new block address of
  each out-of-place update in that range with f2fs_start_dio_opu_track()
  and f2fs_end_dio_opu_track(), and f2fs_write_end_bio() marks the
  entries of the pages of a failed bio segment with
  f2fs_mark_dio_opu_failed(). Entries are looked up by page index in an
  xarray, so bio completion doesn't walk a list.
  A page rewritten in place after a failed update holds newer data, so
  its entry is marked superseded and not rolled back.

- f2fs_finish_dio_bufio() stops tracking, waits until the tracked
  updates have reached the dnode and until their I/O has completed, and
  then rolls back each failed update whose blocks were not changed again
  since. If no checkpoint was done since the update and the old blocks
  are valid in the last checkpoint, the old blocks can't have been
  reused, so the dnode points back to them and they are validated again
  with the new f2fs_revalidate_blocks(). Otherwise the blocks are reset
  to NEW_ADDR, so that they read as zeroes. Page cache of rolled back
  updates is dropped. If the write failed, its page cache is dropped and
  i_size is restored, keeping blocks reserved by
  fallocate(FALLOC_FL_KEEP_SIZE).

The state is allocated on the first fallback of an inode and freed with
the inode, so f2fs_inode_info only grows by one pointer. Entries are
allocated with __GFP_NOFAIL, so the tracking can't fail writeback.

Writeback of the range by other tasks in the meantime (e.g. of mmap
writes, or GC moving pages with f2fs_do_write_data_page()) is tracked
and rolled back on failure as well, but only failures of the bytes
written by this write, and of the flush, make it fail.

Fixes: a1e09b03e6f5 ("f2fs: use iomap for direct I/O")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/data.c    | 163 ++++++++++++++++++++++++
 fs/f2fs/f2fs.h    |  55 +++++++++
 fs/f2fs/file.c    | 306 +++++++++++++++++++++++++++++++++++++++++++---
 fs/f2fs/segment.c |  42 ++++++-
 fs/f2fs/super.c   |   6 +
 5 files changed, 550 insertions(+), 22 deletions(-)

diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index e978b6b18c26..83c414cac252 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -410,6 +410,149 @@ static void f2fs_ffs_put_write_pending(struct folio *folio, unsigned int nr)
 	if (folio_test_writeback(folio))
 		folio_end_writeback(folio);
 }
+
+/*
+ * Out-of-place updates done while an O_DIRECT write falls back to buffered
+ * I/O are tracked, so that they can be rolled back by f2fs_finish_dio_bufio()
+ * if the write fails. Otherwise the dnode would keep pointing to new blocks
+ * that were never written and expose their old contents.
+ */
+struct f2fs_dio_opu_entry *f2fs_start_dio_opu_track(struct inode *inode,
+						    pgoff_t start_idx,
+						    unsigned int nr_pages)
+{
+	/* Pairs with smp_store_release() in f2fs_start_dio_bufio(). */
+	struct f2fs_dio_bufio *b = smp_load_acquire(&F2FS_I(inode)->dio_bufio);
+	struct f2fs_dio_opu_entry *entry;
+	unsigned long flags;
+	unsigned int i;
+
+	if (!b)
+		return NULL;
+
+	/* Failing here would fail writeback, so don't. */
+	entry = kzalloc(struct_size(entry, blkaddr, 2 * nr_pages),
+			GFP_NOFS | __GFP_NOFAIL);
+	entry->old_blkaddr = entry->blkaddr;
+	entry->new_blkaddr = entry->blkaddr + nr_pages;
+	entry->start_idx = start_idx;
+	entry->nr_pages = nr_pages;
+	/* The caller holds f2fs_lock_op(), so this can't change under us. */
+	entry->cp_ver = cur_cp_version(F2FS_CKPT(F2FS_I_SB(inode)));
+
+	/* Reserve the slots, so that storing the entry below won't allocate. */
+	for (i = 0; i < nr_pages; i++)
+		if (xa_reserve_irq(&b->entries, start_idx + i,
+				   GFP_NOFS | __GFP_NOFAIL))
+			break;
+
+	xa_lock_irqsave(&b->entries, flags);
+	if (unlikely(i < nr_pages) || !b->tracking ||
+	    start_idx + nr_pages - 1 < b->start || start_idx > b->end) {
+		for (i = 0; i < nr_pages; i++)
+			__xa_cmpxchg(&b->entries, start_idx + i,
+				     XA_ZERO_ENTRY, NULL, 0);
+		xa_unlock_irqrestore(&b->entries, flags);
+		kfree(entry);
+		return NULL;
+	}
+	for (i = 0; i < nr_pages; i++) {
+		void *old = __xa_store(&b->entries, start_idx + i, entry,
+				       GFP_ATOMIC);
+
+		/* The slots are reserved, so this can't fail. */
+		if (WARN_ON_ONCE(xa_is_err(old)))
+			break;
+	}
+	if (unlikely(i < nr_pages)) {
+		while (i--)
+			__xa_cmpxchg(&b->entries, start_idx + i, entry, NULL, 0);
+		for (i = 0; i < nr_pages; i++)
+			__xa_cmpxchg(&b->entries, start_idx + i,
+				     XA_ZERO_ENTRY, NULL, 0);
+		xa_unlock_irqrestore(&b->entries, flags);
+		kfree(entry);
+		return NULL;
+	}
+	list_add_tail(&entry->list, &b->list);
+	atomic_inc(&b->pending);
+	xa_unlock_irqrestore(&b->entries, flags);
+
+	return entry;
+}
+
+/*
+ * Called after the block addresses of @entry were updated. If the folios were
+ * not submitted, the dnode was not changed and the entry is dropped.
+ */
+void f2fs_end_dio_opu_track(struct inode *inode,
+			    struct f2fs_dio_opu_entry *entry,
+			    bool submitted, bool failed)
+{
+	struct f2fs_dio_bufio *b = F2FS_I(inode)->dio_bufio;
+	unsigned long flags;
+	unsigned int i;
+
+	xa_lock_irqsave(&b->entries, flags);
+	if (unlikely(!submitted)) {
+		for (i = 0; i < entry->nr_pages; i++)
+			__xa_cmpxchg(&b->entries, entry->start_idx + i,
+				     entry, NULL, 0);
+		list_del(&entry->list);
+	} else if (unlikely(failed)) {
+		entry->failed = true;
+	}
+	xa_unlock_irqrestore(&b->entries, flags);
+
+	if (unlikely(!submitted))
+		kfree(entry);
+
+	/* Pairs with wait_var_event() in f2fs_finish_dio_bufio(). */
+	if (atomic_dec_and_test(&b->pending))
+		wake_up_var(&b->pending);
+}
+
+static void f2fs_mark_dio_opu(struct inode *inode, pgoff_t index,
+			      bool failed)
+{
+	struct f2fs_dio_bufio *b;
+	struct f2fs_dio_opu_entry *entry;
+	unsigned long flags;
+
+	if (!inode)
+		return;
+
+	/* Pairs with smp_store_release() in f2fs_start_dio_bufio(). */
+	b = smp_load_acquire(&F2FS_I(inode)->dio_bufio);
+	if (likely(!b) || !READ_ONCE(b->active))
+		return;
+
+	xa_lock_irqsave(&b->entries, flags);
+	if (b->active) {
+		entry = xa_load(&b->entries, index);
+		if (entry && failed)
+			entry->failed = true;
+		else if (entry)
+			entry->superseded = true;
+	}
+	xa_unlock_irqrestore(&b->entries, flags);
+}
+
+/* Called on write I/O failure of the folio at @index, may run in IRQ context. */
+void f2fs_mark_dio_opu_failed(struct inode *inode, pgoff_t index)
+{
+	f2fs_mark_dio_opu(inode, index, true);
+}
+
+/*
+ * Called before the folio at @index is rewritten in place: the block now holds
+ * newer data, so a failed out-of-place update to it must not be rolled back.
+ */
+void f2fs_mark_dio_opu_superseded(struct inode *inode, pgoff_t index)
+{
+	f2fs_mark_dio_opu(inode, index, false);
+}
+
 static void f2fs_write_end_bio(struct bio *bio)
 {
 	struct f2fs_sb_info *sbi = bio->bi_private;
@@ -439,7 +582,13 @@ static void f2fs_write_end_bio(struct bio *bio)
 		type = WB_DATA_TYPE(folio, false);
 
 		if (unlikely(bio->bi_status != BLK_STS_OK)) {
+			pgoff_t idx = folio->index + (fi.offset >> PAGE_SHIFT);
+			unsigned int j;
+
 			mapping_set_error(folio->mapping, -EIO);
+			for (j = 0; j < npages; j++)
+				f2fs_mark_dio_opu_failed(folio->mapping->host,
+							 idx + j);
 			if (type == F2FS_WB_CP_DATA) {
 				f2fs_stop_checkpoint(sbi, true,
 						STOP_CP_REASON_WRITE_FAIL);
@@ -3756,6 +3905,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
 	struct node_info ni;
 	struct f2fs_lock_context lc;
 	struct f2fs_folio_state *ffs;
+	struct f2fs_dio_opu_entry *opu_entry = NULL;
 	bool ipu_force = false;
 	bool atomic_commit;
 	bool pending_added = false;
@@ -3824,6 +3974,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
 	if (ipu_force ||
 		(__is_valid_data_blkaddr(fio->old_blkaddr) &&
 					need_inplace_update(fio))) {
+		f2fs_mark_dio_opu_superseded(inode, F2FS_FIO_LBLK(fio));
 		__folio_start_writeback(folio, folio_test_dirty(folio));
 		f2fs_put_dnode(&dn);
 		if (fio->need_lock == LOCK_REQ)
@@ -3851,6 +4002,14 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
 
 	fio->version = ni.version;
 
+	if (!atomic_commit && !fio->compr_blocks &&
+	    unlikely(f2fs_should_track_dio_opu(inode, F2FS_FIO_LBLK(fio), 1))) {
+		opu_entry = f2fs_start_dio_opu_track(inode, F2FS_FIO_LBLK(fio),
+						     1);
+		if (opu_entry)
+			opu_entry->old_blkaddr[0] = fio->old_blkaddr;
+	}
+
 	__folio_start_writeback(folio, folio_test_dirty(folio));
 	if (f2fs_folio_has_ffs(folio)) {
 		ffs = (struct f2fs_folio_state *)folio->private;
@@ -3863,6 +4022,10 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
 
 	/* LFS mode write path */
 	f2fs_outplace_write_data(&dn, fio);
+	if (unlikely(opu_entry)) {
+		opu_entry->new_blkaddr[0] = fio->new_blkaddr;
+		f2fs_end_dio_opu_track(inode, opu_entry, fio->submitted, false);
+	}
 	trace_f2fs_do_write_data_page(folio, OPU);
 	/*
 	 * The outplace write path failed to submit this block.
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 4fa4d8042f70..7c36b996a09c 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -941,6 +941,35 @@ enum {
 	FI_MAX,			/* max flag, never be used */
 };
 
+/*
+ * Out-of-place block update done while an O_DIRECT write falls back to
+ * buffered I/O, recorded so that it can be rolled back if the write fails.
+ */
+struct f2fs_dio_opu_entry {
+	struct list_head list;		/* in f2fs_dio_bufio->list */
+	unsigned long long cp_ver;	/* checkpoint version of the update */
+	pgoff_t start_idx;		/* first page index */
+	unsigned int nr_pages;		/* number of pages */
+	unsigned int old_compr_blocks;	/* compressed blocks before update */
+	unsigned int new_compr_blocks;	/* compressed blocks after update */
+	bool failed;			/* write I/O failed */
+	bool superseded;		/* rewritten in place since */
+	block_t *old_blkaddr;		/* block addresses before update */
+	block_t *new_blkaddr;		/* block addresses after update */
+	block_t blkaddr[];		/* storage of old_blkaddr/new_blkaddr */
+};
+
+/* State of an O_DIRECT write falling back to buffered I/O */
+struct f2fs_dio_bufio {
+	struct xarray entries;		/* page index -> latest entry */
+	struct list_head list;		/* all entries, oldest first */
+	atomic_t pending;		/* entries with block update in progress */
+	pgoff_t start;			/* first page index of the write */
+	pgoff_t end;			/* last page index of the write */
+	bool tracking;			/* record new block updates */
+	bool active;			/* record write failures */
+};
+
 struct f2fs_inode_info {
 	struct inode vfs_inode;		/* serve a vfs inode */
 	unsigned long i_flags;		/* keep an inode flags for ioctl */
@@ -1009,6 +1038,9 @@ struct f2fs_inode_info {
 	unsigned int i_cluster_size;		/* cluster size */
 	atomic_t writeback;			/* count # of writeback thread */
 
+	/* allocated by the first O_DIRECT write falling back to buffered I/O */
+	struct f2fs_dio_bufio *dio_bufio;
+
 	unsigned int atomic_write_cnt;
 	loff_t original_i_size;		/* original i_size before atomic write */
 #ifdef CONFIG_FS_ENCRYPTION
@@ -3867,6 +3899,19 @@ static inline int f2fs_has_inline_dentry(struct inode *inode)
 	return is_inode_flag_set(inode, FI_INLINE_DENTRY);
 }
 
+static inline bool f2fs_should_track_dio_opu(struct inode *inode,
+					     pgoff_t start_idx,
+					     unsigned int nr_pages)
+{
+	/* Pairs with smp_store_release() in f2fs_start_dio_bufio(). */
+	struct f2fs_dio_bufio *b = smp_load_acquire(&F2FS_I(inode)->dio_bufio);
+
+	if (likely(!b) || !READ_ONCE(b->tracking))
+		return false;
+	return start_idx + nr_pages - 1 >= READ_ONCE(b->start) &&
+	       start_idx <= READ_ONCE(b->end);
+}
+
 static inline int is_file(struct inode *inode, int type)
 {
 	return F2FS_I(inode)->i_advise & type;
@@ -4258,6 +4303,8 @@ int f2fs_flush_device_cache(struct f2fs_sb_info *sbi);
 void f2fs_destroy_flush_cmd_control(struct f2fs_sb_info *sbi, bool free);
 void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr,
 						unsigned int len);
+void f2fs_revalidate_blocks(struct f2fs_sb_info *sbi, block_t addr,
+			    unsigned int len);
 void f2fs_reserve_device_alias(struct f2fs_sb_info *sbi, block_t addr,
 						unsigned int len);
 bool f2fs_is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr);
@@ -4472,6 +4519,14 @@ struct folio *f2fs_get_new_data_folio(struct inode *inode,
 			struct f2fs_cached_block *ientry, pgoff_t index, bool new_i_size);
 void f2fs_ffs_mark_subrange_uptodate(struct folio *folio, size_t offset, size_t len);
 int f2fs_do_write_data_page(struct f2fs_io_info *fio);
+struct f2fs_dio_opu_entry *f2fs_start_dio_opu_track(struct inode *inode,
+						    pgoff_t start_idx,
+						    unsigned int nr_pages);
+void f2fs_end_dio_opu_track(struct inode *inode,
+			    struct f2fs_dio_opu_entry *entry,
+			    bool submitted, bool failed);
+void f2fs_mark_dio_opu_failed(struct inode *inode, pgoff_t index);
+void f2fs_mark_dio_opu_superseded(struct inode *inode, pgoff_t index);
 int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag);
 int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 			u64 start, u64 len);
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 3fcea85c36a2..70ba3f535211 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -5644,23 +5644,30 @@ static const struct iomap_dio_ops f2fs_iomap_dio_write_ops = {
 };
 
 /*
- * Write back and drop the page cache of a buffered fallback of an O_DIRECT
- * write.  Writing a part of a compressed cluster dirties the whole cluster,
- * so extend the range to cluster boundaries for compressed files.
+ * Writing a part of a compressed cluster dirties the whole cluster, so the
+ * page cache of a buffered fallback of an O_DIRECT write must be handled in
+ * whole clusters for compressed files.
  */
+static void f2fs_dio_bufio_range(struct inode *inode, loff_t *start_pos,
+				 loff_t *end_pos)
+{
+	loff_t cluster_bytes;
+
+	if (!f2fs_compressed_file(inode))
+		return;
+
+	cluster_bytes = (loff_t)F2FS_I(inode)->i_cluster_size << PAGE_SHIFT;
+	*start_pos = round_down(*start_pos, cluster_bytes);
+	*end_pos = round_up(*end_pos + 1, cluster_bytes) - 1;
+}
+
+/* Write back and drop the page cache of a buffered fallback of O_DIRECT. */
 static int f2fs_flush_buffered_write(struct address_space *mapping,
 				     loff_t start_pos, loff_t end_pos)
 {
-	struct inode *inode = mapping->host;
 	int ret;
 
-	if (f2fs_compressed_file(inode)) {
-		loff_t cluster_bytes =
-			(loff_t)F2FS_I(inode)->i_cluster_size << PAGE_SHIFT;
-
-		start_pos = round_down(start_pos, cluster_bytes);
-		end_pos = round_up(end_pos + 1, cluster_bytes) - 1;
-	}
+	f2fs_dio_bufio_range(mapping->host, &start_pos, &end_pos);
 
 	ret = filemap_write_and_wait_range(mapping, start_pos, end_pos);
 	if (ret < 0)
@@ -5671,6 +5678,234 @@ static int f2fs_flush_buffered_write(struct address_space *mapping,
 	return 0;
 }
 
+/*
+ * Start tracking out-of-place block updates of the buffered fallback of an
+ * O_DIRECT write of @count bytes at @pos. Called with the inode lock held.
+ */
+static void f2fs_start_dio_bufio(struct inode *inode, loff_t pos, size_t count)
+{
+	struct f2fs_inode_info *fi = F2FS_I(inode);
+	struct f2fs_dio_bufio *b = fi->dio_bufio;
+	loff_t start_pos = pos, end_pos = pos + count - 1;
+	unsigned long flags;
+
+	if (!count)
+		return;
+
+	if (!b) {
+		b = kzalloc_obj(*b, GFP_NOFS | __GFP_NOFAIL);
+		xa_init_flags(&b->entries, XA_FLAGS_LOCK_IRQ);
+		INIT_LIST_HEAD(&b->list);
+		atomic_set(&b->pending, 0);
+		/* Pairs with smp_load_acquire() in the writeback paths. */
+		smp_store_release(&fi->dio_bufio, b);
+	}
+
+	f2fs_dio_bufio_range(inode, &start_pos, &end_pos);
+
+	xa_lock_irqsave(&b->entries, flags);
+	WRITE_ONCE(b->start, start_pos >> PAGE_SHIFT);
+	WRITE_ONCE(b->end, end_pos >> PAGE_SHIFT);
+	WRITE_ONCE(b->active, true);
+	WRITE_ONCE(b->tracking, true);
+	xa_unlock_irqrestore(&b->entries, flags);
+}
+
+/* Roll back the out-of-place update of @entry, whose write failed. */
+static void f2fs_rollback_dio_opu_entry(struct inode *inode,
+					struct f2fs_dio_opu_entry *entry)
+{
+	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+	struct dnode_of_data dn;
+	bool can_revalidate;
+	unsigned int i;
+	int err;
+
+	set_new_dnode(&dn, inode, NULL, NULL, 0);
+	err = f2fs_get_dnode_of_data(&dn, entry->start_idx, LOOKUP_NODE);
+	if (err == -ENOENT)	/* truncated, nothing to roll back */
+		return;
+	if (err) {
+		f2fs_warn_ratelimited(sbi,
+				      "%s: failed to roll back write, ino:%llu, index:%lu, err:%d",
+				      __func__, inode->i_ino, entry->start_idx,
+				      err);
+		set_sbi_flag(sbi, SBI_NEED_FSCK);
+		return;
+	}
+
+	/* Leave the blocks alone if they were updated again since. */
+	for (i = 0; i < entry->nr_pages; i++) {
+		if (data_blkaddr(dn.inode, dn.node_entry,
+				 dn.ofs_in_node + i) != entry->new_blkaddr[i])
+			goto put_dnode;
+	}
+
+	/*
+	 * The update invalidated the old blocks. They can be validated again
+	 * only if they can't have been reused since: no checkpoint was done,
+	 * and they are valid in the last checkpoint (otherwise SSR may reuse
+	 * them, e.g. with checkpoint=disable). If they can't, reset the
+	 * blocks to NEW_ADDR, which reads as zeroes, so the failed write at
+	 * least doesn't expose stale data.
+	 */
+	can_revalidate = cur_cp_version(F2FS_CKPT(sbi)) == entry->cp_ver;
+	for (i = 0; can_revalidate && i < entry->nr_pages; i++) {
+		if (!f2fs_is_checkpointed_data(sbi, entry->old_blkaddr[i]))
+			can_revalidate = false;
+	}
+
+	for (i = 0; i < entry->nr_pages; i++, dn.ofs_in_node++) {
+		block_t old_blk = entry->old_blkaddr[i];
+		block_t new_blk = entry->new_blkaddr[i];
+
+		if (can_revalidate) {
+			f2fs_update_data_blkaddr(&dn, old_blk);
+			if (new_blk == old_blk)
+				continue;
+			if (__is_valid_data_blkaddr(new_blk))
+				f2fs_invalidate_blocks(sbi, new_blk, 1);
+			if (__is_valid_data_blkaddr(old_blk))
+				f2fs_revalidate_blocks(sbi, old_blk, 1);
+		} else {
+			if (__is_valid_data_blkaddr(new_blk))
+				f2fs_invalidate_blocks(sbi, new_blk, 1);
+			f2fs_update_data_blkaddr(&dn, NEW_ADDR);
+		}
+	}
+
+	if (entry->new_compr_blocks)
+		f2fs_i_compr_blocks_update(inode, entry->new_compr_blocks - 1,
+					   false);
+	if (can_revalidate && entry->old_compr_blocks)
+		f2fs_i_compr_blocks_update(inode, entry->old_compr_blocks - 1,
+					   true);
+put_dnode:
+	f2fs_put_dnode(&dn);
+}
+
+/*
+ * Finish the buffered fallback of an O_DIRECT write started by
+ * f2fs_start_dio_bufio(). @len bytes were written at @start_pos to the page
+ * cache, and @err is the result of writing them back. Failed block updates
+ * in the range are rolled back. If the write failed, its page cache is
+ * dropped and i_size is restored. Returns @err, or -EIO if a block update of
+ * the written bytes failed.
+ */
+static int f2fs_finish_dio_bufio(struct inode *inode, loff_t start_pos,
+				 loff_t len, loff_t orig_i_size,
+				 bool keep_isize, int err)
+{
+	struct f2fs_inode_info *fi = F2FS_I(inode);
+	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+	struct address_space *mapping = inode->i_mapping;
+	struct f2fs_dio_bufio *b = fi->dio_bufio;
+	struct f2fs_dio_opu_entry *entry, *tmp;
+	LIST_HEAD(entries);
+	unsigned long flags;
+	loff_t end_pos = start_pos + len - 1;
+	pgoff_t first, last;
+	bool failed = false;
+
+	if (!b || !b->active)
+		return err;
+
+	/* Stop tracking new block updates, ... */
+	xa_lock_irqsave(&b->entries, flags);
+	WRITE_ONCE(b->tracking, false);
+	xa_unlock_irqrestore(&b->entries, flags);
+
+	/* ... wait for the tracked ones to finish updating the dnode, ... */
+	wait_var_event(&b->pending, !atomic_read_acquire(&b->pending));
+
+	/*
+	 * ... and for their I/O to complete, so that failures are recorded.
+	 * Leave errors in the mapping for others to see.
+	 */
+	filemap_fdatawait_range_keep_errors(mapping,
+					    (loff_t)b->start << PAGE_SHIFT,
+					    (loff_t)b->end << PAGE_SHIFT |
+					    (PAGE_SIZE - 1));
+
+	xa_lock_irqsave(&b->entries, flags);
+	WRITE_ONCE(b->active, false);
+	list_splice_init(&b->list, &entries);
+	xa_unlock_irqrestore(&b->entries, flags);
+
+	/*
+	 * Remove the entries from the xarray one by one rather than with
+	 * xa_destroy(), which would also drop slots just reserved by a writer
+	 * of the next fallback.
+	 */
+	f2fs_dio_bufio_range(inode, &start_pos, &end_pos);
+	first = start_pos >> PAGE_SHIFT;
+	last = end_pos >> PAGE_SHIFT;
+	list_for_each_entry(entry, &entries, list) {
+		unsigned int i;
+
+		for (i = 0; i < entry->nr_pages; i++)
+			xa_cmpxchg_irq(&b->entries, entry->start_idx + i,
+				       entry, NULL, 0);
+		if (entry->failed && !entry->superseded) {
+			/* Only failures of the written bytes fail the write. */
+			if (len > 0 && entry->start_idx <= last &&
+			    entry->start_idx + entry->nr_pages - 1 >= first &&
+			    !err)
+				err = -EIO;
+			failed = true;
+		}
+	}
+
+	if (failed || (err && len > 0)) {
+		f2fs_down_write(&fi->i_gc_rwsem[WRITE]);
+		filemap_invalidate_lock(mapping);
+
+		if (failed) {
+			struct f2fs_lock_context lc;
+
+			f2fs_lock_op(sbi, &lc);
+			list_for_each_entry_reverse(entry, &entries, list) {
+				if (entry->failed && !entry->superseded)
+					f2fs_rollback_dio_opu_entry(inode,
+								    entry);
+			}
+			f2fs_unlock_op(sbi, &lc);
+
+			/* Cached data no longer matches the blocks on disk. */
+			list_for_each_entry(entry, &entries, list) {
+				pgoff_t idx = entry->start_idx;
+				pgoff_t end = idx + entry->nr_pages - 1;
+
+				if (!entry->failed || entry->superseded)
+					continue;
+				invalidate_inode_pages2_range(mapping, idx,
+							      end);
+			}
+		}
+
+		/* Drop the page cache of the failed write, as for O_DIRECT. */
+		if (err && len > 0)
+			invalidate_inode_pages2_range(mapping, first, last);
+
+		if (err && i_size_read(inode) > orig_i_size) {
+			truncate_pagecache(inode, orig_i_size);
+			f2fs_i_size_write(inode, orig_i_size);
+			if (!keep_isize)
+				f2fs_truncate_blocks(inode, orig_i_size, true);
+		}
+
+		filemap_invalidate_unlock(mapping);
+		f2fs_up_write(&fi->i_gc_rwsem[WRITE]);
+	}
+
+	list_for_each_entry_safe(entry, tmp, &entries, list) {
+		list_del(&entry->list);
+		kfree(entry);
+	}
+
+	return err;
+}
+
 static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 				   bool *may_need_sync)
 {
@@ -5755,6 +5990,9 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 		ssize_t ret2;
 		loff_t bufio_start_pos = iocb->ki_pos;
 
+		loff_t bufio_orig_size = i_size_read(inode);
+		bool keep_isize = file_keep_isize(inode);
+
 		/* A buffered write can block, so don't fall back for NOWAIT. */
 		if (iocb->ki_flags & IOCB_NOWAIT) {
 			if (!ret)
@@ -5766,16 +6004,22 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 		 * The direct write was partial, so we need to fall back to a
 		 * buffered write for the remainder.
 		 */
-
+		f2fs_start_dio_bufio(inode, bufio_start_pos,
+				     iov_iter_count(from));
 		ret2 = f2fs_buffered_write_iter(iocb, from);
 		if (iov_iter_count(from))
 			f2fs_write_failed(inode, iocb->ki_pos);
 		/*
 		 * Return the number of bytes written by direct I/O, or the
 		 * error if there were none, same as direct_write_fallback().
+		 * Failures of other writes in the range are rolled back, but
+		 * not reported here.
 		 */
-		if (ret2 < 0) {
-			if (!ret)
+		if (ret2 <= 0) {
+			f2fs_finish_dio_bufio(inode, bufio_start_pos, 0,
+					      bufio_orig_size, keep_isize,
+					      ret2);
+			if (ret2 < 0 && !ret)
 				ret = ret2;
 			goto out;
 		}
@@ -5791,6 +6035,9 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 			err = f2fs_flush_buffered_write(file->f_mapping,
 							bufio_start_pos,
 							bufio_end_pos);
+			err = f2fs_finish_dio_bufio(inode, bufio_start_pos,
+						    ret2, bufio_orig_size,
+						    keep_isize, err);
 			if (err) {
 				/* We don't know how much of it was written. */
 				iocb->ki_pos = bufio_start_pos;
@@ -5868,10 +6115,20 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 	if (preallocated < 0) {
 		ret = preallocated;
 	} else {
+		/* Buffered write fails right away for NOWAIT. */
+		bool forced_bufio = !dio && (iocb->ki_flags & IOCB_DIRECT) &&
+				    !(iocb->ki_flags & IOCB_NOWAIT);
+		loff_t orig_i_size = i_size_read(inode);
+		bool keep_isize = file_keep_isize(inode);
+
 		if (trace_f2fs_datawrite_start_enabled())
 			f2fs_trace_rw_file_path(iocb->ki_filp, iocb->ki_pos,
 						orig_count, WRITE);
 
+		if (forced_bufio)
+			f2fs_start_dio_bufio(inode, bufio_start_pos,
+					     iov_iter_count(from));
+
 		/* Do the actual write. */
 		ret = dio ?
 			f2fs_dio_write_iter(iocb, from, &may_need_sync) :
@@ -5883,13 +6140,22 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		 * If buffered IO was forced, flush and drop the data from
 		 * the page cache to preserve O_DIRECT semantics.
 		 */
-		if (ret > 0 && !dio && (iocb->ki_flags & IOCB_DIRECT)) {
-			int err;
+		if (forced_bufio) {
+			struct address_space *mapping = iocb->ki_filp->f_mapping;
+			int err = 0;
 
-			err = f2fs_flush_buffered_write(iocb->ki_filp->f_mapping,
-							bufio_start_pos,
-							bufio_start_pos + ret - 1);
-			if (err) {
+			if (ret > 0) {
+				loff_t bufio_end_pos = bufio_start_pos + ret - 1;
+
+				err = f2fs_flush_buffered_write(mapping,
+								bufio_start_pos,
+								bufio_end_pos);
+			}
+			err = f2fs_finish_dio_bufio(inode, bufio_start_pos,
+						    ret > 0 ? ret : 0,
+						    orig_i_size, keep_isize,
+						    ret < 0 ? ret : err);
+			if (ret > 0 && err) {
 				iocb->ki_pos = bufio_start_pos;
 				ret = err;
 			}
diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index a26cde2dc60e..0cc37788efb8 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -2733,8 +2733,9 @@ void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr,
 	up_write(&sit_i->sentry_lock);
 }
 
-void f2fs_reserve_device_alias(struct f2fs_sb_info *sbi, block_t addr,
-				unsigned int len)
+/* Mark @len blocks starting at @addr as valid in the SIT. */
+static void __validate_blocks(struct f2fs_sb_info *sbi, block_t addr,
+			      unsigned int len)
 {
 	unsigned int segno = GET_SEGNO(sbi, addr);
 	struct sit_info *sit_i = SIT_I(sbi);
@@ -2779,6 +2780,43 @@ void f2fs_reserve_device_alias(struct f2fs_sb_info *sbi, block_t addr,
 	up_write(&sit_i->sentry_lock);
 }
 
+void f2fs_reserve_device_alias(struct f2fs_sb_info *sbi, block_t addr,
+			       unsigned int len)
+{
+	__validate_blocks(sbi, addr, len);
+}
+
+/*
+ * Validate again blocks invalidated by an out-of-place update in the current
+ * checkpoint, when rolling the update back. The caller must make sure the
+ * blocks have not been reused since, see f2fs_is_checkpointed_data().
+ */
+void f2fs_revalidate_blocks(struct f2fs_sb_info *sbi, block_t addr,
+			    unsigned int len)
+{
+	if (len == 0)
+		return;
+
+	f2fs_bug_on(sbi, addr == NULL_ADDR);
+	if (addr == NEW_ADDR || addr == COMPRESS_ADDR)
+		return;
+
+	/* Undo the unusable block accounting done on invalidation. */
+	if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) {
+		unsigned int i;
+
+		for (i = 0; i < len; i++) {
+			if (f2fs_is_checkpointed_data(sbi, addr + i)) {
+				spin_lock(&sbi->stat_lock);
+				sbi->unusable_block_count--;
+				spin_unlock(&sbi->stat_lock);
+			}
+		}
+	}
+
+	__validate_blocks(sbi, addr, len);
+}
+
 bool f2fs_is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr)
 {
 	struct sit_info *sit_i = SIT_I(sbi);
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 78dbf1e92285..cce6bac1cda1 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -1980,6 +1980,12 @@ static void f2fs_dirty_inode(struct inode *inode, int flags)
 
 static void f2fs_free_inode(struct inode *inode)
 {
+	struct f2fs_dio_bufio *b = F2FS_I(inode)->dio_bufio;
+
+	if (b) {
+		xa_destroy(&b->entries);
+		kfree(b);
+	}
 	fscrypt_free_inode(inode);
 	kmem_cache_free(f2fs_inode_cachep, F2FS_I(inode));
 }
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 7/9] f2fs: roll back failed block updates of direct writes

[Jan Prusakowski]

Direct writes to holes (NULL_ADDR), blocks reserved by fallocate()
(NEW_ADDR), or existing blocks in LFS mode allocate new physical blocks
in f2fs_preallocate_blocks() or f2fs_map_blocks(F2FS_GET_BLOCK_DIO) and
map them in the dnode and read extent cache before the bio is submitted.
If the direct write fails (e.g. with dm-error in generic/250 and
generic/252) or stops short, blocks inside i_size or on KEEP_SIZE files
are not truncated by f2fs_file_write_iter(), leaving the dnode and read
extent cache pointing to blocks that were never written and exposing
stale on-disk data.

Fix this while keeping direct writes on the native direct I/O path:

- Record contiguous runs of blocks allocated during a native O_DIRECT
  write per kiocb in f2fs_record_dio_block(), along with their previous
  block address (NULL_ADDR, NEW_ADDR, or old block in LFS mode) and the
  checkpoint version. Skip F2FS_GET_BLOCK_PRE_DIO on KEEP_SIZE files so
  post-EOF truncation on error does not drop blocks reserved by
  fallocate().

- Ensure IOMAP_F_NEW is set whenever a direct write allocated blocks so
  completions run in process context.

- In f2fs_dio_write_end_io() (and error paths where the bio is not
  submitted), free the recorded entries when the full range was written,
  or roll back any unwritten blocks: restore NULL_ADDR (and decrement
  valid block count), restore NEW_ADDR, or restore and revalidate the
  old block in LFS mode if no checkpoint ran since and the old block was
  checkpointed (falling back to NEW_ADDR otherwise).

- In f2fs_file_read_iter(), f2fs_file_splice_read() and
  f2fs_file_write_iter(), wait for any overlapping in-flight allocating
  direct write to finish (or return -EAGAIN for non-blocking I/O) so
  concurrent reads do not see unwritten blocks and concurrent writes do
  not race with rollback, while keeping non-overlapping async direct
  writes parallel.

The tracking state is allocated on the first native O_DIRECT write of an
inode and freed with the inode, so f2fs_inode_info only grows by one
pointer, and extent entries are allocated lazily with __GFP_NOFAIL only
when blocks are actually allocated.

Fixes: d4dd19ec1ea0 ("f2fs: do not expose unwritten blocks to user by DIO")
Fixes: a1e09b03e6f5 ("f2fs: use iomap for direct I/O")
Fixes: f847c699cff3 ("f2fs: allow out-place-update for direct IO in LFS mode")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/data.c  |   6 +-
 fs/f2fs/f2fs.h  |  44 ++++++
 fs/f2fs/file.c  | 392 +++++++++++++++++++++++++++++++++++++++++++++++-
 fs/f2fs/super.c |   5 +
 4 files changed, 438 insertions(+), 9 deletions(-)

diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 83c414cac252..4d7731fbf5a4 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -2087,6 +2087,7 @@ static int __allocate_data_block(struct dnode_of_data *dn, int seg_type)
 	if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO)
 		f2fs_invalidate_internal_cache(sbi, old_blkaddr, 1);
 
+	f2fs_record_dio_block(dn, old_blkaddr);
 	f2fs_update_data_blkaddr(dn, dn->data_blkaddr);
 	return 0;
 }
@@ -2339,7 +2340,7 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag)
 		}
 
 		blkaddr = dn.data_blkaddr;
-		if (is_hole)
+		if (is_hole || flag == F2FS_GET_BLOCK_DIO)
 			map->m_flags |= F2FS_MAP_NEW;
 	} else if (is_hole) {
 		if (f2fs_compressed_file(inode) &&
@@ -6273,7 +6274,8 @@ static int f2fs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
 		iomap->addr = IOMAP_NULL_ADDR;
 	}
 
-	if (map.m_flags & F2FS_MAP_NEW)
+	if ((map.m_flags & F2FS_MAP_NEW) ||
+	    ((flags & IOMAP_WRITE) && f2fs_dio_has_cur(inode)))
 		iomap->flags |= IOMAP_F_NEW;
 	if ((inode_state_read_once(inode) & I_DIRTY_DATASYNC) ||
 	    offset + length > i_size_read(inode))
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 7c36b996a09c..f9bdf681f29e 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -970,6 +970,38 @@ struct f2fs_dio_bufio {
 	bool active;			/* record write failures */
 };
 
+/*
+ * Contiguous run of blocks allocated by a native O_DIRECT write, recorded so
+ * that unwritten blocks can be rolled back if the write fails or stops short.
+ */
+struct f2fs_dio_ext {
+	struct list_head list;		/* in f2fs_dio_write->extents */
+	unsigned long long cp_ver;	/* checkpoint version of the update */
+	pgoff_t fofs;			/* first file block offset */
+	unsigned int len;		/* number of blocks */
+	block_t old_blkaddr;		/* NULL_ADDR, NEW_ADDR, or first old block */
+	block_t new_blkaddr;		/* first allocated block */
+};
+
+struct f2fs_dio_write {
+	struct list_head list;		/* in f2fs_dio_track->inflight */
+	struct kiocb *iocb;		/* owning kiocb */
+	loff_t pos;			/* starting byte offset of the write */
+	pgoff_t min_fofs;		/* first block offset in extents */
+	pgoff_t max_end;		/* end block offset in extents */
+	bool completing;		/* completion in progress */
+	struct list_head extents;	/* f2fs_dio_ext entries, in fofs order */
+};
+
+struct f2fs_dio_track {
+	spinlock_t lock;		/* protects active_*, cur and inflight */
+	atomic_t nr_inflight;		/* number of entries in inflight */
+	struct kiocb *active_iocb;	/* kiocb mapping blocks under inode_lock */
+	loff_t active_pos;		/* starting offset of active_iocb */
+	struct f2fs_dio_write *cur;	/* entry of active_iocb, if any */
+	struct list_head inflight;	/* in-flight DIO writes with allocations */
+};
+
 struct f2fs_inode_info {
 	struct inode vfs_inode;		/* serve a vfs inode */
 	unsigned long i_flags;		/* keep an inode flags for ioctl */
@@ -1040,6 +1072,8 @@ struct f2fs_inode_info {
 
 	/* allocated by the first O_DIRECT write falling back to buffered I/O */
 	struct f2fs_dio_bufio *dio_bufio;
+	/* allocated by the first native O_DIRECT write */
+	struct f2fs_dio_track *dio_track;
 
 	unsigned int atomic_write_cnt;
 	loff_t original_i_size;		/* original i_size before atomic write */
@@ -3912,6 +3946,14 @@ static inline bool f2fs_should_track_dio_opu(struct inode *inode,
 	       start_idx <= READ_ONCE(b->end);
 }
 
+static inline bool f2fs_has_dio_alloc(struct inode *inode)
+{
+	/* Pairs with smp_store_release() in f2fs_start_dio_write(). */
+	struct f2fs_dio_track *t = smp_load_acquire(&F2FS_I(inode)->dio_track);
+
+	return t && atomic_read(&t->nr_inflight) > 0;
+}
+
 static inline int is_file(struct inode *inode, int type)
 {
 	return F2FS_I(inode)->i_advise & type;
@@ -4527,6 +4569,8 @@ void f2fs_end_dio_opu_track(struct inode *inode,
 			    bool submitted, bool failed);
 void f2fs_mark_dio_opu_failed(struct inode *inode, pgoff_t index);
 void f2fs_mark_dio_opu_superseded(struct inode *inode, pgoff_t index);
+void f2fs_record_dio_block(struct dnode_of_data *dn, block_t old_blkaddr);
+bool f2fs_dio_has_cur(struct inode *inode);
 int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map, int flag);
 int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 			u64 start, u64 len);
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 70ba3f535211..36a9aeabf23e 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -25,6 +25,7 @@
 #include <linux/fileattr.h>
 #include <linux/fadvise.h>
 #include <linux/iomap.h>
+#include <linux/overflow.h>
 
 #include "f2fs.h"
 #include "node.h"
@@ -5403,6 +5404,75 @@ static void f2fs_trace_rw_file_path(struct file *file, loff_t pos, size_t count,
 	f2fs_putname(buf);
 }
 
+static bool f2fs_dio_alloc_overlaps(struct inode *inode, pgoff_t start,
+				    pgoff_t end)
+{
+	struct f2fs_dio_track *t = F2FS_I(inode)->dio_track;
+	struct f2fs_dio_write *dw;
+	unsigned long flags;
+	bool found = false;
+
+	if (likely(!t || !atomic_read(&t->nr_inflight)))
+		return false;
+
+	spin_lock_irqsave(&t->lock, flags);
+	list_for_each_entry(dw, &t->inflight, list) {
+		struct f2fs_dio_ext *ext;
+
+		if (end <= dw->min_fofs || start >= dw->max_end)
+			continue;
+		list_for_each_entry(ext, &dw->extents, list) {
+			if (end > ext->fofs && start < ext->fofs + ext->len) {
+				found = true;
+				goto out;
+			}
+		}
+	}
+out:
+	spin_unlock_irqrestore(&t->lock, flags);
+	return found;
+}
+
+/*
+ * Wait for any in-flight allocating O_DIRECT write overlapping [@pos, @pos +
+ * @count) to complete (and roll back if it failed), so reads do not see
+ * unwritten blocks and concurrent writes do not race with rollback.
+ */
+static int f2fs_wait_dio_alloc_range(struct inode *inode, loff_t pos,
+				     size_t count, bool nowait)
+{
+	struct f2fs_sb_info *sbi;
+	struct f2fs_dio_track *t;
+	u64 start_blk;
+	loff_t end_pos;
+	pgoff_t start, end;
+
+	if (likely(!f2fs_has_dio_alloc(inode)) || !count || pos < 0)
+		return 0;
+
+	sbi = F2FS_I_SB(inode);
+	start_blk = F2FS_BYTES_TO_BLK(sbi, pos);
+	if (start_blk >= ULONG_MAX)
+		return 0;
+	start = start_blk;
+	if (check_add_overflow(pos, (loff_t)count - 1, &end_pos))
+		end = ULONG_MAX;
+	else
+		end = min_t(u64, (u64)F2FS_BYTES_TO_BLK(sbi, end_pos) + 1,
+			    ULONG_MAX);
+
+	if (!f2fs_dio_alloc_overlaps(inode, start, end))
+		return 0;
+	if (nowait)
+		return -EAGAIN;
+
+	t = F2FS_I(inode)->dio_track;
+	/* Pairs with wake_up_var() in f2fs_finish_dio_write(). */
+	wait_var_event(&t->nr_inflight,
+		       !f2fs_dio_alloc_overlaps(inode, start, end));
+	return 0;
+}
+
 static ssize_t f2fs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 {
 	struct inode *inode = file_inode(iocb->ki_filp);
@@ -5413,6 +5483,11 @@ static ssize_t f2fs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	if (!f2fs_is_compress_backend_ready(inode))
 		return -EOPNOTSUPP;
 
+	ret = f2fs_wait_dio_alloc_range(inode, pos, iov_iter_count(to),
+					iocb->ki_flags & IOCB_NOWAIT);
+	if (ret)
+		return ret;
+
 	if (trace_f2fs_dataread_start_enabled())
 		f2fs_trace_rw_file_path(iocb->ki_filp, iocb->ki_pos,
 					iov_iter_count(to), READ);
@@ -5448,6 +5523,10 @@ static ssize_t f2fs_file_splice_read(struct file *in, loff_t *ppos,
 	if (!f2fs_is_compress_backend_ready(inode))
 		return -EOPNOTSUPP;
 
+	ret = f2fs_wait_dio_alloc_range(inode, pos, len, false);
+	if (ret)
+		return ret;
+
 	if (trace_f2fs_dataread_start_enabled())
 		f2fs_trace_rw_file_path(in, pos, len, READ);
 
@@ -5510,12 +5589,14 @@ static int f2fs_preallocate_blocks(struct kiocb *iocb, struct iov_iter *iter,
 	if (dio && f2fs_lfs_mode(sbi))
 		return 0;
 	/*
-	 * Don't preallocate holes aligned to DIO_SKIP_HOLES which turns into
-	 * buffered IO, if DIO meets any holes.
+	 * Don't preallocate holes inside i_size, nor past i_size of KEEP_SIZE
+	 * files where post-EOF truncation on error would also drop blocks
+	 * reserved by fallocate().
 	 */
-	if (dio && i_size_read(inode) &&
-		(F2FS_BYTES_TO_BLK(sbi, pos) <
-		 F2FS_BLK_ALIGN(sbi, i_size_read(inode))))
+	if (dio && (file_keep_isize(inode) ||
+		    (i_size_read(inode) &&
+		     (F2FS_BYTES_TO_BLK(sbi, pos) <
+		      F2FS_BLK_ALIGN(sbi, i_size_read(inode))))))
 		return 0;
 
 	/* No-wait I/O can't allocate blocks. */
@@ -5593,11 +5674,289 @@ static ssize_t f2fs_buffered_write_iter(struct kiocb *iocb,
 	return ret;
 }
 
+static void f2fs_start_dio_write(struct inode *inode, struct kiocb *iocb,
+				 loff_t pos)
+{
+	struct f2fs_inode_info *fi = F2FS_I(inode);
+	struct f2fs_dio_track *t = fi->dio_track;
+	unsigned long flags;
+
+	if (!t) {
+		t = kzalloc_obj(*t, GFP_NOFS | __GFP_NOFAIL);
+		spin_lock_init(&t->lock);
+		atomic_set(&t->nr_inflight, 0);
+		INIT_LIST_HEAD(&t->inflight);
+		/* Pairs with smp_load_acquire() in f2fs_has_dio_alloc(). */
+		smp_store_release(&fi->dio_track, t);
+	}
+
+	spin_lock_irqsave(&t->lock, flags);
+	t->active_iocb = iocb;
+	t->active_pos = pos;
+	t->cur = NULL;
+	spin_unlock_irqrestore(&t->lock, flags);
+}
+
+static void f2fs_end_dio_write(struct inode *inode)
+{
+	struct f2fs_dio_track *t = F2FS_I(inode)->dio_track;
+	unsigned long flags;
+
+	if (!t || !t->active_iocb)
+		return;
+
+	spin_lock_irqsave(&t->lock, flags);
+	t->active_iocb = NULL;
+	t->cur = NULL;
+	spin_unlock_irqrestore(&t->lock, flags);
+}
+
+bool f2fs_dio_has_cur(struct inode *inode)
+{
+	struct f2fs_dio_track *t = F2FS_I(inode)->dio_track;
+
+	return t && READ_ONCE(t->cur);
+}
+
+/*
+ * Record a block allocated by a native O_DIRECT write so it can be rolled back
+ * if the write fails or stops short. The caller holds inode_lock() and
+ * f2fs_lock_op().
+ */
+void f2fs_record_dio_block(struct dnode_of_data *dn, block_t old_blkaddr)
+{
+	struct inode *inode = dn->inode;
+	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+	struct f2fs_dio_track *t = F2FS_I(inode)->dio_track;
+	struct f2fs_dio_write *dw;
+	struct f2fs_dio_ext *ext, *last;
+	unsigned long long cp_ver;
+	block_t new_blkaddr = dn->data_blkaddr;
+	pgoff_t fofs;
+	unsigned long flags;
+
+	if (!t || !t->active_iocb)
+		return;
+
+	fofs = f2fs_start_bidx_of_node(ofs_of_node(sbi, dn->node_entry),
+				       inode) + dn->ofs_in_node;
+	cp_ver = cur_cp_version(F2FS_CKPT(sbi));
+	dw = t->cur;
+	if (dw) {
+		bool can_merge;
+
+		last = list_last_entry(&dw->extents, struct f2fs_dio_ext, list);
+		can_merge = last->cp_ver == cp_ver &&
+			    last->fofs + last->len == fofs &&
+			    last->new_blkaddr + last->len == new_blkaddr &&
+			    (__is_valid_data_blkaddr(old_blkaddr) ?
+			     (__is_valid_data_blkaddr(last->old_blkaddr) &&
+			      last->old_blkaddr + last->len == old_blkaddr) :
+			     last->old_blkaddr == old_blkaddr);
+		if (can_merge) {
+			spin_lock_irqsave(&t->lock, flags);
+			last->len++;
+			if (fofs + 1 > dw->max_end)
+				dw->max_end = fofs + 1;
+			spin_unlock_irqrestore(&t->lock, flags);
+			return;
+		}
+	} else {
+		dw = kzalloc_obj(*dw, GFP_NOFS | __GFP_NOFAIL);
+		dw->iocb = t->active_iocb;
+		dw->pos = t->active_pos;
+		dw->min_fofs = fofs;
+		dw->max_end = fofs + 1;
+		INIT_LIST_HEAD(&dw->extents);
+	}
+
+	ext = kzalloc_obj(*ext, GFP_NOFS | __GFP_NOFAIL);
+	ext->cp_ver = cp_ver;
+	ext->fofs = fofs;
+	ext->len = 1;
+	ext->old_blkaddr = old_blkaddr;
+	ext->new_blkaddr = new_blkaddr;
+
+	spin_lock_irqsave(&t->lock, flags);
+	list_add_tail(&ext->list, &dw->extents);
+	if (fofs < dw->min_fofs)
+		dw->min_fofs = fofs;
+	if (fofs + 1 > dw->max_end)
+		dw->max_end = fofs + 1;
+	if (!t->cur) {
+		WRITE_ONCE(t->cur, dw);
+		list_add_tail(&dw->list, &t->inflight);
+		atomic_inc(&t->nr_inflight);
+	}
+	spin_unlock_irqrestore(&t->lock, flags);
+}
+
+static void f2fs_rollback_dio_ext(struct inode *inode, pgoff_t fofs,
+				  unsigned int len, block_t old_blk,
+				  block_t new_blk, unsigned long long cp_ver)
+{
+	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+	bool old_valid = __is_valid_data_blkaddr(old_blk);
+	bool same_cp = cur_cp_version(F2FS_CKPT(sbi)) == cp_ver;
+
+	while (len) {
+		struct dnode_of_data dn;
+		unsigned int end_offset;
+		blkcnt_t freed = 0;
+		int err;
+
+		set_new_dnode(&dn, inode, NULL, NULL, 0);
+		err = f2fs_get_dnode_of_data(&dn, fofs, LOOKUP_NODE);
+		if (err == -ENOENT) {
+			pgoff_t next = f2fs_get_next_page_offset(&dn, fofs);
+			unsigned int step = min_t(pgoff_t, len,
+						  max_t(pgoff_t, 1,
+							next - fofs));
+
+			fofs += step;
+			new_blk += step;
+			if (old_valid)
+				old_blk += step;
+			len -= step;
+			continue;
+		}
+		if (err) {
+			f2fs_warn_ratelimited(sbi,
+				"%s: failed to roll back direct write, ino:%llu, index:%lu, err:%d",
+				__func__, inode->i_ino, fofs, err);
+			set_sbi_flag(sbi, SBI_NEED_FSCK);
+			return;
+		}
+
+		end_offset = ADDRS_PER_PAGE(dn.node_entry, inode);
+		for (; len && dn.ofs_in_node < end_offset;
+		     dn.ofs_in_node++, fofs++, new_blk++, len--) {
+			block_t cur_old = old_blk;
+
+			if (old_valid)
+				old_blk++;
+			if (f2fs_data_blkaddr(&dn) != new_blk)
+				continue;
+
+			if (cur_old == NULL_ADDR) {
+				f2fs_invalidate_blocks(sbi, new_blk, 1);
+				f2fs_update_data_blkaddr(&dn, NULL_ADDR);
+				freed++;
+			} else if (cur_old == NEW_ADDR) {
+				f2fs_invalidate_blocks(sbi, new_blk, 1);
+				f2fs_update_data_blkaddr(&dn, NEW_ADDR);
+			} else if (same_cp &&
+				   f2fs_is_checkpointed_data(sbi, cur_old)) {
+				f2fs_update_data_blkaddr(&dn, cur_old);
+				if (new_blk != cur_old) {
+					f2fs_invalidate_blocks(sbi, new_blk, 1);
+					f2fs_revalidate_blocks(sbi, cur_old, 1);
+				}
+			} else {
+				f2fs_invalidate_blocks(sbi, new_blk, 1);
+				f2fs_update_data_blkaddr(&dn, NEW_ADDR);
+			}
+		}
+		if (freed)
+			dec_valid_block_count(sbi, inode, freed);
+		f2fs_put_dnode(&dn);
+	}
+}
+
+static void f2fs_finish_dio_write(struct inode *inode, struct kiocb *iocb,
+				  ssize_t size, int error)
+{
+	struct f2fs_dio_track *t = F2FS_I(inode)->dio_track;
+	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+	struct f2fs_dio_write *cur, *dw = NULL;
+	struct f2fs_dio_ext *ext, *tmp, *last;
+	ssize_t written = error ? 0 : size;
+	pgoff_t end_fofs = 0;
+	unsigned long flags;
+	bool need_rollback;
+
+	if (likely(!t || !atomic_read(&t->nr_inflight)))
+		return;
+
+	spin_lock_irqsave(&t->lock, flags);
+	list_for_each_entry(cur, &t->inflight, list) {
+		if (cur->iocb == iocb && !cur->completing) {
+			dw = cur;
+			break;
+		}
+	}
+	if (!dw) {
+		spin_unlock_irqrestore(&t->lock, flags);
+		return;
+	}
+
+	if (t->cur == dw)
+		WRITE_ONCE(t->cur, NULL);
+	if (written > 0)
+		end_fofs = F2FS_BLK_ALIGN(sbi, dw->pos + written);
+	last = list_last_entry_or_null(&dw->extents, struct f2fs_dio_ext, list);
+	need_rollback = last &&
+			(written <= 0 || last->fofs + last->len > end_fofs);
+	if (need_rollback) {
+		dw->completing = true;
+	} else {
+		list_del_init(&dw->list);
+		atomic_dec(&t->nr_inflight);
+	}
+	spin_unlock_irqrestore(&t->lock, flags);
+
+	if (need_rollback) {
+		struct f2fs_lock_context lc;
+
+		clear_inode_flag(inode, FI_PREALLOCATED_ALL);
+		f2fs_lock_op(sbi, &lc);
+		list_for_each_entry(ext, &dw->extents, list) {
+			pgoff_t fofs = ext->fofs;
+			unsigned int len = ext->len;
+			block_t old_blk = ext->old_blkaddr;
+			block_t new_blk = ext->new_blkaddr;
+
+			if (written > 0) {
+				if (fofs + len <= end_fofs)
+					continue;
+				if (fofs < end_fofs) {
+					unsigned int done = end_fofs - fofs;
+
+					fofs += done;
+					new_blk += done;
+					if (__is_valid_data_blkaddr(old_blk))
+						old_blk += done;
+					len -= done;
+				}
+			}
+			f2fs_rollback_dio_ext(inode, fofs, len, old_blk,
+					      new_blk, ext->cp_ver);
+		}
+		f2fs_unlock_op(sbi, &lc);
+
+		spin_lock_irqsave(&t->lock, flags);
+		list_del_init(&dw->list);
+		atomic_dec(&t->nr_inflight);
+		spin_unlock_irqrestore(&t->lock, flags);
+	}
+
+	/* Pairs with wait_var_event() in f2fs_wait_dio_alloc_range(). */
+	wake_up_var(&t->nr_inflight);
+
+	list_for_each_entry_safe(ext, tmp, &dw->extents, list) {
+		list_del(&ext->list);
+		kfree(ext);
+	}
+	kfree(dw);
+}
+
 static int f2fs_dio_write_end_io(struct kiocb *iocb, ssize_t size, int error,
 				 unsigned int flags)
 {
-	struct f2fs_sb_info *sbi = F2FS_I_SB(file_inode(iocb->ki_filp));
+	struct inode *inode = file_inode(iocb->ki_filp);
+	struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
 
+	f2fs_finish_dio_write(inode, iocb, size, error);
 	dec_cache_count(sbi, F2FS_DIO_WRITE);
 	if (error)
 		return error;
@@ -5965,12 +6324,15 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 		dio_flags |= IOMAP_DIO_FORCE_WAIT;
 	dio = __iomap_dio_rw(iocb, from, &f2fs_iomap_ops,
 			     &f2fs_iomap_dio_write_ops, dio_flags, iocb, 0);
+	f2fs_end_dio_write(inode);
 	if (IS_ERR_OR_NULL(dio)) {
 		ret = PTR_ERR_OR_ZERO(dio);
 		if (ret == -ENOTBLK)
 			ret = 0;
-		if (ret != -EIOCBQUEUED)
+		if (ret != -EIOCBQUEUED) {
+			f2fs_finish_dio_write(inode, iocb, 0, -ECANCELED);
 			dec_cache_count(sbi, F2FS_DIO_WRITE);
+		}
 	} else {
 		ret = iomap_dio_complete(dio);
 	}
@@ -6053,6 +6415,9 @@ static ssize_t f2fs_dio_write_iter(struct kiocb *iocb, struct iov_iter *from,
 		*may_need_sync = false;
 	}
 out:
+	f2fs_end_dio_write(inode);
+	if (ret != -EIOCBQUEUED)
+		f2fs_finish_dio_write(inode, iocb, 0, -ECANCELED);
 	trace_f2fs_direct_IO_exit(inode, pos, count, WRITE, ret);
 	return ret;
 }
@@ -6098,6 +6463,12 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		goto out_unlock;
 	}
 
+	ret = f2fs_wait_dio_alloc_range(inode, iocb->ki_pos,
+					iov_iter_count(from),
+					iocb->ki_flags & IOCB_NOWAIT);
+	if (ret)
+		goto out_unlock;
+
 	bufio_start_pos = iocb->ki_pos;
 
 	/* Determine whether we will do a direct write or a buffered write. */
@@ -6109,10 +6480,17 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
 		goto out_unlock;
 	}
 
+	if (dio)
+		f2fs_start_dio_write(inode, iocb, iocb->ki_pos);
+
 	/* Possibly preallocate the blocks for the write. */
 	target_size = iocb->ki_pos + iov_iter_count(from);
 	preallocated = f2fs_preallocate_blocks(iocb, from, dio);
 	if (preallocated < 0) {
+		if (dio) {
+			f2fs_end_dio_write(inode);
+			f2fs_finish_dio_write(inode, iocb, 0, preallocated);
+		}
 		ret = preallocated;
 	} else {
 		/* Buffered write fails right away for NOWAIT. */
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index cce6bac1cda1..3bd9f2ab08b9 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -1981,11 +1981,16 @@ static void f2fs_dirty_inode(struct inode *inode, int flags)
 static void f2fs_free_inode(struct inode *inode)
 {
 	struct f2fs_dio_bufio *b = F2FS_I(inode)->dio_bufio;
+	struct f2fs_dio_track *t = F2FS_I(inode)->dio_track;
 
 	if (b) {
 		xa_destroy(&b->entries);
 		kfree(b);
 	}
+	if (t) {
+		WARN_ON_ONCE(!list_empty(&t->inflight));
+		kfree(t);
+	}
 	fscrypt_free_inode(inode);
 	kmem_cache_free(f2fs_inode_cachep, F2FS_I(inode));
 }
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 8/9] f2fs: compress: fix cleanup when a cluster write fails to be submitted

[Jan Prusakowski]

When f2fs_write_compressed_pages() fails to submit a compressed page in
the middle of a cluster, it gives up and returns -EAGAIN, and
f2fs_write_multi_pages() writes the cluster with raw pages instead. But
it leaves two things behind:

- cc->cpages[i - 1] is set to NULL before the submission is checked, so
  the compressed page that was not submitted is not freed by the out_free
  loop, and leaks from the compress page pool.

- The dnode already points to the new blocks of the slots before i, and
  their old blocks were invalidated, while the slots from i on still
  point to the old blocks. The cluster then mixes new and old compressed
  data, and f2fs_write_raw_pages() sees it in that state.

Clear cc->cpages[i - 1] only after the page was submitted, and restore
the old block addresses of the slots before i, validating the old blocks
again. That's safe because they were invalidated in the same
f2fs_lock_op() section, unless they are not valid in the last checkpoint
(checkpoint=disable), where SSR may already have reused them; leave the
cluster as it is then. The old block addresses are kept on the stack for
clusters of up to 16 blocks, and in a __GFP_NOFAIL allocation of at most
1 KiB otherwise.

Also, when a raw page overwrites a compressed cluster, decrement
i_compr_blocks in f2fs_do_write_data_page() only if the page was
submitted, since the dnode keeps COMPRESS_ADDR otherwise.

Submission only fails when block allocation fails, which happens only
after the checkpoint was stopped, so all of this only keeps the
in-memory state consistent until unmount. Note that if the allocation
fails while opening a new segment, f2fs_allocate_data_block() has
already updated the SIT for the block it doesn't return, and that is
not undone here.

Fixes: 4c8ff7095bef ("f2fs: support data compression")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/compress.c | 61 +++++++++++++++++++++++++++++++++++++++++++---
 fs/f2fs/data.c     |  6 ++---
 2 files changed, 61 insertions(+), 6 deletions(-)

diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c
index c5758da7c621..0147a7e05022 100644
--- a/fs/f2fs/compress.c
+++ b/fs/f2fs/compress.c
@@ -1279,6 +1279,45 @@ int f2fs_truncate_partial_cluster(struct inode *inode, u64 from, bool lock)
 	return f2fs_do_truncate_blocks(inode, round_up(from, PAGE_SIZE), lock);
 }
 
+/*
+ * Undo the block address updates of the first @nr slots of a cluster, after a
+ * write of the cluster failed to be submitted in the middle. Submission only
+ * fails if block allocation fails, i.e. after the checkpoint was stopped, so
+ * this only keeps the in-memory state consistent until unmount.
+ *
+ * The old blocks were invalidated in the same f2fs_lock_op() section, so they
+ * can be validated again, unless SSR could have reused them because they are
+ * not valid in the last checkpoint (checkpoint=disable). Leave the cluster as
+ * it is then.
+ */
+static void f2fs_undo_cluster_update(struct dnode_of_data *dn,
+				     block_t *old_blkaddr, int nr)
+{
+	struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
+	int i;
+
+	for (i = 0; i < nr; i++) {
+		if (!f2fs_is_checkpointed_data(sbi, old_blkaddr[i]))
+			return;
+	}
+
+	for (i = 0; i < nr; i++, dn->ofs_in_node++) {
+		block_t old_blk = old_blkaddr[i];
+		block_t new_blk = f2fs_data_blkaddr(dn);
+
+		if (new_blk == old_blk)
+			continue;
+		f2fs_update_data_blkaddr(dn, old_blk);
+		if (__is_valid_data_blkaddr(new_blk))
+			f2fs_invalidate_blocks(sbi, new_blk, 1);
+		if (__is_valid_data_blkaddr(old_blk))
+			f2fs_revalidate_blocks(sbi, old_blk, 1);
+	}
+}
+
+/* Old block addresses of clusters up to this size are kept on the stack. */
+#define F2FS_ONSTACK_CLUSTER_SIZE	16
+
 static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 					int *submitted,
 					struct writeback_control *wbc,
@@ -1306,6 +1345,8 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 	struct node_info ni;
 	struct compress_io_ctx *cic;
 	struct f2fs_lock_context lc;
+	block_t old_blkaddr_onstack[F2FS_ONSTACK_CLUSTER_SIZE];
+	block_t *old_blkaddr = old_blkaddr_onstack;
 	pgoff_t start_idx = start_idx_of_cluster(cc);
 	unsigned int last_index = cc->cluster_size - 1;
 	loff_t psize;
@@ -1363,6 +1404,12 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 
 	cic->nr_rpages = cc->cluster_size;
 
+	if (cc->cluster_size > ARRAY_SIZE(old_blkaddr_onstack)) {
+		/* At most 1 KiB, don't fail the write for it. */
+		old_blkaddr = kmalloc_array(cc->cluster_size, sizeof(block_t),
+					    GFP_NOFS | __GFP_NOFAIL);
+	}
+
 	for (i = 0; i < cc->valid_nr_cpages; i++) {
 		f2fs_set_compressed_page(cc->cpages[i], inode,
 				page_folio(cc->rpages[i + 1])->index, cic);
@@ -1386,6 +1433,7 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 		blkaddr = f2fs_data_blkaddr(&dn);
 		fio.folio = page_folio(cc->rpages[i]);
 		fio.old_blkaddr = blkaddr;
+		old_blkaddr[i] = blkaddr;
 
 		/* cluster header */
 		if (i == 0) {
@@ -1412,14 +1460,16 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 
 		fio.compressed_page = cc->cpages[i - 1];
 
-		cc->cpages[i - 1] = NULL;
 		fio.submitted = 0;
 		f2fs_outplace_write_data(&dn, &fio);
 		if (unlikely(!fio.submitted)) {
 			cancel_cluster_writeback(cc, cic, i);
 			*submitted = 0;
-			goto out_free_page_array;
+			dn.ofs_in_node -= i;
+			f2fs_undo_cluster_update(&dn, old_blkaddr, i);
+			goto out_submit_err;
 		}
+		cc->cpages[i - 1] = NULL;
 		(*submitted)++;
 unlock_continue:
 		inode_dec_dirty_block(cc->inode);
@@ -1431,6 +1481,9 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 	f2fs_i_compr_blocks_update(inode, cc->valid_nr_cpages, true);
 	add_compr_block_stat(inode, cc->valid_nr_cpages);
 
+	if (old_blkaddr != old_blkaddr_onstack)
+		kfree(old_blkaddr);
+
 	set_inode_flag(cc->inode, FI_APPEND_WRITE);
 
 	f2fs_put_dnode(&dn);
@@ -1450,7 +1503,9 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 	f2fs_destroy_compress_ctx(cc, false);
 	return 0;
 
-out_free_page_array:
+out_submit_err:
+	if (old_blkaddr != old_blkaddr_onstack)
+		kfree(old_blkaddr);
 	page_array_free(sbi, cic->rpages, cc->cluster_size);
 out_put_cic:
 	kmem_cache_free(cic_entry_slab, cic);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 4d7731fbf5a4..016aa347880e 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -4018,11 +4018,11 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
 		pending_added = true;
 	}
 
-	if (fio->compr_blocks && fio->old_blkaddr == COMPRESS_ADDR)
-		f2fs_i_compr_blocks_update(inode, fio->compr_blocks - 1, false);
-
 	/* LFS mode write path */
 	f2fs_outplace_write_data(&dn, fio);
+	if (fio->compr_blocks && fio->old_blkaddr == COMPRESS_ADDR &&
+	    likely(fio->submitted))
+		f2fs_i_compr_blocks_update(inode, fio->compr_blocks - 1, false);
 	if (unlikely(opu_entry)) {
 		opu_entry->new_blkaddr[0] = fio->new_blkaddr;
 		f2fs_end_dio_opu_track(inode, opu_entry, fio->submitted, false);
-- 
2.56.0.567.gdb80ae8cd5-goog

[PATCH RESEND 9/9] f2fs: compress: roll back failed cluster writes of O_DIRECT buffered fallback

[Jan Prusakowski]

O_DIRECT writes to compressed files always fall back to buffered I/O.
The out-of-place updates done by writeback of compressed files don't go
through f2fs_do_write_data_page(), so they are not tracked for rollback
by the buffered fallback, and a failed write (e.g. with dm-error in
generic/250 and generic/252) leaves the dnode pointing to blocks that
were never written:

- f2fs_write_compressed_pages() points the slots of the cluster to new
  blocks, sets the header to COMPRESS_ADDR and invalidates the old
  blocks before the bios complete.

- f2fs_write_raw_pages(), when overwriting a compressed cluster, writes
  the pages with compr_blocks set, which f2fs_do_write_data_page() does
  not track, and also changes i_compr_blocks.

Track both as one entry per cluster:

- f2fs_write_compressed_pages() records the old and new block addresses
  of all slots, and the number of compressed blocks of the old and new
  cluster, so that f2fs_rollback_dio_opu_entry() can restore
  i_compr_blocks as well. f2fs_compress_write_end_io() marks the entry
  failed on I/O error; compressed pages carry the index of a page of the
  cluster.

- f2fs_write_raw_pages() records the block addresses of the cluster
  before and after writing it. An error other than a failed I/O can
  leave the cluster partially overwritten, so the entry is marked failed
  then too, to restore the old cluster.

Quota files are not tracked, they are written by the kernel, not with
O_DIRECT.

Fixes: 4c8ff7095bef ("f2fs: support data compression")
Assisted-by: LLM
Signed-off-by: Jan Prusakowski <jprusakowski@google.com>
---
 fs/f2fs/compress.c | 97 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 92 insertions(+), 5 deletions(-)

diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c
index 0147a7e05022..8aed34dcc89e 100644
--- a/fs/f2fs/compress.c
+++ b/fs/f2fs/compress.c
@@ -1345,8 +1345,10 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 	struct node_info ni;
 	struct compress_io_ctx *cic;
 	struct f2fs_lock_context lc;
+	struct f2fs_dio_opu_entry *opu_entry = NULL;
 	block_t old_blkaddr_onstack[F2FS_ONSTACK_CLUSTER_SIZE];
 	block_t *old_blkaddr = old_blkaddr_onstack;
+	block_t *new_blkaddr = NULL;
 	pgoff_t start_idx = start_idx_of_cluster(cc);
 	unsigned int last_index = cc->cluster_size - 1;
 	loff_t psize;
@@ -1404,7 +1406,15 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 
 	cic->nr_rpages = cc->cluster_size;
 
-	if (cc->cluster_size > ARRAY_SIZE(old_blkaddr_onstack)) {
+	if (!quota_inode &&
+	    unlikely(f2fs_should_track_dio_opu(inode, start_idx,
+					       cc->cluster_size)))
+		opu_entry = f2fs_start_dio_opu_track(inode, start_idx,
+						     cc->cluster_size);
+	if (opu_entry) {
+		old_blkaddr = opu_entry->old_blkaddr;
+		new_blkaddr = opu_entry->new_blkaddr;
+	} else if (cc->cluster_size > ARRAY_SIZE(old_blkaddr_onstack)) {
 		/* At most 1 KiB, don't fail the write for it. */
 		old_blkaddr = kmalloc_array(cc->cluster_size, sizeof(block_t),
 					    GFP_NOFS | __GFP_NOFAIL);
@@ -1472,6 +1482,8 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 		cc->cpages[i - 1] = NULL;
 		(*submitted)++;
 unlock_continue:
+		if (new_blkaddr)
+			new_blkaddr[i] = f2fs_data_blkaddr(&dn);
 		inode_dec_dirty_block(cc->inode);
 		folio_unlock(fio.folio);
 	}
@@ -1481,8 +1493,13 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 	f2fs_i_compr_blocks_update(inode, cc->valid_nr_cpages, true);
 	add_compr_block_stat(inode, cc->valid_nr_cpages);
 
-	if (old_blkaddr != old_blkaddr_onstack)
+	if (unlikely(opu_entry)) {
+		opu_entry->old_compr_blocks = fio.compr_blocks;
+		opu_entry->new_compr_blocks = 1 + cc->valid_nr_cpages;
+		f2fs_end_dio_opu_track(inode, opu_entry, true, false);
+	} else if (old_blkaddr != old_blkaddr_onstack) {
 		kfree(old_blkaddr);
+	}
 
 	set_inode_flag(cc->inode, FI_APPEND_WRITE);
 
@@ -1504,7 +1521,9 @@ static int f2fs_write_compressed_pages(struct compress_ctx *cc,
 	return 0;
 
 out_submit_err:
-	if (old_blkaddr != old_blkaddr_onstack)
+	if (unlikely(opu_entry))
+		f2fs_end_dio_opu_track(inode, opu_entry, false, false);
+	else if (old_blkaddr != old_blkaddr_onstack)
 		kfree(old_blkaddr);
 	page_array_free(sbi, cic->rpages, cc->cluster_size);
 out_put_cic:
@@ -1534,8 +1553,10 @@ void f2fs_compress_write_end_io(struct bio *bio, struct folio *folio)
 	enum count_type type = WB_DATA_TYPE(folio, true);
 	int i;
 
-	if (unlikely(bio->bi_status != BLK_STS_OK))
+	if (unlikely(bio->bi_status != BLK_STS_OK)) {
 		mapping_set_error(cic->inode->i_mapping, -EIO);
+		f2fs_mark_dio_opu_failed(cic->inode, folio->index);
+	}
 
 	f2fs_compress_free_page(page);
 
@@ -1565,6 +1586,58 @@ void f2fs_compress_write_end_io(struct bio *bio, struct folio *folio)
 	dec_cache_count(sbi, type);
 }
 
+/*
+ * Track the overwrite of a compressed cluster with raw pages during the
+ * buffered fallback of an O_DIRECT write. If the dnode can't be read, write
+ * without tracking rather than failing the write for it.
+ */
+static struct f2fs_dio_opu_entry *f2fs_track_raw_cluster(struct compress_ctx *cc,
+							 pgoff_t start_idx)
+{
+	struct f2fs_dio_opu_entry *entry;
+	struct dnode_of_data dn;
+	unsigned int i;
+
+	set_new_dnode(&dn, cc->inode, NULL, NULL, 0);
+	if (f2fs_get_dnode_of_data(&dn, start_idx, LOOKUP_NODE))
+		return NULL;
+
+	entry = f2fs_start_dio_opu_track(cc->inode, start_idx,
+					 cc->cluster_size);
+	for (i = 0; entry && i < cc->cluster_size; i++) {
+		entry->old_blkaddr[i] = data_blkaddr(dn.inode, dn.node_entry,
+						     dn.ofs_in_node + i);
+		entry->new_blkaddr[i] = entry->old_blkaddr[i];
+	}
+	f2fs_put_dnode(&dn);
+
+	return entry;
+}
+
+static void f2fs_end_raw_cluster_track(struct compress_ctx *cc,
+				       struct f2fs_dio_opu_entry *entry,
+				       int compr_blocks, bool failed)
+{
+	struct dnode_of_data dn;
+	unsigned int i;
+
+	set_new_dnode(&dn, cc->inode, NULL, NULL, 0);
+	if (!f2fs_get_dnode_of_data(&dn, entry->start_idx, LOOKUP_NODE)) {
+		for (i = 0; i < cc->cluster_size; i++, dn.ofs_in_node++)
+			entry->new_blkaddr[i] = f2fs_data_blkaddr(&dn);
+		f2fs_put_dnode(&dn);
+	} else {
+		failed = true;
+	}
+
+	/* The cluster is no longer compressed if its header was rewritten. */
+	if (entry->old_blkaddr[0] == COMPRESS_ADDR &&
+	    entry->new_blkaddr[0] != COMPRESS_ADDR)
+		entry->old_compr_blocks = compr_blocks;
+
+	f2fs_end_dio_opu_track(cc->inode, entry, true, failed);
+}
+
 static int f2fs_write_raw_pages(struct compress_ctx *cc,
 					int *submitted_p,
 					struct writeback_control *wbc,
@@ -1572,6 +1645,8 @@ static int f2fs_write_raw_pages(struct compress_ctx *cc,
 {
 	struct address_space *mapping = cc->inode->i_mapping;
 	struct f2fs_sb_info *sbi = F2FS_M_SB(mapping);
+	struct f2fs_dio_opu_entry *opu_entry = NULL;
+	pgoff_t start_idx = start_idx_of_cluster(cc);
 	struct f2fs_lock_context lc;
 	int submitted, compr_blocks, i;
 	int ret = 0;
@@ -1590,8 +1665,13 @@ static int f2fs_write_raw_pages(struct compress_ctx *cc,
 		return compr_blocks;
 
 	/* overwrite compressed cluster w/ normal cluster */
-	if (compr_blocks > 0)
+	if (compr_blocks > 0) {
 		f2fs_lock_op(sbi, &lc);
+		if (!IS_NOQUOTA(cc->inode) &&
+		    unlikely(f2fs_should_track_dio_opu(cc->inode, start_idx,
+						       cc->cluster_size)))
+			opu_entry = f2fs_track_raw_cluster(cc, start_idx);
+	}
 
 	for (i = 0; i < cc->cluster_size; i++) {
 		struct folio *folio;
@@ -1646,6 +1726,13 @@ static int f2fs_write_raw_pages(struct compress_ctx *cc,
 	}
 
 out:
+	/*
+	 * An error other than a failed I/O can leave the cluster partially
+	 * overwritten, so mark it failed too, to restore the old cluster.
+	 */
+	if (unlikely(opu_entry))
+		f2fs_end_raw_cluster_track(cc, opu_entry, compr_blocks,
+					   ret || f2fs_cp_error(sbi));
 	if (compr_blocks > 0)
 		f2fs_unlock_op(sbi, &lc);
 
-- 
2.56.0.567.gdb80ae8cd5-goog