* [RFC PATCH v4 01/12] xfs: reexpand iter to original count when upgrading to excl ILOCK
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 02/12] fs: Add counter to track inflight writes that need stable pages Ojaswin Mujoo
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
In xfs_file_write_checks() when we upgrade to an exclusive lock, we
correctly redo the write checks, but we should also reexpand the iter to
the original count incase it was trimmed. This is because a parallel
truncation/expansion, in the window where inode was unlocked, could have
changed the isize so its good to recalculate limits based on the original
count.
Also, s/count/orig_count/ while we are at it to better indicate its purpose
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/xfs/xfs_file.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index d8202da15aca..d53a329b6a58 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -450,7 +450,7 @@ xfs_file_write_checks(
struct xfs_zone_alloc_ctx *ac)
{
struct inode *inode = iocb->ki_filp->f_mapping->host;
- size_t count = iov_iter_count(from);
+ size_t orig_count = iov_iter_count(from);
bool drained_dio = false;
ssize_t error;
@@ -482,6 +482,7 @@ xfs_file_write_checks(
*iolock = 0;
return error;
}
+ iov_iter_reexpand(from, orig_count);
goto restart;
}
@@ -496,7 +497,7 @@ xfs_file_write_checks(
* the slow path when we are at or beyond the current EOF.
*/
if (iocb->ki_pos > i_size_read(inode)) {
- error = xfs_file_write_zero_eof(iocb, from, iolock, count,
+ error = xfs_file_write_zero_eof(iocb, from, iolock, orig_count,
&drained_dio, ac);
if (error == 1)
goto restart;
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 02/12] fs: Add counter to track inflight writes that need stable pages
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 01/12] xfs: reexpand iter to original count when upgrading to excl ILOCK Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 03/12] mm: Refactor folio_clear_dirty_for_io() Ojaswin Mujoo
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
The current flag-style stable write implementation uses idempotent set
and clear functions. This is okay because the users for the most part
just want to set or clear it once based on factors like underlying
device support.
However, this scheme doesn't play well when we have parallel users
wanting to temporarily set and unset stable writes. For example, the
upcoming RWF_WRITETHROUGH patches need stable writes to be enabled for
the duration of the IO. The current scheme can lead to bugs like:
RWF_WRITETHROUGH write 1 RWF_WRITETHROUGH write 2
enable stable write enable stable write
submit IO
disable stable write <---- WRONG
submit IO
disable stable write
The 2nd write loses the stable write guarantee midway which is not
correct. Fix this by introducing a new inflight_stable_write counter
which can be used by parallel users safely.
Unfortunately, due to the way the current users are designed, we cannot
directly migrate them to the counter approach hence for now we will
have to keep both methods till all the users adapt to the counters.
Suggested-by: "Darrick J. Wong" <djwong@kernel.org>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
include/linux/fs.h | 1 +
include/linux/pagemap.h | 14 +++++++++++++-
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 05d05bb9c951..f196f25255ab 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -482,6 +482,7 @@ struct address_space {
errseq_t wb_err;
spinlock_t i_private_lock;
struct rw_semaphore i_mmap_rwsem;
+ atomic_t inflight_stable_writes_count;
} __attribute__((aligned(sizeof(long)))) __randomize_layout;
/*
* On most architectures that alignment is already the case; but
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653..59713769f731 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -306,7 +306,8 @@ static inline void mapping_clear_release_always(struct address_space *mapping)
static inline bool mapping_stable_writes(const struct address_space *mapping)
{
- return test_bit(AS_STABLE_WRITES, &mapping->flags);
+ return test_bit(AS_STABLE_WRITES, &mapping->flags) ||
+ atomic_read(&mapping->inflight_stable_writes_count) > 0;
}
static inline void mapping_set_stable_writes(struct address_space *mapping)
@@ -319,6 +320,17 @@ static inline void mapping_clear_stable_writes(struct address_space *mapping)
clear_bit(AS_STABLE_WRITES, &mapping->flags);
}
+static inline void mapping_inc_inflight_stable_writes(struct address_space *mapping)
+{
+ atomic_inc(&mapping->inflight_stable_writes_count);
+}
+
+static inline void mapping_dec_inflight_stable_writes(struct address_space *mapping)
+{
+ WARN_ON_ONCE(atomic_dec_if_positive(
+ &mapping->inflight_stable_writes_count) < 0);
+}
+
static inline void mapping_set_inaccessible(struct address_space *mapping)
{
/*
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 03/12] mm: Refactor folio_clear_dirty_for_io()
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 01/12] xfs: reexpand iter to original count when upgrading to excl ILOCK Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 02/12] fs: Add counter to track inflight writes that need stable pages Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 04/12] iomap: Add helper to revert iomap iter Ojaswin Mujoo
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
Add a new __folio_clear_dirty_for_io() helper which takes an extra
parameter to indicate folio_mkclean() is needed. This is in preparation
of buffered writethrough support where we already do folio_mkclean()
before calling into this function.
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
mm/page-writeback.c | 39 +++++++++++++++++++++++++--------------
1 file changed, 25 insertions(+), 14 deletions(-)
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index eeab25d6ce36..2f8059ab55b9 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2850,20 +2850,12 @@ void __folio_cancel_dirty(struct folio *folio)
EXPORT_SYMBOL(__folio_cancel_dirty);
/*
- * Clear a folio's dirty flag, while caring for dirty memory accounting.
- * Returns true if the folio was previously dirty.
- *
- * This is for preparing to put the folio under writeout. We leave
- * the folio tagged as dirty in the xarray so that a concurrent
- * write-for-sync can discover it via a PAGECACHE_TAG_DIRTY walk.
- * The ->writepage implementation will run either folio_start_writeback()
- * or folio_mark_dirty(), at which stage we bring the folio's dirty flag
- * and xarray dirty tag back into sync.
- *
- * This incoherency between the folio's dirty flag and xarray tag is
- * unfortunate, but it only exists while the folio is locked.
+ * Internal helper to take care of clearing dirty bit on a folio in preparation
+ * of an IO. For some cases we might not want to do mkclean, eg, if we've
+ * already taken care of it, hence pass the should_mkclean flag to indicate if
+ * its needed.
*/
-bool folio_clear_dirty_for_io(struct folio *folio)
+static bool __folio_clear_dirty_for_io(struct folio *folio, bool should_mkclean)
{
struct address_space *mapping = folio_mapping(folio);
bool ret = false;
@@ -2900,7 +2892,7 @@ bool folio_clear_dirty_for_io(struct folio *folio)
* as a serialization point for all the different
* threads doing their things.
*/
- if (folio_mkclean(folio))
+ if (should_mkclean && folio_mkclean(folio))
folio_mark_dirty(folio);
/*
* We carefully synchronise fault handlers against
@@ -2925,6 +2917,25 @@ bool folio_clear_dirty_for_io(struct folio *folio)
}
return folio_test_clear_dirty(folio);
}
+
+/*
+ * Clear a folio's dirty flag, while caring for dirty memory accounting.
+ * Returns true if the folio was previously dirty.
+ *
+ * This is for preparing to put the folio under writeout. We leave
+ * the folio tagged as dirty in the xarray so that a concurrent
+ * write-for-sync can discover it via a PAGECACHE_TAG_DIRTY walk.
+ * The ->writepage implementation will run either folio_start_writeback()
+ * or folio_mark_dirty(), at which stage we bring the folio's dirty flag
+ * and xarray dirty tag back into sync.
+ *
+ * This incoherency between the folio's dirty flag and xarray tag is
+ * unfortunate, but it only exists while the folio is locked.
+ */
+bool folio_clear_dirty_for_io(struct folio *folio)
+{
+ return __folio_clear_dirty_for_io(folio, true);
+}
EXPORT_SYMBOL(folio_clear_dirty_for_io);
static void wb_inode_writeback_start(struct bdi_writeback *wb)
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 04/12] iomap: Add helper to revert iomap iter
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (2 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 03/12] mm: Refactor folio_clear_dirty_for_io() Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 05/12] iomap: Add initial support for buffered RWF_WRITETHROUGH Ojaswin Mujoo
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
Add helper to revert back the iomap iter by count bytes. This will be
used by upcoming RWF_WRITETHROUGH feature to revert the state of the
iomap in case an error is encountered.
Writethrough can process and park folios so that they can be batched
together for IO later, this also advances the iomap iter. However if we
encounter any error, we would want to cancel the IO on those processed
folios. In this case, we want to revert the iter back so the upper
layers can know how much we were actually able to write before failure.
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/iomap/iter.c | 10 ++++++++++
include/linux/iomap.h | 1 +
2 files changed, 11 insertions(+)
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index c445a38b6285..81af220dca47 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -32,6 +32,16 @@ int iomap_iter_advance(struct iomap_iter *iter, u64 count)
return 0;
}
+int iomap_iter_revert(struct iomap_iter *iter, u64 count)
+{
+ if (WARN_ON_ONCE(iter->pos < count ||
+ iter->pos - count < iter->iter_start_pos))
+ return -EIO;
+ iter->pos -= count;
+ iter->len += count;
+ return 0;
+}
+
static inline void iomap_iter_done(struct iomap_iter *iter)
{
WARN_ON_ONCE(iter->iomap.offset > iter->pos);
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bc7ae6327dbf..a9ebf486e7b4 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -275,6 +275,7 @@ struct iomap_iter {
int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops);
int iomap_iter_advance(struct iomap_iter *iter, u64 count);
+int iomap_iter_revert(struct iomap_iter *iter, u64 count);
/**
* iomap_length_trim - trimmed length of the current iomap iteration
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 05/12] iomap: Add initial support for buffered RWF_WRITETHROUGH
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (3 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 04/12] iomap: Add helper to revert iomap iter Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 06/12] xfs: Add RWF_WRITETHROUGH support to xfs Ojaswin Mujoo
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund,
Dave Chinner
This adds initial support for performing buffered non-aio
RWF_WRITETHROUGH write. The rough flow for a writethrough write is as
follows:
1. Acquire inode lock
2. initialize writethrough context (wt_ctx) and mark
mapping as stable.
3. Start the iomap_iter() loop. For each iomap:
3.1. Acquire folio and folio_lock.
3.2. perform memcpy from user buffer to the folio and mark it
dirty
3.3. Wait for any current writeback to complete and then call
folio_mkclean() to prevent mmap writes from changing it.
3.4. Start writeback on the folio
3.5. Add the folio range under write to wt_ctx->bvec and folio_unlock()
3.6. If bvec is full, submit the current bvecs for IO.
3.7. Repeat 3.2 to 3.6 till the whole iomap is processed. Submit
the final set of bvecs for IO.
4. Repeat step 3 till we have no more data to write.
5. Finally, sleep in the syscall thread till all the IOs are
completed (refcount == 0). Once that happens, the end io handler will
wake us up.
6. Upon waking up, call fs ->end_io() callback (which updates inode
size), record any errors and return.
7. inode_unlock()
This design gives buffered writethrough the same semantics as dio and
any error in the IO is directly returned to the caller. The design has
deliberately open coded the IO submission and completion flow (inspired
by dio) rather than reusing the dio functions as accommodating buffered
writethrough logic in dio code was polluting it with too many if else
conditionals and special cases. For notes on error handling, check the
comment above iomap_file_writethrough_write()
Suggested-by: Jan Kara <jack@suse.cz>
Suggested-by: Dave Chinner <dgc@kernel.org>
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/iomap/buffered-io.c | 495 ++++++++++++++++++++++++++++++++++
include/linux/fs.h | 11 +
include/linux/iomap.h | 42 +++
include/linux/pagemap.h | 1 +
include/uapi/linux/fs.h | 5 +-
mm/page-writeback.c | 10 +
tools/include/uapi/linux/fs.h | 5 +-
7 files changed, 567 insertions(+), 2 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 678f4c5329e7..68b93d96faa3 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -10,6 +10,7 @@
#include <linux/migrate.h>
#include <linux/fserror.h>
#include <linux/fsverity.h>
+#include <linux/rmap.h>
#include "internal.h"
#include "trace.h"
@@ -1185,6 +1186,384 @@ static bool iomap_write_end(struct iomap_iter *iter, size_t len, size_t copied,
return __iomap_write_end(iter->inode, pos, len, copied, folio);
}
+static ssize_t iomap_writethrough_complete(struct iomap_writethrough_ctx *wt_ctx)
+{
+ struct kiocb *iocb = wt_ctx->iocb;
+ struct inode *inode = wt_ctx->inode;
+ ssize_t ret = wt_ctx->error;
+
+ if (wt_ctx->end_io) {
+ int err = wt_ctx->end_io(wt_ctx, wt_ctx->written,
+ wt_ctx->error,
+ wt_ctx->flags);
+ if (err)
+ ret = err;
+ }
+
+ mapping_dec_inflight_stable_writes(inode->i_mapping);
+
+ if (!ret) {
+ ret = wt_ctx->written;
+ iocb->ki_pos += ret;
+ } else
+ mapping_set_error(inode->i_mapping, ret);
+
+ kfree(wt_ctx);
+ return ret;
+}
+
+static void iomap_writethrough_done(struct iomap_writethrough_ctx *wt_ctx)
+{
+ struct task_struct *waiter = wt_ctx->waiter;
+
+ WRITE_ONCE(wt_ctx->waiter, NULL);
+ blk_wake_io_task(waiter);
+}
+
+static void iomap_writethrough_bio_end_io(struct bio *bio)
+{
+ struct iomap_writethrough_ctx *wt_ctx = bio->bi_private;
+ struct folio_iter fi;
+
+ if (bio->bi_status)
+ cmpxchg(&wt_ctx->error, 0,
+ blk_status_to_errno(bio->bi_status));
+ bio_for_each_folio_all(fi, bio)
+ folio_end_writeback(fi.folio);
+
+ bio_put(bio);
+ if (atomic_dec_and_test(&wt_ctx->ref))
+ iomap_writethrough_done(wt_ctx);
+}
+
+static int
+iomap_writethrough_submit_bio(struct iomap_writethrough_ctx *wt_ctx,
+ struct iomap *iomap,
+ const struct iomap_writethrough_ops *wt_ops, int error)
+{
+ struct bio *bio;
+ unsigned int i;
+ u64 len = 0;
+ blk_opf_t opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE;
+
+ if (!wt_ctx->nr_bvecs)
+ goto exit;
+
+ for (i = 0; i < wt_ctx->nr_bvecs; i++)
+ len += wt_ctx->bvec[i].bv_len;
+
+ bio = bio_alloc(iomap->bdev, wt_ctx->nr_bvecs, opf, GFP_NOFS);
+ bio->bi_iter.bi_sector = iomap_sector(iomap, wt_ctx->bio_pos);
+ bio->bi_end_io = iomap_writethrough_bio_end_io;
+ bio->bi_private = wt_ctx;
+
+ for (i = 0; i < wt_ctx->nr_bvecs; i++)
+ __bio_add_page(bio, wt_ctx->bvec[i].bv_page,
+ wt_ctx->bvec[i].bv_len,
+ wt_ctx->bvec[i].bv_offset);
+
+ if (!error && wt_ops->writethrough_submit)
+ error = wt_ops->writethrough_submit(wt_ctx->inode, iomap,
+ wt_ctx->bio_pos, len);
+
+
+ atomic_inc(&wt_ctx->ref);
+
+ /*
+ * In case of error we still need the I/O completion to run so we can
+ * release references and end writeback on the folios.
+ */
+ if (error) {
+ bio->bi_status = errno_to_blk_status(error);
+ bio_endio(bio);
+ return error;
+ }
+
+ submit_bio(bio);
+ wt_ctx->nr_bvecs = 0;
+
+exit:
+ return 0;
+}
+
+/*
+ * Submit any pending bvecs as a bio and account the written bytes.
+ * On failure, iomap_writethrough_submit_bio() has already called the
+ * endio completion handler to record the error.
+ */
+static int
+iomap_writethrough_try_submit(struct iomap_writethrough_ctx *wt_ctx,
+ struct iomap *iomap,
+ const struct iomap_writethrough_ops *wt_ops,
+ ssize_t *pending)
+{
+ int ret = iomap_writethrough_submit_bio(wt_ctx, iomap, wt_ops, 0);
+
+ if (ret < 0)
+ return ret;
+ wt_ctx->written += *pending;
+ *pending = 0;
+ return 0;
+}
+
+/**
+ * iomap_writethrough_begin - prepare the various structures for writethrough
+ * @folio: folio to prepare for writethrough
+ * @off: offset of write within folio
+ * @len: len of write within folio
+ *
+ * This function does the major preparation work needed before starting the
+ * writethrough. The main task is to prepare folio for writeththrough by blocking
+ * mmap writes and setting writeback on it. Further, we must clear the write range
+ * to non-dirty. If this results in the complete folio becoming non-dirty, then we
+ * need to clear the master dirty bit.
+ */
+static void iomap_folio_prepare_writethrough(struct folio *folio, size_t off,
+ size_t len)
+{
+ bool fully_written;
+ u64 zero = 0;
+
+ if (folio_test_writeback(folio))
+ folio_wait_writeback(folio);
+
+ if (folio_mkclean(folio))
+ folio_mark_dirty(folio);
+
+ /*
+ * We might either write through the complete folio or a partial folio
+ * writethrough might result in all blocks becoming non-dirty, so we need to
+ * check and mark the folio clean if that is the case.
+ */
+ fully_written = (off == 0 && len == folio_size(folio));
+ iomap_clear_range_dirty(folio, off, len);
+ if (fully_written ||
+ !iomap_find_dirty_range(folio, &zero, folio_size(folio)))
+ folio_clear_dirty_for_writethrough(folio);
+
+ folio_start_writeback(folio);
+}
+
+/**
+ * iomap_writethrough_iter - perform RWF_WRITETHROUGH buffered write
+ * @wt_ctx: writethrough context
+ * @iter: iomap iter holding mapping information
+ * @i: iov_iter for write
+ * @wt_ops: the fs callbacks needed for writethrough
+ *
+ * This function copies the user buffer to folio similar to usual buffered
+ * IO path, with the difference that we immediately issue the IO. For this we
+ * utilize IO submission and completion mechanism that is inspired by dio.
+ *
+ * Folio handling note: We might be writing through a partial folio so we need
+ * to be careful to not clear the folio dirty bit unless there are no dirty blocks
+ * in the folio after the writethrough.
+ */
+static int iomap_writethrough_iter(struct iomap_writethrough_ctx *wt_ctx,
+ struct iomap_iter *iter, struct iov_iter *i,
+ const struct iomap_writethrough_ops *wt_ops)
+
+{
+ ssize_t total_written = 0, pending = 0;
+ loff_t submit_start_pos;
+ int status = 0;
+ struct address_space *mapping = iter->inode->i_mapping;
+ size_t chunk = mapping_max_folio_size(mapping);
+ unsigned int bdp_flags = (iter->flags & IOMAP_NOWAIT) ? BDP_ASYNC : 0;
+ unsigned int bs = i_blocksize(iter->inode);
+
+ /* copied over based on how DIO handles these flags */
+ if (iter->iomap.type == IOMAP_UNWRITTEN)
+ wt_ctx->flags |= IOMAP_DIO_UNWRITTEN;
+ if (iter->iomap.flags & IOMAP_F_SHARED)
+ wt_ctx->flags |= IOMAP_DIO_COW;
+
+ if (!(iter->flags & IOMAP_WRITETHROUGH))
+ return -EINVAL;
+
+ /*
+ * IOMAP_INLINE mappings have NULL bdev and would cause
+ * iomap_sector() to dereference invalid memory. Reject them.
+ */
+ if (iter->iomap.type == IOMAP_INLINE)
+ return -EINVAL;
+
+ do {
+ struct folio *folio;
+ size_t offset; /* Offset into folio */
+ loff_t old_size;
+ u64 bytes; /* Bytes to write to folio */
+ size_t copied; /* Bytes copied from user */
+ u64 written; /* Bytes have been written */
+ loff_t pos;
+ size_t off_aligned, len_aligned, pos_aligned;
+ size_t prev_pos, prev_len;
+
+ bytes = iov_iter_count(i);
+retry:
+ offset = iter->pos & (chunk - 1);
+ bytes = min(chunk - offset, bytes);
+ status = balance_dirty_pages_ratelimited_flags(mapping,
+ bdp_flags);
+ if (unlikely(status))
+ break;
+
+ /*
+ * If completions already occurred and reported errors, give up
+ * now
+ */
+ status = data_race(wt_ctx->error);
+ if (unlikely(status))
+ break;
+
+ if (bytes > iomap_length(iter))
+ bytes = iomap_length(iter);
+
+ /*
+ * Bring in the user page that we'll copy from _first_.
+ * Otherwise there's a nasty deadlock on copying from the
+ * same page as we're writing to, without it being marked
+ * up-to-date.
+ *
+ * For async buffered writes the assumption is that the user
+ * page has already been faulted in. This can be optimized by
+ * faulting the user page.
+ */
+ if (unlikely(fault_in_iov_iter_readable(i, bytes) == bytes)) {
+ status = -EFAULT;
+ break;
+ }
+
+ status = iomap_write_begin(iter, wt_ops->write_ops, &folio,
+ &offset, &bytes);
+ if (unlikely(status)) {
+ iomap_write_failed(iter->inode, iter->pos, bytes);
+ break;
+ }
+ if (iter->iomap.flags & IOMAP_F_STALE)
+ break;
+
+ pos = iter->pos;
+
+ if (mapping_writably_mapped(mapping))
+ flush_dcache_folio(folio);
+
+ copied = copy_folio_from_iter_atomic(folio, offset, bytes, i);
+ written = iomap_write_end(iter, bytes, copied, folio) ?
+ copied : 0;
+
+ old_size = iter->inode->i_size;
+ if (pos + written > old_size)
+ i_size_write(iter->inode, pos + written);
+
+ if (!written)
+ goto put_folio;
+
+ off_aligned = round_down(offset, bs);
+ pos_aligned = round_down(pos, bs);
+ len_aligned = round_up(offset + written, bs) - off_aligned;
+
+ /*
+ * Because we round down to the block size, a short copy in prev
+ * iteration can result in processing the exact same fs block
+ * again. Hence, we need to be careful to not add the repeating
+ * blocks in the bvec again.
+ */
+ if (wt_ctx->nr_bvecs && prev_pos + prev_len > pos_aligned) {
+ size_t delta = prev_pos + prev_len - pos_aligned;
+
+ /* Everything already added to bvec, nothing to do */
+ if (delta >= len_aligned)
+ goto put_folio;
+
+ pos_aligned += delta;
+ off_aligned += delta;
+ len_aligned -= delta;
+ }
+
+ prev_pos = off_aligned;
+ prev_len = len_aligned;
+
+ iomap_folio_prepare_writethrough(folio, off_aligned,
+ len_aligned);
+
+ if (!wt_ctx->nr_bvecs) {
+ wt_ctx->bio_pos = round_down(pos, bs);
+ submit_start_pos = pos;
+ }
+
+ bvec_set_folio(&wt_ctx->bvec[wt_ctx->nr_bvecs], folio,
+ len_aligned, off_aligned);
+ wt_ctx->nr_bvecs++;
+
+put_folio:
+ __iomap_put_folio(iter, wt_ops->write_ops, written, folio);
+
+ if (old_size < pos)
+ pagecache_isize_extended(iter->inode, old_size, pos);
+
+ cond_resched();
+ if (unlikely(written == 0)) {
+ iomap_write_failed(iter->inode, pos, bytes);
+ iov_iter_revert(i, copied);
+
+ if (chunk > PAGE_SIZE)
+ chunk /= 2;
+ if (copied) {
+ bytes = copied;
+ goto retry;
+ }
+ } else {
+ total_written += written;
+ pending += written;
+ iomap_iter_advance(iter, written);
+ }
+
+ /*
+ * If we fail to submit the bio, we immediately call the
+ * IO completion handler that records the error. We
+ * shall not retry anymore cause this could lead to
+ * infinite loops in case of non-transient errors.
+ */
+ if (wt_ctx->nr_bvecs == wt_ctx->max_bvecs) {
+ status = iomap_writethrough_try_submit(wt_ctx,
+ &iter->iomap, wt_ops, &pending);
+ if (status)
+ goto submit_failed;
+ }
+
+ } while (iov_iter_count(i) && iomap_length(iter));
+
+ if (wt_ctx->nr_bvecs) {
+ int ret;
+
+ ret = iomap_writethrough_try_submit(wt_ctx,
+ &iter->iomap, wt_ops, &pending);
+ if (ret) {
+ status = ret;
+ goto submit_failed;
+ }
+ }
+
+ return status;
+
+submit_failed:
+ WARN_ON(!status);
+ cmpxchg(&wt_ctx->error, 0, status);
+
+ /*
+ * In case of an error, we only consider the bytes we were actually able
+ * to submit IO for as valid data and revert the iters accordingly
+ */
+ if (pending) {
+ iomap_write_failed(iter->inode, submit_start_pos, pending);
+ iomap_iter_revert(iter, pending);
+ iov_iter_revert(i, pending);
+ }
+
+ return status;
+}
+
static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
const struct iomap_write_ops *write_ops)
{
@@ -1345,6 +1724,122 @@ int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
}
EXPORT_SYMBOL_GPL(iomap_fsverity_write);
+/**
+ * iomap_file_writethrough_write - perform a writethrough write
+ * @iocb: kiocb for the write
+ * @i: iov_iter for the write
+ * @wt_ops: writethrough related callbacks
+ * @private: optional private data
+ *
+ * Perform a writethrough write by first copying data to the page cache and then
+ * immediately sending it for IO. This is similar to dio but which maintaining
+ * the page cache coherency.
+ *
+ * There are 2 types of errors RWF_WRITETHROUGH can face:
+ *
+ * 1. Errors that don't leave page cache inconsistent. These are errors that
+ * happen before we have copied any data to folio or where we are able to
+ * successfully submit everything in folio before the error was encountered.
+ * Here the disk contents and page cache are in sync. In this case we return
+ * num. of bytes successfully written. If nothing is written, return the error.
+ *
+ * 2. Errors that leave page cache inconsistent: Any errors that either prevent
+ * us to submit data copied to folio or endio errors fall in this category.
+ * Basically, these errors happen after the folio is made uptodate and unlocked.
+ * Being unable to submit it means applications can't be sure if the content of
+ * the folio are consistent to the disk anymore. In essence these are similar to
+ * writeback failures. Hence, in this case we always mark the error in the
+ * address space and return it to user.
+ *
+ * Return: number of bytes written on success, -EIOCBQUEUED if the operation
+ * was submitted asynchronously, or a negative error code on failure.
+ */
+ssize_t iomap_file_writethrough_write(struct kiocb *iocb, struct iov_iter *i,
+ const struct iomap_writethrough_ops *wt_ops,
+ void *private)
+{
+ struct inode *inode = iocb->ki_filp->f_mapping->host;
+ struct iomap_iter iter = {
+ .inode = inode,
+ .pos = iocb->ki_pos,
+ .len = iov_iter_count(i),
+ .flags = IOMAP_WRITE | IOMAP_WRITETHROUGH,
+ .private = private,
+ };
+ struct iomap_writethrough_ctx *wt_ctx;
+ unsigned int max_bvecs;
+ ssize_t ret;
+ struct blk_plug plug;
+ size_t min_folio_bytes = PAGE_SIZE
+ << mapping_min_folio_order(inode->i_mapping);
+
+ /*
+ * For now we don't support any other flag with WRITETHROUGH
+ */
+ if (!(iocb->ki_flags & IOCB_WRITETHROUGH))
+ return -EINVAL;
+ if (iocb->ki_flags & (IOCB_DONTCACHE))
+ return -EINVAL;
+ if (iocb_is_dsync(iocb))
+ /* D_SYNC support not implemented yet */
+ return -EOPNOTSUPP;
+ if (!is_sync_kiocb(iocb))
+ /* aio support not implemented yet */
+ return -EOPNOTSUPP;
+
+ /*
+ * +1 to max bvecs to account for unaligned write spanning multiple
+ * folios. Guard against overflow since iov_iter_count() returns size_t.
+ */
+ max_bvecs = (unsigned int)min_t(
+ size_t, DIV_ROUND_UP(iov_iter_count(i), min_folio_bytes) + 1,
+ BIO_MAX_VECS);
+
+ wt_ctx = kzalloc(struct_size(wt_ctx, bvec, max_bvecs), GFP_NOFS);
+ if (!wt_ctx)
+ return -ENOMEM;
+
+ wt_ctx->iocb = iocb;
+ wt_ctx->inode = inode;
+ wt_ctx->end_io = wt_ops->end_io;
+ wt_ctx->old_i_size = i_size_read(inode);
+ wt_ctx->max_bvecs = max_bvecs;
+ atomic_set(&wt_ctx->ref, 1);
+ wt_ctx->waiter = current;
+
+ mapping_inc_inflight_stable_writes(inode->i_mapping);
+
+ blk_start_plug(&plug);
+
+ while ((ret = iomap_iter(&iter, wt_ops->ops)) > 0) {
+ WARN_ON(iter.iomap.type != IOMAP_UNWRITTEN &&
+ iter.iomap.type != IOMAP_MAPPED);
+ iter.status = iomap_writethrough_iter(wt_ctx, &iter, i, wt_ops);
+ }
+
+ blk_finish_plug(&plug);
+
+ if (!atomic_dec_and_test(&wt_ctx->ref)) {
+ for (;;) {
+ set_current_state(TASK_UNINTERRUPTIBLE);
+ if (!READ_ONCE(wt_ctx->waiter))
+ break;
+ blk_io_schedule();
+ }
+ __set_current_state(TASK_RUNNING);
+ }
+
+ /*
+ * we encountered an error before submitting any data and there are no
+ * mapping error. We can simply return the error.
+ */
+ if (ret && !wt_ctx->written && !wt_ctx->error)
+ return ret;
+
+ return iomap_writethrough_complete(wt_ctx);
+}
+EXPORT_SYMBOL_GPL(iomap_file_writethrough_write);
+
static void iomap_write_delalloc_ifs_punch(struct inode *inode,
struct folio *folio, loff_t start_byte, loff_t end_byte,
struct iomap *iomap, iomap_punch_t punch)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index f196f25255ab..cbaa65e05d9e 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -344,6 +344,7 @@ struct readahead_control;
#define IOCB_ATOMIC (__force int) RWF_ATOMIC
#define IOCB_DONTCACHE (__force int) RWF_DONTCACHE
#define IOCB_NOSIGNAL (__force int) RWF_NOSIGNAL
+#define IOCB_WRITETHROUGH (__force int) RWF_WRITETHROUGH
/* non-RWF related bits - start at 16 */
#define IOCB_EVENTFD (1 << 16)
@@ -1979,6 +1980,8 @@ struct file_operations {
#define FOP_ASYNC_LOCK ((__force fop_flags_t)(1 << 6))
/* File system supports uncached read/write buffered IO */
#define FOP_DONTCACHE ((__force fop_flags_t)(1 << 7))
+/* File system supports write through buffered IO */
+#define FOP_WRITETHROUGH ((__force fop_flags_t)(1 << 8))
/* Wrap a directory iterator that needs exclusive inode access */
int wrap_directory_iterator(struct file *, struct dir_context *,
@@ -3478,6 +3481,14 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags,
if (IS_DAX(ki->ki_filp->f_mapping->host))
return -EOPNOTSUPP;
}
+ if (flags & RWF_WRITETHROUGH) {
+ /* file system must support it */
+ if (!(ki->ki_filp->f_op->fop_flags & FOP_WRITETHROUGH))
+ return -EOPNOTSUPP;
+ /* DAX mappings not supported */
+ if (IS_DAX(ki->ki_filp->f_mapping->host))
+ return -EOPNOTSUPP;
+ }
kiocb_flags |= (__force int) (flags & RWF_SUPPORTED);
if (flags & RWF_SYNC)
kiocb_flags |= IOCB_DSYNC;
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index a9ebf486e7b4..5a9223856015 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -212,6 +212,7 @@ struct iomap_write_ops {
#endif /* CONFIG_FS_DAX */
#define IOMAP_ATOMIC (1 << 9) /* torn-write protection */
#define IOMAP_DONTCACHE (1 << 10)
+#define IOMAP_WRITETHROUGH (1 << 11)
/*
* Return the existing mapping at pos, or reserve space starting at pos for up
@@ -558,6 +559,29 @@ struct iomap_writepage_ctx {
void *wb_ctx; /* pending writeback context */
};
+struct iomap_writethrough_ctx {
+ struct kiocb *iocb;
+ struct inode *inode;
+ loff_t old_i_size;
+ loff_t new_i_size;
+ loff_t pos;
+ size_t written;
+ atomic_t ref;
+ unsigned int flags;
+ int error;
+
+ /* used during submission and for non-aio completion */
+ struct task_struct *waiter;
+ int (*end_io)(struct iomap_writethrough_ctx *wt_ctx, ssize_t size,
+ int error, unsigned int flags);
+
+ loff_t bio_pos;
+ unsigned int nr_bvecs;
+ unsigned int max_bvecs;
+ struct bio_vec bvec[];
+
+};
+
struct iomap_ioend *iomap_init_ioend(struct inode *inode, struct bio *bio,
loff_t file_offset, u16 ioend_flags);
struct iomap_ioend *iomap_split_ioend(struct iomap_ioend *ioend,
@@ -748,6 +772,24 @@ static __always_inline ssize_t iomap_dio_read_simple(struct kiocb *iocb,
return __iomap_dio_read_simple(iocb, iter, &iomi);
}
+/*
+ * In writethrough, we copy user data to folio first and then send the folio
+ * to writeback via dio path. To achieve this, we need callbacks from iomap_ops
+ * and iomap_write_ops. This struct packs them together.
+ */
+struct iomap_writethrough_ops {
+ const struct iomap_ops *ops;
+ const struct iomap_write_ops *write_ops;
+ int (*writethrough_submit)(struct inode *inode, struct iomap *iomap,
+ loff_t offset, u64 len);
+ int (*end_io)(struct iomap_writethrough_ctx *iocb, ssize_t size,
+ int error, unsigned int flags);
+};
+
+ssize_t iomap_file_writethrough_write(struct kiocb *iocb, struct iov_iter *i,
+ const struct iomap_writethrough_ops *wt_ops,
+ void *private);
+
#ifdef CONFIG_SWAP
struct file;
struct swap_info_struct;
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 59713769f731..8f458ecb8518 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1333,6 +1333,7 @@ static inline void folio_cancel_dirty(struct folio *folio)
__folio_cancel_dirty(folio);
}
bool folio_clear_dirty_for_io(struct folio *folio);
+bool folio_clear_dirty_for_writethrough(struct folio *folio);
bool clear_page_dirty_for_io(struct page *page);
void folio_invalidate(struct folio *folio, size_t offset, size_t length);
bool noop_dirty_folio(struct address_space *mapping, struct folio *folio);
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index a46c33692aa2..67d8987b343d 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -451,10 +451,13 @@ typedef int __bitwise __kernel_rwf_t;
/* prevent pipe and socket writes from raising SIGPIPE */
#define RWF_NOSIGNAL ((__force __kernel_rwf_t)0x00000100)
+/* buffered IO that is asynchronously written through to disk after write */
+#define RWF_WRITETHROUGH ((__force __kernel_rwf_t)0x00000200)
+
/* mask of flags supported by the kernel */
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\
- RWF_DONTCACHE | RWF_NOSIGNAL)
+ RWF_DONTCACHE | RWF_NOSIGNAL | RWF_WRITETHROUGH)
#define PROCFS_IOCTL_MAGIC 'f'
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 2f8059ab55b9..098aa472d368 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2938,6 +2938,16 @@ bool folio_clear_dirty_for_io(struct folio *folio)
}
EXPORT_SYMBOL(folio_clear_dirty_for_io);
+/*
+ * Clear folio dirty in preparation of writethrough. Note that for writethrough
+ * we have already done folkio_mkclean so we avoid it here
+ */
+bool folio_clear_dirty_for_writethrough(struct folio *folio)
+{
+ return __folio_clear_dirty_for_io(folio, false);
+}
+EXPORT_SYMBOL(folio_clear_dirty_for_writethrough);
+
static void wb_inode_writeback_start(struct bdi_writeback *wb)
{
atomic_inc(&wb->writeback_inodes);
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index f0a26309b6d5..63ef6b126b8b 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -344,10 +344,13 @@ typedef int __bitwise __kernel_rwf_t;
/* buffered IO that drops the cache after reading or writing data */
#define RWF_DONTCACHE ((__force __kernel_rwf_t)0x00000080)
+/* buffered IO that is asynchronously written through to disk after write */
+#define RWF_WRITETHROUGH ((__force __kernel_rwf_t)0x00000200)
+
/* mask of flags supported by the kernel */
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\
- RWF_DONTCACHE)
+ RWF_DONTCACHE | RWF_WRITETHROUGH)
#define PROCFS_IOCTL_MAGIC 'f'
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 06/12] xfs: Add RWF_WRITETHROUGH support to xfs
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (4 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 05/12] iomap: Add initial support for buffered RWF_WRITETHROUGH Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 07/12] iomap: Add aio support to RWF_WRITETHROUGH Ojaswin Mujoo
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
Add the boilerplate needed to start supporting RWF_WRITETHROUGH in XFS.
We use the direct write ->iomap_begin() functions to ensure the range
under write through always has a real non-delalloc extent. We reuse the xfs
dio's end IO function to perform extent conversion and i_size handling
for us.
*Note on COW extent over DATA hole case*
In case of an unmapped COW extent over a DATA hole
(due to COW preallocations), leave the extent unmapped until we are just
about to send IO. At that time, use the ->writethrough_submit() call
back to convert the COW extent to written.
We initially tried converting during iomap_begin() time (like dio does)
but that results in a stale data exposure as follows:
1. iomap_begin() - converts COW extent over DATA hole to written and
marks IOMAP_F_NEW to handle zeroing.
2. During iomap_write_begin() -> realise extent is stale and return back
without zeroing.
3. iomap_begin() - Again sees the same COW extent but it's written
this time so we don't mark IOMAP_F_NEW
4. Since IOMAP_F_NEW is unmarked, we never zeroout and hence expose
stale data.
To avoid the above, take the buffered IO approach of converting the
extent just before IO, when we are sure to have zeroed out the folio.
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/xfs/xfs_file.c | 95 ++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 89 insertions(+), 6 deletions(-)
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index d53a329b6a58..3e25baca22c2 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -690,6 +690,48 @@ static const struct iomap_dio_ops xfs_dio_write_ops = {
.end_io = xfs_dio_write_end_io,
};
+static int
+xfs_writethrough_end_io(
+ struct iomap_writethrough_ctx *wt_ctx,
+ ssize_t size,
+ int error,
+ unsigned int flags)
+{
+ struct xfs_inode *ip = XFS_I(wt_ctx->inode);
+ xfs_off_t offset = wt_ctx->iocb->ki_pos;
+
+ if (unlikely(error))
+ goto error;
+
+ /*
+ * writethrough completions are handled same as dio with the exception
+ * that we need to explicitly change the i_disk_size. This is because
+ * unlike dio, we have already updated the i_size and hence the
+ * (i_disk_size < i_size) check will fail in dio code
+ */
+ error = xfs_dio_write_end_io(wt_ctx->iocb, size, error, flags);
+ if (error)
+ goto error;
+ if (!size)
+ goto out;
+
+ /* Fast lockless check similar to how we do in buffered endio */
+ if (offset + size > ip->i_disk_size) {
+ error = xfs_setfilesize(ip, offset, size);
+ if (error)
+ goto error;
+ }
+
+ return 0;
+
+error:
+ if (wt_ctx->flags & IOMAP_DIO_COW)
+ xfs_reflink_cancel_cow_range(ip, offset, size, true);
+
+out:
+ return error;
+}
+
static void
xfs_dio_zoned_submit_io(
const struct iomap_iter *iter,
@@ -1021,6 +1063,39 @@ xfs_file_dax_write(
return ret;
}
+static int
+xfs_writethrough_submit(
+ struct inode *inode,
+ struct iomap *iomap,
+ loff_t offset,
+ u64 count)
+{
+ int error = 0;
+ unsigned int nofs_flag;
+
+ /*
+ * Convert CoW extents to regular.
+ *
+ * We are under writethrough context with folio lock possibly held. To
+ * avoid memory allocation deadlocks, set the task-wide nofs context.
+ */
+ if (iomap->flags & IOMAP_F_SHARED) {
+ nofs_flag = memalloc_nofs_save();
+ error = xfs_reflink_convert_cow(XFS_I(inode), offset, count);
+ memalloc_nofs_restore(nofs_flag);
+ }
+
+ return error;
+}
+
+const struct iomap_writethrough_ops xfs_writethrough_ops = {
+ .ops = &xfs_direct_write_iomap_ops,
+ .write_ops = &xfs_iomap_write_ops,
+ .end_io = xfs_writethrough_end_io,
+ .writethrough_submit = &xfs_writethrough_submit
+};
+
+
STATIC ssize_t
xfs_file_buffered_write(
struct kiocb *iocb,
@@ -1043,9 +1118,13 @@ xfs_file_buffered_write(
goto out;
trace_xfs_file_buffered_write(iocb, from);
- ret = iomap_file_buffered_write(iocb, from,
- &xfs_buffered_write_iomap_ops, &xfs_iomap_write_ops,
- NULL);
+ if (iocb->ki_flags & IOCB_WRITETHROUGH) {
+ ret = iomap_file_writethrough_write(iocb, from,
+ &xfs_writethrough_ops, NULL);
+ } else
+ ret = iomap_file_buffered_write(iocb, from,
+ &xfs_buffered_write_iomap_ops,
+ &xfs_iomap_write_ops, NULL);
/*
* If we hit a space limit, try to free up some lingering preallocated
@@ -1080,8 +1159,12 @@ xfs_file_buffered_write(
if (ret > 0) {
XFS_STATS_ADD(ip->i_mount, xs_write_bytes, ret);
- /* Handle various SYNC-type writes */
- ret = generic_write_sync(iocb, ret);
+ /*
+ * Handle various SYNC-type writes.
+ * For writethrough, we handle sync during completion.
+ */
+ if (!(iocb->ki_flags & IOCB_WRITETHROUGH))
+ ret = generic_write_sync(iocb, ret);
}
return ret;
}
@@ -2176,7 +2259,7 @@ const struct file_operations xfs_file_operations = {
.remap_file_range = xfs_file_remap_range,
.fop_flags = FOP_MMAP_SYNC | FOP_BUFFER_RASYNC |
FOP_BUFFER_WASYNC | FOP_DIO_PARALLEL_WRITE |
- FOP_DONTCACHE,
+ FOP_DONTCACHE | FOP_WRITETHROUGH,
.setlease = generic_setlease,
};
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 07/12] iomap: Add aio support to RWF_WRITETHROUGH
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (5 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 06/12] xfs: Add RWF_WRITETHROUGH support to xfs Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 08/12] iomap: Add DSYNC " Ojaswin Mujoo
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
With aio the only thing we need to be careful off is that writethrough
can be in progress even after dropping inode and folio lock. Due to
this, we need a way to synchronise with other paths where stable write
is not enough, example:
1. Truncate to 0 in xfs sets i_size = 0 before waiting for writeback to
complete. In case of writethrough, the end io completion can again
push the i_size to a non-zero value.
2. Dio reads might race with aio writethrough ->end_io() and read 0s if
unwritten conversion is yet to happen.
Hence use the dio begin/end as it gives us the required guarantees.
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/iomap/buffered-io.c | 57 ++++++++++++++++++++++++++++++++++++------
include/linux/iomap.h | 11 ++++++--
2 files changed, 59 insertions(+), 9 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 68b93d96faa3..2ea45ab67bd9 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -1202,6 +1202,9 @@ static ssize_t iomap_writethrough_complete(struct iomap_writethrough_ctx *wt_ctx
mapping_dec_inflight_stable_writes(inode->i_mapping);
+ if (wt_ctx->is_aio)
+ inode_dio_end(inode);
+
if (!ret) {
ret = wt_ctx->written;
iocb->ki_pos += ret;
@@ -1212,12 +1215,27 @@ static ssize_t iomap_writethrough_complete(struct iomap_writethrough_ctx *wt_ctx
return ret;
}
+static void iomap_writethrough_complete_work(struct work_struct *work)
+{
+ struct iomap_writethrough_ctx *wt_ctx =
+ container_of(work, struct iomap_writethrough_ctx, aio_work);
+ struct kiocb *iocb = wt_ctx->iocb;
+
+ iocb->ki_complete(iocb, iomap_writethrough_complete(wt_ctx));
+}
+
static void iomap_writethrough_done(struct iomap_writethrough_ctx *wt_ctx)
{
- struct task_struct *waiter = wt_ctx->waiter;
+ if (!wt_ctx->is_aio) {
+ struct task_struct *waiter = wt_ctx->waiter;
- WRITE_ONCE(wt_ctx->waiter, NULL);
- blk_wake_io_task(waiter);
+ WRITE_ONCE(wt_ctx->waiter, NULL);
+ blk_wake_io_task(waiter);
+ return;
+ }
+
+ INIT_WORK(&wt_ctx->aio_work, iomap_writethrough_complete_work);
+ queue_work(wt_ctx->inode->i_sb->s_dio_done_wq, &wt_ctx->aio_work);
}
static void iomap_writethrough_bio_end_io(struct bio *bio)
@@ -1783,9 +1801,9 @@ ssize_t iomap_file_writethrough_write(struct kiocb *iocb, struct iov_iter *i,
if (iocb_is_dsync(iocb))
/* D_SYNC support not implemented yet */
return -EOPNOTSUPP;
- if (!is_sync_kiocb(iocb))
- /* aio support not implemented yet */
- return -EOPNOTSUPP;
+
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ iter.flags |= IOMAP_NOWAIT;
/*
* +1 to max bvecs to account for unaligned write spanning multiple
@@ -1804,11 +1822,33 @@ ssize_t iomap_file_writethrough_write(struct kiocb *iocb, struct iov_iter *i,
wt_ctx->end_io = wt_ops->end_io;
wt_ctx->old_i_size = i_size_read(inode);
wt_ctx->max_bvecs = max_bvecs;
+ wt_ctx->is_aio = !is_sync_kiocb(iocb);
atomic_set(&wt_ctx->ref, 1);
- wt_ctx->waiter = current;
+
+ if (!wt_ctx->is_aio)
+ wt_ctx->waiter = current;
+ else
+ /*
+ * With aio, writethrough can be in progress even after dropping
+ * inode and folio lock. Due to this, we need a way to
+ * synchronise with other paths where stable write is not enough
+ * (example truncate). Hence use the dio begin/end as it gives
+ * us the required guarantees.
+ */
+ inode_dio_begin(inode);
mapping_inc_inflight_stable_writes(inode->i_mapping);
+ if (wt_ctx->is_aio && !inode->i_sb->s_dio_done_wq) {
+ ret = sb_init_dio_done_wq(inode->i_sb);
+ if (ret < 0) {
+ mapping_dec_inflight_stable_writes(inode->i_mapping);
+ inode_dio_end(inode);
+ kfree(wt_ctx);
+ return ret;
+ }
+ }
+
blk_start_plug(&plug);
while ((ret = iomap_iter(&iter, wt_ops->ops)) > 0) {
@@ -1820,6 +1860,9 @@ ssize_t iomap_file_writethrough_write(struct kiocb *iocb, struct iov_iter *i,
blk_finish_plug(&plug);
if (!atomic_dec_and_test(&wt_ctx->ref)) {
+ if (wt_ctx->is_aio)
+ return -EIOCBQUEUED;
+
for (;;) {
set_current_state(TASK_UNINTERRUPTIBLE);
if (!READ_ONCE(wt_ctx->waiter))
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 5a9223856015..691f48bc8afa 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -569,9 +569,16 @@ struct iomap_writethrough_ctx {
atomic_t ref;
unsigned int flags;
int error;
+ bool is_aio;
+
+ union {
+ /* used during submission and for non-aio completion */
+ struct task_struct *waiter;
+
+ /* used during aio completion */
+ struct work_struct aio_work;
+ };
- /* used during submission and for non-aio completion */
- struct task_struct *waiter;
int (*end_io)(struct iomap_writethrough_ctx *wt_ctx, ssize_t size,
int error, unsigned int flags);
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 08/12] iomap: Add DSYNC support to RWF_WRITETHROUGH
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (6 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 07/12] iomap: Add aio support to RWF_WRITETHROUGH Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 09/12] fs: Introduce RWF_NOSERIAL flag to indicate parallel reads/writes Ojaswin Mujoo
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund,
Dave Chinner
Add DSYNC support to writethrough buffered writes. Unlike the usual
buffered writes where we call generic_write_sync() inline during the
syscall path, for writethrough we instead sync the data during IO
completion path, just like dio.
This allows aio writethrough to be truly async where the syscall can
return after IO submission and the sync can then be done asynchronously
during IO completion time.
Further, just like dio, we utilize the FUA optimization, if available,
to avoid syncing the data for DSYNC operations.
Suggested-by: Dave Chinner <dgc@kernel.org>
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/iomap/buffered-io.c | 35 ++++++++++++++++++++++++++++++++++-
include/linux/iomap.h | 1 +
2 files changed, 35 insertions(+), 1 deletion(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 2ea45ab67bd9..91dad3550398 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -1208,7 +1208,17 @@ static ssize_t iomap_writethrough_complete(struct iomap_writethrough_ctx *wt_ctx
if (!ret) {
ret = wt_ctx->written;
iocb->ki_pos += ret;
- } else
+
+ /*
+ * If this is a DSYNC write and we couldn't optimize it, make
+ * sure we push it to stable storage now that we've written
+ * data.
+ */
+ if (iocb_is_dsync(wt_ctx->iocb) && !wt_ctx->use_fua)
+ ret = generic_write_sync(iocb, ret);
+ }
+
+ if (ret < 0)
mapping_set_error(inode->i_mapping, ret);
kfree(wt_ctx);
@@ -1270,6 +1280,9 @@ iomap_writethrough_submit_bio(struct iomap_writethrough_ctx *wt_ctx,
for (i = 0; i < wt_ctx->nr_bvecs; i++)
len += wt_ctx->bvec[i].bv_len;
+ if (wt_ctx->use_fua)
+ opf |= REQ_FUA;
+
bio = bio_alloc(iomap->bdev, wt_ctx->nr_bvecs, opf, GFP_NOFS);
bio->bi_iter.bi_sector = iomap_sector(iomap, wt_ctx->bio_pos);
bio->bi_end_io = iomap_writethrough_bio_end_io;
@@ -1406,6 +1419,19 @@ static int iomap_writethrough_iter(struct iomap_writethrough_ctx *wt_ctx,
if (iter->iomap.type == IOMAP_INLINE)
return -EINVAL;
+ /*
+ * If we realise that cache flush is necessary (eg FUA is not present
+ * or we need metadata updates) then we turn off the optimization.
+ */
+ if (wt_ctx->use_fua) {
+ if (iter->iomap.type != IOMAP_MAPPED ||
+ (iter->iomap.flags &
+ (IOMAP_F_NEW | IOMAP_F_SHARED | IOMAP_F_DIRTY)) ||
+ (bdev_write_cache(iter->iomap.bdev) &&
+ !bdev_fua(iter->iomap.bdev)))
+ wt_ctx->use_fua = false;
+ }
+
do {
struct folio *folio;
size_t offset; /* Offset into folio */
@@ -1825,6 +1851,13 @@ ssize_t iomap_file_writethrough_write(struct kiocb *iocb, struct iov_iter *i,
wt_ctx->is_aio = !is_sync_kiocb(iocb);
atomic_set(&wt_ctx->ref, 1);
+ /*
+ * Similar to dio, we optimistically set use_fua=true to avoid explicit
+ * sync. In case we later realise cache flush is needed we set it back
+ * to false.
+ */
+ wt_ctx->use_fua = iocb_is_dsync(iocb) && !(iocb->ki_flags & IOCB_SYNC);
+
if (!wt_ctx->is_aio)
wt_ctx->waiter = current;
else
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 691f48bc8afa..b1c6707c25a4 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -570,6 +570,7 @@ struct iomap_writethrough_ctx {
unsigned int flags;
int error;
bool is_aio;
+ bool use_fua;
union {
/* used during submission and for non-aio completion */
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 09/12] fs: Introduce RWF_NOSERIAL flag to indicate parallel reads/writes
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (7 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 08/12] iomap: Add DSYNC " Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 10/12] xfs: Implement RWF_NOSERIAL to parallelize RWF_WRITETHROUGH writes Ojaswin Mujoo
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
Introduce RWF_NOSERIAL flag to indicate that the application is
okay with its reads and writes going in parallel to other read/writes.
This flag will allow writes and read to go in parallel (for eg, under a
shared lock) increasing performance at the cost of losing the (loosely
implemented) POSIX guarantees wrt to R/W serialization that various
filesystems provide.
In this patch we just introduce the flag and in upcoming patches we will
use it to implement parallel writes to increase performance of
RWF_WRITETHROUGH
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
include/linux/fs.h | 10 +++++++++-
include/uapi/linux/fs.h | 6 +++++-
tools/include/uapi/linux/fs.h | 5 ++++-
3 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/include/linux/fs.h b/include/linux/fs.h
index cbaa65e05d9e..6f84bb41f84a 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -345,6 +345,7 @@ struct readahead_control;
#define IOCB_DONTCACHE (__force int) RWF_DONTCACHE
#define IOCB_NOSIGNAL (__force int) RWF_NOSIGNAL
#define IOCB_WRITETHROUGH (__force int) RWF_WRITETHROUGH
+#define IOCB_NOSERIAL (__force int) RWF_NOSERIAL
/* non-RWF related bits - start at 16 */
#define IOCB_EVENTFD (1 << 16)
@@ -375,7 +376,8 @@ struct readahead_control;
{ IOCB_NOIO, "NOIO" }, \
{ IOCB_ALLOC_CACHE, "ALLOC_CACHE" }, \
{ IOCB_AIO_RW, "AIO_RW" }, \
- { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }
+ { IOCB_HAS_METADATA, "AIO_HAS_METADATA" }, \
+ { IOCB_NOSERIAL, "IOCB_NOSERIAL" }
struct kiocb {
struct file *ki_filp;
@@ -3499,6 +3501,12 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags,
ki->ki_flags &= ~IOCB_APPEND;
}
+ /*
+ * Currently, only writethrough supports noserial IO.
+ */
+ if ((flags & RWF_NOSERIAL) && !(flags & RWF_WRITETHROUGH))
+ return -EOPNOTSUPP;
+
ki->ki_flags |= kiocb_flags;
return 0;
}
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index 67d8987b343d..b2ad2d8548ac 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -454,10 +454,14 @@ typedef int __bitwise __kernel_rwf_t;
/* buffered IO that is asynchronously written through to disk after write */
#define RWF_WRITETHROUGH ((__force __kernel_rwf_t)0x00000200)
+/* buffered IO writes that are non sequential because they use a shared lock */
+#define RWF_NOSERIAL ((__force __kernel_rwf_t)0x00000400)
+
/* mask of flags supported by the kernel */
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\
- RWF_DONTCACHE | RWF_NOSIGNAL | RWF_WRITETHROUGH)
+ RWF_DONTCACHE | RWF_NOSIGNAL | RWF_WRITETHROUGH |\
+ RWF_NOSERIAL)
#define PROCFS_IOCTL_MAGIC 'f'
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index 63ef6b126b8b..c7a1a1d238f0 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -347,10 +347,13 @@ typedef int __bitwise __kernel_rwf_t;
/* buffered IO that is asynchronously written through to disk after write */
#define RWF_WRITETHROUGH ((__force __kernel_rwf_t)0x00000200)
+/* buffered IO writes that are non sequential because they use a shared lock */
+#define RWF_NOSERIAL ((__force __kernel_rwf_t)0x00000400)
+
/* mask of flags supported by the kernel */
#define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\
RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\
- RWF_DONTCACHE | RWF_WRITETHROUGH)
+ RWF_DONTCACHE | RWF_WRITETHROUGH | RWF_NOSERIAL)
#define PROCFS_IOCTL_MAGIC 'f'
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 10/12] xfs: Implement RWF_NOSERIAL to parallelize RWF_WRITETHROUGH writes
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (8 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 09/12] fs: Introduce RWF_NOSERIAL flag to indicate parallel reads/writes Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 11/12] iomap: Avoid folio dirtying in case of RWF_WRITETHROUGH Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 12/12] iomap: Handle deadlock due to repeating folios in RWF_WRITETHROUGH Ojaswin Mujoo
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
In xfs, buffered writethrough writes take an exclusive inode lock
similar to regular buffered write path. However, since writethrough also
submits the write under the inode lock, the increased critical section
really hurts performance when we have parallel writers writing to the
same file.
To mitigate this, implement RWF_NOSERIAL flag which allows us to perform
the write under a shared inode lock, instead of an exclusive lock. This
gives significant performance boost to single file, multiple writer
workloads at the cost of losing write-write serialization and read-write
serialization guarantees which XFS has historically provided.
Let's look at each of the guarantees and what changes with the NOSERIAL
flag:
Write-write guarantee:
Image 2 writers trying to write the same 3 folios. One is write As to
all 3 folios (denoted by AAA) and the other is writing BBB. Then under
exclusive lock the final state of the 3 folios could only be either AAA
or BBB.
However, with NOSERIAL writes, we could end up with mixed data in the
3 folios, like AAB, ABA, BAA etc. Note that this mixing will always
happen at the inter folio level, data contained within the same folio
will not be mixed as it is protected by the folio lock.
Read-write guarantee:
In XFS, reads also take a shared lock, however they don't take a folio
lock when copying from the folio to the user buffer. Due to the shared
lock of read and exclusive lock of write, we are able to guarantee that
the read always reads either completely old data or completely new data.
However, with the NOSERIAL flag, writethrough will take a shared lock for
writes, ie a read can race with a write which is still in middle of
copying data to the folio, hence the read can read a mix of old and new
data.
Despite the above changes in behavior, there might be applications
who would be okay to lose the guarantees because of the nature of their
workloads for example, if they never have multiple readers/writes doing
IO to the same range in the file. Such applications would benefit
significantly by using NOSERIAL writes.
Below are some fio performance numbers of RWF_WRITETHROUGH with and without
NOSERIAL writes.
** Multiple writes, single file (Pure overwrites, DSYNC) **
Fio Workload:
libaio, buffered randwrite, bs=4k, size=2G (pre written), O_DSYNC
iodepth=32
numjobs baseline BW RWF_NOSERIAL BW Δ%
1 350 392 +12.0%
2 526 798 +51.7%
4 597 1591 +166.5%
8 630 1839 +191.9%
16 570 1836 +222.1%
** Multiple writes, single file (Preallocated file, sync_file_range) **
Fio Workload:
libaio, buffered randwrite, bs=16k, size=5G (fallocated)
sync_file_range=wait_before,write:16
numjobs baseline BW RWF_NOSERIAL BW Δ%
1 1099 1279 +16.4%
2 1482 2456 +65.7%
4 2065 2479 +20.1%
8 1778 2500 +40.6%
16 1787 2508 +40.3%
* Multiple writes, single file (Truncated file, DSYNC) *
Fio Workload:
libaio, buffered randwrite, bs=4k, size=6.5G (truncated), O_DSYNC
iodepth=32
numjobs baseline BW RWF_NOSERIAL BW Δ%
1 78 80 +2.6%
2 97 88 -9.3%
4 100 96 -4.0%
8 109 97 -11.0%
16 106 107 +0.9%
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/xfs/xfs_file.c | 57 ++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 51 insertions(+), 6 deletions(-)
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 3e25baca22c2..ef3265f3b6b6 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -508,6 +508,45 @@ xfs_file_write_checks(
return kiocb_modified(iocb);
}
+STATIC ssize_t
+xfs_file_writethrough_checks(
+ struct kiocb *iocb,
+ struct iov_iter *from,
+ unsigned int *iolock,
+ struct xfs_zone_alloc_ctx *ac)
+{
+ struct inode *inode = iocb->ki_filp->f_mapping->host;
+ size_t orig_count = iov_iter_count(from);
+ xfs_fsize_t isize = i_size_read(inode);
+ ssize_t error;
+
+restart:
+ error = xfs_file_write_checks(iocb, from, iolock, ac);
+ if (error < 0)
+ return error;
+
+ if (*iolock == XFS_IOLOCK_EXCL)
+ return 0;
+
+ /* Extending IO needs exclusive lock for i_size change */
+ if (iocb->ki_pos + iov_iter_count(from) > isize)
+ goto upgrade_excl;
+
+ return 0;
+
+upgrade_excl:
+ xfs_iunlock(XFS_I(inode), *iolock);
+ *iolock = XFS_IOLOCK_EXCL;
+ error = xfs_ilock_iocb(iocb, *iolock);
+ if (error) {
+ *iolock = 0;
+ return error;
+ }
+ iov_iter_reexpand(from, orig_count);
+ goto restart;
+
+}
+
static ssize_t
xfs_zoned_write_space_reserve(
struct xfs_mount *mp,
@@ -1108,23 +1147,29 @@ xfs_file_buffered_write(
unsigned int iolock;
write_retry:
- iolock = XFS_IOLOCK_EXCL;
+ if (iocb->ki_flags & IOCB_NOSERIAL)
+ iolock = XFS_IOLOCK_SHARED;
+ else
+ iolock = XFS_IOLOCK_EXCL;
ret = xfs_ilock_iocb(iocb, iolock);
if (ret)
return ret;
- ret = xfs_file_write_checks(iocb, from, &iolock, NULL);
- if (ret)
- goto out;
-
trace_xfs_file_buffered_write(iocb, from);
if (iocb->ki_flags & IOCB_WRITETHROUGH) {
+ ret = xfs_file_writethrough_checks(iocb, from, &iolock, NULL);
+ if (ret)
+ goto out;
ret = iomap_file_writethrough_write(iocb, from,
&xfs_writethrough_ops, NULL);
- } else
+ } else {
+ ret = xfs_file_write_checks(iocb, from, &iolock, NULL);
+ if (ret)
+ goto out;
ret = iomap_file_buffered_write(iocb, from,
&xfs_buffered_write_iomap_ops,
&xfs_iomap_write_ops, NULL);
+ }
/*
* If we hit a space limit, try to free up some lingering preallocated
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 11/12] iomap: Avoid folio dirtying in case of RWF_WRITETHROUGH
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (9 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 10/12] xfs: Implement RWF_NOSERIAL to parallelize RWF_WRITETHROUGH writes Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
2026-09-28 12:03 ` [RFC PATCH v4 12/12] iomap: Handle deadlock due to repeating folios in RWF_WRITETHROUGH Ojaswin Mujoo
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
RWF_WRITETHROUGH dirties the folio only to send it for IO immediately,
in the same context. Due to this, we can optimize away the folio
dirtying and clearing step usually seen in buffered IO. Althrough we can
do away with most of the accounting there are a couple of counters we
need to take care of which we do during IO submission/completion.
Further, now folio_start/end_writeback() can find an inode with no wb
attached and hence ensure its not null before using it.
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/iomap/buffered-io.c | 75 ++++++++++++++++++++++++++++++++++++------
mm/page-writeback.c | 24 +++++++++++---
2 files changed, 84 insertions(+), 15 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 91dad3550398..84310c786abf 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -11,6 +11,7 @@
#include <linux/fserror.h>
#include <linux/fsverity.h>
#include <linux/rmap.h>
+#include <linux/task_io_accounting_ops.h>
#include "internal.h"
#include "trace.h"
@@ -1162,6 +1163,34 @@ static bool iomap_write_end_inline(const struct iomap_iter *iter,
return true;
}
+/*
+ * __iomap_writethrough_end() is almost same as __iomap_write_end() but with the difference
+ * that we don't mark folio dirty since we are about to issue it for IO anyways.
+ * Consequently, most of the accounting is skipped.
+ */
+static bool __iomap_writethrough_end(struct inode *inode, loff_t pos, size_t len,
+ size_t copied, struct folio *folio)
+{
+ flush_dcache_folio(folio);
+
+ /*
+ * The blocks that were entirely written will now be up-to-date, so we
+ * don't have to worry about a read_folio reading them and overwriting a
+ * partial write. However, if we've encountered a short write and only
+ * partially written into a block, it will not be marked up-to-date, so a
+ * read_folio might come in and destroy our partial write.
+ *
+ * Do the simplest thing and just treat any short write to a
+ * non-uptodate page as a zero-length write, and force the caller to
+ * redo the whole thing.
+ */
+ if (unlikely(copied < len && !folio_test_uptodate(folio)))
+ return false;
+ iomap_set_range_uptodate(folio, offset_in_folio(folio, pos), len);
+ return true;
+}
+
+
/*
* Returns true if all copied bytes have been written to the pagecache,
* otherwise return false.
@@ -1183,7 +1212,10 @@ static bool iomap_write_end(struct iomap_iter *iter, size_t len, size_t copied,
return bh_written == copied;
}
- return __iomap_write_end(iter->inode, pos, len, copied, folio);
+ if (iter->flags & IOMAP_WRITETHROUGH)
+ return __iomap_writethrough_end(iter->inode, pos, len, copied, folio);
+ else
+ return __iomap_write_end(iter->inode, pos, len, copied, folio);
}
static ssize_t iomap_writethrough_complete(struct iomap_writethrough_ctx *wt_ctx)
@@ -1302,9 +1334,11 @@ iomap_writethrough_submit_bio(struct iomap_writethrough_ctx *wt_ctx,
/*
* In case of error we still need the I/O completion to run so we can
- * release references and end writeback on the folios.
+ * release references, handle accounting and end writeback on the
+ * folios.
*/
if (error) {
+ task_io_account_cancelled_write(len);
bio->bi_status = errno_to_blk_status(error);
bio_endio(bio);
return error;
@@ -1352,8 +1386,10 @@ iomap_writethrough_try_submit(struct iomap_writethrough_ctx *wt_ctx,
static void iomap_folio_prepare_writethrough(struct folio *folio, size_t off,
size_t len)
{
- bool fully_written;
+ bool needs_cleardirty = false, fully_written = false;
u64 zero = 0;
+ u64 tmp_off = off;
+ struct iomap_folio_state *ifs = folio->private;
if (folio_test_writeback(folio))
folio_wait_writeback(folio);
@@ -1362,16 +1398,35 @@ static void iomap_folio_prepare_writethrough(struct folio *folio, size_t off,
folio_mark_dirty(folio);
/*
- * We might either write through the complete folio or a partial folio
- * writethrough might result in all blocks becoming non-dirty, so we need to
- * check and mark the folio clean if that is the case.
+ * For writethrough, we don't mark the write range dirty but we still
+ * need clear the dirty range if someone else has dirtied it before.
+ * Further, if the clearing results in folio becoming completely clean,
+ * then we need to take care of accounting. If we have ifs, we can
+ * simply use that to ensure this. If we don't have ifs, that implies we
+ * did a complete overwrite of the folio, in which case we can just
+ * check if the folio was dirty earlier
*/
fully_written = (off == 0 && len == folio_size(folio));
- iomap_clear_range_dirty(folio, off, len);
- if (fully_written ||
- !iomap_find_dirty_range(folio, &zero, folio_size(folio)))
- folio_clear_dirty_for_writethrough(folio);
+ if (ifs) {
+ if (iomap_find_dirty_range(folio, &tmp_off, tmp_off + len)) {
+ iomap_clear_range_dirty(folio, off, len);
+
+ /*
+ * iomap_find_dirty_range() only works for folios with ifs
+ */
+ if (!iomap_find_dirty_range(folio, &zero,
+ folio_size(folio)))
+ needs_cleardirty = true;
+ }
+ } else {
+ WARN_ON(!fully_written);
+ if (folio_test_dirty(folio))
+ needs_cleardirty = true;
+ }
+ if (needs_cleardirty)
+ folio_clear_dirty_for_writethrough(folio);
+ task_io_account_write(len);
folio_start_writeback(folio);
}
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 098aa472d368..3ca54716d08e 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2986,11 +2986,18 @@ bool __folio_end_writeback(struct folio *folio)
__xa_clear_mark(&mapping->i_pages, folio->index,
PAGECACHE_TAG_WRITEBACK);
+ /*
+ * With RWF_WRITETHROUGH, we might not have a writeback
+ * associated with the inode
+ */
wb = inode_to_wb(inode);
- wb_stat_mod(wb, WB_WRITEBACK, -nr);
- __wb_writeout_add(wb, nr);
+ if (wb) {
+ wb_stat_mod(wb, WB_WRITEBACK, -nr);
+ __wb_writeout_add(wb, nr);
+ }
if (!mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK)) {
- wb_inode_writeback_end(wb);
+ if (wb)
+ wb_inode_writeback_end(wb);
if (mapping->host)
sb_clear_inode_writeback(mapping->host);
}
@@ -3030,10 +3037,17 @@ void __folio_start_writeback(struct folio *folio, bool keep_write)
on_wblist = mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK);
xas_set_mark(&xas, PAGECACHE_TAG_WRITEBACK);
+
+ /*
+ * With RWF_WRITETHROUGH, we might not have a writeback
+ * associated with the inode
+ */
wb = inode_to_wb(inode);
- wb_stat_mod(wb, WB_WRITEBACK, nr);
+ if (wb)
+ wb_stat_mod(wb, WB_WRITEBACK, nr);
if (!on_wblist) {
- wb_inode_writeback_start(wb);
+ if (wb)
+ wb_inode_writeback_start(wb);
/*
* We can come through here when swapping anonymous
* folios, so we don't necessarily have an inode to
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread* [RFC PATCH v4 12/12] iomap: Handle deadlock due to repeating folios in RWF_WRITETHROUGH
2026-09-28 12:03 [RFC PATCH v4 00/13] Add RWF_WRITETHROUGH support to iomap & xfs Ojaswin Mujoo
` (10 preceding siblings ...)
2026-09-28 12:03 ` [RFC PATCH v4 11/12] iomap: Avoid folio dirtying in case of RWF_WRITETHROUGH Ojaswin Mujoo
@ 2026-09-28 12:03 ` Ojaswin Mujoo
11 siblings, 0 replies; 13+ messages in thread
From: Ojaswin Mujoo @ 2026-09-28 12:03 UTC (permalink / raw)
To: Christian Brauner, linux-fsdevel
Cc: Darrick J . Wong, Carlos Maiolino, Alexander Viro, Jan Kara,
Matthew Wilcox, Andrew Morton, Ritesh Harjani, Zhang Yi,
Christoph Hellwig, Dave Chinner, Daniel Gomez, Pankaj Raghav,
Theodore Tso, linux-xfs, linux-kernel, linux-mm, Andres Freund
In iomap_writethrough_iter() we might encounter repeating folios across
multiple iterations. Repeating folios can occur if, example,
copy_folio_from_iter_atomic() does a short copy due to userspace pages
not faulted in. This is an issue because a previous loop might have
started writeback on them but not yet issued the IO. In the next
iteration trying to get the same folio with FGP_STABLE will result in a
deadlock. Since repeating folios will always be encountered back to
back, we can just use a simple cur != prev check to detect them.
Use this to avoid waiting for writeback or starting writeback on folios
we have already processed. Note that in ->endio() we might end up
calling folio_end_writethrough() twice on the same folio which can cause
issues with folio_xor_flags_has_waiters(). For simplicity, just change
the folio_xor_flags_has_waiters() call to an idempotent variant.
Reported-by: Pankaj Raghav <pankaj.raghav@linux.dev>
Co-developed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
fs/iomap/buffered-io.c | 46 ++++++++++++++++++++++++++++++++++--------
1 file changed, 38 insertions(+), 8 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 84310c786abf..69a2eccff26a 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -802,6 +802,13 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
{
fgf_t fgp = FGP_WRITEBEGIN;
+ /*
+ * For writethrough, we open code the FGP_STABLE logic directly in
+ * iomap_writhrethrough_iter() so disable it here.. See
+ * iomap_writethrough_iter() for details.
+ */
+ if (iter->flags & IOMAP_WRITETHROUGH)
+ fgp &= ~FGP_STABLE;
if (iter->flags & IOMAP_NOWAIT)
fgp |= FGP_NOWAIT;
if (iter->flags & IOMAP_DONTCACHE)
@@ -1284,12 +1291,16 @@ static void iomap_writethrough_bio_end_io(struct bio *bio)
{
struct iomap_writethrough_ctx *wt_ctx = bio->bi_private;
struct folio_iter fi;
+ struct folio *prev_folio = NULL;
if (bio->bi_status)
cmpxchg(&wt_ctx->error, 0,
blk_status_to_errno(bio->bi_status));
- bio_for_each_folio_all(fi, bio)
- folio_end_writeback(fi.folio);
+ bio_for_each_folio_all(fi, bio) {
+ if (fi.folio != prev_folio)
+ folio_end_writeback(fi.folio);
+ prev_folio = fi.folio;
+ }
bio_put(bio);
if (atomic_dec_and_test(&wt_ctx->ref))
@@ -1384,16 +1395,13 @@ iomap_writethrough_try_submit(struct iomap_writethrough_ctx *wt_ctx,
* need to clear the master dirty bit.
*/
static void iomap_folio_prepare_writethrough(struct folio *folio, size_t off,
- size_t len)
+ size_t len, bool already_prepared)
{
bool needs_cleardirty = false, fully_written = false;
u64 zero = 0;
u64 tmp_off = off;
struct iomap_folio_state *ifs = folio->private;
- if (folio_test_writeback(folio))
- folio_wait_writeback(folio);
-
if (folio_mkclean(folio))
folio_mark_dirty(folio);
@@ -1427,7 +1435,8 @@ static void iomap_folio_prepare_writethrough(struct folio *folio, size_t off,
if (needs_cleardirty)
folio_clear_dirty_for_writethrough(folio);
task_io_account_write(len);
- folio_start_writeback(folio);
+ if (!already_prepared)
+ folio_start_writeback(folio);
}
/**
@@ -1444,6 +1453,17 @@ static void iomap_folio_prepare_writethrough(struct folio *folio, size_t off,
* Folio handling note: We might be writing through a partial folio so we need
* to be careful to not clear the folio dirty bit unless there are no dirty blocks
* in the folio after the writethrough.
+ *
+ * **A corner case to be careful about**
+ *
+ * For writethrough, we open code the stable write behavior to handle the case
+ * where we encounter a folio that we already started writeback on but have not
+ * yet submitted. In that case we must not wait for writeback again to avoid
+ * deadlocking. Repeating folios can occur if, example,
+ * copy_folio_from_iter_atomic() does a short copy due to userspace pages not
+ * faulted in. Also, repeating folios will always be encountered back to back so
+ * we can just use a simple cur != prev check to detect them.
+
*/
static int iomap_writethrough_iter(struct iomap_writethrough_ctx *wt_ctx,
struct iomap_iter *iter, struct iov_iter *i,
@@ -1457,6 +1477,7 @@ static int iomap_writethrough_iter(struct iomap_writethrough_ctx *wt_ctx,
size_t chunk = mapping_max_folio_size(mapping);
unsigned int bdp_flags = (iter->flags & IOMAP_NOWAIT) ? BDP_ASYNC : 0;
unsigned int bs = i_blocksize(iter->inode);
+ struct folio *prev_folio = NULL;
/* copied over based on how DIO handles these flags */
if (iter->iomap.type == IOMAP_UNWRITTEN)
@@ -1547,6 +1568,10 @@ static int iomap_writethrough_iter(struct iomap_writethrough_ctx *wt_ctx,
if (mapping_writably_mapped(mapping))
flush_dcache_folio(folio);
+ /* Open coding stable write behavior, see comment on top. */
+ if (prev_folio != folio)
+ folio_wait_writeback(folio);
+
copied = copy_folio_from_iter_atomic(folio, offset, bytes, i);
written = iomap_write_end(iter, bytes, copied, folio) ?
copied : 0;
@@ -1584,7 +1609,10 @@ static int iomap_writethrough_iter(struct iomap_writethrough_ctx *wt_ctx,
prev_len = len_aligned;
iomap_folio_prepare_writethrough(folio, off_aligned,
- len_aligned);
+ len_aligned,
+ prev_folio == folio);
+
+ prev_folio = folio;
if (!wt_ctx->nr_bvecs) {
wt_ctx->bio_pos = round_down(pos, bs);
@@ -1629,6 +1657,8 @@ static int iomap_writethrough_iter(struct iomap_writethrough_ctx *wt_ctx,
&iter->iomap, wt_ops, &pending);
if (status)
goto submit_failed;
+
+ prev_folio = NULL;
}
} while (iov_iter_count(i) && iomap_length(iter));
--
2.55.0
^ permalink raw reply [flat|nested] 13+ messages in thread