[PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[Tal Zussman]

mm/filemap.c has accumulated additional infrastructure over the years
that is not directly related to the page cache. It is currently nearly
5000 lines long. This series splits out the folio bit-lock and wait
queue code into separate files, and moves the VFS-level
generic_file_{read,write}_iter() family of files to fs/read_write.c, in
order to provide better separation of concerns. This also slims down
mm/filemap.c by ~1000 lines.

The folio wait infrastructure is centralized in mm/folio_wait.c and
include/linux/folio_wait.h, with functions moved from mm/filemap.c,
mm/page-writeback.c, and include/linux/pagemap.h. Afterwards, the code
is cleaned up a little, with functions and data types renamed to refer
to folios rather than pages.

generic_file_{read,write}_iter() implement the VFS-level read/write path
for filesystems, including support for direct I/O. These functions and
their helpers are moved to fs/read_write.c, along with other VFS-level
read/write functions. dir_pages() is also moved to include/linux/fs.h.
i_blocks_per_folio() is not moved from include/linux/pagemap.h, as it
requires folio_size(), which is not currently available in
include/linux/fs.h.

No functional change is intended.

Note: I have additional cleanups to mm/filemap.c ready to go, foremost
among them centralizing on the filemap_*() naming convention and making
the exposed page cache API clearer and more consistent, but I've split
these patches off from that in order to avoid sending these logically
separate patches to ~60 maintainers.

---
Tal Zussman (11):
      mm: add folio_wake_writeback() helper
      folio_wait: move folio bit-lock and wait implementation to mm/folio_wait.c
      folio_wait: move folio bit-lock and wait declarations to include/linux/folio_wait.h
      folio_wait: move folio_wait_writeback() family to mm/folio_wait.c
      folio_wait: reformat comments and fix alignment
      folio_wait: rename wait_page_* infrastructure to wait_folio_*
      folio_wait: convert VM_BUG_ON_FOLIO() to VM_WARN_ON_ONCE_FOLIO()
      MAINTAINERS: add folio_wait files to MEMORY MANAGEMENT - CORE
      fs: move dir_pages() from <linux/pagemap.h> to <linux/fs.h>
      fs: move generic_file_read_iter() to fs/read_write.c
      fs: move generic_file_write_iter() family to fs/read_write.c

 MAINTAINERS                |   2 +
 fs/read_write.c            | 358 ++++++++++++++++
 include/linux/folio_wait.h | 183 +++++++++
 include/linux/fs.h         |  19 +-
 include/linux/pagemap.h    | 184 +--------
 io_uring/rw.c              |  14 +-
 io_uring/rw.h              |   6 +-
 mm/Makefile                |   2 +-
 mm/filemap.c               | 993 +--------------------------------------------
 mm/folio_wait.c            | 710 ++++++++++++++++++++++++++++++++
 mm/internal.h              |   4 +
 mm/page-writeback.c        |  66 ---
 12 files changed, 1285 insertions(+), 1256 deletions(-)
---
base-commit: e9add7501ad3297dad9b90ce201266830a68ab47
change-id: 20260511-filemap-split-871b5c18e98c

Best regards,
-- 
Tal Zussman <tz2294@columbia.edu>

[PATCH RFC 01/11] mm: add folio_wake_writeback() helper

[Tal Zussman]

Add a folio_wake_writeback() wrapper for folio_wake_bit() for use in
folio_end_writeback_no_dropbehind(), in preparation for moving the folio
bit lock and wait queue code to a separate file.

No functional change.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 mm/filemap.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 5aaba0d3e81d..567742fbaff0 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1209,6 +1209,14 @@ static void folio_wake_bit(struct folio *folio, int bit_nr)
 	spin_unlock_irqrestore(&q->lock, flags);
 }
 
+/*
+ * Wake waiters on PG_writeback for @folio.
+ */
+static void folio_wake_writeback(struct folio *folio)
+{
+	folio_wake_bit(folio, PG_writeback);
+}
+
 /*
  * A choice of three behaviors for folio_wait_bit_common():
  */
@@ -1664,7 +1672,7 @@ void folio_end_writeback_no_dropbehind(struct folio *folio)
 	}
 
 	if (__folio_end_writeback(folio))
-		folio_wake_bit(folio, PG_writeback);
+		folio_wake_writeback(folio);
 
 	acct_reclaim_writeback(folio);
 }

-- 
2.39.5

[PATCH RFC 02/11] folio_wait: move folio bit-lock and wait implementation to mm/folio_wait.c

[Tal Zussman]

mm/filemap.c contains ~600 lines of folio bit-lock and wait queue
infrastructure that is logically separate from the page cache. Move it
into a new file, mm/folio_wait.c.

folio_wake_writeback(), folio_put_wait_locked(), and
__folio_lock_async() are made non-static and declared in mm/internal.h,
as they are still used in filemap.c.

pagecache_init() is refactored to call folio_wait_init(), which
initializes the wait queue table and page_lock_unfairness sysctl.
filemap_sysctl_table is renamed to folio_wait_sysctl_table.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 mm/Makefile     |   2 +-
 mm/filemap.c    | 640 +-----------------------------------------------------
 mm/folio_wait.c | 662 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/internal.h   |   4 +
 4 files changed, 668 insertions(+), 640 deletions(-)

diff --git a/mm/Makefile b/mm/Makefile
index 8ad2ab08244e..65ce5afe7692 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -52,7 +52,7 @@ obj-y			:= filemap.o mempool.o oom_kill.o fadvise.o \
 			   maccess.o page-writeback.o folio-compat.o \
 			   readahead.o swap.o truncate.o vmscan.o shrinker.o \
 			   shmem.o util.o mmzone.o vmstat.o backing-dev.o \
-			   mm_init.o percpu.o slab_common.o \
+			   mm_init.o percpu.o slab_common.o folio_wait.o \
 			   compaction.o show_mem.o \
 			   interval_tree.o list_lru.o workingset.o \
 			   debug.o gup.o mmap_lock.o vma_init.o $(mmu-y)
diff --git a/mm/filemap.c b/mm/filemap.c
index 567742fbaff0..079f9c3ac8a2 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1053,561 +1053,12 @@ void filemap_invalidate_unlock_two(struct address_space *mapping1,
 }
 EXPORT_SYMBOL(filemap_invalidate_unlock_two);
 
-/*
- * In order to wait for pages to become available there must be
- * waitqueues associated with pages. By using a hash table of
- * waitqueues where the bucket discipline is to maintain all
- * waiters on the same queue and wake all when any of the pages
- * become available, and for the woken contexts to check to be
- * sure the appropriate page became available, this saves space
- * at a cost of "thundering herd" phenomena during rare hash
- * collisions.
- */
-#define PAGE_WAIT_TABLE_BITS 8
-#define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS)
-static wait_queue_head_t folio_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned;
-
-static wait_queue_head_t *folio_waitqueue(struct folio *folio)
-{
-	return &folio_wait_table[hash_ptr(folio, PAGE_WAIT_TABLE_BITS)];
-}
-
-/* How many times do we accept lock stealing from under a waiter? */
-static int sysctl_page_lock_unfairness = 5;
-static const struct ctl_table filemap_sysctl_table[] = {
-	{
-		.procname	= "page_lock_unfairness",
-		.data		= &sysctl_page_lock_unfairness,
-		.maxlen		= sizeof(sysctl_page_lock_unfairness),
-		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
-		.extra1		= SYSCTL_ZERO,
-	}
-};
-
 void __init pagecache_init(void)
 {
-	int i;
-
-	for (i = 0; i < PAGE_WAIT_TABLE_SIZE; i++)
-		init_waitqueue_head(&folio_wait_table[i]);
-
+	folio_wait_init();
 	page_writeback_init();
-	register_sysctl_init("vm", filemap_sysctl_table);
-}
-
-/*
- * The page wait code treats the "wait->flags" somewhat unusually, because
- * we have multiple different kinds of waits, not just the usual "exclusive"
- * one.
- *
- * We have:
- *
- *  (a) no special bits set:
- *
- *	We're just waiting for the bit to be released, and when a waker
- *	calls the wakeup function, we set WQ_FLAG_WOKEN and wake it up,
- *	and remove it from the wait queue.
- *
- *	Simple and straightforward.
- *
- *  (b) WQ_FLAG_EXCLUSIVE:
- *
- *	The waiter is waiting to get the lock, and only one waiter should
- *	be woken up to avoid any thundering herd behavior. We'll set the
- *	WQ_FLAG_WOKEN bit, wake it up, and remove it from the wait queue.
- *
- *	This is the traditional exclusive wait.
- *
- *  (c) WQ_FLAG_EXCLUSIVE | WQ_FLAG_CUSTOM:
- *
- *	The waiter is waiting to get the bit, and additionally wants the
- *	lock to be transferred to it for fair lock behavior. If the lock
- *	cannot be taken, we stop walking the wait queue without waking
- *	the waiter.
- *
- *	This is the "fair lock handoff" case, and in addition to setting
- *	WQ_FLAG_WOKEN, we set WQ_FLAG_DONE to let the waiter easily see
- *	that it now has the lock.
- */
-static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, void *arg)
-{
-	unsigned int flags;
-	struct wait_page_key *key = arg;
-	struct wait_page_queue *wait_page
-		= container_of(wait, struct wait_page_queue, wait);
-
-	if (!wake_page_match(wait_page, key))
-		return 0;
-
-	/*
-	 * If it's a lock handoff wait, we get the bit for it, and
-	 * stop walking (and do not wake it up) if we can't.
-	 */
-	flags = wait->flags;
-	if (flags & WQ_FLAG_EXCLUSIVE) {
-		if (test_bit(key->bit_nr, &key->folio->flags.f))
-			return -1;
-		if (flags & WQ_FLAG_CUSTOM) {
-			if (test_and_set_bit(key->bit_nr, &key->folio->flags.f))
-				return -1;
-			flags |= WQ_FLAG_DONE;
-		}
-	}
-
-	/*
-	 * We are holding the wait-queue lock, but the waiter that
-	 * is waiting for this will be checking the flags without
-	 * any locking.
-	 *
-	 * So update the flags atomically, and wake up the waiter
-	 * afterwards to avoid any races. This store-release pairs
-	 * with the load-acquire in folio_wait_bit_common().
-	 */
-	smp_store_release(&wait->flags, flags | WQ_FLAG_WOKEN);
-	wake_up_state(wait->private, mode);
-
-	/*
-	 * Ok, we have successfully done what we're waiting for,
-	 * and we can unconditionally remove the wait entry.
-	 *
-	 * Note that this pairs with the "finish_wait()" in the
-	 * waiter, and has to be the absolute last thing we do.
-	 * After this list_del_init(&wait->entry) the wait entry
-	 * might be de-allocated and the process might even have
-	 * exited.
-	 */
-	list_del_init_careful(&wait->entry);
-	return (flags & WQ_FLAG_EXCLUSIVE) != 0;
-}
-
-static void folio_wake_bit(struct folio *folio, int bit_nr)
-{
-	wait_queue_head_t *q = folio_waitqueue(folio);
-	struct wait_page_key key;
-	unsigned long flags;
-
-	key.folio = folio;
-	key.bit_nr = bit_nr;
-	key.page_match = 0;
-
-	spin_lock_irqsave(&q->lock, flags);
-	__wake_up_locked_key(q, TASK_NORMAL, &key);
-
-	/*
-	 * It's possible to miss clearing waiters here, when we woke our page
-	 * waiters, but the hashed waitqueue has waiters for other pages on it.
-	 * That's okay, it's a rare case. The next waker will clear it.
-	 *
-	 * Note that, depending on the page pool (buddy, hugetlb, ZONE_DEVICE,
-	 * other), the flag may be cleared in the course of freeing the page;
-	 * but that is not required for correctness.
-	 */
-	if (!waitqueue_active(q) || !key.page_match)
-		folio_clear_waiters(folio);
-
-	spin_unlock_irqrestore(&q->lock, flags);
-}
-
-/*
- * Wake waiters on PG_writeback for @folio.
- */
-static void folio_wake_writeback(struct folio *folio)
-{
-	folio_wake_bit(folio, PG_writeback);
-}
-
-/*
- * A choice of three behaviors for folio_wait_bit_common():
- */
-enum behavior {
-	EXCLUSIVE,	/* Hold ref to page and take the bit when woken, like
-			 * __folio_lock() waiting on then setting PG_locked.
-			 */
-	SHARED,		/* Hold ref to page and check the bit when woken, like
-			 * folio_wait_writeback() waiting on PG_writeback.
-			 */
-	DROP,		/* Drop ref to page before wait, no check when woken,
-			 * like folio_put_wait_locked() on PG_locked.
-			 */
-};
-
-/*
- * Attempt to check (or get) the folio flag, and mark us done
- * if successful.
- */
-static inline bool folio_trylock_flag(struct folio *folio, int bit_nr,
-					struct wait_queue_entry *wait)
-{
-	if (wait->flags & WQ_FLAG_EXCLUSIVE) {
-		if (test_and_set_bit(bit_nr, &folio->flags.f))
-			return false;
-	} else if (test_bit(bit_nr, &folio->flags.f))
-		return false;
-
-	wait->flags |= WQ_FLAG_WOKEN | WQ_FLAG_DONE;
-	return true;
-}
-
-static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
-		int state, enum behavior behavior)
-{
-	wait_queue_head_t *q = folio_waitqueue(folio);
-	int unfairness = sysctl_page_lock_unfairness;
-	struct wait_page_queue wait_page;
-	wait_queue_entry_t *wait = &wait_page.wait;
-	bool thrashing = false;
-	unsigned long pflags;
-	bool in_thrashing;
-
-	if (bit_nr == PG_locked &&
-	    !folio_test_uptodate(folio) && folio_test_workingset(folio)) {
-		delayacct_thrashing_start(&in_thrashing);
-		psi_memstall_enter(&pflags);
-		thrashing = true;
-	}
-
-	init_wait(wait);
-	wait->func = wake_page_function;
-	wait_page.folio = folio;
-	wait_page.bit_nr = bit_nr;
-
-repeat:
-	wait->flags = 0;
-	if (behavior == EXCLUSIVE) {
-		wait->flags = WQ_FLAG_EXCLUSIVE;
-		if (--unfairness < 0)
-			wait->flags |= WQ_FLAG_CUSTOM;
-	}
-
-	/*
-	 * Do one last check whether we can get the
-	 * page bit synchronously.
-	 *
-	 * Do the folio_set_waiters() marking before that
-	 * to let any waker we _just_ missed know they
-	 * need to wake us up (otherwise they'll never
-	 * even go to the slow case that looks at the
-	 * page queue), and add ourselves to the wait
-	 * queue if we need to sleep.
-	 *
-	 * This part needs to be done under the queue
-	 * lock to avoid races.
-	 */
-	spin_lock_irq(&q->lock);
-	folio_set_waiters(folio);
-	if (!folio_trylock_flag(folio, bit_nr, wait))
-		__add_wait_queue_entry_tail(q, wait);
-	spin_unlock_irq(&q->lock);
-
-	/*
-	 * From now on, all the logic will be based on
-	 * the WQ_FLAG_WOKEN and WQ_FLAG_DONE flag, to
-	 * see whether the page bit testing has already
-	 * been done by the wake function.
-	 *
-	 * We can drop our reference to the folio.
-	 */
-	if (behavior == DROP)
-		folio_put(folio);
-
-	/*
-	 * Note that until the "finish_wait()", or until
-	 * we see the WQ_FLAG_WOKEN flag, we need to
-	 * be very careful with the 'wait->flags', because
-	 * we may race with a waker that sets them.
-	 */
-	for (;;) {
-		unsigned int flags;
-
-		set_current_state(state);
-
-		/* Loop until we've been woken or interrupted */
-		flags = smp_load_acquire(&wait->flags);
-		if (!(flags & WQ_FLAG_WOKEN)) {
-			if (signal_pending_state(state, current))
-				break;
-
-			io_schedule();
-			continue;
-		}
-
-		/* If we were non-exclusive, we're done */
-		if (behavior != EXCLUSIVE)
-			break;
-
-		/* If the waker got the lock for us, we're done */
-		if (flags & WQ_FLAG_DONE)
-			break;
-
-		/*
-		 * Otherwise, if we're getting the lock, we need to
-		 * try to get it ourselves.
-		 *
-		 * And if that fails, we'll have to retry this all.
-		 */
-		if (unlikely(test_and_set_bit(bit_nr, folio_flags(folio, 0))))
-			goto repeat;
-
-		wait->flags |= WQ_FLAG_DONE;
-		break;
-	}
-
-	/*
-	 * If a signal happened, this 'finish_wait()' may remove the last
-	 * waiter from the wait-queues, but the folio waiters bit will remain
-	 * set. That's ok. The next wakeup will take care of it, and trying
-	 * to do it here would be difficult and prone to races.
-	 */
-	finish_wait(q, wait);
-
-	if (thrashing) {
-		delayacct_thrashing_end(&in_thrashing);
-		psi_memstall_leave(&pflags);
-	}
-
-	/*
-	 * NOTE! The wait->flags weren't stable until we've done the
-	 * 'finish_wait()', and we could have exited the loop above due
-	 * to a signal, and had a wakeup event happen after the signal
-	 * test but before the 'finish_wait()'.
-	 *
-	 * So only after the finish_wait() can we reliably determine
-	 * if we got woken up or not, so we can now figure out the final
-	 * return value based on that state without races.
-	 *
-	 * Also note that WQ_FLAG_WOKEN is sufficient for a non-exclusive
-	 * waiter, but an exclusive one requires WQ_FLAG_DONE.
-	 */
-	if (behavior == EXCLUSIVE)
-		return wait->flags & WQ_FLAG_DONE ? 0 : -EINTR;
-
-	return wait->flags & WQ_FLAG_WOKEN ? 0 : -EINTR;
 }
 
-#ifdef CONFIG_MIGRATION
-/**
- * softleaf_entry_wait_on_locked - Wait for a migration entry or
- * device_private entry to be removed.
- * @entry: migration or device_private swap entry.
- * @ptl: already locked ptl. This function will drop the lock.
- *
- * Wait for a migration entry referencing the given page, or device_private
- * entry referencing a dvice_private page to be unlocked. This is
- * equivalent to folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE) except
- * this can be called without taking a reference on the page. Instead this
- * should be called while holding the ptl for @entry referencing
- * the page.
- *
- * Returns after unlocking the ptl.
- *
- * This follows the same logic as folio_wait_bit_common() so see the comments
- * there.
- */
-void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *ptl)
-	__releases(ptl)
-{
-	struct wait_page_queue wait_page;
-	wait_queue_entry_t *wait = &wait_page.wait;
-	bool thrashing = false;
-	unsigned long pflags;
-	bool in_thrashing;
-	wait_queue_head_t *q;
-	struct folio *folio = softleaf_to_folio(entry);
-
-	q = folio_waitqueue(folio);
-	if (!folio_test_uptodate(folio) && folio_test_workingset(folio)) {
-		delayacct_thrashing_start(&in_thrashing);
-		psi_memstall_enter(&pflags);
-		thrashing = true;
-	}
-
-	init_wait(wait);
-	wait->func = wake_page_function;
-	wait_page.folio = folio;
-	wait_page.bit_nr = PG_locked;
-	wait->flags = 0;
-
-	spin_lock_irq(&q->lock);
-	folio_set_waiters(folio);
-	if (!folio_trylock_flag(folio, PG_locked, wait))
-		__add_wait_queue_entry_tail(q, wait);
-	spin_unlock_irq(&q->lock);
-
-	/*
-	 * If a migration entry exists for the page the migration path must hold
-	 * a valid reference to the page, and it must take the ptl to remove the
-	 * migration entry. So the page is valid until the ptl is dropped.
-	 * Similarly any path attempting to drop the last reference to a
-	 * device-private page needs to grab the ptl to remove the device-private
-	 * entry.
-	 */
-	spin_unlock(ptl);
-
-	for (;;) {
-		unsigned int flags;
-
-		set_current_state(TASK_UNINTERRUPTIBLE);
-
-		/* Loop until we've been woken or interrupted */
-		flags = smp_load_acquire(&wait->flags);
-		if (!(flags & WQ_FLAG_WOKEN)) {
-			if (signal_pending_state(TASK_UNINTERRUPTIBLE, current))
-				break;
-
-			io_schedule();
-			continue;
-		}
-		break;
-	}
-
-	finish_wait(q, wait);
-
-	if (thrashing) {
-		delayacct_thrashing_end(&in_thrashing);
-		psi_memstall_leave(&pflags);
-	}
-}
-#endif
-
-void folio_wait_bit(struct folio *folio, int bit_nr)
-{
-	folio_wait_bit_common(folio, bit_nr, TASK_UNINTERRUPTIBLE, SHARED);
-}
-EXPORT_SYMBOL(folio_wait_bit);
-
-int folio_wait_bit_killable(struct folio *folio, int bit_nr)
-{
-	return folio_wait_bit_common(folio, bit_nr, TASK_KILLABLE, SHARED);
-}
-EXPORT_SYMBOL(folio_wait_bit_killable);
-
-/**
- * folio_put_wait_locked - Drop a reference and wait for it to be unlocked
- * @folio: The folio to wait for.
- * @state: The sleep state (TASK_KILLABLE, TASK_UNINTERRUPTIBLE, etc).
- *
- * The caller should hold a reference on @folio.  They expect the page to
- * become unlocked relatively soon, but do not wish to hold up migration
- * (for example) by holding the reference while waiting for the folio to
- * come unlocked.  After this function returns, the caller should not
- * dereference @folio.
- *
- * Return: 0 if the folio was unlocked or -EINTR if interrupted by a signal.
- */
-static int folio_put_wait_locked(struct folio *folio, int state)
-{
-	return folio_wait_bit_common(folio, PG_locked, state, DROP);
-}
-
-/**
- * folio_unlock - Unlock a locked folio.
- * @folio: The folio.
- *
- * Unlocks the folio and wakes up any thread sleeping on the page lock.
- *
- * Context: May be called from interrupt or process context.  May not be
- * called from NMI context.
- */
-void folio_unlock(struct folio *folio)
-{
-	/* Bit 7 allows x86 to check the byte's sign bit */
-	BUILD_BUG_ON(PG_waiters != 7);
-	BUILD_BUG_ON(PG_locked > 7);
-	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
-	if (folio_xor_flags_has_waiters(folio, 1 << PG_locked))
-		folio_wake_bit(folio, PG_locked);
-}
-EXPORT_SYMBOL(folio_unlock);
-
-/**
- * folio_end_read - End read on a folio.
- * @folio: The folio.
- * @success: True if all reads completed successfully.
- *
- * When all reads against a folio have completed, filesystems should
- * call this function to let the pagecache know that no more reads
- * are outstanding.  This will unlock the folio and wake up any thread
- * sleeping on the lock.  The folio will also be marked uptodate if all
- * reads succeeded.
- *
- * Context: May be called from interrupt or process context.  May not be
- * called from NMI context.
- */
-void folio_end_read(struct folio *folio, bool success)
-{
-	unsigned long mask = 1 << PG_locked;
-
-	/* Must be in bottom byte for x86 to work */
-	BUILD_BUG_ON(PG_uptodate > 7);
-	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
-	VM_BUG_ON_FOLIO(success && folio_test_uptodate(folio), folio);
-
-	if (likely(success))
-		mask |= 1 << PG_uptodate;
-	if (folio_xor_flags_has_waiters(folio, mask))
-		folio_wake_bit(folio, PG_locked);
-}
-EXPORT_SYMBOL(folio_end_read);
-
-/**
- * folio_end_private_2 - Clear PG_private_2 and wake any waiters.
- * @folio: The folio.
- *
- * Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for
- * it.  The folio reference held for PG_private_2 being set is released.
- *
- * This is, for example, used when a netfs folio is being written to a local
- * disk cache, thereby allowing writes to the cache for the same folio to be
- * serialised.
- */
-void folio_end_private_2(struct folio *folio)
-{
-	VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);
-	clear_bit_unlock(PG_private_2, folio_flags(folio, 0));
-	folio_wake_bit(folio, PG_private_2);
-	folio_put(folio);
-}
-EXPORT_SYMBOL(folio_end_private_2);
-
-/**
- * folio_wait_private_2 - Wait for PG_private_2 to be cleared on a folio.
- * @folio: The folio to wait on.
- *
- * Wait for PG_private_2 to be cleared on a folio.
- */
-void folio_wait_private_2(struct folio *folio)
-{
-	while (folio_test_private_2(folio))
-		folio_wait_bit(folio, PG_private_2);
-}
-EXPORT_SYMBOL(folio_wait_private_2);
-
-/**
- * folio_wait_private_2_killable - Wait for PG_private_2 to be cleared on a folio.
- * @folio: The folio to wait on.
- *
- * Wait for PG_private_2 to be cleared on a folio or until a fatal signal is
- * received by the calling task.
- *
- * Return:
- * - 0 if successful.
- * - -EINTR if a fatal signal was encountered.
- */
-int folio_wait_private_2_killable(struct folio *folio)
-{
-	int ret = 0;
-
-	while (folio_test_private_2(folio)) {
-		ret = folio_wait_bit_killable(folio, PG_private_2);
-		if (ret < 0)
-			break;
-	}
-
-	return ret;
-}
-EXPORT_SYMBOL(folio_wait_private_2_killable);
-
 static void filemap_end_dropbehind(struct folio *folio)
 {
 	struct address_space *mapping = folio->mapping;
@@ -1703,95 +1154,6 @@ void folio_end_writeback(struct folio *folio)
 }
 EXPORT_SYMBOL(folio_end_writeback);
 
-/**
- * __folio_lock - Get a lock on the folio, assuming we need to sleep to get it.
- * @folio: The folio to lock
- */
-void __folio_lock(struct folio *folio)
-{
-	folio_wait_bit_common(folio, PG_locked, TASK_UNINTERRUPTIBLE,
-				EXCLUSIVE);
-}
-EXPORT_SYMBOL(__folio_lock);
-
-int __folio_lock_killable(struct folio *folio)
-{
-	return folio_wait_bit_common(folio, PG_locked, TASK_KILLABLE,
-					EXCLUSIVE);
-}
-EXPORT_SYMBOL_GPL(__folio_lock_killable);
-
-static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait)
-{
-	struct wait_queue_head *q = folio_waitqueue(folio);
-	int ret;
-
-	wait->folio = folio;
-	wait->bit_nr = PG_locked;
-
-	spin_lock_irq(&q->lock);
-	__add_wait_queue_entry_tail(q, &wait->wait);
-	folio_set_waiters(folio);
-	ret = !folio_trylock(folio);
-	/*
-	 * If we were successful now, we know we're still on the
-	 * waitqueue as we're still under the lock. This means it's
-	 * safe to remove and return success, we know the callback
-	 * isn't going to trigger.
-	 */
-	if (!ret)
-		__remove_wait_queue(q, &wait->wait);
-	else
-		ret = -EIOCBQUEUED;
-	spin_unlock_irq(&q->lock);
-	return ret;
-}
-
-/*
- * Return values:
- * 0 - folio is locked.
- * non-zero - folio is not locked.
- *     mmap_lock or per-VMA lock has been released (mmap_read_unlock() or
- *     vma_end_read()), unless flags had both FAULT_FLAG_ALLOW_RETRY and
- *     FAULT_FLAG_RETRY_NOWAIT set, in which case the lock is still held.
- *
- * If neither ALLOW_RETRY nor KILLABLE are set, will always return 0
- * with the folio locked and the mmap_lock/per-VMA lock is left unperturbed.
- */
-vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf)
-{
-	unsigned int flags = vmf->flags;
-
-	if (fault_flag_allow_retry_first(flags)) {
-		/*
-		 * CAUTION! In this case, mmap_lock/per-VMA lock is not
-		 * released even though returning VM_FAULT_RETRY.
-		 */
-		if (flags & FAULT_FLAG_RETRY_NOWAIT)
-			return VM_FAULT_RETRY;
-
-		release_fault_lock(vmf);
-		if (flags & FAULT_FLAG_KILLABLE)
-			folio_wait_locked_killable(folio);
-		else
-			folio_wait_locked(folio);
-		return VM_FAULT_RETRY;
-	}
-	if (flags & FAULT_FLAG_KILLABLE) {
-		bool ret;
-
-		ret = __folio_lock_killable(folio);
-		if (ret) {
-			release_fault_lock(vmf);
-			return VM_FAULT_RETRY;
-		}
-	} else {
-		__folio_lock(folio);
-	}
-
-	return 0;
-}
-
 /**
  * page_cache_next_miss() - Find the next gap in the page cache.
  * @mapping: Mapping.
diff --git a/mm/folio_wait.c b/mm/folio_wait.c
new file mode 100644
index 000000000000..18b42488ce37
--- /dev/null
+++ b/mm/folio_wait.c
@@ -0,0 +1,662 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Folio bit-lock and wait-queue infrastructure.
+ */
+#include <linux/compiler.h>
+#include <linux/export.h>
+#include <linux/sched/signal.h>
+#include <linux/mm.h>
+#include <linux/swap.h>
+#include <linux/leafops.h>
+#include <linux/pagemap.h>
+#include <linux/wait.h>
+#include <linux/hash.h>
+#include <linux/sysctl.h>
+#include <linux/delayacct.h>
+#include <linux/psi.h>
+#include <linux/migrate.h>
+
+#include "internal.h"
+
+/*
+ * In order to wait for pages to become available there must be
+ * waitqueues associated with pages. By using a hash table of
+ * waitqueues where the bucket discipline is to maintain all
+ * waiters on the same queue and wake all when any of the pages
+ * become available, and for the woken contexts to check to be
+ * sure the appropriate page became available, this saves space
+ * at a cost of "thundering herd" phenomena during rare hash
+ * collisions.
+ */
+#define PAGE_WAIT_TABLE_BITS 8
+#define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS)
+static wait_queue_head_t folio_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned;
+
+static wait_queue_head_t *folio_waitqueue(struct folio *folio)
+{
+	return &folio_wait_table[hash_ptr(folio, PAGE_WAIT_TABLE_BITS)];
+}
+
+/* How many times do we accept lock stealing from under a waiter? */
+static int sysctl_page_lock_unfairness = 5;
+static const struct ctl_table folio_wait_sysctl_table[] = {
+	{
+		.procname	= "page_lock_unfairness",
+		.data		= &sysctl_page_lock_unfairness,
+		.maxlen		= sizeof(sysctl_page_lock_unfairness),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+	}
+};
+
+void __init folio_wait_init(void)
+{
+	int i;
+
+	for (i = 0; i < PAGE_WAIT_TABLE_SIZE; i++)
+		init_waitqueue_head(&folio_wait_table[i]);
+
+	register_sysctl_init("vm", folio_wait_sysctl_table);
+}
+
+/*
+ * The page wait code treats the "wait->flags" somewhat unusually, because
+ * we have multiple different kinds of waits, not just the usual "exclusive"
+ * one.
+ *
+ * We have:
+ *
+ *  (a) no special bits set:
+ *
+ *	We're just waiting for the bit to be released, and when a waker
+ *	calls the wakeup function, we set WQ_FLAG_WOKEN and wake it up,
+ *	and remove it from the wait queue.
+ *
+ *	Simple and straightforward.
+ *
+ *  (b) WQ_FLAG_EXCLUSIVE:
+ *
+ *	The waiter is waiting to get the lock, and only one waiter should
+ *	be woken up to avoid any thundering herd behavior. We'll set the
+ *	WQ_FLAG_WOKEN bit, wake it up, and remove it from the wait queue.
+ *
+ *	This is the traditional exclusive wait.
+ *
+ *  (c) WQ_FLAG_EXCLUSIVE | WQ_FLAG_CUSTOM:
+ *
+ *	The waiter is waiting to get the bit, and additionally wants the
+ *	lock to be transferred to it for fair lock behavior. If the lock
+ *	cannot be taken, we stop walking the wait queue without waking
+ *	the waiter.
+ *
+ *	This is the "fair lock handoff" case, and in addition to setting
+ *	WQ_FLAG_WOKEN, we set WQ_FLAG_DONE to let the waiter easily see
+ *	that it now has the lock.
+ */
+static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, void *arg)
+{
+	unsigned int flags;
+	struct wait_page_key *key = arg;
+	struct wait_page_queue *wait_page
+		= container_of(wait, struct wait_page_queue, wait);
+
+	if (!wake_page_match(wait_page, key))
+		return 0;
+
+	/*
+	 * If it's a lock handoff wait, we get the bit for it, and
+	 * stop walking (and do not wake it up) if we can't.
+	 */
+	flags = wait->flags;
+	if (flags & WQ_FLAG_EXCLUSIVE) {
+		if (test_bit(key->bit_nr, &key->folio->flags.f))
+			return -1;
+		if (flags & WQ_FLAG_CUSTOM) {
+			if (test_and_set_bit(key->bit_nr, &key->folio->flags.f))
+				return -1;
+			flags |= WQ_FLAG_DONE;
+		}
+	}
+
+	/*
+	 * We are holding the wait-queue lock, but the waiter that
+	 * is waiting for this will be checking the flags without
+	 * any locking.
+	 *
+	 * So update the flags atomically, and wake up the waiter
+	 * afterwards to avoid any races. This store-release pairs
+	 * with the load-acquire in folio_wait_bit_common().
+	 */
+	smp_store_release(&wait->flags, flags | WQ_FLAG_WOKEN);
+	wake_up_state(wait->private, mode);
+
+	/*
+	 * Ok, we have successfully done what we're waiting for,
+	 * and we can unconditionally remove the wait entry.
+	 *
+	 * Note that this pairs with the "finish_wait()" in the
+	 * waiter, and has to be the absolute last thing we do.
+	 * After this list_del_init(&wait->entry) the wait entry
+	 * might be de-allocated and the process might even have
+	 * exited.
+	 */
+	list_del_init_careful(&wait->entry);
+	return (flags & WQ_FLAG_EXCLUSIVE) != 0;
+}
+
+static void folio_wake_bit(struct folio *folio, int bit_nr)
+{
+	wait_queue_head_t *q = folio_waitqueue(folio);
+	struct wait_page_key key;
+	unsigned long flags;
+
+	key.folio = folio;
+	key.bit_nr = bit_nr;
+	key.page_match = 0;
+
+	spin_lock_irqsave(&q->lock, flags);
+	__wake_up_locked_key(q, TASK_NORMAL, &key);
+
+	/*
+	 * It's possible to miss clearing waiters here, when we woke our page
+	 * waiters, but the hashed waitqueue has waiters for other pages on it.
+	 * That's okay, it's a rare case. The next waker will clear it.
+	 *
+	 * Note that, depending on the page pool (buddy, hugetlb, ZONE_DEVICE,
+	 * other), the flag may be cleared in the course of freeing the page;
+	 * but that is not required for correctness.
+	 */
+	if (!waitqueue_active(q) || !key.page_match)
+		folio_clear_waiters(folio);
+
+	spin_unlock_irqrestore(&q->lock, flags);
+}
+
+/*
+ * Wake waiters on PG_writeback for @folio.
+ */
+void folio_wake_writeback(struct folio *folio)
+{
+	folio_wake_bit(folio, PG_writeback);
+}
+
+/*
+ * A choice of three behaviors for folio_wait_bit_common():
+ */
+enum behavior {
+	EXCLUSIVE,	/* Hold ref to page and take the bit when woken, like
+			 * __folio_lock() waiting on then setting PG_locked.
+			 */
+	SHARED,		/* Hold ref to page and check the bit when woken, like
+			 * folio_wait_writeback() waiting on PG_writeback.
+			 */
+	DROP,		/* Drop ref to page before wait, no check when woken,
+			 * like folio_put_wait_locked() on PG_locked.
+			 */
+};
+
+/*
+ * Attempt to check (or get) the folio flag, and mark us done
+ * if successful.
+ */
+static inline bool folio_trylock_flag(struct folio *folio, int bit_nr,
+					struct wait_queue_entry *wait)
+{
+	if (wait->flags & WQ_FLAG_EXCLUSIVE) {
+		if (test_and_set_bit(bit_nr, &folio->flags.f))
+			return false;
+	} else if (test_bit(bit_nr, &folio->flags.f))
+		return false;
+
+	wait->flags |= WQ_FLAG_WOKEN | WQ_FLAG_DONE;
+	return true;
+}
+
+static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
+		int state, enum behavior behavior)
+{
+	wait_queue_head_t *q = folio_waitqueue(folio);
+	int unfairness = sysctl_page_lock_unfairness;
+	struct wait_page_queue wait_page;
+	wait_queue_entry_t *wait = &wait_page.wait;
+	bool thrashing = false;
+	unsigned long pflags;
+	bool in_thrashing;
+
+	if (bit_nr == PG_locked &&
+	    !folio_test_uptodate(folio) && folio_test_workingset(folio)) {
+		delayacct_thrashing_start(&in_thrashing);
+		psi_memstall_enter(&pflags);
+		thrashing = true;
+	}
+
+	init_wait(wait);
+	wait->func = wake_page_function;
+	wait_page.folio = folio;
+	wait_page.bit_nr = bit_nr;
+
+repeat:
+	wait->flags = 0;
+	if (behavior == EXCLUSIVE) {
+		wait->flags = WQ_FLAG_EXCLUSIVE;
+		if (--unfairness < 0)
+			wait->flags |= WQ_FLAG_CUSTOM;
+	}
+
+	/*
+	 * Do one last check whether we can get the
+	 * page bit synchronously.
+	 *
+	 * Do the folio_set_waiters() marking before that
+	 * to let any waker we _just_ missed know they
+	 * need to wake us up (otherwise they'll never
+	 * even go to the slow case that looks at the
+	 * page queue), and add ourselves to the wait
+	 * queue if we need to sleep.
+	 *
+	 * This part needs to be done under the queue
+	 * lock to avoid races.
+	 */
+	spin_lock_irq(&q->lock);
+	folio_set_waiters(folio);
+	if (!folio_trylock_flag(folio, bit_nr, wait))
+		__add_wait_queue_entry_tail(q, wait);
+	spin_unlock_irq(&q->lock);
+
+	/*
+	 * From now on, all the logic will be based on
+	 * the WQ_FLAG_WOKEN and WQ_FLAG_DONE flag, to
+	 * see whether the page bit testing has already
+	 * been done by the wake function.
+	 *
+	 * We can drop our reference to the folio.
+	 */
+	if (behavior == DROP)
+		folio_put(folio);
+
+	/*
+	 * Note that until the "finish_wait()", or until
+	 * we see the WQ_FLAG_WOKEN flag, we need to
+	 * be very careful with the 'wait->flags', because
+	 * we may race with a waker that sets them.
+	 */
+	for (;;) {
+		unsigned int flags;
+
+		set_current_state(state);
+
+		/* Loop until we've been woken or interrupted */
+		flags = smp_load_acquire(&wait->flags);
+		if (!(flags & WQ_FLAG_WOKEN)) {
+			if (signal_pending_state(state, current))
+				break;
+
+			io_schedule();
+			continue;
+		}
+
+		/* If we were non-exclusive, we're done */
+		if (behavior != EXCLUSIVE)
+			break;
+
+		/* If the waker got the lock for us, we're done */
+		if (flags & WQ_FLAG_DONE)
+			break;
+
+		/*
+		 * Otherwise, if we're getting the lock, we need to
+		 * try to get it ourselves.
+		 *
+		 * And if that fails, we'll have to retry this all.
+		 */
+		if (unlikely(test_and_set_bit(bit_nr, folio_flags(folio, 0))))
+			goto repeat;
+
+		wait->flags |= WQ_FLAG_DONE;
+		break;
+	}
+
+	/*
+	 * If a signal happened, this 'finish_wait()' may remove the last
+	 * waiter from the wait-queues, but the folio waiters bit will remain
+	 * set. That's ok. The next wakeup will take care of it, and trying
+	 * to do it here would be difficult and prone to races.
+	 */
+	finish_wait(q, wait);
+
+	if (thrashing) {
+		delayacct_thrashing_end(&in_thrashing);
+		psi_memstall_leave(&pflags);
+	}
+
+	/*
+	 * NOTE! The wait->flags weren't stable until we've done the
+	 * 'finish_wait()', and we could have exited the loop above due
+	 * to a signal, and had a wakeup event happen after the signal
+	 * test but before the 'finish_wait()'.
+	 *
+	 * So only after the finish_wait() can we reliably determine
+	 * if we got woken up or not, so we can now figure out the final
+	 * return value based on that state without races.
+	 *
+	 * Also note that WQ_FLAG_WOKEN is sufficient for a non-exclusive
+	 * waiter, but an exclusive one requires WQ_FLAG_DONE.
+	 */
+	if (behavior == EXCLUSIVE)
+		return wait->flags & WQ_FLAG_DONE ? 0 : -EINTR;
+
+	return wait->flags & WQ_FLAG_WOKEN ? 0 : -EINTR;
+}
+
+#ifdef CONFIG_MIGRATION
+/**
+ * softleaf_entry_wait_on_locked - Wait for a migration entry or
+ * device_private entry to be removed.
+ * @entry: migration or device_private swap entry.
+ * @ptl: already locked ptl. This function will drop the lock.
+ *
+ * Wait for a migration entry referencing the given page, or device_private
+ * entry referencing a dvice_private page to be unlocked. This is
+ * equivalent to folio_put_wait_locked(folio, TASK_UNINTERRUPTIBLE) except
+ * this can be called without taking a reference on the page. Instead this
+ * should be called while holding the ptl for @entry referencing
+ * the page.
+ *
+ * Returns after unlocking the ptl.
+ *
+ * This follows the same logic as folio_wait_bit_common() so see the comments
+ * there.
+ */
+void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *ptl)
+	__releases(ptl)
+{
+	struct wait_page_queue wait_page;
+	wait_queue_entry_t *wait = &wait_page.wait;
+	bool thrashing = false;
+	unsigned long pflags;
+	bool in_thrashing;
+	wait_queue_head_t *q;
+	struct folio *folio = softleaf_to_folio(entry);
+
+	q = folio_waitqueue(folio);
+	if (!folio_test_uptodate(folio) && folio_test_workingset(folio)) {
+		delayacct_thrashing_start(&in_thrashing);
+		psi_memstall_enter(&pflags);
+		thrashing = true;
+	}
+
+	init_wait(wait);
+	wait->func = wake_page_function;
+	wait_page.folio = folio;
+	wait_page.bit_nr = PG_locked;
+	wait->flags = 0;
+
+	spin_lock_irq(&q->lock);
+	folio_set_waiters(folio);
+	if (!folio_trylock_flag(folio, PG_locked, wait))
+		__add_wait_queue_entry_tail(q, wait);
+	spin_unlock_irq(&q->lock);
+
+	/*
+	 * If a migration entry exists for the page the migration path must hold
+	 * a valid reference to the page, and it must take the ptl to remove the
+	 * migration entry. So the page is valid until the ptl is dropped.
+	 * Similarly any path attempting to drop the last reference to a
+	 * device-private page needs to grab the ptl to remove the device-private
+	 * entry.
+	 */
+	spin_unlock(ptl);
+
+	for (;;) {
+		unsigned int flags;
+
+		set_current_state(TASK_UNINTERRUPTIBLE);
+
+		/* Loop until we've been woken or interrupted */
+		flags = smp_load_acquire(&wait->flags);
+		if (!(flags & WQ_FLAG_WOKEN)) {
+			if (signal_pending_state(TASK_UNINTERRUPTIBLE, current))
+				break;
+
+			io_schedule();
+			continue;
+		}
+		break;
+	}
+
+	finish_wait(q, wait);
+
+	if (thrashing) {
+		delayacct_thrashing_end(&in_thrashing);
+		psi_memstall_leave(&pflags);
+	}
+}
+#endif
+
+void folio_wait_bit(struct folio *folio, int bit_nr)
+{
+	folio_wait_bit_common(folio, bit_nr, TASK_UNINTERRUPTIBLE, SHARED);
+}
+EXPORT_SYMBOL(folio_wait_bit);
+
+int folio_wait_bit_killable(struct folio *folio, int bit_nr)
+{
+	return folio_wait_bit_common(folio, bit_nr, TASK_KILLABLE, SHARED);
+}
+EXPORT_SYMBOL(folio_wait_bit_killable);
+
+/**
+ * folio_put_wait_locked - Drop a reference and wait for it to be unlocked
+ * @folio: The folio to wait for.
+ * @state: The sleep state (TASK_KILLABLE, TASK_UNINTERRUPTIBLE, etc).
+ *
+ * The caller should hold a reference on @folio.  They expect the page to
+ * become unlocked relatively soon, but do not wish to hold up migration
+ * (for example) by holding the reference while waiting for the folio to
+ * come unlocked.  After this function returns, the caller should not
+ * dereference @folio.
+ *
+ * Return: 0 if the folio was unlocked or -EINTR if interrupted by a signal.
+ */
+int folio_put_wait_locked(struct folio *folio, int state)
+{
+	return folio_wait_bit_common(folio, PG_locked, state, DROP);
+}
+
+/**
+ * folio_unlock - Unlock a locked folio.
+ * @folio: The folio.
+ *
+ * Unlocks the folio and wakes up any thread sleeping on the page lock.
+ *
+ * Context: May be called from interrupt or process context.  May not be
+ * called from NMI context.
+ */
+void folio_unlock(struct folio *folio)
+{
+	/* Bit 7 allows x86 to check the byte's sign bit */
+	BUILD_BUG_ON(PG_waiters != 7);
+	BUILD_BUG_ON(PG_locked > 7);
+	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+	if (folio_xor_flags_has_waiters(folio, 1 << PG_locked))
+		folio_wake_bit(folio, PG_locked);
+}
+EXPORT_SYMBOL(folio_unlock);
+
+/**
+ * folio_end_read - End read on a folio.
+ * @folio: The folio.
+ * @success: True if all reads completed successfully.
+ *
+ * When all reads against a folio have completed, filesystems should
+ * call this function to let the pagecache know that no more reads
+ * are outstanding.  This will unlock the folio and wake up any thread
+ * sleeping on the lock.  The folio will also be marked uptodate if all
+ * reads succeeded.
+ *
+ * Context: May be called from interrupt or process context.  May not be
+ * called from NMI context.
+ */
+void folio_end_read(struct folio *folio, bool success)
+{
+	unsigned long mask = 1 << PG_locked;
+
+	/* Must be in bottom byte for x86 to work */
+	BUILD_BUG_ON(PG_uptodate > 7);
+	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+	VM_BUG_ON_FOLIO(success && folio_test_uptodate(folio), folio);
+
+	if (likely(success))
+		mask |= 1 << PG_uptodate;
+	if (folio_xor_flags_has_waiters(folio, mask))
+		folio_wake_bit(folio, PG_locked);
+}
+EXPORT_SYMBOL(folio_end_read);
+
+/**
+ * folio_end_private_2 - Clear PG_private_2 and wake any waiters.
+ * @folio: The folio.
+ *
+ * Clear the PG_private_2 bit on a folio and wake up any sleepers waiting for
+ * it.  The folio reference held for PG_private_2 being set is released.
+ *
+ * This is, for example, used when a netfs folio is being written to a local
+ * disk cache, thereby allowing writes to the cache for the same folio to be
+ * serialised.
+ */
+void folio_end_private_2(struct folio *folio)
+{
+	VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);
+	clear_bit_unlock(PG_private_2, folio_flags(folio, 0));
+	folio_wake_bit(folio, PG_private_2);
+	folio_put(folio);
+}
+EXPORT_SYMBOL(folio_end_private_2);
+
+/**
+ * folio_wait_private_2 - Wait for PG_private_2 to be cleared on a folio.
+ * @folio: The folio to wait on.
+ *
+ * Wait for PG_private_2 to be cleared on a folio.
+ */
+void folio_wait_private_2(struct folio *folio)
+{
+	while (folio_test_private_2(folio))
+		folio_wait_bit(folio, PG_private_2);
+}
+EXPORT_SYMBOL(folio_wait_private_2);
+
+/**
+ * folio_wait_private_2_killable - Wait for PG_private_2 to be cleared on a folio.
+ * @folio: The folio to wait on.
+ *
+ * Wait for PG_private_2 to be cleared on a folio or until a fatal signal is
+ * received by the calling task.
+ *
+ * Return:
+ * - 0 if successful.
+ * - -EINTR if a fatal signal was encountered.
+ */
+int folio_wait_private_2_killable(struct folio *folio)
+{
+	int ret = 0;
+
+	while (folio_test_private_2(folio)) {
+		ret = folio_wait_bit_killable(folio, PG_private_2);
+		if (ret < 0)
+			break;
+	}
+
+	return ret;
+}
+EXPORT_SYMBOL(folio_wait_private_2_killable);
+
+/**
+ * __folio_lock - Get a lock on the folio, assuming we need to sleep to get it.
+ * @folio: The folio to lock
+ */
+void __folio_lock(struct folio *folio)
+{
+	folio_wait_bit_common(folio, PG_locked, TASK_UNINTERRUPTIBLE,
+				EXCLUSIVE);
+}
+EXPORT_SYMBOL(__folio_lock);
+
+int __folio_lock_killable(struct folio *folio)
+{
+	return folio_wait_bit_common(folio, PG_locked, TASK_KILLABLE,
+					EXCLUSIVE);
+}
+EXPORT_SYMBOL_GPL(__folio_lock_killable);
+
+int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait)
+{
+	struct wait_queue_head *q = folio_waitqueue(folio);
+	int ret;
+
+	wait->folio = folio;
+	wait->bit_nr = PG_locked;
+
+	spin_lock_irq(&q->lock);
+	__add_wait_queue_entry_tail(q, &wait->wait);
+	folio_set_waiters(folio);
+	ret = !folio_trylock(folio);
+	/*
+	 * If we were successful now, we know we're still on the
+	 * waitqueue as we're still under the lock. This means it's
+	 * safe to remove and return success, we know the callback
+	 * isn't going to trigger.
+	 */
+	if (!ret)
+		__remove_wait_queue(q, &wait->wait);
+	else
+		ret = -EIOCBQUEUED;
+	spin_unlock_irq(&q->lock);
+	return ret;
+}
+
+/*
+ * Return values:
+ * 0 - folio is locked.
+ * non-zero - folio is not locked.
+ *     mmap_lock or per-VMA lock has been released (mmap_read_unlock() or
+ *     vma_end_read()), unless flags had both FAULT_FLAG_ALLOW_RETRY and
+ *     FAULT_FLAG_RETRY_NOWAIT set, in which case the lock is still held.
+ *
+ * If neither ALLOW_RETRY nor KILLABLE are set, will always return 0
+ * with the folio locked and the mmap_lock/per-VMA lock is left unperturbed.
+ */
+vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf)
+{
+	unsigned int flags = vmf->flags;
+
+	if (fault_flag_allow_retry_first(flags)) {
+		/*
+		 * CAUTION! In this case, mmap_lock/per-VMA lock is not
+		 * released even though returning VM_FAULT_RETRY.
+		 */
+		if (flags & FAULT_FLAG_RETRY_NOWAIT)
+			return VM_FAULT_RETRY;
+
+		release_fault_lock(vmf);
+		if (flags & FAULT_FLAG_KILLABLE)
+			folio_wait_locked_killable(folio);
+		else
+			folio_wait_locked(folio);
+		return VM_FAULT_RETRY;
+	}
+	if (flags & FAULT_FLAG_KILLABLE) {
+		bool ret;
+
+		ret = __folio_lock_killable(folio);
+		if (ret) {
+			release_fault_lock(vmf);
+			return VM_FAULT_RETRY;
+		}
+	} else {
+		__folio_lock(folio);
+	}
+
+	return 0;
+}
diff --git a/mm/internal.h b/mm/internal.h
index 09931b1e535f..a121ca07f75c 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -102,6 +102,10 @@ struct pagetable_move_control {
 })
 
 void page_writeback_init(void);
+void folio_wait_init(void);
+void folio_wake_writeback(struct folio *folio);
+int folio_put_wait_locked(struct folio *folio, int state);
+int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait);
 
 /*
  * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages,

-- 
2.39.5

[PATCH RFC 03/11] folio_wait: move folio bit-lock and wait declarations to include/linux/folio_wait.h

[Tal Zussman]

Move ~150 lines of folio bit-lock and wait queue infrastructure from
pagemap.h to folio_wait.h. pagemap.h includes the new header so existing
users don't break.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 include/linux/folio_wait.h | 181 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pagemap.h    | 172 +-----------------------------------------
 mm/folio_wait.c            |   2 +-
 3 files changed, 183 insertions(+), 172 deletions(-)

diff --git a/include/linux/folio_wait.h b/include/linux/folio_wait.h
new file mode 100644
index 000000000000..80ddf1ffcae4
--- /dev/null
+++ b/include/linux/folio_wait.h
@@ -0,0 +1,181 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_FOLIO_WAIT_H
+#define _LINUX_FOLIO_WAIT_H
+
+#include <linux/bitops.h>
+#include <linux/page-flags.h>
+#include <linux/wait.h>
+
+struct wait_page_key {
+	struct folio *folio;
+	int bit_nr;
+	int page_match;
+};
+
+struct wait_page_queue {
+	struct folio *folio;
+	int bit_nr;
+	wait_queue_entry_t wait;
+};
+
+static inline bool wake_page_match(struct wait_page_queue *wait_page,
+				  struct wait_page_key *key)
+{
+	if (wait_page->folio != key->folio)
+	       return false;
+	key->page_match = 1;
+
+	if (wait_page->bit_nr != key->bit_nr)
+		return false;
+
+	return true;
+}
+
+void __folio_lock(struct folio *folio);
+int __folio_lock_killable(struct folio *folio);
+vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf);
+void unlock_page(struct page *page);
+void folio_unlock(struct folio *folio);
+
+/**
+ * folio_trylock() - Attempt to lock a folio.
+ * @folio: The folio to attempt to lock.
+ *
+ * Sometimes it is undesirable to wait for a folio to be unlocked (eg
+ * when the locks are being taken in the wrong order, or if making
+ * progress through a batch of folios is more important than processing
+ * them in order).  Usually folio_lock() is the correct function to call.
+ *
+ * Context: Any context.
+ * Return: Whether the lock was successfully acquired.
+ */
+static inline bool folio_trylock(struct folio *folio)
+{
+	return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio, 0)));
+}
+
+/*
+ * Return true if the page was successfully locked
+ */
+static inline bool trylock_page(struct page *page)
+{
+	return folio_trylock(page_folio(page));
+}
+
+/**
+ * folio_lock() - Lock this folio.
+ * @folio: The folio to lock.
+ *
+ * The folio lock protects against many things, probably more than it
+ * should.  It is primarily held while a folio is being brought uptodate,
+ * either from its backing file or from swap.  It is also held while a
+ * folio is being truncated from its address_space, so holding the lock
+ * is sufficient to keep folio->mapping stable.
+ *
+ * The folio lock is also held while write() is modifying the page to
+ * provide POSIX atomicity guarantees (as long as the write does not
+ * cross a page boundary).  Other modifications to the data in the folio
+ * do not hold the folio lock and can race with writes, eg DMA and stores
+ * to mapped pages.
+ *
+ * Context: May sleep.  If you need to acquire the locks of two or
+ * more folios, they must be in order of ascending index, if they are
+ * in the same address_space.  If they are in different address_spaces,
+ * acquire the lock of the folio which belongs to the address_space which
+ * has the lowest address in memory first.
+ */
+static inline void folio_lock(struct folio *folio)
+{
+	might_sleep();
+	if (!folio_trylock(folio))
+		__folio_lock(folio);
+}
+
+/**
+ * lock_page() - Lock the folio containing this page.
+ * @page: The page to lock.
+ *
+ * See folio_lock() for a description of what the lock protects.
+ * This is a legacy function and new code should probably use folio_lock()
+ * instead.
+ *
+ * Context: May sleep.  Pages in the same folio share a lock, so do not
+ * attempt to lock two pages which share a folio.
+ */
+static inline void lock_page(struct page *page)
+{
+	struct folio *folio;
+	might_sleep();
+
+	folio = page_folio(page);
+	if (!folio_trylock(folio))
+		__folio_lock(folio);
+}
+
+/**
+ * folio_lock_killable() - Lock this folio, interruptible by a fatal signal.
+ * @folio: The folio to lock.
+ *
+ * Attempts to lock the folio, like folio_lock(), except that the sleep
+ * to acquire the lock is interruptible by a fatal signal.
+ *
+ * Context: May sleep; see folio_lock().
+ * Return: 0 if the lock was acquired; -EINTR if a fatal signal was received.
+ */
+static inline int folio_lock_killable(struct folio *folio)
+{
+	might_sleep();
+	if (!folio_trylock(folio))
+		return __folio_lock_killable(folio);
+	return 0;
+}
+
+/*
+ * folio_lock_or_retry - Lock the folio, unless this would block and the
+ * caller indicated that it can handle a retry.
+ *
+ * Return value and mmap_lock implications depend on flags; see
+ * __folio_lock_or_retry().
+ */
+static inline vm_fault_t folio_lock_or_retry(struct folio *folio,
+					     struct vm_fault *vmf)
+{
+	might_sleep();
+	if (!folio_trylock(folio))
+		return __folio_lock_or_retry(folio, vmf);
+	return 0;
+}
+
+/*
+ * This is exported only for folio_wait_locked/folio_wait_writeback, etc.,
+ * and should not be used directly.
+ */
+void folio_wait_bit(struct folio *folio, int bit_nr);
+int folio_wait_bit_killable(struct folio *folio, int bit_nr);
+
+/*
+ * Wait for a folio to be unlocked.
+ *
+ * This must be called with the caller "holding" the folio,
+ * ie with increased folio reference count so that the folio won't
+ * go away during the wait.
+ */
+static inline void folio_wait_locked(struct folio *folio)
+{
+	if (folio_test_locked(folio))
+		folio_wait_bit(folio, PG_locked);
+}
+
+static inline int folio_wait_locked_killable(struct folio *folio)
+{
+	if (!folio_test_locked(folio))
+		return 0;
+	return folio_wait_bit_killable(folio, PG_locked);
+}
+
+void folio_end_read(struct folio *folio, bool success);
+void folio_end_private_2(struct folio *folio);
+void folio_wait_private_2(struct folio *folio);
+int folio_wait_private_2_killable(struct folio *folio);
+
+#endif /* _LINUX_FOLIO_WAIT_H */
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 627771e82eb1..7f65c2b0097b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -15,6 +15,7 @@
 #include <linux/bitops.h>
 #include <linux/hardirq.h> /* for in_interrupt() */
 #include <linux/hugetlb_inline.h>
+#include <linux/folio_wait.h>
 
 struct folio_batch;
 
@@ -1072,174 +1073,6 @@ static inline pgoff_t linear_page_index(const struct vm_area_struct *vma,
 	return pgoff;
 }
 
-struct wait_page_key {
-	struct folio *folio;
-	int bit_nr;
-	int page_match;
-};
-
-struct wait_page_queue {
-	struct folio *folio;
-	int bit_nr;
-	wait_queue_entry_t wait;
-};
-
-static inline bool wake_page_match(struct wait_page_queue *wait_page,
-				  struct wait_page_key *key)
-{
-	if (wait_page->folio != key->folio)
-	       return false;
-	key->page_match = 1;
-
-	if (wait_page->bit_nr != key->bit_nr)
-		return false;
-
-	return true;
-}
-
-void __folio_lock(struct folio *folio);
-int __folio_lock_killable(struct folio *folio);
-vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf);
-void unlock_page(struct page *page);
-void folio_unlock(struct folio *folio);
-
-/**
- * folio_trylock() - Attempt to lock a folio.
- * @folio: The folio to attempt to lock.
- *
- * Sometimes it is undesirable to wait for a folio to be unlocked (eg
- * when the locks are being taken in the wrong order, or if making
- * progress through a batch of folios is more important than processing
- * them in order).  Usually folio_lock() is the correct function to call.
- *
- * Context: Any context.
- * Return: Whether the lock was successfully acquired.
- */
-static inline bool folio_trylock(struct folio *folio)
-{
-	return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio, 0)));
-}
-
-/*
- * Return true if the page was successfully locked
- */
-static inline bool trylock_page(struct page *page)
-{
-	return folio_trylock(page_folio(page));
-}
-
-/**
- * folio_lock() - Lock this folio.
- * @folio: The folio to lock.
- *
- * The folio lock protects against many things, probably more than it
- * should.  It is primarily held while a folio is being brought uptodate,
- * either from its backing file or from swap.  It is also held while a
- * folio is being truncated from its address_space, so holding the lock
- * is sufficient to keep folio->mapping stable.
- *
- * The folio lock is also held while write() is modifying the page to
- * provide POSIX atomicity guarantees (as long as the write does not
- * cross a page boundary).  Other modifications to the data in the folio
- * do not hold the folio lock and can race with writes, eg DMA and stores
- * to mapped pages.
- *
- * Context: May sleep.  If you need to acquire the locks of two or
- * more folios, they must be in order of ascending index, if they are
- * in the same address_space.  If they are in different address_spaces,
- * acquire the lock of the folio which belongs to the address_space which
- * has the lowest address in memory first.
- */
-static inline void folio_lock(struct folio *folio)
-{
-	might_sleep();
-	if (!folio_trylock(folio))
-		__folio_lock(folio);
-}
-
-/**
- * lock_page() - Lock the folio containing this page.
- * @page: The page to lock.
- *
- * See folio_lock() for a description of what the lock protects.
- * This is a legacy function and new code should probably use folio_lock()
- * instead.
- *
- * Context: May sleep.  Pages in the same folio share a lock, so do not
- * attempt to lock two pages which share a folio.
- */
-static inline void lock_page(struct page *page)
-{
-	struct folio *folio;
-	might_sleep();
-
-	folio = page_folio(page);
-	if (!folio_trylock(folio))
-		__folio_lock(folio);
-}
-
-/**
- * folio_lock_killable() - Lock this folio, interruptible by a fatal signal.
- * @folio: The folio to lock.
- *
- * Attempts to lock the folio, like folio_lock(), except that the sleep
- * to acquire the lock is interruptible by a fatal signal.
- *
- * Context: May sleep; see folio_lock().
- * Return: 0 if the lock was acquired; -EINTR if a fatal signal was received.
- */
-static inline int folio_lock_killable(struct folio *folio)
-{
-	might_sleep();
-	if (!folio_trylock(folio))
-		return __folio_lock_killable(folio);
-	return 0;
-}
-
-/*
- * folio_lock_or_retry - Lock the folio, unless this would block and the
- * caller indicated that it can handle a retry.
- *
- * Return value and mmap_lock implications depend on flags; see
- * __folio_lock_or_retry().
- */
-static inline vm_fault_t folio_lock_or_retry(struct folio *folio,
-					     struct vm_fault *vmf)
-{
-	might_sleep();
-	if (!folio_trylock(folio))
-		return __folio_lock_or_retry(folio, vmf);
-	return 0;
-}
-
-/*
- * This is exported only for folio_wait_locked/folio_wait_writeback, etc.,
- * and should not be used directly.
- */
-void folio_wait_bit(struct folio *folio, int bit_nr);
-int folio_wait_bit_killable(struct folio *folio, int bit_nr);
-
-/* 
- * Wait for a folio to be unlocked.
- *
- * This must be called with the caller "holding" the folio,
- * ie with increased folio reference count so that the folio won't
- * go away during the wait.
- */
-static inline void folio_wait_locked(struct folio *folio)
-{
-	if (folio_test_locked(folio))
-		folio_wait_bit(folio, PG_locked);
-}
-
-static inline int folio_wait_locked_killable(struct folio *folio)
-{
-	if (!folio_test_locked(folio))
-		return 0;
-	return folio_wait_bit_killable(folio, PG_locked);
-}
-
-void folio_end_read(struct folio *folio, bool success);
 void wait_on_page_writeback(struct page *page);
 void folio_wait_writeback(struct folio *folio);
 int folio_wait_writeback_killable(struct folio *folio);
@@ -1268,9 +1101,6 @@ int filemap_migrate_folio(struct address_space *mapping, struct folio *dst,
 #else
 #define filemap_migrate_folio NULL
 #endif
-void folio_end_private_2(struct folio *folio);
-void folio_wait_private_2(struct folio *folio);
-int folio_wait_private_2_killable(struct folio *folio);
 
 /*
  * Fault in userspace address range.
diff --git a/mm/folio_wait.c b/mm/folio_wait.c
index 18b42488ce37..06156e138c09 100644
--- a/mm/folio_wait.c
+++ b/mm/folio_wait.c
@@ -8,7 +8,7 @@
 #include <linux/mm.h>
 #include <linux/swap.h>
 #include <linux/leafops.h>
-#include <linux/pagemap.h>
+#include <linux/folio_wait.h>
 #include <linux/wait.h>
 #include <linux/hash.h>
 #include <linux/sysctl.h>

-- 
2.39.5

[PATCH RFC 04/11] folio_wait: move folio_wait_writeback() family to mm/folio_wait.c

[Tal Zussman]

folio_wait_writeback(), folio_wait_writeback_killable() and
folio_wait_stable() are thin wrappers around folio_wait_bit() and
folio_wait_bit_killable() on PG_writeback. Move them to mm/folio_wait.c,
next to the rest of the folio bit-wait infrastructure.

The legacy wait_on_page_writeback() wrapper stays in folio-compat.c, as
its days are numbered, and it will be deleted once the remaining callers
are converted.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 include/linux/folio_wait.h |  4 +++
 include/linux/pagemap.h    |  3 ---
 mm/folio_wait.c            | 67 ++++++++++++++++++++++++++++++++++++++++++++++
 mm/page-writeback.c        | 66 ---------------------------------------------
 4 files changed, 71 insertions(+), 69 deletions(-)

diff --git a/include/linux/folio_wait.h b/include/linux/folio_wait.h
index 80ddf1ffcae4..4a5cb2fcf046 100644
--- a/include/linux/folio_wait.h
+++ b/include/linux/folio_wait.h
@@ -178,4 +178,8 @@ void folio_end_private_2(struct folio *folio);
 void folio_wait_private_2(struct folio *folio);
 int folio_wait_private_2_killable(struct folio *folio);
 
+void folio_wait_writeback(struct folio *folio);
+int folio_wait_writeback_killable(struct folio *folio);
+void folio_wait_stable(struct folio *folio);
+
 #endif /* _LINUX_FOLIO_WAIT_H */
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 7f65c2b0097b..84ccb682cca8 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1074,13 +1074,10 @@ static inline pgoff_t linear_page_index(const struct vm_area_struct *vma,
 }
 
 void wait_on_page_writeback(struct page *page);
-void folio_wait_writeback(struct folio *folio);
-int folio_wait_writeback_killable(struct folio *folio);
 void end_page_writeback(struct page *page);
 void folio_end_writeback(struct folio *folio);
 void folio_end_writeback_no_dropbehind(struct folio *folio);
 void folio_end_dropbehind(struct folio *folio);
-void folio_wait_stable(struct folio *folio);
 void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
 void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
 void __folio_cancel_dirty(struct folio *folio);
diff --git a/mm/folio_wait.c b/mm/folio_wait.c
index 06156e138c09..9d3328717bb3 100644
--- a/mm/folio_wait.c
+++ b/mm/folio_wait.c
@@ -15,6 +15,7 @@
 #include <linux/delayacct.h>
 #include <linux/psi.h>
 #include <linux/migrate.h>
+#include <trace/events/writeback.h>
 
 #include "internal.h"
 
@@ -572,6 +573,72 @@ int folio_wait_private_2_killable(struct folio *folio)
 }
 EXPORT_SYMBOL(folio_wait_private_2_killable);
 
+/**
+ * folio_wait_writeback - Wait for a folio to finish writeback.
+ * @folio: The folio to wait for.
+ *
+ * If the folio is currently being written back to storage, wait for the
+ * I/O to complete.
+ *
+ * Context: Sleeps.  Must be called in process context and with
+ * no spinlocks held.  Caller should hold a reference on the folio.
+ * If the folio is not locked, writeback may start again after writeback
+ * has finished.
+ */
+void folio_wait_writeback(struct folio *folio)
+{
+	while (folio_test_writeback(folio)) {
+		trace_folio_wait_writeback(folio, folio_mapping(folio));
+		folio_wait_bit(folio, PG_writeback);
+	}
+}
+EXPORT_SYMBOL_GPL(folio_wait_writeback);
+
+/**
+ * folio_wait_writeback_killable - Wait for a folio to finish writeback.
+ * @folio: The folio to wait for.
+ *
+ * If the folio is currently being written back to storage, wait for the
+ * I/O to complete or a fatal signal to arrive.
+ *
+ * Context: Sleeps.  Must be called in process context and with
+ * no spinlocks held.  Caller should hold a reference on the folio.
+ * If the folio is not locked, writeback may start again after writeback
+ * has finished.
+ * Return: 0 on success, -EINTR if we get a fatal signal while waiting.
+ */
+int folio_wait_writeback_killable(struct folio *folio)
+{
+	while (folio_test_writeback(folio)) {
+		trace_folio_wait_writeback(folio, folio_mapping(folio));
+		if (folio_wait_bit_killable(folio, PG_writeback))
+			return -EINTR;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(folio_wait_writeback_killable);
+
+/**
+ * folio_wait_stable() - wait for writeback to finish, if necessary.
+ * @folio: The folio to wait on.
+ *
+ * This function determines if the given folio is related to a backing
+ * device that requires folio contents to be held stable during writeback.
+ * If so, then it will wait for any pending writeback to complete.
+ *
+ * Context: Sleeps.  Must be called in process context and with
+ * no spinlocks held.  Caller should hold a reference on the folio.
+ * If the folio is not locked, writeback may start again after writeback
+ * has finished.
+ */
+void folio_wait_stable(struct folio *folio)
+{
+	if (mapping_stable_writes(folio_mapping(folio)))
+		folio_wait_writeback(folio);
+}
+EXPORT_SYMBOL_GPL(folio_wait_stable);
+
 /**
  * __folio_lock - Get a lock on the folio, assuming we need to sleep to get it.
  * @folio: The folio to lock
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 833f743f309f..50f548bbb375 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -3042,69 +3042,3 @@ void __folio_start_writeback(struct folio *folio, bool keep_write)
 	VM_BUG_ON_FOLIO(access_ret != 0, folio);
 }
 EXPORT_SYMBOL(__folio_start_writeback);
-
-/**
- * folio_wait_writeback - Wait for a folio to finish writeback.
- * @folio: The folio to wait for.
- *
- * If the folio is currently being written back to storage, wait for the
- * I/O to complete.
- *
- * Context: Sleeps.  Must be called in process context and with
- * no spinlocks held.  Caller should hold a reference on the folio.
- * If the folio is not locked, writeback may start again after writeback
- * has finished.
- */
-void folio_wait_writeback(struct folio *folio)
-{
-	while (folio_test_writeback(folio)) {
-		trace_folio_wait_writeback(folio, folio_mapping(folio));
-		folio_wait_bit(folio, PG_writeback);
-	}
-}
-EXPORT_SYMBOL_GPL(folio_wait_writeback);
-
-/**
- * folio_wait_writeback_killable - Wait for a folio to finish writeback.
- * @folio: The folio to wait for.
- *
- * If the folio is currently being written back to storage, wait for the
- * I/O to complete or a fatal signal to arrive.
- *
- * Context: Sleeps.  Must be called in process context and with
- * no spinlocks held.  Caller should hold a reference on the folio.
- * If the folio is not locked, writeback may start again after writeback
- * has finished.
- * Return: 0 on success, -EINTR if we get a fatal signal while waiting.
- */
-int folio_wait_writeback_killable(struct folio *folio)
-{
-	while (folio_test_writeback(folio)) {
-		trace_folio_wait_writeback(folio, folio_mapping(folio));
-		if (folio_wait_bit_killable(folio, PG_writeback))
-			return -EINTR;
-	}
-
-	return 0;
-}
-EXPORT_SYMBOL_GPL(folio_wait_writeback_killable);
-
-/**
- * folio_wait_stable() - wait for writeback to finish, if necessary.
- * @folio: The folio to wait on.
- *
- * This function determines if the given folio is related to a backing
- * device that requires folio contents to be held stable during writeback.
- * If so, then it will wait for any pending writeback to complete.
- *
- * Context: Sleeps.  Must be called in process context and with
- * no spinlocks held.  Caller should hold a reference on the folio.
- * If the folio is not locked, writeback may start again after writeback
- * has finished.
- */
-void folio_wait_stable(struct folio *folio)
-{
-	if (mapping_stable_writes(folio_mapping(folio)))
-		folio_wait_writeback(folio);
-}
-EXPORT_SYMBOL_GPL(folio_wait_stable);

-- 
2.39.5

[PATCH RFC 05/11] folio_wait: reformat comments and fix alignment

[Tal Zussman]

Reflow comments to fill 80 columns and fix indentation issues carried
over from the original locations in pagemap.h and filemap.c.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 include/linux/folio_wait.h |  62 ++++++++-------
 mm/folio_wait.c            | 185 ++++++++++++++++++++-------------------------
 2 files changed, 113 insertions(+), 134 deletions(-)

diff --git a/include/linux/folio_wait.h b/include/linux/folio_wait.h
index 4a5cb2fcf046..57ccf9ffd243 100644
--- a/include/linux/folio_wait.h
+++ b/include/linux/folio_wait.h
@@ -19,10 +19,10 @@ struct wait_page_queue {
 };
 
 static inline bool wake_page_match(struct wait_page_queue *wait_page,
-				  struct wait_page_key *key)
+		struct wait_page_key *key)
 {
 	if (wait_page->folio != key->folio)
-	       return false;
+		return false;
 	key->page_match = 1;
 
 	if (wait_page->bit_nr != key->bit_nr)
@@ -41,10 +41,10 @@ void folio_unlock(struct folio *folio);
  * folio_trylock() - Attempt to lock a folio.
  * @folio: The folio to attempt to lock.
  *
- * Sometimes it is undesirable to wait for a folio to be unlocked (eg
- * when the locks are being taken in the wrong order, or if making
- * progress through a batch of folios is more important than processing
- * them in order).  Usually folio_lock() is the correct function to call.
+ * Sometimes it is undesirable to wait for a folio to be unlocked (e.g. when
+ * the locks are being taken in the wrong order, or if making progress through
+ * a batch of folios is more important than processing them in order). Usually
+ * folio_lock() is the correct function to call.
  *
  * Context: Any context.
  * Return: Whether the lock was successfully acquired.
@@ -66,23 +66,22 @@ static inline bool trylock_page(struct page *page)
  * folio_lock() - Lock this folio.
  * @folio: The folio to lock.
  *
- * The folio lock protects against many things, probably more than it
- * should.  It is primarily held while a folio is being brought uptodate,
- * either from its backing file or from swap.  It is also held while a
- * folio is being truncated from its address_space, so holding the lock
- * is sufficient to keep folio->mapping stable.
+ * The folio lock protects against many things, probably more than it should.
+ * It is primarily held while a folio is being brought uptodate, either from
+ * its backing file or from swap. It is also held while a folio is being
+ * truncated from its address_space, so holding the lock is sufficient to keep
+ * folio->mapping stable.
  *
- * The folio lock is also held while write() is modifying the page to
- * provide POSIX atomicity guarantees (as long as the write does not
- * cross a page boundary).  Other modifications to the data in the folio
- * do not hold the folio lock and can race with writes, eg DMA and stores
- * to mapped pages.
+ * The folio lock is also held while write() is modifying the folio to provide
+ * POSIX atomicity guarantees (as long as the write does not cross a page
+ * boundary). Other modifications to the data in the folio do not hold the
+ * folio lock and can race with writes, e.g. DMA and stores to mapped pages.
  *
- * Context: May sleep.  If you need to acquire the locks of two or
- * more folios, they must be in order of ascending index, if they are
- * in the same address_space.  If they are in different address_spaces,
- * acquire the lock of the folio which belongs to the address_space which
- * has the lowest address in memory first.
+ * Context: May sleep. If you need to acquire the locks of two or more folios,
+ * they must be in order of ascending index, if they are in the same
+ * address_space. If they are in different address_spaces, acquire the lock of
+ * the folio which belongs to the address_space which has the lowest address in
+ * memory first.
  */
 static inline void folio_lock(struct folio *folio)
 {
@@ -99,8 +98,8 @@ static inline void folio_lock(struct folio *folio)
  * This is a legacy function and new code should probably use folio_lock()
  * instead.
  *
- * Context: May sleep.  Pages in the same folio share a lock, so do not
- * attempt to lock two pages which share a folio.
+ * Context: May sleep. Pages in the same folio share a lock, so do not attempt
+ * to lock two pages which share a folio.
  */
 static inline void lock_page(struct page *page)
 {
@@ -116,8 +115,8 @@ static inline void lock_page(struct page *page)
  * folio_lock_killable() - Lock this folio, interruptible by a fatal signal.
  * @folio: The folio to lock.
  *
- * Attempts to lock the folio, like folio_lock(), except that the sleep
- * to acquire the lock is interruptible by a fatal signal.
+ * Attempts to lock the folio, like folio_lock(), except that the sleep to
+ * acquire the lock is interruptible by a fatal signal.
  *
  * Context: May sleep; see folio_lock().
  * Return: 0 if the lock was acquired; -EINTR if a fatal signal was received.
@@ -131,8 +130,8 @@ static inline int folio_lock_killable(struct folio *folio)
 }
 
 /*
- * folio_lock_or_retry - Lock the folio, unless this would block and the
- * caller indicated that it can handle a retry.
+ * folio_lock_or_retry - Lock the folio, unless this would block and the caller
+ * indicated that it can handle a retry.
  *
  * Return value and mmap_lock implications depend on flags; see
  * __folio_lock_or_retry().
@@ -147,8 +146,8 @@ static inline vm_fault_t folio_lock_or_retry(struct folio *folio,
 }
 
 /*
- * This is exported only for folio_wait_locked/folio_wait_writeback, etc.,
- * and should not be used directly.
+ * This is exported only for folio_wait_locked/folio_wait_writeback, etc., and
+ * should not be used directly.
  */
 void folio_wait_bit(struct folio *folio, int bit_nr);
 int folio_wait_bit_killable(struct folio *folio, int bit_nr);
@@ -156,9 +155,8 @@ int folio_wait_bit_killable(struct folio *folio, int bit_nr);
 /*
  * Wait for a folio to be unlocked.
  *
- * This must be called with the caller "holding" the folio,
- * ie with increased folio reference count so that the folio won't
- * go away during the wait.
+ * This must be called with the caller "holding" the folio, i.e. with increased
+ * folio reference count so that the folio won't go away during the wait.
  */
 static inline void folio_wait_locked(struct folio *folio)
 {
diff --git a/mm/folio_wait.c b/mm/folio_wait.c
index 9d3328717bb3..8d8237cdd73b 100644
--- a/mm/folio_wait.c
+++ b/mm/folio_wait.c
@@ -20,14 +20,12 @@
 #include "internal.h"
 
 /*
- * In order to wait for pages to become available there must be
- * waitqueues associated with pages. By using a hash table of
- * waitqueues where the bucket discipline is to maintain all
- * waiters on the same queue and wake all when any of the pages
- * become available, and for the woken contexts to check to be
- * sure the appropriate page became available, this saves space
- * at a cost of "thundering herd" phenomena during rare hash
- * collisions.
+ * In order to wait for pages to become available there must be waitqueues
+ * associated with pages. By using a hash table of waitqueues where the bucket
+ * discipline is to maintain all waiters on the same queue and wake all when any
+ * of the pages become available, and for the woken contexts to check to be
+ * sure the appropriate page became available, this saves space at a cost of
+ * "thundering herd" phenomena during rare hash collisions.
  */
 #define PAGE_WAIT_TABLE_BITS 8
 #define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS)
@@ -70,44 +68,42 @@ void __init folio_wait_init(void)
  *
  *  (a) no special bits set:
  *
- *	We're just waiting for the bit to be released, and when a waker
- *	calls the wakeup function, we set WQ_FLAG_WOKEN and wake it up,
- *	and remove it from the wait queue.
+ *	We're just waiting for the bit to be released, and when a waker calls
+ *	the wakeup function, we set WQ_FLAG_WOKEN and wake it up, and remove
+ *	it from the wait queue.
  *
  *	Simple and straightforward.
  *
  *  (b) WQ_FLAG_EXCLUSIVE:
  *
- *	The waiter is waiting to get the lock, and only one waiter should
- *	be woken up to avoid any thundering herd behavior. We'll set the
+ *	The waiter is waiting to get the lock, and only one waiter should be
+ *	woken up to avoid any thundering herd behavior. We'll set the
  *	WQ_FLAG_WOKEN bit, wake it up, and remove it from the wait queue.
  *
  *	This is the traditional exclusive wait.
  *
  *  (c) WQ_FLAG_EXCLUSIVE | WQ_FLAG_CUSTOM:
  *
- *	The waiter is waiting to get the bit, and additionally wants the
- *	lock to be transferred to it for fair lock behavior. If the lock
- *	cannot be taken, we stop walking the wait queue without waking
- *	the waiter.
+ *	The waiter is waiting to get the bit, and additionally wants the lock
+ *	to be transferred to it for fair lock behavior. If the lock cannot be
+ *	taken, we stop walking the wait queue without waking the waiter.
  *
  *	This is the "fair lock handoff" case, and in addition to setting
- *	WQ_FLAG_WOKEN, we set WQ_FLAG_DONE to let the waiter easily see
- *	that it now has the lock.
+ *	WQ_FLAG_WOKEN, we set WQ_FLAG_DONE to let the waiter easily see that
+ *	it now has the lock.
  */
-static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, void *arg)
+static int wake_page_function(wait_queue_entry_t *wait, unsigned int mode, int sync, void *arg)
 {
 	unsigned int flags;
 	struct wait_page_key *key = arg;
-	struct wait_page_queue *wait_page
-		= container_of(wait, struct wait_page_queue, wait);
+	struct wait_page_queue *wait_page = container_of(wait, struct wait_page_queue, wait);
 
 	if (!wake_page_match(wait_page, key))
 		return 0;
 
 	/*
-	 * If it's a lock handoff wait, we get the bit for it, and
-	 * stop walking (and do not wake it up) if we can't.
+	 * If it's a lock handoff wait, we get the bit for it, and stop walking
+	 * (and do not wake it up) if we can't.
 	 */
 	flags = wait->flags;
 	if (flags & WQ_FLAG_EXCLUSIVE) {
@@ -121,26 +117,24 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync,
 	}
 
 	/*
-	 * We are holding the wait-queue lock, but the waiter that
-	 * is waiting for this will be checking the flags without
-	 * any locking.
+	 * We are holding the wait-queue lock, but the waiter that is waiting
+	 * for this will be checking the flags without any locking.
 	 *
-	 * So update the flags atomically, and wake up the waiter
-	 * afterwards to avoid any races. This store-release pairs
-	 * with the load-acquire in folio_wait_bit_common().
+	 * So update the flags atomically, and wake up the waiter afterwards to
+	 * avoid any races. This store-release pairs with the load-acquire in
+	 * folio_wait_bit_common().
 	 */
 	smp_store_release(&wait->flags, flags | WQ_FLAG_WOKEN);
 	wake_up_state(wait->private, mode);
 
 	/*
-	 * Ok, we have successfully done what we're waiting for,
-	 * and we can unconditionally remove the wait entry.
+	 * Ok, we have successfully done what we're waiting for, and we can
+	 * unconditionally remove the wait entry.
 	 *
-	 * Note that this pairs with the "finish_wait()" in the
-	 * waiter, and has to be the absolute last thing we do.
-	 * After this list_del_init(&wait->entry) the wait entry
-	 * might be de-allocated and the process might even have
-	 * exited.
+	 * Note that this pairs with the "finish_wait()" in the waiter, and has
+	 * to be the absolute last thing we do. After this
+	 * list_del_init(&wait->entry) the wait entry might be de-allocated and
+	 * the process might even have exited.
 	 */
 	list_del_init_careful(&wait->entry);
 	return (flags & WQ_FLAG_EXCLUSIVE) != 0;
@@ -198,11 +192,10 @@ enum behavior {
 };
 
 /*
- * Attempt to check (or get) the folio flag, and mark us done
- * if successful.
+ * Attempt to check (or get) the folio flag, and mark as done if successful.
  */
 static inline bool folio_trylock_flag(struct folio *folio, int bit_nr,
-					struct wait_queue_entry *wait)
+		struct wait_queue_entry *wait)
 {
 	if (wait->flags & WQ_FLAG_EXCLUSIVE) {
 		if (test_and_set_bit(bit_nr, &folio->flags.f))
@@ -246,18 +239,14 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 	}
 
 	/*
-	 * Do one last check whether we can get the
-	 * page bit synchronously.
+	 * Do one last check whether we can get the page bit synchronously.
 	 *
-	 * Do the folio_set_waiters() marking before that
-	 * to let any waker we _just_ missed know they
-	 * need to wake us up (otherwise they'll never
-	 * even go to the slow case that looks at the
-	 * page queue), and add ourselves to the wait
-	 * queue if we need to sleep.
+	 * Do the folio_set_waiters() marking before that to let any waker we
+	 * _just_ missed know they need to wake us up (otherwise they'll never
+	 * even go to the slow case that looks at the wait queue), and add
+	 * ourselves to the wait queue if we need to sleep.
 	 *
-	 * This part needs to be done under the queue
-	 * lock to avoid races.
+	 * This part needs to be done under the queue lock to avoid races.
 	 */
 	spin_lock_irq(&q->lock);
 	folio_set_waiters(folio);
@@ -266,9 +255,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 	spin_unlock_irq(&q->lock);
 
 	/*
-	 * From now on, all the logic will be based on
-	 * the WQ_FLAG_WOKEN and WQ_FLAG_DONE flag, to
-	 * see whether the page bit testing has already
+	 * From now on, all the logic will be based on the WQ_FLAG_WOKEN and
+	 * WQ_FLAG_DONE flag, to see whether the page bit testing has already
 	 * been done by the wake function.
 	 *
 	 * We can drop our reference to the folio.
@@ -277,10 +265,9 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 		folio_put(folio);
 
 	/*
-	 * Note that until the "finish_wait()", or until
-	 * we see the WQ_FLAG_WOKEN flag, we need to
-	 * be very careful with the 'wait->flags', because
-	 * we may race with a waker that sets them.
+	 * Note that until the "finish_wait()", or until we see the
+	 * WQ_FLAG_WOKEN flag, we need to be very careful with the
+	 * 'wait->flags', because we may race with a waker that sets them.
 	 */
 	for (;;) {
 		unsigned int flags;
@@ -306,8 +293,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 			break;
 
 		/*
-		 * Otherwise, if we're getting the lock, we need to
-		 * try to get it ourselves.
+		 * Otherwise, if we're getting the lock, we need to try to get
+		 * it ourselves.
 		 *
 		 * And if that fails, we'll have to retry this all.
 		 */
@@ -333,13 +320,13 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 
 	/*
 	 * NOTE! The wait->flags weren't stable until we've done the
-	 * 'finish_wait()', and we could have exited the loop above due
-	 * to a signal, and had a wakeup event happen after the signal
-	 * test but before the 'finish_wait()'.
+	 * 'finish_wait()', and we could have exited the loop above due to a
+	 * signal, and had a wakeup event happen after the signal test but
+	 * before the 'finish_wait()'.
 	 *
-	 * So only after the finish_wait() can we reliably determine
-	 * if we got woken up or not, so we can now figure out the final
-	 * return value based on that state without races.
+	 * So only after the finish_wait() can we reliably determine if we got
+	 * woken up or not, so we can now figure out the final return value
+	 * based on that state without races.
 	 *
 	 * Also note that WQ_FLAG_WOKEN is sufficient for a non-exclusive
 	 * waiter, but an exclusive one requires WQ_FLAG_DONE.
@@ -452,11 +439,10 @@ EXPORT_SYMBOL(folio_wait_bit_killable);
  * @folio: The folio to wait for.
  * @state: The sleep state (TASK_KILLABLE, TASK_UNINTERRUPTIBLE, etc).
  *
- * The caller should hold a reference on @folio.  They expect the page to
- * become unlocked relatively soon, but do not wish to hold up migration
- * (for example) by holding the reference while waiting for the folio to
- * come unlocked.  After this function returns, the caller should not
- * dereference @folio.
+ * The caller should hold a reference on @folio. They expect the page to become
+ * unlocked relatively soon, but do not wish to hold up migration (for example)
+ * by holding the reference while waiting for the folio to come unlocked. After
+ * this function returns, the caller should not dereference @folio.
  *
  * Return: 0 if the folio was unlocked or -EINTR if interrupted by a signal.
  */
@@ -471,8 +457,8 @@ int folio_put_wait_locked(struct folio *folio, int state)
  *
  * Unlocks the folio and wakes up any thread sleeping on the page lock.
  *
- * Context: May be called from interrupt or process context.  May not be
- * called from NMI context.
+ * Context: May be called from interrupt or process context. May not be called
+ * from NMI context.
  */
 void folio_unlock(struct folio *folio)
 {
@@ -490,14 +476,13 @@ EXPORT_SYMBOL(folio_unlock);
  * @folio: The folio.
  * @success: True if all reads completed successfully.
  *
- * When all reads against a folio have completed, filesystems should
- * call this function to let the pagecache know that no more reads
- * are outstanding.  This will unlock the folio and wake up any thread
- * sleeping on the lock.  The folio will also be marked uptodate if all
- * reads succeeded.
+ * When all reads against a folio have completed, filesystems should call this
+ * function to let the pagecache know that no more reads are outstanding. This
+ * will unlock the folio and wake up any thread sleeping on the lock. The folio
+ * will also be marked uptodate if all reads succeeded.
  *
- * Context: May be called from interrupt or process context.  May not be
- * called from NMI context.
+ * Context: May be called from interrupt or process context. May not be called
+ * from NMI context.
  */
 void folio_end_read(struct folio *folio, bool success)
 {
@@ -577,13 +562,12 @@ EXPORT_SYMBOL(folio_wait_private_2_killable);
  * folio_wait_writeback - Wait for a folio to finish writeback.
  * @folio: The folio to wait for.
  *
- * If the folio is currently being written back to storage, wait for the
- * I/O to complete.
+ * If the folio is currently being written back to storage, wait for the I/O to
+ * complete.
  *
- * Context: Sleeps.  Must be called in process context and with
- * no spinlocks held.  Caller should hold a reference on the folio.
- * If the folio is not locked, writeback may start again after writeback
- * has finished.
+ * Context: Sleeps. Must be called in process context and with no spinlocks
+ * held. Caller should hold a reference on the folio. If the folio is not
+ * locked, writeback may start again after writeback has finished.
  */
 void folio_wait_writeback(struct folio *folio)
 {
@@ -598,13 +582,12 @@ EXPORT_SYMBOL_GPL(folio_wait_writeback);
  * folio_wait_writeback_killable - Wait for a folio to finish writeback.
  * @folio: The folio to wait for.
  *
- * If the folio is currently being written back to storage, wait for the
- * I/O to complete or a fatal signal to arrive.
+ * If the folio is currently being written back to storage, wait for the I/O to
+ * complete or a fatal signal to arrive.
  *
- * Context: Sleeps.  Must be called in process context and with
- * no spinlocks held.  Caller should hold a reference on the folio.
- * If the folio is not locked, writeback may start again after writeback
- * has finished.
+ * Context: Sleeps. Must be called in process context and with no spinlocks
+ * held. Caller should hold a reference on the folio. If the folio is not
+ * locked, writeback may start again after writeback has finished.
  * Return: 0 on success, -EINTR if we get a fatal signal while waiting.
  */
 int folio_wait_writeback_killable(struct folio *folio)
@@ -623,14 +606,13 @@ EXPORT_SYMBOL_GPL(folio_wait_writeback_killable);
  * folio_wait_stable() - wait for writeback to finish, if necessary.
  * @folio: The folio to wait on.
  *
- * This function determines if the given folio is related to a backing
- * device that requires folio contents to be held stable during writeback.
- * If so, then it will wait for any pending writeback to complete.
+ * This function determines if the given folio is related to a backing device
+ * that requires folio contents to be held stable during writeback. If so, then
+ * it will wait for any pending writeback to complete.
  *
- * Context: Sleeps.  Must be called in process context and with
- * no spinlocks held.  Caller should hold a reference on the folio.
- * If the folio is not locked, writeback may start again after writeback
- * has finished.
+ * Context: Sleeps. Must be called in process context and with no spinlocks
+ * held. Caller should hold a reference on the folio. If the folio is not
+ * locked, writeback may start again after writeback has finished.
  */
 void folio_wait_stable(struct folio *folio)
 {
@@ -670,10 +652,9 @@ int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait)
 	folio_set_waiters(folio);
 	ret = !folio_trylock(folio);
 	/*
-	 * If we were successful now, we know we're still on the
-	 * waitqueue as we're still under the lock. This means it's
-	 * safe to remove and return success, we know the callback
-	 * isn't going to trigger.
+	 * If we were successful now, we know we're still on the waitqueue as
+	 * we're still under the lock. This means it's safe to remove and
+	 * return success, we know the callback isn't going to trigger.
 	 */
 	if (!ret)
 		__remove_wait_queue(q, &wait->wait);

-- 
2.39.5

[PATCH RFC 06/11] folio_wait: rename wait_page_* infrastructure to wait_folio_*

[Tal Zussman]

The folio bit-lock wait infrastructure still refers to "page" in the
names of its core types and helpers, even though it operates on folios.
Rename accordingly:

  struct wait_page_key           -> struct wait_folio_key
  struct wait_page_queue         -> struct wait_folio_queue
  wait_page_key.page_match       -> wait_folio_key.folio_match
  wake_page_match()              -> wake_folio_match()
  wake_page_function()           -> wake_folio_function()
  PAGE_WAIT_TABLE_{BITS,SIZE}    -> FOLIO_WAIT_TABLE_{BITS,SIZE}

Also rename local variables and field names, such as io_uring's wpq ->
wfq. Update relevant comments as well.

While at it, update io_uring/rw.h to include folio_wait.h rather than
pagemap.h.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 include/linux/folio_wait.h | 16 +++++-----
 include/linux/fs.h         |  2 +-
 io_uring/rw.c              | 14 ++++-----
 io_uring/rw.h              |  6 ++--
 mm/folio_wait.c            | 74 +++++++++++++++++++++++-----------------------
 mm/internal.h              |  2 +-
 6 files changed, 57 insertions(+), 57 deletions(-)

diff --git a/include/linux/folio_wait.h b/include/linux/folio_wait.h
index 57ccf9ffd243..1732df23d952 100644
--- a/include/linux/folio_wait.h
+++ b/include/linux/folio_wait.h
@@ -6,26 +6,26 @@
 #include <linux/page-flags.h>
 #include <linux/wait.h>
 
-struct wait_page_key {
+struct wait_folio_key {
 	struct folio *folio;
 	int bit_nr;
-	int page_match;
+	int folio_match;
 };
 
-struct wait_page_queue {
+struct wait_folio_queue {
 	struct folio *folio;
 	int bit_nr;
 	wait_queue_entry_t wait;
 };
 
-static inline bool wake_page_match(struct wait_page_queue *wait_page,
-		struct wait_page_key *key)
+static inline bool wake_folio_match(struct wait_folio_queue *wait_folio,
+		struct wait_folio_key *key)
 {
-	if (wait_page->folio != key->folio)
+	if (wait_folio->folio != key->folio)
 		return false;
-	key->page_match = 1;
+	key->folio_match = 1;
 
-	if (wait_page->bit_nr != key->bit_nr)
+	if (wait_folio->bit_nr != key->bit_nr)
 		return false;
 
 	return true;
diff --git a/include/linux/fs.h b/include/linux/fs.h
index bb9cc4f7207c..cd5088dfe9a1 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -390,7 +390,7 @@ struct kiocb {
 	 * waitqueue associated with completing the read.
 	 * Valid IFF IOCB_WAITQ is set.
 	 */
-	struct wait_page_queue	*ki_waitq;
+	struct wait_folio_queue	*ki_waitq;
 };
 
 static inline bool is_sync_kiocb(struct kiocb *kiocb)
diff --git a/io_uring/rw.c b/io_uring/rw.c
index 0c4834645279..fc87baac1911 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -763,14 +763,14 @@ static ssize_t loop_rw_iter(int ddir, struct io_rw *rw, struct iov_iter *iter)
 static int io_async_buf_func(struct wait_queue_entry *wait, unsigned mode,
 			     int sync, void *arg)
 {
-	struct wait_page_queue *wpq;
+	struct wait_folio_queue *wfq;
 	struct io_kiocb *req = wait->private;
 	struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw);
-	struct wait_page_key *key = arg;
+	struct wait_folio_key *key = arg;
 
-	wpq = container_of(wait, struct wait_page_queue, wait);
+	wfq = container_of(wait, struct wait_folio_queue, wait);
 
-	if (!wake_page_match(wpq, key))
+	if (!wake_folio_match(wfq, key))
 		return 0;
 
 	rw->kiocb.ki_flags &= ~IOCB_WAITQ;
@@ -783,7 +783,7 @@ static int io_async_buf_func(struct wait_queue_entry *wait, unsigned mode,
  * This controls whether a given IO request should be armed for async page
  * based retry. If we return false here, the request is handed to the async
  * worker threads for retry. If we're doing buffered reads on a regular file,
- * we prepare a private wait_page_queue entry and retry the operation. This
+ * we prepare a private wait_folio_queue entry and retry the operation. This
  * will either succeed because the page is now uptodate and unlocked, or it
  * will register a callback when the page is unlocked at IO completion. Through
  * that callback, io_uring uses task_work to setup a retry of the operation.
@@ -794,7 +794,7 @@ static int io_async_buf_func(struct wait_queue_entry *wait, unsigned mode,
 static bool io_rw_should_retry(struct io_kiocb *req)
 {
 	struct io_async_rw *io = req->async_data;
-	struct wait_page_queue *wait = &io->wpq;
+	struct wait_folio_queue *wait = &io->wfq;
 	struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw);
 	struct kiocb *kiocb = &rw->kiocb;
 
@@ -897,7 +897,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode, int rw_type)
 			return -EINVAL;
 
 		/*
-		 * We have a union of meta fields with wpq used for buffered-io
+		 * We have a union of meta fields with wfq used for buffered-io
 		 * in io_async_rw, so fail it here.
 		 */
 		if (!(file->f_flags & O_DIRECT))
diff --git a/io_uring/rw.h b/io_uring/rw.h
index 9bd7fbf70ea9..22e9f77c51d6 100644
--- a/io_uring/rw.h
+++ b/io_uring/rw.h
@@ -1,7 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 
+#include <linux/folio_wait.h>
 #include <linux/io_uring_types.h>
-#include <linux/pagemap.h>
 
 struct io_meta_state {
 	u32			seed;
@@ -19,11 +19,11 @@ struct io_async_rw {
 		unsigned			buf_group;
 
 		/*
-		 * wpq is for buffered io, while meta fields are used with
+		 * wfq is for buffered io, while meta fields are used with
 		 * direct io
 		 */
 		union {
-			struct wait_page_queue		wpq;
+			struct wait_folio_queue		wfq;
 			struct {
 				struct uio_meta			meta;
 				struct io_meta_state		meta_state;
diff --git a/mm/folio_wait.c b/mm/folio_wait.c
index 8d8237cdd73b..70f808729f9c 100644
--- a/mm/folio_wait.c
+++ b/mm/folio_wait.c
@@ -20,20 +20,20 @@
 #include "internal.h"
 
 /*
- * In order to wait for pages to become available there must be waitqueues
- * associated with pages. By using a hash table of waitqueues where the bucket
+ * In order to wait for folios to become available there must be waitqueues
+ * associated with folios. By using a hash table of waitqueues where the bucket
  * discipline is to maintain all waiters on the same queue and wake all when any
- * of the pages become available, and for the woken contexts to check to be
- * sure the appropriate page became available, this saves space at a cost of
+ * of the folios become available, and for the woken contexts to check to be
+ * sure the appropriate folio became available, this saves space at a cost of
  * "thundering herd" phenomena during rare hash collisions.
  */
-#define PAGE_WAIT_TABLE_BITS 8
-#define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS)
-static wait_queue_head_t folio_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned;
+#define FOLIO_WAIT_TABLE_BITS 8
+#define FOLIO_WAIT_TABLE_SIZE (1 << FOLIO_WAIT_TABLE_BITS)
+static wait_queue_head_t folio_wait_table[FOLIO_WAIT_TABLE_SIZE] __cacheline_aligned;
 
 static wait_queue_head_t *folio_waitqueue(struct folio *folio)
 {
-	return &folio_wait_table[hash_ptr(folio, PAGE_WAIT_TABLE_BITS)];
+	return &folio_wait_table[hash_ptr(folio, FOLIO_WAIT_TABLE_BITS)];
 }
 
 /* How many times do we accept lock stealing from under a waiter? */
@@ -53,14 +53,14 @@ void __init folio_wait_init(void)
 {
 	int i;
 
-	for (i = 0; i < PAGE_WAIT_TABLE_SIZE; i++)
+	for (i = 0; i < FOLIO_WAIT_TABLE_SIZE; i++)
 		init_waitqueue_head(&folio_wait_table[i]);
 
 	register_sysctl_init("vm", folio_wait_sysctl_table);
 }
 
 /*
- * The page wait code treats the "wait->flags" somewhat unusually, because
+ * The folio wait code treats the "wait->flags" somewhat unusually, because
  * we have multiple different kinds of waits, not just the usual "exclusive"
  * one.
  *
@@ -92,13 +92,13 @@ void __init folio_wait_init(void)
  *	WQ_FLAG_WOKEN, we set WQ_FLAG_DONE to let the waiter easily see that
  *	it now has the lock.
  */
-static int wake_page_function(wait_queue_entry_t *wait, unsigned int mode, int sync, void *arg)
+static int wake_folio_function(wait_queue_entry_t *wait, unsigned int mode, int sync, void *arg)
 {
 	unsigned int flags;
-	struct wait_page_key *key = arg;
-	struct wait_page_queue *wait_page = container_of(wait, struct wait_page_queue, wait);
+	struct wait_folio_key *key = arg;
+	struct wait_folio_queue *wait_folio = container_of(wait, struct wait_folio_queue, wait);
 
-	if (!wake_page_match(wait_page, key))
+	if (!wake_folio_match(wait_folio, key))
 		return 0;
 
 	/*
@@ -143,26 +143,26 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned int mode, int s
 static void folio_wake_bit(struct folio *folio, int bit_nr)
 {
 	wait_queue_head_t *q = folio_waitqueue(folio);
-	struct wait_page_key key;
+	struct wait_folio_key key;
 	unsigned long flags;
 
 	key.folio = folio;
 	key.bit_nr = bit_nr;
-	key.page_match = 0;
+	key.folio_match = 0;
 
 	spin_lock_irqsave(&q->lock, flags);
 	__wake_up_locked_key(q, TASK_NORMAL, &key);
 
 	/*
-	 * It's possible to miss clearing waiters here, when we woke our page
-	 * waiters, but the hashed waitqueue has waiters for other pages on it.
+	 * It's possible to miss clearing waiters here, when we woke our folio
+	 * waiters, but the hashed waitqueue has waiters for other folios on it.
 	 * That's okay, it's a rare case. The next waker will clear it.
 	 *
 	 * Note that, depending on the page pool (buddy, hugetlb, ZONE_DEVICE,
 	 * other), the flag may be cleared in the course of freeing the page;
 	 * but that is not required for correctness.
 	 */
-	if (!waitqueue_active(q) || !key.page_match)
+	if (!waitqueue_active(q) || !key.folio_match)
 		folio_clear_waiters(folio);
 
 	spin_unlock_irqrestore(&q->lock, flags);
@@ -180,13 +180,13 @@ void folio_wake_writeback(struct folio *folio)
  * A choice of three behaviors for folio_wait_bit_common():
  */
 enum behavior {
-	EXCLUSIVE,	/* Hold ref to page and take the bit when woken, like
+	EXCLUSIVE,	/* Hold ref to folio and take the bit when woken, like
 			 * __folio_lock() waiting on then setting PG_locked.
 			 */
-	SHARED,		/* Hold ref to page and check the bit when woken, like
+	SHARED,		/* Hold ref to folio and check the bit when woken, like
 			 * folio_wait_writeback() waiting on PG_writeback.
 			 */
-	DROP,		/* Drop ref to page before wait, no check when woken,
+	DROP,		/* Drop ref to folio before wait, no check when woken,
 			 * like folio_put_wait_locked() on PG_locked.
 			 */
 };
@@ -212,8 +212,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 {
 	wait_queue_head_t *q = folio_waitqueue(folio);
 	int unfairness = sysctl_page_lock_unfairness;
-	struct wait_page_queue wait_page;
-	wait_queue_entry_t *wait = &wait_page.wait;
+	struct wait_folio_queue wait_folio;
+	wait_queue_entry_t *wait = &wait_folio.wait;
 	bool thrashing = false;
 	unsigned long pflags;
 	bool in_thrashing;
@@ -226,9 +226,9 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 	}
 
 	init_wait(wait);
-	wait->func = wake_page_function;
-	wait_page.folio = folio;
-	wait_page.bit_nr = bit_nr;
+	wait->func = wake_folio_function;
+	wait_folio.folio = folio;
+	wait_folio.bit_nr = bit_nr;
 
 repeat:
 	wait->flags = 0;
@@ -239,7 +239,7 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 	}
 
 	/*
-	 * Do one last check whether we can get the page bit synchronously.
+	 * Do one last check whether we can get the folio bit synchronously.
 	 *
 	 * Do the folio_set_waiters() marking before that to let any waker we
 	 * _just_ missed know they need to wake us up (otherwise they'll never
@@ -256,7 +256,7 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 
 	/*
 	 * From now on, all the logic will be based on the WQ_FLAG_WOKEN and
-	 * WQ_FLAG_DONE flag, to see whether the page bit testing has already
+	 * WQ_FLAG_DONE flag, to see whether the folio bit testing has already
 	 * been done by the wake function.
 	 *
 	 * We can drop our reference to the folio.
@@ -359,8 +359,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr,
 void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *ptl)
 	__releases(ptl)
 {
-	struct wait_page_queue wait_page;
-	wait_queue_entry_t *wait = &wait_page.wait;
+	struct wait_folio_queue wait_folio;
+	wait_queue_entry_t *wait = &wait_folio.wait;
 	bool thrashing = false;
 	unsigned long pflags;
 	bool in_thrashing;
@@ -375,9 +375,9 @@ void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *ptl)
 	}
 
 	init_wait(wait);
-	wait->func = wake_page_function;
-	wait_page.folio = folio;
-	wait_page.bit_nr = PG_locked;
+	wait->func = wake_folio_function;
+	wait_folio.folio = folio;
+	wait_folio.bit_nr = PG_locked;
 	wait->flags = 0;
 
 	spin_lock_irq(&q->lock);
@@ -439,7 +439,7 @@ EXPORT_SYMBOL(folio_wait_bit_killable);
  * @folio: The folio to wait for.
  * @state: The sleep state (TASK_KILLABLE, TASK_UNINTERRUPTIBLE, etc).
  *
- * The caller should hold a reference on @folio. They expect the page to become
+ * The caller should hold a reference on @folio. They expect the folio to become
  * unlocked relatively soon, but do not wish to hold up migration (for example)
  * by holding the reference while waiting for the folio to come unlocked. After
  * this function returns, the caller should not dereference @folio.
@@ -455,7 +455,7 @@ int folio_put_wait_locked(struct folio *folio, int state)
  * folio_unlock - Unlock a locked folio.
  * @folio: The folio.
  *
- * Unlocks the folio and wakes up any thread sleeping on the page lock.
+ * Unlocks the folio and wakes up any thread sleeping on the folio lock.
  *
  * Context: May be called from interrupt or process context. May not be called
  * from NMI context.
@@ -639,7 +639,7 @@ int __folio_lock_killable(struct folio *folio)
 }
 EXPORT_SYMBOL_GPL(__folio_lock_killable);
 
-int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait)
+int __folio_lock_async(struct folio *folio, struct wait_folio_queue *wait)
 {
 	struct wait_queue_head *q = folio_waitqueue(folio);
 	int ret;
diff --git a/mm/internal.h b/mm/internal.h
index a121ca07f75c..21b0f4ec2478 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -105,7 +105,7 @@ void page_writeback_init(void);
 void folio_wait_init(void);
 void folio_wake_writeback(struct folio *folio);
 int folio_put_wait_locked(struct folio *folio, int state);
-int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait);
+int __folio_lock_async(struct folio *folio, struct wait_folio_queue *wait);
 
 /*
  * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages,

-- 
2.39.5

[PATCH RFC 07/11] folio_wait: convert VM_BUG_ON_FOLIO() to VM_WARN_ON_ONCE_FOLIO()

[Tal Zussman]

BUG_ON() is deprecated [1]. The VM_BUG_ON_FOLIO() assertions in
folio_unlock(), folio_end_read(), and folio_end_private_2() verify folio
state invariants and are already debug checks. There is no additional
benefit gained by crashing the system.

Convert them to VM_WARN_ON_ONCE_FOLIO(), as is now preferred for such
checks.

[1] https://www.kernel.org/doc/html/latest/process/coding-style.html#use-warn-rather-than-bug

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 mm/folio_wait.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mm/folio_wait.c b/mm/folio_wait.c
index 70f808729f9c..52d336bc7fe0 100644
--- a/mm/folio_wait.c
+++ b/mm/folio_wait.c
@@ -465,7 +465,7 @@ void folio_unlock(struct folio *folio)
 	/* Bit 7 allows x86 to check the byte's sign bit */
 	BUILD_BUG_ON(PG_waiters != 7);
 	BUILD_BUG_ON(PG_locked > 7);
-	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
 	if (folio_xor_flags_has_waiters(folio, 1 << PG_locked))
 		folio_wake_bit(folio, PG_locked);
 }
@@ -490,8 +490,8 @@ void folio_end_read(struct folio *folio, bool success)
 
 	/* Must be in bottom byte for x86 to work */
 	BUILD_BUG_ON(PG_uptodate > 7);
-	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
-	VM_BUG_ON_FOLIO(success && folio_test_uptodate(folio), folio);
+	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+	VM_WARN_ON_ONCE_FOLIO(success && folio_test_uptodate(folio), folio);
 
 	if (likely(success))
 		mask |= 1 << PG_uptodate;
@@ -513,7 +513,7 @@ EXPORT_SYMBOL(folio_end_read);
  */
 void folio_end_private_2(struct folio *folio)
 {
-	VM_BUG_ON_FOLIO(!folio_test_private_2(folio), folio);
+	VM_WARN_ON_ONCE_FOLIO(!folio_test_private_2(folio), folio);
 	clear_bit_unlock(PG_private_2, folio_flags(folio, 0));
 	folio_wake_bit(folio, PG_private_2);
 	folio_put(folio);

-- 
2.39.5

[PATCH RFC 08/11] MAINTAINERS: add folio_wait files to MEMORY MANAGEMENT - CORE

[Tal Zussman]

Add mm/folio_wait.c and include/linux/folio_wait.h after they were split
out from mm/filemap.c, mm/page-writeback.c, and include/linux/pagemap.h.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 MAINTAINERS | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 8cf9ba51d981..bfe1488d9030 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -16781,6 +16781,7 @@ S:	Maintained
 W:	http://www.linux-mm.org
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
 F:	include/linux/folio_batch.h
+F:	include/linux/folio_wait.h
 F:	include/linux/gfp.h
 F:	include/linux/gfp_types.h
 F:	include/linux/highmem.h
@@ -16802,6 +16803,7 @@ F:	kernel/fork.c
 F:	mm/Kconfig
 F:	mm/debug.c
 F:	mm/folio-compat.c
+F:	mm/folio_wait.c
 F:	mm/highmem.c
 F:	mm/init-mm.c
 F:	mm/internal.h

-- 
2.39.5

[PATCH RFC 09/11] fs: move dir_pages() from <linux/pagemap.h> to <linux/fs.h>

[Tal Zussman]

This is an inode-based helper and should live with other inode helpers.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 include/linux/fs.h      | 6 ++++++
 include/linux/pagemap.h | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/include/linux/fs.h b/include/linux/fs.h
index cd5088dfe9a1..776cc82932a7 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1171,6 +1171,12 @@ static inline void i_size_write(struct inode *inode, loff_t i_size)
 #endif
 }
 
+static inline unsigned long dir_pages(const struct inode *inode)
+{
+	return (unsigned long)(inode->i_size + PAGE_SIZE - 1) >>
+			       PAGE_SHIFT;
+}
+
 static inline unsigned iminor(const struct inode *inode)
 {
 	return MINOR(inode->i_rdev);
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 84ccb682cca8..f86a550ad516 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -1356,12 +1356,6 @@ static inline size_t readahead_batch_length(const struct readahead_control *rac)
 	return rac->_batch_count * PAGE_SIZE;
 }
 
-static inline unsigned long dir_pages(const struct inode *inode)
-{
-	return (unsigned long)(inode->i_size + PAGE_SIZE - 1) >>
-			       PAGE_SHIFT;
-}
-
 /**
  * folio_mkwrite_check_truncate - check if folio was truncated
  * @folio: the folio to check

-- 
2.39.5

[PATCH RFC 10/11] fs: move generic_file_read_iter() to fs/read_write.c

[Tal Zussman]

generic_file_read_iter() and its kiocb_write_and_wait() helper are
VFS-level read functions: Their callers are filesystems, and their job
is to glue direct I/O or the page cache (filemap_read) to a struct kiocb
and iov_iter caller.

Move both to fs/read_write.c, alongside vfs_iter_read. Drop the extern
from generic_file_read_iter()'s declaration and reflow the
generic_file_read_iter() definition to fit on one line too.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 fs/read_write.c         | 82 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/fs.h      |  3 +-
 include/linux/pagemap.h |  1 -
 mm/filemap.c            | 82 -------------------------------------------------
 4 files changed, 84 insertions(+), 84 deletions(-)

diff --git a/fs/read_write.c b/fs/read_write.c
index 50bff7edc91f..59ceea85c163 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -989,6 +989,88 @@ ssize_t vfs_iter_write(struct file *file, struct iov_iter *iter, loff_t *ppos,
 }
 EXPORT_SYMBOL(vfs_iter_write);
 
+int kiocb_write_and_wait(struct kiocb *iocb, size_t count)
+{
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
+	loff_t pos = iocb->ki_pos;
+	loff_t end = pos + count - 1;
+
+	if (iocb->ki_flags & IOCB_NOWAIT) {
+		if (filemap_range_needs_writeback(mapping, pos, end))
+			return -EAGAIN;
+		return 0;
+	}
+
+	return filemap_write_and_wait_range(mapping, pos, end);
+}
+EXPORT_SYMBOL_GPL(kiocb_write_and_wait);
+
+/**
+ * generic_file_read_iter - generic filesystem read routine
+ * @iocb:	kernel I/O control block
+ * @iter:	destination for the data read
+ *
+ * This is the "read_iter()" routine for all filesystems
+ * that can use the page cache directly.
+ *
+ * The IOCB_NOWAIT flag in iocb->ki_flags indicates that -EAGAIN shall
+ * be returned when no data can be read without waiting for I/O requests
+ * to complete; it doesn't prevent readahead.
+ *
+ * The IOCB_NOIO flag in iocb->ki_flags indicates that no new I/O
+ * requests shall be made for the read or for readahead.  When no data
+ * can be read, -EAGAIN shall be returned.  When readahead would be
+ * triggered, a partial, possibly empty read shall be returned.
+ *
+ * Return:
+ * * number of bytes copied, even for partial reads
+ * * negative error code (or 0 if IOCB_NOIO) if nothing was read
+ */
+ssize_t generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
+{
+	size_t count = iov_iter_count(iter);
+	ssize_t retval = 0;
+
+	if (!count)
+		return 0; /* skip atime */
+
+	if (iocb->ki_flags & IOCB_DIRECT) {
+		struct file *file = iocb->ki_filp;
+		struct address_space *mapping = file->f_mapping;
+		struct inode *inode = mapping->host;
+
+		retval = kiocb_write_and_wait(iocb, count);
+		if (retval < 0)
+			return retval;
+		file_accessed(file);
+
+		retval = mapping->a_ops->direct_IO(iocb, iter);
+		if (retval >= 0) {
+			iocb->ki_pos += retval;
+			count -= retval;
+		}
+		if (retval != -EIOCBQUEUED)
+			iov_iter_revert(iter, count - iov_iter_count(iter));
+
+		/*
+		 * Btrfs can have a short DIO read if we encounter
+		 * compressed extents, so if there was an error, or if
+		 * we've already read everything we wanted to, or if
+		 * there was a short read because we hit EOF, go ahead
+		 * and return.  Otherwise fallthrough to buffered io for
+		 * the rest of the read.  Buffered reads will not work for
+		 * DAX files, so don't bother trying.
+		 */
+		if (retval < 0 || !count || IS_DAX(inode))
+			return retval;
+		if (iocb->ki_pos >= i_size_read(inode))
+			return retval;
+	}
+
+	return filemap_read(iocb, iter, retval);
+}
+EXPORT_SYMBOL(generic_file_read_iter);
+
 static ssize_t vfs_readv(struct file *file, const struct iovec __user *vec,
 			 unsigned long vlen, loff_t *pos, rwf_t flags)
 {
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 776cc82932a7..c0151ced8e7a 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -3055,7 +3055,8 @@ extern int generic_write_check_limits(struct file *file, loff_t pos,
 extern int generic_file_rw_checks(struct file *file_in, struct file *file_out);
 ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *to,
 		ssize_t already_read);
-extern ssize_t generic_file_read_iter(struct kiocb *, struct iov_iter *);
+ssize_t generic_file_read_iter(struct kiocb *, struct iov_iter *);
+int kiocb_write_and_wait(struct kiocb *iocb, size_t count);
 extern ssize_t __generic_file_write_iter(struct kiocb *, struct iov_iter *);
 extern ssize_t generic_file_write_iter(struct kiocb *, struct iov_iter *);
 extern ssize_t generic_file_direct_write(struct kiocb *, struct iov_iter *);
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index f86a550ad516..46cefd552a51 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -59,7 +59,6 @@ int filemap_fdatawrite_range(struct address_space *mapping,
 		loff_t start, loff_t end);
 int filemap_check_errors(struct address_space *mapping);
 void __filemap_set_wb_err(struct address_space *mapping, int err);
-int kiocb_write_and_wait(struct kiocb *iocb, size_t count);
 
 static inline int filemap_write_and_wait(struct address_space *mapping)
 {
diff --git a/mm/filemap.c b/mm/filemap.c
index 079f9c3ac8a2..db7c53cd681b 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2251,22 +2251,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 }
 EXPORT_SYMBOL_GPL(filemap_read);
 
-int kiocb_write_and_wait(struct kiocb *iocb, size_t count)
-{
-	struct address_space *mapping = iocb->ki_filp->f_mapping;
-	loff_t pos = iocb->ki_pos;
-	loff_t end = pos + count - 1;
-
-	if (iocb->ki_flags & IOCB_NOWAIT) {
-		if (filemap_range_needs_writeback(mapping, pos, end))
-			return -EAGAIN;
-		return 0;
-	}
-
-	return filemap_write_and_wait_range(mapping, pos, end);
-}
-EXPORT_SYMBOL_GPL(kiocb_write_and_wait);
-
 int filemap_invalidate_pages(struct address_space *mapping,
 			     loff_t pos, loff_t end, bool nowait)
 {
@@ -2302,72 +2286,6 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t count)
 }
 EXPORT_SYMBOL_GPL(kiocb_invalidate_pages);
 
-/**
- * generic_file_read_iter - generic filesystem read routine
- * @iocb:	kernel I/O control block
- * @iter:	destination for the data read
- *
- * This is the "read_iter()" routine for all filesystems
- * that can use the page cache directly.
- *
- * The IOCB_NOWAIT flag in iocb->ki_flags indicates that -EAGAIN shall
- * be returned when no data can be read without waiting for I/O requests
- * to complete; it doesn't prevent readahead.
- *
- * The IOCB_NOIO flag in iocb->ki_flags indicates that no new I/O
- * requests shall be made for the read or for readahead.  When no data
- * can be read, -EAGAIN shall be returned.  When readahead would be
- * triggered, a partial, possibly empty read shall be returned.
- *
- * Return:
- * * number of bytes copied, even for partial reads
- * * negative error code (or 0 if IOCB_NOIO) if nothing was read
- */
-ssize_t
-generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
-{
-	size_t count = iov_iter_count(iter);
-	ssize_t retval = 0;
-
-	if (!count)
-		return 0; /* skip atime */
-
-	if (iocb->ki_flags & IOCB_DIRECT) {
-		struct file *file = iocb->ki_filp;
-		struct address_space *mapping = file->f_mapping;
-		struct inode *inode = mapping->host;
-
-		retval = kiocb_write_and_wait(iocb, count);
-		if (retval < 0)
-			return retval;
-		file_accessed(file);
-
-		retval = mapping->a_ops->direct_IO(iocb, iter);
-		if (retval >= 0) {
-			iocb->ki_pos += retval;
-			count -= retval;
-		}
-		if (retval != -EIOCBQUEUED)
-			iov_iter_revert(iter, count - iov_iter_count(iter));
-
-		/*
-		 * Btrfs can have a short DIO read if we encounter
-		 * compressed extents, so if there was an error, or if
-		 * we've already read everything we wanted to, or if
-		 * there was a short read because we hit EOF, go ahead
-		 * and return.  Otherwise fallthrough to buffered io for
-		 * the rest of the read.  Buffered reads will not work for
-		 * DAX files, so don't bother trying.
-		 */
-		if (retval < 0 || !count || IS_DAX(inode))
-			return retval;
-		if (iocb->ki_pos >= i_size_read(inode))
-			return retval;
-	}
-
-	return filemap_read(iocb, iter, retval);
-}
-EXPORT_SYMBOL(generic_file_read_iter);
 
 /*
  * Splice subpages from a folio into a pipe.

-- 
2.39.5

[PATCH RFC 11/11] fs: move generic_file_write_iter() family to fs/read_write.c

[Tal Zussman]

Move the VFS-level generic write path out of mm/filemap.c into
fs/read_write.c next to the just-relocated read path:

  - generic_file_write_iter()
  - __generic_file_write_iter()
  - generic_file_direct_write()
  - generic_perform_write()
  - kiocb_invalidate_pages()
  - kiocb_invalidate_post_direct_write()
  - dio_warn_stale_pagecache()

The kiocb_invalidate_* prototypes move from <linux/pagemap.h> to
<linux/fs.h>, joining kiocb_write_and_wait() and the other generic
read/write declarations. Drop extern from the prototypes of all
five generic_file_* declarations in <linux/fs.h>. Reflow the
generic_file_direct_write() definition to fit on one line.

Signed-off-by: Tal Zussman <tz2294@columbia.edu>
---
 fs/read_write.c         | 276 +++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/fs.h      |   8 +-
 include/linux/pagemap.h |   2 -
 mm/filemap.c            | 277 ------------------------------------------------
 4 files changed, 281 insertions(+), 282 deletions(-)

diff --git a/fs/read_write.c b/fs/read_write.c
index 59ceea85c163..cea5f79fdacf 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -1071,6 +1071,282 @@ ssize_t generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 }
 EXPORT_SYMBOL(generic_file_read_iter);
 
+int kiocb_invalidate_pages(struct kiocb *iocb, size_t count)
+{
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
+
+	return filemap_invalidate_pages(mapping, iocb->ki_pos,
+					iocb->ki_pos + count - 1,
+					iocb->ki_flags & IOCB_NOWAIT);
+}
+EXPORT_SYMBOL_GPL(kiocb_invalidate_pages);
+
+/*
+ * Warn about a page cache invalidation failure during a direct I/O write.
+ */
+static void dio_warn_stale_pagecache(struct file *filp)
+{
+	static DEFINE_RATELIMIT_STATE(_rs, 86400 * HZ, DEFAULT_RATELIMIT_BURST);
+	char pathname[128];
+	char *path;
+
+	errseq_set(&filp->f_mapping->wb_err, -EIO);
+	if (__ratelimit(&_rs)) {
+		path = file_path(filp, pathname, sizeof(pathname));
+		if (IS_ERR(path))
+			path = "(unknown)";
+		pr_crit("Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!\n");
+		pr_crit("File: %s PID: %d Comm: %.20s\n", path, current->pid,
+			current->comm);
+	}
+}
+
+void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count)
+{
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
+
+	if (mapping->nrpages &&
+	    invalidate_inode_pages2_range(mapping,
+			iocb->ki_pos >> PAGE_SHIFT,
+			(iocb->ki_pos + count - 1) >> PAGE_SHIFT))
+		dio_warn_stale_pagecache(iocb->ki_filp);
+}
+
+ssize_t generic_file_direct_write(struct kiocb *iocb, struct iov_iter *from)
+{
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
+	size_t write_len = iov_iter_count(from);
+	ssize_t written;
+
+	/*
+	 * If a page can not be invalidated, return 0 to fall back
+	 * to buffered write.
+	 */
+	written = kiocb_invalidate_pages(iocb, write_len);
+	if (written) {
+		if (written == -EBUSY)
+			return 0;
+		return written;
+	}
+
+	written = mapping->a_ops->direct_IO(iocb, from);
+
+	/*
+	 * Finally, try again to invalidate clean pages which might have been
+	 * cached by non-direct readahead, or faulted in by get_user_pages()
+	 * if the source of the write was an mmap'ed region of the file
+	 * we're writing.  Either one is a pretty crazy thing to do,
+	 * so we don't support it 100%.  If this invalidation
+	 * fails, tough, the write still worked...
+	 *
+	 * Most of the time we do not need this since dio_complete() will do
+	 * the invalidation for us. However there are some file systems that
+	 * do not end up with dio_complete() being called, so let's not break
+	 * them by removing it completely.
+	 *
+	 * Noticeable example is a blkdev_direct_IO().
+	 *
+	 * Skip invalidation for async writes or if mapping has no pages.
+	 */
+	if (written > 0) {
+		struct inode *inode = mapping->host;
+		loff_t pos = iocb->ki_pos;
+
+		kiocb_invalidate_post_direct_write(iocb, written);
+		pos += written;
+		write_len -= written;
+		if (pos > i_size_read(inode) && !S_ISBLK(inode->i_mode)) {
+			i_size_write(inode, pos);
+			mark_inode_dirty(inode);
+		}
+		iocb->ki_pos = pos;
+	}
+	if (written != -EIOCBQUEUED)
+		iov_iter_revert(from, write_len - iov_iter_count(from));
+	return written;
+}
+EXPORT_SYMBOL(generic_file_direct_write);
+
+ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
+{
+	struct file *file = iocb->ki_filp;
+	loff_t pos = iocb->ki_pos;
+	struct address_space *mapping = file->f_mapping;
+	const struct address_space_operations *a_ops = mapping->a_ops;
+	size_t chunk = mapping_max_folio_size(mapping);
+	long status = 0;
+	ssize_t written = 0;
+
+	do {
+		struct folio *folio;
+		size_t offset;		/* Offset into folio */
+		size_t bytes;		/* Bytes to write to folio */
+		size_t copied;		/* Bytes copied from user */
+		void *fsdata = NULL;
+
+		bytes = iov_iter_count(i);
+retry:
+		offset = pos & (chunk - 1);
+		bytes = min(chunk - offset, bytes);
+		balance_dirty_pages_ratelimited(mapping);
+
+		if (fatal_signal_pending(current)) {
+			status = -EINTR;
+			break;
+		}
+
+		status = a_ops->write_begin(iocb, mapping, pos, bytes,
+						&folio, &fsdata);
+		if (unlikely(status < 0))
+			break;
+
+		offset = offset_in_folio(folio, pos);
+		if (bytes > folio_size(folio) - offset)
+			bytes = folio_size(folio) - offset;
+
+		if (mapping_writably_mapped(mapping))
+			flush_dcache_folio(folio);
+
+		/*
+		 * Faults here on mmap()s can recurse into arbitrary
+		 * filesystem code. Lots of locks are held that can
+		 * deadlock. Use an atomic copy to avoid deadlocking
+		 * in page fault handling.
+		 */
+		copied = copy_folio_from_iter_atomic(folio, offset, bytes, i);
+		flush_dcache_folio(folio);
+
+		status = a_ops->write_end(iocb, mapping, pos, bytes, copied,
+						folio, fsdata);
+		if (unlikely(status != copied)) {
+			iov_iter_revert(i, copied - max(status, 0L));
+			if (unlikely(status < 0))
+				break;
+		}
+		cond_resched();
+
+		if (unlikely(status == 0)) {
+			/*
+			 * A short copy made ->write_end() reject the
+			 * thing entirely.  Might be memory poisoning
+			 * halfway through, might be a race with munmap,
+			 * might be severe memory pressure.
+			 */
+			if (chunk > PAGE_SIZE)
+				chunk /= 2;
+			if (copied) {
+				bytes = copied;
+				goto retry;
+			}
+
+			/*
+			 * 'folio' is now unlocked and faults on it can be
+			 * handled. Ensure forward progress by trying to
+			 * fault it in now.
+			 */
+			if (fault_in_iov_iter_readable(i, bytes) == bytes) {
+				status = -EFAULT;
+				break;
+			}
+		} else {
+			pos += status;
+			written += status;
+		}
+	} while (iov_iter_count(i));
+
+	if (!written)
+		return status;
+	iocb->ki_pos += written;
+	return written;
+}
+EXPORT_SYMBOL(generic_perform_write);
+
+/**
+ * __generic_file_write_iter - write data to a file
+ * @iocb:	IO state structure (file, offset, etc.)
+ * @from:	iov_iter with data to write
+ *
+ * This function does all the work needed for actually writing data to a
+ * file. It does all basic checks, removes SUID from the file, updates
+ * modification times and calls proper subroutines depending on whether we
+ * do direct IO or a standard buffered write.
+ *
+ * It expects i_rwsem to be grabbed unless we work on a block device or similar
+ * object which does not need locking at all.
+ *
+ * This function does *not* take care of syncing data in case of O_SYNC write.
+ * A caller has to handle it. This is mainly due to the fact that we want to
+ * avoid syncing under i_rwsem.
+ *
+ * Return:
+ * * number of bytes written, even for truncated writes
+ * * negative error code if no data has been written at all
+ */
+ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
+{
+	struct file *file = iocb->ki_filp;
+	struct address_space *mapping = file->f_mapping;
+	struct inode *inode = mapping->host;
+	ssize_t ret;
+
+	ret = file_remove_privs(file);
+	if (ret)
+		return ret;
+
+	ret = file_update_time(file);
+	if (ret)
+		return ret;
+
+	if (iocb->ki_flags & IOCB_DIRECT) {
+		ret = generic_file_direct_write(iocb, from);
+		/*
+		 * If the write stopped short of completing, fall back to
+		 * buffered writes.  Some filesystems do this for writes to
+		 * holes, for example.  For DAX files, a buffered write will
+		 * not succeed (even if it did, DAX does not handle dirty
+		 * page-cache pages correctly).
+		 */
+		if (ret < 0 || !iov_iter_count(from) || IS_DAX(inode))
+			return ret;
+		return direct_write_fallback(iocb, from, ret,
+				generic_perform_write(iocb, from));
+	}
+
+	return generic_perform_write(iocb, from);
+}
+EXPORT_SYMBOL(__generic_file_write_iter);
+
+/**
+ * generic_file_write_iter - write data to a file
+ * @iocb:	IO state structure
+ * @from:	iov_iter with data to write
+ *
+ * This is a wrapper around __generic_file_write_iter() to be used by most
+ * filesystems. It takes care of syncing the file in case of O_SYNC file
+ * and acquires i_rwsem as needed.
+ * Return:
+ * * negative error code if no data has been written at all of
+ *   vfs_fsync_range() failed for a synchronous write
+ * * number of bytes written, even for truncated writes
+ */
+ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
+{
+	struct file *file = iocb->ki_filp;
+	struct inode *inode = file->f_mapping->host;
+	ssize_t ret;
+
+	inode_lock(inode);
+	ret = generic_write_checks(iocb, from);
+	if (ret > 0)
+		ret = __generic_file_write_iter(iocb, from);
+	inode_unlock(inode);
+
+	if (ret > 0)
+		ret = generic_write_sync(iocb, ret);
+	return ret;
+}
+EXPORT_SYMBOL(generic_file_write_iter);
+
 static ssize_t vfs_readv(struct file *file, const struct iovec __user *vec,
 			 unsigned long vlen, loff_t *pos, rwf_t flags)
 {
diff --git a/include/linux/fs.h b/include/linux/fs.h
index c0151ced8e7a..6cfb9e46bc37 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -3057,9 +3057,11 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *to,
 		ssize_t already_read);
 ssize_t generic_file_read_iter(struct kiocb *, struct iov_iter *);
 int kiocb_write_and_wait(struct kiocb *iocb, size_t count);
-extern ssize_t __generic_file_write_iter(struct kiocb *, struct iov_iter *);
-extern ssize_t generic_file_write_iter(struct kiocb *, struct iov_iter *);
-extern ssize_t generic_file_direct_write(struct kiocb *, struct iov_iter *);
+int kiocb_invalidate_pages(struct kiocb *iocb, size_t count);
+void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count);
+ssize_t __generic_file_write_iter(struct kiocb *, struct iov_iter *);
+ssize_t generic_file_write_iter(struct kiocb *, struct iov_iter *);
+ssize_t generic_file_direct_write(struct kiocb *, struct iov_iter *);
 ssize_t generic_perform_write(struct kiocb *, struct iov_iter *);
 ssize_t direct_write_fallback(struct kiocb *iocb, struct iov_iter *iter,
 		ssize_t direct_written, ssize_t buffered_written);
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 46cefd552a51..b7c2dc8076ab 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -31,8 +31,6 @@ static inline void invalidate_remote_inode(struct inode *inode)
 int invalidate_inode_pages2(struct address_space *mapping);
 int invalidate_inode_pages2_range(struct address_space *mapping,
 		pgoff_t start, pgoff_t end);
-int kiocb_invalidate_pages(struct kiocb *iocb, size_t count);
-void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count);
 int filemap_invalidate_pages(struct address_space *mapping,
 			     loff_t pos, loff_t end, bool nowait);
 
diff --git a/mm/filemap.c b/mm/filemap.c
index db7c53cd681b..284c0296a011 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2276,17 +2276,6 @@ int filemap_invalidate_pages(struct address_space *mapping,
 					     end >> PAGE_SHIFT);
 }
 
-int kiocb_invalidate_pages(struct kiocb *iocb, size_t count)
-{
-	struct address_space *mapping = iocb->ki_filp->f_mapping;
-
-	return filemap_invalidate_pages(mapping, iocb->ki_pos,
-					iocb->ki_pos + count - 1,
-					iocb->ki_flags & IOCB_NOWAIT);
-}
-EXPORT_SYMBOL_GPL(kiocb_invalidate_pages);
-
-
 /*
  * Splice subpages from a folio into a pipe.
  */
@@ -3500,272 +3489,6 @@ struct page *read_cache_page_gfp(struct address_space *mapping,
 }
 EXPORT_SYMBOL(read_cache_page_gfp);
 
-/*
- * Warn about a page cache invalidation failure during a direct I/O write.
- */
-static void dio_warn_stale_pagecache(struct file *filp)
-{
-	static DEFINE_RATELIMIT_STATE(_rs, 86400 * HZ, DEFAULT_RATELIMIT_BURST);
-	char pathname[128];
-	char *path;
-
-	errseq_set(&filp->f_mapping->wb_err, -EIO);
-	if (__ratelimit(&_rs)) {
-		path = file_path(filp, pathname, sizeof(pathname));
-		if (IS_ERR(path))
-			path = "(unknown)";
-		pr_crit("Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!\n");
-		pr_crit("File: %s PID: %d Comm: %.20s\n", path, current->pid,
-			current->comm);
-	}
-}
-
-void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count)
-{
-	struct address_space *mapping = iocb->ki_filp->f_mapping;
-
-	if (mapping->nrpages &&
-	    invalidate_inode_pages2_range(mapping,
-			iocb->ki_pos >> PAGE_SHIFT,
-			(iocb->ki_pos + count - 1) >> PAGE_SHIFT))
-		dio_warn_stale_pagecache(iocb->ki_filp);
-}
-
-ssize_t
-generic_file_direct_write(struct kiocb *iocb, struct iov_iter *from)
-{
-	struct address_space *mapping = iocb->ki_filp->f_mapping;
-	size_t write_len = iov_iter_count(from);
-	ssize_t written;
-
-	/*
-	 * If a page can not be invalidated, return 0 to fall back
-	 * to buffered write.
-	 */
-	written = kiocb_invalidate_pages(iocb, write_len);
-	if (written) {
-		if (written == -EBUSY)
-			return 0;
-		return written;
-	}
-
-	written = mapping->a_ops->direct_IO(iocb, from);
-
-	/*
-	 * Finally, try again to invalidate clean pages which might have been
-	 * cached by non-direct readahead, or faulted in by get_user_pages()
-	 * if the source of the write was an mmap'ed region of the file
-	 * we're writing.  Either one is a pretty crazy thing to do,
-	 * so we don't support it 100%.  If this invalidation
-	 * fails, tough, the write still worked...
-	 *
-	 * Most of the time we do not need this since dio_complete() will do
-	 * the invalidation for us. However there are some file systems that
-	 * do not end up with dio_complete() being called, so let's not break
-	 * them by removing it completely.
-	 *
-	 * Noticeable example is a blkdev_direct_IO().
-	 *
-	 * Skip invalidation for async writes or if mapping has no pages.
-	 */
-	if (written > 0) {
-		struct inode *inode = mapping->host;
-		loff_t pos = iocb->ki_pos;
-
-		kiocb_invalidate_post_direct_write(iocb, written);
-		pos += written;
-		write_len -= written;
-		if (pos > i_size_read(inode) && !S_ISBLK(inode->i_mode)) {
-			i_size_write(inode, pos);
-			mark_inode_dirty(inode);
-		}
-		iocb->ki_pos = pos;
-	}
-	if (written != -EIOCBQUEUED)
-		iov_iter_revert(from, write_len - iov_iter_count(from));
-	return written;
-}
-EXPORT_SYMBOL(generic_file_direct_write);
-
-ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)
-{
-	struct file *file = iocb->ki_filp;
-	loff_t pos = iocb->ki_pos;
-	struct address_space *mapping = file->f_mapping;
-	const struct address_space_operations *a_ops = mapping->a_ops;
-	size_t chunk = mapping_max_folio_size(mapping);
-	long status = 0;
-	ssize_t written = 0;
-
-	do {
-		struct folio *folio;
-		size_t offset;		/* Offset into folio */
-		size_t bytes;		/* Bytes to write to folio */
-		size_t copied;		/* Bytes copied from user */
-		void *fsdata = NULL;
-
-		bytes = iov_iter_count(i);
-retry:
-		offset = pos & (chunk - 1);
-		bytes = min(chunk - offset, bytes);
-		balance_dirty_pages_ratelimited(mapping);
-
-		if (fatal_signal_pending(current)) {
-			status = -EINTR;
-			break;
-		}
-
-		status = a_ops->write_begin(iocb, mapping, pos, bytes,
-						&folio, &fsdata);
-		if (unlikely(status < 0))
-			break;
-
-		offset = offset_in_folio(folio, pos);
-		if (bytes > folio_size(folio) - offset)
-			bytes = folio_size(folio) - offset;
-
-		if (mapping_writably_mapped(mapping))
-			flush_dcache_folio(folio);
-
-		/*
-		 * Faults here on mmap()s can recurse into arbitrary
-		 * filesystem code. Lots of locks are held that can
-		 * deadlock. Use an atomic copy to avoid deadlocking
-		 * in page fault handling.
-		 */
-		copied = copy_folio_from_iter_atomic(folio, offset, bytes, i);
-		flush_dcache_folio(folio);
-
-		status = a_ops->write_end(iocb, mapping, pos, bytes, copied,
-						folio, fsdata);
-		if (unlikely(status != copied)) {
-			iov_iter_revert(i, copied - max(status, 0L));
-			if (unlikely(status < 0))
-				break;
-		}
-		cond_resched();
-
-		if (unlikely(status == 0)) {
-			/*
-			 * A short copy made ->write_end() reject the
-			 * thing entirely.  Might be memory poisoning
-			 * halfway through, might be a race with munmap,
-			 * might be severe memory pressure.
-			 */
-			if (chunk > PAGE_SIZE)
-				chunk /= 2;
-			if (copied) {
-				bytes = copied;
-				goto retry;
-			}
-
-			/*
-			 * 'folio' is now unlocked and faults on it can be
-			 * handled. Ensure forward progress by trying to
-			 * fault it in now.
-			 */
-			if (fault_in_iov_iter_readable(i, bytes) == bytes) {
-				status = -EFAULT;
-				break;
-			}
-		} else {
-			pos += status;
-			written += status;
-		}
-	} while (iov_iter_count(i));
-
-	if (!written)
-		return status;
-	iocb->ki_pos += written;
-	return written;
-}
-EXPORT_SYMBOL(generic_perform_write);
-
-/**
- * __generic_file_write_iter - write data to a file
- * @iocb:	IO state structure (file, offset, etc.)
- * @from:	iov_iter with data to write
- *
- * This function does all the work needed for actually writing data to a
- * file. It does all basic checks, removes SUID from the file, updates
- * modification times and calls proper subroutines depending on whether we
- * do direct IO or a standard buffered write.
- *
- * It expects i_rwsem to be grabbed unless we work on a block device or similar
- * object which does not need locking at all.
- *
- * This function does *not* take care of syncing data in case of O_SYNC write.
- * A caller has to handle it. This is mainly due to the fact that we want to
- * avoid syncing under i_rwsem.
- *
- * Return:
- * * number of bytes written, even for truncated writes
- * * negative error code if no data has been written at all
- */
-ssize_t __generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
-{
-	struct file *file = iocb->ki_filp;
-	struct address_space *mapping = file->f_mapping;
-	struct inode *inode = mapping->host;
-	ssize_t ret;
-
-	ret = file_remove_privs(file);
-	if (ret)
-		return ret;
-
-	ret = file_update_time(file);
-	if (ret)
-		return ret;
-
-	if (iocb->ki_flags & IOCB_DIRECT) {
-		ret = generic_file_direct_write(iocb, from);
-		/*
-		 * If the write stopped short of completing, fall back to
-		 * buffered writes.  Some filesystems do this for writes to
-		 * holes, for example.  For DAX files, a buffered write will
-		 * not succeed (even if it did, DAX does not handle dirty
-		 * page-cache pages correctly).
-		 */
-		if (ret < 0 || !iov_iter_count(from) || IS_DAX(inode))
-			return ret;
-		return direct_write_fallback(iocb, from, ret,
-				generic_perform_write(iocb, from));
-	}
-
-	return generic_perform_write(iocb, from);
-}
-EXPORT_SYMBOL(__generic_file_write_iter);
-
-/**
- * generic_file_write_iter - write data to a file
- * @iocb:	IO state structure
- * @from:	iov_iter with data to write
- *
- * This is a wrapper around __generic_file_write_iter() to be used by most
- * filesystems. It takes care of syncing the file in case of O_SYNC file
- * and acquires i_rwsem as needed.
- * Return:
- * * negative error code if no data has been written at all of
- *   vfs_fsync_range() failed for a synchronous write
- * * number of bytes written, even for truncated writes
- */
-ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
-{
-	struct file *file = iocb->ki_filp;
-	struct inode *inode = file->f_mapping->host;
-	ssize_t ret;
-
-	inode_lock(inode);
-	ret = generic_write_checks(iocb, from);
-	if (ret > 0)
-		ret = __generic_file_write_iter(iocb, from);
-	inode_unlock(inode);
-
-	if (ret > 0)
-		ret = generic_write_sync(iocb, ret);
-	return ret;
-}
-EXPORT_SYMBOL(generic_file_write_iter);
 
 /**
  * filemap_release_folio() - Release fs-specific metadata on a folio.

-- 
2.39.5

Re: [PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[Jan Kara]

On Wed 20-05-26 16:48:51, Tal Zussman wrote:
> mm/filemap.c has accumulated additional infrastructure over the years
> that is not directly related to the page cache. It is currently nearly
> 5000 lines long. This series splits out the folio bit-lock and wait
> queue code into separate files, and moves the VFS-level
> generic_file_{read,write}_iter() family of files to fs/read_write.c, in
> order to provide better separation of concerns. This also slims down
> mm/filemap.c by ~1000 lines.
> 
> The folio wait infrastructure is centralized in mm/folio_wait.c and
> include/linux/folio_wait.h, with functions moved from mm/filemap.c,
> mm/page-writeback.c, and include/linux/pagemap.h. Afterwards, the code
> is cleaned up a little, with functions and data types renamed to refer
> to folios rather than pages.
> 
> generic_file_{read,write}_iter() implement the VFS-level read/write path
> for filesystems, including support for direct I/O. These functions and
> their helpers are moved to fs/read_write.c, along with other VFS-level
> read/write functions. dir_pages() is also moved to include/linux/fs.h.
> i_blocks_per_folio() is not moved from include/linux/pagemap.h, as it
> requires folio_size(), which is not currently available in
> include/linux/fs.h.
> 
> No functional change is intended.
> 
> Note: I have additional cleanups to mm/filemap.c ready to go, foremost
> among them centralizing on the filemap_*() naming convention and making
> the exposed page cache API clearer and more consistent, but I've split
> these patches off from that in order to avoid sending these logically
> separate patches to ~60 maintainers.

Overall this makes sense to me. In particular I agree it makes sense to
move the file read/write helpers into fs. Regarding the page waiting bits
it makes some sense to me as well although there it's more of "I don't
really care" opinion so let's see what Matthew and others think...

								Honza

> 
> ---
> Tal Zussman (11):
>       mm: add folio_wake_writeback() helper
>       folio_wait: move folio bit-lock and wait implementation to mm/folio_wait.c
>       folio_wait: move folio bit-lock and wait declarations to include/linux/folio_wait.h
>       folio_wait: move folio_wait_writeback() family to mm/folio_wait.c
>       folio_wait: reformat comments and fix alignment
>       folio_wait: rename wait_page_* infrastructure to wait_folio_*
>       folio_wait: convert VM_BUG_ON_FOLIO() to VM_WARN_ON_ONCE_FOLIO()
>       MAINTAINERS: add folio_wait files to MEMORY MANAGEMENT - CORE
>       fs: move dir_pages() from <linux/pagemap.h> to <linux/fs.h>
>       fs: move generic_file_read_iter() to fs/read_write.c
>       fs: move generic_file_write_iter() family to fs/read_write.c
> 
>  MAINTAINERS                |   2 +
>  fs/read_write.c            | 358 ++++++++++++++++
>  include/linux/folio_wait.h | 183 +++++++++
>  include/linux/fs.h         |  19 +-
>  include/linux/pagemap.h    | 184 +--------
>  io_uring/rw.c              |  14 +-
>  io_uring/rw.h              |   6 +-
>  mm/Makefile                |   2 +-
>  mm/filemap.c               | 993 +--------------------------------------------
>  mm/folio_wait.c            | 710 ++++++++++++++++++++++++++++++++
>  mm/internal.h              |   4 +
>  mm/page-writeback.c        |  66 ---
>  12 files changed, 1285 insertions(+), 1256 deletions(-)
> ---
> base-commit: e9add7501ad3297dad9b90ce201266830a68ab47
> change-id: 20260511-filemap-split-871b5c18e98c
> 
> Best regards,
> -- 
> Tal Zussman <tz2294@columbia.edu>
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[Christian Brauner]

On Thu, May 28, 2026 at 11:22:37AM +0200, Jan Kara wrote:
> On Wed 20-05-26 16:48:51, Tal Zussman wrote:
> > mm/filemap.c has accumulated additional infrastructure over the years
> > that is not directly related to the page cache. It is currently nearly
> > 5000 lines long. This series splits out the folio bit-lock and wait
> > queue code into separate files, and moves the VFS-level
> > generic_file_{read,write}_iter() family of files to fs/read_write.c, in
> > order to provide better separation of concerns. This also slims down
> > mm/filemap.c by ~1000 lines.
> > 
> > The folio wait infrastructure is centralized in mm/folio_wait.c and
> > include/linux/folio_wait.h, with functions moved from mm/filemap.c,
> > mm/page-writeback.c, and include/linux/pagemap.h. Afterwards, the code
> > is cleaned up a little, with functions and data types renamed to refer
> > to folios rather than pages.
> > 
> > generic_file_{read,write}_iter() implement the VFS-level read/write path
> > for filesystems, including support for direct I/O. These functions and
> > their helpers are moved to fs/read_write.c, along with other VFS-level
> > read/write functions. dir_pages() is also moved to include/linux/fs.h.
> > i_blocks_per_folio() is not moved from include/linux/pagemap.h, as it
> > requires folio_size(), which is not currently available in
> > include/linux/fs.h.
> > 
> > No functional change is intended.
> > 
> > Note: I have additional cleanups to mm/filemap.c ready to go, foremost
> > among them centralizing on the filemap_*() naming convention and making
> > the exposed page cache API clearer and more consistent, but I've split
> > these patches off from that in order to avoid sending these logically
> > separate patches to ~60 maintainers.
> 
> Overall this makes sense to me. In particular I agree it makes sense to
> move the file read/write helpers into fs. Regarding the page waiting bits

Seconded.

> it makes some sense to me as well although there it's more of "I don't
> really care" opinion so let's see what Matthew and others think...

Agreed.

Re: [PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[Christoph Hellwig]

On Thu, May 28, 2026 at 11:22:37AM +0200, Jan Kara wrote:
> Overall this makes sense to me. In particular I agree it makes sense to
> move the file read/write helpers into fs.

I disagree very strongly.  Mixing default implementations with the
higher level APIs is a really bad idea and leads to people taking
stupid shortcuts and other layering violations.

Splitting up filemap.c makes sense, but I'd rather keep the generic copy
into and out of the pagecache code with the MM infrastructure for it,
as it is not VFS code, and making that clear to anyone touching the code
is important.

Re: [PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[Tal Zussman]

On 5/28/26 8:49 AM, Christoph Hellwig wrote:
> On Thu, May 28, 2026 at 11:22:37AM +0200, Jan Kara wrote:
>> Overall this makes sense to me. In particular I agree it makes sense to
>> move the file read/write helpers into fs.
> 
> I disagree very strongly.  Mixing default implementations with the
> higher level APIs is a really bad idea and leads to people taking
> stupid shortcuts and other layering violations.

fs/read_write.c already contains some of these "generic" function
implementations, including generic_write_checks(), which is called by
generic_file_write_iter() in mm/filemap.c. Right now the two files are
unnecessarily interdependent. I do think fs/read_write.c is the natural home
for these functions.

> Splitting up filemap.c makes sense, but I'd rather keep the generic copy
> into and out of the pagecache code with the MM infrastructure for it,
> as it is not VFS code, and making that clear to anyone touching the code
> is important.

About half the code moved is implementing direct I/O or multiplexing between
page cache I/O and direct I/O. It definitely shouldn't be in the page cache,
and I do think it is VFS code. The one exception I see is
generic_perform_write(), which is analogous to filemap_read() and should stay
in filemap.c (and probably be renamed to something like filemap_write()).

Re: [PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[Christoph Hellwig]

On Fri, May 29, 2026 at 06:54:07PM -0400, Tal Zussman wrote:
> On 5/28/26 8:49 AM, Christoph Hellwig wrote:
> > On Thu, May 28, 2026 at 11:22:37AM +0200, Jan Kara wrote:
> >> Overall this makes sense to me. In particular I agree it makes sense to
> >> move the file read/write helpers into fs.
> > 
> > I disagree very strongly.  Mixing default implementations with the
> > higher level APIs is a really bad idea and leads to people taking
> > stupid shortcuts and other layering violations.
> 
> fs/read_write.c already contains some of these "generic" function
> implementations, including generic_write_checks(), which is called by
> generic_file_write_iter() in mm/filemap.c. Right now the two files are
> unnecessarily interdependent. I do think fs/read_write.c is the natural home
> for these functions.

generic_write_checks is a very different beast.  It is a generic helper
that every implementation must call. The implementations have to call
it with the right locks held, and this it can't be done before calling
into the method.

> > Splitting up filemap.c makes sense, but I'd rather keep the generic copy
> > into and out of the pagecache code with the MM infrastructure for it,
> > as it is not VFS code, and making that clear to anyone touching the code
> > is important.
> 
> About half the code moved is implementing direct I/O or multiplexing between
> page cache I/O and direct I/O.

This will hopefully change quite a bit once we move everyone off the
legacy direct I/O code and the helpers for it.  Another reason not to
move the code around for now as it should change a bit.

> It definitely shouldn't be in the page cache,
> and I do think it is VFS code.

For the higher level stuff I'd agree.  But I'm not sure how much
is left after the above.  If we have good helpers left something like
libfs.c or a new library might be a better place.

> The one exception I see is
> generic_perform_write(), which is analogous to filemap_read() and should stay
> in filemap.c (and probably be renamed to something like filemap_write()).

Agreed on that.

Re: [PATCH RFC 00/11] mm/filemap: split out folio wait and VFS code

[Tal Zussman]

On 5/28/26 5:22 AM, Jan Kara wrote:
> On Wed 20-05-26 16:48:51, Tal Zussman wrote:
>> mm/filemap.c has accumulated additional infrastructure over the years
>> that is not directly related to the page cache. It is currently nearly
>> 5000 lines long. This series splits out the folio bit-lock and wait
>> queue code into separate files, and moves the VFS-level
>> generic_file_{read,write}_iter() family of files to fs/read_write.c, in
>> order to provide better separation of concerns. This also slims down
>> mm/filemap.c by ~1000 lines.
>> 
>> The folio wait infrastructure is centralized in mm/folio_wait.c and
>> include/linux/folio_wait.h, with functions moved from mm/filemap.c,
>> mm/page-writeback.c, and include/linux/pagemap.h. Afterwards, the code
>> is cleaned up a little, with functions and data types renamed to refer
>> to folios rather than pages.
>> 
>> generic_file_{read,write}_iter() implement the VFS-level read/write path
>> for filesystems, including support for direct I/O. These functions and
>> their helpers are moved to fs/read_write.c, along with other VFS-level
>> read/write functions. dir_pages() is also moved to include/linux/fs.h.
>> i_blocks_per_folio() is not moved from include/linux/pagemap.h, as it
>> requires folio_size(), which is not currently available in
>> include/linux/fs.h.
>> 
>> No functional change is intended.
>> 
>> Note: I have additional cleanups to mm/filemap.c ready to go, foremost
>> among them centralizing on the filemap_*() naming convention and making
>> the exposed page cache API clearer and more consistent, but I've split
>> these patches off from that in order to avoid sending these logically
>> separate patches to ~60 maintainers.
> 
> Overall this makes sense to me. In particular I agree it makes sense to
> move the file read/write helpers into fs. Regarding the page waiting bits
> it makes some sense to me as well although there it's more of "I don't
> really care" opinion so let's see what Matthew and others think...

Sounds good, thanks. For the folio wait/lock code, my reasoning was that
it's used well beyond the page cache and independent of it, so there's no
reason to clog up filemap.c with 700 lines of infrastructure for a more
generic interface (and splitting up pagemap.h a little is a nice bonus).
But yes, let's see what Matthew thinks.

Thanks,
Tal