[PATCH 0/3] mm: allow freezing mappings

[PATCH 0/3] mm: allow freezing mappings

[Pratyush Yadav]

From: "Pratyush Yadav (Google)" <pratyush@kernel.org>

Hi,

This series allows freezing the mapping of an inode. The main user is
memfd_luo. It serializes the state of the mapping and needs to make sure
it does not go out of sync. Currently it does so via a combination of
pinning and internal flags. But this functionality will be used by
HugeTLB and guest_memfd preservation in the future, so this series makes
the feature generic.

The idea came from David Hildenbrand during my presentation on HugeTLB
preservation at LSF/MM/BPF this year. He originally suggested putting it
in the VFS layer, but I find the page cache to be the appropriate layer
for this, since it is the i_pages that memfd_luo serializes.

Patch 1 is a cleanup that also came as a suggestion from David. It is
useful for this series since the freezing change exposes a race with
shmem_writeout(), and this patch fixes this, along with other
improvements including preservation performance. It can be taken
independent of this series, but for now I am including it here.

Patch 2 is the main one. It adds AS_FROZEN, and adds helpers to set it,
and adds checks in filemap core.

Patch 3 updates shmem and memfd_luo to use AS_FROZEN.

My local sashiko has complained about some more minor things, but this
is my first foray into the page cache and I am not even sure this is the
right approach to begin with. So I'd much rather find out early if this
is the wrong approach.

The series is based on mm-unstable. It has a minor merge conflict with
liveupdate/next, but I can fix that up after the merge window. I see
this mainly as a MM change, so I think this should go through the MM
tree.

Regards,
Pratyush Yadav

Pratyush Yadav (Google) (3):
  mm/memfd_luo: don't use memfd_pin_folios()
  mm: allow freezing i_pages
  mm: shmem: use mapping_set_frozen() for memfd preservation

 fs/inode.c               |   9 ++
 include/linux/pagemap.h  |  35 +++++
 include/linux/shmem_fs.h |  17 ---
 mm/filemap.c             |   7 +
 mm/huge_memory.c         |  10 ++
 mm/khugepaged.c          |  12 ++
 mm/memfd_luo.c           | 291 +++++++++++++++++++++++----------------
 mm/migrate.c             |  14 +-
 mm/shmem.c               |  34 +++--
 mm/truncate.c            |  16 ++-
 mm/vmscan.c              |   3 +
 11 files changed, 302 insertions(+), 146 deletions(-)


base-commit: 39274b454e9993cea319f4ac342a8082a9ee1730
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 1/3] mm/memfd_luo: don't use memfd_pin_folios()

[Pratyush Yadav]

From: "Pratyush Yadav (Google)" <pratyush@kernel.org>

memfd_luo takes a snapshot of the folios in the file's mapping. To make
sure the snapshot does not go out of sync, it pins the folios in the
file via memfd_pin_folios(). The pinning also migrates the folios out of
MIGRATE_MOVABLE to make sure doesn't land in KHO boot memory or in a CMA
region.

memfd_pin_folios() does not lock the folios. So it can race with
shmem_writeout(). A folio returned by memfd_pin_folios() might already
be in swap cache. This happens to work but is fragile. memfd_luo looks
at folio flags and index on a folio that isn't in the memfd's page
cache.

In addition, memfd_pin_folios() needs the caller to allocate an array of
folios. memfd_luo does not really need that since it will vmalloc
folios_ser and store the information there anyway. The folios array is
unnecessary overhead.

Finally, memfd_pin_folios() fills in holes in the memfd. This makes
preserving sparse memfds very expensive in both compute and memory.
While this patch doesn't yet stop filling in the holes, it sets up the
infrastructure to make it simpler.

Fix all this by walking the page cache directly using shmem_get_folio()
with SGP_CACHE. This returns a locked folio with an elevated refcount.
memfd_luo holds that reference until the memfd is unpreserved. This
blocks compaction, migration, etc.

There is still a need to migrate folios out of MIGRATE_MOVABLE page
blocks. This is because those might be CMA or KHO scratch. memfd_luo
will hold a longterm reference to the folios so it will block migration
in that pageblock. Introduce memfd_luo_migrate_folio() to do this.

Also factor out folio unpreservation logic into memfd_luo_put_folios()
and use it from memfd_luo_unpreserve_folios() and from error paths in
memfd_luo_preserve_folios().

Since shmem_get_folio() always returns uptodate folios, drop the folio
zeroing logic.

This change also improves preservation performance, shown in the table
below.

|------------------+----------+------------+-------|
| memfd            | baseline | this patch | delta |
|------------------+----------+------------+-------|
| 4K pages, 32 GiB | 748 ms   | 600 ms     |  -20% |
| THP, 4 GiB       | 0.59 ms  | 0.59 ms    |    0% |
| THP, 16 GiB      | 2.99 ms  | 2.61 ms    |  -13% |
| THP, 32 GiB      | 4.83 ms  | 5.21 ms    |   +8% |
| sparse, 32 GiB   | 8695 ms  | 7269 ms    |  -16% |
|------------------+----------+------------+-------|

Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>

---

The diff is unfortunately hard to read. It would be easier to review the
patch by applying it and looking at the final code.
---
 mm/memfd_luo.c | 280 +++++++++++++++++++++++++++++--------------------
 1 file changed, 165 insertions(+), 115 deletions(-)

diff --git a/mm/memfd_luo.c b/mm/memfd_luo.c
index 59de210bee5f9..fe182d5057890 100644
--- a/mm/memfd_luo.c
+++ b/mm/memfd_luo.c
@@ -77,26 +77,126 @@
 #include <linux/kexec_handover.h>
 #include <linux/kho/abi/memfd.h>
 #include <linux/liveupdate.h>
+#include <linux/migrate.h>
+#include <linux/sched/mm.h>
+#include <linux/sched/signal.h>
 #include <linux/shmem_fs.h>
+#include <linux/swap.h>
 #include <linux/vmalloc.h>
 #include <linux/memfd.h>
 #include <uapi/linux/memfd.h>
 
 #include "internal.h"
 
+static void memfd_luo_put_folios(const struct memfd_luo_folio_ser *folios_ser,
+				 u64 nr_folios)
+{
+	u64 i;
+
+	for (i = 0; i < nr_folios; i++) {
+		const struct memfd_luo_folio_ser *pfolio = &folios_ser[i];
+		struct folio *folio;
+
+		if (!pfolio->pfn)
+			continue;
+
+		folio = pfn_folio(pfolio->pfn);
+
+		kho_unpreserve_folio(folio);
+		folio_put(folio);
+	}
+}
+
+/* How many times to retry isolating a folio someone else has isolated. */
+#define MEMFD_LUO_MIGRATE_RETRIES	3
+
+/*
+ * Move a folio out of ZONE_MOVABLE or CMA, so it can be held until the next
+ * kernel. Called with a reference on the unlocked folio, which is always
+ * dropped. The caller must look up the folio again afterwards.
+ *
+ * Returns 0 on success, -EBUSY if the folio could not be isolated (e.g.
+ * because someone else has it isolated right now), or -ENOMEM if migration
+ * failed.
+ */
+static int memfd_luo_migrate_folio(struct folio *folio)
+{
+	struct migration_target_control mtc = {
+		.nid = NUMA_NO_NODE,
+		/* Not movable, so the target is neither in ZONE_MOVABLE nor CMA. */
+		.gfp_mask = GFP_USER | __GFP_NOWARN,
+		.reason = MR_LONGTERM_PIN,
+	};
+	LIST_HEAD(source);
+	bool isolated;
+
+	/* Flush per-CPU LRU batches holding the folio so isolation works. */
+	lru_cache_drain_for_folio(folio, 1, NULL);
+	isolated = isolate_folio_to_list(folio, &source);
+	/* Isolation holds its own reference. */
+	folio_put(folio);
+	if (!isolated)
+		return -EBUSY;
+
+	if (migrate_pages(&source, alloc_migration_target, NULL,
+			  (unsigned long)&mtc, MIGRATE_SYNC, MR_LONGTERM_PIN,
+			  NULL)) {
+		putback_movable_pages(&source);
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+static int memfd_luo_preserve_folio(struct folio *folio,
+				    struct memfd_luo_folio_ser *pfolio)
+{
+	int err;
+
+	err = kho_preserve_folio(folio);
+	if (err)
+		return err;
+
+	/*
+	 * A dirty folio is one which has been written to. A clean folio
+	 * is its opposite. Since a clean folio does not carry user
+	 * data, it can be freed by page reclaim under memory pressure.
+	 *
+	 * Saving the dirty flag at prepare() time doesn't work since it
+	 * can change later. Saving it at freeze() also won't work
+	 * because the dirty bit is normally synced at unmap and there
+	 * might still be a mapping of the file at freeze().
+	 *
+	 * To see why this is a problem, say a folio is clean at
+	 * preserve, but gets dirtied later. The pfolio flags will mark
+	 * it as clean. After retrieve, the next kernel might try to
+	 * reclaim this folio under memory pressure, losing user data.
+	 *
+	 * Unconditionally mark it dirty to avoid this problem. This
+	 * comes at the cost of making clean folios un-reclaimable after
+	 * live update.
+	 */
+	folio_mark_dirty(folio);
+
+	pfolio->pfn = folio_pfn(folio);
+	pfolio->flags = MEMFD_LUO_FOLIO_DIRTY | MEMFD_LUO_FOLIO_UPTODATE;
+	pfolio->index = folio->index;
+
+	return 0;
+}
+
 static int memfd_luo_preserve_folios(struct file *file,
 				     struct kho_vmalloc *kho_vmalloc,
 				     struct memfd_luo_folio_ser **out_folios_ser,
 				     u64 *nr_foliosp)
 {
+	struct memfd_luo_folio_ser *folios_ser, *shrunk;
 	struct inode *inode = file_inode(file);
-	struct memfd_luo_folio_ser *folios_ser;
-	unsigned int max_folios;
-	long i, size, nr_pinned;
-	struct folio **folios;
-	int err = -EINVAL;
-	pgoff_t offset;
-	u64 nr_folios;
+	u64 nr_folios = 0, max_folios;
+	int err, busy_retries = 0;
+	unsigned int pin_flags;
+	pgoff_t index = 0, end;
+	loff_t size;
 
 	size = i_size_read(inode);
 	/*
@@ -113,99 +213,77 @@ static int memfd_luo_preserve_folios(struct file *file,
 	 * Guess the number of folios based on inode size. Real number might end
 	 * up being smaller if there are higher order folios.
 	 */
-	max_folios = PAGE_ALIGN(size) / PAGE_SIZE;
-	folios = kvmalloc_objs(*folios, max_folios);
-	if (!folios)
+	end = DIV_ROUND_UP_ULL(size, PAGE_SIZE);
+	max_folios = end;
+	folios_ser = vmalloc_array(max_folios, sizeof(*folios_ser));
+	if (!folios_ser)
 		return -ENOMEM;
 
 	/*
-	 * Pin the folios so they don't move around behind our back. This also
-	 * ensures none of the folios are in CMA -- which ensures they don't
-	 * fall in KHO scratch memory. It also moves swapped out folios back to
-	 * memory.
+	 * The folios are held until the next kernel, so keep newly allocated
+	 * ones out of ZONE_MOVABLE and CMA -- which also ensures they don't
+	 * fall in KHO bootmem. Existing ones are migrated below.
 	 *
-	 * A side effect of doing this is that it allocates a folio for all
-	 * indices in the file. This might waste memory on sparse memfds. If
-	 * that is really a problem in the future, we can have a
-	 * memfd_pin_folios() variant that does not allocate a page on empty
-	 * slots.
+	 * A side effect of this is that it allocates a folio for all indices in
+	 * the file. This might waste memory on sparse memfds. If that is really
+	 * a problem in the future, this loop can be taught to skip empty slots.
 	 */
-	nr_pinned = memfd_pin_folios(file, 0, size - 1, folios, max_folios,
-				     &offset);
-	if (nr_pinned < 0) {
-		err = nr_pinned;
-		pr_err("failed to pin folios: %d\n", err);
-		goto err_free_folios;
-	}
-	nr_folios = nr_pinned;
+	pin_flags = memalloc_pin_save();
+	while (index < end) {
+		struct folio *folio;
 
-	folios_ser = vcalloc(nr_folios, sizeof(*folios_ser));
-	if (!folios_ser) {
-		err = -ENOMEM;
-		goto err_unpin;
-	}
+		cond_resched();
 
-	for (i = 0; i < nr_folios; i++) {
-		struct memfd_luo_folio_ser *pfolio = &folios_ser[i];
-		struct folio *folio = folios[i];
+		if (fatal_signal_pending(current)) {
+			err = -EINTR;
+			goto err_restore;
+		}
 
-		err = kho_preserve_folio(folio);
+		if (WARN_ON_ONCE(nr_folios >= max_folios)) {
+			err = -EINVAL;
+			goto err_restore;
+		}
+
+		err = shmem_get_folio(inode, index, 0, &folio, SGP_CACHE);
 		if (err)
-			goto err_unpreserve;
+			goto err_restore;
 
-		folio_lock(folio);
-
-		/*
-		 * A dirty folio is one which has been written to. A clean folio
-		 * is its opposite. Since a clean folio does not carry user
-		 * data, it can be freed by page reclaim under memory pressure.
-		 *
-		 * Saving the dirty flag at prepare() time doesn't work since it
-		 * can change later. Saving it at freeze() also won't work
-		 * because the dirty bit is normally synced at unmap and there
-		 * might still be a mapping of the file at freeze().
-		 *
-		 * To see why this is a problem, say a folio is clean at
-		 * preserve, but gets dirtied later. The pfolio flags will mark
-		 * it as clean. After retrieve, the next kernel might try to
-		 * reclaim this folio under memory pressure, losing user data.
-		 *
-		 * Unconditionally mark it dirty to avoid this problem. This
-		 * comes at the cost of making clean folios un-reclaimable after
-		 * live update.
-		 */
-		folio_mark_dirty(folio);
-
-		/*
-		 * If the folio is not uptodate, it was fallocated but never
-		 * used. Saving this flag at prepare() doesn't work since it
-		 * might change later when someone uses the folio.
-		 *
-		 * Since we have taken the performance penalty of allocating,
-		 * zeroing, and pinning all the folios in the holes, take a bit
-		 * more and zero all non-uptodate folios too.
-		 *
-		 * NOTE: For someone looking to improve preserve performance,
-		 * this is a good place to look.
-		 */
-		if (!folio_test_uptodate(folio)) {
-			folio_zero_range(folio, 0, folio_size(folio));
-			flush_dcache_folio(folio);
-			folio_mark_uptodate(folio);
+		if (unlikely(!folio_is_longterm_pinnable(folio))) {
+			folio_unlock(folio);
+			err = memfd_luo_migrate_folio(folio);
+			/* Someone else might have it isolated, retry a bit. */
+			if (err == -EBUSY && ++busy_retries < MEMFD_LUO_MIGRATE_RETRIES)
+				continue;
+			if (err)
+				goto err_restore;
+			continue;
 		}
 
+		err = memfd_luo_preserve_folio(folio, &folios_ser[nr_folios]);
+		index = folio_next_index(folio);
 		folio_unlock(folio);
-
-		pfolio->pfn = folio_pfn(folio);
-		pfolio->flags = MEMFD_LUO_FOLIO_DIRTY | MEMFD_LUO_FOLIO_UPTODATE;
-		pfolio->index = folio->index;
+		if (err) {
+			folio_put(folio);
+			goto err_restore;
+		}
+		nr_folios++;
+		busy_retries = 0;
 	}
+	memalloc_pin_restore(pin_flags);
+
+	/*
+	 * Free the unused tail so it doesn't get preserved. If vrealloc fails,
+	 * the folios_ser is still valid.
+	 */
+	shrunk = vrealloc(folios_ser, array_size(nr_folios, sizeof(*folios_ser)),
+			  GFP_KERNEL);
+	if (shrunk)
+		folios_ser = shrunk;
 
 	err = kho_preserve_vmalloc(folios_ser, kho_vmalloc);
 	if (err)
-		goto err_unpreserve;
+		goto err_put;
 
-	kvfree(folios);
 	*nr_foliosp = nr_folios;
 	*out_folios_ser = folios_ser;
 
@@ -216,15 +294,11 @@ static int memfd_luo_preserve_folios(struct file *file,
 	 */
 	return 0;
 
-err_unpreserve:
-	for (i = i - 1; i >= 0; i--)
-		kho_unpreserve_folio(folios[i]);
+err_restore:
+	memalloc_pin_restore(pin_flags);
+err_put:
+	memfd_luo_put_folios(folios_ser, nr_folios);
 	vfree(folios_ser);
-err_unpin:
-	unpin_folios(folios, nr_folios);
-err_free_folios:
-	kvfree(folios);
-
 	return err;
 }
 
@@ -232,26 +306,11 @@ static void memfd_luo_unpreserve_folios(struct kho_vmalloc *kho_vmalloc,
 					struct memfd_luo_folio_ser *folios_ser,
 					u64 nr_folios)
 {
-	long i;
-
 	if (!nr_folios)
 		return;
 
 	kho_unpreserve_vmalloc(kho_vmalloc);
-
-	for (i = 0; i < nr_folios; i++) {
-		const struct memfd_luo_folio_ser *pfolio = &folios_ser[i];
-		struct folio *folio;
-
-		if (!pfolio->pfn)
-			continue;
-
-		folio = pfn_folio(pfolio->pfn);
-
-		kho_unpreserve_folio(folio);
-		unpin_folio(folio);
-	}
-
+	memfd_luo_put_folios(folios_ser, nr_folios);
 	vfree(folios_ser);
 }
 
@@ -288,15 +347,6 @@ static int memfd_luo_preserve(struct liveupdate_file_op_args *args)
 	ser->pos = args->file->f_pos;
 	inode_size = i_size_read(inode);
 
-	/*
-	 * memfd_pin_folios() caps at UINT_MAX folios; refuse larger
-	 * files to avoid silently preserving only a prefix.
-	 */
-	if (DIV_ROUND_UP_ULL(inode_size, PAGE_SIZE) > UINT_MAX) {
-		err = -EFBIG;
-		goto err_free_ser;
-	}
-
 	ser->size = inode_size;
 	ser->seals = seals;
 
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 2/3] mm: allow freezing i_pages

[Pratyush Yadav]

From: "Pratyush Yadav (Google)" <pratyush@kernel.org>

Live update preserves file-backed memory across kexec by serializing the
index -> folio relationship of the file's page cache. Once serialized, the
page cache must not change, otherwise the serialized data goes stale. memfd
preservation currently does this with a shmem-only flag that just blocks
size changes and fallocate. This leaves other paths that change the page
cache, like reclaim, migration, and THP split and collapse, unchecked. It
also can't be reused for hugetlb or guest_memfd preservation.

Add AS_FROZEN, a mapping flag that freezes the page cache. While it is
set, no folio can be added to, removed from, or replaced in the mapping.
Check it in the core paths that change i_pages and refuse the change
there. The flag is set and cleared under the i_pages lock.

Setting the flag can race with truncation or fallocate, causing them to
fail mid-way. The file systems must ensure that these operations don't
race with the flag setter.

The freezer must hold a reference to the inode until it is unfrozen.
This is because eviction will naturally result in changes to i_pages.
Evicting a frozen inode is a bug and will trigger a warning, but it
still frees the pages to avoid leaking memory.

Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
 fs/inode.c              |  9 +++++++++
 include/linux/pagemap.h | 35 +++++++++++++++++++++++++++++++++++
 mm/filemap.c            |  7 +++++++
 mm/huge_memory.c        | 10 ++++++++++
 mm/khugepaged.c         | 12 ++++++++++++
 mm/migrate.c            | 14 ++++++++++++--
 mm/shmem.c              | 19 +++++++++++++++++++
 mm/truncate.c           | 16 +++++++++++++++-
 mm/vmscan.c             |  3 +++
 9 files changed, 122 insertions(+), 3 deletions(-)

diff --git a/fs/inode.c b/fs/inode.c
index a9d37be390a17..635083c42419f 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -7,6 +7,7 @@
 #include <linux/fs.h>
 #include <linux/filelock.h>
 #include <linux/mm.h>
+#include <linux/pagemap.h>
 #include <linux/backing-dev.h>
 #include <linux/hash.h>
 #include <linux/swap.h>
@@ -818,6 +819,14 @@ static void evict(struct inode *inode)
 	inode_wait_for_writeback(inode);
 	spin_unlock(&inode->i_lock);
 
+	/*
+	 * Whoever froze the mapping must hold an inode reference until it is
+	 * unfrozen, so this should never happen. If it does, unfreeze it so
+	 * the final truncate does not leak folios.
+	 */
+	if (WARN_ON_ONCE(mapping_frozen(&inode->i_data)))
+		clear_bit(AS_FROZEN, &inode->i_data.flags);
+
 	if (op->evict_inode) {
 		op->evict_inode(inode);
 	} else {
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 73af18a373670..9b9d0459f3992 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -209,6 +209,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_FROZEN = 11,		/* index -> folio mapping must not change */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,
@@ -344,6 +345,40 @@ static inline bool mapping_writeback_may_deadlock_on_reclaim(const struct addres
 	return test_bit(AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM, &mapping->flags);
 }
 
+/*
+ * A frozen mapping disallows any change to its i_pages: no folio can be added,
+ * removed, replaced (migration, swap out), split, or collapsed.
+ *
+ * Only the page cache is frozen, not the file. Operations that need to change
+ * the page cache fail with -EBUSY, e.g. a write or fault that needs to allocate
+ * a new folio. Others, like reading or writing (to pages already in i_pages),
+ * still work.
+ *
+ * Locking: the flag is flipped under the i_pages lock. The filesystem must
+ * ensure mapping_set_frozen() doesn't race with a truncate or fallocate, since
+ * those can fail if frozen mid-way.
+ *
+ * The mapping must be unfrozen before the inode is evicted.
+ */
+static inline void mapping_set_frozen(struct address_space *mapping)
+{
+	xa_lock_irq(&mapping->i_pages);
+	set_bit(AS_FROZEN, &mapping->flags);
+	xa_unlock_irq(&mapping->i_pages);
+}
+
+static inline void mapping_clear_frozen(struct address_space *mapping)
+{
+	xa_lock_irq(&mapping->i_pages);
+	clear_bit(AS_FROZEN, &mapping->flags);
+	xa_unlock_irq(&mapping->i_pages);
+}
+
+static inline bool mapping_frozen(const struct address_space *mapping)
+{
+	return test_bit(AS_FROZEN, &mapping->flags);
+}
+
 static inline gfp_t mapping_gfp_mask(const struct address_space *mapping)
 {
 	return mapping->gfp_mask;
diff --git a/mm/filemap.c b/mm/filemap.c
index b74bc1e5015c6..d040c69177f79 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -138,6 +138,7 @@ static void page_cache_delete(struct address_space *mapping,
 	nr = folio_nr_pages(folio);
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+	VM_WARN_ON_ONCE_FOLIO(mapping_frozen(mapping), folio);
 
 	xas_store(&xas, shadow);
 	xas_init_marks(&xas);
@@ -284,6 +285,7 @@ static void page_cache_delete_batch(struct address_space *mapping,
 	int i = 0;
 	struct folio *folio;
 
+	VM_WARN_ON_ONCE(mapping_frozen(mapping));
 	mapping_set_update(&xas, mapping);
 	xas_for_each(&xas, folio, ULONG_MAX) {
 		if (i >= folio_batch_count(fbatch))
@@ -826,6 +828,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new)
 	mem_cgroup_replace_folio(old, new);
 
 	xas_lock_irq(&xas);
+	VM_WARN_ON_ONCE_FOLIO(mapping_frozen(mapping), old);
 	xas_store(&xas, new);
 
 	old->mapping = NULL;
@@ -873,6 +876,10 @@ noinline int __filemap_add_folio(struct address_space *mapping,
 		void *entry, *old = NULL;
 
 		xas_lock_irq(&xas);
+		if (unlikely(mapping_frozen(mapping))) {
+			xas_set_err(&xas, -EBUSY);
+			goto unlock;
+		}
 		xas_for_each_conflict(&xas, entry) {
 			old = entry;
 			if (!xa_is_value(entry)) {
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index bd44d58e86845..06d2f71ec7498 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3948,6 +3948,10 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
 	if (!folio->mapping && !is_anon)
 		return -EBUSY;
 
+	/* Splitting would change a frozen mapping. */
+	if (!folio_test_anon(folio) && mapping_frozen(folio->mapping))
+		return -EBUSY;
+
 	/* order-1 is not supported for anonymous THP. */
 	if (is_anon && new_order == 1)
 		return -EINVAL;
@@ -4186,6 +4190,12 @@ static int __folio_freeze_split_file(struct folio *folio,
 
 	xas_lock_irq(&xas);
 
+	/* Can't change a frozen mapping. */
+	if (mapping_frozen(mapping)) {
+		ret = -EBUSY;
+		goto fail;
+	}
+
 	/*
 	 * Check if the folio is present in page cache.
 	 * We assume all tail are present too, if folio is there.
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index da91762f6358f..5c78573f57c47 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -2531,6 +2531,12 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
 		i_mmap_lock_read(mapping);
 		xas_lock_irq(&xas);
 
+		/* Can't collapse folios in a frozen mapping. */
+		if (mapping_frozen(mapping)) {
+			result = SCAN_FAIL;
+			goto immap_locked;
+		}
+
 		xas_set(&xas, start);
 		for (index = start; index < end; index++) {
 			if (!xas_next(&xas)) {
@@ -2582,6 +2588,12 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
 		}
 	} else {
 		xas_lock_irq(&xas);
+		/* Can't collapse folios in a frozen mapping. */
+		if (mapping_frozen(mapping)) {
+			xas_unlock_irq(&xas);
+			result = SCAN_FAIL;
+			goto rollback;
+		}
 	}
 
 	if (is_shmem) {
diff --git a/mm/migrate.c b/mm/migrate.c
index d649a6cfce6c0..6ce01871d72e3 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -614,10 +614,16 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 	oldzone = folio_zone(folio);
 	newzone = folio_zone(newfolio);
 
-	if (folio_test_swapcache(folio))
+	if (folio_test_swapcache(folio)) {
 		ci = swap_cluster_get_and_lock_irq(folio);
-	else
+	} else {
 		xas_lock_irq(&xas);
+		/* Can't replace folios in a frozen mapping. */
+		if (mapping_frozen(mapping)) {
+			xas_unlock_irq(&xas);
+			return -EBUSY;
+		}
+	}
 
 	if (!folio_ref_freeze(folio, expected_count)) {
 		if (ci)
@@ -750,6 +756,10 @@ int migrate_huge_page_move_mapping(struct address_space *mapping,
 		return rc;
 
 	xas_lock_irq(&xas);
+	if (mapping_frozen(mapping)) {
+		xas_unlock_irq(&xas);
+		return -EBUSY;
+	}
 	if (!folio_ref_freeze(src, expected_count)) {
 		xas_unlock_irq(&xas);
 		return -EAGAIN;
diff --git a/mm/shmem.c b/mm/shmem.c
index 13e0786436cda..6507b4246c8af 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1121,6 +1121,10 @@ int shmem_add_to_page_cache(struct folio *folio,
 	do {
 		iter = swap;
 		xas_lock_irq(&xas);
+		if (unlikely(mapping_frozen(mapping))) {
+			xas_set_err(&xas, -EBUSY);
+			goto unlock;
+		}
 		xas_for_each_conflict(&xas, entry) {
 			/*
 			 * The range must either be empty, or filled with
@@ -1345,6 +1349,10 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend,
 	if (lend == -1)
 		end = -1;	/* unsigned, so actually very big */
 
+	/* Callers must not try to truncate a frozen mapping. */
+	if (WARN_ON_ONCE(mapping_frozen(mapping)))
+		return;
+
 	if (info->fallocend > start && info->fallocend <= end && !unfalloc)
 		info->fallocend = start;
 
@@ -1822,6 +1830,10 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
 	if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap)
 		goto redirty;
 
+	/* Swapping out would replace the folio in a frozen mapping. */
+	if (mapping_frozen(mapping))
+		goto redirty;
+
 	if (!total_swap_pages)
 		goto redirty;
 
@@ -2483,6 +2495,13 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
 	if (softleaf_is_poison_marker(index_entry))
 		return -EIO;
 
+	/*
+	 * Swapping in would change the i_pages of a frozen mapping. A frozen
+	 * mapping should not have swap entries in the first place.
+	 */
+	if (unlikely(mapping_frozen(mapping)))
+		return -EBUSY;
+
 	si = get_swap_device(index_entry);
 	order = shmem_confirm_swap(mapping, index, index_entry);
 	if (IS_ERR_OR_NULL(si)) {
diff --git a/mm/truncate.c b/mm/truncate.c
index 733d2713a875a..6512f460a28e9 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -171,6 +171,8 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio)
 {
 	if (folio->mapping != mapping)
 		return -EIO;
+	if (mapping_frozen(mapping))
+		return -EBUSY;
 
 	truncate_cleanup_folio(folio);
 	filemap_remove_folio(folio);
@@ -367,6 +369,8 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio)
 		return 0;
 	if (folio_test_dirty(folio) || folio_test_writeback(folio))
 		return 0;
+	if (mapping_frozen(mapping))
+		return 0;
 	/* The refcount will be elevated if any page in the folio is mapped */
 	if (folio_ref_count(folio) >
 			folio_nr_pages(folio) + folio_has_private(folio) + 1)
@@ -419,6 +423,13 @@ void truncate_inode_pages_range(struct address_space *mapping,
 	if (mapping_empty(mapping))
 		return;
 
+	/*
+	 * Callers must not try to truncate a frozen mapping. Filesystems that
+	 * support frozen mappings must refuse shrinking and hole punching
+	 * before getting here.
+	 */
+	if (WARN_ON_ONCE(mapping_frozen(mapping)))
+		return;
 	/*
 	 * 'start' and 'end' always covers the range of folios to be fully
 	 * truncated, with both boundaries aligned inwards to 1 << min_order.
@@ -668,6 +679,9 @@ int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio,
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
 
+	if (mapping_frozen(mapping))
+		return -EBUSY;
+
 	if (folio_mapped(folio))
 		unmap_mapping_folio(folio);
 	BUG_ON(folio_mapped(folio));
@@ -682,7 +696,7 @@ int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio,
 
 	spin_lock(&mapping->host->i_lock);
 	xa_lock_irq(&mapping->i_pages);
-	if (folio_test_dirty(folio))
+	if (folio_test_dirty(folio) || mapping_frozen(mapping))
 		goto failed;
 
 	BUG_ON(folio_has_private(folio));
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce83..86ef233e8752e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -766,6 +766,9 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,
 	} else {
 		spin_lock(&mapping->host->i_lock);
 		xa_lock_irq(&mapping->i_pages);
+		/* Can't remove folios from a frozen mapping. */
+		if (mapping_frozen(mapping))
+			goto cannot_free;
 	}
 
 	/*
-- 
2.56.0.307.g5fe7bd680c-goog

[PATCH 3/3] mm: shmem: use mapping_set_frozen() for memfd preservation

[Pratyush Yadav]

From: "Pratyush Yadav (Google)" <pratyush@kernel.org>

memfd preservation via LUO freezes the shmem mapping with
SHMEM_F_MAPPING_FROZEN. That flag only blocks size changes and
fallocate(), so other paths can still change the page cache of a
preserved memfd. memfd_luo avoids it by pinning all folios and faulting
in all the holes.

The page cache now natively supports freezing mappings. Switch over the
code to use that instead.

Disallow any type of truncation on frozen mappings, even one with same
size, since it might drop fallocated folios past EOF, and would fail in
shmem_undo_range() with a warning.

memfd_luo_preserve() now freezes the mapping after grabbing the folios,
instead of before. The grabbing fills holes in the file and may migrate
folios, both of which a frozen mapping refuses. The inode lock is held
throughout, so truncate, fallocate and writes can't change the file in
between.

Signed-off-by: Pratyush Yadav (Google) <pratyush@kernel.org>
---
 include/linux/shmem_fs.h | 17 -----------------
 mm/memfd_luo.c           | 11 ++++++++---
 mm/shmem.c               | 15 +++++++--------
 3 files changed, 15 insertions(+), 28 deletions(-)

diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index a7c7a96a7cbf9..5b48598a7b952 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -23,14 +23,6 @@
 #define SHMEM_F_NORESERVE	BIT(0)
 /* Disallow swapping. */
 #define SHMEM_F_LOCKED		BIT(1)
-/*
- * Disallow growing, shrinking, or hole punching in the inode. Combined with
- * folio pinning, makes sure the inode's mapping stays fixed.
- *
- * In some ways similar to F_SEAL_GROW | F_SEAL_SHRINK, but can be removed and
- * isn't directly visible to userspace.
- */
-#define SHMEM_F_MAPPING_FROZEN	BIT(2)
 
 struct shmem_inode_info {
 	spinlock_t		lock;
@@ -202,15 +194,6 @@ static inline bool shmem_file(struct file *file)
 	return shmem_mapping(file->f_mapping);
 }
 
-/* Must be called with inode lock taken exclusive. */
-static inline void shmem_freeze(struct inode *inode, bool freeze)
-{
-	if (freeze)
-		SHMEM_I(inode)->flags |= SHMEM_F_MAPPING_FROZEN;
-	else
-		SHMEM_I(inode)->flags &= ~SHMEM_F_MAPPING_FROZEN;
-}
-
 /*
  * If fallocate(FALLOC_FL_KEEP_SIZE) has been used, there may be pages
  * beyond i_size's notion of EOF, which fallocate has committed to reserving:
diff --git a/mm/memfd_luo.c b/mm/memfd_luo.c
index fe182d5057890..ec5abd41ecb8b 100644
--- a/mm/memfd_luo.c
+++ b/mm/memfd_luo.c
@@ -84,6 +84,7 @@
 #include <linux/swap.h>
 #include <linux/vmalloc.h>
 #include <linux/memfd.h>
+#include <linux/pagemap.h>
 #include <uapi/linux/memfd.h>
 
 #include "internal.h"
@@ -323,7 +324,6 @@ static int memfd_luo_preserve(struct liveupdate_file_op_args *args)
 	int err = 0, seals;
 
 	inode_lock(inode);
-	shmem_freeze(inode, true);
 
 	/* Allocate the main serialization structure in preserved memory */
 	ser = kho_alloc_preserve(sizeof(*ser));
@@ -355,6 +355,12 @@ static int memfd_luo_preserve(struct liveupdate_file_op_args *args)
 	if (err)
 		goto err_free_ser;
 
+	/*
+	 * All the folios in the file are now stable. Freeze the mapping to make
+	 * sure new ones can't be added.
+	 */
+	mapping_set_frozen(inode->i_mapping);
+
 	ser->nr_folios = nr_folios;
 	inode_unlock(inode);
 
@@ -366,7 +372,6 @@ static int memfd_luo_preserve(struct liveupdate_file_op_args *args)
 err_free_ser:
 	kho_unpreserve_free(ser);
 err_unlock:
-	shmem_freeze(inode, false);
 	inode_unlock(inode);
 	return err;
 }
@@ -398,7 +403,7 @@ static void memfd_luo_unpreserve(struct liveupdate_file_op_args *args)
 		return;
 
 	inode_lock(inode);
-	shmem_freeze(inode, false);
+	mapping_clear_frozen(inode->i_mapping);
 
 	ser = phys_to_virt(args->serialized_data);
 
diff --git a/mm/shmem.c b/mm/shmem.c
index 6507b4246c8af..1c70fd306bec2 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1570,8 +1570,6 @@ static int shmem_setattr(struct mnt_idmap *idmap,
 			return -EPERM;
 
 		if (newsize != oldsize) {
-			if (info->flags & SHMEM_F_MAPPING_FROZEN)
-				return -EPERM;
 			error = shmem_reacct_size(SHMEM_I(inode)->flags,
 					oldsize, newsize);
 			if (error)
@@ -1583,6 +1581,11 @@ static int shmem_setattr(struct mnt_idmap *idmap,
 		}
 		if (newsize <= oldsize) {
 			loff_t holebegin = round_up(newsize, PAGE_SIZE);
+
+			/* No truncation allowed on a frozen mapping. */
+			if (mapping_frozen(inode->i_mapping))
+				return -EBUSY;
+
 			if (oldsize > holebegin)
 				unmap_mapping_range(inode->i_mapping,
 							holebegin, 0, 1);
@@ -3482,10 +3485,6 @@ shmem_write_begin(const struct kiocb *iocb, struct address_space *mapping,
 			return -EPERM;
 	}
 
-	if (unlikely((info->flags & SHMEM_F_MAPPING_FROZEN) &&
-		     pos + len > inode->i_size))
-		return -EPERM;
-
 	ret = shmem_get_folio(inode, index, pos + len, &folio, SGP_WRITE);
 	if (ret)
 		return ret;
@@ -3860,8 +3859,8 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,
 
 	inode_lock(inode);
 
-	if (info->flags & SHMEM_F_MAPPING_FROZEN) {
-		error = -EPERM;
+	if (mapping_frozen(inode->i_mapping)) {
+		error = -EBUSY;
 		goto out;
 	}
 
-- 
2.56.0.307.g5fe7bd680c-goog

Re: [PATCH 0/3] mm: allow freezing mappings

[David Hildenbrand (Arm)]

On 10/27/26 18:30, Pratyush Yadav wrote:
> From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
> 
> Hi,

Hi,

we're in the merge windows and focused on fixing things (apparently all of the
kernel). This is not the right time to discuss new changes.

-- 
Cheers,

David

Re: [PATCH 0/3] mm: allow freezing mappings

[Pratyush Yadav]

On Tue, Oct 27 2026, David Hildenbrand (Arm) wrote:

> On 10/27/26 18:30, Pratyush Yadav wrote:
>> From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
>> 
>> Hi,
>
> Hi,
>
> we're in the merge windows and focused on fixing things (apparently all of the
> kernel). This is not the right time to discuss new changes.

Sorry. I thought the MM tree was open to changes in the merge window.
Please feel free to ignore this until you are done with the merge
window. I can also resend it after -rc1/-rc2 if you'd like.

-- 
Regards,
Pratyush Yadav

Re: [PATCH 0/3] mm: allow freezing mappings

[David Hildenbrand (Arm)]

On 10/27/26 18:38, Pratyush Yadav wrote:
> On Tue, Oct 27 2026, David Hildenbrand (Arm) wrote:
> 
>> On 10/27/26 18:30, Pratyush Yadav wrote:
>>> From: "Pratyush Yadav (Google)" <pratyush@kernel.org>
>>>
>>> Hi,
>>
>> Hi,
>>
>> we're in the merge windows and focused on fixing things (apparently all of the
>> kernel). This is not the right time to discuss new changes.
> 
> Sorry. I thought the MM tree was open to changes in the merge window.

We usually prefer to get stuff outside the merge window, but right now we have
our hands full with fixes.

So I just wanted to set the expectation that this will likely be ignored and
might need a ping or a resend after the merge window.

(I'm afraid the fix party will not end before the merge window ends)

-- 
Cheers,

David