* [RFC PATCH 1/3] mm: move folio split code into mm/folio_split.c
2026-11-10 3:52 [RFC PATCH 0/3] mm: enable filesystem page-cache large folios without THP Fang Xieyan
@ 2026-11-10 3:52 ` Fang Xieyan
2026-11-10 3:52 ` [RFC PATCH 2/3] mm: build file-backed folio splitting with CONFIG_LARGE_FOLIO Fang Xieyan
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Fang Xieyan @ 2026-11-10 3:52 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Wilcox,
Jan Kara, Hugh Dickins, Carlos Maiolino
Cc: Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Usama Arif, Kiryl Shutsemau,
Lorenzo Stoakes, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Muchun Song, Kairui Song,
linux-mm, linux-kernel, linux-fsdevel, linux-xfs, Liu Chao
Move the folio split implementation out of mm/huge_memory.c into a new
file, mm/folio_split.c, so that the file-backed split path can later be
built without transparent hugepages.
This is a pure move: the functions are copied verbatim and in their
original order. The only edits are
- folio_reset_partially_mapped() is no longer static and gains a
declaration in mm/internal.h, because deferred_split_scan() in
huge_memory.c keeps calling it,
- the new file gets its own include list and is added to
obj-$(CONFIG_TRANSPARENT_HUGEPAGE), and
- mm/folio_split.c is added to the THP entry in MAINTAINERS.
The moved functions are unmap_folio(), remap_anon_folio(),
lru_add_split_folio(), page_range_has_hwpoisoned(),
__split_folio_to_order(), __split_frozen_folio(),
folio_check_splittable(), folio_swapcache_ref_count(),
folio_reset_partially_mapped(), __folio_freeze_split_anon(),
__folio_freeze_split_file(), __folio_split(), folio_split_unmapped(),
__split_huge_page_to_list_to_order(), folio_split(),
min_order_for_split() and split_folio_to_list().
The anonymous freeze path moves along with the file one for now,
because both use unmap_folio(), __split_frozen_folio() and
lru_add_split_folio(). Moving only the file path would have required
four cross-file helpers where one is enough today.
No functional change intended. Verified by building the same
configuration at the parent commit: the symbol sizes of the moved
functions are unchanged, and the disassembly of the moved code only
differs where a jump had to switch to a longer encoding because its
target left the rel8 range.
Reviewed-by: Liu Chao <liuc63@xiaopeng.com>
Signed-off-by: Fang Xieyan <fangxy@xiaopeng.com>
---
MAINTAINERS | 1 +
mm/Makefile | 2 +-
mm/folio_split.c | 947 +++++++++++++++++++++++++++++++++++++++++++++++
mm/huge_memory.c | 920 ---------------------------------------------
mm/internal.h | 1 +
5 files changed, 950 insertions(+), 921 deletions(-)
create mode 100644 mm/folio_split.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 0cf24eee2..d1b1dee6f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17943,6 +17943,7 @@ F: include/linux/khugepaged.h
F: include/trace/events/huge_memory.h
F: include/trace/events/thp.h
F: mm/collapse.h
+F: mm/folio_split.c
F: mm/huge_memory.c
F: mm/khugepaged.c
F: mm/mm_slot.h
diff --git a/mm/Makefile b/mm/Makefile
index 277b97f7f..7f7f2c114 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -100,7 +100,7 @@ obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o
obj-$(CONFIG_NUMA) += memory-tiers.o
obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
-obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
+obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += folio_split.o huge_memory.o khugepaged.o
obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o
obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
diff --git a/mm/folio_split.c b/mm/folio_split.c
new file mode 100644
index 000000000..7770d8215
--- /dev/null
+++ b/mm/folio_split.c
@@ -0,0 +1,947 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Folio split implementation, shared by the file-backed and the anonymous
+ * paths.
+ *
+ * Extracted from mm/huge_memory.c without functional changes, so that the
+ * file-backed path can later be built without transparent hugepages.
+ */
+
+#include <linux/mm.h>
+#include <linux/sched/mm.h>
+#include <linux/rmap.h>
+#include <linux/swap.h>
+#include <linux/mm_inline.h>
+#include <linux/swapops.h>
+#include <linux/backing-dev.h>
+#include <linux/dax.h>
+#include <linux/pagemap.h>
+#include <linux/migrate.h>
+#include <linux/memcontrol.h>
+#include <linux/shmem_fs.h>
+#include <linux/page_owner.h>
+#include <linux/pgalloc_tag.h>
+#include <linux/vmstat.h>
+
+#include "internal.h"
+#include "swap.h"
+
+/*
+ * A return value of 0 does not mean that unmapping succeeded. It might
+ * still have failed, but remap_anon_folio() must be called afterwards,
+ * for anon folios.
+ */
+static int unmap_folio(struct folio *folio)
+{
+ enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SYNC |
+ TTU_BATCH_FLUSH;
+
+ VM_BUG_ON_FOLIO(!folio_test_large(folio), folio);
+
+ /* Racy check if we can split the page, before we split PMDs */
+ if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
+ return -EAGAIN;
+
+ if (folio_test_pmd_mappable(folio))
+ ttu_flags |= TTU_SPLIT_HUGE_PMD;
+
+ /*
+ * Anon pages need migration entries to preserve them, but file
+ * pages can simply be left unmapped, then faulted back on demand.
+ */
+ if (folio_test_anon(folio))
+ try_to_migrate(folio, ttu_flags);
+ else
+ try_to_unmap(folio, ttu_flags | TTU_IGNORE_MLOCK);
+
+ try_to_unmap_flush();
+
+ return 0;
+}
+
+static void remap_anon_folio(struct folio *folio, unsigned long nr, int flags)
+{
+ int i = 0;
+
+ /*
+ * unmap_folio() installs migration entries only for anon folios,
+ * so currently only anon folios need to be remapped. File folios
+ * stay unmapped after the split and are faulted back on demand.
+ */
+ VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
+
+ for (;;) {
+ remove_migration_ptes(folio, folio, TTU_RMAP_LOCKED | flags);
+ i += folio_nr_pages(folio);
+ if (i >= nr)
+ break;
+ folio = folio_next(folio);
+ }
+}
+
+static void lru_add_split_folio(struct folio *folio, struct folio *new_folio,
+ struct lruvec *lruvec, struct list_head *list)
+{
+ VM_BUG_ON_FOLIO(folio_test_lru(new_folio), folio);
+ lockdep_assert_held(&lruvec->lru_lock);
+
+ if (folio_is_device_private(folio))
+ return;
+
+ if (list) {
+ /* page reclaim is reclaiming a huge page */
+ VM_WARN_ON(folio_test_lru(folio));
+ folio_get(new_folio);
+ list_add_tail(&new_folio->lru, list);
+ } else {
+ /* head is still on lru (and we have it frozen) */
+ VM_WARN_ON(!folio_test_lru(folio));
+ if (folio_test_unevictable(folio))
+ new_folio->mlock_count = 0;
+ else
+ list_add_tail(&new_folio->lru, &folio->lru);
+ folio_set_lru(new_folio);
+ }
+}
+
+static bool page_range_has_hwpoisoned(struct page *page, long nr_pages)
+{
+ for (; nr_pages; page++, nr_pages--)
+ if (PageHWPoison(page))
+ return true;
+ return false;
+}
+
+/*
+ * It splits @folio into @new_order folios and copies the @folio metadata to
+ * all the resulting folios.
+ */
+static void __split_folio_to_order(struct folio *folio, int old_order,
+ int new_order)
+{
+ /* Scan poisoned pages when split a poisoned folio to large folios */
+ const bool handle_hwpoison = folio_test_has_hwpoisoned(folio) && new_order;
+ long new_nr_pages = 1 << new_order;
+ long nr_pages = 1 << old_order;
+ long i;
+
+ folio_clear_has_hwpoisoned(folio);
+
+ /* Check first new_nr_pages since the loop below skips them */
+ if (handle_hwpoison &&
+ page_range_has_hwpoisoned(folio_page(folio, 0), new_nr_pages))
+ folio_set_has_hwpoisoned(folio);
+ /*
+ * Skip the first new_nr_pages, since the new folio from them have all
+ * the flags from the original folio.
+ */
+ for (i = new_nr_pages; i < nr_pages; i += new_nr_pages) {
+ struct page *new_head = &folio->page + i;
+ /*
+ * Careful: new_folio is not a "real" folio before we cleared PageTail.
+ * Don't pass it around before clear_compound_head().
+ */
+ struct folio *new_folio = (struct folio *)new_head;
+
+ VM_BUG_ON_PAGE(atomic_read(&new_folio->_mapcount) != -1, new_head);
+
+ /*
+ * Clone page flags before unfreezing refcount.
+ *
+ * After successful get_page_unless_zero() might follow flags change,
+ * for example lock_page() which set PG_waiters.
+ *
+ * Note that for mapped sub-pages of an anonymous THP,
+ * PG_anon_exclusive has been cleared in unmap_folio() and is stored in
+ * the migration entry instead from where remap_anon_folio() will restore it.
+ * We can still have PG_anon_exclusive set on effectively unmapped and
+ * unreferenced sub-pages of an anonymous THP: we can simply drop
+ * PG_anon_exclusive (-> PG_mappedtodisk) for these here.
+ */
+ new_folio->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+ new_folio->flags.f |= (folio->flags.f &
+ ((1L << PG_referenced) |
+ (1L << PG_swapbacked) |
+ (1L << PG_swapcache) |
+ (1L << PG_mlocked) |
+ (1L << PG_uptodate) |
+ (1L << PG_active) |
+ (1L << PG_workingset) |
+ (1L << PG_locked) |
+ (1L << PG_unevictable) |
+#ifdef CONFIG_ARCH_USES_PG_ARCH_2
+ (1L << PG_arch_2) |
+#endif
+#ifdef CONFIG_ARCH_USES_PG_ARCH_3
+ (1L << PG_arch_3) |
+#endif
+ (1L << PG_dirty) |
+ (1L << PG_dropbehind) |
+ LRU_GEN_MASK | LRU_REFS_MASK));
+
+ new_folio->mapping = folio->mapping;
+ new_folio->index = folio->index + i;
+
+ /*
+ * page->private should not be set in tail pages. Warn once
+ * if private is unexpectedly set. Do it before swap.val assignment
+ * since private overlaps with swap.val.
+ */
+ VM_WARN_ON_ONCE_PAGE(new_folio->private, new_head);
+
+ /*
+ * Not all folio fields are valid during a split, so open-code
+ * the swap entry rather than using folio_swap_entry().
+ */
+ if (folio_test_swapcache(folio))
+ new_folio->swap.val = folio->swap.val + i;
+
+ /* Page flags must be visible before we make the page non-compound. */
+ smp_wmb();
+
+ /*
+ * Clear PageTail before unfreezing page refcount.
+ *
+ * After successful get_page_unless_zero() might follow put_page()
+ * which needs correct compound_head().
+ */
+ clear_compound_head(new_head);
+ if (new_order) {
+ prep_compound_page(new_head, new_order);
+ folio_set_large_rmappable(new_folio);
+ }
+
+ /*
+ * PG_has_hwpoisoned is on the 2nd page, so set it after
+ * the compound head is prepped.
+ */
+ if (handle_hwpoison &&
+ page_range_has_hwpoisoned(new_head, new_nr_pages))
+ folio_set_has_hwpoisoned(new_folio);
+
+ if (folio_test_young(folio))
+ folio_set_young(new_folio);
+ if (folio_test_idle(folio))
+ folio_set_idle(new_folio);
+#ifdef CONFIG_MEMCG
+ new_folio->memcg_data = folio->memcg_data;
+#endif
+
+ folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio));
+ }
+
+ if (new_order)
+ folio_set_order(folio, new_order);
+ else
+ ClearPageCompound(&folio->page);
+}
+
+/**
+ * __split_frozen_folio() - splits a frozen @folio to lower order folios
+ * in two ways: uniform split or non-uniform split.
+ * @folio: the to-be-split folio
+ * @new_order: the smallest order of the after split folios (since buddy
+ * allocator like split generates folios with orders from @folio's
+ * order - 1 to new_order).
+ * @split_at: in buddy allocator like split, the folio containing @split_at
+ * will be split until its order becomes @new_order.
+ * @xas: xa_state pointing to folio->mapping->i_pages and locked by caller
+ * @split_type: if the split is uniform or not (buddy allocator like split)
+ *
+ *
+ * 1. uniform split: the given @folio into multiple @new_order small folios,
+ * where all small folios have the same order. This is done when
+ * split_type is SPLIT_TYPE_UNIFORM.
+ * 2. buddy allocator like (non-uniform) split: the given @folio is split into
+ * half and one of the half (containing the given page) is split into half
+ * until the given @folio's order becomes @new_order. This is done when
+ * split_type is SPLIT_TYPE_NON_UNIFORM.
+ *
+ * The high level flow for these two methods are:
+ *
+ * 1. uniform split: @xas is split with no expectation of failure and a single
+ * __split_folio_to_order() is called to split the @folio into @new_order
+ * along with stats update.
+ * 2. non-uniform split: folio_order - @new_order calls to
+ * __split_folio_to_order() are expected to be made in a for loop to split
+ * the @folio to one lower order at a time. The folio containing @split_at
+ * is split in each iteration. @xas is split into half in each iteration and
+ * can fail. A failed @xas split leaves split folios as is without merging
+ * them back.
+ *
+ * After splitting, the caller's folio reference will be transferred to the
+ * folio containing @split_at. The caller needs to unlock and/or free
+ * after-split folios if necessary.
+ *
+ * Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
+ * split but not to @new_order, the caller needs to check)
+ */
+static int __split_frozen_folio(struct folio *folio, int new_order,
+ struct page *split_at, struct xa_state *xas,
+ enum split_type split_type)
+{
+ const bool is_anon = folio_test_anon(folio);
+ int old_order = folio_order(folio);
+ int start_order = split_type == SPLIT_TYPE_UNIFORM ? new_order : old_order - 1;
+ struct folio *old_folio = folio;
+ int split_order;
+
+ /* Frozen implies unmapped, callers unmap before splitting. */
+ VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
+
+ /*
+ * split to new_order one order at a time. For uniform split,
+ * folio is split to new_order directly.
+ */
+ for (split_order = start_order;
+ split_order >= new_order;
+ split_order--) {
+ int nr_new_folios = 1UL << (old_order - split_order);
+
+ /* order-1 anonymous folio is not supported */
+ if (is_anon && split_order == 1)
+ continue;
+
+ if (xas) {
+ /*
+ * uniform split has xas_split_alloc() called before
+ * irq is disabled to allocate enough memory, whereas
+ * non-uniform split can handle ENOMEM.
+ * Use the to-be-split folio, so that a parallel
+ * folio_try_get() waits on it until xarray is updated
+ * with after-split folios and the original one is
+ * unfrozen.
+ */
+ if (split_type == SPLIT_TYPE_UNIFORM) {
+ xas_split(xas, old_folio, old_order);
+ } else {
+ xas_set_order(xas, folio->index, split_order);
+ xas_try_split(xas, old_folio, old_order);
+ if (xas_error(xas))
+ return xas_error(xas);
+ }
+ }
+
+ folio_split_memcg_refs(folio, old_order, split_order);
+ split_page_owner(&folio->page, old_order, split_order);
+ pgalloc_tag_split(folio, old_order, split_order);
+ __split_folio_to_order(folio, old_order, split_order);
+
+ if (is_anon) {
+ mod_mthp_stat(old_order, MTHP_STAT_NR_ANON, -1);
+ mod_mthp_stat(split_order, MTHP_STAT_NR_ANON, nr_new_folios);
+ }
+ /*
+ * If uniform split, the process is complete.
+ * If non-uniform, continue splitting the folio at @split_at
+ * as long as the next @split_order is >= @new_order.
+ */
+ folio = page_folio(split_at);
+ old_order = split_order;
+ }
+
+ return 0;
+}
+
+/**
+ * folio_check_splittable() - check if a folio can be split to a given order
+ * @folio: folio to be split
+ * @new_order: the smallest order of the after split folios (since buddy
+ * allocator like split generates folios with orders from @folio's
+ * order - 1 to new_order).
+ * @split_type: uniform or non-uniform split
+ *
+ * folio_check_splittable() checks if @folio can be split to @new_order using
+ * @split_type method. The truncated folio check must come first.
+ *
+ * Context: folio must be locked.
+ *
+ * Return: 0 - @folio can be split to @new_order, otherwise an error number is
+ * returned.
+ */
+int folio_check_splittable(struct folio *folio, unsigned int new_order,
+ enum split_type split_type)
+{
+ const bool is_anon = folio_test_anon(folio);
+ const bool is_swapcache = folio_test_swapcache(folio);
+
+ VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
+ /*
+ * Folios that just got truncated cannot get split. Signal to the
+ * caller that there was a race.
+ *
+ * TODO: this will also currently refuse folios without a mapping in the
+ * swapcache (shmem or to-be-anon folios).
+ */
+ if (!folio->mapping && !is_anon)
+ return -EBUSY;
+
+ /* order-1 is not supported for anonymous THP. */
+ if (is_anon && new_order == 1)
+ return -EINVAL;
+
+ /*
+ * swapcache folio could only be split to order 0
+ *
+ * non-uniform split creates after-split folios with orders from
+ * folio_order(folio) - 1 to new_order, making it not suitable for any
+ * swapcache folio split. Only uniform split to order-0 can be used
+ * here.
+ */
+ if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && is_swapcache)
+ return -EINVAL;
+
+ if (is_huge_zero_folio(folio))
+ return -EINVAL;
+
+ if (folio_test_writeback(folio))
+ return -EBUSY;
+
+ /*
+ * A non-anon swapcache folio that still has a mapping can only be a
+ * shmem folio under SWAP IO, it's removed from either swap cache or
+ * shmem mapping afterward. There is little benefit in splitting them
+ * hence reject it here up front before touching anything.
+ */
+ if (!is_anon && is_swapcache && folio->mapping)
+ return -EBUSY;
+
+ return 0;
+}
+
+/* Number of folio references from the swapcache. */
+static unsigned int folio_swapcache_ref_count(const struct folio *folio)
+{
+ if (!folio_test_swapcache(folio))
+ return 0;
+ return folio_nr_pages(folio);
+}
+
+void folio_reset_partially_mapped(struct folio *folio)
+{
+ /* Folio must be frozen. */
+ VM_WARN_ON_FOLIO(folio_ref_count(folio), folio);
+
+ if (!folio_test_partially_mapped(folio))
+ return;
+
+ /*
+ * Order-1 folios have no _deferred_list. The flag is only ever set
+ * on folios that do, so the list can be checked after the flag.
+ */
+ VM_WARN_ON_FOLIO(!list_empty(&folio->_deferred_list), folio);
+
+ folio_clear_partially_mapped(folio);
+ mod_mthp_stat(folio_order(folio),
+ MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
+}
+
+static int __folio_freeze_split_anon(struct folio *folio,
+ unsigned int new_order, struct page *split_at, bool do_lru,
+ struct list_head *list, enum split_type split_type)
+{
+ struct folio *end_folio = folio_next(folio);
+ struct swap_cluster_info *ci = NULL;
+ const int old_order = folio_order(folio);
+ struct folio *new_folio, *next;
+ struct anon_vma *anon_vma = NULL;
+ enum ttu_flags ttu_flags = 0;
+ struct lruvec *lruvec;
+ int ret = 0;
+
+ /*
+ * Unmap/remap needs the anon_vma, so we first take a reference on
+ * it to prevent it from disappearing, and lock it for write here,
+ * letting unmap_folio() walk the rmap with TTU_RMAP_LOCKED.
+ *
+ * folio_mapped() is not stable here, but it can only change in
+ * one direction while the folio is locked. The mapcount can drop
+ * to zero at any time, zap_pte_range() takes no folio lock. It
+ * cannot go up: swapin, migration and uffd move all lock the folio
+ * before mapping it, and fork only copies PTEs that already exist.
+ *
+ * So if we see the folio mapped, the worst case is an empty rmap
+ * walk. If we see it unmapped, it stays unmapped and needs neither
+ * the reference nor the lock. Anything else needs a reference
+ * first and folio_ref_freeze() below catches it.
+ *
+ * Note that entirely swapped-out THPs are unmapped but can be split.
+ */
+ if (folio_mapped(folio)) {
+ anon_vma = folio_get_anon_vma(folio);
+ if (!anon_vma)
+ return -EBUSY;
+ anon_vma_lock_write(anon_vma);
+ ret = unmap_folio(folio);
+ if (ret)
+ goto out_unlock;
+ }
+
+ local_irq_disable();
+
+ if (!folio_ref_freeze(folio, folio_swapcache_ref_count(folio) + 1)) {
+ ret = -EAGAIN;
+ goto out_no_split;
+ }
+
+ /* Take off the deferred split queue while frozen and memcg set */
+ folio_unqueue_deferred_split(folio);
+
+ /*
+ * deferred_split_scan() takes the folio off the queue before it
+ * splits it, so the unqueue above finds an empty list and
+ * leaves PG_partially_mapped set.
+ * Clear it here: the flag does not survive the split.
+ */
+ folio_reset_partially_mapped(folio);
+
+ if (folio_test_swapcache(folio))
+ ci = swap_cluster_get_and_lock(folio);
+
+ if (do_lru)
+ lruvec = folio_lruvec_lock(folio);
+
+ ret = __split_frozen_folio(folio, new_order, split_at, NULL, split_type);
+
+ /*
+ * Unfreeze the after-split folios and put them back to the right
+ * place. Keep the head @folio frozen until the end: sub entries
+ * in swap cache must be updated first, so a concurrent
+ * swap_cache_get_folio() cannot return the head folio for a sub
+ * entry (folio_try_get() will fail on the head @folio until unfreeze).
+ */
+ for (new_folio = folio_next(folio); new_folio != end_folio;
+ new_folio = next) {
+ next = folio_next(new_folio);
+ zone_device_private_split_cb(folio, new_folio);
+ folio_ref_unfreeze(new_folio,
+ folio_swapcache_ref_count(new_folio) + 1);
+ if (do_lru)
+ lru_add_split_folio(folio, new_folio, lruvec, list);
+ if (ci)
+ __swap_cache_replace_folio(ci, folio, new_folio);
+ }
+
+ zone_device_private_split_cb(folio, NULL);
+ folio_ref_unfreeze(folio, folio_swapcache_ref_count(folio) + 1);
+
+ if (do_lru)
+ lruvec_unlock(lruvec);
+ if (ci)
+ swap_cluster_unlock(ci);
+out_no_split:
+ local_irq_enable();
+ if (anon_vma) {
+ if (!ret && !folio_is_device_private(folio))
+ ttu_flags = TTU_USE_SHARED_ZEROPAGE;
+ remap_anon_folio(folio, 1 << old_order, ttu_flags);
+ }
+out_unlock:
+ if (anon_vma) {
+ anon_vma_unlock_write(anon_vma);
+ put_anon_vma(anon_vma);
+ }
+
+ return ret;
+}
+
+static int __folio_freeze_split_file(struct folio *folio,
+ unsigned int new_order, struct page *split_at,
+ struct list_head *list, enum split_type split_type)
+{
+ const long old_nr_pages = folio_nr_pages(folio);
+ struct address_space *mapping = folio->mapping;
+ XA_STATE(xas, &mapping->i_pages, folio->index);
+ struct folio *end_folio = folio_next(folio);
+ struct mem_cgroup *memcg, *old_memcg;
+ struct folio *new_folio, *next;
+ int nr_shmem_dropped = 0;
+ unsigned int min_order;
+ struct lruvec *lruvec;
+ pgoff_t end;
+ gfp_t gfp;
+ int ret = 0;
+
+ min_order = mapping_min_folio_order(mapping);
+ if (new_order < min_order)
+ return -EINVAL;
+
+ /*
+ * Switch to folio's memcg as xarray node allocation can happen and
+ * needs to charge to it.
+ */
+ memcg = get_mem_cgroup_from_folio(folio);
+ old_memcg = set_active_memcg(memcg);
+
+ gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK);
+ if (!filemap_release_folio(folio, gfp)) {
+ ret = -EBUSY;
+ goto fail_free;
+ }
+
+ mapping_set_update(&xas, mapping);
+
+ if (split_type == SPLIT_TYPE_UNIFORM) {
+ const int old_order = folio_order(folio);
+
+ xas_set_order(&xas, folio->index, new_order);
+ xas_split_alloc(&xas, folio, old_order, gfp);
+ if (xas_error(&xas)) {
+ ret = xas_error(&xas);
+ goto fail_free;
+ }
+ }
+
+ i_mmap_lock_read(mapping);
+
+ /* Currently device private folios can only back anonymous memory. */
+ VM_WARN_ON_ONCE_FOLIO(folio_is_device_private(folio), folio);
+
+ /*
+ * The loop below may need to trim off pages beyond
+ * EOF: but on 32-bit, i_size_read() takes an irq-unsafe
+ * seqlock, which cannot be nested inside the page tree lock.
+ * So note end now: i_size itself may be changed at any moment,
+ * but folio lock is good enough to serialize the trimming.
+ */
+ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
+ if (shmem_mapping(mapping))
+ end = shmem_fallocend(mapping->host, end);
+
+ ret = unmap_folio(folio);
+ if (ret)
+ goto fail_mmap_unlock;
+
+ xas_lock_irq(&xas);
+
+ /*
+ * Check if the folio is present in page cache.
+ * We assume all tail are present too, if folio is there.
+ */
+ if (xas_load(&xas) != folio) {
+ ret = -EAGAIN;
+ goto fail;
+ }
+
+ if (!folio_ref_freeze(folio, old_nr_pages + 1)) {
+ ret = -EAGAIN;
+ goto fail;
+ }
+
+ if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) {
+ if (folio_test_swapbacked(folio))
+ lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -old_nr_pages);
+ else
+ lruvec_stat_mod_folio(folio, NR_FILE_THPS, -old_nr_pages);
+ }
+
+ /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
+ lruvec = folio_lruvec_lock(folio);
+ ret = __split_frozen_folio(folio, new_order, split_at, &xas, split_type);
+
+ /*
+ * Unfreeze after-split folios and put them back to the right
+ * list. @folio should be kept frozen until page cache
+ * entries are updated with all the other after-split folios
+ * to prevent others seeing stale page cache entries.
+ * As a result, new_folio starts from the next folio of
+ * @folio.
+ */
+ for (new_folio = folio_next(folio); new_folio != end_folio;
+ new_folio = next) {
+ unsigned long nr_pages = folio_nr_pages(new_folio);
+
+ /* compute next before the folio can be freed below */
+ next = folio_next(new_folio);
+
+ folio_ref_unfreeze(new_folio,
+ folio_nr_pages(new_folio) + 1);
+
+ lru_add_split_folio(folio, new_folio, lruvec, list);
+
+ /* Add the new folio to the page cache. */
+ if (new_folio->index < end) {
+ __xa_store(&mapping->i_pages, new_folio->index,
+ new_folio, 0);
+ continue;
+ }
+
+ /* Drop folio beyond EOF: ->index >= end */
+ if (shmem_mapping(mapping))
+ nr_shmem_dropped += nr_pages;
+ else if (folio_test_clear_dirty(new_folio))
+ folio_account_cleaned(
+ new_folio, inode_to_wb(mapping->host));
+ __filemap_remove_folio(new_folio, NULL);
+ folio_put_refs(new_folio, nr_pages);
+ }
+
+ /*
+ * Unfreeze @folio only after all page cache entries, which
+ * used to point to it, have been updated with new folios.
+ * Otherwise, a parallel folio_try_get() can grab @folio
+ * and its caller can see stale page cache entries.
+ */
+ folio_ref_unfreeze(folio, folio_nr_pages(folio) + 1);
+ lruvec_unlock(lruvec);
+fail:
+ xas_unlock_irq(&xas);
+fail_mmap_unlock:
+ if (nr_shmem_dropped)
+ shmem_uncharge(mapping->host, nr_shmem_dropped);
+ /*
+ * Drop the mapping while the inode is still pinned. @folio stays
+ * locked and present in the page cache, so eviction cannot free
+ * the inode yet, nothing past this point may touch the inode or
+ * the mapping.
+ */
+ i_mmap_unlock_read(mapping);
+fail_free:
+ /* Restore the previously active memcg */
+ set_active_memcg(old_memcg);
+ mem_cgroup_put(memcg);
+ xas_destroy(&xas);
+ return ret;
+}
+
+/**
+ * __folio_split() - split a folio at @split_at to a @new_order folio
+ * @folio: folio to split
+ * @new_order: the order of the new folio
+ * @split_at: a page within the new folio
+ * @lock_at: a page within @folio to be left locked to caller
+ * @list: after-split folios will be put on it if non NULL
+ * @split_type: perform uniform split or not (non-uniform split)
+ *
+ * It calls __split_frozen_folio() to perform uniform and non-uniform split.
+ * It is in charge of checking whether the split is supported or not and
+ * preparing @folio for __split_frozen_folio().
+ *
+ * After splitting, the after-split folio containing @lock_at remains locked
+ * and others are unlocked:
+ * 1. for uniform split, @lock_at points to one of @folio's subpages;
+ * 2. for buddy allocator like (non-uniform) split, @lock_at points to @folio.
+ *
+ * Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
+ * split but not to @new_order, the caller needs to check)
+ */
+static int __folio_split(struct folio *folio, unsigned int new_order,
+ struct page *split_at, struct page *lock_at,
+ struct list_head *list, enum split_type split_type)
+{
+ struct folio *end_folio = folio_next(folio);
+ const bool is_anon = folio_test_anon(folio);
+ const bool is_swapcache = folio_test_swapcache(folio);
+ int old_order = folio_order(folio);
+ struct folio *new_folio, *next;
+ int ret;
+
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
+
+ if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (new_order >= old_order) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ ret = folio_check_splittable(folio, new_order, split_type);
+ if (ret) {
+ VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
+ goto out;
+ }
+
+ if (is_anon)
+ ret = __folio_freeze_split_anon(folio, new_order, split_at,
+ true, list, split_type);
+ else
+ ret = __folio_freeze_split_file(folio, new_order, split_at,
+ list, split_type);
+
+ /*
+ * Unlock all after-split folios except the one containing
+ * @lock_at page. If @folio is not split, it will be kept locked.
+ */
+ for (new_folio = folio; new_folio != end_folio; new_folio = next) {
+ next = folio_next(new_folio);
+ if (new_folio == page_folio(lock_at))
+ continue;
+
+ /*
+ * Subpages whose mapping has been zapped may be freed
+ * earlier, but freeing them requires taking the
+ * lru_lock, so we defer folio_put() on tail pages until
+ * after the split completes.
+ */
+ if (is_swapcache && !folio_mapped(new_folio))
+ folio_free_swap(new_folio);
+ folio_unlock(new_folio);
+ folio_put(new_folio);
+ }
+
+out:
+ if (is_pmd_order(old_order))
+ count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
+ count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
+ return ret;
+}
+
+/**
+ * folio_split_unmapped() - split a large anon folio that is already unmapped
+ * @folio: folio to split
+ * @new_order: the order of folios after split
+ *
+ * This function is a helper for splitting folios that have already been
+ * unmapped. The use case is that the device or the CPU can refuse to migrate
+ * THP pages in the middle of migration, due to allocation issues on either
+ * side.
+ *
+ * The anon rmap lock is not required to be held, mmap_read_lock() or
+ * mmap_write_lock() should be held. @folio is expected to be locked by the
+ * caller. device-private and non device-private folios are supported along
+ * with folios that are in the swapcache. @folio should also be unmapped and
+ * isolated from LRU (if applicable)
+ *
+ * Upon return, the folio is not remapped, split folios are not added to LRU,
+ * folio_free_swap() is not called, and new folios remain locked.
+ *
+ * Return: 0 on success, -EAGAIN if the folio cannot be split (e.g., due to
+ * insufficient reference count or extra pins).
+ */
+int folio_split_unmapped(struct folio *folio, unsigned int new_order)
+{
+ VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
+
+ return __folio_freeze_split_anon(folio, new_order, &folio->page,
+ false, NULL, SPLIT_TYPE_UNIFORM);
+}
+
+/*
+ * This function splits a large folio into smaller folios of order @new_order.
+ * @page can point to any page of the large folio to split. The split operation
+ * does not change the position of @page.
+ *
+ * Prerequisites:
+ *
+ * 1) The caller must hold a reference on the @page's owning folio, also known
+ * as the large folio.
+ *
+ * 2) The large folio must be locked.
+ *
+ * 3) The folio must not be pinned. Any unexpected folio references, including
+ * GUP pins, will result in the folio not getting split; instead, the caller
+ * will receive an -EAGAIN.
+ *
+ * 4) @new_order > 1, usually. Splitting to order-1 anonymous folios is not
+ * supported for non-file-backed folios, because folio->_deferred_list, which
+ * is used by partially mapped folios, is stored in subpage 2, but an order-1
+ * folio only has subpages 0 and 1. File-backed order-1 folios are supported,
+ * since they do not use _deferred_list.
+ *
+ * After splitting, the caller's folio reference will be transferred to @page,
+ * resulting in a raised refcount of @page after this call. The other pages may
+ * be freed if they are not mapped.
+ *
+ * If @list is null, tail pages will be added to LRU list, otherwise, to @list.
+ *
+ * Pages in @new_order will inherit the mapping, flags, and so on from the
+ * huge page.
+ *
+ * Returns 0 if the huge page was split successfully.
+ *
+ * Returns -EAGAIN if the folio has unexpected reference (e.g., GUP) or if
+ * the folio was concurrently removed from the page cache.
+ *
+ * Returns -EBUSY when trying to split the huge zeropage, if the folio is
+ * under writeback, if fs-specific folio metadata cannot currently be
+ * released, or if some unexpected race happened (e.g., anon VMA disappeared,
+ * truncation).
+ *
+ * Callers should ensure that the order respects the address space mapping
+ * min-order if one is set for non-anonymous folios.
+ *
+ * Returns -EINVAL when trying to split to an order that is incompatible
+ * with the folio. Splitting to order 0 is compatible with all folios.
+ */
+int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
+ unsigned int new_order)
+{
+ struct folio *folio = page_folio(page);
+
+ return __folio_split(folio, new_order, &folio->page, page, list,
+ SPLIT_TYPE_UNIFORM);
+}
+
+/**
+ * folio_split() - split a folio at @split_at to a @new_order folio
+ * @folio: folio to split
+ * @new_order: the order of the new folio
+ * @split_at: a page within the new folio
+ * @list: after-split folios are added to @list if not null, otherwise to LRU
+ * list
+ *
+ * It has the same prerequisites and returns as
+ * split_huge_page_to_list_to_order().
+ *
+ * Split a folio at @split_at to a new_order folio, leave the
+ * remaining subpages of the original folio as large as possible. For example,
+ * in the case of splitting an order-9 folio at its third order-3 subpages to
+ * an order-3 folio, there are 2^(9-3)=64 order-3 subpages in the order-9 folio.
+ * After the split, there will be a group of folios with different orders and
+ * the new folio containing @split_at is marked in bracket:
+ * [order-4, {order-3}, order-3, order-5, order-6, order-7, order-8].
+ *
+ * After split, folio is left locked for caller.
+ *
+ * Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
+ * split but not to @new_order, the caller needs to check)
+ */
+int folio_split(struct folio *folio, unsigned int new_order,
+ struct page *split_at, struct list_head *list)
+{
+ return __folio_split(folio, new_order, split_at, &folio->page, list,
+ SPLIT_TYPE_NON_UNIFORM);
+}
+
+/**
+ * min_order_for_split() - get the minimum order @folio can be split to
+ * @folio: folio to split
+ *
+ * min_order_for_split() tells the minimum order @folio can be split to.
+ * If a file-backed folio is truncated, 0 will be returned. Any subsequent
+ * split attempt should get -EBUSY from split checking code.
+ *
+ * Context: @folio must be locked.
+ *
+ * Return: @folio's minimum order for split
+ */
+unsigned int min_order_for_split(struct folio *folio)
+{
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
+ if (folio_test_anon(folio))
+ return 0;
+
+ /*
+ * If the folio got truncated, we don't know the previous mapping and
+ * consequently the old min order. But it doesn't matter, as any split
+ * attempt will immediately fail with -EBUSY as the folio cannot get
+ * split until freed.
+ */
+ if (!folio->mapping)
+ return 0;
+
+ return mapping_min_folio_order(folio->mapping);
+}
+
+int split_folio_to_list(struct folio *folio, struct list_head *list)
+{
+ return split_huge_page_to_list_to_order(&folio->page, list, 0);
+}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 70ad60936..5ef28984b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3582,39 +3582,6 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma,
split_huge_leaf(next, end);
}
-/*
- * A return value of 0 does not mean that unmapping succeeded. It might
- * still have failed, but remap_anon_folio() must be called afterwards,
- * for anon folios.
- */
-static int unmap_folio(struct folio *folio)
-{
- enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SYNC |
- TTU_BATCH_FLUSH;
-
- VM_BUG_ON_FOLIO(!folio_test_large(folio), folio);
-
- /* Racy check if we can split the page, before we split PMDs */
- if (folio_expected_ref_count(folio) != folio_ref_count(folio) - 1)
- return -EAGAIN;
-
- if (folio_test_pmd_mappable(folio))
- ttu_flags |= TTU_SPLIT_HUGE_PMD;
-
- /*
- * Anon pages need migration entries to preserve them, but file
- * pages can simply be left unmapped, then faulted back on demand.
- */
- if (folio_test_anon(folio))
- try_to_migrate(folio, ttu_flags);
- else
- try_to_unmap(folio, ttu_flags | TTU_IGNORE_MLOCK);
-
- try_to_unmap_flush();
-
- return 0;
-}
-
static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp,
struct folio *folio)
@@ -3692,893 +3659,6 @@ bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
return __discard_anon_folio_pmd_locked(vma, addr, pmdp, folio);
}
-static void remap_anon_folio(struct folio *folio, unsigned long nr, int flags)
-{
- int i = 0;
-
- /*
- * unmap_folio() installs migration entries only for anon folios,
- * so currently only anon folios need to be remapped. File folios
- * stay unmapped after the split and are faulted back on demand.
- */
- VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
-
- for (;;) {
- remove_migration_ptes(folio, folio, TTU_RMAP_LOCKED | flags);
- i += folio_nr_pages(folio);
- if (i >= nr)
- break;
- folio = folio_next(folio);
- }
-}
-
-static void lru_add_split_folio(struct folio *folio, struct folio *new_folio,
- struct lruvec *lruvec, struct list_head *list)
-{
- VM_BUG_ON_FOLIO(folio_test_lru(new_folio), folio);
- lockdep_assert_held(&lruvec->lru_lock);
-
- if (folio_is_device_private(folio))
- return;
-
- if (list) {
- /* page reclaim is reclaiming a huge page */
- VM_WARN_ON(folio_test_lru(folio));
- folio_get(new_folio);
- list_add_tail(&new_folio->lru, list);
- } else {
- /* head is still on lru (and we have it frozen) */
- VM_WARN_ON(!folio_test_lru(folio));
- if (folio_test_unevictable(folio))
- new_folio->mlock_count = 0;
- else
- list_add_tail(&new_folio->lru, &folio->lru);
- folio_set_lru(new_folio);
- }
-}
-
-static bool page_range_has_hwpoisoned(struct page *page, long nr_pages)
-{
- for (; nr_pages; page++, nr_pages--)
- if (PageHWPoison(page))
- return true;
- return false;
-}
-
-/*
- * It splits @folio into @new_order folios and copies the @folio metadata to
- * all the resulting folios.
- */
-static void __split_folio_to_order(struct folio *folio, int old_order,
- int new_order)
-{
- /* Scan poisoned pages when split a poisoned folio to large folios */
- const bool handle_hwpoison = folio_test_has_hwpoisoned(folio) && new_order;
- long new_nr_pages = 1 << new_order;
- long nr_pages = 1 << old_order;
- long i;
-
- folio_clear_has_hwpoisoned(folio);
-
- /* Check first new_nr_pages since the loop below skips them */
- if (handle_hwpoison &&
- page_range_has_hwpoisoned(folio_page(folio, 0), new_nr_pages))
- folio_set_has_hwpoisoned(folio);
- /*
- * Skip the first new_nr_pages, since the new folio from them have all
- * the flags from the original folio.
- */
- for (i = new_nr_pages; i < nr_pages; i += new_nr_pages) {
- struct page *new_head = &folio->page + i;
- /*
- * Careful: new_folio is not a "real" folio before we cleared PageTail.
- * Don't pass it around before clear_compound_head().
- */
- struct folio *new_folio = (struct folio *)new_head;
-
- VM_BUG_ON_PAGE(atomic_read(&new_folio->_mapcount) != -1, new_head);
-
- /*
- * Clone page flags before unfreezing refcount.
- *
- * After successful get_page_unless_zero() might follow flags change,
- * for example lock_page() which set PG_waiters.
- *
- * Note that for mapped sub-pages of an anonymous THP,
- * PG_anon_exclusive has been cleared in unmap_folio() and is stored in
- * the migration entry instead from where remap_anon_folio() will restore it.
- * We can still have PG_anon_exclusive set on effectively unmapped and
- * unreferenced sub-pages of an anonymous THP: we can simply drop
- * PG_anon_exclusive (-> PG_mappedtodisk) for these here.
- */
- new_folio->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
- new_folio->flags.f |= (folio->flags.f &
- ((1L << PG_referenced) |
- (1L << PG_swapbacked) |
- (1L << PG_swapcache) |
- (1L << PG_mlocked) |
- (1L << PG_uptodate) |
- (1L << PG_active) |
- (1L << PG_workingset) |
- (1L << PG_locked) |
- (1L << PG_unevictable) |
-#ifdef CONFIG_ARCH_USES_PG_ARCH_2
- (1L << PG_arch_2) |
-#endif
-#ifdef CONFIG_ARCH_USES_PG_ARCH_3
- (1L << PG_arch_3) |
-#endif
- (1L << PG_dirty) |
- (1L << PG_dropbehind) |
- LRU_GEN_MASK | LRU_REFS_MASK));
-
- new_folio->mapping = folio->mapping;
- new_folio->index = folio->index + i;
-
- /*
- * page->private should not be set in tail pages. Warn once
- * if private is unexpectedly set. Do it before swap.val assignment
- * since private overlaps with swap.val.
- */
- VM_WARN_ON_ONCE_PAGE(new_folio->private, new_head);
-
- /*
- * Not all folio fields are valid during a split, so open-code
- * the swap entry rather than using folio_swap_entry().
- */
- if (folio_test_swapcache(folio))
- new_folio->swap.val = folio->swap.val + i;
-
- /* Page flags must be visible before we make the page non-compound. */
- smp_wmb();
-
- /*
- * Clear PageTail before unfreezing page refcount.
- *
- * After successful get_page_unless_zero() might follow put_page()
- * which needs correct compound_head().
- */
- clear_compound_head(new_head);
- if (new_order) {
- prep_compound_page(new_head, new_order);
- folio_set_large_rmappable(new_folio);
- }
-
- /*
- * PG_has_hwpoisoned is on the 2nd page, so set it after
- * the compound head is prepped.
- */
- if (handle_hwpoison &&
- page_range_has_hwpoisoned(new_head, new_nr_pages))
- folio_set_has_hwpoisoned(new_folio);
-
- if (folio_test_young(folio))
- folio_set_young(new_folio);
- if (folio_test_idle(folio))
- folio_set_idle(new_folio);
-#ifdef CONFIG_MEMCG
- new_folio->memcg_data = folio->memcg_data;
-#endif
-
- folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio));
- }
-
- if (new_order)
- folio_set_order(folio, new_order);
- else
- ClearPageCompound(&folio->page);
-}
-
-/**
- * __split_frozen_folio() - splits a frozen @folio to lower order folios
- * in two ways: uniform split or non-uniform split.
- * @folio: the to-be-split folio
- * @new_order: the smallest order of the after split folios (since buddy
- * allocator like split generates folios with orders from @folio's
- * order - 1 to new_order).
- * @split_at: in buddy allocator like split, the folio containing @split_at
- * will be split until its order becomes @new_order.
- * @xas: xa_state pointing to folio->mapping->i_pages and locked by caller
- * @split_type: if the split is uniform or not (buddy allocator like split)
- *
- *
- * 1. uniform split: the given @folio into multiple @new_order small folios,
- * where all small folios have the same order. This is done when
- * split_type is SPLIT_TYPE_UNIFORM.
- * 2. buddy allocator like (non-uniform) split: the given @folio is split into
- * half and one of the half (containing the given page) is split into half
- * until the given @folio's order becomes @new_order. This is done when
- * split_type is SPLIT_TYPE_NON_UNIFORM.
- *
- * The high level flow for these two methods are:
- *
- * 1. uniform split: @xas is split with no expectation of failure and a single
- * __split_folio_to_order() is called to split the @folio into @new_order
- * along with stats update.
- * 2. non-uniform split: folio_order - @new_order calls to
- * __split_folio_to_order() are expected to be made in a for loop to split
- * the @folio to one lower order at a time. The folio containing @split_at
- * is split in each iteration. @xas is split into half in each iteration and
- * can fail. A failed @xas split leaves split folios as is without merging
- * them back.
- *
- * After splitting, the caller's folio reference will be transferred to the
- * folio containing @split_at. The caller needs to unlock and/or free
- * after-split folios if necessary.
- *
- * Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
- * split but not to @new_order, the caller needs to check)
- */
-static int __split_frozen_folio(struct folio *folio, int new_order,
- struct page *split_at, struct xa_state *xas,
- enum split_type split_type)
-{
- const bool is_anon = folio_test_anon(folio);
- int old_order = folio_order(folio);
- int start_order = split_type == SPLIT_TYPE_UNIFORM ? new_order : old_order - 1;
- struct folio *old_folio = folio;
- int split_order;
-
- /* Frozen implies unmapped, callers unmap before splitting. */
- VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
-
- /*
- * split to new_order one order at a time. For uniform split,
- * folio is split to new_order directly.
- */
- for (split_order = start_order;
- split_order >= new_order;
- split_order--) {
- int nr_new_folios = 1UL << (old_order - split_order);
-
- /* order-1 anonymous folio is not supported */
- if (is_anon && split_order == 1)
- continue;
-
- if (xas) {
- /*
- * uniform split has xas_split_alloc() called before
- * irq is disabled to allocate enough memory, whereas
- * non-uniform split can handle ENOMEM.
- * Use the to-be-split folio, so that a parallel
- * folio_try_get() waits on it until xarray is updated
- * with after-split folios and the original one is
- * unfrozen.
- */
- if (split_type == SPLIT_TYPE_UNIFORM) {
- xas_split(xas, old_folio, old_order);
- } else {
- xas_set_order(xas, folio->index, split_order);
- xas_try_split(xas, old_folio, old_order);
- if (xas_error(xas))
- return xas_error(xas);
- }
- }
-
- folio_split_memcg_refs(folio, old_order, split_order);
- split_page_owner(&folio->page, old_order, split_order);
- pgalloc_tag_split(folio, old_order, split_order);
- __split_folio_to_order(folio, old_order, split_order);
-
- if (is_anon) {
- mod_mthp_stat(old_order, MTHP_STAT_NR_ANON, -1);
- mod_mthp_stat(split_order, MTHP_STAT_NR_ANON, nr_new_folios);
- }
- /*
- * If uniform split, the process is complete.
- * If non-uniform, continue splitting the folio at @split_at
- * as long as the next @split_order is >= @new_order.
- */
- folio = page_folio(split_at);
- old_order = split_order;
- }
-
- return 0;
-}
-
-/**
- * folio_check_splittable() - check if a folio can be split to a given order
- * @folio: folio to be split
- * @new_order: the smallest order of the after split folios (since buddy
- * allocator like split generates folios with orders from @folio's
- * order - 1 to new_order).
- * @split_type: uniform or non-uniform split
- *
- * folio_check_splittable() checks if @folio can be split to @new_order using
- * @split_type method. The truncated folio check must come first.
- *
- * Context: folio must be locked.
- *
- * Return: 0 - @folio can be split to @new_order, otherwise an error number is
- * returned.
- */
-int folio_check_splittable(struct folio *folio, unsigned int new_order,
- enum split_type split_type)
-{
- const bool is_anon = folio_test_anon(folio);
- const bool is_swapcache = folio_test_swapcache(folio);
-
- VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
- /*
- * Folios that just got truncated cannot get split. Signal to the
- * caller that there was a race.
- *
- * TODO: this will also currently refuse folios without a mapping in the
- * swapcache (shmem or to-be-anon folios).
- */
- if (!folio->mapping && !is_anon)
- return -EBUSY;
-
- /* order-1 is not supported for anonymous THP. */
- if (is_anon && new_order == 1)
- return -EINVAL;
-
- /*
- * swapcache folio could only be split to order 0
- *
- * non-uniform split creates after-split folios with orders from
- * folio_order(folio) - 1 to new_order, making it not suitable for any
- * swapcache folio split. Only uniform split to order-0 can be used
- * here.
- */
- if ((split_type == SPLIT_TYPE_NON_UNIFORM || new_order) && is_swapcache)
- return -EINVAL;
-
- if (is_huge_zero_folio(folio))
- return -EINVAL;
-
- if (folio_test_writeback(folio))
- return -EBUSY;
-
- /*
- * A non-anon swapcache folio that still has a mapping can only be a
- * shmem folio under SWAP IO, it's removed from either swap cache or
- * shmem mapping afterward. There is little benefit in splitting them
- * hence reject it here up front before touching anything.
- */
- if (!is_anon && is_swapcache && folio->mapping)
- return -EBUSY;
-
- return 0;
-}
-
-/* Number of folio references from the swapcache. */
-static unsigned int folio_swapcache_ref_count(const struct folio *folio)
-{
- if (!folio_test_swapcache(folio))
- return 0;
- return folio_nr_pages(folio);
-}
-
-static void folio_reset_partially_mapped(struct folio *folio)
-{
- /* Folio must be frozen. */
- VM_WARN_ON_FOLIO(folio_ref_count(folio), folio);
-
- if (!folio_test_partially_mapped(folio))
- return;
-
- /*
- * Order-1 folios have no _deferred_list. The flag is only ever set
- * on folios that do, so the list can be checked after the flag.
- */
- VM_WARN_ON_FOLIO(!list_empty(&folio->_deferred_list), folio);
-
- folio_clear_partially_mapped(folio);
- mod_mthp_stat(folio_order(folio),
- MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
-}
-
-static int __folio_freeze_split_anon(struct folio *folio,
- unsigned int new_order, struct page *split_at, bool do_lru,
- struct list_head *list, enum split_type split_type)
-{
- struct folio *end_folio = folio_next(folio);
- struct swap_cluster_info *ci = NULL;
- const int old_order = folio_order(folio);
- struct folio *new_folio, *next;
- struct anon_vma *anon_vma = NULL;
- enum ttu_flags ttu_flags = 0;
- struct lruvec *lruvec;
- int ret = 0;
-
- /*
- * Unmap/remap needs the anon_vma, so we first take a reference on
- * it to prevent it from disappearing, and lock it for write here,
- * letting unmap_folio() walk the rmap with TTU_RMAP_LOCKED.
- *
- * folio_mapped() is not stable here, but it can only change in
- * one direction while the folio is locked. The mapcount can drop
- * to zero at any time, zap_pte_range() takes no folio lock. It
- * cannot go up: swapin, migration and uffd move all lock the folio
- * before mapping it, and fork only copies PTEs that already exist.
- *
- * So if we see the folio mapped, the worst case is an empty rmap
- * walk. If we see it unmapped, it stays unmapped and needs neither
- * the reference nor the lock. Anything else needs a reference
- * first and folio_ref_freeze() below catches it.
- *
- * Note that entirely swapped-out THPs are unmapped but can be split.
- */
- if (folio_mapped(folio)) {
- anon_vma = folio_get_anon_vma(folio);
- if (!anon_vma)
- return -EBUSY;
- anon_vma_lock_write(anon_vma);
- ret = unmap_folio(folio);
- if (ret)
- goto out_unlock;
- }
-
- local_irq_disable();
-
- if (!folio_ref_freeze(folio, folio_swapcache_ref_count(folio) + 1)) {
- ret = -EAGAIN;
- goto out_no_split;
- }
-
- /* Take off the deferred split queue while frozen and memcg set */
- folio_unqueue_deferred_split(folio);
-
- /*
- * deferred_split_scan() takes the folio off the queue before it
- * splits it, so the unqueue above finds an empty list and
- * leaves PG_partially_mapped set.
- * Clear it here: the flag does not survive the split.
- */
- folio_reset_partially_mapped(folio);
-
- if (folio_test_swapcache(folio))
- ci = swap_cluster_get_and_lock(folio);
-
- if (do_lru)
- lruvec = folio_lruvec_lock(folio);
-
- ret = __split_frozen_folio(folio, new_order, split_at, NULL, split_type);
-
- /*
- * Unfreeze the after-split folios and put them back to the right
- * place. Keep the head @folio frozen until the end: sub entries
- * in swap cache must be updated first, so a concurrent
- * swap_cache_get_folio() cannot return the head folio for a sub
- * entry (folio_try_get() will fail on the head @folio until unfreeze).
- */
- for (new_folio = folio_next(folio); new_folio != end_folio;
- new_folio = next) {
- next = folio_next(new_folio);
- zone_device_private_split_cb(folio, new_folio);
- folio_ref_unfreeze(new_folio,
- folio_swapcache_ref_count(new_folio) + 1);
- if (do_lru)
- lru_add_split_folio(folio, new_folio, lruvec, list);
- if (ci)
- __swap_cache_replace_folio(ci, folio, new_folio);
- }
-
- zone_device_private_split_cb(folio, NULL);
- folio_ref_unfreeze(folio, folio_swapcache_ref_count(folio) + 1);
-
- if (do_lru)
- lruvec_unlock(lruvec);
- if (ci)
- swap_cluster_unlock(ci);
-out_no_split:
- local_irq_enable();
- if (anon_vma) {
- if (!ret && !folio_is_device_private(folio))
- ttu_flags = TTU_USE_SHARED_ZEROPAGE;
- remap_anon_folio(folio, 1 << old_order, ttu_flags);
- }
-out_unlock:
- if (anon_vma) {
- anon_vma_unlock_write(anon_vma);
- put_anon_vma(anon_vma);
- }
-
- return ret;
-}
-
-static int __folio_freeze_split_file(struct folio *folio,
- unsigned int new_order, struct page *split_at,
- struct list_head *list, enum split_type split_type)
-{
- const long old_nr_pages = folio_nr_pages(folio);
- struct address_space *mapping = folio->mapping;
- XA_STATE(xas, &mapping->i_pages, folio->index);
- struct folio *end_folio = folio_next(folio);
- struct mem_cgroup *memcg, *old_memcg;
- struct folio *new_folio, *next;
- int nr_shmem_dropped = 0;
- unsigned int min_order;
- struct lruvec *lruvec;
- pgoff_t end;
- gfp_t gfp;
- int ret = 0;
-
- min_order = mapping_min_folio_order(mapping);
- if (new_order < min_order)
- return -EINVAL;
-
- /*
- * Switch to folio's memcg as xarray node allocation can happen and
- * needs to charge to it.
- */
- memcg = get_mem_cgroup_from_folio(folio);
- old_memcg = set_active_memcg(memcg);
-
- gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK);
- if (!filemap_release_folio(folio, gfp)) {
- ret = -EBUSY;
- goto fail_free;
- }
-
- mapping_set_update(&xas, mapping);
-
- if (split_type == SPLIT_TYPE_UNIFORM) {
- const int old_order = folio_order(folio);
-
- xas_set_order(&xas, folio->index, new_order);
- xas_split_alloc(&xas, folio, old_order, gfp);
- if (xas_error(&xas)) {
- ret = xas_error(&xas);
- goto fail_free;
- }
- }
-
- i_mmap_lock_read(mapping);
-
- /* Currently device private folios can only back anonymous memory. */
- VM_WARN_ON_ONCE_FOLIO(folio_is_device_private(folio), folio);
-
- /*
- * The loop below may need to trim off pages beyond
- * EOF: but on 32-bit, i_size_read() takes an irq-unsafe
- * seqlock, which cannot be nested inside the page tree lock.
- * So note end now: i_size itself may be changed at any moment,
- * but folio lock is good enough to serialize the trimming.
- */
- end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
- if (shmem_mapping(mapping))
- end = shmem_fallocend(mapping->host, end);
-
- ret = unmap_folio(folio);
- if (ret)
- goto fail_mmap_unlock;
-
- xas_lock_irq(&xas);
-
- /*
- * Check if the folio is present in page cache.
- * We assume all tail are present too, if folio is there.
- */
- if (xas_load(&xas) != folio) {
- ret = -EAGAIN;
- goto fail;
- }
-
- if (!folio_ref_freeze(folio, old_nr_pages + 1)) {
- ret = -EAGAIN;
- goto fail;
- }
-
- if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) {
- if (folio_test_swapbacked(folio))
- lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -old_nr_pages);
- else
- lruvec_stat_mod_folio(folio, NR_FILE_THPS, -old_nr_pages);
- }
-
- /* lock lru list/PageCompound, ref frozen by page_ref_freeze */
- lruvec = folio_lruvec_lock(folio);
- ret = __split_frozen_folio(folio, new_order, split_at, &xas, split_type);
-
- /*
- * Unfreeze after-split folios and put them back to the right
- * list. @folio should be kept frozen until page cache
- * entries are updated with all the other after-split folios
- * to prevent others seeing stale page cache entries.
- * As a result, new_folio starts from the next folio of
- * @folio.
- */
- for (new_folio = folio_next(folio); new_folio != end_folio;
- new_folio = next) {
- unsigned long nr_pages = folio_nr_pages(new_folio);
-
- /* compute next before the folio can be freed below */
- next = folio_next(new_folio);
-
- folio_ref_unfreeze(new_folio,
- folio_nr_pages(new_folio) + 1);
-
- lru_add_split_folio(folio, new_folio, lruvec, list);
-
- /* Add the new folio to the page cache. */
- if (new_folio->index < end) {
- __xa_store(&mapping->i_pages, new_folio->index,
- new_folio, 0);
- continue;
- }
-
- /* Drop folio beyond EOF: ->index >= end */
- if (shmem_mapping(mapping))
- nr_shmem_dropped += nr_pages;
- else if (folio_test_clear_dirty(new_folio))
- folio_account_cleaned(
- new_folio, inode_to_wb(mapping->host));
- __filemap_remove_folio(new_folio, NULL);
- folio_put_refs(new_folio, nr_pages);
- }
-
- /*
- * Unfreeze @folio only after all page cache entries, which
- * used to point to it, have been updated with new folios.
- * Otherwise, a parallel folio_try_get() can grab @folio
- * and its caller can see stale page cache entries.
- */
- folio_ref_unfreeze(folio, folio_nr_pages(folio) + 1);
- lruvec_unlock(lruvec);
-fail:
- xas_unlock_irq(&xas);
-fail_mmap_unlock:
- if (nr_shmem_dropped)
- shmem_uncharge(mapping->host, nr_shmem_dropped);
- /*
- * Drop the mapping while the inode is still pinned. @folio stays
- * locked and present in the page cache, so eviction cannot free
- * the inode yet, nothing past this point may touch the inode or
- * the mapping.
- */
- i_mmap_unlock_read(mapping);
-fail_free:
- /* Restore the previously active memcg */
- set_active_memcg(old_memcg);
- mem_cgroup_put(memcg);
- xas_destroy(&xas);
- return ret;
-}
-
-/**
- * __folio_split() - split a folio at @split_at to a @new_order folio
- * @folio: folio to split
- * @new_order: the order of the new folio
- * @split_at: a page within the new folio
- * @lock_at: a page within @folio to be left locked to caller
- * @list: after-split folios will be put on it if non NULL
- * @split_type: perform uniform split or not (non-uniform split)
- *
- * It calls __split_frozen_folio() to perform uniform and non-uniform split.
- * It is in charge of checking whether the split is supported or not and
- * preparing @folio for __split_frozen_folio().
- *
- * After splitting, the after-split folio containing @lock_at remains locked
- * and others are unlocked:
- * 1. for uniform split, @lock_at points to one of @folio's subpages;
- * 2. for buddy allocator like (non-uniform) split, @lock_at points to @folio.
- *
- * Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
- * split but not to @new_order, the caller needs to check)
- */
-static int __folio_split(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct page *lock_at,
- struct list_head *list, enum split_type split_type)
-{
- struct folio *end_folio = folio_next(folio);
- const bool is_anon = folio_test_anon(folio);
- const bool is_swapcache = folio_test_swapcache(folio);
- int old_order = folio_order(folio);
- struct folio *new_folio, *next;
- int ret;
-
- VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
- VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
-
- if (folio != page_folio(split_at) || folio != page_folio(lock_at)) {
- ret = -EINVAL;
- goto out;
- }
-
- if (new_order >= old_order) {
- ret = -EINVAL;
- goto out;
- }
-
- ret = folio_check_splittable(folio, new_order, split_type);
- if (ret) {
- VM_WARN_ONCE(ret == -EINVAL, "Tried to split an unsplittable folio");
- goto out;
- }
-
- if (is_anon)
- ret = __folio_freeze_split_anon(folio, new_order, split_at,
- true, list, split_type);
- else
- ret = __folio_freeze_split_file(folio, new_order, split_at,
- list, split_type);
-
- /*
- * Unlock all after-split folios except the one containing
- * @lock_at page. If @folio is not split, it will be kept locked.
- */
- for (new_folio = folio; new_folio != end_folio; new_folio = next) {
- next = folio_next(new_folio);
- if (new_folio == page_folio(lock_at))
- continue;
-
- /*
- * Subpages whose mapping has been zapped may be freed
- * earlier, but freeing them requires taking the
- * lru_lock, so we defer folio_put() on tail pages until
- * after the split completes.
- */
- if (is_swapcache && !folio_mapped(new_folio))
- folio_free_swap(new_folio);
- folio_unlock(new_folio);
- folio_put(new_folio);
- }
-
-out:
- if (is_pmd_order(old_order))
- count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
- count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
- return ret;
-}
-
-/**
- * folio_split_unmapped() - split a large anon folio that is already unmapped
- * @folio: folio to split
- * @new_order: the order of folios after split
- *
- * This function is a helper for splitting folios that have already been
- * unmapped. The use case is that the device or the CPU can refuse to migrate
- * THP pages in the middle of migration, due to allocation issues on either
- * side.
- *
- * The anon rmap lock is not required to be held, mmap_read_lock() or
- * mmap_write_lock() should be held. @folio is expected to be locked by the
- * caller. device-private and non device-private folios are supported along
- * with folios that are in the swapcache. @folio should also be unmapped and
- * isolated from LRU (if applicable)
- *
- * Upon return, the folio is not remapped, split folios are not added to LRU,
- * folio_free_swap() is not called, and new folios remain locked.
- *
- * Return: 0 on success, -EAGAIN if the folio cannot be split (e.g., due to
- * insufficient reference count or extra pins).
- */
-int folio_split_unmapped(struct folio *folio, unsigned int new_order)
-{
- VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
- VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
- VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
- VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
-
- return __folio_freeze_split_anon(folio, new_order, &folio->page,
- false, NULL, SPLIT_TYPE_UNIFORM);
-}
-
-/*
- * This function splits a large folio into smaller folios of order @new_order.
- * @page can point to any page of the large folio to split. The split operation
- * does not change the position of @page.
- *
- * Prerequisites:
- *
- * 1) The caller must hold a reference on the @page's owning folio, also known
- * as the large folio.
- *
- * 2) The large folio must be locked.
- *
- * 3) The folio must not be pinned. Any unexpected folio references, including
- * GUP pins, will result in the folio not getting split; instead, the caller
- * will receive an -EAGAIN.
- *
- * 4) @new_order > 1, usually. Splitting to order-1 anonymous folios is not
- * supported for non-file-backed folios, because folio->_deferred_list, which
- * is used by partially mapped folios, is stored in subpage 2, but an order-1
- * folio only has subpages 0 and 1. File-backed order-1 folios are supported,
- * since they do not use _deferred_list.
- *
- * After splitting, the caller's folio reference will be transferred to @page,
- * resulting in a raised refcount of @page after this call. The other pages may
- * be freed if they are not mapped.
- *
- * If @list is null, tail pages will be added to LRU list, otherwise, to @list.
- *
- * Pages in @new_order will inherit the mapping, flags, and so on from the
- * huge page.
- *
- * Returns 0 if the huge page was split successfully.
- *
- * Returns -EAGAIN if the folio has unexpected reference (e.g., GUP) or if
- * the folio was concurrently removed from the page cache.
- *
- * Returns -EBUSY when trying to split the huge zeropage, if the folio is
- * under writeback, if fs-specific folio metadata cannot currently be
- * released, or if some unexpected race happened (e.g., anon VMA disappeared,
- * truncation).
- *
- * Callers should ensure that the order respects the address space mapping
- * min-order if one is set for non-anonymous folios.
- *
- * Returns -EINVAL when trying to split to an order that is incompatible
- * with the folio. Splitting to order 0 is compatible with all folios.
- */
-int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
- unsigned int new_order)
-{
- struct folio *folio = page_folio(page);
-
- return __folio_split(folio, new_order, &folio->page, page, list,
- SPLIT_TYPE_UNIFORM);
-}
-
-/**
- * folio_split() - split a folio at @split_at to a @new_order folio
- * @folio: folio to split
- * @new_order: the order of the new folio
- * @split_at: a page within the new folio
- * @list: after-split folios are added to @list if not null, otherwise to LRU
- * list
- *
- * It has the same prerequisites and returns as
- * split_huge_page_to_list_to_order().
- *
- * Split a folio at @split_at to a new_order folio, leave the
- * remaining subpages of the original folio as large as possible. For example,
- * in the case of splitting an order-9 folio at its third order-3 subpages to
- * an order-3 folio, there are 2^(9-3)=64 order-3 subpages in the order-9 folio.
- * After the split, there will be a group of folios with different orders and
- * the new folio containing @split_at is marked in bracket:
- * [order-4, {order-3}, order-3, order-5, order-6, order-7, order-8].
- *
- * After split, folio is left locked for caller.
- *
- * Return: 0 - successful, <0 - failed (if -ENOMEM is returned, @folio might be
- * split but not to @new_order, the caller needs to check)
- */
-int folio_split(struct folio *folio, unsigned int new_order,
- struct page *split_at, struct list_head *list)
-{
- return __folio_split(folio, new_order, split_at, &folio->page, list,
- SPLIT_TYPE_NON_UNIFORM);
-}
-
-/**
- * min_order_for_split() - get the minimum order @folio can be split to
- * @folio: folio to split
- *
- * min_order_for_split() tells the minimum order @folio can be split to.
- * If a file-backed folio is truncated, 0 will be returned. Any subsequent
- * split attempt should get -EBUSY from split checking code.
- *
- * Context: @folio must be locked.
- *
- * Return: @folio's minimum order for split
- */
-unsigned int min_order_for_split(struct folio *folio)
-{
- VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
-
- if (folio_test_anon(folio))
- return 0;
-
- /*
- * If the folio got truncated, we don't know the previous mapping and
- * consequently the old min order. But it doesn't matter, as any split
- * attempt will immediately fail with -EBUSY as the folio cannot get
- * split until freed.
- */
- if (!folio->mapping)
- return 0;
-
- return mapping_min_folio_order(folio->mapping);
-}
-
-int split_folio_to_list(struct folio *folio, struct list_head *list)
-{
- return split_huge_page_to_list_to_order(&folio->page, list, 0);
-}
-
/*
* __folio_unqueue_deferred_split() is not to be called directly:
* the folio_unqueue_deferred_split() inline wrapper in mm/internal.h
diff --git a/mm/internal.h b/mm/internal.h
index 3a1643afe..ca549ad5e 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -779,6 +779,7 @@ static inline void folio_set_order(struct folio *folio, unsigned int order)
#endif
}
+void folio_reset_partially_mapped(struct folio *folio);
bool __folio_unqueue_deferred_split(struct folio *folio);
static inline bool folio_unqueue_deferred_split(struct folio *folio)
{
--
2.50.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [RFC PATCH 2/3] mm: build file-backed folio splitting with CONFIG_LARGE_FOLIO
2026-11-10 3:52 [RFC PATCH 0/3] mm: enable filesystem page-cache large folios without THP Fang Xieyan
2026-11-10 3:52 ` [RFC PATCH 1/3] mm: move folio split code into mm/folio_split.c Fang Xieyan
@ 2026-11-10 3:52 ` Fang Xieyan
2026-11-10 3:52 ` [RFC PATCH 3/3] mm: enable filesystem page-cache large folios without THP Fang Xieyan
2026-11-10 4:01 ` [RFC PATCH 0/3] " Zi Yan
3 siblings, 0 replies; 7+ messages in thread
From: Fang Xieyan @ 2026-11-10 3:52 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Matthew Wilcox,
Jan Kara, Hugh Dickins, Carlos Maiolino
Cc: Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Usama Arif, Kiryl Shutsemau,
Lorenzo Stoakes, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Muchun Song, Kairui Song,
linux-mm, linux-kernel, linux-fsdevel, linux-xfs, Liu Chao
Transparent hugepages gate the folio split implementation, so a
filesystem cannot split a large page-cache folio when THP is disabled.
Add CONFIG_LARGE_FOLIO to build that code on its own.
This only makes splitting possible; it does not enable large folio
allocation. The page cache still caps folios at order 0 through
mapping_max_folio_order() and friends in pagemap.h, which are left
untouched here.
The anonymous path stays behind CONFIG_TRANSPARENT_HUGEPAGE. Large
anonymous folios are supported only by the THP/mTHP implementation, so
with LARGE_FOLIO alone the anonymous split dispatch returns
-EOPNOTSUPP instead of attempting to walk an anon_vma that cannot
exist, while requests that fail the checks ahead of it keep returning
-EINVAL or -EBUSY. Inside unmap_folio(), the same condition replaces
try_to_migrate() with -EOPNOTSUPP for anon folios;
__folio_freeze_split_anon(), remap_anon_folio() and
folio_swapcache_ref_count() are not compiled at all.
The split API in huge_mm.h that LARGE_FOLIO needs moves out of the THP
block into a gate of its own. folio_split_unmapped() is anonymous
only, so its declaration stays behind CONFIG_TRANSPARENT_HUGEPAGE,
matching its definition. The existing !THP stubs stay for the case
where neither option is set. ClearPageCompound() in page-flags.h needs
the same treatment.
Two THP-specific accounting sites move with the code, and both keep a
CONFIG_TRANSPARENT_HUGEPAGE guard, but for different reasons.
THP_SPLIT_PAGE and THP_SPLIT_PAGE_FAILED only exist in the THP block of
the vm_event enum, so the __folio_split() epilogue that counts them has
to be guarded.
The NR_FILE_THPS/NR_SHMEM_THPS adjustment in
__folio_freeze_split_file() is file-backed code and is guarded for two
other reasons. folio_test_pmd_mappable() is a false stub without THP,
so the test could never fire, and the comparison is written in terms of
HPAGE_PMD_ORDER, which is HPAGE_PMD_SHIFT - PAGE_SHIFT where
HPAGE_PMD_SHIFT expands to BUILD_BUG() unless
CONFIG_PGTABLE_HAS_HUGE_LEAVES is set. That option is def_bool
TRANSPARENT_HUGEPAGE || HUGETLB_PAGE, so HUGETLBFS can provide it
without THP: HPAGE_PMD_ORDER and is_pmd_order() are not THP-only
constructs, only this particular accounting test is dropped.
THP builds are unaffected: with both options set the preprocessed
output is unchanged, and mm/huge_memory.o and mm/folio_split.o are
byte-identical to the previous commit.
Reviewed-by: Liu Chao <liuc63@xiaopeng.com>
Signed-off-by: Fang Xieyan <fangxy@xiaopeng.com>
---
include/linux/huge_mm.h | 60 +++++++++++++++++++++++---------------
include/linux/page-flags.h | 5 +++-
mm/Kconfig | 12 ++++++++
mm/Makefile | 3 +-
mm/folio_split.c | 30 ++++++++++++++++---
5 files changed, 80 insertions(+), 30 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index b4b91b748..458e44044 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -179,8 +179,37 @@ static inline void count_mthp_stat(int order, enum mthp_stat_item item)
}
#endif
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_LARGE_FOLIO)
+enum split_type {
+ SPLIT_TYPE_UNIFORM,
+ SPLIT_TYPE_NON_UNIFORM,
+};
+
+int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
+ unsigned int new_order);
+unsigned int min_order_for_split(struct folio *folio);
+int split_folio_to_list(struct folio *folio, struct list_head *list);
+int folio_check_splittable(struct folio *folio, unsigned int new_order,
+ enum split_type split_type);
+int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
+ struct list_head *list);
+
+static inline int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
+ unsigned int new_order)
+{
+ return __split_huge_page_to_list_to_order(page, list, new_order);
+}
+static inline int split_huge_page_to_order(struct page *page, unsigned int new_order)
+{
+ return split_huge_page_to_list_to_order(page, NULL, new_order);
+}
+#endif
+
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+/* Only anon folios can be split while they are unmapped. */
+int folio_split_unmapped(struct folio *folio, unsigned int new_order);
+
extern unsigned long transparent_hugepage_flags;
extern unsigned long huge_anon_orders_always;
extern unsigned long huge_anon_orders_madvise;
@@ -394,30 +423,6 @@ unsigned long thp_get_unmapped_area_vmaflags(struct file *filp, unsigned long ad
unsigned long len, unsigned long pgoff, unsigned long flags,
vma_flags_t vma_flags);
-enum split_type {
- SPLIT_TYPE_UNIFORM,
- SPLIT_TYPE_NON_UNIFORM,
-};
-
-int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
- unsigned int new_order);
-int folio_split_unmapped(struct folio *folio, unsigned int new_order);
-unsigned int min_order_for_split(struct folio *folio);
-int split_folio_to_list(struct folio *folio, struct list_head *list);
-int folio_check_splittable(struct folio *folio, unsigned int new_order,
- enum split_type split_type);
-int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
- struct list_head *list);
-
-static inline int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
- unsigned int new_order)
-{
- return __split_huge_page_to_list_to_order(page, list, new_order);
-}
-static inline int split_huge_page_to_order(struct page *page, unsigned int new_order)
-{
- return split_huge_page_to_list_to_order(page, NULL, new_order);
-}
static inline int split_huge_page(struct page *page)
{
@@ -641,6 +646,7 @@ thp_get_unmapped_area_vmaflags(struct file *filp, unsigned long addr,
return 0;
}
+#if !IS_ENABLED(CONFIG_LARGE_FOLIO)
static inline int
split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
unsigned int new_order)
@@ -653,12 +659,16 @@ static inline int split_huge_page_to_order(struct page *page, unsigned int new_o
VM_WARN_ON_ONCE_PAGE(1, page);
return -EINVAL;
}
+#endif
+
static inline int split_huge_page(struct page *page)
{
VM_WARN_ON_ONCE_PAGE(1, page);
return -EINVAL;
}
+#if !IS_ENABLED(CONFIG_LARGE_FOLIO)
+
static inline unsigned int min_order_for_split(struct folio *folio)
{
VM_WARN_ON_ONCE_FOLIO(1, folio);
@@ -677,6 +687,8 @@ static inline int folio_split(struct folio *folio, unsigned int new_order,
VM_WARN_ON_ONCE_FOLIO(1, folio);
return -EINVAL;
}
+#endif
+
static inline int folio_memcg_alloc_deferred(struct folio *folio)
{
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index b0ddc652e..6561ee02e 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -869,12 +869,15 @@ static inline bool folio_test_large(const struct folio *folio)
return folio_test_head(folio);
}
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_LARGE_FOLIO)
static inline void ClearPageCompound(struct page *page)
{
BUG_ON(!PageHead(page));
ClearPageHead(page);
}
+#endif
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
FOLIO_FLAG(large_rmappable, FOLIO_SECOND_PAGE)
FOLIO_FLAG(partially_mapped, FOLIO_SECOND_PAGE)
#else
diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b..9f97ad698 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -827,6 +827,18 @@ config PERSISTENT_HUGE_ZERO_FOLIO
config MM_ID
def_bool n
+config LARGE_FOLIO
+ bool "Large folio support for the page cache (without THP)" if !TRANSPARENT_HUGEPAGE
+ depends on MMU
+ select XARRAY_MULTI
+ default y if TRANSPARENT_HUGEPAGE
+ default n
+ help
+ Build the folio split implementation for page-cache large folios
+ even when transparent hugepages are disabled. Transparent
+ hugepages imply this option; this option alone does not enable
+ large folio allocation, it only makes splitting them possible.
+
menuconfig TRANSPARENT_HUGEPAGE
bool "Transparent Hugepage Support"
depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE && !PREEMPT_RT
diff --git a/mm/Makefile b/mm/Makefile
index 7f7f2c114..a1619ea47 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -100,7 +100,8 @@ obj-$(CONFIG_MEMTEST) += memtest.o
obj-$(CONFIG_MIGRATION) += migrate.o
obj-$(CONFIG_NUMA) += memory-tiers.o
obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
-obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += folio_split.o huge_memory.o khugepaged.o
+obj-$(CONFIG_LARGE_FOLIO) += folio_split.o
+obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o
obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
diff --git a/mm/folio_split.c b/mm/folio_split.c
index 7770d8215..91b3d78ba 100644
--- a/mm/folio_split.c
+++ b/mm/folio_split.c
@@ -49,16 +49,22 @@ static int unmap_folio(struct folio *folio)
* Anon pages need migration entries to preserve them, but file
* pages can simply be left unmapped, then faulted back on demand.
*/
- if (folio_test_anon(folio))
+ if (folio_test_anon(folio)) {
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
try_to_migrate(folio, ttu_flags);
- else
+#else
+ return -EOPNOTSUPP;
+#endif
+ } else {
try_to_unmap(folio, ttu_flags | TTU_IGNORE_MLOCK);
+ }
try_to_unmap_flush();
return 0;
}
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
static void remap_anon_folio(struct folio *folio, unsigned long nr, int flags)
{
int i = 0;
@@ -78,6 +84,7 @@ static void remap_anon_folio(struct folio *folio, unsigned long nr, int flags)
folio = folio_next(folio);
}
}
+#endif
static void lru_add_split_folio(struct folio *folio, struct folio *new_folio,
struct lruvec *lruvec, struct list_head *list)
@@ -409,6 +416,7 @@ int folio_check_splittable(struct folio *folio, unsigned int new_order,
return 0;
}
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
/* Number of folio references from the swapcache. */
static unsigned int folio_swapcache_ref_count(const struct folio *folio)
{
@@ -416,6 +424,7 @@ static unsigned int folio_swapcache_ref_count(const struct folio *folio)
return 0;
return folio_nr_pages(folio);
}
+#endif
void folio_reset_partially_mapped(struct folio *folio)
{
@@ -436,6 +445,7 @@ void folio_reset_partially_mapped(struct folio *folio)
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
static int __folio_freeze_split_anon(struct folio *folio,
unsigned int new_order, struct page *split_at, bool do_lru,
struct list_head *list, enum split_type split_type)
@@ -544,6 +554,7 @@ static int __folio_freeze_split_anon(struct folio *folio,
return ret;
}
+#endif
static int __folio_freeze_split_file(struct folio *folio,
unsigned int new_order, struct page *split_at,
@@ -628,12 +639,14 @@ static int __folio_freeze_split_file(struct folio *folio,
goto fail;
}
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
if (folio_test_pmd_mappable(folio) && new_order < HPAGE_PMD_ORDER) {
if (folio_test_swapbacked(folio))
lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -old_nr_pages);
else
lruvec_stat_mod_folio(folio, NR_FILE_THPS, -old_nr_pages);
}
+#endif
/* lock lru list/PageCompound, ref frozen by page_ref_freeze */
lruvec = folio_lruvec_lock(folio);
@@ -755,12 +768,17 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
goto out;
}
- if (is_anon)
+ if (is_anon) {
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
ret = __folio_freeze_split_anon(folio, new_order, split_at,
true, list, split_type);
- else
+#else
+ ret = -EOPNOTSUPP;
+#endif
+ } else {
ret = __folio_freeze_split_file(folio, new_order, split_at,
list, split_type);
+ }
/*
* Unlock all after-split folios except the one containing
@@ -784,12 +802,15 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
}
out:
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
if (is_pmd_order(old_order))
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
+#endif
count_mthp_stat(old_order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED);
return ret;
}
+#if IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)
/**
* folio_split_unmapped() - split a large anon folio that is already unmapped
* @folio: folio to split
@@ -822,6 +843,7 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
return __folio_freeze_split_anon(folio, new_order, &folio->page,
false, NULL, SPLIT_TYPE_UNIFORM);
}
+#endif
/*
* This function splits a large folio into smaller folios of order @new_order.
--
2.50.1
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [RFC PATCH 0/3] mm: enable filesystem page-cache large folios without THP
2026-11-10 3:52 [RFC PATCH 0/3] mm: enable filesystem page-cache large folios without THP Fang Xieyan
` (2 preceding siblings ...)
2026-11-10 3:52 ` [RFC PATCH 3/3] mm: enable filesystem page-cache large folios without THP Fang Xieyan
@ 2026-11-10 4:01 ` Zi Yan
2026-11-10 4:10 ` Zi Yan
3 siblings, 1 reply; 7+ messages in thread
From: Zi Yan @ 2026-11-10 4:01 UTC (permalink / raw)
To: Fang Xieyan, Andrew Morton, David Hildenbrand, Matthew Wilcox,
Jan Kara, Hugh Dickins, Carlos Maiolino, Pankaj Raghav
Cc: Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Usama Arif, Kiryl Shutsemau,
Lorenzo Stoakes, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Muchun Song, Kairui Song,
linux-mm, linux-kernel, linux-fsdevel, linux-xfs
+Pankaj, since he was working on this[1].
[1] https://lore.kernel.org/all/20251206030858.1418814-1-p.raghav@samsung.com/
On Mon Nov 9, 2026 at 10:52 PM EST, Fang Xieyan wrote:
> This series makes filesystem page-cache large folios work on kernels
> built with CONFIG_TRANSPARENT_HUGEPAGE=n, and keeps them splittable
> there.
>
> 1. Why this is needed
>
> TRANSPARENT_HUGEPAGE depends on !PREEMPT_RT (mm/Kconfig), so a
> PREEMPT_RT kernel always has THP off. Today that also removes
> file-backed large folios entirely: the helpers that split them live in
> mm/huge_memory.c under CONFIG_TRANSPARENT_HUGEPAGE, and the order
> accessors in include/linux/pagemap.h short-circuit to "no large
> folios" when THP is off, so every filesystem that opts in silently
> ends up with order-0 folios.
>
> The consequence is not only throughput. XFS allows block sizes up to
> 64K and fs/xfs/xfs_super.c refuses to mount anything larger than
> mapping_max_folio_size_supported(). With THP=n that helper reports
> PAGE_SIZE, so a 64K-block filesystem cannot be mounted at all:
>
> XFS (vda): block size (65536 bytes) not supported; Only block size (4096) or less is supported
>
> There is no degraded mode to fall back to, because a block has to fit
> in a single page-cache folio - which is why
> fs/xfs/libxfs/xfs_ialloc.c derives min_folio_order from
> sb_blocklog - PAGE_SHIFT. Either page-cache large folios work without
> THP, or the filesystem is unusable.
>
> 2. What the series does
>
> The anon and file split helpers were already separated by Kairui
> Song's "mm/huge_memory: clean up and decouple the anon and file split
> helpers" (v6 04/17, in mm-unstable); that is the base this series
> starts from.
>
> 1/3 moves the folio-splitting implementation out of mm/huge_memory.c
> into mm/folio_split.c without changing any behaviour. The new object
> is still built by CONFIG_TRANSPARENT_HUGEPAGE, so a THP build keeps
> exactly the code it had before; a pure move is also the easiest thing
> to rebase on top of further churn in this area.
>
> 2/3 builds that object for CONFIG_LARGE_FOLIO instead, and drops the
> anon, PMD and khugepaged dependencies a file-only build cannot have.
> The symbol comes from Matthew Wilcox's "[PATCH v9 12/15]"; if that
> series is in first, this patch shrinks to its Kconfig default and the
> accessors it already rewrites.
>
> 3/3 enables filesystem page-cache large folios when THP is off. It
> lifts the compile-time short circuits in pagemap.h for
> mapping_max_folio_size_supported(), mapping_set_folio_order_range(),
> mapping_max_folio_order() and mapping_min_folio_order().
> PREFERRED_MAX_PAGECACHE_ORDER is left alone: an LF-only build keeps
> the existing cap of order 8 instead of following HPAGE_PMD_ORDER.
> Without THP such folios cannot be PMD-mapped, so raising the ceiling
> would mainly increase the maximum page-cache allocation size. The
> changelog has the arm64 64K-page arithmetic, which is a calculation
> and not a measurement.
>
> The split side is not rewritten: truncate, migration and memory-failure
> already call the helpers outside any THP ifdef, so with 2/3 in place
> they simply stop resolving to the -EINVAL stubs once 3/3 turns
> allocation on. madvise is inside a THP ifdef and does not take part.
>
> 3/3 also makes the XFS mount failure above say why it failed: when
> CONFIG_LARGE_FOLIO is not set, the warning now names the missing
> option instead of only reporting the block size as unsupported. With
> LF=y the added warning is compiled out completely.
>
> 3. Testing
>
> Six configurations build clean as full kernels, bzImage and the
> vmlinux link included, with no new warnings: A (THP=y), B (THP=n,
> LF=y), C (B plus HUGETLBFS), D (THP=n, LF=n), and RT-B and RT-C with
> PREEMPT_RT=y. The PREEMPT_RT configurations were build-tested only.
> No runtime PREEMPT_RT stress test or livelock validation was
> performed.
>
> THP=y is unchanged: the .text sections of mm/filemap.o,
> mm/readahead.o and mm/shmem.o are byte-identical before and after the
> series (md5 equal, 45896, 7139 and 35314 bytes). With THP=n and LF=n
> mm/filemap.o and mm/readahead.o stay identical (44584 and 6259 bytes)
> and every function in mm/shmem.o keeps its size; the only difference
> there is one dead argument load that the folded-out
> mapping_set_large_folios() call leaves behind, so the section keeps
> its 29778 bytes while its md5 does not match.
>
> In the new LF=y, THP=n configuration mm/shmem.o keeps its size as well,
> 48 bytes shorter at 29730. That difference is not from 3/3: 40 of the
> bytes are shmem_writeout(), which shrinks because 2/3 turns
> split_folio_to_list() from an inline -EINVAL stub into a real call, and
> the other 8 are the padding in front of the following function; the
> same 29730 bytes appear with 2/3 alone, and 3/3's own shmem hunk
> changes no function size. The new call is unreachable in this
> configuration anyway, since shmem_alloc_and_add_folio() still forces
> order 0 without THP.
>
> Where the generated code really changes is the allocation policy:
> mm/readahead.o grows by 880 bytes of .text (6259 to 7139) and
> mm/filemap.o by 134 (44584 to 44718). nm -u mm/folio_split.o lists no
> anon, pmd or khugepaged symbols.
>
> Runtime, x86_64, 4K pages, THP=n, QEMU/KVM, on a 64K-block XFS image
> (mkfs.xfs -b size=64k -s size=4k), with kprobes on
> filemap_alloc_folio, filemap_add_folio and __folio_split:
>
> - writing 64M: 84 allocations - 59 at order-8, 5 at order-7, 5 at
> order-6, 5 at order-5, 10 at order-4, i.e. exactly 64M;
> - reading the same 64M back after drop_caches: 72 allocations,
> 62 at order-8, 1 at order-7, 5 at order-6, 4 at order-4;
> - truncating 1M to 700K: one __folio_split, landing at order-4,
> which is the min_folio_order XFS asked for; order-0 never appears
> in any of the three phases;
> - tmpfs writing 32M: 8192 shmem_alloc_folio calls, all order-0;
> - shmem write vs F_SEAL_GROW, reproducible as: memfd_create(),
> ftruncate() to exactly 4096 bytes, F_ADD_SEALS with F_SEAL_GROW,
> then a single pwrite() of 8192 bytes at offset 0, comparing the
> syscall's return value. It returns 4096, the same as the
> pre-series THP=n kernel. A THP=y kernel returns -EPERM here, and
> so did the LF-only build before shmem's mapping was put back
> behind THP; that -EPERM is pre-existing behaviour of THP kernels,
> this series neither introduces it nor extends it to other
> configurations.
>
> Same image on the pre-series kernel, for reference: it fails to mount
> with -ENOSYS before any of the above can run. The whole probe was
> re-run on the final revision of the series with identical numbers.
>
> 4. What this turns on for existing filesystems
>
> LF=y makes the existing opt-in take effect for every filesystem that
> already asks for large folios, and that list is not ext4 only: ext4,
> xfs, btrfs, f2fs, fuse, afs, erofs, nfs, cifs and zonefs all call
> mapping_set_folio_order_range() or mapping_set_large_folios() from
> their inode or superblock setup, unconditionally, today. Those paths
> run on any THP=y kernel; the new configuration is only THP=n with
> LF=y. With THP off the ceiling is PREFERRED_MAX_PAGECACHE_ORDER,
> which stays at order 8.
>
> On the tested 4K-page configuration, 4K-block ext4 gets min order 0
> and max order 8, while 64K-block XFS gets min order 4. Btrfs derives
> both ends from the mkfs sectorsize and nodesize. A THP=y kernel keeps
> the order-9 ceiling (HPAGE_PMD_ORDER) it has today on 4K pages.
>
> shmem keeps its THP dependency: its mapping is only marked large-folio
> capable when THP is enabled, because shmem_alloc_and_add_folio() still
> forces orders to zero without it. This matters beyond cosmetics.
> generic_perform_write() sizes its write chunk from
> mapping_max_folio_size() and passes that length to ->write_begin()
> before clamping it to folio_size(), and shmem_write_begin() rejects
> pos + len > i_size for F_SEAL_GROW. On a 4K memfd sealed with
> F_SEAL_GROW, an 8K write at offset 0 returns 4096 on the pre-series
> THP=n kernel and -EPERM on a THP=y kernel; advertising order 8 while
> allocating order 0 reproduced the -EPERM on an LF-only build. Tying
> shmem's policy to LARGE_FOLIO means the allocation, swap and split
> sides together, which is a separate series.
>
> 5. What is not covered
>
> Since PREEMPT_RT is the motivation, the RT side needs to be explicit.
> The freeze-to-xarray-update window in __folio_freeze_split_file() is
> the fourth freeze site named by Matthew Wilcox in the RT livelock
> thread started by Dong Hui
> (<c2ff0c66422c4e2d869499a1186ff0ae@xiaomi.com>):
>
> There's also __folio_freeze_split_file(). I know PREEMPT_RT
> disables TRANSPARENT_HUGEPAGE at the moment, but hopefully that
> restriction will be lifted.
>
> This series does not fix that problem and does not claim to be
> RT-safe. It only makes a code path that already exists on THP kernels
> reachable in a configuration that has no THP, which is the same
> exposure Kairui described in 06/17 as "should be fixed generically".
> One detail that may matter if a reader-side fix is chosen: the
> function already holds the xarray lock until the folio is no longer
> frozen and visible, since folio_ref_unfreeze() happens before
> xas_unlock_irq(), so a generic fix of that shape would cover this path
> without touching it.
>
> Also not tested here: the migration, memory-failure and madvise split
> entries at runtime (only truncate was exercised end to end), arm64
> with 64K pages, nommu, KASAN and lockdep, xfstests, and the btrfs
> order range.
>
> 6. Questions
>
> (a) Non-THP split accounting. count_mthp_stat() is a no-op without
> THP, so in an LF-only build a file split updates no counter at all and
> THP_SPLIT_PAGE does not even appear in /proc/vmstat. I have a
> follow-up patch that gives the file path its own counter. Would you
> rather see a new vmstat item, an existing one reused, or nothing?
>
> (b) Should XFS select LARGE_FOLIO itself? Three lines of
> "select LARGE_FOLIO if MMU" in fs/xfs/Kconfig make 64K-block XFS
> mountable on a THP=n kernel with no manual configuration; I built such
> a kernel and mounted the 64K image. The cost is mm/folio_split.o: 3.2K
> of .text in a THP=n build, code that every THP=y build already carries,
> paid also by XFS builds that will never use a block size above the page
> size. Kconfig cannot express "only when a 64K filesystem is mounted".
> I left it out because it changes default coverage for a decision XFS
> maintainers should make, but I can add it as 4/4 if that is preferred.
>
> Comments welcome.
>
> Fang Xieyan (3):
> mm: move folio split code into mm/folio_split.c
> mm: build file-backed folio splitting with CONFIG_LARGE_FOLIO
> mm: enable filesystem page-cache large folios without THP
>
> MAINTAINERS | 1 +
> fs/xfs/xfs_super.c | 2 +
> include/linux/huge_mm.h | 60 ++-
> include/linux/page-flags.h | 5 +-
> include/linux/pagemap.h | 17 +-
> mm/Kconfig | 19 +
> mm/Makefile | 1 +
> mm/folio_split.c | 969 +++++++++++++++++++++++++++++++++++++
> mm/huge_memory.c | 920 -----------------------------------
> mm/internal.h | 1 +
> mm/shmem.c | 11 +-
> 11 files changed, 1054 insertions(+), 952 deletions(-)
> create mode 100644 mm/folio_split.c
>
>
> base-commit: b024a51b0184b1efb6f55b11aaf7b15a490a6307
--
Best Regards,
Yan, Zi
^ permalink raw reply [flat|nested] 7+ messages in thread