[PATCH v4 00/13] mm/collapse: separate a collapse from its callers

[PATCH v4 00/13] mm/collapse: separate a collapse from its callers

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

[ This is the first of the cleanups I said I would front-load ]

There is no line between the collapse engine and the callers that ask for
a collapse.  khugepaged.c holds both, and they reach into each other.

 - Sixteen tests through the collapse path read cc->is_khugepaged to work
   out what they are allowed to do, when every one of those decisions was
   made by the caller before it asked.

 - collapse_single_pmd() does both halves of a collapse behind one call and
   drops mmap_lock somewhere in the middle.  Which of its paths dropped it
   is not something a caller can see, so it hands back a bool and the
   caller keeps track.

 - MADV_COLLAPSE's implementation -- the walk over the user's range, the
   per-PMD loop, the errno translation -- sits in khugepaged.c, which is
   the daemon's file.

So: draw the line.  State what a caller allows in a policy, split the call
in two with the lock as the boundary, and move the syscall to madvise.c.
What the engine offers is then three calls, with the lock state written
down against each, and a policy the caller fills for itself:

    collapse_control_init(cc)         once, before the first table
    collapse_policy_*(&cc->policy)    what this caller allows
    collapse_scan_pmd(vma, addr, ...) per table, under mmap_lock
    collapse_run_pmd(mm, addr, ...)   when a scan found work, no mmap_lock

The engine stays in khugepaged.c for now; what changes is that it has an
interface, and that neither half has to ask about the other.  madvise.c
gains the operation it should have had all along.

Changes since v3
================

  https://lore.kernel.org/all/20260916093145.4022188-1-kirill@shutemov.name/

 - Patch 5: the PTE limits come as two sets, struct collapse_limits for a
   PMD-sized window and one for anything smaller, and the strict_sub_pmd
   flag goes.  The warning for a max_ptes_none between 0 and the maximum
   moves to where khugepaged fills its policy.  The fields only one side
   reads are prefixed anon_ or file_.  collapse_policy_forced() is
   collapse_policy_madvise() (David).

 - Patch 8 is new: collapse_file() is called from collapse_single_pmd()
   rather than from inside collapse_scan_file(), so the file scan stops at
   the decision before the scan/run split.  The writeback retry and the
   tracepoint change move into it, so patch 9 is only the split (David).

 - Patch 9: no collapse_control_release().  It had nothing to release in
   this series; it comes back with the engine, where init allocates.  The
   check that the last scan was run stays at the top of collapse_scan_pmd()
   (David).

 - Patch 10: the changelog says why the lock boundary belongs to the
   caller (David).

 - Patch 11: both callers pass their TVA type as the literal it is rather
   than reading it back from the policy (David).

 - Patch 12: kerneldoc on every function collapse.h declares (David).

 - Acked-by from David Hildenbrand on 1-4, 6 and 7; Reviewed-by from
   Baolin Wang on 10.  Reviewed-by dropped from 5 and 9, which changed.

Changes since v2
================

  https://lore.kernel.org/all/20260910120238.2529819-1-kirill@shutemov.name/

 - Patch 8: the file scan returns SCAN_PTE_MAPPED_HUGEPAGE as it is and
   the run is handed what the scan returned, so it goes straight to
   retracting the PTE table when it sees it.  The scan_retract_only flag
   and the result round trip go, and the two copies of the file put become
   one helper (Zi).  Patches 9-12 follow the new signature.

 - Patch 8: mthp_collapse() and collapse_huge_page() read the orders and
   the referenced and swapped-out counts from collapse_control instead of
   taking them as arguments (Baolin).

 - Patch 11: each interface function is documented where it is defined,
   with the lock state on entry and exit.  The overview in collapse.h
   stays (Zi).

 - Reviewed-by from Zi Yan on 8 and 9, and from Zi Yan and Baolin Wang
   on 5 and 10.

Changes since v1
================

  https://lore.kernel.org/all/cover.1788533997.git.kas@kernel.org/

 - Rebased onto mm-new with Vernon Yang's tracepoint fixes in it.  Patch 8
   no longer merges the two calls to each scan tracepoint, since the base
   already has one; its changelog now says what the status field reports.

 - Patch 3: nr_occupied_ptes is nr_eligible_ptes, and the mthp_collapse()
   comment counts eligible PTEs too (Zi, Baolin).

 - Patch 4: no comments on the two constants (Baolin).

 - Patch 5: one line per policy field (Baolin).

 - Patch 8: the file side is split like the anonymous one (Zi).
   collapse_scan_file() runs under mmap_lock in the scan and only reads;
   collapse_file() runs in the run.  See Behaviour below.

 - Reviewed-by from Zi Yan and Baolin Wang on 1-4, 6 and 7.

Patches
=======

The first three stand alone and can be taken separately:

  1  drop the mmgrab() MADV_COLLAPSE has held since 7d8faaf15545
  2  count collapses in khugepaged's own walk, where the daemon's
     bookkeeping belongs
  3  rename cc->mthp_present_ptes to eligible_ptes, which is what a set
     bit means

Then the interface, in order:

  4  add collapse.h, and move enum scan_result, struct collapse_control
     and the two constants into it
  5  struct collapse_policy, filled by the caller; the is_khugepaged
     tests become field reads, and the flag goes
  6  drop collapse_possible(), a wrapper that only turns a mask into a bool
  7  collapse_control_init_scan() is a per-table reset, so name it
     collapse_scan_reset()
  8  call collapse_file() from collapse_single_pmd(), so the file scan
     stops at the decision
  9  give the scan and the collapse a function each: collapse_scan_pmd()
     and collapse_run_pmd()
 10  open-code the entry point that joined them, so each caller owns the
     lock across the boundary and the bool goes
 11  work out the orders a VMA allows once per VMA, not once per table
 12  declare the three calls in collapse.h, with the lock rules
 13  MADV_COLLAPSE moves to madvise.c

Behaviour
=========

No functional change is intended.  Nothing here changes which tables get
collapsed, into what, or what MADV_COLLAPSE returns.  The tracepoints are
the one place a change can be seen from outside; the rest is where work
happens, not what it does.

 - Patches 8 and 9: mm_khugepaged_scan_file and mm_khugepaged_scan_pmd
   fire before the collapse rather than after it.  For an accepted table
   their status field reads SCAN_SUCCEED, where it used to carry what the
   collapse made of the table; that is now for mm_khugepaged_collapse_file
   and mm_collapse_huge_page to report.

Four things move that a reader should not have to find in the diff:

 - Patch 5: khugepaged fills its policy once per scan pass, so the
   max_ptes_* limits and the defrag setting behind the allocation mask are
   sampled once per pass rather than once per table.  A knob written
   mid-pass takes effect on the next pass instead of the next table.

 - Patch 8: the writeback retry re-runs collapse_file() alone, where it
   used to rescan the page cache first.

 - Patch 9: the file scan runs under mmap_lock, where before the lock was
   given up first.  A file table the scan refuses no longer costs
   khugepaged an unlock, a trip back through khugepaged_do_scan(), a
   relock and a VMA lookup; only a table it goes on to collapse does.

 - Patch 11: which orders a table is scanned for is sampled once per VMA
   rather than once per table.  It cannot widen what a collapse does; the
   order is tested again under the lock the collapse retakes.

For an anonymous table, and for a file table that gets collapsed, the lock
is given up and taken again at exactly the points it was before; the only
difference is that the caller is the one doing it.

selftests/mm khugepaged passes on x86-64 with a KASAN, lockdep and
DEBUG_VM config, in the five forms run_vmtests.sh runs it.  Every patch
builds, CONFIG_TRANSPARENT_HUGEPAGE=n included.

Kiryl Shutsemau (Meta) (13):
  mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()
  mm/khugepaged: count collapses where khugepaged makes them
  mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes
  mm/collapse: add collapse.h for the collapse interface
  mm/collapse: state what a collapse may do in the policy
  mm/collapse: drop the collapse_possible() wrapper
  mm/collapse: name the per-table scan reset for what it resets
  mm/collapse: call collapse_file() from collapse_single_pmd()
  mm/collapse: separate scanning a PTE table from collapsing it
  mm/collapse: open-code collapse_single_pmd() in its two callers
  mm/collapse: work out the orders a VMA allows once per VMA
  mm/collapse: declare the collapse interface in collapse.h
  mm/collapse: implement MADV_COLLAPSE in madvise.c

 MAINTAINERS             |   1 +
 include/linux/huge_mm.h |   9 -
 mm/collapse.h           | 162 +++++++++++
 mm/khugepaged.c         | 620 +++++++++++++++++-----------------------
 mm/madvise.c            | 170 ++++++++++-
 5 files changed, 590 insertions(+), 372 deletions(-)
 create mode 100644 mm/collapse.h


Range-diff against v3
=====================

 1:  c777058f55c5 !  1:  73d87f3454b0 mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()
    @@ Commit message
         Drop the mmgrab()/mmdrop() pair.
     
         Assisted-by: LLM
    +    Acked-by: David Hildenbrand (Arm) <david@kernel.org>
         Reviewed-by: Zi Yan <ziy@nvidia.com>
         Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
 2:  45e08a1d7aca !  2:  095b43905bd6 mm/khugepaged: count collapses where khugepaged makes them
    @@ Commit message
         its own counter without the shared path testing who called.
     
         Assisted-by: LLM
    +    Acked-by: David Hildenbrand (Arm) <david@kernel.org>
         Reviewed-by: Zi Yan <ziy@nvidia.com>
         Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
 3:  8caa0e047559 !  3:  66802b5676a3 mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes
    @@ Commit message
         No functional change.
     
         Assisted-by: LLM
    +    Acked-by: David Hildenbrand (Arm) <david@kernel.org>
         Reviewed-by: Zi Yan <ziy@nvidia.com>
         Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
 4:  b7a8d6ebd4ce !  4:  22f00b9a706d mm/collapse: add collapse.h for the collapse interface
    @@ Commit message
         No functional change.
     
         Assisted-by: LLM
    +    Acked-by: David Hildenbrand (Arm) <david@kernel.org>
         Reviewed-by: Zi Yan <ziy@nvidia.com>
         Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
 5:  c6441dd38fe2 !  5:  4b9e43984bc9 mm/collapse: state what a collapse may do in the policy
    @@ Commit message
         Every test becomes a read of a field, and cc->is_khugepaged goes, having
         no reader left.
     
    +    The PTE limits come as two sets, one for a PMD-sized window and one for
    +    anything smaller, so that the helpers pick a set for the order and read
    +    it.  khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
    +    and empty PTEs only when the knob says all or nothing; the warning for a
    +    knob value in between moves to where khugepaged fills its policy.  The
    +    fields only one side reads say which: anon_ or file_.  David Hildenbrand
    +    asked for both.
    +
         khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
         call.  That is the one change in behaviour.  The max_ptes_* limits and the
         defrag setting behind the allocation mask are sampled once per pass rather
    @@ Commit message
         cc unconditionally, so the check was already dead.
     
         Assisted-by: LLM
    -    Reviewed-by: Zi Yan <ziy@nvidia.com>
    -    Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
      ## mm/collapse.h ##
    @@ mm/collapse.h: enum scan_result {
      	SCAN_PAGE_DIRTY_OR_WRITEBACK,
      };
      
    -+/* What a collapse is allowed to do, decided by the caller that asks for it */
    -+struct collapse_policy {
    -+	/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */
    ++/* How many PTEs of a window may be missing, swapped out or shared */
    ++struct collapse_limits {
    ++	/* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */
     +	unsigned int max_ptes_none;
     +	unsigned int max_ptes_swap;
     +	unsigned int max_ptes_shared;
    ++};
     +
    -+	/* Take no swapped-out or shared PTE into a sub-PMD collapse */
    -+	bool strict_sub_pmd;
    ++/* What a collapse is allowed to do, decided by the caller that asks for it */
    ++struct collapse_policy {
    ++	/* Limits for a PMD-sized window */
    ++	struct collapse_limits pmd;
    ++
    ++	/*
    ++	 * Limits for a smaller window.  Its max_ptes_none is either 0 or
    ++	 * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the
    ++	 * window whatever its order; any other value counts as 0.
    ++	 */
    ++	struct collapse_limits sub_pmd;
     +
     +	/* Leave clean lazyfree folios to reclaim rather than collapse them */
    -+	bool skip_lazyfree;
    ++	bool anon_skip_lazyfree;
     +
    -+	/* Refuse a range with no sign of use */
    -+	bool require_referenced;
    ++	/* Refuse an anonymous range with no sign of use */
    ++	bool anon_require_referenced;
     +
     +	/* Map the PMD over a file collapse instead of leaving it to a fault */
    -+	bool install_pmd;
    ++	bool file_install_pmd;
     +
     +	/* Write dirty pages back and retry once instead of refusing them */
    -+	bool writeback_dirty;
    ++	bool file_writeback_dirty;
     +
     +	/* How hard to try for a destination folio */
     +	gfp_t gfp;
    @@ mm/khugepaged.c: static bool pte_none_or_zero(pte_t pte)
      		struct vm_area_struct *vma, unsigned int order)
      {
     -	const unsigned int max_ptes_none = khugepaged_max_ptes_none;
    -+	const unsigned int max_ptes_none = cc->policy.max_ptes_none;
    ++	unsigned int max_ptes_none;
      
      	if (vma && userfaultfd_armed(vma))
      		return 0;
    @@ mm/khugepaged.c: static bool pte_none_or_zero(pte_t pte)
     -	if (!cc->is_khugepaged)
     -		return HPAGE_PMD_NR;
     -	/* for PMD collapse, respect the user defined maximum */
    --	if (is_pmd_order(order))
    -+	/* The limit as given, at the PMD order and wherever it is not capped */
    -+	if (is_pmd_order(order) || !cc->policy.strict_sub_pmd)
    - 		return max_ptes_none;
    - 	/*
    - 	 * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT,
    -@@ mm/khugepaged.c: static unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
    + 	if (is_pmd_order(order))
    +-		return max_ptes_none;
    +-	/*
    +-	 * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT,
    +-	 * scale the maximum number of PTEs to the order of the collapse.
    +-	 */
    ++		return cc->policy.pmd.max_ptes_none;
    ++
    ++	/* Below PMD order: all but one PTE of the window, or none */
    ++	max_ptes_none = cc->policy.sub_pmd.max_ptes_none;
    + 	if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT)
    + 		return (1 << order) - 1;
    +-	/*
    +-	 * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT,
    +-	 * emit a warning and return 0.
    +-	 */
    +-	if (max_ptes_none)
    +-		pr_warn_once("mTHP collapse does not support max_ptes_none"
    +-		     " values other than 0 or %u, defaulting to 0.\n",
    +-		     COLLAPSE_MAX_PTES_LIMIT);
    + 	return 0;
    + }
    + 
    +@@ mm/khugepaged.c: static unsigned int collapse_max_ptes_none(struct collapse_control *cc,
    + static unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
      		unsigned int order)
      {
    - 	/*
    +-	/*
     -	 * For MADV_COLLAPSE, do not restrict the number of PTEs that map shared
     -	 * anonymous pages.
    -+	 * A sub-PMD window held to the strict rule takes no shared page at all:
    -+	 * an mTHP is not worth the CoW-breaking.
    - 	 */
    +-	 */
     -	if (!cc->is_khugepaged)
     -		return HPAGE_PMD_NR;
     -	/*
    @@ mm/khugepaged.c: static unsigned int collapse_max_ptes_shared(struct collapse_co
     -	 * are shared between processes.
     -	 */
     -	if (!is_pmd_order(order))
    -+	if (!is_pmd_order(order) && cc->policy.strict_sub_pmd)
    - 		return 0;
    +-		return 0;
     -	/* for PMD collapse, respect the user defined maximum */
     -	return khugepaged_max_ptes_shared;
    -+	return cc->policy.max_ptes_shared;
    ++	if (is_pmd_order(order))
    ++		return cc->policy.pmd.max_ptes_shared;
    ++	return cc->policy.sub_pmd.max_ptes_shared;
      }
      
      /**
    -@@ mm/khugepaged.c: static unsigned int collapse_max_ptes_swap(struct collapse_control *cc,
    +@@ mm/khugepaged.c: static unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
    + static unsigned int collapse_max_ptes_swap(struct collapse_control *cc,
      		unsigned int order)
      {
    - 	/*
    +-	/*
     -	 * For MADV_COLLAPSE, do not restrict the number PTEs entries or
     -	 * pagecache entries that are non-present.
    -+	 * A sub-PMD window held to the strict rule takes nothing non-present:
    -+	 * reading pages back to build an mTHP is not worth the latency.
    - 	 */
    +-	 */
     -	if (!cc->is_khugepaged)
     -		return HPAGE_PMD_NR;
     -	/* for mTHP collapse do not allow any non-present PTEs or pagecache entries */
     -	if (!is_pmd_order(order))
    -+	if (!is_pmd_order(order) && cc->policy.strict_sub_pmd)
    - 		return 0;
    +-		return 0;
     -	/* for PMD collapse, respect the user defined maximum */
     -	return khugepaged_max_ptes_swap;
    -+	return cc->policy.max_ptes_swap;
    ++	if (is_pmd_order(order))
    ++		return cc->policy.pmd.max_ptes_swap;
    ++	return cc->policy.sub_pmd.max_ptes_swap;
      }
      
      int hugepage_madvise(struct vm_area_struct *vma,
    @@ mm/khugepaged.c: static enum scan_result __collapse_huge_page_isolate(struct vm_
      		 * preserve the lazyfree property without needing to skip.
      		 */
     -		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
    -+		if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
    ++		if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
      		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
      			result = SCAN_PAGE_LAZYFREE;
      			goto out;
    @@ mm/khugepaged.c: static enum scan_result __collapse_huge_page_isolate(struct vm_
      			list_add_tail(&folio->lru, compound_pagelist);
      next:
     -		if (cc->is_khugepaged &&
    -+		if (cc->policy.require_referenced &&
    ++		if (cc->policy.anon_require_referenced &&
      		    folio_pte_referenced(folio, vma, addr, pteval))
      			referenced++;
      	}
      
     -	if (unlikely(cc->is_khugepaged && !referenced)) {
    -+	if (unlikely(cc->policy.require_referenced && !referenced)) {
    ++	if (unlikely(cc->policy.anon_require_referenced && !referenced)) {
      		result = SCAN_LACK_REFERENCED_PAGE;
      	} else {
      		result = SCAN_SUCCEED;
    @@ mm/khugepaged.c: static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void)
     +/* khugepaged collapses on its own initiative, so it obeys its own settings */
     +static void collapse_policy_khugepaged(struct collapse_policy *p)
     +{
    -+	p->max_ptes_none = READ_ONCE(khugepaged_max_ptes_none);
    -+	p->max_ptes_swap = READ_ONCE(khugepaged_max_ptes_swap);
    -+	p->max_ptes_shared = READ_ONCE(khugepaged_max_ptes_shared);
    -+	p->strict_sub_pmd = true;
    -+	p->skip_lazyfree = true;
    -+	p->require_referenced = true;
    -+	p->install_pmd = false;
    -+	p->writeback_dirty = false;
    ++	p->pmd.max_ptes_none = READ_ONCE(khugepaged_max_ptes_none);
    ++	p->pmd.max_ptes_swap = READ_ONCE(khugepaged_max_ptes_swap);
    ++	p->pmd.max_ptes_shared = READ_ONCE(khugepaged_max_ptes_shared);
    ++
    ++	/*
    ++	 * A sub-PMD window takes no swapped-out and no shared PTE: reading
    ++	 * pages back or breaking CoW is not worth it for an mTHP.  Empty PTEs
    ++	 * it takes all or nothing, since anything in between would let one
    ++	 * collapse feed the next.
    ++	 */
    ++	p->sub_pmd.max_ptes_none = p->pmd.max_ptes_none;
    ++	if (p->sub_pmd.max_ptes_none &&
    ++	    p->sub_pmd.max_ptes_none != COLLAPSE_MAX_PTES_LIMIT)
    ++		pr_warn_once("mTHP collapse does not support max_ptes_none values other than 0 or %u, defaulting to 0.\n",
    ++			     COLLAPSE_MAX_PTES_LIMIT);
    ++	p->sub_pmd.max_ptes_swap = 0;
    ++	p->sub_pmd.max_ptes_shared = 0;
    ++
    ++	p->anon_skip_lazyfree = true;
    ++	p->anon_require_referenced = true;
    ++	p->file_install_pmd = false;
    ++	p->file_writeback_dirty = false;
     +	p->gfp = alloc_hugepage_khugepaged_gfpmask();
     +	p->tva_type = TVA_KHUGEPAGED;
     +}
     +
     +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */
    -+static void collapse_policy_forced(struct collapse_policy *p)
    ++static void collapse_policy_madvise(struct collapse_policy *p)
     +{
    -+	p->max_ptes_none = HPAGE_PMD_NR;
    -+	p->max_ptes_swap = HPAGE_PMD_NR;
    -+	p->max_ptes_shared = HPAGE_PMD_NR;
    -+	p->strict_sub_pmd = false;
    -+	p->skip_lazyfree = false;
    -+	p->require_referenced = false;
    -+	p->install_pmd = true;
    -+	p->writeback_dirty = true;
    ++	p->pmd.max_ptes_none = HPAGE_PMD_NR;
    ++	p->pmd.max_ptes_swap = HPAGE_PMD_NR;
    ++	p->pmd.max_ptes_shared = HPAGE_PMD_NR;
    ++	/* Never read: MADV_COLLAPSE collapses to PMD order only */
    ++	p->sub_pmd = p->pmd;
    ++
    ++	p->anon_skip_lazyfree = false;
    ++	p->anon_require_referenced = false;
    ++	p->file_install_pmd = true;
    ++	p->file_writeback_dirty = true;
     +	p->gfp = GFP_TRANSHUGE;
     +	p->tva_type = TVA_FORCED_COLLAPSE;
     +}
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
      		 * preserve the lazyfree property without needing to skip.
      		 */
     -		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
    -+		if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
    ++		if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
      		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
      			result = SCAN_PAGE_LAZYFREE;
      			failed_pfn = folio_pfn(folio);
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
      		}
      
     -		if (cc->is_khugepaged &&
    -+		if (cc->policy.require_referenced &&
    ++		if (cc->policy.anon_require_referenced &&
      		    folio_pte_referenced(folio, vma, addr, pteval))
      			referenced++;
      	}
     -	if (cc->is_khugepaged &&
     -		   (!referenced ||
     -		    (unmapped && referenced < HPAGE_PMD_NR / 2))) {
    -+	if (cc->policy.require_referenced &&
    ++	if (cc->policy.anon_require_referenced &&
     +	    (!referenced ||
     +	     (unmapped && referenced < HPAGE_PMD_NR / 2))) {
      		result = SCAN_LACK_REFERENCED_PAGE;
    @@ mm/khugepaged.c: static enum scan_result collapse_file(struct mm_struct *mm, uns
      	 */
      	retract_page_tables(mapping, start);
     -	if (cc && !cc->is_khugepaged)
    -+	if (cc->policy.install_pmd)
    ++	if (cc->policy.file_install_pmd)
      		result = SCAN_PTE_MAPPED_HUGEPAGE;
      	folio_unlock(new_folio);
      
    @@ mm/khugepaged.c: static enum scan_result collapse_single_pmd(unsigned long addr,
     -	 */
     -	if (!cc->is_khugepaged && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
     +	/* Dirty pages are worth a writeback and one more try, if asked for */
    -+	if (cc->policy.writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
    ++	if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
      	    !triggered_wb && mapping_can_writeback(file->f_mapping)) {
      		const loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
      		const loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
    @@ mm/khugepaged.c: static enum scan_result collapse_single_pmd(unsigned long addr,
      		else
      			result = try_collapse_pte_mapped_thp(mm, addr,
     -							     !cc->is_khugepaged);
    -+							cc->policy.install_pmd);
    ++							cc->policy.file_install_pmd);
      		if (result == SCAN_PMD_MAPPED)
      			result = SCAN_SUCCEED;
      		mmap_read_unlock(mm);
    @@ mm/khugepaged.c: static void khugepaged_do_scan(struct collapse_control *cc)
      
      	lru_add_drain_all();
      
    -+	/* One policy for the whole pass, so every table is treated the same */
     +	collapse_policy_khugepaged(&cc->policy);
     +
      	cc->progress = 0;
    @@ mm/khugepaged.c: int madvise_collapse(struct vm_area_struct *vma, unsigned long
      	if (!cc)
      		return -ENOMEM;
     -	cc->is_khugepaged = false;
    -+	collapse_policy_forced(&cc->policy);
    ++	collapse_policy_madvise(&cc->policy);
      	cc->progress = 0;
      
      	lru_add_drain_all();
 6:  301d92c2d728 !  6:  9ae83d07f1e0 mm/collapse: drop the collapse_possible() wrapper
    @@ Commit message
         No functional change.
     
         Assisted-by: LLM
    +    Acked-by: David Hildenbrand (Arm) <david@kernel.org>
         Reviewed-by: Zi Yan <ziy@nvidia.com>
         Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
 7:  863f4aec4750 !  7:  bfdf63c56c7e mm/collapse: name the per-table scan reset for what it resets
    @@ Commit message
         No functional change.
     
         Assisted-by: LLM
    +    Acked-by: David Hildenbrand (Arm) <david@kernel.org>
         Reviewed-by: Zi Yan <ziy@nvidia.com>
         Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
 -:  ------------ >  8:  9b8038917bc5 mm/collapse: call collapse_file() from collapse_single_pmd()
 8:  d0e612260e80 !  9:  dd6d94fdad05 mm/collapse: separate scanning a PTE table from collapsing it
    @@ Commit message
     
         Give each half its own function:
     
    -      - collapse_scan_pmd() scans one table and only reads.  The anonymous
    -        scan that used to carry that name keeps its body as
    -        collapse_scan_anon_pmd(), and collapse_scan_pmd() is now the entry
    -        that picks the anonymous or the file side.
    +      - collapse_scan_pmd() scans one table.  The anonymous scan that used to
    +        carry that name keeps its body as collapse_scan_anon_pmd(), and
    +        collapse_scan_pmd() is now the entry that picks the anonymous or the
    +        file side.
     
           - collapse_run_pmd() does the collapse the scan asked for, and is
             handed what the scan returned.  SCAN_SUCCEED means there is
    @@ Commit message
             nothing to do.
     
         collapse_single_pmd() is now the two of them with the mmap_lock drop in
    -    between, so its callers see what they saw before.
    +    between, so its callers see what they saw before.  collapse_control_init()
    +    sets a control up before its first scan.
     
         What the scan found and the run needs travels in collapse_control.  For
         an anonymous table that is the orders and the referenced and swapped-out
         counts, which mthp_collapse() and collapse_huge_page() now read from
         there instead of taking as arguments.  For a file it is the file itself
    -    and the offset in it.
    +    and the offset in it: a file collapse works on the page cache and never
    +    sees a VMA, so the scan takes the reference while it still has one and
    +    the run gives it back.
     
    -    The file side moves with the anonymous one.  collapse_scan_file() used to
    -    run with mmap_lock already given up, and called collapse_file() itself
    -    when the page cache looked worth it.  It now runs under the lock like the
    -    anonymous scan and only reads; the run does the collapse.  A file
    -    collapse works on the page cache and never sees a VMA, so the scan takes
    -    the file reference while it still has one and the run gives it back.
    -
    -    That changes what a refused file table costs khugepaged.  Every file
    -    table it scanned used to end its pass over that mm, because the lock had
    -    been dropped to scan it; now only a table it goes on to collapse does.
    -
    -    Two things on the file side stop being rescanned.  When the page cache
    -    already holds the PMD folio, the scan says so and the run goes straight
    -    to retracting the PTE table.  A run that refuses dirty pages and may
    -    write them back retries collapse_file() alone.  The checks the scan makes
    -    ahead of it are ones collapse_file() repeats under the page cache lock.
    -
    -    Tracing changes with it.  mm_khugepaged_scan_pmd and
    -    mm_khugepaged_scan_file used to fire after the collapse, so for an
    -    accepted table their status field carried what the collapse made of it.
    -    They now fire before it and read SCAN_SUCCEED for an accepted table.  What
    -    the collapse then made of it is for mm_collapse_huge_page and
    -    mm_khugepaged_collapse_file to report.
    +    The file scan moves under mmap_lock with the anonymous one, where before
    +    the lock was given up first.  The lock is now held over the page cache
    +    walk, an RCU walk over one table's worth of slots with no PTL, and taken
    +    fewer times.  collapse_scan_mm_slot() ends its walk whenever the lock was
    +    dropped, so a refused file table used to cost khugepaged an unlock, a
    +    trip back through khugepaged_do_scan(), a relock and a VMA lookup.  Now
    +    only a table that goes on to be collapsed does.
     
         Assisted-by: LLM
    -    Reviewed-by: Zi Yan <ziy@nvidia.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
      ## mm/collapse.h ##
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
      out:
      	trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced,
     @@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm,
    - 	else
    - 		cc->progress += HPAGE_PMD_NR;
    - 
    --	if (result == SCAN_SUCCEED) {
    --		if (present < HPAGE_PMD_NR - max_ptes_none) {
    --			result = SCAN_EXCEED_NONE_PTE;
    --			count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
    --		} else {
    --			result = collapse_file(mm, addr, file, start, cc);
    --		}
    -+	if (result == SCAN_SUCCEED && present < HPAGE_PMD_NR - max_ptes_none) {
    -+		result = SCAN_EXCEED_NONE_PTE;
    -+		count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
    + 		count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
      	}
      
     -	trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result);
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
     +	cc->scan_file = NULL;
     +}
     +
    -+/* A scan that took a file reference should have been run */
    -+static void collapse_put_scan_file(struct collapse_control *cc)
    -+{
    -+	if (WARN_ON_ONCE(cc->scan_file)) {
    -+		fput(cc->scan_file);
    -+		cc->scan_file = NULL;
    -+	}
    -+}
    -+
    -+static void collapse_control_release(struct collapse_control *cc)
    -+{
    -+	collapse_put_scan_file(cc);
    -+}
    -+
     +static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
     +		unsigned long addr, struct collapse_control *cc)
      {
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
     -	mmap_assert_locked(mm);
     +	mmap_assert_locked(vma->vm_mm);
     +	/* Whatever the last scan found has to have been run by now */
    -+	collapse_put_scan_file(cc);
    ++	if (WARN_ON_ONCE(cc->scan_file)) {
    ++		fput(cc->scan_file);
    ++		cc->scan_file = NULL;
    ++	}
      
      	if (vma_is_anonymous(vma))
     -		return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
      
     -	file = get_file(vma->vm_file);
      	pgoff = linear_page_index(vma, addr);
    -+	result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc);
    -+	/*
    -+	 * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds
    -+	 * the PMD folio, and retracting the PTE table is the run's job.
    -+	 */
    -+	if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
    -+		return result;
    - 
    +-
     -	mmap_read_unlock(mm);
     -	*lock_dropped = true;
    +-
    ++	result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc);
    + 	/*
    + 	 * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds
    +-	 * the PMD folio, and only the PTE table is left to retract.
    ++	 * the PMD folio, and retracting the PTE table is the run's job.
    + 	 */
    +-	result = collapse_scan_file(mm, addr, file, pgoff, cc);
    +-	if (result != SCAN_SUCCEED)
    ++	if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
    ++		return result;
    ++
     +	/*
     +	 * A file collapse works on the page cache and never sees a VMA, so take
     +	 * what it needs from this one while it is still here.
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
     +
     +	/* The scan found the PMD folio in place: nothing to collapse */
     +	if (result == SCAN_PTE_MAPPED_HUGEPAGE)
    -+		goto retract;
    + 		goto put;
      retry:
    --	result = collapse_scan_file(mm, addr, file, pgoff, cc);
    -+	result = collapse_file(mm, addr, file, pgoff, cc);
    - 
    - 	/* Dirty pages are worth a writeback and one more try, if asked for */
    - 	if (cc->policy.writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
    + 	result = collapse_file(mm, addr, file, pgoff, cc);
     @@ mm/khugepaged.c: static enum scan_result collapse_single_pmd(unsigned long addr,
    - 		triggered_wb = true;
    - 		goto retry;
    - 	}
    -+retract:
    + put:
      	fput(file);
      
     +	/*
    @@ mm/khugepaged.c: static void khugepaged_do_scan(struct collapse_control *cc)
      	lru_add_drain_all();
      
     +	collapse_control_init(cc);
    - 	/* One policy for the whole pass, so every table is treated the same */
      	collapse_policy_khugepaged(&cc->policy);
      
     -	cc->progress = 0;
      	while (true) {
      		cond_resched();
      
    -@@ mm/khugepaged.c: static void khugepaged_do_scan(struct collapse_control *cc)
    - 			khugepaged_alloc_sleep();
    - 		}
    - 	}
    -+
    -+	collapse_control_release(cc);
    - }
    - 
    - static bool khugepaged_should_wakeup(void)
     @@ mm/khugepaged.c: int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
      	cc = kmalloc_obj(*cc);
      	if (!cc)
      		return -ENOMEM;
     +	collapse_control_init(cc);
    - 	collapse_policy_forced(&cc->policy);
    + 	collapse_policy_madvise(&cc->policy);
     -	cc->progress = 0;
      
      	lru_add_drain_all();
      
    -@@ mm/khugepaged.c: int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
    - 	}
    - out_nolock:
    - 	mmap_assert_locked(mm);
    -+	collapse_control_release(cc);
    - 	kfree(cc);
    - 
    - 	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
 9:  ab4013d31d71 ! 10:  97d9ce4aa4a5 mm/collapse: open-code collapse_single_pmd() in its two callers
    @@ Metadata
      ## Commit message ##
         mm/collapse: open-code collapse_single_pmd() in its two callers
     
    -    collapse_scan_pmd() and collapse_run_pmd() each have a clear locking
    -    contract.  The scan is called with mmap_lock held for reading and returns
    -    with it still held.  The collapse is called without it.
    +    A scan and a collapse want different things from mmap_lock.  The scan
    +    reads one PTE table under the lock the caller holds, refuses most of the
    +    time, and the caller moves on to the next table without letting go.  The
    +    collapse allocates, may sleep in writeback and takes the lock for write
    +    itself, so the lock it is handed is of no use to it.
     
         collapse_single_pmd() kept that boundary inside itself.  It dropped the
    -    lock on some paths and not others, and reported which by way of a bool its
    -    callers had to carry along and then act on.
    +    lock on some paths and not others, and reported which by way of a bool
    +    its callers had to carry along and then act on.  Both callers already
    +    act on a drop, khugepaged by ending its walk and madvise_collapse() by
    +    looking its VMA up again.  The code that has to know is not the code
    +    that does it.
     
         Open-code it in the two callers.  Each scans under the lock it already
    -    holds and, when the scan found work, gives the lock up before running the
    -    collapse.
    -    khugepaged's lock_dropped and madvise_collapse()'s mmap_unlocked both go:
    -    the code dropping the lock is now the code that wanted to know.
    +    holds and, when the scan found work, gives the lock up before running
    +    the collapse.  The scan then has one rule, called locked and returning
    +    locked, and the run another, called unlocked.  Nothing is left to
    +    report, so khugepaged's lock_dropped and madvise_collapse()'s
    +    mmap_unlocked both go.  The engine never touches a lock it did not
    +    take, and how a caller locks its scan is the caller's business alone.
     
         khugepaged's walk carries on to the next table while the scan keeps
         refusing, and ends once a collapse has taken the lock from under it.
    -    madvise_collapse() re-finds its VMA after a collapse, which it did before,
    -    and now uses a NULL vma to say that it has to.  It still reports the drop
    -    to its own caller, from the line that does it.
    +    madvise_collapse() re-finds its VMA after a collapse, which it did
    +    before, and now uses a NULL vma to say that it has to.  It still reports
    +    the drop to its own caller, from the line that does it.
    +
    +    Preparation for moving madvise_collapse() out of khugepaged.c: what it
    +    needs from the engine is then two calls with one lock rule each.
     
         The lock is given up and taken again at the same points as before.  No
         functional change.
     
         Assisted-by: LLM
         Reviewed-by: Zi Yan <ziy@nvidia.com>
    +    Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
      ## mm/khugepaged.c ##
    @@ mm/khugepaged.c: int madvise_collapse(struct vm_area_struct *vma, unsigned long
     -out_nolock:
     +out_locked:
      	mmap_assert_locked(mm);
    - 	collapse_control_release(cc);
      	kfree(cc);
    + 
10:  2782efd229ae ! 11:  3e43fd51b87d mm/collapse: work out the orders a VMA allows once per VMA
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_anon_pmd(struct vm_area_s
      	/*
      	 * If PMD is the only enabled order, enforce max_ptes_none, otherwise
      	 * scan all pages to populate the bitmap for mTHP collapse. The bitmap
    -@@ mm/khugepaged.c: static void collapse_control_release(struct collapse_control *cc)
    +@@ mm/khugepaged.c: static void collapse_control_init(struct collapse_control *cc)
      }
      
      static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
    @@ mm/khugepaged.c: static void collapse_control_release(struct collapse_control *c
      	enum scan_result result;
      	pgoff_t pgoff;
     @@ mm/khugepaged.c: static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
    - 	collapse_put_scan_file(cc);
    + 	}
      
      	if (vma_is_anonymous(vma))
     -		return collapse_scan_anon_pmd(vma, addr, cc);
    @@ mm/khugepaged.c: static void collapse_scan_mm_slot(unsigned int progress_max,
     -					      TVA_KHUGEPAGED)) {
     +		/* One mask for the whole VMA */
     +		orders = collapse_possible_orders(vma, vma->vm_flags,
    -+						  cc->policy.tva_type);
    ++						  TVA_KHUGEPAGED);
     +		if (!orders) {
      			cc->progress++;
      			continue;
    @@ mm/khugepaged.c: int madvise_collapse(struct vm_area_struct *vma, unsigned long
      			vma = found;
      			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
     +			orders = collapse_possible_orders(vma, vma->vm_flags,
    -+							  cc->policy.tva_type);
    ++							  TVA_FORCED_COLLAPSE);
      		}
      
     -		result = collapse_scan_pmd(vma, addr, cc);
11:  7dc745fd037d ! 12:  358bba63f1ab mm/collapse: declare the collapse interface in collapse.h
    @@ Metadata
      ## Commit message ##
         mm/collapse: declare the collapse interface in collapse.h
     
    -    A collapse takes four calls:
    +    A collapse takes three calls:
     
           - collapse_control_init() - set up the control a caller carries;
           - collapse_scan_pmd() - scan one PTE table, under mmap_lock;
    -      - collapse_run_pmd() - collapse what the scan found, no mmap_lock;
    -      - collapse_control_release() - done with the control.
    +      - collapse_run_pmd() - collapse what the scan found, no mmap_lock.
     
    -    All four are static in khugepaged.c, as are collapse_possible_orders(),
    +    All three are static in khugepaged.c, as are collapse_possible_orders(),
         which says what a VMA allows, and the revalidate a caller needs once a
         collapse has given the mmap_lock up.  No other file can ask for a collapse
         without them.
     
         Declare them in collapse.h, with a comment stating the order they are
    -    called in and who holds the lock over each step.  Each function says
    -    what it needs and what it does where it is defined.
    +    called in and who holds the lock over each step.  Each function gets a
    +    kerneldoc comment where it is defined: what it takes, what it does, and
    +    the lock state on entry and exit.
     
         hugepage_vma_revalidate() becomes collapse_vma_revalidate(): it is part of
         what a collapse offers now, not a helper of the daemon.
    @@ mm/collapse.h: struct collapse_control {
     + *     collapse_control_init(cc)              once, before the first table
     + *     collapse_scan_pmd(vma, addr, ...)      per table
     + *     collapse_run_pmd(mm, addr, result, cc) when a scan found work
    -+ *     collapse_control_release(cc)           once, when done with the control
     + *
     + * The caller holds mmap_lock for reading over the scan and passes an address
     + * within @vma, aligned to the PTE table to scan.
     + *
    -+ * The scan returns with that lock still held.  It only reads, and almost every
    -+ * table it is offered has nothing to collapse, so a caller walks a whole VMA
    -+ * under the one lock it took to get there.  SCAN_SUCCEED means there is
    ++ * The scan returns with that lock still held.  Almost every table it is
    ++ * offered has nothing to collapse, so a caller walks a whole VMA under the one
    ++ * lock it took to get there.  SCAN_SUCCEED means there is
     + * something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE means the page cache
     + * already holds the PMD folio and only the PTE table is left to retract.
     + * Both are work for the run, which is handed what the scan returned; anything
    @@ mm/collapse.h: struct collapse_control {
     + * gives it back.
     + */
     +void collapse_control_init(struct collapse_control *cc);
    -+void collapse_control_release(struct collapse_control *cc);
     +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
     +		unsigned long addr, struct collapse_control *cc,
     +		unsigned long orders);
    @@ mm/collapse.h: struct collapse_control {
     
      ## mm/khugepaged.c ##
     @@ mm/khugepaged.c: void __khugepaged_enter(struct mm_struct *mm)
    -  * Check what orders are possible based on the vma and collapse type.
    -  * This is used to determine if mTHP collapse is a viable option.
    + 		wake_up_interruptible(&khugepaged_wait);
    + }
    + 
    +-/*
    +- * Check what orders are possible based on the vma and collapse type.
    +- * This is used to determine if mTHP collapse is a viable option.
    ++/**
    ++ * collapse_possible_orders - which orders a VMA may collapse to
    ++ * @vma: the VMA
    ++ * @vm_flags: its flags, passed separately where they are about to change
    ++ * @tva_flags: who is asking, as thp_vma_allowable_orders() spells it
    ++ *
    ++ * khugepaged may collapse anonymous memory to any enabled order; everything
    ++ * else collapses to PMD order only.
    ++ *
    ++ * Return: the orders as a bitmask, zero when the VMA may not collapse at all.
       */
     -static unsigned long collapse_possible_orders(struct vm_area_struct *vma,
     +unsigned long collapse_possible_orders(struct vm_area_struct *vma,
    @@ mm/khugepaged.c: void __khugepaged_enter(struct mm_struct *mm)
      {
      	unsigned long orders;
     @@ mm/khugepaged.c: static int collapse_find_target_node(struct collapse_control *cc)
    + }
      #endif
      
    - /*
    +-/*
     - * If mmap_lock temporarily dropped, revalidate vma
     - * after taking the mmap_lock again.
     - * Returns enum scan_result value.
    -+ * Find the VMA at @address again once mmap_lock has been given up and taken
    -+ * back, and check it still allows a collapse of @order there.  The VMA has to
    -+ * span the whole PMD whatever @order is; with @expect_anon it also has to be
    -+ * anonymous and have an anon_vma.  *@vmap is the VMA found, if any.
    ++/**
    ++ * collapse_vma_revalidate - look a VMA up again after mmap_lock was dropped
    ++ * @mm: the mm
    ++ * @address: an address within the PTE table being collapsed
    ++ * @expect_anon: the collapse started on an anonymous VMA
    ++ * @vmap: the VMA found, if any
    ++ * @cc: the control, for the policy that says who is asking
    ++ * @order: the order the collapse is going for
    ++ *
    ++ * Called with mmap_lock held, for reading or writing, once it has been given up
    ++ * and taken back.  The VMA has to span the whole PMD whatever @order is; with
    ++ * @expect_anon it also has to be anonymous and have an anon_vma.
    ++ *
    ++ * Return: SCAN_SUCCEED, or why a collapse of @order at @address is off.
       */
     -
     -static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_file(struct mm_struct *mm
      }
      
     -static void collapse_control_init(struct collapse_control *cc)
    -+/* Set up a control before its first scan; cc->policy is the caller's to fill */
    ++/**
    ++ * collapse_control_init - set up a control before its first scan
    ++ * @cc: the control the caller carries across its scans
    ++ *
    ++ * cc->policy is the caller's to fill.
    ++ */
     +void collapse_control_init(struct collapse_control *cc)
      {
      	cc->progress = 0;
      	cc->scan_file = NULL;
    -@@ mm/khugepaged.c: static void collapse_put_scan_file(struct collapse_control *cc)
    - 	}
    - }
    - 
    --static void collapse_control_release(struct collapse_control *cc)
    -+/*
    -+ * Done with a control.  A scan that found something has to have been run by
    -+ * then: the file side takes a reference on the file while it still has the
    -+ * VMA to take it from, and the run is what gives it back.
    -+ */
    -+void collapse_control_release(struct collapse_control *cc)
    - {
    - 	collapse_put_scan_file(cc);
      }
      
     -static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
    -+/*
    -+ * Scan the PTE table of @vma at @addr for a collapse candidate.  @addr is
    -+ * aligned to the table; @orders is what the caller allows there.
    ++/**
    ++ * collapse_scan_pmd - scan one PTE table for a collapse candidate
    ++ * @vma: the VMA the table belongs to
    ++ * @addr: start of the table, PMD aligned
    ++ * @cc: the caller's control
    ++ * @orders: the orders the caller allows for @vma
     + *
    -+ * Called with mmap_lock held for reading and returns with it still held.  It
    -+ * only reads, and almost every table it is offered has nothing to collapse,
    -+ * so a caller walks a whole VMA under the one lock it took to get there.
    ++ * Called with mmap_lock held for reading and returns with it still held.
    ++ * Almost every table it is offered has nothing to collapse, so a caller walks
    ++ * a whole VMA under the one lock it took to get there.
     + *
    -+ * SCAN_SUCCEED means there is something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE
    -+ * means the page cache already holds the PMD folio and only the PTE table is
    -+ * left to retract.  Both are work for collapse_run_pmd(), which is handed
    -+ * what the scan returned; anything else is why there is nothing to do.
    ++ * Return: SCAN_SUCCEED when there is something to collapse;
    ++ * SCAN_PTE_MAPPED_HUGEPAGE when the page cache already holds the PMD folio and
    ++ * only the PTE table is left to retract.  Both are work for collapse_run_pmd(),
    ++ * which is handed what the scan returned.  Anything else is why there is
    ++ * nothing to do.
     + */
     +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
      		unsigned long addr, struct collapse_control *cc,
    @@ mm/khugepaged.c: static enum scan_result collapse_scan_pmd(struct vm_area_struct
      }
      
     -static enum scan_result collapse_run_pmd(struct mm_struct *mm,
    -+/*
    -+ * Collapse the table a scan found work in.  @result is what the scan
    -+ * returned.
    ++/**
    ++ * collapse_run_pmd - collapse the table a scan found work in
    ++ * @mm: the mm
    ++ * @addr: start of the table, as given to the scan
    ++ * @result: what the scan returned
    ++ * @cc: the control the scan ran with
     + *
     + * Called without mmap_lock and returns without it, taking what it needs in
     + * between: what it does -- allocate, isolate, copy, flush -- is slow enough
     + * that a writer would wait behind it.  The caller gives the lock up first,
     + * and with it the VMA and anything derived under it.  The run revalidates for
     + * itself rather than trusting what the scan saw.
    ++ *
    ++ * Return: what the collapse made of the table.
     + */
     +enum scan_result collapse_run_pmd(struct mm_struct *mm,
      		unsigned long addr, enum scan_result result,
12:  7d38fee1a929 ! 13:  0ad10df0a90a mm/collapse: implement MADV_COLLAPSE in madvise.c
    @@ mm/khugepaged.c: static void collapse_policy_khugepaged(struct collapse_policy *
      }
      
     -/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */
    --static void collapse_policy_forced(struct collapse_policy *p)
    +-static void collapse_policy_madvise(struct collapse_policy *p)
     -{
    --	p->max_ptes_none = HPAGE_PMD_NR;
    --	p->max_ptes_swap = HPAGE_PMD_NR;
    --	p->max_ptes_shared = HPAGE_PMD_NR;
    --	p->strict_sub_pmd = false;
    --	p->skip_lazyfree = false;
    --	p->require_referenced = false;
    --	p->install_pmd = true;
    --	p->writeback_dirty = true;
    +-	p->pmd.max_ptes_none = HPAGE_PMD_NR;
    +-	p->pmd.max_ptes_swap = HPAGE_PMD_NR;
    +-	p->pmd.max_ptes_shared = HPAGE_PMD_NR;
    +-	/* Never read: MADV_COLLAPSE collapses to PMD order only */
    +-	p->sub_pmd = p->pmd;
    +-
    +-	p->anon_skip_lazyfree = false;
    +-	p->anon_require_referenced = false;
    +-	p->file_install_pmd = true;
    +-	p->file_writeback_dirty = true;
     -	p->gfp = GFP_TRANSHUGE;
     -	p->tva_type = TVA_FORCED_COLLAPSE;
     -}
    @@ mm/khugepaged.c: static void collapse_policy_khugepaged(struct collapse_policy *
      static int collapse_find_target_node(struct collapse_control *cc)
      {
     @@ mm/khugepaged.c: enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
    -  * and with it the VMA and anything derived under it.  The run revalidates for
    -  * itself rather than trusting what the scan saw.
    +  *
    +  * Return: what the collapse made of the table.
       */
     -enum scan_result collapse_run_pmd(struct mm_struct *mm,
     -		unsigned long addr, enum scan_result result,
    @@ mm/khugepaged.c: bool current_is_khugepaged(void)
     -	if (!cc)
     -		return -ENOMEM;
     -	collapse_control_init(cc);
    --	collapse_policy_forced(&cc->policy);
    +-	collapse_policy_madvise(&cc->policy);
     -
     -	lru_add_drain_all();
     -
    @@ mm/khugepaged.c: bool current_is_khugepaged(void)
     -			vma = found;
     -			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
     -			orders = collapse_possible_orders(vma, vma->vm_flags,
    --							  cc->policy.tva_type);
    +-							  TVA_FORCED_COLLAPSE);
     -		}
     -
     -		result = collapse_scan_pmd(vma, addr, cc, orders);
    @@ mm/khugepaged.c: bool current_is_khugepaged(void)
     -		mmap_read_lock(mm);
     -out_locked:
     -	mmap_assert_locked(mm);
    --	collapse_control_release(cc);
     -	kfree(cc);
     -
     -	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
    @@ mm/madvise.c: bool madvise_dontneed_free_valid_vma(struct madvise_behavior *madv
     +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
     +
     +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */
    -+static void collapse_policy_forced(struct collapse_policy *p)
    ++static void collapse_policy_madvise(struct collapse_policy *p)
     +{
    -+	p->max_ptes_none = HPAGE_PMD_NR;
    -+	p->max_ptes_swap = HPAGE_PMD_NR;
    -+	p->max_ptes_shared = HPAGE_PMD_NR;
    -+	p->strict_sub_pmd = false;
    -+	p->skip_lazyfree = false;
    -+	p->require_referenced = false;
    -+	p->install_pmd = true;
    -+	p->writeback_dirty = true;
    ++	p->pmd.max_ptes_none = HPAGE_PMD_NR;
    ++	p->pmd.max_ptes_swap = HPAGE_PMD_NR;
    ++	p->pmd.max_ptes_shared = HPAGE_PMD_NR;
    ++	/* Never read: MADV_COLLAPSE collapses to PMD order only */
    ++	p->sub_pmd = p->pmd;
    ++
    ++	p->anon_skip_lazyfree = false;
    ++	p->anon_require_referenced = false;
    ++	p->file_install_pmd = true;
    ++	p->file_writeback_dirty = true;
     +	p->gfp = GFP_TRANSHUGE;
     +	p->tva_type = TVA_FORCED_COLLAPSE;
     +}
    @@ mm/madvise.c: bool madvise_dontneed_free_valid_vma(struct madvise_behavior *madv
     +	if (!cc)
     +		return -ENOMEM;
     +	collapse_control_init(cc);
    -+	collapse_policy_forced(&cc->policy);
    ++	collapse_policy_madvise(&cc->policy);
     +
     +	lru_add_drain_all();
     +
    @@ mm/madvise.c: bool madvise_dontneed_free_valid_vma(struct madvise_behavior *madv
     +			vma = found;
     +			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
     +			orders = collapse_possible_orders(vma, vma->vm_flags,
    -+							  cc->policy.tva_type);
    ++							  TVA_FORCED_COLLAPSE);
     +		}
     +
     +		result = collapse_scan_pmd(vma, addr, cc, orders);
    @@ mm/madvise.c: bool madvise_dontneed_free_valid_vma(struct madvise_behavior *madv
     +		mmap_read_lock(mm);
     +out_locked:
     +	mmap_assert_locked(mm);
    -+	collapse_control_release(cc);
     +	kfree(cc);
     +
     +	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0

base-commit: cf558a250cf4475a8936902b8978fbb6c61016f8
-- 
2.54.0

[PATCH v4 01/13] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse()

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

madvise_collapse() holds an mmgrab() reference across its work. It is
redundant. Every caller already holds mm_users:

 - madvise(2) works on current->mm, which lives as long as the task is in
   the syscall;
 - process_madvise(2) reaches a remote mm through mm_access(), which takes
   an mm_users reference and holds it until the syscall returns;
 - io_uring passes current->mm;
 - DAMON takes one with get_task_mm() and drops it after the call.

Drop the mmgrab()/mmdrop() pair.

Assisted-by: LLM
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index e13d233b9967..75c876acf527 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -3229,7 +3229,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	cc->is_khugepaged = false;
 	cc->progress = 0;
 
-	mmgrab(mm);
 	lru_add_drain_all();
 
 	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
@@ -3285,7 +3284,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	}
 out_nolock:
 	mmap_assert_locked(mm);
-	mmdrop(mm);
 	kfree(cc);
 
 	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
-- 
2.54.0

[PATCH v4 02/13] mm/khugepaged: count collapses where khugepaged makes them

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and
tests cc->is_khugepaged to know whether it should: the counter belongs to
the daemon, and MADV_COLLAPSE must not touch it.

The daemon sees every result of every collapse it asks for, so it can keep
its own counter without the shared path testing who called.

Assisted-by: LLM
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 75c876acf527..4979a93e3648 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -2815,10 +2815,8 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 
 	mmap_assert_locked(mm);
 
-	if (vma_is_anonymous(vma)) {
-		result = collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
-		goto end;
-	}
+	if (vma_is_anonymous(vma))
+		return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
 
 	file = get_file(vma->vm_file);
 	pgoff = linear_page_index(vma, addr);
@@ -2854,9 +2852,6 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 			result = SCAN_SUCCEED;
 		mmap_read_unlock(mm);
 	}
-end:
-	if (cc->is_khugepaged && result == SCAN_SUCCEED)
-		++khugepaged_pages_collapsed;
 	return result;
 }
 
@@ -2933,6 +2928,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 
 			*result = collapse_single_pmd(khugepaged_scan.address,
 						      vma, &lock_dropped, cc);
+			if (*result == SCAN_SUCCEED)
+				khugepaged_pages_collapsed++;
 			/* move to next address */
 			khugepaged_scan.address += HPAGE_PMD_SIZE;
 			if (lock_dropped)
-- 
2.54.0

[PATCH v4 03/13] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

The name says less than the bit means.  A set bit means not only that the
PTE is present, but also that it passed the other checks: uffd,
lazyfree, anonymity, sharing.  The PTE can be considered a collapse
source.

mthp_collapse() then reads the bitmap starting at the PMD order, so the
bitmap is not specific to mTHP either.

Name it for what a set bit means, and update the comments that named it.

No functional change.

Assisted-by: LLM
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 4979a93e3648..081f705cfca2 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -115,8 +115,8 @@ struct collapse_control {
 	/* nodemask for allocation fallback */
 	nodemask_t alloc_nmask;
 
-	/* Each bit represents a single occupied (!none/zero) page. */
-	DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE);
+	/* Each bit marks a PTE the scan accepted as a collapse source */
+	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
 };
 
 /**
@@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_control *cc)
 {
 	memset(cc->node_load, 0, sizeof(cc->node_load));
 	nodes_clear(cc->alloc_nmask);
-	bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE);
+	bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE);
 }
 
 static void release_pte_folio(struct folio *folio)
@@ -1482,15 +1482,15 @@ static unsigned int max_order_from_offset(unsigned int offset)
  * mthp_collapse() consumes the bitmap that is generated during
  * collapse_scan_pmd() to determine what regions and mTHP orders fit best.
  *
- * Each bit in cc->mthp_present_ptes represents a single occupied (!none/zero)
- * page. We start at the PMD order and check if it is eligible for collapse;
+ * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collapse
+ * source. We start at the PMD order and check if it is eligible for collapse;
  * if not, we check the left and right halves of the PTE page table we are
  * examining at a lower order.
  *
- * For each of these, we determine how many PTE entries are occupied in the
- * range of PTE entries we propose to collapse, then we compare this to a
- * threshold number of PTE entries which would need to be occupied for a
- * collapse to be permitted at that order (accounting for max_ptes_none).
+ * For each of these, we count the eligible PTEs in the range we propose to
+ * collapse, then we compare this to the number of eligible PTEs the range
+ * would need for a collapse to be permitted at that order (accounting for
+ * max_ptes_none).
  *
  * If a collapse is permitted, we attempt to collapse the PTE range into a
  * mTHP.
@@ -1499,7 +1499,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 		unsigned long address, int referenced, int unmapped,
 		struct collapse_control *cc, unsigned long enabled_orders)
 {
-	unsigned int nr_occupied_ptes, nr_ptes, max_ptes_none;
+	unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none;
 	enum scan_result last_result = SCAN_FAIL;
 	int collapsed = 0;
 	bool alloc_failed = false;
@@ -1514,18 +1514,18 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 			goto next_order;
 
 		max_ptes_none = collapse_max_ptes_none(cc, NULL, order);
-		nr_occupied_ptes = bitmap_weight_from(cc->mthp_present_ptes, offset,
+		nr_eligible_ptes = bitmap_weight_from(cc->eligible_ptes, offset,
 						      offset + nr_ptes);
 
 		/*
 		 * Swap PTEs accepted during the scan are counted in @unmapped,
-		 * not in the present-PTE bitmap. Account them for the PMD-order
+		 * not in cc->eligible_ptes. Account them for the PMD-order
 		 * candidate.
 		 */
 		if (is_pmd_order(order))
-			nr_occupied_ptes += unmapped;
+			nr_eligible_ptes += unmapped;
 
-		if (nr_occupied_ptes >= nr_ptes - max_ptes_none) {
+		if (nr_eligible_ptes >= nr_ptes - max_ptes_none) {
 			enum scan_result ret;
 
 			collapse_address = address + offset * PAGE_SIZE;
@@ -1731,8 +1731,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 			}
 		}
 
-		/* Set bit for occupied pages */
-		__set_bit(i, cc->mthp_present_ptes);
+		/* The scan accepted this PTE as a collapse source */
+		__set_bit(i, cc->eligible_ptes);
 		/*
 		 * Record which node the original page is from and save this
 		 * information to cc->node_load[].
-- 
2.54.0

[PATCH v4 04/13] mm/collapse: add collapse.h for the collapse interface

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

khugepaged.c holds both the users of collapse and the machinery that
performs it.  The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's
entry point and the collapse itself all sit in one file and reach into
each other freely.  Nothing marks where a user ends and the engine
begins.

Start drawing that line.

Add mm/collapse.h for what the two sides have to agree on:

  - enum scan_result - what the engine hands back;
  - struct collapse_control - the state a request carries.

And two constants move with them:

  - KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT;
  - KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER.

Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix.

No functional change.

Assisted-by: LLM
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 MAINTAINERS     |  1 +
 mm/collapse.h   | 64 +++++++++++++++++++++++++++++++++++++++
 mm/khugepaged.c | 79 ++++++++-----------------------------------------
 3 files changed, 78 insertions(+), 66 deletions(-)
 create mode 100644 mm/collapse.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 35262aa3e67e..36315972156b 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17434,6 +17434,7 @@ F:	Documentation/admin-guide/mm/transhuge.rst
 F:	include/linux/huge_mm.h
 F:	include/linux/khugepaged.h
 F:	include/trace/events/huge_memory.h
+F:	mm/collapse.h
 F:	mm/huge_memory.c
 F:	mm/khugepaged.c
 F:	mm/mm_slot.h
diff --git a/mm/collapse.h b/mm/collapse.h
new file mode 100644
index 000000000000..b115034d9018
--- /dev/null
+++ b/mm/collapse.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __MM_COLLAPSE_H
+#define __MM_COLLAPSE_H
+
+#include <linux/mm.h>
+#include <linux/nodemask.h>
+#include <linux/pgtable.h>
+#include <linux/types.h>
+
+#define COLLAPSE_MAX_PTES_LIMIT		(HPAGE_PMD_NR - 1)
+#define COLLAPSE_MIN_MTHP_ORDER		2
+
+enum scan_result {
+	SCAN_FAIL,
+	SCAN_SUCCEED,
+	SCAN_NO_PTE_TABLE,
+	SCAN_PMD_MAPPED,
+	SCAN_EXCEED_NONE_PTE,
+	SCAN_EXCEED_SWAP_PTE,
+	SCAN_EXCEED_SHARED_PTE,
+	SCAN_PTE_NON_PRESENT,
+	SCAN_PTE_UFFD,
+	SCAN_PTE_MAPPED_HUGEPAGE,
+	SCAN_LACK_REFERENCED_PAGE,
+	SCAN_PAGE_NULL,
+	SCAN_SCAN_ABORT,
+	SCAN_PAGE_COUNT,
+	SCAN_PAGE_LRU,
+	SCAN_PAGE_LOCK,
+	SCAN_PAGE_ANON,
+	SCAN_PAGE_LAZYFREE,
+	SCAN_PAGE_COMPOUND,
+	SCAN_ANY_PROCESS,
+	SCAN_VMA_NULL,
+	SCAN_VMA_CHECK,
+	SCAN_ADDRESS_RANGE,
+	SCAN_DEL_PAGE_LRU,
+	SCAN_ALLOC_HUGE_PAGE_FAIL,
+	SCAN_CGROUP_CHARGE_FAIL,
+	SCAN_TRUNCATED,
+	SCAN_PAGE_HAS_PRIVATE,
+	SCAN_STORE_FAILED,
+	SCAN_COPY_MC,
+	SCAN_PAGE_FILLED,
+	SCAN_PAGE_DIRTY_OR_WRITEBACK,
+};
+
+struct collapse_control {
+	bool is_khugepaged;
+
+	/* Num pages scanned per node */
+	u32 node_load[MAX_NUMNODES];
+
+	/* Num pages scanned (see khugepaged_pages_to_scan) */
+	unsigned int progress;
+
+	/* nodemask for allocation fallback */
+	nodemask_t alloc_nmask;
+
+	/* Each bit marks a PTE the scan accepted as a collapse source */
+	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
+};
+
+#endif	/* __MM_COLLAPSE_H */
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 081f705cfca2..8889f75cf45f 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -26,44 +26,10 @@
 #include <linux/cleanup.h>
 
 #include <asm/tlb.h>
+#include "collapse.h"
 #include "internal.h"
-#include "page_alloc.h"
 #include "mm_slot.h"
-
-enum scan_result {
-	SCAN_FAIL,
-	SCAN_SUCCEED,
-	SCAN_NO_PTE_TABLE,
-	SCAN_PMD_MAPPED,
-	SCAN_EXCEED_NONE_PTE,
-	SCAN_EXCEED_SWAP_PTE,
-	SCAN_EXCEED_SHARED_PTE,
-	SCAN_PTE_NON_PRESENT,
-	SCAN_PTE_UFFD,
-	SCAN_PTE_MAPPED_HUGEPAGE,
-	SCAN_LACK_REFERENCED_PAGE,
-	SCAN_PAGE_NULL,
-	SCAN_SCAN_ABORT,
-	SCAN_PAGE_COUNT,
-	SCAN_PAGE_LRU,
-	SCAN_PAGE_LOCK,
-	SCAN_PAGE_ANON,
-	SCAN_PAGE_LAZYFREE,
-	SCAN_PAGE_COMPOUND,
-	SCAN_ANY_PROCESS,
-	SCAN_VMA_NULL,
-	SCAN_VMA_CHECK,
-	SCAN_ADDRESS_RANGE,
-	SCAN_DEL_PAGE_LRU,
-	SCAN_ALLOC_HUGE_PAGE_FAIL,
-	SCAN_CGROUP_CHARGE_FAIL,
-	SCAN_TRUNCATED,
-	SCAN_PAGE_HAS_PRIVATE,
-	SCAN_STORE_FAILED,
-	SCAN_COPY_MC,
-	SCAN_PAGE_FILLED,
-	SCAN_PAGE_DIRTY_OR_WRITEBACK,
-};
+#include "page_alloc.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/huge_memory.h>
@@ -91,7 +57,6 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);
  *
  * Note that these are only respected if collapse was initiated by khugepaged.
  */
-#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1)
 unsigned int khugepaged_max_ptes_none __read_mostly;
 static unsigned int khugepaged_max_ptes_swap __read_mostly;
 static unsigned int khugepaged_max_ptes_shared __read_mostly;
@@ -101,24 +66,6 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);
 
 static struct kmem_cache *mm_slot_cache __ro_after_init;
 
-#define KHUGEPAGED_MIN_MTHP_ORDER	2
-
-struct collapse_control {
-	bool is_khugepaged;
-
-	/* Num pages scanned per node */
-	u32 node_load[MAX_NUMNODES];
-
-	/* Num pages scanned (see khugepaged_pages_to_scan) */
-	unsigned int progress;
-
-	/* nodemask for allocation fallback */
-	nodemask_t alloc_nmask;
-
-	/* Each bit marks a PTE the scan accepted as a collapse source */
-	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
-};
-
 /**
  * struct khugepaged_scan - cursor for scanning
  * @mm_head: the head of the mm list to scan
@@ -267,7 +214,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj,
 	unsigned long max_ptes_none;
 
 	err = kstrtoul(buf, 10, &max_ptes_none);
-	if (err || max_ptes_none > KHUGEPAGED_MAX_PTES_LIMIT)
+	if (err || max_ptes_none > COLLAPSE_MAX_PTES_LIMIT)
 		return -EINVAL;
 
 	khugepaged_max_ptes_none = max_ptes_none;
@@ -292,7 +239,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj,
 	unsigned long max_ptes_swap;
 
 	err  = kstrtoul(buf, 10, &max_ptes_swap);
-	if (err || max_ptes_swap > KHUGEPAGED_MAX_PTES_LIMIT)
+	if (err || max_ptes_swap > COLLAPSE_MAX_PTES_LIMIT)
 		return -EINVAL;
 
 	khugepaged_max_ptes_swap = max_ptes_swap;
@@ -318,7 +265,7 @@ static ssize_t max_ptes_shared_store(struct kobject *kobj,
 	unsigned long max_ptes_shared;
 
 	err  = kstrtoul(buf, 10, &max_ptes_shared);
-	if (err || max_ptes_shared > KHUGEPAGED_MAX_PTES_LIMIT)
+	if (err || max_ptes_shared > COLLAPSE_MAX_PTES_LIMIT)
 		return -EINVAL;
 
 	khugepaged_max_ptes_shared = max_ptes_shared;
@@ -378,19 +325,19 @@ static unsigned int collapse_max_ptes_none(struct collapse_control *cc,
 	if (is_pmd_order(order))
 		return max_ptes_none;
 	/*
-	 * for mTHP collapse with the sysctl value set to KHUGEPAGED_MAX_PTES_LIMIT,
+	 * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT,
 	 * scale the maximum number of PTEs to the order of the collapse.
 	 */
-	if (max_ptes_none == KHUGEPAGED_MAX_PTES_LIMIT)
+	if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT)
 		return (1 << order) - 1;
 	/*
-	 * For mTHP collapse of values other than 0 or KHUGEPAGED_MAX_PTES_LIMIT,
+	 * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT,
 	 * emit a warning and return 0.
 	 */
 	if (max_ptes_none)
 		pr_warn_once("mTHP collapse does not support max_ptes_none"
 		     " values other than 0 or %u, defaulting to 0.\n",
-		     KHUGEPAGED_MAX_PTES_LIMIT);
+		     COLLAPSE_MAX_PTES_LIMIT);
 	return 0;
 }
 
@@ -476,7 +423,7 @@ int __init khugepaged_init(void)
 		return -ENOMEM;
 
 	khugepaged_pages_to_scan = HPAGE_PMD_NR * 8;
-	khugepaged_max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
+	khugepaged_max_ptes_none = COLLAPSE_MAX_PTES_LIMIT;
 	khugepaged_max_ptes_swap = HPAGE_PMD_NR / 8;
 	khugepaged_max_ptes_shared = HPAGE_PMD_NR / 2;
 
@@ -1571,8 +1518,8 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 		 * any smaller order enabled. When at the smallest order
 		 * we must always move to the next offset.
 		 */
-		if (order > KHUGEPAGED_MIN_MTHP_ORDER &&
-			(enabled_orders & GENMASK(order - 1, 0))) {
+		if (order > COLLAPSE_MIN_MTHP_ORDER &&
+		    (enabled_orders & GENMASK(order - 1, 0))) {
 			order--;
 			continue;
 		}
@@ -1636,7 +1583,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 	 * is then checked again in mthp_collapse() for each attempted order.
 	 */
 	if (enabled_orders != BIT(HPAGE_PMD_ORDER))
-		max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT;
+		max_ptes_none = COLLAPSE_MAX_PTES_LIMIT;
 
 	pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl);
 	if (!pte) {
-- 
2.54.0

[PATCH v4 05/13] mm/collapse: state what a collapse may do in the policy

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

Tests scattered through the collapse path decide what a collapse is
allowed to do by asking whether khugepaged started it.  Between them they
settle:

 - which VMAs are eligible, and how hard to try for a folio;
 - how many empty, swapped-out or shared PTEs a window may contain, and
   whether a sub-PMD window is held to a stricter rule than a PMD;
 - whether a range has to look used, and whether a MADV_FREE'd page is
   left alone;
 - whether the PMD is mapped as part of the request, and whether dirty
   pages are worth writing back and retrying.

None of those is a fact about khugepaged.  Each is something the caller
decided before asking, and the collapse code should not have to look up
who called to find out.

Add struct collapse_policy for the caller to fill: khugepaged from its
own settings, MADV_COLLAPSE from the fact that a user asked explicitly.
Every test becomes a read of a field, and cc->is_khugepaged goes, having
no reader left.

The PTE limits come as two sets, one for a PMD-sized window and one for
anything smaller, so that the helpers pick a set for the order and read
it.  khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
and empty PTEs only when the knob says all or nothing; the warning for a
knob value in between moves to where khugepaged fills its policy.  The
fields only one side reads say which: anon_ or file_.  David Hildenbrand
asked for both.

khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
call.  That is the one change in behaviour.  The max_ptes_* limits and the
defrag setting behind the allocation mask are sampled once per pass rather
than on every table.  A table scanned early in a pass and one scanned late
are then treated alike.

collapse_file() also drops a NULL check on the collapse_control.  It has
one call site, reached only from collapse_single_pmd(), which dereferences
cc unconditionally, so the check was already dead.

Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/collapse.h   |  41 +++++++++++++-
 mm/khugepaged.c | 147 +++++++++++++++++++++++++-----------------------
 2 files changed, 118 insertions(+), 70 deletions(-)

diff --git a/mm/collapse.h b/mm/collapse.h
index b115034d9018..dcd117071955 100644
--- a/mm/collapse.h
+++ b/mm/collapse.h
@@ -45,8 +45,47 @@ enum scan_result {
 	SCAN_PAGE_DIRTY_OR_WRITEBACK,
 };
 
+/* How many PTEs of a window may be missing, swapped out or shared */
+struct collapse_limits {
+	/* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */
+	unsigned int max_ptes_none;
+	unsigned int max_ptes_swap;
+	unsigned int max_ptes_shared;
+};
+
+/* What a collapse is allowed to do, decided by the caller that asks for it */
+struct collapse_policy {
+	/* Limits for a PMD-sized window */
+	struct collapse_limits pmd;
+
+	/*
+	 * Limits for a smaller window.  Its max_ptes_none is either 0 or
+	 * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the
+	 * window whatever its order; any other value counts as 0.
+	 */
+	struct collapse_limits sub_pmd;
+
+	/* Leave clean lazyfree folios to reclaim rather than collapse them */
+	bool anon_skip_lazyfree;
+
+	/* Refuse an anonymous range with no sign of use */
+	bool anon_require_referenced;
+
+	/* Map the PMD over a file collapse instead of leaving it to a fault */
+	bool file_install_pmd;
+
+	/* Write dirty pages back and retry once instead of refusing them */
+	bool file_writeback_dirty;
+
+	/* How hard to try for a destination folio */
+	gfp_t gfp;
+
+	/* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */
+	enum tva_type tva_type;
+};
+
 struct collapse_control {
-	bool is_khugepaged;
+	struct collapse_policy policy;
 
 	/* Num pages scanned per node */
 	u32 node_load[MAX_NUMNODES];
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 8889f75cf45f..667216ae15dd 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -314,30 +314,17 @@ static bool pte_none_or_zero(pte_t pte)
 static unsigned int collapse_max_ptes_none(struct collapse_control *cc,
 		struct vm_area_struct *vma, unsigned int order)
 {
-	const unsigned int max_ptes_none = khugepaged_max_ptes_none;
+	unsigned int max_ptes_none;
 
 	if (vma && userfaultfd_armed(vma))
 		return 0;
-	/* for MADV_COLLAPSE, allow any empty/shared zeropage PTEs */
-	if (!cc->is_khugepaged)
-		return HPAGE_PMD_NR;
-	/* for PMD collapse, respect the user defined maximum */
 	if (is_pmd_order(order))
-		return max_ptes_none;
-	/*
-	 * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT,
-	 * scale the maximum number of PTEs to the order of the collapse.
-	 */
+		return cc->policy.pmd.max_ptes_none;
+
+	/* Below PMD order: all but one PTE of the window, or none */
+	max_ptes_none = cc->policy.sub_pmd.max_ptes_none;
 	if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT)
 		return (1 << order) - 1;
-	/*
-	 * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT,
-	 * emit a warning and return 0.
-	 */
-	if (max_ptes_none)
-		pr_warn_once("mTHP collapse does not support max_ptes_none"
-		     " values other than 0 or %u, defaulting to 0.\n",
-		     COLLAPSE_MAX_PTES_LIMIT);
 	return 0;
 }
 
@@ -353,20 +340,9 @@ static unsigned int collapse_max_ptes_none(struct collapse_control *cc,
 static unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
 		unsigned int order)
 {
-	/*
-	 * For MADV_COLLAPSE, do not restrict the number of PTEs that map shared
-	 * anonymous pages.
-	 */
-	if (!cc->is_khugepaged)
-		return HPAGE_PMD_NR;
-	/*
-	 * for mTHP collapse do not allow collapsing anonymous memory pages that
-	 * are shared between processes.
-	 */
-	if (!is_pmd_order(order))
-		return 0;
-	/* for PMD collapse, respect the user defined maximum */
-	return khugepaged_max_ptes_shared;
+	if (is_pmd_order(order))
+		return cc->policy.pmd.max_ptes_shared;
+	return cc->policy.sub_pmd.max_ptes_shared;
 }
 
 /**
@@ -381,17 +357,9 @@ static unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
 static unsigned int collapse_max_ptes_swap(struct collapse_control *cc,
 		unsigned int order)
 {
-	/*
-	 * For MADV_COLLAPSE, do not restrict the number PTEs entries or
-	 * pagecache entries that are non-present.
-	 */
-	if (!cc->is_khugepaged)
-		return HPAGE_PMD_NR;
-	/* for mTHP collapse do not allow any non-present PTEs or pagecache entries */
-	if (!is_pmd_order(order))
-		return 0;
-	/* for PMD collapse, respect the user defined maximum */
-	return khugepaged_max_ptes_swap;
+	if (is_pmd_order(order))
+		return cc->policy.pmd.max_ptes_swap;
+	return cc->policy.sub_pmd.max_ptes_swap;
 }
 
 int hugepage_madvise(struct vm_area_struct *vma,
@@ -678,7 +646,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		 * If the vma has the VM_DROPPABLE flag, the collapse will
 		 * preserve the lazyfree property without needing to skip.
 		 */
-		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
+		if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
 		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
 			result = SCAN_PAGE_LAZYFREE;
 			goto out;
@@ -767,12 +735,12 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 		if (folio_test_large(folio))
 			list_add_tail(&folio->lru, compound_pagelist);
 next:
-		if (cc->is_khugepaged &&
+		if (cc->policy.anon_require_referenced &&
 		    folio_pte_referenced(folio, vma, addr, pteval))
 			referenced++;
 	}
 
-	if (unlikely(cc->is_khugepaged && !referenced)) {
+	if (unlikely(cc->policy.anon_require_referenced && !referenced)) {
 		result = SCAN_LACK_REFERENCED_PAGE;
 	} else {
 		result = SCAN_SUCCEED;
@@ -938,9 +906,7 @@ static void khugepaged_alloc_sleep(void)
 	remove_wait_queue(&khugepaged_wait, &wait);
 }
 
-static struct collapse_control khugepaged_collapse_control = {
-	.is_khugepaged = true,
-};
+static struct collapse_control khugepaged_collapse_control;
 
 static bool collapse_scan_abort(int nid, struct collapse_control *cc)
 {
@@ -976,6 +942,52 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void)
 	return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT;
 }
 
+/* khugepaged collapses on its own initiative, so it obeys its own settings */
+static void collapse_policy_khugepaged(struct collapse_policy *p)
+{
+	p->pmd.max_ptes_none = READ_ONCE(khugepaged_max_ptes_none);
+	p->pmd.max_ptes_swap = READ_ONCE(khugepaged_max_ptes_swap);
+	p->pmd.max_ptes_shared = READ_ONCE(khugepaged_max_ptes_shared);
+
+	/*
+	 * A sub-PMD window takes no swapped-out and no shared PTE: reading
+	 * pages back or breaking CoW is not worth it for an mTHP.  Empty PTEs
+	 * it takes all or nothing, since anything in between would let one
+	 * collapse feed the next.
+	 */
+	p->sub_pmd.max_ptes_none = p->pmd.max_ptes_none;
+	if (p->sub_pmd.max_ptes_none &&
+	    p->sub_pmd.max_ptes_none != COLLAPSE_MAX_PTES_LIMIT)
+		pr_warn_once("mTHP collapse does not support max_ptes_none values other than 0 or %u, defaulting to 0.\n",
+			     COLLAPSE_MAX_PTES_LIMIT);
+	p->sub_pmd.max_ptes_swap = 0;
+	p->sub_pmd.max_ptes_shared = 0;
+
+	p->anon_skip_lazyfree = true;
+	p->anon_require_referenced = true;
+	p->file_install_pmd = false;
+	p->file_writeback_dirty = false;
+	p->gfp = alloc_hugepage_khugepaged_gfpmask();
+	p->tva_type = TVA_KHUGEPAGED;
+}
+
+/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */
+static void collapse_policy_madvise(struct collapse_policy *p)
+{
+	p->pmd.max_ptes_none = HPAGE_PMD_NR;
+	p->pmd.max_ptes_swap = HPAGE_PMD_NR;
+	p->pmd.max_ptes_shared = HPAGE_PMD_NR;
+	/* Never read: MADV_COLLAPSE collapses to PMD order only */
+	p->sub_pmd = p->pmd;
+
+	p->anon_skip_lazyfree = false;
+	p->anon_require_referenced = false;
+	p->file_install_pmd = true;
+	p->file_writeback_dirty = true;
+	p->gfp = GFP_TRANSHUGE;
+	p->tva_type = TVA_FORCED_COLLAPSE;
+}
+
 #ifdef CONFIG_NUMA
 static int collapse_find_target_node(struct collapse_control *cc)
 {
@@ -1013,8 +1025,7 @@ static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned l
 		struct collapse_control *cc, unsigned int order)
 {
 	struct vm_area_struct *vma;
-	enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :
-				 TVA_FORCED_COLLAPSE;
+	enum tva_type type = cc->policy.tva_type;
 
 	if (unlikely(collapse_test_exit_or_disable(mm)))
 		return SCAN_ANY_PROCESS;
@@ -1197,8 +1208,7 @@ static enum scan_result __collapse_huge_page_swapin(struct mm_struct *mm,
 static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,
 		struct collapse_control *cc, unsigned int order)
 {
-	gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
-		     GFP_TRANSHUGE);
+	gfp_t gfp = cc->policy.gfp;
 	int node = collapse_find_target_node(cc);
 	struct folio *folio;
 
@@ -1551,7 +1561,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
 	const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
 	unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
-	enum tva_type tva_flags = cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORCED_COLLAPSE;
+	enum tva_type tva_flags = cc->policy.tva_type;
 	pmd_t *pmd;
 	pte_t *pte, *_pte, pteval;
 	int i;
@@ -1651,7 +1661,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		 * If the vma has the VM_DROPPABLE flag, the collapse will
 		 * preserve the lazyfree property without needing to skip.
 		 */
-		if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) &&
+		if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
 		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
 			result = SCAN_PAGE_LAZYFREE;
 			failed_pfn = folio_pfn(folio);
@@ -1717,13 +1727,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 			goto out_unmap;
 		}
 
-		if (cc->is_khugepaged &&
+		if (cc->policy.anon_require_referenced &&
 		    folio_pte_referenced(folio, vma, addr, pteval))
 			referenced++;
 	}
-	if (cc->is_khugepaged &&
-		   (!referenced ||
-		    (unmapped && referenced < HPAGE_PMD_NR / 2))) {
+	if (cc->policy.anon_require_referenced &&
+	    (!referenced ||
+	     (unmapped && referenced < HPAGE_PMD_NR / 2))) {
 		result = SCAN_LACK_REFERENCED_PAGE;
 	} else {
 		result = SCAN_SUCCEED;
@@ -2582,11 +2592,11 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
 	xas_unlock_irq(&xas);
 
 	/*
-	 * Remove pte page tables, so we can re-fault the page as huge.
-	 * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp().
+	 * Remove pte page tables, so we can re-fault the page as huge.  A
+	 * caller that wants the PMD mapped now is told to go and do that.
 	 */
 	retract_page_tables(mapping, start);
-	if (cc && !cc->is_khugepaged)
+	if (cc->policy.file_install_pmd)
 		result = SCAN_PTE_MAPPED_HUGEPAGE;
 	folio_unlock(new_folio);
 
@@ -2773,11 +2783,8 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 retry:
 	result = collapse_scan_file(mm, addr, file, pgoff, cc);
 
-	/*
-	 * For MADV_COLLAPSE, when encountering dirty pages, try to writeback,
-	 * then retry the collapse one time.
-	 */
-	if (!cc->is_khugepaged && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
+	/* Dirty pages are worth a writeback and one more try, if asked for */
+	if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
 	    !triggered_wb && mapping_can_writeback(file->f_mapping)) {
 		const loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
 		const loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
@@ -2794,7 +2801,7 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 			result = SCAN_ANY_PROCESS;
 		else
 			result = try_collapse_pte_mapped_thp(mm, addr,
-							     !cc->is_khugepaged);
+							cc->policy.file_install_pmd);
 		if (result == SCAN_PMD_MAPPED)
 			result = SCAN_SUCCEED;
 		mmap_read_unlock(mm);
@@ -2943,6 +2950,8 @@ static void khugepaged_do_scan(struct collapse_control *cc)
 
 	lru_add_drain_all();
 
+	collapse_policy_khugepaged(&cc->policy);
+
 	cc->progress = 0;
 	while (true) {
 		cond_resched();
@@ -3170,7 +3179,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	cc = kmalloc_obj(*cc);
 	if (!cc)
 		return -ENOMEM;
-	cc->is_khugepaged = false;
+	collapse_policy_madvise(&cc->policy);
 	cc->progress = 0;
 
 	lru_add_drain_all();
-- 
2.54.0

Re: [PATCH v4 05/13] mm/collapse: state what a collapse may do in the policy

[David Hildenbrand (Arm)]

On 9/28/26 12:06, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> 
> Tests scattered through the collapse path decide what a collapse is
> allowed to do by asking whether khugepaged started it.  Between them they
> settle:
> 
>  - which VMAs are eligible, and how hard to try for a folio;
>  - how many empty, swapped-out or shared PTEs a window may contain, and
>    whether a sub-PMD window is held to a stricter rule than a PMD;
>  - whether a range has to look used, and whether a MADV_FREE'd page is
>    left alone;
>  - whether the PMD is mapped as part of the request, and whether dirty
>    pages are worth writing back and retrying.
> 
> None of those is a fact about khugepaged.  Each is something the caller
> decided before asking, and the collapse code should not have to look up
> who called to find out.
> 
> Add struct collapse_policy for the caller to fill: khugepaged from its
> own settings, MADV_COLLAPSE from the fact that a user asked explicitly.
> Every test becomes a read of a field, and cc->is_khugepaged goes, having
> no reader left.
> 
> The PTE limits come as two sets, one for a PMD-sized window and one for
> anything smaller, so that the helpers pick a set for the order and read
> it.  khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
> and empty PTEs only when the knob says all or nothing; the warning for a
> knob value in between moves to where khugepaged fills its policy.  The
> fields only one side reads say which: anon_ or file_.  David Hildenbrand
> asked for both.
> 
> khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
> call.  That is the one change in behaviour.  The max_ptes_* limits and the
> defrag setting behind the allocation mask are sampled once per pass rather
> than on every table.  A table scanned early in a pass and one scanned late
> are then treated alike.
> 
> collapse_file() also drops a NULL check on the collapse_control.  It has
> one call site, reached only from collapse_single_pmd(), which dereferences
> cc unconditionally, so the check was already dead.
> 
> Assisted-by: LLM
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  mm/collapse.h   |  41 +++++++++++++-
>  mm/khugepaged.c | 147 +++++++++++++++++++++++++-----------------------
>  2 files changed, 118 insertions(+), 70 deletions(-)
> 
> diff --git a/mm/collapse.h b/mm/collapse.h
> index b115034d9018..dcd117071955 100644
> --- a/mm/collapse.h
> +++ b/mm/collapse.h
> @@ -45,8 +45,47 @@ enum scan_result {
>  	SCAN_PAGE_DIRTY_OR_WRITEBACK,
>  };
>  
> +/* How many PTEs of a window may be missing, swapped out or shared */
> +struct collapse_limits {
> +	/* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */
> +	unsigned int max_ptes_none;
> +	unsigned int max_ptes_swap;
> +	unsigned int max_ptes_shared;
> +};
> +
> +/* What a collapse is allowed to do, decided by the caller that asks for it */
> +struct collapse_policy {
> +	/* Limits for a PMD-sized window */
> +	struct collapse_limits pmd;
> +
> +	/*
> +	 * Limits for a smaller window.  Its max_ptes_none is either 0 or
> +	 * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the
> +	 * window whatever its order; any other value counts as 0.
> +	 */
> +	struct collapse_limits sub_pmd;
> +

Yeah, this is cleaner.

Skimmed over the remaining stuff and nothing jumped at me

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v4 05/13] mm/collapse: state what a collapse may do in the policy

[Zi Yan]

On Mon Sep 28, 2026 at 6:06 AM EDT, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
>
> Tests scattered through the collapse path decide what a collapse is
> allowed to do by asking whether khugepaged started it.  Between them they
> settle:
>
>  - which VMAs are eligible, and how hard to try for a folio;
>  - how many empty, swapped-out or shared PTEs a window may contain, and
>    whether a sub-PMD window is held to a stricter rule than a PMD;
>  - whether a range has to look used, and whether a MADV_FREE'd page is
>    left alone;
>  - whether the PMD is mapped as part of the request, and whether dirty
>    pages are worth writing back and retrying.
>
> None of those is a fact about khugepaged.  Each is something the caller
> decided before asking, and the collapse code should not have to look up
> who called to find out.
>
> Add struct collapse_policy for the caller to fill: khugepaged from its
> own settings, MADV_COLLAPSE from the fact that a user asked explicitly.
> Every test becomes a read of a field, and cc->is_khugepaged goes, having
> no reader left.
>
> The PTE limits come as two sets, one for a PMD-sized window and one for
> anything smaller, so that the helpers pick a set for the order and read
> it.  khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
> and empty PTEs only when the knob says all or nothing; the warning for a
> knob value in between moves to where khugepaged fills its policy.  The
> fields only one side reads say which: anon_ or file_.  David Hildenbrand
> asked for both.
>
> khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
> call.  That is the one change in behaviour.  The max_ptes_* limits and the
> defrag setting behind the allocation mask are sampled once per pass rather
> than on every table.  A table scanned early in a pass and one scanned late
> are then treated alike.
>
> collapse_file() also drops a NULL check on the collapse_control.  It has
> one call site, reached only from collapse_single_pmd(), which dereferences
> cc unconditionally, so the check was already dead.
>
> Assisted-by: LLM
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  mm/collapse.h   |  41 +++++++++++++-
>  mm/khugepaged.c | 147 +++++++++++++++++++++++++-----------------------
>  2 files changed, 118 insertions(+), 70 deletions(-)
>
LGTM.

Reviewed-by: Zi Yan <ziy@nvidia.com>


-- 
Best Regards,
Yan, Zi

Re: [PATCH v4 05/13] mm/collapse: state what a collapse may do in the policy

[Baolin Wang]

On 9/28/26 6:06 PM, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> 
> Tests scattered through the collapse path decide what a collapse is
> allowed to do by asking whether khugepaged started it.  Between them they
> settle:
> 
>   - which VMAs are eligible, and how hard to try for a folio;
>   - how many empty, swapped-out or shared PTEs a window may contain, and
>     whether a sub-PMD window is held to a stricter rule than a PMD;
>   - whether a range has to look used, and whether a MADV_FREE'd page is
>     left alone;
>   - whether the PMD is mapped as part of the request, and whether dirty
>     pages are worth writing back and retrying.
> 
> None of those is a fact about khugepaged.  Each is something the caller
> decided before asking, and the collapse code should not have to look up
> who called to find out.
> 
> Add struct collapse_policy for the caller to fill: khugepaged from its
> own settings, MADV_COLLAPSE from the fact that a user asked explicitly.
> Every test becomes a read of a field, and cc->is_khugepaged goes, having
> no reader left.
> 
> The PTE limits come as two sets, one for a PMD-sized window and one for
> anything smaller, so that the helpers pick a set for the order and read
> it.  khugepaged takes no swapped-out or shared PTE into a sub-PMD window,
> and empty PTEs only when the knob says all or nothing; the warning for a
> knob value in between moves to where khugepaged fills its policy.  The
> fields only one side reads say which: anon_ or file_.  David Hildenbrand
> asked for both.
> 
> khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per
> call.  That is the one change in behaviour.  The max_ptes_* limits and the
> defrag setting behind the allocation mask are sampled once per pass rather
> than on every table.  A table scanned early in a pass and one scanned late
> are then treated alike.
> 
> collapse_file() also drops a NULL check on the collapse_control.  It has
> one call site, reached only from collapse_single_pmd(), which dereferences
> cc unconditionally, so the check was already dead.
> 
> Assisted-by: LLM
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>   mm/collapse.h   |  41 +++++++++++++-
>   mm/khugepaged.c | 147 +++++++++++++++++++++++++-----------------------
>   2 files changed, 118 insertions(+), 70 deletions(-)
> 
> diff --git a/mm/collapse.h b/mm/collapse.h
> index b115034d9018..dcd117071955 100644
> --- a/mm/collapse.h
> +++ b/mm/collapse.h
> @@ -45,8 +45,47 @@ enum scan_result {
>   	SCAN_PAGE_DIRTY_OR_WRITEBACK,
>   };
>   
> +/* How many PTEs of a window may be missing, swapped out or shared */
> +struct collapse_limits {
> +	/* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */
> +	unsigned int max_ptes_none;
> +	unsigned int max_ptes_swap;
> +	unsigned int max_ptes_shared;
> +};
> +
> +/* What a collapse is allowed to do, decided by the caller that asks for it */
> +struct collapse_policy {
> +	/* Limits for a PMD-sized window */
> +	struct collapse_limits pmd;
> +
> +	/*
> +	 * Limits for a smaller window.  Its max_ptes_none is either 0 or
> +	 * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the
> +	 * window whatever its order; any other value counts as 0.
> +	 */
> +	struct collapse_limits sub_pmd;

Much better than before. Thanks.

Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>

[PATCH v4 06/13] mm/collapse: drop the collapse_possible() wrapper

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

collapse_possible() only forwards to collapse_possible_orders() and turns
its mask into a bool.  Its three callers can test the mask themselves.

No functional change.

Assisted-by: LLM
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 667216ae15dd..cc69281c542f 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -494,17 +494,11 @@ static unsigned long collapse_possible_orders(struct vm_area_struct *vma,
 	return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders);
 }
 
-static bool collapse_possible(struct vm_area_struct *vma,
-		vm_flags_t vm_flags, enum tva_type tva_flags)
-{
-	return collapse_possible_orders(vma, vm_flags, tva_flags);
-}
-
 void khugepaged_enter_vma(struct vm_area_struct *vma,
 			  vm_flags_t vm_flags)
 {
-	if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled()
-	    && collapse_possible(vma, vm_flags, TVA_KHUGEPAGED))
+	if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() &&
+	    collapse_possible_orders(vma, vm_flags, TVA_KHUGEPAGED))
 		__khugepaged_enter(vma->vm_mm);
 }
 
@@ -2855,7 +2849,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 			cc->progress++;
 			break;
 		}
-		if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) {
+		if (!collapse_possible_orders(vma, vma->vm_flags,
+					      TVA_KHUGEPAGED)) {
 			cc->progress++;
 			continue;
 		}
@@ -3167,7 +3162,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	BUG_ON(vma->vm_start > start);
 	BUG_ON(vma->vm_end < end);
 
-	if (!collapse_possible(vma, vma->vm_flags, TVA_FORCED_COLLAPSE))
+	if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE))
 		return -EINVAL;
 
 	hstart = ALIGN(start, HPAGE_PMD_SIZE);
-- 
2.54.0

[PATCH v4 07/13] mm/collapse: name the per-table scan reset for what it resets

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

collapse_control_init_scan() resets what one scan accumulates: the node
load, the allocation nodemask and the eligible-PTE bitmap.  It runs before
every PTE table a scan is given, not once per control, so its name points
at the wrong thing.

Call it collapse_scan_reset().

Preparation for giving a control a real init and release, run once each
for a whole series of scans.  Two names a word apart would then stand for
two different jobs.

No functional change.

Assisted-by: LLM
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index cc69281c542f..60ec7e80d554 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -532,7 +532,7 @@ void __khugepaged_exit(struct mm_struct *mm)
 	}
 }
 
-static void collapse_control_init_scan(struct collapse_control *cc)
+static void collapse_scan_reset(struct collapse_control *cc)
 {
 	memset(cc->node_load, 0, sizeof(cc->node_load));
 	nodes_clear(cc->alloc_nmask);
@@ -1577,7 +1577,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		goto out;
 	}
 
-	collapse_control_init_scan(cc);
+	collapse_scan_reset(cc);
 
 	enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags);
 
@@ -2655,7 +2655,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 
 	present = 0;
 	swap = 0;
-	collapse_control_init_scan(cc);
+	collapse_scan_reset(cc);
 	rcu_read_lock();
 	xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) {
 		if (xas_retry(&xas, folio))
-- 
2.54.0

[PATCH v4 08/13] mm/collapse: call collapse_file() from collapse_single_pmd()

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

collapse_scan_file() reads the page cache to decide whether a table is
worth collapsing and, when it is, calls collapse_file() itself.  The
caller cannot get between the decision and the collapse.

Move the collapse_file() call up into collapse_single_pmd(), so the scan
stops at the decision.

Two things change with it.  The writeback retry re-runs collapse_file()
alone instead of rescanning first; collapse_file() repeats the scan's
checks under the page cache lock anyway.  And mm_khugepaged_scan_file
fires before the collapse, so for an accepted table its status reads
SCAN_SUCCEED; what the collapse made of the table is for
mm_khugepaged_collapse_file to report.

Preparation for splitting a collapse into a scan under mmap_lock and a
run without it.  The file scan has to stop where the anonymous one will.

Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 60ec7e80d554..89a4c3f5a91c 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -2737,13 +2737,9 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 	else
 		cc->progress += HPAGE_PMD_NR;
 
-	if (result == SCAN_SUCCEED) {
-		if (present < HPAGE_PMD_NR - max_ptes_none) {
-			result = SCAN_EXCEED_NONE_PTE;
-			count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
-		} else {
-			result = collapse_file(mm, addr, file, start, cc);
-		}
+	if (result == SCAN_SUCCEED && present < HPAGE_PMD_NR - max_ptes_none) {
+		result = SCAN_EXCEED_NONE_PTE;
+		count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
 	}
 
 	trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result);
@@ -2774,8 +2770,16 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 
 	mmap_read_unlock(mm);
 	*lock_dropped = true;
-retry:
+
+	/*
+	 * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds
+	 * the PMD folio, and only the PTE table is left to retract.
+	 */
 	result = collapse_scan_file(mm, addr, file, pgoff, cc);
+	if (result != SCAN_SUCCEED)
+		goto put;
+retry:
+	result = collapse_file(mm, addr, file, pgoff, cc);
 
 	/* Dirty pages are worth a writeback and one more try, if asked for */
 	if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
@@ -2787,6 +2791,7 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 		triggered_wb = true;
 		goto retry;
 	}
+put:
 	fput(file);
 
 	if (result == SCAN_PTE_MAPPED_HUGEPAGE) {
-- 
2.54.0

Re: [PATCH v4 08/13] mm/collapse: call collapse_file() from collapse_single_pmd()

[Zi Yan]

On Mon Sep 28, 2026 at 6:06 AM EDT, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
>
> collapse_scan_file() reads the page cache to decide whether a table is
> worth collapsing and, when it is, calls collapse_file() itself.  The
> caller cannot get between the decision and the collapse.
>
> Move the collapse_file() call up into collapse_single_pmd(), so the scan
> stops at the decision.
>
> Two things change with it.  The writeback retry re-runs collapse_file()
> alone instead of rescanning first; collapse_file() repeats the scan's
> checks under the page cache lock anyway.  And mm_khugepaged_scan_file

So the new version is a little faster by saving a file scan?

> fires before the collapse, so for an accepted table its status reads
> SCAN_SUCCEED; what the collapse made of the table is for
> mm_khugepaged_collapse_file to report.
>
> Preparation for splitting a collapse into a scan under mmap_lock and a
> run without it.  The file scan has to stop where the anonymous one will.
>
> Assisted-by: LLM
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  mm/khugepaged.c | 21 +++++++++++++--------
>  1 file changed, 13 insertions(+), 8 deletions(-)
>
LGTM.

Reviewed-by: Zi Yan <ziy@nvidia.com>

-- 
Best Regards,
Yan, Zi

[PATCH v4 09/13] mm/collapse: separate scanning a PTE table from collapsing it

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

A collapse is two jobs.  One reads a PTE table under mmap_lock and decides
whether the range is worth collapsing.  The other allocates, isolates,
copies and flushes, and wants the lock given up first.

collapse_single_pmd() did both, so the boundary between them was somewhere
in the middle of a function.

Give each half its own function:

  - collapse_scan_pmd() scans one table.  The anonymous scan that used to
    carry that name keeps its body as collapse_scan_anon_pmd(), and
    collapse_scan_pmd() is now the entry that picks the anonymous or the
    file side.

  - collapse_run_pmd() does the collapse the scan asked for, and is
    handed what the scan returned.  SCAN_SUCCEED means there is
    something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE means the page
    cache already holds the PMD folio and only the PTE table is left to
    retract.  Both are work for the run; anything else is why there is
    nothing to do.

collapse_single_pmd() is now the two of them with the mmap_lock drop in
between, so its callers see what they saw before.  collapse_control_init()
sets a control up before its first scan.

What the scan found and the run needs travels in collapse_control.  For
an anonymous table that is the orders and the referenced and swapped-out
counts, which mthp_collapse() and collapse_huge_page() now read from
there instead of taking as arguments.  For a file it is the file itself
and the offset in it: a file collapse works on the page cache and never
sees a VMA, so the scan takes the reference while it still has one and
the run gives it back.

The file scan moves under mmap_lock with the anonymous one, where before
the lock was given up first.  The lock is now held over the page cache
walk, an RCU walk over one table's worth of slots with no PTL, and taken
fewer times.  collapse_scan_mm_slot() ends its walk whenever the lock was
dropped, so a refused file table used to cost khugepaged an unlock, a
trip back through khugepaged_do_scan(), a relock and a VMA lookup.  Now
only a table that goes on to be collapsed does.

Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/collapse.h   |  14 +++++
 mm/khugepaged.c | 141 ++++++++++++++++++++++++++++++++----------------
 2 files changed, 108 insertions(+), 47 deletions(-)

diff --git a/mm/collapse.h b/mm/collapse.h
index dcd117071955..ca7b367c89cb 100644
--- a/mm/collapse.h
+++ b/mm/collapse.h
@@ -98,6 +98,20 @@ struct collapse_control {
 
 	/* Each bit marks a PTE the scan accepted as a collapse source */
 	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
+
+	/*
+	 * What a scan found and the run after it needs.  Live only between the
+	 * two, and read by nobody else.
+	 *
+	 * The file side takes a reference while it still has the VMA, since a
+	 * file collapse works on the page cache and never sees one; the run is
+	 * what gives it back.
+	 */
+	unsigned long scan_orders;
+	int scan_referenced;
+	int scan_unmapped;
+	struct file *scan_file;
+	pgoff_t scan_pgoff;
 };
 
 #endif	/* __MM_COLLAPSE_H */
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 89a4c3f5a91c..2b044e63d8c9 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1235,8 +1235,8 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
  * while allocating a THP, as that could trigger direct reclaim/compaction.
  * Note that the VMA must be rechecked after grabbing the mmap_lock again.
  */
-static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long start_addr,
-		int referenced, int unmapped, struct collapse_control *cc,
+static enum scan_result collapse_huge_page(struct mm_struct *mm,
+		unsigned long start_addr, struct collapse_control *cc,
 		unsigned int order)
 {
 	const unsigned long pmd_addr = start_addr & HPAGE_PMD_MASK;
@@ -1275,14 +1275,14 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 		goto out_nolock;
 	}
 
-	if (unmapped) {
+	if (cc->scan_unmapped) {
 		/*
 		 * __collapse_huge_page_swapin() will return with mmap_lock
 		 * released when it fails. So we jump out_nolock directly in
 		 * that case.  Continuing to collapse causes inconsistency.
 		 */
 		result = __collapse_huge_page_swapin(mm, vma, start_addr, pmd,
-						     referenced, order);
+						     cc->scan_referenced, order);
 		if (result != SCAN_SUCCEED)
 			goto out_nolock;
 	}
@@ -1446,9 +1446,8 @@ static unsigned int max_order_from_offset(unsigned int offset)
  * If a collapse is permitted, we attempt to collapse the PTE range into a
  * mTHP.
  */
-static enum scan_result mthp_collapse(struct mm_struct *mm,
-		unsigned long address, int referenced, int unmapped,
-		struct collapse_control *cc, unsigned long enabled_orders)
+static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long address,
+		struct collapse_control *cc)
 {
 	unsigned int nr_eligible_ptes, nr_ptes, max_ptes_none;
 	enum scan_result last_result = SCAN_FAIL;
@@ -1461,7 +1460,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 	while (offset < HPAGE_PMD_NR) {
 		nr_ptes = 1UL << order;
 
-		if (!test_bit(order, &enabled_orders))
+		if (!test_bit(order, &cc->scan_orders))
 			goto next_order;
 
 		max_ptes_none = collapse_max_ptes_none(cc, NULL, order);
@@ -1469,19 +1468,18 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 						      offset + nr_ptes);
 
 		/*
-		 * Swap PTEs accepted during the scan are counted in @unmapped,
-		 * not in cc->eligible_ptes. Account them for the PMD-order
-		 * candidate.
+		 * Swap PTEs accepted during the scan are counted in
+		 * cc->scan_unmapped, not in cc->eligible_ptes. Account them for
+		 * the PMD-order candidate.
 		 */
 		if (is_pmd_order(order))
-			nr_eligible_ptes += unmapped;
+			nr_eligible_ptes += cc->scan_unmapped;
 
 		if (nr_eligible_ptes >= nr_ptes - max_ptes_none) {
 			enum scan_result ret;
 
 			collapse_address = address + offset * PAGE_SIZE;
-			ret = collapse_huge_page(mm, collapse_address, referenced,
-						 unmapped, cc, order);
+			ret = collapse_huge_page(mm, collapse_address, cc, order);
 
 			switch (ret) {
 			/* Cases where we continue to next collapse candidate */
@@ -1523,7 +1521,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 		 * we must always move to the next offset.
 		 */
 		if (order > COLLAPSE_MIN_MTHP_ORDER &&
-		    (enabled_orders & GENMASK(order - 1, 0))) {
+		    (cc->scan_orders & GENMASK(order - 1, 0))) {
 			order--;
 			continue;
 		}
@@ -1548,14 +1546,14 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
 	return last_result;
 }
 
-static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
-		struct vm_area_struct *vma, unsigned long start_addr,
-		bool *lock_dropped, struct collapse_control *cc)
+static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,
+		unsigned long start_addr, struct collapse_control *cc)
 {
 	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
 	const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
 	unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
 	enum tva_type tva_flags = cc->policy.tva_type;
+	struct mm_struct *mm = vma->vm_mm;
 	pmd_t *pmd;
 	pte_t *pte, *_pte, pteval;
 	int i;
@@ -1735,12 +1733,9 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 out_unmap:
 	pte_unmap_unlock(pte, ptl);
 	if (result == SCAN_SUCCEED) {
-		/* collapse_huge_page() expects the lock to be dropped before calling */
-		mmap_read_unlock(mm);
-		result = mthp_collapse(mm, start_addr, referenced,
-				       unmapped, cc, enabled_orders);
-		/* mmap_lock was released above, set lock_dropped */
-		*lock_dropped = true;
+		cc->scan_orders = enabled_orders;
+		cc->scan_referenced = referenced;
+		cc->scan_unmapped = unmapped;
 	}
 out:
 	trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced,
@@ -2742,41 +2737,67 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 		count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
 	}
 
-	trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result);
+	trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap,
+				      result);
 	return result;
 }
 
-/*
- * Try to collapse a single PMD starting at a PMD aligned addr, and return
- * the results.
- */
-static enum scan_result collapse_single_pmd(unsigned long addr,
-		struct vm_area_struct *vma, bool *lock_dropped,
-		struct collapse_control *cc)
+static void collapse_control_init(struct collapse_control *cc)
+{
+	cc->progress = 0;
+	cc->scan_file = NULL;
+}
+
+static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
+		unsigned long addr, struct collapse_control *cc)
 {
-	struct mm_struct *mm = vma->vm_mm;
-	bool triggered_wb = false;
 	enum scan_result result;
-	struct file *file;
 	pgoff_t pgoff;
 
-	mmap_assert_locked(mm);
+	mmap_assert_locked(vma->vm_mm);
+	/* Whatever the last scan found has to have been run by now */
+	if (WARN_ON_ONCE(cc->scan_file)) {
+		fput(cc->scan_file);
+		cc->scan_file = NULL;
+	}
 
 	if (vma_is_anonymous(vma))
-		return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc);
+		return collapse_scan_anon_pmd(vma, addr, cc);
 
-	file = get_file(vma->vm_file);
 	pgoff = linear_page_index(vma, addr);
-
-	mmap_read_unlock(mm);
-	*lock_dropped = true;
-
+	result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc);
 	/*
 	 * SCAN_PTE_MAPPED_HUGEPAGE is work too: the page cache already holds
-	 * the PMD folio, and only the PTE table is left to retract.
+	 * the PMD folio, and retracting the PTE table is the run's job.
 	 */
-	result = collapse_scan_file(mm, addr, file, pgoff, cc);
-	if (result != SCAN_SUCCEED)
+	if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
+		return result;
+
+	/*
+	 * A file collapse works on the page cache and never sees a VMA, so take
+	 * what it needs from this one while it is still here.
+	 */
+	cc->scan_file = get_file(vma->vm_file);
+	cc->scan_pgoff = pgoff;
+	return result;
+}
+
+static enum scan_result collapse_run_pmd(struct mm_struct *mm,
+		unsigned long addr, enum scan_result result,
+		struct collapse_control *cc)
+{
+	struct file *file = cc->scan_file;
+	bool triggered_wb = false;
+	pgoff_t pgoff;
+
+	if (!file)
+		return mthp_collapse(mm, addr, cc);
+
+	cc->scan_file = NULL;
+	pgoff = cc->scan_pgoff;
+
+	/* The scan found the PMD folio in place: nothing to collapse */
+	if (result == SCAN_PTE_MAPPED_HUGEPAGE)
 		goto put;
 retry:
 	result = collapse_file(mm, addr, file, pgoff, cc);
@@ -2794,6 +2815,10 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 put:
 	fput(file);
 
+	/*
+	 * A PMD folio is in the page cache, whether the collapse just put it
+	 * there or found it: retract the PTE table, and map the PMD if asked.
+	 */
 	if (result == SCAN_PTE_MAPPED_HUGEPAGE) {
 		mmap_read_lock(mm);
 		if (collapse_test_exit_or_disable(mm))
@@ -2808,6 +2833,28 @@ static enum scan_result collapse_single_pmd(unsigned long addr,
 	return result;
 }
 
+/*
+ * Try to collapse a single PMD starting at a PMD aligned addr, and return
+ * the results.
+ */
+static enum scan_result collapse_single_pmd(unsigned long addr,
+		struct vm_area_struct *vma, bool *lock_dropped,
+		struct collapse_control *cc)
+{
+	struct mm_struct *mm = vma->vm_mm;
+	enum scan_result result;
+
+	result = collapse_scan_pmd(vma, addr, cc);
+	if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
+		return result;
+
+	/* The collapse takes its own locks, so give this up */
+	mmap_read_unlock(mm);
+	*lock_dropped = true;
+
+	return collapse_run_pmd(mm, addr, result, cc);
+}
+
 static void collapse_scan_mm_slot(unsigned int progress_max,
 		enum scan_result *result, struct collapse_control *cc)
 	__releases(&khugepaged_mm_lock)
@@ -2950,9 +2997,9 @@ static void khugepaged_do_scan(struct collapse_control *cc)
 
 	lru_add_drain_all();
 
+	collapse_control_init(cc);
 	collapse_policy_khugepaged(&cc->policy);
 
-	cc->progress = 0;
 	while (true) {
 		cond_resched();
 
@@ -3179,8 +3226,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	cc = kmalloc_obj(*cc);
 	if (!cc)
 		return -ENOMEM;
+	collapse_control_init(cc);
 	collapse_policy_madvise(&cc->policy);
-	cc->progress = 0;
 
 	lru_add_drain_all();
 
-- 
2.54.0

Re: [PATCH v4 09/13] mm/collapse: separate scanning a PTE table from collapsing it

[Zi Yan]

On Mon Sep 28, 2026 at 6:06 AM EDT, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
>
> A collapse is two jobs.  One reads a PTE table under mmap_lock and decides
> whether the range is worth collapsing.  The other allocates, isolates,
> copies and flushes, and wants the lock given up first.
>
> collapse_single_pmd() did both, so the boundary between them was somewhere
> in the middle of a function.
>
> Give each half its own function:
>
>   - collapse_scan_pmd() scans one table.  The anonymous scan that used to
>     carry that name keeps its body as collapse_scan_anon_pmd(), and
>     collapse_scan_pmd() is now the entry that picks the anonymous or the
>     file side.
>
>   - collapse_run_pmd() does the collapse the scan asked for, and is
>     handed what the scan returned.  SCAN_SUCCEED means there is
>     something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE means the page
>     cache already holds the PMD folio and only the PTE table is left to
>     retract.  Both are work for the run; anything else is why there is
>     nothing to do.
>
> collapse_single_pmd() is now the two of them with the mmap_lock drop in
> between, so its callers see what they saw before.  collapse_control_init()
> sets a control up before its first scan.
>
> What the scan found and the run needs travels in collapse_control.  For
> an anonymous table that is the orders and the referenced and swapped-out
> counts, which mthp_collapse() and collapse_huge_page() now read from
> there instead of taking as arguments.  For a file it is the file itself
> and the offset in it: a file collapse works on the page cache and never
> sees a VMA, so the scan takes the reference while it still has one and
> the run gives it back.
>
> The file scan moves under mmap_lock with the anonymous one, where before
> the lock was given up first.  The lock is now held over the page cache
> walk, an RCU walk over one table's worth of slots with no PTL, and taken
> fewer times.  collapse_scan_mm_slot() ends its walk whenever the lock was
> dropped, so a refused file table used to cost khugepaged an unlock, a
> trip back through khugepaged_do_scan(), a relock and a VMA lookup.  Now
> only a table that goes on to be collapsed does.
>
> Assisted-by: LLM
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  mm/collapse.h   |  14 +++++
>  mm/khugepaged.c | 141 ++++++++++++++++++++++++++++++++----------------
>  2 files changed, 108 insertions(+), 47 deletions(-)
>
> diff --git a/mm/collapse.h b/mm/collapse.h
> index dcd117071955..ca7b367c89cb 100644
> --- a/mm/collapse.h
> +++ b/mm/collapse.h
> @@ -98,6 +98,20 @@ struct collapse_control {
>  
>  	/* Each bit marks a PTE the scan accepted as a collapse source */
>  	DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE);
> +
> +	/*
> +	 * What a scan found and the run after it needs.  Live only between the
> +	 * two, and read by nobody else.
> +	 *
> +	 * The file side takes a reference while it still has the VMA, since a
> +	 * file collapse works on the page cache and never sees one; the run is
> +	 * what gives it back.
> +	 */
> +	unsigned long scan_orders;
> +	int scan_referenced;
> +	int scan_unmapped;

They are used by anon only.

> +	struct file *scan_file;
> +	pgoff_t scan_pgoff;

They are used by file only.

I think the comments need to address them separately.

Otherwise, LGTM.

Acked-by: Zi Yan <ziy@nvidia.com>



-- 
Best Regards,
Yan, Zi

[PATCH v4 10/13] mm/collapse: open-code collapse_single_pmd() in its two callers

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

A scan and a collapse want different things from mmap_lock.  The scan
reads one PTE table under the lock the caller holds, refuses most of the
time, and the caller moves on to the next table without letting go.  The
collapse allocates, may sleep in writeback and takes the lock for write
itself, so the lock it is handed is of no use to it.

collapse_single_pmd() kept that boundary inside itself.  It dropped the
lock on some paths and not others, and reported which by way of a bool
its callers had to carry along and then act on.  Both callers already
act on a drop, khugepaged by ending its walk and madvise_collapse() by
looking its VMA up again.  The code that has to know is not the code
that does it.

Open-code it in the two callers.  Each scans under the lock it already
holds and, when the scan found work, gives the lock up before running
the collapse.  The scan then has one rule, called locked and returning
locked, and the run another, called unlocked.  Nothing is left to
report, so khugepaged's lock_dropped and madvise_collapse()'s
mmap_unlocked both go.  The engine never touches a lock it did not
take, and how a caller locks its scan is the caller's business alone.

khugepaged's walk carries on to the next table while the scan keeps
refusing, and ends once a collapse has taken the lock from under it.
madvise_collapse() re-finds its VMA after a collapse, which it did
before, and now uses a NULL vma to say that it has to.  It still reports
the drop to its own caller, from the line that does it.

Preparation for moving madvise_collapse() out of khugepaged.c: what it
needs from the engine is then two calls with one lock rule each.

The lock is given up and taken again at the same points as before.  No
functional change.

Assisted-by: LLM
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 103 +++++++++++++++++++++++-------------------------
 1 file changed, 50 insertions(+), 53 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 2b044e63d8c9..be8a26e1de49 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -2833,28 +2833,6 @@ static enum scan_result collapse_run_pmd(struct mm_struct *mm,
 	return result;
 }
 
-/*
- * Try to collapse a single PMD starting at a PMD aligned addr, and return
- * the results.
- */
-static enum scan_result collapse_single_pmd(unsigned long addr,
-		struct vm_area_struct *vma, bool *lock_dropped,
-		struct collapse_control *cc)
-{
-	struct mm_struct *mm = vma->vm_mm;
-	enum scan_result result;
-
-	result = collapse_scan_pmd(vma, addr, cc);
-	if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
-		return result;
-
-	/* The collapse takes its own locks, so give this up */
-	mmap_read_unlock(mm);
-	*lock_dropped = true;
-
-	return collapse_run_pmd(mm, addr, result, cc);
-}
-
 static void collapse_scan_mm_slot(unsigned int progress_max,
 		enum scan_result *result, struct collapse_control *cc)
 	__releases(&khugepaged_mm_lock)
@@ -2917,7 +2895,7 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 		VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK);
 
 		while (khugepaged_scan.address < hend) {
-			bool lock_dropped = false;
+			unsigned long addr;
 
 			cond_resched();
 			if (unlikely(collapse_test_exit_or_disable(mm)))
@@ -2927,23 +2905,30 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 				  khugepaged_scan.address + HPAGE_PMD_SIZE >
 				  hend);
 
-			*result = collapse_single_pmd(khugepaged_scan.address,
-						      vma, &lock_dropped, cc);
-			if (*result == SCAN_SUCCEED)
-				khugepaged_pages_collapsed++;
+			addr = khugepaged_scan.address;
 			/* move to next address */
 			khugepaged_scan.address += HPAGE_PMD_SIZE;
-			if (lock_dropped)
-				/*
-				 * We released mmap_lock so break loop.  Note
-				 * that we drop mmap_lock before all hugepage
-				 * allocations, so if allocation fails, we are
-				 * guaranteed to break here and report the
-				 * correct result back to caller.
-				 */
-				goto breakouterloop_mmap_lock;
-			if (cc->progress >= progress_max)
-				goto breakouterloop;
+
+			*result = collapse_scan_pmd(vma, addr, cc);
+			/* Nothing to do here, and the lock is still ours */
+			if (*result != SCAN_SUCCEED &&
+			    *result != SCAN_PTE_MAPPED_HUGEPAGE) {
+				if (cc->progress >= progress_max)
+					goto breakouterloop;
+				continue;
+			}
+
+			/*
+			 * A collapse takes its own locks and is slow enough
+			 * that a writer should not wait behind it, so give the
+			 * lock up.  That ends this walk: vma and the mm are
+			 * whatever the collapse leaves them.
+			 */
+			mmap_read_unlock(mm);
+			*result = collapse_run_pmd(mm, addr, *result, cc);
+			if (*result == SCAN_SUCCEED)
+				khugepaged_pages_collapsed++;
+			goto breakouterloop_mmap_lock;
 		}
 	}
 breakouterloop:
@@ -3209,7 +3194,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	unsigned long hstart, hend, addr;
 	enum scan_result last_fail = SCAN_FAIL;
 	int thps = 0;
-	bool mmap_unlocked = false;
 
 	BUG_ON(vma->vm_start > start);
 	BUG_ON(vma->vm_end < end);
@@ -3232,25 +3216,40 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 	lru_add_drain_all();
 
 	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
-		enum scan_result result = SCAN_FAIL;
+		struct vm_area_struct *found;
+		enum scan_result result;
 
-		if (mmap_unlocked) {
+		/*
+		 * A collapse gives the lock up, so the VMA has to be found
+		 * again after one: it can shrink while nothing is held.  A scan
+		 * that finds nothing to collapse leaves the lock alone, so a
+		 * range that is already collapsed walks on without relocking.
+		 */
+		if (!vma) {
 			cond_resched();
 			mmap_read_lock(mm);
-			mmap_unlocked = false;
-			*lock_dropped = true;
-			result = hugepage_vma_revalidate(mm, addr, false, &vma,
+			result = hugepage_vma_revalidate(mm, addr, false, &found,
 							 cc, HPAGE_PMD_ORDER);
 			if (result != SCAN_SUCCEED) {
 				last_fail = result;
-				goto out_nolock;
+				goto out_locked;
 			}
-
+			vma = found;
 			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
 		}
 
-		result = collapse_single_pmd(addr, vma, &mmap_unlocked, cc);
+		result = collapse_scan_pmd(vma, addr, cc);
+		/* Nothing to do here, and the lock is still ours */
+		if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
+			goto tally;
 
+		/* The collapse takes its own locks, so give this up */
+		mmap_read_unlock(mm);
+		*lock_dropped = true;
+		vma = NULL;
+
+		result = collapse_run_pmd(mm, addr, result, cc);
+tally:
 		switch (result) {
 		case SCAN_SUCCEED:
 		case SCAN_PMD_MAPPED:
@@ -3272,17 +3271,15 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 		default:
 			last_fail = result;
 			/* Other error, exit */
-			goto out_maybelock;
+			goto out;
 		}
 	}
 
-out_maybelock:
+out:
 	/* Caller expects us to hold mmap_lock on return */
-	if (mmap_unlocked) {
-		*lock_dropped = true;
+	if (!vma)
 		mmap_read_lock(mm);
-	}
-out_nolock:
+out_locked:
 	mmap_assert_locked(mm);
 	kfree(cc);
 
-- 
2.54.0

[PATCH v4 11/13] mm/collapse: work out the orders a VMA allows once per VMA

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

The scan asked collapse_possible_orders() for every PTE table, for an
answer that is a property of the VMA.  Both callers walk a VMA a table at
a time, so let them work it out once and pass the mask in.  It is only
good while the lock that produced it is held, so madvise_collapse() takes
it again after every collapse.

The mask is then sampled once per VMA rather than once per table.  A thp
enabled knob written during a walk takes effect one VMA later, and cannot
widen a collapse: hugepage_vma_revalidate() tests the order again under
the lock the collapse retakes.

Assisted-by: LLM
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/khugepaged.c | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index be8a26e1de49..ed74d3268698 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1547,12 +1547,12 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long addres
 }
 
 static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,
-		unsigned long start_addr, struct collapse_control *cc)
+		unsigned long start_addr, struct collapse_control *cc,
+		unsigned long enabled_orders)
 {
 	const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER);
 	const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER);
 	unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER);
-	enum tva_type tva_flags = cc->policy.tva_type;
 	struct mm_struct *mm = vma->vm_mm;
 	pmd_t *pmd;
 	pte_t *pte, *_pte, pteval;
@@ -1563,7 +1563,6 @@ static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,
 	struct folio *folio = NULL;
 	unsigned long failed_pfn = -1;
 	unsigned long addr;
-	unsigned long enabled_orders;
 	spinlock_t *ptl;
 	int node = NUMA_NO_NODE, unmapped = 0;
 
@@ -1577,8 +1576,6 @@ static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma,
 
 	collapse_scan_reset(cc);
 
-	enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags);
-
 	/*
 	 * If PMD is the only enabled order, enforce max_ptes_none, otherwise
 	 * scan all pages to populate the bitmap for mTHP collapse. The bitmap
@@ -2749,7 +2746,8 @@ static void collapse_control_init(struct collapse_control *cc)
 }
 
 static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
-		unsigned long addr, struct collapse_control *cc)
+		unsigned long addr, struct collapse_control *cc,
+		unsigned long orders)
 {
 	enum scan_result result;
 	pgoff_t pgoff;
@@ -2762,7 +2760,7 @@ static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
 	}
 
 	if (vma_is_anonymous(vma))
-		return collapse_scan_anon_pmd(vma, addr, cc);
+		return collapse_scan_anon_pmd(vma, addr, cc, orders);
 
 	pgoff = linear_page_index(vma, addr);
 	result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc);
@@ -2872,15 +2870,17 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 
 	vma_iter_init(&vmi, mm, khugepaged_scan.address);
 	for_each_vma(vmi, vma) {
-		unsigned long hstart, hend;
+		unsigned long hstart, hend, orders;
 
 		cond_resched();
 		if (unlikely(collapse_test_exit_or_disable(mm))) {
 			cc->progress++;
 			break;
 		}
-		if (!collapse_possible_orders(vma, vma->vm_flags,
-					      TVA_KHUGEPAGED)) {
+		/* One mask for the whole VMA */
+		orders = collapse_possible_orders(vma, vma->vm_flags,
+						  TVA_KHUGEPAGED);
+		if (!orders) {
 			cc->progress++;
 			continue;
 		}
@@ -2909,7 +2909,7 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
 			/* move to next address */
 			khugepaged_scan.address += HPAGE_PMD_SIZE;
 
-			*result = collapse_scan_pmd(vma, addr, cc);
+			*result = collapse_scan_pmd(vma, addr, cc, orders);
 			/* Nothing to do here, and the lock is still ours */
 			if (*result != SCAN_SUCCEED &&
 			    *result != SCAN_PTE_MAPPED_HUGEPAGE) {
@@ -3191,14 +3191,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 {
 	struct collapse_control *cc;
 	struct mm_struct *mm = vma->vm_mm;
-	unsigned long hstart, hend, addr;
+	unsigned long hstart, hend, addr, orders;
 	enum scan_result last_fail = SCAN_FAIL;
 	int thps = 0;
 
 	BUG_ON(vma->vm_start > start);
 	BUG_ON(vma->vm_end < end);
 
-	if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE))
+	orders = collapse_possible_orders(vma, vma->vm_flags,
+					  TVA_FORCED_COLLAPSE);
+	if (!orders)
 		return -EINVAL;
 
 	hstart = ALIGN(start, HPAGE_PMD_SIZE);
@@ -3236,9 +3238,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 			}
 			vma = found;
 			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
+			orders = collapse_possible_orders(vma, vma->vm_flags,
+							  TVA_FORCED_COLLAPSE);
 		}
 
-		result = collapse_scan_pmd(vma, addr, cc);
+		result = collapse_scan_pmd(vma, addr, cc, orders);
 		/* Nothing to do here, and the lock is still ours */
 		if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
 			goto tally;
-- 
2.54.0

[PATCH v4 12/13] mm/collapse: declare the collapse interface in collapse.h

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

A collapse takes three calls:

  - collapse_control_init() - set up the control a caller carries;
  - collapse_scan_pmd() - scan one PTE table, under mmap_lock;
  - collapse_run_pmd() - collapse what the scan found, no mmap_lock.

All three are static in khugepaged.c, as are collapse_possible_orders(),
which says what a VMA allows, and the revalidate a caller needs once a
collapse has given the mmap_lock up.  No other file can ask for a collapse
without them.

Declare them in collapse.h, with a comment stating the order they are
called in and who holds the lock over each step.  Each function gets a
kerneldoc comment where it is defined: what it takes, what it does, and
the lock state on entry and exit.

hugepage_vma_revalidate() becomes collapse_vma_revalidate(): it is part of
what a collapse offers now, not a helper of the daemon.

Preparation for implementing MADV_COLLAPSE in madvise.c.

No functional change.

Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/collapse.h   | 45 ++++++++++++++++++++++++++
 mm/khugepaged.c | 86 ++++++++++++++++++++++++++++++++++++++++---------
 2 files changed, 115 insertions(+), 16 deletions(-)

diff --git a/mm/collapse.h b/mm/collapse.h
index ca7b367c89cb..b19351fbe19b 100644
--- a/mm/collapse.h
+++ b/mm/collapse.h
@@ -114,4 +114,49 @@ struct collapse_control {
 	pgoff_t scan_pgoff;
 };
 
+/* Which orders a VMA may collapse to, zero when it may not collapse at all */
+unsigned long collapse_possible_orders(struct vm_area_struct *vma,
+		vm_flags_t vm_flags, enum tva_type tva_flags);
+
+/*
+ * A caller states what it allows in cc->policy and then hands over one PTE
+ * table's worth of a VMA at a time:
+ *
+ *     collapse_control_init(cc)              once, before the first table
+ *     collapse_scan_pmd(vma, addr, ...)      per table
+ *     collapse_run_pmd(mm, addr, result, cc) when a scan found work
+ *
+ * The caller holds mmap_lock for reading over the scan and passes an address
+ * within @vma, aligned to the PTE table to scan.
+ *
+ * The scan returns with that lock still held.  Almost every table it is
+ * offered has nothing to collapse, so a caller walks a whole VMA under the one
+ * lock it took to get there.  SCAN_SUCCEED means there is
+ * something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE means the page cache
+ * already holds the PMD folio and only the PTE table is left to retract.
+ * Both are work for the run, which is handed what the scan returned; anything
+ * else is why there is nothing to do.
+ *
+ * The run is called without the lock and returns without it, taking what it
+ * needs in between: what it does -- allocate, isolate, copy, flush -- is slow
+ * enough that a writer would wait behind it.  The caller gives the lock up
+ * first, and with it @vma and anything derived under it, so a caller carrying
+ * on has to look up again with collapse_vma_revalidate().  The run revalidates
+ * for itself rather than trusting what the scan saw.
+ *
+ * A scan that found something has to be run: the file side takes a reference on
+ * the file while it still has the VMA to take it from, and the run is what
+ * gives it back.
+ */
+void collapse_control_init(struct collapse_control *cc);
+enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
+		unsigned long addr, struct collapse_control *cc,
+		unsigned long orders);
+enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr,
+		enum scan_result result, struct collapse_control *cc);
+enum scan_result collapse_vma_revalidate(struct mm_struct *mm,
+		unsigned long address, bool expect_anon,
+		struct vm_area_struct **vmap, struct collapse_control *cc,
+		unsigned int order);
+
 #endif	/* __MM_COLLAPSE_H */
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index ed74d3268698..f79ba6d1bf49 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -476,11 +476,18 @@ void __khugepaged_enter(struct mm_struct *mm)
 		wake_up_interruptible(&khugepaged_wait);
 }
 
-/*
- * Check what orders are possible based on the vma and collapse type.
- * This is used to determine if mTHP collapse is a viable option.
+/**
+ * collapse_possible_orders - which orders a VMA may collapse to
+ * @vma: the VMA
+ * @vm_flags: its flags, passed separately where they are about to change
+ * @tva_flags: who is asking, as thp_vma_allowable_orders() spells it
+ *
+ * khugepaged may collapse anonymous memory to any enabled order; everything
+ * else collapses to PMD order only.
+ *
+ * Return: the orders as a bitmask, zero when the VMA may not collapse at all.
  */
-static unsigned long collapse_possible_orders(struct vm_area_struct *vma,
+unsigned long collapse_possible_orders(struct vm_area_struct *vma,
 		vm_flags_t vm_flags, enum tva_type tva_flags)
 {
 	unsigned long orders;
@@ -1008,13 +1015,22 @@ static int collapse_find_target_node(struct collapse_control *cc)
 }
 #endif
 
-/*
- * If mmap_lock temporarily dropped, revalidate vma
- * after taking the mmap_lock again.
- * Returns enum scan_result value.
+/**
+ * collapse_vma_revalidate - look a VMA up again after mmap_lock was dropped
+ * @mm: the mm
+ * @address: an address within the PTE table being collapsed
+ * @expect_anon: the collapse started on an anonymous VMA
+ * @vmap: the VMA found, if any
+ * @cc: the control, for the policy that says who is asking
+ * @order: the order the collapse is going for
+ *
+ * Called with mmap_lock held, for reading or writing, once it has been given up
+ * and taken back.  The VMA has to span the whole PMD whatever @order is; with
+ * @expect_anon it also has to be anonymous and have an anon_vma.
+ *
+ * Return: SCAN_SUCCEED, or why a collapse of @order at @address is off.
  */
-
-static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
+enum scan_result collapse_vma_revalidate(struct mm_struct *mm, unsigned long address,
 		bool expect_anon, struct vm_area_struct **vmap,
 		struct collapse_control *cc, unsigned int order)
 {
@@ -1262,7 +1278,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm,
 	}
 
 	mmap_read_lock(mm);
-	result = hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true,
+	result = collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true,
 					 &vma, cc, order);
 	if (result != SCAN_SUCCEED) {
 		mmap_read_unlock(mm);
@@ -1297,7 +1313,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm,
 	 * mmap_lock.
 	 */
 	mmap_write_lock(mm);
-	result = hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true,
+	result = collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true,
 					 &vma, cc, order);
 	if (result != SCAN_SUCCEED)
 		goto out_up_write;
@@ -2739,13 +2755,36 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm,
 	return result;
 }
 
-static void collapse_control_init(struct collapse_control *cc)
+/**
+ * collapse_control_init - set up a control before its first scan
+ * @cc: the control the caller carries across its scans
+ *
+ * cc->policy is the caller's to fill.
+ */
+void collapse_control_init(struct collapse_control *cc)
 {
 	cc->progress = 0;
 	cc->scan_file = NULL;
 }
 
-static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
+/**
+ * collapse_scan_pmd - scan one PTE table for a collapse candidate
+ * @vma: the VMA the table belongs to
+ * @addr: start of the table, PMD aligned
+ * @cc: the caller's control
+ * @orders: the orders the caller allows for @vma
+ *
+ * Called with mmap_lock held for reading and returns with it still held.
+ * Almost every table it is offered has nothing to collapse, so a caller walks
+ * a whole VMA under the one lock it took to get there.
+ *
+ * Return: SCAN_SUCCEED when there is something to collapse;
+ * SCAN_PTE_MAPPED_HUGEPAGE when the page cache already holds the PMD folio and
+ * only the PTE table is left to retract.  Both are work for collapse_run_pmd(),
+ * which is handed what the scan returned.  Anything else is why there is
+ * nothing to do.
+ */
+enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
 		unsigned long addr, struct collapse_control *cc,
 		unsigned long orders)
 {
@@ -2780,7 +2819,22 @@ static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
 	return result;
 }
 
-static enum scan_result collapse_run_pmd(struct mm_struct *mm,
+/**
+ * collapse_run_pmd - collapse the table a scan found work in
+ * @mm: the mm
+ * @addr: start of the table, as given to the scan
+ * @result: what the scan returned
+ * @cc: the control the scan ran with
+ *
+ * Called without mmap_lock and returns without it, taking what it needs in
+ * between: what it does -- allocate, isolate, copy, flush -- is slow enough
+ * that a writer would wait behind it.  The caller gives the lock up first,
+ * and with it the VMA and anything derived under it.  The run revalidates for
+ * itself rather than trusting what the scan saw.
+ *
+ * Return: what the collapse made of the table.
+ */
+enum scan_result collapse_run_pmd(struct mm_struct *mm,
 		unsigned long addr, enum scan_result result,
 		struct collapse_control *cc)
 {
@@ -3230,7 +3284,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
 		if (!vma) {
 			cond_resched();
 			mmap_read_lock(mm);
-			result = hugepage_vma_revalidate(mm, addr, false, &found,
+			result = collapse_vma_revalidate(mm, addr, false, &found,
 							 cc, HPAGE_PMD_ORDER);
 			if (result != SCAN_SUCCEED) {
 				last_fail = result;
-- 
2.54.0

Re: [PATCH v4 12/13] mm/collapse: declare the collapse interface in collapse.h

[Zi Yan]

On Mon Sep 28, 2026 at 6:06 AM EDT, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
>
> A collapse takes three calls:
>
>   - collapse_control_init() - set up the control a caller carries;
>   - collapse_scan_pmd() - scan one PTE table, under mmap_lock;
>   - collapse_run_pmd() - collapse what the scan found, no mmap_lock.
>
> All three are static in khugepaged.c, as are collapse_possible_orders(),
> which says what a VMA allows, and the revalidate a caller needs once a
> collapse has given the mmap_lock up.  No other file can ask for a collapse
> without them.
>
> Declare them in collapse.h, with a comment stating the order they are
> called in and who holds the lock over each step.  Each function gets a
> kerneldoc comment where it is defined: what it takes, what it does, and
> the lock state on entry and exit.
>
> hugepage_vma_revalidate() becomes collapse_vma_revalidate(): it is part of
> what a collapse offers now, not a helper of the daemon.
>
> Preparation for implementing MADV_COLLAPSE in madvise.c.
>
> No functional change.
>
> Assisted-by: LLM
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  mm/collapse.h   | 45 ++++++++++++++++++++++++++
>  mm/khugepaged.c | 86 ++++++++++++++++++++++++++++++++++++++++---------
>  2 files changed, 115 insertions(+), 16 deletions(-)
>
<snip>

> + * lock it took to get there.  SCAN_SUCCEED means there is
> + * something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE means the page cache
> + * already holds the PMD folio and only the PTE table is left to retract.
> + * Both are work for the run, which is handed what the scan returned; anything
> + * else is why there is nothing to do.

These are implmentation details and already documented in the
collapse_scan_pmd()'s kdoc.

> + *
> + * The run is called without the lock and returns without it, taking what it
> + * needs in between: what it does -- allocate, isolate, copy, flush -- is slow
> + * enough that a writer would wait behind it.  The caller gives the lock up
> + * first, and with it @vma and anything derived under it, so a caller carrying
> + * on has to look up again with collapse_vma_revalidate().  The run revalidates
> + * for itself rather than trusting what the scan saw.


The run does ... would be sufficient, since the above is the same
contente in collapse_run_pmd()'s kdoc.

> + *
> + * A scan that found something has to be run: the file side takes a reference on
> + * the file while it still has the VMA to take it from, and the run is what
> + * gives it back.

This is a repeated comment in collapse_control. We only need one.


Otherwise, LGTM.

Acked-by: Zi Yan <ziy@nvidia.com>

-- 
Best Regards,
Yan, Zi

[PATCH v4 13/13] mm/collapse: implement MADV_COLLAPSE in madvise.c

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

MADV_COLLAPSE is a madvise operation, but its implementation sat in
khugepaged.c.  The daemon's file therefore also held a syscall's worth of
code that has nothing to do with the daemon: the walk over the user's
range, the per-PMD loop, and the errno translation.

Move it to madvise.c, among the operations it belongs with, along with the
errno map and the policy it states for itself.  It takes a struct
madvise_behavior like every one of those operations, which is where the
range, the VMA and the lock-dropped flag it used to be handed separately
already live.

It stays a caller of the interface khugepaged uses, so nothing about the
collapse changes.

The !CONFIG_TRANSPARENT_HUGEPAGE stub moves in with it.

Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 include/linux/huge_mm.h |   9 ---
 mm/khugepaged.c         | 159 +------------------------------------
 mm/madvise.c            | 170 +++++++++++++++++++++++++++++++++++++++-
 3 files changed, 170 insertions(+), 168 deletions(-)

diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index c745f7ad2298..8ca0fa3be2ac 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -510,8 +510,6 @@ change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
 
 int hugepage_madvise(struct vm_area_struct *vma, vm_flags_t *vm_flags,
 		     int advice);
-int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
-		     unsigned long end, bool *lock_dropped);
 void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start,
 			   unsigned long end, struct vm_area_struct *next);
 spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma);
@@ -715,13 +713,6 @@ static inline int hugepage_madvise(struct vm_area_struct *vma,
 	return -EINVAL;
 }
 
-static inline int madvise_collapse(struct vm_area_struct *vma,
-				   unsigned long start,
-				   unsigned long end, bool *lock_dropped)
-{
-	return -EINVAL;
-}
-
 static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
 					 unsigned long start,
 					 unsigned long end,
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f79ba6d1bf49..3870d096bb86 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -972,23 +972,6 @@ static void collapse_policy_khugepaged(struct collapse_policy *p)
 	p->tva_type = TVA_KHUGEPAGED;
 }
 
-/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */
-static void collapse_policy_madvise(struct collapse_policy *p)
-{
-	p->pmd.max_ptes_none = HPAGE_PMD_NR;
-	p->pmd.max_ptes_swap = HPAGE_PMD_NR;
-	p->pmd.max_ptes_shared = HPAGE_PMD_NR;
-	/* Never read: MADV_COLLAPSE collapses to PMD order only */
-	p->sub_pmd = p->pmd;
-
-	p->anon_skip_lazyfree = false;
-	p->anon_require_referenced = false;
-	p->file_install_pmd = true;
-	p->file_writeback_dirty = true;
-	p->gfp = GFP_TRANSHUGE;
-	p->tva_type = TVA_FORCED_COLLAPSE;
-}
-
 #ifdef CONFIG_NUMA
 static int collapse_find_target_node(struct collapse_control *cc)
 {
@@ -2834,9 +2817,8 @@ enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
  *
  * Return: what the collapse made of the table.
  */
-enum scan_result collapse_run_pmd(struct mm_struct *mm,
-		unsigned long addr, enum scan_result result,
-		struct collapse_control *cc)
+enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr,
+		enum scan_result result, struct collapse_control *cc)
 {
 	struct file *file = cc->scan_file;
 	bool triggered_wb = false;
@@ -3207,140 +3189,3 @@ bool current_is_khugepaged(void)
 {
 	return kthread_func(current) == khugepaged;
 }
-
-static int madvise_collapse_errno(enum scan_result r)
-{
-	/*
-	 * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide
-	 * actionable feedback to caller, so they may take an appropriate
-	 * fallback measure depending on the nature of the failure.
-	 */
-	switch (r) {
-	case SCAN_ALLOC_HUGE_PAGE_FAIL:
-		return -ENOMEM;
-	case SCAN_CGROUP_CHARGE_FAIL:
-	case SCAN_EXCEED_NONE_PTE:
-		return -EBUSY;
-	/* Resource temporary unavailable - trying again might succeed */
-	case SCAN_PAGE_COUNT:
-	case SCAN_PAGE_LOCK:
-	case SCAN_PAGE_LRU:
-	case SCAN_DEL_PAGE_LRU:
-	case SCAN_PAGE_FILLED:
-	case SCAN_PAGE_HAS_PRIVATE:
-	case SCAN_PAGE_DIRTY_OR_WRITEBACK:
-		return -EAGAIN;
-	/*
-	 * Other: Trying again likely not to succeed / error intrinsic to
-	 * specified memory range. khugepaged likely won't be able to collapse
-	 * either.
-	 */
-	default:
-		return -EINVAL;
-	}
-}
-
-int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
-		     unsigned long end, bool *lock_dropped)
-{
-	struct collapse_control *cc;
-	struct mm_struct *mm = vma->vm_mm;
-	unsigned long hstart, hend, addr, orders;
-	enum scan_result last_fail = SCAN_FAIL;
-	int thps = 0;
-
-	BUG_ON(vma->vm_start > start);
-	BUG_ON(vma->vm_end < end);
-
-	orders = collapse_possible_orders(vma, vma->vm_flags,
-					  TVA_FORCED_COLLAPSE);
-	if (!orders)
-		return -EINVAL;
-
-	hstart = ALIGN(start, HPAGE_PMD_SIZE);
-	hend = ALIGN_DOWN(end, HPAGE_PMD_SIZE);
-
-	if (hstart >= hend)
-		return 0;
-
-	cc = kmalloc_obj(*cc);
-	if (!cc)
-		return -ENOMEM;
-	collapse_control_init(cc);
-	collapse_policy_madvise(&cc->policy);
-
-	lru_add_drain_all();
-
-	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
-		struct vm_area_struct *found;
-		enum scan_result result;
-
-		/*
-		 * A collapse gives the lock up, so the VMA has to be found
-		 * again after one: it can shrink while nothing is held.  A scan
-		 * that finds nothing to collapse leaves the lock alone, so a
-		 * range that is already collapsed walks on without relocking.
-		 */
-		if (!vma) {
-			cond_resched();
-			mmap_read_lock(mm);
-			result = collapse_vma_revalidate(mm, addr, false, &found,
-							 cc, HPAGE_PMD_ORDER);
-			if (result != SCAN_SUCCEED) {
-				last_fail = result;
-				goto out_locked;
-			}
-			vma = found;
-			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
-			orders = collapse_possible_orders(vma, vma->vm_flags,
-							  TVA_FORCED_COLLAPSE);
-		}
-
-		result = collapse_scan_pmd(vma, addr, cc, orders);
-		/* Nothing to do here, and the lock is still ours */
-		if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
-			goto tally;
-
-		/* The collapse takes its own locks, so give this up */
-		mmap_read_unlock(mm);
-		*lock_dropped = true;
-		vma = NULL;
-
-		result = collapse_run_pmd(mm, addr, result, cc);
-tally:
-		switch (result) {
-		case SCAN_SUCCEED:
-		case SCAN_PMD_MAPPED:
-			++thps;
-			break;
-		/* Whitelisted set of results where continuing OK */
-		case SCAN_NO_PTE_TABLE:
-		case SCAN_PTE_NON_PRESENT:
-		case SCAN_PTE_UFFD:
-		case SCAN_LACK_REFERENCED_PAGE:
-		case SCAN_PAGE_NULL:
-		case SCAN_PAGE_COUNT:
-		case SCAN_PAGE_LOCK:
-		case SCAN_PAGE_COMPOUND:
-		case SCAN_PAGE_LRU:
-		case SCAN_DEL_PAGE_LRU:
-			last_fail = result;
-			break;
-		default:
-			last_fail = result;
-			/* Other error, exit */
-			goto out;
-		}
-	}
-
-out:
-	/* Caller expects us to hold mmap_lock on return */
-	if (!vma)
-		mmap_read_lock(mm);
-out_locked:
-	mmap_assert_locked(mm);
-	kfree(cc);
-
-	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
-			: madvise_collapse_errno(last_fail);
-}
diff --git a/mm/madvise.c b/mm/madvise.c
index 963337f93a7a..acb5215e2f07 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -38,6 +38,7 @@
 
 #include "internal.h"
 #include "swap.h"
+#include "collapse.h"
 
 #define __MADV_SET_ANON_VMA_NAME (-1)
 
@@ -906,6 +907,172 @@ bool madvise_dontneed_free_valid_vma(struct madvise_behavior *madv_behavior)
 	return true;
 }
 
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+
+/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */
+static void collapse_policy_madvise(struct collapse_policy *p)
+{
+	p->pmd.max_ptes_none = HPAGE_PMD_NR;
+	p->pmd.max_ptes_swap = HPAGE_PMD_NR;
+	p->pmd.max_ptes_shared = HPAGE_PMD_NR;
+	/* Never read: MADV_COLLAPSE collapses to PMD order only */
+	p->sub_pmd = p->pmd;
+
+	p->anon_skip_lazyfree = false;
+	p->anon_require_referenced = false;
+	p->file_install_pmd = true;
+	p->file_writeback_dirty = true;
+	p->gfp = GFP_TRANSHUGE;
+	p->tva_type = TVA_FORCED_COLLAPSE;
+}
+
+static int madvise_collapse_errno(enum scan_result r)
+{
+	/*
+	 * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide
+	 * actionable feedback to caller, so they may take an appropriate
+	 * fallback measure depending on the nature of the failure.
+	 */
+	switch (r) {
+	case SCAN_ALLOC_HUGE_PAGE_FAIL:
+		return -ENOMEM;
+	case SCAN_CGROUP_CHARGE_FAIL:
+	case SCAN_EXCEED_NONE_PTE:
+		return -EBUSY;
+	/* Resource temporary unavailable - trying again might succeed */
+	case SCAN_PAGE_COUNT:
+	case SCAN_PAGE_LOCK:
+	case SCAN_PAGE_LRU:
+	case SCAN_DEL_PAGE_LRU:
+	case SCAN_PAGE_FILLED:
+	case SCAN_PAGE_HAS_PRIVATE:
+	case SCAN_PAGE_DIRTY_OR_WRITEBACK:
+		return -EAGAIN;
+	/*
+	 * Other: Trying again likely not to succeed / error intrinsic to
+	 * specified memory range. khugepaged likely won't be able to collapse
+	 * either.
+	 */
+	default:
+		return -EINVAL;
+	}
+}
+
+static int madvise_collapse(struct madvise_behavior *madv_behavior)
+{
+	struct madvise_behavior_range *range = &madv_behavior->range;
+	struct vm_area_struct *vma = madv_behavior->vma;
+	struct mm_struct *mm = madv_behavior->mm;
+	struct collapse_control *cc;
+	unsigned long hstart, hend, addr, orders;
+	enum scan_result last_fail = SCAN_FAIL;
+	int thps = 0;
+
+	BUG_ON(vma->vm_start > range->start);
+	BUG_ON(vma->vm_end < range->end);
+
+	orders = collapse_possible_orders(vma, vma->vm_flags,
+					  TVA_FORCED_COLLAPSE);
+	if (!orders)
+		return -EINVAL;
+
+	hstart = ALIGN(range->start, HPAGE_PMD_SIZE);
+	hend = ALIGN_DOWN(range->end, HPAGE_PMD_SIZE);
+
+	if (hstart >= hend)
+		return 0;
+
+	cc = kmalloc_obj(*cc);
+	if (!cc)
+		return -ENOMEM;
+	collapse_control_init(cc);
+	collapse_policy_madvise(&cc->policy);
+
+	lru_add_drain_all();
+
+	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
+		struct vm_area_struct *found;
+		enum scan_result result;
+
+		/*
+		 * A collapse gives the lock up, so the VMA has to be found
+		 * again after one: it can shrink while nothing is held.  A scan
+		 * that finds nothing to collapse leaves the lock alone, so a
+		 * range that is already collapsed walks on without relocking.
+		 */
+		if (!vma) {
+			cond_resched();
+			mmap_read_lock(mm);
+			result = collapse_vma_revalidate(mm, addr, false, &found,
+							 cc, HPAGE_PMD_ORDER);
+			if (result != SCAN_SUCCEED) {
+				last_fail = result;
+				goto out_locked;
+			}
+			vma = found;
+			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
+			orders = collapse_possible_orders(vma, vma->vm_flags,
+							  TVA_FORCED_COLLAPSE);
+		}
+
+		result = collapse_scan_pmd(vma, addr, cc, orders);
+		/* Nothing to do here, and the lock is still ours */
+		if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE)
+			goto tally;
+
+		/* The collapse takes its own locks, so give this up */
+		mmap_read_unlock(mm);
+		mark_mmap_lock_dropped(madv_behavior);
+		vma = NULL;
+
+		result = collapse_run_pmd(mm, addr, result, cc);
+tally:
+		switch (result) {
+		case SCAN_SUCCEED:
+		case SCAN_PMD_MAPPED:
+			++thps;
+			break;
+		/* Whitelisted set of results where continuing OK */
+		case SCAN_NO_PTE_TABLE:
+		case SCAN_PTE_NON_PRESENT:
+		case SCAN_PTE_UFFD:
+		case SCAN_LACK_REFERENCED_PAGE:
+		case SCAN_PAGE_NULL:
+		case SCAN_PAGE_COUNT:
+		case SCAN_PAGE_LOCK:
+		case SCAN_PAGE_COMPOUND:
+		case SCAN_PAGE_LRU:
+		case SCAN_DEL_PAGE_LRU:
+			last_fail = result;
+			break;
+		default:
+			last_fail = result;
+			/* Other error, exit */
+			goto out;
+		}
+	}
+
+out:
+	/* Caller expects us to hold mmap_lock on return */
+	if (!vma)
+		mmap_read_lock(mm);
+out_locked:
+	mmap_assert_locked(mm);
+	kfree(cc);
+
+	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
+			: madvise_collapse_errno(last_fail);
+}
+
+#else	/* CONFIG_TRANSPARENT_HUGEPAGE */
+
+static int madvise_collapse(struct madvise_behavior *madv_behavior)
+{
+	return -EINVAL;
+}
+
+#endif	/* CONFIG_TRANSPARENT_HUGEPAGE */
+
 static long madvise_dontneed_free(struct madvise_behavior *madv_behavior)
 {
 	struct mm_struct *mm = madv_behavior->mm;
@@ -1373,8 +1540,7 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior)
 	case MADV_DONTNEED_LOCKED:
 		return madvise_dontneed_free(madv_behavior);
 	case MADV_COLLAPSE:
-		return madvise_collapse(vma, range->start, range->end,
-			&madv_behavior->lock_dropped);
+		return madvise_collapse(madv_behavior);
 	case MADV_GUARD_INSTALL:
 		return madvise_guard_install(madv_behavior);
 	case MADV_GUARD_REMOVE:
-- 
2.54.0

Re: [PATCH v4 13/13] mm/collapse: implement MADV_COLLAPSE in madvise.c

[Zi Yan]

On Mon Sep 28, 2026 at 6:06 AM EDT, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
>
> MADV_COLLAPSE is a madvise operation, but its implementation sat in
> khugepaged.c.  The daemon's file therefore also held a syscall's worth of
> code that has nothing to do with the daemon: the walk over the user's
> range, the per-PMD loop, and the errno translation.
>
> Move it to madvise.c, among the operations it belongs with, along with the
> errno map and the policy it states for itself.  It takes a struct
> madvise_behavior like every one of those operations, which is where the
> range, the VMA and the lock-dropped flag it used to be handed separately
> already live.
>
> It stays a caller of the interface khugepaged uses, so nothing about the
> collapse changes.
>
> The !CONFIG_TRANSPARENT_HUGEPAGE stub moves in with it.
>
> Assisted-by: LLM
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  include/linux/huge_mm.h |   9 ---
>  mm/khugepaged.c         | 159 +------------------------------------
>  mm/madvise.c            | 170 +++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 170 insertions(+), 168 deletions(-)
>

LGTM.

Acked-by: Zi Yan <ziy@nvidia.com>


-- 
Best Regards,
Yan, Zi

Re: [PATCH v4 00/13] mm/collapse: separate a collapse from its callers

[Andrew Morton]

On Mon, 28 Sep 2026 11:06:14 +0100 Kiryl Shutsemau <kirill@shutemov.name> wrote:

> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> 
> [ This is the first of the cleanups I said I would front-load ]
> 
> There is no line between the collapse engine and the callers that ask for
> a collapse.  khugepaged.c holds both, and they reach into each other.
> 
>  - Sixteen tests through the collapse path read cc->is_khugepaged to work
>    out what they are allowed to do, when every one of those decisions was
>    made by the caller before it asked.
> 
>  - collapse_single_pmd() does both halves of a collapse behind one call and
>    drops mmap_lock somewhere in the middle.  Which of its paths dropped it
>    is not something a caller can see, so it hands back a bool and the
>    caller keeps track.
> 
>  - MADV_COLLAPSE's implementation -- the walk over the user's range, the
>    per-PMD loop, the errno translation -- sits in khugepaged.c, which is
>    the daemon's file.
> 
> So: draw the line.  State what a caller allows in a policy, split the call
> in two with the lock as the boundary, and move the syscall to madvise.c.
> What the engine offers is then three calls, with the lock state written
> down against each, and a policy the caller fills for itself:
> 
>     collapse_control_init(cc)         once, before the first table
>     collapse_policy_*(&cc->policy)    what this caller allows
>     collapse_scan_pmd(vma, addr, ...) per table, under mmap_lock
>     collapse_run_pmd(mm, addr, ...)   when a scan found work, no mmap_lock
> 
> The engine stays in khugepaged.c for now; what changes is that it has an
> interface, and that neither half has to ask about the other.  madvise.c
> gains the operation it should have had all along.

Thanks, I updated mm-unstable to this version.

I added a note in order to track David's v3 comment at
https://lore.kernel.org/abd32250-6ba1-443b-b8e4-7772bcdd325a@kernel.org

> Changes since v3
> ================
> 
>   https://lore.kernel.org/all/20260916093145.4022188-1-kirill@shutemov.name/
> 
>  - Patch 5: the PTE limits come as two sets, struct collapse_limits for a
>    PMD-sized window and one for anything smaller, and the strict_sub_pmd
>    flag goes.  The warning for a max_ptes_none between 0 and the maximum
>    moves to where khugepaged fills its policy.  The fields only one side
>    reads are prefixed anon_ or file_.  collapse_policy_forced() is
>    collapse_policy_madvise() (David).
> 
>  - Patch 8 is new: collapse_file() is called from collapse_single_pmd()
>    rather than from inside collapse_scan_file(), so the file scan stops at
>    the decision before the scan/run split.  The writeback retry and the
>    tracepoint change move into it, so patch 9 is only the split (David).
> 
>  - Patch 9: no collapse_control_release().  It had nothing to release in
>    this series; it comes back with the engine, where init allocates.  The
>    check that the last scan was run stays at the top of collapse_scan_pmd()
>    (David).
> 
>  - Patch 10: the changelog says why the lock boundary belongs to the
>    caller (David).
> 
>  - Patch 11: both callers pass their TVA type as the literal it is rather
>    than reading it back from the policy (David).
> 
>  - Patch 12: kerneldoc on every function collapse.h declares (David).
> 
>  - Acked-by from David Hildenbrand on 1-4, 6 and 7; Reviewed-by from
>    Baolin Wang on 10.  Reviewed-by dropped from 5 and 9, which changed.

Here's how v4 altered mm.git.  It's substantial:


 mm/collapse.h   |   38 +++++---
 mm/khugepaged.c |  199 +++++++++++++++++++++++-----------------------
 mm/madvise.c    |   25 +++--
 3 files changed, 139 insertions(+), 123 deletions(-)

--- a/mm/collapse.h~b
+++ a/mm/collapse.h
@@ -45,27 +45,37 @@ enum scan_result {
 	SCAN_PAGE_DIRTY_OR_WRITEBACK,
 };
 
-/* What a collapse is allowed to do, decided by the caller that asks for it */
-struct collapse_policy {
-	/* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */
+/* How many PTEs of a window may be missing, swapped out or shared */
+struct collapse_limits {
+	/* Counted over a PMD-sized window; HPAGE_PMD_NR means "no limit" */
 	unsigned int max_ptes_none;
 	unsigned int max_ptes_swap;
 	unsigned int max_ptes_shared;
+};
+
+/* What a collapse is allowed to do, decided by the caller that asks for it */
+struct collapse_policy {
+	/* Limits for a PMD-sized window */
+	struct collapse_limits pmd;
 
-	/* Take no swapped-out or shared PTE into a sub-PMD collapse */
-	bool strict_sub_pmd;
+	/*
+	 * Limits for a smaller window.  Its max_ptes_none is either 0 or
+	 * COLLAPSE_MAX_PTES_LIMIT, the latter meaning all but one PTE of the
+	 * window whatever its order; any other value counts as 0.
+	 */
+	struct collapse_limits sub_pmd;
 
 	/* Leave clean lazyfree folios to reclaim rather than collapse them */
-	bool skip_lazyfree;
+	bool anon_skip_lazyfree;
 
-	/* Refuse a range with no sign of use */
-	bool require_referenced;
+	/* Refuse an anonymous range with no sign of use */
+	bool anon_require_referenced;
 
 	/* Map the PMD over a file collapse instead of leaving it to a fault */
-	bool install_pmd;
+	bool file_install_pmd;
 
 	/* Write dirty pages back and retry once instead of refusing them */
-	bool writeback_dirty;
+	bool file_writeback_dirty;
 
 	/* How hard to try for a destination folio */
 	gfp_t gfp;
@@ -115,14 +125,13 @@ unsigned long collapse_possible_orders(s
  *     collapse_control_init(cc)              once, before the first table
  *     collapse_scan_pmd(vma, addr, ...)      per table
  *     collapse_run_pmd(mm, addr, result, cc) when a scan found work
- *     collapse_control_release(cc)           once, when done with the control
  *
  * The caller holds mmap_lock for reading over the scan and passes an address
  * within @vma, aligned to the PTE table to scan.
  *
- * The scan returns with that lock still held.  It only reads, and almost every
- * table it is offered has nothing to collapse, so a caller walks a whole VMA
- * under the one lock it took to get there.  SCAN_SUCCEED means there is
+ * The scan returns with that lock still held.  Almost every table it is
+ * offered has nothing to collapse, so a caller walks a whole VMA under the one
+ * lock it took to get there.  SCAN_SUCCEED means there is
  * something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE means the page cache
  * already holds the PMD folio and only the PTE table is left to retract.
  * Both are work for the run, which is handed what the scan returned; anything
@@ -140,7 +149,6 @@ unsigned long collapse_possible_orders(s
  * gives it back.
  */
 void collapse_control_init(struct collapse_control *cc);
-void collapse_control_release(struct collapse_control *cc);
 enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
 		unsigned long addr, struct collapse_control *cc,
 		unsigned long orders);
--- a/mm/khugepaged.c~b
+++ a/mm/khugepaged.c
@@ -314,27 +314,17 @@ static bool pte_none_or_zero(pte_t pte)
 static unsigned int collapse_max_ptes_none(struct collapse_control *cc,
 		struct vm_area_struct *vma, unsigned int order)
 {
-	const unsigned int max_ptes_none = cc->policy.max_ptes_none;
+	unsigned int max_ptes_none;
 
 	if (vma && userfaultfd_armed(vma))
 		return 0;
-	/* The limit as given, at the PMD order and wherever it is not capped */
-	if (is_pmd_order(order) || !cc->policy.strict_sub_pmd)
-		return max_ptes_none;
-	/*
-	 * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT,
-	 * scale the maximum number of PTEs to the order of the collapse.
-	 */
+	if (is_pmd_order(order))
+		return cc->policy.pmd.max_ptes_none;
+
+	/* Below PMD order: all but one PTE of the window, or none */
+	max_ptes_none = cc->policy.sub_pmd.max_ptes_none;
 	if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT)
 		return (1 << order) - 1;
-	/*
-	 * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT,
-	 * emit a warning and return 0.
-	 */
-	if (max_ptes_none)
-		pr_warn_once("mTHP collapse does not support max_ptes_none"
-		     " values other than 0 or %u, defaulting to 0.\n",
-		     COLLAPSE_MAX_PTES_LIMIT);
 	return 0;
 }
 
@@ -350,13 +340,9 @@ static unsigned int collapse_max_ptes_no
 static unsigned int collapse_max_ptes_shared(struct collapse_control *cc,
 		unsigned int order)
 {
-	/*
-	 * A sub-PMD window held to the strict rule takes no shared page at all:
-	 * an mTHP is not worth the CoW-breaking.
-	 */
-	if (!is_pmd_order(order) && cc->policy.strict_sub_pmd)
-		return 0;
-	return cc->policy.max_ptes_shared;
+	if (is_pmd_order(order))
+		return cc->policy.pmd.max_ptes_shared;
+	return cc->policy.sub_pmd.max_ptes_shared;
 }
 
 /**
@@ -371,13 +357,9 @@ static unsigned int collapse_max_ptes_sh
 static unsigned int collapse_max_ptes_swap(struct collapse_control *cc,
 		unsigned int order)
 {
-	/*
-	 * A sub-PMD window held to the strict rule takes nothing non-present:
-	 * reading pages back to build an mTHP is not worth the latency.
-	 */
-	if (!is_pmd_order(order) && cc->policy.strict_sub_pmd)
-		return 0;
-	return cc->policy.max_ptes_swap;
+	if (is_pmd_order(order))
+		return cc->policy.pmd.max_ptes_swap;
+	return cc->policy.sub_pmd.max_ptes_swap;
 }
 
 int hugepage_madvise(struct vm_area_struct *vma,
@@ -494,9 +476,16 @@ void __khugepaged_enter(struct mm_struct
 		wake_up_interruptible(&khugepaged_wait);
 }
 
-/*
- * Check what orders are possible based on the vma and collapse type.
- * This is used to determine if mTHP collapse is a viable option.
+/**
+ * collapse_possible_orders - which orders a VMA may collapse to
+ * @vma: the VMA
+ * @vm_flags: its flags, passed separately where they are about to change
+ * @tva_flags: who is asking, as thp_vma_allowable_orders() spells it
+ *
+ * khugepaged may collapse anonymous memory to any enabled order; everything
+ * else collapses to PMD order only.
+ *
+ * Return: the orders as a bitmask, zero when the VMA may not collapse at all.
  */
 unsigned long collapse_possible_orders(struct vm_area_struct *vma,
 		vm_flags_t vm_flags, enum tva_type tva_flags)
@@ -658,7 +647,7 @@ static enum scan_result __collapse_huge_
 		 * If the vma has the VM_DROPPABLE flag, the collapse will
 		 * preserve the lazyfree property without needing to skip.
 		 */
-		if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
+		if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
 		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
 			result = SCAN_PAGE_LAZYFREE;
 			goto out;
@@ -747,12 +736,12 @@ static enum scan_result __collapse_huge_
 		if (folio_test_large(folio))
 			list_add_tail(&folio->lru, compound_pagelist);
 next:
-		if (cc->policy.require_referenced &&
+		if (cc->policy.anon_require_referenced &&
 		    folio_pte_referenced(folio, vma, addr, pteval))
 			referenced++;
 	}
 
-	if (unlikely(cc->policy.require_referenced && !referenced)) {
+	if (unlikely(cc->policy.anon_require_referenced && !referenced)) {
 		result = SCAN_LACK_REFERENCED_PAGE;
 	} else {
 		result = SCAN_SUCCEED;
@@ -957,14 +946,28 @@ static inline gfp_t alloc_hugepage_khuge
 /* khugepaged collapses on its own initiative, so it obeys its own settings */
 static void collapse_policy_khugepaged(struct collapse_policy *p)
 {
-	p->max_ptes_none = READ_ONCE(khugepaged_max_ptes_none);
-	p->max_ptes_swap = READ_ONCE(khugepaged_max_ptes_swap);
-	p->max_ptes_shared = READ_ONCE(khugepaged_max_ptes_shared);
-	p->strict_sub_pmd = true;
-	p->skip_lazyfree = true;
-	p->require_referenced = true;
-	p->install_pmd = false;
-	p->writeback_dirty = false;
+	p->pmd.max_ptes_none = READ_ONCE(khugepaged_max_ptes_none);
+	p->pmd.max_ptes_swap = READ_ONCE(khugepaged_max_ptes_swap);
+	p->pmd.max_ptes_shared = READ_ONCE(khugepaged_max_ptes_shared);
+
+	/*
+	 * A sub-PMD window takes no swapped-out and no shared PTE: reading
+	 * pages back or breaking CoW is not worth it for an mTHP.  Empty PTEs
+	 * it takes all or nothing, since anything in between would let one
+	 * collapse feed the next.
+	 */
+	p->sub_pmd.max_ptes_none = p->pmd.max_ptes_none;
+	if (p->sub_pmd.max_ptes_none &&
+	    p->sub_pmd.max_ptes_none != COLLAPSE_MAX_PTES_LIMIT)
+		pr_warn_once("mTHP collapse does not support max_ptes_none values other than 0 or %u, defaulting to 0.\n",
+			     COLLAPSE_MAX_PTES_LIMIT);
+	p->sub_pmd.max_ptes_swap = 0;
+	p->sub_pmd.max_ptes_shared = 0;
+
+	p->anon_skip_lazyfree = true;
+	p->anon_require_referenced = true;
+	p->file_install_pmd = false;
+	p->file_writeback_dirty = false;
 	p->gfp = alloc_hugepage_khugepaged_gfpmask();
 	p->tva_type = TVA_KHUGEPAGED;
 }
@@ -995,11 +998,20 @@ static int collapse_find_target_node(str
 }
 #endif
 
-/*
- * Find the VMA at @address again once mmap_lock has been given up and taken
- * back, and check it still allows a collapse of @order there.  The VMA has to
- * span the whole PMD whatever @order is; with @expect_anon it also has to be
- * anonymous and have an anon_vma.  *@vmap is the VMA found, if any.
+/**
+ * collapse_vma_revalidate - look a VMA up again after mmap_lock was dropped
+ * @mm: the mm
+ * @address: an address within the PTE table being collapsed
+ * @expect_anon: the collapse started on an anonymous VMA
+ * @vmap: the VMA found, if any
+ * @cc: the control, for the policy that says who is asking
+ * @order: the order the collapse is going for
+ *
+ * Called with mmap_lock held, for reading or writing, once it has been given up
+ * and taken back.  The VMA has to span the whole PMD whatever @order is; with
+ * @expect_anon it also has to be anonymous and have an anon_vma.
+ *
+ * Return: SCAN_SUCCEED, or why a collapse of @order at @address is off.
  */
 enum scan_result collapse_vma_revalidate(struct mm_struct *mm, unsigned long address,
 		bool expect_anon, struct vm_area_struct **vmap,
@@ -1667,7 +1679,7 @@ static enum scan_result collapse_scan_an
 		 * If the vma has the VM_DROPPABLE flag, the collapse will
 		 * preserve the lazyfree property without needing to skip.
 		 */
-		if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
+		if (cc->policy.anon_skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) &&
 		    folio_test_lazyfree(folio) && !pte_dirty(pteval)) {
 			result = SCAN_PAGE_LAZYFREE;
 			failed_pfn = folio_pfn(folio);
@@ -1733,11 +1745,11 @@ static enum scan_result collapse_scan_an
 			goto out_unmap;
 		}
 
-		if (cc->policy.require_referenced &&
+		if (cc->policy.anon_require_referenced &&
 		    folio_pte_referenced(folio, vma, addr, pteval))
 			referenced++;
 	}
-	if (cc->policy.require_referenced &&
+	if (cc->policy.anon_require_referenced &&
 	    (!referenced ||
 	     (unmapped && referenced < HPAGE_PMD_NR / 2))) {
 		result = SCAN_LACK_REFERENCED_PAGE;
@@ -2592,7 +2604,7 @@ immap_locked:
 	 * caller that wants the PMD mapped now is told to go and do that.
 	 */
 	retract_page_tables(mapping, start);
-	if (cc->policy.install_pmd)
+	if (cc->policy.file_install_pmd)
 		result = SCAN_PTE_MAPPED_HUGEPAGE;
 	folio_unlock(new_folio);
 
@@ -2749,44 +2761,34 @@ static enum scan_result collapse_scan_fi
 	return result;
 }
 
-/* Set up a control before its first scan; cc->policy is the caller's to fill */
+/**
+ * collapse_control_init - set up a control before its first scan
+ * @cc: the control the caller carries across its scans
+ *
+ * cc->policy is the caller's to fill.
+ */
 void collapse_control_init(struct collapse_control *cc)
 {
 	cc->progress = 0;
 	cc->scan_file = NULL;
 }
 
-/* A scan that took a file reference should have been run */
-static void collapse_put_scan_file(struct collapse_control *cc)
-{
-	if (WARN_ON_ONCE(cc->scan_file)) {
-		fput(cc->scan_file);
-		cc->scan_file = NULL;
-	}
-}
-
-/*
- * Done with a control.  A scan that found something has to have been run by
- * then: the file side takes a reference on the file while it still has the
- * VMA to take it from, and the run is what gives it back.
- */
-void collapse_control_release(struct collapse_control *cc)
-{
-	collapse_put_scan_file(cc);
-}
-
-/*
- * Scan the PTE table of @vma at @addr for a collapse candidate.  @addr is
- * aligned to the table; @orders is what the caller allows there.
+/**
+ * collapse_scan_pmd - scan one PTE table for a collapse candidate
+ * @vma: the VMA the table belongs to
+ * @addr: start of the table, PMD aligned
+ * @cc: the caller's control
+ * @orders: the orders the caller allows for @vma
  *
- * Called with mmap_lock held for reading and returns with it still held.  It
- * only reads, and almost every table it is offered has nothing to collapse,
- * so a caller walks a whole VMA under the one lock it took to get there.
- *
- * SCAN_SUCCEED means there is something to collapse.  SCAN_PTE_MAPPED_HUGEPAGE
- * means the page cache already holds the PMD folio and only the PTE table is
- * left to retract.  Both are work for collapse_run_pmd(), which is handed
- * what the scan returned; anything else is why there is nothing to do.
+ * Called with mmap_lock held for reading and returns with it still held.
+ * Almost every table it is offered has nothing to collapse, so a caller walks
+ * a whole VMA under the one lock it took to get there.
+ *
+ * Return: SCAN_SUCCEED when there is something to collapse;
+ * SCAN_PTE_MAPPED_HUGEPAGE when the page cache already holds the PMD folio and
+ * only the PTE table is left to retract.  Both are work for collapse_run_pmd(),
+ * which is handed what the scan returned.  Anything else is why there is
+ * nothing to do.
  */
 enum scan_result collapse_scan_pmd(struct vm_area_struct *vma,
 		unsigned long addr, struct collapse_control *cc,
@@ -2797,7 +2799,10 @@ enum scan_result collapse_scan_pmd(struc
 
 	mmap_assert_locked(vma->vm_mm);
 	/* Whatever the last scan found has to have been run by now */
-	collapse_put_scan_file(cc);
+	if (WARN_ON_ONCE(cc->scan_file)) {
+		fput(cc->scan_file);
+		cc->scan_file = NULL;
+	}
 
 	if (vma_is_anonymous(vma))
 		return collapse_scan_anon_pmd(vma, addr, cc, orders);
@@ -2820,15 +2825,20 @@ enum scan_result collapse_scan_pmd(struc
 	return result;
 }
 
-/*
- * Collapse the table a scan found work in.  @result is what the scan
- * returned.
+/**
+ * collapse_run_pmd - collapse the table a scan found work in
+ * @mm: the mm
+ * @addr: start of the table, as given to the scan
+ * @result: what the scan returned
+ * @cc: the control the scan ran with
  *
  * Called without mmap_lock and returns without it, taking what it needs in
  * between: what it does -- allocate, isolate, copy, flush -- is slow enough
  * that a writer would wait behind it.  The caller gives the lock up first,
  * and with it the VMA and anything derived under it.  The run revalidates for
  * itself rather than trusting what the scan saw.
+ *
+ * Return: what the collapse made of the table.
  */
 enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr,
 		enum scan_result result, struct collapse_control *cc)
@@ -2845,12 +2855,12 @@ enum scan_result collapse_run_pmd(struct
 
 	/* The scan found the PMD folio in place: nothing to collapse */
 	if (result == SCAN_PTE_MAPPED_HUGEPAGE)
-		goto retract;
+		goto put;
 retry:
 	result = collapse_file(mm, addr, file, pgoff, cc);
 
 	/* Dirty pages are worth a writeback and one more try, if asked for */
-	if (cc->policy.writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
+	if (cc->policy.file_writeback_dirty && result == SCAN_PAGE_DIRTY_OR_WRITEBACK &&
 	    !triggered_wb && mapping_can_writeback(file->f_mapping)) {
 		const loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
 		const loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
@@ -2859,7 +2869,7 @@ retry:
 		triggered_wb = true;
 		goto retry;
 	}
-retract:
+put:
 	fput(file);
 
 	/*
@@ -2872,7 +2882,7 @@ retract:
 			result = SCAN_ANY_PROCESS;
 		else
 			result = try_collapse_pte_mapped_thp(mm, addr,
-							cc->policy.install_pmd);
+							cc->policy.file_install_pmd);
 		if (result == SCAN_PMD_MAPPED)
 			result = SCAN_SUCCEED;
 		mmap_read_unlock(mm);
@@ -2928,7 +2938,7 @@ static void collapse_scan_mm_slot(unsign
 		}
 		/* One mask for the whole VMA */
 		orders = collapse_possible_orders(vma, vma->vm_flags,
-						  cc->policy.tva_type);
+						  TVA_KHUGEPAGED);
 		if (!orders) {
 			cc->progress++;
 			continue;
@@ -3032,7 +3042,6 @@ static void khugepaged_do_scan(struct co
 	lru_add_drain_all();
 
 	collapse_control_init(cc);
-	/* One policy for the whole pass, so every table is treated the same */
 	collapse_policy_khugepaged(&cc->policy);
 
 	while (true) {
@@ -3065,8 +3074,6 @@ static void khugepaged_do_scan(struct co
 			khugepaged_alloc_sleep();
 		}
 	}
-
-	collapse_control_release(cc);
 }
 
 static bool khugepaged_should_wakeup(void)
--- a/mm/madvise.c~b
+++ a/mm/madvise.c
@@ -910,16 +910,18 @@ bool madvise_dontneed_free_valid_vma(str
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 
 /* MADV_COLLAPSE was asked for explicitly, so it is not held to those */
-static void collapse_policy_forced(struct collapse_policy *p)
+static void collapse_policy_madvise(struct collapse_policy *p)
 {
-	p->max_ptes_none = HPAGE_PMD_NR;
-	p->max_ptes_swap = HPAGE_PMD_NR;
-	p->max_ptes_shared = HPAGE_PMD_NR;
-	p->strict_sub_pmd = false;
-	p->skip_lazyfree = false;
-	p->require_referenced = false;
-	p->install_pmd = true;
-	p->writeback_dirty = true;
+	p->pmd.max_ptes_none = HPAGE_PMD_NR;
+	p->pmd.max_ptes_swap = HPAGE_PMD_NR;
+	p->pmd.max_ptes_shared = HPAGE_PMD_NR;
+	/* Never read: MADV_COLLAPSE collapses to PMD order only */
+	p->sub_pmd = p->pmd;
+
+	p->anon_skip_lazyfree = false;
+	p->anon_require_referenced = false;
+	p->file_install_pmd = true;
+	p->file_writeback_dirty = true;
 	p->gfp = GFP_TRANSHUGE;
 	p->tva_type = TVA_FORCED_COLLAPSE;
 }
@@ -984,7 +986,7 @@ static int madvise_collapse(struct madvi
 	if (!cc)
 		return -ENOMEM;
 	collapse_control_init(cc);
-	collapse_policy_forced(&cc->policy);
+	collapse_policy_madvise(&cc->policy);
 
 	lru_add_drain_all();
 
@@ -1010,7 +1012,7 @@ static int madvise_collapse(struct madvi
 			vma = found;
 			hend = min(hend, vma->vm_end & HPAGE_PMD_MASK);
 			orders = collapse_possible_orders(vma, vma->vm_flags,
-							  cc->policy.tva_type);
+							  TVA_FORCED_COLLAPSE);
 		}
 
 		result = collapse_scan_pmd(vma, addr, cc, orders);
@@ -1056,7 +1058,6 @@ out:
 		mmap_read_lock(mm);
 out_locked:
 	mmap_assert_locked(mm);
-	collapse_control_release(cc);
 	kfree(cc);
 
 	return thps == ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0
_