Re: [RFC PATCH 00/57] mm/collapse: rebuild collapse on migration primitives

["David Hildenbrand (Arm)" <david@kernel.org>]

On 8/19/26 19:09, Kiryl Shutsemau wrote:
> On Tue, Aug 18, 2026 at 03:55:55PM +0200, David Hildenbrand (Arm) wrote:
>>> This replaces khugepaged's anonymous collapse with an engine that
>>> can collapse sub-PMD ranges. It is built around migration entries and
>>> frozen folios instead of heavy locking and isolation, aiming for better
>>> scalability and less disruption to the workload being collapsed.
>>
>> I recall us discussing something around using some PTE/PMD markers (e.g.,
>> migration entries) in the past.
>>
>> One thing that needed care is handling concurrent MADV_DONTNEED + faultin after
>> dropping relevant locks.
> 
> Handled at install time.
> 
> I drop the PTL after the freeze to allow allocation and copy, but sample
> the PTE values (see saved_ptes) at freeze time. If something changed
> under us by the time we install the new page table entries, we give up
> on that candidate and roll it back; the rest of the round still
> installs. We allow harmless transitions: zero page to none.

I think there was more to it, and Jann also hinted at some examples in his
reply. But we'll get to that part once it's no longer buried in 39 patches ;)

> 
>>> Which is why hugepage_vma_revalidate() demands that the VMA span the
>>> whole PMD even for an mTHP order -- "we'd need to lock all VMAs in the
>>> PMD range to support this", as the comment there puts it.  A PMD-granular
>>> operation is only safe when one VMA owns the PMD, and that is exactly the
>>> restriction in the way.  The alignment is the symptom; the PMD is the
>>> design.
>>
>> I disagree with "A PMD-granular operation is only safe when one VMA owns the
>> PMD". It's safe when all page table walkers can be stopped (see above).
> 
> Fair, the sentence is too strong.  mmap_write_lock plus a VMA write lock on
> every VMA the PMD covers, plus their rmap locks, would make it safe.

Ack.

> 
> But the mechanism still clears the whole PMD, flushes, IPIs and
> repopulates it to collapse each 16-page window. That's very noisy to the
> workload.
> 

Right. khugepaged itself is pretty noise already, though. So one would have to
understand "how much more noisy and who cares".

I can understand why one would want to make khugepaged less noisy, though.

> And I am not sure how to deal with rmap locking here.  Nothing in mm
> holds two unrelated anon_vma rwsems: vma_prepare() takes one for both VMAs
> it touches, because a merge requires them to share the anon_vma, and
> anon_vma_clone() takes one because "all anon_vma's share the same root".
> The lock ordering in mm/rmap.c has a single anon_vma->rwsem level, so a PMD
> spanning unrelated mappings would need an ordering rule that does not exist
> today.

That's a good point. Try-locking would likely work but might have other effects.
Nobody tried this so far.

Lorenzo is on his way to simplify a lot on that anon_vma front (scalable cow),
and IIRC it would also simplify that case.

> 
>>> Between the two, nothing can reach a source, so the copy runs with no
>>> lock held at all -- and the address space is left alone while it does.
>>
>> Right. Concurrent MADV_DONTNEED can zap migration PTEs and other faults even
>> re-fault fresh anon folios. So that must be detected before replacing migration
>> entries again I guess.
> 
> Yes, that is the install-time check above.
> 
> The copy itself is safe: a zap of a migration entry only clears the slot
> and adjusts rss -- see zap_nonpresent_ptes(), which neither puts the
> folio nor drops its rmap -- so a frozen, locked source cannot go away
> under the copy.

I'll have to think about the impact of having these folios frozen for a longer
time, instead of only very briefly during migration.

E.g., these folios will then be unmovable for the entirety of the collapse
operation, because folio_try_get() by memory offlining/cma/compaction will just
fail.

[...]

> 
> There are no PMD-level migration entries here.

That's good.

> 
> The freeze is always at PTE level, so the pmd keeps pointing at the
> table until the last step, and the PMD leaf goes in as the terminal
> layer: verify, pmdp_collapse_flush(), deposit a fresh table, set the
> leaf, all in one section under the pmd lock with the pte ptl nested
> inside.
> 
> A pmd-level walker sees the old table or the leaf and never pmd_none,
> and faults stay held at pte level by the migration entries throughout,
> which is what lets PMD collapse run under a VMA read lock like
> everything else.
> 
>>
>>> Working in windows rather than whole PMDs takes care of the other root.
>>> A sub-PMD window is collapsed under the page table lock, so a collapse
>>> disturbs only the window it collapses, and each candidate is validated
>>
>> I recall us discussing that holding the PT lock for a longer collapse operation
>> (especially on 64k) is problematic. But I don't get all the details from your
>> description here.
> 
> As I mentioned above, we drop the ptl after the freeze. And take it a
> second time for the install.  Allocation and the copy run in between with
> no lock held -- on 64K the copy at PMD order is 512M of it, which is why
> it cannot sit under either lock.

Hold on, are freezing all folios to collapse? That cannot possibly work with
COW-shared folios that are mapped into other address spaces.

We must only freeze a folio if we are sure that no frozen reference can go away
concurrently.

But maybe I misunderstood or you handle this in a special way elsewhere.

-- 
Cheers,

David