Re: [PATCH] KVM: guest_memfd: Disable VMA merging with VM_DONTEXPAND

["David Hildenbrand (arm)" <david@kernel.org>]

On 2/4/26 22:37, Sean Christopherson wrote:
> On Wed, Feb 04, 2026, Ackerley Tng wrote:
>> Ackerley Tng <ackerleytng@google.com> writes:
>>
>>> #syz test: git://git.kernel.org/pub/scm/virt/kvm/kvm.git next
>>>
>>> guest_memfd VMAs don't need to be merged,
> 
> Why not?  There are benefits to merging VMAs that have nothing to do with folios.
> E.g. map 1GiB of guest_memfd with 512*512 4KiB VMAs, and then it becomes quite
> desirable to merge all of those VMAs into one.
> 
> Creating _hugepages_ doesn't add value, but that's not the same things as merging
> VMAs.
> 
>>> especially now, since guest_memfd only supports PAGE_SIZE folios.
>>>
>>> Set VM_DONTEXPAND on guest_memfd VMAs.
>>
>> Local tests and syzbot agree that this fixes the issue identified. :)
>>
>> I would like to look into madvise(MADV_COLLAPSE) and uprobes triggering
>> mapping/folio collapsing before submitting a full patch series.
>>
>> David, Michael, Vishal, what do you think of the choice of setting
>> VM_DONTEXPAND to disable khugepaged?
> 
> I'm not one of the above, but for me it feels very much like treating a symptom
> and not fixing the underlying cause.

And you are spot-on :)

> 
> It seems like what KVM should do is not block one path that triggers hugepage
> processing, but instead flat out disallow creating hugepages.  Unfortunately,
> AFAICT, there's no existing way to prevent madvise() from clearing VM_NOHUGEPAGE,
> so we can't simply force that flag.
> 
> I'd prefer not to special case guest_memfd, a la devdax, but I also want to address
> this head-on, not by removing a tangentially related trigger.

VM_NOHUGEPAGE also smells like the wrong thing. This is a file limitation.

!thp_vma_allowable_order() must take care of that somehow down in 
__thp_vma_allowable_orders(), by checking the file).

Likely the file_thp_enabled() check is the culprit with 
CONFIG_READ_ONLY_THP_FOR_FS?

Maybe we need a flag to say "even not CONFIG_READ_ONLY_THP_FOR_FS".

I wonder how we handle that for secretmem. Too late for me, going to bed :)

-- 
Cheers,

David