Re: [PATCH] Documentation/userfaultfd: document RWP behavior when the PTE is dropped

["Jose A. Perez de Azpillaga" <azpijr@gmail.com>]

On 11/13/26 1:44 PM, Kiryl Shutsemau wrote:
> On Fri, Nov 13, 2026 at 12:27:42PM +0100, Jose A. Perez de Azpillaga wrote:
>> "Scope of protection" lists only the PTE drops requested by the user and
>> says the range has to be re-armed afterwards. On shmem the kernel drops
>> the PTE as well (page reclaim, collapse), and RWP has no marker by design:
>> the next access is a minor or a missing fault, and the user is expected to
>> register for those. Document this, together with the modes that sync and
>> async tracking need.
>>
>> The VMM example registers guest_mem with RWP and MISSING only, so a vCPU
>> write to an offset that is not mapped there can be lost during eviction.
>> Add MINOR to the registration and describe how the handler treats minor
>> faults.
>>
>> Link: https://lore.kernel.org/linux-mm/856cdf21-7982-475a-b244-a7ceff4eb6aa@kernel.org/
>> Link: https://lore.kernel.org/linux-mm/avWnaR7VVEGa7yNp@thinkstation/
>> Suggested-by: Tal Zussman <tz2294@columbia.edu>
>> Signed-off-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>
>> ---
>> this is the first time I'm writing documentation, so I accept any
>> suggestions to improve it :) and I tried to follow the documentation's
>> own style.
>>
>>  Documentation/admin-guide/mm/userfaultfd.rst | 65 ++++++++++++++++----
>>  1 file changed, 53 insertions(+), 12 deletions(-)
>>
>> diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst
>> index 97daea83b7f1..19bdac668f93 100644
>> --- a/Documentation/admin-guide/mm/userfaultfd.rst
>> +++ b/Documentation/admin-guide/mm/userfaultfd.rst
>> @@ -384,17 +384,45 @@ normal missing path (``do_anonymous_page()``, ``do_swap_page()``,
>>  the population itself, co-register the range with
>>  ``UFFDIO_REGISTER_MODE_MISSING``.
>>  
>> -Protection is preserved across page reclaim: a page swapped out while
>> -RWP-protected carries the marker on its swap entry, and swap-in restores
>> -the PROT_NONE state so the first access after swap-in still faults. The
>> -same applies to pages temporarily replaced by migration entries.
>> -
>> -Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous
>> -memory, hole-punch on shmem, truncation of a file mapping — also drop the
>> -RWP marker: the next access re-populates the range without protection.
>> -Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no
>> -persistent RWP marker today. The user needs to re-arm the range with
>> -``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs.
>> +Protection of anonymous memory is preserved across page reclaim: a page
>> +swapped out while RWP-protected carries the marker on its swap entry, and
>> +swap-in restores the PROT_NONE state so the first access after swap-in
>> +still faults. The same applies to pages temporarily replaced by migration
>> +entries. Shmem is different: reclaiming a shmem page leaves the PTE empty
>> +rather than installing a swap entry, so the protection is dropped along
>> +with the PTE.
> 
> Migration is not anon-only: migration entries carry the uffd bit for
> shmem and hugetlbfs too. Keep that sentence out of the anon part.
> 
> And please call it the uffd bit, not the marker. "Carries the marker"
> next to "there is no persistent RWP marker" reads as a contradiction.
> 
>> +Operations that drop the PTE entirely also drop the RWP protection. Some
>> +are requested by the user — ``MADV_DONTNEED``, hole-punch on shmem,
>> +truncation of a file mapping — and some are initiated by the kernel on
>> +shmem: page reclaim, and collapse into a huge page by khugepaged or
>> +``MADV_COLLAPSE``.
> 
> I would not list them. The list is not complete: splitting a PMD mapping
> of a shmem THP drops it too. The next tool will find the path that is
> missing and report it. State the rule instead: on shmem the kernel can
> drop the PTE at any time while the page stays in the page cache or swap.
> Reclaim and collapse are examples.
> 
>> Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``),
>> +there is no persistent RWP marker, and this is intentional: once the PTE is
>> +gone, the next access takes a page fault that userfaultfd can deliver, and
>> +that fault is the notification. Which fault it is depends on what happened
>> +to the page:
>> +
>> +- The page is still in the page cache or in swap: the access is a minor
>> +  fault, delivered if the range is co-registered with
>> +  ``UFFDIO_REGISTER_MODE_MINOR``.
>> +
>> +- The page is gone: the access is a missing fault, delivered if the range
>> +  is co-registered with ``UFFDIO_REGISTER_MODE_MISSING``.
>> +
>> +WP cannot use the same scheme — a minor fault would trap reads as well as
>> +writes — which is why it needs a marker.
>> +
>> +Without the matching mode the page is mapped back silently and without
>> +protection, so what to co-register depends on the tracking mode:
>> +
>> +- Sync mode: the access would be lost. Co-register shmem and hugetlbfs
>> +  ranges with ``UFFDIO_REGISTER_MODE_MINOR`` (and
>> +  ``UFFDIO_REGISTER_MODE_MISSING`` if the file can have holes), and
>> +  anonymous ranges, where minor faults are not available, with
>> +  ``UFFDIO_REGISTER_MODE_MISSING``.
>> +
>> +- Async mode: neither is needed. A page faulted back in is mapped without
>> +  the uffd bit, so ``PAGEMAP_SCAN`` reports it as accessed.
> 
> The registration is fixed, the mode is not. A context that switches to
> sync with UFFDIO_SET_MODE, as the example does, needs the sync
> registration. That has a cost in async mode: minor faults are always
> synchronous, so "async -- no vCPU stalls" in step 1 is no longer true.
> 
> Getting it back would need MINOR to follow UFFDIO_SET_MODE: in async
> mode, map the page and let the next scan report it as accessed. That
> could be a future project if the stalls turn out to matter. For this
> patch, just fix the comment.
> 
>>  **Fault Handling:**
>>  
>> @@ -486,10 +514,16 @@ never traps on its own ::
>>      ioctl(uffd, UFFDIO_API, &api);
>>      if (!(api.features & UFFD_FEATURE_RWP))
>>          /* RWP unavailable on this kernel/arch -- fall back. */
>> +    /*
>> +     * MINOR: an offset in the page cache but not mapped in guest_mem
>> +     * would be mapped back without a fault, and a vCPU write between
>> +     * the pwrite() and the hole punch in step 4 would be lost.
>> +     */
> 
> Not yours, but the "if (!(api.features & UFFD_FEATURE_RWP))" has only a
> comment as its body, so UFFDIO_REGISTER becomes the body. A second
> comment between them makes that harder to see.
> 
> Adding {} around the body comment of the branch can help a bit.
> 
>>      ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){
>>          .range = { guest_mem, guest_size },
>>          .mode = UFFDIO_REGISTER_MODE_RWP |
>> -                UFFDIO_REGISTER_MODE_MISSING,
>> +                UFFDIO_REGISTER_MODE_MISSING |
>> +                UFFDIO_REGISTER_MODE_MINOR,
>>      });
>>  
>>      /* Tracking loop */
>> @@ -550,6 +584,13 @@ progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE``
>>  fires, the vCPU retries the access, faults as ``MISSING``, and the
>>  handler resolves it with ``UFFDIO_COPY`` from storage.
>>  
>> +The same holds for an offset being evicted that is not mapped in
>> +``guest_mem``, except that the vCPU blocks with a
>> +``UFFD_PAGEFAULT_FLAG_MINOR`` fault, which the handler leaves pending until
>> +the page has been punched out. Minor faults on any other offset are
>> +resolved with ``UFFDIO_CONTINUE``: the page is mapped without RWP
>> +protection, so the next scan reports it as accessed.
> 
> "Any other offset" is too broad. A cold offset that step 4 has not
> reached yet will still be written out and punched. If the handler maps
> it with UFFDIO_CONTINUE, a vCPU write between the pwrite() and the punch
> is lost. That is the case MINOR was added for. A fault on any offset
> this pass will evict has to wait until that offset is punched. The same
> goes for RWP faults.

thanks for the review! I'll address all of it in v2 :)

-- 
cheers,
jose a. p-a