Re: [PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range()

[Rik van Riel <riel@surriel.com>]

On Thu, 2026-09-17 at 16:37 +0800, Yuan-Hao Hsu wrote:
> GUP-fast grabs a PTE-mapped large folio one page at a time.  Every
> PTE
> costs a try_grab_folio_fast() (a refcount cmpxchg, plus the pincount
> atomic, a full barrier and a node stat update for FOLL_PIN), a
> gup_fast_folio_allowed() and a folio_set_referenced(), so a 64 kB
> mTHP
> pays sixteen of each and a PTE-mapped 2 MB THP pays 512.  The PMD and
> PUD leaf paths already take the whole range with one
> try_grab_folio_fast() call, and the slow path is getting the same
> treatment for PTEs in Rik's follow_page_mask() series.
> 

> fio O_DIRECT randread of a null_blk device (CPU bound) with the I/O
> buffer in 64 kB mTHP, median of 15 five-second runs:
> 
>   bs=1M psync                 126.2/125.6 GB/s  ->  220.6 GB/s
>   bs=1M io_uring, iodepth 16   71.2/71.6 GB/s   ->  112.1 GB/s
>   bs=64k io_uring, iodepth 16  50.2/50.7 GB/s   ->   60.1 GB/s
> 
That's huge! Nice.

> 
> +static unsigned int gup_fast_pte_batch(struct folio *folio,
> +		struct page *page, pmd_t pmd, pmd_t *pmdp, pte_t
> *ptep,
> +		pte_t pte, unsigned int flags, unsigned int max_nr,
> +		struct page **pages)
> +{
> +	unsigned int nr, i;
> +
> +	if (max_nr == 1)
> +		return 1;
> +
> +	/*
> +	 * The reference on @folio keeps folio_nr_pages() stable. 
> The PTEs
> +	 * are read without the PTL, hence FPB_LOCKLESS.
> +	 */
> +	nr = folio_pte_batch_flags(folio, NULL, ptep, &pte, max_nr,
> +				   FPB_RESPECT_WRITE |
> FPB_LOCKLESS);
> 
...
> +	if (try_grab_folio(folio, nr - 1, flags))
> +		return 1;
> 

Since this is lockless, the PTEs could change under
us between when we walk them in folio_pte_batch_flags()
and the batched try_grab_folio().

Is that race condition safe, or do we need to re-check
afterward that:
1) the ptes still point at the same folio, and/or
2) the folio still spans the entire range, and was
   not split before we grabbed the refcount?

The equivalent changes to GUP do not need that check,
because that all happens under the pte_lock:

https://lore.kernel.org/all/20260811025157.1632867-1-riel@surriel.com/

-- 
All Rights Reversed.