[PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range()

[PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range()

[Yuan-Hao Hsu]

GUP-fast grabs a PTE-mapped large folio one page at a time.  Every PTE
costs a try_grab_folio_fast() (a refcount cmpxchg, plus the pincount
atomic, a full barrier and a node stat update for FOLL_PIN), a
gup_fast_folio_allowed() and a folio_set_referenced(), so a 64 kB mTHP
pays sixteen of each and a PTE-mapped 2 MB THP pays 512.  The PMD and
PUD leaf paths already take the whole range with one
try_grab_folio_fast() call, and the slow path is getting the same
treatment for PTEs in Rik's follow_page_mask() series.

The users are the hot ones: iov_iter_extract_user_pages() pins every
O_DIRECT buffer through pin_user_pages_fast(), io_uring registers its
buffers through it, and so does RDMA memory registration.  With mTHP
those buffers are PTE-mapped large folios.

After the head page has been grabbed and verified as before, extend the
run to the following PTEs that map the next pages of the same folio
with the same protection bits, take the remaining references for the
run with try_grab_folio(), and fill pages[] from the run.  The PTEs of
the run are read while the reference on the folio is held, so the folio
can neither be freed nor split underneath the scan; a THP collapse can
still detach the page table, so the pmd is checked again after the run
is grabbed and the run is dropped if it changed, leaving the next
iteration to hit the existing pmd check and bail out.  A read-only run
of an anonymous folio stops at the first page that gup_must_unshare()
rejects, so the per-page exclusivity guarantee for FOLL_PIN is kept.

folio_pte_batch_flags() does the scan.  It reads the entries with
ptep_get(), which is not safe without the page table lock on
CONFIG_GUP_GET_PXX_LOW_HIGH (x86 PAE, mips32, sh) or with arm64 contpte,
and GUP-fast reads every PTE with ptep_get_lockless() for that reason.
Add FPB_LOCKLESS to make the helper do the same; it is inlined per call
site, so the other callers do not change.

The order-0 path keeps its instruction sequence.  The folio flags are
tested before folio_set_referenced() on purpose: a load of the flags
right after that locked instruction cost about 25% on order-0 pages in
the gup_test benchmark below, and reading them first brings that path
back to the baseline.

gup_test ioctl over a 256 MB anonymous region, FOLL_WRITE, 65536 pages
per call, median of 15 runs, two baseline boots / one patched boot, on
an x86-64 VM:

                       get_user_pages_fast()   pin_user_pages_fast()
                       before       after      before       after
  4 kB pages           849/789 us   783 us     1174/1138 us 1094 us
  64 kB mTHP           812/811 us   126 us     1215/1113 us  188 us
  1 MB mTHP            799/774 us    59 us     1185/1077 us   64 us
  2 MB THP, PTE-mapped 803/786 us    57 us     1159/1120 us   58 us
  2 MB THP, PMD-mapped  42/40 us     40 us       46/41 us     42 us

fio O_DIRECT randread of a null_blk device (CPU bound) with the I/O
buffer in 64 kB mTHP, median of 15 five-second runs:

  bs=1M psync                 126.2/125.6 GB/s  ->  220.6 GB/s
  bs=1M io_uring, iodepth 16   71.2/71.6 GB/s   ->  112.1 GB/s
  bs=64k io_uring, iodepth 16  50.2/50.7 GB/s   ->   60.1 GB/s

IORING_REGISTER_BUFFERS of 1 GB, median of 15:

  64 kB mTHP           6179/5998 us  ->  1599 us
  1 MB mTHP            5790/5801 us  ->   893 us
  4 kB pages           7173/6741 us  ->  6763 us

Link: https://lore.kernel.org/r/20260811025157.1632867-1-riel@surriel.com/
Assisted-by: LLM sparse
Signed-off-by: Yuan-Hao Hsu <aa9736195201@gmail.com>
---
 mm/gup.c      | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/internal.h | 15 +++++++++-
 2 files changed, 90 insertions(+), 1 deletion(-)

diff --git a/mm/gup.c b/mm/gup.c
index eb898ea1ee22..1b4bbb525609 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2807,6 +2807,64 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags)
 }
 
 #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL
+/*
+ * Extend the run of pages grabbed by gup_fast_pte_range() from @page, mapped
+ * by the PTE at @ptep, to the following PTEs that map the next pages of
+ * @folio with the same protection.  The caller holds one reference for @page
+ * and has verified @pte against the page table; this takes the references
+ * for the rest of the run and stores its pages in @pages.  Returns the
+ * number of pages in the run, including @page.
+ */
+static unsigned int gup_fast_pte_batch(struct folio *folio,
+		struct page *page, pmd_t pmd, pmd_t *pmdp, pte_t *ptep,
+		pte_t pte, unsigned int flags, unsigned int max_nr,
+		struct page **pages)
+{
+	unsigned int nr, i;
+
+	if (max_nr == 1)
+		return 1;
+
+	/*
+	 * The reference on @folio keeps folio_nr_pages() stable.  The PTEs
+	 * are read without the PTL, hence FPB_LOCKLESS.
+	 */
+	nr = folio_pte_batch_flags(folio, NULL, ptep, &pte, max_nr,
+				   FPB_RESPECT_WRITE | FPB_LOCKLESS);
+
+	/*
+	 * gup_must_unshare() is per page: a read-only run of an anonymous
+	 * folio ends at the first page that is not exclusive.
+	 */
+	if (!pte_write(pte)) {
+		for (i = 1; i < nr; i++) {
+			if (gup_must_unshare(NULL, flags, page + i))
+				break;
+		}
+		nr = i;
+	}
+	if (nr == 1)
+		return 1;
+
+	if (try_grab_folio(folio, nr - 1, flags))
+		return 1;
+
+	/*
+	 * The PTEs were read after the reference on @folio was taken, so the
+	 * pages cannot have been freed, but the page table could have been
+	 * detached by a THP collapse meanwhile, leaving stale PTEs.  Drop the
+	 * run and let the next iteration hit the pmd check and bail out.
+	 */
+	if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp)))) {
+		gup_put_folio(folio, nr - 1, flags);
+		return 1;
+	}
+
+	for (i = 1; i < nr; i++)
+		*pages++ = page + i;
+	return nr;
+}
+
 /*
  * GUP-fast relies on pte change detection to avoid concurrent pgtable
  * operations.
@@ -2830,6 +2888,8 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
 		unsigned long end, unsigned int flags, struct page **pages,
 		int *nr)
 {
+	unsigned int nr_batch;
+	bool large;
 	int ret = 0;
 	pte_t *ptep, *ptem;
 
@@ -2891,9 +2951,25 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
 			gup_put_folio(folio, 1, flags);
 			goto pte_unmap;
 		}
+		/*
+		 * Read the folio flags before the atomic in
+		 * folio_set_referenced(); a load right after it has to wait for
+		 * it, which is measurable on the order-0 path.
+		 */
+		large = folio_test_large(folio);
 		folio_set_referenced(folio);
 		pages[*nr] = page;
 		(*nr)++;
+
+		if (likely(!large))
+			continue;
+
+		nr_batch = gup_fast_pte_batch(folio, page, pmd, pmdp, ptep, pte,
+					      flags, (end - addr) >> PAGE_SHIFT,
+					      pages + *nr);
+		*nr += nr_batch - 1;
+		ptep += nr_batch - 1;
+		addr += (nr_batch - 1) * PAGE_SIZE;
 	} while (ptep++, addr += PAGE_SIZE, addr != end);
 
 	ret = 1;
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c9..783754a81960 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -368,6 +368,12 @@ typedef int __bitwise fpb_t;
  */
 #define FPB_MERGE_YOUNG_DIRTY		((__force fpb_t)BIT(4))
 
+/*
+ * Read the page table entries with ptep_get_lockless(): the caller does not
+ * hold the page table lock (GUP-fast).
+ */
+#define FPB_LOCKLESS			((__force fpb_t)BIT(5))
+
 static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
 {
 	if (!(flags & FPB_RESPECT_DIRTY))
@@ -400,6 +406,10 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
  * must be limited by the caller so scanning cannot exceed a single VMA and
  * a single page table.
  *
+ * The caller must hold the page table lock, unless FPB_LOCKLESS is set: then
+ * the entries are read with ptep_get_lockless() and the caller has to make
+ * sure the folio cannot be freed or split, as GUP-fast does.
+ *
  * Depending on the FPB_MERGE_* flags, the pte stored at @ptentp will
  * be updated: it's crucial that a pointer to a COPY of the first
  * page table entry, obtained through ptep_get(), is provided as @ptentp.
@@ -436,7 +446,10 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
 	ptep = ptep + nr;
 
 	while (nr < max_nr) {
-		pte = ptep_get(ptep);
+		if (flags & FPB_LOCKLESS)
+			pte = ptep_get_lockless(ptep);
+		else
+			pte = ptep_get(ptep);
 
 		if (!pte_same(__pte_batch_clear_ignored(pte, flags), expected_pte))
 			break;

base-commit: 9b87fdc9af2fbfcdb5c24a64139685ef80f6573f
-- 
2.43.0

Re: [PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range()

[David Hildenbrand (Arm)]

On 9/17/26 10:37, Yuan-Hao Hsu wrote:
> GUP-fast grabs a PTE-mapped large folio one page at a time.  Every PTE
> costs a try_grab_folio_fast() (a refcount cmpxchg, plus the pincount
> atomic, a full barrier and a node stat update for FOLL_PIN), a
> gup_fast_folio_allowed() and a folio_set_referenced(), so a 64 kB mTHP
> pays sixteen of each and a PTE-mapped 2 MB THP pays 512.  The PMD and
> PUD leaf paths already take the whole range with one
> try_grab_folio_fast() call, and the slow path is getting the same
> treatment for PTEs in Rik's follow_page_mask() series.
> 
> The users are the hot ones: iov_iter_extract_user_pages() pins every
> O_DIRECT buffer through pin_user_pages_fast(), io_uring registers its
> buffers through it, and so does RDMA memory registration.  With mTHP
> those buffers are PTE-mapped large folios.
> 
> After the head page has been grabbed and verified as before, extend the
> run to the following PTEs that map the next pages of the same folio
> with the same protection bits, take the remaining references for the
> run with try_grab_folio(), and fill pages[] from the run.  The PTEs of
> the run are read while the reference on the folio is held, so the folio
> can neither be freed nor split underneath the scan; a THP collapse can
> still detach the page table, so the pmd is checked again after the run
> is grabbed and the run is dropped if it changed, leaving the next
> iteration to hit the existing pmd check and bail out.  A read-only run
> of an anonymous folio stops at the first page that gup_must_unshare()
> rejects, so the per-page exclusivity guarantee for FOLL_PIN is kept.
> 
> folio_pte_batch_flags() does the scan.  It reads the entries with
> ptep_get(), which is not safe without the page table lock on
> CONFIG_GUP_GET_PXX_LOW_HIGH (x86 PAE, mips32, sh) or with arm64 contpte,
> and GUP-fast reads every PTE with ptep_get_lockless() for that reason.
> Add FPB_LOCKLESS to make the helper do the same; it is inlined per call
> site, so the other callers do not change.
> 
> The order-0 path keeps its instruction sequence.  The folio flags are
> tested before folio_set_referenced() on purpose: a load of the flags
> right after that locked instruction cost about 25% on order-0 pages in
> the gup_test benchmark below, and reading them first brings that path
> back to the baseline.
> 
> gup_test ioctl over a 256 MB anonymous region, FOLL_WRITE, 65536 pages
> per call, median of 15 runs, two baseline boots / one patched boot, on
> an x86-64 VM:
> 
>                        get_user_pages_fast()   pin_user_pages_fast()
>                        before       after      before       after
>   4 kB pages           849/789 us   783 us     1174/1138 us 1094 us
>   64 kB mTHP           812/811 us   126 us     1215/1113 us  188 us
>   1 MB mTHP            799/774 us    59 us     1185/1077 us   64 us
>   2 MB THP, PTE-mapped 803/786 us    57 us     1159/1120 us   58 us
>   2 MB THP, PMD-mapped  42/40 us     40 us       46/41 us     42 us
> 
> fio O_DIRECT randread of a null_blk device (CPU bound) with the I/O
> buffer in 64 kB mTHP, median of 15 five-second runs:
> 
>   bs=1M psync                 126.2/125.6 GB/s  ->  220.6 GB/s
>   bs=1M io_uring, iodepth 16   71.2/71.6 GB/s   ->  112.1 GB/s
>   bs=64k io_uring, iodepth 16  50.2/50.7 GB/s   ->   60.1 GB/s
> 
> IORING_REGISTER_BUFFERS of 1 GB, median of 15:
> 
>   64 kB mTHP           6179/5998 us  ->  1599 us
>   1 MB mTHP            5790/5801 us  ->   893 us
>   4 kB pages           7173/6741 us  ->  6763 us
> 

I think Rik was already working on this and sent some patches. Anyhow, there is
quite some GUP review backlog I have t go through, so this will have to wait.

-- 
Cheers,

David

Re: [PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range()

[David Hildenbrand (Arm)]

On 9/17/26 10:42, David Hildenbrand (Arm) wrote:
> On 9/17/26 10:37, Yuan-Hao Hsu wrote:
>> GUP-fast grabs a PTE-mapped large folio one page at a time.  Every PTE
>> costs a try_grab_folio_fast() (a refcount cmpxchg, plus the pincount
>> atomic, a full barrier and a node stat update for FOLL_PIN), a
>> gup_fast_folio_allowed() and a folio_set_referenced(), so a 64 kB mTHP
>> pays sixteen of each and a PTE-mapped 2 MB THP pays 512.  The PMD and
>> PUD leaf paths already take the whole range with one
>> try_grab_folio_fast() call, and the slow path is getting the same
>> treatment for PTEs in Rik's follow_page_mask() series.
>>
>> The users are the hot ones: iov_iter_extract_user_pages() pins every
>> O_DIRECT buffer through pin_user_pages_fast(), io_uring registers its
>> buffers through it, and so does RDMA memory registration.  With mTHP
>> those buffers are PTE-mapped large folios.
>>
>> After the head page has been grabbed and verified as before, extend the
>> run to the following PTEs that map the next pages of the same folio
>> with the same protection bits, take the remaining references for the
>> run with try_grab_folio(), and fill pages[] from the run.  The PTEs of
>> the run are read while the reference on the folio is held, so the folio
>> can neither be freed nor split underneath the scan; a THP collapse can
>> still detach the page table, so the pmd is checked again after the run
>> is grabbed and the run is dropped if it changed, leaving the next
>> iteration to hit the existing pmd check and bail out.  A read-only run
>> of an anonymous folio stops at the first page that gup_must_unshare()
>> rejects, so the per-page exclusivity guarantee for FOLL_PIN is kept.
>>
>> folio_pte_batch_flags() does the scan.  It reads the entries with
>> ptep_get(), which is not safe without the page table lock on
>> CONFIG_GUP_GET_PXX_LOW_HIGH (x86 PAE, mips32, sh) or with arm64 contpte,
>> and GUP-fast reads every PTE with ptep_get_lockless() for that reason.
>> Add FPB_LOCKLESS to make the helper do the same; it is inlined per call
>> site, so the other callers do not change.
>>
>> The order-0 path keeps its instruction sequence.  The folio flags are
>> tested before folio_set_referenced() on purpose: a load of the flags
>> right after that locked instruction cost about 25% on order-0 pages in
>> the gup_test benchmark below, and reading them first brings that path
>> back to the baseline.
>>
>> gup_test ioctl over a 256 MB anonymous region, FOLL_WRITE, 65536 pages
>> per call, median of 15 runs, two baseline boots / one patched boot, on
>> an x86-64 VM:
>>
>>                        get_user_pages_fast()   pin_user_pages_fast()
>>                        before       after      before       after
>>   4 kB pages           849/789 us   783 us     1174/1138 us 1094 us
>>   64 kB mTHP           812/811 us   126 us     1215/1113 us  188 us
>>   1 MB mTHP            799/774 us    59 us     1185/1077 us   64 us
>>   2 MB THP, PTE-mapped 803/786 us    57 us     1159/1120 us   58 us
>>   2 MB THP, PMD-mapped  42/40 us     40 us       46/41 us     42 us
>>
>> fio O_DIRECT randread of a null_blk device (CPU bound) with the I/O
>> buffer in 64 kB mTHP, median of 15 five-second runs:
>>
>>   bs=1M psync                 126.2/125.6 GB/s  ->  220.6 GB/s
>>   bs=1M io_uring, iodepth 16   71.2/71.6 GB/s   ->  112.1 GB/s
>>   bs=64k io_uring, iodepth 16  50.2/50.7 GB/s   ->   60.1 GB/s
>>
>> IORING_REGISTER_BUFFERS of 1 GB, median of 15:
>>
>>   64 kB mTHP           6179/5998 us  ->  1599 us
>>   1 MB mTHP            5790/5801 us  ->   893 us
>>   4 kB pages           7173/6741 us  ->  6763 us
>>
> 
> I think Rik was already working on this and sent some patches. Anyhow, there is
> quite some GUP review backlog I have t go through, so this will have to wait.
> 

Ah, no, Rik looked at GUP not GUP-fast. So I'll have a look but it will take a
while.

-- 
Cheers,

David

Re: [PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range()

[Yuan-Hao Hsu]

On Thu, 17 Sep 2026 10:43:22 +0200, David Hildenbrand (Arm) wrote:
> Ah, no, Rik looked at GUP not GUP-fast. So I'll have a look but it will take a
> while.

Thanks.

One note for whenever you get to it: the patch is against mainline, and
the last hunk has a two-line context conflict with Aristeu's
gup_fast_*() cleanup in mm-everything (pages[nr_pages++] there).  I
have the same change on top of mm-everything-2026-09-15-07-39 if that
base is preferable, and can post FPB_LOCKLESS as a separate first patch
if a two-patch series is easier to review.

Yuan-Hao Hsu