* [PATCH 1/8] mm/gup: break out gup_fill_pages() helper
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-03 8:43 ` David Hildenbrand (Arm)
2026-11-02 19:30 ` [PATCH 2/8] mm/gup: convert follow_page_mask() to return a long Rik van Riel
` (6 subsequent siblings)
7 siblings, 1 reply; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
__get_user_pages() fills pages[] and flushes each page's caches in an
open-coded loop.
Move it into a gup_fill_pages() helper, which the follow_page_mask()
call chain can then use to fill its own pages[] slots.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 27 ++++++++++++++++++---------
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 7d43ce717b246..cdbe2ed3e6442 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -633,6 +633,23 @@ static struct page *no_page_table(struct vm_area_struct *vma,
return NULL;
}
+static void gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
+ struct page *page, unsigned long nr, struct page **pages)
+{
+ unsigned long i;
+
+ if (!pages)
+ return;
+
+ for (i = 0; i < nr; i++) {
+ struct page *subpage = page + i;
+
+ pages[i] = subpage;
+ flush_anon_page(vma, subpage, address + i * PAGE_SIZE);
+ flush_dcache_page(subpage);
+ }
+}
+
#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES
/* FOLL_FORCE can write to even unwritable PUDs in COW mappings. */
static inline bool can_follow_write_pud(pud_t pud, struct page *page,
@@ -1461,9 +1478,6 @@ static long __get_user_pages(struct mm_struct *mm,
page_increm = nr_pages;
if (pages) {
- struct page *subpage;
- unsigned int j;
-
/*
* This must be a large folio (and doesn't need to
* be the whole folio; it can be part of it), do
@@ -1493,12 +1507,7 @@ static long __get_user_pages(struct mm_struct *mm,
}
}
- for (j = 0; j < page_increm; j++) {
- subpage = page + j;
- pages[i + j] = subpage;
- flush_anon_page(vma, subpage, start + j * PAGE_SIZE);
- flush_dcache_page(subpage);
- }
+ gup_fill_pages(vma, start, page, page_increm, pages + i);
}
i += page_increm;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 1/8] mm/gup: break out gup_fill_pages() helper
2026-11-02 19:30 ` [PATCH 1/8] mm/gup: break out gup_fill_pages() helper Rik van Riel
@ 2026-11-03 8:43 ` David Hildenbrand (Arm)
2026-11-03 16:29 ` Rik van Riel
0 siblings, 1 reply; 18+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-03 8:43 UTC (permalink / raw)
To: Rik van Riel, linux-kernel
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On 11/2/26 20:30, Rik van Riel wrote:
> __get_user_pages() fills pages[] and flushes each page's caches in an
> open-coded loop.
>
> Move it into a gup_fill_pages() helper, which the follow_page_mask()
> call chain can then use to fill its own pages[] slots.
>
> No functional changes intended.
>
> Suggested-by: David Hildenbrand <david@kernel.org>
> Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> Signed-off-by: Rik van Riel <riel@surriel.com>
> ---
> mm/gup.c | 27 ++++++++++++++++++---------
> 1 file changed, 18 insertions(+), 9 deletions(-)
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 7d43ce717b246..cdbe2ed3e6442 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -633,6 +633,23 @@ static struct page *no_page_table(struct vm_area_struct *vma,
> return NULL;
> }
>
> +static void gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
> + struct page *page, unsigned long nr, struct page **pages)
unsigned long nr_pages ?
> +{
> + unsigned long i;
> +
> + if (!pages)
> + return;
Why is that check needed? Why should someone try filling something that doesn't
exist?
> +
> + for (i = 0; i < nr; i++) {
> + struct page *subpage = page + i;
There is no such thing as a subpage as Willy keeps saying :)
Why not reuse "page" ?
for (i = 0; i < nr; i++, page++) {
> +
> + pages[i] = subpage;
> + flush_anon_page(vma, subpage, address + i * PAGE_SIZE);
> + flush_dcache_page(subpage);
Now I wonder why GUP-fast doesn't need any of that. Likely because architectures
that support GUP-fast don't need any such magic.
Maybe we should in fact base this on the gup-fast batching series and also use
the helper in there (passing vma=NULL)?
--
Cheers,
David
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 1/8] mm/gup: break out gup_fill_pages() helper
2026-11-03 8:43 ` David Hildenbrand (Arm)
@ 2026-11-03 16:29 ` Rik van Riel
2026-11-03 16:39 ` David Hildenbrand (Arm)
0 siblings, 1 reply; 18+ messages in thread
From: Rik van Riel @ 2026-11-03 16:29 UTC (permalink / raw)
To: David Hildenbrand (Arm), linux-kernel
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On Tue, 2026-11-03 at 09:43 +0100, David Hildenbrand (Arm) wrote:
> On 11/2/26 20:30, Rik van Riel wrote:
> > __get_user_pages() fills pages[] and flushes each page's caches in
> > an
> > open-coded loop.
> >
> > Move it into a gup_fill_pages() helper, which the
> > follow_page_mask()
> > call chain can then use to fill its own pages[] slots.
> >
> > No functional changes intended.
> >
> > Suggested-by: David Hildenbrand <david@kernel.org>
> > Reviewed-by: Suren Baghdasaryan <surenb@google.com>
> > Signed-off-by: Rik van Riel <riel@surriel.com>
> > ---
> > mm/gup.c | 27 ++++++++++++++++++---------
> > 1 file changed, 18 insertions(+), 9 deletions(-)
> >
> > diff --git a/mm/gup.c b/mm/gup.c
> > index 7d43ce717b246..cdbe2ed3e6442 100644
> > --- a/mm/gup.c
> > +++ b/mm/gup.c
> > @@ -633,6 +633,23 @@ static struct page *no_page_table(struct
> > vm_area_struct *vma,
> > return NULL;
> > }
> >
> > +static void gup_fill_pages(struct vm_area_struct *vma, unsigned
> > long address,
> > + struct page *page, unsigned long nr, struct page
> > **pages)
>
> unsigned long nr_pages ?
Will do. Thank you.
>
>
> > +{
> > + unsigned long i;
> > +
> > + if (!pages)
> > + return;
>
> Why is that check needed? Why should someone try filling something
> that doesn't
> exist?
faultin_page_range() and populate_vma_page_range
both call the GUP code with a NULL pages pointer.
>
> > +
> > + for (i = 0; i < nr; i++) {
> > + struct page *subpage = page + i;
>
> There is no such thing as a subpage as Willy keeps saying :)
>
> Why not reuse "page" ?
>
> for (i = 0; i < nr; i++, page++) {
Will do.
>
> > +
> > + pages[i] = subpage;
> > + flush_anon_page(vma, subpage, address + i *
> > PAGE_SIZE);
> > + flush_dcache_page(subpage);
>
> Now I wonder why GUP-fast doesn't need any of that. Likely because
> architectures
> that support GUP-fast don't need any such magic.
These cache flushes seem to be common on architectures
with virtually indexed caches, where the cache needs
to be flushed if the memory is accessed through a
different virtual address than where userspace might
have written data.
On 32 bit ARM, MIPS, and sh, the kernel seems to
implement an actual cache flush here, but also
claims to support GUP-fast.
I don't understand how that would possibly be safe,
since the pages returned by GUP-fast could be
accessed under a different cache alias than where
userspace accesses the memory.
I think one reason GUP-fast can get away without
a dcache flush is that most of the callers take
care of that, for example by using raw_copy_to_user()
to copy data using the user's virtual address, which
takes care of VIVT cache aliasing.
Likewise, drivers that do zero-copy DMA into user
memory will get the cache flushed on VIVT architectures
inside dma_unmap().
The dcache flush inside GUP covers rarer cases, where
the kernel accesses memory at a different address than
where userspace wrote to that memory.
cat /proc/<pid>/cmdline KERNEL
---- -------------------
read(buf=U)
-------------------------------->
get_mm_cmdline(mm, U):
access_remote_vm(mm, arg_start,
kbuf, len, FOLL_ANON):
get_user_page_vma_remote()
(slow GUP + its flushes)
kmap_local_folio(P)
memcpy(kbuf <- K) // read
kunmap_local(K)
copy_to_user(U, kbuf) // write
through U itself
read buf at U
------>
clean: the final write to U went
through U. Same line. No hazard.
In this case, the dcache flush in slow GUP ensures
that data written before the GUP is visible to the
reader.
In the write path, e.g. for PTRACE_POKE, __access_remote_vm
flushes after the write has been done, so the other
process will see the data on the next read.
It looks like process_vm_writev() might be missing that
cache flush after a write, though.
However, now we have a situation where:
1) Only a few oddball callers of GUP need the dcache flush, and
2) Those need to do their OWN flush after a write, and
3) The dcache flush is not needed for most other GUP callers.
We could do two different things here to solve the
asymmetry.
Either we could add dcache flushing to
unpin_user_pages_dirty_lock(), and make sure all
the callers that need to call that function, or
we could move the cache flushing to the GUP callers.
Then we have KSM, which uses GUP, and which would like
to see the latest page contents...
>
> Maybe we should in fact base this on the gup-fast batching series and
> also use
> the helper in there (passing vma=NULL)?
I just moved this code out of __get_user_pages(),
but if we can get rid of it completely by using
a different helper, all the better.
Which GUP-fast batching series are you talking
about, here?
In Yeoreum Yun's series, the batch filling seems
to be done inline in gup_fast_pte_range(), and not
in a helper:
https://lore.kernel.org/all/20261102-refactor_gup_fast-v2-4-bc1ac018c834@arm.com/
Should both paths be using the same gup_fill_pages()
helper?
The cache flushing part is broken out into its own
helper function later in this series, so we could
sequence those differently.
--
All Rights Reversed.
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 1/8] mm/gup: break out gup_fill_pages() helper
2026-11-03 16:29 ` Rik van Riel
@ 2026-11-03 16:39 ` David Hildenbrand (Arm)
2026-11-03 17:00 ` Rik van Riel
0 siblings, 1 reply; 18+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-03 16:39 UTC (permalink / raw)
To: Rik van Riel, linux-kernel
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On 11/3/26 17:29, Rik van Riel wrote:
> On Tue, 2026-11-03 at 09:43 +0100, David Hildenbrand (Arm) wrote:
>> On 11/2/26 20:30, Rik van Riel wrote:
>>> __get_user_pages() fills pages[] and flushes each page's caches in
>>> an
>>> open-coded loop.
>>>
>>> Move it into a gup_fill_pages() helper, which the
>>> follow_page_mask()
>>> call chain can then use to fill its own pages[] slots.
>>>
>>> No functional changes intended.
>>>
>>> Suggested-by: David Hildenbrand <david@kernel.org>
>>> Reviewed-by: Suren Baghdasaryan <surenb@google.com>
>>> Signed-off-by: Rik van Riel <riel@surriel.com>
>>> ---
>>> mm/gup.c | 27 ++++++++++++++++++---------
>>> 1 file changed, 18 insertions(+), 9 deletions(-)
>>>
>>> diff --git a/mm/gup.c b/mm/gup.c
>>> index 7d43ce717b246..cdbe2ed3e6442 100644
>>> --- a/mm/gup.c
>>> +++ b/mm/gup.c
>>> @@ -633,6 +633,23 @@ static struct page *no_page_table(struct
>>> vm_area_struct *vma,
>>> return NULL;
>>> }
>>>
>>> +static void gup_fill_pages(struct vm_area_struct *vma, unsigned
>>> long address,
>>> + struct page *page, unsigned long nr, struct page
>>> **pages)
>>
>> unsigned long nr_pages ?
>
> Will do. Thank you.
>>
>>
>>> +{
>>> + unsigned long i;
>>> +
>>> + if (!pages)
>>> + return;
>>
>> Why is that check needed? Why should someone try filling something
>> that doesn't
>> exist?
>
> faultin_page_range() and populate_vma_page_range
> both call the GUP code with a NULL pages pointer.
That's not what I meant. I know that we have GUP users without
pages array.
In the code change you are performing the check is not required.
That's why there wasn't one in the first place.
>
>>
>>> +
>>> + for (i = 0; i < nr; i++) {
>>> + struct page *subpage = page + i;
>>
>> There is no such thing as a subpage as Willy keeps saying :)
>>
>> Why not reuse "page" ?
>>
>> for (i = 0; i < nr; i++, page++) {
>
> Will do.
>
>>
>>> +
>>> + pages[i] = subpage;
>>> + flush_anon_page(vma, subpage, address + i *
>>> PAGE_SIZE);
>>> + flush_dcache_page(subpage);
>>
>> Now I wonder why GUP-fast doesn't need any of that. Likely because
>> architectures
>> that support GUP-fast don't need any such magic.
>
> These cache flushes seem to be common on architectures
> with virtually indexed caches, where the cache needs
> to be flushed if the memory is accessed through a
> different virtual address than where userspace might
> have written data.
>
> On 32 bit ARM, MIPS, and sh, the kernel seems to
> implement an actual cache flush here, but also
> claims to support GUP-fast.
>
> I don't understand how that would possibly be safe,
> since the pages returned by GUP-fast could be
> accessed under a different cache alias than where
> userspace accesses the memory.
Right. What a mess. Once again :)
>
> I think one reason GUP-fast can get away without
> a dcache flush is that most of the callers take
> care of that, for example by using raw_copy_to_user()
> to copy data using the user's virtual address, which
> takes care of VIVT cache aliasing.
>
> Likewise, drivers that do zero-copy DMA into user
> memory will get the cache flushed on VIVT architectures
> inside dma_unmap().
>
> The dcache flush inside GUP covers rarer cases, where
> the kernel accesses memory at a different address than
> where userspace wrote to that memory.
>
> cat /proc/<pid>/cmdline KERNEL
> ---- -------------------
> read(buf=U)
> -------------------------------->
> get_mm_cmdline(mm, U):
> access_remote_vm(mm, arg_start,
> kbuf, len, FOLL_ANON):
> get_user_page_vma_remote()
> (slow GUP + its flushes)
> kmap_local_folio(P)
> memcpy(kbuf <- K) // read
> kunmap_local(K)
> copy_to_user(U, kbuf) // write
> through U itself
> read buf at U
> ------>
> clean: the final write to U went
> through U. Same line. No hazard.
>
> In this case, the dcache flush in slow GUP ensures
> that data written before the GUP is visible to the
> reader.
>
> In the write path, e.g. for PTRACE_POKE, __access_remote_vm
> flushes after the write has been done, so the other
> process will see the data on the next read.
>
> It looks like process_vm_writev() might be missing that
> cache flush after a write, though.
:(
>
> However, now we have a situation where:
> 1) Only a few oddball callers of GUP need the dcache flush, and
> 2) Those need to do their OWN flush after a write, and
> 3) The dcache flush is not needed for most other GUP callers.
>
> We could do two different things here to solve the
> asymmetry.
>
> Either we could add dcache flushing to
> unpin_user_pages_dirty_lock(), and make sure all
> the callers that need to call that function, or
> we could move the cache flushing to the GUP callers.
>
> Then we have KSM, which uses GUP, and which would like
> to see the latest page contents...
It's all very inconsistent. And undocumented. And error prone.
Could we have GUP-fast also flush the dcache for consistency or
must that only be called with the PT lock held?
>
>>
>> Maybe we should in fact base this on the gup-fast batching series and
>> also use
>> the helper in there (passing vma=NULL)?
>
> I just moved this code out of __get_user_pages(),
> but if we can get rid of it completely by using
> a different helper, all the better.
>
> Which GUP-fast batching series are you talking
> about, here?
>
> In Yeoreum Yun's series, the batch filling seems
> to be done inline in gup_fast_pte_range(), and not
> in a helper:
>
> https://lore.kernel.org/all/20261102-refactor_gup_fast-v2-4-bc1ac018c834@arm.com/
>
> Should both paths be using the same gup_fill_pages()
> helper?
We also have gup_fast_pmd_leaf that fills an array. I think all of these should
just use a helper (instead of manually looping).
For now we can distinguish gup vs. gup-fast using the VMA argument.
--
Cheers,
David
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 1/8] mm/gup: break out gup_fill_pages() helper
2026-11-03 16:39 ` David Hildenbrand (Arm)
@ 2026-11-03 17:00 ` Rik van Riel
2026-11-03 17:04 ` David Hildenbrand (Arm)
0 siblings, 1 reply; 18+ messages in thread
From: Rik van Riel @ 2026-11-03 17:00 UTC (permalink / raw)
To: David Hildenbrand (Arm), linux-kernel
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On Tue, 2026-11-03 at 17:39 +0100, David Hildenbrand (Arm) wrote:
> On 11/3/26 17:29, Rik van Riel wrote:
> > On Tue, 2026-11-03 at 09:43 +0100, David Hildenbrand (Arm) wrote:
> > > On 11/2/26 20:30, Rik van Riel wrote:
> > >
> > >
> > >
> > > > +{
> > > > + unsigned long i;
> > > > +
> > > > + if (!pages)
> > > > + return;
> > >
> > > Why is that check needed? Why should someone try filling
> > > something
> > > that doesn't
> > > exist?
> >
> > faultin_page_range() and populate_vma_page_range
> > both call the GUP code with a NULL pages pointer.
>
> That's not what I meant. I know that we have GUP users without
> pages array.
>
> In the code change you are performing the check is not required.
>
> That's why there wasn't one in the first place.
It looks like 2 of the 3 callers of gup_fill_pages()
check for pages existing. The other does not.
Should we just have the check in this one place, and
not depend on the caller?
>
>
> >
> > However, now we have a situation where:
> > 1) Only a few oddball callers of GUP need the dcache flush, and
> > 2) Those need to do their OWN flush after a write, and
> > 3) The dcache flush is not needed for most other GUP callers.
> >
> > We could do two different things here to solve the
> > asymmetry.
> >
> > Either we could add dcache flushing to
> > unpin_user_pages_dirty_lock(), and make sure all
> > the callers that need to call that function, or
> > we could move the cache flushing to the GUP callers.
> >
> > Then we have KSM, which uses GUP, and which would like
> > to see the latest page contents...
>
> It's all very inconsistent. And undocumented. And error prone.
>
> Could we have GUP-fast also flush the dcache for consistency or
> must that only be called with the PT lock held?
I think we could, but it does not look like any of
them need it.
Out of the 83 callers of gup_fast, many DMA into the
page, and the drivers do their own flushing. Also,
flushing before the DMA write doesn't help.
KVM, Infiniband, and others pin the pages, sometimes
for longer periods of time, and either flush around
their reads and writes, or run only on architectures
where we don't need to.
The others access the page through the user's own
virtual address, which again does not require a
dcache flush.
Only a small number of slow GUP callers need dcache
flushes.
It may make more sense to move the cache flushes
all the way out into __get_user_pages() and away
from the follow_page_mask() call chain, so we can
maybe merge more of the helper functions between
slow GUP and fast GUP.
> >
> > In Yeoreum Yun's series, the batch filling seems
> > to be done inline in gup_fast_pte_range(), and not
> > in a helper:
> >
> > https://lore.kernel.org/all/20261102-refactor_gup_fast-v2-4-bc1ac018c834@arm.com/
> >
> > Should both paths be using the same gup_fill_pages()
> > helper?
>
> We also have gup_fast_pmd_leaf that fills an array. I think all of
> these should
> just use a helper (instead of manually looping).
>
> For now we can distinguish gup vs. gup-fast using the VMA argument.
I think all of those could use the same helper
function I am breaking out here, once the dcache
flushing moves to its own helper.
--
All Rights Reversed.
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 1/8] mm/gup: break out gup_fill_pages() helper
2026-11-03 17:00 ` Rik van Riel
@ 2026-11-03 17:04 ` David Hildenbrand (Arm)
2026-11-03 19:30 ` Rik van Riel
0 siblings, 1 reply; 18+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-03 17:04 UTC (permalink / raw)
To: Rik van Riel, linux-kernel
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On 11/3/26 18:00, Rik van Riel wrote:
> On Tue, 2026-11-03 at 17:39 +0100, David Hildenbrand (Arm) wrote:
>> On 11/3/26 17:29, Rik van Riel wrote:
>>>
>>> faultin_page_range() and populate_vma_page_range
>>> both call the GUP code with a NULL pages pointer.
>>
>> That's not what I meant. I know that we have GUP users without
>> pages array.
>>
>> In the code change you are performing the check is not required.
>>
>> That's why there wasn't one in the first place.
>
> It looks like 2 of the 3 callers of gup_fill_pages()
> check for pages existing. The other does not.
>
> Should we just have the check in this one place, and
> not depend on the caller?
>
That should be spelled out in the patch description.
The change as it stands in this patch looks useless :)
Although trying to fill something that doesn't exist is a bit weird.
But if it makes the calling code nicer, sure.
>>
>>
>>>
>>> However, now we have a situation where:
>>> 1) Only a few oddball callers of GUP need the dcache flush, and
>>> 2) Those need to do their OWN flush after a write, and
>>> 3) The dcache flush is not needed for most other GUP callers.
>>>
>>> We could do two different things here to solve the
>>> asymmetry.
>>>
>>> Either we could add dcache flushing to
>>> unpin_user_pages_dirty_lock(), and make sure all
>>> the callers that need to call that function, or
>>> we could move the cache flushing to the GUP callers.
>>>
>>> Then we have KSM, which uses GUP, and which would like
>>> to see the latest page contents...
>>
>> It's all very inconsistent. And undocumented. And error prone.
>>
>> Could we have GUP-fast also flush the dcache for consistency or
>> must that only be called with the PT lock held?
>
> I think we could, but it does not look like any of
> them need it.
Sure, but it makes the code more consistent and from a GUP PoV easier to understand.
>
> Out of the 83 callers of gup_fast, many DMA into the
> page, and the drivers do their own flushing. Also,
> flushing before the DMA write doesn't help.
>
> KVM, Infiniband, and others pin the pages, sometimes
> for longer periods of time, and either flush around
> their reads and writes, or run only on architectures
> where we don't need to.
>
> The others access the page through the user's own
> virtual address, which again does not require a
> dcache flush.
>
> Only a small number of slow GUP callers need dcache
> flushes.
>
> It may make more sense to move the cache flushes
> all the way out into __get_user_pages() and away
> from the follow_page_mask() call chain, so we can
> maybe merge more of the helper functions between
> slow GUP and fast GUP.
That's likely more work, can you look into that?
--
Cheers,
David
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/8] mm/gup: break out gup_fill_pages() helper
2026-11-03 17:04 ` David Hildenbrand (Arm)
@ 2026-11-03 19:30 ` Rik van Riel
0 siblings, 0 replies; 18+ messages in thread
From: Rik van Riel @ 2026-11-03 19:30 UTC (permalink / raw)
To: David Hildenbrand (Arm), linux-kernel
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On Tue, 2026-11-03 at 18:04 +0100, David Hildenbrand (Arm) wrote:
> On 11/3/26 18:00, Rik van Riel wrote:
> > On Tue, 2026-11-03 at 17:39 +0100, David Hildenbrand (Arm) wrote:
> > > On 11/3/26 17:29, Rik van Riel wrote:
> > > >
> > > > faultin_page_range() and populate_vma_page_range
> > > > both call the GUP code with a NULL pages pointer.
> > >
> > > That's not what I meant. I know that we have GUP users without
> > > pages array.
> > >
> > > In the code change you are performing the check is not required.
> > >
> > > That's why there wasn't one in the first place.
> >
> > It looks like 2 of the 3 callers of gup_fill_pages()
> > check for pages existing. The other does not.
> >
> > Should we just have the check in this one place, and
> > not depend on the caller?
> >
>
> That should be spelled out in the patch description.
>
> The change as it stands in this patch looks useless :)
> Although trying to fill something that doesn't exist is a bit weird.
>
> But if it makes the calling code nicer, sure.
>
I agree this patch by itself isn't super useful.
It's goal is to be able to have this code in one
spot, and not in multiple places in gup.c
I'll split out the array filling from the cache
flushing, so we can move the cache flushing out
from under the lock, and have GUP-fast and regular
GUP share the same helper.
> >
> >
> > Only a small number of slow GUP callers need dcache
> > flushes.
> >
> > It may make more sense to move the cache flushes
> > all the way out into __get_user_pages() and away
> > from the follow_page_mask() call chain, so we can
> > maybe merge more of the helper functions between
> > slow GUP and fast GUP.
>
> That's likely more work, can you look into that?
>
It really doesn't look too bad. Here are all
the GUP callers that need a cache flush:
- kernel/events/uprobes.c (through get_user_pages_remote)
- /proc/pid/cmdline (through access_remote_vm)
- fs/proc/base.c (through access_remote_vm)
- kernel/trace/trace_events_user.c (pin_user_pages_remote)
- arch/x86/kernel/cpu/sgx/ioctl.c (get_user_pages ... urgh)
- kernel/ptrace.c (access_remote_vm)
- mm/process_vm_access.c (pin_user_pages_remote)
- fs/exec.c (pin_user_pages_remote)
- virt/kvm/async_pf.c (get_user_pages_remote)
In other words, if we change the SGX code to just
call get_user_pages_remote(), and then move the
cache flushing to get_user_pages_remote() and
pin_user_pages_remote() we should be all set.
Want me to fold that into this series, or send
it in as a small, separate prequel series?
As a separate series it would:
1) split out the gup_fill_pages() and gup_flush_pages()
helpers
2) convert the SGX code to use get_user_pages_remote()
3) move the flush call up the stack into pin_user_pages_remote()
and get_user_pages_remote()
Just let me know if you prefer that separate, or
folded in.
--
All Rights Reversed.
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 2/8] mm/gup: convert follow_page_mask() to return a long
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
2026-11-02 19:30 ` [PATCH 1/8] mm/gup: break out gup_fill_pages() helper Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-02 19:30 ` [PATCH 3/8] mm/gup: split out folio grab-and-store helper Rik van Riel
` (5 subsequent siblings)
7 siblings, 0 replies; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
follow_page_mask() and its helpers return a struct page pointer: NULL,
ERR_PTR(), or the page found. Change the return type to long instead:
0, a negative errno, or 1 with the page stored in a new pages[0] slot.
This lets the return value carry a page count rather than a single
struct page pointer.
follow_huge_pud(), follow_huge_pmd() and follow_page_pte() now store
the found page in a pages[0] slot. __get_user_pages() itself is
unchanged: it reads that page back from pages[i] and expands a large
folio's remaining subpages on its own, as before. The vsyscall gate
area bypasses follow_page_mask() and fills its own slot the same way.
*page_mask and __get_user_pages()'s handling of a large folio's
remaining subpages are untouched, and mm/gup_test.c
(PIN_LONGTERM_BENCHMARK) shows no measurable difference for 4 kB,
64 kB mTHP, or 2 MB THP.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 254 +++++++++++++++++++++++++++++--------------------------
1 file changed, 134 insertions(+), 120 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index cdbe2ed3e6442..f7d940e7e5ffb 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -608,15 +608,15 @@ static inline bool can_follow_write_common(struct page *page,
return page && PageAnon(page) && PageAnonExclusive(page);
}
-static struct page *no_page_table(struct vm_area_struct *vma,
- unsigned int flags, unsigned long address)
+static long no_page_table(struct vm_area_struct *vma,
+ unsigned int flags, unsigned long address)
{
if (!(flags & FOLL_DUMP))
- return NULL;
+ return 0;
/*
* When core dumping, we don't want to allocate unnecessary pages or
- * page tables. Return error instead of NULL to skip handle_mm_fault,
+ * page tables. Return error instead of 0 to skip handle_mm_fault,
* then get_dump_page() will return NULL to leave a hole in the dump.
* But we can only make this optimization where a hole would surely
* be zero-filled if handle_mm_fault() actually did handle it.
@@ -625,12 +625,12 @@ static struct page *no_page_table(struct vm_area_struct *vma,
struct hstate *h = hstate_vma(vma);
if (!hugetlbfs_pagecache_present(h, vma, address))
- return ERR_PTR(-EFAULT);
+ return -EFAULT;
} else if ((vma_is_anonymous(vma) || !vma->vm_ops->fault)) {
- return ERR_PTR(-EFAULT);
+ return -EFAULT;
}
- return NULL;
+ return 0;
}
static void gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
@@ -663,9 +663,10 @@ static inline bool can_follow_write_pud(pud_t pud, struct page *page,
return can_follow_write_common(page, vma, flags);
}
-static struct page *follow_huge_pud(struct vm_area_struct *vma,
- unsigned long addr, pud_t *pudp,
- int flags, unsigned long *page_mask)
+static long follow_huge_pud(struct vm_area_struct *vma,
+ unsigned long addr, pud_t *pudp,
+ unsigned int flags, unsigned long *page_mask,
+ struct page **pages)
{
struct mm_struct *mm = vma->vm_mm;
struct page *page;
@@ -676,25 +677,27 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma,
assert_spin_locked(pud_lockptr(mm, pudp));
if (!pud_present(pud))
- return NULL;
+ return 0;
if ((flags & FOLL_WRITE) &&
!can_follow_write_pud(pud, pfn_to_page(pfn), vma, flags))
- return NULL;
+ return 0;
pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT;
page = pfn_to_page(pfn);
if (!pud_write(pud) && gup_must_unshare(vma, flags, page))
- return ERR_PTR(-EMLINK);
+ return -EMLINK;
ret = try_grab_folio(page_folio(page), 1, flags);
if (ret)
- page = ERR_PTR(ret);
- else
- *page_mask = HPAGE_PUD_NR - 1;
+ return ret;
+
+ *page_mask = HPAGE_PUD_NR - 1;
- return page;
+ gup_fill_pages(vma, addr, page, 1, pages);
+
+ return 1;
}
/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */
@@ -715,10 +718,10 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
return !userfaultfd_huge_pmd_wp(vma, pmd);
}
-static struct page *follow_huge_pmd(struct vm_area_struct *vma,
- unsigned long addr, pmd_t *pmd,
- unsigned int flags,
- unsigned long *page_mask)
+static long follow_huge_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd,
+ unsigned int flags, unsigned long *page_mask,
+ struct page **pages)
{
struct mm_struct *mm = vma->vm_mm;
pmd_t pmdval = *pmd;
@@ -730,24 +733,24 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma,
page = pmd_page(pmdval);
if ((flags & FOLL_WRITE) &&
!can_follow_write_pmd(pmdval, page, vma, flags))
- return NULL;
+ return 0;
/* Avoid dumping huge zero page */
if ((flags & FOLL_DUMP) && is_huge_zero_pmd(pmdval))
- return ERR_PTR(-EFAULT);
+ return -EFAULT;
if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags))
- return NULL;
+ return 0;
if (!pmd_write(pmdval) && gup_must_unshare(vma, flags, page))
- return ERR_PTR(-EMLINK);
+ return -EMLINK;
VM_WARN_ON_ONCE_PAGE((flags & FOLL_PIN) && PageAnon(page) &&
!PageAnonExclusive(page), page);
ret = try_grab_folio(page_folio(page), 1, flags);
if (ret)
- return ERR_PTR(ret);
+ return ret;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
if (pmd_trans_huge(pmdval) && (flags & FOLL_TOUCH))
@@ -757,23 +760,26 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma,
page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
*page_mask = HPAGE_PMD_NR - 1;
- return page;
+ gup_fill_pages(vma, addr, page, 1, pages);
+
+ return 1;
}
#else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */
-static struct page *follow_huge_pud(struct vm_area_struct *vma,
- unsigned long addr, pud_t *pudp,
- int flags, unsigned long *page_mask)
+static long follow_huge_pud(struct vm_area_struct *vma,
+ unsigned long addr, pud_t *pudp,
+ unsigned int flags, unsigned long *page_mask,
+ struct page **pages)
{
- return NULL;
+ return 0;
}
-static struct page *follow_huge_pmd(struct vm_area_struct *vma,
- unsigned long addr, pmd_t *pmd,
- unsigned int flags,
- unsigned long *page_mask)
+static long follow_huge_pmd(struct vm_area_struct *vma,
+ unsigned long addr, pmd_t *pmd,
+ unsigned int flags, unsigned long *page_mask,
+ struct page **pages)
{
- return NULL;
+ return 0;
}
#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */
@@ -816,15 +822,16 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
return !userfaultfd_pte_wp(vma, pte);
}
-static struct page *follow_page_pte(struct vm_area_struct *vma,
- unsigned long address, pmd_t *pmd, unsigned int flags)
+static long follow_page_pte(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmd, unsigned int flags,
+ struct page **pages)
{
struct mm_struct *mm = vma->vm_mm;
struct folio *folio;
struct page *page;
spinlock_t *ptl;
pte_t *ptep, pte;
- int ret;
+ long ret;
ptep = pte_offset_map_lock(mm, pmd, address, &ptl);
if (!ptep)
@@ -842,14 +849,14 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
*/
if ((flags & FOLL_WRITE) &&
!can_follow_write_pte(pte, page, vma, flags)) {
- page = NULL;
+ ret = 0;
goto out;
}
if (unlikely(!page)) {
if (flags & FOLL_DUMP) {
/* Avoid special (like zero) pages in core dumps */
- page = ERR_PTR(-EFAULT);
+ ret = -EFAULT;
goto out;
}
@@ -857,14 +864,13 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
page = pte_page(pte);
} else {
ret = follow_pfn_pte(vma, address, ptep, flags);
- page = ERR_PTR(ret);
goto out;
}
}
folio = page_folio(page);
if (!pte_write(pte) && gup_must_unshare(vma, flags, page)) {
- page = ERR_PTR(-EMLINK);
+ ret = -EMLINK;
goto out;
}
@@ -873,10 +879,8 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
/* try_grab_folio() does nothing unless FOLL_GET or FOLL_PIN is set. */
ret = try_grab_folio(folio, 1, flags);
- if (unlikely(ret)) {
- page = ERR_PTR(ret);
+ if (unlikely(ret))
goto out;
- }
/*
* We need to make the page accessible if and only if we are going
@@ -886,8 +890,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
if (flags & FOLL_PIN) {
ret = arch_make_folio_accessible(folio);
if (ret) {
- unpin_user_page(page);
- page = ERR_PTR(ret);
+ gup_put_folio(folio, 1, flags);
goto out;
}
}
@@ -902,24 +905,27 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
*/
folio_mark_accessed(folio);
}
+
+ gup_fill_pages(vma, address, page, 1, pages);
+ ret = 1;
out:
pte_unmap_unlock(ptep, ptl);
- return page;
+ return ret;
no_page:
pte_unmap_unlock(ptep, ptl);
if (!pte_none(pte))
- return NULL;
+ return 0;
return no_page_table(vma, flags, address);
}
-static struct page *follow_pmd_mask(struct vm_area_struct *vma,
- unsigned long address, pud_t *pudp,
- unsigned int flags,
- unsigned long *page_mask)
+static long follow_pmd_mask(struct vm_area_struct *vma,
+ unsigned long address, pud_t *pudp,
+ unsigned int flags, unsigned long *page_mask,
+ struct page **pages)
{
pmd_t *pmd, pmdval;
spinlock_t *ptl;
- struct page *page;
+ long ret;
struct mm_struct *mm = vma->vm_mm;
pmd = pmd_offset(pudp, address);
@@ -929,7 +935,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma,
if (!pmd_present(pmdval))
return no_page_table(vma, flags, address);
if (likely(!pmd_leaf(pmdval)))
- return follow_page_pte(vma, address, pmd, flags);
+ return follow_page_pte(vma, address, pmd, flags, pages);
if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags))
return no_page_table(vma, flags, address);
@@ -942,28 +948,28 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma,
}
if (unlikely(!pmd_leaf(pmdval))) {
spin_unlock(ptl);
- return follow_page_pte(vma, address, pmd, flags);
+ return follow_page_pte(vma, address, pmd, flags, pages);
}
if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) {
spin_unlock(ptl);
split_huge_pmd(vma, pmd, address);
/* If pmd was left empty, stuff a page table in there quickly */
- return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) :
- follow_page_pte(vma, address, pmd, flags);
+ return pte_alloc(mm, pmd) ? -ENOMEM :
+ follow_page_pte(vma, address, pmd, flags, pages);
}
- page = follow_huge_pmd(vma, address, pmd, flags, page_mask);
+ ret = follow_huge_pmd(vma, address, pmd, flags, page_mask, pages);
spin_unlock(ptl);
- return page;
+ return ret;
}
-static struct page *follow_pud_mask(struct vm_area_struct *vma,
- unsigned long address, p4d_t *p4dp,
- unsigned int flags,
- unsigned long *page_mask)
+static long follow_pud_mask(struct vm_area_struct *vma,
+ unsigned long address, p4d_t *p4dp,
+ unsigned int flags, unsigned long *page_mask,
+ struct page **pages)
{
pud_t *pudp, pud;
spinlock_t *ptl;
- struct page *page;
+ long ret;
struct mm_struct *mm = vma->vm_mm;
pudp = pud_offset(p4dp, address);
@@ -972,22 +978,22 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma,
return no_page_table(vma, flags, address);
if (pud_leaf(pud)) {
ptl = pud_lock(mm, pudp);
- page = follow_huge_pud(vma, address, pudp, flags, page_mask);
+ ret = follow_huge_pud(vma, address, pudp, flags, page_mask, pages);
spin_unlock(ptl);
- if (page)
- return page;
+ if (ret)
+ return ret;
return no_page_table(vma, flags, address);
}
if (unlikely(pud_bad(pud)))
return no_page_table(vma, flags, address);
- return follow_pmd_mask(vma, address, pudp, flags, page_mask);
+ return follow_pmd_mask(vma, address, pudp, flags, page_mask, pages);
}
-static struct page *follow_p4d_mask(struct vm_area_struct *vma,
- unsigned long address, pgd_t *pgdp,
- unsigned int flags,
- unsigned long *page_mask)
+static long follow_p4d_mask(struct vm_area_struct *vma,
+ unsigned long address, pgd_t *pgdp,
+ unsigned int flags, unsigned long *page_mask,
+ struct page **pages)
{
p4d_t *p4dp, p4d;
@@ -998,7 +1004,7 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
if (!p4d_present(p4d) || p4d_bad(p4d))
return no_page_table(vma, flags, address);
- return follow_pud_mask(vma, address, p4dp, flags, page_mask);
+ return follow_pud_mask(vma, address, p4dp, flags, page_mask, pages);
}
/**
@@ -1007,6 +1013,9 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
* @address: virtual address to look up
* @flags: flags modifying lookup behaviour
* @page_mask: a pointer to output page_mask
+ * @pages: array to receive the page found, refcounted per @flags, or NULL
+ * to walk the page tables (e.g. to fault pages in) without
+ * collecting or refcounting them
*
* @flags can have FOLL_ flags set, defined in <linux/mm.h>
*
@@ -1017,17 +1026,17 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
*
* On output, @page_mask is set according to the size of the page.
*
- * Return: the mapped (struct page *), %NULL if no mapping exists, or
- * an error pointer if there is a mapping to something not represented
- * by a page descriptor (see also vm_normal_page()).
+ * Return: 1 with @pages[0] filled in if a page was found, 0 if no mapping
+ * exists at @address, or a negative errno for a mapping to something not
+ * represented by a page descriptor (see also vm_normal_page()).
*/
-static struct page *follow_page_mask(struct vm_area_struct *vma,
- unsigned long address, unsigned int flags,
- unsigned long *page_mask)
+static long follow_page_mask(struct vm_area_struct *vma,
+ unsigned long address, unsigned int flags,
+ unsigned long *page_mask, struct page **pages)
{
pgd_t *pgd;
struct mm_struct *mm = vma->vm_mm;
- struct page *page;
+ long ret;
vma_pgtable_walk_begin(vma);
@@ -1035,13 +1044,13 @@ static struct page *follow_page_mask(struct vm_area_struct *vma,
pgd = pgd_offset(mm, address);
if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
- page = no_page_table(vma, flags, address);
+ ret = no_page_table(vma, flags, address);
else
- page = follow_p4d_mask(vma, address, pgd, flags, page_mask);
+ ret = follow_p4d_mask(vma, address, pgd, flags, page_mask, pages);
vma_pgtable_walk_end(vma);
- return page;
+ return ret;
}
static int get_gate_page(struct mm_struct *mm, unsigned long address,
@@ -1391,6 +1400,7 @@ static long __get_user_pages(struct mm_struct *mm,
do {
struct page *page;
unsigned int page_increm;
+ long nr;
/* first iteration or cross vma bound */
if (!vma || start >= vma->vm_end) {
@@ -1417,8 +1427,12 @@ static long __get_user_pages(struct mm_struct *mm,
pages ? &page : NULL);
if (ret)
goto out;
- page_mask = 0;
- goto next_page;
+ gup_fill_pages(vma, start, page, 1,
+ pages ? pages + i : NULL);
+ i++;
+ start += PAGE_SIZE;
+ nr_pages--;
+ continue;
}
if (!vma) {
@@ -1440,10 +1454,11 @@ static long __get_user_pages(struct mm_struct *mm,
}
cond_resched();
- page = follow_page_mask(vma, start, gup_flags, &page_mask);
- if (!page || PTR_ERR(page) == -EMLINK) {
+ nr = follow_page_mask(vma, start, gup_flags, &page_mask,
+ pages ? &pages[i] : NULL);
+ if (!nr || nr == -EMLINK) {
ret = faultin_page(vma, start, gup_flags,
- PTR_ERR(page) == -EMLINK, locked);
+ nr == -EMLINK, locked);
switch (ret) {
case 0:
goto retry;
@@ -1457,7 +1472,7 @@ static long __get_user_pages(struct mm_struct *mm,
goto out;
}
BUG();
- } else if (PTR_ERR(page) == -EEXIST) {
+ } else if (nr == -EEXIST) {
/*
* Proper page table entry exists, but no corresponding
* struct page. If the caller expects **pages to be
@@ -1465,49 +1480,48 @@ static long __get_user_pages(struct mm_struct *mm,
* for this page.
*/
if (pages) {
- ret = PTR_ERR(page);
+ ret = nr;
goto out;
}
- } else if (IS_ERR(page)) {
- ret = PTR_ERR(page);
+ } else if (nr < 0) {
+ ret = nr;
goto out;
}
-next_page:
+
page_increm = 1 + (~(start >> PAGE_SHIFT) & page_mask);
if (page_increm > nr_pages)
page_increm = nr_pages;
- if (pages) {
+ /*
+ * This must be a large folio (and doesn't need to
+ * be the whole folio; it can be part of it), do
+ * the refcount work for all the subpages too.
+ *
+ * NOTE: here the page may not be the head page
+ * e.g. when start addr is not thp-size aligned.
+ * try_grab_folio() should have taken care of tail
+ * pages.
+ */
+ if (pages && page_increm > 1) {
+ struct folio *folio = page_folio(pages[i]);
+
/*
- * This must be a large folio (and doesn't need to
- * be the whole folio; it can be part of it), do
- * the refcount work for all the subpages too.
- *
- * NOTE: here the page may not be the head page
- * e.g. when start addr is not thp-size aligned.
- * try_grab_folio() should have taken care of tail
- * pages.
+ * Since we already hold refcount on the
+ * large folio, this should never fail.
*/
- if (page_increm > 1) {
- struct folio *folio = page_folio(page);
-
+ if (try_grab_folio(folio, page_increm - 1,
+ gup_flags)) {
/*
- * Since we already hold refcount on the
- * large folio, this should never fail.
+ * Release the 1st page ref if the
+ * folio is problematic, fail hard.
*/
- if (try_grab_folio(folio, page_increm - 1,
- gup_flags)) {
- /*
- * Release the 1st page ref if the
- * folio is problematic, fail hard.
- */
- gup_put_folio(folio, 1, gup_flags);
- ret = -EFAULT;
- goto out;
- }
+ gup_put_folio(folio, 1, gup_flags);
+ ret = -EFAULT;
+ goto out;
}
- gup_fill_pages(vma, start, page, page_increm, pages + i);
+ gup_fill_pages(vma, start + PAGE_SIZE, pages[i] + 1,
+ page_increm - 1, pages + i + 1);
}
i += page_increm;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH 3/8] mm/gup: split out folio grab-and-store helper
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
2026-11-02 19:30 ` [PATCH 1/8] mm/gup: break out gup_fill_pages() helper Rik van Riel
2026-11-02 19:30 ` [PATCH 2/8] mm/gup: convert follow_page_mask() to return a long Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-02 19:30 ` [PATCH 4/8] mm/gup: break out gup_lookup_pte() helper Rik van Riel
` (4 subsequent siblings)
7 siblings, 0 replies; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
follow_page_pte() does two things once it has resolved a present PTE
to a page: run the per-PTE safety checks (write-fault, unshare), then
grab a folio reference and hand the page back: fault it in if pinning,
mark it dirty/accessed, and fill the caller's array slot.
Split the second part into gup_grab_folio_pages(), which
takes its inputs as parameters, so the checks and the grab-and-store
can be applied at different granularities. The name states what the
helper does, not 'commit', and it takes the PTE's dirtiness as a bool.
The helper stores the pages but leaves the cache flushes to the
caller: split them into gup_flush_pages(), which follow_page_pte()
runs after dropping the PTE lock. Storing one slot under the lock is
cheap; flushing is not, and needs no lock once the folio reference
is held.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 101 +++++++++++++++++++++++++++++++++++++++----------------
1 file changed, 72 insertions(+), 29 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index f7d940e7e5ffb..27ae2a1d7c36d 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -650,6 +650,25 @@ static void gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
}
}
+/*
+ * Flush the caches for @nr pages stored in @pages. The caller already
+ * holds folio references, so the pages cannot go away; this runs
+ * without any page table lock held.
+ */
+static void gup_flush_pages(struct vm_area_struct *vma, unsigned long address,
+ struct page **pages, unsigned long nr)
+{
+ unsigned long i;
+
+ if (!pages)
+ return;
+
+ for (i = 0; i < nr; i++) {
+ flush_anon_page(vma, pages[i], address + i * PAGE_SIZE);
+ flush_dcache_page(pages[i]);
+ }
+}
+
#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES
/* FOLL_FORCE can write to even unwritable PUDs in COW mappings. */
static inline bool can_follow_write_pud(pud_t pud, struct page *page,
@@ -822,6 +841,54 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
return !userfaultfd_pte_wp(vma, pte);
}
+/*
+ * Grab a folio reference for @nr pages starting at @page, mark the folio
+ * accessed (and dirty for FOLL_WRITE), and store the pages for the caller.
+ * The caller flushes the caches after dropping the page table lock.
+ */
+static long gup_grab_folio_pages(struct folio *folio,
+ struct page *page, unsigned long nr, bool dirty,
+ unsigned int flags, struct page **pages)
+{
+ unsigned long i;
+ long ret;
+
+ /* try_grab_folio() does nothing unless FOLL_GET or FOLL_PIN is set. */
+ ret = try_grab_folio(folio, nr, flags);
+ if (unlikely(ret))
+ return ret;
+
+ /*
+ * We need to make the page accessible if and only if we are going
+ * to access its content (the FOLL_PIN case). Please see
+ * Documentation/core-api/pin_user_pages.rst for details.
+ */
+ if (flags & FOLL_PIN) {
+ ret = arch_make_folio_accessible(folio);
+ if (ret) {
+ gup_put_folio(folio, nr, flags);
+ return ret;
+ }
+ }
+ if (flags & FOLL_TOUCH) {
+ if ((flags & FOLL_WRITE) && !dirty && !folio_test_dirty(folio))
+ folio_mark_dirty(folio);
+ /*
+ * pte_mkyoung() would be more correct here, but atomic care
+ * is needed to avoid losing the dirty bit: it is easier to use
+ * folio_mark_accessed().
+ */
+ folio_mark_accessed(folio);
+ }
+
+ if (pages) {
+ for (i = 0; i < nr; i++)
+ pages[i] = page + i;
+ }
+
+ return 0;
+}
+
static long follow_page_pte(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmd, unsigned int flags,
struct page **pages)
@@ -877,39 +944,15 @@ static long follow_page_pte(struct vm_area_struct *vma,
VM_WARN_ON_ONCE_PAGE((flags & FOLL_PIN) && PageAnon(page) &&
!PageAnonExclusive(page), page);
- /* try_grab_folio() does nothing unless FOLL_GET or FOLL_PIN is set. */
- ret = try_grab_folio(folio, 1, flags);
- if (unlikely(ret))
+ ret = gup_grab_folio_pages(folio, page, 1,
+ pte_dirty(pte), flags, pages);
+ if (ret)
goto out;
-
- /*
- * We need to make the page accessible if and only if we are going
- * to access its content (the FOLL_PIN case). Please see
- * Documentation/core-api/pin_user_pages.rst for details.
- */
- if (flags & FOLL_PIN) {
- ret = arch_make_folio_accessible(folio);
- if (ret) {
- gup_put_folio(folio, 1, flags);
- goto out;
- }
- }
- if (flags & FOLL_TOUCH) {
- if ((flags & FOLL_WRITE) &&
- !pte_dirty(pte) && !folio_test_dirty(folio))
- folio_mark_dirty(folio);
- /*
- * pte_mkyoung() would be more correct here, but atomic care
- * is needed to avoid losing the dirty bit: it is easier to use
- * folio_mark_accessed().
- */
- folio_mark_accessed(folio);
- }
-
- gup_fill_pages(vma, address, page, 1, pages);
ret = 1;
out:
pte_unmap_unlock(ptep, ptl);
+ if (ret > 0)
+ gup_flush_pages(vma, address, pages, ret);
return ret;
no_page:
pte_unmap_unlock(ptep, ptl);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH 4/8] mm/gup: break out gup_lookup_pte() helper
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
` (2 preceding siblings ...)
2026-11-02 19:30 ` [PATCH 3/8] mm/gup: split out folio grab-and-store helper Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-02 19:30 ` [PATCH 5/8] mm/gup: fill the pages array outside the pud/pmd lock Rik van Riel
` (3 subsequent siblings)
7 siblings, 0 replies; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
follow_page_pte() is 92 lines and does two separate things: work out
which page a PTE maps, if any, and commit to the page it found. The
first half reaches the second through five exit paths, two of which
unwind the PTE lock in different ways.
Split the resolve half into gup_lookup_pte(), which returns the page it
resolved, NULL when the PTE cannot be followed, or the errno the caller
must report. follow_page_pte() is left with one unlock and one exit.
no_page_table() can look up the page cache, so the case that needs it is
recorded and the call made after dropping the PTE lock, as before.
A none PTE still returns before any write check runs, so a FOLL_WRITE
failure can never reach the recorded call: the caller consults
no_page_table() for exactly the same PTEs the old code did.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 102 +++++++++++++++++++++++++++++++------------------------
1 file changed, 58 insertions(+), 44 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 27ae2a1d7c36d..c2a4f5b987ced 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -889,76 +889,90 @@ static long gup_grab_folio_pages(struct folio *folio,
return 0;
}
-static long follow_page_pte(struct vm_area_struct *vma,
- unsigned long address, pmd_t *pmd, unsigned int flags,
- struct page **pages)
+/*
+ * Resolve one present PTE to the page it maps. Returns no page and no error
+ * when the PTE cannot be followed but the caller may fault it in, and a
+ * negative errno when the caller must report the failure.
+ */
+static long gup_lookup_pte(struct vm_area_struct *vma, unsigned long address,
+ pte_t *ptep, pte_t pte, unsigned int flags, struct page **pagep)
{
- struct mm_struct *mm = vma->vm_mm;
- struct folio *folio;
struct page *page;
- spinlock_t *ptl;
- pte_t *ptep, pte;
- long ret;
- ptep = pte_offset_map_lock(mm, pmd, address, &ptl);
- if (!ptep)
- return no_page_table(vma, flags, address);
- pte = ptep_get(ptep);
+ *pagep = NULL;
+
if (!pte_present(pte))
- goto no_page;
+ return 0;
if (pte_protnone(pte) && !gup_can_follow_protnone(vma, flags))
- goto no_page;
+ return 0;
page = vm_normal_page(vma, address, pte);
/*
* We only care about anon pages in can_follow_write_pte().
*/
- if ((flags & FOLL_WRITE) &&
- !can_follow_write_pte(pte, page, vma, flags)) {
- ret = 0;
- goto out;
- }
+ if ((flags & FOLL_WRITE) && !can_follow_write_pte(pte, page, vma, flags))
+ return 0;
if (unlikely(!page)) {
if (flags & FOLL_DUMP) {
/* Avoid special (like zero) pages in core dumps */
- ret = -EFAULT;
- goto out;
- }
-
- if (is_zero_pfn(pte_pfn(pte))) {
- page = pte_page(pte);
- } else {
- ret = follow_pfn_pte(vma, address, ptep, flags);
- goto out;
+ return -EFAULT;
}
+ if (!is_zero_pfn(pte_pfn(pte)))
+ return follow_pfn_pte(vma, address, ptep, flags);
+ page = pte_page(pte);
}
- folio = page_folio(page);
- if (!pte_write(pte) && gup_must_unshare(vma, flags, page)) {
- ret = -EMLINK;
- goto out;
- }
+ if (!pte_write(pte) && gup_must_unshare(vma, flags, page))
+ return -EMLINK;
VM_WARN_ON_ONCE_PAGE((flags & FOLL_PIN) && PageAnon(page) &&
!PageAnonExclusive(page), page);
- ret = gup_grab_folio_pages(folio, page, 1,
- pte_dirty(pte), flags, pages);
- if (ret)
- goto out;
- ret = 1;
-out:
+ *pagep = page;
+ return 0;
+}
+
+static long follow_page_pte(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmd, unsigned int flags,
+ struct page **pages)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ bool need_no_page_table = false;
+ struct page *page;
+ spinlock_t *ptl;
+ pte_t *ptep, pte;
+ long ret;
+
+ ptep = pte_offset_map_lock(mm, pmd, address, &ptl);
+ if (!ptep)
+ return no_page_table(vma, flags, address);
+ pte = ptep_get(ptep);
+
+ ret = gup_lookup_pte(vma, address, ptep, pte, flags, &page);
+ if (!ret && page) {
+ ret = gup_grab_folio_pages(page_folio(page), page,
+ 1, pte_dirty(pte), flags, pages);
+ if (!ret)
+ ret = 1;
+ } else if (!ret && pte_none(pte)) {
+ /*
+ * no_page_table() may look up the page cache, so it cannot run
+ * under the PTE lock. A none PTE returns from gup_lookup_pte()
+ * before any write check runs, so this is the same call the old
+ * code made -- not a new failure mode for FOLL_WRITE.
+ */
+ need_no_page_table = true;
+ }
+
pte_unmap_unlock(ptep, ptl);
+
+ if (need_no_page_table)
+ return no_page_table(vma, flags, address);
if (ret > 0)
gup_flush_pages(vma, address, pages, ret);
return ret;
-no_page:
- pte_unmap_unlock(ptep, ptl);
- if (!pte_none(pte))
- return 0;
- return no_page_table(vma, flags, address);
}
static long follow_pmd_mask(struct vm_area_struct *vma,
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH 5/8] mm/gup: fill the pages array outside the pud/pmd lock
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
` (3 preceding siblings ...)
2026-11-02 19:30 ` [PATCH 4/8] mm/gup: break out gup_lookup_pte() helper Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-02 19:30 ` [PATCH 6/8] mm/gup: return a huge page's full count from follow_page_mask() Rik van Riel
` (2 subsequent siblings)
7 siblings, 0 replies; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
follow_huge_pud() and follow_huge_pmd() fill pages[] and flush the page's
caches while still holding the pud or pmd lock. Neither flush_anon_page()
nor flush_dcache_page() needs that lock.
Have the huge paths store the page and let follow_pud_mask() and
follow_pmd_mask() do the fill after they unlock, so the flushes happen
outside the critical section.
This should be safe because try_grab_folio() has already taken a folio
reference before the unlock, so nothing can free the page while the fill
runs, and the fill itself touches neither the page tables nor the pud or
pmd entry it was reached through.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index c2a4f5b987ced..2edffc424f272 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -714,7 +714,8 @@ static long follow_huge_pud(struct vm_area_struct *vma,
*page_mask = HPAGE_PUD_NR - 1;
- gup_fill_pages(vma, addr, page, 1, pages);
+ if (pages)
+ pages[0] = page;
return 1;
}
@@ -779,7 +780,8 @@ static long follow_huge_pmd(struct vm_area_struct *vma,
page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
*page_mask = HPAGE_PMD_NR - 1;
- gup_fill_pages(vma, addr, page, 1, pages);
+ if (pages)
+ pages[0] = page;
return 1;
}
@@ -1016,6 +1018,14 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
}
ret = follow_huge_pmd(vma, address, pmd, flags, page_mask, pages);
spin_unlock(ptl);
+
+ /*
+ * The ref is already held, so the page cannot go away: fill the
+ * array and flush caches without the pmd lock.
+ */
+ if (ret > 0 && pages)
+ gup_fill_pages(vma, address, pages[0], ret, pages);
+
return ret;
}
@@ -1037,6 +1047,12 @@ static long follow_pud_mask(struct vm_area_struct *vma,
ptl = pud_lock(mm, pudp);
ret = follow_huge_pud(vma, address, pudp, flags, page_mask, pages);
spin_unlock(ptl);
+ /*
+ * The ref is already held, so the page cannot go away: fill
+ * the array and flush caches without the lock.
+ */
+ if (ret > 0 && pages)
+ gup_fill_pages(vma, address, pages[0], ret, pages);
if (ret)
return ret;
return no_page_table(vma, flags, address);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH 6/8] mm/gup: return a huge page's full count from follow_page_mask()
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
` (4 preceding siblings ...)
2026-11-02 19:30 ` [PATCH 5/8] mm/gup: fill the pages array outside the pud/pmd lock Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-02 19:30 ` [PATCH 7/8] mm/gup: walk multiple PTEs per follow_page_pte() call Rik van Riel
2026-11-02 19:30 ` [PATCH 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab Rik van Riel
7 siblings, 0 replies; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
follow_huge_pud()/follow_huge_pmd() already know the huge page's full
size but report it via a separate *page_mask output; __get_user_pages()
does a second try_grab_folio() call and subpage loop for everything
past the first page.
Have the huge paths report their count as the return value instead,
clamped to the huge page's size and @end. The merged grab returns
whatever error try_grab_folio() gives, instead of forcing -EFAULT on the
second call's failure.
*page_mask and __get_user_pages()'s second-grab/subpage loop are now
dead; remove them. The old silent page_increm clamp becomes a
VM_WARN_ON_ONCE, since @end already bounds the count and refs/pages[]
are already committed by the time the caller sees it -- truncating here
would leak references, not just waste a comparison.
follow_page_pte() is unaffected, still returning at most 1 page.
The -EEXIST path needs an explicit nr = 1 when pages == NULL: it used to
get that from page_mask staying 0 for a PFN-special PTE, but nr holds
-EEXIST there, which would grow nr_pages instead of shrinking it.
mm/gup_test.c (PIN_LONGTERM_BENCHMARK) shows no measurable change; the
lock-hold-time improvement is by inspection, not measurement.
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 150 +++++++++++++++++++++----------------------------------
1 file changed, 58 insertions(+), 92 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 2edffc424f272..9e7eb8a323bbf 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -683,14 +683,14 @@ static inline bool can_follow_write_pud(pud_t pud, struct page *page,
}
static long follow_huge_pud(struct vm_area_struct *vma,
- unsigned long addr, pud_t *pudp,
- unsigned int flags, unsigned long *page_mask,
- struct page **pages)
+ unsigned long addr, unsigned long end, pud_t *pudp,
+ unsigned int flags, struct page **pages)
{
struct mm_struct *mm = vma->vm_mm;
struct page *page;
pud_t pud = *pudp;
unsigned long pfn = pud_pfn(pud);
+ unsigned long off, nr;
int ret;
assert_spin_locked(pud_lockptr(mm, pudp));
@@ -702,22 +702,23 @@ static long follow_huge_pud(struct vm_area_struct *vma,
!can_follow_write_pud(pud, pfn_to_page(pfn), vma, flags))
return 0;
- pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT;
+ off = PFN_DOWN(addr & ~PUD_MASK);
+ pfn += off;
page = pfn_to_page(pfn);
if (!pud_write(pud) && gup_must_unshare(vma, flags, page))
return -EMLINK;
- ret = try_grab_folio(page_folio(page), 1, flags);
+ nr = min(HPAGE_PUD_NR - off, PFN_DOWN(end - addr));
+
+ ret = try_grab_folio(page_folio(page), nr, flags);
if (ret)
return ret;
- *page_mask = HPAGE_PUD_NR - 1;
-
if (pages)
pages[0] = page;
- return 1;
+ return nr;
}
/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */
@@ -739,13 +740,13 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
}
static long follow_huge_pmd(struct vm_area_struct *vma,
- unsigned long addr, pmd_t *pmd,
- unsigned int flags, unsigned long *page_mask,
- struct page **pages)
+ unsigned long addr, unsigned long end, pmd_t *pmd,
+ unsigned int flags, struct page **pages)
{
struct mm_struct *mm = vma->vm_mm;
pmd_t pmdval = *pmd;
struct page *page;
+ unsigned long off, nr;
int ret;
assert_spin_locked(pmd_lockptr(mm, pmd));
@@ -768,7 +769,10 @@ static long follow_huge_pmd(struct vm_area_struct *vma,
VM_WARN_ON_ONCE_PAGE((flags & FOLL_PIN) && PageAnon(page) &&
!PageAnonExclusive(page), page);
- ret = try_grab_folio(page_folio(page), 1, flags);
+ off = PFN_DOWN(addr & ~HPAGE_PMD_MASK);
+ nr = min(HPAGE_PMD_NR - off, PFN_DOWN(end - addr));
+
+ ret = try_grab_folio(page_folio(page), nr, flags);
if (ret)
return ret;
@@ -777,28 +781,25 @@ static long follow_huge_pmd(struct vm_area_struct *vma,
touch_pmd(vma, addr, pmd, flags & FOLL_WRITE);
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
- page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
- *page_mask = HPAGE_PMD_NR - 1;
+ page += off;
if (pages)
pages[0] = page;
- return 1;
+ return nr;
}
#else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */
static long follow_huge_pud(struct vm_area_struct *vma,
- unsigned long addr, pud_t *pudp,
- unsigned int flags, unsigned long *page_mask,
- struct page **pages)
+ unsigned long addr, unsigned long end, pud_t *pudp,
+ unsigned int flags, struct page **pages)
{
return 0;
}
static long follow_huge_pmd(struct vm_area_struct *vma,
- unsigned long addr, pmd_t *pmd,
- unsigned int flags, unsigned long *page_mask,
- struct page **pages)
+ unsigned long addr, unsigned long end, pmd_t *pmd,
+ unsigned int flags, struct page **pages)
{
return 0;
}
@@ -978,9 +979,8 @@ static long follow_page_pte(struct vm_area_struct *vma,
}
static long follow_pmd_mask(struct vm_area_struct *vma,
- unsigned long address, pud_t *pudp,
- unsigned int flags, unsigned long *page_mask,
- struct page **pages)
+ unsigned long address, unsigned long end, pud_t *pudp,
+ unsigned int flags, struct page **pages)
{
pmd_t *pmd, pmdval;
spinlock_t *ptl;
@@ -1016,7 +1016,7 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
return pte_alloc(mm, pmd) ? -ENOMEM :
follow_page_pte(vma, address, pmd, flags, pages);
}
- ret = follow_huge_pmd(vma, address, pmd, flags, page_mask, pages);
+ ret = follow_huge_pmd(vma, address, end, pmd, flags, pages);
spin_unlock(ptl);
/*
@@ -1030,9 +1030,8 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
}
static long follow_pud_mask(struct vm_area_struct *vma,
- unsigned long address, p4d_t *p4dp,
- unsigned int flags, unsigned long *page_mask,
- struct page **pages)
+ unsigned long address, unsigned long end, p4d_t *p4dp,
+ unsigned int flags, struct page **pages)
{
pud_t *pudp, pud;
spinlock_t *ptl;
@@ -1045,11 +1044,13 @@ static long follow_pud_mask(struct vm_area_struct *vma,
return no_page_table(vma, flags, address);
if (pud_leaf(pud)) {
ptl = pud_lock(mm, pudp);
- ret = follow_huge_pud(vma, address, pudp, flags, page_mask, pages);
+ ret = follow_huge_pud(vma, address, end, pudp, flags, pages);
spin_unlock(ptl);
/*
* The ref is already held, so the page cannot go away: fill
- * the array and flush caches without the lock.
+ * the array and flush caches without the lock. A 1 GB folio
+ * can be up to HPAGE_PUD_NR pages, too long to flush under a
+ * spinlock.
*/
if (ret > 0 && pages)
gup_fill_pages(vma, address, pages[0], ret, pages);
@@ -1060,13 +1061,12 @@ static long follow_pud_mask(struct vm_area_struct *vma,
if (unlikely(pud_bad(pud)))
return no_page_table(vma, flags, address);
- return follow_pmd_mask(vma, address, pudp, flags, page_mask, pages);
+ return follow_pmd_mask(vma, address, end, pudp, flags, pages);
}
static long follow_p4d_mask(struct vm_area_struct *vma,
- unsigned long address, pgd_t *pgdp,
- unsigned int flags, unsigned long *page_mask,
- struct page **pages)
+ unsigned long address, unsigned long end, pgd_t *pgdp,
+ unsigned int flags, struct page **pages)
{
p4d_t *p4dp, p4d;
@@ -1077,18 +1077,18 @@ static long follow_p4d_mask(struct vm_area_struct *vma,
if (!p4d_present(p4d) || p4d_bad(p4d))
return no_page_table(vma, flags, address);
- return follow_pud_mask(vma, address, p4dp, flags, page_mask, pages);
+ return follow_pud_mask(vma, address, end, p4dp, flags, pages);
}
/**
- * follow_page_mask - look up a page descriptor from a user-virtual address
+ * follow_page_mask - look up pages at a user-virtual address
* @vma: vm_area_struct mapping @address
* @address: virtual address to look up
+ * @end: virtual address at which to stop batching contiguous pages
* @flags: flags modifying lookup behaviour
- * @page_mask: a pointer to output page_mask
- * @pages: array to receive the page found, refcounted per @flags, or NULL
- * to walk the page tables (e.g. to fault pages in) without
- * collecting or refcounting them
+ * @pages: array to receive the pages, refcounted per @flags, or NULL to
+ * walk the page tables (e.g. to fault pages in) without collecting
+ * or refcounting them
*
* @flags can have FOLL_ flags set, defined in <linux/mm.h>
*
@@ -1097,15 +1097,15 @@ static long follow_p4d_mask(struct vm_area_struct *vma,
* trigger a fault with FAULT_FLAG_UNSHARE set. Note that unsharing is only
* relevant with FOLL_PIN and !FOLL_WRITE.
*
- * On output, @page_mask is set according to the size of the page.
- *
- * Return: 1 with @pages[0] filled in if a page was found, 0 if no mapping
- * exists at @address, or a negative errno for a mapping to something not
- * represented by a page descriptor (see also vm_normal_page()).
+ * Return: the number of contiguous pages starting at @address that were
+ * placed into @pages (if non-NULL), which may be fewer than the pages
+ * requested via @end; 0 if no mapping exists at @address; or a negative
+ * errno for a mapping to something not represented by a page descriptor
+ * (see also vm_normal_page()).
*/
static long follow_page_mask(struct vm_area_struct *vma,
- unsigned long address, unsigned int flags,
- unsigned long *page_mask, struct page **pages)
+ unsigned long address, unsigned long end,
+ unsigned int flags, struct page **pages)
{
pgd_t *pgd;
struct mm_struct *mm = vma->vm_mm;
@@ -1113,13 +1113,12 @@ static long follow_page_mask(struct vm_area_struct *vma,
vma_pgtable_walk_begin(vma);
- *page_mask = 0;
pgd = pgd_offset(mm, address);
if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
ret = no_page_table(vma, flags, address);
else
- ret = follow_p4d_mask(vma, address, pgd, flags, page_mask, pages);
+ ret = follow_p4d_mask(vma, address, end, pgd, flags, pages);
vma_pgtable_walk_end(vma);
@@ -1457,7 +1456,6 @@ static long __get_user_pages(struct mm_struct *mm,
{
long ret = 0, i = 0;
struct vm_area_struct *vma = NULL;
- unsigned long page_mask = 0;
if (!nr_pages)
return 0;
@@ -1472,7 +1470,6 @@ static long __get_user_pages(struct mm_struct *mm,
do {
struct page *page;
- unsigned int page_increm;
long nr;
/* first iteration or cross vma bound */
@@ -1527,8 +1524,8 @@ static long __get_user_pages(struct mm_struct *mm,
}
cond_resched();
- nr = follow_page_mask(vma, start, gup_flags, &page_mask,
- pages ? &pages[i] : NULL);
+ nr = follow_page_mask(vma, start, start + nr_pages * PAGE_SIZE,
+ gup_flags, pages ? &pages[i] : NULL);
if (!nr || nr == -EMLINK) {
ret = faultin_page(vma, start, gup_flags,
nr == -EMLINK, locked);
@@ -1550,56 +1547,25 @@ static long __get_user_pages(struct mm_struct *mm,
* Proper page table entry exists, but no corresponding
* struct page. If the caller expects **pages to be
* filled in, bail out now, because that can't be done
- * for this page.
+ * for this page. Otherwise advance by the one page
+ * follow_page_mask() looked at.
*/
if (pages) {
ret = nr;
goto out;
}
+ nr = 1;
} else if (nr < 0) {
ret = nr;
goto out;
}
- page_increm = 1 + (~(start >> PAGE_SHIFT) & page_mask);
- if (page_increm > nr_pages)
- page_increm = nr_pages;
-
- /*
- * This must be a large folio (and doesn't need to
- * be the whole folio; it can be part of it), do
- * the refcount work for all the subpages too.
- *
- * NOTE: here the page may not be the head page
- * e.g. when start addr is not thp-size aligned.
- * try_grab_folio() should have taken care of tail
- * pages.
- */
- if (pages && page_increm > 1) {
- struct folio *folio = page_folio(pages[i]);
-
- /*
- * Since we already hold refcount on the
- * large folio, this should never fail.
- */
- if (try_grab_folio(folio, page_increm - 1,
- gup_flags)) {
- /*
- * Release the 1st page ref if the
- * folio is problematic, fail hard.
- */
- gup_put_folio(folio, 1, gup_flags);
- ret = -EFAULT;
- goto out;
- }
-
- gup_fill_pages(vma, start + PAGE_SIZE, pages[i] + 1,
- page_increm - 1, pages + i + 1);
- }
+ /* Check that we didn't pin more pages than the caller will free. */
+ VM_WARN_ON_ONCE(nr > nr_pages);
- i += page_increm;
- start += page_increm * PAGE_SIZE;
- nr_pages -= page_increm;
+ i += nr;
+ start += nr * PAGE_SIZE;
+ nr_pages -= nr;
} while (nr_pages);
out:
return i ? i : ret;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH 7/8] mm/gup: walk multiple PTEs per follow_page_pte() call
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
` (5 preceding siblings ...)
2026-11-02 19:30 ` [PATCH 6/8] mm/gup: return a huge page's full count from follow_page_mask() Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-02 19:30 ` [PATCH 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab Rik van Riel
7 siblings, 0 replies; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
follow_page_pte() looks at one PTE per call, so __get_user_pages() calls
it once per page, restarting the pgd/p4d/pud/pmd descent and retaking the
PTE lock each time.
Walk every PTE from @address to the page-table/VMA/@end boundary in one
call instead. Adjacent pages from different folios, or plain base pages
with no folio relationship at all, are covered by the same call under one
lock.
A failure on the first PTE is returned as before. A failure after that
ends the walk, and __get_user_pages() retrying the read gets the error.
The pages[] slots are stored under the lock, but the cache flushes run
after it is dropped, over the pages the walk stored: holding one page
table can cover up to PTRS_PER_PTE pages, and the flush helpers need no
lock once the folio references are held.
Measured with mm/gup_test.c, median get time over 16 iterations on a
256 MB MADV_HUGEPAGE region in a 4 CPU VM. Folio size was confirmed
through the per-size anon_fault_alloc counters, 4100 folios for 64 kB
and 128 for 2 MB:
gup_test -L -m 256 -n 65536 -r 16 -t
before after
4 kB base pages 2753 us 1178 us (2.3x)
64 kB mTHP 2946 us 1361 us (2.2x)
2 MB THP (control) 72 us 70 us
Base pages and mTHP gain about the same amount, because what is saved is
the page-table descent that no longer restarts per page, not anything
folio-specific. 2 MB THP does not reach follow_page_pte(), so it stays
flat.
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 77 ++++++++++++++++++++++++++++++++++++--------------------
1 file changed, 50 insertions(+), 27 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 9e7eb8a323bbf..67e20b326be6e 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -937,44 +937,67 @@ static long gup_lookup_pte(struct vm_area_struct *vma, unsigned long address,
return 0;
}
+/*
+ * Walk the PTEs from the start address to the end of this page table or VMA,
+ * whichever comes first, and commit every page found.
+ *
+ * A failure on the first PTE is returned to the caller. A failure after that
+ * is a short read; __get_user_pages() retrying the read will get the error.
+ *
+ * Return: the number of pages stored in @pages, 0 when no mapping exists at
+ * the start address, or a negative errno for a failure on the first PTE.
+ */
static long follow_page_pte(struct vm_area_struct *vma,
- unsigned long address, pmd_t *pmd, unsigned int flags,
- struct page **pages)
+ unsigned long address, unsigned long end, pmd_t *pmd,
+ unsigned int flags, struct page **pages)
{
struct mm_struct *mm = vma->vm_mm;
bool need_no_page_table = false;
- struct page *page;
+ pte_t *ptep, *orig_ptep;
+ unsigned long walk_end, base;
+ unsigned long nr = 0;
spinlock_t *ptl;
- pte_t *ptep, pte;
- long ret;
+ long ret = 0;
- ptep = pte_offset_map_lock(mm, pmd, address, &ptl);
+ orig_ptep = ptep = pte_offset_map_lock(mm, pmd, address, &ptl);
if (!ptep)
return no_page_table(vma, flags, address);
- pte = ptep_get(ptep);
- ret = gup_lookup_pte(vma, address, ptep, pte, flags, &page);
- if (!ret && page) {
+ base = address;
+ walk_end = min(pmd_addr_end(address, end), vma->vm_end);
+
+ for (; address < walk_end; address += PAGE_SIZE, ptep++) {
+ pte_t pte = ptep_get(ptep);
+ struct page *page;
+
+ ret = gup_lookup_pte(vma, address, ptep, pte, flags, &page);
+ if (ret || !page) {
+ /*
+ * no_page_table() may look up the page cache, so it
+ * cannot run under the PTE lock. A none PTE returns
+ * from gup_lookup_pte() before any write check runs,
+ * so this is the same call the old code made -- not a
+ * new failure mode for FOLL_WRITE.
+ */
+ if (!ret && pte_none(pte))
+ need_no_page_table = true;
+ break;
+ }
ret = gup_grab_folio_pages(page_folio(page), page,
- 1, pte_dirty(pte), flags, pages);
- if (!ret)
- ret = 1;
- } else if (!ret && pte_none(pte)) {
- /*
- * no_page_table() may look up the page cache, so it cannot run
- * under the PTE lock. A none PTE returns from gup_lookup_pte()
- * before any write check runs, so this is the same call the old
- * code made -- not a new failure mode for FOLL_WRITE.
- */
- need_no_page_table = true;
+ 1, pte_dirty(pte), flags,
+ pages ? pages + nr : NULL);
+ if (ret)
+ break;
+ nr++;
}
+ pte_unmap_unlock(orig_ptep, ptl);
- pte_unmap_unlock(ptep, ptl);
-
+ if (nr) {
+ gup_flush_pages(vma, base, pages, nr);
+ return nr;
+ }
if (need_no_page_table)
return no_page_table(vma, flags, address);
- if (ret > 0)
- gup_flush_pages(vma, address, pages, ret);
return ret;
}
@@ -994,7 +1017,7 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
if (!pmd_present(pmdval))
return no_page_table(vma, flags, address);
if (likely(!pmd_leaf(pmdval)))
- return follow_page_pte(vma, address, pmd, flags, pages);
+ return follow_page_pte(vma, address, end, pmd, flags, pages);
if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags))
return no_page_table(vma, flags, address);
@@ -1007,14 +1030,14 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
}
if (unlikely(!pmd_leaf(pmdval))) {
spin_unlock(ptl);
- return follow_page_pte(vma, address, pmd, flags, pages);
+ return follow_page_pte(vma, address, end, pmd, flags, pages);
}
if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) {
spin_unlock(ptl);
split_huge_pmd(vma, pmd, address);
/* If pmd was left empty, stuff a page table in there quickly */
return pte_alloc(mm, pmd) ? -ENOMEM :
- follow_page_pte(vma, address, pmd, flags, pages);
+ follow_page_pte(vma, address, end, pmd, flags, pages);
}
ret = follow_huge_pmd(vma, address, end, pmd, flags, pages);
spin_unlock(ptl);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* [PATCH 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab
2026-11-02 19:30 [RFC PATCH v5 0/8] batch lookups in follow_page_mask() Rik van Riel
` (6 preceding siblings ...)
2026-11-02 19:30 ` [PATCH 7/8] mm/gup: walk multiple PTEs per follow_page_pte() call Rik van Riel
@ 2026-11-02 19:30 ` Rik van Riel
2026-11-03 8:35 ` David Hildenbrand (Arm)
7 siblings, 1 reply; 18+ messages in thread
From: Rik van Riel @ 2026-11-02 19:30 UTC (permalink / raw)
To: linux-kernel
Cc: Andrew Morton, David Hildenbrand, Jason Gunthorpe, John Hubbard,
Peter Xu, Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig,
linux-mm, Rik van Riel
follow_page_pte() now walks a whole page table in one call, but still
resolves and commits each PTE on its own, so a PTE-mapped large folio
(mTHP) pays one try_grab_folio() per subpage.
Add gup_pte_batch_len() and give gup_grab_folio_pages() a run
length. Each run starts with a full per-PTE resolve, then
gup_pte_batch_len() finds how many more PTEs extend it with a pte_same()
scan rather than a re-derivation of each page, and the run is committed
with one refcount grab. Runs stop at folio boundaries, so plain base
pages are unaffected.
Gather the dirty bits from all the PTEs in a batch, in order to mark
the folio dirty for FOLL_WRITE.
Same benchmark as the previous change, with before again taken at the
base of the series:
gup_test -L -m 256 -n 65536 -r 16 -t
before after
64 kB mTHP 2946 us 231 us (12.8x)
4 kB base (control) 2753 us 1198 us (2.3x)
2 MB THP (control) 72 us 71 us
The 4 kB column is the previous change's 2.3x, unchanged: base pages
share no folio, so there is nothing to batch. 64 kB mTHP goes from that
change's 2.2x to 12.8x, a further 5.9x from committing a run with one
refcount grab instead of one per subpage.
With FOLL_TOUCH, folio_mark_accessed() runs once per run instead of
once per page: 512 calls before and 32 after over 2 MB of 64 kB
folios, leaving them referenced but inactive after the first call.
That matches the huge-page and fast paths, which mark once
per folio. (Measured by Yuan-Hao Hsu.)
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 46 insertions(+), 3 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 67e20b326be6e..47f204e21a3a7 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -937,6 +937,35 @@ static long gup_lookup_pte(struct vm_area_struct *vma, unsigned long address,
return 0;
}
+/*
+ * Return how many PTEs map consecutive pages of the same folio and can be
+ * committed as one run. Always at least 1.
+ *
+ * The write-fault and unshare checks in gup_lookup_pte() are per PTE, but a
+ * writable run needs no repeat: a writable anon page is exclusive. A read-only
+ * run under FOLL_WRITE or FOLL_PIN does need the per-page check, so it stays
+ * one page at a time.
+ */
+static unsigned long gup_pte_batch_len(struct vm_area_struct *vma,
+ unsigned long address, unsigned long walk_end,
+ struct folio *folio, pte_t *ptep, pte_t *batch_pte, unsigned int flags)
+{
+ unsigned long max;
+
+ if (!folio_test_large(folio))
+ return 1;
+ if (!pte_write(*batch_pte) && (flags & (FOLL_WRITE | FOLL_PIN)))
+ return 1;
+
+ max = (walk_end - address) >> PAGE_SHIFT;
+ if (max <= 1)
+ return 1;
+
+ /* Merge young/dirty across batch so folio_mark_dirty sees any dirty. */
+ return folio_pte_batch_flags(folio, vma, ptep, batch_pte, max,
+ FPB_RESPECT_WRITE | FPB_MERGE_YOUNG_DIRTY);
+}
+
/*
* Walk the PTEs from the start address to the end of this page table or VMA,
* whichever comes first, and commit every page found.
@@ -968,7 +997,10 @@ static long follow_page_pte(struct vm_area_struct *vma,
for (; address < walk_end; address += PAGE_SIZE, ptep++) {
pte_t pte = ptep_get(ptep);
+ struct folio *folio;
struct page *page;
+ unsigned long batch;
+ pte_t batch_pte;
ret = gup_lookup_pte(vma, address, ptep, pte, flags, &page);
if (ret || !page) {
@@ -983,12 +1015,23 @@ static long follow_page_pte(struct vm_area_struct *vma,
need_no_page_table = true;
break;
}
- ret = gup_grab_folio_pages(page_folio(page), page,
- 1, pte_dirty(pte), flags,
+ folio = page_folio(page);
+ batch_pte = pte;
+
+ batch = gup_pte_batch_len(vma, address, walk_end, folio,
+ ptep, &batch_pte, flags);
+ ret = gup_grab_folio_pages(folio, page, batch,
+ pte_dirty(batch_pte), flags,
pages ? pages + nr : NULL);
if (ret)
break;
- nr++;
+ nr += batch;
+ /*
+ * The loop's own increment covers one PTE; skip
+ * the rest of the batch.
+ */
+ ptep += batch - 1;
+ address += (batch - 1) * PAGE_SIZE;
}
pte_unmap_unlock(orig_ptep, ptl);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [PATCH 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab
2026-11-02 19:30 ` [PATCH 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab Rik van Riel
@ 2026-11-03 8:35 ` David Hildenbrand (Arm)
2026-11-03 16:42 ` Rik van Riel
0 siblings, 1 reply; 18+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-03 8:35 UTC (permalink / raw)
To: Rik van Riel, linux-kernel
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On 11/2/26 20:30, Rik van Riel wrote:
> follow_page_pte() now walks a whole page table in one call, but still
> resolves and commits each PTE on its own, so a PTE-mapped large folio
> (mTHP) pays one try_grab_folio() per subpage.
>
> Add gup_pte_batch_len() and give gup_grab_folio_pages() a run
> length. Each run starts with a full per-PTE resolve, then
> gup_pte_batch_len() finds how many more PTEs extend it with a pte_same()
> scan rather than a re-derivation of each page, and the run is committed
> with one refcount grab. Runs stop at folio boundaries, so plain base
> pages are unaffected.
>
> Gather the dirty bits from all the PTEs in a batch, in order to mark
> the folio dirty for FOLL_WRITE.
>
> Same benchmark as the previous change, with before again taken at the
> base of the series:
>
> gup_test -L -m 256 -n 65536 -r 16 -t
> before after
> 64 kB mTHP 2946 us 231 us (12.8x)
> 4 kB base (control) 2753 us 1198 us (2.3x)
> 2 MB THP (control) 72 us 71 us
>
> The 4 kB column is the previous change's 2.3x, unchanged: base pages
> share no folio, so there is nothing to batch. 64 kB mTHP goes from that
> change's 2.2x to 12.8x, a further 5.9x from committing a run with one
> refcount grab instead of one per subpage.
>
> With FOLL_TOUCH, folio_mark_accessed() runs once per run instead of
> once per page: 512 calls before and 32 after over 2 MB of 64 kB
> folios, leaving them referenced but inactive after the first call.
> That matches the huge-page and fast paths, which mark once
> per folio. (Measured by Yuan-Hao Hsu.)
>
> Suggested-by: David Hildenbrand <david@kernel.org>
> Signed-off-by: Rik van Riel <riel@surriel.com>
> ---
> mm/gup.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 46 insertions(+), 3 deletions(-)
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 67e20b326be6e..47f204e21a3a7 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -937,6 +937,35 @@ static long gup_lookup_pte(struct vm_area_struct *vma, unsigned long address,
> return 0;
> }
>
> +/*
> + * Return how many PTEs map consecutive pages of the same folio and can be
> + * committed as one run. Always at least 1.
> + *
> + * The write-fault and unshare checks in gup_lookup_pte() are per PTE, but a
> + * writable run needs no repeat: a writable anon page is exclusive. A read-only
> + * run under FOLL_WRITE or FOLL_PIN does need the per-page check, so it stays
> + * one page at a time.
That's a clear indication that this needs more work.
I'll try to get to it soon and see how it could be done in a cleaner way, likely
it should be based on top of the gup-fast series where we will have helpers to
check for required unsharing of multiple PTEs.
--
Cheers,
David
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab
2026-11-03 8:35 ` David Hildenbrand (Arm)
@ 2026-11-03 16:42 ` Rik van Riel
2026-11-03 16:55 ` David Hildenbrand (Arm)
0 siblings, 1 reply; 18+ messages in thread
From: Rik van Riel @ 2026-11-03 16:42 UTC (permalink / raw)
To: David Hildenbrand (Arm), linux-kernel, Yeoreum Yun
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On Tue, 2026-11-03 at 09:35 +0100, David Hildenbrand (Arm) wrote:
> On 11/2/26 20:30, Rik van Riel wrote:
> >
> > +/*
> > + * Return how many PTEs map consecutive pages of the same folio
> > and can be
> > + * committed as one run. Always at least 1.
> > + *
> > + * The write-fault and unshare checks in gup_lookup_pte() are per
> > PTE, but a
> > + * writable run needs no repeat: a writable anon page is
> > exclusive. A read-only
> > + * run under FOLL_WRITE or FOLL_PIN does need the per-page check,
> > so it stays
> > + * one page at a time.
>
> That's a clear indication that this needs more work.
>
> I'll try to get to it soon and see how it could be done in a cleaner
> way, likely
> it should be based on top of the gup-fast series where we will have
> helpers to
> check for required unsharing of multiple PTEs.
I'm all for sharing the same helpers.
How would you like to sequence things?
Do you prefer to merge both series separately,
and then get a follow-up series that merges
the helper functions?
Would you like Yeoreum Yun and me to combine
the series, merge the helper functions beforehand,
and then send you one larger, combined series?
I have spent enough time in gup.c that I would
be happy to get rid of some more of that code.
--
All Rights Reversed.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab
2026-11-03 16:42 ` Rik van Riel
@ 2026-11-03 16:55 ` David Hildenbrand (Arm)
0 siblings, 0 replies; 18+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-03 16:55 UTC (permalink / raw)
To: Rik van Riel, linux-kernel, Yeoreum Yun
Cc: Andrew Morton, Jason Gunthorpe, John Hubbard, Peter Xu,
Suren Baghdasaryan, Yuan-Hao Hsu, Christoph Hellwig, linux-mm
On 11/3/26 17:42, Rik van Riel wrote:
> On Tue, 2026-11-03 at 09:35 +0100, David Hildenbrand (Arm) wrote:
>> On 11/2/26 20:30, Rik van Riel wrote:
>>>
>>> +/*
>>> + * Return how many PTEs map consecutive pages of the same folio
>>> and can be
>>> + * committed as one run. Always at least 1.
>>> + *
>>> + * The write-fault and unshare checks in gup_lookup_pte() are per
>>> PTE, but a
>>> + * writable run needs no repeat: a writable anon page is
>>> exclusive. A read-only
>>> + * run under FOLL_WRITE or FOLL_PIN does need the per-page check,
>>> so it stays
>>> + * one page at a time.
>>
>> That's a clear indication that this needs more work.
>>
>> I'll try to get to it soon and see how it could be done in a cleaner
>> way, likely
>> it should be based on top of the gup-fast series where we will have
>> helpers to
>> check for required unsharing of multiple PTEs.
>
> I'm all for sharing the same helpers.
>
> How would you like to sequence things?
>
> Do you prefer to merge both series separately,
> and then get a follow-up series that merges
> the helper functions?
>
> Would you like Yeoreum Yun and me to combine
> the series, merge the helper functions beforehand,
> and then send you one larger, combined series?
>
> I have spent enough time in gup.c that I would
> be happy to get rid of some more of that code.
My plan is to get the small series for gup-fast ready, stage that, and then
build on top of it.
By then I should have been able to dig through most of your series. The cleanups
can stay part of this series.
--
Cheers,
David
^ permalink raw reply [flat|nested] 18+ messages in thread