From: Rik van Riel <riel@surriel.com>
To: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org, David Hildenbrand <david@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Suren Baghdasaryan <surenb@google.com>,
Lorenzo Stoakes <ljs@kernel.org>,
Christoph Hellwig <hch@infradead.org>,
Yuan-Hao Hsu <aa9736195201@gmail.com>,
John Hubbard <jhubbard@nvidia.com>,
Jason Gunthorpe <jgg@ziepe.ca>, Peter Xu <peterx@redhat.com>,
Rik van Riel <riel@surriel.com>
Subject: [PATCH v5 1/8] mm/gup: break out gup_fill_pages() helper
Date: Mon, 16 Nov 2026 16:01:20 -0500 [thread overview]
Message-ID: <20261116210127.3837898-2-riel@surriel.com> (raw)
In-Reply-To: <20261116210127.3837898-1-riel@surriel.com>
__get_user_pages() fills pages[] in an open-coded loop, and the
GUP-fast PTE, PMD and PUD leaf paths each have their own copy of
the same loop.
Move the store into a gup_fill_pages() helper, shared by the slow
and the fast paths. The helper keeps a pages==NULL check, so the
fault-in users that pass no pages array (faultin_page_range(),
populate_vma_page_range()) need no conditionals at their call
sites. Cache flushing stays open-coded for now; it moves into its
own helper in the next patch.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 35 +++++++++++++++++++++++++----------
1 file changed, 25 insertions(+), 10 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index b0a0019de80a..e0698d2075b4 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -710,6 +710,25 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
return gup_pages_not_requiring_unshare(vma, flags, folio, page, 1) != 1;
}
+/*
+ * Store @nr_pages consecutive pages starting at @page into @pages.
+ * The caller must hold at least one reference on the folio, so it
+ * cannot go away, or get split while the array is filled.
+ *
+ * Some GUP users only fault pages in and pass no pages array.
+ */
+static void gup_fill_pages(struct page **pages, struct page *page,
+ unsigned long nr_pages)
+{
+ unsigned long i;
+
+ if (!pages)
+ return;
+
+ for (i = 0; i < nr_pages; i++)
+ pages[i] = page + i;
+}
+
#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES
/* FOLL_FORCE can write to even unwritable PUDs in COW mappings. */
static inline bool can_follow_write_pud(pud_t pud, struct page *page,
@@ -1579,9 +1598,9 @@ static long __get_user_pages(struct mm_struct *mm,
}
}
+ gup_fill_pages(pages + i, page, page_increm);
for (j = 0; j < page_increm; j++) {
subpage = page + j;
- pages[i + j] = subpage;
flush_anon_page(vma, subpage, start + j * PAGE_SIZE);
flush_dcache_page(subpage);
}
@@ -2946,7 +2965,6 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
do {
pte_t pte = ptep_get_lockless(ptep);
struct page *page;
- unsigned long i;
nr_batch = 1;
@@ -3017,8 +3035,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
nr_batch = 1;
folio_set_referenced(folio);
- for (i = 0; i < nr_batch; i++)
- pages[nr_pages + i] = page + i;
+ gup_fill_pages(pages + nr_pages, page, nr_batch);
nr_pages += nr_batch;
folio = NULL;
} while (ptep += nr_batch, addr += nr_batch * PAGE_SIZE, addr != end);
@@ -3054,7 +3071,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
{
struct page *mapped_page, *page;
struct folio *folio;
- unsigned long nr_pages, i;
+ unsigned long nr_pages;
/* See gup_fast_pte_range() */
if (pmd_protnone(orig))
@@ -3089,8 +3106,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
return 0;
}
- for (i = 0; i < nr_pages; i++)
- *(pages++) = page++;
+ gup_fill_pages(pages, page, nr_pages);
folio_set_referenced(folio);
return nr_pages;
}
@@ -3101,7 +3117,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
{
struct page *mapped_page, *page;
struct folio *folio;
- unsigned long nr_pages, i;
+ unsigned long nr_pages;
if (!pud_access_permitted(orig, flags & FOLL_WRITE))
return 0;
@@ -3133,8 +3149,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
return 0;
}
- for (i = 0; i < nr_pages; i++)
- *(pages++) = page++;
+ gup_fill_pages(pages, page, nr_pages);
folio_set_referenced(folio);
return nr_pages;
}
--
2.56.0
next prev parent reply other threads:[~2026-11-16 21:01 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 21:01 [PATCH v5 0/8] mm/gup: walk and batch whole PTE tables in slow GUP Rik van Riel
2026-11-16 21:01 ` Rik van Riel [this message]
2026-11-16 21:01 ` [PATCH v5 2/8] mm/gup: break out gup_flush_pages() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 3/8] mm/gup: convert follow_page_mask() to return a long Rik van Riel
2026-11-16 21:01 ` [PATCH v5 4/8] mm/gup: split out folio grab-and-store helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 5/8] mm/gup: break out gup_lookup_pte() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 6/8] mm/gup: return a huge page's full count from follow_page_mask() Rik van Riel
2026-11-16 21:01 ` [PATCH v5 7/8] mm/gup: walk multiple PTEs per follow_page_pte() call Rik van Riel
2026-11-16 21:01 ` [PATCH v5 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab Rik van Riel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116210127.3837898-2-riel@surriel.com \
--to=riel@surriel.com \
--cc=aa9736195201@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=david@kernel.org \
--cc=hch@infradead.org \
--cc=jgg@ziepe.ca \
--cc=jhubbard@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=peterx@redhat.com \
--cc=surenb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®