mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH v5 0/8] mm/gup: walk and batch whole PTE tables in slow GUP
@ 2026-11-16 21:01 Rik van Riel
  2026-11-16 21:01 ` [PATCH v5 1/8] mm/gup: break out gup_fill_pages() helper Rik van Riel
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Rik van Riel @ 2026-11-16 21:01 UTC (permalink / raw)
  To: linux-kernel
  Cc: linux-mm, David Hildenbrand, Andrew Morton, Suren Baghdasaryan,
	Lorenzo Stoakes, Christoph Hellwig, Yuan-Hao Hsu, John Hubbard,
	Jason Gunthorpe, Peter Xu

Slow GUP resolves and commits one PTE per follow_page_pte() call, so
__get_user_pages() restarts the pgd/p4d/pud/pmd descent and retakes
the PTE lock for every page, and a PTE-mapped large folio (mTHP) pays
one try_grab_folio() per subpage. The fast path already batches
PTE-mapped large folios; the slow path does not.

This series brings the slow path to the same shape in reviewable
steps: three no-functional-change refactors (store, flush, return
type), two splits of follow_page_pte(), huge-page count reporting,
the multi-PTE walk, and finally the same-folio batch commit. The
batch-length helper is shared with GUP-fast through an fpb_flags
parameter selecting the scan.

Patches:
 1. break out gup_fill_pages() helper
 2. break out gup_flush_pages() helper
 3. convert follow_page_mask() to return a long
 4. split out folio grab-and-store helper
 5. break out gup_lookup_pte() helper
 6. return a huge page's full count from follow_page_mask()
 7. walk multiple PTEs per follow_page_pte() call
 8. batch contiguous same-folio PTEs into one refcount grab

Measured with mm/gup_test.c (PIN_LONGTERM_BENCHMARK), median get
time over 16 iterations on a 256 MB MADV_HUGEPAGE region in a 4 CPU
VM, folio sizes confirmed through the anon_fault_alloc counters
(4100 folios at 64 kB, 128 at 2 MB):

  gup_test -L -m 256 -n 65536 -r 16 -t
                        before      after
  4 kB base pages      2753 us     1178 us   (2.3x, patch 7)
  64 kB mTHP           2946 us      231 us  (12.8x, patch 8)
  2 MB THP (control)     72 us       71 us

Patch 7's gain is the page-table descent that no longer restarts per
page; patch 8 adds a further 5.9x on mTHP by committing each run with
one refcount grab. Base pages share no folio and are unaffected by
patch 8; 2 MB THP never reaches follow_page_pte(). With FOLL_TOUCH,
folio_mark_accessed() drops from 512 calls to 32 over 2 MB of 64 kB
folios (measured by Yuan-Hao Hsu), matching the huge-page and fast
paths, which mark once per folio.

This is based on 102d6c3d5892 (merge of "mm: gup: batch PTE-mapped
large folios in gup_fast_pte_range()" from the gup-7.5-ptebatch branch
of mm-core.git)

mm/gup.c only: 347 insertions, 226 deletions.

Suggested-by: David Hildenbrand <david@kernel.org>

Changelog
---------
v4 -> v5:
- Rebased onto 102d6c3d5892 (landed GUP-fast batching and the
  folio-based unshare helpers); adapted to the store-only
  gup_fill_pages() and gup_must_unshare(vma, flags, folio, page).
- Split the store/flush extraction in two: new patch 2 breaks out
  gup_flush_pages()
- Dropped the "fill the pages array outside the pud/pmd lock" patch.
  Separating of the fill and flush helpers made it possible to just
  keep the (fast) fills inline, while keeping the (slow) flush
  out of the loop, at the tail end of __get_user_pages()
- Rewrote patch 8 around a shared gup_pte_batch_len() with an
  fpb_flags scan selector; GUP-fast uses the same helper instead
  of its inline scan and trim. (David Hildenbrand)
- Read-only runs are trimmed to the pages needing no unsharing
  instead of refused; the trim also covers runs of one page, which
  the fast path relies on as its per-page unshare gate.
- Renamed gup_grab_folio_pages() to gup_grab_fill_folio_pages().
- Re-measured the benchmarks.

Link: https://lore.kernel.org/r/20261102193026.3860700-1-riel@surriel.com [v4]
Link: https://lore.kernel.org/r/20260811025157.1632867-1-riel@surriel.com [v3]
Link: https://lore.kernel.org/r/20260801031540.2742891-1-riel@surriel.com [v2]
Link: https://lore.kernel.org/r/20260730035350.1fc95dd8@fangorn [v1]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-11-16 21:01 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 21:01 [PATCH v5 0/8] mm/gup: walk and batch whole PTE tables in slow GUP Rik van Riel
2026-11-16 21:01 ` [PATCH v5 1/8] mm/gup: break out gup_fill_pages() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 2/8] mm/gup: break out gup_flush_pages() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 3/8] mm/gup: convert follow_page_mask() to return a long Rik van Riel
2026-11-16 21:01 ` [PATCH v5 4/8] mm/gup: split out folio grab-and-store helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 5/8] mm/gup: break out gup_lookup_pte() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 6/8] mm/gup: return a huge page's full count from follow_page_mask() Rik van Riel
2026-11-16 21:01 ` [PATCH v5 7/8] mm/gup: walk multiple PTEs per follow_page_pte() call Rik van Riel
2026-11-16 21:01 ` [PATCH v5 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab Rik van Riel

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®