[PATCH 5/8] mm/gup: fill the pages array outside the pud/pmd lock

[Rik van Riel <riel@surriel.com>]

follow_huge_pud() and follow_huge_pmd() fill pages[] and flush the page's
caches while still holding the pud or pmd lock. Neither flush_anon_page()
nor flush_dcache_page() needs that lock.

Have the huge paths store the page and let follow_pud_mask() and
follow_pmd_mask() do the fill after they unlock, so the flushes happen
outside the critical section.

This should be safe because try_grab_folio() has already taken a folio
reference before the unlock, so nothing can free the page while the fill
runs, and the fill itself touches neither the page tables nor the pud or
pmd entry it was reached through.

No functional changes intended.

Suggested-by: David Hildenbrand <david@kernel.org>
Reviewed-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/gup.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/mm/gup.c b/mm/gup.c
index c2a4f5b987ced..2edffc424f272 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -714,7 +714,8 @@ static long follow_huge_pud(struct vm_area_struct *vma,
 
 	*page_mask = HPAGE_PUD_NR - 1;
 
-	gup_fill_pages(vma, addr, page, 1, pages);
+	if (pages)
+		pages[0] = page;
 
 	return 1;
 }
@@ -779,7 +780,8 @@ static long follow_huge_pmd(struct vm_area_struct *vma,
 	page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
 	*page_mask = HPAGE_PMD_NR - 1;
 
-	gup_fill_pages(vma, addr, page, 1, pages);
+	if (pages)
+		pages[0] = page;
 
 	return 1;
 }
@@ -1016,6 +1018,14 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
 	}
 	ret = follow_huge_pmd(vma, address, pmd, flags, page_mask, pages);
 	spin_unlock(ptl);
+
+	/*
+	 * The ref is already held, so the page cannot go away: fill the
+	 * array and flush caches without the pmd lock.
+	 */
+	if (ret > 0 && pages)
+		gup_fill_pages(vma, address, pages[0], ret, pages);
+
 	return ret;
 }
 
@@ -1037,6 +1047,12 @@ static long follow_pud_mask(struct vm_area_struct *vma,
 		ptl = pud_lock(mm, pudp);
 		ret = follow_huge_pud(vma, address, pudp, flags, page_mask, pages);
 		spin_unlock(ptl);
+		/*
+		 * The ref is already held, so the page cannot go away: fill
+		 * the array and flush caches without the lock.
+		 */
+		if (ret > 0 && pages)
+			gup_fill_pages(vma, address, pages[0], ret, pages);
 		if (ret)
 			return ret;
 		return no_page_table(vma, flags, address);
-- 
2.53.0-Meta