[PATCH 00/15] mm: eliminate &folio->page references in mm/

[PATCH 00/15] mm: eliminate &folio->page references in mm/

[Nilay Vaish]

As part of the memdescs effort to separate struct folio from struct page
and allocate struct folio dynamically [1], struct folio will no longer
embed a struct page member (folio->page), and the address of a struct
folio will no longer equal the address of its head struct page in
vmemmap.

This series eliminates all but one of the remaining &folio->page
references across mm/*.c (25 files) by converting them to:

- folio_page(folio, 0) (or folio_page(folio, i) where indexing a
  subpage) when passing a folio's head/subpage struct page * to
  page-based helpers (PageAnonExclusive, SetPageAnonExclusive,
  rmap/split helpers, movable_ops helpers, HugeTLB Vmemmap Optimization
  (HVO) address calculations, page_ext, alloc_tag, etc.);
- VM_BUG_ON_FOLIO(..., old_folio) in place of
  VM_BUG_ON_PAGE(..., &old_folio->page) in hugetlb_wp(), and
  folio_set_hwpoison(folio) in place of SetPageHWPoison(&folio->page) in
  mf_generic_kill_procs();
- ERR_CAST(folio) in place of returning &folio->page on IS_ERR(folio)
  paths in do_read_cache_page() and shmem_read_mapping_page_gfp();
- Operating on struct page * directly in compaction_alloc_noprof()
  before calling page_rmappable_folio(), and adding &page->lru to
  cc->freepages[order] in compaction_free().

The remaining folio->page uses in mm/debug.c (__dump_folio(), which has
three lines: page_has_type(&folio->page) and two reads of
folio->page.page_type) are intentionally left untouched because
__dump_folio() operates on a stack snapshot (&ps.folio_snapshot) rather
than a live struct folio.  Those are the only folio->page uses left in
mm/*.c after this series.

This series is based on the mm-unstable branch of Andrew Morton's mm
tree.  It has been build- and boot-tested on x86_64 under KVM with
CONFIG_TRANSPARENT_HUGEPAGE, CONFIG_HUGETLBFS,
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, CONFIG_CMA, CONFIG_KSM,
CONFIG_USERFAULTFD, CONFIG_MEMORY_FAILURE, CONFIG_PAGE_OWNER,
CONFIG_MEM_ALLOC_PROFILING, and CONFIG_DEBUG_VM enabled.

No functional change is intended while struct folio overlays struct
page.

[1] https://kernelnewbies.org/MatthewWilcox/Memdescs

Nilay Vaish (15):
  mm/khugepaged: drop redundant page parameter from set_huge_pmd()
  mm/huge_memory: use folio_page() in PMD/PUD rmap, zap, and migration
  mm/huge_memory: use folio_page() in THP split paths
  mm/hugetlb_vmemmap: use folio_page() for vmemmap address arithmetic
  mm/hugetlb_cma: use folio_page() in hugetlb_cma_free_frozen_folio()
  mm/hugetlb: replace &folio->page with folio_page() and
    VM_BUG_ON_FOLIO()
  mm/page_isolation: use folio_page() in page_is_unmovable()
  mm/compaction: stop casting free pages to struct folio
  mm/migrate, vmscan: use folio_page() for movable_ops helpers
  mm/memory: use folio_page() in fault and CoW paths
  mm/rmap, gup: use folio_page() for PageAnonExclusive and
    gup_must_unshare
  mm/ksm, userfaultfd: replace &folio->page with folio_page()
  mm/page_alloc, folio: use folio_page() when freeing folios
  mm/page_owner, alloc_tag: use folio_page() for page_ext and codetag
  mm: replace remaining &folio->page uses with folio_page() and helpers

 mm/alloc_tag.c       | 10 +++++-----
 mm/compaction.c      | 15 ++++++---------
 mm/filemap.c         |  6 +++---
 mm/folio.c           |  2 +-
 mm/gup.c             | 11 +++++++----
 mm/gup_test.c        |  6 ++++--
 mm/huge_memory.c     | 37 ++++++++++++++++++++-----------------
 mm/hugetlb.c         | 21 +++++++++++----------
 mm/hugetlb_cma.c     |  3 ++-
 mm/hugetlb_vmemmap.c | 10 +++++-----
 mm/hwpoison-inject.c |  2 +-
 mm/khugepaged.c      |  6 +++---
 mm/ksm.c             | 16 ++++++++++------
 mm/memory-failure.c  |  8 ++++----
 mm/memory.c          | 16 +++++++++-------
 mm/memory_hotplug.c  |  2 +-
 mm/memremap.c        |  2 +-
 mm/migrate.c         | 20 +++++++++++---------
 mm/page_alloc.c      | 18 ++++++++++--------
 mm/page_isolation.c  |  2 +-
 mm/page_owner.c      | 15 +++++++++------
 mm/rmap.c            | 10 +++++-----
 mm/shmem.c           |  2 +-
 mm/userfaultfd.c     | 14 ++++++++------
 mm/vmscan.c          |  2 +-
 25 files changed, 139 insertions(+), 117 deletions(-)


base-commit: 90ddfbd1963659ca4a5d3d7f20717a4222682a8e
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 01/15] mm/khugepaged: drop redundant page parameter from set_huge_pmd()

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

set_huge_pmd() is a static helper with a single caller that passes both
'folio' and '&folio->page'. Drop the redundant struct page parameter and
pass folio_page(folio, 0) to do_set_pmd() so khugepaged no longer
references folio->page directly.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/khugepaged.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 913086eaf17b..63acb13f24eb 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1793,7 +1793,7 @@ static void collect_mm_slot(struct mm_slot *slot)
 
 /* folio must be locked, and mmap_lock must be held */
 static enum scan_result set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,
-		pmd_t *pmdp, struct folio *folio, struct page *page)
+		pmd_t *pmdp, struct folio *folio)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct vm_fault vmf = {
@@ -1821,7 +1821,7 @@ static enum scan_result set_huge_pmd(struct vm_area_struct *vma, unsigned long a
 	}
 
 	vmf.pmd = pmdp;
-	if (do_set_pmd(&vmf, folio, page))
+	if (do_set_pmd(&vmf, folio, folio_page(folio, 0)))
 		return SCAN_FAIL;
 
 	folio_get(folio);
@@ -2032,7 +2032,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
 maybe_install_pmd:
 	/* step 5: install pmd entry */
 	result = install_pmd
-			? set_huge_pmd(vma, haddr, pmd, folio, &folio->page)
+			? set_huge_pmd(vma, haddr, pmd, folio)
 			: SCAN_SUCCEED;
 	goto drop_folio;
 abort:
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 02/15] mm/huge_memory: use folio_page() in PMD/PUD rmap, zap, and migration

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct &folio->page uses with folio_page(folio, 0) in
insert_pmd(), insert_pud(), copy_huge_non_present_pmd(),
zap_huge_pmd_folio(), zap_huge_pmd(), move_pages_huge_pmd(), and
remove_migration_pmd().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/huge_memory.c | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ddc631a388b9..31af817eeded 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1664,7 +1664,8 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 			entry = pmd_mkspecial(entry);
 		} else {
 			folio_get(fop.folio);
-			folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma);
+			folio_add_file_rmap_pmd(fop.folio,
+						folio_page(fop.folio, 0), vma);
 			add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
 		}
 	} else {
@@ -1786,7 +1787,8 @@ static vm_fault_t insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		entry = folio_mk_pud(fop.folio, vma->vm_page_prot);
 
 		folio_get(fop.folio);
-		folio_add_file_rmap_pud(fop.folio, &fop.folio->page, vma);
+		folio_add_file_rmap_pud(fop.folio,
+					folio_page(fop.folio, 0), vma);
 		add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PUD_NR);
 	} else {
 		entry = pud_mkhuge(pfn_pud(fop.pfn, prot));
@@ -1935,7 +1937,7 @@ static void copy_huge_non_present_pmd(
 		 * folio_try_dup_anon_rmap_pmd does not fail for
 		 * device private entries.
 		 */
-		folio_try_dup_anon_rmap_pmd(src_folio, &src_folio->page,
+		folio_try_dup_anon_rmap_pmd(src_folio, folio_page(src_folio, 0),
 					    dst_vma, src_vma);
 	}
 
@@ -2487,7 +2489,7 @@ static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct *vma,
 
 	/* Present and device private folios are rmappable. */
 	if (is_present || is_device_private)
-		folio_remove_rmap_pmd(folio, &folio->page, vma);
+		folio_remove_rmap_pmd(folio, folio_page(folio, 0), vma);
 
 	if (folio_test_anon(folio)) {
 		add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR);
@@ -2572,7 +2574,7 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 
 	spin_unlock(ptl);
 	if (is_present && folio)
-		tlb_remove_page_size(tlb, &folio->page, HPAGE_PMD_SIZE);
+		tlb_remove_page_size(tlb, folio_page(folio, 0), HPAGE_PMD_SIZE);
 	return true;
 }
 
@@ -2947,7 +2949,7 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 	}
 	if (src_folio) {
 		if (folio_maybe_dma_pinned(src_folio) ||
-		    !PageAnonExclusive(&src_folio->page)) {
+		    !PageAnonExclusive(folio_page(src_folio, 0))) {
 			err = -EBUSY;
 			goto unlock_ptls;
 		}
@@ -5186,12 +5188,13 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *folio
 		if (!softleaf_is_migration_read(entry))
 			rmap_flags |= RMAP_EXCLUSIVE;
 
-		folio_add_anon_rmap_pmd(folio, &folio->page, vma, haddr, rmap_flags);
+		folio_add_anon_rmap_pmd(folio, folio_page(folio, 0), vma,
+					haddr, rmap_flags);
 	} else {
-		folio_add_file_rmap_pmd(folio, &folio->page, vma);
+		folio_add_file_rmap_pmd(folio, folio_page(folio, 0), vma);
 	}
 	VM_WARN_ON_ONCE(pmd_write(pmde) && folio_test_anon(folio) &&
-			!PageAnonExclusive(&folio->page));
+			!PageAnonExclusive(folio_page(folio, 0)));
 	set_pmd_at(mm, haddr, pvmw->pmd, pmde);
 
 	/* No need to invalidate - it was non-present before */
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 03/15] mm/huge_memory: use folio_page() in THP split paths

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct &folio->page uses and pointer arithmetic with
folio_page() in __split_folio_to_order(), __split_frozen_folio(),
folio_split_unmapped(), __split_huge_page_to_list_to_order(),
folio_split(), and split_folio_to_list().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/huge_memory.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 31af817eeded..58fbc4d4ddb8 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3711,7 +3711,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 	 * the flags from the original folio.
 	 */
 	for (i = new_nr_pages; i < nr_pages; i += new_nr_pages) {
-		struct page *new_head = &folio->page + i;
+		struct page *new_head = folio_page(folio, i);
 		/*
 		 * Careful: new_folio is not a "real" folio before we cleared PageTail.
 		 * Don't pass it around before clear_compound_head().
@@ -3808,7 +3808,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 	if (new_order)
 		folio_set_order(folio, new_order);
 	else
-		ClearPageCompound(&folio->page);
+		ClearPageCompound(folio_page(folio, 0));
 }
 
 /**
@@ -3898,7 +3898,7 @@ static int __split_frozen_folio(struct folio *folio, int new_order,
 		}
 
 		folio_split_memcg_refs(folio, old_order, split_order);
-		split_page_owner(&folio->page, old_order, split_order);
+		split_page_owner(folio_page(folio, 0), old_order, split_order);
 		pgalloc_tag_split(folio, old_order, split_order);
 		__split_folio_to_order(folio, old_order, split_order);
 
@@ -4394,7 +4394,7 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio);
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
 
-	return __folio_freeze_split_anon(folio, new_order, &folio->page,
+	return __folio_freeze_split_anon(folio, new_order, folio_page(folio, 0),
 					 false, NULL, SPLIT_TYPE_UNIFORM);
 }
 
@@ -4450,7 +4450,7 @@ int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list
 {
 	struct folio *folio = page_folio(page);
 
-	return __folio_split(folio, new_order, &folio->page, page, list,
+	return __folio_split(folio, new_order, folio_page(folio, 0), page, list,
 			     SPLIT_TYPE_UNIFORM);
 }
 
@@ -4481,8 +4481,8 @@ int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list
 int folio_split(struct folio *folio, unsigned int new_order,
 		struct page *split_at, struct list_head *list)
 {
-	return __folio_split(folio, new_order, split_at, &folio->page, list,
-			     SPLIT_TYPE_NON_UNIFORM);
+	return __folio_split(folio, new_order, split_at, folio_page(folio, 0),
+			     list, SPLIT_TYPE_NON_UNIFORM);
 }
 
 /**
@@ -4514,7 +4514,7 @@ unsigned int min_order_for_split(struct folio *folio)
 
 int split_folio_to_list(struct folio *folio, struct list_head *list)
 {
-	return split_huge_page_to_list_to_order(&folio->page, list, 0);
+	return split_huge_page_to_list_to_order(folio_page(folio, 0), list, 0);
 }
 
 /*
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 04/15] mm/hugetlb_vmemmap: use folio_page() for vmemmap address arithmetic

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct (unsigned long)&folio->page casts and the implicit folio
pointer passed to copy_page() with folio_page(folio, 0) so HugeTLB
Vmemmap Optimization (HVO) operates on the vmemmap head page rather than
the struct folio address when struct folio is separately allocated.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/hugetlb_vmemmap.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 0057fa2a16a8..807c6bb23997 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -403,7 +403,7 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h,
 	if (!folio_test_hugetlb_vmemmap_optimized(folio))
 		return 0;
 
-	vmemmap_start	= (unsigned long)&folio->page;
+	vmemmap_start	= (unsigned long)folio_page(folio, 0);
 	vmemmap_end	= vmemmap_start + hugetlb_vmemmap_size(h);
 
 	vmemmap_start	+= HUGETLB_VMEMMAP_RESERVE_SIZE;
@@ -531,11 +531,11 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 		goto out;
 	}
 
-	copy_page(page_to_virt(vmemmap_head), folio);
+	copy_page(page_to_virt(vmemmap_head), folio_page(folio, 0));
 	list_add(&vmemmap_head->lru, vmemmap_pages);
 	memmap_pages_add(1);
 
-	vmemmap_start	= (unsigned long)&folio->page;
+	vmemmap_start	= (unsigned long)folio_page(folio, 0);
 	vmemmap_end	= vmemmap_start + hugetlb_vmemmap_size(h);
 
 	/*
@@ -578,7 +578,7 @@ static int hugetlb_vmemmap_split_folio(const struct hstate *h, struct folio *fol
 	if (!vmemmap_should_optimize_folio(h, folio))
 		return 0;
 
-	vmemmap_start	= (unsigned long)&folio->page;
+	vmemmap_start	= (unsigned long)folio_page(folio, 0);
 	vmemmap_end	= vmemmap_start + hugetlb_vmemmap_size(h);
 
 	/*
@@ -610,7 +610,7 @@ static void __hugetlb_vmemmap_optimize_folios(struct hstate *h,
 			 * Already optimized by pre-HVO, just map the
 			 * mirrored tail page structs RO.
 			 */
-			spfn = (unsigned long)&folio->page;
+			spfn = (unsigned long)folio_page(folio, 0);
 			epfn = spfn + hugetlb_vmemmap_size(h);
 			vmemmap_wrprotect_hvo(spfn, epfn, folio_nid(folio),
 					HUGETLB_VMEMMAP_RESERVE_SIZE);
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 05/15] mm/hugetlb_cma: use folio_page() in hugetlb_cma_free_frozen_folio()

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Pass folio_page(folio, 0) instead of &folio->page to
cma_release_frozen() in hugetlb_cma_free_frozen_folio().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/hugetlb_cma.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c
index 95fd2d190f0d..e0fdd07cc6a9 100644
--- a/mm/hugetlb_cma.c
+++ b/mm/hugetlb_cma.c
@@ -47,7 +47,8 @@ static phys_addr_t __init memblock_node_memory_size(int nid)
 void hugetlb_cma_free_frozen_folio(struct folio *folio)
 {
 	WARN_ON_ONCE(!cma_release_frozen(hugetlb_cma[folio_nid(folio)],
-					 &folio->page, folio_nr_pages(folio)));
+					 folio_page(folio, 0),
+					 folio_nr_pages(folio)));
 }
 
 struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 06/15] mm/hugetlb: replace &folio->page with folio_page() and VM_BUG_ON_FOLIO()

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct &folio->page uses in mm/hugetlb.c with
folio_page(folio, 0) and convert VM_BUG_ON_PAGE(..., &old_folio->page)
in hugetlb_wp() to VM_BUG_ON_FOLIO(..., old_folio) as a 1:1 replacement
that avoids referencing &old_folio->page without changing behavior.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/hugetlb.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index a1b51251103b..31ca8777ccba 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1274,7 +1274,7 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
 		if (folio_test_hwpoison(folio))
 			continue;
 
-		if (is_migrate_isolate_page(&folio->page))
+		if (is_migrate_isolate_page(folio_page(folio, 0)))
 			continue;
 
 		list_move(&folio->lru, &h->hugepage_activelist);
@@ -1511,7 +1511,7 @@ static void __update_and_free_hugetlb_folio(struct hstate *h,
 	if (folio_test_hugetlb_cma(folio))
 		hugetlb_cma_free_frozen_folio(folio);
 	else
-		free_frozen_pages(&folio->page, folio_order(folio));
+		free_frozen_pages(folio_page(folio, 0), folio_order(folio));
 }
 
 /*
@@ -1701,7 +1701,7 @@ void free_huge_folio(struct folio *folio)
 
 	hugetlb_set_folio_subpool(folio, NULL);
 	if (folio_test_anon(folio))
-		__ClearPageAnonExclusive(&folio->page);
+		__ClearPageAnonExclusive(folio_page(folio, 0));
 	folio->mapping = NULL;
 	restore_reserve = folio_test_hugetlb_restore_reserve(folio);
 	folio_clear_hugetlb_restore_reserve(folio);
@@ -3238,7 +3238,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
 	 */
 	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
 		__init_single_page(page, pfn, zone, nid);
-		prep_compound_tail(page, &folio->page, order);
+		prep_compound_tail(page, folio_page(folio, 0), order);
 		set_page_count(page, 0);
 	}
 }
@@ -3259,7 +3259,7 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
 	ret = folio_ref_freeze(folio, 1);
 	VM_BUG_ON(!ret);
 	hugetlb_folio_init_tail_vmemmap(folio, h, 1, nr_pages);
-	prep_compound_head(&folio->page, huge_page_order(h));
+	prep_compound_head(folio_page(folio, 0), huge_page_order(h));
 }
 
 /*
@@ -4004,7 +4004,8 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst,
 
 		list_del(&folio->lru);
 
-		split_page_owner(&folio->page, huge_page_order(src), huge_page_order(dst));
+		split_page_owner(folio_page(folio, 0), huge_page_order(src),
+				 huge_page_order(dst));
 		pgalloc_tag_split(folio, huge_page_order(src), huge_page_order(dst));
 
 		for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) {
@@ -5605,9 +5606,9 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf)
 	 * be leaks between processes, for example, with FOLL_GET users.
 	 */
 	if (folio_mapcount(old_folio) == 1 && folio_test_anon(old_folio)) {
-		if (!PageAnonExclusive(&old_folio->page)) {
+		if (!PageAnonExclusive(folio_page(old_folio, 0))) {
 			folio_move_anon_rmap(old_folio, vma);
-			SetPageAnonExclusive(&old_folio->page);
+			SetPageAnonExclusive(folio_page(old_folio, 0));
 		}
 		if (likely(!unshare))
 			set_huge_ptep_maybe_writable(vma, vmf->address,
@@ -5616,8 +5617,8 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf)
 		delayacct_wpcopy_end();
 		return 0;
 	}
-	VM_BUG_ON_PAGE(folio_test_anon(old_folio) &&
-		       PageAnonExclusive(&old_folio->page), &old_folio->page);
+	VM_BUG_ON_FOLIO(folio_test_anon(old_folio) &&
+			PageAnonExclusive(folio_page(old_folio, 0)), old_folio);
 
 	/*
 	 * If the process that created a MAP_PRIVATE mapping is about to perform
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 07/15] mm/page_isolation: use folio_page() in page_is_unmovable()

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace &folio->page with folio_page(folio, 0) when passing the head
page to compound_order() in page_is_unmovable(). Keep compound_order()
rather than folio_order() since page_is_unmovable() inspects pages
without holding a folio reference.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/page_isolation.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 5aaad037384e..29c9e108b2e5 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -47,7 +47,7 @@ bool page_is_unmovable(struct zone *zone, struct page *page,
 		unsigned long nr_pages, pfn;
 		unsigned int order;
 
-		order = compound_order(&folio->page);
+		order = compound_order(folio_page(folio, 0));
 		if (order > MAX_FOLIO_ORDER)
 			return true;
 
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 08/15] mm/compaction: stop casting free pages to struct folio

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

In compaction_alloc_noprof(), pass freepage directly to
post_alloc_hook(), set_page_refcounted(), prep_compound_page(), and
page_rmappable_folio() instead of casting freepage to struct folio *
prematurely and referencing &dst->page. In compaction_free(), use
folio_page(dst, 0) and add &page->lru to the struct page freelist
cc->freepages[order] after free_pages_prepare().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/compaction.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index f1b2060eb201..d708d973250a 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1845,7 +1845,6 @@ static void isolate_freepages(struct compact_control *cc)
 static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long data)
 {
 	struct compact_control *cc = (struct compact_control *)data;
-	struct folio *dst;
 	int order = folio_order(src);
 	bool has_isolated_pages = false;
 	int start_order;
@@ -1878,15 +1877,13 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da
 
 		list_add(&freepage[size].lru, &cc->freepages[start_order]);
 	}
-	dst = (struct folio *)freepage;
-
-	post_alloc_hook(&dst->page, order, __GFP_MOVABLE, ALLOC_DEFAULT);
-	set_page_refcounted(&dst->page);
+	post_alloc_hook(freepage, order, __GFP_MOVABLE, ALLOC_DEFAULT);
+	set_page_refcounted(freepage);
 	if (order)
-		prep_compound_page(&dst->page, order);
+		prep_compound_page(freepage, order);
 	cc->nr_freepages -= 1 << order;
 	cc->nr_migratepages -= 1 << order;
-	return page_rmappable_folio(&dst->page);
+	return page_rmappable_folio(freepage);
 }
 
 static struct folio *compaction_alloc(struct folio *src, unsigned long data)
@@ -1903,10 +1900,10 @@ static void compaction_free(struct folio *dst, unsigned long data)
 {
 	struct compact_control *cc = (struct compact_control *)data;
 	int order = folio_order(dst);
-	struct page *page = &dst->page;
+	struct page *page = folio_page(dst, 0);
 
 	if (folio_put_testzero(dst) && free_pages_prepare(page, order)) {
-		list_add(&dst->lru, &cc->freepages[order]);
+		list_add(&page->lru, &cc->freepages[order]);
 		cc->nr_freepages += 1 << order;
 	}
 	cc->nr_migratepages += 1 << order;
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 09/15] mm/migrate, vmscan: use folio_page() for movable_ops helpers

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct &folio->page, &src->page, and &dst->page uses with
folio_page(..., 0) when calling page_has_movable_ops(),
putback_movable_ops_page(), isolate_movable_ops_page(), and
migrate_movable_ops_page().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/migrate.c | 20 +++++++++++---------
 mm/vmscan.c  |  2 +-
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 7bdcdb57652f..b027c938d830 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -266,8 +266,8 @@ void putback_movable_pages(struct list_head *l)
 			continue;
 		}
 		list_del(&folio->lru);
-		if (unlikely(page_has_movable_ops(&folio->page))) {
-			putback_movable_ops_page(&folio->page);
+		if (unlikely(page_has_movable_ops(folio_page(folio, 0)))) {
+			putback_movable_ops_page(folio_page(folio, 0));
 		} else {
 			node_stat_mod_folio(folio, NR_ISOLATED_ANON +
 					folio_is_file_lru(folio), -folio_nr_pages(folio));
@@ -282,8 +282,8 @@ bool isolate_folio_to_list(struct folio *folio, struct list_head *list)
 	if (folio_test_hugetlb(folio))
 		return folio_isolate_hugetlb(folio, list);
 
-	if (page_has_movable_ops(&folio->page)) {
-		if (!isolate_movable_ops_page(&folio->page,
+	if (page_has_movable_ops(folio_page(folio, 0))) {
+		if (!isolate_movable_ops_page(folio_page(folio, 0),
 					      ISOLATE_UNEVICTABLE))
 			return false;
 	} else {
@@ -1199,7 +1199,8 @@ static void migrate_folio_undo_dst(struct folio *dst, bool locked,
 static void migrate_folio_done(struct folio *src,
 			       enum migrate_reason reason)
 {
-	if (likely(!page_has_movable_ops(&src->page)) && reason != MR_DEMOTION)
+	if (likely(!page_has_movable_ops(folio_page(src, 0))) &&
+	    reason != MR_DEMOTION)
 		mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON +
 				    folio_is_file_lru(src), -folio_nr_pages(src));
 
@@ -1308,7 +1309,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		goto out;
 	dst_locked = true;
 
-	if (unlikely(page_has_movable_ops(&src->page))) {
+	if (unlikely(page_has_movable_ops(folio_page(src, 0)))) {
 		__migrate_folio_record(dst, old_folio_state, anon_vma);
 		return 0;
 	}
@@ -1375,8 +1376,9 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,
 	prev = dst->lru.prev;
 	list_del(&dst->lru);
 
-	if (unlikely(page_has_movable_ops(&src->page))) {
-		rc = migrate_movable_ops_page(&dst->page, &src->page, mode);
+	if (unlikely(page_has_movable_ops(folio_page(src, 0)))) {
+		rc = migrate_movable_ops_page(folio_page(dst, 0),
+					      folio_page(src, 0), mode);
 		if (rc)
 			goto out;
 		goto out_unlock_both;
@@ -1903,7 +1905,7 @@ static int migrate_pages_batch(struct list_head *from,
 			 * If we are holding the last folio reference, the folio
 			 * was freed from under us, so just drop our reference.
 			 */
-			if (likely(!page_has_movable_ops(&folio->page)) &&
+			if (likely(!page_has_movable_ops(folio_page(folio, 0))) &&
 			    folio_ref_count(folio) == 1) {
 				folio_clear_active(folio);
 				folio_clear_unevictable(folio);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 91295070ca33..150ffeec3d47 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1709,7 +1709,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone,
 
 	list_for_each_entry_safe(folio, next, folio_list, lru) {
 		/* TODO: these pages should not even appear in this list. */
-		if (page_has_movable_ops(&folio->page))
+		if (page_has_movable_ops(folio_page(folio, 0)))
 			continue;
 		if (!folio_test_hugetlb(folio) && folio_is_file_lru(folio) &&
 		    !folio_test_dirty(folio) && !folio_test_unevictable(folio)) {
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 10/15] mm/memory: use folio_page() in fault and CoW paths

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct &folio->page and &new_folio->page uses with
folio_page(..., 0) across copy_present_page(), wp_page_copy(),
do_swap_page(), do_set_pmd(), finish_fault(), and do_cow_fault().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/memory.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 330cde31bf8b..9252ba338112 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1098,7 +1098,8 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma
 	 * over and copy the page & arm it.
 	 */
 
-	if (copy_mc_user_highpage(&new_folio->page, page, addr, src_vma))
+	if (copy_mc_user_highpage(folio_page(new_folio, 0), page, addr,
+				  src_vma))
 		return -EHWPOISON;
 
 	*prealloc = NULL;
@@ -4073,7 +4074,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
 	if (!pfn_is_zero) {
 		int err;
 
-		err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf);
+		err = __wp_page_copy_user(folio_page(new_folio, 0), vmf->page,
+					  vmf);
 		if (err) {
 			/*
 			 * COW failed, if the fault was solved by other,
@@ -4089,7 +4091,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
 			delayacct_wpcopy_end();
 			return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0;
 		}
-		kmsan_copy_page_meta(&new_folio->page, vmf->page);
+		kmsan_copy_page_meta(folio_page(new_folio, 0), vmf->page);
 	}
 
 	__folio_mark_uptodate(new_folio);
@@ -5143,7 +5145,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		ptep = folio_ptep;
 		nr_pages = nr;
 		entry = folio->swap;
-		page = &folio->page;
+		page = folio_page(folio, 0);
 	}
 
 check_folio:
@@ -5697,7 +5699,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa
 
 	if (!is_pmd_order(folio_order(folio)))
 		return ret;
-	page = &folio->page;
+	page = folio_page(folio, 0);
 
 	/*
 	 * Just backoff if any subpage of a THP is corrupted otherwise
@@ -5908,7 +5910,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf)
 		} else {
 			/* Now we can set mappings for the whole large folio. */
 			addr = vmf->address - idx * PAGE_SIZE;
-			page = &folio->page;
+			page = folio_page(folio, 0);
 		}
 	}
 
@@ -6093,7 +6095,7 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf)
 	if (!folio)
 		return VM_FAULT_OOM;
 
-	vmf->cow_page = &folio->page;
+	vmf->cow_page = folio_page(folio, 0);
 
 	ret = __do_fault(vmf);
 	if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 11/15] mm/rmap, gup: use folio_page() for PageAnonExclusive and gup_must_unshare

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace &folio->page with folio_page(folio, 0) when checking or setting
PageAnonExclusive() in folio_add_new_anon_rmap(),
hugetlb_add_anon_rmap(), hugetlb_add_new_anon_rmap(), and
sanity_check_pinned_pages(), and when calling gup_must_unshare() in
gup_fast_pmd_leaf() and gup_fast_pud_leaf().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/gup.c  | 11 +++++++----
 mm/rmap.c | 10 +++++-----
 2 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/mm/gup.c b/mm/gup.c
index 8e9ef5ee7498..1ce7ea5e7abb 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -59,10 +59,11 @@ static inline void sanity_check_pinned_pages(struct page **pages,
 		    !folio_test_anon(folio))
 			continue;
 		if (!folio_test_large(folio) || folio_test_hugetlb(folio))
-			VM_WARN_ON_ONCE_FOLIO(!PageAnonExclusive(&folio->page), folio);
+			VM_WARN_ON_ONCE_FOLIO(!PageAnonExclusive(folio_page(folio, 0)),
+					      folio);
 		else
 			/* Either a PTE-mapped or a PMD-mapped THP. */
-			VM_WARN_ON_ONCE_PAGE(!PageAnonExclusive(&folio->page) &&
+			VM_WARN_ON_ONCE_PAGE(!PageAnonExclusive(folio_page(folio, 0)) &&
 					     !PageAnonExclusive(page), page);
 	}
 }
@@ -2957,7 +2958,8 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
 		gup_put_folio(folio, nr_pages, flags);
 		return 0;
 	}
-	if (!pmd_write(orig) && gup_must_unshare(NULL, flags, &folio->page)) {
+	if (!pmd_write(orig) &&
+	    gup_must_unshare(NULL, flags, folio_page(folio, 0))) {
 		gup_put_folio(folio, nr_pages, flags);
 		return 0;
 	}
@@ -2999,7 +3001,8 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
 		return 0;
 	}
 
-	if (!pud_write(orig) && gup_must_unshare(NULL, flags, &folio->page)) {
+	if (!pud_write(orig) &&
+	    gup_must_unshare(NULL, flags, folio_page(folio, 0))) {
 		gup_put_folio(folio, nr_pages, flags);
 		return 0;
 	}
diff --git a/mm/rmap.c b/mm/rmap.c
index fbd66a2823b7..cd1cf570c770 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1668,7 +1668,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 		/* increment count (starts at -1) */
 		atomic_set(&folio->_mapcount, 0);
 		if (exclusive)
-			SetPageAnonExclusive(&folio->page);
+			SetPageAnonExclusive(folio_page(folio, 0));
 	} else if (!folio_test_pmd_mappable(folio)) {
 		int i;
 
@@ -1694,7 +1694,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 		if (IS_ENABLED(CONFIG_PAGE_MAPCOUNT))
 			atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED);
 		if (exclusive)
-			SetPageAnonExclusive(&folio->page);
+			SetPageAnonExclusive(folio_page(folio, 0));
 		nr_pmdmapped = nr;
 	}
 
@@ -3187,9 +3187,9 @@ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma,
 	atomic_inc(&folio->_entire_mapcount);
 	atomic_inc(&folio->_large_mapcount);
 	if (flags & RMAP_EXCLUSIVE)
-		SetPageAnonExclusive(&folio->page);
+		SetPageAnonExclusive(folio_page(folio, 0));
 	VM_WARN_ON_FOLIO(folio_entire_mapcount(folio) > 1 &&
-			 PageAnonExclusive(&folio->page), folio);
+			 PageAnonExclusive(folio_page(folio, 0)), folio);
 }
 
 void hugetlb_add_new_anon_rmap(struct folio *folio,
@@ -3203,6 +3203,6 @@ void hugetlb_add_new_anon_rmap(struct folio *folio,
 	atomic_set(&folio->_large_mapcount, 0);
 	folio_clear_hugetlb_restore_reserve(folio);
 	__folio_set_anon(folio, vma, address, true);
-	SetPageAnonExclusive(&folio->page);
+	SetPageAnonExclusive(folio_page(folio, 0));
 }
 #endif /* CONFIG_HUGETLB_PAGE */
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 12/15] mm/ksm, userfaultfd: replace &folio->page with folio_page()

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct &folio->page, &kfolio->page, &tree_folio->page, and
&src_folio->page uses with folio_page(..., 0) across KSM tree
comparison/merging and userfaultfd page population/movement paths.

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/ksm.c         | 16 ++++++++++------
 mm/userfaultfd.c | 14 ++++++++------
 2 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index f33523a84a1a..bb8c564d9ffb 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -1112,7 +1112,8 @@ struct ksm_stable_node *folio_stable_node(const struct folio *folio)
 static inline void folio_set_stable_node(struct folio *folio,
 					 struct ksm_stable_node *stable_node)
 {
-	VM_WARN_ON_FOLIO(folio_test_anon(folio) && PageAnonExclusive(&folio->page), folio);
+	VM_WARN_ON_FOLIO(folio_test_anon(folio) &&
+			 PageAnonExclusive(folio_page(folio, 0)), folio);
 	WRITE_ONCE(folio->mapping,
 		   (void *)((unsigned long)stable_node | FOLIO_MAPPING_KSM));
 }
@@ -1324,7 +1325,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct folio *folio,
 	if (unlikely(!pte_present(entry)))
 		goto out_unlock;
 
-	anon_exclusive = PageAnonExclusive(&folio->page);
+	anon_exclusive = PageAnonExclusive(folio_page(folio, 0));
 	if (pte_write(entry) || pte_dirty(entry) ||
 	    anon_exclusive || mm_tlb_flush_pending(mm)) {
 		swapped = folio_test_swapcache(folio);
@@ -1355,7 +1356,8 @@ static int write_protect_page(struct vm_area_struct *vma, struct folio *folio,
 
 		/* See folio_try_share_anon_rmap_pte(): clear PTE first. */
 		if (anon_exclusive &&
-		    folio_try_share_anon_rmap_pte(folio, &folio->page)) {
+		    folio_try_share_anon_rmap_pte(folio,
+						  folio_page(folio, 0))) {
 			set_pte_at(mm, pvmw.address, pvmw.pte, entry);
 			goto out_unlock;
 		}
@@ -1893,7 +1895,7 @@ static struct folio *stable_tree_search(struct page *page)
 			goto again;
 		}
 
-		ret = memcmp_pages(page, &tree_folio->page);
+		ret = memcmp_pages(page, folio_page(tree_folio, 0));
 		folio_put(tree_folio);
 
 		parent = *new;
@@ -2100,7 +2102,8 @@ static struct ksm_stable_node *stable_tree_insert(struct folio *kfolio)
 			goto again;
 		}
 
-		ret = memcmp_pages(&kfolio->page, &tree_folio->page);
+		ret = memcmp_pages(folio_page(kfolio, 0),
+				   folio_page(tree_folio, 0));
 		folio_put(tree_folio);
 
 		parent = *new;
@@ -2333,7 +2336,8 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite
 		if (kfolio == ERR_PTR(-EBUSY))
 			return;
 
-		err = try_to_merge_with_ksm_page(rmap_item, page, &kfolio->page);
+		err = try_to_merge_with_ksm_page(rmap_item, page,
+						 folio_page(kfolio, 0));
 		if (!err) {
 			/*
 			 * The page was successfully merged:
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 3c7fd39deb13..a39d7d40a76b 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -565,7 +565,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 				goto err_folio_put;
 		}
 	} else if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE)) {
-		clear_user_highpage(&folio->page, state->dst_addr);
+		clear_user_highpage(folio_page(folio, 0), state->dst_addr);
 	} else {
 		VM_WARN_ONCE(1, "Unknown UFFDIO operation, flags: %x", flags);
 	}
@@ -584,7 +584,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 	}
 
 	ret = mfill_atomic_install_pte(state->pmd, state->vma, dst_addr,
-				       &folio->page, flags);
+				       folio_page(folio, 0), flags);
 	if (ret)
 		goto err_filemap_remove;
 
@@ -1268,7 +1268,8 @@ static struct folio *check_ptes_for_batched_move(struct vm_area_struct *src_vma,
 	folio = vm_normal_folio(src_vma, src_addr, orig_src_pte);
 	if (!folio || !folio_trylock(folio))
 		return NULL;
-	if (!PageAnonExclusive(&folio->page) || folio_test_large(folio)) {
+	if (!PageAnonExclusive(folio_page(folio, 0)) ||
+	    folio_test_large(folio)) {
 		folio_unlock(folio);
 		return NULL;
 	}
@@ -1306,7 +1307,7 @@ static long move_present_ptes(struct mm_struct *mm,
 	}
 	if (folio_test_large(src_folio) ||
 	    folio_maybe_dma_pinned(src_folio) ||
-	    !PageAnonExclusive(&src_folio->page)) {
+	    !PageAnonExclusive(folio_page(src_folio, 0))) {
 		err = -EBUSY;
 		goto out;
 	}
@@ -1590,7 +1591,8 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd
 			}
 
 			folio = vm_normal_folio(src_vma, src_addr, orig_src_pte);
-			if (!folio || !PageAnonExclusive(&folio->page)) {
+			if (!folio ||
+			    !PageAnonExclusive(folio_page(folio, 0))) {
 				spin_unlock(src_ptl);
 				ret = -EBUSY;
 				goto out;
@@ -2072,7 +2074,7 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 					struct folio *folio = pmd_folio(*src_pmd);
 
 					if (!is_huge_zero_folio(folio) &&
-					    !PageAnonExclusive(&folio->page)) {
+					    !PageAnonExclusive(folio_page(folio, 0))) {
 						spin_unlock(ptl);
 						err = -EBUSY;
 						break;
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 13/15] mm/page_alloc, folio: use folio_page() when freeing folios

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace &folio->page with folio_page(folio, 0) in __folio_put() and
free_unref_folios().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/folio.c      |  2 +-
 mm/page_alloc.c | 18 ++++++++++--------
 2 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/mm/folio.c b/mm/folio.c
index 35e242b48870..978565add078 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -103,7 +103,7 @@ void __folio_put(struct folio *folio)
 	page_cache_release(folio);
 	folio_unqueue_deferred_split(folio);
 	mem_cgroup_uncharge(folio);
-	free_frozen_pages(&folio->page, folio_order(folio));
+	free_frozen_pages(folio_page(folio, 0), folio_order(folio));
 }
 EXPORT_SYMBOL(__folio_put);
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 07e03c7c7771..2f8cd3da199d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3006,17 +3006,18 @@ void free_unref_folios(struct folio_batch *folios)
 	/* Prepare folios for freeing */
 	for (i = 0, j = 0; i < folios->nr; i++) {
 		struct folio *folio = folios->folios[i];
+		struct page *page = folio_page(folio, 0);
 		unsigned long pfn = folio_pfn(folio);
 		unsigned int order = folio_order(folio);
 
-		if (!__free_pages_prepare(&folio->page, order, FPI_NONE))
+		if (!__free_pages_prepare(page, order, FPI_NONE))
 			continue;
 		/*
 		 * Free orders not handled on the PCP directly to the
 		 * allocator.
 		 */
 		if (!pcp_allowed_order(order)) {
-			free_one_page(folio_zone(folio), &folio->page,
+			free_one_page(folio_zone(folio), page,
 				      pfn, order, FPI_NONE);
 			continue;
 		}
@@ -3029,13 +3030,14 @@ void free_unref_folios(struct folio_batch *folios)
 
 	for (i = 0; i < folios->nr; i++) {
 		struct folio *folio = folios->folios[i];
+		struct page *page = folio_page(folio, 0);
 		struct zone *zone = folio_zone(folio);
 		unsigned long pfn = folio_pfn(folio);
 		unsigned int order = (unsigned long)folio->private;
 		int migratetype;
 
 		folio->private = NULL;
-		migratetype = get_pfnblock_migratetype(&folio->page, pfn);
+		migratetype = get_pfnblock_migratetype(page, pfn);
 
 		/* Different zone requires a different pcp lock */
 		if (zone != locked_zone ||
@@ -3051,7 +3053,7 @@ void free_unref_folios(struct folio_batch *folios)
 			 * allocator, see comment in free_frozen_pages.
 			 */
 			if (is_migrate_isolate(migratetype)) {
-				free_one_page(zone, &folio->page, pfn,
+				free_one_page(zone, page, pfn,
 					      order, FPI_NONE);
 				continue;
 			}
@@ -3062,7 +3064,7 @@ void free_unref_folios(struct folio_batch *folios)
 			 */
 			pcp = pcp_spin_trylock(zone->per_cpu_pageset);
 			if (unlikely(!pcp)) {
-				free_one_page(zone, &folio->page, pfn,
+				free_one_page(zone, page, pfn,
 					      order, FPI_NONE);
 				continue;
 			}
@@ -3076,9 +3078,9 @@ void free_unref_folios(struct folio_batch *folios)
 		if (unlikely(migratetype >= MIGRATE_PCPTYPES))
 			migratetype = MIGRATE_MOVABLE;
 
-		trace_mm_page_free_batched(&folio->page);
-		if (!free_frozen_page_commit(zone, pcp, &folio->page,
-				migratetype, order, FPI_NONE)) {
+		trace_mm_page_free_batched(page);
+		if (!free_frozen_page_commit(zone, pcp, page,
+					     migratetype, order, FPI_NONE)) {
 			pcp = NULL;
 			locked_zone = NULL;
 		}
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 14/15] mm/page_owner, alloc_tag: use folio_page() for page_ext and codetag

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Replace direct &folio->page, &old->page, &new->page, and &newfolio->page
uses with folio_page(..., 0) in __folio_set_owner_migrate_reason(),
__folio_copy_owner(), pgalloc_tag_split(), and pgalloc_tag_swap().

No functional change intended.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/alloc_tag.c  | 10 +++++-----
 mm/page_owner.c | 15 +++++++++------
 2 files changed, 14 insertions(+), 11 deletions(-)

diff --git a/mm/alloc_tag.c b/mm/alloc_tag.c
index 82e2c3448dcf..c93efb224ce1 100644
--- a/mm/alloc_tag.c
+++ b/mm/alloc_tag.c
@@ -534,7 +534,7 @@ void pgalloc_tag_split(struct folio *folio, int old_order, int new_order)
 	if (!mem_alloc_profiling_enabled())
 		return;
 
-	tag = __pgalloc_tag_get(&folio->page);
+	tag = __pgalloc_tag_get(folio_page(folio, 0));
 	if (!tag)
 		return;
 
@@ -560,16 +560,16 @@ void pgalloc_tag_swap(struct folio *new, struct folio *old)
 	if (!mem_alloc_profiling_enabled())
 		return;
 
-	tag_old = __pgalloc_tag_get(&old->page);
+	tag_old = __pgalloc_tag_get(folio_page(old, 0));
 	if (!tag_old)
 		return;
-	tag_new = __pgalloc_tag_get(&new->page);
+	tag_new = __pgalloc_tag_get(folio_page(new, 0));
 	if (!tag_new)
 		return;
 
-	if (!get_page_tag_ref(&old->page, &ref_old, &handle_old))
+	if (!get_page_tag_ref(folio_page(old, 0), &ref_old, &handle_old))
 		return;
-	if (!get_page_tag_ref(&new->page, &ref_new, &handle_new)) {
+	if (!get_page_tag_ref(folio_page(new, 0), &ref_new, &handle_new)) {
 		put_page_tag_ref(handle_old);
 		return;
 	}
diff --git a/mm/page_owner.c b/mm/page_owner.c
index cfc31c92d765..39c381699a63 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -365,7 +365,7 @@ noinline void __set_page_owner(struct page *page, unsigned short order,
 
 void __folio_set_owner_migrate_reason(struct folio *folio, enum migrate_reason reason)
 {
-	struct page_ext *page_ext = page_ext_get(&folio->page);
+	struct page_ext *page_ext = page_ext_get(folio_page(folio, 0));
 	struct page_owner *page_owner;
 
 	if (unlikely(!page_ext))
@@ -398,14 +398,14 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old)
 	struct page_owner *new_page_owner;
 	depot_stack_handle_t migrate_handle;
 
-	page_ext = page_ext_get(&old->page);
+	page_ext = page_ext_get(folio_page(old, 0));
 	if (unlikely(!page_ext))
 		return;
 
 	old_page_owner = get_page_owner(page_ext);
 	page_ext_put(page_ext);
 
-	page_ext = page_ext_get(&newfolio->page);
+	page_ext = page_ext_get(folio_page(newfolio, 0));
 	if (unlikely(!page_ext))
 		return;
 
@@ -413,7 +413,8 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old)
 	page_ext_put(page_ext);
 
 	migrate_handle = new_page_owner->handle;
-	__update_page_owner_handle(&newfolio->page, old_page_owner->handle,
+	__update_page_owner_handle(folio_page(newfolio, 0),
+				   old_page_owner->handle,
 				   old_page_owner->order, old_page_owner->gfp_mask,
 				   old_page_owner->last_migrate_reason,
 				   old_page_owner->ts_nsec, old_page_owner->pid,
@@ -423,7 +424,8 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old)
 	 * will be freed after migration. Keep them until then as they may be
 	 * useful.
 	 */
-	__update_page_owner_free_handle(&newfolio->page, 0, old_page_owner->order,
+	__update_page_owner_free_handle(folio_page(newfolio, 0), 0,
+					old_page_owner->order,
 					old_page_owner->free_pid,
 					old_page_owner->free_tgid,
 					old_page_owner->free_ts_nsec);
@@ -433,7 +435,8 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old)
 	 * when subtracting those pages from the stack.
 	 */
 	rcu_read_lock();
-	for_each_page_ext(&old->page, 1 << new_page_owner->order, page_ext, iter) {
+	for_each_page_ext(folio_page(old, 0), 1 << new_page_owner->order,
+			  page_ext, iter) {
 		old_page_owner = get_page_owner(page_ext);
 		old_page_owner->handle = migrate_handle;
 	}
-- 
2.56.0.rc1.315.gc6ed9934b7-goog

[PATCH 15/15] mm: replace remaining &folio->page uses with folio_page() and helpers

[Nilay Vaish]

As part of the memdescs effort, struct folio will be allocated
separately from struct page and will no longer overlay the head page.

Convert the remaining &folio->page uses in mm/memory-failure.c,
mm/hwpoison-inject.c, mm/filemap.c, mm/gup_test.c, mm/memory_hotplug.c,
and mm/memremap.c to folio_page(folio, 0) (and folio_set_hwpoison() in
mf_generic_kill_procs()), and use ERR_CAST(folio) when returning an
error pointer as struct page * in do_read_cache_page() and
shmem_read_mapping_page_gfp().

In free_zone_device_folio() (mm/memremap.c), passing folio_page(folio, 0)
to wake_up_var() ensures the wake-up key continues to match the struct
page address that fs/dax.c waits on via ___wait_var_event(page, ...)
once struct folio is separated from struct page.

No functional change intended while struct folio overlays struct page.

Assisted-by: LLM
Signed-off-by: Nilay Vaish <nilayvaish@google.com>
---
 mm/filemap.c         | 6 +++---
 mm/gup_test.c        | 6 ++++--
 mm/hwpoison-inject.c | 2 +-
 mm/memory-failure.c  | 8 ++++----
 mm/memory_hotplug.c  | 2 +-
 mm/memremap.c        | 2 +-
 mm/shmem.c           | 2 +-
 7 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index b74bc1e5015c..02e0c2af594c 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -156,7 +156,7 @@ static void filemap_unaccount_folio(struct address_space *mapping,
 	if (!IS_ENABLED(CONFIG_DEBUG_VM) && unlikely(folio_mapped(folio))) {
 		pr_alert("BUG: Bad page cache in process %s  pfn:%05lx\n",
 			 current->comm, folio_pfn(folio));
-		dump_page(&folio->page, "still mapped when deleted");
+		dump_page(folio_page(folio, 0), "still mapped when deleted");
 		dump_stack();
 		add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE);
 
@@ -3888,7 +3888,7 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf,
 		unsigned long *rss)
 {
 	vm_fault_t ret = 0;
-	struct page *page = &folio->page;
+	struct page *page = folio_page(folio, 0);
 
 	if (PageHWPoison(page))
 		goto out;
@@ -4239,7 +4239,7 @@ static struct page *do_read_cache_page(struct address_space *mapping,
 
 	folio = do_read_cache_folio(mapping, index, filler, file, gfp);
 	if (IS_ERR(folio))
-		return &folio->page;
+		return ERR_CAST(folio);
 	return folio_file_page(folio, index);
 }
 
diff --git a/mm/gup_test.c b/mm/gup_test.c
index ba74bf3f4104..337b58221c79 100644
--- a/mm/gup_test.c
+++ b/mm/gup_test.c
@@ -59,13 +59,15 @@ static void verify_dma_pinned(unsigned int cmd, struct page **pages,
 			if (WARN(!folio_maybe_dma_pinned(folio),
 				 "pages[%lu] is NOT dma-pinned\n", i)) {
 
-				dump_page(&folio->page, "gup_test failure");
+				dump_page(folio_page(folio, 0),
+					  "gup_test failure");
 				break;
 			} else if (cmd == PIN_LONGTERM_BENCHMARK &&
 				WARN(!folio_is_longterm_pinnable(folio),
 				     "pages[%lu] is NOT pinnable but pinned\n",
 				     i)) {
-				dump_page(&folio->page, "gup_test failure");
+				dump_page(folio_page(folio, 0),
+					  "gup_test failure");
 				break;
 			}
 		}
diff --git a/mm/hwpoison-inject.c b/mm/hwpoison-inject.c
index a11222572f97..127e5d75aa0d 100644
--- a/mm/hwpoison-inject.c
+++ b/mm/hwpoison-inject.c
@@ -131,7 +131,7 @@ static int hwpoison_inject(void *data, u64 val)
 	 * the targeted owner (or on a free page).
 	 * memory_failure() will redo the check reliably inside page lock.
 	 */
-	err = hwpoison_filter(&folio->page);
+	err = hwpoison_filter(folio_page(folio, 0));
 	if (err)
 		return 0;
 
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 0c96eb511997..771a75eb5505 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1400,7 +1400,7 @@ static int __get_hwpoison_page(struct page *page, unsigned long flags)
 	 * unsupported type of folio in order to reduce the risk of unexpected
 	 * races caused by taking a folio refcount.
 	 */
-	if (!HWPoisonHandlable(&folio->page, flags))
+	if (!HWPoisonHandlable(folio_page(folio, 0), flags))
 		return -EBUSY;
 
 	if (folio_try_get(folio)) {
@@ -1777,7 +1777,7 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags,
 	if (!cookie)
 		return -EBUSY;
 
-	if (hwpoison_filter(&folio->page)) {
+	if (hwpoison_filter(folio_page(folio, 0))) {
 		rc = -EOPNOTSUPP;
 		goto unlock;
 	}
@@ -1799,7 +1799,7 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags,
 	 * Use this flag as an indication that the dax page has been
 	 * remapped UC to prevent speculative consumption of poison.
 	 */
-	SetPageHWPoison(&folio->page);
+	folio_set_hwpoison(folio);
 
 	/*
 	 * Unlike System-RAM there is no possibility to swap in a
@@ -1808,7 +1808,7 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags,
 	 * SIGBUS (i.e. MF_MUST_KILL)
 	 */
 	flags |= MF_ACTION_REQUIRED | MF_MUST_KILL;
-	collect_procs(folio, &folio->page, &to_kill, true);
+	collect_procs(folio, folio_page(folio, 0), &to_kill, true);
 
 	unmap_and_kill(&to_kill, pfn, folio->mapping, folio->index, flags);
 unlock:
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 796af1028ee2..47bf3b5d3ff2 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1952,7 +1952,7 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 				if (__ratelimit(&migrate_rs)) {
 					pr_warn("migrating pfn %lx failed ret:%d\n",
 						folio_pfn(folio), ret);
-					dump_page(&folio->page,
+					dump_page(folio_page(folio, 0),
 						  "migration failure");
 				}
 			}
diff --git a/mm/memremap.c b/mm/memremap.c
index accba23aef28..70cfd3f05a25 100644
--- a/mm/memremap.c
+++ b/mm/memremap.c
@@ -467,7 +467,7 @@ void free_zone_device_folio(struct folio *folio)
 		break;
 
 	case MEMORY_DEVICE_FS_DAX:
-		wake_up_var(&folio->page);
+		wake_up_var(folio_page(folio, 0));
 		break;
 
 	case MEMORY_DEVICE_PCI_P2PDMA:
diff --git a/mm/shmem.c b/mm/shmem.c
index 07b2855dfb7b..1dca70978ac9 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -6195,7 +6195,7 @@ struct page *shmem_read_mapping_page_gfp(struct address_space *mapping,
 	struct page *page;
 
 	if (IS_ERR(folio))
-		return &folio->page;
+		return ERR_CAST(folio);
 
 	page = folio_file_page(folio, index);
 	if (PageHWPoison(page)) {
-- 
2.56.0.rc1.315.gc6ed9934b7-goog