* [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 21:17 ` Barry Song
` (2 more replies)
2026-09-14 12:27 ` [RESEND v7 02/29] arm64: mm: add PMD swap-exclusive helpers Usama Arif
` (28 subsequent siblings)
29 siblings, 3 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
pmd_to_softleaf_folio() reads as if it converted a PMD into a folio. What
it does is decode the softleaf entry stored in the PMD and return the
folio that entry references - the direction softleaf_to_folio() already
spells out.
No functional change intended.
Suggested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
include/linux/leafops.h | 4 ++--
mm/huge_memory.c | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/include/linux/leafops.h b/include/linux/leafops.h
index 4c1476ae32343..7c13c58a5e218 100644
--- a/include/linux/leafops.h
+++ b/include/linux/leafops.h
@@ -657,7 +657,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)
}
/**
- * pmd_to_softleaf_folio() - Convert the PMD entry to a folio.
+ * pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio.
* @pmd: PMD entry.
*
* The PMD entry is expected to be a valid PMD softleaf entry.
@@ -665,7 +665,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)
* Returns: the folio the softleaf entry references if this is a valid softleaf
* entry, otherwise NULL.
*/
-static inline struct folio *pmd_to_softleaf_folio(pmd_t pmd)
+static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd)
{
const softleaf_t entry = softleaf_from_pmd(pmd);
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7140a1031fb2e..ee8d46827ffdc 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2518,7 +2518,7 @@ static struct folio *normal_or_softleaf_folio_pmd(struct vm_area_struct *vma,
if (!thp_migration_supported())
WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!");
- return pmd_to_softleaf_folio(pmdval);
+ return pmd_softleaf_to_folio(pmdval);
}
static bool has_deposited_pgtable(struct vm_area_struct *vma, pmd_t pmdval,
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
2026-09-14 12:27 ` [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio() Usama Arif
@ 2026-09-14 21:17 ` Barry Song
2026-09-15 3:06 ` Baoquan He
2026-09-15 3:34 ` Qi Zheng
2 siblings, 0 replies; 35+ messages in thread
From: Barry Song @ 2026-09-14 21:17 UTC (permalink / raw)
To: Usama Arif
Cc: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm,
ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, dev.jain, baolin.wang, Nico Pache,
Liam R. Howlett, ryan.roberts, Vlastimil Babka, lance.yang,
linux-kernel, nphamcs, shikemeng, yosry, qi.zheng, luizcap,
kernel-team
On Mon, Sep 14, 2026 at 8:30 PM Usama Arif <usama.arif@linux.dev> wrote:
>
> pmd_to_softleaf_folio() reads as if it converted a PMD into a folio. What
> it does is decode the softleaf entry stored in the PMD and return the
> folio that entry references - the direction softleaf_to_folio() already
> spells out.
>
> No functional change intended.
>
> Suggested-by: Dev Jain <dev.jain@arm.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>
> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
LGMT,
Reviewed-by: Barry Song <baohua@kernel.org>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
2026-09-14 12:27 ` [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio() Usama Arif
2026-09-14 21:17 ` Barry Song
@ 2026-09-15 3:06 ` Baoquan He
2026-09-15 3:34 ` Qi Zheng
2 siblings, 0 replies; 35+ messages in thread
From: Baoquan He @ 2026-09-15 3:06 UTC (permalink / raw)
To: Usama Arif
Cc: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm,
ying.huang, willy, youngjun.park, hannes, riel, shakeel.butt,
alex, kas, baohua, dev.jain, baolin.wang, Nico Pache,
Liam R. Howlett, ryan.roberts, Vlastimil Babka, lance.yang,
linux-kernel, nphamcs, shikemeng, yosry, qi.zheng, luizcap,
kernel-team
On 09/14/26 at 05:27am, Usama Arif wrote:
> pmd_to_softleaf_folio() reads as if it converted a PMD into a folio. What
> it does is decode the softleaf entry stored in the PMD and return the
> folio that entry references - the direction softleaf_to_folio() already
> spells out.
>
> No functional change intended.
>
> Suggested-by: Dev Jain <dev.jain@arm.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>
> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
> include/linux/leafops.h | 4 ++--
> mm/huge_memory.c | 2 +-
> 2 files changed, 3 insertions(+), 3 deletions(-)
Nice cleanup.
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
>
> diff --git a/include/linux/leafops.h b/include/linux/leafops.h
> index 4c1476ae32343..7c13c58a5e218 100644
> --- a/include/linux/leafops.h
> +++ b/include/linux/leafops.h
> @@ -657,7 +657,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)
> }
>
> /**
> - * pmd_to_softleaf_folio() - Convert the PMD entry to a folio.
> + * pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio.
> * @pmd: PMD entry.
> *
> * The PMD entry is expected to be a valid PMD softleaf entry.
> @@ -665,7 +665,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)
> * Returns: the folio the softleaf entry references if this is a valid softleaf
> * entry, otherwise NULL.
> */
> -static inline struct folio *pmd_to_softleaf_folio(pmd_t pmd)
> +static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd)
> {
> const softleaf_t entry = softleaf_from_pmd(pmd);
>
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 7140a1031fb2e..ee8d46827ffdc 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2518,7 +2518,7 @@ static struct folio *normal_or_softleaf_folio_pmd(struct vm_area_struct *vma,
>
> if (!thp_migration_supported())
> WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!");
> - return pmd_to_softleaf_folio(pmdval);
> + return pmd_softleaf_to_folio(pmdval);
> }
>
> static bool has_deposited_pgtable(struct vm_area_struct *vma, pmd_t pmdval,
> --
> 2.53.0-Meta
>
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio()
2026-09-14 12:27 ` [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio() Usama Arif
2026-09-14 21:17 ` Barry Song
2026-09-15 3:06 ` Baoquan He
@ 2026-09-15 3:34 ` Qi Zheng
2 siblings, 0 replies; 35+ messages in thread
From: Qi Zheng @ 2026-09-15 3:34 UTC (permalink / raw)
To: Usama Arif, Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, luizcap,
kernel-team
On 9/14/26 8:27 PM, Usama Arif wrote:
> pmd_to_softleaf_folio() reads as if it converted a PMD into a folio. What
> it does is decode the softleaf entry stored in the PMD and return the
> folio that entry references - the direction softleaf_to_folio() already
> spells out.
>
> No functional change intended.
>
> Suggested-by: Dev Jain <dev.jain@arm.com>
> Signed-off-by: Usama Arif <usama.arif@linux.dev>
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>
> Reviewed-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> Reviewed-by: Zi Yan <ziy@nvidia.com>
> Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
> include/linux/leafops.h | 4 ++--
> mm/huge_memory.c | 2 +-
> 2 files changed, 3 insertions(+), 3 deletions(-)
Reviewed-by: Qi Zheng <qi.zheng@linux.dev>
Thanks,
Qi
>
> diff --git a/include/linux/leafops.h b/include/linux/leafops.h
> index 4c1476ae32343..7c13c58a5e218 100644
> --- a/include/linux/leafops.h
> +++ b/include/linux/leafops.h
> @@ -657,7 +657,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)
> }
>
> /**
> - * pmd_to_softleaf_folio() - Convert the PMD entry to a folio.
> + * pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio.
> * @pmd: PMD entry.
> *
> * The PMD entry is expected to be a valid PMD softleaf entry.
> @@ -665,7 +665,7 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)
> * Returns: the folio the softleaf entry references if this is a valid softleaf
> * entry, otherwise NULL.
> */
> -static inline struct folio *pmd_to_softleaf_folio(pmd_t pmd)
> +static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd)
> {
> const softleaf_t entry = softleaf_from_pmd(pmd);
>
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 7140a1031fb2e..ee8d46827ffdc 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2518,7 +2518,7 @@ static struct folio *normal_or_softleaf_folio_pmd(struct vm_area_struct *vma,
>
> if (!thp_migration_supported())
> WARN_ONCE(1, "Non present huge pmd without pmd migration enabled!");
> - return pmd_to_softleaf_folio(pmdval);
> + return pmd_softleaf_to_folio(pmdval);
> }
>
> static bool has_deposited_pgtable(struct vm_area_struct *vma, pmd_t pmdval,
^ permalink raw reply [flat|nested] 35+ messages in thread
* [RESEND v7 02/29] arm64: mm: add PMD swap-exclusive helpers
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
2026-09-14 12:27 ` [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio() Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:27 ` [RESEND v7 03/29] loongarch: " Usama Arif
` (27 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif, Catalin Marinas, Will Deacon
A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the
swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and
not just in a swap PTE.
arm64 encodes a swap PMD exactly like a swap PTE, so the new helpers wrap
the PTE ones and reuse PTE_SWP_EXCLUSIVE.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/arm64/include/asm/pgtable.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index e89ec5f4787b4..d3f53a601aed3 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -599,6 +599,12 @@ static inline int pmd_protnone(pmd_t pmd)
#define pmd_swp_clear_uffd(pmd) \
pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)))
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
+#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+#define pmd_swp_exclusive(pmd) pte_swp_exclusive(pmd_pte(pmd))
+#define pmd_swp_mkexclusive(pmd) pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd)))
+#define pmd_swp_clear_exclusive(pmd) \
+ pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd)))
+#endif
#define pmd_write(pmd) pte_write(pmd_pte(pmd))
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 03/29] loongarch: mm: add PMD swap-exclusive helpers
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
2026-09-14 12:27 ` [RESEND v7 01/29] mm: rename pmd_to_softleaf_folio() to pmd_softleaf_to_folio() Usama Arif
2026-09-14 12:27 ` [RESEND v7 02/29] arm64: mm: add PMD swap-exclusive helpers Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:27 ` [RESEND v7 04/29] powerpc: " Usama Arif
` (26 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif, Huacai Chen
A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the
swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and
not just in a swap PTE.
A LoongArch swap PMD is the swap PTE value plus _PAGE_HUGE, and
_PAGE_SWP_EXCLUSIVE sits outside both the type and the offset field, so the
PMD helpers can use the same bit.
Cc: Huacai Chen <chenhuacai@kernel.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/loongarch/include/asm/pgtable.h | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h
index cf29a4c8ac593..87fecc3a51001 100644
--- a/arch/loongarch/include/asm/pgtable.h
+++ b/arch/loongarch/include/asm/pgtable.h
@@ -351,6 +351,25 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
return pte;
}
+#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
+{
+ pmd_val(pmd) |= _PAGE_SWP_EXCLUSIVE;
+ return pmd;
+}
+
+static inline bool pmd_swp_exclusive(pmd_t pmd)
+{
+ return pmd_val(pmd) & _PAGE_SWP_EXCLUSIVE;
+}
+
+static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
+{
+ pmd_val(pmd) &= ~_PAGE_SWP_EXCLUSIVE;
+ return pmd;
+}
+#endif
+
#define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))
#define pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE))
#define pte_no_exec(pte) (pte_val(pte) & _PAGE_NO_EXEC)
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 04/29] powerpc: mm: add PMD swap-exclusive helpers
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (2 preceding siblings ...)
2026-09-14 12:27 ` [RESEND v7 03/29] loongarch: " Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:27 ` [RESEND v7 05/29] riscv: " Usama Arif
` (25 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif, Madhavan Srinivasan
A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the
swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and
not just in a swap PTE.
book3s64 builds a swap PMD by running the PTE encoding over pmd_pte(), so
the PMD helpers use the same _PAGE_SWP_EXCLUSIVE bit. It is also the only
powerpc variant that selects ARCH_HAS_PMD_SOFTLEAVES, via PPC_THP.
Cc: Madhavan Srinivasan <maddy@linux.ibm.com>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/powerpc/include/asm/book3s/64/pgtable.h | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index dff8790a047db..28943ef3c1c80 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -699,6 +699,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
return __pte_raw(pte_raw(pte) & cpu_to_be64(~_PAGE_SWP_EXCLUSIVE));
}
+#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
+{
+ return __pmd_raw(pmd_raw(pmd) | cpu_to_be64(_PAGE_SWP_EXCLUSIVE));
+}
+
+static inline bool pmd_swp_exclusive(pmd_t pmd)
+{
+ return !!(pmd_raw(pmd) & cpu_to_be64(_PAGE_SWP_EXCLUSIVE));
+}
+
+static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
+{
+ return __pmd_raw(pmd_raw(pmd) & cpu_to_be64(~_PAGE_SWP_EXCLUSIVE));
+}
+#endif
+
static inline bool check_pte_access(unsigned long access, unsigned long ptev)
{
/*
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 05/29] riscv: mm: add PMD swap-exclusive helpers
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (3 preceding siblings ...)
2026-09-14 12:27 ` [RESEND v7 04/29] powerpc: " Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:27 ` [RESEND v7 06/29] s390: " Usama Arif
` (24 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif, Paul Walmsley, Palmer Dabbelt,
Albert Ou
A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the
swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and
not just in a swap PTE.
riscv encodes a swap PMD exactly like a swap PTE, so the new helpers wrap
the PTE ones and reuse _PAGE_SWP_EXCLUSIVE.
Cc: Paul Walmsley <pjw@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/riscv/include/asm/pgtable.h | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index d48f90140841e..b644db16bda94 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -1219,6 +1219,21 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
}
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+static inline bool pmd_swp_exclusive(pmd_t pmd)
+{
+ return pte_swp_exclusive(pmd_pte(pmd));
+}
+
+static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
+{
+ return pte_pmd(pte_swp_mkexclusive(pmd_pte(pmd)));
+}
+
+static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
+{
+ return pte_pmd(pte_swp_clear_exclusive(pmd_pte(pmd)));
+}
+
#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pmd(swp) __pmd((swp).val)
#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 06/29] s390: mm: add PMD swap-exclusive helpers
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (4 preceding siblings ...)
2026-09-14 12:27 ` [RESEND v7 05/29] riscv: " Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:27 ` [RESEND v7 07/29] x86: " Usama Arif
` (23 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif, Alexander Gordeev,
Gerald Schaefer, Heiko Carstens, Vasily Gorbik
A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the
swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and
not just in a swap PTE.
s390 is the one architecture where a swap PMD is not a swap PTE in
disguise: it is an RSTE with its own layout, converted to a fake PTE swap
entry for the common code. Give it its own exclusive bit rather than
borrowing the PTE-format macro. The two happen to have the same value, but
that is a coincidence. Bit 52 was documented as unused; document what it
is now.
Cc: Alexander Gordeev <agordeev@linux.ibm.com>
Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/s390/include/asm/pgtable.h | 28 ++++++++++++++++++++++++++--
1 file changed, 26 insertions(+), 2 deletions(-)
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 2d5c2ab06de98..0790a0884cfab 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -333,6 +333,7 @@ void setup_protection_map(void);
/* Common bits in region and segment table entries, for swap entries */
#define _RST_ENTRY_COMM 0x0010 /* Common-Region/Segment, marks swap entry */
#define _RST_ENTRY_INVALID 0x0020 /* invalid region/segment table entry */
+#define _RST_ENTRY_SWP_EXCLUSIVE 0x0800 /* SW exclusive swap bit, see mk_swap_rste() */
#define _CRST_ENTRIES 2048 /* number of region/segment table entries */
#define _PAGE_ENTRIES 256 /* number of page table entries */
@@ -859,6 +860,28 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
return clear_pte_bit(pte, __pgprot(_PAGE_SWP_EXCLUSIVE));
}
+#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+/*
+ * A PMD swap entry is an RSTE, not a PTE, so it needs its own exclusive bit
+ * rather than the PTE-format _PAGE_SWP_EXCLUSIVE. The two happen to have the
+ * same value; see the RSTE swap layout above mk_swap_rste().
+ */
+static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
+{
+ return set_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE));
+}
+
+static inline bool pmd_swp_exclusive(pmd_t pmd)
+{
+ return pmd_val(pmd) & _RST_ENTRY_SWP_EXCLUSIVE;
+}
+
+static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
+{
+ return clear_pmd_bit(pmd, __pgprot(_RST_ENTRY_SWP_EXCLUSIVE));
+}
+#endif
+
static inline int pte_soft_dirty(pte_t pte)
{
return pte_val(pte) & _PAGE_SOFT_DIRTY;
@@ -1900,15 +1923,16 @@ static inline swp_entry_t __swp_entry(unsigned long type, unsigned long offset)
* Bits 59 and 63 are used to indicate the swap entry. Bit 58 marks the rste
* as invalid.
* A swap entry is indicated by bit pattern (rste & 0x011) == 0x010
- * | offset |Xtype |11TT|S0|
+ * | offset |Etype |11TT|S0|
* |0000000000111111111122222222223333333333444444444455|555555|5566|66|
* |0123456789012345678901234567890123456789012345678901|234567|8901|23|
*
* Bits 0-51 store the offset.
+ * Bit 52 (E) is used to remember PG_anon_exclusive
+ * (_RST_ENTRY_SWP_EXCLUSIVE), mirroring bit 52 of a swap pte.
* Bits 53-57 store the type.
* Bit 62 (S) is used for softdirty tracking.
* Bits 60-61 (TT) indicate the table type: 0x01 for REGION3 and 0x00 for SEGMENT.
- * Bit 52 (X) is unused.
*/
#define __SWP_OFFSET_MASK_RSTE ((1UL << 52) - 1)
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 07/29] x86: mm: add PMD swap-exclusive helpers
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (5 preceding siblings ...)
2026-09-14 12:27 ` [RESEND v7 06/29] s390: " Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:27 ` [RESEND v7 08/29] mm: recognize PMD swap entries in the softleaf layer Usama Arif
` (22 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86
A later patch keeps a PMD-mapped anonymous THP mapped by a PMD across the
swap round-trip, so PG_anon_exclusive now has to survive in a swap PMD and
not just in a swap PTE.
x86-64 encodes a swap PMD exactly like a swap PTE, so the new helpers reuse
_PAGE_SWP_EXCLUSIVE, bit 3, which the swap-entry layout already reserves
for PG_anon_exclusive. 32-bit x86 aliases that bit to _PAGE_PSE and does
not select ARCH_HAS_PMD_SOFTLEAVES.
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: x86@kernel.org
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
arch/x86/include/asm/pgtable.h | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index d551120a7c889..a2d1cd03cba23 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -1525,6 +1525,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
return pte_clear_flags(pte, _PAGE_SWP_EXCLUSIVE);
}
+#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
+{
+ return pmd_set_flags(pmd, _PAGE_SWP_EXCLUSIVE);
+}
+
+static inline bool pmd_swp_exclusive(pmd_t pmd)
+{
+ return pmd_flags(pmd) & _PAGE_SWP_EXCLUSIVE;
+}
+
+static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
+{
+ return pmd_clear_flags(pmd, _PAGE_SWP_EXCLUSIVE);
+}
+#endif
+
#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
static inline pte_t pte_swp_mksoft_dirty(pte_t pte)
{
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 08/29] mm: recognize PMD swap entries in the softleaf layer
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (6 preceding siblings ...)
2026-09-14 12:27 ` [RESEND v7 07/29] x86: " Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:27 ` [RESEND v7 09/29] mm/debug_vm_pgtable: test PMD swap-exclusive helpers Usama Arif
` (21 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
Reclaim splits a PMD-mapped anonymous THP into PTE-level swap entries
before unmapping it, so an ordinary swap entry has never had to appear in a
PMD. Later patches install one there instead, and the softleaf layer is
where every consumer decodes non-present PMDs.
Accept swap entries as valid PMD softleaves and add pmd_is_swap_entry().
A swap entry carries no PFN, so make pmd_softleaf_to_folio() warn and
return NULL rather than interpret a swap offset as a page frame number.
Unlike migration and device-private entries, a PMD swap entry can also
carry the swap-exclusive marker, which softleaf_from_pmd() has to strip
before decoding. Strip all three overlays unconditionally while we are
here: each clear is a plain bit clear, so testing first only buys a branch.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
include/linux/leafops.h | 40 ++++++++++++++++++++++++++++------------
include/linux/pgtable.h | 17 +++++++++++++++++
2 files changed, 45 insertions(+), 12 deletions(-)
diff --git a/include/linux/leafops.h b/include/linux/leafops.h
index 7c13c58a5e218..ce176c78cefd4 100644
--- a/include/linux/leafops.h
+++ b/include/linux/leafops.h
@@ -98,10 +98,9 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd)
if (pmd_present(pmd) || pmd_none(pmd))
return softleaf_mk_none();
- if (pmd_swp_soft_dirty(pmd))
- pmd = pmd_swp_clear_soft_dirty(pmd);
- if (pmd_swp_uffd(pmd))
- pmd = pmd_swp_clear_uffd(pmd);
+ pmd = pmd_swp_clear_soft_dirty(pmd);
+ pmd = pmd_swp_clear_uffd(pmd);
+ pmd = pmd_swp_clear_exclusive(pmd);
arch_entry = __pmd_to_swp_entry(pmd);
/* Temporary until swp_entry_t eliminated. */
@@ -634,18 +633,29 @@ static inline bool pmd_is_migration_entry(pmd_t pmd)
*/
static inline bool softleaf_is_valid_pmd_entry(softleaf_t entry)
{
- /* Only device private, migration entries valid for PMD. */
return softleaf_is_device_private(entry) ||
- softleaf_is_migration(entry);
+ softleaf_is_migration(entry) ||
+ softleaf_is_swap(entry);
+}
+
+/**
+ * pmd_is_swap_entry() - Does this PMD entry encode an actual swap entry?
+ * @pmd: PMD entry.
+ *
+ * Returns: true if the PMD encodes a swap entry, otherwise false.
+ */
+static inline bool pmd_is_swap_entry(pmd_t pmd)
+{
+ return softleaf_is_swap(softleaf_from_pmd(pmd));
}
/**
* pmd_is_valid_softleaf() - Is this PMD entry a valid softleaf entry?
* @pmd: PMD entry.
*
- * PMD leaf entries are valid only if they are device private or migration
- * entries. This function asserts that a PMD leaf entry is valid in this
- * respect.
+ * PMD leaf entries are valid only if they are device private, migration,
+ * or swap entries. This function asserts that a PMD leaf entry is valid
+ * in this respect.
*
* Returns: true if the PMD entry is a valid leaf entry, otherwise false.
*/
@@ -660,10 +670,12 @@ static inline bool pmd_is_valid_softleaf(pmd_t pmd)
* pmd_softleaf_to_folio() - Convert the PMD softleaf entry to a folio.
* @pmd: PMD entry.
*
- * The PMD entry is expected to be a valid PMD softleaf entry.
+ * The PMD entry is expected to be a valid PMD softleaf entry that references a
+ * PFN, that is a migration or device private entry. A PMD swap entry is a valid
+ * softleaf entry but encodes swap slots rather than a PFN, so it has no folio.
*
- * Returns: the folio the softleaf entry references if this is a valid softleaf
- * entry, otherwise NULL.
+ * Returns: the folio the softleaf entry references, or NULL if the entry is not
+ * a valid PMD softleaf entry or does not reference a PFN.
*/
static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd)
{
@@ -673,6 +685,10 @@ static inline struct folio *pmd_softleaf_to_folio(pmd_t pmd)
VM_WARN_ON_ONCE(true);
return NULL;
}
+ if (!softleaf_has_pfn(entry)) {
+ VM_WARN_ON_ONCE(true);
+ return NULL;
+ }
return softleaf_to_folio(entry);
}
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index e3c8ab96941c5..3f955f836abfe 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -1917,6 +1917,23 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
}
#endif
+#ifndef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+static inline pmd_t pmd_swp_mkexclusive(pmd_t pmd)
+{
+ return pmd;
+}
+
+static inline bool pmd_swp_exclusive(pmd_t pmd)
+{
+ return false;
+}
+
+static inline pmd_t pmd_swp_clear_exclusive(pmd_t pmd)
+{
+ return pmd;
+}
+#endif
+
#ifndef __HAVE_PFNMAP_TRACKING
/*
* Interfaces that can be used by architecture code to keep track of
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 09/29] mm/debug_vm_pgtable: test PMD swap-exclusive helpers
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (7 preceding siblings ...)
2026-09-14 12:27 ` [RESEND v7 08/29] mm: recognize PMD swap entries in the softleaf layer Usama Arif
@ 2026-09-14 12:27 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 10/29] mm: make PMD migration-entry splitting explicit Usama Arif
` (20 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:27 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
An architecture that picked a PMD exclusive bit overlapping the swap type
or offset field would otherwise only be caught by data corruption at
runtime. Mirror pte_swap_exclusive_tests() at PMD level.
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/debug_vm_pgtable.c | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c
index 2875fd22d7bb0..863111c6d4eb3 100644
--- a/mm/debug_vm_pgtable.c
+++ b/mm/debug_vm_pgtable.c
@@ -802,6 +802,45 @@ static void __init pte_swap_exclusive_tests(struct pgtable_debug_args *args)
WARN_ON(memcmp(&entry, &softleaf, sizeof(entry)));
}
+#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
+static void __init pmd_swap_exclusive_tests(struct pgtable_debug_args *args)
+{
+ swp_entry_t entry;
+ softleaf_t softleaf;
+ pmd_t pmd;
+
+ if (!has_transparent_hugepage())
+ return;
+
+ pr_debug("Validating PMD swap exclusive\n");
+ entry = args->swp_entry;
+
+ pmd = softleaf_to_pmd(entry);
+ softleaf = softleaf_from_pmd(pmd);
+
+ WARN_ON(pmd_swp_exclusive(pmd));
+ WARN_ON(!softleaf_is_swap(softleaf));
+ WARN_ON(memcmp(&entry, &softleaf, sizeof(entry)));
+
+ pmd = pmd_swp_mkexclusive(pmd);
+ softleaf = softleaf_from_pmd(pmd);
+
+ WARN_ON(!pmd_swp_exclusive(pmd));
+ WARN_ON(!softleaf_is_swap(softleaf));
+ WARN_ON(pmd_swp_soft_dirty(pmd));
+ WARN_ON(memcmp(&entry, &softleaf, sizeof(entry)));
+
+ pmd = pmd_swp_clear_exclusive(pmd);
+ softleaf = softleaf_from_pmd(pmd);
+
+ WARN_ON(pmd_swp_exclusive(pmd));
+ WARN_ON(!softleaf_is_swap(softleaf));
+ WARN_ON(memcmp(&entry, &softleaf, sizeof(entry)));
+}
+#else /* !CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
+static void __init pmd_swap_exclusive_tests(struct pgtable_debug_args *args) { }
+#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
+
static void __init pte_swap_tests(struct pgtable_debug_args *args)
{
swp_entry_t arch_entry;
@@ -1322,6 +1361,7 @@ static int __init debug_vm_pgtable(void)
pmd_leaf_soft_dirty_tests(&args);
pte_swap_exclusive_tests(&args);
+ pmd_swap_exclusive_tests(&args);
pte_swap_tests(&args);
pmd_softleaf_tests(&args);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 10/29] mm: make PMD migration-entry splitting explicit
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (8 preceding siblings ...)
2026-09-14 12:27 ` [RESEND v7 09/29] mm/debug_vm_pgtable: test PMD swap-exclusive helpers Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 11/29] mm: split PMD swap entries into PTE swap entries Usama Arif
` (19 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
__split_huge_pmd() and friends take a "freeze" boolean that every caller
has to pass and almost every caller passes as false. The name says nothing
about what it selects, and the one thing it does select - PTE migration
entries instead of PTE mappings - is only ever wanted by the rmap migration
path.
Rename it to use_migration_entries, keep it private to mm/huge_memory.c,
and add split_pmd_to_migration_entries() for try_to_migrate_one(), the only
caller that wants it.
migrate_vma_split_unmapped_folio() also passed freeze=true, but only ever
runs on a PMD that is already a migration entry, which the generic helper
expands into PTE migration entries either way. Its folio_get() only existed
to balance the put_page() that freeze=true performs, so both go.
No functional change intended.
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
include/linux/huge_mm.h | 22 ++++++++-------
mm/huge_memory.c | 60 ++++++++++++++++++++++++-----------------
mm/memory.c | 4 +--
mm/migrate_device.c | 7 +----
mm/mprotect.c | 2 +-
mm/rmap.c | 7 +++--
6 files changed, 55 insertions(+), 47 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 8ca0fa3be2acb..64b6a2eea899d 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -430,7 +430,7 @@ int folio_memcg_alloc_deferred(struct folio *folio);
void deferred_split_folio(struct folio *folio, bool partially_mapped);
void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
- unsigned long address, bool freeze);
+ unsigned long address);
/**
* pmd_is_huge() - Is this PMD either a huge PMD entry or a software leaf entry?
@@ -462,12 +462,10 @@ static inline bool pmd_is_huge(pmd_t pmd)
do { \
pmd_t *____pmd = (__pmd); \
if (pmd_is_huge(*____pmd)) \
- __split_huge_pmd(__vma, __pmd, __address, \
- false); \
+ __split_huge_pmd(__vma, __pmd, __address); \
} while (0)
-void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address,
- bool freeze);
+void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address);
void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud,
unsigned long address);
@@ -590,7 +588,9 @@ static inline bool thp_migration_supported(void)
}
void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
- pmd_t *pmd, bool freeze);
+ pmd_t *pmd);
+void split_pmd_to_migration_entries(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmd);
bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr,
pmd_t *pmdp, struct folio *folio);
void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,
@@ -690,12 +690,14 @@ static inline void deferred_split_folio(struct folio *folio, bool partially_mapp
do { } while (0)
static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
- unsigned long address, bool freeze) {}
+ unsigned long address) {}
static inline void split_huge_pmd_address(struct vm_area_struct *vma,
- unsigned long address, bool freeze) {}
+ unsigned long address) {}
static inline void split_huge_pmd_locked(struct vm_area_struct *vma,
- unsigned long address, pmd_t *pmd,
- bool freeze) {}
+ unsigned long address, pmd_t *pmd) {}
+static inline void
+split_pmd_to_migration_entries(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmd) {}
static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp,
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ee8d46827ffdc..873887aed0bc2 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2033,7 +2033,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
pte_free(dst_mm, pgtable);
spin_unlock(src_ptl);
spin_unlock(dst_ptl);
- __split_huge_pmd(src_vma, src_pmd, addr, false);
+ __split_huge_pmd(src_vma, src_pmd, addr);
return -EAGAIN;
}
add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
@@ -2257,7 +2257,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)
folio_unlock(folio);
spin_unlock(vmf->ptl);
fallback:
- __split_huge_pmd(vma, vmf->pmd, vmf->address, false);
+ __split_huge_pmd(vma, vmf->pmd, vmf->address);
return VM_FAULT_FALLBACK;
}
@@ -3190,7 +3190,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,
}
static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
- unsigned long haddr, bool freeze)
+ unsigned long haddr, bool use_migration_entries)
{
struct mm_struct *mm = vma->vm_mm;
struct folio *folio;
@@ -3291,10 +3291,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
* folios w.r.t anon exclusive handling. See the comments for
* folio handling and anon_exclusive below.
*/
- if (freeze && anon_exclusive &&
+ if (use_migration_entries && anon_exclusive &&
folio_try_share_anon_rmap_pmd(folio, page))
- freeze = false;
- if (!freeze) {
+ use_migration_entries = false;
+ if (!use_migration_entries) {
rmap_t rmap_flags = RMAP_NONE;
folio_ref_add(folio, HPAGE_PMD_NR - 1);
@@ -3344,11 +3344,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
/*
- * Without "freeze", we'll simply split the PMD, propagating the
- * PageAnonExclusive() flag for each PTE by setting it for
+ * Without migration entries, we'll simply split the PMD and
+ * propagate the PageAnonExclusive() flag for each PTE by setting it for
* each subpage -- no need to (temporarily) clear.
*
- * With "freeze" we want to replace mapped pages by
+ * With migration entries we want to replace mapped pages by
* migration entries right away. This is only possible if we
* managed to clear PageAnonExclusive() -- see
* set_pmd_migration_entry().
@@ -3359,10 +3359,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
* See folio_try_share_anon_rmap_pmd(): invalidate PMD first.
*/
anon_exclusive = PageAnonExclusive(page);
- if (freeze && anon_exclusive &&
+ if (use_migration_entries && anon_exclusive &&
folio_try_share_anon_rmap_pmd(folio, page))
- freeze = false;
- if (!freeze) {
+ use_migration_entries = false;
+ if (!use_migration_entries) {
rmap_t rmap_flags = RMAP_NONE;
folio_ref_add(folio, HPAGE_PMD_NR - 1);
@@ -3387,7 +3387,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
* Note that NUMA hinting access restrictions are not transferred to
* avoid any possibility of altering permissions across VMAs.
*/
- if (freeze || pmd_is_migration_entry(old_pmd)) {
+ if (use_migration_entries || pmd_is_migration_entry(old_pmd)) {
pte_t entry;
swp_entry_t swp_entry;
@@ -3420,8 +3420,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) {
/*
* anon_exclusive was already propagated to the relevant
- * pages corresponding to the pte entries when freeze
- * is false.
+ * pages corresponding to the pte entries when
+ * use_migration_entries is false.
*/
if (write)
swp_entry = make_writable_device_private_entry(
@@ -3469,7 +3469,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
if (!pmd_is_migration_entry(*pmd))
folio_remove_rmap_pmd(folio, page, vma);
- if (freeze)
+ if (use_migration_entries)
put_page(page);
smp_wmb(); /* make pte visible before pmd */
@@ -3477,15 +3477,28 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
}
void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address,
- pmd_t *pmd, bool freeze)
+ pmd_t *pmd)
{
VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE));
if (pmd_trans_huge(*pmd) || pmd_is_valid_softleaf(*pmd))
- __split_huge_pmd_locked(vma, pmd, address, freeze);
+ __split_huge_pmd_locked(vma, pmd, address, false);
+}
+
+/*
+ * Split a present PMD into PTE migration entries, for the rmap migration
+ * walker. Like split_huge_pmd_locked(), the caller must hold the PMD lock and
+ * must already be inside an mmu_notifier invalidate range.
+ */
+void split_pmd_to_migration_entries(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmd)
+{
+ VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE));
+ if (pmd_trans_huge(*pmd) || pmd_is_valid_softleaf(*pmd))
+ __split_huge_pmd_locked(vma, pmd, address, true);
}
void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
- unsigned long address, bool freeze)
+ unsigned long address)
{
spinlock_t *ptl;
struct mmu_notifier_range range;
@@ -3495,20 +3508,19 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
(address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE);
mmu_notifier_invalidate_range_start(&range);
ptl = pmd_lock(vma->vm_mm, pmd);
- split_huge_pmd_locked(vma, range.start, pmd, freeze);
+ split_huge_pmd_locked(vma, range.start, pmd);
spin_unlock(ptl);
mmu_notifier_invalidate_range_end(&range);
}
-void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address,
- bool freeze)
+void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address)
{
pmd_t *pmd = mm_find_pmd(vma->vm_mm, address);
if (!pmd)
return;
- __split_huge_pmd(vma, pmd, address, freeze);
+ __split_huge_pmd(vma, pmd, address);
}
static inline void split_huge_pmd_if_needed(struct vm_area_struct *vma, unsigned long address)
@@ -3520,7 +3532,7 @@ static inline void split_huge_pmd_if_needed(struct vm_area_struct *vma, unsigned
if (!IS_ALIGNED(address, HPAGE_PMD_SIZE) &&
range_in_vma(vma, ALIGN_DOWN(address, HPAGE_PMD_SIZE),
ALIGN(address, HPAGE_PMD_SIZE)))
- split_huge_pmd_address(vma, address, false);
+ split_huge_pmd_address(vma, address);
}
void vma_adjust_trans_huge(struct vm_area_struct *vma,
diff --git a/mm/memory.c b/mm/memory.c
index 926276d419202..477d7e359b447 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2096,7 +2096,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb,
next = pmd_addr_end(addr, end);
if (pmd_is_huge(*pmd)) {
if (next - addr != HPAGE_PMD_SIZE)
- __split_huge_pmd(vma, pmd, addr, false);
+ __split_huge_pmd(vma, pmd, addr);
else if (zap_huge_pmd(tlb, vma, pmd, addr)) {
addr = next;
continue;
@@ -6382,7 +6382,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
split:
/* COW or write-notify handled on pte level: split pmd. */
- __split_huge_pmd(vma, vmf->pmd, vmf->address, false);
+ __split_huge_pmd(vma, vmf->pmd, vmf->address);
return VM_FAULT_FALLBACK;
}
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index 0c437004329d9..4a0b61d50d222 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -918,12 +918,7 @@ static int migrate_vma_split_unmapped_folio(struct migrate_vma *migrate,
unsigned long flags;
int ret = 0;
- /*
- * take a reference, since split_huge_pmd_address() with freeze = true
- * drops a reference at the end.
- */
- folio_get(folio);
- split_huge_pmd_address(migrate->vma, addr, true);
+ split_huge_pmd_address(migrate->vma, addr);
ret = folio_split_unmapped(folio, 0);
if (ret)
return ret;
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 2888ee638d872..ee33bbb421008 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -530,7 +530,7 @@ static inline long change_pmd_range(struct mmu_gather *tlb,
if (pmd_is_huge(_pmd)) {
if ((next - addr != HPAGE_PMD_SIZE) ||
pgtable_split_needed(vma, cp_flags)) {
- __split_huge_pmd(vma, pmd, addr, false);
+ __split_huge_pmd(vma, pmd, addr);
/*
* For file-backed, the pmd could have been
* cleared; make sure pmd populated if
diff --git a/mm/rmap.c b/mm/rmap.c
index 5332c52909be1..feb751e29b992 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2290,7 +2290,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
* restart so we can process the PTE-mapped THP.
*/
split_huge_pmd_locked(vma, pvmw.address,
- pvmw.pmd, false);
+ pvmw.pmd);
flags &= ~TTU_SPLIT_HUGE_PMD;
page_vma_mapped_walk_restart(&pvmw);
continue;
@@ -2515,13 +2515,12 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
if (flags & TTU_SPLIT_HUGE_PMD) {
/*
- * split_huge_pmd_locked() might leave the
+ * split_pmd_to_migration_entries() might leave the
* folio mapped through PTEs. Retry the walk
* so we can detect this scenario and properly
* abort the walk.
*/
- split_huge_pmd_locked(vma, pvmw.address,
- pvmw.pmd, true);
+ split_pmd_to_migration_entries(vma, pvmw.address, pvmw.pmd);
flags &= ~TTU_SPLIT_HUGE_PMD;
page_vma_mapped_walk_restart(&pvmw);
continue;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 11/29] mm: split PMD swap entries into PTE swap entries
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (9 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 10/29] mm: make PMD migration-entry splitting explicit Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 12/29] mm: handle PMD swap entries in fork path Usama Arif
` (18 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
Once a PMD can hold a swap entry, everything that splits a PMD - mprotect()
or munmap() over part of the range, MADV_FREE, a pagewalk with no PMD
handler - has to be able to split that entry too, or the callers that rely
on split_huge_pmd() to hand them a PTE table would find the PMD unchanged.
No reference counting is needed: a swap entry pins no folio, and swap_map
is already one per slot, so the PTEs simply take over what the PMD held.
The migration-only entry point cannot reach the new branch, because
page_vma_mapped_walk() never hands back a swap PMD for the folio being
migrated. Warn if that ever changes, and force the regular split anyway,
since the branch leaves folio and page uninitialised.
Test the pre-split old_pmd rather than re-reading *pmd in the trailing
folio_remove_rmap_pmd() gate, so every entry-type test in the function
interrogates the same snapshot. That part is cosmetic: pmdp_invalidate()
leaves the PMD present as far as software is concerned.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/huge_memory.c | 36 +++++++++++++++++++++++++++++++++++-
1 file changed, 35 insertions(+), 1 deletion(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 873887aed0bc2..0e347a545588c 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3304,6 +3304,21 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR,
vma, haddr, rmap_flags);
}
+ } else if (pmd_is_swap_entry(*pmd)) {
+ /*
+ * A PMD swap entry has no page, so it cannot be turned into
+ * PTE migration entries. page_vma_mapped_walk() never hands
+ * one back for the folio being migrated, so this should not
+ * happen; warn, but also force the regular split so that a
+ * broken invariant cannot make the code below dereference the
+ * uninitialised folio and page.
+ */
+ VM_WARN_ON_ONCE(use_migration_entries);
+ use_migration_entries = false;
+ old_pmd = *pmd;
+ soft_dirty = pmd_swp_soft_dirty(old_pmd);
+ uffd_wp = pmd_swp_uffd(old_pmd);
+ anon_exclusive = pmd_swp_exclusive(old_pmd);
} else {
/*
* Up to this point the pmd is present and huge and userland has
@@ -3440,6 +3455,25 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
VM_WARN_ON(!pte_none(ptep_get(pte + i)));
set_pte_at(mm, addr, pte + i, entry);
}
+ } else if (pmd_is_swap_entry(old_pmd)) {
+ const softleaf_t old_entry = softleaf_from_pmd(old_pmd);
+ pte_t pte_swp_entry;
+ swp_entry_t entry;
+
+ for (i = 0, addr = haddr; i < HPAGE_PMD_NR;
+ i++, addr += PAGE_SIZE) {
+ entry = swp_entry(swp_type(old_entry),
+ swp_offset(old_entry) + i);
+ pte_swp_entry = swp_entry_to_pte(entry);
+ if (soft_dirty)
+ pte_swp_entry = pte_swp_mksoft_dirty(pte_swp_entry);
+ if (uffd_wp)
+ pte_swp_entry = pte_swp_mkuffd(pte_swp_entry);
+ if (anon_exclusive)
+ pte_swp_entry = pte_swp_mkexclusive(pte_swp_entry);
+ VM_WARN_ON(!pte_none(ptep_get(pte + i)));
+ set_pte_at(mm, addr, pte + i, pte_swp_entry);
+ }
} else {
pte_t entry;
@@ -3467,7 +3501,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
}
pte_unmap(pte);
- if (!pmd_is_migration_entry(*pmd))
+ if (!pmd_is_migration_entry(old_pmd) && !pmd_is_swap_entry(old_pmd))
folio_remove_rmap_pmd(folio, page, vma);
if (use_migration_entries)
put_page(page);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 12/29] mm: handle PMD swap entries in fork path
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (10 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 11/29] mm: split PMD swap entries into PTE swap entries Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 13/29] mm: zswap: reject high-order swap cache allocations backed by zswap Usama Arif
` (17 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
copy_huge_pmd() only knows about migration and device-private PMDs, so a
PMD swap entry would fall through to the present-PMD path and fork() would
duplicate it without taking a reference on the slots it points at.
Copy it the way copy_nonpresent_pte() copies a PTE swap entry: duplicate
the swap references, clear the exclusive marker on the source, put the
destination mm on mmlist, and account the child's slots to MM_SWAPENTS.
Duplicating HPAGE_PMD_NR slots one at a time would be wasteful, so give
swap_dup_entry_direct() an nr argument and rename it accordingly. Unlike
the put side it hands nr straight to the per-cluster helper, so the range
has to sit inside one cluster - which it does, since SWAPFILE_CLUSTER ==
HPAGE_PMD_NR under CONFIG_THP_SWAP and a PMD-order folio's slots are only
ever allocated at a cluster head. Reject a crossing range with -EINVAL so a
future caller cannot walk off the end of the swap table.
The GFP_ATOMIC extend-table allocation inside the dup can fail;
copy_huge_pmd() then drops both PMD locks and retries once with
GFP_KERNEL. Bound it to one retry, because swap_retry_table_alloc() also
returns 0 when it decides the table is not needed. Normalise any remaining
failure to -ENOMEM: copy_pmd_range() treats every other error as "not a
huge PMD" and would then reach pmd_none_or_clear_bad(), clearing the source
PMD and leaking its swap slots.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
include/linux/swap.h | 4 +--
mm/huge_memory.c | 65 +++++++++++++++++++++++++++++++++++++++-----
mm/memory.c | 4 +--
mm/swap.h | 5 ++--
mm/swapfile.c | 58 +++++++++++++++++++++++++++++----------
5 files changed, 109 insertions(+), 27 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 43155e122b5c3..0c3006ece1ca5 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -414,7 +414,7 @@ sector_t swap_folio_sector(struct folio *folio);
* All entries must be allocated by folio_alloc_swap(). And they must have
* a swap count > 1. See comments of folio_*_swap helpers for more info.
*/
-int swap_dup_entry_direct(swp_entry_t entry);
+int swap_dup_entries_direct(swp_entry_t entry, int nr);
void swap_put_entries_direct(swp_entry_t entry, int nr);
/*
@@ -458,7 +458,7 @@ static inline void free_swap_cache(struct folio *folio)
{
}
-static inline int swap_dup_entry_direct(swp_entry_t ent)
+static inline int swap_dup_entries_direct(swp_entry_t ent, int nr)
{
return 0;
}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 0e347a545588c..6dfe8ef6dd371 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1894,7 +1894,7 @@ bool touch_pmd(struct vm_area_struct *vma, unsigned long addr,
return false;
}
-static void copy_huge_non_present_pmd(
+static int copy_huge_non_present_pmd(
struct mm_struct *dst_mm, struct mm_struct *src_mm,
pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
@@ -1940,14 +1940,40 @@ static void copy_huge_non_present_pmd(
*/
folio_try_dup_anon_rmap_pmd(src_folio, &src_folio->page,
dst_vma, src_vma);
+ } else if (softleaf_is_swap(entry)) {
+ int err;
+
+ /*
+ * PMD swap entry: duplicate swap references and clear
+ * exclusive on source, matching copy_nonpresent_pte().
+ *
+ * A PMD swap entry only exists under CONFIG_THP_SWAP, where
+ * SWAPFILE_CLUSTER == HPAGE_PMD_NR, and it is cluster aligned,
+ * so these HPAGE_PMD_NR slots are exactly one cluster - which
+ * is what swap_dup_entries_direct() requires.
+ */
+ err = swap_dup_entries_direct(entry, HPAGE_PMD_NR);
+ if (err < 0)
+ return err;
+
+ mm_prepare_for_swap_entries(dst_mm);
+
+ if (pmd_swp_exclusive(pmd)) {
+ pmd = pmd_swp_clear_exclusive(pmd);
+ set_pmd_at(src_mm, addr, src_pmd, pmd);
+ }
}
- add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+ if (softleaf_is_swap(entry))
+ add_mm_counter(dst_mm, MM_SWAPENTS, HPAGE_PMD_NR);
+ else
+ add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
mm_inc_nr_ptes(dst_mm);
pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
if (!userfaultfd_protected(dst_vma))
pmd = pmd_swp_clear_uffd(pmd);
set_pmd_at(dst_mm, addr, dst_pmd, pmd);
+ return 0;
}
int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
@@ -1957,6 +1983,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
spinlock_t *dst_ptl, *src_ptl;
struct page *src_page;
struct folio *src_folio;
+ bool retried = false;
pmd_t pmd;
pgtable_t pgtable = NULL;
int ret = -ENOMEM;
@@ -1988,6 +2015,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
if (unlikely(!pgtable))
goto out;
+retry:
dst_ptl = pmd_lock(dst_mm, dst_pmd);
src_ptl = pmd_lockptr(src_mm, src_pmd);
spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
@@ -1995,11 +2023,34 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
ret = -EAGAIN;
pmd = *src_pmd;
- if (unlikely(thp_migration_supported() &&
- pmd_is_valid_softleaf(pmd))) {
- copy_huge_non_present_pmd(dst_mm, src_mm, dst_pmd, src_pmd, addr,
- dst_vma, src_vma, pmd, pgtable);
- ret = 0;
+ if (unlikely(pmd_is_valid_softleaf(pmd))) {
+ ret = copy_huge_non_present_pmd(dst_mm, src_mm, dst_pmd, src_pmd,
+ addr, dst_vma, src_vma, pmd,
+ pgtable);
+ if (ret) {
+ spin_unlock(src_ptl);
+ spin_unlock(dst_ptl);
+ /*
+ * For PMD swap entries -ENOMEM means the per-cluster
+ * swap-extend table couldn't be GFP_ATOMIC-allocated.
+ * Try the GFP_KERNEL fallback once before giving up.
+ * swap_retry_table_alloc() also returns 0 when it
+ * decides the table is not needed after all, so bound
+ * this to a single retry rather than looping on it.
+ */
+ if (ret == -ENOMEM && !retried) {
+ softleaf_t entry = softleaf_from_pmd(pmd);
+
+ retried = true;
+ if (softleaf_is_swap(entry) &&
+ !swap_retry_table_alloc(entry, HPAGE_PMD_NR,
+ GFP_KERNEL))
+ goto retry;
+ }
+ pte_free(dst_mm, pgtable);
+ ret = -ENOMEM;
+ goto out;
+ }
goto out_unlock;
}
diff --git a/mm/memory.c b/mm/memory.c
index 477d7e359b447..84e1e1c22bffa 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -979,7 +979,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
struct page *page;
if (likely(softleaf_is_swap(entry))) {
- if (swap_dup_entry_direct(entry) < 0)
+ if (swap_dup_entries_direct(entry, 1) < 0)
return -EIO;
mm_prepare_for_swap_entries(dst_mm);
@@ -1394,7 +1394,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
if (ret == -EIO) {
VM_WARN_ON_ONCE(!entry.val);
- if (swap_retry_table_alloc(entry, GFP_KERNEL) < 0) {
+ if (swap_retry_table_alloc(entry, 1, GFP_KERNEL) < 0) {
ret = -ENOMEM;
goto out;
}
diff --git a/mm/swap.h b/mm/swap.h
index b3b54c28929a1..2321c9a2c2c58 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -222,7 +222,7 @@ static inline void swap_cluster_unlock_irq(struct swap_cluster_info *ci)
spin_unlock_irq(&ci->lock);
}
-extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp);
+int swap_retry_table_alloc(swp_entry_t entry, unsigned int nr, gfp_t gfp);
/*
* Below are the core routines for doing swap for a folio.
@@ -428,7 +428,8 @@ static inline int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
return 0;
}
-static inline int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
+static inline int swap_retry_table_alloc(swp_entry_t entry, unsigned int nr,
+ gfp_t gfp)
{
return -EINVAL;
}
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 280dd906eb187..27ae3964a158e 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1468,11 +1468,16 @@ static bool swap_sync_discard(void)
static int swap_extend_table_alloc(struct swap_info_struct *si,
struct swap_cluster_info *ci,
- unsigned int ci_off, gfp_t gfp)
+ unsigned int ci_off, unsigned int nr,
+ gfp_t gfp)
{
int count;
+ unsigned int i;
void *table;
+ /* The range must not run past the end of @ci's swap table. */
+ VM_WARN_ON_ONCE(ci_off + nr > SWAPFILE_CLUSTER);
+
table = kzalloc(sizeof(ci->extend_table[0]) * SWAPFILE_CLUSTER, gfp);
if (!table)
return -ENOMEM;
@@ -1486,15 +1491,21 @@ static int swap_extend_table_alloc(struct swap_info_struct *si,
*/
if (!cluster_table_is_alloced(ci))
goto out_free;
- count = swp_tb_get_count(__swap_table_get(ci, ci_off));
- if (count < (SWP_TB_COUNT_MAX - 1))
- goto out_free;
if (ci->extend_table)
goto out_free;
-
- ci->extend_table = table;
- spin_unlock(&ci->lock);
- return 0;
+ /*
+ * The caller may not know which slot in [ci_off, ci_off + nr) hit
+ * SWP_TB_COUNT_MAX - 1. Confirm at least one slot in the range still
+ * needs the extend table before committing the allocation.
+ */
+ for (i = 0; i < nr; i++) {
+ count = swp_tb_get_count(__swap_table_get(ci, ci_off + i));
+ if (count >= (SWP_TB_COUNT_MAX - 1)) {
+ ci->extend_table = table;
+ spin_unlock(&ci->lock);
+ return 0;
+ }
+ }
out_free:
spin_unlock(&ci->lock);
@@ -1502,7 +1513,7 @@ static int swap_extend_table_alloc(struct swap_info_struct *si,
return 0;
}
-int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
+int swap_retry_table_alloc(swp_entry_t entry, unsigned int nr, gfp_t gfp)
{
int ret;
struct swap_info_struct *si;
@@ -1514,7 +1525,8 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
return 0;
ci = __swap_offset_to_cluster(si, offset);
- ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
+ ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), nr,
+ gfp);
put_swap_device(si);
return ret;
@@ -1690,6 +1702,9 @@ static int __swap_cluster_dup_entry(struct swap_cluster_info *ci,
* @offset: start offset of slots.
* @nr: number of slots.
*
+ * The range [offset, offset + nr) must not cross a cluster boundary; the
+ * caller is responsible for splitting a range that can.
+ *
* Context: The specified slots must be pinned by existing swap count or swap
* cache reference, so they won't be released until this helper returns.
* Return: 0 on success. -ENOMEM if the swap count maxed out (SWP_TB_COUNT_MAX)
@@ -1704,6 +1719,7 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
ci_start = offset % SWAPFILE_CLUSTER;
ci_end = ci_start + nr;
+ VM_WARN_ON_ONCE(ci_end > SWAPFILE_CLUSTER);
ci_off = ci_start;
ci = swap_cluster_lock(si, offset);
restart:
@@ -1712,7 +1728,8 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
if (unlikely(err)) {
if (err == -ENOMEM) {
spin_unlock(&ci->lock);
- err = swap_extend_table_alloc(si, ci, ci_off, GFP_ATOMIC);
+ err = swap_extend_table_alloc(si, ci, ci_off, 1,
+ GFP_ATOMIC);
spin_lock(&ci->lock);
if (!err)
goto restart;
@@ -1723,6 +1740,7 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
swap_cluster_unlock(ci);
return 0;
failed:
+ /* The caller's page-table or swap-cache reference pins every slot. */
while (ci_off-- > ci_start)
__swap_cluster_put_entry(ci, ci_off);
swap_cluster_unlock(ci);
@@ -3966,8 +3984,9 @@ void si_swapinfo(struct sysinfo *val)
}
/*
- * swap_dup_entry_direct() - Increase reference count of a swap entry by one.
+ * swap_dup_entries_direct() - Increase reference count of swap entries by one.
* @entry: first swap entry from which we want to increase the refcount.
+ * @nr: number of contiguous swap entries to duplicate.
*
* Returns 0 for success, or -ENOMEM if the extend table is required
* but could not be atomically allocated. Returns -EINVAL if the swap
@@ -3978,8 +3997,16 @@ void si_swapinfo(struct sysinfo *val)
* owner. e.g., locking the PTL of a PTE containing the entry being increased.
* Also the swap entry must have a count >= 1. Otherwise folio_dup_swap should
* be used.
+ *
+ * Unlike swap_put_entries_direct(), the whole range [entry, entry + nr) must
+ * lie within one swap cluster; a range that crosses a cluster boundary is
+ * rejected with -EINVAL. The only caller passing nr > 1 is the PMD swap entry
+ * fork path: a PMD swap entry can only exist with CONFIG_THP_SWAP, where
+ * SWAPFILE_CLUSTER == HPAGE_PMD_NR, and a PMD-order folio's slots are only ever
+ * allocated at a cluster head (see alloc_swap_scan_cluster()), so such a range
+ * is exactly one cluster.
*/
-int swap_dup_entry_direct(swp_entry_t entry)
+int swap_dup_entries_direct(swp_entry_t entry, int nr)
{
struct swap_info_struct *si;
@@ -3989,6 +4016,9 @@ int swap_dup_entry_direct(swp_entry_t entry)
return -EINVAL;
}
+ if (WARN_ON_ONCE(swp_cluster_offset(entry) + nr > SWAPFILE_CLUSTER))
+ return -EINVAL;
+
/*
* The caller must be increasing the swap count from a direct
* reference of the swap slot (e.g. a swap entry in page table).
@@ -3996,7 +4026,7 @@ int swap_dup_entry_direct(swp_entry_t entry)
*/
VM_WARN_ON_ONCE(!swap_entry_swapped(si, entry));
- return swap_dup_entries_cluster(si, swp_offset(entry), 1);
+ return swap_dup_entries_cluster(si, swp_offset(entry), nr);
}
#if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 13/29] mm: zswap: reject high-order swap cache allocations backed by zswap
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (11 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 12/29] mm: handle PMD swap entries in fork path Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 14/29] mm: swap in PMD swap entries as whole THPs during swapoff Usama Arif
` (16 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
zswap stores swapped-out pages one at a time, so a range of swap slots can
be split between zswap entries and the backing device. A high-order
swap-cache folio over such a range cannot be read back in one go: the disk
read would return stale or zero data for every slot zswap owns.
Reject that in __swap_cache_alloc(). Do the check after the folio is in the
swap cache: from that point zswap can neither start storing nor start
writing back any slot in the range, so the answer is stable. Callers
already fall back to order-0, which zswap_load() handles per page.
Export zswap_is_present() for the PMD-order swap-in users added later.
Co-developed-by: Alexandre Ghiti <alex@ghiti.fr>
Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
include/linux/zswap.h | 6 ++++++
mm/swap_state.c | 39 +++++++++++++++++++++++++--------------
mm/zswap.c | 12 +++++++++---
3 files changed, 40 insertions(+), 17 deletions(-)
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index df6cafbe95dc0..94746fb71bb65 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -35,6 +35,7 @@ void zswap_lruvec_state_init(struct lruvec *lruvec);
void zswap_folio_swapin(struct folio *folio);
bool zswap_is_enabled(void);
bool zswap_never_enabled(void);
+bool zswap_is_present(swp_entry_t entry, unsigned int nr);
#else
struct zswap_lruvec_state {};
@@ -73,6 +74,11 @@ static inline bool zswap_never_enabled(void)
return true;
}
+static inline bool zswap_is_present(swp_entry_t entry, unsigned int nr)
+{
+ return false;
+}
+
#endif
#endif /* _LINUX_ZSWAP_H */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 625c185a1ca4d..251b48b2c60e1 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -12,6 +12,7 @@
#include <linux/kernel_stat.h>
#include <linux/mempolicy.h>
#include <linux/swap.h>
+#include <linux/zswap.h>
#include <linux/leafops.h>
#include <linux/init.h>
#include <linux/pagemap.h>
@@ -466,26 +467,27 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
__swap_cache_do_add_folio(ci, folio, entry);
spin_unlock(&ci->lock);
+ /*
+ * Now that the folio is in the swap cache, zswap can no longer start
+ * storing or writing back any slot in the range, so this is a stable
+ * answer. Reject a high-order allocation over a range that already
+ * has per-page zswap entries.
+ */
+ if (order && zswap_is_present(entry, nr_pages)) {
+ err = -EBUSY;
+ goto delete_folio;
+ }
+
if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
vmf ? vmf->vma->vm_mm : NULL, gfp)) {
- spin_lock(&ci->lock);
- __swap_cache_do_del_folio(ci, folio, entry, shadow);
- spin_unlock(&ci->lock);
- folio_unlock(folio);
- /* nr_pages refs from swap cache, 1 from allocation */
- folio_put_refs(folio, nr_pages + 1);
+ err = -ENOMEM;
count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
- return ERR_PTR(-ENOMEM);
+ goto delete_folio;
}
if (order > 1 && folio_memcg_alloc_deferred(folio)) {
- spin_lock(&ci->lock);
- __swap_cache_do_del_folio(ci, folio, entry, shadow);
- spin_unlock(&ci->lock);
- folio_unlock(folio);
- /* nr_pages refs from swap cache, 1 from allocation */
- folio_put_refs(folio, nr_pages + 1);
- return ERR_PTR(-ENOMEM);
+ err = -ENOMEM;
+ goto delete_folio;
}
/* memsw uncharges swap when folio is added to swap cache */
@@ -499,6 +501,15 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
/* Caller will initiate read into locked new_folio */
folio_add_lru(folio);
return folio;
+
+delete_folio:
+ spin_lock(&ci->lock);
+ __swap_cache_do_del_folio(ci, folio, entry, shadow);
+ spin_unlock(&ci->lock);
+ folio_unlock(folio);
+ /* nr_pages refs from swap cache, 1 from allocation */
+ folio_put_refs(folio, nr_pages + 1);
+ return ERR_PTR(err);
}
/**
diff --git a/mm/zswap.c b/mm/zswap.c
index 507f2d19fd2a4..f5b5827d30084 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1587,11 +1587,14 @@ bool zswap_store(struct folio *folio)
* @entry: base swap entry of the range
* @nr: number of contiguous slots to check
*
- * Context: The caller must keep the range pinned, otherwise the answer can
- * change under it.
+ * Context: Unless the caller keeps the range pinned, the answer is only a
+ * hint: zswap can store or write back a slot right after this returns. A
+ * caller that needs a stable answer must pin the range first, as
+ * __swap_cache_alloc() does by inserting the folio into the swap cache before
+ * asking.
* Return: true if at least one slot in the range is in zswap.
*/
-static bool zswap_is_present(swp_entry_t entry, unsigned int nr)
+bool zswap_is_present(swp_entry_t entry, unsigned int nr)
{
pgoff_t offset = swp_offset(entry);
struct xarray *tree = swap_zswap_tree(entry);
@@ -1605,6 +1608,9 @@ static bool zswap_is_present(swp_entry_t entry, unsigned int nr)
*/
BUILD_BUG_ON(SWAPFILE_CLUSTER > ZSWAP_ADDRESS_SPACE_PAGES);
+ if (zswap_never_enabled())
+ return false;
+
return xa_find(tree, &index, offset + nr - 1, XA_PRESENT);
}
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 14/29] mm: swap in PMD swap entries as whole THPs during swapoff
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (12 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 13/29] mm: zswap: reject high-order swap cache allocations backed by zswap Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 15/29] fs/proc: account PMD swap entries in smaps Usama Arif
` (15 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
swapoff walks every mm and faults every slot of the device back in.
unuse_pte_range() only understands PTEs, so a PMD swap entry would never be
found and swapoff would never complete.
A PMD swap entry is a compact encoding for HPAGE_PMD_NR slots, not a
promise that the swap cache holds one folio for them. Add
swap_pmd_cache_lookup() to classify the covered range as empty, one
PMD-sized folio, or already split, and unuse_pmd() to map the first two
cases back in as one THP, preserving soft-dirty, exclusive and UFFD state.
Everything else falls back to PTEs: a split cache, per-page zswap state, a
failed PMD-order allocation or read, or a poisoned subpage. Check
PageHWPoison on every subpage rather than the folio-level flag, which
memory_failure() only sets after taking the folio lock.
All the fallback reasons are observed without the PMD lock and possibly
after sleeping, so they share one exit that re-checks the PMD is still the
entry we were called for before splitting it. That exit also drops a folio
that is not uptodate, or that has never been mapped, from the swap cache:
the PTE path cannot re-read the first, and would add a single-page rmap to
the second.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/internal.h | 16 ++++
mm/swap.h | 17 +++++
mm/swap_state.c | 44 +++++++++++
mm/swapfile.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 266 insertions(+)
diff --git a/mm/internal.h b/mm/internal.h
index 05179c4b2090e..ec7f007bc2c0d 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -24,6 +24,22 @@
struct folio_batch;
+/*
+ * Unlike folio_contain_hwpoisoned_page(), this does not rely on the folio-level
+ * PG_has_hwpoisoned, which memory_failure() only sets after taking the folio
+ * lock and so can lag a tail-page poison.
+ */
+static inline bool folio_has_hwpoisoned_subpage(const struct folio *folio)
+{
+ long nr = folio_nr_pages(folio);
+ long i;
+
+ for (i = 0; i < nr; i++)
+ if (PageHWPoison(folio_page(folio, i)))
+ return true;
+ return false;
+}
+
/* mm/workingset.c */
bool workingset_test_recent(void *shadow, bool file, bool *workingset,
bool flush);
diff --git a/mm/swap.h b/mm/swap.h
index 2321c9a2c2c58..0b54026f32d6f 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -311,6 +311,23 @@ static inline bool folio_matches_swap_entry(const struct folio *folio,
bool swap_cache_has_folio(swp_entry_t entry);
struct folio *swap_cache_get_folio(swp_entry_t entry);
void *swap_cache_get_shadow(swp_entry_t entry);
+enum swap_pmd_cache {
+ SWAP_PMD_CACHE_EMPTY,
+ SWAP_PMD_CACHE_HUGE,
+ SWAP_PMD_CACHE_SPLIT,
+};
+
+#ifdef CONFIG_THP_SWAP
+enum swap_pmd_cache swap_pmd_cache_lookup(swp_entry_t entry,
+ struct folio **foliop);
+#else
+static inline enum swap_pmd_cache swap_pmd_cache_lookup(swp_entry_t entry,
+ struct folio **foliop)
+{
+ *foliop = NULL;
+ return SWAP_PMD_CACHE_EMPTY;
+}
+#endif
void swap_cache_del_folio(struct folio *folio);
struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
unsigned long orders, struct vm_fault *vmf,
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 251b48b2c60e1..15f93b9a3eb2a 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -125,6 +125,50 @@ bool swap_cache_has_folio(swp_entry_t entry)
return swp_tb_is_folio(swp_tb);
}
+#ifdef CONFIG_THP_SWAP
+/**
+ * swap_pmd_cache_lookup - classify the swap cache behind a PMD swap entry
+ * @entry: first swap slot encoded by the PMD swap entry
+ * @foliop: returned PMD-sized folio, with a reference, if present
+ *
+ * A PMD swap entry is a compact page-table encoding for HPAGE_PMD_NR
+ * consecutive swap slots. The swap cache behind those slots can be empty,
+ * one PMD-sized folio, or per-slot folios after the original folio was split.
+ *
+ * Context: Caller must keep @entry valid using the usual swap cache rules.
+ * Return: SWAP_PMD_CACHE_EMPTY if no slot in the PMD range has a cached folio,
+ * SWAP_PMD_CACHE_HUGE if one PMD-sized folio covers the range, or
+ * SWAP_PMD_CACHE_SPLIT if the range needs per-page handling.
+ */
+enum swap_pmd_cache swap_pmd_cache_lookup(swp_entry_t entry,
+ struct folio **foliop)
+{
+ unsigned int type = swp_type(entry);
+ pgoff_t offset = swp_offset(entry);
+ struct folio *folio;
+ int i;
+
+ *foliop = NULL;
+
+ folio = swap_cache_get_folio(entry);
+ if (folio) {
+ if (folio_nr_pages(folio) == HPAGE_PMD_NR) {
+ *foliop = folio;
+ return SWAP_PMD_CACHE_HUGE;
+ }
+ folio_put(folio);
+ return SWAP_PMD_CACHE_SPLIT;
+ }
+
+ for (i = 1; i < HPAGE_PMD_NR; i++) {
+ if (swap_cache_has_folio(swp_entry(type, offset + i)))
+ return SWAP_PMD_CACHE_SPLIT;
+ }
+
+ return SWAP_PMD_CACHE_EMPTY;
+}
+#endif
+
/**
* swap_cache_get_shadow - Looks up a shadow in the swap cache.
* @entry: swap entry used for the lookup.
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 27ae3964a158e..fe3cadfc3a017 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -42,6 +42,7 @@
#include <linux/suspend.h>
#include <linux/zswap.h>
#include <linux/plist.h>
+#include <linux/huge_mm.h>
#include <asm/tlbflush.h>
#include <linux/leafops.h>
@@ -2696,6 +2697,182 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
return 0;
}
+#ifdef CONFIG_THP_SWAP
+/*
+ * unuse_pmd - Map a locked folio at PMD granularity during swapoff.
+ *
+ * The caller provides a locked, swapped-in folio. Returns 0 on success
+ * (PMD was mapped). Returns -EAGAIN if the swap cache folio no longer
+ * matches the entry or the PMD changed under the lock (try_to_unuse will
+ * rescan). Returns -EIO if the folio is not uptodate or contains a poisoned
+ * subpage; in that case the PMD is split so unuse_pte_range() can handle
+ * individual pages.
+ */
+static int unuse_pmd(struct vm_area_struct *vma, pmd_t *pmd,
+ unsigned long addr, softleaf_t entry,
+ struct folio *folio)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ struct page *page;
+ pmd_t new_pmd, old_pmd;
+ spinlock_t *ptl;
+ rmap_t rmap_flags = RMAP_NONE;
+ bool exclusive;
+
+ if (unlikely(!folio_matches_swap_entry(folio, entry)))
+ return -EAGAIN;
+
+ /* Let PTE fallback reread each slot, or isolate the poisoned ones. */
+ if (unlikely(!folio_test_uptodate(folio)))
+ return -EIO;
+ if (unlikely(folio_has_hwpoisoned_subpage(folio)))
+ return -EIO;
+
+ page = folio_page(folio, 0);
+
+ ptl = pmd_lock(mm, pmd);
+ old_pmd = pmdp_get(pmd);
+
+ if (!pmd_is_swap_entry(old_pmd) ||
+ softleaf_from_pmd(old_pmd).val != entry.val) {
+ spin_unlock(ptl);
+ return -EAGAIN;
+ }
+
+ exclusive = pmd_swp_exclusive(old_pmd);
+
+ /*
+ * Some architectures may have to restore extra metadata to the folio
+ * when reading from swap. This metadata may be indexed by swap entry
+ * so this must be called before folio_put_swap().
+ */
+ arch_swap_restore(folio_swap(entry, folio), folio);
+
+ add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR);
+ add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR);
+
+ new_pmd = folio_mk_pmd(folio, vma->vm_page_prot);
+ new_pmd = pmd_mkold(new_pmd);
+ if (pmd_swp_soft_dirty(old_pmd))
+ new_pmd = pmd_mksoft_dirty(new_pmd);
+ if (pmd_swp_uffd(old_pmd))
+ new_pmd = pmd_mkuffd(new_pmd);
+ if (pmd_swp_uffd(old_pmd) && userfaultfd_rwp(vma))
+ new_pmd = pmd_modify(new_pmd, PAGE_NONE);
+
+ if (exclusive)
+ rmap_flags |= RMAP_EXCLUSIVE;
+
+ folio_get(folio);
+ if (!folio_test_anon(folio))
+ folio_add_new_anon_rmap(folio, vma, addr, rmap_flags);
+ else
+ folio_add_anon_rmap_pmd(folio, page, vma, addr, rmap_flags);
+
+ set_pmd_at(mm, addr, pmd, new_pmd);
+ folio_put_swap(folio, NULL);
+
+ spin_unlock(ptl);
+
+ folio_free_swap(folio);
+ return 0;
+}
+
+/*
+ * Try to swap in a PMD swap entry as a whole THP. Returns 0 on success.
+ * If the swap cache no longer has one PMD-sized folio, zswap may require
+ * per-page loading, or a PMD-order allocation/read fails, split the PMD so
+ * the caller can fall back to unuse_pte_range(). Otherwise propagates the
+ * error from unuse_pmd().
+ */
+static int unuse_pmd_entry(struct vm_area_struct *vma, pmd_t *pmd,
+ unsigned long addr, softleaf_t entry)
+{
+ enum swap_pmd_cache cache_state;
+ struct folio *folio;
+ pmd_t pmdval;
+ int ret;
+
+ cache_state = swap_pmd_cache_lookup(entry, &folio);
+ if (cache_state == SWAP_PMD_CACHE_SPLIT) {
+ ret = -EAGAIN;
+ goto split_fallback;
+ }
+ if (!folio) {
+ struct vm_fault vmf = {
+ .vma = vma,
+ .address = addr,
+ .real_address = addr,
+ .pmd = pmd,
+ };
+
+ if (zswap_is_present(entry, HPAGE_PMD_NR)) {
+ ret = -EAGAIN;
+ goto split_fallback;
+ }
+
+ folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
+ BIT(HPAGE_PMD_ORDER), &vmf, NULL, 0);
+ if (IS_ERR_OR_NULL(folio)) {
+ ret = folio ? PTR_ERR(folio) : -ENOMEM;
+ goto split_fallback;
+ }
+ }
+
+ folio_lock(folio);
+ folio_wait_writeback(folio);
+ /*
+ * If the cached folio is no longer PMD-sized (e.g. split in the
+ * swap cache by deferred_split_scan() or memory_failure() while
+ * the PMD swap entry was installed), the PMD swap entry no longer
+ * maps a single contiguous folio. Split the PMD swap entry so
+ * unuse_pte_range() can swap the per-slot folios in individually.
+ */
+ if (folio_nr_pages(folio) != HPAGE_PMD_NR) {
+ folio_unlock(folio);
+ folio_put(folio);
+ ret = -EAGAIN;
+ goto split_fallback;
+ }
+ ret = unuse_pmd(vma, pmd, addr, entry, folio);
+ /*
+ * PTE fallback cannot add a single-page rmap to a newly allocated,
+ * PMD-sized !anon folio, and it cannot do anything useful with a folio
+ * that failed to read. Remove either from the swap cache so each slot
+ * is read back into an order-0 folio. An uptodate anon swap-cache folio
+ * can be mapped one PTE at a time and must stay cached so poisoned
+ * subpages remain visible.
+ */
+ if (ret && folio_matches_swap_entry(folio, entry) &&
+ (!folio_test_uptodate(folio) || !folio_test_anon(folio)))
+ swap_cache_del_folio(folio);
+ folio_unlock(folio);
+ folio_put(folio);
+ if (ret == -EIO)
+ goto split_fallback;
+ return ret;
+
+split_fallback:
+ /*
+ * Every reason we get here was observed without the PMD lock and
+ * possibly after sleeping, so re-check that the PMD is still the entry
+ * we were called for. A racing fault may already have swapped the range
+ * back in as a THP, and splitting that would demote it for nothing.
+ */
+ pmdval = pmdp_get(pmd);
+ if (pmd_is_swap_entry(pmdval) &&
+ softleaf_from_pmd(pmdval).val == entry.val)
+ __split_huge_pmd(vma, pmd, addr);
+ return ret;
+}
+#else /* !CONFIG_THP_SWAP */
+static inline int unuse_pmd_entry(struct vm_area_struct *vma, pmd_t *pmd,
+ unsigned long addr, softleaf_t entry)
+{
+ return -EAGAIN;
+}
+#endif /* CONFIG_THP_SWAP */
+
static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud,
unsigned long addr, unsigned long end,
unsigned int type)
@@ -2706,8 +2883,20 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud,
pmd = pmd_offset(pud, addr);
do {
+ pmd_t pmdval;
+
cond_resched();
next = pmd_addr_end(addr, end);
+
+ pmdval = pmdp_get(pmd);
+ if (IS_ENABLED(CONFIG_THP_SWAP) && pmd_is_swap_entry(pmdval)) {
+ softleaf_t entry = softleaf_from_pmd(pmdval);
+
+ if (swp_type(entry) == type &&
+ !unuse_pmd_entry(vma, pmd, addr, entry))
+ continue;
+ }
+
ret = unuse_pte_range(vma, pmd, addr, next, type);
if (ret)
return ret;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 15/29] fs/proc: account PMD swap entries in smaps
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (13 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 14/29] mm: swap in PMD swap entries as whole THPs during swapoff Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 16/29] mm: handle soft-dirty and uffd-wp on PMD swap entries Usama Arif
` (14 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
smaps_pmd_entry() derives a page from a non-present PMD and gives up if it
cannot. A PMD swap entry carries no PFN, so a swapped-out THP is accounted
nowhere: its bytes show up in neither Swap nor SwapPss, and VmSwap silently
loses them.
The slots of a PMD swap entry can have different swap reference counts, so
SwapPss has to be computed one slot at a time rather than divided once.
Factor that loop out of smaps_pte_entry() and call it from both levels,
then return early: a swapped-out THP is not resident and must not be added
to AnonHugePages or Rss.
pagemap needs no equivalent change - it already emits PM_SWAP and
type | offset+idx for a PFN-less softleaf entry - but its
thp_migration_supported() gate answers a different question than the decode
requires, so use pmd_is_valid_softleaf() there instead.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
fs/proc/task_mmu.c | 42 ++++++++++++++++++++++++++++--------------
1 file changed, 28 insertions(+), 14 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 24425e2308951..aa3f4d54a8580 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -957,6 +957,27 @@ static void smaps_pte_hole_lookup(unsigned long addr, struct mm_walk *walk)
#endif
}
+static void smaps_account_swap(struct mem_size_stats *mss,
+ softleaf_t entry, unsigned long size)
+{
+ unsigned long nr_pages = size >> PAGE_SHIFT;
+
+ mss->swap += size;
+ do {
+ int mapcount = swp_swapcount(entry);
+
+ if (mapcount >= 2) {
+ u64 pss_delta = (u64)PAGE_SIZE << PSS_SHIFT;
+
+ do_div(pss_delta, mapcount);
+ mss->swap_pss += pss_delta;
+ } else {
+ mss->swap_pss += (u64)PAGE_SIZE << PSS_SHIFT;
+ }
+ entry.val++;
+ } while (--nr_pages);
+}
+
static void smaps_pte_entry(pte_t *pte, unsigned long addr,
struct mm_walk *walk)
{
@@ -978,18 +999,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr,
const softleaf_t entry = softleaf_from_pte(ptent);
if (softleaf_is_swap(entry)) {
- int mapcount;
-
- mss->swap += PAGE_SIZE;
- mapcount = swp_swapcount(entry);
- if (mapcount >= 2) {
- u64 pss_delta = (u64)PAGE_SIZE << PSS_SHIFT;
-
- do_div(pss_delta, mapcount);
- mss->swap_pss += pss_delta;
- } else {
- mss->swap_pss += (u64)PAGE_SIZE << PSS_SHIFT;
- }
+ smaps_account_swap(mss, entry, PAGE_SIZE);
} else if (softleaf_has_pfn(entry)) {
if (softleaf_is_device_private(entry))
present = true;
@@ -1019,9 +1029,13 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
if (pmd_present(*pmd)) {
page = vm_normal_page_pmd(vma, addr, *pmd);
present = true;
- } else if (unlikely(thp_migration_supported())) {
+ } else {
const softleaf_t entry = softleaf_from_pmd(*pmd);
+ if (softleaf_is_swap(entry)) {
+ smaps_account_swap(mss, entry, HPAGE_PMD_SIZE);
+ return;
+ }
if (softleaf_has_pfn(entry))
page = softleaf_to_page(entry);
}
@@ -2000,7 +2014,7 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,
flags |= PM_UFFD_WP;
if (pm->show_pfn)
frame = pmd_pfn(pmd) + idx;
- } else if (thp_migration_supported()) {
+ } else if (pmd_is_valid_softleaf(pmd)) {
const softleaf_t entry = softleaf_from_pmd(pmd);
unsigned long offset;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 16/29] mm: handle soft-dirty and uffd-wp on PMD swap entries
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (14 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 15/29] fs/proc: account PMD swap entries in smaps Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 17/29] mm/hmm: fault PMD swap entries on demand Usama Arif
` (13 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
Soft-dirty and uffd-wp live in the PMD, not in the swap entry, so every
path that manipulates them for a PMD migration entry has to do the same for
a PMD swap entry. Otherwise clearing soft-dirty skips a swapped-out THP,
UFFDIO_WRITEPROTECT does not arm it, and mremap does not mark the
destination.
change_huge_pmd() needs nothing beyond dropping its
thp_migration_supported() gate: a PMD swap entry encodes no write
permission, so it falls into the default arm of
change_non_present_huge_pmd() and picks up only the uffd-wp update.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
fs/proc/task_mmu.c | 4 ++--
mm/huge_memory.c | 11 +++++++++--
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index aa3f4d54a8580..ad0d4c7f0988b 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1654,7 +1654,7 @@ static void clear_soft_dirty_pmd(struct vm_area_struct *vma,
pmd = pmd_clear_soft_dirty(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
- } else if (pmd_is_migration_entry(pmd)) {
+ } else if (pmd_is_migration_entry(pmd) || pmd_is_swap_entry(pmd)) {
pmd = pmd_swp_clear_soft_dirty(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
}
@@ -2480,7 +2480,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
old = pmdp_invalidate_ad(vma, addr, pmdp);
pmd = pmd_mkuffd(old);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
- } else if (pmd_is_migration_entry(pmd)) {
+ } else if (pmd_is_migration_entry(pmd) || pmd_is_swap_entry(pmd)) {
pmd = pmd_swp_mkuffd(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 6dfe8ef6dd371..d3d0fec9a5ff3 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2660,7 +2660,8 @@ static inline int pmd_move_must_withdraw(spinlock_t *new_pmd_ptl,
static pmd_t move_soft_dirty_pmd(pmd_t pmd)
{
if (pgtable_supports_soft_dirty()) {
- if (unlikely(pmd_is_migration_entry(pmd)))
+ if (unlikely(pmd_is_migration_entry(pmd) ||
+ pmd_is_swap_entry(pmd)))
pmd = pmd_swp_mksoft_dirty(pmd);
else if (pmd_present(pmd))
pmd = pmd_mksoft_dirty(pmd);
@@ -2752,6 +2753,12 @@ static void change_non_present_huge_pmd(struct mm_struct *mm,
pmd_t newpmd;
VM_WARN_ON(!pmd_is_valid_softleaf(*pmd));
+
+ /*
+ * Note that a PMD swap entry falls into the default branch below: it
+ * does not encode write permission in the entry type, so only the
+ * uffd_wp flag update at the end applies to it.
+ */
if (softleaf_is_migration_write(entry)) {
const struct folio *folio = softleaf_to_folio(entry);
@@ -2812,7 +2819,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
if (!ptl)
return 0;
- if (thp_migration_supported() && pmd_is_valid_softleaf(*pmd)) {
+ if (pmd_is_valid_softleaf(*pmd)) {
change_non_present_huge_pmd(mm, addr, pmd, uffd_prot,
uffd_prot_resolve);
goto unlock;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 17/29] mm/hmm: fault PMD swap entries on demand
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (15 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 16/29] mm: handle soft-dirty and uffd-wp on PMD swap entries Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 18/29] mm: free PMD swap entries in zap_huge_pmd() Usama Arif
` (12 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
hmm_vma_handle_absent_pmd() reports anything non-present that is not
device-private as HMM_PFN_ERROR, so a swapped-out THP would look like a
permanent failure rather than memory that needs paging in.
Treat it like a PTE swap entry: record the fault when one was asked for,
and otherwise report the range as non-resident with no flags, which is what
hmm_vma_handle_pte() already does in snapshot mode.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/hmm.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 2f1e98c6b6440..95575ac378888 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -377,12 +377,21 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
required_fault = hmm_range_need_fault(hmm_vma_walk, hmm_pfns,
npages, 0);
if (required_fault) {
- if (softleaf_is_device_private(entry))
+ if (softleaf_is_device_private(entry) ||
+ softleaf_is_swap(entry))
return hmm_record_fault(addr, end, required_fault, walk);
else
return -EFAULT;
}
+ /*
+ * A swapped-out THP is not resident. Report it as not-valid,
+ * matching what hmm_vma_handle_pte() does for a PTE swap entry when
+ * no fault was requested.
+ */
+ if (softleaf_is_swap(entry))
+ return hmm_pfns_fill(start, end, range, 0);
+
return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
}
#else
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 18/29] mm: free PMD swap entries in zap_huge_pmd()
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (16 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 17/29] mm/hmm: fault PMD swap entries on demand Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 19/29] mm/madvise: free PMD swap entries with MADV_FREE Usama Arif
` (11 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
zap_huge_pmd() assumes any non-present PMD it clears references a folio, so
a PMD swap entry would reach normal_or_softleaf_folio_pmd() and warn.
Worse, nothing would release the slots it points at: unmapping a
swapped-out THP would leak the whole PMD's worth of swap.
Handle it before the folio lookup, as zap_pte_range() already does for a
PTE swap entry.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/huge_memory.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index d3d0fec9a5ff3..689a1f4286471 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2628,6 +2628,16 @@ bool zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
arch_check_zapped_pmd(vma, orig_pmd);
tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
+ if (pmd_is_swap_entry(orig_pmd)) {
+ softleaf_t entry = softleaf_from_pmd(orig_pmd);
+
+ zap_deposited_table(mm, pmd);
+ spin_unlock(ptl);
+ swap_put_entries_direct(entry, HPAGE_PMD_NR);
+ add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR);
+ return true;
+ }
+
is_present = pmd_present(orig_pmd);
folio = normal_or_softleaf_folio_pmd(vma, addr, orig_pmd, is_present);
has_deposit = has_deposited_pgtable(vma, orig_pmd, folio);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 19/29] mm/madvise: free PMD swap entries with MADV_FREE
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (17 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 18/29] mm: free PMD swap entries in zap_huge_pmd() Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 20/29] mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT Usama Arif
` (10 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
madvise_free_pte_range() only calls madvise_free_huge_pmd() for a present
THP, and its PTE loop cannot descend into a non-present PMD, so MADV_FREE
over a swapped-out THP silently does nothing. Over the same range as PTE
swap entries it frees every slot, which is what userspace gets today.
Free it at PMD granularity when the advice covers the whole PMD, and split
and let the PTE path handle the subrange when it does not.
zap_deposited_table() moves above its new caller; it is otherwise
unchanged.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/huge_memory.c | 34 +++++++++++++++++++++++++---------
mm/madvise.c | 2 +-
2 files changed, 26 insertions(+), 10 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 689a1f4286471..5f3d620c64a94 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2447,6 +2447,15 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
return 0;
}
+static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
+{
+ pgtable_t pgtable;
+
+ pgtable = pgtable_trans_huge_withdraw(mm, pmd);
+ pte_free_defer(mm, pgtable);
+ mm_dec_nr_ptes(mm);
+}
+
/*
* Return true if we do MADV_FREE successfully on entire pmd page.
* Otherwise, return false.
@@ -2471,6 +2480,22 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
goto out;
if (unlikely(!pmd_present(orig_pmd))) {
+ if (pmd_is_swap_entry(orig_pmd)) {
+ softleaf_t entry = softleaf_from_pmd(orig_pmd);
+
+ if (next - addr != HPAGE_PMD_SIZE) {
+ spin_unlock(ptl);
+ __split_huge_pmd(vma, pmd, addr);
+ goto out_unlocked;
+ }
+
+ pmdp_huge_get_and_clear(mm, addr, pmd);
+ zap_deposited_table(mm, pmd);
+ spin_unlock(ptl);
+ swap_put_entries_direct(entry, HPAGE_PMD_NR);
+ add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR);
+ return true;
+ }
VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) &&
!pmd_is_device_private_entry(orig_pmd));
goto out;
@@ -2525,15 +2550,6 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
return ret;
}
-static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
-{
- pgtable_t pgtable;
-
- pgtable = pgtable_trans_huge_withdraw(mm, pmd);
- pte_free_defer(mm, pgtable);
- mm_dec_nr_ptes(mm);
-}
-
static void zap_huge_pmd_folio(struct mm_struct *mm, struct vm_area_struct *vma,
pmd_t pmdval, struct folio *folio, bool is_present)
{
diff --git a/mm/madvise.c b/mm/madvise.c
index fbb72ab49aa64..70acb39b8b3dd 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -678,7 +678,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
int nr, max_nr;
next = pmd_addr_end(addr, end);
- if (pmd_trans_huge(*pmd))
+ if (pmd_trans_huge(*pmd) || pmd_is_swap_entry(*pmd))
if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next))
return 0;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 20/29] mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (18 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 19/29] mm/madvise: free PMD swap entries with MADV_FREE Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 21/29] mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE Usama Arif
` (9 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
A swapped-out THP has no resident folio to deactivate or reclaim, which is
also the conclusion the PTE path reaches when it walks the swap entries and
skips every one.
madvise_cold_or_pageout_pte_range() gets there the long way round:
pmd_trans_huge() is false for a PMD swap entry, so it falls out of the huge
branch and tries to map a PTE table that a non-present PMD does not have.
The entry can also appear after that first, lockless test, so re-check
under the PMD lock; otherwise a PMD that turned into a swap entry while
pmd_trans_huge_lock() was being taken trips the VM_WARN_ON_ONCE() that
expects a migration or device-private entry.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/madvise.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/mm/madvise.c b/mm/madvise.c
index 70acb39b8b3dd..d3b5ed7e3cffb 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -385,6 +385,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
!can_do_file_pageout(vma);
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ /*
+ * Swapped-out THPs have no resident folio to deactivate or reclaim.
+ * Avoid descending into or splitting a PMD swap entry.
+ */
+ if (pmd_is_swap_entry(*pmd)) {
+ walk->action = ACTION_CONTINUE;
+ return 0;
+ }
+
if (pmd_trans_huge(*pmd)) {
pmd_t orig_pmd;
unsigned long next = pmd_addr_end(addr, end);
@@ -395,6 +404,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
return 0;
orig_pmd = *pmd;
+ if (pmd_is_swap_entry(orig_pmd))
+ goto huge_unlock;
+
if (unlikely(!pmd_present(orig_pmd))) {
VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) &&
!pmd_is_device_private_entry(orig_pmd));
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 21/29] mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (19 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 20/29] mm/madvise: skip PMD swap entries for MADV_COLD and MADV_PAGEOUT Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 22/29] mm/mincore: report PMD swap-cache residency Usama Arif
` (8 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
Both callbacks test bare pmd_trans_huge(), which is false for a PMD swap
entry, so they return without consuming the PMD and walk_pmd_range() splits
it - only for the PTE callbacks to find the range populated and abort, or
to find no guard markers and do nothing.
Both already short-circuit at PMD level for a present THP. Do the same for
a swapped-out one: it is just as populated, and just as free of guard
markers.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/madvise.c | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/mm/madvise.c b/mm/madvise.c
index d3b5ed7e3cffb..64019668d69fb 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -1269,8 +1269,12 @@ static int guard_install_pmd_entry(pmd_t *pmd, unsigned long addr,
{
pmd_t pmdval = pmdp_get(pmd);
- /* If huge return >0 so we abort the operation + zap. */
- return pmd_trans_huge(pmdval);
+ /*
+ * If huge return >0 so we abort the operation + zap. A PMD swap entry
+ * is a swapped-out THP: also populated, and splitting it here would
+ * only demote it before the zap.
+ */
+ return pmd_trans_huge(pmdval) || pmd_is_swap_entry(pmdval);
}
static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
@@ -1409,8 +1413,12 @@ static int guard_remove_pmd_entry(pmd_t *pmd, unsigned long addr,
{
pmd_t pmdval = pmdp_get(pmd);
- /* If huge, cannot have guard pages present, so no-op - skip. */
- if (pmd_trans_huge(pmdval))
+ /*
+ * If huge, cannot have guard pages present, so no-op - skip. The same
+ * is true of a PMD swap entry, which must not be split just to discover
+ * there is nothing to remove.
+ */
+ if (pmd_trans_huge(pmdval) || pmd_is_swap_entry(pmdval))
walk->action = ACTION_CONTINUE;
return 0;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 22/29] mm/mincore: report PMD swap-cache residency
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (20 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 21/29] mm/madvise: keep PMD swap entries whole for MADV_GUARD_INSTALL/REMOVE Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 23/29] mm/khugepaged: treat PMD swap entries as mapped THPs Usama Arif
` (7 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
mincore_pte_range() reports every page under a huge PMD as resident. That
is wrong for a swapped-out one: mincore() would tell userspace that memory
it will have to fault in is already in core.
A PMD swap entry is only a compact encoding for HPAGE_PMD_NR slots, so
answer from the swap cache instead. One PMD-sized folio covering the range
makes every page resident exactly when that folio is uptodate; an empty
cache makes none of them; and if the folio was split while the entry stayed
in place, fall back to looking up each covered slot as mincore_swap() does
for a PTE swap entry.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/mincore.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 46 insertions(+), 1 deletion(-)
diff --git a/mm/mincore.c b/mm/mincore.c
index c086836bc4bcc..1846c6f68ae63 100644
--- a/mm/mincore.c
+++ b/mm/mincore.c
@@ -85,6 +85,48 @@ static unsigned char mincore_swap(swp_entry_t entry, bool shmem)
return present;
}
+#ifdef CONFIG_THP_SWAP
+static void mincore_pmd_swap(swp_entry_t entry, unsigned long addr,
+ unsigned long end, unsigned char *vec)
+{
+ unsigned long haddr = addr & HPAGE_PMD_MASK;
+ unsigned long start = (addr - haddr) >> PAGE_SHIFT;
+ unsigned long nr = (end - addr) >> PAGE_SHIFT;
+ struct folio *folio;
+ enum swap_pmd_cache state;
+ int i;
+
+ state = swap_pmd_cache_lookup(entry, &folio);
+ if (state == SWAP_PMD_CACHE_HUGE) {
+ memset(vec, folio_test_uptodate(folio), nr);
+ folio_put(folio);
+ return;
+ }
+
+ if (state == SWAP_PMD_CACHE_EMPTY) {
+ memset(vec, 0, nr);
+ return;
+ }
+
+ /*
+ * The PMD swap entry is only a compact encoding for consecutive swap
+ * slots. If the PMD-sized swapcache folio was split, report residency
+ * from the individual slots covered by this mincore() range.
+ */
+ for (i = 0; i < nr; i++)
+ vec[i] = mincore_swap(swp_entry(swp_type(entry),
+ swp_offset(entry) + start + i),
+ false);
+}
+#else /* !CONFIG_THP_SWAP */
+static void mincore_pmd_swap(swp_entry_t entry, unsigned long addr,
+ unsigned long end, unsigned char *vec)
+{
+ /* Nothing produces PMD swap entries without CONFIG_THP_SWAP. */
+ memset(vec, 0, (end - addr) >> PAGE_SHIFT);
+}
+#endif /* CONFIG_THP_SWAP */
+
/*
* Later we can get more picky about what "in core" means precisely.
* For now, simply check to see if the page is in the page cache,
@@ -171,7 +213,10 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
ptl = pmd_trans_huge_lock(pmd, vma);
if (ptl) {
- memset(vec, 1, nr);
+ if (pmd_is_swap_entry(*pmd))
+ mincore_pmd_swap(softleaf_from_pmd(*pmd), addr, end, vec);
+ else
+ memset(vec, 1, nr);
spin_unlock(ptl);
goto out;
}
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 23/29] mm/khugepaged: treat PMD swap entries as mapped THPs
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (21 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 22/29] mm/mincore: report PMD swap-cache residency Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 24/29] mm: handle PMD swap entries in MADV_WILLNEED Usama Arif
` (6 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
check_pmd_state() would report a PMD swap entry as SCAN_NO_PTE_TABLE - it
is not present and, on x86, is also pmd_bad(). The range is already a THP;
it just happens to be on disk.
Return SCAN_PMD_MAPPED next to the existing migration-entry case and before
the pmd_present() and pmd_bad() tests.
This does change MADV_COLLAPSE. Today a swapped-out THP is HPAGE_PMD_NR PTE
swap entries and MADV_COLLAPSE, which ignores khugepaged_max_ptes_swap,
swaps them all in. With a PMD swap entry it reports SCAN_PMD_MAPPED, which
madvise_collapse() counts as success, so the call returns 0 without making
the range resident. It still faults back in as a whole THP on first touch;
only the point at which the I/O happens moves.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/khugepaged.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 0d8cd44b14c57..bbbf96c616295 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1052,6 +1052,12 @@ static inline enum scan_result check_pmd_state(pmd_t *pmd)
*/
if (pmd_is_migration_entry(pmde))
return SCAN_PMD_MAPPED;
+ /*
+ * A PMD-mapped THP that has been swapped out is still a THP from
+ * khugepaged's perspective; treat it like a present huge PMD.
+ */
+ if (pmd_is_swap_entry(pmde))
+ return SCAN_PMD_MAPPED;
if (!pmd_present(pmde))
return SCAN_NO_PTE_TABLE;
if (pmd_trans_huge(pmde))
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 24/29] mm: handle PMD swap entries in MADV_WILLNEED
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (22 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 23/29] mm/khugepaged: treat PMD swap entries as mapped THPs Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 25/29] mm: handle PMD swap entries in UFFDIO_MOVE Usama Arif
` (5 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
swapin_walk_pmd_entry() walks PTEs and cannot descend into a non-present
PMD, so MADV_WILLNEED is a no-op on a PMD swap entry. Reading the slots
back one at a time would not be much better: order-0 readahead populates
per-page swap-cache state, and the fault that follows would then have to
split the entry the prefetch was supposed to help.
Prefetch the whole range as one PMD-order folio instead, leaving the entry
in place so the eventual fault still takes do_huge_pmd_swap_page(). Check
the THP policy first: the range was PMD-mapped when it was swapped out, but
MADV_NOHUGEPAGE or the sysfs knob may have changed since, and reading at
PMD order would then hand do_swap_page() a PMD-sized folio to map with
PTEs.
A split cache, per-page zswap state, or a failed PMD-order read or
allocation falls back to PTEs. Leaving the entry alone would prefetch
nothing at all, while the PTE path can still read the slots at order 0. A
folio that failed to read is dropped from the swap cache first, so the PTE
retry re-reads each slot rather than every later fault returning SIGBUS.
Only split if the PMD still looks like the entry that was observed - the
decision is taken after the PMD lock has been dropped, so a racing fault
may have swapped the range back in as a THP. The test is lockless, so it
narrows that window rather than closing it.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/madvise.c | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 139 insertions(+)
diff --git a/mm/madvise.c b/mm/madvise.c
index 64019668d69fb..ec1cb1b71070a 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -33,6 +33,7 @@
#include <linux/shmem_fs.h>
#include <linux/mmu_notifier.h>
#include <linux/swap_ops.h>
+#include <linux/zswap.h>
#include <asm/tlb.h>
@@ -194,6 +195,108 @@ static int madvise_update_vma(vm_flags_t new_flags,
}
#ifdef CONFIG_SWAP
+/*
+ * Prefetch a whole PMD swap entry as one PMD-order folio.
+ *
+ * Called with the PMD lock held; always drops it. Returns true when the
+ * caller should ask the walker to retry so the PTE path can handle the
+ * covered slots individually.
+ */
+static bool swapin_pmd_swap_entry(struct vm_area_struct *vma, pmd_t *pmd,
+ unsigned long addr, softleaf_t entry,
+ spinlock_t *ptl)
+{
+ struct vm_fault vmf = {
+ .vma = vma,
+ .address = addr,
+ .real_address = addr,
+ .pmd = pmd,
+ };
+ enum swap_pmd_cache cache_state;
+ struct swap_info_struct *si;
+ struct folio *folio;
+ bool split = false;
+
+ /*
+ * The range was PMD-mapped when it was swapped out, but the policy may
+ * have changed since: MADV_NOHUGEPAGE, or the sysfs knob. Reading it
+ * back at PMD order would then hand do_swap_page() a PMD-sized folio to
+ * map with PTEs, which is exactly what the policy forbids. Split now
+ * and let the PTE path prefetch at order 0 - the next fault would split
+ * the entry anyway, so nothing is lost that the VMA still permits.
+ */
+ if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_PAGEFAULT,
+ HPAGE_PMD_ORDER)) {
+ spin_unlock(ptl);
+ return true;
+ }
+
+ cache_state = swap_pmd_cache_lookup(entry, &folio);
+ if (cache_state == SWAP_PMD_CACHE_HUGE) {
+ /* Already cached as one PMD-sized folio, nothing to do. */
+ folio_put(folio);
+ spin_unlock(ptl);
+ return false;
+ }
+ if (cache_state == SWAP_PMD_CACHE_SPLIT ||
+ zswap_is_present(entry, HPAGE_PMD_NR)) {
+ spin_unlock(ptl);
+ return true;
+ }
+
+ /*
+ * Pin the swap device under the PMD lock so the PMD-swap-entry
+ * observation keeps the entry valid for swapin_sync().
+ */
+ si = get_swap_device(entry);
+ spin_unlock(ptl);
+ if (IS_ERR_OR_NULL(si))
+ return false;
+
+ folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE, BIT(HPAGE_PMD_ORDER),
+ &vmf, NULL, 0);
+
+ /*
+ * Fall back to PTE-order swapin: a PMD-order failure does not mean
+ * that individual slots cannot be read.
+ */
+ if (IS_ERR_OR_NULL(folio)) {
+ split = true;
+ goto out;
+ }
+
+ if (folio_nr_pages(folio) != HPAGE_PMD_NR) {
+ split = true;
+ goto out_put;
+ }
+
+ /*
+ * A trylock only succeeds once the read has completed, so this never
+ * blocks MADV_WILLNEED on in-flight I/O. A read that failed - a
+ * PMD-order zswap load that found per-page state, or an I/O error -
+ * leaves the folio clean and not uptodate. Drop it from the swap cache
+ * so the PTE retry reads each slot again; leaving it there would make
+ * the next fault return VM_FAULT_SIGBUS. Another thread may have
+ * removed it already, so revalidate the association first.
+ */
+ if (!folio_trylock(folio))
+ goto out_put;
+
+ if (!folio_test_uptodate(folio)) {
+ if (folio_matches_swap_entry(folio, entry))
+ swap_cache_del_folio(folio);
+ split = true;
+ }
+ folio_unlock(folio);
+
+out_put:
+ folio_put(folio);
+out:
+ /* Keep the device pinned until the last use of @entry. */
+ put_swap_device(si);
+ return split;
+}
+
static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,
unsigned long end, struct mm_walk *walk)
{
@@ -203,6 +306,41 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,
spinlock_t *ptl;
unsigned long addr;
+ ptl = pmd_trans_huge_lock(pmd, vma);
+ if (ptl) {
+ pmd_t pmdval = *pmd;
+
+ if (pmd_is_swap_entry(pmdval)) {
+ /* swapin_pmd_swap_entry() always drops the PMD lock. */
+ if (!swapin_pmd_swap_entry(vma, pmd, start,
+ softleaf_from_pmd(pmdval),
+ ptl))
+ goto ret;
+ /*
+ * Only split if this still looks like the entry we
+ * observed. The fallback was decided after the PMD lock
+ * was dropped, so a racing fault may have swapped the
+ * range back in as a THP, and splitting that would
+ * demote a perfectly good huge mapping for an advisory
+ * hint. The test is lockless, so it narrows that window
+ * rather than closing it.
+ */
+ if (pmd_same(pmdval, pmdp_get_lockless(pmd))) {
+ __split_huge_pmd(vma, pmd, start);
+ walk->action = ACTION_AGAIN;
+ goto ret;
+ }
+ /*
+ * Somebody else changed the PMD. Leave it alone and let
+ * the PTE loop below deal with whatever is there now;
+ * it simply finds no page table if the range came back
+ * as a THP.
+ */
+ } else {
+ spin_unlock(ptl);
+ }
+ }
+
for (addr = start; addr < end; addr += PAGE_SIZE) {
pte_t pte;
softleaf_t entry;
@@ -231,6 +369,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start,
if (ptep)
pte_unmap_unlock(ptep, ptl);
swap_read_submit(&ctx);
+ret:
cond_resched();
return 0;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 25/29] mm: handle PMD swap entries in UFFDIO_MOVE
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (23 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 24/29] mm: handle PMD swap entries in MADV_WILLNEED Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 26/29] mm: don't PTE-batch a swap-in over a hardware-poisoned subpage Usama Arif
` (4 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
move_pages_huge_pmd() returns -ENOENT for any PMD that is neither
trans_huge nor a migration entry, so an aligned UFFDIO_MOVE over a
swapped-out THP fails even though a PMD swap entry is a perfectly good
mapping to move. Falling back to the PTE path is no help either: splitting
yields PTE swap entries pointing at the same swap-cache folio, and
move_pages_ptes() refuses any swap-cache folio that is still large.
move_swap_pmd() is modelled on move_swap_pte(): it moves the entry under
both PMD locks, propagates soft-dirty, arms the UFFD marker for an
RWP-registered destination, carries the deposited page table across, and
requires pmd_swp_exclusive() for the same single-owner semantics.
The entry can only be moved whole while the covered swap cache is empty or
holds one PMD-sized folio. A cached folio is locked and revalidated, then
its anon rmap is re-anchored to the destination VMA; an empty cache is
re-checked slot by slot under both PMD locks, because a per-slot folio that
appeared meanwhile would need the PTE path to fix up its rmap metadata. A
range that is already split is split and retried through PTEs. Revalidation
failure just returns -EAGAIN: its usual cause is a racing fault that made
src_pmd a healthy present THP, which must not be shattered.
Finally, reject a PMD swap entry at the *destination* with -EEXIST. It is
not a hole, and unlike a migration entry it does not resolve on its own:
pte_alloc() skips a !pmd_none PMD, pte_offset_map_rw_nolock() then fails,
and the resulting -EAGAIN would be retried forever.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/huge_memory.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++-
mm/userfaultfd.c | 14 +++++
2 files changed, 171 insertions(+), 1 deletion(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 5f3d620c64a94..497f677a3ef71 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2972,6 +2972,78 @@ int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,
#endif
#ifdef CONFIG_USERFAULTFD
+#ifdef CONFIG_THP_SWAP
+/*
+ * Move a PMD-level swap entry from src_pmd to dst_pmd. Both PMD locks are
+ * acquired here; src_folio (if present) must already be locked. The deposited
+ * page table backing the source THP is moved across with the entry.
+ */
+static int move_swap_pmd(struct mm_struct *mm, struct vm_area_struct *dst_vma,
+ unsigned long dst_addr, unsigned long src_addr,
+ pmd_t *dst_pmd, pmd_t *src_pmd,
+ pmd_t orig_dst_pmd, pmd_t orig_src_pmd,
+ spinlock_t *dst_ptl, spinlock_t *src_ptl,
+ struct folio *src_folio, swp_entry_t entry)
+{
+ pgtable_t src_pgtable;
+ pmd_t moved_pmd;
+
+ /*
+ * The folio may have been freed and reused for a different swap entry
+ * while it was unlocked. Re-verify the association.
+ */
+ if (src_folio && unlikely(!folio_matches_swap_entry(src_folio, entry) ||
+ folio_nr_pages(src_folio) != HPAGE_PMD_NR))
+ return -EAGAIN;
+
+ double_pt_lock(dst_ptl, src_ptl);
+
+ if (!pmd_same(*src_pmd, orig_src_pmd) ||
+ !pmd_same(*dst_pmd, orig_dst_pmd)) {
+ double_pt_unlock(dst_ptl, src_ptl);
+ return -EAGAIN;
+ }
+
+ /*
+ * If the folio is in the swap cache, re-anchor its anon rmap to the
+ * destination VMA so a future swap-in fault at dst_addr finds it.
+ * Otherwise, re-check the whole PMD swap range: a PMD swap entry is
+ * only a compact encoding for HPAGE_PMD_NR swap slots, and any per-slot
+ * cached folio would need the PTE move path to update its rmap
+ * metadata.
+ */
+ if (src_folio) {
+ folio_move_anon_rmap(src_folio, dst_vma);
+ src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
+ } else {
+ unsigned int type = swp_type(entry);
+ pgoff_t offset = swp_offset(entry);
+ int i;
+
+ for (i = 0; i < HPAGE_PMD_NR; i++) {
+ if (swap_cache_has_folio(swp_entry(type, offset + i))) {
+ double_pt_unlock(dst_ptl, src_ptl);
+ return -EAGAIN;
+ }
+ }
+ }
+
+ moved_pmd = pmdp_huge_get_and_clear(mm, src_addr, src_pmd);
+ if (pgtable_supports_soft_dirty())
+ moved_pmd = pmd_swp_mksoft_dirty(moved_pmd);
+ /* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */
+ if (userfaultfd_rwp(dst_vma))
+ moved_pmd = pmd_swp_mkuffd(moved_pmd);
+ set_pmd_at(mm, dst_addr, dst_pmd, moved_pmd);
+
+ src_pgtable = pgtable_trans_huge_withdraw(mm, src_pmd);
+ pgtable_trans_huge_deposit(mm, dst_pmd, src_pgtable);
+
+ double_pt_unlock(dst_ptl, src_ptl);
+ return 0;
+}
+#endif /* CONFIG_THP_SWAP */
+
/*
* The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by
* the caller, but it must return after releasing the page_table_lock.
@@ -3006,11 +3078,95 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
}
if (!pmd_trans_huge(src_pmdval)) {
- spin_unlock(src_ptl);
if (pmd_is_migration_entry(src_pmdval)) {
+ spin_unlock(src_ptl);
pmd_migration_entry_wait(mm, src_pmd);
return -EAGAIN;
}
+#ifdef CONFIG_THP_SWAP
+ if (pmd_is_swap_entry(src_pmdval)) {
+ swp_entry_t entry;
+ struct swap_info_struct *si;
+ enum swap_pmd_cache cache_state;
+
+ /*
+ * UFFDIO_MOVE on anon mappings requires single-owner
+ * semantics; refuse to move a shared swap entry.
+ */
+ if (!pmd_swp_exclusive(src_pmdval)) {
+ spin_unlock(src_ptl);
+ return -EBUSY;
+ }
+
+ entry = softleaf_from_pmd(src_pmdval);
+ spin_unlock(src_ptl);
+
+ /*
+ * Pin the swap device against a racing swapoff. NULL
+ * means swapoff is in progress, which resolves on its
+ * own, so ask the caller to retry. An error pointer
+ * means the entry names no swap device at all: that
+ * never resolves, so report it instead of spinning in
+ * the caller's -EAGAIN loop.
+ */
+ si = get_swap_device(entry);
+ if (!si)
+ return -EAGAIN;
+ if (IS_ERR(si))
+ return PTR_ERR(si);
+
+ src_folio = NULL;
+ cache_state = swap_pmd_cache_lookup(entry, &src_folio);
+ if (cache_state == SWAP_PMD_CACHE_SPLIT) {
+ put_swap_device(si);
+ __split_huge_pmd(src_vma, src_pmd, src_addr);
+ return -EAGAIN;
+ }
+
+ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0,
+ mm, src_addr,
+ src_addr + HPAGE_PMD_SIZE);
+ mmu_notifier_invalidate_range_start(&range);
+
+ if (src_folio) {
+ folio_lock(src_folio);
+ /*
+ * Do not split on failure here. The usual cause
+ * is that a racing fault swapped the range back
+ * in and dropped the folio from the swap cache,
+ * so src_pmd is now a healthy present THP;
+ * splitting it would destroy the very mapping
+ * UFFDIO_MOVE is trying to move whole. The
+ * caller's -EAGAIN retry re-reads src_pmd and
+ * picks the right path, exactly as
+ * move_swap_pte() relies on for the PTE case.
+ */
+ if (!folio_matches_swap_entry(src_folio, entry) ||
+ folio_nr_pages(src_folio) != HPAGE_PMD_NR) {
+ folio_unlock(src_folio);
+ folio_put(src_folio);
+ mmu_notifier_invalidate_range_end(&range);
+ put_swap_device(si);
+ return -EAGAIN;
+ }
+ }
+
+ dst_ptl = pmd_lockptr(mm, dst_pmd);
+ err = move_swap_pmd(mm, dst_vma, dst_addr, src_addr,
+ dst_pmd, src_pmd, dst_pmdval,
+ src_pmdval, dst_ptl, src_ptl,
+ src_folio, entry);
+
+ mmu_notifier_invalidate_range_end(&range);
+ if (src_folio) {
+ folio_unlock(src_folio);
+ folio_put(src_folio);
+ }
+ put_swap_device(si);
+ return err;
+ }
+#endif /* CONFIG_THP_SWAP */
+ spin_unlock(src_ptl);
return -ENOENT;
}
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 79cc7b546f130..e9e1df254fd72 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -2053,6 +2053,20 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
break;
}
+ /*
+ * A PMD swap entry at dst is a swapped-out THP, not a hole,
+ * and unlike a PMD migration entry it will not resolve on its
+ * own. Nothing below faults it back in: pte_alloc() skips a
+ * !pmd_none PMD, pte_offset_map_rw_nolock() then fails on the
+ * non-present PMD, and the -EAGAIN that produces would be
+ * retried forever by the loop below. Be strict, exactly as for
+ * a present THP.
+ */
+ if (unlikely(pmd_is_swap_entry(dst_pmdval))) {
+ err = -EEXIST;
+ break;
+ }
+
ptl = pmd_trans_huge_lock(src_pmd, src_vma);
if (ptl) {
/* Check if we can move the pmd without splitting it. */
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 26/29] mm: don't PTE-batch a swap-in over a hardware-poisoned subpage
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (24 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 25/29] mm: handle PMD swap entries in UFFDIO_MOVE Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 27/29] mm: handle PMD swap entry faults on swap-in Usama Arif
` (3 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
do_swap_page() checks PageHWPoison() on the faulting page, but the
large-folio batching path then maps the rest of the folio without looking
at the other subpages. A poisoned subpage that is not the one being faulted
gets mapped anyway, and the fault that would have reported it never
happens.
Skip batching for such a folio and let each page take its own fault, where
the existing check catches it. Test every subpage rather than the
folio-level PG_has_hwpoisoned: memory_failure() sets PageHWPoison() on the
subpage before it acquires the folio lock, which we hold here.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
mm/memory.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/mm/memory.c b/mm/memory.c
index 84e1e1c22bffa..aa1f67b378587 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5020,7 +5020,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
page_idx = 0;
address = vmf->address;
ptep = vmf->pte;
- if (folio_test_large(folio) && folio_test_swapcache(folio)) {
+ /*
+ * Scan every subpage rather than testing the folio-level
+ * PG_has_hwpoisoned: memory_failure() sets PageHWPoison on the subpage
+ * before it takes the folio lock, and we hold that lock, so the
+ * folio-level flag can still be clear here.
+ */
+ if (folio_test_large(folio) && folio_test_swapcache(folio) &&
+ !folio_has_hwpoisoned_subpage(folio)) {
int nr = folio_nr_pages(folio);
unsigned long idx = folio_page_idx(folio, page);
unsigned long folio_start = address - idx * PAGE_SIZE;
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 27/29] mm: handle PMD swap entry faults on swap-in
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (25 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 26/29] mm: don't PTE-batch a swap-in over a hardware-poisoned subpage Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 28/29] mm: install PMD swap entries on swap-out Usama Arif
` (2 subsequent siblings)
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
Nothing faults a PMD swap entry back in. __handle_mm_fault() recognises a
non-present PMD as device-private or a migration entry and returns 0 for
anything else, so a swapped-out THP would refault forever.
do_huge_pmd_swap_page() resolves the whole mapping in one go, mirroring
do_swap_page() at PMD granularity, and restores soft-dirty, uffd-wp and
write permission the same way. It deliberately skips the order-0 readahead
paths: the fault already asks for the whole range, and readahead would
populate per-page swap-cache state and force the entry to split before the
fault could finish.
A PMD swap entry only promises HPAGE_PMD_NR consecutive slots, not that the
cache still holds one folio for them. When it does not - the VMA no longer
permits PMD-order THPs, the cache has been split, a slot is in zswap, the
allocation or read fails, the memcg charge fails, or the cache insertion
loses a race - split the entry and return 0 so the retry lands in
do_swap_page(). That keeps a transient PMD-order allocation failure from
becoming VM_FAULT_OOM. Only split if the PMD is still the entry we were
called for, since every reason to fall back was observed without the PMD
lock.
Before falling back with a locked folio, drop it from the swap cache if it
has never been mapped or is not uptodate: do_swap_page() would hand an
unmapped PMD-sized folio to folio_add_new_anon_rmap() as a whole while
installing one PTE, and a folio that failed to read would make every later
fault return VM_FAULT_SIGBUS instead of re-reading the slots. An uptodate
anon folio stays cached so a poisoned subpage stays visible. This mirrors
unuse_pmd_entry().
Refuse to map a folio with a poisoned subpage so do_swap_page() can report
VM_FAULT_HWPOISON for the offending page, and drop the exclusive marker
when the folio is under writeback to an SWP_STABLE_WRITES backend such as
zram, so a later write COWs rather than corrupting the writeback.
When the PMD ends up read-only but the fault was a write, call
wp_huge_pmd() from the same handler so the COW does not need a second
fault. Mask VM_FAULT_FALLBACK out of what it returns: splitting to PTE
level is a normal outcome, but the bit is part of VM_FAULT_ERROR and arch
fault handlers BUG() on it without an accompanying signal.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
include/linux/huge_mm.h | 14 +++
mm/huge_memory.c | 257 ++++++++++++++++++++++++++++++++++++++++
mm/internal.h | 42 +++++++
mm/memory.c | 40 +------
4 files changed, 319 insertions(+), 34 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 64b6a2eea899d..b44a228dfe20c 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -548,6 +548,15 @@ vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf);
vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);
+#ifdef CONFIG_THP_SWAP
+vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf);
+#else
+static inline vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf)
+{
+ return 0;
+}
+#endif
+
extern struct folio *huge_zero_folio;
extern unsigned long huge_zero_pfn;
@@ -747,6 +756,11 @@ static inline vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf)
return 0;
}
+static inline vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf)
+{
+ return 0;
+}
+
static inline bool is_huge_zero_folio(const struct folio *folio)
{
return false;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 497f677a3ef71..bd9cc24c2b011 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -42,6 +42,7 @@
#include <linux/pgalloc_tag.h>
#include <linux/pagewalk.h>
#include <linux/cleanup.h>
+#include <linux/zswap.h>
#include <asm/tlb.h>
#include "internal.h"
@@ -2447,6 +2448,262 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
return 0;
}
+#ifdef CONFIG_THP_SWAP
+/**
+ * do_huge_pmd_swap_page() - Handle a fault on a PMD-level swap entry.
+ * @vmf: Fault context. vmf->orig_pmd contains the swap PMD.
+ *
+ * A PMD swap entry is a compact encoding for HPAGE_PMD_NR consecutive swap
+ * slots. If the swap cache still has one PMD-sized folio covering the range,
+ * map it directly at PMD level. If the range has been split into per-page
+ * cache state, or zswap may have per-page state for it, split the PMD swap
+ * entry and retry at PTE granularity.
+ *
+ * Return: VM_FAULT_* flags.
+ */
+vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct mm_struct *mm = vma->vm_mm;
+ struct folio *folio;
+ struct page *page;
+ struct swap_info_struct *si;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
+ softleaf_t entry;
+ swp_entry_t swp_entry;
+ pmd_t pmd;
+ vm_fault_t ret = 0;
+ bool exclusive, stable_writes, rwp_restore = false;
+ bool write = vmf->flags & FAULT_FLAG_WRITE;
+ rmap_t rmap_flags = RMAP_NONE;
+ enum swap_pmd_cache cache_state;
+
+ entry = softleaf_from_pmd(vmf->orig_pmd);
+ if (unlikely(!softleaf_is_swap(entry)))
+ return 0;
+
+ if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_PAGEFAULT,
+ HPAGE_PMD_ORDER)) {
+ __split_huge_pmd(vma, vmf->pmd, haddr);
+ return 0;
+ }
+
+ swp_entry = entry;
+
+ /* Prevent swapoff from happening to us. */
+ si = get_swap_device(swp_entry);
+ if (IS_ERR_OR_NULL(si)) {
+ if (IS_ERR(si))
+ return VM_FAULT_SIGBUS;
+ return 0;
+ }
+
+ cache_state = swap_pmd_cache_lookup(swp_entry, &folio);
+ if (cache_state == SWAP_PMD_CACHE_SPLIT)
+ goto split_fallback;
+ if (!folio) {
+ /*
+ * PMD swap entries encode ordinary per-page swap slots. If any
+ * slot is in zswap, split and let the PTE swap path load the
+ * range per page. Otherwise the range is all on disk and can be
+ * read back as one PMD-sized folio.
+ */
+ if (zswap_is_present(swp_entry, HPAGE_PMD_NR))
+ goto split_fallback;
+
+ folio = swapin_sync(swp_entry, GFP_HIGHUSER_MOVABLE,
+ BIT(HPAGE_PMD_ORDER), vmf, NULL, 0);
+ if (IS_ERR_OR_NULL(folio))
+ goto split_fallback;
+
+ /* Had to read from swap area: Major fault */
+ ret = VM_FAULT_MAJOR;
+ count_vm_event(PGMAJFAULT);
+ count_memcg_event_mm(mm, PGMAJFAULT);
+ }
+
+ ret |= folio_lock_or_retry(folio, vmf);
+ if (ret & VM_FAULT_RETRY)
+ goto out_release;
+
+ /* Verify the folio is still in swap cache and matches our entry */
+ if (unlikely(!folio_matches_swap_entry(folio, swp_entry)))
+ goto out_page;
+
+ /*
+ * Folio should be PMD-sized; if not (e.g. split in swap cache),
+ * split the PMD swap entry and retry at PTE level.
+ */
+ if (folio_nr_pages(folio) != HPAGE_PMD_NR)
+ goto unlock_split_fallback;
+
+ /*
+ * A read that failed - a PMD-order zswap load that found per-page
+ * state, or an I/O error - leaves the folio clean and not uptodate.
+ * Fall back so the PTE retry reads each slot again rather than
+ * returning SIGBUS for the whole range.
+ */
+ if (unlikely(!folio_test_uptodate(folio)))
+ goto unlock_split_fallback;
+
+ /*
+ * If any subpage is hardware-poisoned, split the PMD swap entry and
+ * let the PTE swap-in path handle each page individually so
+ * do_swap_page() can return VM_FAULT_HWPOISON for the poisoned
+ * subpage rather than mapping the corrupted memory as one THP.
+ */
+ if (unlikely(folio_has_hwpoisoned_subpage(folio)))
+ goto unlock_split_fallback;
+
+ page = folio_page(folio, 0);
+ arch_swap_restore(folio_swap(swp_entry, folio), folio);
+
+ folio_throttle_swaprate(folio, GFP_KERNEL);
+
+ /* Lock the PMD and verify it hasn't changed */
+ vmf->ptl = pmd_lock(mm, vmf->pmd);
+ if (unlikely(!pmd_same(vmf->orig_pmd, pmdp_get(vmf->pmd)))) {
+ spin_unlock(vmf->ptl);
+ goto out_page;
+ }
+
+ exclusive = pmd_swp_exclusive(vmf->orig_pmd);
+
+ /*
+ * Some swap backends (e.g. zram) don't support concurrent page
+ * modifications while under writeback. If we map exclusive on such
+ * a backend while the folio is still under writeback, the writeback
+ * may see partial modifications and corrupt the swap slot. Drop the
+ * exclusive marker and only map R/O for that case; further GUP
+ * references can't appear once the page is fully unmapped, so this
+ * is safe.
+ */
+ /* Lockless like do_swap_page(): SWP_STABLE_WRITES never changes. */
+ stable_writes = data_race(si->flags & SWP_STABLE_WRITES);
+ if (exclusive && folio_test_writeback(folio) && stable_writes)
+ exclusive = false;
+
+ /*
+ * Set up the PMD mapping. Similar to do_swap_page() but at PMD level.
+ */
+ add_mm_counter(mm, MM_ANONPAGES, HPAGE_PMD_NR);
+ add_mm_counter(mm, MM_SWAPENTS, -HPAGE_PMD_NR);
+
+ pmd = folio_mk_pmd(folio, vma->vm_page_prot);
+ pmd = pmd_mkyoung(pmd);
+
+ if (pmd_swp_soft_dirty(vmf->orig_pmd))
+ pmd = pmd_mksoft_dirty(pmd);
+ if (pmd_swp_uffd(vmf->orig_pmd))
+ pmd = pmd_mkuffd(pmd);
+ if (pmd_swp_uffd(vmf->orig_pmd) && userfaultfd_rwp(vma)) {
+ pmd = pmd_modify(pmd, PAGE_NONE);
+ rwp_restore = true;
+ }
+
+ /*
+ * Check exclusivity to determine if we can map writable.
+ */
+ if (exclusive) {
+ if (!rwp_restore && (vma->vm_flags & VM_WRITE) &&
+ !userfaultfd_huge_pmd_wp(vma, pmd) &&
+ !pmd_needs_soft_dirty_wp(vma, pmd)) {
+ pmd = pmd_mkwrite(pmd, vma);
+ if (write)
+ pmd = pmd_mkdirty(pmd);
+ }
+ rmap_flags |= RMAP_EXCLUSIVE;
+ }
+
+ flush_icache_pages(vma, page, HPAGE_PMD_NR);
+
+ if (!folio_test_anon(folio))
+ folio_add_new_anon_rmap(folio, vma, haddr, rmap_flags);
+ else
+ folio_add_anon_rmap_pmd(folio, page, vma, haddr, rmap_flags);
+
+ folio_put_swap(folio, NULL);
+
+ set_pmd_at(mm, haddr, vmf->pmd, pmd);
+ update_mmu_cache_pmd(vma, haddr, vmf->pmd);
+
+ /* Update orig_pmd for any follow-up wp_huge_pmd() below. */
+ vmf->orig_pmd = pmd;
+
+ /*
+ * Conditionally try to free up the swap cache. Do it after mapping,
+ * so raced page faults will likely see the folio in swap cache and
+ * wait on the folio lock.
+ */
+ if (should_try_to_free_swap(si, folio, vma, exclusive, vmf->flags))
+ folio_free_swap(folio);
+
+ spin_unlock(vmf->ptl);
+
+ folio_unlock(folio);
+ put_swap_device(si);
+
+ /*
+ * If the write fault wasn't satisfied above (folio is shared without
+ * exclusivity), call wp_huge_pmd() to handle COW or
+ * userfaultfd-wp without forcing a second fault.
+ *
+ * wp_huge_pmd() may return VM_FAULT_FALLBACK if it had to split the
+ * PMD; that's a normal outcome, and the natural PTE-level refault will
+ * complete the COW. Mask it so callers (and the arch fault handler)
+ * don't see VM_FAULT_FALLBACK as a fatal VM_FAULT_ERROR.
+ */
+ if (write && !pmd_write(pmd) && !rwp_restore) {
+ vm_fault_t wp_ret = wp_huge_pmd(vmf);
+
+ wp_ret &= ~VM_FAULT_FALLBACK;
+ ret |= wp_ret;
+ if (ret & VM_FAULT_ERROR)
+ ret &= VM_FAULT_ERROR;
+ }
+
+ return ret;
+
+out_page:
+ folio_unlock(folio);
+out_release:
+ folio_put(folio);
+ put_swap_device(si);
+ return ret;
+
+unlock_split_fallback:
+ /*
+ * PTE fallback cannot add a single-page rmap to a PMD-sized folio that
+ * has never been mapped: do_swap_page() would hand the whole folio to
+ * folio_add_new_anon_rmap() while installing one PTE. Nor can it do
+ * anything useful with a folio that failed to read. Remove either from
+ * the swap cache so each slot is read back into its own order-0 folio.
+ * An uptodate anon swap-cache folio can be mapped one PTE at a time and
+ * must stay cached, so that any poisoned subpage stays visible to
+ * do_swap_page(). This mirrors unuse_pmd_entry().
+ */
+ if (folio_matches_swap_entry(folio, swp_entry) &&
+ (!folio_test_uptodate(folio) || !folio_test_anon(folio)))
+ swap_cache_del_folio(folio);
+ folio_unlock(folio);
+ folio_put(folio);
+
+split_fallback:
+ /*
+ * Only split if the PMD is still the swap entry we were called for.
+ * All the reasons we get here (allocation failure, zswap state, a
+ * split or poisoned cached folio) were observed without the PMD lock,
+ * so a racing thread may already have swapped the range back in as a
+ * THP -- splitting that would silently demote a perfectly good huge
+ * mapping.
+ */
+ if (pmd_same(vmf->orig_pmd, pmdp_get_lockless(vmf->pmd)))
+ __split_huge_pmd(vma, vmf->pmd, haddr);
+ put_swap_device(si);
+ return 0;
+}
+#endif /* CONFIG_THP_SWAP */
+
static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
{
pgtable_t pgtable;
diff --git a/mm/internal.h b/mm/internal.h
index ec7f007bc2c0d..1a5480e4b5071 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -577,6 +577,48 @@ static inline vm_fault_t vmf_anon_prepare(struct vm_fault *vmf)
}
vm_fault_t do_swap_page(struct vm_fault *vmf);
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+vm_fault_t wp_huge_pmd(struct vm_fault *vmf);
+#else
+static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
+{
+ return VM_FAULT_FALLBACK;
+}
+#endif
+
+/*
+ * Check if we should call folio_free_swap to free the swap cache.
+ * folio_free_swap only frees the swap cache to release the slot if swap
+ * count is zero, so we don't need to check the swap count here.
+ */
+static inline bool should_try_to_free_swap(struct swap_info_struct *si,
+ struct folio *folio,
+ struct vm_area_struct *vma,
+ bool exclusive,
+ unsigned int fault_flags)
+{
+ if (!folio_test_swapcache(folio))
+ return false;
+ /*
+ * Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap
+ * cache can help save some IO or memory overhead, but these devices
+ * are fast, and meanwhile, swap cache pinning the slot deferring the
+ * release of metadata or fragmentation is a more critical issue.
+ */
+ if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ return true;
+ if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
+ folio_test_mlocked(folio))
+ return true;
+
+ /*
+ * Free the swapcache only if we are the exclusive user and
+ * this is a write fault.
+ */
+ return (fault_flags & FAULT_FLAG_WRITE) && exclusive;
+}
+
void folio_rotate_reclaimable(struct folio *folio);
bool __folio_end_writeback(struct folio *folio);
void deactivate_file_folio(struct folio *folio);
diff --git a/mm/memory.c b/mm/memory.c
index aa1f67b378587..63b51ba46b0b5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4603,38 +4603,6 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)
return 0;
}
-/*
- * Check if we should call folio_free_swap to free the swap cache.
- * folio_free_swap only frees the swap cache to release the slot if swap
- * count is zero, so we don't need to check the swap count here.
- */
-static inline bool should_try_to_free_swap(struct swap_info_struct *si,
- struct folio *folio,
- struct vm_area_struct *vma,
- bool exclusive,
- unsigned int fault_flags)
-{
- if (!folio_test_swapcache(folio))
- return false;
- /*
- * Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap
- * cache can help save some IO or memory overhead, but these devices
- * are fast, and meanwhile, swap cache pinning the slot deferring the
- * release of metadata or fragmentation is a more critical issue.
- */
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
- return true;
- if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
- folio_test_mlocked(folio))
- return true;
-
- /*
- * Free the swapcache only if we are the exclusive user and
- * this is a write fault.
- */
- return (fault_flags & FAULT_FLAG_WRITE) && exclusive;
-}
-
static vm_fault_t pte_marker_clear(struct vm_fault *vmf)
{
vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
@@ -6362,8 +6330,8 @@ static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf)
return VM_FAULT_FALLBACK;
}
-/* `inline' is required to avoid gcc 4.1.2 build error */
-static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
@@ -6393,6 +6361,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf)
return VM_FAULT_FALLBACK;
}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
static vm_fault_t create_huge_pud(struct vm_fault *vmf)
{
@@ -6656,6 +6625,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
if (pmd_is_migration_entry(vmf.orig_pmd))
pmd_migration_entry_wait(mm, vmf.pmd);
+ else if (IS_ENABLED(CONFIG_THP_SWAP) &&
+ pmd_is_swap_entry(vmf.orig_pmd))
+ return do_huge_pmd_swap_page(&vmf);
return 0;
}
if (pmd_trans_huge(vmf.orig_pmd)) {
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 28/29] mm: install PMD swap entries on swap-out
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (26 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 27/29] mm: handle PMD swap entry faults on swap-in Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-14 12:28 ` [RESEND v7 29/29] selftests/mm: add PMD swap entry tests Usama Arif
2026-09-15 3:32 ` [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Andrew Morton
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
When reclaim swaps out a PMD-mapped anonymous THP it first splits the PMD
via TTU_SPLIT_HUGE_PMD. The huge mapping is lost for the whole swap
round-trip: swapping the range back in takes HPAGE_PMD_NR faults and
leaves as many small mappings, and the process waits for khugepaged to
collapse them again.
The PMD does not have to be split. A contiguous run of slots was already
secured when the folio was added to the swap cache - a non-contiguous
allocation would have split the folio first - so the whole mapping can be
replaced by one PMD-level swap entry encoding the first slot.
shrink_folio_list() therefore stops asking for TTU_SPLIT_HUGE_PMD for a
PMD-mappable folio already in the swap cache, and try_to_unmap_one() grows
a PMD branch. TTU_SPLIT_HUGE_PMD remains the fallback for everything else.
set_pmd_swap_entry() is deliberately close in shape to
set_pmd_migration_entry(): invalidate the mapping while keeping the
original for rollback, take a swap reference on every slot, transfer the
exclusive state, propagate the dirty bit to the folio so writeback is not
lost, add the mm to mmlist before the entry becomes visible, and carry
over soft-dirty and uffd-wp. Any step that can fail restores the mapping
first.
The entry encodes exactly what the PTE entries would, so swap_map
accounting is unchanged: each slot carries a count of one, released
individually on a later split or together on swap-in.
zswap needs no handling here. It stores the folio as order-0 entries, and
the PMD-order swap-in users split and fall back to PTEs if any covered
slot turns out to have a zswap entry.
thp_swpout_pmd counts PMD mappings replaced this way. Unlike thp_swpout it
counts mappings rather than folios, so a fork-shared THP can increment it
once per mapping.
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
Documentation/admin-guide/mm/transhuge.rst | 5 ++
include/linux/huge_mm.h | 2 +
include/linux/vm_event_item.h | 1 +
mm/huge_memory.c | 84 ++++++++++++++++++++++
mm/rmap.c | 19 +++++
mm/vmscan.c | 9 ++-
mm/vmstat.c | 1 +
7 files changed, 120 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
index b187d618452f4..64d413d9fd83e 100644
--- a/Documentation/admin-guide/mm/transhuge.rst
+++ b/Documentation/admin-guide/mm/transhuge.rst
@@ -632,6 +632,11 @@ thp_swpout
is incremented every time a huge page is swapout in one
piece without splitting.
+thp_swpout_pmd
+ is incremented every time a PMD mapping is replaced by a PMD-level
+ swap entry. A fork-shared THP can increment this counter once for each
+ PMD mapping that is swapped out.
+
thp_swpout_fallback
is incremented if a huge page has to be split before swapout.
Usually because failed to allocate some continuous swap space
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index b44a228dfe20c..86c65e3a6c5d1 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -550,6 +550,8 @@ vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);
#ifdef CONFIG_THP_SWAP
vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf);
+int set_pmd_swap_entry(struct page_vma_mapped_walk *pvmw,
+ struct folio *folio);
#else
static inline vm_fault_t do_huge_pmd_swap_page(struct vm_fault *vmf)
{
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 2628ccda076a0..f8fd4e13698c3 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -108,6 +108,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
THP_ZERO_PAGE_ALLOC_FAILED,
THP_SWPOUT,
THP_SWPOUT_FALLBACK,
+ THP_SWPOUT_PMD,
#endif
#ifdef CONFIG_BALLOON
BALLOON_INFLATE,
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index bd9cc24c2b011..317d2108105de 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -5736,3 +5736,87 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *folio
trace_remove_migration_pmd(address, pmd_val(pmde));
}
#endif
+
+#ifdef CONFIG_THP_SWAP
+/**
+ * set_pmd_swap_entry() - Replace a PMD mapping with a PMD-level swap entry.
+ * @pvmw: Page vma mapped walk context, must have pvmw->pmd set and
+ * pvmw->pte NULL (i.e. PMD-mapped).
+ * @folio: The folio being swapped out. Must be in the swap cache.
+ *
+ * This installs a PMD-level swap entry in place of a present PMD mapping,
+ * avoiding the need to split the PMD into PTE-level swap entries.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int set_pmd_swap_entry(struct page_vma_mapped_walk *pvmw,
+ struct folio *folio)
+{
+ struct vm_area_struct *vma = pvmw->vma;
+ struct mm_struct *mm = vma->vm_mm;
+ unsigned long address = pvmw->address;
+ unsigned long haddr = address & HPAGE_PMD_MASK;
+ struct page *page = folio_page(folio, 0);
+ bool anon_exclusive;
+ pmd_t pmdval;
+ swp_entry_t entry;
+ pmd_t pmdswp;
+
+ /*
+ * try_to_unmap_one() only gets here for a PMD-mapped, anonymous,
+ * PMD-sized folio that is already in the swap cache, and a swapcache
+ * folio is always swapbacked. Refuse instead of crashing should that
+ * ever stop being true: the caller aborts the rmap walk and the folio
+ * simply stays mapped.
+ */
+ if (unlikely(!pvmw->pmd || pvmw->pte ||
+ !folio_test_anon(folio) ||
+ !folio_test_swapcache(folio) ||
+ !folio_test_swapbacked(folio) ||
+ folio_nr_pages(folio) != HPAGE_PMD_NR)) {
+ VM_WARN_ON_ONCE_FOLIO(true, folio);
+ return -EBUSY;
+ }
+
+ flush_cache_range(vma, haddr, haddr + HPAGE_PMD_SIZE);
+
+ pmdval = pmdp_invalidate(vma, haddr, pvmw->pmd);
+
+ /* Update high watermark before we lower rss */
+ update_hiwater_rss(mm);
+
+ if (folio_dup_swap(folio, NULL) < 0) {
+ set_pmd_at(mm, haddr, pvmw->pmd, pmdval);
+ return -ENOMEM;
+ }
+
+ /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */
+ anon_exclusive = PageAnonExclusive(page);
+ if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) {
+ folio_put_swap(folio, NULL);
+ set_pmd_at(mm, haddr, pvmw->pmd, pmdval);
+ return -EBUSY;
+ }
+
+ mm_prepare_for_swap_entries(mm);
+
+ if (pmd_dirty(pmdval))
+ folio_mark_dirty(folio);
+
+ entry = folio->swap;
+ pmdswp = softleaf_to_pmd(entry);
+ if (pmd_soft_dirty(pmdval))
+ pmdswp = pmd_swp_mksoft_dirty(pmdswp);
+ if (pmd_uffd(pmdval))
+ pmdswp = pmd_swp_mkuffd(pmdswp);
+ if (anon_exclusive)
+ pmdswp = pmd_swp_mkexclusive(pmdswp);
+ set_pmd_at(mm, haddr, pvmw->pmd, pmdswp);
+
+ folio_remove_rmap_pmd(folio, page, vma);
+ folio_put(folio);
+
+ count_vm_event(THP_SWPOUT_PMD);
+ return 0;
+}
+#endif /* CONFIG_THP_SWAP */
diff --git a/mm/rmap.c b/mm/rmap.c
index feb751e29b992..3745f16cf5716 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2284,6 +2284,25 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
goto walk_abort;
}
+#ifdef CONFIG_THP_SWAP
+ /*
+ * If the folio is in the swap cache and we're not
+ * asked to split, install a PMD-level swap entry.
+ */
+ if (!(flags & TTU_SPLIT_HUGE_PMD) &&
+ folio_test_anon(folio) &&
+ folio_test_swapcache(folio)) {
+ if (set_pmd_swap_entry(&pvmw, folio))
+ goto walk_abort;
+
+ add_mm_counter(mm, MM_ANONPAGES,
+ -HPAGE_PMD_NR);
+ add_mm_counter(mm, MM_SWAPENTS,
+ HPAGE_PMD_NR);
+ goto walk_done;
+ }
+#endif
+
if (flags & TTU_SPLIT_HUGE_PMD) {
/*
* We temporarily have to drop the PTL and
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c2eb8fa9d5e50..7648a2a0d0813 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1408,7 +1408,14 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
enum ttu_flags flags = TTU_BATCH_FLUSH;
bool was_swapbacked = folio_test_swapbacked(folio);
- if (folio_test_pmd_mappable(folio))
+ /*
+ * With THP_SWAP, PMD-mappable folios already in the
+ * swap cache can be unmapped with a PMD-level swap
+ * entry, avoiding the cost of splitting the PMD.
+ */
+ if (folio_test_pmd_mappable(folio) &&
+ !(IS_ENABLED(CONFIG_THP_SWAP) &&
+ folio_test_swapcache(folio)))
flags |= TTU_SPLIT_HUGE_PMD;
/*
* Without TTU_SYNC, try_to_unmap will only begin to
diff --git a/mm/vmstat.c b/mm/vmstat.c
index a3e809c57f295..5badcce8ff0ad 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1435,6 +1435,7 @@ const char * const vmstat_text[] = {
[I(THP_ZERO_PAGE_ALLOC_FAILED)] = "thp_zero_page_alloc_failed",
[I(THP_SWPOUT)] = "thp_swpout",
[I(THP_SWPOUT_FALLBACK)] = "thp_swpout_fallback",
+ [I(THP_SWPOUT_PMD)] = "thp_swpout_pmd",
#endif
#ifdef CONFIG_BALLOON
[I(BALLOON_INFLATE)] = "balloon_inflate",
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* [RESEND v7 29/29] selftests/mm: add PMD swap entry tests
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (27 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 28/29] mm: install PMD swap entries on swap-out Usama Arif
@ 2026-09-14 12:28 ` Usama Arif
2026-09-15 3:32 ` [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Andrew Morton
29 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-14 12:28 UTC (permalink / raw)
To: Andrew Morton, david, chrisl, kasong, ljs, ziy, linux-mm
Cc: ying.huang, Baoquan He, willy, youngjun.park, hannes, riel,
shakeel.butt, alex, kas, baohua, dev.jain, baolin.wang,
Nico Pache, Liam R. Howlett, ryan.roberts, Vlastimil Babka,
lance.yang, linux-kernel, nphamcs, shikemeng, yosry, qi.zheng,
luizcap, kernel-team, Usama Arif
Each test gets a fresh PMD-mapped THP from fixture setup, fills it with a
pattern that differs between base pages so a reordered split is detectable,
swaps it out with MADV_PAGEOUT, and verifies thp_swpout_pmd increased
before the test body runs.
The tests are basic, swapin_sync, fork, fork_cow, write, rwp_swapin,
munmap, mprotect, split_mprotect, split_munmap, uffdio_move, mremap,
pagemap, mincore, madvise_free, madvise_willneed and swapoff.
MADV_PAGEOUT leaves the folio in the swap cache on an asynchronous device,
so a fault would only remap it and never reach swapin_sync(). Fixture setup
therefore asks the current cgroup to reclaim afterwards, and uses mincore()
- which reports a PMD swap entry as resident exactly while the cache holds
it - to tell whether that worked. Tests run either way; swapin_sync skips
if the cache survived. The swapoff test runs only when PMD_SWAP_DEVICE is
the sole active swap device, and restores it at its original priority.
Skip rather than fail where the kernel never promised anything.
thp_swpout_pmd and thp_swpout_fallback are system-wide, so test the
fallback counter first: if reclaim split any THP we cannot be sure it was
not ours. That also covers a kernel without CONFIG_THP_SWAP. A kernel with
no thp_swpout_pmd counter does not implement the feature at all.
With zswap enabled the range may legitimately come back through the PTE
fallback, so those runs skip the PMD-restoration assertions and say so.
Add a /proc/vmstat field reader to vm_util.c so other tests can use it too,
and register pmd_swap with run_vmtests.sh and the default runner.
Assisted-by: LLM
Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
tools/testing/selftests/mm/Makefile | 2 +
tools/testing/selftests/mm/ksft_pmd_swap.sh | 4 +
tools/testing/selftests/mm/pmd_swap.c | 989 ++++++++++++++++++++
tools/testing/selftests/mm/run_vmtests.sh | 4 +
tools/testing/selftests/mm/vm_util.c | 24 +
tools/testing/selftests/mm/vm_util.h | 2 +
6 files changed, 1025 insertions(+)
create mode 100755 tools/testing/selftests/mm/ksft_pmd_swap.sh
create mode 100644 tools/testing/selftests/mm/pmd_swap.c
diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index d3e9bd67904aa..a685988a3d569 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -104,6 +104,7 @@ TEST_GEN_FILES += guard-regions
TEST_GEN_FILES += merge
TEST_GEN_FILES += rmap
TEST_GEN_FILES += folio_split_race_test
+TEST_GEN_FILES += pmd_swap
TEST_GEN_FILES += soft-dirty
ifeq ($(ARCH),x86_64)
@@ -162,6 +163,7 @@ TEST_PROGS += ksft_mremap.sh
TEST_PROGS += ksft_pagemap.sh
TEST_PROGS += ksft_pfnmap.sh
TEST_PROGS += ksft_pkey.sh
+TEST_PROGS += ksft_pmd_swap.sh
TEST_PROGS += ksft_process_madv.sh
TEST_PROGS += ksft_process_mrelease.sh
TEST_PROGS += ksft_rmap.sh
diff --git a/tools/testing/selftests/mm/ksft_pmd_swap.sh b/tools/testing/selftests/mm/ksft_pmd_swap.sh
new file mode 100755
index 0000000000000..0f070b4729a89
--- /dev/null
+++ b/tools/testing/selftests/mm/ksft_pmd_swap.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: GPL-2.0
+
+./run_vmtests.sh -t pmd_swap
diff --git a/tools/testing/selftests/mm/pmd_swap.c b/tools/testing/selftests/mm/pmd_swap.c
new file mode 100644
index 0000000000000..91bac9ae5560e
--- /dev/null
+++ b/tools/testing/selftests/mm/pmd_swap.c
@@ -0,0 +1,989 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Test PMD-level swap entries and their users. */
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <stdint.h>
+#include <sys/random.h>
+#include <sys/stat.h>
+#include <sys/swap.h>
+#include <sys/syscall.h>
+#include <sys/ioctl.h>
+#include <poll.h>
+#include <pthread.h>
+#include <linux/userfaultfd.h>
+#include <time.h>
+
+#include "kselftest_harness.h"
+#include "vm_util.h"
+
+#define ZSWAP_ENABLED_PATH "/sys/module/zswap/parameters/enabled"
+
+/* pagemap: bits 0-54 hold the PFN, or type|offset for a swap entry. */
+#define PM_PFRAME_MASK ((1ULL << 55) - 1)
+/* Must match MAX_SWAPFILES_SHIFT in include/linux/swap.h. */
+#define MAX_SWAPFILES_SHIFT 5
+
+static bool check_swapped(int pagemap_fd, char *addr, unsigned long size)
+{
+ unsigned long off;
+
+ for (off = 0; off < size; off += getpagesize())
+ if (!pagemap_is_swapped(pagemap_fd, addr + off))
+ return false;
+ return true;
+}
+
+static bool zswap_enabled(void)
+{
+ char enabled = 0;
+ FILE *f;
+
+ f = fopen(ZSWAP_ENABLED_PATH, "r");
+ if (!f)
+ return false;
+
+ if (fscanf(f, " %c", &enabled) != 1)
+ enabled = 0;
+ fclose(f);
+
+ return enabled == 'Y' || enabled == 'y' || enabled == '1';
+}
+
+static bool swap_available(unsigned long required_bytes)
+{
+ unsigned long required_kb = (required_bytes + 1023) / 1024;
+ unsigned long size_kb, used_kb;
+ char line[256];
+ bool ret = false;
+ FILE *f;
+
+ f = fopen("/proc/swaps", "r");
+ if (!f)
+ return false;
+
+ /* Skip the header. */
+ if (!fgets(line, sizeof(line), f))
+ goto out;
+
+ while (fgets(line, sizeof(line), f)) {
+ if (sscanf(line, "%*s %*s %lu %lu", &size_kb, &used_kb) == 2 &&
+ size_kb >= used_kb && size_kb - used_kb >= required_kb) {
+ ret = true;
+ break;
+ }
+ }
+
+out:
+ fclose(f);
+ return ret;
+}
+
+static bool same_swap_device(const struct stat *a, const struct stat *b)
+{
+ if (S_ISBLK(a->st_mode) && S_ISBLK(b->st_mode))
+ return a->st_rdev == b->st_rdev;
+ return a->st_dev == b->st_dev && a->st_ino == b->st_ino;
+}
+
+/*
+ * Returns true if @swap_dev is the one and only active swap device, and stores
+ * its /proc/swaps priority in *prio so the caller can put it back the way it
+ * found it.
+ */
+static bool swap_device_is_only_active(const char *swap_dev, int *prio)
+{
+ struct stat expected, active;
+ char path[256], line[512];
+ unsigned int nr_active = 0;
+ bool matches = false;
+ FILE *f;
+
+ *prio = -1;
+
+ if (stat(swap_dev, &expected))
+ return false;
+
+ f = fopen("/proc/swaps", "r");
+ if (!f)
+ return false;
+ if (!fgets(line, sizeof(line), f))
+ goto out;
+
+ while (fgets(line, sizeof(line), f)) {
+ int line_prio;
+
+ if (sscanf(line, "%255s %*s %*s %*s %d", path, &line_prio) != 2)
+ continue;
+ nr_active++;
+ if (nr_active > 1)
+ goto out;
+ matches = !stat(path, &active) &&
+ same_swap_device(&expected, &active);
+ if (matches)
+ *prio = line_prio;
+ }
+
+out:
+ fclose(f);
+ return nr_active == 1 && matches;
+}
+
+/* Re-enable a device swapoff()ed by this test, at its original priority. */
+static int swapon_restore(const char *swap_dev, int prio)
+{
+ int flags = 0;
+
+ if (prio >= 0)
+ flags = SWAP_FLAG_PREFER |
+ ((prio << SWAP_FLAG_PRIO_SHIFT) & SWAP_FLAG_PRIO_MASK);
+ return swapon(swap_dev, flags);
+}
+
+/* Locate this task's cgroup-v2 directory. An empty relative path is the root. */
+static bool cgroup2_self_dir(char *buf, size_t len)
+{
+ char mnt[PATH_MAX] = "", type[64], rel[PATH_MAX] = "";
+ char line[2 * PATH_MAX];
+ size_t rel_len;
+ FILE *f;
+
+ f = fopen("/proc/self/mounts", "r");
+ if (!f)
+ return false;
+ while (fgets(line, sizeof(line), f)) {
+ if (sscanf(line, "%*s %4095s %63s", mnt, type) == 2 &&
+ !strcmp(type, "cgroup2"))
+ goto found_mnt;
+ }
+ mnt[0] = '\0';
+found_mnt:
+ fclose(f);
+ if (!mnt[0])
+ return false;
+
+ f = fopen("/proc/self/cgroup", "r");
+ if (!f)
+ return false;
+ while (fgets(line, sizeof(line), f)) {
+ if (strncmp(line, "0::", 3))
+ continue;
+ rel_len = strcspn(line + 3, "\n");
+ if (rel_len && rel_len < sizeof(rel)) {
+ memcpy(rel, line + 3, rel_len);
+ rel[rel_len] = '\0';
+ }
+ break;
+ }
+ fclose(f);
+ if (!rel[0])
+ return false;
+ if (!strcmp(rel, "/"))
+ rel[0] = '\0';
+ return snprintf(buf, len, "%s%s/memory.reclaim", mnt, rel) < (int)len;
+}
+
+static bool cgroup_reclaim(unsigned long bytes)
+{
+ char path[PATH_MAX], val[32];
+ ssize_t written;
+ int fd, len, err;
+
+ if (!cgroup2_self_dir(path, sizeof(path)))
+ return false;
+
+ fd = open(path, O_WRONLY);
+ if (fd < 0)
+ return false;
+ len = snprintf(val, sizeof(val), "%lu", bytes);
+ written = write(fd, val, len);
+ err = errno;
+ close(fd);
+ /* -EAGAIN means it reclaimed something but fell short, which is fine. */
+ return written == len || err == EAGAIN;
+}
+
+/*
+ * mincore() reports 1 over a PMD swap entry iff the swap cache still holds it.
+ * This samples the first slot only: it is a hint used to decide whether the
+ * swap cache was evicted, not an assertion about the whole range.
+ */
+static bool swapcache_resident(char *mem)
+{
+ unsigned char vec[1];
+
+ if (mincore(mem, getpagesize(), vec))
+ return true;
+ return vec[0] & 1;
+}
+
+/*
+ * MADV_PAGEOUT leaves the folio in the swap cache on asynchronous swap
+ * devices, so faulting would just remap the cached folio and never reach
+ * swapin_sync(). Push it out so the PMD-order swap-in path is exercised.
+ */
+static bool drop_swapcache(char *mem, unsigned long pmd_size)
+{
+ unsigned long want = pmd_size;
+ int i;
+
+ for (i = 0; i < 3 && swapcache_resident(mem); i++, want *= 4)
+ if (!cgroup_reclaim(want))
+ break;
+ return !swapcache_resident(mem);
+}
+
+static unsigned int random_seed(void)
+{
+ unsigned int seed;
+
+ if (getrandom(&seed, sizeof(seed), 0) != sizeof(seed))
+ seed = (unsigned int)time(NULL);
+ return seed;
+}
+
+static unsigned long test_page_size(void)
+{
+ static unsigned long page_size;
+
+ if (!page_size)
+ page_size = getpagesize();
+ return page_size;
+}
+
+/*
+ * Two base pages of the same PMD must never hold identical bytes, or the split
+ * tests cannot tell that the slots came back in the wrong order. A single byte
+ * cannot encode a page index on its own - HPAGE_PMD_NR is 8192 on arm64 with
+ * 64K pages - so spell the index out in the first two bytes of every page.
+ */
+static unsigned char pattern_byte(unsigned int seed, unsigned long off)
+{
+ unsigned long page_size = test_page_size();
+ unsigned long idx = off & (page_size - 1);
+
+ if (idx < 2)
+ return (unsigned char)(seed + ((off / page_size) >> (idx * 8)));
+
+ return (unsigned char)(seed + off + (off >> 8) + (off >> 16));
+}
+
+static void fill_pattern(char *buf, unsigned long size, unsigned int seed)
+{
+ unsigned long i;
+
+ for (i = 0; i < size; i++)
+ buf[i] = (char)pattern_byte(seed, i);
+}
+
+static bool verify_pattern_range(char *buf, unsigned long size,
+ unsigned int seed, unsigned long offset)
+{
+ unsigned long i;
+
+ for (i = 0; i < size; i++)
+ if ((unsigned char)buf[i] != pattern_byte(seed, offset + i))
+ return false;
+ return true;
+}
+
+static bool verify_pattern(char *buf, unsigned long size, unsigned int seed)
+{
+ return verify_pattern_range(buf, size, seed, 0);
+}
+
+static bool verify_zero(char *buf, unsigned long size)
+{
+ unsigned long i;
+
+ for (i = 0; i < size; i++)
+ if (buf[i])
+ return false;
+ return true;
+}
+
+/*
+ * mmap an anonymous PMD-aligned region of pmd_size bytes. Over-allocates
+ * by one PMD and trims the unaligned head/tail so the returned address is
+ * PMD-aligned (required for whole-PMD UFFDIO_MOVE).
+ */
+static char *mmap_pmd_aligned(unsigned long pmd_size)
+{
+ unsigned long pad = pmd_size;
+ char *raw, *aligned;
+
+ raw = mmap(NULL, pmd_size + pad, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (raw == MAP_FAILED)
+ return MAP_FAILED;
+
+ aligned = (char *)(((uintptr_t)raw + pmd_size - 1) & ~(pmd_size - 1));
+ if (aligned != raw)
+ munmap(raw, aligned - raw);
+ if (aligned + pmd_size != raw + pmd_size + pad)
+ munmap(aligned + pmd_size,
+ (raw + pmd_size + pad) - (aligned + pmd_size));
+ return aligned;
+}
+
+/* Per-process swapped size in bytes, from /proc/self/status VmSwap. */
+static unsigned long read_vmswap(void)
+{
+ char line[256];
+ unsigned long kb = 0;
+ FILE *f;
+
+ f = fopen("/proc/self/status", "r");
+ if (!f)
+ return 0;
+ while (fgets(line, sizeof(line), f)) {
+ if (!strncmp(line, "VmSwap:", 7)) {
+ kb = strtoul(line + 7, NULL, 10);
+ break;
+ }
+ }
+ fclose(f);
+ return kb * 1024;
+}
+
+/*
+ * Swap the PMD range out. Returns true if a PMD swap entry was installed.
+ * On failure *swap_failed is set unless this environment simply cannot swap
+ * at PMD granularity, in which case the caller should skip rather than fail.
+ */
+static bool swap_out_pmd(char *mem, unsigned long pmd_size, int pagemap_fd,
+ bool *swap_failed)
+{
+ long pmd_before = read_vmstat("thp_swpout_pmd");
+ long fallback_before = read_vmstat("thp_swpout_fallback");
+ long pmd_after, fallback_after;
+ bool swapped;
+
+ /*
+ * A kernel without PMD swap entry support has no thp_swpout_pmd at
+ * all. Skip rather than report a failure the kernel never promised.
+ */
+ if (pmd_before < 0 || fallback_before < 0) {
+ ksft_print_msg("no thp_swpout_pmd counter; PMD swap entries unsupported\n");
+ return false;
+ }
+
+ if (madvise(mem, pmd_size, MADV_PAGEOUT)) {
+ ksft_print_msg("MADV_PAGEOUT failed: %s\n", strerror(errno));
+ *swap_failed = true;
+ return false;
+ }
+
+ swapped = check_swapped(pagemap_fd, mem, pmd_size);
+ pmd_after = read_vmstat("thp_swpout_pmd");
+ fallback_after = read_vmstat("thp_swpout_fallback");
+ ksft_print_msg("thp_swpout_pmd: %ld -> %ld, fallback: %ld -> %ld\n",
+ pmd_before, pmd_after, fallback_before, fallback_after);
+
+ if (!swapped) {
+ ksft_print_msg("MADV_PAGEOUT did not swap the whole PMD range\n");
+ *swap_failed = true;
+ return false;
+ }
+ /*
+ * Both counters are system-wide, so another task swapping a THP during
+ * the window above can move either of them. Test the fallback counter
+ * first: if reclaim split *any* THP we cannot be sure ours was not the
+ * one, and skipping is the safe direction. That also covers a kernel
+ * built without CONFIG_THP_SWAP, where folio_alloc_swap() returns
+ * -E2BIG for every PMD-order folio.
+ */
+ if (fallback_after > fallback_before) {
+ ksft_print_msg("PMD swap unavailable; reclaim used PTE fallback\n");
+ return false;
+ }
+ if (pmd_after > pmd_before)
+ return true;
+
+ *swap_failed = true;
+ return false;
+}
+
+static char *alloc_fill_swap_thp(unsigned long pmd_size, int pagemap_fd,
+ unsigned int seed, bool *swap_failed)
+{
+ char *mem;
+
+ *swap_failed = false;
+
+ mem = mmap_pmd_aligned(pmd_size);
+ if (mem == MAP_FAILED)
+ return MAP_FAILED;
+
+ if (madvise(mem, pmd_size, MADV_HUGEPAGE)) {
+ ksft_print_msg("MADV_HUGEPAGE failed: %s\n", strerror(errno));
+ munmap(mem, pmd_size);
+ return MAP_FAILED;
+ }
+ fill_pattern(mem, pmd_size, seed);
+
+ if (!check_huge_anon(mem, pmd_size, 1, pmd_size)) {
+ munmap(mem, pmd_size);
+ return MAP_FAILED;
+ }
+ if (!swap_out_pmd(mem, pmd_size, pagemap_fd, swap_failed)) {
+ munmap(mem, pmd_size);
+ return MAP_FAILED;
+ }
+
+ return mem;
+}
+
+struct rwp_access_args {
+ unsigned char *addr;
+ unsigned char expected;
+ bool write;
+ bool ok;
+};
+
+static void *rwp_access_thread(void *data)
+{
+ struct rwp_access_args *args = data;
+
+ if (args->write)
+ *args->addr = args->expected;
+ args->ok = *args->addr == args->expected;
+ return NULL;
+}
+
+static int register_rwp(char *addr, unsigned long size, bool protect)
+{
+ struct uffdio_register reg = {};
+ struct uffdio_rwprotect rwp = {};
+ struct uffdio_api api = {};
+ int uffd;
+
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd < 0)
+ return -1;
+
+ api.api = UFFD_API;
+ api.features = UFFD_FEATURE_RWP;
+ if (ioctl(uffd, UFFDIO_API, &api) ||
+ !(api.features & UFFD_FEATURE_RWP))
+ goto error;
+
+ reg.range.start = (unsigned long)addr;
+ reg.range.len = size;
+ reg.mode = UFFDIO_REGISTER_MODE_RWP;
+ if (ioctl(uffd, UFFDIO_REGISTER, ®))
+ goto error;
+
+ if (!protect)
+ return uffd;
+
+ rwp.range.start = (unsigned long)addr;
+ rwp.range.len = size;
+ rwp.mode = UFFDIO_RWPROTECT_MODE_RWP;
+ if (!ioctl(uffd, UFFDIO_RWPROTECT, &rwp))
+ return uffd;
+
+error:
+ close(uffd);
+ return -1;
+}
+
+static bool expect_rwp_fault(int uffd, char *addr, unsigned long size,
+ unsigned char expected, bool write)
+{
+ struct rwp_access_args args = {
+ .addr = (unsigned char *)addr,
+ .expected = expected,
+ .write = write,
+ };
+ struct uffdio_rwprotect rwp = {
+ .range = {
+ .start = (unsigned long)addr,
+ .len = size,
+ },
+ };
+ struct pollfd pollfd = {
+ .fd = uffd,
+ .events = POLLIN,
+ };
+ struct uffd_msg msg = {};
+ pthread_t thread;
+ bool saw_rwp = false;
+ int ret;
+
+ if (pthread_create(&thread, NULL, rwp_access_thread, &args))
+ return false;
+
+ ret = poll(&pollfd, 1, 5000);
+ if (ret == 1 && (pollfd.revents & POLLIN) &&
+ read(uffd, &msg, sizeof(msg)) == (ssize_t)sizeof(msg)) {
+ saw_rwp = msg.event == UFFD_EVENT_PAGEFAULT &&
+ (msg.arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_RWP);
+ }
+
+ /* Resolve the access even on failure so the worker cannot remain blocked. */
+ ioctl(uffd, UFFDIO_RWPROTECT, &rwp);
+ if (pthread_join(thread, NULL))
+ return false;
+ return saw_rwp && args.ok;
+}
+
+FIXTURE(pmd_swap)
+{
+ unsigned long pmd_size;
+ unsigned long mem_len;
+ int pagemap_fd;
+ int uffd;
+ unsigned int seed;
+ bool zswap_enabled;
+ bool swap_disabled;
+ int swap_prio;
+ bool swapcache_dropped;
+ const char *swap_dev;
+ char *mem;
+ char *aux;
+};
+
+FIXTURE_SETUP(pmd_swap)
+{
+ bool swap_failed;
+
+ self->pagemap_fd = -1;
+ self->uffd = -1;
+ self->mem = MAP_FAILED;
+ self->aux = MAP_FAILED;
+ self->mem_len = 0;
+ self->swap_disabled = false;
+ self->swap_prio = -1;
+ self->swapcache_dropped = false;
+ self->swap_dev = NULL;
+
+ self->pmd_size = read_pmd_pagesize();
+ if (!self->pmd_size)
+ SKIP(return, "Cannot determine PMD size\n");
+
+ self->pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+ if (self->pagemap_fd < 0)
+ SKIP(return, "Cannot open /proc/self/pagemap\n");
+
+ if (!swap_available(self->pmd_size))
+ SKIP(return, "No active swap device has enough free space\n");
+
+ self->seed = random_seed();
+ self->zswap_enabled = zswap_enabled();
+ self->mem = alloc_fill_swap_thp(self->pmd_size, self->pagemap_fd,
+ self->seed, &swap_failed);
+ if (self->mem == MAP_FAILED) {
+ ASSERT_FALSE(swap_failed);
+ SKIP(return, "Could not create swapped THP\n");
+ }
+ self->mem_len = self->pmd_size;
+ self->swapcache_dropped = drop_swapcache(self->mem, self->pmd_size);
+ ksft_print_msg("swap cache %s the swapped-out THP\n",
+ self->swapcache_dropped ? "no longer holds"
+ : "still holds");
+}
+
+FIXTURE_TEARDOWN(pmd_swap)
+{
+ int swap_err = 0;
+ int swap_ret = 0;
+
+ if (self->swap_disabled) {
+ swap_ret = swapon_restore(self->swap_dev, self->swap_prio);
+ swap_err = errno;
+ }
+ if (self->uffd >= 0)
+ close(self->uffd);
+ if (self->aux != MAP_FAILED)
+ munmap(self->aux, self->pmd_size);
+ if (self->mem != MAP_FAILED)
+ munmap(self->mem, self->mem_len);
+ if (self->pagemap_fd >= 0)
+ close(self->pagemap_fd);
+
+ EXPECT_EQ(swap_ret, 0) {
+ TH_LOG("swapon(%s) failed: %s", self->swap_dev,
+ strerror(swap_err));
+ }
+}
+
+TEST_F(pmd_swap, basic)
+{
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+}
+
+/*
+ * With the swap cache evicted, the fault cannot be served by remapping a
+ * cached folio, so this covers the PMD-order swapin_sync() read.
+ */
+TEST_F(pmd_swap, swapin_sync)
+{
+ if (!self->swapcache_dropped)
+ SKIP(return, "Could not evict the folio from the swap cache\n");
+
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+ if (self->zswap_enabled)
+ ksft_print_msg("zswap enabled: PMD restoration not checked\n");
+ else
+ ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1,
+ self->pmd_size));
+}
+
+TEST_F(pmd_swap, fork)
+{
+ pid_t pid;
+ int status;
+
+ pid = fork();
+ ASSERT_GE(pid, 0);
+
+ if (pid == 0)
+ _exit(verify_pattern(self->mem, self->pmd_size,
+ self->seed) ? 0 : 1);
+
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+
+ ASSERT_EQ(waitpid(pid, &status, 0), pid);
+ ASSERT_TRUE(WIFEXITED(status));
+ ASSERT_EQ(WEXITSTATUS(status), 0);
+}
+
+TEST_F(pmd_swap, fork_cow)
+{
+ unsigned int parent_seed = self->seed;
+ unsigned int child_seed = ~self->seed;
+ unsigned int new_seed = self->seed ^ 0xa5a5a5a5;
+ int release_child[2];
+ bool parent_ok;
+ char c = 0;
+ pid_t pid;
+ int status, ret;
+
+ ASSERT_EQ(pipe(release_child), 0);
+
+ pid = fork();
+ ASSERT_GE(pid, 0);
+
+ if (pid == 0) {
+ close(release_child[1]);
+ if (read(release_child[0], &c, 1) != 1)
+ _exit(1);
+ if (!verify_pattern(self->mem, self->pmd_size, parent_seed))
+ _exit(2);
+ fill_pattern(self->mem, self->pmd_size, child_seed);
+ if (!verify_pattern(self->mem, self->pmd_size, child_seed))
+ _exit(3);
+ _exit(0);
+ }
+
+ close(release_child[0]);
+ fill_pattern(self->mem, self->pmd_size, new_seed);
+ parent_ok = verify_pattern(self->mem, self->pmd_size, new_seed);
+ ret = write(release_child[1], &c, 1);
+ close(release_child[1]);
+ ASSERT_EQ(waitpid(pid, &status, 0), pid);
+ ASSERT_EQ(ret, 1);
+ ASSERT_TRUE(parent_ok);
+ ASSERT_TRUE(WIFEXITED(status));
+ ASSERT_EQ(WEXITSTATUS(status), 0);
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, new_seed));
+}
+
+TEST_F(pmd_swap, write)
+{
+ self->mem[0] = 0xbb;
+ ASSERT_EQ(self->mem[0], (char)0xbb);
+ ASSERT_TRUE(verify_pattern_range(self->mem + 1, self->pmd_size - 1,
+ self->seed, 1));
+ if (self->zswap_enabled)
+ ksft_print_msg("zswap enabled: PMD restoration not checked\n");
+ else
+ ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1,
+ self->pmd_size));
+}
+
+TEST_F(pmd_swap, rwp_swapin)
+{
+ self->uffd = register_rwp(self->mem, self->pmd_size, true);
+ if (self->uffd < 0)
+ SKIP(return, "Userfaultfd RWP unsupported\n");
+
+ ASSERT_TRUE(expect_rwp_fault(self->uffd, self->mem, self->pmd_size,
+ pattern_byte(self->seed, 0), false));
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+}
+
+TEST_F(pmd_swap, munmap)
+{
+ unsigned long swap_before, swap_after;
+ int ret;
+
+ swap_before = read_vmswap();
+ ASSERT_GE(swap_before, self->pmd_size);
+
+ ret = munmap(self->mem, self->pmd_size);
+ if (!ret) {
+ self->mem = MAP_FAILED;
+ self->mem_len = 0;
+ }
+ ASSERT_EQ(ret, 0);
+
+ swap_after = read_vmswap();
+ ASSERT_LE(swap_after, swap_before - self->pmd_size);
+}
+
+TEST_F(pmd_swap, mprotect)
+{
+ ASSERT_EQ(mprotect(self->mem, self->pmd_size, PROT_READ), 0);
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem,
+ self->pmd_size));
+ ASSERT_EQ(mprotect(self->mem, self->pmd_size,
+ PROT_READ | PROT_WRITE), 0);
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem,
+ self->pmd_size));
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+}
+
+TEST_F(pmd_swap, split_mprotect)
+{
+ unsigned long half = self->pmd_size / 2;
+
+ ASSERT_EQ(mprotect(self->mem, half, PROT_READ), 0);
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem,
+ self->pmd_size));
+ ASSERT_EQ(mprotect(self->mem, half, PROT_READ | PROT_WRITE), 0);
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+}
+
+TEST_F(pmd_swap, split_munmap)
+{
+ unsigned long half = self->pmd_size / 2;
+ unsigned long swap_before = read_vmswap();
+ unsigned long i;
+ char *base = self->mem;
+ int ret;
+
+ ASSERT_GE(swap_before, half);
+ ret = munmap(base, half);
+ if (!ret) {
+ self->mem = base + half;
+ self->mem_len = half;
+ }
+ ASSERT_EQ(ret, 0);
+ ASSERT_LE(read_vmswap(), swap_before - half);
+
+ for (i = 0; i < half; i += getpagesize())
+ ASSERT_TRUE(pagemap_is_swapped(self->pagemap_fd,
+ self->mem + i));
+ ASSERT_TRUE(verify_pattern_range(self->mem, half, self->seed, half));
+}
+
+TEST_F(pmd_swap, uffdio_move)
+{
+ struct uffdio_register reg = {};
+ struct uffdio_move move = {};
+ struct uffdio_api api = {};
+ bool rwp;
+
+ self->aux = mmap_pmd_aligned(self->pmd_size);
+ if (self->aux == MAP_FAILED)
+ SKIP(return, "Could not mmap aligned dst\n");
+ ASSERT_EQ(madvise(self->aux, self->pmd_size, MADV_HUGEPAGE), 0);
+
+ self->uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (self->uffd < 0)
+ SKIP(return, "userfaultfd unavailable\n");
+
+ api.api = UFFD_API;
+ api.features = UFFD_FEATURE_MOVE | UFFD_FEATURE_RWP;
+ if (ioctl(self->uffd, UFFDIO_API, &api) ||
+ !(api.features & UFFD_FEATURE_MOVE))
+ SKIP(return, "UFFD_FEATURE_MOVE unsupported\n");
+ rwp = api.features & UFFD_FEATURE_RWP;
+
+ reg.range.start = (unsigned long)self->aux;
+ reg.range.len = self->pmd_size;
+ reg.mode = UFFDIO_REGISTER_MODE_MISSING |
+ (rwp ? UFFDIO_REGISTER_MODE_RWP : 0);
+ ASSERT_EQ(ioctl(self->uffd, UFFDIO_REGISTER, ®), 0);
+
+ move.dst = (unsigned long)self->aux;
+ move.src = (unsigned long)self->mem;
+ move.len = self->pmd_size;
+ ASSERT_EQ(ioctl(self->uffd, UFFDIO_MOVE, &move), 0);
+ ASSERT_EQ(move.move, self->pmd_size);
+
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, self->aux,
+ self->pmd_size));
+ if (rwp)
+ ASSERT_TRUE(expect_rwp_fault(self->uffd, self->aux,
+ self->pmd_size,
+ pattern_byte(self->seed, 0), false));
+ ASSERT_TRUE(verify_pattern(self->aux, self->pmd_size, self->seed));
+ if (self->zswap_enabled)
+ ksft_print_msg("zswap enabled: PMD restoration not checked\n");
+ else
+ ASSERT_TRUE(check_huge_anon(self->aux, self->pmd_size, 1,
+ self->pmd_size));
+}
+
+TEST_F(pmd_swap, mremap)
+{
+ char *new_mem, *dst;
+
+ self->aux = mmap_pmd_aligned(self->pmd_size);
+ if (self->aux == MAP_FAILED)
+ SKIP(return, "Could not mmap aligned dst\n");
+ dst = self->aux;
+
+ new_mem = mremap(self->mem, self->pmd_size, self->pmd_size,
+ MREMAP_MAYMOVE | MREMAP_FIXED, dst);
+ if (new_mem != MAP_FAILED) {
+ self->mem = new_mem;
+ self->aux = MAP_FAILED;
+ }
+ ASSERT_NE(new_mem, MAP_FAILED);
+ ASSERT_EQ(new_mem, dst);
+
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, new_mem, self->pmd_size));
+ ASSERT_TRUE(verify_pattern(new_mem, self->pmd_size, self->seed));
+}
+
+TEST_F(pmd_swap, pagemap)
+{
+ uint64_t entry, first;
+ unsigned long off;
+
+ entry = pagemap_get_entry(self->pagemap_fd, self->mem);
+ ASSERT_TRUE(entry & (1ULL << 62));
+ ASSERT_FALSE(entry & (1ULL << 63));
+ first = entry & PM_PFRAME_MASK;
+
+ /*
+ * The kernel zeroes the swap type/offset payload for readers without
+ * CAP_SYS_ADMIN, so the slot-ordering check below would silently test
+ * nothing. Skip instead of passing vacuously.
+ */
+ if (!first)
+ SKIP(return, "pagemap swap offsets need CAP_SYS_ADMIN\n");
+
+ for (off = getpagesize(); off < self->pmd_size; off += getpagesize()) {
+ uint64_t idx = off / getpagesize();
+
+ entry = pagemap_get_entry(self->pagemap_fd, self->mem + off);
+ ASSERT_TRUE(entry & (1ULL << 62));
+ ASSERT_FALSE(entry & (1ULL << 63));
+ ASSERT_EQ(entry & PM_PFRAME_MASK,
+ first + (idx << MAX_SWAPFILES_SHIFT));
+ }
+}
+
+TEST_F(pmd_swap, mincore)
+{
+ unsigned long pages = self->pmd_size / getpagesize();
+ unsigned char vec[pages];
+ unsigned long i;
+
+ ASSERT_EQ(mincore(self->mem, self->pmd_size, vec), 0);
+ /*
+ * Nothing in this test faults the range, and the fixture already
+ * evicted the swap cache, so every slot must report not-resident. The
+ * other direction is not stable - reclaim can drop the cached folio
+ * between fixture setup and here, and a split cache reports per slot -
+ * so only check that mincore() answered.
+ */
+ if (self->swapcache_dropped) {
+ for (i = 0; i < pages; i++)
+ ASSERT_EQ(vec[i] & 1, 0);
+ }
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem,
+ self->pmd_size));
+}
+
+TEST_F(pmd_swap, madvise_free)
+{
+ unsigned long swap_before = read_vmswap();
+ unsigned long i;
+
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem,
+ self->pmd_size));
+ ASSERT_GE(swap_before, self->pmd_size);
+ ASSERT_EQ(madvise(self->mem, self->pmd_size, MADV_FREE), 0);
+ for (i = 0; i < self->pmd_size; i += getpagesize())
+ ASSERT_FALSE(pagemap_is_swapped(self->pagemap_fd,
+ self->mem + i));
+ ASSERT_LE(read_vmswap(), swap_before - self->pmd_size);
+ ASSERT_TRUE(verify_zero(self->mem, self->pmd_size));
+}
+
+TEST_F(pmd_swap, madvise_willneed)
+{
+ ASSERT_EQ(madvise(self->mem, self->pmd_size, MADV_WILLNEED), 0);
+ ASSERT_TRUE(check_swapped(self->pagemap_fd, self->mem,
+ self->pmd_size));
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+ if (self->zswap_enabled)
+ ksft_print_msg("zswap enabled: PMD restoration not checked\n");
+ else
+ ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1,
+ self->pmd_size));
+}
+
+TEST_F(pmd_swap, swapoff)
+{
+ int ret, err;
+
+ self->swap_dev = getenv("PMD_SWAP_DEVICE");
+ if (!self->swap_dev)
+ SKIP(return, "PMD_SWAP_DEVICE env var not set\n");
+ /*
+ * Otherwise a higher-priority device may have taken the PMD swap
+ * entry and swapoff() would operate on the wrong backend.
+ */
+ if (!swap_device_is_only_active(self->swap_dev, &self->swap_prio))
+ SKIP(return, "PMD_SWAP_DEVICE must be the only active swap device\n");
+
+ self->uffd = register_rwp(self->mem, self->pmd_size, true);
+
+ ret = swapoff(self->swap_dev);
+ err = errno;
+ if (!ret)
+ self->swap_disabled = true;
+ ASSERT_EQ(ret, 0) {
+ TH_LOG("swapoff(%s) failed: %s", self->swap_dev, strerror(err));
+ }
+
+ /*
+ * Check residency before touching the memory. If we read
+ * first, a bug that left a PMD swap entry in place after swapoff
+ * would silently trigger do_huge_pmd_swap_page() and reinstall a
+ * PMD mapping, masking the regression.
+ */
+ if (self->zswap_enabled)
+ ksft_print_msg("zswap enabled: PMD restoration not checked\n");
+ else
+ ASSERT_TRUE(check_huge_anon(self->mem, self->pmd_size, 1,
+ self->pmd_size));
+ if (self->uffd >= 0)
+ ASSERT_TRUE(expect_rwp_fault(self->uffd, self->mem,
+ self->pmd_size,
+ pattern_byte(self->seed, 0), false));
+ ASSERT_TRUE(verify_pattern(self->mem, self->pmd_size, self->seed));
+
+ ret = swapon_restore(self->swap_dev, self->swap_prio);
+ err = errno;
+ if (!ret)
+ self->swap_disabled = false;
+ ASSERT_EQ(ret, 0) {
+ TH_LOG("swapon(%s) failed: %s", self->swap_dev, strerror(err));
+ }
+}
+
+TEST_HARNESS_MAIN
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 19755fb4edb94..720ba5cbacd62 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -69,6 +69,8 @@ separated by spaces:
test pagemap_scan IOCTL
- pfnmap
tests for VM_PFNMAP handling
+- pmd_swap
+ tests for PMD-level swap entries
- process_madv
test for process_madv
- cow
@@ -414,6 +416,8 @@ CATEGORY="pagemap" run_test ./pagemap_ioctl
CATEGORY="pfnmap" run_test ./pfnmap
+CATEGORY="pmd_swap" run_test ./pmd_swap
+
# COW tests
CATEGORY="cow" run_test ./cow
diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests/mm/vm_util.c
index 80bc9f597b521..e6133784abad2 100644
--- a/tools/testing/selftests/mm/vm_util.c
+++ b/tools/testing/selftests/mm/vm_util.c
@@ -180,6 +180,30 @@ uint64_t read_pmd_pagesize(void)
return strtoul(buf, NULL, 10);
}
+long read_vmstat(const char *name)
+{
+ char line[MAX_LINE_LENGTH];
+ size_t name_len = strlen(name);
+ long val = -1;
+ FILE *f;
+
+ f = fopen("/proc/vmstat", "r");
+ if (!f)
+ return -1;
+ /*
+ * Match the whole field name: "thp_swpout" is a prefix of
+ * "thp_swpout_fallback" and "thp_swpout_pmd".
+ */
+ while (fgets(line, sizeof(line), f)) {
+ if (!strncmp(line, name, name_len) && line[name_len] == ' ') {
+ val = strtol(line + name_len + 1, NULL, 10);
+ break;
+ }
+ }
+ fclose(f);
+ return val;
+}
+
unsigned long rss_anon(void)
{
unsigned long rss_anon = 0;
diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests/mm/vm_util.h
index 9a49af88702e4..6355d830cd4a8 100644
--- a/tools/testing/selftests/mm/vm_util.h
+++ b/tools/testing/selftests/mm/vm_util.h
@@ -89,6 +89,8 @@ unsigned long pagemap_get_pfn(int fd, char *start);
void clear_softdirty(void);
bool check_for_pattern(FILE *fp, const char *pattern, char *buf, size_t len);
uint64_t read_pmd_pagesize(void);
+/* Returns the counter value, or -1 if /proc/vmstat has no such field. */
+long read_vmstat(const char *name);
unsigned long rss_anon(void);
bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage_size);
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs
2026-09-14 12:27 [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Usama Arif
` (28 preceding siblings ...)
2026-09-14 12:28 ` [RESEND v7 29/29] selftests/mm: add PMD swap entry tests Usama Arif
@ 2026-09-15 3:32 ` Andrew Morton
2026-09-15 14:09 ` Usama Arif
29 siblings, 1 reply; 35+ messages in thread
From: Andrew Morton @ 2026-09-15 3:32 UTC (permalink / raw)
To: Usama Arif
Cc: david, chrisl, kasong, ljs, ziy, linux-mm, ying.huang,
Baoquan He, willy, youngjun.park, hannes, riel, shakeel.butt,
alex, kas, baohua, dev.jain, baolin.wang, Nico Pache,
Liam R. Howlett, ryan.roberts, Vlastimil Babka, lance.yang,
linux-kernel, nphamcs, shikemeng, yosry, qi.zheng, luizcap,
kernel-team
On Mon, 14 Sep 2026 05:27:50 -0700 Usama Arif <usama.arif@linux.dev> wrote:
> When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is
> split into HPAGE_PMD_NR PTE-level swap entries via TTU_SPLIT_HUGE_PMD
> before unmap. This series introduces a PMD-level swap entry so the
> huge mapping can survive the swap round-trip and do_huge_pmd_swap_page()
> can restore the PMD mapping directly on swap-in, without waiting for
> khugepaged to collapse the range later.
Thanks. I'd describe this as an inevitable step along our thp journey.
Has to be done sometime.
I'll await more review, especially of the core parts: pmd swap fault
handler (#27) and swapout producer (#28).
And it's a 29-patch performance optimization without any performance
testing results ;) Is efficacy testing in the works?
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs
2026-09-15 3:32 ` [RESEND v7 00/29] mm: PMD-level swap entries for anonymous THPs Andrew Morton
@ 2026-09-15 14:09 ` Usama Arif
0 siblings, 0 replies; 35+ messages in thread
From: Usama Arif @ 2026-09-15 14:09 UTC (permalink / raw)
To: Andrew Morton
Cc: david, chrisl, kasong, ljs, ziy, linux-mm, ying.huang,
Baoquan He, willy, youngjun.park, hannes, riel, shakeel.butt,
alex, kas, baohua, dev.jain, baolin.wang, Nico Pache,
Liam R. Howlett, ryan.roberts, Vlastimil Babka, lance.yang,
linux-kernel, nphamcs, shikemeng, yosry, qi.zheng, luizcap,
kernel-team
On 15/09/2026 04:32, Andrew Morton wrote:
> On Mon, 14 Sep 2026 05:27:50 -0700 Usama Arif <usama.arif@linux.dev> wrote:
>
>> When reclaim swaps out a PMD-mapped anonymous THP today, the PMD is
>> split into HPAGE_PMD_NR PTE-level swap entries via TTU_SPLIT_HUGE_PMD
>> before unmap. This series introduces a PMD-level swap entry so the
>> huge mapping can survive the swap round-trip and do_huge_pmd_swap_page()
>> can restore the PMD mapping directly on swap-in, without waiting for
>> khugepaged to collapse the range later.
>
> Thanks. I'd describe this as an inevitable step along our thp journey.
> Has to be done sometime.
>
> I'll await more review, especially of the core parts: pmd swap fault
> handler (#27) and swapout producer (#28).
>
> And it's a 29-patch performance optimization without any performance
> testing results ;) Is efficacy testing in the works?
I previously tested an older revision and reran the test this morning
with the current revision. I used vm-scalability's case-swap-w-seq
benchmark [1].
The test configuration was:
- 4 vCPUs and 4 GiB RAM
- 4 pinned workers
- 8 GiB NOCOW raw virtio swap, cache=none,aio=native
- zswap disabled; THP allocation and defrag set to always
- 6 GiB total anonymous working set
- 8 sequential write passes
Four workers repeatedly write the working set, forcing memory to be
swapped out and back in. The results below are the medians of five
interleaved runs per kernel, following one warm-up run:
Metric Baseline median Patched median Change
Aggregate benchmark throughput 584.3 MiB/s 2,408.9 MiB/s +312.2% (4.12x)
Elapsed time 85.43 s 20.72 s -75.7%
Major faults 1,814,699 228,466 -87.4%
Swap I/O rate 1.02 GiB/s 4.10 GiB/s +303.9%
The results look good as this is a swap-intensive workload.
It primarily demonstrates the reduction in swap-fault and page-table
overhead from preserving PMD mappings.
For long-running workloads, where I believe the series will shine, I expect
benefits from fewer TLB misses, less khugepaged work as well as less kernel churn
due to larger folios, although there aren't a lot of benchmarks to test that.
PMD swap entries also move us closer to eliminating page-table deposits for
anonymous THPs, which would provide memory savings.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swap-w-seq
^ permalink raw reply [flat|nested] 35+ messages in thread