[PATCH V2 0/4] arm64/mm: Convert pgtable READ_ONCE() as pxdp_get()

[PATCH V2 0/4] arm64/mm: Convert pgtable READ_ONCE() as pxdp_get()

[Anshuman Khandual]

Route READ_ONCE() accesses on pgtable entries via level specific generic MM
pxdp_get() helpers which platforms could then override when required. These
accessors on arm64, will help in ensuring page table accesses are performed
in an atomic manner while reading 128 bit page table entries later. But for
now they just default to READ_ONCE(), thus preserving existing behaviour on
current D64 translation a well.

This is similar to generic MM where all atomic accesses have already been
converted from READ_ONCE() into pgtable level specific accessors.

These are preparatory patches for D128 enablement, although standing their
own for D64 translation as well. These patches are split from earlier D128
V1 series. There are no functional changes here and besides mm kselftests
do not show any regression.

https://lore.kernel.org/linux-mm/20260729122452.3797443-1-anshuman.khandual@arm.com/

This series applies on v7.3-rc4 but after applying the following patches on
pgtable entry print standardization posted earlier.

https://lore.kernel.org/linux-mm/20260831054331.625505-1-anshuman.khandual@arm.com/

https://lore.kernel.org/linux-mm/20260916044933.2689426-1-anshuman.khandual@arm.com/

These patches have been hosted here as well.

git@git.gitlab.arm.com:linux-arm/linux-anshuman.git (arm64_pxdp_get_v2)

Changes in V2:

- Rebased on v7.3-rc4 along with all latest dependencies
- Updated commit messages per Ryan
- Added tag from Ryan

Changes in V1:

https://lore.kernel.org/all/20260903060657.2717457-1-anshuman.khandual@arm.com/

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Linu Cherian <linu.cherian@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org

Anshuman Khandual (4):
  arm64/mm: Use pmdp_get() for PMD accesses
  arm64/mm: Use pudp_get() for PUD accesses
  arm64/mm: Use p4dp_get() for P4D accesses
  arm64/mm: Use pgdp_get() for PGD accesses

 arch/arm64/include/asm/pgtable.h |  6 ++--
 arch/arm64/mm/fault.c            |  8 ++---
 arch/arm64/mm/fixmap.c           |  6 ++--
 arch/arm64/mm/hugetlbpage.c      | 10 +++---
 arch/arm64/mm/kasan_init.c       | 18 +++++-----
 arch/arm64/mm/mmu.c              | 60 ++++++++++++++++----------------
 arch/arm64/mm/pageattr.c         |  8 ++---
 arch/arm64/mm/trans_pgd.c        | 14 ++++----
 8 files changed, 66 insertions(+), 64 deletions(-)

-- 
2.43.0

[PATCH V2 1/4] arm64/mm: Use pmdp_get() for PMD accesses

[Anshuman Khandual]

Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for
supporting both D64 and D128 translation table formats.

READ_ONCE() cannot currently be used for 128-bit page table entries on
arm64 because it does not provide the required 128-bit single-copy
atomicity, causing builds to fail for accesses wider than 64 bits.

Although LDP/STP provide the required atomicity when FEAT_LSE is
available (as required by FEAT_D128), extending READ_ONCE() to support
128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
so doing so could encourage other 128-bit users that would either fail
to build in configurations without D128 support or, if D128 becomes a
runtime option, silently permit tearing on systems without the required
hardware support.

Instead, standardize PMD accesses on the existing page-table helpers.
These can be overridden on arm64 to provide 128-bit single-copy
atomicity when required. No functional change intended.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rtland <mark.rtland@arm.com>
Cc: linx-arm-kernel@lists.infradead.org
Cc: linx-kernel@vger.kernel.org
Cc: kasan-dev@googlegrops.com
Reviewed-by: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
 arch/arm64/include/asm/pgtable.h |  3 ++-
 arch/arm64/mm/fault.c            |  2 +-
 arch/arm64/mm/fixmap.c           |  2 +-
 arch/arm64/mm/hugetlbpage.c      |  2 +-
 arch/arm64/mm/kasan_init.c       |  4 ++--
 arch/arm64/mm/mmu.c              | 22 +++++++++++-----------
 arch/arm64/mm/pageattr.c         |  2 +-
 arch/arm64/mm/trans_pgd.c        |  2 +-
 8 files changed, 20 insertions(+), 19 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 763c5a411d64..b953ea0ba3ed 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -811,7 +811,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
 }
 
 /* Find an entry in the third-level page table. */
-#define pte_offset_phys(dir,addr)	(pmd_page_paddr(READ_ONCE(*(dir))) + pte_index(addr) * sizeof(pte_t))
+#define pte_offset_phys(dir, addr)	(pmd_page_paddr(pmdp_get(dir)) + \
+					 pte_index(addr) * sizeof(pte_t))
 
 #define pte_set_fixmap(addr)		((pte_t *)set_fixmap_offset(FIX_PTE, addr))
 #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 2cecf6ba6df7..2757ee0c4300 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -188,7 +188,7 @@ static void show_pte(unsigned long addr)
 			break;
 
 		pmdp = pmd_offset_lockless(pudp, pud, addr);
-		pmd = READ_ONCE(*pmdp);
+		pmd = pmdp_get(pmdp);
 		ptval_to_str(pxd_str, pmd_val(pmd));
 		pr_cont(", pmd=%s", pxd_str);
 		if (pmd_none(pmd) || pmd_bad(pmd))
diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
index f66a0016dd02..3cdac8021d4f 100644
--- a/arch/arm64/mm/fixmap.c
+++ b/arch/arm64/mm/fixmap.c
@@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr)
 
 static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr)
 {
-	pmd_t pmd = READ_ONCE(*pmdp);
+	pmd_t pmd = pmdp_get(pmdp);
 	pte_t *ptep;
 
 	if (pmd_none(pmd)) {
diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
index 8e799c1fe0aa..cdaa4500faf9 100644
--- a/arch/arm64/mm/hugetlbpage.c
+++ b/arch/arm64/mm/hugetlbpage.c
@@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 		addr &= CONT_PMD_MASK;
 
 	pmdp = pmd_offset(pudp, addr);
-	pmd = READ_ONCE(*pmdp);
+	pmd = pmdp_get(pmdp);
 	if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) &&
 	    pmd_none(pmd))
 		return NULL;
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 45fbdce684c8..7ca833c5de5e 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node)
 static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
 				      bool early)
 {
-	if (pmd_none(READ_ONCE(*pmdp))) {
+	if (pmd_none(pmdp_get(pmdp))) {
 		phys_addr_t pte_phys = early ?
 				__pa_symbol(kasan_early_shadow_pte)
 					: kasan_alloc_zeroed_page(node);
@@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
 	do {
 		next = pmd_addr_end(addr, end);
 		kasan_pte_populate(pmdp, addr, next, node, early);
-	} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
+	} while (pmdp++, addr = next, addr != end && pmd_none(pmdp_get(pmdp)));
 }
 
 static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index e589fb00d1e3..95621913679e 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -200,7 +200,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,
 			       int flags)
 {
 	unsigned long next;
-	pmd_t pmd = READ_ONCE(*pmdp);
+	pmd_t pmd = pmdp_get(pmdp);
 	pte_t *ptep;
 
 	BUG_ON(pmd_leaf(pmd));
@@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 	unsigned long next;
 
 	do {
-		pmd_t old_pmd = READ_ONCE(*pmdp);
+		pmd_t old_pmd = pmdp_get(pmdp);
 
 		next = pmd_addr_end(addr, end);
 
@@ -272,7 +272,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 			 * only allow updates to the permission attributes.
 			 */
 			BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd),
-						      READ_ONCE(pmd_val(*pmdp))));
+						      pmd_val(pmdp_get(pmdp))));
 		} else {
 			int ret;
 
@@ -282,7 +282,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 				return ret;
 
 			VM_WARN_ON_ONCE(pmd_val(old_pmd) != 0 &&
-					pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp)));
+					pmd_val(old_pmd) != pmd_val(pmdp_get(pmdp)));
 		}
 		phys += next - addr;
 	} while (pmdp++, addr = next, addr != end);
@@ -293,7 +293,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 static bool pmd_range_has_valid_noncont(pmd_t *pmdp)
 {
 	for (int i = 0; i < CONT_PMDS; i++) {
-		pte_t pte = pmd_pte(READ_ONCE(pmdp[i]));
+		pte_t pte = pmd_pte(pmdp_get(pmdp + i));
 
 		if (pte_valid(pte) && !pte_cont(pte))
 			return true;
@@ -1553,7 +1553,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
 	do {
 		next = pmd_addr_end(addr, end);
 		pmdp = pmd_offset(pudp, addr);
-		pmd = READ_ONCE(*pmdp);
+		pmd = pmdp_get(pmdp);
 		if (pmd_none(pmd))
 			continue;
 
@@ -1708,7 +1708,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
 	do {
 		next = pmd_addr_end(addr, end);
 		pmdp = pmd_offset(pudp, addr);
-		pmd = READ_ONCE(*pmdp);
+		pmd = pmdp_get(pmdp);
 		if (pmd_none(pmd))
 			continue;
 
@@ -1729,7 +1729,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
 	 */
 	pmdp = pmd_offset(pudp, 0UL);
 	for (i = 0; i < PTRS_PER_PMD; i++) {
-		if (!pmd_none(READ_ONCE(pmdp[i])))
+		if (!pmd_none(pmdp_get(pmdp + i)))
 			return;
 	}
 
@@ -1881,7 +1881,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
 	pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot));
 
 	/* Only allow permission changes for now */
-	if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
+	if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)),
 				   pmd_val(new_pmd)))
 		return 0;
 
@@ -1906,7 +1906,7 @@ int pud_clear_huge(pud_t *pudp)
 
 int pmd_clear_huge(pmd_t *pmdp)
 {
-	if (!pmd_leaf(READ_ONCE(*pmdp)))
+	if (!pmd_leaf(pmdp_get(pmdp)))
 		return 0;
 	pmd_clear(pmdp);
 	return 1;
@@ -1917,7 +1917,7 @@ int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 	pte_t *table;
 	pmd_t pmd;
 
-	pmd = READ_ONCE(*pmdp);
+	pmd = pmdp_get(pmdp);
 
 	if (!pmd_table(pmd)) {
 		VM_WARN_ON(1);
diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
index bbe98ac9ad8c..0ca07bd5ded9 100644
--- a/arch/arm64/mm/pageattr.c
+++ b/arch/arm64/mm/pageattr.c
@@ -414,7 +414,7 @@ bool kernel_page_present(struct page *page)
 		return pud_valid(pud);
 
 	pmdp = pmd_offset(pudp, addr);
-	pmd = READ_ONCE(*pmdp);
+	pmd = pmdp_get(pmdp);
 	if (pmd_none(pmd))
 		return false;
 	if (pmd_leaf(pmd))
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index cca9706a875c..b27b2d2c20c3 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -74,7 +74,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
 
 	src_pmdp = pmd_offset(src_pudp, start);
 	do {
-		pmd_t pmd = READ_ONCE(*src_pmdp);
+		pmd_t pmd = pmdp_get(src_pmdp);
 
 		next = pmd_addr_end(addr, end);
 		if (pmd_none(pmd))
-- 
2.43.0

Re: [PATCH V2 1/4] arm64/mm: Use pmdp_get() for PMD accesses

[Ryan Roberts]

On 22/09/2026 07:16, Anshuman Khandual wrote:
> Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for
> supporting both D64 and D128 translation table formats.
> 
> READ_ONCE() cannot currently be used for 128-bit page table entries on
> arm64 because it does not provide the required 128-bit single-copy
> atomicity, causing builds to fail for accesses wider than 64 bits.
> 
> Although LDP/STP provide the required atomicity when FEAT_LSE is
> available (as required by FEAT_D128), extending READ_ONCE() to support
> 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> so doing so could encourage other 128-bit users that would either fail
> to build in configurations without D128 support or, if D128 becomes a
> runtime option, silently permit tearing on systems without the required
> hardware support.
> 
> Instead, standardize PMD accesses on the existing page-table helpers.
> These can be overridden on arm64 to provide 128-bit single-copy
> atomicity when required. No functional change intended.

I notice you have an unconverted READ_ONCE(*pmdp) in
pmdp_test_and_clear_young(). Is that intentional?

Thanks,
Ryan


> 
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Ryan Roberts <ryan.roberts@arm.com>
> Cc: Mark Rtland <mark.rtland@arm.com>
> Cc: linx-arm-kernel@lists.infradead.org
> Cc: linx-kernel@vger.kernel.org
> Cc: kasan-dev@googlegrops.com
> Reviewed-by: Ryan Roberts <ryan.roberts@arm.com>
> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> ---
>  arch/arm64/include/asm/pgtable.h |  3 ++-
>  arch/arm64/mm/fault.c            |  2 +-
>  arch/arm64/mm/fixmap.c           |  2 +-
>  arch/arm64/mm/hugetlbpage.c      |  2 +-
>  arch/arm64/mm/kasan_init.c       |  4 ++--
>  arch/arm64/mm/mmu.c              | 22 +++++++++++-----------
>  arch/arm64/mm/pageattr.c         |  2 +-
>  arch/arm64/mm/trans_pgd.c        |  2 +-
>  8 files changed, 20 insertions(+), 19 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index 763c5a411d64..b953ea0ba3ed 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -811,7 +811,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
>  }
>  
>  /* Find an entry in the third-level page table. */
> -#define pte_offset_phys(dir,addr)	(pmd_page_paddr(READ_ONCE(*(dir))) + pte_index(addr) * sizeof(pte_t))
> +#define pte_offset_phys(dir, addr)	(pmd_page_paddr(pmdp_get(dir)) + \
> +					 pte_index(addr) * sizeof(pte_t))
>  
>  #define pte_set_fixmap(addr)		((pte_t *)set_fixmap_offset(FIX_PTE, addr))
>  #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 2cecf6ba6df7..2757ee0c4300 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -188,7 +188,7 @@ static void show_pte(unsigned long addr)
>  			break;
>  
>  		pmdp = pmd_offset_lockless(pudp, pud, addr);
> -		pmd = READ_ONCE(*pmdp);
> +		pmd = pmdp_get(pmdp);
>  		ptval_to_str(pxd_str, pmd_val(pmd));
>  		pr_cont(", pmd=%s", pxd_str);
>  		if (pmd_none(pmd) || pmd_bad(pmd))
> diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
> index f66a0016dd02..3cdac8021d4f 100644
> --- a/arch/arm64/mm/fixmap.c
> +++ b/arch/arm64/mm/fixmap.c
> @@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr)
>  
>  static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr)
>  {
> -	pmd_t pmd = READ_ONCE(*pmdp);
> +	pmd_t pmd = pmdp_get(pmdp);
>  	pte_t *ptep;
>  
>  	if (pmd_none(pmd)) {
> diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
> index 8e799c1fe0aa..cdaa4500faf9 100644
> --- a/arch/arm64/mm/hugetlbpage.c
> +++ b/arch/arm64/mm/hugetlbpage.c
> @@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
>  		addr &= CONT_PMD_MASK;
>  
>  	pmdp = pmd_offset(pudp, addr);
> -	pmd = READ_ONCE(*pmdp);
> +	pmd = pmdp_get(pmdp);
>  	if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) &&
>  	    pmd_none(pmd))
>  		return NULL;
> diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
> index 45fbdce684c8..7ca833c5de5e 100644
> --- a/arch/arm64/mm/kasan_init.c
> +++ b/arch/arm64/mm/kasan_init.c
> @@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node)
>  static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
>  				      bool early)
>  {
> -	if (pmd_none(READ_ONCE(*pmdp))) {
> +	if (pmd_none(pmdp_get(pmdp))) {
>  		phys_addr_t pte_phys = early ?
>  				__pa_symbol(kasan_early_shadow_pte)
>  					: kasan_alloc_zeroed_page(node);
> @@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
>  	do {
>  		next = pmd_addr_end(addr, end);
>  		kasan_pte_populate(pmdp, addr, next, node, early);
> -	} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
> +	} while (pmdp++, addr = next, addr != end && pmd_none(pmdp_get(pmdp)));
>  }
>  
>  static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index e589fb00d1e3..95621913679e 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -200,7 +200,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,
>  			       int flags)
>  {
>  	unsigned long next;
> -	pmd_t pmd = READ_ONCE(*pmdp);
> +	pmd_t pmd = pmdp_get(pmdp);
>  	pte_t *ptep;
>  
>  	BUG_ON(pmd_leaf(pmd));
> @@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>  	unsigned long next;
>  
>  	do {
> -		pmd_t old_pmd = READ_ONCE(*pmdp);
> +		pmd_t old_pmd = pmdp_get(pmdp);
>  
>  		next = pmd_addr_end(addr, end);
>  
> @@ -272,7 +272,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>  			 * only allow updates to the permission attributes.
>  			 */
>  			BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd),
> -						      READ_ONCE(pmd_val(*pmdp))));
> +						      pmd_val(pmdp_get(pmdp))));
>  		} else {
>  			int ret;
>  
> @@ -282,7 +282,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>  				return ret;
>  
>  			VM_WARN_ON_ONCE(pmd_val(old_pmd) != 0 &&
> -					pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp)));
> +					pmd_val(old_pmd) != pmd_val(pmdp_get(pmdp)));
>  		}
>  		phys += next - addr;
>  	} while (pmdp++, addr = next, addr != end);
> @@ -293,7 +293,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>  static bool pmd_range_has_valid_noncont(pmd_t *pmdp)
>  {
>  	for (int i = 0; i < CONT_PMDS; i++) {
> -		pte_t pte = pmd_pte(READ_ONCE(pmdp[i]));
> +		pte_t pte = pmd_pte(pmdp_get(pmdp + i));
>  
>  		if (pte_valid(pte) && !pte_cont(pte))
>  			return true;
> @@ -1553,7 +1553,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
>  	do {
>  		next = pmd_addr_end(addr, end);
>  		pmdp = pmd_offset(pudp, addr);
> -		pmd = READ_ONCE(*pmdp);
> +		pmd = pmdp_get(pmdp);
>  		if (pmd_none(pmd))
>  			continue;
>  
> @@ -1708,7 +1708,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
>  	do {
>  		next = pmd_addr_end(addr, end);
>  		pmdp = pmd_offset(pudp, addr);
> -		pmd = READ_ONCE(*pmdp);
> +		pmd = pmdp_get(pmdp);
>  		if (pmd_none(pmd))
>  			continue;
>  
> @@ -1729,7 +1729,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
>  	 */
>  	pmdp = pmd_offset(pudp, 0UL);
>  	for (i = 0; i < PTRS_PER_PMD; i++) {
> -		if (!pmd_none(READ_ONCE(pmdp[i])))
> +		if (!pmd_none(pmdp_get(pmdp + i)))
>  			return;
>  	}
>  
> @@ -1881,7 +1881,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
>  	pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot));
>  
>  	/* Only allow permission changes for now */
> -	if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
> +	if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)),
>  				   pmd_val(new_pmd)))
>  		return 0;
>  
> @@ -1906,7 +1906,7 @@ int pud_clear_huge(pud_t *pudp)
>  
>  int pmd_clear_huge(pmd_t *pmdp)
>  {
> -	if (!pmd_leaf(READ_ONCE(*pmdp)))
> +	if (!pmd_leaf(pmdp_get(pmdp)))
>  		return 0;
>  	pmd_clear(pmdp);
>  	return 1;
> @@ -1917,7 +1917,7 @@ int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
>  	pte_t *table;
>  	pmd_t pmd;
>  
> -	pmd = READ_ONCE(*pmdp);
> +	pmd = pmdp_get(pmdp);
>  
>  	if (!pmd_table(pmd)) {
>  		VM_WARN_ON(1);
> diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
> index bbe98ac9ad8c..0ca07bd5ded9 100644
> --- a/arch/arm64/mm/pageattr.c
> +++ b/arch/arm64/mm/pageattr.c
> @@ -414,7 +414,7 @@ bool kernel_page_present(struct page *page)
>  		return pud_valid(pud);
>  
>  	pmdp = pmd_offset(pudp, addr);
> -	pmd = READ_ONCE(*pmdp);
> +	pmd = pmdp_get(pmdp);
>  	if (pmd_none(pmd))
>  		return false;
>  	if (pmd_leaf(pmd))
> diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
> index cca9706a875c..b27b2d2c20c3 100644
> --- a/arch/arm64/mm/trans_pgd.c
> +++ b/arch/arm64/mm/trans_pgd.c
> @@ -74,7 +74,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
>  
>  	src_pmdp = pmd_offset(src_pudp, start);
>  	do {
> -		pmd_t pmd = READ_ONCE(*src_pmdp);
> +		pmd_t pmd = pmdp_get(src_pmdp);
>  
>  		next = pmd_addr_end(addr, end);
>  		if (pmd_none(pmd))

Re: [PATCH V2 1/4] arm64/mm: Use pmdp_get() for PMD accesses

[Anshuman Khandual]

On Wed, Sep 23, 2026 at 09:32:17AM +0100, Ryan Roberts wrote:
> On 22/09/2026 07:16, Anshuman Khandual wrote:
> > Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for
> > supporting both D64 and D128 translation table formats.
> > 
> > READ_ONCE() cannot currently be used for 128-bit page table entries on
> > arm64 because it does not provide the required 128-bit single-copy
> > atomicity, causing builds to fail for accesses wider than 64 bits.
> > 
> > Although LDP/STP provide the required atomicity when FEAT_LSE is
> > available (as required by FEAT_D128), extending READ_ONCE() to support
> > 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> > so doing so could encourage other 128-bit users that would either fail
> > to build in configurations without D128 support or, if D128 becomes a
> > runtime option, silently permit tearing on systems without the required
> > hardware support.
> > 
> > Instead, standardize PMD accesses on the existing page-table helpers.
> > These can be overridden on arm64 to provide 128-bit single-copy
> > atomicity when required. No functional change intended.
> 
> I notice you have an unconverted READ_ONCE(*pmdp) in
> pmdp_test_and_clear_young(). Is that intentional?

Converting the above READ_ONCE() instances as pmdp_get() cuases a build
failure as pmpd_get() which is defined in generic pgtable header is not
visible inside platform specific pgtable header, even with a forward
declaration.

In D128 V2 series these conversions were performed later in the series,
when platform specific pxdp_get() overrides were added via ptval_get(). 

But to avoid this, we could just move pmdp_test_and_clear_young() inside
arch/arm64/mm/mmu.c file instead. Folding in the following change builds.

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index ae41946c54f26..13ca85ec89f6a 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1289,14 +1289,8 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,

 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
 #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
-static inline bool pmdp_test_and_clear_young(struct vm_area_struct *vma,
-               unsigned long address, pmd_t *pmdp)
-{
-       /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
-       VM_WARN_ON(pmd_table(READ_ONCE(*pmdp)) && !system_supports_haft());
-       return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
-}
-#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
+bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp);
+#endif

 static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,
                                               unsigned long address,
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 76d8d320aeedf..b3fd06a623db5 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -2440,4 +2440,15 @@ void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte)
                     __func__, pte_str_old, pte_str);
 }
 #endif /* CONFIG_DEBUG_VM */
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
+#define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
+bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp)
+{
+       /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
+       VM_WARN_ON(pmd_table(pmdp_get(pmdp)) && !system_supports_haft());
+       return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
+}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
+
 #endif

> 
> Thanks,
> Ryan
> 
> 
> > 
> > Cc: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Will Deacon <will@kernel.org>
> > Cc: Ryan Roberts <ryan.roberts@arm.com>
> > Cc: Mark Rtland <mark.rtland@arm.com>
> > Cc: linx-arm-kernel@lists.infradead.org
> > Cc: linx-kernel@vger.kernel.org
> > Cc: kasan-dev@googlegrops.com
> > Reviewed-by: Ryan Roberts <ryan.roberts@arm.com>
> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> > ---
> >  arch/arm64/include/asm/pgtable.h |  3 ++-
> >  arch/arm64/mm/fault.c            |  2 +-
> >  arch/arm64/mm/fixmap.c           |  2 +-
> >  arch/arm64/mm/hugetlbpage.c      |  2 +-
> >  arch/arm64/mm/kasan_init.c       |  4 ++--
> >  arch/arm64/mm/mmu.c              | 22 +++++++++++-----------
> >  arch/arm64/mm/pageattr.c         |  2 +-
> >  arch/arm64/mm/trans_pgd.c        |  2 +-
> >  8 files changed, 20 insertions(+), 19 deletions(-)
> > 
> > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> > index 763c5a411d64..b953ea0ba3ed 100644
> > --- a/arch/arm64/include/asm/pgtable.h
> > +++ b/arch/arm64/include/asm/pgtable.h
> > @@ -811,7 +811,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
> >  }
> >  
> >  /* Find an entry in the third-level page table. */
> > -#define pte_offset_phys(dir,addr)	(pmd_page_paddr(READ_ONCE(*(dir))) + pte_index(addr) * sizeof(pte_t))
> > +#define pte_offset_phys(dir, addr)	(pmd_page_paddr(pmdp_get(dir)) + \
> > +					 pte_index(addr) * sizeof(pte_t))
> >  
> >  #define pte_set_fixmap(addr)		((pte_t *)set_fixmap_offset(FIX_PTE, addr))
> >  #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
> > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> > index 2cecf6ba6df7..2757ee0c4300 100644
> > --- a/arch/arm64/mm/fault.c
> > +++ b/arch/arm64/mm/fault.c
> > @@ -188,7 +188,7 @@ static void show_pte(unsigned long addr)
> >  			break;
> >  
> >  		pmdp = pmd_offset_lockless(pudp, pud, addr);
> > -		pmd = READ_ONCE(*pmdp);
> > +		pmd = pmdp_get(pmdp);
> >  		ptval_to_str(pxd_str, pmd_val(pmd));
> >  		pr_cont(", pmd=%s", pxd_str);
> >  		if (pmd_none(pmd) || pmd_bad(pmd))
> > diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
> > index f66a0016dd02..3cdac8021d4f 100644
> > --- a/arch/arm64/mm/fixmap.c
> > +++ b/arch/arm64/mm/fixmap.c
> > @@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr)
> >  
> >  static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr)
> >  {
> > -	pmd_t pmd = READ_ONCE(*pmdp);
> > +	pmd_t pmd = pmdp_get(pmdp);
> >  	pte_t *ptep;
> >  
> >  	if (pmd_none(pmd)) {
> > diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
> > index 8e799c1fe0aa..cdaa4500faf9 100644
> > --- a/arch/arm64/mm/hugetlbpage.c
> > +++ b/arch/arm64/mm/hugetlbpage.c
> > @@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
> >  		addr &= CONT_PMD_MASK;
> >  
> >  	pmdp = pmd_offset(pudp, addr);
> > -	pmd = READ_ONCE(*pmdp);
> > +	pmd = pmdp_get(pmdp);
> >  	if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) &&
> >  	    pmd_none(pmd))
> >  		return NULL;
> > diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
> > index 45fbdce684c8..7ca833c5de5e 100644
> > --- a/arch/arm64/mm/kasan_init.c
> > +++ b/arch/arm64/mm/kasan_init.c
> > @@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node)
> >  static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
> >  				      bool early)
> >  {
> > -	if (pmd_none(READ_ONCE(*pmdp))) {
> > +	if (pmd_none(pmdp_get(pmdp))) {
> >  		phys_addr_t pte_phys = early ?
> >  				__pa_symbol(kasan_early_shadow_pte)
> >  					: kasan_alloc_zeroed_page(node);
> > @@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
> >  	do {
> >  		next = pmd_addr_end(addr, end);
> >  		kasan_pte_populate(pmdp, addr, next, node, early);
> > -	} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
> > +	} while (pmdp++, addr = next, addr != end && pmd_none(pmdp_get(pmdp)));
> >  }
> >  
> >  static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
> > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > index e589fb00d1e3..95621913679e 100644
> > --- a/arch/arm64/mm/mmu.c
> > +++ b/arch/arm64/mm/mmu.c
> > @@ -200,7 +200,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,
> >  			       int flags)
> >  {
> >  	unsigned long next;
> > -	pmd_t pmd = READ_ONCE(*pmdp);
> > +	pmd_t pmd = pmdp_get(pmdp);
> >  	pte_t *ptep;
> >  
> >  	BUG_ON(pmd_leaf(pmd));
> > @@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >  	unsigned long next;
> >  
> >  	do {
> > -		pmd_t old_pmd = READ_ONCE(*pmdp);
> > +		pmd_t old_pmd = pmdp_get(pmdp);
> >  
> >  		next = pmd_addr_end(addr, end);
> >  
> > @@ -272,7 +272,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >  			 * only allow updates to the permission attributes.
> >  			 */
> >  			BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd),
> > -						      READ_ONCE(pmd_val(*pmdp))));
> > +						      pmd_val(pmdp_get(pmdp))));
> >  		} else {
> >  			int ret;
> >  
> > @@ -282,7 +282,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >  				return ret;
> >  
> >  			VM_WARN_ON_ONCE(pmd_val(old_pmd) != 0 &&
> > -					pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp)));
> > +					pmd_val(old_pmd) != pmd_val(pmdp_get(pmdp)));
> >  		}
> >  		phys += next - addr;
> >  	} while (pmdp++, addr = next, addr != end);
> > @@ -293,7 +293,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >  static bool pmd_range_has_valid_noncont(pmd_t *pmdp)
> >  {
> >  	for (int i = 0; i < CONT_PMDS; i++) {
> > -		pte_t pte = pmd_pte(READ_ONCE(pmdp[i]));
> > +		pte_t pte = pmd_pte(pmdp_get(pmdp + i));
> >  
> >  		if (pte_valid(pte) && !pte_cont(pte))
> >  			return true;
> > @@ -1553,7 +1553,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
> >  	do {
> >  		next = pmd_addr_end(addr, end);
> >  		pmdp = pmd_offset(pudp, addr);
> > -		pmd = READ_ONCE(*pmdp);
> > +		pmd = pmdp_get(pmdp);
> >  		if (pmd_none(pmd))
> >  			continue;
> >  
> > @@ -1708,7 +1708,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
> >  	do {
> >  		next = pmd_addr_end(addr, end);
> >  		pmdp = pmd_offset(pudp, addr);
> > -		pmd = READ_ONCE(*pmdp);
> > +		pmd = pmdp_get(pmdp);
> >  		if (pmd_none(pmd))
> >  			continue;
> >  
> > @@ -1729,7 +1729,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
> >  	 */
> >  	pmdp = pmd_offset(pudp, 0UL);
> >  	for (i = 0; i < PTRS_PER_PMD; i++) {
> > -		if (!pmd_none(READ_ONCE(pmdp[i])))
> > +		if (!pmd_none(pmdp_get(pmdp + i)))
> >  			return;
> >  	}
> >  
> > @@ -1881,7 +1881,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
> >  	pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot));
> >  
> >  	/* Only allow permission changes for now */
> > -	if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
> > +	if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)),
> >  				   pmd_val(new_pmd)))
> >  		return 0;
> >  
> > @@ -1906,7 +1906,7 @@ int pud_clear_huge(pud_t *pudp)
> >  
> >  int pmd_clear_huge(pmd_t *pmdp)
> >  {
> > -	if (!pmd_leaf(READ_ONCE(*pmdp)))
> > +	if (!pmd_leaf(pmdp_get(pmdp)))
> >  		return 0;
> >  	pmd_clear(pmdp);
> >  	return 1;
> > @@ -1917,7 +1917,7 @@ int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
> >  	pte_t *table;
> >  	pmd_t pmd;
> >  
> > -	pmd = READ_ONCE(*pmdp);
> > +	pmd = pmdp_get(pmdp);
> >  
> >  	if (!pmd_table(pmd)) {
> >  		VM_WARN_ON(1);
> > diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
> > index bbe98ac9ad8c..0ca07bd5ded9 100644
> > --- a/arch/arm64/mm/pageattr.c
> > +++ b/arch/arm64/mm/pageattr.c
> > @@ -414,7 +414,7 @@ bool kernel_page_present(struct page *page)
> >  		return pud_valid(pud);
> >  
> >  	pmdp = pmd_offset(pudp, addr);
> > -	pmd = READ_ONCE(*pmdp);
> > +	pmd = pmdp_get(pmdp);
> >  	if (pmd_none(pmd))
> >  		return false;
> >  	if (pmd_leaf(pmd))
> > diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
> > index cca9706a875c..b27b2d2c20c3 100644
> > --- a/arch/arm64/mm/trans_pgd.c
> > +++ b/arch/arm64/mm/trans_pgd.c
> > @@ -74,7 +74,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
> >  
> >  	src_pmdp = pmd_offset(src_pudp, start);
> >  	do {
> > -		pmd_t pmd = READ_ONCE(*src_pmdp);
> > +		pmd_t pmd = pmdp_get(src_pmdp);
> >  
> >  		next = pmd_addr_end(addr, end);
> >  		if (pmd_none(pmd))
> 

Re: [PATCH V2 1/4] arm64/mm: Use pmdp_get() for PMD accesses

[Ryan Roberts]

On 23/09/2026 12:01, Anshuman Khandual wrote:
> On Wed, Sep 23, 2026 at 09:32:17AM +0100, Ryan Roberts wrote:
>> On 22/09/2026 07:16, Anshuman Khandual wrote:
>>> Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for
>>> supporting both D64 and D128 translation table formats.
>>>
>>> READ_ONCE() cannot currently be used for 128-bit page table entries on
>>> arm64 because it does not provide the required 128-bit single-copy
>>> atomicity, causing builds to fail for accesses wider than 64 bits.
>>>
>>> Although LDP/STP provide the required atomicity when FEAT_LSE is
>>> available (as required by FEAT_D128), extending READ_ONCE() to support
>>> 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
>>> so doing so could encourage other 128-bit users that would either fail
>>> to build in configurations without D128 support or, if D128 becomes a
>>> runtime option, silently permit tearing on systems without the required
>>> hardware support.
>>>
>>> Instead, standardize PMD accesses on the existing page-table helpers.
>>> These can be overridden on arm64 to provide 128-bit single-copy
>>> atomicity when required. No functional change intended.
>>
>> I notice you have an unconverted READ_ONCE(*pmdp) in
>> pmdp_test_and_clear_young(). Is that intentional?
> 
> Converting the above READ_ONCE() instances as pmdp_get() cuases a build
> failure as pmpd_get() which is defined in generic pgtable header is not
> visible inside platform specific pgtable header, even with a forward
> declaration.
> 
> In D128 V2 series these conversions were performed later in the series,
> when platform specific pxdp_get() overrides were added via ptval_get(). 
> 
> But to avoid this, we could just move pmdp_test_and_clear_young() inside
> arch/arm64/mm/mmu.c file instead. Folding in the following change builds.

I'm not sure if/how this might affect performance? Probably not significant
since they are for higher levels, so not called as frequently as pte helpers.

Another option would be to implement arm64-specific pxdp_get() helpers in
asm/pgtable.h to override the generic versions?

Thanks,
Ryan

> 
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index ae41946c54f26..13ca85ec89f6a 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -1289,14 +1289,8 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
> 
>  #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
>  #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
> -static inline bool pmdp_test_and_clear_young(struct vm_area_struct *vma,
> -               unsigned long address, pmd_t *pmdp)
> -{
> -       /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
> -       VM_WARN_ON(pmd_table(READ_ONCE(*pmdp)) && !system_supports_haft());
> -       return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
> -}
> -#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
> +bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp);
> +#endif
> 
>  static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,
>                                                unsigned long address,
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index 76d8d320aeedf..b3fd06a623db5 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -2440,4 +2440,15 @@ void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte)
>                      __func__, pte_str_old, pte_str);
>  }
>  #endif /* CONFIG_DEBUG_VM */
> +
> +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
> +#define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
> +bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp)
> +{
> +       /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
> +       VM_WARN_ON(pmd_table(pmdp_get(pmdp)) && !system_supports_haft());
> +       return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
> +}
> +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
> +
>  #endif
> 
>>
>> Thanks,
>> Ryan
>>
>>
>>>
>>> Cc: Catalin Marinas <catalin.marinas@arm.com>
>>> Cc: Will Deacon <will@kernel.org>
>>> Cc: Ryan Roberts <ryan.roberts@arm.com>
>>> Cc: Mark Rtland <mark.rtland@arm.com>
>>> Cc: linx-arm-kernel@lists.infradead.org
>>> Cc: linx-kernel@vger.kernel.org
>>> Cc: kasan-dev@googlegrops.com
>>> Reviewed-by: Ryan Roberts <ryan.roberts@arm.com>
>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
>>> ---
>>>  arch/arm64/include/asm/pgtable.h |  3 ++-
>>>  arch/arm64/mm/fault.c            |  2 +-
>>>  arch/arm64/mm/fixmap.c           |  2 +-
>>>  arch/arm64/mm/hugetlbpage.c      |  2 +-
>>>  arch/arm64/mm/kasan_init.c       |  4 ++--
>>>  arch/arm64/mm/mmu.c              | 22 +++++++++++-----------
>>>  arch/arm64/mm/pageattr.c         |  2 +-
>>>  arch/arm64/mm/trans_pgd.c        |  2 +-
>>>  8 files changed, 20 insertions(+), 19 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
>>> index 763c5a411d64..b953ea0ba3ed 100644
>>> --- a/arch/arm64/include/asm/pgtable.h
>>> +++ b/arch/arm64/include/asm/pgtable.h
>>> @@ -811,7 +811,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
>>>  }
>>>  
>>>  /* Find an entry in the third-level page table. */
>>> -#define pte_offset_phys(dir,addr)	(pmd_page_paddr(READ_ONCE(*(dir))) + pte_index(addr) * sizeof(pte_t))
>>> +#define pte_offset_phys(dir, addr)	(pmd_page_paddr(pmdp_get(dir)) + \
>>> +					 pte_index(addr) * sizeof(pte_t))
>>>  
>>>  #define pte_set_fixmap(addr)		((pte_t *)set_fixmap_offset(FIX_PTE, addr))
>>>  #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
>>> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
>>> index 2cecf6ba6df7..2757ee0c4300 100644
>>> --- a/arch/arm64/mm/fault.c
>>> +++ b/arch/arm64/mm/fault.c
>>> @@ -188,7 +188,7 @@ static void show_pte(unsigned long addr)
>>>  			break;
>>>  
>>>  		pmdp = pmd_offset_lockless(pudp, pud, addr);
>>> -		pmd = READ_ONCE(*pmdp);
>>> +		pmd = pmdp_get(pmdp);
>>>  		ptval_to_str(pxd_str, pmd_val(pmd));
>>>  		pr_cont(", pmd=%s", pxd_str);
>>>  		if (pmd_none(pmd) || pmd_bad(pmd))
>>> diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
>>> index f66a0016dd02..3cdac8021d4f 100644
>>> --- a/arch/arm64/mm/fixmap.c
>>> +++ b/arch/arm64/mm/fixmap.c
>>> @@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr)
>>>  
>>>  static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr)
>>>  {
>>> -	pmd_t pmd = READ_ONCE(*pmdp);
>>> +	pmd_t pmd = pmdp_get(pmdp);
>>>  	pte_t *ptep;
>>>  
>>>  	if (pmd_none(pmd)) {
>>> diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
>>> index 8e799c1fe0aa..cdaa4500faf9 100644
>>> --- a/arch/arm64/mm/hugetlbpage.c
>>> +++ b/arch/arm64/mm/hugetlbpage.c
>>> @@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
>>>  		addr &= CONT_PMD_MASK;
>>>  
>>>  	pmdp = pmd_offset(pudp, addr);
>>> -	pmd = READ_ONCE(*pmdp);
>>> +	pmd = pmdp_get(pmdp);
>>>  	if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) &&
>>>  	    pmd_none(pmd))
>>>  		return NULL;
>>> diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
>>> index 45fbdce684c8..7ca833c5de5e 100644
>>> --- a/arch/arm64/mm/kasan_init.c
>>> +++ b/arch/arm64/mm/kasan_init.c
>>> @@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node)
>>>  static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
>>>  				      bool early)
>>>  {
>>> -	if (pmd_none(READ_ONCE(*pmdp))) {
>>> +	if (pmd_none(pmdp_get(pmdp))) {
>>>  		phys_addr_t pte_phys = early ?
>>>  				__pa_symbol(kasan_early_shadow_pte)
>>>  					: kasan_alloc_zeroed_page(node);
>>> @@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
>>>  	do {
>>>  		next = pmd_addr_end(addr, end);
>>>  		kasan_pte_populate(pmdp, addr, next, node, early);
>>> -	} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
>>> +	} while (pmdp++, addr = next, addr != end && pmd_none(pmdp_get(pmdp)));
>>>  }
>>>  
>>>  static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
>>> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
>>> index e589fb00d1e3..95621913679e 100644
>>> --- a/arch/arm64/mm/mmu.c
>>> +++ b/arch/arm64/mm/mmu.c
>>> @@ -200,7 +200,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,
>>>  			       int flags)
>>>  {
>>>  	unsigned long next;
>>> -	pmd_t pmd = READ_ONCE(*pmdp);
>>> +	pmd_t pmd = pmdp_get(pmdp);
>>>  	pte_t *ptep;
>>>  
>>>  	BUG_ON(pmd_leaf(pmd));
>>> @@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>>>  	unsigned long next;
>>>  
>>>  	do {
>>> -		pmd_t old_pmd = READ_ONCE(*pmdp);
>>> +		pmd_t old_pmd = pmdp_get(pmdp);
>>>  
>>>  		next = pmd_addr_end(addr, end);
>>>  
>>> @@ -272,7 +272,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>>>  			 * only allow updates to the permission attributes.
>>>  			 */
>>>  			BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd),
>>> -						      READ_ONCE(pmd_val(*pmdp))));
>>> +						      pmd_val(pmdp_get(pmdp))));
>>>  		} else {
>>>  			int ret;
>>>  
>>> @@ -282,7 +282,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>>>  				return ret;
>>>  
>>>  			VM_WARN_ON_ONCE(pmd_val(old_pmd) != 0 &&
>>> -					pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp)));
>>> +					pmd_val(old_pmd) != pmd_val(pmdp_get(pmdp)));
>>>  		}
>>>  		phys += next - addr;
>>>  	} while (pmdp++, addr = next, addr != end);
>>> @@ -293,7 +293,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
>>>  static bool pmd_range_has_valid_noncont(pmd_t *pmdp)
>>>  {
>>>  	for (int i = 0; i < CONT_PMDS; i++) {
>>> -		pte_t pte = pmd_pte(READ_ONCE(pmdp[i]));
>>> +		pte_t pte = pmd_pte(pmdp_get(pmdp + i));
>>>  
>>>  		if (pte_valid(pte) && !pte_cont(pte))
>>>  			return true;
>>> @@ -1553,7 +1553,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
>>>  	do {
>>>  		next = pmd_addr_end(addr, end);
>>>  		pmdp = pmd_offset(pudp, addr);
>>> -		pmd = READ_ONCE(*pmdp);
>>> +		pmd = pmdp_get(pmdp);
>>>  		if (pmd_none(pmd))
>>>  			continue;
>>>  
>>> @@ -1708,7 +1708,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
>>>  	do {
>>>  		next = pmd_addr_end(addr, end);
>>>  		pmdp = pmd_offset(pudp, addr);
>>> -		pmd = READ_ONCE(*pmdp);
>>> +		pmd = pmdp_get(pmdp);
>>>  		if (pmd_none(pmd))
>>>  			continue;
>>>  
>>> @@ -1729,7 +1729,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
>>>  	 */
>>>  	pmdp = pmd_offset(pudp, 0UL);
>>>  	for (i = 0; i < PTRS_PER_PMD; i++) {
>>> -		if (!pmd_none(READ_ONCE(pmdp[i])))
>>> +		if (!pmd_none(pmdp_get(pmdp + i)))
>>>  			return;
>>>  	}
>>>  
>>> @@ -1881,7 +1881,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
>>>  	pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot));
>>>  
>>>  	/* Only allow permission changes for now */
>>> -	if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
>>> +	if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)),
>>>  				   pmd_val(new_pmd)))
>>>  		return 0;
>>>  
>>> @@ -1906,7 +1906,7 @@ int pud_clear_huge(pud_t *pudp)
>>>  
>>>  int pmd_clear_huge(pmd_t *pmdp)
>>>  {
>>> -	if (!pmd_leaf(READ_ONCE(*pmdp)))
>>> +	if (!pmd_leaf(pmdp_get(pmdp)))
>>>  		return 0;
>>>  	pmd_clear(pmdp);
>>>  	return 1;
>>> @@ -1917,7 +1917,7 @@ int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
>>>  	pte_t *table;
>>>  	pmd_t pmd;
>>>  
>>> -	pmd = READ_ONCE(*pmdp);
>>> +	pmd = pmdp_get(pmdp);
>>>  
>>>  	if (!pmd_table(pmd)) {
>>>  		VM_WARN_ON(1);
>>> diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
>>> index bbe98ac9ad8c..0ca07bd5ded9 100644
>>> --- a/arch/arm64/mm/pageattr.c
>>> +++ b/arch/arm64/mm/pageattr.c
>>> @@ -414,7 +414,7 @@ bool kernel_page_present(struct page *page)
>>>  		return pud_valid(pud);
>>>  
>>>  	pmdp = pmd_offset(pudp, addr);
>>> -	pmd = READ_ONCE(*pmdp);
>>> +	pmd = pmdp_get(pmdp);
>>>  	if (pmd_none(pmd))
>>>  		return false;
>>>  	if (pmd_leaf(pmd))
>>> diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
>>> index cca9706a875c..b27b2d2c20c3 100644
>>> --- a/arch/arm64/mm/trans_pgd.c
>>> +++ b/arch/arm64/mm/trans_pgd.c
>>> @@ -74,7 +74,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
>>>  
>>>  	src_pmdp = pmd_offset(src_pudp, start);
>>>  	do {
>>> -		pmd_t pmd = READ_ONCE(*src_pmdp);
>>> +		pmd_t pmd = pmdp_get(src_pmdp);
>>>  
>>>  		next = pmd_addr_end(addr, end);
>>>  		if (pmd_none(pmd))
>>

Re: [PATCH V2 1/4] arm64/mm: Use pmdp_get() for PMD accesses

[Anshuman Khandual]

On Wed, Sep 23, 2026 at 12:31:42PM +0100, Ryan Roberts wrote:
> On 23/09/2026 12:01, Anshuman Khandual wrote:
> > On Wed, Sep 23, 2026 at 09:32:17AM +0100, Ryan Roberts wrote:
> >> On 22/09/2026 07:16, Anshuman Khandual wrote:
> >>> Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for
> >>> supporting both D64 and D128 translation table formats.
> >>>
> >>> READ_ONCE() cannot currently be used for 128-bit page table entries on
> >>> arm64 because it does not provide the required 128-bit single-copy
> >>> atomicity, causing builds to fail for accesses wider than 64 bits.
> >>>
> >>> Although LDP/STP provide the required atomicity when FEAT_LSE is
> >>> available (as required by FEAT_D128), extending READ_ONCE() to support
> >>> 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> >>> so doing so could encourage other 128-bit users that would either fail
> >>> to build in configurations without D128 support or, if D128 becomes a
> >>> runtime option, silently permit tearing on systems without the required
> >>> hardware support.
> >>>
> >>> Instead, standardize PMD accesses on the existing page-table helpers.
> >>> These can be overridden on arm64 to provide 128-bit single-copy
> >>> atomicity when required. No functional change intended.
> >>
> >> I notice you have an unconverted READ_ONCE(*pmdp) in
> >> pmdp_test_and_clear_young(). Is that intentional?
> > 
> > Converting the above READ_ONCE() instances as pmdp_get() cuases a build
> > failure as pmpd_get() which is defined in generic pgtable header is not
> > visible inside platform specific pgtable header, even with a forward
> > declaration.
> > 
> > In D128 V2 series these conversions were performed later in the series,
> > when platform specific pxdp_get() overrides were added via ptval_get(). 
> > 
> > But to avoid this, we could just move pmdp_test_and_clear_young() inside
> > arch/arm64/mm/mmu.c file instead. Folding in the following change builds.
> 
> I'm not sure if/how this might affect performance? Probably not significant
> since they are for higher levels, so not called as frequently as pte helpers.

Right, should not have much performance impact.

> 
> Another option would be to implement arm64-specific pxdp_get() helpers in
> asm/pgtable.h to override the generic versions?

For now those will still be based on READ_ONCE() just like the default ones.
Although trying to drop similar redundancies from other platforms :)

https://lore.kernel.org/linux-mm/20260923043226.331880-1-anshuman.khandual@arm.com/

But given D128 is going to define pxdp_get() anyway it might make sense to
define them now and withdraw the above series :) Seems like s390 platform
also has similar situation.

>
> Thanks,
> Ryan
> 
> > 
> > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> > index ae41946c54f26..13ca85ec89f6a 100644
> > --- a/arch/arm64/include/asm/pgtable.h
> > +++ b/arch/arm64/include/asm/pgtable.h
> > @@ -1289,14 +1289,8 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
> > 
> >  #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
> >  #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
> > -static inline bool pmdp_test_and_clear_young(struct vm_area_struct *vma,
> > -               unsigned long address, pmd_t *pmdp)
> > -{
> > -       /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
> > -       VM_WARN_ON(pmd_table(READ_ONCE(*pmdp)) && !system_supports_haft());
> > -       return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
> > -}
> > -#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
> > +bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp);
> > +#endif
> > 
> >  static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,
> >                                                unsigned long address,
> > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > index 76d8d320aeedf..b3fd06a623db5 100644
> > --- a/arch/arm64/mm/mmu.c
> > +++ b/arch/arm64/mm/mmu.c
> > @@ -2440,4 +2440,15 @@ void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte)
> >                      __func__, pte_str_old, pte_str);
> >  }
> >  #endif /* CONFIG_DEBUG_VM */
> > +
> > +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
> > +#define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
> > +bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp)
> > +{
> > +       /* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
> > +       VM_WARN_ON(pmd_table(pmdp_get(pmdp)) && !system_supports_haft());
> > +       return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
> > +}
> > +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
> > +
> >  #endif
> > 
> >>
> >> Thanks,
> >> Ryan
> >>
> >>
> >>>
> >>> Cc: Catalin Marinas <catalin.marinas@arm.com>
> >>> Cc: Will Deacon <will@kernel.org>
> >>> Cc: Ryan Roberts <ryan.roberts@arm.com>
> >>> Cc: Mark Rtland <mark.rtland@arm.com>
> >>> Cc: linx-arm-kernel@lists.infradead.org
> >>> Cc: linx-kernel@vger.kernel.org
> >>> Cc: kasan-dev@googlegrops.com
> >>> Reviewed-by: Ryan Roberts <ryan.roberts@arm.com>
> >>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> >>> ---
> >>>  arch/arm64/include/asm/pgtable.h |  3 ++-
> >>>  arch/arm64/mm/fault.c            |  2 +-
> >>>  arch/arm64/mm/fixmap.c           |  2 +-
> >>>  arch/arm64/mm/hugetlbpage.c      |  2 +-
> >>>  arch/arm64/mm/kasan_init.c       |  4 ++--
> >>>  arch/arm64/mm/mmu.c              | 22 +++++++++++-----------
> >>>  arch/arm64/mm/pageattr.c         |  2 +-
> >>>  arch/arm64/mm/trans_pgd.c        |  2 +-
> >>>  8 files changed, 20 insertions(+), 19 deletions(-)
> >>>
> >>> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> >>> index 763c5a411d64..b953ea0ba3ed 100644
> >>> --- a/arch/arm64/include/asm/pgtable.h
> >>> +++ b/arch/arm64/include/asm/pgtable.h
> >>> @@ -811,7 +811,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
> >>>  }
> >>>  
> >>>  /* Find an entry in the third-level page table. */
> >>> -#define pte_offset_phys(dir,addr)	(pmd_page_paddr(READ_ONCE(*(dir))) + pte_index(addr) * sizeof(pte_t))
> >>> +#define pte_offset_phys(dir, addr)	(pmd_page_paddr(pmdp_get(dir)) + \
> >>> +					 pte_index(addr) * sizeof(pte_t))
> >>>  
> >>>  #define pte_set_fixmap(addr)		((pte_t *)set_fixmap_offset(FIX_PTE, addr))
> >>>  #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))
> >>> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> >>> index 2cecf6ba6df7..2757ee0c4300 100644
> >>> --- a/arch/arm64/mm/fault.c
> >>> +++ b/arch/arm64/mm/fault.c
> >>> @@ -188,7 +188,7 @@ static void show_pte(unsigned long addr)
> >>>  			break;
> >>>  
> >>>  		pmdp = pmd_offset_lockless(pudp, pud, addr);
> >>> -		pmd = READ_ONCE(*pmdp);
> >>> +		pmd = pmdp_get(pmdp);
> >>>  		ptval_to_str(pxd_str, pmd_val(pmd));
> >>>  		pr_cont(", pmd=%s", pxd_str);
> >>>  		if (pmd_none(pmd) || pmd_bad(pmd))
> >>> diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
> >>> index f66a0016dd02..3cdac8021d4f 100644
> >>> --- a/arch/arm64/mm/fixmap.c
> >>> +++ b/arch/arm64/mm/fixmap.c
> >>> @@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr)
> >>>  
> >>>  static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr)
> >>>  {
> >>> -	pmd_t pmd = READ_ONCE(*pmdp);
> >>> +	pmd_t pmd = pmdp_get(pmdp);
> >>>  	pte_t *ptep;
> >>>  
> >>>  	if (pmd_none(pmd)) {
> >>> diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
> >>> index 8e799c1fe0aa..cdaa4500faf9 100644
> >>> --- a/arch/arm64/mm/hugetlbpage.c
> >>> +++ b/arch/arm64/mm/hugetlbpage.c
> >>> @@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
> >>>  		addr &= CONT_PMD_MASK;
> >>>  
> >>>  	pmdp = pmd_offset(pudp, addr);
> >>> -	pmd = READ_ONCE(*pmdp);
> >>> +	pmd = pmdp_get(pmdp);
> >>>  	if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) &&
> >>>  	    pmd_none(pmd))
> >>>  		return NULL;
> >>> diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
> >>> index 45fbdce684c8..7ca833c5de5e 100644
> >>> --- a/arch/arm64/mm/kasan_init.c
> >>> +++ b/arch/arm64/mm/kasan_init.c
> >>> @@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node)
> >>>  static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
> >>>  				      bool early)
> >>>  {
> >>> -	if (pmd_none(READ_ONCE(*pmdp))) {
> >>> +	if (pmd_none(pmdp_get(pmdp))) {
> >>>  		phys_addr_t pte_phys = early ?
> >>>  				__pa_symbol(kasan_early_shadow_pte)
> >>>  					: kasan_alloc_zeroed_page(node);
> >>> @@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
> >>>  	do {
> >>>  		next = pmd_addr_end(addr, end);
> >>>  		kasan_pte_populate(pmdp, addr, next, node, early);
> >>> -	} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
> >>> +	} while (pmdp++, addr = next, addr != end && pmd_none(pmdp_get(pmdp)));
> >>>  }
> >>>  
> >>>  static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
> >>> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> >>> index e589fb00d1e3..95621913679e 100644
> >>> --- a/arch/arm64/mm/mmu.c
> >>> +++ b/arch/arm64/mm/mmu.c
> >>> @@ -200,7 +200,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,
> >>>  			       int flags)
> >>>  {
> >>>  	unsigned long next;
> >>> -	pmd_t pmd = READ_ONCE(*pmdp);
> >>> +	pmd_t pmd = pmdp_get(pmdp);
> >>>  	pte_t *ptep;
> >>>  
> >>>  	BUG_ON(pmd_leaf(pmd));
> >>> @@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >>>  	unsigned long next;
> >>>  
> >>>  	do {
> >>> -		pmd_t old_pmd = READ_ONCE(*pmdp);
> >>> +		pmd_t old_pmd = pmdp_get(pmdp);
> >>>  
> >>>  		next = pmd_addr_end(addr, end);
> >>>  
> >>> @@ -272,7 +272,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >>>  			 * only allow updates to the permission attributes.
> >>>  			 */
> >>>  			BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd),
> >>> -						      READ_ONCE(pmd_val(*pmdp))));
> >>> +						      pmd_val(pmdp_get(pmdp))));
> >>>  		} else {
> >>>  			int ret;
> >>>  
> >>> @@ -282,7 +282,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >>>  				return ret;
> >>>  
> >>>  			VM_WARN_ON_ONCE(pmd_val(old_pmd) != 0 &&
> >>> -					pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp)));
> >>> +					pmd_val(old_pmd) != pmd_val(pmdp_get(pmdp)));
> >>>  		}
> >>>  		phys += next - addr;
> >>>  	} while (pmdp++, addr = next, addr != end);
> >>> @@ -293,7 +293,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
> >>>  static bool pmd_range_has_valid_noncont(pmd_t *pmdp)
> >>>  {
> >>>  	for (int i = 0; i < CONT_PMDS; i++) {
> >>> -		pte_t pte = pmd_pte(READ_ONCE(pmdp[i]));
> >>> +		pte_t pte = pmd_pte(pmdp_get(pmdp + i));
> >>>  
> >>>  		if (pte_valid(pte) && !pte_cont(pte))
> >>>  			return true;
> >>> @@ -1553,7 +1553,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
> >>>  	do {
> >>>  		next = pmd_addr_end(addr, end);
> >>>  		pmdp = pmd_offset(pudp, addr);
> >>> -		pmd = READ_ONCE(*pmdp);
> >>> +		pmd = pmdp_get(pmdp);
> >>>  		if (pmd_none(pmd))
> >>>  			continue;
> >>>  
> >>> @@ -1708,7 +1708,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
> >>>  	do {
> >>>  		next = pmd_addr_end(addr, end);
> >>>  		pmdp = pmd_offset(pudp, addr);
> >>> -		pmd = READ_ONCE(*pmdp);
> >>> +		pmd = pmdp_get(pmdp);
> >>>  		if (pmd_none(pmd))
> >>>  			continue;
> >>>  
> >>> @@ -1729,7 +1729,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
> >>>  	 */
> >>>  	pmdp = pmd_offset(pudp, 0UL);
> >>>  	for (i = 0; i < PTRS_PER_PMD; i++) {
> >>> -		if (!pmd_none(READ_ONCE(pmdp[i])))
> >>> +		if (!pmd_none(pmdp_get(pmdp + i)))
> >>>  			return;
> >>>  	}
> >>>  
> >>> @@ -1881,7 +1881,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
> >>>  	pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot));
> >>>  
> >>>  	/* Only allow permission changes for now */
> >>> -	if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
> >>> +	if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)),
> >>>  				   pmd_val(new_pmd)))
> >>>  		return 0;
> >>>  
> >>> @@ -1906,7 +1906,7 @@ int pud_clear_huge(pud_t *pudp)
> >>>  
> >>>  int pmd_clear_huge(pmd_t *pmdp)
> >>>  {
> >>> -	if (!pmd_leaf(READ_ONCE(*pmdp)))
> >>> +	if (!pmd_leaf(pmdp_get(pmdp)))
> >>>  		return 0;
> >>>  	pmd_clear(pmdp);
> >>>  	return 1;
> >>> @@ -1917,7 +1917,7 @@ int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
> >>>  	pte_t *table;
> >>>  	pmd_t pmd;
> >>>  
> >>> -	pmd = READ_ONCE(*pmdp);
> >>> +	pmd = pmdp_get(pmdp);
> >>>  
> >>>  	if (!pmd_table(pmd)) {
> >>>  		VM_WARN_ON(1);
> >>> diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
> >>> index bbe98ac9ad8c..0ca07bd5ded9 100644
> >>> --- a/arch/arm64/mm/pageattr.c
> >>> +++ b/arch/arm64/mm/pageattr.c
> >>> @@ -414,7 +414,7 @@ bool kernel_page_present(struct page *page)
> >>>  		return pud_valid(pud);
> >>>  
> >>>  	pmdp = pmd_offset(pudp, addr);
> >>> -	pmd = READ_ONCE(*pmdp);
> >>> +	pmd = pmdp_get(pmdp);
> >>>  	if (pmd_none(pmd))
> >>>  		return false;
> >>>  	if (pmd_leaf(pmd))
> >>> diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
> >>> index cca9706a875c..b27b2d2c20c3 100644
> >>> --- a/arch/arm64/mm/trans_pgd.c
> >>> +++ b/arch/arm64/mm/trans_pgd.c
> >>> @@ -74,7 +74,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
> >>>  
> >>>  	src_pmdp = pmd_offset(src_pudp, start);
> >>>  	do {
> >>> -		pmd_t pmd = READ_ONCE(*src_pmdp);
> >>> +		pmd_t pmd = pmdp_get(src_pmdp);
> >>>  
> >>>  		next = pmd_addr_end(addr, end);
> >>>  		if (pmd_none(pmd))
> >>
> 

Re: [PATCH V2 1/4] arm64/mm: Use pmdp_get() for PMD accesses

[Ryan Roberts]

On 23/09/2026 13:22, Anshuman Khandual wrote:
> On Wed, Sep 23, 2026 at 12:31:42PM +0100, Ryan Roberts wrote:
>> On 23/09/2026 12:01, Anshuman Khandual wrote:
>>> On Wed, Sep 23, 2026 at 09:32:17AM +0100, Ryan Roberts wrote:
>>>> On 22/09/2026 07:16, Anshuman Khandual wrote:
>>>>> Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for
>>>>> supporting both D64 and D128 translation table formats.
>>>>>
>>>>> READ_ONCE() cannot currently be used for 128-bit page table entries on
>>>>> arm64 because it does not provide the required 128-bit single-copy
>>>>> atomicity, causing builds to fail for accesses wider than 64 bits.
>>>>>
>>>>> Although LDP/STP provide the required atomicity when FEAT_LSE is
>>>>> available (as required by FEAT_D128), extending READ_ONCE() to support
>>>>> 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
>>>>> so doing so could encourage other 128-bit users that would either fail
>>>>> to build in configurations without D128 support or, if D128 becomes a
>>>>> runtime option, silently permit tearing on systems without the required
>>>>> hardware support.
>>>>>
>>>>> Instead, standardize PMD accesses on the existing page-table helpers.
>>>>> These can be overridden on arm64 to provide 128-bit single-copy
>>>>> atomicity when required. No functional change intended.
>>>>
>>>> I notice you have an unconverted READ_ONCE(*pmdp) in
>>>> pmdp_test_and_clear_young(). Is that intentional?
>>>
>>> Converting the above READ_ONCE() instances as pmdp_get() cuases a build
>>> failure as pmpd_get() which is defined in generic pgtable header is not
>>> visible inside platform specific pgtable header, even with a forward
>>> declaration.

I was wondering why the forward declaration doesn't work, but looks like some c
files include asm/pgtable.h directly, which explains it. Let's assume there are
good reasons and unpicking it so that only linux/pgtable.h includes
asm/pgtable.h is not practical.

>>>
>>> In D128 V2 series these conversions were performed later in the series,
>>> when platform specific pxdp_get() overrides were added via ptval_get(). 
>>>
>>> But to avoid this, we could just move pmdp_test_and_clear_young() inside
>>> arch/arm64/mm/mmu.c file instead. Folding in the following change builds.
>>
>> I'm not sure if/how this might affect performance? Probably not significant
>> since they are for higher levels, so not called as frequently as pte helpers.
> 
> Right, should not have much performance impact.
> 
>>
>> Another option would be to implement arm64-specific pxdp_get() helpers in
>> asm/pgtable.h to override the generic versions?
> 
> For now those will still be based on READ_ONCE() just like the default ones.
> Although trying to drop similar redundancies from other platforms :)
> 
> https://lore.kernel.org/linux-mm/20260923043226.331880-1-anshuman.khandual@arm.com/
> 
> But given D128 is going to define pxdp_get() anyway it might make sense to
> define them now and withdraw the above series :) Seems like s390 platform
> also has similar situation.

Given the desired end state is that arm64 defines it's own pxdp_get() helpers, I
think the best approach is just to define them now (as READ_ONCE()).

Thanks,
Ryan

Re: [PATCH V2 1/4] arm64/mm: Use pmdp_get() for PMD accesses

[Anshuman Khandual]

On Wed, Sep 23, 2026 at 01:50:34PM +0100, Ryan Roberts wrote:
> On 23/09/2026 13:22, Anshuman Khandual wrote:
> > On Wed, Sep 23, 2026 at 12:31:42PM +0100, Ryan Roberts wrote:
> >> On 23/09/2026 12:01, Anshuman Khandual wrote:
> >>> On Wed, Sep 23, 2026 at 09:32:17AM +0100, Ryan Roberts wrote:
> >>>> On 22/09/2026 07:16, Anshuman Khandual wrote:
> >>>>> Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for
> >>>>> supporting both D64 and D128 translation table formats.
> >>>>>
> >>>>> READ_ONCE() cannot currently be used for 128-bit page table entries on
> >>>>> arm64 because it does not provide the required 128-bit single-copy
> >>>>> atomicity, causing builds to fail for accesses wider than 64 bits.
> >>>>>
> >>>>> Although LDP/STP provide the required atomicity when FEAT_LSE is
> >>>>> available (as required by FEAT_D128), extending READ_ONCE() to support
> >>>>> 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> >>>>> so doing so could encourage other 128-bit users that would either fail
> >>>>> to build in configurations without D128 support or, if D128 becomes a
> >>>>> runtime option, silently permit tearing on systems without the required
> >>>>> hardware support.
> >>>>>
> >>>>> Instead, standardize PMD accesses on the existing page-table helpers.
> >>>>> These can be overridden on arm64 to provide 128-bit single-copy
> >>>>> atomicity when required. No functional change intended.
> >>>>
> >>>> I notice you have an unconverted READ_ONCE(*pmdp) in
> >>>> pmdp_test_and_clear_young(). Is that intentional?
> >>>
> >>> Converting the above READ_ONCE() instances as pmdp_get() cuases a build
> >>> failure as pmpd_get() which is defined in generic pgtable header is not
> >>> visible inside platform specific pgtable header, even with a forward
> >>> declaration.
> 
> I was wondering why the forward declaration doesn't work, but looks like some c
> files include asm/pgtable.h directly, which explains it. Let's assume there are
> good reasons and unpicking it so that only linux/pgtable.h includes
> asm/pgtable.h is not practical.

Agreed.

> 
> >>>
> >>> In D128 V2 series these conversions were performed later in the series,
> >>> when platform specific pxdp_get() overrides were added via ptval_get(). 
> >>>
> >>> But to avoid this, we could just move pmdp_test_and_clear_young() inside
> >>> arch/arm64/mm/mmu.c file instead. Folding in the following change builds.
> >>
> >> I'm not sure if/how this might affect performance? Probably not significant
> >> since they are for higher levels, so not called as frequently as pte helpers.
> > 
> > Right, should not have much performance impact.
> > 
> >>
> >> Another option would be to implement arm64-specific pxdp_get() helpers in
> >> asm/pgtable.h to override the generic versions?
> > 
> > For now those will still be based on READ_ONCE() just like the default ones.
> > Although trying to drop similar redundancies from other platforms :)
> > 
> > https://lore.kernel.org/linux-mm/20260923043226.331880-1-anshuman.khandual@arm.com/
> > 
> > But given D128 is going to define pxdp_get() anyway it might make sense to
> > define them now and withdraw the above series :) Seems like s390 platform
> > also has similar situation.
> 
> Given the desired end state is that arm64 defines it's own pxdp_get() helpers, I
> think the best approach is just to define them now (as READ_ONCE()).

Sure, will do that.

> 
> Thanks,
> Ryan
> 
> 

[PATCH V2 2/4] arm64/mm: Use pudp_get() for PUD accesses

[Anshuman Khandual]

Replace READ_ONCE() with pudp_get() for PUD accesses in preparation for
supporting both D64 and D128 translation table formats.

READ_ONCE() cannot currently be used for 128-bit page table entries on
arm64 because it does not provide the required 128-bit single-copy
atomicity, causing builds to fail for accesses wider than 64 bits.

Although LDP/STP provide the required atomicity when FEAT_LSE is available
(as required by FEAT_D128), extending READ_ONCE() to support 128-bit
accesses is undesirable. READ_ONCE() is a general-purpose API, so doing so
could encourage other 128-bit users that would either fail to build in
configurations without D128 support or, if D128 becomes a runtime option,
silently permit tearing on systems without the required hardware support.

Instead, standardize PUD accesses on the existing page-table helpers. These
can be overridden on arm64 to provide 128-bit single-copy atomicity when
required. No functional change intended.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Cc: kasan-dev@googlegroups.com
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
 arch/arm64/include/asm/pgtable.h |  3 ++-
 arch/arm64/mm/fault.c            |  2 +-
 arch/arm64/mm/fixmap.c           |  2 +-
 arch/arm64/mm/hugetlbpage.c      |  4 ++--
 arch/arm64/mm/kasan_init.c       |  4 ++--
 arch/arm64/mm/mmu.c              | 20 ++++++++++----------
 arch/arm64/mm/pageattr.c         |  2 +-
 arch/arm64/mm/trans_pgd.c        |  4 ++--
 8 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index b953ea0ba3ed..ae41946c54f2 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -873,7 +873,8 @@ static inline pmd_t *pud_pgtable(pud_t pud)
 }
 
 /* Find an entry in the second-level page table. */
-#define pmd_offset_phys(dir, addr)	(pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t))
+#define pmd_offset_phys(dir, addr)	(pud_page_paddr(pudp_get(dir)) + \
+					 pmd_index(addr) * sizeof(pmd_t))
 
 #define pmd_set_fixmap(addr)		((pmd_t *)set_fixmap_offset(FIX_PMD, addr))
 #define pmd_set_fixmap_offset(pud, addr)	pmd_set_fixmap(pmd_offset_phys(pud, addr))
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 2757ee0c4300..435e2e14c070 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -181,7 +181,7 @@ static void show_pte(unsigned long addr)
 			break;
 
 		pudp = pud_offset_lockless(p4dp, p4d, addr);
-		pud = READ_ONCE(*pudp);
+		pud = pudp_get(pudp);
 		ptval_to_str(pxd_str, pud_val(pud));
 		pr_cont(", pud=%s", pxd_str);
 		if (pud_none(pud) || pud_bad(pud))
diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
index 3cdac8021d4f..d9a870836faf 100644
--- a/arch/arm64/mm/fixmap.c
+++ b/arch/arm64/mm/fixmap.c
@@ -56,7 +56,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr,
 					 unsigned long end)
 {
 	unsigned long next;
-	pud_t pud = READ_ONCE(*pudp);
+	pud_t pud = pudp_get(pudp);
 	pmd_t *pmdp;
 
 	if (pud_none(pud))
diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
index cdaa4500faf9..c9ad5e75b073 100644
--- a/arch/arm64/mm/hugetlbpage.c
+++ b/arch/arm64/mm/hugetlbpage.c
@@ -262,7 +262,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 		WARN_ON(addr & (sz - 1));
 		ptep = pte_alloc_huge(mm, pmdp, addr);
 	} else if (sz == PMD_SIZE) {
-		if (want_pmd_share(vma, addr) && pud_none(READ_ONCE(*pudp)))
+		if (want_pmd_share(vma, addr) && pud_none(pudp_get(pudp)))
 			ptep = huge_pmd_share(mm, vma, addr, pudp);
 		else
 			ptep = (pte_t *)pmd_alloc(mm, pudp, addr);
@@ -292,7 +292,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 		return NULL;
 
 	pudp = pud_offset(p4dp, addr);
-	pud = READ_ONCE(*pudp);
+	pud = pudp_get(pudp);
 	if (sz != PUD_SIZE && pud_none(pud))
 		return NULL;
 	/* hugepage or swap? */
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 7ca833c5de5e..aad29bcc7622 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -76,7 +76,7 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
 static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node,
 				      bool early)
 {
-	if (pud_none(READ_ONCE(*pudp))) {
+	if (pud_none(pudp_get(pudp))) {
 		phys_addr_t pmd_phys = early ?
 				__pa_symbol(kasan_early_shadow_pmd)
 					: kasan_alloc_zeroed_page(node);
@@ -150,7 +150,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
 	do {
 		next = pud_addr_end(addr, end);
 		kasan_pmd_populate(pudp, addr, next, node, early);
-	} while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp)));
+	} while (pudp++, addr = next, addr != end && pud_none(pudp_get(pudp)));
 }
 
 static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 95621913679e..22efaee79293 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -309,7 +309,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,
 {
 	int ret;
 	unsigned long next;
-	pud_t pud = READ_ONCE(*pudp);
+	pud_t pud = pudp_get(pudp);
 	pmd_t *pmdp;
 
 	/*
@@ -390,7 +390,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 	}
 
 	do {
-		pud_t old_pud = READ_ONCE(*pudp);
+		pud_t old_pud = pudp_get(pudp);
 
 		next = pud_addr_end(addr, end);
 
@@ -408,7 +408,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 			 * only allow updates to the permission attributes.
 			 */
 			BUG_ON(!pgattr_change_is_safe(pud_val(old_pud),
-						      READ_ONCE(pud_val(*pudp))));
+						      pud_val(pudp_get(pudp))));
 		} else {
 			ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot,
 						  pgtable_alloc, flags);
@@ -416,7 +416,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 				goto out;
 
 			VM_WARN_ON_ONCE(pud_val(old_pud) != 0 &&
-					pud_val(old_pud) != READ_ONCE(pud_val(*pudp)));
+					pud_val(old_pud) != pud_val(pudp_get(pudp)));
 		}
 		phys += next - addr;
 	} while (pudp++, addr = next, addr != end);
@@ -1591,7 +1591,7 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr,
 	do {
 		next = pud_addr_end(addr, end);
 		pudp = pud_offset(p4dp, addr);
-		pud = READ_ONCE(*pudp);
+		pud = pudp_get(pudp);
 		if (pud_none(pud))
 			continue;
 
@@ -1748,7 +1748,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr,
 	do {
 		next = pud_addr_end(addr, end);
 		pudp = pud_offset(p4dp, addr);
-		pud = READ_ONCE(*pudp);
+		pud = pudp_get(pudp);
 		if (pud_none(pud))
 			continue;
 
@@ -1769,7 +1769,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr,
 	 */
 	pudp = pud_offset(p4dp, 0UL);
 	for (i = 0; i < PTRS_PER_PUD; i++) {
-		if (!pud_none(READ_ONCE(pudp[i])))
+		if (!pud_none(pudp_get(pudp + i)))
 			return;
 	}
 
@@ -1867,7 +1867,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot)
 	pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot));
 
 	/* Only allow permission changes for now */
-	if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)),
+	if (!pgattr_change_is_safe(pud_val(pudp_get(pudp)),
 				   pud_val(new_pud)))
 		return 0;
 
@@ -1898,7 +1898,7 @@ void p4d_clear_huge(p4d_t *p4dp)
 
 int pud_clear_huge(pud_t *pudp)
 {
-	if (!pud_leaf(READ_ONCE(*pudp)))
+	if (!pud_leaf(pudp_get(pudp)))
 		return 0;
 	pud_clear(pudp);
 	return 1;
@@ -1938,7 +1938,7 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 	pud_t pud;
 	unsigned long next, end;
 
-	pud = READ_ONCE(*pudp);
+	pud = pudp_get(pudp);
 
 	if (!pud_table(pud)) {
 		VM_WARN_ON(1);
diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
index 0ca07bd5ded9..07b2fa4de57f 100644
--- a/arch/arm64/mm/pageattr.c
+++ b/arch/arm64/mm/pageattr.c
@@ -407,7 +407,7 @@ bool kernel_page_present(struct page *page)
 		return false;
 
 	pudp = pud_offset(p4dp, addr);
-	pud = READ_ONCE(*pudp);
+	pud = pudp_get(pudp);
 	if (pud_none(pud))
 		return false;
 	if (pud_leaf(pud))
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index b27b2d2c20c3..d119119455f1 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -64,7 +64,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
 	unsigned long next;
 	unsigned long addr = start;
 
-	if (pud_none(READ_ONCE(*dst_pudp))) {
+	if (pud_none(pudp_get(dst_pudp))) {
 		dst_pmdp = trans_alloc(info);
 		if (!dst_pmdp)
 			return -ENOMEM;
@@ -109,7 +109,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
 
 	src_pudp = pud_offset(src_p4dp, start);
 	do {
-		pud_t pud = READ_ONCE(*src_pudp);
+		pud_t pud = pudp_get(src_pudp);
 
 		next = pud_addr_end(addr, end);
 		if (pud_none(pud))
-- 
2.43.0

[PATCH V2 3/4] arm64/mm: Use p4dp_get() for P4D accesses

[Anshuman Khandual]

Replace READ_ONCE() with p4dp_get() for P4D accesses in preparation for
supporting both D64 and D128 translation table formats.

READ_ONCE() cannot currently be used for 128-bit page table entries on
arm64 because it does not provide the required 128-bit single-copy
atomicity, causing builds to fail for accesses wider than 64 bits.

Although LDP/STP provide the required atomicity when FEAT_LSE is
available (as required by FEAT_D128), extending READ_ONCE() to support
128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
so doing so could encourage other 128-bit users that would either fail
to build in configurations without D128 support or, if D128 becomes a
runtime option, silently permit tearing on systems without the required
hardware support.

Instead, standardize P4D accesses on the existing page-table helpers.
These can be overridden on arm64 to provide 128-bit single-copy
atomicity when required. No functional change intended.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Cc: kasan-dev@googlegroups.com
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
 arch/arm64/mm/fault.c       |  2 +-
 arch/arm64/mm/fixmap.c      |  2 +-
 arch/arm64/mm/hugetlbpage.c |  2 +-
 arch/arm64/mm/kasan_init.c  |  4 ++--
 arch/arm64/mm/mmu.c         | 12 ++++++------
 arch/arm64/mm/pageattr.c    |  2 +-
 arch/arm64/mm/trans_pgd.c   |  4 ++--
 7 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 435e2e14c070..a6afd8929a10 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -174,7 +174,7 @@ static void show_pte(unsigned long addr)
 			break;
 
 		p4dp = p4d_offset_lockless(pgdp, pgd, addr);
-		p4d = READ_ONCE(*p4dp);
+		p4d = p4dp_get(p4dp);
 		ptval_to_str(pxd_str, p4d_val(p4d));
 		pr_cont(", p4d=%s", pxd_str);
 		if (p4d_none(p4d) || p4d_bad(p4d))
diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
index d9a870836faf..2a0fe0b0f771 100644
--- a/arch/arm64/mm/fixmap.c
+++ b/arch/arm64/mm/fixmap.c
@@ -74,7 +74,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr,
 static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr,
 					 unsigned long end)
 {
-	p4d_t p4d = READ_ONCE(*p4dp);
+	p4d_t p4d = p4dp_get(p4dp);
 	pud_t *pudp;
 
 	if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) &&
diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
index c9ad5e75b073..5771bf49e1fe 100644
--- a/arch/arm64/mm/hugetlbpage.c
+++ b/arch/arm64/mm/hugetlbpage.c
@@ -288,7 +288,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 		return NULL;
 
 	p4dp = p4d_offset(pgdp, addr);
-	if (!p4d_present(READ_ONCE(*p4dp)))
+	if (!p4d_present(p4dp_get(p4dp)))
 		return NULL;
 
 	pudp = pud_offset(p4dp, addr);
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index aad29bcc7622..4dd8c1186471 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -89,7 +89,7 @@ static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node,
 static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
 				      bool early)
 {
-	if (p4d_none(READ_ONCE(*p4dp))) {
+	if (p4d_none(p4dp_get(p4dp))) {
 		phys_addr_t pud_phys = early ?
 				__pa_symbol(kasan_early_shadow_pud)
 					: kasan_alloc_zeroed_page(node);
@@ -162,7 +162,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,
 	do {
 		next = p4d_addr_end(addr, end);
 		kasan_pud_populate(p4dp, addr, next, node, early);
-	} while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp)));
+	} while (p4dp++, addr = next, addr != end && p4d_none(p4dp_get(p4dp)));
 }
 
 static void __init kasan_pgd_populate(unsigned long addr, unsigned long end,
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 22efaee79293..83709d987e7c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -367,7 +367,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 {
 	int ret = 0;
 	unsigned long next;
-	p4d_t p4d = READ_ONCE(*p4dp);
+	p4d_t p4d = p4dp_get(p4dp);
 	pud_t *pudp;
 
 	if (p4d_none(p4d)) {
@@ -457,7 +457,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
 	}
 
 	do {
-		p4d_t old_p4d = READ_ONCE(*p4dp);
+		p4d_t old_p4d = p4dp_get(p4dp);
 
 		next = p4d_addr_end(addr, end);
 
@@ -467,7 +467,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
 			goto out;
 
 		VM_WARN_ON_ONCE(p4d_val(old_p4d) != 0 &&
-				p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp)));
+				p4d_val(old_p4d) != (p4d_val(p4dp_get(p4dp))));
 
 		phys += next - addr;
 	} while (p4dp++, addr = next, addr != end);
@@ -1622,7 +1622,7 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr,
 	do {
 		next = p4d_addr_end(addr, end);
 		p4dp = p4d_offset(pgdp, addr);
-		p4d = READ_ONCE(*p4dp);
+		p4d = p4dp_get(p4dp);
 		if (p4d_none(p4d))
 			continue;
 
@@ -1788,7 +1788,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
 	do {
 		next = p4d_addr_end(addr, end);
 		p4dp = p4d_offset(pgdp, addr);
-		p4d = READ_ONCE(*p4dp);
+		p4d = p4dp_get(p4dp);
 		if (p4d_none(p4d))
 			continue;
 
@@ -1809,7 +1809,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
 	 */
 	p4dp = p4d_offset(pgdp, 0UL);
 	for (i = 0; i < PTRS_PER_P4D; i++) {
-		if (!p4d_none(READ_ONCE(p4dp[i])))
+		if (!p4d_none(p4dp_get(p4dp + i)))
 			return;
 	}
 
diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
index 07b2fa4de57f..826856ef64a8 100644
--- a/arch/arm64/mm/pageattr.c
+++ b/arch/arm64/mm/pageattr.c
@@ -403,7 +403,7 @@ bool kernel_page_present(struct page *page)
 		return false;
 
 	p4dp = p4d_offset(pgdp, addr);
-	if (p4d_none(READ_ONCE(*p4dp)))
+	if (p4d_none(p4dp_get(p4dp)))
 		return false;
 
 	pudp = pud_offset(p4dp, addr);
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index d119119455f1..7afe2beca4ba 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -99,7 +99,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
 	unsigned long next;
 	unsigned long addr = start;
 
-	if (p4d_none(READ_ONCE(*dst_p4dp))) {
+	if (p4d_none(p4dp_get(dst_p4dp))) {
 		dst_pudp = trans_alloc(info);
 		if (!dst_pudp)
 			return -ENOMEM;
@@ -145,7 +145,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
 	src_p4dp = p4d_offset(src_pgdp, start);
 	do {
 		next = p4d_addr_end(addr, end);
-		if (p4d_none(READ_ONCE(*src_p4dp)))
+		if (p4d_none(p4dp_get(src_p4dp)))
 			continue;
 		if (copy_pud(info, dst_p4dp, src_p4dp, addr, next))
 			return -ENOMEM;
-- 
2.43.0

Re: [PATCH V2 3/4] arm64/mm: Use p4dp_get() for P4D accesses

[Ryan Roberts]

On 22/09/2026 07:16, Anshuman Khandual wrote:
> Replace READ_ONCE() with p4dp_get() for P4D accesses in preparation for
> supporting both D64 and D128 translation table formats.
> 
> READ_ONCE() cannot currently be used for 128-bit page table entries on
> arm64 because it does not provide the required 128-bit single-copy
> atomicity, causing builds to fail for accesses wider than 64 bits.
> 
> Although LDP/STP provide the required atomicity when FEAT_LSE is
> available (as required by FEAT_D128), extending READ_ONCE() to support
> 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> so doing so could encourage other 128-bit users that would either fail
> to build in configurations without D128 support or, if D128 becomes a
> runtime option, silently permit tearing on systems without the required
> hardware support.
> 
> Instead, standardize P4D accesses on the existing page-table helpers.
> These can be overridden on arm64 to provide 128-bit single-copy
> atomicity when required. No functional change intended.

I notice you still have uncoverted READ_ONCE(*p4dp) in:

 - pud_offset_phys()
 - pud_offset()

Is that intentional? If so, perhaps we need a comment explain why it's not using
the helper?

Thanks,
Ryan


> 
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Ryan Roberts <ryan.roberts@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-kernel@vger.kernel.org
> Cc: kasan-dev@googlegroups.com
> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> ---
>  arch/arm64/mm/fault.c       |  2 +-
>  arch/arm64/mm/fixmap.c      |  2 +-
>  arch/arm64/mm/hugetlbpage.c |  2 +-
>  arch/arm64/mm/kasan_init.c  |  4 ++--
>  arch/arm64/mm/mmu.c         | 12 ++++++------
>  arch/arm64/mm/pageattr.c    |  2 +-
>  arch/arm64/mm/trans_pgd.c   |  4 ++--
>  7 files changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 435e2e14c070..a6afd8929a10 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -174,7 +174,7 @@ static void show_pte(unsigned long addr)
>  			break;
>  
>  		p4dp = p4d_offset_lockless(pgdp, pgd, addr);
> -		p4d = READ_ONCE(*p4dp);
> +		p4d = p4dp_get(p4dp);
>  		ptval_to_str(pxd_str, p4d_val(p4d));
>  		pr_cont(", p4d=%s", pxd_str);
>  		if (p4d_none(p4d) || p4d_bad(p4d))
> diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
> index d9a870836faf..2a0fe0b0f771 100644
> --- a/arch/arm64/mm/fixmap.c
> +++ b/arch/arm64/mm/fixmap.c
> @@ -74,7 +74,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr,
>  static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr,
>  					 unsigned long end)
>  {
> -	p4d_t p4d = READ_ONCE(*p4dp);
> +	p4d_t p4d = p4dp_get(p4dp);
>  	pud_t *pudp;
>  
>  	if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) &&
> diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
> index c9ad5e75b073..5771bf49e1fe 100644
> --- a/arch/arm64/mm/hugetlbpage.c
> +++ b/arch/arm64/mm/hugetlbpage.c
> @@ -288,7 +288,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
>  		return NULL;
>  
>  	p4dp = p4d_offset(pgdp, addr);
> -	if (!p4d_present(READ_ONCE(*p4dp)))
> +	if (!p4d_present(p4dp_get(p4dp)))
>  		return NULL;
>  
>  	pudp = pud_offset(p4dp, addr);
> diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
> index aad29bcc7622..4dd8c1186471 100644
> --- a/arch/arm64/mm/kasan_init.c
> +++ b/arch/arm64/mm/kasan_init.c
> @@ -89,7 +89,7 @@ static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node,
>  static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
>  				      bool early)
>  {
> -	if (p4d_none(READ_ONCE(*p4dp))) {
> +	if (p4d_none(p4dp_get(p4dp))) {
>  		phys_addr_t pud_phys = early ?
>  				__pa_symbol(kasan_early_shadow_pud)
>  					: kasan_alloc_zeroed_page(node);
> @@ -162,7 +162,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,
>  	do {
>  		next = p4d_addr_end(addr, end);
>  		kasan_pud_populate(p4dp, addr, next, node, early);
> -	} while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp)));
> +	} while (p4dp++, addr = next, addr != end && p4d_none(p4dp_get(p4dp)));
>  }
>  
>  static void __init kasan_pgd_populate(unsigned long addr, unsigned long end,
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index 22efaee79293..83709d987e7c 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -367,7 +367,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
>  {
>  	int ret = 0;
>  	unsigned long next;
> -	p4d_t p4d = READ_ONCE(*p4dp);
> +	p4d_t p4d = p4dp_get(p4dp);
>  	pud_t *pudp;
>  
>  	if (p4d_none(p4d)) {
> @@ -457,7 +457,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
>  	}
>  
>  	do {
> -		p4d_t old_p4d = READ_ONCE(*p4dp);
> +		p4d_t old_p4d = p4dp_get(p4dp);
>  
>  		next = p4d_addr_end(addr, end);
>  
> @@ -467,7 +467,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
>  			goto out;
>  
>  		VM_WARN_ON_ONCE(p4d_val(old_p4d) != 0 &&
> -				p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp)));
> +				p4d_val(old_p4d) != (p4d_val(p4dp_get(p4dp))));
>  
>  		phys += next - addr;
>  	} while (p4dp++, addr = next, addr != end);
> @@ -1622,7 +1622,7 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr,
>  	do {
>  		next = p4d_addr_end(addr, end);
>  		p4dp = p4d_offset(pgdp, addr);
> -		p4d = READ_ONCE(*p4dp);
> +		p4d = p4dp_get(p4dp);
>  		if (p4d_none(p4d))
>  			continue;
>  
> @@ -1788,7 +1788,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
>  	do {
>  		next = p4d_addr_end(addr, end);
>  		p4dp = p4d_offset(pgdp, addr);
> -		p4d = READ_ONCE(*p4dp);
> +		p4d = p4dp_get(p4dp);
>  		if (p4d_none(p4d))
>  			continue;
>  
> @@ -1809,7 +1809,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
>  	 */
>  	p4dp = p4d_offset(pgdp, 0UL);
>  	for (i = 0; i < PTRS_PER_P4D; i++) {
> -		if (!p4d_none(READ_ONCE(p4dp[i])))
> +		if (!p4d_none(p4dp_get(p4dp + i)))
>  			return;
>  	}
>  
> diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
> index 07b2fa4de57f..826856ef64a8 100644
> --- a/arch/arm64/mm/pageattr.c
> +++ b/arch/arm64/mm/pageattr.c
> @@ -403,7 +403,7 @@ bool kernel_page_present(struct page *page)
>  		return false;
>  
>  	p4dp = p4d_offset(pgdp, addr);
> -	if (p4d_none(READ_ONCE(*p4dp)))
> +	if (p4d_none(p4dp_get(p4dp)))
>  		return false;
>  
>  	pudp = pud_offset(p4dp, addr);
> diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
> index d119119455f1..7afe2beca4ba 100644
> --- a/arch/arm64/mm/trans_pgd.c
> +++ b/arch/arm64/mm/trans_pgd.c
> @@ -99,7 +99,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
>  	unsigned long next;
>  	unsigned long addr = start;
>  
> -	if (p4d_none(READ_ONCE(*dst_p4dp))) {
> +	if (p4d_none(p4dp_get(dst_p4dp))) {
>  		dst_pudp = trans_alloc(info);
>  		if (!dst_pudp)
>  			return -ENOMEM;
> @@ -145,7 +145,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
>  	src_p4dp = p4d_offset(src_pgdp, start);
>  	do {
>  		next = p4d_addr_end(addr, end);
> -		if (p4d_none(READ_ONCE(*src_p4dp)))
> +		if (p4d_none(p4dp_get(src_p4dp)))
>  			continue;
>  		if (copy_pud(info, dst_p4dp, src_p4dp, addr, next))
>  			return -ENOMEM;

Re: [PATCH V2 3/4] arm64/mm: Use p4dp_get() for P4D accesses

[Anshuman Khandual]

On Wed, Sep 23, 2026 at 09:30:26AM +0100, Ryan Roberts wrote:
> On 22/09/2026 07:16, Anshuman Khandual wrote:
> > Replace READ_ONCE() with p4dp_get() for P4D accesses in preparation for
> > supporting both D64 and D128 translation table formats.
> > 
> > READ_ONCE() cannot currently be used for 128-bit page table entries on
> > arm64 because it does not provide the required 128-bit single-copy
> > atomicity, causing builds to fail for accesses wider than 64 bits.
> > 
> > Although LDP/STP provide the required atomicity when FEAT_LSE is
> > available (as required by FEAT_D128), extending READ_ONCE() to support
> > 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> > so doing so could encourage other 128-bit users that would either fail
> > to build in configurations without D128 support or, if D128 becomes a
> > runtime option, silently permit tearing on systems without the required
> > hardware support.
> > 
> > Instead, standardize P4D accesses on the existing page-table helpers.
> > These can be overridden on arm64 to provide 128-bit single-copy
> > atomicity when required. No functional change intended.
> 
> I notice you still have uncoverted READ_ONCE(*p4dp) in:
> 
>  - pud_offset_phys()
>  - pud_offset()
> 
> Is that intentional? If so, perhaps we need a comment explain why it's not using
> the helper?

Platform overrride for p4dp_get() should solve the problem.

> 
> Thanks,
> Ryan
> 
> 
> > 
> > Cc: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Will Deacon <will@kernel.org>
> > Cc: Ryan Roberts <ryan.roberts@arm.com>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: linux-arm-kernel@lists.infradead.org
> > Cc: linux-kernel@vger.kernel.org
> > Cc: kasan-dev@googlegroups.com
> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> > ---
> >  arch/arm64/mm/fault.c       |  2 +-
> >  arch/arm64/mm/fixmap.c      |  2 +-
> >  arch/arm64/mm/hugetlbpage.c |  2 +-
> >  arch/arm64/mm/kasan_init.c  |  4 ++--
> >  arch/arm64/mm/mmu.c         | 12 ++++++------
> >  arch/arm64/mm/pageattr.c    |  2 +-
> >  arch/arm64/mm/trans_pgd.c   |  4 ++--
> >  7 files changed, 14 insertions(+), 14 deletions(-)
> > 
> > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> > index 435e2e14c070..a6afd8929a10 100644
> > --- a/arch/arm64/mm/fault.c
> > +++ b/arch/arm64/mm/fault.c
> > @@ -174,7 +174,7 @@ static void show_pte(unsigned long addr)
> >  			break;
> >  
> >  		p4dp = p4d_offset_lockless(pgdp, pgd, addr);
> > -		p4d = READ_ONCE(*p4dp);
> > +		p4d = p4dp_get(p4dp);
> >  		ptval_to_str(pxd_str, p4d_val(p4d));
> >  		pr_cont(", p4d=%s", pxd_str);
> >  		if (p4d_none(p4d) || p4d_bad(p4d))
> > diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
> > index d9a870836faf..2a0fe0b0f771 100644
> > --- a/arch/arm64/mm/fixmap.c
> > +++ b/arch/arm64/mm/fixmap.c
> > @@ -74,7 +74,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr,
> >  static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr,
> >  					 unsigned long end)
> >  {
> > -	p4d_t p4d = READ_ONCE(*p4dp);
> > +	p4d_t p4d = p4dp_get(p4dp);
> >  	pud_t *pudp;
> >  
> >  	if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) &&
> > diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
> > index c9ad5e75b073..5771bf49e1fe 100644
> > --- a/arch/arm64/mm/hugetlbpage.c
> > +++ b/arch/arm64/mm/hugetlbpage.c
> > @@ -288,7 +288,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
> >  		return NULL;
> >  
> >  	p4dp = p4d_offset(pgdp, addr);
> > -	if (!p4d_present(READ_ONCE(*p4dp)))
> > +	if (!p4d_present(p4dp_get(p4dp)))
> >  		return NULL;
> >  
> >  	pudp = pud_offset(p4dp, addr);
> > diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
> > index aad29bcc7622..4dd8c1186471 100644
> > --- a/arch/arm64/mm/kasan_init.c
> > +++ b/arch/arm64/mm/kasan_init.c
> > @@ -89,7 +89,7 @@ static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node,
> >  static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
> >  				      bool early)
> >  {
> > -	if (p4d_none(READ_ONCE(*p4dp))) {
> > +	if (p4d_none(p4dp_get(p4dp))) {
> >  		phys_addr_t pud_phys = early ?
> >  				__pa_symbol(kasan_early_shadow_pud)
> >  					: kasan_alloc_zeroed_page(node);
> > @@ -162,7 +162,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,
> >  	do {
> >  		next = p4d_addr_end(addr, end);
> >  		kasan_pud_populate(p4dp, addr, next, node, early);
> > -	} while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp)));
> > +	} while (p4dp++, addr = next, addr != end && p4d_none(p4dp_get(p4dp)));
> >  }
> >  
> >  static void __init kasan_pgd_populate(unsigned long addr, unsigned long end,
> > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > index 22efaee79293..83709d987e7c 100644
> > --- a/arch/arm64/mm/mmu.c
> > +++ b/arch/arm64/mm/mmu.c
> > @@ -367,7 +367,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
> >  {
> >  	int ret = 0;
> >  	unsigned long next;
> > -	p4d_t p4d = READ_ONCE(*p4dp);
> > +	p4d_t p4d = p4dp_get(p4dp);
> >  	pud_t *pudp;
> >  
> >  	if (p4d_none(p4d)) {
> > @@ -457,7 +457,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
> >  	}
> >  
> >  	do {
> > -		p4d_t old_p4d = READ_ONCE(*p4dp);
> > +		p4d_t old_p4d = p4dp_get(p4dp);
> >  
> >  		next = p4d_addr_end(addr, end);
> >  
> > @@ -467,7 +467,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
> >  			goto out;
> >  
> >  		VM_WARN_ON_ONCE(p4d_val(old_p4d) != 0 &&
> > -				p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp)));
> > +				p4d_val(old_p4d) != (p4d_val(p4dp_get(p4dp))));
> >  
> >  		phys += next - addr;
> >  	} while (p4dp++, addr = next, addr != end);
> > @@ -1622,7 +1622,7 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr,
> >  	do {
> >  		next = p4d_addr_end(addr, end);
> >  		p4dp = p4d_offset(pgdp, addr);
> > -		p4d = READ_ONCE(*p4dp);
> > +		p4d = p4dp_get(p4dp);
> >  		if (p4d_none(p4d))
> >  			continue;
> >  
> > @@ -1788,7 +1788,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
> >  	do {
> >  		next = p4d_addr_end(addr, end);
> >  		p4dp = p4d_offset(pgdp, addr);
> > -		p4d = READ_ONCE(*p4dp);
> > +		p4d = p4dp_get(p4dp);
> >  		if (p4d_none(p4d))
> >  			continue;
> >  
> > @@ -1809,7 +1809,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
> >  	 */
> >  	p4dp = p4d_offset(pgdp, 0UL);
> >  	for (i = 0; i < PTRS_PER_P4D; i++) {
> > -		if (!p4d_none(READ_ONCE(p4dp[i])))
> > +		if (!p4d_none(p4dp_get(p4dp + i)))
> >  			return;
> >  	}
> >  
> > diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
> > index 07b2fa4de57f..826856ef64a8 100644
> > --- a/arch/arm64/mm/pageattr.c
> > +++ b/arch/arm64/mm/pageattr.c
> > @@ -403,7 +403,7 @@ bool kernel_page_present(struct page *page)
> >  		return false;
> >  
> >  	p4dp = p4d_offset(pgdp, addr);
> > -	if (p4d_none(READ_ONCE(*p4dp)))
> > +	if (p4d_none(p4dp_get(p4dp)))
> >  		return false;
> >  
> >  	pudp = pud_offset(p4dp, addr);
> > diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
> > index d119119455f1..7afe2beca4ba 100644
> > --- a/arch/arm64/mm/trans_pgd.c
> > +++ b/arch/arm64/mm/trans_pgd.c
> > @@ -99,7 +99,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
> >  	unsigned long next;
> >  	unsigned long addr = start;
> >  
> > -	if (p4d_none(READ_ONCE(*dst_p4dp))) {
> > +	if (p4d_none(p4dp_get(dst_p4dp))) {
> >  		dst_pudp = trans_alloc(info);
> >  		if (!dst_pudp)
> >  			return -ENOMEM;
> > @@ -145,7 +145,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
> >  	src_p4dp = p4d_offset(src_pgdp, start);
> >  	do {
> >  		next = p4d_addr_end(addr, end);
> > -		if (p4d_none(READ_ONCE(*src_p4dp)))
> > +		if (p4d_none(p4dp_get(src_p4dp)))
> >  			continue;
> >  		if (copy_pud(info, dst_p4dp, src_p4dp, addr, next))
> >  			return -ENOMEM;
> 

[PATCH V2 4/4] arm64/mm: Use pgdp_get() for PGD accesses

[Anshuman Khandual]

Replace READ_ONCE() with pgdp_get() for PGD accesses in preparation for
supporting both D64 and D128 translation table formats.

READ_ONCE() cannot currently be used for 128-bit page table entries on
arm64 because it does not provide the required 128-bit single-copy
atomicity, causing builds to fail for accesses wider than 64 bits.

Although LDP/STP provide the required atomicity when FEAT_LSE is
available (as required by FEAT_D128), extending READ_ONCE() to support
128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
so doing so could encourage other 128-bit users that would either fail
to build in configurations without D128 support or, if D128 becomes a
runtime option, silently permit tearing on systems without the required
hardware support.

Instead, standardize PGD accesses on the existing page-table helpers.
These can be overridden on arm64 to provide 128-bit single-copy
atomicity when required. No functional change intended.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Cc: kasan-dev@googlegroups.com
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
 arch/arm64/mm/fault.c       | 2 +-
 arch/arm64/mm/hugetlbpage.c | 2 +-
 arch/arm64/mm/kasan_init.c  | 6 +++---
 arch/arm64/mm/mmu.c         | 6 +++---
 arch/arm64/mm/pageattr.c    | 2 +-
 arch/arm64/mm/trans_pgd.c   | 4 ++--
 6 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index a6afd8929a10..43f25418275a 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -160,7 +160,7 @@ static void show_pte(unsigned long addr)
 	guard(irqsave)();
 
 	pgdp = pgd_offset(mm, addr);
-	pgd = READ_ONCE(*pgdp);
+	pgd = pgdp_get(pgdp);
 	ptval_to_str(pxd_str, pgd_val(pgd));
 	pr_alert("[%016lx] pgd=%s", addr, pxd_str);
 
diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
index 5771bf49e1fe..e05e5b41ae50 100644
--- a/arch/arm64/mm/hugetlbpage.c
+++ b/arch/arm64/mm/hugetlbpage.c
@@ -284,7 +284,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 	pmd_t *pmdp, pmd;
 
 	pgdp = pgd_offset(mm, addr);
-	if (!pgd_present(READ_ONCE(*pgdp)))
+	if (!pgd_present(pgdp_get(pgdp)))
 		return NULL;
 
 	p4dp = p4d_offset(pgdp, addr);
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index 4dd8c1186471..f61f5bc09467 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -102,7 +102,7 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
 static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int node,
 				      bool early)
 {
-	if (pgd_none(READ_ONCE(*pgdp))) {
+	if (pgd_none(pgdp_get(pgdp))) {
 		phys_addr_t p4d_phys = early ?
 				__pa_symbol(kasan_early_shadow_p4d)
 					: kasan_alloc_zeroed_page(node);
@@ -256,7 +256,7 @@ static int __init root_level_idx(u64 addr)
 static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pud)
 {
 	int idx = root_level_idx(addr);
-	pgd_t pgd = READ_ONCE(swapper_pg_dir[idx]);
+	pgd_t pgd = pgdp_get(swapper_pg_dir + idx);
 	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
 
 	memcpy(pud, pudp, PAGE_SIZE);
@@ -280,7 +280,7 @@ static int __init next_level_idx(u64 addr)
  */
 static void __init clear_next_level(int pgd_idx, int start, int end)
 {
-	pgd_t pgd = READ_ONCE(swapper_pg_dir[pgd_idx]);
+	pgd_t pgd = pgdp_get(swapper_pg_dir + pgd_idx);
 	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
 
 	memset(&pudp[start], 0, (end - start) * sizeof(pud_t));
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 83709d987e7c..76d8d320aeed 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -434,7 +434,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
 {
 	int ret;
 	unsigned long next;
-	pgd_t pgd = READ_ONCE(*pgdp);
+	pgd_t pgd = pgdp_get(pgdp);
 	p4d_t *p4dp;
 
 	if (pgd_none(pgd)) {
@@ -1649,7 +1649,7 @@ static void unmap_hotplug_range(unsigned long addr, unsigned long end,
 	do {
 		next = pgd_addr_end(addr, end);
 		pgdp = pgd_offset_k(addr);
-		pgd = READ_ONCE(*pgdp);
+		pgd = pgdp_get(pgdp);
 		if (pgd_none(pgd))
 			continue;
 
@@ -1827,7 +1827,7 @@ static void free_empty_tables(unsigned long addr, unsigned long end,
 	do {
 		next = pgd_addr_end(addr, end);
 		pgdp = pgd_offset_k(addr);
-		pgd = READ_ONCE(*pgdp);
+		pgd = pgdp_get(pgdp);
 		if (pgd_none(pgd))
 			continue;
 
diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
index 826856ef64a8..dfa1356daac2 100644
--- a/arch/arm64/mm/pageattr.c
+++ b/arch/arm64/mm/pageattr.c
@@ -399,7 +399,7 @@ bool kernel_page_present(struct page *page)
 	unsigned long addr = (unsigned long)page_address(page);
 
 	pgdp = pgd_offset_k(addr);
-	if (pgd_none(READ_ONCE(*pgdp)))
+	if (pgd_none(pgdp_get(pgdp)))
 		return false;
 
 	p4dp = p4d_offset(pgdp, addr);
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index 7afe2beca4ba..06470d690f9f 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -134,7 +134,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
 	unsigned long next;
 	unsigned long addr = start;
 
-	if (pgd_none(READ_ONCE(*dst_pgdp))) {
+	if (pgd_none(pgdp_get(dst_pgdp))) {
 		dst_p4dp = trans_alloc(info);
 		if (!dst_p4dp)
 			return -ENOMEM;
@@ -164,7 +164,7 @@ static int copy_page_tables(struct trans_pgd_info *info, pgd_t *dst_pgdp,
 	dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
 	do {
 		next = pgd_addr_end(addr, end);
-		if (pgd_none(READ_ONCE(*src_pgdp)))
+		if (pgd_none(pgdp_get(src_pgdp)))
 			continue;
 		if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next))
 			return -ENOMEM;
-- 
2.43.0

Re: [PATCH V2 4/4] arm64/mm: Use pgdp_get() for PGD accesses

[Ryan Roberts]

On 22/09/2026 07:16, Anshuman Khandual wrote:
> Replace READ_ONCE() with pgdp_get() for PGD accesses in preparation for
> supporting both D64 and D128 translation table formats.
> 
> READ_ONCE() cannot currently be used for 128-bit page table entries on
> arm64 because it does not provide the required 128-bit single-copy
> atomicity, causing builds to fail for accesses wider than 64 bits.
> 
> Although LDP/STP provide the required atomicity when FEAT_LSE is
> available (as required by FEAT_D128), extending READ_ONCE() to support
> 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> so doing so could encourage other 128-bit users that would either fail
> to build in configurations without D128 support or, if D128 becomes a
> runtime option, silently permit tearing on systems without the required
> hardware support.
> 
> Instead, standardize PGD accesses on the existing page-table helpers.
> These can be overridden on arm64 to provide 128-bit single-copy
> atomicity when required. No functional change intended.

I notice you have unconverted READ_ONCE(*pgdp) in:

 - p4d_offset_phys()
 - p4d_offset()

Is that intentional?

Thanks,
Ryan


> 
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Ryan Roberts <ryan.roberts@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: linux-arm-kernel@lists.infradead.org
> Cc: linux-kernel@vger.kernel.org
> Cc: kasan-dev@googlegroups.com
> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> ---
>  arch/arm64/mm/fault.c       | 2 +-
>  arch/arm64/mm/hugetlbpage.c | 2 +-
>  arch/arm64/mm/kasan_init.c  | 6 +++---
>  arch/arm64/mm/mmu.c         | 6 +++---
>  arch/arm64/mm/pageattr.c    | 2 +-
>  arch/arm64/mm/trans_pgd.c   | 4 ++--
>  6 files changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index a6afd8929a10..43f25418275a 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -160,7 +160,7 @@ static void show_pte(unsigned long addr)
>  	guard(irqsave)();
>  
>  	pgdp = pgd_offset(mm, addr);
> -	pgd = READ_ONCE(*pgdp);
> +	pgd = pgdp_get(pgdp);
>  	ptval_to_str(pxd_str, pgd_val(pgd));
>  	pr_alert("[%016lx] pgd=%s", addr, pxd_str);
>  
> diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
> index 5771bf49e1fe..e05e5b41ae50 100644
> --- a/arch/arm64/mm/hugetlbpage.c
> +++ b/arch/arm64/mm/hugetlbpage.c
> @@ -284,7 +284,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
>  	pmd_t *pmdp, pmd;
>  
>  	pgdp = pgd_offset(mm, addr);
> -	if (!pgd_present(READ_ONCE(*pgdp)))
> +	if (!pgd_present(pgdp_get(pgdp)))
>  		return NULL;
>  
>  	p4dp = p4d_offset(pgdp, addr);
> diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
> index 4dd8c1186471..f61f5bc09467 100644
> --- a/arch/arm64/mm/kasan_init.c
> +++ b/arch/arm64/mm/kasan_init.c
> @@ -102,7 +102,7 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
>  static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int node,
>  				      bool early)
>  {
> -	if (pgd_none(READ_ONCE(*pgdp))) {
> +	if (pgd_none(pgdp_get(pgdp))) {
>  		phys_addr_t p4d_phys = early ?
>  				__pa_symbol(kasan_early_shadow_p4d)
>  					: kasan_alloc_zeroed_page(node);
> @@ -256,7 +256,7 @@ static int __init root_level_idx(u64 addr)
>  static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pud)
>  {
>  	int idx = root_level_idx(addr);
> -	pgd_t pgd = READ_ONCE(swapper_pg_dir[idx]);
> +	pgd_t pgd = pgdp_get(swapper_pg_dir + idx);
>  	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
>  
>  	memcpy(pud, pudp, PAGE_SIZE);
> @@ -280,7 +280,7 @@ static int __init next_level_idx(u64 addr)
>   */
>  static void __init clear_next_level(int pgd_idx, int start, int end)
>  {
> -	pgd_t pgd = READ_ONCE(swapper_pg_dir[pgd_idx]);
> +	pgd_t pgd = pgdp_get(swapper_pg_dir + pgd_idx);
>  	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
>  
>  	memset(&pudp[start], 0, (end - start) * sizeof(pud_t));
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index 83709d987e7c..76d8d320aeed 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -434,7 +434,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
>  {
>  	int ret;
>  	unsigned long next;
> -	pgd_t pgd = READ_ONCE(*pgdp);
> +	pgd_t pgd = pgdp_get(pgdp);
>  	p4d_t *p4dp;
>  
>  	if (pgd_none(pgd)) {
> @@ -1649,7 +1649,7 @@ static void unmap_hotplug_range(unsigned long addr, unsigned long end,
>  	do {
>  		next = pgd_addr_end(addr, end);
>  		pgdp = pgd_offset_k(addr);
> -		pgd = READ_ONCE(*pgdp);
> +		pgd = pgdp_get(pgdp);
>  		if (pgd_none(pgd))
>  			continue;
>  
> @@ -1827,7 +1827,7 @@ static void free_empty_tables(unsigned long addr, unsigned long end,
>  	do {
>  		next = pgd_addr_end(addr, end);
>  		pgdp = pgd_offset_k(addr);
> -		pgd = READ_ONCE(*pgdp);
> +		pgd = pgdp_get(pgdp);
>  		if (pgd_none(pgd))
>  			continue;
>  
> diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
> index 826856ef64a8..dfa1356daac2 100644
> --- a/arch/arm64/mm/pageattr.c
> +++ b/arch/arm64/mm/pageattr.c
> @@ -399,7 +399,7 @@ bool kernel_page_present(struct page *page)
>  	unsigned long addr = (unsigned long)page_address(page);
>  
>  	pgdp = pgd_offset_k(addr);
> -	if (pgd_none(READ_ONCE(*pgdp)))
> +	if (pgd_none(pgdp_get(pgdp)))
>  		return false;
>  
>  	p4dp = p4d_offset(pgdp, addr);
> diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
> index 7afe2beca4ba..06470d690f9f 100644
> --- a/arch/arm64/mm/trans_pgd.c
> +++ b/arch/arm64/mm/trans_pgd.c
> @@ -134,7 +134,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
>  	unsigned long next;
>  	unsigned long addr = start;
>  
> -	if (pgd_none(READ_ONCE(*dst_pgdp))) {
> +	if (pgd_none(pgdp_get(dst_pgdp))) {
>  		dst_p4dp = trans_alloc(info);
>  		if (!dst_p4dp)
>  			return -ENOMEM;
> @@ -164,7 +164,7 @@ static int copy_page_tables(struct trans_pgd_info *info, pgd_t *dst_pgdp,
>  	dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
>  	do {
>  		next = pgd_addr_end(addr, end);
> -		if (pgd_none(READ_ONCE(*src_pgdp)))
> +		if (pgd_none(pgdp_get(src_pgdp)))
>  			continue;
>  		if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next))
>  			return -ENOMEM;

Re: [PATCH V2 4/4] arm64/mm: Use pgdp_get() for PGD accesses

[Anshuman Khandual]

On Wed, Sep 23, 2026 at 09:31:28AM +0100, Ryan Roberts wrote:
> On 22/09/2026 07:16, Anshuman Khandual wrote:
> > Replace READ_ONCE() with pgdp_get() for PGD accesses in preparation for
> > supporting both D64 and D128 translation table formats.
> > 
> > READ_ONCE() cannot currently be used for 128-bit page table entries on
> > arm64 because it does not provide the required 128-bit single-copy
> > atomicity, causing builds to fail for accesses wider than 64 bits.
> > 
> > Although LDP/STP provide the required atomicity when FEAT_LSE is
> > available (as required by FEAT_D128), extending READ_ONCE() to support
> > 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API,
> > so doing so could encourage other 128-bit users that would either fail
> > to build in configurations without D128 support or, if D128 becomes a
> > runtime option, silently permit tearing on systems without the required
> > hardware support.
> > 
> > Instead, standardize PGD accesses on the existing page-table helpers.
> > These can be overridden on arm64 to provide 128-bit single-copy
> > atomicity when required. No functional change intended.
> 
> I notice you have unconverted READ_ONCE(*pgdp) in:
> 
>  - p4d_offset_phys()
>  - p4d_offset()
> 
> Is that intentional?

Platform overrride for p4dp_get() should solve the problem as well.

> 
> Thanks,
> Ryan
> 
> 
> > 
> > Cc: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: Will Deacon <will@kernel.org>
> > Cc: Ryan Roberts <ryan.roberts@arm.com>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: linux-arm-kernel@lists.infradead.org
> > Cc: linux-kernel@vger.kernel.org
> > Cc: kasan-dev@googlegroups.com
> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
> > ---
> >  arch/arm64/mm/fault.c       | 2 +-
> >  arch/arm64/mm/hugetlbpage.c | 2 +-
> >  arch/arm64/mm/kasan_init.c  | 6 +++---
> >  arch/arm64/mm/mmu.c         | 6 +++---
> >  arch/arm64/mm/pageattr.c    | 2 +-
> >  arch/arm64/mm/trans_pgd.c   | 4 ++--
> >  6 files changed, 11 insertions(+), 11 deletions(-)
> > 
> > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> > index a6afd8929a10..43f25418275a 100644
> > --- a/arch/arm64/mm/fault.c
> > +++ b/arch/arm64/mm/fault.c
> > @@ -160,7 +160,7 @@ static void show_pte(unsigned long addr)
> >  	guard(irqsave)();
> >  
> >  	pgdp = pgd_offset(mm, addr);
> > -	pgd = READ_ONCE(*pgdp);
> > +	pgd = pgdp_get(pgdp);
> >  	ptval_to_str(pxd_str, pgd_val(pgd));
> >  	pr_alert("[%016lx] pgd=%s", addr, pxd_str);
> >  
> > diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
> > index 5771bf49e1fe..e05e5b41ae50 100644
> > --- a/arch/arm64/mm/hugetlbpage.c
> > +++ b/arch/arm64/mm/hugetlbpage.c
> > @@ -284,7 +284,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
> >  	pmd_t *pmdp, pmd;
> >  
> >  	pgdp = pgd_offset(mm, addr);
> > -	if (!pgd_present(READ_ONCE(*pgdp)))
> > +	if (!pgd_present(pgdp_get(pgdp)))
> >  		return NULL;
> >  
> >  	p4dp = p4d_offset(pgdp, addr);
> > diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
> > index 4dd8c1186471..f61f5bc09467 100644
> > --- a/arch/arm64/mm/kasan_init.c
> > +++ b/arch/arm64/mm/kasan_init.c
> > @@ -102,7 +102,7 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
> >  static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int node,
> >  				      bool early)
> >  {
> > -	if (pgd_none(READ_ONCE(*pgdp))) {
> > +	if (pgd_none(pgdp_get(pgdp))) {
> >  		phys_addr_t p4d_phys = early ?
> >  				__pa_symbol(kasan_early_shadow_p4d)
> >  					: kasan_alloc_zeroed_page(node);
> > @@ -256,7 +256,7 @@ static int __init root_level_idx(u64 addr)
> >  static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pud)
> >  {
> >  	int idx = root_level_idx(addr);
> > -	pgd_t pgd = READ_ONCE(swapper_pg_dir[idx]);
> > +	pgd_t pgd = pgdp_get(swapper_pg_dir + idx);
> >  	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
> >  
> >  	memcpy(pud, pudp, PAGE_SIZE);
> > @@ -280,7 +280,7 @@ static int __init next_level_idx(u64 addr)
> >   */
> >  static void __init clear_next_level(int pgd_idx, int start, int end)
> >  {
> > -	pgd_t pgd = READ_ONCE(swapper_pg_dir[pgd_idx]);
> > +	pgd_t pgd = pgdp_get(swapper_pg_dir + pgd_idx);
> >  	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
> >  
> >  	memset(&pudp[start], 0, (end - start) * sizeof(pud_t));
> > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > index 83709d987e7c..76d8d320aeed 100644
> > --- a/arch/arm64/mm/mmu.c
> > +++ b/arch/arm64/mm/mmu.c
> > @@ -434,7 +434,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
> >  {
> >  	int ret;
> >  	unsigned long next;
> > -	pgd_t pgd = READ_ONCE(*pgdp);
> > +	pgd_t pgd = pgdp_get(pgdp);
> >  	p4d_t *p4dp;
> >  
> >  	if (pgd_none(pgd)) {
> > @@ -1649,7 +1649,7 @@ static void unmap_hotplug_range(unsigned long addr, unsigned long end,
> >  	do {
> >  		next = pgd_addr_end(addr, end);
> >  		pgdp = pgd_offset_k(addr);
> > -		pgd = READ_ONCE(*pgdp);
> > +		pgd = pgdp_get(pgdp);
> >  		if (pgd_none(pgd))
> >  			continue;
> >  
> > @@ -1827,7 +1827,7 @@ static void free_empty_tables(unsigned long addr, unsigned long end,
> >  	do {
> >  		next = pgd_addr_end(addr, end);
> >  		pgdp = pgd_offset_k(addr);
> > -		pgd = READ_ONCE(*pgdp);
> > +		pgd = pgdp_get(pgdp);
> >  		if (pgd_none(pgd))
> >  			continue;
> >  
> > diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
> > index 826856ef64a8..dfa1356daac2 100644
> > --- a/arch/arm64/mm/pageattr.c
> > +++ b/arch/arm64/mm/pageattr.c
> > @@ -399,7 +399,7 @@ bool kernel_page_present(struct page *page)
> >  	unsigned long addr = (unsigned long)page_address(page);
> >  
> >  	pgdp = pgd_offset_k(addr);
> > -	if (pgd_none(READ_ONCE(*pgdp)))
> > +	if (pgd_none(pgdp_get(pgdp)))
> >  		return false;
> >  
> >  	p4dp = p4d_offset(pgdp, addr);
> > diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
> > index 7afe2beca4ba..06470d690f9f 100644
> > --- a/arch/arm64/mm/trans_pgd.c
> > +++ b/arch/arm64/mm/trans_pgd.c
> > @@ -134,7 +134,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
> >  	unsigned long next;
> >  	unsigned long addr = start;
> >  
> > -	if (pgd_none(READ_ONCE(*dst_pgdp))) {
> > +	if (pgd_none(pgdp_get(dst_pgdp))) {
> >  		dst_p4dp = trans_alloc(info);
> >  		if (!dst_p4dp)
> >  			return -ENOMEM;
> > @@ -164,7 +164,7 @@ static int copy_page_tables(struct trans_pgd_info *info, pgd_t *dst_pgdp,
> >  	dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
> >  	do {
> >  		next = pgd_addr_end(addr, end);
> > -		if (pgd_none(READ_ONCE(*src_pgdp)))
> > +		if (pgd_none(pgdp_get(src_pgdp)))
> >  			continue;
> >  		if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next))
> >  			return -ENOMEM;
>