From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=217.140.110.172 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794878894; b=1Mj2Fep8Rq6etX0cIZy5nw4ZQM99BvtJbumr4XUEpZSKzolg0lyv/MalXf0qvuXcjqD2 6lrstakU3Mlba53bKr4kb+ptTwrQe/ItbVfJbuXXTZJdl3e00Mg2kIXgx2qOTx3VkD/+w 2oXOR4hcQDp0NIs+MBCwq/4IoftB6sOr46XVDUzg54j6XspeoCBQ9fNpkxl470pU2nBJ2 MtlfWJBD53zyMe42yIRw08Cts0JG0rsxJLZ5Ij1d+1lKJaNJisXscU75LLGpQMBi6tkz3 aRdS2J42u6UaFaIt5ek+tBunxsDCmRmIqpyjQFKxeabW8RlU5LjrzRR8wRPbyAzCBVQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794878894; h=DMARC-Filter:Received:Received:DKIM-Signature:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=SAZOtCyIvrop90LYK6Gfvm6EibkZAQOFm6TE4q9leos=; b=eaUwohWpsrIEoebwYhW4vmTErNBAxWZg4QfVYHMFPJ98a17YYxEtdidzD/ky9xnsVMgP Bmwu/X/0Hu//qSoLHjcZ2+9b6f3rBAsfbSbbvAQyL1b6+/lfRmKXDlT9C0mply9OHz/z3 xxGGWo8sAiGm4qQc7jB+0YCGe/RG1OMIJeVCjZM+X31nMeGyKPxyJBA/+SNsUs5tTHQQD cDhfaiygY36LpHNyQguG2r8/XxKswqHCyv760Gflrkkn2AwAQ8VTijM3zVZtvYQUG2Rmz Jgb56S+aHYseS53hj1e36RjESiaqv2BcyuyvpNT0ekRxEGlqZJj5ehSD6TLxfbHoejA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b="MGNX//NZ"; arc=none smtp.remote-ip=217.140.110.172 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C4934606A31C for ; Tue, 17 Nov 2026 01:28:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org C4934606A31C Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b=MGNX//NZ Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 444B41684; Mon, 16 Nov 2026 17:28:10 -0800 (PST) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7BE203F673; Mon, 16 Nov 2026 17:28:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1794878893; bh=LDZOS8QnU7Z9iKqYNs/7MsU4zmtODqm1Gxg8jYGxZ9g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MGNX//NZL7k/+6Cn2X8zxABlk3H+q2h3m+Fy2L3DIOUGI7yqXgt2HW7W6Rlvt+xT9 1rXA8P0EChC4bfzGJilRO6fyIP2SG3CT9ZG53kOIQDORYd4pTG0YreCAYn8okOZjL0 ATVETc8oNbvuX0wxx+EFvL+WlDmpXgF3Mne5dHto= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org Cc: Anshuman Khandual , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: [PATCH V5 2/4] arm64/mm: Use pudp_get() for PUD accesses Date: Tue, 17 Nov 2026 06:57:56 +0530 Message-ID: <20261117012758.3814093-3-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261117012758.3814093-1-anshuman.khandual@arm.com> References: <20261117012758.3814093-1-anshuman.khandual@arm.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Replace READ_ONCE() with pudp_get() for PUD accesses in preparation for supporting both D64 and D128 translation table formats. READ_ONCE() cannot currently be used for 128-bit page table entries on arm64 because it does not provide the required 128-bit single-copy atomicity, causing builds to fail for accesses wider than 64 bits. Although LDP/STP provide the required atomicity when FEAT_LSE is available (as required by FEAT_D128), extending READ_ONCE() to support 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API, so doing so could encourage other 128-bit users that would either fail to build in configurations without D128 support or, if D128 becomes a runtime option, silently permit tearing on systems without the required hardware support. Instead, standardize PUD accesses on the existing page-table helpers. These can be overridden on arm64 to provide 128-bit single-copy atomicity when required. For now override pudp_get() which is same as the generic memory default in order to be consistent with similar helpers. No functional change intended. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Reviewed-by: Ryan Roberts Signed-off-by: Anshuman Khandual --- arch/arm64/include/asm/pgtable.h | 9 ++++++++- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 4 ++-- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 20 ++++++++++---------- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 8 files changed, 27 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ab5641f60fa3..ad6cd5a71b71 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -90,6 +90,12 @@ static inline pmd_t pmdp_get(const pmd_t *pmdp) return READ_ONCE(*pmdp); } +#define pudp_get pudp_get +static inline pud_t pudp_get(const pud_t *pudp) +{ + return READ_ONCE(*pudp); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE @@ -879,7 +885,8 @@ static inline pmd_t *pud_pgtable(pud_t pud) } /* Find an entry in the second-level page table. */ -#define pmd_offset_phys(dir, addr) (pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t)) +#define pmd_offset_phys(dir, addr) (pud_page_paddr(pudp_get(dir)) + \ + pmd_index(addr) * sizeof(pmd_t)) #define pmd_set_fixmap(addr) ((pmd_t *)set_fixmap_offset(FIX_PMD, addr)) #define pmd_set_fixmap_offset(pud, addr) pmd_set_fixmap(pmd_offset_phys(pud, addr)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 33b05a9f13d7..ea0cdc730768 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -181,7 +181,7 @@ static void show_pte(unsigned long addr) break; pudp = pud_offset_lockless(p4dp, p4d, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); ptval_to_str(pxd_str, pud_val(pud)); pr_cont(", pud=%s", pxd_str); if (pud_none(pud) || pud_bad(pud)) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 3cdac8021d4f..d9a870836faf 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -56,7 +56,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr, unsigned long end) { unsigned long next; - pud_t pud = READ_ONCE(*pudp); + pud_t pud = pudp_get(pudp); pmd_t *pmdp; if (pud_none(pud)) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 301b5b7265a2..ce8ecf5a308f 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -262,7 +262,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, WARN_ON(addr & (sz - 1)); ptep = pte_alloc_huge(mm, pmdp, addr); } else if (sz == PMD_SIZE) { - if (want_pmd_share(vma, addr) && pud_none(READ_ONCE(*pudp))) + if (want_pmd_share(vma, addr) && pud_none(pudp_get(pudp))) ptep = huge_pmd_share(mm, vma, addr, pudp); else ptep = (pte_t *)pmd_alloc(mm, pudp, addr); @@ -292,7 +292,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return NULL; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (sz != PUD_SIZE && pud_none(pud)) return NULL; /* hugepage or swap? */ diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 7ca833c5de5e..aad29bcc7622 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -76,7 +76,7 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node, static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node, bool early) { - if (pud_none(READ_ONCE(*pudp))) { + if (pud_none(pudp_get(pudp))) { phys_addr_t pmd_phys = early ? __pa_symbol(kasan_early_shadow_pmd) : kasan_alloc_zeroed_page(node); @@ -150,7 +150,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); kasan_pmd_populate(pudp, addr, next, node, early); - } while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp))); + } while (pudp++, addr = next, addr != end && pud_none(pudp_get(pudp))); } static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 0433033c0a71..c14e7404f971 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -309,7 +309,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, { int ret; unsigned long next; - pud_t pud = READ_ONCE(*pudp); + pud_t pud = pudp_get(pudp); pmd_t *pmdp; /* @@ -390,7 +390,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, } do { - pud_t old_pud = READ_ONCE(*pudp); + pud_t old_pud = pudp_get(pudp); next = pud_addr_end(addr, end); @@ -408,7 +408,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), - READ_ONCE(pud_val(*pudp)))); + pud_val(pudp_get(pudp)))); } else { ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); @@ -416,7 +416,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, goto out; VM_WARN_ON_ONCE(pud_val(old_pud) != 0 && - pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); + pud_val(old_pud) != pud_val(pudp_get(pudp))); } phys += next - addr; } while (pudp++, addr = next, addr != end); @@ -1591,7 +1591,7 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) continue; @@ -1748,7 +1748,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, do { next = pud_addr_end(addr, end); pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) continue; @@ -1769,7 +1769,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr, */ pudp = pud_offset(p4dp, 0UL); for (i = 0; i < PTRS_PER_PUD; i++) { - if (!pud_none(READ_ONCE(pudp[i]))) + if (!pud_none(pudp_get(pudp + i))) return; } @@ -1867,7 +1867,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); /* Only allow permission changes for now */ - if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)), + if (!pgattr_change_is_safe(pud_val(pudp_get(pudp)), pud_val(new_pud))) return 0; @@ -1898,7 +1898,7 @@ void p4d_clear_huge(p4d_t *p4dp) int pud_clear_huge(pud_t *pudp) { - if (!pud_leaf(READ_ONCE(*pudp))) + if (!pud_leaf(pudp_get(pudp))) return 0; pud_clear(pudp); return 1; @@ -1938,7 +1938,7 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) pud_t pud; unsigned long next, end; - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (!pud_table(pud)) { VM_WARN_ON(1); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index bf1fd1258a75..c544a61a5afd 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -355,7 +355,7 @@ bool kernel_page_present(struct page *page) return false; pudp = pud_offset(p4dp, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get(pudp); if (pud_none(pud)) return false; if (pud_leaf(pud)) diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index b27b2d2c20c3..d119119455f1 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -64,7 +64,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp, unsigned long next; unsigned long addr = start; - if (pud_none(READ_ONCE(*dst_pudp))) { + if (pud_none(pudp_get(dst_pudp))) { dst_pmdp = trans_alloc(info); if (!dst_pmdp) return -ENOMEM; @@ -109,7 +109,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp, src_pudp = pud_offset(src_p4dp, start); do { - pud_t pud = READ_ONCE(*src_pudp); + pud_t pud = pudp_get(src_pudp); next = pud_addr_end(addr, end); if (pud_none(pud)) -- 2.43.0