From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=217.140.110.172 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793853261; b=M7fYJd6Qw4QfUvZWUaJm5OPLwf8BXZkXS/hiEEW59Qc4Io95AxK0740Et6q9ZfqmL6m3 PXVmHt/2UDGqbGLlWAxpRza/IDkfT4kw1dZ9xQ7KE08W7YC8YHcuYJxPjkExvXQmEW4uQ GjscScRmT17MxD7u3yPs55R5xp5Uw4wgicmcdHqpeKN/Nz+lOJUHdVuRZLb8HMHPZjfxs w/qNDwDPPhUQVd8VAjT8F7tPU8gTz2R7zFrv3KR+7o77shxqmZ5cMDb5A28Mi2UXjIvdA 7dBQqSG3d8FqqYR72rUslAiZILNJ8Cb764Du36bB9D3LlzZ7aNVrdPVXELirfrx+95g== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793853261; h=DMARC-Filter:Received:Received:DKIM-Signature:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=mdZJpnovM6lNRF0Mo4Sv+yVKUzBD6Nyll5FbjHeB/EM=; b=LqK3b8SpYxhaf4zSJK1SN540chkkMu+JwG2bCminSF7uAOc0DWV2PHnT1RO2smUrpbpq A1/cN46H0BVmkYrFuH2KgCzMQNbv58G4B95bEi1zlQFSRvXB0vEV/EJ1s91nT5H/gKnOt Z4mkWzwE+oUbVByrq5iRoroxRZMSMq+pvAYUvYJSa62ZYUiD8VEyL732d76iy87bYT4C2 nBcn3EaTUNsqCeqhl8/ed4L2f0arjUzNZEJicNEiLekaulw81rplG3fHb67A6Axc3hx3R sVeLlMgfwJeq1xx3zXdW2Ur+HmFN2U2CYkEw+FMFjcDLBL0Sp52WAjPE8F+g/QpiULg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b=mBfyVHJI; arc=none smtp.remote-ip=217.140.110.172 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 209AA606D80F for ; Thu, 5 Nov 2026 04:34:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 209AA606D80F Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b=mBfyVHJI Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6BE6919F6; Wed, 4 Nov 2026 20:34:17 -0800 (PST) Received: from localhost (a085714.arm.com [10.164.19.28]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BFE343F673; Wed, 4 Nov 2026 20:34:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1793853260; bh=rp7aWu/XoIApf8cmDyYFxTYta3iMIp0GH8RXgCLwKGQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mBfyVHJI3WDJls8iQzO2ir3C/Qtnby7Q3jYY12atOyI16ZW6FgoB5+wGLCv6GIwfh CNdfROy47FKqMUo+kU55cfnxPonFz6FddklfX1nsHdy/BkLqYyaAcTWgx1zznV9+UF q9v1+yP7MrIk4om5xqSuA618OzL7WzAPYg1kQFTY= From: Anshuman Khandual To: linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will@kernel.org Cc: Anshuman Khandual , Ryan Roberts , Mark Rutland , Lorenzo Stoakes , Andrew Morton , David Hildenbrand , Mike Rapoport , Linu Cherian , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: [PATCH V4 3/4] arm64/mm: Use p4dp_get() for P4D accesses Date: Thu, 5 Nov 2026 10:04:02 +0530 Message-ID: <20261105043403.3326445-4-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261105043403.3326445-1-anshuman.khandual@arm.com> References: <20261105043403.3326445-1-anshuman.khandual@arm.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Replace READ_ONCE() with p4dp_get() for P4D accesses in preparation for supporting both D64 and D128 translation table formats. READ_ONCE() cannot currently be used for 128-bit page table entries on arm64 because it does not provide the required 128-bit single-copy atomicity, causing builds to fail for accesses wider than 64 bits. Although LDP/STP provide the required atomicity when FEAT_LSE is available (as required by FEAT_D128), extending READ_ONCE() to support 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API, so doing so could encourage other 128-bit users that would either fail to build in configurations without D128 support or, if D128 becomes a runtime option, silently permit tearing on systems without the required hardware support. Instead, standardize P4D accesses on the existing page-table helpers. These can be overridden on arm64 to provide 128-bit single-copy atomicity when required. Both READ_ONCE() instances in pud_offset_[phys]() could not be converted into p4pd_get() because of cyclic dependency between and . Move the function inside instead to workaround the problem. No functional change intended. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: kasan-dev@googlegroups.com Reviewed-by: Ryan Roberts Signed-off-by: Anshuman Khandual --- Changes in V4: - Dropped p4dp_get() platform override - Moved both pud_offset[_phys]() inside arch/arm64/mm/mmu.c arch/arm64/include/asm/pgtable.h | 12 ++---------- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 26 ++++++++++++++++++++------ arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 8 files changed, 30 insertions(+), 24 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 96cfcc70acaa..8dffa7f787ac 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -960,12 +960,7 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) return (pud_t *)__va(p4d_page_paddr(p4d)); } -static inline phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr) -{ - VM_WARN_ON_ONCE(!pgtable_l4_enabled()); - - return p4d_page_paddr(READ_ONCE(*p4dp)) + pud_index(addr) * sizeof(pud_t); -} +phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr); static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr) @@ -976,10 +971,7 @@ pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr) } #define pud_offset_lockless pud_offset_lockless -static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) -{ - return pud_offset_lockless(p4dp, READ_ONCE(*p4dp), addr); -} +pud_t *pud_offset(p4d_t *p4dp, unsigned long addr); #define pud_offset pud_offset static inline pud_t *pud_set_fixmap(unsigned long addr) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index ea0cdc730768..934b534c5b89 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -174,7 +174,7 @@ static void show_pte(unsigned long addr) break; p4dp = p4d_offset_lockless(pgdp, pgd, addr); - p4d = READ_ONCE(*p4dp); + p4d = p4dp_get(p4dp); ptval_to_str(pxd_str, p4d_val(p4d)); pr_cont(", p4d=%s", pxd_str); if (p4d_none(p4d) || p4d_bad(p4d)) diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index d9a870836faf..2a0fe0b0f771 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -74,7 +74,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr, static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end) { - p4d_t p4d = READ_ONCE(*p4dp); + p4d_t p4d = p4dp_get(p4dp); pud_t *pudp; if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) && diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index c9ad5e75b073..5771bf49e1fe 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -288,7 +288,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return NULL; p4dp = p4d_offset(pgdp, addr); - if (!p4d_present(READ_ONCE(*p4dp))) + if (!p4d_present(p4dp_get(p4dp))) return NULL; pudp = pud_offset(p4dp, addr); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index aad29bcc7622..4dd8c1186471 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -89,7 +89,7 @@ static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node, static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node, bool early) { - if (p4d_none(READ_ONCE(*p4dp))) { + if (p4d_none(p4dp_get(p4dp))) { phys_addr_t pud_phys = early ? __pa_symbol(kasan_early_shadow_pud) : kasan_alloc_zeroed_page(node); @@ -162,7 +162,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr, do { next = p4d_addr_end(addr, end); kasan_pud_populate(p4dp, addr, next, node, early); - } while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp))); + } while (p4dp++, addr = next, addr != end && p4d_none(p4dp_get(p4dp))); } static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 7aa8793f13c5..711d734a63f7 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -367,7 +367,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, { int ret = 0; unsigned long next; - p4d_t p4d = READ_ONCE(*p4dp); + p4d_t p4d = p4dp_get(p4dp); pud_t *pudp; if (p4d_none(p4d)) { @@ -457,7 +457,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, } do { - p4d_t old_p4d = READ_ONCE(*p4dp); + p4d_t old_p4d = p4dp_get(p4dp); next = p4d_addr_end(addr, end); @@ -467,7 +467,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, goto out; VM_WARN_ON_ONCE(p4d_val(old_p4d) != 0 && - p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp))); + p4d_val(old_p4d) != (p4d_val(p4dp_get(p4dp)))); phys += next - addr; } while (p4dp++, addr = next, addr != end); @@ -1622,7 +1622,7 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr, do { next = p4d_addr_end(addr, end); p4dp = p4d_offset(pgdp, addr); - p4d = READ_ONCE(*p4dp); + p4d = p4dp_get(p4dp); if (p4d_none(p4d)) continue; @@ -1788,7 +1788,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, do { next = p4d_addr_end(addr, end); p4dp = p4d_offset(pgdp, addr); - p4d = READ_ONCE(*p4dp); + p4d = p4dp_get(p4dp); if (p4d_none(p4d)) continue; @@ -1809,7 +1809,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr, */ p4dp = p4d_offset(pgdp, 0UL); for (i = 0; i < PTRS_PER_P4D; i++) { - if (!p4d_none(READ_ONCE(p4dp[i]))) + if (!p4d_none(p4dp_get(p4dp + i))) return; } @@ -2450,3 +2450,17 @@ bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ + +#if CONFIG_PGTABLE_LEVELS > 3 +phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr) +{ + VM_WARN_ON_ONCE(!pgtable_l4_enabled()); + + return p4d_page_paddr(p4dp_get(p4dp)) + pud_index(addr) * sizeof(pud_t); +} + +pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) +{ + return pud_offset_lockless(p4dp, p4dp_get(p4dp), addr); +} +#endif /* CONFIG_PGTABLE_LEVELS > 3 */ diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index c544a61a5afd..172bc516164c 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -351,7 +351,7 @@ bool kernel_page_present(struct page *page) return false; p4dp = p4d_offset(pgdp, addr); - if (p4d_none(READ_ONCE(*p4dp))) + if (p4d_none(p4dp_get(p4dp))) return false; pudp = pud_offset(p4dp, addr); diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index d119119455f1..7afe2beca4ba 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -99,7 +99,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp, unsigned long next; unsigned long addr = start; - if (p4d_none(READ_ONCE(*dst_p4dp))) { + if (p4d_none(p4dp_get(dst_p4dp))) { dst_pudp = trans_alloc(info); if (!dst_pudp) return -ENOMEM; @@ -145,7 +145,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp, src_p4dp = p4d_offset(src_pgdp, start); do { next = p4d_addr_end(addr, end); - if (p4d_none(READ_ONCE(*src_p4dp))) + if (p4d_none(p4dp_get(src_p4dp))) continue; if (copy_pud(info, dst_p4dp, src_p4dp, addr, next)) return -ENOMEM; -- 2.43.0