From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=217.140.110.172 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793618605; b=jY8YfDm/AawbNPgN3VmDQ70rYEJ/+YpX8hFtYgrHT9DjuxA+lykCsPYUNUaA0klWR+Ph ocf/Iuqpr6PvmVqOaXLzSkEAW6e0Za7zvF317IZjRBgTX27N64J004CGfvGWNPHyFbT/6 lAE0pGMTXJ4mwJooYyfqv8kVeJwy/rmhZZxfSk5+euTa95A2Z2ZrlB6qd5P3GMi58HjxP z7+1CslBWnUhZBbFJu1REFLBCT2BI9q0XUEK9Tp/SMLsBo3kEwJPSs4ammOAAoeWo4OdC 3JGG4LG8iHDzV7f3aS4Y57HzIOWN328J4xdz3BfBSz0dbeu6PoUkJent18ufuqa2zFg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793618605; h=DMARC-Filter:Received:Received:DKIM-Signature:From:Date:Subject: MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id: References:In-Reply-To:To:Cc:X-Mailer:X-Developer-Signature: X-Developer-Key; bh=bzbkHlCXIOztNCUIQ2oe4PXisYyJ6l3zF3m0g0wGdqo=; b=MzLa+ojm+CfQhjOX4/cDGBWqad/php7UMH5P9Cn34Hj1dW/cnaK0gc+0WvSahyDxULq5 QIa3c55kuXXOpmeO0a7DiP4B1S6B+TFfbT6HG5c32H573hoFBod0ffe1947IxE6UugXmM /tzTgkKU5+vA3CjHUnZ7QkDy1C7RKggOBn0RGqBNPMV198foWtKaddoNe261W/dpdNHUN on4wBfrrL/DOnTXVxDOjJCB5uksyZDpVzxCoPHq2hRHMMLgDi3qpFcg1INe2GRtkXyGFc kxlsIVCXeqfVktLV0AsiRjYS4SNQScqNbcrSiN6IJ3j4D3IbdsdlegIE2nMV95t5uXg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b=O6Y5aTQJ; arc=none smtp.remote-ip=217.140.110.172 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D2DDE60A260B; Mon, 2 Nov 2026 11:23:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org D2DDE60A260B Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=foss header.b=O6Y5aTQJ Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2E0282A31; Mon, 2 Nov 2026 03:23:21 -0800 (PST) Received: from e129823.arm.com (e129823.arm.com [10.2.213.3]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 425403F632; Mon, 2 Nov 2026 03:23:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1793618604; bh=3qwcredkJP47Q0knCs8r2zQMj6PgEaC6pvNuESy4rVM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=O6Y5aTQJGvriCytR9bYPhPc8teP7S38goXBCyi0FrXHhPcgUI/VK5QVSOYRjvKkWX RN0/pAen7AAHAps9w7HSRdvtyoM9DYJ+IN2BHRbHL1MPPhiy2Rrxfh3JKryeMbngMC 35sXBD2JouQDrqVQDe9c4jQWSg46NUeELFmBDt2M= From: Yeoreum Yun Date: Mon, 02 Nov 2026 11:21:30 +0000 Subject: [PATCH RESEND v2 17/21] mm/pgtable: optimize pmdp_get() and friends for folded pagetable levels X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261102-dummy_ptxp3-v2-17-78840b2ec73c@arm.com> References: <20261102-dummy_ptxp3-v2-0-78840b2ec73c@arm.com> In-Reply-To: <20261102-dummy_ptxp3-v2-0-78840b2ec73c@arm.com> To: Russell King , Huacai Chen , WANG Xuerui , Thomas Bogendoerfer , Catalin Marinas , Will Deacon , Arnd Bergmann , Andrew Morton , Kairui Song , Qi Zheng , Shakeel Butt , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Lorenzo Stoakes , Tianrui Zhao , Bibo Mao , Anup Patel , Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jonas Bonn , Stefan Kristiansson , Stafford Horne Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-openrisc@vger.kernel.org X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5359; i=yeoreum.yun@arm.com; h=from:subject:message-id; bh=7UYhDG8FvnoAk5ImIUgRe9XhRYP6zIOaKvnUSG6ABJs=; b=owEB7QES/pANAwAKAW3Vw9FaxTEzAcsmYgBq6HI3ORBjhDgVhiG8NLXgz+05V/q3PbFarKyI1 M1+TFxo5X6JAbMEAAEKAB0WIQQtg+CS3QUzuFh1pJ1t1cPRWsUxMwUCauhyNwAKCRBt1cPRWsUx M4K8C/9RIui2TsgD3POI6zjdB3SZC/Jq7bkgNK0EXZ7pxRiCpRWJ3s+y6XKZKi08pLUrW/0doeo f79FzUimsG5sWq8i7o3Qvs8iLUlOerz7iIeU8CLwgackXEZI8Yp1OtC3Pa3eIgti8dVW4HCVAc2 45fyjV6RiSA/h0N2l5M2E5xAHzKdpGSyG4V1e1z2f/E+xvh4yJaK6PeTyVo7ITroN161YMVMyyH IqXfICPAav4K4QcS7SJFjxr11Dv9S5lhwprM4oAiu7UGSz0RWlaI9dA9WIlYHppYNQ9Ep7wcFKd PZUAI+/RIF5Jx2eFscXFI2ZijjsPo9Z79NpjqsD/oyBAZVh61V9A/nwySidlAV2yzU2RCgjuTIp A6YGZmNpCwkrO+3fkndiCg0vgX5Xw7Tt2Y1osCKh8GORdrTenvXH+Km7wCwjhvE1RI8DwmyU/rj fRQb3IcrX1TH+HW+cpEGyHwUcXlzYYKevuYTiKGU+ESHSEFQ6OAJMtmtmxsbgFsZ6yyW4= X-Developer-Key: i=yeoreum.yun@arm.com; a=openpgp; fpr=2D83E092DD0533B85875A49D6DD5C3D15AC53133 From: "David Hildenbrand (Arm)" Using pmdp_get() and friends in common code on a kernel config with folded page tables is suboptimal: they default to a READ_ONCE(), forcing the compiler to actually read that value even though it will not actually be used afterwards. This was recently reported by Christophe Leroy [1] and block conversion of more common code to pmdp_get() and friends. (using pgdp_get() as one example) Most of the code ignores the result from pgdp_get() on configs with folded page tables entirely, as it's hardcoded: pgd_present()==1 && pgd_leaf()==false Common code will just treat it as a "this is a page table" and call p4d_offset() or p4d_offset_lockless() for the next lower level, where the obtained pgdp_get() result is ignored entirely. So, return a dummy value and avoid any memory reads. There is a catch, though: 1) If code calls pgd_val() and somehow relies on the data, it would now see dummy values. The code really must be aware of folded page table levels. Fortunately, code usually ignores pgd_val() completely for page tables (with ptdump being one exception when calculating effective permissions). it's checked + fixed the x86 ptdump mechanism. 2) If code passes the pgd_t to a function that would work on the result, it would now see dummy values. The only concern is really passing the pgd_t on the stack as a pointer to p4d_offset(). Most code that would do that, should actually use p4d_offset_lockless(), which handles this properly. it's checked + fixed problematic instances. 3) Calling set_pgd() / pgd_page() / pgd_page_vaddr() with a pgd_t obtained through pgdp_get(). There was once such case in riscv code with set_pgd(), which is fixed. As an example, this is the generated code for perf_get_page_size() with PGTABLE_LEVELS=3 on arm64: Before: 00000000000052a0 : ... 52dc: d53b4234 mrs x20, DAIF 52e0: d50343df msr DAIFSet, #0x3 ... 52fc: d35e9a69 ubfx x9, x19, #30, #9 /* pud_offset_lockless() */ 5300: f9403508 ldr x8, [x8, #0x68] 5304: f869790a ldr x10, [x8, x9, lsl #3] /* pudp_get() */ 5308: f90007ea str x10, [sp, #0x8] 530c: f8697908 ldr x8, [x8, x9, lsl #3] /* pudp_get() */ ... 5360: 90000009 adrp x9, 0x5000 5364: 92746908 and x8, x8, #0x7ffffff000 5368: d3557675 ubfx x21, x19, #21, #9 /* pmd_offset_lockless() */ ... 5394: f8757ac8 ldr x8, [x22, x21, lsl #3] /* pmdp_get() */ After: 0000000000052a0 : ... 52dc: d53b4234 mrs x20, DAIF 52e0: d50343df msr DAIFSet, #0x3 ... /* no pud_offset_lockless() and pudp_get() */ 5318: 90000009 adrp x9, 0x5000 531c: 92746908 and x8, x8, #0x7ffffff000 5320: d3557675 ubfx x21, x19, #21, #9 /* pmd_offset_lockless() */ ... 5334: f8757ac8 ldr x8, [x22, x21, lsl #3] /* pmdp_get() */ [1] https://lore.kernel.org/all/0019d675-ce3d-4a5c-89ed-f126c45145c9@kernel.org/ Signed-off-by: David Hildenbrand (Arm) --- include/asm-generic/pgtable-nop4d.h | 8 ++++++++ include/asm-generic/pgtable-nopmd.h | 8 ++++++++ include/asm-generic/pgtable-nopud.h | 8 ++++++++ 3 files changed, 24 insertions(+) diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h index 019c3f074b771..acd62ee233536 100644 --- a/include/asm-generic/pgtable-nop4d.h +++ b/include/asm-generic/pgtable-nop4d.h @@ -34,6 +34,14 @@ static inline bool pgd_leaf(pgd_t pgd) { return false; } */ #define set_pgd(pgdptr, pgdval) set_p4d((p4d_t *)(pgdptr), (p4d_t) { pgdval }) +static inline pgd_t pgdp_get(pgd_t *pgdp) +{ + pgd_t dummy = { 0 }; + + return dummy; +} +#define pgdp_get pgdp_get + static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) { return (p4d_t *)pgd; diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h index 1dd5b165234be..9322f07ddf1ef 100644 --- a/include/asm-generic/pgtable-nopmd.h +++ b/include/asm-generic/pgtable-nopmd.h @@ -43,6 +43,14 @@ static inline void pud_clear(pud_t *pud) { } */ #define set_pud(pudptr, pudval) set_pmd((pmd_t *)(pudptr), (pmd_t) { pudval }) +static inline pud_t pudp_get(pud_t *pudp) +{ + pud_t dummy = { 0 }; + + return dummy; +} +#define pudp_get pudp_get + static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address) { return (pmd_t *)pud; diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h index 5a2b0a81ae197..8f01abbb00503 100644 --- a/include/asm-generic/pgtable-nopud.h +++ b/include/asm-generic/pgtable-nopud.h @@ -41,6 +41,14 @@ static inline bool p4d_leaf(p4d_t p4d) { return false; } */ #define set_p4d(p4dptr, p4dval) set_pud((pud_t *)(p4dptr), (pud_t) { p4dval }) +static inline p4d_t p4dp_get(p4d_t *p4dp) +{ + p4d_t dummy = { 0 }; + + return dummy; +} +#define p4dp_get p4dp_get + static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) { return (pud_t *)p4d; -- 2.43.0