From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8FBE544AB63; Wed, 22 Jul 2026 15:32:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784734350; cv=none; b=FiYoO2OacI4dHXMxKaeNQlJ/G/m0CPsXbNtLMs7GthvoGldxF6u9MHfY3YWMRSZRig0i+Ft/x0F/hXHFtufR2tt118MjQvkL6ILnavkA7l3UEodTP8oTwifU9Jyl5f0wzKrZeJkznyt2E2sgXB4Tqv6JD+8CoHSdV/+/l8I9Kow= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784734350; c=relaxed/simple; bh=vWdfZfN+TiiZA9mxg4SsfZFErqbDJUiR9ggvZZQt5JA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=fb6/6nlSo+yk+JCxPneTBh9kg3xed0QsEqySj5n6MZ85BHgnc7q1euSEMegg0VmkqcIoHr/XgfrQHNZYsmxC585fKpp1KQAMl7fZdLoWVnPEsb4QIhNh2hW4pU2Y0T/S9GA0vJ9nx+CHOY45Bu61JdEkd2l2ST3qCTpPDQl0VH0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=TluBfn6Y; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="TluBfn6Y" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BC0C81595; Wed, 22 Jul 2026 08:32:23 -0700 (PDT) Received: from e129823.arm.com (e129823.arm.com [10.2.213.3]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 239BA3F66F; Wed, 22 Jul 2026 08:32:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1784734347; bh=vWdfZfN+TiiZA9mxg4SsfZFErqbDJUiR9ggvZZQt5JA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=TluBfn6YvynnLDcgkk/QGS/YqHzTC1qQOYuWvEjPBaQRfpNlFivei9Evm3MWLvruP TPbqCkwMeV/455tcZTKmsYaC7xvULM9EWroPcZjh24N/EDwhLaY+XZhbJrM151ixS0 tRinqhTWhBgUWaY0/Q9D4pc2+OS/ow878qXHsExY= From: Yeoreum Yun Date: Wed, 22 Jul 2026 16:30:23 +0100 Subject: [PATCH RFC v2 15/20] mm/pgtable: optimize pmdp_get() and friends for folded pagetable levels Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260722-dummy_ptxp3-v2-15-d9e4bad31e0a@arm.com> References: <20260722-dummy_ptxp3-v2-0-d9e4bad31e0a@arm.com> In-Reply-To: <20260722-dummy_ptxp3-v2-0-d9e4bad31e0a@arm.com> To: Russell King , Huacai Chen , WANG Xuerui , Thomas Bogendoerfer , Catalin Marinas , Will Deacon , Arnd Bergmann , Andrew Morton , Kairui Song , Qi Zheng , Shakeel Butt , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Lorenzo Stoakes , Tianrui Zhao , Bibo Mao , Anup Patel , Atish Patra , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jonas Bonn , Stefan Kristiansson , Stafford Horne Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-openrisc@vger.kernel.org X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5184; i=yeoreum.yun@arm.com; h=from:subject:message-id; bh=cnUlm8DDOUq30iZM7WGkUZANPR8bNBvx+fV94KySXE8=; b=owEB7QES/pANAwAKAW3Vw9FaxTEzAcsmYgBqYOIWDgQIN2ellQ1u+0chCvrQkqScLp7v2R/RU 36cXzhY1xOJAbMEAAEKAB0WIQQtg+CS3QUzuFh1pJ1t1cPRWsUxMwUCamDiFgAKCRBt1cPRWsUx MyDDDACQWZei5YO99pneN8Lyoq4PWeM8DMKhrvzlE+hATCDxZYZYqyuMtbdAqTyetpmWXGW5siE gts8DGY7bYaUgcRAMVhigeJd7bcz7LMDtHhi/u/GW3DDpWGMfA7psXQ6DFVqYMnVUy+wCSNENOc SanzIHDCdilyEwAYEfSV92vfl0r239d6VSMLGN2WKD8fSNNn9WtMRLitSyiFRlcVM7uTDQ1D+9m swQnX0kJjxZd7fbD1Y0sB3pbgfhBXOnu1b93CBRLKHNK97V0z1hHNLSpcqwI5zeE6vgRLHvCAZ+ atJFh938hNaw2px8jQNgsNCNI5gFUuCju1bSSlHelm07/m4HUaE3TMUe7Qo9/+h6h5xuytxKnAG EFEv9pl5Dr/xsuErnxJQDYY/m3S3fQDIuHvAduxYEAuJC4tT25Q/fdA4f0IMJCBKYgFKHO4WE/C BxD1l8i7ha71s9NuT5l/0LKh+2twxhEwhIB1IE99Tv/Df7TZ0p/Yl9HWK5rgCAYq/Y6GI= X-Developer-Key: i=yeoreum.yun@arm.com; a=openpgp; fpr=2D83E092DD0533B85875A49D6DD5C3D15AC53133 From: "David Hildenbrand (Arm)" Using pmdp_get() and friends in common code on a kernel config with folded page tables is suboptimal: they default to a READ_ONCE(), forcing the compiler to actually read that value even though it will not actually be used afterwards. This was recently reported by Christophe Leroy [1] and block conversion of more common code to pmdp_get() and friends. (using pgdp_get() as one example) Most of the code ignores the result from pgdp_get() on configs with folded page tables entirely, as we hardcode: pgd_present()==1 && pgd_leaf()==false Common code will just treat it as a "this is a page table" and call p4d_offset() or p4d_offset_lockless() for the next lower level, where we just ignore the obtained pgdp_get() result entirely. So we can just return a dummy value and avoid any memory reads. There is a catch, though: 1) If code calls pgd_val() and somehow relies on the data, it would now see dummy values. The code really must be aware of folded page table levels. Fortunately, code usually ignores pgd_val() completely for page tables (with ptdump being one exception when calculating effective permissions). We checked + fixed the x86 ptdump mechanism. 2) If code passes the pgd_t to a function that would work on the result, it would now see dummy values. The only concern is really passing the pgd_t on the stack as a pointer to p4d_offset(). Most code that would do that, should actually use p4d_offset_lockless(), which handles this properly. We checked + fixed problematic instances. As an example, this is the generated code for perf_get_page_size() with PGTABLE_LEVELS=3 on arm64: Before: 00000000000052a0 : ... 52dc: d53b4234 mrs x20, DAIF 52e0: d50343df msr DAIFSet, #0x3 ... 52fc: d35e9a69 ubfx x9, x19, #30, #9 /* pud_offset_lockless() */ 5300: f9403508 ldr x8, [x8, #0x68] 5304: f869790a ldr x10, [x8, x9, lsl #3] /* pudp_get() */ 5308: f90007ea str x10, [sp, #0x8] 530c: f8697908 ldr x8, [x8, x9, lsl #3] /* pudp_get() */ ... 5360: 90000009 adrp x9, 0x5000 5364: 92746908 and x8, x8, #0x7ffffff000 5368: d3557675 ubfx x21, x19, #21, #9 /* pmd_offset_lockless() */ ... 5394: f8757ac8 ldr x8, [x22, x21, lsl #3] /* pmdp_get() */ After: 0000000000052a0 : ... 52dc: d53b4234 mrs x20, DAIF 52e0: d50343df msr DAIFSet, #0x3 ... /* no pud_offset_lockless() and pudp_get() */ 5318: 90000009 adrp x9, 0x5000 531c: 92746908 and x8, x8, #0x7ffffff000 5320: d3557675 ubfx x21, x19, #21, #9 /* pmd_offset_lockless() */ ... 5334: f8757ac8 ldr x8, [x22, x21, lsl #3] /* pmdp_get() */ [1] https://lore.kernel.org/all/0019d675-ce3d-4a5c-89ed-f126c45145c9@kernel.org/ Signed-off-by: David Hildenbrand (Arm) --- include/asm-generic/pgtable-nop4d.h | 8 ++++++++ include/asm-generic/pgtable-nopmd.h | 8 ++++++++ include/asm-generic/pgtable-nopud.h | 8 ++++++++ 3 files changed, 24 insertions(+) diff --git a/include/asm-generic/pgtable-nop4d.h b/include/asm-generic/pgtable-nop4d.h index 019c3f074b77..ab4a826b3404 100644 --- a/include/asm-generic/pgtable-nop4d.h +++ b/include/asm-generic/pgtable-nop4d.h @@ -34,6 +34,14 @@ static inline bool pgd_leaf(pgd_t pgd) { return false; } */ #define set_pgd(pgdptr, pgdval) set_p4d((p4d_t *)(pgdptr), (p4d_t) { pgdval }) +static inline pgd_t pgdp_get(pgd_t *p4dp) +{ + pgd_t dummy = { 0 }; + + return dummy; +} +#define pgdp_get pgdp_get + static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) { return (p4d_t *)pgd; diff --git a/include/asm-generic/pgtable-nopmd.h b/include/asm-generic/pgtable-nopmd.h index ae2eff44889a..711e12e5d180 100644 --- a/include/asm-generic/pgtable-nopmd.h +++ b/include/asm-generic/pgtable-nopmd.h @@ -44,6 +44,14 @@ static inline void pud_clear(pud_t *pud) { } */ #define set_pud(pudptr, pudval) set_pmd((pmd_t *)(pudptr), (pmd_t) { pudval }) +static inline pud_t pudp_get(pud_t *pudp) +{ + pud_t dummy = { 0 }; + + return dummy; +} +#define pudp_get pudp_get + static inline pmd_t * pmd_offset(pud_t * pud, unsigned long address) { return (pmd_t *)pud; diff --git a/include/asm-generic/pgtable-nopud.h b/include/asm-generic/pgtable-nopud.h index 5a2b0a81ae19..8f01abbb0050 100644 --- a/include/asm-generic/pgtable-nopud.h +++ b/include/asm-generic/pgtable-nopud.h @@ -41,6 +41,14 @@ static inline bool p4d_leaf(p4d_t p4d) { return false; } */ #define set_p4d(p4dptr, p4dval) set_pud((pud_t *)(p4dptr), (pud_t) { p4dval }) +static inline p4d_t p4dp_get(p4d_t *p4dp) +{ + p4d_t dummy = { 0 }; + + return dummy; +} +#define p4dp_get p4dp_get + static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) { return (pud_t *)p4d; -- 2.43.0