From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD23B3CA486 for ; Sun, 26 Jul 2026 22:23:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.69 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785104620; cv=none; b=t4FqEvFdT2VYsnpy6VhIckOp2vxMsyPV9rcqHrfxAYNCg+TKiopDQ8IXz8R3BOhooWMRTDwt/nLofrtZZ53a5cX5IJKXmgKvXTycOrqJ8d+zznHdyq+7OIX058bI4XhIq8V0rEgGNJ2bVulfq6EVdLmf0n852YjtdoxXOX4jPpQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785104620; c=relaxed/simple; bh=jyfd3j9n84GqOIWNOxkE5J3XhLw1zeZVutD95NY5tkE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SHF99X4wEg0/arszN/J49UBScORENtQ9ZC4ycYKCQdBxzuL1M/s7itjGoSDra9HPOA6OV5rBf6fpGpyvOSNvnGfWCihVlMaUqVv4jHTzB3B8F2YmDzjiuwfLgFDOScpxYulXjm51NPkI15kzxQJpzD4gS+EYMm8xNwJWgHXCLtc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FQN4t3M8; arc=none smtp.client-ip=209.85.208.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FQN4t3M8" Received: by mail-ed1-f69.google.com with SMTP id 4fb4d7f45d1cf-69843ac68d0so2111898a12.3 for ; Sun, 26 Jul 2026 15:23:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785104611; x=1785709411; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=t3PhqaxxZ1xAjFZ/fCk/opbDZbTshcWyWRb6oD1k9p0=; b=FQN4t3M8y0z8jYRb+i0YD16tOyilhyUkf0Z+2NkKjwLmgLFVsnIYyjMUPDsRlq3FLv LYjdJE+AebyBMBxRRjPl2l6ajcjsOwGo/iT5zi0BS9YeKPGTuDjJD5oWerOXYSiJji64 q8hUhSScWoIe2iIA+kLJEJFZcXT3S7WeGExPdUFXC8QJIjm/5lmz5WIJ/iKy5WaElRwH 3K63YBflDDOL48IUtdZPyZPyHuDrSylmkGjZnstvy05E9DE2KIY4lL3rJtXDL1wH/nB7 wGTm9q3p16ofBRrFpjhur7xSYEWlXbkg8UPdcTUvdzRxKpv4OVeHID/EIUdAygYop1AR K2VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785104611; x=1785709411; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=t3PhqaxxZ1xAjFZ/fCk/opbDZbTshcWyWRb6oD1k9p0=; b=cJgoxYXdPPkTUJc/VegbVh3aIvPR8xPCwYuJYc/havfkthE2YG/BNENYq51JlXOfCq a3mLnMGcXL53O5YZ7FG3i9kf7trTN6G1lU4q1YPY70bQbNXPft8HyzzKfblpBDTO505w ADjOYKN7PYf/vP5GYjOWFXgQaTwJsWUMR83wgw0ih52QzbBhjGdF+l4EVHP0ARlTOYjp ryY5GYu6WyJOuBoESwpQqLV4IjTy10utheAkBEoyiliGFBrc5Zz+wHiCwy4DdkFFkzaj 9IFnrpmNVi+bOXASF9kABlGhH3alPMWzZVHaEivFxr+r+TYQEUZR79LMj8GKVY00jrbt iVTA== X-Forwarded-Encrypted: i=1; AHgh+Ron0OsqNy9K1+BM3SvAoSKFUxsFD8KyBVRrc4IoOZwGSqxVhreOf4P2x0DD1foG+qcFlG07B6yauf+HlXk=@vger.kernel.org X-Gm-Message-State: AOJu0YxV9ZlOv1Net4d0QyvlNswsT37lVi0JlCWlA0tL+J4W53aR1A2b F2hSQyDqXMSNLPo55gvoC2qywYmy9IQ5DcSegZE56fTGFEmZNKyNI+SsAnf2bOoSJX5cIy8lcfk cQ15x0FD1YQbLnw== X-Received: from edqj22.prod.google.com ([2002:aa7:c416:0:b0:69f:b160:de14]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:2b8a:b0:698:3b7c:7e56 with SMTP id 4fb4d7f45d1cf-69fc116ad83mr2202096a12.34.1785104610714; Sun, 26 Jul 2026 15:23:30 -0700 (PDT) Date: Sun, 26 Jul 2026 22:22:40 +0000 In-Reply-To: <20260726-page_alloc-unmapped-v3-0-6f5729aa9832@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260726-page_alloc-unmapped-v3-0-6f5729aa9832@google.com> X-Mailer: b4 0.16-dev Message-ID: <20260726-page_alloc-unmapped-v3-7-6f5729aa9832@google.com> Subject: [PATCH v3 07/26] x86/mm: introduce mm-local region From: Brendan Jackman To: Borislav Petkov , Dave Hansen , Peter Zijlstra , Andrew Morton , David Hildenbrand , Vlastimil Babka , Mike Rapoport , Wei Xu , Johannes Weiner , Zi Yan , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, rppt@kernel.org, Sumit Garg , Will Deacon , rientjes@google.com, "Kalyazin, Nikita" , patrick.roy@linux.dev, "Itazuri, Takahiro" , Andy Lutomirski , David Kaplan , Thomas Gleixner , Yosry Ahmed , Patrick Bellasi , Reiji Watanabe , Sean Christopherson , Brendan Jackman Content-Type: text/plain; charset="utf-8" Various security features benefit from having process-local address mappings within the kernel. Examples include no-direct-map guest_memfd [2] and significant optimizations for ASI [1]. With the currently envisaged usecases, there will be many situations where almost no processes have any need for the mm-local region. Therefore, avoid its overhead (memory cost of pagetables, alloc/free overhead during fork/exit) for processes that don't use it by requiring its users to explicitly initialize it via the new mm_local_* API. As pointed out by Andy in [0], x86 already has a PGD entry that is local to the mm, which is used for the LDT. In a subsequent patch, the LDT remap will be unified with the general mm-local region, but to help keep the patch to a manageable size, first just introduce the mm-local region. On 64-bit, give the mm-local region a whole PGD. On 32-bit, just give it one PMD. No investigation has been done into whether it's feasible to expand the region on 32-bit. Most likely there is no strong usecase for that anyway. In order to combine the need for an on-demand mm initialisation, with the desire to transparently handle propagating mappings to userspace under KPTI, the user and kernel pagetables are shared at the highest level possible. For PAE that means the PTE table is shared and for 64-bit the P4D/PUD. This is implemented by pre-allocating the first shared table when the mm-local region is first initialised. [0] https://lore.kernel.org/linux-mm/CALCETrXHbS9VXfZ80kOjiTrreM2EbapYeGp68mvJPbosUtorYA@mail.gmail.com/ [1] https://linuxasi.dev/ [2] https://lore.kernel.org/all/20250924151101.2225820-1-patrick.roy@campus.lmu.de Signed-off-by: Brendan Jackman --- arch/x86/Kconfig | 2 + arch/x86/include/asm/mmu_context.h | 118 +++++++++++++++++++++++++++++++- arch/x86/include/asm/pgtable_32_areas.h | 9 ++- arch/x86/include/asm/pgtable_64_types.h | 12 +++- arch/x86/mm/pgtable.c | 3 + include/linux/mm.h | 13 ++++ include/linux/mm_types.h | 2 + kernel/fork.c | 1 + mm/Kconfig | 7 ++ 9 files changed, 161 insertions(+), 6 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index fb298e2191792..3efab3524a6cf 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -132,6 +132,8 @@ config X86 select ARCH_SUPPORTS_LTO_CLANG select ARCH_SUPPORTS_LTO_CLANG_THIN select ARCH_SUPPORTS_RT + # LDT remap temporarily clashes with mm-local region, can't have both. + select ARCH_SUPPORTS_MM_LOCAL_REGION if X86_64 || X86_PAE && !MODIFY_LDT_SYSCALL select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF select ARCH_USE_MEMTEST diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index ef5b507de34e2..3d4f54673014f 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -8,8 +8,10 @@ #include +#include #include #include +#include #include #include #include @@ -223,10 +225,124 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) return ldt_dup_context(oldmm, mm); } +#ifdef CONFIG_MM_LOCAL_REGION +static inline void mm_local_region_free(struct mm_struct *mm) +{ + if (!mm_local_region_used(mm)) + return; + + struct mmu_gather tlb; + unsigned long start = MM_LOCAL_BASE_ADDR; + unsigned long end = MM_LOCAL_END_ADDR; + + /* + * Although free_pgd_range() is intended for freeing user + * page-tables, it also works out for kernel mappings on x86. + * Use tlb_gather_mmu_fullmm() to avoid confusing the + * range-tracking logic in __tlb_adjust_range(). + */ + tlb_gather_mmu_fullmm(&tlb, mm); + free_pgd_range(&tlb, start, end, start, end); + tlb_finish_mmu(&tlb); + + mm_flags_clear(MMF_LOCAL_REGION_USED, mm); +} + +#if defined(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) && defined(CONFIG_X86_PAE) +static inline pmd_t *pgd_to_pmd_walk(pgd_t *pgd, unsigned long va) +{ + p4d_t *p4d; + pud_t *pud; + + if (pgd->pgd == 0) + return NULL; + + p4d = p4d_offset(pgd, va); + if (p4d_none(*p4d)) + return NULL; + + pud = pud_offset(p4d, va); + if (pud_none(*pud)) + return NULL; + + return pmd_offset(pud, va); +} + +static inline int mm_local_map_to_user(struct mm_struct *mm) +{ + BUILD_BUG_ON(!PREALLOCATED_PMDS); + pgd_t *k_pgd = pgd_offset(mm, MM_LOCAL_BASE_ADDR); + pgd_t *u_pgd = kernel_to_user_pgdp(k_pgd); + pmd_t *k_pmd, *u_pmd; + int err; + + k_pmd = pgd_to_pmd_walk(k_pgd, MM_LOCAL_BASE_ADDR); + u_pmd = pgd_to_pmd_walk(u_pgd, MM_LOCAL_BASE_ADDR); + + BUILD_BUG_ON(MM_LOCAL_END_ADDR - MM_LOCAL_BASE_ADDR > PMD_SIZE); + + /* Preallocate the PTE table so it can be shared. */ + err = pte_alloc(mm, k_pmd); + if (err) + return err; + + /* Point the userspace PMD at the same PTE as the kernel PMD. */ + set_pmd(u_pmd, *k_pmd); + return 0; +} +#elif defined(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) +static inline int mm_local_map_to_user(struct mm_struct *mm) +{ + pgd_t *pgd; + int err; + + err = preallocate_sub_pgd(mm, MM_LOCAL_BASE_ADDR); + if (err) + return err; + + pgd = pgd_offset(mm, MM_LOCAL_BASE_ADDR); + set_pgd(kernel_to_user_pgdp(pgd), *pgd); + return 0; +} +#else +static inline int mm_local_map_to_user(struct mm_struct *mm) +{ + WARN_ONCE(1, "mm_local_map_to_user() not implemented"); + return -EINVAL; +} +#endif + +/* + * Do initial setup of the mm-local region. Call from process context. + * + * Under PTI, userspace shares the pagetables for the mm-local region with the + * kernel (if you map stuff here, it's immediately mapped into userspace too). + * LDT remap. It's assuming nothing gets mapped in here that needs to be + * protected from Meltdown-type attacks from the current process. + */ +static inline int mm_local_region_init(struct mm_struct *mm) +{ + int err; + + if (boot_cpu_has(X86_FEATURE_PTI)) { + err = mm_local_map_to_user(mm); + if (err) + return err; + } + + mm_flags_set(MMF_LOCAL_REGION_USED, mm); + + return 0; +} + +#else +static inline void mm_local_region_free(struct mm_struct *mm) { } +#endif /* CONFIG_MM_LOCAL_REGION */ + static inline void arch_exit_mmap(struct mm_struct *mm) { paravirt_arch_exit_mmap(mm); - ldt_arch_exit_mmap(mm); + mm_local_region_free(mm); } #ifdef CONFIG_X86_64 diff --git a/arch/x86/include/asm/pgtable_32_areas.h b/arch/x86/include/asm/pgtable_32_areas.h index 921148b429676..7fccb887f8b33 100644 --- a/arch/x86/include/asm/pgtable_32_areas.h +++ b/arch/x86/include/asm/pgtable_32_areas.h @@ -30,9 +30,14 @@ extern bool __vmalloc_start_set; /* set once high_memory is set */ #define CPU_ENTRY_AREA_BASE \ ((FIXADDR_TOT_START - PAGE_SIZE*(CPU_ENTRY_AREA_PAGES+1)) & PMD_MASK) -#define LDT_BASE_ADDR \ - ((CPU_ENTRY_AREA_BASE - PAGE_SIZE) & PMD_MASK) +/* + * On 32-bit the mm-local region is currently completely consumed by the LDT + * remap. + */ +#define MM_LOCAL_BASE_ADDR ((CPU_ENTRY_AREA_BASE - PAGE_SIZE) & PMD_MASK) +#define MM_LOCAL_END_ADDR (MM_LOCAL_BASE_ADDR + PMD_SIZE) +#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR #define LDT_END_ADDR (LDT_BASE_ADDR + PMD_SIZE) #define PKMAP_BASE \ diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 7eb61ef6a185f..1181565966405 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -5,8 +5,11 @@ #include #ifndef __ASSEMBLER__ +#include #include #include +#include +#include /* * These are used to make use of C type-checking.. @@ -100,9 +103,12 @@ extern unsigned int ptrs_per_p4d; #define GUARD_HOLE_BASE_ADDR (GUARD_HOLE_PGD_ENTRY << PGDIR_SHIFT) #define GUARD_HOLE_END_ADDR (GUARD_HOLE_BASE_ADDR + GUARD_HOLE_SIZE) -#define LDT_PGD_ENTRY -240UL -#define LDT_BASE_ADDR (LDT_PGD_ENTRY << PGDIR_SHIFT) -#define LDT_END_ADDR (LDT_BASE_ADDR + PGDIR_SIZE) +#define MM_LOCAL_PGD_ENTRY -240UL +#define MM_LOCAL_BASE_ADDR (MM_LOCAL_PGD_ENTRY << PGDIR_SHIFT) +#define MM_LOCAL_END_ADDR ((MM_LOCAL_PGD_ENTRY + 1) << PGDIR_SHIFT) + +#define LDT_BASE_ADDR MM_LOCAL_BASE_ADDR +#define LDT_END_ADDR (LDT_BASE_ADDR + PMD_SIZE) #define __VMALLOC_BASE_L4 0xffffc90000000000UL #define __VMALLOC_BASE_L5 0xffa0000000000000UL diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 8f3505b1eefec..cc768ea8c19c8 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -335,6 +335,9 @@ pgd_t *pgd_alloc(struct mm_struct *mm) void pgd_free(struct mm_struct *mm, pgd_t *pgd) { + /* Should be cleaned up in mmap exit path. */ + VM_WARN_ON_ONCE(mm_local_region_used(mm)); + pgd_mop_up_pmds(mm, pgd); pgd_dtor(pgd); paravirt_pgd_free(mm, pgd); diff --git a/include/linux/mm.h b/include/linux/mm.h index 7fabe6c66b4b7..28022c96edcb5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -989,6 +989,19 @@ static inline void mm_flags_clear_all(struct mm_struct *mm) bitmap_zero(ACCESS_PRIVATE(&mm->flags, __mm_flags), NUM_MM_FLAG_BITS); } +#ifdef CONFIG_MM_LOCAL_REGION +static inline bool mm_local_region_used(struct mm_struct *mm) +{ + return mm_flags_test(MMF_LOCAL_REGION_USED, mm); +} +#else +static inline bool mm_local_region_used(struct mm_struct *mm) +{ + VM_WARN_ON_ONCE(mm_flags_test(MMF_LOCAL_REGION_USED, mm)); + return false; +} +#endif + extern const struct vm_operations_struct vma_dummy_vm_ops; static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index b5d4cd3b067bf..d39fddf57edc8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1997,6 +1997,8 @@ enum { #define MMF_TOPDOWN 31 /* mm searches top down by default */ #define MMF_TOPDOWN_MASK BIT(MMF_TOPDOWN) +#define MMF_LOCAL_REGION_USED 32 + #define MMF_INIT_LEGACY_MASK (MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK) diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5a..4fb23ea33b7da 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1154,6 +1154,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p) fail_nocontext: mm_free_id(mm); fail_noid: + WARN_ON_ONCE(mm_local_region_used(mm)); mm_free_pgd(mm); fail_mm_init: free_mm(mm); diff --git a/mm/Kconfig b/mm/Kconfig index c52ab6afcb159..cb531c1436f77 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1500,6 +1500,13 @@ config LAZY_MMU_MODE_KUNIT_TEST If unsure, say N. +config ARCH_SUPPORTS_MM_LOCAL_REGION + bool + +config MM_LOCAL_REGION + bool + depends on ARCH_SUPPORTS_MM_LOCAL_REGION + source "mm/damon/Kconfig" endmenu -- 2.54.0