From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A765337C93C for ; Sun, 26 Jul 2026 22:23:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.69 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785104618; cv=none; b=n/oBYzl3wYnqvovJX4JSR9z28BO+uLiRzFHicl72ltBPBkjHy34XGtAZzlDOKZiGBE72rvAO4RAE5Snu/XqhqSe+EhdDpmAQ4wZq1GY3DID+cnxULdHUR2HWnBRcWA7JVU7n8Y0f2sVgP0zz9UCKT0kbhYTGM3DIxiu2Uk1a+b4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785104618; c=relaxed/simple; bh=leU4otJNRryxibWooiK3t6+B1PzgqvkRL0+GeVrI9IE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ny6ZsMx2bJR/jmIAhubbWvCurOCl9KKWLZDzyZwrbSGs0Gb+fUGBWRw2LWfMNBW/BZ2sOpswMGqwJkrCPGp/EOZFyA7Z/uqIpKuF+TfXoMcA4euUkehH3YYUPlLcdxSJnpcrhgpK/JG8QuRmuTwKimm+/7IyjQVH3X/Q2fsVYGw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XoD6Ri4p; arc=none smtp.client-ip=209.85.128.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XoD6Ri4p" Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4955e865174so12119425e9.3 for ; Sun, 26 Jul 2026 15:23:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785104615; x=1785709415; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=f4dpUyUWVU4SO+yzE3FO0i+T5bpBbji8PRmiEBEU3h4=; b=XoD6Ri4pTZdX81k0vz+CRw1kkB8sn9mI5C8R3+5j7NzgXeM0eDpraAfqG8cNCzQ6dD TuTaec2Mw49LJ23kUedmuTGbeXjT+l73TWX+2AOiWsTJmmK7oWoJPKqnranvLJeQ9qBC ocFpoT40QOJWpwkwbuRtWVuIwWJD2G+HesoPRhem8HTAPeky6eCVQQPhger29JXscQx5 WvFzIE/f0fv2YtY1B4RGF0MxunuodsLo4mDRwF4xZEu1h3cUn8csOKh+6wtogOE8Y28u CMWm6uPcZpw/CMlpIwUFvHiaPSj4hjQ4N7d0rWkCI6jY6puKWuSWl5/lER5Sw638FNzW YyKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785104615; x=1785709415; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=f4dpUyUWVU4SO+yzE3FO0i+T5bpBbji8PRmiEBEU3h4=; b=cm625ZDN7em7wi/uSgtsY/O5GosFVCJxTgW5zhqb5dIKj86gIKfRkFIFwrd4p0xfNw paPJ094KbwahhGSBewzMpy7Z6nwi8qwioe4yuRvHmF5vCoR8LIxhxerq2WfNPLT/J1GX cfrzChUJcazExEVKmo3WaPKr286xTyNYcjVHDVK77wgU9Lkc5KR27A7AWuLB53eWxs4Y D4c2wF6M/qStNI3U9/FabZfoHbpc/S3vu59EU/G2kg1k7PfzkBqJx0C7af3G5kfCVyFB oZt+B22hcacJgUvCqLGvthNoabzofog7KBto1XwL1lprMalgM2Gs1ZpWm0r5OzrppjQi qKeA== X-Forwarded-Encrypted: i=1; AHgh+RqByzBpvd3CMmcoJVHfjljToIb1IEGytO0Z6PuWtCsbmgUM8L4jWrQeEm8Q6l83bDbaWyRjJm4bThCOhZs=@vger.kernel.org X-Gm-Message-State: AOJu0Yzp6IQRRV3ZiH244jKBxQxOJhowFAu8oHzNwMDbaGgc0dKoWoOk VbwBfRtQPc68zZNYAuSjE54ieMhrPz11HVd5iPydep2oKNVgz0i11NdxFx/zur3OLsBcn1BhKPX +MCbV7uXsCXIEHQ== X-Received: from wmbjz7.prod.google.com ([2002:a05:600c:5807:b0:495:6163:c602]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e86:b0:495:4730:15b2 with SMTP id 5b1f17b1804b1-496b690a8bcmr77698195e9.31.1785104614604; Sun, 26 Jul 2026 15:23:34 -0700 (PDT) Date: Sun, 26 Jul 2026 22:22:43 +0000 In-Reply-To: <20260726-page_alloc-unmapped-v3-0-6f5729aa9832@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260726-page_alloc-unmapped-v3-0-6f5729aa9832@google.com> X-Mailer: b4 0.16-dev Message-ID: <20260726-page_alloc-unmapped-v3-10-6f5729aa9832@google.com> Subject: [PATCH v3 10/26] mm: Add more flags for __apply_to_page_range() From: Brendan Jackman To: Borislav Petkov , Dave Hansen , Peter Zijlstra , Andrew Morton , David Hildenbrand , Vlastimil Babka , Mike Rapoport , Wei Xu , Johannes Weiner , Zi Yan , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, rppt@kernel.org, Sumit Garg , Will Deacon , rientjes@google.com, "Kalyazin, Nikita" , patrick.roy@linux.dev, "Itazuri, Takahiro" , Andy Lutomirski , David Kaplan , Thomas Gleixner , Yosry Ahmed , Patrick Bellasi , Reiji Watanabe , Sean Christopherson , Brendan Jackman Content-Type: text/plain; charset="utf-8" Add two flags to make this API more generic: 1. Separate "create" into two levels - one to allow creating new mappings without allocating pagetables, and one for the current behaviour that allows both of these. 2. Create a new flag to report that the caller has taken care of synchronization and no locks are required. Both of these will serve to allow calling this API from restricted contexts where allocation and pagetable locking are not possible. Signed-off-by: Brendan Jackman --- mm/internal.h | 26 +++++++++++++++++++++++++- mm/memory.c | 59 ++++++++++++++++++++++++++++++++++------------------------- 2 files changed, 59 insertions(+), 26 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 395331a12d62d..5a237d9c5fa96 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1662,9 +1662,33 @@ static inline bool can_spin_trylock(void) /* * Create a mapping if it doesn't exist. (Otherwise, skip regions with no - * existing mapping, and return an error for regions with no leaf pagetable). + * existing mapping). This doesn't allow allocating, most users will want + * PGRANGE_ALLOC. + * + * Do not test this bit directly as it is implied by PGRANGE_ALLOC, use + * pgrange_create() instead. */ #define PGRANGE_CREATE (1 << 0) +/* + * Allocate a pagetable if one is missing. (Otherwise, return an error for + * regions with no leaf pagetable). Also implies PGRANGE_CREATE. + * + * Note that __apply_to_page_range() assumes that pagetables for the area are + * already initialised down to PMD level, so this only affects PTEs in practice. + */ +#define PGRANGE_ALLOC (1 << 1) +/* + * Do not take any locks. This means the caller has taken care of + * synchronisation. This is incompatible with PGRANGE_ALLOC and also with + * mm=&init_mm. + */ +#define PGRANGE_NOLOCK (1 << 2) + + +static inline bool pgrange_create(unsigned int flags) +{ + return flags & (PGRANGE_CREATE | PGRANGE_ALLOC); +} int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, unsigned long size, pte_fn_t fn, diff --git a/mm/memory.c b/mm/memory.c index c4defefea1574..d00508db1021e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3441,30 +3441,35 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, pte_fn_t fn, void *data, unsigned int flags, pgtbl_mod_mask *mask) { - bool create = flags & PGRANGE_CREATE; pte_t *pte, *mapped_pte; int err = 0; spinlock_t *ptl; - if (create) { + if (flags & PGRANGE_ALLOC) { + VM_WARN_ON(flags & PGRANGE_NOLOCK); + mapped_pte = pte = (mm == &init_mm) ? pte_alloc_kernel_track(pmd, addr, mask) : pte_alloc_map_lock(mm, pmd, addr, &ptl); if (!pte) return -ENOMEM; } else { - mapped_pte = pte = (mm == &init_mm) ? - pte_offset_kernel(pmd, addr) : - pte_offset_map_lock(mm, pmd, addr, &ptl); + if (mm == &init_mm) + pte = pte_offset_kernel(pmd, addr); + else if (flags & PGRANGE_NOLOCK) + pte = pte_offset_map(pmd, addr); + else + pte = pte_offset_map_lock(mm, pmd, addr, &ptl); if (!pte) return -EINVAL; + mapped_pte = pte; } lazy_mmu_mode_enable(); if (fn) { do { - if (create || !pte_none(ptep_get(pte))) { + if (pgrange_create(flags) || !pte_none(ptep_get(pte))) { err = fn(pte, addr, data); if (err) break; @@ -3475,8 +3480,13 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, lazy_mmu_mode_disable(); - if (mm != &init_mm) - pte_unmap_unlock(mapped_pte, ptl); + if (mm != &init_mm) { + if (flags & PGRANGE_NOLOCK) + pte_unmap(mapped_pte); + else + pte_unmap_unlock(mapped_pte, ptl); + } + return err; } @@ -3486,13 +3496,12 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, pgtbl_mod_mask *mask) { pmd_t *pmd; - bool create = flags & PGRANGE_CREATE; unsigned long next; int err = 0; BUG_ON(pud_leaf(*pud)); - if (create) { + if (flags & PGRANGE_ALLOC) { pmd = pmd_alloc_track(mm, pud, addr, mask); if (!pmd) return -ENOMEM; @@ -3501,12 +3510,12 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, } do { next = pmd_addr_end(addr, end); - if (pmd_none(*pmd) && !create) + if (pmd_none(*pmd) && !pgrange_create(flags)) continue; if (WARN_ON_ONCE(pmd_leaf(*pmd))) return -EINVAL; if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) { - if (!create) + if (!pgrange_create(flags)) continue; pmd_clear_bad(pmd); } @@ -3525,11 +3534,10 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, pgtbl_mod_mask *mask) { pud_t *pud; - bool create = flags & PGRANGE_CREATE; unsigned long next; int err = 0; - if (create) { + if (flags & PGRANGE_ALLOC) { pud = pud_alloc_track(mm, p4d, addr, mask); if (!pud) return -ENOMEM; @@ -3538,17 +3546,17 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, } do { next = pud_addr_end(addr, end); - if (pud_none(*pud) && !create) + if (pud_none(*pud) && !pgrange_create(flags)) continue; if (WARN_ON_ONCE(pud_leaf(*pud))) return -EINVAL; if (!pud_none(*pud) && WARN_ON_ONCE(pud_bad(*pud))) { - if (!create) + if (!pgrange_create(flags)) continue; pud_clear_bad(pud); } err = apply_to_pmd_range(mm, pud, addr, next, - fn, data, create, mask); + fn, data, flags, mask); if (err) break; } while (pud++, addr = next, addr != end); @@ -3562,11 +3570,10 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, pgtbl_mod_mask *mask) { p4d_t *p4d; - bool create = flags & PGRANGE_CREATE; unsigned long next; int err = 0; - if (create) { + if (flags & PGRANGE_ALLOC) { p4d = p4d_alloc_track(mm, pgd, addr, mask); if (!p4d) return -ENOMEM; @@ -3575,12 +3582,12 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, } do { next = p4d_addr_end(addr, end); - if (p4d_none(*p4d) && !create) + if (p4d_none(*p4d) && !pgrange_create(flags)) continue; if (WARN_ON_ONCE(p4d_leaf(*p4d))) return -EINVAL; if (!p4d_none(*p4d) && WARN_ON_ONCE(p4d_bad(*p4d))) { - if (!create) + if (!pgrange_create(flags)) continue; p4d_clear_bad(p4d); } @@ -3598,7 +3605,6 @@ int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, void *data, unsigned int flags) { pgd_t *pgd; - bool create = flags & PGRANGE_CREATE; unsigned long start = addr, next; unsigned long end = addr + size; pgtbl_mod_mask mask = 0; @@ -3606,18 +3612,21 @@ int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, if (WARN_ON(addr >= end)) return -EINVAL; + if (WARN_ON(flags & PGRANGE_NOLOCK && + (mm == &init_mm || flags & PGRANGE_ALLOC))) + return -EINVAL; pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); - if (pgd_none(*pgd) && !create) + if (pgd_none(*pgd) && !pgrange_create(flags)) continue; if (WARN_ON_ONCE(pgd_leaf(*pgd))) { err = -EINVAL; break; } if (!pgd_none(*pgd) && WARN_ON_ONCE(pgd_bad(*pgd))) { - if (!create) + if (!pgrange_create(flags)) continue; pgd_clear_bad(pgd); } @@ -3640,7 +3649,7 @@ int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, int apply_to_page_range(struct mm_struct *mm, unsigned long addr, unsigned long size, pte_fn_t fn, void *data) { - return __apply_to_page_range(mm, addr, size, fn, data, PGRANGE_CREATE); + return __apply_to_page_range(mm, addr, size, fn, data, PGRANGE_ALLOC); } EXPORT_SYMBOL_GPL(apply_to_page_range); -- 2.54.0