From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FC6F3ED126 for ; Wed, 25 Feb 2026 16:35:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772037305; cv=none; b=XJT+Nf1/rDJTRQ1N94I1Kjj3B77A3z7JKhLTmexDBgH/yf5xeMefOxYRSNSB0mEm72th+I94CJ7RvAjsBt4srXq7S4OJyAEWReuNSExSemuRXmixvOBq1+9e/hcLWbOSSStFTdNTCc18DynUpVqesfgPTn3UhzxmD4Sv4J20PGI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772037305; c=relaxed/simple; bh=ctPEukN6fm0ljH0l7Bo/HryqNSf7WZZpoSOg0AzOQys=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kgwtUY9yFfxQJRV3nq3KFLVdZPBEsWSWSMwssHjfLgf4J1VQ6Y4pjAyiRimaw+g7Q4vQcKTZjBYWk3IsoCmagHtBLXHWl44WLAXiRkklrd5lEZfIJQuMtSDWUTHEg9UElmcX7B8swSqKl91Tq+JAIEk7r7AZwU/zxUFGElwaWPY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mMx+AMCg; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mMx+AMCg" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-4398ad5d81dso1105893f8f.0 for ; Wed, 25 Feb 2026 08:35:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1772037302; x=1772642102; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8IxPs+shi9o6AXlEGCilkkLX6RGxoXvE/7sr5zHVQPU=; b=mMx+AMCgp3sLytjXSrEK/pC2Xez2H547V1cIYYdXjf5lqPVSYD3UKaVaTOTP3rJnJX /kiBYGoB+HSB6x4mOdN/BYvZScWyZIZcWkjb27VP1BulqyfzdMyiAhRZaeFAKtR4/nr9 dELMoF9C2+bbikknl/K/sXCJODTHa7oZK2obsP61yNxOrOxblOFlNFDa1I36Xplu8rH3 5uu70akLFpq41BIqVDEZ5aood1tFelrF0MtgbTZLTys562Hsoysmcf33c1WaSR3zqbLN uCXIpqV489gQsql8I09z18f3hxywL5jdEY4C/rSmpimsAFC5XCJxVuPaBHHxq0R6D5Qg KzrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772037302; x=1772642102; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8IxPs+shi9o6AXlEGCilkkLX6RGxoXvE/7sr5zHVQPU=; b=ZWZH0xgZrtSNjE+JL+FvftNmJl6QG2f6v0ImmgUma8qug7pMKlAZiVOu6DLRgfGXhc rgwZhFsHeRd357t6ObTGuVGEeHmvC2gKPCD7zz6/6D2dOLCzw5dYNXKcJyoRgcrgVo7V CFR7WHUUnmMlZGi2IZGUyPEqLyn+xoQKreqiWX9MSUcwS++4+FGkHFu5UlDiOBeoHyDs nSkQmHaV49RmIgVqqKpQtjmSltNb0d6TzzkfTjElssPrGLJ0QKfdJUn8TMHUYE/qwV6c V5v5WyY9UDxuCUxded87HlUbpu/JuHPb0msSUj4WwOxal1uKEF1d1Jx70hAN0v3E0d3V fkwQ== X-Forwarded-Encrypted: i=1; AJvYcCUfiVM3JHtFUeBKPKACfeZFNEoaX4h0zsfuNdLGdmDeFVVPIpf8wLs8fOm9AMXfaVR/IcbzUl25g3xwU3c=@vger.kernel.org X-Gm-Message-State: AOJu0Yx39GA+1YxHo2MXyKzxSguoclTfQpq98j3V85yPkeeJjv6VPoMS IiuEaKDAN+JjBFzq+E+WNPWY5HnwkRGzu1PpzLfIvfJ8NCdekivwKjrrTvbMhsfaS00v3YgPCo4 PCQ4OTrDeyV1SYw== X-Received: from wrph15.prod.google.com ([2002:adf:f4cf:0:b0:42f:b132:61e6]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:64c6:0:b0:435:b755:c67e with SMTP id ffacd0b85a97d-439942fbd3cmr2235545f8f.49.1772037300603; Wed, 25 Feb 2026 08:35:00 -0800 (PST) Date: Wed, 25 Feb 2026 16:34:43 +0000 In-Reply-To: <20260225-page_alloc-unmapped-v1-0-e8808a03cd66@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225-page_alloc-unmapped-v1-0-e8808a03cd66@google.com> X-Mailer: b4 0.14.3 Message-ID: <20260225-page_alloc-unmapped-v1-18-e8808a03cd66@google.com> Subject: [PATCH RFC 18/19] mm/page_alloc: implement __GFP_UNMAPPED|__GFP_ZERO allocations From: Brendan Jackman To: Borislav Petkov , Dave Hansen , Peter Zijlstra , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Vlastimil Babka , Wei Xu , Johannes Weiner , Zi Yan Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, rppt@kernel.org, Sumit Garg , derkling@google.com, reijiw@google.com, Will Deacon , rientjes@google.com, "Kalyazin, Nikita" , patrick.roy@linux.dev, "Itazuri, Takahiro" , Andy Lutomirski , David Kaplan , Thomas Gleixner , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" The pages being zeroed here are unmapped, so they can't be zeroed via the direct map. Temporarily mapping them in the direct map is not possible because: - In general this requires allocating pagetables, - Unmapping them would require a TLB shootdown, which can't be done in general from the allocator (x86 requires IRQs on). Therefore, use the new mermap mechanism to zero these pages. The main mermap API is expected to fail very often. In order to avoid needing to fail allocations when that happens, instead fallback to the special mermap_get_reserved() variant, which is less efficient. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/pgtable_types.h | 2 + mm/Kconfig | 12 +++++- mm/page_alloc.c | 76 +++++++++++++++++++++++++++++++----- 3 files changed, 79 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 2ec250ba467e2..c3d73bdfff1fa 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -223,6 +223,7 @@ enum page_cache_mode { #define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) #define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G) #define __PAGE_KERNEL (__PP|__RW| 0|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_NOGLOBAL (__PP|__RW| 0|___A|__NX|___D| 0| 0) #define __PAGE_KERNEL_EXEC (__PP|__RW| 0|___A| 0|___D| 0|___G) #define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) #define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G) @@ -245,6 +246,7 @@ enum page_cache_mode { #define __pgprot_mask(x) __pgprot((x) & __default_kernel_pte_mask) #define PAGE_KERNEL __pgprot_mask(__PAGE_KERNEL | _ENC) +#define PAGE_KERNEL_NOGLOBAL __pgprot_mask(__PAGE_KERNEL_NOGLOBAL | _ENC) #define PAGE_KERNEL_NOENC __pgprot_mask(__PAGE_KERNEL | 0) #define PAGE_KERNEL_RO __pgprot_mask(__PAGE_KERNEL_RO | _ENC) #define PAGE_KERNEL_EXEC __pgprot_mask(__PAGE_KERNEL_EXEC | _ENC) diff --git a/mm/Kconfig b/mm/Kconfig index 3200ea8836432..134c6aab6fc50 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1503,7 +1503,15 @@ config MERMAP_KUNIT_TEST If unsure, say N. config PAGE_ALLOC_UNMAPPED - bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST - default COMPILE_TEST + bool "Support allocating pages that aren't in the direct map" if COMPILE_TEST || KUNIT + default COMPILE_TEST || KUNIT + depends on MERMAP + +config PAGE_ALLOC_KUNIT_TESTS + tristate "KUnit tests for the page allocator" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + Builds KUnit tests for the page allocator. endmenu diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f7754080dd25b..9b35e91dadeb5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -14,6 +14,7 @@ * (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ +#include #include #include #include @@ -1365,15 +1366,72 @@ static inline bool should_skip_kasan_poison(struct page *page) return page_kasan_tag(page) == KASAN_TAG_KERNEL; } -static void kernel_init_pages(struct page *page, int numpages) +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED +static inline bool pageblock_unmapped(struct page *page) { - int i; + return freetype_flags(get_pageblock_freetype(page)) & FREETYPE_UNMAPPED; +} - /* s390's use of memset() could override KASAN redzones. */ - kasan_disable_current(); - for (i = 0; i < numpages; i++) - clear_highpage_kasan_tagged(page + i); - kasan_enable_current(); +static inline void clear_page_mermap(struct page *page, unsigned int numpages) +{ + void *mermap; + + BUILD_BUG_ON(IS_ENABLED(CONFIG_HIGHMEM)); + + /* Fast path: single mapping (may fail under preemption). */ + mermap = mermap_get(page, numpages << PAGE_SHIFT, PAGE_KERNEL_NOGLOBAL); + if (mermap) { + void *buf = kasan_reset_tag(mermap_addr(mermap)); + + for (int i = 0; i < numpages; i++) + clear_page(buf + (i << PAGE_SHIFT)); + mermap_put(mermap); + return; + } + + /* Slow path, map each page individually (always succeeds). */ + for (int i = 0; i < numpages; i++) { + unsigned long flags; + + local_irq_save(flags); + mermap = mermap_get_reserved(page + i, PAGE_KERNEL); + clear_page(kasan_reset_tag(mermap_addr(mermap))); + mermap_put(mermap); + local_irq_restore(flags); + } +} +#else +static inline bool pageblock_unmapped(struct page *page) +{ + return false; +} + +static inline void clear_page_mermap(struct page *page, unsigned int numpages) +{ + BUG(); +} +#endif + +static void kernel_init_pages(struct page *page, unsigned int numpages) +{ + int num_blocks = DIV_ROUND_UP(numpages, pageblock_nr_pages); + + for (int block = 0; block < num_blocks; block++) { + struct page *block_page = page + (block << pageblock_order); + bool unmapped = pageblock_unmapped(block_page); + + /* s390's use of memset() could override KASAN redzones. */ + kasan_disable_current(); + if (unmapped) { + clear_page_mermap(page, numpages); + } else { + for (int i = 0; i < min(numpages, pageblock_nr_pages); i++) + clear_highpage_kasan_tagged(block_page + i); + } + kasan_enable_current(); + + numpages -= pageblock_nr_pages; + } } #ifdef CONFIG_MEM_ALLOC_PROFILING @@ -5284,8 +5342,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, ac->nodemask = nodemask; ac->freetype = gfp_freetype(gfp_mask); - /* Not implemented yet. */ - if (freetype_flags(ac->freetype) & FREETYPE_UNMAPPED && gfp_mask & __GFP_ZERO) + if (freetype_flags(ac->freetype) & FREETYPE_UNMAPPED && + WARN_ON(!mermap_ready())) return false; if (cpusets_enabled()) { -- 2.51.2