From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E5DD3E717F for ; Tue, 4 Aug 2026 23:41:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785886900; cv=none; b=Fq24AN3baFI2Vc9X+kPAaXhhCfYDzSt83GvYZD5g4YsDK9whu1UMe3mFnOofgkevlGkUjCW/ROZO9QbC3xYMP+JaHwtyAW3+tb9MgnrBLtELe0X++PEEi9i0Kzg8ntxqfKvTyyhOJZSj7KQ9K735tQuuGHq4TLJOSz0UOYvTmTw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785886900; c=relaxed/simple; bh=lfOznkludt6Es+4cg1kg60+mPRUIrMs3phX+6TkrUMQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=BCBwx3LvMj1qvBkNqmkOxtEBHgxCj7XfqNcNoGn3HgnSejJna5v/5SOu9PHRdzaTQUvKy6zf5kg0UAwuK41k44/AFCX61Pa8TXqF88t1DlQWzWlR0GiCHHRfLYAYELkGpeoSw15N2/Gr4QX7LTt/CPpX+O+Y15svAzUuF59Nkkg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KwDcKFe5; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KwDcKFe5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BD5E21F000E9; Tue, 4 Aug 2026 23:41:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1785886898; bh=2EUCMgHBanhRcpfI756++H/l3l8ZfGyKrWQyzXhKbkY=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=KwDcKFe58ufznJCPvjxMCvF7i7E2j8fGLj09mChK0reeO8Iml6E+/DNaZqGa6vz/P G44jbZMvSbZ/dnNXUsZOAh43aDCsIYtd6Oocrmpues3F+LNyqAc0bFmlEMQZ9rBQYp e/sxYQsDKPxjP7qQ5wNQwTrYskuM4vgR8HEoXV9SHYDLk9ryEqAkGU/IZm/E2Jysnr 67n1UxBR8fySD6NNGMTo57owUnIZXkz1ca4eVosHLT6blttfAcScUVKpM2XF4waWF0 TaspZrmX9L6crp1j1dk5UM2IKnq7sKEQa7lnyhH22vb8NW14yWYAXxAuKZqhEsVRbB 2WA37Q0SpG1ww== Date: Tue, 4 Aug 2026 23:41:36 +0000 From: Yosry Ahmed To: Brendan Jackman Cc: Borislav Petkov , Dave Hansen , Peter Zijlstra , Andrew Morton , David Hildenbrand , Vlastimil Babka , Mike Rapoport , Wei Xu , Johannes Weiner , Zi Yan , Lorenzo Stoakes , linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Sumit Garg , Will Deacon , rientjes@google.com, "Kalyazin, Nikita" , patrick.roy@linux.dev, "Itazuri, Takahiro" , Andy Lutomirski , David Kaplan , Thomas Gleixner , Patrick Bellasi , Reiji Watanabe , Sean Christopherson Subject: Re: [PATCH v3 21/26] mm/page_alloc: implement FREETYPE_UNMAPPED allocations Message-ID: References: <20260726-page_alloc-unmapped-v3-0-6f5729aa9832@google.com> <20260726-page_alloc-unmapped-v3-21-6f5729aa9832@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260726-page_alloc-unmapped-v3-21-6f5729aa9832@google.com> On Sun, Jul 26, 2026 at 10:22:54PM +0000, Brendan Jackman wrote: > Currently FREETYPE_UNMAPPED allocs will always fail because, although the > lists exist to hold them, there is no way to actually create an unmapped > page block. This commit adds one, and also the logic to map it back > again when that's needed. > > Doing this at pageblock granularity ensures that the pageblock flags can > be used to infer which freetype a page belongs to. It also provides nice > batching of TLB flushes, and also avoids creating too much unnecessary > TLB fragmentation in the physmap. > > There are some functional requirements for flipping a block: > > - Unmapping requires a TLB shootdown, meaning IRQs must be enabled. > > - Updating the pagetables might require allocating a pagetable to break > down a huge page. This would deadlock if the zone lock was held. We also need to zero unmapped/sensitive pages before mapping them again, but seems like the current approach is to differ this to the caller, which makes sense. The only annoying part is that want_init_on_{free/alloc}() now silently skip the zeroing for those pages. We should document somewhere that the users of ALLOC_UNMAPPED are responsible for zeroing memory before freeing it? The current users currently always zero the pages on allocation as well, I am not sure if this should also be a general requirement, or perhaps only if want_init_on_alloc() is set? > > This makes allocations that need to change sensitivity _somewhat_ s/sensitivity/direct mapping status (or sth)? > similar to those that need to fallback to a different migratetype. But, > the locking requirements mean that this can't just be squashed into the > existing "fallback" allocator logic, instead a new allocator path just > for this purpose is needed. > > The new path is assumed to be much cheaper than the really heavyweight > stuff like compaction and reclaim. But at present it is treated as less > desirable than the mobility-related "fallback" and "stealing" logic. > This might turn out to need revision (in particular, maybe it's a > problem that __rmqueue_steal(), which causes fragmentation, happens > before __rmqueue_direct_map()), but that should be treated as a subsequent > optimisation project. > > Adding alloc_flags to gfp_freetype() requires moving it to > mm/page_alloc.h so it can refer to ALLOC_UNMAPPED. It was already only > used in internal mm code. > > Now that unmapped pageblocks actually exist, exclude them from > migration. Migrating unmapped pages via the mermap should be possible > but that's something to be added later when needed. > > Signed-off-by: Brendan Jackman [..] > @@ -3400,6 +3426,127 @@ static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, > #endif > } > > +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED > +/* Try to allocate a page by mapping/unmapping a block from the direct map. */ > +static inline struct page * > +__rmqueue_direct_map(struct zone *zone, unsigned int request_order, > + unsigned int alloc_flags, freetype_t freetype) > +{ > + unsigned int ft_flags_other = freetype_flags(freetype) ^ FREETYPE_UNMAPPED; > + freetype_t ft_other = migrate_to_freetype(free_to_migratetype(freetype), > + ft_flags_other); > + bool want_mapped = !(freetype_flags(freetype) & FREETYPE_UNMAPPED); > + enum rmqueue_mode rmqm = RMQUEUE_NORMAL; > + unsigned long irq_flags; > + int nr_pageblocks, nr_freed; > + struct page *page; > + int alloc_order; > + int err; > + > + if (freetype_idx(ft_other) < 0) > + return NULL; > + > + /* > + * Might need a TLB shootdown. Even if IRQs are on this isn't > + * safe if the caller holds a lock (in case the other CPUs need that > + * lock to handle the shootdown IPI). > + */ > + if (alloc_flags & ALLOC_NOBLOCK) > + return NULL; Should we only check this if !want_mapped? IIUC we only need a TLB shootdown when unmapping. > + > + if (!can_set_direct_map() || alloc_flags & ALLOC_NOLOCK) > + return NULL; > + > + lockdep_assert(!irqs_disabled() || unlikely(early_boot_irqs_disabled)); > + > + /* > + * Need to [un]map a whole pageblock (otherwise it might require > + * allocating pagetables). First allocate it. > + */ > + alloc_order = max(request_order, pageblock_order); > + nr_pageblocks = 1 << (alloc_order - pageblock_order); > + spin_lock_irqsave(&zone->lock, irq_flags); > + /* First try a block that already has the right migratetype. */ > + page = __rmqueue(zone, alloc_order, ft_other, alloc_flags, &rmqm); IIUC, this is called after __rmqueue() will have already failed in the caller with request_order (a potentially smaller order), so why are we trying this again here? > + if (!page) { > + /* Fallback to changing a block's migratetype. */ > + rmqm = RMQUEUE_CLAIM; > + page = __rmqueue(zone, alloc_order, ft_other, alloc_flags, &rmqm); > + } > + spin_unlock_irqrestore(&zone->lock, irq_flags); > + if (!page) > + return NULL; > + > + /* > + * Now that IRQs are on it's safe to do a TLB shootdown, and now that we > + * released the zone lock it's possible to allocate a pagetable if > + * needed to split up a huge page. > + * > + * Note that modifying the direct map may need to allocate pagetables. > + * What about unbounded recursion? Here are the assumptions that make it > + * safe: > + * > + * - The direct map starts out fully mapped at boot. (This is not really > + * an "assumption" as it's in direct control of page_alloc.c). > + * > + * - Once pages in the direct map are broken down, they are not > + * re-aggregated into larger pages again. > + * > + * - Pagetables are never allocated with ALLOC_UNMAPPED. > + * > + * Under these assumptions, a pagetable might need to be allocated while > + * _unmapping_ stuff from the direct map during an ALLOC_UNMAPPED > + * allocation. But, the allocation of that pagetable never requires > + * allocating a further pagetable. > + */ > + err = set_direct_map_valid_noflush(page, > + nr_pageblocks << pageblock_order, want_mapped); > + if (err == -ENOMEM || WARN_ONCE(err, "err=%d\n", err)) { > + set_direct_map_valid_noflush(page, > + nr_pageblocks << pageblock_order, !want_mapped); > + spin_lock_irqsave(&zone->lock, irq_flags); > + /* Important: free using _old_ freetype. */ > + __free_one_page(page, page_to_pfn(page), zone, > + alloc_order, ft_other, FPI_SKIP_REPORT_NOTIFY); > + spin_unlock_irqrestore(&zone->lock, irq_flags); > + return NULL; > + } > + > + if (want_mapped) { > + /* Exposing formerly-protected data; scrub it. */ > + clear_highpages_kasan_tagged(page, nr_pageblocks << pageblock_order); Shouldn't all unmapped memory be zeroed on free? If we solidify this assumption we can probably drop this here (and maybe replace it with an assertion)? > + } else { > + unsigned long start = (unsigned long)page_address(page); > + unsigned long end = start + (nr_pageblocks << (pageblock_order + PAGE_SHIFT)); > + > + flush_tlb_kernel_range(start, end); > + } > + > + for (int i = 0; i < nr_pageblocks; i++) { > + struct page *block_page = page + (pageblock_nr_pages * i); > + > + set_pageblock_freetype_flags(block_page, freetype_flags(freetype)); > + } > + > + if (request_order >= alloc_order) > + return page; > + > + /* Free any remaining pages in the block. */ > + spin_lock_irqsave(&zone->lock, irq_flags); > + nr_freed = expand(zone, page, request_order, alloc_order, freetype); > + account_freepages(zone, nr_freed, free_to_migratetype(freetype)); > + spin_unlock_irqrestore(&zone->lock, irq_flags); > + > + return page; > +} > +#else /* CONFIG_PAGE_ALLOC_UNMAPPED */ > +static inline struct page *__rmqueue_direct_map(struct zone *zone, unsigned int request_order, > + unsigned int alloc_flags, freetype_t freetype) > +{ > + return NULL; > +} > +#endif /* CONFIG_PAGE_ALLOC_UNMAPPED */ > + > static __always_inline > struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, > unsigned int order, unsigned int alloc_flags, > @@ -3433,13 +3580,15 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, > */ > if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_HARDER))) > page = __rmqueue_smallest(zone, order, ft_high); > - > - if (!page) { > - spin_unlock_irqrestore(&zone->lock, flags); > - return NULL; > - } > } > spin_unlock_irqrestore(&zone->lock, flags); > + > + /* Try changing direct map, now we've released the zone lock */ > + if (!page) > + page = __rmqueue_direct_map(zone, order, alloc_flags, freetype); > + if (!page) > + return NULL; > + > } while (check_new_pages(page, order)); > > /* > @@ -3660,6 +3809,8 @@ static void reserve_highatomic_pageblock(struct page *page, int order, > return; > > ft_high = freetype_with_migrate(ft, MIGRATE_HIGHATOMIC); > + if (freetype_idx(ft_high) < 0) > + return; Does this belong in "mm/page_alloc: add support for freetypes with no freelist"? > if (order < pageblock_order) { > if (move_freepages_block(zone, page, ft, ft_high) == -1) > return; > @@ -3975,13 +4126,15 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) > } > > /* Must be called after current_gfp_context() which can change gfp_mask */ > -static inline unsigned int alloc_flags_cma(gfp_t gfp_mask) > +static inline unsigned int alloc_flags_cma(gfp_t gfp_mask, unsigned int alloc_flags) > { > #ifdef CONFIG_CMA > - if (free_to_migratetype(gfp_freetype(gfp_mask)) == MIGRATE_MOVABLE) > - return ALLOC_CMA; > + if (free_to_migratetype(gfp_freetype(gfp_mask, alloc_flags)) == MIGRATE_MOVABLE) > + alloc_flags |= ALLOC_CMA; > #endif > - return ALLOC_DEFAULT; > + alloc_flags |= ALLOC_DEFAULT; > + > + return alloc_flags; > } > > /* > @@ -4770,7 +4923,7 @@ alloc_flags_slowpath(gfp_t gfp_mask, unsigned int order) > } else if (unlikely(rt_or_dl_task(current)) && in_task()) > alloc_flags |= ALLOC_MIN_RESERVE; > > - alloc_flags |= alloc_flags_cma(gfp_mask); > + alloc_flags = alloc_flags_cma(gfp_mask, alloc_flags); > > if (defrag_mode) > alloc_flags |= ALLOC_NOFRAGMENT; > @@ -5085,7 +5238,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > > reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); > if (reserve_flags) > - alloc_flags = alloc_flags_cma(gfp_mask) | reserve_flags | > + alloc_flags = alloc_flags_cma(gfp_mask, alloc_flags) | reserve_flags | > ac->alloc_flags | (alloc_flags & ALLOC_KSWAPD); Should we pass in ac->alloc_flags here to maintain equivalent functionality, maybe this: alloc_flags = alloc_flags_cma(gfp_mask, ac->alloc_flags) | reserve_flags | (alloc_flags & ALLOC_KSWAPD); > > /* > @@ -5307,7 +5460,11 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, > ac->highest_zoneidx = gfp_zone(gfp_mask); > ac->zonelist = node_zonelist(preferred_nid, gfp_mask); > ac->nodemask = nodemask; > - ac->freetype = gfp_freetype(gfp_mask); > + ac->freetype = gfp_freetype(gfp_mask, *alloc_flags); > + > + /* Not implemented yet. */ > + if (freetype_flags(ac->freetype) & FREETYPE_UNMAPPED && gfp_mask & __GFP_ZERO) Nit: Add more parentheses for readability? if ((freetype_flags(ac->freetype) & FREETYPE_UNMAPPED) && (gfp_mask & __GFP_ZERO)) > + return false; > > if (cpusets_enabled()) { > *alloc_gfp |= __GFP_HARDWALL;