Re: [PATCH v3 21/26] mm/page_alloc: implement FREETYPE_UNMAPPED allocations

[Yosry Ahmed <yosry@kernel.org>]

On Sun, Jul 26, 2026 at 10:22:54PM +0000, Brendan Jackman wrote:
> Currently FREETYPE_UNMAPPED allocs will always fail because, although the
> lists exist to hold them, there is no way to actually create an unmapped
> page block. This commit adds one, and also the logic to map it back
> again when that's needed.
> 
> Doing this at pageblock granularity ensures that the pageblock flags can
> be used to infer which freetype a page belongs to. It also provides nice
> batching of TLB flushes, and also avoids creating too much unnecessary
> TLB fragmentation in the physmap.
> 
> There are some functional requirements for flipping a block:
> 
>  - Unmapping requires a TLB shootdown, meaning IRQs must be enabled.
> 
>  - Updating the pagetables might require allocating a pagetable to break
>    down a huge page. This would deadlock if the zone lock was held.

We also need to zero unmapped/sensitive pages before mapping them again,
but seems like the current approach is to differ this to the caller,
which makes sense. The only annoying part is that
want_init_on_{free/alloc}() now silently skip the zeroing for those
pages.

We should document somewhere that the users of ALLOC_UNMAPPED are
responsible for zeroing memory before freeing it?

The current users currently always zero the pages on allocation as well,
I am not sure if this should also be a general requirement, or perhaps
only if want_init_on_alloc() is set?

> 
> This makes allocations that need to change sensitivity _somewhat_

s/sensitivity/direct mapping status (or sth)?

> similar to those that need to fallback to a different migratetype. But,
> the locking requirements mean that this can't just be squashed into the
> existing "fallback" allocator logic, instead a new allocator path just
> for this purpose is needed.
> 
> The new path is assumed to be much cheaper than the really heavyweight
> stuff like compaction and reclaim. But at present it is treated as less
> desirable than the mobility-related "fallback" and "stealing" logic.
> This might turn out to need revision (in particular, maybe it's a
> problem that __rmqueue_steal(), which causes fragmentation, happens
> before __rmqueue_direct_map()), but that should be treated as a subsequent
> optimisation project.
> 
> Adding alloc_flags to gfp_freetype() requires moving it to
> mm/page_alloc.h so it can refer to ALLOC_UNMAPPED. It was already only
> used in internal mm code.
> 
> Now that unmapped pageblocks actually exist, exclude them from
> migration. Migrating unmapped pages via the mermap should be possible
> but that's something to be added later when needed.
> 
> Signed-off-by: Brendan Jackman <jackmanb@google.com>
[..]
> @@ -3400,6 +3426,127 @@ static inline void zone_statistics(struct zone *preferred_zone, struct zone *z,
>  #endif
>  }
>  
> +#ifdef CONFIG_PAGE_ALLOC_UNMAPPED
> +/* Try to allocate a page by mapping/unmapping a block from the direct map. */
> +static inline struct page *
> +__rmqueue_direct_map(struct zone *zone, unsigned int request_order,
> +		     unsigned int alloc_flags, freetype_t freetype)
> +{
> +	unsigned int ft_flags_other = freetype_flags(freetype) ^ FREETYPE_UNMAPPED;
> +	freetype_t ft_other = migrate_to_freetype(free_to_migratetype(freetype),
> +						  ft_flags_other);
> +	bool want_mapped = !(freetype_flags(freetype) & FREETYPE_UNMAPPED);
> +	enum rmqueue_mode rmqm = RMQUEUE_NORMAL;
> +	unsigned long irq_flags;
> +	int nr_pageblocks, nr_freed;
> +	struct page *page;
> +	int alloc_order;
> +	int err;
> +
> +	if (freetype_idx(ft_other) < 0)
> +		return NULL;
> +
> +	/*
> +	 * Might need a TLB shootdown. Even if IRQs are on this isn't
> +	 * safe if the caller holds a lock (in case the other CPUs need that
> +	 * lock to handle the shootdown IPI).
> +	 */
> +	if (alloc_flags & ALLOC_NOBLOCK)
> +		return NULL;

Should we only check this if !want_mapped? IIUC we only need a TLB
shootdown when unmapping.

> +
> +	if (!can_set_direct_map() || alloc_flags & ALLOC_NOLOCK)
> +		return NULL;
> +
> +	lockdep_assert(!irqs_disabled() || unlikely(early_boot_irqs_disabled));
> +
> +	/*
> +	 * Need to [un]map a whole pageblock (otherwise it might require
> +	 * allocating pagetables). First allocate it.
> +	 */
> +	alloc_order = max(request_order, pageblock_order);
> +	nr_pageblocks = 1 << (alloc_order - pageblock_order);
> +	spin_lock_irqsave(&zone->lock, irq_flags);
> +	/* First try a block that already has the right migratetype. */
> +	page = __rmqueue(zone, alloc_order, ft_other, alloc_flags, &rmqm);

IIUC, this is called after __rmqueue() will have already failed in the
caller with request_order (a potentially smaller order), so why are we
trying this again here?

> +	if (!page) {
> +		/* Fallback to changing a block's migratetype. */
> +		rmqm = RMQUEUE_CLAIM;
> +		page = __rmqueue(zone, alloc_order, ft_other, alloc_flags, &rmqm);
> +	}
> +	spin_unlock_irqrestore(&zone->lock, irq_flags);
> +	if (!page)
> +		return NULL;
> +
> +	/*
> +	 * Now that IRQs are on it's safe to do a TLB shootdown, and now that we
> +	 * released the zone lock it's possible to allocate a pagetable if
> +	 * needed to split up a huge page.
> +	 *
> +	 * Note that modifying the direct map may need to allocate pagetables.
> +	 * What about unbounded recursion? Here are the assumptions that make it
> +	 * safe:
> +	 *
> +	 * - The direct map starts out fully mapped at boot. (This is not really
> +	 *   an "assumption" as it's in direct control of page_alloc.c).
> +	 *
> +	 * - Once pages in the direct map are broken down, they are not
> +	 *   re-aggregated into larger pages again.
> +	 *
> +	 * - Pagetables are never allocated with ALLOC_UNMAPPED.
> +	 *
> +	 * Under these assumptions, a pagetable might need to be allocated while
> +	 * _unmapping_ stuff from the direct map during an ALLOC_UNMAPPED
> +	 * allocation. But, the allocation of that pagetable never requires
> +	 * allocating a further pagetable.
> +	 */
> +	err = set_direct_map_valid_noflush(page,
> +				nr_pageblocks << pageblock_order, want_mapped);
> +	if (err == -ENOMEM || WARN_ONCE(err, "err=%d\n", err)) {
> +		set_direct_map_valid_noflush(page,
> +				nr_pageblocks << pageblock_order, !want_mapped);
> +		spin_lock_irqsave(&zone->lock, irq_flags);
> +		/* Important: free using _old_ freetype. */
> +		__free_one_page(page, page_to_pfn(page), zone,
> +				alloc_order, ft_other, FPI_SKIP_REPORT_NOTIFY);
> +		spin_unlock_irqrestore(&zone->lock, irq_flags);
> +		return NULL;
> +	}
> +
> +	if (want_mapped) {
> +		/* Exposing formerly-protected data; scrub it. */
> +		clear_highpages_kasan_tagged(page, nr_pageblocks << pageblock_order);

Shouldn't all unmapped memory be zeroed on free? If we solidify this
assumption we can probably drop this here (and maybe replace it with an
assertion)?

> +	} else {
> +		unsigned long start = (unsigned long)page_address(page);
> +		unsigned long end = start + (nr_pageblocks << (pageblock_order + PAGE_SHIFT));
> +
> +		flush_tlb_kernel_range(start, end);
> +	}
> +
> +	for (int i = 0; i < nr_pageblocks; i++) {
> +		struct page *block_page = page + (pageblock_nr_pages * i);
> +
> +		set_pageblock_freetype_flags(block_page, freetype_flags(freetype));
> +	}
> +
> +	if (request_order >= alloc_order)
> +		return page;
> +
> +	/* Free any remaining pages in the block. */
> +	spin_lock_irqsave(&zone->lock, irq_flags);
> +	nr_freed = expand(zone, page, request_order, alloc_order, freetype);
> +	account_freepages(zone, nr_freed, free_to_migratetype(freetype));
> +	spin_unlock_irqrestore(&zone->lock, irq_flags);
> +
> +	return page;
> +}
> +#else /* CONFIG_PAGE_ALLOC_UNMAPPED */
> +static inline struct page *__rmqueue_direct_map(struct zone *zone, unsigned int request_order,
> +				unsigned int alloc_flags, freetype_t freetype)
> +{
> +	return NULL;
> +}
> +#endif /* CONFIG_PAGE_ALLOC_UNMAPPED */
> +
>  static __always_inline
>  struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>  			   unsigned int order, unsigned int alloc_flags,
> @@ -3433,13 +3580,15 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone,
>  			 */
>  			if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_HARDER)))
>  				page = __rmqueue_smallest(zone, order, ft_high);
> -
> -			if (!page) {
> -				spin_unlock_irqrestore(&zone->lock, flags);
> -				return NULL;
> -			}
>  		}
>  		spin_unlock_irqrestore(&zone->lock, flags);
> +
> +		/* Try changing direct map, now we've released the zone lock */
> +		if (!page)
> +			page = __rmqueue_direct_map(zone, order, alloc_flags, freetype);
> +		if (!page)
> +			return NULL;
> +
>  	} while (check_new_pages(page, order));
>  
>  	/*
> @@ -3660,6 +3809,8 @@ static void reserve_highatomic_pageblock(struct page *page, int order,
>  		return;
>  
>  	ft_high = freetype_with_migrate(ft, MIGRATE_HIGHATOMIC);
> +	if (freetype_idx(ft_high) < 0)
> +		return;

Does this belong in "mm/page_alloc: add support for freetypes with no
freelist"?

>  	if (order < pageblock_order) {
>  		if (move_freepages_block(zone, page, ft, ft_high) == -1)
>  			return;
> @@ -3975,13 +4126,15 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
>  }
>  
>  /* Must be called after current_gfp_context() which can change gfp_mask */
> -static inline unsigned int alloc_flags_cma(gfp_t gfp_mask)
> +static inline unsigned int alloc_flags_cma(gfp_t gfp_mask, unsigned int alloc_flags)
>  {
>  #ifdef CONFIG_CMA
> -	if (free_to_migratetype(gfp_freetype(gfp_mask)) == MIGRATE_MOVABLE)
> -		return ALLOC_CMA;
> +	if (free_to_migratetype(gfp_freetype(gfp_mask, alloc_flags)) == MIGRATE_MOVABLE)
> +		alloc_flags |= ALLOC_CMA;
>  #endif
> -	return ALLOC_DEFAULT;
> +	alloc_flags |= ALLOC_DEFAULT;
> +
> +	return alloc_flags;
>  }
>  
>  /*
> @@ -4770,7 +4923,7 @@ alloc_flags_slowpath(gfp_t gfp_mask, unsigned int order)
>  	} else if (unlikely(rt_or_dl_task(current)) && in_task())
>  		alloc_flags |= ALLOC_MIN_RESERVE;
>  
> -	alloc_flags |= alloc_flags_cma(gfp_mask);
> +	alloc_flags = alloc_flags_cma(gfp_mask, alloc_flags);
>  
>  	if (defrag_mode)
>  		alloc_flags |= ALLOC_NOFRAGMENT;
> @@ -5085,7 +5238,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  
>  	reserve_flags = __gfp_pfmemalloc_flags(gfp_mask);
>  	if (reserve_flags)
> -		alloc_flags = alloc_flags_cma(gfp_mask) | reserve_flags |
> +		alloc_flags = alloc_flags_cma(gfp_mask, alloc_flags) | reserve_flags |
>  				ac->alloc_flags | (alloc_flags & ALLOC_KSWAPD);

Should we pass in ac->alloc_flags here to maintain equivalent
functionality, maybe this:

		alloc_flags = alloc_flags_cma(gfp_mask, ac->alloc_flags) |
				reserve_flags | (alloc_flags & ALLOC_KSWAPD);

>  
>  	/*
> @@ -5307,7 +5460,11 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
>  	ac->highest_zoneidx = gfp_zone(gfp_mask);
>  	ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
>  	ac->nodemask = nodemask;
> -	ac->freetype = gfp_freetype(gfp_mask);
> +	ac->freetype = gfp_freetype(gfp_mask, *alloc_flags);
> +
> +	/* Not implemented yet. */
> +	if (freetype_flags(ac->freetype) & FREETYPE_UNMAPPED && gfp_mask & __GFP_ZERO)

Nit: Add more parentheses for readability?
	
	if ((freetype_flags(ac->freetype) & FREETYPE_UNMAPPED) && (gfp_mask & __GFP_ZERO))

> +		return false;
>  
>  	if (cpusets_enabled()) {
>  		*alloc_gfp |= __GFP_HARDWALL;