Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting

[Ryan Roberts <ryan.roberts@arm.com>]

On 30/10/2023 08:18, Huang, Ying wrote:
> Hi, Ryan,
> 
> Ryan Roberts <ryan.roberts@arm.com> writes:
> 
>> The upcoming anonymous small-sized THP feature enables performance
>> improvements by allocating large folios for anonymous memory. However
>> I've observed that on an arm64 system running a parallel workload (e.g.
>> kernel compilation) across many cores, under high memory pressure, the
>> speed regresses. This is due to bottlenecking on the increased number of
>> TLBIs added due to all the extra folio splitting.
>>
>> Therefore, solve this regression by adding support for swapping out
>> small-sized THP without needing to split the folio, just like is already
>> done for PMD-sized THP. This change only applies when CONFIG_THP_SWAP is
>> enabled, and when the swap backing store is a non-rotating block device.
>> These are the same constraints as for the existing PMD-sized THP
>> swap-out support.
>>
>> Note that no attempt is made to swap-in THP here - this is still done
>> page-by-page, like for PMD-sized THP.
>>
>> The main change here is to improve the swap entry allocator so that it
>> can allocate any power-of-2 number of contiguous entries between [1, (1
>> << PMD_ORDER)]. This is done by allocating a cluster for each distinct
>> order and allocating sequentially from it until the cluster is full.
>> This ensures that we don't need to search the map and we get no
>> fragmentation due to alignment padding for different orders in the
>> cluster. If there is no current cluster for a given order, we attempt to
>> allocate a free cluster from the list. If there are no free clusters, we
>> fail the allocation and the caller falls back to splitting the folio and
>> allocates individual entries (as per existing PMD-sized THP fallback).
>>
>> The per-order current clusters are maintained per-cpu using the existing
>> infrastructure. This is done to avoid interleving pages from different
>> tasks, which would prevent IO being batched. This is already done for
>> the order-0 allocations so we follow the same pattern.
>> __scan_swap_map_try_ssd_cluster() is introduced to deal with arbitrary
>> orders and scan_swap_map_try_ssd_cluster() is refactored as a wrapper
>> for order-0.
>>
>> As is done for order-0 per-cpu clusters, the scanner now can steal
>> order-0 entries from any per-cpu-per-order reserved cluster. This
>> ensures that when the swap file is getting full, space doesn't get tied
>> up in the per-cpu reserves.
>>
>> I've run the tests on Ampere Altra (arm64), set up with a 35G block ram
>> device as the swap device and from inside a memcg limited to 40G memory.
>> I've then run `usemem` from vm-scalability with 70 processes (each has
>> its own core), each allocating and writing 1G of memory. I've repeated
>> everything 5 times and taken the mean:
>>
>> Mean Performance Improvement vs 4K/baseline
>>
>> | alloc size |            baseline |       + this series |
>> |            |  v6.6-rc4+anonfolio |                     |
>> |:-----------|--------------------:|--------------------:|
>> | 4K Page    |                0.0% |                4.9% |
>> | 64K THP    |              -44.1% |               10.7% |
>> | 2M THP     |               56.0% |               65.9% |
>>
>> So with this change, the regression for 64K swap performance goes away
>> and 4K and 2M swap improves slightly too.
>>
>> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
>> ---
>>  include/linux/swap.h |  10 +--
>>  mm/swapfile.c        | 149 +++++++++++++++++++++++++++++++------------
>>  mm/vmscan.c          |  10 +--
>>  3 files changed, 119 insertions(+), 50 deletions(-)
>>
>> diff --git a/include/linux/swap.h b/include/linux/swap.h
>> index 0ca8aaa098ba..ccbca5db851b 100644
>> --- a/include/linux/swap.h
>> +++ b/include/linux/swap.h
>> @@ -295,11 +295,11 @@ struct swap_info_struct {
>>  	unsigned int __percpu *cluster_next_cpu; /*percpu index for next allocation */
>>  	unsigned int __percpu *cpu_next;/*
>>  					 * Likely next allocation offset. We
>> -					 * assign a cluster to each CPU, so each
>> -					 * CPU can allocate swap entry from its
>> -					 * own cluster and swapout sequentially.
>> -					 * The purpose is to optimize swapout
>> -					 * throughput.
>> +					 * assign a cluster per-order to each
>> +					 * CPU, so each CPU can allocate swap
>> +					 * entry from its own cluster and
>> +					 * swapout sequentially. The purpose is
>> +					 * to optimize swapout throughput.
>>  					 */
> 
> This is kind of hard to understand.  Better to define some intermediate
> data structure to improve readability.  For example,
> 
> #ifdef CONFIG_THP_SWAP
> #define NR_SWAP_ORDER   PMD_ORDER
> #else
> #define NR_SWAP_ORDER   1
> #endif
> 
> struct percpu_clusters {
>         unsigned int alloc_next[NR_SWAP_ORDER];
> };
> 
> PMD_ORDER isn't a constant on powerpc, but THP_SWAP isn't supported on
> powerpc too.

I get your point, but this is just making it more difficult for powerpc to ever
enable the feature in future - you're implicitly depending on !powerpc, which
seems fragile. How about if I change the first line of the coment to be "per-cpu
array indexed by allocation order"? Would that be enough?

> 
>>  	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
>>  	struct block_device *bdev;	/* swap device or bdev of swap file */
>> diff --git a/mm/swapfile.c b/mm/swapfile.c
>> index 94f7cc225eb9..b50bce50bed9 100644
>> --- a/mm/swapfile.c
>> +++ b/mm/swapfile.c
>> @@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)
>>  
>>  /*
>>   * The cluster corresponding to page_nr will be used. The cluster will be
>> - * removed from free cluster list and its usage counter will be increased.
>> + * removed from free cluster list and its usage counter will be increased by
>> + * count.
>>   */
>> -static void inc_cluster_info_page(struct swap_info_struct *p,
>> -	struct swap_cluster_info *cluster_info, unsigned long page_nr)
>> +static void add_cluster_info_page(struct swap_info_struct *p,
>> +	struct swap_cluster_info *cluster_info, unsigned long page_nr,
>> +	unsigned long count)
>>  {
>>  	unsigned long idx = page_nr / SWAPFILE_CLUSTER;
>>  
>> @@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,
>>  	if (cluster_is_free(&cluster_info[idx]))
>>  		alloc_cluster(p, idx);
>>  
>> -	VM_BUG_ON(cluster_count(&cluster_info[idx]) >= SWAPFILE_CLUSTER);
>> +	VM_BUG_ON(cluster_count(&cluster_info[idx]) + count > SWAPFILE_CLUSTER);
>>  	cluster_set_count(&cluster_info[idx],
>> -		cluster_count(&cluster_info[idx]) + 1);
>> +		cluster_count(&cluster_info[idx]) + count);
>> +}
>> +
>> +/*
>> + * The cluster corresponding to page_nr will be used. The cluster will be
>> + * removed from free cluster list and its usage counter will be increased.
>> + */
>> +static void inc_cluster_info_page(struct swap_info_struct *p,
>> +	struct swap_cluster_info *cluster_info, unsigned long page_nr)
>> +{
>> +	add_cluster_info_page(p, cluster_info, page_nr, 1);
>>  }
>>  
>>  /*
>> @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,
>>   * cluster list. Avoiding such abuse to avoid list corruption.
>>   */
>>  static bool
>> -scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,
>> -	unsigned long offset)
>> +__scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,
>> +	unsigned long offset, int order)
>>  {
>>  	bool conflict;
>>  
>> @@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,
>>  	if (!conflict)
>>  		return false;
>>  
>> -	*this_cpu_ptr(si->cpu_next) = SWAP_NEXT_NULL;
>> +	this_cpu_ptr(si->cpu_next)[order] = SWAP_NEXT_NULL;
> 
> This is added in the previous patch.  I don't think SWAP_NEXT_NULL is a
> good name.  Because NEXT isn't a pointer (while cluster_next is). Better
> to name it as SWAP_NEXT_INVALID, etc.

ACK, will make change for next version.

> 
>>  	return true;
>>  }
>>  
>>  /*
>> - * Try to get a swap entry from current cpu's swap entry pool (a cluster). This
>> - * might involve allocating a new cluster for current CPU too.
>> + * It's possible scan_swap_map_slots() uses a free cluster in the middle of free
>> + * cluster list. Avoiding such abuse to avoid list corruption.
>>   */
>> -static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,
>> -	unsigned long *offset, unsigned long *scan_base)
>> +static bool
>> +scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,
>> +	unsigned long offset)
>> +{
>> +	return __scan_swap_map_ssd_cluster_conflict(si, offset, 0);
>> +}
>> +
>> +/*
>> + * Try to get a swap entry (or size indicated by order) from current cpu's swap
>> + * entry pool (a cluster). This might involve allocating a new cluster for
>> + * current CPU too.
>> + */
>> +static bool __scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,
>> +	unsigned long *offset, unsigned long *scan_base, int order)
>>  {
>>  	struct swap_cluster_info *ci;
>> -	unsigned int tmp, max;
>> +	unsigned int tmp, max, i;
>>  	unsigned int *cpu_next;
>> +	unsigned int nr_pages = 1 << order;
>>  
>>  new_cluster:
>> -	cpu_next = this_cpu_ptr(si->cpu_next);
>> +	cpu_next = &this_cpu_ptr(si->cpu_next)[order];
>>  	tmp = *cpu_next;
>>  	if (tmp == SWAP_NEXT_NULL) {
>>  		if (!cluster_list_empty(&si->free_clusters)) {
>> @@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,
>>  	 * reserve a new cluster.
>>  	 */
>>  	ci = lock_cluster(si, tmp);
>> -	if (si->swap_map[tmp]) {
>> -		unlock_cluster(ci);
>> -		*cpu_next = SWAP_NEXT_NULL;
>> -		goto new_cluster;
>> +	for (i = 0; i < nr_pages; i++) {
>> +		if (si->swap_map[tmp + i]) {
>> +			unlock_cluster(ci);
>> +			*cpu_next = SWAP_NEXT_NULL;
>> +			goto new_cluster;
>> +		}
>>  	}
>>  	unlock_cluster(ci);
>>  
>> @@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,
>>  	*scan_base = tmp;
>>  
>>  	max = ALIGN_DOWN(tmp, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER;
> 
> This line is added in a previous patch.  Can we just use
> 
>         max = ALIGN(tmp + 1, SWAPFILE_CLUSTER);

Sure. This is how I originally had it, but then decided that the other approach
was a bit clearer. But I don't have a strong opinion, so I'll change it as you
suggest.

> 
> Or, add ALIGN_UP() for this?
> 
>> -	tmp += 1;
>> +	tmp += nr_pages;
>>  	*cpu_next = tmp < max ? tmp : SWAP_NEXT_NULL;
>>  
>>  	return true;
>>  }
>>  
>> +/*
>> + * Try to get a swap entry from current cpu's swap entry pool (a cluster). This
>> + * might involve allocating a new cluster for current CPU too.
>> + */
>> +static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,
>> +	unsigned long *offset, unsigned long *scan_base)
>> +{
>> +	return __scan_swap_map_try_ssd_cluster(si, offset, scan_base, 0);
>> +}
>> +
>>  static void __del_from_avail_list(struct swap_info_struct *p)
>>  {
>>  	int nid;
>> @@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
>>  	return n_ret;
>>  }
>>  
>> -static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
>> +static int swap_alloc_large(struct swap_info_struct *si, swp_entry_t *slot,
>> +			    unsigned int nr_pages)
> 
> IMHO, it's better to make scan_swap_map_slots() to support order > 0
> instead of making swap_alloc_cluster() to support order != PMD_ORDER.
> And, we may merge swap_alloc_cluster() with scan_swap_map_slots() after
> that.

I did consider adding a 5th patch to rename swap_alloc_large() to something like
swap_alloc_one_ssd_entry() (which would then be used for order=0 too) and
refactor scan_swap_map_slots() to fully delegate to it for the non-scaning ssd
allocation case. Would something like that suit?

I have reservations about making scan_swap_map_slots() take an order and be the
sole entry point:

  - in the non-ssd case, we can't support order!=0
  - there is a lot of other logic to deal with falling back to scanning which we
    would only want to do for order==0, so we would end up with a few ugly
    conditionals against order.
  - I was concerned the risk of me introducing a bug when refactoring all that
    subtle logic was high

What do you think? Is not making scan_swap_map_slots() support order > 0 a deal
breaker for you?

Thanks,
Ryan


> 
>>  {
>> -	unsigned long idx;
>>  	struct swap_cluster_info *ci;
>> -	unsigned long offset;
>> +	unsigned long offset, scan_base;
>> +	int order = ilog2(nr_pages);
>> +	bool ret;
>>  
>>  	/*
>> -	 * Should not even be attempting cluster allocations when huge
>> +	 * Should not even be attempting large allocations when huge
>>  	 * page swap is disabled.  Warn and fail the allocation.
>>  	 */
>> -	if (!IS_ENABLED(CONFIG_THP_SWAP)) {
>> +	if (!IS_ENABLED(CONFIG_THP_SWAP) ||
>> +	    nr_pages < 2 || nr_pages > SWAPFILE_CLUSTER ||
>> +	    !is_power_of_2(nr_pages)) {
>>  		VM_WARN_ON_ONCE(1);
>>  		return 0;
>>  	}
>>  
>> -	if (cluster_list_empty(&si->free_clusters))
>> +	/*
>> +	 * Swapfile is not block device or not using clusters so unable to
>> +	 * allocate large entries.
>> +	 */
>> +	if (!(si->flags & SWP_BLKDEV) || !si->cluster_info)
>>  		return 0;
>>  
>> -	idx = cluster_list_first(&si->free_clusters);
>> -	offset = idx * SWAPFILE_CLUSTER;
>> -	ci = lock_cluster(si, offset);
>> -	alloc_cluster(si, idx);
>> -	cluster_set_count(ci, SWAPFILE_CLUSTER);
>> +again:
>> +	/*
>> +	 * __scan_swap_map_try_ssd_cluster() may drop si->lock during discard,
>> +	 * so indicate that we are scanning to synchronise with swapoff.
>> +	 */
>> +	si->flags += SWP_SCANNING;
>> +	ret = __scan_swap_map_try_ssd_cluster(si, &offset, &scan_base, order);
>> +	si->flags -= SWP_SCANNING;
>> +
>> +	/*
>> +	 * If we failed to allocate or if swapoff is waiting for us (due to lock
>> +	 * being dropped for discard above), return immediately.
>> +	 */
>> +	if (!ret || !(si->flags & SWP_WRITEOK))
>> +		return 0;
>>  
>> -	memset(si->swap_map + offset, SWAP_HAS_CACHE, SWAPFILE_CLUSTER);
>> +	if (__scan_swap_map_ssd_cluster_conflict(si, offset, order))
>> +		goto again;
>> +
>> +	ci = lock_cluster(si, offset);
>> +	memset(si->swap_map + offset, SWAP_HAS_CACHE, nr_pages);
>> +	add_cluster_info_page(si, si->cluster_info, offset, nr_pages);
>>  	unlock_cluster(ci);
>> -	swap_range_alloc(si, offset, SWAPFILE_CLUSTER);
>> -	*slot = swp_entry(si->type, offset);
>>  
>> +	swap_range_alloc(si, offset, nr_pages);
>> +	*slot = swp_entry(si->type, offset);
>>  	return 1;
>>  }
>>  
>> @@ -1036,7 +1096,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
>>  	int node;
>>  
>>  	/* Only single cluster request supported */
>> -	WARN_ON_ONCE(n_goal > 1 && size == SWAPFILE_CLUSTER);
>> +	WARN_ON_ONCE(n_goal > 1 && size > 1);
>>  
>>  	spin_lock(&swap_avail_lock);
>>  
>> @@ -1073,14 +1133,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
>>  			spin_unlock(&si->lock);
>>  			goto nextsi;
>>  		}
>> -		if (size == SWAPFILE_CLUSTER) {
>> -			if (si->flags & SWP_BLKDEV)
>> -				n_ret = swap_alloc_cluster(si, swp_entries);
>> +		if (size > 1) {
>> +			n_ret = swap_alloc_large(si, swp_entries, size);
>>  		} else
>>  			n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE,
>>  						    n_goal, swp_entries);
>>  		spin_unlock(&si->lock);
>> -		if (n_ret || size == SWAPFILE_CLUSTER)
>> +		if (n_ret || size > 1)
>>  			goto check_out;
>>  		cond_resched();
>>  
>> @@ -3041,6 +3100,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
>>  	if (p->bdev && bdev_nonrot(p->bdev)) {
>>  		int cpu;
>>  		unsigned long ci, nr_cluster;
>> +		int nr_order;
>> +		int i;
>>  
>>  		p->flags |= SWP_SOLIDSTATE;
>>  		p->cluster_next_cpu = alloc_percpu(unsigned int);
>> @@ -3068,13 +3129,19 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
>>  		for (ci = 0; ci < nr_cluster; ci++)
>>  			spin_lock_init(&((cluster_info + ci)->lock));
>>  
>> -		p->cpu_next = alloc_percpu(unsigned int);
>> +		nr_order = IS_ENABLED(CONFIG_THP_SWAP) ? PMD_ORDER + 1 : 1;
>> +		p->cpu_next = __alloc_percpu(sizeof(unsigned int) * nr_order,
>> +					     __alignof__(unsigned int));
>>  		if (!p->cpu_next) {
>>  			error = -ENOMEM;
>>  			goto bad_swap_unlock_inode;
>>  		}
>> -		for_each_possible_cpu(cpu)
>> -			per_cpu(*p->cpu_next, cpu) = SWAP_NEXT_NULL;
>> +		for_each_possible_cpu(cpu) {
>> +			unsigned int *cpu_next = per_cpu_ptr(p->cpu_next, cpu);
>> +
>> +			for (i = 0; i < nr_order; i++)
>> +				cpu_next[i] = SWAP_NEXT_NULL;
>> +		}
>>  	} else {
>>  		atomic_inc(&nr_rotate_swap);
>>  		inced_nr_rotate_swap = true;
>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index 2cc0cb41fb32..ea19710aa4cd 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
>>  					if (!can_split_folio(folio, NULL))
>>  						goto activate_locked;
>>  					/*
>> -					 * Split folios without a PMD map right
>> -					 * away. Chances are some or all of the
>> -					 * tail pages can be freed without IO.
>> +					 * Split PMD-mappable folios without a
>> +					 * PMD map right away. Chances are some
>> +					 * or all of the tail pages can be freed
>> +					 * without IO.
>>  					 */
>> -					if (!folio_entire_mapcount(folio) &&
>> +					if (folio_test_pmd_mappable(folio) &&
>> +					    !folio_entire_mapcount(folio) &&
>>  					    split_folio_to_list(folio,
>>  								folio_list))
>>  						goto activate_locked;
> 
> --
> Best Regards,
> Huang, Ying