Re: [REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Luiz Capitulino <luizcap@redhat.com>]

On 10/23/26 3:16 AM, Vlastimil Babka (SUSE) wrote:
> On 10/23/26 04:31, Luiz Capitulino wrote:
>> Growing a VM_MIXEDMAP VMA with mremap(), with or without MREMAP_MAYMOVE, fails
>> with -ENOMEM when the address range after the VMA is free.
>>
>> This regression was introduced in v6.1 by commit ca3d76b0aa80 ("mm: add merging
>> after mremap resize"). This commit changed mremap() to use vma_merge() instead
>> of vma_adjust() for the expansion case. However, vma_merge() returns NULL for
>> a VMA that has any of the VM_SPECIAL flags set.
> 
> Hi,
> 
> 6.1 was a rather long time ago. How was it discovered just now? I.e. what
> use case broke by the change and prompted creating the reproducer?

The use-case that broke was an application running on an older kernel
and accessing a GPFS volume/FS. I was told that GPFS sets VM_MIXEDMAP on
a VMA because it inserts pages in the process address space with
vm_insert_page(). I then asked an LLM to check if there was anything in
tree that would reproduce that, it found the tcp_mmap() case and created
the attached reproducer. Btw, the LLM also told me there are more cases
in-tree that can reproduce this (but they require special hardware or
setup so I didn't investigate).

> 
> Thanks,
> Vlastimil
> 
>> The issue is also present in latest Linus tree, except that the code has been
>> refactored. In the current tree, the path now is expand_vma_in_place() ->
>> vma_merge_extend() -> vma_merge_new_range(), which performs the check:
>>
>> 	/* Special VMAs are unmergeable, also if no prev/next. */
>> 	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
>> 	    (!prev && !next))
>> 		return NULL;
>>
>> This issue was originally reproduced with an external module that sets
>> VM_MIXEDMAP on a VMA because it wants to insert pages with vm_insert_page().
>> However, there's at least one in-tree reproducer: tcp_mmap().
>>
>> Note that commit ca3d76b0aa80 introduced other VMA expansion regressions,
>> but they were caught and fixed early:
>>
>> - 6f12be792fde ("mm, mremap: fix mremap() expanding vma with addr inside vma")
>> - d014cd7c1c35 ("mm, mremap: fix mremap() expanding for vma's with vm_ops->close()")
>>
>> Below you'll find a reproducer and an attempted fix.
>>
>> Reproducer
>> ==========
>>
>> This reproducer was written by an LLM and uses tcp_mmap(). It reproduces
>> against v6.1 and latest Linus tree (2a08195e4db5). The issue is not
>> reproduced on v6.0.
>>
>> #define _GNU_SOURCE
>> #include <stdio.h>
>> #include <unistd.h>
>> #include <errno.h>
>> #include <sys/mman.h>
>> #include <sys/socket.h>
>>
>> int main(void)
>> {
>>          int fd, pagesize;
>>          void *p, *ret;
>>
>>          fd = socket(AF_INET, SOCK_STREAM, 0);
>>          if (fd < 0) {
>>                  perror("socket");
>>                  return 1;
>>          }
>>
>>          pagesize = getpagesize();
>>
>>          /* tcp_mmap() sets VM_MIXEDMAP on this vma. */
>>          p = mmap(NULL, 2 * pagesize, PROT_READ, MAP_SHARED, fd, 0);
>>          if (p == MAP_FAILED) {
>>                  perror("mmap");
>>                  return 1;
>>          }
>>
>>          /* Drop the upper half, so growing back into it can happen in place. */
>>          if (munmap((char *)p + pagesize, pagesize)) {
>>                  perror("munmap");
>>                  return 1;
>>          }
>>
>>          errno = 0;
>>          ret = mremap(p, pagesize, 2 * pagesize, MREMAP_MAYMOVE);
>>          if (ret == MAP_FAILED) {
>>                  printf("BUG REPRODUCED errno=%d\n", errno);
>>                  return 1;
>>          }
>>
>>          printf("NOT REPRODUCED\n");
>>
>>          return 0;
>> }
>>
>> Sketch fix
>> ==========
>>
>> In the fix below, I started with the same idea as d014cd7c1c35: on v6.1 I
>> made mremap() expand VM_MIXEDMAP VMAs the same way that commit did for
>> VMAs with vm_ops->close(). Then I adapted it to the current tree. It fixes
>> the reproducer and I didn't get regressions in MM kselftests nor the
>> tools/testing/vma test.
>>
>> However, I'm not posting this as a fix because I'm unfamiliar with the VMA
>> code so I'm not confident this is the right approach.
>>
>> I really don't mind someone more familiar with the code taking this over.
>>
>> diff --git a/mm/vma.c b/mm/vma.c
>> index 6c35f0d775ab..ac9a0a8a4be9 100644
>> --- a/mm/vma.c
>> +++ b/mm/vma.c
>> @@ -851,9 +851,15 @@ static int commit_merge(struct vma_merge_struct *vmg)
>>   	return 0;
>>   }
>>   
>> -/* We can only remove VMAs when merging if they do not have a close hook. */
>>   static bool can_merge_remove_vma(struct vm_area_struct *vma)
>>   {
>> +	if (vma_flags_test(&vma->flags, VMA_MIXEDMAP_BIT))
>> +		return false;
>> +
>> +	/*
>> +	 * We can only remove VMAs when merging if they do not have a
>> +	 * close hook.
>> +	 */
>>   	return !vma->vm_ops || !vma->vm_ops->close;
>>   }
>>   
>> @@ -1093,6 +1099,20 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(
>>   	return NULL;
>>   }
>>   
>> +static bool can_merge_vma(const struct vma_merge_struct *vmg)
>> +{
>> +	/* VMAs without prev/next are unmergeable */
>> +	if (!vmg->prev && !vmg->next)
>> +		return false;
>> +
>> +	/* We can't merge VMA_IO, VMA_DONTEXPAND or VMA_PFNMAP */
>> +	if (vma_flags_test_any(&vmg->vma_flags, VMA_IO_BIT, VMA_DONTEXPAND_BIT,
>> +			       VMA_PFNMAP_BIT))
>> +		return false;
>> +
>> +	return true;
>> +}
>> +
>>   /*
>>    * vma_merge_new_range - Attempt to merge a new VMA into address space
>>    *
>> @@ -1152,9 +1172,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
>>   
>>   	vmg->state = VMA_MERGE_NOMERGE;
>>   
>> -	/* Special VMAs are unmergeable, also if no prev/next. */
>> -	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
>> -	    (!prev && !next))
>> +	if (!can_merge_vma(vmg))
>>   		return NULL;
>>   
>>   	can_merge_left = can_vma_merge_left(vmg);
>>
>