[REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Luiz Capitulino]

Growing a VM_MIXEDMAP VMA with mremap(), with or without MREMAP_MAYMOVE, fails
with -ENOMEM when the address range after the VMA is free.

This regression was introduced in v6.1 by commit ca3d76b0aa80 ("mm: add merging
after mremap resize"). This commit changed mremap() to use vma_merge() instead
of vma_adjust() for the expansion case. However, vma_merge() returns NULL for
a VMA that has any of the VM_SPECIAL flags set.

The issue is also present in latest Linus tree, except that the code has been
refactored. In the current tree, the path now is expand_vma_in_place() ->
vma_merge_extend() -> vma_merge_new_range(), which performs the check:

	/* Special VMAs are unmergeable, also if no prev/next. */
	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
	    (!prev && !next))
		return NULL;

This issue was originally reproduced with an external module that sets
VM_MIXEDMAP on a VMA because it wants to insert pages with vm_insert_page().
However, there's at least one in-tree reproducer: tcp_mmap().

Note that commit ca3d76b0aa80 introduced other VMA expansion regressions,
but they were caught and fixed early:

- 6f12be792fde ("mm, mremap: fix mremap() expanding vma with addr inside vma")
- d014cd7c1c35 ("mm, mremap: fix mremap() expanding for vma's with vm_ops->close()")

Below you'll find a reproducer and an attempted fix.

Reproducer
==========

This reproducer was written by an LLM and uses tcp_mmap(). It reproduces
against v6.1 and latest Linus tree (2a08195e4db5). The issue is not
reproduced on v6.0.

#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/socket.h>

int main(void)
{
        int fd, pagesize;
        void *p, *ret;

        fd = socket(AF_INET, SOCK_STREAM, 0);
        if (fd < 0) {
                perror("socket");
                return 1;
        }

        pagesize = getpagesize();

        /* tcp_mmap() sets VM_MIXEDMAP on this vma. */
        p = mmap(NULL, 2 * pagesize, PROT_READ, MAP_SHARED, fd, 0);
        if (p == MAP_FAILED) {
                perror("mmap");
                return 1;
        }

        /* Drop the upper half, so growing back into it can happen in place. */
        if (munmap((char *)p + pagesize, pagesize)) {
                perror("munmap");
                return 1;
        }

        errno = 0;
        ret = mremap(p, pagesize, 2 * pagesize, MREMAP_MAYMOVE);
        if (ret == MAP_FAILED) {
                printf("BUG REPRODUCED errno=%d\n", errno);
                return 1;
        }

        printf("NOT REPRODUCED\n");

        return 0;
}

Sketch fix
==========

In the fix below, I started with the same idea as d014cd7c1c35: on v6.1 I
made mremap() expand VM_MIXEDMAP VMAs the same way that commit did for 
VMAs with vm_ops->close(). Then I adapted it to the current tree. It fixes
the reproducer and I didn't get regressions in MM kselftests nor the
tools/testing/vma test.

However, I'm not posting this as a fix because I'm unfamiliar with the VMA
code so I'm not confident this is the right approach.

I really don't mind someone more familiar with the code taking this over.

diff --git a/mm/vma.c b/mm/vma.c
index 6c35f0d775ab..ac9a0a8a4be9 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -851,9 +851,15 @@ static int commit_merge(struct vma_merge_struct *vmg)
 	return 0;
 }
 
-/* We can only remove VMAs when merging if they do not have a close hook. */
 static bool can_merge_remove_vma(struct vm_area_struct *vma)
 {
+	if (vma_flags_test(&vma->flags, VMA_MIXEDMAP_BIT))
+		return false;
+
+	/*
+	 * We can only remove VMAs when merging if they do not have a
+	 * close hook.
+	 */
 	return !vma->vm_ops || !vma->vm_ops->close;
 }
 
@@ -1093,6 +1099,20 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(
 	return NULL;
 }
 
+static bool can_merge_vma(const struct vma_merge_struct *vmg)
+{
+	/* VMAs without prev/next are unmergeable */
+	if (!vmg->prev && !vmg->next)
+		return false;
+
+	/* We can't merge VMA_IO, VMA_DONTEXPAND or VMA_PFNMAP */
+	if (vma_flags_test_any(&vmg->vma_flags, VMA_IO_BIT, VMA_DONTEXPAND_BIT,
+			       VMA_PFNMAP_BIT))
+		return false;
+
+	return true;
+}
+
 /*
  * vma_merge_new_range - Attempt to merge a new VMA into address space
  *
@@ -1152,9 +1172,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
 
 	vmg->state = VMA_MERGE_NOMERGE;
 
-	/* Special VMAs are unmergeable, also if no prev/next. */
-	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
-	    (!prev && !next))
+	if (!can_merge_vma(vmg))
 		return NULL;
 
 	can_merge_left = can_vma_merge_left(vmg);

Re: [REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Vlastimil Babka (SUSE)]

On 10/23/26 04:31, Luiz Capitulino wrote:
> Growing a VM_MIXEDMAP VMA with mremap(), with or without MREMAP_MAYMOVE, fails
> with -ENOMEM when the address range after the VMA is free.
> 
> This regression was introduced in v6.1 by commit ca3d76b0aa80 ("mm: add merging
> after mremap resize"). This commit changed mremap() to use vma_merge() instead
> of vma_adjust() for the expansion case. However, vma_merge() returns NULL for
> a VMA that has any of the VM_SPECIAL flags set.

Hi,

6.1 was a rather long time ago. How was it discovered just now? I.e. what
use case broke by the change and prompted creating the reproducer?

Thanks,
Vlastimil

> The issue is also present in latest Linus tree, except that the code has been
> refactored. In the current tree, the path now is expand_vma_in_place() ->
> vma_merge_extend() -> vma_merge_new_range(), which performs the check:
> 
> 	/* Special VMAs are unmergeable, also if no prev/next. */
> 	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
> 	    (!prev && !next))
> 		return NULL;
> 
> This issue was originally reproduced with an external module that sets
> VM_MIXEDMAP on a VMA because it wants to insert pages with vm_insert_page().
> However, there's at least one in-tree reproducer: tcp_mmap().
> 
> Note that commit ca3d76b0aa80 introduced other VMA expansion regressions,
> but they were caught and fixed early:
> 
> - 6f12be792fde ("mm, mremap: fix mremap() expanding vma with addr inside vma")
> - d014cd7c1c35 ("mm, mremap: fix mremap() expanding for vma's with vm_ops->close()")
> 
> Below you'll find a reproducer and an attempted fix.
> 
> Reproducer
> ==========
> 
> This reproducer was written by an LLM and uses tcp_mmap(). It reproduces
> against v6.1 and latest Linus tree (2a08195e4db5). The issue is not
> reproduced on v6.0.
> 
> #define _GNU_SOURCE
> #include <stdio.h>
> #include <unistd.h>
> #include <errno.h>
> #include <sys/mman.h>
> #include <sys/socket.h>
> 
> int main(void)
> {
>         int fd, pagesize;
>         void *p, *ret;
> 
>         fd = socket(AF_INET, SOCK_STREAM, 0);
>         if (fd < 0) {
>                 perror("socket");
>                 return 1;
>         }
> 
>         pagesize = getpagesize();
> 
>         /* tcp_mmap() sets VM_MIXEDMAP on this vma. */
>         p = mmap(NULL, 2 * pagesize, PROT_READ, MAP_SHARED, fd, 0);
>         if (p == MAP_FAILED) {
>                 perror("mmap");
>                 return 1;
>         }
> 
>         /* Drop the upper half, so growing back into it can happen in place. */
>         if (munmap((char *)p + pagesize, pagesize)) {
>                 perror("munmap");
>                 return 1;
>         }
> 
>         errno = 0;
>         ret = mremap(p, pagesize, 2 * pagesize, MREMAP_MAYMOVE);
>         if (ret == MAP_FAILED) {
>                 printf("BUG REPRODUCED errno=%d\n", errno);
>                 return 1;
>         }
> 
>         printf("NOT REPRODUCED\n");
> 
>         return 0;
> }
> 
> Sketch fix
> ==========
> 
> In the fix below, I started with the same idea as d014cd7c1c35: on v6.1 I
> made mremap() expand VM_MIXEDMAP VMAs the same way that commit did for 
> VMAs with vm_ops->close(). Then I adapted it to the current tree. It fixes
> the reproducer and I didn't get regressions in MM kselftests nor the
> tools/testing/vma test.
> 
> However, I'm not posting this as a fix because I'm unfamiliar with the VMA
> code so I'm not confident this is the right approach.
> 
> I really don't mind someone more familiar with the code taking this over.
> 
> diff --git a/mm/vma.c b/mm/vma.c
> index 6c35f0d775ab..ac9a0a8a4be9 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -851,9 +851,15 @@ static int commit_merge(struct vma_merge_struct *vmg)
>  	return 0;
>  }
>  
> -/* We can only remove VMAs when merging if they do not have a close hook. */
>  static bool can_merge_remove_vma(struct vm_area_struct *vma)
>  {
> +	if (vma_flags_test(&vma->flags, VMA_MIXEDMAP_BIT))
> +		return false;
> +
> +	/*
> +	 * We can only remove VMAs when merging if they do not have a
> +	 * close hook.
> +	 */
>  	return !vma->vm_ops || !vma->vm_ops->close;
>  }
>  
> @@ -1093,6 +1099,20 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(
>  	return NULL;
>  }
>  
> +static bool can_merge_vma(const struct vma_merge_struct *vmg)
> +{
> +	/* VMAs without prev/next are unmergeable */
> +	if (!vmg->prev && !vmg->next)
> +		return false;
> +
> +	/* We can't merge VMA_IO, VMA_DONTEXPAND or VMA_PFNMAP */
> +	if (vma_flags_test_any(&vmg->vma_flags, VMA_IO_BIT, VMA_DONTEXPAND_BIT,
> +			       VMA_PFNMAP_BIT))
> +		return false;
> +
> +	return true;
> +}
> +
>  /*
>   * vma_merge_new_range - Attempt to merge a new VMA into address space
>   *
> @@ -1152,9 +1172,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
>  
>  	vmg->state = VMA_MERGE_NOMERGE;
>  
> -	/* Special VMAs are unmergeable, also if no prev/next. */
> -	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
> -	    (!prev && !next))
> +	if (!can_merge_vma(vmg))
>  		return NULL;
>  
>  	can_merge_left = can_vma_merge_left(vmg);
> 

Re: [REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Lorenzo Stoakes (ARM)]

I'll look into this but obviously we won't make mixed map regions mergeable (nor
were they ever), that's completely broken.

The most recent version of the mm code (which is going to be send to Linus soon
in this merge window) documents why:

/**
 * vma_flags_can_merge() - Do the specified VMA flags permit the VMA to be
 * merged with another?
 * @flags: The VMA flags to test.
 * Returns: true if the flags permit merging, false otherwise.
 */
static inline bool vma_flags_can_merge(const vma_flags_t *flags)
{
	/*
	 * VMA merging assumes that a VMA's flags and fields completely describe
	 * its state, which is only the case for mm-backed mappings.
	 *
	 * Mappings which are not mm-managed may have established state upon
	 * mapping not embodied in any attribute of the VMA.
	 *
	 * Additionally, private (CoW) PFN maps encode the source PFN of the
	 * range in vma->vm_pgoff, which may otherwise cause spurious merges.
	 */
	if (!vma_flags_is_mm_backed(flags))
		return false;
	/*
	 * hugetlb mappings contain per-VMA state, so their flags and fields do
	 * not completely describe them.
	 */
	return !vma_flags_is_hugetlb(flags);
}

I'll dig in + get back to you.

--
Cheers, Lorenzo

Re: [REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Lorenzo Stoakes (ARM)]

On Fri, Oct 23, 2026 at 09:16:12AM +0200, Vlastimil Babka (SUSE) wrote:
> On 10/23/26 04:31, Luiz Capitulino wrote:

> > This regression was introduced in v6.1 by commit ca3d76b0aa80 ("mm: add merging
> > after mremap resize"). This commit changed mremap() to use vma_merge() instead
> > of vma_adjust() for the expansion case. However, vma_merge() returns NULL for
> > a VMA that has any of the VM_SPECIAL flags set.
>
> Hi,
>
> 6.1 was a rather long time ago. How was it discovered just now? I.e. what
> use case broke by the change and prompted creating the reproducer?
>
> Vlastimil
> Thanks,

Thanks, I have the same questions :)

Do always include that kind of information and whether it was an LLM-discovered
thing or whether you had an actual workload that was impacted, etc.

--
Cheers, Lorenzo