[REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Luiz Capitulino <luizcap@redhat.com>]

Growing a VM_MIXEDMAP VMA with mremap(), with or without MREMAP_MAYMOVE, fails
with -ENOMEM when the address range after the VMA is free.

This regression was introduced in v6.1 by commit ca3d76b0aa80 ("mm: add merging
after mremap resize"). This commit changed mremap() to use vma_merge() instead
of vma_adjust() for the expansion case. However, vma_merge() returns NULL for
a VMA that has any of the VM_SPECIAL flags set.

The issue is also present in latest Linus tree, except that the code has been
refactored. In the current tree, the path now is expand_vma_in_place() ->
vma_merge_extend() -> vma_merge_new_range(), which performs the check:

	/* Special VMAs are unmergeable, also if no prev/next. */
	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
	    (!prev && !next))
		return NULL;

This issue was originally reproduced with an external module that sets
VM_MIXEDMAP on a VMA because it wants to insert pages with vm_insert_page().
However, there's at least one in-tree reproducer: tcp_mmap().

Note that commit ca3d76b0aa80 introduced other VMA expansion regressions,
but they were caught and fixed early:

- 6f12be792fde ("mm, mremap: fix mremap() expanding vma with addr inside vma")
- d014cd7c1c35 ("mm, mremap: fix mremap() expanding for vma's with vm_ops->close()")

Below you'll find a reproducer and an attempted fix.

Reproducer
==========

This reproducer was written by an LLM and uses tcp_mmap(). It reproduces
against v6.1 and latest Linus tree (2a08195e4db5). The issue is not
reproduced on v6.0.

#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/socket.h>

int main(void)
{
        int fd, pagesize;
        void *p, *ret;

        fd = socket(AF_INET, SOCK_STREAM, 0);
        if (fd < 0) {
                perror("socket");
                return 1;
        }

        pagesize = getpagesize();

        /* tcp_mmap() sets VM_MIXEDMAP on this vma. */
        p = mmap(NULL, 2 * pagesize, PROT_READ, MAP_SHARED, fd, 0);
        if (p == MAP_FAILED) {
                perror("mmap");
                return 1;
        }

        /* Drop the upper half, so growing back into it can happen in place. */
        if (munmap((char *)p + pagesize, pagesize)) {
                perror("munmap");
                return 1;
        }

        errno = 0;
        ret = mremap(p, pagesize, 2 * pagesize, MREMAP_MAYMOVE);
        if (ret == MAP_FAILED) {
                printf("BUG REPRODUCED errno=%d\n", errno);
                return 1;
        }

        printf("NOT REPRODUCED\n");

        return 0;
}

Sketch fix
==========

In the fix below, I started with the same idea as d014cd7c1c35: on v6.1 I
made mremap() expand VM_MIXEDMAP VMAs the same way that commit did for 
VMAs with vm_ops->close(). Then I adapted it to the current tree. It fixes
the reproducer and I didn't get regressions in MM kselftests nor the
tools/testing/vma test.

However, I'm not posting this as a fix because I'm unfamiliar with the VMA
code so I'm not confident this is the right approach.

I really don't mind someone more familiar with the code taking this over.

diff --git a/mm/vma.c b/mm/vma.c
index 6c35f0d775ab..ac9a0a8a4be9 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -851,9 +851,15 @@ static int commit_merge(struct vma_merge_struct *vmg)
 	return 0;
 }
 
-/* We can only remove VMAs when merging if they do not have a close hook. */
 static bool can_merge_remove_vma(struct vm_area_struct *vma)
 {
+	if (vma_flags_test(&vma->flags, VMA_MIXEDMAP_BIT))
+		return false;
+
+	/*
+	 * We can only remove VMAs when merging if they do not have a
+	 * close hook.
+	 */
 	return !vma->vm_ops || !vma->vm_ops->close;
 }
 
@@ -1093,6 +1099,20 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(
 	return NULL;
 }
 
+static bool can_merge_vma(const struct vma_merge_struct *vmg)
+{
+	/* VMAs without prev/next are unmergeable */
+	if (!vmg->prev && !vmg->next)
+		return false;
+
+	/* We can't merge VMA_IO, VMA_DONTEXPAND or VMA_PFNMAP */
+	if (vma_flags_test_any(&vmg->vma_flags, VMA_IO_BIT, VMA_DONTEXPAND_BIT,
+			       VMA_PFNMAP_BIT))
+		return false;
+
+	return true;
+}
+
 /*
  * vma_merge_new_range - Attempt to merge a new VMA into address space
  *
@@ -1152,9 +1172,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
 
 	vmg->state = VMA_MERGE_NOMERGE;
 
-	/* Special VMAs are unmergeable, also if no prev/next. */
-	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
-	    (!prev && !next))
+	if (!can_merge_vma(vmg))
 		return NULL;
 
 	can_merge_left = can_vma_merge_left(vmg);