MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

[Demi Marie Obenour <demiobenour@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 1918 bytes --]

Right now, a failed MAP_FIXED mmap() can leave a hole in the address
space.  Another mmap() can then enter this hole.  This was fixed in
the past, but the fix was undone in
4f87153e82c4 ("mm: change failure of MAP_FIXED to restoring the gap on failure").

This is really nasty, and can create serious problems for userspace.
A simple example is MAP_FIXED over a part of a KVM memslot to expose
memory to a guest.  If that leaves a hole, the result is catastrophic:
Subsequent mmap()s (perhaps by malloc()) can get put into that address
range.  KVM will give the guest access to whatever happens to be put
there, and one must assume the guest can turn that into a VM escape.

Other situations don't result in immediate disaster, but are still
really hard to recover from.  One can use MAP_FIXED_NOREPLACE to
refill the hole if nobody else has filled it, but if someone has,
there's no hope of making the original mapping (that the MAP_FIXED had
clobbered) contiguous again.  One must either figure out how to handle
a non-contiguous mapping, rebuild the mapping elsewhere, or crash.

In my case, I can work around the problem by using a single memslot per
file descriptor, but I suspect there are other use-cases that aren't
so lucky.  Yet others may simply not have realized this was possible;
there's even a Stack Overflow answer [1] that claims that the map is
replaced atomically.  I only found out about it from Claude Opus 5.5.
I asked it to write a test suite, and it pointed out that a failed
MAP_FIXED mmap() can leave a hole.  If I hadn't asked, I would have
shipped a security vulnerability.

Should the behavior be changed again, so that mmap() with MAP_FIXED
is atomic?  That would provide a UAPI that actually makes sense and
can be used without horribly ugly workarounds.

[1]: https://stackoverflow.com/a/28576674
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]