MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 1918 bytes --]

Right now, a failed MAP_FIXED mmap() can leave a hole in the address
space.  Another mmap() can then enter this hole.  This was fixed in
the past, but the fix was undone in
4f87153e82c4 ("mm: change failure of MAP_FIXED to restoring the gap on failure").

This is really nasty, and can create serious problems for userspace.
A simple example is MAP_FIXED over a part of a KVM memslot to expose
memory to a guest.  If that leaves a hole, the result is catastrophic:
Subsequent mmap()s (perhaps by malloc()) can get put into that address
range.  KVM will give the guest access to whatever happens to be put
there, and one must assume the guest can turn that into a VM escape.

Other situations don't result in immediate disaster, but are still
really hard to recover from.  One can use MAP_FIXED_NOREPLACE to
refill the hole if nobody else has filled it, but if someone has,
there's no hope of making the original mapping (that the MAP_FIXED had
clobbered) contiguous again.  One must either figure out how to handle
a non-contiguous mapping, rebuild the mapping elsewhere, or crash.

In my case, I can work around the problem by using a single memslot per
file descriptor, but I suspect there are other use-cases that aren't
so lucky.  Yet others may simply not have realized this was possible;
there's even a Stack Overflow answer [1] that claims that the map is
replaced atomically.  I only found out about it from Claude Opus 5.5.
I asked it to write a test suite, and it pointed out that a failed
MAP_FIXED mmap() can leave a hole.  If I hadn't asked, I would have
shipped a security vulnerability.

Should the behavior be changed again, so that mmap() with MAP_FIXED
is atomic?  That would provide a UAPI that actually makes sense and
can be used without horribly ugly workarounds.

[1]: https://stackoverflow.com/a/28576674
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

[Lorenzo Stoakes (ARM)]

[ please check kernel mails in MAINTAINERS - my and Liam's at least were
  incorrect, which makes it easy to miss. Fixed those up. ]

Thanks for the report.

In general linux system calls do not guarantee compound operation
atomicity, and any assumption based on that is incorrect. If you see an
error - the operation failed, perhaps partially.

Further, a failed MAP_FIXED leaving gaps was the case for most of the
history of linux. Avoiding that is infeasible.

I advise you to review the NOTES section
https://man7.org/linux/man-pages/man2/mmap.2.html on MAP_FIXED - your
suggested use of it is not recommended.

--
Cheers, Lorenzo

Re: MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

[Demi Marie Obenour]

[-- Attachment #1.1: Type: text/plain, Size: 1995 bytes --]

On 11/5/26 10:11, Lorenzo Stoakes (ARM) wrote:
> [ please check kernel mails in MAINTAINERS - my and Liam's at least were
>   incorrect, which makes it easy to miss. Fixed those up. ]

I thought I did that, but obviously I didn't do it properly.

> Thanks for the report.
> 
> In general linux system calls do not guarantee compound operation
> atomicity, and any assumption based on that is incorrect. If you see an
> error - the operation failed, perhaps partially.
> 
> Further, a failed MAP_FIXED leaving gaps was the case for most of the> history of linux. Avoiding that is infeasible.
> 
> I advise you to review the NOTES section
> https://man7.org/linux/man-pages/man2/mmap.2.html on MAP_FIXED - your
> suggested use of it is not recommended.

Does that mean that MAP_FIXED should be deprecated?  It seems that the
safe use-cases can all use MAP_FIXED_NOREPLACE, and that the use-cases
that cannot be served by MAP_FIXED_NOREPLACE cannot be served by current
Linux.

Specifically, if a MAP_FIXED mmap() fails,  the mapping might either
still be there (in which case one _must_ call munmap() to cleanup)
or it might not be there (in which case one _must not_ call munmap()).
There's no race-free way to tell which is the case.  This means that
MAP_FIXED is unusable in multithreaded programs, unless one is okay
with leaking address space.

Are there cases, such as anonymous mappings, that cannot leave holes?
I believe there are some programs that depend on this to handle
SIGBUS faults from truncated mapped files.  If even that can leave
a hole, the only thread-safe alternative would be longjmp() out of
the SIGBUS handler.

An alternative behavior would be to unmap the pages, but keep the
address space reserved, as if the mapping was a PROT_NONE anonymous one.
That would avoid the "it may or may not have been freed" situation,
as it would be safe to retry the mapping or call munmap().
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

[Vlastimil Babka (SUSE)]

On 11/6/26 00:59, Demi Marie Obenour wrote:
> On 11/5/26 10:11, Lorenzo Stoakes (ARM) wrote:
>> [ please check kernel mails in MAINTAINERS - my and Liam's at least were
>>   incorrect, which makes it easy to miss. Fixed those up. ]
> 
> I thought I did that, but obviously I didn't do it properly.
> 
>> Thanks for the report.
>> 
>> In general linux system calls do not guarantee compound operation
>> atomicity, and any assumption based on that is incorrect. If you see an
>> error - the operation failed, perhaps partially.

I'd think we only cared about not exposing the temporary state between unmap
old and map new, right?

>> Further, a failed MAP_FIXED leaving gaps was the case for most of the> history of linux. Avoiding that is infeasible.
>> 
>> I advise you to review the NOTES section
>> https://man7.org/linux/man-pages/man2/mmap.2.html on MAP_FIXED - your
>> suggested use of it is not recommended.
> 
> Does that mean that MAP_FIXED should be deprecated?  It seems that the
> safe use-cases can all use MAP_FIXED_NOREPLACE, and that the use-cases
> that cannot be served by MAP_FIXED_NOREPLACE cannot be served by current
> Linux.
> 
> Specifically, if a MAP_FIXED mmap() fails,  the mapping might either
> still be there (in which case one _must_ call munmap() to cleanup)
> or it might not be there (in which case one _must not_ call munmap()).
> There's no race-free way to tell which is the case.  This means that
> MAP_FIXED is unusable in multithreaded programs, unless one is okay
> with leaking address space.

You said "This was fixed in the past, but the fix was undone in
4f87153e82c4". Any idea when exactly it was "fixed", whether the change was
done consciously (or some refactoring side-effect) and if somebody started
depending on it?
4f87153e82c4 is from 6.12 (2 years old by now) and nobody complained so far...

> Are there cases, such as anonymous mappings, that cannot leave holes?

(I don't have an immediate answer. I'd guess that if e.g. some kernel
allocation can fail, this can happen)

> I believe there are some programs that depend on this to handle
> SIGBUS faults from truncated mapped files.  If even that can leave
> a hole, the only thread-safe alternative would be longjmp() out of
> the SIGBUS handler.

A concrete example would help I guess.

> An alternative behavior would be to unmap the pages, but keep the
> address space reserved, as if the mapping was a PROT_NONE anonymous one.
> That would avoid the "it may or may not have been freed" situation,
> as it would be safe to retry the mapping or call munmap().

It would probably have to be a new flag as that's a different semantic from
what we have now. And the use case for adding it would have to be strong IMHO.