Re: MAP_FIXED leaving holes on failure is very difficult to recover from and can even be a security hole

["Vlastimil Babka (SUSE)" <vbabka@kernel.org>]

On 11/6/26 00:59, Demi Marie Obenour wrote:
> On 11/5/26 10:11, Lorenzo Stoakes (ARM) wrote:
>> [ please check kernel mails in MAINTAINERS - my and Liam's at least were
>>   incorrect, which makes it easy to miss. Fixed those up. ]
> 
> I thought I did that, but obviously I didn't do it properly.
> 
>> Thanks for the report.
>> 
>> In general linux system calls do not guarantee compound operation
>> atomicity, and any assumption based on that is incorrect. If you see an
>> error - the operation failed, perhaps partially.

I'd think we only cared about not exposing the temporary state between unmap
old and map new, right?

>> Further, a failed MAP_FIXED leaving gaps was the case for most of the> history of linux. Avoiding that is infeasible.
>> 
>> I advise you to review the NOTES section
>> https://man7.org/linux/man-pages/man2/mmap.2.html on MAP_FIXED - your
>> suggested use of it is not recommended.
> 
> Does that mean that MAP_FIXED should be deprecated?  It seems that the
> safe use-cases can all use MAP_FIXED_NOREPLACE, and that the use-cases
> that cannot be served by MAP_FIXED_NOREPLACE cannot be served by current
> Linux.
> 
> Specifically, if a MAP_FIXED mmap() fails,  the mapping might either
> still be there (in which case one _must_ call munmap() to cleanup)
> or it might not be there (in which case one _must not_ call munmap()).
> There's no race-free way to tell which is the case.  This means that
> MAP_FIXED is unusable in multithreaded programs, unless one is okay
> with leaking address space.

You said "This was fixed in the past, but the fix was undone in
4f87153e82c4". Any idea when exactly it was "fixed", whether the change was
done consciously (or some refactoring side-effect) and if somebody started
depending on it?
4f87153e82c4 is from 6.12 (2 years old by now) and nobody complained so far...

> Are there cases, such as anonymous mappings, that cannot leave holes?

(I don't have an immediate answer. I'd guess that if e.g. some kernel
allocation can fail, this can happen)

> I believe there are some programs that depend on this to handle
> SIGBUS faults from truncated mapped files.  If even that can leave
> a hole, the only thread-safe alternative would be longjmp() out of
> the SIGBUS handler.

A concrete example would help I guess.

> An alternative behavior would be to unmap the pages, but keep the
> address space reserved, as if the mapping was a PROT_NONE anonymous one.
> That would avoid the "it may or may not have been freed" situation,
> as it would be safe to retry the mapping or call munmap().

It would probably have to be a new flag as that's a different semantic from
what we have now. And the use case for adding it would have to be strong IMHO.