Re: OOM behavior in constrained memory situations

[Andrew Morton <akpm@osdl.org>]

Christoph Lameter <clameter@engr.sgi.com> wrote:
>
> On Mon, 6 Feb 2006, Andi Kleen wrote:
> 
> > At least remnants from my old 80% hack to avoid this (huge_page_needed)
> > seem to be still there in mainline:
> > 
> > fs/hugetlbfs/inode.c:hugetlbfs_file_mmap
> > 
> >    bytes = huge_pages_needed(mapping, vma);
> >    if (!is_hugepage_mem_enough(bytes))
> >           return -ENOMEM;
> > 
> > 
> > So something must be broken if this doesn't work. Or did you allocate
> > the pages in some other way? 
> 
> huge pages are now allocated in the huge fault handler. If it would be 
> returning an OOM then the OOM killer may be activated.

?

The oom-killer is invoked from the page allocator.  A hugetlb pagefault
won't use the page allocator.  So there shouldn't be an oom-killing on
hugepage exhaustion.

I think this comment is just wrong:

		/* Logically this is OOM, not a SIGBUS, but an OOM
		 * could cause the kernel to go killing other
		 * processes which won't help the hugepage situation
		 * at all (?) */

A VM_FAULT_OOM from there won't cause the oom-killer to do anything.  We
should return VM_FAULT_OOM and let do_page_fault() commit suicide with
SIGKILL.