[PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Alexis Bouzigues via B4 Relay]

From: Alexis Bouzigues <BouziguesAlexis@JohnDeere.com>

Commit 212925802454 ("mm: oom: let oom_reap_task and exit_mmap run
concurrently") moved setting MMF_OOM_SKIP from __mmput() into the MMU
exit_mmap(), so it is no longer set when a NOMMU mm is torn down.

As a result, a NOMMU OOM victim that has exited and freed its memory
keeps the OOM killer from selecting another victim until it is reaped.

Set MMF_OOM_SKIP in the NOMMU exit_mmap() once the VMAs are freed, as
the MMU version does.

Fixes: 212925802454 ("mm: oom: let oom_reap_task and exit_mmap run concurrently")
Assisted-by: LLM
Signed-off-by: Alexis Bouzigues <BouziguesAlexis@JohnDeere.com>
---
Restore MMF_OOM_SKIP on NOMMU mm teardown, which was lost when it moved
into the MMU-only exit_mmap().
---
 mm/nommu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/nommu.c b/mm/nommu.c
index 498e01ee40b0..913b2a1b5b3b 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1555,6 +1555,7 @@ void exit_mmap(struct mm_struct *mm)
 		delete_vma(mm, vma);
 		cond_resched();
 	}
+	mm_flags_set(MMF_OOM_SKIP, mm);
 	__mt_destroy(&mm->mm_mt);
 	mmap_write_unlock(mm);
 }

---
base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
change-id: 20261013-fix-nommu-exit-mmap-oom-skip-b8e68bf34b6d

Best regards,
-- 
Alexis Bouzigues <BouziguesAlexis@JohnDeere.com>

Re: [PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Michal Hocko]

On Tue 13-10-26 17:56:19, Alexis Bouzigues via B4 Relay wrote:
> From: Alexis Bouzigues <BouziguesAlexis@JohnDeere.com>
> 
> Commit 212925802454 ("mm: oom: let oom_reap_task and exit_mmap run
> concurrently") moved setting MMF_OOM_SKIP from __mmput() into the MMU
> exit_mmap(), so it is no longer set when a NOMMU mm is torn down.
> 
> As a result, a NOMMU OOM victim that has exited and freed its memory
> keeps the OOM killer from selecting another victim until it is reaped.
> 
> Set MMF_OOM_SKIP in the NOMMU exit_mmap() once the VMAs are freed, as
> the MMU version does.

Is this a real problem that you can actually hit?

> Fixes: 212925802454 ("mm: oom: let oom_reap_task and exit_mmap run concurrently")
> Assisted-by: LLM
> Signed-off-by: Alexis Bouzigues <BouziguesAlexis@JohnDeere.com>
> ---
> Restore MMF_OOM_SKIP on NOMMU mm teardown, which was lost when it moved
> into the MMU-only exit_mmap().
> ---
>  mm/nommu.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/mm/nommu.c b/mm/nommu.c
> index 498e01ee40b0..913b2a1b5b3b 100644
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -1555,6 +1555,7 @@ void exit_mmap(struct mm_struct *mm)
>  		delete_vma(mm, vma);
>  		cond_resched();
>  	}
> +	mm_flags_set(MMF_OOM_SKIP, mm);
>  	__mt_destroy(&mm->mm_mt);
>  	mmap_write_unlock(mm);
>  }
> 
> ---
> base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
> change-id: 20261013-fix-nommu-exit-mmap-oom-skip-b8e68bf34b6d
> 
> Best regards,
> -- 
> Alexis Bouzigues <BouziguesAlexis@JohnDeere.com>
> 
> 

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Alexis Bouzigues]

On Wed, Oct 14, 2026 at 09:50:30AM +0200, Michal Hocko wrote:
> > Set MMF_OOM_SKIP in the NOMMU exit_mmap() once the VMAs are freed, as
> > the MMU version does.
> Is this a real problem that you can actually hit?

Yes, without this change the oom killer can fail to make progress to
resolve a global oom. This gets seeded by a parent PID making an
allocation that triggers oom killer before reaping an exited child oom
victim. The parent pid expected to reap the victim indefinitely retries
in the page allocator because the oom killer does not mark a new victim
until the old victim is reaped.

A /init reproducer is provided below. I tested this patch on both the
reproducer and original benchmark where I encountered the livelocks.

---8<---
#include <poll.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/wait.h>
#include <signal.h>
#include <unistd.h>

#define TEST_TIME_MS 10000

static char *self;

static void hog(void)
{
 for (;;) {
  char *p = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
                 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
  if (p != MAP_FAILED) memset(p, 0xa5, 4096);
 }
}

static pid_t spawn(char *role, int *rfd)
{
 char *argv[] = { self, role, NULL };
 char *envp[] = { NULL };
 int p[2];
 pid_t pid;

 if (pipe(p)) return -1;

 pid = vfork();
 if (pid == 0) {
  close(p[0]);
  execve(self, argv, envp);
  _exit(127);
 }

 close(p[1]);
 *rfd = p[0];

 return pid;
}

static long wait_eof(int fd, long timeout_ms)
{
 struct pollfd pfd = { .fd = fd, .events = POLLIN };
 long ms;

 for (ms = 0; timeout_ms < 0 || ms < timeout_ms; ms += 100)
  if (poll(&pfd, 1, 100) > 0)
   return ms;

 return -1;
}

static void parent(void)
{
 int fd;
 pid_t child = spawn("child", &fd);

 if (child < 0 || wait_eof(fd, -1) < 0)
  _exit(1);

 hog();
}

int main(int argc, char **argv)
{
 int fd;
 long ms;
 pid_t p;

 self = argv[0];

 if (argc > 1 && !strcmp(argv[1], "child")) hog();
 if (argc > 1 && !strcmp(argv[1], "parent")) parent();

 p = spawn("parent", &fd);
 if (p < 0)
  return 1;

 ms = wait_eof(fd, TEST_TIME_MS);
 if (ms < 0) kill(p, SIGKILL);

 int status = 0;

 if (!(waitpid(p, &status, 0) == p && WIFSIGNALED(status) &&
       WTERMSIG(status) == SIGKILL))
  printf("Parent pid exited unexpectedly\n");
 else if (ms >= 0)
  printf("Parent pid was killed\n");
 else
  printf("Parent pid was still alive!\n");

 if (getpid() == 1)
  for (;;)
   poll(NULL, 0, -1);

 return ms < 0;
}
--->8---

Re: [PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Michal Hocko]

On Wed 14-10-26 21:58:52, Alexis Bouzigues wrote:
> On Wed, Oct 14, 2026 at 09:50:30AM +0200, Michal Hocko wrote:
> > > Set MMF_OOM_SKIP in the NOMMU exit_mmap() once the VMAs are freed, as
> > > the MMU version does.
> > Is this a real problem that you can actually hit?
> 
> Yes, without this change the oom killer can fail to make progress to
> resolve a global oom.

Are you really looking into the code or just relying on an LLM?
Go and check to learn that the oom reaper is only compiled in with
CONFIG_MMU=y

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Alexis Bouzigues]

On Thu, Oct 15, 2026 at 09:03:57AM +0200, Michal Hocko wrote:
> Are you really looking into the code or just relying on an LLM?
> Go and check to learn that the oom reaper is only compiled in with
> CONFIG_MMU=y

Hello,

Sorry about that, my earlier message used "reap" in the sense of
cleaning up the exited child in zombie state (waitpid), not in the sense
of oom-reaper.

I also went through the code in detail today manually using a 
backtrace from the reproducer. I set a breakpoint at oom_evaluate_task 
(containing the MMF_OOM_SKIP check) ran the reproducer on v7.3-rc7
through qemu-system-arm with gdb. I iterated to confirm the
breakpoint keeps getting hit.

Here is a backtrace of the reproducer when the parent PID is livelocked:

    oom_evaluate_task                  mm/oom_kill.c:354
    select_bad_process                 mm/oom_kill.c:373
    out_of_memory                      mm/oom_kill.c:1155
    __alloc_pages_may_oom              mm/page_alloc.c:4113
    __alloc_pages_slowpath             mm/page_alloc.c:5030
    __alloc_frozen_pages_noprof        mm/page_alloc.c:5449
    __alloc_pages_noprof               mm/page_alloc.c:5471
    alloc_pages_node_noprof            mm/page_alloc.c:5485
    alloc_pages_noprof                 include/linux/gfp.h:284
    get_free_pages_noprof              mm/page_alloc.c:5507
    alloc_pages_exact_noprof           mm/page_alloc.c:5641
    do_mmap_private                    mm/nommu.c:959
    do_mmap                            mm/nommu.c:1182
    vm_mmap_pgoff                      mm/util.c:581
    ksys_mmap_pgoff                    mm/nommu.c:1287


Inside oom_evaluate_task(), when the task is the zombie child pid
victim, without MMF_OOM_SKIP it aborts the loop from select_bad_process()
and sets oc->chosen to -1, preventing the oom killer from calling
oom_kill_process on oc->chosen

Since oc->chosen is not NULL, it does not hit the deadlock branch in
out_of_memory instead it returns !!oc->chosen (1).

Returning to __alloc_pages_may_oom(), out_of_memory(&oc) evaluates
non-zero, so did_some_progress from __alloc_pages_slowpath gets set
to 1.

Then in __alloc_pages_slowpath(), since did_some_progress was set
to 1, it jumps to the retry label

As long as the system is out of memory, __alloc_pages_may_oom and 
out_of_memory will get called, meaning that oom killer does not make
progress and that the parent PID busy spins in the allocation.

If the MMF_OOM_SKIP bit were to be set for the child zombie process, then
oom_evaluate_task would not abort and would select the parent, meaning the
oom killer can now call oom_kill_process on oc->chosen and progress.

So setting MMF_OOM_SKIP as soon as its memory gets freed (in exit_mmap)
would guard against the parent's mmap stalling oom killer, since
oom_evaluate_task would rule the child out and proceed.

Re: [PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Michal Hocko]

On Fri 16-10-26 00:42:39, Alexis Bouzigues wrote:
> On Thu, Oct 15, 2026 at 09:03:57AM +0200, Michal Hocko wrote:
> > Are you really looking into the code or just relying on an LLM?
> > Go and check to learn that the oom reaper is only compiled in with
> > CONFIG_MMU=y
> 
> Hello,
> 
> Sorry about that, my earlier message used "reap" in the sense of
> cleaning up the exited child in zombie state (waitpid), not in the sense
> of oom-reaper.

OK, this is what confused me indeed. Especially along with the changelog
explicitly mentioning coordination with the oom reaper
: Fixes: 212925802454 ("mm: oom: let oom_reap_task and exit_mmap run concurrently")

The whole oom reaper chnages should have no impact on the nommu OOM
workflow.

> I also went through the code in detail today manually using a 
> backtrace from the reproducer. I set a breakpoint at oom_evaluate_task 
> (containing the MMF_OOM_SKIP check) ran the reproducer on v7.3-rc7
> through qemu-system-arm with gdb. I iterated to confirm the
> breakpoint keeps getting hit.
> 
> Here is a backtrace of the reproducer when the parent PID is livelocked:
> 
>     oom_evaluate_task                  mm/oom_kill.c:354
>     select_bad_process                 mm/oom_kill.c:373
>     out_of_memory                      mm/oom_kill.c:1155
>     __alloc_pages_may_oom              mm/page_alloc.c:4113
>     __alloc_pages_slowpath             mm/page_alloc.c:5030
>     __alloc_frozen_pages_noprof        mm/page_alloc.c:5449
>     __alloc_pages_noprof               mm/page_alloc.c:5471
>     alloc_pages_node_noprof            mm/page_alloc.c:5485
>     alloc_pages_noprof                 include/linux/gfp.h:284
>     get_free_pages_noprof              mm/page_alloc.c:5507
>     alloc_pages_exact_noprof           mm/page_alloc.c:5641
>     do_mmap_private                    mm/nommu.c:959
>     do_mmap                            mm/nommu.c:1182
>     vm_mmap_pgoff                      mm/util.c:581
>     ksys_mmap_pgoff                    mm/nommu.c:1287
> 
> 
> Inside oom_evaluate_task(), when the task is the zombie child pid
> victim, without MMF_OOM_SKIP it aborts the loop from select_bad_process()
> and sets oc->chosen to -1, preventing the oom killer from calling
> oom_kill_process on oc->chosen

Correc.

> Since oc->chosen is not NULL, it does not hit the deadlock branch in
> out_of_memory instead it returns !!oc->chosen (1).
> 
> Returning to __alloc_pages_may_oom(), out_of_memory(&oc) evaluates
> non-zero, so did_some_progress from __alloc_pages_slowpath gets set
> to 1.
> 
> Then in __alloc_pages_slowpath(), since did_some_progress was set
> to 1, it jumps to the retry label
> 
> As long as the system is out of memory, __alloc_pages_may_oom and 
> out_of_memory will get called, meaning that oom killer does not make
> progress and that the parent PID busy spins in the allocation.

Correct. The question is why killing the child, which has passed
exit_mmap stage sucessfully didn't help.

> If the MMF_OOM_SKIP bit were to be set for the child zombie process, then
> oom_evaluate_task would not abort and would select the parent, meaning the
> oom killer can now call oom_kill_process on oc->chosen and progress.
> 
> So setting MMF_OOM_SKIP as soon as its memory gets freed (in exit_mmap)
> would guard against the parent's mmap stalling oom killer, since
> oom_evaluate_task would rule the child out and proceed.

right MMF_OOM_SKIP will help to hide the killed child process. The
question is whether this is just working around a propblem or it is a
correct solution/workaround. There is absolutely no guarantee of a
forward progress for nommu (due to missing oom_reaper) and overall
nommu OOM handling is rather fragile and there is an implicit assumption
that memory usage in this platform is very carefully scrutinized to make
sure this doesn't really happen.

So the main question you should focus on to explain is whether your
specific example is  
	a) a corner case of a valid nommu application or  this is 
	   a theoretical problem
	b) why this doesn't break otherwise very fragile nommu oom
	   handling
Thanks!
-- 
Michal Hocko
SUSE Labs