* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
@ 2017-07-27 8:29 Manish Jaggi
2017-07-27 9:24 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Manish Jaggi @ 2017-07-27 8:29 UTC (permalink / raw)
To: linux-kernel, Michal Hocko, Andrew Morton, Nair, Jayachandran,
Andrea Arcangeli, David Rientjes, Oleg Nesterov, Lomovtsev,
Vadim
Hi Michal,
On Mon, Jul 24, 2017 at 09:23:32AM +0200, Michal Hocko wrote:
>From: Michal Hocko <mhocko@suse.com>
>
>David has noticed that the oom killer might kill additional tasks while
>the exiting oom victim hasn't terminated yet because the oom_reaper marks
>the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
>to 0. The race is as follows
>
>oom_reap_task do_exit
> exit_mm
> __oom_reap_task_mm
> mmput
> __mmput
> mmget_not_zero # fails
> exit_mmap # frees memory
> set_bit(MMF_OOM_SKIP)
>
>The victim is still visible to the OOM killer until it is unhashed.
>
>Currently we try to reduce a risk of this race by taking oom_lock
>and wait for out_of_memory sleep while holding the lock to give the
>victim some time to exit. This is quite suboptimal approach because
>there is no guarantee the victim (especially a large one) will manage
>to unmap its address space and free enough memory to the particular oom
>domain which needs a memory (e.g. a specific NUMA node).
>
>Fix this problem by allowing __oom_reap_task_mm and __mmput path to
>race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
>to run in parallel with other unmappers (hence the mmap_sem for read).
>
>The only tricky part is to exclude page tables tear down and all
>operations which modify the address space in the __mmput path. exit_mmap
>doesn't expect any other users so it doesn't use any locking. Nothing
>really forbids us to use mmap_sem for write, though. In fact we are
>already relying on this lock earlier in the __mmput path to synchronize
>with ksm and khugepaged.
>
>Take the exclusive mmap_sem when calling free_pgtables and destroying
>vmas to sync with __oom_reap_task_mm which take the lock for read. All
>other operations can safely race with the parallel unmap.
>
>Changes
>- bail on null mm->mmap early as per David Rientjes
With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
The issue experienced was as follows
that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
that pid was marked as killed but somewhere there is a race and the process didnt get killed.
and the oom01/oom02 test started killing further processes, till it panics.
IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
the killing of victim.
Log (https://pastebin.com/hg5iXRj2)
As a subtest of oom02 starts, it prints out the victim - In this case 4578
oom02 0 TINFO : start OOM testing for mlocked pages.
oom02 0 TINFO : expected victim is 4578.
When oom02 thread invokes oom-killer, it did select 4578 for killing...
[ 364.737486] oom02:4583 invoked oom-killer: gfp_mask=0x16080c0(GFP_KERNEL|__GFP_ZERO|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
[...] snip
[ 365.036127] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 365.044691] [ 1905] 0 1905 3236 1714 10 4 0 0 systemd-journal
[...] snip
[ 365.222325] [ 4491] 0 4491 27965 1022 8 3 0 0 bash
[ 365.230849] [ 4513] 0 4513 670 365 5 3 0 0 oom02
[ 365.239459] [ 4578] 0 4578 37776030 32890957 64257 138 0 0 oom02
[ 365.248067] Out of memory: Kill process 4578 (oom02) score 952 or sacrifice child
[ 365.255581] Killed process 4578 (oom02) total-vm:151104120kB, anon-rss:131562528kB, file-rss:1300kB, shmem-rss:0kB
[ 365.266829] out_of_memory: Current (4583) has a pending SIGKILL
[ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
[ 365.282658] oom_reaper: reaped process 4583 (oom02), now anon-rss:131561664kB, file-rss:0kB, shmem-rss:0kB
==> At this point, the test should have completed with a TPASS or TFAIL, but it didnt and it continues invoking oom-killer again.
[ 365.283361] oom02:4586 invoked oom-killer: gfp_mask=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
[ 365.283368] oom02 cpuset=/ mems_allowed=0-1
later it panics...
[ 365.576298] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 365.576338] [ 2421] 0 2421 3241 878 9 3 0 -1000 systemd-udevd
[ 365.576342] [ 3125] 0 3125 3834 719 9 4 0 -1000 auditd
[ 365.576347] [ 3309] 0 3309 3332 616 10 3 0 -1000 sshd
[ 365.576356] [ 4580] 0 4578 37776030 32890417 64258 138 0 0 oom02
[ 365.576361] Kernel panic - not syncing: Out of memory and no killable processes...
[ 365.576361]
-Thanks
Manish Jaggi
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-27 8:29 [PATCH] mm, oom: allow oom reaper to race with exit_mmap Manish Jaggi
@ 2017-07-27 9:24 ` Michal Hocko
2017-07-28 11:52 ` Possible race condition in oom-killer Manish Jaggi
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-27 9:24 UTC (permalink / raw)
To: Manish Jaggi
Cc: linux-kernel, Andrew Morton, Nair, Jayachandran,
Andrea Arcangeli, David Rientjes, Oleg Nesterov, Lomovtsev,
Vadim
On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
[...]
> With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
> The issue experienced was as follows
> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
> and the oom01/oom02 test started killing further processes, till it panics.
>
> IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
> If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
> the killing of victim.
>
> Log (https://pastebin.com/hg5iXRj2)
>
> As a subtest of oom02 starts, it prints out the victim - In this case 4578
>
> oom02 0 TINFO : start OOM testing for mlocked pages.
> oom02 0 TINFO : expected victim is 4578.
>
> When oom02 thread invokes oom-killer, it did select 4578 for killing...
I will definitely have a look. Can you report it in a separate email
thread please? Are you able to reproduce with the current Linus or
linux-next trees?
>
>
> [ 364.737486] oom02:4583 invoked oom-killer: gfp_mask=0x16080c0(GFP_KERNEL|__GFP_ZERO|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
> [...] snip
> [ 365.036127] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
> [ 365.044691] [ 1905] 0 1905 3236 1714 10 4 0 0 systemd-journal
> [...] snip
> [ 365.222325] [ 4491] 0 4491 27965 1022 8 3 0 0 bash
> [ 365.230849] [ 4513] 0 4513 670 365 5 3 0 0 oom02
> [ 365.239459] [ 4578] 0 4578 37776030 32890957 64257 138 0 0 oom02
> [ 365.248067] Out of memory: Kill process 4578 (oom02) score 952 or sacrifice child
> [ 365.255581] Killed process 4578 (oom02) total-vm:151104120kB, anon-rss:131562528kB, file-rss:1300kB, shmem-rss:0kB
> [ 365.266829] out_of_memory: Current (4583) has a pending SIGKILL
> [ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
> [ 365.282658] oom_reaper: reaped process 4583 (oom02), now anon-rss:131561664kB, file-rss:0kB, shmem-rss:0kB
>
> ==> At this point, the test should have completed with a TPASS or TFAIL, but it didnt and it continues invoking oom-killer again.
>
> [ 365.283361] oom02:4586 invoked oom-killer: gfp_mask=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
Yes because
[ 365.283499] Node 1 Normal free:19500kB min:33804kB low:165916kB high:298028kB active_anon:13312kB inactive_anon:172kB active_file:0kB inactive_file:1044kB unevictable:131560064kB writepending:0kB present:134213632kB managed:132113248kB mlocked:131560064kB slab_reclaimable:5748kB slab_unreclaimable:17808kB kernel_stack:2720kB pagetables:254636kB bounce:0kB free_pcp:10476kB local_pcp:144kB free_cma:0kB
Although we have killed and reaped oom02 process Node1 is still below
min watermark and that is why we have hit the oom killer again. It
is not immediatelly clear to me why, that would require a deeper
inspection.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Possible race condition in oom-killer
2017-07-27 9:24 ` Michal Hocko
@ 2017-07-28 11:52 ` Manish Jaggi
2017-07-28 12:32 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Manish Jaggi @ 2017-07-28 11:52 UTC (permalink / raw)
To: Michal Hocko, linux-kernel
was: Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
Hi Michal,
On 7/27/2017 2:54 PM, Michal Hocko wrote:
> On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
> [...]
>> With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
>> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
>> The issue experienced was as follows
>> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
>> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
>> and the oom01/oom02 test started killing further processes, till it panics.
>> IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
>> If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
>> the killing of victim.
>>
>> Log (https://pastebin.com/hg5iXRj2)
>>
>> As a subtest of oom02 starts, it prints out the victim - In this case 4578
>>
>> oom02 0 TINFO : start OOM testing for mlocked pages.
>> oom02 0 TINFO : expected victim is 4578.
>>
>> When oom02 thread invokes oom-killer, it did select 4578 for killing...
> I will definitely have a look. Can you report it in a separate email
> thread please? Are you able to reproduce with the current Linus or
> linux-next trees?
Yes this issue is visible with linux-next.
>>
>> [ 364.737486] oom02:4583 invoked oom-killer: gfp_mask=0x16080c0(GFP_KERNEL|__GFP_ZERO|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
>> [...] snip
>> [ 365.036127] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
>> [ 365.044691] [ 1905] 0 1905 3236 1714 10 4 0 0 systemd-journal
>> [...] snip
>> [ 365.222325] [ 4491] 0 4491 27965 1022 8 3 0 0 bash
>> [ 365.230849] [ 4513] 0 4513 670 365 5 3 0 0 oom02
>> [ 365.239459] [ 4578] 0 4578 37776030 32890957 64257 138 0 0 oom02
>> [ 365.248067] Out of memory: Kill process 4578 (oom02) score 952 or sacrifice child
>> [ 365.255581] Killed process 4578 (oom02) total-vm:151104120kB, anon-rss:131562528kB, file-rss:1300kB, shmem-rss:0kB
>> [ 365.266829] out_of_memory: Current (4583) has a pending SIGKILL
>> [ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
>> [ 365.282658] oom_reaper: reaped process 4583 (oom02), now anon-rss:131561664kB, file-rss:0kB, shmem-rss:0kB
>>
>> ==> At this point, the test should have completed with a TPASS or TFAIL, but it didnt and it continues invoking oom-killer again.
>>
>> [ 365.283361] oom02:4586 invoked oom-killer: gfp_mask=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
> Yes because
> [ 365.283499] Node 1 Normal free:19500kB min:33804kB low:165916kB high:298028kB active_anon:13312kB inactive_anon:172kB active_file:0kB inactive_file:1044kB unevictable:131560064kB writepending:0kB present:134213632kB managed:132113248kB mlocked:131560064kB slab_reclaimable:5748kB slab_unreclaimable:17808kB kernel_stack:2720kB pagetables:254636kB bounce:0kB free_pcp:10476kB local_pcp:144kB free_cma:0kB
>
> Although we have killed and reaped oom02 process Node1 is still below
> min watermark and that is why we have hit the oom killer again. It
> is not immediatelly clear to me why, that would require a deeper
> inspection.
I have a doubt here
my understanding of oom test: oom() function basically forks itself and
starts n threads each thread has a loop which allocates and touches
memory thus will trigger oom-killer and will kill the process. the
parent process is on a wait() and will print pass/fail.
So IIUC when 4578 is reaped all the child threads should be terminated,
which happens in pass case (line 152)
But even after being killed and reaped, the oom killer is invoked again
which doesn't seem right.
Could it be that the process is just marked hidden from oom including
its threads, thus oom-killer continues.
What do you think ?
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 11:52 ` Possible race condition in oom-killer Manish Jaggi
@ 2017-07-28 12:32 ` Michal Hocko
2017-07-28 12:59 ` Tetsuo Handa
2017-07-28 13:50 ` Manish Jaggi
0 siblings, 2 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-28 12:32 UTC (permalink / raw)
To: Manish Jaggi; +Cc: linux-kernel, linux-mm
[CC linux-mm]
On Fri 28-07-17 17:22:25, Manish Jaggi wrote:
> was: Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
>
> Hi Michal,
> On 7/27/2017 2:54 PM, Michal Hocko wrote:
> >On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
> >[...]
> >>With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
> >> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
> >>The issue experienced was as follows
> >> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
> >> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
> >> and the oom01/oom02 test started killing further processes, till it panics.
> >>IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
> >>If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
> >>the killing of victim.
> >>
> >>Log (https://pastebin.com/hg5iXRj2)
> >>
> >>As a subtest of oom02 starts, it prints out the victim - In this case 4578
> >>
> >>oom02 0 TINFO : start OOM testing for mlocked pages.
> >>oom02 0 TINFO : expected victim is 4578.
> >>
> >>When oom02 thread invokes oom-killer, it did select 4578 for killing...
> >I will definitely have a look. Can you report it in a separate email
> >thread please? Are you able to reproduce with the current Linus or
> >linux-next trees?
> Yes this issue is visible with linux-next.
Could you provide the full kernel log from this run please? I do not
expect there to be much difference but just to be sure that the code I
am looking at matches logs.
[...]
> >>[ 365.283361] oom02:4586 invoked oom-killer: gfp_mask=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
> >Yes because
> >[ 365.283499] Node 1 Normal free:19500kB min:33804kB low:165916kB high:298028kB active_anon:13312kB inactive_anon:172kB active_file:0kB inactive_file:1044kB unevictable:131560064kB writepending:0kB present:134213632kB managed:132113248kB mlocked:131560064kB slab_reclaimable:5748kB slab_unreclaimable:17808kB kernel_stack:2720kB pagetables:254636kB bounce:0kB free_pcp:10476kB local_pcp:144kB free_cma:0kB
> >
> >Although we have killed and reaped oom02 process Node1 is still below
> >min watermark and that is why we have hit the oom killer again. It
> >is not immediatelly clear to me why, that would require a deeper
> >inspection.
> I have a doubt here
> my understanding of oom test: oom() function basically forks itself and
> starts n threads each thread has a loop which allocates and touches memory
> thus will trigger oom-killer and will kill the process. the parent process
> is on a wait() and will print pass/fail.
>
> So IIUC when 4578 is reaped all the child threads should be terminated,
> which happens in pass case (line 152)
> But even after being killed and reaped, the oom killer is invoked again
> which doesn't seem right.
As I've said the OOM killer hits because the memory from Node 1 didn't
get freed for some reasov or got immediatally populated.
> Could it be that the process is just marked hidden from oom including its
> threads, thus oom-killer continues.
The whole process should be killed and the OOM reaper should only mark
the victim oom invisible _after_ the address space has been reaped (and
memory freed). You said the patch from
http://lkml.kernel.org/r/20170724072332.31903-1-mhocko@kernel.org didn't
help so it shouldn't be a race with the last __mmput.
Thanks!
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 12:32 ` Michal Hocko
@ 2017-07-28 12:59 ` Tetsuo Handa
2017-07-28 13:07 ` Michal Hocko
2017-07-28 13:15 ` Manish Jaggi
2017-07-28 13:50 ` Manish Jaggi
1 sibling, 2 replies; 54+ messages in thread
From: Tetsuo Handa @ 2017-07-28 12:59 UTC (permalink / raw)
To: Michal Hocko, Manish Jaggi; +Cc: linux-kernel, linux-mm
(Oops. Forgot to add CC.)
On 2017/07/28 21:32, Michal Hocko wrote:
> [CC linux-mm]
>
> On Fri 28-07-17 17:22:25, Manish Jaggi wrote:
>> was: Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
>>
>> Hi Michal,
>> On 7/27/2017 2:54 PM, Michal Hocko wrote:
>>> On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
>>> [...]
>>>> With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
>>>> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
>>>> The issue experienced was as follows
>>>> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
>>>> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
>>>> and the oom01/oom02 test started killing further processes, till it panics.
>>>> IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
>>>> If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
>>>> the killing of victim.
>>>>
>>>> Log (https://pastebin.com/hg5iXRj2)
>>>>
>>>> As a subtest of oom02 starts, it prints out the victim - In this case 4578
>>>>
>>>> oom02 0 TINFO : start OOM testing for mlocked pages.
>>>> oom02 0 TINFO : expected victim is 4578.
>>>>
>>>> When oom02 thread invokes oom-killer, it did select 4578 for killing...
>>> I will definitely have a look. Can you report it in a separate email
>>> thread please? Are you able to reproduce with the current Linus or
>>> linux-next trees?
>> Yes this issue is visible with linux-next.
>
> Could you provide the full kernel log from this run please? I do not
> expect there to be much difference but just to be sure that the code I
> am looking at matches logs.
4578 is consuming memory as mlocked pages. But the OOM reaper cannot reclaim
mlocked pages (i.e. can_madv_dontneed_vma() returns false due to VM_LOCKED), can it?
oom02 0 TINFO : start OOM testing for mlocked pages.
oom02 0 TINFO : expected victim is 4578.
[ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
As a result, MMF_OOM_SKIP is set without reclaiming much memory.
Thus, it is natural that subsequent OOM victims are selected immediately because
almost all memory is still in use. Since 4578 is multi-threaded (isn't it?),
it will take time to call final __mmput() because mm->users are large.
Since there are many threads, it is possible that all OOM killable processes are
killed before final __mmput() of 4578 (which releases mlocked pages) is called.
>
> [...]
>>>> [ 365.283361] oom02:4586 invoked oom-killer: gfp_mask=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
>>> Yes because
>>> [ 365.283499] Node 1 Normal free:19500kB min:33804kB low:165916kB high:298028kB active_anon:13312kB inactive_anon:172kB active_file:0kB inactive_file:1044kB unevictable:131560064kB writepending:0kB present:134213632kB managed:132113248kB mlocked:131560064kB slab_reclaimable:5748kB slab_unreclaimable:17808kB kernel_stack:2720kB pagetables:254636kB bounce:0kB free_pcp:10476kB local_pcp:144kB free_cma:0kB
>>>
>>> Although we have killed and reaped oom02 process Node1 is still below
>>> min watermark and that is why we have hit the oom killer again. It
>>> is not immediatelly clear to me why, that would require a deeper
>>> inspection.
>> I have a doubt here
>> my understanding of oom test: oom() function basically forks itself and
>> starts n threads each thread has a loop which allocates and touches memory
>> thus will trigger oom-killer and will kill the process. the parent process
>> is on a wait() and will print pass/fail.
>>
>> So IIUC when 4578 is reaped all the child threads should be terminated,
>> which happens in pass case (line 152)
>> But even after being killed and reaped, the oom killer is invoked again
>> which doesn't seem right.
>
> As I've said the OOM killer hits because the memory from Node 1 didn't
> get freed for some reasov or got immediatally populated.
Because of mlocked pages by multi threaded process, it will take time to
reclaim mlocked pages.
>
>> Could it be that the process is just marked hidden from oom including its
>> threads, thus oom-killer continues.
>
> The whole process should be killed and the OOM reaper should only mark
> the victim oom invisible _after_ the address space has been reaped (and
> memory freed). You said the patch from
> http://lkml.kernel.org/r/20170724072332.31903-1-mhocko@kernel.org didn't
> help so it shouldn't be a race with the last __mmput.
>
> Thanks!
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 12:59 ` Tetsuo Handa
@ 2017-07-28 13:07 ` Michal Hocko
2017-07-28 13:15 ` Tetsuo Handa
2017-07-28 13:15 ` Manish Jaggi
1 sibling, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-28 13:07 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: Manish Jaggi, linux-kernel, linux-mm
On Fri 28-07-17 21:59:50, Tetsuo Handa wrote:
> (Oops. Forgot to add CC.)
>
> On 2017/07/28 21:32, Michal Hocko wrote:
> > [CC linux-mm]
> >
> > On Fri 28-07-17 17:22:25, Manish Jaggi wrote:
> >> was: Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
> >>
> >> Hi Michal,
> >> On 7/27/2017 2:54 PM, Michal Hocko wrote:
> >>> On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
> >>> [...]
> >>>> With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
> >>>> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
> >>>> The issue experienced was as follows
> >>>> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
> >>>> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
> >>>> and the oom01/oom02 test started killing further processes, till it panics.
> >>>> IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
> >>>> If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
> >>>> the killing of victim.
> >>>>
> >>>> Log (https://pastebin.com/hg5iXRj2)
> >>>>
> >>>> As a subtest of oom02 starts, it prints out the victim - In this case 4578
> >>>>
> >>>> oom02 0 TINFO : start OOM testing for mlocked pages.
> >>>> oom02 0 TINFO : expected victim is 4578.
> >>>>
> >>>> When oom02 thread invokes oom-killer, it did select 4578 for killing...
> >>> I will definitely have a look. Can you report it in a separate email
> >>> thread please? Are you able to reproduce with the current Linus or
> >>> linux-next trees?
> >> Yes this issue is visible with linux-next.
> >
> > Could you provide the full kernel log from this run please? I do not
> > expect there to be much difference but just to be sure that the code I
> > am looking at matches logs.
>
> 4578 is consuming memory as mlocked pages. But the OOM reaper cannot reclaim
> mlocked pages (i.e. can_madv_dontneed_vma() returns false due to VM_LOCKED), can it?
You are absolutely right. I am pretty sure I've checked mlocked counter
as the first thing but that must be from one of the earlier oom reports.
My fault I haven't checked it in the critical one
[ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
[ 365.282658] oom_reaper: reaped process 4583 (oom02), now anon-rss:131561664kB, file-rss:0kB, shmem-rss:0kB
and the above screemed about the fact I was just completely blind.
mlock pages handling is on my todo list for quite some time already but
I didn't get around it to implement that. mlock code is very tricky.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 13:07 ` Michal Hocko
@ 2017-07-28 13:15 ` Tetsuo Handa
2017-07-28 13:29 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Tetsuo Handa @ 2017-07-28 13:15 UTC (permalink / raw)
To: mhocko; +Cc: mjaggi, linux-kernel, linux-mm
Michal Hocko wrote:
> > 4578 is consuming memory as mlocked pages. But the OOM reaper cannot reclaim
> > mlocked pages (i.e. can_madv_dontneed_vma() returns false due to VM_LOCKED), can it?
>
> You are absolutely right. I am pretty sure I've checked mlocked counter
> as the first thing but that must be from one of the earlier oom reports.
> My fault I haven't checked it in the critical one
>
> [ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
> [ 365.282658] oom_reaper: reaped process 4583 (oom02), now anon-rss:131561664kB, file-rss:0kB, shmem-rss:0kB
>
> and the above screemed about the fact I was just completely blind.
>
> mlock pages handling is on my todo list for quite some time already but
> I didn't get around it to implement that. mlock code is very tricky.
task_will_free_mem(current) in out_of_memory() returning false due to
MMF_OOM_SKIP already set allowed each thread sharing that mm to select a new
OOM victim. If task_will_free_mem(current) in out_of_memory() did not return
false, threads sharing MMF_OOM_SKIP mm would not have selected new victims
to the level where all OOM killable processes are killed and calls panic().
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 12:59 ` Tetsuo Handa
2017-07-28 13:07 ` Michal Hocko
@ 2017-07-28 13:15 ` Manish Jaggi
1 sibling, 0 replies; 54+ messages in thread
From: Manish Jaggi @ 2017-07-28 13:15 UTC (permalink / raw)
To: Tetsuo Handa, Michal Hocko; +Cc: linux-kernel, linux-mm
Hello Tetsuo Handa,
On 7/28/2017 6:29 PM, Tetsuo Handa wrote:
> (Oops. Forgot to add CC.)
>
> On 2017/07/28 21:32, Michal Hocko wrote:
>> [CC linux-mm]
>>
>> On Fri 28-07-17 17:22:25, Manish Jaggi wrote:
>>> was: Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
>>>
>>> Hi Michal,
>>> On 7/27/2017 2:54 PM, Michal Hocko wrote:
>>>> On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
>>>> [...]
>>>>> With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
>>>>> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
>>>>> The issue experienced was as follows
>>>>> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
>>>>> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
>>>>> and the oom01/oom02 test started killing further processes, till it panics.
>>>>> IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
>>>>> If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
>>>>> the killing of victim.
>>>>>
>>>>> Log (https://pastebin.com/hg5iXRj2)
>>>>>
>>>>> As a subtest of oom02 starts, it prints out the victim - In this case 4578
>>>>>
>>>>> oom02 0 TINFO : start OOM testing for mlocked pages.
>>>>> oom02 0 TINFO : expected victim is 4578.
>>>>>
>>>>> When oom02 thread invokes oom-killer, it did select 4578 for killing...
>>>> I will definitely have a look. Can you report it in a separate email
>>>> thread please? Are you able to reproduce with the current Linus or
>>>> linux-next trees?
>>> Yes this issue is visible with linux-next.
>> Could you provide the full kernel log from this run please? I do not
>> expect there to be much difference but just to be sure that the code I
>> am looking at matches logs.
> 4578 is consuming memory as mlocked pages. But the OOM reaper cannot reclaim
> mlocked pages (i.e. can_madv_dontneed_vma() returns false due to VM_LOCKED), can it?
>
> oom02 0 TINFO : start OOM testing for mlocked pages.
> oom02 0 TINFO : expected victim is 4578.
> [ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
>
> As a result, MMF_OOM_SKIP is set without reclaiming much memory.
> Thus, it is natural that subsequent OOM victims are selected immediately because
> almost all memory is still in use. Since 4578 is multi-threaded (isn't it?),
> it will take time to call final __mmput() because mm->users are large.
> Since there are many threads, it is possible that all OOM killable processes are
> killed before final __mmput() of 4578 (which releases mlocked pages) is called.
My setup has 95 or more cores, so the large number of cores could be the
reason for the random failure?
>> [...]
>>>>> [ 365.283361] oom02:4586 invoked oom-killer: gfp_mask=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
>>>> Yes because
>>>> [ 365.283499] Node 1 Normal free:19500kB min:33804kB low:165916kB high:298028kB active_anon:13312kB inactive_anon:172kB active_file:0kB inactive_file:1044kB unevictable:131560064kB writepending:0kB present:134213632kB managed:132113248kB mlocked:131560064kB slab_reclaimable:5748kB slab_unreclaimable:17808kB kernel_stack:2720kB pagetables:254636kB bounce:0kB free_pcp:10476kB local_pcp:144kB free_cma:0kB
>>>>
>>>> Although we have killed and reaped oom02 process Node1 is still below
>>>> min watermark and that is why we have hit the oom killer again. It
>>>> is not immediatelly clear to me why, that would require a deeper
>>>> inspection.
>>> I have a doubt here
>>> my understanding of oom test: oom() function basically forks itself and
>>> starts n threads each thread has a loop which allocates and touches memory
>>> thus will trigger oom-killer and will kill the process. the parent process
>>> is on a wait() and will print pass/fail.
>>>
>>> So IIUC when 4578 is reaped all the child threads should be terminated,
>>> which happens in pass case (line 152)
>>> But even after being killed and reaped, the oom killer is invoked again
>>> which doesn't seem right.
>> As I've said the OOM killer hits because the memory from Node 1 didn't
>> get freed for some reasov or got immediatally populated.
> Because of mlocked pages by multi threaded process, it will take time to
> reclaim mlocked pages.
>
>>> Could it be that the process is just marked hidden from oom including its
>>> threads, thus oom-killer continues.
>> The whole process should be killed and the OOM reaper should only mark
>> the victim oom invisible _after_ the address space has been reaped (and
>> memory freed). You said the patch from
>> http://lkml.kernel.org/r/20170724072332.31903-1-mhocko@kernel.org didn't
>> help so it shouldn't be a race with the last __mmput.
>>
>> Thanks!
>>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 13:15 ` Tetsuo Handa
@ 2017-07-28 13:29 ` Michal Hocko
2017-07-28 13:55 ` Tetsuo Handa
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-28 13:29 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: mjaggi, linux-kernel, linux-mm
On Fri 28-07-17 22:15:01, Tetsuo Handa wrote:
> Michal Hocko wrote:
> > > 4578 is consuming memory as mlocked pages. But the OOM reaper cannot reclaim
> > > mlocked pages (i.e. can_madv_dontneed_vma() returns false due to VM_LOCKED), can it?
> >
> > You are absolutely right. I am pretty sure I've checked mlocked counter
> > as the first thing but that must be from one of the earlier oom reports.
> > My fault I haven't checked it in the critical one
> >
> > [ 365.267347] oom_reaper: reaped process 4578 (oom02), now anon-rss:131559616kB, file-rss:0kB, shmem-rss:0kB
> > [ 365.282658] oom_reaper: reaped process 4583 (oom02), now anon-rss:131561664kB, file-rss:0kB, shmem-rss:0kB
> >
> > and the above screemed about the fact I was just completely blind.
> >
> > mlock pages handling is on my todo list for quite some time already but
> > I didn't get around it to implement that. mlock code is very tricky.
>
> task_will_free_mem(current) in out_of_memory() returning false due to
> MMF_OOM_SKIP already set allowed each thread sharing that mm to select a new
> OOM victim. If task_will_free_mem(current) in out_of_memory() did not return
> false, threads sharing MMF_OOM_SKIP mm would not have selected new victims
> to the level where all OOM killable processes are killed and calls panic().
I am not sure I understand. Do you mean this?
---
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f030c1c..671e4a4107d0 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -779,13 +779,6 @@ static bool task_will_free_mem(struct task_struct *task)
if (!__task_will_free_mem(task))
return false;
- /*
- * This task has already been drained by the oom reaper so there are
- * only small chances it will free some more
- */
- if (test_bit(MMF_OOM_SKIP, &mm->flags))
- return false;
-
if (atomic_read(&mm->mm_users) <= 1)
return true;
If yes I would have to think about this some more because that might
have weird side effects (e.g. oom_victims counting after threads passed
exit_oom_victim).
Anyway the proper fix for this is to allow reaping mlocked pages. Is
something other than the LTP test affected to give this more priority?
Do we have other usecases where something mlocks the whole memory?
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 12:32 ` Michal Hocko
2017-07-28 12:59 ` Tetsuo Handa
@ 2017-07-28 13:50 ` Manish Jaggi
2017-07-28 14:12 ` Michal Hocko
1 sibling, 1 reply; 54+ messages in thread
From: Manish Jaggi @ 2017-07-28 13:50 UTC (permalink / raw)
To: Michal Hocko; +Cc: linux-kernel, linux-mm
Hi Michal,
On 7/28/2017 6:02 PM, Michal Hocko wrote:
> [CC linux-mm]
>
> On Fri 28-07-17 17:22:25, Manish Jaggi wrote:
>> was: Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
>>
>> Hi Michal,
>> On 7/27/2017 2:54 PM, Michal Hocko wrote:
>>> On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
>>> [...]
>>>> With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
>>>> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
>>>> The issue experienced was as follows
>>>> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
>>>> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
>>>> and the oom01/oom02 test started killing further processes, till it panics.
>>>> IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
>>>> If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
>>>> the killing of victim.
>>>>
>>>> Log (https://pastebin.com/hg5iXRj2)
>>>>
>>>> As a subtest of oom02 starts, it prints out the victim - In this case 4578
>>>>
>>>> oom02 0 TINFO : start OOM testing for mlocked pages.
>>>> oom02 0 TINFO : expected victim is 4578.
>>>>
>>>> When oom02 thread invokes oom-killer, it did select 4578 for killing...
>>> I will definitely have a look. Can you report it in a separate email
>>> thread please? Are you able to reproduce with the current Linus or
>>> linux-next trees?
>> Yes this issue is visible with linux-next.
> Could you provide the full kernel log from this run please? I do not
> expect there to be much difference but just to be sure that the code I
> am looking at matches logs.
The log is here: https://pastebin.com/Pmn5ZwEM
mlocked memory keeps on increasing till panic.
> [...]
>>>> [ 365.283361] oom02:4586 invoked oom-killer: gfp_mask=0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=1, order=0, oom_score_adj=0
>>> Yes because
>>> [ 365.283499] Node 1 Normal free:19500kB min:33804kB low:165916kB high:298028kB active_anon:13312kB inactive_anon:172kB active_file:0kB inactive_file:1044kB unevictable:131560064kB writepending:0kB present:134213632kB managed:132113248kB mlocked:131560064kB slab_reclaimable:5748kB slab_unreclaimable:17808kB kernel_stack:2720kB pagetables:254636kB bounce:0kB free_pcp:10476kB local_pcp:144kB free_cma:0kB
>>>
>>> Although we have killed and reaped oom02 process Node1 is still below
>>> min watermark and that is why we have hit the oom killer again. It
>>> is not immediatelly clear to me why, that would require a deeper
>>> inspection.
>> I have a doubt here
>> my understanding of oom test: oom() function basically forks itself and
>> starts n threads each thread has a loop which allocates and touches memory
>> thus will trigger oom-killer and will kill the process. the parent process
>> is on a wait() and will print pass/fail.
>>
>> So IIUC when 4578 is reaped all the child threads should be terminated,
>> which happens in pass case (line 152)
>> But even after being killed and reaped, the oom killer is invoked again
>> which doesn't seem right.
> As I've said the OOM killer hits because the memory from Node 1 didn't
> get freed for some reasov or got immediatally populated.
>
>> Could it be that the process is just marked hidden from oom including its
>> threads, thus oom-killer continues.
> The whole process should be killed and the OOM reaper should only mark
> the victim oom invisible _after_ the address space has been reaped (and
> memory freed). You said the patch from
> http://lkml.kernel.org/r/20170724072332.31903-1-mhocko@kernel.org didn't
> help so it shouldn't be a race with the last __mmput.
>
> Thanks!
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 13:29 ` Michal Hocko
@ 2017-07-28 13:55 ` Tetsuo Handa
2017-07-28 14:07 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Tetsuo Handa @ 2017-07-28 13:55 UTC (permalink / raw)
To: mhocko; +Cc: mjaggi, linux-kernel, linux-mm
Michal Hocko wrote:
> On Fri 28-07-17 22:15:01, Tetsuo Handa wrote:
> > task_will_free_mem(current) in out_of_memory() returning false due to
> > MMF_OOM_SKIP already set allowed each thread sharing that mm to select a new
> > OOM victim. If task_will_free_mem(current) in out_of_memory() did not return
> > false, threads sharing MMF_OOM_SKIP mm would not have selected new victims
> > to the level where all OOM killable processes are killed and calls panic().
>
> I am not sure I understand. Do you mean this?
Yes.
> ---
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index 9e8b4f030c1c..671e4a4107d0 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -779,13 +779,6 @@ static bool task_will_free_mem(struct task_struct *task)
> if (!__task_will_free_mem(task))
> return false;
>
> - /*
> - * This task has already been drained by the oom reaper so there are
> - * only small chances it will free some more
> - */
> - if (test_bit(MMF_OOM_SKIP, &mm->flags))
> - return false;
> -
> if (atomic_read(&mm->mm_users) <= 1)
> return true;
>
> If yes I would have to think about this some more because that might
> have weird side effects (e.g. oom_victims counting after threads passed
> exit_oom_victim).
But this check should not be removed unconditionally. We should still return
false if returning true was not sufficient to solve the OOM situation, for
we need to select next OOM victim in that case.
>
> Anyway the proper fix for this is to allow reaping mlocked pages.
Different approach is to set TIF_MEMDIE to all threads sharing the same
memory so that threads sharing MMF_OOM_SKIP mm do not need to call
out_of_memory() in order to get TIF_MEMDIE.
Yet another apporach is to use __GFP_KILLABLE (we can start it as
best effort basis).
> Is
> something other than the LTP test affected to give this more priority?
> Do we have other usecases where something mlocks the whole memory?
This panic was caused by 50 threads sharing MMF_OOM_SKIP mm exceeding
number of OOM killable processes. Whether memory is locked or not isn't
important. If a multi-threaded process which consumes little memory was
selected as an OOM victim (and reaped by the OOM reaper and MMF_OOM_SKIP
was set immediately), it might be still possible to select next OOM victims
needlessly.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 13:55 ` Tetsuo Handa
@ 2017-07-28 14:07 ` Michal Hocko
2017-07-29 4:31 ` Tetsuo Handa
2017-08-01 10:46 ` Tetsuo Handa
0 siblings, 2 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-28 14:07 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: mjaggi, linux-kernel, linux-mm
On Fri 28-07-17 22:55:51, Tetsuo Handa wrote:
> Michal Hocko wrote:
> > On Fri 28-07-17 22:15:01, Tetsuo Handa wrote:
> > > task_will_free_mem(current) in out_of_memory() returning false due to
> > > MMF_OOM_SKIP already set allowed each thread sharing that mm to select a new
> > > OOM victim. If task_will_free_mem(current) in out_of_memory() did not return
> > > false, threads sharing MMF_OOM_SKIP mm would not have selected new victims
> > > to the level where all OOM killable processes are killed and calls panic().
> >
> > I am not sure I understand. Do you mean this?
>
> Yes.
>
> > ---
> > diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> > index 9e8b4f030c1c..671e4a4107d0 100644
> > --- a/mm/oom_kill.c
> > +++ b/mm/oom_kill.c
> > @@ -779,13 +779,6 @@ static bool task_will_free_mem(struct task_struct *task)
> > if (!__task_will_free_mem(task))
> > return false;
> >
> > - /*
> > - * This task has already been drained by the oom reaper so there are
> > - * only small chances it will free some more
> > - */
> > - if (test_bit(MMF_OOM_SKIP, &mm->flags))
> > - return false;
> > -
> > if (atomic_read(&mm->mm_users) <= 1)
> > return true;
> >
> > If yes I would have to think about this some more because that might
> > have weird side effects (e.g. oom_victims counting after threads passed
> > exit_oom_victim).
>
> But this check should not be removed unconditionally. We should still return
> false if returning true was not sufficient to solve the OOM situation, for
> we need to select next OOM victim in that case.
>
> >
> > Anyway the proper fix for this is to allow reaping mlocked pages.
>
> Different approach is to set TIF_MEMDIE to all threads sharing the same
> memory so that threads sharing MMF_OOM_SKIP mm do not need to call
> out_of_memory() in order to get TIF_MEMDIE.
This is not so simple. If it were we could simply remove TIF_MEMDIE
altogether and rely on tsk_is_oom_victim.
> Yet another apporach is to use __GFP_KILLABLE (we can start it as
> best effort basis).
>
> > Is
> > something other than the LTP test affected to give this more priority?
> > Do we have other usecases where something mlocks the whole memory?
>
> This panic was caused by 50 threads sharing MMF_OOM_SKIP mm exceeding
> number of OOM killable processes. Whether memory is locked or not isn't
> important.
You are wrong here I believe. The whole problem is that the OOM victim
is consuming basically all the memory (that is what the test case
actually does IIRC) and that memory is mlocked. oom_reaper is much
faster to evaluate the mm of the victim and bail out sooner than the
exit path actually manages to tear down the address space. And so we
have to find other oom victims until we simply kill everything and
panic.
> If a multi-threaded process which consumes little memory was
> selected as an OOM victim (and reaped by the OOM reaper and MMF_OOM_SKIP
> was set immediately), it might be still possible to select next OOM victims
> needlessly.
This would be true if the address space itself only contained a little
amount of memory and the large part of the memory was in page tables or
other resources which oom_reaper cannot work with. This is not a usual
case though.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 13:50 ` Manish Jaggi
@ 2017-07-28 14:12 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-28 14:12 UTC (permalink / raw)
To: Manish Jaggi; +Cc: linux-kernel, linux-mm
On Fri 28-07-17 19:20:42, Manish Jaggi wrote:
>
> Hi Michal,
> On 7/28/2017 6:02 PM, Michal Hocko wrote:
> >[CC linux-mm]
> >
> >On Fri 28-07-17 17:22:25, Manish Jaggi wrote:
> >>was: Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
> >>
> >>Hi Michal,
> >>On 7/27/2017 2:54 PM, Michal Hocko wrote:
> >>>On Thu 27-07-17 13:59:09, Manish Jaggi wrote:
> >>>[...]
> >>>>With 4.11.6 I was getting random kernel panics (Out of memory - No process left to kill),
> >>>> when running LTP oom01 /oom02 ltp tests on our arm64 hardware with ~256G memory and high core count.
> >>>>The issue experienced was as follows
> >>>> that either test (oom01/oom02) selected a pid as victim and waited for the pid to be killed.
> >>>> that pid was marked as killed but somewhere there is a race and the process didnt get killed.
> >>>> and the oom01/oom02 test started killing further processes, till it panics.
> >>>>IIUC this issue is quite similar to your patch description. But applying your patch I still see the issue.
> >>>>If it is not related to this patch, can you please suggest by looking at the log, what could be preventing
> >>>>the killing of victim.
> >>>>
> >>>>Log (https://pastebin.com/hg5iXRj2)
> >>>>
> >>>>As a subtest of oom02 starts, it prints out the victim - In this case 4578
> >>>>
> >>>>oom02 0 TINFO : start OOM testing for mlocked pages.
> >>>>oom02 0 TINFO : expected victim is 4578.
> >>>>
> >>>>When oom02 thread invokes oom-killer, it did select 4578 for killing...
> >>>I will definitely have a look. Can you report it in a separate email
> >>>thread please? Are you able to reproduce with the current Linus or
> >>>linux-next trees?
> >>Yes this issue is visible with linux-next.
> >Could you provide the full kernel log from this run please? I do not
> >expect there to be much difference but just to be sure that the code I
> >am looking at matches logs.
> The log is here: https://pastebin.com/Pmn5ZwEM
> mlocked memory keeps on increasing till panic.
Thank you for retesting. I confirm the issue is that the oom reaper
hides the oom victim too early because the whole address space is
mlocked basically and there is not much to free. As the exit of the test
takes some time a new instance of the test pid 4625 in this case will go
and consume more than the exiting frees and that would go on an on until
we kill other eligible tasks until we panic due to no more eligible
tasks.
This is a bad situation and as I've said elsewhere in the thread the
proper fix is to teach the oom reaper to handle mlocked pages. This is
not a trivial task. We could play some other dirty tricks but I am not
sure it is worth it considering this is rather artificial test.
Thanks!
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 14:07 ` Michal Hocko
@ 2017-07-29 4:31 ` Tetsuo Handa
2017-08-01 12:14 ` Michal Hocko
2017-08-01 10:46 ` Tetsuo Handa
1 sibling, 1 reply; 54+ messages in thread
From: Tetsuo Handa @ 2017-07-29 4:31 UTC (permalink / raw)
To: mhocko; +Cc: mjaggi, linux-kernel, linux-mm
Michal Hocko wrote:
> On Fri 28-07-17 22:55:51, Tetsuo Handa wrote:
> > Michal Hocko wrote:
> > > On Fri 28-07-17 22:15:01, Tetsuo Handa wrote:
> > > > task_will_free_mem(current) in out_of_memory() returning false due to
> > > > MMF_OOM_SKIP already set allowed each thread sharing that mm to select a new
> > > > OOM victim. If task_will_free_mem(current) in out_of_memory() did not return
> > > > false, threads sharing MMF_OOM_SKIP mm would not have selected new victims
> > > > to the level where all OOM killable processes are killed and calls panic().
> > >
> > > I am not sure I understand. Do you mean this?
> >
> > Yes.
> >
> > > ---
> > > diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> > > index 9e8b4f030c1c..671e4a4107d0 100644
> > > --- a/mm/oom_kill.c
> > > +++ b/mm/oom_kill.c
> > > @@ -779,13 +779,6 @@ static bool task_will_free_mem(struct task_struct *task)
> > > if (!__task_will_free_mem(task))
> > > return false;
> > >
> > > - /*
> > > - * This task has already been drained by the oom reaper so there are
> > > - * only small chances it will free some more
> > > - */
> > > - if (test_bit(MMF_OOM_SKIP, &mm->flags))
> > > - return false;
> > > -
> > > if (atomic_read(&mm->mm_users) <= 1)
> > > return true;
> > >
> > > If yes I would have to think about this some more because that might
> > > have weird side effects (e.g. oom_victims counting after threads passed
> > > exit_oom_victim).
> >
> > But this check should not be removed unconditionally. We should still return
> > false if returning true was not sufficient to solve the OOM situation, for
> > we need to select next OOM victim in that case.
> >
I think that below one can manage this race condition.
---
include/linux/sched.h | 1 +
mm/oom_kill.c | 21 ++++++++++++++-------
2 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 0db4870..3fccf72 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -652,6 +652,7 @@ struct task_struct {
/* disallow userland-initiated cgroup migration */
unsigned no_cgroup_migration:1;
#endif
+ unsigned oom_kill_free_check_raced:1;
unsigned long atomic_flags; /* Flags requiring atomic access. */
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f0..a093193 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -779,13 +779,6 @@ static bool task_will_free_mem(struct task_struct *task)
if (!__task_will_free_mem(task))
return false;
- /*
- * This task has already been drained by the oom reaper so there are
- * only small chances it will free some more
- */
- if (test_bit(MMF_OOM_SKIP, &mm->flags))
- return false;
-
if (atomic_read(&mm->mm_users) <= 1)
return true;
@@ -806,6 +799,20 @@ static bool task_will_free_mem(struct task_struct *task)
}
rcu_read_unlock();
+ /*
+ * It is possible that current thread fails to try allocation from
+ * memory reserves if the OOM reaper set MMF_OOM_SKIP on this mm before
+ * current thread calls out_of_memory() in order to get TIF_MEMDIE.
+ * In that case, allow current thread to try TIF_MEMDIE allocation
+ * before start selecting next OOM victims.
+ */
+ if (ret && test_bit(MMF_OOM_SKIP, &mm->flags)) {
+ if (task == current && !task->oom_kill_free_check_raced)
+ task->oom_kill_free_check_raced = true;
+ else
+ ret = false;
+ }
+
return ret;
}
--
1.8.3.1
What is "oom_victims counting after threads passed exit_oom_victim" ?
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-28 14:07 ` Michal Hocko
2017-07-29 4:31 ` Tetsuo Handa
@ 2017-08-01 10:46 ` Tetsuo Handa
2017-08-01 11:30 ` Michal Hocko
1 sibling, 1 reply; 54+ messages in thread
From: Tetsuo Handa @ 2017-08-01 10:46 UTC (permalink / raw)
To: mhocko; +Cc: mjaggi, linux-kernel, linux-mm
Michal Hocko wrote:
> > > Is
> > > something other than the LTP test affected to give this more priority?
> > > Do we have other usecases where something mlocks the whole memory?
> >
> > This panic was caused by 50 threads sharing MMF_OOM_SKIP mm exceeding
> > number of OOM killable processes. Whether memory is locked or not isn't
> > important.
>
> You are wrong here I believe. The whole problem is that the OOM victim
> is consuming basically all the memory (that is what the test case
> actually does IIRC) and that memory is mlocked. oom_reaper is much
> faster to evaluate the mm of the victim and bail out sooner than the
> exit path actually manages to tear down the address space. And so we
> have to find other oom victims until we simply kill everything and
> panic.
Again, whether memory is locked or not isn't important. I can easily
reproduce unnecessary OOM victim selection as a local unprivileged user
using below program.
----------
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sched.h>
#include <sys/mman.h>
#define NUMTHREADS 128
#define MMAPSIZE 128 * 1048576
#define STACKSIZE 4096
static int pipe_fd[2] = { EOF, EOF };
static int memory_eater(void *unused)
{
int fd = open("/dev/zero", O_RDONLY);
char *buf = mmap(NULL, MMAPSIZE, PROT_WRITE | PROT_READ,
MAP_ANONYMOUS | MAP_SHARED, EOF, 0);
read(pipe_fd[0], buf, 1);
read(fd, buf, MMAPSIZE);
pause();
return 0;
}
int main(int argc, char *argv[])
{
int i;
char *stack;
if (fork() || fork() || setsid() == EOF || pipe(pipe_fd))
_exit(0);
stack = mmap(NULL, STACKSIZE * NUMTHREADS, PROT_WRITE | PROT_READ,
MAP_ANONYMOUS | MAP_SHARED, EOF, 0);
for (i = 0; i < NUMTHREADS; i++)
if (clone(memory_eater, stack + (i + 1) * STACKSIZE,
CLONE_THREAD | CLONE_SIGHAND | CLONE_VM | CLONE_FS |
CLONE_FILES, NULL) == -1)
break;
sleep(1);
close(pipe_fd[1]);
pause();
return 0;
}
----------
Complete log is at http://I-love.SAKURA.ne.jp/tmp/serial-20170801-2.txt.xz :
----------
[ 237.792768] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 237.795575] [ 451] 0 451 9206 639 21 3 0 0 systemd-journal
[ 237.798515] [ 478] 0 478 11138 740 25 3 0 -1000 systemd-udevd
[ 237.801430] [ 488] 0 488 13856 100 26 3 0 -1000 auditd
[ 237.804212] [ 592] 81 592 6135 119 18 3 0 -900 dbus-daemon
[ 237.807166] [ 668] 0 668 1094 23 8 3 0 0 rngd
[ 237.809927] [ 671] 70 671 7029 75 19 4 0 0 avahi-daemon
[ 237.812809] [ 672] 0 672 53144 402 57 4 0 0 abrtd
[ 237.815611] [ 675] 0 675 26372 246 54 3 0 -1000 sshd
[ 237.818358] [ 679] 0 679 52573 341 54 3 0 0 abrt-watch-log
[ 237.821274] [ 680] 0 680 6050 79 17 3 0 0 systemd-logind
[ 237.824279] [ 683] 0 683 4831 82 16 3 0 0 irqbalance
[ 237.827119] [ 698] 0 698 56014 630 40 4 0 0 rsyslogd
[ 237.829929] [ 715] 70 715 6997 59 18 4 0 0 avahi-daemon
[ 237.832799] [ 832] 0 832 65453 228 44 3 0 0 vmtoolsd
[ 237.835605] [ 852] 0 852 57168 353 58 3 0 0 vmtoolsd
[ 237.838409] [ 909] 0 909 31558 155 20 3 0 0 crond
[ 237.841160] [ 986] 0 986 84330 393 114 4 0 0 nmbd
[ 237.843878] [ 1041] 0 1041 23728 168 51 3 0 0 login
[ 237.846623] [ 2019] 0 2019 22261 252 43 3 0 0 master
[ 237.849307] [ 2034] 0 2034 27511 33 13 3 0 0 agetty
[ 237.851977] [ 2100] 89 2100 22287 250 45 3 0 0 pickup
[ 237.854607] [ 2101] 89 2101 22304 251 45 3 0 0 qmgr
[ 237.857179] [ 2597] 0 2597 102073 568 150 3 0 0 smbd
[ 237.859773] [ 3905] 1000 3905 28885 133 15 4 0 0 bash
[ 237.862337] [ 3952] 0 3952 27511 32 10 3 0 0 agetty
[ 237.864905] [ 4772] 0 4772 102073 568 140 3 0 0 cleanupd
[ 237.867488] [ 4775] 1000 4775 4195473 860912 1814 19 0 0 a.out
[ 237.869991] Out of memory: Kill process 4775 (a.out) score 924 or sacrifice child
[ 240.940617] Killed process 4775 (a.out) total-vm:16781892kB, anon-rss:88kB, file-rss:0kB, shmem-rss:3443560kB
[ 240.962810] oom_reaper: reaped process 4775 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.965513] oom_reaper: reaped process 4863 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.968192] oom_reaper: reaped process 4896 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.970789] oom_reaper: reaped process 4851 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.973352] oom_reaper: reaped process 4788 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.978376] oom_reaper: reaped process 4781 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.981007] oom_reaper: reaped process 4780 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.983527] oom_reaper: reaped process 4903 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.986038] oom_reaper: reaped process 4891 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.988697] oom_reaper: reaped process 4809 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 240.999681] oom_reaper: reaped process 4806 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 241.002165] oom_reaper: reaped process 4783 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 241.004591] oom_reaper: reaped process 4816 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 241.007022] oom_reaper: reaped process 4873 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3445880kB
[ 241.009522] a.out invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[ 241.157201] a.out cpuset=/ mems_allowed=0
[ 241.158983] CPU: 1 PID: 4805 Comm: a.out Not tainted 4.13.0-rc2-next-20170728 #649
(...snipped...)
[ 357.797379] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 357.800119] [ 478] 0 478 11138 740 25 3 0 -1000 systemd-udevd
[ 357.803005] [ 488] 0 488 13856 110 26 3 0 -1000 auditd
[ 357.805800] [ 592] 81 592 6135 119 18 3 0 -900 dbus-daemon
[ 357.808710] [ 668] 0 668 1094 23 8 3 0 0 rngd
[ 357.811437] [ 671] 70 671 7029 75 19 4 0 0 avahi-daemon
[ 357.814308] [ 675] 0 675 26372 246 54 3 0 -1000 sshd
[ 357.817048] [ 679] 0 679 52573 341 54 3 0 0 abrt-watch-log
[ 357.819969] [ 680] 0 680 6050 79 17 3 0 0 systemd-logind
[ 357.822863] [ 683] 0 683 4831 82 16 3 0 0 irqbalance
[ 357.825699] [ 715] 70 715 6997 59 18 4 0 0 avahi-daemon
[ 357.828567] [ 832] 0 832 65453 228 44 3 0 0 vmtoolsd
[ 357.831373] [ 909] 0 909 31558 155 20 3 0 0 crond
[ 357.834160] [ 1041] 0 1041 23728 168 51 3 0 0 login
[ 357.836882] [ 2019] 0 2019 22261 258 43 3 0 0 master
[ 357.839597] [ 2034] 0 2034 27511 33 13 3 0 0 agetty
[ 357.842472] [ 2100] 89 2100 22287 250 45 3 0 0 pickup
[ 357.845358] [ 2101] 89 2101 22304 250 45 3 0 0 qmgr
[ 357.848247] [ 3905] 1000 3905 28885 134 15 4 0 0 bash
[ 357.851074] [ 3952] 0 3952 27511 32 10 3 0 0 agetty
[ 357.853790] [ 5040] 0 5040 9207 365 20 3 0 0 systemd-journal
[ 357.856619] [ 5459] 0 5459 58062 437 42 3 0 0 rsyslogd
[ 357.859296] [ 5465] 1000 5465 4195473 862629 1767 19 0 0 a.out
[ 357.861907] Out of memory: Kill process 5465 (a.out) score 926 or sacrifice child
[ 358.789961] Killed process 5465 (a.out) total-vm:16781892kB, anon-rss:88kB, file-rss:0kB, shmem-rss:3450428kB
[ 358.903547] oom_reaper: reaped process 5465 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.906409] oom_reaper: reaped process 5556 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.909182] oom_reaper: reaped process 5579 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.911930] oom_reaper: reaped process 5500 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.914625] oom_reaper: reaped process 5514 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.917425] oom_reaper: reaped process 5569 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.923629] oom_reaper: reaped process 5533 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.926289] oom_reaper: reaped process 5501 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.928867] oom_reaper: reaped process 5534 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
[ 358.935580] oom_reaper: reaped process 5505 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3451792kB
(...snipped...)
[ 359.863069] Out of memory: Kill process 5459 (rsyslogd) score 0 or sacrifice child
[ 359.868030] Killed process 5459 (rsyslogd) total-vm:232248kB, anon-rss:648kB, file-rss:4kB, shmem-rss:1096kB
(...snipped...)
[ 364.311536] Out of memory: Kill process 679 (abrt-watch-log) score 0 or sacrifice child
[ 364.317978] Killed process 679 (abrt-watch-log) total-vm:210292kB, anon-rss:1360kB, file-rss:4kB, shmem-rss:0kB
(...snipped...)
[ 364.490986] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 364.493818] [ 478] 0 478 11138 740 25 3 0 -1000 systemd-udevd
[ 364.496727] [ 488] 0 488 13856 110 26 3 0 -1000 auditd
[ 364.499669] [ 592] 81 592 6135 119 18 3 0 -900 dbus-daemon
[ 364.502571] [ 668] 0 668 1094 23 8 3 0 0 rngd
[ 364.505303] [ 671] 70 671 7029 75 19 4 0 0 avahi-daemon
[ 364.508067] [ 675] 0 675 26372 246 54 3 0 -1000 sshd
[ 364.510723] [ 680] 0 680 6050 79 17 3 0 0 systemd-logind
[ 364.513735] [ 683] 0 683 4831 81 16 3 0 0 irqbalance
[ 364.516457] [ 715] 70 715 6997 59 18 4 0 0 avahi-daemon
[ 364.519210] [ 832] 0 832 65453 228 44 3 0 0 vmtoolsd
[ 364.521919] [ 909] 0 909 31558 155 20 3 0 0 crond
[ 364.524817] [ 1041] 0 1041 23728 168 51 3 0 0 login
[ 364.527471] [ 2019] 0 2019 22261 258 43 3 0 0 master
[ 364.530118] [ 2034] 0 2034 27511 33 13 3 0 0 agetty
[ 364.532933] [ 2100] 89 2100 22287 250 45 3 0 0 pickup
[ 364.535522] [ 2101] 89 2101 22304 250 45 3 0 0 qmgr
[ 364.538160] [ 3905] 1000 3905 28885 134 15 4 0 0 bash
[ 364.540763] [ 3952] 0 3952 27511 32 10 3 0 0 agetty
[ 364.543292] [ 5040] 0 5040 9207 364 20 3 0 0 systemd-journal
[ 364.546028] [ 5484] 1000 5465 4195473 867043 1767 19 0 0 a.out
[ 364.548515] Out of memory: Kill process 5040 (systemd-journal) score 0 or sacrifice child
[ 364.551666] Killed process 5040 (systemd-journal) total-vm:36828kB, anon-rss:260kB, file-rss:0kB, shmem-rss:1196kB
----------
Usually OOM killable tasks up to number of available CPUs are killed by one OOM killer
invocation, but similar reproducer (CLONE_THREAD | CLONE_SIGHAND removed from above one)
had killed all OOM killable tasks and panic()ed (so far only once).
Complete log is at http://I-love.SAKURA.ne.jp/tmp/serial-20170801.txt.xz :
----------
[ 1301.010587] Mem-Info:
[ 1301.021217] active_anon:3744 inactive_anon:872936 isolated_anon:0
[ 1301.021217] active_file:59 inactive_file:22 isolated_file:37
[ 1301.021217] unevictable:0 dirty:16 writeback:1 unstable:0
[ 1301.021217] slab_reclaimable:0 slab_unreclaimable:17
[ 1301.021217] mapped:853542 shmem:874817 pagetables:2039 bounce:0
[ 1301.021217] free:21371 free_pcp:93 free_cma:0
[ 1301.030807] Node 0 active_anon:14976kB inactive_anon:3491956kB active_file:104kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):92kB mapped:3414188kB dirty:60kB writeback:4kB shmem:3499480kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1301.038255] Node 0 DMA free:14756kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1108kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1301.044649] lowmem_reserve[]: 0 2688 3624 3624
[ 1301.046306] Node 0 DMA32 free:53376kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682284kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5328kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1301.053203] lowmem_reserve[]: 0 0 936 936
[ 1301.054748] Node 0 Normal free:16756kB min:17384kB low:21728kB high:26072kB active_anon:14960kB inactive_anon:808292kB active_file:20kB inactive_file:16kB unevictable:0kB writepending:64kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:4800kB pagetables:2836kB bounce:0kB free_pcp:392kB local_pcp:288kB free_cma:0kB
[ 1301.062749] lowmem_reserve[]: 0 0 0 0
[ 1301.065274] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1301.070034] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1301.074902] Node 0 Normal: 85*4kB (UMEH) 56*8kB (UMEH) 38*16kB (UMEH) 14*32kB (UMEH) 9*64kB (UEH) 12*128kB (UMEH) 26*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 16756kB
[ 1301.080143] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1301.082796] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1301.085387] 874919 total pagecache pages
[ 1301.086962] 0 pages in swap cache
[ 1301.088505] Swap cache stats: add 0, delete 0, find 0/0
[ 1301.090370] Free swap = 0kB
[ 1301.091743] Total swap = 0kB
[ 1301.093140] 1048445 pages RAM
[ 1301.094677] 0 pages HighMem/MovableOnly
[ 1301.096600] 116531 pages reserved
[ 1301.098143] 0 pages hwpoisoned
[ 1301.099690] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1301.103027] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1301.106142] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1301.108737] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1301.111466] [ 6535] 0 6535 9207 423 21 3 0 0 systemd-journal
[ 1301.114370] [ 6536] 0 6536 6050 78 16 3 0 0 systemd-logind
[ 1301.117136] [ 6537] 0 6537 27511 32 13 3 0 0 agetty
[ 1301.119727] [ 6538] 81 6538 6103 78 17 3 0 -900 dbus-daemon
[ 1301.122372] [ 6699] 0 6699 76537 1249 50 3 0 0 rsyslogd
[ 1301.125020] [ 6731] 1000 6731 4195473 848108 1790 19 0 0 a.out
[ 1301.127633] [ 6732] 1000 6732 4195473 848108 1790 19 0 0 a.out
[ 1301.130415] [ 6733] 1000 6733 4195473 848108 1790 19 0 0 a.out
(...snipped...)
[ 1301.386215] [ 6859] 1000 6859 4195473 848108 1790 19 0 0 a.out
[ 1301.388102] [ 6862] 0 6862 27511 32 10 3 0 0 agetty
[ 1301.390015] Out of memory: Kill process 6731 (a.out) score 910 or sacrifice child
[ 1301.392740] Killed process 6731 (a.out) total-vm:16781892kB, anon-rss:84kB, file-rss:0kB, shmem-rss:3392348kB
[ 1301.395605] oom_reaper: reaped process 6731 (a.out), now anon-rss:0kB, file-rss:0kB, shmem-rss:3392400kB
[ 1301.397946] a.out invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[ 1301.400456] a.out cpuset=/ mems_allowed=0
[ 1301.401574] CPU: 2 PID: 6855 Comm: a.out Not tainted 4.13.0-rc2-next-20170728 #649
[ 1301.403392] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1301.407439] Call Trace:
[ 1301.408321] dump_stack+0x67/0x9e
[ 1301.409352] dump_header+0x9d/0x3fa
[ 1301.410411] ? trace_hardirqs_on+0xd/0x10
[ 1301.411589] oom_kill_process+0x226/0x650
[ 1301.412776] out_of_memory+0x136/0x560
[ 1301.413898] ? out_of_memory+0x206/0x560
[ 1301.415062] __alloc_pages_nodemask+0xdce/0xeb0
[ 1301.416373] alloc_pages_vma+0x76/0x1a0
[ 1301.417531] shmem_alloc_page+0x6e/0xa0
[ 1301.418700] ? native_sched_clock+0x36/0xa0
[ 1301.419944] shmem_alloc_and_acct_page+0x6d/0x1f0
[ 1301.421299] shmem_getpage_gfp+0x1b6/0xde0
[ 1301.422536] ? current_kernel_time64+0x80/0xa0
[ 1301.423844] shmem_fault+0x91/0x1f0
[ 1301.424969] ? __lock_acquire+0x506/0x1a90
[ 1301.426227] __do_fault+0x19/0x120
[ 1301.427341] __handle_mm_fault+0x873/0x1160
[ 1301.428639] ? native_sched_clock+0x36/0xa0
[ 1301.429964] handle_mm_fault+0x186/0x360
[ 1301.431223] ? handle_mm_fault+0x44/0x360
[ 1301.432487] __do_page_fault+0x1da/0x510
[ 1301.433730] ? __lock_acquire+0x506/0x1a90
[ 1301.435032] do_page_fault+0x21/0x70
[ 1301.436223] page_fault+0x22/0x30
[ 1301.437366] RIP: 0010:__clear_user+0x3d/0x70
[ 1301.438696] RSP: 0000:ffff8801395ffd70 EFLAGS: 00010206
[ 1301.441909] RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000200
[ 1301.443823] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00007f9bda247000
[ 1301.445863] RBP: ffff8801395ffd80 R08: 0000000000000001 R09: 0000000000000000
[ 1301.447812] R10: 0000000000000001 R11: 0000000000000001 R12: 00007f9bda247000
[ 1301.449757] R13: ffff8801395ffe30 R14: 0000000001992000 R15: 0000000000001000
[ 1301.451723] clear_user+0x34/0x50
[ 1301.452935] iov_iter_zero+0x88/0x380
[ 1301.454219] read_iter_zero+0x38/0xb0
[ 1301.455511] __vfs_read+0xe3/0x140
[ 1301.456752] vfs_read+0x9c/0x150
[ 1301.457958] SyS_read+0x53/0xc0
[ 1301.459151] do_syscall_64+0x61/0x1d0
[ 1301.460451] entry_SYSCALL64_slow_path+0x25/0x25
[ 1301.461940] RIP: 0033:0x7f9ec099dc30
[ 1301.463225] RSP: 002b:00007f9ec0e81fd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 1301.465286] RAX: ffffffffffffffda RBX: 00007f9bd88b5000 RCX: 00007f9ec099dc30
[ 1301.467242] RDX: 0000000008000000 RSI: 00007f9bd88b5000 RDI: 0000000000000061
[ 1301.469187] RBP: 0000000000000061 R08: ffffffffffffffff R09: 0000000000000000
[ 1301.471111] R10: 0000000000000021 R11: 0000000000000246 R12: 00000000004007d7
[ 1301.473044] R13: 00007ffeb6830c90 R14: 0000000000000000 R15: 0000000000000000
[ 1301.475012] Mem-Info:
[ 1302.770130] active_anon:3723 inactive_anon:873018 isolated_anon:0
[ 1302.770130] active_file:3 inactive_file:0 isolated_file:0
[ 1302.770130] unevictable:0 dirty:1 writeback:1 unstable:0
[ 1302.770130] slab_reclaimable:0 slab_unreclaimable:2
[ 1302.770130] mapped:853537 shmem:874899 pagetables:2050 bounce:0
[ 1302.770130] free:21373 free_pcp:35 free_cma:0
[ 1302.787545] Node 0 active_anon:14892kB inactive_anon:3492072kB active_file:8kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3414148kB dirty:4kB writeback:4kB shmem:3499596kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1302.794036] Node 0 DMA free:14752kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1116kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1302.803259] lowmem_reserve[]: 0 2688 3624 3624
[ 1302.804839] Node 0 DMA32 free:53516kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682496kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1302.817760] lowmem_reserve[]: 0 0 936 936
[ 1302.819371] Node 0 Normal free:17224kB min:17384kB low:21728kB high:26072kB active_anon:14876kB inactive_anon:808460kB active_file:108kB inactive_file:0kB unevictable:0kB writepending:8kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:4784kB pagetables:2844kB bounce:0kB free_pcp:140kB local_pcp:4kB free_cma:0kB
[ 1302.829597] lowmem_reserve[]: 0 0 0 0
[ 1302.831194] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1302.835771] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1302.842805] Node 0 Normal: 138*4kB (UMEH) 62*8kB (UMEH) 49*16kB (UMEH) 15*32kB (UMEH) 9*64kB (UEH) 12*128kB (UMEH) 26*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 17224kB
[ 1302.850292] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1302.853053] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1302.860191] 874902 total pagecache pages
[ 1302.862020] 0 pages in swap cache
[ 1302.863688] Swap cache stats: add 0, delete 0, find 0/0
[ 1302.865828] Free swap = 0kB
[ 1302.867400] Total swap = 0kB
[ 1302.868968] 1048445 pages RAM
[ 1303.699719] 0 pages HighMem/MovableOnly
[ 1303.701531] 116531 pages reserved
[ 1303.703194] 0 pages hwpoisoned
[ 1303.704793] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1303.707509] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1303.710392] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1303.713153] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1303.715875] [ 6535] 0 6535 9207 423 21 3 0 0 systemd-journal
[ 1303.719739] [ 6536] 0 6536 6050 78 16 3 0 0 systemd-logind
[ 1303.722604] [ 6537] 0 6537 27511 32 13 3 0 0 agetty
[ 1303.725318] [ 6538] 81 6538 6103 78 17 3 0 -900 dbus-daemon
[ 1303.728122] [ 6699] 0 6699 76537 1231 50 3 0 0 rsyslogd
[ 1303.730883] [ 6732] 1000 6732 4195473 848100 1790 19 0 0 a.out
[ 1303.733591] [ 6733] 1000 6733 4195473 848100 1790 19 0 0 a.out
[ 1303.736272] [ 6734] 1000 6734 4195473 848100 1790 19 0 0 a.out
(...snipped...)
[ 1303.996897] [ 6859] 1000 6859 4195473 848100 1790 19 0 0 a.out
[ 1303.998810] [ 6862] 0 6862 27511 32 10 3 0 0 agetty
[ 1304.000763] Out of memory: Kill process 6699 (rsyslogd) score 1 or sacrifice child
...(noisy "page allocation stalls" lines snipped)...
[ 1378.224508] kworker/u16:2 invoked oom-killer: gfp_mask=0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null), order=2, oom_score_adj=0
[ 1378.224513] kworker/u16:2 cpuset=/ mems_allowed=0
[ 1378.224520] CPU: 3 PID: 6523 Comm: kworker/u16:2 Not tainted 4.13.0-rc2-next-20170728 #649
[ 1378.224521] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1378.224527] Workqueue: events_unbound call_usermodehelper_exec_work
[ 1378.224528] Call Trace:
[ 1378.224535] dump_stack+0x67/0x9e
[ 1378.224539] dump_header+0x9d/0x3fa
[ 1378.224545] ? trace_hardirqs_on+0xd/0x10
[ 1378.224552] oom_kill_process+0x226/0x650
[ 1378.224558] out_of_memory+0x136/0x560
[ 1378.224560] ? out_of_memory+0x206/0x560
[ 1378.224565] __alloc_pages_nodemask+0xdce/0xeb0
[ 1378.224568] ? copy_process.part.39+0x7eb/0x1e30
[ 1378.224585] copy_process.part.39+0x13f/0x1e30
[ 1378.224587] ? __lock_acquire+0x506/0x1a90
[ 1378.224590] ? load_balance+0x1b0/0xaf0
[ 1378.224598] ? umh_complete+0x30/0x30
[ 1378.224604] _do_fork+0xea/0x5f0
[ 1378.224610] ? native_sched_clock+0x36/0xa0
[ 1378.224619] kernel_thread+0x24/0x30
[ 1378.224622] call_usermodehelper_exec_work+0x35/0xc0
[ 1378.224625] process_one_work+0x1d0/0x3e0
[ 1378.224626] ? process_one_work+0x16a/0x3e0
[ 1378.224634] worker_thread+0x48/0x3c0
[ 1378.224642] kthread+0x10d/0x140
[ 1378.224643] ? process_one_work+0x3e0/0x3e0
[ 1378.224646] ? kthread_create_on_node+0x60/0x60
[ 1378.224651] ret_from_fork+0x27/0x40
[ 1378.224662] Mem-Info:
[ 1378.224666] active_anon:3649 inactive_anon:873071 isolated_anon:0
[ 1378.224666] active_file:34 inactive_file:373 isolated_file:0
[ 1378.224666] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1378.224666] slab_reclaimable:0 slab_unreclaimable:0
[ 1378.224666] mapped:852813 shmem:875026 pagetables:2050 bounce:0
[ 1378.224666] free:21442 free_pcp:86 free_cma:0
[ 1378.224668] Node 0 active_anon:14596kB inactive_anon:3492284kB active_file:136kB inactive_file:1492kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3411252kB dirty:0kB writeback:0kB shmem:3500104kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1378.224669] Node 0 DMA free:14752kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1116kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.224672] lowmem_reserve[]: 0 2688 3624 3624
[ 1378.224677] Node 0 DMA32 free:53516kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682496kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.224680] lowmem_reserve[]: 0 0 936 936
[ 1378.224684] Node 0 Normal free:17500kB min:17384kB low:21728kB high:26072kB active_anon:14776kB inactive_anon:808668kB active_file:0kB inactive_file:1940kB unevictable:0kB writepending:0kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:2736kB pagetables:2844kB bounce:0kB free_pcp:344kB local_pcp:224kB free_cma:0kB
[ 1378.224687] lowmem_reserve[]: 0 0 0 0
[ 1378.224691] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1378.224756] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1378.224783] Node 0 Normal: 107*4kB (UEH) 68*8kB (UEH) 50*16kB (UMEH) 61*32kB (UEH) 15*64kB (UEH) 12*128kB (UMEH) 19*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 17228kB
[ 1378.224802] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1378.224803] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1378.224804] 875607 total pagecache pages
[ 1378.224807] 0 pages in swap cache
[ 1378.224808] Swap cache stats: add 0, delete 0, find 0/0
[ 1378.224809] Free swap = 0kB
[ 1378.224809] Total swap = 0kB
[ 1378.224811] 1048445 pages RAM
[ 1378.224812] 0 pages HighMem/MovableOnly
[ 1378.224813] 116531 pages reserved
[ 1378.224814] 0 pages hwpoisoned
[ 1378.224814] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1378.224824] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1378.224827] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1378.224830] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1378.224833] [ 6535] 0 6535 9207 422 21 3 0 0 systemd-journal
[ 1378.224836] [ 6536] 0 6536 6050 78 16 3 0 0 systemd-logind
[ 1378.224839] [ 6537] 0 6537 27511 32 13 3 0 0 agetty
[ 1378.224841] [ 6538] 81 6538 6103 78 17 3 0 -900 dbus-daemon
[ 1378.224930] [ 6855] 1000 6855 4195473 852376 1790 19 0 0 a.out
[ 1378.224935] [ 6862] 0 6862 27511 32 10 3 0 0 agetty
[ 1378.224937] Out of memory: Kill process 6535 (systemd-journal) score 0 or sacrifice child
[ 1378.225006] Killed process 6535 (systemd-journal) total-vm:36828kB, anon-rss:260kB, file-rss:0kB, shmem-rss:1428kB
[ 1378.232809] kworker/u16:2 invoked oom-killer: gfp_mask=0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null), order=2, oom_score_adj=0
[ 1378.232814] kworker/u16:2 cpuset=/ mems_allowed=0
[ 1378.232821] CPU: 3 PID: 6523 Comm: kworker/u16:2 Not tainted 4.13.0-rc2-next-20170728 #649
[ 1378.232822] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1378.232827] Workqueue: events_unbound call_usermodehelper_exec_work
[ 1378.232829] Call Trace:
[ 1378.232835] dump_stack+0x67/0x9e
[ 1378.232839] dump_header+0x9d/0x3fa
[ 1378.232845] ? trace_hardirqs_on+0xd/0x10
[ 1378.232852] oom_kill_process+0x226/0x650
[ 1378.232859] out_of_memory+0x136/0x560
[ 1378.232860] ? out_of_memory+0x206/0x560
[ 1378.232866] __alloc_pages_nodemask+0xdce/0xeb0
[ 1378.232869] ? copy_process.part.39+0x7eb/0x1e30
[ 1378.232885] copy_process.part.39+0x13f/0x1e30
[ 1378.232888] ? __lock_acquire+0x506/0x1a90
[ 1378.232890] ? load_balance+0x1b0/0xaf0
[ 1378.232898] ? umh_complete+0x30/0x30
[ 1378.232904] _do_fork+0xea/0x5f0
[ 1378.232910] ? native_sched_clock+0x36/0xa0
[ 1378.232919] kernel_thread+0x24/0x30
[ 1378.232922] call_usermodehelper_exec_work+0x35/0xc0
[ 1378.232925] process_one_work+0x1d0/0x3e0
[ 1378.232926] ? process_one_work+0x16a/0x3e0
[ 1378.232934] worker_thread+0x48/0x3c0
[ 1378.232943] kthread+0x10d/0x140
[ 1378.232944] ? process_one_work+0x3e0/0x3e0
[ 1378.232946] ? kthread_create_on_node+0x60/0x60
[ 1378.232951] ret_from_fork+0x27/0x40
[ 1378.232962] Mem-Info:
[ 1378.232965] active_anon:3723 inactive_anon:872923 isolated_anon:0
[ 1378.232965] active_file:34 inactive_file:262 isolated_file:0
[ 1378.232965] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1378.232965] slab_reclaimable:0 slab_unreclaimable:0
[ 1378.232965] mapped:852480 shmem:875026 pagetables:2050 bounce:0
[ 1378.232965] free:21442 free_pcp:442 free_cma:0
[ 1378.232968] Node 0 active_anon:14892kB inactive_anon:3491692kB active_file:136kB inactive_file:1048kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3409920kB dirty:0kB writeback:0kB shmem:3500104kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1378.232969] Node 0 DMA free:14752kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1116kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.232972] lowmem_reserve[]: 0 2688 3624 3624
[ 1378.232977] Node 0 DMA32 free:53516kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682496kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.232980] lowmem_reserve[]: 0 0 936 936
[ 1378.232984] Node 0 Normal free:17500kB min:17384kB low:21728kB high:26072kB active_anon:14776kB inactive_anon:808164kB active_file:0kB inactive_file:1436kB unevictable:0kB writepending:0kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:2736kB pagetables:2844kB bounce:0kB free_pcp:1768kB local_pcp:676kB free_cma:0kB
[ 1378.232987] lowmem_reserve[]: 0 0 0 0
[ 1378.232991] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1378.233010] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1378.233028] Node 0 Normal: 76*4kB (UEH) 68*8kB (UEH) 53*16kB (UMEH) 61*32kB (UEH) 15*64kB (UEH) 12*128kB (UMEH) 19*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 17152kB
[ 1378.233047] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1378.233048] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1378.233049] 875385 total pagecache pages
[ 1378.233052] 0 pages in swap cache
[ 1378.233053] Swap cache stats: add 0, delete 0, find 0/0
[ 1378.233054] Free swap = 0kB
[ 1378.233055] Total swap = 0kB
[ 1378.233056] 1048445 pages RAM
[ 1378.233057] 0 pages HighMem/MovableOnly
[ 1378.233058] 116531 pages reserved
[ 1378.233058] 0 pages hwpoisoned
[ 1378.233059] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1378.233068] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1378.233072] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1378.233075] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1378.233079] [ 6536] 0 6536 6050 78 16 3 0 0 systemd-logind
[ 1378.233082] [ 6537] 0 6537 27511 32 13 3 0 0 agetty
[ 1378.233085] [ 6538] 81 6538 6103 78 17 3 0 -900 dbus-daemon
[ 1378.233172] [ 6855] 1000 6855 4195473 852376 1790 19 0 0 a.out
[ 1378.233177] [ 6862] 0 6862 27511 32 10 3 0 0 agetty
[ 1378.233179] Out of memory: Kill process 6536 (systemd-logind) score 0 or sacrifice child
[ 1378.233276] Killed process 6536 (systemd-logind) total-vm:24200kB, anon-rss:308kB, file-rss:4kB, shmem-rss:0kB
[ 1378.236782] kworker/u16:1 invoked oom-killer: gfp_mask=0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null), order=2, oom_score_adj=0
[ 1378.236786] kworker/u16:1 cpuset=/ mems_allowed=0
[ 1378.236792] CPU: 0 PID: 6520 Comm: kworker/u16:1 Not tainted 4.13.0-rc2-next-20170728 #649
[ 1378.236793] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1378.236797] Workqueue: events_unbound call_usermodehelper_exec_work
[ 1378.236798] Call Trace:
[ 1378.236803] dump_stack+0x67/0x9e
[ 1378.236807] dump_header+0x9d/0x3fa
[ 1378.236812] ? trace_hardirqs_on+0xd/0x10
[ 1378.236819] oom_kill_process+0x226/0x650
[ 1378.236825] out_of_memory+0x136/0x560
[ 1378.236827] ? out_of_memory+0x206/0x560
[ 1378.236832] __alloc_pages_nodemask+0xdce/0xeb0
[ 1378.236834] ? copy_process.part.39+0x7eb/0x1e30
[ 1378.236851] copy_process.part.39+0x13f/0x1e30
[ 1378.236853] ? __lock_acquire+0x506/0x1a90
[ 1378.236861] ? umh_complete+0x30/0x30
[ 1378.236868] _do_fork+0xea/0x5f0
[ 1378.236873] ? native_sched_clock+0x36/0xa0
[ 1378.236882] kernel_thread+0x24/0x30
[ 1378.236885] call_usermodehelper_exec_work+0x35/0xc0
[ 1378.236888] process_one_work+0x1d0/0x3e0
[ 1378.236889] ? process_one_work+0x16a/0x3e0
[ 1378.236897] worker_thread+0x48/0x3c0
[ 1378.236905] kthread+0x10d/0x140
[ 1378.236906] ? process_one_work+0x3e0/0x3e0
[ 1378.236909] ? kthread_create_on_node+0x60/0x60
[ 1378.236913] ret_from_fork+0x27/0x40
[ 1378.236924] Mem-Info:
[ 1378.236927] active_anon:3649 inactive_anon:872923 isolated_anon:0
[ 1378.236927] active_file:34 inactive_file:55 isolated_file:0
[ 1378.236927] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1378.236927] slab_reclaimable:0 slab_unreclaimable:0
[ 1378.236927] mapped:852480 shmem:875026 pagetables:2050 bounce:0
[ 1378.236927] free:21694 free_pcp:471 free_cma:0
[ 1378.236929] Node 0 active_anon:14596kB inactive_anon:3491692kB active_file:136kB inactive_file:220kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3409920kB dirty:0kB writeback:0kB shmem:3500104kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1378.236930] Node 0 DMA free:14752kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1116kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.236933] lowmem_reserve[]: 0 2688 3624 3624
[ 1378.236938] Node 0 DMA32 free:53516kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682496kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.236941] lowmem_reserve[]: 0 0 936 936
[ 1378.236945] Node 0 Normal free:18508kB min:17384kB low:21728kB high:26072kB active_anon:14776kB inactive_anon:808164kB active_file:0kB inactive_file:384kB unevictable:0kB writepending:0kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:2736kB pagetables:2844kB bounce:0kB free_pcp:1884kB local_pcp:716kB free_cma:0kB
[ 1378.236949] lowmem_reserve[]: 0 0 0 0
[ 1378.236953] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1378.236972] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1378.236990] Node 0 Normal: 216*4kB (UMEH) 89*8kB (UMEH) 71*16kB (UMEH) 66*32kB (UMEH) 16*64kB (UMEH) 12*128kB (UMEH) 19*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 18392kB
[ 1378.237008] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1378.237009] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1378.237010] 875163 total pagecache pages
[ 1378.237013] 0 pages in swap cache
[ 1378.237014] Swap cache stats: add 0, delete 0, find 0/0
[ 1378.237015] Free swap = 0kB
[ 1378.237016] Total swap = 0kB
[ 1378.237017] 1048445 pages RAM
[ 1378.237018] 0 pages HighMem/MovableOnly
[ 1378.237018] 116531 pages reserved
[ 1378.237019] 0 pages hwpoisoned
[ 1378.237020] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1378.237029] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1378.237031] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1378.237034] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1378.237039] [ 6537] 0 6537 27511 32 13 3 0 0 agetty
[ 1378.237041] [ 6538] 81 6538 6103 78 17 3 0 -900 dbus-daemon
[ 1378.237127] [ 6855] 1000 6855 4195473 852376 1790 19 0 0 a.out
[ 1378.237132] [ 6862] 0 6862 27511 32 10 3 0 0 agetty
[ 1378.237134] Out of memory: Kill process 6537 (agetty) score 0 or sacrifice child
[ 1378.237146] Killed process 6537 (agetty) total-vm:110044kB, anon-rss:124kB, file-rss:4kB, shmem-rss:0kB
[ 1378.247858] kworker/u16:2 invoked oom-killer: gfp_mask=0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null), order=2, oom_score_adj=0
[ 1378.247864] kworker/u16:2 cpuset=/ mems_allowed=0
[ 1378.247871] CPU: 3 PID: 6523 Comm: kworker/u16:2 Not tainted 4.13.0-rc2-next-20170728 #649
[ 1378.247872] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1378.247879] Workqueue: events_unbound call_usermodehelper_exec_work
[ 1378.247881] Call Trace:
[ 1378.247888] dump_stack+0x67/0x9e
[ 1378.247913] dump_header+0x9d/0x3fa
[ 1378.247924] ? trace_hardirqs_on+0xd/0x10
[ 1378.247931] oom_kill_process+0x226/0x650
[ 1378.247938] out_of_memory+0x136/0x560
[ 1378.247940] ? out_of_memory+0x206/0x560
[ 1378.247945] __alloc_pages_nodemask+0xdce/0xeb0
[ 1378.247948] ? copy_process.part.39+0x7eb/0x1e30
[ 1378.247966] copy_process.part.39+0x13f/0x1e30
[ 1378.247968] ? __lock_acquire+0x506/0x1a90
[ 1378.247971] ? load_balance+0x1b0/0xaf0
[ 1378.248035] ? umh_complete+0x30/0x30
[ 1378.248054] _do_fork+0xea/0x5f0
[ 1378.248060] ? native_sched_clock+0x36/0xa0
[ 1378.248187] kernel_thread+0x24/0x30
[ 1378.248191] call_usermodehelper_exec_work+0x35/0xc0
[ 1378.248194] process_one_work+0x1d0/0x3e0
[ 1378.248225] ? process_one_work+0x16a/0x3e0
[ 1378.248234] worker_thread+0x48/0x3c0
[ 1378.248265] kthread+0x10d/0x140
[ 1378.248267] ? process_one_work+0x3e0/0x3e0
[ 1378.248269] ? kthread_create_on_node+0x60/0x60
[ 1378.248274] ret_from_fork+0x27/0x40
[ 1378.248301] Mem-Info:
[ 1378.248304] active_anon:3649 inactive_anon:872923 isolated_anon:0
[ 1378.248304] active_file:34 inactive_file:18 isolated_file:0
[ 1378.248304] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1378.248304] slab_reclaimable:0 slab_unreclaimable:0
[ 1378.248304] mapped:852443 shmem:875026 pagetables:2050 bounce:0
[ 1378.248304] free:21820 free_pcp:444 free_cma:0
[ 1378.248307] Node 0 active_anon:14596kB inactive_anon:3491692kB active_file:136kB inactive_file:72kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3409772kB dirty:0kB writeback:0kB shmem:3500104kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1378.248308] Node 0 DMA free:14752kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1116kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.248312] lowmem_reserve[]: 0 2688 3624 3624
[ 1378.248316] Node 0 DMA32 free:53516kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682496kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.248319] lowmem_reserve[]: 0 0 936 936
[ 1378.248324] Node 0 Normal free:19012kB min:17384kB low:21728kB high:26072kB active_anon:14776kB inactive_anon:808164kB active_file:0kB inactive_file:384kB unevictable:0kB writepending:0kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:2736kB pagetables:2844kB bounce:0kB free_pcp:1776kB local_pcp:692kB free_cma:0kB
[ 1378.248326] lowmem_reserve[]: 0 0 0 0
[ 1378.248331] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1378.248350] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1378.248368] Node 0 Normal: 285*4kB (UMEH) 92*8kB (UMEH) 77*16kB (UMEH) 69*32kB (UMEH) 18*64kB (UMEH) 12*128kB (UMEH) 19*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 19012kB
[ 1378.248387] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1378.248388] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1378.248389] 875052 total pagecache pages
[ 1378.248392] 0 pages in swap cache
[ 1378.248393] Swap cache stats: add 0, delete 0, find 0/0
[ 1378.248394] Free swap = 0kB
[ 1378.248395] Total swap = 0kB
[ 1378.248396] 1048445 pages RAM
[ 1378.248397] 0 pages HighMem/MovableOnly
[ 1378.248398] 116531 pages reserved
[ 1378.248399] 0 pages hwpoisoned
[ 1378.248400] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1378.248412] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1378.248416] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1378.248419] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1378.248426] [ 6538] 81 6538 6103 78 17 3 0 -900 dbus-daemon
[ 1378.248527] [ 6855] 1000 6855 4195473 852376 1790 19 0 0 a.out
[ 1378.248533] [ 6862] 0 6862 27511 32 10 3 0 0 agetty
[ 1378.248535] Out of memory: Kill process 6862 (agetty) score 0 or sacrifice child
[ 1378.248566] Killed process 6862 (agetty) total-vm:110044kB, anon-rss:124kB, file-rss:4kB, shmem-rss:0kB
[ 1378.252379] kworker/u16:26 invoked oom-killer: gfp_mask=0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null), order=2, oom_score_adj=0
[ 1378.252384] kworker/u16:26 cpuset=/ mems_allowed=0
[ 1378.252390] CPU: 0 PID: 353 Comm: kworker/u16:26 Not tainted 4.13.0-rc2-next-20170728 #649
[ 1378.252391] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1378.252396] Workqueue: events_unbound call_usermodehelper_exec_work
[ 1378.252397] Call Trace:
[ 1378.252402] dump_stack+0x67/0x9e
[ 1378.252407] dump_header+0x9d/0x3fa
[ 1378.252413] ? trace_hardirqs_on+0xd/0x10
[ 1378.252420] oom_kill_process+0x226/0x650
[ 1378.252426] out_of_memory+0x136/0x560
[ 1378.252428] ? out_of_memory+0x206/0x560
[ 1378.252433] __alloc_pages_nodemask+0xdce/0xeb0
[ 1378.252436] ? copy_process.part.39+0x7eb/0x1e30
[ 1378.252453] copy_process.part.39+0x13f/0x1e30
[ 1378.252455] ? __lock_acquire+0x506/0x1a90
[ 1378.252458] ? load_balance+0x1b0/0xaf0
[ 1378.252465] ? umh_complete+0x30/0x30
[ 1378.252472] _do_fork+0xea/0x5f0
[ 1378.252477] ? native_sched_clock+0x36/0xa0
[ 1378.252487] kernel_thread+0x24/0x30
[ 1378.252490] call_usermodehelper_exec_work+0x35/0xc0
[ 1378.252493] process_one_work+0x1d0/0x3e0
[ 1378.252494] ? process_one_work+0x16a/0x3e0
[ 1378.252502] worker_thread+0x48/0x3c0
[ 1378.252510] kthread+0x10d/0x140
[ 1378.252511] ? process_one_work+0x3e0/0x3e0
[ 1378.252514] ? kthread_create_on_node+0x60/0x60
[ 1378.252519] ret_from_fork+0x27/0x40
[ 1378.252530] Mem-Info:
[ 1378.252533] active_anon:3649 inactive_anon:872923 isolated_anon:0
[ 1378.252533] active_file:34 inactive_file:0 isolated_file:0
[ 1378.252533] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1378.252533] slab_reclaimable:0 slab_unreclaimable:0
[ 1378.252533] mapped:852443 shmem:875026 pagetables:2050 bounce:0
[ 1378.252533] free:21820 free_pcp:501 free_cma:0
[ 1378.252535] Node 0 active_anon:14596kB inactive_anon:3491692kB active_file:136kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3409772kB dirty:0kB writeback:0kB shmem:3500104kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1378.252536] Node 0 DMA free:14752kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1116kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.252539] lowmem_reserve[]: 0 2688 3624 3624
[ 1378.252544] Node 0 DMA32 free:53516kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682496kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1378.252547] lowmem_reserve[]: 0 0 936 936
[ 1378.252551] Node 0 Normal free:19012kB min:17384kB low:21728kB high:26072kB active_anon:14776kB inactive_anon:808164kB active_file:0kB inactive_file:384kB unevictable:0kB writepending:0kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:2736kB pagetables:2844kB bounce:0kB free_pcp:2004kB local_pcp:668kB free_cma:0kB
[ 1378.252554] lowmem_reserve[]: 0 0 0 0
[ 1378.252559] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1378.252578] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1378.252596] Node 0 Normal: 271*4kB (UMEH) 97*8kB (UMEH) 79*16kB (UMEH) 69*32kB (UMEH) 18*64kB (UMEH) 12*128kB (UMEH) 19*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 19028kB
[ 1378.252614] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1378.252616] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1378.252617] 875015 total pagecache pages
[ 1378.252620] 0 pages in swap cache
[ 1378.252621] Swap cache stats: add 0, delete 0, find 0/0
[ 1378.252622] Free swap = 0kB
[ 1378.252622] Total swap = 0kB
[ 1378.252624] 1048445 pages RAM
[ 1378.252625] 0 pages HighMem/MovableOnly
[ 1378.252626] 116531 pages reserved
[ 1378.252627] 0 pages hwpoisoned
[ 1378.252628] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1378.252634] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1378.252638] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1378.252641] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1378.252646] [ 6538] 81 6538 6103 78 17 3 0 -900 dbus-daemon
[ 1378.252859] [ 6855] 1000 6855 4195473 852376 1790 19 0 0 a.out
[ 1378.252866] Out of memory: Kill process 6538 (dbus-daemon) score 0 or sacrifice child
[ 1378.252879] Killed process 6538 (dbus-daemon) total-vm:24412kB, anon-rss:312kB, file-rss:0kB, shmem-rss:0kB
[ 1379.068645] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1379.068647] 875105 total pagecache pages
[ 1379.068650] 0 pages in swap cache
[ 1379.068651] Swap cache stats: add 0, delete 0, find 0/0
[ 1379.068652] Free swap = 0kB
[ 1379.068653] Total swap = 0kB
[ 1379.068655] 1048445 pages RAM
[ 1379.068656] 0 pages HighMem/MovableOnly
[ 1379.068656] 116531 pages reserved
[ 1379.068657] 0 pages hwpoisoned
[ 1379.101641] kworker/u16:29 invoked oom-killer: gfp_mask=0x17080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null), order=2, oom_score_adj=0
[ 1379.105795] kworker/u16:29 cpuset=/ mems_allowed=0
[ 1379.107976] CPU: 0 PID: 356 Comm: kworker/u16:29 Not tainted 4.13.0-rc2-next-20170728 #649
[ 1379.110761] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1379.114084] Workqueue: events_unbound call_usermodehelper_exec_work
[ 1379.116440] Call Trace:
[ 1379.117938] dump_stack+0x67/0x9e
[ 1379.119603] dump_header+0x9d/0x3fa
[ 1379.121301] out_of_memory+0x470/0x560
[ 1379.122979] ? out_of_memory+0x206/0x560
[ 1379.124689] __alloc_pages_nodemask+0xdce/0xeb0
[ 1379.126519] ? copy_process.part.39+0x7eb/0x1e30
[ 1379.128346] copy_process.part.39+0x13f/0x1e30
[ 1379.130084] ? __lock_acquire+0x506/0x1a90
[ 1379.131719] ? umh_complete+0x30/0x30
[ 1379.133265] _do_fork+0xea/0x5f0
[ 1379.134689] ? native_sched_clock+0x36/0xa0
[ 1379.136239] kernel_thread+0x24/0x30
[ 1379.137628] call_usermodehelper_exec_work+0x35/0xc0
[ 1379.139364] process_one_work+0x1d0/0x3e0
[ 1379.140857] ? process_one_work+0x16a/0x3e0
[ 1379.142361] worker_thread+0x48/0x3c0
[ 1379.143813] kthread+0x10d/0x140
[ 1379.145138] ? process_one_work+0x3e0/0x3e0
[ 1379.146657] ? kthread_create_on_node+0x60/0x60
[ 1379.148252] ret_from_fork+0x27/0x40
[ 1379.149659] Mem-Info:
[ 1379.151818] active_anon:3493 inactive_anon:872905 isolated_anon:0
[ 1379.151818] active_file:44 inactive_file:82 isolated_file:0
[ 1379.151818] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1379.151818] slab_reclaimable:0 slab_unreclaimable:0
[ 1379.151818] mapped:532931 shmem:875026 pagetables:1923 bounce:0
[ 1379.151818] free:21712 free_pcp:565 free_cma:0
[ 1379.163532] Node 0 active_anon:13972kB inactive_anon:3491620kB active_file:176kB inactive_file:204kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1943792kB dirty:0kB writeback:0kB shmem:3500104kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1379.171228] Node 0 DMA free:14752kB min:288kB low:360kB high:432kB active_anon:0kB inactive_anon:1116kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1379.178826] lowmem_reserve[]: 0 2688 3624 3624
[ 1379.180764] Node 0 DMA32 free:53516kB min:49908kB low:62384kB high:74860kB active_anon:16kB inactive_anon:2682496kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129216kB managed:2752884kB mlocked:0kB kernel_stack:0kB pagetables:5352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1379.188610] lowmem_reserve[]: 0 0 936 936
[ 1379.191335] Node 0 Normal free:19124kB min:17384kB low:21728kB high:26072kB active_anon:13956kB inactive_anon:808008kB active_file:100kB inactive_file:516kB unevictable:0kB writepending:0kB present:1048576kB managed:958868kB mlocked:0kB kernel_stack:2640kB pagetables:2336kB bounce:0kB free_pcp:2156kB local_pcp:540kB free_cma:0kB
[ 1379.200442] lowmem_reserve[]: 0 0 0 0
[ 1379.202105] Node 0 DMA: 2*4kB (UM) 1*8kB (U) 1*16kB (U) 2*32kB (UM) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (U) 0*2048kB 3*4096kB (ME) = 14752kB
[ 1379.207076] Node 0 DMA32: 9*4kB (UM) 7*8kB (UM) 7*16kB (UME) 6*32kB (UME) 4*64kB (UME) 1*128kB (M) 2*256kB (ME) 0*512kB 1*1024kB (U) 1*2048kB (E) 12*4096kB (UM) = 53516kB
[ 1379.212387] Node 0 Normal: 72*4kB (UMEH) 108*8kB (UMEH) 97*16kB (UMEH) 83*32kB (UMEH) 21*64kB (UMEH) 12*128kB (UMEH) 19*256kB (UMEH) 10*512kB (UME) 1*1024kB (H) 0*2048kB 0*4096kB = 19248kB
[ 1379.217936] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1379.220795] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1379.223635] 875203 total pagecache pages
[ 1379.225549] 0 pages in swap cache
[ 1379.228278] Swap cache stats: add 0, delete 0, find 0/0
[ 1379.230716] Free swap = 0kB
[ 1379.232374] Total swap = 0kB
[ 1379.233975] 1048445 pages RAM
[ 1379.235603] 0 pages HighMem/MovableOnly
[ 1379.237382] 116531 pages reserved
[ 1379.239170] 0 pages hwpoisoned
[ 1379.240891] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 1379.243705] [ 476] 0 476 10952 553 24 3 0 -1000 systemd-udevd
[ 1379.246883] [ 486] 0 486 13856 110 26 3 0 -1000 auditd
[ 1379.249890] [ 660] 0 660 26372 247 55 3 0 -1000 sshd
[ 1379.253002] Kernel panic - not syncing: Out of memory and no killable processes...
[ 1379.253002]
[ 1379.256943] CPU: 0 PID: 356 Comm: kworker/u16:29 Not tainted 4.13.0-rc2-next-20170728 #649
[ 1379.259739] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 1379.263081] Workqueue: events_unbound call_usermodehelper_exec_work
[ 1379.265496] Call Trace:
[ 1379.267085] dump_stack+0x67/0x9e
[ 1379.268897] panic+0xe5/0x23f
[ 1379.270588] out_of_memory+0x47e/0x560
[ 1379.272475] ? out_of_memory+0x206/0x560
[ 1379.274397] __alloc_pages_nodemask+0xdce/0xeb0
[ 1379.276422] ? copy_process.part.39+0x7eb/0x1e30
[ 1379.278475] copy_process.part.39+0x13f/0x1e30
[ 1379.280491] ? __lock_acquire+0x506/0x1a90
[ 1379.282561] ? umh_complete+0x30/0x30
[ 1379.284514] _do_fork+0xea/0x5f0
[ 1379.286341] ? native_sched_clock+0x36/0xa0
[ 1379.288370] kernel_thread+0x24/0x30
----------
>
> > If a multi-threaded process which consumes little memory was
> > selected as an OOM victim (and reaped by the OOM reaper and MMF_OOM_SKIP
> > was set immediately), it might be still possible to select next OOM victims
> > needlessly.
>
> This would be true if the address space itself only contained a little
> amount of memory and the large part of the memory was in page tables or
> other resources which oom_reaper cannot work with. This is not a usual
> case though.
mlock()ing whole memory needs CAP_IPC_LOCK, but consuming whole memory as
MAP_SHARED does not need CAP_IPC_LOCK. And I think we can relax MMF_OOM_SKIP
test in task_will_free_mem() to ignore MMF_OOM_SKIP for once, for "mm, oom:
do not grant oom victims full memory reserves access" might be too large change
for older kernels which next version of LTS distributions would choose.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-08-01 10:46 ` Tetsuo Handa
@ 2017-08-01 11:30 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-08-01 11:30 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: mjaggi, linux-kernel, linux-mm
On Tue 01-08-17 19:46:38, Tetsuo Handa wrote:
> Michal Hocko wrote:
> > > > Is
> > > > something other than the LTP test affected to give this more priority?
> > > > Do we have other usecases where something mlocks the whole memory?
> > >
> > > This panic was caused by 50 threads sharing MMF_OOM_SKIP mm exceeding
> > > number of OOM killable processes. Whether memory is locked or not isn't
> > > important.
> >
> > You are wrong here I believe. The whole problem is that the OOM victim
> > is consuming basically all the memory (that is what the test case
> > actually does IIRC) and that memory is mlocked. oom_reaper is much
> > faster to evaluate the mm of the victim and bail out sooner than the
> > exit path actually manages to tear down the address space. And so we
> > have to find other oom victims until we simply kill everything and
> > panic.
>
> Again, whether memory is locked or not isn't important. I can easily
> reproduce unnecessary OOM victim selection as a local unprivileged user
> using below program.
>
> ----------
> #define _GNU_SOURCE
> #include <stdio.h>
> #include <stdlib.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <fcntl.h>
> #include <unistd.h>
> #include <sched.h>
> #include <sys/mman.h>
>
> #define NUMTHREADS 128
> #define MMAPSIZE 128 * 1048576
> #define STACKSIZE 4096
> static int pipe_fd[2] = { EOF, EOF };
> static int memory_eater(void *unused)
> {
> int fd = open("/dev/zero", O_RDONLY);
> char *buf = mmap(NULL, MMAPSIZE, PROT_WRITE | PROT_READ,
> MAP_ANONYMOUS | MAP_SHARED, EOF, 0);
> read(pipe_fd[0], buf, 1);
> read(fd, buf, MMAPSIZE);
> pause();
> return 0;
> }
> int main(int argc, char *argv[])
> {
> int i;
> char *stack;
> if (fork() || fork() || setsid() == EOF || pipe(pipe_fd))
> _exit(0);
> stack = mmap(NULL, STACKSIZE * NUMTHREADS, PROT_WRITE | PROT_READ,
> MAP_ANONYMOUS | MAP_SHARED, EOF, 0);
> for (i = 0; i < NUMTHREADS; i++)
> if (clone(memory_eater, stack + (i + 1) * STACKSIZE,
> CLONE_THREAD | CLONE_SIGHAND | CLONE_VM | CLONE_FS |
> CLONE_FILES, NULL) == -1)
> break;
> sleep(1);
> close(pipe_fd[1]);
> pause();
> return 0;
> }
This is a clear DoS. There is sadly^Wsimply no implicit limit for the
amount of shared anonymous memory. This is very close to consuming shmem
via fs interface except the fs interface has an upper bound for the size.
I do not thing this is anything new. If you are creative enough you can
DoS the system the same way regardless of the oom reaper by passing
shmem fds around AFAICS...
[...]
> > > If a multi-threaded process which consumes little memory was
> > > selected as an OOM victim (and reaped by the OOM reaper and MMF_OOM_SKIP
> > > was set immediately), it might be still possible to select next OOM victims
> > > needlessly.
> >
> > This would be true if the address space itself only contained a little
> > amount of memory and the large part of the memory was in page tables or
> > other resources which oom_reaper cannot work with. This is not a usual
> > case though.
>
> mlock()ing whole memory needs CAP_IPC_LOCK, but consuming whole memory as
> MAP_SHARED does not need CAP_IPC_LOCK. And I think we can relax MMF_OOM_SKIP
> test in task_will_free_mem() to ignore MMF_OOM_SKIP for once
As I've said it is not that simple. I will comment on your other email.
> for "mm, oom: do not grant oom victims full memory reserves access"
> might be too large change for older kernels which next version of LTS
> distributions would choose.
While this is annoying I do not think this is something new. If you have
an untrusted user on the system you better contain it (you can use memcg
for example).
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-07-29 4:31 ` Tetsuo Handa
@ 2017-08-01 12:14 ` Michal Hocko
2017-08-01 14:16 ` Tetsuo Handa
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-08-01 12:14 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: mjaggi, linux-kernel, linux-mm
On Sat 29-07-17 13:31:44, Tetsuo Handa wrote:
[...]
> @@ -806,6 +799,20 @@ static bool task_will_free_mem(struct task_struct *task)
> }
> rcu_read_unlock();
>
> + /*
> + * It is possible that current thread fails to try allocation from
> + * memory reserves if the OOM reaper set MMF_OOM_SKIP on this mm before
> + * current thread calls out_of_memory() in order to get TIF_MEMDIE.
> + * In that case, allow current thread to try TIF_MEMDIE allocation
> + * before start selecting next OOM victims.
> + */
> + if (ret && test_bit(MMF_OOM_SKIP, &mm->flags)) {
> + if (task == current && !task->oom_kill_free_check_raced)
> + task->oom_kill_free_check_raced = true;
> + else
> + ret = false;
> + }
> +
> return ret;
> }
I was going to argue that this will not work because we could mark a
former OOM victim again after it passed exit_oom_victim but this seems
impossible because task_will_free_mem checks task->mm and that will be
NULL by that time. This is still an ugly hack and it doesn't provide any
additional guarantee. Once we merge [1] then the oom victim wouldn't
need to get TIF_MEMDIE to access memory reserves.
[1] http://lkml.kernel.org/r/20170727090357.3205-2-mhocko@kernel.org
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-08-01 12:14 ` Michal Hocko
@ 2017-08-01 14:16 ` Tetsuo Handa
2017-08-01 14:47 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Tetsuo Handa @ 2017-08-01 14:16 UTC (permalink / raw)
To: mhocko; +Cc: mjaggi, linux-kernel, linux-mm
Michal Hocko wrote:
> Once we merge [1] then the oom victim wouldn't
> need to get TIF_MEMDIE to access memory reserves.
>
> [1] http://lkml.kernel.org/r/20170727090357.3205-2-mhocko@kernel.org
False. We are not setting oom_mm to all thread groups (!CLONE_THREAD) sharing
that mm (CLONE_VM). Thus, one thread from each thread group sharing that mm
will have to call out_of_memory() in order to set oom_mm, and they will find
task_will_free_mem() returning false due to MMF_OOM_SKIP already set, and
after all goes to next OOM victim selection.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: Possible race condition in oom-killer
2017-08-01 14:16 ` Tetsuo Handa
@ 2017-08-01 14:47 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-08-01 14:47 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: mjaggi, linux-kernel, linux-mm
On Tue 01-08-17 23:16:13, Tetsuo Handa wrote:
> Michal Hocko wrote:
> > Once we merge [1] then the oom victim wouldn't
> > need to get TIF_MEMDIE to access memory reserves.
> >
> > [1] http://lkml.kernel.org/r/20170727090357.3205-2-mhocko@kernel.org
>
> False. We are not setting oom_mm to all thread groups (!CLONE_THREAD) sharing
> that mm (CLONE_VM). Thus, one thread from each thread group sharing that mm
> will have to call out_of_memory() in order to set oom_mm, and they will find
> task_will_free_mem() returning false due to MMF_OOM_SKIP already set, and
> after all goes to next OOM victim selection.
Once the patch is merged we can mark_oom_victim all of them as well.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-26 16:29 ` Andrea Arcangeli
2017-07-26 16:43 ` Andrea Arcangeli
2017-07-27 6:50 ` Michal Hocko
@ 2017-08-15 0:20 ` David Rientjes
2 siblings, 0 replies; 54+ messages in thread
From: David Rientjes @ 2017-08-15 0:20 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Michal Hocko, Kirill A. Shutemov, Andrew Morton, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Wed, 26 Jul 2017, Andrea Arcangeli wrote:
> From 3d9001490ee1a71f39c7bfaf19e96821f9d3ff16 Mon Sep 17 00:00:00 2001
> From: Andrea Arcangeli <aarcange@redhat.com>
> Date: Tue, 25 Jul 2017 20:02:27 +0200
> Subject: [PATCH 1/1] mm: oom: let oom_reap_task and exit_mmap to run
> concurrently
>
> This is purely required because exit_aio() may block and exit_mmap() may
> never start, if the oom_reap_task cannot start running on a mm with
> mm_users == 0.
>
> At the same time if the OOM reaper doesn't wait at all for the memory
> of the current OOM candidate to be freed by exit_mmap->unmap_vmas, it
> would generate a spurious OOM kill.
>
> If it wasn't because of the exit_aio or similar blocking functions in
> the last mmput, it would be enough to change the oom_reap_task() in
> the case it finds mm_users == 0, to wait for a timeout or to wait for
> __mmput to set MMF_OOM_SKIP itself, but it's not just exit_mmap the
> problem here so the concurrency of exit_mmap and oom_reap_task is
> apparently warranted.
>
> It's a non standard runtime, exit_mmap() runs without mmap_sem, and
> oom_reap_task runs with the mmap_sem for reading as usual (kind of
> MADV_DONTNEED).
>
> The race between the two is solved with a combination of
> tsk_is_oom_victim() (serialized by task_lock) and MMF_OOM_SKIP
> (serialized by a dummy down_write/up_write cycle on the same lines of
> the ksm_exit method).
>
> If the oom_reap_task() may be running concurrently during exit_mmap,
> exit_mmap will wait it to finish in down_write (before taking down mm
> structures that would make the oom_reap_task fail with use after
> free).
>
> If exit_mmap comes first, oom_reap_task() will skip the mm if
> MMF_OOM_SKIP is already set and in turn all memory is already freed
> and furthermore the mm data structures may already have been taken
> down by free_pgtables.
>
> Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
With your follow-up one liner to include linux/oom.h folded in:
Tested-by: David Rientjes <rientjes@google.com>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-08-10 18:51 ` Michal Hocko
@ 2017-08-10 20:36 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-08-10 20:36 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Andrew Morton, Tetsuo Handa, David Rientjes, Oleg Nesterov,
Andrea Argangeli, Hugh Dickins, Kirill A. Shutemov, linux-mm,
LKML
On Thu 10-08-17 20:51:38, Michal Hocko wrote:
[...]
> OK, let's agree to disagree. As I've said I like when the critical
> section is explicit and we _know_ what it protects. In this case it is
> clear that we have to protect from the page tables tear down and the
> vma destructions. But as I've said I am not going to argue about this
> more. It is more important to finally fix this.
Now that I've reread, it may sound different than I thought. I meant to
say that I will not argue about which solution is better and both
patches are good to go. I will let others to decide but I would be glad
if we go with something finally.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-08-10 18:05 ` Andrea Arcangeli
@ 2017-08-10 18:51 ` Michal Hocko
2017-08-10 20:36 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-08-10 18:51 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Andrew Morton, Tetsuo Handa, David Rientjes, Oleg Nesterov,
Andrea Argangeli, Hugh Dickins, Kirill A. Shutemov, linux-mm,
LKML
On Thu 10-08-17 20:05:54, Andrea Arcangeli wrote:
> On Thu, Aug 10, 2017 at 10:16:32AM +0200, Michal Hocko wrote:
> > Andrea has proposed and alternative solution [4] which should be
> > equivalent functionally similar to {ksm,khugepaged}_exit. I have to
> > confess I really don't like that approach but I can live with it if
> > that is a preferred way (to be honest I would like to drop the empty
>
> Well you added two branches, when only one is necessary. It's more or
> less like preferring a rwsem when a mutex is enough, because you're
> more used to use rwsems.
>
> > down_write();up_write() from the other two callers as well). In fact I
> > have asked Andrea to post his patch [5] but that hasn't happened. I do
> > not think we should wait much longer and finally merge some fix.
>
> It's posted in [4] already below I didn't think it was necessary to
> resend it.
it was deep in the email thread and I've asked you explicitly to repost
which I've done for the same reason.
> The only other improvement I can think of is an unlikely
> around tsk_is_oom_victim() in exit_mmap, but your patch below would
> need it too, and two of them.
with
diff --git a/mm/mmap.c b/mm/mmap.c
index 822e8860b9d2..9d4a5a488f72 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -3002,7 +3002,7 @@ void exit_mmap(struct mm_struct *mm)
* with tsk->mm != NULL checked on !current tasks which synchronizes
* with exit_mm and so we cannot race here.
*/
- if (tsk_is_oom_victim(current)) {
+ if (unlikely(tsk_is_oom_victim(current))) {
down_write(&mm->mmap_sem);
locked = true;
}
@@ -3020,7 +3020,7 @@ void exit_mmap(struct mm_struct *mm)
}
mm->mmap = NULL;
vm_unacct_memory(nr_accounted);
- if (locked)
+ if (unlikely(locked))
up_write(&mm->mmap_sem);
}
The generated code is identical. But I do not have any objection of
course.
> > [1] http://lkml.kernel.org/r/20170724072332.31903-1-mhocko@kernel.org
> > [2] http://lkml.kernel.org/r/20170725142626.GJ26723@dhcp22.suse.cz
> > [3] http://lkml.kernel.org/r/20170725160359.GO26723@dhcp22.suse.cz
> > [4] http://lkml.kernel.org/r/20170726162912.GA29716@redhat.com
> > [5] http://lkml.kernel.org/r/20170728062345.GA2274@dhcp22.suse.cz
> >
> > + if (tsk_is_oom_victim(current)) {
> > + down_write(&mm->mmap_sem);
> > + locked = true;
> > + }
> > free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> > tlb_finish_mmu(&tlb, 0, -1);
> >
> > @@ -3005,7 +3018,10 @@ void exit_mmap(struct mm_struct *mm)
> > nr_accounted += vma_pages(vma);
> > vma = remove_vma(vma);
> > }
> > + mm->mmap = NULL;
> > vm_unacct_memory(nr_accounted);
> > + if (locked)
> > + up_write(&mm->mmap_sem);
>
> I wouldn't normally repost to add an unlikely when I'm not sure if it
> gets merged, but if it gets merged I would immediately tell to Andrew
> about the microoptimization being missing there so he can fold it
> later.
>
> Before reposting about the unlikely I thought we should agree which
> version to merge: [4] or the above double branch (for no good as far
> as I tangibly can tell).
>
> I think down_write;up_write is the correct thing to do here because
> holding the lock for any additional instruction has zero benefits, and
> if it has zero benefits it only adds up confusion and makes the code
> partly senseless, and that ultimately hurts the reader when it tries
> to understand why you're holding the lock for so long when it's not
> needed.
OK, let's agree to disagree. As I've said I like when the critical
section is explicit and we _know_ what it protects. In this case it is
clear that we have to protect from the page tables tear down and the
vma destructions. But as I've said I am not going to argue about this
more. It is more important to finally fix this.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-08-10 8:16 [PATCH] mm, oom: allow oom reaper to race with exit_mmap Michal Hocko
@ 2017-08-10 18:05 ` Andrea Arcangeli
2017-08-10 18:51 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Andrea Arcangeli @ 2017-08-10 18:05 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, Tetsuo Handa, David Rientjes, Oleg Nesterov,
Andrea Argangeli, Hugh Dickins, Kirill A. Shutemov, linux-mm,
LKML, Michal Hocko
On Thu, Aug 10, 2017 at 10:16:32AM +0200, Michal Hocko wrote:
> Andrea has proposed and alternative solution [4] which should be
> equivalent functionally similar to {ksm,khugepaged}_exit. I have to
> confess I really don't like that approach but I can live with it if
> that is a preferred way (to be honest I would like to drop the empty
Well you added two branches, when only one is necessary. It's more or
less like preferring a rwsem when a mutex is enough, because you're
more used to use rwsems.
> down_write();up_write() from the other two callers as well). In fact I
> have asked Andrea to post his patch [5] but that hasn't happened. I do
> not think we should wait much longer and finally merge some fix.
It's posted in [4] already below I didn't think it was necessary to
resend it. The only other improvement I can think of is an unlikely
around tsk_is_oom_victim() in exit_mmap, but your patch below would
need it too, and two of them.
> [1] http://lkml.kernel.org/r/20170724072332.31903-1-mhocko@kernel.org
> [2] http://lkml.kernel.org/r/20170725142626.GJ26723@dhcp22.suse.cz
> [3] http://lkml.kernel.org/r/20170725160359.GO26723@dhcp22.suse.cz
> [4] http://lkml.kernel.org/r/20170726162912.GA29716@redhat.com
> [5] http://lkml.kernel.org/r/20170728062345.GA2274@dhcp22.suse.cz
>
> + if (tsk_is_oom_victim(current)) {
> + down_write(&mm->mmap_sem);
> + locked = true;
> + }
> free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> tlb_finish_mmu(&tlb, 0, -1);
>
> @@ -3005,7 +3018,10 @@ void exit_mmap(struct mm_struct *mm)
> nr_accounted += vma_pages(vma);
> vma = remove_vma(vma);
> }
> + mm->mmap = NULL;
> vm_unacct_memory(nr_accounted);
> + if (locked)
> + up_write(&mm->mmap_sem);
I wouldn't normally repost to add an unlikely when I'm not sure if it
gets merged, but if it gets merged I would immediately tell to Andrew
about the microoptimization being missing there so he can fold it
later.
Before reposting about the unlikely I thought we should agree which
version to merge: [4] or the above double branch (for no good as far
as I tangibly can tell).
I think down_write;up_write is the correct thing to do here because
holding the lock for any additional instruction has zero benefits, and
if it has zero benefits it only adds up confusion and makes the code
partly senseless, and that ultimately hurts the reader when it tries
to understand why you're holding the lock for so long when it's not
needed.
I just read other code yesterday for another bug about the rss going
off by one in some older kernel, that calls add_mm_rss_vec(mm, rss);
where rss is on the stack and mm->rss_stat is mm global, under the PT
lock, and again I had to ask myself why is it done there, and if the
PT lock could possibly help. My evaluation was no, it's just done in
the wrong place, but then I'm not 100% sure because there's a chance I
misread something very subtle.
add_mm_rss_vec(mm, rss);
arch_leave_lazy_mmu_mode();
/* Do the actual TLB flush before dropping ptl */
if (force_flush)
tlb_flush_mmu_tlbonly(tlb);
pte_unmap_unlock(start_pte, ptl);
The tlb flushing doesn't seem to check the stats either, so why is
add_mm_rss_vec isn't called after pte_unmap_unlock?
And yes it looks offtopic (and there's no bug in the rss accounting, I
was just reading around it just in case) but it's not, it's precisely
the kind of issue I have with your patch because it'll introduce
another case like above that I can't explain why it's done under a
lock when it doesn't need it, and it's hard to guess it was just your
dislike for down_read;up_write that made you choose to hold the lock
for no good reason arbitrarily a bit longer.
Thanks,
Andrea
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH] mm, oom: allow oom reaper to race with exit_mmap
@ 2017-08-10 8:16 Michal Hocko
2017-08-10 18:05 ` Andrea Arcangeli
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-08-10 8:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Tetsuo Handa, David Rientjes, Oleg Nesterov, Andrea Argangeli,
Hugh Dickins, Kirill A. Shutemov, linux-mm, LKML, Michal Hocko
From: Michal Hocko <mhocko@suse.com>
David has noticed that the oom killer might kill additional tasks while
the exiting oom victim hasn't terminated yet because the oom_reaper marks
the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
to 0. The race is as follows
oom_reap_task do_exit
exit_mm
__oom_reap_task_mm
mmput
__mmput
mmget_not_zero # fails
exit_mmap # frees memory
set_bit(MMF_OOM_SKIP)
The victim is still visible to the OOM killer until it is unhashed.
Currently we try to reduce a risk of this race by taking oom_lock
and wait for out_of_memory sleep while holding the lock to give the
victim some time to exit. This is quite suboptimal approach because
there is no guarantee the victim (especially a large one) will manage
to unmap its address space and free enough memory to the particular oom
domain which needs a memory (e.g. a specific NUMA node).
Fix this problem by allowing __oom_reap_task_mm and __mmput path to
race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
to run in parallel with other unmappers (hence the mmap_sem for read).
The only tricky part is to exclude page tables tear down and all
operations which modify the address space in the __mmput path. exit_mmap
doesn't expect any other users so it doesn't use any locking. Nothing
really forbids us to use mmap_sem for write, though. In fact we are
already relying on this lock earlier in the __mmput path to synchronize
with ksm and khugepaged.
Take the exclusive mmap_sem when calling free_pgtables and destroying
vmas to sync with __oom_reap_task_mm which take the lock for read. All
other operations can safely race with the parallel unmap.
Changes v1
- bail on null mm->mmap early as per David Rientjes
- take exclusive mmap_sem in exit_mmap only for oom victims to reduce
the lock overhead
Reported-by: David Rientjes <rientjes@google.com>
Fixes: 26db62f179d1 ("oom: keep mm of the killed task available")
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
Hi,
the previous version of the patch has been posted here [1]. The original
patch has taken mmap_sem in exit_mmap unconditionally but Kirill was
worried this could have a performance impact (we should exercise the
fast path most of the time because nobody should be holding lock at that
stage). An artificial testcase [2] has shown ~3% difference but numbers
are quite noisy [3] so it is effect is not all that clear. Anyway I have
made the lock conditional for oom victims.
Andrea has proposed and alternative solution [4] which should be
equivalent functionally similar to {ksm,khugepaged}_exit. I have to
confess I really don't like that approach but I can live with it if
that is a preferred way (to be honest I would like to drop the empty
down_write();up_write() from the other two callers as well). In fact I
have asked Andrea to post his patch [5] but that hasn't happened. I do
not think we should wait much longer and finally merge some fix.
[1] http://lkml.kernel.org/r/20170724072332.31903-1-mhocko@kernel.org
[2] http://lkml.kernel.org/r/20170725142626.GJ26723@dhcp22.suse.cz
[3] http://lkml.kernel.org/r/20170725160359.GO26723@dhcp22.suse.cz
[4] http://lkml.kernel.org/r/20170726162912.GA29716@redhat.com
[5] http://lkml.kernel.org/r/20170728062345.GA2274@dhcp22.suse.cz
mm/mmap.c | 16 ++++++++++++++++
mm/oom_kill.c | 47 ++++++++---------------------------------------
2 files changed, 24 insertions(+), 39 deletions(-)
diff --git a/mm/mmap.c b/mm/mmap.c
index 24e9261bdcc0..822e8860b9d2 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -44,6 +44,7 @@
#include <linux/userfaultfd_k.h>
#include <linux/moduleparam.h>
#include <linux/pkeys.h>
+#include <linux/oom.h>
#include <linux/uaccess.h>
#include <asm/cacheflush.h>
@@ -2967,6 +2968,7 @@ void exit_mmap(struct mm_struct *mm)
struct mmu_gather tlb;
struct vm_area_struct *vma;
unsigned long nr_accounted = 0;
+ bool locked = false;
/* mm's last user has gone, and its about to be pulled down */
mmu_notifier_release(mm);
@@ -2993,6 +2995,17 @@ void exit_mmap(struct mm_struct *mm)
/* Use -1 here to ensure all VMAs in the mm are unmapped */
unmap_vmas(&tlb, vma, 0, -1);
+ /*
+ * oom reaper might race with exit_mmap so make sure we won't free
+ * page tables or unmap VMAs under its feet
+ * Please note that mark_oom_victim is always called under task_lock
+ * with tsk->mm != NULL checked on !current tasks which synchronizes
+ * with exit_mm and so we cannot race here.
+ */
+ if (tsk_is_oom_victim(current)) {
+ down_write(&mm->mmap_sem);
+ locked = true;
+ }
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(&tlb, 0, -1);
@@ -3005,7 +3018,10 @@ void exit_mmap(struct mm_struct *mm)
nr_accounted += vma_pages(vma);
vma = remove_vma(vma);
}
+ mm->mmap = NULL;
vm_unacct_memory(nr_accounted);
+ if (locked)
+ up_write(&mm->mmap_sem);
}
/* Insert vm structure into process list sorted by address
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f030c1c..b1c96e1910f2 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -470,40 +470,15 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
{
struct mmu_gather tlb;
struct vm_area_struct *vma;
- bool ret = true;
-
- /*
- * We have to make sure to not race with the victim exit path
- * and cause premature new oom victim selection:
- * __oom_reap_task_mm exit_mm
- * mmget_not_zero
- * mmput
- * atomic_dec_and_test
- * exit_oom_victim
- * [...]
- * out_of_memory
- * select_bad_process
- * # no TIF_MEMDIE task selects new victim
- * unmap_page_range # frees some memory
- */
- mutex_lock(&oom_lock);
if (!down_read_trylock(&mm->mmap_sem)) {
- ret = false;
trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
+ return false;
}
- /*
- * increase mm_users only after we know we will reap something so
- * that the mmput_async is called only when we have reaped something
- * and delayed __mmput doesn't matter that much
- */
- if (!mmget_not_zero(mm)) {
- up_read(&mm->mmap_sem);
- trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
- }
+ /* There is nothing to reap so bail out without signs in the log */
+ if (!mm->mmap)
+ goto unlock;
trace_start_task_reaping(tsk->pid);
@@ -540,18 +515,12 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
K(get_mm_counter(mm, MM_ANONPAGES)),
K(get_mm_counter(mm, MM_FILEPAGES)),
K(get_mm_counter(mm, MM_SHMEMPAGES)));
- up_read(&mm->mmap_sem);
- /*
- * Drop our reference but make sure the mmput slow path is called from a
- * different context because we shouldn't risk we get stuck there and
- * put the oom_reaper out of the way.
- */
- mmput_async(mm);
trace_finish_task_reaping(tsk->pid);
-unlock_oom:
- mutex_unlock(&oom_lock);
- return ret;
+unlock:
+ up_read(&mm->mmap_sem);
+
+ return true;
}
#define MAX_OOM_REAP_RETRIES 10
--
2.13.2
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-27 14:55 ` Andrea Arcangeli
@ 2017-07-28 6:23 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-28 6:23 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Kirill A. Shutemov, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Thu 27-07-17 16:55:59, Andrea Arcangeli wrote:
> On Thu, Jul 27, 2017 at 08:50:24AM +0200, Michal Hocko wrote:
> > Yes this will work and it won't depend on the oom_lock. But isn't it
> > just more ugly than simply doing
> >
> > if (tsk_is_oom_victim) {
> > down_write(&mm->mmap_sem);
> > locked = true;
> > }
> > free_pgtables(...)
> > [...]
> > if (locked)
> > down_up(&mm->mmap_sem);
>
> To me not doing if (tsk_is_oom...) { down_write; up_write } is by
> default a confusing implementation, because it's not strict and not
> strict code is not self documenting and you've to think twice of why
> you're doing something the way you're doing it.
I disagree. But this is a matter of taste, I guess. I prefer when
locking is explicit and clear about the scope. An empty locked region
used for synchronization is just obscure and you have to think much more
what it means when you can see what the lock actually protects. It is
also less error prone I believe.
> The doubt on what was the point to hold the mmap_sem during
> free_pgtables is precisely why I started digging into this issue
> because it didn't look possible you could truly benefit from holding
> the mmap_sem during free_pgtables.
The point is that you cannot remove page tables while somebody is
walking them. This is enforced in all other paths except for exit which
was kind of special because nobody could race there.
> I also don't like having a new invariant that your solution relies on,
> that is mm->mmap = NULL, when we can make just set the MMF_OOM_SKIP a
> bit earlier that it gets set anyway and use that to control the other
> side of the race.
Well, again a matter of taste. I prefer making it clear that mm->mmap is
no longer valid because it points to a freed pointer. Relying on
MMF_OOM_SKIP for the handshake is just abusing the flag for a different
purpose than it was intended.
> I like strict code that uses as fewer invariants as possible and that
> never holds a lock for any instruction more than it is required (again
> purely for self documenting reasons, the CPU won't notice much one
> instruction more or less).
>
> Even with your patch the two branches are unnecessary, that may not be
> measurable, but it's still wasted CPU. It's all about setting mm->mmap
> before the up_write. In fact my patch should at least put an incremental
> unlikely around my single branch added to exit_mmap.
>
> I see the {down_write;up_write} Hugh's ksm_exit-like as a strict
> solution to this issue and I wrote it specifically while trying to
> research a way to be more strict because from the start it didn't look
> the holding of the mmap_sem during free_pgtables was necessary.
While we have discussed that with Hugh he has shown an interest to
actually get rid of this (peculiar) construct which would be possible if
the down_write was implicit in exit_mmap [1].
> I'm also fine to drop the oom_lock but I think it can be done
> incrementally as it's a separate issue, my second patch should allow
> for it with no adverse side effects.
>
> All I care about is the exit_mmap path because it runs too many times
> not to pay deep attention to every bit of it ;).
Well, all I care about is to fix this over-eager oom killing due to oom
reaper setting MMF_OOM_SKIP too early. That is a regression. I think we
can agree that both proposed solutions are functionally equivalent. I do
not want to push mine too hard so if you _believe_ that yours is really
better feel free to post it for inclusion.
[1] http://lkml.kernel.org/r/alpine.LSU.2.11.1707191716030.2055@eggly.anvils
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-27 6:50 ` Michal Hocko
@ 2017-07-27 14:55 ` Andrea Arcangeli
2017-07-28 6:23 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Andrea Arcangeli @ 2017-07-27 14:55 UTC (permalink / raw)
To: Michal Hocko
Cc: Kirill A. Shutemov, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Thu, Jul 27, 2017 at 08:50:24AM +0200, Michal Hocko wrote:
> Yes this will work and it won't depend on the oom_lock. But isn't it
> just more ugly than simply doing
>
> if (tsk_is_oom_victim) {
> down_write(&mm->mmap_sem);
> locked = true;
> }
> free_pgtables(...)
> [...]
> if (locked)
> down_up(&mm->mmap_sem);
To me not doing if (tsk_is_oom...) { down_write; up_write } is by
default a confusing implementation, because it's not strict and not
strict code is not self documenting and you've to think twice of why
you're doing something the way you're doing it.
The doubt on what was the point to hold the mmap_sem during
free_pgtables is precisely why I started digging into this issue
because it didn't look possible you could truly benefit from holding
the mmap_sem during free_pgtables.
I also don't like having a new invariant that your solution relies on,
that is mm->mmap = NULL, when we can make just set the MMF_OOM_SKIP a
bit earlier that it gets set anyway and use that to control the other
side of the race.
I like strict code that uses as fewer invariants as possible and that
never holds a lock for any instruction more than it is required (again
purely for self documenting reasons, the CPU won't notice much one
instruction more or less).
Even with your patch the two branches are unnecessary, that may not be
measurable, but it's still wasted CPU. It's all about setting mm->mmap
before the up_write. In fact my patch should at least put an incremental
unlikely around my single branch added to exit_mmap.
I see the {down_write;up_write} Hugh's ksm_exit-like as a strict
solution to this issue and I wrote it specifically while trying to
research a way to be more strict because from the start it didn't look
the holding of the mmap_sem during free_pgtables was necessary.
I'm also fine to drop the oom_lock but I think it can be done
incrementally as it's a separate issue, my second patch should allow
for it with no adverse side effects.
All I care about is the exit_mmap path because it runs too many times
not to pay deep attention to every bit of it ;).
Thanks,
Andrea
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-26 16:29 ` Andrea Arcangeli
2017-07-26 16:43 ` Andrea Arcangeli
@ 2017-07-27 6:50 ` Michal Hocko
2017-07-27 14:55 ` Andrea Arcangeli
2017-08-15 0:20 ` David Rientjes
2 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-27 6:50 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Kirill A. Shutemov, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Wed 26-07-17 18:29:12, Andrea Arcangeli wrote:
> On Wed, Jul 26, 2017 at 07:45:57AM +0200, Michal Hocko wrote:
> > On Tue 25-07-17 21:19:52, Andrea Arcangeli wrote:
> > > On Tue, Jul 25, 2017 at 06:04:00PM +0200, Michal Hocko wrote:
> > > > - down_write(&mm->mmap_sem);
> > > > + if (tsk_is_oom_victim(current))
> > > > + down_write(&mm->mmap_sem);
> > > > free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> > > > tlb_finish_mmu(&tlb, 0, -1);
> > > >
> > > > @@ -3012,7 +3014,8 @@ void exit_mmap(struct mm_struct *mm)
> > > > }
> > > > mm->mmap = NULL;
> > > > vm_unacct_memory(nr_accounted);
> > > > - up_write(&mm->mmap_sem);
> > > > + if (tsk_is_oom_victim(current))
> > > > + up_write(&mm->mmap_sem);
> > >
> > > How is this possibly safe? mark_oom_victim can run while exit_mmap is
> > > running.
> >
> > I believe it cannot. We always call mark_oom_victim (on !current) with
> > task_lock held and check task->mm != NULL and we call do_exit->mmput after
> > mm is set to NULL under the same lock.
>
> Holding the mmap_sem for writing and setting mm->mmap to NULL to
> filter which tasks already released the mmap_sem for writing post
> free_pgtables still look unnecessary to solve this.
>
> Using MMF_OOM_SKIP as flag had side effects of oom_badness() skipping
> it, but we can use the same tsk_is_oom_victim instead and relay on the
> locking in mark_oom_victim you pointed out above instead of the
> test_and_set_bit of my patch, because current->mm is already NULL at
> that point.
>
> A race at the light of the above now is, because current->mm is NULL by the
> time mmput is called, how can you start the oom_reap_task on a process
> with current->mm NULL that called the last mmput and is blocked
> in exit_aio?
Because we have that mm available. See tsk->signal->oom_mm in
oom_reap_task
> It looks like no false positive can get fixed until this
> is solved first because
>
> Isn't this enough? If this is enough it avoids other modification to
> the exit_mmap runtime that looks unnecessary: mm->mmap = NULL replaced
> by MMF_OOM_SKIP that has to be set anyway by __mmput later and one
> unnecessary branch to call the up_write.
>
[...]
> diff --git a/mm/mmap.c b/mm/mmap.c
> index f19efcf75418..bdab595ce25c 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -2993,6 +2993,23 @@ void exit_mmap(struct mm_struct *mm)
> /* Use -1 here to ensure all VMAs in the mm are unmapped */
> unmap_vmas(&tlb, vma, 0, -1);
>
> + set_bit(MMF_OOM_SKIP, &mm->flags);
> + if (tsk_is_oom_victim(current)) {
> + /*
> + * Wait for oom_reap_task() to stop working on this
> + * mm. Because MMF_OOM_SKIP is already set before
> + * calling down_read(), oom_reap_task() will not run
> + * on this "mm" post up_write().
> + *
> + * tsk_is_oom_victim() cannot be set from under us
> + * either because current->mm is already set to NULL
> + * under task_lock before calling mmput and oom_mm is
> + * set not NULL by the OOM killer only if current->mm
> + * is found not NULL while holding the task_lock.
> + */
> + down_write(&mm->mmap_sem);
> + up_write(&mm->mmap_sem);
> + }
> free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> tlb_finish_mmu(&tlb, 0, -1);
Yes this will work and it won't depend on the oom_lock. But isn't it
just more ugly than simply doing
if (tsk_is_oom_victim) {
down_write(&mm->mmap_sem);
locked = true;
}
free_pgtables(...)
[...]
if (locked)
down_up(&mm->mmap_sem);
in general I do not like empty locked sections much, to be honest. Now
with the conditional locking my patch looks as follows. It should be
pretty much equivalent to your patch. Would that be acceptable for you
or do you think there is a strong reason to go with yours?
---
>From 2198654be88d11efb1f372e8579761f65e219206 Mon Sep 17 00:00:00 2001
From: Michal Hocko <mhocko@suse.com>
Date: Thu, 27 Jul 2017 08:48:15 +0200
Subject: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
David has noticed that the oom killer might kill additional tasks while
the exiting oom victim hasn't terminated yet because the oom_reaper marks
the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
to 0. The race is as follows
oom_reap_task do_exit
exit_mm
__oom_reap_task_mm
mmput
__mmput
mmget_not_zero # fails
exit_mmap # frees memory
set_bit(MMF_OOM_SKIP)
The victim is still visible to the OOM killer until it is unhashed.
Currently we try to reduce a risk of this race by taking oom_lock
and wait for out_of_memory sleep while holding the lock to give the
victim some time to exit. This is quite suboptimal approach because
there is no guarantee the victim (especially a large one) will manage
to unmap its address space and free enough memory to the particular oom
domain which needs a memory (e.g. a specific NUMA node).
Fix this problem by allowing __oom_reap_task_mm and __mmput path to
race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
to run in parallel with other unmappers (hence the mmap_sem for read).
The only tricky part is to exclude page tables tear down and all
operations which modify the address space in the __mmput path. exit_mmap
doesn't expect any other users so it doesn't use any locking. Nothing
really forbids us to use mmap_sem for write, though. In fact we are
already relying on this lock earlier in the __mmput path to synchronize
with ksm and khugepaged.
Take the exclusive mmap_sem when calling free_pgtables and destroying
vmas to sync with __oom_reap_task_mm which take the lock for read. All
other operations can safely race with the parallel unmap.
Changes v1
- bail on null mm->mmap early as per David Rientjes
- take exclusive mmap_sem in exit_mmap only for oom victims to reduce
the lock overhead
Reported-by: David Rientjes <rientjes@google.com>
Fixes: 26db62f179d1 ("oom: keep mm of the killed task available")
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
mm/mmap.c | 16 ++++++++++++++++
mm/oom_kill.c | 47 ++++++++---------------------------------------
2 files changed, 24 insertions(+), 39 deletions(-)
diff --git a/mm/mmap.c b/mm/mmap.c
index 24e9261bdcc0..822e8860b9d2 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -44,6 +44,7 @@
#include <linux/userfaultfd_k.h>
#include <linux/moduleparam.h>
#include <linux/pkeys.h>
+#include <linux/oom.h>
#include <linux/uaccess.h>
#include <asm/cacheflush.h>
@@ -2967,6 +2968,7 @@ void exit_mmap(struct mm_struct *mm)
struct mmu_gather tlb;
struct vm_area_struct *vma;
unsigned long nr_accounted = 0;
+ bool locked = false;
/* mm's last user has gone, and its about to be pulled down */
mmu_notifier_release(mm);
@@ -2993,6 +2995,17 @@ void exit_mmap(struct mm_struct *mm)
/* Use -1 here to ensure all VMAs in the mm are unmapped */
unmap_vmas(&tlb, vma, 0, -1);
+ /*
+ * oom reaper might race with exit_mmap so make sure we won't free
+ * page tables or unmap VMAs under its feet
+ * Please note that mark_oom_victim is always called under task_lock
+ * with tsk->mm != NULL checked on !current tasks which synchronizes
+ * with exit_mm and so we cannot race here.
+ */
+ if (tsk_is_oom_victim(current)) {
+ down_write(&mm->mmap_sem);
+ locked = true;
+ }
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(&tlb, 0, -1);
@@ -3005,7 +3018,10 @@ void exit_mmap(struct mm_struct *mm)
nr_accounted += vma_pages(vma);
vma = remove_vma(vma);
}
+ mm->mmap = NULL;
vm_unacct_memory(nr_accounted);
+ if (locked)
+ up_write(&mm->mmap_sem);
}
/* Insert vm structure into process list sorted by address
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f030c1c..b1c96e1910f2 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -470,40 +470,15 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
{
struct mmu_gather tlb;
struct vm_area_struct *vma;
- bool ret = true;
-
- /*
- * We have to make sure to not race with the victim exit path
- * and cause premature new oom victim selection:
- * __oom_reap_task_mm exit_mm
- * mmget_not_zero
- * mmput
- * atomic_dec_and_test
- * exit_oom_victim
- * [...]
- * out_of_memory
- * select_bad_process
- * # no TIF_MEMDIE task selects new victim
- * unmap_page_range # frees some memory
- */
- mutex_lock(&oom_lock);
if (!down_read_trylock(&mm->mmap_sem)) {
- ret = false;
trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
+ return false;
}
- /*
- * increase mm_users only after we know we will reap something so
- * that the mmput_async is called only when we have reaped something
- * and delayed __mmput doesn't matter that much
- */
- if (!mmget_not_zero(mm)) {
- up_read(&mm->mmap_sem);
- trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
- }
+ /* There is nothing to reap so bail out without signs in the log */
+ if (!mm->mmap)
+ goto unlock;
trace_start_task_reaping(tsk->pid);
@@ -540,18 +515,12 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
K(get_mm_counter(mm, MM_ANONPAGES)),
K(get_mm_counter(mm, MM_FILEPAGES)),
K(get_mm_counter(mm, MM_SHMEMPAGES)));
- up_read(&mm->mmap_sem);
- /*
- * Drop our reference but make sure the mmput slow path is called from a
- * different context because we shouldn't risk we get stuck there and
- * put the oom_reaper out of the way.
- */
- mmput_async(mm);
trace_finish_task_reaping(tsk->pid);
-unlock_oom:
- mutex_unlock(&oom_lock);
- return ret;
+unlock:
+ up_read(&mm->mmap_sem);
+
+ return true;
}
#define MAX_OOM_REAP_RETRIES 10
--
2.13.2
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-26 16:39 ` Andrea Arcangeli
@ 2017-07-27 6:32 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-27 6:32 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, linux-mm, LKML
On Wed 26-07-17 18:39:28, Andrea Arcangeli wrote:
> On Wed, Jul 26, 2017 at 07:45:33AM +0200, Michal Hocko wrote:
> > Yes, exit_aio is the only blocking call I know of currently. But I would
> > like this to be as robust as possible and so I do not want to rely on
> > the current implementation. This can change in future and I can
> > guarantee that nobody will think about the oom path when adding
> > something to the final __mmput path.
>
> I think ksm_exit may block too waiting for allocations, the generic
> idea is those calls before exit_mmap can cause a problem yes.
I thought that ksm used __GFP_NORETRY but haven't checked too deeply.
Anyway I guess we agree that enabling oom_reaper to race with the final
__mmput is desirable?
[...]
> > This will work more or less the same to what we have currently.
> >
> > [victim] [oom reaper] [oom killer]
> > do_exit __oom_reap_task_mm
> > mmput
> > __mmput
> > mmget_not_zero
> > test_and_set_bit(MMF_OOM_SKIP)
> > oom_evaluate_task
> > # select next victim
> > # reap the mm
> > unmap_vmas
> >
> > so we can select a next victim while the current one is still not
> > completely torn down.
>
> How does oom_evaluate_task possibly run at the same time of
> test_and_set_bit in __oom_reap_task_mm considering both are running
> under the oom_lock?
You are absolutely right. This race is impossible. It was just me
assuming we are going to get rid of the oom_lock because I have that
idea in the back of my head and I would really like to get rid of
it. Global locks are nasty and I would prefer dropping it if we can.
[...]
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-26 16:29 ` Andrea Arcangeli
@ 2017-07-26 16:43 ` Andrea Arcangeli
2017-07-27 6:50 ` Michal Hocko
2017-08-15 0:20 ` David Rientjes
2 siblings, 0 replies; 54+ messages in thread
From: Andrea Arcangeli @ 2017-07-26 16:43 UTC (permalink / raw)
To: Michal Hocko
Cc: Kirill A. Shutemov, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Wed, Jul 26, 2017 at 06:29:12PM +0200, Andrea Arcangeli wrote:
> From 3d9001490ee1a71f39c7bfaf19e96821f9d3ff16 Mon Sep 17 00:00:00 2001
> From: Andrea Arcangeli <aarcange@redhat.com>
> Date: Tue, 25 Jul 2017 20:02:27 +0200
> Subject: [PATCH 1/1] mm: oom: let oom_reap_task and exit_mmap to run
> concurrently
This needs an incremental one liner...
diff --git a/mm/mmap.c b/mm/mmap.c
index bdab595ce25c..fd16996ee0a8 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -44,6 +44,7 @@
#include <linux/userfaultfd_k.h>
#include <linux/moduleparam.h>
#include <linux/pkeys.h>
+#include <linux/oom.h>
#include <linux/uaccess.h>
#include <asm/cacheflush.h>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-26 5:45 ` Michal Hocko
@ 2017-07-26 16:39 ` Andrea Arcangeli
2017-07-27 6:32 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Andrea Arcangeli @ 2017-07-26 16:39 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, linux-mm, LKML
On Wed, Jul 26, 2017 at 07:45:33AM +0200, Michal Hocko wrote:
> Yes, exit_aio is the only blocking call I know of currently. But I would
> like this to be as robust as possible and so I do not want to rely on
> the current implementation. This can change in future and I can
> guarantee that nobody will think about the oom path when adding
> something to the final __mmput path.
I think ksm_exit may block too waiting for allocations, the generic
idea is those calls before exit_mmap can cause a problem yes.
> > exit_mmap would have no issue, if there was enough time in the
> > lifetime CPU to allocate the memory, sure the memory will also be
> > freed in finite amount of time by exit_mmap.
>
> I am not sure I understand. Say that any call prior to unmap_vmas blocks
> on a lock which is held by another call path which cannot proceed with
> the allocation...
What I meant was, if three was no prior call to exit_mmap->unmap_vmas.
> I really do not want to rely on any timing. This just too fragile. Once
> we have killed a task then we shouldn't pick another victim until it
> passed exit_mmap or the oom_reaper did its job. Otherwise we just risk
> false positives while we have already disrupted the workload.
On smaller systems lack or parallelism in OOM killing surely isn't a
problem.
> This will work more or less the same to what we have currently.
>
> [victim] [oom reaper] [oom killer]
> do_exit __oom_reap_task_mm
> mmput
> __mmput
> mmget_not_zero
> test_and_set_bit(MMF_OOM_SKIP)
> oom_evaluate_task
> # select next victim
> # reap the mm
> unmap_vmas
>
> so we can select a next victim while the current one is still not
> completely torn down.
How does oom_evaluate_task possibly run at the same time of
test_and_set_bit in __oom_reap_task_mm considering both are running
under the oom_lock? It's hard to see how what you describe above could
materialize as second and third column cannot run in parallel because
of the oom_lock.
I don't think there was any issue, but then you pointed out the
locking on signal->oom_mm that is protected by the task_lock vs
current->mm NULL check, so I can replace in my patch the
test_and_set_bit with set_bit on one side and the oom_mm task_lock
protected locking on the other side. This way I can put back a set_bit
in the __mmput fast path (instead of test_and_set_bit) and it's even
more efficient. With such a change, I'll also stop depending on the
oom_lock to prevent second and third column to run in parallel.
I still didn't remove the oom_lock outright that seems orthogonal
change unrelated to this issue but now you could remove it as far as
the above is concerned.
> I hope 3f70dc38cec2 ("mm: make sure that kthreads will not refault oom
> reaped memory") will clarify this code. If not please start a new thread
> so that we do not conflate different things together.
I'll look into that, thanks.
Andrea
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-26 5:45 ` Michal Hocko
@ 2017-07-26 16:29 ` Andrea Arcangeli
2017-07-26 16:43 ` Andrea Arcangeli
` (2 more replies)
0 siblings, 3 replies; 54+ messages in thread
From: Andrea Arcangeli @ 2017-07-26 16:29 UTC (permalink / raw)
To: Michal Hocko
Cc: Kirill A. Shutemov, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Wed, Jul 26, 2017 at 07:45:57AM +0200, Michal Hocko wrote:
> On Tue 25-07-17 21:19:52, Andrea Arcangeli wrote:
> > On Tue, Jul 25, 2017 at 06:04:00PM +0200, Michal Hocko wrote:
> > > - down_write(&mm->mmap_sem);
> > > + if (tsk_is_oom_victim(current))
> > > + down_write(&mm->mmap_sem);
> > > free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> > > tlb_finish_mmu(&tlb, 0, -1);
> > >
> > > @@ -3012,7 +3014,8 @@ void exit_mmap(struct mm_struct *mm)
> > > }
> > > mm->mmap = NULL;
> > > vm_unacct_memory(nr_accounted);
> > > - up_write(&mm->mmap_sem);
> > > + if (tsk_is_oom_victim(current))
> > > + up_write(&mm->mmap_sem);
> >
> > How is this possibly safe? mark_oom_victim can run while exit_mmap is
> > running.
>
> I believe it cannot. We always call mark_oom_victim (on !current) with
> task_lock held and check task->mm != NULL and we call do_exit->mmput after
> mm is set to NULL under the same lock.
Holding the mmap_sem for writing and setting mm->mmap to NULL to
filter which tasks already released the mmap_sem for writing post
free_pgtables still look unnecessary to solve this.
Using MMF_OOM_SKIP as flag had side effects of oom_badness() skipping
it, but we can use the same tsk_is_oom_victim instead and relay on the
locking in mark_oom_victim you pointed out above instead of the
test_and_set_bit of my patch, because current->mm is already NULL at
that point.
A race at the light of the above now is, because current->mm is NULL by the
time mmput is called, how can you start the oom_reap_task on a process
with current->mm NULL that called the last mmput and is blocked
in exit_aio? It looks like no false positive can get fixed until this
is solved first because
Isn't this enough? If this is enough it avoids other modification to
the exit_mmap runtime that looks unnecessary: mm->mmap = NULL replaced
by MMF_OOM_SKIP that has to be set anyway by __mmput later and one
unnecessary branch to call the up_write.
>From 3d9001490ee1a71f39c7bfaf19e96821f9d3ff16 Mon Sep 17 00:00:00 2001
From: Andrea Arcangeli <aarcange@redhat.com>
Date: Tue, 25 Jul 2017 20:02:27 +0200
Subject: [PATCH 1/1] mm: oom: let oom_reap_task and exit_mmap to run
concurrently
This is purely required because exit_aio() may block and exit_mmap() may
never start, if the oom_reap_task cannot start running on a mm with
mm_users == 0.
At the same time if the OOM reaper doesn't wait at all for the memory
of the current OOM candidate to be freed by exit_mmap->unmap_vmas, it
would generate a spurious OOM kill.
If it wasn't because of the exit_aio or similar blocking functions in
the last mmput, it would be enough to change the oom_reap_task() in
the case it finds mm_users == 0, to wait for a timeout or to wait for
__mmput to set MMF_OOM_SKIP itself, but it's not just exit_mmap the
problem here so the concurrency of exit_mmap and oom_reap_task is
apparently warranted.
It's a non standard runtime, exit_mmap() runs without mmap_sem, and
oom_reap_task runs with the mmap_sem for reading as usual (kind of
MADV_DONTNEED).
The race between the two is solved with a combination of
tsk_is_oom_victim() (serialized by task_lock) and MMF_OOM_SKIP
(serialized by a dummy down_write/up_write cycle on the same lines of
the ksm_exit method).
If the oom_reap_task() may be running concurrently during exit_mmap,
exit_mmap will wait it to finish in down_write (before taking down mm
structures that would make the oom_reap_task fail with use after
free).
If exit_mmap comes first, oom_reap_task() will skip the mm if
MMF_OOM_SKIP is already set and in turn all memory is already freed
and furthermore the mm data structures may already have been taken
down by free_pgtables.
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---
kernel/fork.c | 1 -
mm/mmap.c | 17 +++++++++++++++++
mm/oom_kill.c | 15 +++++----------
3 files changed, 22 insertions(+), 11 deletions(-)
diff --git a/kernel/fork.c b/kernel/fork.c
index 9ec98b0c4675..ed412d85a596 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -910,7 +910,6 @@ static inline void __mmput(struct mm_struct *mm)
}
if (mm->binfmt)
module_put(mm->binfmt->module);
- set_bit(MMF_OOM_SKIP, &mm->flags);
mmdrop(mm);
}
diff --git a/mm/mmap.c b/mm/mmap.c
index f19efcf75418..bdab595ce25c 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2993,6 +2993,23 @@ void exit_mmap(struct mm_struct *mm)
/* Use -1 here to ensure all VMAs in the mm are unmapped */
unmap_vmas(&tlb, vma, 0, -1);
+ set_bit(MMF_OOM_SKIP, &mm->flags);
+ if (tsk_is_oom_victim(current)) {
+ /*
+ * Wait for oom_reap_task() to stop working on this
+ * mm. Because MMF_OOM_SKIP is already set before
+ * calling down_read(), oom_reap_task() will not run
+ * on this "mm" post up_write().
+ *
+ * tsk_is_oom_victim() cannot be set from under us
+ * either because current->mm is already set to NULL
+ * under task_lock before calling mmput and oom_mm is
+ * set not NULL by the OOM killer only if current->mm
+ * is found not NULL while holding the task_lock.
+ */
+ down_write(&mm->mmap_sem);
+ up_write(&mm->mmap_sem);
+ }
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(&tlb, 0, -1);
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f030c1c..242a1f0d579b 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -495,11 +495,12 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
}
/*
- * increase mm_users only after we know we will reap something so
- * that the mmput_async is called only when we have reaped something
- * and delayed __mmput doesn't matter that much
+ * MMF_OOM_SKIP is set by exit_mmap when the OOM reaper can't
+ * work on the mm anymore. The check for MMF_OOM_SKIP must run
+ * under mmap_sem for reading because it serializes against the
+ * down_write();up_write() cycle in exit_mmap().
*/
- if (!mmget_not_zero(mm)) {
+ if (test_bit(MMF_OOM_SKIP, &mm->flags)) {
up_read(&mm->mmap_sem);
trace_skip_task_reaping(tsk->pid);
goto unlock_oom;
@@ -542,12 +543,6 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
K(get_mm_counter(mm, MM_SHMEMPAGES)));
up_read(&mm->mmap_sem);
- /*
- * Drop our reference but make sure the mmput slow path is called from a
- * different context because we shouldn't risk we get stuck there and
- * put the oom_reaper out of the way.
- */
- mmput_async(mm);
trace_finish_task_reaping(tsk->pid);
unlock_oom:
mutex_unlock(&oom_lock);
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 19:19 ` Andrea Arcangeli
@ 2017-07-26 5:45 ` Michal Hocko
2017-07-26 16:29 ` Andrea Arcangeli
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-26 5:45 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Kirill A. Shutemov, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Tue 25-07-17 21:19:52, Andrea Arcangeli wrote:
> On Tue, Jul 25, 2017 at 06:04:00PM +0200, Michal Hocko wrote:
> > - down_write(&mm->mmap_sem);
> > + if (tsk_is_oom_victim(current))
> > + down_write(&mm->mmap_sem);
> > free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> > tlb_finish_mmu(&tlb, 0, -1);
> >
> > @@ -3012,7 +3014,8 @@ void exit_mmap(struct mm_struct *mm)
> > }
> > mm->mmap = NULL;
> > vm_unacct_memory(nr_accounted);
> > - up_write(&mm->mmap_sem);
> > + if (tsk_is_oom_victim(current))
> > + up_write(&mm->mmap_sem);
>
> How is this possibly safe? mark_oom_victim can run while exit_mmap is
> running.
I believe it cannot. We always call mark_oom_victim (on !current) with
task_lock held and check task->mm != NULL and we call do_exit->mmput after
mm is set to NULL under the same lock.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 18:26 ` Andrea Arcangeli
@ 2017-07-26 5:45 ` Michal Hocko
2017-07-26 16:39 ` Andrea Arcangeli
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-26 5:45 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, linux-mm, LKML
On Tue 25-07-17 20:26:19, Andrea Arcangeli wrote:
> On Tue, Jul 25, 2017 at 05:45:14PM +0200, Michal Hocko wrote:
> > That problem is real though as reported by David.
>
> I'm not against fixing it, I just think it's not a major concern, and
> the solution doesn't seem optimal as measured by Kirill.
>
> I'm just skeptical it's the best to solve that tiny race, 99.9% of the
> time such down_write is unnecessary.
>
> > it is not only about exit_mmap. __mmput calls into exit_aio and that can
> > wait for completion and there is no way to guarantee this will finish in
> > finite time.
>
> exit_aio blocking is actually the only good point for wanting this
> concurrency where exit_mmap->unmap_vmas and
> oom_reap_task->unmap_page_range have to run concurrently on the same
> mm.
Yes, exit_aio is the only blocking call I know of currently. But I would
like this to be as robust as possible and so I do not want to rely on
the current implementation. This can change in future and I can
guarantee that nobody will think about the oom path when adding
something to the final __mmput path.
> exit_mmap would have no issue, if there was enough time in the
> lifetime CPU to allocate the memory, sure the memory will also be
> freed in finite amount of time by exit_mmap.
I am not sure I understand. Say that any call prior to unmap_vmas blocks
on a lock which is held by another call path which cannot proceed with
the allocation...
> In fact you mentioned multiple OOM in the NUMA case, exit_mmap may not
> solve that, so depending on the runtime it may have been better not to
> wait all memory of the process to be freed before moving to the next
> task, but only a couple of seconds before the OOM reaper moves to a
> new candidate. Again this is only a tradeoff between solving the OOM
> faster vs risk of false positives OOM.
I really do not want to rely on any timing. This just too fragile. Once
we have killed a task then we shouldn't pick another victim until it
passed exit_mmap or the oom_reaper did its job. Otherwise we just risk
false positives while we have already disrupted the workload.
> If it wasn't because of exit_aio (which may have to wait I/O
> completion), changing the OOM reaper to return "false" if
> mmget_not_zero returns zero and MMF_OOM_SKIP is not set yet, would
> have been enough (and depending on the runtime it may have solved OOM
> faster in NUMA) and there would be absolutely no need to run OOM
> reaper and exit_mmap concurrently on the same mm. However there's such
> exit_aio..
>
> Raw I/O mempools never require memory allocations, although aio if it
> involves a filesystem to complete may run into filesystem or buffering
> locks which are known to loop forever or depend on other tasks stuck
> in kernel allocations, so I didn't go down that chain too long.
Exactly. We simply cannot assume anything here because veryfying this
basically impossible.
[...]
> diff --git a/mm/mmap.c b/mm/mmap.c
> index f19efcf75418..615133762b99 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -2993,6 +2993,11 @@ void exit_mmap(struct mm_struct *mm)
> /* Use -1 here to ensure all VMAs in the mm are unmapped */
> unmap_vmas(&tlb, vma, 0, -1);
>
> + if (test_and_set_bit(MMF_OOM_SKIP, &mm->flags)) {
> + /* wait the OOM reaper to stop working on this mm */
> + down_write(&mm->mmap_sem);
> + up_write(&mm->mmap_sem);
> + }
> free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> tlb_finish_mmu(&tlb, 0, -1);
>
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index 9e8b4f030c1c..2a7000995784 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -471,6 +471,7 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
> struct mmu_gather tlb;
> struct vm_area_struct *vma;
> bool ret = true;
> + bool mmgot = true;
>
> /*
> * We have to make sure to not race with the victim exit path
> @@ -500,9 +501,16 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
> * and delayed __mmput doesn't matter that much
> */
> if (!mmget_not_zero(mm)) {
> - up_read(&mm->mmap_sem);
> trace_skip_task_reaping(tsk->pid);
> - goto unlock_oom;
> + /*
> + * MMF_OOM_SKIP is set by exit_mmap when the OOM
> + * reaper can't work on the mm anymore.
> + */
> + if (test_and_set_bit(MMF_OOM_SKIP, &mm->flags)) {
> + up_read(&mm->mmap_sem);
> + goto unlock_oom;
> + }
> + mmgot = false;
> }
This will work more or less the same to what we have currently.
[victim] [oom reaper] [oom killer]
do_exit __oom_reap_task_mm
mmput
__mmput
mmget_not_zero
test_and_set_bit(MMF_OOM_SKIP)
oom_evaluate_task
# select next victim
# reap the mm
unmap_vmas
so we can select a next victim while the current one is still not
completely torn down.
> > > 4) how is it safe to overwrite a VM_FAULT_RETRY that returns without
> > > mmap_sem and then the arch code will release the mmap_sem despite
> > > it was already released by handle_mm_fault? Anonymous memory faults
> > > aren't common to return VM_FAULT_RETRY but an userfault
> > > can. Shouldn't there be a block that prevents overwriting if
> > > VM_FAULT_RETRY is set below? (not only VM_FAULT_ERROR)
> > >
> > > if (unlikely((current->flags & PF_KTHREAD) && !(ret & VM_FAULT_ERROR)
> > > && test_bit(MMF_UNSTABLE, &vma->vm_mm->flags)))
> > > ret = VM_FAULT_SIGBUS;
> >
> > I am not sure I understand what you mean and how this is related to the
> > patch?
>
> It's not related to the patch but it involves the OOM reaper as it
> only happens when MMF_UNSTABLE is set which is set only by the OOM
> reaper. I was simply reading the OOM reaper code and following up what
> MMF_UNSTABLE does and it ringed a bell.
I hope 3f70dc38cec2 ("mm: make sure that kthreads will not refault oom
reaped memory") will clarify this code. If not please start a new thread
so that we do not conflate different things together.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 16:04 ` Michal Hocko
@ 2017-07-25 19:19 ` Andrea Arcangeli
2017-07-26 5:45 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Andrea Arcangeli @ 2017-07-25 19:19 UTC (permalink / raw)
To: Michal Hocko
Cc: Kirill A. Shutemov, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, linux-mm, LKML
On Tue, Jul 25, 2017 at 06:04:00PM +0200, Michal Hocko wrote:
> - down_write(&mm->mmap_sem);
> + if (tsk_is_oom_victim(current))
> + down_write(&mm->mmap_sem);
> free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
> tlb_finish_mmu(&tlb, 0, -1);
>
> @@ -3012,7 +3014,8 @@ void exit_mmap(struct mm_struct *mm)
> }
> mm->mmap = NULL;
> vm_unacct_memory(nr_accounted);
> - up_write(&mm->mmap_sem);
> + if (tsk_is_oom_victim(current))
> + up_write(&mm->mmap_sem);
How is this possibly safe? mark_oom_victim can run while exit_mmap is
running. Even if you cache the first read in the local stack, failure
to notice you marked it, could lead to use after free. Or at least
there's no comment on which lock should prevent the use after free
with the above.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 15:45 ` Michal Hocko
@ 2017-07-25 18:26 ` Andrea Arcangeli
2017-07-26 5:45 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Andrea Arcangeli @ 2017-07-25 18:26 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, linux-mm, LKML
On Tue, Jul 25, 2017 at 05:45:14PM +0200, Michal Hocko wrote:
> That problem is real though as reported by David.
I'm not against fixing it, I just think it's not a major concern, and
the solution doesn't seem optimal as measured by Kirill.
I'm just skeptical it's the best to solve that tiny race, 99.9% of the
time such down_write is unnecessary.
> it is not only about exit_mmap. __mmput calls into exit_aio and that can
> wait for completion and there is no way to guarantee this will finish in
> finite time.
exit_aio blocking is actually the only good point for wanting this
concurrency where exit_mmap->unmap_vmas and
oom_reap_task->unmap_page_range have to run concurrently on the same
mm.
exit_mmap would have no issue, if there was enough time in the
lifetime CPU to allocate the memory, sure the memory will also be
freed in finite amount of time by exit_mmap.
In fact you mentioned multiple OOM in the NUMA case, exit_mmap may not
solve that, so depending on the runtime it may have been better not to
wait all memory of the process to be freed before moving to the next
task, but only a couple of seconds before the OOM reaper moves to a
new candidate. Again this is only a tradeoff between solving the OOM
faster vs risk of false positives OOM.
If it wasn't because of exit_aio (which may have to wait I/O
completion), changing the OOM reaper to return "false" if
mmget_not_zero returns zero and MMF_OOM_SKIP is not set yet, would
have been enough (and depending on the runtime it may have solved OOM
faster in NUMA) and there would be absolutely no need to run OOM
reaper and exit_mmap concurrently on the same mm. However there's such
exit_aio..
Raw I/O mempools never require memory allocations, although aio if it
involves a filesystem to complete may run into filesystem or buffering
locks which are known to loop forever or depend on other tasks stuck
in kernel allocations, so I didn't go down that chain too long.
So the simplest is to use a similar trick to what ksm_exit uses, this
is untested just to show the idea it may require further adjustment as
the bit isn't used only for the test_and_set_bit locking, but I didn't
see much issues with other set_bit/test_bit.
>From f414244480fdc1f771b3148feb3fac77ec813e4c Mon Sep 17 00:00:00 2001
From: Andrea Arcangeli <aarcange@redhat.com>
Date: Tue, 25 Jul 2017 20:02:27 +0200
Subject: [PATCH 1/1] mm: oom: let oom_reap_task and exit_mmap to run
concurrently
This is purely required because exit_aio() may block and exit_mmap() may
never start, if the oom_reap_task cannot start running on a mm with
mm_users == 0.
At the same time if the OOM reaper doesn't wait at all for the memory
of the current OOM candidate to be freed by exit_mmap->unmap_vmas, it
would generate a spurious OOM kill.
If it wasn't because of the exit_aio it would be enough to change the
oom_reap_task in the case it finds mm_users == 0, to wait for a
timeout or to wait for __mmput to set MMF_OOM_SKIP itself, but it's
not just exit_mmap the problem here so the concurrency of exit_mmap
and oom_reap_task is apparently warranted.
It's a non standard runtime, exit_mmap runs without mmap_sem, and
oom_reap_task runs with the mmap_sem for reading as usual (kind of
MADV_DONTNEED).
The race between the two is solved with pair of test_and_set_bit and a
dummy down_write/up_write cycle on the same lines of the ksm_exit
method.
If the OOM reaper sets the bit first, exit_mmap will wait it to finish
in down_write (before taking down mm structures that would make the
oom_reap_task fail with use after free).
If exit_mmap comes first, oom reaper will skip the mm as it's already
all freed.
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---
kernel/fork.c | 1 -
mm/mmap.c | 5 +++++
mm/oom_kill.c | 15 ++++++++++++---
3 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/kernel/fork.c b/kernel/fork.c
index 9ec98b0c4675..ed412d85a596 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -910,7 +910,6 @@ static inline void __mmput(struct mm_struct *mm)
}
if (mm->binfmt)
module_put(mm->binfmt->module);
- set_bit(MMF_OOM_SKIP, &mm->flags);
mmdrop(mm);
}
diff --git a/mm/mmap.c b/mm/mmap.c
index f19efcf75418..615133762b99 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2993,6 +2993,11 @@ void exit_mmap(struct mm_struct *mm)
/* Use -1 here to ensure all VMAs in the mm are unmapped */
unmap_vmas(&tlb, vma, 0, -1);
+ if (test_and_set_bit(MMF_OOM_SKIP, &mm->flags)) {
+ /* wait the OOM reaper to stop working on this mm */
+ down_write(&mm->mmap_sem);
+ up_write(&mm->mmap_sem);
+ }
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(&tlb, 0, -1);
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f030c1c..2a7000995784 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -471,6 +471,7 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
struct mmu_gather tlb;
struct vm_area_struct *vma;
bool ret = true;
+ bool mmgot = true;
/*
* We have to make sure to not race with the victim exit path
@@ -500,9 +501,16 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
* and delayed __mmput doesn't matter that much
*/
if (!mmget_not_zero(mm)) {
- up_read(&mm->mmap_sem);
trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
+ /*
+ * MMF_OOM_SKIP is set by exit_mmap when the OOM
+ * reaper can't work on the mm anymore.
+ */
+ if (test_and_set_bit(MMF_OOM_SKIP, &mm->flags)) {
+ up_read(&mm->mmap_sem);
+ goto unlock_oom;
+ }
+ mmgot = false;
}
trace_start_task_reaping(tsk->pid);
@@ -547,7 +555,8 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
* different context because we shouldn't risk we get stuck there and
* put the oom_reaper out of the way.
*/
- mmput_async(mm);
+ if (mmgot)
+ mmput_async(mm);
trace_finish_task_reaping(tsk->pid);
unlock_oom:
mutex_unlock(&oom_lock);
This will perform the same as then the set_bit in __mmput can be
removed and test_and_set_bit doesn't cost more (at least on x86, on
other archs it requires an smp_mb too, but it's marginal difference),
still an atomic op that is.
> > 4) how is it safe to overwrite a VM_FAULT_RETRY that returns without
> > mmap_sem and then the arch code will release the mmap_sem despite
> > it was already released by handle_mm_fault? Anonymous memory faults
> > aren't common to return VM_FAULT_RETRY but an userfault
> > can. Shouldn't there be a block that prevents overwriting if
> > VM_FAULT_RETRY is set below? (not only VM_FAULT_ERROR)
> >
> > if (unlikely((current->flags & PF_KTHREAD) && !(ret & VM_FAULT_ERROR)
> > && test_bit(MMF_UNSTABLE, &vma->vm_mm->flags)))
> > ret = VM_FAULT_SIGBUS;
>
> I am not sure I understand what you mean and how this is related to the
> patch?
It's not related to the patch but it involves the OOM reaper as it
only happens when MMF_UNSTABLE is set which is set only by the OOM
reaper. I was simply reading the OOM reaper code and following up what
MMF_UNSTABLE does and it ringed a bell.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 15:31 ` Kirill A. Shutemov
@ 2017-07-25 16:04 ` Michal Hocko
2017-07-25 19:19 ` Andrea Arcangeli
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-25 16:04 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue 25-07-17 18:31:10, Kirill A. Shutemov wrote:
> On Tue, Jul 25, 2017 at 05:23:00PM +0200, Michal Hocko wrote:
> > what is stdev?
>
> Updated tables:
>
> 3 runs before the patch:
> Min. 1st Qu. Median Mean 3rd Qu. Max. Stdev
> 177200 205000 212900 217800 223700 2377000 32868
> 172400 201700 209700 214300 220600 1343000 31191
> 175700 203800 212300 217100 223000 1061000 31195
>
> 3 runs after the patch:
> Min. 1st Qu. Median Mean 3rd Qu. Max. Stdev
> 175900 204800 213000 216400 223600 1989000 27210
> 180300 210900 219600 223600 230200 3184000 32609
> 182100 212500 222000 226200 232700 1473000 32138
High std/avg ~15% matches my measurements (mine were even higher ~20%)
and that would suggest that 3% average difference is still somehing
within a "noise".
Anyway, I do not really need to take the lock unless the task is the
oom victim. Could you try whether those numbers improve if the lock is
conditional?
Thanks!
---
diff --git a/mm/mmap.c b/mm/mmap.c
index 0eeb658caa30..ca8a274485f8 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -44,6 +44,7 @@
#include <linux/userfaultfd_k.h>
#include <linux/moduleparam.h>
#include <linux/pkeys.h>
+#include <linux/oom.h>
#include <linux/uaccess.h>
#include <asm/cacheflush.h>
@@ -2997,7 +2998,8 @@ void exit_mmap(struct mm_struct *mm)
* oom reaper might race with exit_mmap so make sure we won't free
* page tables or unmap VMAs under its feet
*/
- down_write(&mm->mmap_sem);
+ if (tsk_is_oom_victim(current))
+ down_write(&mm->mmap_sem);
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(&tlb, 0, -1);
@@ -3012,7 +3014,8 @@ void exit_mmap(struct mm_struct *mm)
}
mm->mmap = NULL;
vm_unacct_memory(nr_accounted);
- up_write(&mm->mmap_sem);
+ if (tsk_is_oom_victim(current))
+ up_write(&mm->mmap_sem);
}
/* Insert vm structure into process list sorted by address
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 15:26 ` Andrea Arcangeli
@ 2017-07-25 15:45 ` Michal Hocko
2017-07-25 18:26 ` Andrea Arcangeli
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-25 15:45 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, linux-mm, LKML
On Tue 25-07-17 17:26:39, Andrea Arcangeli wrote:
> On Mon, Jul 24, 2017 at 09:23:32AM +0200, Michal Hocko wrote:
> > From: Michal Hocko <mhocko@suse.com>
> >
> > David has noticed that the oom killer might kill additional tasks while
> > the exiting oom victim hasn't terminated yet because the oom_reaper marks
> > the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
> > to 0. The race is as follows
> >
> > oom_reap_task do_exit
> > exit_mm
> > __oom_reap_task_mm
> > mmput
> > __mmput
> > mmget_not_zero # fails
> > exit_mmap # frees memory
> > set_bit(MMF_OOM_SKIP)
> >
> > The victim is still visible to the OOM killer until it is unhashed.
>
> I think this is a very minor problem, in the worst case you get a
> false positive oom kill, and it requires a race condition for it to
> happen. I wouldn't add mmap_sem in exit_mmap just for this considering
> the mmget_not_zero is already enough to leave exit_mmap alone.
That problem is real though as reported by David.
> Could you first clarify these points then I'll understand better what
> the above is about:
>
> 1) if exit_mmap runs for a long time with terabytes of RAM with
> mmap_sem held for writing like your patch does, wouldn't then
> oom_reap_task_mm fail the same way after a few tries on
> down_read_trylock? Despite your patch got applied? Isn't that
> simply moving the failure that leads to set_bit(MMF_OOM_SKIP) from
> mmget_not_zero to down_read_trylock?
No, it's not because the exclusive lock in exit_mmap is taken _after_ we
unmapped the address space. unmap_vmas will happily race with the oom
reaper.
> 2) why isn't __oom_reap_task_mm returning different retvals in case
> mmget_not_zero fails? What is the point to schedule_timeout
> and retry MAX_OOM_REAP_RETRIES times if mmget_not_zero caused it to
> return null as it can't do anything about such task anymore? Why
> are we scheduling those RETRIES times if mm_users is 0?
We are not. __oom_reap_task_mm will return true if the mm_users is 0 and
bail out.
> 3) if exit_mmap is freeing lots of memory already, why should there be
> another OOM immediately?
Because the memory can be freed from a different oom domain (e.g. a
different NUMA node).
> I thought oom reaper only was needed when
> the task on the right column couldn't reach the final mmput to set
> mm_users to 0. Why exactly is a problem that MMF_OOM_SKIP gets set
> on the mm, if exit_mmap is already guaranteed to be running?
MMF_OOM_SKIP will hide this task from the OOM killer and so we will
select another victim if we are still under oom. We _want_ to postpone
setting MMF_OOM_SKIP until we know that the oom victim no longer
interesting and we can go on to select another one.
> Why
> isn't the oom reaper happy to just stop in such case and wait it to
> complete?
Because there is no _guarantee_ that the final __mmput will release the
memory in finite time. And we cannot guarantee that longterm.
> exit_mmap doesn't even take the mmap_sem and it's running
> in R state, how would it block in a way that requires the OOM
> reaper to free memory from another process to complete?
it is not only about exit_mmap. __mmput calls into exit_aio and that can
wait for completion and there is no way to guarantee this will finish in
finite time.
> 4) how is it safe to overwrite a VM_FAULT_RETRY that returns without
> mmap_sem and then the arch code will release the mmap_sem despite
> it was already released by handle_mm_fault? Anonymous memory faults
> aren't common to return VM_FAULT_RETRY but an userfault
> can. Shouldn't there be a block that prevents overwriting if
> VM_FAULT_RETRY is set below? (not only VM_FAULT_ERROR)
>
> if (unlikely((current->flags & PF_KTHREAD) && !(ret & VM_FAULT_ERROR)
> && test_bit(MMF_UNSTABLE, &vma->vm_mm->flags)))
> ret = VM_FAULT_SIGBUS;
I am not sure I understand what you mean and how this is related to the
patch?
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 15:23 ` Michal Hocko
@ 2017-07-25 15:31 ` Kirill A. Shutemov
2017-07-25 16:04 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Kirill A. Shutemov @ 2017-07-25 15:31 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue, Jul 25, 2017 at 05:23:00PM +0200, Michal Hocko wrote:
> what is stdev?
Updated tables:
3 runs before the patch:
Min. 1st Qu. Median Mean 3rd Qu. Max. Stdev
177200 205000 212900 217800 223700 2377000 32868
172400 201700 209700 214300 220600 1343000 31191
175700 203800 212300 217100 223000 1061000 31195
3 runs after the patch:
Min. 1st Qu. Median Mean 3rd Qu. Max. Stdev
175900 204800 213000 216400 223600 1989000 27210
180300 210900 219600 223600 230200 3184000 32609
182100 212500 222000 226200 232700 1473000 32138
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 7:23 Michal Hocko
` (2 preceding siblings ...)
2017-07-24 16:42 ` kbuild test robot
@ 2017-07-25 15:26 ` Andrea Arcangeli
2017-07-25 15:45 ` Michal Hocko
3 siblings, 1 reply; 54+ messages in thread
From: Andrea Arcangeli @ 2017-07-25 15:26 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, linux-mm, LKML, Michal Hocko
On Mon, Jul 24, 2017 at 09:23:32AM +0200, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> David has noticed that the oom killer might kill additional tasks while
> the exiting oom victim hasn't terminated yet because the oom_reaper marks
> the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
> to 0. The race is as follows
>
> oom_reap_task do_exit
> exit_mm
> __oom_reap_task_mm
> mmput
> __mmput
> mmget_not_zero # fails
> exit_mmap # frees memory
> set_bit(MMF_OOM_SKIP)
>
> The victim is still visible to the OOM killer until it is unhashed.
I think this is a very minor problem, in the worst case you get a
false positive oom kill, and it requires a race condition for it to
happen. I wouldn't add mmap_sem in exit_mmap just for this considering
the mmget_not_zero is already enough to leave exit_mmap alone.
Could you first clarify these points then I'll understand better what
the above is about:
1) if exit_mmap runs for a long time with terabytes of RAM with
mmap_sem held for writing like your patch does, wouldn't then
oom_reap_task_mm fail the same way after a few tries on
down_read_trylock? Despite your patch got applied? Isn't that
simply moving the failure that leads to set_bit(MMF_OOM_SKIP) from
mmget_not_zero to down_read_trylock?
2) why isn't __oom_reap_task_mm returning different retvals in case
mmget_not_zero fails? What is the point to schedule_timeout
and retry MAX_OOM_REAP_RETRIES times if mmget_not_zero caused it to
return null as it can't do anything about such task anymore? Why
are we scheduling those RETRIES times if mm_users is 0?
3) if exit_mmap is freeing lots of memory already, why should there be
another OOM immediately? I thought oom reaper only was needed when
the task on the right column couldn't reach the final mmput to set
mm_users to 0. Why exactly is a problem that MMF_OOM_SKIP gets set
on the mm, if exit_mmap is already guaranteed to be running? Why
isn't the oom reaper happy to just stop in such case and wait it to
complete? exit_mmap doesn't even take the mmap_sem and it's running
in R state, how would it block in a way that requires the OOM
reaper to free memory from another process to complete?
4) how is it safe to overwrite a VM_FAULT_RETRY that returns without
mmap_sem and then the arch code will release the mmap_sem despite
it was already released by handle_mm_fault? Anonymous memory faults
aren't common to return VM_FAULT_RETRY but an userfault
can. Shouldn't there be a block that prevents overwriting if
VM_FAULT_RETRY is set below? (not only VM_FAULT_ERROR)
if (unlikely((current->flags & PF_KTHREAD) && !(ret & VM_FAULT_ERROR)
&& test_bit(MMF_UNSTABLE, &vma->vm_mm->flags)))
ret = VM_FAULT_SIGBUS;
Thanks,
Andrea
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 15:17 ` Kirill A. Shutemov
@ 2017-07-25 15:23 ` Michal Hocko
2017-07-25 15:31 ` Kirill A. Shutemov
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-25 15:23 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue 25-07-17 18:17:54, Kirill A. Shutemov wrote:
> > before the patch
> > min: 306300.00 max: 6731916.00 avg: 437962.07 std: 92898.30 nr: 100000
> >
> > after
> > min: 303196.00 max: 5728080.00 avg: 436081.87 std: 96165.98 nr: 100000
> >
> > The results are well withing noise as I would expect.
>
> I've silightly modified your test case: replaced cpuid + rdtsc with
> rdtscp. cpuid overhead is measurable in such tight loop.
>
> 3 runs before the patch:
> Min. 1st Qu. Median Mean 3rd Qu. Max.
> 177200 205000 212900 217800 223700 2377000
> 172400 201700 209700 214300 220600 1343000
> 175700 203800 212300 217100 223000 1061000
>
> 3 runs after the patch:
> Min. 1st Qu. Median Mean 3rd Qu. Max.
> 175900 204800 213000 216400 223600 1989000
> 180300 210900 219600 223600 230200 3184000
> 182100 212500 222000 226200 232700 1473000
>
> The difference is still measuarble. Around 3%.
what is stdev?
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 14:26 ` Michal Hocko
@ 2017-07-25 15:17 ` Kirill A. Shutemov
2017-07-25 15:23 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Kirill A. Shutemov @ 2017-07-25 15:17 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue, Jul 25, 2017 at 04:26:26PM +0200, Michal Hocko wrote:
> On Mon 24-07-17 18:11:46, Michal Hocko wrote:
> > On Mon 24-07-17 17:51:42, Kirill A. Shutemov wrote:
> > > On Mon, Jul 24, 2017 at 04:15:26PM +0200, Michal Hocko wrote:
> > [...]
> > > > What kind of scalability implication you have in mind? There is
> > > > basically a zero contention on the mmap_sem that late in the exit path
> > > > so this should be pretty much a fast path of the down_write. I agree it
> > > > is not 0 cost but the cost of the address space freeing should basically
> > > > make it a noise.
> > >
> > > Even in fast path case, it adds two atomic operation per-process. If the
> > > cache line is not exclusive to the core by the time of exit(2) it can be
> > > noticible.
> > >
> > > ... but I guess it's not very hot scenario.
> > >
> > > I guess I'm just too cautious here. :)
> >
> > I definitely did not want to handwave your concern. I just think we can
> > rule out the slow path and didn't think about the fast path overhead.
> >
> > > > > Should we do performance/scalability evaluation of the patch before
> > > > > getting it applied?
> > > >
> > > > What kind of test(s) would you be interested in?
> > >
> > > Can we at lest check that number of /bin/true we can spawn per second
> > > wouldn't be harmed by the patch? ;)
> >
> > OK, so measuring a single /bin/true doesn't tell anything so I've done
> > root@test1:~# cat a.sh
> > #!/bin/sh
> >
> > NR=$1
> > for i in $(seq $NR)
> > do
> > /bin/true
> > done
>
> I wanted to reduce a potential shell side effects so I've come with a
> simple program which forks and saves the timestamp before child exit and
> right after waitpid (see attached) and then measured it 100k times. Sure
> this still measures waitpid overhead and the signal delivery but this
> should be more or less constant on an idle system, right? See attached.
>
> before the patch
> min: 306300.00 max: 6731916.00 avg: 437962.07 std: 92898.30 nr: 100000
>
> after
> min: 303196.00 max: 5728080.00 avg: 436081.87 std: 96165.98 nr: 100000
>
> The results are well withing noise as I would expect.
I've silightly modified your test case: replaced cpuid + rdtsc with
rdtscp. cpuid overhead is measurable in such tight loop.
3 runs before the patch:
Min. 1st Qu. Median Mean 3rd Qu. Max.
177200 205000 212900 217800 223700 2377000
172400 201700 209700 214300 220600 1343000
175700 203800 212300 217100 223000 1061000
3 runs after the patch:
Min. 1st Qu. Median Mean 3rd Qu. Max.
175900 204800 213000 216400 223600 1989000
180300 210900 219600 223600 230200 3184000
182100 212500 222000 226200 232700 1473000
The difference is still measuarble. Around 3%.
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 15:07 ` Kirill A. Shutemov
@ 2017-07-25 15:15 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-25 15:15 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue 25-07-17 18:07:19, Kirill A. Shutemov wrote:
> On Tue, Jul 25, 2017 at 04:26:17PM +0200, Michal Hocko wrote:
[...]
> > Thanks for retesting Kirill. Are those numbers stable over runs? E.g.
> > the run without the patch has ~3% variance while the one with the patch
> > has it smaller. This sounds suspicious to me. There shouldn't be any
> > lock contention (except for the oom killer) so the lock shouldn't make
> > any difference wrt. variability.
>
> There's run-to-tun variability. I'll post new numbers for your new test.
That's what I've seen and the variance was quite large. I suspected
shell but if you look at the more dedicated test, the std over avg is
still quite large.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 14:26 ` Michal Hocko
@ 2017-07-25 15:07 ` Kirill A. Shutemov
2017-07-25 15:15 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Kirill A. Shutemov @ 2017-07-25 15:07 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue, Jul 25, 2017 at 04:26:17PM +0200, Michal Hocko wrote:
> On Tue 25-07-17 17:17:23, Kirill A. Shutemov wrote:
> [...]
> > Below are numbers for the same test case, but from bigger machine (48
> > threads, 64GiB of RAM).
> >
> > v4.13-rc2:
> >
> > Performance counter stats for './a.sh 100000' (5 runs):
> >
> > 159857.233790 task-clock:u (msec) # 1.000 CPUs utilized ( +- 3.21% )
> > 0 context-switches:u # 0.000 K/sec
> > 0 cpu-migrations:u # 0.000 K/sec
> > 8,761,843 page-faults:u # 0.055 M/sec ( +- 0.64% )
> > 38,725,763,026 cycles:u # 0.242 GHz ( +- 0.18% )
> > 272,691,643,016 stalled-cycles-frontend:u # 704.16% frontend cycles idle ( +- 3.16% )
> > 22,221,416,575 instructions:u # 0.57 insn per cycle
> > # 12.27 stalled cycles per insn ( +- 0.00% )
> > 5,306,829,649 branches:u # 33.197 M/sec ( +- 0.00% )
> > 240,783,599 branch-misses:u # 4.54% of all branches ( +- 0.15% )
> >
> > 159.808721098 seconds time elapsed ( +- 3.15% )
> >
> > v4.13-rc2 + the patch:
> >
> > Performance counter stats for './a.sh 100000' (5 runs):
> >
> > 167628.094556 task-clock:u (msec) # 1.007 CPUs utilized ( +- 1.63% )
> > 0 context-switches:u # 0.000 K/sec
> > 0 cpu-migrations:u # 0.000 K/sec
> > 8,838,314 page-faults:u # 0.053 M/sec ( +- 0.26% )
> > 38,862,240,137 cycles:u # 0.232 GHz ( +- 0.10% )
> > 282,105,057,553 stalled-cycles-frontend:u # 725.91% frontend cycles idle ( +- 1.64% )
> > 22,219,273,623 instructions:u # 0.57 insn per cycle
> > # 12.70 stalled cycles per insn ( +- 0.00% )
> > 5,306,165,194 branches:u # 31.654 M/sec ( +- 0.00% )
> > 240,473,075 branch-misses:u # 4.53% of all branches ( +- 0.07% )
> >
> > 166.497005412 seconds time elapsed ( +- 1.61% )
> >
> > IMO, there is something to think about. ~4% slowdown is not insignificant.
> > I expect effect to be bigger for larger machines.
>
> Thanks for retesting Kirill. Are those numbers stable over runs? E.g.
> the run without the patch has ~3% variance while the one with the patch
> has it smaller. This sounds suspicious to me. There shouldn't be any
> lock contention (except for the oom killer) so the lock shouldn't make
> any difference wrt. variability.
There's run-to-tun variability. I'll post new numbers for your new test.
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 16:11 ` Michal Hocko
2017-07-25 14:17 ` Kirill A. Shutemov
@ 2017-07-25 14:26 ` Michal Hocko
2017-07-25 15:17 ` Kirill A. Shutemov
1 sibling, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-25 14:26 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]
On Mon 24-07-17 18:11:46, Michal Hocko wrote:
> On Mon 24-07-17 17:51:42, Kirill A. Shutemov wrote:
> > On Mon, Jul 24, 2017 at 04:15:26PM +0200, Michal Hocko wrote:
> [...]
> > > What kind of scalability implication you have in mind? There is
> > > basically a zero contention on the mmap_sem that late in the exit path
> > > so this should be pretty much a fast path of the down_write. I agree it
> > > is not 0 cost but the cost of the address space freeing should basically
> > > make it a noise.
> >
> > Even in fast path case, it adds two atomic operation per-process. If the
> > cache line is not exclusive to the core by the time of exit(2) it can be
> > noticible.
> >
> > ... but I guess it's not very hot scenario.
> >
> > I guess I'm just too cautious here. :)
>
> I definitely did not want to handwave your concern. I just think we can
> rule out the slow path and didn't think about the fast path overhead.
>
> > > > Should we do performance/scalability evaluation of the patch before
> > > > getting it applied?
> > >
> > > What kind of test(s) would you be interested in?
> >
> > Can we at lest check that number of /bin/true we can spawn per second
> > wouldn't be harmed by the patch? ;)
>
> OK, so measuring a single /bin/true doesn't tell anything so I've done
> root@test1:~# cat a.sh
> #!/bin/sh
>
> NR=$1
> for i in $(seq $NR)
> do
> /bin/true
> done
I wanted to reduce a potential shell side effects so I've come with a
simple program which forks and saves the timestamp before child exit and
right after waitpid (see attached) and then measured it 100k times. Sure
this still measures waitpid overhead and the signal delivery but this
should be more or less constant on an idle system, right? See attached.
before the patch
min: 306300.00 max: 6731916.00 avg: 437962.07 std: 92898.30 nr: 100000
after
min: 303196.00 max: 5728080.00 avg: 436081.87 std: 96165.98 nr: 100000
The results are well withing noise as I would expect.
--
Michal Hocko
SUSE Labs
[-- Attachment #2: exit_time.c --]
[-- Type: text/x-csrc, Size: 985 bytes --]
#include <sys/mman.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <inttypes.h>
#include <unistd.h>
#define NR_FORKS 100000
static inline uint64_t get_cycles(void)
{
uint64_t t;
volatile int dont_remove __attribute__((unused));
unsigned tmp;
__asm volatile ("cpuid" : "=a"(tmp), "=b"(tmp), "=c"(tmp), "=d"(tmp): "a" (0));
dont_remove = tmp;
__asm volatile ("rdtsc" : "=A"(t));
return t;
}
int main(int argc, char **argv)
{
void *addr = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_ANON|MAP_SHARED, -1, 0);
int i = NR_FORKS, j = 1;
assert(addr != MAP_FAILED);
while (i-- > 0) {
pid_t child = fork();
uint64_t before, after;
assert(child != -1);
if (!child) {
*(uint64_t *)addr = get_cycles();
return 0;
}
assert(child == waitpid(child, NULL, 0));
before = *(uint64_t *)addr;
after = get_cycles();
printf("%u\n", (unsigned)(after - before));
fflush(stdout);
}
return 0;
}
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-25 14:17 ` Kirill A. Shutemov
@ 2017-07-25 14:26 ` Michal Hocko
2017-07-25 15:07 ` Kirill A. Shutemov
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-25 14:26 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue 25-07-17 17:17:23, Kirill A. Shutemov wrote:
[...]
> Below are numbers for the same test case, but from bigger machine (48
> threads, 64GiB of RAM).
>
> v4.13-rc2:
>
> Performance counter stats for './a.sh 100000' (5 runs):
>
> 159857.233790 task-clock:u (msec) # 1.000 CPUs utilized ( +- 3.21% )
> 0 context-switches:u # 0.000 K/sec
> 0 cpu-migrations:u # 0.000 K/sec
> 8,761,843 page-faults:u # 0.055 M/sec ( +- 0.64% )
> 38,725,763,026 cycles:u # 0.242 GHz ( +- 0.18% )
> 272,691,643,016 stalled-cycles-frontend:u # 704.16% frontend cycles idle ( +- 3.16% )
> 22,221,416,575 instructions:u # 0.57 insn per cycle
> # 12.27 stalled cycles per insn ( +- 0.00% )
> 5,306,829,649 branches:u # 33.197 M/sec ( +- 0.00% )
> 240,783,599 branch-misses:u # 4.54% of all branches ( +- 0.15% )
>
> 159.808721098 seconds time elapsed ( +- 3.15% )
>
> v4.13-rc2 + the patch:
>
> Performance counter stats for './a.sh 100000' (5 runs):
>
> 167628.094556 task-clock:u (msec) # 1.007 CPUs utilized ( +- 1.63% )
> 0 context-switches:u # 0.000 K/sec
> 0 cpu-migrations:u # 0.000 K/sec
> 8,838,314 page-faults:u # 0.053 M/sec ( +- 0.26% )
> 38,862,240,137 cycles:u # 0.232 GHz ( +- 0.10% )
> 282,105,057,553 stalled-cycles-frontend:u # 725.91% frontend cycles idle ( +- 1.64% )
> 22,219,273,623 instructions:u # 0.57 insn per cycle
> # 12.70 stalled cycles per insn ( +- 0.00% )
> 5,306,165,194 branches:u # 31.654 M/sec ( +- 0.00% )
> 240,473,075 branch-misses:u # 4.53% of all branches ( +- 0.07% )
>
> 166.497005412 seconds time elapsed ( +- 1.61% )
>
> IMO, there is something to think about. ~4% slowdown is not insignificant.
> I expect effect to be bigger for larger machines.
Thanks for retesting Kirill. Are those numbers stable over runs? E.g.
the run without the patch has ~3% variance while the one with the patch
has it smaller. This sounds suspicious to me. There shouldn't be any
lock contention (except for the oom killer) so the lock shouldn't make
any difference wrt. variability.
Also I was about to post a more targeted test. Could you try it with it
as well, please?
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 16:11 ` Michal Hocko
@ 2017-07-25 14:17 ` Kirill A. Shutemov
2017-07-25 14:26 ` Michal Hocko
2017-07-25 14:26 ` Michal Hocko
1 sibling, 1 reply; 54+ messages in thread
From: Kirill A. Shutemov @ 2017-07-25 14:17 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Mon, Jul 24, 2017 at 06:11:47PM +0200, Michal Hocko wrote:
> On Mon 24-07-17 17:51:42, Kirill A. Shutemov wrote:
> > On Mon, Jul 24, 2017 at 04:15:26PM +0200, Michal Hocko wrote:
> [...]
> > > What kind of scalability implication you have in mind? There is
> > > basically a zero contention on the mmap_sem that late in the exit path
> > > so this should be pretty much a fast path of the down_write. I agree it
> > > is not 0 cost but the cost of the address space freeing should basically
> > > make it a noise.
> >
> > Even in fast path case, it adds two atomic operation per-process. If the
> > cache line is not exclusive to the core by the time of exit(2) it can be
> > noticible.
> >
> > ... but I guess it's not very hot scenario.
> >
> > I guess I'm just too cautious here. :)
>
> I definitely did not want to handwave your concern. I just think we can
> rule out the slow path and didn't think about the fast path overhead.
>
> > > > Should we do performance/scalability evaluation of the patch before
> > > > getting it applied?
> > >
> > > What kind of test(s) would you be interested in?
> >
> > Can we at lest check that number of /bin/true we can spawn per second
> > wouldn't be harmed by the patch? ;)
>
> OK, so measuring a single /bin/true doesn't tell anything so I've done
> root@test1:~# cat a.sh
> #!/bin/sh
>
> NR=$1
> for i in $(seq $NR)
> do
> /bin/true
> done
>
> in my virtual machine (on a otherwise idle host) with 4 cpus and 2GB of
> RAM
>
> Unpatched kernel
> root@test1:~# /usr/bin/time -v ./a.sh 100000
> Command being timed: "./a.sh 100000"
> User time (seconds): 53.57
> System time (seconds): 26.12
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.46
> root@test1:~# /usr/bin/time -v ./a.sh 100000
> Command being timed: "./a.sh 100000"
> User time (seconds): 53.90
> System time (seconds): 26.23
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.77
> root@test1:~# /usr/bin/time -v ./a.sh 100000
> Command being timed: "./a.sh 100000"
> User time (seconds): 54.02
> System time (seconds): 26.18
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.92
>
> patched kernel
> root@test1:~# /usr/bin/time -v ./a.sh 100000
> Command being timed: "./a.sh 100000"
> User time (seconds): 53.81
> System time (seconds): 26.55
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.99
> root@test1:~# /usr/bin/time -v ./a.sh 100000
> Command being timed: "./a.sh 100000"
> User time (seconds): 53.78
> System time (seconds): 26.15
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.67
> root@test1:~# /usr/bin/time -v ./a.sh 100000
> Command being timed: "./a.sh 100000"
> User time (seconds): 54.08
> System time (seconds): 26.87
> Percent of CPU this job got: 100%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:20.52
>
> the results very quite a lot (have a look at the user time which
> shouldn't have no reason to vary at all - maybe the virtual machine
> aspect?). I would say that we are still reasonably close to a noise
> here. Considering that /bin/true would close to the worst case I think
> this looks reasonably. What do you think?
>
> If you absolutely insist, I can make the lock conditional only for oom
> victims. That would still mean current->signal->oom_mm pointers fetches
> and a 2 branches.
Below are numbers for the same test case, but from bigger machine (48
threads, 64GiB of RAM).
v4.13-rc2:
Performance counter stats for './a.sh 100000' (5 runs):
159857.233790 task-clock:u (msec) # 1.000 CPUs utilized ( +- 3.21% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
8,761,843 page-faults:u # 0.055 M/sec ( +- 0.64% )
38,725,763,026 cycles:u # 0.242 GHz ( +- 0.18% )
272,691,643,016 stalled-cycles-frontend:u # 704.16% frontend cycles idle ( +- 3.16% )
22,221,416,575 instructions:u # 0.57 insn per cycle
# 12.27 stalled cycles per insn ( +- 0.00% )
5,306,829,649 branches:u # 33.197 M/sec ( +- 0.00% )
240,783,599 branch-misses:u # 4.54% of all branches ( +- 0.15% )
159.808721098 seconds time elapsed ( +- 3.15% )
v4.13-rc2 + the patch:
Performance counter stats for './a.sh 100000' (5 runs):
167628.094556 task-clock:u (msec) # 1.007 CPUs utilized ( +- 1.63% )
0 context-switches:u # 0.000 K/sec
0 cpu-migrations:u # 0.000 K/sec
8,838,314 page-faults:u # 0.053 M/sec ( +- 0.26% )
38,862,240,137 cycles:u # 0.232 GHz ( +- 0.10% )
282,105,057,553 stalled-cycles-frontend:u # 725.91% frontend cycles idle ( +- 1.64% )
22,219,273,623 instructions:u # 0.57 insn per cycle
# 12.70 stalled cycles per insn ( +- 0.00% )
5,306,165,194 branches:u # 31.654 M/sec ( +- 0.00% )
240,473,075 branch-misses:u # 4.53% of all branches ( +- 0.07% )
166.497005412 seconds time elapsed ( +- 1.61% )
IMO, there is something to think about. ~4% slowdown is not insignificant.
I expect effect to be bigger for larger machines.
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 16:42 ` kbuild test robot
@ 2017-07-24 18:12 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-24 18:12 UTC (permalink / raw)
To: kbuild test robot
Cc: kbuild-all, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Tue 25-07-17 00:42:05, kbuild test robot wrote:
> Hi Michal,
>
> [auto build test ERROR on mmotm/master]
> [also build test ERROR on v4.13-rc2 next-20170724]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
>
> url: https://github.com/0day-ci/linux/commits/Michal-Hocko/mm-oom-allow-oom-reaper-to-race-with-exit_mmap/20170724-233159
> base: git://git.cmpxchg.org/linux-mmotm.git master
> config: x86_64-randconfig-x016-201730 (attached as .config)
> compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
> reproduce:
> # save the attached .config to linux build tree
> make ARCH=x86_64
>
> All error/warnings (new ones prefixed by >>):
>
> mm/oom_kill.c: In function '__oom_reap_task_mm':
> >> mm/oom_kill.c:523:9: error: 'ret' undeclared (first use in this function)
> return ret;
> ^~~
Fixed by http://lkml.kernel.org/r/20170724152703.GP25221@dhcp22.suse.cz
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 7:23 Michal Hocko
2017-07-24 14:00 ` Kirill A. Shutemov
2017-07-24 15:27 ` Michal Hocko
@ 2017-07-24 16:42 ` kbuild test robot
2017-07-24 18:12 ` Michal Hocko
2017-07-25 15:26 ` Andrea Arcangeli
3 siblings, 1 reply; 54+ messages in thread
From: kbuild test robot @ 2017-07-24 16:42 UTC (permalink / raw)
To: Michal Hocko
Cc: kbuild-all, Andrew Morton, David Rientjes, Tetsuo Handa,
Oleg Nesterov, Hugh Dickins, Andrea Arcangeli, linux-mm, LKML,
Michal Hocko
[-- Attachment #1: Type: text/plain, Size: 5793 bytes --]
Hi Michal,
[auto build test ERROR on mmotm/master]
[also build test ERROR on v4.13-rc2 next-20170724]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Michal-Hocko/mm-oom-allow-oom-reaper-to-race-with-exit_mmap/20170724-233159
base: git://git.cmpxchg.org/linux-mmotm.git master
config: x86_64-randconfig-x016-201730 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All error/warnings (new ones prefixed by >>):
mm/oom_kill.c: In function '__oom_reap_task_mm':
>> mm/oom_kill.c:523:9: error: 'ret' undeclared (first use in this function)
return ret;
^~~
mm/oom_kill.c:523:9: note: each undeclared identifier is reported only once for each function it appears in
>> mm/oom_kill.c:524:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
vim +/ret +523 mm/oom_kill.c
03049269d Michal Hocko 2016-03-25 468
7ebffa455 Tetsuo Handa 2016-10-07 469 static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
aac453635 Michal Hocko 2016-03-25 470 {
aac453635 Michal Hocko 2016-03-25 471 struct mmu_gather tlb;
aac453635 Michal Hocko 2016-03-25 472 struct vm_area_struct *vma;
e2fe14564 Michal Hocko 2016-05-27 473
aac453635 Michal Hocko 2016-03-25 474 if (!down_read_trylock(&mm->mmap_sem)) {
fa60da35c Andrew Morton 2017-07-14 475 trace_skip_task_reaping(tsk->pid);
ed7a155c6 Michal Hocko 2017-07-24 476 return false;
e5e3f4c4f Michal Hocko 2016-07-26 477 }
e5e3f4c4f Michal Hocko 2016-07-26 478
ed7a155c6 Michal Hocko 2017-07-24 479 /* There is nothing to reap so bail out without signs in the log */
ed7a155c6 Michal Hocko 2017-07-24 480 if (!mm->mmap)
ed7a155c6 Michal Hocko 2017-07-24 481 goto unlock;
aac453635 Michal Hocko 2016-03-25 482
fa60da35c Andrew Morton 2017-07-14 483 trace_start_task_reaping(tsk->pid);
fa60da35c Andrew Morton 2017-07-14 484
3f70dc38c Michal Hocko 2016-10-07 485 /*
3f70dc38c Michal Hocko 2016-10-07 486 * Tell all users of get_user/copy_from_user etc... that the content
3f70dc38c Michal Hocko 2016-10-07 487 * is no longer stable. No barriers really needed because unmapping
3f70dc38c Michal Hocko 2016-10-07 488 * should imply barriers already and the reader would hit a page fault
3f70dc38c Michal Hocko 2016-10-07 489 * if it stumbled over a reaped memory.
3f70dc38c Michal Hocko 2016-10-07 490 */
3f70dc38c Michal Hocko 2016-10-07 491 set_bit(MMF_UNSTABLE, &mm->flags);
3f70dc38c Michal Hocko 2016-10-07 492
aac453635 Michal Hocko 2016-03-25 493 tlb_gather_mmu(&tlb, mm, 0, -1);
aac453635 Michal Hocko 2016-03-25 494 for (vma = mm->mmap ; vma; vma = vma->vm_next) {
235190738 Kirill A. Shutemov 2017-02-22 495 if (!can_madv_dontneed_vma(vma))
aac453635 Michal Hocko 2016-03-25 496 continue;
aac453635 Michal Hocko 2016-03-25 497
aac453635 Michal Hocko 2016-03-25 498 /*
aac453635 Michal Hocko 2016-03-25 499 * Only anonymous pages have a good chance to be dropped
aac453635 Michal Hocko 2016-03-25 500 * without additional steps which we cannot afford as we
aac453635 Michal Hocko 2016-03-25 501 * are OOM already.
aac453635 Michal Hocko 2016-03-25 502 *
aac453635 Michal Hocko 2016-03-25 503 * We do not even care about fs backed pages because all
aac453635 Michal Hocko 2016-03-25 504 * which are reclaimable have already been reclaimed and
aac453635 Michal Hocko 2016-03-25 505 * we do not want to block exit_mmap by keeping mm ref
aac453635 Michal Hocko 2016-03-25 506 * count elevated without a good reason.
aac453635 Michal Hocko 2016-03-25 507 */
aac453635 Michal Hocko 2016-03-25 508 if (vma_is_anonymous(vma) || !(vma->vm_flags & VM_SHARED))
aac453635 Michal Hocko 2016-03-25 509 unmap_page_range(&tlb, vma, vma->vm_start, vma->vm_end,
3e8715fdc Kirill A. Shutemov 2017-02-22 510 NULL);
aac453635 Michal Hocko 2016-03-25 511 }
aac453635 Michal Hocko 2016-03-25 512 tlb_finish_mmu(&tlb, 0, -1);
bc448e897 Michal Hocko 2016-03-25 513 pr_info("oom_reaper: reaped process %d (%s), now anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",
bc448e897 Michal Hocko 2016-03-25 514 task_pid_nr(tsk), tsk->comm,
bc448e897 Michal Hocko 2016-03-25 515 K(get_mm_counter(mm, MM_ANONPAGES)),
bc448e897 Michal Hocko 2016-03-25 516 K(get_mm_counter(mm, MM_FILEPAGES)),
bc448e897 Michal Hocko 2016-03-25 517 K(get_mm_counter(mm, MM_SHMEMPAGES)));
36324a990 Michal Hocko 2016-03-25 518
fa60da35c Andrew Morton 2017-07-14 519 trace_finish_task_reaping(tsk->pid);
ed7a155c6 Michal Hocko 2017-07-24 520 unlock:
ed7a155c6 Michal Hocko 2017-07-24 521 up_read(&mm->mmap_sem);
ed7a155c6 Michal Hocko 2017-07-24 522
aac453635 Michal Hocko 2016-03-25 @523 return ret;
aac453635 Michal Hocko 2016-03-25 @524 }
aac453635 Michal Hocko 2016-03-25 525
:::::: The code at line 523 was first introduced by commit
:::::: aac453635549699c13a84ea1456d5b0e574ef855 mm, oom: introduce oom reaper
:::::: TO: Michal Hocko <mhocko@suse.com>
:::::: CC: Linus Torvalds <torvalds@linux-foundation.org>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 31959 bytes --]
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 14:51 ` Kirill A. Shutemov
@ 2017-07-24 16:11 ` Michal Hocko
2017-07-25 14:17 ` Kirill A. Shutemov
2017-07-25 14:26 ` Michal Hocko
0 siblings, 2 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-24 16:11 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Mon 24-07-17 17:51:42, Kirill A. Shutemov wrote:
> On Mon, Jul 24, 2017 at 04:15:26PM +0200, Michal Hocko wrote:
[...]
> > What kind of scalability implication you have in mind? There is
> > basically a zero contention on the mmap_sem that late in the exit path
> > so this should be pretty much a fast path of the down_write. I agree it
> > is not 0 cost but the cost of the address space freeing should basically
> > make it a noise.
>
> Even in fast path case, it adds two atomic operation per-process. If the
> cache line is not exclusive to the core by the time of exit(2) it can be
> noticible.
>
> ... but I guess it's not very hot scenario.
>
> I guess I'm just too cautious here. :)
I definitely did not want to handwave your concern. I just think we can
rule out the slow path and didn't think about the fast path overhead.
> > > Should we do performance/scalability evaluation of the patch before
> > > getting it applied?
> >
> > What kind of test(s) would you be interested in?
>
> Can we at lest check that number of /bin/true we can spawn per second
> wouldn't be harmed by the patch? ;)
OK, so measuring a single /bin/true doesn't tell anything so I've done
root@test1:~# cat a.sh
#!/bin/sh
NR=$1
for i in $(seq $NR)
do
/bin/true
done
in my virtual machine (on a otherwise idle host) with 4 cpus and 2GB of
RAM
Unpatched kernel
root@test1:~# /usr/bin/time -v ./a.sh 100000
Command being timed: "./a.sh 100000"
User time (seconds): 53.57
System time (seconds): 26.12
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.46
root@test1:~# /usr/bin/time -v ./a.sh 100000
Command being timed: "./a.sh 100000"
User time (seconds): 53.90
System time (seconds): 26.23
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.77
root@test1:~# /usr/bin/time -v ./a.sh 100000
Command being timed: "./a.sh 100000"
User time (seconds): 54.02
System time (seconds): 26.18
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.92
patched kernel
root@test1:~# /usr/bin/time -v ./a.sh 100000
Command being timed: "./a.sh 100000"
User time (seconds): 53.81
System time (seconds): 26.55
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.99
root@test1:~# /usr/bin/time -v ./a.sh 100000
Command being timed: "./a.sh 100000"
User time (seconds): 53.78
System time (seconds): 26.15
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:19.67
root@test1:~# /usr/bin/time -v ./a.sh 100000
Command being timed: "./a.sh 100000"
User time (seconds): 54.08
System time (seconds): 26.87
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:20.52
the results very quite a lot (have a look at the user time which
shouldn't have no reason to vary at all - maybe the virtual machine
aspect?). I would say that we are still reasonably close to a noise
here. Considering that /bin/true would close to the worst case I think
this looks reasonably. What do you think?
If you absolutely insist, I can make the lock conditional only for oom
victims. That would still mean current->signal->oom_mm pointers fetches
and a 2 branches.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 7:23 Michal Hocko
2017-07-24 14:00 ` Kirill A. Shutemov
@ 2017-07-24 15:27 ` Michal Hocko
2017-07-24 16:42 ` kbuild test robot
2017-07-25 15:26 ` Andrea Arcangeli
3 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-24 15:27 UTC (permalink / raw)
To: Andrew Morton
Cc: David Rientjes, Tetsuo Handa, Oleg Nesterov, Hugh Dickins,
Andrea Arcangeli, linux-mm, LKML
Dohh, the full conflict resolution didn't make it into the commit. The
full patch is below
---
>From be69b355a56649167dce901d24c2296ef3a3f7ec Mon Sep 17 00:00:00 2001
From: Michal Hocko <mhocko@suse.com>
Date: Mon, 24 Jul 2017 09:23:32 +0200
Subject: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
David has noticed that the oom killer might kill additional tasks while
the exiting oom victim hasn't terminated yet because the oom_reaper marks
the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
to 0. The race is as follows
oom_reap_task do_exit
exit_mm
__oom_reap_task_mm
mmput
__mmput
mmget_not_zero # fails
exit_mmap # frees memory
set_bit(MMF_OOM_SKIP)
The victim is still visible to the OOM killer until it is unhashed.
Currently we try to reduce a risk of this race by taking oom_lock
and wait for out_of_memory sleep while holding the lock to give the
victim some time to exit. This is quite suboptimal approach because
there is no guarantee the victim (especially a large one) will manage
to unmap its address space and free enough memory to the particular oom
domain which needs a memory (e.g. a specific NUMA node).
Fix this problem by allowing __oom_reap_task_mm and __mmput path to
race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
to run in parallel with other unmappers (hence the mmap_sem for read).
The only tricky part is to exclude page tables tear down and all
operations which modify the address space in the __mmput path. exit_mmap
doesn't expect any other users so it doesn't use any locking. Nothing
really forbids us to use mmap_sem for write, though. In fact we are
already relying on this lock earlier in the __mmput path to synchronize
with ksm and khugepaged.
Take the exclusive mmap_sem when calling free_pgtables and destroying
vmas to sync with __oom_reap_task_mm which take the lock for read. All
other operations can safely race with the parallel unmap.
Changes
- bail on null mm->mmap early as per David Rientjes
Reported-by: David Rientjes <rientjes@google.com>
Fixes: 26db62f179d1 ("oom: keep mm of the killed task available")
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
mm/mmap.c | 7 +++++++
mm/oom_kill.c | 47 ++++++++---------------------------------------
2 files changed, 15 insertions(+), 39 deletions(-)
diff --git a/mm/mmap.c b/mm/mmap.c
index 24e9261bdcc0..0eeb658caa30 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2993,6 +2993,11 @@ void exit_mmap(struct mm_struct *mm)
/* Use -1 here to ensure all VMAs in the mm are unmapped */
unmap_vmas(&tlb, vma, 0, -1);
+ /*
+ * oom reaper might race with exit_mmap so make sure we won't free
+ * page tables or unmap VMAs under its feet
+ */
+ down_write(&mm->mmap_sem);
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(&tlb, 0, -1);
@@ -3005,7 +3010,9 @@ void exit_mmap(struct mm_struct *mm)
nr_accounted += vma_pages(vma);
vma = remove_vma(vma);
}
+ mm->mmap = NULL;
vm_unacct_memory(nr_accounted);
+ up_write(&mm->mmap_sem);
}
/* Insert vm structure into process list sorted by address
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f030c1c..b1c96e1910f2 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -470,40 +470,15 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
{
struct mmu_gather tlb;
struct vm_area_struct *vma;
- bool ret = true;
-
- /*
- * We have to make sure to not race with the victim exit path
- * and cause premature new oom victim selection:
- * __oom_reap_task_mm exit_mm
- * mmget_not_zero
- * mmput
- * atomic_dec_and_test
- * exit_oom_victim
- * [...]
- * out_of_memory
- * select_bad_process
- * # no TIF_MEMDIE task selects new victim
- * unmap_page_range # frees some memory
- */
- mutex_lock(&oom_lock);
if (!down_read_trylock(&mm->mmap_sem)) {
- ret = false;
trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
+ return false;
}
- /*
- * increase mm_users only after we know we will reap something so
- * that the mmput_async is called only when we have reaped something
- * and delayed __mmput doesn't matter that much
- */
- if (!mmget_not_zero(mm)) {
- up_read(&mm->mmap_sem);
- trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
- }
+ /* There is nothing to reap so bail out without signs in the log */
+ if (!mm->mmap)
+ goto unlock;
trace_start_task_reaping(tsk->pid);
@@ -540,18 +515,12 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
K(get_mm_counter(mm, MM_ANONPAGES)),
K(get_mm_counter(mm, MM_FILEPAGES)),
K(get_mm_counter(mm, MM_SHMEMPAGES)));
- up_read(&mm->mmap_sem);
- /*
- * Drop our reference but make sure the mmput slow path is called from a
- * different context because we shouldn't risk we get stuck there and
- * put the oom_reaper out of the way.
- */
- mmput_async(mm);
trace_finish_task_reaping(tsk->pid);
-unlock_oom:
- mutex_unlock(&oom_lock);
- return ret;
+unlock:
+ up_read(&mm->mmap_sem);
+
+ return true;
}
#define MAX_OOM_REAP_RETRIES 10
--
2.13.2
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 14:15 ` Michal Hocko
@ 2017-07-24 14:51 ` Kirill A. Shutemov
2017-07-24 16:11 ` Michal Hocko
0 siblings, 1 reply; 54+ messages in thread
From: Kirill A. Shutemov @ 2017-07-24 14:51 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Mon, Jul 24, 2017 at 04:15:26PM +0200, Michal Hocko wrote:
> On Mon 24-07-17 17:00:08, Kirill A. Shutemov wrote:
> > On Mon, Jul 24, 2017 at 09:23:32AM +0200, Michal Hocko wrote:
> > > From: Michal Hocko <mhocko@suse.com>
> > >
> > > David has noticed that the oom killer might kill additional tasks while
> > > the exiting oom victim hasn't terminated yet because the oom_reaper marks
> > > the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
> > > to 0. The race is as follows
> > >
> > > oom_reap_task do_exit
> > > exit_mm
> > > __oom_reap_task_mm
> > > mmput
> > > __mmput
> > > mmget_not_zero # fails
> > > exit_mmap # frees memory
> > > set_bit(MMF_OOM_SKIP)
> > >
> > > The victim is still visible to the OOM killer until it is unhashed.
> > >
> > > Currently we try to reduce a risk of this race by taking oom_lock
> > > and wait for out_of_memory sleep while holding the lock to give the
> > > victim some time to exit. This is quite suboptimal approach because
> > > there is no guarantee the victim (especially a large one) will manage
> > > to unmap its address space and free enough memory to the particular oom
> > > domain which needs a memory (e.g. a specific NUMA node).
> > >
> > > Fix this problem by allowing __oom_reap_task_mm and __mmput path to
> > > race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
> > > to run in parallel with other unmappers (hence the mmap_sem for read).
> > >
> > > The only tricky part is to exclude page tables tear down and all
> > > operations which modify the address space in the __mmput path. exit_mmap
> > > doesn't expect any other users so it doesn't use any locking. Nothing
> > > really forbids us to use mmap_sem for write, though. In fact we are
> > > already relying on this lock earlier in the __mmput path to synchronize
> > > with ksm and khugepaged.
> >
> > That's true, but we take mmap_sem there for small portion of cases.
> >
> > It's quite different from taking the lock unconditionally. I'm worry about
> > scalability implication of such move. On bigger machines it can be big
> > hit.
>
> What kind of scalability implication you have in mind? There is
> basically a zero contention on the mmap_sem that late in the exit path
> so this should be pretty much a fast path of the down_write. I agree it
> is not 0 cost but the cost of the address space freeing should basically
> make it a noise.
Even in fast path case, it adds two atomic operation per-process. If the
cache line is not exclusive to the core by the time of exit(2) it can be
noticible.
... but I guess it's not very hot scenario.
I guess I'm just too cautious here. :)
> > Should we do performance/scalability evaluation of the patch before
> > getting it applied?
>
> What kind of test(s) would you be interested in?
Can we at lest check that number of /bin/true we can spawn per second
wouldn't be harmed by the patch? ;)
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 14:00 ` Kirill A. Shutemov
@ 2017-07-24 14:15 ` Michal Hocko
2017-07-24 14:51 ` Kirill A. Shutemov
0 siblings, 1 reply; 54+ messages in thread
From: Michal Hocko @ 2017-07-24 14:15 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML
On Mon 24-07-17 17:00:08, Kirill A. Shutemov wrote:
> On Mon, Jul 24, 2017 at 09:23:32AM +0200, Michal Hocko wrote:
> > From: Michal Hocko <mhocko@suse.com>
> >
> > David has noticed that the oom killer might kill additional tasks while
> > the exiting oom victim hasn't terminated yet because the oom_reaper marks
> > the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
> > to 0. The race is as follows
> >
> > oom_reap_task do_exit
> > exit_mm
> > __oom_reap_task_mm
> > mmput
> > __mmput
> > mmget_not_zero # fails
> > exit_mmap # frees memory
> > set_bit(MMF_OOM_SKIP)
> >
> > The victim is still visible to the OOM killer until it is unhashed.
> >
> > Currently we try to reduce a risk of this race by taking oom_lock
> > and wait for out_of_memory sleep while holding the lock to give the
> > victim some time to exit. This is quite suboptimal approach because
> > there is no guarantee the victim (especially a large one) will manage
> > to unmap its address space and free enough memory to the particular oom
> > domain which needs a memory (e.g. a specific NUMA node).
> >
> > Fix this problem by allowing __oom_reap_task_mm and __mmput path to
> > race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
> > to run in parallel with other unmappers (hence the mmap_sem for read).
> >
> > The only tricky part is to exclude page tables tear down and all
> > operations which modify the address space in the __mmput path. exit_mmap
> > doesn't expect any other users so it doesn't use any locking. Nothing
> > really forbids us to use mmap_sem for write, though. In fact we are
> > already relying on this lock earlier in the __mmput path to synchronize
> > with ksm and khugepaged.
>
> That's true, but we take mmap_sem there for small portion of cases.
>
> It's quite different from taking the lock unconditionally. I'm worry about
> scalability implication of such move. On bigger machines it can be big
> hit.
What kind of scalability implication you have in mind? There is
basically a zero contention on the mmap_sem that late in the exit path
so this should be pretty much a fast path of the down_write. I agree it
is not 0 cost but the cost of the address space freeing should basically
make it a noise.
> Should we do performance/scalability evaluation of the patch before
> getting it applied?
What kind of test(s) would you be interested in?
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH] mm, oom: allow oom reaper to race with exit_mmap
2017-07-24 7:23 Michal Hocko
@ 2017-07-24 14:00 ` Kirill A. Shutemov
2017-07-24 14:15 ` Michal Hocko
2017-07-24 15:27 ` Michal Hocko
` (2 subsequent siblings)
3 siblings, 1 reply; 54+ messages in thread
From: Kirill A. Shutemov @ 2017-07-24 14:00 UTC (permalink / raw)
To: Michal Hocko
Cc: Andrew Morton, David Rientjes, Tetsuo Handa, Oleg Nesterov,
Hugh Dickins, Andrea Arcangeli, linux-mm, LKML, Michal Hocko
On Mon, Jul 24, 2017 at 09:23:32AM +0200, Michal Hocko wrote:
> From: Michal Hocko <mhocko@suse.com>
>
> David has noticed that the oom killer might kill additional tasks while
> the exiting oom victim hasn't terminated yet because the oom_reaper marks
> the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
> to 0. The race is as follows
>
> oom_reap_task do_exit
> exit_mm
> __oom_reap_task_mm
> mmput
> __mmput
> mmget_not_zero # fails
> exit_mmap # frees memory
> set_bit(MMF_OOM_SKIP)
>
> The victim is still visible to the OOM killer until it is unhashed.
>
> Currently we try to reduce a risk of this race by taking oom_lock
> and wait for out_of_memory sleep while holding the lock to give the
> victim some time to exit. This is quite suboptimal approach because
> there is no guarantee the victim (especially a large one) will manage
> to unmap its address space and free enough memory to the particular oom
> domain which needs a memory (e.g. a specific NUMA node).
>
> Fix this problem by allowing __oom_reap_task_mm and __mmput path to
> race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
> to run in parallel with other unmappers (hence the mmap_sem for read).
>
> The only tricky part is to exclude page tables tear down and all
> operations which modify the address space in the __mmput path. exit_mmap
> doesn't expect any other users so it doesn't use any locking. Nothing
> really forbids us to use mmap_sem for write, though. In fact we are
> already relying on this lock earlier in the __mmput path to synchronize
> with ksm and khugepaged.
That's true, but we take mmap_sem there for small portion of cases.
It's quite different from taking the lock unconditionally. I'm worry about
scalability implication of such move. On bigger machines it can be big
hit.
Should we do performance/scalability evaluation of the patch before
getting it applied?
--
Kirill A. Shutemov
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH] mm, oom: allow oom reaper to race with exit_mmap
@ 2017-07-24 7:23 Michal Hocko
2017-07-24 14:00 ` Kirill A. Shutemov
` (3 more replies)
0 siblings, 4 replies; 54+ messages in thread
From: Michal Hocko @ 2017-07-24 7:23 UTC (permalink / raw)
To: Andrew Morton
Cc: David Rientjes, Tetsuo Handa, Oleg Nesterov, Hugh Dickins,
Andrea Arcangeli, linux-mm, LKML, Michal Hocko
From: Michal Hocko <mhocko@suse.com>
David has noticed that the oom killer might kill additional tasks while
the exiting oom victim hasn't terminated yet because the oom_reaper marks
the curent victim MMF_OOM_SKIP too early when mm->mm_users dropped down
to 0. The race is as follows
oom_reap_task do_exit
exit_mm
__oom_reap_task_mm
mmput
__mmput
mmget_not_zero # fails
exit_mmap # frees memory
set_bit(MMF_OOM_SKIP)
The victim is still visible to the OOM killer until it is unhashed.
Currently we try to reduce a risk of this race by taking oom_lock
and wait for out_of_memory sleep while holding the lock to give the
victim some time to exit. This is quite suboptimal approach because
there is no guarantee the victim (especially a large one) will manage
to unmap its address space and free enough memory to the particular oom
domain which needs a memory (e.g. a specific NUMA node).
Fix this problem by allowing __oom_reap_task_mm and __mmput path to
race. __oom_reap_task_mm is basically MADV_DONTNEED and that is allowed
to run in parallel with other unmappers (hence the mmap_sem for read).
The only tricky part is to exclude page tables tear down and all
operations which modify the address space in the __mmput path. exit_mmap
doesn't expect any other users so it doesn't use any locking. Nothing
really forbids us to use mmap_sem for write, though. In fact we are
already relying on this lock earlier in the __mmput path to synchronize
with ksm and khugepaged.
Take the exclusive mmap_sem when calling free_pgtables and destroying
vmas to sync with __oom_reap_task_mm which take the lock for read. All
other operations can safely race with the parallel unmap.
Changes
- bail on null mm->mmap early as per David Rientjes
Reported-by: David Rientjes <rientjes@google.com>
Fixes: 26db62f179d1 ("oom: keep mm of the killed task available")
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
Hi,
I've sent this as an RFC [1] previously and it seems that the original
issue has been resolved [2], although an explicit tested-by would be
appreciated of course. Hugh has pointed out [3] that using mmap_sem
in exit_mmap will allow to drop the tricky
down_write(mmap_sem);
up_write(mmap_sem);
in both paths. I hope I will get to that in a forseeable future.
I am not yet sure this is important enough to merge to stable trees,
I would rather wait for a report to show up.
[1] http://lkml.kernel.org/r/20170626130346.26314-1-mhocko@kernel.org
[2] http://lkml.kernel.org/r/alpine.DEB.2.10.1707111336250.60183@chino.kir.corp.google.com
[3] http://lkml.kernel.org/r/alpine.LSU.2.11.1707191716030.2055@eggly.anvils
mm/mmap.c | 7 +++++++
mm/oom_kill.c | 45 +++++++--------------------------------------
2 files changed, 14 insertions(+), 38 deletions(-)
diff --git a/mm/mmap.c b/mm/mmap.c
index 24e9261bdcc0..0eeb658caa30 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2993,6 +2993,11 @@ void exit_mmap(struct mm_struct *mm)
/* Use -1 here to ensure all VMAs in the mm are unmapped */
unmap_vmas(&tlb, vma, 0, -1);
+ /*
+ * oom reaper might race with exit_mmap so make sure we won't free
+ * page tables or unmap VMAs under its feet
+ */
+ down_write(&mm->mmap_sem);
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING);
tlb_finish_mmu(&tlb, 0, -1);
@@ -3005,7 +3010,9 @@ void exit_mmap(struct mm_struct *mm)
nr_accounted += vma_pages(vma);
vma = remove_vma(vma);
}
+ mm->mmap = NULL;
vm_unacct_memory(nr_accounted);
+ up_write(&mm->mmap_sem);
}
/* Insert vm structure into process list sorted by address
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9e8b4f030c1c..a6dabe3691c1 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -470,40 +470,15 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
{
struct mmu_gather tlb;
struct vm_area_struct *vma;
- bool ret = true;
-
- /*
- * We have to make sure to not race with the victim exit path
- * and cause premature new oom victim selection:
- * __oom_reap_task_mm exit_mm
- * mmget_not_zero
- * mmput
- * atomic_dec_and_test
- * exit_oom_victim
- * [...]
- * out_of_memory
- * select_bad_process
- * # no TIF_MEMDIE task selects new victim
- * unmap_page_range # frees some memory
- */
- mutex_lock(&oom_lock);
if (!down_read_trylock(&mm->mmap_sem)) {
- ret = false;
trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
+ return false;
}
- /*
- * increase mm_users only after we know we will reap something so
- * that the mmput_async is called only when we have reaped something
- * and delayed __mmput doesn't matter that much
- */
- if (!mmget_not_zero(mm)) {
- up_read(&mm->mmap_sem);
- trace_skip_task_reaping(tsk->pid);
- goto unlock_oom;
- }
+ /* There is nothing to reap so bail out without signs in the log */
+ if (!mm->mmap)
+ goto unlock;
trace_start_task_reaping(tsk->pid);
@@ -540,17 +515,11 @@ static bool __oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)
K(get_mm_counter(mm, MM_ANONPAGES)),
K(get_mm_counter(mm, MM_FILEPAGES)),
K(get_mm_counter(mm, MM_SHMEMPAGES)));
- up_read(&mm->mmap_sem);
- /*
- * Drop our reference but make sure the mmput slow path is called from a
- * different context because we shouldn't risk we get stuck there and
- * put the oom_reaper out of the way.
- */
- mmput_async(mm);
trace_finish_task_reaping(tsk->pid);
-unlock_oom:
- mutex_unlock(&oom_lock);
+unlock:
+ up_read(&mm->mmap_sem);
+
return ret;
}
--
2.13.2
^ permalink raw reply [flat|nested] 54+ messages in thread
end of thread, other threads:[~2017-08-15 0:21 UTC | newest]
Thread overview: 54+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-27 8:29 [PATCH] mm, oom: allow oom reaper to race with exit_mmap Manish Jaggi
2017-07-27 9:24 ` Michal Hocko
2017-07-28 11:52 ` Possible race condition in oom-killer Manish Jaggi
2017-07-28 12:32 ` Michal Hocko
2017-07-28 12:59 ` Tetsuo Handa
2017-07-28 13:07 ` Michal Hocko
2017-07-28 13:15 ` Tetsuo Handa
2017-07-28 13:29 ` Michal Hocko
2017-07-28 13:55 ` Tetsuo Handa
2017-07-28 14:07 ` Michal Hocko
2017-07-29 4:31 ` Tetsuo Handa
2017-08-01 12:14 ` Michal Hocko
2017-08-01 14:16 ` Tetsuo Handa
2017-08-01 14:47 ` Michal Hocko
2017-08-01 10:46 ` Tetsuo Handa
2017-08-01 11:30 ` Michal Hocko
2017-07-28 13:15 ` Manish Jaggi
2017-07-28 13:50 ` Manish Jaggi
2017-07-28 14:12 ` Michal Hocko
-- strict thread matches above, loose matches on Subject: below --
2017-08-10 8:16 [PATCH] mm, oom: allow oom reaper to race with exit_mmap Michal Hocko
2017-08-10 18:05 ` Andrea Arcangeli
2017-08-10 18:51 ` Michal Hocko
2017-08-10 20:36 ` Michal Hocko
2017-07-24 7:23 Michal Hocko
2017-07-24 14:00 ` Kirill A. Shutemov
2017-07-24 14:15 ` Michal Hocko
2017-07-24 14:51 ` Kirill A. Shutemov
2017-07-24 16:11 ` Michal Hocko
2017-07-25 14:17 ` Kirill A. Shutemov
2017-07-25 14:26 ` Michal Hocko
2017-07-25 15:07 ` Kirill A. Shutemov
2017-07-25 15:15 ` Michal Hocko
2017-07-25 14:26 ` Michal Hocko
2017-07-25 15:17 ` Kirill A. Shutemov
2017-07-25 15:23 ` Michal Hocko
2017-07-25 15:31 ` Kirill A. Shutemov
2017-07-25 16:04 ` Michal Hocko
2017-07-25 19:19 ` Andrea Arcangeli
2017-07-26 5:45 ` Michal Hocko
2017-07-26 16:29 ` Andrea Arcangeli
2017-07-26 16:43 ` Andrea Arcangeli
2017-07-27 6:50 ` Michal Hocko
2017-07-27 14:55 ` Andrea Arcangeli
2017-07-28 6:23 ` Michal Hocko
2017-08-15 0:20 ` David Rientjes
2017-07-24 15:27 ` Michal Hocko
2017-07-24 16:42 ` kbuild test robot
2017-07-24 18:12 ` Michal Hocko
2017-07-25 15:26 ` Andrea Arcangeli
2017-07-25 15:45 ` Michal Hocko
2017-07-25 18:26 ` Andrea Arcangeli
2017-07-26 5:45 ` Michal Hocko
2017-07-26 16:39 ` Andrea Arcangeli
2017-07-27 6:32 ` Michal Hocko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®