How to immunize a process from the OOM Killer

How to immunize a process from the OOM Killer

[Juan Miscaro]

Hi, I'm running 2.6.24 on Ubuntu.

I've got a OOM Killer gone wild.  I have plenty of free memory (over
40 GB) and lots of processes are being murdered.  Anyway, I would like
to first prevent a few processes from being killed in the hopes of
buying me time (users can work) to discover why the killer is being
invoked in the first place.

I've read in a few places [1] that in order to immunize a process you
need to modify the value of /proc/PID/oom_adj.  Thing is, such files
cannot be edited directly.  How do I achieve my goal?

Thank you,

--
jm

[1]
http://linux-mm.org/OOM_Killer
http://lwn.net/Articles/317814/

Re: How to immunize a process from the OOM Killer

[David Rientjes]

On Thu, 29 Oct 2009, Juan Miscaro wrote:

> Hi, I'm running 2.6.24 on Ubuntu.
> 
> I've got a OOM Killer gone wild.  I have plenty of free memory (over
> 40 GB) and lots of processes are being murdered.  Anyway, I would like
> to first prevent a few processes from being killed in the hopes of
> buying me time (users can work) to discover why the killer is being
> invoked in the first place.
> 
> I've read in a few places [1] that in order to immunize a process you
> need to modify the value of /proc/PID/oom_adj.  Thing is, such files
> cannot be edited directly.  How do I achieve my goal?
> 

You need to do it as the superuser:

	$ sudo su
	# echo -17 > /proc/pid/oom_adj

or

	sudo sh -c "echo -17 > /proc/pid/oom_adj"

Re: How to immunize a process from the OOM Killer

[KAMEZAWA Hiroyuki]

On Thu, 29 Oct 2009 15:01:22 -0400
Juan Miscaro <jmiscaro@gmail.com> wrote:

> Hi, I'm running 2.6.24 on Ubuntu.
> 
> I've got a OOM Killer gone wild.  I have plenty of free memory (over
> 40 GB) and lots of processes are being murdered.  Anyway, I would like
> to first prevent a few processes from being killed in the hopes of
> buying me time (users can work) to discover why the killer is being
> invoked in the first place.

Could you show us your message log of OOM-Killer ?

Thanks,
-Kame

Re: How to immunize a process from the OOM Killer

[Juan Miscaro]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

2009/10/29 KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>:
> On Thu, 29 Oct 2009 15:01:22 -0400
> Juan Miscaro <jmiscaro@gmail.com> wrote:
>
>> Hi, I'm running 2.6.24 on Ubuntu.
>>
>> I've got a OOM Killer gone wild.  I have plenty of free memory (over
>> 40 GB) and lots of processes are being murdered.  Anyway, I would like
>> to first prevent a few processes from being killed in the hopes of
>> buying me time (users can work) to discover why the killer is being
>> invoked in the first place.
>
> Could you show us your message log of OOM-Killer ?

Hi, by message log I guess you mean parts of the kern.log?  I've
attached a snippet of that log file that covers one process being
killed.  I have many others like this.

-- 
/jm

[-- Attachment #2: oom-killer_stanza-host.txt --]
[-- Type: text/plain, Size: 10732 bytes --]

Oct 15 12:36:05 host kernel: [2695651.038485] nxssh invoked oom-killer: gfp_mask=0x44d0, order=2, oomkilladj=0
Oct 15 12:36:05 host kernel: [2695651.038496] Pid: 15247, comm: nxssh Not tainted 2.6.24.6-030909 #1
Oct 15 12:36:05 host kernel: [2695651.038519]  [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120
Oct 15 12:36:05 host kernel: [2695651.038537]  [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0
Oct 15 12:36:05 host kernel: [2695651.038549]  [nfs:__alloc_pages+0x34c/0x380] __alloc_pages+0x34c/0x380
Oct 15 12:36:05 host kernel: [2695651.038564]  [nfs:__get_free_pages+0x39/0x7b0] __get_free_pages+0x39/0x50
Oct 15 12:36:05 host kernel: [2695651.038573]  [tun:__alloc_skb+0x55/0x1250] __alloc_skb+0x55/0x120
Oct 15 12:36:05 host kernel: [2695651.038585]  [sock_alloc_send_skb+0x170/0x1c0] sock_alloc_send_skb+0x170/0x1c0
Oct 15 12:36:05 host kernel: [2695651.038598]  [unix_stream_sendmsg+0x28f/0x390] unix_stream_sendmsg+0x28f/0x390
Oct 15 12:36:05 host kernel: [2695651.038609]  [unix_stream_sendmsg+0x189/0x390] unix_stream_sendmsg+0x189/0x390
Oct 15 12:36:05 host kernel: [2695651.038624]  [sock_aio_write+0x109/0x120] sock_aio_write+0x109/0x120
Oct 15 12:36:05 host kernel: [2695651.038635]  [ktime_get_ts+0x19/0x50] ktime_get_ts+0x19/0x50
Oct 15 12:36:05 host kernel: [2695651.038650]  [tun:do_sync_write+0xd5/0x120] do_sync_write+0xd5/0x120
Oct 15 12:36:05 host kernel: [2695651.038665]  [<c0142d70>] autoremove_wake_function+0x0/0x40
Oct 15 12:36:05 host kernel: [2695651.038679]  [aa_file_permission+0x3/0xd0] aa_file_permission+0x3/0xd0
Oct 15 12:36:05 host kernel: [2695651.038692]  [vfs_write+0x15a/0x170] vfs_write+0x15a/0x170
Oct 15 12:36:05 host kernel: [2695651.038701]  [sys_write+0x41/0x70] sys_write+0x41/0x70
Oct 15 12:36:05 host kernel: [2695651.038710]  [sysenter_past_esp+0x6b/0xa1] sysenter_past_esp+0x6b/0xa1
Oct 15 12:36:05 host kernel: [2695651.038725]  =======================
Oct 15 12:36:05 host kernel: [2695651.038729] Mem-info:
Oct 15 12:36:05 host kernel: [2695651.038733] DMA per-cpu:
Oct 15 12:36:05 host kernel: [2695651.038737] CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038742] CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038746] CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038752] CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038757] CPU    4: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038762] CPU    5: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038767] CPU    6: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038772] CPU    7: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038777] CPU    8: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038782] CPU    9: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038788] CPU   10: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038792] CPU   11: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038797] CPU   12: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038801] CPU   13: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038806] CPU   14: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038812] CPU   15: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038816] Normal per-cpu:
Oct 15 12:36:05 host kernel: [2695651.038820] CPU    0: Hot: hi:  186, btch:  31 usd:  29   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038825] CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038830] CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038836] CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038841] CPU    4: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038846] CPU    5: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038852] CPU    6: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038856] CPU    7: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038861] CPU    8: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038865] CPU    9: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038870] CPU   10: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038875] CPU   11: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038881] CPU   12: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038886] CPU   13: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038892] CPU   14: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038897] CPU   15: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038902] HighMem per-cpu:
Oct 15 12:36:05 host kernel: [2695651.038906] CPU    0: Hot: hi:  186, btch:  31 usd:  24   Cold: hi:   62, btch:  15 usd:  11
Oct 15 12:36:05 host kernel: [2695651.038911] CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038916] CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038922] CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038926] CPU    4: Hot: hi:  186, btch:  31 usd:  24   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038932] CPU    5: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038937] CPU    6: Hot: hi:  186, btch:  31 usd:  57   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038941] CPU    7: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038946] CPU    8: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038950] CPU    9: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038955] CPU   10: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038960] CPU   11: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038965] CPU   12: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038970] CPU   13: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038976] CPU   14: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038981] CPU   15: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Oct 15 12:36:05 host kernel: [2695651.038987] Active:4175196 inactive:867151 dirty:15 writeback:10 unstable:0
Oct 15 12:36:05 host kernel: [2695651.038988]  free:9400621 slab:69457 mapped:182830 pagetables:21155 bounce:0
Oct 15 12:36:05 host kernel: [2695651.038994] DMA free:940kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? no
Oct 15 12:36:05 host kernel: [2695651.038998] lowmem_reserve[]: 0 873 57642 57642
Oct 15 12:36:05 host kernel: [2695651.039010] Normal free:85688kB min:3744kB low:4680kB high:5616kB active:0kB inactive:280kB present:894080kB pages_scanned:0 all_unreclaimable? no
Oct 15 12:36:05 host kernel: [2695651.039014] lowmem_reserve[]: 0 0 454152 454152
Oct 15 12:36:05 host kernel: [2695651.039028] HighMem free:37515856kB min:512kB low:61428kB high:122348kB active:16701060kB inactive:3468324kB present:58131456kB pages_scanned:0 all_unreclaimable? no
Oct 15 12:36:05 host kernel: [2695651.039033] lowmem_reserve[]: 0 0 0 0
Oct 15 12:36:05 host kernel: [2695651.039043] DMA: 1*4kB 1*8kB 2*16kB 4*32kB 4*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 940kB
Oct 15 12:36:05 host kernel: [2695651.039071] Normal: 20427*4kB 409*8kB 4*16kB 21*32kB 7*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 86420kB
Oct 15 12:36:05 host kernel: [2695651.039099] HighMem: 31689*4kB 507549*8kB 176346*16kB 57445*32kB 16385*64kB 4522*128kB 1214*256kB 428*512kB 305*1024kB 185*2048kB 6304*4096kB = 37516684kB
Oct 15 12:36:05 host kernel: [2695651.039128] Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct 15 12:36:05 host kernel: [2695651.039132] Free swap  = 29294516kB
Oct 15 12:36:05 host kernel: [2695651.039136] Total swap = 29294516kB
Oct 15 12:36:05 host kernel: [2695651.039140] Free swap:       29294516kB
Oct 15 12:36:05 host kernel: [2695651.249865] 14876671 pages of RAM
Oct 15 12:36:05 host kernel: [2695651.249875] 14647295 pages of HIGHMEM
Oct 15 12:36:05 host kernel: [2695651.249876] 314941 reserved pages
Oct 15 12:36:05 host kernel: [2695651.249879] 2921946 pages shared
Oct 15 12:36:05 host kernel: [2695651.249883] 0 pages swap cached
Oct 15 12:36:05 host kernel: [2695651.249885] 20 pages dirty
Oct 15 12:36:05 host kernel: [2695651.249886] 10 pages writeback
Oct 15 12:36:05 host kernel: [2695651.249888] 183270 pages mapped
Oct 15 12:36:05 host kernel: [2695651.249889] 69454 pages slab
Oct 15 12:36:05 host kernel: [2695651.249891] 21174 pages pagetables

Re: How to immunize a process from the OOM Killer

[KAMEZAWA Hiroyuki]

On Fri, 30 Oct 2009 13:26:56 -0400
Juan Miscaro <jmiscaro@gmail.com> wrote:

> 2009/10/29 KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>:
> > On Thu, 29 Oct 2009 15:01:22 -0400
> > Juan Miscaro <jmiscaro@gmail.com> wrote:
> >
> >> Hi, I'm running 2.6.24 on Ubuntu.
> >>
> >> I've got a OOM Killer gone wild.  I have plenty of free memory (over
> >> 40 GB) and lots of processes are being murdered.  Anyway, I would like
> >> to first prevent a few processes from being killed in the hopes of
> >> buying me time (users can work) to discover why the killer is being
> >> invoked in the first place.
> >
> > Could you show us your message log of OOM-Killer ?
> 
> Hi, by message log I guess you mean parts of the kern.log?  I've
> attached a snippet of that log file that covers one process being
> killed.  I have many others like this.
> 
Thank for log.
Hmm, from this log. x86-32 ? Then, normal (memory for kernel) area is too small.
Highmem area (39GB+) cannot be used for this socket's memory allocation.

It seems this is caused by 2 reasons.
  - normal area is small and fragmented.
  - unix domain socket allocates order=2(16KB) continuous area.

For avoid killing, using oom_adj is an idea (as David pointed out)

But considering the situation, if you want to avoid this temporaly,
reducing size of socket buffer is another idea.(for avoiding 16KB allocation)
(But yes, that will make throuput worse and another program may hit this
 fragmentation problem in other reason.)
It seems your system is not robust against fragmentation problem, now.

40GB is not good for x86-32 system, in general.

Thanks,
-Kame
==

Oct 15 12:36:05 host kernel: [2695651.038987] Active:4175196 inactive:867151 dirty:15 writeback:10 unstable:0
Oct 15 12:36:05 host kernel: [2695651.038988]  free:9400621 slab:69457 mapped:182830 pagetables:21155 bounce:0
Oct 15 12:36:05 host kernel: [2695651.038994] DMA free:940kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16256kB pages_scanned:0 all_unreclaimable? no
Oct 15 12:36:05 host kernel: [2695651.038998] lowmem_reserve[]: 0 873 57642 57642
Oct 15 12:36:05 host kernel: [2695651.039010] Normal free:85688kB min:3744kB low:4680kB high:5616kB active:0kB inactive:280kB present:894080kB pages_scanned:0 all_unreclaimable? no
Oct 15 12:36:05 host kernel: [2695651.039014] lowmem_reserve[]: 0 0 454152 454152
Oct 15 12:36:05 host kernel: [2695651.039028] HighMem free:37515856kB min:512kB low:61428kB high:122348kB active:16701060kB inactive:3468324kB present:58131456kB pages_scanned:0 all_unreclaimable? no
Oct 15 12:36:05 host kernel: [2695651.039033] lowmem_reserve[]: 0 0 0 0
Oct 15 12:36:05 host kernel: [2695651.039043] DMA: 1*4kB 1*8kB 2*16kB 4*32kB 4*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 940kB
Oct 15 12:36:05 host kernel: [2695651.039071] Normal: 20427*4kB 409*8kB 4*16kB 21*32kB 7*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 86420kB
Oct 15 12:36:05 host kernel: [2695651.039099] HighMem: 31689*4kB 507549*8kB 176346*16kB 57445*32kB 16385*64kB 4522*128kB 1214*256kB 428*512kB 305*1024kB 185*2048kB 6304*4096kB = 37516684kB
Oct 15 12:36:05 host kernel: [2695651.039128] Swap cache: add 0, delete 0, find 0/0, race 0+0
Oct 15 12:36:05 host kernel: [2695651.039132] Free swap  = 29294516kB
Oct 15 12:36:05 host kernel: [2695651.039136] Total swap = 29294516kB
Oct 15 12:36:05 host kernel: [2695651.039140] Free swap:       29294516kB
Oct 15 12:36:05 host kernel: [2695651.249865] 14876671 pages of RAM
Oct 15 12:36:05 host kernel: [2695651.249875] 14647295 pages of HIGHMEM
Oct 15 12:36:05 host kernel: [2695651.249876] 314941 reserved pages
Oct 15 12:36:05 host kernel: [2695651.249879] 2921946 pages shared
Oct 15 12:36:05 host kernel: [2695651.249883] 0 pages swap cached
Oct 15 12:36:05 host kernel: [2695651.249885] 20 pages dirty
Oct 15 12:36:05 host kernel: [2695651.249886] 10 pages writeback
Oct 15 12:36:05 host kernel: [2695651.249888] 183270 pages mapped
Oct 15 12:36:05 host kernel: [2695651.249889] 69454 pages slab
Oct 15 12:36:05 host kernel: [2695651.249891] 21174 pages pagetables