From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751986AbdBJBO4 (ORCPT ); Thu, 9 Feb 2017 20:14:56 -0500 Received: from szxga01-in.huawei.com ([58.251.152.64]:36171 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751656AbdBJBOy (ORCPT ); Thu, 9 Feb 2017 20:14:54 -0500 Subject: Re: [RFC] 3.10 kernel- oom with about 24G free memory To: Michal Hocko References: <9a22aefd-dfb8-2e4c-d280-fc172893bcb4@huawei.com> <20170209132628.GI10257@dhcp22.suse.cz> <20170209134131.GJ10257@dhcp22.suse.cz> CC: Vlastimil Babka , , , Tetsuo Handa , Hanjun Guo From: Yisheng Xie Message-ID: Date: Fri, 10 Feb 2017 09:13:58 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <20170209134131.GJ10257@dhcp22.suse.cz> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.177.29.40] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.589D13E2.0219,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: d94e270584a53f14a8a69b8fa4d5f8f8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org hi Michal, Thanks for your comment. On 2017/2/9 21:41, Michal Hocko wrote: > On Thu 09-02-17 14:26:28, Michal Hocko wrote: >> On Thu 09-02-17 20:54:49, Yisheng Xie wrote: >>> Hi all, >>> I get an oom on a linux 3.10 kvm guest OS. when it triggers the oom >>> it have about 24G free memory(and host OS have about 10G free memory) >>> and watermark is sure ok. >>> >>> I also check about about memcg limit value, also cannot find the >>> root cause. >>> >>> Is there anybody ever meet similar problem and have any idea about it? >>> >>> Any comment is more than welcome! >>> >>> Thanks >>> Yisheng Xie >>> >>> ------------- >>> [ 81.234289] DefSch0200 invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0 >>> [ 81.234295] DefSch0200 cpuset=/ mems_allowed=0 >>> [ 81.234299] CPU: 3 PID: 8284 Comm: DefSch0200 Tainted: G O E ----V------- 3.10.0-229.42.1.105.x86_64 #1 >>> [ 81.234301] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20161111_105425-HGH1000008200 04/01/2014 >>> [ 81.234303] ffff880ae2900000 000000002b3489d7 ffff880b6cec7c58 ffffffff81608d3d >>> [ 81.234307] ffff880b6cec7ce8 ffffffff81603d1c 0000000000000000 ffff880b6cd09000 >>> [ 81.234311] ffff880b6cec7cd8 000000002b3489d7 ffff880b6cec7ce0 ffffffff811bdd77 >>> [ 81.234314] Call Trace: >>> [ 81.234323] [] dump_stack+0x19/0x1b >>> [ 81.234327] [] dump_header+0x8e/0x214 >>> [ 81.234333] [] ? mem_cgroup_iter+0x177/0x2b0 >>> [ 81.234339] [] check_panic_on_oom+0x2e/0x60 >>> [ 81.234342] [] mem_cgroup_oom_synchronize+0x34f/0x580 >> >> OK, so this is a memcg OOM killer which panics because the configuration >> says so. The OOM report doesn't say so and that is the bug. dump_header >> is memcg aware and mem_cgroup_out_of_memory initializes oom_control >> properly. Is this Vanilla kernel? That means we should raise the limit of that memcg to avoid memcg OOM killer, right? > > I have only now noticed this is 3.10 rather than 4.10 kernel. > > There we simply do > dump_header(NULL, gfp_mask, order, NULL, nodemask); > > so memcg is NULL and that's why we report global counters. You need > 2415b9f5cb04 ("memcg: print cgroup information when system panics due to > panic_on_oom") > I will add this patch to check the memory stat of that memcg. Thanks again for your help. Thanks. Yisheng Xie