From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753203AbdHXO6c (ORCPT ); Thu, 24 Aug 2017 10:58:32 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:49445 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752563AbdHXO6a (ORCPT ); Thu, 24 Aug 2017 10:58:30 -0400 Date: Thu, 24 Aug 2017 15:58:01 +0100 From: Roman Gushchin To: Michal Hocko CC: , Vladimir Davydov , Johannes Weiner , Tetsuo Handa , David Rientjes , Tejun Heo , , , , Subject: Re: [v6 2/4] mm, oom: cgroup-aware OOM killer Message-ID: <20170824145801.GA23457@castle.DHCP.thefacebook.com> References: <20170823165201.24086-1-guro@fb.com> <20170823165201.24086-3-guro@fb.com> <20170824114706.GG5943@dhcp22.suse.cz> <20170824122846.GA15916@castle.DHCP.thefacebook.com> <20170824125811.GK5943@dhcp22.suse.cz> <20170824135842.GA21167@castle.DHCP.thefacebook.com> <20170824141336.GP5943@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170824141336.GP5943@dhcp22.suse.cz> User-Agent: Mutt/1.8.3 (2017-05-23) X-Originating-IP: [2620:10d:c092:200::1:4649] X-ClientProxiedBy: AM5P194CA0010.EURP194.PROD.OUTLOOK.COM (2603:10a6:203:8f::20) To SN2PR15MB1088.namprd15.prod.outlook.com (2603:10b6:804:22::10) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: eac01c96-7beb-405f-2358-08d4eb008aff X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:SN2PR15MB1088; X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB1088;3:3Rc6RN8p2uhL7dbQFQi7o4P66pnDEqC8COWbGTcKQO/vj0YJk6U6ERC2e1yt6db7f3A+sIvNd7vvhk04YeMEUBog3u0Hjcl+nuycQ9FL87lXuWzjre2jOmtMj6zuawWU1aLrI40zTwFv00QyhLTXCjMBlEYbq46W3BhSdjG0PubMxtVWTRZ6+fwb+iYPhhIIrLnntSqm43q0aWwTEsp4yTDf8WwK+M3y7CC9upFJaSMiSLbQExOoB/RdWMwEeD+p;25:t34KreT2Lg038aypX4IXbka33lo1NIbC1eAsIFA7+/zeplDlsXHaroLdNbEvBTkSwAFQhvNGfFpDpA431K5E1ncpOEIKDnlEqqmB8mZO7koxizGD/0pnFoDKFqoUzQ6rEiMOhlP2XJp3lwZfg9RDjb+pM3K4TmnolWd7vQiW5uAYUcCZ2bmJGNQeA26UA2XbycA1QARRAaBzsM7KiBpZdbBeRtZGuNxIUlXvryGtPQP9j3TAQ9s/sy0lx2EGtHw4X/qjHFnqJUe9o6UEWSWlTKaAfo9ENRAcaYjrwtXMVb/UXK0UvB/ACiDvkA5J7GY+Dt+ilWEXkUnkNwjU/5cVsw==;31:9G6N/cpqiZ4sx1oHCPYA/cESDcj7PcpjYLBcAfsIVM+xbH08BzbIvIsNJhfPYAsgDREGooj2VWw3twv3IwqFXpuLaEswzXyRXHdO/ZgjCW+WYCxTcm3Y+AkP5gzvixnq1ie6FoF97USTq/hJ9moRMHoAIyydglwyaNj8NzQBSs9LJh9JLpTn7PhrpXkQZ4lCO/D/3tB9SPxlznI//1gKcXqABd3wxpN4DvRgUXE6ZOE= X-MS-TrafficTypeDiagnostic: SN2PR15MB1088: X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB1088;20:dAPl74C1qbRwWsINGh+IyEX85477CAeVJkFhGguZnDwGUUehaAq0mZZthpyIe5ukbuslnCPM+JeJd5ZX+nMPdf6j0YhVILo19YoSD4N/oTa5dJp1lhNBfPP/7+30MQAnPI1w3jvUs1fHnva/Trec8GEOsBCJ/dd8Lh9Pnjy9ZbWDRFzNRrRdtBAhaIyK+B8WnSjuoC4AGELtpuBpMjf89pvcBg/K1fZL0TN/9YLpdpLO15+wVYCTISGTf8D40KoSrJYK6uq0qqf5c3mq8QEobOqqK/JtyPNFTwQ1wo4KTnOeuL9CVH8uu+Lkb2+Ja2KXy2kklKBs+jVviZjdh3D/eSsNqLM5QNqLGLk67vrHhzYZ0Tuj8X+Cl/APJmRAvwvypesN6/Oac2CmWwSvs4PlAHnuLWTNGbpnJviDVZ8+irpFZQ6sjgN5Czygho0HfIHGmW1ImhqMrF1SDRQScXNXR3hoH1DxvWJxDruvlkK09TcJRpj9vbGEitYV7qxU+Hed;4:NGZ8lnR9a+0LemA0lub5Y0nHzxn1Z1YD7MmhRXZCfQZ/nxIYFKOTnvgzIDj9NkpoTXoV8tQDGATbi5hZ8y5JGiDvgMeQmyZ+oaDdqNMOD5/qg7yILdQNh+2O9OSAwQWCnBiUe+lTV2JsLqQGPrGTKJiIIJ2NJ9O5uo4aZYbWGHfBW5InZEVpPgb1pJE2n0xOJCZTh+TNvpYfwbB+pWMoSNHX+UJIbVRXCZrBZ3AcvFILVmfHh8ciND/dAI8vHR0N X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(3002001)(100000703101)(100105400095)(93006095)(93001095)(10201501046)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123555025)(20161123564025)(20161123560025)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:SN2PR15MB1088;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:SN2PR15MB1088; X-Forefront-PRVS: 04097B7F7F X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(7370300001)(979002)(6009001)(189002)(24454002)(377424004)(199003)(6116002)(7416002)(23726003)(86362001)(305945005)(9686003)(8676002)(6666003)(81166006)(106356001)(2906002)(54906002)(229853002)(6506006)(53936002)(6246003)(42186005)(105586002)(189998001)(55016002)(6916009)(2950100002)(101416001)(1076002)(7736002)(81156014)(50986999)(4001350100001)(25786009)(33656002)(97736004)(76176999)(5660300001)(7350300001)(50466002)(83506001)(93886005)(68736007)(110136004)(478600001)(54356999)(4326008)(47776003)(18370500001)(42262002)(969003)(989001)(999001)(1009001)(1019001);DIR:OUT;SFP:1102;SCL:1;SRVR:SN2PR15MB1088;H:castle.DHCP.thefacebook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;SN2PR15MB1088;23:W7kCbU2FtYLIBcWnlSmQpB/LEaAoomGW2BxwLnBfX?= =?us-ascii?Q?HmQsMzp6QW5JuwtlukJW4xUVUZHQvbORE2jtjMzOmbN/keBXEO9liozWVSh/?= =?us-ascii?Q?UHJXK3m3FZT7Z93A9ObfSiMsb63xNO9opiS2svjlrsVTNr4dPRQZfjegYlS4?= =?us-ascii?Q?oD/Qgc6zFgrmw+Mq0rSAyZ96qsGeHRZAr63W1na69cTko7rCVF/KTASKyslO?= =?us-ascii?Q?UWmE9rxnVPZ3I39CymsM7Fv0W9Xh+ciUNEQMmrSw/UTHstvrT9sdXTiFkkiB?= =?us-ascii?Q?pKqlXNPR0K7PZcYdUmDTRGPGKSW1cZVK6XYpIBl44UJWJRj5NhUmMdMDcENd?= =?us-ascii?Q?RqQMfa0zgVRQKTJrVmntua8Bq49+LBU23BROk60xbLqwYF8hA794SQg9mPf0?= =?us-ascii?Q?XGBQt4wVF4MbOouIA9MswT1bB2rKKqsuXgWWBP4ianDEUBYXz5BS+RMBohpX?= =?us-ascii?Q?BeiLIA5cV6/42szo3SjIp0oIxRw4d9SfnE12/YiW1U2bldhNjTujmIQLq5Ob?= =?us-ascii?Q?Tz4bSHh49fBF8PNDVaLpWxk9eHoHCLLq6ZWVFsUDJq2+dsS7iniI+zz48QDn?= =?us-ascii?Q?zu1aG0LUQSoZX8YPUMJlW/0WV07H08WgHt0b+rDclckc7X/40sQ9kOoGek2k?= =?us-ascii?Q?KkLY6A6TS4n0dncyO6bDH6zYQIF/Z3OpGtOceYdAgHxmn1f7se10WaUD6OCA?= =?us-ascii?Q?a2g20eX5GBkSWxjW9nonfMnYhIVdhFf+4D/TtM/vTVmyZOtofcVuFPp4sRAv?= =?us-ascii?Q?z+M8HAmMKZc4/yMQKLcrLn5jZgLHlbAPyAsDb4hNia70CtEyTVhFAVuKev/W?= =?us-ascii?Q?LJfYIDaKPeOeH4soTfOLIciXlo2uStSrJognLFmlvK9VtNP4VUt4ePmlD8Xq?= =?us-ascii?Q?jxMx2YaW+l3tGxuWdgCWU1NnXcfTCO4tsRHDoXmVqhnxpfl3BSsQfgyb84ei?= =?us-ascii?Q?T7dE7E+8L2DbCfHzzpkWlVhovYbMiSeqBINTf7anuJK4D7fpf/YgMyKmX9Ez?= =?us-ascii?Q?Yv6fW80lnoZm1Dhb23VWfYhpJShWbNxex07Z1p1L/5Cb9CIDukk6/kB6hfyc?= =?us-ascii?Q?lD6aOBImK+57aOzOivemGBBu4S9CGUa+vavvO1ZQqwtoGMgHBBh2wgzOMmIy?= =?us-ascii?Q?h4VRnzGJ9rJnCwSW+eFzMtmJnU6TKD5CA2xFAthoIsthQwd+suvaG80+Ezpb?= =?us-ascii?Q?U6sOvnVM80+TIrt/hjl2gb6sTrlKTVEiJiHp/oT9+SDf5Vtqx3J+4AK5EOk+?= =?us-ascii?Q?S0RBU3TPj5qmXY36FYyiBMBUSPx+uoLAqxq+rBjL1B2nZjxJDU2grFcA6t1Y?= =?us-ascii?Q?B4BBTd6HMDhIMilMrh9TKiRQ5HsCeH8JspPsj8WSnMC9HwKfEdWU5HAIG58L?= =?us-ascii?Q?rpleQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB1088;6:LCzfrzYeVKlZ+HWx38uRkM5hsNmt7S+9IUPsZDqNuru5s/LX6XNz6keJCkYMtgpNysG4FsFEnwBRzBDsaozvQ6Q0mXBsM06pK1Ks58iPyV1XK+vh9QqbVN6DhJUO/7z8p03L5u1li9CvH7cw6tXx4cBcTIndl/vHBQLAyNIty/jk7MU5V6haZ508VrTSOv01S2h928Bd0fg4lmD2LrLQx9d6aNkVV+GST2MKpkjPNq3+nQi1lKrTeFPtmtR02hvKV+gf3LsnSIB1pgNUihWeY1GMXVAi+IVBTEp0mANhL09sAJI94W0BGEEGznPeS7WyGWApVZFuOSTp1XFnrdZqHg==;5:mkL4lt7iKjekHmf1KXxkP35EBWLxOzBsKxbeWQ5LZ9w1/xqa/GlkYByUJhyvkNFzLmg+GrksoxulGKs7QKzCNnSyEaXqDchaK+ns03Qy0u1hXaxceIxVpmP7k94XzSo8xOsgRkqwJTc6YCVPgESPbA==;24:etOhoNUZSA7aOHxhdFwzQGDbV4eWF81/1krpqqON4cgkpdqJj4rQkebcoFEaluATtnhAJApTK0bSwCbo2uBnucZI10n8tIRcVpakKSuMnao=;7:uTZb7Q8e33LldaMgUfV8ypheIBVO9JfFB8g3Vidfv3Aj1039uUQLGOPeSYRv8HLySSCsH2+vewRJTniulKwmS/FfQhFW4EdxGLwheU7zWafVfJth09Ot1h7pKmti38+LplV0Vvzf9kvFTqQPnKxETqDCLcILMtpaa0lQ4a8z7d6956W+mbqyqSo/aNr7kOvLC0dfXumUeW+g0qLHYQRaUwtxKunyVBs4EwcIcbnpdLo= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;SN2PR15MB1088;20:rgal1DhfiaEj4Pt3QAYfTlKrXMyRc28Y7eUG8/8FRUbiWZ03eUwM0r6+5aNl6ru0qiw7hxJCF6NNgGLGL9edcgVtQE9A/fjcIdwsB6GpCXuIwUlRDb4tv9TnyNbxALwiqWARe7eqUnJGVpsiZrAmZm8KjLlkf4kHqicFT44v000= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Aug 2017 14:58:11.3554 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR15MB1088 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-08-24_06:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Aug 24, 2017 at 04:13:37PM +0200, Michal Hocko wrote: > On Thu 24-08-17 14:58:42, Roman Gushchin wrote: > > On Thu, Aug 24, 2017 at 02:58:11PM +0200, Michal Hocko wrote: > > > On Thu 24-08-17 13:28:46, Roman Gushchin wrote: > > > > Hi Michal! > > > > > > > There is nothing like a "better victim". We are pretty much in a > > > catastrophic situation when we try to survive by killing a userspace. > > > > Not necessary, it can be a cgroup OOM. > > memcg OOM is no different. The catastrophic is scoped to the specific > hierarchy but tasks in that hierarchy still fail to make a further > progress. > > > > We try to kill the largest because that assumes that we return the > > > most memory from it. Now I do understand that you want to treat the > > > memcg as a single killable entity but I find it really questionable > > > to do a per-memcg metric and then do not treat it like that and kill > > > only a single task. Just imagine a single memcg with zillions of taks > > > each very small and you select it as the largest while a small taks > > > itself doesn't help to help to get us out of the OOM. > > > > I don't think it's different from a non-containerized state: if you > > have a zillion of small tasks in the system, you'll meet the same issues. > > Yes this is possible but usually you are comparing apples to apples so > you will kill the largest offender and then go on. To be honest I really > do hate how we try to kill a children rather than the selected victim > for the same reason. I do hate it too. > > > > > > I guess I have asked already and we haven't reached any consensus. I do > > > > > not like how you treat memcgs and tasks differently. Why cannot we have > > > > > a memcg score a sum of all its tasks? > > > > > > > > It sounds like a more expensive way to get almost the same with less accuracy. > > > > Why it's better? > > > > > > because then you are comparing apples to apples? > > > > Well, I can say that I compare some number of pages against some other number > > of pages. And the relation between a page and memcg is more obvious, than a > > relation between a page and a process. > > But you are comparing different accounting systems. > > > Both ways are not ideal, and sum of the processes is not ideal too. > > Especially, if you take oom_score_adj into account. Will you respect it? > > Yes, and I do not see any reason why we shouldn't. It makes things even more complicated. Right now task's oom_score can be in (~ -total_memory, ~ +2*total_memory) range, and it you're starting summing it, it can be multiplied by number of tasks... Weird. It also will be different in case of system and memcg-wide OOM. > > > I've started actually with such approach, but then found it weird. > > > > > Besides that you have > > > to check each task for over-killing anyway. So I do not see any > > > performance merits here. > > > > It's an implementation detail, and we can hopefully get rid of it at some point. > > Well, we might do some estimations and ignore oom scopes but I that > sounds really complicated and error prone. Unless we have anything like > that then I would start from tasks and build up the necessary to make a > decision at the higher level. Seriously speaking, do you have an example, when summing per-process oom_score will work better? Especially, if we're talking about customizing oom_score calculation, it makes no sence to me. How you will sum process timestamps? > > > > > > How do you want to compare memcg score with tasks score? > > > > > > > > I have to do it for tasks in root cgroups, but it shouldn't be a common case. > > > > > > How come? I can easily imagine a setup where only some memcgs which > > > really do need a kill-all semantic while all others can live with single > > > task killed perfectly fine. > > > > I mean taking a unified cgroup hierarchy into an account, there should not > > be lot of tasks in the root cgroup, if any. > > Is that really the case? I would assume that memory controller would be > enabled only in those subtrees which really use the functionality and > the rest will be sitting in the root memcg. It might be the case if you > are running only containers but I am not really sure this is true in > general. Agree.