From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753243AbdHNM3b (ORCPT ); Mon, 14 Aug 2017 08:29:31 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:51993 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753128AbdHNM31 (ORCPT ); Mon, 14 Aug 2017 08:29:27 -0400 Date: Mon, 14 Aug 2017 13:28:32 +0100 From: Roman Gushchin To: David Rientjes CC: , Michal Hocko , Vladimir Davydov , Johannes Weiner , Tetsuo Handa , Tejun Heo , , , , Subject: Re: [v4 4/4] mm, oom, docs: describe the cgroup-aware OOM killer Message-ID: <20170814122832.GB24393@castle.DHCP.thefacebook.com> References: <20170726132718.14806-1-guro@fb.com> <20170726132718.14806-5-guro@fb.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.8.3 (2017-05-23) X-Originating-IP: [2620:10d:c092:200::1:aa9f] X-ClientProxiedBy: HE1PR0502CA0012.eurprd05.prod.outlook.com (2603:10a6:3:e3::22) To CO1PR15MB1079.namprd15.prod.outlook.com (2a01:111:e400:7b66::9) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2a6d7479-b685-49be-d779-08d4e3100583 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(2017030254152)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:CO1PR15MB1079; X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1079;3:YWSlozvDBTNAEP+NZTUnKBTb0VBIZmDsqx66L50zThAECxscPyw8iC7aNz9K2TIkQhad0zl5rXpPhtzE/KMxYvW8fcRCCk1zzz46yDdZRUhDhBtKna09nltL+0ymNhQTlaWg7tN2z5PIVcY5U3Kpde/NUc8av728HGi6IWX5T/GpMKSYZ+vdcxTi7KXC6vijVtGYjTXQJBfmv2PGSjhR0ot0zRAcCljGxB+AAYJMMMOhypvlBuN2wiehJUpMIT1T;25:feP+K4brZ1xUEdtPaH7SbiTFRJ2U1brbv8VkeqNC1ZcacEui5mPTGH8b30ibtPMKqpjumxFOOY9JEOj5ZJuZND2wi3SxoEzRg1cwnN0TMufKSgi2Ku7B95PtKqOW1fV/dVrxlBvmFVbPj2jv4wVk4PHk/Gn/CZuHl0FmnkHk+Eb3fhiNeW8E1XGY9pXt3jk9zeUCa/YJwEMzamS/j25DWV1orpi4IyIP7v48hKeHDb9MPiIf/jYeQ2/rUzp77G1b1MXYeub3MnVL6LYXDwifC1DeV/3AwK5Gjv+wAq75kGwh8LLCvZAW07RJO//0EDGVbm9+XRL+SgTCB7Ql6pAcYw==;31:sXlwt3ufs7ya0AUkCkymRcVDDxIyZvWkqvE/0SR10vTKADJHC5lBqgaBQGw7ImncaPxtHxNsINH/NzKQ/xmvBgyC/ZaSASY9p08AApif0AuOypuBl41Q2ng5zYxBRzZy4ev1upZM30d3Q3om4zZp60jUzyHz/vsefHb7LcYIDgV2ueFy6l4JImY5sjgz2K7BwoOc48eXNk4hI6ZKlsjtUo1yvVKRz6VWxkcIacpBCcM= X-MS-TrafficTypeDiagnostic: CO1PR15MB1079: X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1079;20:bJ7GVjuug0FWC9np/gG/l7VfINpPippWbBwaqVQVz75hnFhl8ZUtzrM4HXDEnUt9NIqxJsqyhFiDdsAq5EtvR8avC7nwSRAufiauiyc/t9tdkmhYmfrSwauUVDROepsYxifMOZSaSgc/rCavoj3nD2mYeUTGoBiQioAL2G2PKpXSLFXFtLAUACIpQANkK2RLu+plLzhX14XsrowjnPOMd62YJlHc/o7n+joraVjKO6nh2hlY+HDPm0dIudko/aMzzsDY1D94yGtf1ZG+nVY/TFJo67v3myK80anom6gPZl1Vo13Tb5hPOo+KJH5/mrzeYOVaOyClV0vGnONjZR0/0WkvsDbhXOJZ1grI3A7q+s/oyG6UyBUQmM1sTo2N241hfXasFFLye+HPRizt5/gumNRB0zZtv/agwt8/bQ7B7ysnF5s91Ze/med3wFn9957QO9FXxY1/sp4NFiKSR9/AyVzsn4jhii4qIED32s9mX8wkSd8HTBjzo8XdUcAUVfBz;4:WxLPgcR0Xob/e0/YnYYh/G14vfAINVi0tYZT/mf2Ts/rpiWtJXpNq+J6UvlTGm4k5VSp/L03VJaTIU0RH7k+a+3IHCGbIK+iXcA4zewfVj0+fkbPgjLDtc2vLEGH72rzDHURlNKPhXHHu+8zrbC3dneivbd5/WSScUkfl44cc+B3fmUJAe+3RlyG91eD50tFDGsc3uYiGdHhCwx1M6npM515D9U/UK14YUTianVAg2LJBIW/fJv3dlUNF7FbMANQ8fP2UdYbow3mYfO3lY05wIjjor1/wgdonX5KeXusD4Y= X-Exchange-Antispam-Report-Test: UriScan:(788757137089); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(100000703101)(100105400095)(93006095)(93001095)(6041248)(20161123564025)(20161123560025)(20161123562025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:CO1PR15MB1079;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:CO1PR15MB1079; X-Forefront-PRVS: 039975700A X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(7370300001)(4630300001)(6009001)(209900001)(199003)(189002)(24454002)(42186005)(6246003)(110136004)(6506006)(9686003)(53376002)(33656002)(106356001)(189998001)(83506001)(105586002)(53936002)(5890100001)(6916009)(4001350100001)(55016002)(478600001)(6306002)(97736004)(54906002)(966005)(6666003)(229853002)(2950100002)(50466002)(68736007)(7350300001)(5660300001)(305945005)(7416002)(54356999)(1076002)(8676002)(23726003)(47776003)(101416001)(6116002)(25786009)(2906002)(7736002)(4326008)(81156014)(50986999)(86362001)(81166006)(76176999)(18370500001)(42262002)(6606295002);DIR:OUT;SFP:1102;SCL:1;SRVR:CO1PR15MB1079;H:castle.DHCP.thefacebook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;CO1PR15MB1079;23:gc5d1VvjzMpASdxZYmi8w7h54zm9HsPj1w31yXg7A?= =?us-ascii?Q?Qn/q4AiuK7ivdq6/QzK2Krs6OhpsJLT9vmamNBB9zUhfUhn6UCwpsLj/aqce?= =?us-ascii?Q?Mz6PgbBWfRWHIEJaFqaarjRp85LDovRkGwSmV+E80Ddw7dOqdQ8zqITYpau9?= =?us-ascii?Q?NvzgpCXGN1FrOME3tatXaDMOlnwHjgDO83ZnEc3dkfQzQAivVLVBm75Wp4IY?= =?us-ascii?Q?jDc9IAxOac+OJDtfZ4AFCT11EKfCJWBH/qmng4oIFBM8SrB0RvjSJAnShP/j?= =?us-ascii?Q?dfGr9DzM6P6MKq3TBEuxw4Ox/ZStU9cgkbGnVVvLgG7v+SlNmaEf5THIDexW?= =?us-ascii?Q?4GVg1mF7wneOgjC6KlHPrtXi75GBWhgDpC7l9P5yUrYYkO9Rzd0P5S+dydi3?= =?us-ascii?Q?Z1sA4XuRwzp/lTXxY7Vz4CuPit4N1Xn3uFPpeY4+ddESymcZBBGvtV4AW+q8?= =?us-ascii?Q?tJh5lbpRflNtT9gDPBGCrkvoykawbWMmgdoobQT3ivZKV/im5bA3tSKt+1fS?= =?us-ascii?Q?6/KaBzHbrIkgxYcAjjUiZjyopxQzgVNjuZHMRCX+p4DolAxb7SNuaCCq3AVq?= =?us-ascii?Q?FZJd4PGSEwp25HrpOOpTwVDUn2zf2onvYmwqciGKSkRRosXhzSy/i6tLT6Mz?= =?us-ascii?Q?WoXD7tHu1gkvO+5r8nTkVECDpbB0LFBcKg38hwK0cpU2I5/MLXTfACfsJ3/q?= =?us-ascii?Q?seSMSR42bjGnkd0BeIEmZlzdPWZG5+wpAraY1iyY+wlfc/G49hZ1BA36bdwV?= =?us-ascii?Q?ZoW3eOgkccienGpBqe1bAeRVKCgzrf1GXiwHz1aYK5iwyKUBk3QsmP77t1Qc?= =?us-ascii?Q?zguRBhJzzcLk86+g+PFGN0xIWwatRwmfeNaEChkQc6OGZpUYkzUEp9Lohk0k?= =?us-ascii?Q?M8p+lJuWpbJWUFb3FvVnU6Pn2AFSJFYRdxYhq9JKTp6V3O+f1/y+dFDsOpLh?= =?us-ascii?Q?uZGyqThT75BR6LZJI+2wvr1u/iZ8E6n0jyNkUBuwPo3TfHuzYVnbZQNKf1N+?= =?us-ascii?Q?pwbF0M1HndnplcrJh8ljG3h1J7s6C9hdP/WJkeaFI+itsvogSMKjUe9QmlQM?= =?us-ascii?Q?0Ezve3ToEs8mWTQgeC9//3379iCBgwbJmFv8awwed3z0o7dsJXFIrSaGg6dw?= =?us-ascii?Q?FTqrCj9LGRNCKnoUg0Qdd1c+E2UvCNyAM91PWYN2EyDG+TrzhG5AoFzVLX5+?= =?us-ascii?Q?0XxKXA2WDSsqCYTkmIrqRhoc8Pf0QsDsqy6K03VgI1+22ag7BP/aPamnXih0?= =?us-ascii?Q?iWbz/dw/40gykuMSoOtcA82WmFKCtZ8HDp0Z0Maan5k7GHAY305/1N39y29n?= =?us-ascii?Q?xZw39vnJLwusF6SX/AgPOD71MwT/2jCPdpip8rC7Dm7?= X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1079;6:xrJ9eHfVn3wtqS7BkhytMwTwP2Ay/hcwboRY++9mxATXASbmPH1xget1qy2STHo6k3roue21OovT/uyhxJOYAdZkxPKzK3wAjwvjR+0p+5SK8YAleDAHL2Yo0+E8FkLmSwIiCGyLg1P1KBs7+zVUTEA26uymNauxz1JqWHRMrdJi0OwucFUrD31vyP2sZADIul7lSqu50AiTj5w7IMHyDjvmNLvM/M8L1QInFCFaaWg9V02uU/n/n/+zbrD9qek9+aPaOc+A69n+6uNhJ2IBRodb1Xv7jusACcqgWJkIlReFWv0lThe0JTSz+3K1BT850W7fNY7Iiu/+ZFeW4l21Yw==;5:q7hMyBXIGkYCJ4AQRDBwml0pFON+0r6z43mTYghY1xllR8pc9VpxO/omj4JvX/j9eX5Izh33oHzr41gAuhCY1Guc814k01Qgz3P3LVZMSZDdGHRaWVXpmyfFTV9lKSp39pg5oOLchTQihcEbrNOXDA==;24:FWhEENtiJRcWOgHsHyMNvH1eHrFSHLGvk25k2JmqXz1c1b0MY5T4/+7AL5Y0hKG1HJx8hkXHyOBGWonDWTfYdCSk8Zhf/glMy7CDdMA6XOM=;7:fSEETiltmLPqawj3bQExQMTuLI1s60jAhb93ANofe7W76SNDIIuxUx0/OjBLarhU0Kch13zgidt5CfP13Bp5c/wMX/Hfv1Y9AAARxegxgg7T+dsrUYXDNMqxd0iZK8AxNz0JdrG/NOHnyd2wZmO2wVp/nnbB2Q9nXCqnp/KFM+dTD4wcoHqLNlq8n9kt8KBulDzN13lrJzyqR2KqOCyHCMA30uZQxoiAv1Eub7j0RDw= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1079;20:X/H1WJCAVFnf6Zmml1xZmnuBH2xhjhRnB+85GD2RTOOTfd/MzXKOD+p3EQqdH9/wodSg3GhseMWMpk1i3Tqu4+2l5ygk/wv9ngZkHHmM8xHJVgaZYKKoY1h6ywgP+BNVg3Fg0K8YLiZxDxlHBX6koHtwWU5JKREAhfHkIQSuclo= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Aug 2017 12:28:48.7776 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR15MB1079 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-08-14_10:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 08, 2017 at 04:24:32PM -0700, David Rientjes wrote: > On Wed, 26 Jul 2017, Roman Gushchin wrote: > > > +Cgroup-aware OOM Killer > > +~~~~~~~~~~~~~~~~~~~~~~~ > > + > > +Cgroup v2 memory controller implements a cgroup-aware OOM killer. > > +It means that it treats memory cgroups as first class OOM entities. > > + > > +Under OOM conditions the memory controller tries to make the best > > +choise of a victim, hierarchically looking for the largest memory > > +consumer. By default, it will look for the biggest task in the > > +biggest leaf cgroup. > > + > > +Be default, all cgroups have oom_priority 0, and OOM killer will > > +chose the largest cgroup recursively on each level. For non-root > > +cgroups it's possible to change the oom_priority, and it will cause > > +the OOM killer to look athe the priority value first, and compare > > +sizes only of cgroups with equal priority. > > + > > +But a user can change this behavior by enabling the per-cgroup > > +oom_kill_all_tasks option. If set, it causes the OOM killer treat > > +the whole cgroup as an indivisible memory consumer. In case if it's > > +selected as on OOM victim, all belonging tasks will be killed. > > + > > +Tasks in the root cgroup are treated as independent memory consumers, > > +and are compared with other memory consumers (e.g. leaf cgroups). > > +The root cgroup doesn't support the oom_kill_all_tasks feature. > > + > > +This affects both system- and cgroup-wide OOMs. For a cgroup-wide OOM > > +the memory controller considers only cgroups belonging to the sub-tree > > +of the OOM'ing cgroup. > > + > > IO > > -- > > Thanks very much for following through with this. > > As described in http://marc.info/?l=linux-kernel&m=149980660611610 this is > very similar to what we do for priority based oom killing. > > I'm wondering your comments on extending it one step further, however: > include process priority as part of the selection rather than simply memcg > priority. > > memory.oom_priority will dictate which memcg the kill will originate from, > but processes have no ability to specify that they should actually be > killed as opposed to a leaf memcg. I'm not sure how important this is for > your usecase, but we have found it useful to be able to specify process > priority as part of the decisionmaking. > > At each level of consideration, we simply kill a process with lower > /proc/pid/oom_priority if there are no memcgs with a lower > memory.oom_priority. This allows us to define the exact process that will > be oom killed, absent oom_kill_all_tasks, and not require that the process > be attached to leaf memcg. Most notably these are processes that are best > effort: stats collection, logging, etc. I'm focused on cgroup v2 interface, that means, that there are no processes belonging to non-leaf cgroups. So, cgroups are compared only with root-cgroup processes, and I'm not sure we really need a way to prioritize them. > > Do you think it would be helpful to introduce per-process oom priority as > well? I'm not against per-process oom_priority, and it might be a good idea to replace the existing oom_score_adj with it at some point. I might be wrong, but I think users mostly using the extereme oom_score_adj values; no one really needs the tiebreaking based on some percentages of the total memory. And oom_priority will be just a simpler and more clear way to express the same intention. But it's not directly related to this patchset, and it's more arguable, so I think it can be done later.