From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752994AbdHXOL5 (ORCPT ); Thu, 24 Aug 2017 10:11:57 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:43534 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751186AbdHXOLz (ORCPT ); Thu, 24 Aug 2017 10:11:55 -0400 Date: Thu, 24 Aug 2017 15:11:08 +0100 From: Roman Gushchin To: Michal Hocko CC: , Vladimir Davydov , Johannes Weiner , Tetsuo Handa , David Rientjes , Tejun Heo , , , , Subject: Re: [v6 3/4] mm, oom: introduce oom_priority for memory cgroups Message-ID: <20170824141108.GB21167@castle.DHCP.thefacebook.com> References: <20170823165201.24086-1-guro@fb.com> <20170823165201.24086-4-guro@fb.com> <20170824121054.GI5943@dhcp22.suse.cz> <20170824125113.GB15916@castle.DHCP.thefacebook.com> <20170824134859.GO5943@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20170824134859.GO5943@dhcp22.suse.cz> User-Agent: Mutt/1.8.3 (2017-05-23) X-Originating-IP: [2620:10d:c092:200::1:ec5a] X-ClientProxiedBy: VI1P194CA0023.EURP194.PROD.OUTLOOK.COM (2603:10a6:800:be::33) To CO1PR15MB1080.namprd15.prod.outlook.com (2a01:111:e400:7b66::10) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: befefd76-814b-471d-31fc-08d4eafa014f X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(300000502095)(300135100095)(22001)(2017030254152)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:CO1PR15MB1080; X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1080;3:l8UMaZMpxcx2sg8/C9W1JI1PwDVW1B3uiT2AaRxFdIWTugZTx2McR7TNS1aIRe+EgTkK1kce4f6LGaclOt6zvBDskhrdt3LMLpWvuNz1HV9oD+XXGiowsd5z8RsgjW4dskpuWhGrBADbbF/wGMedqdUEefI8MVcG5+OIYdmE3obLR6Kme70Pa5WA6I6VFuE5HBuQabOkkJQGzMttgtgLTDnJnNeEt1ATr3iutyFZofholDK+s+mg6ls/Dtrszjku;25:oZGsKNFb6qjg6CadZlaGuzfPRhLYABzBq35cNQF0DvA2qz5W1Fdj3w+wFDT3B/qzdL7r2ggE1lChrWtWyhwGfUwNwEHt+vSYxBciHU8Z4H8aM8Wy8G9ziTj3UCqR8Kgm3RmA8446bet9Cpux3yyCaYYwgbZfOPvtyBQxfRZrD+yeFVo8G+HuKylD7trT/OEgsmSq37Zy2gMsrutxUMUOjoaBxLA14e2A5zWuWP21dC9EcZLx3/t+3RThR1xW7XhKQ3rgenefK71hjRMmIuJ72LBy2txt8gGJKajMTnaIayi3jj3G7gglHEwqEmPmWlO8j8i/FLqErF279oQDq8luPA==;31:yhlpMnBicV3Nx4yzwJcLQQJjLmrIPGulZO5V2K9MxhNi9guFGpTSeAdjPYyDxir9rpp3KzC3qjV6GVZzBoB747A94ZrzrgWiOaQ7+d7VBXBEI0JMD2Du17Cbsgh04ooyVrprts+ACdRBV5zw0csIS6qhhHD2A8nEiI6t2JiAX10MF2Dj4Brdp2RFbAFA9vki7m5/c9eE/AGYUFjNLR2ujeUDlTtKQF1ukzTrLxujz14= X-MS-TrafficTypeDiagnostic: CO1PR15MB1080: X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1080;20:A8rmSWU+4EinG0U9qeFcckbLWgEofmhtRfS/GKP/X1br4v546GgPeWU04kfn8yP28rD3N6MpYxKejXkTumB4NtJbLaiAhduMcX8eoA9Dbw436rO/bFmS915A5ggXP8LuIcOPg1FAgeWu/mvSThrYF6DeBV0NvWvWytxPAVcd7dVpHpbxWG4rthGC8yW3eAdewCAoyjnk6EFG7HboayGsYwOuQ5w3nVc0s4oCoSW2IEKUDKuS0gslaiKE5ssc+z5WPDnRmEUjlbubAx7/SdUWU4vMImatyj7FcYfJkFsSbPMIPe9Qk16pqU+a0Dlzal1pe4/3z8507HuizzYTDNh9MhBJZX/PRluDrB+dL9XRuzMV8STlwp8UKUso1otvGFbrB7BEQ0m3sNP4FUQagqRhuum6U+TlB6U9vt029Zx/UVhIxXM1m0+nkMzhykhmc9lVA6h9oGFleYiFA8epZTwVxLmW7XRX7SaJTn6yyhcQfoeHRq6ZxZ/xUtQAJGLcp4Et;4:OyTAtDIEJVQfRZaG/HJFpDWyh2oEKII0YatNLBZC/e3tU2aSKud8vYxDAvM7ve5yB9dE3DeXF6l/LgxQ9HZ0a+SVHj7rhynDqhhGJakUQB5Eh9IcIZIoDo+v0mAmwftCP4TcM2spexfj/+B7JbRwzaxaGZUzojx+BkmTnVmC0Jx4JqLNlnFIax8F/v6wRpvhp4/MUBxdheMaQhnuX8OOJKWdoE56ahkRuGYS3mIbqm0HkZxxyjxg9xrNt04Q84ta8pIo4v9YDkPJktl3VctbvDzHicp1SotRDrAzA27HoZM= X-Exchange-Antispam-Report-Test: UriScan:(788757137089); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(10201501046)(100000703101)(100105400095)(6041248)(20161123558100)(20161123560025)(20161123555025)(20161123564025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:CO1PR15MB1080;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:CO1PR15MB1080; X-Forefront-PRVS: 04097B7F7F X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(7370300001)(6009001)(377424004)(189002)(57704003)(199003)(24454002)(54534003)(2906002)(68736007)(5660300001)(55016002)(1076002)(47776003)(33656002)(81166006)(42186005)(81156014)(83506001)(7350300001)(54906002)(23726003)(2950100002)(8676002)(6666003)(76176999)(6916009)(54356999)(9686003)(50986999)(6116002)(7416002)(86362001)(101416001)(6506006)(7736002)(50466002)(25786009)(93886005)(97736004)(478600001)(4326008)(4001350100001)(189998001)(105586002)(110136004)(53936002)(229853002)(106356001)(305945005)(6246003)(18370500001)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:CO1PR15MB1080;H:castle.DHCP.thefacebook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;CO1PR15MB1080;23:LTyy/7GjCzjdt40857aFfkB3FyyxSrHT5gqgfl0J6?= =?us-ascii?Q?x9K6ak+HFuJVsfg1tsEe5H2dwc+DARTh15pekoRS48k93JB30Z7ti7SQAGK9?= =?us-ascii?Q?/TnhSYGUpPj+ZQNwxD025yzDZFFE4QIr83d7DM9acWgQkeGuC4yYicvOgpNc?= =?us-ascii?Q?rLMkWFiaWPFiPK1h/H0DGagh2oiQS+K1TilPjuoUmCP8Ce+oLXOPESo8hFBA?= =?us-ascii?Q?cIh5gvsmP/SKDqnY6IIlUjxSqSIlCJtHde9nbRbvYX+zPmYmfb4N6WHXVNTf?= =?us-ascii?Q?aCJ6+NZvJYxlAvORqWb4P6B8EFuLeaUZyaW1CLJCsttmNfVhxVTAhqCUwAM6?= =?us-ascii?Q?9kTqV0yAar7qgsnTYZ6qaW1pmZDG0r8I6G/ER6poPSQfq1sIJZvp1bMqQT3G?= =?us-ascii?Q?wTC6xuJxZqfqzzi3Cl1heVQ0lxEqplMUf+LnEudS9K6rzC6Y56Em2TgUwPxk?= =?us-ascii?Q?7e77CCsGwqARbc/2lmSQf1ftLjLcNOImDlPHIyHXxHPWmxqPU8PDhrhh2Wqk?= =?us-ascii?Q?ff/Xu70DtnZoil9DXYV8BC8CJ3M3lq0diILkOn+HKNrZMrok7xkCczZlzKS4?= =?us-ascii?Q?5+9GMhK4S92qiXYghMmUb2ojxRk13+Yuvlu1VjnKRARdvRKqbNvAzWg2/Owp?= =?us-ascii?Q?5rXG15LN3679itt9x4HkKuf3PBYJLZdfOE8JgN3I+Ghgb62wzkpy3hAH+sqq?= =?us-ascii?Q?WkqdyHyhr8t/3BG7+2qXLIvHCyWgKvbPxqCdRf7mDd9jX9qQ/lgYGtqCxAiR?= =?us-ascii?Q?8M8kj90l801rf7j97Mr2hH/eNd7IQOM/ZGfToAlJClVWPH0x1e8VZTaI10Vo?= =?us-ascii?Q?cS7ehAM1TMGfzNCXlzu9TSy+isXwoVmscajE85pg0jQHLiou7Xzqy5+62Srg?= =?us-ascii?Q?J2EX491+Ju2JM8WIRuLWSkSb8ZU9TYjkVqoa9niXZ3SWXbYHRJ6wfgTjxM8z?= =?us-ascii?Q?5P2nrEm5VhZdH4RRF1G8LnxjDd0DbmtvMKRmAQdKyWeGLZrqs8n0I3BpGk80?= =?us-ascii?Q?bDBcQmFiSnIrHmm35IP6EGK76KAwZAzn7FIO7ZASsCl1+qJ2h8+FH8WyHWNW?= =?us-ascii?Q?unjhilSNwJGyB2kJ7Jw+eqZLzcYzopkf0uEIIc3FuvPkFMIm7vNeCoL8D/Jm?= =?us-ascii?Q?0nS1FK8zY8sblrU0RmLS3RQS5St82jh7mLmfsb2mIum+lS/oz+iLAXXUMg9Q?= =?us-ascii?Q?mshIC2ZLwpJ7QYgTwFtL+wCIrB7H/9Xa17CT9FYRWriZLEzHj+AKGfCKRxSC?= =?us-ascii?Q?5JFHk4FZYcKIzE51GKsorR96z2FdVgr/2nfejGJxGB4onNjh+gPjQ5wkeUNB?= =?us-ascii?B?QT09?= X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1080;6:/v0e7imKvVsdQ9HvoO53K0mQME0kc3cbug0V9CQi2dxpene9MqR2XM5ZwYcmJGqu3Hrtj7Oy3EXmINsiyyCGm4XO0wzF1TOWUI/Qx72F7axZyhkBSaLAIYVSxS89ytV3sbIMkGFkKkyw+6LU6OJn0FXjaJhBGpa6dBypm6uy0UaH3Ike1UVfeTy+wmQF6El/FK774UF7WMvAMvGWoziiVwAzNRt+EHC4J2CKB1AvnpT/cheMp5hlVarq93KbmA3VzcojaHkNyS9T1zpVtfjumAPTlkjxaS+xBzWmixnp4qO662WxVZpjx5RgqlHrbcyupy7b2M/xCgP8OZh3H7KeRg==;5:4HHq5tbjw72Dd2rL2F4fSYoiMvBRv0mU47HayZJImVigOYvvbffMg75W6xOge/dRHPI0/NYoHAaHfgKBLQriESaa2ZDqMom64FsoH6135VNSkWKJ/PR4CG59HHsCPeoK8Ysh9F7p353usCB/C/PNAmP2xMs1Wn5g4jbTMVbcslU=;24:+iEdE6VYT+lEyNHhYZJA2O6zjMEcrL2etEYxLLTfIBu/bDNS4pQlQkeOfokXXTT/NY2cSFNEW9xDdDh8f4XgcCXpBXjUxdsdNZ6ewhAfOpw=;7:JjMcw2mgKtbNPDgx2qs4wa4fS9nEEGv0aG8wUX//j3fmLGu1JB7GxZ8PyC4Zvy3qU7+eiorgB2bH3gBzI4BRXd/UizUrn+ykQKrJ3img1nzQvARDh4fvdr0UZjQOeC12qClbgRsWuH2XRNddutDc2SjLOppwleR2jHFZx73AocjRuxJ4w/jf7hLRovEo8ouIsuym+GbBG0LopF0XhKjAZLFpihjZCknca5l2MkE/aVQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CO1PR15MB1080;20:qpydU+wH8SnduHw/RPMyJkQg6lLHaiKd5taevkXCn6b16QUafYUkHzKuqT9Fy8JVAj9rEIWHR3g+Xi+o4sipyrbdZCBjxemv4GUc1Qi7OaMzILB/2vMJgUwxq21QeGbZ5lo3RwSx3TLO3pnDWiC36itlMmG2C3vVDn5Sl3ewxmg= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Aug 2017 14:11:21.9522 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR15MB1080 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-08-24_05:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Aug 24, 2017 at 03:48:59PM +0200, Michal Hocko wrote: > On Thu 24-08-17 13:51:13, Roman Gushchin wrote: > > On Thu, Aug 24, 2017 at 02:10:54PM +0200, Michal Hocko wrote: > > > On Wed 23-08-17 17:52:00, Roman Gushchin wrote: > > > > Introduce a per-memory-cgroup oom_priority setting: an integer number > > > > within the [-10000, 10000] range, which defines the order in which > > > > the OOM killer selects victim memory cgroups. > > > > > > Why do we need a range here? > > > > No specific reason, both [INT_MIN, INT_MAX] and [-10000, 10000] will > > work equally. > > Then do not enforce a range because this just reduces possible usecases > (e.g. timestamp one...). I agree. > > > We should be able to predefine an OOM killing order for > > any reasonable amount of cgroups. > > > > > > > > > OOM killer prefers memory cgroups with larger priority if they are > > > > populated with eligible tasks. > > > > > > So this is basically orthogonal to the score based selection and the > > > real size is only the tiebreaker for same priorities? Could you describe > > > the usecase? Becasuse to me this sounds like a separate oom killer > > > strategy. I can imagine somebody might be interested (e.g. always kill > > > the oldest memcgs...) but an explicit range wouldn't fly with such a > > > usecase very well. > > > > The usecase: you have a machine with several containerized workloads > > of different importance, and some system-level stuff, also in (memory) > > cgroups. > > In case of global memory shortage, some workloads should be killed in > > a first order, others should be killed only if there is no other option. > > Several workloads can have equal importance. Size-based tiebreaking > > is very useful to catch memory leakers amongst them. > > OK, please document that in the changelog. Sure. > > > > That brings me back to my original suggestion. Wouldn't a "register an > > > oom strategy" approach much better than blending things together and > > > then have to wrap heads around different combinations of tunables? > > > > Well, I believe that 90% of this patchset is still relevant; > > agreed and didn't say otherwise. > > > the only > > thing you might want to customize/replace size-based tiebreaking with > > something else (like timestamp-based tiebreaking, mentioned by David earlier). > > > What about tunables, there are two, and they are completely orthogonal: > > 1) oom_priority allows to define an order, in which cgroups will be OOMed > > 2) oom_kill_all defines if all or just one task should be killed > > > > So, I don't think it's a too complex interface. > > > > Again, I'm not against oom strategy approach, it just looks as a much bigger > > project, and I do not see a big need. > > Well, I was thinking that our current oom victim selection code is > quite extensible already. Your patches will teach it kill the whole > group semantic which is already very useful. Now we can talk about the > selection criteria and this is something to be replaceable. Because even > the current discussion has shown that different people might and will > have different requirements. Can we structure the code in such a way > that new comparison algorithm would be simple to add without reworking > the whole selection logic? I'd say that extended oom_priority range and potentially customizable memcg_oom_badness() should do the job for memcgroups. We can extract a part of the oom_badness() into something like this: unsigned long task_oom_badness(struct task_struct *p) { /* * The baseline for the badness score is the proportion of RAM that each * task's rss, pagetable and swap space use. */ return get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS) + atomic_long_read(&p->mm->nr_ptes) + mm_nr_pmds(p->mm); } Also, it would be nice to introduce an oom_priority for tasks, as David suggested. > > > Do you have an example, which can't be effectively handled by an approach > > I'm suggesting? > > No, I do not have any which would be _explicitly_ requested but I do > envision new requirements will emerge. The most probable one would be > kill the youngest container because that would imply the least amount of > work wasted. I agree, this a nice feature. It can be implemented in userspace by setting oom_priority. Thanks! Roman