From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753007AbdLEQCM (ORCPT ); Tue, 5 Dec 2017 11:02:12 -0500 Received: from mail-eopbgr10117.outbound.protection.outlook.com ([40.107.1.117]:57216 "EHLO EUR02-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752915AbdLEQCG (ORCPT ); Tue, 5 Dec 2017 11:02:06 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=ktkhai@virtuozzo.com; Subject: Re: [PATCH] aio: Add memcg accounting of user used data To: Michal Hocko Cc: axboe@kernel.dk, oleg@redhat.com, bcrl@kvack.org, tj@kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-aio@kvack.org, jmoyer@redhat.com References: <151246799787.29619.13497559380263553588.stgit@localhost.localdomain> <20171205151532.5evofjjma2zg3kte@dhcp22.suse.cz> <7f440fd7-872e-ff17-a152-6ef6dbd10a4e@virtuozzo.com> <20171205154301.p53ut2osrykveoso@dhcp22.suse.cz> From: Kirill Tkhai Message-ID: <82853c8c-ed5f-4ac2-cdc1-f03fcca7505d@virtuozzo.com> Date: Tue, 5 Dec 2017 19:02:00 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20171205154301.p53ut2osrykveoso@dhcp22.suse.cz> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [195.214.232.6] X-ClientProxiedBy: HE1PR0102CA0034.eurprd01.prod.exchangelabs.com (2603:10a6:7:14::47) To HE1PR0801MB1338.eurprd08.prod.outlook.com (2603:10a6:3:39::28) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e9e6f061-396f-428b-db43-08d53bf986d2 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(4534020)(4602075)(7168020)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603286);SRVR:HE1PR0801MB1338; X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;3:0Zj677DnxYIfTxkHGBLHpD3xdoNYazJAt2QTcVp10qQ7kCVBjEV5GKKwsfwB+Aqh2c2PA6hmJKJbzpnMboOSLJ2XEB5YyCfKi2AsqpnWBxzwrVgdjXqIDpH9KqJyYkmOZS07e+eV/Md2eZU6KcM4BcmZ3L7TzSVOdG6aoy3mA2HlQmLrTiMh0xJ+dZRIDlNSBJeSc5ZfDWdJ2dDfa/xKivLG63OlUI2Hfv4nFLhQlIRIfYmfbH5TQ6AWoPQvzgS4;25:XvpgIKU80dnKbDfSqJ98h7eTDEm9V7YqzOD+IHVPDv1wJYjdlBloiXzNBftlKYY3TVepXII9Abh/+2wENWqczgPR5KUxcg90b9WDl08QFX1pn8IBUN2gFtcHzkcQINhy8HInSLIxrNISSVE+ZnebKmBn0FddhXE3U74fm4WXrX4hlP5ktCPIEDIdIF/+ZzMDvcqYpNjEtmtnM3e3ZAvJHb/DI61J5IK+lSCzEYo+ZbKAwWdIAxqlfnw1qrREY19mpWUzG+aazucHbU2rL/XXr/Z+Rd+17BNW8zA6v7swMPTjO3YJb8thF/aphBfnn+3vu8f+esLRUU24IxABu1YJPQ==;31:/xmlyDQfJuZgQL/uhhWP2IFLrzoNxxOgCTU/a+rBPGu67Am9wCb+UImzCcACKHGSaQAaxpULLdqOjlbW+9F2N/q/EjlyIKuO5pxeTVvNPmrtksHIhqIdTtxGSUGd3mCMqrUkrPBnbsyOP4q8X+eHa8VOSt2cJ8QiQVDaH9ktpN663OUqr6ieNiZpBC3uxog6Esg/4+tgSiNIT3M41T8Dce+XTUffInUMDBGjWo/S7+U= X-MS-TrafficTypeDiagnostic: HE1PR0801MB1338: X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;20:q9CxGLnNJICIbyN6QtEWecVhK+5/3E0lFxzH88DkV0BNkOGeZGh5zZTg4Z9WGizIXAc7k2vRFK4/mDg2wT5jW6dkha1HBvXNClnd2hn8I2NS3rshH9d2AfjVwRdwQeN6UX2ErBxfOBhVhymzozt2akMoE31IYtP1MNN9aynFo6cPNZEfifOlZxKNqPQNgIzZ4qWpCUm7TJmWQoCwmZFTzL9N96his8D/fIYziIs1sm6smYGx7AtamP6A1E0TazARrtTTi6GWuEYqG5Ssnh9dP1y4tUfh5DblxL9S7qheOf3Vph8M0lzgaWGMgk6OrG+4/cT5aVEVe8CIvt4Y9bQlGiv/qxuiTmw09QtxIGrAwqfdKRYYfgQZpAKfdEjUWPXF5Ekh8HNbuv4k0llH2c7l5YGOowjjjPj+mIShwz+ptz0=;4:xVybwKSPPA4sgRa09o5Xuj+HBUX/AnbKIXHi8QzFplR0b90I1/KCQ/WTyQXx5oNPJ2mnPahqSFQF/PrmLY0jYlKtAUUoLGRdNf6FbiPUSxTEjPrC6Pflr7hVY7xhB1zVW7lIPZ97lAWQF40H5TAUWWuzYo61MAP7EEMvCjHoELBCaXkU78NHMzC8UjtZBf6U8d4CmthDjuyK5jsB4Vk95IEKc2JqhwUlbYDlv60/giePH2P8nTCX9XNOzH+rHr0PpT0Y96HoiVdv7PtU4T0dBQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3231022)(3002001)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123555025)(20161123560025)(20161123564025)(20161123558100)(6072148)(201708071742011);SRVR:HE1PR0801MB1338;BCL:0;PCL:0;RULEID:(100000803101)(100110400095);SRVR:HE1PR0801MB1338; X-Forefront-PRVS: 0512CC5201 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(6049001)(6009001)(376002)(346002)(366004)(377424004)(199003)(24454002)(189002)(64126003)(58126008)(55236003)(5660300001)(50466002)(65826007)(93886005)(52116002)(16526018)(2486003)(52146003)(23676004)(230700001)(68736007)(36756003)(83506002)(15650500001)(189998001)(2950100002)(81156014)(6916009)(316002)(2906002)(7736002)(6116002)(3846002)(76176011)(16576012)(97736004)(81166006)(53936002)(65806001)(106356001)(33646002)(47776003)(65956001)(66066001)(8676002)(229853002)(8936002)(105586002)(53546010)(25786009)(31696002)(86362001)(77096006)(6486002)(305945005)(6246003)(478600001)(4326008)(101416001)(31686004);DIR:OUT;SFP:1102;SCL:1;SRVR:HE1PR0801MB1338;H:[172.16.25.196];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtIRTFQUjA4MDFNQjEzMzg7MjM6bzZUV1gwSHpaQVk5eGltODdXbnJ2b2Zq?= =?utf-8?B?T2tDMkRCb2VvbGFqRE4weGNLYmUrYkZGMWpNUmVCcTlGMkRjOVhHZTZlT0Np?= =?utf-8?B?YlptMjlqc1FWZUFhMFpYaGhCT2FqVGFnLzNLVUc1bUgyYzZ4alNYTjRwdlp1?= =?utf-8?B?TGNKTUJrcHU2aFZCd3FZeHk2MXNyc3JPbU5WTDN6SFJySlJPb0pPdzJOK2ti?= =?utf-8?B?dytCQ0tvVWNDcUJHTXpIT1JmdFdBUW94YW1MQk9tbysyMWlJcUx4Y2UycjYx?= =?utf-8?B?RjVaQmNYYjlWMnBNK0NlbUVSTTJOQXhnMWdreS9UMW5meGVLQWE0MUlIbFhE?= =?utf-8?B?Y3ZRVFZwVUI3ZTdyV29jM1ZpckZHUkg3K0g3Qmx1dXRHMy81WGZpRXF3SEpP?= =?utf-8?B?RHpCcXJDYTkxMGc4WUwzRytQWDNpejlhQWZzbGsrNVZRcVhNN3k3K3dSQTVo?= =?utf-8?B?a2dOR0FMcFhzRE5FSlpvK3JJRzBRNXNDTk55c3NOUUF1LytuRlFOQjJJODNF?= =?utf-8?B?TlVBUktiSS9MbDAydVZTdi9TdEp3THlTRC9YZjM4aHlaeStpeXJHWk5uaTMv?= =?utf-8?B?S0xXQnBka0xvRlpEQWc2TGJhZmQ1aW9rVkk5Ymo3ZnpubHQ4SThTZER5UThG?= =?utf-8?B?Rngrcmp4cmMyQ0xBQXg3SmNBMDVvSVh0aktoOUFPeW1TcllmakxMZm9ZdytX?= =?utf-8?B?aGo2aCswKzJScVdyS3JXbERFVDE3NXJHcnZoNFZqZFF1RkVpODFKempwODhu?= =?utf-8?B?ejN2d0dhemVzYVRleHo5bFZyazFBcEMzMTBtUWpsZWFQdGNlNHBLZkR4MFha?= =?utf-8?B?N21CTzdTamlreUdBR3VRR0pEa3JvSUsxMXlzdGROSmd0d2pibFJwTStGQ0Uw?= =?utf-8?B?a0xVL0hmVUV3czZiYXBVSnVEakRDMUhzVEUxeG1pcXBaT1NyYTVtUWtLd1ds?= =?utf-8?B?WXE3Z2pEa3pMZDdsaUxPSERrVHIvZGcrQWU1d1RCMi9NdE1tWCtHRzlVRXUx?= =?utf-8?B?VFg0Ym13SlZYWG5heUEzNXpVVUszMU5HaXZVcDFmK1I3WjVZR1QrYzdzZUE3?= =?utf-8?B?QlpxSGVqcitReXlPRG1IeCtvQWJEQWVZRWhFZFJORTgyV2VjeGlnb3MzMkVr?= =?utf-8?B?SEVGTVgzeW96KzFiNnZUVk51TThXQU1CRWtGL2haQjhzTXN6ek1iclZsTzVK?= =?utf-8?B?MUNlRlpIdERINnd0NWlWMXpKY1lsNE9wVUdQWSs4UnpHLy9BN1d3bWdPS3F5?= =?utf-8?B?REJNMzJCam1IbEJxbjVsVHRobFhPMzZGRXFQSGhEcS9Wb09PUTVKVmgvMlFp?= =?utf-8?B?TEhsU2swOGVMWHRTeXBuU3BvVm01UyszM005NjVZZStDVzBlMXVBMW1NTHZh?= =?utf-8?B?Zyt3Q2s1aU9SdnhLRUFiZkJON2pNQ0xtQktwQWpqNCtxaTZtS1grdlJuTmt3?= =?utf-8?B?b1pOcDNkd2hBZUQ2eDVvUGRRbUtJbEtZaTZHa0VlS3VNb3FsN0RlMSt2WGZS?= =?utf-8?B?YkdIOE0zN01TeUUzbGw3bmI1UHVZbGJYaVR4eURPejZsak9OcW9vLys5WE5y?= =?utf-8?B?K2tmcStOb0M4YmFSMHVJY3NVc3NaSUI1cThHcmpreUdzd01TQ3FRWi83Z3F2?= =?utf-8?B?dTNEekRabE1NZEV2VUgzcmNqS3llZTJKWENtcytIb0ZhL1ZJUncrRFltUDlr?= =?utf-8?B?SUxZcmhMME5mNGlMdnpDSEMrUnNETjRNNkV0dlFwUVovUkk3QW9WaDN0OUxw?= =?utf-8?B?cm5laHI4S0t2d015dGYzT25WU2NIT0xoVG9ZVWoxZnI4SHNTZjUvWXI2eGJK?= =?utf-8?B?OHBqaW5TMDRuT2NITzV6Mnp3T29mZHpEVlVka3RaaDR2MWtoemVmSzNmUFl3?= =?utf-8?B?TDBnWGU4OFFiWmVUNUtoN3Z5OVlpVHZ0VVdXdFhoanFnV1V3RXNSY20zU04w?= =?utf-8?B?R0ZrNEVQejh2S0E9PQ==?= X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;6:er0Y4Ks7aUXp+75X/hig3JBcoUHLxkIbg/Zz6Wba6RuVqpsIyNArxNT+pd86HXa/dj/zyCV2XJFWjxCCaTk229DXWG18QOJO/iZTLQ4644CcwKamPlPN+JR3SfC//B+THDPKaKcVI3hKQNftDPYp3205uNnpphOLsrMPBIRcZMu8AWh6wVe+XCLXqat4/fZBnkUqAl0JAwit/NHrX+GjHmSvYRbdJdStTc1Ycm472dmEijHD1ZjNccr49sozH843zBDDYmOoQtQFV7b/OPybfd4zPhpVyb9OmrpumRhoVQ4L6r+1DiErWafBNRqUxhycATDvdzIcG5pwJr4yzxtH+ilfGqqv7GHGYNSUfmIB5Vg=;5:rNADyHFPF87h84gGOTk0D1B1xpGwd/zkV4SRiU40Z1KCMcRz+3oznK/PDbH3vb2ssMjVT/TDrTWrnOk3T8scwnWsNIB5TNDap7fORIw25MEV28aZ2AuXBXzN4tLkuYKy9o0TF1XoQTUZSnMFmlY0K6gXB4j5sPFhbT3ECc2eAg0=;24:++XiMzimy3WYL3Yt5ZEgdAvGnilCUtDZkz9SwbYxFEvFZ7V4E/NJ6TQ6NfWGPZq8RED3jN7TuHDTCJ7WhcKMipEpVbiiEWgZoOFSvdA8WGs=;7:PlUTw7QmlsHcxAwVrZrI9YQfLupy07xXB4t/Ol1lfEb8oGT+QyuzKCCQODexAJZJSWO/3h43KitXzU7IJk6yy5ub/iaMAEKuL6InEnaO1d1Hbjqr0lZ7Jfz4CYjkH/3Otf9haYQ8zqfDLPD0oVzKnyPkb1jDX07ABRGF2/0vZgyYKiRjA5L3p4JQoT4kowyoG46ScVydDXUVT05UvRR56Zj+QCdL3VP53OTb0Fw4AJmUOIpHD29GosFpvnkCYCaS SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1338;20:KVe3gvNdSIAfVPJ828LynKdaFwHyrICCz1gLdyYbgRcpkgtNkDTKqN3/0r0Ptn3iIfIGtAj62Xm2ykGotV5eaEIKr6jD0vb9S3Jwzvxa3K1XAkCctpqVLpC5Tir+DxwEn0fAUVh3Em5tiLdt/QclAIvKDl0QZqqwgZt8HRe9e54= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2017 16:02:03.3747 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e9e6f061-396f-428b-db43-08d53bf986d2 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0801MB1338 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05.12.2017 18:43, Michal Hocko wrote: > On Tue 05-12-17 18:34:59, Kirill Tkhai wrote: >> On 05.12.2017 18:15, Michal Hocko wrote: >>> On Tue 05-12-17 13:00:54, Kirill Tkhai wrote: >>>> Currently, number of available aio requests may be >>>> limited only globally. There are two sysctl variables >>>> aio_max_nr and aio_nr, which implement the limitation >>>> and request accounting. They help to avoid >>>> the situation, when all the memory is eaten in-flight >>>> requests, which are written by slow block device, >>>> and which can't be reclaimed by shrinker. >>>> >>>> This meets the problem in case of many containers >>>> are used on the hardware node. Since aio_max_nr is >>>> a global limit, any container may occupy the whole >>>> available aio requests, and to deprive others the >>>> possibility to use aio at all. The situation may >>>> happen because of evil intentions of the container's >>>> user or because of the program error, when the user >>>> makes this occasionally >>>> >>>> The patch allows to fix the problem. It adds memcg >>>> accounting of user used aio data (the biggest is >>>> the bunch of aio_kiocb; ring buffer is the second >>>> biggest), so a user of a certain memcg won't be able >>>> to allocate more aio requests memory, then the cgroup >>>> allows, and he will bumped into the limit. >>> >>> So what happens when we hit the hard limit and oom kill somebody? >>> Are those charged objects somehow bound to a process context? >> >> There is exit_aio() called from __mmput(), which waits till >> the charged objects complete and decrement reference counter. > > OK, so it is bound to _a_ process context. The oom killer will not know > about which process has consumed those objects but the effect will be at > least reduced to a memcg. > >> If there was a problem with oom in memcg, there would be >> the same problem on global oom, as it can be seen there is >> no __GFP_NOFAIL flags anywhere in aio code. >> >> But it seems everything is safe. > > Could you share your testing scenario and the way how the system behaved > during a heavy aio? > > I am not saying the patch is wrong, I am just trying to undestand all > the consequences. My test is simple program, which creates aio context and then starts infinity io_submit() cycle. I've tested the cases, when certain stages fail: io_setup() meets oom, io_submit() meets oom, io_getevents() meets oom. This was simply tested by inserting sleep() before the stage, and moving the task to appropriate cgroup with low memory limit. The most cases, I get bash killed (I moved it to cgroup too). Also, I've executed the test in parallel. If you want I can send you the source code, but I don't think it will be easy to use it if you are not the author. Kirill