From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756108AbbIXAMp (ORCPT ); Wed, 23 Sep 2015 20:12:45 -0400 Received: from mail-bl2on0136.outbound.protection.outlook.com ([65.55.169.136]:60964 "EHLO na01-bl2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754981AbbIXAMm convert rfc822-to-8bit (ORCPT ); Wed, 23 Sep 2015 20:12:42 -0400 X-Greylist: delayed 36436 seconds by postgrey-1.27 at vger.kernel.org; Wed, 23 Sep 2015 20:12:41 EDT Authentication-Results: spf=pass (sender IP is 206.191.229.116) smtp.mailfrom=microsoft.com; fb.com; dkim=none (message not signed) header.d=none;fb.com; dmarc=pass action=none header.from=microsoft.com; From: Dexuan Cui To: Chris Mason CC: "Theodore Ts'o" , Andreas Dilger , Tejun Heo , "linux-ext4@vger.kernel.org" , "linux-fsdevel@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: RE: ext4: performance regression introduced by the cgroup writeback support Thread-Topic: ext4: performance regression introduced by the cgroup writeback support Thread-Index: AdD142fYZSxUpRKBTkye5CjsMXMB2AAN3K2AABCd4YA= Date: Thu, 24 Sep 2015 00:12:30 +0000 Message-ID: References: <20150923161359.GB25218@ret.masoncoding.com> In-Reply-To: <20150923161359.GB25218@ret.masoncoding.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.57.133] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;BN1BFFO11FD018;1:AA1KmSBavfmmbiH19YHUMPVVJZOzgpRFndISTqXMIao1Q3UYu5tm5QgWuD0Jtq7oKITdIIW7zg3FnuO4fLmVIxEuwTXTX4ZPgpRW+uPwTV3+7LOi8hZrDZMUcYbJbeUOds2qlHzQXLSiy/kvOIAVyAGVtBackjvuiFIxQ9JWIdZYILkCSZanAUF4KjLidL+2mRwf+vkn6bHjZDVb8aXHm0Q35oISZ0BDPR3K9cRVlFHo9O7QpP9xDMv9w9K7YpPX5GQpYwg4yQSp9RFOcLuxrmt2mCV7F8GD5cY51qY4ZdnJj0r6vuhCVf//VT17uCkZdF1rLUzozMk4wxwtVJM+IzY8/i21V93PZfqbj7a9pMI= X-Forefront-Antispam-Report: CIP:206.191.229.116;CTRY:US;IPV:CAL;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10019020)(6009001)(2980300002)(438002)(13464003)(69234005)(189002)(199003)(3905003)(164054003)(53754006)(24454002)(50986999)(66066001)(54356999)(108616004)(10090500001)(76176999)(47776003)(86612001)(46102003)(33646002)(10400500002)(5005710100001)(10290500002)(19580405001)(19580395003)(23726002)(110136002)(5001960100002)(5003600100002)(4001540100001)(81156007)(189998001)(46406003)(24736003)(26826002)(97736004)(5001860100001)(5001830100001)(87936001)(62966003)(77156002)(6806005)(68736005)(2950100001)(2900100001)(5890100001)(97756001)(50466002)(15975445007)(106466001)(64706001)(16796002)(575784001)(69596002)(102836002)(86146001)(92566002)(5007970100001)(5004730100002)(86362001);DIR:OUT;SFP:1102;SCL:1;SRVR:DM2PR03MB368;H:064-smtp-out.microsoft.com;FPR:;SPF:Pass;PTR:ErrorRetry;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB368;2:dP47hbPOhhdslkThbVasnA0rUSEwYvRFLVImkFt1HtnmWDA7K/SEo8KPnLQV6ew2So67fvmze2cIQlae0VBbWDm+huE42bkeSr61/pWl1pfN8fuR/HqdE3E1pqRmc9+HnPG5n+fpRxDcI0uI8as/1QQBGoQGgF1rIlkIZlHn5Ao=;3:0Xoa5pOi+cjCLWW6Mh4CtkdTvaHUcs+6z6cQBzz633vhoKQ4HkxUUDXRfs3ospxl6fVA64LNrF0j6NQS1p4bIbV5uok/Aidu2RYemAxVk766tr5HGCEyJxnbxp4hxZ61V7XbrHOw9ZgzYOHAROUU7BUT7Az3Z1TDgTXhMZJnec7+xRZYH/BJDxQzkM8lNQKcp2H/uKA3+Mr59xO0APUOqq29G6C649nEtgjDjAFtNlVwuhPFuYpYrLn50Gp0JzCbhnMvsiA0klgSJ5fFnHJOCA==;25:ILNklRBRmnqXpWTrhKP/EZGnR1tnBhJNzkLmjqZ3dxShLax1yM23z3CBajD1PZUErt7YJwsDBTZYZJTrIT0QjWVck0tHp5cLt8PL8vQWBKyMDjc6QGBg8MSUCgihZ8p6SJrCbZnlFwPqTKsUVNyOy4rBkfvvX/B8iklQRNC7gbNLWYEGoatskRieoyLir9mRp6/NtOqarcrZUVsMKHtlVp8CKN/qMD8HECHAKO3yXMb8n0ovWqO4JywFyd/jx33ajUOPi9c87yA4X7CmlA+YmA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501001);SRVR:DM2PR03MB368; X-O365EOP-Header: O365_EOP: AllowList from IP - set SCL to -1 X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB368;20:3MsLXdDL5fBd0KEZrlkyHl1qt1W9AUvB9i1mmejCjS3ceaDLwpdsqXhxsK66IN33cJ3lUA5hdxWx4SjKvoB59lWdrkyaHQtChUQ23ox//+xHP6rlxM8T/RBGbwq8p7d0tA2iZ3oF/Nfxp3tko1XG5LanvkjkeM7IR6zj+Mp8Hf65w5b1H62B+IiA9/mgEtdKa1gCqqeuOPRL+5z0s15Tz4+T6c9ztlsL5MJ78Q96HTEHKQqVCpeJH9XldmhRQNMc1vQ7dl0IbmSIKu8nHwKExK1SFPnzhjnAWo0TUjb5MyPWpA3r4n8k8Sbz4GrXSwkd3+zopcZ8zfV287K29KH+u2lvIJwuj73yEZWKVaCV3ZGBPIWST1fFyExkkjBHYFWDh34/jY/jyl6VSqcSYLE5C49g7QjsjVAbigChlIuJfnGGX824QKVT8sNlkrTOXv6hzRdcgbdtzoYW54cDkBaLc7mBhTI0UHmDiTLC6HsMJYbIzI33AQy/9zfj03XDlLEDgu5knJOTS21bcU9DCv/KI8oT6de9VAxtLVqZpyz+QyTDbAhzcuo1AlrmVyvhIEyk3+cMuZOrKIhNuEZG6wB0k6vtkuHV0YNmE+m0T+KokFo= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(108003899814671); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(61425024)(601004)(2401047)(520078)(5005006)(8121501046)(3002001)(61426024)(61427024);SRVR:DM2PR03MB368;BCL:0;PCL:0;RULEID:;SRVR:DM2PR03MB368; X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB368;4:YmV85RYaOVTPfZ/HWIqKxyRBh5syJ4Xv846b2KhRmQZOnyjzUAiCDn4R1LNz00vPXNXIsuV5xnILBR9j94s3MvMczqadtTrm1d3JfJcluoWrOe7x9wAwcJnO8ba+R2mUAhEL8KsFLXxvlzPnuBl+6uv/rauWLpAeEWUuI8wIMsEPWaqVk3kAPHbv7TlbbJiKS/+vU+IRqsn7fwZnpDa3kph+W5qo545OtmJxins2lhnaoHM4MaNPSboZ+KEJrFQbEAdQ8EbPjGi4i08yYcpHAFyfmRHUbI4x5IsnJ9BvSK6oRSWIbbL05TZw2pB7NZm7VymK6kbHuY9BDWueA2yW7VzQk8NcxAlApIUmCxJfEqqLBH02oX0P+XPfixv7c3Bi2EIbds5FObK9e9xNjjFzc0zX3W2o3np2mw6WKw4sIyk= X-Forefront-PRVS: 070912876F X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;DM2PR03MB368;23:sk4LwbEvB0hJqnHdLdP1Q4b1Pu6wxKe/WchpYcYaPk?= =?us-ascii?Q?6IH1/5Xj470m53c/83sNGb4HXTajmIJU2H+EC0F5Iuwq5WaK6z8IipQqLkEF?= =?us-ascii?Q?NjlMiFNiddosL+yZeSycgh3Vi48so4Pj5/Nk0hl5g1LRCpxdQri7+gSU8btl?= =?us-ascii?Q?g6Tq7kqMMYCEaTfoAWErnPsqXHLGW9k6dHF1P9EmQKDfXs9f0r9eU7LHoJRj?= =?us-ascii?Q?6NhKBqz0v6AH4d1iwqLH6NSLKxxQlBsoLZ58/zal90V402NFj8CHV7/AD+kx?= =?us-ascii?Q?SFHJx7k4/WzlPi5RnOrC7g6e9HnRjjQFhuR9+Iki1i2oxVxDaKAn+nGe0r2A?= =?us-ascii?Q?N0XKeULdk5XGbiIg823a8Fuiw0Sfir810DQh/xoI49DpONYEG0ZkxhMP8Qxn?= =?us-ascii?Q?eZRyiE2wKoWw0IVv4/vzdXSMf8PADTUzKLENGO9SQPRLAPW08+7xQx+6KTvk?= =?us-ascii?Q?Pi/CtRWFGBw16yjungtcAq84G2gm935m8Xd3yzfDAIfDYN0OeVibpRBuKS+e?= =?us-ascii?Q?ZUntaPxkLAkrdHzu/w63KVurtQomr73ApBkbA+ojbWit5O022oQS21aAfTbs?= =?us-ascii?Q?OvYNPqJ1ptpDMXicV1B9SYSsK/09/DwvXyva5dJsmiStx27cW9rWonF5KvfP?= =?us-ascii?Q?aUbhKewggf/G1rxXtzLrX6caZ9pkiOE9TWJAGx6fzCgVpc9c4IpMrAQSOMLz?= =?us-ascii?Q?2vr7nrhTi4hh/5B+YU6TWEPpMQVRhYcNqlS4BfPsUR8mAv2Ovavy5sgX5bqN?= =?us-ascii?Q?GQfSQ0um6ebGheA9W4WXf/39ivYCoIX48u1fMBNiorJG6HP4D0wjEGbr4cCX?= =?us-ascii?Q?+sdL7H+A2E8iTqdSbbR2+KksUKDVS+07O1gM0i6jSCuQ4wRjzn/Uq2aT8NLk?= =?us-ascii?Q?fwCZ3pBwzx56vaGOvUqHhZBU3DXAmVdDDSgb4lye7muA2rr5tDIKh72Wb+JC?= =?us-ascii?Q?bm39r0kTtbLIc4Gy97zBzWLaVBHDh7B+PVA43gbfT94GSySZfCTZzjHYuysm?= =?us-ascii?Q?aXfxrQAmsjT6kXvmUWh32qILD6xp4M1uhi0cYxaQQbyfdoZsrBmrypOOgpB/?= =?us-ascii?Q?YHkqbvQUGO1uvd2T3G22RybbmT24HOZC9jgCj+KxAoaWN4+949mWPtTCYmF+?= =?us-ascii?Q?J8rjfNdu8DkUZB+9Q5K+3ax2h3UmovJ6IXvULsrTsEKQppemS/wQMNtvodDS?= =?us-ascii?Q?9Q6WH0mRz4isc5HIUDbugBf8QW0ExV9gT+JYWm0bHJiY4OXCtrP4faRAkO1R?= =?us-ascii?Q?inCRsIxLg+4blgfIuM39D5iYJbecXGDVeV4osvU5WcrNKFQjrdqDj/KLJATZ?= =?us-ascii?Q?BtKfqw1AQqQ1RNSPdO/XOE2NqO9dayzvsG5Sle6tzWv1KK8GvXJPWtdiPooL?= =?us-ascii?Q?UuK+utqrBgj/5KhvmyCZeAe2K5N6PRXiKxknGvVyTlpUZXTwoGoRr60Ho6w0?= =?us-ascii?Q?94CZtAqd4zehnQwt31YV6xzfC999nohv/cT0+1fW9ebF1sDE/MIpfm876Nts?= =?us-ascii?Q?3L94K95gyYudW4gLCWHRJHfnirKf1uLcXoa8kj2DOL3klhQuTFKMkGn8lXvj?= =?us-ascii?Q?Xw+0nxb3pAXJyyBQwawqVp7OujRWX2tn+a3hAUsHbcOW095SC/+g6VOsbC?= X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB368;5:eHP1ErqRDfHnQWNk3i329uWXw4XJzvyx+W4AavKc8EaKX/eMaILHpUlN7xZsOv6RWtqzMJ31IVbDwYD/Wdg+ME9ejAxBq6MZxRitD0fd+h8ZOKImISNt4N6lnGhKo2jWW9ArgsfjhOO85Z3NaV+WWA==;24:QOdZSiTt4Jq/ovRw10RdfgIzgFqhAcIj7yRP51rLUNJp+GulrCDpYeaDle8c7C59g1w11NIwzAzCX0b0uOsnHBat0KqEPCppocY32Y9OETg=;20:/fNNtUKwCx8KeCu5qIhqFtFNhPPxBuyqmggB4NVbxPQzI4YSJ38S86+1qz3RVWkPM/CbE/QB8A2H672tWD5Xzw== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Sep 2015 00:12:33.3400 (UTC) X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47;Ip=[206.191.229.116];Helo=[064-smtp-out.microsoft.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR03MB368 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > -----Original Message----- > From: Chris Mason [mailto:clm@fb.com] > Sent: Thursday, September 24, 2015 0:14 > To: Dexuan Cui > Cc: Theodore Ts'o ; Andreas Dilger ; > Tejun Heo ; linux-ext4@vger.kernel.org; linux- > fsdevel@vger.kernel.org; linux-kernel@vger.kernel.org > Subject: Re: ext4: performance regression introduced by the cgroup writeback > support > > On Wed, Sep 23, 2015 at 01:49:31PM +0000, Dexuan Cui wrote: > > Hi all, > > Since some point between July and Sep, I have been suffered from a strange > "very slow write" issue and on Sep 9 I reported it to LKML (but got no reply): > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2flkml.org% > 2flkml%2f2015%2f9%2f9%2f290&data=01%7c01%7cdecui%40064d.mgd.micros > oft.com%7c8001aa10249f41a0363608d2c432042d%7c72f988bf86f141af91ab2 > d7cd011db47%7c1&sdata=oJBsP55jdg86TNt2X71s0gfPlwbMTzaJN9QIcsXsSmA% > 3d > > > > The issue is: under high CPU and disk I/O pressure, *some* processes can > suffer from a very slow write speed (e.g., <1MB/s or even only 20KB/s), while > the normal write speed should be at least dozens of MB/s. > > > > I think I identified the commit which introduced the regression: > > ext4: implement cgroup writeback support > (https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgit.kernel. > org%2fcgit%2flinux%2fkernel%2fgit%2fnext%2flinux- > next.git%2fcommit%2f%3fid%3d001e4a8775f6e8ad52a89e0072f09aee47d5d25 > 2&data=01%7c01%7cdecui%40064d.mgd.microsoft.com%7c8001aa10249f41a0 > 363608d2c432042d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=QIcX > R%2flZMqkK2afIxV%2fYxZDug26vj5yx%2bkoh6ugJB2A%3d) > > > > This commit is already in the mainline tree, so I can reproduce the issue there > too: > > With the latest mainline, I can reproduce the issue; after I revert the patch, I > can't reproduce the issue. > > > > When the issue happens: > > 1. the read speed is pretty normal, e.g.. it's still >100MB/s. > > 2. 'top' shows both the 'user' and 'sys' utilization is about 0%, but the IO-wait is > always about 100%. > > 3. 'iotop' shows the read speed is 0 (this is correct because there is indeed no > read request) and the write speed is pretty slow (the average is <1MB/s or even > 20KB/s). > > 4. when the issue happens, sometimes any new process suffers from the slow > write issue, but sometimes it looks not all the new processes suffers from the > issue. > > 5. The " WARNING: CPU: 7 PID: 6782 at fs/inode.c:390 ihold+0x30/0x40() " in > my Sep-9 mail may be another different issue. > > 6. To reproduce the issue, I need to run my workload for enough long time > (see the below). > > > > My workload is simple: I just repeatedly build the kernel source ("make clean; > make -j16"). My kernel config is attached FYI. > > > > I can reproduce the issue on a physical machine: e.g., in my kernel building test > with my .config, it took only ~5 minutes in the first 176 runs, but since the 177th > run, it could take from 10 hours to 5 minutes - very unstable. > > > > It looks it's easier to reproduce the issue in a Hyper-V VM: usually I can > reproduce the issue within the first 10 or 20 runs. > > > > Any idea? > > Are you using cgroups? That patch really shouldn't impact load unless > there are actual IO controls in place. > > -chris I'm not using cgroups here. Tejun just now found the root cause: "Separate wb domains shouldn't have been enabled on traditional hierarchies " and supplied a fix. Thanks, -- Dexuan