From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932166AbbIXAbh (ORCPT ); Wed, 23 Sep 2015 20:31:37 -0400 Received: from mail-bn1on0132.outbound.protection.outlook.com ([157.56.110.132]:33997 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1755794AbbIXAbf convert rfc822-to-8bit (ORCPT ); Wed, 23 Sep 2015 20:31:35 -0400 Authentication-Results: spf=pass (sender IP is 206.191.229.116) smtp.mailfrom=microsoft.com; kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=pass action=none header.from=microsoft.com; From: Dexuan Cui To: Tejun Heo , Chris Mason , "Theodore Ts'o" , Andreas Dilger , "linux-ext4@vger.kernel.org" , "linux-fsdevel@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: RE: ext4: performance regression introduced by the cgroup writeback support Thread-Topic: ext4: performance regression introduced by the cgroup writeback support Thread-Index: AdD142fYZSxUpRKBTkye5CjsMXMB2AAN3K2AAAWTO4AACyRGwA== Date: Thu, 24 Sep 2015 00:15:37 +0000 Message-ID: <08750e508f1643b2a2c0bb495c91d450@SIXPR30MB031.064d.mgd.msft.net> References: <20150923161359.GB25218@ret.masoncoding.com> <20150923185337.GK26647@mtj.duckdns.org> In-Reply-To: <20150923185337.GK26647@mtj.duckdns.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [141.251.57.133] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;BN1BFFO11FD030;1:eK6aIyZ/eKA7ovm/yCsaTdKq7FoHUqwzLux6eLIX6R79pVH5QsQHjmGs54yKpyti2J/FxEoYAlYjAsyDIk6nuxbDa4+7n1037LaEQGsDYBBgBOgBerV7W6olZ9b6+C1+yXuSZGrYwZ3PJEULkW0n90vAlP54Y/QA2fNYhl5IiTAaQw1PCysT5AdSedMEHEgBFwP/LjkZVUlQZkK/gfYTHklS5jYGf7jzWYbS36ixerF+PzFee+LSrzExQAh6fwyutFUad0SkOynAWOBVB7bTHvEwIJnV+wqowq35Znj9xHrKBsPn3OMlJCQIhnwiMtlHArwTtg7KvMufq3DmoLF6ZaO3JZ7Y5jB1jSJGXCgsdOY= X-Forefront-Antispam-Report: CIP:206.191.229.116;CTRY:US;IPV:CAL;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10019020)(6009001)(2980300002)(438002)(13464003)(189002)(24454002)(199003)(10400500002)(26826002)(46406003)(24736003)(50466002)(86612001)(5005710100001)(50986999)(33646002)(46102003)(2501003)(10090500001)(54356999)(69596002)(19580405001)(108616004)(86362001)(10290500002)(19580395003)(2171001)(5007970100001)(106466001)(2201001)(6806005)(16796002)(64706001)(62966003)(47776003)(66066001)(87936001)(5001830100001)(86146001)(4001540100001)(5001960100002)(76176999)(107886002)(5001860100001)(5001770100001)(81156007)(97736004)(5001920100001)(77156002)(2950100001)(2900100001)(92566002)(97756001)(102836002)(189998001)(68736005)(23726002)(5003600100002)(5004730100002);DIR:OUT;SFP:1102;SCL:1;SRVR:DM2PR03MB303;H:064-smtp-out.microsoft.com;FPR:;SPF:Pass;PTR:ErrorRetry;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB303;2:bdeFmUSu7FvzrVdQ9HDMCbp2MXFtxa6XJYLAcxruJ/u7L12IsllvQTmTCcICoPIf2mg5+LQ8OJz6lr53h02V/BHaI057Cjdc/C8BYNcNGXEUKmxh/rtdwLcVncXx7Czaw1hP7LxsfdxLn5eWeJw7uQ6ngjFAVPxUVhYFdwk+Agw=;3:XEeJ+cJRnpL1v0oTQgucR8UT0f2psytognJTJNQJcjyQ+7QBUUrEt/koHKR8OSzICbfP3vXQFR92wF7IJRGLmiYF7sZhaVX2+PEajAfvwhPfk2W8vaf0oQtl4x/w+rbA7MLOOaX6RZe3aZjCJgc0Ws8WC7XrSKITG1eERTqyZd5/ZX2lUpPDcVV5CkPrvIjGHNtsepc28runOkgn7yf7pG2tSzIQLSaGM5cbbNP/UVtRwMN4HiME7nLrQGlmG2/SubLfpsJsl26XNMveGl1BwQ==;25:5oQ5RepwWsWjPSD9tOrtnJGuLY6bkSsbkn0Sz7d73utsPaGyR1+i7ariNiZ+S8b7QDIkbd3NEVChGBAnCElDby0xhr5yHxh/rCrPu9IKtwcrOgBmmfUBMEZgQnAXfjVnHxCntGR9p9SCLMUn7BLg20ZoDOlz5r+R+HuwSBIpWQue1Io0dUGEdjq4VQlma9jYqpwZYP/1/8ePmu+VJuJkQQmi3uEuziw4g3T6QZCF/R4V2zyoP9TAuxmydt+/4RlA X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501001);SRVR:DM2PR03MB303; X-O365EOP-Header: O365_EOP: AllowList from IP - set SCL to -1 X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB303;20:mCdJYH1NVNTdXYyfg7H69VzxGksQ2glYzz7E9kCdWU76gxcgxh2EJXjzwPnEyTJYrNa4ScOAXU/M3HhvCvJ20aOBB6ywyUsH5P1rSRGENPm84395UbEKRbE0Wi+mBEG6g1kN7JJmhcBg12ux5NCMx6qqcDhN8p25hxReFDZ7iGq5Ny9wMKtPgWzRsJZOcE0AC8R29qF1LvkfJ6P9Vdof/2ysvRV/bjL1EbZlfdNQ7EqLldh8RIxQviZpx+zDRuYAeRpsbJzhTfYE9SA1tgoChSywhw4tiXsPE2SFggzKBDEK5IQr675Kqonus2X0ioWWiXAlkA1vgZpkvTUz4hqJzNfdJaUIwrJ0pKkBbwolbiDyKaovSuHWjWpkA+ltD9gBkHeAZ5ppYtLlc1MGhPEQ9a2Fvj0IuMt955QkQwoIV64C+Z5mocE59GQ8WS8jywE/gpzKlfIoblHWj9gNq8mv2xm3VuoR7Ge+pkxNvdGdHLTOTdUq/RsjMBwZaIstfh+ytvwVdrOVdmJzUcqDJ2I/Q0lNw59iREXist/AnOVLDmzhoh6paSoT7B63R7wXRz5TA18RmZZKNsWSmKme86AlSksuIXDBza9kbAufiBXmRUo= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(108003899814671); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(61425024)(601004)(2401047)(520078)(5005006)(8121501046)(3002001)(61426024)(61427024);SRVR:DM2PR03MB303;BCL:0;PCL:0;RULEID:;SRVR:DM2PR03MB303; X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB303;4:vTDAl7rE5x6COc+SfTItlulm4Kk1jgfX5swDoUr+0foZ5bJd0BNXrMy5Vwuhu7hYQ3DK5w0/k3QmFt120j7WSCBztqVrjRUUoOPGKHCG5w2Azuql/DsV0pJVWe4WGdKMU01H+lZxoGy49aYXswG9s64j2fvCZb2aPW0xPi36IZ8ysMZv7YkF6BK3I0Mvc7/ERWH1zT4vIKcypmK29j2KcARpNNdRpSZWExFZ5SRkeUgrS9kGHd4+6NRvIy5XzKYbJrUWrQ78aBv4WV0Rwfo4fO+jL4aA35h3YQ3HN+YPOscCZwPGYXH+A2gnLSzV/LoAIvqrRxf2J90OQRlXdFkIadnHsLdSwPZvVkecjVxqzcszysGs0THGxWXsReACby9ZRBSXvOwezNZs1oGchA9rThlRsZyLpZFclHNM1jNaPZs= X-Forefront-PRVS: 070912876F X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;DM2PR03MB303;23:824l4NrAg5Z8mz5a9w94+N4GgxDG7mi/MrEjRuT6Zb?= =?us-ascii?Q?QV90K8AoYyQfJDgXmsYgcFVWi5j4KWNikYGIxRXfWsDrdwG0P0br7wEbZTmN?= =?us-ascii?Q?HVlnJPRlqvGS7kMVJhaK8d0bsM3zJLxTRUxARs2HmSZo5wnK/GHPMiuPWrwI?= =?us-ascii?Q?bC/jrB0staIjaoD2pTEVt349cccuhbiAD+OM0DzR6C1t2uAoydwfbARKeJk6?= =?us-ascii?Q?lUSbAdPOnV6eLXRqEqMpUj2BHIaLi4/Gv8Vh1qv1px7nIC9p37wzVpC/wo4I?= =?us-ascii?Q?jpdaI/T5Vo3yLG1vafFpSuTjQ+7vYBDydTcwX3A3DTbVuRYNwOE9X9G2GZQf?= =?us-ascii?Q?Sc5yXYQqvUn2+y5bMJw//jLZMGjKck9J+eIJZLhXiqOWIreenC8HOEqqGLHX?= =?us-ascii?Q?XAZA5c3jD+MiiTv9VdE2NaBYmR4SKFYUibunMXlFN0gHYQTZGjRQ2E0K4k2V?= =?us-ascii?Q?+f9oIy2MHk6QLnH11IYxe1AM0Pj+JNBiAC5ID8g4/18ICX44K0AEbEvor5lS?= =?us-ascii?Q?eWsmEFdOSoNkOU6YXqRqsRnX3PNMeP6ZuddnXx2UpV5BI2AeSVkRpRItMv4G?= =?us-ascii?Q?A2c5gEl0dkOfKomIVBRuaK1HGPv8RG/+FvchVgtjUkvANXkxuqp8rWwoXJCd?= =?us-ascii?Q?C+Ov9eU5/kFh6Uafe/A0Hjj3PuvXwBooKH1+zu/N7j5tjR/f/cKr6diGNQOC?= =?us-ascii?Q?jwXKz891WvqUZ0PNU2cEVATDA/TDDKpP16EztBLh5BSF1/FLi6hD03UTUeRE?= =?us-ascii?Q?2RghBlLqGK2z+OjRlMNdXyM5lHq/mCbkPYBpSFbgAcAVSVIACVzRUyvhFRAN?= =?us-ascii?Q?mSzpK4Bc0J3jZtCTWTNeGJ47hlR2UxcQhxSXC+KPCNzLX9qYMKh9zcJlupkc?= =?us-ascii?Q?VdfPnl3wnxXMDPCxzTEpiilAEFyAFvBdShN+5zWUsHZuYZU2FT7cgIZaab0t?= =?us-ascii?Q?XfBX4jBHEI1/ADHV3xJQCKQYjqo9eg0fdfJyZlcDUq9IbHFgnEHjZOH1/wux?= =?us-ascii?Q?NCSADAeacN30m7JEA01mJnPnK2YYSi2DD6un7ERW0ObYicSWSNvY/YgRwbF2?= =?us-ascii?Q?bqZAYULeR1dhC0Rz6b7Y+w6JKgoIXwDMbY7G2rQPBYFvVMeVrNptRyLQKbc4?= =?us-ascii?Q?mkm7Kc2CJgI0RUpJX5gf05c80h8krqE2kEcoaQNw5zu+7RAhO3/+IOPrAlad?= =?us-ascii?Q?+gS9J4bT6PAEXU2daD6Kd88JZm2xvSzMwK4ImTC1Or39vwNeHMSt42z0Ffrb?= =?us-ascii?Q?MR79EOmv0VKfEbpK3oFo3U8qxbKb7RN/8p+8tmW8J3VEi/q1iqQipMl8w3Em?= =?us-ascii?Q?lWojERJxbNLUm18UbKsT5SSvZoSJ+6/AQ7KSRB/SwS66hbAdCcyinXImyZKC?= =?us-ascii?Q?iTPI4+0ZwCGQ6IwtKpUFGdstj9TFLNKxXi8EVh6F6ySujOm93b6ha21QlBKI?= =?us-ascii?Q?vsVTkEvJGwYHh7SLKO/S5qUhNUDDbTV+hmuLsH1auavF4FRR7bUdua5Wo9rZ?= =?us-ascii?Q?5fPOOQEPKgZ7047OfSX5TyTNVzp/5uNJy6KV0VWcOH4YhBd65t+d/xNzCMG/?= =?us-ascii?Q?R6hcqvJulVW6g4PQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;DM2PR03MB303;5:UcC+MyV67kANGxyVvS+NTAZVeRMEPtNlgG2nXCuumXMdIrU9n6ZAtACFWU/+tR7M4yAoA5AhCescy9Whm7t1wK5mN4CQfwrn9I3LMn1NYPpVWkUEBquKoV2YguDFlBAXcBS7bz/+OZGwu/TnUoO+Sg==;24:bFqEvqtpA8QceUeaHhzxAwMyYLlcNAzrmzvjQqt9DIh11RcvMz4H4BCqCz/YvSaW3BbeFFmvAxqNsTkSJdD+bohCa1FMfYHmwi2Sy3ugKFM=;20:h/V4txk10gpB2fHgzkjBBRuVKAu6hkK16JfF9fmbV0NJ0YLOB/XzoyiEHqkn5CjGeUIvOX9dNXBd0ZXh3vvKtg== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Sep 2015 00:15:40.3827 (UTC) X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47;Ip=[206.191.229.116];Helo=[064-smtp-out.microsoft.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR03MB303 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org > -----Original Message----- > From: Tejun Heo [mailto:htejun@gmail.com] On Behalf Of Tejun Heo > Sent: Thursday, September 24, 2015 2:54 > To: Chris Mason ; Dexuan Cui ; > Theodore Ts'o ; Andreas Dilger ; > linux-ext4@vger.kernel.org; linux-fsdevel@vger.kernel.org; linux- > kernel@vger.kernel.org > Subject: Re: ext4: performance regression introduced by the cgroup writeback > support > > On Wed, Sep 23, 2015 at 12:13:59PM -0400, Chris Mason wrote: > > > The issue is: under high CPU and disk I/O pressure, *some* processes can > suffer from a very slow write speed (e.g., <1MB/s or even only 20KB/s), while > the normal write speed should be at least dozens of MB/s. > > So, I think I know what caused this regression. Separate wb domains > shouldn't have been enabled on traditional hierarchies. It doesn't > work there and leads to multiple wb domains competing on the same > blkcg and the bw estimation would go completely haywire. Will update > soon. > > Thanks. > > -- > tejun Thanks a lot for the quick fix, Tejun! I'll test the fix. I'll report back in case it can't fix the issue --I think this is unlikely. :-) -- Dexuan