From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754861AbdEHOJt (ORCPT ); Mon, 8 May 2017 10:09:49 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:32882 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751691AbdEHOJq (ORCPT ); Mon, 8 May 2017 10:09:46 -0400 Authentication-Results: lixom.net; dkim=none (message not signed) header.d=none;lixom.net; dmarc=none action=none header.from=fb.com; Subject: Re: FIO performance regression in 4.11 kernel vs. 4.10 kernel observed on ARM64 To: Arnd Bergmann , Will Deacon References: <50f528e8-8d8d-b168-784d-607a602ba808@broadcom.com> <20170508110754.GI8526@arm.com> CC: Scott Branden , "linux-arm-kernel@lists.infradead.org" , Mark Rutland , Russell King , Catalin Marinas , Linux Kernel Mailing List , bcm-kernel-feedback-list , Olof Johansson From: Jens Axboe Message-ID: Date: Mon, 8 May 2017 08:08:55 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [216.160.245.98] X-ClientProxiedBy: DM5PR12CA0023.namprd12.prod.outlook.com (10.172.32.161) To DM5PR15MB1194.namprd15.prod.outlook.com (10.173.209.12) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1891e229-0c19-4f78-c140-08d4961bc5f2 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:DM5PR15MB1194; X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1194;3:TbnJxxF1zBpW57AEp0N3L+HssQOBARnm1vGs0rZ55vwTKgVDO6h/xjfSH5U4NiCxoeWXMVIuk0A/BovNqBwiEbmEus3aaIuwgwhcpjaAOIDGupMfOFk69rLZUMAhvnFbAPRBzk+yjDMOZQFeXi102xHdqcS2e1Ik92/XAaFAPxtn82vWMsH7/ZeE2VXiihHxVAAG7e0dk0UKgYgGcvyKFSgxVmxlulrqkTXG514LATxb9NVKOGfOFmCyBobBYJ54z6+8dFM4C0TPQaqSrz7n3M31jNFhK3sLyrJ8OcQWqf1+8ZwamdX1VuR6XB7Ne/BUV1JFSkUo1oKZcNn1JR5cqQ==;25:n+qxbtFdZOqqFCvdEz6HaD6DNub3S17xlz3NXzwVcgLVy7d8vQnabsuZt8JDhy+uuHeRYBGZE6fV6559iwv57OYbxp5rLm/uekC7jCWeWSH/hKnjQbm700jnijV3Q3NC3Y4dowShaiLCnVCr8mbq7kmM6VmxAYB3pXXIXi/xNUdEccB/E8BKUAVov4V0rRwq2peL+W7q0XYByVms3JYFBxruc9v9oTulyyET6qYdyipymPuxJQ17YBONcJQtKRRqpfjg5nAFUf8O0LMWlF4rGs9zh/DaI807m9QFCIHoqZBxpKHI1tm/EyG9GkSV8c6OVnq4ltw5TV5CWKkuV1J/Tgqz0QBXcIFeH6zduDAblRGXJm97tqabPT2fQ6ucCrb3gk71ZNaiLC1XKFJrJdsYp3ucvyCiMW7UJRj0H0wXIFf4UKwHqZfDLj0aJo3/ABflxEe9WagrT6/efTD/cX2SIKxdvq7Ij2QiHkrTB1kCoAo= X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1194;31:1sQLCUB43Mb5YQlgt9uhazg6dNf437CJnony4cG/COuYDoBTIM6vzLnOATI2OXOhlPAixqhAXa7SqxTI+sxytGDHxopVCnEd+66Ceqo5XtdT+jYvEidL972QUzhJBO4mcsCn8EVMZ6s/ATH4qjfmZOrhJNNGVYxYLrO25NrMqStdCZbKW6jXzxjVHthN9nH3i/b1rNrx97GxS1zP9aKQiVPmyeXJHJqQ2ymPV+fA8x+jNF7W3EzTWAv/Jb9VUKG9BpxPa0TTwem0431JjuIVtg==;20:Bd0zXdBiZ6rOoblt+ONFjy2y0hrQ7RndqpHKX5YWbkIKpFGpRc07UxWLkRzTbauoSxfgkKDj4jU8MwyqB4S6fk6QZwII473LjYr2bBCx+T+WvjMSQI+GiLLqmyUqlVYwRBWTQv9ebZE6qBJbahJ4qac3/j/b/I/aDJw2JdBNvQ1B0yZyLe29FyCTY4KY/eaVZRJHi30g2u+CLWGCw4B1QPeehk/6LENQuD2xeRPNihhUAItdPp75EqgtkGDt1wOUuJ9s5gA0P57PoblhxmB7vE28xPQPprIcmaxlC3P4ym9DY2mdLvDIV/iEhJL9G2dzchpaNmIrzrVY7K6dRaHSG5q6pA+hDxFlRbHUMTgsxqy4CD/PBFQPhAxO7GTHXMU6qCuglHIZ37WR4RVD7CyVZru+Eaz8/neDzNGqHSWX0gG0OL0N0FsZkhgRlP3xgbaXjwxnTPKTE1e+7Bglq7XNTFvfZ8H5IfuBoRVgXtpEj5ATewWKurxkatWcWnIDhm1X X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(180628864354917); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(93001095)(6041248)(20161123558100)(20161123560025)(20161123555025)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148);SRVR:DM5PR15MB1194;BCL:0;PCL:0;RULEID:;SRVR:DM5PR15MB1194; X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1194;4:K1CSRe6zqyA/YIKrGtA0253wRgWCRLVr+CD0iTzFvPzWVkf0TjSJqu3pWg92QMlpYFsLsTVnWMyx+M5nt1mWMcWFBFlZZiLI7TRI4kNtPCWKDuBbSEC/oMTETRq4uP31oG9EfBPti/hdZKJpsFDV6N6vIs0NRAkIhAlIhrAFzlE3FX4k4aFK7cnYjejlpx+mEhtuh3r27nyCyMySk98JvENnZrtqzF79GO/T1pnS8lD+/mzjoUN5oyRzas+jIHDquXB3Od89vLQXKqW3eHXSM/CT2JL410nEBR5NOfXrZZlo9QQ4zXnYFdALO5+JANjvhYA+ubRNc89VLpfBPq2i5+5PGVPuuQPhgzjwRV/6+9IjfhPMglMt+dvfGVBPEs9bxOYr+UgmvQ2HwxijmBUNaPEGOirzN6qcqboyBeS25DT82ebSVa+E9U0tgDhzap50U2qpmei6Fd+rvl21Ts8oWfsWPPfyHO5KANPGmg68Cv3OS2NQov0O5LQyAAvTM1/mXevWRiKroSo1tDe5EACvja8D4BWtlqJ1etNvojHILZKZETL36e7OkfvkYfOcqYvWpdzYAdbeIlyr4IlO/v3kSqE+LSg9NumYy16Mk1k4RB/yPfCAt8yukgJxRaChkcycH6yQzhgS/+dgvgH8/uoGggZkO5LOcwv38Lz4suYxN5pXm54XAo1JgYhvki23REmhpFtQm/1jUGE0BGLa8IDY1cDFPHryDOMj6Cavlq7YVAKlOrH6MsUV7UkA2i6WL9t8gRSQKgo1OjBYcBUquAGu9CaJixzR8EeHTdD2my8FxHpVPTyQiUDpnDk7fQMkOXONV70oHZH6CI8RvHBfU95yYQ== X-Forefront-PRVS: 0301360BF5 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(39850400002)(39450400003)(39840400002)(39410400002)(39400400002)(377454003)(24454002)(51914003)(65956001)(3846002)(6116002)(2906002)(77096006)(47776003)(66066001)(305945005)(7416002)(7736002)(36756003)(64126003)(25786009)(53546009)(31696002)(86362001)(65826007)(50986999)(5660300001)(65806001)(478600001)(117156002)(230700001)(76176999)(5890100001)(83506001)(50466002)(4326008)(54356999)(6666003)(2950100002)(6486002)(53936002)(90366009)(189998001)(4001350100001)(229853002)(38730400002)(6246003)(23676002)(42186005)(33646002)(81166006)(8676002);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR15MB1194;H:[192.168.1.154];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjE1TUIxMTk0OzIzOlIxMkJQdW5oQzNpelJhVVA5NmpiMlBsdDVF?= =?utf-8?B?czRuNjRqU1A5UTBpTW1seG95ekxValIyZ1Q0MWVTelRJb3cwaC9wRnpwZUVq?= =?utf-8?B?QXVyY1h6dEVWcWgveXREWE9BRkRpaGtjOGNiamt3SGRrN3FySEZaZG9WL1Nu?= =?utf-8?B?RDN6dGZrTmt4QlJuSHlmWUlTZWJmMXZDR0NMVUlROTNqSURyUTdSaFRFK0Z2?= =?utf-8?B?bTZ2R05GWWNHMTBZdmhjQk5FTExFYXBnQzhQSkFMbm1GUzVxaVh1YjBmUi8r?= =?utf-8?B?SXpUeVJXa1V3R2pudU1Sd0JIS0JNUnlVV0pabTFnRTFqUHNmeUEveHZzVk1F?= =?utf-8?B?c29vVTVyUGZiQi9Sd1B6aWd4MUV4M0RKQVNTRlVjRlhnejRNTHo2RW1JSVFO?= =?utf-8?B?T2hiWTAyKzQ1dFVFZGFnM3krY1pRYWdnQ0N6OFMxKzROLzYycVR3ekZDaGRE?= =?utf-8?B?NTBkeDJDMHl0bCtUMHY1Q0JUdk9EOXhBU2lrQ3RPMUdqTzVoSzdWWUZ5bUQr?= =?utf-8?B?WEFhdERXcGdKTERqZzJKK0JUanZvV3FDL0ZhcVpnN2RlVG8vN3VMbWRFUDZ1?= =?utf-8?B?TGdNZ2lOcjNVUHBjWHcyUUZmdVcxQ2JJQmdrL2pvWHlaL3BjQXJKc1Mrb0li?= =?utf-8?B?WElwNFV2eDlUR2NuUmZXWHJ5clVZMVhBOGhkTU9ZUkhCNlBjNTRXY0E4Y3JT?= =?utf-8?B?MTZlUjRlb1Z5SnFuWkJzV1I1cDNuRmw5U2wwMjVYNURYckc3VmNxVmIyTmwz?= =?utf-8?B?d292MWJSNVZYYVViTVBqMjlrL2pXcVh0S2ZkU2FRZGVWSkFmcWt3V0R6MW9i?= =?utf-8?B?RGtCNFhzaCtnZWdoVVpPTDhzRzNsUmlSalJBRjRmL1QxODNKbGlrcHBnMEhy?= =?utf-8?B?VUowekY1WHgzenNyRHVZaC9JdFE5M3cxOXh4Y2s4TC9nMTZtdmNyTEhWZ0gx?= =?utf-8?B?QllaMzVqbFNYZ1FOUGVjTzUybklIMkdLVjVyNGcwemd1bUY4KzBBVE9tZ0l5?= =?utf-8?B?VVJwR0ZJRnd2R09Hc0dEaUd4M1hiampxM0F6VnJQKzdwWTBFdDVQNk1aUnFL?= =?utf-8?B?MkF4YUxJb2dwT2VNQjdCVFcyTS9jMjB4QnRJYisrTVVIa3FtY3VBVW5MbTRP?= =?utf-8?B?bmtnaENEeklOM1dIVjRKT0VrZGNMTGN3bTllcmg2MDAyY1c2b3lyQ0RxS25C?= =?utf-8?B?VU9Yam1mLzhvZHBpTDl2N0ZDVW9PaERFQ0c5UW9tU3oyMSt0cmY5WkZRZmsz?= =?utf-8?B?aDdIUm9sdFc5b0tzK3VVemRQeTM1S3hoMlBydGxOcGdzSGJwclBZeGhLWTcx?= =?utf-8?B?d09STUpKWTZJb2FsbzZ1aGUwVUluTUpzMmltTmwzc2FnZFdqOXRIK2x2NFZr?= =?utf-8?B?S25VM3JXRmF5S014WElLOGZrTVBEcGwzSUZpbkxxQ1RPZEFWV0U1UzNycXFt?= =?utf-8?B?Skt1bjY5QVIybFVtb1J4RTByQnBDbVNJSVJoWmFXYnhvMlMwcGM4cUV0U3hS?= =?utf-8?B?RDRkbGFJNVRvN3Y2NGcwV3Zrc1prK3I5d1p1RmwvemlqeFYvYW1nY3VIRVo3?= =?utf-8?B?VmUrZUZBdDlMZll0cU96M00zZ2pjN2xRSit3VWpndlY0cjZuNWcrUi84Skxv?= =?utf-8?B?TjEzV3NzRWdvMXJNeUF1TERTMWpkSEhWSnV0clIwK25rMlM3U1BIQjNReGJi?= =?utf-8?B?UWU5TkpGT0liajBHRko2WU5BZVUwVjFkV2hvSEs2OFQ4aWxJczFSYXZxR1hK?= =?utf-8?B?amxGTFpIR0VIcG56SzJNRjhFcitJNEVZRXFlRUZBUDlCZHRwSjJva3RqUkVt?= =?utf-8?B?WC9iRXU1MzJBMEtvRXFCbTU5NGN4d040MzAwTUh4NTJjU2c9PQ==?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1194;6:hUOg+TuSF4e/a1pCZA5SyxNZ2kV6eJCIqQLjfhdKfJn3OX9uSwwWo8xjmWD6a5VEUATDyPd3kb2SN+pSME/ZzCoBt29QtBLWGvTK2ELyvJzq9UR2gMfOnTJY9DnGSHXgWLBMHlVOGW7kwscP4lGBuzXqQ2+xfgI24cTU9SlUSNYQefpiynkTeEBHcRtREoMLLjNNeXzcizK+digoO5vuQUsQ6QOA9hZGhNivO0uf+/RN4bsFcMrPLOBwi0f0fWsT2uuJSJE8jFWvk0PXVdEgAm77nMKVAhXM3VVOzTKjFt/v0FQmkHbqxiOYmy7Ui71oyuUIIcQA5jTQGrOPc27fT6AYiaZCYN6MomHqlN2faaWziK9s9OP7cVm+FI3rvXweRXy9ORxfq+bI6aOsBiv0+4lqADHmAmABl7Y/GyUlvMkUygLAl5GUXg1V/nqvEAoX7/Roy0O89Jd8Qfa3VpUylMaeF8oEfR9yk01Dj2DWll7WLWRU8+nz/prF94LpK68FyCouT3TxXADgeK6NuNogNw==;5:8tZjUXTAYvHHN8UC9CRU2A+CaCPvuc5rWWjll1q9vtpSEnF846GdXoO4JpTzJ2b9DiQIGrMnn8tMdRMPZXrdTxrd3/MZ7glWccZUUEZNGjtXJxvtEEfrzXSU72G9nqu7mCF3IHn9ihxgRHcVLGB/DAYGc4UhA6nWg3YdOtH8M2c=;24:cXDPqj4ZrGIdmNvLW3xlfi8InRcP3ngONVJr3mh7+pmN8fl3KWTpQvskZd6z6VGmy+Uu9/rowW6Y9mRkIbgTM8SZhvR1LNpBh12gYJdDOeo= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR15MB1194;7:v3oD7+b5PYx5LmvAm8iQjBHpwMh8QFDt9p/QUySjMRw2+u3iZLMJ/fLDuuU3idFRXUrPB8VquTUxXt0RqtUbz1K9uXZGfvSKbvY8YTXeNCqYbm5uQivMc//xCBZ0ISUOI3uckI6tSecYj8wc8eOAKWyUKY1MwBj2T72C8woriU4P47tqjtTF0KtEIL9895otabLerwcwUtrlTnKBjPnXsUAK5k0qOoFIQ8qr9/U7i7yBP3XYtLwjJ0ltHame1Ee/AbCe4HKl8D49JcR1Dx5V2hKfTBsuPjVmZDmBAiwkAOYp5rdsOfkllwvmiKdHdotfrnV6JmX1ayqMsk+HyPWXHg==;20:ESKjdHm9NGsnIRUCu86l/0PqrqV1d+sQyGRtEnZSQKH2ngDTbEsbi6EbXLXv5Y9ajRXZ/U2lYU2WpSqG0UIqVzD+yjW4O3Vd+xgKaCsjRF+MyMasmIycSRcu7v9/qAbC/UPtNX6Cp1736d0H2Bg14Ns0UiiTT9VH9nRbgOzMlfY= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2017 14:08:58.9341 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1194 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-08_10:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/08/2017 05:19 AM, Arnd Bergmann wrote: > On Mon, May 8, 2017 at 1:07 PM, Will Deacon wrote: >> Hi Scott, >> >> Thanks for the report. >> >> On Fri, May 05, 2017 at 06:37:55PM -0700, Scott Branden wrote: >>> I have updated the kernel to 4.11 and see significant performance >>> drops using fio-2.9. >>> >>> Using FIO the performanced drops from 281 KIOPS to 207 KIOPS using >>> single core and task. >>> Percent performance drop becomes even worse if multi-cores and multi- >>> threads are used. >>> >>> Platform is ARM64 based A72. Can somebody reproduce the results or >>> know what may have changed to make such a dramatic change? >>> >>> FIO command and resulting log output below using null_blk to remove >>> as many hardware specific driver dependencies as possible. >>> >>> modprobe null_blk queue_mode=2 irqmode=0 completion_nsec=0 >>> submit_queues=1 bs=4096 >>> >>> taskset 0x1 fio --randrepeat=1 --ioengine=libaio --direct=1 --numjobs=1 >>> --gtod_reduce=1 --name=readtest --filename=/dev/nullb0 --bs=4k >>> --iodepth=128 --time_based --runtime=15 --readwrite=read >> >> I can confirm that I also see a ~20% drop in results from 4.10 to 4.11 on >> my AMD Seattle board w/ defconfig, but I can't see anything obvious in the >> log. >> >> Things you could try: >> >> 1. Try disabling CONFIG_NUMA in the 4.11 kernel (this was enabled in >> defconfig between the releases). >> >> 2. Try to reproduce on an x86 box >> >> 3. Have a go at bisecting the issue, so we can revert the offender if >> necessary. > > One more thing to try early: As 4.11 gained support for blk-mq I/O > schedulers compared to 4.10, null_blk will now also need some extra > cycles for each I/O request. Try loading the driver with "queue_mode=0" > or "queue_mode=1" instead of "queue_mode=2". Since you have 1 submit queues set, you are being loaded with deadline attached. To compare 4.10 and 4.11, with queue_mode=2 and submit_queues=1, after loading null_blk in 4.11, do: # echo none > /sys/block/nullb0/queue/scheduler and re-test. -- Jens Axboe