From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755519AbdEHP3b (ORCPT ); Mon, 8 May 2017 11:29:31 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:54639 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750985AbdEHP3N (ORCPT ); Mon, 8 May 2017 11:29:13 -0400 Authentication-Results: lixom.net; dkim=none (message not signed) header.d=none;lixom.net; dmarc=none action=none header.from=fb.com; Subject: Re: FIO performance regression in 4.11 kernel vs. 4.10 kernel observed on ARM64 To: Will Deacon References: <50f528e8-8d8d-b168-784d-607a602ba808@broadcom.com> <20170508110754.GI8526@arm.com> <20170508152400.GB7131@arm.com> CC: Arnd Bergmann , Scott Branden , "linux-arm-kernel@lists.infradead.org" , Mark Rutland , Russell King , Catalin Marinas , Linux Kernel Mailing List , bcm-kernel-feedback-list , Olof Johansson From: Jens Axboe Message-ID: <49bb6a78-e57f-b4ad-a593-0dc9d24074f1@fb.com> Date: Mon, 8 May 2017 09:28:07 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170508152400.GB7131@arm.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [216.160.245.98] X-ClientProxiedBy: DM5PR21CA0014.namprd21.prod.outlook.com (10.173.176.152) To BN6PR15MB1186.namprd15.prod.outlook.com (10.172.205.140) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2ed6c041-a7ee-4058-1dce-08d49626d6a5 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:BN6PR15MB1186; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;3:z7x6Ek3pGLwJsjVohQoqbuxkjl5952IWuTWbF2aFLilFfosRFrFRgAoJhIJYTj94EUjPh//3SS3sUaXRpzNMn94+X7EMd9Ni0dKQDZGYlKG4WL3QL1pXZ4l8rSYJ14SPDR1hTCv8Dq09USCgQ423zMqKnTArOonW0t2Tkvc0xvMqGIRloJ5TVY1Dxl/ZKW+VwTFaf7m66rpHx+KZm30S1VhG5vSOBrSvgxigHATDpey709zTmJ4GLXOyvyHd53Bqc6BXD/D63Ccwfj4CF2srrKMb8OqVW70NAdh32o6JHEpYR3MQnH1diXVJAYXjNV+pR31CBbzrHRqdMToQxfe9yA==;25:dgiXhTIGO983zXvvYp7Ikqry15RRPu+uJBEyQVBdEOdcdueiNDrQhJ4zmRCiqDSMCOpuHBW3jLmtDRybVz6ZWgsmwsdhQQL18cbYPf28ZF9HWgpTYjY81UpgbQYKwIIc6ipRL60Paz3gopbxGtnH5G0NEZS1nbuYNiqgVXa3p0wR0hpNRsJxQh4Ch8R+dFuOWh9IDXU0SV8RPxMAk5EuHmXSdyaJh3rM4OlRdEeM0e3ZPN8/fG1nF4EMfPZmFaQ4eIrqiNOLbkD0IedGkofaJgbwhRVE4Ox+RRZQ8duOpDn/aL+2nt084VMNzoCfOVD1WT/5yqhThkWqqa7dsPJBaoVVykwDUVDBXQh9LB/cQv3KXakqOUYG0mHZM2hGbMkfNuPA97ugiOU0w8TOYRZN2RKQ/dvxYWW/vXPfNSe6TBZUm/hJXD+xeUQOGsPlVTqsnqzreyzU88OGJN/wvEOR4ZJNt4dXJScmOV6CUG06MaQ= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;31:U8H+8r+mpVUGdWITmxIJkbjWfC8wqOW94FPYgmRmmcMMLFCf4ls/9XeZkqy05ruPDqNuX3Ldf1++P2tOTJ4+c61+BTplLQihxICivJr5BQqphB0nwtSusXVCQUvqBwSgf+JE53zdzuAUDg3CsMTnxQn76q9hfN79NE7PP6OILCO61W+L7G37c7JpAALEYccrdb+bZ98/NoKZfgkOPf1kkZ4RVReAzph0FngleV/NtR0IpRnn8kcS77D+6z0WXrzxTTC670qMF9d5Vu+7p9H0yLD4NXboxE7S2Y5uSNK6klA=;20:PUkh0I+ag8GyLySIioStu8gvMacC/8BnLDkY/lYSOSU65m7U2FuaJp9vqxoUEUPjuRHFHnBu3sLJoCLPVL/jiXqdFxyTQOHeW9E99LhGx2XeLHRlu0u0RmNJ6Ghgxe0VEbUQ5sVM+KXRyIAQ0MPJS/efOBQMWGL3XDDGmJ7UsH/yXnd2bG6KVf38DEmRSUg5G9C0LPsEjcd6kpZo2k7f2GjPC3WTqyj+2WgLEsWlO19qHHaEX0P/tzsTRuNiWy5pW3m9+M31+e3vLnQ0qwvQscjywDhrznpEwGR5+vdRECVBkHzOS9KCEvreNsNtRanEbLw5+KSzr1eMQPYuurFP0JPX7k6IUwcjjDInneZ9WOl4LLlcEPIufMYi98x39lpC5IS2KLJhkNGScTTGfX8poJskDfqgzvEWeDl8SpzZ2DWEGSEHgZLXugXtVrOBeSwdukIQZclkGp+oJw2VFs7rHU8dMA78O476OE1tpVI//0wO87007tSQCwJGABTY1WNk X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(180628864354917); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123564025)(20161123560025)(20161123555025)(20161123558100)(6072148);SRVR:BN6PR15MB1186;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1186; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;4:dXjGYaP6EKcaNVIzAHAtQircsQMmWbLP5Y/Myg9uhOgxVTwMkqMWKKkYVhwMnvPN7+LVqB5R0SkLczT6Yfd7Ure7cEWbP07bN6TdinPHMIryk+nnczfU0F43ZTmZ3raJ1cPbskoGzygcAxBaIkXCJJgOSYeVwsgajCQljjBLZQ4VSY6OoSUS7XHmmjK0F1tt6bB30ynbQt1sjlJ+5E6qcgKgfliMEGxOW+4LmDcV2VGV6TVUs97t0rgWwpovEaX/ETH3NbRbP0c3bUyHEySR2n7rq5uLjeedRxYdgc2PL4iVtMBgLK6Oio6/Fgf8TloCNqud3p7ZXNpLxz0Z5vyflN+n4KcBdayPByLlF9qfHfAlc4IjC9S1i0DTyvggtBomiBuMBSF8XA7GhcweSfbSdqfd1QWwdgLA1AGOjG4QzYkLUqkhUTrDzBJNlWOj/YfVgnkKYmODvcvvkflSPjR7oxDaEw8qaHRGUz+pcVm3HtM0EBN7/rJZ3p4JwEy64o69eGf87KF3aYYq+1sP86sNDi1MvziGY2hU5tqwvpykw/6pqgsRXkrpcEVwUSw9zLYL2JOqa+JmOkTc6S6pQkqFvD8DdD5zmVIzke7m1LeB31O4eTv1CTKoXWX8YweJB8xXHb0U7+OCiL3oTjuKUPcT1lSzK8T6ePaAP5DtLw9wEugALGEFj+jLraCrnpluVAufX693nke8XcjnI9FVf05Xuk2NvFJotHf6ew9nz5dhGi9B+Ufm4Qt/vJfcYowS+wSZpSSqY2OVNBSRNbfTY/u8VwoM4EkeJrXAM/tA7HrFjc9aF4OL2McTKHFALE9+VWjmXlbgWPl7Z1xSJviEbOOoVQ== X-Forefront-PRVS: 0301360BF5 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(39410400002)(39850400002)(39840400002)(39450400003)(377454003)(24454002)(4326008)(83506001)(117156002)(31686004)(8676002)(53546009)(36756003)(38730400002)(4001350100001)(81166006)(77096006)(110136004)(7416002)(5890100001)(25786009)(50466002)(478600001)(50986999)(54356999)(76176999)(189998001)(53936002)(229853002)(23676002)(86362001)(90366009)(6486002)(6666003)(31696002)(93886004)(5660300001)(6916009)(64126003)(2950100002)(2906002)(33646002)(47776003)(230700001)(6116002)(3846002)(54906002)(65956001)(305945005)(42186005)(7736002);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR15MB1186;H:[192.168.1.154];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjE1TUIxMTg2OzIzOmhhYk4xRmhubUszUDVSU1BMNDJTcGdHMXhR?= =?utf-8?B?bGNTTXYwaHhRcVpPWnNLYk9ISG9OWk1tbm44cVZFQW41YTM2NGRpUGtNSlg4?= =?utf-8?B?RjhYWm1FOTc5d1lUeG5wUDdEKzhQS2VwTlBEbDBIS0w0ZU5iN3QwZm0zcytN?= =?utf-8?B?ZVJyeGc3dWJGd25PQllWRm1Va3FQRGxybUZsc3pDK2g4L0pvZ24vSWpqUVFN?= =?utf-8?B?VlhWY1FUUVBVV3IwWnFidnJFRmRJWnRYTi9LdjE4SHU4Ty9BaE9HRHZkaWhE?= =?utf-8?B?anJUYU9JRGp6YncvSFpTVEt4eE0wNk9VMENxQWZqUVlMaEhjMXhXcWdPeUdw?= =?utf-8?B?dmxzTU1JL29tbFNmMFVjbnhkaXRYbXZqK3QwSzROR0x1NmVDWWRHN3VlMTY3?= =?utf-8?B?YkxZckdYSTdyb24xT0kraCtVbUNJQjlvSmdYMUhnT1BNWlRXS01DRGd3MEJv?= =?utf-8?B?bjFPeFlNWTJPY0dTcy9QODVRaWs3UWR5c3VUa1N0RVczdG9kaWtVMExFd0VH?= =?utf-8?B?cGxXa3pTaTRQeGwzdExrcU8vNlFzeDhuK21pMVh3WXA0b3l5SFNza2Z4SHI2?= =?utf-8?B?TUJoa2pYdlhmVjFHejlmNWVXOGZBclZ5Vm9SdjZSbDRYWGFJVEtNYlhqWGd5?= =?utf-8?B?dzBsdVJXWEhMU3k1YlNaMHdwWmliUEozNVJRNys2WGRvQnBMYWlzUEUweWxt?= =?utf-8?B?cEVZdjA3OGdkcE1pR3BuNElzeFVyemV1WFZOZmE0MkVmVGduYWFrM2F5Q3hI?= =?utf-8?B?aFZPT2YwR1RzTXZwTmpFTjVIRGYxczN5cmhaeGpLMk9WSDdPNm9OaG03TkNy?= =?utf-8?B?ZndUakg0QUN6VlAwSk5ZWnVZRStSWXFUM0hlelVaMVM3L05wMFIybUxlSU5W?= =?utf-8?B?M0MwSUY4L2gzTUlOdEh2MU13WXFLSzh4Qi8rY3JrQm1CRERiT29vYTNBV1hU?= =?utf-8?B?dU54dGNvREZKckhxOVJNZ0dsS0NFMHc0VXdCL0RLS1BlQ3pUeDZPaE0vTXda?= =?utf-8?B?N1VSdGNyc0FVaUl6ZE5sck14Qm51VmV5SUJRN2ErakNRVzRPSDlLR2NyTUFO?= =?utf-8?B?N3RvOTRRbmVEYVJtSGs3MndRTG90bnlRb0FKVHNlTDF3YTVvemluaFdaYUVB?= =?utf-8?B?N29sSmZoNDU0QjBKeWZyRE16WEwycHA4UDY1MjRHUStNZWRUR1lXb3cvYkFE?= =?utf-8?B?cnJzU3QyUUYrUC9pNEwrVkdzUmdPOUxyOGNhVnprS0pjOHhnbkZ6NmozU2lN?= =?utf-8?B?ZXNKMWg0ajYvWFdjcmtNWW02NGlDL1o2YVJ2ZmhGOWVjMXZDbWxLVmFnRmtH?= =?utf-8?B?S1RxVDN3dDFicWd5MGZ3eUJEdFJvZTVLWWhjejg4SGZUdFNEQ3lsdXk3Q1Rr?= =?utf-8?B?bTZLWUdOdnRyamhKRWQxYmxoU0NtMUxjaUx6SWlOTTZML2FWWkk2cVZncXlF?= =?utf-8?B?VzljWmlHUWFpbGgrdW5kUnhTbXZPSWpRd2Z6ZndZckIrRjRwOWtQbkdJOVZa?= =?utf-8?B?bmhTV1JQbzlPOUN0dk4yOWRUMnMzT1V6US9mWTJsNjFJbDE2SGJPNEtYbHJE?= =?utf-8?B?ZkhQd3g2MVBYK1piWVBlWkJZWDVkYUs2dnA3dTRnaGhuZmpaNitKbmFpeTMz?= =?utf-8?B?YnM0M1hTQnVkWUphU3VqekRPN0hYYjlmVTNIMHE2ekl0ZEZ0WkN0UnRBWlRS?= =?utf-8?B?NFRwK2JpMmVMcDZRVzl6UWtKNGthMXcxaWJOWkYrOGhkc1JvSE1aRWFkVlJW?= =?utf-8?B?QnN3SVpsRXRlaEEyVnlRd2RNenRjbnlZQ0J5c2Vja1A4QUpUN1djaTlrVnF3?= =?utf-8?Q?zN7KdBoF+kq5U?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;6:hdfSTs9qlT6VCeA4KiYEX+36eGue3Ne90jzQ/QHXlpAmby/0XqLpHQriq+ktpBbVDLF7cWo/thIcCew9NSFNxB46YfWpocxywojMTC3OKU5/WW0xSVYlYVng9HjbDE3MwexPkGGts7oe0XHuiEdkqzWPG2RIi6aZqrJaldFJuRHioE3uFqN7wLpdA7TOJ6i5525toXwlE2zs2sbosbxwThGQ/ecr4HljNwV8r143fUr5iCYwA8bpEkS0ZvZftafZKweIR2CXs7rSvo58d6WKcmqhzxpnuFIqjHZZZZCKicInFH09zhgPAkZb0yF/tpVq+5ugjf9p4vq78nzT0r99J8tWk3C/kphzVwzcaYGnWblRHPgqsH770ANjJi3DZ/Z1WT0OA6+D143yTegS3W4nKIyXcisO5j/Gd1NBdN1Z7+hE+Tbm8aOArW+zHAAAZHIx9vb0U2mX2WZiHhXTb8cffoPMd2/qOv5nXLRcXfi6efIv79/U+ouhsMVWbgGvRrJ0eWs9oj6tgn9wzFZuPBIblw==;5:B5R4WZLdkJAJBEl9MIItqZiRxLYw2Q9K++aunCb8Dbj68TyXUdYBoUMpf0rsuEh2Hlv30OeMFXlkZZ/RdztSE67c+C/IZE8HieRKjePNGzZ5A74aIVW8hlbAROrswpK75hmhYHW+cnyrVL2FPvzEWg==;24:UMzm+3OdLLaxblIwDT6Gora+tmXwsrhSfRbRdig4Afpd+E0WorowKrHCwegbn+yT4Djr4CEaaTWBZF++uikveTnRdIS/iEveYSFRdNcoQgU= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;7:c5IphB0sdXQ+mjPEtRegoWGmRrSKo33PIOYVMEDcnt1BVmxPeQ7Lz2YPcI79n1bj72kQFtDmnAdv2pmINRpSXGinq295jR3ckGG9lYXMXh+dlUmUaaaXdO2bcl1YLOGVpYkX4isFYHp58cFOsizxCnvaJJhYgDhhrZn8vMcbEQOFCCShpykI8U5WLH9Jb78xOlkdx5Zwo50FiBOSFM4nuUS48fzfVAhrLijL04QN2CcTYS7FPzZOKMX6d1sF6UGvxnv73VroFLHSBmZBHd4SFclXPZk79mK56YFks5uLrpVPv8Pv8xCyZWy8hbrMZ5XLIk84a4314sh1BciMSiJRQQ==;20:UVHrSJdR1Ukt+JEAUqC1V7b56viVPLkMuX7+g65vVpLDlYegYux79nkaEfgIsyGTFC1ejv2uyqh9lCb1EscFcIoWQjJFAxPndEFJ9PATEbUQJ2Y6oc0ZnoboQDvLsd7TGM771LM/EkeW/2HbREkFkJvGBBIobODCymuxTehORxI= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2017 15:28:11.0043 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1186 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-08_10:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/08/2017 09:24 AM, Will Deacon wrote: > On Mon, May 08, 2017 at 08:08:55AM -0600, Jens Axboe wrote: >> On 05/08/2017 05:19 AM, Arnd Bergmann wrote: >>> On Mon, May 8, 2017 at 1:07 PM, Will Deacon wrote: >>>> On Fri, May 05, 2017 at 06:37:55PM -0700, Scott Branden wrote: >>>>> I have updated the kernel to 4.11 and see significant performance >>>>> drops using fio-2.9. >>>>> >>>>> Using FIO the performanced drops from 281 KIOPS to 207 KIOPS using >>>>> single core and task. >>>>> Percent performance drop becomes even worse if multi-cores and multi- >>>>> threads are used. >>>>> >>>>> Platform is ARM64 based A72. Can somebody reproduce the results or >>>>> know what may have changed to make such a dramatic change? >>>>> >>>>> FIO command and resulting log output below using null_blk to remove >>>>> as many hardware specific driver dependencies as possible. >>>>> >>>>> modprobe null_blk queue_mode=2 irqmode=0 completion_nsec=0 >>>>> submit_queues=1 bs=4096 >>>>> >>>>> taskset 0x1 fio --randrepeat=1 --ioengine=libaio --direct=1 --numjobs=1 >>>>> --gtod_reduce=1 --name=readtest --filename=/dev/nullb0 --bs=4k >>>>> --iodepth=128 --time_based --runtime=15 --readwrite=read >>>> >>>> I can confirm that I also see a ~20% drop in results from 4.10 to 4.11 on >>>> my AMD Seattle board w/ defconfig, but I can't see anything obvious in the >>>> log. >>>> >>>> Things you could try: >>>> >>>> 1. Try disabling CONFIG_NUMA in the 4.11 kernel (this was enabled in >>>> defconfig between the releases). >>>> >>>> 2. Try to reproduce on an x86 box >>>> >>>> 3. Have a go at bisecting the issue, so we can revert the offender if >>>> necessary. >>> >>> One more thing to try early: As 4.11 gained support for blk-mq I/O >>> schedulers compared to 4.10, null_blk will now also need some extra >>> cycles for each I/O request. Try loading the driver with "queue_mode=0" >>> or "queue_mode=1" instead of "queue_mode=2". >> >> Since you have 1 submit queues set, you are being loaded with deadline >> attached. To compare 4.10 and 4.11, with queue_mode=2 and submit_queues=1, >> after loading null_blk in 4.11, do: >> >> # echo none > /sys/block/nullb0/queue/scheduler >> >> and re-test. > > On my setup, doing this restored a bunch of the performance, but the numbers > are still ~5% worse than 4.10 (as opposed to ~20% worse with mq-deadline). > Disabling NUMA as well cuts this down to ~2%. So we're down to 2%. How stable are these numbers? With mq-deadline attached, I'm not surprised there's a drop for a null_blk type of test. Maybe a perf profile comparison between the two kernels would help? -- Jens Axboe