From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754542AbdJIMto (ORCPT ); Mon, 9 Oct 2017 08:49:44 -0400 Received: from mail-sn1nam01on0064.outbound.protection.outlook.com ([104.47.32.64]:4949 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751550AbdJIMtj (ORCPT ); Mon, 9 Oct 2017 08:49:39 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Yuri.Norov@cavium.com; Date: Mon, 9 Oct 2017 15:49:21 +0300 From: Yury Norov To: Will Deacon Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Jeremy.Linton@arm.com, peterz@infradead.org, mingo@redhat.com, longman@redhat.com, boqun.feng@gmail.com, paulmck@linux.vnet.ibm.com Subject: Re: [PATCH v2 0/5] Switch arm64 over to qrwlock Message-ID: <20171009124921.wtbzvqagges44brq@yury-thinkpad> References: <1507296882-18721-1-git-send-email-will.deacon@arm.com> <20171008213052.ojyxpr56d2ypscjy@yury-thinkpad> <20171009095935.GC5127@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20171009095935.GC5127@arm.com> User-Agent: NeoMutt/20170113 (1.7.2) X-Originating-IP: [176.59.107.168] X-ClientProxiedBy: AM0PR0402CA0007.eurprd04.prod.outlook.com (2603:10a6:208:15::20) To CY4PR0701MB3827.namprd07.prod.outlook.com (2603:10b6:910:94::33) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4137886f-120e-4a26-3d18-08d50f14333b X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075);SRVR:CY4PR0701MB3827; X-Microsoft-Exchange-Diagnostics: 1;CY4PR0701MB3827;3:Fzkay6HG7S9qAtM0RYP0JywJIXGmjzY0W1b+hSsn1QViabVByyVQuRiPg95OZga7E1YSazIL/1EAUoMh2Sfyw7395fODCQHOBtrQixhkaJGtN9+ZHJzhtAiJBQdz+/4lyvXKQNXRdDtyP8FlVcxU2zz3ZOQUJAfqfmC0KxtzujZUNfIxN4fouDvvuwzZJJmQfa4ccmx565abvPtjtgUqfrPCXh+0XVxO8RcQRmxCTFNQ/OI7r97jhPtYK25CBFg1;25:Db4YT8/Fp+erfButait38O/sOk8ANjc0fNJLmEY8qG8ZxJcvSrD/+fuTlRN27FFOfXgVoY6PyxFVLj/LsyiyLjKaZkSP/04b4L3CjanfbHZNiI19XT1v7U+5pwSIduul0vTYokVvSfBcsSTdK2o1sFRzB4bj2pu/B+805vr3qnyqlnMKbssKKWy6mBienJPgJTbE7B5gJBIwpNot7SNGbun4o3iX61kUlbIfhdH+uTEJj5o4Gq0ZgUCjrTr2SmCd3RIslVuHGexxpAemg3i5C73/psvvFEfyDhMvZg6TcjMCTpThXCU9YH+MsfHRkMSw7ZnVsf239k6GaSesh5HzJA==;31:ALu6oDXJRu7aMrrzhOa0HLOcnHWcyAU2SKY/Fzzn2fqp1OxeVZHmiuyp6WYZStSXdIidoQZkDBoa+bViiHJz8eCmX/bDL18//qlpzKf6x2+gYlrHw8We2r3Ic0/dwz4UmpIyFB1uVbjDVSHd0izNCmaNJyRPKayWCXK+Bcix+IdSozJJw/d4QOfdrt0irlwmG89i/eR6oaPssCGvGKM7MkCBAuZrpCvCIMQreYHh2Vo= X-MS-TrafficTypeDiagnostic: CY4PR0701MB3827: X-Microsoft-Exchange-Diagnostics: 1;CY4PR0701MB3827;20:Yhte1Rdcpoyhm7VnIeacqwUlApm2+j1pUyGhC3YwuI1XyVfXzKMZLGyddvtdswv6r+sRj8M5jj+oKl3k0Jt/qs7Ue+QQHwU9MqaDekcZzsIx0AKfKX1Qa8RJ4DExQ8GRa60qH4J6Wsk4O0xpbgmV8QIlIGWjNzkWjW1eokoFQRHm4MYhdXjraCUdNmoM5JNDcCLZMITF7F4AefyjksXzstz1wTbLh6OOYgZrZ2Hqx2loPG6MMsKAG00tO2pZlhjIcNan5Gajtp4b5NyUcKR3SSYXyYvt8eX+9aMnMB35JP+H6aaKDH3dwkROoCMzkhX/AOX9eZfqaEUqLwc5Hms6NRv2oWx1tSzF7NIiJ8LsHnvfsWPSWZFPRDo9leo/ZOMkHvvr1aptXsxYxbzhKNMGhrQ1ac9znjU/NdkHYKeAlXRg9rAGso5xe1F7/WlixTOoHBSKqdKTNrSpUXYQZEoRoQBNl1Gys3jcsakq/OepxJvCCLf3XSwTK8/Wvi0Dhv9nU3f4+F6yuVyaj1fcm5bhRVaZNOp+tB6OeOAg+TrIbF2OvyZdGPX8gD5yx+NVkOwYP66nVoM2kFQLe5QAorOp3KE47lPF19e8x/hvaC21gnM= X-Exchange-Antispam-Report-Test: UriScan:(258649278758335)(84791874153150); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(93006095)(3002001)(100000703101)(100105400095)(10201501046)(6041248)(20161123558100)(20161123555025)(20161123562025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:CY4PR0701MB3827;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:CY4PR0701MB3827; X-Microsoft-Exchange-Diagnostics: 1;CY4PR0701MB3827;4:J0W7lgqcMsNiBsGx0aKkvbLQUe0yYzWOjrcH1GrLiWCcOuTi8nCQcsGPsKYWjBGpopPRVII6bXmuL0h7I5SzpqxLqaobwBtTnFAODkMiVn5HvbfF2SwfQ0tK80DfPcXjyzPU6ZUHfn9SZOLfIoe14Em1yRlnXyhILFVlJjQ52R/pwy8hZ8qOe/7t8+FlgjiMAzdmI3u/MZMvbx9zAOFh8IuyowdjsNVO93W4iec8h5KVFaPrUpBK8HJTqQUS2q+P0xCmB6WhUIPFohoJab38pMqebZ52rlkGBeM94M29noOfltNsfEghH90Z/aFLuPBCATVFR0FTfmOYko2Jt2yP+Q== X-Forefront-PRVS: 045584D28C X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6069001)(6009001)(7916004)(346002)(376002)(189002)(24454002)(199003)(105586002)(6116002)(7736002)(6306002)(83506001)(305945005)(6916009)(42882006)(2950100002)(478600001)(6496005)(9686003)(33646002)(6246003)(53936002)(316002)(3846002)(25786009)(16586007)(97736004)(68736007)(8936002)(6666003)(16526018)(23726003)(1076002)(8676002)(966005)(72206003)(81156014)(229853002)(5660300001)(58126008)(6486002)(81166006)(76176999)(50986999)(54356999)(2906002)(50466002)(101416001)(189998001)(4326008)(76506005)(106356001)(33716001)(39060400002)(66066001)(47776003)(533714002);DIR:OUT;SFP:1101;SCL:1;SRVR:CY4PR0701MB3827;H:localhost;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;CY4PR0701MB3827;23:fR+QVkUK8W6JlgXrNOlD3PLJ1m/gRKJml6n3S9q?= =?us-ascii?Q?TM/NfD9TZBd2TColOEi/Oj6Oz2BLR9tINcZqUmDoIuzpgBEkX/jLcUUVEmea?= =?us-ascii?Q?vvlb+KfTDdbtGcSYi6rX38BckY5jvkpYsLqy/R3hv5pDrUry1ImwLb7Sde+D?= =?us-ascii?Q?RS0Nhpzryp9wW1YP3Z2DJGckaMf6UFAyOJmUap7yMpHqigzsk0CN8VQWZuaV?= =?us-ascii?Q?U0B0C7Qhb8g/XI2vVf4b+RVVj7WTQRr5B7PNrTun8qy/XjbEhUyiKYPNnl8Q?= =?us-ascii?Q?cU85wEHtuBB33e3y3RWx2M/hB2amN5qYrUG88Pl5c8M5vdvv/4Scy5MezWys?= =?us-ascii?Q?O9rntkr0N9+bxPnCII+yI9eozvioY+E6Gd6+oynOlPxsJ8w89eSZ3OzfZ89B?= =?us-ascii?Q?9ic06UeYHtUjQU8xK5miAjDXqcGAOjzt23f0BNerGnJ15nGl5M8iijf5nEEh?= =?us-ascii?Q?hgn3debunSJwJ/YiW0ZRAXN3X6PGZ3yPAs6Eaq3KV1pmjUGSFxRwVGsAcD09?= =?us-ascii?Q?salbo7EvU/hQe8/Cjlljmk7oya01T4ze2rPhYouHXXS1WqKVH+PI6lEPJ0CH?= =?us-ascii?Q?jdH20G2QiRoKjXQP+rDAWqc4pbBhms/2ZxTspvnKsvIX4Kh0eBFzorP3pxef?= =?us-ascii?Q?w1MYbdnbE64Dy60jW1QuY6MuJXe5/RO2t7zCORad8iHZMQwP8NiP8FvBdQB4?= =?us-ascii?Q?5qseC3i2fWpdoCsF2CCKMDeC8Hw7TDRUAuh3t4D/xDFX4QjU0U6U5x5Jw6P+?= =?us-ascii?Q?lFBOlpsHB2LWUKQZ06N2PT7Tm/sAccFiG7TZ6IKv2kPKx1yjvSmSDJzifMKg?= =?us-ascii?Q?IeAaIOq9TtqzZ1bZLUm78sWcJTO7qolfai9g6q2wFBDxLJUTi/SV38j4ik0F?= =?us-ascii?Q?6cggaiwxYIEzCk+2oaUqpXI3e+HqH8X6iilstU90/hAmtFX81TAynQ+B6dIp?= =?us-ascii?Q?zhCtJl30OKbI+rQf3+Brm6kRumLYYvqFl8HhIADy4CTFaEceG9tZZfW76uVz?= =?us-ascii?Q?mr2Sl9fTVOE8nvEBrFEpbhewktdwotrNbqXGvIJCR7EcsaIhStgUheNS8CEv?= =?us-ascii?Q?BZtGo6l0bRULIVjkUARD1Ik9s00ea0W35cVXmf6VmfOENDrbnsxT3rMrlBJZ?= =?us-ascii?Q?KEja5SG3EMKrSJNgSpJZF0lmSE0EedcbC/qQ48WKxBKy3HVKNXsTnqnJdWCn?= =?us-ascii?Q?L7SP3Tx3XKZoDBdZL81xoNZ1g+qRopRyfMYGNVXtCY8q4Dr5lgCs/XzZU++J?= =?us-ascii?Q?G9Y2GjLRGL/g6uy8VIM7KXW+JjkmC6A56zJ45EXiPsBkI1lZTAtrV1k8WAhN?= =?us-ascii?Q?v9U2gWZ0qO5PHMUBk/sYbv2o=3D?= X-Microsoft-Exchange-Diagnostics: 1;CY4PR0701MB3827;6:D6ml+26AyKl199rH/VS09He3a+miaWV57yrcZcdFEi5aJ5MDYBQni2uSUkwm+/AapkTEDGYPXNNh3kmQ66FnoFZWuwsSthfAOprmzCyoUS60+yrb6xQ17MGIhAvluNvRx5GicqIg/VCluGi/nsc5w6BFU2WqIIrerxreW6cctAmJbru/bWBF4W272WPQOMKg2lpdQft9mPDKXrSSdPdPozUoeTdmbQ88o9jSj6Oy3uo9vVJtltIXorL0nnfHIX5SPxZlhOXgQlCHy4JkdURgA/HFC/KVft67+dYWScc1IYA9UZbfRoFpX0DUIvKpucyXYzthC7sA9RjnhNfha59FSw==;5:VtaEwkNWFlXhglfrztX7XyW9kH+uHcSUj7eWGFlsueOEqWC8coCOcK21p0ebXZVWcCMaMERDVlZa/Lg9TRDApVS3Q9T0MqLvSf6ao0KJD7ZkZAjdXjD+VUteY/kiiY0iqGt0sCaX04vq/RQMYpa8Qg==;24:NiNuDnV7f/c/5UxTDFz0gG0JV4vrVh3sxOVBzj1c9L6pgAwPQv9Cy/V71NWlxMcN9ZV0xGxNcgjpjBD/R4Fvw/JQUOtERMwg4WQQwNXm804=;7:B2YM6zdXytTqr0TtI0f6VXoRmza48PPJ5DbmJZA3zTpseyagCo2a2XthlolOhwmXdq+Vn/L164gnJmWCli5ax/NOF8zGikwRBNwIjv8VZ4o5aQAKUp705n5rto5lmZtR6qHDip891PW/OMjyowS1HZKBo71WRLiCUJyDk+pubOmKIlyVxt5jik2KyjU405vW/P0jeXOmHsTrXJ6J4bavMCRcFds4G2twqimlFgTBGpA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2017 12:49:36.9200 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR0701MB3827 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Oct 09, 2017 at 10:59:36AM +0100, Will Deacon wrote: > Hi Yury, > > On Mon, Oct 09, 2017 at 12:30:52AM +0300, Yury Norov wrote: > > On Fri, Oct 06, 2017 at 02:34:37PM +0100, Will Deacon wrote: > > > This is version two of the patches I posted yesterday: > > > > > > http://lists.infradead.org/pipermail/linux-arm-kernel/2017-October/534666.html > > > > > > I'd normally leave it longer before posting again, but Peter had a good > > > suggestion to rework the layout of the lock word, so I wanted to post a > > > version that follows that approach. > > > > > > I've updated my branch if you're after the full patch stack: > > > > > > git://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git qrwlock > > > > > > As before, all comments (particularly related to testing and performance) > > > welcome! > > > > > I tested your patches with locktorture and found measurable performance > > regression. I also respin the patch of Jan Glauber [1], and I also > > tried Jan's patch with patch 5 from this series. Numbers differ a lot > > from my previous measurements, but since that I changed working > > station and use qemu with the support of parallel threads. > > Spinlock Read-RW lock Write-RW lock > > Vanilla: 129804626 12340895 14716138 > > This series: 113718002 10982159 13068934 > > Jan patch: 117977108 11363462 13615449 > > Jan patch + #5: 121483176 11696728 13618967 > > > > The bottomline of discussion [1] was that queued locks are more > > effective when SoC has many CPUs. And 4 is not many. My measurement > > was made on the 4-CPU machine, and it seems it confirms that. Does > > it make sense to make queued locks default for many-CPU machines only? > > Just to confirm, you're running this under qemu on an x86 host, using full > AArch64 system emulation? If so, I really don't think we should base the > merits of qrwlocks on arm64 around this type of configuration. Given that > you work for a silicon vendor, could you try running on real arm64 hardware > instead, please? I don't have the hardware access at the moment. I'll run the test when I'll get it. > My measurements on 6-core and 8-core systems look a lot > better with qrwlock than what we currently have in mainline, and they > also fix a real starvation issue reported by Jeremy [1]. > > I'd also add that lock fairness comes at a cost, so I'd expect a small drop > in total throughput for some workloads. I encourage you to try passing > different arguments to locktorture to see this in action. For example, on > an 8-core machine: > > # insmod ./locktorture.ko nwriters_stress=2 nreaders_stress=8 torture_type="rw_lock_irq" stat_interval=2 > > -rc3: > > Writes: Total: 6612 Max/Min: 0/0 Fail: 0 > Reads : Total: 1265230 Max/Min: 0/0 Fail: 0 > Writes: Total: 6709 Max/Min: 0/0 Fail: 0 > Reads : Total: 1916418 Max/Min: 0/0 Fail: 0 > Writes: Total: 6725 Max/Min: 0/0 Fail: 0 > Reads : Total: 5103727 Max/Min: 0/0 Fail: 0 > > notice how the writers are really struggling here (you only have to tweak a > bit more and you get RCU stalls, lose interrupts etc). > > With the qrwlock: > > Writes: Total: 47962 Max/Min: 0/0 Fail: 0 > Reads : Total: 277903 Max/Min: 0/0 Fail: 0 > Writes: Total: 100151 Max/Min: 0/0 Fail: 0 > Reads : Total: 525781 Max/Min: 0/0 Fail: 0 > Writes: Total: 155284 Max/Min: 0/0 Fail: 0 > Reads : Total: 767703 Max/Min: 0/0 Fail: 0 > > which is an awful lot better for maximum latency and fairness, despite the > much lower reader count. > > > There were 2 preparing patches in the series: > > [PATCH 1/3] kernel/locking: #include in qrwlock > > and > > [PATCH 2/3] asm-generic: don't #include in qspinlock_types.h > > > > 1st patch is not needed anymore because Babu Moger submitted similar patch that > > is already in mainline: 9ab6055f95903 ("kernel/locking: Fix compile error with > > qrwlock.c"). Could you revisit second patch? > > Sorry, not sure what you're asking me to do here. It removes unneeded #include in include/asm-generic/qspinlock_types.h. Could you or someone else take it upstream? > Will > > [1] http://lists.infradead.org/pipermail/linux-arm-kernel/2017-October/534299.html