From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753124AbeCFBOS (ORCPT ); Mon, 5 Mar 2018 20:14:18 -0500 Received: from ale.deltatee.com ([207.54.116.67]:37326 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753039AbeCFBOP (ORCPT ); Mon, 5 Mar 2018 20:14:15 -0500 To: Oliver Cc: Keith Busch , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, "linux-nvdimm@lists.01.org" , linux-block@vger.kernel.org, Jens Axboe , Benjamin Herrenschmidt , Alex Williamson , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-8-logang@deltatee.com> <20180305160004.GA30975@localhost.localdomain> <3f56c76d-6a5c-7c2f-5442-c9209749b598@deltatee.com> From: Logan Gunthorpe Message-ID: <416d8d27-200f-befb-1c30-14544fffcba0@deltatee.com> Date: Mon, 5 Mar 2018 18:14:02 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: hch@lst.de, maxg@mellanox.com, bhelgaas@google.com, jgg@mellanox.com, jglisse@redhat.com, alex.williamson@redhat.com, benh@kernel.crashing.org, axboe@kernel.dk, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, keith.busch@intel.com, oohall@gmail.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v2 07/10] nvme-pci: Use PCI p2pmem subsystem to manage the CMB X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/03/18 05:49 PM, Oliver wrote: > It's in arch/powerpc/kernel/io.c as _memcpy_toio() and it has two full barriers! > > Awesome! > > Our io.h indicates that our iomem accessors are designed to provide x86ish > strong ordering of accesses to MMIO space. The git log indicates > arch/powerpc/kernel/io.c has barely been touched in the last decade so > odds are most of that code was written in the elder days when people > were less aware of ordering issues. It might just be overly conservative > by today's standards, but maybe not (see below). Yes, that seems overly conservative. > (I'm not going to suggest ditching the lwsync trick. mpe is not going > to take that patch > without a really good reason) Well, that's pretty gross. Is this not exactly the situation mmiowb() is meant to solve? See [1]. Though, you're right in principle. Even if power was similar to other systems in this way, it's still a risk that if these pages get passed somewhere in the kernel that uses a spin lock like that without an mmiowb() call, then it's going to have a bug. For now, the risk is pretty low as we know exactly where all the p2pmem pages will be used but if it gets into other places, all bets are off. I did do some work trying to make a safe version of io-pages and also trying to change from pages to pfn_t in large areas but neither approach seemed likely to get any traction in the community, at least not in the near term. Logan [1] ACQUIRES VS I/O ACCESSES in https://www.kernel.org/doc/Documentation/memory-barriers.txt