From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751927AbeEKPs6 (ORCPT ); Fri, 11 May 2018 11:48:58 -0400 Received: from ale.deltatee.com ([207.54.116.67]:55176 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751366AbeEKPs4 (ORCPT ); Fri, 11 May 2018 11:48:56 -0400 To: =?UTF-8?Q?Christian_K=c3=b6nig?= , Stephen Bates , Jerome Glisse Cc: Alex Williamson , Bjorn Helgaas , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , Benjamin Herrenschmidt References: <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <4e0d0b96-ab02-2662-adf3-fa956efd294c@deltatee.com> <2fc61d29-9eb4-d168-a3e5-955c36e5d821@amd.com> <94C8FE12-7FC3-48BD-9DCA-E6A427E71810@raithlin.com> <868B49CE-4F0E-4A48-BE78-12149F85F1A4@raithlin.com> <8113cba8-62b9-1801-7a77-f82be223b183@deltatee.com> From: Logan Gunthorpe Message-ID: <53fa38ce-40dd-6bc1-9b83-c6ccf228b755@deltatee.com> Date: Fri, 11 May 2018 09:48:27 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-SA-Exim-Connect-IP: 172.16.1.162 X-SA-Exim-Rcpt-To: benh@kernel.crashing.org, dan.j.williams@intel.com, maxg@mellanox.com, jgg@mellanox.com, bhelgaas@google.com, sagi@grimberg.me, keith.busch@intel.com, axboe@kernel.dk, hch@lst.de, linux-block@vger.kernel.org, linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, helgaas@kernel.org, alex.williamson@redhat.com, jglisse@redhat.com, sbates@raithlin.com, christian.koenig@amd.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 5/11/2018 2:52 AM, Christian König wrote: > This only works when the IOVA and the PCI bus addresses never overlap. > I'm not sure how the IOVA allocation works but I don't think we > guarantee that on Linux. I find this hard to believe. There's always the possibility that some part of the system doesn't support ACS so if the PCI bus addresses and IOVA overlap there's a good chance that P2P and ATS won't work at all on some hardware. > If we really want to enable P2P without ATS and IOMMU enabled I think we > should probably approach it like this: > > a) Make double sure that IOVA in an IOMMU group never overlap with PCI > BARs in that group. > > b) Add configuration options to put a whole PCI branch of devices (e.g. > a bridge) into a single IOMMU group. > > c) Add a configuration option to disable the ACS bit on bridges in the > same IOMMU group. I think a configuration option to manage IOMMU groups as you suggest would be a very complex interface and difficult to implement. I prefer the option to disable the ACS bit on boot and let the existing code put the devices into their own IOMMU group (as it should already do to support hardware that doesn't have ACS support). Logan