From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935608AbdAJPHQ (ORCPT ); Tue, 10 Jan 2017 10:07:16 -0500 Received: from mout.kundenserver.de ([212.227.17.24]:54370 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1760754AbdAJPHI (ORCPT ); Tue, 10 Jan 2017 10:07:08 -0500 From: Arnd Bergmann To: Robin Murphy Cc: Nikita Yushchenko , Will Deacon , linux-arm-kernel@lists.infradead.org, Catalin Marinas , linux-kernel@vger.kernel.org, linux-renesas-soc@vger.kernel.org, Simon Horman , Bjorn Helgaas , artemi.ivanov@cogentembedded.com, fkan@apm.com, Christoph Hellwig Subject: Re: [PATCH v2] arm64: do not set dma masks that device connection can't handle Date: Tue, 10 Jan 2017 16:06:29 +0100 Message-ID: <1900630.ldF24KWUdV@wuerfel> User-Agent: KMail/5.1.3 (Linux/4.4.0-34-generic; KDE/5.18.0; x86_64; ; ) In-Reply-To: <5c5cd4fd-4854-a2dd-10b6-9cc98e63a85c@arm.com> References: <1483947002-16410-1-git-send-email-nikita.yoush@cogentembedded.com> <6116278.nQQUSuo3l4@wuerfel> <5c5cd4fd-4854-a2dd-10b6-9cc98e63a85c@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Provags-ID: V03:K0:QhmCTTOGrih/QGWm281v1D4ky07UZHRGclA3uqbwmFSRGkxiVuH yUjZOAIGb910z+iT2F6Hddphx3CqHPNezKiaMx/ssJjo94SBgZEixu1yLu0woawBKjS/54M +xcEgrEreA6aCJANr1kNuwF330Yeb4M3qcOat0m8vHtJF57HwQy1Rwuml14BtS9wrVplvmu zqYqPQfDE+a7n2U7uKpAw== X-UI-Out-Filterresults: notjunk:1;V01:K0:NXkr1p9L5ZI=:LRahna/+b+PZhwOFNKqx2E kZHdOhMRptRv2dBvvsxtQc1JkQmRWvRNn9tKjQcImjJCeMyRUywGn33WS4opa0Pd1ujGfWWp6 0SNPlaVAyHP7mAIlKhW0Cne5MT2aQjDo112uyO3X4Yn7bP/4ustBOvziCd93ieemg//D7k6WV BN4awNi7Mo2Zp4YQSzz2ivEne23eRDA407qnZlAwmEQGIiGNY5PGEv94xXfIynh0r2zKKBhQc u0NdBQW7kdEE4UHMTiDovjJW+x4F8ZejYJ/Wb4SBMf5RXFOGGwxEDNfS5HxWIpeo+BWKSK7Cm Qbh9mXuXU9zm/6QNxfOYHSDpCOFWpXn/BpWByb6yUweEEpW1/ZYrkJ9mZ0BVDG8gn761XHSHH 717jS31yrEfYLp1Rr23F9Tvk4FvH8jSimi692RPGOrpoCQlwJqI6kUgG9NTMq/5iKbeLVwYuc PA5s2aCB36H+NXreV5ryNYmVBib9TsDH9u60rTxOQfQrl1OMbuXFmHLXaTJo96rQTePzK2mrZ h7DuBuUUf7i38rHJt/PqY97SxAUBGnNBEMDT6zX5esmMu7AgLmuG/g/XfsXyXPZfJr1c9m0yY zbKKHJ3sRf5OttvxGp+8MqWS/o2KGiuLwIs7IHFLpkrTbRav0R9LxUlX7icqS+YQSuURyuVzi QU5q8uPKcRAJi2ACwPUYac6r+RE5v1cDTKgbUVX24izsz/sjJzoGRL9OtWwZbM/Cmmplwfm3b yAdTy4OaWSF1vunC7rPAZljeF/nKSACYh3PeFENj/+lar6W5IgbP4iXqZYGY6WdpFQI6Q0dS+ nxq3Z8p1WFxxuk7q8u4jyEL47qtUw== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tuesday, January 10, 2017 2:16:57 PM CET Robin Murphy wrote: > On 10/01/17 13:42, Arnd Bergmann wrote: > > On Tuesday, January 10, 2017 1:25:12 PM CET Robin Murphy wrote: > >> On 10/01/17 12:47, Nikita Yushchenko wrote: > >>>> The point here is that an IOMMU doesn't solve your issue, and the > >>>> IOMMU-backed DMA ops need the same treatment. In light of that, it really > >>>> feels to me like the DMA masks should be restricted in of_dma_configure > >>>> so that the parent mask is taken into account there, rather than hook > >>>> into each set of DMA ops to intercept set_dma_mask. We'd still need to > >>>> do something to stop dma_set_mask widening the mask if it was restricted > >>>> by of_dma_configure, but I think Robin (cc'd) was playing with that. > >>> > >>> What issue "IOMMU doesn't solve"? > >>> > >>> Issue I'm trying to address is - inconsistency within swiotlb > >>> dma_map_ops, where (1) any wide mask is silently accepted, but (2) then > >>> mask is used to decide if bounce buffers are needed or not. This > >>> inconsistency causes NVMe+R-Car cobmo not working (and breaking memory > >>> instead). > >> > >> The fundamental underlying problem is the "any wide mask is silently > >> accepted" part, and that applies equally to IOMMU ops as well. > > > > It's a much rarer problem for the IOMMU case though, because it only > > impacts devices that are restricted to addressing of less than 32-bits. > > > > If you have an IOMMU enabled, the dma-mapping interface does not care > > if the device can do wider than 32 bit addressing, as it will never > > hand out IOVAs above 0xffffffff. > > I can assure you that it will - we constrain allocations to the > intersection of the IOMMU domain aperture (normally the IOMMU's physical > input address width) and the given device's DMA mask. If both of those > are >32 bits then >32-bit IOVAs will fall out. For the arm64/common > implementation I have prototyped a copy of the x86 optimisation which > always first tries to get 32-bit IOVAs for PCI devices, but even then it > can start returning higher addresses if the 32-bit space fills up. Ok, got it. I have to admit that most of my knowledge about the internals of IOMMUs is from PowerPC of a few years ago, which couldn't do this at all. I agree that we need to do the same thing on swiotlb and iommu then. > >> The thread Will linked to describes that equivalent version of your > >> problem - the IOMMU gives the device 48-bit addresses which get > >> erroneously truncated because it doesn't know that only 42 bits are > >> actually wired up. That situation still requires the device's DMA mask > >> to correctly describe its addressing capability just as yours does. > > > > That problem should only impact virtual machines which have a guest > > bus address space covering more than 42 bits of physical RAM, whereas > > the problem we have with swiotlb is for the dma-mapping interface. > > > I actually have a third variation of this problem involving a PCI root > complex which *could* drive full-width (40-bit) addresses, but won't, > due to the way its PCI<->AXI interface is programmed. That would require > even more complicated dma-ranges handling to describe the windows of > valid physical addresses which it *will* pass, so I'm not pressing the > issue - let's just get the basic DMA mask case fixed first. Can you describe this a little more? We should at least try to not make it harder to solve the next problem while solving this one, so I'd like to understand the exact limitation you are hitting there. Arnd