From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751200AbdE3RAB (ORCPT ); Tue, 30 May 2017 13:00:01 -0400 Received: from foss.arm.com ([217.140.101.70]:33520 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751002AbdE3Q76 (ORCPT ); Tue, 30 May 2017 12:59:58 -0400 Subject: Re: Device address specific mapping of arm,mmu-500 To: Ray Jui , Will Deacon References: <1b79efe2-6835-7a7a-f5ad-361391a7b967@broadcom.com> <20170530151437.GC23067@arm.com> <81637642-22d9-4868-156f-052f64bd042f@broadcom.com> Cc: Robin Murphy , Mark Rutland , Joerg Roedel , linux-arm-kernel@lists.infradead.org, iommu@lists.linux-foundation.org, "linux-kernel@vger.kernel.org" From: Marc Zyngier Organization: ARM Ltd Message-ID: <226bcebc-3902-90d3-24e5-51f2e1f3affb@arm.com> Date: Tue, 30 May 2017 17:59:54 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <81637642-22d9-4868-156f-052f64bd042f@broadcom.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 30/05/17 17:49, Ray Jui wrote: > Hi Will, > > On 5/30/17 8:14 AM, Will Deacon wrote: >> On Mon, May 29, 2017 at 06:18:45PM -0700, Ray Jui wrote: >>> I'm writing to check with you to see if the latest arm-smmu.c driver in >>> v4.12-rc Linux for smmu-500 can support mapping that is only specific to >>> a particular physical address range while leave the rest still to be >>> handled by the client device. I believe this can already be supported by >>> the device tree binding of the generic IOMMU framework; however, it is >>> not clear to me whether or not the arm-smmu.c driver can support it. >>> >>> To give you some background information: >>> >>> We have a SoC that has PCIe root complex that has a build-in logic block >>> to forward MSI writes to ARM GICv3 ITS. Unfortunately, this logic block >>> has a HW bug that causes the MSI writes not parsed properly and can >>> potentially corrupt data in the internal FIFO. A workaround is to have >>> ARM MMU-500 takes care of all inbound transactions. I found that is >>> working after hooking up our PCIe root complex to MMU-500; however, even >>> with this optimized arm-smmu driver in v4.12, I'm still seeing a >>> significant Ethernet throughput drop in both the TX and RX directions. >>> The throughput drop is very significant at around 50% (but is already >>> much improved compared to other prior kernel versions at 70~90%). >> >> Did Robin's experiments help at all with this? >> >> http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/perf >> > > It looks like these are new optimizations that have not yet been merged > in v4.12? I'm going to give it a try. > >>> One alternative is to only use MMU-500 for MSI writes towards >>> GITS_TRANSLATER register in the GICv3, i.e., if I can define a specific >>> region of physical address that I want MMU-500 to act on and leave the >>> rest of inbound transactions to be handled directly by our PCIe >>> controller, it can potentially work around the HW bug we have and at the >>> same time achieve optimal throughput. >> >> I don't think you can bypass the SMMU for MSIs unless you give them their >> own StreamIDs, which is likely to break things horribly in the kernel. You >> could try to create an identity mapping, but you'll still have the >> translation overhead and you'd probably end up having to supply your own DMA >> ops to manage the address space. I'm assuming that you need to prevent the >> physical address of the ITS from being allocated as an IOVA? > > Will, is that a HW limitation that the SMMU cannot be used, only for MSI > writes, in which case, the physical address range is very specific in > our ASIC that falls in the device memory region (e.g., below 0x80000000)? > > In fact, what I need in this case is a static mapping from IOMMU on the > physical address of the GITS_TRANSLATER of the GICv3 ITS, which is the > address that MSI writes go to. This is to bypass the MSI forwarding > logic in our PCIe controller. At the same time, I can leave the rest of > inbound transactions to be handled by our PCIe controller without going > through the MMU. How is that going to work for DMA? I imagine your network interfaces do have to access memory, don't they? How can the transactions be terminated in the PCIe controller? Thanks, M. -- Jazz is not dead. It just smells funny...