From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61433C4338F for ; Tue, 10 Aug 2021 09:52:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 44D5361051 for ; Tue, 10 Aug 2021 09:52:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235877AbhHJJwW (ORCPT ); Tue, 10 Aug 2021 05:52:22 -0400 Received: from foss.arm.com ([217.140.110.172]:52514 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229474AbhHJJwV (ORCPT ); Tue, 10 Aug 2021 05:52:21 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D7BEF1063; Tue, 10 Aug 2021 02:51:59 -0700 (PDT) Received: from [10.57.36.146] (unknown [10.57.36.146]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 920723F70D; Tue, 10 Aug 2021 02:51:58 -0700 (PDT) Subject: Re: [RFC PATCH 2/3] iommu/dma-iommu: Support iovad->granule > PAGE_SIZE To: Sven Peter , Sven Peter Cc: Arnd Bergmann , Hector Martin , linux-kernel@vger.kernel.org, Alexander Graf , Mohamed Mediouni , Will Deacon References: <20210806155523.50429-1-sven@svenpeter.dev> <20210806155523.50429-3-sven@svenpeter.dev> <5002ed91-416c-d7ee-b1ab-a50c590749c2@arm.com> From: Robin Murphy Message-ID: Date: Tue, 10 Aug 2021 10:51:52 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2021-08-09 21:45, Sven Peter wrote: > > > On Mon, Aug 9, 2021, at 19:41, Robin Murphy wrote: >> On 2021-08-07 12:47, Sven Peter via iommu wrote: >>> >>> >>> On Fri, Aug 6, 2021, at 20:04, Robin Murphy wrote: >>>> On 2021-08-06 16:55, Sven Peter via iommu wrote: >>>>> @@ -1006,6 +1019,31 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, >>>>> if (dev_is_untrusted(dev)) >>>>> return iommu_dma_map_sg_swiotlb(dev, sg, nents, dir, attrs); >>>>> >>>>> + /* >>>>> + * If the IOMMU pagesize is larger than the CPU pagesize we will >>>>> + * very likely run into sgs with a physical address that is not aligned >>>>> + * to an IOMMU page boundary. Fall back to just mapping every entry >>>>> + * independently with __iommu_dma_map then. >>>> >>>> Scatterlist segments often don't have nicely aligned ends, which is why >>>> we already align things to IOVA granules in main loop here. I think in >>>> principle we'd just need to move the non-IOVA-aligned part of the >>>> address from sg->page to sg->offset in the temporary transformation for >>>> the rest of the assumptions to hold. I don't blame you for being timid >>>> about touching that, though - it took me 3 tries to get right when I >>>> first wrote it... >>>> >>> >>> >>> I've spent some time with that code now and I think we cannot use it >>> but have to fall back to iommu_dma_map_sg_swiotlb (even though that swiotlb >>> part is a lie then): >>> >>> When we have sg_phys(s) = 0x802e65000 with s->offset = 0 the paddr >>> is aligned to PAGE_SIZE but has an offset of 0x1000 from something >>> the IOMMU can map. >>> Now this would result in s->offset = -0x1000 which is already weird >>> enough. >>> Offset is unsigned (and 32bit) so this will actually look like >>> s->offset = 0xfffff000 then, which isn't much better. >>> And then sg_phys(s) = 0x902e64000 (instead of 0x802e64000) and >>> we'll map some random memory in iommu_map_sg_atomic and a little bit later >>> everything explodes. >>> >>> Now I could probably adjust the phys addr backwards and make sure offset is >>> always positive (and possibly larger than PAGE_SIZE) and later restore it >>> in __finalise_sg then but I feel like that's pushing this a little bit too far. >> >> Yes, that's what I meant. At a quick guess, something like the >> completely untested diff below. > > That unfortunately results in unaligned mappings You mean it even compiles!? :D > [ 9.630334] iommu: unaligned: iova 0xbff40000 pa 0x0000000801a3b000 size 0x4000 min_pagesz 0x4000 > > I'll take a closer look later this week and see if I can fix it. On reflection, "s->offset ^ s_iova_off" is definitely wrong, that more likely wants to be "s->offset & ~s_iova_off". Robin. >> It really comes down to what we want to >> achieve here - if it's just to make this thing work at all, then I'd >> favour bolting on the absolute minimum changes, possibly even cheating >> by tainting the kernel and saying all bets are off instead of trying to >> handle the more involved corners really properly. However if you want to >> work towards this being a properly-supported thing, then I think it's >> worth generalising the existing assumptions of page alignment from the >> beginning. > > I'd like to try and see if we can make this a properly-supported thing. > > That will likely take a few iterations but realistically the rest of the drivers > required to make this platform actually useful (and especially the display controller > and GPU drivers) won't be ready for a few more months anyway. And even on 4KB PAGE_SIZE > kernels half the USB ports and NVMe will work fine, which should be enough to install > a distro and some third-party package that just ships the distro kernel with 16KB > pages. > > > > > Sven >