From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.2 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33B7DC433DB for ; Mon, 1 Feb 2021 18:25:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DD3FF64EA1 for ; Mon, 1 Feb 2021 18:25:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231599AbhBASZn (ORCPT ); Mon, 1 Feb 2021 13:25:43 -0500 Received: from mail29.static.mailgun.info ([104.130.122.29]:41649 "EHLO mail29.static.mailgun.info" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232773AbhBASVe (ORCPT ); Mon, 1 Feb 2021 13:21:34 -0500 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg.codeaurora.org; q=dns/txt; s=smtp; t=1612203657; h=In-Reply-To: Content-Type: MIME-Version: References: Message-ID: Subject: Cc: To: From: Date: Sender; bh=KCP5gKkHPVjwLZx5nyh5N3OYo5vrT4dmcu2MEeNyjnI=; b=h/hwr8pewfuHt0OH1Lp3/E6XaqH6vsdgT7EPQ4aq4DS6Gg7Bxh194y4HmkA4UE8+Ttzy+qoK Scn7WAx5kNX3vSUNOoAEJOIwiD4H2o7GheH3TidkdCAJc31EV1Oo2k3wrWXL7G7GkGU9VvoX 5T5qx7slJccxB91wQscj/BhJZuw= X-Mailgun-Sending-Ip: 104.130.122.29 X-Mailgun-Sid: WyI0MWYwYSIsICJsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnIiwgImJlOWU0YSJd Received: from smtp.codeaurora.org (ec2-35-166-182-171.us-west-2.compute.amazonaws.com [35.166.182.171]) by smtp-out-n02.prod.us-west-2.postgun.com with SMTP id 60184666ab96aecb9fd1de78 (version=TLS1.2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Mon, 01 Feb 2021 18:20:22 GMT Sender: jcrouse=codeaurora.org@mg.codeaurora.org Received: by smtp.codeaurora.org (Postfix, from userid 1001) id 67FA9C43462; Mon, 1 Feb 2021 18:20:22 +0000 (UTC) Received: from jcrouse1-lnx.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: jcrouse) by smtp.codeaurora.org (Postfix) with ESMTPSA id 5F93FC433C6; Mon, 1 Feb 2021 18:20:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 5F93FC433C6 Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: aws-us-west-2-caf-mail-1.web.codeaurora.org; spf=fail smtp.mailfrom=jcrouse@codeaurora.org Date: Mon, 1 Feb 2021 11:20:16 -0700 From: Jordan Crouse To: Rob Clark Cc: Will Deacon , Sai Prakash Ranjan , Robin Murphy , Joerg Roedel , Akhil P Oommen , "Isaac J. Manjarres" , "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , Linux Kernel Mailing List , linux-arm-msm , freedreno , Kristian H Kristensen , Sean Paul , David Airlie , Daniel Vetter , dri-devel Subject: Re: [PATCH 2/3] iommu/io-pgtable-arm: Add IOMMU_LLC page protection flag Message-ID: <20210201182016.GA21629@jcrouse1-lnx.qualcomm.com> Mail-Followup-To: Rob Clark , Will Deacon , Sai Prakash Ranjan , Robin Murphy , Joerg Roedel , Akhil P Oommen , "Isaac J. Manjarres" , "list@263.net:IOMMU DRIVERS , Joerg Roedel ," , "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE" , Linux Kernel Mailing List , linux-arm-msm , freedreno , Kristian H Kristensen , Sean Paul , David Airlie , Daniel Vetter , dri-devel References: <3f589e7de3f9fa93e84c83420c5270c546a0c368.1610372717.git.saiprakash.ranjan@codeaurora.org> <20210129090516.GB3998@willie-the-truck> <5d23fce629323bcda71594010824aad0@codeaurora.org> <20210201111556.GA7172@willie-the-truck> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Feb 01, 2021 at 08:20:44AM -0800, Rob Clark wrote: > On Mon, Feb 1, 2021 at 3:16 AM Will Deacon wrote: > > > > On Fri, Jan 29, 2021 at 03:12:59PM +0530, Sai Prakash Ranjan wrote: > > > On 2021-01-29 14:35, Will Deacon wrote: > > > > On Mon, Jan 11, 2021 at 07:45:04PM +0530, Sai Prakash Ranjan wrote: > > > > > Add a new page protection flag IOMMU_LLC which can be used > > > > > by non-coherent masters to set cacheable memory attributes > > > > > for an outer level of cache called as last-level cache or > > > > > system cache. Initial user of this page protection flag is > > > > > the adreno gpu and then can later be used by other clients > > > > > such as video where this can be used for per-buffer based > > > > > mapping. > > > > > > > > > > Signed-off-by: Sai Prakash Ranjan > > > > > --- > > > > > drivers/iommu/io-pgtable-arm.c | 3 +++ > > > > > include/linux/iommu.h | 6 ++++++ > > > > > 2 files changed, 9 insertions(+) > > > > > > > > > > diff --git a/drivers/iommu/io-pgtable-arm.c > > > > > b/drivers/iommu/io-pgtable-arm.c > > > > > index 7439ee7fdcdb..ebe653ef601b 100644 > > > > > --- a/drivers/iommu/io-pgtable-arm.c > > > > > +++ b/drivers/iommu/io-pgtable-arm.c > > > > > @@ -415,6 +415,9 @@ static arm_lpae_iopte > > > > > arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data, > > > > > else if (prot & IOMMU_CACHE) > > > > > pte |= (ARM_LPAE_MAIR_ATTR_IDX_CACHE > > > > > << ARM_LPAE_PTE_ATTRINDX_SHIFT); > > > > > + else if (prot & IOMMU_LLC) > > > > > + pte |= (ARM_LPAE_MAIR_ATTR_IDX_INC_OCACHE > > > > > + << ARM_LPAE_PTE_ATTRINDX_SHIFT); > > > > > } > > > > > > > > > > if (prot & IOMMU_CACHE) > > > > > diff --git a/include/linux/iommu.h b/include/linux/iommu.h > > > > > index ffaa389ea128..1f82057df531 100644 > > > > > --- a/include/linux/iommu.h > > > > > +++ b/include/linux/iommu.h > > > > > @@ -31,6 +31,12 @@ > > > > > * if the IOMMU page table format is equivalent. > > > > > */ > > > > > #define IOMMU_PRIV (1 << 5) > > > > > +/* > > > > > + * Non-coherent masters can use this page protection flag to set > > > > > cacheable > > > > > + * memory attributes for only a transparent outer level of cache, > > > > > also known as > > > > > + * the last-level or system cache. > > > > > + */ > > > > > +#define IOMMU_LLC (1 << 6) > > > > > > > > On reflection, I'm a bit worried about exposing this because I think it > > > > will > > > > introduce a mismatched virtual alias with the CPU (we don't even have a > > > > MAIR > > > > set up for this memory type). Now, we also have that issue for the PTW, > > > > but > > > > since we always use cache maintenance (i.e. the streaming API) for > > > > publishing the page-tables to a non-coheren walker, it works out. > > > > However, > > > > if somebody expects IOMMU_LLC to be coherent with a DMA API coherent > > > > allocation, then they're potentially in for a nasty surprise due to the > > > > mismatched outer-cacheability attributes. > > > > > > > > > > Can't we add the syscached memory type similar to what is done on android? > > > > Maybe. How does the GPU driver map these things on the CPU side? > > Currently we use writecombine mappings for everything, although there > are some cases that we'd like to use cached (but have not merged > patches that would give userspace a way to flush/invalidate) > > BR, > -R LLC/system cache doesn't have a relationship with the CPU cache. Its just a little accelerator that sits on the connection from the GPU to DDR and caches accesses. The hint that Sai is suggesting is used to mark the buffers as 'no-write-allocate' to prevent GPU write operations from being cached in the LLC which a) isn't interesting and b) takes up cache space for read operations. Its easiest to think of the LLC as a bonus accelerator that has no cost for us to use outside of the unfortunate per buffer hint. We do have to worry about the CPU cache w.r.t I/O coherency (which is a different hint) and in that case we have all of concerns that Will identified. Jordan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project