From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8C25C48BE5 for ; Tue, 22 Jun 2021 12:01:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B8B4160FF1 for ; Tue, 22 Jun 2021 12:01:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231201AbhFVMEC (ORCPT ); Tue, 22 Jun 2021 08:04:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229913AbhFVMEA (ORCPT ); Tue, 22 Jun 2021 08:04:00 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A708C061574 for ; Tue, 22 Jun 2021 05:01:44 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id f70so38608229qke.13 for ; Tue, 22 Jun 2021 05:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=uZXjrCktHYqiHY7Zuave3Zqe+nWlABu14O38dLKiRSc=; b=V1vGTztZADrEdhzlzySX29OAyznJd+LoMCj0Tl5HgSKC/gYfSvRa4RbGq1dowEhGfV daLDhPvM9+xafGLmtfuTCbex8YEiqsKJyrpDW4po9e16l0EgzUkVl44Hmyz39LGoO7xT TzLfvUBU/DW8f9Bi5bQpi35/7V2g6v30a9D8yfcA8rTZQB+vxYyM9vAI2U0vDLGGfIJy vutkFVFlx3zvB/LwkqKxO1Zhegmmb3Lh6V0Sj8MNmRXtTiRqKmKokko1doEKvfc01llt m1k5v66LhFEesVNI9Fs78siXiJEJKb+BhvYsnd/58x+uaR7+OOZwrnkeKi2AGgaPMM8m J+VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=uZXjrCktHYqiHY7Zuave3Zqe+nWlABu14O38dLKiRSc=; b=UDTj2HgG6BJqIbDkGlGt09kyWH57l+C/FfWRSFUS7gsdwqalskJlOf5/4yMqupQOU8 0E0HJLidJ6Uip0Q/PAMds+aZ4BTszcwMt3tNhHrN+RK+CwfURpqqGaDi50V7cRxgknei ynN51YgfXPI4UIxfhvbd5c8sOYMM0y9Ry9ahR0J56H1iJa5i/k6zIC3su/Oiq3YaWrsp 8iBvoagC8LxFg71kN8X1RiGX4WHR+LmW34lnD17mH9fsVvzEORYoKbKdna0S5O5H9pGh /vZeV8sltnduAb88c3ikIWhtnJvtqOrWRjzuKfWseMxvY/YiN+HCr+1vU3R3TYNnDcqs dF+Q== X-Gm-Message-State: AOAM532Rgv+WAkY/49w6o73wnM53pq46Q/t8g8cLx5Qsw9QaYjnFqHJA pSu7/Y9L1MH47d3MMYopzTYrqg== X-Google-Smtp-Source: ABdhPJya7BBhAOKrUhmySN3jYp1EpbKfOqKBSj+U+j//RgFUf85U7EK52a+yYa92TMoPjVXQfaSu4g== X-Received: by 2002:a37:496:: with SMTP id 144mr3716033qke.456.1624363303727; Tue, 22 Jun 2021 05:01:43 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-113-94.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.113.94]) by smtp.gmail.com with ESMTPSA id d23sm1485085qto.74.2021.06.22.05.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 05:01:43 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lvf66-00A9JB-Im; Tue, 22 Jun 2021 09:01:42 -0300 Date: Tue, 22 Jun 2021 09:01:42 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: Christian =?utf-8?B?S8O2bmln?= , Gal Pressman , sleybo@amazon.com, linux-rdma , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , Christian =?utf-8?B?S8O2bmln?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Message-ID: <20210622120142.GL1096940@ziepe.ca> References: <20210621141217.GE1096940@ziepe.ca> <20210621175511.GI1096940@ziepe.ca> <20210621232912.GK1096940@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > On Tue, Jun 22, 2021 at 9:37 AM Christian König > wrote: > > > > Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: > > > On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote: > > > > > >> Another thing I want to emphasize is that we are doing p2p only > > >> through the export/import of the FD. We do *not* allow the user to > > >> mmap the dma-buf as we do not support direct IO. So there is no access > > >> to these pages through the userspace. > > > Arguably mmaping the memory is a better choice, and is the direction > > > that Logan's series goes in. Here the use of DMABUF was specifically > > > designed to allow hitless revokation of the memory, which this isn't > > > even using. > > > > The major problem with this approach is that DMA-buf is also used for > > memory which isn't CPU accessible. That isn't an issue here because the memory is only intended to be used with P2P transfers so it must be CPU accessible. > > That was one of the reasons we didn't even considered using the mapping > > memory approach for GPUs. Well, now we have DEVICE_PRIVATE memory that can meet this need too.. Just nobody has wired it up to hmm_range_fault() > > > So you are taking the hit of very limited hardware support and reduced > > > performance just to squeeze into DMABUF.. > > Thanks Jason for the clarification, but I honestly prefer to use > DMA-BUF at the moment. > It gives us just what we need (even more than what we need as you > pointed out), it is *already* integrated and tested in the RDMA > subsystem, and I'm feeling comfortable using it as I'm somewhat > familiar with it from my AMD days. You still have the issue that this patch is doing all of this P2P stuff wrong - following the already NAK'd AMD approach. > I'll go and read Logan's patch-set to see if that will work for us in > the future. Please remember, as Daniel said, we don't have struct page > backing our device memory, so if that is a requirement to connect to > Logan's work, then I don't think we will want to do it at this point. It is trivial to get the struct page for a PCI BAR. Jason