From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 163DEC49EA4 for ; Tue, 22 Jun 2021 16:05:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 023A5610C7 for ; Tue, 22 Jun 2021 16:05:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230031AbhFVQH6 (ORCPT ); Tue, 22 Jun 2021 12:07:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229751AbhFVQH5 (ORCPT ); Tue, 22 Jun 2021 12:07:57 -0400 Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62B30C06175F for ; Tue, 22 Jun 2021 09:05:40 -0700 (PDT) Received: by mail-qt1-x834.google.com with SMTP id x21so7242063qtq.9 for ; Tue, 22 Jun 2021 09:05:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=DZ3pXIZRpSkwolJsuMb/jDRoPgB3Tr5Ed0gJwqNhEn0=; b=bPcDo+rlbwvx6J+BbENOrwkAhwr/ApYX6dANLjI5IldJpUZEty9WP/A+WSRSkVnfVp fd2e4y/tXjqXDkPAhIKeQEV9eOqRiGPykVl6hBy9OPbjguGQbCXYh8BSABRpZ+0/loeK k5B7tYHxXO+OYAqc2XDEiodfylo30HXpiBw3o5xdgwyyFytUxxOswWBt61pJyEvA82JG jeswvAaYdQ80zAeU2QU/ihXizpJ995HOOdf81AhQWyQIhwQZLacM2brku76mBW2AwSnt rLC2zGxEtGcdZECkACvvAcUZ2O3xmGuKK/Qrbfiv04IEijyo0iWQN51AxkzmtGCV9rNa dlIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=DZ3pXIZRpSkwolJsuMb/jDRoPgB3Tr5Ed0gJwqNhEn0=; b=dsVdyS5pyCX1uA3TpP5Yv+1fvlAxwqCgJpkfinWQag4gRK0WyAcJLa9VOAHKrFshvQ GuK5rAPXXjymtEpkFbUo0n8YwTACna2kjPBjzEp9W+ILo60oXuebDoGR9MDqZVHvQKo1 eX3yK0YOaYLGavWa5gJ+ZmcbhtyIi7IFPLbguNnrf/Q6nTxemBYKFdvPlQL+GhuhisdC LbXECVWtZ5SQJPKxxc2Qar9OxRyROHieqBRl09hXbzGkwlBnbCeKSNoVATXozRpdrTBy DGVlDELGuz5R/+ii0c865BV7Z63DuDFZ+djAwi6/BfpmJdzB89T1cYTd724gyei54llL EB1A== X-Gm-Message-State: AOAM533R2qEi+6biPq+m6g+hi8KqZ0zZ+AGwAohzo0AmRD2/CfdDxPkn oxScvrH9X0a34JshSqGbpcC+uw== X-Google-Smtp-Source: ABdhPJwsZvQD0cl9e1oHhoP7iCcZiW+TSb8cEQVE7Z3CfQWJtDN54cMi/qDxEpHPFsGzcH9l1+8wSg== X-Received: by 2002:a05:622a:13cd:: with SMTP id p13mr4098685qtk.235.1624377939521; Tue, 22 Jun 2021 09:05:39 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-113-94.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.113.94]) by smtp.gmail.com with ESMTPSA id j7sm13254363qkd.21.2021.06.22.09.05.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 09:05:38 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lviuA-00ADrH-BE; Tue, 22 Jun 2021 13:05:38 -0300 Date: Tue, 22 Jun 2021 13:05:38 -0300 From: Jason Gunthorpe To: Christian =?utf-8?B?S8O2bmln?= Cc: Oded Gabbay , Christian =?utf-8?B?S8O2bmln?= , Gal Pressman , sleybo@amazon.com, linux-rdma , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Message-ID: <20210622160538.GT1096940@ziepe.ca> References: <20210621232912.GK1096940@ziepe.ca> <20210622120142.GL1096940@ziepe.ca> <20210622152343.GO1096940@ziepe.ca> <3fabe8b7-7174-bf49-5ffe-26db30968a27@amd.com> <20210622154027.GS1096940@ziepe.ca> <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jun 22, 2021 at 05:48:10PM +0200, Christian König wrote: > Am 22.06.21 um 17:40 schrieb Jason Gunthorpe: > > On Tue, Jun 22, 2021 at 05:29:01PM +0200, Christian König wrote: > > > [SNIP] > > > No absolutely not. NVidia GPUs work exactly the same way. > > > > > > And you have tons of similar cases in embedded and SoC systems where > > > intermediate memory between devices isn't directly addressable with the CPU. > > None of that is PCI P2P. > > > > It is all some specialty direct transfer. > > > > You can't reasonably call dma_map_resource() on non CPU mapped memory > > for instance, what address would you pass? > > > > Do not confuse "I am doing transfers between two HW blocks" with PCI > > Peer to Peer DMA transfers - the latter is a very narrow subcase. > > > > > No, just using the dma_map_resource() interface. > > Ik, but yes that does "work". Logan's series is better. > > No it isn't. It makes devices depend on allocating struct pages for their > BARs which is not necessary nor desired. Which dramatically reduces the cost of establishing DMA mappings, a loop of dma_map_resource() is very expensive. > How do you prevent direct I/O on those pages for example? GUP fails. > Allocating a struct pages has their use case, for example for exposing VRAM > as memory for HMM. But that is something very specific and should not limit > PCIe P2P DMA in general. Sure, but that is an ideal we are far from obtaining, and nobody wants to work on it prefering to do hacky hacky like this. If you believe in this then remove the scatter list from dmabuf, add a new set of dma_map* APIs to work on physical addresses and all the other stuff needed. Otherwise, we have what we have and drivers don't get to opt out. This is why the stuff in AMDGPU was NAK'd. Jason