From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE1AC49EA4 for ; Tue, 22 Jun 2021 15:11:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 878EF6128C for ; Tue, 22 Jun 2021 15:11:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232097AbhFVPOC (ORCPT ); Tue, 22 Jun 2021 11:14:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230047AbhFVPOA (ORCPT ); Tue, 22 Jun 2021 11:14:00 -0400 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08B1BC06175F for ; Tue, 22 Jun 2021 08:11:45 -0700 (PDT) Received: by mail-qt1-x835.google.com with SMTP id d9so6832810qtx.8 for ; Tue, 22 Jun 2021 08:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=GfBcan5Oq/+F6rqd3wteDDUo+4mA0t9y3pSH6r/kVdI=; b=oQKSqDMY5ui7ysN6RXa17KyJ2Wzpm89WOS/VTDpLsBMmjFt0xKup1/rN3Iu2Ysn7kj 0o4QpiviU+XCzwAjOmd3ASe2G50DR4wQM1cxJ/IcnSoJE+UREjasIpFci4SAI/zjLeO4 BaVYbbQSqm0nAX/Y00oQm+1Fl/s1cHNEiBtnrRSnwG+/c0ZPfPFGw36xuMKNGuUPMk6/ S9czK9NLULrCof4fVnQi6scmnRJl9iMhmlDxM2xpQloNB9qi5QwwBnyh0jFdiD4S4nEx BNdzQik55X4yiVc76B5/N6HeFax4G1G02Yo0H2X2M1eLLt5C2jCamVulCHYNUMYd6Nkl GXvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=GfBcan5Oq/+F6rqd3wteDDUo+4mA0t9y3pSH6r/kVdI=; b=edqcncEgQHPRDCMUzf3yfz6YSWDQgRD+UM16ylv3bbsEgz+wiSXbqQHeiyeZHbUpf0 PWd/pYvxvpcLQfhKspIUWsysyofuWQlZPHKTssGAEenJcmQN4XcL/1ag11DuSLbQ6OYj 69bjdaUP0zo0gFk9JCdRLMpcb+2ZBwJsjPeNEE9TtclOyueIl8oWzQ+QyT5saklEt+M0 x9JW8TO/GwFNw3zCk5A0Mgk3IMDCrXCq134Veo8peGaw8TXZ/fDI8uzi8lMB7eDdwypA e+NlZ6VFZEBt5tHZY+3+5KAeXJbKqD9FrGU3c+OeFmF5q3llWFGbvgI81BYjeG9+aR2w 8nlQ== X-Gm-Message-State: AOAM532lJvTge0U9Yjx6ft+H5B19alJjJKFb9rb/faFaRat/BBCBKlN/ ewF77xaZFoYSGOxBUydrd6zdww== X-Google-Smtp-Source: ABdhPJyIvo3ZVkwIuidF8Fxvcg1uTF/rMIUzGf5xwZlLSDUGqkiRqy7l/P12S4XnoNrAjy14Ik1xNg== X-Received: by 2002:ac8:5dd2:: with SMTP id e18mr3979783qtx.263.1624374704154; Tue, 22 Jun 2021 08:11:44 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-113-94.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.113.94]) by smtp.gmail.com with ESMTPSA id m3sm10504480qkk.27.2021.06.22.08.11.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 08:11:43 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lvi3y-00AD4o-Si; Tue, 22 Jun 2021 12:11:42 -0300 Date: Tue, 22 Jun 2021 12:11:42 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: Christian =?utf-8?B?S8O2bmln?= , Gal Pressman , sleybo@amazon.com, linux-rdma , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , Christian =?utf-8?B?S8O2bmln?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Message-ID: <20210622151142.GA2431880@ziepe.ca> References: <20210621175511.GI1096940@ziepe.ca> <20210621232912.GK1096940@ziepe.ca> <20210622120142.GL1096940@ziepe.ca> <20210622121546.GN1096940@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jun 22, 2021 at 04:12:26PM +0300, Oded Gabbay wrote: > > 1) Setting sg_page to NULL > > 2) 'mapping' pages for P2P DMA without going through the iommu > > 3) Allowing P2P DMA without using the p2p dma API to validate that it > > can work at all in the first place. > > > > All of these result in functional bugs in certain system > > configurations. > > > > Jason > > Hi Jason, > Thanks for the feedback. > Regarding point 1, why is that a problem if we disable the option to > mmap the dma-buf from user-space ? Userspace has nothing to do with needing struct pages or not Point 1 and 2 mostly go together, you supporting the iommu is not nice if you dont have struct pages. You should study Logan's patches I pointed you at as they are solving exactly this problem. > In addition, I didn't see any problem with sg_page being NULL in the > RDMA p2p dma-buf code. Did I miss something here ? No, the design of the dmabuf requires the exporter to do the dma maps and so it is only the exporter that is wrong to omit all the iommu and p2p logic. RDMA is OK today only because nobody has implemented dma buf support in rxe/si - mainly because the only implementations of exporters don't set the struct page and are thus buggy. > I will take two GAUDI devices and use one as an exporter and one as an > importer. I want to see that the solution works end-to-end, with real > device DMA from importer to exporter. I can tell you it doesn't. Stuffing physical addresses directly into the sg list doesn't involve any of the IOMMU code so any configuration that requires IOMMU page table setup will not work. Jason