From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D003C43465 for ; Fri, 18 Sep 2020 13:59:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0310121D42 for ; Fri, 18 Sep 2020 13:59:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="dZS+Pz8u" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726931AbgIRN7S (ORCPT ); Fri, 18 Sep 2020 09:59:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726253AbgIRN7R (ORCPT ); Fri, 18 Sep 2020 09:59:17 -0400 Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28A34C0613CF for ; Fri, 18 Sep 2020 06:59:17 -0700 (PDT) Received: by mail-qt1-x842.google.com with SMTP id z2so4969258qtv.12 for ; Fri, 18 Sep 2020 06:59:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=iDzy1sMgIGxVws9oU0uRMBwks6chpSBoUJBA5ZAPs6Y=; b=dZS+Pz8uPa7xZedC6ywyuba3H9rxNa8iD2FLUWjM9LYjV/vp0Nnwuxl4sIAH/WzJmo Z71YmWaPcUeza3JssOVlprHU4cGQ3yjZxah7wyneWO2mh86SvelVUNGd45Cqk9Yie1oG 617Nev2Z7twgiRUh132ng7fDDjipACTM4bU0rYD4M+AA9ZUYUyf0U4tF3fwh8UPfh7cR ijQl8xM6r/btbx2DzAHfRisLO48r+cfseEBb/dQ/NLLKA9FIWvNEWjokBXGDc2fndUqW ScKssfiqwnp2XUoY0uD9YyQ4SnhhADwwA+/HFxDHtK2d76LmBhaSyZ9tPLqOZUnoOxnr 5sUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=iDzy1sMgIGxVws9oU0uRMBwks6chpSBoUJBA5ZAPs6Y=; b=iTEUPUa+r0GpBrGEIeFEZZtq1+fyJ7niCn/17XlbNEUJWk/L09tpMWbWd+sExqYyZH yGgdbRkxFUYng6ToTgK9huc9PToDWl55r1TWvAIgSI6GdAdwmKrMDZctjq/57P2ye9lv nQs8IarDEg5LSYYlTgvifI9VvnoooaH79FK+UkzVcPWs5Zl0v20xSevePoGHsBphK2QY iH1Y8VbLvD4P24+RHOjv+0613O+/6cLtsb0+DGRFJk2wR9auzNPhIvbGlByd2McrCTzK K4MIod3MW6OWCnf+fynCTdwrIz7eq7gQpr8F4eIMFevNPRhAhLmp3AqaGHs2zxaMNZfn g/9g== X-Gm-Message-State: AOAM5314GucVaHgAdsbO8EQgwYG7HyIpiYuE+L3Q8h+mDyr/kkq6pBZh cQbAhq2KIqCFXRXMj2PVOo++Fg== X-Google-Smtp-Source: ABdhPJxJEjIjl7tbKTQubndOek+3OibCxEDhsqhs1uDEM/H/kHLq/g77HjdghQ/1oNTy7EDlsPqX1w== X-Received: by 2002:ac8:6e90:: with SMTP id c16mr33479531qtv.17.1600437556361; Fri, 18 Sep 2020 06:59:16 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id v18sm2144966qtq.15.2020.09.18.06.59.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2020 06:59:15 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kJGux-001HHG-3h; Fri, 18 Sep 2020 10:59:15 -0300 Date: Fri, 18 Sep 2020 10:59:15 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: Greg Kroah-Hartman , izur@habana.ai, Gal Pressman , Jakub Kicinski , "Linux-Kernel@Vger. Kernel. Org" , netdev@vger.kernel.org, SW_Drivers , "David S. Miller" , Andrew Lunn , Florian Fainelli , linux-rdma@vger.kernel.org, Olof Johansson Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Message-ID: <20200918135915.GT8409@ziepe.ca> References: <20200917171833.GJ8409@ziepe.ca> <0b21db8d-1061-6453-960b-8043951b3bad@amazon.com> <20200918115601.GP8409@ziepe.ca> <20200918121621.GQ8409@ziepe.ca> <20200918125014.GR8409@ziepe.ca> <20200918132645.GS8409@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Sep 18, 2020 at 04:49:25PM +0300, Oded Gabbay wrote: > On Fri, Sep 18, 2020 at 4:26 PM Jason Gunthorpe wrote: > > > > On Fri, Sep 18, 2020 at 04:02:24PM +0300, Oded Gabbay wrote: > > > > > The problem with MR is that the API doesn't let us return a new VA. It > > > forces us to use the original VA that the Host OS allocated. > > > > If using the common MR API you'd have to assign a unique linear range > > in the single device address map and record both the IOVA and the MMU > > VA in the kernel struct. > > > > Then when submitting work using that MR lkey the kernel will adjust > > the work VA using the equation (WORK_VA - IOVA) + MMU_VA before > > forwarding to HW. > > > We can't do that. That will kill the performance. If for every > submission I need to modify the packet's contents, the throughput will > go downhill. You clearly didn't read where I explained there is a fast path and slow path expectation. > Also, submissions to our RDMA qmans are coupled with submissions to > our DMA/Compute QMANs. We can't separate those to different API calls. > That will also kill performance and in addition, will prevent us from > synchronizing all the engines. Not sure I see why this is a problem. I already explained the fast device specific path. As long as the kernel maintains proper security when it processes submissions the driver can allow objects to cross between the two domains. Jason