From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S967878AbdAFQ4x (ORCPT ); Fri, 6 Jan 2017 11:56:53 -0500 Received: from mail-bn3nam01on0074.outbound.protection.outlook.com ([104.47.33.74]:42720 "EHLO NAM01-BN3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754290AbdAFQ4p (ORCPT ); Fri, 6 Jan 2017 11:56:45 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Serguei.Sagalovitch@amd.com; Subject: Re: Enabling peer to peer device transactions for PCIe devices To: Jerome Glisse , Jason Gunthorpe References: <20170105183927.GA5324@gmail.com> <20170105190113.GA12587@obsidianresearch.com> <20170105195424.GB2166@redhat.com> <20170105200719.GB31047@obsidianresearch.com> <20170105201935.GC2166@redhat.com> <20170105224215.GA3855@obsidianresearch.com> <20170105232352.GB6426@redhat.com> <20170106003034.GB4670@obsidianresearch.com> <20170106015831.GA2226@gmail.com> CC: Jerome Glisse , "Deucher, Alexander" , "'linux-kernel@vger.kernel.org'" , "'linux-rdma@vger.kernel.org'" , "'linux-nvdimm@lists.01.org'" , "'Linux-media@vger.kernel.org'" , "'dri-devel@lists.freedesktop.org'" , "'linux-pci@vger.kernel.org'" , "Kuehling, Felix" , "Blinzer, Paul" , "Koenig, Christian" , "Suthikulpanit, Suravee" , "Sander, Ben" , , , From: Serguei Sagalovitch Message-ID: Date: Fri, 6 Jan 2017 11:56:30 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170106015831.GA2226@gmail.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: BN6PR19CA0042.namprd19.prod.outlook.com (10.173.148.156) To BLUPR12MB0689.namprd12.prod.outlook.com (10.163.218.14) X-MS-Office365-Filtering-Correlation-Id: 8d4ed261-9c61-4f82-b3a2-08d43654fd1c X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:BLUPR12MB0689; X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0689;3:wtIaGR01WckSzRqbVeBHNkrMFnN2nQ5/3KWE0ObswYNULHy8OIbm6pwvfDTFVICPCH4zkRfEOzasnVPuOgxWO1J0bncRY7V0EllI+Sovf1OkI/lZCQjUINxIkmsi0yfz2NHevBdkdmuvmIhLN9iGLMXvyisHxWM6UcSnJGH5V0LUAFqWJCc2gRBT2JAIUhOuA6BRJFVxfOvw3xQxb9FHX+bXPxyG+beueyOa9fQjIvnoxZ8q8THuOnC30DV7UnphVId9zaQ+jq3fXXm3WwjThg==;25:xLLmGsV5VqEv/jiacm0ZetnSmDU2dk3IqSGmloFIiV8/tJHM+5M4CxOuUTWy4l4tFB19Nb8D/esbVgyRuQ4Cx6CMNxfhkVaVjNSb+YJ3I68SbInzozF1+LiHmVG1KHNoogSBlAq2qdtn2qYj8dsWKOwrOfHrIeWflZoHNaYYj0L19gPQA3xegAjBrU9ENr+3HE9PvR/O9WofqMX/fqV2ZSBjsabq0F1PRJvVzaWR6SB7ZLt9cHIx63flzPV0YXFYzN4J4EiNsbox4odOQwk+Ka3G+Y3wGWNyKUU1s6DvSfJY3zoL9wZ9WvVkaXe4N9YGVQUBXyCWBa9IHK+DMW5u430zMbD35Y5LXOAoKuER5YFrTh0DeWjcPdA5LQ/ZB/yLYn1gZZPcQLdpkmGNO73VlvL2uLRcrOEXrlomtycO4mBDnEXtUjBStMR3B0jDmxrD1GtvvtjRXRVdCG8vbTu5VA== X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0689;31:LAPRRN0wEonU5qz98f65l2L1vrtQHY8H92UZx/k4O0EvltfenK7rHpJESsIrUNKV1noWpXtNQGmI58cVEtCZSND7vukC3HS9EZhebn4uMnlNfv5++VLg4qW3zRo2oO5X2hB0He0cTrvNbAE47wy/WRYIs/3Q2p1qvCa3eswZPI5IzMpI5QMRgk8WmtfNJC0jrukcCxlwlCeqLcQ0fd0yqsEc8urQpHRcA3eGfusYTUx0ZtEt5MZ8A+sIoOQCQ5eq;20:CBFsJoCtLcBUs6YMHryrOBdav2Vr+zOp5auUV2K948sRoa2NPtQG8lNAwggMimTq/dcJ3Tlrqgr/vI9840Vek5Aou7DBdNwu+ygyHy5JTGrq8Rnqvxl+eEcpqWDO8FS2pwLgvLf0dy6+nSbndFaTe/5CVCPMjv8FWlUHy496zsNPk6jkF0bvQbrF164K7rFctJJicgBQLOzLp3TluNiyDlhP92A7l0SHeYXe/+6eIGBKt7aAUFYhF9Vo4KFci0xF1VLWYhH1uW64NjxwitOFiTdA6CZEymB+n1437fJhggVEPBFXhq1bpASxnqxrMeNR0uc6O/pvLT3M7Rn0lbh0bzhYgNFpSPDkysOfOoLtumgjK8ipC3X+x1xpb5Kp5RikUPrYr72ASwIy1m6hxaoD40ThvtcPcPP2IedxiC3kOlNUXqow3YPVPcfOcY+RYhfnBkUYQDtfsANSuyL9xpKlbmHhd1sTnQHqTD/wuhGM1tWM7l/kuFWDR+DkrY9KTEFt X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026)(6041248)(20161123564025)(20161123562025)(20161123555025)(20161123560025)(6072148);SRVR:BLUPR12MB0689;BCL:0;PCL:0;RULEID:;SRVR:BLUPR12MB0689; X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0689;4:nd2z0Izl9GF+kCmWf7XQeksz11vxxdm764GUOq+2D6VZwRQbXPovPtoqGH2GNVZmboiI880ZGm9joDCQ2eHtKrCBQW2jximUgi+vt8gI1hxeOneeEXYLN8E8MXAY2m+0Spok1r2sE8161wx8knhAMklUQMfSk4LXIVVb8cyYCH0l+tTIivSENQ63Oo9Xd4XhXMEBdFoAf84FUAuFCQ8PeQxoSwwWyWyReJWe6wTGMLBLvuGEzO4hJFnrKiKNHxAEhIAlImyMpfTszPKmyFi1Fkj4QyN8xKYA9Qo/+e6xC9H54F3xbSP+TJI0KDhV8d2Sxk0P4ItUgaYiS/+huWRlJrsJsQCjyn5oVSuAUShM8232GYexsB+PMobfxv0+UsbJejD9dLuutwDL3bQxyxWZZKZ+YeVFYq6dL3ztqB1L09d184dR/XDYL7hbhdDhn3tWX+zH3RdiyFTuc0mqYnK2nsP7vAHCbF5KYdIgiPZ0TyUQVBis62ARvsILAcys7IMNZNORyeFIwGZ3N388NUq0y/Q1VFMq3ZmzUFbrvu1dHDNXDJwnMHrzsDmSR/vXqEfJ9XfD6ZTLHCVLKk9pEitdGB2pS7MSDKJ/djh6Abg+gvaUDh4DtsNn3sXfvo8Gb+y9AMnP9JBzgjv4HIIUQsv2Mw== X-Forefront-PRVS: 01792087B6 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6049001)(6009001)(7916002)(39410400002)(39840400002)(39860400002)(39450400003)(39850400002)(24454002)(51444003)(377454003)(199003)(377424004)(189002)(31686004)(7736002)(106356001)(105586002)(3846002)(6116002)(42186005)(81156014)(81166006)(50986999)(68736007)(54356999)(76176999)(8676002)(54906002)(83506001)(189998001)(101416001)(7416002)(2906002)(5660300001)(65826007)(23746002)(97736004)(93886004)(4001350100001)(47776003)(31696002)(66066001)(65806001)(65956001)(5001770100001)(86362001)(25786008)(90366009)(77096006)(6486002)(36756003)(38730400001)(4326007)(2950100002)(230700001)(64126003)(92566002)(50466002)(305945005)(229853002)(39060400001)(6666003)(33646002);DIR:OUT;SFP:1101;SCL:1;SRVR:BLUPR12MB0689;H:[172.27.224.67];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BLUPR12MB0689;23:i33TkepNKdTIa/ClIdnXkqG41x968c1Mf1omi?= =?Windows-1252?Q?NjMv4JYz0LmRob37iDh00JtszWtP8kVqLjAqnZspQJg2tVxl2LDft9y0?= =?Windows-1252?Q?HpaIbIWieKIOY3nfIDeMCvLemkPry4duYzYeCHtKs1LC+PehCElwkGIS?= =?Windows-1252?Q?n7+DZ961B+8tDJIrxzEYvKm2+5POT17F26W8Wqr2jktTEVQvN3Yg5GxM?= =?Windows-1252?Q?cvCJAee4W56AXKbSTbwkOdtmsqN3vRCpgTk9ltUeluKSyZIiL1R4a6nS?= =?Windows-1252?Q?Ug4gFnnJGKfB1NoZIPsUZjdr3Cn+CFInbpYlX4LeZQdp/DG/lIHAT+uC?= =?Windows-1252?Q?O7WcJSMLNrSFgGrlB0PzGL8BB0Lncz4G91wpDNQ0EhHhrKEB3rhhqrjU?= =?Windows-1252?Q?zGE4Usz9BOQLvooiNmkvmksSLzRAd56eWvYlJad2SIPeTmSJd8lv+uxP?= =?Windows-1252?Q?71r4sJ3otro/owlhS9IeTnLtQWLCm2I4Kb9zdmBVmM/Fte0btAs76Ths?= =?Windows-1252?Q?oCUT9ubnSZcX2fE/ot+Dg1ia+vAEtnc9xCanONGaZmcHWoTnmaCkOI7T?= =?Windows-1252?Q?Dx8JVAz3rk+XWtIozPySjE24uCmD/E4xr0MvfUPvaEU1MnEU2e/CVYYa?= =?Windows-1252?Q?SF0haBWC4vsbB8yQR6gWqeHkEuRxa/rt2/N5TB2nb/RRWN8H5GoxkUqG?= =?Windows-1252?Q?32FtbPeuFQiRI1En3VvQqqIZfj2QKx3H71v1EqvBic4xMSVBrvnJj9rI?= =?Windows-1252?Q?vTrmHp2x9Db/RMzLn4Qjmc5oYbAxDaiuAB9RoShQE4ol2HgvG0SdYv6n?= =?Windows-1252?Q?tsnHDsOAvJ6kAKd3h07VrioehUTrFp8vXQozSKpz3YQ5yyFpsCrhctPP?= =?Windows-1252?Q?NCBxvdsMbucHqmL59VzXXOLh5C4R+8YeLquAeFSiIKdFYs1Y60nVQnRA?= =?Windows-1252?Q?ZtqJbEeQLuF1Tq/QjSNgwNPzYYAh/4al9CkZHUI6SDihQy8gwOSzlGqG?= =?Windows-1252?Q?Yn6qql5tCP0UAmQa+8Vj1uh1lQSRySIo3bJBqiaXqTkO3CYeEAmTwvvJ?= =?Windows-1252?Q?doSwNPS/FesvtuWk0hx7I17GmnGPQw8r9grbSVV9m8HDwXm4v+JZILae?= =?Windows-1252?Q?IYfurHZQKlG48IZt09GMzm77AqXDi+oQMYsKV4UGDn61fVRTfImxG7QY?= =?Windows-1252?Q?oyXTN7bTxj2B6dbQGYGuy7jqvseobWzNsy7dOGpNklaHqbcs1Kd7IS/W?= =?Windows-1252?Q?hnZI/ZbxsTIwnAjiZkx9BqJDZhtRs6gCb6Ut3D9NsS7dN8Qz/bKCovDc?= =?Windows-1252?Q?GEbTcVRB5icgMNdcCU/RAgROgFMA8+uYdsdkiJjII+Hxn91IORdEcxqP?= =?Windows-1252?Q?L3Qc4NKJeqW3H/Cpf0m0o065uxXba7Z1bsxtst9JRljAuB/H+szxfgBO?= =?Windows-1252?Q?fgpwuauwigKl0b/j8TudioIAVdcILfW+AD5AclwzZPykElhMuH9rOl1D?= =?Windows-1252?Q?dUKs94JrD2YpRzgxJ6ruiiBfvyarI70u2k1Cfvtsdc1LnMy2RJyekcD5?= =?Windows-1252?Q?nv3+Z0KTJ2FL8tSBfArSvggjR9nDtbHjNdoqEtDD7yy8P97dUGhxSfG2?= =?Windows-1252?Q?SPD3D+IVWLq5B5OLp5ku64Mj5loFISH4B6Iin6wPInnj2cYaaO9Xc3nc?= =?Windows-1252?Q?LabDogE2XemEtF5cdm0iOxB1aHeHJE=3D?= X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0689;6:7pSuKTJxQ4lNPnzJwLiT+/I1R+ZDVFJUaOXGgT1AQmnDFD8Xw76n6HKKHLOp12eR6LBJZOdMENo+MrzFB61Fr3rxxAL8l9tlh02d3fshVqM1ksj88qb9GYjZlaVkEw9P5xuiOIJUeQsQ3VzBQP5k75CtFISn9wfY3bj6QhNKrVod7X7DqnUEPhKADQO/2AgL8fKR54lwVgVWS5IXkZwnY9c5BSpJZ52f9sWTnCm1XvOV7vFp2GlRnhmTSxXi4v4aRb9QJWGhwewes+0OxMTW6Rvv3tr9bHoG2ohR2dyydRfUiPCbGQS3BzqVdjDh6VJ6IEx+d3rJ9I3cG8riytO++CGmt584/0amqCr+TBkSH5yHpkoGX6ERTEzCOp63LewM0UVpsJZTejZE7g3Fw+K6BCrCe1vLON0EdO7qDWNsMhq5elDS91tx5wOiPUB/BH19EiL+KVol4215sRuT7oHvMQ==;5:q8PmD1NaRRII3egMervSwdxqy1zCdCMp1WvXy5nQGQUCpCoVkP4Lnl18k3ruVH9oaZRFqa13W6eV/EzbqO4aTkrj12Q6kvmEUZQ1Sx28yGdYZyrkjtZSxUEY+ofiECsl4QYUQJ72hHBd7tAxNgXjEw==;24:HvtSkVPz8pXXhd94gP8Lx1ijxxOqd8Ov45ZYxn5fPsE1j2wnp4p5Qi+5pTDEQrd7GczN+YU59LszaFcrZOAy+1GACR1dlbIiHUXERZv/wEk= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BLUPR12MB0689;7:ky9EbkP+u4GVfqpOVouKgMjxGzT/y74iKercTePezWWYzkjDzY3QhSZrGMZHex6sPljTEFq0hQmi+7wOdWZEv5TNjJomIddKH0guVpqGF5bsWjRJj0WZHeKJ/aKEUjiA70Z/rXYYOqjnTd6AsGNNiSoXwUQ2CKgCl6jFEoGKGk6bX2BsAVJ6QeRIK4xRq+GyF6OsUK6o5OnwWLNddckHtoqSLUXZMW6cA3gePGue5DhQABR/ESDWYBuCNc6TCEc+qrL1rG0iuzUs4M8FSPz5yLdQbrCjcDf7dfW1qz8n7PC3WlBB5xAo6b/evLntWJC+FAB/lEUN/EkwZ2AujKf3Ej4IOj6Sr2SoflctovacrX/+bCVkfQM50MgfCzSqrbxqb14aaumiiz1OtsjGgggk97diIj/S8xRSr3bAN4M5x3t63yy3RQlCVU6v3IAlnCRqDu8Z71/2yxbLdZ+3Q/3Eeg==;20:1qgrfynkcbupsQ78at/dnTSWbl2qMQ3jGE67G/rVi9Xefx+k1Dwf2a0CTzenNuP6yFvxannj2KbUaI7eSssD6EEiA4DAR2UA1WEuOz3RhevJfz363IH6kxxqMD5xdmQAaKf99wb/6f2RiiHAvNzqUo2AmZBe6VbGHfa11hlbFQbYBiQx87qe5uOiDiPoxjX8IrRgnYNhHXwlFTXP0HKxJ1ZdxeVwDPAbmQJ92b1uE3WdgWsVAYDZJoIE8AJ9Ycbg X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jan 2017 16:56:41.0034 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR12MB0689 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2017-01-05 08:58 PM, Jerome Glisse wrote: > On Thu, Jan 05, 2017 at 05:30:34PM -0700, Jason Gunthorpe wrote: >> On Thu, Jan 05, 2017 at 06:23:52PM -0500, Jerome Glisse wrote: >> >>>> I still don't understand what you driving at - you've said in both >>>> cases a user VMA exists. >>> In the former case no, there is no VMA directly but if you want one than >>> a device can provide one. But such VMA is useless as CPU access is not >>> expected. >> I disagree it is useless, the VMA is going to be necessary to support >> upcoming things like CAPI, you need it to support O_DIRECT from the >> filesystem, DPDK, etc. This is why I am opposed to any model that is >> not VMA based for setting up RDMA - that is shorted sighted and does >> not seem to reflect where the industry is going. >> >> So focus on having VMA backed by actual physical memory that covers >> your GPU objects and ask how do we wire up the '__user *' to the DMA >> API in the best way so the DMA API still has enough information to >> setup IOMMUs and whatnot. > I am talking about 2 different thing. Existing hardware and API where you > _do not_ have a vma and you do not need one. This is just existing stuff. I do not understand why you assume that existing API doesn't need one. I would say that a lot of __existing__ user level API and their support in kernel (especially outside of graphics domain) assumes that we have vma and deal with __user * pointers. > Some close driver provide a functionality on top of this design. Question > is do we want to do the same ? If yes and you insist on having a vma we > could provide one but this is does not apply and is useless for where we > are going with new hardware. > > With new hardware you just use malloc or mmap to allocate memory and then > you use it directly with the device. Device driver can migrate any part of > the process address space to device memory. In this scheme you have your > usual VMAs but there is nothing special about them. Assuming that the whole device memory is CPU accessible and it looks like the direction where we are going: - You forgot about use case when we want or need to allocate memory directly on device (why we need to migrate anything if not needed?). - We may want to use CPU to access such memory on device to avoid any unnecessary migration back. - We may have more device memory than the system one. E.g. if you have 12 GPUs w/64GB each it will already give us ~0.7 TB not mentioning NVDIMM cards which could also be used as memory storage for other device access. - We also may want/need to share GPU memory between different processes. > Now when you try to do get_user_page() on any page that is inside the > device it will fails because we do not allow any device memory to be pin. > There is various reasons for that and they are not going away in any hw > in the planing (so for next few years). > > Still we do want to support peer to peer mapping. Plan is to only do so > with ODP capable hardware. Still we need to solve the IOMMU issue and > it needs special handling inside the RDMA device. The way it works is > that RDMA ask for a GPU page, GPU check if it has place inside its PCI > bar to map this page for the device, this can fail. If it succeed then > you need the IOMMU to let the RDMA device access the GPU PCI bar. > > So here we have 2 orthogonal problem. First one is how to make 2 drivers > talks to each other to setup mapping to allow peer to peer But I would assume and second is > about IOMMU. > I think that there is the third problem: A lot of existing user level API (MPI, IB Verbs, file i/o, etc.) deal with pointers to the buffers. Potentially it would be ideally to support use cases when those buffers are located in device memory avoiding any unnecessary migration / double-buffering. Currently a lot of infrastructure in kernel assumes that this is the user pointer and call "get_user_pages" to get s/g. What is your opinion how it should be changed to deal with cases when "buffer" is in device memory?