From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932145AbdGSLxT (ORCPT ); Wed, 19 Jul 2017 07:53:19 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:10323 "EHLO szxga02-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754504AbdGSLxQ (ORCPT ); Wed, 19 Jul 2017 07:53:16 -0400 Subject: Re: [PATCH 00/15] HMM (Heterogeneous Memory Management) v24 To: =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , , , References: <20170628180047.5386-1-jglisse@redhat.com> CC: John Hubbard , Dan Williams , David Nellans From: Yisheng Xie Message-ID: <19d4aa0e-a428-ed6d-c524-9b1cdcf6aa30@huawei.com> Date: Wed, 19 Jul 2017 19:48:08 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 In-Reply-To: <20170628180047.5386-1-jglisse@redhat.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.177.29.40] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020203.596F4769.00C1,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 4b5dafdbfac82287d58705970c68fc6e Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Jérôme On 2017/6/29 2:00, Jérôme Glisse wrote: > > Patchset is on top of git://git.cmpxchg.org/linux-mmotm.git so i > test same kernel as kbuild system, git branch: > > https://cgit.freedesktop.org/~glisse/linux/log/?h=hmm-v24 > > Change since v23 is code comment fixes, simplify kernel configuration and > improve allocation of new page on migration do device memory (last patch > in this patchset). > > Everything else is the same. Below is the long description of what HMM > is about and why. At the end of this email i describe briefly each patch > and suggest reviewers for each of them. > > > Heterogeneous Memory Management (HMM) (description and justification) > > Today device driver expose dedicated memory allocation API through their > device file, often relying on a combination of IOCTL and mmap calls. The > device can only access and use memory allocated through this API. This > effectively split the program address space into object allocated for the > device and useable by the device and other regular memory (malloc, mmap > of a file, share memory, â) only accessible by CPU (or in a very limited > way by a device by pinning memory). > > Allowing different isolated component of a program to use a device thus > require duplication of the input data structure using device memory > allocator. This is reasonable for simple data structure (array, grid, > image, â) but this get extremely complex with advance data structure > (list, tree, graph, â) that rely on a web of memory pointers. This is > becoming a serious limitation on the kind of work load that can be > offloaded to device like GPU. > > New industry standard like C++, OpenCL or CUDA are pushing to remove this > barrier. This require a shared address space between GPU device and CPU so > that GPU can access any memory of a process (while still obeying memory > protection like read only). This kind of feature is also appearing in > various other operating systems. > > HMM is a set of helpers to facilitate several aspects of address space > sharing and device memory management. Unlike existing sharing mechanism > that rely on pining pages use by a device, HMM relies on mmu_notifier to > propagate CPU page table update to device page table. > > Duplicating CPU page table is only one aspect necessary for efficiently > using device like GPU. GPU local memory have bandwidth in the TeraBytes/ > second range but they are connected to main memory through a system bus > like PCIE that is limited to 32GigaBytes/second (PCIE 4.0 16x). Thus it > is necessary to allow migration of process memory from main system memory > to device memory. Issue is that on platform that only have PCIE the device > memory is not accessible by the CPU with the same properties as main > memory (cache coherency, atomic operations, ...). > > To allow migration from main memory to device memory HMM provides a set > of helper to hotplug device memory as a new type of ZONE_DEVICE memory > which is un-addressable by CPU but still has struct page representing it. > This allow most of the core kernel logic that deals with a process memory > to stay oblivious of the peculiarity of device memory. > > When page backing an address of a process is migrated to device memory > the CPU page table entry is set to a new specific swap entry. CPU access > to such address triggers a migration back to system memory, just like if > the page was swap on disk. > [...] > To allow efficient migration between device memory and main memory a new > migrate_vma() helpers is added with this patchset. It allows to leverage > device DMA engine to perform the copy operation. > Is this means that when CPU access an address of a process is migrated to device memory, it should call migrate_vma() to migrate a range of address back to CPU ? If it is so, I think it should somewhere call this function in this patchset, however, I do not find anywhere in this patchset call this function. Or am I miss anything? Thanks Yisheng Xie