From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793686645; b=fGDDZW1q1LwKBXh/8VszPYpIbf+/bcg/pOJib/N1kS3tEYxF4j2aI8WXX3lE1K5gOZW/ rIpgnmQ3wy3kqu+MgQzTCpvNwEuofcTlqaL88U4tEDpOHgZJIxL7HOVP8uYzRAfZ+9pGr 16XofBkeZuj0iCSAbVR2fSJVErsHy8IoFKdZ1MHUk3K5FZifb5mWFDcvk7pTbYIQU/3Jf YtbcgfujtcmKLUzlmWpl8M11vV4iPR2rHVBWSCUvaB+eQIdDEVMhyrNfhfOE61NDLW6Wy ZAfqWYr4Lsze9c3cz/fNhP05CIbfSoVLRrlrf6r5MVaqETxfriftHsXI+3oKp2UgBDw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793686645; h=DMARC-Filter:DKIM-Signature:Received:X-MC-Unique: X-Mimecast-MFC-AGG-ID:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received: X-Received:Received:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:Content-Type: Content-Transfer-Encoding; bh=YJKtASFXSozFleqVLs7kYWFKCja/REnxHfhs5K4R3gw=; b=F3MOSsdXuKhjkb7r9e7Ag0bPg04HFGlMtdPIPBc+NSLaN9lFX80fbnznIzV3kTgetQNM xkpv0bnZR3qO8avjw6VxKXwGmQUkFbkn7Jh9uqJH7UACq1QQo4/jLoGctYK2lFoZmpFsj YV+DFLAvHuJPVEJtuEyvYGGq2KjYuLUVrb3k9pfiDiCZhu4GO4zAzAExjU5MGmk7fatYM E+7CNHc5WH3DDM16cZ/uPNbeA8htOO/PDIoDOpeluOExDSuGwkAfnTSEzv3NlwwDouC4s 70w7e2CDQLgwCXIecGJpDdx7TXeuiJjgINoAIxem67aAGVBB7id6vJXjuCMdxkQ05+Q== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=I8yVdndx; dkim=pass header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=google header.b=bpWpaHyZ; arc=none smtp.remote-ip=170.10.133.124 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4658606A329 for ; Tue, 3 Nov 2026 06:17:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org C4658606A329 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=I8yVdndx; dkim=pass (2048-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=google header.b=bpWpaHyZ DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1793686643; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YJKtASFXSozFleqVLs7kYWFKCja/REnxHfhs5K4R3gw=; b=I8yVdndxpPFESg6mVXwoCa9Kjh2UOqAzdHEgpDJS3E9QBNm4zgcQbmXBaLV/jrutn2VyIF SkqcytNufFXFAfAklt5kSjZYl7O+ABcjt1N6gPXLgiSqRBVelvsx7pAfWJhRo/XHlKWE4x yoG2G8iFUs3WTCFjWEJUaPusr+mbECo= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-618-OeYVDRS0O7ykyGTnr1m6YA-1; Tue, 03 Nov 2026 01:17:20 -0500 X-MC-Unique: OeYVDRS0O7ykyGTnr1m6YA-1 X-Mimecast-MFC-AGG-ID: OeYVDRS0O7ykyGTnr1m6YA_1793686639 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-5be3e0d9c34so2050370e87.3 for ; Mon, 02 Nov 2026 22:17:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1793686639; x=1794291439; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=YJKtASFXSozFleqVLs7kYWFKCja/REnxHfhs5K4R3gw=; b=bpWpaHyZGIK/4odJCtZFaAxDgoS+9dkLlQ3hdEr/xvrsKd+0OMxkQ/i1IXOS6H+Sqi F3aTaLuMwv8xNfBzaLIaEsfRaIApMQzn5sJiBCwrBUF8OvOnRy/HbwF0I8ObP19lkn4+ yUoeGkFPtZ/btjd+muu8NroK+Oz0ybchqMI9ytvW5cLrnk0NN6q+i3hsNV6sEobyl+CG Fi4X2YOG+171l5rBg8MSmo+vBODI9J1r4M3Ehc5cYSJwjOhIN4Zk7uItOImox1+ySPTA FFpjiYm1laAioQAx2NxZQaaMzSAniXtmCU/yXaCjdV1Lw006NAWi0N+fSkgSQgKdh5cu Ei0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793686639; x=1794291439; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=YJKtASFXSozFleqVLs7kYWFKCja/REnxHfhs5K4R3gw=; b=CnGN0zRy+4Jcv2PLNN+2wKxUbOtrxitQUf/0aeE241WWPAHMIT2GyIWOtRcSvcBBjE lUf3h2+cDuCo8Pga/9S0o73V0GAs95Ycxyi2krI/5xNgioyBTlGDXXkB0D6TUA6UxKPW oRKtM5ltmwe9sY6hLzyGab4UyjZu15JXiQo+SYkAfrNg4XD/dLGIMLP6mfV3fJdIzVeL HVp0tRWE2kgaCOaz7QXYu+7kO6R28SseGG5mzdGK5zoaTeh6m+L8czn0bmhgy2f1Pq/B i8ICJ+sVjoP8fwLwhpVDdbaCdgUy3wR/ng+ra2FGPqrHpbuTZvWZ80lUMGTJ5mgCw4a1 yY5g== X-Forwarded-Encrypted: i=1; AKwUvBwYKdL6a67XcEwwzi/caIJJ/Dn623RPe+R/+YZqL1yYB6NEFzfSsCfrXRCUhPWCnN/ZnPVf+ITUovArz68=@vger.kernel.org X-Gm-Message-State: APuU4Z8OMxaw0Joqqldvqg4sP0IkY/I66OuIEQXgD49yu6uCue33XlmQ Z8mT0qXd2KtZqDFfOjo0kpCXT5N1Or+7VbCS8DrntxNTMHDiZlgkVjQ2wPs8Q2FYltdXFkizCma o6prYpJnU6TteUpExk9tJRT6zxvWGo9v7uks/P63I3ttIGmIiiez1AhfGOMgCqhaF X-Gm-Gg: AYBFou3WIFAjiOaiETn3Q23ig3MMWXxTF5sPjewsiJdH6ilLS1QefqdVouowwCgcCNG Z6UGjKk286eUHISBaX01/Bqaa/6KHKwI6vGwSucy6rhxkvBl2DQ99l4RleICAyDiDEfYYpTTqEv FIFh/VvrrPYXZU5Qil8ST2KO5BDlwE/LcaNgP8ERbYuvGS83k74t24TOhLoa6J6ogVfWGd8pUuF B65lHxlbd8yPiHa8uD8GO++KNZYTHhfpLEu37Bju82MlM3FfbtditXvHR8fj6TEtJjplHkWkdCt 1c1/ZEHH1REuKVrfjfE8PQRRMXRcz0SFHvp7iv5nMHDFGxBZKXvjB1qEwg/ClU+200SIann+N27 3pygpx7COzRUPV0EtatjSZeC5mpeQb9rcWtQLblwcTNA4OespoS+fFTEHOx5rL6wxMdz/bw== X-Received: by 2002:a05:6512:3a84:b0:5bf:be4d:13d0 with SMTP id 2adb3069b0e04-5bfd956226emr3849923e87.19.1793686638711; Mon, 02 Nov 2026 22:17:18 -0800 (PST) X-Received: by 2002:a05:6512:3a84:b0:5bf:be4d:13d0 with SMTP id 2adb3069b0e04-5bfd956226emr3849909e87.19.1793686638234; Mon, 02 Nov 2026 22:17:18 -0800 (PST) Received: from fedora (89-27-86-246.bb.dnainternet.fi. [89.27.86.246]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5bfe073e0f9sm1964634e87.54.2026.11.02.22.17.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Nov 2026 22:17:17 -0800 (PST) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Lyude Paul , Danilo Krummrich , Marco Pagani Subject: [PATCH v17 10/12] lib/test_hmm: add a new testcase for the migrate on fault Date: Tue, 3 Nov 2026 08:16:27 +0200 Message-ID: <20261103061629.972669-11-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261103061629.972669-1-mpenttil@redhat.com> References: <20261103061629.972669-1-mpenttil@redhat.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: Mika Penttilä Enhance the hmm test driver (lib/test_hmm) with a migrate on fault case: a single hmm_range_fault() walk with HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE faults in and collects the range, which is then migrated to device private memory and mapped into the dmirror page table. The selftest leaves the first two pages unwritten, so page 0 has to be faulted in and page 1 maps the zero page. If mmu_interval_read_retry() trips after migrate_vma_pages(), the dmirror mapping is skipped and the -EBUSY retry re-walks. Normally the trip comes from an unmap or zap: the migration ptes are gone, migrate_vma_finalize() releases the destination pages and the retry migrates again. Pages committed to device memory without a dmirror mapping (e.g. after a huge zero pmd split in migrate_vma_insert_page()) are still consistent, since any invalidation also clears the dmirror page table, and a real device would map them on its next fault. The selftest doesn't hit that case: the unwritten pages share a pmd with written ones, so there is no huge zero pmd. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Signed-off-by: Marco Pagani Signed-off-by: Mika Penttilä --- lib/test_hmm.c | 172 +++++++++++++++++++++++-- lib/test_hmm_uapi.h | 21 +-- tools/testing/selftests/mm/hmm-tests.c | 56 ++++++++ 3 files changed, 226 insertions(+), 23 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index f05f801b8404..2045896493ae 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -36,6 +36,7 @@ #define DMIRROR_RANGE_FAULT_TIMEOUT 1000 #define DEVMEM_CHUNK_SIZE (256 * 1024 * 1024U) #define DEVMEM_CHUNKS_RESERVE 16 +#define PFNS_ARRAY_SIZE 64 /* * For device_private pages, dpage is just a dummy struct page @@ -355,6 +356,7 @@ static int dmirror_range_fault(struct dmirror *dmirror, struct mm_struct *mm = dmirror->notifier.mm; unsigned long timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + bool migrate = range->default_flags & HMM_PFN_REQ_MIGRATE; int ret; while (true) { @@ -364,9 +366,15 @@ static int dmirror_range_fault(struct dmirror *dmirror, } range->notifier_seq = mmu_interval_read_begin(range->notifier); - mmap_read_lock(mm); - ret = hmm_range_fault(range); - mmap_read_unlock(mm); + + /* mmap lock held for whole migrate on fault operation */ + if (!migrate) { + mmap_read_lock(mm); + ret = hmm_range_fault(range); + mmap_read_unlock(mm); + } else { + ret = hmm_range_fault(range); + } if (ret) { if (ret == -EBUSY) continue; @@ -382,7 +390,9 @@ static int dmirror_range_fault(struct dmirror *dmirror, break; } - ret = dmirror_do_fault(dmirror, range); + /* update device page table after migration */ + if (!migrate) + ret = dmirror_do_fault(dmirror, range); mutex_unlock(&dmirror->mutex); out: @@ -994,8 +1004,12 @@ static int dmirror_atomic_map(unsigned long addr, struct page *page, return 0; } -static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, - struct dmirror *dmirror) +/* + * Map the migrated pages into the device's page table. Caller must hold + * dmirror->mutex. + */ +static int __dmirror_migrate_map_locked(struct migrate_vma *args, + struct dmirror *dmirror) { unsigned long start = args->start; unsigned long end = args->end; @@ -1005,9 +1019,6 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, const unsigned long start_pfn = start >> PAGE_SHIFT; const unsigned long end_pfn = end >> PAGE_SHIFT; - /* Map the migrated pages into the device's page tables. */ - mutex_lock(&dmirror->mutex); - for (pfn = start_pfn; pfn < end_pfn; pfn++, src++, dst++) { struct page *dpage; void *entry; @@ -1036,17 +1047,27 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, if (*dst & MIGRATE_PFN_WRITE) entry = xa_tag_pointer(entry, DPT_XA_TAG_WRITE); entry = xa_store(&dmirror->pt, pfn + i, entry, GFP_ATOMIC); - if (xa_is_err(entry)) { - mutex_unlock(&dmirror->mutex); + if (xa_is_err(entry)) return xa_err(entry); - } } } - mutex_unlock(&dmirror->mutex); return 0; } +static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, + struct dmirror *dmirror) +{ + int ret; + + /* Map the migrated pages into the device's page tables. */ + mutex_lock(&dmirror->mutex); + ret = __dmirror_migrate_map_locked(args, dmirror); + mutex_unlock(&dmirror->mutex); + + return ret; +} + static int dmirror_exclusive(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) { @@ -1377,6 +1398,127 @@ static int dmirror_migrate_to_device(struct dmirror *dmirror, return ret; } +static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range *range) +{ + struct migrate_vma *migrate = range->migrate; + unsigned long timeout = + jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + int ret; + + do { + mmap_read_lock(dmirror->notifier.mm); + + /* Fault-in pages for migration */ + ret = dmirror_range_fault(dmirror, range); + /* + * Set this up even on error: hmm_range_fault() may have collected + * part of the range before failing. + */ + migrate_hmm_range_setup(range); + if (ret) { + /* + * dst[] entries are empty, so this marks every collected + * migration as failed. finalize then restores the source + * mappings and drops the associated locks/references. + */ + migrate_vma_pages(migrate); + migrate_vma_finalize(migrate); + goto unlock; + } + + pr_debug("Migrating from sys mem to device mem\n"); + + dmirror_migrate_alloc_and_copy(migrate, dmirror); + migrate_vma_pages(migrate); + + mutex_lock(&dmirror->mutex); + if (mmu_interval_read_retry(&dmirror->notifier, + range->notifier_seq)) + ret = -EBUSY; + else + ret = __dmirror_migrate_map_locked(migrate, dmirror); + mutex_unlock(&dmirror->mutex); + + migrate_vma_finalize(migrate); +unlock: + mmap_read_unlock(dmirror->notifier.mm); + } while (ret == -EBUSY && !time_after(jiffies, timeout)); + + return ret; +} + +static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror, + struct hmm_dmirror_cmd *cmd) +{ + unsigned long start, size, end, next; + unsigned long src_pfns[PFNS_ARRAY_SIZE] = { 0 }; + unsigned long dst_pfns[PFNS_ARRAY_SIZE] = { 0 }; + struct migrate_vma migrate = { 0 }; + struct hmm_range range = { 0 }; + struct dmirror_bounce bounce; + int ret = 0; + + /* Whole range */ + start = cmd->addr; + size = cmd->npages << PAGE_SHIFT; + end = start + size; + + if (!mmget_not_zero(dmirror->notifier.mm)) { + ret = -EFAULT; + goto out; + } + + migrate.pgmap_owner = dmirror->mdevice; + migrate.src = src_pfns; + migrate.dst = dst_pfns; + migrate.flags = MIGRATE_VMA_SELECT_SYSTEM; + + range.migrate = &migrate; + range.hmm_pfns = src_pfns; + range.pfn_flags_mask = 0; + range.default_flags = HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE; + range.dev_private_owner = dmirror->mdevice; + range.notifier = &dmirror->notifier; + + for (next = start; next < end; next = range.end) { + range.start = next; + range.end = min(end, next + (PFNS_ARRAY_SIZE << PAGE_SHIFT)); + + pr_debug("Fault and migrate range start:%#lx end:%#lx\n", + range.start, range.end); + + ret = do_fault_and_migrate(dmirror, &range); + if (ret) + goto out_mmput; + } + + /* + * Return the migrated data for verification. + * Only for pages in device zone + */ + ret = dmirror_bounce_init(&bounce, start, size); + if (ret) + goto out_mmput; + + mutex_lock(&dmirror->mutex); + ret = dmirror_do_read(dmirror, start, end, &bounce); + mutex_unlock(&dmirror->mutex); + if (ret == 0) { + ret = copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, bounce.size); + if (ret) + ret = -EFAULT; + } + + cmd->cpages = bounce.cpages; + dmirror_bounce_fini(&bounce); + + +out_mmput: + mmput(dmirror->notifier.mm); +out: + return ret; +} + static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *range, unsigned char *perm, unsigned long entry) { @@ -1645,6 +1787,10 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp, ret = dmirror_migrate_to_device(dmirror, &cmd); break; + case HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV: + ret = dmirror_fault_and_migrate_to_device(dmirror, &cmd); + break; + case HMM_DMIRROR_MIGRATE_TO_SYS: ret = dmirror_migrate_to_system(dmirror, &cmd); break; diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h index ea9b0ec404fb..67d267a3b999 100644 --- a/lib/test_hmm_uapi.h +++ b/lib/test_hmm_uapi.h @@ -29,16 +29,17 @@ struct hmm_dmirror_cmd { }; /* Expose the address space of the calling process through hmm device file */ -#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV _IOWR('H', 0x0a, struct hmm_dmirror_cmd) #define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0) diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c index e5f273ca84c1..bacf573d83a9 100644 --- a/tools/testing/selftests/mm/hmm-tests.c +++ b/tools/testing/selftests/mm/hmm-tests.c @@ -333,6 +333,13 @@ static int hmm_migrate_sys_to_dev(int fd, return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_TO_DEV, buffer, npages); } +static int hmm_migrate_on_fault_sys_to_dev(int fd, + struct hmm_buffer *buffer, + unsigned long npages) +{ + return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV, buffer, npages); +} + static int hmm_migrate_dev_to_sys(int fd, struct hmm_buffer *buffer, unsigned long npages) @@ -937,6 +944,55 @@ TEST_F(hmm, migrate) hmm_buffer_free(buffer); } + +/* + * Fault and migrate anonymous memory to device private memory. + */ +TEST_F(hmm, migrate_on_fault) +{ + struct hmm_buffer *buffer; + unsigned long npages; + unsigned long size; + unsigned long i; + int *ptr; + int ret; + int val; + + npages = ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift; + ASSERT_NE(npages, 0); + size = npages << self->page_shift; + + buffer = hmm_buffer_alloc(size, size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1); + ASSERT_NE(buffer, NULL); + + /* + * Initialize buffer in system memory but leave the first two pages + * zero: page 0 stays pte_none so that hmm_range_fault() has to fault + * it in, page 1 is mapped to the zero page. + */ + i = 2 * self->page_size / sizeof(*ptr); + for (ptr = buffer->ptr; i < size / sizeof(*ptr); ++i) + ptr[i] = i; + + val = *(int *)(buffer->ptr + self->page_size); + ASSERT_EQ(val, 0); + + /* Fault and migrate memory to device. */ + ret = hmm_migrate_on_fault_sys_to_dev(self->fd, buffer, npages); + ASSERT_EQ(ret, 0); + ASSERT_EQ(buffer->cpages, npages); + + /* Check what the device read. */ + for (i = 0, ptr = buffer->mirror; i < 2 * self->page_size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], 0); + for (; i < size / sizeof(*ptr); ++i) + ASSERT_EQ(ptr[i], i); + + hmm_buffer_free(buffer); +} + /* * Migrate private file memory to device private memory. */ -- 2.55.0