From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFFE23A452D for ; Tue, 17 Mar 2026 10:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773743313; cv=none; b=bNpdFRBbRRzN8X1Wj98SdhfOtEn1Oje9PQAgPMfIspBvJnAfG4o0IN4UK/5RKJH1qJhbBzO9LW4t/QNq8S3weKmQDkG0KOQVy6doiAozHsGBIQRXBQIaLexiR8Zyl5lPKJHkCTlHxjy8OnMeuJ5m6R18rmwV+kC00cCmDpAG9d8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773743313; c=relaxed/simple; bh=+w95Ye0EjbRqDvfbzPByRgK6Y7uCqrIjhZi0S31b5Xs=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=RWo5fydmgdVlcRso/5fTGvFGp2GL+VsAboiejv3yST1GREwQS+NX0RRQ0DJX4zz0z6tIuJFs9ECJ0C0xhJpruPVP/GRhd8vU2JoEYUcPmm20eGIonI3c7QxTuyMxFhkBFEi2PTqObFbvKsJcYgP3GVTMnEPLOcqZzCyb/TmzLKg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VJyK3294; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=Xszh2uGr; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VJyK3294"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="Xszh2uGr" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1773743310; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7WLfM9OhXfwsNZO2Z5cPG4Mf2ehRGF5Bqst0QMfR6J8=; b=VJyK3294rZrCzO5sVBQCCJDr5x+ucdMOOiEXA5VpeLAcxvSx5K0pp3EdRaHmKG/uwZZNPz dBwRbbliFt+YrKVF0gZc3Xm3c/aifdaqblGLXuZHm5IJgEzwjNxBbtsTVe7ExSjltiltyo zwopqu/5yoIcSwv1rkNMVuBmN4yZNsk= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-526-WdUjlh_cNXSqj6bR9cKi_Q-1; Tue, 17 Mar 2026 06:28:29 -0400 X-MC-Unique: WdUjlh_cNXSqj6bR9cKi_Q-1 X-Mimecast-MFC-AGG-ID: WdUjlh_cNXSqj6bR9cKi_Q_1773743308 Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-5a277331b57so236263e87.1 for ; Tue, 17 Mar 2026 03:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1773743308; x=1774348108; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=7WLfM9OhXfwsNZO2Z5cPG4Mf2ehRGF5Bqst0QMfR6J8=; b=Xszh2uGrX+8LbLt1mTO0wLP0i9uj8mzKdVadioKHKDCBGMG40+ztZAkG4Ysy/ouYSE +947w/57lBccbQQ6KmvINdbcQQrRLR6PY5GRpVIXKTD2dqOtyQdj2eHusWtnClnZ2GAZ by/KMJL5Hmt5HMSkGVUg1NSX7IuIV9/ESJ4/baTV+6WTMQ9csDKBKr5haZEaKEacRjg7 ByItEWrYjfngr7Sx0RXjW1RBm8+lGoAHhhaWQKgcF0abpEUIh42rwdaWbP0urERSWlrn Ji4ddV3mXZHqAvg2kr2mcLOrfxV+NxhDQBxnWo9dE7aikUsH1fYecQz2zIf/+dqIk4AV vsSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773743308; x=1774348108; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7WLfM9OhXfwsNZO2Z5cPG4Mf2ehRGF5Bqst0QMfR6J8=; b=TUV0uM+XAuF1S7q8ZJCvyLoNps0vxXxWUdYx6ei+LcgtlLLFtrV9BDeTxp+Q2ANOwa SzzFfQ6ZucrTfIo3efztD+wIwl+aIXf2GwJNsS+4vX7d1po55/hPtxHmnryT0H9ZTEr+ IIhqVg6VFTZbJpYAUz+wbyh1XO0BtENCIJVDl0wQQ9nrBUWfbc+ME5Rrh65vG1kra8C+ 5XXrUqfkS33T2UUVaZeb+uPKV2AyIO2iDwPVBXyG9bF6aQNg5JsM2l9U3CwajtxQKS0f SQVz+hzXTjaR8Es2FkL8CENp9k7usBbf4tkStUQWX8ZGJR1nsGFjtIgERLo5Znzy1eqX ibEA== X-Gm-Message-State: AOJu0YyQfs1sdmcT1ES8+T2fpr/nj2pIJm6IuktwaaLGZgThQxXWaSas xe0o/2kVc5KTkosATzYM3xHHqcQvfFZJco3di2ALRRLnMcd0+f+Kul/vqSXeiMNGVe6vpsdT2KX YUVNB7wPeseJ4CHVjeGbBVTFQnSi7zi+rCLgxKNxLNmT6uxhrtJ3wrDsg37dfmUsy X-Gm-Gg: ATEYQzzwBXKqENJ6XEN9d59q3mecRq5cctm6/KGpxIUZdWiWvt/59WhEX/LJwiW8Stw qX5WmOVBPEkbn92DXkOaD5FGsJacEeY6LsLyfKwQtylOA76XXoZfl5Io53O07hOI54gN4DCyZok BJoxgeMrxJuI1YeCYMqfg9DsdxAI3O8XvNDWzQfLalD3eJicktpQFqWyoU8a53NONU7dXP4ZwYF z1LRPUPLZ+zN6g8Ie28AX2YpGYGb0KV/z/DBXad5Y0hN0+9m8eORJtJ1Ce6JwSLskbp+F9A2es6 46DERi3yQg70eA+rAtrWdaVfXVqAxFv+3rNk3nY1ve2IQQkDh3tE+ojAq/7Sk7lUc2x+xzIKgep hE9jRhZvN0gaTaWmF95NkVp8De0wL5fpD1C+vapG0xwqH8cI= X-Received: by 2002:ac2:5b1e:0:b0:5a1:343d:8177 with SMTP id 2adb3069b0e04-5a1626fdddbmr3853592e87.11.1773743307701; Tue, 17 Mar 2026 03:28:27 -0700 (PDT) X-Received: by 2002:ac2:5b1e:0:b0:5a1:343d:8177 with SMTP id 2adb3069b0e04-5a1626fdddbmr3853576e87.11.1773743307143; Tue, 17 Mar 2026 03:28:27 -0700 (PDT) Received: from [192.168.1.86] (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5a15603407csm4024915e87.38.2026.03.17.03.28.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 17 Mar 2026 03:28:26 -0700 (PDT) Message-ID: <412cb576-397d-4f02-b91e-ccb9075dad48@redhat.com> Date: Tue, 17 Mar 2026 12:28:26 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 0/6] Migrate on fault for device pages To: Balbir Singh , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Zi Yan , Matthew Brost References: <20260211081301.2940672-1-mpenttil@redhat.com> <856095a1-3784-4e70-acfc-888d2a8027ae@nvidia.com> Content-Language: en-US From: =?UTF-8?Q?Mika_Penttil=C3=A4?= In-Reply-To: <856095a1-3784-4e70-acfc-888d2a8027ae@nvidia.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi, On 3/17/26 12:06, Balbir Singh wrote: > On 2/11/26 19:12, mpenttil@redhat.com wrote: >> From: Mika Penttilä >> >> Currently, the way device page faulting and migration works >> is not optimal, if you want to do both fault handling and >> migration at once. >> >> Being able to migrate not present pages (or pages mapped with incorrect >> permissions, eg. COW) to the GPU requires doing either of the >> following sequences: >> >> 1. hmm_range_fault() - fault in non-present pages with correct permissions, etc. >> 2. migrate_vma_*() - migrate the pages >> >> Or: >> >> 1. migrate_vma_*() - migrate present pages >> 2. If non-present pages detected by migrate_vma_*(): >> a) call hmm_range_fault() to fault pages in >> b) call migrate_vma_*() again to migrate now present pages >> >> The problem with the first sequence is that you always have to do two >> page walks even when most of the time the pages are present or zero page >> mappings so the common case takes a performance hit. >> >> The second sequence is better for the common case, but far worse if >> pages aren't present because now you have to walk the page tables three >> times (once to find the page is not present, once so hmm_range_fault() >> can find a non-present page to fault in and once again to setup the >> migration). It is also tricky to code correctly. One page table walk >> could costs over 1000 cpu cycles on X86-64, which is a significant hit. >> >> We should be able to walk the page table once, faulting >> pages in as required and replacing them with migration entries if >> requested. >> >> Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE, >> which tells to prepare for migration also during fault handling. >> Also, for the migrate_vma_setup() call paths, a flag, MIGRATE_VMA_FAULT, >> is added to tell to add fault handling to migrate. >> >> One extra benefit of migrating with hmm_range_fault() path >> is the migrate_vma.vma gets populated, so no need to >> retrieve that separataly. >> >> Tested in X86-64 VM with HMM test device, passing the selftests. >> For performance, the migrate throughput tests from the selftests >> show similar numbers (within error margin) as unmodified kernel. >> Tested also rebased on the >> "Remove device private pages from physical address space" series: >> https://lore.kernel.org/linux-mm/20260130111050.53670-1-jniethe@nvidia.com/ >> plus a small patch to adjust with no problems. >> >> Changes v4-v5 >> - rebase on 6.19 >> - fixed David's email address >> - fixed link issue without CONFIG_TRANSPARENT_HUGEPAGE >> - refactored into smaller commits >> - added more comments to code >> >> Changes v3-v4: >> - rebase on 6.19-rc8 >> - fixed issues found by kernel test robot with random configs >> - fixed typos >> >> Changes v2-v3: >> - rebase on 6.19-rc7 >> - fixed issues found by kernel test robot >> - fixed smatch issues reported by Dan Carpenter >> - fixes to lock handling (pmd/pte) on errors >> - added assertions for pmd/pte lock states >> - other issues discovered by Matthew, thanks! >> >> Changes v1-v2: >> - rebase on 6.19-rc6 >> - fixed issues found by kernel test robot >> - fixed locking (pmd/ptl) to cover handle_ and prepare_ regions >> parts if migrating >> - other issues discovered by Matthew, thanks! >> >> Changes RFC-v1: >> - rebase on 6.19-rc5 >> - adjust for the device THP >> - changes from feedback >> >> Revisions: >> - RFC https://lore.kernel.org/linux-mm/20250814072045.3637192-1-mpenttil@redhat.com/ >> - v1: https://lore.kernel.org/all/20260114091923.3950465-1-mpenttil@redhat.com/ >> - v2: https://lore.kernel.org/all/20260119112502.645059-1-mpenttil@redhat.com/ >> - v3: https://lore.kernel.org/all/20260126111939.1332983-2-mpenttil@redhat.com/ >> - v4: https://lore.kernel.org/all/20260202112622.2104213-1-mpenttil@redhat.com/ >> >> Cc: David Hildenbrand >> Cc: Jason Gunthorpe >> Cc: Leon Romanovsky >> Cc: Alistair Popple >> Cc: Balbir Singh >> Cc: Zi Yan >> Cc: Matthew Brost >> >> Mika Penttilä (6): >> mm:/Kconfig changes for migrate on fault for device pages >> mm: Add helper to convert HMM pfn to migrate pfn >> mm/hmm: do the plumbing for HMM to participate in migration >> mm: implement device page migration for HMM pagewalk >> mm: add new testcase for the migrate on fault case >> mm:/migrate_device.c: remove migrate_vma_collect_*() >> >> include/linux/hmm.h | 18 +- >> include/linux/migrate.h | 27 +- >> lib/test_hmm.c | 100 ++- >> lib/test_hmm_uapi.h | 19 +- >> mm/Kconfig | 2 + >> mm/hmm.c | 814 +++++++++++++++++++++++-- >> mm/migrate_device.c | 602 +++--------------- >> tools/testing/selftests/mm/hmm-tests.c | 54 ++ >> 8 files changed, 1053 insertions(+), 583 deletions(-) >> >> >> base-commit: 05f7e89ab9731565d8a62e3b5d1ec206485eeb0b > > Thanks for your patience, I am just trying out the patches and reviewing them in parallel > > Balbir > No problem, and thanks! btw there's also v6 it has only two minor changes: https://lore.kernel.org/linux-mm/20260316062407.3354636-1-mpenttil@redhat.com/ --Mika