* [PATCH v16 00/12] migrate on fault for device pages
@ 2026-10-21 6:04 mpenttil
2026-10-21 6:04 ` [PATCH v16 01/12] mm/Kconfig: changes for " mpenttil
` (11 more replies)
0 siblings, 12 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:04 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
Currently, the way device page faulting and migration works
is not optimal, if you want to do both fault handling and
migration at once.
Being able to migrate not present pages (or pages mapped with incorrect
permissions, eg. COW) to the GPU requires doing either of the
following sequences:
1. hmm_range_fault() - fault in non-present pages with correct permissions, etc.
2. migrate_vma_*() - migrate the pages
Or:
1. migrate_vma_*() - migrate present pages
2. If non-present pages detected by migrate_vma_*():
a) call hmm_range_fault() to fault pages in
b) call migrate_vma_*() again to migrate now present pages
The problem with the first sequence is that you always have to do two
page walks even when most of the time the pages are present or zero page
mappings so the common case takes a performance hit.
The second sequence is better for the common case, but far worse if
pages aren't present because now you have to walk the page tables three
times (once to find the page is not present, once so hmm_range_fault()
can find a non-present page to fault in and once again to setup the
migration). It is also tricky to code correctly. One page table walk
could costs over 1000 cpu cycles on X86-64, which is a significant hit.
We should be able to walk the page table once, faulting
pages in as required and replacing them with migration entries if
requested.
Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE,
which tells to prepare for migration also during fault handling.
For the migrate_vma_setup() call paths, new flags, MIGRATE_VMA_FAULT,
and MIGRATE_VMA_WRITE are added to tell to add fault handling to migrate.
An extra benefit of migrating with hmm_range_fault() path
is the migrate_vma.vma gets populated, so no need to
retrieve that separataly.
This series avoids the problems in current collecting implementation,
where racing MADV_DONTNEED could make the collecting arrays overflow.
This is achieved mainly by having a rollback mechanism and
populating the collecting array as the last step.
Tested in X86-64 VM with HMM test device, passing the selftests.
The migrate throughput tests from the selftests show similar numbers
(within error margin) as an unmodified kernel, verifying not
regressing with current workloads.
For nouveau migrate-on-fault test harness was : Fedora 44,
Mesa Rusticl, and a fine-grained-system-SVM OpenCL program.
Nouveau SVM is opt-in via Mesa. The OpenCL program passes a plain
malloc pointer (64 pages, array of integers) to the GPU. Increments
each element on GPU, and verifies on the CPU the result. So it drives
both directions of migration. With svm_migrate_on_fault=1, no
migration at bind happens so it's full lazy. The GPU used in testing
was Pascal generation GP102 (selected based on what nouveau
supports for SVM).
Tested also rebased on the
"Remove device private pages from physical address space" series:
https://lore.kernel.org/linux-mm/20260130111050.53670-1-jniethe@nvidia.com/
plus a small patch to adjust with no problems.
Changes in v16:
- rebased to 7.3.0 (drm-tip)
- addressed sashiko's concerns
- added patch 12 - drm/nouveau/svm: add support for migrate on fault
Sashiko's concerns :
https://sashiko.dev/#/patchset/20260924065313.899730-1-mpenttil%40redhat.com
patch 01 mm/Kconfig: changes for migrate on fault for device pages
Q "Does selecting HMM_MIRROR under DEVICE_MIGRATION require an explicit
dependency on MMU?"
A A DEVICE_MIGRATION is def_bool MIGRATION && ZONE_DEVICE, and MIGRATION depends on MMU,
which HMM_MIRROR also depends on. DEVICE_MIGRATION is not user selectable, and nothing
selects it explicitly now. If some config would select it in the future, it should have MMU.
patch 03 mm/hmm: preparations for HMM to participate in migration
Q "This is not a bug, but could we add MIGRATE_VMA_SELECT_NONE to this enum
under CONFIG_MIGRATION?"
A Yes, added.
patch 05 mm/hmm: migrate collection in HMM pagewalk - pte level
Q "Can this cause a livelock when splitting the folio?"
A the ref is taken under ptl before dropping it (no UAF), and the split either succeeds
(progress) or fails -> HMM_PFN_ERROR + return 0 (termination). No unbounded retry,
no livelock. The -EAGAIN -> goto again path only re-walks when state actually changed.
Q "Does this code operate on uninitialized folio metadata if a driver maps an
uninitialized compound page to userspace?"
A The folio is either from vm_normal_page(), or a device private/coherent page
and properly initialized so no.
Q "Can a transient split failure result in valid pages being falsely marked as
inaccessible?"
A a split failure reports HMM_PFN_ERROR for that walk only; the consumer can re-fault
and a later split can succeed. The page isn't lost or permanently marked.
patch 06 mm/hmm: migrate collection in HMM pagewalk - pmd level
Q "Can this lead to a use-after-free on an unlocked tail page?"
A For that branch, it has to be fault folio. fault_page (of fault folio)
is always device-private -> the tail new_fault_folio is a ZONE_DEVICE
page, which reclaim never scans. So not possible.
patch 08 mm/hmm: implement rollback for device page migration in HMM pagewalk
Q "Will hmm_pfn_rollback_collected() destroy the user's per-PFN request flags?"
A migration is entirely driven by default_flags, and (HMM_PFN_VALID|HMM_PFN_MIGRATE)
is used as a sentinel for already pinned and migrate collected pfns. The sentinel
is safe to be erased from per pfn flags.
Q "Can this regression lead to memory corruption via an uninitialized PFN
unlock/put?"
A "rollback only unlocks/puts entries carrying the (HMM_PFN_VALID|HMM_PFN_MIGRATE) sentinel
,i.e. ones actually collected (locked+ref'd), so no uninitialized pfn is ever unlocked or put
Q "Is there an infinite loop in hmm_range_fault_locked() when a THP split fails?"
A yes, fixed in v16. A split failure now fills HMM_PFN_ERROR and returns 0 instead of -EBUSY
patch 10 lib/test_hmm: add a new testcase for the migrate on fault
Q "Will skipping the device page table update here create an inconsistent state
that migrate_vma_finalize() permanently commits? (in case parallel unmap)"
A Concurrent unmap is fine. In that case migrate_vma_finalize() frees dst pages,
and retry migrates on next round. So no ping-pong, it is self-healing.
Link to v15: https://lore.kernel.org/linux-mm/20260924065313.899730-1-mpenttil@redhat.com/
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Lyude Paul <lyude@redhat.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Mika Penttilä (12):
mm/Kconfig: changes for migrate on fault for device pages
mm: add helper to convert HMM pfn to migrate pfn
mm/hmm: preparations for HMM to participate in migration
mm/hmm: do the plumbing for HMM to participate in migration
mm/hmm: migrate collection in HMM pagewalk - pte level
mm/hmm: migrate collection in HMM pagewalk - pmd level
mm/hmm: add lazy MMU mode support for migration in HMM pagewalk
mm/hmm: implement rollback for device page migration in HMM pagewalk
mm: enable device page migration from HMM pagewalk
lib/test_hmm: add a new testcase for the migrate on fault
Documentation/mm/hmm: document migration through hmm_range_fault()
drm/nouveau/svm: add support for migrate on fault
Documentation/mm/hmm.rst | 39 +
drivers/gpu/drm/nouveau/nouveau_dmem.c | 281 +++++++-
drivers/gpu/drm/nouveau/nouveau_dmem.h | 12 +
drivers/gpu/drm/nouveau/nouveau_svm.c | 141 ++--
drivers/gpu/drm/nouveau/nouveau_svm.h | 5 +
include/linux/hmm.h | 51 +-
include/linux/migrate.h | 59 +-
lib/test_hmm.c | 174 ++++-
lib/test_hmm_uapi.h | 21 +-
mm/Kconfig | 1 +
mm/hmm.c | 959 +++++++++++++++++++++++--
mm/migrate_device.c | 617 +++-------------
tools/testing/selftests/mm/hmm-tests.c | 54 ++
13 files changed, 1740 insertions(+), 674 deletions(-)
drm-tip
base-commit: b0311e0de50840c3f35aafa2e95c14bf38f4a767
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 01/12] mm/Kconfig: changes for migrate on fault for device pages
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
@ 2026-10-21 6:04 ` mpenttil
2026-10-21 6:04 ` [PATCH v16 02/12] mm: add helper to convert HMM pfn to migrate pfn mpenttil
` (10 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:04 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
HMM depends on MMU notifiers. With the unified HMM/migrate_device
page table walk migrate_device needs HMM enabled.
Enable them explicitly to avoid breaking random configs.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Liam R. Howlett" <Liam.Howlett@oracle.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Balbir Singh <balbirs@nvidia.com>
---
mm/Kconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/mm/Kconfig b/mm/Kconfig
index 604c58199acb..b7a485beb9a7 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -687,6 +687,7 @@ config MIGRATION
config DEVICE_MIGRATION
def_bool MIGRATION && ZONE_DEVICE
+ select HMM_MIRROR
config ARCH_ENABLE_HUGEPAGE_MIGRATION
bool
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 02/12] mm: add helper to convert HMM pfn to migrate pfn
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
2026-10-21 6:04 ` [PATCH v16 01/12] mm/Kconfig: changes for " mpenttil
@ 2026-10-21 6:04 ` mpenttil
2026-10-21 6:04 ` [PATCH v16 03/12] mm/hmm: preparations for HMM to participate in migration mpenttil
` (9 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:04 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
The unified HMM/migrate_device pagewalk does the "collecting"
on the HMM side, so we need a helper to transfer pfns to the
migrate_vma world.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
include/linux/hmm.h | 29 +++++++++++++++++-----
include/linux/migrate.h | 3 ++-
mm/migrate_device.c | 55 +++++++++++++++++++++++++++++++++++++++++
3 files changed, 80 insertions(+), 7 deletions(-)
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index 6f04e3932f5b..4f56f3419cb4 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -13,6 +13,8 @@
struct mmu_interval_notifier;
+struct migrate_vma;
+
/*
* On output:
* 0 - The page is faultable and a future call with
@@ -27,13 +29,22 @@ struct mmu_interval_notifier;
* HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer
* HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation
* to mark that page is already DMA mapped
+ * HMM_PFN_MIGRATE - The entry is to be migrated. Note, HMM_PFN_MIGRATE
+ * alone without HMM_PFN_VALID denotes the
+ * empty page.
+ * This flag optionally together with HMM_PFN_COMPOUND
+ * are indicators for migrate_hmm_range_setup() to
+ * setup the migrate pfns.
+ * HMM_PFN_COMPOUND - The entry represents a > 0 order page
*
* On input:
- * 0 - Return the current state of the page, do not fault it.
- * HMM_PFN_REQ_FAULT - The output must have HMM_PFN_VALID or hmm_range_fault()
- * will fail
- * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fault()
- * will fail. Must be combined with HMM_PFN_REQ_FAULT.
+ * 0 - Return the current state of the page, do not fault it.
+ * HMM_PFN_REQ_FAULT - The output must have HMM_PFN_VALID or hmm_range_fault()
+ * will fail
+ * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fault()
+ * will fail. Must be combined with HMM_PFN_REQ_FAULT.
+ * HMM_PFN_REQ_MIGRATE - For default_flags only, request to migrate the range,
+ * according to hmm_range.migrate.flags
*/
enum hmm_pfn_flags {
/* Output fields and flags */
@@ -48,11 +59,15 @@ enum hmm_pfn_flags {
HMM_PFN_P2PDMA = 1UL << (BITS_PER_LONG - 5),
HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
- HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
+ /* Migrate request */
+ HMM_PFN_MIGRATE = 1UL << (BITS_PER_LONG - 7),
+ HMM_PFN_COMPOUND = 1UL << (BITS_PER_LONG - 8),
+ HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 13),
/* Input flags */
HMM_PFN_REQ_FAULT = HMM_PFN_VALID,
HMM_PFN_REQ_WRITE = HMM_PFN_WRITE,
+ HMM_PFN_REQ_MIGRATE = HMM_PFN_MIGRATE,
HMM_PFN_FLAGS = ~((1UL << HMM_PFN_ORDER_SHIFT) - 1),
};
@@ -107,6 +122,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
* @default_flags: default flags for the range (write, read, ... see hmm doc)
* @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter
* @dev_private_owner: owner of device private pages
+ * @migrate: structure for migrating a range of a VMA
*/
struct hmm_range {
struct mmu_interval_notifier *notifier;
@@ -117,6 +133,7 @@ struct hmm_range {
unsigned long default_flags;
unsigned long pfn_flags_mask;
void *dev_private_owner;
+ struct migrate_vma *migrate;
};
/*
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 78424b3824c2..280ec9a866c7 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -3,6 +3,7 @@
#define _LINUX_MIGRATE_H
#include <linux/mm.h>
+#include <linux/hmm.h>
#include <linux/mempolicy.h>
#include <linux/migrate_mode.h>
#include <linux/hugetlb.h>
@@ -202,7 +203,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
unsigned long npages);
void migrate_device_finalize(unsigned long *src_pfns,
unsigned long *dst_pfns, unsigned long npages);
-
+void migrate_hmm_range_setup(struct hmm_range *range);
#endif /* CONFIG_MIGRATION */
#endif /* _LINUX_MIGRATE_H */
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index 009bfa8b212d..43e571e82cca 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -1528,3 +1528,58 @@ int migrate_device_coherent_folio(struct folio *folio)
return 0;
return -EBUSY;
}
+
+/**
+ * migrate_hmm_range_setup() - prepare to migrate a range of memory
+ * @range: contains pointer to struct migrate_vma to be set up.
+ *
+ * When collecting has been done with hmm_range_fault(), this
+ * should be called next, and completes range->migrate by
+ * populating migrate->src[] and migrate->dst[]
+ * using range->hmm_pfns[].
+ * Also, migrate->cpages and migrate->npages get initialized.
+ * After migrate_hmm_range_setup(), range->migrate is good
+ * for the rest of the migrate_vma_* flow.
+ */
+void migrate_hmm_range_setup(struct hmm_range *range)
+{
+ struct migrate_vma *migrate = range->migrate;
+
+ if (!migrate)
+ return;
+
+ migrate->npages = (migrate->end - migrate->start) >> PAGE_SHIFT;
+ migrate->cpages = 0;
+
+ for (unsigned long i = 0; i < migrate->npages; i++) {
+ unsigned long pfn = range->hmm_pfns[i];
+
+ /*
+ * We are only interested in entries to be
+ * migrated.
+ */
+ if (!(pfn & HMM_PFN_MIGRATE)) {
+ migrate->src[i] = 0;
+ migrate->dst[i] = 0;
+ continue;
+ }
+
+ migrate->cpages++;
+
+ /* HMM_PFN_MIGRATE without HMM_PFN_VALID denotes the special zero page */
+ if (pfn & HMM_PFN_VALID)
+ migrate->src[i] = migrate_pfn(page_to_pfn(hmm_pfn_to_page(pfn)));
+ else
+ migrate->src[i] = 0;
+
+ migrate->src[i] |= MIGRATE_PFN_MIGRATE;
+ migrate->src[i] |= (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0;
+ migrate->src[i] |= (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0;
+ migrate->dst[i] = 0;
+ }
+
+ if (migrate->cpages)
+ migrate_vma_unmap(migrate);
+
+}
+EXPORT_SYMBOL(migrate_hmm_range_setup);
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 03/12] mm/hmm: preparations for HMM to participate in migration
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
2026-10-21 6:04 ` [PATCH v16 01/12] mm/Kconfig: changes for " mpenttil
2026-10-21 6:04 ` [PATCH v16 02/12] mm: add helper to convert HMM pfn to migrate pfn mpenttil
@ 2026-10-21 6:04 ` mpenttil
2026-10-21 7:17 ` Balbir Singh
2026-10-21 6:04 ` [PATCH v16 04/12] mm/hmm: do the plumbing " mpenttil
` (8 subsequent siblings)
11 siblings, 1 reply; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:04 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
For migration to happen after hmm_range_fault(), the vma has to
retrieved. Luckily, pagewalk already resolves that for us, so
just have to save it for further use. For migration, mmap_lock
has to be hold for the whole operation, so vma stays stable.
In case lock-drop behavior is requested, return an error while
migrating. Also, rename some struct hmm_vma_walk fields to solve
namespace collisions, and we are later introducing more
xxlocked fields so be unambiguous.
Take care also of firing the mmu_notifier when migrating.
Also, prepare for differencies of how mm_struct * is retrieved
in fault and migrate_vma paths.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
include/linux/migrate.h | 50 +++++++++++++++++-
lib/test_hmm.c | 2 +-
mm/hmm.c | 110 ++++++++++++++++++++++++++++++++++------
3 files changed, 145 insertions(+), 17 deletions(-)
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 280ec9a866c7..dd7dd7ec6995 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -5,6 +5,7 @@
#include <linux/mm.h>
#include <linux/hmm.h>
#include <linux/mempolicy.h>
+#include <linux/mmu_notifier.h>
#include <linux/migrate_mode.h>
#include <linux/hugetlb.h>
@@ -108,6 +109,29 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
spin_unlock(ptl);
}
+enum migrate_vma_info {
+ MIGRATE_VMA_SELECT_NONE = 0,
+ MIGRATE_VMA_SELECT_COMPOUND = MIGRATE_VMA_SELECT_NONE,
+};
+
+static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
+{
+ return MIGRATE_VMA_SELECT_NONE;
+}
+
+static inline void hmm_fill_migrate_vma(struct hmm_range *range,
+ struct vm_area_struct *vma,
+ unsigned long start,
+ unsigned long end)
+{
+}
+
+#ifdef CONFIG_MMU_NOTIFIER
+static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
+{
+ return range->notifier->mm;
+}
+#endif
#endif /* CONFIG_MIGRATION */
#ifdef CONFIG_NUMA_BALANCING
@@ -151,7 +175,8 @@ static inline unsigned long migrate_pfn(unsigned long pfn)
return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID;
}
-enum migrate_vma_direction {
+enum migrate_vma_info {
+ MIGRATE_VMA_SELECT_NONE = 0,
MIGRATE_VMA_SELECT_SYSTEM = 1 << 0,
MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
@@ -193,6 +218,29 @@ struct migrate_vma {
struct page *fault_page;
};
+// TODO: enable migration
+static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
+{
+ return 0;
+}
+
+#ifdef CONFIG_MMU_NOTIFIER
+static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
+{
+ return range->notifier ? range->notifier->mm : range->migrate->vma->vm_mm;
+}
+#endif
+
+static inline void hmm_fill_migrate_vma(struct hmm_range *range,
+ struct vm_area_struct *vma,
+ unsigned long start,
+ unsigned long end)
+{
+ range->migrate->vma = vma;
+ range->migrate->start = start;
+ range->migrate->end = end;
+}
+
int migrate_vma_setup(struct migrate_vma *args);
void migrate_vma_pages(struct migrate_vma *migrate);
void migrate_vma_finalize(struct migrate_vma *migrate);
diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index 6911daa9f854..cd88e8177d0a 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_device *mdevice)
HMM_DMIRROR_MEMORY_DEVICE_PRIVATE);
}
-static enum migrate_vma_direction
+static enum migrate_vma_info
dmirror_select_device(struct dmirror *dmirror)
{
return (dmirror->mdevice->zone_device_type ==
diff --git a/mm/hmm.c b/mm/hmm.c
index 2f1e98c6b644..4e9542aedc25 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -21,6 +21,7 @@
#include <linux/pagemap.h>
#include <linux/leafops.h>
#include <linux/hugetlb.h>
+#include <linux/migrate.h>
#include <linux/memremap.h>
#include <linux/sched/mm.h>
#include <linux/jump_label.h>
@@ -28,15 +29,23 @@
#include <linux/pci-p2pdma.h>
#include <linux/mmu_notifier.h>
#include <linux/memory_hotplug.h>
+#include <asm/tlbflush.h>
#include "internal.h"
struct hmm_vma_walk {
- struct hmm_range *range;
- bool *locked;
- unsigned long last;
- unsigned long end;
- unsigned int required_fault;
+ struct mmu_notifier_range mmu_range;
+ /* Set once mmu_range is armed with mmu_notifier_invalidate_range_start() */
+ bool mmu_armed;
+ struct vm_area_struct *vma;
+ struct hmm_range *range;
+ unsigned long start;
+ unsigned long end;
+ unsigned long last;
+ /* fault and lock drop related fields */
+ bool *mmlocked;
+ unsigned long fault_end;
+ unsigned int required_fault;
};
/*
@@ -95,7 +104,7 @@ static int hmm_record_fault(unsigned long addr, unsigned long end,
WARN_ON_ONCE(!required_fault);
hmm_vma_walk->last = addr;
- hmm_vma_walk->end = end;
+ hmm_vma_walk->fault_end = end;
hmm_vma_walk->required_fault = required_fault;
return HMM_FAULT_PENDING;
}
@@ -400,6 +409,58 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
}
#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
+static int hmm_vma_capture_migrate_range(unsigned long start,
+ unsigned long end,
+ struct mm_walk *walk)
+{
+ struct hmm_vma_walk *hmm_vma_walk = walk->private;
+ struct hmm_range *range = hmm_vma_walk->range;
+
+ if (!hmm_select_migrate(range))
+ return 0;
+
+ if (hmm_vma_walk->vma && (hmm_vma_walk->vma != walk->vma))
+ return -ERANGE;
+
+ hmm_vma_walk->vma = walk->vma;
+
+ if (end - start > range->end - range->start)
+ return -ERANGE;
+
+ if (!hmm_vma_walk->mmu_armed) {
+ hmm_vma_walk->start = start;
+ hmm_vma_walk->end = end;
+ mmu_notifier_range_init_owner(&hmm_vma_walk->mmu_range, MMU_NOTIFY_MIGRATE, 0,
+ walk->vma->vm_mm, start, end,
+ range->dev_private_owner);
+ mmu_notifier_invalidate_range_start(&hmm_vma_walk->mmu_range);
+ hmm_vma_walk->mmu_armed = true;
+ }
+
+ return 0;
+}
+
+static void hmm_vma_post_range_fault(struct hmm_vma_walk *hmm_vma_walk)
+{
+
+ struct hmm_range *range = hmm_vma_walk->range;
+
+ if (hmm_select_migrate(range) &&
+ hmm_vma_walk->mmu_armed) {
+ /*
+ * The migrate_vma path has the following initialized,
+ * so take care of fault path below.
+ */
+ if (range->notifier) {
+ hmm_fill_migrate_vma(range,
+ hmm_vma_walk->vma,
+ hmm_vma_walk->start,
+ hmm_vma_walk->end);
+ }
+ mmu_notifier_invalidate_range_end(&hmm_vma_walk->mmu_range);
+ }
+}
+
static int hmm_vma_walk_pmd(pmd_t *pmdp,
unsigned long start,
unsigned long end,
@@ -594,6 +655,11 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end,
struct hmm_vma_walk *hmm_vma_walk = walk->private;
struct hmm_range *range = hmm_vma_walk->range;
struct vm_area_struct *vma = walk->vma;
+ int r;
+
+ r = hmm_vma_capture_migrate_range(start, end, walk);
+ if (r)
+ return r;
if (!(vma->vm_flags & (VM_IO | VM_PFNMAP)) &&
vma->vm_flags & VM_READ)
@@ -646,12 +712,12 @@ static int hmm_do_fault(struct mm_struct *mm,
struct hmm_vma_walk *hmm_vma_walk)
{
unsigned long addr = hmm_vma_walk->last;
- unsigned long end = hmm_vma_walk->end;
+ unsigned long end = hmm_vma_walk->fault_end;
unsigned int required_fault = hmm_vma_walk->required_fault;
unsigned int fault_flags = FAULT_FLAG_REMOTE;
struct vm_area_struct *vma;
- if (hmm_vma_walk->locked)
+ if (hmm_vma_walk->mmlocked)
fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
vma = vma_lookup(mm, addr);
@@ -670,8 +736,8 @@ static int hmm_do_fault(struct mm_struct *mm,
ret = handle_mm_fault(vma, addr, fault_flags, NULL);
if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
- if (hmm_vma_walk->locked) /* needed by sparse */
- *hmm_vma_walk->locked = false;
+ if (hmm_vma_walk->mmlocked) /* needed by sparse */
+ *hmm_vma_walk->mmlocked = false;
else
WARN_ON_ONCE(1); /* broken fault handler */
return HMM_FAULT_UNLOCKED;
@@ -694,19 +760,31 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
{
struct hmm_vma_walk hmm_vma_walk = {
.range = range,
- .locked = locked,
+ .mmlocked = locked,
.last = range->start,
};
- struct mm_struct *mm = range->notifier->mm;
+ /*
+ * Could be serving a device fault or come from migrate
+ * entry point. For the former we have not resolved the vma
+ * yet, and the latter we don't have a notifier (but have a vma).
+ *
+ */
+ struct mm_struct *mm = hmm_range_fault_mm(range);
int ret;
+ /* Migration is incompatible with mmap lock drop */
+ if (locked && hmm_select_migrate(range))
+ return -EINVAL;
+
mmap_assert_locked(mm);
do {
/* If range is no longer valid force retry. */
- if (mmu_interval_check_retry(range->notifier,
- range->notifier_seq))
- return -EBUSY;
+ if (range->notifier && mmu_interval_check_retry(range->notifier,
+ range->notifier_seq)) {
+ ret = -EBUSY;
+ break;
+ }
ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
&hmm_walk_ops, &hmm_vma_walk);
/*
@@ -732,6 +810,8 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
* output, and all >= are still at their input values.
*/
} while (ret == -EBUSY);
+
+ hmm_vma_post_range_fault(&hmm_vma_walk);
return ret;
}
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 04/12] mm/hmm: do the plumbing for HMM to participate in migration
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (2 preceding siblings ...)
2026-10-21 6:04 ` [PATCH v16 03/12] mm/hmm: preparations for HMM to participate in migration mpenttil
@ 2026-10-21 6:04 ` mpenttil
2026-10-21 6:04 ` [PATCH v16 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level mpenttil
` (7 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:04 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
Do the preparations in hmm_range_fault() and pagewalk callbacks to
do the "collecting" part of migration, needed for migration.
These steps include locking for pmd/pte if migrating, and calling the
still dummy hmm_vma_handle_migrate_prepare_pmd() and
hmm_vma_handle_migrate_prepare() functions in the pagewalk.
When doing migration, have to have pmd/pte table locked for the
duration of both hmm_vma_handle*() and hmm_vma_handle_migrate_prepare*()
walks, and unlock when handling the faults.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
mm/hmm.c | 343 ++++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 289 insertions(+), 54 deletions(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 4e9542aedc25..119e70dbca56 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -42,6 +42,16 @@ struct hmm_vma_walk {
unsigned long start;
unsigned long end;
unsigned long last;
+ /*
+ * For migration we need pte/pmd locked for the handle_* and
+ * prepare_* regions. While faulting we have to drop the locks
+ * and start again. ptelocked and pmdlocked hold the state
+ * and tell if need to drop locks before faulting.
+ * ptl is the lock held for pte or pmd.
+ */
+ bool ptelocked;
+ bool pmdlocked;
+ spinlock_t *ptl;
/* fault and lock drop related fields */
bool *mmlocked;
unsigned long fault_end;
@@ -63,6 +73,16 @@ struct hmm_vma_walk {
*/
#define HMM_FAULT_UNLOCKED -ENOLCK
+#define HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, locked) \
+ WARN_ON_ONCE((hmm_vma_walk)->ptelocked != locked)
+
+#define HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, locked) \
+ WARN_ON_ONCE((hmm_vma_walk)->pmdlocked != locked)
+
+#define HMM_ASSERT_UNLOCKED(hmm_vma_walk) \
+ WARN_ON_ONCE((hmm_vma_walk)->ptelocked || \
+ (hmm_vma_walk)->pmdlocked)
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -76,14 +96,38 @@ enum {
};
static int hmm_pfns_fill(unsigned long addr, unsigned long end,
- struct hmm_range *range, unsigned long cpu_flags)
+ struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags)
{
+ struct hmm_range *range = hmm_vma_walk->range;
unsigned long i = (addr - range->start) >> PAGE_SHIFT;
+ enum migrate_vma_info minfo;
+ bool migrate = false;
+
+ minfo = hmm_select_migrate(range);
+ if (cpu_flags != HMM_PFN_ERROR) {
+ if (minfo && (vma_is_anonymous(hmm_vma_walk->vma))) {
+ cpu_flags |= HMM_PFN_MIGRATE;
+ migrate = true;
+ }
+ }
+
+ if (migrate && thp_migration_supported() &&
+ (minfo & MIGRATE_VMA_SELECT_COMPOUND) &&
+ IS_ALIGNED(addr, HPAGE_PMD_SIZE) &&
+ IS_ALIGNED(end, HPAGE_PMD_SIZE) &&
+ end-addr == HPAGE_PMD_SIZE) {
+ range->hmm_pfns[i] &= HMM_PFN_INOUT_FLAGS;
+ range->hmm_pfns[i] |= cpu_flags | HMM_PFN_COMPOUND;
+ addr += PAGE_SIZE;
+ i++;
+ cpu_flags = 0;
+ }
for (; addr < end; addr += PAGE_SIZE, i++) {
range->hmm_pfns[i] &= HMM_PFN_INOUT_FLAGS;
range->hmm_pfns[i] |= cpu_flags;
}
+
return 0;
}
@@ -103,6 +147,7 @@ static int hmm_record_fault(unsigned long addr, unsigned long end,
struct hmm_vma_walk *hmm_vma_walk = walk->private;
WARN_ON_ONCE(!required_fault);
+ HMM_ASSERT_UNLOCKED(hmm_vma_walk);
hmm_vma_walk->last = addr;
hmm_vma_walk->fault_end = end;
hmm_vma_walk->required_fault = required_fault;
@@ -187,11 +232,16 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
if (!walk->vma) {
if (required_fault)
return -EFAULT;
- return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR);
+ return hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR);
}
- if (required_fault)
+ if (required_fault) {
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+ }
return hmm_record_fault(addr, end, required_fault, walk);
- return hmm_pfns_fill(addr, end, range, 0);
+ }
+ return hmm_pfns_fill(addr, end, hmm_vma_walk, 0);
}
static inline unsigned long hmm_pfn_flags_order(unsigned long order)
@@ -224,8 +274,13 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
cpu_flags = pmd_to_hmm_pfn_flags(range, pmd);
required_fault =
hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags);
- if (required_fault)
+ if (required_fault) {
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+ }
return hmm_record_fault(addr, end, required_fault, walk);
+ }
pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) {
@@ -305,14 +360,25 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
goto fault;
if (softleaf_is_migration(entry)) {
- pte_unmap(ptep);
- hmm_vma_walk->last = addr;
- migration_entry_wait(walk->mm, pmdp, addr);
- return -EBUSY;
+ if (!hmm_select_migrate(range)) {
+ HMM_ASSERT_UNLOCKED(hmm_vma_walk);
+ pte_unmap(ptep);
+ hmm_vma_walk->last = addr;
+ migration_entry_wait(walk->mm, pmdp, addr);
+ return -EBUSY;
+ }
+ return 0;
}
/* Report error for everything else */
- pte_unmap(ptep);
+
+ if (hmm_vma_walk->ptelocked) {
+ pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+ hmm_vma_walk->ptelocked = false;
+ } else {
+ pte_unmap(ptep);
+ }
+
return -EFAULT;
}
@@ -329,7 +395,13 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
if (!vm_normal_page(walk->vma, addr, pte) &&
!is_zero_pfn(pte_pfn(pte))) {
if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) {
- pte_unmap(ptep);
+ if (hmm_vma_walk->ptelocked) {
+ pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+ hmm_vma_walk->ptelocked = false;
+ } else {
+ pte_unmap(ptep);
+ }
+
return -EFAULT;
}
new_pfn_flags = HMM_PFN_ERROR;
@@ -342,7 +414,12 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
return 0;
fault:
- pte_unmap(ptep);
+ if (hmm_vma_walk->ptelocked) {
+ pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+ hmm_vma_walk->ptelocked = false;
+ } else {
+ pte_unmap(ptep);
+ }
/* Fault any virtual address we were asked to fault */
return hmm_record_fault(addr, end, required_fault, walk);
}
@@ -386,13 +463,18 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
required_fault = hmm_range_need_fault(hmm_vma_walk, hmm_pfns,
npages, 0);
if (required_fault) {
- if (softleaf_is_device_private(entry))
+ if (softleaf_is_device_private(entry)) {
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+ }
return hmm_record_fault(addr, end, required_fault, walk);
+ }
else
return -EFAULT;
}
- return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+ return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
}
#else
static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
@@ -400,15 +482,58 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
pmd_t pmd)
{
struct hmm_vma_walk *hmm_vma_walk = walk->private;
- struct hmm_range *range = hmm_vma_walk->range;
unsigned long npages = (end - start) >> PAGE_SHIFT;
if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0))
return -EFAULT;
- return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+ return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
}
#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
+#ifdef CONFIG_DEVICE_MIGRATION
+static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
+ pmd_t *pmdp,
+ unsigned long start,
+ unsigned long end,
+ unsigned long *hmm_pfn)
+{
+ // TODO: implement migration entry insertion
+ return 0;
+}
+
+static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
+ pmd_t *pmdp,
+ pte_t *ptep,
+ unsigned long addr,
+ unsigned long *hmm_pfn,
+ bool *unmapped)
+{
+ // TODO: implement migration entry insertion
+ return 0;
+}
+
+#else
+static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
+ pmd_t *pmdp,
+ unsigned long start,
+ unsigned long end,
+ unsigned long *hmm_pfn)
+{
+ return 0;
+}
+
+static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
+ pmd_t *pmdp,
+ pte_t *ptep,
+ unsigned long addr,
+ unsigned long *hmm_pfn,
+ bool *unmapped)
+{
+ return 0;
+}
+
+#endif
+
static int hmm_vma_capture_migrate_range(unsigned long start,
unsigned long end,
struct mm_walk *walk)
@@ -468,46 +593,116 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
{
struct hmm_vma_walk *hmm_vma_walk = walk->private;
struct hmm_range *range = hmm_vma_walk->range;
- unsigned long *hmm_pfns =
- &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT];
unsigned long npages = (end - start) >> PAGE_SHIFT;
+ struct mm_struct *mm = walk->vma->vm_mm;
+ unsigned long *hmm_pfns, *hmm_pfns_start;
+ enum migrate_vma_info minfo;
unsigned long addr = start;
+ bool unmapped = false;
+ unsigned long i;
pte_t *ptep;
pmd_t pmd;
+ int r = 0;
+ minfo = hmm_select_migrate(range);
+ hmm_pfns_start = &range->hmm_pfns[(start - range->start) >> PAGE_SHIFT];
again:
- pmd = pmdp_get_lockless(pmdp);
- if (pmd_none(pmd))
- return hmm_vma_walk_hole(start, end, -1, walk);
+ hmm_pfns = &range->hmm_pfns[(addr - range->start) >> PAGE_SHIFT];
+ hmm_vma_walk->ptelocked = false;
+ hmm_vma_walk->pmdlocked = false;
+
+ if (minfo) {
+ hmm_vma_walk->ptl = pmd_lock(mm, pmdp);
+ hmm_vma_walk->pmdlocked = true;
+ pmd = pmdp_get(pmdp);
+ } else
+ pmd = pmdp_get_lockless(pmdp);
+
+ if (pmd_none(pmd)) {
+ r = hmm_vma_walk_hole(start, end, -1, walk);
+
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+ }
+ return r;
+ }
if (thp_migration_supported() && pmd_is_migration_entry(pmd)) {
- if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0)) {
- hmm_vma_walk->last = addr;
- pmd_migration_entry_wait(walk->mm, pmdp);
- return -EBUSY;
+ if (!minfo) {
+ if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0)) {
+ hmm_vma_walk->last = addr;
+ pmd_migration_entry_wait(walk->mm, pmdp);
+ return -EBUSY;
+ }
+ for (i = 0; start < end; start += PAGE_SIZE, i++)
+ hmm_pfns_start[i] &= HMM_PFN_INOUT_FLAGS;
+ }
+
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
}
- return hmm_pfns_fill(start, end, range, 0);
+
+ return 0;
}
- if (!pmd_present(pmd))
- return hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns,
- pmd);
+ if (pmd_trans_huge(pmd) || !pmd_present(pmd)) {
+ if (!pmd_present(pmd)) {
+ r = hmm_vma_handle_absent_pmd(walk, start, end, hmm_pfns_start,
+ pmd);
+ // If not migrating we are done
+ if (r || !minfo) {
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+ }
+ return r;
+ }
+ }
- if (pmd_trans_huge(pmd)) {
- /*
- * No need to take pmd_lock here, even if some other thread
- * is splitting the huge pmd we will get that event through
- * mmu_notifier callback.
- *
- * So just read pmd value and check again it's a transparent
- * huge or device mapping one and compute corresponding pfn
- * values.
- */
- pmd = pmdp_get_lockless(pmdp);
- if (!pmd_trans_huge(pmd))
- goto again;
+ if (pmd_trans_huge(pmd)) {
+ /*
+ * No need to take pmd_lock here if not migrating,
+ * even if some other thread is splitting the huge
+ * pmd we will get that event through mmu_notifier callback.
+ *
+ * So just read pmd value and check again it's a transparent
+ * huge or device mapping one and compute corresponding pfn
+ * values.
+ */
+
+ if (!minfo) {
+ pmd = pmdp_get_lockless(pmdp);
+ if (!pmd_trans_huge(pmd))
+ goto again;
+ }
- return hmm_vma_handle_pmd(walk, addr, end, hmm_pfns, pmd);
+ r = hmm_vma_handle_pmd(walk, start, end, hmm_pfns_start, pmd);
+
+ // If not migrating we are done
+ if (r || !minfo) {
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+ }
+ return r;
+ }
+ }
+
+ r = hmm_vma_handle_migrate_prepare_pmd(walk, pmdp, start, end, hmm_pfns_start);
+
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+ }
+
+ return r;
+ }
+
+ if (hmm_vma_walk->pmdlocked) {
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
}
/*
@@ -517,24 +712,64 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
* recover.
*/
if (pmd_bad(pmd)) {
- if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, 0))
+ if (hmm_range_need_fault(hmm_vma_walk, hmm_pfns_start, npages, 0))
return -EFAULT;
- return hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+ return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
}
- ptep = pte_offset_map(pmdp, addr);
- if (!ptep)
+ if (minfo) {
+ ptep = pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl);
+ if (ptep)
+ hmm_vma_walk->ptelocked = true;
+ } else {
+ ptep = pte_offset_map(pmdp, addr);
+ }
+ if (!ptep) {
+ addr = start;
goto again;
+ }
+
for (; addr < end; addr += PAGE_SIZE, ptep++, hmm_pfns++) {
- int r;
r = hmm_vma_handle_pte(walk, addr, end, pmdp, ptep, hmm_pfns);
if (r) {
- /* hmm_vma_handle_pte() did pte_unmap() */
+ /* hmm_vma_handle_pte() did pte_unmap() / pte_unmap_unlock */
+ if (unmapped)
+ flush_tlb_range(walk->vma, start, addr);
return r;
}
+
+ r = hmm_vma_handle_migrate_prepare(walk, pmdp, ptep, addr, hmm_pfns, &unmapped);
+ if (r == -EAGAIN) {
+ HMM_ASSERT_UNLOCKED(hmm_vma_walk);
+ if (unmapped) {
+ flush_tlb_range(walk->vma, start, addr);
+ unmapped = false;
+ }
+ goto again;
+ }
+ if (r) {
+ /* A non -EAGAIN error here means migrate_vma_split_folio()
+ * already dropped the PTE lock and cleared ptelocked.
+ */
+ HMM_ASSERT_UNLOCKED(hmm_vma_walk);
+ if (unmapped)
+ flush_tlb_range(walk->vma, start, addr);
+ hmm_pfns_fill(addr, end, hmm_vma_walk, HMM_PFN_ERROR);
+ return 0;
+ }
}
- pte_unmap(ptep - 1);
+
+ if (unmapped)
+ flush_tlb_range(walk->vma, start, addr);
+
+ if (hmm_vma_walk->ptelocked) {
+ pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl);
+ hmm_vma_walk->ptelocked = false;
+ } else {
+ pte_unmap(ptep - 1);
+ }
+
return 0;
}
@@ -682,7 +917,7 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end,
(end - start) >> PAGE_SHIFT, 0))
return -EFAULT;
- hmm_pfns_fill(start, end, range, HMM_PFN_ERROR);
+ hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
/* Skip this vma and continue processing the next vma. */
return 1;
@@ -798,9 +1033,9 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
if (ret == HMM_FAULT_PENDING) {
ret = hmm_do_fault(mm, &hmm_vma_walk);
if (ret == HMM_FAULT_UNLOCKED) {
- if (fatal_signal_pending(current))
- return -EINTR;
- return -EBUSY;
+ ret = fatal_signal_pending(current) ? -EINTR :
+ -EBUSY;
+ break;
}
}
/*
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (3 preceding siblings ...)
2026-10-21 6:04 ` [PATCH v16 04/12] mm/hmm: do the plumbing " mpenttil
@ 2026-10-21 6:04 ` mpenttil
2026-10-21 6:56 ` Balbir Singh
2026-10-21 6:05 ` [PATCH v16 06/12] mm/hmm: migrate collection in HMM pagewalk - pmd level mpenttil
` (6 subsequent siblings)
11 siblings, 1 reply; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:04 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
Implement the needed hmm_vma_handle_migrate_prepare() function
which is mostly carried over from migrate_device.c's
migrate_vma_collect_pmd() function.
Also implement the migrate_vma_split_folio(), for splitting
pte mapped large folios. It is also mostly from migrate_device.c,
with care taken to reference folio before relasing page table
lock.
With HMM pagewalk based migration, the idea is that
hmm_vma_handle_*() are responsible for faulting,
and the pfn collecting part. hmm_vma_handle_migrate_prepare*()
do the migration decisions (with HMM_PFN_MIGRATE), possibly split
folios, and insert migration ptes/pmds.
HMM pagewalk based migration is enabled in later commit, for now
now hmm_select_migrate() returns 0.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
mm/hmm.c | 263 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 262 insertions(+), 1 deletion(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 119e70dbca56..0ce947d0e572 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -491,6 +491,63 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
#endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
#ifdef CONFIG_DEVICE_MIGRATION
+/**
+ * migrate_vma_split_folio() - Helper function to split a THP folio
+ * @folio: the folio to split
+ * @fault_page: struct page associated with the fault if any
+ * @hmm_vma_walk: walk in progress
+ * @ptep: pte_t * for unmap and unlock ptl
+ *
+ * Returns 0 on success
+ */
+static int migrate_vma_split_folio(struct folio *folio,
+ struct page *fault_page,
+ struct hmm_vma_walk *hmm_vma_walk,
+ pte_t *ptep)
+{
+ int ret;
+ struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
+ struct folio *new_fault_folio = NULL;
+
+ if (folio != fault_folio)
+ folio_get(folio);
+
+ pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+ hmm_vma_walk->ptelocked = false;
+
+ if (folio != fault_folio)
+ folio_lock(folio);
+
+ ret = split_folio(folio);
+ if (ret) {
+ if (folio != fault_folio) {
+ folio_unlock(folio);
+ folio_put(folio);
+ }
+ return ret;
+ }
+
+ new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
+
+ /*
+ * Ensure the lock is held on the correct
+ * folio after the split
+ */
+ if (!new_fault_folio) {
+ folio_unlock(folio);
+ folio_put(folio);
+ } else if (folio != new_fault_folio) {
+ if (new_fault_folio != fault_folio) {
+ folio_get(new_fault_folio);
+ folio_lock(new_fault_folio);
+ }
+ folio_unlock(folio);
+ folio_put(folio);
+ }
+
+ return 0;
+}
+
static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
pmd_t *pmdp,
unsigned long start,
@@ -501,6 +558,11 @@ static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
return 0;
}
+/*
+ * Install migration entries if migration requested, either from fault
+ * or migrate paths.
+ *
+ */
static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
pmd_t *pmdp,
pte_t *ptep,
@@ -508,8 +570,207 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
unsigned long *hmm_pfn,
bool *unmapped)
{
- // TODO: implement migration entry insertion
+ struct hmm_vma_walk *hmm_vma_walk = walk->private;
+ struct hmm_range *range = hmm_vma_walk->range;
+ struct migrate_vma *migrate = range->migrate;
+ struct mm_struct *mm = walk->vma->vm_mm;
+ struct folio *fault_folio = NULL;
+ enum migrate_vma_info minfo;
+ struct dev_pagemap *pgmap;
+ bool anon_exclusive;
+ struct folio *folio;
+ unsigned long pfn;
+ struct page *page;
+ softleaf_t entry;
+ pte_t pte, swp_pte;
+ bool writable = false;
+
+ // Do we want to migrate at all?
+ minfo = hmm_select_migrate(range);
+ if (!minfo)
+ return 0;
+
+ WARN_ON_ONCE(!migrate);
+ HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, true);
+
+ fault_folio = migrate->fault_page ?
+ page_folio(migrate->fault_page) : NULL;
+
+ pte = ptep_get(ptep);
+
+ if (pte_none(pte)) {
+ if (vma_is_anonymous(walk->vma)) {
+ *hmm_pfn &= HMM_PFN_INOUT_FLAGS;
+ *hmm_pfn |= HMM_PFN_MIGRATE;
+ goto out;
+ }
+ }
+
+ if (!(hmm_pfn[0] & HMM_PFN_VALID))
+ goto out;
+
+ if (!pte_present(pte)) {
+ /*
+ * Only care about unaddressable device page special
+ * page table entry. Other special swap entries are not
+ * migratable, and we ignore regular swapped page.
+ */
+ entry = softleaf_from_pte(pte);
+ if (!softleaf_is_device_private(entry))
+ goto out;
+
+ if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE))
+ goto out;
+
+ page = softleaf_to_page(entry);
+ folio = page_folio(page);
+ if (folio->pgmap->owner != migrate->pgmap_owner)
+ goto out;
+
+ if (folio_test_large(folio)) {
+ int ret;
+
+ ret = migrate_vma_split_folio(folio,
+ migrate->fault_page,
+ hmm_vma_walk,
+ ptep);
+ if (ret)
+ goto out_error;
+ return -EAGAIN;
+ }
+
+ pfn = page_to_pfn(page);
+ if (softleaf_is_device_private_write(entry))
+ writable = true;
+ } else {
+ pfn = pte_pfn(pte);
+ if (is_zero_pfn(pfn) &&
+ (minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
+ *hmm_pfn = HMM_PFN_MIGRATE;
+ goto out;
+ }
+ page = vm_normal_page(walk->vma, addr, pte);
+ if (page && !is_zone_device_page(page) &&
+ !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
+ goto out;
+ } else if (page && is_device_coherent_page(page)) {
+ pgmap = page_pgmap(page);
+
+ if (!(minfo &
+ MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
+ pgmap->owner != migrate->pgmap_owner)
+ goto out;
+ }
+
+ folio = page ? page_folio(page) : NULL;
+ if (folio && folio_test_large(folio)) {
+ int ret;
+
+ ret = migrate_vma_split_folio(folio,
+ migrate->fault_page,
+ hmm_vma_walk,
+ ptep);
+ if (ret)
+ goto out_error;
+ return -EAGAIN;
+ }
+
+ writable = pte_write(pte);
+ }
+
+ if (!page || !page->mapping)
+ goto out;
+
+ /*
+ * By getting a reference on the folio we pin it and that blocks
+ * any kind of migration. Side effect is that it "freezes" the
+ * pte.
+ *
+ * We drop this reference after isolating the folio from the lru
+ * for non device folio (device folio are not on the lru and thus
+ * can't be dropped from it).
+ */
+ folio = page_folio(page);
+ folio_get(folio);
+
+ /*
+ * We rely on folio_trylock() to avoid deadlock between
+ * concurrent migrations where each is waiting on the others
+ * folio lock. If we can't immediately lock the folio we fail this
+ * migration as it is only best effort anyway.
+ *
+ * If we can lock the folio it's safe to set up a migration entry
+ * now. In the common case where the folio is mapped once in a
+ * single process setting up the migration entry now is an
+ * optimisation to avoid walking the rmap later with
+ * try_to_migrate().
+ */
+
+ if (fault_folio == folio || folio_trylock(folio)) {
+ anon_exclusive = folio_test_anon(folio) &&
+ PageAnonExclusive(page);
+
+ if (pte_present(pte))
+ flush_cache_page(walk->vma, addr, pfn);
+
+ if (anon_exclusive) {
+ pte = ptep_clear_flush(walk->vma, addr, ptep);
+
+ if (folio_try_share_anon_rmap_pte(folio, page)) {
+ set_pte_at(mm, addr, ptep, pte);
+ if (fault_folio != folio)
+ folio_unlock(folio);
+ folio_put(folio);
+ goto out;
+ }
+ } else {
+ pte = ptep_get_and_clear(mm, addr, ptep);
+ }
+
+ if (pte_present(pte) && pte_dirty(pte))
+ folio_mark_dirty(folio);
+
+ /* Setup special migration page table entry */
+ if (writable)
+ entry = make_writable_migration_entry(pfn);
+ else if (anon_exclusive)
+ entry = make_readable_exclusive_migration_entry(pfn);
+ else
+ entry = make_readable_migration_entry(pfn);
+
+ if (pte_present(pte)) {
+ if (pte_young(pte))
+ entry = make_migration_entry_young(entry);
+ if (pte_dirty(pte))
+ entry = make_migration_entry_dirty(entry);
+ }
+
+ swp_pte = swp_entry_to_pte(entry);
+ if (pte_present(pte)) {
+ if (pte_soft_dirty(pte))
+ swp_pte = pte_swp_mksoft_dirty(swp_pte);
+ if (pte_uffd(pte))
+ swp_pte = pte_swp_mkuffd(swp_pte);
+ } else {
+ if (pte_swp_soft_dirty(pte))
+ swp_pte = pte_swp_mksoft_dirty(swp_pte);
+ if (pte_swp_uffd(pte))
+ swp_pte = pte_swp_mkuffd(swp_pte);
+ }
+
+ set_pte_at(mm, addr, ptep, swp_pte);
+ folio_remove_rmap_pte(folio, page, walk->vma);
+ folio_put(folio);
+ *hmm_pfn |= HMM_PFN_MIGRATE;
+ if (pte_present(pte))
+ *unmapped = true;
+ } else {
+ folio_put(folio);
+ }
+out:
return 0;
+out_error:
+ return -EFAULT;
}
#else
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 06/12] mm/hmm: migrate collection in HMM pagewalk - pmd level
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (4 preceding siblings ...)
2026-10-21 6:04 ` [PATCH v16 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level mpenttil
@ 2026-10-21 6:05 ` mpenttil
2026-10-21 6:05 ` [PATCH v16 07/12] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk mpenttil
` (5 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:05 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
Implement the needed hmm_vma_handle_migrate_prepare_pmd() function
which is mostly carried over from migrate_device.c's
migrate_vma_collect_huge_pmd() function.
With HMM pagewalk based migration, the idea is that
hmm_vma_handle_*() are responsible for faulting, and the pfn
collecting part. hmm_vma_handle_migrate_prepare*() do the
migration decisions (with HMM_PFN_MIGRATE), possibly split
folios, and insert migration ptes/pmds.
HMM pagewalk based migration is enabled in later commit, for now
now hmm_select_migrate() returns 0.
Also avoid the problems present in the migrate_vma_collect_huge_pmd()
split fallback path this HMM path supersedes:
- extra refcount for fault_folio makes split_folio() fail
- wrong folio unlocked after split_folio()
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
mm/hmm.c | 172 ++++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 151 insertions(+), 21 deletions(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 0ce947d0e572..5891efee3e18 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -492,31 +492,19 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
#ifdef CONFIG_DEVICE_MIGRATION
/**
- * migrate_vma_split_folio() - Helper function to split a THP folio
+ * __migrate_vma_split_folio() - split a folio and move the lock/ref to the
+ * order-0 folio backing @fault_page after the split
* @folio: the folio to split
- * @fault_page: struct page associated with the fault if any
- * @hmm_vma_walk: walk in progress
- * @ptep: pte_t * for unmap and unlock ptl
+ * @fault_page: fault page if any
*
- * Returns 0 on success
+ * Returns 0 on success.
*/
-static int migrate_vma_split_folio(struct folio *folio,
- struct page *fault_page,
- struct hmm_vma_walk *hmm_vma_walk,
- pte_t *ptep)
+static int __migrate_vma_split_folio(struct folio *folio,
+ struct page *fault_page)
{
- int ret;
struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
struct folio *new_fault_folio = NULL;
-
- if (folio != fault_folio)
- folio_get(folio);
-
- pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
- hmm_vma_walk->ptelocked = false;
-
- if (folio != fault_folio)
- folio_lock(folio);
+ int ret;
ret = split_folio(folio);
if (ret) {
@@ -548,14 +536,145 @@ static int migrate_vma_split_folio(struct folio *folio,
return 0;
}
+/**
+ * migrate_vma_split_folio() - drop the pte lock and split a THP folio
+ * @folio: the folio to split
+ * @fault_page: struct page associated with the fault if any
+ * @hmm_vma_walk: walk in progress
+ * @ptep: pte_t * for unmap and unlock ptl
+ *
+ * Returns 0 on success
+ */
+static int migrate_vma_split_folio(struct folio *folio,
+ struct page *fault_page,
+ struct hmm_vma_walk *hmm_vma_walk,
+ pte_t *ptep)
+{
+ struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
+
+ if (folio != fault_folio)
+ folio_get(folio);
+
+ pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
+ hmm_vma_walk->ptelocked = false;
+
+ if (folio != fault_folio)
+ folio_lock(folio);
+
+ return __migrate_vma_split_folio(folio, fault_page);
+}
+
static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
pmd_t *pmdp,
unsigned long start,
unsigned long end,
unsigned long *hmm_pfn)
{
- // TODO: implement migration entry insertion
- return 0;
+ struct hmm_vma_walk *hmm_vma_walk = walk->private;
+ struct hmm_range *range = hmm_vma_walk->range;
+ struct migrate_vma *migrate = range->migrate;
+ struct folio *fault_folio = NULL;
+ enum migrate_vma_info minfo;
+ struct folio *folio;
+ unsigned long i;
+ int r = 0;
+
+ // Do we want to migrate at all?
+ minfo = hmm_select_migrate(range);
+ if (!minfo)
+ return r;
+
+ WARN_ON_ONCE(!migrate);
+ HMM_ASSERT_PMD_LOCKED(hmm_vma_walk, true);
+
+ fault_folio = migrate->fault_page ?
+ page_folio(migrate->fault_page) : NULL;
+
+ if (pmd_none(*pmdp))
+ return hmm_pfns_fill(start, end, hmm_vma_walk, 0);
+
+ if (!(hmm_pfn[0] & HMM_PFN_VALID))
+ goto out;
+
+ if (pmd_trans_huge(*pmdp)) {
+ if (!(minfo & MIGRATE_VMA_SELECT_SYSTEM))
+ goto out;
+
+ folio = pmd_folio(*pmdp);
+ if (is_huge_zero_folio(folio))
+ return hmm_pfns_fill(start, end, hmm_vma_walk, 0);
+
+ } else if (!pmd_present(*pmdp)) {
+ const softleaf_t entry = softleaf_from_pmd(*pmdp);
+
+ if (!softleaf_is_device_private(entry))
+ goto out;
+
+ if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE))
+ goto out;
+
+ folio = softleaf_to_folio(entry);
+ if (folio->pgmap->owner != migrate->pgmap_owner)
+ goto out;
+ } else {
+ hmm_vma_walk->last = start;
+ return -EBUSY;
+ }
+
+ folio_get(folio);
+
+ if (folio != fault_folio && unlikely(!folio_trylock(folio))) {
+ folio_put(folio);
+ hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
+ return 0;
+ }
+
+ if (thp_migration_supported() &&
+ (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
+ (IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
+ IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
+ struct page_vma_mapped_walk pvmw = {
+ .ptl = hmm_vma_walk->ptl,
+ .address = start,
+ .pmd = pmdp,
+ .vma = walk->vma,
+ };
+
+ hmm_pfn[0] |= HMM_PFN_MIGRATE | HMM_PFN_COMPOUND;
+
+ r = set_pmd_migration_entry(&pvmw, folio_page(folio, 0));
+ if (r) {
+ hmm_pfn[0] &= ~(HMM_PFN_MIGRATE | HMM_PFN_COMPOUND);
+ goto split; /* fall back to splitting the pmd */
+ }
+ for (i = 1, start += PAGE_SIZE; start < end; start += PAGE_SIZE, i++)
+ hmm_pfn[i] &= HMM_PFN_INOUT_FLAGS;
+
+ } else {
+ goto split; /* fall back to splitting the pmd */
+ }
+
+out:
+ return r;
+
+split:
+ spin_unlock(hmm_vma_walk->ptl);
+ hmm_vma_walk->pmdlocked = false;
+
+ /*
+ * folio_get() above took an extra reference. For the fault folio the
+ * caller still holds a reference and the lock, which is the
+ * precondition of __migrate_vma_split_folio(), so drop the extra one.
+ */
+ if (folio == fault_folio)
+ folio_put(folio);
+
+ r = __migrate_vma_split_folio(folio, migrate->fault_page);
+ if (r)
+ return -EFAULT;
+
+ hmm_vma_walk->last = start;
+ return -EBUSY;
}
/*
@@ -958,6 +1077,17 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
hmm_vma_walk->pmdlocked = false;
}
+ /*
+ * hmm_vma_handle_migrate_prepare_pmd() splits the huge pmd in
+ * place when needed and returns -EBUSY to re-walk the range as
+ * PTEs; any other error means the split failed.
+ */
+ if (r == -EBUSY)
+ return -EBUSY;
+ if (r) {
+ /* Split not successful, skip */
+ return hmm_pfns_fill(start, end, hmm_vma_walk, HMM_PFN_ERROR);
+ }
return r;
}
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 07/12] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (5 preceding siblings ...)
2026-10-21 6:05 ` [PATCH v16 06/12] mm/hmm: migrate collection in HMM pagewalk - pmd level mpenttil
@ 2026-10-21 6:05 ` mpenttil
2026-10-21 6:05 ` [PATCH v16 08/12] mm/hmm: implement rollback for device page " mpenttil
` (4 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:05 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
Add calls to lazy_mmu_mode_enable() and lazy_mmu_mode_disable()
while doing migration.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
mm/hmm.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index 5891efee3e18..36a4edde9823 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -373,6 +373,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
/* Report error for everything else */
if (hmm_vma_walk->ptelocked) {
+ lazy_mmu_mode_disable();
pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
hmm_vma_walk->ptelocked = false;
} else {
@@ -396,6 +397,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
!is_zero_pfn(pte_pfn(pte))) {
if (hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0)) {
if (hmm_vma_walk->ptelocked) {
+ lazy_mmu_mode_disable();
pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
hmm_vma_walk->ptelocked = false;
} else {
@@ -415,6 +417,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
fault:
if (hmm_vma_walk->ptelocked) {
+ lazy_mmu_mode_disable();
pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
hmm_vma_walk->ptelocked = false;
} else {
@@ -555,6 +558,7 @@ static int migrate_vma_split_folio(struct folio *folio,
if (folio != fault_folio)
folio_get(folio);
+ lazy_mmu_mode_disable();
pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
hmm_vma_walk->ptelocked = false;
@@ -1110,8 +1114,10 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
if (minfo) {
ptep = pte_offset_map_lock(mm, pmdp, addr, &hmm_vma_walk->ptl);
- if (ptep)
+ if (ptep) {
+ lazy_mmu_mode_enable();
hmm_vma_walk->ptelocked = true;
+ }
} else {
ptep = pte_offset_map(pmdp, addr);
}
@@ -1155,6 +1161,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
flush_tlb_range(walk->vma, start, addr);
if (hmm_vma_walk->ptelocked) {
+ lazy_mmu_mode_disable();
pte_unmap_unlock(ptep - 1, hmm_vma_walk->ptl);
hmm_vma_walk->ptelocked = false;
} else {
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 08/12] mm/hmm: implement rollback for device page migration in HMM pagewalk
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (6 preceding siblings ...)
2026-10-21 6:05 ` [PATCH v16 07/12] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk mpenttil
@ 2026-10-21 6:05 ` mpenttil
2026-10-21 6:05 ` [PATCH v16 09/12] mm: enable device page migration from " mpenttil
` (3 subsequent siblings)
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:05 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
During the migration pagewalk, the PTE table could be cleared and/or
changed into PMD leaf or even another PTE table while dropped locks.
In these cases the possibly inserted migration ptes are gone. We have
to however undo the collecting done so far, so unlock the folios and
drop reference taken.
During the pagewalk we notice such scenarios if going to recollect a pfn
but have already committed to migrate the entry with HMM_PFN_MIGRATE, in
which case do rollback. Rollback clears (HMM_PFN_VALID | HMM_PFN_MIGRATE)
which is safe, as migration is entirely driven by default_flags.
If we encounter migration ptes they are just skipped to allow for restart
own walks.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
include/linux/hmm.h | 22 +++++++++++++
mm/hmm.c | 79 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 101 insertions(+)
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index 4f56f3419cb4..b08ebc1343dd 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -111,6 +111,28 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
return (hmm_pfn >> HMM_PFN_ORDER_SHIFT) & 0x1F;
}
+/*
+ * hmm_pfn_collected() - is this pfn entry prepared for migration ?
+ * If collected the folio's refcount is increased and the folio
+ * is locked.
+ */
+static inline bool hmm_pfn_collected(unsigned long hmm_pfn)
+{
+ return (hmm_pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) ==
+ (HMM_PFN_VALID | HMM_PFN_MIGRATE);
+}
+
+/*
+ * hmm_pfn_rollback_collected() - undoes the collection of hmm_pfn
+ *
+ * Note for total rollback the folio's refcount has to be put
+ * and folio has to be unlocked.
+ */
+static inline unsigned long hmm_pfn_rollback_collected(unsigned long hmm_pfn)
+{
+ return hmm_pfn & ~(HMM_PFN_VALID | HMM_PFN_MIGRATE | HMM_PFN_COMPOUND);
+}
+
/*
* struct hmm_range - track invalidation lock on virtual address range
*
diff --git a/mm/hmm.c b/mm/hmm.c
index 36a4edde9823..5a1471fd5d28 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -95,6 +95,11 @@ enum {
HMM_PFN_P2PDMA_BUS,
};
+static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk,
+ unsigned long start,
+ unsigned long end,
+ unsigned long *hmm_pfn);
+
static int hmm_pfns_fill(unsigned long addr, unsigned long end,
struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags)
{
@@ -111,6 +116,8 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
}
}
+ hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, end, &range->hmm_pfns[i]);
+
if (migrate && thp_migration_supported() &&
(minfo & MIGRATE_VMA_SELECT_COMPOUND) &&
IS_ALIGNED(addr, HPAGE_PMD_SIZE) &&
@@ -282,6 +289,8 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
return hmm_record_fault(addr, end, required_fault, walk);
}
+ hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr,
+ end, hmm_pfns);
pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) {
hmm_pfns[i] &= HMM_PFN_INOUT_FLAGS;
@@ -412,6 +421,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
new_pfn_flags = pte_pfn(pte) | cpu_flags;
out:
+ hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr,
+ addr + PAGE_SIZE,
+ hmm_pfn);
*hmm_pfn = (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags;
return 0;
@@ -450,6 +462,9 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
if (softleaf_is_device_private_write(entry))
cpu_flags |= HMM_PFN_WRITE;
+ hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk,
+ start, end,
+ hmm_pfns);
/*
* Fully populate the PFN list though subsequent PFNs could be
* inferred, because drivers which are not yet aware of large
@@ -568,6 +583,51 @@ static int migrate_vma_split_folio(struct folio *folio,
return __migrate_vma_split_folio(folio, fault_page);
}
+/*
+ * Due to dropping ptl locks for splitting for instance, would we
+ * overwrite already collected pfns? This could happen when pmd
+ * pointing to a page table has vanished and been replaced with
+ * a leaf pmd, or another page table. In that case unref and unlock
+ * the folios, the pfns of which were collected from the disappeared
+ * page tables.
+ *
+ * Migrate mode drives fault/migration requests only via range->default_flags.
+ * (HMM_PFN_VALID | HMM_PFN_MIGRATE) denotes a collected entry and can safely
+ * be cleared from hmm_pfn[].
+ */
+static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk,
+ unsigned long start,
+ unsigned long end,
+ unsigned long *hmm_pfn)
+{
+ struct hmm_range *range = hmm_vma_walk->range;
+ struct migrate_vma *migrate = range->migrate;
+ struct folio *fault_folio = NULL;
+ enum migrate_vma_info minfo;
+ struct folio *folio;
+ unsigned long i;
+
+ minfo = hmm_select_migrate(range);
+ if (!minfo)
+ return;
+
+ WARN_ON_ONCE(!migrate);
+
+ fault_folio = migrate->fault_page ?
+ page_folio(migrate->fault_page) : NULL;
+
+ for (i = 0; start < end; start += PAGE_SIZE, i++) {
+ if (hmm_pfn_collected(hmm_pfn[i])) {
+ folio = page_folio(hmm_pfn_to_page(hmm_pfn[i]));
+ if (folio != fault_folio)
+ folio_unlock(folio);
+ folio_put(folio);
+ hmm_pfn[i] = hmm_pfn_rollback_collected(hmm_pfn[i]);
+
+ }
+ }
+}
+
static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
pmd_t *pmdp,
unsigned long start,
@@ -722,6 +782,11 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
pte = ptep_get(ptep);
if (pte_none(pte)) {
+ hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk,
+ addr,
+ addr + PAGE_SIZE,
+ hmm_pfn);
+
if (vma_is_anonymous(walk->vma)) {
*hmm_pfn &= HMM_PFN_INOUT_FLAGS;
*hmm_pfn |= HMM_PFN_MIGRATE;
@@ -769,6 +834,10 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
pfn = pte_pfn(pte);
if (is_zero_pfn(pfn) &&
(minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
+ hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk,
+ addr,
+ addr + PAGE_SIZE,
+ hmm_pfn);
*hmm_pfn = HMM_PFN_MIGRATE;
goto out;
}
@@ -897,6 +966,13 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
}
#else
+static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk,
+ unsigned long start,
+ unsigned long end,
+ unsigned long *hmm_pfn)
+{
+}
+
static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
pmd_t *pmdp,
unsigned long start,
@@ -1117,6 +1193,9 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
if (ptep) {
lazy_mmu_mode_enable();
hmm_vma_walk->ptelocked = true;
+ } else {
+ /* The pte table is gone */
+ hmm_vma_handle_migrate_prepare_rollback(walk->private, addr, end, hmm_pfns);
}
} else {
ptep = pte_offset_map(pmdp, addr);
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 09/12] mm: enable device page migration from HMM pagewalk
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (7 preceding siblings ...)
2026-10-21 6:05 ` [PATCH v16 08/12] mm/hmm: implement rollback for device page " mpenttil
@ 2026-10-21 6:05 ` mpenttil
2026-10-21 6:19 ` Balbir Singh
2026-10-21 6:05 ` [PATCH v16 10/12] lib/test_hmm: add a new testcase for the migrate on fault mpenttil
` (2 subsequent siblings)
11 siblings, 1 reply; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:05 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
HMM pagewalk has now the machinery to do the first phase of
device page migration, collecting the pfns and installing
migration ptes.
Enable migration in hmm_range_fault(), and change migrate_vma_setup()
to use HMM pagewalk path. Two new flags, MIGRATE_VMA_FAULT and
MIGRATE_VMA_WRITE are introduced for migrate_vma_setup(),
to request for faulting missing pages, and requesting write
access.
Also, the migrate_vma_collect*() based paths are now unused,
so delete them.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Suggested-by: Alistair Popple <apopple@nvidia.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
include/linux/migrate.h | 10 +-
mm/hmm.c | 33 ++-
mm/migrate_device.c | 562 ++++------------------------------------
3 files changed, 83 insertions(+), 522 deletions(-)
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index dd7dd7ec6995..3780c4c116b8 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -181,6 +181,8 @@ enum migrate_vma_info {
MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
MIGRATE_VMA_SELECT_COMPOUND = 1 << 3,
+ MIGRATE_VMA_FAULT = 1 << 4,
+ MIGRATE_VMA_WRITE = 1 << 5,
};
struct migrate_vma {
@@ -218,10 +220,14 @@ struct migrate_vma {
struct page *fault_page;
};
-// TODO: enable migration
static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
{
- return 0;
+ enum migrate_vma_info minfo;
+
+ minfo = (range->default_flags & HMM_PFN_REQ_MIGRATE) ?
+ range->migrate->flags : 0;
+
+ return minfo;
}
#ifdef CONFIG_MMU_NOTIFIER
diff --git a/mm/hmm.c b/mm/hmm.c
index 5a1471fd5d28..4d853ee8e8b8 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -1542,14 +1542,43 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
* the invalidation to finish.
* -EFAULT: A page was requested to be valid and could not be made valid
* ie it has no backing VMA or it is illegal to access
+ * -ERANGE: The range crosses multiple VMAs, or space for hmm_pfns array
+ * is too low.
*
* This is similar to get_user_pages(), except that it can read the page tables
* without mutating them (ie causing faults).
*
* The mmap lock must be held by the caller and will remain held on return.
* New users should prefer hmm_range_fault_unlocked_timeout() unless they
- * specifically need to keep the mmap lock held across the call. This helper
- * cannot support VMAs whose fault handlers need to drop the mmap lock.
+ * specifically need to keep the mmap lock held across the call like while
+ * migrating. This helper cannot support VMAs whose fault handlers need to
+ * drop the mmap lock.
+ *
+ * If want to do migration after faulting, call hmm_range_fault() with
+ * range.default_flags of HMM_PFN_REQ_MIGRATE, and optionally
+ * HMM_PFN_REQ_FAULT|HMM_PFN_REQ_WRITE, and initialize range->migrate field.
+ * range->migrate->vma will be populated during the call,
+ * and must be stable across the whole migrate process, which is
+ * why mmap_lock must be held around this call.
+ *
+ * When HMM_PFN_REQ_MIGRATE is set, migration collection may be partial on
+ * return and the caller takes responsibility for completing or aborting it.
+ *
+ * On success, the caller must call migrate_hmm_range_setup() and may then
+ * proceed with the normal migrate_vma sequence: prepare destination pages,
+ * call migrate_vma_pages(), update device mappings as needed, and finally
+ * call migrate_vma_finalize().
+ *
+ * On -EBUSY, the caller may retry hmm_range_fault() using the same range and
+ * PFN array without undoing entries collected by the previous attempt. If
+ * the caller stops retrying, it must abort the partial migration.
+ *
+ * On any other error, or when abandoning a retry, the caller must call
+ * migrate_hmm_range_setup(), migrate_vma_pages() with no valid destination
+ * entries, and migrate_vma_finalize() to abort the partial migration.
+ *
+ * The mmap read lock must remain held until the migration has either been
+ * completed or aborted.
*/
int hmm_range_fault(struct hmm_range *range)
{
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index 43e571e82cca..1e1a592e1267 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -18,519 +18,6 @@
#include <asm/tlbflush.h>
#include "internal.h"
-static int migrate_vma_collect_skip(unsigned long start,
- unsigned long end,
- struct mm_walk *walk)
-{
- struct migrate_vma *migrate = walk->private;
- unsigned long addr;
-
- for (addr = start; addr < end; addr += PAGE_SIZE) {
- migrate->dst[migrate->npages] = 0;
- migrate->src[migrate->npages++] = 0;
- }
-
- return 0;
-}
-
-static int migrate_vma_collect_hole(unsigned long start,
- unsigned long end,
- __always_unused int depth,
- struct mm_walk *walk)
-{
- struct migrate_vma *migrate = walk->private;
- unsigned long addr;
-
- /* Only allow populating anonymous memory. */
- if (!vma_is_anonymous(walk->vma))
- return migrate_vma_collect_skip(start, end, walk);
-
- if (thp_migration_supported() &&
- (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
- (IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
- IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
- migrate->src[migrate->npages] = MIGRATE_PFN_MIGRATE |
- MIGRATE_PFN_COMPOUND;
- migrate->dst[migrate->npages] = 0;
- migrate->npages++;
- migrate->cpages++;
-
- /*
- * Collect the remaining entries as holes, in case we
- * need to split later
- */
- return migrate_vma_collect_skip(start + PAGE_SIZE, end, walk);
- }
-
- for (addr = start; addr < end; addr += PAGE_SIZE) {
- migrate->src[migrate->npages] = MIGRATE_PFN_MIGRATE;
- migrate->dst[migrate->npages] = 0;
- migrate->npages++;
- migrate->cpages++;
- }
-
- return 0;
-}
-
-/**
- * migrate_vma_split_folio() - Helper function to split a THP folio
- * @folio: the folio to split
- * @fault_page: struct page associated with the fault if any
- *
- * If @folio is not the folio containing @fault_page, the caller must hold a
- * reference on @folio. The helper consumes that reference.
- *
- * Returns 0 on success
- */
-static int migrate_vma_split_folio(struct folio *folio,
- struct page *fault_page)
-{
- int ret;
- struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
- struct folio *new_fault_folio = NULL;
-
- if (folio != fault_folio)
- folio_lock(folio);
-
- ret = split_folio(folio);
- if (ret) {
- if (folio != fault_folio) {
- folio_unlock(folio);
- folio_put(folio);
- }
- return ret;
- }
-
- new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
-
- /*
- * Ensure the lock is held on the correct
- * folio after the split
- */
- if (!new_fault_folio) {
- folio_unlock(folio);
- folio_put(folio);
- } else if (folio != new_fault_folio) {
- if (new_fault_folio != fault_folio) {
- folio_get(new_fault_folio);
- folio_lock(new_fault_folio);
- }
- folio_unlock(folio);
- folio_put(folio);
- }
-
- return 0;
-}
-
-/** migrate_vma_collect_huge_pmd - collect THP pages without splitting the
- * folio for device private pages.
- * @pmdp: pointer to pmd entry
- * @start: start address of the range for migration
- * @end: end address of the range for migration
- * @walk: mm_walk callback structure
- * @fault_folio: folio associated with the fault if any
- *
- * Collect the huge pmd entry at @pmdp for migration and set the
- * MIGRATE_PFN_COMPOUND flag in the migrate src entry to indicate that
- * migration will occur at HPAGE_PMD granularity
- */
-static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start,
- unsigned long end, struct mm_walk *walk,
- struct folio *fault_folio)
-{
- struct mm_struct *mm = walk->mm;
- struct folio *folio;
- struct migrate_vma *migrate = walk->private;
- spinlock_t *ptl;
- int ret;
- unsigned long write = 0;
-
- ptl = pmd_lock(mm, pmdp);
- if (pmd_none(*pmdp)) {
- spin_unlock(ptl);
- return migrate_vma_collect_hole(start, end, -1, walk);
- }
-
- if (pmd_trans_huge(*pmdp)) {
- if (!(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
- spin_unlock(ptl);
- return migrate_vma_collect_skip(start, end, walk);
- }
-
- folio = pmd_folio(*pmdp);
- if (is_huge_zero_folio(folio)) {
- spin_unlock(ptl);
- return migrate_vma_collect_hole(start, end, -1, walk);
- }
- if (pmd_write(*pmdp))
- write = MIGRATE_PFN_WRITE;
- } else if (!pmd_present(*pmdp)) {
- const softleaf_t entry = softleaf_from_pmd(*pmdp);
-
- if (!softleaf_is_device_private(entry) ||
- !(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE)) {
- spin_unlock(ptl);
- return migrate_vma_collect_skip(start, end, walk);
- }
-
- folio = softleaf_to_folio(entry);
- if (folio->pgmap->owner != migrate->pgmap_owner) {
- spin_unlock(ptl);
- return migrate_vma_collect_skip(start, end, walk);
- }
-
- if (softleaf_is_device_private_write(entry))
- write = MIGRATE_PFN_WRITE;
- } else {
- spin_unlock(ptl);
- return -EAGAIN;
- }
-
- folio_get(folio);
- if (folio != fault_folio && unlikely(!folio_trylock(folio))) {
- spin_unlock(ptl);
- folio_put(folio);
- return migrate_vma_collect_skip(start, end, walk);
- }
-
- if (thp_migration_supported() &&
- (migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
- (IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
- IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
-
- struct page_vma_mapped_walk pvmw = {
- .ptl = ptl,
- .address = start,
- .pmd = pmdp,
- .vma = walk->vma,
- };
-
- unsigned long pfn = page_to_pfn(folio_page(folio, 0));
-
- migrate->src[migrate->npages] = migrate_pfn(pfn) | write
- | MIGRATE_PFN_MIGRATE
- | MIGRATE_PFN_COMPOUND;
- migrate->dst[migrate->npages++] = 0;
- migrate->cpages++;
- ret = set_pmd_migration_entry(&pvmw, folio_page(folio, 0));
- if (ret) {
- migrate->npages--;
- migrate->cpages--;
- migrate->src[migrate->npages] = 0;
- migrate->dst[migrate->npages] = 0;
- goto fallback;
- }
- migrate_vma_collect_skip(start + PAGE_SIZE, end, walk);
- spin_unlock(ptl);
- return 0;
- }
-
-fallback:
- spin_unlock(ptl);
- if (!folio_test_large(folio))
- goto done;
- ret = split_folio(folio);
- if (fault_folio != folio)
- folio_unlock(folio);
- folio_put(folio);
- if (ret)
- return migrate_vma_collect_skip(start, end, walk);
- if (pmd_none(pmdp_get_lockless(pmdp)))
- return migrate_vma_collect_hole(start, end, -1, walk);
-
-done:
- return -ENOENT;
-}
-
-static int migrate_vma_collect_pmd(pmd_t *pmdp,
- unsigned long start,
- unsigned long end,
- struct mm_walk *walk)
-{
- struct migrate_vma *migrate = walk->private;
- struct vm_area_struct *vma = walk->vma;
- struct mm_struct *mm = vma->vm_mm;
- unsigned long addr = start, unmapped = 0;
- spinlock_t *ptl;
- struct folio *fault_folio = migrate->fault_page ?
- page_folio(migrate->fault_page) : NULL;
- pte_t *ptep;
-
-again:
- if (pmd_trans_huge(*pmdp) || !pmd_present(*pmdp)) {
- int ret = migrate_vma_collect_huge_pmd(pmdp, start, end, walk, fault_folio);
-
- if (ret == -EAGAIN)
- goto again;
- if (ret == 0)
- return 0;
- }
-
- ptep = pte_offset_map_lock(mm, pmdp, start, &ptl);
- if (!ptep)
- goto again;
- lazy_mmu_mode_enable();
- ptep += (addr - start) / PAGE_SIZE;
-
- for (; addr < end; addr += PAGE_SIZE, ptep++) {
- struct dev_pagemap *pgmap;
- unsigned long mpfn = 0, pfn;
- struct folio *folio;
- struct page *page;
- softleaf_t entry;
- pte_t pte;
-
- pte = ptep_get(ptep);
-
- if (pte_none(pte)) {
- if (vma_is_anonymous(vma)) {
- mpfn = MIGRATE_PFN_MIGRATE;
- migrate->cpages++;
- }
- goto next;
- }
-
- if (!pte_present(pte)) {
- /*
- * Only care about unaddressable device page special
- * page table entry. Other special swap entries are not
- * migratable, and we ignore regular swapped page.
- */
- entry = softleaf_from_pte(pte);
- if (!softleaf_is_device_private(entry))
- goto next;
-
- page = softleaf_to_page(entry);
- pgmap = page_pgmap(page);
- if (!(migrate->flags &
- MIGRATE_VMA_SELECT_DEVICE_PRIVATE) ||
- pgmap->owner != migrate->pgmap_owner)
- goto next;
-
- folio = page_folio(page);
- if (folio_test_large(folio)) {
- int ret;
-
- /* migrate_vma_split_folio() consumes this reference */
- if (folio != fault_folio)
- folio_get(folio);
- lazy_mmu_mode_disable();
- pte_unmap_unlock(ptep, ptl);
- ret = migrate_vma_split_folio(folio,
- migrate->fault_page);
-
- if (ret) {
- if (unmapped)
- flush_tlb_range(walk->vma, start, end);
-
- return migrate_vma_collect_skip(addr, end, walk);
- }
-
- goto again;
- }
-
- mpfn = migrate_pfn(page_to_pfn(page)) |
- MIGRATE_PFN_MIGRATE;
- if (softleaf_is_device_private_write(entry))
- mpfn |= MIGRATE_PFN_WRITE;
- } else {
- pfn = pte_pfn(pte);
- if (is_zero_pfn(pfn) &&
- (migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
- mpfn = MIGRATE_PFN_MIGRATE;
- migrate->cpages++;
- goto next;
- }
- page = vm_normal_page(migrate->vma, addr, pte);
- if (page && !is_zone_device_page(page) &&
- !(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
- goto next;
- } else if (page && is_device_coherent_page(page)) {
- pgmap = page_pgmap(page);
-
- if (!(migrate->flags &
- MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
- pgmap->owner != migrate->pgmap_owner)
- goto next;
- }
- folio = page ? page_folio(page) : NULL;
- if (folio && folio_test_large(folio)) {
- int ret;
-
- /* migrate_vma_split_folio() consumes this reference */
- if (folio != fault_folio)
- folio_get(folio);
- lazy_mmu_mode_disable();
- pte_unmap_unlock(ptep, ptl);
- ret = migrate_vma_split_folio(folio,
- migrate->fault_page);
-
- if (ret) {
- if (unmapped)
- flush_tlb_range(walk->vma, start, end);
-
- return migrate_vma_collect_skip(addr, end, walk);
- }
-
- goto again;
- }
- mpfn = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
- mpfn |= pte_write(pte) ? MIGRATE_PFN_WRITE : 0;
- }
-
- if (!page || !page->mapping) {
- mpfn = 0;
- goto next;
- }
-
- /*
- * By getting a reference on the folio we pin it and that blocks
- * any kind of migration. Side effect is that it "freezes" the
- * pte.
- *
- * We drop this reference after isolating the folio from the lru
- * for non device folio (device folio are not on the lru and thus
- * can't be dropped from it).
- */
- folio = page_folio(page);
- folio_get(folio);
-
- /*
- * We rely on folio_trylock() to avoid deadlock between
- * concurrent migrations where each is waiting on the others
- * folio lock. If we can't immediately lock the folio we fail this
- * migration as it is only best effort anyway.
- *
- * If we can lock the folio it's safe to set up a migration entry
- * now. In the common case where the folio is mapped once in a
- * single process setting up the migration entry now is an
- * optimisation to avoid walking the rmap later with
- * try_to_migrate().
- */
- if (fault_folio == folio || folio_trylock(folio)) {
- bool anon_exclusive;
- pte_t swp_pte;
-
- if (pte_present(pte))
- flush_cache_page(vma, addr, pte_pfn(pte));
- anon_exclusive = folio_test_anon(folio) &&
- PageAnonExclusive(page);
- if (anon_exclusive) {
- pte = ptep_clear_flush(vma, addr, ptep);
-
- if (folio_try_share_anon_rmap_pte(folio, page)) {
- set_pte_at(mm, addr, ptep, pte);
- if (fault_folio != folio)
- folio_unlock(folio);
- folio_put(folio);
- mpfn = 0;
- goto next;
- }
- } else {
- pte = ptep_get_and_clear(mm, addr, ptep);
- }
-
- migrate->cpages++;
-
- /* Set the dirty flag on the folio now the pte is gone. */
- if (pte_present(pte) && pte_dirty(pte))
- folio_mark_dirty(folio);
-
- /* Setup special migration page table entry */
- if (mpfn & MIGRATE_PFN_WRITE)
- entry = make_writable_migration_entry(
- page_to_pfn(page));
- else if (anon_exclusive)
- entry = make_readable_exclusive_migration_entry(
- page_to_pfn(page));
- else
- entry = make_readable_migration_entry(
- page_to_pfn(page));
- if (pte_present(pte)) {
- if (pte_young(pte))
- entry = make_migration_entry_young(entry);
- if (pte_dirty(pte))
- entry = make_migration_entry_dirty(entry);
- }
- swp_pte = swp_entry_to_pte(entry);
- if (pte_present(pte)) {
- if (pte_soft_dirty(pte))
- swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_uffd(pte))
- swp_pte = pte_swp_mkuffd(swp_pte);
- } else {
- if (pte_swp_soft_dirty(pte))
- swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_swp_uffd(pte))
- swp_pte = pte_swp_mkuffd(swp_pte);
- }
- set_pte_at(mm, addr, ptep, swp_pte);
-
- /*
- * This is like regular unmap: we remove the rmap and
- * drop the folio refcount. The folio won't be freed, as
- * we took a reference just above.
- */
- folio_remove_rmap_pte(folio, page, vma);
- folio_put(folio);
-
- if (pte_present(pte))
- unmapped++;
- } else {
- folio_put(folio);
- mpfn = 0;
- }
-
-next:
- migrate->dst[migrate->npages] = 0;
- migrate->src[migrate->npages++] = mpfn;
- }
-
- /* Only flush the TLB if we actually modified any entries */
- if (unmapped)
- flush_tlb_range(walk->vma, start, end);
-
- lazy_mmu_mode_disable();
- pte_unmap_unlock(ptep - 1, ptl);
-
- return 0;
-}
-
-static const struct mm_walk_ops migrate_vma_walk_ops = {
- .pmd_entry = migrate_vma_collect_pmd,
- .pte_hole = migrate_vma_collect_hole,
- .walk_lock = PGWALK_RDLOCK,
-};
-
-/*
- * migrate_vma_collect() - collect pages over a range of virtual addresses
- * @migrate: migrate struct containing all migration information
- *
- * This will walk the CPU page table. For each virtual address backed by a
- * valid page, it updates the src array and takes a reference on the page, in
- * order to pin the page until we lock it and unmap it.
- */
-static void migrate_vma_collect(struct migrate_vma *migrate)
-{
- struct mmu_notifier_range range;
-
- /*
- * Note that the pgmap_owner is passed to the mmu notifier callback so
- * that the registered device driver can skip invalidating device
- * private page mappings that won't be migrated.
- */
- mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0,
- migrate->vma->vm_mm, migrate->start, migrate->end,
- migrate->pgmap_owner);
- mmu_notifier_invalidate_range_start(&range);
-
- walk_page_range_vma(migrate->vma, migrate->start, migrate->end,
- &migrate_vma_walk_ops, migrate);
-
- mmu_notifier_invalidate_range_end(&range);
- migrate->end = migrate->start + (migrate->npages << PAGE_SHIFT);
-}
-
/*
* migrate_vma_check_page() - check if page is pinned or not
* @page: struct page to check
@@ -677,8 +164,9 @@ static void migrate_vma_unmap(struct migrate_vma *migrate)
* migrate_vma_setup() - prepare to migrate a range of memory
* @args: contains the vma, start, and pfns arrays for the migration
*
- * Returns: negative errno on failures, 0 when 0 or more pages were migrated
- * without an error.
+ * Returns: -EINVAL on invalid arguments, 0 otherwise. A 0 return does not
+ * imply anything was collected; on return @args->cpages holds the number of
+ * pages prepared for migration (possibly 0). See the note on faulting below.
*
* Prepare to migrate a range of memory virtual address range by collecting all
* the pages backing each virtual address in the range, saving them inside the
@@ -688,6 +176,20 @@ static void migrate_vma_unmap(struct migrate_vma *migrate)
* corresponding src array entry. Then restores any pages that are pinned, by
* remapping and unlocking those pages.
*
+ * By default migrate_vma_setup() is best-effort: virtual addresses that cannot
+ * be collected are left with an empty src[] entry and the rest of the range
+ * still proceeds.
+ *
+ * The optional flags MIGRATE_VMA_FAULT and MIGRATE_VMA_WRITE in @args->flags
+ * additionally fault in missing pages before collecting them, for read and for
+ * write respectively. Unlike the best-effort default, if such a fault cannot be
+ * satisfied the whole range is aborted: the partial collection is rolled back,
+ * @args->src is cleared, @args->cpages and @args->npages are reset to 0,
+ * and migrate_vma_setup() still returns 0. The fault error is therefore not
+ * observable through this interface. Callers that need to act on fault failures
+ * should instead drive the migration directly via hmm_range_fault() followed by
+ * migrate_hmm_range_setup(), which preserves the error code.
+ *
* The caller should then allocate destination memory and copy source memory to
* it for all those entries (ie with MIGRATE_PFN_VALID and MIGRATE_PFN_MIGRATE
* flag set). Once these are allocated and copied, the caller must update each
@@ -739,10 +241,20 @@ static void migrate_vma_unmap(struct migrate_vma *migrate)
*/
int migrate_vma_setup(struct migrate_vma *args)
{
+ int ret;
long nr_pages = (args->end - args->start) >> PAGE_SHIFT;
+ struct hmm_range range = {
+ .notifier = NULL,
+ .hmm_pfns = args->src,
+ .dev_private_owner = args->pgmap_owner,
+ .migrate = args,
+ .default_flags = HMM_PFN_REQ_MIGRATE
+ };
args->start &= PAGE_MASK;
args->end &= PAGE_MASK;
+ range.start = args->start;
+ range.end = args->end;
if (!args->vma || is_vm_hugetlb_page(args->vma) ||
(args->vma->vm_flags & VM_SPECIAL) || vma_is_dax(args->vma))
return -EINVAL;
@@ -764,10 +276,24 @@ int migrate_vma_setup(struct migrate_vma *args)
args->cpages = 0;
args->npages = 0;
- migrate_vma_collect(args);
+ if (args->flags & MIGRATE_VMA_FAULT)
+ range.default_flags |= HMM_PFN_REQ_FAULT;
- if (args->cpages)
- migrate_vma_unmap(args);
+ if (args->flags & MIGRATE_VMA_WRITE)
+ range.default_flags |= HMM_PFN_REQ_FAULT | HMM_PFN_REQ_WRITE;
+
+ ret = hmm_range_fault(&range);
+
+ migrate_hmm_range_setup(&range);
+
+ /* Remove migration PTEs */
+ if (ret) {
+ migrate_vma_pages(args);
+ migrate_vma_finalize(args);
+ memset(args->src, 0, sizeof(*args->src) * nr_pages);
+ args->cpages = 0;
+ args->npages = 0;
+ }
/*
* At this point pages are locked and unmapped, and thus they have
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 10/12] lib/test_hmm: add a new testcase for the migrate on fault
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (8 preceding siblings ...)
2026-10-21 6:05 ` [PATCH v16 09/12] mm: enable device page migration from " mpenttil
@ 2026-10-21 6:05 ` mpenttil
2026-10-21 6:05 ` [PATCH v16 11/12] Documentation/mm/hmm: document migration through hmm_range_fault() mpenttil
2026-10-21 6:05 ` [PATCH v16 12/12] drm/nouveau/svm: add support for migrate on fault mpenttil
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:05 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich, Marco Pagani
From: Mika Penttilä <mpenttil@redhat.com>
Enhance the hmm test driver (lib/test_hmm) with migrate on fault case.
Cc: David Hildenbrand <david@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Balbir Singh <balbirs@nvidia.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Marco Pagani <marpagan@redhat.com>
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
lib/test_hmm.c | 172 +++++++++++++++++++++++--
lib/test_hmm_uapi.h | 21 +--
tools/testing/selftests/mm/hmm-tests.c | 54 ++++++++
3 files changed, 224 insertions(+), 23 deletions(-)
diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index cd88e8177d0a..31608b6998bb 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -36,6 +36,7 @@
#define DMIRROR_RANGE_FAULT_TIMEOUT 1000
#define DEVMEM_CHUNK_SIZE (256 * 1024 * 1024U)
#define DEVMEM_CHUNKS_RESERVE 16
+#define PFNS_ARRAY_SIZE 64
/*
* For device_private pages, dpage is just a dummy struct page
@@ -355,6 +356,7 @@ static int dmirror_range_fault(struct dmirror *dmirror,
struct mm_struct *mm = dmirror->notifier.mm;
unsigned long timeout =
jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ bool migrate = range->default_flags & HMM_PFN_REQ_MIGRATE;
int ret;
while (true) {
@@ -364,9 +366,15 @@ static int dmirror_range_fault(struct dmirror *dmirror,
}
range->notifier_seq = mmu_interval_read_begin(range->notifier);
- mmap_read_lock(mm);
- ret = hmm_range_fault(range);
- mmap_read_unlock(mm);
+
+ /* mmap lock held for whole migrate on fault operation */
+ if (!migrate) {
+ mmap_read_lock(mm);
+ ret = hmm_range_fault(range);
+ mmap_read_unlock(mm);
+ } else {
+ ret = hmm_range_fault(range);
+ }
if (ret) {
if (ret == -EBUSY)
continue;
@@ -382,7 +390,9 @@ static int dmirror_range_fault(struct dmirror *dmirror,
break;
}
- ret = dmirror_do_fault(dmirror, range);
+ /* update device page table after migration */
+ if (!migrate)
+ ret = dmirror_do_fault(dmirror, range);
mutex_unlock(&dmirror->mutex);
out:
@@ -994,8 +1004,12 @@ static int dmirror_atomic_map(unsigned long addr, struct page *page,
return 0;
}
-static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
- struct dmirror *dmirror)
+/*
+ * Map the migrated pages into the device's page table. Caller must hold
+ * dmirror->mutex.
+ */
+static int __dmirror_migrate_map_locked(struct migrate_vma *args,
+ struct dmirror *dmirror)
{
unsigned long start = args->start;
unsigned long end = args->end;
@@ -1005,9 +1019,6 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
const unsigned long start_pfn = start >> PAGE_SHIFT;
const unsigned long end_pfn = end >> PAGE_SHIFT;
- /* Map the migrated pages into the device's page tables. */
- mutex_lock(&dmirror->mutex);
-
for (pfn = start_pfn; pfn < end_pfn; pfn++, src++, dst++) {
struct page *dpage;
void *entry;
@@ -1036,17 +1047,27 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
if (*dst & MIGRATE_PFN_WRITE)
entry = xa_tag_pointer(entry, DPT_XA_TAG_WRITE);
entry = xa_store(&dmirror->pt, pfn + i, entry, GFP_ATOMIC);
- if (xa_is_err(entry)) {
- mutex_unlock(&dmirror->mutex);
+ if (xa_is_err(entry))
return xa_err(entry);
- }
}
}
- mutex_unlock(&dmirror->mutex);
return 0;
}
+static int dmirror_migrate_finalize_and_map(struct migrate_vma *args,
+ struct dmirror *dmirror)
+{
+ int ret;
+
+ /* Map the migrated pages into the device's page tables. */
+ mutex_lock(&dmirror->mutex);
+ ret = __dmirror_migrate_map_locked(args, dmirror);
+ mutex_unlock(&dmirror->mutex);
+
+ return ret;
+}
+
static int dmirror_exclusive(struct dmirror *dmirror,
struct hmm_dmirror_cmd *cmd)
{
@@ -1377,6 +1398,127 @@ static int dmirror_migrate_to_device(struct dmirror *dmirror,
return ret;
}
+static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range *range)
+{
+ struct migrate_vma *migrate = range->migrate;
+ unsigned long timeout =
+ jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ int ret;
+
+ do {
+ mmap_read_lock(dmirror->notifier.mm);
+
+ /* Fault-in pages for migration */
+ ret = dmirror_range_fault(dmirror, range);
+ /*
+ * Set this up even on error: hmm_range_fault() may have collected
+ * part of the range before failing.
+ */
+ migrate_hmm_range_setup(range);
+ if (ret) {
+ /*
+ * dst[] entries are empty, so this marks every collected
+ * migration as failed. finalize then restores the source
+ * mappings and drops the associated locks/references.
+ */
+ migrate_vma_pages(migrate);
+ migrate_vma_finalize(migrate);
+ goto unlock;
+ }
+
+ pr_debug("Migrating from sys mem to device mem\n");
+
+ dmirror_migrate_alloc_and_copy(migrate, dmirror);
+ migrate_vma_pages(migrate);
+
+ mutex_lock(&dmirror->mutex);
+ if (mmu_interval_read_retry(&dmirror->notifier,
+ range->notifier_seq))
+ ret = -EBUSY;
+ else
+ ret = __dmirror_migrate_map_locked(migrate, dmirror);
+ mutex_unlock(&dmirror->mutex);
+
+ migrate_vma_finalize(migrate);
+unlock:
+ mmap_read_unlock(dmirror->notifier.mm);
+ } while (ret == -EBUSY && !time_after(jiffies, timeout));
+
+ return ret;
+}
+
+static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror,
+ struct hmm_dmirror_cmd *cmd)
+{
+ unsigned long start, size, end, next;
+ unsigned long src_pfns[PFNS_ARRAY_SIZE] = { 0 };
+ unsigned long dst_pfns[PFNS_ARRAY_SIZE] = { 0 };
+ struct migrate_vma migrate = { 0 };
+ struct hmm_range range = { 0 };
+ struct dmirror_bounce bounce;
+ int ret = 0;
+
+ /* Whole range */
+ start = cmd->addr;
+ size = cmd->npages << PAGE_SHIFT;
+ end = start + size;
+
+ if (!mmget_not_zero(dmirror->notifier.mm)) {
+ ret = -EFAULT;
+ goto out;
+ }
+
+ migrate.pgmap_owner = dmirror->mdevice;
+ migrate.src = src_pfns;
+ migrate.dst = dst_pfns;
+ migrate.flags = MIGRATE_VMA_SELECT_SYSTEM;
+
+ range.migrate = &migrate;
+ range.hmm_pfns = src_pfns;
+ range.pfn_flags_mask = 0;
+ range.default_flags = HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE;
+ range.dev_private_owner = dmirror->mdevice;
+ range.notifier = &dmirror->notifier;
+
+ for (next = start; next < end; next = range.end) {
+ range.start = next;
+ range.end = min(end, next + (PFNS_ARRAY_SIZE << PAGE_SHIFT));
+
+ pr_debug("Fault and migrate range start:%#lx end:%#lx\n",
+ range.start, range.end);
+
+ ret = do_fault_and_migrate(dmirror, &range);
+ if (ret)
+ goto out_mmput;
+ }
+
+ /*
+ * Return the migrated data for verification.
+ * Only for pages in device zone
+ */
+ ret = dmirror_bounce_init(&bounce, start, size);
+ if (ret)
+ goto out_mmput;
+
+ mutex_lock(&dmirror->mutex);
+ ret = dmirror_do_read(dmirror, start, end, &bounce);
+ mutex_unlock(&dmirror->mutex);
+ if (ret == 0) {
+ ret = copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, bounce.size);
+ if (ret)
+ ret = -EFAULT;
+ }
+
+ cmd->cpages = bounce.cpages;
+ dmirror_bounce_fini(&bounce);
+
+
+out_mmput:
+ mmput(dmirror->notifier.mm);
+out:
+ return ret;
+}
+
static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *range,
unsigned char *perm, unsigned long entry)
{
@@ -1643,6 +1785,10 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,
ret = dmirror_migrate_to_device(dmirror, &cmd);
break;
+ case HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV:
+ ret = dmirror_fault_and_migrate_to_device(dmirror, &cmd);
+ break;
+
case HMM_DMIRROR_MIGRATE_TO_SYS:
ret = dmirror_migrate_to_system(dmirror, &cmd);
break;
diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h
index ea9b0ec404fb..67d267a3b999 100644
--- a/lib/test_hmm_uapi.h
+++ b/lib/test_hmm_uapi.h
@@ -29,16 +29,17 @@ struct hmm_dmirror_cmd {
};
/* Expose the address space of the calling process through hmm device file */
-#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV _IOWR('H', 0x0a, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0)
diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c
index e2642eca0d02..a2d1b8f494b6 100644
--- a/tools/testing/selftests/mm/hmm-tests.c
+++ b/tools/testing/selftests/mm/hmm-tests.c
@@ -334,6 +334,13 @@ static int hmm_migrate_sys_to_dev(int fd,
return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_TO_DEV, buffer, npages);
}
+static int hmm_migrate_on_fault_sys_to_dev(int fd,
+ struct hmm_buffer *buffer,
+ unsigned long npages)
+{
+ return hmm_dmirror_cmd(fd, HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV, buffer, npages);
+}
+
static int hmm_migrate_dev_to_sys(int fd,
struct hmm_buffer *buffer,
unsigned long npages)
@@ -938,6 +945,53 @@ TEST_F(hmm, migrate)
hmm_buffer_free(buffer);
}
+
+/*
+ * Fault and migrate anonymous memory to device private memory.
+ */
+TEST_F(hmm, migrate_on_fault)
+{
+ struct hmm_buffer *buffer;
+ unsigned long npages;
+ unsigned long size;
+ unsigned long i;
+ int *ptr;
+ int ret;
+
+ npages = ALIGN(HMM_BUFFER_SIZE, self->page_size) >> self->page_shift;
+ ASSERT_NE(npages, 0);
+ size = npages << self->page_shift;
+
+ buffer = malloc(sizeof(*buffer));
+ ASSERT_NE(buffer, NULL);
+
+ buffer->fd = -1;
+ buffer->size = size;
+ buffer->mirror = malloc(size);
+ ASSERT_NE(buffer->mirror, NULL);
+
+ buffer->ptr = mmap(NULL, size,
+ PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS,
+ buffer->fd, 0);
+ ASSERT_NE(buffer->ptr, MAP_FAILED);
+
+ /* Initialize buffer in system memory. */
+ for (i = 0, ptr = buffer->ptr; i < size / sizeof(*ptr); ++i)
+ ptr[i] = i;
+
+ /* Fault and migrate memory to device. */
+ ret = hmm_migrate_on_fault_sys_to_dev(self->fd, buffer, npages);
+ ASSERT_EQ(ret, 0);
+ ASSERT_EQ(buffer->cpages, npages);
+
+ /* Check what the device read. */
+ for (i = 0, ptr = buffer->mirror; i < size / sizeof(*ptr); ++i)
+ ASSERT_EQ(ptr[i], i);
+
+ hmm_buffer_free(buffer);
+}
+
/*
* Migrate private file memory to device private memory.
*/
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 11/12] Documentation/mm/hmm: document migration through hmm_range_fault()
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (9 preceding siblings ...)
2026-10-21 6:05 ` [PATCH v16 10/12] lib/test_hmm: add a new testcase for the migrate on fault mpenttil
@ 2026-10-21 6:05 ` mpenttil
2026-10-21 6:05 ` [PATCH v16 12/12] drm/nouveau/svm: add support for migrate on fault mpenttil
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:05 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
Describe the two new MIGRATE_VMA_FAULT / MIGRATE_VMA_WRITE flags of
migrate_vma_setup(), and add a section on driving the migration
collection phase directly from hmm_range_fault() via HMM_PFN_REQ_MIGRATE
and migrate_hmm_range_setup().
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
Documentation/mm/hmm.rst | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
index fc1b8dc19825..58424e5f5873 100644
--- a/Documentation/mm/hmm.rst
+++ b/Documentation/mm/hmm.rst
@@ -348,6 +348,13 @@ between device driver specific code and shared common code:
Currently only anonymous private VMA ranges can be migrated to or from
system memory and device private memory.
+ By default only pages already present are collected. Two additional flags
+ ask migrate_vma_setup() to fault in missing pages first:
+
+ * ``MIGRATE_VMA_FAULT`` faults in missing pages with read access.
+ * ``MIGRATE_VMA_WRITE`` faults in missing pages with write access
+ (implies faulting).
+
One of the first steps migrate_vma_setup() does is to invalidate other
device's MMUs with the ``mmu_notifier_invalidate_range_start()`` and
``mmu_notifier_invalidate_range_end()`` calls around the page table
@@ -427,6 +434,38 @@ between device driver specific code and shared common code:
The lock can now be released.
+Migration collection through hmm_range_fault()
+==============================================
+
+The collection phase of migration (steps 1 and 2 above) can also be driven by
+hmm_range_fault() directly, sharing its page table walk. This lets a driver
+fault in and collect a range for migration in one walk, which is useful for
+migrate on fault.
+
+To do so, the driver sets ``HMM_PFN_REQ_MIGRATE`` in ``range->default_flags``
+and points ``range->migrate`` at a ``struct migrate_vma`` it has filled in
+(``flags``, ``src``, ``dst``, ``pgmap_owner``). Usually ``HMM_PFN_REQ_FAULT``
+(and ``HMM_PFN_REQ_WRITE``) is set as well, so missing pages are faulted in
+before being collected. The mmap_read_lock() has to be held for the whole
+migration, since the vma must stay stable.
+
+hmm_range_fault() collects the entries the same way migrate_vma_setup() does,
+taking a folio reference, locking it and installing a migration PTE. Collected
+entries are marked with ``HMM_PFN_VALID | HMM_PFN_MIGRATE`` in
+``range->hmm_pfns``. If the page tables change while locks are dropped the
+partially collected entries are rolled back automatically.
+
+After hmm_range_fault() returns, the driver calls::
+
+ void migrate_hmm_range_setup(struct hmm_range *range);
+
+to translate ``range->hmm_pfns`` into ``migrate->src[]`` (``migrate->dst[]`` is
+zeroed) and initialize ``migrate->cpages`` and ``migrate->npages``. From here on
+the ``range->migrate`` struct is ready for the rest of the flow (steps 3
+onwards), i.e. migrate_vma_pages() and migrate_vma_finalize(). This should be
+called even on error, since hmm_range_fault() may have collected part of the
+range before failing.
+
Exclusive access memory
=======================
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v16 12/12] drm/nouveau/svm: add support for migrate on fault
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
` (10 preceding siblings ...)
2026-10-21 6:05 ` [PATCH v16 11/12] Documentation/mm/hmm: document migration through hmm_range_fault() mpenttil
@ 2026-10-21 6:05 ` mpenttil
11 siblings, 0 replies; 19+ messages in thread
From: mpenttil @ 2026-10-21 6:05 UTC (permalink / raw)
To: linux-mm
Cc: dri-devel, intel-xe, linux-kernel, Mika Penttilä,
David Hildenbrand, Jason Gunthorpe, Leon Romanovsky,
Alistair Popple, Balbir Singh, Zi Yan, Matthew Brost,
Andrew Morton, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Lyude Paul,
Danilo Krummrich
From: Mika Penttilä <mpenttil@redhat.com>
Add a direct user for the new HMM_PFN_REQ_MIGRATE flag and
implement migrate on fault for nouveau driver. It is gated
by the svm_migrate_on_fault module parameter, default off.
Added nouveau_dmem_migrate_fault() that faults a range in and
does migrate collecting in a single hmm_range_fault() walk.
As an optimization, a migration policy - deciding which faulting
ranges are worth migrating to VRAM and which are better left
mirrored in system memory - would be useful. This patch
establishes the migrate-on-fault mechanism as a basis for later
refinement.
Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
---
drivers/gpu/drm/nouveau/nouveau_dmem.c | 281 ++++++++++++++++++++++++-
drivers/gpu/drm/nouveau/nouveau_dmem.h | 12 ++
drivers/gpu/drm/nouveau/nouveau_svm.c | 141 +++++++++----
drivers/gpu/drm/nouveau/nouveau_svm.h | 5 +
4 files changed, 384 insertions(+), 55 deletions(-)
diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c
index e74d7bb975a8..998c63e2ae78 100644
--- a/drivers/gpu/drm/nouveau/nouveau_dmem.c
+++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c
@@ -40,6 +40,9 @@
#include <linux/hmm.h>
#include <linux/memremap.h>
#include <linux/migrate.h>
+#include <linux/jiffies.h>
+#include <linux/delay.h>
+#include <linux/mmu_notifier.h>
/*
* FIXME: this is ugly right now we are using TTM to allocate vram and we pin
@@ -139,10 +142,12 @@ static void nouveau_dmem_folio_free(struct folio *folio)
spin_unlock(&dmem->lock);
}
-static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
+static int nouveau_dmem_fence_done(struct nouveau_fence **fence)
{
+ int ret = 0;
+
if (fence) {
- nouveau_fence_wait(*fence, true, false);
+ ret = nouveau_fence_wait(*fence, true, false);
nouveau_fence_unref(fence);
} else {
/*
@@ -150,6 +155,32 @@ static void nouveau_dmem_fence_done(struct nouveau_fence **fence)
* the hmem object.
*/
}
+
+ /* 0 on completion, -EBUSY on timeout, or other errno; propagated. */
+ return ret;
+}
+
+/*
+ * Poll the copy fence instead of the lazy (interrupt-driven) wait since
+ * completion IRQ is randomly missed, hitting the timeout.
+ *
+ * Returns 0 on completion, -EBUSY on timeout. Consumes the fence reference.
+ */
+static int nouveau_dmem_fence_poll(struct nouveau_fence **fence)
+{
+ unsigned long deadline = jiffies + 30 * HZ;
+ int ret = 0;
+
+ while (!nouveau_fence_done(*fence)) {
+ if (time_after(jiffies, deadline)) {
+ ret = -EBUSY;
+ break;
+ }
+ schedule_timeout_uninterruptible(msecs_to_jiffies(1));
+ }
+ nouveau_fence_unref(fence);
+
+ return ret;
}
static int nouveau_dmem_copy_folio(struct nouveau_drm *drm,
@@ -188,7 +219,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
struct nouveau_svmm *svmm;
struct page *dpage;
vm_fault_t ret = 0;
- int err;
+ int err, fret = 0;
struct migrate_vma args = {
.vma = vmf->vma,
.pgmap_owner = drm->dev,
@@ -233,10 +264,12 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
* than just one page on CPU fault. When such fault happens it is very
* likely that more surrounding page will CPU fault too.
*/
- if (migrate_vma_setup(&args) < 0)
- return VM_FAULT_SIGBUS;
+ if (migrate_vma_setup(&args) < 0) {
+ ret = VM_FAULT_SIGBUS;
+ goto err;
+ }
if (!args.cpages)
- return 0;
+ goto done;
if (order)
dpage = folio_page(vma_alloc_folio(GFP_HIGHUSER | __GFP_ZERO,
@@ -266,9 +299,12 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
nouveau_fence_new(&fence, dmem->migrate.chan);
migrate_vma_pages(&args);
- nouveau_dmem_fence_done(&fence);
+ fret = nouveau_dmem_fence_poll(&fence);
dma_unmap_page(drm->dev->dev, dma_info.dma_addr, dma_info.size,
DMA_BIDIRECTIONAL);
+ if (fret)
+ /* Copy failed; the pte now points at possibly stale sysram data. */
+ ret = VM_FAULT_SIGBUS;
done:
migrate_vma_finalize(&args);
err:
@@ -794,11 +830,12 @@ static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
return 0;
}
-static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
+static unsigned long
+nouveau_dmem_migrate_alloc_and_copy(struct nouveau_drm *drm,
struct nouveau_svmm *svmm, struct migrate_vma *args,
- struct nouveau_dmem_dma_info *dma_info, u64 *pfns)
+ struct nouveau_dmem_dma_info *dma_info, u64 *pfns,
+ unsigned long *nr_dma_out, unsigned long *order_out)
{
- struct nouveau_fence *fence;
unsigned long addr = args->start, nr_dma = 0, i;
unsigned long order = 0;
@@ -820,6 +857,21 @@ static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
addr += (1 << order) * PAGE_SIZE;
}
+ *nr_dma_out = nr_dma;
+ *order_out = order;
+ return i;
+}
+
+static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
+ struct nouveau_svmm *svmm, struct migrate_vma *args,
+ struct nouveau_dmem_dma_info *dma_info, u64 *pfns)
+{
+ struct nouveau_fence *fence;
+ unsigned long nr_dma, order, i;
+
+ i = nouveau_dmem_migrate_alloc_and_copy(drm, svmm, args, dma_info, pfns,
+ &nr_dma, &order);
+
nouveau_fence_new(&fence, drm->dmem->migrate.chan);
migrate_vma_pages(args);
nouveau_dmem_fence_done(&fence);
@@ -832,6 +884,215 @@ static void nouveau_dmem_migrate_chunk(struct nouveau_drm *drm,
migrate_vma_finalize(args);
}
+/*
+ * Encode an hmm_range_fault() pfn for @start into the nouveau PFNMAP format,
+ * shared by nouveau_hmm_convert_pfn() and the in-place map in
+ * nouveau_dmem_migrate_fault().
+ */
+u64 nouveau_dmem_hmm_pfn_to_map(unsigned long hmm_pfn, unsigned long start,
+ unsigned long *map_addr, unsigned int *page_shift)
+{
+ unsigned long order;
+ struct page *page;
+ u64 pfn;
+
+ *map_addr = start;
+ *page_shift = PAGE_SHIFT;
+
+ if (!(hmm_pfn & HMM_PFN_VALID))
+ return NVIF_VMM_PFNMAP_V0_NONE;
+
+ page = hmm_pfn_to_page(hmm_pfn);
+
+ order = hmm_pfn_to_map_order(hmm_pfn);
+ if (order) {
+ *page_shift = order + PAGE_SHIFT;
+ *map_addr = start & ~((1UL << *page_shift) - 1);
+ page -= (start - *map_addr) >> PAGE_SHIFT;
+ }
+
+ if (is_device_private_page(page))
+ pfn = nouveau_dmem_page_addr(page) |
+ NVIF_VMM_PFNMAP_V0_V | NVIF_VMM_PFNMAP_V0_VRAM;
+ else
+ pfn = page_to_phys(page) |
+ NVIF_VMM_PFNMAP_V0_V | NVIF_VMM_PFNMAP_V0_HOST;
+ if (hmm_pfn & HMM_PFN_WRITE)
+ pfn |= NVIF_VMM_PFNMAP_V0_W;
+
+ return pfn;
+}
+
+/*
+ * Fault a range in and migrate it to VRAM.
+ * Returns 0 on success, or an error (so the caller
+ * falls back to nouveau_range_fault()) if nothing could be mapped.
+ */
+int
+nouveau_dmem_migrate_fault(struct nouveau_drm *drm, struct nouveau_svmm *svmm,
+ struct mmu_interval_notifier *notifier,
+ unsigned long start, unsigned long end,
+ unsigned long hmm_flags,
+ u64 *out_pfn0, unsigned long *out_addr,
+ unsigned int *out_shift)
+{
+ unsigned long timeout =
+ jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ unsigned long npages = (end - start) >> PAGE_SHIFT;
+ struct mm_struct *mm = notifier->mm;
+ struct nouveau_dmem_dma_info *dma_info;
+ struct nouveau_fence *fence = NULL;
+ struct migrate_vma migrate = {
+ .start = start,
+ .end = end,
+ .pgmap_owner = drm->dev,
+ .flags = MIGRATE_VMA_SELECT_SYSTEM,
+ };
+ struct hmm_range range = {
+ .notifier = notifier,
+ .start = start,
+ .end = end,
+ .dev_private_owner = drm->dev,
+ .migrate = &migrate,
+ .default_flags = hmm_flags | HMM_PFN_REQ_MIGRATE,
+ };
+ unsigned long nr_dma, order, n, i;
+ unsigned long hmm_pfn0 = 0;
+ bool migrated;
+ u64 *pfns;
+ int ret, fret = 0;
+
+ if (!drm->dmem)
+ return -ENODEV;
+
+ migrate.src = kcalloc(npages, sizeof(*migrate.src), GFP_KERNEL);
+ if (!migrate.src)
+ return -ENOMEM;
+ migrate.dst = kcalloc(npages, sizeof(*migrate.dst), GFP_KERNEL);
+ if (!migrate.dst) {
+ ret = -ENOMEM;
+ goto out_free_src;
+ }
+ dma_info = kmalloc_objs(*dma_info, npages);
+ if (!dma_info) {
+ ret = -ENOMEM;
+ goto out_free_dst;
+ }
+ pfns = nouveau_pfns_alloc(npages);
+ if (!pfns) {
+ ret = -ENOMEM;
+ goto out_free_dma;
+ }
+
+ /* hmm_range_fault() shares the source pfn array with migrate_vma. */
+ range.hmm_pfns = migrate.src;
+
+ do {
+ mmap_read_lock(mm);
+ range.notifier_seq = mmu_interval_read_begin(notifier);
+
+ ret = hmm_range_fault(&range);
+ /*
+ * Snapshot the pfn before migrate_hmm_range_setup() overwrites the
+ * shared array; used to map in place if nothing migrates.
+ */
+ if (!ret)
+ hmm_pfn0 = range.hmm_pfns[0];
+ migrate_hmm_range_setup(&range);
+ if (ret) {
+ migrate_vma_pages(&migrate);
+ migrate_vma_finalize(&migrate);
+ goto unlock;
+ }
+
+ n = nouveau_dmem_migrate_alloc_and_copy(drm, svmm, &migrate,
+ dma_info, pfns,
+ &nr_dma, &order);
+
+ /* Only a migrated page has an in-flight CE copy to unblock below. */
+ for (i = 0, migrated = false; i < npages; i++) {
+ if (migrate.dst[i]) {
+ migrated = true;
+ break;
+ }
+ }
+
+ /*
+ * Track a copy fence only when a copy was enqueued; waiting on an
+ * empty fence would block until the fault's CTXSW_TIMEOUT.
+ */
+ if (migrated)
+ nouveau_fence_new(&fence, drm->dmem->migrate.chan);
+ migrate_vma_pages(&migrate);
+
+ /*
+ * The fault-in path must replay the faulting channel before waiting on
+ * the copy fence: with the fault still unreplayed the copy does not
+ * make progress. GPU has no mapping so it might refault, until it
+ * gets the migrated or mirrored mapping.
+ */
+ if (drm->svm && migrated)
+ nouveau_svm_fault_replay(drm->svm);
+
+ if (migrated)
+ fret = nouveau_dmem_fence_poll(&fence);
+
+ mutex_lock(&svmm->mutex);
+ if (fret) {
+ /* Copy did not complete; don't map a half-copied page. */
+ ret = -EIO;
+ } else if (mmu_interval_read_retry(notifier, range.notifier_seq)) {
+ ret = -EBUSY;
+ } else {
+ unsigned long map_addr = migrate.start;
+ unsigned int page_shift = PAGE_SHIFT + order;
+
+ /*
+ * If nothing migrated might still be able to mirror.
+ * In that case use the snapshot hmm_pfn.
+ * npages == 1 always today.
+ */
+ if (npages == 1 && !(pfns[0] & NVIF_VMM_PFNMAP_V0_V))
+ pfns[0] = nouveau_dmem_hmm_pfn_to_map(hmm_pfn0,
+ start, &map_addr, &page_shift);
+
+ __nouveau_pfns_map_locked(svmm, map_addr, pfns, n,
+ page_shift);
+
+ *out_pfn0 = pfns[0];
+ *out_addr = map_addr;
+ *out_shift = page_shift;
+ }
+ mutex_unlock(&svmm->mutex);
+
+ if (fret)
+ NV_ERROR(drm,
+ "svm: migrate copy did not complete (%d) for %lx-%lx\n",
+ fret, start, end);
+
+ while (nr_dma--) {
+ dma_unmap_page(drm->dev->dev, dma_info[nr_dma].dma_addr,
+ dma_info[nr_dma].size, DMA_BIDIRECTIONAL);
+ }
+ migrate_vma_finalize(&migrate);
+unlock:
+ mmap_read_unlock(mm);
+ } while (ret == -EBUSY && !time_after(jiffies, timeout));
+
+ /* Nothing migrated or mappable in place, fallback to nouveau_range_fault(). */
+ if (!ret && !(pfns[0] & NVIF_VMM_PFNMAP_V0_V))
+ ret = -ENOENT;
+
+ nouveau_pfns_free(pfns);
+out_free_dma:
+ kfree(dma_info);
+out_free_dst:
+ kfree(migrate.dst);
+out_free_src:
+ kfree(migrate.src);
+ return ret;
+}
+
int
nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
struct nouveau_svmm *svmm,
diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.h b/drivers/gpu/drm/nouveau/nouveau_dmem.h
index 64da5d3635c8..91e8910cadbf 100644
--- a/drivers/gpu/drm/nouveau/nouveau_dmem.h
+++ b/drivers/gpu/drm/nouveau/nouveau_dmem.h
@@ -27,6 +27,7 @@ struct drm_file;
struct nouveau_drm;
struct nouveau_svmm;
struct hmm_range;
+struct mmu_interval_notifier;
#if IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM)
void nouveau_dmem_init(struct nouveau_drm *);
@@ -39,7 +40,18 @@ int nouveau_dmem_migrate_vma(struct nouveau_drm *drm,
struct vm_area_struct *vma,
unsigned long start,
unsigned long end);
+int nouveau_dmem_migrate_fault(struct nouveau_drm *drm,
+ struct nouveau_svmm *svmm,
+ struct mmu_interval_notifier *notifier,
+ unsigned long start,
+ unsigned long end,
+ unsigned long hmm_flags,
+ u64 *out_pfn0,
+ unsigned long *out_addr,
+ unsigned int *out_shift);
unsigned long nouveau_dmem_page_addr(struct page *page);
+u64 nouveau_dmem_hmm_pfn_to_map(unsigned long hmm_pfn, unsigned long start,
+ unsigned long *map_addr, unsigned int *page_shift);
#else /* IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM) */
static inline void nouveau_dmem_init(struct nouveau_drm *drm) {}
diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c
index c519c28164fa..960cbf8af974 100644
--- a/drivers/gpu/drm/nouveau/nouveau_svm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_svm.c
@@ -37,6 +37,7 @@
#include <linux/hmm.h>
#include <linux/memremap.h>
#include <linux/rmap.h>
+#include <linux/moduleparam.h>
struct nouveau_svm {
struct nouveau_drm *drm;
@@ -78,6 +79,13 @@ struct nouveau_svm {
#define SVM_DBG(s,f,a...) NV_DEBUG((s)->drm, "svm: "f"\n", ##a)
#define SVM_ERR(s,f,a...) NV_WARN((s)->drm, "svm: "f"\n", ##a)
+/* When set, migrate the faulting range to VRAM on GPU fault (default off). */
+static bool nouveau_svm_migrate_on_fault;
+module_param_named(svm_migrate_on_fault, nouveau_svm_migrate_on_fault, bool,
+ 0644);
+MODULE_PARM_DESC(svm_migrate_on_fault,
+ "migrate system memory to VRAM on GPU fault (default off)");
+
struct nouveau_pfnmap_args {
struct nvif_ioctl_v0_hdr i;
struct nvif_ioctl_mthd_v0_hdr m;
@@ -182,9 +190,10 @@ nouveau_svmm_bind(struct drm_device *dev, void *data,
addr = max(addr, vma->vm_start);
next = min(vma->vm_end, end);
- /* This is a best effort so we ignore errors */
- nouveau_dmem_migrate_vma(cli->drm, cli->svm.svmm, vma, addr,
- next);
+ /* Best effort; skipped when migrate-on-fault does the migration. */
+ if (!nouveau_svm_migrate_on_fault)
+ nouveau_dmem_migrate_vma(cli->drm, cli->svm.svmm, vma,
+ addr, next);
addr = next;
}
@@ -390,7 +399,7 @@ nouveau_svmm_init(struct drm_device *dev, void *data,
}
/* Issue fault replay for GPU to retry accesses that faulted previously. */
-static void
+void
nouveau_svm_fault_replay(struct nouveau_svm *svm)
{
SVM_DBG(svm, "replay");
@@ -527,6 +536,14 @@ static bool nouveau_svm_range_invalidate(struct mmu_interval_notifier *mni,
range->owner == sn->svmm->vmm->cli->drm->dev)
return true;
+ /*
+ * Ignore our own migrate-on-fault invalidation; otherwise its
+ * MMU_NOTIFY_MIGRATE bumps the seq and read_retry() spins to timeout.
+ */
+ if (range->event == MMU_NOTIFY_MIGRATE &&
+ range->owner == sn->svmm->vmm->cli->drm->dev)
+ return true;
+
/*
* serializes the update to mni->invalidate_seq done by caller and
* prevents invalidation of the PTE from progressing while HW is being
@@ -551,48 +568,25 @@ static void nouveau_hmm_convert_pfn(struct nouveau_drm *drm,
struct hmm_range *range,
struct nouveau_pfnmap_args *args)
{
- struct page *page;
+ unsigned long map_addr;
+ unsigned int page_shift;
/*
* The address prepared here is passed through nvif_object_ioctl()
* to an eventual DMA map in something like gp100_vmm_pgt_pfn()
*
* This is all just encoding the internal hmm representation into a
- * different nouveau internal representation.
+ * different nouveau internal representation; see the shared encoder
+ * nouveau_dmem_hmm_pfn_to_map() (also widens to the CPU folio order).
*/
- if (!(range->hmm_pfns[0] & HMM_PFN_VALID)) {
- args->p.phys[0] = 0;
- return;
+ args->p.phys[0] = nouveau_dmem_hmm_pfn_to_map(range->hmm_pfns[0],
+ args->p.addr, &map_addr,
+ &page_shift);
+ if (page_shift != PAGE_SHIFT) {
+ args->p.addr = map_addr;
+ args->p.page = page_shift;
+ args->p.size = 1UL << page_shift;
}
-
- page = hmm_pfn_to_page(range->hmm_pfns[0]);
- /*
- * Only map compound pages to the GPU if the CPU is also mapping the
- * page as a compound page. Otherwise, the PTE protections might not be
- * consistent (e.g., CPU only maps part of a compound page).
- * Note that the underlying page might still be larger than the
- * CPU mapping (e.g., a PUD sized compound page partially mapped with
- * a PMD sized page table entry).
- */
- if (hmm_pfn_to_map_order(range->hmm_pfns[0])) {
- unsigned long addr = args->p.addr;
-
- args->p.page = hmm_pfn_to_map_order(range->hmm_pfns[0]) +
- PAGE_SHIFT;
- args->p.size = 1UL << args->p.page;
- args->p.addr &= ~(args->p.size - 1);
- page -= (addr - args->p.addr) >> PAGE_SHIFT;
- }
- if (is_device_private_page(page))
- args->p.phys[0] = nouveau_dmem_page_addr(page) |
- NVIF_VMM_PFNMAP_V0_V |
- NVIF_VMM_PFNMAP_V0_VRAM;
- else
- args->p.phys[0] = page_to_phys(page) |
- NVIF_VMM_PFNMAP_V0_V |
- NVIF_VMM_PFNMAP_V0_HOST;
- if (range->hmm_pfns[0] & HMM_PFN_WRITE)
- args->p.phys[0] |= NVIF_VMM_PFNMAP_V0_W;
}
static int nouveau_atomic_range_fault(struct nouveau_svmm *svmm,
@@ -728,6 +722,44 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm,
return ret;
}
+static int nouveau_range_fault_and_migrate(struct nouveau_svmm *svmm,
+ struct nouveau_drm *drm,
+ struct nouveau_pfnmap_args *args,
+ unsigned long hmm_flags,
+ struct svm_notifier *notifier)
+{
+ struct mm_struct *mm = svmm->notifier.mm;
+ unsigned long start = args->p.addr;
+ unsigned long end = start + args->p.size;
+ unsigned long map_addr;
+ unsigned int page_shift;
+ u64 pfn0;
+ int ret;
+
+ ret = mmu_interval_notifier_insert(¬ifier->notifier, mm,
+ start, args->p.size,
+ &nouveau_svm_mni_ops);
+ if (ret)
+ return ret;
+
+ ret = nouveau_dmem_migrate_fault(drm, svmm, ¬ifier->notifier,
+ start, end, hmm_flags,
+ &pfn0, &map_addr, &page_shift);
+
+ if (!ret) {
+ args->p.phys[0] = pfn0;
+ if (page_shift != PAGE_SHIFT) {
+ args->p.addr = map_addr;
+ args->p.page = page_shift;
+ args->p.size = 1UL << page_shift;
+ }
+ }
+
+ mmu_interval_notifier_remove(¬ifier->notifier);
+
+ return ret;
+}
+
static void
nouveau_svm_fault(struct work_struct *work)
{
@@ -843,14 +875,25 @@ nouveau_svm_fault(struct work_struct *work)
}
notifier.svmm = svmm;
- if (atomic)
+ if (atomic) {
ret = nouveau_atomic_range_fault(svmm, svm->drm, args,
__struct_size(args),
¬ifier);
- else
+ } else if (nouveau_svm_migrate_on_fault &&
+ buffer->fault[fi]->access != FAULT_ACCESS_PREFETCH) {
+ /* Fault+migrate in one walk; fall back if nothing migrated. */
+ ret = nouveau_range_fault_and_migrate(svmm, svm->drm,
+ args, hmm_flags,
+ ¬ifier);
+ if (ret)
+ ret = nouveau_range_fault(svmm, svm->drm, args,
+ __struct_size(args),
+ hmm_flags, ¬ifier);
+ } else {
ret = nouveau_range_fault(svmm, svm->drm, args,
__struct_size(args),
hmm_flags, ¬ifier);
+ }
mmput(mm);
limit = args->p.addr + args->p.size;
@@ -935,21 +978,29 @@ nouveau_pfns_free(u64 *pfns)
}
void
-nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
- unsigned long addr, u64 *pfns, unsigned long npages,
- unsigned int page_shift)
+__nouveau_pfns_map_locked(struct nouveau_svmm *svmm, unsigned long addr,
+ u64 *pfns, unsigned long npages,
+ unsigned int page_shift)
{
struct nouveau_pfnmap_args *args = nouveau_pfns_to_args(pfns);
+ lockdep_assert_held(&svmm->mutex);
+
args->p.addr = addr;
args->p.size = npages << page_shift;
args->p.page = page_shift;
- mutex_lock(&svmm->mutex);
-
nvif_object_ioctl(&svmm->vmm->vmm.object, args,
struct_size(args, p.phys, npages), NULL);
+}
+void
+nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
+ unsigned long addr, u64 *pfns, unsigned long npages,
+ unsigned int page_shift)
+{
+ mutex_lock(&svmm->mutex);
+ __nouveau_pfns_map_locked(svmm, addr, pfns, npages, page_shift);
mutex_unlock(&svmm->mutex);
}
diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.h b/drivers/gpu/drm/nouveau/nouveau_svm.h
index 3fd78662f17e..0e7ff03f80ea 100644
--- a/drivers/gpu/drm/nouveau/nouveau_svm.h
+++ b/drivers/gpu/drm/nouveau/nouveau_svm.h
@@ -5,6 +5,7 @@
struct drm_device;
struct drm_file;
struct nouveau_drm;
+struct nouveau_svm;
struct nouveau_svmm {
struct mmu_notifier notifier;
@@ -35,6 +36,10 @@ void nouveau_pfns_free(u64 *pfns);
void nouveau_pfns_map(struct nouveau_svmm *svmm, struct mm_struct *mm,
unsigned long addr, u64 *pfns, unsigned long npages,
unsigned int page_shift);
+void __nouveau_pfns_map_locked(struct nouveau_svmm *svmm, unsigned long addr,
+ u64 *pfns, unsigned long npages,
+ unsigned int page_shift);
+void nouveau_svm_fault_replay(struct nouveau_svm *svm);
#else /* IS_ENABLED(CONFIG_DRM_NOUVEAU_SVM) */
static inline void nouveau_svm_init(struct nouveau_drm *drm) {}
static inline void nouveau_svm_fini(struct nouveau_drm *drm) {}
--
2.55.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v16 09/12] mm: enable device page migration from HMM pagewalk
2026-10-21 6:05 ` [PATCH v16 09/12] mm: enable device page migration from " mpenttil
@ 2026-10-21 6:19 ` Balbir Singh
2026-10-21 6:30 ` Mika Penttilä
0 siblings, 1 reply; 19+ messages in thread
From: Balbir Singh @ 2026-10-21 6:19 UTC (permalink / raw)
To: mpenttil, linux-mm
Cc: dri-devel, intel-xe, linux-kernel, David Hildenbrand,
Jason Gunthorpe, Leon Romanovsky, Alistair Popple, Zi Yan,
Matthew Brost, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Lyude Paul, Danilo Krummrich
On 10/21/26 5:05 PM, mpenttil@redhat.com wrote:
> From: Mika Penttilä <mpenttil@redhat.com>
>
> HMM pagewalk has now the machinery to do the first phase of
> device page migration, collecting the pfns and installing
> migration ptes.
>
> Enable migration in hmm_range_fault(), and change migrate_vma_setup()
> to use HMM pagewalk path. Two new flags, MIGRATE_VMA_FAULT and
> MIGRATE_VMA_WRITE are introduced for migrate_vma_setup(),
> to request for faulting missing pages, and requesting write
> access.
>
> Also, the migrate_vma_collect*() based paths are now unused,
> so delete them.
>
> Cc: David Hildenbrand <david@kernel.org>
> Cc: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Leon Romanovsky <leonro@nvidia.com>
> Cc: Alistair Popple <apopple@nvidia.com>
> Cc: Balbir Singh <balbirs@nvidia.com>
> Cc: Zi Yan <ziy@nvidia.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Suggested-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
> ---
> include/linux/migrate.h | 10 +-
> mm/hmm.c | 33 ++-
> mm/migrate_device.c | 562 ++++------------------------------------
> 3 files changed, 83 insertions(+), 522 deletions(-)
>
I got a conflict applying this patch on the latest mm-everything with
mm: avoid use of VMA_SPECIAL_FLAGS in migrate_vma_setup(), quite a simple fixup
Balbir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v16 09/12] mm: enable device page migration from HMM pagewalk
2026-10-21 6:19 ` Balbir Singh
@ 2026-10-21 6:30 ` Mika Penttilä
0 siblings, 0 replies; 19+ messages in thread
From: Mika Penttilä @ 2026-10-21 6:30 UTC (permalink / raw)
To: Balbir Singh, linux-mm
Cc: dri-devel, intel-xe, linux-kernel, David Hildenbrand,
Jason Gunthorpe, Leon Romanovsky, Alistair Popple, Zi Yan,
Matthew Brost, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Lyude Paul, Danilo Krummrich
On 10/21/26 09:19, Balbir Singh wrote:
> On 10/21/26 5:05 PM, mpenttil@redhat.com wrote:
>> From: Mika Penttilä <mpenttil@redhat.com>
>>
>> HMM pagewalk has now the machinery to do the first phase of
>> device page migration, collecting the pfns and installing
>> migration ptes.
>>
>> Enable migration in hmm_range_fault(), and change migrate_vma_setup()
>> to use HMM pagewalk path. Two new flags, MIGRATE_VMA_FAULT and
>> MIGRATE_VMA_WRITE are introduced for migrate_vma_setup(),
>> to request for faulting missing pages, and requesting write
>> access.
>>
>> Also, the migrate_vma_collect*() based paths are now unused,
>> so delete them.
>>
>> Cc: David Hildenbrand <david@kernel.org>
>> Cc: Jason Gunthorpe <jgg@nvidia.com>
>> Cc: Leon Romanovsky <leonro@nvidia.com>
>> Cc: Alistair Popple <apopple@nvidia.com>
>> Cc: Balbir Singh <balbirs@nvidia.com>
>> Cc: Zi Yan <ziy@nvidia.com>
>> Cc: Matthew Brost <matthew.brost@intel.com>
>> Suggested-by: Alistair Popple <apopple@nvidia.com>
>> Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
>> ---
>> include/linux/migrate.h | 10 +-
>> mm/hmm.c | 33 ++-
>> mm/migrate_device.c | 562 ++++------------------------------------
>> 3 files changed, 83 insertions(+), 522 deletions(-)
>>
> I got a conflict applying this patch on the latest mm-everything with
> mm: avoid use of VMA_SPECIAL_FLAGS in migrate_vma_setup(), quite a simple fixup
>
> Balbir
Ah okay - yes this is on drm-tip
--Mika
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v16 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level
2026-10-21 6:04 ` [PATCH v16 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level mpenttil
@ 2026-10-21 6:56 ` Balbir Singh
2026-10-21 7:09 ` Mika Penttilä
0 siblings, 1 reply; 19+ messages in thread
From: Balbir Singh @ 2026-10-21 6:56 UTC (permalink / raw)
To: mpenttil, linux-mm
Cc: dri-devel, intel-xe, linux-kernel, David Hildenbrand,
Jason Gunthorpe, Leon Romanovsky, Alistair Popple, Zi Yan,
Matthew Brost, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Lyude Paul, Danilo Krummrich
On 10/21/26 5:04 PM, mpenttil@redhat.com wrote:
> From: Mika Penttilä <mpenttil@redhat.com>
>
> Implement the needed hmm_vma_handle_migrate_prepare() function
> which is mostly carried over from migrate_device.c's
> migrate_vma_collect_pmd() function.
>
> Also implement the migrate_vma_split_folio(), for splitting
> pte mapped large folios. It is also mostly from migrate_device.c,
> with care taken to reference folio before relasing page table
> lock.
Typo - releasing
>
> With HMM pagewalk based migration, the idea is that
> hmm_vma_handle_*() are responsible for faulting,
> and the pfn collecting part. hmm_vma_handle_migrate_prepare*()
> do the migration decisions (with HMM_PFN_MIGRATE), possibly split
> folios, and insert migration ptes/pmds.
>
> HMM pagewalk based migration is enabled in later commit, for now
> now hmm_select_migrate() returns 0.
>
now now -> now
> Cc: David Hildenbrand <david@kernel.org>
> Cc: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Leon Romanovsky <leonro@nvidia.com>
> Cc: Alistair Popple <apopple@nvidia.com>
> Cc: Balbir Singh <balbirs@nvidia.com>
> Cc: Zi Yan <ziy@nvidia.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Suggested-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
> ---
> mm/hmm.c | 263 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 262 insertions(+), 1 deletion(-)
>
> diff --git a/mm/hmm.c b/mm/hmm.c
> index 119e70dbca56..0ce947d0e572 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c
> @@ -491,6 +491,63 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
> #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
>
> #ifdef CONFIG_DEVICE_MIGRATION
> +/**
> + * migrate_vma_split_folio() - Helper function to split a THP folio
> + * @folio: the folio to split
> + * @fault_page: struct page associated with the fault if any
> + * @hmm_vma_walk: walk in progress
> + * @ptep: pte_t * for unmap and unlock ptl
> + *
> + * Returns 0 on success
and drops the pte_lock associated with the pte
> + */
> +static int migrate_vma_split_folio(struct folio *folio,
> + struct page *fault_page,
> + struct hmm_vma_walk *hmm_vma_walk,
> + pte_t *ptep)
> +{
> + int ret;
> + struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
> + struct folio *new_fault_folio = NULL;
> +
> + if (folio != fault_folio)
> + folio_get(folio);
> +
> + pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
> + hmm_vma_walk->ptelocked = false;
> +
> + if (folio != fault_folio)
> + folio_lock(folio);
> +
Could you add a comment on why the get and lock is split into separate steps
with the pte lock dropped in between
> + ret = split_folio(folio);
> + if (ret) {
> + if (folio != fault_folio) {
> + folio_unlock(folio);
> + folio_put(folio);
> + }
> + return ret;
> + }
> +
> + new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
> +
> + /*
> + * Ensure the lock is held on the correct
> + * folio after the split
> + */
> + if (!new_fault_folio) {
> + folio_unlock(folio);
> + folio_put(folio);
> + } else if (folio != new_fault_folio) {
> + if (new_fault_folio != fault_folio) {
> + folio_get(new_fault_folio);
> + folio_lock(new_fault_folio);
> + }
> + folio_unlock(folio);
> + folio_put(folio);
> + }
> +
> + return 0;
> +}
> +
> static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
> pmd_t *pmdp,
> unsigned long start,
> @@ -501,6 +558,11 @@ static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
> return 0;
> }
>
> +/*
> + * Install migration entries if migration requested, either from fault
> + * or migrate paths.
> + *
> + */
> static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
> pmd_t *pmdp,
> pte_t *ptep,
> @@ -508,8 +570,207 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
> unsigned long *hmm_pfn,
> bool *unmapped)
> {
> - // TODO: implement migration entry insertion
> + struct hmm_vma_walk *hmm_vma_walk = walk->private;
> + struct hmm_range *range = hmm_vma_walk->range;
> + struct migrate_vma *migrate = range->migrate;
> + struct mm_struct *mm = walk->vma->vm_mm;
> + struct folio *fault_folio = NULL;
> + enum migrate_vma_info minfo;
> + struct dev_pagemap *pgmap;
> + bool anon_exclusive;
> + struct folio *folio;
> + unsigned long pfn;
> + struct page *page;
> + softleaf_t entry;
> + pte_t pte, swp_pte;
> + bool writable = false;
> +
> + // Do we want to migrate at all?
> + minfo = hmm_select_migrate(range);
> + if (!minfo)
> + return 0;
> +
> + WARN_ON_ONCE(!migrate);
> + HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, true);
> +
> + fault_folio = migrate->fault_page ?
> + page_folio(migrate->fault_page) : NULL;
> +
> + pte = ptep_get(ptep);
> +
> + if (pte_none(pte)) {
> + if (vma_is_anonymous(walk->vma)) {
> + *hmm_pfn &= HMM_PFN_INOUT_FLAGS;
> + *hmm_pfn |= HMM_PFN_MIGRATE;
> + goto out;
> + }
> + }
> +
> + if (!(hmm_pfn[0] & HMM_PFN_VALID))
> + goto out;
> +
We use *hmm_pfn and then hmm_pfn[0], could we be consistent. Based on this I
assuming that there is just one entry in hmm_pfn?
> + if (!pte_present(pte)) {
> + /*
> + * Only care about unaddressable device page special
> + * page table entry. Other special swap entries are not
> + * migratable, and we ignore regular swapped page.
> + */
> + entry = softleaf_from_pte(pte);
> + if (!softleaf_is_device_private(entry))
> + goto out;
> +
> + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE))
> + goto out;
> +
> + page = softleaf_to_page(entry);
> + folio = page_folio(page);
> + if (folio->pgmap->owner != migrate->pgmap_owner)
> + goto out;
> +
> + if (folio_test_large(folio)) {
> + int ret;
> +
> + ret = migrate_vma_split_folio(folio,
> + migrate->fault_page,
> + hmm_vma_walk,
> + ptep);
> + if (ret)
> + goto out_error;
> + return -EAGAIN;
> + }
> +
> + pfn = page_to_pfn(page);
> + if (softleaf_is_device_private_write(entry))
> + writable = true;
> + } else {
> + pfn = pte_pfn(pte);
> + if (is_zero_pfn(pfn) &&
> + (minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
> + *hmm_pfn = HMM_PFN_MIGRATE;
> + goto out;
> + }
> + page = vm_normal_page(walk->vma, addr, pte);
> + if (page && !is_zone_device_page(page) &&
> + !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
> + goto out;
> + } else if (page && is_device_coherent_page(page)) {
> + pgmap = page_pgmap(page);
> +
> + if (!(minfo &
> + MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
> + pgmap->owner != migrate->pgmap_owner)
> + goto out;
> + }
> +
> + folio = page ? page_folio(page) : NULL;
> + if (folio && folio_test_large(folio)) {
> + int ret;
> +
> + ret = migrate_vma_split_folio(folio,
> + migrate->fault_page,
> + hmm_vma_walk,
> + ptep);
> + if (ret)
> + goto out_error;
> + return -EAGAIN;
> + }
> +
> + writable = pte_write(pte);
> + }
> +
> + if (!page || !page->mapping)
> + goto out;
> +
> + /*
> + * By getting a reference on the folio we pin it and that blocks
> + * any kind of migration. Side effect is that it "freezes" the
> + * pte.
> + *
> + * We drop this reference after isolating the folio from the lru
> + * for non device folio (device folio are not on the lru and thus
> + * can't be dropped from it).
> + */
> + folio = page_folio(page);
> + folio_get(folio);
> +
> + /*
> + * We rely on folio_trylock() to avoid deadlock between
> + * concurrent migrations where each is waiting on the others
> + * folio lock. If we can't immediately lock the folio we fail this
> + * migration as it is only best effort anyway.
> + *
> + * If we can lock the folio it's safe to set up a migration entry
> + * now. In the common case where the folio is mapped once in a
> + * single process setting up the migration entry now is an
> + * optimisation to avoid walking the rmap later with
> + * try_to_migrate().
> + */
> +
> + if (fault_folio == folio || folio_trylock(folio)) {
> + anon_exclusive = folio_test_anon(folio) &&
> + PageAnonExclusive(page);
> +
> + if (pte_present(pte))
> + flush_cache_page(walk->vma, addr, pfn);
> +
> + if (anon_exclusive) {
> + pte = ptep_clear_flush(walk->vma, addr, ptep);
> +
> + if (folio_try_share_anon_rmap_pte(folio, page)) {
> + set_pte_at(mm, addr, ptep, pte);
> + if (fault_folio != folio)
> + folio_unlock(folio);
> + folio_put(folio);
> + goto out;
> + }
> + } else {
> + pte = ptep_get_and_clear(mm, addr, ptep);
> + }
> +
> + if (pte_present(pte) && pte_dirty(pte))
> + folio_mark_dirty(folio);
> +
> + /* Setup special migration page table entry */
> + if (writable)
> + entry = make_writable_migration_entry(pfn);
> + else if (anon_exclusive)
> + entry = make_readable_exclusive_migration_entry(pfn);
> + else
> + entry = make_readable_migration_entry(pfn);
> +
> + if (pte_present(pte)) {
> + if (pte_young(pte))
> + entry = make_migration_entry_young(entry);
> + if (pte_dirty(pte))
> + entry = make_migration_entry_dirty(entry);
> + }
> +
> + swp_pte = swp_entry_to_pte(entry);
> + if (pte_present(pte)) {
> + if (pte_soft_dirty(pte))
> + swp_pte = pte_swp_mksoft_dirty(swp_pte);
> + if (pte_uffd(pte))
> + swp_pte = pte_swp_mkuffd(swp_pte);
> + } else {
> + if (pte_swp_soft_dirty(pte))
> + swp_pte = pte_swp_mksoft_dirty(swp_pte);
> + if (pte_swp_uffd(pte))
> + swp_pte = pte_swp_mkuffd(swp_pte);
> + }
> +
> + set_pte_at(mm, addr, ptep, swp_pte);
> + folio_remove_rmap_pte(folio, page, walk->vma);
> + folio_put(folio);
> + *hmm_pfn |= HMM_PFN_MIGRATE;
> + if (pte_present(pte))
> + *unmapped = true;
> + } else {
> + folio_put(folio);
> + }
> +out:
> return 0;
> +out_error:
> + return -EFAULT;
> }
>
> #else
Thanks,
Balbir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v16 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level
2026-10-21 6:56 ` Balbir Singh
@ 2026-10-21 7:09 ` Mika Penttilä
0 siblings, 0 replies; 19+ messages in thread
From: Mika Penttilä @ 2026-10-21 7:09 UTC (permalink / raw)
To: Balbir Singh, linux-mm
Cc: dri-devel, intel-xe, linux-kernel, David Hildenbrand,
Jason Gunthorpe, Leon Romanovsky, Alistair Popple, Zi Yan,
Matthew Brost, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Lyude Paul, Danilo Krummrich
On 10/21/26 09:56, Balbir Singh wrote:
> On 10/21/26 5:04 PM, mpenttil@redhat.com wrote:
>> From: Mika Penttilä <mpenttil@redhat.com>
>>
>> Implement the needed hmm_vma_handle_migrate_prepare() function
>> which is mostly carried over from migrate_device.c's
>> migrate_vma_collect_pmd() function.
>>
>> Also implement the migrate_vma_split_folio(), for splitting
>> pte mapped large folios. It is also mostly from migrate_device.c,
>> with care taken to reference folio before relasing page table
>> lock.
> Typo - releasing
ack
>
>> With HMM pagewalk based migration, the idea is that
>> hmm_vma_handle_*() are responsible for faulting,
>> and the pfn collecting part. hmm_vma_handle_migrate_prepare*()
>> do the migration decisions (with HMM_PFN_MIGRATE), possibly split
>> folios, and insert migration ptes/pmds.
>>
>> HMM pagewalk based migration is enabled in later commit, for now
>> now hmm_select_migrate() returns 0.
>>
> now now -> now
thanks
>
>> Cc: David Hildenbrand <david@kernel.org>
>> Cc: Jason Gunthorpe <jgg@nvidia.com>
>> Cc: Leon Romanovsky <leonro@nvidia.com>
>> Cc: Alistair Popple <apopple@nvidia.com>
>> Cc: Balbir Singh <balbirs@nvidia.com>
>> Cc: Zi Yan <ziy@nvidia.com>
>> Cc: Matthew Brost <matthew.brost@intel.com>
>> Suggested-by: Alistair Popple <apopple@nvidia.com>
>> Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
>> ---
>> mm/hmm.c | 263 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>> 1 file changed, 262 insertions(+), 1 deletion(-)
>>
>> diff --git a/mm/hmm.c b/mm/hmm.c
>> index 119e70dbca56..0ce947d0e572 100644
>> --- a/mm/hmm.c
>> +++ b/mm/hmm.c
>> @@ -491,6 +491,63 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
>> #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
>>
>> #ifdef CONFIG_DEVICE_MIGRATION
>> +/**
>> + * migrate_vma_split_folio() - Helper function to split a THP folio
>> + * @folio: the folio to split
>> + * @fault_page: struct page associated with the fault if any
>> + * @hmm_vma_walk: walk in progress
>> + * @ptep: pte_t * for unmap and unlock ptl
>> + *
>> + * Returns 0 on success
> and drops the pte_lock associated with the pte
agreed
>
>> + */
>> +static int migrate_vma_split_folio(struct folio *folio,
>> + struct page *fault_page,
>> + struct hmm_vma_walk *hmm_vma_walk,
>> + pte_t *ptep)
>> +{
>> + int ret;
>> + struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
>> + struct folio *new_fault_folio = NULL;
>> +
>> + if (folio != fault_folio)
>> + folio_get(folio);
>> +
>> + pte_unmap_unlock(ptep, hmm_vma_walk->ptl);
>> + hmm_vma_walk->ptelocked = false;
>> +
>> + if (folio != fault_folio)
>> + folio_lock(folio);
>> +
> Could you add a comment on why the get and lock is split into separate steps
> with the pte lock dropped in between
yes will add
>> + ret = split_folio(folio);
>> + if (ret) {
>> + if (folio != fault_folio) {
>> + folio_unlock(folio);
>> + folio_put(folio);
>> + }
>> + return ret;
>> + }
>> +
>> + new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
>> +
>> + /*
>> + * Ensure the lock is held on the correct
>> + * folio after the split
>> + */
>> + if (!new_fault_folio) {
>> + folio_unlock(folio);
>> + folio_put(folio);
>> + } else if (folio != new_fault_folio) {
>> + if (new_fault_folio != fault_folio) {
>> + folio_get(new_fault_folio);
>> + folio_lock(new_fault_folio);
>> + }
>> + folio_unlock(folio);
>> + folio_put(folio);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
>> pmd_t *pmdp,
>> unsigned long start,
>> @@ -501,6 +558,11 @@ static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk,
>> return 0;
>> }
>>
>> +/*
>> + * Install migration entries if migration requested, either from fault
>> + * or migrate paths.
>> + *
>> + */
>> static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
>> pmd_t *pmdp,
>> pte_t *ptep,
>> @@ -508,8 +570,207 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk,
>> unsigned long *hmm_pfn,
>> bool *unmapped)
>> {
>> - // TODO: implement migration entry insertion
>> + struct hmm_vma_walk *hmm_vma_walk = walk->private;
>> + struct hmm_range *range = hmm_vma_walk->range;
>> + struct migrate_vma *migrate = range->migrate;
>> + struct mm_struct *mm = walk->vma->vm_mm;
>> + struct folio *fault_folio = NULL;
>> + enum migrate_vma_info minfo;
>> + struct dev_pagemap *pgmap;
>> + bool anon_exclusive;
>> + struct folio *folio;
>> + unsigned long pfn;
>> + struct page *page;
>> + softleaf_t entry;
>> + pte_t pte, swp_pte;
>> + bool writable = false;
>> +
>> + // Do we want to migrate at all?
>> + minfo = hmm_select_migrate(range);
>> + if (!minfo)
>> + return 0;
>> +
>> + WARN_ON_ONCE(!migrate);
>> + HMM_ASSERT_PTE_LOCKED(hmm_vma_walk, true);
>> +
>> + fault_folio = migrate->fault_page ?
>> + page_folio(migrate->fault_page) : NULL;
>> +
>> + pte = ptep_get(ptep);
>> +
>> + if (pte_none(pte)) {
>> + if (vma_is_anonymous(walk->vma)) {
>> + *hmm_pfn &= HMM_PFN_INOUT_FLAGS;
>> + *hmm_pfn |= HMM_PFN_MIGRATE;
>> + goto out;
>> + }
>> + }
>> +
>> + if (!(hmm_pfn[0] & HMM_PFN_VALID))
>> + goto out;
>> +
> We use *hmm_pfn and then hmm_pfn[0], could we be consistent. Based on this I
> assuming that there is just one entry in hmm_pfn?
yes
>
>> + if (!pte_present(pte)) {
>> + /*
>> + * Only care about unaddressable device page special
>> + * page table entry. Other special swap entries are not
>> + * migratable, and we ignore regular swapped page.
>> + */
>> + entry = softleaf_from_pte(pte);
>> + if (!softleaf_is_device_private(entry))
>> + goto out;
>> +
>> + if (!(minfo & MIGRATE_VMA_SELECT_DEVICE_PRIVATE))
>> + goto out;
>> +
>> + page = softleaf_to_page(entry);
>> + folio = page_folio(page);
>> + if (folio->pgmap->owner != migrate->pgmap_owner)
>> + goto out;
>> +
>> + if (folio_test_large(folio)) {
>> + int ret;
>> +
>> + ret = migrate_vma_split_folio(folio,
>> + migrate->fault_page,
>> + hmm_vma_walk,
>> + ptep);
>> + if (ret)
>> + goto out_error;
>> + return -EAGAIN;
>> + }
>> +
>> + pfn = page_to_pfn(page);
>> + if (softleaf_is_device_private_write(entry))
>> + writable = true;
>> + } else {
>> + pfn = pte_pfn(pte);
>> + if (is_zero_pfn(pfn) &&
>> + (minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
>> + *hmm_pfn = HMM_PFN_MIGRATE;
>> + goto out;
>> + }
>> + page = vm_normal_page(walk->vma, addr, pte);
>> + if (page && !is_zone_device_page(page) &&
>> + !(minfo & MIGRATE_VMA_SELECT_SYSTEM)) {
>> + goto out;
>> + } else if (page && is_device_coherent_page(page)) {
>> + pgmap = page_pgmap(page);
>> +
>> + if (!(minfo &
>> + MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
>> + pgmap->owner != migrate->pgmap_owner)
>> + goto out;
>> + }
>> +
>> + folio = page ? page_folio(page) : NULL;
>> + if (folio && folio_test_large(folio)) {
>> + int ret;
>> +
>> + ret = migrate_vma_split_folio(folio,
>> + migrate->fault_page,
>> + hmm_vma_walk,
>> + ptep);
>> + if (ret)
>> + goto out_error;
>> + return -EAGAIN;
>> + }
>> +
>> + writable = pte_write(pte);
>> + }
>> +
>> + if (!page || !page->mapping)
>> + goto out;
>> +
>> + /*
>> + * By getting a reference on the folio we pin it and that blocks
>> + * any kind of migration. Side effect is that it "freezes" the
>> + * pte.
>> + *
>> + * We drop this reference after isolating the folio from the lru
>> + * for non device folio (device folio are not on the lru and thus
>> + * can't be dropped from it).
>> + */
>> + folio = page_folio(page);
>> + folio_get(folio);
>> +
>> + /*
>> + * We rely on folio_trylock() to avoid deadlock between
>> + * concurrent migrations where each is waiting on the others
>> + * folio lock. If we can't immediately lock the folio we fail this
>> + * migration as it is only best effort anyway.
>> + *
>> + * If we can lock the folio it's safe to set up a migration entry
>> + * now. In the common case where the folio is mapped once in a
>> + * single process setting up the migration entry now is an
>> + * optimisation to avoid walking the rmap later with
>> + * try_to_migrate().
>> + */
>> +
>> + if (fault_folio == folio || folio_trylock(folio)) {
>> + anon_exclusive = folio_test_anon(folio) &&
>> + PageAnonExclusive(page);
>> +
>> + if (pte_present(pte))
>> + flush_cache_page(walk->vma, addr, pfn);
>> +
>> + if (anon_exclusive) {
>> + pte = ptep_clear_flush(walk->vma, addr, ptep);
>> +
>> + if (folio_try_share_anon_rmap_pte(folio, page)) {
>> + set_pte_at(mm, addr, ptep, pte);
>> + if (fault_folio != folio)
>> + folio_unlock(folio);
>> + folio_put(folio);
>> + goto out;
>> + }
>> + } else {
>> + pte = ptep_get_and_clear(mm, addr, ptep);
>> + }
>> +
>> + if (pte_present(pte) && pte_dirty(pte))
>> + folio_mark_dirty(folio);
>> +
>> + /* Setup special migration page table entry */
>> + if (writable)
>> + entry = make_writable_migration_entry(pfn);
>> + else if (anon_exclusive)
>> + entry = make_readable_exclusive_migration_entry(pfn);
>> + else
>> + entry = make_readable_migration_entry(pfn);
>> +
>> + if (pte_present(pte)) {
>> + if (pte_young(pte))
>> + entry = make_migration_entry_young(entry);
>> + if (pte_dirty(pte))
>> + entry = make_migration_entry_dirty(entry);
>> + }
>> +
>> + swp_pte = swp_entry_to_pte(entry);
>> + if (pte_present(pte)) {
>> + if (pte_soft_dirty(pte))
>> + swp_pte = pte_swp_mksoft_dirty(swp_pte);
>> + if (pte_uffd(pte))
>> + swp_pte = pte_swp_mkuffd(swp_pte);
>> + } else {
>> + if (pte_swp_soft_dirty(pte))
>> + swp_pte = pte_swp_mksoft_dirty(swp_pte);
>> + if (pte_swp_uffd(pte))
>> + swp_pte = pte_swp_mkuffd(swp_pte);
>> + }
>> +
>> + set_pte_at(mm, addr, ptep, swp_pte);
>> + folio_remove_rmap_pte(folio, page, walk->vma);
>> + folio_put(folio);
>> + *hmm_pfn |= HMM_PFN_MIGRATE;
>> + if (pte_present(pte))
>> + *unmapped = true;
>> + } else {
>> + folio_put(folio);
>> + }
>> +out:
>> return 0;
>> +out_error:
>> + return -EFAULT;
>> }
>>
>> #else
> Thanks,
> Balbir
Thanks Balbir!
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v16 03/12] mm/hmm: preparations for HMM to participate in migration
2026-10-21 6:04 ` [PATCH v16 03/12] mm/hmm: preparations for HMM to participate in migration mpenttil
@ 2026-10-21 7:17 ` Balbir Singh
2026-10-21 8:11 ` Mika Penttilä
0 siblings, 1 reply; 19+ messages in thread
From: Balbir Singh @ 2026-10-21 7:17 UTC (permalink / raw)
To: mpenttil, linux-mm
Cc: dri-devel, intel-xe, linux-kernel, David Hildenbrand,
Jason Gunthorpe, Leon Romanovsky, Alistair Popple, Zi Yan,
Matthew Brost, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Lyude Paul, Danilo Krummrich
On 10/21/26 5:04 PM, mpenttil@redhat.com wrote:
> From: Mika Penttilä <mpenttil@redhat.com>
>
> For migration to happen after hmm_range_fault(), the vma has to
> retrieved. Luckily, pagewalk already resolves that for us, so
> just have to save it for further use. For migration, mmap_lock
> has to be hold for the whole operation, so vma stays stable.
> In case lock-drop behavior is requested, return an error while
> migrating. Also, rename some struct hmm_vma_walk fields to solve
> namespace collisions, and we are later introducing more
> xxlocked fields so be unambiguous.
>
> Take care also of firing the mmu_notifier when migrating.
>
> Also, prepare for differencies of how mm_struct * is retrieved
> in fault and migrate_vma paths.
>
> Cc: David Hildenbrand <david@kernel.org>
> Cc: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Leon Romanovsky <leonro@nvidia.com>
> Cc: Alistair Popple <apopple@nvidia.com>
> Cc: Balbir Singh <balbirs@nvidia.com>
> Cc: Zi Yan <ziy@nvidia.com>
> Cc: Matthew Brost <matthew.brost@intel.com>
> Suggested-by: Alistair Popple <apopple@nvidia.com>
> Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
> ---
> include/linux/migrate.h | 50 +++++++++++++++++-
> lib/test_hmm.c | 2 +-
> mm/hmm.c | 110 ++++++++++++++++++++++++++++++++++------
> 3 files changed, 145 insertions(+), 17 deletions(-)
>
> diff --git a/include/linux/migrate.h b/include/linux/migrate.h
> index 280ec9a866c7..dd7dd7ec6995 100644
> --- a/include/linux/migrate.h
> +++ b/include/linux/migrate.h
> @@ -5,6 +5,7 @@
> #include <linux/mm.h>
> #include <linux/hmm.h>
> #include <linux/mempolicy.h>
> +#include <linux/mmu_notifier.h>
> #include <linux/migrate_mode.h>
> #include <linux/hugetlb.h>
>
> @@ -108,6 +109,29 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
> spin_unlock(ptl);
> }
>
> +enum migrate_vma_info {
> + MIGRATE_VMA_SELECT_NONE = 0,
> + MIGRATE_VMA_SELECT_COMPOUND = MIGRATE_VMA_SELECT_NONE,
> +};
> +
> +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
> +{
> + return MIGRATE_VMA_SELECT_NONE;
> +}
> +
> +static inline void hmm_fill_migrate_vma(struct hmm_range *range,
> + struct vm_area_struct *vma,
> + unsigned long start,
> + unsigned long end)
> +{
> +}
> +
> +#ifdef CONFIG_MMU_NOTIFIER
> +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
> +{
> + return range->notifier->mm;
> +}
> +#endif
> #endif /* CONFIG_MIGRATION */
>
> #ifdef CONFIG_NUMA_BALANCING
> @@ -151,7 +175,8 @@ static inline unsigned long migrate_pfn(unsigned long pfn)
> return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID;
> }
>
> -enum migrate_vma_direction {
> +enum migrate_vma_info {
> + MIGRATE_VMA_SELECT_NONE = 0,
> MIGRATE_VMA_SELECT_SYSTEM = 1 << 0,
> MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
> MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
> @@ -193,6 +218,29 @@ struct migrate_vma {
> struct page *fault_page;
> };
>
> +// TODO: enable migration
> +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
> +{
> + return 0;
> +}
> +
> +#ifdef CONFIG_MMU_NOTIFIER
> +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
> +{
> + return range->notifier ? range->notifier->mm : range->migrate->vma->vm_mm;
> +}
> +#endif
> +
> +static inline void hmm_fill_migrate_vma(struct hmm_range *range,
> + struct vm_area_struct *vma,
> + unsigned long start,
> + unsigned long end)
> +{
> + range->migrate->vma = vma;
> + range->migrate->start = start;
> + range->migrate->end = end;
> +}
> +
> int migrate_vma_setup(struct migrate_vma *args);
> void migrate_vma_pages(struct migrate_vma *migrate);
> void migrate_vma_finalize(struct migrate_vma *migrate);
> diff --git a/lib/test_hmm.c b/lib/test_hmm.c
> index 6911daa9f854..cd88e8177d0a 100644
> --- a/lib/test_hmm.c
> +++ b/lib/test_hmm.c
> @@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_device *mdevice)
> HMM_DMIRROR_MEMORY_DEVICE_PRIVATE);
> }
>
> -static enum migrate_vma_direction
> +static enum migrate_vma_info
> dmirror_select_device(struct dmirror *dmirror)
> {
> return (dmirror->mdevice->zone_device_type ==
> diff --git a/mm/hmm.c b/mm/hmm.c
> index 2f1e98c6b644..4e9542aedc25 100644
> --- a/mm/hmm.c
> +++ b/mm/hmm.c
> @@ -21,6 +21,7 @@
> #include <linux/pagemap.h>
> #include <linux/leafops.h>
> #include <linux/hugetlb.h>
> +#include <linux/migrate.h>
> #include <linux/memremap.h>
> #include <linux/sched/mm.h>
> #include <linux/jump_label.h>
> @@ -28,15 +29,23 @@
> #include <linux/pci-p2pdma.h>
> #include <linux/mmu_notifier.h>
> #include <linux/memory_hotplug.h>
> +#include <asm/tlbflush.h>
>
> #include "internal.h"
>
> struct hmm_vma_walk {
> - struct hmm_range *range;
> - bool *locked;
> - unsigned long last;
> - unsigned long end;
> - unsigned int required_fault;
> + struct mmu_notifier_range mmu_range;
> + /* Set once mmu_range is armed with mmu_notifier_invalidate_range_start() */
> + bool mmu_armed;
> + struct vm_area_struct *vma;
> + struct hmm_range *range;
> + unsigned long start;
> + unsigned long end;
> + unsigned long last;
> + /* fault and lock drop related fields */
> + bool *mmlocked;
> + unsigned long fault_end;
> + unsigned int required_fault;
> };
>
> /*
> @@ -95,7 +104,7 @@ static int hmm_record_fault(unsigned long addr, unsigned long end,
>
> WARN_ON_ONCE(!required_fault);
> hmm_vma_walk->last = addr;
> - hmm_vma_walk->end = end;
> + hmm_vma_walk->fault_end = end;
> hmm_vma_walk->required_fault = required_fault;
> return HMM_FAULT_PENDING;
> }
> @@ -400,6 +409,58 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
> }
> #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
>
> +static int hmm_vma_capture_migrate_range(unsigned long start,
> + unsigned long end,
> + struct mm_walk *walk)
> +{
> + struct hmm_vma_walk *hmm_vma_walk = walk->private;
> + struct hmm_range *range = hmm_vma_walk->range;
> +
> + if (!hmm_select_migrate(range))
> + return 0;
> +
> + if (hmm_vma_walk->vma && (hmm_vma_walk->vma != walk->vma))
> + return -ERANGE;
> +
> + hmm_vma_walk->vma = walk->vma;
I don't quite follow the check for -ERANGE, it checks for hmm_vma_walk->vma !=
walk->vma, but then set's it later to be the same?
> +
> + if (end - start > range->end - range->start)
> + return -ERANGE;
> +
Have we established that end > start and same for range->end > range->start?
> + if (!hmm_vma_walk->mmu_armed) {
> + hmm_vma_walk->start = start;
> + hmm_vma_walk->end = end;
Does the walk range matter here?
> + mmu_notifier_range_init_owner(&hmm_vma_walk->mmu_range, MMU_NOTIFY_MIGRATE, 0,
> + walk->vma->vm_mm, start, end,
> + range->dev_private_owner);
> + mmu_notifier_invalidate_range_start(&hmm_vma_walk->mmu_range);
> + hmm_vma_walk->mmu_armed = true;
> + }
> +
> + return 0;
> +}
> +
> +static void hmm_vma_post_range_fault(struct hmm_vma_walk *hmm_vma_walk)
> +{
> +
> + struct hmm_range *range = hmm_vma_walk->range;
> +
> + if (hmm_select_migrate(range) &&
> + hmm_vma_walk->mmu_armed) {
> + /*
> + * The migrate_vma path has the following initialized,
> + * so take care of fault path below.
> + */
> + if (range->notifier) {
> + hmm_fill_migrate_vma(range,
> + hmm_vma_walk->vma,
> + hmm_vma_walk->start,
> + hmm_vma_walk->end);
> + }
> + mmu_notifier_invalidate_range_end(&hmm_vma_walk->mmu_range);
> + }
> +}
> +
> static int hmm_vma_walk_pmd(pmd_t *pmdp,
> unsigned long start,
> unsigned long end,
> @@ -594,6 +655,11 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end,
> struct hmm_vma_walk *hmm_vma_walk = walk->private;
> struct hmm_range *range = hmm_vma_walk->range;
> struct vm_area_struct *vma = walk->vma;
> + int r;
> +
> + r = hmm_vma_capture_migrate_range(start, end, walk);
> + if (r)
> + return r;
>
> if (!(vma->vm_flags & (VM_IO | VM_PFNMAP)) &&
> vma->vm_flags & VM_READ)
> @@ -646,12 +712,12 @@ static int hmm_do_fault(struct mm_struct *mm,
> struct hmm_vma_walk *hmm_vma_walk)
> {
> unsigned long addr = hmm_vma_walk->last;
> - unsigned long end = hmm_vma_walk->end;
> + unsigned long end = hmm_vma_walk->fault_end;
> unsigned int required_fault = hmm_vma_walk->required_fault;
> unsigned int fault_flags = FAULT_FLAG_REMOTE;
> struct vm_area_struct *vma;
>
> - if (hmm_vma_walk->locked)
> + if (hmm_vma_walk->mmlocked)
> fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
>
> vma = vma_lookup(mm, addr);
> @@ -670,8 +736,8 @@ static int hmm_do_fault(struct mm_struct *mm,
> ret = handle_mm_fault(vma, addr, fault_flags, NULL);
>
> if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
> - if (hmm_vma_walk->locked) /* needed by sparse */
> - *hmm_vma_walk->locked = false;
> + if (hmm_vma_walk->mmlocked) /* needed by sparse */
> + *hmm_vma_walk->mmlocked = false;
> else
> WARN_ON_ONCE(1); /* broken fault handler */
> return HMM_FAULT_UNLOCKED;
> @@ -694,19 +760,31 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
> {
> struct hmm_vma_walk hmm_vma_walk = {
> .range = range,
> - .locked = locked,
> + .mmlocked = locked,
> .last = range->start,
> };
> - struct mm_struct *mm = range->notifier->mm;
> + /*
> + * Could be serving a device fault or come from migrate
> + * entry point. For the former we have not resolved the vma
> + * yet, and the latter we don't have a notifier (but have a vma).
> + *
> + */
> + struct mm_struct *mm = hmm_range_fault_mm(range);
> int ret;
>
> + /* Migration is incompatible with mmap lock drop */
> + if (locked && hmm_select_migrate(range))
> + return -EINVAL;
> +
> mmap_assert_locked(mm);
>
> do {
> /* If range is no longer valid force retry. */
> - if (mmu_interval_check_retry(range->notifier,
> - range->notifier_seq))
> - return -EBUSY;
> + if (range->notifier && mmu_interval_check_retry(range->notifier,
> + range->notifier_seq)) {
> + ret = -EBUSY;
> + break;
> + }
> ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
> &hmm_walk_ops, &hmm_vma_walk);
> /*
> @@ -732,6 +810,8 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
> * output, and all >= are still at their input values.
> */
> } while (ret == -EBUSY);
> +
> + hmm_vma_post_range_fault(&hmm_vma_walk);
Have you checked to see if we could end up with notifier start/end inbalance?
For example if hmm_select_migrate returns false?
> return ret;
> }
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v16 03/12] mm/hmm: preparations for HMM to participate in migration
2026-10-21 7:17 ` Balbir Singh
@ 2026-10-21 8:11 ` Mika Penttilä
0 siblings, 0 replies; 19+ messages in thread
From: Mika Penttilä @ 2026-10-21 8:11 UTC (permalink / raw)
To: Balbir Singh, linux-mm
Cc: dri-devel, intel-xe, linux-kernel, David Hildenbrand,
Jason Gunthorpe, Leon Romanovsky, Alistair Popple, Zi Yan,
Matthew Brost, Andrew Morton, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Lyude Paul, Danilo Krummrich
On 10/21/26 10:17, Balbir Singh wrote:
> On 10/21/26 5:04 PM, mpenttil@redhat.com wrote:
>> From: Mika Penttilä <mpenttil@redhat.com>
>>
>> For migration to happen after hmm_range_fault(), the vma has to
>> retrieved. Luckily, pagewalk already resolves that for us, so
>> just have to save it for further use. For migration, mmap_lock
>> has to be hold for the whole operation, so vma stays stable.
>> In case lock-drop behavior is requested, return an error while
>> migrating. Also, rename some struct hmm_vma_walk fields to solve
>> namespace collisions, and we are later introducing more
>> xxlocked fields so be unambiguous.
>>
>> Take care also of firing the mmu_notifier when migrating.
>>
>> Also, prepare for differencies of how mm_struct * is retrieved
>> in fault and migrate_vma paths.
>>
>> Cc: David Hildenbrand <david@kernel.org>
>> Cc: Jason Gunthorpe <jgg@nvidia.com>
>> Cc: Leon Romanovsky <leonro@nvidia.com>
>> Cc: Alistair Popple <apopple@nvidia.com>
>> Cc: Balbir Singh <balbirs@nvidia.com>
>> Cc: Zi Yan <ziy@nvidia.com>
>> Cc: Matthew Brost <matthew.brost@intel.com>
>> Suggested-by: Alistair Popple <apopple@nvidia.com>
>> Signed-off-by: Mika Penttilä <mpenttil@redhat.com>
>> ---
>> include/linux/migrate.h | 50 +++++++++++++++++-
>> lib/test_hmm.c | 2 +-
>> mm/hmm.c | 110 ++++++++++++++++++++++++++++++++++------
>> 3 files changed, 145 insertions(+), 17 deletions(-)
>>
>> diff --git a/include/linux/migrate.h b/include/linux/migrate.h
>> index 280ec9a866c7..dd7dd7ec6995 100644
>> --- a/include/linux/migrate.h
>> +++ b/include/linux/migrate.h
>> @@ -5,6 +5,7 @@
>> #include <linux/mm.h>
>> #include <linux/hmm.h>
>> #include <linux/mempolicy.h>
>> +#include <linux/mmu_notifier.h>
>> #include <linux/migrate_mode.h>
>> #include <linux/hugetlb.h>
>>
>> @@ -108,6 +109,29 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
>> spin_unlock(ptl);
>> }
>>
>> +enum migrate_vma_info {
>> + MIGRATE_VMA_SELECT_NONE = 0,
>> + MIGRATE_VMA_SELECT_COMPOUND = MIGRATE_VMA_SELECT_NONE,
>> +};
>> +
>> +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
>> +{
>> + return MIGRATE_VMA_SELECT_NONE;
>> +}
>> +
>> +static inline void hmm_fill_migrate_vma(struct hmm_range *range,
>> + struct vm_area_struct *vma,
>> + unsigned long start,
>> + unsigned long end)
>> +{
>> +}
>> +
>> +#ifdef CONFIG_MMU_NOTIFIER
>> +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
>> +{
>> + return range->notifier->mm;
>> +}
>> +#endif
>> #endif /* CONFIG_MIGRATION */
>>
>> #ifdef CONFIG_NUMA_BALANCING
>> @@ -151,7 +175,8 @@ static inline unsigned long migrate_pfn(unsigned long pfn)
>> return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID;
>> }
>>
>> -enum migrate_vma_direction {
>> +enum migrate_vma_info {
>> + MIGRATE_VMA_SELECT_NONE = 0,
>> MIGRATE_VMA_SELECT_SYSTEM = 1 << 0,
>> MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
>> MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
>> @@ -193,6 +218,29 @@ struct migrate_vma {
>> struct page *fault_page;
>> };
>>
>> +// TODO: enable migration
>> +static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
>> +{
>> + return 0;
>> +}
>> +
>> +#ifdef CONFIG_MMU_NOTIFIER
>> +static inline struct mm_struct *hmm_range_fault_mm(struct hmm_range *range)
>> +{
>> + return range->notifier ? range->notifier->mm : range->migrate->vma->vm_mm;
>> +}
>> +#endif
>> +
>> +static inline void hmm_fill_migrate_vma(struct hmm_range *range,
>> + struct vm_area_struct *vma,
>> + unsigned long start,
>> + unsigned long end)
>> +{
>> + range->migrate->vma = vma;
>> + range->migrate->start = start;
>> + range->migrate->end = end;
>> +}
>> +
>> int migrate_vma_setup(struct migrate_vma *args);
>> void migrate_vma_pages(struct migrate_vma *migrate);
>> void migrate_vma_finalize(struct migrate_vma *migrate);
>> diff --git a/lib/test_hmm.c b/lib/test_hmm.c
>> index 6911daa9f854..cd88e8177d0a 100644
>> --- a/lib/test_hmm.c
>> +++ b/lib/test_hmm.c
>> @@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_device *mdevice)
>> HMM_DMIRROR_MEMORY_DEVICE_PRIVATE);
>> }
>>
>> -static enum migrate_vma_direction
>> +static enum migrate_vma_info
>> dmirror_select_device(struct dmirror *dmirror)
>> {
>> return (dmirror->mdevice->zone_device_type ==
>> diff --git a/mm/hmm.c b/mm/hmm.c
>> index 2f1e98c6b644..4e9542aedc25 100644
>> --- a/mm/hmm.c
>> +++ b/mm/hmm.c
>> @@ -21,6 +21,7 @@
>> #include <linux/pagemap.h>
>> #include <linux/leafops.h>
>> #include <linux/hugetlb.h>
>> +#include <linux/migrate.h>
>> #include <linux/memremap.h>
>> #include <linux/sched/mm.h>
>> #include <linux/jump_label.h>
>> @@ -28,15 +29,23 @@
>> #include <linux/pci-p2pdma.h>
>> #include <linux/mmu_notifier.h>
>> #include <linux/memory_hotplug.h>
>> +#include <asm/tlbflush.h>
>>
>> #include "internal.h"
>>
>> struct hmm_vma_walk {
>> - struct hmm_range *range;
>> - bool *locked;
>> - unsigned long last;
>> - unsigned long end;
>> - unsigned int required_fault;
>> + struct mmu_notifier_range mmu_range;
>> + /* Set once mmu_range is armed with mmu_notifier_invalidate_range_start() */
>> + bool mmu_armed;
>> + struct vm_area_struct *vma;
>> + struct hmm_range *range;
>> + unsigned long start;
>> + unsigned long end;
>> + unsigned long last;
>> + /* fault and lock drop related fields */
>> + bool *mmlocked;
>> + unsigned long fault_end;
>> + unsigned int required_fault;
>> };
>>
>> /*
>> @@ -95,7 +104,7 @@ static int hmm_record_fault(unsigned long addr, unsigned long end,
>>
>> WARN_ON_ONCE(!required_fault);
>> hmm_vma_walk->last = addr;
>> - hmm_vma_walk->end = end;
>> + hmm_vma_walk->fault_end = end;
>> hmm_vma_walk->required_fault = required_fault;
>> return HMM_FAULT_PENDING;
>> }
>> @@ -400,6 +409,58 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
>> }
>> #endif /* CONFIG_ARCH_HAS_PMD_SOFTLEAVES */
>>
>> +static int hmm_vma_capture_migrate_range(unsigned long start,
>> + unsigned long end,
>> + struct mm_walk *walk)
>> +{
>> + struct hmm_vma_walk *hmm_vma_walk = walk->private;
>> + struct hmm_range *range = hmm_vma_walk->range;
>> +
>> + if (!hmm_select_migrate(range))
>> + return 0;
>> +
>> + if (hmm_vma_walk->vma && (hmm_vma_walk->vma != walk->vma))
>> + return -ERANGE;
>> +
>> + hmm_vma_walk->vma = walk->vma;
> I don't quite follow the check for -ERANGE, it checks for hmm_vma_walk->vma !=
> walk->vma, but then set's it later to be the same?
The intention is to catch a range that crosses a vma boundary. hmm_vma_walk lives across the restart loop in hmm_range_fault_locked(),
so test_walk can run again for the same vma after an -EBUSY re-walk. The first call records the vma, a re-walk of the same vma passes,
and a different vma means the range spans more than one vma, which migration doesn't support, so -ERANGE.
The assignment only takes effect on the first call. I'll restructure it as "record if unset, else -ERANGE if different" and add a comment.
>
>> +
>> + if (end - start > range->end - range->start)
>> + return -ERANGE;
>> +
> Have we established that end > start and same for range->end > range->start?
Yes. walk_page_range() rejects start >= end before calling any callback, and hmm_range_fault_locked() walks from hmm_vma_walk.last
(initially range->start) to range->end, so range->start < range->end holds whenever test_walk runs.
The test_walk start/end are a sub-range of that walk clamped to the vma, so start < end too.
That also means end - start can never exceed range->end - range->start, so the check is redundant. I'll drop it in v17.
>
>> + if (!hmm_vma_walk->mmu_armed) {
>> + hmm_vma_walk->start = start;
>> + hmm_vma_walk->end = end;
> Does the walk range matter here?
The walk range is the caller's range clamped to the vma: with the caller's range inside a single vma the two are the same,
and the clamp keeps the walk range's end within the vma (migrate_vma needs that) if the caller's range extends past
the vma into a hole. The walk range is used as both the MMU_NOTIFY_MIGRATE invalidation range and the migrate_vma start/end,
and is set once on the first test_walk call. I'll also add a check that the walk range starts at range->start,
since migrate_vma indexes src[] from there.
>> + mmu_notifier_range_init_owner(&hmm_vma_walk->mmu_range, MMU_NOTIFY_MIGRATE, 0,
>> + walk->vma->vm_mm, start, end,
>> + range->dev_private_owner);
>> + mmu_notifier_invalidate_range_start(&hmm_vma_walk->mmu_range);
>> + hmm_vma_walk->mmu_armed = true;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static void hmm_vma_post_range_fault(struct hmm_vma_walk *hmm_vma_walk)
>> +{
>> +
>> + struct hmm_range *range = hmm_vma_walk->range;
>> +
>> + if (hmm_select_migrate(range) &&
>> + hmm_vma_walk->mmu_armed) {
>> + /*
>> + * The migrate_vma path has the following initialized,
>> + * so take care of fault path below.
>> + */
>> + if (range->notifier) {
>> + hmm_fill_migrate_vma(range,
>> + hmm_vma_walk->vma,
>> + hmm_vma_walk->start,
>> + hmm_vma_walk->end);
>> + }
>> + mmu_notifier_invalidate_range_end(&hmm_vma_walk->mmu_range);
>> + }
>> +}
>> +
>> static int hmm_vma_walk_pmd(pmd_t *pmdp,
>> unsigned long start,
>> unsigned long end,
>> @@ -594,6 +655,11 @@ static int hmm_vma_walk_test(unsigned long start, unsigned long end,
>> struct hmm_vma_walk *hmm_vma_walk = walk->private;
>> struct hmm_range *range = hmm_vma_walk->range;
>> struct vm_area_struct *vma = walk->vma;
>> + int r;
>> +
>> + r = hmm_vma_capture_migrate_range(start, end, walk);
>> + if (r)
>> + return r;
>>
>> if (!(vma->vm_flags & (VM_IO | VM_PFNMAP)) &&
>> vma->vm_flags & VM_READ)
>> @@ -646,12 +712,12 @@ static int hmm_do_fault(struct mm_struct *mm,
>> struct hmm_vma_walk *hmm_vma_walk)
>> {
>> unsigned long addr = hmm_vma_walk->last;
>> - unsigned long end = hmm_vma_walk->end;
>> + unsigned long end = hmm_vma_walk->fault_end;
>> unsigned int required_fault = hmm_vma_walk->required_fault;
>> unsigned int fault_flags = FAULT_FLAG_REMOTE;
>> struct vm_area_struct *vma;
>>
>> - if (hmm_vma_walk->locked)
>> + if (hmm_vma_walk->mmlocked)
>> fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
>>
>> vma = vma_lookup(mm, addr);
>> @@ -670,8 +736,8 @@ static int hmm_do_fault(struct mm_struct *mm,
>> ret = handle_mm_fault(vma, addr, fault_flags, NULL);
>>
>> if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
>> - if (hmm_vma_walk->locked) /* needed by sparse */
>> - *hmm_vma_walk->locked = false;
>> + if (hmm_vma_walk->mmlocked) /* needed by sparse */
>> + *hmm_vma_walk->mmlocked = false;
>> else
>> WARN_ON_ONCE(1); /* broken fault handler */
>> return HMM_FAULT_UNLOCKED;
>> @@ -694,19 +760,31 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
>> {
>> struct hmm_vma_walk hmm_vma_walk = {
>> .range = range,
>> - .locked = locked,
>> + .mmlocked = locked,
>> .last = range->start,
>> };
>> - struct mm_struct *mm = range->notifier->mm;
>> + /*
>> + * Could be serving a device fault or come from migrate
>> + * entry point. For the former we have not resolved the vma
>> + * yet, and the latter we don't have a notifier (but have a vma).
>> + *
>> + */
>> + struct mm_struct *mm = hmm_range_fault_mm(range);
>> int ret;
>>
>> + /* Migration is incompatible with mmap lock drop */
>> + if (locked && hmm_select_migrate(range))
>> + return -EINVAL;
>> +
>> mmap_assert_locked(mm);
>>
>> do {
>> /* If range is no longer valid force retry. */
>> - if (mmu_interval_check_retry(range->notifier,
>> - range->notifier_seq))
>> - return -EBUSY;
>> + if (range->notifier && mmu_interval_check_retry(range->notifier,
>> + range->notifier_seq)) {
>> + ret = -EBUSY;
>> + break;
>> + }
>> ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
>> &hmm_walk_ops, &hmm_vma_walk);
>> /*
>> @@ -732,6 +810,8 @@ static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
>> * output, and all >= are still at their input values.
>> */
>> } while (ret == -EBUSY);
>> +
>> + hmm_vma_post_range_fault(&hmm_vma_walk);
> Have you checked to see if we could end up with notifier start/end inbalance?
> For example if hmm_select_migrate returns false?
Yes. invalidate_range_start() is only called from hmm_vma_capture_migrate_range(), after the hmm_select_migrate()
check and only once (mmu_armed). invalidate_range_end() is only called from hmm_vma_post_range_fault() when mmu_armed
is set. hmm_select_migrate() doesn't change during the call, so if it returns false neither is called.
Every exit from the loop goes through hmm_vma_post_range_fault(); except after test for lock drop which happens before walks.
>
>> return ret;
>> }
>>
Thanks,
--Mika
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2026-10-21 8:12 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-21 6:04 [PATCH v16 00/12] migrate on fault for device pages mpenttil
2026-10-21 6:04 ` [PATCH v16 01/12] mm/Kconfig: changes for " mpenttil
2026-10-21 6:04 ` [PATCH v16 02/12] mm: add helper to convert HMM pfn to migrate pfn mpenttil
2026-10-21 6:04 ` [PATCH v16 03/12] mm/hmm: preparations for HMM to participate in migration mpenttil
2026-10-21 7:17 ` Balbir Singh
2026-10-21 8:11 ` Mika Penttilä
2026-10-21 6:04 ` [PATCH v16 04/12] mm/hmm: do the plumbing " mpenttil
2026-10-21 6:04 ` [PATCH v16 05/12] mm/hmm: migrate collection in HMM pagewalk - pte level mpenttil
2026-10-21 6:56 ` Balbir Singh
2026-10-21 7:09 ` Mika Penttilä
2026-10-21 6:05 ` [PATCH v16 06/12] mm/hmm: migrate collection in HMM pagewalk - pmd level mpenttil
2026-10-21 6:05 ` [PATCH v16 07/12] mm/hmm: add lazy MMU mode support for migration in HMM pagewalk mpenttil
2026-10-21 6:05 ` [PATCH v16 08/12] mm/hmm: implement rollback for device page " mpenttil
2026-10-21 6:05 ` [PATCH v16 09/12] mm: enable device page migration from " mpenttil
2026-10-21 6:19 ` Balbir Singh
2026-10-21 6:30 ` Mika Penttilä
2026-10-21 6:05 ` [PATCH v16 10/12] lib/test_hmm: add a new testcase for the migrate on fault mpenttil
2026-10-21 6:05 ` [PATCH v16 11/12] Documentation/mm/hmm: document migration through hmm_range_fault() mpenttil
2026-10-21 6:05 ` [PATCH v16 12/12] drm/nouveau/svm: add support for migrate on fault mpenttil
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®