[PATCH v3 0/7] sched/numa: stop VMA scan filters from gating promotion

[PATCH v3 0/7] sched/numa: stop VMA scan filters from gating promotion

[Gregory Price]

NUMA balancing uses hinting faults for both task placement and
memory-tier promotion. Several filters designed to avoid unproductive
socket-placement faults also prevent promotion.

In a production deployment using kernel.numa_balancing=2 on a host
with 768 GB of DRAM and 256 GB of CXL memory running two ~430 GB
database workloads with large shmem VMAs - we discovered poor NUMA
balancing behavior:

Before patch series:
  - 150-200 GB/s DRAM bandwidth
  - 40-45 GB/s CXL bandwidth, saturating the device
  - request latency above 5 ms with longer tails

After patch series:
  - 250+ GB/s sustained DRAM bandwidth
  - approximately 5-10 GB/s sustained CXL bandwidth
  - request latency between 800 us and 2 ms

The core issue is that the global balancing mode says which mechanisms
are enabled, but it cannot describe the intent of an individual
VMA/PTE walk (placement vs promotion).

This plumbs the scan reasoning into the prot_none injection while
retaining the placement-scan filtering.

Patches 1-5 form the backportable fix set:

  1. Add promotion-only NUMA protection walks and derive private VMA
     state in the folio eligibility check.
  2. Permit eligible shared folios to be promoted to a fast tier and
     name the lower-tier predicate according to its use.
  3. Scan mappings covered by the legacy file placement filter using
     promotion-only scans.
  4. Separate VMA placement eligibility from partial-scan continuation.
  5. Scan PID-inactive VMAs for promotion and account for placement scans
     separately.

Patch 4 and 5 are technically part of the same fix, but are separated to
make it easier to review - they must be backported together.

Patches 6-7 are independently requested cleanups.

  6. Use BIT() for the change_protection() flags.
  7. Use the VMA flag API in the touched NUMA-balancing code.

Gregory Price (Meta) (7):
  mm: support promotion-only NUMA hinting scans
  mm: allow shared folios to be promoted to a fast tier
  sched/numa: scan read-only file mappings in tiering mode
  sched/numa: separate VMA placement from scan continuation
  sched/numa: scan PID-inactive VMAs for promotion
  mm: use BIT() for change_protection() flags
  mm: use VMA flag helpers in NUMA balancing

 include/linux/mm.h       | 23 +++++-----
 include/linux/mm_types.h | 10 +++++
 kernel/sched/fair.c      | 95 ++++++++++++++++++++++++++++------------
 mm/huge_memory.c         |  3 +-
 mm/internal.h            |  6 +--
 mm/memory-tiers.c        |  9 ++--
 mm/memory.c              |  2 +-
 mm/mempolicy.c           | 37 +++++++++-------
 mm/migrate.c             | 14 ++++--
 mm/mprotect.c            |  7 +--
 10 files changed, 133 insertions(+), 73 deletions(-)

-- 
2.53.0-Meta

base-commit: 8d29b5365d528da545c1fe0768a55babadee95d0

[PATCH v3 1/7] mm: support promotion-only NUMA hinting scans

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

folio_can_map_prot_numa() derives folio eligibility from the global
balancing mode. The mode (normal, tiering, or combined) describes
which balancing mechanisms are enabled (placement vs promotion).

The global setting itself cannot describe the intent of an individual
protection walk because normal-mode tempers its scanning activity based
on a number of heuristics (read-only VMAs, activeness of VMA, etc).

In tiering or combined mode, applying these normal-mode optimizations
to an entire VMA is incorrect and breaks tiering.

 - Opting VMAs out of scanning because they became inactive obviously
   breaks tiering - because the intent is to identify when a VMA
   becomes active.  Applying it in tiering modes causes

 - Opting Read-only file VMAs out of scanning is just incorrect for
   tiering modes because its intent is to prevent bouncing between
   sockets (east-west), while tiering controls tier migration
   (north-south).  The result is hot read-only files can overload
   lower tier bandwidth.

Add MM_CP_PROT_NUMA_PROMO_ONLY and let task_numa_work() select the type
of walk. Build the change-protection flags there and pass them unchanged
through change_prot_numa() so its PTE/PMD paths use the same decision.

Have folio_can_map_prot_numa() derive the single-threaded private state at
the point of use instead of adding another precomputed boolean to the
protection-walk interface. This keeps the interface focused on scan intent
and avoids plumbing VMA-derived state beside cp_flags.

The tradeoff for the cleaner interface is an atomic mm_users read for each
folio, rather than once per PTE range. Check the promotion-only top-tier
exclusion bit first as a mild optimization.

The scan-intent interface is required by the following memory-tiering
fixes and must accompany them when backported.

Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
Cc: stable@vger.kernel.org
Suggested-by: David Hildenbrand <david@kernel.org>
Link: https://lore.kernel.org/r/4d2853c9-edf4-4685-b186-7214ed6abd84@kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 include/linux/mm.h  |  6 ++++--
 kernel/sched/fair.c |  7 ++++++-
 mm/huge_memory.c    |  3 +--
 mm/internal.h       |  4 ++--
 mm/mempolicy.c      | 27 ++++++++++++---------------
 mm/mprotect.c       |  7 +------
 6 files changed, 26 insertions(+), 28 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index e3d29f87567f8..beab621e6f88d 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3608,6 +3608,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 #define  MM_CP_UFFD_RWP_RESOLVE            (1UL << 5) /* resolve rwp */
 #define  MM_CP_UFFD_RWP_ALL                (MM_CP_UFFD_RWP | \
 					    MM_CP_UFFD_RWP_RESOLVE)
+/* Whether a MM_CP_PROT_NUMA change is for promotion only */
+#define  MM_CP_PROT_NUMA_PROMO_ONLY        (1UL << 6)
 
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
@@ -4980,8 +4982,8 @@ static inline void vma_set_page_prot(struct vm_area_struct *vma)
 void vma_set_file(struct vm_area_struct *vma, struct file *file);
 
 #ifdef CONFIG_NUMA_BALANCING
-unsigned long change_prot_numa(struct vm_area_struct *vma,
-			unsigned long start, unsigned long end);
+unsigned long change_prot_numa(struct vm_area_struct *vma, unsigned long start,
+		unsigned long end, unsigned long cp_flags);
 #endif
 
 struct vm_area_struct *find_extend_vma_locked(struct mm_struct *,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ae6c1a606eb5d..9f544e3df9490 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4124,6 +4124,7 @@ static void task_numa_work(struct callback_head *work)
 	struct mm_struct *mm = p->mm;
 	u64 runtime = p->se.sum_exec_runtime;
 	struct vm_area_struct *vma;
+	unsigned long cp_flags = MM_CP_PROT_NUMA;
 	unsigned long start, end;
 	unsigned long nr_pte_updates = 0;
 	long pages, virtpages;
@@ -4131,6 +4132,9 @@ static void task_numa_work(struct callback_head *work)
 	bool vma_pids_skipped;
 	bool vma_pids_forced = false;
 
+	if (!(READ_ONCE(sysctl_numa_balancing_mode) & NUMA_BALANCING_NORMAL))
+		cp_flags |= MM_CP_PROT_NUMA_PROMO_ONLY;
+
 	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
 
 	work->next = work;
@@ -4307,7 +4311,8 @@ static void task_numa_work(struct callback_head *work)
 			start = max(start, vma->vm_start);
 			end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
 			end = min(end, vma->vm_end);
-			nr_pte_updates = change_prot_numa(vma, start, end);
+			nr_pte_updates = change_prot_numa(vma, start, end,
+							  cp_flags);
 
 			/*
 			 * Try to scan sysctl_numa_balancing_size worth of
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 493b29cabee59..da9cf87903a55 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2769,8 +2769,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		if (is_huge_zero_pmd(*pmd))
 			goto unlock;
 
-		if (!folio_can_map_prot_numa(pmd_folio(*pmd), vma,
-					     vma_is_single_threaded_private(vma)))
+		if (!folio_can_map_prot_numa(pmd_folio(*pmd), vma, cp_flags))
 			goto unlock;
 	}
 	/*
diff --git a/mm/internal.h b/mm/internal.h
index 0434dfcfc36f1..cbeaf1fc38cfc 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1248,11 +1248,11 @@ static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
 
 #ifdef CONFIG_NUMA_BALANCING
 bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
-		bool is_private_single_threaded);
+		unsigned long cp_flags);
 
 #else
 static inline bool folio_can_map_prot_numa(struct folio *folio,
-		struct vm_area_struct *vma, bool is_private_single_threaded)
+		struct vm_area_struct *vma, unsigned long cp_flags)
 {
 	return false;
 }
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 4c0b8ff1a7e66..1427e1b6213b6 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -847,7 +847,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
  * folio_can_map_prot_numa() - check whether the folio can map prot numa
  * @folio: The folio whose mapping considered for being made NUMA hintable
  * @vma: The VMA that the folio belongs to.
- * @is_private_single_threaded: Is this a single-threaded private VMA or not
+ * @cp_flags: Flags describing the protection change
  *
  * This function checks to see if the folio actually indicates that
  * we need to make the mapping one which causes a NUMA hinting fault,
@@ -857,7 +857,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
  * Return: True if the mapping of the folio needs to be changed, false otherwise.
  */
 bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
-		bool is_private_single_threaded)
+		unsigned long cp_flags)
 {
 	int nid;
 
@@ -880,22 +880,17 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	if (folio_is_file_lru(folio) && folio_test_dirty(folio))
 		return false;
 
-	/*
-	 * Don't mess with PTEs if folio is already on the node
-	 * a single-threaded process is running on.
-	 */
 	nid = folio_nid(folio);
-	if (is_private_single_threaded && (nid == numa_node_id()))
+	/* Promotion-only scans do not mark top-tier folios. */
+	if ((cp_flags & MM_CP_PROT_NUMA_PROMO_ONLY) && node_is_toptier(nid))
 		return false;
 
 	/*
-	 * Skip scanning top tier node if normal numa
-	 * balancing is disabled
+	 * Don't mess with PTEs if folio is already on the node
+	 * a single-threaded process is running on.
 	 */
-	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
-	    node_is_toptier(nid))
+	if (vma_is_single_threaded_private(vma) && nid == numa_node_id())
 		return false;
-
 	if (folio_use_access_time(folio))
 		folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
 
@@ -907,19 +902,21 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
  * These are later cleared by a NUMA hinting fault. Depending on these
  * faults, pages may be migrated for better NUMA placement.
  *
+ * @cp_flags carries the NUMA scan policy through the protection walk.
+ *
  * This is assuming that NUMA faults are handled using PROT_NONE. If
  * an architecture makes a different choice, it will need further
  * changes to the core.
  */
-unsigned long change_prot_numa(struct vm_area_struct *vma,
-			unsigned long addr, unsigned long end)
+unsigned long change_prot_numa(struct vm_area_struct *vma, unsigned long addr,
+		unsigned long end, unsigned long cp_flags)
 {
 	struct mmu_gather tlb;
 	long nr_updated;
 
 	tlb_gather_mmu(&tlb, vma->vm_mm);
 
-	nr_updated = change_protection(&tlb, vma, addr, end, MM_CP_PROT_NUMA);
+	nr_updated = change_protection(&tlb, vma, addr, end, cp_flags);
 	if (nr_updated > 0) {
 		count_vm_numa_events(NUMA_PTE_UPDATES, nr_updated);
 		count_memcg_events_mm(vma->vm_mm, NUMA_PTE_UPDATES, nr_updated);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index e59c69cb5a238..7a247ed4cd55f 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -335,7 +335,6 @@ static long change_pte_range(struct mmu_gather *tlb,
 	pte_t *pte, oldpte;
 	spinlock_t *ptl;
 	long pages = 0;
-	bool is_private_single_threaded;
 	bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
 	bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
@@ -346,9 +345,6 @@ static long change_pte_range(struct mmu_gather *tlb,
 	if (!pte)
 		return -EAGAIN;
 
-	if (prot_numa)
-		is_private_single_threaded = vma_is_single_threaded_private(vma);
-
 	flush_tlb_batched_pending(vma->vm_mm);
 	lazy_mmu_mode_enable();
 	do {
@@ -383,8 +379,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 			 * must set protnone regardless of NUMA placement.
 			 */
 			if (prot_numa &&
-			    !folio_can_map_prot_numa(folio, vma,
-						is_private_single_threaded)) {
+			    !folio_can_map_prot_numa(folio, vma, cp_flags)) {
 
 				/* determine batch to skip */
 				nr_ptes = mprotect_folio_pte_batch(folio,
-- 
2.53.0-Meta

Re: [PATCH v3 1/7] mm: support promotion-only NUMA hinting scans

[David Hildenbrand (Arm)]

On 9/22/26 20:29, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> folio_can_map_prot_numa() derives folio eligibility from the global
> balancing mode. The mode (normal, tiering, or combined) describes
> which balancing mechanisms are enabled (placement vs promotion).
> 
> The global setting itself cannot describe the intent of an individual
> protection walk because normal-mode tempers its scanning activity based
> on a number of heuristics (read-only VMAs, activeness of VMA, etc).
> 
> In tiering or combined mode, applying these normal-mode optimizations
> to an entire VMA is incorrect and breaks tiering.
> 
>  - Opting VMAs out of scanning because they became inactive obviously
>    breaks tiering - because the intent is to identify when a VMA
>    becomes active.  Applying it in tiering modes causes

Incomplete sentence.

> 
>  - Opting Read-only file VMAs out of scanning is just incorrect for
>    tiering modes because its intent is to prevent bouncing between
>    sockets (east-west), while tiering controls tier migration
>    (north-south).  The result is hot read-only files can overload
>    lower tier bandwidth.
> 
> Add MM_CP_PROT_NUMA_PROMO_ONLY and let task_numa_work() select the type
> of walk. Build the change-protection flags there and pass them unchanged
> through change_prot_numa() so its PTE/PMD paths use the same decision.
> 
> Have folio_can_map_prot_numa() derive the single-threaded private state at
> the point of use instead of adding another precomputed boolean to the
> protection-walk interface. This keeps the interface focused on scan intent
> and avoids plumbing VMA-derived state beside cp_flags.
> 
> The tradeoff for the cleaner interface is an atomic mm_users read for each
> folio, rather than once per PTE range. Check the promotion-only top-tier
> exclusion bit first as a mild optimization.
> 
> The scan-intent interface is required by the following memory-tiering
> fixes and must accompany them when backported.
> 
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Suggested-by: David Hildenbrand <david@kernel.org>
> Link: https://lore.kernel.org/r/4d2853c9-edf4-4685-b186-7214ed6abd84@kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---

LGTM

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v3 1/7] mm: support promotion-only NUMA hinting scans

[Gregory Price]

On Thu, Sep 24, 2026 at 01:45:24PM +0200, David Hildenbrand (Arm) wrote:
> >  - Opting VMAs out of scanning because they became inactive obviously
> >    breaks tiering - because the intent is to identify when a VMA
> >    becomes active.  Applying it in tiering modes causes
> 
> Incomplete sentence.
>

oi

Applying it in tiering modes causes VMAs to become permanently stranded
on lower tiers.

> > 
> >  - Opting Read-only file VMAs out of scanning is just incorrect for
> >    tiering modes because its intent is to prevent bouncing between
> >    sockets (east-west), while tiering controls tier migration
> >    (north-south).  The result is hot read-only files can overload
> >    lower tier bandwidth.
> > 
> > Add MM_CP_PROT_NUMA_PROMO_ONLY and let task_numa_work() select the type
> > of walk. Build the change-protection flags there and pass them unchanged
> > through change_prot_numa() so its PTE/PMD paths use the same decision.
> > 
> > Have folio_can_map_prot_numa() derive the single-threaded private state at
> > the point of use instead of adding another precomputed boolean to the
> > protection-walk interface. This keeps the interface focused on scan intent
> > and avoids plumbing VMA-derived state beside cp_flags.
> > 
> > The tradeoff for the cleaner interface is an atomic mm_users read for each
> > folio, rather than once per PTE range. Check the promotion-only top-tier
> > exclusion bit first as a mild optimization.
> > 
> > The scan-intent interface is required by the following memory-tiering
> > fixes and must accompany them when backported.
> > 
> > Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> > Cc: stable@vger.kernel.org
> > Suggested-by: David Hildenbrand <david@kernel.org>
> > Link: https://lore.kernel.org/r/4d2853c9-edf4-4685-b186-7214ed6abd84@kernel.org
> > Assisted-by: LLM
> > Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> > ---
> 
> LGTM
> 
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>
> 
> -- 
> Cheers,
> 
> David

[PATCH v3 2/7] mm: allow shared folios to be promoted to a fast tier

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

NUMA balancing rejects shared copy-on-write folios and executable
file folios mapped by multiple processes to avoid east-west
migration bouncing. These checks break promotion from slow memory.

Allow such folios to participate when the folio being migrated is
a low-tier folio and the destination is top-tier (south->north).
This allows promotion, but prevents east-west or north->south
migrations (north->south is handled by reclaim demotion).

Keep the existing restrictions for ordinary placement and for
migrations that are not slow-to-top-tier promotions.

Rename folio_use_access_time() to folio_in_lowtier() so the source-tier
condition is more obvious (timing is the mechanism, not the condition).
The helper retains its existing behavior.

Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
Cc: stable@vger.kernel.org
Suggested-by: Zi Yan <ziy@nvidia.com>
Link: https://lore.kernel.org/r/DLHS4KFPQ86I.1J4LN3352RI71@nvidia.com
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 include/linux/mm.h  |  5 +++--
 kernel/sched/fair.c |  2 +-
 mm/memory-tiers.c   |  9 ++++++---
 mm/memory.c         |  2 +-
 mm/mempolicy.c      | 12 +++++++++---
 mm/migrate.c        | 14 ++++++++++----
 6 files changed, 30 insertions(+), 14 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index beab621e6f88d..225ba26c9d503 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2671,7 +2671,7 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 	}
 }
 
-bool folio_use_access_time(struct folio *folio);
+bool folio_in_lowtier(struct folio *folio);
 #else /* !CONFIG_NUMA_BALANCING */
 static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
 {
@@ -2725,7 +2725,8 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 }
-static inline bool folio_use_access_time(struct folio *folio)
+
+static inline bool folio_in_lowtier(struct folio *folio)
 {
 	return false;
 }
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9f544e3df9490..dc78d24ed8bc0 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -2734,7 +2734,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
 	 */
-	if (folio_use_access_time(folio)) {
+	if (folio_in_lowtier(folio)) {
 		struct pglist_data *pgdat;
 		unsigned long rate_limit;
 		unsigned int latency, th, def_th;
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 25e121851b586..082ca74d51ae9 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -53,16 +53,19 @@ static const struct bus_type memory_tier_subsys = {
 
 #ifdef CONFIG_NUMA_BALANCING
 /**
- * folio_use_access_time - check if a folio reuses cpupid for page access time
+ * folio_in_lowtier - check if a folio is in a tiering-managed lower tier
  * @folio: folio to check
  *
  * folio's _last_cpupid field is repurposed by memory tiering. In memory
  * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
  * record page access time.
  *
- * Return: the folio _last_cpupid is used to record page access time
+ * If memory tiering is disabled, then lowtier has no appreciable meaning,
+ * so we return false (the folio should not be migrated on this distinction).
+ *
+ * Return: true if memory tiering can promote the folio.
  */
-bool folio_use_access_time(struct folio *folio)
+bool folio_in_lowtier(struct folio *folio)
 {
 	return (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 	       !node_is_toptier(folio_nid(folio));
diff --git a/mm/memory.c b/mm/memory.c
index 79fa57a381ce0..f05c469e32a2a 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6240,7 +6240,7 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	 * For memory tiering mode, cpupid of slow memory page is used
 	 * to record page access time.  So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_in_lowtier(folio))
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
 	else
 		*last_cpupid = folio_last_cpupid(folio);
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 1427e1b6213b6..3c3ee28f0142f 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -864,8 +864,13 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
 		return false;
 
-	/* Also skip shared copy-on-write folios */
-	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
+	/*
+	 * Shared copy-on-write folios are poor east-west placement candidates.
+	 * When tiering is enabled, folio_in_lowtier() identifies a promotable
+	 * folio on a low tier, which needs a hint fault for promotion.
+	 */
+	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio) &&
+	    !folio_in_lowtier(folio))
 		return false;
 
 	/* Folios are pinned and can't be migrated */
@@ -891,7 +896,8 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	 */
 	if (vma_is_single_threaded_private(vma) && nid == numa_node_id())
 		return false;
-	if (folio_use_access_time(folio))
+
+	if (folio_in_lowtier(folio))
 		folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
 
 	return true;
diff --git a/mm/migrate.c b/mm/migrate.c
index 7bdcdb57652f8..6a08690cfe219 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2694,14 +2694,20 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 
 	if (folio_is_file_lru(folio)) {
 		/*
-		 * Do not migrate file folios that are mapped in multiple
-		 * processes with execute permissions as they are probably
-		 * shared libraries.
+		 * Limit east-east migration of file folios mapped in
+		 * multiple processes with execute permissions as they
+		 * are probably shared libraries (limits bouncing).
+		 *
+		 * If this is a low-tier folio, only migrate if the target
+		 * node is toptier (this allows south->north migration while
+		 * disallowing east-west migration between slow tiers).
 		 *
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if ((vma->vm_flags & VM_EXEC) &&
+		    folio_maybe_mapped_shared(folio) &&
+		    (!folio_in_lowtier(folio) || !node_is_toptier(node)))
 			return -EACCES;
 
 		/*
-- 
2.53.0-Meta

Re: [PATCH v3 2/7] mm: allow shared folios to be promoted to a fast tier

[David Hildenbrand (Arm)]

On 9/22/26 20:29, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> NUMA balancing rejects shared copy-on-write folios and executable
> file folios mapped by multiple processes to avoid east-west
> migration bouncing. These checks break promotion from slow memory.

Makes sense. So in general we're happy to promote if there is no chance of
bouncing (as would happen with ordinary numa migration).

> 
> Allow such folios to participate when the folio being migrated is
> a low-tier folio and the destination is top-tier (south->north).
> This allows promotion, but prevents east-west or north->south
> migrations (north->south is handled by reclaim demotion).
> 
> Keep the existing restrictions for ordinary placement and for
> migrations that are not slow-to-top-tier promotions.
> 
> Rename folio_use_access_time() to folio_in_lowtier() so the source-tier
> condition is more obvious (timing is the mechanism, not the condition).
> The helper retains its existing behavior.
> 
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Suggested-by: Zi Yan <ziy@nvidia.com>
> Link: https://lore.kernel.org/r/DLHS4KFPQ86I.1J4LN3352RI71@nvidia.com
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---

[...]

> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -53,16 +53,19 @@ static const struct bus_type memory_tier_subsys = {
>  
>  #ifdef CONFIG_NUMA_BALANCING
>  /**
> - * folio_use_access_time - check if a folio reuses cpupid for page access time
> + * folio_in_lowtier - check if a folio is in a tiering-managed lower tier
>   * @folio: folio to check
>   *
>   * folio's _last_cpupid field is repurposed by memory tiering. In memory
>   * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
>   * record page access time.
>   *
> - * Return: the folio _last_cpupid is used to record page access time
> + * If memory tiering is disabled, then lowtier has no appreciable meaning,
> + * so we return false (the folio should not be migrated on this distinction).
> + *
> + * Return: true if memory tiering can promote the folio.

I wonder if a better name would just make the comment above (about memory
tiering) implicit and make more sense of the sysctl_numa_balancing_mode &
NUMA_BALANCING_MEMORY_TIERING) check.

	folio_promotable()

	folio_is_promotable()

	folio_numa_promotable()

etc.

because that seems to be what we really test with both things combined.

> +++ b/mm/mempolicy.c
> @@ -864,8 +864,13 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
>  	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
>  		return false;
>  
> -	/* Also skip shared copy-on-write folios */
> -	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
> +	/*
> +	 * Shared copy-on-write folios are poor east-west placement candidates.
> +	 * When tiering is enabled, folio_in_lowtier() identifies a promotable
> +	 * folio on a low tier, which needs a hint fault for promotion.
> +	 */
> +	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio) &&
> +	    !folio_in_lowtier(folio))
>  		return false;
>  
>  	/* Folios are pinned and can't be migrated */
> @@ -891,7 +896,8 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
>  	 */
>  	if (vma_is_single_threaded_private(vma) && nid == numa_node_id())
>  		return false;
> -	if (folio_use_access_time(folio))
> +
> +	if (folio_in_lowtier(folio))
>  		folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
>  
>  	return true;
> diff --git a/mm/migrate.c b/mm/migrate.c
> index 7bdcdb57652f8..6a08690cfe219 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -2694,14 +2694,20 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
>  
>  	if (folio_is_file_lru(folio)) {
>  		/*
> -		 * Do not migrate file folios that are mapped in multiple
> -		 * processes with execute permissions as they are probably
> -		 * shared libraries.
> +		 * Limit east-east migration of file folios mapped in
> +		 * multiple processes with execute permissions as they
> +		 * are probably shared libraries (limits bouncing).
> +		 *
> +		 * If this is a low-tier folio, only migrate if the target
> +		 * node is toptier (this allows south->north migration while
> +		 * disallowing east-west migration between slow tiers).
>  		 *
>  		 * See folio_maybe_mapped_shared() on possible imprecision
>  		 * when we cannot easily detect if a folio is shared.
>  		 */
> -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
> +		if ((vma->vm_flags & VM_EXEC) &&
> +		    folio_maybe_mapped_shared(folio) &&
> +		    (!folio_in_lowtier(folio) || !node_is_toptier(node)))
>  			return -EACCES;
>  
>  		/*

Sashiko has some comment about anonymous shared COW folios. I think it has a
point, but didn't look too closely.

-- 
Cheers,

David

Re: [PATCH v3 2/7] mm: allow shared folios to be promoted to a fast tier

[Gregory Price]

On Thu, Sep 24, 2026 at 01:59:15PM +0200, David Hildenbrand (Arm) wrote:
> I wonder if a better name would just make the comment above (about memory
> tiering) implicit and make more sense of the sysctl_numa_balancing_mode &
> NUMA_BALANCING_MEMORY_TIERING) check.
> 
> 	folio_promotable()
> 
> 	folio_is_promotable()
> 
> 	folio_numa_promotable()
>
> etc.
> 
> because that seems to be what we really test with both things combined.
>

maybe:  folio_numab_promotable() ?

> > -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
> > +		if ((vma->vm_flags & VM_EXEC) &&
> > +		    folio_maybe_mapped_shared(folio) &&
> > +		    (!folio_in_lowtier(folio) || !node_is_toptier(node)))
> >  			return -EACCES;
> >  
> >  		/*
> 
> Sashiko has some comment about anonymous shared COW folios. I think it has a
> point, but didn't look too closely.
>

its tl;dr:

`skipping this check means those folios could end up freely migrating east-west
 between slow tiers`

Which cannot happen (we only migrate to top tier, numab will never
migrate *to* a slow tier).

basically false-positive

~Gregory

Re: [PATCH v3 2/7] mm: allow shared folios to be promoted to a fast tier

[David Hildenbrand (Arm)]

On 9/24/26 16:07, Gregory Price wrote:
> On Thu, Sep 24, 2026 at 01:59:15PM +0200, David Hildenbrand (Arm) wrote:
>> I wonder if a better name would just make the comment above (about memory
>> tiering) implicit and make more sense of the sysctl_numa_balancing_mode &
>> NUMA_BALANCING_MEMORY_TIERING) check.
>>
>> 	folio_promotable()
>>
>> 	folio_is_promotable()
>>
>> 	folio_numa_promotable()
>>
>> etc.
>>
>> because that seems to be what we really test with both things combined.
>>
> 
> maybe:  folio_numab_promotable() ?

Works for me if "numab" means numa balancing :)

> 
>>> -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
>>> +		if ((vma->vm_flags & VM_EXEC) &&
>>> +		    folio_maybe_mapped_shared(folio) &&
>>> +		    (!folio_in_lowtier(folio) || !node_is_toptier(node)))
>>>  			return -EACCES;
>>>  
>>>  		/*
>>
>> Sashiko has some comment about anonymous shared COW folios. I think it has a
>> point, but didn't look too closely.
>>
> 
> its tl;dr:
> 
> `skipping this check means those folios could end up freely migrating east-west
>  between slow tiers`
> 
> Which cannot happen (we only migrate to top tier, numab will never
> migrate *to* a slow tier).
> 
> basically false-positive

False-positive? From Sashiko? I am shocked!

-- 
Cheers,

David

Re: [PATCH v3 2/7] mm: allow shared folios to be promoted to a fast tier

[Gregory Price]

On Thu, Sep 24, 2026 at 05:32:22PM +0200, David Hildenbrand (Arm) wrote:
> On 9/24/26 16:07, Gregory Price wrote:
> > On Thu, Sep 24, 2026 at 01:59:15PM +0200, David Hildenbrand (Arm) wrote:
> >> I wonder if a better name would just make the comment above (about memory
> >> tiering) implicit and make more sense of the sysctl_numa_balancing_mode &
> >> NUMA_BALANCING_MEMORY_TIERING) check.
> >>
> >> 	folio_promotable()
> >>
> >> 	folio_is_promotable()
> >>
> >> 	folio_numa_promotable()
> >>
> >> etc.
> >>
> >> because that seems to be what we really test with both things combined.
> >>
> > 
> > maybe:  folio_numab_promotable() ?
> 
> Works for me if "numab" means numa balancing :)
>

Sounds good, i'll let this rest for a bit before spinning a v4. I think
the rest is pretty solid, but there's one other sashiko thing i need to
think about whether it's real or not.

~Gregory

Re: [PATCH v3 2/7] mm: allow shared folios to be promoted to a fast tier

[Zi Yan]

On 24 Sep 2026, at 11:37, Gregory Price wrote:

> On Thu, Sep 24, 2026 at 05:32:22PM +0200, David Hildenbrand (Arm) wrote:
>> On 9/24/26 16:07, Gregory Price wrote:
>>> On Thu, Sep 24, 2026 at 01:59:15PM +0200, David Hildenbrand (Arm) wrote:
>>>> I wonder if a better name would just make the comment above (about memory
>>>> tiering) implicit and make more sense of the sysctl_numa_balancing_mode &
>>>> NUMA_BALANCING_MEMORY_TIERING) check.
>>>>
>>>> 	folio_promotable()
>>>>
>>>> 	folio_is_promotable()
>>>>
>>>> 	folio_numa_promotable()
>>>>
>>>> etc.
>>>>
>>>> because that seems to be what we really test with both things combined.
>>>>
>>>
>>> maybe:  folio_numab_promotable() ?
>>
>> Works for me if "numab" means numa balancing :)
>>
>
> Sounds good, i'll let this rest for a bit before spinning a v4. I think
> the rest is pretty solid, but there's one other sashiko thing i need to
> think about whether it's real or not.

folio_numab_promotable() sounds good to me.


Best Regards,
Yan, Zi

[PATCH v3 3/7] sched/numa: scan read-only file mappings in tiering mode

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

Commit 4591ce4f2d22 ("sched/numa: Do not trap hinting faults for
shared libraries") excludes file-backed read-only VMAs from NUMA
hint faulting to prevent east-west placement bouncing.

This filter hides hot file folios on slow memory from promotion.

Scan those VMAs when tiering is enabled, but make their scans promotion
only to retains the existing restriction. Keep the historical VMA
predicate unchanged for backport-ability.

Read the balancing mode once per task_numa_work() invocation and build
the protection flags for each VMA from that snapshot. This keeps the PTE
and PMD paths on the same policy for an entire protection walk (which
may be split across multiple scanning periods).

On a host with 768 GB of DRAM and 256 GB of CXL memory running two
database services using ~430GB each, 169 MB of their shared 185 MB
main binary accumulated on CXL before permanently stuck there.

With the change, the binary tier residency tracks its runtime hotness.

Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 kernel/sched/fair.c | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index dc78d24ed8bc0..8a4687f67d82f 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4119,21 +4119,21 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
  */
 static void task_numa_work(struct callback_head *work)
 {
+	const unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
+	const bool tiering = numab_mode & NUMA_BALANCING_MEMORY_TIERING;
 	unsigned long migrate, next_scan, now = jiffies;
 	struct task_struct *p = current;
 	struct mm_struct *mm = p->mm;
 	u64 runtime = p->se.sum_exec_runtime;
 	struct vm_area_struct *vma;
-	unsigned long cp_flags = MM_CP_PROT_NUMA;
+	unsigned long cp_flags;
 	unsigned long start, end;
 	unsigned long nr_pte_updates = 0;
 	long pages, virtpages;
 	struct vma_iterator vmi;
 	bool vma_pids_skipped;
 	bool vma_pids_forced = false;
-
-	if (!(READ_ONCE(sysctl_numa_balancing_mode) & NUMA_BALANCING_NORMAL))
-		cp_flags |= MM_CP_PROT_NUMA_PROMO_ONLY;
+	bool placement_scan;
 
 	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
 
@@ -4221,13 +4221,19 @@ static void task_numa_work(struct callback_head *work)
 		}
 
 		/*
-		 * Shared library pages mapped by multiple processes are not
-		 * migrated as it is expected they are cache replicated. Avoid
-		 * hinting faults in read-only file-backed mappings or the vDSO
-		 * as migrating the pages will be of marginal benefit.
+		 * Shared library pages mapped by multiple processes are limited
+		 * to south->north migrations as it is expected they are cache
+		 * replicated. The benefit of east-west migration in this case
+		 * is at best marginal and may be harmful due to TLB/cache
+		 * invalidation.
+		 *
+		 * Allow promotion as a cold page incurring many cache-misses
+		 * under cache pressure can drive considerable bandwidth.
 		 */
-		if (!vma->vm_mm ||
-		    (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ))) {
+		placement_scan = !(vma->vm_file &&
+			(vma->vm_flags & (VM_READ | VM_WRITE)) == VM_READ);
+
+		if (!vma->vm_mm || (!placement_scan && !tiering)) {
 			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_SHARED_RO);
 			continue;
 		}
@@ -4307,6 +4313,11 @@ static void task_numa_work(struct callback_head *work)
 			continue;
 		}
 
+		placement_scan &= numab_mode & NUMA_BALANCING_NORMAL;
+		cp_flags = MM_CP_PROT_NUMA;
+		if (!placement_scan)
+			cp_flags |= MM_CP_PROT_NUMA_PROMO_ONLY;
+
 		do {
 			start = max(start, vma->vm_start);
 			end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
-- 
2.53.0-Meta

Re: [PATCH v3 3/7] sched/numa: scan read-only file mappings in tiering mode

[David Hildenbrand (Arm)]

On 9/22/26 20:29, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> Commit 4591ce4f2d22 ("sched/numa: Do not trap hinting faults for
> shared libraries") excludes file-backed read-only VMAs from NUMA
> hint faulting to prevent east-west placement bouncing.
> 
> This filter hides hot file folios on slow memory from promotion.
> 
> Scan those VMAs when tiering is enabled, but make their scans promotion
> only to retains the existing restriction. Keep the historical VMA
> predicate unchanged for backport-ability.
> 
> Read the balancing mode once per task_numa_work() invocation and build
> the protection flags for each VMA from that snapshot. This keeps the PTE
> and PMD paths on the same policy for an entire protection walk (which
> may be split across multiple scanning periods).
> 
> On a host with 768 GB of DRAM and 256 GB of CXL memory running two
> database services using ~430GB each, 169 MB of their shared 185 MB
> main binary accumulated on CXL before permanently stuck there.
> 
> With the change, the binary tier residency tracks its runtime hotness.
> 
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  kernel/sched/fair.c | 31 +++++++++++++++++++++----------
>  1 file changed, 21 insertions(+), 10 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index dc78d24ed8bc0..8a4687f67d82f 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4119,21 +4119,21 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
>   */
>  static void task_numa_work(struct callback_head *work)
>  {
> +	const unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
> +	const bool tiering = numab_mode & NUMA_BALANCING_MEMORY_TIERING;
>  	unsigned long migrate, next_scan, now = jiffies;
>  	struct task_struct *p = current;
>  	struct mm_struct *mm = p->mm;
>  	u64 runtime = p->se.sum_exec_runtime;
>  	struct vm_area_struct *vma;
> -	unsigned long cp_flags = MM_CP_PROT_NUMA;
> +	unsigned long cp_flags;
>  	unsigned long start, end;
>  	unsigned long nr_pte_updates = 0;
>  	long pages, virtpages;
>  	struct vma_iterator vmi;
>  	bool vma_pids_skipped;
>  	bool vma_pids_forced = false;
> -
> -	if (!(READ_ONCE(sysctl_numa_balancing_mode) & NUMA_BALANCING_NORMAL))
> -		cp_flags |= MM_CP_PROT_NUMA_PROMO_ONLY;
> +	bool placement_scan;
>  
>  	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
>  
> @@ -4221,13 +4221,19 @@ static void task_numa_work(struct callback_head *work)
>  		}
>  
>  		/*
> -		 * Shared library pages mapped by multiple processes are not
> -		 * migrated as it is expected they are cache replicated. Avoid
> -		 * hinting faults in read-only file-backed mappings or the vDSO
> -		 * as migrating the pages will be of marginal benefit.
> +		 * Shared library pages mapped by multiple processes are limited
> +		 * to south->north migrations as it is expected they are cache
> +		 * replicated. The benefit of east-west migration in this case
> +		 * is at best marginal and may be harmful due to TLB/cache
> +		 * invalidation.
> +		 *
> +		 * Allow promotion as a cold page incurring many cache-misses
> +		 * under cache pressure can drive considerable bandwidth.
>  		 */
> -		if (!vma->vm_mm ||
> -		    (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ))) {
> +		placement_scan = !(vma->vm_file &&
> +			(vma->vm_flags & (VM_READ | VM_WRITE)) == VM_READ);

Indentation looks weird here as well. Maybe it's my mail client.

> +
> +		if (!vma->vm_mm || (!placement_scan && !tiering)) {
>  			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_SHARED_RO);
>  			continue;
>  		}
> @@ -4307,6 +4313,11 @@ static void task_numa_work(struct callback_head *work)
>  			continue;
>  		}
>  
> +		placement_scan &= numab_mode & NUMA_BALANCING_NORMAL;

Do we have to update placement_scan at all?

Can't we do:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8a4687f67d82f..fc8862ee31442 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4121,6 +4121,7 @@ static void task_numa_work(struct callback_head *work)
 {
        const unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
        const bool tiering = numab_mode & NUMA_BALANCING_MEMORY_TIERING;
+       const bool balancing = numab_mode & NUMA_BALANCING_NORMAL;
        unsigned long migrate, next_scan, now = jiffies;
        struct task_struct *p = current;
        struct mm_struct *mm = p->mm;
@@ -4313,9 +4314,8 @@ static void task_numa_work(struct callback_head *work)
                        continue;
                }

-               placement_scan &= numab_mode & NUMA_BALANCING_NORMAL;
                cp_flags = MM_CP_PROT_NUMA;
-               if (!placement_scan)
+               if (!placement_scan || !balancing)
                        cp_flags |= MM_CP_PROT_NUMA_PROMO_ONLY;

                do {

(maybe I messed up my boolean algreba, it's late)

-- 
Cheers,

David

[PATCH v3 4/7] sched/numa: separate VMA placement from scan continuation

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

vma_is_accessed() returns true both when a VMA needs placement
(east-west) sampling and when a VMA is already mid-scan across
multiple scan windows.

Callers cannot distinguish placement eligibility from scan progress.

Rename the helper to vma_needs_placement_scan() and handle continuation
state in task_numa_work(). Cache the scan eligibility decision while a
VMA is scanned in chunks so a resumed scan retains the existing policy.

This separation allows PID-inactive VMAs to be scanned for promotion
without treating scan continuation as evidence of placement eligibility.
It is a prerequisite for the following fix and must accompany it when
backported.

Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 include/linux/mm_types.h |  3 +++
 kernel/sched/fair.c      | 42 ++++++++++++++++++++++++----------------
 2 files changed, 28 insertions(+), 17 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 77c90b1994131..fd35db969bc94 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -803,6 +803,9 @@ struct vma_numab_state {
 	 * A VMA is not eligible for scanning if prev_scan_seq == numa_scan_seq
 	 */
 	int prev_scan_seq;
+
+	/* Preserve placement-scan eligibility during an in-progress scan. */
+	bool placement_scan;
 };
 
 #ifdef __HAVE_PFNMAP_TRACKING
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8a4687f67d82f..a2849e72c4e26 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4074,7 +4074,8 @@ static void reset_ptenuma_scan(struct task_struct *p)
 	p->mm->numa_scan_offset = 0;
 }
 
-static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
+static bool vma_needs_placement_scan(struct mm_struct *mm,
+		struct vm_area_struct *vma)
 {
 	unsigned long pids;
 	/*
@@ -4090,15 +4091,6 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
 	if (test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids))
 		return true;
 
-	/*
-	 * Complete a scan that has already started regardless of PID access, or
-	 * some VMAs may never be scanned in multi-threaded applications:
-	 */
-	if (mm->numa_scan_offset > vma->vm_start) {
-		trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
-		return true;
-	}
-
 	/*
 	 * This vma has not been accessed for a while, and if the number
 	 * the threads in the same process is low, which means no other
@@ -4133,7 +4125,8 @@ static void task_numa_work(struct callback_head *work)
 	struct vma_iterator vmi;
 	bool vma_pids_skipped;
 	bool vma_pids_forced = false;
-	bool placement_scan;
+	bool pid_scan_allowed, placement_due;
+	bool placement_scan, scan_started;
 
 	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
 
@@ -4304,16 +4297,30 @@ static void task_numa_work(struct callback_head *work)
 		}
 
 		/*
-		 * Do not scan the VMA if task has not accessed it, unless no other
-		 * VMA candidate exists.
+		 * Do not scan the VMA if a task has not accessed it, unless no other
+		 * VMA candidate exists. If a scan is already in-progress, finish it,
+		 * but track continuation separately from starting a new one.
 		 */
-		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
-			vma_pids_skipped = true;
-			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
-			continue;
+		placement_due = vma_needs_placement_scan(mm, vma);
+		scan_started = mm->numa_scan_offset > vma->vm_start;
+		pid_scan_allowed = vma_pids_forced || placement_due;
+
+		if (!pid_scan_allowed) {
+			if (scan_started) {
+				trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
+			} else {
+				vma_pids_skipped = true;
+				trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
+				continue;
+			}
 		}
 
+		/* Keep scan policy stable while processing a VMA in chunks.*/
 		placement_scan &= numab_mode & NUMA_BALANCING_NORMAL;
+		if (scan_started)
+			placement_scan &= vma->numab_state->placement_scan;
+
+		vma->numab_state->placement_scan = placement_scan;
 		cp_flags = MM_CP_PROT_NUMA;
 		if (!placement_scan)
 			cp_flags |= MM_CP_PROT_NUMA_PROMO_ONLY;
@@ -4346,6 +4353,7 @@ static void task_numa_work(struct callback_head *work)
 
 		/* VMA scan is complete, do not scan until next sequence. */
 		vma->numab_state->prev_scan_seq = mm->numa_scan_seq;
+		vma->numab_state->placement_scan = false;
 
 		/*
 		 * Only force scan within one VMA at a time, to limit the
-- 
2.53.0-Meta

[PATCH v3 5/7] sched/numa: scan PID-inactive VMAs for promotion

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

Commit fc137c0ddab2 ("sched/numa: enhance vma scanning logic") skips
VMAs without recent PID activity. Since only NUMA hint faults record
that activity, the filter can suppress the fault needed to promote hot
slow-tier memory.

Let memory tiering bypass the PID scan filter. In combined mode, use the
placement decision to restrict top-tier sampling to VMAs that need it,
while inactive VMAs still receive promotion-only scans.

Track the last completed placement scan separately from scans of any
kind. Promotion-only scans still update prev_scan_seq, but do not
advance the placement-starvation horizon.

On a host with 768 GB of DRAM and 256 GB of CXL memory, one large shmem
VMA consumed most scanning activity, while 2,537 other VMAs covering
84 GB were skipped as inactive. One stand-out result: a hot 20 GB hash
table ended up trapped entirely on CXL and drove CXL bandwidth
utilization beyond sustainable levels - resulting in a large regression.

With this series, the hot hash table ends up split evenly between DRAM
and CXL, tier residency tracked runtime load, and CXL bandwidth
utilization drops from 45GB/s (maxed) to 5-10GB/s, while DRAM bandwidth
utilization increases from ~200GB/s to 250GB/s+, resulting in major
throughput improvements for the database workload.

Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
Cc: stable@vger.kernel.org
Assisted-by: OpenAI:gpt-5
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 include/linux/mm_types.h |  7 +++++++
 kernel/sched/fair.c      | 26 +++++++++++++++++++++-----
 2 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index fd35db969bc94..dcca3ead9db59 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -804,6 +804,13 @@ struct vma_numab_state {
 	 */
 	int prev_scan_seq;
 
+	/*
+	 * MM scan sequence ID when the VMA was last scanned for placement.
+	 * The starvation horizon in vma_needs_placement_scan() counts against
+	 * this, so promotion-only scans cannot postpone placement indefinitely.
+	 */
+	int prev_placement_scan_seq;
+
 	/* Preserve placement-scan eligibility during an in-progress scan. */
 	bool placement_scan;
 };
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index a2849e72c4e26..412c72084a63d 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4097,7 +4097,7 @@ static bool vma_needs_placement_scan(struct mm_struct *mm,
 	 * threads can help scan this vma, force a vma scan.
 	 */
 	if (READ_ONCE(mm->numa_scan_seq) >
-	   (vma->numab_state->prev_scan_seq + get_nr_threads(current)))
+	   (vma->numab_state->prev_placement_scan_seq + get_nr_threads(current)))
 		return true;
 
 	return false;
@@ -4267,7 +4267,8 @@ static void task_numa_work(struct callback_head *work)
 			 * to prevent VMAs being skipped prematurely on the
 			 * first scan:
 			 */
-			 vma->numab_state->prev_scan_seq = mm->numa_scan_seq - 1;
+			vma->numab_state->prev_scan_seq = mm->numa_scan_seq - 1;
+			vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq - 1;
 		}
 
 		/*
@@ -4300,10 +4301,13 @@ static void task_numa_work(struct callback_head *work)
 		 * Do not scan the VMA if a task has not accessed it, unless no other
 		 * VMA candidate exists. If a scan is already in-progress, finish it,
 		 * but track continuation separately from starting a new one.
+		 *
+		 * The PID filter must not gate promotion. Allow PID-inactive VMAs
+		 * to proceed when memory tiering is enabled.
 		 */
 		placement_due = vma_needs_placement_scan(mm, vma);
 		scan_started = mm->numa_scan_offset > vma->vm_start;
-		pid_scan_allowed = vma_pids_forced || placement_due;
+		pid_scan_allowed = tiering || vma_pids_forced || placement_due;
 
 		if (!pid_scan_allowed) {
 			if (scan_started) {
@@ -4315,10 +4319,16 @@ static void task_numa_work(struct callback_head *work)
 			}
 		}
 
-		/* Keep scan policy stable while processing a VMA in chunks.*/
+		/*
+		 * Keep scan policy stable while processing a VMA in chunks.
+		 * A fault in one chunk can make a VMA placement-eligible. Keep a
+		 * promotion-only decision sticky for the rest of a partial scan.
+		 */
 		placement_scan &= numab_mode & NUMA_BALANCING_NORMAL;
 		if (scan_started)
 			placement_scan &= vma->numab_state->placement_scan;
+		else if (tiering)
+			placement_scan &= placement_due;
 
 		vma->numab_state->placement_scan = placement_scan;
 		cp_flags = MM_CP_PROT_NUMA;
@@ -4351,8 +4361,14 @@ static void task_numa_work(struct callback_head *work)
 			cond_resched();
 		} while (end != vma->vm_end);
 
-		/* VMA scan is complete, do not scan until next sequence. */
+		/*
+		 * VMA scan is complete, do not scan until next sequence.
+		 * A promotion-only scan did not cover top-tier folios, so it
+		 * does not count towards the placement-scan starvation check.
+		 */
 		vma->numab_state->prev_scan_seq = mm->numa_scan_seq;
+		if (placement_scan)
+			vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq;
 		vma->numab_state->placement_scan = false;
 
 		/*
-- 
2.53.0-Meta

[PATCH v3 6/7] mm: use BIT() for change_protection() flags

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

The MM_CP_* flags are individual bits expressed as open-coded shifts.
Use BIT() consistently for the complete block.

No functional change intended.

Suggested-by: David Hildenbrand <david@kernel.org>
Link: https://lore.kernel.org/r/9b5e1573-6fd8-48a5-a274-a6e7ff83d2fe@kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 include/linux/mm.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 225ba26c9d503..c2ffba42019c0 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3596,21 +3596,21 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
  * because something (e.g., COW, uffd-wp) blocks that from happening for all
  * PTEs automatically in a writable mapping.
  */
-#define  MM_CP_TRY_CHANGE_WRITABLE	   (1UL << 0)
+#define  MM_CP_TRY_CHANGE_WRITABLE	   BIT(0)
 /* Whether this protection change is for NUMA hints */
-#define  MM_CP_PROT_NUMA                   (1UL << 1)
+#define  MM_CP_PROT_NUMA                   BIT(1)
 /* Whether this change is for write protecting */
-#define  MM_CP_UFFD_WP                     (1UL << 2) /* do wp */
-#define  MM_CP_UFFD_WP_RESOLVE             (1UL << 3) /* Resolve wp */
+#define  MM_CP_UFFD_WP                     BIT(2) /* do wp */
+#define  MM_CP_UFFD_WP_RESOLVE             BIT(3) /* Resolve wp */
 #define  MM_CP_UFFD_WP_ALL                 (MM_CP_UFFD_WP | \
 					    MM_CP_UFFD_WP_RESOLVE)
 /* Whether this change is for uffd RWP */
-#define  MM_CP_UFFD_RWP                    (1UL << 4) /* do rwp */
-#define  MM_CP_UFFD_RWP_RESOLVE            (1UL << 5) /* resolve rwp */
+#define  MM_CP_UFFD_RWP                    BIT(4) /* do rwp */
+#define  MM_CP_UFFD_RWP_RESOLVE            BIT(5) /* resolve rwp */
 #define  MM_CP_UFFD_RWP_ALL                (MM_CP_UFFD_RWP | \
 					    MM_CP_UFFD_RWP_RESOLVE)
 /* Whether a MM_CP_PROT_NUMA change is for promotion only */
-#define  MM_CP_PROT_NUMA_PROMO_ONLY        (1UL << 6)
+#define  MM_CP_PROT_NUMA_PROMO_ONLY        BIT(6)
 
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
-- 
2.53.0-Meta

Re: [PATCH v3 6/7] mm: use BIT() for change_protection() flags

[David Hildenbrand (Arm)]

On 9/22/26 20:29, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> The MM_CP_* flags are individual bits expressed as open-coded shifts.
> Use BIT() consistently for the complete block.
> 
> No functional change intended.
> 
> Suggested-by: David Hildenbrand <david@kernel.org>
> Link: https://lore.kernel.org/r/9b5e1573-6fd8-48a5-a274-a6e7ff83d2fe@kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

[PATCH v3 7/7] mm: use VMA flag helpers in NUMA balancing

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

Use vma_test() instead of accessing vm_flags directly in the NUMA
balancing code touched by the preceding fixes. Keep the existing flag
predicates unchanged.

No functional change intended.

Suggested-by: Lorenzo Stoakes <ljs@kernel.org>
Link: https://lore.kernel.org/r/aq1FDhy00epXXtgd@gremlin
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 kernel/sched/fair.c | 3 ++-
 mm/internal.h       | 2 +-
 mm/migrate.c        | 2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 412c72084a63d..0d48d595740bf 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4224,7 +4224,8 @@ static void task_numa_work(struct callback_head *work)
 		 * under cache pressure can drive considerable bandwidth.
 		 */
 		placement_scan = !(vma->vm_file &&
-			(vma->vm_flags & (VM_READ | VM_WRITE)) == VM_READ);
+		    vma_test(vma, VMA_READ_BIT) &&
+		    !vma_test(vma, VMA_WRITE_BIT));
 
 		if (!vma->vm_mm || (!placement_scan && !tiering)) {
 			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_SHARED_RO);
diff --git a/mm/internal.h b/mm/internal.h
index cbeaf1fc38cfc..560406c36a757 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1240,7 +1240,7 @@ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,
 
 static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
 {
-	if (vma->vm_flags & VM_SHARED)
+	if (vma_test(vma, VMA_SHARED_BIT))
 		return false;
 
 	return atomic_read(&vma->vm_mm->mm_users) == 1;
diff --git a/mm/migrate.c b/mm/migrate.c
index 6a08690cfe219..8e1e5ff1cb580 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2705,7 +2705,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) &&
+		if (vma_test(vma, VMA_EXEC_BIT) &&
 		    folio_maybe_mapped_shared(folio) &&
 		    (!folio_in_lowtier(folio) || !node_is_toptier(node)))
 			return -EACCES;
-- 
2.53.0-Meta

Re: [PATCH v3 7/7] mm: use VMA flag helpers in NUMA balancing

[David Hildenbrand (Arm)]

On 9/22/26 20:29, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> Use vma_test() instead of accessing vm_flags directly in the NUMA
> balancing code touched by the preceding fixes. Keep the existing flag
> predicates unchanged.
> 
> No functional change intended.
> 
> Suggested-by: Lorenzo Stoakes <ljs@kernel.org>
> Link: https://lore.kernel.org/r/aq1FDhy00epXXtgd@gremlin
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  kernel/sched/fair.c | 3 ++-
>  mm/internal.h       | 2 +-
>  mm/migrate.c        | 2 +-
>  3 files changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 412c72084a63d..0d48d595740bf 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4224,7 +4224,8 @@ static void task_numa_work(struct callback_head *work)
>  		 * under cache pressure can drive considerable bandwidth.
>  		 */
>  		placement_scan = !(vma->vm_file &&
> -			(vma->vm_flags & (VM_READ | VM_WRITE)) == VM_READ);
> +		    vma_test(vma, VMA_READ_BIT) &&
> +		    !vma_test(vma, VMA_WRITE_BIT));
>  

Indentation looks wrong.

Apart from that LGTM (did not check if there are other cases).

-- 
Cheers,

David