[PATCH v2 0/4] sched/numa: stop VMA scan filters from gating promotion

[PATCH v2 0/4] sched/numa: stop VMA scan filters from gating promotion

[Gregory Price]

NUMA balancing uses hinting faults for both task placement and memory-tier
promotion. Several filters designed to avoid unproductive socket-placement
faults can also prevent promotion.

Read-only file mappings and VMAs without recent PID activity may never be
scanned, while lower MM layers reject some shared folios. Hot memory in
these mappings can therefore remain on a slow tier indefinitely.

Separate promotion-only scans from socket-placement scans.

MM_CP_PROT_NUMA_PROMO_ONLY carries that choice for one protection walk, so
the PTE and PMD paths can restrict hinting faults to promotion candidates.
Keeping this transient state in the protection flags avoids per-mm state and
its associated lifetime, concurrency, and VMA identity problems.

  - Allow eligible shared folios to be promoted to a fast tier.
  - Scan read-only file mappings and PID-inactive VMAs for promotion
    without re-enabling placement sampling.
  - Track the last placement scan separately so promotion-only scans
    cannot postpone the existing placement-starvation fallback.

The series is ordered as follows:

  1. Add promotion-only NUMA protection walks without changing behavior.
  2. Permit eligible shared folios to be promoted to a fast tier.
  3. Scan read-only file mappings using promotion-only scans.
  4. Scan PID-inactive VMAs for promotion and account for placement scans
     separately.

Tested on a host with 768GB/256GB DRAM/CXL.
Ran 2 ~430GB database workloads with large (>300GB) shmem VMAs.

Before change:
  - 150-200GB/s DRAM bandwidth usage
  - 40-45GB/s CXL bandwidth usage (maxed out)
  - request latencies over 5ms (longer tails)

After chage:
  - 250GB/s+ sustained DRAM bandwidth usage
  - ~10GB/s sustained CXL bandwidth usage
  - request latencies 800us-2ms.

Functional observation:
  A 20 GB hash table VMA that previously remained entirely on CXL was
  split evenly between DRAM and CXL after the changes - and tier
  residency tracked hotness.  This was previously affected by the
  stavation issue caused by the "unaccessed VMA" filter.

Gregory Price (Meta) (4):
  mm: support promotion-only NUMA hinting scans
  mm: allow shared folios to be promoted to a fast tier
  sched/numa: scan read-only file mappings in tiering mode
  sched/numa: do not let VMA PID activity gate promotion

 include/linux/mm.h       |  4 ++-
 include/linux/mm_types.h |  7 ++++
 kernel/sched/fair.c      | 72 +++++++++++++++++++++++++++++-----------
 mm/huge_memory.c         |  3 +-
 mm/internal.h            |  5 +--
 mm/mempolicy.c           | 30 +++++++++++------
 mm/migrate.c             |  6 ++--
 mm/mprotect.c            |  4 ++-
 8 files changed, 93 insertions(+), 38 deletions(-)

-- 
2.55.0

[PATCH v2 1/4] mm: support promotion-only NUMA hinting scans

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

folio_can_map_prot_numa() derives the eligible memory from the global
balancing mode. Combined mode needs the scanner to distinguish placement
scans from promotion-only scans.

Add MM_CP_PROT_NUMA_PROMO_ONLY and let change_prot_numa() callers request
a promotion-only protection walk. Carry the choice with each walk so the
PTE and PMD paths use the same decision.

Initially derive the value from the normal balancing mode, preserving the
existing behavior for the following fixes.

Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 include/linux/mm.h  |  4 +++-
 kernel/sched/fair.c |  7 ++++++-
 mm/huge_memory.c    |  3 ++-
 mm/internal.h       |  5 +++--
 mm/mempolicy.c      | 22 +++++++++++++---------
 mm/mprotect.c       |  4 +++-
 6 files changed, 30 insertions(+), 15 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 969594074fd2..2d1b59a27629 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3408,6 +3408,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 #define  MM_CP_UFFD_RWP_RESOLVE            (1UL << 5) /* resolve rwp */
 #define  MM_CP_UFFD_RWP_ALL                (MM_CP_UFFD_RWP | \
 					    MM_CP_UFFD_RWP_RESOLVE)
+/* Whether a MM_CP_PROT_NUMA change is for promotion only */
+#define  MM_CP_PROT_NUMA_PROMO_ONLY        (1UL << 6)
 
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
@@ -4736,7 +4738,7 @@ void vma_set_file(struct vm_area_struct *vma, struct file *file);
 
 #ifdef CONFIG_NUMA_BALANCING
 unsigned long change_prot_numa(struct vm_area_struct *vma,
-			unsigned long start, unsigned long end);
+			unsigned long start, unsigned long end, bool promo_only);
 #endif
 
 struct vm_area_struct *find_extend_vma_locked(struct mm_struct *,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8dff37059faf..81359b414947 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4129,8 +4129,10 @@ static void task_numa_work(struct callback_head *work)
 	unsigned long nr_pte_updates = 0;
 	long pages, virtpages;
 	struct vma_iterator vmi;
+	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
 	bool vma_pids_skipped;
 	bool vma_pids_forced = false;
+	bool promo_only;
 
 	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
 
@@ -4304,11 +4306,14 @@ static void task_numa_work(struct callback_head *work)
 			continue;
 		}
 
+		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL);
+
 		do {
 			start = max(start, vma->vm_start);
 			end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
 			end = min(end, vma->vm_end);
-			nr_pte_updates = change_prot_numa(vma, start, end);
+			nr_pte_updates = change_prot_numa(vma, start, end,
+							  promo_only);
 
 			/*
 			 * Try to scan sysctl_numa_balancing_size worth of
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 30b7c63b0e35..2f9ada1bbcfc 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2784,7 +2784,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 			goto unlock;
 
 		if (!folio_can_map_prot_numa(pmd_folio(*pmd), vma,
-					     vma_is_single_threaded_private(vma)))
+					     vma_is_single_threaded_private(vma),
+					     cp_flags & MM_CP_PROT_NUMA_PROMO_ONLY))
 			goto unlock;
 	}
 	/*
diff --git a/mm/internal.h b/mm/internal.h
index 0dca33db068f..f3e093168f84 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1237,11 +1237,12 @@ static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
 
 #ifdef CONFIG_NUMA_BALANCING
 bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
-		bool is_private_single_threaded);
+		bool is_private_single_threaded, bool promo_only);
 
 #else
 static inline bool folio_can_map_prot_numa(struct folio *folio,
-		struct vm_area_struct *vma, bool is_private_single_threaded)
+		struct vm_area_struct *vma, bool is_private_single_threaded,
+		bool promo_only)
 {
 	return false;
 }
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 2ad0a5f18280..a082ccfa09ec 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -846,6 +846,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
  * @folio: The folio whose mapping considered for being made NUMA hintable
  * @vma: The VMA that the folio belongs to.
  * @is_private_single_threaded: Is this a single-threaded private VMA or not
+ * @promo_only: Whether this scan should only collect promotion candidates
  *
  * This function checks to see if the folio actually indicates that
  * we need to make the mapping one which causes a NUMA hinting fault,
@@ -855,7 +856,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
  * Return: True if the mapping of the folio needs to be changed, false otherwise.
  */
 bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
-		bool is_private_single_threaded)
+		bool is_private_single_threaded, bool promo_only)
 {
 	int nid;
 
@@ -886,12 +887,8 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	if (is_private_single_threaded && (nid == numa_node_id()))
 		return false;
 
-	/*
-	 * Skip scanning top tier node if normal numa
-	 * balancing is disabled
-	 */
-	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
-	    node_is_toptier(nid))
+	/* Promotion-only scans do not collect NUMA placement samples */
+	if (promo_only && node_is_toptier(nid))
 		return false;
 
 	if (folio_use_access_time(folio))
@@ -905,19 +902,26 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
  * These are later cleared by a NUMA hinting fault. Depending on these
  * faults, pages may be migrated for better NUMA placement.
  *
+ * With @promo_only only folios eligible for promotion are made
+ * hint-faultable, without also sampling for task placement.
+ *
  * This is assuming that NUMA faults are handled using PROT_NONE. If
  * an architecture makes a different choice, it will need further
  * changes to the core.
  */
 unsigned long change_prot_numa(struct vm_area_struct *vma,
-			unsigned long addr, unsigned long end)
+			unsigned long addr, unsigned long end, bool promo_only)
 {
+	unsigned long cp_flags = MM_CP_PROT_NUMA;
 	struct mmu_gather tlb;
 	long nr_updated;
 
+	if (promo_only)
+		cp_flags |= MM_CP_PROT_NUMA_PROMO_ONLY;
+
 	tlb_gather_mmu(&tlb, vma->vm_mm);
 
-	nr_updated = change_protection(&tlb, vma, addr, end, MM_CP_PROT_NUMA);
+	nr_updated = change_protection(&tlb, vma, addr, end, cp_flags);
 	if (nr_updated > 0) {
 		count_vm_numa_events(NUMA_PTE_UPDATES, nr_updated);
 		count_memcg_events_mm(vma->vm_mm, NUMA_PTE_UPDATES, nr_updated);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 2888ee638d87..4e66e4c7665b 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -337,6 +337,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 	long pages = 0;
 	bool is_private_single_threaded;
 	bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
+	bool numa_promo_only = cp_flags & MM_CP_PROT_NUMA_PROMO_ONLY;
 	bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	int nr_ptes;
@@ -384,7 +385,8 @@ static long change_pte_range(struct mmu_gather *tlb,
 			 */
 			if (prot_numa &&
 			    !folio_can_map_prot_numa(folio, vma,
-						is_private_single_threaded)) {
+						is_private_single_threaded,
+						numa_promo_only)) {
 
 				/* determine batch to skip */
 				nr_ptes = mprotect_folio_pte_batch(folio,
-- 
2.55.0

[PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

NUMA balancing rejects shared copy-on-write folios and executable
file folios mapped by multiple processes to avoid placement bouncing.
These checks also block promotion from slow memory.

Allow such folios to participate when moving from a slow tier to a fast
tier. Keep the existing restrictions for ordinary placement.

Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 mm/mempolicy.c | 8 ++++++--
 mm/migrate.c   | 6 ++++--
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index a082ccfa09ec..19b599bc2dd1 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -863,8 +863,12 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
 		return false;
 
-	/* Also skip shared copy-on-write folios */
-	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
+	/*
+	 * Shared copy-on-write folios are poor NUMA placement candidates, but
+	 * a hot folio on a slow tier still needs a hint fault for promotion.
+	 */
+	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio) &&
+	    !folio_use_access_time(folio))
 		return false;
 
 	/* Folios are pinned and can't be migrated */
diff --git a/mm/migrate.c b/mm/migrate.c
index a369d0c95c38..afd9c97d2389 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2697,12 +2697,14 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		/*
 		 * Do not migrate file folios that are mapped in multiple
 		 * processes with execute permissions as they are probably
-		 * shared libraries.
+		 * shared libraries, unless this is a promotion from a slow tier.
 		 *
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if ((vma->vm_flags & VM_EXEC) &&
+		    folio_maybe_mapped_shared(folio) &&
+		    (!folio_use_access_time(folio) || !node_is_toptier(node)))
 			return -EACCES;
 
 		/*
-- 
2.55.0

[PATCH v2 3/4] sched/numa: scan read-only file mappings in tiering mode

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

Commit 4591ce4f2d22 ("sched/numa: Do not trap hinting faults for
shared libraries") excludes read-only file mappings from NUMA hinting to
prevent placement bouncing. This also hides hot file folios on slow memory
from the tiering code.

Scan these mappings when memory tiering is enabled, but make their scans
promotion-only. Ordinary NUMA placement retains the existing restriction.

On a host with 768 GB of DRAM and 256 GB of CXL memory running a roughly
430 GB database service, 169 MB of its 185 MB main binary accumulated on
CXL before this change. Afterwards its tier residency tracked runtime load.

Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 kernel/sched/fair.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 81359b414947..e636e8de53f1 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4061,6 +4061,16 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	p->numa_faults_locality[local] += pages;
 }
 
+/*
+ * Read-only file-backed mappings are expected to be cache replicated between
+ * accessor nodes, so they are not worth sampling for placement.  They can
+ * still strand on the slow tier like anything else.
+ */
+static bool vma_is_ro_file(struct vm_area_struct *vma)
+{
+	return vma->vm_file && (vma->vm_flags & (VM_READ | VM_WRITE)) == VM_READ;
+}
+
 static void reset_ptenuma_scan(struct task_struct *p)
 {
 	/*
@@ -4220,13 +4230,13 @@ static void task_numa_work(struct callback_head *work)
 		}
 
 		/*
-		 * Shared library pages mapped by multiple processes are not
-		 * migrated as it is expected they are cache replicated. Avoid
-		 * hinting faults in read-only file-backed mappings or the vDSO
-		 * as migrating the pages will be of marginal benefit.
+		 * Read-only file-backed folios are poor NUMA placement
+		 * candidates, but slow-tier folios still need to be scanned for
+		 * promotion.
 		 */
 		if (!vma->vm_mm ||
-		    (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ))) {
+		    (vma_is_ro_file(vma) &&
+		     !(numab_mode & NUMA_BALANCING_MEMORY_TIERING))) {
 			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_SHARED_RO);
 			continue;
 		}
@@ -4306,7 +4316,8 @@ static void task_numa_work(struct callback_head *work)
 			continue;
 		}
 
-		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL);
+		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL) ||
+			     vma_is_ro_file(vma);
 
 		do {
 			start = max(start, vma->vm_start);
-- 
2.55.0

[PATCH v2 4/4] sched/numa: do not let VMA PID activity gate promotion

[Gregory Price]

From: "Gregory Price (Meta)" <gourry@gourry.net>

Commit fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
skips VMAs without recent PID activity. Since only NUMA hint faults record
that activity, the filter can suppress the fault needed to promote hot
slow-tier memory.

In tiering mode, scan PID-inactive VMAs using promotion-only scans.
Reevaluate this choice whenever the scanner visits a VMA and pass it with
each protection walk.

Record socket-placement scans in prev_placement_scan_seq. Promotion-only
scans still update prev_scan_seq, but no longer postpone the starvation
fallback for placement scans.

On a host with 768 GB of DRAM and 256 GB of CXL memory running two roughly
430 GB database workloads, a large shmem VMA occupied each scan while 2,537
other VMAs covering 84 GB were skipped as inactive. A hot 20 GB hash table
remained entirely on CXL before this change and was split evenly between
DRAM and CXL afterwards.

Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
---
 include/linux/mm_types.h |  7 ++++++
 kernel/sched/fair.c      | 48 ++++++++++++++++++++++++++--------------
 2 files changed, 39 insertions(+), 16 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 5413bd10fff2..9f042d6ad465 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -803,6 +803,13 @@ struct vma_numab_state {
 	 * A VMA is not eligible for scanning if prev_scan_seq == numa_scan_seq
 	 */
 	int prev_scan_seq;
+
+	/*
+	 * MM scan sequence ID when the VMA was last scanned for placement.
+	 * The starvation horizon in vma_is_accessed() counts against this, so
+	 * promotion-only scans cannot postpone placement indefinitely.
+	 */
+	int prev_placement_scan_seq;
 };
 
 #ifdef __HAVE_PFNMAP_TRACKING
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index e636e8de53f1..6d1da13a2ef5 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4085,6 +4085,10 @@ static void reset_ptenuma_scan(struct task_struct *p)
 	p->mm->numa_scan_offset = 0;
 }
 
+/*
+ * Decide whether this VMA should be sampled for NUMA placement.  In addition
+ * to recent accesses, periodically allow a scan to avoid starvation.
+ */
 static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
 {
 	unsigned long pids;
@@ -4101,22 +4105,13 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
 	if (test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids))
 		return true;
 
-	/*
-	 * Complete a scan that has already started regardless of PID access, or
-	 * some VMAs may never be scanned in multi-threaded applications:
-	 */
-	if (mm->numa_scan_offset > vma->vm_start) {
-		trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
-		return true;
-	}
-
 	/*
 	 * This vma has not been accessed for a while, and if the number
 	 * the threads in the same process is low, which means no other
 	 * threads can help scan this vma, force a vma scan.
 	 */
 	if (READ_ONCE(mm->numa_scan_seq) >
-	   (vma->numab_state->prev_scan_seq + get_nr_threads(current)))
+	   (vma->numab_state->prev_placement_scan_seq + get_nr_threads(current)))
 		return true;
 
 	return false;
@@ -4142,7 +4137,7 @@ static void task_numa_work(struct callback_head *work)
 	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
 	bool vma_pids_skipped;
 	bool vma_pids_forced = false;
-	bool promo_only;
+	bool accessed, scan_started, promo_only;
 
 	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
 
@@ -4278,6 +4273,7 @@ static void task_numa_work(struct callback_head *work)
 			 * first scan:
 			 */
 			 vma->numab_state->prev_scan_seq = mm->numa_scan_seq - 1;
+			 vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq - 1;
 		}
 
 		/*
@@ -4307,17 +4303,30 @@ static void task_numa_work(struct callback_head *work)
 		}
 
 		/*
-		 * Do not scan the VMA if task has not accessed it, unless no other
-		 * VMA candidate exists.
+		 * The PID filter must not gate promotion.  Scan PID-inactive
+		 * VMAs in tiering mode using promotion-only scans.
 		 */
-		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
+		accessed = vma_is_accessed(mm, vma);
+		scan_started = mm->numa_scan_offset > vma->vm_start;
+
+		if (!vma_pids_forced && !accessed && !scan_started &&
+		    !(numab_mode & NUMA_BALANCING_MEMORY_TIERING)) {
 			vma_pids_skipped = true;
 			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
 			continue;
 		}
+		if (!vma_pids_forced && !accessed &&
+		    !(numab_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+		    scan_started)
+			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
 
+		/*
+		 * In combined mode, only VMAs the task uses need placement
+		 * samples. Without tiering, every VMA reaching here does.
+		 */
 		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL) ||
-			     vma_is_ro_file(vma);
+			     ((numab_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+			      !accessed) || vma_is_ro_file(vma);
 
 		do {
 			start = max(start, vma->vm_start);
@@ -4345,8 +4354,15 @@ static void task_numa_work(struct callback_head *work)
 			cond_resched();
 		} while (end != vma->vm_end);
 
-		/* VMA scan is complete, do not scan until next sequence. */
+		/*
+		 * VMA scan is complete, do not scan until next sequence.  A
+		 * promotion-only scan reached the end of the VMA but did not
+		 * sample placement, so it does not count towards the starvation
+		 * horizon in vma_is_accessed().
+		 */
 		vma->numab_state->prev_scan_seq = mm->numa_scan_seq;
+		if (!promo_only)
+			vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq;
 
 		/*
 		 * Only force scan within one VMA at a time, to limit the
-- 
2.55.0

Re: [PATCH v2 0/4] sched/numa: stop VMA scan filters from gating promotion

[Gregory Price]

On Thu, Sep 10, 2026 at 08:18:22PM -0400, Gregory Price wrote:
> NUMA balancing uses hinting faults for both task placement and memory-tier
> promotion. Several filters designed to avoid unproductive socket-placement
> faults can also prevent promotion.
> 

sashiko notes - remaining notes appear to be false positives.

patch 2: "... bounce horizontally between slow tiers" <- nonsense
patch 3 and 4: it's existing code extracted into a helper, using the
               same checks we're already using.  If we want it updated,
               that can come in a separate patch.

note: this work has some backport conflicts, but tiered numa balancing
has effectively been broken multiple times since 2022 and 2023 due to
these filtering mechanisms.

~Gregory

Re: [PATCH v2 0/4] sched/numa: stop VMA scan filters from gating promotion

[Andrew Morton]

On Thu, 10 Sep 2026 20:18:22 -0400 Gregory Price <gourry@gourry.net> wrote:

> NUMA balancing uses hinting faults for both task placement and memory-tier
> promotion. Several filters designed to avoid unproductive socket-placement
> faults can also prevent promotion.
> 
> Read-only file mappings and VMAs without recent PID activity may never be
> scanned, while lower MM layers reject some shared folios. Hot memory in
> these mappings can therefore remain on a slow tier indefinitely.
> 
> Separate promotion-only scans from socket-placement scans.
> 
> MM_CP_PROT_NUMA_PROMO_ONLY carries that choice for one protection walk, so
> the PTE and PMD paths can restrict hinting faults to promotion candidates.
> Keeping this transient state in the protection flags avoids per-mm state and
> its associated lifetime, concurrency, and VMA identity problems.
> 
>   - Allow eligible shared folios to be promoted to a fast tier.
>   - Scan read-only file mappings and PID-inactive VMAs for promotion
>     without re-enabling placement sampling.
>   - Track the last placement scan separately so promotion-only scans
>     cannot postpone the existing placement-starvation fallback.
> 
> The series is ordered as follows:
> 
>   1. Add promotion-only NUMA protection walks without changing behavior.
>   2. Permit eligible shared folios to be promoted to a fast tier.
>   3. Scan read-only file mappings using promotion-only scans.
>   4. Scan PID-inactive VMAs for promotion and account for placement scans
>      separately.
> 
> Tested on a host with 768GB/256GB DRAM/CXL.
> Ran 2 ~430GB database workloads with large (>300GB) shmem VMAs.
> 
> Before change:
>   - 150-200GB/s DRAM bandwidth usage
>   - 40-45GB/s CXL bandwidth usage (maxed out)
>   - request latencies over 5ms (longer tails)
> 
> After chage:
>   - 250GB/s+ sustained DRAM bandwidth usage
>   - ~10GB/s sustained CXL bandwidth usage
>   - request latencies 800us-2ms.
> 
> Functional observation:
>   A 20 GB hash table VMA that previously remained entirely on CXL was
>   split evenly between DRAM and CXL after the changes - and tier
>   residency tracked hotness.  This was previously affected by the
>   stavation issue caused by the "unaccessed VMA" filter.

This seems very significant?

Why cc:stable and Fixes:?  Is this something which ran at these sorts
of speeds before the offending commits?

Re: [PATCH v2 0/4] sched/numa: stop VMA scan filters from gating promotion

[Gregory Price]

On Wed, Sep 16, 2026 at 10:35:25PM -0700, Andrew Morton wrote:
> On Thu, 10 Sep 2026 20:18:22 -0400 Gregory Price <gourry@gourry.net> wrote:
> 
> > 
> > Functional observation:
> >   A 20 GB hash table VMA that previously remained entirely on CXL was
> >   split evenly between DRAM and CXL after the changes - and tier
> >   residency tracked hotness.  This was previously affected by the
> >   stavation issue caused by the "unaccessed VMA" filter.
> 
> This seems very significant?
> 
> Why cc:stable and Fixes:?  Is this something which ran at these sorts
> of speeds before the offending commits?
> 

It's actually that the changes functionally broke numa balancing to the
point that it just became completely ineffective for very common use
cases.

We only didn't notice because another bug:
https://lore.kernel.org/all/20260629163337.1264881-1-hannes@cmpxchg.org/

hid almost all these issues from being apparent on large workloads
(which was differently horrendous - that bug made ~70-90% of some
 database workload memory ineligible for tiering)

Once the shmem fix went in, all the starvation and filtering issues
become more apparent when the bandwidth numbers stopped making sense
(see above).

Basically numa balancing has been broken since 2022/2023, and anyone
who has done any testing with it since then has been working off of
bad data (incomplete VMA coverage).

~Gregory

Re: [PATCH v2 0/4] sched/numa: stop VMA scan filters from gating promotion

[David Hildenbrand (Arm)]

On 9/17/26 08:59, Gregory Price wrote:
> On Wed, Sep 16, 2026 at 10:35:25PM -0700, Andrew Morton wrote:
>> On Thu, 10 Sep 2026 20:18:22 -0400 Gregory Price <gourry@gourry.net> wrote:
>>
>>>
>>> Functional observation:
>>>   A 20 GB hash table VMA that previously remained entirely on CXL was
>>>   split evenly between DRAM and CXL after the changes - and tier
>>>   residency tracked hotness.  This was previously affected by the
>>>   stavation issue caused by the "unaccessed VMA" filter.
>>
>> This seems very significant?
>>
>> Why cc:stable and Fixes:?  Is this something which ran at these sorts
>> of speeds before the offending commits?
>>
> 
> It's actually that the changes functionally broke numa balancing to the
> point that it just became completely ineffective for very common use
> cases.
> 
> We only didn't notice because another bug:
> https://lore.kernel.org/all/20260629163337.1264881-1-hannes@cmpxchg.org/
> 
> hid almost all these issues from being apparent on large workloads
> (which was differently horrendous - that bug made ~70-90% of some
>  database workload memory ineligible for tiering)
> 
> Once the shmem fix went in, all the starvation and filtering issues
> become more apparent when the bandwidth numbers stopped making sense
> (see above).
> 
> Basically numa balancing has been broken since 2022/2023, and anyone
> who has done any testing with it since then has been working off of
> bad data (incomplete VMA coverage).

I'll go through the patches from a MM side. I am not a big expert on
kernel/sched/fair.c side of things. Would be good if someone could have a look
on that.

-- 
Cheers,

David

Re: [PATCH v2 1/4] mm: support promotion-only NUMA hinting scans

[Peter Zijlstra]

On Thu, Sep 10, 2026 at 08:18:23PM -0400, Gregory Price wrote:

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 8dff37059faf..81359b414947 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4129,8 +4129,10 @@ static void task_numa_work(struct callback_head *work)
>  	unsigned long nr_pte_updates = 0;
>  	long pages, virtpages;
>  	struct vma_iterator vmi;
> +	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);

This thing sticks out like a sort thumb, does that want to be on top?
Typically reverse xmas is preferred and all that.

>  	bool vma_pids_skipped;
>  	bool vma_pids_forced = false;
> +	bool promo_only;
>  
>  	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
>  

> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -1237,11 +1237,12 @@ static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
>  
>  #ifdef CONFIG_NUMA_BALANCING
>  bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
> -		bool is_private_single_threaded);
> +		bool is_private_single_threaded, bool promo_only);

Two bools rather than updating the existing bool to a flags?

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[Peter Zijlstra]

On Thu, Sep 10, 2026 at 08:18:24PM -0400, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> NUMA balancing rejects shared copy-on-write folios and executable
> file folios mapped by multiple processes to avoid placement bouncing.
> These checks also block promotion from slow memory.
> 
> Allow such folios to participate when moving from a slow tier to a fast
> tier. Keep the existing restrictions for ordinary placement.
> 
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  mm/mempolicy.c | 8 ++++++--
>  mm/migrate.c   | 6 ++++--
>  2 files changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index a082ccfa09ec..19b599bc2dd1 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -863,8 +863,12 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
>  	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
>  		return false;
>  
> -	/* Also skip shared copy-on-write folios */
> -	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
> +	/*
> +	 * Shared copy-on-write folios are poor NUMA placement candidates, but
> +	 * a hot folio on a slow tier still needs a hint fault for promotion.
> +	 */
> +	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio) &&
> +	    !folio_use_access_time(folio))
>  		return false;
>  
>  	/* Folios are pinned and can't be migrated */
> diff --git a/mm/migrate.c b/mm/migrate.c
> index a369d0c95c38..afd9c97d2389 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -2697,12 +2697,14 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
>  		/*
>  		 * Do not migrate file folios that are mapped in multiple
>  		 * processes with execute permissions as they are probably
> -		 * shared libraries.
> +		 * shared libraries, unless this is a promotion from a slow tier.
>  		 *
>  		 * See folio_maybe_mapped_shared() on possible imprecision
>  		 * when we cannot easily detect if a folio is shared.
>  		 */
> -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
> +		if ((vma->vm_flags & VM_EXEC) &&
> +		    folio_maybe_mapped_shared(folio) &&
> +		    (!folio_use_access_time(folio) || !node_is_toptier(node)))
>  			return -EACCES;
>  

Semi related; I've often wondered if we should still account shared and
pinned vmas in the fault statistic, even though we should not migrate
them.

After all, those pages are still used and by not accounting them in the
fault statistics, it becomes easier to migrate a task away from them.

Using the scanning for two different things has made a mess of things
though :/

Re: [PATCH v2 1/4] mm: support promotion-only NUMA hinting scans

[Gregory Price]

On Thu, Sep 17, 2026 at 06:03:28PM +0200, Peter Zijlstra wrote:
> On Thu, Sep 10, 2026 at 08:18:23PM -0400, Gregory Price wrote:
> 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 8dff37059faf..81359b414947 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -4129,8 +4129,10 @@ static void task_numa_work(struct callback_head *work)
> >  	unsigned long nr_pte_updates = 0;
> >  	long pages, virtpages;
> >  	struct vma_iterator vmi;
> > +	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
> 
> This thing sticks out like a sort thumb, does that want to be on top?
> Typically reverse xmas is preferred and all that.
> 

oi yeah sorry, should have given this a closer style look.

will fix it up.

> >  	bool vma_pids_skipped;
> >  	bool vma_pids_forced = false;
> > +	bool promo_only;
> >  
> >  	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
> >  
> 
> > --- a/mm/internal.h
> > +++ b/mm/internal.h
> > @@ -1237,11 +1237,12 @@ static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
> >  
> >  #ifdef CONFIG_NUMA_BALANCING
> >  bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
> > -		bool is_private_single_threaded);
> > +		bool is_private_single_threaded, bool promo_only);
> 
> Two bools rather than updating the existing bool to a flags?

every time i've added a flag field someone screeches at me, so I tend
to avoid it now - but i can do that's the preference here.

Otherwise, i tend to follow rule of 3 for generalizing.

~Gregory

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[Gregory Price]

On Thu, Sep 17, 2026 at 06:08:27PM +0200, Peter Zijlstra wrote:
> > diff --git a/mm/migrate.c b/mm/migrate.c
> > index a369d0c95c38..afd9c97d2389 100644
> > --- a/mm/migrate.c
> > +++ b/mm/migrate.c
> > @@ -2697,12 +2697,14 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
> >  		/*
> >  		 * Do not migrate file folios that are mapped in multiple
> >  		 * processes with execute permissions as they are probably
> > -		 * shared libraries.
> > +		 * shared libraries, unless this is a promotion from a slow tier.
> >  		 *
> >  		 * See folio_maybe_mapped_shared() on possible imprecision
> >  		 * when we cannot easily detect if a folio is shared.
> >  		 */
> > -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
> > +		if ((vma->vm_flags & VM_EXEC) &&
> > +		    folio_maybe_mapped_shared(folio) &&
> > +		    (!folio_use_access_time(folio) || !node_is_toptier(node)))
> >  			return -EACCES;
> >  
> 
> Semi related; I've often wondered if we should still account shared and
> pinned vmas in the fault statistic, even though we should not migrate
> them.
> 
> After all, those pages are still used and by not accounting them in the
> fault statistics, it becomes easier to migrate a task away from them.
>

I don't have a strong opinion here to be honest, I'm just trying to get
tiering back on track.  There's some scheduler voodoo there that I will
happily claim ignorance on, so I just tried to keep things as-is here.

> Using the scanning for two different things has made a mess of things
> though :/

This has been my takeaway from this fix as well.

Honestly I'm starting to think hint faults are a big hammer making up
for the lack of hardware support for getting this data.

Would be nice to just have the hardware report what's hot (and how hot)
rather than depending on a software-heuristic like deriving hotness
from a page fault.

~Gregory

Re: [PATCH v2 4/4] sched/numa: do not let VMA PID activity gate promotion

[Peter Zijlstra]

On Thu, Sep 10, 2026 at 08:18:26PM -0400, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> Commit fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
> skips VMAs without recent PID activity. Since only NUMA hint faults record
> that activity, the filter can suppress the fault needed to promote hot
> slow-tier memory.
> 
> In tiering mode, scan PID-inactive VMAs using promotion-only scans.
> Reevaluate this choice whenever the scanner visits a VMA and pass it with
> each protection walk.
> 
> Record socket-placement scans in prev_placement_scan_seq. Promotion-only
> scans still update prev_scan_seq, but no longer postpone the starvation
> fallback for placement scans.
> 
> On a host with 768 GB of DRAM and 256 GB of CXL memory running two roughly
> 430 GB database workloads, a large shmem VMA occupied each scan while 2,537
> other VMAs covering 84 GB were skipped as inactive. A hot 20 GB hash table
> remained entirely on CXL before this change and was split evenly between
> DRAM and CXL afterwards.
> 
> Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  include/linux/mm_types.h |  7 ++++++
>  kernel/sched/fair.c      | 48 ++++++++++++++++++++++++++--------------
>  2 files changed, 39 insertions(+), 16 deletions(-)
> 
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index 5413bd10fff2..9f042d6ad465 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -803,6 +803,13 @@ struct vma_numab_state {
>  	 * A VMA is not eligible for scanning if prev_scan_seq == numa_scan_seq
>  	 */
>  	int prev_scan_seq;
> +
> +	/*
> +	 * MM scan sequence ID when the VMA was last scanned for placement.
> +	 * The starvation horizon in vma_is_accessed() counts against this, so
> +	 * promotion-only scans cannot postpone placement indefinitely.
> +	 */
> +	int prev_placement_scan_seq;
>  };
>  
>  #ifdef __HAVE_PFNMAP_TRACKING
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index e636e8de53f1..6d1da13a2ef5 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4085,6 +4085,10 @@ static void reset_ptenuma_scan(struct task_struct *p)
>  	p->mm->numa_scan_offset = 0;
>  }
>  
> +/*
> + * Decide whether this VMA should be sampled for NUMA placement.  In addition
> + * to recent accesses, periodically allow a scan to avoid starvation.
> + */
>  static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
>  {
>  	unsigned long pids;
> @@ -4101,22 +4105,13 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
>  	if (test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids))
>  		return true;
>  
> -	/*
> -	 * Complete a scan that has already started regardless of PID access, or
> -	 * some VMAs may never be scanned in multi-threaded applications:
> -	 */
> -	if (mm->numa_scan_offset > vma->vm_start) {
> -		trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
> -		return true;
> -	}
> -
>  	/*
>  	 * This vma has not been accessed for a while, and if the number
>  	 * the threads in the same process is low, which means no other
>  	 * threads can help scan this vma, force a vma scan.
>  	 */
>  	if (READ_ONCE(mm->numa_scan_seq) >
> -	   (vma->numab_state->prev_scan_seq + get_nr_threads(current)))
> +	   (vma->numab_state->prev_placement_scan_seq + get_nr_threads(current)))
>  		return true;
>  
>  	return false;
> @@ -4142,7 +4137,7 @@ static void task_numa_work(struct callback_head *work)
>  	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
>  	bool vma_pids_skipped;
>  	bool vma_pids_forced = false;
> -	bool promo_only;
> +	bool accessed, scan_started, promo_only;
>  
>  	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
>  
> @@ -4278,6 +4273,7 @@ static void task_numa_work(struct callback_head *work)
>  			 * first scan:
>  			 */
>  			 vma->numab_state->prev_scan_seq = mm->numa_scan_seq - 1;
> +			 vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq - 1;
>  		}
>  
>  		/*
> @@ -4307,17 +4303,30 @@ static void task_numa_work(struct callback_head *work)
>  		}
>  
>  		/*
> -		 * Do not scan the VMA if task has not accessed it, unless no other
> -		 * VMA candidate exists.
> +		 * The PID filter must not gate promotion.  Scan PID-inactive
> +		 * VMAs in tiering mode using promotion-only scans.
>  		 */
> -		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
> +		accessed = vma_is_accessed(mm, vma);
> +		scan_started = mm->numa_scan_offset > vma->vm_start;
> +
> +		if (!vma_pids_forced && !accessed && !scan_started &&
> +		    !(numab_mode & NUMA_BALANCING_MEMORY_TIERING)) {
>  			vma_pids_skipped = true;
>  			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
>  			continue;
>  		}
> +		if (!vma_pids_forced && !accessed &&
> +		    !(numab_mode & NUMA_BALANCING_MEMORY_TIERING) &&
> +		    scan_started)
> +			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
>  
> +		/*
> +		 * In combined mode, only VMAs the task uses need placement
> +		 * samples. Without tiering, every VMA reaching here does.
> +		 */
>  		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL) ||
> -			     vma_is_ro_file(vma);
> +			     ((numab_mode & NUMA_BALANCING_MEMORY_TIERING) &&
> +			      !accessed) || vma_is_ro_file(vma);
>  
>  		do {
>  			start = max(start, vma->vm_start);
> @@ -4345,8 +4354,15 @@ static void task_numa_work(struct callback_head *work)
>  			cond_resched();
>  		} while (end != vma->vm_end);
>  
> -		/* VMA scan is complete, do not scan until next sequence. */
> +		/*
> +		 * VMA scan is complete, do not scan until next sequence.  A
> +		 * promotion-only scan reached the end of the VMA but did not
> +		 * sample placement, so it does not count towards the starvation
> +		 * horizon in vma_is_accessed().
> +		 */
>  		vma->numab_state->prev_scan_seq = mm->numa_scan_seq;
> +		if (!promo_only)
> +			vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq;
>  
>  		/*
>  		 * Only force scan within one VMA at a time, to limit the

Not a fan of what that tiering code is causing :/

But I suppose this will do; Mel?

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[Peter Zijlstra]

On Thu, Sep 17, 2026 at 12:18:24PM -0400, Gregory Price wrote:
> On Thu, Sep 17, 2026 at 06:08:27PM +0200, Peter Zijlstra wrote:
> > > diff --git a/mm/migrate.c b/mm/migrate.c
> > > index a369d0c95c38..afd9c97d2389 100644
> > > --- a/mm/migrate.c
> > > +++ b/mm/migrate.c
> > > @@ -2697,12 +2697,14 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
> > >  		/*
> > >  		 * Do not migrate file folios that are mapped in multiple
> > >  		 * processes with execute permissions as they are probably
> > > -		 * shared libraries.
> > > +		 * shared libraries, unless this is a promotion from a slow tier.
> > >  		 *
> > >  		 * See folio_maybe_mapped_shared() on possible imprecision
> > >  		 * when we cannot easily detect if a folio is shared.
> > >  		 */
> > > -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
> > > +		if ((vma->vm_flags & VM_EXEC) &&
> > > +		    folio_maybe_mapped_shared(folio) &&
> > > +		    (!folio_use_access_time(folio) || !node_is_toptier(node)))
> > >  			return -EACCES;
> > >  
> > 
> > Semi related; I've often wondered if we should still account shared and
> > pinned vmas in the fault statistic, even though we should not migrate
> > them.
> > 
> > After all, those pages are still used and by not accounting them in the
> > fault statistics, it becomes easier to migrate a task away from them.
> >
> 
> I don't have a strong opinion here to be honest, I'm just trying to get
> tiering back on track.  There's some scheduler voodoo there that I will
> happily claim ignorance on, so I just tried to keep things as-is here.

Yeah, fair enough.

> > Using the scanning for two different things has made a mess of things
> > though :/
> 
> This has been my takeaway from this fix as well.
> 
> Honestly I'm starting to think hint faults are a big hammer making up
> for the lack of hardware support for getting this data.
> 
> Would be nice to just have the hardware report what's hot (and how hot)
> rather than depending on a software-heuristic like deriving hotness
> from a page fault.

Yeah, there is/was this patch-set from AMD that uses their IBS counters
for this, but 'ab'-using the performance counters for this also has ick.
PMU data isn't ideal either. Mostly they generate a ton of data that
needs to be analyzed as well. Its not clear cut and easy.

I'm not sure there's been proposals for better hardware support.

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[Gregory Price]

On Thu, Sep 17, 2026 at 06:23:24PM +0200, Peter Zijlstra wrote:
> > Would be nice to just have the hardware report what's hot (and how hot)
> > rather than depending on a software-heuristic like deriving hotness
> > from a page fault.
> 
> Yeah, there is/was this patch-set from AMD that uses their IBS counters
> for this, but 'ab'-using the performance counters for this also has ick.
> PMU data isn't ideal either. Mostly they generate a ton of data that
> needs to be analyzed as well. Its not clear cut and easy.
> 
> I'm not sure there's been proposals for better hardware support.

In the CXL Spec there's a "CXL Hot-range Monitoring Unit" mechanism that
will report the N hottest regions of the device.

Not sure we actually need this for the CPU-tiers. LRU already does a
decent job of recording coldness.

I think bandwidth data provides the missing glue between the two.

I've been a proponent of just using bandwidth data to drive *when* to
promote, and let the devices dictate *what* to promote.

https://lore.kernel.org/all/aPJ7qe_2xznFSUMZ@gourry-fedora-PF4VCD3F/

e.g. if the top-tier looks under-utilized in terms of bandwidth, get
some chunk of hot data from below and see if it shifts the needle.
If so, repeat until numbers stabilize or the top tier reaches
saturation - otherwise chill out.

</food for thought | ramblings>

~Gregory

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[Zi Yan]

On Thu Sep 10, 2026 at 8:18 PM EDT, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
>
> NUMA balancing rejects shared copy-on-write folios and executable
> file folios mapped by multiple processes to avoid placement bouncing.
> These checks also block promotion from slow memory.
>
> Allow such folios to participate when moving from a slow tier to a fast
> tier. Keep the existing restrictions for ordinary placement.
>
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  mm/mempolicy.c | 8 ++++++--
>  mm/migrate.c   | 6 ++++--
>  2 files changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index a082ccfa09ec..19b599bc2dd1 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -863,8 +863,12 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
>  	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
>  		return false;
>  
> -	/* Also skip shared copy-on-write folios */
> -	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
> +	/*
> +	 * Shared copy-on-write folios are poor NUMA placement candidates, but
> +	 * a hot folio on a slow tier still needs a hint fault for promotion.
> +	 */
> +	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio) &&
> +	    !folio_use_access_time(folio))
>  		return false;
>  
>  	/* Folios are pinned and can't be migrated */
> diff --git a/mm/migrate.c b/mm/migrate.c
> index a369d0c95c38..afd9c97d2389 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -2697,12 +2697,14 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
>  		/*
>  		 * Do not migrate file folios that are mapped in multiple
>  		 * processes with execute permissions as they are probably
> -		 * shared libraries.
> +		 * shared libraries, unless this is a promotion from a slow tier.
>  		 *
>  		 * See folio_maybe_mapped_shared() on possible imprecision
>  		 * when we cannot easily detect if a folio is shared.
>  		 */
> -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
> +		if ((vma->vm_flags & VM_EXEC) &&
> +		    folio_maybe_mapped_shared(folio) &&
> +		    (!folio_use_access_time(folio) || !node_is_toptier(node)))
>  			return -EACCES;
>  
>  		/*

Should we rename folio_use_access_time() to folio_in_lowtier()?
Otherwise the code is really hard to understand. I admit that I
introduced folio_use_access_time() and it was probably because it
decides the use of folio_xchg_access_time() in
folio_can_map_prot_numa(). But in the other callsites, folio_in_lowtier()
makes more sense.

After the rename, a comment "only record access time of folios in low tier"
above folio_xchg_access_time() should work.

-- 
Best Regards,
Yan, Zi

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[Bharata B Rao]

> On Thu, Sep 17, 2026 at 12:18:24PM -0400, Gregory Price wrote:
> > On Thu, Sep 17, 2026 at 06:08:27PM +0200, Peter Zijlstra wrote:
> > 
> > Honestly I'm starting to think hint faults are a big hammer making up
> > for the lack of hardware support for getting this data.
> > 
> > Would be nice to just have the hardware report what's hot (and how hot)
> > rather than depending on a software-heuristic like deriving hotness
> > from a page fault.
> 
> Yeah, there is/was this patch-set from AMD that uses their IBS counters
> for this, but 'ab'-using the performance counters for this also has ick.
> PMU data isn't ideal either. Mostly they generate a ton of data that
> needs to be analyzed as well. Its not clear cut and easy.

Here are my experiments with using IBS for memory access profiling:

1. My very early attempt was to replace NUMA hint faults with IBS
provided memory access information to drive NUMA Balancing. This
worked for all the modes of NUMA Balancing (kernel.numa_balancing=1
or 2) [1]

2. The next attempt was to use IBS data to drive only hot page promotion.
In this approach, IBS was used as one of the sources of page hotness
information to pghot (existing NUMA hintfaults being the other) [2]

Both the above approaches used the primary IBS instance that was being
used by perf sub-system also and I had made the use mutually exclusive.

> 
> I'm not sure there's been proposals for better hardware support.

Then AMD Zen6 processors introduced a 2nd light-weight IBS instance
called IBS Memory Profiler which is separate, works independently
of the primary IBS instance (which continues to be used by perf)
and which is primarily targeted for memory access profiling.

I have used this as source of page hotness with pghot (pghot-hwhints)
and the benchmark results are encouraging. [3]

Also it is worth reiterating here that neither primary IBS nor this
new IBS Memory Profiler have got anything to do with PMU sub-system.

In this context, I would also like to point out that pghot patchset [4]
is trying to move hot page promotion from scheduler to its own dedicated
sub-system. I have done the following till now:

- Extracted out hot page promotion engine and moved it to pghot
  so that the same gets used for other sources of page hotness.
- Moved fault-time migration to async and batched kernel-thread driven
  migration (kmigrated)
- Used NUMA hint faults as page hotness source to pghot (pghot-hintfaults)

I have often wondered if it makes sense to move out complete hint faulting
mechanism out of scheduler but then I see that task-follows-memory part,
the scanning logic, fault stats heuristics are tightly tied to the scheduler.

Also apt is to remember the PTE-A bit scanning approach [5] that was started
as a potential replacement to NUMA hint faults based scanning. We are
planning to revive that effort and make it as another source for pghot
if we get good results with different benchmarks.

Regards,
Bharata.

[1] Primary IBS instance driving NUMA Balancing
    https://lore.kernel.org/lkml/20230208073533.715-1-bharata@amd.com/
[2] The last pghot version (v5)  which used primary IBS instance as page hotness source
    https://lore.kernel.org/linux-mm/20260129144043.231636-1-bharata@amd.com/
[3] IBS Memory Profiler as page hotness source for pghot
    https://lore.kernel.org/linux-mm/92c26cce-0608-4c0d-bb13-fe87afc225ba@amd.com/T/#m6d17d0d58a5026c16d63c34e1abfd67a57a66256
[4] The last posted pghot (v8) patchset
    https://lore.kernel.org/linux-mm/20260728054356.291998-1-bharata@amd.com/
[5] Kscand - PTE A bit based scanning
    https://lore.kernel.org/linux-mm/20250814153307.1553061-1-raghavendra.kt@amd.com/

Re: [PATCH v2 1/4] mm: support promotion-only NUMA hinting scans

[David Hildenbrand (Arm)]

On 9/17/26 18:14, Gregory Price wrote:
> On Thu, Sep 17, 2026 at 06:03:28PM +0200, Peter Zijlstra wrote:
>> On Thu, Sep 10, 2026 at 08:18:23PM -0400, Gregory Price wrote:
>>
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index 8dff37059faf..81359b414947 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -4129,8 +4129,10 @@ static void task_numa_work(struct callback_head *work)
>>>  	unsigned long nr_pte_updates = 0;
>>>  	long pages, virtpages;
>>>  	struct vma_iterator vmi;
>>> +	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
>>
>> This thing sticks out like a sort thumb, does that want to be on top?
>> Typically reverse xmas is preferred and all that.
>>
> 
> oi yeah sorry, should have given this a closer style look.
> 
> will fix it up.
> 
>>>  	bool vma_pids_skipped;
>>>  	bool vma_pids_forced = false;
>>> +	bool promo_only;
>>>  
>>>  	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
>>>  
>>
>>> --- a/mm/internal.h
>>> +++ b/mm/internal.h
>>> @@ -1237,11 +1237,12 @@ static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
>>>  
>>>  #ifdef CONFIG_NUMA_BALANCING
>>>  bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
>>> -		bool is_private_single_threaded);
>>> +		bool is_private_single_threaded, bool promo_only);
>>
>> Two bools rather than updating the existing bool to a flags?
> 
> every time i've added a flag field someone screeches at me, so I tend
> to avoid it now - but i can do that's the preference here.
> 
> Otherwise, i tend to follow rule of 3 for generalizing.

The is_private_single_threaded only exists to repeatedly call
vma_is_single_threaded_private().

Maybe we don't care about that optimization.

So maybe move the vma_is_single_threaded_private() call into
folio_can_map_prot_numa() and avoid one bool entirely ... and then maybe just
pass the cp_flags?

-- 
Cheers,

David

Re: [PATCH v2 1/4] mm: support promotion-only NUMA hinting scans

[David Hildenbrand (Arm)]

On 9/11/26 02:18, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> folio_can_map_prot_numa() derives the eligible memory from the global
> balancing mode. Combined mode needs the scanner to distinguish placement
> scans from promotion-only scans.
> 
> Add MM_CP_PROT_NUMA_PROMO_ONLY and let change_prot_numa() callers request
> a promotion-only protection walk. Carry the choice with each walk so the
> PTE and PMD paths use the same decision.
> 
> Initially derive the value from the normal balancing mode, preserving the
> existing behavior for the following fixes.
> 
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  include/linux/mm.h  |  4 +++-
>  kernel/sched/fair.c |  7 ++++++-
>  mm/huge_memory.c    |  3 ++-
>  mm/internal.h       |  5 +++--
>  mm/mempolicy.c      | 22 +++++++++++++---------
>  mm/mprotect.c       |  4 +++-
>  6 files changed, 30 insertions(+), 15 deletions(-)
> 
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 969594074fd2..2d1b59a27629 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -3408,6 +3408,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
>  #define  MM_CP_UFFD_RWP_RESOLVE            (1UL << 5) /* resolve rwp */
>  #define  MM_CP_UFFD_RWP_ALL                (MM_CP_UFFD_RWP | \
>  					    MM_CP_UFFD_RWP_RESOLVE)
> +/* Whether a MM_CP_PROT_NUMA change is for promotion only */
> +#define  MM_CP_PROT_NUMA_PROMO_ONLY        (1UL << 6)

BTW, shouldn't we just be using BIT()?

>  
>  bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
>  			     pte_t pte);
> @@ -4736,7 +4738,7 @@ void vma_set_file(struct vm_area_struct *vma, struct file *file);
>  
>  #ifdef CONFIG_NUMA_BALANCING
>  unsigned long change_prot_numa(struct vm_area_struct *vma,
> -			unsigned long start, unsigned long end);
> +			unsigned long start, unsigned long end, bool promo_only);

While at it use two tabs.

>  #endif
>  
>  struct vm_area_struct *find_extend_vma_locked(struct mm_struct *,
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 8dff37059faf..81359b414947 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4129,8 +4129,10 @@ static void task_numa_work(struct callback_head *work)
>  	unsigned long nr_pte_updates = 0;
>  	long pages, virtpages;
>  	struct vma_iterator vmi;
> +	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);

const and all the way to the top?

>  	bool vma_pids_skipped;
>  	bool vma_pids_forced = false;
> +	bool promo_only;
>  
>  	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
>  
> @@ -4304,11 +4306,14 @@ static void task_numa_work(struct callback_head *work)
>  			continue;
>  		}
>  
> +		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL);

bool promo_only = !(numab_mode & NUMA_BALANCING_NORMAL);


and in the later patch

if (vma_is_ro_file(vma))
	promo_only = true;

?

But ...

> +
>  		do {
>  			start = max(start, vma->vm_start);
>  			end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);
>  			end = min(end, vma->vm_end);
> -			nr_pte_updates = change_prot_numa(vma, start, end);
> +			nr_pte_updates = change_prot_numa(vma, start, end,
> +							  promo_only);


Stupid question: why can't/shouldn't change_prot_numa() query
sysctl_numa_balancing_mode? Why do we have to query this outside of the function
and forward it?

I mean, change_prot_numa() gets the vma and can query
sysctl_numa_balancing_mode. Why not move that into the function and avoid the
boolean parameter?

We do have a single change_prot_numa() caller in the tree ...

>  
>  			/*
>  			 * Try to scan sysctl_numa_balancing_size worth of
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 30b7c63b0e35..2f9ada1bbcfc 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2784,7 +2784,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  			goto unlock;
>  
>  		if (!folio_can_map_prot_numa(pmd_folio(*pmd), vma,
> -					     vma_is_single_threaded_private(vma)))
> +					     vma_is_single_threaded_private(vma),
> +					     cp_flags & MM_CP_PROT_NUMA_PROMO_ONLY))

As raised, maybe just forward cp_flags



-- 
Cheers,

David

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[David Hildenbrand (Arm)]

On 9/11/26 02:18, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> NUMA balancing rejects shared copy-on-write folios and executable
> file folios mapped by multiple processes to avoid placement bouncing.
> These checks also block promotion from slow memory.

Right. How to we know the direction of promotion vs. demotion, though?

> 
> Allow such folios to participate when moving from a slow tier to a fast
> tier. Keep the existing restrictions for ordinary placement.
> 
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  mm/mempolicy.c | 8 ++++++--
>  mm/migrate.c   | 6 ++++--
>  2 files changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index a082ccfa09ec..19b599bc2dd1 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -863,8 +863,12 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
>  	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
>  		return false;
>  
> -	/* Also skip shared copy-on-write folios */
> -	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
> +	/*
> +	 * Shared copy-on-write folios are poor NUMA placement candidates, but
> +	 * a hot folio on a slow tier still needs a hint fault for promotion.
> +	 */
> +	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio) &&
> +	    !folio_use_access_time(folio))


While I understand a "!node_is_toptier(node)" to say "this is a slow tier", I am
clueless about the folio_use_access_time() check.

Confusing.

>  		return false;
>  
>  	/* Folios are pinned and can't be migrated */
> diff --git a/mm/migrate.c b/mm/migrate.c
> index a369d0c95c38..afd9c97d2389 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -2697,12 +2697,14 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
>  		/*
>  		 * Do not migrate file folios that are mapped in multiple
>  		 * processes with execute permissions as they are probably
> -		 * shared libraries.
> +		 * shared libraries, unless this is a promotion from a slow tier.
>  		 *
>  		 * See folio_maybe_mapped_shared() on possible imprecision
>  		 * when we cannot easily detect if a folio is shared.
>  		 */
> -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
> +		if ((vma->vm_flags & VM_EXEC) &&
> +		    folio_maybe_mapped_shared(folio) &&
> +		    (!folio_use_access_time(folio) || !node_is_toptier(node)))
>  			return -EACCES;

This screams for a readable helper function.

And my same comment for folio_use_access_time() applies, completely unclear how
that fits into the picture here.

-- 
Cheers,

David

Re: [PATCH v2 2/4] mm: allow shared folios to be promoted to a fast tier

[David Hildenbrand (Arm)]

On 9/17/26 19:49, Zi Yan wrote:
> On Thu Sep 10, 2026 at 8:18 PM EDT, Gregory Price wrote:
>> From: "Gregory Price (Meta)" <gourry@gourry.net>
>>
>> NUMA balancing rejects shared copy-on-write folios and executable
>> file folios mapped by multiple processes to avoid placement bouncing.
>> These checks also block promotion from slow memory.
>>
>> Allow such folios to participate when moving from a slow tier to a fast
>> tier. Keep the existing restrictions for ordinary placement.
>>
>> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
>> Cc: stable@vger.kernel.org
>> Assisted-by: LLM
>> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
>> ---
>>  mm/mempolicy.c | 8 ++++++--
>>  mm/migrate.c   | 6 ++++--
>>  2 files changed, 10 insertions(+), 4 deletions(-)
>>
>> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
>> index a082ccfa09ec..19b599bc2dd1 100644
>> --- a/mm/mempolicy.c
>> +++ b/mm/mempolicy.c
>> @@ -863,8 +863,12 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
>>  	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
>>  		return false;
>>  
>> -	/* Also skip shared copy-on-write folios */
>> -	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio))
>> +	/*
>> +	 * Shared copy-on-write folios are poor NUMA placement candidates, but
>> +	 * a hot folio on a slow tier still needs a hint fault for promotion.
>> +	 */
>> +	if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio) &&
>> +	    !folio_use_access_time(folio))
>>  		return false;
>>  
>>  	/* Folios are pinned and can't be migrated */
>> diff --git a/mm/migrate.c b/mm/migrate.c
>> index a369d0c95c38..afd9c97d2389 100644
>> --- a/mm/migrate.c
>> +++ b/mm/migrate.c
>> @@ -2697,12 +2697,14 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
>>  		/*
>>  		 * Do not migrate file folios that are mapped in multiple
>>  		 * processes with execute permissions as they are probably
>> -		 * shared libraries.
>> +		 * shared libraries, unless this is a promotion from a slow tier.
>>  		 *
>>  		 * See folio_maybe_mapped_shared() on possible imprecision
>>  		 * when we cannot easily detect if a folio is shared.
>>  		 */
>> -		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
>> +		if ((vma->vm_flags & VM_EXEC) &&
>> +		    folio_maybe_mapped_shared(folio) &&
>> +		    (!folio_use_access_time(folio) || !node_is_toptier(node)))
>>  			return -EACCES;
>>  
>>  		/*
> 
> Should we rename folio_use_access_time() to folio_in_lowtier()?
> Otherwise the code is really hard to understand. 

I just stumbled over that myself and I agree.

> I admit that I
> introduced folio_use_access_time() and it was probably because it
> decides the use of folio_xchg_access_time() in
> folio_can_map_prot_numa(). But in the other callsites, folio_in_lowtier()
> makes more sense.

We can just have an alias function if it makes the code easier to get.

-- 
Cheers,

David

Re: [PATCH v2 3/4] sched/numa: scan read-only file mappings in tiering mode

[David Hildenbrand (Arm)]

On 9/11/26 02:18, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> Commit 4591ce4f2d22 ("sched/numa: Do not trap hinting faults for
> shared libraries") excludes read-only file mappings from NUMA hinting to
> prevent placement bouncing. This also hides hot file folios on slow memory
> from the tiering code.
> 
> Scan these mappings when memory tiering is enabled, but make their scans
> promotion-only. Ordinary NUMA placement retains the existing restriction.
> 
> On a host with 768 GB of DRAM and 256 GB of CXL memory running a roughly
> 430 GB database service, 169 MB of its 185 MB main binary accumulated on
> CXL before this change. Afterwards its tier residency tracked runtime load.
> 
> Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  kernel/sched/fair.c | 23 +++++++++++++++++------
>  1 file changed, 17 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 81359b414947..e636e8de53f1 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4061,6 +4061,16 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
>  	p->numa_faults_locality[local] += pages;
>  }
>  
> +/*
> + * Read-only file-backed mappings are expected to be cache replicated between
> + * accessor nodes, so they are not worth sampling for placement.  They can
> + * still strand on the slow tier like anything else.
> + */
> +static bool vma_is_ro_file(struct vm_area_struct *vma)
> +{
> +	return vma->vm_file && (vma->vm_flags & (VM_READ | VM_WRITE)) == VM_READ;


MAP_PRIVATE can easily map a read-only file with write permissions. So the
function name is a bit misleading.

This smells like a helper that should go next to other vma helpers and have
clear semantics.

> +}
> +
>  static void reset_ptenuma_scan(struct task_struct *p)
>  {
>  	/*
> @@ -4220,13 +4230,13 @@ static void task_numa_work(struct callback_head *work)
>  		}
>  
>  		/*
> -		 * Shared library pages mapped by multiple processes are not
> -		 * migrated as it is expected they are cache replicated. Avoid
> -		 * hinting faults in read-only file-backed mappings or the vDSO
> -		 * as migrating the pages will be of marginal benefit.
> +		 * Read-only file-backed folios are poor NUMA placement
> +		 * candidates, but slow-tier folios still need to be scanned for
> +		 * promotion.
>  		 */
>  		if (!vma->vm_mm ||
> -		    (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ))) {
> +		    (vma_is_ro_file(vma) &&
> +		     !(numab_mode & NUMA_BALANCING_MEMORY_TIERING))) {

I'd vote for >80c here and but it into a singe line.

Or just use a magical helper

const bool tiering = numab_mode & NUMA_BALANCING_MEMORY_TIERING;

or sth like that.

>  			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_SHARED_RO);
>  			continue;
>  		}
> @@ -4306,7 +4316,8 @@ static void task_numa_work(struct callback_head *work)
>  			continue;
>  		}
>  
> -		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL);
> +		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL) ||
> +			     vma_is_ro_file(vma);

As I said, maybe that flag could be voided.

-- 
Cheers,

David

Re: [PATCH v2 4/4] sched/numa: do not let VMA PID activity gate promotion

[David Hildenbrand (Arm)]

On 9/11/26 02:18, Gregory Price wrote:
> From: "Gregory Price (Meta)" <gourry@gourry.net>
> 
> Commit fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
> skips VMAs without recent PID activity. Since only NUMA hint faults record
> that activity, the filter can suppress the fault needed to promote hot
> slow-tier memory.
> 
> In tiering mode, scan PID-inactive VMAs using promotion-only scans.
> Reevaluate this choice whenever the scanner visits a VMA and pass it with
> each protection walk.
> 
> Record socket-placement scans in prev_placement_scan_seq. Promotion-only
> scans still update prev_scan_seq, but no longer postpone the starvation
> fallback for placement scans.
> 
> On a host with 768 GB of DRAM and 256 GB of CXL memory running two roughly
> 430 GB database workloads, a large shmem VMA occupied each scan while 2,537
> other VMAs covering 84 GB were skipped as inactive. A hot 20 GB hash table
> remained entirely on CXL before this change and was split evenly between
> DRAM and CXL afterwards.
> 
> Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
> Cc: stable@vger.kernel.org
> Assisted-by: LLM
> Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
> ---
>  include/linux/mm_types.h |  7 ++++++
>  kernel/sched/fair.c      | 48 ++++++++++++++++++++++++++--------------
>  2 files changed, 39 insertions(+), 16 deletions(-)
> 
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index 5413bd10fff2..9f042d6ad465 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -803,6 +803,13 @@ struct vma_numab_state {
>  	 * A VMA is not eligible for scanning if prev_scan_seq == numa_scan_seq
>  	 */
>  	int prev_scan_seq;
> +
> +	/*
> +	 * MM scan sequence ID when the VMA was last scanned for placement.
> +	 * The starvation horizon in vma_is_accessed() counts against this, so
> +	 * promotion-only scans cannot postpone placement indefinitely.
> +	 */
> +	int prev_placement_scan_seq;
>  };
>  
>  #ifdef __HAVE_PFNMAP_TRACKING
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index e636e8de53f1..6d1da13a2ef5 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4085,6 +4085,10 @@ static void reset_ptenuma_scan(struct task_struct *p)
>  	p->mm->numa_scan_offset = 0;
>  }
>  
> +/*
> + * Decide whether this VMA should be sampled for NUMA placement.  In addition
> + * to recent accesses, periodically allow a scan to avoid starvation.
> + */
>  static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
>  {
>  	unsigned long pids;
> @@ -4101,22 +4105,13 @@ static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
>  	if (test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids))
>  		return true;
>  
> -	/*
> -	 * Complete a scan that has already started regardless of PID access, or
> -	 * some VMAs may never be scanned in multi-threaded applications:
> -	 */
> -	if (mm->numa_scan_offset > vma->vm_start) {
> -		trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
> -		return true;
> -	}
> -
>  	/*
>  	 * This vma has not been accessed for a while, and if the number
>  	 * the threads in the same process is low, which means no other
>  	 * threads can help scan this vma, force a vma scan.
>  	 */
>  	if (READ_ONCE(mm->numa_scan_seq) >
> -	   (vma->numab_state->prev_scan_seq + get_nr_threads(current)))
> +	   (vma->numab_state->prev_placement_scan_seq + get_nr_threads(current)))
>  		return true;
>  
>  	return false;
> @@ -4142,7 +4137,7 @@ static void task_numa_work(struct callback_head *work)
>  	unsigned int numab_mode = READ_ONCE(sysctl_numa_balancing_mode);
>  	bool vma_pids_skipped;
>  	bool vma_pids_forced = false;
> -	bool promo_only;
> +	bool accessed, scan_started, promo_only;
>  
>  	WARN_ON_ONCE(p != container_of(work, struct task_struct, numa_work));
>  
> @@ -4278,6 +4273,7 @@ static void task_numa_work(struct callback_head *work)
>  			 * first scan:
>  			 */
>  			 vma->numab_state->prev_scan_seq = mm->numa_scan_seq - 1;
> +			 vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq - 1;
>  		}
>  
>  		/*
> @@ -4307,17 +4303,30 @@ static void task_numa_work(struct callback_head *work)
>  		}
>  
>  		/*
> -		 * Do not scan the VMA if task has not accessed it, unless no other
> -		 * VMA candidate exists.
> +		 * The PID filter must not gate promotion.  Scan PID-inactive
> +		 * VMAs in tiering mode using promotion-only scans.
>  		 */
> -		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
> +		accessed = vma_is_accessed(mm, vma);
> +		scan_started = mm->numa_scan_offset > vma->vm_start;
> +
> +		if (!vma_pids_forced && !accessed && !scan_started &&
> +		    !(numab_mode & NUMA_BALANCING_MEMORY_TIERING)) {
>  			vma_pids_skipped = true;
>  			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
>  			continue;
>  		}
> +		if (!vma_pids_forced && !accessed &&
> +		    !(numab_mode & NUMA_BALANCING_MEMORY_TIERING) &&
> +		    scan_started)
> +			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_IGNORE_PID);
>  
> +		/*
> +		 * In combined mode, only VMAs the task uses need placement
> +		 * samples. Without tiering, every VMA reaching here does.
> +		 */
>  		promo_only = !(numab_mode & NUMA_BALANCING_NORMAL) ||
> -			     vma_is_ro_file(vma);
> +			     ((numab_mode & NUMA_BALANCING_MEMORY_TIERING) &&
> +			      !accessed) || vma_is_ro_file(vma);
>  
>  		do {
>  			start = max(start, vma->vm_start);
> @@ -4345,8 +4354,15 @@ static void task_numa_work(struct callback_head *work)
>  			cond_resched();
>  		} while (end != vma->vm_end);
>  
> -		/* VMA scan is complete, do not scan until next sequence. */
> +		/*
> +		 * VMA scan is complete, do not scan until next sequence.  A
> +		 * promotion-only scan reached the end of the VMA but did not
> +		 * sample placement, so it does not count towards the starvation
> +		 * horizon in vma_is_accessed().
> +		 */
>  		vma->numab_state->prev_scan_seq = mm->numa_scan_seq;
> +		if (!promo_only)
> +			vma->numab_state->prev_placement_scan_seq = mm->numa_scan_seq;


This is all rather ... messy. I mean, just look at that promo_only() computation ...

There must be a cleaner way ;)
-- 
Cheers,

David