[PATCH 0/2] sched/numa: stop VMA scan filters from gating promotion

[PATCH 0/2] sched/numa: stop VMA scan filters from gating promotion

[Gregory Price]

task_numa_work() applies a series of VMA-level filters before it will mark
a VMA for hint faults.  Those filters were written when the hint fault was
a socket-residency signal. Skipping a VMA only cost some resolution.

Under memory tiering the hint fault is not a signal, it is the mechanism.
A folio on the slow tier is considered for promotion only because the scan
marked it and something then touched it.  A VMA excluded from the scan is
excluded from promotion, and stays excluded, because the filters take
their input from faults the scan itself produces.

Patch 1 covers the per-VMA PID filter, which is self-referential: a VMA
needs a hint fault to earn a scan, and a scan to produce a hint fault.

Patch 2 covers the read-only file mapping exclusion, whose premise:
"if we are never going to migrate the page, it is overhead for no gain"
only applies to socket-residency, not tier-residency.

Neither patch removes a filter.

Patch 1 adds vma->numab_state->slow_only, which restricts one scan of one
VMA to folios on non-toptier nodes, and scans the rejected VMAs under it.

Patch 2 reuses it.  A toptier folio is never marked anywhere the filters
would not already have marked it, in any mode, so placement behaviour is
unchanged and only promotion candidates are added.

On a 768G DRAM + 256G CXL host running two ~430G database services:

  before - DRAM bandwidth decayed from 200GB/s to 150GB/s, CXL climbed
           from 5GB/s to 45GB/s and capped, request latency went from
           800us to 5ms and tracked CXL bandwidth

  after  - DRAM holds 200-250GB/s, CXL holds 7-10GB/s, request latency
           sits at 800us-2ms and tracks request load

Gregory Price (2):
  sched/numa: do not let the per-VMA PID filter gate promotion
  sched/numa: scan read-only file mappings in tiering mode

 include/linux/mm_types.h |  7 +++++++
 kernel/sched/fair.c      | 41 ++++++++++++++++++++++++++++++++++------
 mm/mempolicy.c           |  9 +++++----
 3 files changed, 47 insertions(+), 10 deletions(-)

-- 
2.55.0

[PATCH 1/2] sched/numa: do not let the per-VMA PID filter gate promotion

[Gregory Price]

task_numa_work() will not scan a VMA that vma_is_accessed() rejects:

  if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
      vma_pids_skipped = true;
      trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
      continue;
  }

vma_is_accessed() wants the scanning thread's hash bit in
vma->numab_state->pids_active[].  Those bits have one setter,
vma_set_access_pid_bit(), called on a NUMA hint fault.

The PROT_NONE scan is the only source of hint faults, so a VMA the
task has not hint-faulted looks untouched - preventing new scans
from causing faults for potentially hours or days.

The existing escape mechanisms do not solve this problem:

  numa_scan_offset      - rescues only the VMA holding the scan cursor

  get_nr_threads()      - the horizon from commit f22cde4371f3
                          ("sched/numa: Fix the vma scan starving issue")
                          costs nr_threads scan sequences, which
                          can be incredibly large.

  vma_pids_forced retry - fires only at the end of the VMA list, and then
                          scans exactly one VMA

Observed effect in the following environment:

  - 768G DRAM + 256G CXL Host
  - two ~430GB highly-threaded database workload
  - a single large (300GB+) shmem/tmpfs VMA in each workload
  - numa_balancing = 2   (default tuning)

In this environment, the large shmem VMAs held the cursor for 100% of a
scan, while 84G across 2537 other VMAs were skipped due to "inactivity".

Among them, a 20G hash table ended up 100% on CXL.  After this patch it
was consistently split 50:50 between DRAM and CXL.

Scan the rejected VMAs, looking only at folios on non-toptier nodes.
Promotion candidates then come from the whole address space, while toptier
folios are only marked where vma_is_accessed() already allowed it, so
top-tier marking is unchanged in every mode.

A restricted scan reaches the end of the VMA but may not completely scan
it, so it leaves prev_scan_seq alone.  Setting it there would permanently
disarm the get_nr_threads() horizon above.

Before this change:

  DRAM Bandwidth utilization dropped from 200GB/s to 150GB/s
  CXL Bandwidth utilization increased from 5GB/s to 45GB/s (capped)
  Request Latency increases from 800us to 5ms tracked with CXL bandwidth

After this change:

  DRAM Bandwidth maintains between 200GB/s and 250GB/s
  CXL Bandwidth maintains at ~7-10GB/s
  Request Latency hovers between 800us-2ms tracked with request load.

commit d230991493b5 ("mm: mempolicy: fix automatic numa balancing for shmem")
surfaced this issue when it fixed shmem numa balance scanning.

Fixes: fc137c0ddab2 ("sched/numa: enhance vma scanning logic")
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Assisted-by: Claude:claude-opus-5
---
 include/linux/mm_types.h |  7 +++++++
 kernel/sched/fair.c      | 25 ++++++++++++++++++++-----
 mm/mempolicy.c           |  9 +++++----
 3 files changed, 32 insertions(+), 9 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 5413bd10fff2..d986fcab9c8f 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -803,6 +803,13 @@ struct vma_numab_state {
 	 * A VMA is not eligible for scanning if prev_scan_seq == numa_scan_seq
 	 */
 	int prev_scan_seq;
+
+	/*
+	 * Set by the scanner for the duration of a scan of this VMA when only
+	 * folios on non-toptier nodes should be made NUMA hintable, because
+	 * the scanning task has not accessed the VMA recently.
+	 */
+	bool slow_only;
 };
 
 #ifdef __HAVE_PFNMAP_TRACKING
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 6d881e530f89..990af0e45d69 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4297,11 +4297,20 @@ static void task_numa_work(struct callback_head *work)
 		/*
 		 * Do not scan the VMA if task has not accessed it, unless no other
 		 * VMA candidate exists.
+		 *
+		 * Force-scan only slow-tier folios when in tiering mode, as a large
+		 * VMA can cause others to become "permanently unaccessed" if a scan
+		 * cycle is consumed entirely by the large VMA.
 		 */
+		vma->numab_state->slow_only = false;
 		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
-			vma_pids_skipped = true;
-			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
-			continue;
+			if (!(sysctl_numa_balancing_mode &
+			      NUMA_BALANCING_MEMORY_TIERING)) {
+				vma_pids_skipped = true;
+				trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
+				continue;
+			}
+			vma->numab_state->slow_only = true;
 		}
 
 		do {
@@ -4329,8 +4338,14 @@ static void task_numa_work(struct callback_head *work)
 			cond_resched();
 		} while (end != vma->vm_end);
 
-		/* VMA scan is complete, do not scan until next sequence. */
-		vma->numab_state->prev_scan_seq = mm->numa_scan_seq;
+		/*
+		 * VMA scan is complete, do not scan until next sequence.
+		 * A restricted scan reached the end of the VMA but may not
+		 * have completely scanned it - it must not disarm the sequence
+		 * counting horizon in vma_is_accessed() for this VMA.
+		 */
+		if (!vma->numab_state->slow_only)
+			vma->numab_state->prev_scan_seq = mm->numa_scan_seq;
 
 		/*
 		 * Only force scan within one VMA at a time, to limit the
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index ce10ce437464..e3fe08c4f109 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -887,11 +887,12 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 		return false;
 
 	/*
-	 * Skip scanning top tier node if normal numa
-	 * balancing is disabled
+	 * Skip scanning top tier node if normal numa balancing is disabled,
+	 * or if the scanner only wants promotion candidates out of this VMA.
 	 */
-	if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) &&
-	    node_is_toptier(nid))
+	if (node_is_toptier(nid) &&
+	    (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL) ||
+	     (vma->numab_state && vma->numab_state->slow_only)))
 		return false;
 
 	if (folio_use_access_time(folio))
-- 
2.55.0

Re: [PATCH 1/2] sched/numa: do not let the per-VMA PID filter gate promotion

[Gregory Price]

>  		/*
>  		 * Do not scan the VMA if task has not accessed it, unless no other
>  		 * VMA candidate exists.
> +		 *
> +		 * Force-scan only slow-tier folios when in tiering mode, as a large
> +		 * VMA can cause others to become "permanently unaccessed" if a scan
> +		 * cycle is consumed entirely by the large VMA.
>  		 */
> +		vma->numab_state->slow_only = false;
>  		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
> -			vma_pids_skipped = true;
> -			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
> -			continue;
> +			if (!(sysctl_numa_balancing_mode &
> +			      NUMA_BALANCING_MEMORY_TIERING)) {
> +				vma_pids_skipped = true;
> +				trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_PID_INACTIVE);
> +				continue;
> +			}
> +			vma->numab_state->slow_only = true;
>  		}

sashiko (correctly) reported that slow_only=false set unconditionally
here causes a VMA that's scanned over two passes to be prematurely
re-admitted regardless of vma_is_accessed() activity.

Patch looks simple:

+               if (mm->numa_scan_offset <= vma->vm_start)
+                       vma->numab_state->slow_only = false;

and on patch two:

-                       vma->numab_state->slow_only = false;
+                       vma->numab_state->slow_only = ro_file;

I'll follow up with a v2 next week after I test it.

~Gregory

[PATCH 2/2] sched/numa: scan read-only file mappings in tiering mode

[Gregory Price]

task_numa_work() refuses to scan any read-only file-backed mapping:

  if (vma->vm_file &&
      (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ)) {
          trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_SHARED_RO);
          continue;
  }

commit 4591ce4f2d22 ("sched/numa: Do not trap hinting faults for shared
libraries") added this because such pages are expected to be cache
replicated and bounce between accessor nodes, so:

  If we are never going to migrate the page, it is overhead for no gain

Memory tiering broke that premise.  A slow tier folio is promoted on
hint fault based on hotness, so these can migrate now - they just never
get the chance, because the VMA is never scanned.

Observed on a 768G DRAM + 256G CXL host running a ~430G database service:

  91% of its main binary (169M of 185M) accumulated on CXL.

After this patch the binary's tier residency tracked runtime load.

Scan these VMAs with the slow_only restriction. Only non-toptier folios
are made hint-faultable, so the scan never marks a toptier folio in a
read-only file mapping in any mode.  A page already on the fast tier costs
nothing, and a page on the slow tier costs one hint fault, which is what
buys the promotion.

That fault is not free of placement effect: a successful promotion reports
its toptier destination node to task_numa_fault(), so it lands in
numa_faults[] like any other promotion.  Only the set of VMAs that can
produce one changes.

Fixes: c574bbe91703 ("NUMA balancing: optimize page placement for memory tiering system")
Signed-off-by: Gregory Price (Meta) <gourry@gourry.net>
Assisted-by: Claude:claude-opus-5
---
 kernel/sched/fair.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 990af0e45d69..5c5c80b17649 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4075,6 +4075,11 @@ static void reset_ptenuma_scan(struct task_struct *p)
 	p->mm->numa_scan_offset = 0;
 }
 
+static bool vma_is_ro_file(struct vm_area_struct *vma)
+{
+	return vma->vm_file && (vma->vm_flags & (VM_READ | VM_WRITE)) == VM_READ;
+}
+
 static bool vma_is_accessed(struct mm_struct *mm, struct vm_area_struct *vma)
 {
 	unsigned long pids;
@@ -4211,6 +4216,8 @@ static void task_numa_work(struct callback_head *work)
 	}
 
 	for (; vma; vma = vma_next(&vmi)) {
+		bool ro_file;
+
 		if (!vma_migratable(vma) || !vma_policy_mof(vma) ||
 			is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) {
 			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_UNSUITABLE);
@@ -4222,9 +4229,16 @@ static void task_numa_work(struct callback_head *work)
 		 * migrated as it is expected they are cache replicated. Avoid
 		 * hinting faults in read-only file-backed mappings or the vDSO
 		 * as migrating the pages will be of marginal benefit.
+		 *
+		 * Tiering still wants them.  A read-only mapping strands just
+		 * as much on the slow tier as any other, so scan it for slow
+		 * tier folios alone.  Nothing on the top tier is marked here,
+		 * in any mode.
 		 */
+		ro_file = vma_is_ro_file(vma);
 		if (!vma->vm_mm ||
-		    (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ))) {
+		    (ro_file && !(sysctl_numa_balancing_mode &
+				  NUMA_BALANCING_MEMORY_TIERING))) {
 			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_SHARED_RO);
 			continue;
 		}
@@ -4302,7 +4316,7 @@ static void task_numa_work(struct callback_head *work)
 		 * VMA can cause others to become "permanently unaccessed" if a scan
 		 * cycle is consumed entirely by the large VMA.
 		 */
-		vma->numab_state->slow_only = false;
+		vma->numab_state->slow_only = ro_file;
 		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
 			if (!(sysctl_numa_balancing_mode &
 			      NUMA_BALANCING_MEMORY_TIERING)) {
-- 
2.55.0

Re: [PATCH 2/2] sched/numa: scan read-only file mappings in tiering mode

[Gregory Price]

On Fri, Sep 04, 2026 at 02:20:06PM -0400, Gregory Price wrote:
> @@ -4302,7 +4316,7 @@ static void task_numa_work(struct callback_head *work)
>  		 * VMA can cause others to become "permanently unaccessed" if a scan
>  		 * cycle is consumed entirely by the large VMA.
>  		 */
> -		vma->numab_state->slow_only = false;
> +		vma->numab_state->slow_only = ro_file;
>  		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
>  			if (!(sysctl_numa_balancing_mode &
>  			      NUMA_BALANCING_MEMORY_TIERING)) {

Sashiko thinks this causes unneeded rescanning, but I don't think it
does. The cursor prevents it from restarting, and when we loop back
around reset_ptenuma_scan() bumps the sequence.

It also had concerns about concurrent scanners and ptl lock contention,
but mm->numa_next_scan only admits one per scan period.

Either way on a rewalk the scan hits pte_protnone() and moves on.

~Gregory

Re: [PATCH 2/2] sched/numa: scan read-only file mappings in tiering mode

[Gregory Price]

On Fri, Sep 04, 2026 at 03:31:19PM -0400, Gregory Price wrote:
> On Fri, Sep 04, 2026 at 02:20:06PM -0400, Gregory Price wrote:
> > @@ -4302,7 +4316,7 @@ static void task_numa_work(struct callback_head *work)
> >  		 * VMA can cause others to become "permanently unaccessed" if a scan
> >  		 * cycle is consumed entirely by the large VMA.
> >  		 */
> > -		vma->numab_state->slow_only = false;
> > +		vma->numab_state->slow_only = ro_file;
> >  		if (!vma_pids_forced && !vma_is_accessed(mm, vma)) {
> >  			if (!(sysctl_numa_balancing_mode &
> >  			      NUMA_BALANCING_MEMORY_TIERING)) {
> 
> Sashiko thinks this causes unneeded rescanning, but I don't think it
> does. The cursor prevents it from restarting, and when we loop back
> around reset_ptenuma_scan() bumps the sequence.
> 
> It also had concerns about concurrent scanners and ptl lock contention,
> but mm->numa_next_scan only admits one per scan period.
> 
> Either way on a rewalk the scan hits pte_protnone() and moves on.
> 

On a second look, there are a few issues bundled in with this, and the
change required to handle the concurrent scan + mode=3 (_NORMAL |
_TIERING) requires a bit more complex of a change to fully resolve.

Will push a v2 after some additional testing - but that said the
existing tests still stand.  As-is, any mode with _TIERING set is very
broken at the moment because of the filtering mechanisms.

~Gregory