[PATCH 0/6] sched/numa: fix several NUMA balancing issues

[PATCH 0/6] sched/numa: fix several NUMA balancing issues

[Li Zhe]

This series fixes several NUMA balancing issues around scan-period
adjustment, memory-tiering promotion feedback, and stale task-placement
state.

Patch 1 clears retained NUMA fault-window and locality state across
execve(), so the new image does not inherit feedback from the old one.

Patch 2 fixes the scan-period ratio denominator checks. The existing code
checks a different sum than the one later used as the divisor.

Patch 3 fixes scan-period adjustment for shared faults. Shared or local
faults should slow scanning down, while speeding up should require enough
faults that are both private and remote.

Patch 4 makes the free-space memory-tiering promotion bypass configurable.
The default behavior is preserved, but users can require the normal hotness
and rate-limit checks even when the top tier has enough free space.

Patch 5 prevents memory-tiering-only promotion faults from feeding normal
AutoNUMA task-placement statistics.

Patch 6 ignores stale preferred-node placement state while normal NUMA
balancing is disabled, so memory-tiering-only mode does not consume old
task-placement decisions.

Li Zhe (6):
  sched/numa: clear fault locality state on execve
  sched/numa: fix scan-period ratio denominator checks
  sched/numa: fix scan-period adjustment for shared faults
  sched/numa: make free-space tiering promotion bypass configurable
  sched/numa: do not feed tiering promotions into task placement stats
  sched/numa: ignore preferred nodes without normal balancing

 Documentation/admin-guide/sysctl/kernel.rst |  16 ++
 include/linux/mm.h                          |   5 +-
 include/linux/mmzone.h                      |   5 +-
 include/linux/sched.h                       |  17 +-
 kernel/sched/fair.c                         | 181 ++++++++++++++------
 mm/memory.c                                 |   2 +-
 6 files changed, 163 insertions(+), 63 deletions(-)

-- 
2.20.1

[PATCH 1/6] sched/numa: clear fault locality state on execve

[Li Zhe]

task_numa_free(p, false) is used by execve() to reset NUMA balancing
state without freeing p->numa_faults, because concurrent readers may
still observe the task.

The non-final path clears p->total_numa_faults and the averaged half of
the retained p->numa_faults array, but leaves the current-window buffer
half and p->numa_faults_locality unchanged. Those are also NUMA fault
feedback consumed by task_numa_placement() and update_task_scan_period(),
so a newly executed image can inherit stale fault feedback from the old
image.

Clear the whole retained p->numa_faults array and p->numa_faults_locality
together in the non-final reset path.

Fixes: 04bb2f947505 ("sched/numa: Adjust scan rate in task_numa_placement")
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 kernel/sched/fair.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 57360f5cdde4..1053437933ee 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4177,8 +4177,9 @@ void task_numa_free(struct task_struct *p, bool final)
 		kfree(numa_faults);
 	} else {
 		p->total_numa_faults = 0;
-		for (i = 0; i < NR_NUMA_HINT_FAULT_STATS * nr_node_ids; i++)
-			numa_faults[i] = 0;
+		memset(p->numa_faults_locality, 0, sizeof(p->numa_faults_locality));
+		memset(numa_faults, 0, sizeof(*numa_faults) *
+		       NR_NUMA_HINT_FAULT_BUCKETS * nr_node_ids);
 	}
 }
 
-- 
2.20.1

[PATCH 2/6] sched/numa: fix scan-period ratio denominator checks

[Li Zhe]

update_task_scan_period() scales the scan period with two ratios:
local/(local + remote) and private/(private + shared).

The existing guard only checks local + shared. That normally implies
that both ratio denominators are non-zero because the locality and type
counters are updated from the same hint faults. Memory hotplug can make
that assumption fragile because task_numa_placement() folds the
private/shared buffers by walking online nodes, while the locality counters
were already charged when the fault was recorded.

The divide-by-zero was reproduced on v7.3-rc1 in a QEMU guest by
hot-removing a NUMA node after NUMA balancing had collected fault feedback.
The next NUMA hint fault enters task_numa_placement() with locality
feedback that was charged before node1 was removed. The per-node
private/shared buffers for node1 are skipped by for_each_online_node(), so
update_task_scan_period() can see local > 0 while private and shared are
both 0. The old local + shared guard lets that state pass, and the
subsequent private/(private + shared) calculation divides by zero.

Keep the slow-scan fallback for empty or unusable samples, but test the
actual denominators before calculating the ratios.

Fixes: 37ec97deb3a8 ("sched/numa: Slow down scan rate if shared faults dominate")
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 kernel/sched/fair.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 1053437933ee..76532b8202dc 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3698,6 +3698,8 @@ static void update_task_scan_period(struct task_struct *p,
 
 	unsigned long remote = p->numa_faults_locality[0];
 	unsigned long local = p->numa_faults_locality[1];
+	unsigned long locality_faults = local + remote;
+	unsigned long type_faults = private + shared;
 
 	/*
 	 * If there were no record hinting faults then either the task is
@@ -3706,7 +3708,7 @@ static void update_task_scan_period(struct task_struct *p,
 	 * migration then it implies we are migrating too quickly or the local
 	 * node is overloaded. In either case, scan slower
 	 */
-	if (local + shared == 0 || p->numa_faults_locality[2]) {
+	if (!locality_faults || !type_faults || p->numa_faults_locality[2]) {
 		p->numa_scan_period = min(p->numa_scan_period_max,
 			p->numa_scan_period << 1);
 
@@ -3723,8 +3725,8 @@ static void update_task_scan_period(struct task_struct *p,
 	 *	 >= NUMA_PERIOD_THRESHOLD scan period increases (scan slower)
 	 */
 	period_slot = DIV_ROUND_UP(p->numa_scan_period, NUMA_PERIOD_SLOTS);
-	lr_ratio = (local * NUMA_PERIOD_SLOTS) / (local + remote);
-	ps_ratio = (private * NUMA_PERIOD_SLOTS) / (private + shared);
+	lr_ratio = (local * NUMA_PERIOD_SLOTS) / locality_faults;
+	ps_ratio = (private * NUMA_PERIOD_SLOTS) / type_faults;
 
 	if (ps_ratio >= NUMA_PERIOD_THRESHOLD) {
 		/*
-- 
2.20.1

[PATCH 3/6] sched/numa: fix scan-period adjustment for shared faults

[Li Zhe]

Commit 37ec97deb3a8 ("sched/numa: Slow down scan rate if shared faults
dominate") intended to slow scanning when either shared or local faults
dominate, and to speed scanning up only when many private faults are on
remote memory pages.

The implementation uses the private-fault ratio as the slow-scan signal.
That makes private-dominated samples slow scanning, while
shared-dominated samples can still fall through to the speed-up path.
That does not match the stated intent.

Replacing the private ratio with a shared ratio fixes the
shared-dominated case, but the scan-period code still only has separate
locality and sharing ratios. Those ratios cannot distinguish a window
with many remote shared faults and only a few remote private faults from
one with substantial remote private faults.

Track faults that are both remote and private, and only speed up scanning
when that signal is large enough. This keeps local or shared samples
slowing scanning and avoids extra hinting overhead when the remote
activity is mostly shared.

Fixes: 37ec97deb3a8 ("sched/numa: Slow down scan rate if shared faults dominate")
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/sched.h | 17 ++++++++---
 kernel/sched/fair.c   | 68 ++++++++++++++++++++++---------------------
 2 files changed, 48 insertions(+), 37 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index d35ae49a991f..1be7a9ba7553 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -832,6 +832,16 @@ struct task_ipi_mask {
 struct task_ipi_mask { };
 #endif
 
+#ifdef CONFIG_NUMA_BALANCING
+enum numa_faults_locality {
+	NUMA_FAULT_REMOTE,
+	NUMA_FAULT_LOCAL,
+	NUMA_FAULT_MIGRATE_FAIL,
+	NUMA_FAULT_REMOTE_PRIVATE,
+	NR_NUMA_FAULT_LOCALITY,
+};
+#endif
+
 struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -1422,11 +1432,10 @@ struct task_struct {
 
 	/*
 	 * numa_faults_locality tracks if faults recorded during the last
-	 * scan window were remote/local or failed to migrate. The task scan
-	 * period is adapted based on the locality of the faults with different
-	 * weights depending on whether they were shared or private faults
+	 * scan window were remote/local, remote private, or failed to migrate.
+	 * The task scan period is adapted based on those fault patterns.
 	 */
-	unsigned long			numa_faults_locality[3];
+	unsigned long			numa_faults_locality[NR_NUMA_FAULT_LOCALITY];
 
 	unsigned long			numa_pages_migrated;
 #endif /* CONFIG_NUMA_BALANCING */
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 76532b8202dc..ec8a6bfeb2c7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3675,10 +3675,10 @@ static void numa_group_count_active_nodes(struct numa_group *numa_group)
 
 /*
  * When adapting the scan rate, the period is divided into NUMA_PERIOD_SLOTS
- * increments. The more local the fault statistics are, the higher the scan
- * period will be for the next scan window. If local/(local+remote) ratio is
- * below NUMA_PERIOD_THRESHOLD (where range of ratio is 1..NUMA_PERIOD_SLOTS)
- * the scan period will decrease. Aim for 70% local accesses.
+ * increments. The more local or shared the fault statistics are, the higher
+ * the scan period will be for the next scan window. If the private remote
+ * ratio reaches NUMA_PERIOD_SLOTS - NUMA_PERIOD_THRESHOLD, the scan period
+ * will decrease. Aim for 70% local or shared accesses.
  */
 #define NUMA_PERIOD_SLOTS 10
 #define NUMA_PERIOD_THRESHOLD 7
@@ -3693,11 +3693,13 @@ static void update_task_scan_period(struct task_struct *p,
 			unsigned long shared, unsigned long private)
 {
 	unsigned int period_slot;
-	int lr_ratio, ps_ratio;
+	int lr_ratio, sr_ratio, rp_ratio, ratio;
 	int diff;
 
-	unsigned long remote = p->numa_faults_locality[0];
-	unsigned long local = p->numa_faults_locality[1];
+	unsigned long remote = p->numa_faults_locality[NUMA_FAULT_REMOTE];
+	unsigned long local = p->numa_faults_locality[NUMA_FAULT_LOCAL];
+	unsigned long remote_private =
+		p->numa_faults_locality[NUMA_FAULT_REMOTE_PRIVATE];
 	unsigned long locality_faults = local + remote;
 	unsigned long type_faults = private + shared;
 
@@ -3708,7 +3710,8 @@ static void update_task_scan_period(struct task_struct *p,
 	 * migration then it implies we are migrating too quickly or the local
 	 * node is overloaded. In either case, scan slower
 	 */
-	if (!locality_faults || !type_faults || p->numa_faults_locality[2]) {
+	if (!locality_faults || !type_faults ||
+	    p->numa_faults_locality[NUMA_FAULT_MIGRATE_FAIL]) {
 		p->numa_scan_period = min(p->numa_scan_period_max,
 			p->numa_scan_period << 1);
 
@@ -3721,40 +3724,36 @@ static void update_task_scan_period(struct task_struct *p,
 	/*
 	 * Prepare to scale scan period relative to the current period.
 	 *	 == NUMA_PERIOD_THRESHOLD scan period stays the same
-	 *       <  NUMA_PERIOD_THRESHOLD scan period decreases (scan faster)
+	 *       <  NUMA_PERIOD_THRESHOLD scan period may decrease if the
+	 *                            remote-private signal is strong enough
 	 *	 >= NUMA_PERIOD_THRESHOLD scan period increases (scan slower)
 	 */
 	period_slot = DIV_ROUND_UP(p->numa_scan_period, NUMA_PERIOD_SLOTS);
 	lr_ratio = (local * NUMA_PERIOD_SLOTS) / locality_faults;
-	ps_ratio = (private * NUMA_PERIOD_SLOTS) / type_faults;
+	sr_ratio = (shared * NUMA_PERIOD_SLOTS) / type_faults;
+	rp_ratio = (remote_private * NUMA_PERIOD_SLOTS) / locality_faults;
+	ratio = max(lr_ratio, sr_ratio);
 
-	if (ps_ratio >= NUMA_PERIOD_THRESHOLD) {
+	if (ratio >= NUMA_PERIOD_THRESHOLD) {
 		/*
-		 * Most memory accesses are local. There is no need to
-		 * do fast NUMA scanning, since memory is already local.
+		 * Most memory accesses are local or shared. Local faults do not
+		 * need fast scanning; shared faults may just be moved elsewhere by
+		 * other tasks.
 		 */
-		int slot = ps_ratio - NUMA_PERIOD_THRESHOLD;
-		if (!slot)
-			slot = 1;
-		diff = slot * period_slot;
-	} else if (lr_ratio >= NUMA_PERIOD_THRESHOLD) {
-		/*
-		 * Most memory accesses are shared with other tasks.
-		 * There is no point in continuing fast NUMA scanning,
-		 * since other tasks may just move the memory elsewhere.
-		 */
-		int slot = lr_ratio - NUMA_PERIOD_THRESHOLD;
+		int slot = ratio - NUMA_PERIOD_THRESHOLD;
 		if (!slot)
 			slot = 1;
 		diff = slot * period_slot;
 	} else {
 		/*
-		 * Private memory faults exceed (SLOTS-THRESHOLD)/SLOTS,
-		 * yet they are not on the local NUMA node. Speed up
-		 * NUMA scanning to get the memory moved over.
+		 * Only speed up scanning when enough faults are both private
+		 * and remote. Mixed windows dominated by shared remote faults
+		 * should not create extra hinting overhead.
 		 */
-		int ratio = max(lr_ratio, ps_ratio);
-		diff = -(NUMA_PERIOD_THRESHOLD - ratio) * period_slot;
+		if (rp_ratio >= NUMA_PERIOD_SLOTS - NUMA_PERIOD_THRESHOLD)
+			diff = -(NUMA_PERIOD_THRESHOLD - ratio) * period_slot;
+		else
+			diff = 0;
 	}
 
 	p->numa_scan_period = clamp(p->numa_scan_period + diff,
@@ -3906,8 +3905,8 @@ static void task_numa_placement(struct task_struct *p)
 	p->numa_scan_seq = seq;
 	p->numa_scan_period_max = task_scan_max(p);
 
-	total_faults = p->numa_faults_locality[0] +
-		       p->numa_faults_locality[1];
+	total_faults = p->numa_faults_locality[NUMA_FAULT_REMOTE] +
+		       p->numa_faults_locality[NUMA_FAULT_LOCAL];
 	runtime = numa_get_avg_runtime(p, &period);
 
 	/* If the task is part of a group prevent parallel updates to group stats */
@@ -4262,11 +4261,14 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	if (migrated)
 		p->numa_pages_migrated += pages;
 	if (flags & TNF_MIGRATE_FAIL)
-		p->numa_faults_locality[2] += pages;
+		p->numa_faults_locality[NUMA_FAULT_MIGRATE_FAIL] += pages;
 
 	p->numa_faults[task_faults_idx(NUMA_MEMBUF, mem_node, priv)] += pages;
 	p->numa_faults[task_faults_idx(NUMA_CPUBUF, cpu_node, priv)] += pages;
-	p->numa_faults_locality[local] += pages;
+	if (!local && priv)
+		p->numa_faults_locality[NUMA_FAULT_REMOTE_PRIVATE] += pages;
+	p->numa_faults_locality[local ? NUMA_FAULT_LOCAL :
+				 NUMA_FAULT_REMOTE] += pages;
 }
 
 static void reset_ptenuma_scan(struct task_struct *p)
-- 
2.20.1

[PATCH 4/6] sched/numa: make free-space tiering promotion bypass configurable

[Li Zhe]

Memory tiering normally promotes any recently accessed slow-tier page if
the target fast-tier node has enough free space. This preserves the
existing policy of filling available fast memory before relying on
hotness and rate-limit control.

That is not always the desired policy. Some workloads may intentionally
place memory across CXL-attached memory and DRAM to use aggregate memory
bandwidth. With the free-space bypass, cold one-shot accesses and bursts
of hot accesses can refill the fast tier and pull the workload away from
that distribution whenever DRAM has free space.

Add kernel.numa_balancing_promote_free_space_bypass. It defaults to 1,
preserving the existing behavior. When cleared, enough free space in the
target fast-tier node no longer bypasses the normal hotness and
rate-limit controls.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 Documentation/admin-guide/sysctl/kernel.rst | 16 ++++++++++++++
 include/linux/mmzone.h                      |  5 ++---
 kernel/sched/fair.c                         | 23 +++++++++++++++------
 3 files changed, 35 insertions(+), 9 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index ffea61d448eb..4a035b18aff4 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -765,6 +765,22 @@ will be limited to be no more than the set value.
 A rule of thumb is to set this to less than 1/10 of the PMEM node
 write bandwidth.
 
+numa_balancing_promote_free_space_bypass
+========================================
+
+Controls whether memory tiering promotion may bypass the normal
+hint-fault hotness check and promotion rate limit when the target
+fast-tier node has enough free space.
+
+The default value is 1.  In this mode, if the fast tier has enough free
+space, recently accessed slow-tier pages can be promoted without the
+hotness check or the promotion rate limit, so the kernel can fill
+available fast-tier capacity.
+
+When set to 0, enough free space no longer bypasses these controls.
+A slow-tier page must pass the normal hotness and rate-limit checks before
+promotion.
+
 oops_all_cpu_backtrace
 ======================
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff5416..5d583b17dfa2 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -297,9 +297,8 @@ enum node_stat_item {
 	 */
 	PGPROMOTE_CANDIDATE,
 	/**
-	 * Not rate-limited (NRL) candidate pages for those can be promoted
-	 * without considering hot threshold because of enough free pages in
-	 * fast-tier node.  These promotions bypass the regular hotness checks
+	 * Not rate-limited (NRL) candidate pages that are promoted through the
+	 * free-space bypass. These promotions bypass the regular hotness checks
 	 * and do NOT influence the promotion rate-limiter or
 	 * threshold-adjustment logic.
 	 * This is for statistics/monitoring purposes.
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ec8a6bfeb2c7..2e11107bf5ca 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -134,6 +134,8 @@ static unsigned int sysctl_sched_cfs_bandwidth_slice		= 5000UL;
 #ifdef CONFIG_NUMA_BALANCING
 /* Restrict the NUMA promotion throughput (MB/s) for each target node. */
 static unsigned int sysctl_numa_balancing_promote_rate_limit = 65536;
+/* Preserve the existing default of using free fast-tier space eagerly. */
+static unsigned int sysctl_numa_balancing_promote_free_space_bypass = 1;
 #endif
 
 #ifdef CONFIG_SYSCTL
@@ -157,6 +159,15 @@ static const struct ctl_table sched_fair_sysctls[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_ZERO,
 	},
+	{
+		.procname	= "numa_balancing_promote_free_space_bypass",
+		.data		= &sysctl_numa_balancing_promote_free_space_bypass,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_douintvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE,
+	},
 #endif /* CONFIG_NUMA_BALANCING */
 };
 
@@ -2816,11 +2827,10 @@ static inline bool cpupid_valid(int cpupid)
 }
 
 /*
- * For memory tiering mode, if there are enough free pages (more than
- * enough watermark defined here) in fast memory node, to take full
- * advantage of fast memory capacity, all recently accessed slow
- * memory pages will be migrated to fast memory node without
- * considering hot threshold.
+ * For memory tiering mode, if the target fast memory node has enough free
+ * pages, recently accessed slow memory pages can bypass the normal hotness
+ * check and rate limit so the fast tier can be filled. The bypass can be
+ * disabled by numa_balancing_promote_free_space_bypass.
  */
 static bool pgdat_free_space_enough(struct pglist_data *pgdat)
 {
@@ -2941,7 +2951,8 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 		long nr = folio_nr_pages(folio);
 
 		pgdat = NODE_DATA(dst_nid);
-		if (pgdat_free_space_enough(pgdat)) {
+		if (sysctl_numa_balancing_promote_free_space_bypass &&
+		    pgdat_free_space_enough(pgdat)) {
 			/* workload changed, reset hot threshold */
 			pgdat->nbp_threshold = 0;
 			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
-- 
2.20.1

[PATCH 5/6] sched/numa: do not feed tiering promotions into task placement stats

[Li Zhe]

NUMA balancing has separate mode bits for normal task placement and
memory tiering. In memory-tiering-only mode, NUMA hinting faults on
slow-tier pages are used to detect hot pages and promote them to a top
tier. They are not normal task-placement faults.

task_numa_fault() already avoids normal placement accounting for slow-tier
faults while memory tiering is enabled. However, a successful slow-to-fast
promotion can bypass that check. do_numa_page() reports the destination
node to task_numa_fault() after migration, so the fault is seen as a
top-tier fault and can update p->numa_faults, task placement state and
the preferred NUMA node.

Slow-tier PTEs store access-time metadata in the folio last-cpupid field.
Before this change, numa_migrate_check() returned the same all-ones
last_cpupid value for those faults that normal AutoNUMA uses for a first
or unknown access. Those cases are different: the ordinary unset value
means there is no previous CPU/PID sample, while a slow-tier access-time
value means the field does not contain a cpupid at all.

Name the ordinary all-ones sentinel LAST_CPUPID_UNSET and use it for the
normal first/unknown cpupid state. Return a distinct
LAST_CPUPID_ACCESS_TIME value while the folio still carries access-time
metadata. If normal NUMA balancing is disabled, keep the migrated-page
accounting but skip normal task-placement accounting for those faults.

When normal NUMA balancing and memory tiering are both enabled, promoted
pages can still feed task placement. In that mode, the user has enabled
both CPU placement feedback and tiering promotion, so a promoted top-tier
fault is treated as an ordinary unset-cpupid AutoNUMA fault.

A stale access-time value can also survive after memory tiering is
disabled and later reach task_numa_fault() from a top-tier page, where
the slow-tier validation no longer applies. Treat any remaining invalid
last_cpupid as LAST_CPUPID_UNSET before task_numa_group() can decode a
CPU from it.

This does not change whether a page is promoted. It only prevents
memory-tiering-only promotion faults from being consumed by the normal
AutoNUMA placement heuristic.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/mm.h  |  5 ++++-
 kernel/sched/fair.c | 26 +++++++++++++++++++++++++-
 mm/memory.c         |  2 +-
 3 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index dd09c438fa23..bb0c72640152 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2420,6 +2420,9 @@ static inline int folio_nid(const struct folio *folio)
 #define PAGE_ACCESS_TIME_MASK				\
 	(LAST_CPUPID_MASK << PAGE_ACCESS_TIME_BUCKETS)
 
+#define LAST_CPUPID_UNSET		(-1 & LAST_CPUPID_MASK)
+#define LAST_CPUPID_ACCESS_TIME		(-2 & LAST_CPUPID_MASK)
+
 static inline int cpu_pid_to_cpupid(int cpu, int pid)
 {
 	return ((cpu & LAST__CPU_MASK) << LAST__PID_SHIFT) | (pid & LAST__PID_MASK);
@@ -2468,7 +2471,7 @@ static inline int folio_last_cpupid(struct folio *folio)
 }
 static inline void page_cpupid_reset_last(struct page *page)
 {
-	page->_last_cpupid = -1 & LAST_CPUPID_MASK;
+	page->_last_cpupid = LAST_CPUPID_UNSET;
 }
 #else
 static inline int folio_last_cpupid(struct folio *folio)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 2e11107bf5ca..ec08406d1364 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4214,6 +4214,21 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	if (!p->mm)
 		return;
 
+	/*
+	 * Slow-tier pages use last_cpupid as access-time metadata instead of
+	 * a real cpupid. Do not feed these faults into normal task placement
+	 * when normal NUMA balancing is disabled.
+	 */
+	if (last_cpupid == LAST_CPUPID_ACCESS_TIME) {
+		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL)) {
+			if (migrated)
+				p->numa_pages_migrated += pages;
+			return;
+		}
+
+		last_cpupid = LAST_CPUPID_UNSET;
+	}
+
 	/*
 	 * NUMA faults statistics are unnecessary for the slow memory
 	 * node for memory tiering mode.
@@ -4223,6 +4238,15 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	     !cpupid_valid(last_cpupid)))
 		return;
 
+	/*
+	 * A stale slow-tier access timestamp can survive after memory tiering
+	 * is disabled and later reach here as if it was a cpupid from a
+	 * top-tier page. Treat any non-cpupid value as an ordinary first
+	 * access before task_numa_group() decodes a CPU from it.
+	 */
+	if (unlikely(!cpupid_valid(last_cpupid)))
+		last_cpupid = LAST_CPUPID_UNSET;
+
 	/* Allocate buffer to track faults on a per-node basis */
 	if (unlikely(!p->numa_faults)) {
 		int size = sizeof(*p->numa_faults) *
@@ -4240,7 +4264,7 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	 * First accesses are treated as private, otherwise consider accesses
 	 * to be private if the accessing pid has not changed
 	 */
-	if (unlikely(last_cpupid == (-1 & LAST_CPUPID_MASK))) {
+	if (unlikely(last_cpupid == LAST_CPUPID_UNSET)) {
 		priv = 1;
 	} else {
 		priv = cpupid_match_pid(p, last_cpupid);
diff --git a/mm/memory.c b/mm/memory.c
index 8b0c2c735d3d..212e9d1d5b64 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6173,7 +6173,7 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	 * to record page access time.  So use default value.
 	 */
 	if (folio_use_access_time(folio))
-		*last_cpupid = (-1 & LAST_CPUPID_MASK);
+		*last_cpupid = LAST_CPUPID_ACCESS_TIME;
 	else
 		*last_cpupid = folio_last_cpupid(folio);
 
-- 
2.20.1

[PATCH 6/6] sched/numa: ignore preferred nodes without normal balancing

[Li Zhe]

numa_preferred_nid is normal AutoNUMA task-placement state.
Memory-tiering-only mode uses NUMA hint faults to drive page promotion,
not CPU placement.

Even after tiering-only faults stop updating normal placement statistics,
a task can still have a preferred NUMA node from earlier normal balancing
or from previously polluted state. Several scheduler paths consume that
stale state when the NUMA balancing static key is enabled, including
explicit NUMA task migration, sched-cache preferred LLC and scan CPU mask
selection, load-balance locality checks, fbq classification, and wakeup
group selection.

Add a small helper for normal NUMA placement being enabled and use it to
ignore numa_preferred_nid when NUMA_BALANCING_NORMAL is disabled. Keep
the preferred-node migration retry timestamp update before the normal-mode
check so accepted non-normal sysctl states do not leave stale retry state
behind.

Keep the rq nr_numa_running and nr_preferred_running accounting unchanged
so runtime mode changes do not make enqueue/dequeue accounting asymmetric;
the load balancer simply stops consuming those counters while normal
balancing is off.

This does not clear numa_preferred_nid and does not change memory-tiering
promotion decisions.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index ec08406d1364..11b54bb353ad 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1328,6 +1328,21 @@ static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
 static unsigned long task_h_load(struct task_struct *p);
 static unsigned long capacity_of(int cpu);
 
+#ifdef CONFIG_NUMA_BALANCING
+static inline bool numa_balancing_normal_enabled(void)
+{
+	return sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL;
+}
+
+static inline int task_numa_preferred_nid(struct task_struct *p)
+{
+	if (!numa_balancing_normal_enabled())
+		return NUMA_NO_NODE;
+
+	return p->numa_preferred_nid;
+}
+#endif
+
 /* Give new sched_entity start runnable values to heavy its load in infant time */
 void init_entity_runnable_average(struct sched_entity *se)
 {
@@ -1890,10 +1905,12 @@ static int get_pref_llc(struct task_struct *p, struct sched_cache_group *grp)
 		 * than sched_setnuma() at least -- and thus the
 		 * conflict only exists for a short period of time.
 		 */
-		if (static_branch_likely(&sched_numa_balancing) &&
-		    p->numa_preferred_nid >= 0 &&
-		    cpu_to_node(mm_sched_cpu) != p->numa_preferred_nid)
-			mm_sched_llc = -1;
+		if (static_branch_likely(&sched_numa_balancing)) {
+			int pref_nid = task_numa_preferred_nid(p);
+
+			if (pref_nid >= 0 && cpu_to_node(mm_sched_cpu) != pref_nid)
+				mm_sched_llc = -1;
+		}
 #endif
 	}
 
@@ -2009,7 +2026,7 @@ static void get_scan_cpumasks(cpumask_var_t cpus, struct task_struct *p,
 	 * current running node to mitigate the bouncing risk.
 	 * TBD: numa_group should be considered during task aggregation.
 	 */
-	pref_nid = p->numa_preferred_nid;
+	pref_nid = task_numa_preferred_nid(p);
 	/* honor the task's preferred node */
 	if (pref_nid == NUMA_NO_NODE)
 		goto out;
@@ -3649,6 +3666,9 @@ static void numa_migrate_preferred(struct task_struct *p)
 	interval = min(interval, msecs_to_jiffies(p->numa_scan_period) / 16);
 	p->numa_migrate_retry = jiffies + interval;
 
+	if (!numa_balancing_normal_enabled())
+		return;
+
 	/* Success if task is already running on preferred CPU */
 	if (task_node(p) == p->numa_preferred_nid)
 		return;
@@ -10740,6 +10760,9 @@ static long migrate_degrades_locality(struct task_struct *p, struct lb_env *env)
 	if (!static_branch_likely(&sched_numa_balancing))
 		return 0;
 
+	if (!numa_balancing_normal_enabled())
+		return 0;
+
 	if (!p->numa_faults || !(env->sd->flags & SD_NUMA))
 		return 0;
 
@@ -12533,6 +12556,9 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 #ifdef CONFIG_NUMA_BALANCING
 static inline enum fbq_type fbq_classify_group(struct sg_lb_stats *sgs)
 {
+	if (!numa_balancing_normal_enabled())
+		return all;
+
 	if (sgs->sum_h_nr_running > sgs->nr_numa_running)
 		return regular;
 	if (sgs->sum_h_nr_running > sgs->nr_preferred_running)
@@ -12542,6 +12568,9 @@ static inline enum fbq_type fbq_classify_group(struct sg_lb_stats *sgs)
 
 static inline enum fbq_type fbq_classify_rq(struct rq *rq)
 {
+	if (!numa_balancing_normal_enabled())
+		return regular;
+
 	if (rq->nr_running > rq->nr_numa_running)
 		return regular;
 	if (rq->nr_running > rq->nr_preferred_running)
@@ -12843,17 +12872,21 @@ sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, int
 		if (sd->flags & SD_NUMA) {
 			int imb_numa_nr = sd->imb_numa_nr;
 #ifdef CONFIG_NUMA_BALANCING
-			int idlest_cpu;
+			int pref_nid = task_numa_preferred_nid(p);
 			/*
 			 * If there is spare capacity at NUMA, try to select
 			 * the preferred node
 			 */
-			if (cpu_to_node(this_cpu) == p->numa_preferred_nid)
-				return NULL;
+			if (pref_nid != NUMA_NO_NODE) {
+				int idlest_cpu;
 
-			idlest_cpu = cpumask_first(sched_group_span(idlest));
-			if (cpu_to_node(idlest_cpu) == p->numa_preferred_nid)
-				return idlest;
+				if (cpu_to_node(this_cpu) == pref_nid)
+					return NULL;
+
+				idlest_cpu = cpumask_first(sched_group_span(idlest));
+				if (cpu_to_node(idlest_cpu) == pref_nid)
+					return idlest;
+			}
 #endif /* CONFIG_NUMA_BALANCING */
 			/*
 			 * Otherwise, keep the task close to the wakeup source
-- 
2.20.1

Re: [PATCH 2/6] sched/numa: fix scan-period ratio denominator checks

[Gregory Price]

On Wed, Oct 14, 2026 at 05:43:43PM +0800, Li Zhe wrote:
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 1053437933ee..76532b8202dc 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3698,6 +3698,8 @@ static void update_task_scan_period(struct task_struct *p,
>  
>  	unsigned long remote = p->numa_faults_locality[0];
>  	unsigned long local = p->numa_faults_locality[1];
> +	unsigned long locality_faults = local + remote;
> +	unsigned long type_faults = private + shared;
>  
>  	/*
>  	 * If there were no record hinting faults then either the task is
> @@ -3706,7 +3708,7 @@ static void update_task_scan_period(struct task_struct *p,
>  	 * migration then it implies we are migrating too quickly or the local
>  	 * node is overloaded. In either case, scan slower
>  	 */
> -	if (local + shared == 0 || p->numa_faults_locality[2]) {
> +	if (!locality_faults || !type_faults || p->numa_faults_locality[2]) {
>  		p->numa_scan_period = min(p->numa_scan_period_max,
>  			p->numa_scan_period << 1);
> 

This does more than just change the div/o below, it changes the behavior
of the function overall.  It's not clear to me this change is correct.

> @@ -3723,8 +3725,8 @@ static void update_task_scan_period(struct task_struct *p,
>  	 *	 >= NUMA_PERIOD_THRESHOLD scan period increases (scan slower)
>  	 */
>  	period_slot = DIV_ROUND_UP(p->numa_scan_period, NUMA_PERIOD_SLOTS);
> -	lr_ratio = (local * NUMA_PERIOD_SLOTS) / (local + remote);
> -	ps_ratio = (private * NUMA_PERIOD_SLOTS) / (private + shared);
> +	lr_ratio = (local * NUMA_PERIOD_SLOTS) / locality_faults;
> +	ps_ratio = (private * NUMA_PERIOD_SLOTS) / type_faults;
>  

This otherwise looks fine.

~Gregory

Re: [PATCH 1/6] sched/numa: clear fault locality state on execve

[Gregory Price]

On Wed, Oct 14, 2026 at 05:43:42PM +0800, Li Zhe wrote:
> task_numa_free(p, false) is used by execve() to reset NUMA balancing
> state without freeing p->numa_faults, because concurrent readers may
> still observe the task.
> 

Is a process that exec()'s going to run long enough to actually collect
any real numa fault data?

> the retained p->numa_faults array, but leaves the current-window buffer
> half and p->numa_faults_locality unchanged. Those are also NUMA fault
> feedback consumed by task_numa_placement() and update_task_scan_period(),
> so a newly executed image can inherit stale fault feedback from the old
> image.

I'm not sure this actually corrects much unless you have a long running
program that eventually exec()'s - and even then it seems like this all
self-corrects within a couple seconds.

Do you have some clear reproduction that makes an obvious wrong effect,
or was this just some static analysis?

It's "not wrong", but I'm trying to understand what impact this actually
has.

> The non-final path clears p->total_numa_faults and the averaged half of
> 
> Clear the whole retained p->numa_faults array and p->numa_faults_locality
> together in the non-final reset path.
> 
> Fixes: 04bb2f947505 ("sched/numa: Adjust scan rate in task_numa_placement")
> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
> ---
>  kernel/sched/fair.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 57360f5cdde4..1053437933ee 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -4177,8 +4177,9 @@ void task_numa_free(struct task_struct *p, bool final)
>  		kfree(numa_faults);
>  	} else {
>  		p->total_numa_faults = 0;
> -		for (i = 0; i < NR_NUMA_HINT_FAULT_STATS * nr_node_ids; i++)
> -			numa_faults[i] = 0;
> +		memset(p->numa_faults_locality, 0, sizeof(p->numa_faults_locality));
> +		memset(numa_faults, 0, sizeof(*numa_faults) *
> +		       NR_NUMA_HINT_FAULT_BUCKETS * nr_node_ids);
>  	}
>  }
>  
> -- 
> 2.20.1

Re: [PATCH 3/6] sched/numa: fix scan-period adjustment for shared faults

[Gregory Price]

On Wed, Oct 14, 2026 at 05:43:44PM +0800, Li Zhe wrote:
> Commit 37ec97deb3a8 ("sched/numa: Slow down scan rate if shared faults
> dominate") intended to slow scanning when either shared or local faults
> dominate, and to speed scanning up only when many private faults are on
> remote memory pages.
> 
> The implementation uses the private-fault ratio as the slow-scan signal.
> That makes private-dominated samples slow scanning, while
> shared-dominated samples can still fall through to the speed-up path.
> That does not match the stated intent.
> 
> Replacing the private ratio with a shared ratio fixes the
> shared-dominated case, but the scan-period code still only has separate
> locality and sharing ratios. Those ratios cannot distinguish a window
> with many remote shared faults and only a few remote private faults from
> one with substantial remote private faults.
> 
> Track faults that are both remote and private, and only speed up scanning
> when that signal is large enough. This keeps local or shared samples
> slowing scanning and avoids extra hinting overhead when the remote
> activity is mostly shared.
> 

This changes NUMA balancing behavior that's been around since 2017.

While it's not the stated intent, it's been consistent since then,
and i'm concerned that this could cause unexpected regressions.

Do you have performance numbers on this change to show it's actually
beneficial compared to the existing behavior?

~Gregory

Re: [PATCH 4/6] sched/numa: make free-space tiering promotion bypass configurable

[Gregory Price]

On Wed, Oct 14, 2026 at 05:43:45PM +0800, Li Zhe wrote:
> Memory tiering normally promotes any recently accessed slow-tier page if
> the target fast-tier node has enough free space. This preserves the
> existing policy of filling available fast memory before relying on
> hotness and rate-limit control.
> 
> That is not always the desired policy. Some workloads may intentionally
> place memory across CXL-attached memory and DRAM to use aggregate memory
> bandwidth. With the free-space bypass, cold one-shot accesses and bursts
> of hot accesses can refill the fast tier and pull the workload away from
> that distribution whenever DRAM has free space.
> 
> Add kernel.numa_balancing_promote_free_space_bypass. It defaults to 1,
> preserving the existing behavior. When cleared, enough free space in the
> target fast-tier node no longer bypasses the normal hotness and
> rate-limit controls.
>

A global switch for this is a poor solution for this problem.

Also, if the memory was spread out intentionally, it presumably was done
so with a memory policy.  In that case, if the user did not set
MPOL_F_NUMA_BALANCING - that spread is already protected because the
mempolicy overrides the numa balancing fault and no migration occurs.

I don't think this is the right direction.

~Gregory

Re: [PATCH 5/6] sched/numa: do not feed tiering promotions into task placement stats

[Gregory Price]

On Wed, Oct 14, 2026 at 05:43:46PM +0800, Li Zhe wrote:
> NUMA balancing has separate mode bits for normal task placement and
> memory tiering. In memory-tiering-only mode, NUMA hinting faults on
> slow-tier pages are used to detect hot pages and promote them to a top
> tier. They are not normal task-placement faults.
> 
> task_numa_fault() already avoids normal placement accounting for slow-tier
> faults while memory tiering is enabled. However, a successful slow-to-fast
> promotion can bypass that check. do_numa_page() reports the destination
> node to task_numa_fault() after migration, so the fault is seen as a
> top-tier fault and can update p->numa_faults, task placement state and
> the preferred NUMA node.
> 
> Slow-tier PTEs store access-time metadata in the folio last-cpupid field.
> Before this change, numa_migrate_check() returned the same all-ones
> last_cpupid value for those faults that normal AutoNUMA uses for a first
> or unknown access. Those cases are different: the ordinary unset value
> means there is no previous CPU/PID sample, while a slow-tier access-time
> value means the field does not contain a cpupid at all.
> 
> Name the ordinary all-ones sentinel LAST_CPUPID_UNSET and use it for the
> normal first/unknown cpupid state. Return a distinct
> LAST_CPUPID_ACCESS_TIME value while the folio still carries access-time
> metadata. If normal NUMA balancing is disabled, keep the migrated-page
> accounting but skip normal task-placement accounting for those faults.
> 
> When normal NUMA balancing and memory tiering are both enabled, promoted
> pages can still feed task placement. In that mode, the user has enabled
> both CPU placement feedback and tiering promotion, so a promoted top-tier
> fault is treated as an ordinary unset-cpupid AutoNUMA fault.
> 
> A stale access-time value can also survive after memory tiering is
> disabled and later reach task_numa_fault() from a top-tier page, where
> the slow-tier validation no longer applies. Treat any remaining invalid
> last_cpupid as LAST_CPUPID_UNSET before task_numa_group() can decode a
> CPU from it.
> 
> This does not change whether a page is promoted. It only prevents
> memory-tiering-only promotion faults from being consumed by the normal
> AutoNUMA placement heuristic.
>

Similarly with prior patches - this is a pretty major change. Although I
agree with the premise, do you have any performance numbers that show
this change is beneficial?

It's quite difficult to determine what the user-face effects of this
change would be other than expecting some kind of performance benefit.

~Gregory

Re: [PATCH 6/6] sched/numa: ignore preferred nodes without normal balancing

[Gregory Price]

On Wed, Oct 14, 2026 at 05:43:47PM +0800, Li Zhe wrote:
> numa_preferred_nid is normal AutoNUMA task-placement state.
> Memory-tiering-only mode uses NUMA hint faults to drive page promotion,
> not CPU placement.
> 
> Even after tiering-only faults stop updating normal placement statistics,
> a task can still have a preferred NUMA node from earlier normal balancing
> or from previously polluted state. Several scheduler paths consume that
> stale state when the NUMA balancing static key is enabled, including
> explicit NUMA task migration, sched-cache preferred LLC and scan CPU mask
> selection, load-balance locality checks, fbq classification, and wakeup
> group selection.
> 
> Add a small helper for normal NUMA placement being enabled and use it to
> ignore numa_preferred_nid when NUMA_BALANCING_NORMAL is disabled. Keep
> the preferred-node migration retry timestamp update before the normal-mode
> check so accepted non-normal sysctl states do not leave stale retry state
> behind.
> 
> Keep the rq nr_numa_running and nr_preferred_running accounting unchanged
> so runtime mode changes do not make enqueue/dequeue accounting asymmetric;
> the load balancer simply stops consuming those counters while normal
> balancing is off.
> 
> This does not clear numa_preferred_nid and does not change memory-tiering
> promotion decisions.
> 

Same feedback - need to present user-facing performance changes to show
the impact of this change.

Sashiko also has some correctness feedback.

~Gregory