[PATCH v2 0/3] sched/numa: refine memory tiering promotion heuristics

[PATCH v2 0/3] sched/numa: refine memory tiering promotion heuristics

[Li Zhe]

This version narrows the series to memory tiering.  The generic normal
NUMA balancing cleanups and heuristic changes from v1 are not included.

Patch 1 makes the free-space promotion bypass configurable while keeping
the existing eager-promotion behavior as the default.  Disabling the
bypass forces slow-tier promotions through the normal hotness and
rate-limit checks, which helps workloads that intentionally keep part of
their memory on a slow tier avoid short cold bursts refilling the fast
tier.

Patch 2 keeps memory-tiering-only promotion faults out of normal AutoNUMA
task-placement statistics.  Slow-tier NUMA hint faults use the folio
last-cpupid field as access-time metadata, so those faults should not be
interpreted as normal placement feedback.  The patch marks them explicitly
and skips the placement accounting path.

Patch 3 builds on that separation.  Once tiering faults are no longer fed
into task placement stats, tiering-only mode can use tiering-specific
outcomes to tune the scan period: hot feedback speeds scanning up, while
cold, throttled, failed or empty windows slow it down.  This prevents
tiering-only promotion feedback from being handled by the normal placement
heuristic, where repeated promotions can otherwise push the scan period
towards the maximum.

Li Zhe (3):
  sched/numa: make free-space tiering promotion bypass configurable
  sched/numa: do not feed tiering promotions into task placement stats
  sched/numa: fix scan-period adjustment for tiering-only mode

 include/linux/mmzone.h               |   5 +-
 include/linux/sched.h                |  17 ++++
 include/linux/sched/numa_balancing.h |  10 ++
 kernel/sched/debug.c                 |   2 +
 kernel/sched/fair.c                  | 140 +++++++++++++++++++++++++--
 kernel/sched/sched.h                 |   1 +
 mm/memory.c                          |   6 +-
 7 files changed, 168 insertions(+), 13 deletions(-)

---
v1: https://lore.kernel.org/all/20261014094347.99157-1-lizhe.67@bytedance.com/

Changes since v1:
- Dropped the generic normal NUMA balancing patches.
- Added patch 3 to dynamically adjust the scan period in
  memory-tiering-only mode.

-- 
2.20.1

[PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Li Zhe]

Memory tiering normally promotes any recently accessed slow-tier page if
the target fast-tier node has enough free space. This preserves the
existing policy of filling available fast memory before relying on
hotness and rate-limit control.

That is not always the desired policy. Some deployments may intentionally
keep part of a workload's memory on CXL-attached memory, for example to
preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
the free-space bypass, short cold accesses and bursts of hot accesses can
refill the fast tier and pull the workload away from that distribution
whenever DRAM has free space.

Redis testing exercises this policy issue. On a CXL system with node0 as
DRAM and node3 as slow-tier memory, a Redis test was run with
memory-tiering-only mode enabled. The test used a Redis dataset with 12
million 4KB values. It reclaimed memory until about 85% of Redis
anonymous memory was on node3, used a Redis benchmark configuration with
a 10% hot-key working set, and then issued a short 256MB access burst to
cold keys outside that working set. The cold-key burst mainly touches
memory outside the hot working set, which is expected to remain mostly on
the slow tier.

The upstream default promoted the cold-key burst through the free-space
bypass. The patched default preserved that behavior, while disabling the
bypass kept the same promotions behind the hotness and rate-limit checks:

                              upstream     patched default    bypass disabled
  cold-key burst ops/s          117250         115481             142505
  cold-key burst P99.9 ms        0.854          0.859              0.686
  pgpromote_candidate              0              0               1488328

Compared to upstream default, disabling the bypass improved cold-key burst
throughput by 21.5% and P99.9 latency by 19.7%.

Add a sched debugfs knob:

  /sys/kernel/debug/sched/numa_balancing/promote_free_space_bypass

It defaults to enabled, preserving the existing behavior. When disabled,
enough free space in the target fast-tier node no longer bypasses the
normal hotness and rate-limit controls.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/mmzone.h |  5 ++---
 kernel/sched/debug.c   |  2 ++
 kernel/sched/fair.c    | 14 ++++++++------
 kernel/sched/sched.h   |  1 +
 4 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecd1ebd49756..bf2d604c04b3 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -278,9 +278,8 @@ enum node_stat_item {
 	 */
 	PGPROMOTE_CANDIDATE,
 	/**
-	 * Not rate-limited (NRL) candidate pages for those can be promoted
-	 * without considering hot threshold because of enough free pages in
-	 * fast-tier node.  These promotions bypass the regular hotness checks
+	 * Not rate-limited (NRL) candidate pages selected for promotion through
+	 * the free-space bypass. These candidates bypass the regular hotness checks
 	 * and do NOT influence the promotion rate-limiter or
 	 * threshold-adjustment logic.
 	 * This is for statistics/monitoring purposes.
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e6a3b516c703..a28f0f09c91e 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -736,6 +736,8 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
 	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
+	debugfs_create_bool("promote_free_space_bypass", 0644, numa,
+			    &numa_balancing_promote_free_space_bypass);
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_SCHED_CACHE
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cfc982205b98..46a11f5ddafb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -134,6 +134,8 @@ static unsigned int sysctl_sched_cfs_bandwidth_slice		= 5000UL;
 #ifdef CONFIG_NUMA_BALANCING
 /* Restrict the NUMA promotion throughput (MB/s) for each target node. */
 static unsigned int sysctl_numa_balancing_promote_rate_limit = 65536;
+/* Preserve the existing default of using free fast-tier space eagerly. */
+bool numa_balancing_promote_free_space_bypass = true;
 #endif
 
 #ifdef CONFIG_SYSCTL
@@ -2759,11 +2761,10 @@ static inline bool cpupid_valid(int cpupid)
 }
 
 /*
- * For memory tiering mode, if there are enough free pages (more than
- * enough watermark defined here) in fast memory node, to take full
- * advantage of fast memory capacity, all recently accessed slow
- * memory pages will be migrated to fast memory node without
- * considering hot threshold.
+ * For memory tiering mode, if the target fast memory node has enough free
+ * pages, recently accessed slow memory pages can bypass the normal hotness
+ * check and rate limit so the fast tier can be filled. The bypass can be
+ * disabled by numa_balancing_promote_free_space_bypass.
  */
 static bool pgdat_free_space_enough(struct pglist_data *pgdat)
 {
@@ -2884,7 +2885,8 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 		long nr = folio_nr_pages(folio);
 
 		pgdat = NODE_DATA(dst_nid);
-		if (pgdat_free_space_enough(pgdat)) {
+		if (numa_balancing_promote_free_space_bypass &&
+		    pgdat_free_space_enough(pgdat)) {
 			/* workload changed, reset hot threshold */
 			pgdat->nbp_threshold = 0;
 			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 87c5335add37..662b11bd401e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3218,6 +3218,7 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
 extern unsigned int sysctl_numa_balancing_hot_threshold;
+extern bool numa_balancing_promote_free_space_bypass;
 
 #ifdef CONFIG_SCHED_HRTICK
 
-- 
2.20.1

[PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Li Zhe]

NUMA balancing has separate mode bits for normal task placement and
memory tiering. In memory-tiering-only mode, NUMA hinting faults on
slow-tier pages are used to detect hot pages and promote them to a top
tier. They are not normal task-placement faults.

For slow-tier folios, memory tiering reuses the folio last-cpupid field
to store access-time metadata. numa_migrate_check() therefore cannot
return a real cpupid for those faults. Before this change, it returned the
same all-ones last_cpupid value that normal AutoNUMA uses for a first or
unknown access.

That can make a memory-tiering-only promotion look like an ordinary first
private AutoNUMA fault. After a successful promotion, do_numa_page()
reports the target top-tier node to task_numa_fault(), so the existing
slow-node skip no longer applies. task_numa_fault() then accounts the
fault in p->numa_faults and p->numa_faults_locality, and
task_numa_placement() can consume that sample to update normal task
placement state even though normal NUMA balancing is disabled. In other
words, hot-page promotion feedback can leak into the CPU placement
heuristic.

Mark faults that carried access-time metadata with TNF_ACCESS_TIME. If
normal NUMA balancing is disabled, keep the migrated-page accounting but
skip normal task-placement accounting for those faults.

When normal NUMA balancing and memory tiering are both enabled, promoted
pages can still feed task placement. In that mode, the user has enabled
both CPU placement feedback and tiering promotion, so a promoted top-tier
fault is treated as an ordinary unset-cpupid AutoNUMA fault.

This does not change whether a page is promoted. It only prevents
memory-tiering-only promotion faults from being consumed by the normal
AutoNUMA placement heuristic.

Using a fault flag avoids reserving an in-band value from the cpupid
namespace, where any masked value may also be produced by a real CPU/PID
pair.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/sched/numa_balancing.h |  1 +
 kernel/sched/fair.c                  | 13 +++++++++++++
 mm/memory.c                          |  6 ++++--
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
index 52b22c5c396d..200f50075d5c 100644
--- a/include/linux/sched/numa_balancing.h
+++ b/include/linux/sched/numa_balancing.h
@@ -14,6 +14,7 @@
 #define TNF_SHARED	0x04
 #define TNF_FAULT_LOCAL	0x08
 #define TNF_MIGRATE_FAIL 0x10
+#define TNF_ACCESS_TIME	0x20
 
 enum numa_vmaskip_reason {
 	NUMAB_SKIP_UNSUITABLE,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 46a11f5ddafb..136207b62d87 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4149,6 +4149,19 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	if (!p->mm)
 		return;
 
+	/*
+	 * Slow-tier pages use last_cpupid as access-time metadata instead of
+	 * a real cpupid. Do not feed these faults into normal task placement
+	 * when normal NUMA balancing is disabled.
+	 */
+	if (flags & TNF_ACCESS_TIME) {
+		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL)) {
+			if (migrated)
+				p->numa_pages_migrated += pages;
+			return;
+		}
+	}
+
 	/*
 	 * NUMA faults statistics are unnecessary for the slow memory
 	 * node for memory tiering mode.
diff --git a/mm/memory.c b/mm/memory.c
index 0fb676dd2bd6..00b171e2fe23 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6241,10 +6241,12 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	 * For memory tiering mode, cpupid of slow memory page is used
 	 * to record page access time.  So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_use_access_time(folio)) {
+		*flags |= TNF_ACCESS_TIME;
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
-	else
+	} else {
 		*last_cpupid = folio_last_cpupid(folio);
+	}
 
 	/* Record the current PID accessing VMA */
 	vma_set_access_pid_bit(vma);
-- 
2.20.1

[PATCH v2 3/3] sched/numa: fix scan-period adjustment for tiering-only mode

[Li Zhe]

When automatic NUMA balancing runs in memory-tiering-only mode, NUMA hint
faults identify slow-memory pages that are worth promoting. They are not
normal task-placement feedback.

Without separate tiering feedback, successful tiering promotions are fed
to update_task_scan_period() as private remote faults. That leaves
local == 0 and shared == 0, so the function hits its early slow-down path
and doubles p->numa_scan_period. Repeated promotion feedback can therefore
drive the scan period towards the maximum.

Track memory-tiering feedback separately from normal task-placement stats.
Successful free-space bypass promotions are not counted as tiering scan
feedback because they did not pass the hotness and rate-limit checks. Cold,
hot, and throttled feedback is generated only by the hotness/rate-limit
path, while migration failures slow scanning like the normal placement
heuristic does.

Use the same 70% threshold style as the normal scan-period heuristic. A
hot-dominated window speeds scanning up, a cold-dominated window slows it
down, and a throttled ratio of at least 30% also slows it down. If a
window has no tiering feedback, for example because promotions used the
free-space bypass, double the scan period, following the same
no-useful-signal slow-down direction as normal scan-period adjustment.

Evaluate the tiering window when a scan sequence completes as well as when
tiering faults are reported. This lets an empty window back off the scan
period after qualifying tiering feedback stops.

On a CXL system with node0 as DRAM and node3 as slow-tier memory, a
STREAM Triad-derived OpenMP workload was used to model a
bandwidth-sensitive working-set shift. It first accessed one 100G range,
then switched to another 100G range. The workload was first-touch placed
on node3 and then run from node0 with memory-tiering-only mode enabled.

With the free-space bypass disabled, so that promotions have to pass the
hotness and rate-limit checks, a kernel with only the bypass control was
used as the no-feedback baseline. Tiering scan-period feedback improved
the transition window after the active range switch:

                              no feedback       tiering feedback
  first 80s after switch       122.4 GB/s             129.3 GB/s  (+5.7%)
  200-240s after switch        139.9 GB/s             140.6 GB/s  (+0.5%)

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/sched.h                |  17 ++++
 include/linux/sched/numa_balancing.h |   9 +++
 kernel/sched/fair.c                  | 113 ++++++++++++++++++++++++++-
 3 files changed, 137 insertions(+), 2 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1c93da75226e..c4959747374b 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -833,6 +833,16 @@ struct task_ipi_mask {
 struct task_ipi_mask { };
 #endif
 
+#ifdef CONFIG_NUMA_BALANCING
+enum numa_tiering_fault_stats {
+	NUMA_TIERING_FAULT_COLD,
+	NUMA_TIERING_FAULT_HOT,
+	NUMA_TIERING_FAULT_THROTTLED,
+	NUMA_TIERING_FAULT_MIGRATE_FAIL,
+	NR_NUMA_TIERING_FAULT_STATS
+};
+#endif
+
 struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -1433,6 +1443,13 @@ struct task_struct {
 	 */
 	unsigned long			numa_faults_locality[3];
 
+	/*
+	 * Memory tiering faults are tracked separately from normal NUMA
+	 * placement statistics. In tiering-only mode, slow-memory hint faults
+	 * are used to decide promotion, not task placement.
+	 */
+	unsigned long			numa_tiering_faults[NR_NUMA_TIERING_FAULT_STATS];
+
 	unsigned long			numa_pages_migrated;
 #endif /* CONFIG_NUMA_BALANCING */
 
diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
index 200f50075d5c..12f31ed56d66 100644
--- a/include/linux/sched/numa_balancing.h
+++ b/include/linux/sched/numa_balancing.h
@@ -8,6 +8,7 @@
  */
 
 #include <linux/sched.h>
+#include <linux/sched/sysctl.h>
 
 #define TNF_MIGRATED	0x01
 #define TNF_NO_GROUP	0x02
@@ -16,6 +17,14 @@
 #define TNF_MIGRATE_FAIL 0x10
 #define TNF_ACCESS_TIME	0x20
 
+static inline bool numa_balancing_tiering_only(void)
+{
+	int mode = sysctl_numa_balancing_mode;
+
+	return (mode & NUMA_BALANCING_MEMORY_TIERING) &&
+	       !(mode & NUMA_BALANCING_NORMAL);
+}
+
 enum numa_vmaskip_reason {
 	NUMAB_SKIP_UNSUITABLE,
 	NUMAB_SKIP_SHARED_RO,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 136207b62d87..1d9efc2b7427 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -2760,6 +2760,32 @@ static inline bool cpupid_valid(int cpupid)
 	return cpupid_to_cpu(cpupid) < nr_cpu_ids;
 }
 
+static void task_numa_tiering_placement(struct task_struct *p);
+
+static bool task_numa_tiering_update(struct task_struct *p)
+{
+	if (!numa_balancing_tiering_only() || !p->mm)
+		return false;
+
+	task_numa_tiering_placement(p);
+	return true;
+}
+
+static void task_numa_tiering_no_feedback(struct task_struct *p)
+{
+	task_numa_tiering_update(p);
+}
+
+static void task_numa_tiering_fault(struct task_struct *p,
+				    enum numa_tiering_fault_stats stat,
+				    unsigned long pages)
+{
+	if (!task_numa_tiering_update(p))
+		return;
+
+	p->numa_tiering_faults[stat] += pages;
+}
+
 /*
  * For memory tiering mode, if the target fast memory node has enough free
  * pages, recently accessed slow memory pages can bypass the normal hotness
@@ -2887,6 +2913,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 		pgdat = NODE_DATA(dst_nid);
 		if (numa_balancing_promote_free_space_bypass &&
 		    pgdat_free_space_enough(pgdat)) {
+			task_numa_tiering_no_feedback(p);
 			/* workload changed, reset hot threshold */
 			pgdat->nbp_threshold = 0;
 			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
@@ -2899,10 +2926,18 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 
 		th = pgdat->nbp_threshold ? : def_th;
 		latency = numa_hint_fault_latency(folio);
-		if (latency >= th)
+		if (latency >= th) {
+			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_COLD, nr);
+			return false;
+		}
+
+		if (numa_promotion_rate_limit(pgdat, rate_limit, nr)) {
+			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_THROTTLED, nr);
 			return false;
+		}
 
-		return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
+		task_numa_tiering_fault(p, NUMA_TIERING_FAULT_HOT, nr);
+		return true;
 	}
 
 	this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
@@ -3708,6 +3743,48 @@ static void update_task_scan_period(struct task_struct *p,
 	       sizeof(p->numa_faults_locality));
 }
 
+/*
+ * Memory tiering uses NUMA hint faults to identify recently accessed slow
+ * memory that should be promoted. In tiering-only mode, those faults are not
+ * normal task-placement feedback, so use tiering-specific outcomes to adjust
+ * the scan period.
+ */
+static void update_task_tiering_scan_period(struct task_struct *p)
+{
+	unsigned long cold, hot, throttled, failed;
+	unsigned long total;
+	s64 period, period_slot;
+	s64 diff = 0;
+
+	cold = p->numa_tiering_faults[NUMA_TIERING_FAULT_COLD];
+	hot = p->numa_tiering_faults[NUMA_TIERING_FAULT_HOT];
+	throttled = p->numa_tiering_faults[NUMA_TIERING_FAULT_THROTTLED];
+	failed = p->numa_tiering_faults[NUMA_TIERING_FAULT_MIGRATE_FAIL];
+	total = cold + hot + throttled + failed;
+
+	period = p->numa_scan_period;
+	period_slot = DIV_ROUND_UP_ULL(period, NUMA_PERIOD_SLOTS);
+
+	if (!total) {
+		diff = period;
+	} else if (failed) {
+		diff = period_slot;
+	} else if (throttled * NUMA_PERIOD_SLOTS >=
+		   total * (NUMA_PERIOD_SLOTS - NUMA_PERIOD_THRESHOLD)) {
+		diff = period_slot;
+	} else if (cold * NUMA_PERIOD_SLOTS >=
+		   total * NUMA_PERIOD_THRESHOLD) {
+		diff = period_slot;
+	} else if (hot * NUMA_PERIOD_SLOTS >=
+		   total * NUMA_PERIOD_THRESHOLD) {
+		diff = -period_slot;
+	}
+
+	p->numa_scan_period = clamp_t(s64, period + diff,
+				      task_scan_min(p), task_scan_max(p));
+	memset(p->numa_tiering_faults, 0, sizeof(p->numa_tiering_faults));
+}
+
 /*
  * Get the fraction of time the task has been running since the last
  * NUMA placement cycle. The scheduler keeps similar statistics, but
@@ -3974,6 +4051,22 @@ static void task_numa_placement(struct task_struct *p)
 	update_task_scan_period(p, fault_types[0], fault_types[1]);
 }
 
+static void task_numa_tiering_placement(struct task_struct *p)
+{
+	int seq;
+
+	if (!p->mm)
+		return;
+
+	seq = READ_ONCE(p->mm->numa_scan_seq);
+	if (p->numa_scan_seq == seq)
+		return;
+	p->numa_scan_seq = seq;
+	p->numa_scan_period_max = task_scan_max(p);
+
+	update_task_tiering_scan_period(p);
+}
+
 static inline int get_numa_group(struct numa_group *grp)
 {
 	return refcount_inc_not_zero(&grp->refcount);
@@ -4105,6 +4198,8 @@ void task_numa_free(struct task_struct *p, bool final)
 	unsigned long flags;
 	int i;
 
+	memset(p->numa_tiering_faults, 0, sizeof(p->numa_tiering_faults));
+
 	if (!numa_faults)
 		return;
 
@@ -4158,6 +4253,11 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL)) {
 			if (migrated)
 				p->numa_pages_migrated += pages;
+			if (numa_balancing_tiering_only() &&
+			    (flags & TNF_MIGRATE_FAIL))
+				task_numa_tiering_fault(p,
+						NUMA_TIERING_FAULT_MIGRATE_FAIL,
+						pages);
 			return;
 		}
 	}
@@ -4529,6 +4629,14 @@ static void task_numa_work(struct callback_head *work)
 		reset_ptenuma_scan(p);
 	mmap_read_unlock(mm);
 
+	/*
+	 * Tiering-only placement feedback is still scan-window based. Consume
+	 * the window when a scan sequence completes, so an empty tiering
+	 * window can back off after qualifying tiering faults stop.
+	 */
+	if (numa_balancing_tiering_only())
+		task_numa_tiering_placement(p);
+
 	/*
 	 * Make sure tasks use at least 32x as much time to run other code
 	 * than they used here, to limit NUMA PTE scanning overhead to 3% max.
@@ -4562,6 +4670,7 @@ void init_numa_balancing(u64 clone_flags, struct task_struct *p)
 	p->numa_faults			= NULL;
 	p->numa_pages_migrated		= 0;
 	p->total_numa_faults		= 0;
+	memset(p->numa_tiering_faults, 0, sizeof(p->numa_tiering_faults));
 	RCU_INIT_POINTER(p->numa_group, NULL);
 	p->last_task_numa_placement	= 0;
 	p->last_sum_exec_runtime	= 0;
-- 
2.20.1

Re: [PATCH v2 0/3] sched/numa: refine memory tiering promotion heuristics

[Peter Zijlstra]

On Wed, Oct 28, 2026 at 12:12:50PM +0800, Li Zhe wrote:
> This version narrows the series to memory tiering.  The generic normal
> NUMA balancing cleanups and heuristic changes from v1 are not included.

Should we not look at Bharata's pghot stuff first and untangle this numa
balancing vs tiering nonsense, before making an even bigger mess of
things?

Re: [PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Joshua Hahn]

On Wed, Oct 28, 2026 at 12:14 AM Li Zhe <lizhe.67@bytedance.com> wrote:
>
> NUMA balancing has separate mode bits for normal task placement and
> memory tiering. In memory-tiering-only mode, NUMA hinting faults on
> slow-tier pages are used to detect hot pages and promote them to a top
> tier. They are not normal task-placement faults.
>
> For slow-tier folios, memory tiering reuses the folio last-cpupid field
> to store access-time metadata. numa_migrate_check() therefore cannot
> return a real cpupid for those faults. Before this change, it returned the
> same all-ones last_cpupid value that normal AutoNUMA uses for a first or
> unknown access.
>
> That can make a memory-tiering-only promotion look like an ordinary first
> private AutoNUMA fault. After a successful promotion, do_numa_page()
> reports the target top-tier node to task_numa_fault(), so the existing
> slow-node skip no longer applies. task_numa_fault() then accounts the
> fault in p->numa_faults and p->numa_faults_locality, and
> task_numa_placement() can consume that sample to update normal task
> placement state even though normal NUMA balancing is disabled. In other
> words, hot-page promotion feedback can leak into the CPU placement
> heuristic.

Hi Zhe,

Thanks for the revision. To me, this looks more like a fix than an
optimization, since there really is incorrect NUMA balancing behavior
happening that could be visible in userspace (unintended
task-placement and related accounting). On the other hand patch 3/3
doesn't seem like a fix to me despite the name, since this one would
mitigate the incorrect behavior.

How about we pull this one out of the series and submit it
independently with an appropriate Fixes: tag and cc:stable, and
discuss 1 and 3 later on? Unfortunately this is also not a great time
to submit new features or behaviors as we are in the middle of a large
merge window, and there are a bunch of fixes that maintainers might be
busy looking into : -(

Regardless, thank you for the fix, Zhe! I don't have a strong
preference on the ordering of this against pghot, but if other folks
are concerned that this could create more churn, I think it would be
reasonable to wait for patches 1 and 3 after we discuss with Bharata.

I hope you have a great day : -)
Joshua

Re: [PATCH v2 0/3] sched/numa: refine memory tiering promotion heuristics

[Li Zhe]

On 10/28/26 5:20 PM, Peter Zijlstra wrote:
> On Wed, Oct 28, 2026 at 12:12:50PM +0800, Li Zhe wrote:
>> This version narrows the series to memory tiering.  The generic normal
>> NUMA balancing cleanups and heuristic changes from v1 are not included.
> Should we not look at Bharata's pghot stuff first and untangle this numa
> balancing vs tiering nonsense, before making an even bigger mess of
> things?
I agree that pghot is relevant to the longer-term direction.  My
understanding is that the two ideas are not conflicting with each other:
this series tries to refine the current NUMA-balancing-based tiering
implementation, while pghot would be a larger rework of the promotion
infrastructure.

Let's also copy Bharata here so he is aware of this discussion.

Based on Joshua's comment, I can also split patch 2 out as a standalone
fix with a proper Fixes tag, and keep patches 1 and 3 for the broader
tiering heuristic discussion.

Thanks,
Zhe

Re: [PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Li Zhe]

On 10/28/26 11:03 PM, Joshua Hahn wrote:
> On Wed, Oct 28, 2026 at 12:14 AM Li Zhe <lizhe.67@bytedance.com> wrote:
>> NUMA balancing has separate mode bits for normal task placement and
>> memory tiering. In memory-tiering-only mode, NUMA hinting faults on
>> slow-tier pages are used to detect hot pages and promote them to a top
>> tier. They are not normal task-placement faults.
>>
>> For slow-tier folios, memory tiering reuses the folio last-cpupid field
>> to store access-time metadata. numa_migrate_check() therefore cannot
>> return a real cpupid for those faults. Before this change, it returned the
>> same all-ones last_cpupid value that normal AutoNUMA uses for a first or
>> unknown access.
>>
>> That can make a memory-tiering-only promotion look like an ordinary first
>> private AutoNUMA fault. After a successful promotion, do_numa_page()
>> reports the target top-tier node to task_numa_fault(), so the existing
>> slow-node skip no longer applies. task_numa_fault() then accounts the
>> fault in p->numa_faults and p->numa_faults_locality, and
>> task_numa_placement() can consume that sample to update normal task
>> placement state even though normal NUMA balancing is disabled. In other
>> words, hot-page promotion feedback can leak into the CPU placement
>> heuristic.
> Hi Zhe,
>
> Thanks for the revision. To me, this looks more like a fix than an
> optimization, since there really is incorrect NUMA balancing behavior
> happening that could be visible in userspace (unintended
> task-placement and related accounting). On the other hand patch 3/3
> doesn't seem like a fix to me despite the name, since this one would
> mitigate the incorrect behavior.
>
> How about we pull this one out of the series and submit it
> independently with an appropriate Fixes: tag and cc:stable, and
> discuss 1 and 3 later on? Unfortunately this is also not a great time
> to submit new features or behaviors as we are in the middle of a large
> merge window, and there are a bunch of fixes that maintainers might be
> busy looking into : -(
>
> Regardless, thank you for the fix, Zhe! I don't have a strong
> preference on the ordering of this against pghot, but if other folks
> are concerned that this could create more churn, I think it would be
> reasonable to wait for patches 1 and 3 after we discuss with Bharata.
Thanks for the review.

Agreed, patch 2 is better framed as a fix.  I will check the exact
commit that introduced this path and can resend it separately with a
proper Fixes tag and Cc stable after rc1 is released.

For patches 1 and 3, I agree they are more about the tiering heuristic
and policy behavior.  I will keep them separate from the fix discussion
and continue that part together with the pghot-related discussion.

Thanks,
Zhe
>
> I hope you have a great day : -)
> Joshua

Re: [PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Bharata B Rao]

On 28-Oct-26 9:42 AM, Li Zhe wrote:
> Memory tiering normally promotes any recently accessed slow-tier page if
> the target fast-tier node has enough free space. This preserves the
> existing policy of filling available fast memory before relying on
> hotness and rate-limit control.
> 
> That is not always the desired policy. Some deployments may intentionally
> keep part of a workload's memory on CXL-attached memory, for example to
> preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
> the free-space bypass, short cold accesses and bursts of hot accesses can
> refill the fast tier and pull the workload away from that distribution
> whenever DRAM has free space.

Currently promotion happens on 1st access. Won't it be more natural to tweak
this frequency of access and also the time between successive accesses to
prevent promotion due to short cold accesses and bursts of hot accesses
rather than free space bypass configuration?

Regards,
Bharata.

Re: [PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Bharata B Rao]

On 28-Oct-26 8:33 PM, Joshua Hahn wrote:
> On Wed, Oct 28, 2026 at 12:14 AM Li Zhe <lizhe.67@bytedance.com> wrote:
>>
>> NUMA balancing has separate mode bits for normal task placement and
>> memory tiering. In memory-tiering-only mode, NUMA hinting faults on
>> slow-tier pages are used to detect hot pages and promote them to a top
>> tier. They are not normal task-placement faults.
>>
>> For slow-tier folios, memory tiering reuses the folio last-cpupid field
>> to store access-time metadata. numa_migrate_check() therefore cannot
>> return a real cpupid for those faults. Before this change, it returned the
>> same all-ones last_cpupid value that normal AutoNUMA uses for a first or
>> unknown access.
>>
>> That can make a memory-tiering-only promotion look like an ordinary first
>> private AutoNUMA fault. After a successful promotion, do_numa_page()
>> reports the target top-tier node to task_numa_fault(), so the existing
>> slow-node skip no longer applies. task_numa_fault() then accounts the
>> fault in p->numa_faults and p->numa_faults_locality, and
>> task_numa_placement() can consume that sample to update normal task
>> placement state even though normal NUMA balancing is disabled. In other
>> words, hot-page promotion feedback can leak into the CPU placement
>> heuristic.
> 
> Hi Zhe,
> 
> Thanks for the revision. To me, this looks more like a fix than an
> optimization, since there really is incorrect NUMA balancing behavior
> happening that could be visible in userspace (unintended
> task-placement and related accounting). On the other hand patch 3/3
> doesn't seem like a fix to me despite the name, since this one would
> mitigate the incorrect behavior.
> 
> How about we pull this one out of the series and submit it
> independently with an appropriate Fixes: tag and cc:stable, and
> discuss 1 and 3 later on?

Currently only successful promotions feed stats to scan period update.
This change stops that and nothing updates the scan period for tiering
mode after this change.

Though the direction of the fix is right, I think this can't be an
independent fix until there is some alternative which patch 3 in this
series brings in.

Also it would be good to get a view from Ying Huang as to why only
successful promotions drove the fault stats while unsuccessful promotions
didn't in the tiering mode.

Regards,
Bharata.

Re: [PATCH v2 3/3] sched/numa: fix scan-period adjustment for tiering-only mode

[Bharata B Rao]

On 28-Oct-26 9:42 AM, Li Zhe wrote:
> When automatic NUMA balancing runs in memory-tiering-only mode, NUMA hint
> faults identify slow-memory pages that are worth promoting. They are not
> normal task-placement feedback.
> 
> Without separate tiering feedback, successful tiering promotions are fed
> to update_task_scan_period() as private remote faults. That leaves
> local == 0 and shared == 0, so the function hits its early slow-down path
> and doubles p->numa_scan_period. Repeated promotion feedback can therefore
> drive the scan period towards the maximum.
> 
> Track memory-tiering feedback separately from normal task-placement stats.
> Successful free-space bypass promotions are not counted as tiering scan
> feedback because they did not pass the hotness and rate-limit checks. Cold,
> hot, and throttled feedback is generated only by the hotness/rate-limit
> path, while migration failures slow scanning like the normal placement
> heuristic does.
> 
> Use the same 70% threshold style as the normal scan-period heuristic. A
> hot-dominated window speeds scanning up, a cold-dominated window slows it
> down, and a throttled ratio of at least 30% also slows it down. If a
> window has no tiering feedback, for example because promotions used the
> free-space bypass, double the scan period, following the same
> no-useful-signal slow-down direction as normal scan-period adjustment.
> 
> Evaluate the tiering window when a scan sequence completes as well as when
> tiering faults are reported. This lets an empty window back off the scan
> period after qualifying tiering feedback stops.
> 
> On a CXL system with node0 as DRAM and node3 as slow-tier memory, a
> STREAM Triad-derived OpenMP workload was used to model a
> bandwidth-sensitive working-set shift. It first accessed one 100G range,
> then switched to another 100G range. The workload was first-touch placed
> on node3 and then run from node0 with memory-tiering-only mode enabled.
> 
> With the free-space bypass disabled, so that promotions have to pass the
> hotness and rate-limit checks, a kernel with only the bypass control was
> used as the no-feedback baseline. Tiering scan-period feedback improved
> the transition window after the active range switch:
> 
>                               no feedback       tiering feedback
>   first 80s after switch       122.4 GB/s             129.3 GB/s  (+5.7%)
>   200-240s after switch        139.9 GB/s             140.6 GB/s  (+0.5%)
> 
> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
> ---
>  include/linux/sched.h                |  17 ++++
>  include/linux/sched/numa_balancing.h |   9 +++
>  kernel/sched/fair.c                  | 113 ++++++++++++++++++++++++++-
>  3 files changed, 137 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 1c93da75226e..c4959747374b 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -833,6 +833,16 @@ struct task_ipi_mask {
>  struct task_ipi_mask { };
>  #endif
>  
> +#ifdef CONFIG_NUMA_BALANCING
> +enum numa_tiering_fault_stats {
> +	NUMA_TIERING_FAULT_COLD,
> +	NUMA_TIERING_FAULT_HOT,
> +	NUMA_TIERING_FAULT_THROTTLED,
> +	NUMA_TIERING_FAULT_MIGRATE_FAIL,
> +	NR_NUMA_TIERING_FAULT_STATS
> +};
> +#endif
> +
>  struct task_struct {
>  #ifdef CONFIG_THREAD_INFO_IN_TASK
>  	/*
> @@ -1433,6 +1443,13 @@ struct task_struct {
>  	 */
>  	unsigned long			numa_faults_locality[3];
>  
> +	/*
> +	 * Memory tiering faults are tracked separately from normal NUMA
> +	 * placement statistics. In tiering-only mode, slow-memory hint faults
> +	 * are used to decide promotion, not task placement.
> +	 */
> +	unsigned long			numa_tiering_faults[NR_NUMA_TIERING_FAULT_STATS];
> +
>  	unsigned long			numa_pages_migrated;
>  #endif /* CONFIG_NUMA_BALANCING */
>  
> diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
> index 200f50075d5c..12f31ed56d66 100644
> --- a/include/linux/sched/numa_balancing.h
> +++ b/include/linux/sched/numa_balancing.h
> @@ -8,6 +8,7 @@
>   */
>  
>  #include <linux/sched.h>
> +#include <linux/sched/sysctl.h>
>  
>  #define TNF_MIGRATED	0x01
>  #define TNF_NO_GROUP	0x02
> @@ -16,6 +17,14 @@
>  #define TNF_MIGRATE_FAIL 0x10
>  #define TNF_ACCESS_TIME	0x20
>  
> +static inline bool numa_balancing_tiering_only(void)
> +{
> +	int mode = sysctl_numa_balancing_mode;
> +
> +	return (mode & NUMA_BALANCING_MEMORY_TIERING) &&
> +	       !(mode & NUMA_BALANCING_NORMAL);
> +}
> +
>  enum numa_vmaskip_reason {
>  	NUMAB_SKIP_UNSUITABLE,
>  	NUMAB_SKIP_SHARED_RO,
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 136207b62d87..1d9efc2b7427 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -2760,6 +2760,32 @@ static inline bool cpupid_valid(int cpupid)
>  	return cpupid_to_cpu(cpupid) < nr_cpu_ids;
>  }
>  
> +static void task_numa_tiering_placement(struct task_struct *p);
> +
> +static bool task_numa_tiering_update(struct task_struct *p)
> +{
> +	if (!numa_balancing_tiering_only() || !p->mm)
> +		return false;
> +
> +	task_numa_tiering_placement(p);
> +	return true;
> +}
> +
> +static void task_numa_tiering_no_feedback(struct task_struct *p)
> +{
> +	task_numa_tiering_update(p);
> +}
> +
> +static void task_numa_tiering_fault(struct task_struct *p,
> +				    enum numa_tiering_fault_stats stat,
> +				    unsigned long pages)
> +{
> +	if (!task_numa_tiering_update(p))
> +		return;
> +
> +	p->numa_tiering_faults[stat] += pages;
> +}
> +
>  /*
>   * For memory tiering mode, if the target fast memory node has enough free
>   * pages, recently accessed slow memory pages can bypass the normal hotness
> @@ -2887,6 +2913,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
>  		pgdat = NODE_DATA(dst_nid);
>  		if (numa_balancing_promote_free_space_bypass &&
>  		    pgdat_free_space_enough(pgdat)) {
> +			task_numa_tiering_no_feedback(p);
>  			/* workload changed, reset hot threshold */
>  			pgdat->nbp_threshold = 0;
>  			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
> @@ -2899,10 +2926,18 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
>  
>  		th = pgdat->nbp_threshold ? : def_th;
>  		latency = numa_hint_fault_latency(folio);
> -		if (latency >= th)
> +		if (latency >= th) {
> +			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_COLD, nr);
> +			return false;
> +		}
> +
> +		if (numa_promotion_rate_limit(pgdat, rate_limit, nr)) {
> +			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_THROTTLED, nr);
>  			return false;
> +		}
>  
> -		return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
> +		task_numa_tiering_fault(p, NUMA_TIERING_FAULT_HOT, nr);
> +		return true;
>  	}

I like the separation of scan-period updation of tiering mode from the regular
mode as the existing update scheme was dependent on private/shared faults which
made little sense in tiering mode.

However I am not sure if you should drive the scan period updates from
should_numa_migrate_memory(). The scan period belongs to the hint fault
scanner. I am pulling the promotion heuristic out of
should_numa_migrate_memory() in the pghot series. The hint fault path
only reports an access and shared engine does the classification and
migration for every source. The other sources may not have numa_scan_period,
so this feedback has to stay with the hint fault path.

pghot supporting multiple sources may not see the light of the day in
near future but it is still good to keep the source specific
aspects and promotion engine separate.

Regards,
Bharata.

Re: [PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Li Zhe]

On 10/29/26 12:16 PM, Bharata B Rao wrote:
> On 28-Oct-26 9:42 AM, Li Zhe wrote:
>> Memory tiering normally promotes any recently accessed slow-tier page if
>> the target fast-tier node has enough free space. This preserves the
>> existing policy of filling available fast memory before relying on
>> hotness and rate-limit control.
>>
>> That is not always the desired policy. Some deployments may intentionally
>> keep part of a workload's memory on CXL-attached memory, for example to
>> preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
>> the free-space bypass, short cold accesses and bursts of hot accesses can
>> refill the fast tier and pull the workload away from that distribution
>> whenever DRAM has free space.
> Currently promotion happens on 1st access. Won't it be more natural to tweak
> this frequency of access and also the time between successive accesses to
> prevent promotion due to short cold accesses and bursts of hot accesses
> rather than free space bypass configuration?
I agree that using hotness and rate-limit controls is the more natural
policy here.  That is exactly what this knob is trying to make possible.

The current free-space bypass promotes before reaching those existing
checks.  So when the target fast-tier node has enough free space, short
accesses can still be promoted even if they would not pass the
hotness/rate-limit path.

With this knob disabled, the free-space shortcut is skipped and those
faults fall back to the existing hotness and rate-limit controls instead.
So the patch does not add a new hotness policy; it only makes the
existing one effective even when the fast tier has free space.

Thanks,
Zhe

>
> Regards,
> Bharata.

Re: [PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Li Zhe]

On 10/29/26 1:51 PM, Bharata B Rao wrote:
> On 28-Oct-26 8:33 PM, Joshua Hahn wrote:
>> On Wed, Oct 28, 2026 at 12:14 AM Li Zhe <lizhe.67@bytedance.com> wrote:
>>> NUMA balancing has separate mode bits for normal task placement and
>>> memory tiering. In memory-tiering-only mode, NUMA hinting faults on
>>> slow-tier pages are used to detect hot pages and promote them to a top
>>> tier. They are not normal task-placement faults.
>>>
>>> For slow-tier folios, memory tiering reuses the folio last-cpupid field
>>> to store access-time metadata. numa_migrate_check() therefore cannot
>>> return a real cpupid for those faults. Before this change, it returned the
>>> same all-ones last_cpupid value that normal AutoNUMA uses for a first or
>>> unknown access.
>>>
>>> That can make a memory-tiering-only promotion look like an ordinary first
>>> private AutoNUMA fault. After a successful promotion, do_numa_page()
>>> reports the target top-tier node to task_numa_fault(), so the existing
>>> slow-node skip no longer applies. task_numa_fault() then accounts the
>>> fault in p->numa_faults and p->numa_faults_locality, and
>>> task_numa_placement() can consume that sample to update normal task
>>> placement state even though normal NUMA balancing is disabled. In other
>>> words, hot-page promotion feedback can leak into the CPU placement
>>> heuristic.
>> Hi Zhe,
>>
>> Thanks for the revision. To me, this looks more like a fix than an
>> optimization, since there really is incorrect NUMA balancing behavior
>> happening that could be visible in userspace (unintended
>> task-placement and related accounting). On the other hand patch 3/3
>> doesn't seem like a fix to me despite the name, since this one would
>> mitigate the incorrect behavior.
>>
>> How about we pull this one out of the series and submit it
>> independently with an appropriate Fixes: tag and cc:stable, and
>> discuss 1 and 3 later on?
> Currently only successful promotions feed stats to scan period update.
> This change stops that and nothing updates the scan period for tiering
> mode after this change.
>
> Though the direction of the fix is right, I think this can't be an
> independent fix until there is some alternative which patch 3 in this
> series brings in.
Agreed, thanks for pointing this out.  Patch 2 should not be sent as a
standalone fix without the replacement scan-period feedback path.

Thanks,
Zhe

>
> Also it would be good to get a view from Ying Huang as to why only
> successful promotions drove the fault stats while unsuccessful promotions
> didn't in the tiering mode.
>
> Regards,
> Bharata.

Re: [PATCH v2 3/3] sched/numa: fix scan-period adjustment for tiering-only mode

[Li Zhe]

On 10/29/26 2:21 PM, Bharata B Rao wrote:
> On 28-Oct-26 9:42 AM, Li Zhe wrote:
>> When automatic NUMA balancing runs in memory-tiering-only mode, NUMA hint
>> faults identify slow-memory pages that are worth promoting. They are not
>> normal task-placement feedback.
>>
>> Without separate tiering feedback, successful tiering promotions are fed
>> to update_task_scan_period() as private remote faults. That leaves
>> local == 0 and shared == 0, so the function hits its early slow-down path
>> and doubles p->numa_scan_period. Repeated promotion feedback can therefore
>> drive the scan period towards the maximum.
>>
>> Track memory-tiering feedback separately from normal task-placement stats.
>> Successful free-space bypass promotions are not counted as tiering scan
>> feedback because they did not pass the hotness and rate-limit checks. Cold,
>> hot, and throttled feedback is generated only by the hotness/rate-limit
>> path, while migration failures slow scanning like the normal placement
>> heuristic does.
>>
>> Use the same 70% threshold style as the normal scan-period heuristic. A
>> hot-dominated window speeds scanning up, a cold-dominated window slows it
>> down, and a throttled ratio of at least 30% also slows it down. If a
>> window has no tiering feedback, for example because promotions used the
>> free-space bypass, double the scan period, following the same
>> no-useful-signal slow-down direction as normal scan-period adjustment.
>>
>> Evaluate the tiering window when a scan sequence completes as well as when
>> tiering faults are reported. This lets an empty window back off the scan
>> period after qualifying tiering feedback stops.
>>
>> On a CXL system with node0 as DRAM and node3 as slow-tier memory, a
>> STREAM Triad-derived OpenMP workload was used to model a
>> bandwidth-sensitive working-set shift. It first accessed one 100G range,
>> then switched to another 100G range. The workload was first-touch placed
>> on node3 and then run from node0 with memory-tiering-only mode enabled.
>>
>> With the free-space bypass disabled, so that promotions have to pass the
>> hotness and rate-limit checks, a kernel with only the bypass control was
>> used as the no-feedback baseline. Tiering scan-period feedback improved
>> the transition window after the active range switch:
>>
>>                                no feedback       tiering feedback
>>    first 80s after switch       122.4 GB/s             129.3 GB/s  (+5.7%)
>>    200-240s after switch        139.9 GB/s             140.6 GB/s  (+0.5%)
>>
>> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
>> ---
>>   include/linux/sched.h                |  17 ++++
>>   include/linux/sched/numa_balancing.h |   9 +++
>>   kernel/sched/fair.c                  | 113 ++++++++++++++++++++++++++-
>>   3 files changed, 137 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>> index 1c93da75226e..c4959747374b 100644
>> --- a/include/linux/sched.h
>> +++ b/include/linux/sched.h
>> @@ -833,6 +833,16 @@ struct task_ipi_mask {
>>   struct task_ipi_mask { };
>>   #endif
>>   
>> +#ifdef CONFIG_NUMA_BALANCING
>> +enum numa_tiering_fault_stats {
>> +	NUMA_TIERING_FAULT_COLD,
>> +	NUMA_TIERING_FAULT_HOT,
>> +	NUMA_TIERING_FAULT_THROTTLED,
>> +	NUMA_TIERING_FAULT_MIGRATE_FAIL,
>> +	NR_NUMA_TIERING_FAULT_STATS
>> +};
>> +#endif
>> +
>>   struct task_struct {
>>   #ifdef CONFIG_THREAD_INFO_IN_TASK
>>   	/*
>> @@ -1433,6 +1443,13 @@ struct task_struct {
>>   	 */
>>   	unsigned long			numa_faults_locality[3];
>>   
>> +	/*
>> +	 * Memory tiering faults are tracked separately from normal NUMA
>> +	 * placement statistics. In tiering-only mode, slow-memory hint faults
>> +	 * are used to decide promotion, not task placement.
>> +	 */
>> +	unsigned long			numa_tiering_faults[NR_NUMA_TIERING_FAULT_STATS];
>> +
>>   	unsigned long			numa_pages_migrated;
>>   #endif /* CONFIG_NUMA_BALANCING */
>>   
>> diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
>> index 200f50075d5c..12f31ed56d66 100644
>> --- a/include/linux/sched/numa_balancing.h
>> +++ b/include/linux/sched/numa_balancing.h
>> @@ -8,6 +8,7 @@
>>    */
>>   
>>   #include <linux/sched.h>
>> +#include <linux/sched/sysctl.h>
>>   
>>   #define TNF_MIGRATED	0x01
>>   #define TNF_NO_GROUP	0x02
>> @@ -16,6 +17,14 @@
>>   #define TNF_MIGRATE_FAIL 0x10
>>   #define TNF_ACCESS_TIME	0x20
>>   
>> +static inline bool numa_balancing_tiering_only(void)
>> +{
>> +	int mode = sysctl_numa_balancing_mode;
>> +
>> +	return (mode & NUMA_BALANCING_MEMORY_TIERING) &&
>> +	       !(mode & NUMA_BALANCING_NORMAL);
>> +}
>> +
>>   enum numa_vmaskip_reason {
>>   	NUMAB_SKIP_UNSUITABLE,
>>   	NUMAB_SKIP_SHARED_RO,
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index 136207b62d87..1d9efc2b7427 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -2760,6 +2760,32 @@ static inline bool cpupid_valid(int cpupid)
>>   	return cpupid_to_cpu(cpupid) < nr_cpu_ids;
>>   }
>>   
>> +static void task_numa_tiering_placement(struct task_struct *p);
>> +
>> +static bool task_numa_tiering_update(struct task_struct *p)
>> +{
>> +	if (!numa_balancing_tiering_only() || !p->mm)
>> +		return false;
>> +
>> +	task_numa_tiering_placement(p);
>> +	return true;
>> +}
>> +
>> +static void task_numa_tiering_no_feedback(struct task_struct *p)
>> +{
>> +	task_numa_tiering_update(p);
>> +}
>> +
>> +static void task_numa_tiering_fault(struct task_struct *p,
>> +				    enum numa_tiering_fault_stats stat,
>> +				    unsigned long pages)
>> +{
>> +	if (!task_numa_tiering_update(p))
>> +		return;
>> +
>> +	p->numa_tiering_faults[stat] += pages;
>> +}
>> +
>>   /*
>>    * For memory tiering mode, if the target fast memory node has enough free
>>    * pages, recently accessed slow memory pages can bypass the normal hotness
>> @@ -2887,6 +2913,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
>>   		pgdat = NODE_DATA(dst_nid);
>>   		if (numa_balancing_promote_free_space_bypass &&
>>   		    pgdat_free_space_enough(pgdat)) {
>> +			task_numa_tiering_no_feedback(p);
>>   			/* workload changed, reset hot threshold */
>>   			pgdat->nbp_threshold = 0;
>>   			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
>> @@ -2899,10 +2926,18 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
>>   
>>   		th = pgdat->nbp_threshold ? : def_th;
>>   		latency = numa_hint_fault_latency(folio);
>> -		if (latency >= th)
>> +		if (latency >= th) {
>> +			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_COLD, nr);
>> +			return false;
>> +		}
>> +
>> +		if (numa_promotion_rate_limit(pgdat, rate_limit, nr)) {
>> +			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_THROTTLED, nr);
>>   			return false;
>> +		}
>>   
>> -		return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
>> +		task_numa_tiering_fault(p, NUMA_TIERING_FAULT_HOT, nr);
>> +		return true;
>>   	}
> I like the separation of scan-period updation of tiering mode from the regular
> mode as the existing update scheme was dependent on private/shared faults which
> made little sense in tiering mode.
>
> However I am not sure if you should drive the scan period updates from
> should_numa_migrate_memory(). The scan period belongs to the hint fault
> scanner. I am pulling the promotion heuristic out of
> should_numa_migrate_memory() in the pghot series. The hint fault path
> only reports an access and shared engine does the classification and
> migration for every source. The other sources may not have numa_scan_period,
> so this feedback has to stay with the hint fault path.
>
> pghot supporting multiple sources may not see the light of the day in
> near future but it is still good to keep the source specific
> aspects and promotion engine separate.


Agreed, that layering makes sense.

I am wondering whether that separation is better handled together with
the larger promotion-engine rework, though.  Without the shared engine in
place, moving the outcome reporting out of should_numa_migrate_memory()
would add interface changes mainly for a future direction.

For this series, my goal was narrower: keep tiering-only scan-period
feedback separate from normal task-placement stats while staying close
to the current numabalance implementation.

Thanks,
Zhe

>
> Regards,
> Bharata.

Re: [PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Bharata B Rao]

On 28-Oct-26 9:42 AM, Li Zhe wrote:
> Memory tiering normally promotes any recently accessed slow-tier page if
> the target fast-tier node has enough free space. This preserves the
> existing policy of filling available fast memory before relying on
> hotness and rate-limit control.
> 
> That is not always the desired policy. Some deployments may intentionally
> keep part of a workload's memory on CXL-attached memory, for example to
> preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
> the free-space bypass, short cold accesses and bursts of hot accesses can
> refill the fast tier and pull the workload away from that distribution
> whenever DRAM has free space.
> 
> Redis testing exercises this policy issue. On a CXL system with node0 as
> DRAM and node3 as slow-tier memory, a Redis test was run with
> memory-tiering-only mode enabled. The test used a Redis dataset with 12
> million 4KB values. It reclaimed memory until about 85% of Redis
> anonymous memory was on node3, used a Redis benchmark configuration with
> a 10% hot-key working set, and then issued a short 256MB access burst to
> cold keys outside that working set. The cold-key burst mainly touches
> memory outside the hot working set, which is expected to remain mostly on
> the slow tier.

Can you please clarify if 85% of redis memory ended up on lower tier naturally
due to overflow from toptier? I ask this question because, if it is a natural
overflow/demotion, then it will be hard to find head room in the toptier to
allow promotions that bypass hotness or rate limit checks."

Regards,
Bharata.

Re: [PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Li Zhe]

On 10/29/26 5:34 PM, Bharata B Rao wrote:
> On 28-Oct-26 9:42 AM, Li Zhe wrote:
>> Memory tiering normally promotes any recently accessed slow-tier page if
>> the target fast-tier node has enough free space. This preserves the
>> existing policy of filling available fast memory before relying on
>> hotness and rate-limit control.
>>
>> That is not always the desired policy. Some deployments may intentionally
>> keep part of a workload's memory on CXL-attached memory, for example to
>> preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
>> the free-space bypass, short cold accesses and bursts of hot accesses can
>> refill the fast tier and pull the workload away from that distribution
>> whenever DRAM has free space.
>>
>> Redis testing exercises this policy issue. On a CXL system with node0 as
>> DRAM and node3 as slow-tier memory, a Redis test was run with
>> memory-tiering-only mode enabled. The test used a Redis dataset with 12
>> million 4KB values. It reclaimed memory until about 85% of Redis
>> anonymous memory was on node3, used a Redis benchmark configuration with
>> a 10% hot-key working set, and then issued a short 256MB access burst to
>> cold keys outside that working set. The cold-key burst mainly touches
>> memory outside the hot working set, which is expected to remain mostly on
>> the slow tier.
> Can you please clarify if 85% of redis memory ended up on lower tier naturally
> due to overflow from toptier? I ask this question because, if it is a natural
> overflow/demotion, then it will be hard to find head room in the toptier to
> allow promotions that bypass hotness or rate limit checks."


It was not a natural overflow from a full top tier.

The test intentionally reclaimed/demoted Redis memory to the lower tier
to model a deployment that keeps only a limited fraction of the workload
in DRAM and leaves the rest on CXL memory.  In that setup the top tier can
still have enough free space, so the current free-space bypass can promote
short cold accesses back to DRAM.

Those pages are not hot enough to be promoted by the normal
hotness/rate-limit path, but the free-space bypass can still promote them
when the top tier has enough free space.

Thanks for pointing this out.  I will clarify this setup in the commit
message.

Thanks,
Zhe

>
> Regards,
> Bharata.

Re: [PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Bharata B Rao]

On 29-Oct-26 3:16 PM, Li Zhe wrote:
> [You don't often get email from lizhe.67@bytedance.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> On 10/29/26 5:34 PM, Bharata B Rao wrote:
>> On 28-Oct-26 9:42 AM, Li Zhe wrote:
>>> Memory tiering normally promotes any recently accessed slow-tier page if
>>> the target fast-tier node has enough free space. This preserves the
>>> existing policy of filling available fast memory before relying on
>>> hotness and rate-limit control.
>>>
>>> That is not always the desired policy. Some deployments may intentionally
>>> keep part of a workload's memory on CXL-attached memory, for example to
>>> preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
>>> the free-space bypass, short cold accesses and bursts of hot accesses can
>>> refill the fast tier and pull the workload away from that distribution
>>> whenever DRAM has free space.
>>>
>>> Redis testing exercises this policy issue. On a CXL system with node0 as
>>> DRAM and node3 as slow-tier memory, a Redis test was run with
>>> memory-tiering-only mode enabled. The test used a Redis dataset with 12
>>> million 4KB values. It reclaimed memory until about 85% of Redis
>>> anonymous memory was on node3, used a Redis benchmark configuration with
>>> a 10% hot-key working set, and then issued a short 256MB access burst to
>>> cold keys outside that working set. The cold-key burst mainly touches
>>> memory outside the hot working set, which is expected to remain mostly on
>>> the slow tier.
>> Can you please clarify if 85% of redis memory ended up on lower tier naturally
>> due to overflow from toptier? I ask this question because, if it is a natural
>> overflow/demotion, then it will be hard to find head room in the toptier to
>> allow promotions that bypass hotness or rate limit checks."
> 
> 
> It was not a natural overflow from a full top tier.

Thanks for confirming this.

> 
> The test intentionally reclaimed/demoted Redis memory to the lower tier
> to model a deployment that keeps only a limited fraction of the workload
> in DRAM and leaves the rest on CXL memory.  In that setup the top tier can
> still have enough free space, so the current free-space bypass can promote
> short cold accesses back to DRAM.
With pghot, I have been providing benchmark numbers from two scenarios:

1. Non-overcommitted case: The workload memory is pushed to lower tier before
the measurement phase begins, the toptier has enough memory to accept the
the incoming promotions.

2. Overcommitted case: The workload memory naturally overflows/demotes into
the lower tier and every promotion undergoes the hotness + ratelimiting check.

I gather that your measurement technique is similar to case 1 above.

The feedback I have received for pghot is this: Non-overcommitted case is not
the right methodology to use but the natural spillover is. Hence I am not sure
if optimizing the free-space bypass case would benefit any naturally run
cases.

Regards,
Bharata.

Re: [PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Li Zhe]

On 10/29/26 6:22 PM, Bharata B Rao wrote:
> On 29-Oct-26 3:16 PM, Li Zhe wrote:
>> [You don't often get email from lizhe.67@bytedance.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> On 10/29/26 5:34 PM, Bharata B Rao wrote:
>>> On 28-Oct-26 9:42 AM, Li Zhe wrote:
>>>> Memory tiering normally promotes any recently accessed slow-tier page if
>>>> the target fast-tier node has enough free space. This preserves the
>>>> existing policy of filling available fast memory before relying on
>>>> hotness and rate-limit control.
>>>>
>>>> That is not always the desired policy. Some deployments may intentionally
>>>> keep part of a workload's memory on CXL-attached memory, for example to
>>>> preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
>>>> the free-space bypass, short cold accesses and bursts of hot accesses can
>>>> refill the fast tier and pull the workload away from that distribution
>>>> whenever DRAM has free space.
>>>>
>>>> Redis testing exercises this policy issue. On a CXL system with node0 as
>>>> DRAM and node3 as slow-tier memory, a Redis test was run with
>>>> memory-tiering-only mode enabled. The test used a Redis dataset with 12
>>>> million 4KB values. It reclaimed memory until about 85% of Redis
>>>> anonymous memory was on node3, used a Redis benchmark configuration with
>>>> a 10% hot-key working set, and then issued a short 256MB access burst to
>>>> cold keys outside that working set. The cold-key burst mainly touches
>>>> memory outside the hot working set, which is expected to remain mostly on
>>>> the slow tier.
>>> Can you please clarify if 85% of redis memory ended up on lower tier naturally
>>> due to overflow from toptier? I ask this question because, if it is a natural
>>> overflow/demotion, then it will be hard to find head room in the toptier to
>>> allow promotions that bypass hotness or rate limit checks."
>>
>> It was not a natural overflow from a full top tier.
> Thanks for confirming this.
>
>> The test intentionally reclaimed/demoted Redis memory to the lower tier
>> to model a deployment that keeps only a limited fraction of the workload
>> in DRAM and leaves the rest on CXL memory.  In that setup the top tier can
>> still have enough free space, so the current free-space bypass can promote
>> short cold accesses back to DRAM.
> With pghot, I have been providing benchmark numbers from two scenarios:
>
> 1. Non-overcommitted case: The workload memory is pushed to lower tier before
> the measurement phase begins, the toptier has enough memory to accept the
> the incoming promotions.
>
> 2. Overcommitted case: The workload memory naturally overflows/demotes into
> the lower tier and every promotion undergoes the hotness + ratelimiting check.
>
> I gather that your measurement technique is similar to case 1 above.
>
> The feedback I have received for pghot is this: Non-overcommitted case is not
> the right methodology to use but the natural spillover is. Hence I am not sure
> if optimizing the free-space bypass case would benefit any naturally run
> cases.


Thanks for sharing this feedback.

Yes, the case I am trying to cover is closer to the non-overcommitted
case.  We do have deployments where memory is intentionally placed across
DRAM and CXL memory, instead of relying only on natural spillover from a
full top tier.

For example, the orchestration layer may want a cgroup to use a planned
DRAM/CXL ratio, so that DRAM capacity remains available for later
placement decisions or other workloads.

In such setups, the top tier can still have free space, but we may not
want short cold accesses to refill it simply because the free-space
bypass is available.  The intent of this patch is to let those
deployments keep using the existing hotness and rate-limit checks even
when there is top-tier headroom.

Thanks,
Zhe

>
> Regards,
> Bharata.