[PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

["Li Zhe" <lizhe.67@bytedance.com>]

Memory tiering normally promotes any recently accessed slow-tier page if
the target fast-tier node has enough free space. This preserves the
existing policy of filling available fast memory before relying on
hotness and rate-limit control.

That is not always the desired policy. Some deployments may intentionally
keep part of a workload's memory on CXL-attached memory, for example to
preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
the free-space bypass, short cold accesses and bursts of hot accesses can
refill the fast tier and pull the workload away from that distribution
whenever DRAM has free space.

Redis testing exercises this policy issue. On a CXL system with node0 as
DRAM and node3 as slow-tier memory, a Redis test was run with
memory-tiering-only mode enabled. The test used a Redis dataset with 12
million 4KB values. It reclaimed memory until about 85% of Redis
anonymous memory was on node3, used a Redis benchmark configuration with
a 10% hot-key working set, and then issued a short 256MB access burst to
cold keys outside that working set. The cold-key burst mainly touches
memory outside the hot working set, which is expected to remain mostly on
the slow tier.

The upstream default promoted the cold-key burst through the free-space
bypass. The patched default preserved that behavior, while disabling the
bypass kept the same promotions behind the hotness and rate-limit checks:

                              upstream     patched default    bypass disabled
  cold-key burst ops/s          117250         115481             142505
  cold-key burst P99.9 ms        0.854          0.859              0.686
  pgpromote_candidate              0              0               1488328

Compared to upstream default, disabling the bypass improved cold-key burst
throughput by 21.5% and P99.9 latency by 19.7%.

Add a sched debugfs knob:

  /sys/kernel/debug/sched/numa_balancing/promote_free_space_bypass

It defaults to enabled, preserving the existing behavior. When disabled,
enough free space in the target fast-tier node no longer bypasses the
normal hotness and rate-limit controls.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/mmzone.h |  5 ++---
 kernel/sched/debug.c   |  2 ++
 kernel/sched/fair.c    | 14 ++++++++------
 kernel/sched/sched.h   |  1 +
 4 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecd1ebd49756..bf2d604c04b3 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -278,9 +278,8 @@ enum node_stat_item {
 	 */
 	PGPROMOTE_CANDIDATE,
 	/**
-	 * Not rate-limited (NRL) candidate pages for those can be promoted
-	 * without considering hot threshold because of enough free pages in
-	 * fast-tier node.  These promotions bypass the regular hotness checks
+	 * Not rate-limited (NRL) candidate pages selected for promotion through
+	 * the free-space bypass. These candidates bypass the regular hotness checks
 	 * and do NOT influence the promotion rate-limiter or
 	 * threshold-adjustment logic.
 	 * This is for statistics/monitoring purposes.
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e6a3b516c703..a28f0f09c91e 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -736,6 +736,8 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
 	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
+	debugfs_create_bool("promote_free_space_bypass", 0644, numa,
+			    &numa_balancing_promote_free_space_bypass);
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_SCHED_CACHE
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cfc982205b98..46a11f5ddafb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -134,6 +134,8 @@ static unsigned int sysctl_sched_cfs_bandwidth_slice		= 5000UL;
 #ifdef CONFIG_NUMA_BALANCING
 /* Restrict the NUMA promotion throughput (MB/s) for each target node. */
 static unsigned int sysctl_numa_balancing_promote_rate_limit = 65536;
+/* Preserve the existing default of using free fast-tier space eagerly. */
+bool numa_balancing_promote_free_space_bypass = true;
 #endif
 
 #ifdef CONFIG_SYSCTL
@@ -2759,11 +2761,10 @@ static inline bool cpupid_valid(int cpupid)
 }
 
 /*
- * For memory tiering mode, if there are enough free pages (more than
- * enough watermark defined here) in fast memory node, to take full
- * advantage of fast memory capacity, all recently accessed slow
- * memory pages will be migrated to fast memory node without
- * considering hot threshold.
+ * For memory tiering mode, if the target fast memory node has enough free
+ * pages, recently accessed slow memory pages can bypass the normal hotness
+ * check and rate limit so the fast tier can be filled. The bypass can be
+ * disabled by numa_balancing_promote_free_space_bypass.
  */
 static bool pgdat_free_space_enough(struct pglist_data *pgdat)
 {
@@ -2884,7 +2885,8 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 		long nr = folio_nr_pages(folio);
 
 		pgdat = NODE_DATA(dst_nid);
-		if (pgdat_free_space_enough(pgdat)) {
+		if (numa_balancing_promote_free_space_bypass &&
+		    pgdat_free_space_enough(pgdat)) {
 			/* workload changed, reset hot threshold */
 			pgdat->nbp_threshold = 0;
 			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 87c5335add37..662b11bd401e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3218,6 +3218,7 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
 extern unsigned int sysctl_numa_balancing_hot_threshold;
+extern bool numa_balancing_promote_free_space_bypass;
 
 #ifdef CONFIG_SCHED_HRTICK
 
-- 
2.20.1