[RFC PATCH 0/6] sched/fair: Introduce the SMT_BUDDY_ASSIST feature

[RFC PATCH 0/6] sched/fair: Introduce the SMT_BUDDY_ASSIST feature

[Li kunyu]

[Background]

In high-performance computing (HPC) and cloud computing scenarios,
maximizing overall compute throughput and sustaining peak CPU capacity
remain critical optimization goals. Within the scheduling domain, the
primary optimization focuses include:

1. Timely responsiveness for task wakeup and execution startup.

2. Fast and balanced distribution of CPU utilization across cores.

3. Preferring task migration within shared-cache domains (e.g., L3)
before burst workloads arrive, so as to reduce cross-die or
cross-node overhead.

During development, the following issues were encountered. The above
approach ultimately yielded measurable performance improvements.


[Problem]

It was found that on servers equipped with the Hygon 7447v CPU (Model 7,
384 logical cores), migrating tasks across LLC scheduling domains during
load balancing caused significant performance degradation:

1. A low L3 cache hitting rate (due to factors such as non-shared L3
caches).

2. The traversal overhead of higher-level scheduling domains (e.g.,
cross-NUMA scheduling domains).

Similar issues were also observed on CPUs from other vendors during
testing, though the degree of memory bandwidth increase and other
latency effects varied. Consequently, a series of patches was developed
to address and improve these issues.


[Solution]

This RFC patch series introduces the SMT_BUDDY_ASSIST scheduling
feature. The main work is as follows:

1. Push CFS tasks to SMT sibling CPUs during periodic load balancing.

2. Skip SMT-domain balancing after a successful push and avoid pulling
the pushed task back.

3. Restrict newidle balancing so it does not pull from higher-level
scheduling domains.

4. Compensate for the resulting load imbalance with a burst-task wakeup
path that sends tasks to idle CPUs.


The detailed implementation approach is as follows:

Load balancing-related procedures are primarily categorized into two
major directions:

1. The periodic load balancing path, comprising periodic load balancing
and NOHZ kick, retains the original logic and continues to maintain
overall CPU balance.

2. The newidle CPU path logic has been updated solely to quickly obtain
efficient tasks. By restricting task acquisition to the LLC (L3)
scheduling domain and avoiding traversal and task migration at levels
above it, this change avoids jitter in scenarios where the entire
machine is idle or the CPU load is relatively balanced. This is because
each traversal of higher-level scheduling domains remains costly, and
migrating tasks across dies (LLCs) incurs substantial short-term
overhead, such as TLB flushes/refills and page migration:

  2.1. When overall CPU utilization is relatively low, avoid the
scenario where the whole CPU is idle yet cross-die (high-latency) tasks
are being pulled. In such cases, it is better to run only one logical
CPU per core to avoid cache interference and frequent TLB flushes. This
applies when:
    2.1.1. the CPU has multiple sleeping tasks (with delayed dequeue
disabled).
    2.1.2. the number of runnable tasks on the CPU is less than 2.

  2.2. When overall CPU load is relatively balanced, refrain from
pulling cross-die (high-latency) tasks. This avoids task jitter and
prevents resource exhaustion (similar to the issues in 2.1).


Of course, the change to the newidle CPU path logic has also introduced
new issues, including: 
1. bursty task creation, where a few CPUs spawn a large number of
processes or threads in a short period of time and these tasks cannot be
migrated to low-load CPUs in a timely manner.
2. uneven system-state load across CPUs. For example, some CPUs show
0% to 2% user-mode and 30% to 80% kernel-mode utilization, while others
show 0% to 2% user-mode and 10% to 30% kernel-mode utilization. In addition
, more than 10% of CPUs remain idle.


To address the aforementioned issues, a more lightweight implementation
was adopted to avoid the interference caused by the original newidle CPU
path. Subsequently, select_task_rq_fair() is used to quickly select the
most suitable CPU, as follows:

1. The original task wake-up logic remains unchanged: it first runs the
current-CPU wake-up, affinity wake-up, and similar paths. when the task
count on the found CPU exceeds a threshold (default: 4) a burst-task CPU
allocation process is used.

2. For newly created processes or threads, there is less cause for
concern. Since the CPU selection in the selected scheduling domain meets
the burst-task condition during the slow path, applying the burst-task
CPU allocation process is optimal, as it can quickly utilize idle CPUs
and thereby achieve rapid overall task distribution.


This series of patches also introduces a notable design: in the periodic
load balancing path, tasks are pushed to SMT sibling CPUs to quickly
distribute load within the same core, while also pulling tasks from
other CPUs (excluding SMT siblings) during the current scheduling domain
traversal. This provides a new mechanism for evenly distributing tasks
across the entire machine and strongly supports the change to the
newidle CPU path.


[Implementation]

This patchset gathers 3 different parts:

- Patch 1-3 In periodic load balancing, tasks are pushed to SMT sibling
  CPUs, and pulling them back should be avoided.

- Patch 4 For newidle CPUs, avoid pulling tasks from cross-die
  (high-latency) CPUs.

- Patch 5 This change fixes the local limitations introduced by patch 4
  (e.g., the hackbench performance regression), improves overall
  performance, and quickly distributes tasks to idle CPUs based on the
  task surge threshold.


For detailed information on each patch, please refer to the respective
patch description.


Enable feature:
    echo "SMT_BUDDY_ASSIST" > /sys/kernel/debug/sched/features


[The test results are as follows (Hygon 7447v CPU)]

The overall average improvement is approximately +9.26%.
Testing on other hardware platforms is working in progress.

hackbench:
group   base    smt_buddy pct
g=1	0.262	0.2646	  -0.99%(degeneration)
g=2	0.2878	0.2954	  -2.64%(degeneration)
g=4	0.4074	0.347	  +14.83%(improve)
g=8	0.5674	0.5328	  +6.10%(improve)
g=16	0.8358	0.8418	  -0.72%(degeneration)
g=32	1.411	1.2166	  +13.78%(improve)
g=64	2.4726	2.3564	  +4.70%(improve)
g=128	4.5024	4.5424	  -0.89%(degeneration)
Average improvement: +4.27%.


tbench(Throughput):
clients base       smt_buddy    pct
1	475.968	   487.169	+2.35%(improve)
2	714.484	   808.752	+13.19%(improve)
4	1349.733   1435.15	+6.33%(improve)
8	2058.997   2085.303	+1.28%(improve)
16	4364.247   3692.093	-15.40%(degeneration)
32	8325.17	   6772.693	-18.65%(degeneration)
64	12614.197  12448.367	-1.31%(degeneration)
128	15845.067  17133.367	+8.13%(improve)
256	20837.933  22958.567	+10.18%(improve)
512	29820.667  28242.767	-5.29%(degeneration)
1024	25969.933  22917.733	-11.75%(degeneration)
Average improvement: -0.99%.


schbench(99th):
threads base    smt_buddy  pct
1	10.75	7.6	   +29.30%(improve)
2	11.2	10.8	   +3.57%(improve)
4	12.8	12.2	   +4.69%(improve)
8	13.8	15.0	   -8.70%(degeneration)
16	14.8	15.0	   -1.35%(degeneration)
32	17.0	17.4	   -2.35%(degeneration)
64	27.2	28.8	   -5.88%(degeneration)
128	2844	2845.6	   -0.06%(degeneration)
256	5656	5672	   -0.28%(degeneration)
512	9676	10096	   -4.34%(degeneration)
Average improvement: +1.46%.


stream-10:
operation  base         smt_buddy       pct
Copy	   103034.5	126128.4	+22.41%(improve)
Scale	   100012.0	131812.34	+31.80%(improve)
Add	   128469.0	151501.48	+17.93%(improve)
Triad	   114401.24	158818.78	+38.83%(improve)
Average improvement: +27.74%.


stream-100:
operation  base         smt_buddy       pct
Copy	   122156.6	169662.65	+38.89%(improve)
Scale	   127074.8	168375.375	+32.50%(improve)
Add	   140021.52	154840.075	+10.58%(improve)
Triad	   136455.38	151062.275	+10.70%(improve)
Average improvement: +23.17%.


unixbench:
cpu     base      smt_buddy     pct
1 	1808.3	  1853.1	+2.48%(improve)
384 	113834.2  122562.6	+7.67%(improve)
Average improvement: +5.08%.


sysbench_mysql_test:
operation    base       smt_buddy     pct
TPS	     128.955	134.555	      +4.34%(improve)
QPS	     2579.085	2691.055      +4.34%(improve)
95% latency  1060.31 ms	1022.805 ms   +3.54%(improve)
Average improvement: +4.07%.


Several benchmarks show performance regressions:

1. tbench 32      8325.17    6772.693     -18.65%(degeneration):
Each tbench sample was run three times. Initially, some measurement
error was observed. Moreover, this patch series essentially utilized all
idle CPUs, whereas without the feature enabled, 5% to 10% of idle CPUs may
remain idle. This led to more CPUs experiencing cache interference and
other effects. The results are benign and within an acceptable range.

2. schbench(99th) 8       13.8    15.0       -8.70%(degeneration):
The situation is similar to that observed with tbench. Each sample was
tested five times. The primary regression occurs under bursty task
scenarios, in which tasks are spread across idle CPUs, resulting in
increased memory bandwidth contention, cache latency, and other delays
over a short interval. This is also considered acceptable.


Li kunyu (6):
  sched/fair: Add SMT sibling push in periodic load balance
  sched/fair: Add is_smt_buddy_run() to skip SMT domain
  sched/fair: SMT sibling exclusion after successful push
  sched/fair: Restrict newidle balance to LLC domain
  sched/fair: Add task burst threshold for large-scale bursts
  sched/docs: Add documentation for SMT Buddy Assist feature

 Documentation/scheduler/smt-buddy-assist.rst | 126 ++++++++++
 kernel/sched/fair.c                          | 239 ++++++++++++++++++-
 kernel/sched/features.h                      |   9 +
 3 files changed, 364 insertions(+), 10 deletions(-)
 create mode 100644 Documentation/scheduler/smt-buddy-assist.rst

-- 
2.47.3

[RFC PATCH 1/6] sched/fair: Add SMT sibling push in periodic load balance

[Li kunyu]

[Purpose]

An attempt is made to migrate the task to an SMT sibling CPU. If the
migration fails (i.e., the conditions are not met), it generally incurs
no additional overhead.

Migration fails in the following scenarios: (1) the current CPU is in
the CPU_IDLE state; (2) the current CPU load is relatively low; (3) the
overall CPU load is evenly distributed. All three cases are expected, so
no intervention is necessary.

Migration succeeds in the following scenarios: (1) the current CPU load
is moderately high; (2) the current CPU spawns a large number of
processes or threads within a short period (i.e., bursty tasks);  (3)
the SMT sibling CPUs are idle while the current CPU meets the minimum
migration requirements. All three cases are expected and can achieve
rapid task distribution.

[Implementation]

When conceptualizing this plan, we attempted to add push functions in
multiple locations, and finally tested and analyzed that adding this
function in periodic load balancing is the most appropriate:

1. There is no need to consider complex locking logic, as the CPU is
already idle and the task is no longer running.

2. At this point, it is only necessary for the CFS load of the SMT
sibling CPU to be lower than a certain proportion of the current CPU,
which is more efficient.

3. After the current CPU load drops (task migration successful), there
is an opportunity to pull CFS tasks from other CPUs, which is meaningful
for the overall situation.

Tested-by: Li Wang (Hieco) <li.wang@linux.dev>
Signed-off-by: Li kunyu <likunyu@open-hieco.net>
---
 kernel/sched/fair.c     | 108 ++++++++++++++++++++++++++++++++++++++++
 kernel/sched/features.h |   9 ++++
 2 files changed, 117 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 6e005f1e8dc9..a43a168f23ba 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -136,6 +136,15 @@ static unsigned int sysctl_sched_cfs_bandwidth_slice		= 5000UL;
 static unsigned int sysctl_numa_balancing_promote_rate_limit = 65536;
 #endif
 
+/*
+ * The imbalance percentage condition for pushing tasks to SMT sibling CPUs.
+ *
+ * (default: ~25%)
+ */
+static unsigned int sysctl_smt_buddy_imbalance_pct = 125;
+static unsigned int sysctl_smt_buddy_imbalance_pct_min = 105;
+static unsigned int sysctl_smt_buddy_imbalance_pct_max = 200;
+
 #ifdef CONFIG_SYSCTL
 static const struct ctl_table sched_fair_sysctls[] = {
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -158,6 +167,15 @@ static const struct ctl_table sched_fair_sysctls[] = {
 		.extra1		= SYSCTL_ZERO,
 	},
 #endif /* CONFIG_NUMA_BALANCING */
+	{
+		.procname	= "smt_buddy_imbalance_pct",
+		.data		= &sysctl_smt_buddy_imbalance_pct,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= &sysctl_smt_buddy_imbalance_pct_min,
+		.extra2		= &sysctl_smt_buddy_imbalance_pct_max,
+	},
 };
 
 static int __init sched_fair_sysctl_init(void)
@@ -14226,6 +14244,94 @@ update_newidle_cost(struct sched_domain *sd, u64 cost, unsigned int success)
 	return false;
 }
 
+/*
+ * An exciting thing is that we have implemented an efficient push
+ * function process.
+ *
+ * It is applied in cycle load balancing and nohz_kick process.
+ * It can help us solve the problem of high workload of CFS tasks in
+ * the RQ queue.
+ */
+static bool smt_buddy_move_task(struct rq *src_rq)
+{
+	bool is_overload, is_move = false;
+	int src_cpu = cpu_of(src_rq), smt_cpu;
+	unsigned long src_cpu_runnable, smt_cpu_runnable;
+	struct sched_domain *smt_sd;
+	struct task_struct *p;
+	struct rq *smt_rq;
+	struct rq_flags rf;
+
+	if (!sched_feat(SMT_BUDDY_ASSIST) || !sched_smt_active())
+		goto exit;
+
+restart:
+	if (src_rq->nr_running < 2)
+		goto exit;
+
+	rcu_read_lock();
+
+	is_overload = false;
+	src_cpu_runnable = cpu_runnable(src_rq);
+	for_each_cpu(smt_cpu, cpu_smt_mask(src_cpu)) {
+		if (smt_cpu == src_cpu)
+			continue;
+
+		smt_cpu_runnable = cpu_runnable(cpu_rq(smt_cpu));
+		if ((src_cpu_runnable * 100 > smt_cpu_runnable * sysctl_smt_buddy_imbalance_pct)
+			&& cpu_active(smt_cpu)) {
+			is_overload = true;
+			break;
+		}
+	}
+
+	if (!is_overload) {
+		rcu_read_unlock();
+		goto exit;
+	}
+
+	smt_sd = lowest_flag_domain(src_cpu, SD_SHARE_CPUCAPACITY);
+	if (!smt_sd) {
+		rcu_read_unlock();
+		goto exit;
+	}
+
+	rcu_read_unlock();
+
+	rq_lock_irqsave(src_rq, &rf);
+	smt_rq = cpu_rq(smt_cpu);
+
+	struct lb_env env = {
+		.sd             = smt_sd,
+		.dst_cpu        = smt_cpu,
+		.dst_rq         = smt_rq,
+		.src_cpu        = src_cpu,
+		.src_rq         = src_rq,
+		.idle           = 0,		/* __CPU_NOT_IDLE */
+		.flags          = LBF_ACTIVE_LB,
+	};
+
+	/* Update the src_rq clock only once when migrating tasks. */
+	if (!is_move)
+		update_rq_clock(src_rq);
+
+	p = detach_one_task(&env);
+	if (!p) {
+		rq_unlock_irqrestore(src_rq, &rf);
+		goto exit;
+	}
+
+	rq_unlock(src_rq, &rf);
+	attach_one_task(smt_rq, p);
+	local_irq_restore(rf.flags);
+	is_move = true;
+
+	goto restart;
+
+exit:
+	return is_move;
+}
+
 /*
  * It checks each scheduling domain to see if it is due to be balanced,
  * and initiates a balancing operation if so.
@@ -14245,6 +14351,8 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 	int need_decay = 0;
 	u64 max_cost = 0;
 
+	smt_buddy_move_task(rq);
+
 	rcu_read_lock();
 	for_each_domain(cpu, sd) {
 		/*
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 8f0dee8fc475..173e39bb950a 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -142,3 +142,12 @@ SCHED_FEAT(LATENCY_WARN, false)
  */
 SCHED_FEAT(NI_RANDOM, true)
 SCHED_FEAT(NI_RATE, true)
+
+/*
+ * Based on the scalability feature added on the basis of SMT, by pushing
+ * CFS tasks to SMT sibling CPUs, the load can be rapidly reduced.
+ * When the load of SMT sibling CPUs is relatively high compared to
+ * the scheduling domain (above SMT), CFS tasks can be migrated to other
+ * CPUs in the scheduling domain, achieving fast and efficient migration of CFS tasks.
+ */
+SCHED_FEAT(SMT_BUDDY_ASSIST, false)
-- 
2.47.3

[RFC PATCH 2/6] sched/fair: Add is_smt_buddy_run() to skip SMT domain

[Li kunyu]

When a CFS task is successfully pushed to an SMT sibling CPU during
periodic load balancing or nohz_kick, scanning the SMT scheduling domain
afterwards is unnecessary. It also risks pulling the task back to the
original CPU, defeating the purpose of the push.

The is_smt_buddy_run() helper is introduced to detect whether a task is
already running on its SMT sibling. When this condition holds, the SMT
scheduling domain is skipped in the subsequent load balance path.

This improves efficiency by avoiding redundant SMT domain scans and
prevents reverse pulling of tasks that were just pushed to the sibling.

Tested-by: Li Wang (Hieco) <li.wang@linux.dev>
Signed-off-by: Li kunyu <likunyu@open-hieco.net>
---
 kernel/sched/fair.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index a43a168f23ba..12b3223c6403 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -14244,6 +14244,22 @@ update_newidle_cost(struct sched_domain *sd, u64 cost, unsigned int success)
 	return false;
 }
 
+/*
+ * When the CFS task is successfully pushed to the SMT sibling CPU in
+ * the cyclic load balancing and nohz_kick process, skipping the SMT
+ * scheduling domain check can improve efficiency to a certain extent.
+ */
+static bool is_smt_buddy_run(struct sched_domain *sd, bool is_smt_move_task)
+{
+	if (!sched_feat(SMT_BUDDY_ASSIST))
+		return false;
+
+	if ((sd->flags & SD_SHARE_CPUCAPACITY) && is_smt_move_task)
+		return true;
+
+	return false;
+}
+
 /*
  * An exciting thing is that we have implemented an efficient push
  * function process.
@@ -14350,8 +14366,7 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 	int update_next_balance = 0;
 	int need_decay = 0;
 	u64 max_cost = 0;
-
-	smt_buddy_move_task(rq);
+	bool is_smt_move_task = smt_buddy_move_task(rq);
 
 	rcu_read_lock();
 	for_each_domain(cpu, sd) {
@@ -14375,7 +14390,8 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 
 		interval = get_sd_balance_interval(sd, busy);
 		if (time_after_eq(jiffies, sd->last_balance + interval)) {
-			if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
+			if (!is_smt_buddy_run(sd, is_smt_move_task) &&
+				sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
 				/*
 				 * The LBF_DST_PINNED logic could have changed
 				 * env->dst_cpu, so we can't know our idle
-- 
2.47.3

[RFC PATCH 3/6] sched/fair: SMT sibling exclusion after successful push

[Li kunyu]

In sched_balance_domains(), after a task is successfully pushed to its
SMT sibling CPU, the pull process continues. To prevent the already-pushed
task from being pulled again, both the SMT scheduling domain and the
sibling CPUs are skipped during selection. As a result, SMT sibling CPUs
are not selected.

This patch skips SMT sibling CPUs.

Tested-by: Li Wang (Hieco) <li.wang@linux.dev>
Signed-off-by: Li kunyu <likunyu@open-hieco.net>
---
 kernel/sched/fair.c | 32 ++++++++++++++++++++++++++++----
 1 file changed, 28 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 12b3223c6403..6e2271484c86 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10657,6 +10657,8 @@ struct lb_env {
 	enum fbq_type		fbq_type;
 	enum migration_type	migration_type;
 	struct list_head	tasks;
+
+	bool			is_smt_move_task;
 };
 
 /*
@@ -13377,7 +13379,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 	struct sched_domain __maybe_unused *sd_tmp;
 	unsigned int busiest_nr = 0;
 	int __maybe_unused dst_llc;
-	int i;
+	int i, smt_cpu, dst_cpu = env->dst_cpu;
+
+	if (sched_feat(SMT_BUDDY_ASSIST) && env->is_smt_move_task) {
+		/*
+		 * Remove the SMT sibling CPU from the mask to avoid the
+		 * extremely rare case of pulling tasks from it.
+		 */
+		for_each_cpu(smt_cpu, cpu_smt_mask(dst_cpu)) {
+			if (smt_cpu == dst_cpu)
+				continue;
+
+			cpumask_clear_cpu(smt_cpu, env->cpus);
+		}
+	}
 
 	for_each_cpu_and(i, sched_group_span(group), env->cpus) {
 		unsigned long capacity, load, util;
@@ -13743,7 +13758,7 @@ static atomic_t sched_balance_running = ATOMIC_INIT(0);
  */
 static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 			struct sched_domain *sd, enum cpu_idle_type idle,
-			int *continue_balancing)
+			int *continue_balancing, bool is_smt_move_task)
 {
 	int ld_moved, cur_ld_moved, active_balance = 0;
 	struct sched_domain *sd_parent = sd->parent;
@@ -13761,6 +13776,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
 		.cpus		= cpus,
 		.fbq_type	= all,
 		.tasks		= LIST_HEAD_INIT(env.tasks),
+		.is_smt_move_task = is_smt_move_task,
 	};
 	bool need_unlock = false;
 
@@ -14390,8 +14406,15 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 
 		interval = get_sd_balance_interval(sd, busy);
 		if (time_after_eq(jiffies, sd->last_balance + interval)) {
+			/*
+			 * When a task is successfully pushed to its SMT sibling CPU:
+			 * 1. Skip traversal of the SMT scheduling domain.
+			 * 2. Continue pulling tasks from heavily loaded CPUs.
+			 * 3. Avoid pulling tasks from SMT sibling CPUs.
+			 */
 			if (!is_smt_buddy_run(sd, is_smt_move_task) &&
-				sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
+				sched_balance_rq(cpu, rq, sd, idle, &continue_balancing,
+					is_smt_move_task)) {
 				/*
 				 * The LBF_DST_PINNED logic could have changed
 				 * env->dst_cpu, so we can't know our idle
@@ -15059,7 +15082,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
 
 			pulled_task = sched_balance_rq(this_cpu, this_rq,
 						   sd, CPU_NEWLY_IDLE,
-						   &continue_balancing);
+						   &continue_balancing,
+							false);
 
 			t1 = sched_clock_cpu(this_cpu);
 			domain_cost = t1 - t0;
-- 
2.47.3

[RFC PATCH 4/6] sched/fair: Restrict newidle balance to LLC domain

[Li kunyu]

[Purpose]

The newidle CPU path logic update is solely intended to quickly obtain
eligible tasks. By restricting task acquisition to the LLC (L3)
scheduling domain and avoiding traversal and task migration at levels
above it, this change avoids jitter in scenarios where the entire
machine is idle or the CPU load is relatively balanced. This is because
each traversal of higher-level scheduling domains remains costly, and
migrating tasks across dies (LLCs) incurs substantial short-term
overhead, such as TLB flushes/refills and page migration:

1. When overall CPU utilization is relatively low, avoid the
scenario where the whole CPU is idle yet cross-die (high-latency) tasks
are being pulled. In such cases, it is better to run only one logical
CPU per core to avoid cache interference and frequent TLB flushes. This
applies when:
    1.1. the CPU has multiple sleeping tasks (with delayed dequeue
disabled).
    1.2. the number of runnable tasks on the CPU is less than 2.

2. When overall CPU load is relatively balanced, refrain from
pulling cross-die (high-latency) tasks. This avoids task jitter and
prevents resource exhaustion (similar to the issues in 2.1).

[Implementation]

After extensive testing on a Hygon 7447v CPU server (Model 7, with 384
logical cores), several issues were identified. For example, when the CPU
first enters idle mode, cross-NUMA-node task pulling performs poorly in
certain test scenarios, including MySQL, UnixBench, and tbench. Likewise,
certain advantages were observed in tests using hackbench -l 2560 -g 4 and
in additional test groups.

However, its benefits in some respects are simply too great: it avoids
more time-consuming and extensive scheduling-domain traversal, as well
as cross-NUMA-node task pulling (avoiding the problem of low hit rates
from memory to L3 cache). Limiting the task-pulling range to the L3
cache is more efficient.

In actual testing, the following scenarios have shown significant
improvements:

1. unixbench: Overall improvement of 5% to 8% across 384 logical CPUs.

2. mysql: TPS and QPS increased by approximately 5% (this may not be
entirely accurate), while the 95th percentile latency decreased by
3% to 5%.

Tested-by: Li Wang (Hieco) <li.wang@linux.dev>
Signed-off-by: Li kunyu <likunyu@open-hieco.net>
---
 kernel/sched/fair.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 6e2271484c86..f2bf4b53d8a9 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -14983,6 +14983,24 @@ static inline bool nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle
 static inline void nohz_newidle_balance(struct rq *this_rq) { }
 #endif /* !CONFIG_NO_HZ_COMMON */
 
+/*
+ * is_smt_buddy_newidle_constrain - After enabling the smt_buddy_assist
+ *                                  feature, check whether the current
+ *                                  scheduling domain meets the conditions
+ *                                  for skipping statistics.
+ * @sd: The current scheduling domain that is to be traversed.
+ */
+static bool is_smt_buddy_newidle_constrain(struct sched_domain *sd)
+{
+	if (!sched_feat(SMT_BUDDY_ASSIST))
+		return false;
+
+	if (sd->flags & SD_NUMA)
+		return true;
+
+	return false;
+}
+
 /*
  * sched_balance_newidle is called by schedule() if this_cpu is about to become
  * idle. Attempts to pull tasks from other CPUs.
@@ -15080,7 +15098,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
 				weight = (1024 + weight/2) / weight;
 			}
 
-			pulled_task = sched_balance_rq(this_cpu, this_rq,
+			if (!is_smt_buddy_newidle_constrain(sd))
+				pulled_task = sched_balance_rq(this_cpu, this_rq,
 						   sd, CPU_NEWLY_IDLE,
 						   &continue_balancing,
 							false);
-- 
2.47.3

[RFC PATCH 5/6] sched/fair: Add task burst threshold for large-scale bursts

[Li kunyu]

In practice, bursty tasks can lead to uneven task distribution. For
example, some CPUs may have kernel-mode utilization above 50%, while
more than 10% of CPUs in the system remain idle. To address this, a task
burst threshold is introduced:

1. When a task is woken up, the scheduler first follows the original
wake-up logic (e.g., wake-up on the current CPU or affinity wake-up)
to select a CPU. If no CPU is selected, the task burst threshold of
new_cpu is then checked.

2. For newly created child processes or threads, only the task burst
threshold needs to be checked.

Benefits:

1. The performance degradation observed in the hackbench scenario is
resolved, allowing tasks to be distributed quickly.

2. On a Hygon 7447v CPU (Model 7) server with 384 logical CPUs, UnixBench
shows approximately 3% to 5% performance improvement for this patch, with
a cumulative improvement of 8% to 10% across the patch series.

Tested-by: Li Wang (Hieco) <li.wang@linux.dev>
Signed-off-by: Li kunyu <likunyu@open-hieco.net>
---
 kernel/sched/fair.c | 62 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 57 insertions(+), 5 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index f2bf4b53d8a9..0cd0c5a5f158 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -145,6 +145,15 @@ static unsigned int sysctl_smt_buddy_imbalance_pct = 125;
 static unsigned int sysctl_smt_buddy_imbalance_pct_min = 105;
 static unsigned int sysctl_smt_buddy_imbalance_pct_max = 200;
 
+/*
+ * In certain scenarios, the task burst threshold helps distribute tasks to
+ * idle CPUs quickly.
+ *
+ * (default: 4)
+ */
+static unsigned int sysctl_smt_buddy_balance_task = 4;
+static unsigned int sysctl_smt_buddy_balance_task_min = 2;
+
 #ifdef CONFIG_SYSCTL
 static const struct ctl_table sched_fair_sysctls[] = {
 #ifdef CONFIG_CFS_BANDWIDTH
@@ -176,6 +185,14 @@ static const struct ctl_table sched_fair_sysctls[] = {
 		.extra1		= &sysctl_smt_buddy_imbalance_pct_min,
 		.extra2		= &sysctl_smt_buddy_imbalance_pct_max,
 	},
+	{
+		.procname       = "smt_buddy_balance_task",
+		.data           = &sysctl_smt_buddy_balance_task,
+		.maxlen         = sizeof(unsigned int),
+		.mode           = 0644,
+		.proc_handler   = proc_dointvec_minmax,
+		.extra1         = &sysctl_smt_buddy_balance_task_min,
+	},
 };
 
 static int __init sched_fair_sysctl_init(void)
@@ -9937,7 +9954,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	int sync = (wake_flags & WF_SYNC) && !(current->flags & PF_EXITING);
 	struct sched_domain *tmp, *sd = NULL;
 	int cpu = smp_processor_id();
-	int new_cpu = prev_cpu;
+	int new_cpu = prev_cpu, nohz_idle_cpu = 0;
 	int want_affine = 0;
 	/* SD_flags and WF_flags share the first nibble */
 	int sd_flag = wake_flags & 0xF;
@@ -9990,11 +10007,46 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 
 	/* Slow path */
 	if (unlikely(sd))
-		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
-
+		new_cpu = sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
 	/* Fast path */
-	if (wake_flags & WF_TTWU)
-		return select_idle_sibling(p, prev_cpu, new_cpu);
+	else if (wake_flags & WF_TTWU)
+		new_cpu = select_idle_sibling(p, prev_cpu, new_cpu);
+
+#ifdef CONFIG_NO_HZ_COMMON
+	/*
+	 * In practice, bursty tasks can lead to uneven task distribution.
+	 * For example, some CPUs may have kernel-mode utilization above 50%,
+	 * while more than 10% of CPUs in the system remain idle.
+	 * Therefore, we define a task burst threshold:
+	 *
+	 * 1. For a task to be woken up, the scheduler first follows the
+	 * original wake-up logic (such as wake-up on the current CPU or
+	 * affinity wake-up) to return a CPU. If no CPU is returned, it
+	 * then checks the task burst threshold of new_cpu.
+	 * 2. Newly created child processes or threads have fewer
+	 * considerations; only the task burst threshold needs to be checked.
+	 */
+	if (sched_feat(SMT_BUDDY_ASSIST) && cpu_rq(new_cpu)->nr_running >=
+			sysctl_smt_buddy_balance_task)
+		for_each_cpu_wrap(nohz_idle_cpu,  nohz.idle_cpus_mask, new_cpu + 1) {
+			if (!idle_cpu(nohz_idle_cpu))
+				continue;
+
+			/*
+			 * In general-purpose scenarios, the CPU can
+			 * typically meet task requirements.
+			 */
+			if (!cpumask_test_cpu(nohz_idle_cpu, p->cpus_ptr))
+				continue;
+
+		/*
+		 * Simply returning nohz_idle_cpu is sufficient, since the
+		 * subsequent NOHZ kick will clear the corresponding bit in
+		 * nohz.idle_cpus_mask.
+		 */
+		return nohz_idle_cpu;
+	}
+#endif
 
 	return new_cpu;
 }
-- 
2.47.3

[RFC PATCH 6/6] sched/docs: Add documentation for SMT Buddy Assist feature

[Li kunyu]

Document the SMT Buddy Assist mechanism, including its motivation,
enabling method (sysctl interface), and implementation overview. This
helps users understand when and why the feature engages, and how it
interacts with existing CFS load balancing.

Tested-by: Li Wang (Hieco) <li.wang@linux.dev>
Signed-off-by: Li kunyu <likunyu@open-hieco.net>
---
 Documentation/scheduler/smt-buddy-assist.rst | 126 +++++++++++++++++++
 1 file changed, 126 insertions(+)
 create mode 100644 Documentation/scheduler/smt-buddy-assist.rst

diff --git a/Documentation/scheduler/smt-buddy-assist.rst b/Documentation/scheduler/smt-buddy-assist.rst
new file mode 100644
index 000000000000..c252f4fe955f
--- /dev/null
+++ b/Documentation/scheduler/smt-buddy-assist.rst
@@ -0,0 +1,126 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+================
+SMT Buddy Assist
+================
+
+1. Introduction
+===============
+
+SMT introduces scalability features that constrain the scheduling domain
+within the LLC (L3) cache via the newidle CPU path. Combined with the ability
+to push CFS tasks to SMT sibling CPUs and to quickly distribute tasks to idle
+CPUs using select_task_rq_fair, this improves overall CPU performance under
+varying loads.
+
+2. Design
+=========
+
+The detailed implementation approach is as follows:
+
+Load balancing-related procedures are primarily categorized into two
+major directions:
+
+1. The periodic load balancing path, comprising periodic load balancing
+and NOHZ kick, retains the original logic and continues to maintain
+overall CPU balance.
+
+2. The newidle CPU path logic has been updated solely to quickly obtain
+efficient tasks. By restricting task acquisition to the LLC (L3)
+scheduling domain and avoiding traversal and task migration at levels
+above it, this change avoids jitter in scenarios where the entire
+machine is idle or the CPU load is relatively balanced. This is because
+each traversal of higher-level scheduling domains remains costly, and
+migrating tasks across dies (LLCs) incurs substantial short-term
+overhead, such as TLB flushes/refills and page migration:
+
+  2.1. When overall CPU utilization is relatively low, avoid the
+scenario where the whole CPU is idle yet cross-die (high-latency) tasks
+are being pulled. In such cases, it is better to run only one logical
+CPU per core to avoid cache interference and frequent TLB flushes. This
+applies when:
+    2.1.1. the CPU has multiple sleeping tasks (with delayed dequeue
+disabled).
+    2.1.2. the number of runnable tasks on the CPU is less than 2.
+
+  2.2. When overall CPU load is relatively balanced, refrain from
+pulling cross-die (high-latency) tasks. This avoids task jitter and
+prevents resource exhaustion (similar to the issues in 2.1).
+
+Of course, the change to the newidle CPU path logic has also introduced
+new issues, including:
+1. bursty task creation, where a few CPUs spawn a large number of
+processes or threads in a short period of time and these tasks cannot be
+migrated to low-load CPUs in a timely manner.
+2. uneven system-state load across CPUs. For example, some CPUs show
+0% to 2% user-mode and 30% to 80% kernel-mode utilization, while others show
+0% to 2% user-mode and 10% to 30% kernel-mode utilization. In addition, more
+than 10% of CPUs remain idle.
+
+To address the aforementioned issues, a more lightweight implementation
+was adopted to avoid the interference caused by the original newidle CPU
+path. Subsequently, select_task_rq_fair() is used to quickly select the
+most suitable CPU, as follows:
+
+1. The original task wake-up logic remains unchanged: it first runs the
+current-CPU wake-up, affinity wake-up, and similar paths. when the task
+count on the found CPU exceeds a threshold (default: 4) a burst-task CPU
+allocation process is used.
+
+2. For newly created processes or threads, there is less cause for
+concern. Since the CPU selection in the selected scheduling domain meets
+the burst-task condition during the slow path, applying the burst-task
+CPU allocation process is optimal, as it can quickly utilize idle CPUs
+and thereby achieve rapid overall task distribution.
+
+This series of patches also introduces a notable design: in the periodic
+load balancing path, tasks are pushed to SMT sibling CPUs to quickly
+distribute load within the same core, while also pulling tasks from
+other CPUs (excluding SMT siblings) during the current scheduling domain
+traversal. This provides a new mechanism for evenly distributing tasks
+across the entire machine and strongly supports the change to the
+newidle CPU path.
+
+2.1 SMT_BUDDY_ASSIST
+--------------------
+
+Enable feature:
+    echo "SMT_BUDDY_ASSIST" > /sys/kernel/debug/sched/features
+
+3. System interface
+-------------------
+
+3.1. smt_buddy_imbalance_pct
+----------------------------
+
+When pushing tasks to SMT siblings, the current CPU must exceed a certain
+imbalance ratio before its CFS runqueue is considered heavily loaded.
+
+The imbalance ratio threshold is constrained to a range of 105% to 200%,
+covering both mild deviation and severe imbalance scenarios.
+
+Mild deviation:
+For example, when the pull imbalance ratio within the SMT domain is around
+110%, this feature provides a comparable and configurable trigger condition.
+
+Severe imbalance:
+For example, when the runnable time ratio of the local CPU's cfs_rq is 20%
+while that of its SMT sibling is 40%, load balancing is explicitly required
+to restore fairness.
+
+3.2. smt_buddy_balance_task
+---------------------------
+
+The task-burst detection mechanism employs a configurable threshold derived
+from two primary metrics: the runqueue waiting time of a task on the local
+CPU, and the comparative cost of migrating that task to a remote (cross-die)
+CPU versus allowing it to execute locally after a prolonged wait.
+
+A default threshold of 4 is selected such that migration is triggered when
+the queuing latency approaches the execution duration of an individual task.
+The marginal benefit of migration increases monotonically with the number of
+tasks enqueued on the local CPU.
+
+A lower bound of 2 accounts for scenarios involving long-running tasks, which
+require more aggressive distribution across the CPU topology to avoid localized
+congestion.
-- 
2.47.3

Re: [RFC PATCH 1/6] sched/fair: Add SMT sibling push in periodic load balance

[Peter Zijlstra]

On Tue, Oct 20, 2026 at 04:37:41PM +0800, Li kunyu wrote:

Just a few remarks on coding style while I try and make sense of the
whole thing. Don't go rush out a new version with this stuff fixed
before you've seen feedback on the higher level stuff.

> @@ -14226,6 +14244,94 @@ update_newidle_cost(struct sched_domain *sd, u64 cost, unsigned int success)
>  	return false;
>  }
>  
> +/*
> + * An exciting thing is that we have implemented an efficient push
> + * function process.
> + *
> + * It is applied in cycle load balancing and nohz_kick process.
> + * It can help us solve the problem of high workload of CFS tasks in
> + * the RQ queue.
> + */
> +static bool smt_buddy_move_task(struct rq *src_rq)
> +{
> +	bool is_overload, is_move = false;
> +	int src_cpu = cpu_of(src_rq), smt_cpu;
> +	unsigned long src_cpu_runnable, smt_cpu_runnable;
> +	struct sched_domain *smt_sd;
> +	struct task_struct *p;
> +	struct rq *smt_rq;
> +	struct rq_flags rf;
> +
> +	if (!sched_feat(SMT_BUDDY_ASSIST) || !sched_smt_active())
> +		goto exit;
> +
> +restart:
> +	if (src_rq->nr_running < 2)
> +		goto exit;

Did this want to be:

	while (src_rq->nr_running > 1) {

> +
> +	rcu_read_lock();

	scoped_guard (rcu) {

> +
> +	is_overload = false;
> +	src_cpu_runnable = cpu_runnable(src_rq);
> +	for_each_cpu(smt_cpu, cpu_smt_mask(src_cpu)) {
> +		if (smt_cpu == src_cpu)
> +			continue;
> +
> +		smt_cpu_runnable = cpu_runnable(cpu_rq(smt_cpu));
> +		if ((src_cpu_runnable * 100 > smt_cpu_runnable * sysctl_smt_buddy_imbalance_pct)
> +			&& cpu_active(smt_cpu)) {

This is horrifically bad indenting. Operators go at the end of the
previous line and then you indent to the level of whatever ( is active.
If you're a vim user 'cino=(0' gets you that.

> +			is_overload = true;
> +			break;
> +		}
> +	}
> +
> +	if (!is_overload) {
> +		rcu_read_unlock();
> +		goto exit;

This pattern should scream at you for doing it wrong.

> +	}
> +
> +	smt_sd = lowest_flag_domain(src_cpu, SD_SHARE_CPUCAPACITY);
> +	if (!smt_sd) {
> +		rcu_read_unlock();
> +		goto exit;

And again.

> +	}
> +
> +	rcu_read_unlock();
	}
> +
> +	rq_lock_irqsave(src_rq, &rf);

	scoped_guard (irqsave) {
		scoped_guard (rq_lock, src_rq)

> +	smt_rq = cpu_rq(smt_cpu);
> +
> +	struct lb_env env = {
> +		.sd             = smt_sd,
> +		.dst_cpu        = smt_cpu,
> +		.dst_rq         = smt_rq,
> +		.src_cpu        = src_cpu,
> +		.src_rq         = src_rq,
> +		.idle           = 0,		/* __CPU_NOT_IDLE */
> +		.flags          = LBF_ACTIVE_LB,
> +	};
> +
> +	/* Update the src_rq clock only once when migrating tasks. */
> +	if (!is_move)
> +		update_rq_clock(src_rq);
> +
> +	p = detach_one_task(&env);
> +	if (!p) {
> +		rq_unlock_irqrestore(src_rq, &rf);
> +		goto exit;

More of the same bad pattern.
	
> +	}
> +
> +	rq_unlock(src_rq, &rf);
		}
> +	attach_one_task(smt_rq, p);
> +	local_irq_restore(rf.flags);
	}
> +	is_move = true;
> +
> +	goto restart;
	}
	
> +
> +exit:
> +	return is_move;
> +}

Re: [RFC PATCH 2/6] sched/fair: Add is_smt_buddy_run() to skip SMT domain

[Peter Zijlstra]

On Tue, Oct 20, 2026 at 04:37:42PM +0800, Li kunyu wrote:
> @@ -14375,7 +14390,8 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
>  
>  		interval = get_sd_balance_interval(sd, busy);
>  		if (time_after_eq(jiffies, sd->last_balance + interval)) {
> -			if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
> +			if (!is_smt_buddy_run(sd, is_smt_move_task) &&
> +				sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {

Same horrific indenting fail. Surely you've read surrounding code?

>  				/*
>  				 * The LBF_DST_PINNED logic could have changed
>  				 * env->dst_cpu, so we can't know our idle
> -- 
> 2.47.3
> 

Re: [RFC PATCH 3/6] sched/fair: SMT sibling exclusion after successful push

[Peter Zijlstra]

On Tue, Oct 20, 2026 at 04:37:43PM +0800, Li kunyu wrote:

> @@ -13377,7 +13379,20 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
>  	struct sched_domain __maybe_unused *sd_tmp;
>  	unsigned int busiest_nr = 0;
>  	int __maybe_unused dst_llc;
> -	int i;
> +	int i, smt_cpu, dst_cpu = env->dst_cpu;
> +
> +	if (sched_feat(SMT_BUDDY_ASSIST) && env->is_smt_move_task) {
> +		/*
> +		 * Remove the SMT sibling CPU from the mask to avoid the
> +		 * extremely rare case of pulling tasks from it.

Your comment seems to think the SMT mask is only ever 2 cpus, we have a
ton of architectures where this is not true.

> +		 */
> +		for_each_cpu(smt_cpu, cpu_smt_mask(dst_cpu)) {
> +			if (smt_cpu == dst_cpu)
> +				continue;
> +
> +			cpumask_clear_cpu(smt_cpu, env->cpus);
> +		}

This is terrifyingly horrible. You're using a loop and atomic bit clear
to clear bits in a local cpumask.

		cpumask_andnot(env->cpus, env->cpus, cpu_smt_mask(dst_cpu));
		__cpumask_set_cpu(dst_cpu, env->cpus);

Re: [RFC PATCH 4/6] sched/fair: Restrict newidle balance to LLC domain

[Peter Zijlstra]

On Tue, Oct 20, 2026 at 04:37:44PM +0800, Li kunyu wrote:

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 6e2271484c86..f2bf4b53d8a9 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -14983,6 +14983,24 @@ static inline bool nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle
>  static inline void nohz_newidle_balance(struct rq *this_rq) { }
>  #endif /* !CONFIG_NO_HZ_COMMON */
>  
> +/*
> + * is_smt_buddy_newidle_constrain - After enabling the smt_buddy_assist
> + *                                  feature, check whether the current
> + *                                  scheduling domain meets the conditions
> + *                                  for skipping statistics.
> + * @sd: The current scheduling domain that is to be traversed.
> + */
> +static bool is_smt_buddy_newidle_constrain(struct sched_domain *sd)
> +{
> +	if (!sched_feat(SMT_BUDDY_ASSIST))
> +		return false;
> +
> +	if (sd->flags & SD_NUMA)
> +		return true;
> +
> +	return false;
> +}
> +
>  /*
>   * sched_balance_newidle is called by schedule() if this_cpu is about to become
>   * idle. Attempts to pull tasks from other CPUs.
> @@ -15080,7 +15098,8 @@ static int sched_balance_newidle(struct rq *this_rq, struct rq_flags *rf)
>  				weight = (1024 + weight/2) / weight;
>  			}
>  
> -			pulled_task = sched_balance_rq(this_cpu, this_rq,
> +			if (!is_smt_buddy_newidle_constrain(sd))
> +				pulled_task = sched_balance_rq(this_cpu, this_rq,
>  						   sd, CPU_NEWLY_IDLE,
>  						   &continue_balancing,
>  							false);

You could have just fiddled with SD_BALANCE_NEWIDLE, no? But instead you
went and did something inscrutable and violated coding style *again*.
Any multi-line block must have {} on.

Re: [RFC PATCH 0/6] sched/fair: Introduce the SMT_BUDDY_ASSIST feature

[Kayra Cizmeci]

On Tue, Oct 20, 2026 at 04:37:40PM +0800, Li kunyu wrote:

> Several benchmarks show performance regressions:
> 
> 1. tbench 32      8325.17    6772.693     -18.65%(degeneration):
> Each tbench sample was run three times. Initially, some measurement
> error was observed. Moreover, this patch series essentially utilized all
> idle CPUs, whereas without the feature enabled, 5% to 10% of idle CPUs may
> remain idle. This led to more CPUs experiencing cache interference and
> other effects. The results are benign and within an acceptable range.

According to what the results are 'acceptable'? If there were some error
why even talk about them?

> 
> 2. schbench(99th) 8       13.8    15.0       -8.70%(degeneration):
> The situation is similar to that observed with tbench. Each sample was
> tested five times. The primary regression occurs under bursty task
> scenarios, in which tasks are spread across idle CPUs, resulting in
> increased memory bandwidth contention, cache latency, and other delays
> over a short interval. This is also considered acceptable.
> 
> 
> Li kunyu (6):
>   sched/fair: Add SMT sibling push in periodic load balance
>   sched/fair: Add is_smt_buddy_run() to skip SMT domain
>   sched/fair: SMT sibling exclusion after successful push
>   sched/fair: Restrict newidle balance to LLC domain
>   sched/fair: Add task burst threshold for large-scale bursts
>   sched/docs: Add documentation for SMT Buddy Assist feature
> 
>  Documentation/scheduler/smt-buddy-assist.rst | 126 ++++++++++
>  kernel/sched/fair.c                          | 239 ++++++++++++++++++-
>  kernel/sched/features.h                      |   9 +
>  3 files changed, 364 insertions(+), 10 deletions(-)
>  create mode 100644 Documentation/scheduler/smt-buddy-assist.rst
> 
> -- 
> 2.47.3
> 

Re: [RFC PATCH 5/6] sched/fair: Add task burst threshold for large-scale bursts

[Peter Zijlstra]

On Tue, Oct 20, 2026 at 04:37:45PM +0800, Li kunyu wrote:

> +#ifdef CONFIG_NO_HZ_COMMON
> +	/*
> +	 * In practice, bursty tasks can lead to uneven task distribution.
> +	 * For example, some CPUs may have kernel-mode utilization above 50%,
> +	 * while more than 10% of CPUs in the system remain idle.
> +	 * Therefore, we define a task burst threshold:
> +	 *
> +	 * 1. For a task to be woken up, the scheduler first follows the
> +	 * original wake-up logic (such as wake-up on the current CPU or
> +	 * affinity wake-up) to return a CPU. If no CPU is returned, it
> +	 * then checks the task burst threshold of new_cpu.
> +	 * 2. Newly created child processes or threads have fewer
> +	 * considerations; only the task burst threshold needs to be checked.
> +	 */
> +	if (sched_feat(SMT_BUDDY_ASSIST) && cpu_rq(new_cpu)->nr_running >=
> +			sysctl_smt_buddy_balance_task)
> +		for_each_cpu_wrap(nohz_idle_cpu,  nohz.idle_cpus_mask, new_cpu + 1) {
> +			if (!idle_cpu(nohz_idle_cpu))
> +				continue;
> +
> +			/*
> +			 * In general-purpose scenarios, the CPU can
> +			 * typically meet task requirements.
> +			 */
> +			if (!cpumask_test_cpu(nohz_idle_cpu, p->cpus_ptr))
> +				continue;
> +
> +		/*
> +		 * Simply returning nohz_idle_cpu is sufficient, since the
> +		 * subsequent NOHZ kick will clear the corresponding bit in
> +		 * nohz.idle_cpus_mask.
> +		 */
> +		return nohz_idle_cpu;
> +	}
> +#endif

I don't understand why this depends on NO_HZ. I also don't understand
why this only considers idle CPUs.

Also, static thresholds are horrible.


Someone that burst wakes a ton of tasks, even on a non-idle system, will
cause an uneven task distribution. Irrespective of there being idle CPUs
in the system or NOHZ whatever.

So how about we track an average rq-depth in the periodic load-balance
pass, in say the root domain or so. And then, when the the local
rq-depth reaches some multiple (3 sounds like a good number?) of that,
we'll fall back and do something utterly dumb like:

	cpu = cpumask_any_distribute(p->cpus_ptr);

Eg, simply start rotating new tasks around the system *and* reset the
periodic load balance period.

Re: [RFC PATCH 6/6] sched/docs: Add documentation for SMT Buddy Assist feature

[Peter Zijlstra]

On Tue, Oct 20, 2026 at 04:37:46PM +0800, Li kunyu wrote:
> +2.1 SMT_BUDDY_ASSIST
> +--------------------
> +
> +Enable feature:
> +    echo "SMT_BUDDY_ASSIST" > /sys/kernel/debug/sched/features

NO! this is not an interface. Note the 'debug' there.