Re: [RFC PATCH 0/6] sched/fair: Introduce the SMT_BUDDY_ASSIST feature

[Li kunyu <likunyu@open-hieco.net>]

Hi Tim,

On 2026/10/23 4:25, Tim Chen wrote:
> On Tue, 2026-10-20 at 16:37 +0800, Li kunyu wrote:
>> [Background]
>>
>> In high-performance computing (HPC) and cloud computing scenarios,
>> maximizing overall compute throughput and sustaining peak CPU capacity
>> remain critical optimization goals. Within the scheduling domain, the
>> primary optimization focuses include:
>>
>> 1. Timely responsiveness for task wakeup and execution startup.
>>
>> 2. Fast and balanced distribution of CPU utilization across cores.
>>
>> 3. Preferring task migration within shared-cache domains (e.g., L3)
>> before burst workloads arrive, so as to reduce cross-die or
>> cross-node overhead.
>>
>> During development, the following issues were encountered. The above
>> approach ultimately yielded measurable performance improvements.
>>
>>
>> [Problem]
>>
>> It was found that on servers equipped with the Hygon 7447v CPU (Model 7,
>> 384 logical cores), migrating tasks across LLC scheduling domains during
>> load balancing caused significant performance degradation:
>>
>> 1. A low L3 cache hitting rate (due to factors such as non-shared L3
>> caches).
>>
>> 2. The traversal overhead of higher-level scheduling domains (e.g.,
>> cross-NUMA scheduling domains).
>>
>> Similar issues were also observed on CPUs from other vendors during
>> testing, though the degree of memory bandwidth increase and other
>> latency effects varied. Consequently, a series of patches was developed
>> to address and improve these issues.
>>
>>
>> [Solution]
>>
>> This RFC patch series introduces the SMT_BUDDY_ASSIST scheduling
>> feature. The main work is as follows:
>>
>> 1. Push CFS tasks to SMT sibling CPUs during periodic load balancing.
>>
>> 2. Skip SMT-domain balancing after a successful push and avoid pulling
>> the pushed task back.
>>
>> 3. Restrict newidle balancing so it does not pull from higher-level
>> scheduling domains.
> Kunyu,
> 
> Have you tried to set relax_domain_level= in boot parameter to
> prevent new idle balance across NUMA nodes?
> 
> grep . /sys/kernel/debug/sched/domains/cpu0/domain*/{name,level,flags}
> 
> to find the right level for NUMA and also if the SD_BALANCE_NEWIDLE
> is turned off for NUMA after you set relax_domain_level.
> 
> Wonder how much this will improve things for you.


I conducted a comparative test between patch4 and relax_domain_level=4
(scheduling domain levels 3, SMT, LLC, and node). Their effective scheduling
levels were the same, and their overall averages were quite close (+3.28%). The
trends for each item were consistent.

At present, relax_domain_level has not solved the problems encountered. It
merely serves as a reference for patch4. Subsequent versions of patch4 will be
optimized following the style of relax_domain_level.

I would like to seek your opinion on the modification of patch4. Please review
the following:

1. If relax_domain_level is set, patch4 is disabled.

2. If relax_domain_level is not set and the feature is enabled, patch4 takes effect.

The overall average values and the data for each test are shown below. Here,
smt_buddy denotes the patch4 code, and relax denotes the relax_domain_level = 4
code:


type           smt_buddy_pct  relax_pct
hackbench	-14.94%	       -17.65%
tbench	        +12.54%	       +8.70%
schbench	-4.27%	       +0.48%
STREAM	        +21.80%	       +23.29%
unixBench	+1.19%	       +1.58%
overall average	+3.27%	       +3.28%


hackbench:
group   base  smt_buddy relax   smt_buddy_pct  relax_pct
g1	0.262	0.215	0.202	+17.79%	        +22.82%
g2	0.288	0.322	0.334	-11.81%	        -16.12%
g4	0.407	0.421	0.494	-3.29%	        -21.26%
g8	0.567	0.613	0.756	-7.97%	        -33.17%
g16	0.836	1.100	1.103	-31.61%	        -31.99%
g32	1.411	1.907	1.646	-35.15%	        -16.63%
g64	2.473	3.077	3.204	-24.44%	        -29.58%
g128	4.502	5.538	5.190	-23.01%	        -15.28%


tbench:
clients   base        smt_buddy         relax      smt_buddy_pct  relax_pct
1	475.97	       493.30	        489.21	        +3.64%	  +2.78%
2	714.48	       680.53	        853.52	        -4.75%	  +19.46%
4	1349.73	       1569.21	        1493.59	        +16.26%	  +10.66%
8	2059.00	       2240.22	        2285.86	        +8.80%	  +11.02%
16	4364.25	       4326.11	        3381.46	        -0.87%	  -22.52%
32	8325.17	       7740.42	        7052.11	        -7.02%	  -15.29%
64	12614.20       12955.47	        14898.40	+2.71%	  +18.11%
128	15845.07       22412.27	        19323.67	+41.45%	  +21.95%
256	20837.93       30456.97	        29053.70	+46.16%	  +39.43%
512	29820.67       32227.33	        28809.13	+8.07%	  -3.39%
1024	25969.93       32078.90	        29487.20	+23.52%	  +13.54%


schbench:
threads   base        smt_buddy         relax      smt_buddy_pct  relax_pct
t1	10.75	      9.20	        9.20	       +14.42%	  +14.42%
t2	11.20	      13.00	        10.80	       -16.07%	  +3.57%
t4	12.80	      14.80	        13.00	       -15.63%	  -1.56%
t8	13.80	      14.20	        15.60	       -2.90%	  -13.04%
t16	14.80	      16.20	        14.80	       -9.46%	  0.00%
t32	17.00	      17.00	        16.60	       0.00%	  +2.35%
t64	27.20	      1713.20	        1139.00	       abnormal   abnormal
t128	2844.00	      2855.20	        2856.80	       -0.39%	  -0.45%
t256	5656.00	      5668.80	        5672.00	       -0.23%	  -0.28%
t512	9680.00	      10467.20	        9744.00	       -8.13%	  -0.66%


STREAM-10:
operation base        smt_buddy         relax      smt_buddy_pct  relax_pct
Copy	  103034.5	114194.9	120734.5	+10.83%	   +17.18%
Scale	  100012.0	113639.7	119969.1	+13.63%	   +19.96%
Add	  128469.0	134292.8	140613.4	+4.53%	   +9.45%
Triad	  114401.2	138207.1	138510.9	+20.81%	   +21.07%


STREAM-100:
operation base        smt_buddy         relax      smt_buddy_pct  relax_pct
Copy	  122156.6	176227.5	167672.5	+44.26%	   +37.26%
Scale	  127074.8	167997.0	158140.2	+32.20%	   +24.45%
Add	  140021.5	165604.5	175501.2	+18.27%	   +25.34%
Triad	  136455.4	177236.9	179608.7	+29.89%	   +31.63%


unixbench:
cpu     base        smt_buddy   relax      smt_buddy_pct  relax_pct
1	1808.3	    1835.5	1854.8	    +1.50%	   +2.57%
384     113834.2    114835.7	114505.4    +0.88%	   +0.59%


-- 
thanks,
kunyu