* [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression
@ 2025-12-18 4:59 kernel test robot
2025-12-18 8:37 ` Peter Zijlstra
0 siblings, 1 reply; 7+ messages in thread
From: kernel test robot @ 2025-12-18 4:59 UTC (permalink / raw)
To: Peter Zijlstra
Cc: oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Linus Torvalds,
Dietmar Eggemann, Juri Lelli, Mel Gorman, Shrikanth Hegde,
Valentin Schneider, Vincent Guittot, aubrey.li, yu.c.chen,
oliver.sang
Hello,
kernel test robot noticed a 52.4% regression of pts.schbench.32.usec,_99.9th_latency_percentile on:
commit: 089d84203ad42bc8fd6dbf41683e162ac6e848cd ("sched/fair: Fold the sched_avg update")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core
testcase: pts
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
parameters:
test: schbench-1.1.0
option_a: 32
option_b: 2
cpufreq_governor: performance
In addition to that, the commit also has significant impact on the following tests:
+------------------+-----------------------------------------------------------------------------------------------+
| testcase: change | pts: pts.stress-ng.Semaphores.bogo_ops_s 17.0% improvement |
| test machine | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory |
| test parameters | cpufreq_governor=performance |
| | option_a=Semaphores |
| | test=stress-ng-1.11.0 |
+------------------+-----------------------------------------------------------------------------------------------+
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202512181208.753b9f6e-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251218/202512181208.753b9f6e-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/option_b/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/32/2/debian-12-x86_64-phoronix/lkp-csl-2sp7/schbench-1.1.0/pts
commit:
38a68b982d ("<linux/compiler_types.h>: Add the __signed_scalar_typeof() helper")
089d84203a ("sched/fair: Fold the sched_avg update")
38a68b982dd0b10e 089d84203ad42bc8fd6dbf41683
---------------- ---------------------------
%stddev %change %stddev
\ | \
22925 -8.7% 20922 vmstat.system.cs
9350 ± 3% +66.2% 15536 pts.schbench.32.usec,_50.0th_latency_percentile
15508 ± 2% +68.7% 26163 pts.schbench.32.usec,_75.0th_latency_percentile
21587 ± 2% +66.3% 35891 pts.schbench.32.usec,_90.0th_latency_percentile
45209 ± 8% +52.4% 68896 ± 2% pts.schbench.32.usec,_99.9th_latency_percentile
836593 -13.5% 723826 pts.time.involuntary_context_switches
263006 -5.9% 247368 pts.time.voluntary_context_switches
19.44 ± 9% -1.9 17.56 ± 10% perf-stat.i.cache-miss-rate%
23851 -10.1% 21445 perf-stat.i.context-switches
2482 ± 3% +55.0% 3848 perf-stat.i.cpu-migrations
58297 ± 6% +22.2% 71251 ± 5% perf-stat.i.cycles-between-cache-misses
4.28 +1.9% 4.37 perf-stat.i.major-faults
459621 ± 4% -10.2% 412737 ± 6% perf-stat.i.node-load-misses
59.21 ± 2% -3.7 55.48 ± 3% perf-stat.i.node-store-miss-rate%
23548 -9.1% 21411 perf-stat.ps.context-switches
2453 ± 3% +57.1% 3853 perf-stat.ps.cpu-migrations
451933 ± 4% -10.2% 405766 ± 6% perf-stat.ps.node-load-misses
3.80 ±101% -2.4 1.37 ±158% perf-profile.calltrace.cycles-pp.shmem_alloc_and_add_folio.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter
3.80 ±101% -2.4 1.37 ±158% perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter.vfs_write
3.80 ±101% -2.4 1.37 ±158% perf-profile.calltrace.cycles-pp.shmem_write_begin.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write
5.04 ± 89% -2.2 2.88 ±136% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.04 ± 89% -2.2 2.88 ±136% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call.do_syscall_64
5.04 ± 89% -2.2 2.88 ±136% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.04 ± 89% -2.2 2.88 ±136% perf-profile.calltrace.cycles-pp.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.80 ±101% -2.4 1.37 ±158% perf-profile.children.cycles-pp.shmem_alloc_and_add_folio
3.80 ±101% -2.4 1.37 ±158% perf-profile.children.cycles-pp.shmem_get_folio_gfp
3.80 ±101% -2.4 1.37 ±158% perf-profile.children.cycles-pp.shmem_write_begin
5.04 ± 89% -2.2 2.88 ±136% perf-profile.children.cycles-pp.__x64_sys_exit_group
5.04 ± 89% -2.2 2.88 ±136% perf-profile.children.cycles-pp.x64_sys_call
3398 ± 14% +45.6% 4947 ± 16% sched_debug.cfs_rq:/.avg_vruntime.stddev
10.87 ± 38% +268.9% 40.11 ± 40% sched_debug.cfs_rq:/.load_avg.avg
195.51 ± 8% -32.1% 132.77 ± 19% sched_debug.cfs_rq:/.runnable_avg.avg
195.49 ± 8% -32.1% 132.67 ± 19% sched_debug.cfs_rq:/.util_avg.avg
16931 ± 18% +59.7% 27048 ± 19% sched_debug.cfs_rq:/.zero_vruntime.max
2742 ± 11% +48.8% 4080 ± 9% sched_debug.cfs_rq:/.zero_vruntime.stddev
9.71 ± 10% +1542.6% 159.44 ± 41% sched_debug.cfs_rq:/system.slice.load_avg.avg
49.20 ± 29% +1273.6% 675.80 ± 24% sched_debug.cfs_rq:/system.slice.load_avg.max
10.96 ± 9% +1016.0% 122.30 ± 26% sched_debug.cfs_rq:/system.slice.load_avg.stddev
230.81 ± 9% -33.2% 154.14 ± 20% sched_debug.cfs_rq:/system.slice.runnable_avg.avg
8.42 ± 49% -55.9% 3.71 ± 14% sched_debug.cfs_rq:/system.slice.se->avg.load_avg.avg
180.50 ± 57% -79.6% 36.90 ± 46% sched_debug.cfs_rq:/system.slice.se->avg.load_avg.max
22.27 ± 48% -71.5% 6.34 ± 22% sched_debug.cfs_rq:/system.slice.se->avg.load_avg.stddev
230.83 ± 9% -33.3% 153.85 ± 20% sched_debug.cfs_rq:/system.slice.se->avg.runnable_avg.avg
230.83 ± 9% -33.4% 153.79 ± 20% sched_debug.cfs_rq:/system.slice.se->avg.util_avg.avg
3241 ± 31% -81.3% 607.00 ± 25% sched_debug.cfs_rq:/system.slice.se->load.weight.min
1893 ± 40% +627.5% 13772 ± 35% sched_debug.cfs_rq:/system.slice.tg_load_avg.avg
2803 ± 30% +556.7% 18406 ± 25% sched_debug.cfs_rq:/system.slice.tg_load_avg.max
1715 ± 44% +604.5% 12086 ± 37% sched_debug.cfs_rq:/system.slice.tg_load_avg.min
231.68 ± 59% +643.9% 1723 ± 29% sched_debug.cfs_rq:/system.slice.tg_load_avg.stddev
22.27 ± 41% +641.4% 165.13 ± 38% sched_debug.cfs_rq:/system.slice.tg_load_avg_contrib.avg
230.81 ± 9% -33.2% 154.07 ± 20% sched_debug.cfs_rq:/system.slice.util_avg.avg
0.03 ± 33% -51.4% 0.02 ± 24% perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
0.00 ± 31% +569.0% 0.03 ± 27% perf-sched.sch_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
0.24 ± 18% +43.3% 0.34 ± 7% perf-sched.sch_delay.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown]
0.24 ± 17% +40.8% 0.34 ± 5% perf-sched.sch_delay.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
0.01 ± 13% +313.3% 0.04 ± 25% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
25.39 ± 24% +103.1% 51.58 ± 35% perf-sched.sch_delay.max.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
42.11 ± 55% +115.8% 90.86 ± 40% perf-sched.sch_delay.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown]
36.90 ± 8% +134.0% 86.33 ± 35% perf-sched.sch_delay.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
29.57 ± 17% +94.8% 57.62 ± 46% perf-sched.sch_delay.max.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown]
27.31 ± 18% +98.5% 54.21 ± 49% perf-sched.sch_delay.max.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown].[unknown]
1.28 ± 54% +617.0% 9.21 ± 82% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
0.15 ± 15% +30.9% 0.20 ± 5% perf-sched.total_sch_delay.average.ms
24.08 ± 3% +12.1% 27.00 perf-sched.total_wait_and_delay.average.ms
23.93 ± 3% +12.0% 26.80 perf-sched.total_wait_time.average.ms
12.02 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
8.86 ± 9% -33.9% 5.86 ± 3% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
9.27 ± 6% +37.9% 12.79 ± 5% perf-sched.wait_and_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
23.08 ± 2% +19.1% 27.49 ± 2% perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown]
23.25 ± 2% +18.7% 27.60 perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
24.82 ± 11% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown]
24.47 ± 12% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown].[unknown]
27.22 ± 7% +14.8% 31.24 ± 2% perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown]
27.68 ± 7% +14.5% 31.70 ± 2% perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown].[unknown]
561.21 +15.2% 646.76 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
1151 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
9330 ± 12% +85.2% 17277 ± 3% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
49629 ± 5% -15.4% 41983 ± 3% perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown]
48187 ± 5% -15.4% 40747 ± 3% perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
4019 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_call_function_single.[unknown]
3792 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_call_function_single.[unknown].[unknown]
10992 ± 4% -9.9% 9903 ± 2% perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown]
183.81 ±149% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
150.43 ± 9% +39.7% 210.17 ± 4% perf-sched.wait_and_delay.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown]
138.97 ± 7% +48.9% 206.98 ± 6% perf-sched.wait_and_delay.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
120.74 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown]
117.76 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown].[unknown]
1.04 ± 31% -71.6% 0.29 ± 44% perf-sched.wait_time.avg.ms.__cond_resched.dput.part.0.__fput
8.83 ± 8% -33.9% 5.84 ± 3% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
9.27 ± 6% +37.7% 12.76 ± 5% perf-sched.wait_time.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
22.84 ± 2% +18.9% 27.15 ± 2% perf-sched.wait_time.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown]
23.01 +18.5% 27.26 perf-sched.wait_time.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
27.09 ± 7% +14.7% 31.07 ± 2% perf-sched.wait_time.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown]
27.55 ± 7% +14.5% 31.53 ± 2% perf-sched.wait_time.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown].[unknown]
561.20 +15.2% 646.75 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
131.24 ± 9% +37.8% 180.91 ± 6% perf-sched.wait_time.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown]
128.37 ± 7% +41.2% 181.24 ± 7% perf-sched.wait_time.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
115.53 ± 7% +17.9% 136.21 ± 7% perf-sched.wait_time.max.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown]
112.57 ± 5% +47.8% 166.40 ± 67% perf-sched.wait_time.max.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown].[unknown]
7.69 ± 33% +6561.5% 512.09 ±234% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/Semaphores/debian-12-x86_64-phoronix/lkp-csl-2sp7/stress-ng-1.11.0/pts
commit:
38a68b982d ("<linux/compiler_types.h>: Add the __signed_scalar_typeof() helper")
089d84203a ("sched/fair: Fold the sched_avg update")
38a68b982dd0b10e 089d84203ad42bc8fd6dbf41683
---------------- ---------------------------
%stddev %change %stddev
\ | \
0.94 ± 12% +0.1 1.05 ± 11% mpstat.cpu.all.irq%
22175775 ± 2% -29.7% 15587298 ± 2% vmstat.system.cs
43933492 +17.0% 51381109 pts.stress-ng.Semaphores.bogo_ops_s
1.201e+09 -29.9% 8.415e+08 ± 2% pts.time.involuntary_context_switches
2047 -2.8% 1990 pts.time.system_time
774.22 +6.9% 827.70 pts.time.user_time
18302 ± 19% +182.0% 51617 ± 19% pts.time.voluntary_context_switches
1.385e+08 +9.9% 1.522e+08 perf-stat.i.branch-misses
23793135 ± 2% -29.7% 16716565 ± 2% perf-stat.i.context-switches
165.66 ± 4% +49.0% 246.91 perf-stat.i.cpu-migrations
2.678e+10 -3.7% 2.58e+10 perf-stat.i.dTLB-loads
1.557e+10 -3.4% 1.504e+10 perf-stat.i.dTLB-stores
21025195 +24.2% 26111221 ± 2% perf-stat.i.iTLB-load-misses
1.017e+11 -2.9% 9.874e+10 perf-stat.i.instructions
32500 ± 30% -51.7% 15693 ± 10% perf-stat.i.instructions-per-iTLB-miss
686.98 -2.6% 668.88 perf-stat.i.metric.M/sec
0.59 +0.1 0.65 perf-stat.overall.branch-miss-rate%
1.50 +3.9% 1.56 perf-stat.overall.cpi
4831 ± 2% -21.8% 3780 ± 2% perf-stat.overall.instructions-per-iTLB-miss
0.67 -3.8% 0.64 perf-stat.overall.ipc
1.358e+08 +9.9% 1.493e+08 perf-stat.ps.branch-misses
23332701 ± 2% -29.7% 16391899 ± 2% perf-stat.ps.context-switches
162.46 ± 4% +49.1% 242.15 perf-stat.ps.cpu-migrations
2.627e+10 -3.7% 2.53e+10 perf-stat.ps.dTLB-loads
1.527e+10 -3.4% 1.475e+10 perf-stat.ps.dTLB-stores
20659015 +24.1% 25643823 ± 2% perf-stat.ps.iTLB-load-misses
9.977e+10 -2.9% 9.686e+10 perf-stat.ps.instructions
5.136e+12 -3.2% 4.973e+12 perf-stat.total.instructions
6.19 ±141% +614.4% 44.23 ± 67% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
3.01 ±222% +936.1% 31.16 ± 92% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
4.75 ± 67% -89.4% 0.50 ± 88% perf-sched.sch_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
0.01 ± 14% +25.7% 0.01 ± 19% perf-sched.sch_delay.avg.ms.irq_wait_for_interrupt.irq_thread.kthread.ret_from_fork.ret_from_fork_asm
27.62 ±134% -97.8% 0.60 ±219% perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
4.19 ±217% +514.4% 25.74 ±112% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone.__x64_sys_vfork
122.79 ± 13% +312.2% 506.13 ± 32% perf-sched.sch_delay.max.ms.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
2181 ± 41% -38.5% 1340 ± 35% perf-sched.sch_delay.max.ms.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read.do_syscall_64
2680 ± 41% -62.5% 1004 ± 99% perf-sched.sch_delay.max.ms.anon_pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
333.74 ±141% +199.8% 1000 perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
167.85 ±223% +598.7% 1172 ± 76% perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
245.85 ±130% -97.6% 5.86 ±222% perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm
0.05 ± 5% +52.0% 0.08 ± 2% perf-sched.total_wait_and_delay.average.ms
20881143 ± 3% -32.1% 14170213 ± 2% perf-sched.total_wait_and_delay.count.ms
0.05 ± 4% +52.8% 0.08 ± 3% perf-sched.total_wait_time.average.ms
0.02 ± 3% +56.0% 0.04 ± 2% perf-sched.wait_and_delay.avg.ms.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
9.51 ±209% +745.8% 80.40 ± 54% perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
30.35 ± 50% -71.3% 8.70 ± 28% perf-sched.wait_and_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
20874873 ± 3% -32.2% 14163492 ± 2% perf-sched.wait_and_delay.count.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
165.00 ± 90% +378.6% 789.67 ± 30% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
382.50 ± 28% +91.8% 733.67 ± 12% perf-sched.wait_and_delay.count.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
503.33 ± 49% +105.4% 1033 ± 24% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.do_epoll_pwait.part
85.17 ± 24% +85.3% 157.83 ± 28% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread.ret_from_fork
514.37 ± 18% +192.2% 1502 ± 24% perf-sched.wait_and_delay.max.ms.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
5701 ± 19% -32.4% 3852 ± 23% perf-sched.wait_and_delay.max.ms.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read.do_syscall_64
5530 ± 34% -45.5% 3015 ± 47% perf-sched.wait_and_delay.max.ms.anon_pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.01 ± 18% +230.2% 0.05 ± 86% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write
0.02 ± 3% +55.3% 0.04 ± 2% perf-sched.wait_time.avg.ms.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
0.01 ± 18% +56.3% 0.02 ± 26% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
25.61 ± 50% -68.0% 8.20 ± 26% perf-sched.wait_time.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex
489.21 ± 21% +171.2% 1326 ± 28% perf-sched.wait_time.max.ms.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown].[unknown]
1185 ± 26% -15.6% 1000 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
0.02 ± 20% +300.7% 0.10 ± 98% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
1281 ± 10% +37.3% 1759 ± 10% sched_debug.cfs_rq:/.avg_vruntime.avg
3255 ± 16% +39.8% 4549 ± 9% sched_debug.cfs_rq:/.avg_vruntime.stddev
10.24 ± 50% +266.8% 37.54 ± 44% sched_debug.cfs_rq:/.load_avg.avg
1086 ± 13% +41.4% 1536 ± 7% sched_debug.cfs_rq:/.zero_vruntime.avg
16641 ± 20% +71.5% 28532 ± 16% sched_debug.cfs_rq:/.zero_vruntime.max
2628 ± 7% +55.5% 4088 ± 8% sched_debug.cfs_rq:/.zero_vruntime.stddev
9.57 ± 13% +1560.4% 158.98 ± 43% sched_debug.cfs_rq:/system.slice.load_avg.avg
54.00 ± 7% +1167.0% 684.17 ± 40% sched_debug.cfs_rq:/system.slice.load_avg.max
10.89 ± 10% +1043.2% 124.50 ± 39% sched_debug.cfs_rq:/system.slice.load_avg.stddev
3621 ± 24% -82.5% 633.83 ± 25% sched_debug.cfs_rq:/system.slice.se->load.weight.min
1382 ± 10% +38.3% 1912 ± 11% sched_debug.cfs_rq:/system.slice.se->vruntime.avg
3453 ± 15% +39.3% 4812 ± 10% sched_debug.cfs_rq:/system.slice.se->vruntime.stddev
1411 ± 54% +818.0% 12960 ± 38% sched_debug.cfs_rq:/system.slice.tg_load_avg.avg
1983 ± 44% +699.0% 15847 ± 40% sched_debug.cfs_rq:/system.slice.tg_load_avg.max
1244 ± 59% +823.7% 11495 ± 38% sched_debug.cfs_rq:/system.slice.tg_load_avg.min
18.22 ± 45% +791.4% 162.37 ± 41% sched_debug.cfs_rq:/system.slice.tg_load_avg_contrib.avg
0.39 ± 40% +163.4% 1.02 ± 17% sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.avg
1.83 ± 37% +118.2% 4.00 ± 25% sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.max
0.70 ± 33% +111.6% 1.48 ± 17% sched_debug.cfs_rq:/system.slice/containerd.service.load_avg.stddev
0.44 ± 28% +146.7% 1.09 ± 16% sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.avg
2.00 ± 28% +108.3% 4.17 ± 25% sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.max
0.77 ± 23% +105.5% 1.57 ± 19% sched_debug.cfs_rq:/system.slice/containerd.service.runnable_avg.stddev
0.29 ± 53% +224.9% 0.94 ± 12% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.load_avg.avg
0.63 ± 54% +119.4% 1.38 ± 28% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.load_avg.stddev
0.44 ± 28% +135.1% 1.04 ± 18% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.avg
2.00 ± 28% +100.0% 4.00 ± 25% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.max
0.77 ± 23% +95.4% 1.50 ± 18% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.runnable_avg.stddev
0.44 ± 28% +135.1% 1.04 ± 18% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.util_avg.avg
2.00 ± 28% +100.0% 4.00 ± 25% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.util_avg.max
0.77 ± 23% +95.4% 1.50 ± 18% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.util_avg.stddev
2.50 ± 47% +133.5% 5.83 ± 28% sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.avg
3.00 ± 50% +105.6% 6.17 ± 28% sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.max
2.17 ± 49% +161.5% 5.67 ± 29% sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg.min
0.39 ± 40% +163.4% 1.02 ± 17% sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.avg
1.83 ± 37% +118.2% 4.00 ± 25% sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.max
0.70 ± 33% +111.6% 1.48 ± 17% sched_debug.cfs_rq:/system.slice/containerd.service.tg_load_avg_contrib.stddev
0.44 ± 28% +146.7% 1.09 ± 16% sched_debug.cfs_rq:/system.slice/containerd.service.util_avg.avg
2.00 ± 28% +108.3% 4.17 ± 25% sched_debug.cfs_rq:/system.slice/containerd.service.util_avg.max
0.77 ± 23% +105.5% 1.57 ± 19% sched_debug.cfs_rq:/system.slice/containerd.service.util_avg.stddev
7.74 ± 3% -39.7% 4.67 ± 35% sched_debug.cpu.clock.stddev
1495 ± 4% +8.4% 1620 ± 5% sched_debug.cpu.nr_switches.avg
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression 2025-12-18 4:59 [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression kernel test robot @ 2025-12-18 8:37 ` Peter Zijlstra 2025-12-18 10:11 ` Shrikanth Hegde 0 siblings, 1 reply; 7+ messages in thread From: Peter Zijlstra @ 2025-12-18 8:37 UTC (permalink / raw) To: kernel test robot Cc: oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Linus Torvalds, Dietmar Eggemann, Juri Lelli, Mel Gorman, Shrikanth Hegde, Valentin Schneider, Vincent Guittot, aubrey.li, yu.c.chen On Thu, Dec 18, 2025 at 12:59:53PM +0800, kernel test robot wrote: > > > Hello, > > kernel test robot noticed a 52.4% regression of pts.schbench.32.usec,_99.9th_latency_percentile on: > > > commit: 089d84203ad42bc8fd6dbf41683e162ac6e848cd ("sched/fair: Fold the sched_avg update") > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core Well, that obviously wasn't the intention. Let me pull that patch :/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression 2025-12-18 8:37 ` Peter Zijlstra @ 2025-12-18 10:11 ` Shrikanth Hegde 2025-12-18 10:20 ` Peter Zijlstra 0 siblings, 1 reply; 7+ messages in thread From: Shrikanth Hegde @ 2025-12-18 10:11 UTC (permalink / raw) To: Peter Zijlstra, kernel test robot Cc: oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Linus Torvalds, Dietmar Eggemann, Juri Lelli, Mel Gorman, Valentin Schneider, Vincent Guittot, aubrey.li, yu.c.chen On 12/18/25 2:07 PM, Peter Zijlstra wrote: > On Thu, Dec 18, 2025 at 12:59:53PM +0800, kernel test robot wrote: >> >> >> Hello, >> >> kernel test robot noticed a 52.4% regression of pts.schbench.32.usec,_99.9th_latency_percentile on: >> >> >> commit: 089d84203ad42bc8fd6dbf41683e162ac6e848cd ("sched/fair: Fold the sched_avg update") >> https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core > > Well, that obviously wasn't the intention. Let me pull that patch :/ Is it possible because it missed scaling by se_weight(se) ?? +#define __update_sa(sa, name, delta_avg, delta_sum) do { \ + add_positive(&(sa)->name##_avg, delta_avg); \ + add_positive(&(sa)->name##_sum, delta_sum); \ + (sa)->name##_sum = max_t(typeof((sa)->name##_sum), \ + (sa)->name##_sum, \ + (sa)->name##_avg * PELT_MIN_DIVIDER); \ +} while (0) + static inline void enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { - cfs_rq->avg.load_avg += se->avg.load_avg; - cfs_rq->avg.load_sum += se_weight(se) * se->avg.load_sum; + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); } ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression 2025-12-18 10:11 ` Shrikanth Hegde @ 2025-12-18 10:20 ` Peter Zijlstra 2025-12-18 10:37 ` Vincent Guittot ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Peter Zijlstra @ 2025-12-18 10:20 UTC (permalink / raw) To: Shrikanth Hegde Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Linus Torvalds, Dietmar Eggemann, Juri Lelli, Mel Gorman, Valentin Schneider, Vincent Guittot, aubrey.li, yu.c.chen On Thu, Dec 18, 2025 at 03:41:55PM +0530, Shrikanth Hegde wrote: > On 12/18/25 2:07 PM, Peter Zijlstra wrote: > > On Thu, Dec 18, 2025 at 12:59:53PM +0800, kernel test robot wrote: > > > > > > > > > Hello, > > > > > > kernel test robot noticed a 52.4% regression of pts.schbench.32.usec,_99.9th_latency_percentile on: > > > > > > > > > commit: 089d84203ad42bc8fd6dbf41683e162ac6e848cd ("sched/fair: Fold the sched_avg update") > > > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core > > > > Well, that obviously wasn't the intention. Let me pull that patch :/ > > Is it possible because it missed scaling by se_weight(se) ?? > static inline void > enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) > { > - cfs_rq->avg.load_avg += se->avg.load_avg; > - cfs_rq->avg.load_sum += se_weight(se) * se->avg.load_sum; > + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); > } Ah, indeed, something like so then? Can the robot (Oliver/Philip) verify? (I was going to shelf it and look at it after the holidays, but if this is it, we can get it fixed before I dissapear). --- diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 76f5e4b78b30..7377f9117501 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3775,13 +3775,15 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) static inline void enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { - __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, + se_weight(se) * se->avg.load_sum); } static inline void dequeue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { - __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, -se->avg.load_sum); + __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, + se_weight(se) * -se->avg.load_sum); } static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags); ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression 2025-12-18 10:20 ` Peter Zijlstra @ 2025-12-18 10:37 ` Vincent Guittot 2025-12-19 8:16 ` [tip: sched/core] sched/fair: Fix sched_avg fold tip-bot2 for Peter Zijlstra 2025-12-21 9:37 ` [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression Oliver Sang 2 siblings, 0 replies; 7+ messages in thread From: Vincent Guittot @ 2025-12-18 10:37 UTC (permalink / raw) To: Peter Zijlstra Cc: Shrikanth Hegde, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Linus Torvalds, Dietmar Eggemann, Juri Lelli, Mel Gorman, Valentin Schneider, aubrey.li, yu.c.chen On Thu, 18 Dec 2025 at 11:20, Peter Zijlstra <peterz@infradead.org> wrote: > > On Thu, Dec 18, 2025 at 03:41:55PM +0530, Shrikanth Hegde wrote: > > On 12/18/25 2:07 PM, Peter Zijlstra wrote: > > > On Thu, Dec 18, 2025 at 12:59:53PM +0800, kernel test robot wrote: > > > > > > > > > > > > Hello, > > > > > > > > kernel test robot noticed a 52.4% regression of pts.schbench.32.usec,_99.9th_latency_percentile on: > > > > > > > > > > > > commit: 089d84203ad42bc8fd6dbf41683e162ac6e848cd ("sched/fair: Fold the sched_avg update") > > > > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core > > > > > > Well, that obviously wasn't the intention. Let me pull that patch :/ > > > > Is it possible because it missed scaling by se_weight(se) ?? > > > static inline void > > enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) > > { > > - cfs_rq->avg.load_avg += se->avg.load_avg; > > - cfs_rq->avg.load_sum += se_weight(se) * se->avg.load_sum; > > + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); > > } > > Ah, indeed, something like so then? Can the robot (Oliver/Philip) > verify? yes, rq tracks the weighted sum whereas se tracks the unweighted sum > > (I was going to shelf it and look at it after the holidays, but if this > is it, we can get it fixed before I dissapear). > > --- > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 76f5e4b78b30..7377f9117501 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -3775,13 +3775,15 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) > static inline void > enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) > { > - __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); > + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, > + se_weight(se) * se->avg.load_sum); > } > > static inline void > dequeue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) > { > - __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, -se->avg.load_sum); > + __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, > + se_weight(se) * -se->avg.load_sum); > } > > static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags); ^ permalink raw reply [flat|nested] 7+ messages in thread
* [tip: sched/core] sched/fair: Fix sched_avg fold 2025-12-18 10:20 ` Peter Zijlstra 2025-12-18 10:37 ` Vincent Guittot @ 2025-12-19 8:16 ` tip-bot2 for Peter Zijlstra 2025-12-21 9:37 ` [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression Oliver Sang 2 siblings, 0 replies; 7+ messages in thread From: tip-bot2 for Peter Zijlstra @ 2025-12-19 8:16 UTC (permalink / raw) To: linux-tip-commits Cc: kernel test robot, Peter Zijlstra (Intel), x86, linux-kernel The following commit has been merged into the sched/core branch of tip: Commit-ID: 6ab7973f254071faf20fe5fcc502a3fe9ca14a47 Gitweb: https://git.kernel.org/tip/6ab7973f254071faf20fe5fcc502a3fe9ca14a47 Author: Peter Zijlstra <peterz@infradead.org> AuthorDate: Fri, 19 Dec 2025 09:04:45 +01:00 Committer: Peter Zijlstra <peterz@infradead.org> CommitterDate: Fri, 19 Dec 2025 09:09:38 +01:00 sched/fair: Fix sched_avg fold After the robot reported a regression wrt commit: 089d84203ad4 ("sched/fair: Fold the sched_avg update"), Shrikanth noted that two spots missed a factor se_weight(). Fixes: 089d84203ad4 ("sched/fair: Fold the sched_avg update") Reported-by: kernel test robot <oliver.sang@intel.com> Closes: https://lore.kernel.org/oe-lkp/202512181208.753b9f6e-lkp@intel.com Debugged-by: Shrikanth Hegde <sshegde@linux.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20251218102020.GO3707891@noisy.programming.kicks-ass.net --- kernel/sched/fair.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 76f5e4b..7377f91 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3775,13 +3775,15 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) static inline void enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { - __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, + se_weight(se) * se->avg.load_sum); } static inline void dequeue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) { - __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, -se->avg.load_sum); + __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, + se_weight(se) * -se->avg.load_sum); } static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags); ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression 2025-12-18 10:20 ` Peter Zijlstra 2025-12-18 10:37 ` Vincent Guittot 2025-12-19 8:16 ` [tip: sched/core] sched/fair: Fix sched_avg fold tip-bot2 for Peter Zijlstra @ 2025-12-21 9:37 ` Oliver Sang 2 siblings, 0 replies; 7+ messages in thread From: Oliver Sang @ 2025-12-21 9:37 UTC (permalink / raw) To: Peter Zijlstra Cc: Shrikanth Hegde, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Linus Torvalds, Dietmar Eggemann, Juri Lelli, Mel Gorman, Valentin Schneider, Vincent Guittot, aubrey.li, yu.c.chen, oliver.sang hi, Peter Zijlstra, On Thu, Dec 18, 2025 at 11:20:20AM +0100, Peter Zijlstra wrote: > On Thu, Dec 18, 2025 at 03:41:55PM +0530, Shrikanth Hegde wrote: > > On 12/18/25 2:07 PM, Peter Zijlstra wrote: > > > On Thu, Dec 18, 2025 at 12:59:53PM +0800, kernel test robot wrote: > > > > > > > > > > > > Hello, > > > > > > > > kernel test robot noticed a 52.4% regression of pts.schbench.32.usec,_99.9th_latency_percentile on: > > > > > > > > > > > > commit: 089d84203ad42bc8fd6dbf41683e162ac6e848cd ("sched/fair: Fold the sched_avg update") > > > > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core > > > > > > Well, that obviously wasn't the intention. Let me pull that patch :/ > > > > Is it possible because it missed scaling by se_weight(se) ?? > > > static inline void > > enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) > > { > > - cfs_rq->avg.load_avg += se->avg.load_avg; > > - cfs_rq->avg.load_sum += se_weight(se) * se->avg.load_sum; > > + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); > > } > > Ah, indeed, something like so then? Can the robot (Oliver/Philip) > verify? sorry for late. it happened that the server (Cascade Lake) used for original bisect/report was converted for other usage. so we pick another server: test machine: 256 threads 2 sockets Intel(R) Xeon(R) 6767P CPU @ 2.4GHz (Granite Rapids) with 256G memory to reproduce the regression (another improvement actually) then tested your patch below. based on the results, the performance restored to be similar to 38a68b982d (parent of 089d84203a) (both regression and improvement have lower pecentage on this Granite Rapids server) Tested-by: kernel test robot <oliver.sang@intel.com> for the regression ========================================================================================= compiler/cpufreq_governor/kconfig/option_a/option_b/rootfs/tbox_group/test/testcase: gcc-14/performance/x86_64-rhel-9.4/32/2/debian-12-x86_64-phoronix/lkp-gnr-2sp3/schbench-1.1.0/pts commit: 38a68b982d ("<linux/compiler_types.h>: Add the __signed_scalar_typeof() helper") 089d84203a ("sched/fair: Fold the sched_avg update") d936730940 ("sched/fair: Fix sched_avg fold") 38a68b982dd0b10e 089d84203ad42bc8fd6dbf41683 d936730940bfff3f3b22770cfe9 ---------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 18752 +16.0% 21749 -0.1% 18730 pts.schbench.32.usec,_50.0th_latency_percentile 29493 ± 2% +17.3% 34581 +0.3% 29568 pts.schbench.32.usec,_75.0th_latency_percentile 40810 +15.2% 46997 +0.3% 40938 pts.schbench.32.usec,_90.0th_latency_percentile 84437 +7.2% 90496 +1.1% 85376 pts.schbench.32.usec,_99.9th_latency_percentile full comparison is as below [1] BTW, in our original report https://lore.kernel.org/all/202512181208.753b9f6e-lkp@intel.com/ we also reported an improvment still on that Cascade Lake server. +------------------+-----------------------------------------------------------------------------------------------+ | testcase: change | pts: pts.stress-ng.Semaphores.bogo_ops_s 17.0% improvement | | test machine | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory | | test parameters | cpufreq_governor=performance | | | option_a=Semaphores | | | test=stress-ng-1.11.0 | +------------------+-----------------------------------------------------------------------------------------------+ in our tests, the performance also restored by your patch. ========================================================================================= compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase: gcc-14/performance/x86_64-rhel-9.4/Semaphores/debian-12-x86_64-phoronix/lkp-gnr-2sp3/stress-ng-1.11.0/pts commit: 38a68b982d ("<linux/compiler_types.h>: Add the __signed_scalar_typeof() helper") 089d84203a ("sched/fair: Fold the sched_avg update") d936730940 ("sched/fair: Fix sched_avg fold") 38a68b982dd0b10e 089d84203ad42bc8fd6dbf41683 d936730940bfff3f3b22770cfe9 ---------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 3.533e+08 +11.3% 3.934e+08 +1.1% 3.573e+08 pts.stress-ng.Semaphores.bogo_ops_s the full comparison is as below [2] > > (I was going to shelf it and look at it after the holidays, but if this > is it, we can get it fixed before I dissapear). > > --- > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 76f5e4b78b30..7377f9117501 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -3775,13 +3775,15 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) > static inline void > enqueue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) > { > - __update_sa(&cfs_rq->avg, load, se->avg.load_avg, se->avg.load_sum); > + __update_sa(&cfs_rq->avg, load, se->avg.load_avg, > + se_weight(se) * se->avg.load_sum); > } > > static inline void > dequeue_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) > { > - __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, -se->avg.load_sum); > + __update_sa(&cfs_rq->avg, load, -se->avg.load_avg, > + se_weight(se) * -se->avg.load_sum); > } > > static void place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags); [1] ========================================================================================= compiler/cpufreq_governor/kconfig/option_a/option_b/rootfs/tbox_group/test/testcase: gcc-14/performance/x86_64-rhel-9.4/32/2/debian-12-x86_64-phoronix/lkp-gnr-2sp3/schbench-1.1.0/pts commit: 38a68b982d ("<linux/compiler_types.h>: Add the __signed_scalar_typeof() helper") 089d84203a ("sched/fair: Fold the sched_avg update") d936730940 ("sched/fair: Fix sched_avg fold") 38a68b982dd0b10e 089d84203ad42bc8fd6dbf41683 d936730940bfff3f3b22770cfe9 ---------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 0.06 ± 2% +0.0 0.07 -0.0 0.06 ± 2% mpstat.cpu.all.soft% 58411 -7.1% 54274 -0.3% 58244 vmstat.system.cs 291141 -1.2% 287740 -0.5% 289788 vmstat.system.in 2018606 -9.1% 1834308 -0.1% 2015969 time.involuntary_context_switches 7153 -1.0% 7079 +0.1% 7162 time.user_time 607814 -3.1% 589005 -0.0% 607669 time.voluntary_context_switches 18752 +16.0% 21749 -0.1% 18730 pts.schbench.32.usec,_50.0th_latency_percentile 29493 ± 2% +17.3% 34581 +0.3% 29568 pts.schbench.32.usec,_75.0th_latency_percentile 40810 +15.2% 46997 +0.3% 40938 pts.schbench.32.usec,_90.0th_latency_percentile 84437 +7.2% 90496 +1.1% 85376 pts.schbench.32.usec,_99.9th_latency_percentile 2018606 -9.1% 1834308 -0.1% 2015969 pts.time.involuntary_context_switches 7153 -1.0% 7079 +0.1% 7162 pts.time.user_time 607814 -3.1% 589005 -0.0% 607669 pts.time.voluntary_context_switches 0.27 ± 4% -0.1% 0.27 ± 5% +7.9% 0.29 ± 2% perf-stat.i.MPKI 17.00 ± 2% -0.8 16.23 +0.4 17.44 ± 2% perf-stat.i.cache-miss-rate% 60336 -7.5% 55810 -0.9% 59784 perf-stat.i.context-switches 5.501e+11 -1.5% 5.419e+11 -0.6% 5.468e+11 perf-stat.i.cpu-cycles 6222 +14.4% 7119 -0.7% 6178 perf-stat.i.cpu-migrations 154698 +3.3% 159875 ± 2% -0.1% 154599 ± 4% perf-stat.i.cycles-between-cache-misses 891574 -3.0% 864626 -1.4% 879513 perf-stat.i.dTLB-store-misses 4.84 ± 5% +6.0% 5.13 +3.2% 4.99 perf-stat.i.major-faults 2.15 -1.6% 2.11 -0.6% 2.13 perf-stat.i.metric.GHz 60586 -6.9% 56433 -0.8% 60112 perf-stat.ps.context-switches 6258 +15.3% 7212 -0.6% 6221 perf-stat.ps.cpu-migrations 888660 -2.5% 866297 -1.2% 877931 perf-stat.ps.dTLB-store-misses 4.49 ± 5% +4.3% 4.68 +3.0% 4.62 perf-stat.ps.major-faults 3.94 ± 92% -3.2 0.71 ±147% -1.9 2.00 ±134% perf-profile.calltrace.cycles-pp.arch_show_interrupts.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read 0.43 ±142% +1.7 2.11 ± 42% +1.6 2.07 ± 61% perf-profile.calltrace.cycles-pp.pv_native_safe_halt.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state 1.08 ± 77% +1.7 2.82 ± 34% +1.7 2.80 ± 77% perf-profile.calltrace.cycles-pp.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 1.08 ± 77% +1.7 2.82 ± 34% +1.7 2.80 ± 77% perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 1.08 ± 77% +1.7 2.82 ± 34% +1.7 2.80 ± 77% perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter 1.08 ± 77% +2.0 3.08 ± 25% +1.7 2.80 ± 77% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 1.08 ± 77% +2.0 3.08 ± 25% +1.7 2.80 ± 77% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 12.35 ± 67% +10.7 23.07 ± 14% -0.1 12.28 ±101% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.09 ± 71% +11.0 23.07 ± 14% +0.2 12.28 ±101% perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.94 ± 92% -3.2 0.71 ±147% -1.9 2.00 ±134% perf-profile.children.cycles-pp.arch_show_interrupts 0.24 ±223% +1.3 1.55 ± 9% +0.2 0.42 ±141% perf-profile.children.cycles-pp.__irq_exit_rcu 0.24 ±223% +1.3 1.55 ± 9% +0.2 0.42 ±141% perf-profile.children.cycles-pp.handle_softirqs 1.08 ± 77% +1.7 2.82 ± 34% +1.7 2.80 ± 77% perf-profile.children.cycles-pp.acpi_idle_do_entry 1.08 ± 77% +1.7 2.82 ± 34% +1.7 2.80 ± 77% perf-profile.children.cycles-pp.acpi_idle_enter 1.08 ± 77% +1.7 2.82 ± 34% +1.7 2.80 ± 77% perf-profile.children.cycles-pp.acpi_safe_halt 1.08 ± 77% +1.7 2.82 ± 34% +1.7 2.80 ± 77% perf-profile.children.cycles-pp.pv_native_safe_halt 1.08 ± 77% +2.0 3.08 ± 25% +1.7 2.80 ± 77% perf-profile.children.cycles-pp.cpuidle_enter 1.08 ± 77% +2.0 3.08 ± 25% +1.7 2.80 ± 77% perf-profile.children.cycles-pp.cpuidle_enter_state 1.24 ±147% +3.2 4.44 ± 37% +0.9 2.14 ± 80% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.02 ± 25% +209.1% 0.05 ± 26% +21.2% 0.02 ± 27% perf-sched.sch_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex 0.25 ± 6% +36.2% 0.34 ± 10% +0.1% 0.25 ± 6% perf-sched.sch_delay.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown] 0.25 ± 7% +35.8% 0.34 ± 9% -0.5% 0.25 ± 6% perf-sched.sch_delay.avg.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 0.21 ± 74% -83.8% 0.03 ±136% +92.6% 0.40 ±145% perf-sched.sch_delay.avg.ms.irqentry_exit.asm_sysvec_call_function.[unknown].[unknown] 0.16 ± 12% +31.2% 0.21 ± 16% +4.4% 0.17 ± 12% perf-sched.sch_delay.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown] 7.88 ± 62% -82.6% 1.38 ±165% +195.7% 23.31 ±174% perf-sched.sch_delay.max.ms.irqentry_exit.asm_sysvec_call_function.[unknown].[unknown] 0.17 ± 7% +37.0% 0.23 ± 13% +0.8% 0.17 ± 8% perf-sched.total_sch_delay.average.ms 144.28 ± 10% +122.9% 321.60 ± 96% +0.6% 145.18 ± 11% perf-sched.total_sch_delay.max.ms 8.72 +32.1% 11.51 -0.3% 8.69 perf-sched.wait_and_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex 25.19 ± 2% +10.4% 27.81 ± 3% +0.4% 25.30 ± 2% perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown] 29.05 +11.9% 32.50 +0.1% 29.06 perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown] 28.93 +12.4% 32.53 +0.0% 28.94 perf-sched.wait_and_delay.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown].[unknown] 120587 -11.1% 107144 -0.4% 120153 perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown] 116894 -11.0% 103989 -0.2% 116681 perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 7275 ± 13% -47.7% 3808 ± 15% -11.6% 6429 ± 12% perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_call_function_single.[unknown] 6801 ± 13% -48.5% 3500 ± 14% -10.9% 6059 ± 12% perf-sched.wait_and_delay.count.irqentry_exit.asm_sysvec_call_function_single.[unknown].[unknown] 811.07 ± 83% -92.9% 57.96 ±106% +7.6% 873.05 ± 93% perf-sched.wait_and_delay.max.ms.__cond_resched.shmem_inode_acct_blocks.shmem_alloc_and_add_folio.shmem_get_folio_gfp.shmem_write_begin 213.79 ± 9% +41.7% 303.02 ± 23% +4.5% 223.47 ± 6% perf-sched.wait_and_delay.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown] 8.27 ± 62% -81.4% 1.54 ±150% -35.3% 5.36 ± 66% perf-sched.wait_time.avg.ms.__cond_resched.migrate_pages_batch.migrate_pages.migrate_misplaced_folio.do_huge_pmd_numa_page 8.70 +31.7% 11.46 -0.3% 8.67 perf-sched.wait_time.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex.__x64_sys_futex 24.73 ± 73% +128.9% 56.61 ± 30% +12.5% 27.81 ± 57% perf-sched.wait_time.avg.ms.irqentry_exit.asm_exc_page_fault.[unknown] 25.06 ± 2% +10.2% 27.61 ± 3% +0.4% 25.17 ± 2% perf-sched.wait_time.avg.ms.irqentry_exit.asm_sysvec_call_function_single.[unknown] 28.88 +11.8% 32.29 +0.0% 28.89 perf-sched.wait_time.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown] 28.76 +12.3% 32.31 +0.0% 28.77 perf-sched.wait_time.avg.ms.irqentry_exit.asm_sysvec_reschedule_ipi.[unknown].[unknown] 0.98 ±140% -99.2% 0.01 ± 11% -32.3% 0.66 ±220% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 203.03 ± 6% +12.8% 229.11 ± 4% +2.9% 208.98 ± 4% perf-sched.wait_time.max.ms.irqentry_exit.asm_sysvec_apic_timer_interrupt.[unknown] 1196 ± 9% +32.8% 1589 ± 14% +10.1% 1317 ± 14% sched_debug.cfs_rq:/.avg_vruntime.avg 18675 ± 15% +48.2% 27668 ± 31% +20.2% 22441 ± 39% sched_debug.cfs_rq:/.avg_vruntime.max 2404 ± 7% +29.7% 3118 ± 13% +8.1% 2599 ± 17% sched_debug.cfs_rq:/.avg_vruntime.stddev 6.09 ± 45% +557.7% 40.05 ± 14% -24.1% 4.62 ± 28% sched_debug.cfs_rq:/.load_avg.avg 6.69 ± 34% -33.3% 4.46 ± 33% -47.7% 3.49 ± 39% sched_debug.cfs_rq:/.util_est.avg 55.43 ± 18% -19.2% 44.80 ± 17% -26.6% 40.69 ± 19% sched_debug.cfs_rq:/.util_est.stddev 1126 ± 8% +30.1% 1464 ± 14% +9.5% 1233 ± 15% sched_debug.cfs_rq:/.zero_vruntime.avg 18675 ± 15% +48.2% 27668 ± 31% +20.2% 22441 ± 39% sched_debug.cfs_rq:/.zero_vruntime.max 2293 ± 7% +23.8% 2840 ± 11% +7.7% 2470 ± 19% sched_debug.cfs_rq:/.zero_vruntime.stddev 4.69 ± 11% +5177.9% 247.68 ± 6% -2.9% 4.56 ± 18% sched_debug.cfs_rq:/system.slice.load_avg.avg 33.33 ± 10% +2518.5% 872.83 ± 14% -3.5% 32.17 ± 10% sched_debug.cfs_rq:/system.slice.load_avg.max 7.52 ± 10% +2180.0% 171.48 ± 6% -1.7% 7.39 ± 14% sched_debug.cfs_rq:/system.slice.load_avg.stddev 24021 ± 43% -53.8% 11098 ± 29% +4.4% 25077 ± 34% sched_debug.cfs_rq:/system.slice.se->load.weight.avg 950.50 ± 17% -86.0% 132.83 ± 22% -3.2% 919.67 ± 23% sched_debug.cfs_rq:/system.slice.se->load.weight.min 1426 ± 14% +39.1% 1984 ± 11% +11.7% 1593 ± 20% sched_debug.cfs_rq:/system.slice.se->vruntime.avg 17114 ± 14% +61.8% 27689 ± 31% +28.8% 22038 ± 41% sched_debug.cfs_rq:/system.slice.se->vruntime.max 2553 ± 7% +45.5% 3714 ± 16% +15.7% 2954 ± 25% sched_debug.cfs_rq:/system.slice.se->vruntime.stddev 2077 ± 52% +1663.6% 36636 ± 8% +14.1% 2369 ± 32% sched_debug.cfs_rq:/system.slice.tg_load_avg.avg 4280 ± 33% +1577.3% 71794 ± 3% +1.4% 4341 ± 23% sched_debug.cfs_rq:/system.slice.tg_load_avg.max 1662 ± 73% +1347.5% 24056 ± 12% +22.7% 2039 ± 32% sched_debug.cfs_rq:/system.slice.tg_load_avg.min 509.03 ± 42% +2013.2% 10756 ± 9% -32.2% 345.01 ± 35% sched_debug.cfs_rq:/system.slice.tg_load_avg.stddev 14.23 ± 60% +1647.1% 248.69 ± 6% +27.0% 18.07 ± 37% sched_debug.cfs_rq:/system.slice.tg_load_avg_contrib.avg 79.49 ± 68% +117.9% 173.24 ± 7% +41.0% 112.06 ± 25% sched_debug.cfs_rq:/system.slice.tg_load_avg_contrib.stddev 0.75 ± 37% -198.7% -0.74 -156.6% -0.43 sched_debug.cfs_rq:/system.slice/containerd.service.avg_vruntime.max 0.73 ± 17% -84.6% 0.11 ±215% -65.3% 0.25 ±140% sched_debug.cfs_rq:/system.slice/containerd.service.avg_vruntime.stddev 1.01 ± 27% -44.6% 0.56 ± 52% -16.0% 0.85 ± 17% sched_debug.cfs_rq:/system.slice/containerd.service.se->avg.load_avg.stddev 0.75 ± 37% -198.7% -0.74 -156.6% -0.43 sched_debug.cfs_rq:/system.slice/containerd.service.zero_vruntime.max 0.73 ± 17% -84.6% 0.11 ±215% -65.3% 0.25 ±140% sched_debug.cfs_rq:/system.slice/containerd.service.zero_vruntime.stddev 281.30 ± 10% +35.7% 381.81 ± 7% -1.9% 276.09 ± 16% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.load_avg.avg 11670 ± 4% -24.4% 8826 ± 5% -3.3% 11288 ± 4% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->load.weight.min 386.83 ± 26% +33.2% 515.16 ± 12% +27.2% 492.08 ± 19% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->vruntime.avg 42031 ± 10% +30.9% 55022 ± 8% -0.5% 41811 ± 8% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.avg 81249 ± 4% +39.2% 113112 ± 7% +0.4% 81550 ± 8% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.max 9990 ± 6% +50.0% 14989 ± 8% +1.8% 10172 ± 15% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.stddev 280.84 ± 11% +31.5% 369.28 ± 5% +0.8% 283.22 ± 14% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg_contrib.avg 2.91 ± 82% -67.6% 0.94 ± 34% -19.0% 2.36 ± 91% sched_debug.cfs_rq:/system.slice/systemd-journald.service.se->sum_exec_runtime.max -19.83 +53.8% -30.50 +75.6% -34.83 sched_debug.cpu.nr_uninterruptible.min [2] ========================================================================================= compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase: gcc-14/performance/x86_64-rhel-9.4/Semaphores/debian-12-x86_64-phoronix/lkp-gnr-2sp3/stress-ng-1.11.0/pts commit: 38a68b982d ("<linux/compiler_types.h>: Add the __signed_scalar_typeof() helper") 089d84203a ("sched/fair: Fold the sched_avg update") d936730940 ("sched/fair: Fix sched_avg fold") 38a68b982dd0b10e 089d84203ad42bc8fd6dbf41683 d936730940bfff3f3b22770cfe9 ---------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 0.04 ± 2% +0.0 0.04 ± 3% -0.0 0.03 ± 2% mpstat.cpu.all.soft% 1.71e+08 ± 3% -12.3% 1.5e+08 ± 2% -0.8% 1.696e+08 ± 2% vmstat.system.cs 59578 ± 22% -16.5% 49752 ± 4% -24.1% 45240 ± 22% proc-vmstat.numa_hint_faults 59099 ± 22% -16.4% 49416 ± 4% -24.3% 44745 ± 23% proc-vmstat.numa_hint_faults_local 8.289e+09 -13.2% 7.198e+09 ± 2% -1.8% 8.142e+09 ± 2% time.involuntary_context_switches 1715 +2.9% 1764 -0.7% 1703 time.user_time 56162 ± 15% +79.7% 100932 ± 5% +16.7% 65551 ± 9% time.voluntary_context_switches 3.533e+08 +11.3% 3.934e+08 +1.1% 3.573e+08 pts.stress-ng.Semaphores.bogo_ops_s 8.289e+09 -13.2% 7.198e+09 ± 2% -1.8% 8.142e+09 ± 2% pts.time.involuntary_context_switches 1715 +2.9% 1764 -0.7% 1703 pts.time.user_time 56162 ± 15% +79.7% 100932 ± 5% +16.7% 65551 ± 9% pts.time.voluntary_context_switches 3.20 ± 74% -1.8 1.44 ±101% -2.2 1.00 ± 89% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 2.79 ± 62% -1.4 1.44 ±101% -1.8 1.00 ± 89% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault 2.63 ± 63% -1.2 1.44 ±101% -1.6 1.00 ± 89% perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 1.41 ± 43% -0.5 0.86 ± 99% -1.0 0.43 ±100% perf-profile.calltrace.cycles-pp.shmem_add_to_page_cache.shmem_alloc_and_add_folio.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write 0.86 ± 45% +0.1 0.97 ± 57% +1.5 2.32 ± 38% perf-profile.calltrace.cycles-pp.lookup_fast.open_last_lookups.path_openat.do_filp_open.do_sys_openat2 2.06 ± 40% +0.4 2.50 ± 22% +2.2 4.22 ± 49% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 2.06 ± 40% +0.4 2.50 ± 22% +2.2 4.22 ± 49% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 2.06 ± 40% +0.4 2.50 ± 22% +2.2 4.22 ± 49% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 2.06 ± 40% +0.4 2.50 ± 22% +2.2 4.22 ± 49% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64 1.19 ± 58% +0.6 1.77 ± 52% +2.0 3.20 ± 59% perf-profile.calltrace.cycles-pp.open_last_lookups.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat 3.69 ± 30% +1.4 5.11 ± 47% +3.4 7.05 ± 29% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.69 ± 30% +1.4 5.11 ± 47% +3.4 7.05 ± 29% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64 3.20 ± 74% -1.8 1.44 ±101% -2.2 1.00 ± 89% perf-profile.children.cycles-pp.do_fault 2.44 ± 36% -1.5 0.92 ±104% -1.7 0.72 ± 81% perf-profile.children.cycles-pp.mutex_unlock 2.63 ± 63% -1.2 1.44 ±101% -1.6 1.00 ± 89% perf-profile.children.cycles-pp.do_read_fault 2.63 ± 63% -1.2 1.44 ±101% -1.6 1.00 ± 89% perf-profile.children.cycles-pp.filemap_map_pages 1.41 ± 43% -0.5 0.86 ± 99% -1.0 0.43 ±100% perf-profile.children.cycles-pp.shmem_add_to_page_cache 0.18 ±223% +0.6 0.77 ± 51% +1.3 1.46 ± 46% perf-profile.children.cycles-pp.__d_lookup_rcu 4.21 ± 31% +1.5 5.69 ± 45% +3.0 7.19 ± 32% perf-profile.children.cycles-pp.do_filp_open 4.21 ± 31% +1.5 5.69 ± 45% +3.0 7.19 ± 32% perf-profile.children.cycles-pp.path_openat 2.44 ± 36% -1.5 0.92 ±104% -1.7 0.72 ± 81% perf-profile.self.cycles-pp.mutex_unlock 0.18 ±223% +0.6 0.77 ± 51% +1.3 1.46 ± 46% perf-profile.self.cycles-pp.__d_lookup_rcu 1.586e+11 ± 3% +3.7% 1.644e+11 +2.2% 1.621e+11 perf-stat.i.branch-instructions 0.30 ± 4% -0.0 0.29 +0.1 0.37 perf-stat.i.branch-miss-rate% 2.701e+08 ± 3% -1.2% 2.669e+08 +68.1% 4.54e+08 perf-stat.i.branch-misses 1.833e+08 ± 4% -11.7% 1.619e+08 ± 2% -0.0% 1.832e+08 ± 2% perf-stat.i.context-switches 1.64 ± 7% -4.9% 1.56 -6.0% 1.55 ± 2% perf-stat.i.cpi 407.20 +24.9% 508.59 ± 2% +1.3% 412.44 ± 2% perf-stat.i.cpu-migrations 0.02 ± 6% -0.0 0.02 ± 3% -0.0 0.02 ± 2% perf-stat.i.dTLB-load-miss-rate% 2.08e+11 ± 3% +3.1% 2.144e+11 +2.2% 2.125e+11 perf-stat.i.dTLB-loads 741999 ± 5% -27.0% 541820 ± 8% -30.3% 516965 ± 2% perf-stat.i.dTLB-store-misses 1.219e+11 ± 3% +3.5% 1.262e+11 +2.6% 1.251e+11 perf-stat.i.dTLB-stores 7.463e+11 ± 3% +3.3% 7.707e+11 +2.2% 7.628e+11 perf-stat.i.instructions 0.92 ± 3% +3.9% 0.96 +2.4% 0.94 perf-stat.i.metric.G/sec 12710 -1.8% 12482 -0.8% 12610 ± 2% perf-stat.i.minor-faults 12715 -1.8% 12487 -0.8% 12615 ± 2% perf-stat.i.page-faults 0.17 -0.0 0.16 +0.1 0.28 perf-stat.overall.branch-miss-rate% 0.02 -0.0 0.02 ± 3% -0.0 0.02 perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 5% -0.0 0.00 ± 8% -0.0 0.00 ± 2% perf-stat.overall.dTLB-store-miss-rate% 1.551e+11 ± 2% +3.6% 1.606e+11 +2.1% 1.584e+11 perf-stat.ps.branch-instructions 2.64e+08 ± 3% -1.2% 2.607e+08 +68.0% 4.434e+08 perf-stat.ps.branch-misses 1.791e+08 ± 4% -11.8% 1.58e+08 ± 2% -0.1% 1.789e+08 ± 2% perf-stat.ps.context-switches 398.46 +24.8% 497.28 ± 2% +1.2% 403.37 ± 2% perf-stat.ps.cpu-migrations 2.033e+11 ± 3% +3.0% 2.095e+11 +2.1% 2.076e+11 perf-stat.ps.dTLB-loads 725199 ± 5% -27.0% 529185 ± 8% -30.4% 504934 ± 2% perf-stat.ps.dTLB-store-misses 1.192e+11 ± 3% +3.4% 1.232e+11 +2.5% 1.222e+11 perf-stat.ps.dTLB-stores 7.299e+11 ± 2% +3.2% 7.531e+11 +2.1% 7.454e+11 perf-stat.ps.instructions 12417 -1.8% 12194 -0.8% 12317 ± 2% perf-stat.ps.minor-faults 12422 -1.8% 12199 -0.8% 12322 ± 2% perf-stat.ps.page-faults 3.379e+13 +1.5% 3.431e+13 +0.4% 3.394e+13 perf-stat.total.instructions 21355 ± 17% +30.6% 27891 ± 9% -9.7% 19290 ± 14% sched_debug.cfs_rq:/.avg_vruntime.max 2572 ± 8% +23.2% 3167 ± 10% -7.0% 2391 ± 9% sched_debug.cfs_rq:/.avg_vruntime.stddev 3.46 ± 20% +1005.0% 38.27 ± 20% +22.4% 4.24 ± 44% sched_debug.cfs_rq:/.load_avg.avg 172.33 ± 33% +353.4% 781.40 ± 10% +88.5% 324.83 ± 74% sched_debug.cfs_rq:/.load_avg.max 14.69 ± 27% +578.7% 99.70 ± 21% +66.6% 24.48 ± 59% sched_debug.cfs_rq:/.load_avg.stddev 1391 ± 11% -23.6% 1063 ± 16% +9.2% 1518 ± 20% sched_debug.cfs_rq:/system.slice.load.avg 6227 ± 9% -22.2% 4846 ± 13% +4.5% 6509 ± 18% sched_debug.cfs_rq:/system.slice.load.stddev 4.52 ± 8% +5661.0% 260.22 ± 7% -5.1% 4.28 ± 11% sched_debug.cfs_rq:/system.slice.load_avg.avg 41.17 ± 22% +1960.9% 848.40 ± 12% -6.1% 38.67 ± 28% sched_debug.cfs_rq:/system.slice.load_avg.max 7.68 ± 8% +2068.9% 166.64 ± 9% -1.5% 7.57 ± 16% sched_debug.cfs_rq:/system.slice.load_avg.stddev 45.50 ± 85% -70.1% 13.60 ± 46% +44.7% 65.83 ± 95% sched_debug.cfs_rq:/system.slice.se->avg.load_avg.max 6.40 ± 50% -59.6% 2.58 ± 17% +26.1% 8.07 ± 87% sched_debug.cfs_rq:/system.slice.se->avg.load_avg.stddev 23349 ± 27% -58.9% 9596 ± 14% -7.7% 21550 ± 32% sched_debug.cfs_rq:/system.slice.se->load.weight.avg 462482 ± 18% -65.8% 158309 ± 59% -2.9% 448934 ± 25% sched_debug.cfs_rq:/system.slice.se->load.weight.max 1010 ± 10% -89.4% 107.20 ± 10% -7.0% 939.50 ± 34% sched_debug.cfs_rq:/system.slice.se->load.weight.min 71887 ± 20% -76.9% 16583 ± 50% -8.7% 65645 ± 25% sched_debug.cfs_rq:/system.slice.se->load.weight.stddev 1677 ± 27% +2348.6% 41062 ± 8% +30.5% 2188 ± 35% sched_debug.cfs_rq:/system.slice.tg_load_avg.avg 4973 ± 43% +1480.9% 78625 ± 6% -8.8% 4538 ± 36% sched_debug.cfs_rq:/system.slice.tg_load_avg.max 1361 ± 27% +1922.4% 27528 ± 6% +37.7% 1874 ± 42% sched_debug.cfs_rq:/system.slice.tg_load_avg.min 554.12 ± 61% +2079.3% 12075 ± 12% -12.5% 484.93 ± 46% sched_debug.cfs_rq:/system.slice.tg_load_avg.stddev 12.35 ± 36% +2045.2% 265.00 ± 7% +15.0% 14.21 ± 45% sched_debug.cfs_rq:/system.slice.tg_load_avg_contrib.avg 81.21 ± 44% +112.1% 172.28 ± 11% +11.4% 90.50 ± 47% sched_debug.cfs_rq:/system.slice.tg_load_avg_contrib.stddev -0.34 -1648.0% 5.20 ±122% -2398.6% 7.72 ±181% sched_debug.cfs_rq:/system.slice/containerd.service.se->vruntime.min 271.19 ± 5% +53.5% 416.15 ± 11% -3.8% 260.79 ± 8% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.load_avg.avg 4.52 ± 8% -21.1% 3.56 ± 14% -5.3% 4.28 ± 12% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->avg.load_avg.avg 11670 ± 2% -27.0% 8517 ± 6% -2.3% 11398 ± 3% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.se->load.weight.min 41222 ± 5% +48.6% 61266 ± 10% +1.9% 41999 ± 12% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.avg 83749 ± 7% +44.4% 120949 ± 6% +0.9% 84484 ± 4% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.max 28726 ± 8% +52.0% 43652 ± 11% -1.6% 28263 ± 12% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.min 10569 ± 14% +48.9% 15737 ± 13% +11.8% 11814 ± 8% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg.stddev 269.98 ± 8% +48.4% 400.63 ± 12% -1.7% 265.28 ± 10% sched_debug.cfs_rq:/system.slice/lkp-bootstrap.service.tg_load_avg_contrib.avg ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2025-12-21 9:38 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-12-18 4:59 [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression kernel test robot 2025-12-18 8:37 ` Peter Zijlstra 2025-12-18 10:11 ` Shrikanth Hegde 2025-12-18 10:20 ` Peter Zijlstra 2025-12-18 10:37 ` Vincent Guittot 2025-12-19 8:16 ` [tip: sched/core] sched/fair: Fix sched_avg fold tip-bot2 for Peter Zijlstra 2025-12-21 9:37 ` [tip:sched/core] [sched/fair] 089d84203a: pts.schbench.32.usec,_99.9th_latency_percentile 52.4% regression Oliver Sang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®