Hello, kernel test robot noticed a -90.3% regression of stress-ng.sem.ops_per_sec on: commit: f879827b83e4d654841ce0163381f4c2604a8c73 ("sched/fair: Add trivial fair server") https://git.kernel.org/cgit/linux/kernel/git/peterz/queue.git sched/more testcase: stress-ng test machine: 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480CTDX (Sapphire Rapids) with 256G memory parameters: nr_threads: 100% testtime: 60s sc_pid_max: 4194304 class: scheduler test: sem cpufreq_governor: performance If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags | Reported-by: kernel test robot | Closes: https://lore.kernel.org/oe-lkp/202312022113.56f340d7-oliver.sang@intel.com Details are as below: --------------------------------------------------------------------------------------------------> ========================================================================================= class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/sc_pid_max/tbox_group/test/testcase/testtime: scheduler/gcc-12/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/4194304/lkp-spr-r02/sem/stress-ng/60s commit: c8b1132966 ("Merge branch 'tip/sched/core'") f879827b83 ("sched/fair: Add trivial fair server") c8b113296686eae8 f879827b83e4d654841ce016338 ---------------- --------------------------- %stddev %change %stddev \ | \ 16993 ± 2% -51.0% 8320 ± 3% uptime.idle 9.186e+09 ± 4% -89.7% 9.464e+08 ± 11% cpuidle..time 5.297e+08 -99.8% 1188850 ± 11% cpuidle..usage 1162733 ± 27% -67.3% 380755 ± 33% numa-numastat.node0.local_node 1267872 ± 29% -62.1% 481138 ± 20% numa-numastat.node0.numa_hit 3002458 ± 11% -51.9% 1444925 ± 9% numa-numastat.node1.local_node 3132909 ± 12% -49.6% 1579678 ± 6% numa-numastat.node1.numa_hit 81.76 -75.0 6.77 ± 10% mpstat.cpu.all.idle% 9.28 +82.1 91.40 mpstat.cpu.all.irq% 0.08 ± 5% +0.0 0.11 mpstat.cpu.all.soft% 5.12 ± 2% -4.3 0.81 ± 2% mpstat.cpu.all.sys% 3.76 -2.9 0.90 ± 2% mpstat.cpu.all.usr% 1791 ± 24% -94.1% 105.17 ± 6% perf-c2c.DRAM.local 1980 ± 13% +214.4% 6226 ± 2% perf-c2c.DRAM.remote 106442 ± 13% -86.6% 14265 ± 2% perf-c2c.HITM.local 1578 ± 16% +191.8% 4604 ± 2% perf-c2c.HITM.remote 108020 ± 13% -82.5% 18869 ± 2% perf-c2c.HITM.total 82.23 -88.3% 9.59 ± 6% vmstat.cpu.id 14.01 ± 2% +539.6% 89.64 vmstat.cpu.sy 12656118 ± 2% -51.0% 6201855 vmstat.memory.cache 58.07 ± 5% +112.4% 123.32 vmstat.procs.r 17534897 ± 2% -90.3% 1696212 vmstat.system.cs 4024915 ± 2% -82.2% 714621 vmstat.system.in 1.174e+09 -90.3% 1.136e+08 stress-ng.sem.ops 19571061 -90.3% 1904041 stress-ng.sem.ops_per_sec 19718706 +29.5% 25530326 ± 2% stress-ng.time.involuntary_context_switches 36256 -30.7% 25109 stress-ng.time.minor_page_faults 4366 -98.9% 47.17 ± 3% stress-ng.time.percent_of_cpu_this_job_got 1647 -98.7% 21.74 ± 2% stress-ng.time.system_time 1073 -99.3% 8.05 ± 3% stress-ng.time.user_time 5.861e+08 -90.3% 56773051 stress-ng.time.voluntary_context_switches 150483 ± 6% +1760.0% 2799019 meminfo.Active 150339 ± 6% +1761.7% 2798880 meminfo.Active(anon) 906793 -11.6% 801565 meminfo.AnonPages 12435789 ± 2% -51.7% 6007432 meminfo.Cached 15890214 -40.8% 9408048 meminfo.Committed_AS 10339450 ± 3% -88.8% 1157751 meminfo.Inactive 10339267 ± 3% -88.8% 1157559 meminfo.Inactive(anon) 1453467 ± 5% -17.1% 1204306 ± 3% meminfo.Mapped 16565293 -41.3% 9716895 meminfo.Memused 9583816 ± 3% -67.1% 3155465 meminfo.Shmem 16747749 ± 2% -39.3% 10167965 meminfo.max_used_kB 555955 ± 7% -21.7% 435391 ± 17% numa-meminfo.node0.AnonPages 2715806 ± 44% -82.4% 478040 ± 9% numa-meminfo.node0.Inactive 2715803 ± 44% -82.4% 477916 ± 9% numa-meminfo.node0.Inactive(anon) 398024 ± 3% -30.6% 276060 ± 12% numa-meminfo.node0.Mapped 2173148 ± 55% -96.9% 67203 ±141% numa-meminfo.node0.Shmem 137867 ± 12% +1914.7% 2777580 ± 2% numa-meminfo.node1.Active 137723 ± 12% +1916.7% 2777533 ± 2% numa-meminfo.node1.Active(anon) 8099761 ± 26% -55.2% 3627680 ± 26% numa-meminfo.node1.FilePages 7628203 ± 14% -91.1% 681239 ± 6% numa-meminfo.node1.Inactive 7628022 ± 14% -91.1% 681171 ± 6% numa-meminfo.node1.Inactive(anon) 1058065 ± 6% -11.7% 934797 ± 5% numa-meminfo.node1.Mapped 9932600 ± 22% -46.5% 5311639 ± 18% numa-meminfo.node1.MemUsed 7415183 ± 15% -58.3% 3092575 ± 4% numa-meminfo.node1.Shmem 1114 ± 2% +143.1% 2709 turbostat.Avg_MHz 40.44 ± 2% +53.4 93.82 turbostat.Busy% 2755 +5.2% 2900 turbostat.Bzy_MHz 12464511 -99.2% 103978 ± 8% turbostat.C1 0.50 ± 3% -0.5 0.01 turbostat.C1% 5.144e+08 -99.9% 539221 ± 15% turbostat.C1E 58.77 -56.1 2.70 ± 15% turbostat.C1E% 59.56 -89.6% 6.17 ± 15% turbostat.CPU%c1 0.32 -81.3% 0.06 turbostat.IPC 2.665e+08 -82.4% 46858207 turbostat.IRQ 2138496 ± 2% -99.5% 9676 ± 18% turbostat.POLL 0.07 ± 5% -0.1 0.00 turbostat.POLL% 522.28 +4.7% 546.57 turbostat.PkgWatt 139000 ± 7% -21.7% 108843 ± 17% numa-vmstat.node0.nr_anon_pages 679013 ± 44% -82.4% 119474 ± 9% numa-vmstat.node0.nr_inactive_anon 99535 ± 3% -30.7% 68966 ± 12% numa-vmstat.node0.nr_mapped 543340 ± 55% -96.9% 16803 ±141% numa-vmstat.node0.nr_shmem 679012 ± 44% -82.4% 119474 ± 9% numa-vmstat.node0.nr_zone_inactive_anon 1267743 ± 29% -62.0% 481114 ± 20% numa-vmstat.node0.numa_hit 1162604 ± 27% -67.3% 380731 ± 33% numa-vmstat.node0.numa_local 34449 ± 12% +1916.9% 694818 ± 2% numa-vmstat.node1.nr_active_anon 2025232 ± 26% -55.2% 907007 ± 26% numa-vmstat.node1.nr_file_pages 1907277 ± 14% -91.1% 169906 ± 6% numa-vmstat.node1.nr_inactive_anon 264719 ± 6% -11.8% 233359 ± 5% numa-vmstat.node1.nr_mapped 1854088 ± 15% -58.3% 773231 ± 4% numa-vmstat.node1.nr_shmem 34449 ± 12% +1916.9% 694818 ± 2% numa-vmstat.node1.nr_zone_active_anon 1907273 ± 14% -91.1% 169906 ± 6% numa-vmstat.node1.nr_zone_inactive_anon 3132736 ± 12% -49.6% 1579500 ± 6% numa-vmstat.node1.numa_hit 3002285 ± 11% -51.9% 1444747 ± 9% numa-vmstat.node1.numa_local 37599 ± 6% +1761.2% 699812 proc-vmstat.nr_active_anon 226710 -11.6% 200389 proc-vmstat.nr_anon_pages 6139688 +2.8% 6310661 proc-vmstat.nr_dirty_background_threshold 12294389 +2.8% 12636753 proc-vmstat.nr_dirty_threshold 3109211 ± 2% -51.7% 1502012 proc-vmstat.nr_file_pages 61771524 +2.8% 63483758 proc-vmstat.nr_free_pages 2585074 ± 3% -88.8% 289447 proc-vmstat.nr_inactive_anon 363506 ± 5% -17.1% 301175 ± 3% proc-vmstat.nr_mapped 2396217 ± 3% -67.1% 789019 proc-vmstat.nr_shmem 41464 -8.7% 37865 proc-vmstat.nr_slab_reclaimable 37599 ± 6% +1761.2% 699812 proc-vmstat.nr_zone_active_anon 2585074 ± 3% -88.8% 289447 proc-vmstat.nr_zone_inactive_anon 332134 ± 14% -33.6% 220616 proc-vmstat.numa_hint_faults 215047 ± 19% -47.7% 112463 ± 6% proc-vmstat.numa_hint_faults_local 4402321 ± 2% -53.2% 2062331 proc-vmstat.numa_hit 4166718 ± 2% -56.1% 1827195 proc-vmstat.numa_local 91936 ± 9% -84.8% 13930 ± 60% proc-vmstat.numa_pages_migrated 999347 ± 5% -51.0% 489647 ± 2% proc-vmstat.numa_pte_updates 72752 ± 12% +119.3% 159518 ± 34% proc-vmstat.pgactivate 4696592 ± 2% -51.5% 2280066 proc-vmstat.pgalloc_normal 1280516 ± 3% -13.3% 1110055 proc-vmstat.pgfault 913260 ± 4% +12.6% 1028187 ± 5% proc-vmstat.pgfree 91936 ± 9% -84.8% 13930 ± 60% proc-vmstat.pgmigrate_success 297408 -91.4% 25656 ± 2% sched_debug.cfs_rq:/.avg_vruntime.avg 719020 ± 14% +83.3% 1318084 ± 18% sched_debug.cfs_rq:/.avg_vruntime.max 264879 ± 4% -96.8% 8457 ± 7% sched_debug.cfs_rq:/.avg_vruntime.min 35247 ± 14% +191.0% 102567 ± 11% sched_debug.cfs_rq:/.avg_vruntime.stddev 0.19 ± 3% +206.3% 0.58 ± 3% sched_debug.cfs_rq:/.h_nr_running.avg 0.38 ± 4% +29.3% 0.49 ± 4% sched_debug.cfs_rq:/.h_nr_running.stddev 31519 ± 21% -43.0% 17980 ± 6% sched_debug.cfs_rq:/.left_deadline.avg 373527 ± 15% +245.4% 1290149 ± 18% sched_debug.cfs_rq:/.left_deadline.max 31512 ± 21% -43.5% 17812 ± 6% sched_debug.cfs_rq:/.left_vruntime.avg 373456 ± 15% +245.4% 1289977 ± 18% sched_debug.cfs_rq:/.left_vruntime.max 43747 ± 3% -19.2% 35341 ± 38% sched_debug.cfs_rq:/.load_avg.max 297408 -91.4% 25656 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg 719020 ± 14% +83.3% 1318084 ± 18% sched_debug.cfs_rq:/.min_vruntime.max 264879 ± 4% -96.8% 8457 ± 7% sched_debug.cfs_rq:/.min_vruntime.min 35247 ± 14% +191.0% 102567 ± 11% sched_debug.cfs_rq:/.min_vruntime.stddev 0.18 +146.8% 0.44 ± 2% sched_debug.cfs_rq:/.nr_running.avg 178.33 ±117% -94.7% 9.42 ± 57% sched_debug.cfs_rq:/.removed.load_avg.avg 1879 ±102% -96.7% 61.46 ± 48% sched_debug.cfs_rq:/.removed.load_avg.stddev 31512 ± 21% -43.5% 17812 ± 6% sched_debug.cfs_rq:/.right_vruntime.avg 373456 ± 15% +245.4% 1290045 ± 18% sched_debug.cfs_rq:/.right_vruntime.max 191.08 +138.7% 456.02 ± 3% sched_debug.cfs_rq:/.runnable_avg.avg 1282 ± 9% +40.4% 1799 ± 9% sched_debug.cfs_rq:/.runnable_avg.max 163.12 ± 3% +81.7% 296.46 ± 3% sched_debug.cfs_rq:/.runnable_avg.stddev 180.90 -60.3% 71.75 ± 3% sched_debug.cfs_rq:/.util_avg.avg 50.25 ± 7% -99.8% 0.08 ±223% sched_debug.cfs_rq:/.util_avg.min 8.02 ± 5% +2670.6% 222.15 ± 5% sched_debug.cfs_rq:/.util_est_enqueued.avg 840.75 ± 13% +28.8% 1082 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.max 67.76 ± 5% +188.8% 195.67 ± 5% sched_debug.cfs_rq:/.util_est_enqueued.stddev 476824 ± 3% +17.1% 558339 sched_debug.cpu.avg_idle.avg 929342 ± 31% +46.9% 1365278 ± 9% sched_debug.cpu.avg_idle.max 11.88 ± 11% +136.8% 28.13 ± 11% sched_debug.cpu.clock.stddev 66753 ± 2% -18.9% 54165 sched_debug.cpu.clock_task.avg 54515 ± 3% -23.1% 41928 ± 2% sched_debug.cpu.clock_task.min 840.52 ± 3% +29.3% 1086 ± 6% sched_debug.cpu.clock_task.stddev 931.88 ± 12% +205.3% 2844 ± 9% sched_debug.cpu.curr->pid.avg 2207 ± 5% +15.1% 2539 ± 3% sched_debug.cpu.curr->pid.stddev 506520 ± 2% +36.7% 692380 sched_debug.cpu.max_idle_balance_cost.avg 737290 ± 23% +71.0% 1260499 ± 3% sched_debug.cpu.max_idle_balance_cost.max 26530 ± 95% +430.8% 140813 ± 3% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 16% +162.0% 0.00 ± 26% sched_debug.cpu.next_balance.stddev 0.15 ± 12% +81.1% 0.26 ± 13% sched_debug.cpu.nr_running.avg 1.92 ± 17% +52.2% 2.92 ± 6% sched_debug.cpu.nr_running.max 0.36 ± 7% +51.6% 0.54 ± 2% sched_debug.cpu.nr_running.stddev 2462095 -90.3% 239220 sched_debug.cpu.nr_switches.avg 2628391 -86.1% 365699 ± 7% sched_debug.cpu.nr_switches.max 1248238 ± 21% -86.1% 174002 ± 7% sched_debug.cpu.nr_switches.min 127417 ± 12% -84.0% 20368 ± 15% sched_debug.cpu.nr_switches.stddev 0.24 ± 20% +139.9% 0.58 ± 2% perf-stat.i.MPKI 2.812e+10 ± 2% -47.1% 1.488e+10 perf-stat.i.branch-instructions 1.10 -0.6 0.50 perf-stat.i.branch-miss-rate% 2.658e+08 ± 2% -75.1% 66118672 ± 2% perf-stat.i.branch-misses 3.33 ± 14% +9.5 12.83 perf-stat.i.cache-miss-rate% 30262369 +18.4% 35827586 perf-stat.i.cache-misses 1.114e+09 ± 2% -74.7% 2.815e+08 ± 2% perf-stat.i.cache-references 18360627 ± 2% -90.3% 1773866 ± 2% perf-stat.i.context-switches 1.86 +411.9% 9.53 perf-stat.i.cpi 2.344e+11 ± 2% +160.9% 6.115e+11 perf-stat.i.cpu-cycles 6853870 ± 2% -98.0% 134793 ± 3% perf-stat.i.cpu-migrations 11232 ± 5% +57.8% 17719 perf-stat.i.cycles-between-cache-misses 0.14 ± 42% -0.1 0.03 ± 4% perf-stat.i.dTLB-load-miss-rate% 42868589 ± 2% -88.8% 4789483 ± 4% perf-stat.i.dTLB-load-misses 3.729e+10 ± 2% -57.4% 1.59e+10 perf-stat.i.dTLB-loads 0.04 ± 6% -0.0 0.02 perf-stat.i.dTLB-store-miss-rate% 8763493 -93.2% 598257 ± 2% perf-stat.i.dTLB-store-misses 2.19e+10 ± 2% -87.9% 2.646e+09 ± 2% perf-stat.i.dTLB-stores 1.398e+11 ± 2% -55.0% 6.284e+10 perf-stat.i.instructions 0.57 -76.3% 0.14 ± 4% perf-stat.i.ipc 1.04 ± 2% +161.3% 2.72 perf-stat.i.metric.GHz 218.03 ± 3% -60.8% 85.57 ± 15% perf-stat.i.metric.K/sec 393.18 ± 2% -61.9% 149.94 perf-stat.i.metric.M/sec 71.59 ± 3% +23.9 95.50 perf-stat.i.node-load-miss-rate% 7319623 ± 3% +28.2% 9383972 ± 2% perf-stat.i.node-load-misses 3974657 ± 7% -90.4% 382649 ± 14% perf-stat.i.node-loads 0.22 ± 3% +159.8% 0.56 perf-stat.overall.MPKI 0.95 -0.5 0.44 perf-stat.overall.branch-miss-rate% 2.70 ± 3% +9.7 12.43 perf-stat.overall.cache-miss-rate% 1.69 +482.7% 9.84 perf-stat.overall.cpi 7807 ± 2% +124.1% 17495 perf-stat.overall.cycles-between-cache-misses 0.12 ± 3% -0.1 0.04 ± 5% perf-stat.overall.dTLB-load-miss-rate% 0.04 ± 2% -0.0 0.02 perf-stat.overall.dTLB-store-miss-rate% 0.59 -82.8% 0.10 perf-stat.overall.ipc 63.40 ± 3% +32.7 96.08 perf-stat.overall.node-load-miss-rate% 2.738e+10 ± 2% -47.2% 1.447e+10 perf-stat.ps.branch-instructions 2.605e+08 ± 2% -75.7% 63224241 ± 2% perf-stat.ps.branch-misses 29446874 +16.5% 34311040 perf-stat.ps.cache-misses 1.093e+09 ± 2% -74.7% 2.761e+08 ± 2% perf-stat.ps.cache-references 18057188 ± 2% -90.3% 1743595 perf-stat.ps.context-switches 2.299e+11 ± 2% +161.1% 6.002e+11 perf-stat.ps.cpu-cycles 6741178 ± 2% -98.1% 129365 ± 3% perf-stat.ps.cpu-migrations 41997044 ± 2% -86.9% 5493471 ± 5% perf-stat.ps.dTLB-load-misses 3.64e+10 ± 2% -57.6% 1.544e+10 perf-stat.ps.dTLB-loads 8615360 -93.3% 573832 ± 2% perf-stat.ps.dTLB-store-misses 2.14e+10 ± 2% -88.2% 2.519e+09 perf-stat.ps.dTLB-stores 1.362e+11 ± 2% -55.2% 6.103e+10 perf-stat.ps.instructions 17871 ± 7% -12.6% 15624 ± 2% perf-stat.ps.minor-faults 7129405 ± 3% +27.7% 9105497 perf-stat.ps.node-load-misses 4118428 ± 6% -91.0% 371774 ± 12% perf-stat.ps.node-loads 17871 ± 7% -12.6% 15625 ± 2% perf-stat.ps.page-faults 8.744e+12 -56.0% 3.843e+12 perf-stat.total.instructions 37.43 -37.4 0.00 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 35.63 -35.6 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 34.74 -34.7 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 21.03 -21.0 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 12.96 -13.0 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 11.38 -11.4 0.00 perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 10.68 -10.7 0.00 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 10.44 -10.4 0.00 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 8.57 -8.6 0.00 perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 7.18 -7.2 0.00 perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary 56.38 -6.3 50.06 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 6.30 -6.3 0.00 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry 56.56 -6.3 50.28 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 56.33 -6.3 50.06 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 56.23 -6.2 50.06 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 5.10 -5.1 0.00 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle 5.00 -5.0 0.00 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue 3.74 -3.1 0.65 ± 2% perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt 3.63 -3.0 0.64 ± 3% perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues 3.50 -2.9 0.64 ± 2% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.hrtimer_wakeup 2.33 -1.8 0.57 ± 2% perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up 3.25 -1.4 1.90 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.__do_softirq 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.__do_softirq.irq_exit_rcu 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 0.00 +0.7 0.70 perf-profile.calltrace.cycles-pp.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch 0.00 +0.7 0.71 perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch.__schedule 0.00 +1.0 1.00 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues 0.00 +1.0 1.01 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt 0.00 +1.3 1.28 ± 2% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.finish_task_switch 0.00 +1.3 1.28 ± 2% perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.finish_task_switch.__schedule 0.00 +1.3 1.29 perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.finish_task_switch.__schedule.schedule_idle 0.00 +1.3 1.29 ± 2% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.finish_task_switch.__schedule.schedule_idle.do_idle 0.00 +1.9 1.86 perf-profile.calltrace.cycles-pp.cpudl_set.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending 0.00 +1.9 1.90 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single 0.00 +1.9 1.90 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single 0.00 +1.9 1.91 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single 26.97 +21.6 48.60 perf-profile.calltrace.cycles-pp.clock_nanosleep 23.47 +25.0 48.50 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep 23.38 +25.1 48.50 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 19.08 +29.3 48.43 perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 18.04 +30.4 48.40 perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 18.00 +30.4 48.39 perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe 17.28 +31.1 48.37 perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64 13.50 ± 2% +34.8 48.31 perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 13.04 ± 2% +35.3 48.30 perf-profile.calltrace.cycles-pp.__schedule.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep 11.64 +36.0 47.64 perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 10.12 +37.3 47.43 perf-profile.calltrace.cycles-pp.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 10.07 +37.4 47.43 perf-profile.calltrace.cycles-pp.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 8.01 +42.0 50.02 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 7.69 +42.3 50.01 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 3.52 +44.5 48.02 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.do_nanosleep.hrtimer_nanosleep 1.12 ± 2% +44.6 45.70 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt 0.96 ± 2% +44.7 45.68 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues 0.93 ± 3% +44.7 45.68 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.hrtimer_wakeup 0.00 +45.5 45.45 perf-profile.calltrace.cycles-pp.cpudl_set.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 1.27 +46.4 47.70 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch.__schedule 1.24 +46.4 47.68 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch 0.00 +47.1 47.11 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.cpudl_set.enqueue_task_fair.activate_task 0.00 +47.2 47.16 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.cpudl_set.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +47.6 47.63 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.cpudl_clear.__dequeue_dl_entity.dl_server_stop 0.00 +47.7 47.68 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.cpudl_clear.__dequeue_dl_entity.dl_server_stop.dequeue_task_fair 0.59 +47.8 48.42 perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch.__schedule.schedule_idle 0.00 +47.8 47.84 perf-profile.calltrace.cycles-pp.cpudl_clear.__dequeue_dl_entity.dl_server_stop.dequeue_task_fair.__schedule 0.64 +47.8 48.48 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.finish_task_switch.__schedule.schedule_idle.do_idle 0.00 +47.9 47.85 perf-profile.calltrace.cycles-pp.__dequeue_dl_entity.dl_server_stop.dequeue_task_fair.__schedule.schedule 0.00 +47.9 47.93 perf-profile.calltrace.cycles-pp.dl_server_stop.dequeue_task_fair.__schedule.schedule.do_nanosleep 0.90 +48.9 49.76 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry 37.60 -37.6 0.01 ±223% perf-profile.children.cycles-pp.cpuidle_idle_call 35.76 -35.7 0.01 ±223% perf-profile.children.cycles-pp.cpuidle_enter 35.71 -35.7 0.01 ±223% perf-profile.children.cycles-pp.cpuidle_enter_state 21.11 -21.1 0.00 perf-profile.children.cycles-pp.intel_idle 8.67 -8.7 0.00 perf-profile.children.cycles-pp.flush_smp_call_function_queue 56.38 -6.3 50.06 perf-profile.children.cycles-pp.start_secondary 56.56 -6.3 50.28 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 56.56 -6.3 50.28 perf-profile.children.cycles-pp.cpu_startup_entry 56.47 -6.2 50.28 perf-profile.children.cycles-pp.do_idle 7.92 -6.0 1.95 perf-profile.children.cycles-pp.__flush_smp_call_function_queue 6.90 -5.0 1.94 perf-profile.children.cycles-pp.sched_ttwu_pending 5.48 ± 2% -4.8 0.66 ± 2% perf-profile.children.cycles-pp.select_task_rq 5.35 ± 2% -4.7 0.66 ± 2% perf-profile.children.cycles-pp.select_task_rq_fair 5.17 ± 2% -4.5 0.66 ± 2% perf-profile.children.cycles-pp.select_idle_sibling 4.57 ± 7% -4.5 0.10 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 3.99 -3.9 0.08 ± 6% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 4.20 -3.8 0.36 ± 2% perf-profile.children.cycles-pp.__sched_yield 3.82 -3.8 0.08 ± 6% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 3.69 -3.6 0.06 ± 8% perf-profile.children.cycles-pp.switch_fpu_return 3.75 -3.6 0.13 perf-profile.children.cycles-pp.enqueue_entity 3.11 -3.1 0.05 perf-profile.children.cycles-pp.dequeue_entity 3.35 ± 2% -3.0 0.33 ± 2% perf-profile.children.cycles-pp.available_idle_cpu 3.52 -2.9 0.63 ± 2% perf-profile.children.cycles-pp.select_idle_cpu 2.97 -2.9 0.11 perf-profile.children.cycles-pp.switch_mm_irqs_off 2.73 -2.6 0.13 ± 3% perf-profile.children.cycles-pp.update_load_avg 2.86 -2.6 0.27 ± 2% perf-profile.children.cycles-pp.__x64_sys_sched_yield 2.31 -2.3 0.06 perf-profile.children.cycles-pp.semaphore_posix_thrash 2.20 -2.0 0.18 ± 2% perf-profile.children.cycles-pp.hrtimer_start_range_ns 1.86 -1.8 0.05 perf-profile.children.cycles-pp.prepare_task_switch 1.68 ± 2% -1.6 0.06 ± 6% perf-profile.children.cycles-pp.sem_getvalue@@GLIBC_2.2.5 1.10 ± 4% -1.1 0.05 perf-profile.children.cycles-pp.ktime_get 0.99 ± 2% -0.9 0.11 ± 3% perf-profile.children.cycles-pp.clockevents_program_event 1.99 -0.9 1.12 perf-profile.children.cycles-pp._raw_spin_lock 1.08 ± 10% -0.8 0.24 ± 18% perf-profile.children.cycles-pp.queue_event 1.08 ± 10% -0.8 0.24 ± 17% perf-profile.children.cycles-pp.ordered_events__queue 1.08 ± 10% -0.8 0.26 ± 17% perf-profile.children.cycles-pp.process_simple 0.94 -0.8 0.13 ± 2% perf-profile.children.cycles-pp.__hrtimer_start_range_ns 0.79 -0.7 0.06 perf-profile.children.cycles-pp.set_next_entity 1.10 ± 11% -0.7 0.37 ± 16% perf-profile.children.cycles-pp.reader__read_event 1.10 ± 11% -0.7 0.38 ± 16% perf-profile.children.cycles-pp.record__finish_output 1.10 ± 11% -0.7 0.38 ± 16% perf-profile.children.cycles-pp.perf_session__process_events 1.12 ± 10% -0.7 0.44 ± 16% perf-profile.children.cycles-pp.__cmd_record 0.74 -0.7 0.09 ± 4% perf-profile.children.cycles-pp.lapic_next_deadline 0.44 ± 5% -0.3 0.13 ± 3% perf-profile.children.cycles-pp.tick_nohz_highres_handler 0.40 ± 4% -0.3 0.13 ± 2% perf-profile.children.cycles-pp.tick_sched_handle 0.40 ± 4% -0.3 0.13 ± 2% perf-profile.children.cycles-pp.update_process_times 0.35 ± 4% -0.2 0.12 ± 3% perf-profile.children.cycles-pp.scheduler_tick 0.28 -0.2 0.08 ± 4% perf-profile.children.cycles-pp._find_next_bit 0.28 -0.2 0.10 ± 3% perf-profile.children.cycles-pp.perf_event_task_tick 0.28 -0.2 0.10 ± 3% perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context 0.16 ± 4% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.00 +0.1 0.06 ± 19% perf-profile.children.cycles-pp.cmd_record 0.00 +0.1 0.07 ± 21% perf-profile.children.cycles-pp.__libc_start_main 0.00 +0.1 0.07 ± 21% perf-profile.children.cycles-pp.main 0.00 +0.1 0.07 ± 21% perf-profile.children.cycles-pp.run_builtin 0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.enqueue_dl_entity 0.00 +0.1 0.09 ± 5% perf-profile.children.cycles-pp.pick_task_dl 0.00 +0.1 0.09 ± 15% perf-profile.children.cycles-pp.perf_session__process_user_event 0.00 +0.1 0.09 ± 16% perf-profile.children.cycles-pp.perf_session__deliver_event 0.00 +0.1 0.09 ± 15% perf-profile.children.cycles-pp.__ordered_events__flush 0.00 +0.1 0.12 ± 3% perf-profile.children.cycles-pp.cpudl_heapify_up 0.00 +0.2 0.16 ± 6% perf-profile.children.cycles-pp.cpupri_set 0.00 +0.3 0.25 ± 2% perf-profile.children.cycles-pp.pick_next_task_dl 0.46 +0.3 0.72 perf-profile.children.cycles-pp.irq_exit_rcu 0.23 ± 6% +0.5 0.70 perf-profile.children.cycles-pp.__do_softirq 0.92 +1.0 1.95 perf-profile.children.cycles-pp.asm_sysvec_call_function_single 0.78 +1.2 1.94 perf-profile.children.cycles-pp.sysvec_call_function_single 0.74 +1.2 1.93 perf-profile.children.cycles-pp.__sysvec_call_function_single 27.34 +21.3 48.61 perf-profile.children.cycles-pp.clock_nanosleep 27.09 +21.8 48.89 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 26.70 +22.2 48.88 perf-profile.children.cycles-pp.do_syscall_64 19.11 +29.3 48.43 perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep 18.20 +30.2 48.40 perf-profile.children.cycles-pp.common_nsleep 18.02 +30.4 48.40 perf-profile.children.cycles-pp.hrtimer_nanosleep 17.36 +31.0 48.38 perf-profile.children.cycles-pp.do_nanosleep 16.52 +32.2 48.77 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 15.82 +32.7 48.56 perf-profile.children.cycles-pp.schedule 15.30 +33.4 48.69 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 14.48 +33.5 47.97 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 14.15 +33.8 47.95 perf-profile.children.cycles-pp.hrtimer_interrupt 13.30 +34.6 47.90 perf-profile.children.cycles-pp.__hrtimer_run_queues 12.14 +35.6 47.69 perf-profile.children.cycles-pp.hrtimer_wakeup 12.12 +35.6 47.69 perf-profile.children.cycles-pp.try_to_wake_up 6.79 +41.0 47.82 perf-profile.children.cycles-pp.ttwu_do_activate 6.51 +41.3 47.83 perf-profile.children.cycles-pp.activate_task 8.07 +42.2 50.23 perf-profile.children.cycles-pp.schedule_idle 4.59 +43.2 47.83 perf-profile.children.cycles-pp.enqueue_task_fair 3.56 +44.5 48.05 perf-profile.children.cycles-pp.dequeue_task_fair 3.78 +46.3 50.09 perf-profile.children.cycles-pp.finish_task_switch 0.00 +47.6 47.57 perf-profile.children.cycles-pp.cpudl_set 0.00 +47.9 47.87 perf-profile.children.cycles-pp.cpudl_clear 0.00 +47.9 47.88 perf-profile.children.cycles-pp.__dequeue_dl_entity 0.00 +48.0 47.96 perf-profile.children.cycles-pp.dl_server_stop 23.24 +75.5 98.78 perf-profile.children.cycles-pp.__schedule 1.53 +93.6 95.14 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.14 ± 7% +95.9 96.09 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 21.11 -21.1 0.00 perf-profile.self.cycles-pp.intel_idle 3.73 -3.6 0.11 ± 3% perf-profile.self.cycles-pp.__schedule 3.32 ± 2% -3.0 0.30 ± 3% perf-profile.self.cycles-pp.available_idle_cpu 2.93 -2.8 0.11 ± 3% perf-profile.self.cycles-pp.switch_mm_irqs_off 2.08 -2.0 0.06 ± 8% perf-profile.self.cycles-pp.semaphore_posix_thrash 1.82 -1.8 0.04 ± 44% perf-profile.self.cycles-pp._raw_spin_lock 1.84 -1.7 0.11 ± 3% perf-profile.self.cycles-pp.finish_task_switch 1.51 -1.4 0.12 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 1.43 ± 2% -1.4 0.05 perf-profile.self.cycles-pp.sem_getvalue@@GLIBC_2.2.5 0.97 -0.9 0.05 perf-profile.self.cycles-pp.update_load_avg 1.07 ± 11% -0.8 0.24 ± 16% perf-profile.self.cycles-pp.queue_event 0.74 -0.7 0.09 ± 5% perf-profile.self.cycles-pp.lapic_next_deadline 0.25 -0.2 0.07 ± 5% perf-profile.self.cycles-pp._find_next_bit 0.16 ± 4% -0.1 0.05 ± 8% perf-profile.self.cycles-pp.__intel_pmu_enable_all 0.00 +0.1 0.10 ± 3% perf-profile.self.cycles-pp.cpudl_clear 0.00 +0.1 0.10 perf-profile.self.cycles-pp.cpudl_set 0.00 +0.1 0.12 ± 3% perf-profile.self.cycles-pp.cpudl_heapify_up 0.00 +0.2 0.16 ± 6% perf-profile.self.cycles-pp.cpupri_set 0.12 ± 3% +0.2 0.28 ± 2% perf-profile.self.cycles-pp.select_idle_cpu 0.14 ± 7% +95.9 96.09 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki