Greeting, FYI, we noticed a 2.6% improvement of will-it-scale.per_thread_ops due to commit: commit: 492e639f0c222784e2e0f121966375f641c61b15 ("bpf: Add bpf_seq_printf and bpf_seq_write helpers") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory with following parameters: nr_task: 100% mode: thread test: pthread_mutex2 cpufreq_governor: performance ucode: 0x11 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale In addition to that, the commit also has significant impact on the following tests: +------------------+----------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops 2.5% improvement | | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=100% | | | test=malloc2 | | | ucode=0x11 | +------------------+----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/pthread_mutex2/will-it-scale/0x11 commit: b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support") 492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers") b121b341e5983bdc 492e639f0c222784e2e0f121966 ---------------- --------------------------- %stddev %change %stddev \ | \ 6105969 +2.6% 6265564 will-it-scale.per_thread_ops 9855 +1196.5% 127775 ± 92% will-it-scale.time.maximum_resident_set_size 1.759e+09 +2.6% 1.804e+09 will-it-scale.workload 330.50 ± 13% +71.4% 566.50 ± 21% cpuidle.POLL.usage 149.50 +3.3% 154.50 ± 2% proc-vmstat.nr_anon_transparent_hugepages 24750 +2.5% 25362 proc-vmstat.nr_shmem 22056 ± 6% -21.5% 17321 ± 27% proc-vmstat.numa_hint_faults 22056 ± 6% -21.5% 17321 ± 27% proc-vmstat.numa_hint_faults_local 451.64 ± 11% +27.3% 574.96 ± 21% sched_debug.cfs_rq:/.util_avg.min 0.46 ± 18% -58.7% 0.19 ±105% sched_debug.rt_rq:/.rt_time.avg 133.72 ± 18% -58.7% 55.26 ±105% sched_debug.rt_rq:/.rt_time.max 7.87 ± 18% -58.7% 3.25 ±105% sched_debug.rt_rq:/.rt_time.stddev 123790 ± 14% -13.1% 107602 ± 3% softirqs.CPU100.TIMER 124472 ± 14% -13.0% 108248 ± 5% softirqs.CPU142.TIMER 125239 ± 14% -12.4% 109741 ± 5% softirqs.CPU166.TIMER 130036 ± 11% -15.7% 109609 ± 3% softirqs.CPU170.TIMER 123083 ± 12% -14.6% 105115 ± 3% softirqs.CPU207.TIMER 137740 ± 13% -21.5% 108150 ± 4% softirqs.CPU209.TIMER 126203 ± 11% -11.0% 112261 ± 2% softirqs.CPU213.TIMER 126552 ± 13% -14.7% 107985 ± 2% softirqs.CPU226.TIMER 126084 ± 11% -13.6% 108956 ± 4% softirqs.CPU73.TIMER 3.162e+10 +2.8% 3.252e+10 perf-stat.i.branch-instructions 22.58 +0.3 22.90 perf-stat.i.branch-miss-rate% 7.165e+09 +4.3% 7.47e+09 perf-stat.i.branch-misses 82101720 +1.4% 83214202 perf-stat.i.cache-references 4.361e+11 +2.2% 4.455e+11 perf-stat.i.cpu-cycles 38454 ± 2% +5.2% 40469 ± 2% perf-stat.i.cycles-between-cache-misses 3.53 +0.0 3.57 perf-stat.i.iTLB-load-miss-rate% 3.786e+09 +3.8% 3.931e+09 perf-stat.i.iTLB-load-misses 1.032e+11 +2.4% 1.057e+11 perf-stat.i.iTLB-loads 1.028e+11 +2.7% 1.055e+11 perf-stat.i.instructions 27.45 -1.3% 27.08 perf-stat.i.instructions-per-iTLB-miss 1.51 +2.2% 1.55 perf-stat.i.metric.GHz 468.30 +2.5% 480.18 perf-stat.i.metric.M/sec 22.69 +0.3 23.00 perf-stat.overall.branch-miss-rate% 37831 ± 2% +5.2% 39799 perf-stat.overall.cycles-between-cache-misses 3.54 +0.0 3.59 perf-stat.overall.iTLB-load-miss-rate% 27.14 -1.1% 26.84 perf-stat.overall.instructions-per-iTLB-miss 3.152e+10 +2.8% 3.241e+10 perf-stat.ps.branch-instructions 7.152e+09 +4.2% 7.454e+09 perf-stat.ps.branch-misses 82017742 +1.3% 83120324 perf-stat.ps.cache-references 4.348e+11 +2.1% 4.441e+11 perf-stat.ps.cpu-cycles 3.775e+09 +3.8% 3.919e+09 perf-stat.ps.iTLB-load-misses 1.028e+11 +2.4% 1.053e+11 perf-stat.ps.iTLB-loads 1.025e+11 +2.6% 1.052e+11 perf-stat.ps.instructions 3.122e+13 +2.5% 3.2e+13 perf-stat.total.instructions 10.82 -0.1 10.70 perf-profile.calltrace.cycles-pp.pthread_mutex_unlock 0.72 ± 10% +0.2 0.89 ± 10% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt 0.79 ± 11% +0.2 0.96 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt 0.84 ± 11% +0.3 1.09 ± 10% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt 0.48 ± 58% +0.3 0.76 ± 9% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.11 ± 11% +0.3 1.39 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt 1.34 ± 10% +0.3 1.64 ± 11% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt 1.47 ± 10% +0.3 1.78 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.__pthread_mutex_unlock_usercnt 0.14 ±173% +0.5 0.60 ± 9% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.64 ± 12% +0.5 2.15 ± 10% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock 2.17 ± 11% +0.6 2.74 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock 2.62 ± 10% +0.6 3.24 ± 11% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock 2.89 ± 10% +0.6 3.51 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.pthread_mutex_lock 0.14 ±173% +0.7 0.86 ± 10% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.47 ± 59% +1.4 1.83 ± 10% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.48 ± 59% +1.4 1.90 ± 11% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.67 ± 17% +1.5 2.19 ± 11% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.74 ± 12% -0.6 0.17 ± 10% perf-profile.children.cycles-pp.__remove_hrtimer 0.23 ± 7% -0.2 0.08 ± 20% perf-profile.children.cycles-pp.rb_next 0.29 ± 7% -0.2 0.14 ± 11% perf-profile.children.cycles-pp.timerqueue_del 0.41 ± 10% -0.1 0.32 ± 12% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 8.81 -0.1 8.72 perf-profile.children.cycles-pp.pthread_mutex_unlock 0.11 ± 18% -0.0 0.07 ± 30% perf-profile.children.cycles-pp.rcu_irq_enter 0.11 ± 10% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.timerqueue_add 0.12 ± 11% +0.0 0.15 ± 5% perf-profile.children.cycles-pp.enqueue_hrtimer 0.17 ± 14% +0.0 0.21 ± 12% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.07 ± 15% +0.0 0.12 ± 13% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.06 ± 20% +0.0 0.11 ± 10% perf-profile.children.cycles-pp.cpuacct_charge 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.native_sched_clock 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.sched_clock 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.__calc_delta 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.sched_clock_cpu 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.raise_softirq 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.cpuacct_account_field 0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.reweight_entity 0.01 ±173% +0.1 0.08 ± 6% perf-profile.children.cycles-pp.arch_scale_freq_tick 0.05 ± 58% +0.1 0.12 ± 20% perf-profile.children.cycles-pp.run_timer_softirq 0.01 ±173% +0.1 0.08 ± 29% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.00 +0.1 0.07 ± 11% perf-profile.children.cycles-pp.hrtimer_active 0.00 +0.1 0.08 ± 24% perf-profile.children.cycles-pp.x86_pmu_disable 0.14 ± 15% +0.1 0.22 ± 8% perf-profile.children.cycles-pp.clockevents_program_event 0.09 ± 15% +0.1 0.18 ± 8% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.06 ± 60% +0.1 0.14 ± 12% perf-profile.children.cycles-pp.account_user_time 0.09 ± 19% +0.1 0.18 ± 7% perf-profile.children.cycles-pp._raw_spin_lock 0.09 ± 20% +0.1 0.18 ± 10% perf-profile.children.cycles-pp.__update_load_avg_se 0.01 ±173% +0.1 0.10 ± 15% perf-profile.children.cycles-pp.profile_tick 0.14 ± 12% +0.1 0.25 ± 11% perf-profile.children.cycles-pp.ktime_get 0.08 ± 21% +0.1 0.21 ± 13% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.24 ± 17% +0.1 0.37 ± 7% perf-profile.children.cycles-pp.update_curr 0.16 ± 11% +0.1 0.30 ± 18% perf-profile.children.cycles-pp.__softirqentry_text_start 0.01 ±173% +0.2 0.17 ± 15% perf-profile.children.cycles-pp.run_local_timers 0.04 ± 59% +0.2 0.20 ± 20% perf-profile.children.cycles-pp.update_rq_clock 0.26 ± 12% +0.2 0.43 ± 17% perf-profile.children.cycles-pp.irq_exit 0.59 ± 17% +0.5 1.14 ± 8% perf-profile.children.cycles-pp.task_tick_fair 0.88 ± 18% +0.9 1.76 ± 9% perf-profile.children.cycles-pp.scheduler_tick 3.39 ± 12% +1.1 4.47 ± 9% perf-profile.children.cycles-pp.__hrtimer_run_queues 4.47 ± 11% +1.2 5.63 ± 10% perf-profile.children.cycles-pp.hrtimer_interrupt 5.38 ± 10% +1.2 6.62 ± 11% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 5.93 ± 10% +1.3 7.19 ± 11% perf-profile.children.cycles-pp.apic_timer_interrupt 1.19 ± 18% +1.3 2.48 ± 10% perf-profile.children.cycles-pp.update_process_times 1.23 ± 18% +1.4 2.58 ± 10% perf-profile.children.cycles-pp.tick_sched_handle 1.39 ± 17% +1.6 2.96 ± 10% perf-profile.children.cycles-pp.tick_sched_timer 0.22 ± 6% -0.1 0.07 ± 21% perf-profile.self.cycles-pp.rb_next 5.98 -0.1 5.85 perf-profile.self.cycles-pp.pthread_mutex_unlock 0.25 ± 10% -0.1 0.14 ± 15% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.11 ± 18% -0.1 0.06 ± 65% perf-profile.self.cycles-pp.rcu_irq_enter 0.04 ± 58% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.prepare_exit_to_usermode 0.07 ± 10% +0.0 0.11 ± 14% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.15 ± 16% +0.0 0.19 ± 6% perf-profile.self.cycles-pp.update_curr 0.06 ± 20% +0.0 0.11 ± 10% perf-profile.self.cycles-pp.cpuacct_charge 0.04 ± 58% +0.1 0.09 ± 19% perf-profile.self.cycles-pp.__softirqentry_text_start 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.raise_softirq 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.cpuacct_account_field 0.03 ±100% +0.1 0.09 ± 17% perf-profile.self.cycles-pp.run_timer_softirq 0.01 ±173% +0.1 0.08 ± 21% perf-profile.self.cycles-pp.account_user_time 0.08 ± 17% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.01 ±173% +0.1 0.08 ± 29% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.00 +0.1 0.07 ± 11% perf-profile.self.cycles-pp.hrtimer_active 0.00 +0.1 0.08 ± 24% perf-profile.self.cycles-pp.x86_pmu_disable 0.07 ± 20% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.__update_load_avg_se 0.09 ± 19% +0.1 0.18 ± 7% perf-profile.self.cycles-pp._raw_spin_lock 0.01 ±173% +0.1 0.10 ± 14% perf-profile.self.cycles-pp.profile_tick 0.04 ± 58% +0.1 0.13 ± 12% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.03 ±102% +0.1 0.12 ± 13% perf-profile.self.cycles-pp.tick_sched_timer 0.11 ± 11% +0.1 0.21 ± 12% perf-profile.self.cycles-pp.ktime_get 0.00 +0.1 0.13 ± 16% perf-profile.self.cycles-pp.run_local_timers 0.00 +0.1 0.14 ± 22% perf-profile.self.cycles-pp.update_rq_clock 0.11 ± 20% +0.1 0.25 ± 10% perf-profile.self.cycles-pp.task_tick_fair 3759 +52.3% 5725 ± 33% interrupts.CPU0.NMI:Non-maskable_interrupts 3759 +52.3% 5725 ± 33% interrupts.CPU0.PMI:Performance_monitoring_interrupts 4752 ± 34% +40.9% 6695 ± 24% interrupts.CPU1.NMI:Non-maskable_interrupts 4752 ± 34% +40.9% 6695 ± 24% interrupts.CPU1.PMI:Performance_monitoring_interrupts 607.50 ± 12% +38.6% 842.25 ± 12% interrupts.CPU10.RES:Rescheduling_interrupts 4635 ± 34% +62.9% 7551 interrupts.CPU104.NMI:Non-maskable_interrupts 4635 ± 34% +62.9% 7551 interrupts.CPU104.PMI:Performance_monitoring_interrupts 896.00 ± 40% -45.1% 492.25 ± 79% interrupts.CPU106.RES:Rescheduling_interrupts 28.50 ± 54% +860.5% 273.75 ± 64% interrupts.CPU107.RES:Rescheduling_interrupts 115.00 ± 48% -64.8% 40.50 ± 21% interrupts.CPU119.RES:Rescheduling_interrupts 4633 ± 34% +22.5% 5675 ± 33% interrupts.CPU124.NMI:Non-maskable_interrupts 4633 ± 34% +22.5% 5675 ± 33% interrupts.CPU124.PMI:Performance_monitoring_interrupts 3724 +52.3% 5673 ± 33% interrupts.CPU129.NMI:Non-maskable_interrupts 3724 +52.3% 5673 ± 33% interrupts.CPU129.PMI:Performance_monitoring_interrupts 3705 +54.9% 5741 ± 33% interrupts.CPU13.NMI:Non-maskable_interrupts 3705 +54.9% 5741 ± 33% interrupts.CPU13.PMI:Performance_monitoring_interrupts 537.50 ± 57% +136.8% 1273 ± 53% interrupts.CPU130.RES:Rescheduling_interrupts 4635 ± 34% +22.8% 5692 ± 33% interrupts.CPU131.NMI:Non-maskable_interrupts 4635 ± 34% +22.8% 5692 ± 33% interrupts.CPU131.PMI:Performance_monitoring_interrupts 4648 ± 34% +42.1% 6603 ± 24% interrupts.CPU147.NMI:Non-maskable_interrupts 4648 ± 34% +42.1% 6603 ± 24% interrupts.CPU147.PMI:Performance_monitoring_interrupts 3720 +52.9% 5689 ± 33% interrupts.CPU148.NMI:Non-maskable_interrupts 3720 +52.9% 5689 ± 33% interrupts.CPU148.PMI:Performance_monitoring_interrupts 3725 +52.9% 5695 ± 33% interrupts.CPU149.NMI:Non-maskable_interrupts 3725 +52.9% 5695 ± 33% interrupts.CPU149.PMI:Performance_monitoring_interrupts 4626 ± 34% +23.3% 5704 ± 33% interrupts.CPU152.NMI:Non-maskable_interrupts 4626 ± 34% +23.3% 5704 ± 33% interrupts.CPU152.PMI:Performance_monitoring_interrupts 4657 ± 34% +42.4% 6630 ± 24% interrupts.CPU154.NMI:Non-maskable_interrupts 4657 ± 34% +42.4% 6630 ± 24% interrupts.CPU154.PMI:Performance_monitoring_interrupts 260.00 ± 59% -55.9% 114.75 ± 31% interrupts.CPU156.RES:Rescheduling_interrupts 3729 +77.2% 6608 ± 24% interrupts.CPU163.NMI:Non-maskable_interrupts 3729 +77.2% 6608 ± 24% interrupts.CPU163.PMI:Performance_monitoring_interrupts 4634 ± 34% +22.6% 5681 ± 33% interrupts.CPU164.NMI:Non-maskable_interrupts 4634 ± 34% +22.6% 5681 ± 33% interrupts.CPU164.PMI:Performance_monitoring_interrupts 95.25 ± 48% +415.7% 491.25 ± 64% interrupts.CPU164.RES:Rescheduling_interrupts 7422 -48.8% 3798 interrupts.CPU168.NMI:Non-maskable_interrupts 7422 -48.8% 3798 interrupts.CPU168.PMI:Performance_monitoring_interrupts 3717 +52.6% 5672 ± 33% interrupts.CPU173.NMI:Non-maskable_interrupts 3717 +52.6% 5672 ± 33% interrupts.CPU173.PMI:Performance_monitoring_interrupts 4682 ± 34% +21.4% 5682 ± 33% interrupts.CPU179.NMI:Non-maskable_interrupts 4682 ± 34% +21.4% 5682 ± 33% interrupts.CPU179.PMI:Performance_monitoring_interrupts 4675 ± 33% +23.6% 5778 ± 33% interrupts.CPU19.NMI:Non-maskable_interrupts 4675 ± 33% +23.6% 5778 ± 33% interrupts.CPU19.PMI:Performance_monitoring_interrupts 3718 +52.6% 5675 ± 33% interrupts.CPU197.NMI:Non-maskable_interrupts 3718 +52.6% 5675 ± 33% interrupts.CPU197.PMI:Performance_monitoring_interrupts 3801 +77.3% 6740 ± 24% interrupts.CPU2.NMI:Non-maskable_interrupts 3801 +77.3% 6740 ± 24% interrupts.CPU2.PMI:Performance_monitoring_interrupts 3702 +78.6% 6612 ± 24% interrupts.CPU204.NMI:Non-maskable_interrupts 3702 +78.6% 6612 ± 24% interrupts.CPU204.PMI:Performance_monitoring_interrupts 3701 +52.9% 5660 ± 33% interrupts.CPU205.NMI:Non-maskable_interrupts 3701 +52.9% 5660 ± 33% interrupts.CPU205.PMI:Performance_monitoring_interrupts 156.50 ± 49% +160.7% 408.00 ± 18% interrupts.CPU206.RES:Rescheduling_interrupts 4657 ± 34% +21.6% 5662 ± 33% interrupts.CPU208.NMI:Non-maskable_interrupts 4657 ± 34% +21.6% 5662 ± 33% interrupts.CPU208.PMI:Performance_monitoring_interrupts 3725 +51.9% 5659 ± 33% interrupts.CPU209.NMI:Non-maskable_interrupts 3725 +51.9% 5659 ± 33% interrupts.CPU209.PMI:Performance_monitoring_interrupts 4616 ± 34% +23.2% 5684 ± 33% interrupts.CPU210.NMI:Non-maskable_interrupts 4616 ± 34% +23.2% 5684 ± 33% interrupts.CPU210.PMI:Performance_monitoring_interrupts 4632 ± 34% +43.5% 6645 ± 24% interrupts.CPU211.NMI:Non-maskable_interrupts 4632 ± 34% +43.5% 6645 ± 24% interrupts.CPU211.PMI:Performance_monitoring_interrupts 4570 ± 34% +43.5% 6557 ± 24% interrupts.CPU214.NMI:Non-maskable_interrupts 4570 ± 34% +43.5% 6557 ± 24% interrupts.CPU214.PMI:Performance_monitoring_interrupts 3698 +54.0% 5695 ± 33% interrupts.CPU217.NMI:Non-maskable_interrupts 3698 +54.0% 5695 ± 33% interrupts.CPU217.PMI:Performance_monitoring_interrupts 4659 ± 34% +42.1% 6622 ± 24% interrupts.CPU218.NMI:Non-maskable_interrupts 4659 ± 34% +42.1% 6622 ± 24% interrupts.CPU218.PMI:Performance_monitoring_interrupts 4647 ± 35% +42.8% 6636 ± 24% interrupts.CPU219.NMI:Non-maskable_interrupts 4647 ± 35% +42.8% 6636 ± 24% interrupts.CPU219.PMI:Performance_monitoring_interrupts 46.75 ± 39% +387.2% 227.75 ± 78% interrupts.CPU222.RES:Rescheduling_interrupts 18.00 ±102% +212.5% 56.25 ± 71% interrupts.CPU225.RES:Rescheduling_interrupts 4667 ± 34% +21.4% 5668 ± 33% interrupts.CPU231.NMI:Non-maskable_interrupts 4667 ± 34% +21.4% 5668 ± 33% interrupts.CPU231.PMI:Performance_monitoring_interrupts 4660 ± 34% +21.8% 5674 ± 33% interrupts.CPU232.NMI:Non-maskable_interrupts 4660 ± 34% +21.8% 5674 ± 33% interrupts.CPU232.PMI:Performance_monitoring_interrupts 4633 ± 34% +22.6% 5679 ± 33% interrupts.CPU237.NMI:Non-maskable_interrupts 4633 ± 34% +22.6% 5679 ± 33% interrupts.CPU237.PMI:Performance_monitoring_interrupts 4646 ± 34% +21.3% 5638 ± 33% interrupts.CPU238.NMI:Non-maskable_interrupts 4646 ± 34% +21.3% 5638 ± 33% interrupts.CPU238.PMI:Performance_monitoring_interrupts 3719 +52.1% 5657 ± 33% interrupts.CPU245.NMI:Non-maskable_interrupts 3719 +52.1% 5657 ± 33% interrupts.CPU245.PMI:Performance_monitoring_interrupts 4635 ± 34% +22.1% 5660 ± 32% interrupts.CPU255.NMI:Non-maskable_interrupts 4635 ± 34% +22.1% 5660 ± 32% interrupts.CPU255.PMI:Performance_monitoring_interrupts 54.50 ± 28% +242.2% 186.50 ± 54% interrupts.CPU256.RES:Rescheduling_interrupts 39.50 ± 19% +162.0% 103.50 ± 45% interrupts.CPU260.RES:Rescheduling_interrupts 4662 ± 34% +22.9% 5730 ± 33% interrupts.CPU267.NMI:Non-maskable_interrupts 4662 ± 34% +22.9% 5730 ± 33% interrupts.CPU267.PMI:Performance_monitoring_interrupts 4613 ± 34% +24.1% 5724 ± 33% interrupts.CPU269.NMI:Non-maskable_interrupts 4613 ± 34% +24.1% 5724 ± 33% interrupts.CPU269.PMI:Performance_monitoring_interrupts 4669 ± 34% +22.6% 5723 ± 33% interrupts.CPU27.NMI:Non-maskable_interrupts 4669 ± 34% +22.6% 5723 ± 33% interrupts.CPU27.PMI:Performance_monitoring_interrupts 4616 ± 34% +24.1% 5727 ± 33% interrupts.CPU274.NMI:Non-maskable_interrupts 4616 ± 34% +24.1% 5727 ± 33% interrupts.CPU274.PMI:Performance_monitoring_interrupts 3732 +52.4% 5689 ± 33% interrupts.CPU276.NMI:Non-maskable_interrupts 3732 +52.4% 5689 ± 33% interrupts.CPU276.PMI:Performance_monitoring_interrupts 3718 +53.7% 5713 ± 33% interrupts.CPU277.NMI:Non-maskable_interrupts 3718 +53.7% 5713 ± 33% interrupts.CPU277.PMI:Performance_monitoring_interrupts 4639 ± 34% +44.0% 6682 ± 24% interrupts.CPU278.NMI:Non-maskable_interrupts 4639 ± 34% +44.0% 6682 ± 24% interrupts.CPU278.PMI:Performance_monitoring_interrupts 3707 +79.7% 6663 ± 24% interrupts.CPU279.NMI:Non-maskable_interrupts 3707 +79.7% 6663 ± 24% interrupts.CPU279.PMI:Performance_monitoring_interrupts 133.50 ± 44% -80.1% 26.50 ± 36% interrupts.CPU279.RES:Rescheduling_interrupts 3722 +79.1% 6668 ± 24% interrupts.CPU281.NMI:Non-maskable_interrupts 3722 +79.1% 6668 ± 24% interrupts.CPU281.PMI:Performance_monitoring_interrupts 3726 +53.5% 5719 ± 33% interrupts.CPU283.NMI:Non-maskable_interrupts 3726 +53.5% 5719 ± 33% interrupts.CPU283.PMI:Performance_monitoring_interrupts 4642 ± 34% +43.8% 6678 ± 24% interrupts.CPU285.NMI:Non-maskable_interrupts 4642 ± 34% +43.8% 6678 ± 24% interrupts.CPU285.PMI:Performance_monitoring_interrupts 3775 +77.7% 6709 ± 24% interrupts.CPU32.NMI:Non-maskable_interrupts 3775 +77.7% 6709 ± 24% interrupts.CPU32.PMI:Performance_monitoring_interrupts 4681 ± 33% +22.9% 5755 ± 33% interrupts.CPU34.NMI:Non-maskable_interrupts 4681 ± 33% +22.9% 5755 ± 33% interrupts.CPU34.PMI:Performance_monitoring_interrupts 354.25 ± 9% +30.3% 461.75 ± 20% interrupts.CPU36.RES:Rescheduling_interrupts 337.75 ± 4% +30.6% 441.25 ± 19% interrupts.CPU38.RES:Rescheduling_interrupts 4633 ± 34% +23.7% 5729 ± 33% interrupts.CPU48.NMI:Non-maskable_interrupts 4633 ± 34% +23.7% 5729 ± 33% interrupts.CPU48.PMI:Performance_monitoring_interrupts 233.50 ± 18% +47.9% 345.25 ± 7% interrupts.CPU54.RES:Rescheduling_interrupts 4682 ± 34% +22.9% 5756 ± 33% interrupts.CPU6.NMI:Non-maskable_interrupts 4682 ± 34% +22.9% 5756 ± 33% interrupts.CPU6.PMI:Performance_monitoring_interrupts 656.50 ± 17% +54.4% 1013 ± 16% interrupts.CPU6.RES:Rescheduling_interrupts 3726 +53.9% 5736 ± 33% interrupts.CPU64.NMI:Non-maskable_interrupts 3726 +53.9% 5736 ± 33% interrupts.CPU64.PMI:Performance_monitoring_interrupts 3724 +53.3% 5708 ± 33% interrupts.CPU65.NMI:Non-maskable_interrupts 3724 +53.3% 5708 ± 33% interrupts.CPU65.PMI:Performance_monitoring_interrupts 4543 ± 33% +23.7% 5618 ± 33% interrupts.CPU71.NMI:Non-maskable_interrupts 4543 ± 33% +23.7% 5618 ± 33% interrupts.CPU71.PMI:Performance_monitoring_interrupts 902.50 ± 69% -68.4% 285.00 ± 27% interrupts.CPU72.RES:Rescheduling_interrupts 4647 ± 34% +22.2% 5677 ± 33% interrupts.CPU73.NMI:Non-maskable_interrupts 4647 ± 34% +22.2% 5677 ± 33% interrupts.CPU73.PMI:Performance_monitoring_interrupts 3736 +77.2% 6622 ± 24% interrupts.CPU79.NMI:Non-maskable_interrupts 3736 +77.2% 6622 ± 24% interrupts.CPU79.PMI:Performance_monitoring_interrupts 4705 ± 34% +22.2% 5750 ± 33% interrupts.CPU8.NMI:Non-maskable_interrupts 4705 ± 34% +22.2% 5750 ± 33% interrupts.CPU8.PMI:Performance_monitoring_interrupts 4666 ± 34% +41.7% 6613 ± 24% interrupts.CPU83.NMI:Non-maskable_interrupts 4666 ± 34% +41.7% 6613 ± 24% interrupts.CPU83.PMI:Performance_monitoring_interrupts 4644 ± 34% +22.3% 5680 ± 33% interrupts.CPU84.NMI:Non-maskable_interrupts 4644 ± 34% +22.3% 5680 ± 33% interrupts.CPU84.PMI:Performance_monitoring_interrupts 4653 ± 35% +22.1% 5682 ± 33% interrupts.CPU86.NMI:Non-maskable_interrupts 4653 ± 35% +22.1% 5682 ± 33% interrupts.CPU86.PMI:Performance_monitoring_interrupts 3712 +52.5% 5660 ± 33% interrupts.CPU87.NMI:Non-maskable_interrupts 3712 +52.5% 5660 ± 33% interrupts.CPU87.PMI:Performance_monitoring_interrupts 3741 +77.5% 6640 ± 24% interrupts.CPU88.NMI:Non-maskable_interrupts 3741 +77.5% 6640 ± 24% interrupts.CPU88.PMI:Performance_monitoring_interrupts 1278 ± 17% -49.1% 650.50 ± 51% interrupts.CPU88.RES:Rescheduling_interrupts 3716 +77.9% 6611 ± 24% interrupts.CPU89.NMI:Non-maskable_interrupts 3716 +77.9% 6611 ± 24% interrupts.CPU89.PMI:Performance_monitoring_interrupts 511.25 ± 14% +63.9% 837.75 ± 33% interrupts.CPU9.RES:Rescheduling_interrupts 4648 ± 34% +42.8% 6639 ± 24% interrupts.CPU91.NMI:Non-maskable_interrupts 4648 ± 34% +42.8% 6639 ± 24% interrupts.CPU91.PMI:Performance_monitoring_interrupts 4648 ± 34% +22.4% 5689 ± 33% interrupts.CPU92.NMI:Non-maskable_interrupts 4648 ± 34% +22.4% 5689 ± 33% interrupts.CPU92.PMI:Performance_monitoring_interrupts 3716 +103.5% 7565 interrupts.CPU95.NMI:Non-maskable_interrupts 3716 +103.5% 7565 interrupts.CPU95.PMI:Performance_monitoring_interrupts 4639 ± 34% +22.1% 5664 ± 33% interrupts.CPU96.NMI:Non-maskable_interrupts 4639 ± 34% +22.1% 5664 ± 33% interrupts.CPU96.PMI:Performance_monitoring_interrupts 3722 +53.1% 5699 ± 33% interrupts.CPU98.NMI:Non-maskable_interrupts 3722 +53.1% 5699 ± 33% interrupts.CPU98.PMI:Performance_monitoring_interrupts will-it-scale.per_thread_ops 7e+06 +-------------------------------------------------------------------+ | O O O O O O O O O O O O O O O O O O O | 6e+06 |.+..+.+.+ +.+..+.+.+..+.+.+..+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.| | : : | 5e+06 |-+ : : | | : : | 4e+06 |-+ : : | | : : | 3e+06 |-+ : : | | : : | 2e+06 |-+ : : | | : : | 1e+06 |-+ : | | : | 0 +-------------------------------------------------------------------+ will-it-scale.workload 2e+09 +-----------------------------------------------------------------+ 1.8e+09 |-O O O O O O O O O O O O O O O O O O O O | |.+..+.+.+ +.+.+.+..+.+.+..+.+.+.+..+.+.+..+.+.+.+..+.+.+.+..+.| 1.6e+09 |-+ : : | 1.4e+09 |-+ : : | | : : | 1.2e+09 |-+ : : | 1e+09 |-+ : : | 8e+08 |-+ : : | | : : | 6e+08 |-+ : : | 4e+08 |-+ :: | | :: | 2e+08 |-+ : | 0 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/malloc2/will-it-scale/0x11 commit: b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support") 492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers") b121b341e5983bdc 492e639f0c222784e2e0f121966 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 4:4 -100% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x %stddev %change %stddev \ | \ 969613 +2.5% 993613 will-it-scale.per_process_ops 9816 +1.7% 9984 will-it-scale.time.maximum_resident_set_size 2.792e+08 +2.5% 2.862e+08 will-it-scale.workload 329.25 ± 13% +43.7% 473.00 cpuidle.POLL.usage 7397888 ± 5% +17.6% 8699904 meminfo.DirectMap2M 2259 ± 2% +7.5% 2429 vmstat.system.cs 5771 ± 5% -7.8% 5323 numa-vmstat.node1.nr_slab_reclaimable 3003 ± 13% +34.0% 4024 numa-vmstat.node1.nr_slab_unreclaimable 23875 +3.1% 24612 proc-vmstat.nr_shmem 1959 +3.3% 2024 proc-vmstat.pgactivate 23087 ± 5% -7.8% 21292 numa-meminfo.node1.KReclaimable 23087 ± 5% -7.8% 21292 numa-meminfo.node1.SReclaimable 12013 ± 13% +34.0% 16096 numa-meminfo.node1.SUnreclaim 1094 ± 9% +20.5% 1319 slabinfo.dmaengine-unmap-16.active_objs 1094 ± 9% +20.5% 1319 slabinfo.dmaengine-unmap-16.num_objs 1573 ± 4% -18.9% 1275 slabinfo.file_lock_cache.active_objs 1573 ± 4% -18.9% 1275 slabinfo.file_lock_cache.num_objs 20496 ± 2% -10.1% 18432 slabinfo.skbuff_head_cache.active_objs 20496 ± 2% -10.1% 18432 slabinfo.skbuff_head_cache.num_objs 4.371e+10 +2.6% 4.486e+10 perf-stat.i.branch-instructions 4.12 -0.1 4.05 perf-stat.i.branch-miss-rate% 2.36 -0.0 2.31 perf-stat.i.cache-miss-rate% 8.193e+08 +2.1% 8.365e+08 perf-stat.i.cache-references 2224 ± 2% +7.9% 2400 perf-stat.i.context-switches 4.33e+11 +2.3% 4.428e+11 perf-stat.i.cpu-cycles 279.15 -1.5% 275.01 perf-stat.i.cpu-migrations 9.822e+08 +1.7% 9.991e+08 perf-stat.i.iTLB-load-misses 1.56e+11 +2.3% 1.596e+11 perf-stat.i.iTLB-loads 1.554e+11 +2.6% 1.594e+11 perf-stat.i.instructions 1.52 +2.2% 1.55 perf-stat.i.metric.GHz 0.03 +3.7% 0.03 perf-stat.i.metric.K/sec 701.66 +2.3% 717.66 perf-stat.i.metric.M/sec 2740 +2.8% 2816 perf-stat.i.minor-faults 2740 +2.8% 2816 perf-stat.i.page-faults 4.10 -0.1 4.04 perf-stat.overall.branch-miss-rate% 2.24 -0.1 2.19 perf-stat.overall.cache-miss-rate% 4.36e+10 +2.6% 4.473e+10 perf-stat.ps.branch-instructions 8.182e+08 +2.1% 8.35e+08 perf-stat.ps.cache-references 2185 ± 2% +7.9% 2359 perf-stat.ps.context-switches 4.32e+11 +2.3% 4.418e+11 perf-stat.ps.cpu-cycles 269.08 -1.6% 264.85 perf-stat.ps.cpu-migrations 9.792e+08 +1.7% 9.958e+08 perf-stat.ps.iTLB-load-misses 1.556e+11 +2.3% 1.591e+11 perf-stat.ps.iTLB-loads 1.55e+11 +2.5% 1.59e+11 perf-stat.ps.instructions 2694 +2.5% 2760 perf-stat.ps.minor-faults 2694 +2.5% 2760 perf-stat.ps.page-faults 4.754e+13 +2.7% 4.882e+13 perf-stat.total.instructions 3888 ± 8% -14.7% 3317 sched_debug.cfs_rq:/.load.avg 137829 ± 62% -80.9% 26264 sched_debug.cfs_rq:/.load.max 2980 -19.8% 2390 sched_debug.cfs_rq:/.load.min 9697 ± 53% -75.3% 2397 sched_debug.cfs_rq:/.load.stddev 3.52 ± 9% -15.9% 2.97 sched_debug.cfs_rq:/.load_avg.avg 105.71 ± 63% -72.1% 29.50 sched_debug.cfs_rq:/.load_avg.max 8.21 ± 50% -66.4% 2.76 sched_debug.cfs_rq:/.load_avg.stddev 1.30 -13.4% 1.12 sched_debug.cfs_rq:/.nr_spread_over.avg 407305 ± 5% +17.4% 478222 sched_debug.cfs_rq:/.spread0.max -5404888 -3.2% -5231353 sched_debug.cfs_rq:/.spread0.min 0.83 +160.0% 2.17 sched_debug.cfs_rq:/.util_est_enqueued.min 7576 -9.1% 6888 sched_debug.cpu.curr->pid.max 335.97 ± 5% +11.0% 373.05 sched_debug.cpu.curr->pid.stddev 23888 ± 52% +214.7% 75185 sched_debug.cpu.nr_switches.max 878.67 ± 17% -25.1% 658.33 sched_debug.cpu.nr_switches.min 2016 ± 30% +129.0% 4618 sched_debug.cpu.nr_switches.stddev 0.00 ± 20% -100.0% 0.00 sched_debug.cpu.nr_uninterruptible.avg 16513 ± 86% +329.7% 70964 sched_debug.cpu.sched_count.max 1557 ± 42% +176.4% 4305 sched_debug.cpu.sched_count.stddev 2278 +29.0% 2939 sched_debug.cpu.sched_goidle.max 202.41 +11.1% 224.95 sched_debug.cpu.sched_goidle.stddev 6738 ± 44% +230.4% 22261 sched_debug.cpu.ttwu_count.max 673.86 ± 23% +112.5% 1431 sched_debug.cpu.ttwu_count.stddev 4944 ± 77% +343.2% 21913 sched_debug.cpu.ttwu_local.max 463.98 ± 43% +189.1% 1341 sched_debug.cpu.ttwu_local.stddev 0.20 ± 74% -100.0% 0.00 sched_debug.rt_rq:/.rt_time.avg 57.79 ± 74% -100.0% 0.00 sched_debug.rt_rq:/.rt_time.max 3.40 ± 74% -100.0% 0.00 sched_debug.rt_rq:/.rt_time.stddev 50.90 ± 10% -11.0 39.91 perf-profile.calltrace.cycles-pp.malloc 17.95 ± 10% -3.5 14.47 perf-profile.calltrace.cycles-pp.free 2.32 ± 30% -1.5 0.84 perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 2.02 ± 14% -1.0 1.02 perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.05 ± 28% -0.3 0.75 perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.65 ± 17% +0.3 0.97 perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.13 ±173% +0.4 0.55 perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.89 ± 13% +0.4 2.34 perf-profile.calltrace.cycles-pp.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop.prepare_exit_to_usermode 1.89 ± 13% +0.5 2.34 perf-profile.calltrace.cycles-pp.change_protection.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop 0.30 ±101% +0.5 0.75 perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.92 ± 13% +0.5 2.38 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 1.90 ± 13% +0.5 2.36 perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode 0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 +0.5 0.51 perf-profile.calltrace.cycles-pp.__acct_update_integrals.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.00 +0.6 0.55 perf-profile.calltrace.cycles-pp.__run_timers.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt 0.00 +0.6 0.60 perf-profile.calltrace.cycles-pp.cpuacct_charge.update_curr.task_tick_fair.scheduler_tick.update_process_times 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp._raw_spin_lock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 0.13 ±173% +0.7 0.84 perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.task_tick_fair.scheduler_tick.update_process_times 0.13 ±173% +0.8 0.91 perf-profile.calltrace.cycles-pp.__update_load_avg_se.update_load_avg.task_tick_fair.scheduler_tick.update_process_times 0.00 +0.9 0.89 perf-profile.calltrace.cycles-pp.run_local_timers.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.00 +0.9 0.95 perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.00 +1.0 0.98 perf-profile.calltrace.cycles-pp.account_user_time.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 1.02 ± 22% +1.0 2.01 perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 0.00 +1.0 0.99 perf-profile.calltrace.cycles-pp.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 +1.0 1.00 perf-profile.calltrace.cycles-pp.ksoftirqd_running.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 2.62 ± 16% +1.0 3.63 perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.00 +1.1 1.11 perf-profile.calltrace.cycles-pp.run_posix_cpu_timers.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +1.1 1.15 perf-profile.calltrace.cycles-pp.update_rq_clock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 1.15 ± 23% +1.6 2.78 perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle 0.71 ± 36% +1.8 2.54 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 1.20 ± 30% +2.8 4.00 perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 2.75 ± 24% +3.7 6.47 perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer 4.15 ± 25% +6.5 10.63 perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 6.49 ± 18% +8.8 15.25 perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 14.71 ± 23% +8.8 23.52 perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 19.68 ± 24% +9.5 29.17 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 6.81 ± 19% +9.8 16.62 perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 7.62 ± 19% +10.9 18.52 perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 26.15 ± 25% +13.0 39.11 perf-profile.calltrace.cycles-pp.apic_timer_interrupt 23.59 ± 25% +13.2 36.78 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt 51.00 ± 10% -11.0 40.04 perf-profile.children.cycles-pp.malloc 16.65 ± 10% -3.3 13.31 perf-profile.children.cycles-pp.free 2.71 ± 25% -1.8 0.92 perf-profile.children.cycles-pp.__remove_hrtimer 2.36 ± 11% -1.2 1.15 perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 1.86 ± 12% -0.4 1.43 perf-profile.children.cycles-pp.malloc@plt 1.26 ± 23% -0.4 0.84 perf-profile.children.cycles-pp.timerqueue_del 0.88 ± 29% -0.4 0.48 perf-profile.children.cycles-pp.rb_next 0.67 ± 28% -0.3 0.36 perf-profile.children.cycles-pp.rcu_irq_enter 0.17 ± 12% -0.1 0.05 perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.37 ± 12% -0.0 0.32 perf-profile.children.cycles-pp.native_write_msr 0.25 ± 9% -0.0 0.21 perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.14 ± 5% -0.0 0.12 perf-profile.children.cycles-pp.irq_work_tick 0.07 ± 13% +0.0 0.10 perf-profile.children.cycles-pp.irq_work_run_list 0.00 +0.1 0.05 perf-profile.children.cycles-pp.account_entity_enqueue 0.00 +0.1 0.05 perf-profile.children.cycles-pp.rebalance_domains 0.00 +0.1 0.05 perf-profile.children.cycles-pp.account_process_tick 0.14 ± 19% +0.1 0.19 perf-profile.children.cycles-pp.ret_from_intr 0.30 ± 7% +0.1 0.36 perf-profile.children.cycles-pp.vm_normal_page 0.18 ± 13% +0.1 0.24 perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.00 +0.1 0.06 perf-profile.children.cycles-pp.rcu_qs 0.17 ± 9% +0.1 0.24 perf-profile.children.cycles-pp.x86_pmu_disable 0.01 ±173% +0.1 0.08 perf-profile.children.cycles-pp.nohz_balance_exit_idle 0.00 +0.1 0.07 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.00 +0.1 0.07 perf-profile.children.cycles-pp.__note_gp_changes 0.23 ± 19% +0.1 0.30 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.01 ±173% +0.1 0.10 perf-profile.children.cycles-pp.update_min_vruntime 0.03 ±102% +0.1 0.13 perf-profile.children.cycles-pp.account_entity_dequeue 0.11 ± 17% +0.1 0.22 perf-profile.children.cycles-pp.update_cfs_group 0.10 ± 15% +0.1 0.20 perf-profile.children.cycles-pp.native_read_msr 0.39 ± 11% +0.1 0.50 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.28 ± 18% +0.1 0.39 perf-profile.children.cycles-pp.lapic_next_deadline 0.09 ± 19% +0.1 0.20 perf-profile.children.cycles-pp.__accumulate_pelt_segments 0.39 ± 9% +0.1 0.50 perf-profile.children.cycles-pp.timerqueue_add 0.48 ± 12% +0.1 0.61 perf-profile.children.cycles-pp.enqueue_hrtimer 0.11 ± 23% +0.1 0.25 perf-profile.children.cycles-pp.__calc_delta 0.05 ± 67% +0.2 0.21 perf-profile.children.cycles-pp.note_gp_changes 0.14 ± 27% +0.2 0.31 perf-profile.children.cycles-pp.trigger_load_balance 0.02 ±173% +0.2 0.19 perf-profile.children.cycles-pp.calc_global_load_tick 0.09 ± 17% +0.2 0.27 perf-profile.children.cycles-pp.native_sched_clock 0.07 ± 6% +0.2 0.25 perf-profile.children.cycles-pp.hrtimer_run_queues 0.18 ± 18% +0.2 0.37 perf-profile.children.cycles-pp.arch_scale_freq_tick 0.10 ± 17% +0.2 0.29 perf-profile.children.cycles-pp.sched_clock 0.16 ± 23% +0.2 0.39 perf-profile.children.cycles-pp.reweight_entity 0.68 ± 13% +0.2 0.91 perf-profile.children.cycles-pp.ktime_get 0.11 ± 17% +0.2 0.34 perf-profile.children.cycles-pp.sched_clock_cpu 0.61 ± 17% +0.2 0.85 perf-profile.children.cycles-pp.clockevents_program_event 0.21 ± 7% +0.3 0.50 perf-profile.children.cycles-pp.cpuacct_account_field 0.09 ± 19% +0.3 0.42 perf-profile.children.cycles-pp.hrtimer_active 2.15 ± 8% +0.3 2.48 perf-profile.children.cycles-pp.change_p4d_range 2.13 ± 8% +0.3 2.46 perf-profile.children.cycles-pp.change_pte_range 0.81 ± 14% +0.3 1.15 perf-profile.children.cycles-pp.ktime_get_update_offsets_now 2.15 ± 8% +0.3 2.49 perf-profile.children.cycles-pp.change_protection 2.15 ± 8% +0.3 2.49 perf-profile.children.cycles-pp.change_prot_numa 0.15 ± 13% +0.3 0.49 perf-profile.children.cycles-pp.raise_softirq 0.23 ± 9% +0.3 0.57 perf-profile.children.cycles-pp.__acct_update_integrals 2.16 ± 8% +0.3 2.51 perf-profile.children.cycles-pp.task_work_run 2.16 ± 8% +0.3 2.51 perf-profile.children.cycles-pp.task_numa_work 0.15 ± 16% +0.3 0.50 perf-profile.children.cycles-pp.profile_tick 2.19 ± 8% +0.3 2.54 perf-profile.children.cycles-pp.exit_to_usermode_loop 0.31 ± 9% +0.4 0.67 perf-profile.children.cycles-pp.cpuacct_charge 0.37 ± 15% +0.4 0.75 perf-profile.children.cycles-pp._raw_spin_lock_irq 0.55 ± 10% +0.4 0.94 perf-profile.children.cycles-pp._raw_spin_lock 0.14 ± 34% +0.4 0.58 perf-profile.children.cycles-pp.rcu_core 2.58 ± 9% +0.5 3.04 perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.46 ± 18% +0.5 0.95 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.44 ± 16% +0.6 1.02 perf-profile.children.cycles-pp.__update_load_avg_se 0.24 ± 30% +0.6 0.84 perf-profile.children.cycles-pp.__run_timers 0.47 ± 3% +0.6 1.08 perf-profile.children.cycles-pp.account_user_time 0.45 ± 14% +0.6 1.07 perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.30 ± 7% +0.7 0.99 perf-profile.children.cycles-pp.run_local_timers 0.28 ± 31% +0.8 1.10 perf-profile.children.cycles-pp.run_timer_softirq 0.17 ± 44% +0.9 1.11 perf-profile.children.cycles-pp.ksoftirqd_running 2.99 ± 11% +0.9 3.94 perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.26 ± 45% +1.0 1.23 perf-profile.children.cycles-pp.run_posix_cpu_timers 0.25 ± 23% +1.0 1.27 perf-profile.children.cycles-pp.update_rq_clock 1.23 ± 19% +1.1 2.31 perf-profile.children.cycles-pp.update_curr 1.35 ± 19% +1.8 3.10 perf-profile.children.cycles-pp.update_load_avg 0.83 ± 31% +2.0 2.85 perf-profile.children.cycles-pp.__softirqentry_text_start 1.40 ± 27% +3.1 4.46 perf-profile.children.cycles-pp.irq_exit 3.20 ± 20% +4.0 7.18 perf-profile.children.cycles-pp.task_tick_fair 4.84 ± 21% +6.1 10.97 perf-profile.children.cycles-pp.scheduler_tick 6.83 ± 17% +8.9 15.73 perf-profile.children.cycles-pp.update_process_times 17.23 ± 19% +9.0 26.21 perf-profile.children.cycles-pp.__hrtimer_run_queues 22.90 ± 19% +9.5 32.43 perf-profile.children.cycles-pp.hrtimer_interrupt 7.16 ± 18% +10.0 17.12 perf-profile.children.cycles-pp.tick_sched_handle 8.02 ± 18% +11.1 19.09 perf-profile.children.cycles-pp.tick_sched_timer 29.66 ± 21% +13.1 42.74 perf-profile.children.cycles-pp.apic_timer_interrupt 27.48 ± 21% +13.2 40.69 perf-profile.children.cycles-pp.smp_apic_timer_interrupt 46.00 ± 10% -10.4 35.62 perf-profile.self.cycles-pp.malloc 13.39 ± 11% -2.9 10.47 perf-profile.self.cycles-pp.free 1.45 ± 27% -1.4 0.09 perf-profile.self.cycles-pp.__remove_hrtimer 1.41 ± 12% -0.9 0.55 perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 1.75 ± 12% -0.4 1.31 perf-profile.self.cycles-pp.malloc@plt 0.84 ± 29% -0.4 0.47 perf-profile.self.cycles-pp.rb_next 0.66 ± 29% -0.3 0.36 perf-profile.self.cycles-pp.rcu_irq_enter 0.25 ± 14% -0.0 0.20 perf-profile.self.cycles-pp.timerqueue_del 0.24 ± 9% -0.0 0.20 perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.36 ± 12% -0.0 0.32 perf-profile.self.cycles-pp.native_write_msr 0.14 ± 5% -0.0 0.11 perf-profile.self.cycles-pp.irq_work_tick 0.10 ± 17% +0.0 0.13 perf-profile.self.cycles-pp.lapic_next_deadline 0.07 ± 7% +0.0 0.11 perf-profile.self.cycles-pp.update_process_times 0.26 ± 9% +0.0 0.31 perf-profile.self.cycles-pp.vm_normal_page 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__note_gp_changes 0.00 +0.1 0.05 perf-profile.self.cycles-pp.account_entity_enqueue 0.00 +0.1 0.06 perf-profile.self.cycles-pp.rcu_qs 0.11 ± 14% +0.1 0.17 perf-profile.self.cycles-pp.clockevents_program_event 0.01 ±173% +0.1 0.08 perf-profile.self.cycles-pp.irq_work_run_list 0.01 ±173% +0.1 0.08 perf-profile.self.cycles-pp.nohz_balance_exit_idle 0.00 +0.1 0.07 perf-profile.self.cycles-pp.sched_clock_cpu 0.00 +0.1 0.07 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.17 ± 12% +0.1 0.24 perf-profile.self.cycles-pp.x86_pmu_disable 0.01 ±173% +0.1 0.09 perf-profile.self.cycles-pp.update_min_vruntime 0.21 ± 18% +0.1 0.29 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.09 ± 21% +0.1 0.17 perf-profile.self.cycles-pp.arch_scale_freq_tick 0.23 ± 13% +0.1 0.32 perf-profile.self.cycles-pp.prepare_exit_to_usermode 0.06 ± 20% +0.1 0.15 perf-profile.self.cycles-pp.tick_sched_handle 0.00 +0.1 0.09 perf-profile.self.cycles-pp.note_gp_changes 0.03 ±100% +0.1 0.12 perf-profile.self.cycles-pp.account_entity_dequeue 0.30 ± 9% +0.1 0.39 perf-profile.self.cycles-pp.irq_exit 0.11 ± 19% +0.1 0.21 perf-profile.self.cycles-pp.update_cfs_group 0.39 ± 11% +0.1 0.49 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.08 ± 26% +0.1 0.19 perf-profile.self.cycles-pp.reweight_entity 0.09 ± 17% +0.1 0.20 perf-profile.self.cycles-pp.native_read_msr 0.09 ± 24% +0.1 0.20 perf-profile.self.cycles-pp.__accumulate_pelt_segments 0.11 ± 26% +0.1 0.23 perf-profile.self.cycles-pp.trigger_load_balance 0.31 ± 7% +0.1 0.43 perf-profile.self.cycles-pp.timerqueue_add 0.11 ± 19% +0.1 0.25 perf-profile.self.cycles-pp.__calc_delta 0.02 ±173% +0.2 0.18 perf-profile.self.cycles-pp.rcu_core 0.02 ±173% +0.2 0.18 perf-profile.self.cycles-pp.calc_global_load_tick 0.09 ± 19% +0.2 0.26 perf-profile.self.cycles-pp.native_sched_clock 0.07 ± 7% +0.2 0.25 perf-profile.self.cycles-pp.hrtimer_run_queues 0.53 ± 14% +0.2 0.72 perf-profile.self.cycles-pp.ktime_get 0.21 ± 7% +0.3 0.50 perf-profile.self.cycles-pp.cpuacct_account_field 0.27 +0.3 0.58 perf-profile.self.cycles-pp.account_user_time 0.09 ± 15% +0.3 0.42 perf-profile.self.cycles-pp.hrtimer_active 0.23 ± 9% +0.3 0.57 perf-profile.self.cycles-pp.__acct_update_integrals 0.15 ± 14% +0.3 0.49 perf-profile.self.cycles-pp.raise_softirq 0.15 ± 16% +0.3 0.50 perf-profile.self.cycles-pp.profile_tick 0.67 ± 12% +0.4 1.03 perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.30 ± 9% +0.4 0.67 perf-profile.self.cycles-pp.cpuacct_charge 0.12 ± 29% +0.4 0.49 perf-profile.self.cycles-pp.run_timer_softirq 0.37 ± 14% +0.4 0.75 perf-profile.self.cycles-pp._raw_spin_lock_irq 0.54 ± 9% +0.4 0.93 perf-profile.self.cycles-pp._raw_spin_lock 0.16 ± 32% +0.4 0.59 perf-profile.self.cycles-pp.__run_timers 0.42 ± 17% +0.5 0.87 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.23 ± 11% +0.5 0.69 perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.27 ± 19% +0.5 0.75 perf-profile.self.cycles-pp.tick_sched_timer 0.42 ± 25% +0.5 0.90 perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.23 ± 8% +0.5 0.73 perf-profile.self.cycles-pp.run_local_timers 0.39 ± 16% +0.5 0.90 perf-profile.self.cycles-pp.__update_load_avg_se 0.78 ± 24% +0.5 1.29 perf-profile.self.cycles-pp.update_curr 0.28 ± 29% +0.5 0.80 perf-profile.self.cycles-pp.task_tick_fair 0.27 ± 33% +0.5 0.81 perf-profile.self.cycles-pp.__softirqentry_text_start 0.45 ± 24% +0.7 1.14 perf-profile.self.cycles-pp.update_load_avg 0.14 ± 31% +0.8 0.93 perf-profile.self.cycles-pp.update_rq_clock 0.17 ± 44% +0.9 1.10 perf-profile.self.cycles-pp.ksoftirqd_running 0.26 ± 45% +1.0 1.22 perf-profile.self.cycles-pp.run_posix_cpu_timers 1251 ±134% +6253.9% 79487 interrupts.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3 1867 ± 12% -27.6% 1351 interrupts.CPU0.RES:Rescheduling_interrupts 551.25 ± 53% -45.2% 302.00 interrupts.CPU10.RES:Rescheduling_interrupts 140.00 ± 87% -92.9% 10.00 interrupts.CPU101.RES:Rescheduling_interrupts 1121 ± 88% -79.4% 231.00 interrupts.CPU102.RES:Rescheduling_interrupts 118.50 ± 31% -68.8% 37.00 interrupts.CPU103.RES:Rescheduling_interrupts 387.25 ± 43% +187.4% 1113 interrupts.CPU104.RES:Rescheduling_interrupts 1326 ± 84% -78.1% 291.00 interrupts.CPU106.RES:Rescheduling_interrupts 86.50 ± 19% -52.6% 41.00 interrupts.CPU109.RES:Rescheduling_interrupts 589.75 ± 18% -54.6% 268.00 interrupts.CPU11.RES:Rescheduling_interrupts 1711 ± 39% -85.9% 242.00 interrupts.CPU110.RES:Rescheduling_interrupts 97.75 ± 62% -83.6% 16.00 interrupts.CPU113.RES:Rescheduling_interrupts 897.75 ± 44% +145.7% 2206 interrupts.CPU120.RES:Rescheduling_interrupts 1079 ± 71% +310.7% 4433 interrupts.CPU122.RES:Rescheduling_interrupts 4608 ± 34% +63.4% 7532 interrupts.CPU123.NMI:Non-maskable_interrupts 4608 ± 34% +63.4% 7532 interrupts.CPU123.PMI:Performance_monitoring_interrupts 240.75 ±105% -93.8% 15.00 interrupts.CPU125.RES:Rescheduling_interrupts 134.00 ± 62% -95.5% 6.00 interrupts.CPU129.RES:Rescheduling_interrupts 1251 ±134% +6253.9% 79487 interrupts.CPU13.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3 607.50 ± 34% -58.7% 251.00 interrupts.CPU13.RES:Rescheduling_interrupts 3690 +103.5% 7510 interrupts.CPU130.NMI:Non-maskable_interrupts 3690 +103.5% 7510 interrupts.CPU130.PMI:Performance_monitoring_interrupts 3693 +103.6% 7521 interrupts.CPU131.NMI:Non-maskable_interrupts 3693 +103.6% 7521 interrupts.CPU131.PMI:Performance_monitoring_interrupts 4588 ± 33% +63.9% 7522 interrupts.CPU132.NMI:Non-maskable_interrupts 4588 ± 33% +63.9% 7522 interrupts.CPU132.PMI:Performance_monitoring_interrupts 1734 ± 91% -79.5% 356.00 interrupts.CPU132.RES:Rescheduling_interrupts 1104 ± 87% -85.2% 163.00 interrupts.CPU134.RES:Rescheduling_interrupts 3664 +105.1% 7515 interrupts.CPU136.NMI:Non-maskable_interrupts 3664 +105.1% 7515 interrupts.CPU136.PMI:Performance_monitoring_interrupts 685.00 ± 73% +233.7% 2286 interrupts.CPU138.RES:Rescheduling_interrupts 481.75 ± 25% -35.4% 311.00 interrupts.CPU14.RES:Rescheduling_interrupts 110.25 ± 27% -69.2% 34.00 interrupts.CPU143.RES:Rescheduling_interrupts 248.25 ± 46% +102.2% 502.00 interrupts.CPU144.RES:Rescheduling_interrupts 4612 ± 35% +62.9% 7514 interrupts.CPU145.NMI:Non-maskable_interrupts 4612 ± 35% +62.9% 7514 interrupts.CPU145.PMI:Performance_monitoring_interrupts 180.00 ± 75% -79.4% 37.00 interrupts.CPU145.RES:Rescheduling_interrupts 4620 ± 34% +62.4% 7506 interrupts.CPU149.NMI:Non-maskable_interrupts 4620 ± 34% +62.4% 7506 interrupts.CPU149.PMI:Performance_monitoring_interrupts 276.75 ±116% -97.8% 6.00 interrupts.CPU149.RES:Rescheduling_interrupts 4688 ± 35% +62.8% 7635 interrupts.CPU15.NMI:Non-maskable_interrupts 4688 ± 35% +62.8% 7635 interrupts.CPU15.PMI:Performance_monitoring_interrupts 495.25 ± 12% -47.7% 259.00 interrupts.CPU15.RES:Rescheduling_interrupts 409.25 ± 30% +97.2% 807.00 interrupts.CPU150.RES:Rescheduling_interrupts 4628 ± 34% +62.7% 7530 interrupts.CPU152.NMI:Non-maskable_interrupts 4628 ± 34% +62.7% 7530 interrupts.CPU152.PMI:Performance_monitoring_interrupts 4614 ± 34% +64.1% 7573 interrupts.CPU153.NMI:Non-maskable_interrupts 4614 ± 34% +64.1% 7573 interrupts.CPU153.PMI:Performance_monitoring_interrupts 212.00 ± 84% +230.2% 700.00 interrupts.CPU156.RES:Rescheduling_interrupts 4621 ± 34% +62.5% 7512 interrupts.CPU159.NMI:Non-maskable_interrupts 4621 ± 34% +62.5% 7512 interrupts.CPU159.PMI:Performance_monitoring_interrupts 4633 ± 34% +64.9% 7640 interrupts.CPU16.NMI:Non-maskable_interrupts 4633 ± 34% +64.9% 7640 interrupts.CPU16.PMI:Performance_monitoring_interrupts 4623 ± 35% +62.8% 7525 interrupts.CPU163.NMI:Non-maskable_interrupts 4623 ± 35% +62.8% 7525 interrupts.CPU163.PMI:Performance_monitoring_interrupts 4623 ± 35% +62.4% 7509 interrupts.CPU164.NMI:Non-maskable_interrupts 4623 ± 35% +62.4% 7509 interrupts.CPU164.PMI:Performance_monitoring_interrupts 4649 ± 34% +61.9% 7527 interrupts.CPU165.NMI:Non-maskable_interrupts 4649 ± 34% +61.9% 7527 interrupts.CPU165.PMI:Performance_monitoring_interrupts 226.75 ± 63% -78.4% 49.00 interrupts.CPU166.RES:Rescheduling_interrupts 100.75 ± 69% -94.0% 6.00 interrupts.CPU169.RES:Rescheduling_interrupts 425.25 ± 20% -42.4% 245.00 interrupts.CPU17.RES:Rescheduling_interrupts 4603 ± 35% +63.7% 7537 interrupts.CPU170.NMI:Non-maskable_interrupts 4603 ± 35% +63.7% 7537 interrupts.CPU170.PMI:Performance_monitoring_interrupts 4635 ± 34% +62.0% 7508 interrupts.CPU171.NMI:Non-maskable_interrupts 4635 ± 34% +62.0% 7508 interrupts.CPU171.PMI:Performance_monitoring_interrupts 165.00 ± 50% +269.1% 609.00 interrupts.CPU174.RES:Rescheduling_interrupts 219.50 ± 55% +134.2% 514.00 interrupts.CPU176.RES:Rescheduling_interrupts 3689 +104.2% 7535 interrupts.CPU177.NMI:Non-maskable_interrupts 3689 +104.2% 7535 interrupts.CPU177.PMI:Performance_monitoring_interrupts 250.50 ± 35% +108.8% 523.00 interrupts.CPU178.RES:Rescheduling_interrupts 534.50 ± 7% -44.1% 299.00 interrupts.CPU18.RES:Rescheduling_interrupts 1195 -48.7% 613.00 interrupts.CPU181.CAL:Function_call_interrupts 3684 +104.3% 7525 interrupts.CPU184.NMI:Non-maskable_interrupts 3684 +104.3% 7525 interrupts.CPU184.PMI:Performance_monitoring_interrupts 243.75 ± 19% +35.4% 330.00 interrupts.CPU190.RES:Rescheduling_interrupts 61.25 ± 59% -91.8% 5.00 interrupts.CPU191.RES:Rescheduling_interrupts 114.00 ±120% -94.7% 6.00 interrupts.CPU195.RES:Rescheduling_interrupts 538.75 ± 84% -77.4% 122.00 interrupts.CPU196.RES:Rescheduling_interrupts 193.50 ± 10% -95.3% 9.00 interrupts.CPU197.RES:Rescheduling_interrupts 454.00 ± 22% -55.5% 202.00 interrupts.CPU198.RES:Rescheduling_interrupts 764.75 ± 56% -58.9% 314.00 interrupts.CPU2.RES:Rescheduling_interrupts 463.50 ± 9% -23.0% 357.00 interrupts.CPU20.RES:Rescheduling_interrupts 243.50 ± 48% -96.3% 9.00 interrupts.CPU200.RES:Rescheduling_interrupts 82.25 ± 49% -83.0% 14.00 interrupts.CPU205.RES:Rescheduling_interrupts 349.75 ± 12% +120.4% 771.00 interrupts.CPU206.RES:Rescheduling_interrupts 222.25 ± 56% -80.7% 43.00 interrupts.CPU208.RES:Rescheduling_interrupts 4685 ± 35% +61.5% 7565 interrupts.CPU21.NMI:Non-maskable_interrupts 4685 ± 35% +61.5% 7565 interrupts.CPU21.PMI:Performance_monitoring_interrupts 394.50 ± 48% -92.1% 31.00 interrupts.CPU210.RES:Rescheduling_interrupts 457.50 ± 74% -78.6% 98.00 interrupts.CPU212.RES:Rescheduling_interrupts 107.50 ± 51% -86.0% 15.00 interrupts.CPU213.RES:Rescheduling_interrupts 79.75 ± 53% -88.7% 9.00 interrupts.CPU215.RES:Rescheduling_interrupts 137.25 ± 24% -46.8% 73.00 interrupts.CPU218.RES:Rescheduling_interrupts 84.25 ± 29% -90.5% 8.00 interrupts.CPU219.RES:Rescheduling_interrupts 3707 +105.5% 7619 interrupts.CPU22.NMI:Non-maskable_interrupts 3707 +105.5% 7619 interrupts.CPU22.PMI:Performance_monitoring_interrupts 512.75 ± 29% -43.6% 289.00 interrupts.CPU22.RES:Rescheduling_interrupts 157.75 ± 42% -77.8% 35.00 interrupts.CPU220.RES:Rescheduling_interrupts 93.00 ±103% -92.5% 7.00 interrupts.CPU221.RES:Rescheduling_interrupts 73.75 ± 22% -67.5% 24.00 interrupts.CPU222.RES:Rescheduling_interrupts 4615 ± 35% +63.0% 7521 interrupts.CPU225.NMI:Non-maskable_interrupts 4615 ± 35% +63.0% 7521 interrupts.CPU225.PMI:Performance_monitoring_interrupts 75.00 ± 44% -93.3% 5.00 interrupts.CPU225.RES:Rescheduling_interrupts 144.75 ± 46% -76.5% 34.00 interrupts.CPU226.RES:Rescheduling_interrupts 3740 +104.5% 7648 interrupts.CPU23.NMI:Non-maskable_interrupts 3740 +104.5% 7648 interrupts.CPU23.PMI:Performance_monitoring_interrupts 404.75 ± 23% -38.7% 248.00 interrupts.CPU23.RES:Rescheduling_interrupts 154.75 ± 62% +122.9% 345.00 interrupts.CPU232.RES:Rescheduling_interrupts 69.25 ± 32% -85.6% 10.00 interrupts.CPU233.RES:Rescheduling_interrupts 121.50 ± 53% -76.1% 29.00 interrupts.CPU236.RES:Rescheduling_interrupts 3772 +102.4% 7636 interrupts.CPU24.NMI:Non-maskable_interrupts 3772 +102.4% 7636 interrupts.CPU24.PMI:Performance_monitoring_interrupts 683.25 ± 43% -58.4% 284.00 interrupts.CPU24.RES:Rescheduling_interrupts 218.25 ±101% -78.9% 46.00 interrupts.CPU240.RES:Rescheduling_interrupts 4616 ± 34% +62.7% 7509 interrupts.CPU242.NMI:Non-maskable_interrupts 4616 ± 34% +62.7% 7509 interrupts.CPU242.PMI:Performance_monitoring_interrupts 108.50 ± 30% -39.2% 66.00 interrupts.CPU242.RES:Rescheduling_interrupts 113.25 ± 44% -93.8% 7.00 interrupts.CPU243.RES:Rescheduling_interrupts 60.50 ± 83% -88.4% 7.00 interrupts.CPU247.RES:Rescheduling_interrupts 161.75 ±125% -96.9% 5.00 interrupts.CPU249.RES:Rescheduling_interrupts 239.50 ± 65% -80.0% 48.00 interrupts.CPU250.RES:Rescheduling_interrupts 105.25 ± 51% +286.7% 407.00 interrupts.CPU252.RES:Rescheduling_interrupts 4622 ± 35% +62.5% 7510 interrupts.CPU253.NMI:Non-maskable_interrupts 4622 ± 35% +62.5% 7510 interrupts.CPU253.PMI:Performance_monitoring_interrupts 151.50 ± 36% -48.5% 78.00 interrupts.CPU256.RES:Rescheduling_interrupts 3669 +105.7% 7547 interrupts.CPU257.NMI:Non-maskable_interrupts 3669 +105.7% 7547 interrupts.CPU257.PMI:Performance_monitoring_interrupts 63.75 ± 52% -90.6% 6.00 interrupts.CPU257.RES:Rescheduling_interrupts 168.00 ± 56% -64.9% 59.00 interrupts.CPU258.RES:Rescheduling_interrupts 90.75 ± 86% -91.2% 8.00 interrupts.CPU259.RES:Rescheduling_interrupts 1164 ± 63% -75.5% 285.00 interrupts.CPU26.RES:Rescheduling_interrupts 4592 ± 34% +63.5% 7507 interrupts.CPU260.NMI:Non-maskable_interrupts 4592 ± 34% +63.5% 7507 interrupts.CPU260.PMI:Performance_monitoring_interrupts 119.25 ± 28% -74.0% 31.00 interrupts.CPU260.RES:Rescheduling_interrupts 4595 ± 35% +63.3% 7505 interrupts.CPU262.NMI:Non-maskable_interrupts 4595 ± 35% +63.3% 7505 interrupts.CPU262.PMI:Performance_monitoring_interrupts 4590 ± 34% +63.7% 7514 interrupts.CPU263.NMI:Non-maskable_interrupts 4590 ± 34% +63.7% 7514 interrupts.CPU263.PMI:Performance_monitoring_interrupts 62.25 ± 56% -92.0% 5.00 interrupts.CPU263.RES:Rescheduling_interrupts 98.50 ± 36% +383.2% 476.00 interrupts.CPU264.RES:Rescheduling_interrupts 51.25 ± 80% -86.3% 7.00 interrupts.CPU265.RES:Rescheduling_interrupts 244.75 ± 84% -86.1% 34.00 interrupts.CPU266.RES:Rescheduling_interrupts 114.50 ± 13% -79.0% 24.00 interrupts.CPU268.RES:Rescheduling_interrupts 4610 ± 35% +64.0% 7559 interrupts.CPU269.NMI:Non-maskable_interrupts 4610 ± 35% +64.0% 7559 interrupts.CPU269.PMI:Performance_monitoring_interrupts 441.50 ± 27% -36.6% 280.00 interrupts.CPU27.RES:Rescheduling_interrupts 129.00 ± 62% +20168.2% 26146 interrupts.CPU272.RES:Rescheduling_interrupts 109.25 ± 65% -93.6% 7.00 interrupts.CPU279.RES:Rescheduling_interrupts 3728 +104.8% 7636 interrupts.CPU28.NMI:Non-maskable_interrupts 3728 +104.8% 7636 interrupts.CPU28.PMI:Performance_monitoring_interrupts 556.75 ± 23% -47.9% 290.00 interrupts.CPU28.RES:Rescheduling_interrupts 358.50 ± 79% -86.1% 50.00 interrupts.CPU282.RES:Rescheduling_interrupts 202.75 ± 54% -83.2% 34.00 interrupts.CPU284.RES:Rescheduling_interrupts 242.25 ± 69% -88.0% 29.00 interrupts.CPU287.RES:Rescheduling_interrupts 3718 +105.0% 7623 interrupts.CPU29.NMI:Non-maskable_interrupts 3718 +105.0% 7623 interrupts.CPU29.PMI:Performance_monitoring_interrupts 389.25 ± 14% -32.2% 264.00 interrupts.CPU29.RES:Rescheduling_interrupts 3757 +102.6% 7612 interrupts.CPU30.NMI:Non-maskable_interrupts 3757 +102.6% 7612 interrupts.CPU30.PMI:Performance_monitoring_interrupts 941.25 ± 19% -63.3% 345.00 interrupts.CPU30.RES:Rescheduling_interrupts 3730 +104.4% 7625 interrupts.CPU31.NMI:Non-maskable_interrupts 3730 +104.4% 7625 interrupts.CPU31.PMI:Performance_monitoring_interrupts 3744 +103.8% 7632 interrupts.CPU32.NMI:Non-maskable_interrupts 3744 +103.8% 7632 interrupts.CPU32.PMI:Performance_monitoring_interrupts 637.75 ± 41% -56.6% 277.00 interrupts.CPU32.RES:Rescheduling_interrupts 4649 ± 34% +64.3% 7637 interrupts.CPU34.NMI:Non-maskable_interrupts 4649 ± 34% +64.3% 7637 interrupts.CPU34.PMI:Performance_monitoring_interrupts 416.25 ± 16% -30.3% 290.00 interrupts.CPU34.RES:Rescheduling_interrupts 3725 +104.1% 7602 interrupts.CPU35.NMI:Non-maskable_interrupts 3725 +104.1% 7602 interrupts.CPU35.PMI:Performance_monitoring_interrupts 3695 +104.7% 7562 interrupts.CPU36.NMI:Non-maskable_interrupts 3695 +104.7% 7562 interrupts.CPU36.PMI:Performance_monitoring_interrupts 3748 +104.3% 7657 interrupts.CPU37.NMI:Non-maskable_interrupts 3748 +104.3% 7657 interrupts.CPU37.PMI:Performance_monitoring_interrupts 324.00 ± 9% -18.8% 263.00 interrupts.CPU37.RES:Rescheduling_interrupts 436.00 ± 5% -27.8% 315.00 interrupts.CPU38.RES:Rescheduling_interrupts 4671 ± 34% +61.0% 7521 interrupts.CPU39.NMI:Non-maskable_interrupts 4671 ± 34% +61.0% 7521 interrupts.CPU39.PMI:Performance_monitoring_interrupts 349.25 ± 14% -30.7% 242.00 interrupts.CPU39.RES:Rescheduling_interrupts 710.50 ± 32% -52.9% 335.00 interrupts.CPU4.RES:Rescheduling_interrupts 3707 +103.9% 7560 interrupts.CPU42.NMI:Non-maskable_interrupts 3707 +103.9% 7560 interrupts.CPU42.PMI:Performance_monitoring_interrupts 4597 ± 35% +63.4% 7513 interrupts.CPU43.NMI:Non-maskable_interrupts 4597 ± 35% +63.4% 7513 interrupts.CPU43.PMI:Performance_monitoring_interrupts 4610 ± 35% +64.1% 7568 interrupts.CPU44.NMI:Non-maskable_interrupts 4610 ± 35% +64.1% 7568 interrupts.CPU44.PMI:Performance_monitoring_interrupts 887.25 ± 95% -67.7% 287.00 interrupts.CPU44.RES:Rescheduling_interrupts 3669 +106.6% 7582 interrupts.CPU45.NMI:Non-maskable_interrupts 3669 +106.6% 7582 interrupts.CPU45.PMI:Performance_monitoring_interrupts 604.50 ± 66% -55.0% 272.00 interrupts.CPU47.RES:Rescheduling_interrupts 3717 +103.6% 7570 interrupts.CPU49.NMI:Non-maskable_interrupts 3717 +103.6% 7570 interrupts.CPU49.PMI:Performance_monitoring_interrupts 160.50 ± 20% +63.9% 263.00 interrupts.CPU49.RES:Rescheduling_interrupts 4601 ± 34% +64.8% 7585 interrupts.CPU50.NMI:Non-maskable_interrupts 4601 ± 34% +64.8% 7585 interrupts.CPU50.PMI:Performance_monitoring_interrupts 170.50 ± 36% +67.2% 285.00 interrupts.CPU50.RES:Rescheduling_interrupts 4590 ± 34% +65.5% 7595 interrupts.CPU51.NMI:Non-maskable_interrupts 4590 ± 34% +65.5% 7595 interrupts.CPU51.PMI:Performance_monitoring_interrupts 146.75 ± 43% -78.9% 31.00 interrupts.CPU51.RES:Rescheduling_interrupts 4627 ± 35% +62.4% 7514 interrupts.CPU52.NMI:Non-maskable_interrupts 4627 ± 35% +62.4% 7514 interrupts.CPU52.PMI:Performance_monitoring_interrupts 266.25 ± 57% -58.3% 111.00 interrupts.CPU54.RES:Rescheduling_interrupts 204.00 ± 71% -87.7% 25.00 interrupts.CPU55.RES:Rescheduling_interrupts 3703 +105.1% 7596 interrupts.CPU58.NMI:Non-maskable_interrupts 3703 +105.1% 7596 interrupts.CPU58.PMI:Performance_monitoring_interrupts 589.00 ± 41% -45.8% 319.00 interrupts.CPU6.RES:Rescheduling_interrupts 255.25 ± 57% -63.6% 93.00 interrupts.CPU60.RES:Rescheduling_interrupts 174.25 ± 19% -62.1% 66.00 interrupts.CPU62.RES:Rescheduling_interrupts 207.25 ± 50% -65.3% 72.00 interrupts.CPU66.RES:Rescheduling_interrupts 169.00 ± 54% -65.1% 59.00 interrupts.CPU68.RES:Rescheduling_interrupts 103.25 ± 66% -84.5% 16.00 interrupts.CPU69.RES:Rescheduling_interrupts 501.75 ± 24% -43.4% 284.00 interrupts.CPU7.RES:Rescheduling_interrupts 286.50 ± 53% +388.3% 1399 interrupts.CPU72.RES:Rescheduling_interrupts 634.25 ± 71% +406.0% 3209 interrupts.CPU74.RES:Rescheduling_interrupts 484.00 ± 55% +147.3% 1197 interrupts.CPU76.RES:Rescheduling_interrupts 85.75 ± 53% -63.8% 31.00 interrupts.CPU77.RES:Rescheduling_interrupts 539.25 ± 18% +73.9% 938.00 interrupts.CPU8.RES:Rescheduling_interrupts 755.25 ± 91% -77.0% 174.00 interrupts.CPU80.RES:Rescheduling_interrupts 63.25 ± 53% +119.8% 139.00 interrupts.CPU81.RES:Rescheduling_interrupts 519.25 ± 33% -65.1% 181.00 interrupts.CPU82.RES:Rescheduling_interrupts 517.25 ± 28% -60.8% 203.00 interrupts.CPU84.RES:Rescheduling_interrupts 132.25 ± 50% -61.4% 51.00 interrupts.CPU85.RES:Rescheduling_interrupts 382.00 ± 35% +77.2% 677.00 interrupts.CPU86.RES:Rescheduling_interrupts 62.50 ± 62% -74.4% 16.00 interrupts.CPU87.RES:Rescheduling_interrupts 92.00 ± 32% -66.3% 31.00 interrupts.CPU89.RES:Rescheduling_interrupts 436.75 ± 19% -39.1% 266.00 interrupts.CPU9.RES:Rescheduling_interrupts 1072 ± 76% -73.7% 282.00 interrupts.CPU92.RES:Rescheduling_interrupts 167.00 ± 75% -82.6% 29.00 interrupts.CPU93.RES:Rescheduling_interrupts 952.75 ± 66% -68.1% 304.00 interrupts.CPU96.RES:Rescheduling_interrupts 96.50 ± 25% +217.1% 306.00 interrupts.CPU97.RES:Rescheduling_interrupts 54.25 ± 22% +351.6% 245.00 interrupts.CPU99.RES:Rescheduling_interrupts 39164 ± 11% +56.5% 61282 softirqs.CPU0.RCU 119211 ± 14% +35.1% 161073 softirqs.CPU0.TIMER 39132 ± 15% +54.9% 60633 softirqs.CPU1.RCU 106485 ± 10% +39.0% 148019 softirqs.CPU1.TIMER 40469 ± 13% +46.2% 59148 softirqs.CPU10.RCU 33205 ± 16% +58.8% 52724 softirqs.CPU100.RCU 33017 ± 16% +58.1% 52202 softirqs.CPU101.RCU 32946 ± 16% +59.9% 52688 softirqs.CPU102.RCU 32954 ± 16% +58.7% 52303 softirqs.CPU103.RCU 32712 ± 16% +59.2% 52093 softirqs.CPU104.RCU 32374 ± 16% +62.4% 52576 softirqs.CPU105.RCU 32420 ± 16% +63.4% 52971 softirqs.CPU106.RCU 32316 ± 15% +62.9% 52648 softirqs.CPU107.RCU 32685 ± 15% +62.1% 52968 softirqs.CPU108.RCU 132926 ± 9% +18.9% 157999 softirqs.CPU108.TIMER 32436 ± 16% +61.6% 52414 softirqs.CPU109.RCU 40413 ± 14% +60.2% 64751 softirqs.CPU11.RCU 115232 ± 12% +21.9% 140473 softirqs.CPU11.TIMER 32751 ± 16% +60.0% 52403 softirqs.CPU110.RCU 32850 ± 16% +60.6% 52742 softirqs.CPU111.RCU 32532 ± 17% +60.8% 52319 softirqs.CPU112.RCU 32501 ± 16% +60.8% 52267 softirqs.CPU113.RCU 31957 ± 16% +64.6% 52601 softirqs.CPU114.RCU 32115 ± 15% +62.2% 52104 softirqs.CPU115.RCU 32767 ± 16% +59.3% 52200 softirqs.CPU116.RCU 32625 ± 15% +61.7% 52741 softirqs.CPU117.RCU 32319 ± 16% +61.3% 52142 softirqs.CPU118.RCU 32227 ± 15% +61.7% 52110 softirqs.CPU119.RCU 43084 ± 21% +41.9% 61121 softirqs.CPU12.RCU 31813 ± 17% +64.6% 52360 softirqs.CPU120.RCU 31512 ± 15% +66.8% 52559 softirqs.CPU121.RCU 32569 ± 15% +61.6% 52619 softirqs.CPU122.RCU 32337 ± 16% +61.7% 52276 softirqs.CPU123.RCU 32442 ± 15% +61.9% 52524 softirqs.CPU124.RCU 32258 ± 15% +63.3% 52680 softirqs.CPU125.RCU 32480 ± 16% +61.6% 52500 softirqs.CPU126.RCU 32535 ± 17% +60.8% 52330 softirqs.CPU127.RCU 32059 ± 16% +60.4% 51435 softirqs.CPU128.RCU 32312 ± 16% +61.8% 52278 softirqs.CPU129.RCU 1056 ±137% +3671.0% 39822 softirqs.CPU13.NET_RX 40034 ± 11% +51.2% 60514 softirqs.CPU13.RCU 108802 ± 10% +29.6% 141012 softirqs.CPU13.TIMER 32215 ± 16% +62.1% 52205 softirqs.CPU130.RCU 32315 ± 16% +61.1% 52075 softirqs.CPU131.RCU 33006 ± 15% +59.0% 52476 softirqs.CPU132.RCU 32266 ± 17% +61.9% 52250 softirqs.CPU133.RCU 32626 ± 16% +62.9% 53136 softirqs.CPU134.RCU 32433 ± 18% +58.8% 51494 softirqs.CPU135.RCU 32294 ± 14% +58.7% 51235 softirqs.CPU136.RCU 31820 ± 16% +63.5% 52021 softirqs.CPU137.RCU 32892 ± 16% +58.8% 52227 softirqs.CPU138.RCU 32976 ± 15% +57.9% 52060 softirqs.CPU139.RCU 40138 ± 12% +60.0% 64221 softirqs.CPU14.RCU 117850 ± 9% +15.9% 136643 softirqs.CPU14.TIMER 32440 ± 16% +59.4% 51714 softirqs.CPU140.RCU 32239 ± 16% +65.8% 53461 softirqs.CPU141.RCU 32639 ± 16% +59.7% 52137 softirqs.CPU142.RCU 32350 ± 16% +62.3% 52492 softirqs.CPU143.RCU 32203 ± 16% +61.8% 52101 softirqs.CPU144.RCU 32170 ± 17% +60.4% 51609 softirqs.CPU145.RCU 32892 ± 16% +65.1% 54293 softirqs.CPU146.RCU 32355 ± 16% +61.3% 52182 softirqs.CPU147.RCU 32385 ± 16% +60.7% 52047 softirqs.CPU148.RCU 32356 ± 17% +62.5% 52581 softirqs.CPU149.RCU 38976 ± 16% +61.0% 62748 softirqs.CPU15.RCU 110967 ± 12% +24.7% 138390 softirqs.CPU15.TIMER 32399 ± 16% +63.3% 52903 softirqs.CPU150.RCU 32238 ± 15% +57.9% 50889 softirqs.CPU151.RCU 32360 ± 16% +62.0% 52413 softirqs.CPU152.RCU 32475 ± 16% +60.2% 52015 softirqs.CPU153.RCU 32209 ± 16% +61.4% 51998 softirqs.CPU154.RCU 32373 ± 16% +61.2% 52181 softirqs.CPU155.RCU 32295 ± 15% +65.9% 53590 softirqs.CPU156.RCU 32127 ± 16% +57.1% 50483 softirqs.CPU157.RCU 32514 ± 16% +60.6% 52204 softirqs.CPU158.RCU 32857 ± 16% +58.2% 51977 softirqs.CPU159.RCU 39215 ± 11% +54.0% 60374 softirqs.CPU16.RCU 116903 ± 13% +28.9% 150659 softirqs.CPU16.TIMER 32327 ± 16% +61.5% 52199 softirqs.CPU160.RCU 32410 ± 16% +60.4% 51970 softirqs.CPU161.RCU 32024 ± 16% +60.8% 51503 softirqs.CPU162.RCU 31125 ± 14% +67.1% 52016 softirqs.CPU163.RCU 32266 ± 16% +60.0% 51633 softirqs.CPU164.RCU 32209 ± 17% +61.9% 52135 softirqs.CPU165.RCU 32200 ± 16% +61.4% 51956 softirqs.CPU166.RCU 31951 ± 16% +62.5% 51932 softirqs.CPU167.RCU 31914 ± 16% +62.4% 51836 softirqs.CPU168.RCU 32081 ± 16% +62.3% 52066 softirqs.CPU169.RCU 39400 ± 11% +52.2% 59956 softirqs.CPU17.RCU 107863 ± 7% +37.2% 147984 softirqs.CPU17.TIMER 32016 ± 17% +63.9% 52460 softirqs.CPU170.RCU 31961 ± 16% +62.7% 51996 softirqs.CPU171.RCU 32978 ± 15% +56.1% 51468 softirqs.CPU172.RCU 32233 ± 16% +62.1% 52246 softirqs.CPU173.RCU 32152 ± 16% +63.4% 52543 softirqs.CPU174.RCU 32075 ± 16% +62.0% 51952 softirqs.CPU175.RCU 32196 ± 16% +60.3% 51620 softirqs.CPU176.RCU 32269 ± 17% +59.5% 51462 softirqs.CPU177.RCU 32096 ± 15% +61.7% 51886 softirqs.CPU178.RCU 32182 ± 16% +60.2% 51570 softirqs.CPU179.RCU 38279 ± 15% +55.2% 59426 softirqs.CPU18.RCU 117158 ± 13% +31.0% 153465 softirqs.CPU18.TIMER 31735 ± 16% +62.2% 51480 softirqs.CPU180.RCU 29335 ± 5% +75.2% 51409 softirqs.CPU181.RCU 31981 ± 16% +60.1% 51205 softirqs.CPU182.RCU 31948 ± 15% +57.6% 50365 softirqs.CPU183.RCU 32189 ± 17% +59.1% 51205 softirqs.CPU184.RCU 31957 ± 16% +60.7% 51344 softirqs.CPU185.RCU 32298 ± 17% +58.2% 51080 softirqs.CPU186.RCU 32335 ± 16% +58.8% 51344 softirqs.CPU187.RCU 32287 ± 16% +59.6% 51540 softirqs.CPU188.RCU 31965 ± 16% +59.1% 50857 softirqs.CPU189.RCU 39495 ± 14% +53.9% 60800 softirqs.CPU19.RCU 114131 ± 9% +20.9% 138034 softirqs.CPU19.TIMER 31213 ± 17% +65.4% 51631 softirqs.CPU190.RCU 31471 ± 17% +64.1% 51645 softirqs.CPU191.RCU 31760 ± 17% +62.9% 51728 softirqs.CPU192.RCU 32147 ± 16% +61.8% 52016 softirqs.CPU193.RCU 32079 ± 17% +62.9% 52256 softirqs.CPU194.RCU 31755 ± 16% +63.2% 51815 softirqs.CPU195.RCU 32240 ± 16% +62.6% 52417 softirqs.CPU196.RCU 32033 ± 16% +60.6% 51435 softirqs.CPU197.RCU 32041 ± 17% +62.4% 52022 softirqs.CPU198.RCU 31849 ± 17% +64.2% 52294 softirqs.CPU199.RCU 39674 ± 10% +54.8% 61436 softirqs.CPU2.RCU 40657 ± 17% +48.6% 60422 softirqs.CPU20.RCU 113465 ± 13% +32.5% 150390 softirqs.CPU20.TIMER 31959 ± 16% +62.8% 52023 softirqs.CPU200.RCU 31710 ± 16% +62.6% 51548 softirqs.CPU201.RCU 32021 ± 17% +61.8% 51801 softirqs.CPU202.RCU 32269 ± 16% +59.5% 51463 softirqs.CPU203.RCU 31922 ± 15% +60.0% 51087 softirqs.CPU204.RCU 31672 ± 17% +63.4% 51740 softirqs.CPU205.RCU 31875 ± 16% +63.2% 52006 softirqs.CPU206.RCU 31746 ± 16% +63.5% 51898 softirqs.CPU207.RCU 32414 ± 16% +61.0% 52186 softirqs.CPU208.RCU 32251 ± 16% +60.8% 51875 softirqs.CPU209.RCU 40977 ± 13% +46.2% 59928 softirqs.CPU21.RCU 109618 ± 13% +23.9% 135789 softirqs.CPU21.TIMER 32763 ± 15% +58.2% 51825 softirqs.CPU210.RCU 32054 ± 15% +62.8% 52195 softirqs.CPU211.RCU 32462 ± 16% +61.6% 52468 softirqs.CPU212.RCU 32438 ± 16% +61.8% 52499 softirqs.CPU213.RCU 31689 ± 17% +62.5% 51498 softirqs.CPU214.RCU 32181 ± 16% +63.6% 52657 softirqs.CPU215.RCU 32243 ± 16% +60.5% 51745 softirqs.CPU216.RCU 32122 ± 15% +59.8% 51345 softirqs.CPU217.RCU 32194 ± 16% +62.6% 52340 softirqs.CPU218.RCU 33401 ± 15% +58.2% 52834 softirqs.CPU219.RCU 39309 ± 19% +53.5% 60334 softirqs.CPU22.RCU 119248 ± 16% +23.3% 147000 softirqs.CPU22.TIMER 31767 ± 16% +63.9% 52051 softirqs.CPU220.RCU 32483 ± 15% +60.4% 52113 softirqs.CPU221.RCU 32330 ± 16% +61.1% 52088 softirqs.CPU222.RCU 32645 ± 15% +59.9% 52206 softirqs.CPU223.RCU 32122 ± 17% +59.0% 51071 softirqs.CPU224.RCU 32307 ± 16% +60.0% 51695 softirqs.CPU225.RCU 32176 ± 16% +61.2% 51868 softirqs.CPU226.RCU 32405 ± 16% +60.4% 51994 softirqs.CPU227.RCU 32186 ± 20% +61.2% 51874 softirqs.CPU228.RCU 31808 ± 16% +63.5% 52008 softirqs.CPU229.RCU 40782 ± 17% +48.6% 60604 softirqs.CPU23.RCU 32003 ± 16% +58.8% 50807 softirqs.CPU230.RCU 32424 ± 16% +58.7% 51456 softirqs.CPU231.RCU 32316 ± 16% +60.7% 51924 softirqs.CPU232.RCU 32725 ± 15% +59.2% 52098 softirqs.CPU233.RCU 31237 ± 15% +65.4% 51666 softirqs.CPU234.RCU 32479 ± 15% +60.3% 52063 softirqs.CPU235.RCU 133723 ± 7% +15.4% 154367 softirqs.CPU235.TIMER 32028 ± 16% +62.2% 51963 softirqs.CPU236.RCU 32136 ± 16% +62.4% 52192 softirqs.CPU237.RCU 32023 ± 16% +60.5% 51408 softirqs.CPU238.RCU 32055 ± 16% +60.7% 51524 softirqs.CPU239.RCU 39888 ± 17% +53.7% 61300 softirqs.CPU24.RCU 112430 ± 15% +27.1% 142949 softirqs.CPU24.TIMER 31745 ± 17% +63.8% 52007 softirqs.CPU240.RCU 32474 ± 16% +60.9% 52242 softirqs.CPU241.RCU 31904 ± 16% +62.7% 51899 softirqs.CPU242.RCU 32032 ± 16% +61.9% 51868 softirqs.CPU243.RCU 31357 ± 18% +64.8% 51685 softirqs.CPU244.RCU 132812 ± 8% +15.3% 153157 softirqs.CPU244.TIMER 31549 ± 16% +64.3% 51841 softirqs.CPU245.RCU 32027 ± 16% +60.8% 51504 softirqs.CPU246.RCU 32033 ± 16% +77.1% 56745 softirqs.CPU247.RCU 141833 ± 5% +19.5% 169427 softirqs.CPU247.TIMER 31859 ± 17% +65.0% 52580 softirqs.CPU248.RCU 32090 ± 16% +62.2% 52055 softirqs.CPU249.RCU 39971 ± 13% +52.6% 61007 softirqs.CPU25.RCU 114330 ± 12% +28.9% 147423 softirqs.CPU25.TIMER 31491 ± 16% +64.9% 51918 softirqs.CPU250.RCU 31766 ± 15% +64.4% 52234 softirqs.CPU251.RCU 32019 ± 16% +61.0% 51544 softirqs.CPU252.RCU 31922 ± 17% +59.7% 50971 softirqs.CPU253.RCU 31514 ± 16% +64.1% 51705 softirqs.CPU254.RCU 31765 ± 17% +64.6% 52276 softirqs.CPU255.RCU 31738 ± 16% +60.0% 50780 softirqs.CPU256.RCU 33985 ± 11% +48.7% 50523 softirqs.CPU257.RCU 31187 ± 17% +57.1% 49000 softirqs.CPU258.RCU 32031 ± 14% +50.2% 48097 softirqs.CPU259.RCU 39939 ± 11% +53.3% 61220 softirqs.CPU26.RCU 115081 ± 12% +24.6% 143406 softirqs.CPU26.TIMER 31401 ± 16% +60.2% 50314 softirqs.CPU260.RCU 30452 ± 16% +62.8% 49574 softirqs.CPU261.RCU 30819 ± 16% +62.0% 49913 softirqs.CPU262.RCU 31668 ± 22% +57.4% 49847 softirqs.CPU263.RCU 32298 ± 16% +58.0% 51038 softirqs.CPU264.RCU 31601 ± 13% +57.3% 49722 softirqs.CPU265.RCU 123387 ± 6% +17.4% 144849 softirqs.CPU265.TIMER 31810 ± 16% +60.5% 51049 softirqs.CPU266.RCU 32986 ± 14% +51.1% 49844 softirqs.CPU267.RCU 32309 ± 16% +58.0% 51060 softirqs.CPU268.RCU 32278 ± 16% +59.2% 51371 softirqs.CPU269.RCU 40744 ± 17% +58.9% 64726 softirqs.CPU27.RCU 108716 ± 13% +41.4% 153739 softirqs.CPU27.TIMER 32431 ± 16% +56.5% 50753 softirqs.CPU270.RCU 32242 ± 16% +55.4% 50109 softirqs.CPU271.RCU 33991 ± 15% -25.3% 25388 softirqs.CPU272.RCU 122760 ± 13% +26.1% 154830 softirqs.CPU272.TIMER 33868 ± 16% +45.9% 49423 softirqs.CPU273.RCU 125322 ± 12% +22.9% 154051 softirqs.CPU273.TIMER 33902 ± 15% +57.9% 53546 softirqs.CPU274.RCU 33622 ± 16% +53.0% 51450 softirqs.CPU275.RCU 133137 ± 8% +16.5% 155075 softirqs.CPU275.TIMER 33881 ± 16% +59.1% 53894 softirqs.CPU276.RCU 33983 ± 16% +60.2% 54449 softirqs.CPU277.RCU 33462 ± 15% +57.7% 52778 softirqs.CPU278.RCU 126550 ± 9% +18.6% 150046 softirqs.CPU278.TIMER 34169 ± 16% +58.3% 54100 softirqs.CPU279.RCU 40134 ± 12% +54.1% 61856 softirqs.CPU28.RCU 106399 ± 7% +36.8% 145511 softirqs.CPU28.TIMER 34216 ± 15% +54.1% 52717 softirqs.CPU280.RCU 34772 ± 14% +49.9% 52127 softirqs.CPU281.RCU 34184 ± 16% +55.3% 53074 softirqs.CPU282.RCU 33411 ± 17% +57.4% 52591 softirqs.CPU283.RCU 33482 ± 17% +59.2% 53316 softirqs.CPU284.RCU 124921 ± 7% +18.4% 147846 softirqs.CPU284.TIMER 34232 ± 16% +59.9% 54735 softirqs.CPU285.RCU 33099 ± 16% +61.5% 53454 softirqs.CPU286.RCU 128032 ± 6% +15.8% 148252 softirqs.CPU286.TIMER 33660 ± 14% +60.2% 53931 softirqs.CPU287.RCU 39735 ± 14% +52.4% 60570 softirqs.CPU29.RCU 109107 ± 10% +31.4% 143385 softirqs.CPU29.TIMER 38978 ± 17% +58.8% 61916 softirqs.CPU3.RCU 114894 ± 13% +24.5% 143095 softirqs.CPU3.TIMER 39122 ± 13% +57.3% 61555 softirqs.CPU30.RCU 112105 ± 11% +29.4% 145113 softirqs.CPU30.TIMER 40231 ± 15% +58.2% 63642 softirqs.CPU31.RCU 106758 ± 9% +28.5% 137237 softirqs.CPU31.TIMER 36607 ± 12% +49.8% 54850 softirqs.CPU32.RCU 107614 ± 10% +30.6% 140576 softirqs.CPU32.TIMER 38637 ± 11% +41.5% 54675 softirqs.CPU33.RCU 111703 ± 10% +25.3% 139965 softirqs.CPU33.TIMER 37654 ± 14% +48.0% 55715 softirqs.CPU34.RCU 108130 ± 10% +29.2% 139690 softirqs.CPU34.TIMER 37742 ± 14% +48.5% 56044 softirqs.CPU35.RCU 111809 ± 13% +35.7% 151748 softirqs.CPU35.TIMER 37760 ± 13% +50.7% 56894 softirqs.CPU36.RCU 104801 ± 6% +41.3% 148074 softirqs.CPU36.TIMER 38759 ± 10% +44.3% 55925 softirqs.CPU37.RCU 110484 ± 10% +31.8% 145580 softirqs.CPU37.TIMER 37881 ± 13% +52.6% 57806 softirqs.CPU38.RCU 111036 ± 14% +28.4% 142517 softirqs.CPU38.TIMER 37751 ± 13% +53.5% 57963 softirqs.CPU39.RCU 109975 ± 12% +29.8% 142703 softirqs.CPU39.TIMER 40700 ± 15% +55.2% 63155 softirqs.CPU4.RCU 33795 ± 15% +53.5% 51887 softirqs.CPU40.RCU 32637 ± 15% +67.7% 54725 softirqs.CPU41.RCU 33955 ± 15% +66.4% 56488 softirqs.CPU42.RCU 128703 ± 11% +31.3% 168987 softirqs.CPU42.TIMER 34606 ± 14% +54.9% 53594 softirqs.CPU43.RCU 124225 ± 11% +22.8% 152583 softirqs.CPU43.TIMER 34280 ± 14% +61.1% 55211 softirqs.CPU44.RCU 33700 ± 16% +64.4% 55416 softirqs.CPU45.RCU 122489 ± 7% +23.4% 151156 softirqs.CPU45.TIMER 34421 ± 14% +60.7% 55327 softirqs.CPU46.RCU 124335 ± 12% +22.7% 152608 softirqs.CPU46.TIMER 32622 ± 17% +63.8% 53430 softirqs.CPU47.RCU 34127 ± 16% +59.8% 54523 softirqs.CPU48.RCU 35165 ± 16% +58.9% 55872 softirqs.CPU49.RCU 40442 ± 11% +56.7% 63356 softirqs.CPU5.RCU 109994 ± 13% +22.7% 135002 softirqs.CPU5.TIMER 34820 ± 16% +61.1% 56093 softirqs.CPU50.RCU 35464 ± 15% +59.4% 56524 softirqs.CPU51.RCU 125051 ± 11% +24.2% 155354 softirqs.CPU51.TIMER 36036 ± 18% +53.7% 55396 softirqs.CPU52.RCU 35654 ± 16% +52.6% 54416 softirqs.CPU53.RCU 35691 ± 11% +52.4% 54411 softirqs.CPU54.RCU 34660 ± 13% +53.3% 53123 softirqs.CPU55.RCU 126943 ± 9% +21.1% 153725 softirqs.CPU55.TIMER 34264 ± 15% +62.4% 55653 softirqs.CPU56.RCU 125229 ± 7% +20.9% 151461 softirqs.CPU56.TIMER 34576 ± 16% +62.1% 56057 softirqs.CPU57.RCU 34536 ± 16% +61.4% 55726 softirqs.CPU58.RCU 129172 ± 5% +16.5% 150459 softirqs.CPU58.TIMER 35042 ± 16% +57.1% 55050 softirqs.CPU59.RCU 40606 ± 12% +60.2% 65069 softirqs.CPU6.RCU 109977 ± 9% +23.9% 136257 softirqs.CPU6.TIMER 35058 ± 16% +57.7% 55277 softirqs.CPU60.RCU 34951 ± 15% +52.2% 53188 softirqs.CPU61.RCU 35594 ± 14% +54.4% 54950 softirqs.CPU62.RCU 35217 ± 15% +52.7% 53790 softirqs.CPU63.RCU 33270 ± 15% +59.3% 52990 softirqs.CPU64.RCU 33326 ± 16% +55.1% 51696 softirqs.CPU65.RCU 33573 ± 15% +56.8% 52631 softirqs.CPU66.RCU 33582 ± 15% +53.8% 51642 softirqs.CPU67.RCU 32917 ± 15% +59.9% 52644 softirqs.CPU68.RCU 33037 ± 15% +63.0% 53849 softirqs.CPU69.RCU 39522 ± 10% +60.3% 63357 softirqs.CPU7.RCU 33226 ± 16% +59.2% 52908 softirqs.CPU70.RCU 32781 ± 15% +58.6% 51989 softirqs.CPU71.RCU 33064 ± 13% +80.6% 59702 softirqs.CPU72.RCU 32784 ± 14% +56.8% 51422 softirqs.CPU73.RCU 32988 ± 17% +60.6% 52968 softirqs.CPU74.RCU 32197 ± 15% +61.3% 51922 softirqs.CPU75.RCU 32210 ± 15% +55.6% 50127 softirqs.CPU76.RCU 31828 ± 15% +57.7% 50193 softirqs.CPU77.RCU 33704 ± 14% +52.1% 51273 softirqs.CPU78.RCU 33146 ± 17% +53.4% 50850 softirqs.CPU79.RCU 40019 ± 14% +53.5% 61416 softirqs.CPU8.RCU 122353 ± 9% +21.5% 148650 softirqs.CPU8.TIMER 32597 ± 15% +64.3% 53549 softirqs.CPU80.RCU 134840 ± 5% +15.2% 155292 softirqs.CPU80.TIMER 32221 ± 15% +64.6% 53052 softirqs.CPU81.RCU 33039 ± 15% +59.6% 52743 softirqs.CPU82.RCU 32787 ± 16% +59.1% 52164 softirqs.CPU83.RCU 32695 ± 15% +58.8% 51926 softirqs.CPU84.RCU 32467 ± 16% +64.0% 53258 softirqs.CPU85.RCU 136153 ± 7% +14.7% 156116 softirqs.CPU85.TIMER 32705 ± 16% +62.3% 53086 softirqs.CPU86.RCU 32601 ± 16% +62.1% 52854 softirqs.CPU87.RCU 32995 ± 15% +59.9% 52757 softirqs.CPU88.RCU 32822 ± 16% +59.3% 52298 softirqs.CPU89.RCU 40364 ± 14% +52.0% 61352 softirqs.CPU9.RCU 112908 ± 14% +33.1% 150306 softirqs.CPU9.TIMER 32824 ± 16% +62.8% 53450 softirqs.CPU90.RCU 32906 ± 16% +64.4% 54093 softirqs.CPU91.RCU 32596 ± 15% +61.9% 52772 softirqs.CPU92.RCU 32472 ± 16% +66.6% 54103 softirqs.CPU93.RCU 32897 ± 16% +60.2% 52703 softirqs.CPU94.RCU 32672 ± 16% +62.5% 53086 softirqs.CPU95.RCU 32311 ± 15% +57.7% 50960 softirqs.CPU96.RCU 32505 ± 15% +58.6% 51551 softirqs.CPU97.RCU 32357 ± 13% +64.3% 53148 softirqs.CPU98.RCU 32434 ± 16% +63.6% 53073 softirqs.CPU99.RCU 9259 ±127% +353.8% 42022 softirqs.NET_RX 9669747 ± 15% +59.0% 15377610 softirqs.RCU *************************************************************************************************** lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/context_switch1/will-it-scale/0x11 commit: b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support") 492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers") b121b341e5983bdc 492e639f0c222784e2e0f121966 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 2:2 -100% :2 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 2:2 -100% :2 dmesg.WARNING:stack_recursion Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen