Greeting, FYI, we noticed a -27.3% regression of netperf.Throughput_Mbps due to commit: commit: 59901cb4520c44bfce81f523bc61e7284a931ad1 ("[PATCH] sched/fair: Don't pull task if local group is more loaded than busiest group") url: https://github.com/0day-ci/linux/commits/Aubrey-Li/sched-fair-Don-t-pull-task-if-local-group-is-more-loaded-than-busiest-group/20200326-080334 in testcase: netperf on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory with following parameters: ip: ipv4 runtime: 300s nr_threads: 25% cluster: cs-localhost send_size: 5K test: TCP_SENDFILE cpufreq_governor: performance ucode: 0x2000065 test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance. test-url: http://www.netperf.org/netperf/ In addition to that, the commit also has significant impact on the following tests: +------------------+------------------------------------------------------------------------+ | testcase: change | netperf: netperf.Throughput_Mbps -31.1% regression | | test machine | 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | ip=ipv4 | | | nr_threads=25% | | | runtime=300s | | | send_size=10K | | | test=SCTP_STREAM_MANY | | | ucode=0xb000038 | +------------------+------------------------------------------------------------------------+ | testcase: change | unixbench: unixbench.score -4.0% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | nr_task=30% | | | runtime=300s | | | test=whetstone-double | | | ucode=0x2000065 | +------------------+------------------------------------------------------------------------+ | testcase: change | unixbench: unixbench.score 29.1% improvement | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | nr_task=30% | | | runtime=300s | | | test=context1 | | | ucode=0x2000065 | +------------------+------------------------------------------------------------------------+ | testcase: change | kbuild: kbuild.buildtime_per_iteration 7.5% regression | | test machine | 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | nr_task=50% | | | runtime=300s | | | target=vmlinux_prereq | | | ucode=0x27 | +------------------+------------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/300s/5K/lkp-skl-2sp7/TCP_SENDFILE/netperf/0x2000065 commit: 9c40365a65 ("threads: Update PID limit comment according to futex UAPI change") 59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group") 9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x :4 50% 2:4 dmesg.WARNING:at_ip__fsnotify_parent/0x %stddev %change %stddev \ | \ 18708 -27.3% 13593 netperf.Throughput_Mbps 336757 -27.3% 244677 netperf.Throughput_total_Mbps 8293 ± 2% +82.7% 15156 ± 57% netperf.time.involuntary_context_switches 2291 +2.6% 2350 netperf.time.maximum_resident_set_size 5103 -8.9% 4651 netperf.time.minor_page_faults 4704 +1.4% 4771 netperf.time.system_time 695.32 -9.7% 627.59 netperf.time.user_time 431.25 ± 51% +394.1% 2131 ± 59% netperf.time.voluntary_context_switches 2.466e+09 -27.3% 1.792e+09 netperf.workload 526188 ± 5% -21.3% 414060 ± 7% meminfo.DirectMap4k 53993 ± 3% -13.9% 46507 ± 5% meminfo.Shmem 1482325 ± 2% -24.4% 1120083 vmstat.system.cs 150508 -1.5% 148218 vmstat.system.in 3999 ± 8% -10.8% 3566 ± 5% slabinfo.kmalloc-rcl-64.active_objs 3999 ± 8% -10.8% 3566 ± 5% slabinfo.kmalloc-rcl-64.num_objs 10897 +12.6% 12274 ± 4% slabinfo.vmap_area.active_objs 10898 +12.7% 12277 ± 4% slabinfo.vmap_area.num_objs 2.208e+08 -20.9% 1.746e+08 cpuidle.C1.usage 5894730 ± 24% +7005.1% 4.188e+08 ± 25% cpuidle.C6.time 19279 ± 3% +2594.5% 519470 ± 17% cpuidle.C6.usage 35085456 ± 9% -39.3% 21307532 ± 7% cpuidle.POLL.time 4822644 ± 12% -49.7% 2423791 ± 9% cpuidle.POLL.usage 76807 -2.7% 74740 proc-vmstat.nr_active_anon 13500 ± 3% -13.8% 11639 ± 5% proc-vmstat.nr_shmem 76807 -2.7% 74740 proc-vmstat.nr_zone_active_anon 15013 ± 91% -86.2% 2071 ±151% proc-vmstat.numa_hint_faults 13589 ± 5% -15.5% 11476 ± 11% proc-vmstat.pgactivate 833769 -2.1% 815970 proc-vmstat.pgfault 754892 +1.1% 762828 proc-vmstat.pgfree 152278 ± 42% -96.3% 5569 ± 23% sched_debug.cfs_rq:/.min_vruntime.min 1169341 +30.4% 1524450 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev 1169341 +30.4% 1524462 ± 7% sched_debug.cfs_rq:/.spread0.stddev 1.58 ± 6% +22.0% 1.93 ± 7% sched_debug.cpu.clock.stddev 1.58 ± 6% +22.1% 1.93 ± 7% sched_debug.cpu.clock_task.stddev 3098435 -28.6% 2211323 ± 9% sched_debug.cpu.nr_switches.avg 1573 ± 16% -21.6% 1233 ± 9% sched_debug.cpu.nr_switches.min 56.75 ± 11% +18.5% 67.25 ± 4% sched_debug.cpu.nr_uninterruptible.max 6543 ± 8% +16.6% 7631 ± 5% numa-vmstat.node0.nr_kernel_stack 574.25 ± 30% +86.8% 1072 ± 22% numa-vmstat.node0.nr_page_table_pages 8042 ± 19% +23.3% 9919 ± 13% numa-vmstat.node0.nr_slab_reclaimable 34409 ± 22% -56.6% 14946 ± 54% numa-vmstat.node1.nr_active_anon 25847 ± 30% -67.3% 8458 ± 93% numa-vmstat.node1.nr_anon_pages 6123 ± 8% -18.0% 5022 ± 8% numa-vmstat.node1.nr_kernel_stack 854.50 ± 19% -56.5% 372.00 ± 65% numa-vmstat.node1.nr_page_table_pages 18421 ± 7% -17.9% 15125 ± 7% numa-vmstat.node1.nr_slab_unreclaimable 34409 ± 22% -56.6% 14946 ± 54% numa-vmstat.node1.nr_zone_active_anon 21046 ± 47% +55.4% 32703 ± 30% numa-vmstat.node1.numa_other 32169 ± 19% +23.3% 39680 ± 13% numa-meminfo.node0.KReclaimable 6543 ± 8% +16.6% 7630 ± 5% numa-meminfo.node0.KernelStack 1001821 ± 9% +14.4% 1145780 ± 4% numa-meminfo.node0.MemUsed 2299 ± 30% +86.6% 4290 ± 22% numa-meminfo.node0.PageTables 32169 ± 19% +23.3% 39680 ± 13% numa-meminfo.node0.SReclaimable 102778 ± 7% +19.9% 123222 ± 6% numa-meminfo.node0.Slab 137587 ± 22% -56.6% 59688 ± 54% numa-meminfo.node1.Active 137493 ± 22% -56.6% 59648 ± 54% numa-meminfo.node1.Active(anon) 52091 ± 42% -68.4% 16453 ±161% numa-meminfo.node1.AnonHugePages 103405 ± 30% -67.3% 33836 ± 93% numa-meminfo.node1.AnonPages 6116 ± 8% -18.0% 5017 ± 8% numa-meminfo.node1.KernelStack 1050635 ± 8% -13.4% 909860 ± 5% numa-meminfo.node1.MemUsed 3416 ± 19% -56.6% 1483 ± 65% numa-meminfo.node1.PageTables 73684 ± 7% -17.9% 60503 ± 7% numa-meminfo.node1.SUnreclaim 108980 ± 7% -19.1% 88152 ± 10% numa-meminfo.node1.Slab 81937 ± 6% +25.2% 102562 ± 9% softirqs.CPU16.RCU 21615 ± 70% +204.7% 65858 ± 22% softirqs.CPU16.SCHED 5010947 ± 28% -67.3% 1637461 ±148% softirqs.CPU25.NET_RX 45054 ± 5% -21.6% 35310 ± 26% softirqs.CPU25.SCHED 4691793 ± 34% -69.2% 1444457 ±170% softirqs.CPU26.NET_RX 48190 ± 9% -29.8% 33825 ± 34% softirqs.CPU26.SCHED 5045331 ± 36% -77.2% 1148265 ±171% softirqs.CPU29.NET_RX 51391 ± 23% -69.3% 15752 ±133% softirqs.CPU36.SCHED 52040 ± 11% -73.0% 14042 ±111% softirqs.CPU46.SCHED 37936 ± 50% -78.3% 8234 ±104% softirqs.CPU47.SCHED 46988 ± 16% -67.0% 15513 ± 99% softirqs.CPU52.SCHED 5967380 ± 30% -80.4% 1169737 ±173% softirqs.CPU54.NET_RX 6170696 ± 42% -76.1% 1472055 ±173% softirqs.CPU57.NET_RX 6718083 ± 40% -77.3% 1528221 ±173% softirqs.CPU58.NET_RX 6397847 ± 27% -74.5% 1629273 ±173% softirqs.CPU59.NET_RX 57352 ± 5% -35.6% 36922 ± 10% softirqs.CPU63.SCHED 4947229 ± 28% -76.6% 1159904 ±173% softirqs.CPU64.NET_RX 4902059 ± 41% -72.8% 1331256 ±169% softirqs.CPU68.NET_RX 3.636e+08 -23.0% 2.799e+08 softirqs.NET_RX 106.50 ± 96% -96.7% 3.50 ±125% interrupts.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19 24013 -1.8% 23590 interrupts.CAL:Function_call_interrupts 14701 ± 60% -99.2% 123.75 ± 89% interrupts.CPU11.RES:Rescheduling_interrupts 106.00 ± 97% -97.2% 3.00 ±135% interrupts.CPU19.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19 23189 ± 24% -94.6% 1242 ±139% interrupts.CPU25.RES:Rescheduling_interrupts 23527 ± 34% -95.2% 1129 ±152% interrupts.CPU26.RES:Rescheduling_interrupts 13122 ± 50% -98.4% 207.50 ±143% interrupts.CPU29.RES:Rescheduling_interrupts 21974 ± 29% -77.8% 4872 ±172% interrupts.CPU3.RES:Rescheduling_interrupts 28365 ± 52% -79.1% 5941 ±171% interrupts.CPU36.RES:Rescheduling_interrupts 27230 ± 58% -98.1% 518.00 ± 98% interrupts.CPU46.RES:Rescheduling_interrupts 21075 ± 63% -99.0% 205.25 ± 58% interrupts.CPU47.RES:Rescheduling_interrupts 6718 ± 19% -43.9% 3772 ± 63% interrupts.CPU48.NMI:Non-maskable_interrupts 6718 ± 19% -43.9% 3772 ± 63% interrupts.CPU48.PMI:Performance_monitoring_interrupts 7806 ± 72% -73.1% 2100 ±147% interrupts.CPU52.RES:Rescheduling_interrupts 7343 ± 23% -57.7% 3106 ± 96% interrupts.CPU54.NMI:Non-maskable_interrupts 7343 ± 23% -57.7% 3106 ± 96% interrupts.CPU54.PMI:Performance_monitoring_interrupts 7679 ± 84% -98.4% 123.75 ±104% interrupts.CPU54.RES:Rescheduling_interrupts 35789 ± 57% -75.5% 8758 ±173% interrupts.CPU56.RES:Rescheduling_interrupts 23882 ± 72% -94.9% 1212 ±172% interrupts.CPU57.RES:Rescheduling_interrupts 21256 ± 72% -94.7% 1134 ±170% interrupts.CPU58.RES:Rescheduling_interrupts 73.50 ±145% -90.8% 6.75 ± 72% interrupts.CPU6.TLB:TLB_shootdowns 32631 ± 47% -68.2% 10387 ± 72% interrupts.CPU61.RES:Rescheduling_interrupts 29653 ± 29% -81.4% 5518 ±171% interrupts.CPU62.RES:Rescheduling_interrupts 25421 ± 21% -91.3% 2218 ±165% interrupts.CPU63.RES:Rescheduling_interrupts 14735 ± 54% -98.6% 200.50 ±155% interrupts.CPU64.RES:Rescheduling_interrupts 11691 ± 85% -64.2% 4184 ±172% interrupts.CPU66.RES:Rescheduling_interrupts 6240 ± 32% -68.2% 1982 ±140% interrupts.CPU67.NMI:Non-maskable_interrupts 6240 ± 32% -68.2% 1982 ±140% interrupts.CPU67.PMI:Performance_monitoring_interrupts 6781 ±120% -93.8% 423.50 ±145% interrupts.CPU68.RES:Rescheduling_interrupts 12981 ± 64% -98.7% 174.50 ±105% interrupts.CPU70.RES:Rescheduling_interrupts 14238 ± 44% -65.1% 4975 ±168% interrupts.CPU71.RES:Rescheduling_interrupts 107.00 ± 5% -17.1% 88.75 ± 11% interrupts.IWI:IRQ_work_interrupts 389189 ± 4% -31.5% 266460 ± 8% interrupts.NMI:Non-maskable_interrupts 389189 ± 4% -31.5% 266460 ± 8% interrupts.PMI:Performance_monitoring_interrupts 1296392 ± 4% -34.6% 847860 ± 22% interrupts.RES:Rescheduling_interrupts 1588 ± 29% -61.8% 607.25 ± 35% interrupts.TLB:TLB_shootdowns 10.18 -4.0% 9.77 perf-stat.i.MPKI 1.552e+10 -25.5% 1.157e+10 perf-stat.i.branch-instructions 3.48e+08 -25.2% 2.605e+08 perf-stat.i.branch-misses 1.52 ± 28% -1.1 0.43 ± 27% perf-stat.i.cache-miss-rate% 11586741 ± 29% -83.7% 1894187 ± 32% perf-stat.i.cache-misses 8.042e+08 -28.5% 5.749e+08 perf-stat.i.cache-references 1492579 ± 2% -24.4% 1127729 perf-stat.i.context-switches 1.22 +23.5% 1.51 ± 2% perf-stat.i.cpi 9.616e+10 -7.9% 8.853e+10 perf-stat.i.cpu-cycles 162.88 ± 9% +39.8% 227.77 ± 4% perf-stat.i.cpu-migrations 10318 ± 43% +758.6% 88595 ± 30% perf-stat.i.cycles-between-cache-misses 10338031 -26.2% 7628495 perf-stat.i.dTLB-load-misses 2.437e+10 -25.5% 1.815e+10 perf-stat.i.dTLB-loads 0.00 ± 5% +0.0 0.00 ± 2% perf-stat.i.dTLB-store-miss-rate% 12952 ± 3% +12.2% 14538 ± 2% perf-stat.i.dTLB-store-misses 1.401e+10 -25.6% 1.041e+10 perf-stat.i.dTLB-stores 49.00 -13.4 35.64 ± 6% perf-stat.i.iTLB-load-miss-rate% 11130067 ± 2% -29.9% 7798895 ± 3% perf-stat.i.iTLB-load-misses 11481481 +22.6% 14080265 ± 7% perf-stat.i.iTLB-loads 7.868e+10 -25.5% 5.862e+10 perf-stat.i.instructions 7290 ± 2% +6.5% 7762 perf-stat.i.instructions-per-iTLB-miss 0.82 -18.9% 0.66 ± 2% perf-stat.i.ipc 2680 -2.2% 2621 perf-stat.i.minor-faults 95.89 -3.7 92.22 perf-stat.i.node-load-miss-rate% 1617319 ± 41% -83.8% 262514 ± 32% perf-stat.i.node-load-misses 53805 ± 29% -57.3% 22960 ± 19% perf-stat.i.node-loads 96.01 -7.8 88.18 ± 2% perf-stat.i.node-store-miss-rate% 213867 ± 31% -86.0% 29841 ± 10% perf-stat.i.node-store-misses 6788 ± 7% -22.0% 5294 ± 12% perf-stat.i.node-stores 2680 -2.2% 2621 perf-stat.i.page-faults 10.22 -4.1% 9.81 perf-stat.overall.MPKI 1.44 ± 29% -1.1 0.33 ± 32% perf-stat.overall.cache-miss-rate% 1.22 +23.6% 1.51 ± 2% perf-stat.overall.cpi 9226 ± 34% +470.3% 52623 ± 35% perf-stat.overall.cycles-between-cache-misses 0.00 ± 3% +0.0 0.00 ± 3% perf-stat.overall.dTLB-store-miss-rate% 49.22 -13.5 35.72 ± 6% perf-stat.overall.iTLB-load-miss-rate% 7072 ± 2% +6.3% 7520 perf-stat.overall.instructions-per-iTLB-miss 0.82 -19.0% 0.66 ± 2% perf-stat.overall.ipc 96.27 -4.6 91.64 perf-stat.overall.node-load-miss-rate% 96.53 -11.8 84.69 ± 3% perf-stat.overall.node-store-miss-rate% 9587 +2.7% 9844 perf-stat.overall.path-length 1.547e+10 -25.5% 1.153e+10 perf-stat.ps.branch-instructions 3.468e+08 -25.1% 2.596e+08 perf-stat.ps.branch-misses 11548381 ± 29% -83.6% 1888636 ± 32% perf-stat.ps.cache-misses 8.015e+08 -28.5% 5.73e+08 perf-stat.ps.cache-references 1487561 ± 2% -24.4% 1123929 perf-stat.ps.context-switches 9.583e+10 -7.9% 8.823e+10 perf-stat.ps.cpu-cycles 162.41 ± 9% +40.1% 227.57 ± 4% perf-stat.ps.cpu-migrations 10302904 -26.2% 7603035 perf-stat.ps.dTLB-load-misses 2.429e+10 -25.5% 1.809e+10 perf-stat.ps.dTLB-loads 12914 ± 3% +12.4% 14511 ± 2% perf-stat.ps.dTLB-store-misses 1.396e+10 -25.6% 1.038e+10 perf-stat.ps.dTLB-stores 11092764 ± 2% -29.9% 7772733 ± 3% perf-stat.ps.iTLB-load-misses 11443703 +22.6% 14033574 ± 7% perf-stat.ps.iTLB-loads 7.841e+10 -25.5% 5.843e+10 perf-stat.ps.instructions 2671 -2.2% 2612 perf-stat.ps.minor-faults 1611848 ± 41% -83.8% 261634 ± 32% perf-stat.ps.node-load-misses 53714 ± 29% -57.3% 22938 ± 19% perf-stat.ps.node-loads 213258 ± 31% -86.0% 29755 ± 10% perf-stat.ps.node-store-misses 6777 ± 7% -21.8% 5301 ± 12% perf-stat.ps.node-stores 2671 -2.2% 2612 perf-stat.ps.page-faults 2.365e+13 -25.4% 1.764e+13 perf-stat.total.instructions 37.98 ± 2% -20.6 17.33 ± 5% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 38.08 ± 2% -20.1 17.96 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 38.03 ± 2% -20.1 17.91 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 40.82 ± 2% -19.6 21.26 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64 40.23 ± 2% -19.3 20.91 ± 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 40.24 ± 2% -19.3 20.93 ± 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 40.24 ± 2% -19.3 20.94 ± 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 0.89 ± 3% +0.1 1.03 ± 2% perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established 0.62 ± 4% +0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64 0.59 ± 5% +0.2 0.75 ± 6% perf-profile.calltrace.cycles-pp.__inet_lookup_established.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 0.54 ± 4% +0.2 0.73 ± 4% perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_release.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor 0.42 ± 57% +0.2 0.62 ± 2% perf-profile.calltrace.cycles-pp.page_cache_pipe_buf_confirm.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor 0.58 ± 5% +0.2 0.80 ± 4% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken 1.74 +0.2 1.96 perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg 0.57 ± 4% +0.2 0.79 ± 3% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 0.68 ± 2% +0.3 0.94 perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 0.63 ± 4% +0.3 0.89 ± 2% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.68 ± 4% +0.3 0.93 ± 3% perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 1.09 ± 4% +0.3 1.37 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv 2.04 ± 2% +0.3 2.31 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 0.61 ± 5% +0.3 0.89 perf-profile.calltrace.cycles-pp.copy_page_to_iter.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct 0.63 ± 6% +0.3 0.91 ± 3% perf-profile.calltrace.cycles-pp.ipv4_mtu.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked 1.91 ± 2% +0.3 2.19 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.83 ± 4% +0.3 1.14 perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.69 ± 2% +0.3 1.02 perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage 2.07 ± 6% +0.3 2.40 ± 4% perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor 0.72 ± 3% +0.3 1.06 perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 2.17 ± 6% +0.4 2.53 ± 3% perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct 2.04 +0.4 2.41 perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 0.84 ± 3% +0.4 1.21 ± 4% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary 0.87 ± 3% +0.4 1.24 ± 4% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.15 ± 3% +0.4 1.59 ± 3% perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg 1.03 ± 3% +0.4 1.48 ± 3% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data 1.06 ± 3% +0.5 1.52 ± 3% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg 0.13 ±173% +0.5 0.60 ± 3% perf-profile.calltrace.cycles-pp._copy_from_user.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.13 ±173% +0.5 0.64 perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile 1.70 ± 4% +0.5 2.20 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 1.25 ± 3% +0.5 1.76 ± 3% perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 1.38 ± 3% +0.5 1.90 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv 0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64 0.00 +0.5 0.53 ± 2% perf-profile.calltrace.cycles-pp.__x86_indirect_thunk_rax 3.01 ± 2% +0.5 3.55 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 0.00 +0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg 1.50 ± 3% +0.5 2.04 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 1.25 +0.6 1.81 ± 2% perf-profile.calltrace.cycles-pp.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage 0.00 +0.6 0.56 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 0.13 ±173% +0.6 0.69 ± 5% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout 1.60 ± 3% +0.6 2.16 ± 3% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 0.00 +0.6 0.57 ± 3% perf-profile.calltrace.cycles-pp.fsnotify.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.15 ± 2% +0.6 3.74 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg 1.20 ± 21% +0.6 1.78 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established 1.66 ± 3% +0.6 2.26 ± 3% perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 0.00 +0.6 0.60 ± 2% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output 0.00 +0.6 0.61 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage 0.00 +0.6 0.62 ± 13% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 1.41 +0.6 2.03 ± 2% perf-profile.calltrace.cycles-pp.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 0.00 +0.6 0.64 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 0.00 +0.7 0.68 ± 3% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg 1.30 ± 8% +0.7 2.00 ± 9% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage 1.82 ± 2% +0.7 2.52 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages 2.09 ± 4% +0.7 2.79 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 1.33 ± 9% +0.7 2.04 ± 9% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage.inet_sendpage 0.98 ± 28% +0.7 1.73 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb 2.00 +0.8 2.77 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked 3.62 ± 3% +0.8 4.41 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 1.57 ± 9% +0.8 2.37 ± 9% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 1.57 ± 9% +0.8 2.38 ± 9% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 0.00 +0.9 0.90 ± 29% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit 2.21 ± 9% +1.1 3.30 ± 9% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage 2.56 ± 4% +1.1 3.65 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 2.33 ± 3% +1.1 3.43 ± 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established 2.57 ± 4% +1.1 3.68 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 4.25 ± 2% +1.1 5.36 perf-profile.calltrace.cycles-pp.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile 3.06 ± 2% +1.2 4.27 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage 3.09 +1.2 4.32 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 3.83 ± 4% +1.4 5.18 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage 6.54 ± 2% +1.4 7.91 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 6.57 ± 2% +1.4 7.96 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 2.85 ± 7% +1.4 4.27 ± 7% perf-profile.calltrace.cycles-pp.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage 4.88 ± 2% +1.5 6.33 perf-profile.calltrace.cycles-pp.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 3.58 +1.5 5.04 ± 2% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit 4.26 ± 3% +1.5 5.76 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage 3.34 +1.5 4.88 ± 4% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv 3.74 +1.6 5.38 ± 2% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames 4.67 ± 4% +1.7 6.36 ± 2% perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage 7.08 +2.7 9.80 perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage 7.22 ± 2% +2.7 9.96 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 7.26 ± 2% +2.8 10.01 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 7.66 +2.8 10.47 perf-profile.calltrace.cycles-pp.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage 8.00 ± 2% +3.2 11.16 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 8.20 ± 2% +3.3 11.50 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg 8.45 ± 2% +3.3 11.79 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 8.32 ± 2% +3.3 11.65 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg 9.22 +3.5 12.69 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv 9.33 +3.5 12.84 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 9.37 +3.5 12.90 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 9.48 +3.6 13.05 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 9.67 +3.7 13.39 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start 9.87 +3.9 13.76 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack 10.15 +4.0 14.14 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq 10.35 +4.1 14.43 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 10.48 +4.2 14.65 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output 10.46 +4.2 14.64 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2 10.55 +4.2 14.76 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit 10.55 +4.2 14.78 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb 15.87 ± 2% +6.1 21.99 ± 2% perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage 16.60 ± 2% +6.3 22.89 ± 2% perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe 16.70 ± 2% +6.4 23.05 ± 2% perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe 17.05 ± 2% +6.4 23.46 ± 2% perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor 17.48 ± 2% +6.5 24.01 ± 2% perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor 20.28 +6.8 27.05 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 20.33 +6.8 27.13 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.76 +6.9 27.67 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.80 +6.9 27.72 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 19.00 ± 2% +7.0 26.05 ± 2% perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct 19.36 ± 2% +7.1 26.49 ± 2% perf-profile.calltrace.cycles-pp.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile 19.67 ± 2% +7.3 26.97 ± 2% perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 25.77 ± 2% +9.3 35.08 perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64 26.29 ± 2% +9.5 35.75 perf-profile.calltrace.cycles-pp.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 28.65 ± 2% +10.4 39.06 perf-profile.calltrace.cycles-pp.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 29.77 ± 2% +10.7 40.51 perf-profile.calltrace.cycles-pp.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 54.03 +18.5 72.54 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 54.22 +18.5 72.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 37.98 ± 2% -20.6 17.34 ± 5% perf-profile.children.cycles-pp.intel_idle 38.62 ± 2% -20.4 18.26 ± 5% perf-profile.children.cycles-pp.cpuidle_enter 38.62 ± 2% -20.4 18.26 ± 5% perf-profile.children.cycles-pp.cpuidle_enter_state 40.82 ± 2% -19.6 21.26 ± 4% perf-profile.children.cycles-pp.secondary_startup_64 40.82 ± 2% -19.6 21.26 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 40.82 ± 2% -19.6 21.26 ± 4% perf-profile.children.cycles-pp.do_idle 40.24 ± 2% -19.3 20.94 ± 4% perf-profile.children.cycles-pp.start_secondary 0.05 +0.0 0.06 ± 6% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.08 +0.0 0.10 ± 5% perf-profile.children.cycles-pp.tcp_update_skb_after_send 0.11 ± 3% +0.0 0.13 ± 5% perf-profile.children.cycles-pp.nr_iowait_cpu 0.07 ± 6% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.05 +0.0 0.07 ± 12% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.06 ± 7% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rb_next 0.07 ± 10% +0.0 0.09 ± 7% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.10 ± 10% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.09 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64 0.16 ± 6% +0.0 0.18 ± 4% perf-profile.children.cycles-pp.tcp_check_space 0.05 ± 9% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.__x86_indirect_thunk_r14 0.11 ± 8% +0.0 0.13 ± 3% perf-profile.children.cycles-pp.rcu_idle_exit 0.06 ± 6% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.pick_next_entity 0.09 ± 7% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.sock_rfree 0.05 ± 8% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.tcp_tx_timestamp 0.12 ± 7% +0.0 0.14 ± 5% perf-profile.children.cycles-pp.remove_wait_queue 0.11 ± 11% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.native_write_msr 0.20 ± 2% +0.0 0.22 ± 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.05 ± 9% +0.0 0.08 ± 8% perf-profile.children.cycles-pp.tcp_recv_timestamp 0.23 ± 2% +0.0 0.25 ± 6% perf-profile.children.cycles-pp.ip_send_check 0.08 ± 5% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.__x86_indirect_thunk_r10 0.07 ± 6% +0.0 0.10 ± 9% perf-profile.children.cycles-pp.cpus_share_cache 0.11 ± 4% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.check_stack_object 0.09 ± 8% +0.0 0.12 ± 10% perf-profile.children.cycles-pp.update_cfs_group 0.07 ± 5% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.finish_task_switch 0.18 ± 4% +0.0 0.21 ± 5% perf-profile.children.cycles-pp.generic_splice_sendpage 0.12 +0.0 0.15 ± 2% perf-profile.children.cycles-pp.aa_sk_perm 0.12 ± 4% +0.0 0.16 ± 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.11 ± 8% +0.0 0.14 ± 6% perf-profile.children.cycles-pp.update_ts_time_stats 0.10 ± 8% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.sched_clock 0.06 +0.0 0.09 ± 4% perf-profile.children.cycles-pp.tcp_event_data_recv 0.04 ± 57% +0.0 0.07 perf-profile.children.cycles-pp.task_tick_fair 0.04 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.bictcp_acked 0.10 ± 7% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.native_sched_clock 0.08 ± 8% +0.0 0.12 ± 7% perf-profile.children.cycles-pp.check_preempt_curr 0.05 ± 8% +0.0 0.09 ± 9% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.05 +0.0 0.09 ± 4% perf-profile.children.cycles-pp.kfree_skbmem 0.06 ± 9% +0.0 0.09 ± 4% perf-profile.children.cycles-pp.__copy_skb_header 0.16 ± 2% +0.0 0.20 ± 3% perf-profile.children.cycles-pp.tcp_wfree 0.14 ± 3% +0.0 0.17 ± 4% perf-profile.children.cycles-pp.sockfd_lookup_light 0.06 +0.0 0.10 ± 4% perf-profile.children.cycles-pp.newidle_balance 0.11 ± 7% +0.0 0.15 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu 0.15 ± 3% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.sock_recvmsg 0.07 +0.0 0.11 perf-profile.children.cycles-pp.inet_ehashfn 0.07 ± 11% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.tcp_update_pacing_rate 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.tcp_chrono_stop 0.13 ± 3% +0.0 0.17 ± 2% perf-profile.children.cycles-pp.security_socket_recvmsg 0.10 ± 5% +0.0 0.14 ± 6% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.18 ± 4% +0.0 0.22 ± 3% perf-profile.children.cycles-pp.inet_send_prepare 0.06 ± 9% +0.0 0.10 ± 8% perf-profile.children.cycles-pp.tcp_push 0.22 ± 3% +0.0 0.26 ± 3% perf-profile.children.cycles-pp.select_task_rq_fair 0.12 ± 4% +0.0 0.17 ± 2% perf-profile.children.cycles-pp.__switch_to_asm 0.09 ± 7% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.__tcp_select_window 0.21 ± 4% +0.0 0.26 ± 4% perf-profile.children.cycles-pp.tcp_rcv_space_adjust 0.07 ± 10% +0.0 0.12 ± 9% perf-profile.children.cycles-pp.netif_skb_features 0.01 ±173% +0.0 0.06 ± 14% perf-profile.children.cycles-pp.raw_local_deliver 0.21 ± 6% +0.0 0.26 ± 6% perf-profile.children.cycles-pp.__put_user_8 0.05 ± 8% +0.0 0.10 perf-profile.children.cycles-pp.___perf_sw_event 0.32 ± 3% +0.0 0.36 perf-profile.children.cycles-pp.tcp_rate_check_app_limited 0.00 +0.1 0.05 perf-profile.children.cycles-pp.default_wake_function 0.08 ± 10% +0.1 0.13 ± 9% perf-profile.children.cycles-pp.sanity 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash 0.03 ±100% +0.1 0.08 ± 10% perf-profile.children.cycles-pp.ip_queue_xmit 0.21 ± 3% +0.1 0.26 ± 3% perf-profile.children.cycles-pp.aa_file_perm 0.15 ± 5% +0.1 0.20 ± 5% perf-profile.children.cycles-pp.__switch_to 0.15 ± 3% +0.1 0.20 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.iov_iter_pipe 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.__usecs_to_jiffies 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.bictcp_cong_avoid 0.07 ± 6% +0.1 0.12 ± 11% perf-profile.children.cycles-pp.__list_del_entry_valid 0.01 ±173% +0.1 0.07 ± 10% perf-profile.children.cycles-pp.resched_curr 0.15 ± 9% +0.1 0.21 ± 3% perf-profile.children.cycles-pp.rcu_all_qs 0.09 ± 7% +0.1 0.15 ± 12% perf-profile.children.cycles-pp._find_next_bit 0.07 ± 12% +0.1 0.13 ± 6% perf-profile.children.cycles-pp.kfree 0.07 ± 7% +0.1 0.12 ± 6% perf-profile.children.cycles-pp.tcp_ack_update_rtt 0.04 ± 57% +0.1 0.10 ± 4% perf-profile.children.cycles-pp.apparmor_ipv4_postroute 0.00 +0.1 0.06 perf-profile.children.cycles-pp.tcp_v4_fill_cb 0.00 +0.1 0.06 perf-profile.children.cycles-pp._raw_spin_unlock_bh 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.tcp_options_write 0.08 ± 10% +0.1 0.14 ± 10% perf-profile.children.cycles-pp.scheduler_tick 0.26 +0.1 0.32 ± 4% perf-profile.children.cycles-pp.__ip_local_out 0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.eth_type_trans 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.rcu_eqs_enter 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.tcp_stream_memory_free 0.28 +0.1 0.34 ± 5% perf-profile.children.cycles-pp.ip_local_out 0.07 ± 10% +0.1 0.14 ± 8% perf-profile.children.cycles-pp.ip_rcv_finish 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.ksoftirqd_running 0.11 ± 4% +0.1 0.17 ± 6% perf-profile.children.cycles-pp.tcp_v4_send_check 0.35 +0.1 0.42 ± 5% perf-profile.children.cycles-pp.__ksize 0.12 ± 7% +0.1 0.18 ± 6% perf-profile.children.cycles-pp.validate_xmit_skb 0.09 ± 4% +0.1 0.16 ± 5% perf-profile.children.cycles-pp.skb_entail 0.24 ± 2% +0.1 0.32 ± 4% perf-profile.children.cycles-pp.__sk_dst_check 0.20 ± 5% +0.1 0.27 ± 4% perf-profile.children.cycles-pp.sock_put 0.00 +0.1 0.07 perf-profile.children.cycles-pp.put_prev_task_fair 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.account_entity_dequeue 0.09 +0.1 0.16 ± 7% perf-profile.children.cycles-pp.ip_rcv_core 0.42 +0.1 0.49 ± 3% perf-profile.children.cycles-pp.skb_release_head_state 0.24 ± 4% +0.1 0.31 ± 2% perf-profile.children.cycles-pp.rw_verify_area 0.21 ± 2% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.ipv4_dst_check 0.44 +0.1 0.51 ± 3% perf-profile.children.cycles-pp.skb_release_all 0.04 ± 57% +0.1 0.11 ± 7% perf-profile.children.cycles-pp.tcp_rearm_rto 0.01 ±173% +0.1 0.09 ± 4% perf-profile.children.cycles-pp.ip_rcv_finish_core 0.23 ± 6% +0.1 0.30 ± 2% perf-profile.children.cycles-pp.update_curr 0.00 +0.1 0.08 ± 11% perf-profile.children.cycles-pp.__tcp_ack_snd_check 0.17 ± 7% +0.1 0.25 ± 5% perf-profile.children.cycles-pp.skb_clone 0.11 ± 17% +0.1 0.19 ± 3% perf-profile.children.cycles-pp.xas_start 0.17 ± 4% +0.1 0.25 ± 5% perf-profile.children.cycles-pp.kmem_cache_free 0.00 +0.1 0.08 ± 10% perf-profile.children.cycles-pp.tcp_rack_advance 0.55 ± 4% +0.1 0.64 ± 2% perf-profile.children.cycles-pp.page_cache_pipe_buf_confirm 0.12 ± 4% +0.1 0.21 ± 3% perf-profile.children.cycles-pp.timestamp_truncate 0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.rb_first 0.30 ± 8% +0.1 0.39 ± 2% perf-profile.children.cycles-pp._cond_resched 0.16 ± 2% +0.1 0.25 ± 7% perf-profile.children.cycles-pp.tcp_add_backlog 0.17 ± 4% +0.1 0.26 ± 9% perf-profile.children.cycles-pp.__next_timer_interrupt 0.38 +0.1 0.47 ± 5% perf-profile.children.cycles-pp.__skb_clone 0.17 ± 2% +0.1 0.26 ± 4% perf-profile.children.cycles-pp.ktime_get_with_offset 0.12 ± 4% +0.1 0.22 ± 11% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.00 +0.1 0.09 ± 4% perf-profile.children.cycles-pp.tcp_cleanup_rbuf 0.11 ± 6% +0.1 0.20 ± 7% perf-profile.children.cycles-pp.__update_load_avg_se 0.00 +0.1 0.10 ± 11% perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb 0.37 +0.1 0.47 ± 3% perf-profile.children.cycles-pp.dst_release 0.25 ± 3% +0.1 0.35 ± 2% perf-profile.children.cycles-pp.enqueue_to_backlog 0.17 ± 6% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.__ip_finish_output 0.11 ± 6% +0.1 0.22 ± 8% perf-profile.children.cycles-pp.nf_hook_slow 0.36 ± 3% +0.1 0.47 ± 4% perf-profile.children.cycles-pp.set_next_entity 0.20 ± 2% +0.1 0.31 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.12 ± 5% +0.1 0.23 ± 4% perf-profile.children.cycles-pp.update_process_times 0.00 +0.1 0.11 ± 15% perf-profile.children.cycles-pp.rcu_core 0.04 ± 57% +0.1 0.15 ± 13% perf-profile.children.cycles-pp.security_sock_rcv_skb 0.26 ± 4% +0.1 0.38 ± 6% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.49 ± 3% +0.1 0.61 ± 3% perf-profile.children.cycles-pp._copy_from_user 0.12 ± 5% +0.1 0.24 ± 5% perf-profile.children.cycles-pp.tick_sched_handle 0.27 ± 8% +0.1 0.39 perf-profile.children.cycles-pp.xas_load 0.24 ± 8% +0.1 0.37 ± 4% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.21 ± 2% +0.1 0.34 perf-profile.children.cycles-pp.tcp_schedule_loss_probe 0.48 ± 4% +0.1 0.60 perf-profile.children.cycles-pp.__fget_light 0.21 ± 3% +0.1 0.34 ± 5% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 0.23 ± 3% +0.1 0.36 ± 6% perf-profile.children.cycles-pp.__kmalloc_reserve 0.32 ± 3% +0.1 0.46 ± 6% perf-profile.children.cycles-pp.__tcp_send_ack 0.32 ± 4% +0.1 0.45 ± 5% perf-profile.children.cycles-pp.tick_nohz_next_event 0.10 ± 11% +0.1 0.24 ± 8% perf-profile.children.cycles-pp.sk_filter_trim_cap 0.18 ± 10% +0.1 0.32 ± 2% perf-profile.children.cycles-pp.__netif_receive_skb_core 0.12 ± 10% +0.1 0.27 ± 14% perf-profile.children.cycles-pp.clockevents_program_event 0.53 +0.1 0.68 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.31 ± 3% +0.1 0.45 ± 4% perf-profile.children.cycles-pp.tcp_established_options 0.14 ± 5% +0.1 0.29 ± 5% perf-profile.children.cycles-pp.tick_sched_timer 0.10 ± 4% +0.1 0.25 ± 5% perf-profile.children.cycles-pp.tcp_release_cb 0.35 ± 2% +0.2 0.50 perf-profile.children.cycles-pp.__might_sleep 0.30 ± 8% +0.2 0.45 ± 7% perf-profile.children.cycles-pp.__virt_addr_valid 0.34 ± 2% +0.2 0.50 ± 6% perf-profile.children.cycles-pp.tcp_mstamp_refresh 0.41 ± 5% +0.2 0.57 ± 4% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.29 ± 2% +0.2 0.46 perf-profile.children.cycles-pp.__fsnotify_parent 0.26 ± 2% +0.2 0.43 ± 4% perf-profile.children.cycles-pp.__slab_free 0.34 ± 5% +0.2 0.52 ± 2% perf-profile.children.cycles-pp.mod_timer 0.08 ± 13% +0.2 0.26 ± 15% perf-profile.children.cycles-pp.irq_exit 0.18 ± 8% +0.2 0.36 ± 3% perf-profile.children.cycles-pp.apparmor_file_permission 0.33 ± 4% +0.2 0.51 ± 4% perf-profile.children.cycles-pp.current_time 0.35 ± 5% +0.2 0.54 ± 2% perf-profile.children.cycles-pp.sk_reset_timer 0.55 ± 4% +0.2 0.73 ± 4% perf-profile.children.cycles-pp.page_cache_pipe_buf_release 0.31 ± 6% +0.2 0.51 ± 2% perf-profile.children.cycles-pp.tcp_event_new_data_sent 0.78 ± 4% +0.2 0.97 ± 3% perf-profile.children.cycles-pp.__inet_lookup_established 1.16 +0.2 1.37 ± 2% perf-profile.children.cycles-pp.skb_release_data 0.44 +0.2 0.65 perf-profile.children.cycles-pp.netif_rx_internal 0.45 ± 2% +0.2 0.66 ± 2% perf-profile.children.cycles-pp.netif_rx 0.48 ± 5% +0.2 0.70 ± 5% perf-profile.children.cycles-pp.dequeue_entity 0.48 ± 3% +0.2 0.70 ± 3% perf-profile.children.cycles-pp.__might_fault 0.59 ± 5% +0.2 0.81 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair 0.76 +0.2 0.99 ± 4% perf-profile.children.cycles-pp.read_tsc 0.53 ± 2% +0.2 0.76 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair 1.77 +0.2 2.00 perf-profile.children.cycles-pp.__check_object_size 0.48 ± 4% +0.2 0.72 ± 3% perf-profile.children.cycles-pp.enqueue_entity 1.24 ± 4% +0.2 1.48 perf-profile.children.cycles-pp.common_file_perm 0.69 ± 2% +0.2 0.94 perf-profile.children.cycles-pp.do_splice_to 0.65 ± 5% +0.3 0.90 ± 2% perf-profile.children.cycles-pp.menu_select 0.49 ± 5% +0.3 0.76 ± 2% perf-profile.children.cycles-pp.fsnotify 0.62 ± 5% +0.3 0.90 perf-profile.children.cycles-pp.copy_page_to_iter 0.63 ± 2% +0.3 0.90 perf-profile.children.cycles-pp._raw_spin_lock 2.04 ± 2% +0.3 2.32 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.42 ± 10% +0.3 0.71 ± 10% perf-profile.children.cycles-pp.hrtimer_interrupt 0.44 ± 4% +0.3 0.74 ± 4% perf-profile.children.cycles-pp.update_load_avg 0.89 ± 2% +0.3 1.20 perf-profile.children.cycles-pp.___might_sleep 0.57 ± 3% +0.3 0.90 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair 2.17 ± 2% +0.3 2.49 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.58 ± 3% +0.3 0.93 ± 3% perf-profile.children.cycles-pp.ttwu_do_activate 0.58 ± 3% +0.3 0.92 ± 3% perf-profile.children.cycles-pp.activate_task 0.53 ± 4% +0.3 0.87 ± 2% perf-profile.children.cycles-pp.atime_needs_update 0.72 ± 2% +0.3 1.07 perf-profile.children.cycles-pp.sk_stream_alloc_skb 2.08 ± 6% +0.3 2.42 ± 4% perf-profile.children.cycles-pp.find_get_entry 0.77 ± 6% +0.4 1.13 ± 3% perf-profile.children.cycles-pp.ipv4_mtu 2.06 +0.4 2.42 perf-profile.children.cycles-pp.simple_copy_to_iter 2.19 ± 6% +0.4 2.55 ± 3% perf-profile.children.cycles-pp.pagecache_get_page 0.88 ± 3% +0.4 1.25 ± 3% perf-profile.children.cycles-pp.schedule_idle 0.96 ± 2% +0.4 1.37 perf-profile.children.cycles-pp.loopback_xmit 1.16 ± 3% +0.4 1.59 ± 3% perf-profile.children.cycles-pp.schedule_timeout 1.01 ± 2% +0.4 1.46 perf-profile.children.cycles-pp.dev_hard_start_xmit 0.98 +0.4 1.43 ± 2% perf-profile.children.cycles-pp.__alloc_skb 0.66 ± 4% +0.5 1.12 ± 2% perf-profile.children.cycles-pp.touch_atime 1.06 ± 3% +0.5 1.52 ± 3% perf-profile.children.cycles-pp.schedule 1.32 ± 3% +0.5 1.78 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 0.98 ± 2% +0.5 1.45 perf-profile.children.cycles-pp.ktime_get 1.84 +0.5 2.31 ± 2% perf-profile.children.cycles-pp.__kfree_skb 0.77 ± 2% +0.5 1.25 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.56 ± 11% +0.5 1.04 ± 7% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.65 ± 12% +0.5 1.12 ± 6% perf-profile.children.cycles-pp.apic_timer_interrupt 1.25 ± 3% +0.5 1.76 ± 3% perf-profile.children.cycles-pp.wait_woken 1.38 ± 3% +0.5 1.91 ± 3% perf-profile.children.cycles-pp.__wake_up_common 1.49 ± 3% +0.6 2.04 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock 1.77 ± 3% +0.6 2.33 perf-profile.children.cycles-pp.security_file_permission 3.19 ± 2% +0.6 3.78 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 1.66 ± 3% +0.6 2.25 ± 2% perf-profile.children.cycles-pp.sock_def_readable 3.17 ± 2% +0.6 3.75 perf-profile.children.cycles-pp.copyout 1.67 ± 3% +0.6 2.27 ± 3% perf-profile.children.cycles-pp.sk_wait_data 1.36 +0.6 1.97 ± 2% perf-profile.children.cycles-pp.tcp_current_mss 1.41 +0.6 2.05 ± 2% perf-profile.children.cycles-pp.tcp_send_mss 2.01 ± 2% +0.7 2.66 ± 2% perf-profile.children.cycles-pp.tcp_clean_rtx_queue 1.54 ± 2% +0.7 2.21 perf-profile.children.cycles-pp.__dev_queue_xmit 3.65 ± 3% +0.8 4.44 perf-profile.children.cycles-pp._copy_to_iter 1.91 ± 3% +0.8 2.73 ± 3% perf-profile.children.cycles-pp.__sched_text_start 2.45 ± 2% +0.9 3.35 ± 2% perf-profile.children.cycles-pp.tcp_ack 4.28 ± 2% +1.1 5.41 perf-profile.children.cycles-pp.generic_file_read_iter 2.78 ± 7% +1.3 4.10 ± 7% perf-profile.children.cycles-pp.__release_sock 6.55 ± 2% +1.4 7.93 perf-profile.children.cycles-pp.__skb_datagram_iter 6.57 ± 2% +1.4 7.96 perf-profile.children.cycles-pp.skb_copy_datagram_iter 3.92 ± 4% +1.4 5.33 ± 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 4.88 ± 2% +1.5 6.34 perf-profile.children.cycles-pp.generic_file_splice_read 3.58 ± 6% +1.7 5.26 ± 6% perf-profile.children.cycles-pp.release_sock 4.98 ± 3% +1.7 6.70 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_bh 4.96 ± 3% +1.8 6.72 ± 2% perf-profile.children.cycles-pp.lock_sock_nested 7.11 +2.7 9.84 perf-profile.children.cycles-pp.do_tcp_sendpages 7.67 +2.8 10.50 perf-profile.children.cycles-pp.tcp_sendpage_locked 7.20 +3.1 10.33 ± 2% perf-profile.children.cycles-pp.tcp_write_xmit 7.25 +3.1 10.39 ± 2% perf-profile.children.cycles-pp.__tcp_push_pending_frames 9.24 +3.5 12.71 perf-profile.children.cycles-pp.tcp_v4_rcv 9.33 +3.5 12.84 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 9.37 +3.5 12.90 perf-profile.children.cycles-pp.ip_local_deliver_finish 9.49 +3.6 13.07 perf-profile.children.cycles-pp.ip_local_deliver 8.91 +3.6 12.53 ± 2% perf-profile.children.cycles-pp.tcp_rcv_established 9.07 +3.7 12.73 ± 2% perf-profile.children.cycles-pp.tcp_v4_do_rcv 9.67 +3.7 13.41 perf-profile.children.cycles-pp.ip_rcv 9.88 +3.9 13.77 perf-profile.children.cycles-pp.__netif_receive_skb_one_core 10.16 +4.0 14.16 perf-profile.children.cycles-pp.process_backlog 10.36 +4.1 14.44 perf-profile.children.cycles-pp.net_rx_action 10.48 +4.2 14.66 perf-profile.children.cycles-pp.do_softirq_own_stack 10.56 +4.2 14.79 perf-profile.children.cycles-pp.do_softirq 10.52 +4.3 14.84 perf-profile.children.cycles-pp.__softirqentry_text_start 10.80 +4.3 15.14 perf-profile.children.cycles-pp.__local_bh_enable_ip 11.79 +4.8 16.56 perf-profile.children.cycles-pp.ip_finish_output2 12.06 +5.0 17.04 perf-profile.children.cycles-pp.ip_output 12.61 +5.1 17.74 perf-profile.children.cycles-pp.__ip_queue_xmit 13.35 +5.5 18.82 perf-profile.children.cycles-pp.__tcp_transmit_skb 15.91 ± 2% +6.1 22.05 ± 2% perf-profile.children.cycles-pp.tcp_sendpage 16.62 ± 2% +6.3 22.92 ± 2% perf-profile.children.cycles-pp.inet_sendpage 16.71 ± 2% +6.4 23.07 ± 2% perf-profile.children.cycles-pp.kernel_sendpage 17.06 ± 2% +6.4 23.48 ± 2% perf-profile.children.cycles-pp.sock_sendpage 17.48 ± 2% +6.5 24.02 ± 2% perf-profile.children.cycles-pp.pipe_to_sendpage 20.29 +6.8 27.07 perf-profile.children.cycles-pp.tcp_recvmsg 20.33 +6.8 27.13 perf-profile.children.cycles-pp.inet_recvmsg 20.76 +6.9 27.67 perf-profile.children.cycles-pp.__sys_recvfrom 20.81 +6.9 27.73 perf-profile.children.cycles-pp.__x64_sys_recvfrom 19.04 ± 2% +7.1 26.10 ± 2% perf-profile.children.cycles-pp.__splice_from_pipe 19.37 ± 2% +7.1 26.51 ± 2% perf-profile.children.cycles-pp.splice_from_pipe 19.68 ± 2% +7.3 26.97 ± 2% perf-profile.children.cycles-pp.direct_splice_actor 25.77 ± 2% +9.3 35.09 perf-profile.children.cycles-pp.splice_direct_to_actor 26.31 ± 2% +9.5 35.77 perf-profile.children.cycles-pp.do_splice_direct 28.69 ± 2% +10.4 39.12 perf-profile.children.cycles-pp.do_sendfile 29.79 ± 2% +10.7 40.54 perf-profile.children.cycles-pp.__x64_sys_sendfile64 54.10 +18.5 72.61 perf-profile.children.cycles-pp.do_syscall_64 54.27 +18.5 72.80 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 37.98 ± 2% -20.6 17.34 ± 5% perf-profile.self.cycles-pp.intel_idle 0.10 ± 4% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.schedule_timeout 0.09 +0.0 0.11 ± 4% perf-profile.self.cycles-pp.check_stack_object 0.06 +0.0 0.08 ± 6% perf-profile.self.cycles-pp.finish_task_switch 0.08 ± 5% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.tcp_rcv_space_adjust 0.06 ± 6% +0.0 0.08 perf-profile.self.cycles-pp.pick_next_entity 0.06 ± 9% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.do_softirq 0.05 ± 9% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.tcp_recv_timestamp 0.11 ± 6% +0.0 0.13 ± 5% perf-profile.self.cycles-pp.nr_iowait_cpu 0.09 ± 7% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.sock_rfree 0.08 ± 5% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.__x86_indirect_thunk_r10 0.17 ± 4% +0.0 0.19 ± 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.10 ± 5% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.__sk_dst_check 0.11 ± 11% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.native_write_msr 0.10 ± 5% +0.0 0.12 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair 0.05 +0.0 0.08 ± 6% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu 0.11 ± 4% +0.0 0.14 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.10 ± 9% +0.0 0.12 ± 8% perf-profile.self.cycles-pp.__sys_recvfrom 0.08 ± 6% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.do_splice_direct 0.07 ± 10% +0.0 0.10 ± 11% perf-profile.self.cycles-pp.kmem_cache_free 0.07 ± 6% +0.0 0.10 ± 9% perf-profile.self.cycles-pp.cpus_share_cache 0.09 ± 9% +0.0 0.12 ± 7% perf-profile.self.cycles-pp.update_cfs_group 0.22 ± 3% +0.0 0.25 ± 4% perf-profile.self.cycles-pp.ip_send_check 0.17 ± 4% +0.0 0.20 ± 3% perf-profile.self.cycles-pp.inet_send_prepare 0.09 ± 4% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.aa_sk_perm 0.10 ± 4% +0.0 0.13 ± 6% perf-profile.self.cycles-pp.native_sched_clock 0.06 +0.0 0.09 ± 7% perf-profile.self.cycles-pp.do_splice_to 0.11 ± 7% +0.0 0.14 ± 5% perf-profile.self.cycles-pp.pagecache_get_page 0.15 ± 5% +0.0 0.18 perf-profile.self.cycles-pp._cond_resched 0.05 +0.0 0.08 ± 5% perf-profile.self.cycles-pp.kfree_skbmem 0.15 ± 3% +0.0 0.19 ± 4% perf-profile.self.cycles-pp.tcp_wfree 0.15 ± 4% +0.0 0.18 ± 5% perf-profile.self.cycles-pp.generic_splice_sendpage 0.05 ± 9% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.ktime_get_with_offset 0.05 ± 8% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.04 ± 57% +0.0 0.07 ± 10% perf-profile.self.cycles-pp._copy_from_user 0.04 ± 57% +0.0 0.07 perf-profile.self.cycles-pp.validate_xmit_skb 0.04 ± 58% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.rb_next 0.07 ± 10% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.__next_timer_interrupt 0.05 ± 8% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.dev_hard_start_xmit 0.16 ± 2% +0.0 0.20 ± 5% perf-profile.self.cycles-pp.sock_def_readable 0.07 ± 7% +0.0 0.10 ± 7% perf-profile.self.cycles-pp.copyout 0.06 +0.0 0.10 ± 5% perf-profile.self.cycles-pp.newidle_balance 0.04 ± 57% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.skb_release_head_state 0.04 ± 58% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.bictcp_acked 0.05 +0.0 0.09 ± 9% perf-profile.self.cycles-pp.tcp_push 0.05 ± 8% +0.0 0.09 ± 7% perf-profile.self.cycles-pp.tcp_event_data_recv 0.05 +0.0 0.09 ± 14% perf-profile.self.cycles-pp.nf_hook_slow 0.06 ± 9% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.__copy_skb_header 0.06 ± 6% +0.0 0.10 perf-profile.self.cycles-pp.tcp_schedule_loss_probe 0.07 ± 6% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.inet_ehashfn 0.11 ± 4% +0.0 0.15 ± 3% perf-profile.self.cycles-pp.ip_local_deliver 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.raw_local_deliver 0.04 ± 58% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.netif_skb_features 0.12 ± 4% +0.0 0.17 ± 2% perf-profile.self.cycles-pp.__switch_to_asm 0.09 ± 7% +0.0 0.13 ± 6% perf-profile.self.cycles-pp.__tcp_select_window 0.20 ± 4% +0.0 0.24 ± 2% perf-profile.self.cycles-pp.set_next_entity 0.08 ± 8% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.skb_entail 0.07 ± 6% +0.0 0.11 ± 3% perf-profile.self.cycles-pp.tcp_update_pacing_rate 0.32 ± 3% +0.0 0.36 perf-profile.self.cycles-pp.tcp_rate_check_app_limited 0.23 ± 4% +0.0 0.28 ± 6% perf-profile.self.cycles-pp.security_file_permission 0.21 ± 6% +0.0 0.26 ± 4% perf-profile.self.cycles-pp.__put_user_8 0.01 ±173% +0.0 0.06 perf-profile.self.cycles-pp.__x86_indirect_thunk_r14 0.24 ± 2% +0.0 0.29 ± 6% perf-profile.self.cycles-pp.tcp_sendpage_locked 0.10 ± 8% +0.0 0.15 ± 2% perf-profile.self.cycles-pp.rcu_all_qs 0.04 ± 58% +0.0 0.09 perf-profile.self.cycles-pp.sk_filter_trim_cap 0.06 ± 9% +0.1 0.11 ± 8% perf-profile.self.cycles-pp.tcp_ack_update_rtt 0.21 ± 3% +0.1 0.26 ± 2% perf-profile.self.cycles-pp.aa_file_perm 0.07 ± 5% +0.1 0.12 ± 4% perf-profile.self.cycles-pp.tcp_event_new_data_sent 0.09 ± 5% +0.1 0.14 ± 9% perf-profile.self.cycles-pp.kernel_sendpage 0.01 ±173% +0.1 0.07 ± 13% perf-profile.self.cycles-pp.inet_recvmsg 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.tcp_v4_fill_cb 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.ip_local_deliver_finish 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp._raw_spin_unlock_bh 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.iov_iter_pipe 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.bictcp_cong_avoid 0.33 +0.1 0.38 ± 6% perf-profile.self.cycles-pp.__skb_clone 0.17 +0.1 0.23 ± 3% perf-profile.self.cycles-pp.net_rx_action 0.14 ± 7% +0.1 0.20 ± 4% perf-profile.self.cycles-pp.__switch_to 0.07 ± 6% +0.1 0.12 ± 8% perf-profile.self.cycles-pp.sanity 0.00 +0.1 0.05 ± 9% perf-profile.self.cycles-pp.tcp_rack_advance 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.tcp_chrono_stop 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.__tcp_send_ack 0.08 ± 5% +0.1 0.13 ± 6% perf-profile.self.cycles-pp.__ip_finish_output 0.15 ± 4% +0.1 0.21 ± 2% perf-profile.self.cycles-pp.xas_load 0.15 ± 4% +0.1 0.21 ± 2% perf-profile.self.cycles-pp.enqueue_to_backlog 0.09 ± 7% +0.1 0.15 ± 15% perf-profile.self.cycles-pp._find_next_bit 0.14 ± 14% +0.1 0.20 ± 2% perf-profile.self.cycles-pp.lock_sock_nested 0.07 ± 7% +0.1 0.12 ± 8% perf-profile.self.cycles-pp.__list_del_entry_valid 0.07 ± 7% +0.1 0.12 ± 12% perf-profile.self.cycles-pp.kfree 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.tcp_options_write 0.01 ±173% +0.1 0.07 ± 10% perf-profile.self.cycles-pp.resched_curr 0.16 ± 5% +0.1 0.22 perf-profile.self.cycles-pp.process_backlog 0.15 ± 5% +0.1 0.21 ± 7% perf-profile.self.cycles-pp.release_sock 0.00 +0.1 0.06 perf-profile.self.cycles-pp.__ip_local_out 0.00 +0.1 0.06 perf-profile.self.cycles-pp.tcp_tx_timestamp 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.rcu_eqs_enter 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.netif_rx_internal 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.tcp_stream_memory_free 0.10 ± 4% +0.1 0.16 ± 6% perf-profile.self.cycles-pp.tcp_v4_send_check 0.13 ± 8% +0.1 0.19 ± 3% perf-profile.self.cycles-pp.update_curr 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.eth_type_trans 0.09 ± 7% +0.1 0.15 ± 5% perf-profile.self.cycles-pp.ip_output 0.35 +0.1 0.42 ± 5% perf-profile.self.cycles-pp.__ksize 0.11 ± 17% +0.1 0.17 ± 4% perf-profile.self.cycles-pp.xas_start 0.20 ± 2% +0.1 0.26 ± 3% perf-profile.self.cycles-pp.ipv4_dst_check 0.00 +0.1 0.07 ± 7% perf-profile.self.cycles-pp.ksoftirqd_running 0.20 ± 3% +0.1 0.27 ± 5% perf-profile.self.cycles-pp.sock_put 0.17 ± 7% +0.1 0.24 ± 5% perf-profile.self.cycles-pp.menu_select 0.09 +0.1 0.16 ± 5% perf-profile.self.cycles-pp.ip_rcv_core 0.00 +0.1 0.07 ± 16% perf-profile.self.cycles-pp.account_entity_dequeue 0.14 ± 5% +0.1 0.21 ± 6% perf-profile.self.cycles-pp.enqueue_entity 0.15 ± 7% +0.1 0.22 ± 12% perf-profile.self.cycles-pp.__might_fault 0.00 +0.1 0.07 perf-profile.self.cycles-pp.__tcp_push_pending_frames 0.00 +0.1 0.07 ± 10% perf-profile.self.cycles-pp.__tcp_ack_snd_check 0.00 +0.1 0.07 ± 10% perf-profile.self.cycles-pp.ip_queue_xmit 0.01 ±173% +0.1 0.08 ± 5% perf-profile.self.cycles-pp.ip_rcv_finish_core 0.53 ± 5% +0.1 0.60 ± 2% perf-profile.self.cycles-pp.page_cache_pipe_buf_confirm 0.22 ± 5% +0.1 0.29 ± 2% perf-profile.self.cycles-pp.rw_verify_area 0.13 ± 6% +0.1 0.21 ± 3% perf-profile.self.cycles-pp.mod_timer 0.00 +0.1 0.08 ± 6% perf-profile.self.cycles-pp.rb_first 0.00 +0.1 0.08 ± 6% perf-profile.self.cycles-pp.tcp_mstamp_refresh 0.29 +0.1 0.36 perf-profile.self.cycles-pp.sock_sendpage 0.28 ± 2% +0.1 0.36 ± 4% perf-profile.self.cycles-pp.__x64_sys_sendfile64 0.11 ± 4% +0.1 0.19 ± 4% perf-profile.self.cycles-pp.timestamp_truncate 0.00 +0.1 0.08 ± 5% perf-profile.self.cycles-pp.apparmor_ipv4_postroute 0.16 ± 10% +0.1 0.24 ± 3% perf-profile.self.cycles-pp.skb_clone 0.14 ± 3% +0.1 0.21 ± 5% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.10 ± 8% +0.1 0.18 ± 8% perf-profile.self.cycles-pp.enqueue_task_fair 0.00 +0.1 0.08 perf-profile.self.cycles-pp.___perf_sw_event 0.00 +0.1 0.08 ± 8% perf-profile.self.cycles-pp.tcp_rearm_rto 0.14 +0.1 0.22 ± 3% perf-profile.self.cycles-pp.tcp_send_mss 0.13 ± 3% +0.1 0.22 ± 6% perf-profile.self.cycles-pp.__kmalloc_node_track_caller 0.08 ± 5% +0.1 0.17 ± 9% perf-profile.self.cycles-pp.wait_woken 0.11 ± 6% +0.1 0.20 ± 7% perf-profile.self.cycles-pp.__update_load_avg_se 0.13 ± 6% +0.1 0.21 ± 5% perf-profile.self.cycles-pp.current_time 0.00 +0.1 0.09 ± 14% perf-profile.self.cycles-pp.apparmor_socket_sock_rcv_skb 0.12 ± 3% +0.1 0.21 ± 11% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.00 +0.1 0.09 ± 7% perf-profile.self.cycles-pp.tcp_cleanup_rbuf 0.52 +0.1 0.61 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.15 ± 8% +0.1 0.24 ± 4% perf-profile.self.cycles-pp.direct_splice_actor 0.36 +0.1 0.45 ± 4% perf-profile.self.cycles-pp.dst_release 0.16 ± 5% +0.1 0.25 perf-profile.self.cycles-pp.loopback_xmit 0.10 ± 8% +0.1 0.19 ± 7% perf-profile.self.cycles-pp.__softirqentry_text_start 0.40 +0.1 0.50 ± 2% perf-profile.self.cycles-pp.pipe_to_sendpage 0.26 ± 4% +0.1 0.36 ± 2% perf-profile.self.cycles-pp.splice_from_pipe 0.35 ± 5% +0.1 0.46 ± 4% perf-profile.self.cycles-pp.__sched_text_start 0.28 ± 2% +0.1 0.39 ± 2% perf-profile.self.cycles-pp.simple_copy_to_iter 0.13 ± 5% +0.1 0.24 ± 5% perf-profile.self.cycles-pp.touch_atime 0.26 ± 7% +0.1 0.37 perf-profile.self.cycles-pp.__local_bh_enable_ip 0.28 +0.1 0.39 ± 3% perf-profile.self.cycles-pp.__ip_queue_xmit 0.27 ± 7% +0.1 0.38 ± 3% perf-profile.self.cycles-pp.ip_finish_output2 0.40 ± 2% +0.1 0.52 ± 3% perf-profile.self.cycles-pp.tcp_rcv_established 0.35 ± 2% +0.1 0.47 ± 4% perf-profile.self.cycles-pp.tcp_clean_rtx_queue 0.44 ± 4% +0.1 0.56 perf-profile.self.cycles-pp.tcp_sendpage 0.48 ± 3% +0.1 0.60 ± 2% perf-profile.self.cycles-pp.inet_sendpage 0.29 ± 4% +0.1 0.41 ± 2% perf-profile.self.cycles-pp._copy_to_iter 0.21 ± 6% +0.1 0.34 ± 2% perf-profile.self.cycles-pp.update_load_avg 0.46 ± 4% +0.1 0.59 perf-profile.self.cycles-pp.__fget_light 0.38 ± 2% +0.1 0.51 ± 3% perf-profile.self.cycles-pp.tcp_v4_rcv 0.41 +0.1 0.55 ± 3% perf-profile.self.cycles-pp.tcp_recvmsg 0.20 ± 4% +0.1 0.33 ± 3% perf-profile.self.cycles-pp.__alloc_skb 0.35 ± 3% +0.1 0.49 ± 2% perf-profile.self.cycles-pp.generic_file_splice_read 0.29 ± 4% +0.1 0.43 ± 4% perf-profile.self.cycles-pp.tcp_established_options 0.24 ± 3% +0.1 0.38 ± 4% perf-profile.self.cycles-pp.tcp_ack 0.09 ± 4% +0.1 0.23 ± 6% perf-profile.self.cycles-pp.tcp_release_cb 0.33 +0.1 0.47 ± 2% perf-profile.self.cycles-pp.__might_sleep 0.38 ± 4% +0.1 0.52 ± 4% perf-profile.self.cycles-pp.__dev_queue_xmit 0.17 ± 8% +0.1 0.32 ± 2% perf-profile.self.cycles-pp.__netif_receive_skb_core 0.60 +0.1 0.75 perf-profile.self.cycles-pp.__skb_datagram_iter 0.28 ± 3% +0.1 0.43 ± 2% perf-profile.self.cycles-pp.__fsnotify_parent 0.28 ± 7% +0.2 0.44 ± 7% perf-profile.self.cycles-pp.__virt_addr_valid 0.15 ± 12% +0.2 0.30 ± 5% perf-profile.self.cycles-pp.apparmor_file_permission 0.70 ± 4% +0.2 0.86 ± 4% perf-profile.self.cycles-pp.__inet_lookup_established 0.20 ± 6% +0.2 0.36 ± 4% perf-profile.self.cycles-pp.atime_needs_update 0.52 ± 3% +0.2 0.69 ± 3% perf-profile.self.cycles-pp.page_cache_pipe_buf_release 0.25 ± 11% +0.2 0.42 ± 3% perf-profile.self.cycles-pp.splice_direct_to_actor 0.26 ± 3% +0.2 0.43 ± 4% perf-profile.self.cycles-pp.__slab_free 0.45 ± 9% +0.2 0.63 ± 5% perf-profile.self.cycles-pp.tcp_current_mss 1.00 ± 5% +0.2 1.20 perf-profile.self.cycles-pp.common_file_perm 0.33 ± 3% +0.2 0.53 ± 3% perf-profile.self.cycles-pp.__splice_from_pipe 0.73 ± 2% +0.2 0.93 ± 4% perf-profile.self.cycles-pp.read_tsc 1.15 +0.2 1.36 perf-profile.self.cycles-pp.skb_release_data 1.77 ± 5% +0.2 1.98 ± 4% perf-profile.self.cycles-pp.find_get_entry 0.54 ± 5% +0.2 0.77 perf-profile.self.cycles-pp.copy_page_to_iter 0.51 ± 4% +0.2 0.75 perf-profile.self.cycles-pp.do_sendfile 0.59 +0.3 0.84 ± 3% perf-profile.self.cycles-pp.__tcp_transmit_skb 0.47 ± 3% +0.3 0.73 perf-profile.self.cycles-pp.fsnotify 0.93 ± 3% +0.3 1.19 perf-profile.self.cycles-pp.generic_file_read_iter 0.59 +0.3 0.85 perf-profile.self.cycles-pp._raw_spin_lock 2.04 ± 2% +0.3 2.32 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.34 ± 6% +0.3 0.63 ± 5% perf-profile.self.cycles-pp.ktime_get 0.86 ± 2% +0.3 1.15 perf-profile.self.cycles-pp.___might_sleep 2.17 ± 2% +0.3 2.49 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.74 ± 6% +0.3 1.07 ± 4% perf-profile.self.cycles-pp.ipv4_mtu 1.08 +0.3 1.42 perf-profile.self.cycles-pp._raw_spin_lock_bh 1.67 ± 2% +0.3 2.01 ± 2% perf-profile.self.cycles-pp.do_tcp_sendpages 0.81 ± 2% +0.4 1.18 ± 2% perf-profile.self.cycles-pp.tcp_write_xmit 0.72 ± 3% +0.4 1.16 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 3.17 ± 2% +0.6 3.75 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 3.36 +0.8 4.15 perf-profile.self.cycles-pp.do_syscall_64 3.89 ± 4% +1.4 5.29 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath netperf.Throughput_Mbps 20000 +-------------------------------------------------------------------+ 18000 |.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.| | | 16000 |-+ | 14000 |-O O O O O O | | O O O O O O O O O O O O | 12000 |-+ | 10000 |-+ | 8000 |-+ | | | 6000 |-+ | 4000 |-+ | | | 2000 |-+ | 0 +-------------------------------------------------------------------+ netperf.Throughput_total_Mbps 350000 +------------------------------------------------------------------+ |.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+.+.+..+.+ +..+.+.+ +.+.| 300000 |-+ | | | 250000 |-O O O O O O O O O O O | | O O O O O O O | 200000 |-+ | | | 150000 |-+ | | | 100000 |-+ | | | 50000 |-+ | | | 0 +------------------------------------------------------------------+ netperf.workload 2.5e+09 +-----------------------------------------------------------------+ | +.+. | | | 2e+09 |-+ | | O O O O O O O O O O O O O O | | O O O O | 1.5e+09 |-+ | | | 1e+09 |-+ | | | | | 5e+08 |-+ | | | | | 0 +-----------------------------------------------------------------+ netperf.time.user_time 700 +---------------------------------------------------------------------+ | O O + O O O O O O | 600 |-+ O O O O O O O O O O | | | 500 |-+ | | | 400 |-+ | | | 300 |-+ | | | 200 |-+ | | | 100 |-+ | | | 0 +---------------------------------------------------------------------+ netperf.time.system_time 5000 +--------------------------------------------------------------------+ 4500 |.+..+.+.+..+.+.+.+..+.+.+..+.+.+..+.+.+.+..+.+.+..+.+.+..+.+.+.+..+.| | | 4000 |-+ | 3500 |-+ | | | 3000 |-+ | 2500 |-+ | 2000 |-+ | | | 1500 |-+ | 1000 |-+ | | | 500 |-+ | 0 +--------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-bdw-ex2: 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/25%/debian-x86_64-20191114.cgz/300s/10K/lkp-bdw-ex2/SCTP_STREAM_MANY/netperf/0xb000038 commit: 9c40365a65 ("threads: Update PID limit comment according to futex UAPI change") 59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group") 9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 8310 -31.1% 5723 ± 8% netperf.Throughput_Mbps 332426 -31.1% 228942 ± 8% netperf.Throughput_total_Mbps 13748 ± 2% -15.8% 11583 ± 7% netperf.time.minor_page_faults 587.05 -26.0% 434.31 ± 6% netperf.time.user_time 1.217e+09 -31.1% 8.384e+08 ± 8% netperf.workload 763112 ± 5% +19.0% 908201 ± 4% meminfo.Committed_AS 10662925 ± 10% +16.3% 12404278 meminfo.DirectMap2M 3.24 -0.8 2.48 ± 11% mpstat.cpu.all.soft% 2.02 -0.5 1.50 ± 6% mpstat.cpu.all.usr% 72.00 +2.1% 73.50 vmstat.cpu.id 2290192 -31.3% 1573919 ± 8% vmstat.system.cs 6.88e+09 ± 6% -54.1% 3.157e+09 ± 16% cpuidle.C1.time 3.412e+08 ± 2% -50.4% 1.691e+08 ± 17% cpuidle.C1.usage 45203190 ± 48% +165.0% 1.198e+08 ± 22% cpuidle.C1E.usage 25963 -4.1% 24904 proc-vmstat.nr_slab_reclaimable 66104 -6.4% 61871 proc-vmstat.nr_slab_unreclaimable 9019 ± 34% -77.7% 2009 ± 72% proc-vmstat.numa_hint_faults_local 1.565e+09 -31.1% 1.078e+09 ± 8% proc-vmstat.numa_hit 1.565e+09 -31.1% 1.078e+09 ± 8% proc-vmstat.numa_local 9.044e+09 -31.1% 6.228e+09 ± 8% proc-vmstat.pgalloc_normal 9.044e+09 -31.1% 6.228e+09 ± 8% proc-vmstat.pgfree 39376 ± 22% -56.0% 17319 ± 38% sched_debug.cfs_rq:/.MIN_vruntime.avg 3360482 ± 13% -57.8% 1416646 ± 30% sched_debug.cfs_rq:/.MIN_vruntime.max 333186 ± 18% -56.4% 145249 ± 31% sched_debug.cfs_rq:/.MIN_vruntime.stddev 39376 ± 22% -56.0% 17319 ± 38% sched_debug.cfs_rq:/.max_vruntime.avg 3360482 ± 13% -57.8% 1416646 ± 30% sched_debug.cfs_rq:/.max_vruntime.max 333186 ± 18% -56.4% 145249 ± 31% sched_debug.cfs_rq:/.max_vruntime.stddev 54158 ± 49% -81.3% 10104 ± 32% sched_debug.cfs_rq:/.min_vruntime.min 1726678 ± 2% +32.7% 2291681 ± 13% sched_debug.cfs_rq:/.min_vruntime.stddev 1387 ± 7% +21.8% 1689 ± 12% sched_debug.cfs_rq:/.runnable_avg.max 460.02 ± 2% +11.2% 511.65 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev 1726688 ± 2% +32.7% 2291718 ± 13% sched_debug.cfs_rq:/.spread0.stddev 14277 ± 28% -30.3% 9949 ± 4% sched_debug.cpu.avg_idle.min 1784858 -34.9% 1161189 ± 4% sched_debug.cpu.nr_switches.avg 6522465 ± 11% +32.0% 8612184 ± 16% sched_debug.cpu.nr_switches.max 2464 ± 11% -21.9% 1924 ± 2% sched_debug.cpu.nr_switches.min 7868 ± 7% +10.8% 8717 ± 5% numa-vmstat.node0.nr_kernel_stack 677.50 ± 50% +88.2% 1274 ± 29% numa-vmstat.node0.nr_page_table_pages 5910 ± 22% +56.5% 9251 ± 7% numa-vmstat.node0.nr_slab_reclaimable 4052 ± 9% -77.4% 917.50 ± 98% numa-vmstat.node1.nr_inactive_anon 6752 ± 2% -16.2% 5660 numa-vmstat.node1.nr_kernel_stack 2939 ± 17% -39.0% 1793 ± 20% numa-vmstat.node1.nr_mapped 434.25 ± 54% -90.8% 39.75 ± 53% numa-vmstat.node1.nr_page_table_pages 4467 ± 8% -75.3% 1102 ± 82% numa-vmstat.node1.nr_shmem 6931 ± 22% -49.3% 3517 ± 13% numa-vmstat.node1.nr_slab_reclaimable 17688 ± 8% -37.7% 11019 ± 7% numa-vmstat.node1.nr_slab_unreclaimable 4052 ± 9% -77.4% 917.50 ± 98% numa-vmstat.node1.nr_zone_inactive_anon 1.873e+08 ± 2% -51.3% 91255968 ± 63% numa-vmstat.node1.numa_hit 1.872e+08 ± 2% -51.3% 91208577 ± 63% numa-vmstat.node1.numa_local 25632 ± 87% -75.5% 6288 ±130% numa-vmstat.node2.nr_active_anon 25596 ± 87% -75.5% 6267 ±131% numa-vmstat.node2.nr_anon_pages 25632 ± 87% -75.5% 6288 ±130% numa-vmstat.node2.nr_zone_active_anon 23643 ± 22% +56.5% 37005 ± 7% numa-meminfo.node0.KReclaimable 7871 ± 7% +10.7% 8713 ± 5% numa-meminfo.node0.KernelStack 7269 ± 24% +24.6% 9060 ± 20% numa-meminfo.node0.Mapped 737950 ± 7% +23.2% 909436 numa-meminfo.node0.MemUsed 2710 ± 50% +88.1% 5098 ± 29% numa-meminfo.node0.PageTables 23643 ± 22% +56.5% 37005 ± 7% numa-meminfo.node0.SReclaimable 98061 ± 8% +26.1% 123682 ± 6% numa-meminfo.node0.Slab 16278 ± 10% -77.1% 3725 ± 96% numa-meminfo.node1.Inactive 16210 ± 9% -77.3% 3672 ± 98% numa-meminfo.node1.Inactive(anon) 27728 ± 22% -49.3% 14069 ± 13% numa-meminfo.node1.KReclaimable 6748 ± 2% -16.0% 5670 numa-meminfo.node1.KernelStack 11640 ± 18% -39.3% 7065 ± 21% numa-meminfo.node1.Mapped 593633 ± 3% -10.8% 529651 ± 5% numa-meminfo.node1.MemUsed 1736 ± 55% -90.4% 167.00 ± 51% numa-meminfo.node1.PageTables 27728 ± 22% -49.3% 14069 ± 13% numa-meminfo.node1.SReclaimable 70762 ± 8% -37.7% 44079 ± 7% numa-meminfo.node1.SUnreclaim 17873 ± 8% -75.3% 4415 ± 82% numa-meminfo.node1.Shmem 98490 ± 5% -41.0% 58148 ± 8% numa-meminfo.node1.Slab 102626 ± 86% -75.4% 25195 ±130% numa-meminfo.node2.Active 102494 ± 87% -75.4% 25163 ±130% numa-meminfo.node2.Active(anon) 71781 ±102% -84.3% 11236 ±162% numa-meminfo.node2.AnonHugePages 102350 ± 87% -75.5% 25082 ±131% numa-meminfo.node2.AnonPages 9418 ± 4% -25.7% 6997 ± 9% slabinfo.Acpi-State.active_objs 9418 ± 4% -25.7% 6997 ± 9% slabinfo.Acpi-State.num_objs 10164 -9.7% 9182 ± 3% slabinfo.cred_jar.active_objs 10164 -9.7% 9182 ± 3% slabinfo.cred_jar.num_objs 7263 ± 6% -30.8% 5027 ± 10% slabinfo.files_cache.active_objs 7263 ± 6% -30.8% 5027 ± 10% slabinfo.files_cache.num_objs 52454 ± 5% -16.9% 43590 ± 4% slabinfo.filp.active_objs 820.50 ± 5% -16.6% 684.50 ± 4% slabinfo.filp.active_slabs 52535 ± 5% -16.6% 43839 ± 4% slabinfo.filp.num_objs 820.50 ± 5% -16.6% 684.50 ± 4% slabinfo.filp.num_slabs 5691 ± 2% -25.7% 4226 ± 7% slabinfo.mm_struct.active_objs 5691 ± 2% -25.7% 4226 ± 7% slabinfo.mm_struct.num_objs 18729 -13.4% 16227 ± 6% slabinfo.pde_opener.active_objs 18729 -13.4% 16227 ± 6% slabinfo.pde_opener.num_objs 11365 -23.0% 8755 ± 12% slabinfo.pool_workqueue.active_objs 11366 -23.0% 8756 ± 12% slabinfo.pool_workqueue.num_objs 3857 ± 4% -18.9% 3126 ± 5% slabinfo.sighand_cache.active_objs 3859 ± 4% -18.8% 3135 ± 4% slabinfo.sighand_cache.num_objs 6370 ± 5% -22.4% 4942 ± 6% slabinfo.signal_cache.active_objs 6372 ± 5% -22.3% 4950 ± 6% slabinfo.signal_cache.num_objs 9525 ± 4% -24.2% 7218 ± 9% slabinfo.task_delay_info.active_objs 9525 ± 4% -24.2% 7218 ± 9% slabinfo.task_delay_info.num_objs 2720 ± 3% -12.5% 2379 slabinfo.task_struct.active_objs 909.25 ± 3% -11.9% 801.00 slabinfo.task_struct.active_slabs 2728 ± 3% -11.9% 2403 slabinfo.task_struct.num_objs 909.25 ± 3% -11.9% 801.00 slabinfo.task_struct.num_slabs 62.27 -6.7% 58.10 perf-stat.i.MPKI 1.506e+10 -18.1% 1.233e+10 ± 4% perf-stat.i.branch-instructions 1.93 -0.2 1.68 ± 4% perf-stat.i.branch-miss-rate% 2.872e+08 -28.3% 2.058e+08 ± 8% perf-stat.i.branch-misses 0.75 ± 11% -0.4 0.38 ± 30% perf-stat.i.cache-miss-rate% 34087325 ± 11% -61.2% 13229676 ± 41% perf-stat.i.cache-misses 4.561e+09 -25.3% 3.405e+09 ± 6% perf-stat.i.cache-references 2306970 -31.2% 1587675 ± 8% perf-stat.i.context-switches 2.02 +21.0% 2.44 ± 2% perf-stat.i.cpi 1.472e+11 -4.0% 1.413e+11 ± 2% perf-stat.i.cpu-cycles 778.21 ± 23% +1447.9% 12045 ± 71% perf-stat.i.cpu-migrations 4566 ± 11% +244.8% 15744 ± 36% perf-stat.i.cycles-between-cache-misses 23558623 -23.0% 18146040 ± 8% perf-stat.i.dTLB-load-misses 2.475e+10 -23.0% 1.905e+10 ± 5% perf-stat.i.dTLB-loads 0.06 +0.0 0.09 ± 9% perf-stat.i.dTLB-store-miss-rate% 1.639e+10 -29.7% 1.153e+10 ± 8% perf-stat.i.dTLB-stores 32.24 ± 6% -7.0 25.20 ± 6% perf-stat.i.iTLB-load-miss-rate% 17483265 ± 6% -24.0% 13290898 ± 7% perf-stat.i.iTLB-load-misses 36909400 ± 4% +7.2% 39555285 perf-stat.i.iTLB-loads 7.295e+10 -20.2% 5.82e+10 ± 4% perf-stat.i.instructions 0.50 -17.0% 0.41 ± 3% perf-stat.i.ipc 7892760 ± 14% -56.1% 3466546 ± 33% perf-stat.i.node-load-misses 82559 ± 5% -49.5% 41717 ± 30% perf-stat.i.node-loads 85.39 ± 2% -21.2 64.20 ± 12% perf-stat.i.node-store-miss-rate% 4935653 ± 12% -64.6% 1747970 ± 45% perf-stat.i.node-store-misses 62.52 -6.5% 58.45 ± 2% perf-stat.overall.MPKI 1.91 -0.2 1.67 ± 4% perf-stat.overall.branch-miss-rate% 0.75 ± 11% -0.4 0.38 ± 34% perf-stat.overall.cache-miss-rate% 2.02 +20.6% 2.43 ± 3% perf-stat.overall.cpi 4380 ± 12% +188.2% 12623 ± 39% perf-stat.overall.cycles-between-cache-misses 0.06 +0.0 0.09 ± 10% perf-stat.overall.dTLB-store-miss-rate% 32.16 ± 6% -7.0 25.13 ± 6% perf-stat.overall.iTLB-load-miss-rate% 0.50 -17.0% 0.41 ± 3% perf-stat.overall.ipc 85.86 ± 2% -19.6 66.27 ± 14% perf-stat.overall.node-store-miss-rate% 18033 +16.1% 20929 ± 3% perf-stat.overall.path-length 1.501e+10 -18.2% 1.228e+10 ± 4% perf-stat.ps.branch-instructions 2.862e+08 -28.4% 2.048e+08 ± 8% perf-stat.ps.branch-misses 33952691 ± 11% -61.3% 13129750 ± 41% perf-stat.ps.cache-misses 4.544e+09 -25.4% 3.39e+09 ± 6% perf-stat.ps.cache-references 2298503 -31.3% 1580015 ± 8% perf-stat.ps.context-switches 1.466e+11 -4.0% 1.408e+11 ± 2% perf-stat.ps.cpu-cycles 794.78 ± 24% +1419.0% 12072 ± 70% perf-stat.ps.cpu-migrations 23475449 -23.0% 18074421 ± 8% perf-stat.ps.dTLB-load-misses 2.466e+10 -23.1% 1.896e+10 ± 5% perf-stat.ps.dTLB-loads 1.633e+10 -29.8% 1.147e+10 ± 8% perf-stat.ps.dTLB-stores 17419953 ± 6% -24.0% 13232122 ± 7% perf-stat.ps.iTLB-load-misses 36786011 ± 4% +7.1% 39405794 ± 2% perf-stat.ps.iTLB-loads 7.268e+10 -20.3% 5.795e+10 ± 4% perf-stat.ps.instructions 7861951 ± 14% -56.2% 3443289 ± 33% perf-stat.ps.node-load-misses 82432 ± 5% -49.5% 41606 ± 30% perf-stat.ps.node-loads 4916204 ± 12% -64.7% 1734707 ± 45% perf-stat.ps.node-store-misses 2.195e+13 -20.3% 1.75e+13 ± 4% perf-stat.total.instructions 3037 ± 66% +68.9% 5130 ± 41% interrupts.CPU10.NMI:Non-maskable_interrupts 3037 ± 66% +68.9% 5130 ± 41% interrupts.CPU10.PMI:Performance_monitoring_interrupts 2687 ± 97% +145.3% 6592 ± 13% interrupts.CPU103.NMI:Non-maskable_interrupts 2687 ± 97% +145.3% 6592 ± 13% interrupts.CPU103.PMI:Performance_monitoring_interrupts 2179 ± 35% +216.1% 6887 ± 2% interrupts.CPU106.NMI:Non-maskable_interrupts 2179 ± 35% +216.1% 6887 ± 2% interrupts.CPU106.PMI:Performance_monitoring_interrupts 3006 ± 56% +126.1% 6798 ± 9% interrupts.CPU107.RES:Rescheduling_interrupts 1996 ± 27% +197.3% 5935 ± 31% interrupts.CPU108.NMI:Non-maskable_interrupts 1996 ± 27% +197.3% 5935 ± 31% interrupts.CPU108.PMI:Performance_monitoring_interrupts 2101 ± 92% +151.9% 5294 ± 49% interrupts.CPU109.RES:Rescheduling_interrupts 1580 ± 57% +330.4% 6801 ± 38% interrupts.CPU118.RES:Rescheduling_interrupts 1362 ± 14% +274.5% 5101 ± 39% interrupts.CPU119.RES:Rescheduling_interrupts 3530 ± 35% -83.7% 574.75 ±115% interrupts.CPU122.RES:Rescheduling_interrupts 1343 ± 43% -74.1% 348.00 ±147% interrupts.CPU137.RES:Rescheduling_interrupts 2221 ± 39% -60.7% 874.25 ±131% interrupts.CPU141.NMI:Non-maskable_interrupts 2221 ± 39% -60.7% 874.25 ±131% interrupts.CPU141.PMI:Performance_monitoring_interrupts 3578 ± 57% -80.8% 688.00 ±134% interrupts.CPU142.NMI:Non-maskable_interrupts 3578 ± 57% -80.8% 688.00 ±134% interrupts.CPU142.PMI:Performance_monitoring_interrupts 2935 ± 38% -73.7% 772.00 ±171% interrupts.CPU146.RES:Rescheduling_interrupts 3052 ± 84% +1035.4% 34656 ±135% interrupts.CPU15.RES:Rescheduling_interrupts 2687 ± 13% -59.0% 1102 ± 98% interrupts.CPU157.RES:Rescheduling_interrupts 4192 ± 70% -65.3% 1455 ±141% interrupts.CPU166.NMI:Non-maskable_interrupts 4192 ± 70% -65.3% 1455 ±141% interrupts.CPU166.PMI:Performance_monitoring_interrupts 3234 ± 75% -72.2% 898.25 ±139% interrupts.CPU167.NMI:Non-maskable_interrupts 3234 ± 75% -72.2% 898.25 ±139% interrupts.CPU167.PMI:Performance_monitoring_interrupts 3496 ± 55% -74.3% 897.00 ±133% interrupts.CPU172.NMI:Non-maskable_interrupts 3496 ± 55% -74.3% 897.00 ±133% interrupts.CPU172.PMI:Performance_monitoring_interrupts 3794 ± 51% -88.5% 435.00 ± 68% interrupts.CPU173.NMI:Non-maskable_interrupts 3794 ± 51% -88.5% 435.00 ± 68% interrupts.CPU173.PMI:Performance_monitoring_interrupts 4717 ± 56% -79.4% 972.75 ±136% interrupts.CPU174.NMI:Non-maskable_interrupts 4717 ± 56% -79.4% 972.75 ±136% interrupts.CPU174.PMI:Performance_monitoring_interrupts 4786 ± 13% -95.5% 213.25 ± 87% interrupts.CPU175.RES:Rescheduling_interrupts 3500 ± 64% -76.2% 832.50 ±107% interrupts.CPU184.NMI:Non-maskable_interrupts 3500 ± 64% -76.2% 832.50 ±107% interrupts.CPU184.PMI:Performance_monitoring_interrupts 2748 ± 4% -71.9% 772.25 ± 95% interrupts.CPU185.NMI:Non-maskable_interrupts 2748 ± 4% -71.9% 772.25 ± 95% interrupts.CPU185.PMI:Performance_monitoring_interrupts 3896 ± 50% -84.2% 616.50 ±115% interrupts.CPU186.NMI:Non-maskable_interrupts 3896 ± 50% -84.2% 616.50 ±115% interrupts.CPU186.PMI:Performance_monitoring_interrupts 6332 ± 22% -90.6% 597.50 ±110% interrupts.CPU187.NMI:Non-maskable_interrupts 6332 ± 22% -90.6% 597.50 ±110% interrupts.CPU187.PMI:Performance_monitoring_interrupts 2798 ± 62% -91.2% 246.25 ±163% interrupts.CPU188.RES:Rescheduling_interrupts 7445 ± 30% -89.6% 776.00 ±126% interrupts.CPU33.RES:Rescheduling_interrupts 2500 ± 31% -88.3% 293.75 ±158% interrupts.CPU35.RES:Rescheduling_interrupts 3157 ± 15% -81.9% 572.25 ±164% interrupts.CPU36.RES:Rescheduling_interrupts 4402 ± 61% -80.8% 844.50 ±168% interrupts.CPU39.RES:Rescheduling_interrupts 6686 ± 71% -91.7% 553.00 ±169% interrupts.CPU41.RES:Rescheduling_interrupts 5143 ± 37% -64.2% 1841 ±149% interrupts.CPU42.NMI:Non-maskable_interrupts 5143 ± 37% -64.2% 1841 ±149% interrupts.CPU42.PMI:Performance_monitoring_interrupts 5837 ± 32% -68.2% 1855 ±159% interrupts.CPU43.NMI:Non-maskable_interrupts 5837 ± 32% -68.2% 1855 ±159% interrupts.CPU43.PMI:Performance_monitoring_interrupts 3557 ± 47% -97.9% 75.25 ±126% interrupts.CPU44.RES:Rescheduling_interrupts 3370 ± 37% +130.1% 7754 ± 39% interrupts.CPU5.RES:Rescheduling_interrupts 4376 ± 37% -78.9% 924.75 ± 65% interrupts.CPU58.RES:Rescheduling_interrupts 2807 ± 64% +108.3% 5847 ± 36% interrupts.CPU6.RES:Rescheduling_interrupts 7502 ± 52% -83.6% 1226 ±112% interrupts.CPU61.RES:Rescheduling_interrupts 3688 ± 67% -88.3% 430.75 ±167% interrupts.CPU65.RES:Rescheduling_interrupts 2911 ± 16% -51.7% 1405 ± 62% interrupts.CPU75.NMI:Non-maskable_interrupts 2911 ± 16% -51.7% 1405 ± 62% interrupts.CPU75.PMI:Performance_monitoring_interrupts 2318 ± 21% -66.1% 786.25 ±117% interrupts.CPU76.NMI:Non-maskable_interrupts 2318 ± 21% -66.1% 786.25 ±117% interrupts.CPU76.PMI:Performance_monitoring_interrupts 2279 ± 27% -86.4% 310.75 ± 34% interrupts.CPU77.NMI:Non-maskable_interrupts 2279 ± 27% -86.4% 310.75 ± 34% interrupts.CPU77.PMI:Performance_monitoring_interrupts 5185 ± 35% -91.0% 465.00 ± 34% interrupts.CPU77.RES:Rescheduling_interrupts 4118 ± 34% -74.6% 1047 ± 87% interrupts.CPU79.RES:Rescheduling_interrupts 5165 ± 22% -63.7% 1875 ±107% interrupts.CPU86.RES:Rescheduling_interrupts 3514 ± 65% -70.4% 1041 ±106% interrupts.CPU88.NMI:Non-maskable_interrupts 3514 ± 65% -70.4% 1041 ±106% interrupts.CPU88.PMI:Performance_monitoring_interrupts 4784 ± 50% -78.3% 1037 ± 96% interrupts.CPU89.NMI:Non-maskable_interrupts 4784 ± 50% -78.3% 1037 ± 96% interrupts.CPU89.PMI:Performance_monitoring_interrupts 2389 ± 60% +193.3% 7009 ± 21% interrupts.CPU9.RES:Rescheduling_interrupts 3639 ± 19% -78.3% 789.75 ±128% interrupts.CPU90.NMI:Non-maskable_interrupts 3639 ± 19% -78.3% 789.75 ±128% interrupts.CPU90.PMI:Performance_monitoring_interrupts 2414 ± 45% -68.2% 768.50 ±126% interrupts.CPU91.NMI:Non-maskable_interrupts 2414 ± 45% -68.2% 768.50 ±126% interrupts.CPU91.PMI:Performance_monitoring_interrupts 3940 ± 28% -99.7% 11.00 ± 94% interrupts.CPU93.RES:Rescheduling_interrupts 4768 ± 51% -82.2% 849.75 ±136% interrupts.CPU94.NMI:Non-maskable_interrupts 4768 ± 51% -82.2% 849.75 ±136% interrupts.CPU94.PMI:Performance_monitoring_interrupts 3071 ± 51% -96.3% 114.25 ±129% interrupts.CPU94.RES:Rescheduling_interrupts 2672 ± 40% -73.5% 707.25 ±171% interrupts.CPU95.RES:Rescheduling_interrupts 644992 ± 20% +28.1% 826353 ± 9% interrupts.RES:Rescheduling_interrupts 141316 ± 2% -10.7% 126249 ± 8% softirqs.CPU10.RCU 38660 ± 4% -39.7% 23314 ± 40% softirqs.CPU10.SCHED 40648 -24.3% 30768 ± 18% softirqs.CPU101.SCHED 135335 ± 2% -10.6% 120927 ± 7% softirqs.CPU102.RCU 39708 ± 24% -48.1% 20612 ± 50% softirqs.CPU102.SCHED 357756 ± 92% +233.0% 1191209 ± 45% softirqs.CPU108.NET_RX 140033 ± 4% -11.5% 123936 ± 10% softirqs.CPU108.RCU 39682 ± 2% -40.4% 23651 ± 53% softirqs.CPU108.SCHED 1162992 ± 22% -62.5% 436694 ± 91% softirqs.CPU113.NET_RX 36929 ± 7% -37.5% 23090 ± 43% softirqs.CPU114.SCHED 39419 ± 15% -29.6% 27760 ± 23% softirqs.CPU116.SCHED 37471 ± 14% -20.4% 29821 ± 8% softirqs.CPU117.SCHED 38725 ± 4% -29.5% 27287 ± 24% softirqs.CPU122.SCHED 380987 ±112% -100.0% 4.75 ± 60% softirqs.CPU124.NET_RX 684082 ±108% -100.0% 6.50 ± 82% softirqs.CPU126.NET_RX 35779 ± 14% +17.7% 42122 ± 5% softirqs.CPU126.SCHED 136179 ± 4% -7.4% 126126 ± 5% softirqs.CPU136.RCU 136482 ± 3% -10.7% 121892 ± 3% softirqs.CPU137.RCU 37701 ± 13% -19.4% 30389 ± 19% softirqs.CPU14.SCHED 138037 ± 5% -7.9% 127161 ± 6% softirqs.CPU143.RCU 135813 ± 4% -11.1% 120675 ± 2% softirqs.CPU144.RCU 136752 ± 3% -9.5% 123693 ± 2% softirqs.CPU146.RCU 140989 ± 2% -7.0% 131118 ± 4% softirqs.CPU147.RCU 140477 ± 4% -7.0% 130636 ± 5% softirqs.CPU151.RCU 140183 ± 3% -9.2% 127229 ± 4% softirqs.CPU157.RCU 41010 ± 3% -41.3% 24083 ± 51% softirqs.CPU157.SCHED 34115 ± 11% +19.5% 40766 ± 12% softirqs.CPU159.SCHED 1225282 ± 94% -94.6% 66471 ±173% softirqs.CPU160.NET_RX 38414 ± 9% -38.0% 23813 ± 45% softirqs.CPU161.SCHED 137423 ± 4% -7.0% 127861 ± 3% softirqs.CPU162.RCU 30810 ± 31% +27.3% 39228 ± 6% softirqs.CPU164.SCHED 138560 ± 3% -13.0% 120523 ± 5% softirqs.CPU165.RCU 42022 ± 5% -24.5% 31709 ± 38% softirqs.CPU165.SCHED 973700 ± 95% -96.8% 30927 ±173% softirqs.CPU172.NET_RX 977062 ± 77% -100.0% 2.75 ± 69% softirqs.CPU173.NET_RX 547399 ±162% -100.0% 2.75 ± 86% softirqs.CPU179.NET_RX 64145 ±114% -100.0% 7.00 ± 66% softirqs.CPU183.NET_RX 39496 +10.4% 43614 ± 6% softirqs.CPU183.SCHED 26478 ± 28% +53.6% 40670 softirqs.CPU186.SCHED 1601031 ± 85% -100.0% 477.00 ±173% softirqs.CPU187.NET_RX 133259 ± 2% -6.6% 124422 ± 4% softirqs.CPU189.RCU 137106 ± 2% -8.4% 125547 ± 2% softirqs.CPU190.RCU 102176 ± 68% -72.1% 28546 ±173% softirqs.CPU191.NET_RX 1500188 ± 32% -71.1% 433553 ± 99% softirqs.CPU23.NET_RX 1456825 ± 48% -78.7% 310211 ±145% softirqs.CPU27.NET_RX 30211 ± 21% +32.5% 40018 ± 16% softirqs.CPU27.SCHED 1249314 ± 45% -83.4% 207506 ±173% softirqs.CPU28.NET_RX 32688 ± 12% +22.9% 40167 ± 10% softirqs.CPU28.SCHED 614013 ± 33% -91.8% 50330 ±172% softirqs.CPU31.NET_RX 856129 ± 54% -92.1% 67308 ±172% softirqs.CPU32.NET_RX 826453 ± 88% -92.6% 61272 ±153% softirqs.CPU33.NET_RX 572160 ± 58% -92.0% 45490 ±172% softirqs.CPU34.NET_RX 1198932 ± 26% -80.2% 237504 ±173% softirqs.CPU35.NET_RX 764186 ± 42% -98.9% 8411 ±170% softirqs.CPU36.NET_RX 1957332 ± 39% -97.6% 47130 ±172% softirqs.CPU38.NET_RX 25563 ± 24% +51.2% 38644 ± 12% softirqs.CPU38.SCHED 1904800 ± 68% -69.5% 581766 ±173% softirqs.CPU39.NET_RX 136554 -10.5% 122273 ± 3% softirqs.CPU41.RCU 1937742 ± 65% -83.7% 316430 ±173% softirqs.CPU42.NET_RX 1965972 ± 41% -84.5% 305614 ±173% softirqs.CPU43.NET_RX 138112 ± 5% -8.5% 126429 ± 4% softirqs.CPU44.RCU 938187 ± 79% -88.1% 111510 ±172% softirqs.CPU45.NET_RX 1859081 ± 63% -100.0% 142.50 ± 6% softirqs.CPU47.NET_RX 1418766 ± 55% -96.3% 51993 ±172% softirqs.CPU56.NET_RX 31415 ± 16% +33.4% 41922 ± 8% softirqs.CPU56.SCHED 1608741 ± 92% -93.1% 110943 ±172% softirqs.CPU57.NET_RX 1185884 ± 49% -52.3% 565791 ±113% softirqs.CPU58.NET_RX 2280687 ± 64% -86.8% 301429 ±153% softirqs.CPU60.NET_RX 989103 ± 66% -84.0% 158376 ±173% softirqs.CPU63.NET_RX 1357266 ± 62% -87.3% 172102 ±173% softirqs.CPU64.NET_RX 1353050 ± 14% -77.5% 305024 ±170% softirqs.CPU67.NET_RX 1061562 ± 62% -70.1% 317101 ±173% softirqs.CPU68.NET_RX 135927 ± 4% -9.8% 122670 ± 7% softirqs.CPU70.RCU 1219305 ± 63% -66.8% 404442 ±163% softirqs.CPU71.NET_RX 138945 ± 3% -12.6% 121504 ± 7% softirqs.CPU71.RCU 1342914 ± 22% -57.5% 570854 ±101% softirqs.CPU75.NET_RX 1370885 ± 17% -60.3% 543996 ±107% softirqs.CPU76.NET_RX 38700 ± 6% -20.6% 30718 ± 21% softirqs.CPU77.SCHED 37200 ± 11% -32.2% 25215 ± 53% softirqs.CPU8.SCHED 2042132 ± 26% -76.3% 484633 ±140% softirqs.CPU84.NET_RX 1162850 ± 36% -69.4% 355800 ±116% softirqs.CPU85.NET_RX 1454966 ± 30% -92.4% 109880 ±173% softirqs.CPU87.NET_RX 29191 ± 28% +48.1% 43243 ± 10% softirqs.CPU87.SCHED 1982705 ± 30% -90.8% 181832 ± 87% softirqs.CPU88.NET_RX 26016 ± 21% +60.9% 41872 ± 10% softirqs.CPU88.SCHED 2230453 ± 9% -92.5% 166989 ±173% softirqs.CPU89.NET_RX 23850 ± 15% +65.7% 39517 ± 6% softirqs.CPU89.SCHED 1324547 ± 34% -100.0% 3.25 ± 70% softirqs.CPU90.NET_RX 30624 ± 11% +34.9% 41296 ± 10% softirqs.CPU90.SCHED 392004 ±131% -100.0% 3.75 ± 87% softirqs.CPU91.NET_RX 716600 ± 98% -97.2% 20120 ±173% softirqs.CPU92.NET_RX 1.739e+08 -31.1% 1.198e+08 ± 8% softirqs.NET_RX 35.60 -2.7 32.88 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 38.93 -2.7 36.26 ± 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 38.93 -2.7 36.26 ± 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 38.91 -2.7 36.25 ± 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 39.06 -2.6 36.42 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64 36.62 -2.5 34.09 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 36.69 -2.5 34.17 ± 4% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 7.72 ± 2% -1.7 5.99 ± 4% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm 7.62 ± 2% -1.7 5.91 ± 4% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter 7.53 ± 2% -1.7 5.83 ± 4% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush 7.26 ± 2% -1.7 5.60 ± 4% perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.sctp_packet_transmit 7.24 ± 2% -1.7 5.58 ± 4% perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit 7.16 ± 2% -1.6 5.52 ± 4% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2 7.18 ± 2% -1.6 5.54 ± 4% perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output 7.06 ± 2% -1.6 5.45 ± 4% perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 6.98 ± 2% -1.6 5.36 ± 4% perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq 6.87 ± 2% -1.6 5.27 ± 4% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack 6.71 ± 2% -1.6 5.14 ± 4% perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start 6.57 ± 2% -1.6 5.01 ± 4% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 6.51 ± 2% -1.6 4.96 ± 4% perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv 6.62 ± 2% -1.6 5.07 ± 4% perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 6.58 ± 2% -1.5 5.03 ± 4% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 10.09 ± 2% -1.4 8.72 ± 3% perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg 8.06 -1.3 6.75 ± 4% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv 5.28 ± 2% -1.2 4.05 ± 4% perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 5.26 -1.2 4.09 ± 5% perf-profile.calltrace.cycles-pp.memcpy_erms.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm 4.95 ± 2% -1.2 3.78 ± 4% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 8.95 ± 2% -1.1 7.82 ± 3% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc 4.45 ± 2% -1.1 3.36 ± 4% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu 4.45 -1.0 3.45 ± 5% perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg 4.31 -1.0 3.33 ± 5% perf-profile.calltrace.cycles-pp._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg 4.20 -1.0 3.23 ± 5% perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc 4.17 -0.9 3.22 ± 5% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.sctp_user_addto_chunk.sctp_datamsg_from_user 8.12 -0.9 7.23 ± 4% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv 7.75 -0.9 6.87 ± 3% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND 3.15 ± 2% -0.8 2.36 ± 5% perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv 7.22 ± 2% -0.7 6.56 ± 3% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg 7.06 ± 2% -0.6 6.44 ± 3% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg 4.18 ± 3% -0.6 3.59 ± 7% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg 4.17 ± 3% -0.6 3.57 ± 7% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg.____sys_recvmsg 1.42 -0.6 0.86 ± 57% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event 1.39 -0.6 0.84 ± 57% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sctp_data_ready 3.76 ± 3% -0.5 3.21 ± 7% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg.inet_recvmsg 2.45 -0.5 1.90 ± 3% perf-profile.calltrace.cycles-pp.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg 3.66 ± 3% -0.5 3.12 ± 7% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.sctp_recvmsg 3.63 ± 3% -0.5 3.10 ± 7% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 1.92 -0.5 1.44 ± 5% perf-profile.calltrace.cycles-pp.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv 1.75 -0.4 1.33 ± 5% perf-profile.calltrace.cycles-pp.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm 1.08 -0.4 0.72 ± 5% perf-profile.calltrace.cycles-pp.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg 1.56 -0.4 1.21 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data.sctp_cmd_interpreter 1.46 -0.3 1.14 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sctp_data_ready.sctp_ulpq_tail_event.sctp_ulpq_tail_data 0.96 -0.3 0.64 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg 0.72 ± 2% -0.3 0.40 ± 57% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv 0.93 -0.3 0.62 ± 6% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.sctp_skb_recv_datagram.sctp_recvmsg 0.65 ± 2% -0.2 0.41 ± 57% perf-profile.calltrace.cycles-pp.copy_msghdr_from_user.sendmsg_copy_msghdr.___sys_sendmsg.__sys_sendmsg.do_syscall_64 0.65 ± 2% -0.2 0.41 ± 57% perf-profile.calltrace.cycles-pp.sendmsg_copy_msghdr.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.92 ± 2% -0.2 0.70 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 0.87 ± 2% -0.2 0.67 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.76 ± 4% -0.2 0.57 ± 4% perf-profile.calltrace.cycles-pp.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg 0.87 ± 3% -0.1 0.74 ± 6% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.85 ± 3% -0.1 0.72 ± 7% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.00 ± 3% -0.1 0.89 ± 3% perf-profile.calltrace.cycles-pp.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.____sys_recvmsg 0.98 ± 3% -0.1 0.88 ± 3% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg 0.96 ± 3% -0.1 0.86 ± 3% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg 2.75 ± 5% +0.9 3.69 ± 13% perf-profile.calltrace.cycles-pp.kfree_skb.sctp_recvmsg.inet_recvmsg.____sys_recvmsg.___sys_recvmsg 2.07 ± 5% +1.1 3.14 ± 14% perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.kfree_skb.sctp_recvmsg.inet_recvmsg 2.21 ± 6% +1.1 3.27 ± 14% perf-profile.calltrace.cycles-pp.__free_pages_ok.kfree_skb.sctp_recvmsg.inet_recvmsg.____sys_recvmsg 1.90 ± 6% +1.1 3.02 ± 14% perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.kfree_skb.sctp_recvmsg 1.75 ± 6% +1.2 2.92 ± 15% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.kfree_skb 1.99 +1.3 3.29 ± 3% perf-profile.calltrace.cycles-pp.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm 1.85 +1.3 3.19 ± 3% perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter 1.82 ± 2% +1.4 3.17 ± 3% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit.sctp_outq_flush 1.81 +1.4 3.17 ± 3% perf-profile.calltrace.cycles-pp.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.sctp_packet_transmit 23.40 +2.0 25.38 ± 2% perf-profile.calltrace.cycles-pp.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg 58.47 +3.2 61.64 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 58.27 +3.2 61.47 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 16.29 +3.2 19.53 ± 2% perf-profile.calltrace.cycles-pp.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg 16.19 +3.3 19.45 ± 2% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg.____sys_sendmsg 16.10 +3.3 19.40 ± 2% perf-profile.calltrace.cycles-pp.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg.sock_sendmsg 12.86 +3.4 16.27 ± 2% perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg.____sys_sendmsg 14.14 +3.7 17.89 ± 2% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock 13.93 +3.8 17.70 ± 2% perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock.sctp_sendmsg 13.93 +3.8 17.70 ± 2% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv.__release_sock.release_sock 7.65 ± 2% +4.2 11.90 perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_backlog_rcv 6.70 ± 2% +4.5 11.22 perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv 6.50 ± 2% +4.6 11.08 perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm 42.95 +4.6 47.55 ± 2% perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.73 +4.6 12.34 ± 2% perf-profile.calltrace.cycles-pp.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.sock_sendmsg 42.69 +4.7 47.35 ± 2% perf-profile.calltrace.cycles-pp.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.90 ± 2% +4.7 10.56 ± 2% perf-profile.calltrace.cycles-pp.__free_pages_ok.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter 5.67 ± 2% +4.7 10.33 ± 2% perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.consume_skb.sctp_chunk_put.sctp_outq_sack 7.36 +4.7 12.04 ± 2% perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg 41.99 +4.8 46.77 ± 2% perf-profile.calltrace.cycles-pp.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.82 +4.8 11.62 ± 2% perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc 5.18 ± 2% +4.8 9.99 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.consume_skb.sctp_chunk_put 41.20 +4.9 46.11 ± 2% perf-profile.calltrace.cycles-pp.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64 4.89 ± 2% +4.9 9.80 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.consume_skb 40.83 +5.0 45.80 ± 2% perf-profile.calltrace.cycles-pp.sctp_sendmsg.sock_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg 5.99 +5.1 11.06 ± 2% perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user 5.96 +5.1 11.03 ± 2% perf-profile.calltrace.cycles-pp.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb._sctp_make_chunk 5.97 +5.1 11.05 ± 2% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty 7.63 +6.5 14.09 ± 3% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb 7.33 +6.5 13.86 ± 3% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller.__kmalloc_reserve 5.58 ± 2% +6.9 12.47 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node.__kmalloc_node_track_caller 5.26 ± 2% +7.0 12.24 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.kmalloc_large_node 35.73 -2.7 33.02 ± 5% perf-profile.children.cycles-pp.intel_idle 38.93 -2.7 36.26 ± 4% perf-profile.children.cycles-pp.start_secondary 39.06 -2.6 36.42 ± 4% perf-profile.children.cycles-pp.do_idle 39.06 -2.6 36.42 ± 4% perf-profile.children.cycles-pp.secondary_startup_64 39.06 -2.6 36.42 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 36.82 -2.5 34.32 ± 4% perf-profile.children.cycles-pp.cpuidle_enter 36.81 -2.5 34.32 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state 17.84 ± 2% -2.2 15.63 ± 4% perf-profile.children.cycles-pp.sctp_outq_flush 7.99 ± 2% -1.8 6.19 ± 4% perf-profile.children.cycles-pp.__ip_queue_xmit 15.83 -1.8 14.03 ± 4% perf-profile.children.cycles-pp.sctp_packet_transmit 7.83 ± 2% -1.8 6.07 ± 4% perf-profile.children.cycles-pp.ip_output 7.69 ± 2% -1.7 5.95 ± 4% perf-profile.children.cycles-pp.ip_finish_output2 7.33 ± 2% -1.7 5.66 ± 4% perf-profile.children.cycles-pp.__local_bh_enable_ip 7.25 ± 2% -1.7 5.59 ± 4% perf-profile.children.cycles-pp.do_softirq 7.19 ± 2% -1.6 5.54 ± 4% perf-profile.children.cycles-pp.do_softirq_own_stack 7.07 ± 2% -1.6 5.45 ± 4% perf-profile.children.cycles-pp.net_rx_action 6.98 ± 2% -1.6 5.37 ± 4% perf-profile.children.cycles-pp.process_backlog 6.87 ± 2% -1.6 5.28 ± 4% perf-profile.children.cycles-pp.__netif_receive_skb_one_core 7.43 ± 2% -1.6 5.85 ± 5% perf-profile.children.cycles-pp.__softirqentry_text_start 6.71 ± 2% -1.6 5.15 ± 4% perf-profile.children.cycles-pp.ip_rcv 6.63 ± 2% -1.6 5.07 ± 4% perf-profile.children.cycles-pp.ip_local_deliver 6.59 ± 2% -1.6 5.04 ± 4% perf-profile.children.cycles-pp.ip_local_deliver_finish 6.52 ± 2% -1.6 4.97 ± 4% perf-profile.children.cycles-pp.sctp_rcv 6.57 ± 2% -1.5 5.02 ± 4% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 8.00 -1.5 6.48 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 10.09 ± 2% -1.4 8.72 ± 3% perf-profile.children.cycles-pp.sctp_primitive_SEND 5.45 -1.2 4.23 ± 5% perf-profile.children.cycles-pp.memcpy_erms 4.45 -1.0 3.46 ± 5% perf-profile.children.cycles-pp.sctp_user_addto_chunk 4.32 -1.0 3.33 ± 5% perf-profile.children.cycles-pp._copy_from_iter_full 4.21 -1.0 3.24 ± 5% perf-profile.children.cycles-pp.copyin 3.16 ± 2% -0.8 2.37 ± 5% perf-profile.children.cycles-pp.sctp_ulpq_tail_data 4.19 ± 3% -0.6 3.59 ± 7% perf-profile.children.cycles-pp.skb_copy_datagram_iter 4.17 ± 3% -0.6 3.58 ± 7% perf-profile.children.cycles-pp.__skb_datagram_iter 3.77 ± 3% -0.5 3.22 ± 7% perf-profile.children.cycles-pp._copy_to_iter 2.45 -0.5 1.91 ± 3% perf-profile.children.cycles-pp.sctp_skb_recv_datagram 3.66 ± 3% -0.5 3.12 ± 7% perf-profile.children.cycles-pp.copyout 1.93 -0.5 1.46 ± 5% perf-profile.children.cycles-pp.sctp_ulpq_tail_event 1.75 -0.4 1.33 ± 5% perf-profile.children.cycles-pp.sctp_data_ready 1.33 -0.4 0.93 ± 4% perf-profile.children.cycles-pp.lock_sock_nested 1.22 -0.4 0.85 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_bh 1.57 -0.4 1.21 ± 6% perf-profile.children.cycles-pp.__wake_up_common_lock 1.46 -0.3 1.14 ± 6% perf-profile.children.cycles-pp.__wake_up_common 1.42 -0.3 1.10 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function 1.39 -0.3 1.08 ± 6% perf-profile.children.cycles-pp.try_to_wake_up 1.07 ± 6% -0.3 0.77 ± 5% perf-profile.children.cycles-pp.sctp_ulpevent_make_rcvmsg 0.83 ± 2% -0.2 0.58 ± 5% perf-profile.children.cycles-pp.__list_del_entry_valid 1.84 ± 3% -0.2 1.61 ± 4% perf-profile.children.cycles-pp.__sched_text_start 0.56 ± 3% -0.2 0.33 ± 7% perf-profile.children.cycles-pp.sctp_datamsg_put 1.00 -0.2 0.77 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.53 ± 3% -0.2 0.31 ± 7% perf-profile.children.cycles-pp.sctp_chunk_free 0.74 ± 8% -0.2 0.52 ± 4% perf-profile.children.cycles-pp.sctp_outq_select_transport 0.92 ± 2% -0.2 0.70 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.67 ± 9% -0.2 0.46 ± 2% perf-profile.children.cycles-pp.sctp_packet_config 0.77 ± 4% -0.2 0.57 ± 4% perf-profile.children.cycles-pp.sctp_ulpevent_free 0.73 -0.2 0.54 ± 3% perf-profile.children.cycles-pp.kmem_cache_free 0.85 -0.2 0.67 perf-profile.children.cycles-pp.skb_release_all 0.84 -0.2 0.66 perf-profile.children.cycles-pp.skb_release_head_state 1.10 -0.2 0.92 ± 4% perf-profile.children.cycles-pp.copy_msghdr_from_user 0.54 ± 8% -0.2 0.37 ± 5% perf-profile.children.cycles-pp.__skb_clone 0.89 -0.1 0.74 ± 3% perf-profile.children.cycles-pp._copy_from_user 0.55 ± 2% -0.1 0.41 ± 6% perf-profile.children.cycles-pp.enqueue_entity 0.65 ± 2% -0.1 0.51 ± 7% perf-profile.children.cycles-pp.ttwu_do_activate 0.64 -0.1 0.50 ± 7% perf-profile.children.cycles-pp.enqueue_task_fair 0.87 ± 3% -0.1 0.74 ± 6% perf-profile.children.cycles-pp.schedule_idle 0.64 -0.1 0.51 ± 8% perf-profile.children.cycles-pp.activate_task 0.58 -0.1 0.45 ± 5% perf-profile.children.cycles-pp.sctp_chunkify 0.40 -0.1 0.28 ± 5% perf-profile.children.cycles-pp.__list_add_valid 0.66 ± 2% -0.1 0.54 ± 5% perf-profile.children.cycles-pp.sendmsg_copy_msghdr 0.41 ± 2% -0.1 0.29 ± 5% perf-profile.children.cycles-pp.__slab_free 0.31 ± 14% -0.1 0.20 ± 4% perf-profile.children.cycles-pp.ipv4_dst_check 1.00 ± 3% -0.1 0.89 ± 3% perf-profile.children.cycles-pp.schedule_timeout 0.41 ± 7% -0.1 0.30 ± 5% perf-profile.children.cycles-pp.sctp_hash_cmp 0.99 ± 3% -0.1 0.88 ± 3% perf-profile.children.cycles-pp.schedule 0.28 ± 15% -0.1 0.17 ± 4% perf-profile.children.cycles-pp.__copy_skb_header 0.39 ± 2% -0.1 0.28 ± 4% perf-profile.children.cycles-pp.sctp_addrs_lookup_transport 0.42 ± 2% -0.1 0.32 ± 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.22 ± 12% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.sctp_transport_hold 0.41 ± 8% -0.1 0.33 ± 6% perf-profile.children.cycles-pp.sctp_endpoint_lookup_assoc 0.39 ± 7% -0.1 0.30 ± 8% perf-profile.children.cycles-pp.sctp_epaddr_lookup_transport 0.45 -0.1 0.36 ± 4% perf-profile.children.cycles-pp.__dev_queue_xmit 0.48 -0.1 0.40 ± 7% perf-profile.children.cycles-pp.kmem_cache_alloc 0.41 -0.1 0.34 ± 5% perf-profile.children.cycles-pp.___might_sleep 0.32 ± 2% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.sctp_addto_chunk 0.32 ± 4% -0.1 0.24 ± 4% perf-profile.children.cycles-pp.sctp_eat_data 0.47 ± 2% -0.1 0.40 ± 4% perf-profile.children.cycles-pp.import_iovec 0.36 -0.1 0.29 ± 4% perf-profile.children.cycles-pp.sockfd_lookup_light 0.36 ± 2% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.32 -0.1 0.25 ± 4% perf-profile.children.cycles-pp.security_socket_sendmsg 0.48 -0.1 0.41 ± 4% perf-profile.children.cycles-pp.update_load_avg 0.43 -0.1 0.36 ± 3% perf-profile.children.cycles-pp.sctp_sf_eat_data_6_2 0.23 ± 4% -0.1 0.16 ± 6% perf-profile.children.cycles-pp.skb_release_data 0.35 ± 3% -0.1 0.28 ± 4% perf-profile.children.cycles-pp.sctp_wfree 0.41 ± 3% -0.1 0.35 ± 5% perf-profile.children.cycles-pp.rw_copy_check_uvector 0.23 ± 3% -0.1 0.17 ± 5% perf-profile.children.cycles-pp.memset_erms 0.25 ± 2% -0.1 0.19 ± 9% perf-profile.children.cycles-pp.sctp_sock_rfree 0.21 ± 3% -0.1 0.15 ± 7% perf-profile.children.cycles-pp.__switch_to 0.15 ± 9% -0.1 0.09 ± 8% perf-profile.children.cycles-pp.ip_local_out 0.22 ± 5% -0.1 0.16 ± 6% perf-profile.children.cycles-pp.skb_set_owner_w 0.46 ± 5% -0.1 0.40 ± 8% perf-profile.children.cycles-pp.recvmsg_copy_msghdr 0.52 ± 3% -0.1 0.46 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair 0.29 ± 2% -0.1 0.23 ± 2% perf-profile.children.cycles-pp.__might_sleep 0.36 ± 5% -0.1 0.31 ± 3% perf-profile.children.cycles-pp.aa_sk_perm 0.29 -0.1 0.24 ± 5% perf-profile.children.cycles-pp.__fget_light 0.16 ± 2% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.36 ± 3% -0.1 0.30 ± 3% perf-profile.children.cycles-pp.sctp_sched_fcfs_dequeue 0.15 ± 5% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.rcu_idle_exit 0.16 ± 2% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.60 ± 2% -0.1 0.55 ± 2% perf-profile.children.cycles-pp.__check_object_size 0.30 -0.1 0.24 ± 3% perf-profile.children.cycles-pp.dev_hard_start_xmit 0.26 ± 3% -0.1 0.21 ± 5% perf-profile.children.cycles-pp.sctp_make_sack 0.27 ± 4% -0.1 0.22 perf-profile.children.cycles-pp.loopback_xmit 0.16 ± 9% -0.1 0.11 ± 7% perf-profile.children.cycles-pp.ipv4_mtu 0.13 ± 6% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.rcu_eqs_exit 0.22 -0.0 0.17 ± 6% perf-profile.children.cycles-pp.sctp_inq_pop 0.15 ± 3% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.check_preempt_curr 0.12 ± 5% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.ip_send_check 0.17 ± 4% -0.0 0.12 ± 8% perf-profile.children.cycles-pp.sctp_outq_flush_ctrl 0.11 ± 3% -0.0 0.07 ± 17% perf-profile.children.cycles-pp.resched_curr 0.22 ± 3% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.sock_kmalloc 0.14 ± 5% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.sock_wfree 0.14 ± 8% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.__ip_local_out 0.25 ± 3% -0.0 0.20 ± 7% perf-profile.children.cycles-pp.sctp_check_transmitted 0.15 ± 3% -0.0 0.11 ± 3% perf-profile.children.cycles-pp.sctp_transport_put 0.22 ± 3% -0.0 0.18 ± 9% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.44 ± 5% -0.0 0.40 ± 3% perf-profile.children.cycles-pp.dequeue_entity 0.18 ± 6% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.__update_load_avg_se 0.12 ± 8% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.sctp_chunk_hold 0.11 ± 15% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.__ip_finish_output 0.07 ± 7% -0.0 0.03 ±100% perf-profile.children.cycles-pp.sctp_hash_key 0.14 ± 11% -0.0 0.10 ± 11% perf-profile.children.cycles-pp.dst_release 0.23 ± 2% -0.0 0.20 ± 4% perf-profile.children.cycles-pp.move_addr_to_kernel 0.19 ± 5% -0.0 0.15 ± 4% perf-profile.children.cycles-pp.sctp_ulpevent_receive_data 0.24 ± 2% -0.0 0.21 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair 0.17 ± 2% -0.0 0.14 ± 6% perf-profile.children.cycles-pp.sctp_v4_addr_valid 0.08 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit 0.06 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.sctp_bind_addr_match 0.17 ± 5% -0.0 0.13 perf-profile.children.cycles-pp.sctp_association_put 0.16 ± 2% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.sock_kfree_s 0.19 -0.0 0.16 ± 7% perf-profile.children.cycles-pp.update_rq_clock 0.17 -0.0 0.14 ± 6% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.18 ± 2% -0.0 0.15 ± 8% perf-profile.children.cycles-pp.__zone_watermark_ok 0.13 ± 9% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.__netif_receive_skb_core 0.36 -0.0 0.33 ± 5% perf-profile.children.cycles-pp.kfree 0.20 ± 4% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.sctp_sched_dequeue_common 0.11 ± 4% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.sctp_association_hold 0.07 ± 5% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.sctp_v4_from_skb 0.07 ± 6% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.pick_next_entity 0.07 ± 7% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.validate_xmit_skb 0.30 -0.0 0.28 ± 5% perf-profile.children.cycles-pp.sctp_packet_transmit_chunk 0.20 ± 4% -0.0 0.18 ± 6% perf-profile.children.cycles-pp.update_curr 0.14 -0.0 0.12 ± 7% perf-profile.children.cycles-pp.sctp_sockaddr_af 0.14 ± 3% -0.0 0.11 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.10 ± 7% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.del_timer 0.09 ± 9% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.sctp_transport_reset_t3_rtx 0.12 ± 8% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.09 ± 4% -0.0 0.07 ± 14% perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase 0.13 ± 6% -0.0 0.11 ± 10% perf-profile.children.cycles-pp.skb_clone 0.08 ± 10% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.sctp_v4_ecn_capable 0.12 ± 5% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.mod_timer 0.14 ± 6% -0.0 0.12 ± 5% perf-profile.children.cycles-pp.__kmalloc 0.08 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.account_entity_dequeue 0.08 ± 5% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.nr_iowait_cpu 0.09 ± 4% -0.0 0.07 perf-profile.children.cycles-pp.lock_timer_base 0.14 ± 6% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.mod_node_page_state 0.10 ± 4% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.sctp_validate_data 0.08 ± 5% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.enqueue_to_backlog 0.11 ± 3% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.__switch_to_asm 0.09 ± 4% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.netif_rx 0.09 -0.0 0.08 ± 6% perf-profile.children.cycles-pp.netif_rx_internal 0.08 -0.0 0.07 ± 6% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.07 -0.0 0.06 perf-profile.children.cycles-pp.__x64_sys_sendmsg 0.06 -0.0 0.05 perf-profile.children.cycles-pp.__free_pages 0.07 ± 10% +0.0 0.09 ± 7% perf-profile.children.cycles-pp.___perf_sw_event 0.05 ± 60% +0.0 0.09 ± 15% perf-profile.children.cycles-pp.irq_enter 0.01 ±173% +0.1 0.07 ± 17% perf-profile.children.cycles-pp.tick_irq_enter 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.load_balance 2.75 ± 5% +0.9 3.69 ± 13% perf-profile.children.cycles-pp.kfree_skb 25.85 +2.0 27.81 ± 2% perf-profile.children.cycles-pp.sctp_cmd_interpreter 23.41 +2.0 25.38 ± 2% perf-profile.children.cycles-pp.sctp_sendmsg_to_asoc 25.96 +2.0 27.94 ± 2% perf-profile.children.cycles-pp.sctp_do_sm 19.22 +2.5 21.76 ± 2% perf-profile.children.cycles-pp.sctp_assoc_bh_rcv 58.51 +3.2 61.68 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 58.31 +3.2 61.52 ± 2% perf-profile.children.cycles-pp.do_syscall_64 16.37 +3.2 19.60 ± 2% perf-profile.children.cycles-pp.release_sock 16.19 +3.3 19.46 ± 2% perf-profile.children.cycles-pp.__release_sock 16.11 +3.3 19.40 ± 2% perf-profile.children.cycles-pp.sctp_backlog_rcv 12.87 +3.4 16.28 ± 2% perf-profile.children.cycles-pp.sctp_datamsg_from_user 7.65 ± 2% +4.3 11.90 perf-profile.children.cycles-pp.sctp_outq_sack 7.37 ± 2% +4.4 11.72 perf-profile.children.cycles-pp.sctp_chunk_put 6.81 ± 2% +4.5 11.31 perf-profile.children.cycles-pp.consume_skb 42.95 +4.6 47.55 ± 2% perf-profile.children.cycles-pp.__sys_sendmsg 7.74 +4.6 12.34 ± 2% perf-profile.children.cycles-pp.sctp_make_datafrag_empty 42.70 +4.6 47.35 ± 2% perf-profile.children.cycles-pp.___sys_sendmsg 7.52 +4.7 12.17 ± 2% perf-profile.children.cycles-pp._sctp_make_chunk 42.00 +4.8 46.78 ± 2% perf-profile.children.cycles-pp.____sys_sendmsg 41.21 +4.9 46.12 ± 2% perf-profile.children.cycles-pp.sock_sendmsg 40.84 +5.0 45.82 ± 2% perf-profile.children.cycles-pp.sctp_sendmsg 7.75 ± 2% +5.7 13.48 ± 4% perf-profile.children.cycles-pp.free_one_page 8.12 ± 2% +5.7 13.86 ± 4% perf-profile.children.cycles-pp.__free_pages_ok 7.47 ± 2% +5.8 13.30 ± 4% perf-profile.children.cycles-pp._raw_spin_lock 8.98 +6.1 15.05 ± 2% perf-profile.children.cycles-pp.__alloc_skb 7.92 +6.4 14.31 ± 3% perf-profile.children.cycles-pp.__kmalloc_reserve 7.87 +6.4 14.29 ± 2% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 7.77 +6.4 14.21 ± 2% perf-profile.children.cycles-pp.kmalloc_large_node 7.64 +6.5 14.10 ± 3% perf-profile.children.cycles-pp.__alloc_pages_nodemask 7.35 +6.5 13.88 ± 3% perf-profile.children.cycles-pp.get_page_from_freelist 6.01 ± 2% +6.8 12.84 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 12.88 ± 2% +12.8 25.66 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 35.73 -2.7 33.02 ± 5% perf-profile.self.cycles-pp.intel_idle 7.95 -1.5 6.43 ± 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 5.40 -1.2 4.20 ± 5% perf-profile.self.cycles-pp.memcpy_erms 2.65 -0.6 2.05 perf-profile.self.cycles-pp.do_syscall_64 0.78 -0.3 0.50 ± 4% perf-profile.self.cycles-pp.__alloc_skb 0.81 -0.3 0.56 perf-profile.self.cycles-pp._raw_spin_lock 0.81 ± 2% -0.2 0.56 ± 5% perf-profile.self.cycles-pp.__list_del_entry_valid 1.00 -0.2 0.77 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.92 ± 2% -0.2 0.70 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.51 ± 4% -0.2 0.29 ± 7% perf-profile.self.cycles-pp.sctp_datamsg_put 0.57 -0.2 0.38 ± 2% perf-profile.self.cycles-pp.sctp_datamsg_from_user 0.70 ± 3% -0.2 0.53 ± 5% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.75 ± 2% -0.1 0.61 ± 5% perf-profile.self.cycles-pp.get_page_from_freelist 0.38 -0.1 0.26 ± 7% perf-profile.self.cycles-pp.__list_add_valid 0.41 -0.1 0.30 ± 6% perf-profile.self.cycles-pp.sctp_recvmsg 0.30 ± 14% -0.1 0.19 ± 6% perf-profile.self.cycles-pp.ipv4_dst_check 0.28 ± 15% -0.1 0.17 ± 7% perf-profile.self.cycles-pp.__copy_skb_header 0.40 ± 2% -0.1 0.29 ± 5% perf-profile.self.cycles-pp.__slab_free 0.33 ± 2% -0.1 0.24 ± 7% perf-profile.self.cycles-pp.free_one_page 0.39 ± 2% -0.1 0.30 ± 3% perf-profile.self.cycles-pp.copy_user_generic_unrolled 0.34 ± 13% -0.1 0.25 perf-profile.self.cycles-pp.sctp_cmd_interpreter 0.21 ± 13% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.sctp_transport_hold 0.39 ± 5% -0.1 0.31 ± 5% perf-profile.self.cycles-pp.sctp_packet_transmit 0.46 ± 2% -0.1 0.38 ± 5% perf-profile.self.cycles-pp.sctp_outq_flush 0.30 ± 2% -0.1 0.22 ± 3% perf-profile.self.cycles-pp.sctp_chunk_put 0.33 -0.1 0.25 ± 7% perf-profile.self.cycles-pp.kmem_cache_free 0.40 -0.1 0.33 ± 6% perf-profile.self.cycles-pp.___might_sleep 0.27 ± 5% -0.1 0.20 ± 6% perf-profile.self.cycles-pp.sctp_rcv 0.26 ± 7% -0.1 0.18 ± 8% perf-profile.self.cycles-pp.sctp_packet_config 0.18 ± 3% -0.1 0.11 ± 11% perf-profile.self.cycles-pp.sctp_ulpq_tail_event 0.18 ± 2% -0.1 0.11 perf-profile.self.cycles-pp.sctp_chunkify 0.23 ± 4% -0.1 0.16 ± 6% perf-profile.self.cycles-pp.skb_release_data 0.23 ± 6% -0.1 0.16 ± 5% perf-profile.self.cycles-pp.__alloc_pages_nodemask 0.33 ± 3% -0.1 0.27 ± 7% perf-profile.self.cycles-pp.__sched_text_start 0.26 ± 2% -0.1 0.20 ± 7% perf-profile.self.cycles-pp.__skb_clone 0.22 ± 3% -0.1 0.16 ± 4% perf-profile.self.cycles-pp.memset_erms 0.22 ± 5% -0.1 0.16 ± 5% perf-profile.self.cycles-pp.skb_set_owner_w 0.20 ± 4% -0.1 0.14 ± 7% perf-profile.self.cycles-pp.__switch_to 0.15 ± 3% -0.1 0.10 ± 5% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.24 -0.1 0.18 ± 9% perf-profile.self.cycles-pp.sctp_sock_rfree 0.28 -0.1 0.22 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_bh 0.31 ± 2% -0.1 0.26 ± 3% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.18 ± 10% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.sctp_data_ready 0.27 ± 2% -0.1 0.21 ± 4% perf-profile.self.cycles-pp.__might_sleep 0.29 ± 2% -0.1 0.24 ± 5% perf-profile.self.cycles-pp.__fget_light 0.12 ± 7% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.ip_send_check 0.29 ± 6% -0.0 0.24 ± 4% perf-profile.self.cycles-pp.__check_object_size 0.27 ± 4% -0.0 0.22 perf-profile.self.cycles-pp.sctp_wfree 0.26 ± 7% -0.0 0.21 ± 5% perf-profile.self.cycles-pp.sctp_sendmsg 0.15 ± 10% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.ipv4_mtu 0.11 ± 3% -0.0 0.07 ± 17% perf-profile.self.cycles-pp.resched_curr 0.22 ± 9% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.sctp_skb_recv_datagram 0.14 ± 5% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.sock_wfree 0.15 ± 3% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.sctp_eat_data 0.13 ± 13% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.dst_release 0.15 ± 4% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.sctp_transport_put 0.08 ± 12% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.sctp_ulpevent_free 0.07 ± 7% -0.0 0.03 ±100% perf-profile.self.cycles-pp.pick_next_entity 0.07 ± 7% -0.0 0.03 ±100% perf-profile.self.cycles-pp.sctp_hash_key 0.33 ± 2% -0.0 0.29 ± 6% perf-profile.self.cycles-pp.__free_pages_ok 0.12 ± 5% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.sctp_chunk_hold 0.08 ± 6% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.14 ± 3% -0.0 0.10 ± 8% perf-profile.self.cycles-pp.enqueue_entity 0.18 ± 7% -0.0 0.14 ± 5% perf-profile.self.cycles-pp.__update_load_avg_se 0.17 ± 5% -0.0 0.13 perf-profile.self.cycles-pp.sctp_association_put 0.21 ± 5% -0.0 0.18 ± 7% perf-profile.self.cycles-pp.____sys_recvmsg 0.20 ± 2% -0.0 0.17 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.15 ± 3% -0.0 0.12 ± 6% perf-profile.self.cycles-pp.sctp_v4_addr_valid 0.14 ± 8% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.sctp_epaddr_lookup_transport 0.14 ± 3% -0.0 0.11 ± 3% perf-profile.self.cycles-pp.set_next_entity 0.16 ± 4% -0.0 0.13 perf-profile.self.cycles-pp.do_idle 0.16 ± 2% -0.0 0.13 ± 8% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.13 ± 9% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.__netif_receive_skb_core 0.18 ± 2% -0.0 0.15 ± 8% perf-profile.self.cycles-pp.__zone_watermark_ok 0.20 ± 4% -0.0 0.17 ± 7% perf-profile.self.cycles-pp.sctp_sendmsg_to_asoc 0.09 ± 7% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.sctp_assoc_rwnd_increase 0.15 ± 4% -0.0 0.12 ± 12% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.12 ± 7% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.__softirqentry_text_start 0.11 ± 4% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.sctp_check_transmitted 0.10 ± 7% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.sctp_ulpevent_make_rcvmsg 0.14 ± 3% -0.0 0.11 ± 6% perf-profile.self.cycles-pp.__skb_datagram_iter 0.16 ± 2% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.sctp_sched_fcfs_dequeue 0.12 ± 4% -0.0 0.10 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.12 ± 5% -0.0 0.10 ± 11% perf-profile.self.cycles-pp.ip_finish_output2 0.08 ± 5% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.sock_kmalloc 0.06 ± 6% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.sockfd_lookup_light 0.10 ± 7% -0.0 0.08 ± 14% perf-profile.self.cycles-pp.sctp_association_hold 0.09 ± 4% -0.0 0.07 ± 10% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.11 ± 4% -0.0 0.09 ± 5% perf-profile.self.cycles-pp.sctp_ulpevent_receive_data 0.11 ± 4% -0.0 0.09 ± 5% perf-profile.self.cycles-pp.sctp_make_datafrag_empty 0.09 ± 4% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.consume_skb 0.09 ± 4% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.sctp_inq_pop 0.08 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.__kmalloc_node_track_caller 0.14 ± 3% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.mod_node_page_state 0.07 ± 5% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.account_entity_dequeue 0.08 ± 6% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.sctp_v4_ecn_capable 0.09 ± 10% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.rw_copy_check_uvector 0.11 ± 3% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.__switch_to_asm 0.11 ± 4% -0.0 0.10 ± 8% perf-profile.self.cycles-pp.sctp_sched_dequeue_common 0.08 ± 5% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.nr_iowait_cpu 0.11 ± 4% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.update_curr 0.09 ± 4% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.copy_msghdr_from_user 0.08 -0.0 0.07 ± 6% perf-profile.self.cycles-pp.__sctp_packet_append_chunk 0.08 ± 5% -0.0 0.07 perf-profile.self.cycles-pp.sctp_addrs_lookup_transport 0.06 -0.0 0.05 perf-profile.self.cycles-pp.__free_pages 0.06 ± 7% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.___perf_sw_event 12.88 ± 2% +12.8 25.66 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath *************************************************************************************************** lkp-skl-fpga01: 104 threads Skylake with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-20191114.cgz/300s/lkp-skl-fpga01/whetstone-double/unixbench/0x2000065 commit: 9c40365a65 ("threads: Update PID limit comment according to futex UAPI change") 59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group") 9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 ---------------- --------------------------- %stddev %change %stddev \ | \ 23098 -4.0% 22179 unixbench.score 474.19 +2.0% 483.52 unixbench.time.elapsed_time 474.19 +2.0% 483.52 unixbench.time.elapsed_time.max 20407 -15.6% 17220 unixbench.time.involuntary_context_switches 11653 +2.5% 11940 unixbench.time.user_time 6674 -2.7% 6494 unixbench.time.voluntary_context_switches 60218701 -2.2% 58918844 unixbench.workload 51504 ± 2% +61.4% 83147 ± 16% cpuidle.C1.usage 45407 ± 4% +9.0% 49503 ± 6% meminfo.Shmem 0.00 ±117% +0.0 0.03 ± 78% mpstat.cpu.all.soft% 1505 +5.3% 1584 vmstat.system.cs 9163 -9.2% 8320 ± 6% numa-meminfo.node0.KernelStack 992.75 ± 12% -49.9% 497.00 ± 65% numa-meminfo.node1.Mlocked 535099 ± 2% -7.3% 496178 ± 3% numa-meminfo.node1.Unevictable 9165 -9.1% 8327 ± 6% numa-vmstat.node0.nr_kernel_stack 133774 ± 2% -7.3% 124044 ± 3% numa-vmstat.node1.nr_unevictable 133774 ± 2% -7.3% 124044 ± 3% numa-vmstat.node1.nr_zone_unevictable 1.24 ± 16% -0.7 0.49 ± 32% perf-profile.children.cycles-pp.ktime_get 0.94 ± 18% -0.5 0.43 ± 29% perf-profile.children.cycles-pp.clockevents_program_event 0.11 ± 36% +0.1 0.20 ± 32% perf-profile.children.cycles-pp.tick_irq_enter 1.12 ± 17% -0.7 0.38 ± 49% perf-profile.self.cycles-pp.ktime_get 842912 ± 13% -42.0% 489022 ± 36% numa-numastat.node0.local_node 851323 ± 14% -39.6% 514237 ± 31% numa-numastat.node0.numa_hit 8420 ±172% +199.5% 25221 ± 57% numa-numastat.node0.other_node 780061 ± 13% +47.3% 1149237 ± 15% numa-numastat.node1.local_node 805331 ± 14% +43.8% 1157736 ± 14% numa-numastat.node1.numa_hit 92.00 ± 6% -5.4% 87.00 proc-vmstat.nr_dirtied 11349 ± 4% +9.0% 12374 ± 6% proc-vmstat.nr_shmem 19448 -1.5% 19147 proc-vmstat.nr_slab_reclaimable 44561 -3.1% 43181 proc-vmstat.nr_slab_unreclaimable 13330 ± 44% -84.1% 2121 ± 82% proc-vmstat.numa_hint_faults 5558 ±107% -99.3% 37.25 ±148% proc-vmstat.numa_hint_faults_local 51552 ± 33% -54.6% 23426 ± 53% proc-vmstat.numa_pte_updates 5345 ± 12% +24.6% 6662 ± 16% proc-vmstat.pgactivate 1739692 +1.0% 1756902 proc-vmstat.pgfree 1018 ± 7% -16.4% 850.75 ± 5% slabinfo.file_lock_cache.active_objs 1018 ± 7% -16.4% 850.75 ± 5% slabinfo.file_lock_cache.num_objs 4826 -11.2% 4285 ± 3% slabinfo.files_cache.active_objs 4826 -11.2% 4285 ± 3% slabinfo.files_cache.num_objs 2826 ± 2% -11.0% 2515 slabinfo.sighand_cache.active_objs 2828 ± 2% -11.0% 2517 slabinfo.sighand_cache.num_objs 1151 ± 2% -24.1% 873.25 ± 22% slabinfo.task_group.active_objs 1151 ± 2% -24.1% 873.25 ± 22% slabinfo.task_group.num_objs 59534 ± 2% -9.2% 54079 ± 3% slabinfo.vm_area_struct.active_objs 1490 ± 2% -9.1% 1354 ± 3% slabinfo.vm_area_struct.active_slabs 59640 ± 2% -9.1% 54195 ± 3% slabinfo.vm_area_struct.num_objs 1490 ± 2% -9.1% 1354 ± 3% slabinfo.vm_area_struct.num_slabs 7.107e+09 -2.7% 6.915e+09 perf-stat.i.branch-instructions 12.39 ± 15% -6.0 6.37 ± 23% perf-stat.i.cache-miss-rate% 1477 +5.5% 1558 perf-stat.i.context-switches 111.88 -1.3% 110.46 perf-stat.i.cpu-migrations 1.099e+10 -2.5% 1.071e+10 perf-stat.i.dTLB-loads 3.823e+09 -2.6% 3.724e+09 perf-stat.i.dTLB-stores 2132180 -20.1% 1703908 ± 14% perf-stat.i.iTLB-loads 6.802e+10 -2.7% 6.618e+10 perf-stat.i.instructions 0.88 -4.5% 0.84 ± 2% perf-stat.i.ipc 4076 -1.5% 4014 perf-stat.i.minor-faults 107204 ± 2% -31.1% 73884 ± 11% perf-stat.i.node-load-misses 14357 ± 5% -9.6% 12974 ± 3% perf-stat.i.node-loads 26949 ± 4% -51.0% 13212 ± 18% perf-stat.i.node-store-misses 4076 -1.5% 4014 perf-stat.i.page-faults 12.57 ± 16% -6.1 6.42 ± 24% perf-stat.overall.cache-miss-rate% 1.04 +3.8% 1.08 perf-stat.overall.cpi 0.96 -3.6% 0.93 perf-stat.overall.ipc 88.19 -3.3 84.92 perf-stat.overall.node-load-miss-rate% 78.98 -11.9 67.04 ± 8% perf-stat.overall.node-store-miss-rate% 536500 +1.3% 543249 perf-stat.overall.path-length 7.106e+09 -2.8% 6.907e+09 perf-stat.ps.branch-instructions 1474 +5.5% 1554 perf-stat.ps.context-switches 111.66 -1.3% 110.22 perf-stat.ps.cpu-migrations 1.098e+10 -2.6% 1.07e+10 perf-stat.ps.dTLB-loads 3.823e+09 -2.7% 3.719e+09 perf-stat.ps.dTLB-stores 2128273 -20.1% 1700626 ± 14% perf-stat.ps.iTLB-loads 6.801e+10 -2.8% 6.61e+10 perf-stat.ps.instructions 4061 -1.4% 4004 perf-stat.ps.minor-faults 107032 ± 2% -31.1% 73785 ± 11% perf-stat.ps.node-load-misses 14320 ± 5% -9.5% 12960 ± 3% perf-stat.ps.node-loads 26906 ± 4% -51.0% 13193 ± 18% perf-stat.ps.node-store-misses 4061 -1.4% 4004 perf-stat.ps.page-faults 51615 ± 9% +38.7% 71599 ± 14% sched_debug.cfs_rq:/.exec_clock.avg 89952 ± 7% +79.0% 160981 ± 14% sched_debug.cfs_rq:/.exec_clock.max 6590 ± 39% -88.3% 770.93 ±168% sched_debug.cfs_rq:/.exec_clock.min 21438 ± 5% +140.7% 51593 ± 19% sched_debug.cfs_rq:/.exec_clock.stddev 1649313 ± 8% +39.8% 2305637 ± 14% sched_debug.cfs_rq:/.min_vruntime.avg 2846100 ± 7% +82.5% 5195365 ± 14% sched_debug.cfs_rq:/.min_vruntime.max 225248 ± 36% -87.1% 29096 ±143% sched_debug.cfs_rq:/.min_vruntime.min 677866 ± 5% +145.1% 1661129 ± 19% sched_debug.cfs_rq:/.min_vruntime.stddev 130.54 ± 7% -54.9% 58.93 ±100% sched_debug.cfs_rq:/.removed.load_avg.max 19.52 ± 32% -67.2% 6.41 ±100% sched_debug.cfs_rq:/.removed.load_avg.stddev 130.54 ± 7% -54.9% 58.93 ±100% sched_debug.cfs_rq:/.removed.runnable_avg.max 19.52 ± 32% -67.2% 6.41 ±100% sched_debug.cfs_rq:/.removed.runnable_avg.stddev -1075371 -254.5% 1660977 ± 63% sched_debug.cfs_rq:/.spread0.avg 121453 ± 95% +3646.9% 4550704 ± 34% sched_debug.cfs_rq:/.spread0.max -2499411 -75.4% -615566 sched_debug.cfs_rq:/.spread0.min 677861 ± 5% +145.1% 1661130 ± 19% sched_debug.cfs_rq:/.spread0.stddev 6.00 ± 7% -31.6% 4.10 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.avg 25.29 ± 7% -22.8% 19.52 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.stddev 241338 ± 5% +12.8% 272122 ± 4% sched_debug.cpu.clock.avg 241342 ± 5% +12.8% 272127 ± 4% sched_debug.cpu.clock.max 241333 ± 5% +12.8% 272117 ± 4% sched_debug.cpu.clock.min 2.47 ± 2% +12.0% 2.76 ± 4% sched_debug.cpu.clock.stddev 241338 ± 5% +12.8% 272122 ± 4% sched_debug.cpu.clock_task.avg 241342 ± 5% +12.8% 272127 ± 4% sched_debug.cpu.clock_task.max 241333 ± 5% +12.8% 272117 ± 4% sched_debug.cpu.clock_task.min 2.47 ± 2% +12.0% 2.76 ± 4% sched_debug.cpu.clock_task.stddev 4932 ± 3% +12.7% 5557 ± 3% sched_debug.cpu.nr_switches.avg 3525 ± 5% +43.7% 5065 ± 17% sched_debug.cpu.nr_switches.stddev 2933 ± 5% +20.6% 3536 ± 5% sched_debug.cpu.sched_count.avg 19139 ± 27% +56.8% 30011 ± 48% sched_debug.cpu.sched_count.max 2773 ± 10% +55.8% 4320 ± 18% sched_debug.cpu.sched_count.stddev 1280 ± 5% +22.6% 1568 ± 6% sched_debug.cpu.sched_goidle.avg 9370 ± 27% +57.8% 14789 ± 48% sched_debug.cpu.sched_goidle.max 1374 ± 10% +56.0% 2144 ± 19% sched_debug.cpu.sched_goidle.stddev 1302 ± 5% +21.4% 1581 ± 6% sched_debug.cpu.ttwu_count.avg 1455 ± 4% +56.4% 2276 ± 19% sched_debug.cpu.ttwu_count.stddev 594.81 ± 4% +28.4% 763.77 ± 7% sched_debug.cpu.ttwu_local.avg 2311 ± 24% +79.3% 4143 ± 10% sched_debug.cpu.ttwu_local.max 345.75 ± 7% +105.0% 708.72 ± 4% sched_debug.cpu.ttwu_local.stddev 241334 ± 5% +12.8% 272118 ± 4% sched_debug.cpu_clk 240831 ± 5% +12.8% 271615 ± 4% sched_debug.ktime 241752 ± 5% +12.7% 272537 ± 4% sched_debug.sched_clk 74.25 ±159% -97.0% 2.25 ±101% interrupts.CPU1.TLB:TLB_shootdowns 105.75 ± 40% +3409.2% 3711 ± 64% interrupts.CPU100.NMI:Non-maskable_interrupts 105.75 ± 40% +3409.2% 3711 ± 64% interrupts.CPU100.PMI:Performance_monitoring_interrupts 159.00 ± 10% +64.2% 261.00 ± 22% interrupts.CPU100.RES:Rescheduling_interrupts 1558 ±163% +190.8% 4532 ± 53% interrupts.CPU101.NMI:Non-maskable_interrupts 1558 ±163% +190.8% 4532 ± 53% interrupts.CPU101.PMI:Performance_monitoring_interrupts 179.25 ± 6% +48.5% 266.25 ± 12% interrupts.CPU101.RES:Rescheduling_interrupts 143.00 ± 82% +2066.4% 3098 ± 83% interrupts.CPU102.NMI:Non-maskable_interrupts 143.00 ± 82% +2066.4% 3098 ± 83% interrupts.CPU102.PMI:Performance_monitoring_interrupts 173.50 ± 19% +66.9% 289.50 ± 9% interrupts.CPU102.RES:Rescheduling_interrupts 193.75 ± 6% +68.5% 326.50 ± 17% interrupts.CPU103.RES:Rescheduling_interrupts 3261 ±140% -96.0% 129.00 ±128% interrupts.CPU12.RES:Rescheduling_interrupts 398.25 ±110% -99.4% 2.50 ±103% interrupts.CPU16.TLB:TLB_shootdowns 146.25 ±160% -97.9% 3.00 ± 62% interrupts.CPU26.TLB:TLB_shootdowns 372.25 ± 79% -99.0% 3.75 ± 51% interrupts.CPU29.TLB:TLB_shootdowns 101.50 ±107% -97.0% 3.00 ± 84% interrupts.CPU3.TLB:TLB_shootdowns 98.75 ± 21% +6523.3% 6540 ±168% interrupts.CPU31.RES:Rescheduling_interrupts 94.50 ± 38% +110.3% 198.75 ± 22% interrupts.CPU33.RES:Rescheduling_interrupts 100.25 ± 38% +114.2% 214.75 ± 31% interrupts.CPU34.RES:Rescheduling_interrupts 81.50 ± 18% +140.8% 196.25 ± 49% interrupts.CPU35.RES:Rescheduling_interrupts 88.75 ± 45% +5932.7% 5354 ±167% interrupts.CPU36.RES:Rescheduling_interrupts 71.25 ± 26% +111.2% 150.50 ± 16% interrupts.CPU38.RES:Rescheduling_interrupts 65.25 ± 28% +136.8% 154.50 ± 20% interrupts.CPU39.RES:Rescheduling_interrupts 45.75 ± 12% +288.0% 177.50 ± 51% interrupts.CPU40.RES:Rescheduling_interrupts 75.00 ± 21% +69.0% 126.75 ± 7% interrupts.CPU41.RES:Rescheduling_interrupts 618.25 ±144% +554.8% 4048 ± 51% interrupts.CPU42.NMI:Non-maskable_interrupts 618.25 ±144% +554.8% 4048 ± 51% interrupts.CPU42.PMI:Performance_monitoring_interrupts 80.25 ± 16% +132.1% 186.25 ± 23% interrupts.CPU42.RES:Rescheduling_interrupts 861.75 ± 99% +596.3% 6000 ± 30% interrupts.CPU43.NMI:Non-maskable_interrupts 861.75 ± 99% +596.3% 6000 ± 30% interrupts.CPU43.PMI:Performance_monitoring_interrupts 69.25 ± 17% +173.6% 189.50 ± 10% interrupts.CPU43.RES:Rescheduling_interrupts 96.25 ± 50% +99.0% 191.50 ± 13% interrupts.CPU44.RES:Rescheduling_interrupts 1133 ± 91% +246.8% 3931 ± 27% interrupts.CPU45.NMI:Non-maskable_interrupts 1133 ± 91% +246.8% 3931 ± 27% interrupts.CPU45.PMI:Performance_monitoring_interrupts 617.00 ±146% +614.5% 4408 ± 38% interrupts.CPU47.NMI:Non-maskable_interrupts 617.00 ±146% +614.5% 4408 ± 38% interrupts.CPU47.PMI:Performance_monitoring_interrupts 617.00 ±146% +945.3% 6449 ± 22% interrupts.CPU49.NMI:Non-maskable_interrupts 617.00 ±146% +945.3% 6449 ± 22% interrupts.CPU49.PMI:Performance_monitoring_interrupts 81.00 ± 55% +2323.5% 1963 ±159% interrupts.CPU49.RES:Rescheduling_interrupts 3679 ± 63% -96.8% 116.25 ± 40% interrupts.CPU5.NMI:Non-maskable_interrupts 3679 ± 63% -96.8% 116.25 ± 40% interrupts.CPU5.PMI:Performance_monitoring_interrupts 3501 ± 37% -96.4% 124.50 ± 43% interrupts.CPU57.NMI:Non-maskable_interrupts 3501 ± 37% -96.4% 124.50 ± 43% interrupts.CPU57.PMI:Performance_monitoring_interrupts 4911 ± 6% -72.3% 1362 ±157% interrupts.CPU62.NMI:Non-maskable_interrupts 4911 ± 6% -72.3% 1362 ±157% interrupts.CPU62.PMI:Performance_monitoring_interrupts 4086 ± 36% -67.1% 1342 ±157% interrupts.CPU64.NMI:Non-maskable_interrupts 4086 ± 36% -67.1% 1342 ±157% interrupts.CPU64.PMI:Performance_monitoring_interrupts 245.25 ± 44% -85.4% 35.75 ± 97% interrupts.CPU7.RES:Rescheduling_interrupts 63.00 ± 71% -94.4% 3.50 ± 95% interrupts.CPU77.RES:Rescheduling_interrupts 2334 ± 51% +217.0% 7399 ± 6% interrupts.CPU78.NMI:Non-maskable_interrupts 2334 ± 51% +217.0% 7399 ± 6% interrupts.CPU78.PMI:Performance_monitoring_interrupts 132.75 ± 21% +157.6% 342.00 ± 22% interrupts.CPU78.RES:Rescheduling_interrupts 128.25 ± 27% +103.5% 261.00 ± 14% interrupts.CPU79.RES:Rescheduling_interrupts 3236 ±143% -99.0% 33.00 ±101% interrupts.CPU8.RES:Rescheduling_interrupts 2481 ± 43% +179.1% 6924 ± 17% interrupts.CPU80.NMI:Non-maskable_interrupts 2481 ± 43% +179.1% 6924 ± 17% interrupts.CPU80.PMI:Performance_monitoring_interrupts 132.50 ± 9% +100.9% 266.25 ± 14% interrupts.CPU80.RES:Rescheduling_interrupts 131.50 ± 33% +90.3% 250.25 ± 20% interrupts.CPU81.RES:Rescheduling_interrupts 1570 ± 63% +371.1% 7400 ± 6% interrupts.CPU82.NMI:Non-maskable_interrupts 1570 ± 63% +371.1% 7400 ± 6% interrupts.CPU82.PMI:Performance_monitoring_interrupts 131.75 ± 13% +2020.3% 2793 ±159% interrupts.CPU82.RES:Rescheduling_interrupts 769.50 ±130% -87.3% 97.50 ±105% interrupts.CPU9.RES:Rescheduling_interrupts 147.50 ± 19% +79.3% 264.50 ± 21% interrupts.CPU90.RES:Rescheduling_interrupts 180.50 ± 9% +1677.8% 3209 ±158% interrupts.CPU91.RES:Rescheduling_interrupts 164.50 ± 20% +119.1% 360.50 ± 26% interrupts.CPU93.RES:Rescheduling_interrupts 1573 ±162% +307.6% 6411 ± 23% interrupts.CPU94.NMI:Non-maskable_interrupts 1573 ±162% +307.6% 6411 ± 23% interrupts.CPU94.PMI:Performance_monitoring_interrupts 148.25 ± 20% +76.9% 262.25 ± 17% interrupts.CPU94.RES:Rescheduling_interrupts 143.75 ± 8% +68.5% 242.25 ± 16% interrupts.CPU95.RES:Rescheduling_interrupts 168.75 ± 6% +127.0% 383.00 ± 48% interrupts.CPU96.RES:Rescheduling_interrupts 2281 ±106% +181.2% 6415 ± 23% interrupts.CPU97.NMI:Non-maskable_interrupts 2281 ±106% +181.2% 6415 ± 23% interrupts.CPU97.PMI:Performance_monitoring_interrupts 163.75 ± 17% +71.3% 280.50 ± 16% interrupts.CPU97.RES:Rescheduling_interrupts 84.75 ± 27% +5459.6% 4711 ± 64% interrupts.CPU98.NMI:Non-maskable_interrupts 84.75 ± 27% +5459.6% 4711 ± 64% interrupts.CPU98.PMI:Performance_monitoring_interrupts 176.25 ± 8% +91.2% 337.00 ± 23% interrupts.CPU98.RES:Rescheduling_interrupts 164.25 ± 8% +61.5% 265.25 ± 20% interrupts.CPU99.RES:Rescheduling_interrupts 242630 ± 11% +27.8% 310009 ± 13% interrupts.NMI:Non-maskable_interrupts 242630 ± 11% +27.8% 310009 ± 13% interrupts.PMI:Performance_monitoring_interrupts 3842 ± 14% -81.4% 714.00 ± 62% interrupts.TLB:TLB_shootdowns 39213 ± 2% +59.4% 62496 ± 8% softirqs.CPU1.SCHED 112479 ± 3% +11.6% 125580 ± 6% softirqs.CPU10.RCU 47544 ± 5% +34.8% 64105 ± 6% softirqs.CPU10.SCHED 115525 ± 5% +34.5% 155325 ± 5% softirqs.CPU100.RCU 42079 ± 4% -34.4% 27593 ± 14% softirqs.CPU100.SCHED 115645 ± 4% +33.4% 154231 ± 4% softirqs.CPU101.RCU 42390 ± 5% -32.4% 28661 ± 9% softirqs.CPU101.SCHED 113237 ± 5% +37.9% 156145 ± 4% softirqs.CPU102.RCU 43877 ± 3% -42.3% 25316 ± 19% softirqs.CPU102.SCHED 115048 ± 4% +35.3% 155692 ± 6% softirqs.CPU103.RCU 42057 ± 2% -34.8% 27428 ± 19% softirqs.CPU103.SCHED 50287 ± 2% +26.8% 63739 ± 2% softirqs.CPU12.SCHED 51055 ± 3% +24.5% 63567 ± 2% softirqs.CPU13.SCHED 111557 ± 4% +15.1% 128441 ± 6% softirqs.CPU14.RCU 51201 ± 8% +21.9% 62437 ± 6% softirqs.CPU14.SCHED 106509 ± 4% +31.4% 139927 ± 7% softirqs.CPU15.RCU 106554 ± 5% +31.4% 140038 ± 10% softirqs.CPU16.RCU 102888 ± 3% +22.9% 126467 ± 7% softirqs.CPU17.RCU 102645 ± 4% +23.8% 127097 ± 8% softirqs.CPU18.RCU 103074 ± 3% +27.2% 131067 ± 8% softirqs.CPU19.RCU 40123 ± 2% +64.7% 66094 ± 16% softirqs.CPU2.SCHED 105947 ± 4% +31.3% 139080 ± 11% softirqs.CPU20.RCU 56698 ± 3% +10.3% 62534 ± 4% softirqs.CPU20.SCHED 107845 ± 4% +25.5% 135380 ± 7% softirqs.CPU21.RCU 104635 ± 2% +22.9% 128563 ± 7% softirqs.CPU22.RCU 57459 ± 2% +9.0% 62636 ± 4% softirqs.CPU22.SCHED 107558 ± 4% +26.9% 136472 ± 9% softirqs.CPU23.RCU 103404 ± 2% +21.9% 126077 ± 9% softirqs.CPU24.RCU 57845 +8.9% 63018 ± 3% softirqs.CPU24.SCHED 103795 ± 3% +27.4% 132281 ± 10% softirqs.CPU25.RCU 116855 ± 3% +39.2% 162714 ± 3% softirqs.CPU26.RCU 40401 ± 3% +14.5% 46275 ± 10% softirqs.CPU26.SCHED 117905 ± 5% +38.2% 163001 ± 3% softirqs.CPU27.RCU 119161 +36.5% 162677 ± 2% softirqs.CPU28.RCU 155833 ± 4% +31.8% 205398 ± 11% softirqs.CPU28.TIMER 120298 ± 6% +39.4% 167637 ± 3% softirqs.CPU29.RCU 165459 ± 5% +28.9% 213346 ± 9% softirqs.CPU29.TIMER 41776 ± 2% +47.3% 61528 ± 12% softirqs.CPU3.SCHED 111332 ± 3% +27.8% 142261 ± 3% softirqs.CPU30.RCU 109782 ± 4% +19.3% 130961 ± 4% softirqs.CPU31.RCU 163967 ± 3% +27.8% 209524 ± 10% softirqs.CPU31.TIMER 110345 ± 4% +25.3% 138304 ± 12% softirqs.CPU32.RCU 110912 ± 4% +27.2% 141121 ± 4% softirqs.CPU33.RCU 111899 ± 5% +27.5% 142697 ± 8% softirqs.CPU34.RCU 164801 ± 4% +25.8% 207277 ± 10% softirqs.CPU34.TIMER 110069 ± 3% +27.6% 140414 ± 11% softirqs.CPU35.RCU 166154 ± 2% +26.4% 210021 ± 9% softirqs.CPU35.TIMER 107927 ± 3% +26.8% 136882 ± 4% softirqs.CPU36.RCU 167316 ± 2% +23.4% 206530 ± 11% softirqs.CPU36.TIMER 107149 ± 3% +28.0% 137185 ± 7% softirqs.CPU37.RCU 52823 ± 5% -24.0% 40128 ± 16% softirqs.CPU37.SCHED 172148 ± 2% +20.5% 207427 ± 11% softirqs.CPU37.TIMER 106831 ± 4% +18.7% 126777 ± 6% softirqs.CPU38.RCU 169088 ± 3% +24.9% 211164 ± 9% softirqs.CPU38.TIMER 109008 ± 4% +29.2% 140872 ± 10% softirqs.CPU39.RCU 52842 ± 4% -12.9% 46051 ± 3% softirqs.CPU39.SCHED 168352 ± 2% +25.8% 211848 ± 8% softirqs.CPU39.TIMER 43344 ± 5% +40.9% 61065 ± 9% softirqs.CPU4.SCHED 109019 ± 4% +29.5% 141157 ± 8% softirqs.CPU40.RCU 53829 ± 2% -23.8% 41002 ± 25% softirqs.CPU40.SCHED 107752 ± 9% +19.7% 128985 ± 5% softirqs.CPU41.RCU 57665 -16.3% 48293 ± 5% softirqs.CPU41.SCHED 104372 ± 4% +29.1% 134762 ± 6% softirqs.CPU42.RCU 56657 -22.2% 44093 ± 8% softirqs.CPU42.SCHED 176519 ± 2% +18.9% 209864 ± 9% softirqs.CPU42.TIMER 102140 ± 4% +30.8% 133570 ± 3% softirqs.CPU43.RCU 57875 ± 2% -28.5% 41392 ± 6% softirqs.CPU43.SCHED 177283 ± 3% +17.1% 207557 ± 9% softirqs.CPU43.TIMER 102750 ± 4% +27.6% 131155 ± 6% softirqs.CPU44.RCU 57039 ± 5% -22.0% 44481 ± 7% softirqs.CPU44.SCHED 180258 ± 3% +17.0% 210978 ± 8% softirqs.CPU44.TIMER 103951 ± 3% +47.1% 152906 ± 2% softirqs.CPU45.RCU 57950 ± 2% -19.4% 46698 ± 7% softirqs.CPU45.SCHED 180059 ± 2% +17.5% 211546 ± 8% softirqs.CPU45.TIMER 106543 ± 4% +46.6% 156185 ± 3% softirqs.CPU46.RCU 58936 ± 3% -24.8% 44309 ± 6% softirqs.CPU46.SCHED 175875 ± 2% +16.8% 205464 ± 11% softirqs.CPU46.TIMER 107599 ± 4% +45.6% 156681 ± 2% softirqs.CPU47.RCU 61135 ± 11% -26.8% 44767 ± 9% softirqs.CPU47.SCHED 175037 ± 3% +18.6% 207548 ± 10% softirqs.CPU47.TIMER 104196 ± 4% +47.8% 154014 ± 3% softirqs.CPU48.RCU 56551 ± 6% -19.3% 45611 ± 16% softirqs.CPU48.SCHED 178334 ± 5% +18.4% 211119 ± 8% softirqs.CPU48.TIMER 105321 ± 4% +46.8% 154628 ± 3% softirqs.CPU49.RCU 176284 ± 2% +18.9% 209524 ± 9% softirqs.CPU49.TIMER 43643 ± 3% +40.7% 61423 ± 11% softirqs.CPU5.SCHED 104189 ± 4% +46.2% 152326 ± 3% softirqs.CPU50.RCU 58720 ± 2% -15.3% 49718 ± 11% softirqs.CPU50.SCHED 180182 ± 3% +16.7% 210254 ± 9% softirqs.CPU50.TIMER 105511 ± 3% +46.2% 154243 ± 2% softirqs.CPU51.RCU 106737 ± 4% +39.1% 148420 ± 10% softirqs.CPU52.RCU 108068 ± 4% +35.5% 146416 ± 11% softirqs.CPU53.RCU 50659 ± 3% +21.0% 61284 ± 9% softirqs.CPU53.SCHED 110776 ± 5% +29.5% 143423 ± 8% softirqs.CPU54.RCU 49820 ± 3% +24.6% 62062 ± 6% softirqs.CPU54.SCHED 108892 ± 4% +29.6% 141149 ± 9% softirqs.CPU55.RCU 48807 ± 3% +28.1% 62502 ± 6% softirqs.CPU55.SCHED 109303 ± 4% +30.2% 142335 ± 10% softirqs.CPU56.RCU 48744 ± 4% +29.7% 63244 ± 4% softirqs.CPU56.SCHED 107665 ± 4% +32.4% 142500 ± 8% softirqs.CPU57.RCU 46939 ± 4% +32.8% 62320 ± 6% softirqs.CPU57.SCHED 107899 ± 5% +32.1% 142541 ± 9% softirqs.CPU58.RCU 47890 ± 6% +29.7% 62112 ± 10% softirqs.CPU58.SCHED 112252 ± 5% +31.6% 147720 ± 9% softirqs.CPU59.RCU 45201 ± 5% +33.9% 60536 ± 11% softirqs.CPU59.SCHED 43271 ± 3% +44.9% 62698 ± 5% softirqs.CPU6.SCHED 42947 ± 4% +41.6% 60815 ± 10% softirqs.CPU60.SCHED 41118 ± 3% +48.2% 60919 ± 10% softirqs.CPU61.SCHED 43558 ± 9% +41.0% 61422 ± 8% softirqs.CPU62.SCHED 42404 ± 5% +45.0% 61473 ± 8% softirqs.CPU63.SCHED 44782 ± 6% +35.2% 60539 ± 11% softirqs.CPU65.SCHED 45270 ± 8% +34.9% 61049 ± 9% softirqs.CPU67.SCHED 46135 ± 6% +32.4% 61099 ± 12% softirqs.CPU69.SCHED 114120 ± 4% +13.0% 128950 ± 9% softirqs.CPU7.RCU 43859 ± 4% +42.6% 62558 ± 4% softirqs.CPU7.SCHED 45007 ± 4% +42.1% 63939 ± 2% softirqs.CPU70.SCHED 46279 ± 4% +31.2% 60706 ± 10% softirqs.CPU71.SCHED 47194 ± 7% +29.5% 61097 ± 9% softirqs.CPU72.SCHED 45037 ± 4% +37.5% 61916 ± 7% softirqs.CPU73.SCHED 44473 ± 7% +38.4% 61533 ± 8% softirqs.CPU74.SCHED 116513 ± 5% +32.7% 154642 ± 7% softirqs.CPU75.RCU 43065 ± 8% +41.5% 60952 ± 10% softirqs.CPU75.SCHED 114917 ± 4% +33.8% 153786 ± 7% softirqs.CPU76.RCU 44462 +40.3% 62390 ± 14% softirqs.CPU76.SCHED 114722 ± 3% +34.1% 153870 ± 7% softirqs.CPU77.RCU 46507 ± 2% +31.3% 61066 ± 8% softirqs.CPU77.SCHED 109126 ± 4% +39.8% 152515 ± 3% softirqs.CPU78.RCU 52943 ± 4% -44.1% 29615 ± 22% softirqs.CPU78.SCHED 113274 ± 2% +34.9% 152829 ± 3% softirqs.CPU79.RCU 49330 ± 4% -39.3% 29950 ± 19% softirqs.CPU79.SCHED 114525 ± 4% +10.7% 126743 ± 6% softirqs.CPU8.RCU 45993 ± 5% +35.6% 62367 ± 5% softirqs.CPU8.SCHED 107326 ± 4% +40.3% 150622 ± 3% softirqs.CPU80.RCU 51636 ± 4% -40.0% 30997 ± 16% softirqs.CPU80.SCHED 103564 ± 4% +48.3% 153578 ± 3% softirqs.CPU81.RCU 54621 ± 3% -45.0% 30058 ± 18% softirqs.CPU81.SCHED 106692 ± 5% +38.1% 147383 ± 6% softirqs.CPU82.RCU 107429 ± 4% +38.3% 148549 ± 3% softirqs.CPU83.RCU 48429 ± 3% -29.7% 34026 ± 15% softirqs.CPU83.SCHED 109037 ± 3% +30.6% 142391 ± 11% softirqs.CPU84.RCU 111642 ± 5% +30.6% 145830 ± 7% softirqs.CPU85.RCU 110949 ± 5% +33.3% 147882 ± 3% softirqs.CPU86.RCU 110815 ± 5% +31.2% 145429 ± 7% softirqs.CPU87.RCU 111916 ± 5% +34.2% 150213 ± 3% softirqs.CPU88.RCU 42154 ± 5% -22.1% 32856 ± 15% softirqs.CPU88.SCHED 111872 ± 4% +33.9% 149766 ± 3% softirqs.CPU89.RCU 47628 ± 3% +32.5% 63122 ± 3% softirqs.CPU9.SCHED 112803 ± 5% +38.3% 155970 ± 5% softirqs.CPU90.RCU 45236 ± 5% -37.4% 28321 ± 23% softirqs.CPU90.SCHED 117379 ± 5% +32.4% 155399 ± 4% softirqs.CPU91.RCU 116981 ± 3% +26.6% 148101 ± 6% softirqs.CPU92.RCU 113561 ± 5% +39.4% 158341 ± 4% softirqs.CPU93.RCU 45469 ± 3% -43.2% 25823 ± 27% softirqs.CPU93.SCHED 112230 ± 5% +38.5% 155456 ± 3% softirqs.CPU94.RCU 113604 ± 4% +34.2% 152470 ± 5% softirqs.CPU95.RCU 45422 ± 4% -33.9% 30021 ± 17% softirqs.CPU95.SCHED 112933 ± 4% +38.3% 156199 ± 4% softirqs.CPU96.RCU 44320 ± 6% -40.9% 26191 ± 20% softirqs.CPU96.SCHED 113319 ± 5% +36.9% 155146 ± 6% softirqs.CPU97.RCU 43834 ± 2% -40.0% 26290 ± 19% softirqs.CPU97.SCHED 116386 ± 6% +32.5% 154262 ± 7% softirqs.CPU98.RCU 41001 ± 8% -31.1% 28239 ± 22% softirqs.CPU98.SCHED 116165 ± 5% +33.7% 155277 ± 5% softirqs.CPU99.RCU 41299 ± 5% -33.7% 27372 ± 20% softirqs.CPU99.SCHED 11518008 ± 4% +25.8% 14489368 ± 4% softirqs.RCU 17583486 ± 3% +16.4% 20469522 ± 4% softirqs.TIMER *************************************************************************************************** lkp-skl-fpga01: 104 threads Skylake with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-20191114.cgz/300s/lkp-skl-fpga01/context1/unixbench/0x2000065 commit: 9c40365a65 ("threads: Update PID limit comment according to futex UAPI change") 59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group") 9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 kmsg.ipmi_si_dmi-ipmi-si.#:IRQ_index#not_found %stddev %change %stddev \ | \ 3115 +29.1% 4022 ± 3% unixbench.score 15433 ± 2% -36.2% 9850 ± 12% unixbench.time.involuntary_context_switches 2114 -15.0% 1797 ± 3% unixbench.time.percent_of_cpu_this_job_got 7797 -17.5% 6433 ± 3% unixbench.time.system_time 477.52 +24.2% 593.20 unixbench.time.user_time 3.69e+08 +23.6% 4.561e+08 ± 2% unixbench.time.voluntary_context_switches 4.873e+08 +29.0% 6.288e+08 ± 3% unixbench.workload 747856 ± 6% -13.1% 649552 ± 6% meminfo.DirectMap4k 1.40 +0.6 1.99 ± 3% mpstat.cpu.all.usr% 181.50 ± 14% +205.5% 554.50 ± 61% numa-meminfo.node0.Mlocked 45.00 ± 14% +207.2% 138.25 ± 61% numa-vmstat.node0.nr_mlock 76.00 +1.3% 77.00 vmstat.cpu.id 3747208 +20.0% 4498355 ± 2% vmstat.system.cs 237644 -4.2% 227776 vmstat.system.in 2932 ± 2% -11.5% 2596 ± 5% slabinfo.khugepaged_mm_slot.active_objs 2932 ± 2% -11.5% 2596 ± 5% slabinfo.khugepaged_mm_slot.num_objs 1160 ± 8% -22.8% 895.75 ± 10% slabinfo.task_group.active_objs 1160 ± 8% -22.8% 895.75 ± 10% slabinfo.task_group.num_objs 7.874e+09 -24.7% 5.932e+09 ± 5% cpuidle.C1.time 8.948e+09 ± 82% +99.3% 1.783e+10 ± 30% cpuidle.C1E.time 28927686 ± 31% +37.6% 39802094 ± 20% cpuidle.C1E.usage 32968312 +1040.8% 3.761e+08 ± 16% cpuidle.POLL.time 3321066 +3571.2% 1.219e+08 ± 17% cpuidle.POLL.usage 77470 -1.2% 76570 proc-vmstat.nr_active_anon 44723 -1.6% 44026 proc-vmstat.nr_slab_unreclaimable 77470 -1.2% 76570 proc-vmstat.nr_zone_active_anon 11008 ± 26% -56.0% 4844 ± 56% proc-vmstat.numa_hint_faults 1806 ± 10% -87.9% 218.75 ± 52% proc-vmstat.numa_hint_faults_local 10143 ± 5% -11.1% 9016 ± 3% proc-vmstat.pgactivate 3687542 ± 3% -16.9% 3066128 ± 10% sched_debug.cfs_rq:/.MIN_vruntime.max 1247933 ± 4% -15.6% 1053151 ± 11% sched_debug.cfs_rq:/.MIN_vruntime.stddev 73039 -14.6% 62394 ± 5% sched_debug.cfs_rq:/.exec_clock.avg 46502 -38.8% 28463 ± 28% sched_debug.cfs_rq:/.exec_clock.min 10133 ± 6% +94.8% 19737 ± 27% sched_debug.cfs_rq:/.exec_clock.stddev 12925 ± 8% +37.1% 17715 ± 27% sched_debug.cfs_rq:/.load.avg 3687542 ± 3% -16.9% 3066129 ± 10% sched_debug.cfs_rq:/.max_vruntime.max 1247933 ± 4% -15.6% 1053152 ± 11% sched_debug.cfs_rq:/.max_vruntime.stddev 3425325 -21.7% 2680459 ± 5% sched_debug.cfs_rq:/.min_vruntime.avg 2191093 -44.6% 1213462 ± 28% sched_debug.cfs_rq:/.min_vruntime.min 471746 ± 6% +77.8% 838686 ± 25% sched_debug.cfs_rq:/.min_vruntime.stddev 0.24 ± 26% +90.8% 0.46 ± 19% sched_debug.cfs_rq:/.nr_spread_over.avg 2.64 ± 36% +151.8% 6.65 ± 55% sched_debug.cfs_rq:/.nr_spread_over.max 0.51 ± 26% +137.0% 1.20 ± 44% sched_debug.cfs_rq:/.nr_spread_over.stddev 1437 ± 3% +37.1% 1970 ± 6% sched_debug.cfs_rq:/.runnable_avg.max 375.43 ± 2% +15.1% 432.07 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev -96957 +1058.8% -1123518 sched_debug.cfs_rq:/.spread0.min 471746 ± 6% +77.8% 838694 ± 25% sched_debug.cfs_rq:/.spread0.stddev 858.93 ± 5% +45.9% 1253 ± 13% sched_debug.cfs_rq:/.util_avg.max 649.39 ± 6% +56.7% 1017 ± 15% sched_debug.cfs_rq:/.util_est_enqueued.max 442164 ± 3% +19.3% 527581 ± 3% sched_debug.cpu.avg_idle.avg 514800 ± 4% +15.5% 594740 ± 13% sched_debug.cpu.max_idle_balance_cost.max 1444 ±148% +613.0% 10298 ± 72% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 3% -16.2% 0.00 ± 8% sched_debug.cpu.next_balance.stddev 1.04 ± 5% +81.0% 1.88 ± 22% sched_debug.cpu.nr_running.max 0.41 +12.1% 0.46 ± 6% sched_debug.cpu.nr_running.stddev 8064604 ± 2% +33.5% 10769101 ± 11% sched_debug.cpu.nr_switches.max 915034 ± 6% +206.7% 2806624 ± 32% sched_debug.cpu.nr_switches.stddev 8061279 ± 2% +33.5% 10761752 ± 11% sched_debug.cpu.sched_count.max 914923 ± 6% +206.7% 2806145 ± 32% sched_debug.cpu.sched_count.stddev 4029443 ± 2% +29.7% 5227316 ± 11% sched_debug.cpu.sched_goidle.max 457354 ± 6% +196.6% 1356319 ± 31% sched_debug.cpu.sched_goidle.stddev 3230031 +17.6% 3799652 ± 8% sched_debug.cpu.ttwu_count.avg 4032123 +37.2% 5533499 ± 11% sched_debug.cpu.ttwu_count.max 457261 ± 6% +217.1% 1449903 ± 33% sched_debug.cpu.ttwu_count.stddev 497.05 +952.2% 5229 ± 17% sched_debug.cpu.ttwu_local.avg 1512 +449.7% 8311 ± 24% sched_debug.cpu.ttwu_local.max 354.64 ± 4% +688.8% 2797 ± 29% sched_debug.cpu.ttwu_local.min 130.66 ± 6% +1584.8% 2201 ± 49% sched_debug.cpu.ttwu_local.stddev 97364 ± 2% -15.3% 82438 ± 10% softirqs.CPU100.SCHED 95732 -13.2% 83091 ± 10% softirqs.CPU101.SCHED 98020 -15.3% 83044 ± 11% softirqs.CPU102.SCHED 134854 ± 2% +9.5% 147696 ± 3% softirqs.CPU33.TIMER 100296 ± 4% -9.8% 90488 ± 8% softirqs.CPU34.SCHED 98665 ± 4% -10.0% 88752 ± 8% softirqs.CPU36.SCHED 101562 ± 2% -12.5% 88834 ± 8% softirqs.CPU37.SCHED 103241 -14.5% 88256 ± 10% softirqs.CPU38.SCHED 102016 -15.1% 86641 ± 9% softirqs.CPU39.SCHED 103072 ± 2% -15.9% 86642 ± 11% softirqs.CPU40.SCHED 103642 -16.7% 86299 ± 10% softirqs.CPU41.SCHED 136281 +8.1% 147364 ± 3% softirqs.CPU41.TIMER 104080 -18.4% 84916 ± 11% softirqs.CPU42.SCHED 104161 -25.6% 77445 ± 19% softirqs.CPU43.SCHED 103794 ± 2% -18.3% 84779 ± 11% softirqs.CPU44.SCHED 104881 -19.0% 84947 ± 12% softirqs.CPU45.SCHED 106249 -21.7% 83209 ± 11% softirqs.CPU46.SCHED 105901 -21.9% 82708 ± 12% softirqs.CPU47.SCHED 106109 -21.0% 83838 ± 10% softirqs.CPU48.SCHED 104208 ± 3% -20.5% 82800 ± 11% softirqs.CPU49.SCHED 105449 -20.8% 83521 ± 10% softirqs.CPU50.SCHED 106650 -12.8% 92988 ± 6% softirqs.CPU52.SCHED 110131 -23.6% 84129 ± 11% softirqs.CPU78.SCHED 107950 ± 3% -22.3% 83897 ± 11% softirqs.CPU79.SCHED 110414 -23.5% 84412 ± 10% softirqs.CPU80.SCHED 107954 ± 2% -21.3% 84966 ± 10% softirqs.CPU81.SCHED 107635 -22.6% 83318 ± 13% softirqs.CPU82.SCHED 107434 ± 2% -22.4% 83344 ± 11% softirqs.CPU83.SCHED 106151 ± 2% -21.5% 83345 ± 12% softirqs.CPU84.SCHED 104935 ± 5% -20.0% 83933 ± 11% softirqs.CPU85.SCHED 103760 ± 2% -19.3% 83770 ± 11% softirqs.CPU86.SCHED 103250 ± 4% -18.6% 84093 ± 11% softirqs.CPU87.SCHED 104851 ± 2% -20.2% 83629 ± 12% softirqs.CPU88.SCHED 104007 -19.5% 83676 ± 12% softirqs.CPU89.SCHED 101221 -17.8% 83192 ± 11% softirqs.CPU90.SCHED 103382 ± 2% -19.4% 83322 ± 11% softirqs.CPU91.SCHED 99759 ± 3% -16.9% 82889 ± 12% softirqs.CPU92.SCHED 99723 -15.8% 84012 ± 11% softirqs.CPU93.SCHED 99706 ± 2% -15.8% 83927 ± 11% softirqs.CPU94.SCHED 100802 -17.3% 83351 ± 11% softirqs.CPU95.SCHED 99622 ± 2% -16.3% 83351 ± 10% softirqs.CPU96.SCHED 98714 ± 2% -16.4% 82483 ± 11% softirqs.CPU97.SCHED 99272 -16.5% 82919 ± 11% softirqs.CPU98.SCHED 97699 ± 2% -15.7% 82320 ± 11% softirqs.CPU99.SCHED 30.70 ± 17% -41.8% 17.86 ± 2% perf-stat.i.MPKI 6.229e+09 +21.0% 7.537e+09 ± 2% perf-stat.i.branch-instructions 87521494 ± 2% +26.1% 1.103e+08 ± 2% perf-stat.i.branch-misses 26161990 -61.0% 10190442 ± 20% perf-stat.i.cache-misses 7.094e+08 -3.2% 6.866e+08 perf-stat.i.cache-references 3766575 +20.0% 4520959 ± 2% perf-stat.i.context-switches 3.51 ± 8% -22.6% 2.72 ± 3% perf-stat.i.cpi 8.561e+10 +2.8% 8.8e+10 perf-stat.i.cpu-cycles 1199 ± 10% +12547.8% 151679 ± 18% perf-stat.i.cpu-migrations 3326 ± 8% +1896.7% 66426 ± 59% perf-stat.i.cycles-between-cache-misses 4368880 ± 5% +260.9% 15765930 ± 11% perf-stat.i.dTLB-load-misses 7.644e+09 +21.3% 9.275e+09 ± 2% perf-stat.i.dTLB-loads 83528 ± 41% +1345.4% 1207309 ± 15% perf-stat.i.dTLB-store-misses 4.5e+09 +22.5% 5.512e+09 ± 2% perf-stat.i.dTLB-stores 4412580 +49.5% 6597934 ± 4% perf-stat.i.iTLB-load-misses 29081328 +25.9% 36603347 perf-stat.i.iTLB-loads 2.835e+10 +21.5% 3.444e+10 ± 2% perf-stat.i.instructions 0.31 ± 4% +21.5% 0.38 ± 3% perf-stat.i.ipc 94.83 -2.4 92.45 perf-stat.i.node-load-miss-rate% 4078999 -58.8% 1678711 ± 18% perf-stat.i.node-load-misses 146125 -60.1% 58372 ± 13% perf-stat.i.node-loads 95.95 -10.6 85.36 perf-stat.i.node-store-miss-rate% 3822185 -59.7% 1539813 ± 20% perf-stat.i.node-store-misses 11551 ± 3% -8.5% 10571 ± 5% perf-stat.i.node-stores 25.02 -20.3% 19.94 ± 2% perf-stat.overall.MPKI 1.41 +0.1 1.46 perf-stat.overall.branch-miss-rate% 3.69 -2.2 1.49 ± 21% perf-stat.overall.cache-miss-rate% 3.02 -15.4% 2.56 ± 2% perf-stat.overall.cpi 3272 +178.9% 9127 ± 25% perf-stat.overall.cycles-between-cache-misses 0.06 ± 5% +0.1 0.17 ± 9% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 41% +0.0 0.02 ± 13% perf-stat.overall.dTLB-store-miss-rate% 13.18 +2.1 15.27 ± 3% perf-stat.overall.iTLB-load-miss-rate% 6425 -18.7% 5224 ± 2% perf-stat.overall.instructions-per-iTLB-miss 0.33 +18.2% 0.39 ± 2% perf-stat.overall.ipc 22735 -5.8% 21419 perf-stat.overall.path-length 6.21e+09 +21.1% 7.518e+09 ± 2% perf-stat.ps.branch-instructions 87263324 ± 2% +26.1% 1.101e+08 ± 2% perf-stat.ps.branch-misses 26081704 -61.1% 10158408 ± 20% perf-stat.ps.cache-misses 7.072e+08 -3.2% 6.848e+08 perf-stat.ps.cache-references 3754979 +20.1% 4509479 ± 2% perf-stat.ps.context-switches 8.535e+10 +2.8% 8.777e+10 perf-stat.ps.cpu-cycles 1197 ± 10% +12540.5% 151373 ± 18% perf-stat.ps.cpu-migrations 4356788 ± 5% +261.1% 15731108 ± 11% perf-stat.ps.dTLB-load-misses 7.621e+09 +21.4% 9.251e+09 ± 2% perf-stat.ps.dTLB-loads 83463 ± 41% +1343.5% 1204761 ± 15% perf-stat.ps.dTLB-store-misses 4.487e+09 +22.5% 5.498e+09 ± 2% perf-stat.ps.dTLB-stores 4399472 +49.6% 6581866 ± 4% perf-stat.ps.iTLB-load-misses 28993919 +25.9% 36508403 perf-stat.ps.iTLB-loads 2.826e+10 +21.6% 3.436e+10 ± 2% perf-stat.ps.instructions 4066345 -58.8% 1673367 ± 18% perf-stat.ps.node-load-misses 145705 -60.0% 58229 ± 13% perf-stat.ps.node-loads 3810301 -59.7% 1534753 ± 20% perf-stat.ps.node-store-misses 11533 ± 3% -8.5% 10555 ± 5% perf-stat.ps.node-stores 1.108e+13 +21.5% 1.346e+13 ± 2% perf-stat.total.instructions 49518 -1.5% 48784 interrupts.CAL:Function_call_interrupts 75573 ± 2% -21.1% 59624 ± 19% interrupts.CPU0.RES:Rescheduling_interrupts 293.50 ± 46% -83.7% 47.75 ± 39% interrupts.CPU0.TLB:TLB_shootdowns 1651 ± 43% -90.6% 155.75 ± 89% interrupts.CPU1.TLB:TLB_shootdowns 107991 ± 2% -25.4% 80516 ± 13% interrupts.CPU10.RES:Rescheduling_interrupts 53.00 ± 95% -70.3% 15.75 ± 34% interrupts.CPU10.TLB:TLB_shootdowns 104860 ± 3% -45.3% 57407 ± 29% interrupts.CPU100.RES:Rescheduling_interrupts 101815 ± 3% -42.4% 58608 ± 28% interrupts.CPU101.RES:Rescheduling_interrupts 104224 ± 3% -45.2% 57146 ± 32% interrupts.CPU102.RES:Rescheduling_interrupts 3026 ± 8% -70.6% 891.00 ±149% interrupts.CPU103.NMI:Non-maskable_interrupts 3026 ± 8% -70.6% 891.00 ±149% interrupts.CPU103.PMI:Performance_monitoring_interrupts 97748 ± 3% -43.9% 54866 ± 25% interrupts.CPU103.RES:Rescheduling_interrupts 106274 ± 2% -25.2% 79442 ± 18% interrupts.CPU11.RES:Rescheduling_interrupts 109794 ± 3% -26.2% 81077 ± 13% interrupts.CPU12.RES:Rescheduling_interrupts 110641 ± 4% -28.4% 79170 ± 17% interrupts.CPU13.RES:Rescheduling_interrupts 112578 ± 4% -28.9% 80064 ± 13% interrupts.CPU14.RES:Rescheduling_interrupts 203.50 ±101% -92.1% 16.00 ± 76% interrupts.CPU14.TLB:TLB_shootdowns 115653 ± 3% -29.9% 81035 ± 16% interrupts.CPU15.RES:Rescheduling_interrupts 2336 ± 30% +56.1% 3646 ± 24% interrupts.CPU16.NMI:Non-maskable_interrupts 2336 ± 30% +56.1% 3646 ± 24% interrupts.CPU16.PMI:Performance_monitoring_interrupts 113166 ± 4% -31.0% 78040 ± 14% interrupts.CPU16.RES:Rescheduling_interrupts 117545 ± 6% -34.2% 77376 ± 15% interrupts.CPU17.RES:Rescheduling_interrupts 119816 ± 5% -35.9% 76821 ± 18% interrupts.CPU18.RES:Rescheduling_interrupts 208.00 ±115% -93.0% 14.50 ± 51% interrupts.CPU18.TLB:TLB_shootdowns 116298 ± 5% -33.1% 77798 ± 18% interrupts.CPU19.RES:Rescheduling_interrupts 118.50 ± 93% -87.8% 14.50 ± 51% interrupts.CPU19.TLB:TLB_shootdowns 471.50 ± 54% -97.1% 13.75 ± 37% interrupts.CPU2.TLB:TLB_shootdowns 119931 ± 4% -34.2% 78873 ± 16% interrupts.CPU20.RES:Rescheduling_interrupts 119193 ± 3% -33.7% 79035 ± 10% interrupts.CPU21.RES:Rescheduling_interrupts 119705 ± 4% -33.6% 79531 ± 21% interrupts.CPU22.RES:Rescheduling_interrupts 119224 -33.6% 79145 ± 15% interrupts.CPU23.RES:Rescheduling_interrupts 239.50 ± 82% -94.8% 12.50 ± 47% interrupts.CPU23.TLB:TLB_shootdowns 114268 ± 5% -32.0% 77747 ± 18% interrupts.CPU24.RES:Rescheduling_interrupts 130.75 ± 54% -87.2% 16.75 ± 80% interrupts.CPU24.TLB:TLB_shootdowns 114570 ± 2% -27.4% 83205 ± 15% interrupts.CPU25.RES:Rescheduling_interrupts 401.00 ± 34% -90.8% 36.75 ± 37% interrupts.CPU26.TLB:TLB_shootdowns 2215 ± 26% -81.2% 417.25 ± 96% interrupts.CPU27.TLB:TLB_shootdowns 1056 ± 71% -76.4% 249.75 ± 54% interrupts.CPU29.TLB:TLB_shootdowns 94273 ± 4% -14.7% 80458 ± 17% interrupts.CPU3.RES:Rescheduling_interrupts 438.75 ± 54% -80.6% 85.25 ±138% interrupts.CPU3.TLB:TLB_shootdowns 297.75 ± 52% -69.4% 91.00 ±123% interrupts.CPU30.TLB:TLB_shootdowns 99323 ± 4% -17.3% 82125 ± 18% interrupts.CPU31.RES:Rescheduling_interrupts 101813 ± 4% -22.1% 79310 ± 22% interrupts.CPU32.RES:Rescheduling_interrupts 104108 ± 4% -23.9% 79216 ± 21% interrupts.CPU33.RES:Rescheduling_interrupts 108408 ± 6% -30.2% 75649 ± 19% interrupts.CPU34.RES:Rescheduling_interrupts 107851 ± 7% -30.2% 75255 ± 22% interrupts.CPU35.RES:Rescheduling_interrupts 3050 ± 5% -38.5% 1875 ± 56% interrupts.CPU36.NMI:Non-maskable_interrupts 3050 ± 5% -38.5% 1875 ± 56% interrupts.CPU36.PMI:Performance_monitoring_interrupts 110860 ± 5% -36.0% 70961 ± 21% interrupts.CPU36.RES:Rescheduling_interrupts 2994 ± 5% -47.5% 1573 ± 56% interrupts.CPU37.NMI:Non-maskable_interrupts 2994 ± 5% -47.5% 1573 ± 56% interrupts.CPU37.PMI:Performance_monitoring_interrupts 110701 ± 6% -35.9% 70912 ± 21% interrupts.CPU37.RES:Rescheduling_interrupts 2950 ± 3% -60.7% 1158 ± 70% interrupts.CPU38.NMI:Non-maskable_interrupts 2950 ± 3% -60.7% 1158 ± 70% interrupts.CPU38.PMI:Performance_monitoring_interrupts 114683 ± 2% -39.3% 69605 ± 27% interrupts.CPU38.RES:Rescheduling_interrupts 2959 ± 3% -53.1% 1387 ± 64% interrupts.CPU39.NMI:Non-maskable_interrupts 2959 ± 3% -53.1% 1387 ± 64% interrupts.CPU39.PMI:Performance_monitoring_interrupts 112794 ± 3% -41.0% 66597 ± 21% interrupts.CPU39.RES:Rescheduling_interrupts 96299 ± 6% -19.1% 77895 ± 15% interrupts.CPU4.RES:Rescheduling_interrupts 3084 ± 6% -62.4% 1158 ± 67% interrupts.CPU40.NMI:Non-maskable_interrupts 3084 ± 6% -62.4% 1158 ± 67% interrupts.CPU40.PMI:Performance_monitoring_interrupts 115740 ± 6% -42.7% 66275 ± 28% interrupts.CPU40.RES:Rescheduling_interrupts 336.75 ± 73% -94.3% 19.25 ± 40% interrupts.CPU40.TLB:TLB_shootdowns 3109 ± 5% -59.0% 1273 ± 57% interrupts.CPU41.NMI:Non-maskable_interrupts 3109 ± 5% -59.0% 1273 ± 57% interrupts.CPU41.PMI:Performance_monitoring_interrupts 117515 ± 3% -45.4% 64133 ± 26% interrupts.CPU41.RES:Rescheduling_interrupts 3074 ± 12% -64.3% 1097 ± 89% interrupts.CPU42.NMI:Non-maskable_interrupts 3074 ± 12% -64.3% 1097 ± 89% interrupts.CPU42.PMI:Performance_monitoring_interrupts 118729 ± 3% -47.7% 62131 ± 30% interrupts.CPU42.RES:Rescheduling_interrupts 3026 ± 7% -70.3% 898.50 ± 83% interrupts.CPU43.NMI:Non-maskable_interrupts 3026 ± 7% -70.3% 898.50 ± 83% interrupts.CPU43.PMI:Performance_monitoring_interrupts 116750 ± 3% -48.2% 60473 ± 29% interrupts.CPU43.RES:Rescheduling_interrupts 2999 ± 13% -71.5% 853.50 ±116% interrupts.CPU44.NMI:Non-maskable_interrupts 2999 ± 13% -71.5% 853.50 ±116% interrupts.CPU44.PMI:Performance_monitoring_interrupts 119184 -48.6% 61233 ± 30% interrupts.CPU44.RES:Rescheduling_interrupts 119035 ± 2% -48.7% 61050 ± 32% interrupts.CPU45.RES:Rescheduling_interrupts 122760 ± 3% -52.1% 58842 ± 30% interrupts.CPU46.RES:Rescheduling_interrupts 122662 ± 2% -53.5% 57035 ± 33% interrupts.CPU47.RES:Rescheduling_interrupts 123349 ± 2% -51.7% 59540 ± 29% interrupts.CPU48.RES:Rescheduling_interrupts 123.00 ±107% -88.6% 14.00 ± 36% interrupts.CPU48.TLB:TLB_shootdowns 123038 ± 2% -53.4% 57374 ± 32% interrupts.CPU49.RES:Rescheduling_interrupts 174.50 ± 70% -92.3% 13.50 ± 24% interrupts.CPU49.TLB:TLB_shootdowns 94491 ± 3% -16.2% 79161 ± 17% interrupts.CPU5.RES:Rescheduling_interrupts 120948 ± 2% -51.3% 58853 ± 28% interrupts.CPU50.RES:Rescheduling_interrupts 121202 ± 3% -53.5% 56355 ± 35% interrupts.CPU51.RES:Rescheduling_interrupts 127597 ± 3% -32.8% 85784 ± 14% interrupts.CPU52.RES:Rescheduling_interrupts 122288 ± 4% -33.4% 81461 ± 14% interrupts.CPU53.RES:Rescheduling_interrupts 119757 ± 3% -32.0% 81484 ± 19% interrupts.CPU54.RES:Rescheduling_interrupts 232.50 ± 97% -94.3% 13.25 ± 86% interrupts.CPU54.TLB:TLB_shootdowns 114657 ± 4% -27.8% 82728 ± 16% interrupts.CPU55.RES:Rescheduling_interrupts 91.50 ±132% -84.7% 14.00 ± 59% interrupts.CPU55.TLB:TLB_shootdowns 112537 ± 6% -28.0% 80980 ± 19% interrupts.CPU56.RES:Rescheduling_interrupts 180.00 ± 94% -95.1% 8.75 ± 74% interrupts.CPU56.TLB:TLB_shootdowns 110747 ± 4% -26.4% 81457 ± 17% interrupts.CPU57.RES:Rescheduling_interrupts 309.25 ± 99% -97.1% 9.00 ± 73% interrupts.CPU57.TLB:TLB_shootdowns 108297 -26.4% 79689 ± 20% interrupts.CPU58.RES:Rescheduling_interrupts 96801 ± 2% -18.7% 78664 ± 16% interrupts.CPU6.RES:Rescheduling_interrupts 104449 ± 5% -21.7% 81825 ± 18% interrupts.CPU60.RES:Rescheduling_interrupts 150.50 ± 85% -93.7% 9.50 ± 28% interrupts.CPU60.TLB:TLB_shootdowns 103863 ± 5% -21.9% 81091 ± 19% interrupts.CPU61.RES:Rescheduling_interrupts 99849 ± 5% -18.8% 81035 ± 19% interrupts.CPU62.RES:Rescheduling_interrupts 99399 ± 4% -18.8% 80732 ± 16% interrupts.CPU63.RES:Rescheduling_interrupts 134.50 ± 88% -92.8% 9.75 ± 38% interrupts.CPU64.TLB:TLB_shootdowns 94400 ± 3% -21.2% 74427 ± 22% interrupts.CPU66.RES:Rescheduling_interrupts 96.75 ±136% -89.1% 10.50 ± 86% interrupts.CPU66.TLB:TLB_shootdowns 93400 ± 6% -21.2% 73562 ± 23% interrupts.CPU68.RES:Rescheduling_interrupts 147.25 ±147% -94.2% 8.50 ± 55% interrupts.CPU69.TLB:TLB_shootdowns 101127 ± 6% -21.5% 79428 ± 17% interrupts.CPU7.RES:Rescheduling_interrupts 43.25 ±101% -76.9% 10.00 ± 86% interrupts.CPU71.TLB:TLB_shootdowns 93.00 ±134% -90.6% 8.75 ± 45% interrupts.CPU73.TLB:TLB_shootdowns 73.50 ±129% -86.4% 10.00 ± 57% interrupts.CPU74.TLB:TLB_shootdowns 78700 ± 4% -18.1% 64461 ± 26% interrupts.CPU75.RES:Rescheduling_interrupts 82332 ± 5% -23.2% 63213 ± 19% interrupts.CPU76.RES:Rescheduling_interrupts 80750 ± 3% -20.5% 64217 ± 21% interrupts.CPU77.RES:Rescheduling_interrupts 167.00 ± 86% -53.9% 77.00 ±158% interrupts.CPU77.TLB:TLB_shootdowns 132925 ± 2% -55.5% 59127 ± 32% interrupts.CPU78.RES:Rescheduling_interrupts 130753 ± 3% -52.1% 62630 ± 28% interrupts.CPU79.RES:Rescheduling_interrupts 2129 ± 28% +106.7% 4402 ± 20% interrupts.CPU8.NMI:Non-maskable_interrupts 2129 ± 28% +106.7% 4402 ± 20% interrupts.CPU8.PMI:Performance_monitoring_interrupts 102071 ± 3% -20.1% 81537 ± 15% interrupts.CPU8.RES:Rescheduling_interrupts 130307 ± 5% -54.5% 59229 ± 29% interrupts.CPU80.RES:Rescheduling_interrupts 130176 ± 3% -52.9% 61349 ± 29% interrupts.CPU81.RES:Rescheduling_interrupts 125592 ± 2% -52.8% 59330 ± 35% interrupts.CPU82.RES:Rescheduling_interrupts 126379 ± 2% -53.4% 58871 ± 30% interrupts.CPU83.RES:Rescheduling_interrupts 122387 ± 4% -52.3% 58375 ± 32% interrupts.CPU84.RES:Rescheduling_interrupts 123520 ± 5% -51.6% 59756 ± 29% interrupts.CPU85.RES:Rescheduling_interrupts 122586 ± 6% -50.6% 60556 ± 30% interrupts.CPU86.RES:Rescheduling_interrupts 119406 ± 5% -49.8% 59963 ± 29% interrupts.CPU87.RES:Rescheduling_interrupts 119861 ± 4% -51.1% 58631 ± 33% interrupts.CPU88.RES:Rescheduling_interrupts 117115 ± 4% -50.0% 58574 ± 32% interrupts.CPU89.RES:Rescheduling_interrupts 104723 ± 2% -25.1% 78456 ± 20% interrupts.CPU9.RES:Rescheduling_interrupts 113197 ± 3% -49.1% 57667 ± 31% interrupts.CPU90.RES:Rescheduling_interrupts 117511 ± 3% -49.1% 59777 ± 31% interrupts.CPU91.RES:Rescheduling_interrupts 113502 ± 2% -48.6% 58311 ± 35% interrupts.CPU92.RES:Rescheduling_interrupts 111247 -46.3% 59729 ± 29% interrupts.CPU93.RES:Rescheduling_interrupts 109067 ± 4% -45.0% 60027 ± 29% interrupts.CPU94.RES:Rescheduling_interrupts 109639 ± 2% -46.4% 58724 ± 29% interrupts.CPU95.RES:Rescheduling_interrupts 107661 ± 3% -45.2% 59031 ± 29% interrupts.CPU96.RES:Rescheduling_interrupts 2987 ± 7% -69.8% 900.75 ±117% interrupts.CPU97.NMI:Non-maskable_interrupts 2987 ± 7% -69.8% 900.75 ±117% interrupts.CPU97.PMI:Performance_monitoring_interrupts 107034 ± 2% -47.6% 56109 ± 29% interrupts.CPU97.RES:Rescheduling_interrupts 96.00 ±125% -86.5% 13.00 ± 12% interrupts.CPU97.TLB:TLB_shootdowns 107585 ± 2% -46.3% 57788 ± 31% interrupts.CPU98.RES:Rescheduling_interrupts 104668 ± 5% -46.5% 55991 ± 33% interrupts.CPU99.RES:Rescheduling_interrupts 93.25 ±108% -84.2% 14.75 ± 14% interrupts.CPU99.TLB:TLB_shootdowns 11221741 -34.2% 7378335 ± 9% interrupts.RES:Rescheduling_interrupts 19833 ± 39% -56.4% 8648 ± 38% interrupts.TLB:TLB_shootdowns 30.31 -8.5 21.77 ± 28% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 31.04 -8.5 22.53 ± 27% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task 34.59 -6.9 27.65 ± 18% perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 36.44 -6.3 30.10 ± 15% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 36.85 -6.1 30.70 ± 14% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 36.88 -6.1 30.77 ± 14% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 36.89 -6.1 30.79 ± 14% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 39.76 -5.9 33.83 ± 14% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 40.65 -4.6 36.05 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 40.72 -4.6 36.15 ± 8% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 38.95 -4.4 34.52 ± 8% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 39.08 -4.4 34.68 ± 8% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 39.31 -4.4 34.93 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 39.59 -4.3 35.27 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 40.70 -3.9 36.76 ± 7% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 40.77 -3.9 36.87 ± 7% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 46.74 -1.8 44.95 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 46.77 -1.8 45.00 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 46.78 -1.8 45.02 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 47.29 -1.5 45.81 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64 6.01 ± 7% -1.3 4.68 ± 9% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 6.03 ± 7% -1.3 4.73 ± 9% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 6.17 ± 7% -1.2 4.99 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 6.16 ± 6% -1.2 4.98 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 6.33 ± 6% -1.1 5.26 ± 9% perf-profile.calltrace.cycles-pp.write 0.53 ± 2% +0.1 0.63 ± 14% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read 0.55 ± 2% +0.2 0.74 ± 23% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.70 ± 2% +0.2 0.91 ± 19% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.70 ± 2% +0.2 0.93 ± 21% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 0.54 +0.2 0.79 ± 24% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule 1.16 +0.3 1.43 ± 11% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle 0.64 ± 4% +0.3 0.95 ± 22% perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle 0.78 ± 3% +0.3 1.12 ± 21% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 1.53 +0.4 1.94 ± 10% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 1.02 ± 2% +0.4 1.43 ± 21% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 1.27 ± 2% +0.5 1.73 ± 22% perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity 0.26 ±100% +0.6 0.85 ± 46% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.60 ± 3% +0.7 2.28 ± 21% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.02 ± 6% +0.8 1.83 ± 9% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.05 ± 6% +0.8 1.87 ± 9% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.24 +0.8 2.08 ± 38% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.pipe_read 1.22 ± 7% +0.9 2.16 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.23 ± 7% +0.9 2.18 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 1.44 ± 7% +1.1 2.56 ± 9% perf-profile.calltrace.cycles-pp.read 1.46 +1.4 2.90 ± 25% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read 2.73 +1.5 4.22 ± 28% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary 2.78 +1.5 4.30 ± 27% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +1.6 1.55 ± 85% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 3.23 ± 6% +1.6 4.84 ± 21% perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task 2.29 ± 2% +1.8 4.12 ± 23% perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 3.08 ± 7% +2.1 5.15 ± 26% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.pipe_read.new_sync_read.vfs_read 3.17 ± 6% +2.1 5.27 ± 26% perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read 5.35 +2.2 7.56 ± 25% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.50 +2.3 7.76 ± 25% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.79 +2.7 8.48 ± 21% perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 5.90 +2.8 8.67 ± 22% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 31.89 -8.3 23.59 ± 25% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 34.59 -6.9 27.66 ± 18% perf-profile.children.cycles-pp.__account_scheduler_latency 30.51 -6.9 23.60 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 36.58 -6.3 30.26 ± 15% perf-profile.children.cycles-pp.enqueue_entity 36.85 -6.1 30.71 ± 14% perf-profile.children.cycles-pp.enqueue_task_fair 36.88 -6.1 30.77 ± 14% perf-profile.children.cycles-pp.activate_task 36.90 -6.1 30.80 ± 14% perf-profile.children.cycles-pp.ttwu_do_activate 39.93 -6.0 33.95 ± 13% perf-profile.children.cycles-pp.intel_idle 38.96 -4.4 34.52 ± 8% perf-profile.children.cycles-pp.try_to_wake_up 39.09 -4.4 34.69 ± 8% perf-profile.children.cycles-pp.autoremove_wake_function 41.17 -4.4 36.79 ± 8% perf-profile.children.cycles-pp.cpuidle_enter_state 39.31 -4.4 34.93 ± 8% perf-profile.children.cycles-pp.__wake_up_common 41.18 -4.4 36.81 ± 8% perf-profile.children.cycles-pp.cpuidle_enter 39.59 -4.3 35.28 ± 8% perf-profile.children.cycles-pp.__wake_up_common_lock 40.71 -3.9 36.77 ± 7% perf-profile.children.cycles-pp.pipe_write 40.79 -3.9 36.89 ± 7% perf-profile.children.cycles-pp.new_sync_write 41.10 -3.8 37.30 ± 6% perf-profile.children.cycles-pp.vfs_write 41.23 -3.7 37.53 ± 6% perf-profile.children.cycles-pp.ksys_write 46.78 -1.8 45.02 ± 2% perf-profile.children.cycles-pp.start_secondary 47.29 -1.5 45.81 ± 2% perf-profile.children.cycles-pp.secondary_startup_64 47.29 -1.5 45.81 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry 47.29 -1.5 45.82 ± 2% perf-profile.children.cycles-pp.do_idle 6.34 ± 6% -1.1 5.29 ± 9% perf-profile.children.cycles-pp.write 0.34 ± 5% -0.1 0.29 ± 12% perf-profile.children.cycles-pp.tick_sched_handle 0.33 ± 5% -0.0 0.29 ± 11% perf-profile.children.cycles-pp.update_process_times 0.24 ± 3% -0.0 0.20 ± 14% perf-profile.children.cycles-pp.scheduler_tick 0.05 ± 8% +0.0 0.08 ± 16% perf-profile.children.cycles-pp.generic_pipe_buf_confirm 0.23 +0.0 0.26 ± 4% perf-profile.children.cycles-pp.nr_iowait_cpu 0.12 ± 4% +0.0 0.16 ± 7% perf-profile.children.cycles-pp.cpus_share_cache 0.06 +0.0 0.09 ± 26% perf-profile.children.cycles-pp.atime_needs_update 0.26 ± 4% +0.0 0.30 ± 9% perf-profile.children.cycles-pp.rcu_idle_exit 0.08 ± 5% +0.0 0.12 ± 26% perf-profile.children.cycles-pp.copyin 0.09 ± 7% +0.0 0.13 ± 25% perf-profile.children.cycles-pp.stack_access_ok 0.09 +0.0 0.13 ± 20% perf-profile.children.cycles-pp.__calc_delta 0.04 ± 57% +0.0 0.08 ± 20% perf-profile.children.cycles-pp.is_bpf_text_address 0.23 +0.0 0.27 ± 6% perf-profile.children.cycles-pp.update_ts_time_stats 0.12 ± 7% +0.0 0.16 ± 19% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.09 ± 4% +0.0 0.13 ± 25% perf-profile.children.cycles-pp.touch_atime 0.06 ± 6% +0.0 0.10 ± 31% perf-profile.children.cycles-pp.__might_fault 0.03 ±100% +0.0 0.07 ± 6% perf-profile.children.cycles-pp.rcu_core 0.12 ± 6% +0.0 0.17 ± 13% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.21 ± 3% +0.0 0.26 ± 10% perf-profile.children.cycles-pp.newidle_balance 0.11 +0.0 0.16 ± 23% perf-profile.children.cycles-pp.in_sched_functions 0.09 ± 5% +0.0 0.13 ± 26% perf-profile.children.cycles-pp.__fsnotify_parent 0.01 ±173% +0.0 0.06 ± 11% perf-profile.children.cycles-pp.lapic_next_deadline 0.01 ±173% +0.1 0.06 ± 26% perf-profile.children.cycles-pp.__x64_sys_write 0.04 ± 57% +0.1 0.09 ± 40% perf-profile.children.cycles-pp.set_next_task_idle 0.06 ± 6% +0.1 0.12 ± 39% perf-profile.children.cycles-pp.pick_next_task_idle 0.09 ± 7% +0.1 0.14 ± 26% perf-profile.children.cycles-pp.menu_reflect 0.06 ± 7% +0.1 0.11 ± 25% perf-profile.children.cycles-pp.find_next_bit 0.01 ±173% +0.1 0.07 ± 19% perf-profile.children.cycles-pp.sched_ttwu_pending 0.00 +0.1 0.06 ± 15% perf-profile.children.cycles-pp.bpf_prog_kallsyms_find 0.00 +0.1 0.06 ± 15% perf-profile.children.cycles-pp.get_cpu_device 0.29 ± 4% +0.1 0.35 ± 8% perf-profile.children.cycles-pp.copyout 0.00 +0.1 0.06 ± 28% perf-profile.children.cycles-pp.in_lock_functions 0.09 ± 5% +0.1 0.15 ± 26% perf-profile.children.cycles-pp.current_time 0.23 +0.1 0.29 ± 10% perf-profile.children.cycles-pp.mutex_unlock 0.12 ± 5% +0.1 0.18 ± 25% perf-profile.children.cycles-pp.fsnotify 0.08 ± 5% +0.1 0.15 ± 30% perf-profile.children.cycles-pp.file_update_time 0.20 ± 2% +0.1 0.26 ± 3% perf-profile.children.cycles-pp.__list_del_entry_valid 0.07 +0.1 0.13 ± 38% perf-profile.children.cycles-pp.__x86_indirect_thunk_rbx 0.04 ± 57% +0.1 0.11 ± 24% perf-profile.children.cycles-pp.put_prev_entity 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.tick_nohz_irq_exit 0.00 +0.1 0.06 ± 17% perf-profile.children.cycles-pp.tick_nohz_tick_stopped 0.14 ± 3% +0.1 0.21 ± 21% perf-profile.children.cycles-pp.resched_curr 0.00 +0.1 0.07 ± 31% perf-profile.children.cycles-pp.timestamp_truncate 0.00 +0.1 0.07 ± 30% perf-profile.children.cycles-pp.apparmor_file_permission 0.11 ± 6% +0.1 0.18 ± 21% perf-profile.children.cycles-pp.rcu_eqs_enter 0.32 ± 6% +0.1 0.39 ± 11% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.11 ± 4% +0.1 0.19 ± 23% perf-profile.children.cycles-pp.account_entity_dequeue 0.00 +0.1 0.07 ± 43% perf-profile.children.cycles-pp.__update_idle_core 0.28 ± 3% +0.1 0.35 ± 6% perf-profile.children.cycles-pp.irq_exit 0.17 ± 4% +0.1 0.25 ± 24% perf-profile.children.cycles-pp.__fget_light 0.23 ± 3% +0.1 0.31 ± 20% perf-profile.children.cycles-pp.reweight_entity 0.19 ± 4% +0.1 0.27 ± 13% perf-profile.children.cycles-pp.pick_next_entity 0.23 ± 2% +0.1 0.30 ± 16% perf-profile.children.cycles-pp.check_preempt_curr 0.27 ± 4% +0.1 0.35 ± 14% perf-profile.children.cycles-pp.native_write_msr 0.11 ± 11% +0.1 0.19 ± 21% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.11 ± 3% +0.1 0.20 ± 25% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.38 +0.1 0.46 ± 9% perf-profile.children.cycles-pp.___might_sleep 0.19 ± 3% +0.1 0.28 ± 22% perf-profile.children.cycles-pp.__fdget_pos 0.13 ± 9% +0.1 0.22 ± 15% perf-profile.children.cycles-pp.clockevents_program_event 0.13 ± 6% +0.1 0.23 ± 25% perf-profile.children.cycles-pp.__might_sleep 0.53 ± 3% +0.1 0.63 ± 11% perf-profile.children.cycles-pp.update_rq_clock 0.19 ± 5% +0.1 0.29 ± 17% perf-profile.children.cycles-pp.read_tsc 0.26 ± 2% +0.1 0.36 ± 17% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.26 ± 3% +0.1 0.36 ± 22% perf-profile.children.cycles-pp.common_file_perm 0.18 ± 2% +0.1 0.28 ± 16% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.09 ± 8% +0.1 0.19 ± 28% perf-profile.children.cycles-pp.put_prev_task_fair 0.16 ± 4% +0.1 0.26 ± 32% perf-profile.children.cycles-pp.switch_fpu_return 0.16 ± 2% +0.1 0.28 ± 30% perf-profile.children.cycles-pp.___perf_sw_event 0.25 +0.1 0.38 ± 20% perf-profile.children.cycles-pp.native_sched_clock 0.19 ± 4% +0.1 0.32 ± 31% perf-profile.children.cycles-pp.orc_find 0.34 +0.1 0.47 ± 16% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.29 ± 2% +0.1 0.43 ± 21% perf-profile.children.cycles-pp.__orc_find 0.26 +0.1 0.40 ± 20% perf-profile.children.cycles-pp.sched_clock 0.24 ± 3% +0.1 0.38 ± 27% perf-profile.children.cycles-pp.finish_task_switch 0.23 ± 7% +0.1 0.37 ± 22% perf-profile.children.cycles-pp.kernel_text_address 0.28 ± 3% +0.1 0.42 ± 22% perf-profile.children.cycles-pp.__update_load_avg_se 0.74 +0.1 0.89 ± 6% perf-profile.children.cycles-pp.mutex_lock 0.23 +0.1 0.38 ± 26% perf-profile.children.cycles-pp.copy_page_from_iter 0.56 ± 5% +0.2 0.71 ± 11% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.28 +0.2 0.43 ± 21% perf-profile.children.cycles-pp.sched_clock_cpu 0.45 ± 3% +0.2 0.60 ± 13% perf-profile.children.cycles-pp.copy_page_to_iter 0.63 +0.2 0.79 ± 13% perf-profile.children.cycles-pp.prepare_to_wait_event 0.52 ± 4% +0.2 0.69 ± 19% perf-profile.children.cycles-pp.__switch_to 0.39 ± 2% +0.2 0.55 ± 18% perf-profile.children.cycles-pp.__switch_to_asm 0.35 ± 3% +0.2 0.52 ± 23% perf-profile.children.cycles-pp.security_file_permission 0.24 +0.2 0.41 ± 32% perf-profile.children.cycles-pp._find_next_bit 1.16 ± 5% +0.2 1.33 ± 5% perf-profile.children.cycles-pp.apic_timer_interrupt 0.35 ± 4% +0.2 0.53 ± 28% perf-profile.children.cycles-pp.update_cfs_group 0.28 ± 5% +0.2 0.45 ± 24% perf-profile.children.cycles-pp.__kernel_text_address 0.44 +0.2 0.65 ± 23% perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched 0.16 ± 2% +0.2 0.38 ± 51% perf-profile.children.cycles-pp.available_idle_cpu 0.33 ± 3% +0.2 0.55 ± 26% perf-profile.children.cycles-pp.unwind_get_return_address 0.43 ± 4% +0.2 0.67 ± 24% perf-profile.children.cycles-pp.__next_timer_interrupt 0.35 ± 4% +0.3 0.61 ± 13% perf-profile.children.cycles-pp.ktime_get 0.60 ± 2% +0.3 0.88 ± 16% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.40 ± 2% +0.3 0.69 ± 28% perf-profile.children.cycles-pp.__unwind_start 0.85 +0.3 1.17 ± 19% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.65 ± 3% +0.3 0.97 ± 22% perf-profile.children.cycles-pp.get_next_timer_interrupt 1.18 +0.3 1.50 ± 12% perf-profile.children.cycles-pp.set_next_entity 0.94 +0.3 1.28 ± 17% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.79 ± 3% +0.4 1.15 ± 21% perf-profile.children.cycles-pp.tick_nohz_next_event 1.03 ± 2% +0.4 1.45 ± 21% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.74 ± 2% +0.4 1.17 ± 24% perf-profile.children.cycles-pp.update_curr 0.59 ± 2% +0.5 1.13 ± 37% perf-profile.children.cycles-pp.select_task_rq_fair 1.77 +0.6 2.33 ± 13% perf-profile.children.cycles-pp.pick_next_task_fair 1.62 ± 3% +0.7 2.33 ± 21% perf-profile.children.cycles-pp.menu_select 1.73 +0.8 2.57 ± 22% perf-profile.children.cycles-pp.unwind_next_frame 1.52 ± 2% +0.9 2.44 ± 24% perf-profile.children.cycles-pp.update_load_avg 1.51 +0.9 2.46 ± 25% perf-profile.children.cycles-pp.dequeue_entity 1.46 ± 6% +1.1 2.59 ± 9% perf-profile.children.cycles-pp.read 1.76 +1.1 2.91 ± 25% perf-profile.children.cycles-pp.dequeue_task_fair 2.76 +1.4 4.17 ± 23% perf-profile.children.cycles-pp.arch_stack_walk 0.14 ± 5% +1.4 1.59 ± 85% perf-profile.children.cycles-pp.poll_idle 3.35 +1.5 4.84 ± 21% perf-profile.children.cycles-pp.stack_trace_save_tsk 2.80 +1.6 4.37 ± 28% perf-profile.children.cycles-pp.schedule_idle 0.55 ± 2% +1.8 2.33 ± 71% perf-profile.children.cycles-pp._raw_spin_lock 3.29 +2.0 5.28 ± 26% perf-profile.children.cycles-pp.schedule 5.81 +2.7 8.51 ± 21% perf-profile.children.cycles-pp.pipe_read 5.91 +2.8 8.68 ± 22% perf-profile.children.cycles-pp.new_sync_read 6.38 +3.0 9.40 ± 21% perf-profile.children.cycles-pp.vfs_read 6.56 +3.1 9.63 ± 21% perf-profile.children.cycles-pp.ksys_read 5.99 +3.5 9.51 ± 27% perf-profile.children.cycles-pp.__sched_text_start 30.51 -6.9 23.59 ± 19% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 39.93 -6.0 33.94 ± 13% perf-profile.self.cycles-pp.intel_idle 0.05 +0.0 0.07 ± 13% perf-profile.self.cycles-pp.tick_nohz_next_event 0.08 ± 5% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.check_preempt_curr 0.23 +0.0 0.26 ± 4% perf-profile.self.cycles-pp.nr_iowait_cpu 0.08 +0.0 0.11 ± 19% perf-profile.self.cycles-pp.in_sched_functions 0.06 ± 6% +0.0 0.09 ± 16% perf-profile.self.cycles-pp.schedule 0.05 +0.0 0.08 ± 20% perf-profile.self.cycles-pp.generic_pipe_buf_confirm 0.05 +0.0 0.08 ± 30% perf-profile.self.cycles-pp.__kernel_text_address 0.05 ± 8% +0.0 0.08 ± 27% perf-profile.self.cycles-pp.put_task_stack 0.12 ± 4% +0.0 0.16 ± 7% perf-profile.self.cycles-pp.cpus_share_cache 0.08 ± 5% +0.0 0.11 ± 23% perf-profile.self.cycles-pp.stack_access_ok 0.08 ± 5% +0.0 0.11 ± 17% perf-profile.self.cycles-pp.ksys_read 0.06 ± 11% +0.0 0.09 ± 23% perf-profile.self.cycles-pp.kernel_text_address 0.06 ± 6% +0.0 0.10 ± 21% perf-profile.self.cycles-pp.menu_reflect 0.04 ± 58% +0.0 0.08 ± 19% perf-profile.self.cycles-pp.copy_page_from_iter 0.20 ± 4% +0.0 0.24 ± 9% perf-profile.self.cycles-pp.newidle_balance 0.12 ± 6% +0.0 0.16 ± 14% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.05 ± 8% +0.0 0.10 ± 39% perf-profile.self.cycles-pp.unwind_get_return_address 0.10 ± 4% +0.0 0.15 ± 27% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.08 ± 8% +0.0 0.13 ± 24% perf-profile.self.cycles-pp.__fsnotify_parent 0.01 ±173% +0.0 0.06 ± 20% perf-profile.self.cycles-pp.security_file_permission 0.05 ± 9% +0.1 0.10 ± 35% perf-profile.self.cycles-pp.__x86_indirect_thunk_rbx 0.18 ± 2% +0.1 0.24 ± 15% perf-profile.self.cycles-pp.prepare_to_wait_event 0.04 ± 57% +0.1 0.09 ± 27% perf-profile.self.cycles-pp.put_prev_entity 0.09 ± 5% +0.1 0.14 ± 19% perf-profile.self.cycles-pp.copy_page_to_iter 0.05 ± 8% +0.1 0.11 ± 40% perf-profile.self.cycles-pp.pick_next_task_fair 0.01 ±173% +0.1 0.07 ± 19% perf-profile.self.cycles-pp.sched_ttwu_pending 0.00 +0.1 0.06 ± 15% perf-profile.self.cycles-pp.bpf_prog_kallsyms_find 0.00 +0.1 0.06 ± 15% perf-profile.self.cycles-pp.get_cpu_device 0.23 ± 3% +0.1 0.29 ± 14% perf-profile.self.cycles-pp.reweight_entity 0.20 ± 2% +0.1 0.25 ± 3% perf-profile.self.cycles-pp.__list_del_entry_valid 0.18 ± 3% +0.1 0.24 ± 9% perf-profile.self.cycles-pp.pick_next_entity 0.00 +0.1 0.06 ± 22% perf-profile.self.cycles-pp.activate_task 0.00 +0.1 0.06 ± 14% perf-profile.self.cycles-pp.__is_insn_slot_addr 0.08 ± 8% +0.1 0.14 ± 32% perf-profile.self.cycles-pp.arch_stack_walk 0.00 +0.1 0.06 ± 28% perf-profile.self.cycles-pp.__x64_sys_write 0.23 +0.1 0.29 ± 10% perf-profile.self.cycles-pp.mutex_unlock 0.11 ± 7% +0.1 0.18 ± 25% perf-profile.self.cycles-pp.fsnotify 0.30 ± 2% +0.1 0.36 ± 16% perf-profile.self.cycles-pp.try_to_wake_up 0.11 ± 4% +0.1 0.17 ± 23% perf-profile.self.cycles-pp.rcu_eqs_enter 0.00 +0.1 0.06 ± 17% perf-profile.self.cycles-pp.cpuidle_governor_latency_req 0.17 ± 2% +0.1 0.24 ± 24% perf-profile.self.cycles-pp.__unwind_start 0.14 ± 3% +0.1 0.21 ± 21% perf-profile.self.cycles-pp.resched_curr 0.09 ± 7% +0.1 0.15 ± 33% perf-profile.self.cycles-pp.new_sync_read 0.00 +0.1 0.07 ± 31% perf-profile.self.cycles-pp.timestamp_truncate 0.17 ± 2% +0.1 0.24 ± 17% perf-profile.self.cycles-pp.__account_scheduler_latency 0.00 +0.1 0.07 ± 12% perf-profile.self.cycles-pp.ksys_write 0.11 ± 6% +0.1 0.18 ± 24% perf-profile.self.cycles-pp.account_entity_dequeue 0.31 ± 6% +0.1 0.38 ± 11% perf-profile.self.cycles-pp.copy_user_generic_unrolled 0.00 +0.1 0.07 ± 39% perf-profile.self.cycles-pp.__update_idle_core 0.39 ± 4% +0.1 0.46 ± 8% perf-profile.self.cycles-pp.mutex_lock 0.17 ± 2% +0.1 0.24 ± 23% perf-profile.self.cycles-pp.__fget_light 0.11 ± 4% +0.1 0.18 ± 24% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.12 ± 10% +0.1 0.20 ± 25% perf-profile.self.cycles-pp.__might_sleep 0.08 ± 8% +0.1 0.16 ± 28% perf-profile.self.cycles-pp.dequeue_entity 0.00 +0.1 0.08 ± 34% perf-profile.self.cycles-pp.put_prev_task_fair 0.37 +0.1 0.46 ± 9% perf-profile.self.cycles-pp.___might_sleep 0.15 ± 3% +0.1 0.24 ± 17% perf-profile.self.cycles-pp.vfs_read 0.26 ± 4% +0.1 0.35 ± 14% perf-profile.self.cycles-pp.native_write_msr 0.23 ± 2% +0.1 0.31 ± 21% perf-profile.self.cycles-pp.common_file_perm 0.20 ± 2% +0.1 0.28 ± 24% perf-profile.self.cycles-pp.finish_task_switch 0.18 ± 4% +0.1 0.27 ± 16% perf-profile.self.cycles-pp.read_tsc 0.17 ± 8% +0.1 0.26 ± 22% perf-profile.self.cycles-pp.__next_timer_interrupt 0.15 ± 4% +0.1 0.25 ± 31% perf-profile.self.cycles-pp.switch_fpu_return 0.42 +0.1 0.53 ± 17% perf-profile.self.cycles-pp.do_idle 0.12 ± 4% +0.1 0.24 ± 31% perf-profile.self.cycles-pp.___perf_sw_event 0.38 ± 2% +0.1 0.49 ± 16% perf-profile.self.cycles-pp.pipe_read 0.27 ± 3% +0.1 0.39 ± 19% perf-profile.self.cycles-pp.enqueue_task_fair 0.18 ± 4% +0.1 0.31 ± 29% perf-profile.self.cycles-pp.orc_find 0.24 +0.1 0.37 ± 19% perf-profile.self.cycles-pp.native_sched_clock 0.28 ± 2% +0.1 0.41 ± 21% perf-profile.self.cycles-pp.__orc_find 0.30 +0.1 0.44 ± 24% perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched 0.21 ± 2% +0.1 0.34 ± 22% perf-profile.self.cycles-pp.dequeue_task_fair 0.38 ± 3% +0.1 0.52 ± 17% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.27 ± 3% +0.1 0.41 ± 22% perf-profile.self.cycles-pp.__update_load_avg_se 0.48 ± 4% +0.2 0.64 ± 18% perf-profile.self.cycles-pp.__switch_to 0.74 +0.2 0.91 ± 11% perf-profile.self.cycles-pp.enqueue_entity 0.39 ± 2% +0.2 0.55 ± 19% perf-profile.self.cycles-pp.__switch_to_asm 0.23 +0.2 0.40 ± 32% perf-profile.self.cycles-pp._find_next_bit 0.16 ± 14% +0.2 0.34 ± 13% perf-profile.self.cycles-pp.ktime_get 0.37 ± 3% +0.2 0.55 ± 20% perf-profile.self.cycles-pp._raw_spin_lock 0.44 ± 6% +0.2 0.62 ± 22% perf-profile.self.cycles-pp.menu_select 0.28 ± 4% +0.2 0.46 ± 37% perf-profile.self.cycles-pp.update_cfs_group 0.16 ± 2% +0.2 0.37 ± 51% perf-profile.self.cycles-pp.available_idle_cpu 1.55 +0.2 1.76 ± 9% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.30 ± 3% +0.2 0.53 ± 28% perf-profile.self.cycles-pp.select_task_rq_fair 1.09 ± 2% +0.2 1.32 ± 15% perf-profile.self.cycles-pp.__sched_text_start 0.59 ± 3% +0.3 0.86 ± 17% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.83 +0.3 1.14 ± 19% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.94 ± 2% +0.3 1.27 ± 17% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.55 ± 2% +0.3 0.90 ± 24% perf-profile.self.cycles-pp.update_curr 0.62 ± 2% +0.4 1.07 ± 29% perf-profile.self.cycles-pp.update_load_avg 1.15 +0.5 1.68 ± 21% perf-profile.self.cycles-pp.unwind_next_frame 1.58 +0.8 2.35 ± 22% perf-profile.self.cycles-pp.do_syscall_64 0.09 ± 5% +1.4 1.50 ± 89% perf-profile.self.cycles-pp.poll_idle *************************************************************************************************** lkp-hsw-d01: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/target/tbox_group/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/50%/debian-x86_64-20191114.cgz/300s/vmlinux_prereq/lkp-hsw-d01/kbuild/0x27 commit: 9c40365a65 ("threads: Update PID limit comment according to futex UAPI change") 59901cb452 ("sched/fair: Don't pull task if local group is more loaded than busiest group") 9c40365a65d62d7c 59901cb4520c44bfce81f523bc6 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.BUG:Bad_page_map_in_process 1:4 -25% :4 dmesg.BUG:Bad_rss-counter_state_mm:#type:MM_ANONPAGES_val 1:4 -25% :4 dmesg.BUG:Bad_rss-counter_state_mm:#type:MM_SWAPENTS_val :4 25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception :4 25% 1:4 dmesg.RIP:__d_lookup_rcu :4 25% 1:4 dmesg.canonical_address#:#[##] 1:4 -25% :4 kmsg.file:libuuid.so.#fault:filemap_fault_mmap:generic_file_mmap_readpage:simple_readpage 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a07fffffa0ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a17fffffa1ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a27fffffa2ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a37fffffa3ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a47fffffa4ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a57fffffa5ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a67fffffa6ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a77fffffa7ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff8aff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a7fffff9aff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a87fffffa8ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#a97fffffa9ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#aa7fffffaaff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ab7fffffabff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ac7fffffacff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ad7fffffadff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ae7fffffaeff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#af7fffffafff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b07fffffb0ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b17fffffb1ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b27fffffb2ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b37fffffb3ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b47fffffb4ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b57fffffb5ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b67fffffb6ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b77fffffb7ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff8bff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b7fffff9bff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b87fffffb8ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#b97fffffb9ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ba7fffffbaff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bb7fffffbbff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bc7fffffbcff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bd7fffffbdff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#be7fffffbeff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#bf7fffffbfff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c07fffffc0ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c17fffffc1ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c27fffffc2ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c37fffffc3ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c47fffffc4ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c57fffffc5ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c67fffffc6ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c77fffffc7ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff8cff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c7fffff9cff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c87fffffc8ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#c97fffffc9ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ca7fffffcaff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cb7fffffcbff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cc7fffffccff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cd7fffffcdff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ce7fffffceff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#cf7fffffcfff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d07fffffd0ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d17fffffd1ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d27fffffd2ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d37fffffd3ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d47fffffd4ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d57fffffd5ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d67fffffd6ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d77fffffd7ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff8dff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d7fffff9dff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d87fffffd8ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#d97fffffd9ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#da7fffffdaff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#db7fffffdbff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#dc7fffffdcff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#dd7fffffddff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#de7fffffdeff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#df7fffffdfff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e07fffffe0ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e17fffffe1ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e27fffffe2ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e37fffffe3ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e47fffffe4ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e57fffffe5ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e67fffffe6ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e77fffffe7ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff8eff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e7fffff9eff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e87fffffe8ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#e97fffffe9ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ea7fffffeaff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#eb7fffffebff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ec7fffffecff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ed7fffffedff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ee7fffffeeff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ef7fffffefff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f07ffffff0ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f17ffffff1ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f27ffffff2ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f37ffffff3ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f47ffffff4ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f57ffffff5ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f67ffffff6ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f77ffffff7ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff8fff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f7fffff9fff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f87ffffff8ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#f97ffffff9ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fa7ffffffaff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fb7ffffffbff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fc7ffffffcff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fd7ffffffdff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fe7ffffffeff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#ff7fffffffff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff80ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff81ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff82ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff83ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff84ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff85ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff86ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff87ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff88ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff89ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff90ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff91ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff92ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff93ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff94ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff95ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff96ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff97ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff98ff 1:4 -25% :4 kmsg.swap_info_get:Bad_swap_file_entry#fffff99ff 0:4 0% 0:4 perf-profile.children.cycles-pp.warning_at 0:4 3% 0:4 perf-profile.children.cycles-pp.warning 0:4 4% 1:4 perf-profile.children.cycles-pp.error_entry 0:4 4% 0:4 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 173.17 +7.5% 186.17 kbuild.buildtime_per_iteration 346.13 +7.5% 372.13 kbuild.time.elapsed_time 346.13 +7.5% 372.13 kbuild.time.elapsed_time.max 62328 -4.4% 59613 kbuild.time.involuntary_context_switches 119.99 +3.7% 124.42 kbuild.time.system_time 1258 +7.8% 1357 kbuild.time.user_time 3976 ± 2% -5.2% 3768 ± 2% proc-vmstat.pgactivate 8581 -7.6% 7925 vmstat.system.cs 29.08 ± 6% +27.4% 37.06 ± 21% boot-time.boot 190.33 ± 5% +31.2% 249.80 ± 20% boot-time.idle 2.474e+08 ± 62% +85.2% 4.58e+08 ± 21% cpuidle.C6.time 361421 ± 60% +70.5% 616268 ± 5% cpuidle.C6.usage 2265 ± 6% -16.4% 1895 ± 10% slabinfo.cred_jar.active_objs 2301 ± 5% -15.3% 1950 ± 9% slabinfo.cred_jar.num_objs 793.33 +14.3% 907.00 ± 4% slabinfo.task_delay_info.active_objs 793.33 +14.3% 907.00 ± 4% slabinfo.task_delay_info.num_objs 249422 +10.7% 276098 ± 4% softirqs.CPU0.RCU 138168 +30.3% 180101 ± 12% softirqs.CPU0.TIMER 139666 +9.2% 152539 ± 2% softirqs.CPU2.TIMER 138194 ± 2% +11.1% 153534 ± 2% softirqs.CPU4.TIMER 139323 ± 2% +11.9% 155966 softirqs.CPU7.TIMER 1153221 +10.4% 1273097 softirqs.TIMER 267.67 ± 15% -37.9% 166.33 ± 9% interrupts.CPU0.CAL:Function_call_interrupts 3697 ± 14% -17.8% 3040 ± 9% interrupts.CPU0.RES:Rescheduling_interrupts 245.00 ± 19% -48.6% 126.00 ± 8% interrupts.CPU0.TLB:TLB_shootdowns 188.67 ± 19% -39.2% 114.67 ± 22% interrupts.CPU1.TLB:TLB_shootdowns 292.33 ± 9% -13.1% 254.00 ± 12% interrupts.CPU2.CAL:Function_call_interrupts 241.67 ± 23% -31.6% 165.33 ± 7% interrupts.CPU2.TLB:TLB_shootdowns 7195 ± 19% -37.5% 4497 ± 29% interrupts.CPU4.NMI:Non-maskable_interrupts 7195 ± 19% -37.5% 4497 ± 29% interrupts.CPU4.PMI:Performance_monitoring_interrupts 1361 ± 5% -16.8% 1132 ± 10% interrupts.TLB:TLB_shootdowns 0.00 +1.4e+11% 1382 ±114% sched_debug.cfs_rq:/.MIN_vruntime.avg 0.00 +1.1e+12% 11058 ±114% sched_debug.cfs_rq:/.MIN_vruntime.max 77852 +20.1% 93480 sched_debug.cfs_rq:/.exec_clock.avg 82443 +20.3% 99210 sched_debug.cfs_rq:/.exec_clock.max 72993 +18.4% 86426 sched_debug.cfs_rq:/.exec_clock.min 2882 ± 6% +33.9% 3858 ± 9% sched_debug.cfs_rq:/.exec_clock.stddev 119503 ± 2% +20.2% 143687 ± 8% sched_debug.cfs_rq:/.load.avg 0.00 +1.4e+11% 1382 ±114% sched_debug.cfs_rq:/.max_vruntime.avg 0.00 +1.1e+12% 11058 ±114% sched_debug.cfs_rq:/.max_vruntime.max 463036 +18.8% 549986 sched_debug.cfs_rq:/.min_vruntime.avg 489987 +18.7% 581626 sched_debug.cfs_rq:/.min_vruntime.max 434924 +17.3% 510306 sched_debug.cfs_rq:/.min_vruntime.min 17242 +24.4% 21446 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev 0.68 ± 6% +18.1% 0.80 ± 3% sched_debug.cfs_rq:/.nr_running.avg 0.44 ± 3% -18.2% 0.36 ± 5% sched_debug.cfs_rq:/.nr_running.stddev 17242 +24.4% 21446 ± 6% sched_debug.cfs_rq:/.spread0.stddev 180483 +21.0% 218362 ± 3% sched_debug.cpu.clock.avg 180483 +21.0% 218363 ± 3% sched_debug.cpu.clock.max 180482 +21.0% 218361 ± 3% sched_debug.cpu.clock.min 180483 +21.0% 218362 ± 3% sched_debug.cpu.clock_task.avg 180483 +21.0% 218363 ± 3% sched_debug.cpu.clock_task.max 180482 +21.0% 218361 ± 3% sched_debug.cpu.clock_task.min 9075 ± 6% +37.6% 12486 sched_debug.cpu.curr->pid.avg 13910 +37.2% 19088 sched_debug.cpu.curr->pid.max 6416 ± 2% +37.6% 8831 sched_debug.cpu.curr->pid.stddev 1.83 ± 19% -24.7% 1.38 ± 12% sched_debug.cpu.nr_running.max 0.66 ± 18% -30.6% 0.46 ± 4% sched_debug.cpu.nr_running.stddev 22.33 ± 28% -53.9% 10.29 ± 11% sched_debug.cpu.nr_uninterruptible.max 10.96 ± 9% -35.3% 7.10 ± 7% sched_debug.cpu.nr_uninterruptible.stddev 180482 +21.0% 218362 ± 3% sched_debug.cpu_clk 179957 +21.0% 217836 ± 3% sched_debug.ktime 181437 +20.8% 219109 ± 3% sched_debug.sched_clk 13.61 +6.4% 14.47 perf-stat.i.MPKI 3.762e+09 -7.5% 3.478e+09 perf-stat.i.branch-instructions 3.00 +0.1 3.08 perf-stat.i.branch-miss-rate% 1.126e+08 -5.5% 1.064e+08 perf-stat.i.branch-misses 8.73 -0.6 8.17 ± 2% perf-stat.i.cache-miss-rate% 20492578 -10.3% 18373371 perf-stat.i.cache-misses 2.365e+08 -2.6% 2.303e+08 perf-stat.i.cache-references 8639 -7.7% 7977 perf-stat.i.context-switches 0.92 +9.1% 1.00 ± 2% perf-stat.i.cpi 1.582e+10 -1.2% 1.562e+10 perf-stat.i.cpu-cycles 182.40 -11.9% 160.76 perf-stat.i.cpu-migrations 799.50 +9.7% 877.23 perf-stat.i.cycles-between-cache-misses 0.31 +0.0 0.33 perf-stat.i.dTLB-load-miss-rate% 14810335 -4.5% 14149852 perf-stat.i.dTLB-load-misses 4.694e+09 -7.5% 4.341e+09 perf-stat.i.dTLB-loads 2.55e+09 -7.0% 2.373e+09 perf-stat.i.dTLB-stores 4617509 +8.3% 4998953 ± 2% perf-stat.i.iTLB-load-misses 17720319 +7.6% 19060873 perf-stat.i.iTLB-loads 1.731e+10 -7.5% 1.601e+10 perf-stat.i.instructions 3841 -13.1% 3339 ± 2% perf-stat.i.instructions-per-iTLB-miss 1.09 -6.6% 1.02 perf-stat.i.ipc 164253 -7.3% 152206 perf-stat.i.minor-faults 13452948 -11.3% 11934691 perf-stat.i.node-loads 1875909 ± 3% -11.5% 1659475 ± 3% perf-stat.i.node-stores 164253 -7.3% 152206 perf-stat.i.page-faults 13.66 +5.3% 14.38 perf-stat.overall.MPKI 2.99 +0.1 3.06 perf-stat.overall.branch-miss-rate% 8.67 -0.7 7.98 perf-stat.overall.cache-miss-rate% 0.91 +6.8% 0.98 perf-stat.overall.cpi 771.90 +10.2% 850.28 perf-stat.overall.cycles-between-cache-misses 0.31 +0.0 0.32 perf-stat.overall.dTLB-load-miss-rate% 3749 -14.5% 3205 ± 3% perf-stat.overall.instructions-per-iTLB-miss 1.09 -6.4% 1.02 perf-stat.overall.ipc 3.751e+09 -7.5% 3.469e+09 perf-stat.ps.branch-instructions 1.123e+08 -5.5% 1.062e+08 perf-stat.ps.branch-misses 20432412 -10.3% 18323590 perf-stat.ps.cache-misses 2.358e+08 -2.6% 2.297e+08 perf-stat.ps.cache-references 8613 -7.6% 7955 perf-stat.ps.context-switches 1.577e+10 -1.2% 1.558e+10 perf-stat.ps.cpu-cycles 181.89 -11.9% 160.32 perf-stat.ps.cpu-migrations 14766724 -4.4% 14111258 perf-stat.ps.dTLB-load-misses 4.68e+09 -7.5% 4.33e+09 perf-stat.ps.dTLB-loads 2.543e+09 -6.9% 2.366e+09 perf-stat.ps.dTLB-stores 4603933 +8.3% 4985424 ± 2% perf-stat.ps.iTLB-load-misses 17668225 +7.6% 19009302 perf-stat.ps.iTLB-loads 1.726e+10 -7.5% 1.597e+10 perf-stat.ps.instructions 163774 -7.3% 151790 perf-stat.ps.minor-faults 13413384 -11.3% 11902353 perf-stat.ps.node-loads 1870377 ± 3% -11.5% 1654965 ± 3% perf-stat.ps.node-stores 163774 -7.3% 151790 perf-stat.ps.page-faults 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.write._fini 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp._fini 4.07 ± 13% -1.8 2.30 ± 21% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write._fini 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write._fini 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.devkmsg_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.new_sync_write.vfs_write.ksys_write 3.46 ± 14% -1.4 2.03 ± 21% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write 3.27 ± 13% -1.4 1.90 ± 20% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit 3.20 ± 14% -1.1 2.08 ± 29% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock 3.20 ± 14% -1.1 2.08 ± 29% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit 2.16 ± 14% -0.9 1.27 ± 18% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write 1.02 ± 14% -0.6 0.45 ± 72% perf-profile.calltrace.cycles-pp.delay_tsc.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write 1.24 ± 2% -0.3 0.95 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 1.24 ± 2% -0.3 0.95 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.56 ± 3% -0.2 0.36 ± 70% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.84 ± 8% -0.2 0.66 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.84 ± 8% -0.2 0.66 ± 3% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.84 ± 8% -0.2 0.66 ± 3% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.80 ± 8% -0.2 0.63 ± 3% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.80 ± 8% -0.2 0.63 ± 3% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 0.79 ± 3% -0.1 0.71 ± 6% perf-profile.calltrace.cycles-pp.read 0.73 ± 3% -0.1 0.65 ± 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 0.73 ± 3% -0.1 0.65 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.63 ± 4% -0.1 0.56 ± 6% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.98 ± 3% -0.1 0.91 ± 6% perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.65 ± 4% -0.1 0.58 ± 4% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.58 ± 2% +0.1 0.67 ± 7% perf-profile.calltrace.cycles-pp.wide_int_to_tree 0.66 ± 11% +0.1 0.78 ± 11% perf-profile.calltrace.cycles-pp.c_lex_with_flags 0.65 ± 6% +0.2 0.81 ± 7% perf-profile.calltrace.cycles-pp.linemap_lookup 0.34 ± 70% +0.2 0.54 ± 2% perf-profile.calltrace.cycles-pp.pop_scope 2.11 ± 5% +0.2 2.36 ± 3% perf-profile.calltrace.cycles-pp._cpp_lex_direct 1.14 ± 2% +0.3 1.44 ± 11% perf-profile.calltrace.cycles-pp.memchr 8.11 ± 5% -2.4 5.75 ± 11% perf-profile.children.cycles-pp.do_syscall_64 8.13 ± 5% -2.4 5.77 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 4.19 ± 12% -1.8 2.36 ± 20% perf-profile.children.cycles-pp.ksys_write 4.21 ± 12% -1.8 2.38 ± 20% perf-profile.children.cycles-pp.write 4.18 ± 12% -1.8 2.36 ± 20% perf-profile.children.cycles-pp.vfs_write 4.17 ± 13% -1.8 2.36 ± 20% perf-profile.children.cycles-pp.new_sync_write 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.children.cycles-pp._fini 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.children.cycles-pp.devkmsg_write 4.07 ± 13% -1.8 2.30 ± 20% perf-profile.children.cycles-pp.devkmsg_emit 4.65 ± 13% -1.5 3.13 ± 20% perf-profile.children.cycles-pp.vprintk_emit 4.73 ± 14% -1.5 3.25 ± 16% perf-profile.children.cycles-pp.console_unlock 4.09 ± 15% -1.2 2.94 ± 16% perf-profile.children.cycles-pp.serial8250_console_write 3.99 ± 15% -1.1 2.88 ± 16% perf-profile.children.cycles-pp.wait_for_xmitr 3.89 ± 15% -1.1 2.80 ± 16% perf-profile.children.cycles-pp.uart_console_write 3.80 ± 15% -1.0 2.75 ± 15% perf-profile.children.cycles-pp.serial8250_console_putchar 2.71 ± 15% -0.8 1.94 ± 15% perf-profile.children.cycles-pp.io_serial_in 1.27 ± 15% -0.3 0.94 ± 18% perf-profile.children.cycles-pp.delay_tsc 0.63 ± 7% -0.3 0.31 ± 18% perf-profile.children.cycles-pp.vt_console_print 0.62 ± 9% -0.3 0.31 ± 18% perf-profile.children.cycles-pp.lf 0.62 ± 9% -0.3 0.31 ± 18% perf-profile.children.cycles-pp.con_scroll 0.62 ± 9% -0.3 0.31 ± 18% perf-profile.children.cycles-pp.fbcon_scroll 0.61 ± 8% -0.3 0.30 ± 17% perf-profile.children.cycles-pp.fbcon_redraw 0.59 ± 7% -0.3 0.29 ± 18% perf-profile.children.cycles-pp.fbcon_putcs 0.59 ± 7% -0.3 0.29 ± 18% perf-profile.children.cycles-pp.bit_putcs 0.50 ± 4% -0.3 0.24 ± 22% perf-profile.children.cycles-pp.drm_fb_helper_cfb_imageblit 0.49 ± 3% -0.2 0.24 ± 22% perf-profile.children.cycles-pp.cfb_imageblit 0.86 ± 7% -0.2 0.66 ± 3% perf-profile.children.cycles-pp.mmput 0.84 ± 8% -0.2 0.66 ± 3% perf-profile.children.cycles-pp.__x64_sys_exit_group 0.84 ± 8% -0.2 0.66 ± 3% perf-profile.children.cycles-pp.do_group_exit 0.84 ± 8% -0.2 0.66 ± 3% perf-profile.children.cycles-pp.do_exit 0.85 ± 7% -0.2 0.66 ± 3% perf-profile.children.cycles-pp.exit_mmap 0.62 ± 6% -0.2 0.47 ± 3% perf-profile.children.cycles-pp.unmap_vmas 0.61 ± 7% -0.1 0.47 ± 3% perf-profile.children.cycles-pp.unmap_page_range 0.52 ± 11% -0.1 0.41 ± 7% perf-profile.children.cycles-pp.tlb_flush_mmu 0.84 ± 3% -0.1 0.73 ± 6% perf-profile.children.cycles-pp.read 0.92 ± 7% -0.1 0.82 ± 5% perf-profile.children.cycles-pp.do_sys_open 0.70 ± 5% -0.1 0.61 ± 4% perf-profile.children.cycles-pp.ksys_read 0.90 ± 7% -0.1 0.81 ± 5% perf-profile.children.cycles-pp.do_sys_openat2 0.69 ± 7% -0.1 0.59 ± 5% perf-profile.children.cycles-pp.vfs_read 1.25 -0.1 1.16 ± 4% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.49 ± 9% -0.1 0.41 ± 7% perf-profile.children.cycles-pp.release_pages 0.45 -0.1 0.37 ± 11% perf-profile.children.cycles-pp.bitmap_clear_bit 0.22 ± 9% -0.1 0.14 ± 11% perf-profile.children.cycles-pp.execve 0.24 ± 5% -0.1 0.17 ± 12% perf-profile.children.cycles-pp.__x64_sys_execve 0.24 ± 5% -0.1 0.17 ± 12% perf-profile.children.cycles-pp.__do_execve_file 0.20 ± 8% -0.1 0.13 ± 24% perf-profile.children.cycles-pp.statistics_fini_pass 0.10 ± 16% -0.1 0.04 ± 70% perf-profile.children.cycles-pp.__libc_fork 0.20 ± 8% -0.1 0.14 ± 17% perf-profile.children.cycles-pp.filemap_map_pages 0.32 ± 7% -0.1 0.26 ± 9% perf-profile.children.cycles-pp.__list_del_entry_valid 0.29 ± 9% -0.1 0.24 ± 5% perf-profile.children.cycles-pp.irq_exit 0.27 ± 7% -0.1 0.21 ± 17% perf-profile.children.cycles-pp.bitmap_bit_p 0.12 ± 6% -0.1 0.07 ± 25% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.20 ± 8% -0.0 0.16 ± 13% perf-profile.children.cycles-pp.side_effects_p 0.12 ± 25% -0.0 0.08 ± 30% perf-profile.children.cycles-pp.alloc_empty_file 0.09 ± 13% -0.0 0.05 ± 72% perf-profile.children.cycles-pp.io_serial_out 0.17 ± 10% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.note_stores 0.18 ± 10% -0.0 0.13 ± 16% perf-profile.children.cycles-pp.bitmap_copy 0.14 ± 8% -0.0 0.10 ± 24% perf-profile.children.cycles-pp.kmem_cache_alloc 0.24 ± 7% -0.0 0.19 ± 14% perf-profile.children.cycles-pp.rcu_core 0.14 ± 14% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.load_elf_binary 0.09 ± 24% -0.0 0.05 ± 78% perf-profile.children.cycles-pp.copy_page 0.25 ± 5% -0.0 0.20 ± 6% perf-profile.children.cycles-pp.init_alias_analysis 0.12 ± 14% -0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_unreachable_blocks 0.16 ± 10% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.14 ± 14% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.search_binary_handler 0.11 ± 25% -0.0 0.07 ± 17% perf-profile.children.cycles-pp.ret_from_fork 0.11 ± 27% -0.0 0.07 ± 35% perf-profile.children.cycles-pp.__alloc_file 0.09 ± 14% -0.0 0.05 ± 72% perf-profile.children.cycles-pp.lra_inheritance 0.09 ± 5% -0.0 0.05 ± 72% perf-profile.children.cycles-pp.page_remove_rmap 0.19 ± 11% -0.0 0.15 ± 16% perf-profile.children.cycles-pp.get_attribute_name 0.12 ± 10% -0.0 0.08 ± 32% perf-profile.children.cycles-pp.for_each_inc_dec 0.15 ± 8% -0.0 0.11 ± 22% perf-profile.children.cycles-pp.update_stmt_operands 0.07 ± 6% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.wrapup_global_declaration_2 0.18 ± 9% -0.0 0.15 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.12 ± 14% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.gimple_simplify 0.10 ± 25% -0.0 0.06 ± 19% perf-profile.children.cycles-pp.kthread 0.16 ± 10% -0.0 0.13 ± 10% perf-profile.children.cycles-pp.free_pcppages_bulk 0.17 ± 9% -0.0 0.14 ± 17% perf-profile.children.cycles-pp.walk_stmt_load_store_addr_ops 0.11 ± 7% -0.0 0.08 ± 22% perf-profile.children.cycles-pp.execute_update_addresses_taken 0.09 ± 14% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.bitmap_and 0.08 ± 5% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.alloc_set_pte 0.10 ± 14% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.et_set_father 0.12 ± 7% -0.0 0.10 ± 14% perf-profile.children.cycles-pp.hash_rtx_cb 0.09 ± 15% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.__do_munmap 0.13 ± 6% -0.0 0.11 ± 11% perf-profile.children.cycles-pp.is_gimple_reg 0.09 ± 10% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.find_get_entry 0.10 ± 8% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.pagecache_get_page 0.10 ± 4% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.substitute_and_fold_dom_walker::before_dom_children 0.09 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.realloc 0.09 ± 10% +0.0 0.11 ± 12% perf-profile.children.cycles-pp.rtx_equal_p 0.07 ± 6% +0.0 0.10 ± 12% perf-profile.children.cycles-pp.c_fully_fold 0.05 ± 8% +0.0 0.08 ± 12% perf-profile.children.cycles-pp._cond_resched 0.09 ± 5% +0.0 0.12 ± 8% perf-profile.children.cycles-pp.build_unary_op 0.04 ± 71% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.cpp_interpret_integer 0.13 ± 6% +0.0 0.16 ± 5% perf-profile.children.cycles-pp.declspecs_add_type 0.10 ± 17% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.task_work_run 0.09 ± 9% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.pushdecl 0.08 ± 16% +0.0 0.11 ± 12% perf-profile.children.cycles-pp.force_fit_type 0.04 ± 73% +0.0 0.07 ± 12% perf-profile.children.cycles-pp.__fput 0.09 ± 13% +0.0 0.13 ± 13% perf-profile.children.cycles-pp.is_typedef_decl 0.11 ± 14% +0.0 0.15 ± 11% perf-profile.children.cycles-pp.convert 0.03 ± 70% +0.0 0.07 ± 20% perf-profile.children.cycles-pp.__mod_node_page_state 0.03 ± 70% +0.0 0.07 perf-profile.children.cycles-pp.do_dentry_open 0.04 ± 73% +0.0 0.08 ± 16% perf-profile.children.cycles-pp.page_add_new_anon_rmap 0.04 ± 70% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.start_function 0.16 ± 11% +0.0 0.21 ± 6% perf-profile.children.cycles-pp.decl_attributes 0.04 ± 71% +0.0 0.08 perf-profile.children.cycles-pp.alloc_stmt_list 0.02 ±141% +0.0 0.06 ± 14% perf-profile.children.cycles-pp.tree_int_cst_sgn 0.02 ±141% +0.0 0.06 ± 13% perf-profile.children.cycles-pp.gimple_call_flags 0.02 ±141% +0.0 0.06 ± 13% perf-profile.children.cycles-pp.build_asm_expr 0.08 ± 11% +0.0 0.13 ± 12% perf-profile.children.cycles-pp.wi::divmod_internal 0.49 ± 4% +0.0 0.54 ± 2% perf-profile.children.cycles-pp.pop_scope 0.03 ± 70% +0.1 0.09 ± 19% perf-profile.children.cycles-pp.pop_stmt_list 0.10 ± 4% +0.1 0.16 ± 10% perf-profile.children.cycles-pp.get_range_from_loc 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.build_component_ref 0.16 ± 10% +0.1 0.23 ± 6% perf-profile.children.cycles-pp.build_null_declspecs 0.00 +0.1 0.07 ± 35% perf-profile.children.cycles-pp.vma_merge 0.34 ± 13% +0.1 0.41 ± 6% perf-profile.children.cycles-pp.fold_binary_loc 0.59 ± 3% +0.1 0.69 ± 7% perf-profile.children.cycles-pp.wide_int_to_tree 0.17 ± 14% +0.1 0.28 ± 12% perf-profile.children.cycles-pp.linemap_add_macro_token 0.72 ± 10% +0.1 0.85 ± 11% perf-profile.children.cycles-pp.c_lex_with_flags 0.58 ± 5% +0.2 0.74 ± 16% perf-profile.children.cycles-pp.htab_find_slot_with_hash 0.76 ± 4% +0.2 0.96 ± 10% perf-profile.children.cycles-pp.linemap_lookup 0.58 ± 11% +0.2 0.83 ± 22% perf-profile.children.cycles-pp.irq_work_interrupt 0.58 ± 11% +0.2 0.83 ± 22% perf-profile.children.cycles-pp.smp_irq_work_interrupt 0.58 ± 11% +0.2 0.83 ± 22% perf-profile.children.cycles-pp.irq_work_run 0.58 ± 11% +0.2 0.83 ± 22% perf-profile.children.cycles-pp.printk 2.21 ± 5% +0.3 2.47 ± 3% perf-profile.children.cycles-pp._cpp_lex_direct 1.16 ± 3% +0.3 1.46 ± 12% perf-profile.children.cycles-pp.memchr 0.66 ± 26% +0.3 0.97 ± 9% perf-profile.children.cycles-pp.irq_work_run_list 2.71 ± 15% -0.8 1.94 ± 15% perf-profile.self.cycles-pp.io_serial_in 1.27 ± 15% -0.3 0.94 ± 18% perf-profile.self.cycles-pp.delay_tsc 0.49 ± 3% -0.2 0.24 ± 22% perf-profile.self.cycles-pp.cfb_imageblit 0.43 ± 3% -0.1 0.35 ± 12% perf-profile.self.cycles-pp.bitmap_clear_bit 0.20 ± 8% -0.1 0.13 ± 24% perf-profile.self.cycles-pp.statistics_fini_pass 0.22 ± 5% -0.1 0.16 ± 8% perf-profile.self.cycles-pp.unmap_page_range 0.32 ± 7% -0.1 0.26 ± 11% perf-profile.self.cycles-pp.__list_del_entry_valid 0.26 ± 8% -0.1 0.21 ± 19% perf-profile.self.cycles-pp.bitmap_bit_p 0.09 ± 18% -0.1 0.04 ± 73% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.18 ± 10% -0.1 0.12 ± 19% perf-profile.self.cycles-pp.bitmap_copy 0.20 ± 8% -0.1 0.15 ± 13% perf-profile.self.cycles-pp.side_effects_p 0.09 ± 13% -0.0 0.05 ± 72% perf-profile.self.cycles-pp.io_serial_out 0.17 ± 5% -0.0 0.12 ± 6% perf-profile.self.cycles-pp.note_stores 0.16 ± 10% -0.0 0.12 ± 10% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.09 ± 24% -0.0 0.05 ± 78% perf-profile.self.cycles-pp.copy_page 0.17 ± 9% -0.0 0.13 ± 9% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.18 ± 12% -0.0 0.14 ± 9% perf-profile.self.cycles-pp.get_attribute_name 0.09 ± 14% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.lra_inheritance 0.24 ± 5% -0.0 0.20 ± 8% perf-profile.self.cycles-pp.init_alias_analysis 0.11 ± 18% -0.0 0.07 ± 28% perf-profile.self.cycles-pp.find_unreachable_blocks 0.12 ± 10% -0.0 0.08 ± 32% perf-profile.self.cycles-pp.for_each_inc_dec 0.07 ± 6% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.wrapup_global_declaration_2 0.13 ± 12% -0.0 0.10 ± 14% perf-profile.self.cycles-pp.free_pcppages_bulk 0.09 ± 14% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.bitmap_and 0.07 ± 7% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.page_remove_rmap 0.10 ± 8% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.update_stmt_operands 0.12 ± 7% -0.0 0.10 ± 14% perf-profile.self.cycles-pp.hash_rtx_cb 0.10 ± 9% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.et_set_father 0.10 ± 4% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.substitute_and_fold_dom_walker::before_dom_children 0.08 +0.0 0.09 perf-profile.self.cycles-pp.realloc 0.07 ± 6% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.htab_traverse 0.08 ± 5% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.build_unary_op 0.08 ± 5% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.rtx_equal_p 0.07 ± 11% +0.0 0.10 ± 9% perf-profile.self.cycles-pp.force_fit_type 0.07 ± 7% +0.0 0.09 ± 10% perf-profile.self.cycles-pp.c_fully_fold 0.13 ± 6% +0.0 0.16 ± 5% perf-profile.self.cycles-pp.declspecs_add_type 0.10 ± 12% +0.0 0.13 ± 6% perf-profile.self.cycles-pp.convert 0.08 ± 10% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.pushdecl 0.04 ± 71% +0.0 0.07 ± 11% perf-profile.self.cycles-pp.start_function 0.15 ± 11% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.decl_attributes 0.03 ± 70% +0.0 0.07 ± 11% perf-profile.self.cycles-pp.malloc@plt 0.09 ± 10% +0.0 0.13 ± 13% perf-profile.self.cycles-pp.is_typedef_decl 0.04 ± 71% +0.0 0.08 perf-profile.self.cycles-pp.alloc_stmt_list 0.02 ±141% +0.0 0.06 ± 13% perf-profile.self.cycles-pp.mark_irreducible_loops 0.08 ± 17% +0.1 0.13 ± 12% perf-profile.self.cycles-pp.wi::divmod_internal 0.02 ±141% +0.1 0.07 ± 25% perf-profile.self.cycles-pp.__mod_node_page_state 0.49 ± 3% +0.1 0.54 ± 2% perf-profile.self.cycles-pp.pop_scope 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.build_component_ref 0.00 +0.1 0.06 ± 16% perf-profile.self.cycles-pp.gimple_call_flags 0.09 ± 5% +0.1 0.15 ± 12% perf-profile.self.cycles-pp.get_range_from_loc 0.02 ±141% +0.1 0.08 ± 12% perf-profile.self.cycles-pp.pop_stmt_list 0.16 ± 10% +0.1 0.23 ± 4% perf-profile.self.cycles-pp.build_null_declspecs 0.29 ± 9% +0.1 0.35 ± 9% perf-profile.self.cycles-pp.fold_binary_loc 0.13 ± 6% +0.1 0.20 ± 2% perf-profile.self.cycles-pp.linemap_add_macro_token 0.57 ± 4% +0.1 0.67 ± 8% perf-profile.self.cycles-pp.wide_int_to_tree 0.70 ± 3% +0.2 0.89 ± 10% perf-profile.self.cycles-pp.linemap_lookup 2.14 ± 6% +0.2 2.39 ± 3% perf-profile.self.cycles-pp._cpp_lex_direct 1.12 ± 2% +0.3 1.40 ± 12% perf-profile.self.cycles-pp.memchr Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen