Greeting, FYI, we noticed a -6.6% regression of netperf.Throughput_Mbps due to commit: commit: b99259a61450bb6403bbbbf279f8754a441eae8b ("[PATCH] net/sock: move memory_allocated over to percpu_counter variables") url: https://github.com/0day-ci/linux/commits/Olof-Johansson/net-sock-move-memory_allocated-over-to-percpu_counter-variables/20180907-095536 in testcase: netperf on test machine: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory with following parameters: ip: ipv4 runtime: 300s nr_threads: 200% cluster: cs-localhost send_size: 5K test: TCP_SENDFILE ucode: 0x25 cpufreq_governor: performance test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance. test-url: http://www.netperf.org/netperf/ In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | netperf: netperf.Throughput_total_tps -49.3% improvement | | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | ip=ipv4 | | | nr_threads=50% | | | runtime=300s | | | test=UDP_RR | +------------------+-----------------------------------------------------------------------+ | testcase: change | netperf: netperf.Throughput_total_tps -13.5% improvement | | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | ip=ipv4 | | | nr_threads=25% | | | runtime=300s | | | test=UDP_RR | +------------------+-----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.2/200%/debian-x86_64-2018-04-03.cgz/300s/5K/lkp-hsw-d01/TCP_SENDFILE/netperf/0x25 commit: 6da410d97f (" mlx5e-fixes-2018-09-05") b99259a614 ("net/sock: move memory_allocated over to percpu_counter variables") 6da410d97ffa486e b99259a61450bb6403bbbbf279 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.RIP:drm_wait_one_vblank[drm] 1:4 -25% :4 dmesg.WARNING:at_drivers/gpu/drm/drm_vblank.c:#drm_wait_one_vblank[drm] :4 25% 1:4 dmesg.WARNING:at_ip_fsnotify/0x 1:4 -25% :4 kmsg.drm:drm_atomic_helper_wait_for_dependencies[drm_kms_helper]]*ERROR*[CONNECTOR:#:VGA-#]flip_done_timed_out 1:4 -25% :4 kmsg.drm:drm_atomic_helper_wait_for_dependencies[drm_kms_helper]]*ERROR*[CRTC:#:pipe_A]flip_done_timed_out 1:4 -25% :4 kmsg.drm:drm_atomic_helper_wait_for_dependencies[drm_kms_helper]]*ERROR*[PLANE:#:primary_A]flip_done_timed_out 1:4 -25% :4 kmsg.drm:drm_atomic_helper_wait_for_flip_done[drm_kms_helper]]*ERROR*[CRTC:#:pipe_A]flip_done_timed_out 0:4 0% 0:4 perf-profile.children.cycles-pp.schedule_timeout %stddev %change %stddev \ | \ 7990 -6.6% 7460 netperf.Throughput_Mbps 127841 -6.6% 119362 netperf.Throughput_total_Mbps 1874683 -2.1% 1835777 netperf.time.involuntary_context_switches 643.25 +1.3% 651.50 netperf.time.percent_of_cpu_this_job_got 1792 +2.1% 1830 netperf.time.system_time 144.93 -8.1% 133.19 netperf.time.user_time 1099277 -7.7% 1014302 netperf.time.voluntary_context_switches 9.363e+08 -6.6% 8.742e+08 netperf.workload 30722 -4.2% 29428 vmstat.system.cs 25888 ± 31% +77.6% 45981 ± 25% sched_debug.cfs_rq:/.min_vruntime.stddev 25888 ± 31% +77.6% 45981 ± 25% sched_debug.cfs_rq:/.spread0.stddev 7151 ± 4% -20.0% 5720 ± 5% slabinfo.anon_vma_chain.active_objs 7409 ± 3% -21.7% 5804 ± 6% slabinfo.anon_vma_chain.num_objs 38618 ± 10% -31.7% 26365 ± 22% turbostat.C1E 0.13 ± 15% -0.1 0.08 ± 44% turbostat.C1E% 161917 ± 4% -24.6% 122121 ± 15% cpuidle.C1.time 3215895 ± 15% -38.8% 1966692 ± 43% cpuidle.C1E.time 38639 ± 10% -31.8% 26368 ± 22% cpuidle.C1E.usage 1.305e+12 -1.4% 1.287e+12 perf-stat.branch-instructions 3.00 -0.0 2.97 perf-stat.branch-miss-rate% 3.92e+10 -2.5% 3.822e+10 perf-stat.branch-misses 0.69 ± 9% -0.2 0.51 ± 14% perf-stat.cache-miss-rate% 2.1e+10 +17.9% 2.477e+10 perf-stat.cache-references 9364894 -4.2% 8976073 perf-stat.context-switches 1.31 +2.0% 1.33 perf-stat.cpi 0.09 -0.0 0.08 perf-stat.dTLB-load-miss-rate% 2.02e+09 -15.8% 1.701e+09 perf-stat.dTLB-load-misses 2.201e+12 -3.0% 2.135e+12 perf-stat.dTLB-loads 1.38e+12 -4.1% 1.325e+12 perf-stat.dTLB-stores 4.827e+08 ± 8% -19.0% 3.908e+08 ± 9% perf-stat.iTLB-loads 6.749e+12 -1.9% 6.62e+12 perf-stat.instructions 0.76 -1.9% 0.75 perf-stat.ipc 7207 +5.1% 7572 perf-stat.path-length 18.44 -1.4 17.04 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.46 -1.4 17.06 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.25 -1.4 16.86 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.17 -1.4 16.80 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 13.45 -1.2 12.24 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 12.68 -1.1 11.57 perf-profile.calltrace.cycles-pp.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 11.95 -0.9 11.00 ± 2% perf-profile.calltrace.cycles-pp.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 9.53 ± 2% -0.9 8.68 perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 9.35 ± 2% -0.8 8.54 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg 9.50 -0.8 8.73 ± 2% perf-profile.calltrace.cycles-pp.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile 10.73 ± 2% -0.5 10.21 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 10.56 ± 2% -0.4 10.11 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage 2.99 ± 3% -0.4 2.59 ± 4% perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64 4.94 ± 2% -0.3 4.60 ± 3% perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.70 ± 14% -0.3 0.43 ± 58% perf-profile.calltrace.cycles-pp.__inode_security_revalidate.selinux_file_permission.security_file_permission.do_splice_direct.do_sendfile 3.04 ± 2% -0.3 2.77 ± 4% perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 2.62 ± 3% -0.2 2.39 ± 2% perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor.do_splice_direct 2.31 ± 2% -0.2 2.09 ± 2% perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.generic_file_read_iter.generic_file_splice_read.splice_direct_to_actor 1.52 ± 5% -0.2 1.31 ± 8% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64 1.14 ± 4% -0.2 0.93 ± 10% perf-profile.calltrace.cycles-pp.__fget_light.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.88 ± 2% -0.2 1.68 perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage 1.11 ± 16% -0.2 0.92 ± 3% perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 3.21 -0.2 3.02 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.74 ± 23% -0.2 0.56 ± 6% perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 1.19 ± 7% -0.2 1.02 ± 5% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64 1.20 ± 3% -0.1 1.05 ± 5% perf-profile.calltrace.cycles-pp.__might_fault.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 1.19 ± 5% -0.1 1.06 ± 6% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64 0.80 ± 4% -0.1 0.68 perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault.copy_page_to_iter.skb_copy_datagram_iter.tcp_recvmsg 0.90 ± 6% -0.1 0.78 ± 6% perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.splice_direct_to_actor.do_splice_direct.do_sendfile 0.71 ± 2% -0.1 0.61 ± 5% perf-profile.calltrace.cycles-pp.sched_clock_cpu.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked 1.07 ± 3% -0.1 0.96 ± 3% perf-profile.calltrace.cycles-pp.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.generic_file_read_iter.generic_file_splice_read 0.74 ± 6% -0.1 0.64 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage 0.63 ± 3% -0.1 0.55 ± 6% perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock.sched_clock_cpu.tcp_write_xmit.__tcp_push_pending_frames 0.70 ± 3% -0.1 0.63 ± 4% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv 0.78 ± 3% -0.1 0.71 ± 5% perf-profile.calltrace.cycles-pp.__radix_tree_lookup.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.generic_file_read_iter 0.82 ± 3% -0.1 0.76 ± 3% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish 5.58 +0.1 5.71 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 5.35 +0.1 5.49 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq 0.91 ± 4% +0.1 1.04 ± 3% perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 90.58 +0.4 90.99 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.5 0.55 ± 6% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 0.00 +0.6 0.60 ± 6% perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 0.00 +0.6 0.65 ± 18% perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish 0.00 +0.7 0.68 ± 2% perf-profile.calltrace.cycles-pp.find_next_bit.cpumask_next.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule 0.00 +0.8 0.78 ± 13% perf-profile.calltrace.cycles-pp.percpu_counter_add_batch.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked 0.00 +0.8 0.84 perf-profile.calltrace.cycles-pp.cpumask_next.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages 0.00 +1.7 1.65 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule 70.71 +2.0 72.70 perf-profile.calltrace.cycles-pp.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 68.26 +2.2 70.47 perf-profile.calltrace.cycles-pp.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +2.8 2.85 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages 57.51 +3.2 60.70 perf-profile.calltrace.cycles-pp.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 54.10 +3.6 57.72 perf-profile.calltrace.cycles-pp.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64 36.84 +4.9 41.77 perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 35.89 +5.0 40.88 perf-profile.calltrace.cycles-pp.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile 35.00 +5.0 40.01 perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct 30.95 +5.0 36.00 perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor 29.68 +5.2 34.87 perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor 28.59 +5.3 33.84 perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe 28.87 +5.3 34.14 perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe 27.07 +5.3 32.36 perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage 21.40 +5.7 27.15 perf-profile.calltrace.cycles-pp.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage 20.54 +5.8 26.38 perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage 0.00 +5.9 5.93 ± 4% perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked 1.22 ± 4% +6.2 7.47 ± 3% perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage 1.32 ± 3% +6.3 7.60 ± 3% perf-profile.calltrace.cycles-pp.__sk_mem_schedule.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 18.44 -1.4 17.04 perf-profile.children.cycles-pp.__sys_recvfrom 18.46 -1.4 17.06 perf-profile.children.cycles-pp.__x64_sys_recvfrom 18.25 -1.4 16.86 perf-profile.children.cycles-pp.inet_recvmsg 18.19 -1.4 16.81 perf-profile.children.cycles-pp.tcp_recvmsg 15.06 -1.2 13.82 perf-profile.children.cycles-pp.copy_page_to_iter 13.47 -1.2 12.25 perf-profile.children.cycles-pp.skb_copy_datagram_iter 12.00 -1.0 11.05 ± 2% perf-profile.children.cycles-pp.generic_file_splice_read 9.58 ± 2% -0.9 8.72 perf-profile.children.cycles-pp.copyout 10.09 -0.9 9.23 ± 2% perf-profile.children.cycles-pp.security_file_permission 9.42 ± 2% -0.8 8.60 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 9.68 -0.8 8.90 ± 2% perf-profile.children.cycles-pp.generic_file_read_iter 11.60 ± 2% -0.6 11.03 perf-profile.children.cycles-pp.__tcp_push_pending_frames 11.47 ± 2% -0.5 10.94 perf-profile.children.cycles-pp.tcp_write_xmit 6.05 ± 2% -0.5 5.57 ± 2% perf-profile.children.cycles-pp.selinux_file_permission 3.54 ± 3% -0.3 3.24 ± 2% perf-profile.children.cycles-pp.___might_sleep 2.50 ± 3% -0.3 2.21 ± 5% perf-profile.children.cycles-pp.file_has_perm 3.05 ± 2% -0.3 2.78 ± 4% perf-profile.children.cycles-pp.do_splice_to 2.06 -0.3 1.81 ± 2% perf-profile.children.cycles-pp.fsnotify 2.70 ± 2% -0.2 2.45 ± 2% perf-profile.children.cycles-pp.pagecache_get_page 3.39 ± 3% -0.2 3.15 perf-profile.children.cycles-pp.touch_atime 2.10 ± 2% -0.2 1.87 ± 2% perf-profile.children.cycles-pp.lock_sock_nested 1.29 ± 13% -0.2 1.07 ± 2% perf-profile.children.cycles-pp.__kfree_skb 2.57 ± 4% -0.2 2.35 perf-profile.children.cycles-pp.atime_needs_update 2.37 -0.2 2.15 ± 3% perf-profile.children.cycles-pp.__might_fault 2.34 ± 2% -0.2 2.13 ± 2% perf-profile.children.cycles-pp.find_get_entry 1.21 ± 5% -0.2 1.00 ± 9% perf-profile.children.cycles-pp.__fget_light 3.67 -0.2 3.46 ± 3% perf-profile.children.cycles-pp.syscall_return_via_sysret 2.72 ± 4% -0.2 2.52 ± 2% perf-profile.children.cycles-pp.__inode_security_revalidate 1.61 ± 4% -0.2 1.41 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_bh 0.86 ± 20% -0.2 0.67 ± 5% perf-profile.children.cycles-pp.skb_release_data 0.47 ± 18% -0.2 0.30 ± 28% perf-profile.children.cycles-pp.hrtimer_interrupt 0.99 ± 5% -0.2 0.83 ± 5% perf-profile.children.cycles-pp.__fsnotify_parent 0.41 ± 21% -0.2 0.25 ± 33% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.52 ± 16% -0.2 0.36 ± 24% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.56 ± 15% -0.2 0.40 ± 22% perf-profile.children.cycles-pp.apic_timer_interrupt 1.49 ± 2% -0.1 1.34 ± 4% perf-profile.children.cycles-pp.current_time 0.32 ± 25% -0.1 0.18 ± 48% perf-profile.children.cycles-pp.tick_sched_timer 1.77 ± 4% -0.1 1.65 ± 2% perf-profile.children.cycles-pp.__might_sleep 1.10 ± 3% -0.1 0.98 ± 2% perf-profile.children.cycles-pp.radix_tree_lookup_slot 0.70 ± 11% -0.1 0.58 ± 9% perf-profile.children.cycles-pp.nf_hook_slow 0.96 ± 2% -0.1 0.85 ± 4% perf-profile.children.cycles-pp.sched_clock_cpu 0.88 ± 2% -0.1 0.78 ± 3% perf-profile.children.cycles-pp.sched_clock 0.85 ± 2% -0.1 0.75 ± 3% perf-profile.children.cycles-pp.native_sched_clock 0.46 ± 4% -0.1 0.37 ± 8% perf-profile.children.cycles-pp.selinux_ip_postroute 0.37 ± 5% -0.1 0.30 ± 8% perf-profile.children.cycles-pp.selinux_ip_postroute_compat 0.44 ± 3% -0.1 0.38 ± 4% perf-profile.children.cycles-pp.tcp_rate_check_app_limited 0.12 ± 27% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.__copy_skb_header 0.20 ± 16% -0.0 0.15 ± 12% perf-profile.children.cycles-pp.lock_timer_base 0.16 ± 13% -0.0 0.11 ± 18% perf-profile.children.cycles-pp.ip_rcv_core 0.21 ± 7% -0.0 0.17 ± 7% perf-profile.children.cycles-pp.enqueue_to_backlog 0.37 ± 3% -0.0 0.33 ± 6% perf-profile.children.cycles-pp.generic_splice_sendpage 0.25 ± 7% -0.0 0.21 ± 7% perf-profile.children.cycles-pp.iov_iter_pipe 0.06 ± 11% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.__list_del_entry_valid 0.06 ± 13% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.get_nohz_timer_target 0.09 ± 18% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.sk_stream_wait_memory 0.09 ± 11% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.tcp_rearm_rto 0.04 ± 60% +0.1 0.11 ± 14% perf-profile.children.cycles-pp.splice_from_pipe_next 0.91 ± 4% +0.1 1.04 ± 3% perf-profile.children.cycles-pp.sk_stream_alloc_skb 0.11 ± 7% +0.2 0.34 ± 9% perf-profile.children.cycles-pp.tcp_leave_memory_pressure 0.33 ± 10% +0.2 0.58 ± 4% perf-profile.children.cycles-pp.__release_sock 3.19 ± 2% +0.4 3.62 ± 4% perf-profile.children.cycles-pp.tcp_v4_do_rcv 0.55 ± 12% +0.4 0.99 ± 10% perf-profile.children.cycles-pp.tcp_data_queue 3.07 ± 2% +0.4 3.51 ± 5% perf-profile.children.cycles-pp.tcp_rcv_established 0.18 ± 27% +0.4 0.63 ± 15% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 90.71 +0.5 91.20 perf-profile.children.cycles-pp.do_syscall_64 0.08 ± 13% +0.5 0.60 ± 12% perf-profile.children.cycles-pp.tcp_try_rmem_schedule 0.00 +0.8 0.77 ± 2% perf-profile.children.cycles-pp.find_next_bit 0.00 +0.9 0.94 ± 10% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.00 +1.1 1.14 ± 3% perf-profile.children.cycles-pp.cpumask_next 0.00 +1.9 1.92 ± 6% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 70.92 +2.0 72.87 perf-profile.children.cycles-pp.__x64_sys_sendfile64 68.42 +2.2 70.62 perf-profile.children.cycles-pp.do_sendfile 57.60 +3.2 60.79 perf-profile.children.cycles-pp.do_splice_direct 0.25 ± 13% +3.3 3.54 ± 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 54.16 +3.6 57.77 perf-profile.children.cycles-pp.splice_direct_to_actor 36.88 +5.0 41.83 perf-profile.children.cycles-pp.direct_splice_actor 35.98 +5.0 40.97 perf-profile.children.cycles-pp.splice_from_pipe 35.13 +5.0 40.13 perf-profile.children.cycles-pp.__splice_from_pipe 31.00 +5.0 36.03 perf-profile.children.cycles-pp.pipe_to_sendpage 29.75 +5.2 34.97 perf-profile.children.cycles-pp.sock_sendpage 28.64 +5.3 33.89 perf-profile.children.cycles-pp.inet_sendpage 28.94 +5.3 34.22 perf-profile.children.cycles-pp.kernel_sendpage 27.21 +5.3 32.51 perf-profile.children.cycles-pp.tcp_sendpage 21.48 +5.7 27.23 perf-profile.children.cycles-pp.tcp_sendpage_locked 20.69 +5.8 26.53 perf-profile.children.cycles-pp.do_tcp_sendpages 0.00 +6.7 6.67 ± 4% perf-profile.children.cycles-pp.__percpu_counter_sum 1.34 ± 3% +7.0 8.32 ± 3% perf-profile.children.cycles-pp.__sk_mem_raise_allocated 1.46 ± 3% +7.0 8.46 ± 3% perf-profile.children.cycles-pp.__sk_mem_schedule 1.21 ± 3% -0.8 0.40 ± 9% perf-profile.self.cycles-pp.__sk_mem_raise_allocated 9.27 -0.8 8.50 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 3.41 ± 3% -0.3 3.08 perf-profile.self.cycles-pp.___might_sleep 3.12 ± 3% -0.3 2.86 perf-profile.self.cycles-pp.selinux_file_permission 2.01 ± 2% -0.2 1.76 ± 2% perf-profile.self.cycles-pp.fsnotify 2.52 -0.2 2.28 perf-profile.self.cycles-pp.generic_file_read_iter 2.42 -0.2 2.20 ± 2% perf-profile.self.cycles-pp.do_sendfile 3.66 -0.2 3.45 ± 3% perf-profile.self.cycles-pp.syscall_return_via_sysret 1.18 ± 5% -0.2 0.98 ± 9% perf-profile.self.cycles-pp.__fget_light 1.62 ± 3% -0.2 1.45 ± 5% perf-profile.self.cycles-pp.tcp_write_xmit 1.19 ± 2% -0.2 1.02 ± 5% perf-profile.self.cycles-pp.pipe_to_sendpage 1.44 ± 5% -0.2 1.28 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_bh 0.89 ± 5% -0.1 0.74 ± 4% perf-profile.self.cycles-pp.__fsnotify_parent 1.12 ± 2% -0.1 0.98 ± 4% perf-profile.self.cycles-pp.do_syscall_64 1.02 ± 3% -0.1 0.90 ± 5% perf-profile.self.cycles-pp.__x64_sys_sendfile64 0.81 ± 7% -0.1 0.70 ± 6% perf-profile.self.cycles-pp.file_has_perm 0.71 ± 7% -0.1 0.61 ± 7% perf-profile.self.cycles-pp.skb_copy_datagram_iter 0.89 ± 6% -0.1 0.79 ± 6% perf-profile.self.cycles-pp.tcp_current_mss 0.14 ± 9% -0.1 0.04 ± 59% perf-profile.self.cycles-pp.sched_clock 0.75 ± 4% -0.1 0.66 ± 6% perf-profile.self.cycles-pp.skb_release_data 0.97 ± 3% -0.1 0.88 ± 6% perf-profile.self.cycles-pp.page_cache_pipe_buf_confirm 0.82 -0.1 0.74 ± 6% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.29 ± 8% -0.1 0.21 ± 8% perf-profile.self.cycles-pp.radix_tree_lookup_slot 0.42 ± 5% -0.1 0.35 ± 4% perf-profile.self.cycles-pp.tcp_rate_check_app_limited 1.57 -0.1 1.51 ± 2% perf-profile.self.cycles-pp.__might_sleep 0.12 ± 27% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.__copy_skb_header 0.15 ± 7% -0.1 0.10 ± 15% perf-profile.self.cycles-pp.selinux_ip_postroute_compat 0.15 ± 14% -0.0 0.11 ± 15% perf-profile.self.cycles-pp.ip_rcv_core 0.17 ± 9% -0.0 0.12 ± 8% perf-profile.self.cycles-pp.ip_output 0.32 -0.0 0.28 ± 4% perf-profile.self.cycles-pp.generic_splice_sendpage 0.14 ± 10% -0.0 0.10 ± 14% perf-profile.self.cycles-pp.enqueue_to_backlog 0.23 ± 9% -0.0 0.19 ± 2% perf-profile.self.cycles-pp.iov_iter_pipe 0.22 ± 9% -0.0 0.18 ± 7% perf-profile.self.cycles-pp.__ip_queue_xmit 0.11 ± 19% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.sock_def_readable 0.11 ± 4% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.bpf_fd_pass 0.11 ± 7% +0.0 0.15 ± 10% perf-profile.self.cycles-pp.tcp_ack 0.01 ±173% +0.1 0.10 ± 8% perf-profile.self.cycles-pp.splice_from_pipe_next 0.09 ± 7% +0.2 0.32 ± 9% perf-profile.self.cycles-pp.tcp_leave_memory_pressure 0.15 ± 16% +0.4 0.52 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.00 +0.4 0.38 ± 12% perf-profile.self.cycles-pp.cpumask_next 0.00 +0.7 0.67 ± 2% perf-profile.self.cycles-pp.find_next_bit 0.00 +0.7 0.72 ± 6% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.24 ± 14% +1.4 1.63 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.00 +1.9 1.90 ± 6% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.00 +2.1 2.08 ± 3% perf-profile.self.cycles-pp.__percpu_counter_sum netperf.Throughput_Mbps 8100 +-+------------------------------------------------------------------+ | ++.+ .++.+++.+ .++. | 8000 +-+++.++. + + +.++.++.++.++.++.++.+++.++.++ ++.++.++.+| 7900 +-+ ++ | | | 7800 +-+ | | | 7700 +-+ | | | 7600 +-+ | 7500 +-+ O O | | O O O OO OOO OO OO | 7400 +-+ O O | OO OO O | 7300 +-+------------------------------------------------------------------+ netperf.Throughput_total_Mbps 130000 +-+----------------------------------------------------------------+ | +.+ +.+++.+ +.+ | 128000 +-+++.++. : +.+ : ++.+ +.++.+++.++.+ .+++.++.+ ++.++.++.+| |+ ++ : : + + | 126000 +-+ : : | | : : | 124000 +-+ :: | | :: | 122000 +-+ :: | | : | 120000 +-+ OO O OO O O | | O O OO O OO O | 118000 O-+OO O | |O O | 116000 +-+----------------------------------------------------------------+ netperf.workload 9.5e+08 +-+---------------------------------------------------------------+ 9.4e+08 +-+ +.++ .++.+++ +.+ | |+.++.++ : + : +.+++.++.++.+++.++.+++.++.++ +.+++.++.+| 9.3e+08 +-+ +.+ : : | 9.2e+08 +-+ : : | | : : | 9.1e+08 +-+ : : | 9e+08 +-+ :: | 8.9e+08 +-+ : | | : | 8.8e+08 +-+ OO OOO OO O | 8.7e+08 +-+ O OOO O OO | O OO O | 8.6e+08 +O+ O | 8.5e+08 +-+---------------------------------------------------------------+ netperf.time.user_time 152 +-+-------------------------------------------------------------------+ 150 +-+ + | | :: | 148 +-+ :: | 146 +-+ +. + + + : :+. +.+| 144 +-++ + + .+ .++.+ .++.+ :+ +.++.++.+ :+ +.++. +.+ + + | 142 +-+ + .++ + +.++ +.+ + +.+ + + | | + | 140 +-+ | 138 +-+ | 136 +-+ O | 134 +-+ O O OO O O O | | O O O OO O O | 132 OO+OO OO | 130 +-+-------------------------------------------------------------------+ netperf.time.system_time 1840 +-+-O----------------------------------------------------------------+ |O O O | 1830 O-+ O O O O OO O OO | | OO O O | | O O O | 1820 +-+ | | | 1810 +-+ | | | 1800 +-+ | | +.+ + +. +. +.+ .++.++. +. ++. +.++. .+ | | .++.+ +. + +. + + : + + + + + + ++.+ .++ +.+| 1790 +-+ ++ + + + +: + | | + + | 1780 +-+------------------------------------------------------------------+ netperf.time.voluntary_context_switches 1.12e+06 +-+--------------------------------------------------------------+ | + + + + + + +. .+ : + .+| 1.1e+06 +-+++. : + + : + + +.++ + + .++.+ :.+ : + .+++ :: +.+++.++ | 1.08e+06 +-+ + :.+ + : + + + + + + + | | + + | 1.06e+06 +-+ | | | 1.04e+06 +-+ | | | 1.02e+06 +-+ OO O OO O | 1e+06 +-+ OO OO O O O | |O O O | 980000 O-+OO O | | | 960000 +-+----O---------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-bdw-ep2: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.2/50%/debian-x86_64-2018-04-03.cgz/300s/lkp-bdw-ep2/UDP_RR/netperf commit: 6da410d97f (" mlx5e-fixes-2018-09-05") b99259a614 ("net/sock: move memory_allocated over to percpu_counter variables") 6da410d97ffa486e b99259a61450bb6403bbbbf279 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x 1:4 -25% :4 dmesg.WARNING:at_ip_native_sched_clock/0x 11:4 -149% 5:4 perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout 4:4 -54% 2:4 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.switch_mm.__sched_text_start.schedule.schedule_timeout 31:4 -378% 16:4 perf-profile.children.cycles-pp.schedule_timeout 1:4 -12% 0:4 perf-profile.self.cycles-pp.schedule_timeout %stddev %change %stddev \ | \ 3140615 ± 2% -49.3% 1593635 netperf.Throughput_total_tps 71377 ± 2% -49.3% 36218 netperf.Throughput_tps 6877 ± 4% +11.8% 7689 netperf.time.involuntary_context_switches 2795 -10.5% 2501 netperf.time.percent_of_cpu_this_job_got 7688 -6.4% 7194 netperf.time.system_time 727.56 ± 3% -53.7% 336.63 ± 6% netperf.time.user_time 9.335e+08 ± 3% -48.8% 4.781e+08 netperf.time.voluntary_context_switches 9.422e+08 ± 2% -49.3% 4.781e+08 netperf.workload 497.45 -10.7% 444.23 pmeter.Average_Active_Power 143.48 -43.2% 81.53 pmeter.performance_per_watt 25025 ± 50% -50.2% 12452 ± 98% proc-vmstat.numa_pte_updates 823677 -1.0% 815119 proc-vmstat.pgfault 14.68 +19.6 34.31 ± 2% mpstat.cpu.soft% 37.84 -21.0 16.88 ± 5% mpstat.cpu.sys% 4.73 ± 3% -2.4 2.30 ± 6% mpstat.cpu.usr% 231917 ± 7% +48.8% 345161 ± 22% numa-numastat.node0.numa_hit 458545 ± 3% -25.1% 343375 ± 24% numa-numastat.node1.local_node 467255 ± 4% -25.0% 350578 ± 22% numa-numastat.node1.numa_hit 53.00 -9.4% 48.00 vmstat.procs.r 12208228 ± 3% -48.6% 6279977 vmstat.system.cs 197402 -5.8% 185888 vmstat.system.in 9.422e+08 ± 2% -49.3% 4.781e+08 softirqs.NET_RX 2090351 ± 22% -23.7% 1595486 ± 3% softirqs.RCU 6824307 -13.3% 5915651 softirqs.SCHED 8055584 +10.9% 8929857 softirqs.TIMER 7.34e+09 +40.3% 1.03e+10 cpuidle.C1.time 1.833e+09 ± 3% -47.4% 9.638e+08 cpuidle.C1.usage 3.942e+08 ± 63% -68.5% 1.243e+08 ± 29% cpuidle.C6.time 558983 ± 49% -64.4% 199205 ± 18% cpuidle.C6.usage 1.824e+08 ± 3% -47.9% 95056831 cpuidle.POLL.time 34485351 ± 5% -82.4% 6062003 cpuidle.POLL.usage 1997 -14.5% 1708 turbostat.Avg_MHz 71.97 -10.8 61.14 turbostat.Busy% 1.833e+09 ± 3% -47.4% 9.638e+08 turbostat.C1 27.42 ± 2% +11.2 38.62 turbostat.C1% 556509 ± 50% -64.8% 196082 ± 18% turbostat.C6 1.46 ± 64% -1.0 0.45 ± 30% turbostat.C6% 27.21 ± 2% +40.7% 38.28 turbostat.CPU%c1 270.14 -13.5% 233.54 turbostat.PkgWatt 15.33 +2.1% 15.65 turbostat.RAMWatt 360260 ± 11% -57.2% 154078 ± 6% sched_debug.cfs_rq:/.MIN_vruntime.avg 2840994 -20.3% 2263123 sched_debug.cfs_rq:/.MIN_vruntime.max 922773 ± 6% -40.1% 552528 ± 3% sched_debug.cfs_rq:/.MIN_vruntime.stddev 360260 ± 11% -57.2% 154078 ± 6% sched_debug.cfs_rq:/.max_vruntime.avg 2840994 -20.3% 2263123 sched_debug.cfs_rq:/.max_vruntime.max 922773 ± 6% -40.1% 552528 ± 3% sched_debug.cfs_rq:/.max_vruntime.stddev 2804769 ± 2% -19.7% 2251378 sched_debug.cfs_rq:/.min_vruntime.avg 2905503 -20.3% 2314293 sched_debug.cfs_rq:/.min_vruntime.max 2731218 ± 3% -18.3% 2232433 sched_debug.cfs_rq:/.min_vruntime.min 29631 ± 28% -61.3% 11469 ± 21% sched_debug.cfs_rq:/.min_vruntime.stddev 0.58 ± 4% -9.9% 0.52 ± 3% sched_debug.cfs_rq:/.nr_running.avg 10.43 ± 9% -15.4% 8.83 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.avg 29606 ± 28% -61.3% 11465 ± 21% sched_debug.cfs_rq:/.spread0.stddev 387.56 -21.9% 302.54 ± 4% sched_debug.cfs_rq:/.util_est_enqueued.avg 356998 ± 25% -45.1% 196064 ± 3% sched_debug.cpu.avg_idle.max 58785 ± 15% -30.2% 41011 ± 2% sched_debug.cpu.avg_idle.stddev 0.46 ± 15% -81.8% 0.08 ± 99% sched_debug.cpu.cpu_load[1].min 2.25 ± 8% -42.6% 1.29 ± 10% sched_debug.cpu.cpu_load[2].min 4.17 ± 14% -21.0% 3.29 ± 4% sched_debug.cpu.cpu_load[3].min 21059143 ± 3% -49.0% 10741980 sched_debug.cpu.nr_switches.avg 21578752 ± 2% -49.7% 10846007 sched_debug.cpu.nr_switches.max 20097461 ± 6% -47.8% 10493962 sched_debug.cpu.nr_switches.min 262381 ± 31% -79.3% 54430 ± 13% sched_debug.cpu.nr_switches.stddev 5.784e+12 ± 2% -27.2% 4.212e+12 perf-stat.branch-instructions 2.76 -0.8 1.96 perf-stat.branch-miss-rate% 1.599e+11 ± 2% -48.2% 8.276e+10 perf-stat.branch-misses 0.73 ± 8% +0.4 1.14 perf-stat.cache-miss-rate% 2.527e+09 ± 7% +11.0% 2.805e+09 perf-stat.cache-misses 3.482e+11 -29.1% 2.47e+11 perf-stat.cache-references 3.733e+09 ± 3% -48.8% 1.912e+09 perf-stat.context-switches 1.93 +20.0% 2.32 perf-stat.cpi 5.633e+13 -17.5% 4.646e+13 perf-stat.cpu-cycles 7.772e+09 ± 4% -43.6% 4.386e+09 ± 15% perf-stat.dTLB-load-misses 8.521e+12 ± 2% -33.9% 5.63e+12 perf-stat.dTLB-loads 0.00 ± 8% +0.0 0.00 ± 7% perf-stat.dTLB-store-miss-rate% 1.095e+08 ± 9% -13.5% 94683885 ± 7% perf-stat.dTLB-store-misses 5.49e+12 ± 2% -48.1% 2.851e+12 perf-stat.dTLB-stores 10.06 ± 5% +3.6 13.65 ± 14% perf-stat.iTLB-load-miss-rate% 9.169e+09 ± 5% -36.1% 5.859e+09 ± 16% perf-stat.iTLB-load-misses 8.203e+10 ± 3% -54.9% 3.696e+10 perf-stat.iTLB-loads 2.915e+13 ± 2% -31.3% 2.002e+13 perf-stat.instructions 0.52 -16.7% 0.43 perf-stat.ipc 801554 -1.1% 792718 perf-stat.minor-faults 7.603e+08 ± 5% +108.2% 1.583e+09 perf-stat.node-load-misses 93.23 ± 3% -35.4 57.88 ± 2% perf-stat.node-store-miss-rate% 1.072e+09 ± 5% -40.6% 6.373e+08 ± 2% perf-stat.node-store-misses 80515351 ± 49% +476.1% 4.638e+08 ± 3% perf-stat.node-stores 801557 -1.1% 792720 perf-stat.page-faults 30931 +35.4% 41869 perf-stat.path-length 15.38 -7.4 7.94 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 15.18 -7.3 7.85 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.40 -6.5 6.93 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.14 -6.3 6.82 perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 8.62 ± 6% -5.9 2.74 perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 10.50 -5.1 5.43 perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 8.98 ± 2% -4.4 4.62 perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom 5.03 ± 12% -4.2 0.88 ± 3% perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 9.64 -3.9 5.70 perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish 9.42 -3.9 5.49 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv 4.54 ± 13% -3.9 0.65 ± 4% perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg 44.22 -3.9 40.32 perf-profile.calltrace.cycles-pp.secondary_startup_64 4.46 ± 13% -3.9 0.60 ± 5% perf-profile.calltrace.cycles-pp.ip_idents_reserve.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg 43.77 -3.8 39.95 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 43.76 -3.8 39.94 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 7.84 ± 2% -3.8 4.06 perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg 43.66 -3.8 39.89 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 8.92 -3.7 5.20 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb 7.58 ± 2% -3.7 3.91 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 7.46 ± 2% -3.6 3.82 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp 8.60 -3.6 5.00 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb 8.49 -3.6 4.94 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable 6.10 -2.8 3.27 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 5.95 -2.7 3.20 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary 4.93 -2.1 2.88 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 4.64 -1.9 2.69 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 3.65 -1.8 1.84 perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 4.00 -1.7 2.31 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 3.24 ± 2% -1.6 1.63 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets 3.27 -1.6 1.69 perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 2.52 -1.3 1.19 perf-profile.calltrace.cycles-pp.__entry_SYSCALL_64_trampoline 2.48 ± 2% -1.3 1.20 perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 2.23 ± 4% -1.1 1.12 ± 8% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb 2.24 -1.1 1.18 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 2.16 ± 2% -1.0 1.17 perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 1.94 ± 2% -1.0 0.96 perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 1.66 ± 5% -0.8 0.84 ± 10% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb 1.57 ± 5% -0.8 0.78 ± 10% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output 1.55 ± 2% -0.8 0.78 perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle 1.65 ± 2% -0.8 0.90 perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg 1.52 -0.7 0.77 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 1.57 ± 2% -0.7 0.85 perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb 1.46 -0.7 0.79 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle 1.29 ± 5% -0.6 0.65 ± 7% perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 0.89 ± 2% -0.6 0.25 ±100% perf-profile.calltrace.cycles-pp.resched_curr.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function 1.26 ± 5% -0.6 0.62 ± 7% perf-profile.calltrace.cycles-pp.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto 1.46 -0.6 0.86 perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up 1.20 ± 5% -0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.ip_route_output_key_hash_rcu.ip_route_output_key_hash.ip_route_output_flow.udp_sendmsg.sock_sendmsg 1.34 -0.6 0.77 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 1.20 -0.6 0.63 ± 2% perf-profile.calltrace.cycles-pp.switch_mm.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets 1.00 ± 2% -0.5 0.52 perf-profile.calltrace.cycles-pp.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.06 ± 2% -0.5 0.59 perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.10 ± 3% -0.5 0.65 perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.95 ± 3% -0.4 0.54 ± 2% perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common 30.84 +2.3 33.16 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 29.71 +2.7 32.39 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 49.06 +7.3 56.32 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 48.63 +7.5 56.11 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 32.67 +15.2 47.87 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 32.48 +15.3 47.77 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 31.55 +15.7 47.30 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.98 +16.0 47.02 perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 20.10 +23.1 43.17 perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 19.82 +23.2 43.02 perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 19.32 +23.4 42.76 perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg 17.83 +24.2 42.03 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg 15.21 +25.5 40.72 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb 15.07 +25.6 40.64 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb 14.83 +25.7 40.52 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output 14.73 +25.7 40.45 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2 14.31 +25.9 40.17 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 13.95 +26.0 39.97 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq 13.17 +26.4 39.61 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack 12.85 +26.6 39.43 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start 12.43 +26.7 39.16 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 12.27 +26.8 39.07 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 11.92 +27.0 38.90 perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 11.37 +27.2 38.58 perf-profile.calltrace.cycles-pp.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv 10.29 +27.7 38.01 perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver 0.00 +30.3 30.29 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb 0.00 +30.5 30.48 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb 0.00 +31.8 31.84 perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv 0.00 +32.0 31.98 perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish 15.40 -7.4 7.96 perf-profile.children.cycles-pp.__x64_sys_recvfrom 15.20 -7.3 7.85 perf-profile.children.cycles-pp.__sys_recvfrom 13.61 -6.5 7.12 perf-profile.children.cycles-pp.__sched_text_start 13.40 -6.5 6.94 perf-profile.children.cycles-pp.inet_recvmsg 13.15 -6.3 6.83 perf-profile.children.cycles-pp.udp_recvmsg 8.64 ± 6% -5.9 2.75 perf-profile.children.cycles-pp.ip_make_skb 10.53 -5.1 5.44 perf-profile.children.cycles-pp.__skb_recv_udp 8.98 ± 2% -4.4 4.63 perf-profile.children.cycles-pp.__skb_wait_for_more_packets 5.04 ± 12% -4.2 0.88 ± 3% perf-profile.children.cycles-pp.__ip_make_skb 9.65 -3.9 5.71 perf-profile.children.cycles-pp.sock_def_readable 9.43 -3.9 5.49 perf-profile.children.cycles-pp.__wake_up_common_lock 4.54 ± 13% -3.9 0.65 ± 4% perf-profile.children.cycles-pp.__ip_select_ident 44.22 -3.9 40.32 perf-profile.children.cycles-pp.secondary_startup_64 44.22 -3.9 40.32 perf-profile.children.cycles-pp.cpu_startup_entry 44.20 -3.9 40.31 perf-profile.children.cycles-pp.do_idle 4.46 ± 13% -3.9 0.60 ± 5% perf-profile.children.cycles-pp.ip_idents_reserve 43.77 -3.8 39.95 perf-profile.children.cycles-pp.start_secondary 8.93 -3.7 5.20 perf-profile.children.cycles-pp.__wake_up_common 7.59 ± 2% -3.7 3.92 perf-profile.children.cycles-pp.schedule 8.61 -3.6 5.01 perf-profile.children.cycles-pp.autoremove_wake_function 8.51 -3.6 4.95 perf-profile.children.cycles-pp.try_to_wake_up 6.16 -2.9 3.31 perf-profile.children.cycles-pp.schedule_idle 4.93 -2.1 2.88 perf-profile.children.cycles-pp.ttwu_do_activate 4.64 -1.9 2.69 perf-profile.children.cycles-pp.enqueue_task_fair 3.70 -1.8 1.87 perf-profile.children.cycles-pp.menu_select 4.09 -1.7 2.36 perf-profile.children.cycles-pp.enqueue_entity 3.32 ± 2% -1.7 1.67 perf-profile.children.cycles-pp.dequeue_task_fair 3.28 -1.6 1.69 perf-profile.children.cycles-pp.__ip_append_data 2.98 -1.5 1.44 ± 2% perf-profile.children.cycles-pp.__entry_SYSCALL_64_trampoline 3.02 ± 2% -1.5 1.51 perf-profile.children.cycles-pp.dequeue_entity 2.93 -1.3 1.61 perf-profile.children.cycles-pp.update_load_avg 2.78 -1.3 1.47 perf-profile.children.cycles-pp.pick_next_task_fair 2.51 -1.3 1.22 perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 2.52 -1.1 1.38 perf-profile.children.cycles-pp.switch_mm_irqs_off 2.24 ± 4% -1.1 1.13 ± 8% perf-profile.children.cycles-pp.__dev_queue_xmit 1.97 ± 2% -1.0 0.97 perf-profile.children.cycles-pp.tick_nohz_next_event 2.16 -1.0 1.18 perf-profile.children.cycles-pp.sock_alloc_send_pskb 1.74 -0.8 0.89 perf-profile.children.cycles-pp.syscall_return_via_sysret 1.67 ± 4% -0.8 0.84 ± 10% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.58 ± 5% -0.8 0.79 ± 11% perf-profile.children.cycles-pp.loopback_xmit 1.57 ± 2% -0.8 0.79 perf-profile.children.cycles-pp.get_next_timer_interrupt 1.46 -0.8 0.70 perf-profile.children.cycles-pp._raw_spin_lock 1.65 ± 2% -0.8 0.90 perf-profile.children.cycles-pp.alloc_skb_with_frags 1.58 ± 2% -0.7 0.85 perf-profile.children.cycles-pp.__alloc_skb 1.49 -0.7 0.80 perf-profile.children.cycles-pp.set_next_entity 1.36 ± 2% -0.7 0.67 perf-profile.children.cycles-pp.load_new_mm_cr3 1.29 ± 5% -0.6 0.65 ± 7% perf-profile.children.cycles-pp.ip_route_output_flow 1.26 ± 5% -0.6 0.63 ± 7% perf-profile.children.cycles-pp.ip_route_output_key_hash 1.21 ± 5% -0.6 0.60 ± 7% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu 1.24 ± 2% -0.6 0.65 perf-profile.children.cycles-pp.switch_mm 1.17 -0.6 0.59 perf-profile.children.cycles-pp.update_curr 1.10 ± 3% -0.6 0.54 ± 3% perf-profile.children.cycles-pp.nf_hook_slow 1.08 ± 3% -0.6 0.52 perf-profile.children.cycles-pp.__next_timer_interrupt 1.09 -0.5 0.60 ± 2% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.95 ± 2% -0.5 0.45 ± 3% perf-profile.children.cycles-pp.___perf_sw_event 1.01 ± 2% -0.5 0.52 perf-profile.children.cycles-pp.move_addr_to_user 1.06 ± 2% -0.5 0.59 perf-profile.children.cycles-pp.ttwu_do_wakeup 1.10 ± 3% -0.5 0.65 perf-profile.children.cycles-pp.select_task_rq_fair 1.02 ± 2% -0.4 0.57 perf-profile.children.cycles-pp.check_preempt_curr 0.83 ± 2% -0.4 0.41 ± 2% perf-profile.children.cycles-pp.__switch_to_asm 0.87 ± 2% -0.4 0.46 ± 3% perf-profile.children.cycles-pp._copy_to_iter 0.77 ± 5% -0.4 0.37 ± 3% perf-profile.children.cycles-pp.selinux_ip_postroute 0.84 -0.4 0.45 ± 3% perf-profile.children.cycles-pp.sk_filter_trim_cap 0.89 ± 2% -0.4 0.50 ± 2% perf-profile.children.cycles-pp.resched_curr 0.87 ± 2% -0.4 0.48 perf-profile.children.cycles-pp.__consume_stateless_skb 0.75 ± 2% -0.4 0.36 ± 2% perf-profile.children.cycles-pp.__switch_to 0.79 -0.4 0.42 ± 2% perf-profile.children.cycles-pp.security_sock_rcv_skb 0.72 ± 7% -0.4 0.36 ± 3% perf-profile.children.cycles-pp.update_cfs_group 0.64 ± 2% -0.4 0.29 perf-profile.children.cycles-pp.ip_generic_getfrag 0.73 ± 3% -0.4 0.38 ± 2% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.72 -0.3 0.37 ± 3% perf-profile.children.cycles-pp.finish_task_switch 0.73 -0.3 0.39 ± 4% perf-profile.children.cycles-pp.sock_wfree 0.74 -0.3 0.40 ± 3% perf-profile.children.cycles-pp.selinux_socket_sock_rcv_skb 0.69 ± 10% -0.3 0.35 ± 12% perf-profile.children.cycles-pp.fib_table_lookup 0.72 ± 2% -0.3 0.39 ± 3% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.62 ± 5% -0.3 0.30 ± 4% perf-profile.children.cycles-pp.selinux_ip_postroute_compat 0.73 -0.3 0.42 perf-profile.children.cycles-pp._raw_spin_lock_bh 0.67 ± 2% -0.3 0.35 ± 3% perf-profile.children.cycles-pp.__slab_free 0.53 ± 2% -0.3 0.23 ± 3% perf-profile.children.cycles-pp._copy_from_iter_full 0.61 ± 2% -0.3 0.33 ± 2% perf-profile.children.cycles-pp.sched_clock_cpu 0.69 ± 3% -0.3 0.40 perf-profile.children.cycles-pp.poll_idle 0.61 ± 2% -0.3 0.34 ± 5% perf-profile.children.cycles-pp.account_entity_enqueue 0.56 ± 2% -0.3 0.30 ± 3% perf-profile.children.cycles-pp.sched_clock 0.47 ± 21% -0.3 0.20 ± 30% perf-profile.children.cycles-pp.netif_rx_internal 0.51 ± 3% -0.3 0.25 ± 4% perf-profile.children.cycles-pp.sockfd_lookup_light 0.54 ± 2% -0.3 0.28 ± 2% perf-profile.children.cycles-pp.native_sched_clock 0.46 ± 4% -0.2 0.21 ± 3% perf-profile.children.cycles-pp.pick_next_task_idle 0.55 -0.2 0.30 ± 4% perf-profile.children.cycles-pp.ktime_get 0.48 ± 3% -0.2 0.24 perf-profile.children.cycles-pp.__might_fault 0.43 ± 4% -0.2 0.18 ± 2% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.54 -0.2 0.30 ± 2% perf-profile.children.cycles-pp.reweight_entity 0.53 -0.2 0.29 perf-profile.children.cycles-pp.prepare_to_wait_exclusive 0.56 ± 2% -0.2 0.33 perf-profile.children.cycles-pp.__update_load_avg_se 0.72 ± 4% -0.2 0.51 perf-profile.children.cycles-pp.find_next_bit 0.46 ± 3% -0.2 0.24 perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.41 ± 10% -0.2 0.19 ± 7% perf-profile.children.cycles-pp.avc_has_perm 0.41 ± 2% -0.2 0.20 ± 4% perf-profile.children.cycles-pp.__fget_light 0.43 ± 10% -0.2 0.21 ± 11% perf-profile.children.cycles-pp.read_tsc 0.51 ± 3% -0.2 0.30 ± 2% perf-profile.children.cycles-pp.update_rq_clock 0.48 -0.2 0.27 ± 3% perf-profile.children.cycles-pp.selinux_sock_rcv_skb_compat 0.45 ± 2% -0.2 0.25 perf-profile.children.cycles-pp.native_write_msr 0.46 ± 2% -0.2 0.26 ± 4% perf-profile.children.cycles-pp.sock_def_write_space 0.38 -0.2 0.18 ± 2% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.49 ± 2% -0.2 0.30 ± 2% perf-profile.children.cycles-pp.__kmalloc_reserve 0.42 ± 2% -0.2 0.23 ± 2% perf-profile.children.cycles-pp.update_ts_time_stats 0.46 ± 4% -0.2 0.27 ± 3% perf-profile.children.cycles-pp.copyout 0.40 ± 2% -0.2 0.21 ± 4% perf-profile.children.cycles-pp.ip_local_out 0.42 -0.2 0.23 perf-profile.children.cycles-pp.skb_set_owner_w 0.36 -0.2 0.18 ± 4% perf-profile.children.cycles-pp.pick_next_entity 0.36 ± 3% -0.2 0.18 ± 4% perf-profile.children.cycles-pp.__ip_local_out 0.33 ± 2% -0.2 0.16 ± 5% perf-profile.children.cycles-pp.validate_xmit_skb 0.32 ± 2% -0.2 0.15 ± 3% perf-profile.children.cycles-pp.enqueue_to_backlog 0.31 ± 3% -0.2 0.15 ± 3% perf-profile.children.cycles-pp.put_prev_task_fair 0.39 ± 7% -0.2 0.23 ± 4% perf-profile.children.cycles-pp.ksize 0.44 -0.2 0.28 perf-profile.children.cycles-pp.__kmalloc_node_track_caller 0.41 ± 7% -0.2 0.26 perf-profile.children.cycles-pp.select_idle_sibling 0.34 ± 2% -0.2 0.19 ± 6% perf-profile.children.cycles-pp.ipv4_mtu 0.30 ± 2% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.___might_sleep 0.35 ± 3% -0.1 0.21 ± 2% perf-profile.children.cycles-pp.nr_iowait_cpu 0.28 -0.1 0.14 ± 3% perf-profile.children.cycles-pp.__skb_try_recv_from_queue 0.27 ± 5% -0.1 0.12 ± 8% perf-profile.children.cycles-pp.security_socket_sendmsg 0.29 ± 2% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.__udp4_lib_lookup 0.24 ± 3% -0.1 0.10 perf-profile.children.cycles-pp.copyin 0.32 ± 3% -0.1 0.18 ± 3% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.25 ± 4% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.kmem_cache_free 0.24 ± 2% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_stage2 0.28 ± 2% -0.1 0.15 ± 2% perf-profile.children.cycles-pp.move_addr_to_kernel 0.27 ± 4% -0.1 0.14 ± 3% perf-profile.children.cycles-pp.sock_has_perm 0.28 -0.1 0.15 ± 5% perf-profile.children.cycles-pp.selinux_parse_skb 0.35 ± 3% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.available_idle_cpu 0.25 -0.1 0.12 ± 4% perf-profile.children.cycles-pp._copy_from_user 0.27 ± 20% -0.1 0.15 ± 7% perf-profile.children.cycles-pp.__netif_receive_skb_core 0.29 ± 2% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.__list_del_entry_valid 0.20 ± 8% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.22 ± 4% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.account_entity_dequeue 0.32 -0.1 0.20 ± 5% perf-profile.children.cycles-pp.__enqueue_entity 0.29 ± 2% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.skb_release_data 0.22 ± 3% -0.1 0.11 perf-profile.children.cycles-pp._copy_to_user 0.20 ± 4% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.menu_reflect 0.21 ± 2% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.put_prev_entity 0.23 -0.1 0.12 ± 3% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.22 ± 3% -0.1 0.11 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.22 ± 3% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.rcu_idle_exit 0.21 ± 2% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.security_sk_classify_flow 0.21 ± 3% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.sock_recvmsg 0.18 ± 3% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.__get_user_4 0.51 ± 3% -0.1 0.41 ± 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.22 -0.1 0.12 ± 8% perf-profile.children.cycles-pp.ip_setup_cork 0.19 ± 4% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.inet_sendmsg 0.21 ± 3% -0.1 0.11 ± 3% perf-profile.children.cycles-pp.__might_sleep 0.20 ± 2% -0.1 0.10 perf-profile.children.cycles-pp.__calc_delta 0.18 ± 2% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.security_socket_recvmsg 0.18 ± 2% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.call_cpuidle 0.23 ± 2% -0.1 0.14 ± 8% perf-profile.children.cycles-pp.__list_add_valid 0.12 ± 4% -0.1 0.04 ± 57% perf-profile.children.cycles-pp.sched_ttwu_pending 0.19 ± 3% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.rcu_eqs_enter 0.23 ± 3% -0.1 0.15 ± 4% perf-profile.children.cycles-pp.activate_task 0.18 ± 2% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.netif_skb_features 0.18 ± 4% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.rb_erase_cached 0.15 ± 2% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.rcu_needs_cpu 0.17 ± 4% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.ip_finish_output 0.16 ± 2% -0.1 0.10 perf-profile.children.cycles-pp.ip_rcv_core 0.13 ± 3% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.selinux_ipv4_output 0.13 -0.1 0.07 perf-profile.children.cycles-pp.selinux_sk_getsecid 0.11 ± 6% -0.1 0.05 perf-profile.children.cycles-pp.update_min_vruntime 0.08 ± 5% -0.1 0.03 ±100% perf-profile.children.cycles-pp.ip_send_check 0.12 ± 9% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.compute_score 0.10 ± 5% -0.1 0.04 ± 57% perf-profile.children.cycles-pp.deactivate_task 0.11 ± 6% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.ipv4_pktinfo_prepare 0.11 -0.1 0.06 ± 9% perf-profile.children.cycles-pp.dst_release 0.08 ± 8% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__slab_alloc 0.11 ± 4% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.pm_qos_request 0.10 ± 4% -0.1 0.05 perf-profile.children.cycles-pp.udp_rcv 0.10 ± 8% -0.1 0.05 perf-profile.children.cycles-pp.__update_idle_core 0.13 ± 10% -0.1 0.08 ± 6% perf-profile.children.cycles-pp.import_single_range 0.10 ± 4% -0.0 0.05 perf-profile.children.cycles-pp.rcu_eqs_exit 0.11 ± 7% -0.0 0.07 perf-profile.children.cycles-pp.rb_insert_color_cached 0.08 ± 8% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.tick_check_broadcast_expired 0.13 -0.0 0.10 ± 4% perf-profile.children.cycles-pp.irq_exit 0.07 ± 5% -0.0 0.05 perf-profile.children.cycles-pp.rebalance_domains 0.07 ± 5% -0.0 0.05 ± 9% perf-profile.children.cycles-pp.receiver_wake_function 0.00 +0.1 0.10 ± 5% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.00 +0.3 0.28 perf-profile.children.cycles-pp.cpumask_next 31.18 +2.3 33.50 perf-profile.children.cycles-pp.cpuidle_enter_state 30.02 +2.7 32.70 perf-profile.children.cycles-pp.intel_idle 49.12 +7.3 56.39 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 48.68 +7.5 56.17 perf-profile.children.cycles-pp.do_syscall_64 32.69 +15.2 47.88 perf-profile.children.cycles-pp.__x64_sys_sendto 32.49 +15.3 47.77 perf-profile.children.cycles-pp.__sys_sendto 31.56 +15.7 47.30 perf-profile.children.cycles-pp.sock_sendmsg 31.00 +16.0 47.03 perf-profile.children.cycles-pp.udp_sendmsg 20.10 +23.1 43.18 perf-profile.children.cycles-pp.udp_send_skb 19.82 +23.2 43.03 perf-profile.children.cycles-pp.ip_send_skb 19.33 +23.4 42.77 perf-profile.children.cycles-pp.ip_output 17.85 +24.2 42.05 perf-profile.children.cycles-pp.ip_finish_output2 15.28 +25.5 40.76 perf-profile.children.cycles-pp.__local_bh_enable_ip 15.10 +25.6 40.66 perf-profile.children.cycles-pp.do_softirq 14.84 +25.7 40.52 perf-profile.children.cycles-pp.do_softirq_own_stack 14.85 +25.7 40.54 perf-profile.children.cycles-pp.__softirqentry_text_start 14.32 +25.9 40.18 perf-profile.children.cycles-pp.net_rx_action 13.96 +26.0 39.98 perf-profile.children.cycles-pp.process_backlog 13.18 +26.4 39.61 perf-profile.children.cycles-pp.__netif_receive_skb_one_core 12.86 +26.6 39.44 perf-profile.children.cycles-pp.ip_rcv 12.43 +26.7 39.16 perf-profile.children.cycles-pp.ip_local_deliver 12.27 +26.8 39.08 perf-profile.children.cycles-pp.ip_local_deliver_finish 11.93 +27.0 38.90 perf-profile.children.cycles-pp.__udp4_lib_rcv 11.38 +27.2 38.59 perf-profile.children.cycles-pp.udp_queue_rcv_skb 10.30 +27.7 38.02 perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb 1.43 ± 3% +29.9 31.29 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.06 ± 87% +30.2 30.31 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.15 ± 3% +31.8 31.98 perf-profile.children.cycles-pp.__sk_mem_raise_allocated 0.00 +31.9 31.87 perf-profile.children.cycles-pp.__percpu_counter_sum 4.44 ± 13% -3.8 0.60 ± 5% perf-profile.self.cycles-pp.ip_idents_reserve 2.90 -1.5 1.39 perf-profile.self.cycles-pp.__entry_SYSCALL_64_trampoline 2.17 -1.0 1.20 perf-profile.self.cycles-pp.__sched_text_start 1.74 -0.8 0.89 perf-profile.self.cycles-pp.syscall_return_via_sysret 1.39 -0.7 0.67 perf-profile.self.cycles-pp._raw_spin_lock 1.36 ± 2% -0.7 0.67 perf-profile.self.cycles-pp.load_new_mm_cr3 1.25 -0.6 0.67 perf-profile.self.cycles-pp.update_load_avg 1.08 ± 2% -0.5 0.59 ± 2% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.87 ± 2% -0.5 0.41 ± 3% perf-profile.self.cycles-pp.___perf_sw_event 1.16 -0.4 0.71 perf-profile.self.cycles-pp.switch_mm_irqs_off 0.83 ± 2% -0.4 0.41 ± 2% perf-profile.self.cycles-pp.__switch_to_asm 0.85 -0.4 0.43 ± 2% perf-profile.self.cycles-pp.update_curr 1.38 ± 2% -0.4 0.98 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.89 ± 2% -0.4 0.49 perf-profile.self.cycles-pp.menu_select 0.71 ± 2% -0.4 0.35 ± 2% perf-profile.self.cycles-pp.__switch_to 0.79 -0.4 0.43 perf-profile.self.cycles-pp.do_idle 0.82 ± 3% -0.4 0.46 ± 2% perf-profile.self.cycles-pp.resched_curr 0.68 ± 2% -0.4 0.32 ± 3% perf-profile.self.cycles-pp.udp_sendmsg 0.70 ± 7% -0.3 0.35 ± 3% perf-profile.self.cycles-pp.update_cfs_group 0.68 ± 11% -0.3 0.35 ± 12% perf-profile.self.cycles-pp.fib_table_lookup 0.71 -0.3 0.41 perf-profile.self.cycles-pp._raw_spin_lock_bh 0.68 ± 3% -0.3 0.38 ± 2% perf-profile.self.cycles-pp.copy_user_generic_unrolled 0.66 ± 2% -0.3 0.35 ± 3% perf-profile.self.cycles-pp.__slab_free 0.62 -0.3 0.32 ± 2% perf-profile.self.cycles-pp.pick_next_task_fair 0.67 ± 2% -0.3 0.38 perf-profile.self.cycles-pp.select_task_rq_fair 0.65 -0.3 0.36 perf-profile.self.cycles-pp.set_next_entity 0.59 -0.3 0.32 perf-profile.self.cycles-pp.udp_recvmsg 0.57 -0.3 0.30 ± 2% perf-profile.self.cycles-pp.finish_task_switch 0.65 -0.3 0.39 perf-profile.self.cycles-pp.enqueue_entity 0.51 ± 2% -0.3 0.26 ± 3% perf-profile.self.cycles-pp.do_syscall_64 0.49 -0.3 0.24 ± 3% perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu 0.52 ± 2% -0.2 0.27 ± 2% perf-profile.self.cycles-pp.native_sched_clock 0.54 ± 3% -0.2 0.29 ± 2% perf-profile.self.cycles-pp.dequeue_entity 0.54 -0.2 0.30 ± 2% perf-profile.self.cycles-pp.reweight_entity 0.45 ± 2% -0.2 0.21 ± 7% perf-profile.self.cycles-pp.__ip_append_data 0.47 ± 3% -0.2 0.24 perf-profile.self.cycles-pp.__next_timer_interrupt 0.57 ± 3% -0.2 0.34 ± 3% perf-profile.self.cycles-pp.poll_idle 0.42 ± 2% -0.2 0.19 ± 3% perf-profile.self.cycles-pp.__skb_wait_for_more_packets 0.54 ± 2% -0.2 0.32 perf-profile.self.cycles-pp.__update_load_avg_se 0.44 -0.2 0.21 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.39 ± 2% -0.2 0.16 ± 6% perf-profile.self.cycles-pp.ip_output 0.55 -0.2 0.33 ± 2% perf-profile.self.cycles-pp.enqueue_task_fair 0.41 ± 3% -0.2 0.20 ± 4% perf-profile.self.cycles-pp.__fget_light 0.40 ± 10% -0.2 0.18 ± 6% perf-profile.self.cycles-pp.avc_has_perm 0.48 -0.2 0.28 perf-profile.self.cycles-pp.move_addr_to_user 0.45 -0.2 0.24 ± 2% perf-profile.self.cycles-pp.native_write_msr 0.40 ± 10% -0.2 0.20 ± 13% perf-profile.self.cycles-pp.read_tsc 0.36 ± 4% -0.2 0.16 ± 4% perf-profile.self.cycles-pp.__skb_recv_udp 0.45 ± 2% -0.2 0.25 ± 5% perf-profile.self.cycles-pp.sock_def_write_space 0.42 ± 2% -0.2 0.23 ± 2% perf-profile.self.cycles-pp.account_entity_enqueue 0.38 ± 5% -0.2 0.19 ± 2% perf-profile.self.cycles-pp.dequeue_task_fair 0.39 ± 8% -0.2 0.20 ± 3% perf-profile.self.cycles-pp.ip_finish_output2 0.41 -0.2 0.23 perf-profile.self.cycles-pp.skb_set_owner_w 0.60 ± 4% -0.2 0.42 perf-profile.self.cycles-pp.find_next_bit 0.34 -0.2 0.17 ± 3% perf-profile.self.cycles-pp.__ip_make_skb 0.36 ± 3% -0.2 0.19 ± 2% perf-profile.self.cycles-pp.process_backlog 0.45 ± 4% -0.2 0.28 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.31 ± 2% -0.2 0.15 ± 2% perf-profile.self.cycles-pp.__alloc_skb 0.33 -0.2 0.17 ± 4% perf-profile.self.cycles-pp.pick_next_entity 0.38 ± 6% -0.2 0.22 ± 3% perf-profile.self.cycles-pp.ksize 0.38 ± 4% -0.2 0.22 ± 3% perf-profile.self.cycles-pp.__softirqentry_text_start 0.31 ± 2% -0.2 0.15 ± 2% perf-profile.self.cycles-pp.__sys_sendto 0.29 ± 3% -0.2 0.14 ± 8% perf-profile.self.cycles-pp._copy_to_iter 0.30 ± 3% -0.1 0.15 perf-profile.self.cycles-pp.___might_sleep 0.27 -0.1 0.12 ± 5% perf-profile.self.cycles-pp.sock_wfree 0.40 ± 2% -0.1 0.26 ± 3% perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb 0.27 ± 3% -0.1 0.13 ± 3% perf-profile.self.cycles-pp.__sys_recvfrom 0.31 -0.1 0.17 ± 6% perf-profile.self.cycles-pp.ipv4_mtu 0.34 ± 3% -0.1 0.21 ± 2% perf-profile.self.cycles-pp.nr_iowait_cpu 0.27 ± 4% -0.1 0.13 ± 3% perf-profile.self.cycles-pp.sock_has_perm 0.29 -0.1 0.16 ± 5% perf-profile.self.cycles-pp.net_rx_action 0.24 ± 3% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_stage2 0.28 ± 2% -0.1 0.15 ± 4% perf-profile.self.cycles-pp.loopback_xmit 0.24 ± 3% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.inet_recvmsg 0.27 -0.1 0.14 ± 3% perf-profile.self.cycles-pp.do_softirq 0.27 -0.1 0.15 ± 5% perf-profile.self.cycles-pp.selinux_parse_skb 0.27 ± 20% -0.1 0.15 ± 5% perf-profile.self.cycles-pp.__netif_receive_skb_core 0.35 ± 4% -0.1 0.23 perf-profile.self.cycles-pp.available_idle_cpu 0.31 ± 2% -0.1 0.19 ± 3% perf-profile.self.cycles-pp.__enqueue_entity 0.24 -0.1 0.12 ± 3% perf-profile.self.cycles-pp.selinux_socket_sock_rcv_skb 0.29 ± 3% -0.1 0.17 ± 4% perf-profile.self.cycles-pp.__list_del_entry_valid 0.28 ± 3% -0.1 0.17 ± 3% perf-profile.self.cycles-pp.skb_release_data 0.27 ± 2% -0.1 0.16 ± 4% perf-profile.self.cycles-pp.__udp4_lib_rcv 0.21 ± 5% -0.1 0.10 ± 5% perf-profile.self.cycles-pp.account_entity_dequeue 0.21 ± 2% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.18 ± 4% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.tick_nohz_next_event 0.23 -0.1 0.12 ± 4% perf-profile.self.cycles-pp.udp_send_skb 0.21 ± 3% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.26 -0.1 0.16 ± 4% perf-profile.self.cycles-pp.try_to_wake_up 0.20 ± 4% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.put_prev_task_fair 0.24 ± 3% -0.1 0.14 ± 3% perf-profile.self.cycles-pp.__wake_up_common 0.20 ± 4% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.__x64_sys_recvfrom 0.18 ± 3% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.selinux_ip_postroute_compat 0.18 ± 2% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.__get_user_4 0.20 ± 8% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.__dev_queue_xmit 0.26 ± 5% -0.1 0.16 ± 2% perf-profile.self.cycles-pp.update_rq_clock 0.26 -0.1 0.16 ± 4% perf-profile.self.cycles-pp.__kmalloc_node_track_caller 0.20 ± 2% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.19 ± 3% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.enqueue_to_backlog 0.18 ± 2% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.call_cpuidle 0.22 ± 3% -0.1 0.13 ± 9% perf-profile.self.cycles-pp.__list_add_valid 0.19 ± 3% -0.1 0.10 perf-profile.self.cycles-pp.__calc_delta 0.17 ± 2% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.inet_sendmsg 0.20 ± 4% -0.1 0.11 ± 6% perf-profile.self.cycles-pp.__x64_sys_sendto 0.19 ± 3% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.rcu_eqs_enter 0.12 ± 4% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.sched_ttwu_pending 0.23 ± 3% -0.1 0.15 ± 4% perf-profile.self.cycles-pp.activate_task 0.19 ± 4% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.check_preempt_curr 0.18 ± 2% -0.1 0.10 perf-profile.self.cycles-pp.__might_sleep 0.18 ± 2% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.__udp4_lib_lookup 0.19 ± 6% -0.1 0.11 ± 18% perf-profile.self.cycles-pp.ktime_get 0.14 ± 3% -0.1 0.06 ± 15% perf-profile.self.cycles-pp.validate_xmit_skb 0.16 ± 2% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.selinux_sock_rcv_skb_compat 0.15 ± 4% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.selinux_ip_postroute 0.13 ± 3% -0.1 0.05 perf-profile.self.cycles-pp._copy_from_iter_full 0.15 ± 3% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.rcu_needs_cpu 0.16 ± 2% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.prepare_to_wait_exclusive 0.17 ± 5% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.ip_local_deliver_finish 0.14 ± 5% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.netif_skb_features 0.10 ± 4% -0.1 0.03 ±100% perf-profile.self.cycles-pp.put_prev_entity 0.14 ± 3% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.__might_fault 0.14 ± 3% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.menu_reflect 0.16 ± 5% -0.1 0.09 perf-profile.self.cycles-pp.rb_erase_cached 0.16 ± 2% -0.1 0.10 ± 5% perf-profile.self.cycles-pp.ip_rcv_core 0.15 ± 3% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.ip_rcv 0.09 ± 4% -0.1 0.03 ±100% perf-profile.self.cycles-pp.udp_rcv 0.15 -0.1 0.08 ± 5% perf-profile.self.cycles-pp.ip_local_deliver 0.13 ± 3% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.09 -0.1 0.03 ±100% perf-profile.self.cycles-pp.selinux_ipv4_output 0.09 ± 7% -0.1 0.03 ±100% perf-profile.self.cycles-pp.deactivate_task 0.12 ± 3% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.rcu_idle_exit 0.12 -0.1 0.06 perf-profile.self.cycles-pp.selinux_sk_getsecid 0.11 ± 4% -0.1 0.05 perf-profile.self.cycles-pp.ip_make_skb 0.10 ± 11% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.__update_idle_core 0.08 -0.1 0.03 ±100% perf-profile.self.cycles-pp.ip_send_check 0.11 ± 4% -0.1 0.06 ± 11% perf-profile.self.cycles-pp.pm_qos_request 0.11 ± 7% -0.1 0.06 ± 11% perf-profile.self.cycles-pp.compute_score 0.11 ± 7% -0.1 0.05 ± 9% perf-profile.self.cycles-pp.sock_sendmsg 0.11 ± 7% -0.1 0.06 ± 9% perf-profile.self.cycles-pp.ip_generic_getfrag 0.11 ± 3% -0.1 0.06 perf-profile.self.cycles-pp.switch_mm 0.11 ± 4% -0.1 0.06 ± 9% perf-profile.self.cycles-pp.nf_hook_slow 0.08 ± 6% -0.1 0.03 ±100% perf-profile.self.cycles-pp.alloc_skb_with_frags 0.10 ± 7% -0.1 0.05 perf-profile.self.cycles-pp.__skb_try_recv_from_queue 0.10 -0.1 0.05 perf-profile.self.cycles-pp.dst_release 0.12 ± 10% -0.0 0.07 perf-profile.self.cycles-pp.import_single_range 0.11 ± 6% -0.0 0.06 perf-profile.self.cycles-pp.rb_insert_color_cached 0.10 ± 4% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.sock_alloc_send_pskb 0.12 ± 5% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.cpuidle_enter_state 0.09 ± 4% -0.0 0.05 perf-profile.self.cycles-pp.ip_finish_output 0.10 -0.0 0.06 perf-profile.self.cycles-pp.ip_send_skb 0.09 ± 7% -0.0 0.05 perf-profile.self.cycles-pp.schedule_idle 0.09 ± 4% -0.0 0.05 perf-profile.self.cycles-pp.get_next_timer_interrupt 0.09 ± 4% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.udp_queue_rcv_skb 0.08 ± 6% -0.0 0.04 ± 58% perf-profile.self.cycles-pp.tick_check_broadcast_expired 0.09 ± 5% -0.0 0.05 perf-profile.self.cycles-pp.dev_hard_start_xmit 0.08 ± 5% -0.0 0.05 perf-profile.self.cycles-pp.ip_setup_cork 0.10 ± 7% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.do_softirq_own_stack 0.07 -0.0 0.05 perf-profile.self.cycles-pp.receiver_wake_function 0.00 +0.1 0.10 ± 5% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.00 +0.1 0.10 ± 4% perf-profile.self.cycles-pp.cpumask_next 0.00 +1.0 1.01 perf-profile.self.cycles-pp.__percpu_counter_sum 30.01 +2.7 32.69 perf-profile.self.cycles-pp.intel_idle 0.06 ± 87% +30.2 30.31 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath *************************************************************************************************** lkp-bdw-ep2: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.2/25%/debian-x86_64-2018-04-03.cgz/300s/lkp-bdw-ep2/UDP_RR/netperf commit: 6da410d97f (" mlx5e-fixes-2018-09-05") b99259a614 ("net/sock: move memory_allocated over to percpu_counter variables") 6da410d97ffa486e b99259a61450bb6403bbbbf279 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x 9:4 -43% 7:4 perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout 3:4 -15% 3:4 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.switch_mm.__sched_text_start.schedule.schedule_timeout 26:4 -109% 21:4 perf-profile.children.cycles-pp.schedule_timeout 1:4 -11% 1:4 perf-profile.self.cycles-pp.schedule_timeout %stddev %change %stddev \ | \ 1930589 -13.5% 1670226 netperf.Throughput_total_tps 87754 -13.5% 75919 netperf.Throughput_tps 1426 -3.9% 1371 ± 2% netperf.time.percent_of_cpu_this_job_got 3911 -2.6% 3809 ± 2% netperf.time.system_time 385.12 -17.0% 319.68 ± 4% netperf.time.user_time 5.792e+08 -13.5% 5.011e+08 netperf.time.voluntary_context_switches 5.792e+08 -13.5% 5.011e+08 netperf.workload 5.792e+08 -12.5% 5.068e+08 softirqs.NET_RX 7590488 -13.1% 6595363 vmstat.system.cs 2095 ± 50% +740.9% 17617 ± 87% numa-meminfo.node0.Shmem 40809 ± 4% -14.3% 34971 ± 11% numa-meminfo.node1.SReclaimable 415.39 -1.8% 408.09 pmeter.Average_Active_Power 211.25 -11.9% 186.02 pmeter.performance_per_watt 18997 ± 6% +8.2% 20555 ± 3% sched_debug.cpu.load.avg 13008831 -13.3% 11273636 sched_debug.cpu.nr_switches.avg 1.133e+09 -12.8% 9.877e+08 cpuidle.C1.usage 8.879e+09 ± 18% -43.5% 5.013e+09 ± 54% cpuidle.C6.time 25831364 ± 3% -27.4% 18752863 cpuidle.POLL.usage 7.97 +2.5 10.43 mpstat.cpu.soft% 17.09 -2.4 14.66 mpstat.cpu.sys% 2.34 -0.3 2.00 ± 2% mpstat.cpu.usr% 1037 -3.8% 998.50 turbostat.Avg_MHz 1.133e+09 -12.8% 9.877e+08 turbostat.C1 33.21 ± 18% -14.4 18.80 ± 55% turbostat.C6% 0.96 ± 96% +201.3% 2.88 ± 30% turbostat.CPU%c3 216.23 -2.2% 211.46 turbostat.PkgWatt 523.00 ± 50% +742.2% 4404 ± 87% numa-vmstat.node0.nr_shmem 471766 ± 12% +27.2% 600222 ± 6% numa-vmstat.node0.numa_hit 454541 ± 12% +28.6% 584644 ± 7% numa-vmstat.node0.numa_local 10202 ± 4% -14.3% 8743 ± 11% numa-vmstat.node1.nr_slab_reclaimable 665328 ± 9% -20.2% 530868 ± 7% numa-vmstat.node1.numa_hit 532036 ± 11% -25.5% 396167 ± 9% numa-vmstat.node1.numa_local 3.597e+12 -6.9% 3.348e+12 perf-stat.branch-instructions 2.70 -0.2 2.54 perf-stat.branch-miss-rate% 9.714e+10 -12.4% 8.509e+10 perf-stat.branch-misses 3.173e+11 ± 2% -14.9% 2.701e+11 perf-stat.cache-references 2.317e+09 -13.3% 2.01e+09 perf-stat.context-switches 1.59 +3.4% 1.64 perf-stat.cpi 2.871e+13 -4.5% 2.742e+13 perf-stat.cpu-cycles 0.09 -0.0 0.09 perf-stat.dTLB-load-miss-rate% 4.816e+09 -10.7% 4.302e+09 perf-stat.dTLB-load-misses 5.29e+12 -8.4% 4.846e+12 perf-stat.dTLB-loads 0.00 ± 27% +0.0 0.00 ± 9% perf-stat.dTLB-store-miss-rate% 50665696 ± 27% +87.6% 95071493 ± 9% perf-stat.dTLB-store-misses 3.418e+12 -11.6% 3.023e+12 perf-stat.dTLB-stores 11.55 ± 5% +1.9 13.47 ± 7% perf-stat.iTLB-load-miss-rate% 4.068e+10 -16.8% 3.384e+10 ± 2% perf-stat.iTLB-loads 1.81e+13 -7.6% 1.673e+13 perf-stat.instructions 0.63 -3.3% 0.61 perf-stat.ipc 7.63e+08 ± 20% +87.8% 1.433e+09 ± 3% perf-stat.node-load-misses 86.73 ± 2% -27.7 59.01 perf-stat.node-store-miss-rate% 5.359e+08 ± 9% +17.5% 6.296e+08 perf-stat.node-store-misses 83392223 ± 25% +424.4% 4.373e+08 perf-stat.node-stores 31255 +6.8% 33385 perf-stat.path-length 13.66 ± 2% -2.4 11.21 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.45 ± 2% -2.4 11.05 ± 6% perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.88 ± 2% -2.1 9.73 ± 6% perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.67 ± 2% -2.1 9.58 ± 6% perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 9.09 ± 2% -1.6 7.45 ± 7% perf-profile.calltrace.cycles-pp.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 7.67 ± 2% -1.4 6.27 ± 7% perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom 9.02 ± 2% -1.3 7.69 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv 9.25 ± 2% -1.3 7.99 ± 6% perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish 8.48 ± 2% -1.2 7.24 ± 7% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_skb 8.14 ± 3% -1.1 6.99 ± 7% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.__udp_enqueue_schedule_skb 8.03 ± 3% -1.1 6.89 ± 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable 6.50 ± 2% -1.1 5.41 ± 7% perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg 6.11 ± 2% -1.0 5.12 ± 7% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 5.97 ± 2% -1.0 5.02 ± 7% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp 5.49 -0.9 4.60 ± 5% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 5.35 -0.9 4.48 ± 5% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary 4.79 -0.8 3.96 ± 6% perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 4.50 ± 2% -0.7 3.85 ± 7% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 4.24 ± 2% -0.6 3.62 ± 8% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 3.50 ± 3% -0.5 2.99 ± 8% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 2.62 ± 2% -0.5 2.16 ± 7% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets 2.88 -0.4 2.46 ± 6% perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 1.68 -0.4 1.29 ± 4% perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 2.78 ± 3% -0.4 2.42 ± 7% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.84 -0.4 1.48 ± 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 0.64 ± 3% -0.4 0.28 ±100% perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select 1.89 ± 3% -0.3 1.56 ± 7% perf-profile.calltrace.cycles-pp.__entry_SYSCALL_64_trampoline 2.04 ± 4% -0.3 1.73 ± 8% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb 1.29 ± 2% -0.3 1.00 ± 4% perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg 1.22 ± 2% -0.3 0.94 ± 4% perf-profile.calltrace.cycles-pp.ip_idents_reserve.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg 1.32 -0.3 1.04 ± 6% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle 2.11 ± 2% -0.3 1.83 ± 6% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 1.77 ± 3% -0.3 1.50 ± 8% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 1.49 -0.3 1.22 ± 5% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 1.58 ± 4% -0.2 1.36 ± 7% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb 1.60 -0.2 1.40 ± 4% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 1.50 ± 4% -0.2 1.30 ± 7% perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output 0.60 ± 2% -0.2 0.41 ± 58% perf-profile.calltrace.cycles-pp.selinux_socket_sock_rcv_skb.security_sock_rcv_skb.sk_filter_trim_cap.udp_queue_rcv_skb.__udp4_lib_rcv 1.48 ± 2% -0.2 1.29 ± 7% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg 1.41 ± 2% -0.2 1.23 ± 7% perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb 0.90 -0.2 0.73 ± 4% perf-profile.calltrace.cycles-pp._copy_to_iter.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 1.09 ± 3% -0.2 0.93 ± 8% perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle 0.81 ± 3% -0.2 0.66 ± 4% perf-profile.calltrace.cycles-pp.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 1.02 ± 3% -0.2 0.87 ± 7% perf-profile.calltrace.cycles-pp.switch_mm.__sched_text_start.schedule.schedule_timeout.__skb_wait_for_more_packets 0.94 -0.2 0.79 ± 6% perf-profile.calltrace.cycles-pp.move_addr_to_user.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 -0.2 0.57 ± 4% perf-profile.calltrace.cycles-pp.__switch_to_asm 0.90 ± 3% -0.2 0.75 ± 5% perf-profile.calltrace.cycles-pp.__consume_stateless_skb.udp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 0.70 ± 3% -0.2 0.54 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.71 ± 2% -0.1 0.58 ± 9% perf-profile.calltrace.cycles-pp.nf_hook_slow.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg 0.94 ± 3% -0.1 0.82 ± 8% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.99 ± 4% -0.1 0.88 ± 5% perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 1.07 ± 5% -0.1 0.96 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.73 ± 3% -0.1 0.62 ± 4% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 1.02 ± 5% -0.1 0.92 ± 5% perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.68 ± 2% -0.1 0.59 ± 7% perf-profile.calltrace.cycles-pp.sk_filter_trim_cap.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver 0.69 ± 4% -0.1 0.60 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.__skb_recv_udp.udp_recvmsg.inet_recvmsg.__sys_recvfrom 0.64 ± 2% -0.1 0.55 ± 6% perf-profile.calltrace.cycles-pp.security_sock_rcv_skb.sk_filter_trim_cap.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish 26.69 ± 2% +1.9 28.61 ± 5% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 26.51 ± 2% +1.9 28.45 ± 5% perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 25.75 ± 2% +2.1 27.80 ± 5% perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 25.20 ± 2% +2.2 27.38 ± 5% perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 18.57 ± 2% +3.3 21.89 ± 5% perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 18.32 ± 2% +3.4 21.68 ± 5% perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 17.95 ± 2% +3.4 21.32 ± 5% perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg 16.70 ± 2% +3.6 20.34 ± 5% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg 14.36 ± 2% +4.0 18.32 ± 5% perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb 14.21 ± 2% +4.0 18.22 ± 5% perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.ip_send_skb 13.99 ± 2% +4.0 18.01 ± 5% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2 14.04 ± 2% +4.0 18.07 ± 5% perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output 13.60 ± 2% +4.1 17.66 ± 5% perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 13.30 ± 2% +4.1 17.42 ± 5% perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq 0.00 +4.1 4.14 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb 12.53 ± 2% +4.2 16.77 ± 5% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack 12.28 ± 2% +4.2 16.52 ± 5% perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start 11.88 ± 3% +4.3 16.14 ± 5% perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 11.73 ± 3% +4.3 16.00 ± 5% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 11.40 ± 2% +4.4 15.77 ± 6% perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 0.00 +4.4 4.44 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb 10.89 ± 2% +4.4 15.34 ± 6% perf-profile.calltrace.cycles-pp.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv 9.96 ± 2% +4.6 14.57 ± 6% perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish.ip_local_deliver 0.00 +5.8 5.80 ± 5% perf-profile.calltrace.cycles-pp.__percpu_counter_sum.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv 0.00 +6.0 6.01 ± 5% perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__udp_enqueue_schedule_skb.udp_queue_rcv_skb.__udp4_lib_rcv.ip_local_deliver_finish 13.67 ± 2% -2.5 11.22 ± 6% perf-profile.children.cycles-pp.__x64_sys_recvfrom 13.46 ± 2% -2.4 11.06 ± 6% perf-profile.children.cycles-pp.__sys_recvfrom 11.88 ± 2% -2.1 9.73 ± 6% perf-profile.children.cycles-pp.inet_recvmsg 11.69 ± 2% -2.1 9.59 ± 6% perf-profile.children.cycles-pp.udp_recvmsg 11.45 -1.8 9.62 ± 6% perf-profile.children.cycles-pp.__sched_text_start 9.11 ± 2% -1.6 7.47 ± 7% perf-profile.children.cycles-pp.__skb_recv_udp 7.67 ± 2% -1.4 6.28 ± 7% perf-profile.children.cycles-pp.__skb_wait_for_more_packets 9.03 ± 2% -1.3 7.70 ± 6% perf-profile.children.cycles-pp.__wake_up_common_lock 9.25 ± 3% -1.3 7.99 ± 6% perf-profile.children.cycles-pp.sock_def_readable 8.48 ± 2% -1.2 7.24 ± 7% perf-profile.children.cycles-pp.__wake_up_common 8.06 ± 3% -1.2 6.90 ± 7% perf-profile.children.cycles-pp.try_to_wake_up 8.15 ± 3% -1.1 7.00 ± 7% perf-profile.children.cycles-pp.autoremove_wake_function 6.11 ± 2% -1.0 5.12 ± 7% perf-profile.children.cycles-pp.schedule 5.51 -0.9 4.62 ± 5% perf-profile.children.cycles-pp.schedule_idle 4.80 -0.8 3.96 ± 6% perf-profile.children.cycles-pp.ip_make_skb 4.50 ± 2% -0.6 3.86 ± 7% perf-profile.children.cycles-pp.ttwu_do_activate 4.25 ± 2% -0.6 3.63 ± 8% perf-profile.children.cycles-pp.enqueue_task_fair 3.66 ± 3% -0.5 3.14 ± 8% perf-profile.children.cycles-pp.enqueue_entity 2.65 ± 2% -0.5 2.18 ± 7% perf-profile.children.cycles-pp.dequeue_task_fair 2.32 ± 2% -0.4 1.88 ± 7% perf-profile.children.cycles-pp.pick_next_task_fair 2.38 ± 2% -0.4 1.95 ± 8% perf-profile.children.cycles-pp.dequeue_entity 2.90 -0.4 2.47 ± 6% perf-profile.children.cycles-pp.__ip_append_data 2.31 ± 3% -0.4 1.91 ± 6% perf-profile.children.cycles-pp.__entry_SYSCALL_64_trampoline 1.68 -0.4 1.29 ± 4% perf-profile.children.cycles-pp.__ip_make_skb 2.81 ± 3% -0.4 2.44 ± 7% perf-profile.children.cycles-pp.menu_select 2.60 -0.4 2.24 ± 5% perf-profile.children.cycles-pp.switch_mm_irqs_off 2.42 ± 3% -0.3 2.11 ± 8% perf-profile.children.cycles-pp.update_load_avg 2.04 ± 4% -0.3 1.74 ± 7% perf-profile.children.cycles-pp.__dev_queue_xmit 1.69 ± 2% -0.3 1.40 ± 5% perf-profile.children.cycles-pp.syscall_return_via_sysret 1.29 ± 2% -0.3 1.00 ± 4% perf-profile.children.cycles-pp.__ip_select_ident 1.23 ± 2% -0.3 0.94 ± 4% perf-profile.children.cycles-pp.ip_idents_reserve 1.34 -0.3 1.06 ± 6% perf-profile.children.cycles-pp.set_next_entity 2.12 ± 2% -0.3 1.84 ± 6% perf-profile.children.cycles-pp.sock_alloc_send_pskb 1.79 ± 3% -0.3 1.52 ± 8% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 1.48 ± 2% -0.3 1.23 ± 3% perf-profile.children.cycles-pp._raw_spin_lock 1.24 ± 2% -0.2 1.03 ± 6% perf-profile.children.cycles-pp.load_new_mm_cr3 1.58 ± 4% -0.2 1.37 ± 7% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.51 ± 4% -0.2 1.31 ± 7% perf-profile.children.cycles-pp.loopback_xmit 1.48 ± 2% -0.2 1.29 ± 7% perf-profile.children.cycles-pp.alloc_skb_with_frags 0.83 ± 3% -0.2 0.66 ± 8% perf-profile.children.cycles-pp.update_curr 1.42 ± 2% -0.2 1.24 ± 7% perf-profile.children.cycles-pp.__alloc_skb 0.73 ± 3% -0.2 0.55 ± 4% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.91 -0.2 0.73 ± 4% perf-profile.children.cycles-pp._copy_to_iter 1.07 ± 2% -0.2 0.90 ± 6% perf-profile.children.cycles-pp.switch_mm 1.10 ± 3% -0.2 0.94 ± 8% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.81 ± 3% -0.2 0.66 ± 4% perf-profile.children.cycles-pp.sock_wfree 0.95 -0.2 0.80 ± 6% perf-profile.children.cycles-pp.move_addr_to_user 0.90 ± 2% -0.2 0.75 ± 5% perf-profile.children.cycles-pp.__consume_stateless_skb 0.74 -0.2 0.59 ± 4% perf-profile.children.cycles-pp.__switch_to_asm 0.78 -0.1 0.64 ± 6% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.49 -0.1 0.35 ± 5% perf-profile.children.cycles-pp.update_ts_time_stats 0.66 ± 2% -0.1 0.53 ± 5% perf-profile.children.cycles-pp.__switch_to 0.84 ± 3% -0.1 0.70 ± 8% perf-profile.children.cycles-pp.nf_hook_slow 0.71 ± 3% -0.1 0.58 ± 9% perf-profile.children.cycles-pp.__next_timer_interrupt 0.95 ± 3% -0.1 0.82 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair 0.68 ± 3% -0.1 0.55 ± 12% perf-profile.children.cycles-pp.update_cfs_group 0.57 ± 4% -0.1 0.44 ± 7% perf-profile.children.cycles-pp.prepare_to_wait_exclusive 0.43 ± 7% -0.1 0.31 ± 10% perf-profile.children.cycles-pp.__enqueue_entity 0.69 ± 3% -0.1 0.56 ± 5% perf-profile.children.cycles-pp.__slab_free 0.43 -0.1 0.31 ± 7% perf-profile.children.cycles-pp.ip_generic_getfrag 0.74 ± 3% -0.1 0.63 ± 4% perf-profile.children.cycles-pp.poll_idle 0.39 ± 2% -0.1 0.28 ± 7% perf-profile.children.cycles-pp._copy_from_iter_full 1.07 ± 5% -0.1 0.97 ± 5% perf-profile.children.cycles-pp.ttwu_do_wakeup 1.04 ± 5% -0.1 0.94 ± 5% perf-profile.children.cycles-pp.check_preempt_curr 0.54 -0.1 0.44 ± 7% perf-profile.children.cycles-pp.update_rq_clock 0.57 ± 2% -0.1 0.47 ± 9% perf-profile.children.cycles-pp.selinux_ip_postroute 0.53 ± 2% -0.1 0.43 ± 4% perf-profile.children.cycles-pp.sock_def_write_space 0.68 ± 2% -0.1 0.59 ± 7% perf-profile.children.cycles-pp.sk_filter_trim_cap 0.57 -0.1 0.48 ± 5% perf-profile.children.cycles-pp.sched_clock_cpu 0.53 -0.1 0.44 ± 4% perf-profile.children.cycles-pp.copyout 0.64 ± 2% -0.1 0.55 ± 6% perf-profile.children.cycles-pp.security_sock_rcv_skb 0.56 -0.1 0.47 ± 4% perf-profile.children.cycles-pp.skb_set_owner_w 0.69 ± 5% -0.1 0.60 ± 8% perf-profile.children.cycles-pp._raw_spin_lock_bh 0.53 ± 2% -0.1 0.44 ± 4% perf-profile.children.cycles-pp.sched_clock 0.39 ± 5% -0.1 0.30 ± 8% perf-profile.children.cycles-pp.__update_load_avg_se 0.35 ± 4% -0.1 0.26 ± 6% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.42 -0.1 0.34 ± 5% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.61 ± 2% -0.1 0.53 ± 6% perf-profile.children.cycles-pp.selinux_socket_sock_rcv_skb 0.38 ± 6% -0.1 0.30 ± 14% perf-profile.children.cycles-pp.___perf_sw_event 0.50 ± 2% -0.1 0.42 ± 5% perf-profile.children.cycles-pp.native_sched_clock 0.40 ± 2% -0.1 0.33 ± 4% perf-profile.children.cycles-pp.nr_iowait_cpu 0.41 ± 3% -0.1 0.34 ± 8% perf-profile.children.cycles-pp.sockfd_lookup_light 0.45 ± 3% -0.1 0.37 ± 11% perf-profile.children.cycles-pp.selinux_ip_postroute_compat 0.45 -0.1 0.37 ± 8% perf-profile.children.cycles-pp.native_write_msr 0.56 ± 4% -0.1 0.48 ± 6% perf-profile.children.cycles-pp.finish_task_switch 0.43 -0.1 0.36 ± 9% perf-profile.children.cycles-pp.ksize 0.30 ± 6% -0.1 0.23 ± 5% perf-profile.children.cycles-pp.__udp4_lib_lookup 0.33 ± 3% -0.1 0.27 ± 8% perf-profile.children.cycles-pp.__fget_light 0.31 -0.1 0.25 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.20 -0.1 0.14 ± 3% perf-profile.children.cycles-pp.copyin 0.35 -0.1 0.28 ± 8% perf-profile.children.cycles-pp.__might_fault 0.28 ± 9% -0.1 0.22 ± 5% perf-profile.children.cycles-pp.pick_next_task_idle 0.21 ± 5% -0.1 0.15 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_stage2 0.49 -0.1 0.43 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.27 ± 3% -0.1 0.21 ± 7% perf-profile.children.cycles-pp.validate_xmit_skb 0.09 ± 8% -0.1 0.04 ± 57% perf-profile.children.cycles-pp.kfree 0.24 ± 5% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.rb_erase_cached 0.44 ± 2% -0.1 0.39 ± 5% perf-profile.children.cycles-pp.reweight_entity 0.26 ± 3% -0.1 0.21 ± 7% perf-profile.children.cycles-pp.enqueue_to_backlog 0.24 -0.0 0.20 ± 5% perf-profile.children.cycles-pp.security_socket_sendmsg 0.21 ± 5% -0.0 0.16 ± 6% perf-profile.children.cycles-pp.kmem_cache_free 0.15 ± 7% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.07 ± 12% -0.0 0.03 ±100% perf-profile.children.cycles-pp.rb_next 0.24 -0.0 0.20 ± 9% perf-profile.children.cycles-pp.___might_sleep 0.18 ± 12% -0.0 0.14 perf-profile.children.cycles-pp.inet_sendmsg 0.20 ± 4% -0.0 0.16 ± 6% perf-profile.children.cycles-pp.rcu_idle_exit 0.19 ± 2% -0.0 0.15 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.17 ± 7% -0.0 0.13 ± 11% perf-profile.children.cycles-pp.put_prev_task_fair 0.14 ± 10% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.compute_score 0.22 ± 4% -0.0 0.19 ± 10% perf-profile.children.cycles-pp.selinux_parse_skb 0.29 ± 4% -0.0 0.25 ± 9% perf-profile.children.cycles-pp.ipv4_mtu 0.16 ± 4% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.__calc_delta 0.24 -0.0 0.21 ± 8% perf-profile.children.cycles-pp.move_addr_to_kernel 0.22 ± 4% -0.0 0.19 ± 3% perf-profile.children.cycles-pp.sock_has_perm 0.23 ± 3% -0.0 0.19 ± 9% perf-profile.children.cycles-pp.__list_del_entry_valid 0.26 ± 3% -0.0 0.22 ± 4% perf-profile.children.cycles-pp.pick_next_entity 0.21 ± 3% -0.0 0.18 ± 7% perf-profile.children.cycles-pp._copy_from_user 0.19 ± 6% -0.0 0.16 ± 6% perf-profile.children.cycles-pp.__skb_try_recv_from_queue 0.20 ± 2% -0.0 0.17 ± 5% perf-profile.children.cycles-pp.rcu_eqs_enter 0.18 ± 2% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.security_sk_classify_flow 0.15 ± 7% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.menu_reflect 0.10 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.selinux_socket_sendmsg 0.19 ± 2% -0.0 0.16 ± 7% perf-profile.children.cycles-pp._copy_to_user 0.19 ± 4% -0.0 0.16 ± 8% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.17 ± 4% -0.0 0.15 ± 11% perf-profile.children.cycles-pp.__might_sleep 0.17 ± 6% -0.0 0.14 perf-profile.children.cycles-pp.sock_recvmsg 0.10 ± 7% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.ipv4_pktinfo_prepare 0.14 ± 7% -0.0 0.12 perf-profile.children.cycles-pp.security_socket_recvmsg 0.13 ± 3% -0.0 0.11 ± 6% perf-profile.children.cycles-pp.__get_user_4 0.13 ± 3% -0.0 0.11 ± 6% perf-profile.children.cycles-pp.selinux_sk_getsecid 0.11 ± 8% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.dst_release 0.10 ± 8% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.selinux_ipv4_postroute 0.11 ± 4% -0.0 0.10 ± 11% perf-profile.children.cycles-pp.pm_qos_request 0.11 ± 4% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.udp_rcv 0.09 ± 4% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.ip_send_check 0.05 +0.0 0.07 ± 13% perf-profile.children.cycles-pp.selinux_netlbl_sock_rcv_skb 0.06 ± 11% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rebalance_domains 0.07 ± 7% +0.0 0.09 ± 7% perf-profile.children.cycles-pp.ip_rcv_finish_core 0.09 ± 4% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.ip_rcv_finish 0.15 ± 8% +0.0 0.20 ± 8% perf-profile.children.cycles-pp.irq_exit 0.11 ± 31% +0.1 0.16 ± 22% perf-profile.children.cycles-pp.clockevents_program_event 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.netlbl_enabled 0.45 ± 5% +0.1 0.57 ± 7% perf-profile.children.cycles-pp.find_next_bit 0.00 +0.2 0.17 ± 4% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.00 +0.3 0.34 ± 5% perf-profile.children.cycles-pp.cpumask_next 26.71 ± 2% +1.9 28.63 ± 5% perf-profile.children.cycles-pp.__x64_sys_sendto 26.51 ± 2% +1.9 28.46 ± 5% perf-profile.children.cycles-pp.__sys_sendto 25.75 ± 2% +2.1 27.81 ± 5% perf-profile.children.cycles-pp.sock_sendmsg 25.23 ± 2% +2.2 27.40 ± 5% perf-profile.children.cycles-pp.udp_sendmsg 18.58 ± 2% +3.3 21.90 ± 5% perf-profile.children.cycles-pp.udp_send_skb 18.33 ± 2% +3.4 21.68 ± 5% perf-profile.children.cycles-pp.ip_send_skb 17.95 ± 2% +3.4 21.33 ± 5% perf-profile.children.cycles-pp.ip_output 16.73 ± 2% +3.6 20.36 ± 5% perf-profile.children.cycles-pp.ip_finish_output2 14.40 ± 2% +4.0 18.36 ± 5% perf-profile.children.cycles-pp.__local_bh_enable_ip 14.24 ± 2% +4.0 18.25 ± 5% perf-profile.children.cycles-pp.do_softirq 14.05 ± 2% +4.0 18.08 ± 5% perf-profile.children.cycles-pp.do_softirq_own_stack 14.12 ± 2% +4.0 18.16 ± 5% perf-profile.children.cycles-pp.__softirqentry_text_start 13.61 ± 2% +4.1 17.67 ± 5% perf-profile.children.cycles-pp.net_rx_action 13.31 ± 2% +4.1 17.43 ± 5% perf-profile.children.cycles-pp.process_backlog 1.50 ± 2% +4.2 5.68 ± 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.00 +4.2 4.19 ± 5% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 12.28 ± 2% +4.2 16.52 ± 5% perf-profile.children.cycles-pp.ip_rcv 12.53 ± 2% +4.2 16.78 ± 5% perf-profile.children.cycles-pp.__netif_receive_skb_one_core 11.89 ± 3% +4.3 16.15 ± 5% perf-profile.children.cycles-pp.ip_local_deliver 11.73 ± 3% +4.3 16.00 ± 5% perf-profile.children.cycles-pp.ip_local_deliver_finish 11.41 ± 2% +4.4 15.78 ± 6% perf-profile.children.cycles-pp.__udp4_lib_rcv 10.90 ± 2% +4.4 15.35 ± 6% perf-profile.children.cycles-pp.udp_queue_rcv_skb 9.97 ± 2% +4.6 14.58 ± 6% perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb 0.00 +5.8 5.84 ± 5% perf-profile.children.cycles-pp.__percpu_counter_sum 0.15 ± 3% +5.9 6.01 ± 5% perf-profile.children.cycles-pp.__sk_mem_raise_allocated 2.25 ± 3% -0.4 1.85 ± 6% perf-profile.self.cycles-pp.__entry_SYSCALL_64_trampoline 2.20 -0.3 1.89 ± 5% perf-profile.self.cycles-pp.__sched_text_start 1.69 ± 2% -0.3 1.39 ± 5% perf-profile.self.cycles-pp.syscall_return_via_sysret 1.22 ± 2% -0.3 0.93 ± 4% perf-profile.self.cycles-pp.ip_idents_reserve 1.43 ± 2% -0.3 1.16 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 1.24 ± 2% -0.2 1.03 ± 6% perf-profile.self.cycles-pp.load_new_mm_cr3 1.01 -0.2 0.82 ± 7% perf-profile.self.cycles-pp.do_idle 0.45 -0.2 0.29 ± 5% perf-profile.self.cycles-pp.__skb_wait_for_more_packets 1.05 ± 2% -0.2 0.90 ± 6% perf-profile.self.cycles-pp.update_load_avg 0.74 -0.2 0.59 ± 4% perf-profile.self.cycles-pp.__switch_to_asm 0.60 ± 2% -0.1 0.45 ± 6% perf-profile.self.cycles-pp.udp_sendmsg 0.76 -0.1 0.62 ± 5% perf-profile.self.cycles-pp.enqueue_entity 1.35 -0.1 1.21 ± 5% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.63 ± 2% -0.1 0.50 ± 5% perf-profile.self.cycles-pp.__switch_to 0.76 -0.1 0.62 ± 6% perf-profile.self.cycles-pp.copy_user_generic_unrolled 0.67 ± 3% -0.1 0.54 ± 12% perf-profile.self.cycles-pp.update_cfs_group 0.56 ± 3% -0.1 0.44 ± 8% perf-profile.self.cycles-pp.update_curr 0.68 ± 3% -0.1 0.56 ± 5% perf-profile.self.cycles-pp.__slab_free 0.42 ± 7% -0.1 0.30 ± 11% perf-profile.self.cycles-pp.__enqueue_entity 0.66 -0.1 0.54 ± 5% perf-profile.self.cycles-pp.set_next_entity 0.34 ± 2% -0.1 0.23 ± 3% perf-profile.self.cycles-pp.ip_output 0.59 ± 2% -0.1 0.49 ± 4% perf-profile.self.cycles-pp.enqueue_task_fair 0.53 ± 3% -0.1 0.42 ± 9% perf-profile.self.cycles-pp.pick_next_task_fair 0.51 -0.1 0.42 ± 7% perf-profile.self.cycles-pp.do_syscall_64 0.64 -0.1 0.55 ± 4% perf-profile.self.cycles-pp.poll_idle 0.54 -0.1 0.45 ± 5% perf-profile.self.cycles-pp.move_addr_to_user 0.63 ± 2% -0.1 0.54 ± 6% perf-profile.self.cycles-pp.select_task_rq_fair 0.40 ± 4% -0.1 0.31 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.56 -0.1 0.47 ± 4% perf-profile.self.cycles-pp.skb_set_owner_w 0.52 ± 2% -0.1 0.43 ± 5% perf-profile.self.cycles-pp.sock_def_write_space 0.20 ± 6% -0.1 0.11 ± 7% perf-profile.self.cycles-pp.check_preempt_curr 0.68 ± 5% -0.1 0.60 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_bh 0.53 ± 5% -0.1 0.45 ± 5% perf-profile.self.cycles-pp.udp_recvmsg 0.38 ± 5% -0.1 0.30 ± 8% perf-profile.self.cycles-pp.__update_load_avg_se 0.31 ± 2% -0.1 0.23 ± 8% perf-profile.self.cycles-pp.__skb_recv_udp 0.51 ± 2% -0.1 0.43 ± 10% perf-profile.self.cycles-pp.dequeue_entity 0.26 ± 4% -0.1 0.18 ± 6% perf-profile.self.cycles-pp.__ip_make_skb 0.48 ± 2% -0.1 0.41 ± 4% perf-profile.self.cycles-pp.native_sched_clock 0.29 ± 4% -0.1 0.22 ± 5% perf-profile.self.cycles-pp._copy_to_iter 0.44 -0.1 0.37 ± 7% perf-profile.self.cycles-pp.native_write_msr 0.44 -0.1 0.37 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.40 -0.1 0.33 ± 4% perf-profile.self.cycles-pp.nr_iowait_cpu 0.25 ± 4% -0.1 0.18 ± 8% perf-profile.self.cycles-pp.__wake_up_common 0.42 -0.1 0.35 ± 9% perf-profile.self.cycles-pp.ksize 0.32 -0.1 0.26 ± 9% perf-profile.self.cycles-pp.__next_timer_interrupt 0.28 ± 3% -0.1 0.21 ± 5% perf-profile.self.cycles-pp.sock_wfree 0.32 ± 2% -0.1 0.26 ± 8% perf-profile.self.cycles-pp.__fget_light 0.31 -0.1 0.25 ± 5% perf-profile.self.cycles-pp.update_rq_clock 0.49 -0.1 0.43 ± 6% perf-profile.self.cycles-pp.__udp_enqueue_schedule_skb 0.21 ± 4% -0.1 0.15 ± 5% perf-profile.self.cycles-pp.__sys_recvfrom 0.29 ± 5% -0.1 0.24 ± 6% perf-profile.self.cycles-pp.try_to_wake_up 0.09 ± 8% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.kfree 0.20 ± 7% -0.1 0.15 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_stage2 0.43 ± 2% -0.0 0.39 ± 6% perf-profile.self.cycles-pp.reweight_entity 0.35 ± 4% -0.0 0.31 ± 6% perf-profile.self.cycles-pp.__softirqentry_text_start 0.17 ± 2% -0.0 0.12 ± 8% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.25 ± 4% -0.0 0.21 ± 7% perf-profile.self.cycles-pp.net_rx_action 0.08 ± 5% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.tick_nohz_idle_exit 0.26 ± 4% -0.0 0.21 ± 7% perf-profile.self.cycles-pp.__sys_sendto 0.20 ± 4% -0.0 0.16 ± 5% perf-profile.self.cycles-pp.__x64_sys_recvfrom 0.12 ± 4% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.udp_queue_rcv_skb 0.27 -0.0 0.22 ± 7% perf-profile.self.cycles-pp.loopback_xmit 0.18 ± 7% -0.0 0.14 ± 5% perf-profile.self.cycles-pp.inet_recvmsg 0.14 ± 10% -0.0 0.10 ± 8% perf-profile.self.cycles-pp.compute_score 0.19 ± 6% -0.0 0.15 ± 8% perf-profile.self.cycles-pp.selinux_socket_sock_rcv_skb 0.15 ± 13% -0.0 0.11 ± 7% perf-profile.self.cycles-pp.ip_local_deliver_finish 0.20 ± 4% -0.0 0.16 ± 7% perf-profile.self.cycles-pp.rb_erase_cached 0.22 ± 5% -0.0 0.18 ± 8% perf-profile.self.cycles-pp.__alloc_skb 0.18 ± 2% -0.0 0.14 ± 9% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.20 ± 2% -0.0 0.16 ± 8% perf-profile.self.cycles-pp.rcu_eqs_enter 0.18 ± 3% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.17 ± 15% -0.0 0.13 perf-profile.self.cycles-pp.inet_sendmsg 0.29 ± 2% -0.0 0.26 ± 4% perf-profile.self.cycles-pp.process_backlog 0.23 ± 3% -0.0 0.19 ± 2% perf-profile.self.cycles-pp.pick_next_entity 0.15 ± 5% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.selinux_ip_postroute_compat 0.16 ± 6% -0.0 0.12 ± 6% perf-profile.self.cycles-pp.__calc_delta 0.11 ± 3% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.validate_xmit_skb 0.09 ± 5% -0.0 0.05 ± 9% perf-profile.self.cycles-pp._copy_from_iter_full 0.09 ± 5% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.selinux_socket_sendmsg 0.22 ± 5% -0.0 0.19 ± 3% perf-profile.self.cycles-pp.sock_has_perm 0.15 ± 5% -0.0 0.12 ± 4% perf-profile.self.cycles-pp.__udp4_lib_lookup 0.15 ± 4% -0.0 0.12 ± 10% perf-profile.self.cycles-pp.__might_sleep 0.09 ± 4% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.nf_hook_slow 0.13 ± 6% -0.0 0.10 ± 8% perf-profile.self.cycles-pp.prepare_to_wait_exclusive 0.15 ± 2% -0.0 0.12 ± 5% perf-profile.self.cycles-pp.enqueue_to_backlog 0.08 ± 5% -0.0 0.06 ± 15% perf-profile.self.cycles-pp.ip_make_skb 0.14 ± 6% -0.0 0.11 ± 3% perf-profile.self.cycles-pp.ip_rcv 0.13 -0.0 0.11 ± 4% perf-profile.self.cycles-pp.selinux_ip_postroute 0.10 -0.0 0.08 ± 6% perf-profile.self.cycles-pp.sock_sendmsg 0.10 -0.0 0.08 ± 10% perf-profile.self.cycles-pp.__might_fault 0.09 ± 9% -0.0 0.07 ± 13% perf-profile.self.cycles-pp.put_prev_task_fair 0.12 ± 3% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.rcu_idle_exit 0.12 ± 8% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.menu_reflect 0.06 -0.0 0.04 ± 57% perf-profile.self.cycles-pp.__ip_local_out 0.13 ± 3% -0.0 0.11 ± 7% perf-profile.self.cycles-pp.selinux_sock_rcv_skb_compat 0.08 ± 5% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.__skb_try_recv_from_queue 0.09 ± 4% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.selinux_ipv4_postroute 0.10 ± 9% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.dst_release 0.08 ± 5% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.schedule 0.13 ± 6% -0.0 0.11 ± 6% perf-profile.self.cycles-pp.__get_user_4 0.11 ± 7% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.netif_skb_features 0.11 ± 7% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.18 ± 4% -0.0 0.17 ± 5% perf-profile.self.cycles-pp.__x64_sys_sendto 0.11 ± 3% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.switch_mm 0.12 ± 3% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.selinux_sk_getsecid 0.09 ± 4% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.udp_rcv 0.07 -0.0 0.05 ± 9% perf-profile.self.cycles-pp.update_min_vruntime 0.09 ± 5% -0.0 0.07 perf-profile.self.cycles-pp.ip_send_check 0.08 ± 10% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.sockfd_lookup_light 0.06 ± 11% +0.0 0.09 ± 7% perf-profile.self.cycles-pp.ip_rcv_finish_core 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.netlbl_enabled 0.00 +0.1 0.06 ± 15% perf-profile.self.cycles-pp.hrtimer_get_next_event 0.21 ± 7% +0.1 0.29 ± 3% perf-profile.self.cycles-pp.sock_def_readable 0.00 +0.1 0.11 ± 7% perf-profile.self.cycles-pp.cpumask_next 0.38 ± 6% +0.1 0.50 ± 7% perf-profile.self.cycles-pp.find_next_bit 0.00 +0.2 0.17 ± 4% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.00 +1.1 1.07 ± 5% perf-profile.self.cycles-pp.__percpu_counter_sum 0.00 +4.2 4.19 ± 5% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen