Greeting, FYI, we noticed a -15.1% regression of aim7.jobs-per-min due to commit: commit: 44fe89de7d5157a4f31f13d94802c7619e23f462 ("btrfs: Do mandatory tree block check before submitting bio") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: aim7 on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory with following parameters: disk: 4BRD_12G md: RAID0 fs: btrfs test: sync_disk_rw load: 20 cpufreq_governor: performance test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | aim7: aim7.jobs-per-min -20.8% regression | | test machine | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory | | test parameters | cpufreq_governor=performance | | | disk=4BRD_12G | | | fs=btrfs | | | load=20 | | | md=RAID0 | | | test=sync_disk_rw | +------------------+-----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-skl-2sp7/sync_disk_rw/aim7 commit: a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()") 44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio") a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 50% 2:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 1204 -15.1% 1022 aim7.jobs-per-min 99.66 +18.3% 117.93 aim7.time.elapsed_time 99.66 +18.3% 117.93 aim7.time.elapsed_time.max 3014 ± 2% -3.1% 2921 aim7.time.minor_page_faults 1.236e+09 +44.6% 1.787e+09 ± 27% cpuidle.C1.time 90473 +15.8% 104758 ± 2% meminfo.AnonHugePages 102742 -16.5% 85841 ± 2% meminfo.max_used_kB 11.68 -1.8 9.85 mpstat.cpu.-1.sys% 0.07 ± 7% -0.0 0.05 ± 17% mpstat.cpu.-1.usr% 664.25 ± 4% +18.1% 784.25 ± 6% slabinfo.dmaengine-unmap-16.active_objs 664.25 ± 4% +18.1% 784.25 ± 6% slabinfo.dmaengine-unmap-16.num_objs 88.00 +2.3% 90.00 vmstat.cpu.id 346712 -15.5% 293086 vmstat.io.bo 558192 -15.4% 472096 vmstat.system.cs 88.42 +2.0% 90.19 iostat.cpu.idle 11.50 -15.3% 9.74 iostat.cpu.system 19892 -15.2% 16877 iostat.md0.w/s 451087 -15.4% 381436 iostat.md0.wkB/s 252.75 ± 2% -19.8% 202.67 ± 16% sched_debug.cfs_rq:/.removed.util_avg.max 57.62 ± 2% +71.4% 98.79 ± 51% sched_debug.cpu.cpu_load[1].max 47.75 ± 13% +81.6% 86.71 ± 43% sched_debug.cpu.cpu_load[2].max 7.35 ± 13% -16.4% 6.14 ± 4% sched_debug.cpu.cpu_load[4].avg 422.75 -14.3% 362.50 turbostat.Avg_MHz 14.00 -2.0 12.04 turbostat.Busy% 15068782 +15.6% 17424116 ± 2% turbostat.IRQ 163.49 -2.2% 159.86 turbostat.PkgWatt 516444 +2.0% 526872 proc-vmstat.nr_active_file 890319 +1.4% 902554 proc-vmstat.nr_file_pages 4668 -1.7% 4591 proc-vmstat.nr_inactive_anon 516444 +2.0% 526872 proc-vmstat.nr_zone_active_file 4668 -1.7% 4591 proc-vmstat.nr_zone_inactive_anon 288573 +14.8% 331243 proc-vmstat.pgfault 716355 +7.5% 770096 proc-vmstat.pgfree 24648391 -12.7% 21521966 ± 5% perf-stat.i.branch-misses 27.60 -1.9 25.66 ± 2% perf-stat.i.cache-miss-rate% 33238901 -22.5% 25763877 ± 2% perf-stat.i.cache-misses 1.163e+08 -13.8% 1.002e+08 ± 3% perf-stat.i.cache-references 570181 -15.7% 480734 perf-stat.i.context-switches 4.53 ± 3% -20.6% 3.60 ± 2% perf-stat.i.cpi 3.02e+10 -14.5% 2.581e+10 perf-stat.i.cpu-cycles 12.93 ± 11% -23.2% 9.92 ± 7% perf-stat.i.cpu-migrations 1.81e+09 +8.7% 1.968e+09 ± 3% perf-stat.i.dTLB-loads 1.018e+09 +10.9% 1.129e+09 perf-stat.i.dTLB-stores 8329645 -15.2% 7063121 perf-stat.i.iTLB-loads 6.662e+09 +9.5% 7.297e+09 perf-stat.i.instructions 93411 ± 7% +20.1% 112199 ± 6% perf-stat.i.instructions-per-iTLB-miss 0.24 ± 4% +21.3% 0.30 perf-stat.i.ipc 2719 -2.6% 2650 perf-stat.i.minor-faults 10179150 -22.8% 7861388 ± 2% perf-stat.i.node-load-misses 819501 -18.0% 672068 perf-stat.i.node-loads 79.98 -3.6 76.35 perf-stat.i.node-store-miss-rate% 3971266 -23.9% 3020404 perf-stat.i.node-store-misses 760058 ± 3% -7.2% 705596 perf-stat.i.node-stores 2720 -2.6% 2650 perf-stat.i.page-faults 17.46 -21.4% 13.73 ± 2% perf-stat.overall.MPKI 1.87 -0.3 1.59 ± 3% perf-stat.overall.branch-miss-rate% 28.58 -2.8 25.75 ± 4% perf-stat.overall.cache-miss-rate% 4.53 -22.0% 3.54 perf-stat.overall.cpi 908.69 +10.3% 1002 ± 2% perf-stat.overall.cycles-between-cache-misses 0.22 +28.1% 0.28 perf-stat.overall.ipc 83.93 -2.9 81.06 perf-stat.overall.node-store-miss-rate% 24408968 -12.5% 21345782 ± 5% perf-stat.ps.branch-misses 32896998 -22.4% 25539476 ± 2% perf-stat.ps.cache-misses 1.151e+08 -13.7% 99343827 ± 3% perf-stat.ps.cache-references 564308 -15.6% 476544 perf-stat.ps.context-switches 2.989e+10 -14.4% 2.558e+10 perf-stat.ps.cpu-cycles 12.79 ± 11% -23.1% 9.84 ± 7% perf-stat.ps.cpu-migrations 1.792e+09 +8.9% 1.951e+09 ± 3% perf-stat.ps.dTLB-loads 1.008e+09 +11.1% 1.12e+09 perf-stat.ps.dTLB-stores 8244056 -15.1% 7001712 perf-stat.ps.iTLB-loads 6.595e+09 +9.7% 7.235e+09 perf-stat.ps.instructions 2693 -2.4% 2628 perf-stat.ps.minor-faults 10074272 -22.6% 7792778 ± 2% perf-stat.ps.node-load-misses 811079 -17.9% 666222 perf-stat.ps.node-loads 3930342 -23.8% 2994037 perf-stat.ps.node-store-misses 752248 ± 3% -7.0% 699463 perf-stat.ps.node-stores 2693 -2.4% 2628 perf-stat.ps.page-faults 6.6e+11 +29.8% 8.564e+11 perf-stat.total.instructions 44.42 -3.1 41.36 ± 3% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 46.97 -2.2 44.77 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 50.60 -1.9 48.74 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64 49.88 -1.7 48.21 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 49.90 -1.7 48.24 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 49.90 -1.7 48.24 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 4.89 ± 12% -1.0 3.84 ± 6% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io 4.89 ± 12% -1.0 3.84 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper 5.75 ± 6% -0.7 5.07 ± 3% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents 2.90 ± 11% -0.6 2.30 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent 5.82 ± 2% -0.5 5.35 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 1.78 ± 14% -0.4 1.34 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent 4.25 ± 7% -0.4 3.82 perf-profile.calltrace.cycles-pp.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents 2.73 ± 7% -0.3 2.43 ± 4% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 8.78 -0.3 8.49 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 1.78 ± 8% -0.3 1.52 ± 3% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 8.54 -0.3 8.28 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node 2.15 ± 2% -0.2 1.99 ± 5% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 1.43 -0.1 1.36 ± 3% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum 1.02 ± 2% -0.1 0.97 perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode 1.02 ± 3% -0.0 0.97 perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_csum_file_blocks.btrfs_log_changed_extents 0.95 ± 5% +0.2 1.11 ± 7% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range 1.12 ± 2% +0.2 1.32 ± 6% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file 1.05 ± 3% +0.2 1.24 ± 5% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range 1.25 ± 2% +0.2 1.46 ± 5% perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter 0.68 ± 6% +0.2 0.92 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.45 ± 2% +0.2 1.70 ± 4% perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write 0.84 ± 13% +0.3 1.10 ± 3% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.70 ± 6% +0.3 0.95 ± 6% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.25 ±100% +0.3 0.56 ± 5% perf-profile.calltrace.cycles-pp.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages 1.55 ± 2% +0.3 1.87 ± 5% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file 1.55 ± 2% +0.3 1.87 ± 5% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops 1.57 ± 2% +0.3 1.89 ± 5% perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write 1.56 ± 2% +0.3 1.88 ± 5% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter 1.56 ± 2% +0.3 1.88 ± 5% perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.__vfs_write 0.26 ±100% +0.4 0.62 ± 6% perf-profile.calltrace.cycles-pp.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range 0.26 ±100% +0.4 0.62 ± 6% perf-profile.calltrace.cycles-pp.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range 0.00 +0.6 0.56 ± 5% perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages 0.00 +0.6 0.56 ± 5% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.btrfs_map_bio.btree_submit_bio_hook.submit_one_bio 0.00 +0.6 0.58 ± 6% perf-profile.calltrace.cycles-pp.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages.do_writepages 0.19 ±173% +0.6 0.78 ± 27% perf-profile.calltrace.cycles-pp.btrfs_next_old_leaf.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent 0.00 +0.8 0.84 ± 5% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook 0.00 +1.3 1.32 ± 6% perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 0.00 +1.4 1.39 ± 5% perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages 0.00 +1.4 1.39 ± 6% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages 0.75 ± 4% +1.5 2.23 ± 5% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log 0.76 ± 3% +1.5 2.24 ± 5% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write 0.75 ± 4% +1.5 2.23 ± 5% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter 0.75 ± 4% +1.5 2.23 ± 5% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file 0.38 ± 57% +1.6 1.96 ± 6% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range 0.38 ± 57% +1.6 1.96 ± 5% perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents 2.42 ± 2% +1.8 4.18 ± 4% perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write 38.62 +2.5 41.15 ± 3% perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write 39.95 +2.6 42.52 ± 3% perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 39.95 +2.6 42.52 ± 3% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64 39.99 +2.6 42.56 ± 3% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 39.98 +2.6 42.55 ± 3% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 40.19 +2.6 42.81 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 40.19 +2.6 42.81 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 97.75 ± 21% +77.5% 173.50 ± 21% interrupts.70:PCI-MSI.12589059-edge.eth3-TxRx-2 63.25 ± 9% +51.4% 95.75 ± 25% interrupts.71:PCI-MSI.12589060-edge.eth3-TxRx-3 202.50 +17.8% 238.50 interrupts.9:IO-APIC.9-fasteoi.acpi 88724 +12.8% 100056 interrupts.CAL:Function_call_interrupts 1255 +12.4% 1410 interrupts.CPU0.CAL:Function_call_interrupts 201136 +17.1% 235497 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts 202.50 +17.8% 238.50 interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi 1242 +13.2% 1407 interrupts.CPU1.CAL:Function_call_interrupts 200860 +16.6% 234199 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts 3195 ± 24% -60.2% 1273 ± 95% interrupts.CPU1.NMI:Non-maskable_interrupts 3195 ± 24% -60.2% 1273 ± 95% interrupts.CPU1.PMI:Performance_monitoring_interrupts 200768 +16.8% 234566 interrupts.CPU10.LOC:Local_timer_interrupts 200322 +17.1% 234482 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts 1174 ± 11% +18.3% 1389 interrupts.CPU12.CAL:Function_call_interrupts 200965 +16.4% 233853 interrupts.CPU12.LOC:Local_timer_interrupts 1209 ± 4% +15.4% 1396 interrupts.CPU13.CAL:Function_call_interrupts 199915 +17.6% 235085 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts 1187 ± 8% +18.1% 1403 interrupts.CPU14.CAL:Function_call_interrupts 200677 +17.2% 235290 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts 1208 ± 5% +13.2% 1367 ± 4% interrupts.CPU15.CAL:Function_call_interrupts 200566 +17.3% 235226 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts 200800 +17.1% 235099 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts 1217 ± 4% +15.3% 1403 interrupts.CPU17.CAL:Function_call_interrupts 200473 +16.6% 233816 interrupts.CPU17.LOC:Local_timer_interrupts 1246 +12.8% 1406 interrupts.CPU18.CAL:Function_call_interrupts 200863 +16.4% 233829 ± 3% interrupts.CPU18.LOC:Local_timer_interrupts 1243 +12.7% 1400 interrupts.CPU19.CAL:Function_call_interrupts 200576 +16.0% 232732 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts 1247 +12.9% 1407 interrupts.CPU2.CAL:Function_call_interrupts 200971 +16.9% 234895 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts 1247 +12.7% 1406 interrupts.CPU20.CAL:Function_call_interrupts 200473 +16.6% 233803 ± 3% interrupts.CPU20.LOC:Local_timer_interrupts 1247 +12.6% 1404 interrupts.CPU21.CAL:Function_call_interrupts 200696 +16.2% 233299 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts 1247 +13.1% 1411 interrupts.CPU22.CAL:Function_call_interrupts 200814 +16.4% 233758 ± 3% interrupts.CPU22.LOC:Local_timer_interrupts 4891 ± 12% -24.7% 3681 ± 12% interrupts.CPU22.RES:Rescheduling_interrupts 1247 +13.0% 1410 interrupts.CPU23.CAL:Function_call_interrupts 200828 +16.4% 233716 ± 3% interrupts.CPU23.LOC:Local_timer_interrupts 1247 +14.3% 1426 ± 3% interrupts.CPU24.CAL:Function_call_interrupts 200833 +16.5% 233997 ± 3% interrupts.CPU24.LOC:Local_timer_interrupts 1246 +12.9% 1407 interrupts.CPU25.CAL:Function_call_interrupts 201055 +16.3% 233920 ± 3% interrupts.CPU25.LOC:Local_timer_interrupts 1265 ± 2% +12.7% 1425 ± 2% interrupts.CPU26.CAL:Function_call_interrupts 201132 +15.9% 233180 ± 3% interrupts.CPU26.LOC:Local_timer_interrupts 1173 ± 10% +20.0% 1407 interrupts.CPU27.CAL:Function_call_interrupts 201166 +15.9% 233210 ± 3% interrupts.CPU27.LOC:Local_timer_interrupts 1263 ± 2% +11.4% 1407 interrupts.CPU28.CAL:Function_call_interrupts 201049 +16.3% 233848 ± 3% interrupts.CPU28.LOC:Local_timer_interrupts 1241 +13.5% 1409 interrupts.CPU29.CAL:Function_call_interrupts 200963 +16.3% 233774 ± 3% interrupts.CPU29.LOC:Local_timer_interrupts 97.75 ± 21% +77.5% 173.50 ± 21% interrupts.CPU3.70:PCI-MSI.12589059-edge.eth3-TxRx-2 1210 ± 5% +13.2% 1369 ± 3% interrupts.CPU3.CAL:Function_call_interrupts 199954 +17.1% 234167 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts 1181 ± 9% +19.1% 1407 interrupts.CPU30.CAL:Function_call_interrupts 200986 +16.4% 233892 ± 3% interrupts.CPU30.LOC:Local_timer_interrupts 1127 ± 18% +24.7% 1406 interrupts.CPU31.CAL:Function_call_interrupts 201110 +16.2% 233756 ± 3% interrupts.CPU31.LOC:Local_timer_interrupts 1249 +13.0% 1412 interrupts.CPU32.CAL:Function_call_interrupts 200832 +16.4% 233787 ± 3% interrupts.CPU32.LOC:Local_timer_interrupts 1249 +13.3% 1416 interrupts.CPU33.CAL:Function_call_interrupts 200969 +16.3% 233758 ± 3% interrupts.CPU33.LOC:Local_timer_interrupts 1246 +13.4% 1413 interrupts.CPU34.CAL:Function_call_interrupts 199891 +17.1% 234012 ± 3% interrupts.CPU34.LOC:Local_timer_interrupts 1234 +13.3% 1399 interrupts.CPU35.CAL:Function_call_interrupts 200721 +16.2% 233256 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts 3994 ± 14% -50.7% 1970 ± 48% interrupts.CPU35.RES:Rescheduling_interrupts 1242 +12.9% 1402 ± 2% interrupts.CPU36.CAL:Function_call_interrupts 199765 +17.7% 235071 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts 200843 +16.6% 234175 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts 1248 +13.4% 1415 interrupts.CPU38.CAL:Function_call_interrupts 200016 +17.0% 234095 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts 1249 +13.0% 1411 interrupts.CPU39.CAL:Function_call_interrupts 200714 +17.2% 235250 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts 63.25 ± 9% +51.4% 95.75 ± 25% interrupts.CPU4.71:PCI-MSI.12589060-edge.eth3-TxRx-3 1246 +13.9% 1419 ± 2% interrupts.CPU4.CAL:Function_call_interrupts 200637 +17.2% 235179 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts 200593 +17.2% 235093 ± 2% interrupts.CPU40.LOC:Local_timer_interrupts 1248 +13.0% 1410 interrupts.CPU41.CAL:Function_call_interrupts 200998 +16.6% 234380 interrupts.CPU41.LOC:Local_timer_interrupts 1249 +12.9% 1410 interrupts.CPU42.CAL:Function_call_interrupts 200934 +16.6% 234357 interrupts.CPU42.LOC:Local_timer_interrupts 1246 +13.2% 1411 interrupts.CPU43.CAL:Function_call_interrupts 200894 +16.9% 234942 ± 2% interrupts.CPU43.LOC:Local_timer_interrupts 1211 ± 5% +16.3% 1408 interrupts.CPU44.CAL:Function_call_interrupts 200154 +17.4% 235070 ± 2% interrupts.CPU44.LOC:Local_timer_interrupts 2454 ± 23% +50.0% 3682 ± 14% interrupts.CPU44.RES:Rescheduling_interrupts 1248 +13.0% 1410 interrupts.CPU45.CAL:Function_call_interrupts 200774 +16.7% 234259 ± 2% interrupts.CPU45.LOC:Local_timer_interrupts 1246 +13.1% 1410 interrupts.CPU46.CAL:Function_call_interrupts 201184 +16.8% 234966 ± 2% interrupts.CPU46.LOC:Local_timer_interrupts 1246 +13.1% 1409 interrupts.CPU47.CAL:Function_call_interrupts 200672 +16.7% 234200 interrupts.CPU47.LOC:Local_timer_interrupts 200070 +17.7% 235463 ± 2% interrupts.CPU48.LOC:Local_timer_interrupts 1251 +12.6% 1409 interrupts.CPU49.CAL:Function_call_interrupts 200232 +17.4% 235071 ± 2% interrupts.CPU49.LOC:Local_timer_interrupts 200450 +17.4% 235328 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts 1216 ± 5% +15.7% 1407 interrupts.CPU50.CAL:Function_call_interrupts 201051 +16.5% 234316 interrupts.CPU50.LOC:Local_timer_interrupts 1252 +12.6% 1410 interrupts.CPU51.CAL:Function_call_interrupts 200007 +17.5% 235046 ± 2% interrupts.CPU51.LOC:Local_timer_interrupts 1253 +12.4% 1408 interrupts.CPU52.CAL:Function_call_interrupts 200770 +17.0% 234811 ± 2% interrupts.CPU52.LOC:Local_timer_interrupts 1252 +12.2% 1405 interrupts.CPU53.CAL:Function_call_interrupts 200095 +17.4% 235004 ± 2% interrupts.CPU53.LOC:Local_timer_interrupts 1252 +12.6% 1410 interrupts.CPU54.CAL:Function_call_interrupts 200697 +16.5% 233818 ± 3% interrupts.CPU54.LOC:Local_timer_interrupts 1252 +12.6% 1409 interrupts.CPU55.CAL:Function_call_interrupts 200852 +16.5% 234024 ± 3% interrupts.CPU55.LOC:Local_timer_interrupts 1251 +12.5% 1407 interrupts.CPU56.CAL:Function_call_interrupts 200874 +16.4% 233828 ± 3% interrupts.CPU56.LOC:Local_timer_interrupts 1251 +12.3% 1404 interrupts.CPU57.CAL:Function_call_interrupts 200887 +16.5% 233974 ± 3% interrupts.CPU57.LOC:Local_timer_interrupts 1250 +12.8% 1411 interrupts.CPU58.CAL:Function_call_interrupts 200837 +16.4% 233702 ± 3% interrupts.CPU58.LOC:Local_timer_interrupts 1250 +12.8% 1410 interrupts.CPU59.CAL:Function_call_interrupts 199809 +17.1% 233932 ± 3% interrupts.CPU59.LOC:Local_timer_interrupts 1262 ± 2% +10.5% 1394 interrupts.CPU6.CAL:Function_call_interrupts 201108 +16.6% 234545 interrupts.CPU6.LOC:Local_timer_interrupts 1253 +12.6% 1411 interrupts.CPU60.CAL:Function_call_interrupts 200941 +16.4% 233857 ± 3% interrupts.CPU60.LOC:Local_timer_interrupts 2285 ± 53% -55.3% 1021 ± 85% interrupts.CPU60.NMI:Non-maskable_interrupts 2285 ± 53% -55.3% 1021 ± 85% interrupts.CPU60.PMI:Performance_monitoring_interrupts 1213 ± 4% +16.2% 1410 interrupts.CPU61.CAL:Function_call_interrupts 200849 +16.4% 233812 ± 3% interrupts.CPU61.LOC:Local_timer_interrupts 1250 +12.7% 1408 interrupts.CPU62.CAL:Function_call_interrupts 200858 +16.4% 233847 ± 3% interrupts.CPU62.LOC:Local_timer_interrupts 1249 +12.7% 1408 interrupts.CPU63.CAL:Function_call_interrupts 200846 +16.3% 233509 ± 3% interrupts.CPU63.LOC:Local_timer_interrupts 1249 +12.6% 1407 interrupts.CPU64.CAL:Function_call_interrupts 201010 +16.3% 233868 ± 3% interrupts.CPU64.LOC:Local_timer_interrupts 1249 +12.6% 1407 interrupts.CPU65.CAL:Function_call_interrupts 201188 +16.2% 233840 ± 3% interrupts.CPU65.LOC:Local_timer_interrupts 1249 +12.6% 1405 interrupts.CPU66.CAL:Function_call_interrupts 201142 +16.3% 233885 ± 3% interrupts.CPU66.LOC:Local_timer_interrupts 1249 +12.6% 1406 interrupts.CPU67.CAL:Function_call_interrupts 200840 +16.5% 233913 ± 3% interrupts.CPU67.LOC:Local_timer_interrupts 1249 +12.7% 1408 interrupts.CPU68.CAL:Function_call_interrupts 200815 +16.4% 233774 ± 3% interrupts.CPU68.LOC:Local_timer_interrupts 1249 +12.8% 1409 interrupts.CPU69.CAL:Function_call_interrupts 200983 +16.3% 233829 ± 3% interrupts.CPU69.LOC:Local_timer_interrupts 200912 +17.1% 235169 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts 1249 +13.0% 1412 interrupts.CPU70.CAL:Function_call_interrupts 200821 +16.4% 233823 ± 3% interrupts.CPU70.LOC:Local_timer_interrupts 1247 +12.9% 1408 interrupts.CPU71.CAL:Function_call_interrupts 200673 +16.4% 233586 ± 3% interrupts.CPU71.LOC:Local_timer_interrupts 3112 ± 28% -55.8% 1376 ± 92% interrupts.CPU71.NMI:Non-maskable_interrupts 3112 ± 28% -55.8% 1376 ± 92% interrupts.CPU71.PMI:Performance_monitoring_interrupts 2655 ± 18% +53.3% 4071 ± 21% interrupts.CPU71.RES:Rescheduling_interrupts 1088 ± 16% +25.1% 1361 ± 4% interrupts.CPU8.CAL:Function_call_interrupts 201291 +16.9% 235317 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts 201246 +16.4% 234250 interrupts.CPU9.LOC:Local_timer_interrupts 14452726 +16.7% 16865906 ± 2% interrupts.LOC:Local_timer_interrupts 25553 ± 7% +26.4% 32295 softirqs.CPU0.RCU 17578 +14.6% 20147 ± 2% softirqs.CPU0.SCHED 40851 +20.4% 49167 ± 3% softirqs.CPU0.TIMER 40546 +17.0% 47455 ± 5% softirqs.CPU1.TIMER 25814 +27.1% 32807 ± 10% softirqs.CPU10.RCU 38717 ± 2% +14.6% 44386 ± 5% softirqs.CPU10.TIMER 38245 +16.3% 44468 ± 5% softirqs.CPU11.TIMER 26439 ± 4% +21.3% 32083 ± 4% softirqs.CPU12.RCU 15582 ± 4% +14.4% 17827 ± 2% softirqs.CPU12.SCHED 38686 +17.5% 45474 ± 4% softirqs.CPU12.TIMER 25962 ± 4% +18.3% 30720 ± 4% softirqs.CPU13.RCU 16126 ± 5% +17.9% 19013 ± 5% softirqs.CPU13.SCHED 25846 ± 2% +20.4% 31108 ± 5% softirqs.CPU14.RCU 15629 ± 4% +21.3% 18953 ± 6% softirqs.CPU14.SCHED 37591 +17.9% 44326 ± 5% softirqs.CPU14.TIMER 24676 ± 2% +20.2% 29658 softirqs.CPU15.RCU 38217 +16.0% 44331 ± 4% softirqs.CPU15.TIMER 24680 ± 2% +18.6% 29278 ± 5% softirqs.CPU16.RCU 38783 +17.9% 45709 ± 4% softirqs.CPU16.TIMER 24738 +20.6% 29833 ± 10% softirqs.CPU17.RCU 39088 +16.3% 45476 ± 5% softirqs.CPU17.TIMER 27971 +24.7% 34880 ± 4% softirqs.CPU18.RCU 27800 +19.7% 33276 ± 2% softirqs.CPU19.RCU 16069 ± 5% +14.6% 18417 ± 3% softirqs.CPU19.SCHED 39339 +18.2% 46516 ± 5% softirqs.CPU19.TIMER 26084 +10.4% 28804 ± 5% softirqs.CPU2.RCU 16601 ± 8% +15.0% 19085 ± 10% softirqs.CPU2.SCHED 38625 +22.4% 47291 ± 5% softirqs.CPU2.TIMER 27472 +27.0% 34877 ± 7% softirqs.CPU20.RCU 16088 ± 3% +10.7% 17811 ± 4% softirqs.CPU20.SCHED 38845 +17.6% 45674 ± 3% softirqs.CPU20.TIMER 28205 ± 6% +20.2% 33906 ± 7% softirqs.CPU21.RCU 39324 +13.8% 44753 ± 4% softirqs.CPU21.TIMER 27282 ± 9% +20.0% 32736 ± 4% softirqs.CPU22.RCU 39542 +13.9% 45038 ± 3% softirqs.CPU22.TIMER 27295 ± 7% +22.3% 33387 ± 2% softirqs.CPU23.RCU 15694 ± 3% +13.2% 17772 ± 3% softirqs.CPU23.SCHED 37452 +16.5% 43645 ± 5% softirqs.CPU23.TIMER 25529 ± 10% +30.1% 33204 softirqs.CPU24.RCU 16068 ± 6% +13.8% 18289 ± 4% softirqs.CPU24.SCHED 39129 +29.4% 50648 ± 17% softirqs.CPU24.TIMER 27444 ± 3% +18.3% 32463 ± 4% softirqs.CPU25.RCU 38658 +14.6% 44286 ± 6% softirqs.CPU25.TIMER 27586 ± 2% +21.4% 33496 softirqs.CPU26.RCU 39631 +13.9% 45136 ± 4% softirqs.CPU26.TIMER 27618 ± 3% +15.9% 32011 ± 2% softirqs.CPU27.RCU 26884 +24.1% 33355 ± 7% softirqs.CPU28.RCU 39120 +14.6% 44813 ± 5% softirqs.CPU28.TIMER 27542 ± 4% +21.4% 33433 ± 5% softirqs.CPU29.RCU 38659 ± 2% +15.9% 44815 ± 5% softirqs.CPU29.TIMER 27539 ± 3% +14.2% 31446 softirqs.CPU3.RCU 16465 ± 4% +16.6% 19192 ± 6% softirqs.CPU3.SCHED 38875 +17.9% 45849 ± 5% softirqs.CPU3.TIMER 26139 +23.5% 32277 ± 4% softirqs.CPU30.RCU 39666 +11.6% 44285 ± 5% softirqs.CPU30.TIMER 26559 ± 2% +23.7% 32855 ± 8% softirqs.CPU31.RCU 16278 ± 2% +19.3% 19420 ± 10% softirqs.CPU31.SCHED 39178 +15.2% 45121 ± 4% softirqs.CPU31.TIMER 25781 +20.9% 31163 ± 2% softirqs.CPU32.RCU 37670 +13.6% 42811 ± 5% softirqs.CPU32.TIMER 27107 ± 5% +16.7% 31625 ± 4% softirqs.CPU33.RCU 16098 ± 2% +12.3% 18071 ± 2% softirqs.CPU33.SCHED 38565 +15.6% 44572 ± 4% softirqs.CPU33.TIMER 26241 ± 3% +18.9% 31209 ± 5% softirqs.CPU34.RCU 38223 +18.1% 45160 ± 5% softirqs.CPU34.TIMER 25360 ± 6% +27.0% 32210 ± 2% softirqs.CPU35.RCU 39986 ± 2% +13.8% 45494 ± 3% softirqs.CPU35.TIMER 26626 ± 2% +23.9% 32987 ± 4% softirqs.CPU36.RCU 16281 ± 5% +12.6% 18329 softirqs.CPU36.SCHED 27227 +23.8% 33703 ± 2% softirqs.CPU37.RCU 26701 ± 2% +29.1% 34462 ± 8% softirqs.CPU38.RCU 15157 ± 5% +15.5% 17507 ± 4% softirqs.CPU38.SCHED 37119 ± 4% +37.9% 51191 ± 27% softirqs.CPU38.TIMER 28007 ± 2% +21.4% 33990 ± 2% softirqs.CPU39.RCU 15700 ± 2% +11.8% 17545 ± 3% softirqs.CPU39.SCHED 37509 ± 2% +14.9% 43109 ± 6% softirqs.CPU39.TIMER 25877 +24.6% 32254 ± 3% softirqs.CPU4.RCU 38648 +16.9% 45191 ± 5% softirqs.CPU4.TIMER 27903 +17.8% 32867 ± 2% softirqs.CPU40.RCU 15256 ± 2% +12.8% 17202 ± 7% softirqs.CPU40.SCHED 36371 ± 2% +17.0% 42572 ± 4% softirqs.CPU40.TIMER 26560 ± 5% +19.9% 31840 ± 6% softirqs.CPU41.RCU 15067 ± 3% +13.8% 17142 ± 4% softirqs.CPU41.SCHED 36473 ± 3% +14.2% 41657 ± 4% softirqs.CPU41.TIMER 27250 +22.5% 33387 ± 2% softirqs.CPU42.RCU 15376 ± 4% +15.4% 17744 ± 3% softirqs.CPU42.SCHED 37838 +13.4% 42895 ± 3% softirqs.CPU42.TIMER 27137 ± 4% +23.2% 33425 softirqs.CPU43.RCU 27818 ± 3% +22.5% 34073 ± 6% softirqs.CPU44.RCU 14305 ± 4% +22.5% 17522 ± 2% softirqs.CPU44.SCHED 36151 ± 4% +16.7% 42187 ± 4% softirqs.CPU44.TIMER 25734 ± 3% +25.6% 32319 ± 4% softirqs.CPU45.RCU 14522 +14.8% 16674 ± 6% softirqs.CPU45.SCHED 36594 +15.8% 42364 ± 5% softirqs.CPU45.TIMER 28660 ± 3% +18.8% 34052 ± 7% softirqs.CPU46.RCU 15306 ± 4% +13.5% 17370 ± 7% softirqs.CPU46.SCHED 37095 ± 3% +14.3% 42388 ± 5% softirqs.CPU46.TIMER 28906 ± 5% +15.3% 33324 ± 7% softirqs.CPU47.RCU 14758 ± 5% +13.7% 16775 ± 5% softirqs.CPU47.SCHED 36736 ± 3% +14.2% 41969 ± 4% softirqs.CPU47.TIMER 28049 ± 4% +24.0% 34791 ± 2% softirqs.CPU48.RCU 14583 ± 4% +18.9% 17340 ± 8% softirqs.CPU48.SCHED 36095 ± 4% +19.5% 43117 ± 6% softirqs.CPU48.TIMER 27551 ± 5% +21.0% 33349 ± 4% softirqs.CPU49.RCU 36024 +18.1% 42545 ± 4% softirqs.CPU49.TIMER 25870 ± 2% +22.4% 31677 ± 3% softirqs.CPU5.RCU 15740 ± 6% +26.8% 19951 ± 9% softirqs.CPU5.SCHED 37622 +33.5% 50234 ± 18% softirqs.CPU5.TIMER 27424 +19.6% 32798 ± 2% softirqs.CPU50.RCU 14010 ± 2% +21.8% 17060 ± 6% softirqs.CPU50.SCHED 36485 ± 3% +14.7% 41862 ± 4% softirqs.CPU50.TIMER 27884 +18.9% 33160 ± 3% softirqs.CPU51.RCU 36212 ± 3% +16.6% 42210 ± 4% softirqs.CPU51.TIMER 27031 ± 7% +29.8% 35075 ± 5% softirqs.CPU52.RCU 14772 ± 2% +18.1% 17451 ± 4% softirqs.CPU52.SCHED 37031 ± 4% +17.9% 43648 ± 4% softirqs.CPU52.TIMER 25188 ± 12% +28.7% 32423 ± 3% softirqs.CPU53.RCU 16289 ± 6% +18.4% 19280 ± 5% softirqs.CPU53.SCHED 37616 ± 3% +35.6% 51010 ± 21% softirqs.CPU53.TIMER 26693 ± 3% +19.4% 31870 ± 2% softirqs.CPU54.RCU 38094 +16.5% 44379 ± 4% softirqs.CPU54.TIMER 26082 ± 2% +23.1% 32100 softirqs.CPU55.RCU 37589 ± 2% +16.5% 43779 ± 5% softirqs.CPU55.TIMER 26352 ± 2% +18.7% 31284 softirqs.CPU56.RCU 15591 ± 4% +15.9% 18066 ± 8% softirqs.CPU56.SCHED 37070 +16.8% 43309 ± 2% softirqs.CPU56.TIMER 27520 ± 4% +15.7% 31841 ± 5% softirqs.CPU57.RCU 37916 ± 2% +13.9% 43170 ± 2% softirqs.CPU57.TIMER 27067 +20.8% 32709 ± 7% softirqs.CPU58.RCU 15059 +19.1% 17935 ± 8% softirqs.CPU58.SCHED 37123 +13.8% 42241 softirqs.CPU58.TIMER 25199 ± 4% +19.6% 30126 ± 6% softirqs.CPU59.RCU 36384 ± 2% +16.8% 42479 ± 2% softirqs.CPU59.TIMER 25871 ± 3% +17.5% 30409 ± 3% softirqs.CPU6.RCU 16584 ± 2% +14.8% 19032 ± 5% softirqs.CPU6.SCHED 38784 +15.6% 44850 ± 4% softirqs.CPU6.TIMER 24777 ± 5% +15.0% 28502 ± 5% softirqs.CPU60.RCU 37468 +14.2% 42794 ± 6% softirqs.CPU60.TIMER 23108 ± 3% +30.1% 30065 ± 2% softirqs.CPU61.RCU 15059 ± 6% +16.5% 17540 ± 6% softirqs.CPU61.SCHED 37002 +16.8% 43227 ± 3% softirqs.CPU61.TIMER 23023 ± 7% +33.6% 30769 ± 6% softirqs.CPU62.RCU 37164 ± 2% +16.2% 43203 ± 2% softirqs.CPU62.TIMER 23069 ± 2% +25.1% 28850 softirqs.CPU63.RCU 14642 ± 5% +15.4% 16900 ± 8% softirqs.CPU63.SCHED 36254 ± 2% +15.4% 41854 ± 3% softirqs.CPU63.TIMER 24304 ± 2% +25.9% 30593 ± 2% softirqs.CPU64.RCU 14666 +15.5% 16938 ± 3% softirqs.CPU64.SCHED 37048 +15.7% 42877 ± 4% softirqs.CPU64.TIMER 24438 ± 9% +23.6% 30212 ± 5% softirqs.CPU65.RCU 37505 +12.8% 42288 ± 2% softirqs.CPU65.TIMER 24568 +20.4% 29569 softirqs.CPU66.RCU 15290 ± 5% +8.4% 16569 ± 4% softirqs.CPU66.SCHED 37381 +14.9% 42968 ± 3% softirqs.CPU66.TIMER 23253 ± 9% +29.7% 30150 ± 2% softirqs.CPU67.RCU 14645 ± 6% +13.4% 16613 ± 6% softirqs.CPU67.SCHED 37012 +14.1% 42221 ± 2% softirqs.CPU67.TIMER 36993 +15.0% 42536 ± 4% softirqs.CPU68.TIMER 24695 ± 7% +23.7% 30551 ± 3% softirqs.CPU69.RCU 14441 ± 3% +11.4% 16089 ± 4% softirqs.CPU69.SCHED 36999 +14.3% 42299 softirqs.CPU69.TIMER 26022 ± 3% +21.4% 31579 ± 2% softirqs.CPU7.RCU 38481 +18.7% 45661 ± 3% softirqs.CPU7.TIMER 25083 ± 8% +18.5% 29721 ± 2% softirqs.CPU70.RCU 14845 ± 7% +15.7% 17173 ± 4% softirqs.CPU70.SCHED 37094 +18.5% 43948 ± 3% softirqs.CPU70.TIMER 24438 ± 2% +20.6% 29461 ± 2% softirqs.CPU71.RCU 15821 ± 4% +13.6% 17978 ± 4% softirqs.CPU71.SCHED 26912 ± 7% +22.3% 32925 ± 6% softirqs.CPU8.RCU 39193 +15.2% 45167 ± 4% softirqs.CPU8.TIMER 25829 ± 2% +20.7% 31178 softirqs.CPU9.RCU 37434 ± 2% +16.6% 43665 ± 5% softirqs.CPU9.TIMER 1897274 +21.5% 2304570 ± 2% softirqs.RCU 1150548 +11.9% 1287616 softirqs.SCHED 2757329 +15.8% 3193002 ± 3% softirqs.TIMER aim7.jobs-per-min 1400 +-+------------------------------------------------------------------+ | | 1200 +-+++.++.++.+++.++.++ + ++.+ + +++.++ + ++.++.++.++.+++.++.++.+| | : : : : : : : : : | 1000 OO+OO OO OO OOO OO OO OOO OO OO OO OOO OO OO OOO O | | : : : : : : : :: : | 800 +-+ : : : : :: : : :: : | | : : : : :: : : :: : | 600 +-+ : ::: :: :: : :: : | | :: :: :: :: :: :: | 400 +-+ :: :: :: :: :: :: | | : :: :: :: : : | 200 +-+ : : : : : : | | : : : : : : | 0 +-+------------------------------------------------------------------+ aim7.time.elapsed_time 140 +-+-------------------------------------------------------------------+ | | 120 OO+OO OO OO OO OO O O OO OO O OO OOO OO OO OO OO O | | O O | 100 +-+++.++.++.++.++.+++ + +.++ + ++.++.+ + ++.++.+++.++.++.++.++.+| | : : : : : : : : : | 80 +-+ : : : : : : : : : | | : :: : : :: : : :: : | 60 +-+ : :: : : :: : : :: : | | : :: : : :: : :: :: | 40 +-+ :: :: :: :: :: :: | | :: :: :: :: :: :: | 20 +-+ : : : : : : | | : : : : : : | 0 +-+-------------------------------------------------------------------+ aim7.time.elapsed_time.max 140 +-+-------------------------------------------------------------------+ | | 120 OO+OO OO OO OO OO O O OO OO O OO OOO OO OO OO OO O | | O O | 100 +-+++.++.++.++.++.+++ + +.++ + ++.++.+ + ++.++.+++.++.++.++.++.+| | : : : : : : : : : | 80 +-+ : : : : : : : : : | | : :: : : :: : : :: : | 60 +-+ : :: : : :: : : :: : | | : :: : : :: : :: :: | 40 +-+ :: :: :: :: :: :: | | :: :: :: :: :: :: | 20 +-+ : : : : : : | | : : : : : : | 0 +-+-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-8/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7 commit: a92e9c3a3c ("btrfs: extent_io: Handle error better in extent_writepages()") 44fe89de7d ("btrfs: Do mandatory tree block check before submitting bio") a92e9c3a3cef4c0c 44fe89de7d5157a4f31f13d9480 ---------------- --------------------------- %stddev %change %stddev \ | \ 474.71 -20.8% 376.09 aim7.jobs-per-min 252.89 +26.2% 319.19 aim7.time.elapsed_time 252.89 +26.2% 319.19 aim7.time.elapsed_time.max 8376 ± 3% +9.4% 9167 ± 3% aim7.time.minor_page_faults 1760 +6.3% 1872 aim7.time.system_time 22.58 -1.5% 22.24 boot-time.dhcp 40945 ± 2% -22.3% 31813 ± 3% meminfo.max_used_kB 670014 +24.2% 832471 proc-vmstat.pgfault 1013780 +13.7% 1152374 proc-vmstat.pgfree 3.022e+09 ± 4% +50.2% 4.54e+09 ± 19% cpuidle.C1.time 34330913 ± 2% +16.1% 39848597 ± 9% cpuidle.C1.usage 2.611e+09 ± 16% +48.9% 3.887e+09 ± 3% cpuidle.C6.time 0.02 ± 15% -0.0 0.00 ±113% mpstat.cpu.-1.soft% 21.36 -3.4 17.98 mpstat.cpu.-1.sys% 0.14 -0.0 0.12 mpstat.cpu.-1.usr% 78.00 +4.8% 81.75 vmstat.cpu.id 138672 -21.1% 109467 vmstat.io.bo 299636 -21.0% 236781 vmstat.system.cs 78.64 +4.3% 81.99 iostat.cpu.idle 21.21 -15.7% 17.88 iostat.cpu.system 7947 -20.5% 6318 iostat.md0.w/s 180043 -20.9% 142377 iostat.md0.wkB/s 3098 ± 30% +270.4% 11475 ± 34% numa-meminfo.node0.Inactive(anon) 9526 +37.0% 13052 ± 15% numa-meminfo.node0.Mapped 5474 ± 26% +188.7% 15807 ± 23% numa-meminfo.node0.Shmem 75040 ± 2% +10.4% 82807 ± 7% numa-meminfo.node0.Slab 16192 ± 5% -51.6% 7838 ± 51% numa-meminfo.node1.Inactive(anon) 20174 ± 7% -50.1% 10076 ± 36% numa-meminfo.node1.Shmem 774.25 ± 30% +270.4% 2867 ± 34% numa-vmstat.node0.nr_inactive_anon 2474 ± 2% +35.6% 3354 ± 14% numa-vmstat.node0.nr_mapped 1367 ± 26% +188.9% 3950 ± 23% numa-vmstat.node0.nr_shmem 774.25 ± 30% +270.4% 2867 ± 34% numa-vmstat.node0.nr_zone_inactive_anon 4046 ± 5% -51.6% 1958 ± 51% numa-vmstat.node1.nr_inactive_anon 5042 ± 7% -50.1% 2518 ± 36% numa-vmstat.node1.nr_shmem 4046 ± 5% -51.6% 1958 ± 51% numa-vmstat.node1.nr_zone_inactive_anon 312.25 -15.1% 265.25 turbostat.Avg_MHz 25.87 -3.9 22.02 turbostat.Busy% 34326534 ± 2% +16.1% 39845190 ± 9% turbostat.C1 33.45 -4.6% 31.92 turbostat.CorWatt 21163841 +25.1% 26466542 turbostat.IRQ 59.45 -2.7% 57.87 turbostat.PkgWatt 32.25 -1.7% 31.71 turbostat.RAMWatt 20330 +26.1% 25630 turbostat.SMI 6.46 ± 2% -0.6 5.89 ± 3% perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe 2.75 ± 5% -0.6 2.19 ± 11% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 5.46 ± 2% -0.5 4.92 ± 3% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode 5.46 ± 2% -0.5 4.93 ± 3% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent 0.69 ± 9% -0.4 0.27 ±100% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write 1.74 ± 6% -0.4 1.39 ± 8% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.__vfs_write.vfs_write.ksys_write 0.63 ± 3% -0.3 0.30 ±100% perf-profile.calltrace.cycles-pp.submit_bio.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio.extent_writepages 2.35 ± 4% -0.2 2.13 ± 3% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents 0.92 ± 9% -0.1 0.80 ± 7% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 0.86 ± 10% -0.1 0.74 ± 9% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages 0.85 ± 9% -0.1 0.73 ± 8% perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages 0.70 ± 5% -0.1 0.57 ± 3% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 1.40 ± 5% +0.2 1.55 perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.65 ± 11% +0.3 0.94 ± 23% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.79 ± 8% +0.3 1.13 ± 21% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.26 ±100% +0.3 0.61 ± 7% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 1.44 ± 8% +0.4 1.87 ± 17% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.14 ±173% +0.6 0.70 ± 27% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 2.44 ± 5% +0.6 3.04 ± 7% perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write.vfs_write 0.00 +0.7 0.72 ± 2% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook 2.51 ± 6% +0.8 3.30 ± 16% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 2.71 ± 6% +0.8 3.53 ± 16% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 0.73 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.__vfs_write 0.72 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file 0.72 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter 0.72 ± 6% +1.0 1.69 ± 2% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log 0.00 +1.0 1.05 perf-profile.calltrace.cycles-pp.check_leaf.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 0.00 +1.1 1.08 perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages 0.00 +1.1 1.09 perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages 0.00 +1.5 1.46 perf-profile.calltrace.cycles-pp.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents 0.00 +1.5 1.46 perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range 71.60 ± 7% -16.2% 59.98 ± 8% sched_debug.cfs_rq:/.load_avg.avg 96.11 ± 10% -16.2% 80.58 ± 6% sched_debug.cfs_rq:/.load_avg.stddev 187997 -8.8% 171461 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg 226472 ± 2% -10.8% 202016 ± 6% sched_debug.cfs_rq:/.min_vruntime.max 17261 ± 7% -21.5% 13550 ± 15% sched_debug.cfs_rq:/.min_vruntime.stddev 8.89 ± 47% -64.7% 3.14 ± 57% sched_debug.cfs_rq:/.removed.load_avg.avg 204.15 -38.5% 125.58 ± 57% sched_debug.cfs_rq:/.removed.load_avg.max 40.32 ± 22% -51.4% 19.61 ± 57% sched_debug.cfs_rq:/.removed.load_avg.stddev 409.91 ± 47% -64.8% 144.41 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.avg 9437 -38.8% 5776 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.max 1859 ± 22% -51.5% 901.87 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 2.95 ± 44% -79.5% 0.60 ± 75% sched_debug.cfs_rq:/.removed.util_avg.avg 72.25 ± 8% -66.6% 24.17 ± 75% sched_debug.cfs_rq:/.removed.util_avg.max 13.53 ± 23% -72.1% 3.77 ± 75% sched_debug.cfs_rq:/.removed.util_avg.stddev 9.43 ± 5% +27.0% 11.97 ± 14% sched_debug.cfs_rq:/.runnable_load_avg.avg 103.80 ± 10% +62.1% 168.29 ± 27% sched_debug.cfs_rq:/.runnable_load_avg.max 22.41 ± 7% +39.0% 31.15 ± 20% sched_debug.cfs_rq:/.runnable_load_avg.stddev 12421 ± 59% +64.4% 20419 ± 40% sched_debug.cfs_rq:/.spread0.avg -24890 -69.6% -7575 sched_debug.cfs_rq:/.spread0.min 17263 ± 7% -21.5% 13551 ± 15% sched_debug.cfs_rq:/.spread0.stddev 325.68 ± 2% -14.6% 278.27 ± 3% sched_debug.cfs_rq:/.util_avg.avg 186.27 ± 3% -11.0% 165.84 ± 2% sched_debug.cfs_rq:/.util_avg.stddev 132.45 -12.4% 116.05 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.stddev 155113 +19.3% 185086 sched_debug.cpu.clock.avg 155121 +19.3% 185093 sched_debug.cpu.clock.max 155106 +19.3% 185079 sched_debug.cpu.clock.min 155113 +19.3% 185086 sched_debug.cpu.clock_task.avg 155121 +19.3% 185093 sched_debug.cpu.clock_task.max 155106 +19.3% 185079 sched_debug.cpu.clock_task.min 104.40 ± 7% +61.5% 168.62 ± 27% sched_debug.cpu.cpu_load[0].max 96.20 ± 8% +74.1% 167.46 ± 30% sched_debug.cpu.cpu_load[1].max 4871 +14.8% 5592 sched_debug.cpu.curr->pid.max 127731 +23.5% 157685 sched_debug.cpu.nr_load_updates.avg 134899 +20.9% 163058 sched_debug.cpu.nr_load_updates.max 125325 +24.0% 155428 sched_debug.cpu.nr_load_updates.min 155106 +19.3% 185079 sched_debug.cpu_clk 151513 +19.8% 181485 sched_debug.ktime 155537 +19.3% 185526 sched_debug.sched_clk 19.77 -15.1% 16.78 ± 3% perf-stat.i.MPKI 8.783e+08 -9.0% 7.989e+08 perf-stat.i.branch-instructions 19.44 -0.2 19.21 perf-stat.i.cache-miss-rate% 14932557 -19.1% 12081970 perf-stat.i.cache-misses 77250252 -18.0% 63363496 perf-stat.i.cache-references 302132 -21.2% 238146 perf-stat.i.context-switches 3.13 -11.6% 2.77 ± 2% perf-stat.i.cpi 1.254e+10 -15.4% 1.061e+10 perf-stat.i.cpu-cycles 229.12 ± 3% -21.3% 180.36 ± 3% perf-stat.i.cpu-migrations 840.12 +4.5% 877.89 perf-stat.i.cycles-between-cache-misses 1.129e+09 -5.4% 1.068e+09 perf-stat.i.dTLB-loads 0.10 ± 4% -0.0 0.09 ± 5% perf-stat.i.dTLB-store-miss-rate% 538827 ± 3% -16.1% 452028 ± 4% perf-stat.i.dTLB-store-misses 47.32 +3.3 50.67 ± 3% perf-stat.i.iTLB-load-miss-rate% 2556829 -20.4% 2034861 ± 2% perf-stat.i.iTLB-loads 4.153e+09 -4.5% 3.964e+09 perf-stat.i.instructions 0.33 +12.2% 0.37 ± 2% perf-stat.i.ipc 6672298 -19.6% 5364393 perf-stat.i.node-load-misses 7193961 -19.6% 5786904 perf-stat.i.node-loads 4765580 -20.6% 3784168 perf-stat.i.node-store-misses 6965089 -20.3% 5548679 perf-stat.i.node-stores 18.60 -14.0% 15.99 ± 2% perf-stat.overall.MPKI 19.33 -0.3 19.07 perf-stat.overall.cache-miss-rate% 3.02 -11.3% 2.68 ± 2% perf-stat.overall.cpi 839.71 +4.6% 878.19 perf-stat.overall.cycles-between-cache-misses 0.10 ± 4% -0.0 0.08 ± 5% perf-stat.overall.dTLB-store-miss-rate% 45.63 +3.8 49.47 ± 3% perf-stat.overall.iTLB-load-miss-rate% 0.33 +12.8% 0.37 ± 2% perf-stat.overall.ipc 8.748e+08 -9.0% 7.964e+08 perf-stat.ps.branch-instructions 14872846 -19.0% 12043692 perf-stat.ps.cache-misses 76941024 -17.9% 63162394 perf-stat.ps.cache-references 300917 -21.1% 237386 perf-stat.ps.context-switches 1.249e+10 -15.3% 1.058e+10 perf-stat.ps.cpu-cycles 228.20 ± 3% -21.2% 179.78 ± 3% perf-stat.ps.cpu-migrations 1.124e+09 -5.3% 1.065e+09 perf-stat.ps.dTLB-loads 536681 ± 3% -16.0% 450604 ± 4% perf-stat.ps.dTLB-store-misses 2546543 -20.3% 2028370 ± 2% perf-stat.ps.iTLB-loads 4.137e+09 -4.5% 3.952e+09 perf-stat.ps.instructions 6645469 -19.5% 5347283 perf-stat.ps.node-load-misses 7165042 -19.5% 5768454 perf-stat.ps.node-loads 4746415 -20.5% 3772102 perf-stat.ps.node-store-misses 6937096 -20.3% 5531006 perf-stat.ps.node-stores 1.049e+12 +20.4% 1.263e+12 perf-stat.total.instructions 160.75 ± 18% +49.5% 240.25 ± 27% interrupts.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5 123.75 +26.3% 156.25 interrupts.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 100313 ± 7% +13.4% 113737 interrupts.CAL:Function_call_interrupts 508519 +26.0% 640668 interrupts.CPU0.LOC:Local_timer_interrupts 2437 ± 12% +18.1% 2877 ± 4% interrupts.CPU1.CAL:Function_call_interrupts 508036 +26.1% 640579 interrupts.CPU1.LOC:Local_timer_interrupts 2521 ± 10% +14.3% 2882 ± 5% interrupts.CPU10.CAL:Function_call_interrupts 507477 +26.1% 639993 interrupts.CPU10.LOC:Local_timer_interrupts 2519 ± 2% +17.9% 2969 ± 6% interrupts.CPU11.CAL:Function_call_interrupts 508051 +26.1% 640531 interrupts.CPU11.LOC:Local_timer_interrupts 508353 +25.8% 639418 interrupts.CPU12.LOC:Local_timer_interrupts 508289 +26.0% 640385 interrupts.CPU13.LOC:Local_timer_interrupts 508136 +25.8% 639470 interrupts.CPU14.LOC:Local_timer_interrupts 2417 ± 5% +20.2% 2906 ± 6% interrupts.CPU15.CAL:Function_call_interrupts 508330 +25.9% 640229 interrupts.CPU15.LOC:Local_timer_interrupts 508351 +25.9% 639790 interrupts.CPU16.LOC:Local_timer_interrupts 2508 ± 10% +18.4% 2968 ± 6% interrupts.CPU17.CAL:Function_call_interrupts 508084 +26.0% 640420 interrupts.CPU17.LOC:Local_timer_interrupts 17327 ± 18% +26.1% 21856 ± 4% interrupts.CPU17.RES:Rescheduling_interrupts 508457 +25.8% 639737 interrupts.CPU18.LOC:Local_timer_interrupts 508140 +26.0% 640214 interrupts.CPU19.LOC:Local_timer_interrupts 2280 ± 8% +22.2% 2786 ± 5% interrupts.CPU2.CAL:Function_call_interrupts 508336 +26.0% 640548 interrupts.CPU2.LOC:Local_timer_interrupts 507912 +26.1% 640315 interrupts.CPU20.LOC:Local_timer_interrupts 2595 ± 7% +13.4% 2943 ± 5% interrupts.CPU21.CAL:Function_call_interrupts 508108 +26.1% 640485 interrupts.CPU21.LOC:Local_timer_interrupts 508306 +26.0% 640244 interrupts.CPU22.LOC:Local_timer_interrupts 1303 ± 34% -29.2% 922.75 ± 32% interrupts.CPU22.NMI:Non-maskable_interrupts 1303 ± 34% -29.2% 922.75 ± 32% interrupts.CPU22.PMI:Performance_monitoring_interrupts 2522 ± 9% +17.2% 2956 ± 4% interrupts.CPU23.CAL:Function_call_interrupts 508123 +26.1% 640526 interrupts.CPU23.LOC:Local_timer_interrupts 508372 +26.0% 640372 interrupts.CPU24.LOC:Local_timer_interrupts 507678 +26.1% 640435 interrupts.CPU25.LOC:Local_timer_interrupts 508375 +26.0% 640407 interrupts.CPU26.LOC:Local_timer_interrupts 2406 ± 11% +19.9% 2885 ± 5% interrupts.CPU27.CAL:Function_call_interrupts 508242 +26.0% 640265 interrupts.CPU27.LOC:Local_timer_interrupts 2449 ± 11% +21.8% 2982 ± 5% interrupts.CPU28.CAL:Function_call_interrupts 508292 +26.0% 640304 interrupts.CPU28.LOC:Local_timer_interrupts 2510 ± 6% +14.4% 2870 ± 5% interrupts.CPU29.CAL:Function_call_interrupts 508085 +26.0% 640167 interrupts.CPU29.LOC:Local_timer_interrupts 508172 +26.0% 640443 interrupts.CPU3.LOC:Local_timer_interrupts 508342 +25.9% 639891 interrupts.CPU30.LOC:Local_timer_interrupts 508009 +26.1% 640402 interrupts.CPU31.LOC:Local_timer_interrupts 2522 ± 11% +16.8% 2946 ± 4% interrupts.CPU32.CAL:Function_call_interrupts 508138 +25.9% 639578 interrupts.CPU32.LOC:Local_timer_interrupts 508010 +26.1% 640577 interrupts.CPU33.LOC:Local_timer_interrupts 508029 +25.9% 639405 interrupts.CPU34.LOC:Local_timer_interrupts 2464 ± 8% +15.3% 2840 ± 5% interrupts.CPU35.CAL:Function_call_interrupts 508216 +26.0% 640171 interrupts.CPU35.LOC:Local_timer_interrupts 160.75 ± 18% +49.5% 240.25 ± 27% interrupts.CPU36.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5 507990 +25.9% 639575 interrupts.CPU36.LOC:Local_timer_interrupts 508010 +26.0% 640200 interrupts.CPU37.LOC:Local_timer_interrupts 2536 ± 8% +13.9% 2888 ± 6% interrupts.CPU38.CAL:Function_call_interrupts 507853 +25.9% 639351 interrupts.CPU38.LOC:Local_timer_interrupts 2414 ± 7% +19.6% 2887 ± 7% interrupts.CPU39.CAL:Function_call_interrupts 507619 +26.0% 639416 interrupts.CPU39.LOC:Local_timer_interrupts 508318 +25.9% 640069 interrupts.CPU4.LOC:Local_timer_interrupts 2463 ± 12% +26.6% 3119 ± 2% interrupts.CPU5.CAL:Function_call_interrupts 508112 +26.1% 640497 interrupts.CPU5.LOC:Local_timer_interrupts 123.75 +26.3% 156.25 interrupts.CPU6.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 2351 ± 9% +21.5% 2856 ± 5% interrupts.CPU6.CAL:Function_call_interrupts 508563 +25.8% 639582 interrupts.CPU6.LOC:Local_timer_interrupts 2592 ± 4% +13.6% 2943 ± 4% interrupts.CPU7.CAL:Function_call_interrupts 508150 +26.0% 640378 interrupts.CPU7.LOC:Local_timer_interrupts 508498 +25.9% 640139 interrupts.CPU8.LOC:Local_timer_interrupts 2501 ± 10% +18.4% 2960 ± 7% interrupts.CPU9.CAL:Function_call_interrupts 508092 +26.0% 640432 interrupts.CPU9.LOC:Local_timer_interrupts 20326190 +26.0% 25605623 interrupts.LOC:Local_timer_interrupts 26401 ± 5% -9.9% 23785 ± 4% interrupts.NMI:Non-maskable_interrupts 26401 ± 5% -9.9% 23785 ± 4% interrupts.PMI:Performance_monitoring_interrupts 69901 ± 3% +27.9% 89422 ± 3% softirqs.CPU0.RCU 45503 ± 2% +18.1% 53752 softirqs.CPU0.SCHED 90281 ± 2% +17.4% 105987 ± 2% softirqs.CPU0.TIMER 72001 ± 4% +30.7% 94130 ± 2% softirqs.CPU1.RCU 42373 +18.9% 50374 ± 2% softirqs.CPU1.SCHED 89928 ± 3% +18.5% 106554 ± 4% softirqs.CPU1.TIMER 73216 ± 6% +24.5% 91156 ± 3% softirqs.CPU10.RCU 42693 +16.8% 49881 ± 2% softirqs.CPU10.SCHED 86975 ± 2% +19.4% 103822 ± 3% softirqs.CPU10.TIMER 71035 +31.2% 93194 ± 2% softirqs.CPU11.RCU 42794 +19.6% 51168 ± 2% softirqs.CPU11.SCHED 86533 +18.9% 102845 ± 4% softirqs.CPU11.TIMER 70358 ± 3% +33.3% 93810 ± 3% softirqs.CPU12.RCU 42548 +17.2% 49878 softirqs.CPU12.SCHED 87303 ± 2% +21.8% 106341 ± 3% softirqs.CPU12.TIMER 70364 ± 3% +31.7% 92648 ± 4% softirqs.CPU13.RCU 42421 +18.9% 50428 softirqs.CPU13.SCHED 86875 +19.5% 103817 ± 3% softirqs.CPU13.TIMER 69128 ± 2% +30.7% 90331 ± 3% softirqs.CPU14.RCU 42975 ± 2% +15.4% 49575 softirqs.CPU14.SCHED 84688 ± 3% +21.6% 102994 ± 3% softirqs.CPU14.TIMER 72010 ± 2% +30.6% 94068 ± 3% softirqs.CPU15.RCU 42893 +17.0% 50178 softirqs.CPU15.SCHED 86458 +20.4% 104117 ± 4% softirqs.CPU15.TIMER 70886 ± 2% +30.3% 92387 ± 2% softirqs.CPU16.RCU 42970 +16.3% 49975 softirqs.CPU16.SCHED 87944 ± 2% +19.7% 105244 ± 2% softirqs.CPU16.TIMER 70757 ± 2% +30.5% 92359 ± 2% softirqs.CPU17.RCU 42742 +17.7% 50310 ± 2% softirqs.CPU17.SCHED 87096 ± 2% +17.9% 102652 ± 3% softirqs.CPU17.TIMER 69077 ± 3% +32.6% 91630 ± 3% softirqs.CPU18.RCU 42942 +17.0% 50245 ± 2% softirqs.CPU18.SCHED 68722 ± 2% +29.7% 89150 ± 2% softirqs.CPU19.RCU 42964 ± 2% +17.2% 50334 ± 2% softirqs.CPU19.SCHED 84910 ± 4% +24.0% 105326 ± 7% softirqs.CPU19.TIMER 72687 ± 4% +25.6% 91315 ± 3% softirqs.CPU2.RCU 42688 +16.2% 49622 softirqs.CPU2.SCHED 88869 ± 5% +16.3% 103332 softirqs.CPU2.TIMER 70693 ± 2% +26.6% 89525 ± 4% softirqs.CPU20.RCU 42228 +17.5% 49611 ± 2% softirqs.CPU20.SCHED 88647 ± 2% +16.4% 103227 ± 2% softirqs.CPU20.TIMER 70857 ± 3% +32.1% 93633 ± 3% softirqs.CPU21.RCU 42878 +17.2% 50239 softirqs.CPU21.SCHED 87325 +18.2% 103214 ± 4% softirqs.CPU21.TIMER 71099 ± 3% +29.0% 91711 ± 3% softirqs.CPU22.RCU 42534 +17.3% 49888 softirqs.CPU22.SCHED 87698 ± 3% +19.3% 104642 softirqs.CPU22.TIMER 72098 ± 2% +30.3% 93930 ± 3% softirqs.CPU23.RCU 42771 +17.0% 50022 ± 2% softirqs.CPU23.SCHED 87931 +17.5% 103319 ± 3% softirqs.CPU23.TIMER 70804 ± 3% +33.4% 94432 ± 6% softirqs.CPU24.RCU 42554 ± 3% +16.4% 49516 ± 2% softirqs.CPU24.SCHED 71435 ± 3% +35.0% 96465 ± 3% softirqs.CPU25.RCU 42605 +16.9% 49815 softirqs.CPU25.SCHED 87163 +21.2% 105672 softirqs.CPU25.TIMER 70216 +31.0% 91964 ± 3% softirqs.CPU26.RCU 42147 +19.0% 50165 ± 2% softirqs.CPU26.SCHED 88215 ± 3% +36.3% 120217 ± 21% softirqs.CPU26.TIMER 70713 +32.5% 93695 ± 6% softirqs.CPU27.RCU 42528 +18.2% 50249 ± 2% softirqs.CPU27.SCHED 88110 +38.5% 122002 ± 24% softirqs.CPU27.TIMER 69654 ± 2% +34.1% 93398 ± 3% softirqs.CPU28.RCU 42052 +18.0% 49636 ± 2% softirqs.CPU28.SCHED 86876 ± 2% +21.6% 105634 ± 2% softirqs.CPU28.TIMER 70230 ± 2% +37.8% 96776 ± 5% softirqs.CPU29.RCU 42440 +17.0% 49673 ± 2% softirqs.CPU29.SCHED 87058 +18.9% 103502 ± 2% softirqs.CPU29.TIMER 70519 ± 3% +34.2% 94624 ± 2% softirqs.CPU3.RCU 43299 +16.2% 50322 ± 2% softirqs.CPU3.SCHED 87572 +19.2% 104386 ± 4% softirqs.CPU3.TIMER 70538 ± 2% +31.0% 92391 ± 5% softirqs.CPU30.RCU 42038 ± 2% +17.9% 49574 softirqs.CPU30.SCHED 86446 ± 3% +20.6% 104284 softirqs.CPU30.TIMER 71140 ± 2% +33.6% 95050 ± 3% softirqs.CPU31.RCU 42599 +16.5% 49608 ± 2% softirqs.CPU31.SCHED 86417 +19.6% 103393 ± 3% softirqs.CPU31.TIMER 59293 ± 2% +29.7% 76894 ± 4% softirqs.CPU32.RCU 42200 +17.8% 49711 softirqs.CPU32.SCHED 86726 +18.8% 103045 ± 3% softirqs.CPU32.TIMER 58507 +31.6% 76967 softirqs.CPU33.RCU 42587 +16.7% 49683 softirqs.CPU33.SCHED 86567 +19.1% 103101 ± 3% softirqs.CPU33.TIMER 59394 ± 2% +27.1% 75493 ± 5% softirqs.CPU34.RCU 42682 +16.6% 49753 softirqs.CPU34.SCHED 59949 ± 3% +29.2% 77459 softirqs.CPU35.RCU 42681 ± 2% +17.3% 50047 ± 2% softirqs.CPU35.SCHED 85785 +19.9% 102825 ± 2% softirqs.CPU35.TIMER 61089 +25.4% 76612 softirqs.CPU36.RCU 42662 +16.9% 49870 softirqs.CPU36.SCHED 87784 ± 2% +17.3% 102973 ± 3% softirqs.CPU36.TIMER 60018 +29.3% 77596 softirqs.CPU37.RCU 42378 ± 2% +18.0% 50012 softirqs.CPU37.SCHED 85919 +20.3% 103386 ± 2% softirqs.CPU37.TIMER 59781 ± 2% +28.6% 76877 softirqs.CPU38.RCU 42779 +17.1% 50105 softirqs.CPU38.SCHED 85246 ± 4% +20.7% 102859 ± 2% softirqs.CPU38.TIMER 59657 +26.9% 75699 ± 2% softirqs.CPU39.RCU 43011 ± 2% +17.6% 50580 ± 3% softirqs.CPU39.SCHED 98669 ± 20% +36.3% 134491 ± 24% softirqs.CPU39.TIMER 68912 ± 2% +32.7% 91479 ± 2% softirqs.CPU4.RCU 43101 +15.9% 49946 softirqs.CPU4.SCHED 86425 ± 5% +18.4% 102344 ± 3% softirqs.CPU4.TIMER 70893 ± 3% +32.1% 93618 ± 3% softirqs.CPU5.RCU 42977 +17.2% 50381 softirqs.CPU5.SCHED 87301 +17.7% 102788 ± 3% softirqs.CPU5.TIMER 71151 +25.0% 88965 ± 3% softirqs.CPU6.RCU 42864 +17.3% 50282 softirqs.CPU6.SCHED 87741 ± 2% +22.5% 107458 ± 4% softirqs.CPU6.TIMER 71242 ± 4% +30.5% 92999 ± 3% softirqs.CPU7.RCU 43014 ± 2% +17.7% 50619 softirqs.CPU7.SCHED 87907 +15.8% 101781 ± 5% softirqs.CPU7.TIMER 69638 ± 2% +32.6% 92309 ± 4% softirqs.CPU8.RCU 42479 +17.6% 49947 softirqs.CPU8.SCHED 86964 ± 2% +18.6% 103128 ± 3% softirqs.CPU8.TIMER 69585 ± 2% +36.1% 94684 ± 2% softirqs.CPU9.RCU 42667 +18.0% 50343 softirqs.CPU9.SCHED 86913 +19.2% 103596 ± 4% softirqs.CPU9.TIMER 2739272 +30.7% 3580875 ± 2% softirqs.RCU 1709285 +17.3% 2005342 softirqs.SCHED 3539953 +19.4% 4226728 ± 2% softirqs.TIMER Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen