Greeting, FYI, we noticed a -86.0% regression of aim7.jobs-per-min due to commit: commit: 4bc034d35377196c854236133b07730a777c4aba ("Revert "MD: fix lock contention for flush bios"") https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git for-5.2/block in testcase: aim7 on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory with following parameters: disk: 4BRD_12G md: RAID0 fs: xfs test: sync_disk_rw load: 300 cpufreq_governor: performance test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/300/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7 commit: 4f4fd7c579 ("Don't jump to compute_result state from check_result state") 4bc034d353 ("Revert "MD: fix lock contention for flush bios"") 4f4fd7c5798bbdd5 4bc034d35377196c854236133b0 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 30282 -86.0% 4234 aim7.jobs-per-min 59.51 +615.0% 425.47 aim7.time.elapsed_time 59.51 +615.0% 425.47 aim7.time.elapsed_time.max 661580 +3562.5% 24230472 aim7.time.involuntary_context_switches 21614 +255.2% 76779 ± 3% aim7.time.minor_page_faults 1450 +927.6% 14908 aim7.time.system_time 15.71 +10.6% 17.38 aim7.time.user_time 35275823 +506.3% 2.139e+08 aim7.time.voluntary_context_switches 41.04 ± 9% -78.6% 8.77 ± 3% iostat.cpu.idle 58.07 ± 6% +56.8% 91.07 iostat.cpu.system 465169 ± 6% -84.0% 74280 iostat.md0.w/s 987203 ± 6% -83.6% 161868 iostat.md0.wkB/s 2121170 +19.1% 2526977 numa-numastat.node0.local_node 2128094 +18.9% 2530235 numa-numastat.node0.numa_hit 2085181 +16.0% 2418368 numa-numastat.node1.local_node 2091318 +16.1% 2428211 numa-numastat.node1.numa_hit 39.33 ± 10% -31.0 8.34 ± 2% mpstat.cpu.all.idle% 0.13 ± 8% -0.1 0.00 ± 50% mpstat.cpu.all.iowait% 0.03 ± 76% -0.0 0.00 ± 35% mpstat.cpu.all.soft% 59.76 ± 6% +31.7 91.51 mpstat.cpu.all.sys% 0.75 ± 12% -0.6 0.15 ± 5% mpstat.cpu.all.usr% 67834 ± 9% +168.7% 182251 meminfo.AnonHugePages 38178 ± 8% +21.3% 46322 meminfo.Dirty 136721 ± 6% +28.6% 175883 meminfo.Inactive 116365 ± 7% +33.8% 155661 meminfo.Inactive(file) 32594 +46.9% 47891 ± 7% meminfo.Shmem 37104 ± 6% -84.2% 5856 meminfo.max_used_kB 40.67 ± 9% -79.7% 8.25 ± 5% vmstat.cpu.id 57.67 ± 6% +57.4% 90.75 vmstat.cpu.sy 984524 ± 6% -83.6% 161406 vmstat.io.bo 62.00 ± 16% +124.6% 139.25 vmstat.procs.r 741410 ± 6% -5.4% 701270 vmstat.system.cs 239168 ± 4% +12.6% 269384 ± 2% vmstat.system.in 3.594e+08 +145.6% 8.829e+08 cpuidle.C1.time 13251444 +193.7% 38919005 cpuidle.C1.usage 8302780 ± 3% +274.0% 31050667 ± 89% cpuidle.C1E.time 199348 ± 4% +155.4% 509135 ± 47% cpuidle.C1E.usage 2.557e+08 ± 28% -78.4% 55352560 ± 87% cpuidle.C3.time 721299 ± 23% -65.1% 251650 ± 71% cpuidle.C3.usage 290310 ± 3% +124.2% 650941 ± 10% cpuidle.POLL.time 131887 +19.0% 156975 ± 2% cpuidle.POLL.usage 32021 ± 88% +191.6% 93372 ± 11% numa-meminfo.node0.AnonHugePages 18534 ± 8% +28.0% 23724 numa-meminfo.node0.Dirty 67343 ± 8% +35.2% 91023 ± 2% numa-meminfo.node0.Inactive 58866 ± 7% +35.3% 79629 numa-meminfo.node0.Inactive(file) 36057 ± 80% +146.6% 88911 ± 12% numa-meminfo.node1.AnonHugePages 18655 ± 7% +21.3% 22635 numa-meminfo.node1.Dirty 69191 ± 11% +22.5% 84740 ± 2% numa-meminfo.node1.Inactive 57378 ± 7% +32.3% 75902 numa-meminfo.node1.Inactive(file) 3792 ± 11% +76.1% 6677 ± 38% numa-meminfo.node1.KernelStack 1117084 ± 7% +5.4% 1177261 ± 6% numa-meminfo.node1.MemUsed 4815 ± 9% +22.9% 5920 numa-vmstat.node0.nr_dirty 14782 ± 8% +34.8% 19925 numa-vmstat.node0.nr_inactive_file 14783 ± 8% +34.8% 19925 numa-vmstat.node0.nr_zone_inactive_file 4754 ± 8% +27.4% 6058 numa-vmstat.node0.nr_zone_write_pending 1355519 ± 6% +22.3% 1658417 ± 3% numa-vmstat.node0.numa_hit 1348673 ± 6% +22.7% 1655017 ± 3% numa-vmstat.node0.numa_local 4847 ± 6% +16.6% 5652 numa-vmstat.node1.nr_dirty 14427 ± 5% +31.6% 18989 numa-vmstat.node1.nr_inactive_file 3792 ± 10% +76.0% 6676 ± 38% numa-vmstat.node1.nr_kernel_stack 14427 ± 5% +31.6% 18989 numa-vmstat.node1.nr_zone_inactive_file 4712 ± 5% +22.9% 5792 numa-vmstat.node1.nr_zone_write_pending 1279822 ± 9% +24.8% 1597807 ± 3% numa-vmstat.node1.numa_hit 1083191 ± 10% +29.0% 1397507 ± 4% numa-vmstat.node1.numa_local 1706 ± 6% +61.4% 2754 turbostat.Avg_MHz 61.91 ± 6% +30.2 92.13 turbostat.Busy% 13247955 +193.7% 38914065 turbostat.C1 13.47 ± 7% -8.3 5.17 turbostat.C1% 199030 ± 4% +155.6% 508725 ± 47% turbostat.C1E 720658 ± 23% -65.2% 250930 ± 71% turbostat.C3 9.40 ± 24% -9.1 0.32 ± 87% turbostat.C3% 15.30 ± 18% -13.1 2.25 ± 11% turbostat.C6% 27.43 ± 16% -77.1% 6.27 ± 6% turbostat.CPU%c1 3.55 ± 37% -94.4% 0.20 ±115% turbostat.CPU%c3 7.11 ± 18% -80.4% 1.39 ± 31% turbostat.CPU%c6 99.14 ± 4% +40.4% 139.14 turbostat.CorWatt 63.67 ± 5% +10.7% 70.50 turbostat.CoreTmp 16370255 +605.1% 1.154e+08 ± 2% turbostat.IRQ 7.31 ± 2% -87.4% 0.92 ± 30% turbostat.Pkg%pc2 63.33 ± 5% +11.3% 70.50 turbostat.PkgTmp 126.14 ± 3% +32.0% 166.50 turbostat.PkgWatt 38.35 -10.7% 34.26 turbostat.RAMWatt 5373 ± 5% +535.9% 34170 turbostat.SMI 63371 +8.7% 68901 proc-vmstat.nr_active_anon 60172 +3.1% 62025 proc-vmstat.nr_anon_pages 9332 ± 7% +24.1% 11581 proc-vmstat.nr_dirty 350500 +3.9% 364042 proc-vmstat.nr_file_pages 29173 ± 7% +33.3% 38875 proc-vmstat.nr_inactive_file 13447 ± 2% +6.4% 14313 proc-vmstat.nr_kernel_stack 6155 -2.9% 5977 proc-vmstat.nr_mapped 8144 +47.1% 11976 ± 7% proc-vmstat.nr_shmem 16872 +2.2% 17235 proc-vmstat.nr_slab_reclaimable 20465 +4.6% 21403 proc-vmstat.nr_slab_unreclaimable 63371 +8.7% 68901 proc-vmstat.nr_zone_active_anon 29173 ± 7% +33.3% 38875 proc-vmstat.nr_zone_inactive_file 8903 ± 6% +33.1% 11852 proc-vmstat.nr_zone_write_pending 5134 ± 7% +1135.3% 63418 ± 8% proc-vmstat.numa_hint_faults 2712 ± 9% +1452.4% 42101 ± 11% proc-vmstat.numa_hint_faults_local 4245003 +17.4% 4985607 proc-vmstat.numa_hit 4231936 +17.5% 4972502 proc-vmstat.numa_local 2422 ± 6% +316.6% 10089 ± 2% proc-vmstat.numa_pages_migrated 6322 ± 14% +944.4% 66031 ± 11% proc-vmstat.numa_pte_updates 4423 ± 3% +44.7% 6400 ± 13% proc-vmstat.pgactivate 4263668 +18.3% 5042078 proc-vmstat.pgalloc_normal 211570 ± 5% +452.3% 1168563 proc-vmstat.pgfault 4180467 +18.7% 4964108 proc-vmstat.pgfree 2422 ± 6% +316.6% 10089 ± 2% proc-vmstat.pgmigrate_success 66934980 +3.3% 69136453 proc-vmstat.pgpgout 1029 ± 4% +16.6% 1200 ± 4% slabinfo.UNIX.active_objs 1029 ± 4% +16.6% 1200 ± 4% slabinfo.UNIX.num_objs 2102 +17.3% 2464 slabinfo.ebitmap_node.active_objs 2102 +17.3% 2464 slabinfo.ebitmap_node.num_objs 1485 +15.4% 1715 slabinfo.ip6-frags.active_objs 1485 +15.4% 1715 slabinfo.ip6-frags.num_objs 783.33 +25.4% 982.00 slabinfo.kmalloc-4k.active_objs 796.00 +26.8% 1009 slabinfo.kmalloc-4k.num_objs 4231 +43.1% 6056 ± 2% slabinfo.kmalloc-96.active_objs 4326 ± 2% +40.2% 6066 ± 2% slabinfo.kmalloc-96.num_objs 94.67 ± 14% +201.6% 285.50 ± 8% slabinfo.nfs_commit_data.active_objs 94.67 ± 14% +201.6% 285.50 ± 8% slabinfo.nfs_commit_data.num_objs 61.67 ± 9% +252.3% 217.25 ± 12% slabinfo.nfs_read_data.active_objs 61.67 ± 9% +252.3% 217.25 ± 12% slabinfo.nfs_read_data.num_objs 331.33 ± 12% +37.6% 455.75 ± 16% slabinfo.skbuff_ext_cache.active_objs 331.33 ± 12% +37.6% 455.75 ± 16% slabinfo.skbuff_ext_cache.num_objs 296.67 ± 11% +47.4% 437.25 ± 11% slabinfo.skbuff_fclone_cache.active_objs 296.67 ± 11% +47.4% 437.25 ± 11% slabinfo.skbuff_fclone_cache.num_objs 1861 ± 7% +13.1% 2104 ± 3% slabinfo.sock_inode_cache.active_objs 1861 ± 7% +13.1% 2104 ± 3% slabinfo.sock_inode_cache.num_objs 1730 +14.9% 1988 slabinfo.xfrm_dst_cache.active_objs 1730 +14.9% 1988 slabinfo.xfrm_dst_cache.num_objs 1210 +15.9% 1402 slabinfo.xfs_btree_cur.active_objs 1210 +15.9% 1402 slabinfo.xfs_btree_cur.num_objs 1124 +14.5% 1287 slabinfo.xfs_buf_item.active_objs 1124 +14.5% 1287 slabinfo.xfs_buf_item.num_objs 1197 +18.6% 1420 slabinfo.xfs_efd_item.active_objs 1197 +18.6% 1420 slabinfo.xfs_efd_item.num_objs 1147 ± 2% +16.8% 1339 slabinfo.xfs_inode.active_objs 1147 ± 2% +16.8% 1339 slabinfo.xfs_inode.num_objs 27.09 ± 12% -72.5% 7.44 ± 7% perf-stat.i.MPKI 4.056e+09 ± 4% +41.5% 5.739e+09 perf-stat.i.branch-instructions 3.69 ± 17% -3.0 0.68 ± 8% perf-stat.i.branch-miss-rate% 50722991 ± 3% -54.4% 23140156 perf-stat.i.branch-misses 19.17 -2.7 16.51 perf-stat.i.cache-miss-rate% 54086897 ± 5% -54.7% 24479387 perf-stat.i.cache-misses 2.929e+08 ± 4% -48.7% 1.501e+08 perf-stat.i.cache-references 772970 ± 5% -8.9% 704502 perf-stat.i.context-switches 6.913e+10 ± 4% +58.8% 1.098e+11 perf-stat.i.cpu-cycles 89123 ± 5% -59.7% 35907 perf-stat.i.cpu-migrations 1133 ± 2% +289.5% 4415 perf-stat.i.cycles-between-cache-misses 1.03 -0.7 0.35 ± 7% perf-stat.i.dTLB-load-miss-rate% 55146957 ± 5% -62.3% 20764703 ± 5% perf-stat.i.dTLB-load-misses 5.071e+09 ± 4% +21.3% 6.15e+09 perf-stat.i.dTLB-loads 0.17 ± 2% +0.0 0.20 ± 5% perf-stat.i.dTLB-store-miss-rate% 3860806 ± 4% -57.1% 1657096 ± 6% perf-stat.i.dTLB-store-misses 2.019e+09 ± 4% -58.5% 8.387e+08 perf-stat.i.dTLB-stores 87.20 +3.3 90.51 perf-stat.i.iTLB-load-miss-rate% 16459537 ± 5% -63.6% 5994301 perf-stat.i.iTLB-load-misses 2753032 ± 3% -77.2% 628945 ± 5% perf-stat.i.iTLB-loads 1.833e+10 ± 4% +30.4% 2.389e+10 perf-stat.i.instructions 1127 ± 2% +250.8% 3956 ± 2% perf-stat.i.instructions-per-iTLB-miss 0.24 ± 2% -10.3% 0.22 perf-stat.i.ipc 2981 -10.9% 2656 perf-stat.i.minor-faults 43.54 +3.9 47.45 perf-stat.i.node-load-miss-rate% 19607511 ± 7% -30.5% 13627815 perf-stat.i.node-load-misses 25859572 ± 6% -41.9% 15014790 perf-stat.i.node-loads 18026722 ± 5% -69.0% 5593547 perf-stat.i.node-store-misses 28667327 ± 5% -67.0% 9470534 perf-stat.i.node-stores 2981 -10.9% 2656 perf-stat.i.page-faults 15.98 -60.7% 6.28 perf-stat.overall.MPKI 1.25 -0.8 0.40 perf-stat.overall.branch-miss-rate% 18.46 -2.2 16.31 perf-stat.overall.cache-miss-rate% 3.77 +21.8% 4.60 perf-stat.overall.cpi 1278 +250.9% 4485 perf-stat.overall.cycles-between-cache-misses 1.08 -0.7 0.34 ± 5% perf-stat.overall.dTLB-load-miss-rate% 85.65 +4.9 90.50 perf-stat.overall.iTLB-load-miss-rate% 1114 +257.8% 3986 perf-stat.overall.instructions-per-iTLB-miss 0.27 -17.9% 0.22 perf-stat.overall.ipc 43.11 +4.5 47.58 perf-stat.overall.node-load-miss-rate% 38.60 -1.5 37.13 perf-stat.overall.node-store-miss-rate% 3.992e+09 ± 4% +43.4% 5.725e+09 perf-stat.ps.branch-instructions 49933802 ± 3% -53.8% 23085684 perf-stat.ps.branch-misses 53231168 ± 5% -54.1% 24422253 perf-stat.ps.cache-misses 2.883e+08 ± 4% -48.0% 1.498e+08 perf-stat.ps.cache-references 760697 ± 5% -7.6% 702869 perf-stat.ps.context-switches 6.804e+10 ± 4% +61.0% 1.095e+11 perf-stat.ps.cpu-cycles 87708 ± 5% -59.2% 35824 perf-stat.ps.cpu-migrations 54271302 ± 5% -61.8% 20716539 ± 5% perf-stat.ps.dTLB-load-misses 4.991e+09 ± 4% +22.9% 6.135e+09 perf-stat.ps.dTLB-loads 3799715 ± 4% -56.5% 1653235 ± 6% perf-stat.ps.dTLB-store-misses 1.987e+09 ± 4% -57.9% 8.367e+08 perf-stat.ps.dTLB-stores 16198464 ± 5% -63.1% 5980364 perf-stat.ps.iTLB-load-misses 2709431 ± 3% -76.8% 627483 ± 5% perf-stat.ps.iTLB-loads 1.804e+10 ± 4% +32.2% 2.384e+10 perf-stat.ps.instructions 2937 -9.8% 2650 perf-stat.ps.minor-faults 78775 +1.3% 79809 perf-stat.ps.msec 19295918 ± 7% -29.5% 13596222 perf-stat.ps.node-load-misses 25448471 ± 6% -41.1% 14979981 perf-stat.ps.node-loads 17740171 ± 5% -68.5% 5580578 perf-stat.ps.node-store-misses 28211759 ± 5% -66.5% 9448596 perf-stat.ps.node-stores 2937 -9.7% 2650 perf-stat.ps.page-faults 1.191e+12 ± 2% +754.0% 1.017e+13 perf-stat.total.instructions 26147 ± 74% +1249.4% 352831 ± 41% sched_debug.cfs_rq:/.MIN_vruntime.avg 419652 ± 70% +1206.8% 5484055 ± 18% sched_debug.cfs_rq:/.MIN_vruntime.max 100681 ± 71% +1199.0% 1307904 ± 29% sched_debug.cfs_rq:/.MIN_vruntime.stddev 700.02 ± 25% -48.6% 359.80 ± 2% sched_debug.cfs_rq:/.load_avg.avg 128.00 ± 10% -63.8% 46.38 ± 3% sched_debug.cfs_rq:/.load_avg.min 26147 ± 74% +1249.4% 352831 ± 41% sched_debug.cfs_rq:/.max_vruntime.avg 419652 ± 70% +1206.8% 5484055 ± 18% sched_debug.cfs_rq:/.max_vruntime.max 100682 ± 71% +1199.0% 1307905 ± 29% sched_debug.cfs_rq:/.max_vruntime.stddev 631056 +923.3% 6457752 sched_debug.cfs_rq:/.min_vruntime.avg 657496 +893.4% 6531324 sched_debug.cfs_rq:/.min_vruntime.max 622268 +927.2% 6392164 sched_debug.cfs_rq:/.min_vruntime.min 6424 ± 21% +491.2% 37984 ± 8% sched_debug.cfs_rq:/.min_vruntime.stddev 0.53 ± 10% +59.6% 0.84 ± 5% sched_debug.cfs_rq:/.nr_running.avg 0.48 ± 11% -14.3% 0.41 ± 9% sched_debug.cfs_rq:/.nr_running.stddev 33.75 ± 17% -79.2% 7.04 ± 49% sched_debug.cfs_rq:/.removed.load_avg.avg 512.00 -75.4% 126.00 sched_debug.cfs_rq:/.removed.load_avg.max 125.70 ± 8% -77.8% 27.85 ± 23% sched_debug.cfs_rq:/.removed.load_avg.stddev 1558 ± 17% -79.2% 324.85 ± 48% sched_debug.cfs_rq:/.removed.runnable_sum.avg 23640 -75.3% 5832 sched_debug.cfs_rq:/.removed.runnable_sum.max 5805 ± 8% -77.8% 1286 ± 23% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 10.35 ± 11% -79.6% 2.11 ± 44% sched_debug.cfs_rq:/.removed.util_avg.avg 210.50 ± 15% -75.7% 51.25 ± 24% sched_debug.cfs_rq:/.removed.util_avg.max 43.22 ± 11% -79.0% 9.08 ± 28% sched_debug.cfs_rq:/.removed.util_avg.stddev -7180 +1028.8% -81052 sched_debug.cfs_rq:/.spread0.min 6424 ± 21% +491.2% 37980 ± 8% sched_debug.cfs_rq:/.spread0.stddev 124.22 ± 17% +165.5% 329.82 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.avg 725.50 ± 17% +26.8% 919.75 sched_debug.cfs_rq:/.util_est_enqueued.max 163.23 ± 3% +55.2% 253.33 sched_debug.cfs_rq:/.util_est_enqueued.stddev 249918 ± 8% -67.7% 80842 ± 3% sched_debug.cpu.avg_idle.avg 613303 ± 6% -69.6% 186565 ± 3% sched_debug.cpu.avg_idle.max 36370 ± 28% -58.9% 14938 ± 29% sched_debug.cpu.avg_idle.min 137943 ± 3% -69.8% 41695 ± 7% sched_debug.cpu.avg_idle.stddev 69483 +259.8% 249970 sched_debug.cpu.clock.avg 69485 +259.8% 249975 sched_debug.cpu.clock.max 69481 +259.8% 249965 sched_debug.cpu.clock.min 1.29 ± 10% +120.8% 2.84 ± 15% sched_debug.cpu.clock.stddev 69483 +259.8% 249970 sched_debug.cpu.clock_task.avg 69485 +259.8% 249975 sched_debug.cpu.clock_task.max 69481 +259.8% 249965 sched_debug.cpu.clock_task.min 1.29 ± 10% +120.8% 2.84 ± 15% sched_debug.cpu.clock_task.stddev 6.17 ± 30% +52.5% 9.41 ± 7% sched_debug.cpu.cpu_load[2].min 178.83 ± 30% -54.3% 81.69 ± 14% sched_debug.cpu.cpu_load[3].max 8.67 ± 26% +54.7% 13.41 ± 3% sched_debug.cpu.cpu_load[3].min 32.77 ± 36% -64.6% 11.60 ± 15% sched_debug.cpu.cpu_load[3].stddev 224.67 ± 13% -59.8% 90.22 ± 9% sched_debug.cpu.cpu_load[4].max 9.17 ± 24% +75.6% 16.09 ± 3% sched_debug.cpu.cpu_load[4].min 38.70 ± 18% -68.1% 12.34 ± 11% sched_debug.cpu.cpu_load[4].stddev 932.68 ± 19% +81.7% 1694 ± 5% sched_debug.cpu.curr->pid.avg 2647 +173.0% 7228 sched_debug.cpu.curr->pid.max 793.79 ± 7% +51.0% 1198 ± 4% sched_debug.cpu.curr->pid.stddev 0.00 ± 15% +83.5% 0.00 ± 2% sched_debug.cpu.next_balance.stddev 40300 +445.7% 219939 sched_debug.cpu.nr_load_updates.avg 50191 ± 2% +353.4% 227592 sched_debug.cpu.nr_load_updates.max 37603 +475.9% 216550 sched_debug.cpu.nr_load_updates.min 0.88 ± 26% +220.9% 2.81 ± 11% sched_debug.cpu.nr_running.avg 4.17 ± 53% +80.0% 7.50 ± 2% sched_debug.cpu.nr_running.max 0.98 ± 42% +112.5% 2.09 ± 3% sched_debug.cpu.nr_running.stddev 629284 +474.1% 3612460 sched_debug.cpu.nr_switches.avg 638714 +472.3% 3655423 sched_debug.cpu.nr_switches.max 622843 +473.1% 3569313 sched_debug.cpu.nr_switches.min 3755 ± 10% +669.9% 28910 ± 4% sched_debug.cpu.nr_switches.stddev 1.93 ± 65% +96.8% 3.81 ± 8% sched_debug.cpu.nr_uninterruptible.avg 414.50 ± 9% +94.8% 807.34 ± 9% sched_debug.cpu.nr_uninterruptible.max -322.00 +70.9% -550.16 sched_debug.cpu.nr_uninterruptible.min 211.68 ± 13% +88.3% 398.65 ± 3% sched_debug.cpu.nr_uninterruptible.stddev 69481 +259.8% 249965 sched_debug.cpu_clk 65881 +274.0% 246371 sched_debug.ktime 69932 +258.1% 250410 sched_debug.sched_clk 43.10 -38.7 4.43 ± 2% perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write 22.52 -20.3 2.21 perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write 20.53 -18.3 2.21 ± 4% perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write 12.59 -12.6 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write 12.49 -12.5 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync 11.12 -10.4 0.71 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write 13.95 -10.2 3.73 perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write 10.86 -10.2 0.69 ± 6% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync 9.65 ± 2% -8.9 0.74 ± 5% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write 9.43 ± 2% -8.7 0.71 ± 5% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write 9.36 ± 2% -8.7 0.70 ± 5% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write 9.34 ± 2% -8.6 0.70 ± 4% perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync 11.71 -8.2 3.51 perf-profile.calltrace.cycles-pp.secondary_startup_64 11.00 -7.5 3.46 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 11.00 -7.5 3.46 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 10.99 -7.5 3.46 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 8.61 -6.5 2.13 perf-profile.calltrace.cycles-pp._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write 8.57 -6.5 2.12 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write 6.19 -6.2 0.00 perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write 6.14 -6.1 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync 6.11 -6.1 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn 8.87 -6.0 2.86 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 8.61 -5.8 2.79 ± 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 5.46 ± 2% -5.5 0.00 perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range 5.43 ± 2% -5.4 0.00 perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range 5.26 ± 2% -5.3 0.00 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages 5.08 ± 2% -4.5 0.55 perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write 4.82 ± 2% -4.4 0.38 ± 57% perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write 4.74 -3.6 1.11 ± 4% perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write 4.71 -3.6 1.10 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write 4.68 -3.6 1.09 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync 3.79 -2.8 0.99 ± 9% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write 3.76 -2.8 0.98 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync 3.72 -2.7 0.98 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn 5.50 -1.9 3.57 perf-profile.calltrace.cycles-pp.ret_from_fork 5.50 -1.9 3.57 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 5.49 -1.9 3.56 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 5.29 -1.9 3.39 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 2.73 -1.6 1.16 perf-profile.calltrace.cycles-pp.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write 2.25 ± 2% -1.3 0.99 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync 2.22 ± 2% -1.2 0.98 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn.xfs_log_force_lsn 2.20 ± 2% -1.2 0.97 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait.__xfs_log_force_lsn 1.81 ± 2% -1.0 0.78 perf-profile.calltrace.cycles-pp.xlog_cil_push.process_one_work.worker_thread.kthread.ret_from_fork 0.88 ± 3% -0.3 0.59 ± 2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push.process_one_work.worker_thread.kthread 0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.md_flush_request.raid0_make_request 0.00 +0.6 0.59 ± 4% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.00 +0.6 0.64 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.md_flush_request.raid0_make_request 0.00 +0.7 0.68 ± 4% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.md_flush_request.raid0_make_request 0.00 +0.7 0.72 ± 4% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.md_flush_request.raid0_make_request.md_handle_request 0.00 +0.8 0.79 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.md_flush_request.raid0_make_request.md_handle_request 0.00 +0.8 0.83 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.md_flush_request.raid0_make_request 0.00 +0.8 0.84 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.md_flush_request.raid0_make_request.md_handle_request 0.00 +0.8 0.85 ± 4% perf-profile.calltrace.cycles-pp.reschedule_interrupt.md_flush_request.raid0_make_request.md_handle_request.md_make_request 0.00 +0.9 0.85 perf-profile.calltrace.cycles-pp.finish_wait.md_flush_request.raid0_make_request.md_handle_request.md_make_request 0.00 +1.0 1.04 perf-profile.calltrace.cycles-pp.prepare_to_wait_event.md_flush_request.raid0_make_request.md_handle_request.md_make_request 0.00 +1.4 1.41 ± 2% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.md_flush_request.raid0_make_request.md_handle_request 0.00 +1.4 1.42 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.md_submit_flush_data 0.00 +1.4 1.43 ± 2% perf-profile.calltrace.cycles-pp.schedule.md_flush_request.raid0_make_request.md_handle_request.md_make_request 0.00 +1.5 1.53 ± 3% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.md_submit_flush_data.process_one_work 0.00 +1.6 1.57 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.md_submit_flush_data.process_one_work.worker_thread 0.00 +1.8 1.78 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.md_submit_flush_data.process_one_work.worker_thread.kthread 0.00 +1.8 1.78 ± 2% perf-profile.calltrace.cycles-pp.md_submit_flush_data.process_one_work.worker_thread.kthread.ret_from_fork 81.32 +11.3 92.58 perf-profile.calltrace.cycles-pp.write 80.05 +12.1 92.17 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 80.00 +12.2 92.16 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 79.89 +12.3 92.15 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 79.81 +12.3 92.14 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 79.30 +12.8 92.06 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 79.07 +12.9 92.02 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 71.03 +20.1 91.10 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write 0.00 +76.8 76.79 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.md_flush_request.raid0_make_request.md_handle_request 0.00 +77.0 77.02 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.md_flush_request.raid0_make_request.md_handle_request.md_make_request 4.03 ± 2% +78.1 82.15 perf-profile.calltrace.cycles-pp.blkdev_issue_flush.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write 3.72 ± 2% +78.4 82.10 perf-profile.calltrace.cycles-pp.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write 3.45 ± 2% +78.5 81.97 perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush 3.27 ± 3% +78.7 81.95 perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.submit_bio_wait 2.99 ± 3% +78.8 81.79 perf-profile.calltrace.cycles-pp.md_flush_request.raid0_make_request.md_handle_request.md_make_request.generic_make_request 3.07 ± 3% +78.9 81.93 perf-profile.calltrace.cycles-pp.raid0_make_request.md_handle_request.md_make_request.generic_make_request.submit_bio 2.76 ± 2% +79.2 82.00 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync 2.75 ± 2% +79.2 82.00 perf-profile.calltrace.cycles-pp.submit_bio.submit_bio_wait.blkdev_issue_flush.xfs_file_fsync.xfs_file_buffered_aio_write 14881 ± 6% +486.4% 87257 softirqs.CPU0.RCU 24878 +270.9% 92277 softirqs.CPU0.SCHED 32103 ± 6% +414.9% 165306 softirqs.CPU0.TIMER 15176 ± 2% +471.7% 86759 softirqs.CPU1.RCU 22055 ± 2% +306.8% 89728 softirqs.CPU1.SCHED 32444 ± 3% +411.3% 165897 ± 3% softirqs.CPU1.TIMER 14864 ± 4% +489.2% 87587 softirqs.CPU10.RCU 21273 ± 2% +315.1% 88301 softirqs.CPU10.SCHED 27880 ± 5% +528.9% 175353 ± 16% softirqs.CPU10.TIMER 14830 ± 11% +483.3% 86503 softirqs.CPU11.RCU 21874 +308.2% 89296 softirqs.CPU11.SCHED 30818 ± 4% +426.0% 162109 softirqs.CPU11.TIMER 17879 ± 13% +408.6% 90939 softirqs.CPU12.RCU 22410 ± 2% +295.3% 88582 softirqs.CPU12.SCHED 34255 ± 2% +385.3% 166243 softirqs.CPU12.TIMER 18805 ± 7% +359.6% 86430 softirqs.CPU13.RCU 21827 +309.3% 89328 softirqs.CPU13.SCHED 29925 ± 4% +441.7% 162101 softirqs.CPU13.TIMER 14412 ± 2% +510.1% 87931 ± 4% softirqs.CPU14.RCU 22144 ± 2% +300.4% 88659 softirqs.CPU14.SCHED 39515 ± 27% +315.1% 164042 ± 2% softirqs.CPU14.TIMER 14377 ± 6% +504.6% 86919 softirqs.CPU15.RCU 21897 ± 2% +308.0% 89347 softirqs.CPU15.SCHED 30163 ± 3% +438.9% 162561 softirqs.CPU15.TIMER 14409 ± 4% +507.2% 87490 softirqs.CPU16.RCU 22024 ± 3% +303.3% 88831 softirqs.CPU16.SCHED 31665 ± 4% +415.9% 163359 softirqs.CPU16.TIMER 14241 ± 4% +516.5% 87798 softirqs.CPU17.RCU 22039 +307.1% 89714 softirqs.CPU17.SCHED 30190 ± 2% +438.1% 162438 softirqs.CPU17.TIMER 14232 +508.6% 86615 softirqs.CPU18.RCU 22227 ± 2% +298.6% 88600 softirqs.CPU18.SCHED 31872 ± 5% +412.8% 163432 ± 2% softirqs.CPU18.TIMER 14326 ± 6% +511.9% 87667 softirqs.CPU19.RCU 21860 ± 2% +309.2% 89450 softirqs.CPU19.SCHED 30517 ± 3% +432.4% 162482 softirqs.CPU19.TIMER 15468 ± 3% +459.8% 86599 softirqs.CPU2.RCU 21947 +306.9% 89296 softirqs.CPU2.SCHED 29942 ± 7% +438.8% 161329 softirqs.CPU2.TIMER 15524 ± 10% +460.9% 87075 ± 2% softirqs.CPU20.RCU 22247 ± 3% +297.6% 88449 softirqs.CPU20.SCHED 32511 ± 9% +399.9% 162531 softirqs.CPU20.TIMER 16124 ± 3% +438.1% 86772 ± 2% softirqs.CPU21.RCU 21867 ± 2% +308.2% 89270 softirqs.CPU21.SCHED 32222 ± 7% +403.3% 162187 softirqs.CPU21.TIMER 15245 ± 6% +469.9% 86890 softirqs.CPU22.RCU 22141 ± 2% +301.4% 88866 softirqs.CPU22.SCHED 30630 ± 7% +438.4% 164905 ± 2% softirqs.CPU22.TIMER 16203 ± 16% +433.7% 86477 softirqs.CPU23.RCU 21894 +309.1% 89560 softirqs.CPU23.SCHED 32179 ± 8% +404.7% 162403 softirqs.CPU23.TIMER 14825 ± 2% +492.1% 87781 softirqs.CPU24.RCU 22102 +301.8% 88810 softirqs.CPU24.SCHED 30067 ± 3% +443.4% 163384 softirqs.CPU24.TIMER 14649 ± 6% +494.4% 87072 softirqs.CPU25.RCU 21811 +310.6% 89558 softirqs.CPU25.SCHED 32069 ± 8% +406.6% 162452 softirqs.CPU25.TIMER 15524 ± 12% +463.1% 87416 softirqs.CPU26.RCU 22060 +304.3% 89198 softirqs.CPU26.SCHED 30608 ± 2% +431.6% 162708 softirqs.CPU26.TIMER 14375 ± 4% +511.8% 87948 softirqs.CPU27.RCU 21802 +311.4% 89696 softirqs.CPU27.SCHED 31032 ± 6% +422.6% 162165 softirqs.CPU27.TIMER 15014 ± 6% +477.5% 86704 softirqs.CPU28.RCU 21935 +303.6% 88529 softirqs.CPU28.SCHED 29871 +441.4% 161730 softirqs.CPU28.TIMER 14457 ± 7% +508.1% 87911 softirqs.CPU29.RCU 21845 +310.4% 89653 softirqs.CPU29.SCHED 30666 ± 5% +430.1% 162564 softirqs.CPU29.TIMER 15720 ± 12% +449.2% 86335 softirqs.CPU3.RCU 21911 +307.6% 89305 softirqs.CPU3.SCHED 32375 ± 7% +400.3% 161979 softirqs.CPU3.TIMER 16382 ± 3% +436.7% 87921 softirqs.CPU30.RCU 21787 +308.4% 88981 softirqs.CPU30.SCHED 29038 ± 2% +455.4% 161277 softirqs.CPU30.TIMER 14836 ± 8% +488.9% 87372 softirqs.CPU31.RCU 21773 +312.8% 89874 softirqs.CPU31.SCHED 30245 ± 5% +437.4% 162532 softirqs.CPU31.TIMER 14911 ± 14% +469.4% 84903 ± 2% softirqs.CPU32.RCU 22122 ± 2% +303.3% 89218 softirqs.CPU32.SCHED 31883 ± 10% +409.2% 162341 softirqs.CPU32.TIMER 13353 ± 4% +534.8% 84767 softirqs.CPU33.RCU 21842 ± 2% +310.5% 89664 softirqs.CPU33.SCHED 30126 ± 3% +438.3% 162162 softirqs.CPU33.TIMER 12271 ± 2% +587.1% 84320 softirqs.CPU34.RCU 22168 +302.3% 89183 softirqs.CPU34.SCHED 30379 ± 4% +433.9% 162194 softirqs.CPU34.TIMER 13700 ± 10% +531.4% 86500 softirqs.CPU35.RCU 21773 ± 2% +311.6% 89617 softirqs.CPU35.SCHED 29374 ± 4% +457.8% 163844 ± 2% softirqs.CPU35.TIMER 14197 ± 7% +504.9% 85881 softirqs.CPU36.RCU 22217 ± 2% +300.1% 88890 softirqs.CPU36.SCHED 34490 ± 9% +369.5% 161930 softirqs.CPU36.TIMER 12721 ± 9% +567.6% 84926 softirqs.CPU37.RCU 21844 ± 2% +308.0% 89118 softirqs.CPU37.SCHED 37740 ± 30% +328.7% 161776 softirqs.CPU37.TIMER 12972 ± 8% +555.4% 85014 softirqs.CPU38.RCU 22224 ± 2% +298.8% 88630 softirqs.CPU38.SCHED 31492 ± 4% +464.2% 177686 ± 14% softirqs.CPU38.TIMER 13739 ± 8% +520.3% 85230 ± 2% softirqs.CPU39.RCU 22027 ± 2% +305.3% 89286 softirqs.CPU39.SCHED 30457 ± 3% +484.2% 177924 ± 14% softirqs.CPU39.TIMER 15752 ± 4% +458.5% 87970 softirqs.CPU4.RCU 22056 +304.4% 89195 softirqs.CPU4.SCHED 34247 ± 11% +375.5% 162860 softirqs.CPU4.TIMER 15513 ± 5% +458.2% 86595 softirqs.CPU5.RCU 21876 ± 2% +308.6% 89388 softirqs.CPU5.SCHED 32238 ± 8% +449.3% 177102 ± 14% softirqs.CPU5.TIMER 15190 ± 7% +470.0% 86580 softirqs.CPU6.RCU 22013 +304.2% 88985 softirqs.CPU6.SCHED 30196 +437.6% 162339 softirqs.CPU6.TIMER 14199 ± 4% +520.5% 88112 softirqs.CPU7.RCU 21843 ± 2% +310.9% 89752 softirqs.CPU7.SCHED 31088 ± 6% +424.4% 163017 softirqs.CPU7.TIMER 15641 ± 7% +458.1% 87287 softirqs.CPU8.RCU 21911 +304.4% 88603 softirqs.CPU8.SCHED 30063 ± 2% +441.0% 162632 softirqs.CPU8.TIMER 14905 ± 6% +483.9% 87032 softirqs.CPU9.RCU 21715 +309.3% 88870 softirqs.CPU9.SCHED 30645 ± 5% +430.4% 162556 softirqs.CPU9.TIMER 595866 ± 3% +483.2% 3475309 softirqs.RCU 881282 +305.0% 3569582 softirqs.SCHED 1259248 ± 4% +421.7% 6569855 softirqs.TIMER 382.00 ± 97% +151.4% 960.50 ± 14% interrupts.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0 126.00 ± 77% +255.4% 447.75 ± 35% interrupts.38:IR-PCI-MSI.524290-edge.eth0-TxRx-1 127.33 ± 63% +620.4% 917.25 ± 97% interrupts.39:IR-PCI-MSI.524291-edge.eth0-TxRx-2 73.00 ± 36% +1571.6% 1220 ±102% interrupts.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3 67.00 ± 51% +492.9% 397.25 ± 39% interrupts.41:IR-PCI-MSI.524293-edge.eth0-TxRx-4 50.67 ± 26% +693.9% 402.25 ± 49% interrupts.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5 94.00 ± 76% +425.5% 494.00 ± 64% interrupts.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6 39316 ± 19% +290.3% 153433 ± 3% interrupts.CAL:Function_call_interrupts 1048 ± 18% +254.6% 3717 ± 5% interrupts.CPU0.CAL:Function_call_interrupts 133588 ± 6% +506.6% 810374 ± 9% interrupts.CPU0.LOC:Local_timer_interrupts 6938 ± 2% -70.0% 2078 ± 93% interrupts.CPU0.NMI:Non-maskable_interrupts 6938 ± 2% -70.0% 2078 ± 93% interrupts.CPU0.PMI:Performance_monitoring_interrupts 270741 +676.9% 2103498 ± 2% interrupts.CPU0.RES:Rescheduling_interrupts 992.00 ± 18% +297.3% 3941 ± 10% interrupts.CPU1.CAL:Function_call_interrupts 133024 ± 5% +509.4% 810700 ± 8% interrupts.CPU1.LOC:Local_timer_interrupts 268182 +661.2% 2041439 ± 3% interrupts.CPU1.RES:Rescheduling_interrupts 919.33 ± 27% +324.9% 3906 ± 6% interrupts.CPU10.CAL:Function_call_interrupts 133250 ± 6% +507.4% 809341 ± 9% interrupts.CPU10.LOC:Local_timer_interrupts 271163 +666.7% 2078939 ± 2% interrupts.CPU10.RES:Rescheduling_interrupts 1017 ± 20% +316.2% 4232 ± 3% interrupts.CPU11.CAL:Function_call_interrupts 133396 ± 6% +507.1% 809829 ± 9% interrupts.CPU11.LOC:Local_timer_interrupts 2329 ± 70% +159.9% 6053 ± 33% interrupts.CPU11.NMI:Non-maskable_interrupts 2329 ± 70% +159.9% 6053 ± 33% interrupts.CPU11.PMI:Performance_monitoring_interrupts 269666 +648.2% 2017734 ± 2% interrupts.CPU11.RES:Rescheduling_interrupts 1008 ± 17% +302.7% 4059 ± 4% interrupts.CPU12.CAL:Function_call_interrupts 133147 ± 6% +507.8% 809269 ± 9% interrupts.CPU12.LOC:Local_timer_interrupts 269665 +667.6% 2069971 ± 2% interrupts.CPU12.RES:Rescheduling_interrupts 1035 ± 19% +307.4% 4216 ± 4% interrupts.CPU13.CAL:Function_call_interrupts 132326 ± 5% +511.6% 809355 ± 9% interrupts.CPU13.LOC:Local_timer_interrupts 268233 +651.0% 2014555 ± 2% interrupts.CPU13.RES:Rescheduling_interrupts 955.00 ± 22% +310.5% 3920 ± 9% interrupts.CPU14.CAL:Function_call_interrupts 132902 ± 6% +508.2% 808332 ± 9% interrupts.CPU14.LOC:Local_timer_interrupts 270232 +669.5% 2079387 ± 2% interrupts.CPU14.RES:Rescheduling_interrupts 958.00 ± 15% +331.2% 4130 ± 10% interrupts.CPU15.CAL:Function_call_interrupts 133491 ± 6% +506.1% 809122 ± 9% interrupts.CPU15.LOC:Local_timer_interrupts 268772 +653.5% 2025140 ± 3% interrupts.CPU15.RES:Rescheduling_interrupts 982.33 ± 20% +283.5% 3767 ± 9% interrupts.CPU16.CAL:Function_call_interrupts 133026 ± 6% +508.4% 809297 ± 9% interrupts.CPU16.LOC:Local_timer_interrupts 32.00 ±139% +6310.9% 2051 ± 97% interrupts.CPU16.NMI:Non-maskable_interrupts 32.00 ±139% +6310.9% 2051 ± 97% interrupts.CPU16.PMI:Performance_monitoring_interrupts 270141 +674.6% 2092564 ± 2% interrupts.CPU16.RES:Rescheduling_interrupts 932.33 ± 11% +285.7% 3596 ± 6% interrupts.CPU17.CAL:Function_call_interrupts 133459 ± 6% +506.5% 809457 ± 9% interrupts.CPU17.LOC:Local_timer_interrupts 268937 +664.9% 2057174 ± 2% interrupts.CPU17.RES:Rescheduling_interrupts 997.33 ± 22% +272.8% 3717 ± 11% interrupts.CPU18.CAL:Function_call_interrupts 133142 ± 6% +508.0% 809548 ± 9% interrupts.CPU18.LOC:Local_timer_interrupts 1161 ±141% +247.1% 4030 ± 70% interrupts.CPU18.NMI:Non-maskable_interrupts 1161 ±141% +247.1% 4030 ± 70% interrupts.CPU18.PMI:Performance_monitoring_interrupts 271465 +671.4% 2094183 ± 2% interrupts.CPU18.RES:Rescheduling_interrupts 984.67 ± 15% +290.8% 3848 ± 9% interrupts.CPU19.CAL:Function_call_interrupts 133426 ± 6% +506.5% 809213 ± 9% interrupts.CPU19.LOC:Local_timer_interrupts 269250 +658.3% 2041823 ± 3% interrupts.CPU19.RES:Rescheduling_interrupts 790.33 ± 10% +339.7% 3475 ± 5% interrupts.CPU2.CAL:Function_call_interrupts 133443 ± 6% +506.4% 809224 ± 9% interrupts.CPU2.LOC:Local_timer_interrupts 5796 ± 29% -99.5% 28.75 ±167% interrupts.CPU2.NMI:Non-maskable_interrupts 5796 ± 29% -99.5% 28.75 ±167% interrupts.CPU2.PMI:Performance_monitoring_interrupts 270575 +681.7% 2115196 ± 2% interrupts.CPU2.RES:Rescheduling_interrupts 972.00 ± 24% +281.5% 3708 ± 6% interrupts.CPU20.CAL:Function_call_interrupts 133230 ± 6% +507.9% 809922 ± 9% interrupts.CPU20.LOC:Local_timer_interrupts 269876 +675.8% 2093778 ± 2% interrupts.CPU20.RES:Rescheduling_interrupts 1005 ± 20% +292.3% 3944 ± 5% interrupts.CPU21.CAL:Function_call_interrupts 133184 ± 5% +508.8% 810772 ± 8% interrupts.CPU21.LOC:Local_timer_interrupts 268330 +658.8% 2036131 interrupts.CPU21.RES:Rescheduling_interrupts 967.33 ± 20% +304.5% 3913 ± 8% interrupts.CPU22.CAL:Function_call_interrupts 133003 ± 6% +508.8% 809723 ± 9% interrupts.CPU22.LOC:Local_timer_interrupts 9.00 ± 86% +78477.8% 7072 ± 24% interrupts.CPU22.NMI:Non-maskable_interrupts 9.00 ± 86% +78477.8% 7072 ± 24% interrupts.CPU22.PMI:Performance_monitoring_interrupts 269563 +668.4% 2071335 ± 2% interrupts.CPU22.RES:Rescheduling_interrupts 954.33 ± 18% +295.1% 3770 ± 6% interrupts.CPU23.CAL:Function_call_interrupts 133483 ± 6% +507.2% 810519 ± 9% interrupts.CPU23.LOC:Local_timer_interrupts 40.67 ±139% +9827.7% 4037 ± 70% interrupts.CPU23.NMI:Non-maskable_interrupts 40.67 ±139% +9827.7% 4037 ± 70% interrupts.CPU23.PMI:Performance_monitoring_interrupts 268426 +661.6% 2044292 ± 2% interrupts.CPU23.RES:Rescheduling_interrupts 972.33 ± 20% +297.8% 3867 ± 9% interrupts.CPU24.CAL:Function_call_interrupts 133172 ± 5% +508.1% 809821 ± 9% interrupts.CPU24.LOC:Local_timer_interrupts 270065 +670.5% 2080975 interrupts.CPU24.RES:Rescheduling_interrupts 951.00 ± 18% +298.8% 3792 ± 6% interrupts.CPU25.CAL:Function_call_interrupts 133180 ± 5% +508.4% 810270 ± 9% interrupts.CPU25.LOC:Local_timer_interrupts 268233 +663.0% 2046605 ± 2% interrupts.CPU25.RES:Rescheduling_interrupts 382.00 ± 97% +151.4% 960.50 ± 14% interrupts.CPU26.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0 994.00 ± 22% +252.5% 3503 ± 7% interrupts.CPU26.CAL:Function_call_interrupts 133368 ± 6% +507.1% 809711 ± 9% interrupts.CPU26.LOC:Local_timer_interrupts 269891 +679.0% 2102493 interrupts.CPU26.RES:Rescheduling_interrupts 898.33 ± 11% +333.8% 3897 ± 5% interrupts.CPU27.CAL:Function_call_interrupts 133388 ± 6% +507.4% 810264 ± 9% interrupts.CPU27.LOC:Local_timer_interrupts 268700 +657.8% 2036093 interrupts.CPU27.RES:Rescheduling_interrupts 126.00 ± 77% +255.4% 447.75 ± 35% interrupts.CPU28.38:IR-PCI-MSI.524290-edge.eth0-TxRx-1 944.33 ± 22% +302.1% 3797 ± 7% interrupts.CPU28.CAL:Function_call_interrupts 133140 ± 6% +507.8% 809290 ± 9% interrupts.CPU28.LOC:Local_timer_interrupts 270210 +677.1% 2099843 ± 3% interrupts.CPU28.RES:Rescheduling_interrupts 969.67 ± 20% +297.5% 3854 ± 8% interrupts.CPU29.CAL:Function_call_interrupts 133520 ± 6% +507.1% 810570 ± 9% interrupts.CPU29.LOC:Local_timer_interrupts 268826 +663.3% 2051921 ± 2% interrupts.CPU29.RES:Rescheduling_interrupts 1038 ± 19% +293.7% 4089 ± 8% interrupts.CPU3.CAL:Function_call_interrupts 133191 ± 5% +508.1% 809881 ± 8% interrupts.CPU3.LOC:Local_timer_interrupts 6940 ± 2% -56.4% 3024 ± 57% interrupts.CPU3.NMI:Non-maskable_interrupts 6940 ± 2% -56.4% 3024 ± 57% interrupts.CPU3.PMI:Performance_monitoring_interrupts 268572 +654.7% 2026828 ± 2% interrupts.CPU3.RES:Rescheduling_interrupts 127.33 ± 63% +620.4% 917.25 ± 97% interrupts.CPU30.39:IR-PCI-MSI.524291-edge.eth0-TxRx-2 963.00 ± 26% +281.9% 3678 ± 4% interrupts.CPU30.CAL:Function_call_interrupts 133054 ± 6% +508.3% 809434 ± 9% interrupts.CPU30.LOC:Local_timer_interrupts 271207 +674.4% 2100122 ± 2% interrupts.CPU30.RES:Rescheduling_interrupts 995.00 ± 18% +267.3% 3654 ± 4% interrupts.CPU31.CAL:Function_call_interrupts 133386 ± 6% +507.3% 810065 ± 9% interrupts.CPU31.LOC:Local_timer_interrupts 269250 +664.1% 2057223 ± 2% interrupts.CPU31.RES:Rescheduling_interrupts 73.00 ± 36% +1571.6% 1220 ±102% interrupts.CPU32.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3 1002 ± 22% +258.4% 3592 ± 4% interrupts.CPU32.CAL:Function_call_interrupts 132940 ± 6% +508.8% 809389 ± 9% interrupts.CPU32.LOC:Local_timer_interrupts 269559 +683.9% 2113083 ± 2% interrupts.CPU32.RES:Rescheduling_interrupts 975.33 ± 19% +275.5% 3662 ± 4% interrupts.CPU33.CAL:Function_call_interrupts 133459 ± 5% +506.4% 809294 ± 9% interrupts.CPU33.LOC:Local_timer_interrupts 267896 +667.4% 2055845 ± 2% interrupts.CPU33.RES:Rescheduling_interrupts 67.00 ± 51% +492.9% 397.25 ± 39% interrupts.CPU34.41:IR-PCI-MSI.524293-edge.eth0-TxRx-4 1017 ± 20% +245.0% 3508 ± 8% interrupts.CPU34.CAL:Function_call_interrupts 133243 ± 6% +507.5% 809390 ± 9% interrupts.CPU34.LOC:Local_timer_interrupts 269293 +680.6% 2102164 ± 3% interrupts.CPU34.RES:Rescheduling_interrupts 1031 ± 20% +271.9% 3834 ± 6% interrupts.CPU35.CAL:Function_call_interrupts 132493 ± 7% +510.7% 809145 ± 9% interrupts.CPU35.LOC:Local_timer_interrupts 268627 +662.3% 2047686 interrupts.CPU35.RES:Rescheduling_interrupts 50.67 ± 26% +693.9% 402.25 ± 49% interrupts.CPU36.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5 1035 ± 18% +275.0% 3880 ± 3% interrupts.CPU36.CAL:Function_call_interrupts 132759 ± 6% +509.0% 808458 ± 9% interrupts.CPU36.LOC:Local_timer_interrupts 269908 +675.6% 2093509 ± 3% interrupts.CPU36.RES:Rescheduling_interrupts 1008 ± 18% +296.3% 3997 ± 4% interrupts.CPU37.CAL:Function_call_interrupts 133226 ± 6% +507.3% 809019 ± 9% interrupts.CPU37.LOC:Local_timer_interrupts 2505 ±131% +141.5% 6051 ± 33% interrupts.CPU37.NMI:Non-maskable_interrupts 2505 ±131% +141.5% 6051 ± 33% interrupts.CPU37.PMI:Performance_monitoring_interrupts 268397 +655.3% 2027303 interrupts.CPU37.RES:Rescheduling_interrupts 94.00 ± 76% +425.5% 494.00 ± 64% interrupts.CPU38.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6 943.00 ± 21% +306.9% 3837 ± 5% interrupts.CPU38.CAL:Function_call_interrupts 132971 ± 6% +508.5% 809161 ± 9% interrupts.CPU38.LOC:Local_timer_interrupts 270860 +668.1% 2080363 ± 2% interrupts.CPU38.RES:Rescheduling_interrupts 1005 ± 22% +294.7% 3966 ± 5% interrupts.CPU39.CAL:Function_call_interrupts 132952 ± 6% +507.9% 808170 ± 9% interrupts.CPU39.LOC:Local_timer_interrupts 268740 +658.0% 2037012 interrupts.CPU39.RES:Rescheduling_interrupts 955.00 ± 30% +259.7% 3435 ± 7% interrupts.CPU4.CAL:Function_call_interrupts 133712 ± 6% +505.5% 809605 ± 9% interrupts.CPU4.LOC:Local_timer_interrupts 270806 +680.7% 2114232 ± 2% interrupts.CPU4.RES:Rescheduling_interrupts 1039 ± 19% +288.5% 4037 ± 7% interrupts.CPU5.CAL:Function_call_interrupts 133190 ± 5% +508.5% 810503 ± 9% interrupts.CPU5.LOC:Local_timer_interrupts 6988 ± 2% -55.5% 3107 ± 52% interrupts.CPU5.NMI:Non-maskable_interrupts 6988 ± 2% -55.5% 3107 ± 52% interrupts.CPU5.PMI:Performance_monitoring_interrupts 269305 +652.6% 2026721 ± 2% interrupts.CPU5.RES:Rescheduling_interrupts 1037 ± 18% +283.8% 3980 ± 6% interrupts.CPU6.CAL:Function_call_interrupts 133350 ± 6% +507.2% 809668 ± 9% interrupts.CPU6.LOC:Local_timer_interrupts 270185 +671.4% 2084081 ± 3% interrupts.CPU6.RES:Rescheduling_interrupts 1023 ± 16% +286.0% 3951 ± 10% interrupts.CPU7.CAL:Function_call_interrupts 133398 ± 6% +507.7% 810619 ± 9% interrupts.CPU7.LOC:Local_timer_interrupts 268918 +657.6% 2037320 ± 3% interrupts.CPU7.RES:Rescheduling_interrupts 959.00 ± 26% +285.6% 3697 ± 6% interrupts.CPU8.CAL:Function_call_interrupts 133200 ± 6% +508.3% 810215 ± 9% interrupts.CPU8.LOC:Local_timer_interrupts 270558 +676.3% 2100284 interrupts.CPU8.RES:Rescheduling_interrupts 1040 ± 19% +289.8% 4054 ± 6% interrupts.CPU9.CAL:Function_call_interrupts 133076 ± 5% +508.7% 810082 ± 9% interrupts.CPU9.LOC:Local_timer_interrupts 268991 +655.1% 2031167 ± 2% interrupts.CPU9.RES:Rescheduling_interrupts 5327898 ± 6% +507.9% 32386035 ± 9% interrupts.LOC:Local_timer_interrupts 10780226 +666.5% 82630024 interrupts.RES:Rescheduling_interrupts 223.67 ± 6% +595.5% 1555 ± 4% interrupts.TLB:TLB_shootdowns aim7.jobs-per-min 35000 +-+-----------------------------------------------------------------+ | | 30000 +-+.+.+..+.+.+ +.+.+ + +.+.+.+ +..+.+.+.+.+ +.+..+.+.+ | | : : : : : : : : : : : | 25000 +-+ : : : : : : : : : : | | : : : : : : : : : : : : | 20000 +-+ : : : : : : : : : : : :| |: : : : : : : : : : : : :| 15000 +-+ : : : : : : : : : : : :| |: O O: :O O O:O: O: :O O :O: : : : :| 10000 O-+ O : : : : : : : : : : : :| |: :: :: :: :: :: ::| 5000 +-+ : :: : : O O O O : : | | : : : : : : | 0 +-O---O----O---O------------O---O---O-------------------------------+ aim7.time.system_time 16000 +-+-----------------------------------------------------------------+ | O O O O | 14000 +-+ | 12000 +-+ | | | 10000 +-+ | | | 8000 +-+ | | | 6000 +-+ | 4000 O-+ O O O O O O | | O O O O O | 2000 +-+ | |.+.+.+..+.+.+. .+.+.+. .+. .+.+.+.+. .+..+.+.+.+.+. .+.+..+.+.+. .| 0 +-O---O----O---O------------O---O---O-------------------------------+ aim7.time.elapsed_time 450 +-+-------------------------------------------------------------------+ | O O O O | 400 +-+ | 350 +-+ | | | 300 +-+ | 250 +-+ | | | 200 +-+ | 150 O-+ O O O O O O O O O | | O O | 100 +-+ | 50 +-+.+..+.+.+.+ +.+.+ + +.+.+.+ +.+.+.+.+..+ +.+.+.+..+ | |+ + .. + + + .. + .. + + + +| 0 +-O----O---O---O------------O----O---O--------------------------------+ aim7.time.elapsed_time.max 450 +-+-------------------------------------------------------------------+ | O O O O | 400 +-+ | 350 +-+ | | | 300 +-+ | 250 +-+ | | | 200 +-+ | 150 O-+ O O O O O O O O O | | O O | 100 +-+ | 50 +-+.+..+.+.+.+ +.+.+ + +.+.+.+ +.+.+.+.+..+ +.+.+.+..+ | |+ + .. + + + .. + .. + + + +| 0 +-O----O---O---O------------O----O---O--------------------------------+ aim7.time.minor_page_faults 90000 +-+-----------------------------------------------------------------+ | O | 80000 +-+ O O | 70000 +-+ O | | | 60000 +-+ | 50000 +-+ | | | 40000 +-+ | 30000 O-+ O O O O O O O O O O | | O | 20000 +-+.+.+..+.+.+ +.+.+ + +.+.+.+ +..+.+.+.+.+ +.+..+.+.+ | 10000 +-+ : : : + : : : : : : : :| |: : : : + : : : : : : : :| 0 +-O---O----O---O------------O---O---O-------------------------------+ aim7.time.voluntary_context_switches 2.5e+08 +-+---------------------------------------------------------------+ | | | O O O O | 2e+08 +-+ | | | | | 1.5e+08 +-+ | | | 1e+08 +-+ | | | | | 5e+07 O-+ O O O O O O O O O O O | | +.+.+.+.+.+ +.+.+ + +.+.+.+ +.+.+.+.+..+ +.+.+.+.+ | |+ + .. + + + + + + + + + +| 0 +-O---O---O---O------------O---O---O------------------------------+ aim7.time.involuntary_context_switches 2.5e+07 +-+----------------------------------------O-O--------------------+ | O O | | | 2e+07 +-+ | | | | | 1.5e+07 +-+ | | | 1e+07 +-+ | | | | | 5e+06 +-+ | O O O O O O O O O O O O | | | 0 +-O---O---O---O------------O---O---O------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen