From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932347AbcHNOvC (ORCPT ); Sun, 14 Aug 2016 10:51:02 -0400 Received: from mga02.intel.com ([134.134.136.20]:54190 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932171AbcHNOvB (ORCPT ); Sun, 14 Aug 2016 10:51:01 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.28,520,1464678000"; d="scan'208";a="748584811" Date: Sun, 14 Aug 2016 22:50:53 +0800 From: Fengguang Wu To: Christoph Hellwig Cc: Dave Chinner , Ye Xiaolong , Linus Torvalds , LKML , Bob Peterson , LKP Subject: Re: [LKP] [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression Message-ID: <20160814145053.GA17428@wfg-t540p.sh.intel.com> References: <20160812060433.GS19025@dastard> <20160812062934.GA17589@yexl-desktop> <20160812085124.GB19354@yexl-desktop> <20160812100208.GA16044@dastard> <20160813003054.GA3101@lst.de> <20160813214825.GA31667@lst.de> <20160813220727.GA4901@wfg-t540p.sh.intel.com> <20160813221507.GA1368@lst.de> <20160813225128.GA6416@wfg-t540p.sh.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20160813225128.GA6416@wfg-t540p.sh.intel.com> User-Agent: Mutt/1.6.0 (2016-04-01) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Christoph, On Sun, Aug 14, 2016 at 06:51:28AM +0800, Fengguang Wu wrote: >Hi Christoph, > >On Sun, Aug 14, 2016 at 12:15:08AM +0200, Christoph Hellwig wrote: >>Hi Fengguang, >> >>feel free to try this git tree: >> >> git://git.infradead.org/users/hch/vfs.git iomap-fixes > >I just queued some test jobs for it. > >% queue -q vip -t ivb44 -b hch-vfs/iomap-fixes aim7-fs-1brd.yaml fs=xfs -r3 -k fe9c2c81ed073878768785a985295cbacc349e42 -k ca2edab2e1d8f30dda874b7f717c2d4664991e9b -k 99091700659f4df965e138b38b4fa26a29b7eade > >That job file can be found here: > > https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/jobs/aim7-fs-1brd.yaml > >It specifies a matrix of the below atom tests: > > wfg /c/lkp-tests% split-job jobs/aim7-fs-1brd.yaml -s 'fs: xfs' > > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_src-3000-performance.yaml > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_rr-3000-performance.yaml > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_rw-3000-performance.yaml > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_cp-3000-performance.yaml > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_wrt-3000-performance.yaml > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-sync_disk_rw-600-performance.yaml > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-creat-clo-1500-performance.yaml > jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_rd-9000-performance.yaml I got some results now. The several finished aim7 tests have some performance regressions for commit fe9c2c81 ("xfs: rewrite and optimize the delalloc write path") comparing to its parent commit ca2edab2e and their base mainline commit 990917006 ("Merge tag 'nfs-for-4.8-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs"). wfg@inn ~% compare -g aim7 -ai 99091700659f4df965e138b38b4fa26a29b7eade ca2edab2e1d8f30dda874b7f717c2d4664991e9b fe9c2c81ed073878768785a985295cbacc349e42 tests: 4 60 perf-index fe9c2c81ed073878768785a985295cbacc349e42 97 power-index fe9c2c81ed073878768785a985295cbacc349e42 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 testcase/testparams/testbox ---------------- -------------------------- -------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 270459 272267 ± 3% -48% 139834 ± 3% aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44 473257 468546 5% 497512 aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44 360578 -18% 296589 -60% 144974 aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44 358701 -6% 335712 -40% 216057 GEO-MEAN aim7.jobs-per-min 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 ---------------- -------------------------- -------------------------- 48.48 48.15 36% 65.85 aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44 89.50 89.76 88.75 aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44 35.78 23% 43.93 76% 63.09 aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44 53.75 7% 57.48 33% 71.71 GEO-MEAN turbostat.%Busy 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 ---------------- -------------------------- -------------------------- 1439 1431 36% 1964 aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44 2671 2674 2650 aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44 1057 23% 1303 78% 1883 aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44 1595 7% 1708 34% 2139 GEO-MEAN turbostat.Avg_MHz 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 ---------------- -------------------------- -------------------------- 167 167 6% 177 aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44 175 175 176 aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44 150 8% 162 19% 178 aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44 164 168 8% 177 GEO-MEAN turbostat.PkgWatt 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 ---------------- -------------------------- -------------------------- 10.27 10.43 -14% 8.79 aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44 6.85 6.66 6.88 aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44 9.96 14% 11.36 -7% 9.23 aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44 8.88 4% 9.24 -7% 8.23 GEO-MEAN turbostat.RAMWatt Here are the detailed numbers for each test case. The perf-profile and latency_stats numbers are sorted by absolute change in each sub-category now. perf-profile numbers > 5 are all shown. It may be more pleasant to view the long trace.call.funcs lines with vim ":set nowrap" option. aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 ---------------- -------------------------- -------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 360578 -18% 294351 -60% 144974 aim7.jobs-per-min 12835 458% 71658 480% 74445 aim7.time.involuntary_context_switches 755 50% 1136 373% 3570 aim7.time.system_time 155970 152810 73% 269438 aim7.time.minor_page_faults 50.15 22% 61.39 148% 124.39 aim7.time.elapsed_time 50.15 22% 61.39 148% 124.39 aim7.time.elapsed_time.max 438660 428601 -7% 407807 aim7.time.voluntary_context_switches 2452 2480 5% 2584 aim7.time.maximum_resident_set_size 34293 ± 4% 70% 58129 ± 19% 213% 107483 interrupts.CAL:Function_call_interrupts 79.70 ± 6% 16% 92.63 ± 6% 89% 150.33 uptime.boot 2890 ± 8% 6% 3077 ± 8% 15% 3329 uptime.idle 150186 ± 9% 41% 212090 122% 333727 softirqs.RCU 161166 9% 176318 16% 186527 softirqs.SCHED 648051 33% 864346 222% 2089349 softirqs.TIMER 50.15 22% 61.39 148% 124.39 time.elapsed_time 50.15 22% 61.39 148% 124.39 time.elapsed_time.max 12835 458% 71658 480% 74445 time.involuntary_context_switches 155970 152810 73% 269438 time.minor_page_faults 1563 21% 1898 85% 2895 time.percent_of_cpu_this_job_got 755 50% 1136 373% 3570 time.system_time 4564660 ± 4% 68% 7651587 79% 8159302 numa-numastat.node0.numa_foreign 3929898 81% 7129718 46% 5733813 numa-numastat.node0.numa_miss 0 2 ± 20% 2 numa-numastat.node1.other_node 4569811 ± 4% 68% 7654689 79% 8163206 numa-numastat.node1.numa_miss 3935075 81% 7132850 46% 5737410 numa-numastat.node1.numa_foreign 34767917 4% 36214694 11% 38627727 numa-numastat.node1.numa_hit 34767917 4% 36214691 11% 38627725 numa-numastat.node1.local_node 12377 ± 18% 3615% 459790 2848% 364868 vmstat.io.bo 119 -8% 110 ± 4% -16% 101 vmstat.memory.buff 18826454 -16% 15748045 -37% 11882562 vmstat.memory.free 16 25% 20 106% 33 vmstat.procs.r 19407 469% 110509 520% 120350 vmstat.system.cs 48215 10% 52977 3% 49819 vmstat.system.in 142459 -11% 126667 -23% 109481 cpuidle.C1-IVT.usage 29494441 ± 3% -18% 24206809 -36% 18889149 cpuidle.C1-IVT.time 5736732 28% 7315830 525% 35868316 cpuidle.C1E-IVT.time 51148 9% 55743 98% 101021 cpuidle.C1E-IVT.usage 18347890 27% 23243942 21% 22154105 cpuidle.C3-IVT.time 96127 9% 104487 -29% 68552 cpuidle.C3-IVT.usage 1.525e+09 6% 1.617e+09 41% 2.147e+09 cpuidle.C6-IVT.time 1805218 11% 1998052 33% 2397285 cpuidle.C6-IVT.usage 286 ± 11% 14% 328 ± 7% 389% 1402 cpuidle.POLL.usage 1013526 ± 41% 98% 2003264 ± 20% 272% 3774675 cpuidle.POLL.time 35.78 24% 44.22 76% 63.09 turbostat.%Busy 1057 24% 1312 78% 1883 turbostat.Avg_MHz 34.80 -3% 33.63 -22% 27.18 turbostat.CPU%c1 0.34 -5% 0.33 -77% 0.08 turbostat.CPU%c3 29.07 -25% 21.82 -67% 9.65 turbostat.CPU%c6 118 11% 130 23% 145 turbostat.CorWatt 9.39 ± 13% -19% 7.61 ± 6% -61% 3.67 turbostat.Pkg%pc2 3.04 ± 33% -49% 1.55 ± 14% -76% 0.72 turbostat.Pkg%pc6 150 9% 164 19% 178 turbostat.PkgWatt 9.96 14% 11.34 -7% 9.23 turbostat.RAMWatt 18232 ± 8% -8% 16747 ± 10% 11% 20267 meminfo.AnonHugePages 80723 78330 -24% 61572 meminfo.CmaFree 4690642 ± 10% -15% 3981312 -15% 3983392 meminfo.DirectMap2M 1060897 -21% 834807 -22% 828755 meminfo.Dirty 2362330 26% 2983603 44% 3391287 meminfo.Inactive 2353250 26% 2974520 44% 3382139 meminfo.Inactive(file) 19388991 -18% 15966408 -38% 12038822 meminfo.MemFree 1186231 4% 1236627 13% 1341728 meminfo.SReclaimable 179570 3% 185696 14% 204382 meminfo.SUnreclaim 1365802 4% 1422323 13% 1546111 meminfo.Slab 318863 10% 352026 16% 368386 meminfo.Unevictable 0.00 0.00 9.15 perf-profile.cycles-pp.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 0.00 0.00 8.90 perf-profile.cycles-pp.xfs_inode_set_eofblocks_tag.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write 0.00 0.00 8.61 perf-profile.cycles-pp._raw_spin_lock.xfs_inode_set_eofblocks_tag.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin.iomap_apply 0.00 0.00 8.50 perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xfs_inode_set_eofblocks_tag.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin 6.05 -11% 5.42 ± 4% -15% 5.14 perf-profile.cycles-pp.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter 6.54 -11% 5.80 ± 4% -16% 5.51 perf-profile.cycles-pp.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle 16.78 -9% 15.34 ± 9% -11% 14.90 perf-profile.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary 16.51 ± 3% -9% 14.99 ± 9% -12% 14.49 perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry 0.23 ± 23% 20% 0.28 ± 12% 3683% 8.70 perf-profile.func.cycles-pp.native_queued_spin_lock_slowpath 4.369e+11 ± 4% 20% 5.239e+11 97% 8.601e+11 perf-stat.branch-instructions 0.38 5% 0.40 -27% 0.28 perf-stat.branch-miss-rate 1.678e+09 ± 3% 26% 2.117e+09 44% 2.413e+09 perf-stat.branch-misses 42.30 -7% 39.31 -5% 40.38 perf-stat.cache-miss-rate 6.874e+09 ± 4% 19% 8.21e+09 51% 1.041e+10 perf-stat.cache-misses 1.625e+10 ± 3% 29% 2.089e+10 59% 2.578e+10 perf-stat.cache-references 1017846 588% 7005227 1401% 15273586 perf-stat.context-switches 2.757e+12 ± 4% 48% 4.092e+12 318% 1.151e+13 perf-stat.cpu-cycles 177918 15% 204776 35% 241051 perf-stat.cpu-migrations 0.37 ± 14% 60% 0.60 ± 3% 45% 0.54 perf-stat.dTLB-load-miss-rate 2.413e+09 ± 14% 97% 4.757e+09 ± 4% 149% 6.001e+09 perf-stat.dTLB-load-misses 6.438e+11 23% 7.893e+11 71% 1.103e+12 perf-stat.dTLB-loads 0.06 ± 38% 100% 0.11 ± 6% 207% 0.17 perf-stat.dTLB-store-miss-rate 2.656e+08 ± 34% 123% 5.91e+08 ± 7% 203% 8.038e+08 perf-stat.dTLB-store-misses 45.99 ± 5% 8% 49.56 ± 11% 14% 52.61 perf-stat.iTLB-load-miss-rate 45151945 45832755 72% 77697494 perf-stat.iTLB-load-misses 53205262 ± 7% -10% 47792612 ± 21% 32% 69997751 perf-stat.iTLB-loads 2.457e+12 ± 4% 16% 2.851e+12 66% 4.084e+12 perf-stat.instructions 0.89 -22% 0.70 -60% 0.35 perf-stat.ipc 286640 8% 310690 99% 571225 perf-stat.minor-faults 29.16 7% 31.25 8% 31.42 perf-stat.node-load-miss-rate 4.86e+08 ± 3% 123% 1.084e+09 250% 1.7e+09 perf-stat.node-load-misses 1.18e+09 102% 2.385e+09 214% 3.711e+09 perf-stat.node-loads 21.51 30% 27.95 62% 34.86 perf-stat.node-store-miss-rate 1.262e+09 58% 1.989e+09 177% 3.499e+09 perf-stat.node-store-misses 4.606e+09 11% 5.126e+09 42% 6.539e+09 perf-stat.node-stores 286617 8% 310730 99% 571253 perf-stat.page-faults 1166432 23% 1429828 42% 1653754 numa-meminfo.node0.Inactive(file) 1175123 22% 1434274 41% 1662351 numa-meminfo.node0.Inactive 513534 -23% 394773 -24% 392567 numa-meminfo.node0.Dirty 9717968 -17% 8082393 -37% 6159862 numa-meminfo.node0.MemFree 159470 11% 176717 16% 184229 numa-meminfo.node0.Unevictable 23148226 7% 24783802 15% 26706333 numa-meminfo.node0.MemUsed 103531 ± 32% -10% 93669 ± 40% 40% 144469 numa-meminfo.node0.SUnreclaim 1187035 30% 1549075 46% 1727751 numa-meminfo.node1.Inactive 1186646 30% 1544438 46% 1727201 numa-meminfo.node1.Inactive(file) 21000905 3% 21647702 13% 23741428 numa-meminfo.node1.Active(file) 21083707 3% 21748741 13% 23822391 numa-meminfo.node1.Active 547021 -20% 438525 -21% 433706 numa-meminfo.node1.Dirty 9663240 -19% 7870896 -39% 5869977 numa-meminfo.node1.MemFree 561241 12% 625903 21% 679671 numa-meminfo.node1.SReclaimable 637259 ± 4% 13% 717863 ± 5% 16% 739482 numa-meminfo.node1.Slab 23329350 8% 25121687 16% 27122606 numa-meminfo.node1.MemUsed 159394 10% 175315 16% 184159 numa-meminfo.node1.Unevictable 521615 33% 695562 267% 1916159 latency_stats.avg.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath 500644 33% 667614 261% 1805608 latency_stats.avg.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath 8932 ± 46% -70% 2717 ± 4% -95% 464 latency_stats.avg.wait_on_page_bit.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.dentry_unlink_inode.__dentry_kill.dput.__fput.____fput.task_work_run.exit_to_usermode_loop 0 0 73327 latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 43 ± 37% 7923% 3503 ± 4% 31792% 13926 latency_stats.hits.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent 1422573 30% 1852368 ± 5% 228% 4672496 latency_stats.max.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath 1423130 30% 1851873 ± 5% 228% 4661765 latency_stats.max.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath 627 ± 66% 3788% 24404 ± 17% 6254% 39883 latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent 3922 ± 18% 56% 6134 ± 29% 634% 28786 latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create 0 0 16665 latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 5.15e+10 25% 6.454e+10 220% 1.649e+11 latency_stats.sum.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath 0 0 1.385e+08 latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 11666476 45% 16905624 755% 99756088 latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode 2216 ± 69% 80030% 1775681 ± 4% 3e+06% 67521154 latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent 1601815 28% 2053992 288% 6213577 latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create 1774397 20% 2120576 244% 6099374 latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_droplink.xfs_remove.xfs_vn_unlink.vfs_unlink 628 ±141% 125% 1416 ± 5% 4e+05% 2677036 latency_stats.sum.xfs_iget.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath 6087 ± 92% 1277% 83839 ± 3% 11105% 682063 latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write.SyS_write 0 0 116108 latency_stats.sum.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write 1212 ± 59% 1842% 23546 ± 7% 4861% 60149 latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_vn_update_time.touch_atime.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.__vfs_read.vfs_read.SyS_read 1624 ± 22% 1356% 23637 ± 3% 1596% 27545 latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_itruncate_extents.xfs_free_eofblocks.xfs_release.xfs_file_release.__fput.____fput.task_work_run 2068 ± 27% 834% 19319 ± 23% 1125% 25334 latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_itruncate_extents.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput 0 0 22155 latency_stats.sum.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput.dentry_unlink_inode 39 ± 71% 41280% 16414 ± 14% 51951% 20647 latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_defer_trans_roll.xfs_defer_finish.xfs_itruncate_extents.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode 0 0 15600 latency_stats.sum.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput.dentry_unlink_inode 10 ±141% 6795% 689 ± 70% 1e+05% 10637 latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput.dentry_unlink_inode.__dentry_kill 99 ±112% 86% 185 ± 80% 9978% 10011 latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_imap_to_bp.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode 18232 ±134% -16% 15260 ± 54% -40% 10975 latency_stats.sum.xfs_lock_two_inodes.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath 647 ± 3% -97% 21 ± 19% 34% 868 proc-vmstat.kswapd_high_wmark_hit_quickly 1091 -97% 36 ± 9% 29% 1411 proc-vmstat.kswapd_low_wmark_hit_quickly 265066 -21% 208142 -22% 206344 proc-vmstat.nr_dirty 20118 19574 -23% 15432 proc-vmstat.nr_free_cma 4844108 -18% 3988031 -38% 3008251 proc-vmstat.nr_free_pages 588262 26% 743537 44% 845765 proc-vmstat.nr_inactive_file 50 ± 25% 192% 148 ± 15% 103% 103 proc-vmstat.nr_pages_scanned 296623 4% 309201 13% 335474 proc-vmstat.nr_slab_reclaimable 44880 3% 46405 14% 51078 proc-vmstat.nr_slab_unreclaimable 79716 10% 88008 16% 92097 proc-vmstat.nr_unevictable 167 ± 9% 9e+06% 14513434 2e+06% 3569348 proc-vmstat.nr_vmscan_immediate_reclaim 162380 ± 18% 4392% 7294622 7024% 11567602 proc-vmstat.nr_written 588257 26% 743537 44% 845784 proc-vmstat.nr_zone_inactive_file 79716 10% 88008 16% 92097 proc-vmstat.nr_zone_unevictable 265092 -21% 208154 -22% 206388 proc-vmstat.nr_zone_write_pending 8507451 ± 3% 74% 14784261 64% 13918067 proc-vmstat.numa_foreign 10 ± 4% 10 ± 4% 6e+05% 57855 proc-vmstat.numa_hint_faults 8507451 ± 3% 74% 14784187 64% 13918067 proc-vmstat.numa_miss 72 72 3e+05% 213175 proc-vmstat.numa_pte_updates 1740 -97% 59 ± 12% 33% 2306 proc-vmstat.pageoutrun 5322372 1068% 62167111 1024% 59824114 proc-vmstat.pgactivate 2816355 27% 3575784 14% 3203214 proc-vmstat.pgalloc_dma32 74392338 11% 82333943 14% 84954110 proc-vmstat.pgalloc_normal 60958397 -18% 49976330 -26% 45055885 proc-vmstat.pgdeactivate 302790 9% 329088 94% 586116 proc-vmstat.pgfault 61061205 14% 69758545 18% 72000453 proc-vmstat.pgfree 655652 ± 18% 4352% 29190304 6967% 46338056 proc-vmstat.pgpgout 60965725 -18% 49983704 -26% 45063375 proc-vmstat.pgrefill 2 ± 17% 4e+07% 985929 ± 8% 7e+07% 1952629 proc-vmstat.pgrotated 82046 ± 36% 50634% 41625211 5397% 4510385 proc-vmstat.pgscan_direct 60128369 -37% 38068394 10% 66306637 proc-vmstat.pgscan_kswapd 2030 ± 46% 1e+06% 27038054 ± 3% 78642% 1598733 proc-vmstat.pgsteal_direct 0 2414551 ± 3% 3694833 proc-vmstat.workingset_activate 0 2414551 ± 3% 3694833 proc-vmstat.workingset_refault 26 ± 39% 1e+07% 2657286 3e+06% 658792 numa-vmstat.node0.nr_vmscan_immediate_reclaim 40449 ± 22% 3135% 1308601 ± 4% 4723% 1950670 numa-vmstat.node0.nr_written 291648 22% 357059 42% 413612 numa-vmstat.node0.nr_zone_inactive_file 291655 22% 357053 42% 413596 numa-vmstat.node0.nr_inactive_file 1542314 ± 5% 77% 2731911 98% 3056411 numa-vmstat.node0.numa_foreign 1366073 ± 4% 103% 2766780 ± 3% 68% 2293117 numa-vmstat.node0.numa_miss 128634 -23% 99104 -24% 98062 numa-vmstat.node0.nr_dirty 128663 -23% 99130 -24% 98051 numa-vmstat.node0.nr_zone_write_pending 2424918 -16% 2033425 -37% 1537826 numa-vmstat.node0.nr_free_pages 14037168 10% 15473174 20% 16883787 numa-vmstat.node0.numa_local 14037172 10% 15473174 20% 16883790 numa-vmstat.node0.numa_hit 39867 10% 44022 16% 46058 numa-vmstat.node0.nr_zone_unevictable 39867 10% 44022 16% 46058 numa-vmstat.node0.nr_unevictable 25871 ± 32% -9% 23414 ± 40% 40% 36094 numa-vmstat.node0.nr_slab_unreclaimable 14851187 6% 15749527 11% 16497187 numa-vmstat.node0.nr_dirtied 0 1225299 ± 4% 2008478 numa-vmstat.node1.workingset_refault 0 1225299 ± 4% 2008478 numa-vmstat.node1.workingset_activate 23 ± 35% 1e+07% 2974198 ± 3% 3e+06% 683002 numa-vmstat.node1.nr_vmscan_immediate_reclaim 40769 ± 26% 3264% 1371611 ± 3% 5569% 2311374 numa-vmstat.node1.nr_written 25 ± 8% 216% 81 ± 3% 356% 117 numa-vmstat.node1.nr_pages_scanned 296681 30% 385708 45% 431591 numa-vmstat.node1.nr_zone_inactive_file 296681 30% 385709 45% 431591 numa-vmstat.node1.nr_inactive_file 5252547 5401234 13% 5936151 numa-vmstat.node1.nr_zone_active_file 5252547 5401238 13% 5936151 numa-vmstat.node1.nr_active_file 136060 -19% 110021 -21% 107114 numa-vmstat.node1.nr_zone_write_pending 136060 -19% 110019 -21% 107107 numa-vmstat.node1.nr_dirty 1520682 ± 3% 76% 2681012 98% 3008493 numa-vmstat.node1.numa_miss 2413468 -18% 1980184 -39% 1466738 numa-vmstat.node1.nr_free_pages 1344474 ± 3% 102% 2715690 ± 4% 67% 2245159 numa-vmstat.node1.numa_foreign 20160 19698 -22% 15673 numa-vmstat.node1.nr_free_cma 14350439 12% 16005551 27% 18257157 numa-vmstat.node1.numa_local 14350440 12% 16005552 27% 18257158 numa-vmstat.node1.numa_hit 15381788 9% 16829619 21% 18645441 numa-vmstat.node1.nr_dirtied 140354 11% 156202 21% 169950 numa-vmstat.node1.nr_slab_reclaimable 39848 10% 43676 16% 46041 numa-vmstat.node1.nr_zone_unevictable 39848 10% 43676 16% 46041 numa-vmstat.node1.nr_unevictable 377 ± 9% 370 ± 5% 24% 468 slabinfo.bdev_cache.active_objs 377 ± 9% 370 ± 5% 24% 468 slabinfo.bdev_cache.num_objs 389 ± 13% 604% 2737 ± 23% 3371% 13501 slabinfo.bio-1.active_objs 389 ± 13% 612% 2770 ± 24% 3441% 13774 slabinfo.bio-1.num_objs 7 ± 17% 1039% 83 ± 24% 3623% 273 slabinfo.bio-1.active_slabs 7 ± 17% 1039% 83 ± 24% 3623% 273 slabinfo.bio-1.num_slabs 978 ± 4% 10% 1075 17% 1144 slabinfo.blkdev_requests.active_objs 978 ± 4% 10% 1075 17% 1144 slabinfo.blkdev_requests.num_objs 10942119 3% 11286505 13% 12389701 slabinfo.buffer_head.num_objs 280566 3% 289397 13% 317684 slabinfo.buffer_head.active_slabs 280566 3% 289397 13% 317684 slabinfo.buffer_head.num_slabs 10941627 10693692 11% 12140372 slabinfo.buffer_head.active_objs 7436 ± 3% 7558 20% 8922 slabinfo.cred_jar.active_objs 7436 ± 3% 7558 20% 8922 slabinfo.cred_jar.num_objs 4734 85% 8767 ± 8% 60% 7554 slabinfo.kmalloc-128.num_objs 4734 78% 8418 ± 8% 45% 6848 slabinfo.kmalloc-128.active_objs 17074 -11% 15121 -10% 15379 slabinfo.kmalloc-256.num_objs 3105 4% 3216 14% 3527 slabinfo.kmalloc-4096.num_objs 3061 4% 3170 12% 3419 slabinfo.kmalloc-4096.active_objs 13131 ± 3% 17% 15379 12% 14714 slabinfo.kmalloc-512.num_objs 1623 ± 3% 1664 ± 3% 16% 1889 slabinfo.mnt_cache.active_objs 1623 ± 3% 1664 ± 3% 16% 1889 slabinfo.mnt_cache.num_objs 2670 6% 2821 19% 3178 slabinfo.nsproxy.active_objs 2670 6% 2821 19% 3178 slabinfo.nsproxy.num_objs 2532 5% 2656 17% 2959 slabinfo.posix_timers_cache.active_objs 2532 5% 2656 17% 2959 slabinfo.posix_timers_cache.num_objs 20689 87% 38595 ± 13% 47% 30452 slabinfo.radix_tree_node.active_objs 399 83% 730 ± 13% 47% 587 slabinfo.radix_tree_node.active_slabs 399 83% 730 ± 13% 47% 587 slabinfo.radix_tree_node.num_slabs 22379 83% 40931 ± 13% 47% 32872 slabinfo.radix_tree_node.num_objs 4688 4706 22% 5712 slabinfo.sigqueue.active_objs 4688 4706 22% 5712 slabinfo.sigqueue.num_objs 979 ± 4% 7% 1046 ± 3% -15% 833 slabinfo.task_group.active_objs 979 ± 4% 7% 1046 ± 3% -15% 833 slabinfo.task_group.num_objs 1344 5% 1410 17% 1570 slabinfo.xfs_btree_cur.active_objs 1344 5% 1410 17% 1570 slabinfo.xfs_btree_cur.num_objs 2500 5% 2632 18% 2946 slabinfo.xfs_da_state.active_objs 2500 5% 2632 18% 2946 slabinfo.xfs_da_state.num_objs 1299 279% 4917 ± 17% 134% 3035 slabinfo.xfs_efd_item.num_objs 1299 278% 4911 ± 17% 126% 2940 slabinfo.xfs_efd_item.active_objs 1904 ± 3% 4% 1982 42% 2703 slabinfo.xfs_inode.num_objs 1904 ± 3% 4% 1982 39% 2644 slabinfo.xfs_inode.active_objs 1659 113% 3538 ± 27% 1360% 24227 slabinfo.xfs_log_ticket.active_objs 1659 116% 3588 ± 27% 1369% 24383 slabinfo.xfs_log_ticket.num_objs 37 169% 99 ± 29% 1405% 557 slabinfo.xfs_log_ticket.active_slabs 37 169% 99 ± 29% 1405% 557 slabinfo.xfs_log_ticket.num_slabs 2615 84% 4821 ± 28% 1549% 43132 slabinfo.xfs_trans.active_objs 2615 86% 4860 ± 28% 1551% 43171 slabinfo.xfs_trans.num_objs 37 162% 97 ± 30% 1614% 634 slabinfo.xfs_trans.active_slabs 37 162% 97 ± 30% 1614% 634 slabinfo.xfs_trans.num_slabs 3255 ± 12% 9210% 303094 38966% 1271810 sched_debug.cfs_rq:/.min_vruntime.avg 8273 ± 10% 382% 39836 ± 17% 309% 33806 sched_debug.cfs_rq:/.load.avg 716 ± 34% 28783% 206899 1e+05% 1034000 sched_debug.cfs_rq:/.min_vruntime.min 1830 ± 5% 4365% 81731 10579% 195502 sched_debug.cfs_rq:/.min_vruntime.stddev 1845 ± 4% 4330% 81754 10503% 195683 sched_debug.cfs_rq:/.spread0.stddev 73578 ± 34% 1043% 841209 ± 34% 452% 405848 sched_debug.cfs_rq:/.load.max 12.67 ± 35% 3999% 519.25 1979% 263.33 sched_debug.cfs_rq:/.runnable_load_avg.max 2.34 ± 33% 4268% 102.01 1854% 45.63 sched_debug.cfs_rq:/.runnable_load_avg.stddev 10284 ± 12% 4107% 432665 ± 7% 15350% 1588973 sched_debug.cfs_rq:/.min_vruntime.max 1.05 ± 20% 2335% 25.54 1631% 18.15 sched_debug.cfs_rq:/.runnable_load_avg.avg 44.06 ± 28% 254% 155.90 ± 16% 310% 180.49 sched_debug.cfs_rq:/.util_avg.stddev 15448 ± 19% 831% 143829 ± 22% 422% 80585 sched_debug.cfs_rq:/.load.stddev 597 ± 13% -39% 367 ± 17% -49% 303 sched_debug.cfs_rq:/.util_avg.min 1464 ± 23% -55% 664 ± 30% -63% 546 sched_debug.cfs_rq:/.load_avg.min 1830 ± 3% -50% 911 ± 5% -65% 642 sched_debug.cfs_rq:/.load_avg.avg 0.30 ± 13% 22% 0.36 ± 11% 86% 0.56 sched_debug.cfs_rq:/.nr_running.avg 2302 ± 11% -31% 1589 -50% 1157 sched_debug.cfs_rq:/.load_avg.max 819 ± 3% 36% 1116 15% 940 sched_debug.cfs_rq:/.util_avg.max 728 -14% 630 -9% 664 sched_debug.cfs_rq:/.util_avg.avg 73578 ± 34% 1043% 841209 ± 34% 452% 405848 sched_debug.cpu.load.max 1.81 ± 11% 77% 3.22 395% 8.98 sched_debug.cpu.clock.stddev 1.81 ± 11% 77% 3.22 395% 8.98 sched_debug.cpu.clock_task.stddev 8278 ± 10% 379% 39671 ± 18% 305% 33517 sched_debug.cpu.load.avg 3600 385% 17452 1023% 40419 sched_debug.cpu.nr_load_updates.min 5446 305% 22069 754% 46492 sched_debug.cpu.nr_load_updates.avg 8627 ± 5% 217% 27314 517% 53222 sched_debug.cpu.nr_load_updates.max 6221 ± 3% 2137% 139191 3486% 223092 sched_debug.cpu.nr_switches.max 15.67 ± 40% 3187% 515.00 1579% 263.00 sched_debug.cpu.cpu_load[0].max 2.55 ± 33% 3886% 101.45 1697% 45.73 sched_debug.cpu.cpu_load[0].stddev 15452 ± 19% 831% 143937 ± 22% 421% 80431 sched_debug.cpu.load.stddev 1144 236% 3839 329% 4911 sched_debug.cpu.nr_load_updates.stddev 23.67 ± 41% 709% 191.50 ± 6% 637% 174.33 sched_debug.cpu.nr_uninterruptible.max 978 7241% 71831 ± 3% 13746% 135493 sched_debug.cpu.nr_switches.avg 0.96 ± 19% 2503% 24.95 1720% 17.44 sched_debug.cpu.cpu_load[0].avg 957 ± 4% 3406% 33568 3626% 35679 sched_debug.cpu.nr_switches.stddev 29644 ± 16% 107% 61350 ± 8% 190% 86111 sched_debug.cpu.clock.max 29644 ± 16% 107% 61350 ± 8% 190% 86111 sched_debug.cpu.clock_task.max 29640 ± 16% 107% 61344 ± 8% 190% 86096 sched_debug.cpu.clock.avg 29640 ± 16% 107% 61344 ± 8% 190% 86096 sched_debug.cpu.clock_task.avg 29635 ± 16% 107% 61338 ± 8% 190% 86079 sched_debug.cpu.clock.min 29635 ± 16% 107% 61338 ± 8% 190% 86079 sched_debug.cpu.clock_task.min 335 ± 4% 7948% 27014 22596% 76183 sched_debug.cpu.nr_switches.min 1.62 ± 32% 1784% 30.61 ± 3% 1100% 19.51 sched_debug.cpu.cpu_load[4].avg 5.46 ± 15% 2325% 132.40 1031% 61.73 sched_debug.cpu.nr_uninterruptible.stddev 424 ± 11% 106% 875 ± 13% 263% 1541 sched_debug.cpu.curr->pid.avg 1400 166% 3721 264% 5100 sched_debug.cpu.curr->pid.max 610 ± 3% 108% 1269 126% 1380 sched_debug.cpu.curr->pid.stddev 0.43 ± 15% 4% 0.45 ± 16% 48% 0.64 sched_debug.cpu.nr_running.avg 253789 ± 13% -5% 241499 ± 3% -22% 198383 sched_debug.cpu.avg_idle.stddev 29638 ± 16% 107% 61339 ± 8% 190% 86079 sched_debug.cpu_clk 28529 ± 17% 111% 60238 ± 8% 198% 84957 sched_debug.ktime 0.17 -74% 0.04 ± 8% -83% 0.03 sched_debug.rt_rq:/.rt_time.avg 0.85 ± 3% -74% 0.22 ± 8% -83% 0.14 sched_debug.rt_rq:/.rt_time.stddev 5.14 ± 10% -75% 1.28 ± 6% -83% 0.88 sched_debug.rt_rq:/.rt_time.max 29638 ± 16% 107% 61339 ± 8% 190% 86079 sched_debug.sched_clk aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 ---------------- -------------------------- -------------------------- 473257 468546 5% 497512 aim7.jobs-per-min 613996 11% 681283 -7% 571701 aim7.time.involuntary_context_switches 4914 4977 -6% 4634 aim7.time.system_time 114.83 115.98 -5% 109.23 aim7.time.elapsed_time 114.83 115.98 -5% 109.23 aim7.time.elapsed_time.max 60711 ± 8% 20% 73007 -9% 55449 aim7.time.voluntary_context_switches 2509 -6% 2360 -4% 2416 aim7.time.maximum_resident_set_size 362268 19% 430263 -8% 332046 softirqs.RCU 352 ± 7% -32% 238 -35% 230 vmstat.procs.r 5 ± 16% 80% 9 -40% 3 vmstat.procs.b 9584 7% 10255 -10% 8585 vmstat.system.cs 20442 ± 5% 38% 28201 -40% 12270 cpuidle.C1-IVT.usage 3.95 -3% 3.81 9% 4.29 turbostat.CPU%c1 0.81 ± 14% 44% 1.17 28% 1.04 turbostat.Pkg%pc6 19711 ± 5% -7% 18413 -17% 16384 meminfo.AnonHugePages 3974485 3977216 27% 5046310 meminfo.DirectMap2M 139742 ± 4% 137012 -17% 116493 meminfo.DirectMap4k 244933 ± 4% -7% 228626 15% 280670 meminfo.PageTables 12.47 ± 39% 84% 22.89 64% 20.46 perf-profile.func.cycles-pp.poll_idle 57.44 ± 6% -10% 51.55 -13% 50.13 perf-profile.func.cycles-pp.intel_idle 0.20 3% 0.20 -5% 0.19 perf-stat.branch-miss-rate 5.356e+08 4% 5.552e+08 -6% 5.046e+08 perf-stat.branch-misses 1113549 7% 1187535 -15% 951607 perf-stat.context-switches 1.48e+13 1.491e+13 -6% 1.397e+13 perf-stat.cpu-cycles 101697 ± 3% 9% 111167 -3% 98319 perf-stat.cpu-migrations 0.69 ± 20% -17% 0.57 139% 1.65 perf-stat.dTLB-load-miss-rate 3.264e+09 ± 19% -17% 2.712e+09 148% 8.084e+09 perf-stat.dTLB-load-misses 4.695e+11 4.718e+11 4.818e+11 perf-stat.dTLB-loads 3.276e+11 ± 3% 3.303e+11 8% 3.528e+11 perf-stat.dTLB-stores 56.47 ± 19% 41% 79.48 -58% 23.96 perf-stat.iTLB-load-miss-rate 48864487 ± 4% 7% 52183944 -12% 43166037 perf-stat.iTLB-load-misses 40455495 ± 41% -67% 13468883 239% 1.37e+08 perf-stat.iTLB-loads 29278 ± 4% -6% 27480 12% 32844 perf-stat.instructions-per-iTLB-miss 0.10 0.10 5% 0.10 perf-stat.ipc 47.16 46.36 46.51 perf-stat.node-store-miss-rate 6568 ± 44% -59% 2721 -71% 1916 numa-meminfo.node0.Shmem 194395 7% 207086 15% 224164 numa-meminfo.node0.Active 10218 ± 24% -37% 6471 -36% 6494 numa-meminfo.node0.Mapped 7496 ± 34% -97% 204 37% 10278 numa-meminfo.node0.AnonHugePages 178888 6% 188799 16% 208213 numa-meminfo.node0.AnonPages 179468 6% 191062 17% 209704 numa-meminfo.node0.Active(anon) 256890 -15% 219489 -15% 219503 numa-meminfo.node1.Active 12213 ± 24% 49% 18208 -50% 6105 numa-meminfo.node1.AnonHugePages 45080 ± 23% -33% 30138 87% 84468 numa-meminfo.node1.PageTables 241623 -15% 204604 -16% 203913 numa-meminfo.node1.Active(anon) 240637 -15% 204491 -15% 203847 numa-meminfo.node1.AnonPages 23782392 ±139% 673% 1.838e+08 -100% 0 latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 61157 ± 4% -6% 57187 14% 69751 proc-vmstat.nr_page_table_pages 1641 ± 44% -59% 679 -71% 478 numa-vmstat.node0.nr_shmem 2655 ± 23% -35% 1715 -35% 1726 numa-vmstat.node0.nr_mapped 44867 5% 47231 16% 52261 numa-vmstat.node0.nr_anon_pages 45014 6% 47793 17% 52636 numa-vmstat.node0.nr_zone_active_anon 45014 6% 47793 17% 52636 numa-vmstat.node0.nr_active_anon 11300 ± 23% -33% 7542 88% 21209 numa-vmstat.node1.nr_page_table_pages 60581 -16% 51156 -15% 51193 numa-vmstat.node1.nr_zone_active_anon 60581 -16% 51156 -15% 51193 numa-vmstat.node1.nr_active_anon 60328 -15% 51127 -15% 51174 numa-vmstat.node1.nr_anon_pages 13671 13608 11% 15190 slabinfo.cred_jar.active_objs 13707 13608 11% 15231 slabinfo.cred_jar.num_objs 24109 24386 -11% 21574 slabinfo.kmalloc-16.active_objs 24109 24386 -11% 21574 slabinfo.kmalloc-16.num_objs 13709 ± 6% 13391 -15% 11600 slabinfo.kmalloc-512.active_objs 13808 ± 6% 13454 -16% 11657 slabinfo.kmalloc-512.num_objs 1456658 4% 1511260 15% 1675984 sched_debug.cfs_rq:/.min_vruntime.min 441613 ± 3% -28% 316751 -76% 105734 sched_debug.cfs_rq:/.min_vruntime.stddev 443999 ± 3% -28% 318033 -76% 106909 sched_debug.cfs_rq:/.spread0.stddev 2657974 2625551 -19% 2158111 sched_debug.cfs_rq:/.min_vruntime.max 0.22 ± 23% 96% 0.43 109% 0.46 sched_debug.cfs_rq:/.nr_spread_over.stddev 1.50 100% 3.00 133% 3.50 sched_debug.cfs_rq:/.nr_spread_over.max 111.95 ± 26% 15% 128.92 128% 254.81 sched_debug.cfs_rq:/.exec_clock.stddev 802 3% 829 -16% 671 sched_debug.cfs_rq:/.load_avg.min 874 879 -11% 780 sched_debug.cfs_rq:/.load_avg.avg 1256 ± 17% -20% 1011 -24% 957 sched_debug.cfs_rq:/.load_avg.max 1.33 ± 35% -100% 0.00 200% 4.00 sched_debug.cpu.cpu_load[4].min 4.56 ± 6% -11% 4.07 -27% 3.33 sched_debug.cpu.cpu_load[4].stddev 4.76 ± 3% -13% 4.14 -30% 3.35 sched_debug.cpu.cpu_load[3].stddev 25.17 ± 12% -26% 18.50 -21% 20.00 sched_debug.cpu.cpu_load[3].max 25.67 ± 9% -32% 17.50 -24% 19.50 sched_debug.cpu.cpu_load[0].max 4.67 ± 3% -17% 3.90 -22% 3.62 sched_debug.cpu.cpu_load[0].stddev 4.88 -15% 4.14 -31% 3.39 sched_debug.cpu.cpu_load[2].stddev 26.17 ± 10% -29% 18.50 -25% 19.50 sched_debug.cpu.cpu_load[2].max 7265 4% 7556 -12% 6419 sched_debug.cpu.nr_switches.avg 9.41 ± 10% 9.67 21% 11.38 sched_debug.cpu.cpu_load[1].avg 9.03 ± 12% 3% 9.32 23% 11.09 sched_debug.cpu.cpu_load[0].avg 4140 ± 4% -11% 3698 -11% 3703 sched_debug.cpu.nr_switches.stddev 9.41 ± 10% 3% 9.71 22% 11.49 sched_debug.cpu.cpu_load[3].avg 4690 4821 -9% 4257 sched_debug.cpu.nr_switches.min 9.39 ± 9% 3% 9.69 23% 11.52 sched_debug.cpu.cpu_load[4].avg 9.43 ± 10% 9.71 21% 11.44 sched_debug.cpu.cpu_load[2].avg 57.92 ± 18% -4% 55.55 -23% 44.50 sched_debug.cpu.nr_uninterruptible.stddev 3002 ± 3% 10% 3288 31% 3919 sched_debug.cpu.curr->pid.avg 6666 6652 -10% 6025 sched_debug.cpu.curr->pid.max 1379 1361 -19% 1118 sched_debug.cpu.ttwu_local.avg 1849 ± 3% -12% 1628 -18% 1517 sched_debug.cpu.ttwu_local.stddev 1679 ± 8% 4% 1738 -15% 1423 sched_debug.cpu.curr->pid.stddev 1.58 ± 33% -11% 1.41 65% 2.60 sched_debug.cpu.nr_running.avg 1767 6% 1880 -16% 1489 sched_debug.cpu.ttwu_count.avg 506 ± 6% -15% 430 -17% 419 sched_debug.cpu.ttwu_count.min 7139 8% 7745 -11% 6355 sched_debug.cpu.sched_count.avg 4355 6% 4625 -11% 3884 sched_debug.cpu.sched_count.min 4.91 ± 3% -16% 4.13 -28% 3.52 sched_debug.cpu.cpu_load[1].stddev 26.67 ± 9% -29% 19.00 -27% 19.50 sched_debug.cpu.cpu_load[1].max 209 ± 8% 19% 247 -15% 178 sched_debug.cpu.sched_goidle.avg 5.67 ± 27% -12% 5.00 50% 8.50 sched_debug.cpu.nr_running.max 36072 ± 7% 70% 61152 17% 42236 sched_debug.cpu.sched_count.max 2008 -8% 1847 -18% 1645 sched_debug.cpu.ttwu_count.stddev 0.07 ± 19% -20% 0.06 186% 0.21 sched_debug.rt_rq:/.rt_time.avg 0.36 ± 17% -23% 0.28 142% 0.88 sched_debug.rt_rq:/.rt_time.stddev 2.33 ± 15% -27% 1.70 87% 4.35 sched_debug.rt_rq:/.rt_time.max aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44 99091700659f4df9 ca2edab2e1d8f30dda874b7f71 fe9c2c81ed073878768785a985 ---------------- -------------------------- -------------------------- 270459 272267 ± 3% -48% 139834 ± 3% aim7.jobs-per-min 21229 ± 5% 20896 ± 3% 449% 116516 ± 6% aim7.time.involuntary_context_switches 1461 ± 5% 1454 ± 5% 174% 3998 ± 3% aim7.time.system_time 155368 153149 149% 386164 aim7.time.minor_page_faults 66.84 66.41 ± 3% 93% 129.07 ± 3% aim7.time.elapsed_time 66.84 66.41 ± 3% 93% 129.07 ± 3% aim7.time.elapsed_time.max 328369 3% 339077 96% 644393 aim7.time.voluntary_context_switches 49489 ± 11% -8% 45459 39% 68941 ± 4% interrupts.CAL:Function_call_interrupts 96.62 ± 7% 97.09 61% 155.12 uptime.boot 186640 ± 10% 186707 127% 424522 ± 4% softirqs.RCU 146596 147043 37% 201373 softirqs.SCHED 1005660 ± 3% 991053 ± 4% 118% 2196513 softirqs.TIMER 66.84 66.41 ± 3% 93% 129.07 ± 3% time.elapsed_time 66.84 66.41 ± 3% 93% 129.07 ± 3% time.elapsed_time.max 21229 ± 5% 20896 ± 3% 449% 116516 ± 6% time.involuntary_context_switches 155368 153149 149% 386164 time.minor_page_faults 2212 2215 41% 3112 time.percent_of_cpu_this_job_got 1461 ± 5% 1454 ± 5% 174% 3998 ± 3% time.system_time 328369 3% 339077 96% 644393 time.voluntary_context_switches 1197810 ± 16% -67% 393936 ± 40% -56% 530668 ± 43% numa-numastat.node0.numa_miss 1196269 ± 16% -68% 387751 ± 40% -55% 533013 ± 42% numa-numastat.node1.numa_foreign 112 159% 292 ± 4% 146% 277 vmstat.memory.buff 16422228 16461619 -28% 11832310 vmstat.memory.free 22 -3% 22 87% 42 ± 3% vmstat.procs.r 48853 48768 50273 vmstat.system.in 125202 8% 135626 51% 189515 ± 4% cpuidle.C1-IVT.usage 28088338 ± 3% 11% 31082173 17% 32997314 ± 5% cpuidle.C1-IVT.time 3471814 27% 4422338 ± 15% 2877% 1.034e+08 ± 3% cpuidle.C1E-IVT.time 33353 8% 36128 703% 267725 cpuidle.C1E-IVT.usage 11371800 9% 12381174 244% 39113028 cpuidle.C3-IVT.time 64048 5% 67490 62% 103940 cpuidle.C3-IVT.usage 1.637e+09 1.631e+09 20% 1.959e+09 cpuidle.C6-IVT.time 1861259 4% 1931551 19% 2223599 cpuidle.C6-IVT.usage 230 ± 9% 42% 326 1631% 3986 cpuidle.POLL.usage 1724995 ± 41% 54% 2656939 ± 10% 112% 3662791 cpuidle.POLL.time 48.48 48.15 36% 65.85 turbostat.%Busy 1439 1431 36% 1964 turbostat.Avg_MHz 33.28 33.45 -25% 24.85 turbostat.CPU%c1 18.09 ± 3% 18.24 ± 4% -49% 9.16 turbostat.CPU%c6 134 133 8% 144 turbostat.CorWatt 5.39 ± 17% 4% 5.63 ± 8% -34% 3.54 turbostat.Pkg%pc2 2.97 ± 44% -17% 2.48 ± 32% -70% 0.91 ± 22% turbostat.Pkg%pc6 167 167 6% 177 turbostat.PkgWatt 10.27 10.43 -14% 8.79 turbostat.RAMWatt 44376005 -100% 205734 -100% 214640 meminfo.Active 44199835 -100% 30412 -100% 30241 meminfo.Active(file) 103029 ± 3% 27% 130507 ± 6% 29% 133114 ± 8% meminfo.CmaFree 124701 ± 4% 123685 ± 14% 16% 144180 ± 3% meminfo.DirectMap4k 7886 ± 4% 7993 ± 5% 144% 19231 ± 7% meminfo.Dirty 2472446 1791% 46747572 1976% 51320420 meminfo.Inactive 2463353 1797% 46738477 1983% 51311261 meminfo.Inactive(file) 16631615 16664565 -28% 11936074 meminfo.MemFree 4.125e+11 -5% 3.927e+11 103% 8.36e+11 perf-stat.branch-instructions 0.41 -20% 0.33 -43% 0.23 perf-stat.branch-miss-rate 1.671e+09 -23% 1.28e+09 16% 1.946e+09 perf-stat.branch-misses 7.138e+09 -3% 6.917e+09 23% 8.746e+09 perf-stat.cache-misses 2.036e+10 -4% 1.956e+10 22% 2.476e+10 perf-stat.cache-references 821470 4% 851532 88% 1548125 ± 3% perf-stat.context-switches 4.93e+12 ± 3% -4% 4.755e+12 ± 4% 154% 1.25e+13 perf-stat.cpu-cycles 125073 4% 129993 167% 333599 perf-stat.cpu-migrations 3.595e+09 ± 16% -19% 2.895e+09 ± 17% 39% 4.987e+09 ± 10% perf-stat.dTLB-load-misses 6.411e+11 6.339e+11 ± 3% 57% 1.004e+12 perf-stat.dTLB-loads 0.06 ± 3% -42% 0.04 87% 0.12 ± 3% perf-stat.dTLB-store-miss-rate 2.738e+08 -39% 1.675e+08 64% 4.502e+08 ± 5% perf-stat.dTLB-store-misses 4.321e+11 5% 4.552e+11 -12% 3.81e+11 ± 8% perf-stat.dTLB-stores 2.343e+12 -5% 2.229e+12 67% 3.918e+12 perf-stat.instructions 46162 ± 41% 46733 ± 3% 55% 71500 perf-stat.instructions-per-iTLB-miss 0.48 ± 4% 0.47 ± 5% -34% 0.31 perf-stat.ipc 325877 322934 115% 699924 perf-stat.minor-faults 42.88 3% 44.33 43.65 perf-stat.node-load-miss-rate 9.499e+08 9.578e+08 66% 1.581e+09 perf-stat.node-load-misses 1.266e+09 -5% 1.203e+09 61% 2.04e+09 perf-stat.node-loads 39.17 40.00 8% 42.12 perf-stat.node-store-miss-rate 3.198e+09 4% 3.318e+09 36% 4.344e+09 perf-stat.node-store-misses 4.966e+09 4.977e+09 20% 5.968e+09 perf-stat.node-stores 325852 322963 115% 699918 perf-stat.page-faults 21719324 -100% 15215 ± 3% -100% 14631 numa-meminfo.node0.Active(file) 1221037 1806% 23278263 1969% 25269114 numa-meminfo.node0.Inactive(file) 1223564 1803% 23286857 1965% 25269597 numa-meminfo.node0.Inactive 21811771 -100% 102448 -100% 104424 numa-meminfo.node0.Active 2971 ± 13% -8% 2734 ± 3% 157% 7626 ± 4% numa-meminfo.node0.Dirty 8476780 8356206 -27% 6162743 numa-meminfo.node0.MemFree 617361 611434 11% 687829 numa-meminfo.node0.SReclaimable 1249068 1779% 23471025 1985% 26046948 numa-meminfo.node1.Inactive 1242501 1789% 23470523 1996% 26038272 numa-meminfo.node1.Inactive(file) 22500867 -100% 15202 ± 4% -100% 15613 numa-meminfo.node1.Active(file) 22584509 -100% 103192 ± 6% -100% 109976 numa-meminfo.node1.Active 4814 ± 13% 4957 ± 5% 135% 11335 numa-meminfo.node1.Dirty 8132889 8297084 ± 3% -29% 5777419 ± 3% numa-meminfo.node1.MemFree 83641 ± 7% 5% 87990 ± 7% 13% 94363 numa-meminfo.node1.Active(anon) 82877 ± 7% 4% 86528 ± 6% 13% 93620 numa-meminfo.node1.AnonPages 0 0 842360 ±100% latency_stats.avg.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.do_syscall_64.return_from_SYSCALL_64 113 ±173% 232% 376 ±100% 2e+05% 203269 ± 4% latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 5245 ± 14% 5325 ± 3% 535% 33286 ± 23% latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create 1133 ±173% 113% 2416 ±100% 1351% 16434 latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 0 0 842360 ±100% latency_stats.max.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.do_syscall_64.return_from_SYSCALL_64 7813 ± 13% -33% 5197 ± 9% 403% 39305 ± 18% latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode 5271 ± 13% -3% 5091 ± 5% 288% 20467 latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_droplink.xfs_remove.xfs_vn_unlink.vfs_unlink 10369 ± 17% -41% 6086 ± 21% -96% 385 ±100% latency_stats.max.wait_on_page_bit.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.dentry_unlink_inode.__dentry_kill.dput.__fput.____fput.task_work_run.exit_to_usermode_loop 94417 ±173% 556% 619712 ±100% 3e+05% 3.061e+08 ± 5% latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 22126648 ± 4% 22776886 1311% 3.123e+08 ± 7% latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode 2536 ±117% -98% 48 ± 43% 2059% 54765 ±100% latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent 1702264 ± 3% 5% 1790192 509% 10359205 ± 6% latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create 1180839 ± 3% 5% 1238547 453% 6527115 ± 5% latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_droplink.xfs_remove.xfs_vn_unlink.vfs_unlink 467 ±173% 680% 3644 ± 4% 7e+05% 3196407 ± 3% latency_stats.sum.xfs_iget.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath 0 0 842360 ±100% latency_stats.sum.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.do_syscall_64.return_from_SYSCALL_64 159018 ± 43% -49% 81514 ± 19% -99% 999 ±100% latency_stats.sum.wait_on_page_bit.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.dentry_unlink_inode.__dentry_kill.dput.__fput.____fput.task_work_run.exit_to_usermode_loop 1084 ± 5% 21% 1313 ± 3% 86% 2018 proc-vmstat.kswapd_high_wmark_hit_quickly 1817 ± 3% 38% 2511 ± 3% 175% 4989 proc-vmstat.kswapd_low_wmark_hit_quickly 11055004 -100% 7603 -100% 7559 proc-vmstat.nr_active_file 1993 2013 ± 4% 128% 4553 ± 5% proc-vmstat.nr_dirty 25746 ± 3% 26% 32494 ± 6% 29% 33319 ± 8% proc-vmstat.nr_free_cma 4152484 4162399 -28% 2984494 proc-vmstat.nr_free_pages 615907 1798% 11688190 1983% 12827366 proc-vmstat.nr_inactive_file 11055042 -100% 7603 -100% 7559 proc-vmstat.nr_zone_active_file 615904 1798% 11688234 1983% 12827434 proc-vmstat.nr_zone_inactive_file 2016 ± 3% 2025 ± 4% 127% 4582 ± 4% proc-vmstat.nr_zone_write_pending 2912 ± 3% 32% 3834 ± 3% 141% 7009 proc-vmstat.pageoutrun 5380414 -100% 2502 ± 3% -100% 2602 ± 3% proc-vmstat.pgactivate 61925072 -100% 0 -100% 0 proc-vmstat.pgdeactivate 348105 343315 108% 723517 proc-vmstat.pgfault 61932469 -100% 0 -100% 0 proc-vmstat.pgrefill 5432311 -100% 3802 ± 3% -100% 3657 numa-vmstat.node0.nr_zone_active_file 5432276 -100% 3802 ± 3% -100% 3657 numa-vmstat.node0.nr_active_file 305236 1802% 5806215 1969% 6314975 numa-vmstat.node0.nr_zone_inactive_file 305239 1802% 5806170 1969% 6314910 numa-vmstat.node0.nr_inactive_file 748 ± 7% -20% 597 ± 10% 114% 1602 numa-vmstat.node0.nr_dirty 775 ± 7% -21% 610 ± 12% 112% 1642 numa-vmstat.node0.nr_zone_write_pending 2116796 2102494 ± 3% -27% 1543100 numa-vmstat.node0.nr_free_pages 154392 152538 11% 171898 numa-vmstat.node0.nr_slab_reclaimable 310642 1784% 5853811 1995% 6507801 numa-vmstat.node1.nr_zone_inactive_file 310642 1784% 5853814 1995% 6507801 numa-vmstat.node1.nr_inactive_file 5627293 -100% 3799 ± 4% -100% 3903 numa-vmstat.node1.nr_zone_active_file 5627293 -100% 3799 ± 4% -100% 3903 numa-vmstat.node1.nr_active_file 1206 ± 16% 14% 1373 129% 2758 ± 10% numa-vmstat.node1.nr_zone_write_pending 1205 ± 16% 14% 1373 129% 2757 ± 10% numa-vmstat.node1.nr_dirty 2031121 2088592 ± 3% -29% 1446172 ± 3% numa-vmstat.node1.nr_free_pages 25743 ± 3% 27% 32608 ± 7% 30% 33415 ± 8% numa-vmstat.node1.nr_free_cma 20877 ± 7% 6% 22077 ± 6% 13% 23620 numa-vmstat.node1.nr_zone_active_anon 20877 ± 7% 6% 22077 ± 6% 13% 23620 numa-vmstat.node1.nr_active_anon 20684 ± 7% 5% 21709 ± 6% 13% 23431 numa-vmstat.node1.nr_anon_pages 4687 4704 11% 5205 ± 3% slabinfo.kmalloc-128.num_objs 4687 4704 11% 5205 ± 3% slabinfo.kmalloc-128.active_objs 1401 -19% 1142 8% 1516 ± 6% slabinfo.xfs_efd_item.num_objs 1401 -19% 1142 8% 1516 ± 6% slabinfo.xfs_efd_item.active_objs 1725 ± 5% -8% 1589 -12% 1518 slabinfo.xfs_inode.num_objs 1725 ± 5% -8% 1589 -12% 1518 slabinfo.xfs_inode.active_objs 382810 ± 4% 383813 ± 3% 301% 1535378 sched_debug.cfs_rq:/.min_vruntime.avg 249011 ± 6% 245840 ± 3% 420% 1294704 sched_debug.cfs_rq:/.min_vruntime.min 105216 106278 79% 188096 sched_debug.cfs_rq:/.min_vruntime.stddev 105260 106358 79% 188314 sched_debug.cfs_rq:/.spread0.stddev 9414 ± 4% 9361 ± 4% 230% 31092 sched_debug.cfs_rq:/.exec_clock.min 541056 ± 9% 540188 ± 3% 236% 1820030 sched_debug.cfs_rq:/.min_vruntime.max 150.87 ± 11% -21% 119.80 ± 10% 34% 202.73 ± 7% sched_debug.cfs_rq:/.util_avg.stddev 13783 13656 170% 37192 sched_debug.cfs_rq:/.exec_clock.avg 17625 17508 141% 42564 sched_debug.cfs_rq:/.exec_clock.max 3410.74 ± 3% 3458.30 38% 4706.14 sched_debug.cfs_rq:/.exec_clock.stddev 732 ± 11% 11% 809 ± 3% -34% 480 ± 7% sched_debug.cfs_rq:/.load_avg.min 844 ± 8% 7% 901 -33% 569 ± 4% sched_debug.cfs_rq:/.load_avg.avg 0.41 ± 7% 11% 0.46 ± 11% 21% 0.50 ± 5% sched_debug.cfs_rq:/.nr_running.avg 1339 ± 5% 1338 -32% 909 sched_debug.cfs_rq:/.load_avg.max 0.53 ± 4% -4% 0.51 32% 0.70 sched_debug.cfs_rq:/.nr_spread_over.avg 0.50 0.50 33% 0.67 sched_debug.cfs_rq:/.nr_spread_over.min 355.00 ± 26% -67% 118.75 ± 4% -82% 64.83 ± 20% sched_debug.cpu.cpu_load[4].max 18042 17697 135% 42380 sched_debug.cpu.nr_load_updates.min 51.83 ± 22% -66% 17.44 -78% 11.18 ± 5% sched_debug.cpu.cpu_load[4].stddev 22708 22546 111% 47986 sched_debug.cpu.nr_load_updates.avg 29633 ± 7% -7% 27554 83% 54243 sched_debug.cpu.nr_load_updates.max 48.83 ± 29% -65% 16.91 ± 29% -73% 13.34 ± 13% sched_debug.cpu.cpu_load[3].stddev 329.25 ± 34% -65% 113.75 ± 30% -76% 79.67 ± 28% sched_debug.cpu.cpu_load[3].max 17106 14% 19541 ± 19% 34% 22978 ± 6% sched_debug.cpu.nr_switches.max 1168 ± 4% -3% 1131 ± 4% 144% 2846 ± 21% sched_debug.cpu.ttwu_local.max 3826 ± 3% 3766 17% 4487 sched_debug.cpu.nr_load_updates.stddev 19.73 ± 12% -4% 18.86 ± 14% 59% 31.42 ± 8% sched_debug.cpu.nr_uninterruptible.avg 149.75 ± 8% 150.00 ± 11% 42% 212.50 sched_debug.cpu.nr_uninterruptible.max 98147 ± 34% 97985 ± 42% 59% 156085 ± 8% sched_debug.cpu.avg_idle.min 8554 ± 3% 4% 8896 ± 5% 62% 13822 sched_debug.cpu.nr_switches.avg 2582 ± 3% 11% 2857 ± 11% 19% 3083 ± 3% sched_debug.cpu.nr_switches.stddev 60029 ± 9% 60817 ± 7% 44% 86205 sched_debug.cpu.clock.max 60029 ± 9% 60817 ± 7% 44% 86205 sched_debug.cpu.clock_task.max 60020 ± 9% 60807 ± 7% 44% 86188 sched_debug.cpu.clock.avg 60020 ± 9% 60807 ± 7% 44% 86188 sched_debug.cpu.clock_task.avg 60008 ± 9% 60793 ± 7% 44% 86169 sched_debug.cpu.clock.min 60008 ± 9% 60793 ± 7% 44% 86169 sched_debug.cpu.clock_task.min 18.36 ± 7% -37% 11.60 ± 5% -33% 12.21 sched_debug.cpu.cpu_load[3].avg 5577 ± 6% 3% 5772 ± 6% 81% 10121 sched_debug.cpu.nr_switches.min 19.14 ± 3% -36% 12.24 -36% 12.33 sched_debug.cpu.cpu_load[4].avg 17.21 ± 14% -31% 11.90 ± 18% -27% 12.56 ± 6% sched_debug.cpu.cpu_load[2].avg 83.49 ± 7% 5% 87.64 ± 3% 17% 97.56 ± 4% sched_debug.cpu.nr_uninterruptible.stddev 3729 3735 18% 4409 ± 13% sched_debug.cpu.curr->pid.max 374 ± 9% -4% 360 ± 9% 157% 962 sched_debug.cpu.ttwu_local.min 665 671 122% 1479 sched_debug.cpu.ttwu_local.avg 196 ± 7% 5% 207 ± 8% 88% 369 ± 14% sched_debug.cpu.ttwu_local.stddev 1196 ± 4% 5% 1261 ± 6% 11% 1333 ± 10% sched_debug.cpu.curr->pid.stddev 0.45 ± 7% 17% 0.53 ± 16% 29% 0.58 ± 16% sched_debug.cpu.nr_running.avg 6738 ± 16% 8% 7296 ± 20% 52% 10236 sched_debug.cpu.ttwu_count.max 3952 ± 4% 5% 4150 ± 5% 75% 6917 sched_debug.cpu.ttwu_count.avg 913 22% 1117 ± 18% 42% 1302 ± 3% sched_debug.cpu.sched_goidle.stddev 2546 ± 4% 4% 2653 ± 7% 89% 4816 sched_debug.cpu.ttwu_count.min 5301 ± 6% 36% 7190 ± 33% 61% 8513 ± 8% sched_debug.cpu.sched_goidle.max 4683 ± 16% 14% 5355 ± 25% 52% 7125 sched_debug.cpu.sched_count.stddev 8262 ± 3% 6% 8746 ± 7% 68% 13912 sched_debug.cpu.sched_count.avg 5139 ± 5% 4% 5362 ± 6% 90% 9773 sched_debug.cpu.sched_count.min 2088 ± 6% 7% 2229 ± 5% 55% 3242 sched_debug.cpu.sched_goidle.min 3258 ± 4% 6% 3445 ± 6% 44% 4706 sched_debug.cpu.sched_goidle.avg 37088 ± 17% 12% 41540 ± 23% 60% 59447 sched_debug.cpu.sched_count.max 1007 ± 7% 13% 1139 ± 14% 38% 1386 ± 3% sched_debug.cpu.ttwu_count.stddev 262591 ± 4% -3% 253748 ± 4% -11% 232974 sched_debug.cpu.avg_idle.stddev 60009 ± 9% 60795 ± 7% 44% 86169 sched_debug.cpu_clk 58763 ± 9% 59673 ± 7% 45% 85068 sched_debug.ktime 60009 ± 9% 60795 ± 7% 44% 86169 sched_debug.sched_clk aim7/1BRD_48G-xfs-creat-clo-1500-performance/ivb44 99091700659f4df9 fe9c2c81ed073878768785a985 ---------------- -------------------------- 69789 5% 73162 aim7.jobs-per-min 81603 -7% 75897 ± 5% aim7.time.involuntary_context_switches 3825 -6% 3583 aim7.time.system_time 129.08 -5% 123.16 aim7.time.elapsed_time 129.08 -5% 123.16 aim7.time.elapsed_time.max 2536 -4% 2424 aim7.time.maximum_resident_set_size 3145 131% 7253 ± 20% numa-numastat.node1.numa_miss 3145 131% 7253 ± 20% numa-numastat.node1.numa_foreign 7059 4% 7362 vmstat.system.cs 7481848 40% 10487336 ± 8% cpuidle.C1-IVT.time 1491314 75% 2607219 ± 10% cpuidle.POLL.time 67 10% 73 ± 4% turbostat.CoreTmp 66 12% 73 ± 4% turbostat.PkgTmp 5025792 -21% 3973802 meminfo.DirectMap2M 49098 12% 54859 meminfo.PageTables 3.94 97% 7.76 ± 18% perf-profile.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry 11.88 -24% 8.99 ± 14% perf-profile.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary 11.63 -25% 8.78 ± 13% perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry 8.412e+11 -7% 7.83e+11 perf-stat.branch-instructions 0.30 0.29 perf-stat.branch-miss-rate 2.495e+09 -8% 2.292e+09 perf-stat.branch-misses 4.277e+09 -6% 4.003e+09 perf-stat.cache-misses 1.396e+10 -5% 1.327e+10 perf-stat.cache-references 1.224e+13 -8% 1.12e+13 perf-stat.cpu-cycles 0.58 -57% 0.25 ± 16% perf-stat.dTLB-load-miss-rate 5.407e+09 -60% 2.175e+09 ± 18% perf-stat.dTLB-load-misses 9.243e+11 -6% 8.708e+11 perf-stat.dTLB-loads 0.17 -58% 0.07 ± 4% perf-stat.dTLB-store-miss-rate 4.368e+08 -50% 2.177e+08 ± 3% perf-stat.dTLB-store-misses 2.549e+11 19% 3.041e+11 perf-stat.dTLB-stores 3.737e+12 -6% 3.498e+12 perf-stat.instructions 0.31 0.31 perf-stat.ipc 439716 426816 perf-stat.minor-faults 2.164e+09 -7% 2.012e+09 perf-stat.node-load-misses 2.417e+09 -7% 2.259e+09 perf-stat.node-loads 1.24e+09 -3% 1.198e+09 perf-stat.node-store-misses 1.556e+09 -4% 1.501e+09 perf-stat.node-stores 439435 426823 perf-stat.page-faults 51452 14% 58403 ± 8% numa-meminfo.node0.Active(anon) 10472 -36% 6692 ± 45% numa-meminfo.node1.Shmem 7665 74% 13316 numa-meminfo.node1.PageTables 6724 144% 16416 ± 43% latency_stats.avg.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath 6724 144% 16416 ± 43% latency_stats.max.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath 6724 144% 16416 ± 43% latency_stats.sum.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath 12237 12% 13693 proc-vmstat.nr_page_table_pages 12824 14% 14578 ± 8% numa-vmstat.node0.nr_zone_active_anon 12824 14% 14578 ± 8% numa-vmstat.node0.nr_active_anon 2618 -36% 1672 ± 45% numa-vmstat.node1.nr_shmem 17453 24% 21726 ± 6% numa-vmstat.node1.numa_miss 1909 74% 3323 numa-vmstat.node1.nr_page_table_pages 17453 24% 21726 ± 6% numa-vmstat.node1.numa_foreign 922 24% 1143 ± 6% slabinfo.blkdev_requests.active_objs 922 24% 1143 ± 6% slabinfo.blkdev_requests.num_objs 569 21% 686 ± 11% slabinfo.file_lock_cache.active_objs 569 21% 686 ± 11% slabinfo.file_lock_cache.num_objs 9.07 16% 10.56 ± 9% sched_debug.cfs_rq:/.runnable_load_avg.avg 18406 -14% 15835 ± 10% sched_debug.cfs_rq:/.load.stddev 0.67 150% 1.67 ± 43% sched_debug.cfs_rq:/.nr_spread_over.max 581 -11% 517 ± 4% sched_debug.cfs_rq:/.load_avg.min 659 -10% 596 ± 4% sched_debug.cfs_rq:/.load_avg.avg 784 -12% 692 ± 4% sched_debug.cfs_rq:/.load_avg.max 18086 -12% 15845 ± 9% sched_debug.cpu.load.stddev 18.72 -17% 15.49 ± 8% sched_debug.cpu.nr_uninterruptible.avg 69.33 42% 98.67 ± 7% sched_debug.cpu.nr_uninterruptible.max 317829 -12% 280218 ± 4% sched_debug.cpu.avg_idle.min 9.80 18% 11.54 ± 10% sched_debug.cpu.cpu_load[1].avg 8.91 15% 10.28 ± 9% sched_debug.cpu.cpu_load[0].avg 9.53 22% 11.64 ± 10% sched_debug.cpu.cpu_load[3].avg 7083 11% 7853 sched_debug.cpu.nr_switches.min 9.73 22% 11.90 ± 7% sched_debug.cpu.cpu_load[4].avg 9.68 20% 11.59 ± 11% sched_debug.cpu.cpu_load[2].avg 24.59 49% 36.53 ± 17% sched_debug.cpu.nr_uninterruptible.stddev 1176 12% 1319 ± 4% sched_debug.cpu.curr->pid.avg 373 35% 502 ± 6% sched_debug.cpu.ttwu_local.min 3644 13% 4120 ± 3% sched_debug.cpu.ttwu_count.min 4855 13% 5463 ± 6% sched_debug.cpu.sched_goidle.max 7019 10% 7745 sched_debug.cpu.sched_count.min 2305 10% 2529 ± 3% sched_debug.cpu.sched_goidle.min 0.00 -19% 0.00 ± 7% sched_debug.cpu.next_balance.stddev 0.68 -17% 0.57 ± 11% sched_debug.cpu.nr_running.stddev 0.05 27% 0.06 ± 14% sched_debug.rt_rq:/.rt_nr_running.stddev Thanks, Fengguang