Greeting, FYI, we noticed a -11.1% regression of fsmark.files_per_sec due to commit: commit: 5e56dfbd837421b7fa3c6c06018c6701e2704917 ("mm, vmscan: consider eligible zones in get_scan_count") https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master in testcase: fsmark on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory with following parameters: iterations: 1 nr_threads: 64 disk: 3HDD md: RAID5 fs: btrfs filesize: 4M test_size: 130G sync_method: NoSync cpufreq_governor: performance test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload. test-url: https://sourceforge.net/projects/fsmark/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml testcase/path_params/tbox_group/run: fsmark/1-64-3HDD-RAID5-btrfs-4M-130G-NoSync-performance/ivb44 69ec9c5d23a4b83a 5e56dfbd837421b7fa3c6c0601 ---------------- -------------------------- %stddev change %stddev \ | \ 57.60 -11% 51.20 fsmark.files_per_sec 606 9% 659 fsmark.time.elapsed_time 606 9% 659 fsmark.time.elapsed_time.max 139 4% 145 fsmark.time.system_time 610856 626380 fsmark.time.voluntary_context_switches 311298 15% 358279 ± 8% interrupts.CAL:Function_call_interrupts 208429 -8% 191541 vmstat.io.bo 93 -5% 89 turbostat.Avg_MHz 3.17 -6% 3.00 turbostat.%Busy 576085 ± 62% -5e+05 49030 ±173% latency_stats.avg.wait_current_trans.[btrfs].start_transaction.[btrfs].btrfs_join_transaction.[btrfs].btrfs_alloc_data_chunk_ondemand.[btrfs].btrfs_check_data_free_space.[btrfs].__btrfs_buffered_write.[btrfs].btrfs_file_write_iter.[btrfs].__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 597933 ± 61% -5e+05 54999 ±173% latency_stats.max.wait_current_trans.[btrfs].start_transaction.[btrfs].btrfs_join_transaction.[btrfs].btrfs_alloc_data_chunk_ondemand.[btrfs].btrfs_check_data_free_space.[btrfs].__btrfs_buffered_write.[btrfs].btrfs_file_write_iter.[btrfs].__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 16948 ±149% -7e+03 9562 ± 56% latency_stats.sum.btrfs_insert_delayed_dir_index.[btrfs].btrfs_insert_dir_item.[btrfs].btrfs_add_link.[btrfs].btrfs_create.[btrfs].path_openat.do_filp_open.do_sys_open.SyS_open.entry_SYSCALL_64_fastpath 915770 ±170% -9e+05 12746 ± 8% latency_stats.sum.btrfs_tree_read_lock.[btrfs].btrfs_read_lock_root_node.[btrfs].btrfs_search_slot.[btrfs].btrfs_lookup_dir_item.[btrfs].btrfs_lookup_dentry.[btrfs].btrfs_lookup.[btrfs].path_openat.do_filp_open.do_sys_open.SyS_open.entry_SYSCALL_64_fastpath 34543093 ± 63% -3e+07 1519937 ±173% latency_stats.sum.wait_current_trans.[btrfs].start_transaction.[btrfs].btrfs_join_transaction.[btrfs].btrfs_alloc_data_chunk_ondemand.[btrfs].btrfs_check_data_free_space.[btrfs].__btrfs_buffered_write.[btrfs].btrfs_file_write_iter.[btrfs].__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 1.812e+08 ± 5% 16% 2.107e+08 ± 5% perf-stat.iTLB-loads 1152482 9% 1252381 perf-stat.minor-faults 1152506 9% 1252385 perf-stat.page-faults 4041609 9% 4389968 perf-stat.context-switches 2.927e+11 ± 5% 9% 3.177e+11 perf-stat.dTLB-loads 2.207e+08 7% 2.361e+08 perf-stat.iTLB-load-misses 47548 -4% 45696 perf-stat.cpu-migrations 5106 -6% 4812 ± 6% perf-stat.instructions-per-iTLB-miss 49.90 -12% 44.12 perf-stat.cache-miss-rate% 9.607e+09 ± 3% -13% 8.317e+09 ± 4% perf-stat.cache-misses 4.745e+09 ± 3% -20% 3.792e+09 ± 5% perf-stat.node-stores 208303 -8% 191422 iostat.md0.wkB/s 11.74 ± 3% 47% 17.26 ± 4% iostat.sda.rrqm/s 104126 -8% 95733 iostat.sda.wkB/s 25814 -8% 23674 iostat.sda.wrqm/s 931 -14% 801 iostat.sda.avgrq-sz 118.10 -34% 78.02 ± 10% iostat.sda.r_await 26.34 -48% 13.59 ± 12% iostat.sda.avgqu-sz 118.11 -50% 59.59 ± 12% iostat.sda.await 117.87 -50% 59.32 ± 12% iostat.sda.w_await 15.27 ± 4% 35% 20.61 ± 6% iostat.sdb.rrqm/s 104103 -8% 95711 iostat.sdb.wkB/s 25808 -8% 23668 iostat.sdb.wrqm/s 930 -14% 800 iostat.sdb.avgrq-sz 122.79 -33% 82.36 ± 7% iostat.sdb.r_await 27.23 -48% 14.15 ± 12% iostat.sdb.avgqu-sz 121.89 -49% 62.10 ± 13% iostat.sdb.await 121.67 -49% 61.83 ± 13% iostat.sdb.w_await 12.00 ± 5% 39% 16.63 ± 7% iostat.sdc.rrqm/s 229 17% 267 iostat.sdc.w/s 104125 -8% 95734 iostat.sdc.wkB/s 25812 -8% 23665 iostat.sdc.wrqm/s 904 -19% 733 iostat.sdc.avgrq-sz 115.80 -36% 74.64 ± 8% iostat.sdc.r_await 26.99 -47% 14.41 ± 10% iostat.sdc.avgqu-sz 118.17 -49% 59.81 ± 11% iostat.sdc.await 117.98 -49% 59.60 ± 12% iostat.sdc.w_await perf-stat.page-faults 1.3e+06 ++---------------------------------------------------------------+ | O | 1.28e+06 O+O O O O O | 1.26e+06 ++ O OO O O O | | O O | 1.24e+06 ++ O | | O | 1.22e+06 ++ | | | 1.2e+06 ++ | 1.18e+06 ++ | | | 1.16e+06 *+ .* .*. *. .* *.* *. * | | **.* *.** * *.**. *.* *.*.* *.*.* *.**.**. + *.*.**.**.* 1.14e+06 ++---------------------*----------------------------*------------+ perf-stat.minor-faults 1.3e+06 ++---------------------------------------------------------------+ | O | 1.28e+06 O+O O O O O | 1.26e+06 ++ O OO O O O | | O O | 1.24e+06 ++ O | | O | 1.22e+06 ++ | | | 1.2e+06 ++ | 1.18e+06 ++ | | | 1.16e+06 *+ .* .*. *. .* *.* *. * | | **.* *.** * *.**. *.* *.*.* *.*.* *.**.**. + *.*.**.**.* 1.14e+06 ++---------------------*----------------------------*------------+ perf-stat.cache-miss-rate_ 54 ++---------------------------------------------------------------------+ | | 52 ++ * * | 50 ++*. .* .*.*. *.* ** *. *. + + :: *. *.*. .* |+ **.*.* *.* * + + + + *.**. + **.* * : + *.* *.**.* | 48 *+ * * * * | | | 46 ++ | | O O O | 44 ++ O O O O O O O O | 42 ++ O O O | | O | 40 ++ O | O | 38 ++---------------------------------------------------------------------+ fsmark.time.elapsed_time 680 ++--------------------------------------------------------------------+ O O O O O | 670 ++O O O O O O O O | 660 ++ O O | | O | 650 ++ | | O | 640 ++ | | | 630 ++ | 620 ++ | | | 610 *+ .*. .**. .*. .* | | *.**.*.*.** * *.*.**.* **.*.* *.*.*.**.*.*.**.*.*.**.*.*.**.* 600 ++--------------------------------------------------------------------+ fsmark.time.elapsed_time.max 680 ++--------------------------------------------------------------------+ O O O O O | 670 ++O O O O O O O O | 660 ++ O O | | O | 650 ++ | | O | 640 ++ | | | 630 ++ | 620 ++ | | | 610 *+ .*. .**. .*. .* | | *.**.*.*.** * *.*.**.* **.*.* *.*.*.**.*.*.**.*.*.**.*.*.**.* 600 ++--------------------------------------------------------------------+ fsmark.files_per_sec 58 ++---------------------------------------------------------------------+ |.*.**.*.*.**.*.*.*.**.*.*.**. .*.*.**. .*.**.*.*.**.*.*.*.**.*.*.**.*.* 57 *+ * * | | | 56 ++ | | | 55 ++ | | | 54 ++ | | | 53 ++ | | | 52 ++ | | | 51 O+O-OO-O-O-OO-O-O-O-OO-O-O-OO------------------------------------------+ vmstat.io.bo 210000 ++*--------------------*------*--------*---------------------------+ |+ *.*.**.*.* .*.**.*.* *.* + *.**. + *.*.**.*.**.*.**.*.**.*.**.* * * * * | 205000 ++ | | | | | 200000 ++ | | | 195000 ++ O | | | | O | 190000 ++ O O O | O O O O O O O O | | O O O O | 185000 ++-----------------------------------------------------------------+ iostat.sdb.wrqm_s 26000 ++*---------------------*------*-------*--------*-------------------+ |+ *.*.*.**.*. *.*.**.* **. + *.*.* + *.**.* *.*.**.*.*.**.*.**.* 25500 *+ * * * | | | | | 25000 ++ | | | 24500 ++ | | | 24000 ++ O | | O | | | 23500 ++ O O O O | O OO O O O OO O O O | 23000 ++------------------------------------------------------------------+ iostat.sdb.wkB_s 106000 ++-----------------------------------------------------------------+ | *. | 104000 ++**.*.**.*.* .*.**.*.* *.* .*.*.**. .**.*.**.*.**.*.**.*.**.*.**.* * * * * | 102000 ++ | | | 100000 ++ | | | 98000 ++ O | | | 96000 ++ O | | O O O | 94000 O+OO O O O OO O O O O | | | 92000 ++-----------------------------------------------------------------+ iostat.sdb.avgrq-sz 940 ++--------------------------------------------------------------------+ *.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.* 920 ++ | 900 ++ | | | 880 ++ | 860 ++ | | | 840 ++ | 820 ++ O | | O | 800 ++ O | 780 ++ O O O O O | O O O OO O O O O | 760 ++--------------------------------------------------------------------+ iostat.sdb.avgqu-sz 28 ++*--------*--------*--*--------*--------*------------------*----------+ * **.*.* *.*.*.* * *.**.* *.**.* **.*.*.**.*.*.*.* *.*.**.*.* 26 ++ | 24 ++ | | | 22 ++ | 20 ++ | | | 18 ++ | 16 ++ O | | O | 14 ++ O | 12 ++O OO O O O O O OO | O O O O | 10 ++---------------------------------------------------------------------+ iostat.sdb.await 130 ++--------------------------------------------------------------------+ *.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.* 120 ++ | 110 ++ | | | 100 ++ | 90 ++ | | | 80 ++ | 70 ++ O | | O | 60 ++ O O | 50 O+O OO O OO O O O O O O | | | 40 ++--------------------------------------------------------------------+ iostat.sdb.w_await 130 ++--------------------------------------------------------------------+ *.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.* 120 ++ | 110 ++ | | | 100 ++ | 90 ++ | | | 80 ++ | 70 ++ O | | O | 60 ++ O O | 50 ++O OO O OO O O O O O O | O | 40 ++--------------------------------------------------------------------+ iostat.sda.wrqm_s 26000 ++*---------------------*------*-------*--------*-------------------+ |+ *.*.*.**.*. *.*.**.* **. + *.*.* + *.**.* *.*.**.*.*.**.*.**.* 25500 *+ * * * | | | | | 25000 ++ | | | 24500 ++ | | | 24000 ++ O | | O | | | 23500 ++ O O O O | O OO O O O OO O O O | 23000 ++------------------------------------------------------------------+ iostat.sda.wkB_s 106000 ++-----------------------------------------------------------------+ | * *. *. * | 104000 ++ *.*.**.*.* .*.**.*.* *.* + *.**. + *.*.**.*.**.*.**.*.**.*.**.* * * * * | 102000 ++ | | | 100000 ++ | | | 98000 ++ O | | | 96000 ++ O | | O O O | 94000 O+OO O O O OO O O O O | | | 92000 ++-----------------------------------------------------------------+ iostat.sda.avgrq-sz 940 ++--------------------------------------------------------------------+ *.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.* 920 ++ | 900 ++ | | | 880 ++ | 860 ++ | | | 840 ++ | 820 ++ O | | O | 800 ++ O | 780 ++ OO O O O O | O O OO O O O O | 760 ++--------------------------------------------------------------------+ iostat.sda.avgqu-sz 28 ++---------------------------------------------------------------------+ |.*. *.*.*.* .*. .**.*.*.**. .*. .**. .*. *.*.*.**.*.*.*.**.*.*.**.*.* 26 *+ * *.* * * * * * | 24 ++ | | | 22 ++ | 20 ++ | | | 18 ++ | 16 ++ O | | | 14 ++ O | 12 ++ O O O O O | O O O OO O O O O O | 10 ++---------------------------------------------------------------------+ iostat.sda.await 130 ++--------------------------------------------------------------------+ | .* | 120 *+*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.* *.*.*.**.* 110 ++ | | | 100 ++ | 90 ++ | | | 80 ++ | 70 ++ O | | | 60 ++ O O | 50 ++O OO O O OO O O O O | O O O | 40 ++--------------------------------------------------------------------+ iostat.sda.w_await 130 ++--------------------------------------------------------------------+ | .* | 120 *+*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.* *.*.*.**.* 110 ++ | | | 100 ++ | 90 ++ | | | 80 ++ | 70 ++ O | | | 60 ++ O O | 50 ++O OO O O OO O O O O | O O O | 40 ++--------------------------------------------------------------------+ iostat.sdc.wrqm_s 26000 ++*---------------------*------*-------*--------*-------------------+ |+ *.*.*.**.*. *.*.**.* **. + *.*.* + *.**.* *.*.**.*.*.**.*.**.* 25500 *+ * * * | | | | | 25000 ++ | | | 24500 ++ | | | 24000 ++ O | | O | | | 23500 ++ O O O O | O OO O O O OO O O O | 23000 ++------------------------------------------------------------------+ iostat.sdc.w_s 280 O+--------------------------------------------------------------------+ | O O O O O O O | 270 ++ OO O O O O | | | | O O O | 260 ++ | | | 250 ++ | | | 240 ++ | | | | | 230 *+*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.* | | 220 ++--------------------------------------------------------------------+ iostat.sdc.wkB_s 106000 ++-----------------------------------------------------------------+ | *. | 104000 ++**.*.**.*.* .*.**.*.* *.* .*.*.**. .**.*.**.*.**.*.**.*.**.*.**.* * * * * | 102000 ++ | | | 100000 ++ | | | 98000 ++ O | | | 96000 ++ O | | O O O | 94000 O+OO O O O OO O O O O | | | 92000 ++-----------------------------------------------------------------+ iostat.sdc.avgrq-sz 950 ++--------------------------------------------------------------------+ | | 900 *+*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.* | | | | 850 ++ | | | 800 ++ | | | 750 ++ O | | O O | | OO O O O O O O O | 700 O+O O O O | | | 650 ++--------------------------------------------------------------------+ iostat.sdc.avgqu-sz 28 ++----------------------------------------------------------*----------+ *.*.**.*.*.**.*.*.*.**.*.*.**.*.*.*.**.*.*.**.*.*.**.*.*.*.* *.*.**.*.* 26 ++ | 24 ++ | | | 22 ++ | 20 ++ | | | 18 ++ | 16 ++ O | | O | 14 ++ O O O | 12 ++O OO O OO O O O O O | O | 10 ++---------------------------------------------------------------------+ iostat.sdc.await 130 ++--------------------------------------------------------------------+ | .* | 120 *+*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.* *.*.*.**.* 110 ++ | | | 100 ++ | 90 ++ | | | 80 ++ | 70 ++ O | | O | 60 ++ O | 50 ++O OO O O OO O O O O | O O O | 40 ++--------------------------------------------------------------------+ iostat.sdc.r_await 130 ++--------------------------------------------------------------------+ | * | 120 *+*.* + * * *.**. .**.*. *. .*. *.*. * | *.*.* + .*.**.*. + *. + *.* *.* *.*.** *. : *.* +| 110 ++ * * * * * | | | 100 ++ | | | 90 ++ | | O | 80 ++ | | O | 70 ++ O O O O O O O O O | O O O O O O | 60 ++--------------------------------------------------------------------+ iostat.sdc.w_await 130 ++--------------------------------------------------------------------+ | .* | 120 *+*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.*.**.*.* *.*.*.**.* 110 ++ | | | 100 ++ | 90 ++ | | | 80 ++ | 70 ++ O | | O | 60 ++ O | 50 ++O OO O O OO O O O O | O O O | 40 ++--------------------------------------------------------------------+ iostat.md0.wkB_s 210000 ++---------------------*-------------------------------------------+ |.**.*.**.*.* .*.**.*.* *.* .*.*.**. .**.*.**.*.**.*.**.*.**.*.**.* * * * * | 205000 ++ | | | | | 200000 ++ | | | 195000 ++ O | | | | O | 190000 ++ O O O | O O O O O O O O | | O O O O | 185000 ++-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Xiaolong