From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752445AbcHKALU (ORCPT ); Wed, 10 Aug 2016 20:11:20 -0400 Received: from mga14.intel.com ([192.55.52.115]:20042 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752152AbcHKALR (ORCPT ); Wed, 10 Aug 2016 20:11:17 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.28,502,1464678000"; d="scan'208";a="1012228226" From: "Huang\, Ying" To: "Huang\, Ying" Cc: Linus Torvalds , Dave Chinner , LKML , Bob Peterson , Wu Fengguang , LKP , Christoph Hellwig Subject: Re: [LKP] [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression References: <20160809143359.GA11220@yexl-desktop> <20160810230840.GS16044@dastard> <87eg5w18iu.fsf@yhuang-mobile.sh.intel.com> Date: Wed, 10 Aug 2016 17:11:16 -0700 In-Reply-To: <87eg5w18iu.fsf@yhuang-mobile.sh.intel.com> (Ying Huang's message of "Wed, 10 Aug 2016 16:58:17 -0700") Message-ID: <87a8gk17x7.fsf@yhuang-mobile.sh.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org "Huang, Ying" writes: > Hi, Linus, > > Linus Torvalds writes: > >> On Wed, Aug 10, 2016 at 4:08 PM, Dave Chinner wrote: >>> >>> That, to me, says there's a change in lock contention behaviour in >>> the workload (which we know aim7 is good at exposing). i.e. the >>> iomap change shifted contention from a sleeping lock to a spinning >>> lock, or maybe we now trigger optimistic spinning behaviour on a >>> lock we previously didn't spin on at all. >> >> Hmm. Possibly. I reacted to the lower cpu load number, but yeah, I >> could easily imagine some locking primitive difference too. >> >>> We really need instruction level perf profiles to understand >>> this - I don't have a machine with this many cpu cores available >>> locally, so I'm not sure I'm going to be able to make any progress >>> tracking it down in the short term. Maybe the lkp team has more >>> in-depth cpu usage profiles they can share? >> >> Yeah, I've occasionally wanted to see some kind of "top-25 kernel >> functions in the profile" thing. That said, when the load isn't all >> that familiar, the profiles usually are not all that easy to make >> sense of either. But comparing the before and after state might give >> us clues. > > I have started perf-profile data collection, will send out the > comparison result soon. Here is the comparison result with perf-profile data. ========================================================================================= compiler/cpufreq_governor/debug-setup/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase: gcc-6/performance/profile/1BRD_48G/xfs/x86_64-rhel/3000/debian-x86_64-2015-02-07.cgz/ivb44/disk_wrt/aim7 commit: f0c6bcba74ac51cb77aadb33ad35cb2dc1ad1506 68a9f5e7007c1afa2cf6830b690a90d0187c0684 f0c6bcba74ac51cb 68a9f5e7007c1afa2cf6830b69 ---------------- -------------------------- %stddev %change %stddev \ | \ 484435 ± 0% -13.3% 420004 ± 0% aim7.jobs-per-min 37.37 ± 0% +15.3% 43.09 ± 0% aim7.time.elapsed_time 37.37 ± 0% +15.3% 43.09 ± 0% aim7.time.elapsed_time.max 6491 ± 3% +30.8% 8491 ± 0% aim7.time.involuntary_context_switches 376.89 ± 0% +28.4% 484.11 ± 0% aim7.time.system_time 430512 ± 0% -20.1% 343838 ± 0% aim7.time.voluntary_context_switches 26816 ± 8% +10.2% 29542 ± 1% interrupts.CAL:Function_call_interrupts 125122 ± 10% -10.7% 111758 ± 12% softirqs.SCHED 24772 ± 0% -28.6% 17675 ± 0% vmstat.system.cs 53477 ± 2% +5.6% 56453 ± 0% vmstat.system.in 15627 ± 0% +27.7% 19956 ± 1% meminfo.Active(file) 16103 ± 3% +14.3% 18405 ± 8% meminfo.AnonHugePages 132898 ± 9% +15.4% 153380 ± 1% meminfo.DirectMap4k 13777 ± 5% +43.1% 19709 ± 0% meminfo.Shmem 3906 ± 0% +28.8% 5032 ± 2% proc-vmstat.nr_active_file 919.33 ± 5% +14.8% 1055 ± 8% proc-vmstat.nr_dirty 3444 ± 5% +41.8% 4884 ± 0% proc-vmstat.nr_shmem 4092 ± 14% +61.2% 6595 ± 1% proc-vmstat.pgactivate 1975 ± 15% +63.2% 3224 ± 17% slabinfo.scsi_data_buffer.active_objs 1975 ± 15% +63.2% 3224 ± 17% slabinfo.scsi_data_buffer.num_objs 464.33 ± 15% +63.3% 758.33 ± 17% slabinfo.xfs_efd_item.active_objs 464.33 ± 15% +63.3% 758.33 ± 17% slabinfo.xfs_efd_item.num_objs 1724300 ± 27% -40.5% 1025538 ± 1% sched_debug.cfs_rq:/.load.max 96.36 ± 3% +18.6% 114.32 ± 15% sched_debug.cfs_rq:/.util_avg.stddev 1724300 ± 27% -40.5% 1025538 ± 1% sched_debug.cpu.load.max 2887 ± 30% -28.2% 2073 ± 48% sched_debug.cpu.nr_load_updates.min 7.66 ± 20% -24.9% 5.75 ± 15% sched_debug.cpu.nr_uninterruptible.stddev 37.37 ± 0% +15.3% 43.09 ± 0% time.elapsed_time 37.37 ± 0% +15.3% 43.09 ± 0% time.elapsed_time.max 6491 ± 3% +30.8% 8491 ± 0% time.involuntary_context_switches 1037 ± 0% +10.8% 1148 ± 0% time.percent_of_cpu_this_job_got 376.89 ± 0% +28.4% 484.11 ± 0% time.system_time 430512 ± 0% -20.1% 343838 ± 0% time.voluntary_context_switches 24.18 ± 0% +9.0% 26.35 ± 0% turbostat.%Busy 686.00 ± 0% +9.5% 751.00 ± 0% turbostat.Avg_MHz 0.28 ± 0% -25.0% 0.21 ± 0% turbostat.CPU%c3 93.33 ± 1% +3.0% 96.15 ± 0% turbostat.CorWatt 124.61 ± 0% +2.1% 127.17 ± 0% turbostat.PkgWatt 4.74 ± 0% -2.7% 4.61 ± 1% turbostat.RAMWatt 7723 ± 0% +32.6% 10238 ± 5% numa-meminfo.node0.Active(file) 1589 ± 17% +45.5% 2313 ± 24% numa-meminfo.node0.Dirty 56052 ± 3% +58.2% 88666 ± 17% numa-meminfo.node1.Active 48142 ± 4% +64.0% 78943 ± 19% numa-meminfo.node1.Active(anon) 7908 ± 1% +22.9% 9722 ± 3% numa-meminfo.node1.Active(file) 46721 ± 3% +55.9% 72837 ± 24% numa-meminfo.node1.AnonPages 4789 ± 69% +102.3% 9687 ± 9% numa-meminfo.node1.Shmem 52991525 ± 1% -19.4% 42687208 ± 0% cpuidle.C1-IVT.time 319584 ± 1% -26.5% 234868 ± 1% cpuidle.C1-IVT.usage 3468808 ± 2% -19.8% 2783341 ± 3% cpuidle.C1E-IVT.time 46760 ± 0% -22.4% 36298 ± 0% cpuidle.C1E-IVT.usage 12590471 ± 0% -22.3% 9788585 ± 1% cpuidle.C3-IVT.time 79965 ± 0% -19.0% 64749 ± 0% cpuidle.C3-IVT.usage 1.3e+09 ± 0% +13.3% 1.473e+09 ± 0% cpuidle.C6-IVT.time 352.33 ± 8% -24.7% 265.33 ± 1% cpuidle.POLL.usage 1930 ± 0% +33.9% 2585 ± 3% numa-vmstat.node0.nr_active_file 4468 ± 7% -8.5% 4089 ± 5% numa-vmstat.node0.nr_alloc_batch 466.67 ± 4% +29.3% 603.33 ± 14% numa-vmstat.node0.nr_dirty 12026 ± 4% +64.1% 19734 ± 20% numa-vmstat.node1.nr_active_anon 1977 ± 1% +23.6% 2444 ± 1% numa-vmstat.node1.nr_active_file 3809 ± 6% +16.1% 4422 ± 4% numa-vmstat.node1.nr_alloc_batch 11671 ± 3% +55.9% 18197 ± 24% numa-vmstat.node1.nr_anon_pages 1197 ± 69% +102.3% 2422 ± 9% numa-vmstat.node1.nr_shmem 456.33 ± 57% -75.6% 111.33 ± 86% numa-vmstat.node1.nr_written 2.658e+11 ± 4% +24.7% 3.316e+11 ± 2% perf-stat.branch-instructions 0.41 ± 1% -9.1% 0.37 ± 1% perf-stat.branch-miss-rate 1.09e+09 ± 3% +13.4% 1.237e+09 ± 1% perf-stat.branch-misses 981138 ± 0% -18.1% 803696 ± 0% perf-stat.context-switches 1.511e+12 ± 5% +23.4% 1.864e+12 ± 3% perf-stat.cpu-cycles 102600 ± 1% -7.3% 95075 ± 1% perf-stat.cpu-migrations 0.26 ± 12% -30.8% 0.18 ± 10% perf-stat.dTLB-load-miss-rate 3.164e+11 ± 1% +39.9% 4.426e+11 ± 4% perf-stat.dTLB-loads 0.03 ± 26% -41.3% 0.02 ± 13% perf-stat.dTLB-store-miss-rate 2.247e+11 ± 6% +26.4% 2.839e+11 ± 2% perf-stat.dTLB-stores 1.49e+12 ± 4% +30.1% 1.939e+12 ± 2% perf-stat.instructions 43348 ± 2% +34.2% 58161 ± 12% perf-stat.instructions-per-iTLB-miss 0.99 ± 0% +5.5% 1.04 ± 0% perf-stat.ipc 262799 ± 0% +4.4% 274251 ± 1% perf-stat.minor-faults 34.12 ± 1% +2.1% 34.83 ± 0% perf-stat.node-load-miss-rate 46476754 ± 2% +4.6% 48601269 ± 1% perf-stat.node-load-misses 9.96 ± 0% +13.4% 11.30 ± 0% perf-stat.node-store-miss-rate 24460859 ± 1% +14.4% 27971097 ± 1% perf-stat.node-store-misses 262780 ± 0% +4.4% 274227 ± 1% perf-stat.page-faults 11.31 ± 1% -18.1% 9.27 ± 0% perf-profile.cycles-pp.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 1.68 ± 1% perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin 1.80 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin 2.55 ± 3% -14.2% 2.19 ± 2% perf-profile.cycles-pp.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin 0.00 ± -1% +Inf% 4.45 ± 1% perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor.iomap_apply 5.93 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write 13.71 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter 10.36 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write 0.00 ± -1% +Inf% 3.64 ± 0% perf-profile.cycles-pp.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write 1.04 ± 2% -18.9% 0.84 ± 1% perf-profile.cycles-pp.__delete_from_page_cache.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final 11.24 ± 2% -18.1% 9.21 ± 0% perf-profile.cycles-pp.__dentry_kill.dput.__fput.____fput.task_work_run 11.31 ± 2% -18.1% 9.26 ± 0% perf-profile.cycles-pp.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath 0.00 ± -1% +Inf% 1.09 ± 2% perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write 1.32 ± 4% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write 0.00 ± -1% +Inf% 2.68 ± 2% perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor 3.04 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write 1.00 ± 1% -18.0% 0.82 ± 1% perf-profile.cycles-pp.__radix_tree_lookup.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin 1.12 ± 2% -17.6% 0.92 ± 4% perf-profile.cycles-pp.__sb_start_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath 1.38 ± 2% -13.3% 1.19 ± 1% perf-profile.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end 54.10 ± 1% +13.1% 61.20 ± 0% perf-profile.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath 6.34 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin 0.00 ± -1% +Inf% 3.69 ± 1% perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor 4.02 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write 0.98 ± 5% -100.0% 0.00 ± -1% perf-profile.cycles-pp.alloc_page_buffers.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin 0.00 ± -1% +Inf% 2.56 ± 2% perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin 2.91 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin 3.42 ± 0% -20.9% 2.71 ± 2% perf-profile.cycles-pp.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final 0.00 ± -1% +Inf% 4.69 ± 0% perf-profile.cycles-pp.block_write_end.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write 6.24 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write 19.18 ± 5% -9.3% 17.40 ± 0% perf-profile.cycles-pp.call_cpuidle.cpu_startup_entry.start_secondary 0.94 ± 4% -19.8% 0.76 ± 0% perf-profile.cycles-pp.cancel_dirty_page.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage 3.95 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.copy_user_enhanced_fast_string.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write 0.00 ± -1% +Inf% 3.22 ± 0% perf-profile.cycles-pp.copy_user_enhanced_fast_string.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 19.75 ± 5% -9.8% 17.81 ± 0% perf-profile.cycles-pp.cpu_startup_entry.start_secondary 19.18 ± 5% -9.3% 17.40 ± 0% perf-profile.cycles-pp.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary 18.45 ± 5% -9.2% 16.75 ± 0% perf-profile.cycles-pp.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary 1.44 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin 0.00 ± -1% +Inf% 1.18 ± 1% perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor 1.86 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write 0.00 ± -1% +Inf% 1.53 ± 1% perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply 1.74 ± 2% -19.9% 1.40 ± 3% perf-profile.cycles-pp.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict 1.27 ± 0% -22.5% 0.99 ± 4% perf-profile.cycles-pp.destroy_inode.evict.iput.__dentry_kill.dput 2.61 ± 1% -24.3% 1.98 ± 1% perf-profile.cycles-pp.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath 2.66 ± 1% -24.3% 2.01 ± 1% perf-profile.cycles-pp.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath 1.79 ± 2% -28.2% 1.28 ± 3% perf-profile.cycles-pp.do_unlinkat.sys_unlink.entry_SYSCALL_64_fastpath 1.07 ± 3% -23.3% 0.82 ± 3% perf-profile.cycles-pp.down_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write 1.01 ± 3% -17.9% 0.83 ± 2% perf-profile.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write 11.26 ± 2% -18.1% 9.23 ± 0% perf-profile.cycles-pp.dput.__fput.____fput.task_work_run.exit_to_usermode_loop 11.21 ± 2% -18.1% 9.18 ± 0% perf-profile.cycles-pp.evict.iput.__dentry_kill.dput.__fput 11.34 ± 2% -18.1% 9.29 ± 0% perf-profile.cycles-pp.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 1.55 ± 3% perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor 1.83 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write 43.95 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write 0.00 ± -1% +Inf% 7.91 ± 1% perf-profile.cycles-pp.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 10.68 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter 1.91 ± 3% -16.4% 1.59 ± 1% perf-profile.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page 0.00 ± -1% +Inf% 9.85 ± 0% perf-profile.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write 10.96 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter 0.00 ± -1% +Inf% 52.29 ± 0% perf-profile.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write 0.00 ± -1% +Inf% 52.94 ± 0% perf-profile.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write 0.00 ± -1% +Inf% 34.35 ± 0% perf-profile.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter 0.00 ± -1% +Inf% 16.48 ± 0% perf-profile.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 11.22 ± 2% -18.1% 9.19 ± 0% perf-profile.cycles-pp.iput.__dentry_kill.dput.__fput.____fput 0.00 ± -1% +Inf% 1.55 ± 1% perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin 1.72 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin 0.00 ± -1% +Inf% 2.78 ± 0% perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor 3.39 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end 0.00 ± -1% +Inf% 3.44 ± 1% perf-profile.cycles-pp.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 3.03 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.memset_erms.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write 0.00 ± -1% +Inf% 2.43 ± 0% perf-profile.cycles-pp.memset_erms.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write 0.00 ± -1% +Inf% 9.25 ± 0% perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply 10.37 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write 2.58 ± 1% -24.1% 1.96 ± 0% perf-profile.cycles-pp.path_openat.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath 1.17 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin 2.06 ± 3% -22.5% 1.60 ± 2% perf-profile.cycles-pp.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath 1.79 ± 3% -22.2% 1.39 ± 0% perf-profile.cycles-pp.security_file_permission.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath 1.32 ± 4% -21.4% 1.04 ± 0% perf-profile.cycles-pp.selinux_file_permission.security_file_permission.rw_verify_area.vfs_write.sys_write 19.79 ± 5% -9.9% 17.84 ± 0% perf-profile.cycles-pp.start_secondary 2.67 ± 1% -24.2% 2.02 ± 1% perf-profile.cycles-pp.sys_creat.entry_SYSCALL_64_fastpath 1.79 ± 3% -27.9% 1.29 ± 3% perf-profile.cycles-pp.sys_unlink.entry_SYSCALL_64_fastpath 60.98 ± 1% +9.5% 66.76 ± 0% perf-profile.cycles-pp.sys_write.entry_SYSCALL_64_fastpath 11.34 ± 1% -18.1% 9.29 ± 0% perf-profile.cycles-pp.syscall_return_slowpath.entry_SYSCALL_64_fastpath 11.32 ± 1% -18.0% 9.28 ± 0% perf-profile.cycles-pp.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath 5.96 ± 1% -20.0% 4.77 ± 0% perf-profile.cycles-pp.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput 9.89 ± 2% -17.4% 8.17 ± 0% perf-profile.cycles-pp.truncate_inode_pages_final.evict.iput.__dentry_kill.dput 9.87 ± 2% -17.5% 8.15 ± 0% perf-profile.cycles-pp.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.__dentry_kill 2.07 ± 1% -20.4% 1.65 ± 2% perf-profile.cycles-pp.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage 2.40 ± 1% -21.0% 1.89 ± 2% perf-profile.cycles-pp.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range 0.00 ± -1% +Inf% 1.36 ± 1% perf-profile.cycles-pp.unlock_page.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write 1.72 ± 4% -100.0% 0.00 ± -1% perf-profile.cycles-pp.unlock_page.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write 59.63 ± 1% +10.2% 65.72 ± 0% perf-profile.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 1.52 ± 2% perf-profile.cycles-pp.workingset_activation.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write 0.00 ± -1% +Inf% 1.73 ± 1% perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply 0.00 ± -1% +Inf% 1.97 ± 2% perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write 0.00 ± -1% +Inf% 1.61 ± 2% perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin 0.00 ± -1% +Inf% 1.24 ± 2% perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin 0.00 ± -1% +Inf% 1.46 ± 1% perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply 0.00 ± -1% +Inf% 1.21 ± 2% perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay 1.25 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int 0.00 ± -1% +Inf% 3.06 ± 1% perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write 1.04 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_bmapi_read.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin 0.00 ± -1% +Inf% 3.04 ± 1% perf-profile.cycles-pp.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 0.00 ± -1% +Inf% 3.05 ± 1% perf-profile.cycles-pp.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply 1.32 ± 2% -21.5% 1.04 ± 1% perf-profile.cycles-pp.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat 51.83 ± 1% +14.3% 59.25 ± 0% perf-profile.cycles-pp.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write.sys_write 0.00 ± -1% +Inf% 16.05 ± 0% perf-profile.cycles-pp.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter 53.16 ± 1% +13.6% 60.40 ± 0% perf-profile.cycles-pp.xfs_file_write_iter.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath 1.24 ± 1% -23.1% 0.95 ± 4% perf-profile.cycles-pp.xfs_fs_destroy_inode.destroy_inode.evict.iput.__dentry_kill 1.42 ± 2% -21.2% 1.12 ± 1% perf-profile.cycles-pp.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open 6.46 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write 1.29 ± 3% -18.9% 1.04 ± 1% perf-profile.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write 0.00 ± -1% +Inf% 1.14 ± 3% perf-profile.cycles-pp.xfs_ilock.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 1.21 ± 1% -23.4% 0.93 ± 4% perf-profile.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput 1.23 ± 4% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.6.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int 0.00 ± -1% +Inf% 4.14 ± 0% perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write 3.28 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin 0.00 ± -1% +Inf% 9.08 ± 0% perf-profile.cycles-pp.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write 3.54 ± 0% -20.8% 2.81 ± 1% perf-profile.cycles-pp.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict 2.35 ± 1% -21.0% 1.86 ± 1% perf-profile.cycles-pp.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page 25.10 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write 11.03 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write 1.42 ± 2% -20.7% 1.13 ± 1% perf-profile.cycles-pp.xfs_vn_create.path_openat.do_filp_open.do_sys_open.sys_creat 1.42 ± 2% -20.5% 1.13 ± 1% perf-profile.cycles-pp.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open 2.27 ± 1% -10.6% 2.03 ± 0% perf-profile.func.cycles-pp.___might_sleep 2.49 ± 0% -34.5% 1.63 ± 1% perf-profile.func.cycles-pp.__block_commit_write.isra.24 1.51 ± 2% +15.4% 1.75 ± 1% perf-profile.func.cycles-pp.__block_write_begin_int 1.79 ± 4% -16.8% 1.49 ± 1% perf-profile.func.cycles-pp.__mark_inode_dirty 1.32 ± 0% -16.4% 1.10 ± 1% perf-profile.func.cycles-pp.__radix_tree_lookup 1.08 ± 2% -100.0% 0.00 ± -1% perf-profile.func.cycles-pp.__xfs_get_blocks 1.16 ± 0% -18.1% 0.95 ± 1% perf-profile.func.cycles-pp._raw_spin_lock 3.96 ± 2% -18.4% 3.23 ± 0% perf-profile.func.cycles-pp.copy_user_enhanced_fast_string 1.41 ± 3% -20.6% 1.12 ± 3% perf-profile.func.cycles-pp.entry_SYSCALL_64_fastpath 1.30 ± 2% -100.0% 0.00 ± -1% perf-profile.func.cycles-pp.generic_perform_write 1.31 ± 2% -46.7% 0.70 ± 0% perf-profile.func.cycles-pp.generic_write_end 18.43 ± 5% -9.1% 16.76 ± 0% perf-profile.func.cycles-pp.intel_idle 0.00 ± -1% +Inf% 1.12 ± 1% perf-profile.func.cycles-pp.iomap_write_actor 1.50 ± 1% -20.9% 1.19 ± 1% perf-profile.func.cycles-pp.mark_buffer_dirty 0.00 ± -1% +Inf% 1.91 ± 1% perf-profile.func.cycles-pp.mark_page_accessed 3.24 ± 0% -19.8% 2.60 ± 0% perf-profile.func.cycles-pp.memset_erms 1.75 ± 2% -18.9% 1.42 ± 1% perf-profile.func.cycles-pp.unlock_page 1.16 ± 1% -21.6% 0.91 ± 1% perf-profile.func.cycles-pp.vfs_write 0.37 ± 2% +243.6% 1.26 ± 2% perf-profile.func.cycles-pp.xfs_bmap_search_extents 0.41 ± 1% +198.4% 1.22 ± 2% perf-profile.func.cycles-pp.xfs_bmap_search_multi_extents 0.70 ± 5% +219.5% 2.24 ± 0% perf-profile.func.cycles-pp.xfs_bmapi_read 1.05 ± 2% -15.6% 0.88 ± 3% perf-profile.func.cycles-pp.xfs_file_write_iter 0.64 ± 1% +182.8% 1.81 ± 4% perf-profile.func.cycles-pp.xfs_iext_bno_to_ext 0.00 ± -1% +Inf% 1.10 ± 3% perf-profile.func.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8 0.46 ± 4% +161.6% 1.20 ± 1% perf-profile.func.cycles-pp.xfs_iomap_write_delay Best Regards, Huang, Ying