Greeting, FYI, we noticed a -42.5% regression of stress-ng.kill.ops_per_sec due to commit: commit: 1ad22fcc4d0d2fb2e0f35aed555a86d016d5e590 ("apparmor: rework profile->rules to be a list") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: stress-ng on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory with following parameters: nr_threads: 10% disk: 1HDD testtime: 60s fs: ext4 class: os test: kill cpufreq_governor: performance If you fix the issue, kindly add following tag | Reported-by: kernel test robot | Link: https://lore.kernel.org/oe-lkp/202212311546.755a3ed7-oliver.sang@intel.com Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests sudo bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run sudo bin/lkp run generated-yaml-file # if come across any failure that blocks the test, # please remove ~/.lkp and /lkp dir to run from a clean state. ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: os/gcc-11/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp7/kill/stress-ng/60s commit: 217af7e2f4 ("apparmor: refactor profile rules and attachments") 1ad22fcc4d ("apparmor: rework profile->rules to be a list") 217af7e2f4deb629 1ad22fcc4d0d2fb2e0f35aed555 ---------------- --------------------------- %stddev %change %stddev \ | \ 2820 ± 2% -42.7% 1616 stress-ng.kill.kill_calls_per_sec 511090 -42.5% 293660 stress-ng.kill.ops 8518 -42.5% 4894 stress-ng.kill.ops_per_sec 3778 ± 2% -58.4% 1571 ± 3% stress-ng.time.involuntary_context_switches 859945 ± 3% -29.7% 604857 ± 2% stress-ng.time.voluntary_context_switches 70351 ± 12% +22.0% 85795 ± 8% meminfo.AnonHugePages 0.05 ± 7% -22.6% 0.04 turbostat.IPC 28627 ± 3% -27.0% 20903 ± 2% vmstat.system.cs 128187 ± 92% +78.6% 228921 ± 49% sched_debug.cfs_rq:/.min_vruntime.max 20901 ±107% +124.7% 46965 ± 59% sched_debug.cfs_rq:/.min_vruntime.stddev -24201 -149.5% 11969 ± 74% sched_debug.cfs_rq:/.spread0.avg 69818 ±123% +170.9% 189126 ± 59% sched_debug.cfs_rq:/.spread0.max 20904 ±107% +124.7% 46965 ± 59% sched_debug.cfs_rq:/.spread0.stddev 1.19 ± 8% +6.3 7.48 ± 10% perf-profile.calltrace.cycles-pp.aa_may_signal.apparmor_task_kill.security_task_kill.kill_something_info.__x64_sys_kill 0.58 ± 8% -0.2 0.40 ± 13% perf-profile.children.cycles-pp.check_kill_permission 0.30 ± 12% -0.1 0.16 ± 8% perf-profile.children.cycles-pp.profile_signal_perm 0.20 ± 10% -0.1 0.12 ± 13% perf-profile.children.cycles-pp.audit_signal_info_syscall 0.21 ± 23% -0.1 0.14 ± 12% perf-profile.children.cycles-pp.pause 0.18 ± 16% -0.1 0.11 ± 20% perf-profile.children.cycles-pp.__task_pid_nr_ns 0.15 ± 10% -0.1 0.09 ± 14% perf-profile.children.cycles-pp.__kill_pgrp_info 0.16 ± 13% -0.1 0.10 ± 14% perf-profile.children.cycles-pp.exit_to_user_mode_loop 0.16 ± 15% -0.1 0.11 ± 12% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.10 ± 9% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.audit_signal_info 0.07 ± 18% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.alloc_empty_file 0.07 ± 18% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.__alloc_file 0.11 ± 7% -0.0 0.08 ± 16% perf-profile.children.cycles-pp.do_send_sig_info 0.10 ± 12% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.__send_signal_locked 0.10 ± 17% -0.0 0.06 ± 19% perf-profile.children.cycles-pp.arch_do_signal_or_restart 0.06 ± 14% +0.0 0.11 ± 18% perf-profile.children.cycles-pp.ns_capable 0.06 ± 14% +0.0 0.11 ± 18% perf-profile.children.cycles-pp.security_capable 0.06 ± 14% +0.0 0.11 ± 18% perf-profile.children.cycles-pp.apparmor_capable 0.09 ± 11% +0.0 0.14 ± 23% perf-profile.children.cycles-pp.kernel_clone 0.09 ± 8% +0.0 0.14 ± 24% perf-profile.children.cycles-pp.copy_process 0.04 ± 71% +0.0 0.08 ± 18% perf-profile.children.cycles-pp.__do_sys_clone 0.07 ± 18% +0.1 0.16 ± 9% perf-profile.children.cycles-pp.__x64_sys_exit_group 0.07 ± 18% +0.1 0.16 ± 9% perf-profile.children.cycles-pp.do_group_exit 0.07 ± 18% +0.1 0.16 ± 9% perf-profile.children.cycles-pp.do_exit 0.06 ± 47% +0.1 0.15 ± 8% perf-profile.children.cycles-pp.exit_notify 0.29 ± 12% +0.1 0.38 ± 11% perf-profile.children.cycles-pp.queued_read_lock_slowpath 0.15 ± 10% +0.1 0.28 ± 13% perf-profile.children.cycles-pp.queued_write_lock_slowpath 0.36 ± 10% +0.1 0.50 ± 10% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.24 ± 8% +6.3 7.56 ± 10% perf-profile.children.cycles-pp.aa_may_signal 0.28 ± 12% -0.1 0.15 ± 10% perf-profile.self.cycles-pp.profile_signal_perm 0.23 ± 16% -0.1 0.12 ± 17% perf-profile.self.cycles-pp.check_kill_permission 0.20 ± 10% -0.1 0.12 ± 13% perf-profile.self.cycles-pp.audit_signal_info_syscall 0.15 ± 11% -0.1 0.08 ± 13% perf-profile.self.cycles-pp.kill_something_info 0.16 ± 6% -0.1 0.10 ± 20% perf-profile.self.cycles-pp.security_task_kill 0.14 ± 11% -0.1 0.09 ± 14% perf-profile.self.cycles-pp.__kill_pgrp_info 0.16 ± 15% -0.1 0.10 ± 22% perf-profile.self.cycles-pp.__task_pid_nr_ns 0.08 ± 13% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.audit_signal_info 0.06 ± 14% +0.0 0.11 ± 16% perf-profile.self.cycles-pp.apparmor_capable 0.36 ± 10% +0.1 0.50 ± 10% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.96 ± 8% +6.4 7.38 ± 10% perf-profile.self.cycles-pp.aa_may_signal 16.17 +18.8% 19.21 perf-stat.i.MPKI 8.529e+08 -30.2% 5.952e+08 perf-stat.i.branch-instructions 0.68 ± 3% +0.1 0.79 ± 2% perf-stat.i.branch-miss-rate% 7256065 ± 3% -7.7% 6698864 ± 2% perf-stat.i.branch-misses 13.08 ± 3% +6.0 19.07 perf-stat.i.cache-miss-rate% 9361692 ± 2% +16.3% 10884924 ± 2% perf-stat.i.cache-misses 71450858 -20.7% 56687571 ± 2% perf-stat.i.cache-references 29596 ± 3% -27.5% 21452 ± 2% perf-stat.i.context-switches 6.21 +50.4% 9.34 perf-stat.i.cpi 2974 ± 2% -14.4% 2547 ± 2% perf-stat.i.cycles-between-cache-misses 0.02 ± 13% +0.0 0.03 ± 6% perf-stat.i.dTLB-load-miss-rate% 1.408e+09 -32.2% 9.544e+08 perf-stat.i.dTLB-loads 7.763e+08 -33.2% 5.183e+08 perf-stat.i.dTLB-stores 34.02 -3.0 31.06 perf-stat.i.iTLB-load-miss-rate% 1109352 ± 2% -18.6% 903511 perf-stat.i.iTLB-load-misses 2150307 -6.8% 2003881 perf-stat.i.iTLB-loads 4.552e+09 -30.7% 3.153e+09 perf-stat.i.instructions 4134 ± 2% -15.1% 3508 perf-stat.i.instructions-per-iTLB-miss 0.19 -26.4% 0.14 perf-stat.i.ipc 814.65 -16.5% 680.21 perf-stat.i.metric.K/sec 31.63 -31.9% 21.53 perf-stat.i.metric.M/sec 80.72 ± 3% +9.8 90.56 perf-stat.i.node-load-miss-rate% 325431 ± 5% +970.6% 3483935 ± 2% perf-stat.i.node-load-misses 79936 ± 19% +323.7% 338676 ± 6% perf-stat.i.node-loads 4260901 ± 2% -34.5% 2792405 perf-stat.i.node-store-misses 15.70 +14.6% 18.00 perf-stat.overall.MPKI 0.85 ± 2% +0.3 1.12 ± 2% perf-stat.overall.branch-miss-rate% 13.11 ± 2% +6.1 19.20 perf-stat.overall.cache-miss-rate% 6.03 +44.8% 8.73 perf-stat.overall.cpi 2931 -13.8% 2527 ± 2% perf-stat.overall.cycles-between-cache-misses 0.02 ± 10% +0.0 0.03 ± 7% perf-stat.overall.dTLB-load-miss-rate% 34.03 -3.0 31.07 perf-stat.overall.iTLB-load-miss-rate% 4104 ± 2% -15.0% 3487 perf-stat.overall.instructions-per-iTLB-miss 0.17 -30.9% 0.11 perf-stat.overall.ipc 80.44 ± 3% +10.7 91.14 perf-stat.overall.node-load-miss-rate% 8.39e+08 -30.3% 5.852e+08 perf-stat.ps.branch-instructions 7111005 ± 2% -7.8% 6555177 ± 2% perf-stat.ps.branch-misses 9215998 ± 2% +16.3% 10714798 ± 2% perf-stat.ps.cache-misses 70327211 -20.7% 55796363 ± 2% perf-stat.ps.cache-references 29136 ± 3% -27.5% 21118 ± 2% perf-stat.ps.context-switches 1.385e+09 -32.2% 9.386e+08 perf-stat.ps.dTLB-loads 7.64e+08 -33.3% 5.099e+08 perf-stat.ps.dTLB-stores 1091797 ± 2% -18.6% 888828 perf-stat.ps.iTLB-load-misses 2116297 -6.8% 1971620 perf-stat.ps.iTLB-loads 4.478e+09 -30.8% 3.1e+09 perf-stat.ps.instructions 320258 ± 5% +970.9% 3429628 ± 2% perf-stat.ps.node-load-misses 78661 ± 19% +323.8% 333386 ± 6% perf-stat.ps.node-loads 4195116 ± 2% -34.5% 2749032 perf-stat.ps.node-store-misses 2.847e+11 -31.2% 1.959e+11 perf-stat.total.instructions Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests