From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752726AbcAFSuW (ORCPT ); Wed, 6 Jan 2016 13:50:22 -0500 Received: from terminus.zytor.com ([198.137.202.10]:54695 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751959AbcAFSuQ (ORCPT ); Wed, 6 Jan 2016 13:50:16 -0500 Date: Wed, 6 Jan 2016 10:49:34 -0800 From: tip-bot for Jiri Olsa Message-ID: Cc: linux-kernel@vger.kernel.org, efault@gmx.de, dzickus@redhat.com, peterz@infradead.org, torvalds@linux-foundation.org, acme@kernel.org, tglx@linutronix.de, jolsa@kernel.org, jmario@redhat.com, hpa@zytor.com, mingo@kernel.org Reply-To: torvalds@linux-foundation.org, acme@kernel.org, linux-kernel@vger.kernel.org, dzickus@redhat.com, efault@gmx.de, peterz@infradead.org, jmario@redhat.com, hpa@zytor.com, mingo@kernel.org, tglx@linutronix.de, jolsa@kernel.org In-Reply-To: <1449606239-28602-1-git-send-email-jolsa@kernel.org> References: <1449606239-28602-1-git-send-email-jolsa@kernel.org> To: linux-tip-commits@vger.kernel.org Subject: [tip:sched/core] sched/core: Move sched_entity:: avg into separate cache line Git-Commit-ID: 5a1078043f844074cbd53981432778a8d5dd56e9 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 5a1078043f844074cbd53981432778a8d5dd56e9 Gitweb: http://git.kernel.org/tip/5a1078043f844074cbd53981432778a8d5dd56e9 Author: Jiri Olsa AuthorDate: Tue, 8 Dec 2015 21:23:59 +0100 Committer: Ingo Molnar CommitDate: Wed, 6 Jan 2016 11:06:14 +0100 sched/core: Move sched_entity::avg into separate cache line The sched_entity::avg collides with read-mostly sched_entity data. The perf c2c tool showed many read HITM accesses across many CPUs for sched_entity's cfs_rq and my_q, while having at the same time tons of stores for avg. After placing sched_entity::avg into separate cache line, the perf bench sched pipe showed around 20 seconds speedup. NOTE I cut out all perf events except for cycles and instructions from following output. Before: $ perf stat -r 5 perf bench sched pipe -l 10000000 # Running 'sched/pipe' benchmark: # Executed 10000000 pipe operations between two processes Total time: 270.348 [sec] 27.034805 usecs/op 36989 ops/sec ... 245,537,074,035 cycles # 1.433 GHz 187,264,548,519 instructions # 0.77 insns per cycle 272.653840535 seconds time elapsed ( +- 1.31% ) After: $ perf stat -r 5 perf bench sched pipe -l 10000000 # Running 'sched/pipe' benchmark: # Executed 10000000 pipe operations between two processes Total time: 251.076 [sec] 25.107678 usecs/op 39828 ops/sec ... 244,573,513,928 cycles # 1.572 GHz 187,409,641,157 instructions # 0.76 insns per cycle 251.679315188 seconds time elapsed ( +- 0.31% ) Signed-off-by: Jiri Olsa Signed-off-by: Peter Zijlstra (Intel) Cc: Arnaldo Carvalho de Melo Cc: Don Zickus Cc: Joe Mario Cc: Linus Torvalds Cc: Mike Galbraith Cc: Peter Zijlstra Cc: Thomas Gleixner Link: http://lkml.kernel.org/r/1449606239-28602-1-git-send-email-jolsa@kernel.org Signed-off-by: Ingo Molnar --- include/linux/sched.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 791b47e..0c0e781 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1268,8 +1268,13 @@ struct sched_entity { #endif #ifdef CONFIG_SMP - /* Per entity load average tracking */ - struct sched_avg avg; + /* + * Per entity load average tracking. + * + * Put into separate cache line so it does not + * collide with read-mostly values above. + */ + struct sched_avg avg ____cacheline_aligned_in_smp; #endif };