From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753688AbbIAWmf (ORCPT ); Tue, 1 Sep 2015 18:42:35 -0400 Received: from mail.kernel.org ([198.145.29.136]:47827 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753586AbbIAWm3 (ORCPT ); Tue, 1 Sep 2015 18:42:29 -0400 From: Andy Lutomirski To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Brian Gerst , Denys Vlasenko , Linus Torvalds , Borislav Petkov , Andy Lutomirski Subject: [RFC 29/30] x86/entry: Force inlining of 32-bit syscall code Date: Tue, 1 Sep 2015 15:41:29 -0700 Message-Id: <3c6ceff654f1df8552f5d8e442475e5eec56aa7c.1441146105.git.luto@kernel.org> X-Mailer: git-send-email 2.4.3 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On systems that support fast syscalls, we only really care about the performance of the fast syscall path. Forcibly inline it and add a likely annotation. This saves 4-6 cycles. Signed-off-by: Andy Lutomirski --- arch/x86/entry/common.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 0fa55f0736f6..a3c4e7aa573b 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -324,9 +324,11 @@ __visible void syscall_return_slowpath(struct pt_regs *regs) #if defined(CONFIG_X86_32) || defined(CONFIG_IA32_EMULATION) /* * Does a 32-bit syscall. Called with IRQs on and does all entry and - * exit work and returns with IRQs off. + * exit work and returns with IRQs off. This function is extremely hot + * in workloads that use it, and it's usually called from + * do_fast_syscall_32, so forcibly inline it to improve performance. */ -static void do_syscall_32_irqs_on(struct pt_regs *regs) +static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs) { struct thread_info *ti = pt_regs_to_thread_info(regs); unsigned int nr = (unsigned int)regs->orig_ax; @@ -345,7 +347,7 @@ static void do_syscall_32_irqs_on(struct pt_regs *regs) nr = syscall_trace_enter(regs); } - if (nr < IA32_NR_syscalls) { + if (likely(nr < IA32_NR_syscalls)) { /* * It's possible that a 32-bit syscall implementation * takes a 64-bit parameter but nonetheless assumes that -- 2.4.3