From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752704AbdLAGaX (ORCPT ); Fri, 1 Dec 2017 01:30:23 -0500 Received: from mail.kernel.org ([198.145.29.99]:43216 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752592AbdLAG3v (ORCPT ); Fri, 1 Dec 2017 01:29:51 -0500 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 642E0219AA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=luto@kernel.org From: Andy Lutomirski To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, Borislav Petkov , Brian Gerst , David Laight , Kees Cook , Peter Zijlstra , Andy Lutomirski Subject: [PATCH 4/6] Unsuck "x86/entry/64: Create a percpu SYSCALL entry trampoline" Date: Thu, 30 Nov 2017 22:29:44 -0800 Message-Id: <8000c2851af491df898d232ecb7500857e2716ea.1512109321.git.luto@kernel.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This fixes a huge performance regression. Please add to the changelog: This patch actually seems to be a small speedup. With this patch, SYSCALL touches an extra cache line and an extra virtual page, but the pipeline no longer stalls waiting for SWAPGS. It seems that, at least in a tight loop, the latter outweights the former. Thanks to David Laight for an optimization tip. [end addition to changelog] Signed-off-by: Andy Lutomirski --- arch/x86/entry/entry_64.S | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index caf74a1bb3de..28f4e7553c26 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -180,14 +180,24 @@ ENTRY(entry_SYSCALL_64_trampoline) /* * x86 lacks a near absolute jump, and we can't jump to the real - * entry text with a relative jump, so we fake it using retq. + * entry text with a relative jump. We could push the target + * address and then use retq, but this destroys the pipeline on + * many CPUs (wasting over 20 cycles on Sandy Bridge). Instead, + * spill RDI and restore it in a second-stage trampoline. */ - pushq $entry_SYSCALL_64_after_hwframe - retq + pushq %rdi + movq $entry_SYSCALL_64_stage2, %rdi + jmp *%rdi END(entry_SYSCALL_64_trampoline) .popsection +ENTRY(entry_SYSCALL_64_stage2) + UNWIND_HINT_EMPTY + popq %rdi + jmp entry_SYSCALL_64_after_hwframe +END(entry_SYSCALL_64_stage2) + ENTRY(entry_SYSCALL_64) UNWIND_HINT_EMPTY /* -- 2.13.6