From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9045CC43381 for ; Wed, 13 Feb 2019 21:37:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6847F2086C for ; Wed, 13 Feb 2019 21:37:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436851AbfBMVhC (ORCPT ); Wed, 13 Feb 2019 16:37:02 -0500 Received: from mga02.intel.com ([134.134.136.20]:41474 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2436839AbfBMVg7 (ORCPT ); Wed, 13 Feb 2019 16:36:59 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Feb 2019 13:36:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,366,1544515200"; d="scan'208";a="274859833" Received: from chang-linux-2.sc.intel.com ([10.3.52.165]) by orsmga004.jf.intel.com with ESMTP; 13 Feb 2019 13:36:58 -0800 From: "Chang S. Bae" To: Andy Lutomirski , Thomas Gleixner , Ingo Molnar , "H . Peter Anvin" , Andi Kleen Cc: Ravi Shankar , "Chang S . Bae" , LKML , Andrew Cooper Subject: [PATCH v6 04/12] x86/fsgsbase/64: Enable FSGSBASE instructions in the helper functions Date: Wed, 13 Feb 2019 13:36:16 -0800 Message-Id: <20190213213624.14354-5-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190213213624.14354-1-chang.seok.bae@intel.com> References: <20190213213624.14354-1-chang.seok.bae@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The helper functions will switch on faster accesses to FSBASE and GSBASE when the FSGSBASE feature is enabled. Accessing user GSBASE needs a couple of SWAPGS operations. It is avoidable if the user GSBASE is saved at kernel entry, being updated as changes, and restored back at kernel exit. However, it seems to spend more cycles for savings and restorations. Little or no benefit was measured from experiments. Also, introduce __{rd,wr}gsbase_inactive() as helpers to access user GSBASE with SWAPGS. Note, for Xen PV, paravirt hooks can be added, since it may allow a very efficient but different implementation. Signed-off-by: Chang S. Bae Cc: Any Lutomirski Cc: H. Peter Anvin Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Andrew Cooper --- arch/x86/include/asm/fsgsbase.h | 27 +++++++--------- arch/x86/kernel/process_64.c | 56 +++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/fsgsbase.h b/arch/x86/include/asm/fsgsbase.h index fdd1177499b4..aefd53767a5d 100644 --- a/arch/x86/include/asm/fsgsbase.h +++ b/arch/x86/include/asm/fsgsbase.h @@ -49,35 +49,32 @@ static __always_inline void wrgsbase(unsigned long gsbase) asm volatile("wrgsbase %0" :: "r" (gsbase) : "memory"); } +#include + /* Helper functions for reading/writing FS/GS base */ static inline unsigned long x86_fsbase_read_cpu(void) { unsigned long fsbase; - rdmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + fsbase = rdfsbase(); + else + rdmsrl(MSR_FS_BASE, fsbase); return fsbase; } -static inline unsigned long x86_gsbase_read_cpu_inactive(void) -{ - unsigned long gsbase; - - rdmsrl(MSR_KERNEL_GS_BASE, gsbase); - - return gsbase; -} - static inline void x86_fsbase_write_cpu(unsigned long fsbase) { - wrmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + wrfsbase(fsbase); + else + wrmsrl(MSR_FS_BASE, fsbase); } -static inline void x86_gsbase_write_cpu_inactive(unsigned long gsbase) -{ - wrmsrl(MSR_KERNEL_GS_BASE, gsbase); -} +extern unsigned long x86_gsbase_read_cpu_inactive(void); +extern void x86_gsbase_write_cpu_inactive(unsigned long gsbase); #endif /* CONFIG_X86_64 */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 6a62f4af9fcf..87b5ffce2a47 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -160,6 +160,42 @@ enum which_selector { GS }; +/* + * Interrupts are disabled here. Out of line to be protected + * from kprobes. It is not used on Xen paravirt. When paravirt + * support is needed, it needs to be renamed with native_ prefix. + */ +static noinline unsigned long __rdgsbase_inactive(void) +{ + unsigned long gsbase, flags; + + local_irq_save(flags); + native_swapgs(); + gsbase = rdgsbase(); + native_swapgs(); + local_irq_restore(flags); + + return gsbase; +} +NOKPROBE_SYMBOL(__rdgsbase_inactive); + +/* + * Interrupts are disabled here. Out of line to be protected + * from kprobes. It is not used on Xen paravirt. When paravirt + * support is needed, it needs to be renamed with native_ prefix. + */ +static noinline void __wrgsbase_inactive(unsigned long gsbase) +{ + unsigned long flags; + + local_irq_save(flags); + native_swapgs(); + wrgsbase(gsbase); + native_swapgs(); + local_irq_restore(flags); +} +NOKPROBE_SYMBOL(__wrgsbase_inactive); + /* * Saves the FS or GS base for an outgoing thread if FSGSBASE extensions are * not available. The goal is to be reasonably fast on non-FSGSBASE systems. @@ -338,6 +374,26 @@ static unsigned long x86_fsgsbase_read_task(struct task_struct *task, return base; } +unsigned long x86_gsbase_read_cpu_inactive(void) +{ + unsigned long gsbase; + + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + gsbase = __rdgsbase_inactive(); + else + rdmsrl(MSR_KERNEL_GS_BASE, gsbase); + + return gsbase; +} + +void x86_gsbase_write_cpu_inactive(unsigned long gsbase) +{ + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + __wrgsbase_inactive(gsbase); + else + wrmsrl(MSR_KERNEL_GS_BASE, gsbase); +} + unsigned long x86_fsbase_read_task(struct task_struct *task) { unsigned long fsbase; -- 2.19.1