From: Willy Tarreau <w@1wt.eu>
To: linux-kernel@vger.kernel.org, x86@kernel.org
Cc: tglx@linutronix.de, gnomes@lxorguk.ukuu.org.uk,
torvalds@linux-foundation.org, Willy Tarreau <w@1wt.eu>
Subject: [PATCH RFC 4/4] x86/entry/pti: don't switch PGD on tasks holding flag TIF_NOPTI
Date: Mon, 8 Jan 2018 17:12:19 +0100 [thread overview]
Message-ID: <1515427939-10999-5-git-send-email-w@1wt.eu> (raw)
In-Reply-To: <1515427939-10999-1-git-send-email-w@1wt.eu>
If a task has the TIF_NOPTI flag set, it doesn't want to experience
page table isolation. In this case, returns from kernel to user will
not switch the CR3, leaving it to the kernel one which already maps
both user and kernel pages. Upon entry in the kernel, we can't check
this flag so we simply check if CR3 was pointing to the kernel's PGD,
indicating an earlier absence of switch, and in this case we don't
change it.
Thanks to these changes, haproxy running under KVM went back from
12400 conn/s to 21000 once loaded after calling prctl().
Signed-off-by: Willy Tarreau <w@1wt.eu>
---
arch/x86/entry/calling.h | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 45a63e0..054b8b7 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -1,5 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/jump_label.h>
+#include <asm/thread_info.h>
#include <asm/unwind_hints.h>
#include <asm/cpufeatures.h>
#include <asm/page_types.h>
@@ -214,6 +215,11 @@
.macro SWITCH_TO_KERNEL_CR3 scratch_reg:req
ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI
mov %cr3, \scratch_reg
+
+ /* if we're already on the kernel PGD, we don't switch */
+ testq $(PTI_SWITCH_PGTABLES_MASK), \scratch_reg
+ jz .Lend_\@
+
ADJUST_KERNEL_CR3 \scratch_reg
mov \scratch_reg, %cr3
.Lend_\@:
@@ -224,6 +230,12 @@
.macro SWITCH_TO_USER_CR3_NOSTACK scratch_reg:req scratch_reg2:req
ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI
+
+ /* "NOPTI" taskflag avoids the switch */
+ movq PER_CPU_VAR(current_task), \scratch_reg
+ btq $TIF_NOPTI, TASK_TI_flags(\scratch_reg)
+ jc .Lend_\@
+
mov %cr3, \scratch_reg
ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID
@@ -262,6 +274,13 @@
ALTERNATIVE "jmp .Ldone_\@", "", X86_FEATURE_PTI
movq %cr3, \scratch_reg
movq \scratch_reg, \save_reg
+
+ /* if we're already on the kernel PGD, we don't switch,
+ * we just save the current cr3.
+ */
+ testq $(PTI_SWITCH_PGTABLES_MASK), \scratch_reg
+ jz .Ldone_\@
+
/*
* Is the "switch mask" all zero? That means that both of
* these are zero:
@@ -284,6 +303,10 @@
.macro RESTORE_CR3 scratch_reg:req save_reg:req
ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI
+ /* if we saved a kernel context, we didn't switch so we don't switch */
+ testq $(PTI_SWITCH_PGTABLES_MASK), \save_reg
+ jz .Lend_\@
+
ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID
/*
--
1.7.12.1
next prev parent reply other threads:[~2018-01-08 16:13 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-08 16:12 [PATCH RFC 0/4] Per-task PTI activation Willy Tarreau
2018-01-08 16:12 ` [PATCH RFC 1/4] x86/thread_info: add TIF_NOPTI to disable PTI per task Willy Tarreau
2018-01-08 16:57 ` Thomas Gleixner
2018-01-08 17:03 ` Willy Tarreau
2018-01-08 17:14 ` Ingo Molnar
2018-01-08 16:12 ` [PATCH RFC 2/4] x86/arch_prctl: add ARCH_GET_NOPTI and ARCH_SET_NOPTI to enable/disable PTI Willy Tarreau
2018-01-08 16:49 ` Peter Zijlstra
2018-01-08 16:56 ` Willy Tarreau
2018-01-08 17:02 ` Thomas Gleixner
2018-01-08 17:10 ` Willy Tarreau
2018-01-08 17:17 ` Ingo Molnar
2018-01-08 17:26 ` Thomas Gleixner
2018-01-08 17:46 ` Ingo Molnar
2018-01-08 17:05 ` Ingo Molnar
2018-01-08 17:19 ` Peter Zijlstra
2018-01-08 17:26 ` Ingo Molnar
2018-01-08 17:50 ` Borislav Petkov
2018-01-08 17:54 ` Linus Torvalds
2018-01-08 18:22 ` Willy Tarreau
2018-01-08 20:49 ` Thomas Gleixner
2018-01-08 21:03 ` Willy Tarreau
2018-01-08 20:35 ` Willy Tarreau
2018-01-08 16:12 ` [PATCH RFC 3/4] x86/pti: don't mark the user PGD with _PAGE_NX Willy Tarreau
2018-01-08 17:03 ` Dave Hansen
2018-01-08 17:17 ` Willy Tarreau
2018-01-08 17:23 ` Dave Hansen
2018-01-08 17:30 ` Peter Zijlstra
2018-01-08 17:49 ` Willy Tarreau
2018-01-08 17:21 ` Ingo Molnar
2018-01-08 23:05 ` Andy Lutomirski
2018-01-08 23:09 ` Kees Cook
2018-01-09 4:22 ` Willy Tarreau
2018-01-08 17:05 ` Thomas Gleixner
2018-01-08 17:28 ` Dave Hansen
2018-01-08 17:50 ` Ingo Molnar
2018-01-08 18:25 ` Alan Cox
2018-01-08 18:35 ` Ingo Molnar
2018-01-08 18:35 ` Linus Torvalds
2018-01-08 18:44 ` Dave Hansen
2018-01-08 16:12 ` Willy Tarreau [this message]
2018-01-08 17:11 ` [PATCH RFC 4/4] x86/entry/pti: don't switch PGD on tasks holding flag TIF_NOPTI Ingo Molnar
2018-01-08 17:20 ` Dave Hansen
2018-01-08 18:12 ` Willy Tarreau
2018-01-08 23:01 ` Andy Lutomirski
2018-01-08 16:59 ` [PATCH RFC 0/4] Per-task PTI activation Dave Hansen
2018-01-08 17:06 ` Willy Tarreau
2018-01-08 17:17 ` Dave Hansen
2018-01-08 17:13 ` Ingo Molnar
2018-01-09 15:31 ` Eric W. Biederman
2018-01-09 16:02 ` Willy Tarreau
2018-01-09 18:11 ` Zhi Wang
2018-01-09 21:07 ` Eric W. Biederman
2018-01-09 21:57 ` Willy Tarreau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1515427939-10999-5-git-send-email-w@1wt.eu \
--to=w@1wt.eu \
--cc=gnomes@lxorguk.ukuu.org.uk \
--cc=linux-kernel@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®