From: Valentin Schneider <vschneid@redhat.com>
To: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
rcu@vger.kernel.org, x86@kernel.org,
linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev,
linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org,
linux-trace-kernel@vger.kernel.org
Cc: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
Dave Hansen <dave.hansen@linux.intel.com>,
"H. Peter Anvin" <hpa@zytor.com>,
Andy Lutomirski <luto@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Josh Poimboeuf <jpoimboe@kernel.org>,
Paolo Bonzini <pbonzini@redhat.com>,
Arnd Bergmann <arnd@arndb.de>,
Frederic Weisbecker <frederic@kernel.org>,
"Paul E. McKenney" <paulmck@kernel.org>,
Jason Baron <jbaron@akamai.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ard Biesheuvel <ardb@kernel.org>,
Sami Tolvanen <samitolvanen@google.com>,
"David S. Miller" <davem@davemloft.net>,
Neeraj Upadhyay <neeraj.upadhyay@kernel.org>,
Joel Fernandes <joelagnelf@nvidia.com>,
Josh Triplett <josh@joshtriplett.org>,
Boqun Feng <boqun.feng@gmail.com>,
Uladzislau Rezki <urezki@gmail.com>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Mel Gorman <mgorman@suse.de>,
Andrew Morton <akpm@linux-foundation.org>,
Masahiro Yamada <masahiroy@kernel.org>,
Han Shen <shenhan@google.com>, Rik van Riel <riel@surriel.com>,
Jann Horn <jannh@google.com>,
Dan Carpenter <dan.carpenter@linaro.org>,
Oleg Nesterov <oleg@redhat.com>,
Juri Lelli <juri.lelli@redhat.com>,
Clark Williams <williams@redhat.com>,
Yair Podemsky <ypodemsk@redhat.com>,
Marcelo Tosatti <mtosatti@redhat.com>,
Daniel Wagner <dwagner@suse.de>, Petr Tesarik <ptesarik@suse.com>
Subject: [RFC PATCH v6 27/29] x86/mm/pti: Implement a TLB flush immediately after a switch to kernel CR3
Date: Fri, 10 Oct 2025 17:38:37 +0200 [thread overview]
Message-ID: <20251010153839.151763-28-vschneid@redhat.com> (raw)
In-Reply-To: <20251010153839.151763-1-vschneid@redhat.com>
Deferring kernel range TLB flushes requires the guarantee that upon
entering the kernel, no stale entry may be accessed. The simplest way to
provide such a guarantee is to issue an unconditional flush upon switching
to the kernel CR3, as this is the pivoting point where such stale entries
may be accessed.
As this is only relevant to NOHZ_FULL, restrict the mechanism to NOHZ_FULL
CPUs.
Note that the COALESCE_TLBI config option is introduced in a later commit,
when the whole feature is implemented.
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
---
arch/x86/entry/calling.h | 26 +++++++++++++++++++++++---
arch/x86/kernel/asm-offsets.c | 1 +
2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 813451b1ddecc..19fb6de276eac 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -9,6 +9,7 @@
#include <asm/ptrace-abi.h>
#include <asm/msr.h>
#include <asm/nospec-branch.h>
+#include <asm/invpcid.h>
/*
@@ -171,8 +172,27 @@ For 32-bit we have the following conventions - kernel is built with
andq $(~PTI_USER_PGTABLE_AND_PCID_MASK), \reg
.endm
-.macro COALESCE_TLBI
+.macro COALESCE_TLBI scratch_reg:req
#ifdef CONFIG_COALESCE_TLBI
+ /* No point in doing this for housekeeping CPUs */
+ movslq PER_CPU_VAR(cpu_number), \scratch_reg
+ bt \scratch_reg, tick_nohz_full_mask(%rip)
+ jnc .Lend_tlbi_\@
+
+ ALTERNATIVE "jmp .Lcr4_\@", "", X86_FEATURE_INVPCID
+ movq $(INVPCID_TYPE_ALL_INCL_GLOBAL), \scratch_reg
+ /* descriptor is all zeroes, point at the zero page */
+ invpcid empty_zero_page(%rip), \scratch_reg
+ jmp .Lend_tlbi_\@
+.Lcr4_\@:
+ /* Note: this gives CR4 pinning the finger */
+ movq PER_CPU_VAR(cpu_tlbstate + TLB_STATE_cr4), \scratch_reg
+ xorq $(X86_CR4_PGE), \scratch_reg
+ movq \scratch_reg, %cr4
+ xorq $(X86_CR4_PGE), \scratch_reg
+ movq \scratch_reg, %cr4
+
+.Lend_tlbi_\@:
movl $1, PER_CPU_VAR(kernel_cr3_loaded)
#endif // CONFIG_COALESCE_TLBI
.endm
@@ -188,7 +208,7 @@ For 32-bit we have the following conventions - kernel is built with
mov %cr3, \scratch_reg
ADJUST_KERNEL_CR3 \scratch_reg
mov \scratch_reg, %cr3
- COALESCE_TLBI
+ COALESCE_TLBI \scratch_reg
.Lend_\@:
.endm
@@ -256,7 +276,7 @@ For 32-bit we have the following conventions - kernel is built with
ADJUST_KERNEL_CR3 \scratch_reg
movq \scratch_reg, %cr3
- COALESCE_TLBI
+ COALESCE_TLBI \scratch_reg
.Ldone_\@:
.endm
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index 6259b474073bc..f5abdcbb150d9 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -105,6 +105,7 @@ static void __used common(void)
/* TLB state for the entry code */
OFFSET(TLB_STATE_user_pcid_flush_mask, tlb_state, user_pcid_flush_mask);
+ OFFSET(TLB_STATE_cr4, tlb_state, cr4);
/* Layout info for cpu_entry_area */
OFFSET(CPU_ENTRY_AREA_entry_stack, cpu_entry_area, entry_stack_page);
--
2.51.0
next prev parent reply other threads:[~2025-10-10 15:47 UTC|newest]
Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-10 15:38 [PATCH v6 00/29] context_tracking,x86: Defer some IPIs until a user->kernel transition Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 01/29] objtool: Make validate_call() recognize indirect calls to pv_ops[] Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 02/29] objtool: Flesh out warning related to pv_ops[] calls Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 03/29] rcu: Add a small-width RCU watching counter debug option Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 04/29] rcutorture: Make TREE04 use CONFIG_RCU_DYNTICKS_TORTURE Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 05/29] jump_label: Add annotations for validating noinstr usage Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 06/29] static_call: Add read-only-after-init static calls Valentin Schneider
2025-10-30 10:25 ` Petr Tesarik
2025-10-31 11:52 ` Valentin Schneider
2025-11-03 8:37 ` Petr Tesarik
2025-10-10 15:38 ` [PATCH v6 07/29] x86/paravirt: Mark pv_sched_clock static call as __ro_after_init Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 08/29] x86/idle: Mark x86_idle " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 09/29] x86/paravirt: Mark pv_steal_clock " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 10/29] riscv/paravirt: " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 11/29] loongarch/paravirt: " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 12/29] arm64/paravirt: " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 13/29] arm/paravirt: " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 14/29] perf/x86/amd: Mark perf_lopwr_cb " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 15/29] sched/clock: Mark sched_clock_running key " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 16/29] KVM: VMX: Mark __kvm_is_using_evmcs static " Valentin Schneider
2025-10-14 0:02 ` Sean Christopherson
2025-10-14 11:20 ` Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 17/29] x86/speculation/mds: Mark cpu_buf_idle_clear key as allowed in .noinstr Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 18/29] sched/clock, x86: Mark __sched_clock_stable " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 19/29] KVM: VMX: Mark vmx_l1d_should flush and vmx_l1d_flush_cond keys " Valentin Schneider
2025-10-14 0:01 ` Sean Christopherson
2025-10-14 11:02 ` Valentin Schneider
2025-10-14 19:06 ` Sean Christopherson
2025-10-10 15:38 ` [PATCH v6 20/29] stackleack: Mark stack_erasing_bypass key " Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 21/29] objtool: Add noinstr validation for static branches/calls Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 22/29] module: Add MOD_NOINSTR_TEXT mem_type Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 23/29] context-tracking: Introduce work deferral infrastructure Valentin Schneider
2025-10-28 14:00 ` Frederic Weisbecker
2025-10-29 10:09 ` Valentin Schneider
2025-10-29 14:52 ` Frederic Weisbecker
2025-11-03 8:32 ` Shrikanth Hegde
2025-11-04 13:45 ` Valentin Schneider
2025-10-10 15:38 ` [PATCH v6 24/29] context_tracking,x86: Defer kernel text patching IPIs Valentin Schneider
2025-10-28 14:49 ` Frederic Weisbecker
2025-10-10 15:38 ` [PATCH v6 25/29] x86/mm: Make INVPCID type macros available to assembly Valentin Schneider
2025-10-10 15:38 ` [RFC PATCH v6 26/29] x86/mm/pti: Introduce a kernel/user CR3 software signal Valentin Schneider
2025-10-10 15:38 ` Valentin Schneider [this message]
2025-10-28 15:59 ` [RFC PATCH v6 27/29] x86/mm/pti: Implement a TLB flush immediately after a switch to kernel CR3 Frederic Weisbecker
2025-10-29 10:16 ` Valentin Schneider
2025-10-29 10:31 ` Frederic Weisbecker
2025-10-29 14:13 ` Valentin Schneider
2025-10-29 14:49 ` Frederic Weisbecker
2025-10-31 9:55 ` Valentin Schneider
2025-10-10 15:38 ` [RFC PATCH v6 28/29] x86/mm, mm/vmalloc: Defer kernel TLB flush IPIs under CONFIG_COALESCE_TLBI=y Valentin Schneider
2025-10-10 15:38 ` [RFC PATCH v6 29/29] x86/entry: Add an option to coalesce TLB flushes Valentin Schneider
2025-10-14 12:58 ` [PATCH v6 00/29] context_tracking,x86: Defer some IPIs until a user->kernel transition Juri Lelli
2025-10-14 15:26 ` Valentin Schneider
2025-10-15 13:16 ` Valentin Schneider
2025-10-15 14:28 ` Juri Lelli
2025-10-28 16:25 ` Frederic Weisbecker
2025-10-29 10:32 ` Valentin Schneider
2025-10-29 17:15 ` Frederic Weisbecker
2025-11-05 16:24 ` Valentin Schneider
2025-11-05 17:46 ` Frederic Weisbecker
2025-11-06 10:02 ` Valentin Schneider
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20251010153839.151763-28-vschneid@redhat.com \
--to=vschneid@redhat.com \
--cc=acme@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=ardb@kernel.org \
--cc=arnd@arndb.de \
--cc=boqun.feng@gmail.com \
--cc=bp@alien8.de \
--cc=dan.carpenter@linaro.org \
--cc=dave.hansen@linux.intel.com \
--cc=davem@davemloft.net \
--cc=dwagner@suse.de \
--cc=frederic@kernel.org \
--cc=hpa@zytor.com \
--cc=jannh@google.com \
--cc=jbaron@akamai.com \
--cc=joelagnelf@nvidia.com \
--cc=josh@joshtriplett.org \
--cc=jpoimboe@kernel.org \
--cc=juri.lelli@redhat.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-riscv@lists.infradead.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=loongarch@lists.linux.dev \
--cc=luto@kernel.org \
--cc=masahiroy@kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=mtosatti@redhat.com \
--cc=neeraj.upadhyay@kernel.org \
--cc=oleg@redhat.com \
--cc=paulmck@kernel.org \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=ptesarik@suse.com \
--cc=rcu@vger.kernel.org \
--cc=riel@surriel.com \
--cc=rostedt@goodmis.org \
--cc=samitolvanen@google.com \
--cc=shenhan@google.com \
--cc=tglx@linutronix.de \
--cc=urezki@gmail.com \
--cc=williams@redhat.com \
--cc=x86@kernel.org \
--cc=ypodemsk@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®