From: Dave Hansen <dave.hansen@linux.intel.com>
To: linux-kernel@vger.kernel.org
Cc: Dave Hansen <dave.hansen@linux.intel.com>,
Andy Lutomirski <luto@kernel.org>, Borislav Petkov <bp@alien8.de>,
David Hildenbrand <david@kernel.org>,
"H. Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>,
"Liam R. Howlett" <liam@infradead.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
linux-arch@vger.kernel.org, linux-mm@kvack.org,
Lorenzo Stoakes <ljs@kernel.org>, Michal Hocko <mhocko@suse.com>,
Mike Rapoport <rppt@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Rik van Riel <riel@surriel.com>,
Suren Baghdasaryan <surenb@google.com>,
Thomas Gleixner <tglx@kernel.org>,
Vlastimil Babka <vbabka@kernel.org>,
x86@kernel.org
Subject: [PATCH] smp: Add memory barrier before many-CPU IPIs
Date: Fri, 20 Nov 2026 14:47:58 -0800 [thread overview]
Message-ID: <20261120224758.04F70CD8@davehans-spike.ostc.intel.com> (raw)
From: Dave Hansen <dave.hansen@linux.intel.com>
This innocent-looking snippet:
mm->foo = bar;
on_each_cpu_mask(mm_cpumask(mm), ipi_func, ...);
is potentially buggy because the variable update is not ordered with
respect to the mm_cpumask() read. There were at least two examples of
this pattern on x86 that we think were real bugs. See commit
0dc2a4dac3d0 ("x86/mm: Ensure write visibility during LDT
installation") for one of them.
Add a barrier to the core IPI code to make this pattern more foolproof.
Other arch maintainers: You might want to take a look over your IPIs
and see if this pattern is biting you anywhere. Also, for those of
you that *do* have correct barriers in place for this pattern, they
may become redundant if this gets applied.
This specific idea came from Linus (and his LLM), with some tweaking
on top from me. I'm planning to send it up in the next merge window
via an x86 tree, barring any objections or performance issues that
surface. Please speak up if this causes you any heartburn.
Longer discussion follows.
== Background ==
There's a basic pattern in the kernel where some in-memory structure
is changed and then remote CPUs are notified via an IPI about the
change. TLB flushes are the canonical example but there are plenty
of others.
The IPI is generally sent to the CPUs using an mm found via
mm_cpumask(). CPUs set themselves in that mask when they context
switch to an mm.
So, there are two sides to this pattern. Both write some data and the
other side consumes it. Bugs happen if, for instance, the IPI-sending
CPU uses a stale mm_cpumask(). Or, if the IPI-receiving CPU happened
to read the data before setting its bit in mm_cpumask().
== Problem ==
There was a recent x86 bug where mm_cpumask() was read before the
newly-written data was visible to remote CPUs. This was the reverse of
program order, but allowed without a memory barrier.
But this case is not special. There are other examples on x86,
including one in the LAM code which is probably a real but largely
inconsequential bug today. There are a few other x86 cases which use
atomics (and have their own memory barriers), but they are probably
correct by accident more than design.
== Solution ==
Add a full smp_mb() in the core IPI code and documentation of the new
rules. Place the smp_mb() just before the cpumask is acted on. This
will ensure that any data written before the IPI is visible to remote
CPUs.
This makes the smp_call_function() code a wee bit more expensive, but
substantially harder to get wrong.
Combined with the barriers in mm switching, this ensures that remote
CPUs either get the IPI or can see the write from the IPI sender.
Place the barrier so it covers every entry point that takes a cpumask:
on_each_cpu_mask(), on_each_cpu_cond_mask() and
smp_call_function_many(). Callers that sample mm_cpumask() themselves
to pick targets must still order the update against their own mask
read.
Remove the now-redundant smp_mb() in x86's install_ldt().
== Impact ==
Many IPI users already have working barriers in place. The new barrier
may make some of those redundant.
== Alternatives ==
My first thought was to add a barrier in mm_cpumask() itself. But it
only computes a pointer to the cpumask. A barrier there would be odd
because there is no actual data consumed.
Andy Lutomirski suggested adding a barrier to on_each_cpu_mm(). That
would work for the x86 cases, but might leave issues in code that does
not deal with mm's.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: x86@kernel.org
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: linux-mm@kvack.org
Cc: linux-arch@vger.kernel.org
---
b/arch/x86/kernel/ldt.c | 12 ++++--------
b/arch/x86/mm/tlb.c | 5 +++++
b/include/linux/mm_types.h | 14 +++++++++++++-
b/include/linux/smp.h | 4 ++++
b/kernel/smp.c | 17 +++++++++++++++++
5 files changed, 43 insertions(+), 9 deletions(-)
diff -puN arch/x86/kernel/ldt.c~ipi-mb2-generic arch/x86/kernel/ldt.c
--- a/arch/x86/kernel/ldt.c~ipi-mb2-generic 2026-11-20 11:28:32.775743722 -0800
+++ b/arch/x86/kernel/ldt.c 2026-11-20 12:04:14.780113172 -0800
@@ -422,16 +422,12 @@ static void install_ldt(struct mm_struct
{
mutex_lock(&mm->context.lock);
- /*
- * Ensure that new mm user CPUs either see the new LDT value
- * or get an IPI. Make the store visible before reading
- * mm_cpumask(). New mm users make their mm_cpumask() writes
- * visible before reading mm->context.ldt.
- */
WRITE_ONCE(mm->context.ldt, ldt);
- smp_mb();
- /* Activate the LDT for all CPUs using currents mm. */
+ /*
+ * Activate the LDT for all mm-user CPUs. New mm users either
+ * see the new LDT or get the IPI, see mm_cpumask().
+ */
on_each_cpu_mask(mm_cpumask(mm), flush_ldt, mm, true);
mutex_unlock(&mm->context.lock);
diff -puN arch/x86/mm/tlb.c~ipi-mb2-generic arch/x86/mm/tlb.c
--- a/arch/x86/mm/tlb.c~ipi-mb2-generic 2026-11-20 11:28:32.777743889 -0800
+++ b/arch/x86/mm/tlb.c 2026-11-20 11:54:03.546906165 -0800
@@ -937,6 +937,11 @@ void switch_mm_irqs_off(struct mm_struct
*
* This way switch_mm() must see the new tlb_gen or
* flush_tlb_mm_range() must see the new loaded_mm, or both.
+ *
+ * This also protects other per-mm state read below, like the
+ * LDT. Changes to that state are followed by an IPI to the
+ * CPUs in mm_cpumask(). Setting the bit first means this CPU
+ * either reads the new value or gets the IPI.
*/
if (next != &init_mm && !cpumask_test_cpu(cpu, mm_cpumask(next)))
cpumask_set_cpu(cpu, mm_cpumask(next));
diff -puN include/linux/mm_types.h~ipi-mb2-generic include/linux/mm_types.h
--- a/include/linux/mm_types.h~ipi-mb2-generic 2026-11-20 11:28:32.779744056 -0800
+++ b/include/linux/mm_types.h 2026-11-20 11:28:32.785744556 -0800
@@ -1514,7 +1514,19 @@ static inline void mm_init_cpumask(struc
cpumask_clear((struct cpumask *)cpu_bitmap);
}
-/* Future-safe accessor for struct mm_struct's cpu_vm_mask. */
+/*
+ * mm_cpumask() - CPUs that may hold state derived from @mm (TLB entries,
+ * cached per-mm context). Code that updates per-mm state and then notifies
+ * the CPUs in the mask relies on:
+ *
+ * - A CPU sets its bit, then a full barrier, then reads that state or
+ * loads @mm's page tables.
+ * - The updater has a full barrier between the update and reading the
+ * mask. on_each_cpu_mask() and smp_call_function_many() provide it;
+ * open-coded mask tests (e.g. "only this CPU?") must do it themselves.
+ * - An architecture that clears bits must revalidate that state when the
+ * CPU sets its bit again.
+ */
static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
{
return (struct cpumask *)&mm->flexible_array;
diff -puN kernel/smp.c~ipi-mb2-generic kernel/smp.c
--- a/kernel/smp.c~ipi-mb2-generic 2026-11-20 11:28:32.780744139 -0800
+++ b/kernel/smp.c 2026-11-20 11:48:27.863451170 -0800
@@ -950,6 +950,15 @@ static void smp_call_function_many_cond(
*/
WARN_ON_ONCE(!in_task());
+ /*
+ * Callers update some state and then IPI the CPUs in @mask to act on
+ * it. A CPU that sets its bit in @mask and then reads that state
+ * must either see the update or get the IPI, so order the caller's
+ * prior updates before reading @mask. Pairs with a full barrier
+ * between setting the bit and reading the state, e.g. in switch_mm().
+ */
+ smp_mb();
+
/* Check if we need remote execution, i.e., any CPU excluding this one. */
if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
cpumask_and(cpumask, mask, cpu_online_mask);
@@ -1043,6 +1052,10 @@ static void smp_call_function_many_cond(
*
* @func is not called on the local CPU even if @mask contains it. Consider
* using on_each_cpu_cond_mask() instead if this is not desirable.
+ *
+ * Stores made by the caller before this call are ordered before the
+ * read of @mask, so a CPU that sets its bit in @mask and then reads
+ * that data will either see the stores or receive the IPI.
*/
void smp_call_function_many(const struct cpumask *mask,
smp_call_func_t func, void *info, bool wait)
@@ -1181,6 +1194,10 @@ void __init smp_init(void)
* to be seen or sent an IPI. If @wait is true, the final wait for remote
* completion happens after that preemption-disabled section.
*
+ * Stores made by the caller before this call are ordered before the
+ * read of @mask, so a CPU that sets its bit in @mask and then reads
+ * that data will either see the stores or receive the IPI.
+ *
* You must not call this function with disabled interrupts or
* from a hardware interrupt handler or from a bottom half handler.
*/
diff -puN include/linux/smp.h~ipi-mb2-generic include/linux/smp.h
--- a/include/linux/smp.h~ipi-mb2-generic 2026-11-20 11:48:08.343859262 -0800
+++ b/include/linux/smp.h 2026-11-20 11:48:27.863451170 -0800
@@ -82,6 +82,10 @@ static inline void on_each_cpu(smp_call_
*
* If @wait is true, then returns once @func has returned.
*
+ * Stores made by the caller before this call are ordered before the
+ * read of @mask, so a CPU that sets its bit in @mask and then reads
+ * that data will either see the stores or receive the IPI.
+ *
* You must not call this function with disabled interrupts or from a
* hardware interrupt handler or from a bottom half handler. The
* exception is that it may be used during early boot while
_
reply other threads:[~2026-11-20 22:47 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120224758.04F70CD8@davehans-spike.ostc.intel.com \
--to=dave.hansen@linux.intel.com \
--cc=bp@alien8.de \
--cc=david@kernel.org \
--cc=hpa@zytor.com \
--cc=liam@infradead.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=luto@kernel.org \
--cc=mhocko@suse.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=riel@surriel.com \
--cc=rppt@kernel.org \
--cc=surenb@google.com \
--cc=tglx@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=vbabka@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®