[PATCH 00/11] Allow preemption during IPI completion waiting to improve real-time performance

[PATCH 00/11] Allow preemption during IPI completion waiting to improve real-time performance

[Chuyi Zhou]

Introduction
============

The vast majority of smp_call_function*() callers block until remote CPUs
complete the IPI function execution. As smp_call_function*() runs with
preemption disabled throughout, scheduling latency increases dramatically
with the number of remote CPUs and other factors (such as interrupts being
disabled).

On x86-64 architectures, TLB flushes are performed via IPIs; thus, during
process exit or when process-mapped pages are reclaimed, numerous IPI
operations must be awaited, leading to increased scheduling latency for
other threads on the current CPU. In our production environment, we
observed IPI wait-induced scheduling latency reaching up to 16ms on a
16-core machine. Our goal is to allow preemption during IPI completion
waiting to improve real-time performance.

Background
============

In our production environments, latency-sensitive workloads (DPDK) are
configured with the highest priority to preempt lower-priority tasks at any
time. We discovered that DPDK's wake-up latency is primarily caused by the
current CPU having preemption disabled. Therefore, we collected the maximum
preemption disabled events within every 30-second interval and then
calculated the P50/P99 of these max preemption disabled events:

                        p50(ns)               p99(ns)
cpu0                   254956                 5465050
cpu1                   115801                 120782
cpu2                   43324                  72957
cpu3                   256637                 16723307
cpu4                   58979                  87237
cpu5                   47464                  79815
cpu6                   48881                  81371
cpu7                   52263                  82294
cpu8                   263555                 4657713
cpu9                   44935                  73962
cpu10                  37659                  65026
cpu11                  257008                 2706878
cpu12                  49669                  90006
cpu13                  45186                  74666
cpu14                  60705                  83866
cpu15                  51311                  86885

Meanwhile, we have collected the distribution of preemption disabling
events exceeding 1ms across different CPUs over several hours(I omitted
CPU data that were all zeros):

CPU        1~10ms   10~50ms   50~100ms
cpu0        29       5       0
cpu3        38       13      0
cpu8        34       6       0
cpu11       24       10      0

The preemption disabled for several milliseconds or even 10ms+ mostly
originates from TLB flush:

@stack[
    trace_preempt_on+143
    trace_preempt_on+143
    preempt_count_sub+67
    arch_tlbbatch_flush/flush_tlb_mm_range
    task_exit/page_reclaim/...
]

Further analysis confirms that the majority of the time is consumed in
csd_lock_wait().

Now smp_call*() always needs to disable preemption, mainly to protect its
internal per‑CPU data structures and synchronize with CPU offline
operations. This patchset attempts to make csd_lock_wait() preemptible,
thereby reducing the preemption‑disabled critical section and improving
kernel real‑time performance.

Effect
======

After applying this patchset, we no longer observe preemption disabled for
more than 1ms on the arch_tlbbatch_flush/flush_tlb_mm_range path. The
overall P99 of max preemption disabled events in every 30-second is
reduced to around 1.5ms (the remaining latency is primarily due to lock
contention.

                     before patch    after patch    reduced by
                     -----------    --------------  ------------
p99(ns)                16723307        1556034        ~90.70%

Chuyi Zhou (11):
  smp: Disable preemption explicitly in __csd_lock_wait
  smp: Enable preemption early in smp_call_function_single
  smp: Remove get_cpu from smp_call_function_any
  smp: Use on-stack cpumask in smp_call_function_many_cond
  smp: Enable preemption early in smp_call_function_many_cond
  smp: Remove preempt_disable from smp_call_function
  smp: Remove preempt_disable from on_each_cpu_cond_mask
  scftorture: Remove preempt_disable in scftorture_invoke_one
  x86/mm: Move flush_tlb_info back to the stack
  x86/mm: Enable preemption during native_flush_tlb_multi
  x86/mm: Enable preemption during flush_tlb_kernel_range

 arch/x86/hyperv/mmu.c |   2 +
 arch/x86/kernel/kvm.c |   4 +-
 arch/x86/mm/tlb.c     | 135 ++++++++++++++++++------------------------
 arch/x86/xen/mmu_pv.c |   1 +
 kernel/scftorture.c   |   9 +--
 kernel/smp.c          |  88 ++++++++++++++++++++++-----
 6 files changed, 136 insertions(+), 103 deletions(-)

-- 
2.20.1

[PATCH 01/11] smp: Disable preemption explicitly in __csd_lock_wait

[Chuyi Zhou]

The latter patches will enable preemption before csd_lock_wait(), which
could break csdlock_debug. Because the slice of other tasks on the CPU may
be accounted between ktime_get_mono_fast_ns() calls. Disable preemption
explicitly in __csd_lock_wait(). This is a preparation for the next
patches.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/smp.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/smp.c b/kernel/smp.c
index f349960f79ca..fc1f7a964616 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -323,6 +323,8 @@ static void __csd_lock_wait(call_single_data_t *csd)
 	int bug_id = 0;
 	u64 ts0, ts1;
 
+	guard(preempt)();
+
 	ts1 = ts0 = ktime_get_mono_fast_ns();
 	for (;;) {
 		if (csd_lock_wait_toolong(csd, ts0, &ts1, &bug_id, &nmessages))
-- 
2.20.1

[PATCH 02/11] smp: Enable preemption early in smp_call_function_single

[Chuyi Zhou]

Now smp_call_function_single() disables preemption mainly for the following
reasons:

- To protect the per-cpu csd_data from concurrent modification by other
tasks on the current CPU in the !wait case. For the wait case,
synchronization is not a concern as on-stack csd is used.

- To prevent the remote online CPU from being offlined. Specifically, we
want to ensure that no new IPIs are queued after smpcfd_dying_cpu() has
finished.

Disabling preemption for the entire execution is unnecessary, especially
csd_lock_wait() part does not require preemption protection. This patch
enables preemption before csd_lock_wait() to reduce the preemption-disabled
critical section.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/smp.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index fc1f7a964616..0858553f3666 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -685,11 +685,24 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
 
 	err = generic_exec_single(cpu, csd);
 
+	/*
+	 * We may block in csd_lock_wait() for a significant amount of time (e.g., if the
+	 * remote CPU has interrupts disabled). Disabling preemption throughout the entire
+	 * smp_call_function_single() impacts the scheduling latency and is unnecessary.
+	 *
+	 * - Preemption must be disabled before sending the IPI to ensure no new IPIs are
+	 * queued after smpcfd_dying_cpu() finishes.
+	 *
+	 * @csd is stack-allocated when @wait is true. No concurrent access except
+	 * from the IPI completion path, so we can re-enable preemption early
+	 * to reduce latency.
+	 *
+	 */
+	put_cpu();
+
 	if (wait)
 		csd_lock_wait(csd);
 
-	put_cpu();
-
 	return err;
 }
 EXPORT_SYMBOL(smp_call_function_single);
-- 
2.20.1

[PATCH 03/11] smp: Remove get_cpu from smp_call_function_any

[Chuyi Zhou]

Now smp_call_function_single() would enable preemption before
csd_lock_wait() to reduce the critical section. To allow callers of
smp_call_function_any() to also benefit from this optimization, remove
get_cpu()/put_cpu() from smp_call_function_any().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/smp.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index 0858553f3666..f572716c3c7d 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -772,13 +772,18 @@ int smp_call_function_any(const struct cpumask *mask,
 	unsigned int cpu;
 	int ret;
 
+	/*
+	 * Prevent migration to another CPU after selecting the current CPU
+	 * as the target.
+	 */
+	guard(migrate)();
+
 	/* Try for same CPU (cheapest) */
-	cpu = get_cpu();
+	cpu = smp_processor_id();
 	if (!cpumask_test_cpu(cpu, mask))
 		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu));
 
 	ret = smp_call_function_single(cpu, func, info, wait);
-	put_cpu();
 	return ret;
 }
 EXPORT_SYMBOL_GPL(smp_call_function_any);
-- 
2.20.1

[PATCH 04/11] smp: Use on-stack cpumask in smp_call_function_many_cond

[Chuyi Zhou]

This patch use on-stack cpumask to replace percpu cfd cpumask in
smp_call_function_many_cond(). alloc_cpumask_var() may fail when
CONFIG_CPUMASK_OFFSTACK is enabled. In such extreme case, fall back to
cfd->cpumask. This is a preparation for the next patch.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/smp.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index f572716c3c7d..35948afced2e 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -805,11 +805,17 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 	int cpu, last_cpu, this_cpu = smp_processor_id();
 	struct call_function_data *cfd;
 	bool wait = scf_flags & SCF_WAIT;
+	bool preemptible_wait = true;
+	cpumask_var_t cpumask_stack;
+	struct cpumask *cpumask;
 	int nr_cpus = 0;
 	bool run_remote = false;
 
 	lockdep_assert_preemption_disabled();
 
+	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
+		preemptible_wait = false;
+
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -831,15 +837,18 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 	/* Check if we need remote execution, i.e., any CPU excluding this one. */
 	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
 		cfd = this_cpu_ptr(&cfd_data);
-		cpumask_and(cfd->cpumask, mask, cpu_online_mask);
-		__cpumask_clear_cpu(this_cpu, cfd->cpumask);
+
+		cpumask = preemptible_wait ? cpumask_stack : cfd->cpumask;
+
+		cpumask_and(cpumask, mask, cpu_online_mask);
+		__cpumask_clear_cpu(this_cpu, cpumask);
 
 		cpumask_clear(cfd->cpumask_ipi);
-		for_each_cpu(cpu, cfd->cpumask) {
+		for_each_cpu(cpu, cpumask) {
 			call_single_data_t *csd = per_cpu_ptr(cfd->csd, cpu);
 
 			if (cond_func && !cond_func(cpu, info)) {
-				__cpumask_clear_cpu(cpu, cfd->cpumask);
+				__cpumask_clear_cpu(cpu, cpumask);
 				continue;
 			}
 
@@ -890,13 +899,16 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 	}
 
 	if (run_remote && wait) {
-		for_each_cpu(cpu, cfd->cpumask) {
+		for_each_cpu(cpu, cpumask) {
 			call_single_data_t *csd;
 
 			csd = per_cpu_ptr(cfd->csd, cpu);
 			csd_lock_wait(csd);
 		}
 	}
+
+	if (preemptible_wait)
+		free_cpumask_var(cpumask_stack);
 }
 
 /**
-- 
2.20.1

[PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Chuyi Zhou]

Now smp_call_function_many_cond() disables preemption mainly for the
following reasons:

- To prevent the remote online CPU from going offline. Specifically, we
want to ensure that no new csds are queued after smpcfd_dying_cpu() has
finished. Therefore, preemption must be disabled until all necessary IPIs
are sent.

- To prevent migration to another CPU, which also implicitly prevents the
current CPU from going offline (since stop_machine requires preempting the
current task to execute offline callbacks). This can be achieved equally
using migrate_disable(), as tasks must be migrated to other CPUs before
takedown_cpu().

- To protect the per-cpu cfd_data from concurrent modification by other
smp_call_*() on the current CPU. cfd_data contains cpumasks and per-cpu
csds. Before enqueueing a csd, we block on the csd_lock() to ensure the
previous asyc csd->func() has completed, and then initialize csd->func and
csd->info. After sending the IPI, we spin-wait for the remote CPU to call
csd_unlock(). Actually the csd_lock mechanism already guarantees csd
serialization. If preemption occurs during csd_lock_wait, other concurrent
smp_call_function_many_cond calls will simply block until the previous
csd->func() completes:

task A                    task B

sd->func = fun_a
send ipis

                preempted by B
               --------------->
                        csd_lock(csd); // block until last
                                       // fun_a finished

                        csd->func = func_b;
                        csd->info = info;
                            ...
                        send ipis

                switch back to A
                <---------------

csd_lock_wait(csd); // block until remote finish func_*

This patch use migrate_disable() to protect the scope of
smp_call_function_many_cond() and enables preemption before csd_lock_wait.
This makes the potentially unpredictable csd_lock_wait preemptible. Using
cpumask_stack can avoid concurrency modification issues, and we can
fall back to the default logic if alloc_cpumask_var() fails.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/smp.c | 37 ++++++++++++++++++++++++++++++++-----
 1 file changed, 32 insertions(+), 5 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index 35948afced2e..af9cee7d4939 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -802,7 +802,7 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 					unsigned int scf_flags,
 					smp_cond_func_t cond_func)
 {
-	int cpu, last_cpu, this_cpu = smp_processor_id();
+	int cpu, last_cpu, this_cpu;
 	struct call_function_data *cfd;
 	bool wait = scf_flags & SCF_WAIT;
 	bool preemptible_wait = true;
@@ -811,11 +811,18 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 	int nr_cpus = 0;
 	bool run_remote = false;
 
-	lockdep_assert_preemption_disabled();
-
-	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
+	if (!wait || !alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
 		preemptible_wait = false;
 
+	/*
+	 * Prevent the current CPU from going offline.
+	 * Being migrated to another CPU and calling csd_lock_wait() may cause
+	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
+	 */
+	migrate_disable();
+
+	this_cpu = get_cpu();
+
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -898,6 +905,22 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 		local_irq_restore(flags);
 	}
 
+	/*
+	 * We may block in csd_lock_wait() for a significant amount of time, especially
+	 * when interrupts are disabled or with a large number of remote CPUs.
+	 * Try to enable preemption before csd_lock_wait().
+	 *
+	 * - If @wait is true, we try to use the cpumask_stack instead of cfd->cpumask to
+	 * avoid concurrency modification from tasks on the same cpu. If alloc_cpumask_var()
+	 * return false, fallback to the default logic.
+	 *
+	 * - If preemption occurs during csd_lock_wait, other concurrent
+	 * smp_call_function_many_cond() calls will simply block until the previous csd->func()
+	 * complete.
+	 */
+	if (preemptible_wait)
+		put_cpu();
+
 	if (run_remote && wait) {
 		for_each_cpu(cpu, cpumask) {
 			call_single_data_t *csd;
@@ -907,8 +930,12 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
 		}
 	}
 
-	if (preemptible_wait)
+	if (!preemptible_wait)
+		put_cpu();
+	else
 		free_cpumask_var(cpumask_stack);
+
+	migrate_enable();
 }
 
 /**
-- 
2.20.1

[PATCH 06/11] smp: Remove preempt_disable from smp_call_function

[Chuyi Zhou]

Now smp_call_function_many_cond() internally handles the preemption logic,
so smp_call_function() does not need to explicitly disable preemption.
Remove preempt_{enable, disable} from smp_call_function().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/smp.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index af9cee7d4939..088b581003fb 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -977,9 +977,8 @@ EXPORT_SYMBOL(smp_call_function_many);
  */
 void smp_call_function(smp_call_func_t func, void *info, int wait)
 {
-	preempt_disable();
-	smp_call_function_many(cpu_online_mask, func, info, wait);
-	preempt_enable();
+	smp_call_function_many_cond(cpu_online_mask, func, info,
+			wait ? SCF_WAIT : 0, NULL);
 }
 EXPORT_SYMBOL(smp_call_function);
 
-- 
2.20.1

[PATCH 07/11] smp: Remove preempt_disable from on_each_cpu_cond_mask

[Chuyi Zhou]

Now smp_call_function_many_cond() internally handles the preemption logic,
so on_each_cpu_cond_mask does not need to explicitly disable preemption.
Remove preempt_{enable, disable} from on_each_cpu_cond_mask().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/smp.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/kernel/smp.c b/kernel/smp.c
index 088b581003fb..c859076239c4 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -1097,9 +1097,7 @@ void on_each_cpu_cond_mask(smp_cond_func_t cond_func, smp_call_func_t func,
 	if (wait)
 		scf_flags |= SCF_WAIT;
 
-	preempt_disable();
 	smp_call_function_many_cond(mask, func, info, scf_flags, cond_func);
-	preempt_enable();
 }
 EXPORT_SYMBOL(on_each_cpu_cond_mask);
 
-- 
2.20.1

[PATCH 08/11] scftorture: Remove preempt_disable in scftorture_invoke_one

[Chuyi Zhou]

Now we no longer need explicit preempt_disable calls before smp_call_*(),
because the smp_call*() internally handle preemption logic themselves.
Remove preempt_{enable, disable} in scftorture_invoke_one.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 kernel/scftorture.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/kernel/scftorture.c b/kernel/scftorture.c
index d86d2d9c4624..3fb1742f3129 100644
--- a/kernel/scftorture.c
+++ b/kernel/scftorture.c
@@ -364,8 +364,6 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 	}
 	if (use_cpus_read_lock)
 		cpus_read_lock();
-	else
-		preempt_disable();
 	switch (scfsp->scfs_prim) {
 	case SCF_PRIM_RESCHED:
 		if (IS_BUILTIN(CONFIG_SCF_TORTURE_TEST)) {
@@ -411,13 +409,10 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 		if (!ret) {
 			if (use_cpus_read_lock)
 				cpus_read_unlock();
-			else
-				preempt_enable();
+
 			wait_for_completion(&scfcp->scfc_completion);
 			if (use_cpus_read_lock)
 				cpus_read_lock();
-			else
-				preempt_disable();
 		} else {
 			scfp->n_single_rpc_ofl++;
 			scf_add_to_free_list(scfcp);
@@ -463,8 +458,6 @@ static void scftorture_invoke_one(struct scf_statistics *scfp, struct torture_ra
 	}
 	if (use_cpus_read_lock)
 		cpus_read_unlock();
-	else
-		preempt_enable();
 	if (allocfail)
 		schedule_timeout_idle((1 + longwait) * HZ);  // Let no-wait handlers complete.
 	else if (!(torture_random(trsp) & 0xfff))
-- 
2.20.1

[PATCH 09/11] x86/mm: Move flush_tlb_info back to the stack

[Chuyi Zhou]

Commit 3db6d5a5ecaf ("x86/mm/tlb: Remove 'struct flush_tlb_info' from the
stack") changed flush_tlb_info from stack variables to per-CPU variables.
This brought about a performance improvement of around 3% in extreme test.
However, it also required that all flush_tlb* operations keep preemption
disabled entirely to prevent concurrent modifications of flush_tlb_info.
flush_tlb* needs to send IPIs to remote CPUs and synchronously wait for all
remote CPUs to complete their local TLB flushes. The process could take
tens of milliseconds when interrupts are disabled or with a large number of
remote CPUs.

From the perspective of improving kernel real-time performance, this patch
reverts flush_tlb_info back to stack variables. This is a preparation for
enabling preemption during TLB flush in next patch.

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 arch/x86/mm/tlb.c | 124 ++++++++++++++++++----------------------------
 1 file changed, 49 insertions(+), 75 deletions(-)

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index f5b93e01e347..2d68297ed35b 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1394,71 +1394,30 @@ void flush_tlb_multi(const struct cpumask *cpumask,
  */
 unsigned long tlb_single_page_flush_ceiling __read_mostly = 33;
 
-static DEFINE_PER_CPU_SHARED_ALIGNED(struct flush_tlb_info, flush_tlb_info);
-
-#ifdef CONFIG_DEBUG_VM
-static DEFINE_PER_CPU(unsigned int, flush_tlb_info_idx);
-#endif
-
-static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm,
-			unsigned long start, unsigned long end,
-			unsigned int stride_shift, bool freed_tables,
-			u64 new_tlb_gen)
+void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
+				unsigned long end, unsigned int stride_shift,
+				bool freed_tables)
 {
-	struct flush_tlb_info *info = this_cpu_ptr(&flush_tlb_info);
+	int cpu = get_cpu();
 
-#ifdef CONFIG_DEBUG_VM
-	/*
-	 * Ensure that the following code is non-reentrant and flush_tlb_info
-	 * is not overwritten. This means no TLB flushing is initiated by
-	 * interrupt handlers and machine-check exception handlers.
-	 */
-	BUG_ON(this_cpu_inc_return(flush_tlb_info_idx) != 1);
-#endif
+	struct flush_tlb_info info = {
+		.mm = mm,
+		.stride_shift = stride_shift,
+		.freed_tables = freed_tables,
+		.trim_cpumask = 0,
+		.initiating_cpu = cpu
+	};
 
-	/*
-	 * If the number of flushes is so large that a full flush
-	 * would be faster, do a full flush.
-	 */
 	if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) {
 		start = 0;
 		end = TLB_FLUSH_ALL;
 	}
 
-	info->start		= start;
-	info->end		= end;
-	info->mm		= mm;
-	info->stride_shift	= stride_shift;
-	info->freed_tables	= freed_tables;
-	info->new_tlb_gen	= new_tlb_gen;
-	info->initiating_cpu	= smp_processor_id();
-	info->trim_cpumask	= 0;
-
-	return info;
-}
-
-static void put_flush_tlb_info(void)
-{
-#ifdef CONFIG_DEBUG_VM
-	/* Complete reentrancy prevention checks */
-	barrier();
-	this_cpu_dec(flush_tlb_info_idx);
-#endif
-}
-
-void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
-				unsigned long end, unsigned int stride_shift,
-				bool freed_tables)
-{
-	struct flush_tlb_info *info;
-	int cpu = get_cpu();
-	u64 new_tlb_gen;
-
 	/* This is also a barrier that synchronizes with switch_mm(). */
-	new_tlb_gen = inc_mm_tlb_gen(mm);
+	info.new_tlb_gen = inc_mm_tlb_gen(mm);
 
-	info = get_flush_tlb_info(mm, start, end, stride_shift, freed_tables,
-				  new_tlb_gen);
+	info.start = start;
+	info.end = end;
 
 	/*
 	 * flush_tlb_multi() is not optimized for the common case in which only
@@ -1466,19 +1425,18 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 	 * flush_tlb_func_local() directly in this case.
 	 */
 	if (mm_global_asid(mm)) {
-		broadcast_tlb_flush(info);
+		broadcast_tlb_flush(&info);
 	} else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) {
-		info->trim_cpumask = should_trim_cpumask(mm);
-		flush_tlb_multi(mm_cpumask(mm), info);
+		info.trim_cpumask = should_trim_cpumask(mm);
+		flush_tlb_multi(mm_cpumask(mm), &info);
 		consider_global_asid(mm);
 	} else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
 		lockdep_assert_irqs_enabled();
 		local_irq_disable();
-		flush_tlb_func(info);
+		flush_tlb_func(&info);
 		local_irq_enable();
 	}
 
-	put_flush_tlb_info();
 	put_cpu();
 	mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end);
 }
@@ -1548,19 +1506,29 @@ static void kernel_tlb_flush_range(struct flush_tlb_info *info)
 
 void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
-	struct flush_tlb_info *info;
+	struct flush_tlb_info info = {
+		.mm = NULL,
+		.stride_shift = PAGE_SHIFT,
+		.freed_tables = false,
+		.trim_cpumask = 0,
+		.new_tlb_gen = TLB_GENERATION_INVALID
+	};
 
 	guard(preempt)();
 
-	info = get_flush_tlb_info(NULL, start, end, PAGE_SHIFT, false,
-				  TLB_GENERATION_INVALID);
+	if ((end - start) >> PAGE_SHIFT > tlb_single_page_flush_ceiling) {
+		start = 0;
+		end = TLB_FLUSH_ALL;
+	}
 
-	if (info->end == TLB_FLUSH_ALL)
-		kernel_tlb_flush_all(info);
-	else
-		kernel_tlb_flush_range(info);
+	info.initiating_cpu = smp_processor_id(),
+	info.start = start;
+	info.end = end;
 
-	put_flush_tlb_info();
+	if (info.end == TLB_FLUSH_ALL)
+		kernel_tlb_flush_all(&info);
+	else
+		kernel_tlb_flush_range(&info);
 }
 
 /*
@@ -1728,12 +1696,19 @@ EXPORT_SYMBOL_FOR_KVM(__flush_tlb_all);
 
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 {
-	struct flush_tlb_info *info;
-
 	int cpu = get_cpu();
 
-	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false,
-				  TLB_GENERATION_INVALID);
+	struct flush_tlb_info info = {
+		.start = 0,
+		.end = TLB_FLUSH_ALL,
+		.mm = NULL,
+		.stride_shift = 0,
+		.freed_tables = false,
+		.new_tlb_gen = TLB_GENERATION_INVALID,
+		.initiating_cpu = cpu,
+		.trim_cpumask = 0,
+	};
+
 	/*
 	 * flush_tlb_multi() is not optimized for the common case in which only
 	 * a local TLB flush is needed. Optimize this use-case by calling
@@ -1743,17 +1718,16 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 		invlpgb_flush_all_nonglobals();
 		batch->unmapped_pages = false;
 	} else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) {
-		flush_tlb_multi(&batch->cpumask, info);
+		flush_tlb_multi(&batch->cpumask, &info);
 	} else if (cpumask_test_cpu(cpu, &batch->cpumask)) {
 		lockdep_assert_irqs_enabled();
 		local_irq_disable();
-		flush_tlb_func(info);
+		flush_tlb_func(&info);
 		local_irq_enable();
 	}
 
 	cpumask_clear(&batch->cpumask);
 
-	put_flush_tlb_info();
 	put_cpu();
 }
 
-- 
2.20.1

[PATCH 10/11] x86/mm: Enable preemption during native_flush_tlb_multi

[Chuyi Zhou]

flush_tlb_mm_range()/arch_tlbbatch_flush() -> native_flush_tlb_multi() is a
common triggering path in real production environments. When pages are
reclaimed or process exit, native_flush_tlb_multi() sends IPIs to remote
CPUs and waits for all remote CPUs to complete their local TLB flushes. The
overall latency may reach tens of milliseconds due to a large number of
remote CPUs and other factors (such as interrupts being disabled). Since
flush_tlb_mm_range()/arch_tlbbatch_flush() always disable preemption, which
may cause increased scheduling latency for other threads on the current
CPU.

Previous patche convert flush_tlb_info from per-cpu variable to on-stack
variable. Additionally, it's no longer necessary to explicitly disable
preemption before calling smp_call*() since they internally handles the
preemption logic. Now is's safe to enable preemption during
native_flush_tlb_multi().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 arch/x86/hyperv/mmu.c |  2 ++
 arch/x86/kernel/kvm.c |  4 +++-
 arch/x86/mm/tlb.c     | 23 +++++++++++++----------
 arch/x86/xen/mmu_pv.c |  1 +
 4 files changed, 19 insertions(+), 11 deletions(-)

diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c
index cfcb60468b01..394f849af10a 100644
--- a/arch/x86/hyperv/mmu.c
+++ b/arch/x86/hyperv/mmu.c
@@ -65,6 +65,8 @@ static void hyperv_flush_tlb_multi(const struct cpumask *cpus,
 	unsigned long flags;
 	bool do_lazy = !info->freed_tables;
 
+	guard(preempt)();
+
 	trace_hyperv_mmu_flush_tlb_multi(cpus, info);
 
 	if (!hv_hypercall_pg)
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index df78ddee0abb..6b56dab28e66 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -654,8 +654,10 @@ static void kvm_flush_tlb_multi(const struct cpumask *cpumask,
 	u8 state;
 	int cpu;
 	struct kvm_steal_time *src;
-	struct cpumask *flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask);
+	struct cpumask *flushmask;
 
+	guard(preempt)();
+	flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask);
 	cpumask_copy(flushmask, cpumask);
 	/*
 	 * We have to call flush only on online vCPUs. And
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 2d68297ed35b..4162d7ff024f 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1398,21 +1398,23 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 				unsigned long end, unsigned int stride_shift,
 				bool freed_tables)
 {
-	int cpu = get_cpu();
-
 	struct flush_tlb_info info = {
 		.mm = mm,
 		.stride_shift = stride_shift,
 		.freed_tables = freed_tables,
-		.trim_cpumask = 0,
-		.initiating_cpu = cpu
+		.trim_cpumask = 0
 	};
+	int cpu;
 
 	if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) {
 		start = 0;
 		end = TLB_FLUSH_ALL;
 	}
 
+	migrate_disable();
+
+	cpu = info.initiating_cpu = smp_processor_id();
+
 	/* This is also a barrier that synchronizes with switch_mm(). */
 	info.new_tlb_gen = inc_mm_tlb_gen(mm);
 
@@ -1425,6 +1427,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 	 * flush_tlb_func_local() directly in this case.
 	 */
 	if (mm_global_asid(mm)) {
+		guard(preempt)();
 		broadcast_tlb_flush(&info);
 	} else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) {
 		info.trim_cpumask = should_trim_cpumask(mm);
@@ -1437,7 +1440,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 		local_irq_enable();
 	}
 
-	put_cpu();
+	migrate_enable();
 	mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end);
 }
 
@@ -1696,8 +1699,6 @@ EXPORT_SYMBOL_FOR_KVM(__flush_tlb_all);
 
 void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 {
-	int cpu = get_cpu();
-
 	struct flush_tlb_info info = {
 		.start = 0,
 		.end = TLB_FLUSH_ALL,
@@ -1705,9 +1706,13 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 		.stride_shift = 0,
 		.freed_tables = false,
 		.new_tlb_gen = TLB_GENERATION_INVALID,
-		.initiating_cpu = cpu,
 		.trim_cpumask = 0,
 	};
+	int cpu;
+
+	guard(migrate)();
+
+	info.initiating_cpu = cpu = smp_processor_id();
 
 	/*
 	 * flush_tlb_multi() is not optimized for the common case in which only
@@ -1727,8 +1732,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 	}
 
 	cpumask_clear(&batch->cpumask);
-
-	put_cpu();
 }
 
 /*
diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c
index 2a4a8deaf612..b801721050f7 100644
--- a/arch/x86/xen/mmu_pv.c
+++ b/arch/x86/xen/mmu_pv.c
@@ -1330,6 +1330,7 @@ static void xen_flush_tlb_multi(const struct cpumask *cpus,
 	const size_t mc_entry_size = sizeof(args->op) +
 		sizeof(args->mask[0]) * BITS_TO_LONGS(num_possible_cpus());
 
+	guard(preempt)();
 	trace_xen_mmu_flush_tlb_multi(cpus, info->mm, info->start, info->end);
 
 	if (cpumask_empty(cpus))
-- 
2.20.1

[PATCH 11/11] x86/mm: Enable preemption during flush_tlb_kernel_range

[Chuyi Zhou]

flush_tlb_kernel_range() is invoked when kernel memory mapping changes.
On x86 platforms without the INVLPGB feature enabled, we need to send IPIs
to every online CPU and synchronously wait for them to complete
do_kernel_range_flush(). This process can be time-consuming due to factors
such as a large number of CPUs or other issues (like interrupts being
disabled). flush_tlb_kernel_range() always disables preemption, this may
affect the scheduling latency of other tasks on the current CPU.

Previous patch convert flush_tlb_info from per-cpu variable to on-stack
variable. Additionally, it's no longer necessary to explicitly disable
preemption before calling smp_call*() since they internally handles the
preemption logic. Now is's safe to enable preemption during
flush_tlb_kernel_range().

Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
 arch/x86/mm/tlb.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 4162d7ff024f..f0de6c1e387f 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1467,6 +1467,8 @@ static void invlpgb_kernel_range_flush(struct flush_tlb_info *info)
 {
 	unsigned long addr, nr;
 
+	guard(preempt)();
+
 	for (addr = info->start; addr < info->end; addr += nr << PAGE_SHIFT) {
 		nr = (info->end - addr) >> PAGE_SHIFT;
 
@@ -1517,7 +1519,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 		.new_tlb_gen = TLB_GENERATION_INVALID
 	};
 
-	guard(preempt)();
+	guard(migrate)();
 
 	if ((end - start) >> PAGE_SHIFT > tlb_single_page_flush_ceiling) {
 		start = 0;
-- 
2.20.1

Re: [PATCH 02/11] smp: Enable preemption early in smp_call_function_single

[Muchun Song]

> On Feb 3, 2026, at 19:23, Chuyi Zhou <zhouchuyi@bytedance.com> wrote:
> 
> Now smp_call_function_single() disables preemption mainly for the following
> reasons:
> 
> - To protect the per-cpu csd_data from concurrent modification by other
> tasks on the current CPU in the !wait case. For the wait case,
> synchronization is not a concern as on-stack csd is used.
> 
> - To prevent the remote online CPU from being offlined. Specifically, we
> want to ensure that no new IPIs are queued after smpcfd_dying_cpu() has
> finished.
> 
> Disabling preemption for the entire execution is unnecessary, especially
> csd_lock_wait() part does not require preemption protection. This patch
> enables preemption before csd_lock_wait() to reduce the preemption-disabled
> critical section.
> 
> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
> ---
> kernel/smp.c | 17 +++++++++++++++--
> 1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/smp.c b/kernel/smp.c
> index fc1f7a964616..0858553f3666 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -685,11 +685,24 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
> 
> 	err = generic_exec_single(cpu, csd);
> 
> + 	/*
> +	 * We may block in csd_lock_wait() for a significant amount of time (e.g., if the
> +	 * remote CPU has interrupts disabled). Disabling preemption throughout the entire
> +	 * smp_call_function_single() impacts the scheduling latency and is unnecessary.
> +	 *
> +	 * - Preemption must be disabled before sending the IPI to ensure no new IPIs are

            ^
That looks odd. Why use a dash ('-')?

> +	 * queued after smpcfd_dying_cpu() finishes.
> +	 *
> +	 * @csd is stack-allocated when @wait is true. No concurrent access except
> +	 * from the IPI completion path, so we can re-enable preemption early
> +	 * to reduce latency.
> +	 *

A blank line. Better to remove.

Reviewed-by: Muchun Song <muchun.song@linux.dev>

Thanks.

> +	 */
> + 	put_cpu();
> +
> 	if (wait)
> 		csd_lock_wait(csd);
> 
> - 	put_cpu();
> -
> 	return err;
> }
> EXPORT_SYMBOL(smp_call_function_single);
> -- 
> 2.20.1

Re: [PATCH 03/11] smp: Remove get_cpu from smp_call_function_any

[Muchun Song]

> On Feb 3, 2026, at 19:23, Chuyi Zhou <zhouchuyi@bytedance.com> wrote:
> 
> Now smp_call_function_single() would enable preemption before
> csd_lock_wait() to reduce the critical section. To allow callers of
> smp_call_function_any() to also benefit from this optimization, remove
> get_cpu()/put_cpu() from smp_call_function_any().
> 
> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>

Reviewed-by: Muchun Song <muchun.song@linux.dev>

Thanks.

Re: [PATCH 01/11] smp: Disable preemption explicitly in __csd_lock_wait

[Muchun Song]

> On Feb 3, 2026, at 19:23, Chuyi Zhou <zhouchuyi@bytedance.com> wrote:
> 
> The latter patches will enable preemption before csd_lock_wait(), which
> could break csdlock_debug. Because the slice of other tasks on the CPU may
> be accounted between ktime_get_mono_fast_ns() calls. Disable preemption
> explicitly in __csd_lock_wait(). This is a preparation for the next
> patches.
> 
> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>

Acked-by: Muchun Song <muchun.song@linux.dev>

Thanks.

Re: [PATCH 02/11] smp: Enable preemption early in smp_call_function_single

[Peter Zijlstra]

On Tue, Feb 03, 2026 at 07:23:52PM +0800, Chuyi Zhou wrote:
> Now smp_call_function_single() disables preemption mainly for the following
> reasons:
> 
> - To protect the per-cpu csd_data from concurrent modification by other
> tasks on the current CPU in the !wait case. For the wait case,
> synchronization is not a concern as on-stack csd is used.
> 
> - To prevent the remote online CPU from being offlined. Specifically, we
> want to ensure that no new IPIs are queued after smpcfd_dying_cpu() has
> finished.
> 
> Disabling preemption for the entire execution is unnecessary, especially
> csd_lock_wait() part does not require preemption protection. This patch
> enables preemption before csd_lock_wait() to reduce the preemption-disabled
> critical section.
> 
> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
> ---
>  kernel/smp.c | 17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/smp.c b/kernel/smp.c
> index fc1f7a964616..0858553f3666 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -685,11 +685,24 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
>  
>  	err = generic_exec_single(cpu, csd);
>  
> +	/*
> +	 * We may block in csd_lock_wait() for a significant amount of time (e.g., if the
> +	 * remote CPU has interrupts disabled). Disabling preemption throughout the entire
> +	 * smp_call_function_single() impacts the scheduling latency and is unnecessary.
> +	 *
> +	 * - Preemption must be disabled before sending the IPI to ensure no new IPIs are
> +	 * queued after smpcfd_dying_cpu() finishes.
> +	 *
> +	 * @csd is stack-allocated when @wait is true. No concurrent access except
> +	 * from the IPI completion path, so we can re-enable preemption early
> +	 * to reduce latency.
> +	 *
> +	 */

Those lines are too long for no reason, please reflow at 78 or so.

Re: [PATCH 03/11] smp: Remove get_cpu from smp_call_function_any

[Peter Zijlstra]

On Tue, Feb 03, 2026 at 07:23:53PM +0800, Chuyi Zhou wrote:
> Now smp_call_function_single() would enable preemption before
> csd_lock_wait() to reduce the critical section. To allow callers of
> smp_call_function_any() to also benefit from this optimization, remove
> get_cpu()/put_cpu() from smp_call_function_any().
> 
> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
> ---
>  kernel/smp.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/smp.c b/kernel/smp.c
> index 0858553f3666..f572716c3c7d 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -772,13 +772,18 @@ int smp_call_function_any(const struct cpumask *mask,
>  	unsigned int cpu;
>  	int ret;
>  
> +	/*
> +	 * Prevent migration to another CPU after selecting the current CPU
> +	 * as the target.
> +	 */
> +	guard(migrate)();
> +
>  	/* Try for same CPU (cheapest) */
> -	cpu = get_cpu();
> +	cpu = smp_processor_id();
>  	if (!cpumask_test_cpu(cpu, mask))
>  		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu));
>  
>  	ret = smp_call_function_single(cpu, func, info, wait);
> -	put_cpu();
>  	return ret;

Urgh, that's horrible.

Basically what you want is something like so:

	bool enable = true;
	unsigned int cpu;
	int ret;

	preempt_disable();
	cpu = smp_processor_id();
	if (!cpumask_test_cpu(cpu, mask)) {
		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu))
		enable = false;
		preempt_enable();
	}

	ret = smp_call_function_single(cpu, func, info, wait);
	if (enable)
		preempt_enable();
	return ret;

Re: [PATCH 04/11] smp: Use on-stack cpumask in smp_call_function_many_cond

[Peter Zijlstra]

On Tue, Feb 03, 2026 at 07:23:54PM +0800, Chuyi Zhou wrote:
> This patch use on-stack cpumask to replace percpu cfd cpumask in
> smp_call_function_many_cond(). alloc_cpumask_var() may fail when
> CONFIG_CPUMASK_OFFSTACK is enabled. In such extreme case, fall back to
> cfd->cpumask. This is a preparation for the next patch.
> 
> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
> ---
>  kernel/smp.c | 22 +++++++++++++++++-----
>  1 file changed, 17 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/smp.c b/kernel/smp.c
> index f572716c3c7d..35948afced2e 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -805,11 +805,17 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
>  	int cpu, last_cpu, this_cpu = smp_processor_id();
>  	struct call_function_data *cfd;
>  	bool wait = scf_flags & SCF_WAIT;
> +	bool preemptible_wait = true;
> +	cpumask_var_t cpumask_stack;
> +	struct cpumask *cpumask;
>  	int nr_cpus = 0;
>  	bool run_remote = false;
>  
>  	lockdep_assert_preemption_disabled();
>  
> +	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
> +		preemptible_wait = false;

IIRC this breaks RT, must not allocate with preemption disabled.

Re: [PATCH 04/11] smp: Use on-stack cpumask in smp_call_function_many_cond

[Peter Zijlstra]

On Tue, Feb 03, 2026 at 07:23:54PM +0800, Chuyi Zhou wrote:
> This patch use on-stack cpumask to replace percpu cfd cpumask in
> smp_call_function_many_cond(). alloc_cpumask_var() may fail when
> CONFIG_CPUMASK_OFFSTACK is enabled. In such extreme case, fall back to
> cfd->cpumask. This is a preparation for the next patch.
> 
> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
> ---
>  kernel/smp.c | 22 +++++++++++++++++-----
>  1 file changed, 17 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/smp.c b/kernel/smp.c
> index f572716c3c7d..35948afced2e 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -805,11 +805,17 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
>  	int cpu, last_cpu, this_cpu = smp_processor_id();
>  	struct call_function_data *cfd;
>  	bool wait = scf_flags & SCF_WAIT;
> +	bool preemptible_wait = true;
> +	cpumask_var_t cpumask_stack;
> +	struct cpumask *cpumask;
>  	int nr_cpus = 0;
>  	bool run_remote = false;
>  
>  	lockdep_assert_preemption_disabled();
>  
> +	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
> +		preemptible_wait = false;
> +
>  	/*
>  	 * Can deadlock when called with interrupts disabled.
>  	 * We allow cpu's that are not yet online though, as no one else can
> @@ -831,15 +837,18 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
>  	/* Check if we need remote execution, i.e., any CPU excluding this one. */
>  	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
>  		cfd = this_cpu_ptr(&cfd_data);
> -		cpumask_and(cfd->cpumask, mask, cpu_online_mask);
> -		__cpumask_clear_cpu(this_cpu, cfd->cpumask);
> +
> +		cpumask = preemptible_wait ? cpumask_stack : cfd->cpumask;
> +
> +		cpumask_and(cpumask, mask, cpu_online_mask);
> +		__cpumask_clear_cpu(this_cpu, cpumask);
>  
>  		cpumask_clear(cfd->cpumask_ipi);
> -		for_each_cpu(cpu, cfd->cpumask) {
> +		for_each_cpu(cpu, cpumask) {
>  			call_single_data_t *csd = per_cpu_ptr(cfd->csd, cpu);
>  
>  			if (cond_func && !cond_func(cpu, info)) {
> -				__cpumask_clear_cpu(cpu, cfd->cpumask);
> +				__cpumask_clear_cpu(cpu, cpumask);
>  				continue;
>  			}
>  
> @@ -890,13 +899,16 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
>  	}
>  
>  	if (run_remote && wait) {
> -		for_each_cpu(cpu, cfd->cpumask) {
> +		for_each_cpu(cpu, cpumask) {
>  			call_single_data_t *csd;
>  
>  			csd = per_cpu_ptr(cfd->csd, cpu);
>  			csd_lock_wait(csd);
>  		}
>  	}
> +
> +	if (preemptible_wait)
> +		free_cpumask_var(cpumask_stack);
>  }

*sigh*, even if you don't break RT, this is quite terrible, what is
wrong with something like so?

---
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -802,19 +802,18 @@ static void smp_call_function_many_cond(
 					unsigned int scf_flags,
 					smp_cond_func_t cond_func)
 {
+	struct call_function_data *cfd = this_cpu_ptr(&cfd_data);
 	int cpu, last_cpu, this_cpu = smp_processor_id();
-	struct call_function_data *cfd;
+	struct cpumask *cpumask = cfd->cpumask;
 	bool wait = scf_flags & SCF_WAIT;
-	bool preemptible_wait = true;
 	cpumask_var_t cpumask_stack;
-	struct cpumask *cpumask;
 	int nr_cpus = 0;
 	bool run_remote = false;
 
 	lockdep_assert_preemption_disabled();
 
 	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
-		preemptible_wait = false;
+		cpumask = cpumask_stack;
 
 	/*
 	 * Can deadlock when called with interrupts disabled.
@@ -836,10 +835,6 @@ static void smp_call_function_many_cond(
 
 	/* Check if we need remote execution, i.e., any CPU excluding this one. */
 	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
-		cfd = this_cpu_ptr(&cfd_data);
-
-		cpumask = preemptible_wait ? cpumask_stack : cfd->cpumask;
-
 		cpumask_and(cpumask, mask, cpu_online_mask);
 		__cpumask_clear_cpu(this_cpu, cpumask);
 
@@ -907,8 +902,7 @@ static void smp_call_function_many_cond(
 		}
 	}
 
-	if (preemptible_wait)
-		free_cpumask_var(cpumask_stack);
+	free_cpumask_var(cpumask_stack);
 }
 
 /**

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Peter Zijlstra]

On Tue, Feb 03, 2026 at 07:23:55PM +0800, Chuyi Zhou wrote:

> +	/*
> +	 * Prevent the current CPU from going offline.
> +	 * Being migrated to another CPU and calling csd_lock_wait() may cause
> +	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
> +	 */
> +	migrate_disable();

This is horrible crap. migrate_disable() is *NOT* supposed to be used to
serialize cpu hotplug.

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Peter Zijlstra]

On Thu, Feb 05, 2026 at 10:52:36AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 03, 2026 at 07:23:55PM +0800, Chuyi Zhou wrote:
> 
> > +	/*
> > +	 * Prevent the current CPU from going offline.
> > +	 * Being migrated to another CPU and calling csd_lock_wait() may cause
> > +	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
> > +	 */
> > +	migrate_disable();
> 
> This is horrible crap. migrate_disable() is *NOT* supposed to be used to
> serialize cpu hotplug.

This was too complicated or something?

--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -802,19 +802,20 @@ static void smp_call_function_many_cond(
 					unsigned int scf_flags,
 					smp_cond_func_t cond_func)
 {
-	int cpu, last_cpu, this_cpu = smp_processor_id();
-	struct call_function_data *cfd;
+	struct call_function_data *cfd = this_cpu_ptr(&cfd_data);
+	struct cpumask *cpumask = cfd->cpumask;
 	bool wait = scf_flags & SCF_WAIT;
-	bool preemptible_wait = true;
 	cpumask_var_t cpumask_stack;
-	struct cpumask *cpumask;
+	int cpu, last_cpu, this_cpu;
 	int nr_cpus = 0;
 	bool run_remote = false;
 
-	lockdep_assert_preemption_disabled();
+	if (wait && !alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
+		cpumask = cpumask_stack;
 
-	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
-		preemptible_wait = false;
+	cpus_read_lock();
+	preempt_disable();
+	this_cpu = smp_processor_id();
 
 	/*
 	 * Can deadlock when called with interrupts disabled.
@@ -836,10 +837,6 @@ static void smp_call_function_many_cond(
 
 	/* Check if we need remote execution, i.e., any CPU excluding this one. */
 	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
-		cfd = this_cpu_ptr(&cfd_data);
-
-		cpumask = preemptible_wait ? cpumask_stack : cfd->cpumask;
-
 		cpumask_and(cpumask, mask, cpu_online_mask);
 		__cpumask_clear_cpu(this_cpu, cpumask);
 
@@ -897,6 +894,7 @@ static void smp_call_function_many_cond(
 		csd_do_func(func, info, NULL);
 		local_irq_restore(flags);
 	}
+	preempt_enable();
 
 	if (run_remote && wait) {
 		for_each_cpu(cpu, cpumask) {
@@ -907,8 +905,8 @@ static void smp_call_function_many_cond(
 		}
 	}
 
-	if (preemptible_wait)
-		free_cpumask_var(cpumask_stack);
+	cpus_read_unlock();
+	free_cpumask_var(cpumask_stack);
 }
 
 /**

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Chuyi Zhou]

Hi Peter,

在 2026/2/5 18:57, Peter Zijlstra 写道:
> On Thu, Feb 05, 2026 at 10:52:36AM +0100, Peter Zijlstra wrote:
>> On Tue, Feb 03, 2026 at 07:23:55PM +0800, Chuyi Zhou wrote:
>>
>>> +	/*
>>> +	 * Prevent the current CPU from going offline.
>>> +	 * Being migrated to another CPU and calling csd_lock_wait() may cause
>>> +	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
>>> +	 */
>>> +	migrate_disable();
>>
>> This is horrible crap. migrate_disable() is *NOT* supposed to be used to
>> serialize cpu hotplug.
> 
> This was too complicated or something?
> 

Now most callers of smp_call*() explicitly use preempt_disable(). IIUC, 
if we want to use cpus_read_lock(), we first need to clean up all these 
preempt_disable() calls.

Maybe a stupid question: Why can't migrate_disable prevent CPU removal?

Before takedown_cpu(), all tasks need to be migrated to other CPUs, and 
all kthreads on that CPU must be parked, except the stopper thread and 
the hotplug thread.



> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -802,19 +802,20 @@ static void smp_call_function_many_cond(
>   					unsigned int scf_flags,
>   					smp_cond_func_t cond_func)
>   {
> -	int cpu, last_cpu, this_cpu = smp_processor_id();
> -	struct call_function_data *cfd;
> +	struct call_function_data *cfd = this_cpu_ptr(&cfd_data);
> +	struct cpumask *cpumask = cfd->cpumask;
>   	bool wait = scf_flags & SCF_WAIT;
> -	bool preemptible_wait = true;
>   	cpumask_var_t cpumask_stack;
> -	struct cpumask *cpumask;
> +	int cpu, last_cpu, this_cpu;
>   	int nr_cpus = 0;
>   	bool run_remote = false;
>   
> -	lockdep_assert_preemption_disabled();
> +	if (wait && !alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
> +		cpumask = cpumask_stack;
>   
> -	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
> -		preemptible_wait = false;
> +	cpus_read_lock();
> +	preempt_disable();
> +	this_cpu = smp_processor_id();
>   
>   	/*
>   	 * Can deadlock when called with interrupts disabled.
> @@ -836,10 +837,6 @@ static void smp_call_function_many_cond(
>   
>   	/* Check if we need remote execution, i.e., any CPU excluding this one. */
>   	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
> -		cfd = this_cpu_ptr(&cfd_data);
> -
> -		cpumask = preemptible_wait ? cpumask_stack : cfd->cpumask;
> -
>   		cpumask_and(cpumask, mask, cpu_online_mask);
>   		__cpumask_clear_cpu(this_cpu, cpumask);
>   
> @@ -897,6 +894,7 @@ static void smp_call_function_many_cond(
>   		csd_do_func(func, info, NULL);
>   		local_irq_restore(flags);
>   	}
> +	preempt_enable();
>   
>   	if (run_remote && wait) {
>   		for_each_cpu(cpu, cpumask) {
> @@ -907,8 +905,8 @@ static void smp_call_function_many_cond(
>   		}
>   	}
>   
> -	if (preemptible_wait)
> -		free_cpumask_var(cpumask_stack);
> +	cpus_read_unlock();
> +	free_cpumask_var(cpumask_stack);
>   }
>   
>   /**

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Peter Zijlstra]

On Thu, Feb 05, 2026 at 10:29:51PM +0800, Chuyi Zhou wrote:
> Hi Peter,
> 
> 在 2026/2/5 18:57, Peter Zijlstra 写道:
> > On Thu, Feb 05, 2026 at 10:52:36AM +0100, Peter Zijlstra wrote:
> >> On Tue, Feb 03, 2026 at 07:23:55PM +0800, Chuyi Zhou wrote:
> >>
> >>> +	/*
> >>> +	 * Prevent the current CPU from going offline.
> >>> +	 * Being migrated to another CPU and calling csd_lock_wait() may cause
> >>> +	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
> >>> +	 */
> >>> +	migrate_disable();
> >>
> >> This is horrible crap. migrate_disable() is *NOT* supposed to be used to
> >> serialize cpu hotplug.
> > 
> > This was too complicated or something?
> > 
> 
> Now most callers of smp_call*() explicitly use preempt_disable(). IIUC, 
> if we want to use cpus_read_lock(), we first need to clean up all these 
> preempt_disable() calls.
> 
> Maybe a stupid question: Why can't migrate_disable prevent CPU removal?

It can, but migrate_disable() is horrible, it should not be used if at
all possible.

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Chuyi Zhou]

Hi Peter,

在 2026/2/5 22:59, Peter Zijlstra 写道:
> On Thu, Feb 05, 2026 at 10:29:51PM +0800, Chuyi Zhou wrote:
>> Hi Peter,
>>
>> 在 2026/2/5 18:57, Peter Zijlstra 写道:
>>> On Thu, Feb 05, 2026 at 10:52:36AM +0100, Peter Zijlstra wrote:
>>>> On Tue, Feb 03, 2026 at 07:23:55PM +0800, Chuyi Zhou wrote:
>>>>
>>>>> +	/*
>>>>> +	 * Prevent the current CPU from going offline.
>>>>> +	 * Being migrated to another CPU and calling csd_lock_wait() may cause
>>>>> +	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
>>>>> +	 */
>>>>> +	migrate_disable();
>>>>
>>>> This is horrible crap. migrate_disable() is *NOT* supposed to be used to
>>>> serialize cpu hotplug.
>>>
>>> This was too complicated or something?
>>>
>>
>> Now most callers of smp_call*() explicitly use preempt_disable(). IIUC,
>> if we want to use cpus_read_lock(), we first need to clean up all these
>> preempt_disable() calls.
>>
>> Maybe a stupid question: Why can't migrate_disable prevent CPU removal?
> 
> It can, but migrate_disable() is horrible, it should not be used if at
> all possible.


As you pointed out, using cpus_read_lock() is the simplest approach, and 
indeed, that was the first solution we considered.

However, 99% of callers have preemption disabled, some of them even 
invoking it within spin_locks (for example, we might trigger a TLB flush 
while holding pte spinlocks).

It's difficult for us to eliminate all these preempt_disable(), 
especially for callers that disable preemption for other purposes, 
making the use of cpus_read_lock almost impossible.

In our production environment, we observed that the overhead of 
csd_lock_wait can be as high as several milliseconds, and in extreme 
cases, even exceed 10ms+. Generally speaking, the time spent on 
csd_lock_wait far exceeds the overhead of sending the IPI.

Disabling preemption for the entire duration would obviously affect the 
preemption latency of high-priority tasks, which is unacceptable. This 
optimization primarily targets PREEMPT, although PREEMPT_RT can also 
benefit from it.

Compared to the cost of disabling preemption entirely, maybe using 
migrate_disable() here seems to be an acceptable trade-off.

Re: [PATCH 03/11] smp: Remove get_cpu from smp_call_function_any

[Chuyi Zhou]

在 2026/2/5 17:42, Peter Zijlstra 写道:
> On Tue, Feb 03, 2026 at 07:23:53PM +0800, Chuyi Zhou wrote:
>> Now smp_call_function_single() would enable preemption before
>> csd_lock_wait() to reduce the critical section. To allow callers of
>> smp_call_function_any() to also benefit from this optimization, remove
>> get_cpu()/put_cpu() from smp_call_function_any().
>>
>> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
>> ---
>>   kernel/smp.c | 9 +++++++--
>>   1 file changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/kernel/smp.c b/kernel/smp.c
>> index 0858553f3666..f572716c3c7d 100644
>> --- a/kernel/smp.c
>> +++ b/kernel/smp.c
>> @@ -772,13 +772,18 @@ int smp_call_function_any(const struct cpumask *mask,
>>   	unsigned int cpu;
>>   	int ret;
>>   
>> +	/*
>> +	 * Prevent migration to another CPU after selecting the current CPU
>> +	 * as the target.
>> +	 */
>> +	guard(migrate)();
>> +
>>   	/* Try for same CPU (cheapest) */
>> -	cpu = get_cpu();
>> +	cpu = smp_processor_id();
>>   	if (!cpumask_test_cpu(cpu, mask))
>>   		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu));
>>   
>>   	ret = smp_call_function_single(cpu, func, info, wait);
>> -	put_cpu();
>>   	return ret;
> 
> Urgh, that's horrible.
> 
> Basically what you want is something like so:
> 
> 	bool enable = true;
> 	unsigned int cpu;
> 	int ret;
> 
> 	preempt_disable();
> 	cpu = smp_processor_id();
> 	if (!cpumask_test_cpu(cpu, mask)) {
> 		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu))
> 		enable = false;
> 		preempt_enable();
> 	}
> 
> 	ret = smp_call_function_single(cpu, func, info, wait);
> 	if (enable)
> 		preempt_enable();
> 	return ret;

OK, I will update it in the next version.

Re: [PATCH 04/11] smp: Use on-stack cpumask in smp_call_function_many_cond

[Chuyi Zhou]

在 2026/2/5 17:44, Peter Zijlstra 写道:
> On Tue, Feb 03, 2026 at 07:23:54PM +0800, Chuyi Zhou wrote:
>> This patch use on-stack cpumask to replace percpu cfd cpumask in
>> smp_call_function_many_cond(). alloc_cpumask_var() may fail when
>> CONFIG_CPUMASK_OFFSTACK is enabled. In such extreme case, fall back to
>> cfd->cpumask. This is a preparation for the next patch.
>>
>> Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
>> ---
>>   kernel/smp.c | 22 +++++++++++++++++-----
>>   1 file changed, 17 insertions(+), 5 deletions(-)
>>
>> diff --git a/kernel/smp.c b/kernel/smp.c
>> index f572716c3c7d..35948afced2e 100644
>> --- a/kernel/smp.c
>> +++ b/kernel/smp.c
>> @@ -805,11 +805,17 @@ static void smp_call_function_many_cond(const struct cpumask *mask,
>>   	int cpu, last_cpu, this_cpu = smp_processor_id();
>>   	struct call_function_data *cfd;
>>   	bool wait = scf_flags & SCF_WAIT;
>> +	bool preemptible_wait = true;
>> +	cpumask_var_t cpumask_stack;
>> +	struct cpumask *cpumask;
>>   	int nr_cpus = 0;
>>   	bool run_remote = false;
>>   
>>   	lockdep_assert_preemption_disabled();
>>   
>> +	if (!alloc_cpumask_var(&cpumask_stack, GFP_ATOMIC))
>> +		preemptible_wait = false;
> 
> IIRC this breaks RT, must not allocate with preemption disabled.

Thank you for the reminder.

Perhaps another feasible approach is only consider 
CONFIG_CPUMASK_OFFSTACK=n.

Of course, if we use cpus_read_lock and ensure that the caller’s context 
is sleepable, this issue would also be eliminated.

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Peter Zijlstra]

On Fri, Feb 06, 2026 at 04:43:48PM +0800, Chuyi Zhou wrote:

> However, 99% of callers have preemption disabled, some of them even 
> invoking it within spin_locks (for example, we might trigger a TLB flush 
> while holding pte spinlocks).

Then 99% of the callers don't benefit from this and won't see your
latency reduction -- and will be broken on PREEMPT_RT, no?

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Chuyi Zhou]

Hello,

在 2026/2/6 17:47, Peter Zijlstra 写道:
> On Fri, Feb 06, 2026 at 04:43:48PM +0800, Chuyi Zhou wrote:
> 
>> However, 99% of callers have preemption disabled, some of them even
>> invoking it within spin_locks (for example, we might trigger a TLB flush
>> while holding pte spinlocks).
> 
> Then 99% of the callers don't benefit from this and won't see your
> latency reduction -- and will be broken on PREEMPT_RT, no?

Once we make the preemption logic self-contained within smp_call, the 
disabling of preemption by most callers becomes unnecessary and can 
therefore be removed. We can preserve the few instances where it is 
still meaningful or apply special optimizations to them, much like the 
subsequent patch does for arch_tlbbatch_flush/flush_tlb_mm_range.

Consequently, the vast majority of callers stand to benefit. For the RT 
kernel, this optimization provides additional benefits when smp_call is 
invoked within spinlocks, as it reduces the length of non-preemptible 
critical sections.

We can invoke alloc_cpumask_var only when CONFIG_CPUMASK_OFFSTACK is 
disabled, thereby avoiding broken on RT.

However, using cpus_read_lock requires us to absolutely guarantee that 
the current context is sleepable, which is difficult to ensure.

Re: [PATCH 05/11] smp: Enable preemption early in smp_call_function_many_cond

[Chuyi Zhou]

在 2026/2/5 17:52, Peter Zijlstra 写道:
> On Tue, Feb 03, 2026 at 07:23:55PM +0800, Chuyi Zhou wrote:
> 
>> +	/*
>> +	 * Prevent the current CPU from going offline.
>> +	 * Being migrated to another CPU and calling csd_lock_wait() may cause
>> +	 * UAF due to smpcfd_dead_cpu() during the current CPU offline process.
>> +	 */
>> +	migrate_disable();
> 
> This is horrible crap. migrate_disable() is *NOT* supposed to be used to
> serialize cpu hotplug.



Here we can use rcu_read_lock to replace migrate_disable/cpus_read_lock, 
and in smpcfd_dead_cpu(), wait for all rcu read critical sections to 
exit before releasing percpu csd data.

This allows csd_lock_wait() to be preemptible and migratable, while 
avoiding concurrency issues between smpcfd_dead_cpu() and csd_lock_wait.

Thanks.