* [PATCH 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes
@ 2026-09-21 9:53 Chuyi Zhou
2026-09-21 9:53 ` [PATCH 1/5] x86/mm: Account for remote kernel TLB flush requests Chuyi Zhou
` (4 more replies)
0 siblings, 5 replies; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-21 9:53 UTC (permalink / raw)
To: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, bigeasy, clrkwllms, rostedt, nadav.amit
Cc: linux-kernel, Chuyi Zhou
This series follows up on the IPI completion preemption work [1] and
addresses the deferred flush_tlb_kernel_range() changes. The generic SMP
completion waits are already preemptible, and commit a5a162fe1ae1
("x86/mm: Re-enable preemption before flush_tlb_multi()") allows the mm
flush paths to use them. The kernel-range change was deferred during
review [2].
Container teardown and BPF map destruction can trigger kernel TLB
flushes through reclamation of unused per-CPU memory. Services that
spawn and reap many workers can also trigger flushes when vmalloc-backed
kernel stacks are reclaimed.
On the IPI backend, flush_tlb_kernel_range() flushes the local CPU,
sends flush requests to all other online CPUs, and waits synchronously
for completion. The target set is system-wide even when the container
or application is confined to a small subset of CPUs.
The synchronous wait can become longer as the number of online CPUs
grows. Completion depends on the slowest participating CPU, so a remote
CPU with interrupts disabled can delay the entire operation.
flush_tlb_kernel_range() keeps preemption disabled throughout that wait,
delaying higher-priority tasks on the initiating CPU.
The kernel path still uses init_flush_tlb_info(), which initializes
initiating_cpu with smp_processor_id(). Simply removing the outer
preemption guard would allow that initialization to run in a preemptible
context and could trigger a CONFIG_DEBUG_PREEMPT warning. The earlier
version used raw_smp_processor_id() to suppress that warning, but this
also removed the CPU-pinning check from the shared initializer used by
the mm paths.
A smaller change could keep preemption disabled only around the
init_flush_tlb_info() call in flush_tlb_kernel_range() and restore it
before dispatching the flush. With separate protection for INVLPGB and
TLBSYNC, this would also allow the final IPI wait to be preempted while
preserving the smp_processor_id() check.
Kernel flushes do not use initiating_cpu or the other mm-specific
fields. This series separates their data from flush_tlb_info to remove
the unused initialization and its preemption requirement. Full flushes
need no descriptor, and range flushes need only start/end. The
smp_processor_id() check remains in the initializer for the mm paths.
Removing the outer preemption guard then lets higher-priority tasks
preempt the final IPI completion wait when the calling context permits
it. Both flush backends remain synchronous, and the INVLPGB helpers
keep their required preemption protection.
The changes are split into five patches:
1. Account for kernel TLB flush requests in NR_TLB_REMOTE_FLUSH,
including ranges promoted to full flushes and both IPI and INVLPGB
backends.
2. Make flush_tlb_all() use kernel_tlb_flush_all(), sharing backend
dispatch and accounting while counting each request once.
3. Extract the range-to-full-flush threshold predicate without changing
its arithmetic or the existing flush policy.
4. Decouple kernel flushes from flush_tlb_info. Full flushes need no
descriptor, and range flushes need only start/end. Use a private
stack descriptor for IPI callbacks, preserving its lifetime through
the synchronous wait. Keep the smp_processor_id() check in the mm
descriptor initializer.
5. Remove the outer preemption guard from flush_tlb_kernel_range().
Keep the INVLPGB range loop and TLBSYNC protected inside their
backend helper; the full INVLPGB helper already has that protection.
[1] https://lore.kernel.org/lkml/20260709122933.4021501-1-zhouchuyi@bytedance.com/
[2] https://lore.kernel.org/9cd743e8-4d60-4a5b-906f-07e4ae82dafb@bytedance.com/
Chuyi Zhou (5):
x86/mm: Account for remote kernel TLB flush requests
x86/mm: Share the full TLB flush dispatch
x86/mm: Extract the TLB range flush threshold check
x86/mm: Decouple kernel TLB flushes from flush_tlb_info
x86/mm: Re-enable preemption before waiting for kernel TLB flushes
arch/x86/mm/tlb.c | 71 +++++++++++++++++++++++++++++++++----------------------
1 file changed, 43 insertions(+), 28 deletions(-)
base-commit: e81ee06308379a5f2ededf997bcf17551bce5db7
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/5] x86/mm: Account for remote kernel TLB flush requests
2026-09-21 9:53 [PATCH 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes Chuyi Zhou
@ 2026-09-21 9:53 ` Chuyi Zhou
2026-09-21 9:53 ` [PATCH 2/5] x86/mm: Share the full TLB flush dispatch Chuyi Zhou
` (3 subsequent siblings)
4 siblings, 0 replies; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-21 9:53 UTC (permalink / raw)
To: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, bigeasy, clrkwllms, rostedt, nadav.amit
Cc: linux-kernel, Chuyi Zhou
NR_TLB_REMOTE_FLUSH counts requests to invalidate TLB entries on other
CPUs. flush_tlb_all() and native_flush_tlb_multi() account for these
requests, but flush_tlb_kernel_range() bypasses both functions and does
not update the counter. Kernel range flushes are therefore missing from
the sender-side statistics, including ranges promoted to a full flush.
Count each request in kernel_tlb_flush_all() and
kernel_tlb_flush_range(). Account for both IPI and INVLPGB backends, as
flush_tlb_all() already does, and count once per request regardless of
the number of target CPUs or invalidation instructions.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
arch/x86/mm/tlb.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 0c4da320831c..da7c408f921c 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1488,6 +1488,8 @@ static void do_kernel_range_flush(void *info)
static void kernel_tlb_flush_all(struct flush_tlb_info *info)
{
+ count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
invlpgb_flush_all();
else
@@ -1496,6 +1498,8 @@ static void kernel_tlb_flush_all(struct flush_tlb_info *info)
static void kernel_tlb_flush_range(struct flush_tlb_info *info)
{
+ count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
+
if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
invlpgb_kernel_range_flush(info);
else
base-commit: e81ee06308379a5f2ededf997bcf17551bce5db7
--
2.20.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 2/5] x86/mm: Share the full TLB flush dispatch
2026-09-21 9:53 [PATCH 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes Chuyi Zhou
2026-09-21 9:53 ` [PATCH 1/5] x86/mm: Account for remote kernel TLB flush requests Chuyi Zhou
@ 2026-09-21 9:53 ` Chuyi Zhou
2026-09-23 9:17 ` Sebastian Andrzej Siewior
2026-09-21 9:53 ` [PATCH 3/5] x86/mm: Extract the TLB range flush threshold check Chuyi Zhou
` (2 subsequent siblings)
4 siblings, 1 reply; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-21 9:53 UTC (permalink / raw)
To: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, bigeasy, clrkwllms, rostedt, nadav.amit
Cc: linux-kernel, Chuyi Zhou
flush_tlb_all() and kernel_tlb_flush_all() select the same INVLPGB or
IPI backend and account for the same sender-side event. Keeping two
implementations duplicates the backend selection and accounting.
Use kernel_tlb_flush_all() for both the public full-flush interface and
kernel ranges promoted to a full flush. Drop its unused flush_tlb_info
argument and keep the event accounting in the shared helper so each
request is counted once.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
arch/x86/mm/tlb.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index da7c408f921c..5e9d1689f605 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1445,7 +1445,7 @@ static void do_flush_tlb_all(void *info)
__flush_tlb_all();
}
-void flush_tlb_all(void)
+static void kernel_tlb_flush_all(void)
{
count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
@@ -1457,6 +1457,11 @@ void flush_tlb_all(void)
on_each_cpu(do_flush_tlb_all, NULL, 1);
}
+void flush_tlb_all(void)
+{
+ kernel_tlb_flush_all();
+}
+
/* Flush an arbitrarily large range of memory with INVLPGB. */
static void invlpgb_kernel_range_flush(struct flush_tlb_info *info)
{
@@ -1486,16 +1491,6 @@ static void do_kernel_range_flush(void *info)
flush_tlb_one_kernel(addr);
}
-static void kernel_tlb_flush_all(struct flush_tlb_info *info)
-{
- count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
-
- if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
- invlpgb_flush_all();
- else
- on_each_cpu(do_flush_tlb_all, NULL, 1);
-}
-
static void kernel_tlb_flush_range(struct flush_tlb_info *info)
{
count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
@@ -1515,7 +1510,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
TLB_GENERATION_INVALID);
if (info.end == TLB_FLUSH_ALL)
- kernel_tlb_flush_all(&info);
+ kernel_tlb_flush_all();
else
kernel_tlb_flush_range(&info);
}
--
2.20.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 3/5] x86/mm: Extract the TLB range flush threshold check
2026-09-21 9:53 [PATCH 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes Chuyi Zhou
2026-09-21 9:53 ` [PATCH 1/5] x86/mm: Account for remote kernel TLB flush requests Chuyi Zhou
2026-09-21 9:53 ` [PATCH 2/5] x86/mm: Share the full TLB flush dispatch Chuyi Zhou
@ 2026-09-21 9:53 ` Chuyi Zhou
2026-09-21 9:53 ` [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info Chuyi Zhou
2026-09-21 9:59 ` [PATCH 5/5] x86/mm: Re-enable preemption before waiting for kernel TLB flushes Chuyi Zhou
4 siblings, 0 replies; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-21 9:53 UTC (permalink / raw)
To: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, bigeasy, clrkwllms, rostedt, nadav.amit
Cc: linux-kernel, Chuyi Zhou
The decision to replace a range flush with a full flush is embedded in
init_flush_tlb_info(). Both mm and kernel flushes use this policy, but
the kernel path only needs the range and the decision, without the
mm-specific descriptor initialization.
Extract the threshold predicate into tlb_range_exceeds_ceiling()
and use it in init_flush_tlb_info(). Preserve the unsigned range
arithmetic, right-shift rounding and strict greater-than comparison.
Descriptor initialization and both flush paths remain unchanged.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
arch/x86/mm/tlb.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 5e9d1689f605..f0dfeb271c66 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1373,6 +1373,12 @@ void flush_tlb_multi(const struct cpumask *cpumask,
*/
unsigned long tlb_single_page_flush_ceiling __read_mostly = 33;
+static bool tlb_range_exceeds_ceiling(unsigned long start, unsigned long end,
+ unsigned int stride_shift)
+{
+ return ((end - start) >> stride_shift) > tlb_single_page_flush_ceiling;
+}
+
static void init_flush_tlb_info(struct flush_tlb_info *info,
struct mm_struct *mm,
unsigned long start, unsigned long end,
@@ -1383,7 +1389,7 @@ static void init_flush_tlb_info(struct flush_tlb_info *info,
* If the number of flushes is so large that a full flush
* would be faster, do a full flush.
*/
- if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) {
+ if (tlb_range_exceeds_ceiling(start, end, stride_shift)) {
start = 0;
end = TLB_FLUSH_ALL;
}
--
2.20.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info
2026-09-21 9:53 [PATCH 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes Chuyi Zhou
` (2 preceding siblings ...)
2026-09-21 9:53 ` [PATCH 3/5] x86/mm: Extract the TLB range flush threshold check Chuyi Zhou
@ 2026-09-21 9:53 ` Chuyi Zhou
2026-09-23 9:28 ` Sebastian Andrzej Siewior
2026-09-21 9:59 ` [PATCH 5/5] x86/mm: Re-enable preemption before waiting for kernel TLB flushes Chuyi Zhou
4 siblings, 1 reply; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-21 9:53 UTC (permalink / raw)
To: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, bigeasy, clrkwllms, rostedt, nadav.amit
Cc: linux-kernel, Chuyi Zhou
Kernel range flushes only need the start and end addresses, but reuse
flush_tlb_info and its initialization of mm state, TLB generations and
the initiating CPU. None of those fields is consumed by the kernel
flush callbacks. In particular, initializing initiating_cpu imposes a
CPU-pinning requirement on a path that does not use it.
Select the full or range flush directly in flush_tlb_kernel_range(),
using the shared threshold predicate and an explicit TLB_FLUSH_ALL
check. Keep init_flush_tlb_info() and its smp_processor_id() check for
the mm paths.
Pass start and end directly through the kernel range helpers. Package
them in a private kernel_tlb_range only for the IPI callback, retaining
the existing payload alignment. The synchronous on_each_cpu() call
keeps the stack descriptor alive until all callbacks have completed.
Retain the outer preemption guard in flush_tlb_kernel_range() so the
descriptor refactoring does not change the preemption behavior.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
Link: https://lore.kernel.org/20260522104818.CbT5fyN8@linutronix.de/
---
arch/x86/mm/tlb.c | 41 +++++++++++++++++++++++++----------------
1 file changed, 25 insertions(+), 16 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index f0dfeb271c66..4f9f0a18dbbc 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1469,12 +1469,12 @@ void flush_tlb_all(void)
}
/* Flush an arbitrarily large range of memory with INVLPGB. */
-static void invlpgb_kernel_range_flush(struct flush_tlb_info *info)
+static void invlpgb_kernel_range_flush(unsigned long start, unsigned long end)
{
unsigned long addr, nr;
- for (addr = info->start; addr < info->end; addr += nr << PAGE_SHIFT) {
- nr = (info->end - addr) >> PAGE_SHIFT;
+ for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
+ nr = (end - addr) >> PAGE_SHIFT;
/*
* INVLPGB has a limit on the size of ranges it can
@@ -1487,38 +1487,47 @@ static void invlpgb_kernel_range_flush(struct flush_tlb_info *info)
__tlbsync();
}
+/* Preserve the alignment of the IPI payload shared with remote CPUs. */
+struct kernel_tlb_range {
+ unsigned long start;
+ unsigned long end;
+} __aligned(FLUSH_TLB_INFO_ALIGN);
+
static void do_kernel_range_flush(void *info)
{
- struct flush_tlb_info *f = info;
+ const struct kernel_tlb_range *range = info;
unsigned long addr;
/* flush range by one by one 'invlpg' */
- for (addr = f->start; addr < f->end; addr += PAGE_SIZE)
+ for (addr = range->start; addr < range->end; addr += PAGE_SIZE)
flush_tlb_one_kernel(addr);
}
-static void kernel_tlb_flush_range(struct flush_tlb_info *info)
+static void kernel_tlb_flush_range(unsigned long start, unsigned long end)
{
count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
- if (cpu_feature_enabled(X86_FEATURE_INVLPGB))
- invlpgb_kernel_range_flush(info);
- else
- on_each_cpu(do_kernel_range_flush, info, 1);
+ if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) {
+ invlpgb_kernel_range_flush(start, end);
+ } else {
+ struct kernel_tlb_range range = {
+ .start = start,
+ .end = end,
+ };
+
+ on_each_cpu(do_kernel_range_flush, &range, 1);
+ }
}
void flush_tlb_kernel_range(unsigned long start, unsigned long end)
{
- struct flush_tlb_info info;
-
guard(preempt)();
- init_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false,
- TLB_GENERATION_INVALID);
- if (info.end == TLB_FLUSH_ALL)
+ if (end == TLB_FLUSH_ALL ||
+ tlb_range_exceeds_ceiling(start, end, PAGE_SHIFT))
kernel_tlb_flush_all();
else
- kernel_tlb_flush_range(&info);
+ kernel_tlb_flush_range(start, end);
}
/*
--
2.20.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 5/5] x86/mm: Re-enable preemption before waiting for kernel TLB flushes
2026-09-21 9:53 [PATCH 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes Chuyi Zhou
` (3 preceding siblings ...)
2026-09-21 9:53 ` [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info Chuyi Zhou
@ 2026-09-21 9:59 ` Chuyi Zhou
4 siblings, 0 replies; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-21 9:59 UTC (permalink / raw)
To: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, bigeasy, clrkwllms, rostedt, nadav.amit
Cc: linux-kernel, Chuyi Zhou
flush_tlb_kernel_range() uses on_each_cpu() to synchronously flush
kernel mappings on all online CPUs when using the IPI backend.
The synchronous wait can become longer as the number of online CPUs
grows, and a remote CPU with interrupts disabled can further delay
completion. Preemption remains disabled throughout this wait, delaying
higher-priority tasks on the initiating CPU. The outer guard prevents
the SMP layer from making its final completion wait preemptible.
The kernel range descriptor is private stack storage and remains valid
until on_each_cpu() returns. The SMP layer protects CPU selection, IPI
queueing and local callback execution, so the kernel flush caller does
not need to remain pinned while waiting for remote completion.
INVLPGB requires separate protection: TLBSYNC only waits for operations
issued on the same CPU. Disable preemption inside
invlpgb_kernel_range_flush() across the broadcast loop and TLBSYNC.
The full-flush helper invlpgb_flush_all() already provides this
protection.
Remove the outer preemption guard from flush_tlb_kernel_range() so the
IPI completion wait can be preempted when the caller's context allows
it. Keep both flush backends synchronous.
Signed-off-by: Chuyi Zhou <zhouchuyi@bytedance.com>
---
arch/x86/mm/tlb.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 4f9f0a18dbbc..f7773ba4a152 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -1473,6 +1473,9 @@ static void invlpgb_kernel_range_flush(unsigned long start, unsigned long end)
{
unsigned long addr, nr;
+ /* Keep the INVLPGB operations and TLBSYNC on the same CPU. */
+ guard(preempt)();
+
for (addr = start; addr < end; addr += nr << PAGE_SHIFT) {
nr = (end - addr) >> PAGE_SHIFT;
@@ -1521,8 +1524,6 @@ static void kernel_tlb_flush_range(unsigned long start, unsigned long end)
void flush_tlb_kernel_range(unsigned long start, unsigned long end)
{
- guard(preempt)();
-
if (end == TLB_FLUSH_ALL ||
tlb_range_exceeds_ceiling(start, end, PAGE_SHIFT))
kernel_tlb_flush_all();
--
2.20.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/5] x86/mm: Share the full TLB flush dispatch
2026-09-21 9:53 ` [PATCH 2/5] x86/mm: Share the full TLB flush dispatch Chuyi Zhou
@ 2026-09-23 9:17 ` Sebastian Andrzej Siewior
2026-09-23 9:25 ` Chuyi Zhou
0 siblings, 1 reply; 12+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-09-23 9:17 UTC (permalink / raw)
To: Chuyi Zhou
Cc: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, clrkwllms, rostedt, nadav.amit, linux-kernel
On 2026-09-21 17:53:56 [+0800], Chuyi Zhou wrote:
> --- a/arch/x86/mm/tlb.c
> +++ b/arch/x86/mm/tlb.c
> @@ -1445,7 +1445,7 @@ static void do_flush_tlb_all(void *info)
> __flush_tlb_all();
> }
>
> -void flush_tlb_all(void)
> +static void kernel_tlb_flush_all(void)
> {
> count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
>
> @@ -1457,6 +1457,11 @@ void flush_tlb_all(void)
> on_each_cpu(do_flush_tlb_all, NULL, 1);
> }
>
> +void flush_tlb_all(void)
> +{
> + kernel_tlb_flush_all();
> +}
> +
Would it make sense to drop kernel_tlb_flush_all() which is used only
here and instead keep flush_tlb_all() which has also external users?
Sebastian
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 2/5] x86/mm: Share the full TLB flush dispatch
2026-09-23 9:17 ` Sebastian Andrzej Siewior
@ 2026-09-23 9:25 ` Chuyi Zhou
0 siblings, 0 replies; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-23 9:25 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, clrkwllms, rostedt, nadav.amit, linux-kernel
Hi
On 2026-09-23 5:17 p.m., Sebastian Andrzej Siewior wrote:
> On 2026-09-21 17:53:56 [+0800], Chuyi Zhou wrote:
>> --- a/arch/x86/mm/tlb.c
>> +++ b/arch/x86/mm/tlb.c
>> @@ -1445,7 +1445,7 @@ static void do_flush_tlb_all(void *info)
>> __flush_tlb_all();
>> }
>>
>> -void flush_tlb_all(void)
>> +static void kernel_tlb_flush_all(void)
>> {
>> count_vm_tlb_event(NR_TLB_REMOTE_FLUSH);
>>
>> @@ -1457,6 +1457,11 @@ void flush_tlb_all(void)
>> on_each_cpu(do_flush_tlb_all, NULL, 1);
>> }
>>
>> +void flush_tlb_all(void)
>> +{
>> + kernel_tlb_flush_all();
>> +}
>> +
>
> Would it make sense to drop kernel_tlb_flush_all() which is used only
> here and instead keep flush_tlb_all() which has also external users?
Yes, that makes sense. flush_tlb_all() already provides the same
accounting and backend dispatch. I'll drop kernel_tlb_flush_all() and
call flush_tlb_all() directly from flush_tlb_kernel_range().
>
> Sebastian
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info
2026-09-21 9:53 ` [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info Chuyi Zhou
@ 2026-09-23 9:28 ` Sebastian Andrzej Siewior
2026-09-23 10:43 ` Chuyi Zhou
0 siblings, 1 reply; 12+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-09-23 9:28 UTC (permalink / raw)
To: Chuyi Zhou
Cc: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, clrkwllms, rostedt, nadav.amit, linux-kernel
On 2026-09-21 17:53:58 [+0800], Chuyi Zhou wrote:
> --- a/arch/x86/mm/tlb.c
> +++ b/arch/x86/mm/tlb.c
> @@ -1487,38 +1487,47 @@ static void invlpgb_kernel_range_flush(struct flush_tlb_info *info)
…
> void flush_tlb_kernel_range(unsigned long start, unsigned long end)
> {
> - struct flush_tlb_info info;
> -
> guard(preempt)();
> - init_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false,
> - TLB_GENERATION_INVALID);
>
> - if (info.end == TLB_FLUSH_ALL)
> + if (end == TLB_FLUSH_ALL ||
info.end might be TLB_FLUSH_ALL because init_flush_tlb_info() might set
it so. But 'end', which is passed as an argument, should not be
TLB_FLUSH_ALL or can it?
And if so, wouldn't the check below cover it anyway?
> + tlb_range_exceeds_ceiling(start, end, PAGE_SHIFT))
> kernel_tlb_flush_all();
> else
> - kernel_tlb_flush_range(&info);
> + kernel_tlb_flush_range(start, end);
> }
Sebastian
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info
2026-09-23 9:28 ` Sebastian Andrzej Siewior
@ 2026-09-23 10:43 ` Chuyi Zhou
2026-09-23 10:50 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-23 10:43 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, clrkwllms, rostedt, nadav.amit, linux-kernel
On 2026-09-23 5:28 p.m., Sebastian Andrzej Siewior wrote:
> On 2026-09-21 17:53:58 [+0800], Chuyi Zhou wrote:
>> --- a/arch/x86/mm/tlb.c
>> +++ b/arch/x86/mm/tlb.c
>> @@ -1487,38 +1487,47 @@ static void invlpgb_kernel_range_flush(struct flush_tlb_info *info)
> …
>> void flush_tlb_kernel_range(unsigned long start, unsigned long end)
>> {
>> - struct flush_tlb_info info;
>> -
>> guard(preempt)();
>> - init_flush_tlb_info(&info, NULL, start, end, PAGE_SHIFT, false,
>> - TLB_GENERATION_INVALID);
>>
>> - if (info.end == TLB_FLUSH_ALL)
>> + if (end == TLB_FLUSH_ALL ||
>
> info.end might be TLB_FLUSH_ALL because init_flush_tlb_info() might set
> it so. But 'end', which is passed as an argument, should not be
> TLB_FLUSH_ALL or can it?
> And if so, wouldn't the check below cover it anyway?
I couldn't find any callers passing TLB_FLUSH_ALL as end.
flush_tlb_kernel_range(start, end) is used to flush actual address
ranges, and flush_tlb_all() is available for unconditional full flushes.
So I agree that we can drop the explicit check.
The ceiling check does not cover all possible inputs, though. For
example, with start = ULONG_MAX - PAGE_SIZE + 1 and end = TLB_FLUSH_ALL,
the expression (end - start) >> PAGE_SHIFT evaluates to zero. The
ceiling check would therefore select a range flush, whereas the explicit
check would select a full flush.
I added the explicit check to preserve that behavior of the old code,
but the current callers do not need it. I'll remove it.
>
>> + tlb_range_exceeds_ceiling(start, end, PAGE_SHIFT))
>> kernel_tlb_flush_all();
>> else
>> - kernel_tlb_flush_range(&info);
>> + kernel_tlb_flush_range(start, end);
>> }
>
> Sebastian
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info
2026-09-23 10:43 ` Chuyi Zhou
@ 2026-09-23 10:50 ` Sebastian Andrzej Siewior
2026-09-23 10:54 ` Chuyi Zhou
0 siblings, 1 reply; 12+ messages in thread
From: Sebastian Andrzej Siewior @ 2026-09-23 10:50 UTC (permalink / raw)
To: Chuyi Zhou
Cc: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, clrkwllms, rostedt, nadav.amit, linux-kernel
On 2026-09-23 18:43:55 [+0800], Chuyi Zhou wrote:
> The ceiling check does not cover all possible inputs, though. For
> example, with start = ULONG_MAX - PAGE_SIZE + 1 and end = TLB_FLUSH_ALL,
Well, due to the number of pages when it is considered to do a full
flush. But "ULONG_MAX - PAGE_SIZE + 1" isn't a valid start input, is
it?
Sebastian
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info
2026-09-23 10:50 ` Sebastian Andrzej Siewior
@ 2026-09-23 10:54 ` Chuyi Zhou
0 siblings, 0 replies; 12+ messages in thread
From: Chuyi Zhou @ 2026-09-23 10:54 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: tglx, mingo, luto, peterz, paulmck, muchun.song, bp, dave.hansen,
pbonzini, clrkwllms, rostedt, nadav.amit, linux-kernel
On 2026-09-23 6:50 p.m., Sebastian Andrzej Siewior wrote:
> On 2026-09-23 18:43:55 [+0800], Chuyi Zhou wrote:
>> The ceiling check does not cover all possible inputs, though. For
>> example, with start = ULONG_MAX - PAGE_SIZE + 1 and end = TLB_FLUSH_ALL,
>
> Well, due to the number of pages when it is considered to do a full
> flush. But "ULONG_MAX - PAGE_SIZE + 1" isn't a valid start input, is
> it?
Correct. On x86-64 that address falls in the unused hole at the top
of the address space, so it does not represent the start of an actual
kernel mapping.
My example only illustrated a difference for arbitrary input values;
it did not demonstrate a valid use case that needs the explicit check.
I'll drop it.
>
> Sebastian
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-09-23 10:55 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-21 9:53 [PATCH 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes Chuyi Zhou
2026-09-21 9:53 ` [PATCH 1/5] x86/mm: Account for remote kernel TLB flush requests Chuyi Zhou
2026-09-21 9:53 ` [PATCH 2/5] x86/mm: Share the full TLB flush dispatch Chuyi Zhou
2026-09-23 9:17 ` Sebastian Andrzej Siewior
2026-09-23 9:25 ` Chuyi Zhou
2026-09-21 9:53 ` [PATCH 3/5] x86/mm: Extract the TLB range flush threshold check Chuyi Zhou
2026-09-21 9:53 ` [PATCH 4/5] x86/mm: Decouple kernel TLB flushes from flush_tlb_info Chuyi Zhou
2026-09-23 9:28 ` Sebastian Andrzej Siewior
2026-09-23 10:43 ` Chuyi Zhou
2026-09-23 10:50 ` Sebastian Andrzej Siewior
2026-09-23 10:54 ` Chuyi Zhou
2026-09-21 9:59 ` [PATCH 5/5] x86/mm: Re-enable preemption before waiting for kernel TLB flushes Chuyi Zhou
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®