From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from va-1-111.ptr.blmpb.com (va-1-111.ptr.blmpb.com [209.127.230.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C64F3F8893 for ; Tue, 9 Jun 2026 09:21:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.111 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780996881; cv=none; b=JluTPPp6aDbAupVcazhW47aRWe2pKIeXkg/YKV111/PopmJ1aE8+ppeABYU5F1Vyyz8XWHfzNyig6nEtZ70dD5jXyVtVxXT1obKUJWw3JQZETOQX9UMviLzdtXbyQISCsNX+nOIOLyEj2lETXCHJ32gGDWFMVQ7+Y1usn/npois= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780996881; c=relaxed/simple; bh=w9dSdJ+5vDbjuf88O9Bd5OEVwIGlUhKELtvsFDUqj08=; h=To:Cc:Subject:References:Message-Id:From:Date:Mime-Version: In-Reply-To:Content-Type; b=PVxNIrLzfWpQUwI9BFR0fOum+EEZ6o6ugJvepUj45YeLPE14lDKM0+brCqYVYJ3EN6XlGjYHf7kAhRdZPvhFMKQ/z/j3Lc8pJGqNFq4IvYJja8qgYxxr2Q80W+b0w68M0N0eidkjLg1PUC0TDnAR/dzZY8nbBqDcIxEbYwehG7I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ShemyTYy; arc=none smtp.client-ip=209.127.230.111 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ShemyTYy" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780996874; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=yld4OBcK0s0WsexcSGJJ89I/GAAeAhu40g6J5PvfICk=; b=ShemyTYyNOpRJXJL+iT0UU5Z7ji3N/R3eQHOTiQh2mDvcL+Hv7tXTktnsu3Tm0qvedpKSU yd6b3JhNOOkcOB2LE6tfR5K3f1bdgKEQSCm40mpivlsyI3jvhA1zvM2D8kepDza4XzNRNw UXfsGXzYo4zh1yr9AJbjvSIWrWdGoGt/IVdwvJXOujlmdvYEnz71fkYcz8dMv+ctyEHgP2 HW9K83nvQjWrsPDfehZdREAQ3+uIL0p9ZMBto+sTrPpdp/mmQBzrRM3w0af4a5cQo5tzVW 1AGtcZPN9xJvqcwhNNdsIgNfqt3xC2V7Tjp3PBQy7o9mfY/AXQg9B/ZPl2cZNg== To: , , , , , , , , , , , , , Cc: , "Chuyi Zhou" Subject: [PATCH v7 14/14] x86/mm: Re-enable preemption before flush_tlb_multi() Content-Transfer-Encoding: 7bit X-Original-From: Chuyi Zhou References: <20260609091710.404083-1-zhouchuyi@bytedance.com> Message-Id: <20260609091710.404083-15-zhouchuyi@bytedance.com> From: "Chuyi Zhou" Date: Tue, 9 Jun 2026 17:17:10 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 In-Reply-To: <20260609091710.404083-1-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 X-Lms-Return-Path: Content-Type: text/plain; charset=UTF-8 flush_tlb_mm_range() and arch_tlbbatch_flush() pin the current CPU while they decide whether the flush can be handled locally or must be sent to remote CPUs. The CPU pinning is needed for the current CPU number and for the local TLB flush path, which reads per-CPU TLB state. It is not needed while waiting for a remote TLB flush to complete. After the remote-flush path has been selected, flush_tlb_info is caller-private stack storage, so the caller no longer has to stay on the same CPU to protect a shared per-CPU flush_tlb_info object. flush_tlb_multi() may also route through x86 PV backends. Those backends must protect their own CPU-local scratch state instead of relying on the caller to stay pinned. Hyper-V already does this by disabling interrupts while using hyperv_pcpu_input_arg, and Xen's multicall path brackets its per-CPU multicall buffer with xen_mc_batch()/xen_mc_issue(). The previous patch makes the KVM backend do the same for __pv_cpu_mask. Remote TLB flushes may synchronously wait for many CPUs, and the wait can take tens of milliseconds when remote CPUs have interrupts disabled or when many CPUs are involved. Keeping preemption disabled for that whole wait unnecessarily increases scheduling latency on the initiating CPU. Drop the CPU pinning before calling flush_tlb_multi() in the remote paths of flush_tlb_mm_range() and arch_tlbbatch_flush(). Keep the local paths inside the pinned section because they still access this CPU's TLB state. Signed-off-by: Chuyi Zhou Reviewed-by: Sebastian Andrzej Siewior Tested-by: Paul E. McKenney --- arch/x86/mm/tlb.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 0c55ee84d50c..807a15b1af19 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1404,6 +1404,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, { struct flush_tlb_info _info; struct flush_tlb_info *info = &_info; + bool remote_flush = false; int cpu = get_cpu(); u64 new_tlb_gen; @@ -1421,9 +1422,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, if (mm_global_asid(mm)) { broadcast_tlb_flush(info); } else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { - info->trim_cpumask = should_trim_cpumask(mm); - flush_tlb_multi(mm_cpumask(mm), info); - consider_global_asid(mm); + remote_flush = true; } else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1432,6 +1431,13 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, } put_cpu(); + + if (remote_flush) { + info->trim_cpumask = should_trim_cpumask(mm); + flush_tlb_multi(mm_cpumask(mm), info); + consider_global_asid(mm); + } + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } @@ -1678,7 +1684,7 @@ EXPORT_SYMBOL_FOR_KVM(__flush_tlb_all); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { struct flush_tlb_info info; - + bool remote_flush = false; int cpu = get_cpu(); init_flush_tlb_info(&info, NULL, 0, TLB_FLUSH_ALL, 0, false, @@ -1692,7 +1698,7 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) invlpgb_flush_all_nonglobals(); batch->unmapped_pages = false; } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { - flush_tlb_multi(&batch->cpumask, &info); + remote_flush = true; } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1700,9 +1706,12 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } - cpumask_clear(&batch->cpumask); - put_cpu(); + + if (remote_flush) + flush_tlb_multi(&batch->cpumask, &info); + + cpumask_clear(&batch->cpumask); } /* -- 2.20.1