From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sg-1-103.ptr.blmpb.com (sg-1-103.ptr.blmpb.com [118.26.132.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68C33387348 for ; Mon, 2 Mar 2026 07:55:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.103 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772438111; cv=none; b=aNChTO1a0n+8gH76zFGSoTAlLp/kcBToel4bxH67fjxRsCnhzQb+6ntqVDiUn939ZQqTcVzVNSxxaHr+SxZQtkmJjb7kpgHg6ZEytogpCf5hMAddudAh85zNN1U7OXJaMrTYJpg6vh4xxwOXnw9bsNxBRgd8Ifu4jKdmZVlYfxY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772438111; c=relaxed/simple; bh=y59GHL9JlZyhg7fkewWl8OtZk6FJ0uRTr7r6nAfQRHM=; h=Subject:In-Reply-To:To:Cc:Content-Type:Mime-Version:References: From:Message-Id:Date; b=V480FMco/mcVeQ3m7MiVRtC7OQTvbvcsuK62tYzO+7XlsedrpXpHHVLmUEh4kcvTB3j4+00suD2l9Yk/tI5dac7gSKZC98mqepSrn9q+fyGBzSq4ASWzWVx8czZJEQ4N4hZO6k4JJYKs69o0BpP108/hyEmOIHDEBA645Yo2U7U= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=nNTCtimC; arc=none smtp.client-ip=118.26.132.103 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="nNTCtimC" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1772438103; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=tuek/6tM4UVEV2b2X9kHS269FPRLDmhOYMEZPzb5Qw8=; b=nNTCtimCswJg4iBTQ0KIj+8y6ZQCPALl7EfkK6t2N8KdmXxIQ4h42acSrRius0HNhIN2HW vIhh1u1EFc67wHR0LO30IksxXYfeogygAYBZhqf8Tk2LOEoGp4K4esVPi4FzNmUFwEcGTA uMOeG56Bqksl0ds8BQq1G47Fyydt/0yr++zV3Nlvi2f8jIK6KTe9CCSrP7in9EmEsQ/oTZ z+GZqllmD22uagcFbkf0TlgfRJGnoiHFCrKXOAGoNnkXjIsU5jhFB1NDo6S15umiLTyl7i iwmtKmKnCSpOOcwbsQquBp8FVFlKKLBSrBgZsig0dR5eTevg3ann4dKCHtiLQA== Subject: [PATCH v2 11/12] x86/mm: Enable preemption during native_flush_tlb_multi X-Mailer: git-send-email 2.20.1 In-Reply-To: <20260302075216.2170675-1-zhouchuyi@bytedance.com> X-Original-From: Chuyi Zhou To: , , , , , , , , , , , Cc: , "Chuyi Zhou" Content-Type: text/plain; charset=UTF-8 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: 7bit References: <20260302075216.2170675-1-zhouchuyi@bytedance.com> From: "Chuyi Zhou" Message-Id: <20260302075216.2170675-12-zhouchuyi@bytedance.com> Date: Mon, 2 Mar 2026 15:52:15 +0800 X-Lms-Return-Path: flush_tlb_mm_range()/arch_tlbbatch_flush() -> native_flush_tlb_multi() is a common path in real production environments. When pages are reclaimed or process exit, native_flush_tlb_multi() sends IPIs to remote CPUs and waits for all remote CPUs to complete their local TLB flushes. The overall latency may reach tens of milliseconds due to a large number of remote CPUs and other factors (such as interrupts being disabled). Since flush_tlb_mm_range()/arch_tlbbatch_flush() always disable preemption, which may cause increased scheduling latency for other threads on the current CPU. Previous patch converted flush_tlb_info from per-cpu variable to on-stack variable. Additionally, it's no longer necessary to explicitly disable preemption before calling smp_call*() since they internally handles the preemption logic. Now is's safe to enable preemption during native_flush_tlb_multi(). Signed-off-by: Chuyi Zhou --- arch/x86/kernel/kvm.c | 4 +++- arch/x86/mm/tlb.c | 9 +++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 3bc062363814..4f7f4c1149b9 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -668,8 +668,10 @@ static void kvm_flush_tlb_multi(const struct cpumask *cpumask, u8 state; int cpu; struct kvm_steal_time *src; - struct cpumask *flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask); + struct cpumask *flushmask; + guard(preempt)(); + flushmask = this_cpu_cpumask_var_ptr(__pv_cpu_mask); cpumask_copy(flushmask, cpumask); /* * We have to call flush only on online vCPUs. And diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 91a0fb389303..86d9c208e424 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1427,9 +1427,11 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, if (mm_global_asid(mm)) { broadcast_tlb_flush(&info); } else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + put_cpu(); info.trim_cpumask = should_trim_cpumask(mm); flush_tlb_multi(mm_cpumask(mm), &info); consider_global_asid(mm); + goto invalidate; } else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1438,6 +1440,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, } put_cpu(); +invalidate: mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } @@ -1718,7 +1721,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) invlpgb_flush_all_nonglobals(); batch->unmapped_pages = false; } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + put_cpu(); flush_tlb_multi(&batch->cpumask, &info); + goto clear; } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled(); local_irq_disable(); @@ -1726,9 +1731,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } - cpumask_clear(&batch->cpumask); - put_cpu(); +clear: + cpumask_clear(&batch->cpumask); } /* -- 2.20.1