From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sg-1-102.ptr.blmpb.com (sg-1-102.ptr.blmpb.com [118.26.132.102]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 565DB1DF73C for ; Mon, 2 Mar 2026 07:52:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=118.26.132.102 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772437980; cv=none; b=YbnOGlgzfgGXNwvoYvvb28frivD8r0MPuOkd5Z3idX5+ME/eM7BrpTam1htwIkYfM4yNVTsPUe8+iob3Xqs97bhZjNEo5n2k9Adi6Tor+Ka5JsWrcN7PUffTmTURBZVyspTPkrHg1MRVx6vqy/yNNY0l7n15iLGl58x4ffhTe98= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772437980; c=relaxed/simple; bh=WswNYHMA+0O12fKzr4Hf5gUKh+sscj9bCUdkVbqr9J8=; h=From:Subject:Mime-Version:To:Cc:Message-Id:Content-Type:Date; b=BkvaX/ZCUuhpRNrk/LpJLsIcG1Oz/KZhg6bSQ2kaYfsuOsvyQW8+T0XhfQzIfAIlMLcawHhWJeclQMOyd1wzOHmUoXvsZ6GR3nJulVmcuTysINA94rOktcyMl5Wx4xzKOVayhmjL+wZpfNrC2ozypBXfgrXu+/n5eiEicamjNK8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=kqdwOcEp; arc=none smtp.client-ip=118.26.132.102 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="kqdwOcEp" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1772437965; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=yQWoJp/cNjEtKnoOWZ8wjbAq2t0fTyQfdNLzWbV4+/E=; b=kqdwOcEpKZEJqUsRuyznpPtTHcyTshXVGw9mvrdWphwKzvCVc/oa8oh9DIZHLwluWY6JUt UpAKQwSHDlT6Gtsec0ozsirrdZ/JfKx2y7MMwtIS+wVBbQDVHWlCuzfuCfiUan8d37LQKd 7lx1hmwJ8JTLJUjjYsl4YjS4KqJdPJgajCpu4vM9adJf73RIees8sBpHUeV7TXmCxcUnhE 9OcDLK7CwfeQ/mapCa4fxzjFoLewXIMevtXXGiFLqE4UZeQCpwwnxffZqBeL42635qvQex SRy1xjkINmxSGh1/U95YxmJaYBpQcLfwg5U89FT062KIgJg7xUnNB6Y1t4HzNQ== From: "Chuyi Zhou" Subject: [PATCH v2 00/12] Allow preemption during IPI completion waiting to improve real-time performance Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Original-From: Chuyi Zhou To: , , , , , , , , , , , Cc: , "Chuyi Zhou" Message-Id: <20260302075216.2170675-1-zhouchuyi@bytedance.com> Content-Type: text/plain; charset=UTF-8 Date: Mon, 2 Mar 2026 15:52:04 +0800 X-Lms-Return-Path: X-Mailer: git-send-email 2.20.1 Changes in v2: - Simplify the code comments in [PATCH v2 2/12] (pointed by peter and muchun) - Adjust the preemption disabling logic in smp_call_function_any() in [PATCH v2 3/12] (suggested by peter). - Use on-stack cpumask only when !CONFIG_CPUMASK_OFFSTACK in [PATCH V2 4/12] (pointed by peter) - Add [PATCH v2 5/12] to replace migrate_disable with the rcu mechanism - Adjust the preemption disabling logic to allow flush_tlb_multi() to be preemptible and migratable in [PATCH v2 11/12] - Collect Acked-bys and Reviewed-bys Introduction =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The vast majority of smp_call_function*() callers block until remote CPUs complete the IPI function execution. As smp_call_function*() runs with preemption disabled throughout, scheduling latency increases dramatically with the number of remote CPUs and other factors (such as interrupts being disabled). On x86-64 architectures, TLB flushes are performed via IPIs; thus, during process exit or when process-mapped pages are reclaimed, numerous IPI operations must be awaited, leading to increased scheduling latency for other threads on the current CPU. In our production environment, we observed IPI wait-induced scheduling latency reaching up to 16ms on a 16-core machine. Our goal is to allow preemption during IPI completion waiting to improve real-time performance. Background =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D In our production environments, latency-sensitive workloads (DPDK) are configured with the highest priority to preempt lower-priority tasks at any time. We discovered that DPDK's wake-up latency is primarily caused by the current CPU having preemption disabled. Therefore, we collected the maximum preemption disabled events within every 30-second interval and then calculated the P50/P99 of these max preemption disabled events: p50(ns) p99(ns) cpu0 254956 5465050 cpu1 115801 120782 cpu2 43324 72957 cpu3 256637 16723307 cpu4 58979 87237 cpu5 47464 79815 cpu6 48881 81371 cpu7 52263 82294 cpu8 263555 4657713 cpu9 44935 73962 cpu10 37659 65026 cpu11 257008 2706878 cpu12 49669 90006 cpu13 45186 74666 cpu14 60705 83866 cpu15 51311 86885 Meanwhile, we have collected the distribution of preemption disabling events exceeding 1ms across different CPUs over several hours(I omitted CPU data that were all zeros): CPU 1~10ms 10~50ms 50~100ms cpu0 29 5 0 cpu3 38 13 0 cpu8 34 6 0 cpu11 24 10 0 The preemption disabled for several milliseconds or even 10ms+ mostly originates from TLB flush: @stack[ trace_preempt_on+143 trace_preempt_on+143 preempt_count_sub+67 arch_tlbbatch_flush/flush_tlb_mm_range task_exit/page_reclaim/... ] Further analysis confirms that the majority of the time is consumed in csd_lock_wait(). Now smp_call*() always needs to disable preemption, mainly to protect its internal per=E2=80=91CPU data structures and synchronize with CPU offline operations. This patchset attempts to make csd_lock_wait() preemptible, thereby reducing the preemption=E2=80=91disabled critical section and impro= ving kernel real=E2=80=91time performance. Effect =3D=3D=3D=3D=3D=3D After applying this patchset, we no longer observe preemption disabled for more than 1ms on the arch_tlbbatch_flush/flush_tlb_mm_range path. The overall P99 of max preemption disabled events in every 30-second is reduced to around 1.5ms (the remaining latency is primarily due to lock contention. before patch after patch reduced by ----------- -------------- ------------ p99(ns) 16723307 1556034 ~90.70% Chuyi Zhou (12): smp: Disable preemption explicitly in __csd_lock_wait smp: Enable preemption early in smp_call_function_single smp: Remove get_cpu from smp_call_function_any smp: Use on-stack cpumask in smp_call_function_many_cond smp: Free call_function_data via RCU in smpcfd_dead_cpu smp: Enable preemption early in smp_call_function_many_cond smp: Remove preempt_disable from smp_call_function smp: Remove preempt_disable from on_each_cpu_cond_mask scftorture: Remove preempt_disable in scftorture_invoke_one x86/mm: Move flush_tlb_info back to the stack x86/mm: Enable preemption during native_flush_tlb_multi x86/mm: Enable preemption during flush_tlb_kernel_range arch/x86/kernel/kvm.c | 4 +- arch/x86/mm/tlb.c | 137 ++++++++++++++++++------------------------ kernel/scftorture.c | 9 +-- kernel/smp.c | 81 +++++++++++++++++++------ 4 files changed, 125 insertions(+), 106 deletions(-) --=20 2.20.1