From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from va-1-113.ptr.blmpb.com (va-1-113.ptr.blmpb.com [209.127.230.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16EFD3DCD9D for ; Wed, 20 May 2026 12:45:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.113 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779281131; cv=none; b=KWoV0gKVqVxxkEhY99WptZ1FfJNuQeOKqXdtSiK1cCbAzyN5z0q7K9WIgqmW4+HdrfNxkqc29F02/7NJ4QDMryP0GI/5AX2WnJaUbJRrZQsqSRHAmoK9p7m/Y0CWSqBc6jTf6YqirUeTmpOnbDBULSNLvs2dYDAe1XWEuqp480E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779281131; c=relaxed/simple; bh=oY+V/gq09xVifFjBP4CDM8Thw4H6ELJzL8GYxKrInDM=; h=Subject:Message-Id:To:Cc:From:Date:Content-Type:Mime-Version: References:In-Reply-To; b=kBZx6SvY1Tq5b7OygSfDEa6arYW1Wcvjntc6RDQ9e4+Iph2itw+FfCkxqNWNSjCwrv1+ajmvdNS+WlnMQeS2NvHIMeSNi9kJvxfpsJP0bcLyi7EJhiP7YbVvjDp//XTNysUpNdr6J4nKYFkT3u7jJJ3TOTWhKqzQIOV4CSbsmmg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Gzpvt6ww; arc=none smtp.client-ip=209.127.230.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Gzpvt6ww" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1779281117; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=q1T0Y68HLcTjep2TVHcJO7PhKeC/HTIcxWdiWcVHVYc=; b=Gzpvt6wwhSX1NSqxq/Mn569gCPU9QDFxaAoeB4w0FvwYkd7L3UtCNqvBW1ISLa62j75P3e fY+e3C64Qw7ZWvQAB4wVsHcN51hCETcynoHj9M4FnUylqmTfuoWK6clqX0kfMHrxARMcpP XgA9Db8y04NU1biBxxLpvFHU4j1iFwTCSxLOpYgGlXNxn8zVUdHFmHsBhkk/WDvblK9bHJ 6cV2RGc7QlxTG2Hn3WXDO/JMx6PMzIOoqnTp0iNVx/JgAWTfgE7CY0qI70Vpm6Ism15L42 NZK9wX18PH1HuyU/tuPLxKqOFbNICa9dnASUDDSxM2eJWbng5yYMa+7BxVOsMA== Subject: Re: [RESEND PATCH v5 00/12] Allow preemption during IPI completion waiting to improve real-time performance Message-Id: <71e3cdb3-ec6a-4b88-b18c-b1f383a5e8e0@bytedance.com> To: , , , , , , , , , , , , Cc: From: "Chuyi Zhou" Date: Wed, 20 May 2026 20:44:54 +0800 User-Agent: Mozilla Thunderbird Content-Type: text/plain; charset=UTF-8 X-Original-From: Chuyi Zhou X-Lms-Return-Path: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260513124524.2569867-1-zhouchuyi@bytedance.com> In-Reply-To: <20260513124524.2569867-1-zhouchuyi@bytedance.com> Content-Transfer-Encoding: quoted-printable friendly ping... On 2026-05-13 8:45 p.m., Chuyi Zhou wrote: > Changes in v5: > - Replace "smp: Remove get_cpu from smp_call_function_any" with a new > approach that extracts a common __smp_call_function_single() to safel= y > keep the remote CPU selection and IPI dispatch process within a singl= e > preemption-disabled region in [PATCH v5 3/12]. > - Fix a typo in comments (s/cpumask_stack/task_mask/) and remove the > obsolete "Preemption must be disabled" constraint from the kernel-doc > in [PATCH v5 6/12]. > - Adjust the WARN_ON_ONCE() validation condition to avoid a false posit= ive > warning caused by CPU hotplug races when use_cpus_read_lock is false = in > [PATCH v5 9/12]. > - Move the preemptible() check in smp_call_function_many_cond() from > [PATCH v5 4/12] to [PATCH v5 6/12]. > - Rebase to commit 4ac4d6549a65 ("sched: Use trace_call__() to save= a > static branch"). >=20 > Changes in v4: > - Use task-local IPI cpumask rather than on-stack cpumask in > [PATCH v4 4/12] (suggested by sebastian). > - Skip to free csd memory in smpcfd_dead_cpu() to guarantee csd memory > access safety, instead of using RCU mechanism in [PATCH v4 5/12] > (suggested by sebastian). > - Align flush_tlb_info with SMP_CACHE_BYTES to avoid performance > degradation caused by unnecessary cache line movements in [PATCH v4 > 10/12](suggested by sebastian and Nadav). > - Collect Acked-bys and Reviewed-bys. >=20 > Changes in v3: > - Add benchmarks to measure the performance impact of changing > flush_tlb_info to stack variable in [PATCH v3 10/12] (suggested by > peter) > - Adjust the rcu_read_unlock() location in [PATCH v3 5/12] (suggested > by muchun) > - Use raw_smp_processor_id() to prevent warning[1] from > check_preemption_disabled() in [PATCH v3 12/12]. > - Collect Acked-bys and Reviewed-by. >=20 > [1]: https://lore.kernel.org/lkml/20260302075216.2170675-1-zhouchuyi@byte= dance.com/T/#mc39999cbeb3f50be176f0903d0fa4075688b073d >=20 > Changes in v2: > - Simplify the code comments in [PATCH v2 2/12] (pointed by peter and > muchun) > - Adjust the preemption disabling logic in smp_call_function_any() in > [PATCH v2 3/12] (suggested by peter). > - Use on-stack cpumask only when !CONFIG_CPUMASK_OFFSTACK in [PATCH V2 > 4/12] (pointed by peter) > - Add [PATCH v2 5/12] to replace migrate_disable with the rcu mechanism > - Adjust the preemption disabling logic to allow flush_tlb_multi() to b= e > preemptible and migratable in [PATCH v2 11/12] > - Collect Acked-bys and Reviewed-bys >=20 > Introduction > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > The vast majority of smp_call_function*() callers block until remote CPUs > complete the IPI function execution. As smp_call_function*() runs with > preemption disabled throughout, scheduling latency increases dramatically > with the number of remote CPUs and other factors (such as interrupts bein= g > disabled). >=20 > On x86-64 architectures, TLB flushes are performed via IPIs; thus, during > process exit or when process-mapped pages are reclaimed, numerous IPI > operations must be awaited, leading to increased scheduling latency for > other threads on the current CPU. In our production environment, we > observed IPI wait-induced scheduling latency reaching up to 16ms on a > 16-core machine. Our goal is to allow preemption during IPI completion > waiting to improve real-time performance. >=20 > Background > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > In our production environments, latency-sensitive workloads (DPDK) are > configured with the highest priority to preempt lower-priority tasks at a= ny > time. We discovered that DPDK's wake-up latency is primarily caused by th= e > current CPU having preemption disabled. Therefore, we collected the maxim= um > preemption disabled events within every 30-second interval and then > calculated the P50/P99 of these max preemption disabled events: >=20 >=20 > p50(ns) p99(ns) > cpu0 254956 5465050 > cpu1 115801 120782 > cpu2 43324 72957 > cpu3 256637 16723307 > cpu4 58979 87237 > cpu5 47464 79815 > cpu6 48881 81371 > cpu7 52263 82294 > cpu8 263555 4657713 > cpu9 44935 73962 > cpu10 37659 65026 > cpu11 257008 2706878 > cpu12 49669 90006 > cpu13 45186 74666 > cpu14 60705 83866 > cpu15 51311 86885 >=20 > Meanwhile, we have collected the distribution of preemption disabling > events exceeding 1ms across different CPUs over several hours(I omitted > CPU data that were all zeros): >=20 > CPU 1~10ms 10~50ms 50~100ms > cpu0 29 5 0 > cpu3 38 13 0 > cpu8 34 6 0 > cpu11 24 10 0 >=20 > The preemption disabled for several milliseconds or even 10ms+ mostly > originates from TLB flush: >=20 > @stack[ > trace_preempt_on+143 > trace_preempt_on+143 > preempt_count_sub+67 > arch_tlbbatch_flush/flush_tlb_mm_range > task_exit/page_reclaim/... > ] >=20 > Further analysis confirms that the majority of the time is consumed in > csd_lock_wait(). >=20 > Now smp_call*() always needs to disable preemption, mainly to protect its > internal per=E2=80=91CPU data structures and synchronize with CPU offline > operations. This patchset attempts to make csd_lock_wait() preemptible, > thereby reducing the preemption=E2=80=91disabled critical section and imp= roving > kernel real=E2=80=91time performance. >=20 > Effect >=20 > =3D=3D=3D=3D=3D=3D >=20 > After applying this patchset, we no longer observe preemption disabled fo= r > more than 1ms on the arch_tlbbatch_flush/flush_tlb_mm_range path. The > overall P99 of max preemption disabled events in every 30-second is > reduced to around 1.5ms (the remaining latency is primarily due to lock > contention. >=20 > before patch after patch reduced by > ----------- -------------- ------------ > p99(ns) 16723307 1556034 ~90.70% >=20 > Chuyi Zhou (12): > smp: Disable preemption explicitly in __csd_lock_wait > smp: Enable preemption early in smp_call_function_single > smp: Refactor remote CPU selection in smp_call_function_any() > smp: Use task-local IPI cpumask in smp_call_function_many_cond() > smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once > smp: Enable preemption early in smp_call_function_many_cond > smp: Remove preempt_disable from smp_call_function > smp: Remove preempt_disable from on_each_cpu_cond_mask > scftorture: Remove preempt_disable in scftorture_invoke_one > x86/mm: Move flush_tlb_info back to the stack > x86/mm: Enable preemption during native_flush_tlb_multi > x86/mm: Enable preemption during flush_tlb_kernel_range >=20 > arch/x86/include/asm/tlbflush.h | 8 +- > arch/x86/kernel/kvm.c | 4 +- > arch/x86/mm/tlb.c | 86 ++++++----------- > include/linux/sched.h | 6 ++ > include/linux/smp.h | 20 ++++ > kernel/fork.c | 9 +- > kernel/scftorture.c | 13 +-- > kernel/smp.c | 157 ++++++++++++++++++++++++-------- > 8 files changed, 195 insertions(+), 108 deletions(-) >