From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from va-1-112.ptr.blmpb.com (va-1-112.ptr.blmpb.com [209.127.230.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E68124248C2 for ; Thu, 16 Jul 2026 14:15:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.112 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784211307; cv=none; b=LSgIBiI2620dqUitI7FiylzdvKxRtIWa6xRkx5PY4A3IDd5UyeNjTQXD1rPGZUBGbHm1CIehpzlB/vtT8Ebkwxjduib7VlUxHhYuJ01iTzAU9iv4Goq6nyROjApB658qBvU1Vp3OQr3F8u8z/GpMIumnjQ0kEw3AEIcdtyz7EHQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784211307; c=relaxed/simple; bh=jA73SH7ZQwCHOVMPhCa1qrzjMbaqzW485srwWVrf8bg=; h=Cc:Subject:Message-Id:Mime-Version:In-Reply-To:Content-Type: References:To:From:Date; b=o8eM67ANauZzT3vgVzPnyOERdAVAEN2/eAmFBnzuEtVtSveTkEIonXhhcquCsHtH/0z+SZ6O2qI1w0Pegre2SPKj/B94kc9bNus25co4cgSYYUMq6E3F1C0bwkgt99l7da2kJcqY/bgsPJBx/IqvfnWpUOfcgoWxSaNCTVTJMks= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=htdgXVOC; arc=none smtp.client-ip=209.127.230.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="htdgXVOC" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1784211297; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=9GPM2SNP/HeM+MSGIYgNdx3utYG1a9VC15zFKPxDOp0=; b=htdgXVOCX3YLKIkCF5PbKMqHQeoBNKoBsXSd+WWlgn2W2kvQOdsBPXSNK133hbWQtY0LvR 9e39JQU3ZMijJjnyv0WK0kUc6cc7HBTn0xDt0VpYKmR4a55ZQEb8amIwqEO2UCdrYWNgKS A7T2ypGBHk6jwKwwp8gWKK6SI9zs0lFcsGSTprlSuJXwJ5N55nZF+IOKcZu9+aAu1y/xMi 9o0iMlS3yf3Awe7F/EeyhY3MR0CuRJtQ2YIDX9d0Zm0pnWPrNNIbIfoGzyvxCC2o2S0nhD r60cHtyOkXLGaVBxCk1nQ3kvU6twBLBqXZ1VflKFH3VtO16TePuIWjy6sbGwDA== Cc: Subject: Re: [PATCH v10 00/14] Allow preemption during IPI completion waiting to improve real-time performance Message-Id: <546a1811-929e-4aff-af84-52bf81268d55@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 In-Reply-To: <20260709122933.4021501-1-zhouchuyi@bytedance.com> Content-Transfer-Encoding: 7bit User-Agent: Mozilla Thunderbird Content-Type: text/plain; charset=UTF-8 References: <20260709122933.4021501-1-zhouchuyi@bytedance.com> To: , , , , , , , , , , , , , From: "Chuyi Zhou" Date: Thu, 16 Jul 2026 22:14:32 +0800 X-Lms-Return-Path: X-Original-From: Chuyi Zhou friendly ping :) On 2026-07-09 8:29 p.m., Chuyi Zhou wrote: > Changes in v10: > - Hide the task-local IPI mask storage behind a struct task_ipi_mask > __private field and access its members only through ACCESS_PRIVATE() > in [PATCH v10 4/14] (suggested by Thomas Gleixner). > - Clarify the x86-64 NR_CPUS=8192 memory cost as 1 KiB per task in > [PATCH v10 4/14]. > - Update stale on_each_cpu_cond_mask() kernel-doc in [PATCH v10 8/14] > to avoid implying that the whole call runs with preemption disabled. > > Changes in v9: > - Rewrite changelogs in [PATCH v9 1/14], [PATCH v9 2/14], > [PATCH v9 4/14], [PATCH v9 5/14], [PATCH v9 6/14], > [PATCH v9 7/14], and [PATCH v9 11/14]. > - Change smp_call_function_single()'s wait argument to bool and fix > function/kernel-doc argument alignment in [PATCH v9 3/14]. > - Clean up task_struct IPI mask indentation, declaration ordering, and > related initialization placement in [PATCH v9 4/14]. > - Update the CSD allocation comment in [PATCH v9 5/14]. > - In [PATCH v9 6/14], make put_cpu() unconditional after IPI > queue/dispatch and clarify that the final csd_lock_wait() only waits > for completion. > - Add a comment for FLUSH_TLB_INFO_ALIGN, reference commit 780e0106d468 > directly, and add Sebastian's Reviewed-by tag in [PATCH v9 11/14]. > > Changes in v8: > - Use MIN(SMP_CACHE_BYTES, 64) for FLUSH_TLB_INFO_ALIGN in > [PATCH v8 11/14]. > - In flush_tlb_mm_range(), name the stack object "info" directly instead > of keeping a temporary "_info" plus pointer in [PATCH v8 12/14]. > > Changes in v7: > - Split the x86/mm flush_tlb_info stack-storage change into > [PATCH v7 10/14], [PATCH v7 11/14], and [PATCH v7 12/14]. > - Rewrite the flush_tlb_info changelog with the historical context, test > data, Link tag, and Acked-by tag. > - Add a preparatory KVM patch for kvm_flush_tlb_multi() in > [PATCH v7 13/14]. > - Refactor flush_tlb_mm_range() and arch_tlbbatch_flush() to avoid gotos > in [PATCH v7 14/14]. > - Drop the flush_tlb_kernel_range() preemption optimization for now. > - Tighten the smp_task_ipi_mask_alloc()/smp_task_ipi_mask_free() > declarations in [PATCH v7 4/14]. > - Update the smp_call_function_many_cond() comment wording. > - Add Paul E. McKenney's Tested-by tags and Sebastian's Reviewed-by tag. > > Changes in v6: > - Make the task-local cpumask selection explicit and drop preemptible() > check in smp_call_function_many_cond(). The early put_cpu() decision now > depends only on whether a task-local cpumask is available. > - Keep smp_task_ipi_mask() private to kernel/smp.c in [PATCH v6 4/12]. > - Add #include to kernel/smp.c in [PATCH v6 4/12] for > kmalloc()/kfree(), fixing the kernel test robot build failure reported > at: > https://lore.kernel.org/oe-kbuild-all/202605241101.w6T2LApw-lkp@intel.com/ > - Update the csd_lock_wait() comment in [PATCH v6 6/12]. > - Add Sebastian's Reviewed-by tags to the reviewed patches. > > Changes in v5: > - Replace "smp: Remove get_cpu from smp_call_function_any" with a new > approach that extracts a common __smp_call_function_single() to safely > keep the remote CPU selection and IPI dispatch process within a single > preemption-disabled region in [PATCH v5 3/12]. > - Fix a typo in comments (s/cpumask_stack/task_mask/) and remove the > obsolete "Preemption must be disabled" constraint from the kernel-doc > in [PATCH v5 6/12]. > - Adjust the WARN_ON_ONCE() validation condition to avoid a false positive > warning caused by CPU hotplug races when use_cpus_read_lock is false in > [PATCH v5 9/12]. > - Move the preemptible() check in smp_call_function_many_cond() from > [PATCH v5 4/12] to [PATCH v5 6/12]. > - Rebase to commit 4ac4d6549a65 ("sched: Use trace_call__() to save a > static branch"). > > Changes in v4: > - Use task-local IPI cpumask rather than on-stack cpumask in > [PATCH v4 4/12] (suggested by sebastian). > - Skip to free csd memory in smpcfd_dead_cpu() to guarantee csd memory > access safety, instead of using RCU mechanism in [PATCH v4 5/12] > (suggested by sebastian). > - Align flush_tlb_info with SMP_CACHE_BYTES to avoid performance > degradation caused by unnecessary cache line movements in [PATCH v4 > 10/12](suggested by sebastian and Nadav). > - Collect Acked-bys and Reviewed-bys. > > Changes in v3: > - Add benchmarks to measure the performance impact of changing > flush_tlb_info to stack variable in [PATCH v3 10/12] (suggested by > peter) > - Adjust the rcu_read_unlock() location in [PATCH v3 5/12] (suggested > by muchun) > - Use raw_smp_processor_id() to prevent warning[1] from > check_preemption_disabled() in [PATCH v3 12/12]. > - Collect Acked-bys and Reviewed-by. > > [1]: https://lore.kernel.org/lkml/20260302075216.2170675-1-zhouchuyi@bytedance.com/T/#mc39999cbeb3f50be176f0903d0fa4075688b073d > > Changes in v2: > - Simplify the code comments in [PATCH v2 2/12] (pointed by peter and > muchun) > - Adjust the preemption disabling logic in smp_call_function_any() in > [PATCH v2 3/12] (suggested by peter). > - Use on-stack cpumask only when !CONFIG_CPUMASK_OFFSTACK in [PATCH V2 > 4/12] (pointed by peter) > - Add [PATCH v2 5/12] to replace migrate_disable with the rcu mechanism > - Adjust the preemption disabling logic to allow flush_tlb_multi() to be > preemptible and migratable in [PATCH v2 11/12] > - Collect Acked-bys and Reviewed-bys > > > Introduction > ============ > > The vast majority of smp_call_function*() callers block until remote CPUs > complete the IPI function execution. As smp_call_function*() runs with > preemption disabled throughout, scheduling latency increases dramatically > with the number of remote CPUs and other factors (such as interrupts being > disabled). > > On x86-64 architectures, TLB flushes are performed via IPIs; thus, during > process exit or when process-mapped pages are reclaimed, numerous IPI > operations must be awaited, leading to increased scheduling latency for > other threads on the current CPU. In our production environment, we > observed IPI wait-induced scheduling latency reaching up to 16ms on a > 16-core machine. Our goal is to allow preemption during IPI completion > waiting to improve real-time performance. > > Background > ========== > > In our production environments, latency-sensitive workloads (DPDK) are > configured with the highest priority to preempt lower-priority tasks at any > time. We discovered that DPDK's wake-up latency is primarily caused by the > current CPU having preemption disabled. Therefore, we collected the maximum > preemption disabled events within every 30-second interval and then > calculated the P50/P99 of these max preemption disabled events: > > > p50(ns) p99(ns) > cpu0 254956 5465050 > cpu1 115801 120782 > cpu2 43324 72957 > cpu3 256637 16723307 > cpu4 58979 87237 > cpu5 47464 79815 > cpu6 48881 81371 > cpu7 52263 82294 > cpu8 263555 4657713 > cpu9 44935 73962 > cpu10 37659 65026 > cpu11 257008 2706878 > cpu12 49669 90006 > cpu13 45186 74666 > cpu14 60705 83866 > cpu15 51311 86885 > > Meanwhile, we have collected the distribution of preemption disabling > events exceeding 1ms across different CPUs over several hours(I omitted > CPU data that were all zeros): > > CPU 1~10ms 10~50ms 50~100ms > cpu0 29 5 0 > cpu3 38 13 0 > cpu8 34 6 0 > cpu11 24 10 0 > > The preemption disabled for several milliseconds or even 10ms+ mostly > originates from TLB flush: > > @stack[ > trace_preempt_on+143 > trace_preempt_on+143 > preempt_count_sub+67 > arch_tlbbatch_flush/flush_tlb_mm_range > task_exit/page_reclaim/... > ] > > Further analysis confirms that the majority of the time is consumed in > csd_lock_wait(). > > Now smp_call*() always needs to disable preemption, mainly to protect its > internal per-CPU data structures and synchronize with CPU offline > operations. This patchset attempts to make csd_lock_wait() preemptible, > thereby reducing the preemption-disabled critical section and improving > kernel real-time performance. > > Effect > ====== > > After applying this patchset, we no longer observe preemption disabled for > more than 1ms on the arch_tlbbatch_flush/flush_tlb_mm_range path. The > overall P99 of max preemption disabled events in every 30-second interval > is reduced to around 1.5ms. The remaining latency is primarily due to lock > contention. > > before patch after patch reduced by > ----------- -------------- ------------ > p99(ns) 16723307 1556034 ~90.70% > > > Chuyi Zhou (14): > smp: Disable preemption explicitly in __csd_lock_wait() > smp: Enable preemption early in smp_call_function_single() > smp: Refactor remote CPU selection in smp_call_function_any() > smp: Use task-local IPI cpumask in smp_call_function_many_cond() > smp: Alloc percpu csd data in smpcfd_prepare_cpu() only once > smp: Enable preemption early in smp_call_function_many_cond() > smp: Remove preempt_disable() from smp_call_function() > smp: Remove preempt_disable() from on_each_cpu_cond_mask() > scftorture: Remove preempt_disable() in scftorture_invoke_one() > x86/mm: Factor out flush_tlb_info initialization > x86/mm: Cap flush_tlb_info alignment at 64 bytes > x86/mm: Move flush_tlb_info back to the stack > x86/kvm: Disable preemption in kvm_flush_tlb_multi() > x86/mm: Re-enable preemption before flush_tlb_multi() > > arch/x86/include/asm/tlbflush.h | 9 +- > arch/x86/kernel/kvm.c | 4 +- > arch/x86/mm/tlb.c | 95 ++++++--------- > include/linux/sched.h | 12 ++ > include/linux/smp.h | 15 ++- > kernel/fork.c | 9 +- > kernel/scftorture.c | 13 +-- > kernel/smp.c | 197 ++++++++++++++++++++++---------- > kernel/up.c | 3 +- > 9 files changed, 223 insertions(+), 134 deletions(-)