From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from va-1-114.ptr.blmpb.com (va-1-114.ptr.blmpb.com [209.127.230.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D751F43E9CB for ; Wed, 3 Jun 2026 13:21:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.230.114 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780492898; cv=none; b=YEzNstDKtTIBB6kR1hRhmoyT5pEzwFLW/olF7jHbGpBdLnmSyZwUzyvdx6jegN4dGr/9vzgrPSvDm/okdSd5p1NbwpulFCKDfkoMHcIjpmRIkIoxTSY+3GmYE4gste+KchciTKP5xhQu2nhc5Hq+xV01jBHAIsIKZLSs1ajzauw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780492898; c=relaxed/simple; bh=bdJGLu8KZlWqZ9ez43L79RKNPxKprkuqUDnWSL2DdEs=; h=To:Message-Id:In-Reply-To:From:Subject:Date:Cc:Mime-Version: References:Content-Type; b=Hpu2nkGMs8r4Ah21uGyp2h+Eb2++uBru6tekAYOMSCCF1Hml5/OTL+cTxG+IBd3Lh0zRj8NJ8pK207igPZCjRpmQLIzcByQNsAVa3852qPbKvuRYbT3V6758Utn8ty9z2lVk63efreAbxyNPHPXKfPYJz/jYPi3y2CiepEwHxgU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=M63ZwhoF; arc=none smtp.client-ip=209.127.230.114 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="M63ZwhoF" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=2212171451; d=bytedance.com; t=1780492888; h=from:subject: mime-version:from:date:message-id:subject:to:cc:reply-to:content-type: mime-version:in-reply-to:message-id; bh=aVapjvPGMvYUsBRJsO/89prIeC4V8NNDtO3BWpQkB/k=; b=M63ZwhoFwRXc4qRYMdwWRwUFmXnIiB0iQfxs4GcdrqnNbdkoLYHWPsLGf4aVfoqWKjX5UZ 7m7s5/0jTIMm700313x98aXosPdFa2Yir1TNdm/Z40a6wTzJvTQuhnRex2/OYdpkn+Bszp x9G96CtCLwlGYmzNd3X69ANKO4QGxh85eZOqGrkcMZ92/y0nXUSyc8NxVxOo7WXUcl5/tM NbyGwap2NO2PxdXCeZ7NGlQtfKVxh96tytj5nTlX+H+HNQ/L8S+3qEVxrutEWvOh0HKpBj xE56trhsowLkLyPVS3jKX+XxV10wt4j151ZOCubdyZ6ISkfX87104xlzjUNS4w== To: Message-Id: In-Reply-To: <20260523042703.862592-1-zhouchuyi@bytedance.com> From: "Chuyi Zhou" Subject: Re: [RESEND PATCH] smp: Avoid invalid per-CPU CSD lookup with CSD lock debug Date: Wed, 3 Jun 2026 21:21:10 +0800 X-Lms-Return-Path: User-Agent: Mozilla Thunderbird Cc: , , , , , , , Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260523042703.862592-1-zhouchuyi@bytedance.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=UTF-8 X-Original-From: Chuyi Zhou Hi Paul, On 2026-05-23 12:27 p.m., Chuyi Zhou wrote: > Commit b0473dcd4b1d ("smp: Improve smp_call_function_single() > CSD-lock diagnostics") made smp_call_function_single() use the destination > CPU's csd_data when CSD lock debugging is enabled. That lets the debug code > associate a stuck CSD lock with the target CPU, but it also means the CPU > argument is used in per_cpu_ptr() before generic_exec_single() has a chance > to validate it. > > This becomes unsafe when smp_call_function_any() cannot find an online CPU > in the supplied mask. In that case the selected CPU can be nr_cpu_ids, and > the !wait path calls get_single_csd_data(cpu) before generic_exec_single() > returns -ENXIO. With csdlock_debug_enabled set, that indexes the per-CPU > offset array with an invalid CPU number. > > Use the destination CPU's csd_data only when the CPU number is within > nr_cpu_ids. For invalid CPU numbers, fall back to the local CPU's csd_data > and let generic_exec_single() perform the existing validation and return > -ENXIO. > > Fixes: b0473dcd4b1d ("smp: Improve smp_call_function_single() CSD-lock diagnostics") > Signed-off-by: Chuyi Zhou > Reviewed-by: Paul E. McKenney I hit a soft-lockup panic while running scftorture on a large x86_64 SMP machine with a kernel based on origin/sched/core. The issue appears to be a regression from: b0473dcd4b1d ("smp: Improve smp_call_function_single() CSD-lock diagnostics") The reproduction environment was: base: origin/sched/core machine: large x86_64 SMP system; CONFIG_CSD_LOCK_WAIT_DEBUG=y The workload was: modprobe scftorture nthreads=1024 onoff_holdoff=50 onoff_interval=100 The visible panic was a secondary soft lockup. CPU 78 was exiting a process and got stuck waiting for CPU 81 to run a synchronous TLB shootdown callback: watchdog: BUG: soft lockup - CPU#78 stuck for 30s! [nc:13732] Kernel panic - not syncing: softlockup: hung tasks smp: csd: Detected non-responsive CSD lock (#118) on CPU#78, waiting 5000000032 ns for CPU#81 flush_tlb_func+0x0/0x290(...) smp: csd: Re-sending CSD lock (#118) IPI from CPU#78 to CPU#81 CPU 78's stack was: csd_lock_wait_toolong smp_call_function_many_cond on_each_cpu_cond_mask flush_tlb_mm_range tlb_finish_mmu exit_mmap __mmput do_exit do_group_exit __x64_sys_exit_group CPU 81 was running a scftorture invoker and was already inside the call-function-single interrupt path: CPU: 81 PID: 13823 Comm: scftorture_invo RIP: __flush_smp_call_function_queue+0x70/0x4d0 __sysvec_call_function_single sysvec_call_function_single asm_sysvec_call_function_single ktime_get_mono_fast_ns csd_lock_wait_toolong smp_call_function_single scftorture_invoker The dump showed CPU 81 scanning a corrupted async scftorture CSD: ff157dfa7fe72e40: ff157dfa7fe72e40 0051007000000001 func = scf_handler_1 [scftorture] Decoding that CSD: u_flags = 0x1 -> CSD_FLAG_LOCK | CSD_TYPE_ASYNC src = 112 dst = 81 The first word is the llist next pointer, and it points back to the CSD itself: csd->node.llist.next == csd That self-loop explains why CPU 81 made no progress in __flush_smp_call_function_queue(). There was later work still pending on CPU 81's live call_single_queue: call_single_queue[81].first = ff157dfa7fe79c00 That queued entry was CPU 78's synchronous TLB callback: u_flags = 0x11 -> CSD_FLAG_LOCK | CSD_TYPE_SYNC src = 78 dst = 81 func = flush_tlb_func So CPU 78's TLB shootdown CSD was queued, but CPU 81 was already stuck in an earlier invocation of __flush_smp_call_function_queue(). The root cause seems to be that b0473dcd4b1d changes the !wait path of smp_call_function_single() to use the destination CPU's csd_data when csdlock_debug_enabled is set: if (static_branch_unlikely(&csdlock_debug_enabled)) return per_cpu_ptr(&csd_data, cpu); return this_cpu_ptr(&csd_data); The important invariant is that the internal smp-call CSDs are normally single-writer objects. With the old this_cpu_ptr(&csd_data) allocation, smp_call_function_single(..., wait = 0) used the current CPU's csd_data, so each sending CPU had its own CSD. smp_call_function() goes through smp_call_function_many_cond(), and that path also starts from the current CPU's cfd_data: cfd = this_cpu_ptr(&cfd_data); csd = per_cpu_ptr(cfd->csd, cpu); get_cpu() keeps preemption disabled while the CSD is prepared and queued, so another task on the same CPU cannot concurrently reuse that sender's cfd/csd storage either. This is why the non-atomic csd_lock() worked before: the target CPU could still observe and unlock the CSD from the previous request, but there was only one CPU that could prepare and enqueue that particular CSD object. With the debug-mode destination-CPU allocation, all CPUs sending async single calls to CPU 81 contend for the same object: csd_data[81] == ff157dfa7fe72e40 However, csd_lock() is not an atomic test-and-set lock: csd_lock_wait(csd); csd->node.u_flags |= CSD_FLAG_LOCK; That is safe under the previous single-writer rule, but it is not safe once many remote CPUs can acquire the same destination CPU CSD. Two senders can both observe the lock clear, both set CSD_FLAG_LOCK, overwrite func/info/src/dst and then both enqueue the same llist node: llist_add(&csd->node.llist, &per_cpu(call_single_queue, 81)) If the same node is added again while it is already the queue head, llist_add() writes the old head into new->next. Since old head == new, this produces: csd->node.llist.next = csd That exactly matches the dump. I see two possible fixes: 1. Revert b0473dcd4b1d. This restores the old source-CPU csd_data allocation and therefore restores the single-writer property, at the cost of losing the improved destination-CPU CSD lock diagnostics. 2. Keep the destination-CPU csd_data behavior for CSD lock debugging, but make csd_lock() an atomic test-and-set lock when csdlock_debug_enabled is set. One possible implementation is: kernel/smp.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/kernel/smp.c b/kernel/smp.c index 15799f842746..e1d2fd3c9d9f 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -362,6 +362,34 @@ static __always_inline void csd_lock_wait(call_single_data_t *csd) } #endif +#ifdef CONFIG_CSD_LOCK_WAIT_DEBUG +static __always_inline void csd_lock(call_single_data_t *csd) +{ + if (static_branch_unlikely(&csdlock_debug_enabled)) { + unsigned int flags; + + for (;;) { + __csd_lock_wait(csd); + flags = READ_ONCE(csd->node.u_flags); + if (!(flags & CSD_FLAG_LOCK) && + try_cmpxchg_acquire(&csd->node.u_flags, &flags, + flags | CSD_FLAG_LOCK)) + break; + cpu_relax(); + } + } else { + csd_lock_wait(csd); + csd->node.u_flags |= CSD_FLAG_LOCK; + } + + /* + * prevent CPU from reordering the above assignment + * to ->flags with any subsequent assignments to other + * fields of the specified call_single_data_t structure: + */ + smp_wmb(); +} +#else static __always_inline void csd_lock(call_single_data_t *csd) { csd_lock_wait(csd); @@ -374,6 +402,7 @@ static __always_inline void csd_lock(call_single_data_t *csd) */ smp_wmb(); } +#endif static __always_inline void csd_unlock(call_single_data_t *csd) { --