From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from latitanza.investici.org (latitanza.investici.org [185.218.207.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36C07248867 for ; Sat, 25 Jul 2026 19:04:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=185.218.207.228 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785006287; cv=none; b=Rh9JZwO8+zV4txrrj73dTQWuQdL3Bhtmy1iaNT0BHVkJ6vHLBBFbNKaS9AGjeX7vp1ERuuzJ9F+cj13rQ/ZsPlD+xQ6DTO+iUuwonChiCm8/EOE+1dqoQBm48W7BkS9k2an2TDeC+uYq5ypfe6ju4Ihs3DqwW8bxj5SzQc+vkgw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785006287; c=relaxed/simple; bh=jfiL2VCglRmgUyFSkdkS+gjmWY76FSE7s/1ZsR87v0o=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=hMKik7Lc4Cv231q4BVfP3oEiQnm1wlC+3fWkzj3pJItYdOUBe5rzMzmvnvgB6VQAY5GmO5O0Xb80M/yJ1WmR4nPK85q+G5gWEZ3dfZbiGcQMcwaAresj/dFf15Sh7r8IfVlxLEE8oWPbK6mjpmzM0BJ1D5tmO41FnYrLXGJpD2c= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=grrlz.net; spf=pass smtp.mailfrom=grrlz.net; dkim=pass (1024-bit key) header.d=grrlz.net header.i=@grrlz.net header.b=KV00JULR; arc=none smtp.client-ip=185.218.207.228 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=grrlz.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=grrlz.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=grrlz.net header.i=@grrlz.net header.b="KV00JULR" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=grrlz.net; s=stigmate; t=1785006275; bh=5Ut4bg/BM6r5bsYWE6cae8c7OFDbSJ32qADk6nDw55c=; h=From:To:Cc:Subject:Date:From; b=KV00JULRU8hSlrHnKVS5AHhtCSFmcuXnjj7Mrw2ELygxB8a/ZbeJ57b8wOOsa/DSg mJcD9vLXY6rtcCJY3/ktKbSM4J7MnhC+EJUGLkb2m5fp8QPTYMJdkLjDHSwUa5LH9i 0dU/HksGlYCt0rT2Z2GRvJa8DphqbHkoWOf/MR0Q= Received: from mx3.investici.org (unknown [127.0.0.1]) by latitanza.investici.org (Postfix) with ESMTP id 4h6vRl6bKmzGpVQ; Sat, 25 Jul 2026 19:04:35 +0000 (UTC) Received: by mx3.investici.org (Postfix) id 4h6vRl4XMHzGpVP; Sat, 25 Jul 2026 19:04:35 +0000 (UTC) From: Bradley Morgan To: Andrew Morton Cc: linux-kernel@vger.kernel.org, Bradley Morgan Subject: [PATCH] stop_machine: Warn when a stop state stalls Date: Sat, 25 Jul 2026 19:04:33 +0000 Message-ID: <20260725190433.13571-1-include@grrlz.net> X-Mailer: git-send-email 2.47.3 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit multi_cpu_stop() feeds the NMI watchdog and rcu_momentary_eqs() on every spin. so when a CPU never acks a state transition the machine hangs, and the two detectors that wouldve caught it are the ones being fed. no tick either, irqs are off everywhere, hung task is blind too. the hang logs exactly nothing. Stamp every state transition in set_state() and let waiters yell once after ten seconds stuck in the same state. every waiter reports itself, the culprit is the CPU missing from the output. its local_clock() not jiffies, jiffies is dead once everyone has irqs off. One branch in a path that already pokes two watchdogs per spin. nothing changes while stop_machine() behaves. Signed-off-by: Bradley Morgan --- kernel/stop_machine.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c index 773d8e9ae30c..1e1d1d46c24d 100644 --- a/kernel/stop_machine.c +++ b/kernel/stop_machine.c @@ -15,12 +15,14 @@ #include #include #include +#include #include #include #include #include #include #include +#include #include /* @@ -174,6 +176,7 @@ struct multi_stop_data { enum multi_stop_state state; atomic_t thread_ack; + u64 tstamp; }; static void set_state(struct multi_stop_data *msdata, @@ -181,6 +184,7 @@ static void set_state(struct multi_stop_data *msdata, { /* Reset ack counter. */ atomic_set(&msdata->thread_ack, msdata->num_threads); + WRITE_ONCE(msdata->tstamp, local_clock()); smp_wmb(); WRITE_ONCE(msdata->state, newstate); } @@ -197,6 +201,9 @@ notrace void __weak stop_machine_yield(const struct cpumask *cpumask) cpu_relax(); } +/* how long a state may sit unacked before waiting CPUs complain */ +#define MULTI_STOP_STUCK_NS (10ULL * NSEC_PER_SEC) + /* This is the cpu_stop function which stops the CPU. */ static int multi_cpu_stop(void *data) { @@ -205,7 +212,7 @@ static int multi_cpu_stop(void *data) int cpu = smp_processor_id(), err = 0; const struct cpumask *cpumask; unsigned long flags; - bool is_active; + bool is_active, stuck = false; /* * When called from stop_machine_from_inactive_cpu(), irq might @@ -250,6 +257,18 @@ static int multi_cpu_stop(void *data) touch_nmi_watchdog(); /* Also suppress RCU CPU stall warnings. */ rcu_momentary_eqs(); + /* + * the watchdogs above eat the evidence, a CPU that + * never acks hangs us silently. every waiter yells + * once, the culprit is the one missing from the log. + */ + if (!stuck && + local_clock() - READ_ONCE(msdata->tstamp) > MULTI_STOP_STUCK_NS) { + stuck = true; + pr_warn("%s: CPU %d stuck in state %d (%ps), %d acks missing\n", + __func__, cpu, curstate, msdata->fn, + atomic_read(&msdata->thread_ack)); + } } } while (curstate != MULTI_STOP_EXIT); -- 2.47.3