From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34AB082866 for ; Fri, 22 May 2026 01:46:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.101 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779414393; cv=none; b=mfExAAQ9Nc4NhuJ1nCxZhTdzuQkOzHZphezBva1PpFVR2eMCWDSFGS02DIE3s/gfHjAyHGmXcEIrnclJK8WKQYnS0sWDCxbJVTeDe+xc8pDmzw5qFIdAYzRoBSW8nNQbAR5RWwYYoObpG15tQbrOtfayAmY3GBnn38q9iVcBeIo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779414393; c=relaxed/simple; bh=FH9O3Cn/qFVddalyDa9Pf/eLKEQB/1thOb0XTlUKkCw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Mars95NU6+xlPHn8W+U6+6x5kggCpWQiV63tF6nBttB4mo6/U3af8n5WgE7dEDfbiiPh9p4Y6Y12zepcXLFUdIL9LtPZy1t93mFxVKRYWKQN41SOcrnCzqRWa37ETVJ50U6um2APILo+wgWaTllPzgdlZDKcUsvzCQ08fUhFvBc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=RrbqTsvQ; arc=none smtp.client-ip=115.124.30.101 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="RrbqTsvQ" DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1779414388; h=Date:From:To:Subject:Message-ID:MIME-Version:Content-Type; bh=oFTKo6YSOxiaL3K8gDq60EyZTlhjZbkfwuVgakTqlP8=; b=RrbqTsvQz5cahCgXkmwaOmUFZXgHcR4s9zn4djqoySWXrHoSRPw6M7OxowZAOebZQfzIMoclVEcQd0ale9Ge6t3OUHrl1pGU1JpvwIpNbaTXCHY3LrWlGM9PNefaJerJMChsl41yREQ09maLzWlOXmdVq5W8Tgv7x7Ulw17TXj8= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037033178;MF=feng.tang@linux.alibaba.com;NM=1;PH=DS;RN=5;SR=0;TI=SMTPD_---0X3Mz8GZ_1779414386; Received: from localhost(mailfrom:feng.tang@linux.alibaba.com fp:SMTPD_---0X3Mz8GZ_1779414386 cluster:ay36) by smtp.aliyun-inc.com; Fri, 22 May 2026 09:46:27 +0800 Date: Fri, 22 May 2026 09:46:26 +0800 From: Feng Tang To: Andrew Morton Cc: paulmck@kernel.org, Petr Mladek , linux-kernel@vger.kernel.org Subject: Re: [PATCH] lib/nmi_backtrace: print out the CPUs which fail to respond to NMI Message-ID: References: <20260521030336.92172-1-feng.tang@linux.alibaba.com> <20260521153716.0e6da8366d3e2004e9e7cf97@linux-foundation.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20260521153716.0e6da8366d3e2004e9e7cf97@linux-foundation.org> Hi Andrew, Thanks for the review! On Thu, May 21, 2026 at 03:37:16PM -0700, Andrew Morton wrote: > On Thu, 21 May 2026 11:03:36 +0800 Feng Tang wrote: > > > When debugging RCU stall cases, usually all CPUs will respond to the > > NMI and print out the backtrace. But in some nasty or hardware related > > cases, some CPUs may fail to respond in 10 seconds, and very likely > > this is sign of severe issues. > > > > Paul E. McKenney has implemented the NMI backtrace stall check for x86, > > and for other architectures, it should be also helpful to at least > > print out those CPUs which failed to repond to the NMI, so that users > > can get an early heads-up for possible CPU hard stall. > > That must be one messed up machine. Is this something you've > encountered in real life? Yes. A big parf of my worktime is to play with panic/lockup/rcustall/hung bugs :). And we did see some real case, and if there is such warning, it could have given us good hint to focus on the not-responding CPU. In one case, kernel requested 31 CPUs to do the CPU backtrace, and only 30 CPUs really did, while the left unnoticed CPU is the root cause. > > --- a/lib/nmi_backtrace.c > > +++ b/lib/nmi_backtrace.c > > @@ -75,7 +75,13 @@ void nmi_trigger_cpumask_backtrace(const cpumask_t *mask, > > mdelay(1); > > touch_softlockup_watchdog(); > > } > > - nmi_backtrace_stall_check(to_cpumask(backtrace_mask)); > > + > > + if (!cpumask_empty(to_cpumask(backtrace_mask))) { > > + pr_warn("After 10 seconds, these CPUS still haven't responded to the NMI: %*pbl\n", > > + cpumask_pr_args(to_cpumask(backtrace_mask))); > > + > > + nmi_backtrace_stall_check(to_cpumask(backtrace_mask)); > > + } > > It's a nitpick, but > > : /* Wait for up to 10 seconds for all CPUs to do the backtrace */ > : for (i = 0; i < 10 * 1000; i++) { > : if (cpumask_empty(to_cpumask(backtrace_mask))) > : break; > : mdelay(1); > : touch_softlockup_watchdog(); > : } > : > : if (!cpumask_empty(to_cpumask(backtrace_mask))) { > : pr_warn("After 10 seconds, these CPUS still haven't responded to the NMI: %*pbl\n", > > Here we're hard-coding "10" in two places and in a comment. It would > be nicer to do > > #define FOO_TIMEOUT 10 > > then use that throughout. > > (bonus points for figuring out how to paste that "10" into the > pr_warn() control string rather than using %d!) How about this followon patch? --- diff --git a/lib/nmi_backtrace.c b/lib/nmi_backtrace.c index a113d3d669be..2810b8f478a4 100644 --- a/lib/nmi_backtrace.c +++ b/lib/nmi_backtrace.c @@ -27,6 +27,8 @@ static DECLARE_BITMAP(backtrace_mask, NR_CPUS) __read_mostly; /* "in progress" flag of arch_trigger_cpumask_backtrace */ static unsigned long backtrace_flag; +#define NMI_BT_TIMEOUT_SEC 10 + /* * When raise() is called it will be passed a pointer to the * backtrace_mask. Architectures that call nmi_cpu_backtrace() @@ -68,8 +70,8 @@ void nmi_trigger_cpumask_backtrace(const cpumask_t *mask, raise(to_cpumask(backtrace_mask)); } - /* Wait for up to 10 seconds for all CPUs to do the backtrace */ - for (i = 0; i < 10 * 1000; i++) { + /* Wait for up to NMI_BT_TIMEOUT_SEC seconds for all CPUs to do the backtrace */ + for (i = 0; i < NMI_BT_TIMEOUT_SEC * 1000; i++) { if (cpumask_empty(to_cpumask(backtrace_mask))) break; mdelay(1); @@ -77,8 +79,8 @@ void nmi_trigger_cpumask_backtrace(const cpumask_t *mask, } if (!cpumask_empty(to_cpumask(backtrace_mask))) { - pr_warn("After 10 seconds, these CPUS still haven't responded to the NMI: %*pbl\n", - cpumask_pr_args(to_cpumask(backtrace_mask))); + pr_warn("After %d seconds, these CPUS still haven't responded to the NMI: %*pbl\n", + NMI_BT_TIMEOUT_SEC, cpumask_pr_args(to_cpumask(backtrace_mask))); nmi_backtrace_stall_check(to_cpumask(backtrace_mask)); }