Re: [PATCH 2/2 v4] watchdog: Printing traces for all cpus on lockup detection

[Andrew Morton <akpm@linux-foundation.org>]

On Wed, 23 Apr 2014 16:40:05 -0400 Don Zickus <dzickus@redhat.com> wrote:

> From: Aaron Tomlin <atomlin@redhat.com>
> 
> A 'softlockup' is defined as a bug that causes the kernel to
> loop in kernel mode for more than a predefined period to
> time, without giving other tasks a chance to run.
> 
> Currently, upon detection of this condition by the per-cpu
> watchdog task, debug information (including a stack trace)
> is sent to the system log.
> 
> On some occasions, we have observed that the "victim" rather
> than the actual "culprit" (i.e. the owner/holder of the
> contended resource) is reported to the user.
> Often this information has proven to be insufficient to
> assist debugging efforts.
> 
> To avoid loss of useful debug information, for architectures
> which support NMI, this patch makes it possible to improve
> soft lockup reporting. This is accomplished by issuing an
> NMI to each cpu to obtain a stack trace.
> 
> If NMI is not supported we just revert back to the old method.
> A sysctl and boot-time parameter is available to toggle this
> feature.
> 
> --- a/include/linux/nmi.h
> +++ b/include/linux/nmi.h
> @@ -57,6 +57,9 @@ int hw_nmi_is_cpu_stuck(struct pt_regs *);
>  u64 hw_nmi_get_sample_period(int watchdog_thresh);
>  extern int watchdog_user_enabled;
>  extern int watchdog_thresh;
> +#ifdef CONFIG_SMP
> +extern int sysctl_softlockup_all_cpu_backtrace;
> +#endif

The ifdefs aren't really needed here.  If we omit them then error
reporting happens at link time rather than at compile time, but that's
a small price to pay for cleaning up the code.

> +		if (softlockup_all_cpu_backtrace) {
> +			/* Prevent multiple soft-lockup reports if one cpu is already
> +			 * engaged in dumping cpu back traces
> +			 */
> +			if (test_and_set_bit(0, &soft_lockup_nmi_warn)) {
> +				/* Someone else will report us. Let's give up */
> +				__this_cpu_write(soft_watchdog_warn, true);
> +				return HRTIMER_RESTART;
> +			}
> +		}

You missed my suggestion here.

   text    data     bss     dec     hex filename
   1519     524      24    2067     813 kernel/watchdog.o-before
   1471     520      16    2007     7d7 kernel/watchdog.o-after


--- a/include/linux/nmi.h~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix
+++ a/include/linux/nmi.h
@@ -57,9 +57,7 @@ int hw_nmi_is_cpu_stuck(struct pt_regs *
 u64 hw_nmi_get_sample_period(int watchdog_thresh);
 extern int watchdog_user_enabled;
 extern int watchdog_thresh;
-#ifdef CONFIG_SMP
 extern int sysctl_softlockup_all_cpu_backtrace;
-#endif
 struct ctl_table;
 extern int proc_dowatchdog(struct ctl_table *, int ,
 			   void __user *, size_t *, loff_t *);
--- a/kernel/watchdog.c~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix
+++ a/kernel/watchdog.c
@@ -31,7 +31,12 @@
 
 int watchdog_user_enabled = 1;
 int __read_mostly watchdog_thresh = 10;
+#ifdef CONFIG_SMP
 int __read_mostly sysctl_softlockup_all_cpu_backtrace;
+#else
+#define sysctl_softlockup_all_cpu_backtrace 0
+#endif
+
 static int __read_mostly watchdog_running;
 static u64 __read_mostly sample_period;
 
_