From: John Stultz <john.stultz@linaro.org>
To: lkml <linux-kernel@vger.kernel.org>
Cc: Tom Hromatka <tom.hromatka@oracle.com>,
Ingo Molnar <mingo@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Daniel Lezcano <daniel.lezcano@linaro.org>,
Richard Cochran <richardcochran@gmail.com>,
Prarit Bhargava <prarit@redhat.com>,
Stephen Boyd <sboyd@codeaurora.org>,
John Stultz <john.stultz@linaro.org>
Subject: [PATCH 9/9] sysrq: Reset the watchdog timers while displaying high-resolution timers
Date: Thu, 30 Mar 2017 14:01:24 -0700 [thread overview]
Message-ID: <1490907684-11186-10-git-send-email-john.stultz@linaro.org> (raw)
In-Reply-To: <1490907684-11186-1-git-send-email-john.stultz@linaro.org>
From: Tom Hromatka <tom.hromatka@oracle.com>
On systems with a large number of CPUs, running sysrq-<q> can cause
watchdog timeouts. There are two slow sections of code in the sysrq-<q>
path in timer_list.c.
1. print_active_timers() - This function is called by print_cpu() and
contains a slow goto loop. On a machine with hundreds of CPUs, this
loop took approximately 100ms for the first CPU in a NUMA node.
(Subsequent CPUs in the same node ran much quicker.) The total time
to print all of the CPUs is ultimately long enough to trigger the
soft lockup watchdog.
2. print_tickdevice() - This function outputs a large amount of textual
information. This function also took approximately 100ms per CPU.
Since sysrq-<q> is not a performance critical path, there should be no
harm in touching the nmi watchdog in both slow sections above. Touching
it in just one location was insufficient on systems with hundreds of
CPUs as occasional timeouts were still observed during testing.
This issue was observed on an Oracle T7 machine with 128 CPUs, but I
anticipate it may affect other systems with similarly large numbers of
CPUs.
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Prarit Bhargava <prarit@redhat.com>
Cc: Stephen Boyd <sboyd@codeaurora.org>
Signed-off-by: Tom Hromatka <tom.hromatka@oracle.com>
Reviewed-by: Rob Gardner <rob.gardner@oracle.com>
Signed-off-by: John Stultz <john.stultz@linaro.org>
---
kernel/time/timer_list.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/kernel/time/timer_list.c b/kernel/time/timer_list.c
index ff8d5c1..0e7f542 100644
--- a/kernel/time/timer_list.c
+++ b/kernel/time/timer_list.c
@@ -16,6 +16,7 @@
#include <linux/sched.h>
#include <linux/seq_file.h>
#include <linux/kallsyms.h>
+#include <linux/nmi.h>
#include <linux/uaccess.h>
@@ -86,6 +87,9 @@ print_active_timers(struct seq_file *m, struct hrtimer_clock_base *base,
next_one:
i = 0;
+
+ touch_nmi_watchdog();
+
raw_spin_lock_irqsave(&base->cpu_base->lock, flags);
curr = timerqueue_getnext(&base->active);
@@ -197,6 +201,8 @@ print_tickdevice(struct seq_file *m, struct tick_device *td, int cpu)
{
struct clock_event_device *dev = td->evtdev;
+ touch_nmi_watchdog();
+
SEQ_printf(m, "Tick Device: mode: %d\n", td->mode);
if (cpu < 0)
SEQ_printf(m, "Broadcast device\n");
--
2.7.4
prev parent reply other threads:[~2017-03-30 21:01 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-30 21:01 [GIT PULL][PATCH 0/9] Timekeeping changes for 4.12 John Stultz
2017-03-30 21:01 ` [PATCH 1/9] clocksource: sh_cmt: Compute rate before registration again John Stultz
2017-03-30 21:01 ` [PATCH 2/9] clocksource: sh_tmu: " John Stultz
2017-03-30 21:01 ` [PATCH 3/9] clocksource: em_sti: Split clock prepare and enable steps John Stultz
2017-03-30 21:01 ` [PATCH 4/9] clocksource: em_sti: Compute rate before registration John Stultz
2017-03-30 21:01 ` [PATCH 5/9] clocksource: h8300_timer8: Don't reset rate in ->set_state_oneshot() John Stultz
2017-03-30 21:01 ` [PATCH 6/9] clockevents: Make clockevents_config() static John Stultz
2017-03-30 21:01 ` [PATCH 7/9] MAINTAINERS: Add Stephen Boyd as timekeeping reviewer John Stultz
2017-03-30 21:01 ` [PATCH 8/9] timers, sched_clock: Update timeout for clock wrap John Stultz
2017-03-30 21:01 ` John Stultz [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1490907684-11186-10-git-send-email-john.stultz@linaro.org \
--to=john.stultz@linaro.org \
--cc=daniel.lezcano@linaro.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=prarit@redhat.com \
--cc=richardcochran@gmail.com \
--cc=sboyd@codeaurora.org \
--cc=tglx@linutronix.de \
--cc=tom.hromatka@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®