From: Steven Rostedt <rostedt@goodmis.org>
To: linux-kernel@vger.kernel.org
Cc: Ingo Molnar <mingo@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Jon Masters <jcm@redhat.com>
Subject: [for-next][PATCH 5/7] tracing: Add documentation for hwlat_detector tracer
Date: Tue, 06 Sep 2016 09:39:37 -0400 [thread overview]
Message-ID: <20160906134000.810139578@goodmis.org> (raw)
In-Reply-To: <20160906133932.473440583@goodmis.org>
[-- Attachment #1: 0005-tracing-Add-documentation-for-hwlat_detector-tracer.patch --]
[-- Type: text/plain, Size: 4251 bytes --]
From: Jon Masters <jcm@redhat.com>
Added the documentation on how to use th hwlat_detector.
Signed-off-by: Jon Masters <jcm@redhat.com>
[ Various updates and modified to show hwlat as a tracer ]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
Documentation/trace/hwlat_detector.txt | 73 ++++++++++++++++++++++++++++++++++
1 file changed, 73 insertions(+)
create mode 100644 Documentation/trace/hwlat_detector.txt
diff --git a/Documentation/trace/hwlat_detector.txt b/Documentation/trace/hwlat_detector.txt
new file mode 100644
index 000000000000..c02e8ef800cf
--- /dev/null
+++ b/Documentation/trace/hwlat_detector.txt
@@ -0,0 +1,73 @@
+Introduction:
+-------------
+
+The tracer hwlat_detector is a special purpose tracer that is used to
+detect large system latencies induced by the behavior of certain underlying
+hardware or firmware, independent of Linux itself. The code was developed
+originally to detect SMIs (System Management Interrupts) on x86 systems,
+however there is nothing x86 specific about this patchset. It was
+originally written for use by the "RT" patch since the Real Time
+kernel is highly latency sensitive.
+
+SMIs are not serviced by the Linux kernel, which means that it does not
+even know that they are occuring. SMIs are instead set up by BIOS code
+and are serviced by BIOS code, usually for "critical" events such as
+management of thermal sensors and fans. Sometimes though, SMIs are used for
+other tasks and those tasks can spend an inordinate amount of time in the
+handler (sometimes measured in milliseconds). Obviously this is a problem if
+you are trying to keep event service latencies down in the microsecond range.
+
+The hardware latency detector works by hogging one of the cpus for configurable
+amounts of time (with interrupts disabled), polling the CPU Time Stamp Counter
+for some period, then looking for gaps in the TSC data. Any gap indicates a
+time when the polling was interrupted and since the interrupts are disabled,
+the only thing that could do that would be an SMI or other hardware hiccup
+(or an NMI, but those can be tracked).
+
+Note that the hwlat detector should *NEVER* be used in a production environment.
+It is intended to be run manually to determine if the hardware platform has a
+problem with long system firmware service routines.
+
+Usage:
+------
+
+Write the ASCII text "hwlat" into the current_tracer file of the tracing system
+(mounted at /sys/kernel/tracing or /sys/kernel/tracing). It is possible to
+redefine the threshold in microseconds (us) above which latency spikes will
+be taken into account.
+
+Example:
+
+ # echo hwlat > /sys/kernel/tracing/current_tracer
+ # echo 100 > /sys/kernel/tracing/tracing_thresh
+
+The /sys/kernel/tracing/hwlat_detector interface contains the following files:
+
+width - time period to sample with CPUs held (usecs)
+ must be less than the total window size (enforced)
+window - total period of sampling, width being inside (usecs)
+
+By default the width is set to 500,000 and window to 1,000,000, meaning that
+for every 1,000,000 usecs (1s) the hwlat detector will spin for 500,000 usecs
+(0.5s). If tracing_thresh contains zero when hwlat tracer is enabled, it will
+change to a default of 10 usecs. If any latencies that exceed the threshold is
+observed then the data will be written to the tracing ring buffer.
+
+The minimum sleep time between periods is 1 millisecond. Even if width
+is less than 1 millisecond apart from window, to allow the system to not
+be totally starved.
+
+If tracing_thresh was zero when hwlat detector was started, it will be set
+back to zero if another tracer is loaded. Note, the last value in
+tracing_thresh that hwlat detector had will be saved and this value will
+be restored in tracing_thresh if it is still zero when hwlat detector is
+started again.
+
+The following tracing directory files are used by the hwlat_detector:
+
+in /sys/kernel/tracing:
+
+ tracing_threshold - minimum latency value to be considered (usecs)
+ tracing_max_latency - maximum hardware latency actually observed (usecs)
+ hwlat_detector/width - specified amount of time to spin within window (usecs)
+ hwlat_detector/window - amount of time between (width) runs (usecs)
--
2.8.1
next prev parent reply other threads:[~2016-09-06 13:41 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-09-06 13:39 [for-next][PATCH 0/7] tracing: Updates for 4.9 Steven Rostedt
2016-09-06 13:39 ` [for-next][PATCH 1/7] tracing/uprobe: Drop isdigit() check in create_trace_uprobe Steven Rostedt
2016-09-06 13:39 ` [for-next][PATCH 2/7] function_graph: Handle TRACE_BPUTS in print_graph_comment Steven Rostedt
2016-09-06 13:39 ` [for-next][PATCH 3/7] ftrace: Access ret_stack->subtime only in the function profiler Steven Rostedt
2016-09-06 13:39 ` [for-next][PATCH 4/7] tracing: Added hardware latency tracer Steven Rostedt
2016-09-06 13:39 ` Steven Rostedt [this message]
2016-09-06 13:39 ` [for-next][PATCH 6/7] tracing: Have hwlat trace migrate across tracing_cpumask CPUs Steven Rostedt
2016-09-06 13:39 ` [for-next][PATCH 7/7] tracing: Add NMI tracing in hwlat detector Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160906134000.810139578@goodmis.org \
--to=rostedt@goodmis.org \
--cc=akpm@linux-foundation.org \
--cc=jcm@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome