From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756048AbYCCXxd (ORCPT ); Mon, 3 Mar 2008 18:53:33 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1757102AbYCCXwK (ORCPT ); Mon, 3 Mar 2008 18:52:10 -0500 Received: from e5.ny.us.ibm.com ([32.97.182.145]:43876 "EHLO e5.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756767AbYCCXwH (ORCPT ); Mon, 3 Mar 2008 18:52:07 -0500 Subject: [patch 2/3] Trace code and documentation From: "David J. Wilder" To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org Cc: systemtap@sourceware.org Content-Type: text/plain Date: Mon, 03 Mar 2008 15:52:01 -0800 Message-Id: <1204588321.32186.8.camel@lc4eb748232119.ibm.com> Mime-Version: 1.0 X-Mailer: Evolution 2.0.2 (2.0.2-35.0.4.el4) Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Trace - Provides tracing primitives Signed-off-by: Tom Zanussi Signed-off-by: Martin Hunt Signed-off-by: David Wilder --- Documentation/trace.txt | 210 ++++++++++++++++++ include/linux/trace.h | 99 +++++++++ lib/Kconfig | 9 + lib/Makefile | 2 + lib/trace.c | 563 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 883 insertions(+), 0 deletions(-) diff --git a/Documentation/trace.txt b/Documentation/trace.txt new file mode 100644 index 0000000..af9a22e --- /dev/null +++ b/Documentation/trace.txt @@ -0,0 +1,210 @@ +Trace Setup and Control +======================= +In the kernel, the trace interface provides a simple mechanism for +starting and managing data channels (traces) to user space. The +trace interface builds on the relay interface. For a complete +description of the relay interface, please see: +Documentation/filesystems/relay.txt. + +The trace interface provides a single layer in a complete tracing +application. Trace provides a kernel API that can be used for the setup +and control of tracing channels. User of trace must provide a data layer +responsible for formatting and writing data into the trace channels. + +A layered approach to tracing +============================= +A complete kernel tracing application consists of a data provider and +a data consumer. Both provider and consumer contain three layers; each +layer works in tandem with the corresponding layer in the opposite side. +The layers are represented in the following diagram. + +Provider Data layer + Formats raw trace data and provides data-related service. + For example, adding timestamps used by consumer to sort data. + +Provider Control layer + Provided by the trace interface, this layer creates trace channels + and informs the data layer and consumer of the current state + of the trace channels. + +Provider Buffering layer + Provided by relay. This layer buffers data in the + kernel for consumption by the consumer's buffer + layer. + +Provider (in-kernel facility) +----------------------------------------------------------------------------- +Consumer (user application) + + +Consumer Buffer layer + Reads/consumes data from the provider's data buffers. + +Consumer Control layer + Communicates to the provider's control layer to control the state + of the trace channels. + +Consumer Data layer + Sorts and formats data as provided by the provider's data layer. + +The provider is coded as a kernel facility. The consumer is coded as +a user application. + + +Trace - Features +================ +Trace exploits services and features provided by relay. These features +are: +- The creation and destruction of relay channels. +- Buffer management. Overwrite or non-overwrite modes can be selected + as well as global or per-CPU buffering. + +Overwrite mode can be called "flight recorder mode". Flight recorder +mode is selected by setting the TRACE_FLIGHT_CHANNEL flag when +creating trace channels. In flight mode when a tracing buffer is +full, the oldest records in the buffer will be discarded to make room +as new records arrive. In the default non-overwrite mode, new records +may be written only if the buffer has room. In either case, to +prevent data loss, a user space reader must keep the buffers +drained. Trace provides a means to detect the number of records that +have been dropped due to a buffer-full condition (non-overwrite mode +only). + +When per-CPU buffers are used, relay creates one debugfs file for each +running CPU. The user-space consumer of the data is responsible for +reading the per-CPU buffers and collating the records presumably using +a time stamp or sequence number included in the trace records. The +use of global buffers eliminates this extra work of sequencing +records; however the provider's data layer must hold a lock when +writing records. The lock prevents writers running on different CPUs +from overwriting each other's data. However, buffering may be slower +because writes to the buffer are serialized. Global buffering is +selected by setting the TRACE_GLOBAL_CHANNEL flag when creating trace +channels. + +Trace User Interface +=================== +When a trace channel is created and started, the following +directories and files are created in the root of the mounted debugfs. + +/debug (root of the debugfs) + / + / + trace[0...N-1] Per-CPU trace data, one + file per CPU. + + state Start or stop tracing by + by writing the strings + "start" or "stop" to this + file. Read the file to get the + current state. + + dropped The number of records dropped + due to a full-buffer condition, + for non-TRACE_FLIGHT_CHANNELs + only. + + rewind Trigger a rewind by writing + to this file. i.e. start + next read at the beginning + again. Only available for + TRACE_FLIGHT_CHANNELS. + + + nr_sub Number of sub-buffers + in the channel. + + sub_size Size of sub-buffers in + the channel. + +Trace data is gathered from the trace[0...N-1] files using one of the +available interfaces provided by relay. + +When using the read(2) interface, as data is read it is marked as +consumed by the relay subsystem. Therefore, subsequent reads will +only return unconsumed data. + +Trace Kernel API +=============== +An overview of the trace Kernel API is now given. More details of the +API can be found in linux/trace.h. + +The steps a kernel data provider takes to utilize the trace interface are: +1) Set up a trace channel - trace_setup() +2) Start the trace channel - trace_start() +3) Write one or more trace records into the channel (using the relay API). + + Important: When writing a trace record the provider must insure that + preemption is disabled and that trace state is set to "running". A + typical function used to write records into a trace channel should + follow the following semantics: + + rcu_read_lock(); // disables preemption + if (trace_running(trace)){ + relay_write(....); // use any available relay data + // function + } + rcu_read_unlock(); // enables preemption + +4) Stop and start tracing as desired - trace_start()/trace_stop() +5) Destroy the trace channel and underlying relay channel - + trace_cleanup(). + +Kernel Configuration +-------------------- +To use trace, configure your kernel with CONFIG_TRACE=y. Trace depends on +both CONFIG_RELAY and CONFIG_DEBUG_FS, these will be automatically configured +when CONFIG_TRACE is selected (if not already configured). + +Using the User Interface +------------------------ +Reading trace data and controlling the trace can be done using commands such +as cat, echo and sort. However, If you are logging binary trace data a +custom application may be required to read and process the trace data. +This section shows several examples of reading trace data and controling +the trace. All examples assume that the "trace" directory is your current +working directory. + +Viewing the current trace state: +$cat state + +Turning the trace on and off: +$echo start > state +$echo stop > state + +Reading data when using global buffers (USE_GLOBAL_BUFFERS): +$echo stop > state +$cat trace0 +$echo start > state + +Reading data when using per-cpu buffers: +When using per-cpu buffers the tracer should add a time stamp or sequence +number to each trace records. This is used by the consumer to sort the trace +records into chronological order. In the following example the tracer has +placed a time stamp at the front of each the record. The format of a record +is now shown. + +: