[PATCH 00/44] perf tools: Introduce an abstraction for AUX Area and Instruction Tracing

[Adrian Hunter <adrian.hunter@intel.com>]

Hi

Here are patches for the introduction of an abstraction for
using the AUX area and Instruction tracing. This patch
set now include support for Intel PT / Intel BTS.

The first 25 patches have been sent before and some have
Jiri's acks. One patch that Jiri acked "perf evlist: Add
support for mmapping an AUX area buffer" has been since
modified slightly so I dropped the ack.

Intel BTS can be used on most recent Intel CPUs. Intel PT
is available on Broadwell.

Examples:

	Trace 'ls' with Intel BTS userspace only

	perf record --per-thread -e intel_bts//u ls
	perf report
	perf script

	Trace 'ls' with Intel BTS kernel and userspace

	~/libexec/perf-core/perf-with-kcore record bts-ls --per-thread -e intel_bts// -- ls
	~/libexec/perf-core/perf-with-kcore report bts-ls
	~/libexec/perf-core/perf-with-kcore script bts-ls

	Trace 'ls' with Intel PT userspace only

	perf record -e intel_pt//u ls
	perf report
	perf script

	Trace 'ls' with Intel PT kernel and userspace

	~/libexec/perf-core/perf-with-kcore record pt-ls -e intel_pt// -- ls
	~/libexec/perf-core/perf-with-kcore report pt-ls
	~/libexec/perf-core/perf-with-kcore script pt-ls


The abstraction has two separate aspects:
	1. recording AUX area data
	2. processing AUX area data

Recording consists of mmapping a separate buffer and copying
the data into the perf.data file.  The buffer is an AUX area
buffer.  The data is written preceded by a new user event
PERF_RECORD_AUXTRACE.  The data is too big to fit in the event
but follows immediately afterward. Session processing has to
skip to get to the next event header in a similar fashion to
the existing PERF_RECORD_HEADER_TRACING_DATA
event.  The main recording patches are:

      perf evlist: Add support for mmapping an AUX area buffer
      perf tools: Add user events for AUX area tracing
      perf tools: Add support for AUX area recording
      perf record: Add basic AUX area tracing support

Processing consists of providing hooks in session processing
to enable a decoder to see all the events and deliver synthesized
events transparently into the event stream.  The main processing
patch is:

      perf session: Add hooks to allow transparent decoding of AUX area tracing data


Adrian Hunter (44):
      perf header: Add AUX area tracing feature
      perf evlist: Add support for mmapping an AUX area buffer
      perf tools: Add user events for AUX area tracing
      perf tools: Add support for AUX area recording
      perf record: Add basic AUX area tracing support
      perf record: Extend -m option for AUX area tracing mmap pages
      perf tools: Add a user event for AUX area tracing errors
      perf session: Add hooks to allow transparent decoding of AUX area tracing data
      perf session: Add instruction tracing options
      perf auxtrace: Add helpers for AUX area tracing errors
      perf auxtrace: Add helpers for queuing AUX area tracing data
      perf auxtrace: Add a heap for sorting AUX area tracing queues
      perf auxtrace: Add processing for AUX area tracing events
      perf auxtrace: Add a hashtable for caching
      perf tools: Add member to struct dso for an instruction cache
      perf script: Add Instruction Tracing support
      perf script: Always allow fields 'addr' and 'cpu' for auxtrace
      perf report: Add Instruction Tracing support
      perf inject: Re-pipe AUX area tracing events
      perf inject: Add Instruction Tracing support
      perf tools: Add AUX area tracing index
      perf tools: Hit all build ids when AUX area tracing
      perf tools: Add build option NO_AUXTRACE to exclude AUX area tracing
      perf auxtrace: Add option to synthesize events for transactions
      perf script: Add field option 'flags' to print sample flags
      perf tools: Add aux_watermark member of struct perf_event_attr
      perf tools: Add support for PERF_RECORD_AUX
      perf tools: Add support for PERF_RECORD_ITRACE_START
      perf tools: Add AUX area tracing Snapshot Mode
      perf record: Add AUX area tracing Snapshot Mode support
      perf auxtrace: Add Intel PT as an AUX area tracing type
      perf tools: Add Intel PT packet decoder
      perf tools: Add Intel PT instruction decoder
      perf tools: Add Intel PT log
      perf tools: Add Intel PT decoder
      perf tools: Add Intel PT support
      perf tools: Take Intel PT into use
      perf tools: Allow auxtrace data alignment
      perf tools: Add Intel BTS support
      perf tools: Output sample flags and insn_len from intel_pt
      perf tools: Output sample flags and insn_len from intel_bts
      perf tools: Intel PT to always update thread stack trace number
      perf tools: Intel BTS to always update thread stack trace number
      perf tools: Add example call-graph script

 tools/build/Makefile.build                         |    2 +
 tools/perf/.gitignore                              |    2 +
 tools/perf/Documentation/intel-bts.txt             |   67 +
 tools/perf/Documentation/intel-pt.txt              |  537 ++++
 tools/perf/Documentation/perf-inject.txt           |   27 +
 tools/perf/Documentation/perf-record.txt           |    9 +
 tools/perf/Documentation/perf-report.txt           |   29 +
 tools/perf/Documentation/perf-script.txt           |   38 +-
 tools/perf/Makefile.perf                           |    8 +-
 tools/perf/arch/x86/util/Build                     |    3 +
 tools/perf/arch/x86/util/auxtrace.c                |   82 +
 tools/perf/arch/x86/util/pmu.c                     |   15 +
 tools/perf/builtin-buildid-list.c                  |    9 +
 tools/perf/builtin-inject.c                        |  174 +-
 tools/perf/builtin-record.c                        |  270 +-
 tools/perf/builtin-report.c                        |   11 +
 tools/perf/builtin-script.c                        |   74 +-
 tools/perf/config/Makefile                         |    5 +
 tools/perf/perf.h                                  |    5 +
 .../scripts/python/call-graph-from-postgresql.py   |  285 +++
 tools/perf/tests/make                              |    2 +
 tools/perf/util/Build                              |    4 +
 tools/perf/util/auxtrace.c                         | 1362 ++++++++++
 tools/perf/util/auxtrace.h                         |  646 +++++
 tools/perf/util/dso.c                              |    2 +
 tools/perf/util/dso.h                              |    3 +
 tools/perf/util/event.c                            |   42 +
 tools/perf/util/event.h                            |   70 +
 tools/perf/util/evlist.c                           |   71 +-
 tools/perf/util/evlist.h                           |    6 +
 tools/perf/util/evsel.c                            |    1 +
 tools/perf/util/header.c                           |   37 +
 tools/perf/util/header.h                           |    1 +
 tools/perf/util/intel-bts.c                        | 1353 ++++++++++
 tools/perf/util/intel-bts.h                        |   31 +
 tools/perf/util/intel-pt-decoder/Build             |   14 +
 .../perf/util/intel-pt-decoder/intel-pt-decoder.c  | 1738 +++++++++++++
 .../perf/util/intel-pt-decoder/intel-pt-decoder.h  |   89 +
 .../util/intel-pt-decoder/intel-pt-insn-decoder.c  |  246 ++
 .../util/intel-pt-decoder/intel-pt-insn-decoder.h  |   65 +
 tools/perf/util/intel-pt-decoder/intel-pt-log.c    |  155 ++
 tools/perf/util/intel-pt-decoder/intel-pt-log.h    |   52 +
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.c   |  400 +++
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.h   |   64 +
 tools/perf/util/intel-pt.c                         | 2637 ++++++++++++++++++++
 tools/perf/util/intel-pt.h                         |   35 +
 tools/perf/util/machine.c                          |   21 +
 tools/perf/util/machine.h                          |    4 +
 tools/perf/util/parse-options.h                    |    4 +
 tools/perf/util/record.c                           |   11 +-
 tools/perf/util/session.c                          |  184 +-
 tools/perf/util/session.h                          |    6 +
 tools/perf/util/tool.h                             |   12 +-
 53 files changed, 10971 insertions(+), 49 deletions(-)
 create mode 100644 tools/perf/Documentation/intel-bts.txt
 create mode 100644 tools/perf/Documentation/intel-pt.txt
 create mode 100644 tools/perf/arch/x86/util/auxtrace.c
 create mode 100644 tools/perf/arch/x86/util/pmu.c
 create mode 100644 tools/perf/scripts/python/call-graph-from-postgresql.py
 create mode 100644 tools/perf/util/auxtrace.c
 create mode 100644 tools/perf/util/auxtrace.h
 create mode 100644 tools/perf/util/intel-bts.c
 create mode 100644 tools/perf/util/intel-bts.h
 create mode 100644 tools/perf/util/intel-pt-decoder/Build
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.h
 create mode 100644 tools/perf/util/intel-pt.c
 create mode 100644 tools/perf/util/intel-pt.h


Regards
Adrian