* [RFC 0/5] perf tools: Add perf data CTF conversion
@ 2014-08-06 12:13 Jiri Olsa
2014-08-06 12:13 ` [PATCH 1/5] perf tools: Add feature check for libbabeltrace Jiri Olsa
` (5 more replies)
0 siblings, 6 replies; 11+ messages in thread
From: Jiri Olsa @ 2014-08-06 12:13 UTC (permalink / raw)
To: linux-kernel
Cc: Arnaldo Carvalho de Melo, David Ahern, Dominique Toupin,
Frederic Weisbecker, Jeremie Galarneau, Jiri Olsa,
Mathieu Desnoyers, Namhyung Kim, Paul Mackerras, Peter Zijlstra,
Sebastian Andrzej Siewior, Tom Zanussi
hi,
this is initial post of the CTF converter for perf.
Basically adding 'perf data convert' command to allow
conversion of perf data file into CTF [1] data.
To convert perf data file run:
$ perf data convert --to-ctf=./ctf-data/
[ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
[ perf data convert: Converted and wrote 11.268 MB (100230 samples) ]
To display converted CTF data run [2]:
$ babeltrace ./ctf-data/
[03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
[03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
[03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
[03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
[03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 }
[03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 }
[03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 }
[03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 }
[03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 }
It has still some limitations (noted in patches), but it has
reached the point of the useful perf feature. More features
and supports are about to come and it'll be better to continue
publicly.
Also reachable in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_ctf_convert
thanks for comments,
jirka
[1] Common Trace Format - http://www.efficios.com/ctf
[2] babeltrace - http://www.efficios.com/babeltrace
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Dominique Toupin <dominique.toupin@ericsson.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
---
Jiri Olsa (3):
perf tools: Add feature check for libbabeltrace
perf tools: Add new perf data command
perf tools data: Add perf data to CTF convertion support
Sebastian Andrzej Siewior (2):
perf tools data: Add a 'perf' prefix to the generic fields
perf tools data: Add tracepoint events fields CTF convertion support
tools/perf/Documentation/perf-data.txt | 40 ++++
tools/perf/Documentation/perf.txt | 7 +-
tools/perf/Makefile.perf | 8 +
tools/perf/builtin-data.c | 112 ++++++++++
tools/perf/builtin.h | 1 +
tools/perf/command-list.txt | 1 +
tools/perf/config/Makefile | 28 ++-
tools/perf/config/feature-checks/Makefile | 10 +-
tools/perf/config/feature-checks/test-all.c | 5 +
tools/perf/config/feature-checks/test-libbabeltrace.c | 8 +
tools/perf/perf.c | 1 +
tools/perf/util/data-bt.c | 859 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
tools/perf/util/data-bt.h | 8 +
tools/perf/util/debug.c | 4 +-
tools/perf/util/debug.h | 1 +
15 files changed, 1086 insertions(+), 7 deletions(-)
create mode 100644 tools/perf/Documentation/perf-data.txt
create mode 100644 tools/perf/builtin-data.c
create mode 100644 tools/perf/config/feature-checks/test-libbabeltrace.c
create mode 100644 tools/perf/util/data-bt.c
create mode 100644 tools/perf/util/data-bt.h
^ permalink raw reply [flat|nested] 11+ messages in thread* [PATCH 1/5] perf tools: Add feature check for libbabeltrace 2014-08-06 12:13 [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa @ 2014-08-06 12:13 ` Jiri Olsa 2014-08-06 12:13 ` [PATCH 2/5] perf tools: Add new perf data command Jiri Olsa ` (4 subsequent siblings) 5 siblings, 0 replies; 11+ messages in thread From: Jiri Olsa @ 2014-08-06 12:13 UTC (permalink / raw) To: linux-kernel Cc: Jiri Olsa, Arnaldo Carvalho de Melo, David Ahern, Dominique Toupin, Frederic Weisbecker, Jeremie Galarneau, Mathieu Desnoyers, Namhyung Kim, Paul Mackerras, Peter Zijlstra, Sebastian Andrzej Siewior, Tom Zanussi Adding feature check for babeltrace library [1], which will be used for perf data file CTF [2] conversion in following patches. The babeltrace library is now automatically detected as standard feature. It's possible to specify LIBBABELTRACE_DIR make variable to specify location of installed libbabeltrace, like: $ make LIBBABELTRACE_DIR=/opt/libbabeltrace/ BUILD: Doing 'make -j4' parallel build Auto-detecting system features: ... dwarf: [ on ] ... glibc: [ on ] ... gtk2: [ on ] ... libaudit: [ on ] ... libbfd: [ on ] ... libelf: [ on ] ... libnuma: [ on ] ... libperl: [ on ] ... libpython: [ on ] ... libslang: [ on ] ... libunwind: [ on ] ... libdw-dwarf-unwind: [ on ] ... libbabeltrace: [ on ] ... DWARF post unwind library: libunwind [1] babeltrace - http://www.efficios.com/babeltrace [2] Common Trace Format - http://www.efficios.com/ctf Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: David Ahern <dsahern@gmail.com> Cc: Dominique Toupin <dominique.toupin@ericsson.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Tom Zanussi <tzanussi@gmail.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Jiri Olsa <jolsa@kernel.org> --- tools/perf/Makefile.perf | 3 +++ tools/perf/config/Makefile | 27 ++++++++++++++++++++-- tools/perf/config/feature-checks/Makefile | 10 +++++--- tools/perf/config/feature-checks/test-all.c | 5 ++++ .../config/feature-checks/test-libbabeltrace.c | 8 +++++++ 5 files changed, 48 insertions(+), 5 deletions(-) create mode 100644 tools/perf/config/feature-checks/test-libbabeltrace.c diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf index 1ea31e275b4d..4c5219b8e209 100644 --- a/tools/perf/Makefile.perf +++ b/tools/perf/Makefile.perf @@ -60,6 +60,9 @@ include config/utilities.mak # # Define NO_LIBDW_DWARF_UNWIND if you do not want libdw support # for dwarf backtrace post unwind. +# +# Define NO_LIBBABELTRACE if you do not want libbabeltrace support +# for CTF data format. ifeq ($(srctree),) srctree := $(patsubst %/,%,$(dir $(shell pwd))) diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile index 75d4c237b03d..b99a0323328c 100644 --- a/tools/perf/config/Makefile +++ b/tools/perf/config/Makefile @@ -88,6 +88,17 @@ ifndef NO_LIBELF FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind := $(LIBDW_LDFLAGS) -ldw endif +ifndef NO_LIBBABELTRACE + # for linking with debug library, run like: + # make DEBUG=1 LIBBABELTRACE_DIR=/opt/libbabeltrace + ifdef LIBBABELTRACE_DIR + LIBBABELTRACE_CFLAGS := -I$(LIBBABELTRACE_DIR)/include + LIBBABELTRACE_LDFLAGS := -L$(LIBBABELTRACE_DIR)/lib + endif + FEATURE_CHECK_CFLAGS-libbabeltrace := $(LIBBABELTRACE_CFLAGS) + FEATURE_CHECK_LDFLAGS-libbabeltrace := $(LIBBABELTRACE_LDFLAGS) -lbabeltrace-ctf +endif + # include ARCH specific config -include $(src-perf)/arch/$(ARCH)/Makefile @@ -204,7 +215,8 @@ CORE_FEATURE_TESTS = \ libunwind \ stackprotector-all \ timerfd \ - libdw-dwarf-unwind + libdw-dwarf-unwind \ + libbabeltrace LIB_FEATURE_TESTS = \ dwarf \ @@ -218,7 +230,8 @@ LIB_FEATURE_TESTS = \ libpython \ libslang \ libunwind \ - libdw-dwarf-unwind + libdw-dwarf-unwind \ + libbabeltrace VF_FEATURE_TESTS = \ backtrace \ @@ -620,6 +633,16 @@ ifdef HAVE_KVM_STAT_SUPPORT CFLAGS += -DHAVE_KVM_STAT_SUPPORT endif +ifndef NO_LIBBABELTRACE + ifeq ($(feature-libbabeltrace), 0) + msg := $(warning No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-devel/libbabeltrace-dev); + NO_LIBBABELTRACE := 1 + else + CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT + EXTLIBS += -lbabeltrace + endif +endif + # Among the variables below, these: # perfexecdir # template_dir diff --git a/tools/perf/config/feature-checks/Makefile b/tools/perf/config/feature-checks/Makefile index 72ab2984718e..fb6454eecd2a 100644 --- a/tools/perf/config/feature-checks/Makefile +++ b/tools/perf/config/feature-checks/Makefile @@ -27,7 +27,8 @@ FILES= \ test-libunwind-debug-frame.bin \ test-stackprotector-all.bin \ test-timerfd.bin \ - test-libdw-dwarf-unwind.bin + test-libdw-dwarf-unwind.bin \ + test-libbabeltrace.bin CC := $(CROSS_COMPILE)gcc -MD PKG_CONFIG := $(CROSS_COMPILE)pkg-config @@ -39,7 +40,7 @@ BUILD = $(CC) $(CFLAGS) -o $(OUTPUT)$@ $(patsubst %.bin,%.c,$@) $(LDFLAGS) ############################### test-all.bin: - $(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl + $(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lbabeltrace test-hello.bin: $(BUILD) @@ -126,7 +127,10 @@ test-timerfd.bin: $(BUILD) test-libdw-dwarf-unwind.bin: - $(BUILD) + $(BUILD) # -ldw provided by $(FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind) + +test-libbabeltrace.bin: + $(BUILD) # -lbabeltrace provided by $(FEATURE_CHECK_LDFLAGS-libbabeltrace) test-sync-compare-and-swap.bin: $(BUILD) -Werror diff --git a/tools/perf/config/feature-checks/test-all.c b/tools/perf/config/feature-checks/test-all.c index a7d022e161c0..9fb4f1c07463 100644 --- a/tools/perf/config/feature-checks/test-all.c +++ b/tools/perf/config/feature-checks/test-all.c @@ -93,6 +93,10 @@ # include "test-sync-compare-and-swap.c" #undef main +#define main main_test_libbabeltrace +# include "test-libbabeltrace.c" +#undef main + int main(int argc, char *argv[]) { main_test_libpython(); @@ -116,6 +120,7 @@ int main(int argc, char *argv[]) main_test_stackprotector_all(); main_test_libdw_dwarf_unwind(); main_test_sync_compare_and_swap(argc, argv); + main_test_libbabeltrace(); return 0; } diff --git a/tools/perf/config/feature-checks/test-libbabeltrace.c b/tools/perf/config/feature-checks/test-libbabeltrace.c new file mode 100644 index 000000000000..123b5b962ea5 --- /dev/null +++ b/tools/perf/config/feature-checks/test-libbabeltrace.c @@ -0,0 +1,8 @@ + +#include <babeltrace/ctf-writer/writer.h> + +int main(void) +{ + bt_ctf_writer_create(NULL); + return 0; +} -- 1.8.3.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 2/5] perf tools: Add new perf data command 2014-08-06 12:13 [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa 2014-08-06 12:13 ` [PATCH 1/5] perf tools: Add feature check for libbabeltrace Jiri Olsa @ 2014-08-06 12:13 ` Jiri Olsa 2014-08-06 12:13 ` [PATCH 3/5] perf tools data: Add perf data to CTF convertion support Jiri Olsa ` (3 subsequent siblings) 5 siblings, 0 replies; 11+ messages in thread From: Jiri Olsa @ 2014-08-06 12:13 UTC (permalink / raw) To: linux-kernel Cc: Jiri Olsa, Arnaldo Carvalho de Melo, David Ahern, Dominique Toupin, Frederic Weisbecker, Jeremie Galarneau, Mathieu Desnoyers, Namhyung Kim, Paul Mackerras, Peter Zijlstra, Sebastian Andrzej Siewior, Tom Zanussi Adding new 'perf data' command to provide operations over data files. The 'perf data convert' sub command is coming in following patch, but there's possibility for other useful commands like 'perf data ls' (to display perf data file in directory in ls style). Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: David Ahern <dsahern@gmail.com> Cc: Dominique Toupin <dominique.toupin@ericsson.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Tom Zanussi <tzanussi@gmail.com> Signed-off-by: Jiri Olsa <jolsa@kernel.org> --- tools/perf/Documentation/perf-data.txt | 15 ++++++ tools/perf/Makefile.perf | 1 + tools/perf/builtin-data.c | 98 ++++++++++++++++++++++++++++++++++ tools/perf/builtin.h | 1 + tools/perf/command-list.txt | 1 + tools/perf/perf.c | 1 + 6 files changed, 117 insertions(+) create mode 100644 tools/perf/Documentation/perf-data.txt create mode 100644 tools/perf/builtin-data.c diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt new file mode 100644 index 000000000000..b8c83947715c --- /dev/null +++ b/tools/perf/Documentation/perf-data.txt @@ -0,0 +1,15 @@ +perf-data(1) +============== + +NAME +---- +perf-data - Data file related processing + +SYNOPSIS +-------- +[verse] +'perf data' [<common options>] <command> [<options>]", + +DESCRIPTION +----------- +Data file related processing. diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf index 4c5219b8e209..f25d848a9ab6 100644 --- a/tools/perf/Makefile.perf +++ b/tools/perf/Makefile.perf @@ -464,6 +464,7 @@ BUILTIN_OBJS += $(OUTPUT)builtin-kvm.o BUILTIN_OBJS += $(OUTPUT)builtin-inject.o BUILTIN_OBJS += $(OUTPUT)tests/builtin-test.o BUILTIN_OBJS += $(OUTPUT)builtin-mem.o +BUILTIN_OBJS += $(OUTPUT)builtin-data.o PERFLIBS = $(LIB_FILE) $(LIBAPIKFS) $(LIBTRACEEVENT) diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c new file mode 100644 index 000000000000..999bdd0e4c7f --- /dev/null +++ b/tools/perf/builtin-data.c @@ -0,0 +1,98 @@ +#include <linux/compiler.h> +#include "builtin.h" +#include "perf.h" +#include "debug.h" +#include "parse-options.h" + +typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix); + +struct data_cmd { + const char *name; + const char *summary; + data_cmd_fn_t fn; +}; + +static struct data_cmd data_cmds[]; + +#define for_each_cmd(cmd) \ + for (cmd = data_cmds; cmd->name; cmd++) + +static const struct option data_options[] = { + OPT_END() +}; + +static const char * const data_usage[] = { + "perf data [<common options>] <command> [<options>]", + NULL +}; + +static void print_usage(void) +{ + struct data_cmd *cmd; + + printf("Usage:\n"); + printf("\t%s\n\n", data_usage[0]); + printf("\tAvailable commands:\n"); + + for_each_cmd(cmd) { + printf("\t %s\t- %s\n", cmd->name, cmd->summary); + } + + printf("\n"); +} + +static const char * const data_convert_usage[] = { + "perf data convert [<options>]", + NULL +}; + +static int cmd_data_convert(int argc, const char **argv, + const char *prefix __maybe_unused) +{ + const struct option options[] = { + OPT_INCR('v', "verbose", &verbose, "be more verbose"), + OPT_STRING('i', "input", &input_name, "file", + "input file name"), + OPT_END() + }; + + argc = parse_options(argc, argv, options, + data_convert_usage, 0); + if (argc) { + usage_with_options(data_convert_usage, options); + return -1; + } + + return 0; +} + +static struct data_cmd data_cmds[] = { + { "convert", "converts data file between formats", cmd_data_convert }, + { NULL, NULL, NULL }, +}; + +int cmd_data(int argc __maybe_unused, const char **argv __maybe_unused, + const char *prefix __maybe_unused) +{ + struct data_cmd *cmd; + + /* No command specified. */ + if (argc < 2) + goto usage; + + argc = parse_options(argc, argv, data_options, data_usage, + PARSE_OPT_STOP_AT_NON_OPTION); + if (argc < 1) + goto usage; + + for_each_cmd(cmd) { + if (strcmp(cmd->name, argv[0])) + continue; + + return cmd->fn(argc, argv, prefix); + } + +usage: + print_usage(); + return -1; +} diff --git a/tools/perf/builtin.h b/tools/perf/builtin.h index b210d62907e4..3688ad29085f 100644 --- a/tools/perf/builtin.h +++ b/tools/perf/builtin.h @@ -37,6 +37,7 @@ extern int cmd_test(int argc, const char **argv, const char *prefix); extern int cmd_trace(int argc, const char **argv, const char *prefix); extern int cmd_inject(int argc, const char **argv, const char *prefix); extern int cmd_mem(int argc, const char **argv, const char *prefix); +extern int cmd_data(int argc, const char **argv, const char *prefix); extern int find_scripts(char **scripts_array, char **scripts_path_array); #endif diff --git a/tools/perf/command-list.txt b/tools/perf/command-list.txt index 0906fc401c52..00fcaf8a5b8d 100644 --- a/tools/perf/command-list.txt +++ b/tools/perf/command-list.txt @@ -7,6 +7,7 @@ perf-archive mainporcelain common perf-bench mainporcelain common perf-buildid-cache mainporcelain common perf-buildid-list mainporcelain common +perf-data mainporcelain common perf-diff mainporcelain common perf-evlist mainporcelain common perf-inject mainporcelain common diff --git a/tools/perf/perf.c b/tools/perf/perf.c index 2282d41879a2..7dcd5f70b742 100644 --- a/tools/perf/perf.c +++ b/tools/perf/perf.c @@ -62,6 +62,7 @@ static struct cmd_struct commands[] = { #endif { "inject", cmd_inject, 0 }, { "mem", cmd_mem, 0 }, + { "data", cmd_data, 0 }, }; struct pager_config { -- 1.8.3.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 3/5] perf tools data: Add perf data to CTF convertion support 2014-08-06 12:13 [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa 2014-08-06 12:13 ` [PATCH 1/5] perf tools: Add feature check for libbabeltrace Jiri Olsa 2014-08-06 12:13 ` [PATCH 2/5] perf tools: Add new perf data command Jiri Olsa @ 2014-08-06 12:13 ` Jiri Olsa 2014-08-06 12:13 ` [PATCH 4/5] perf tools data: Add a 'perf' prefix to the generic fields Jiri Olsa ` (2 subsequent siblings) 5 siblings, 0 replies; 11+ messages in thread From: Jiri Olsa @ 2014-08-06 12:13 UTC (permalink / raw) To: linux-kernel Cc: Jiri Olsa, Arnaldo Carvalho de Melo, David Ahern, Dominique Toupin, Frederic Weisbecker, Jeremie Galarneau, Mathieu Desnoyers, Namhyung Kim, Paul Mackerras, Peter Zijlstra, Sebastian Andrzej Siewior, Tom Zanussi Adding 'perf data convert' to convert perf data file into different format. This patch adds support for CTF format conversion. To convert perf.data into CTF run: $ perf data convert --to-ctf=./ctf-data/ [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ] [ perf data convert: Converted and wrote 11.268 MB (100230 samples) ] The command will create CTF metadata out of perf.data file (or one specified via -i option) and then convert all sample events into single CTF stream. Each sample_type bit is translated into separated CTF event field apart from following exceptions: PERF_SAMPLE_RAW - added in next patch PERF_SAMPLE_READ - TODO PERF_SAMPLE_CALLCHAIN - TODO PERF_SAMPLE_BRANCH_STACK - TODO PERF_SAMPLE_REGS_USER - TODO PERF_SAMPLE_STACK_USER - TODO $ perf --debug=data-convert=2 data convert ... The converted CTF data could be analyzed by CTF tools, like: $ babeltrace ./ctf-data/ [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 } [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 } [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 } [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 } [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 } [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 } [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 } [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 } [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 } Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: David Ahern <dsahern@gmail.com> Cc: Dominique Toupin <dominique.toupin@ericsson.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Tom Zanussi <tzanussi@gmail.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Jiri Olsa <jolsa@kernel.org> --- tools/perf/Documentation/perf-data.txt | 25 ++ tools/perf/Documentation/perf.txt | 7 +- tools/perf/Makefile.perf | 4 + tools/perf/builtin-data.c | 14 + tools/perf/config/Makefile | 5 +- tools/perf/util/data-bt.c | 611 +++++++++++++++++++++++++++++++++ tools/perf/util/data-bt.h | 8 + tools/perf/util/debug.c | 4 +- tools/perf/util/debug.h | 1 + 9 files changed, 675 insertions(+), 4 deletions(-) create mode 100644 tools/perf/util/data-bt.c create mode 100644 tools/perf/util/data-bt.h diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt index b8c83947715c..be8fa1a0a97e 100644 --- a/tools/perf/Documentation/perf-data.txt +++ b/tools/perf/Documentation/perf-data.txt @@ -13,3 +13,28 @@ SYNOPSIS DESCRIPTION ----------- Data file related processing. + +COMMANDS +-------- +convert:: + Converts perf data file into another format (only CTF [1] format is support by now). + It's possible to set data-convert debug variable to get debug messages from conversion, + like: + perf --debug data-convert data convert ... + +OPTIONS for 'convert' +--------------------- +--to-ctf:: + Triggers the CTF conversion, specify the path of CTF data directory. + +-i:: + Specify input perf data file path. + +-v:: +--verbose:: + Be more verbose (show counter open errors, etc). + +SEE ALSO +-------- +linkperf:perf[1] +[1] Common Trace Format - http://www.efficios.com/ctf diff --git a/tools/perf/Documentation/perf.txt b/tools/perf/Documentation/perf.txt index d240bb2e5b22..eacdc2a63a9c 100644 --- a/tools/perf/Documentation/perf.txt +++ b/tools/perf/Documentation/perf.txt @@ -13,11 +13,16 @@ SYNOPSIS OPTIONS ------- --debug:: - Setup debug variable (just verbose for now) in value + Setup debug variable (see list below) in value range (0, 10). Use like: --debug verbose # sets verbose = 1 --debug verbose=2 # sets verbose = 2 + List of debug variables allowed to set: + verbose - general debug messages + ordered-events - ordered events object debug messages + data-convert - data convert command debug messages + DESCRIPTION ----------- Performance counters for Linux are a new kernel-based subsystem diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf index f25d848a9ab6..bc9b2e7c0fd7 100644 --- a/tools/perf/Makefile.perf +++ b/tools/perf/Makefile.perf @@ -383,6 +383,10 @@ LIB_OBJS += $(OUTPUT)util/data.o LIB_OBJS += $(OUTPUT)util/tsc.o LIB_OBJS += $(OUTPUT)util/cloexec.o +ifneq ($(NO_LIBBABELTRACE),1) +LIB_OBJS += $(OUTPUT)util/data-bt.o +endif + LIB_OBJS += $(OUTPUT)ui/setup.o LIB_OBJS += $(OUTPUT)ui/helpline.o LIB_OBJS += $(OUTPUT)ui/progress.o diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c index 999bdd0e4c7f..7233dcee03a1 100644 --- a/tools/perf/builtin-data.c +++ b/tools/perf/builtin-data.c @@ -3,6 +3,7 @@ #include "perf.h" #include "debug.h" #include "parse-options.h" +#include "data-bt.h" typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix); @@ -49,10 +50,14 @@ static const char * const data_convert_usage[] = { static int cmd_data_convert(int argc, const char **argv, const char *prefix __maybe_unused) { + const char *to_ctf = NULL; const struct option options[] = { OPT_INCR('v', "verbose", &verbose, "be more verbose"), OPT_STRING('i', "input", &input_name, "file", "input file name"), +#ifdef HAVE_LIBBABELTRACE_SUPPORT + OPT_STRING(0, "to-ctf", &to_ctf, NULL, "Convert to CTF format"), +#endif OPT_END() }; @@ -63,6 +68,15 @@ static int cmd_data_convert(int argc, const char **argv, return -1; } + if (to_ctf) { +#ifdef HAVE_LIBBABELTRACE_SUPPORT + return bt_convert__perf2ctf(input_name, to_ctf); +#else + pr_err("The libbabeltrace support is not compiled in\n"); + return -1; +#endif + } + return 0; } diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile index b99a0323328c..3537fd226f4d 100644 --- a/tools/perf/config/Makefile +++ b/tools/perf/config/Makefile @@ -638,8 +638,9 @@ ifndef NO_LIBBABELTRACE msg := $(warning No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-devel/libbabeltrace-dev); NO_LIBBABELTRACE := 1 else - CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT - EXTLIBS += -lbabeltrace + CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT $(LIBBABELTRACE_CFLAGS) + LDFLAGS += $(LIBBABELTRACE_LDFLAGS) + EXTLIBS += -lbabeltrace-ctf endif endif diff --git a/tools/perf/util/data-bt.c b/tools/perf/util/data-bt.c new file mode 100644 index 000000000000..67d7be0e76bc --- /dev/null +++ b/tools/perf/util/data-bt.c @@ -0,0 +1,611 @@ +/* + * CTF writting support via babeltrace. + * + * Copyright (C) 2014, Jiri Olsa <jolsa@redhat.com> + * Copyright (C) 2014, Sebastian Andrzej Siewior <bigeasy@linutronix.de> + * + * Released under the GPL v2. (and only v2, not any later version) + */ +#include <linux/compiler.h> +#include <babeltrace/ctf-writer/writer.h> +#include <babeltrace/ctf-writer/clock.h> +#include <babeltrace/ctf-writer/stream.h> +#include <babeltrace/ctf-writer/event.h> +#include <babeltrace/ctf-writer/event-types.h> +#include <babeltrace/ctf-writer/event-fields.h> +#include <babeltrace/ctf/events.h> +#include <traceevent/event-parse.h> +#include "asm/bug.h" +#include "data-bt.h" +#include "session.h" +#include "util.h" +#include "debug.h" +#include "tool.h" +#include "evlist.h" +#include "evsel.h" +#include "machine.h" + +#define pr_N(n, fmt, ...) \ + eprintf(n, debug_data_convert, fmt, ##__VA_ARGS__) + +#define pr(fmt, ...) pr_N(1, pr_fmt(fmt), ##__VA_ARGS__) +#define pr2(fmt, ...) pr_N(2, pr_fmt(fmt), ##__VA_ARGS__) + +#define pr_time2(t, fmt, ...) pr_time_N(2, debug_data_convert, t, \ + pr_fmt(fmt), ##__VA_ARGS__) + +struct evsel_priv { + struct bt_ctf_event_class *event_class; +}; + +struct ctf_writer { + /* writer primitives */ + struct bt_ctf_writer *writer; + struct bt_ctf_stream *stream; + struct bt_ctf_stream_class *stream_class; + struct bt_ctf_clock *clock; + + /* data types */ + union { + struct { + struct bt_ctf_field_type *s64; + struct bt_ctf_field_type *u64; + struct bt_ctf_field_type *s32; + struct bt_ctf_field_type *u32; + struct bt_ctf_field_type *string; + struct bt_ctf_field_type *u64_hex; + }; + struct bt_ctf_field_type *array[6]; + } data; +}; + +struct convert { + struct perf_tool tool; + struct ctf_writer writer; + + u64 events_size; + u64 events_count; +}; + +static int value_set(struct bt_ctf_field_type *type, + struct bt_ctf_event *event, + const char *name, u64 val) +{ + struct bt_ctf_field *field; + bool sign = bt_ctf_field_type_integer_get_signed(type); + int ret; + + field = bt_ctf_field_create(type); + if (!field) { + pr_err("failed to create a field %s\n", name); + return -1; + } + + if (sign) { + ret = bt_ctf_field_signed_integer_set_value(field, val); + if (ret) { + pr_err("failed to set field value %s\n", name); + goto err; + } + } else { + ret = bt_ctf_field_unsigned_integer_set_value(field, val); + if (ret) { + pr_err("failed to set field value %s\n", name); + goto err; + } + } + + ret = bt_ctf_event_set_payload(event, name, field); + if (ret) { + pr_err("failed to set payload %s\n", name); + goto err; + } + + pr2(" SET [%s = %" PRIu64 "]\n", name, val); + +err: + bt_ctf_field_put(field); + return ret; +} + +#define __FUNC_VALUE_SET(_name, _val_type) \ +static int __maybe_unused value_set_##_name(struct ctf_writer *cw, \ + struct bt_ctf_event *event, \ + const char *name, \ + _val_type val) \ +{ \ + struct bt_ctf_field_type *type = cw->data._name; \ + return value_set(type, event, name, (u64) val); \ +} + +#define FUNC_VALUE_SET(_name) __FUNC_VALUE_SET(_name, _name) + +FUNC_VALUE_SET(s32) +FUNC_VALUE_SET(u32) +FUNC_VALUE_SET(s64) +FUNC_VALUE_SET(u64) +__FUNC_VALUE_SET(u64_hex, u64) + +static int add_generic_values(struct ctf_writer *cw, + struct bt_ctf_event *event, + struct perf_evsel *evsel, + struct perf_sample *sample) +{ + u64 type = evsel->attr.sample_type; + int ret; + + /* + * missing: + * PERF_SAMPLE_TIME - not needed as we have it in + * ctf event header + * PERF_SAMPLE_READ - TODO + * PERF_SAMPLE_CALLCHAIN - TODO + * PERF_SAMPLE_RAW - tracepoint fields are handled separately + * PERF_SAMPLE_BRANCH_STACK - TODO + * PERF_SAMPLE_REGS_USER - TODO + * PERF_SAMPLE_STACK_USER - TODO + */ + + if (type & PERF_SAMPLE_IP) { + ret = value_set_u64_hex(cw, event, "ip", sample->ip); + if (ret) + return -1; + } + + if (type & PERF_SAMPLE_TID) { + ret = value_set_s32(cw, event, "tid", sample->tid); + if (ret) + return -1; + + ret = value_set_s32(cw, event, "pid", sample->tid); + if (ret) + return -1; + } + + if ((type & PERF_SAMPLE_ID) || + (type & PERF_SAMPLE_IDENTIFIER)) { + ret = value_set_u64(cw, event, "id", sample->id); + if (ret) + return -1; + } + + if (type & PERF_SAMPLE_STREAM_ID) { + ret = value_set_u64(cw, event, "stream_id", sample->id); + if (ret) + return -1; + } + + if (type & PERF_SAMPLE_CPU) { + ret = value_set_u32(cw, event, "cpu", sample->cpu); + if (ret) + return -1; + } + + if (type & PERF_SAMPLE_PERIOD) { + ret = value_set_u64(cw, event, "period", sample->period); + if (ret) + return -1; + } + + if (type & PERF_SAMPLE_WEIGHT) { + ret = value_set_u64(cw, event, "weight", sample->weight); + if (ret) + return -1; + } + + if (type & PERF_SAMPLE_DATA_SRC) { + ret = value_set_u64(cw, event, "data_src", sample->data_src); + if (ret) + return -1; + } + + if (type & PERF_SAMPLE_TRANSACTION) { + ret = value_set_u64(cw, event, "transaction", sample->transaction); + if (ret) + return -1; + } + + return 0; +} + +static int process_sample_event(struct perf_tool *tool, + union perf_event *_event __maybe_unused, + struct perf_sample *sample, + struct perf_evsel *evsel, + struct machine *machine __maybe_unused) +{ + struct convert *c = container_of(tool, struct convert, tool); + struct evsel_priv *priv = evsel->priv; + struct ctf_writer *cw = &c->writer; + struct bt_ctf_event_class *event_class; + struct bt_ctf_event *event; + int ret; + + if (WARN_ONCE(!priv, "Failed to setup all events.\n")) + return 0; + + event_class = priv->event_class; + + /* update stats */ + c->events_count++; + c->events_size += _event->header.size; + + pr_time2(sample->time, "sample %" PRIu64 "\n", c->events_count); + + event = bt_ctf_event_create(event_class); + if (!event) { + pr_err("Failed to create an CTF event\n"); + return -1; + } + + bt_ctf_clock_set_time(cw->clock, sample->time); + + ret = add_generic_values(cw, event, evsel, sample); + if (ret) + return -1; + + bt_ctf_stream_append_event(cw->stream, event); + bt_ctf_event_put(event); + return 0; +} + +static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel, + struct bt_ctf_event_class *event_class) +{ + u64 type = evsel->attr.sample_type; + + /* + * missing: + * PERF_SAMPLE_TIME - not needed as we have it in + * ctf event header + * PERF_SAMPLE_READ - TODO + * PERF_SAMPLE_CALLCHAIN - TODO + * PERF_SAMPLE_RAW - tracepoint fields are handled separately + * PERF_SAMPLE_BRANCH_STACK - TODO + * PERF_SAMPLE_REGS_USER - TODO + * PERF_SAMPLE_STACK_USER - TODO + */ + +#define ADD_FIELD(cl, t, n) \ + do { \ + pr2(" field '%s'\n", n); \ + if (bt_ctf_event_class_add_field(cl, t, n)) { \ + pr_err("Failed to add field '%s;\n", n); \ + return -1; \ + } \ + } while (0) + + if (type & PERF_SAMPLE_IP) + ADD_FIELD(event_class, cw->data.u64_hex, "ip"); + + if (type & PERF_SAMPLE_TID) { + ADD_FIELD(event_class, cw->data.s32, "tid"); + ADD_FIELD(event_class, cw->data.s32, "pid"); + } + + if ((type & PERF_SAMPLE_ID) || + (type & PERF_SAMPLE_IDENTIFIER)) + ADD_FIELD(event_class, cw->data.u64, "id"); + + if (type & PERF_SAMPLE_STREAM_ID) + ADD_FIELD(event_class, cw->data.u64, "stream_id"); + + if (type & PERF_SAMPLE_CPU) + ADD_FIELD(event_class, cw->data.u32, "cpu"); + + if (type & PERF_SAMPLE_PERIOD) + ADD_FIELD(event_class, cw->data.u64, "period"); + + if (type & PERF_SAMPLE_WEIGHT) + ADD_FIELD(event_class, cw->data.u64, "weight"); + + if (type & PERF_SAMPLE_DATA_SRC) + ADD_FIELD(event_class, cw->data.u64, "data_src"); + + if (type & PERF_SAMPLE_TRANSACTION) + ADD_FIELD(event_class, cw->data.u64, "transaction"); + +#undef ADD_FIELD + return 0; +} + +static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel) +{ + struct bt_ctf_event_class *event_class; + struct evsel_priv *priv; + const char *name = perf_evsel__name(evsel); + int ret; + + pr("Adding event '%s' (type %d)\n", name, evsel->attr.type); + + event_class = bt_ctf_event_class_create(name); + if (!event_class) + return -1; + + ret = add_generic_types(cw, evsel, event_class); + if (ret) + goto err; + + ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class); + if (ret) { + pr("Failed to add event class into stream.\n"); + goto err; + } + + priv = malloc(sizeof(*priv)); + if (!priv) + goto err; + + priv->event_class = event_class; + evsel->priv = priv; + return 0; + +err: + bt_ctf_event_class_put(event_class); + pr_err("Failed to add event '%s'.\n", name); + return -1; +} + +static int setup_events(struct ctf_writer *cw, struct perf_session *session) +{ + struct perf_evlist *evlist = session->evlist; + struct perf_evsel *evsel; + int ret; + + evlist__for_each(evlist, evsel) { + ret = add_event(cw, evsel); + if (ret) + return ret; + } + return 0; +} + +static int ctf_writer__setup_env(struct ctf_writer *cw, + struct perf_session *session) +{ + struct perf_header *header = &session->header; + struct bt_ctf_writer *writer = cw->writer; + +#define ADD(__n, __v) \ + do { \ + if (bt_ctf_writer_add_environment_field(writer, __n, __v))\ + return -1; \ + } while (0) + + ADD("host", header->env.hostname); + ADD("sysname", "Linux"); + ADD("release", header->env.os_release); + ADD("machine", header->env.arch); + + /* TODO ADD nodename, version */ + +#undef ADD + return 0; +} + +static int ctf_writer__setup_clock(struct ctf_writer *cw) +{ + struct bt_ctf_clock *clock = cw->clock; + + bt_ctf_clock_set_description(clock, "perf clock"); + +#define SET(__n, __v) \ + do { \ + if (bt_ctf_clock_set_##__n(clock, __v)) \ + return -1; \ + } while (0) + + SET(frequency, 1000000000); + SET(offset_s, 0); + SET(offset, 0); + SET(precision, 10); + SET(is_absolute, 0); + +#undef SET + return 0; +} + +static struct bt_ctf_field_type *create_int_type(int size, bool sign, bool hex) +{ + struct bt_ctf_field_type *type; + + type = bt_ctf_field_type_integer_create(size); + if (!type) + return NULL; + + if (sign && + bt_ctf_field_type_integer_set_signed(type, 1)) + goto err; + + if (hex && + bt_ctf_field_type_integer_set_base(type, BT_CTF_INTEGER_BASE_HEXADECIMAL)) + goto err; + + pr2("Created type: INTEGER %d-bit %ssigned %s\n", + size, sign ? "un" : "", hex ? "hex" : ""); + return type; + +err: + bt_ctf_field_type_put(type); + return NULL; +} + +static void ctf_writer__cleanup_data(struct ctf_writer *cw) +{ + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(cw->data.array); i++) + bt_ctf_field_type_put(cw->data.array[i]); +} + +static int ctf_writer__init_data(struct ctf_writer *cw) +{ +#define CREATE_INT_TYPE(type, size, sign, hex) \ + do { \ + (type) = create_int_type(size, sign, hex); \ + if (!(type)) \ + goto err; \ + } while (0) + + CREATE_INT_TYPE(cw->data.s64, 64, true, false); + CREATE_INT_TYPE(cw->data.u64, 64, false, false); + CREATE_INT_TYPE(cw->data.s32, 32, true, false); + CREATE_INT_TYPE(cw->data.u32, 32, false, false); + CREATE_INT_TYPE(cw->data.u64_hex, 64, false, true); + +#undef CREATE_INT_TYPE + + cw->data.string = bt_ctf_field_type_string_create(); + if (cw->data.string) + return 0; + +err: + ctf_writer__cleanup_data(cw); + pr_err("Failed to create data types.\n"); + return -1; +} + +static void ctf_writer__cleanup(struct ctf_writer *cw) +{ + ctf_writer__cleanup_data(cw); + + bt_ctf_clock_put(cw->clock); + bt_ctf_stream_put(cw->stream); + bt_ctf_stream_class_put(cw->stream_class); + bt_ctf_writer_put(cw->writer); + + /* and NULL all the pointers */ + memset(cw, 0, sizeof(*cw)); +} + +static int ctf_writer__init(struct ctf_writer *cw, const char *path) +{ + struct bt_ctf_writer *writer; + struct bt_ctf_stream_class *stream_class; + struct bt_ctf_stream *stream; + struct bt_ctf_clock *clock; + + /* CTF writer */ + writer = bt_ctf_writer_create(path); + if (!writer) + goto err; + + cw->writer = writer; + + /* CTF clock */ + clock = bt_ctf_clock_create("perf_clock"); + if (!clock) { + pr("Failed to create CTF clock.\n"); + goto err_cleanup; + } + + cw->clock = clock; + + if (ctf_writer__setup_clock(cw)) { + pr("Failed to setup CTF clock.\n"); + goto err_cleanup; + } + + /* CTF stream class */ + stream_class = bt_ctf_stream_class_create("perf_stream"); + if (!stream_class) { + pr("Failed to create CTF stream class.\n"); + goto err_cleanup; + } + + cw->stream_class = stream_class; + + /* CTF clock stream setup */ + if (bt_ctf_stream_class_set_clock(stream_class, clock)) { + pr("Failed to assign CTF clock to stream class.\n"); + goto err_cleanup; + } + + /* CTF stream instance */ + stream = bt_ctf_writer_create_stream(writer, stream_class); + if (!stream) { + pr("Failed to create CTF stream.\n"); + goto err_cleanup; + } + + cw->stream = stream; + + /* CTF clock writer setup */ + if (bt_ctf_writer_add_clock(writer, clock)) { + pr("Failed to assign CTF clock to writer.\n"); + goto err_cleanup; + } + + if (!ctf_writer__init_data(cw)) + return 0; + +err_cleanup: + ctf_writer__cleanup(cw); +err: + pr_err("Failed to setup CTF writer.\n"); + return -1; +} + +int bt_convert__perf2ctf(const char *input, const char *path) +{ + struct perf_session *session; + struct perf_data_file file = { + .path = input, + .mode = PERF_DATA_MODE_READ, + }; + struct convert c = { + .tool = { + .sample = process_sample_event, + .mmap = perf_event__process_mmap, + .mmap2 = perf_event__process_mmap2, + .comm = perf_event__process_comm, + .exit = perf_event__process_exit, + .fork = perf_event__process_fork, + .lost = perf_event__process_lost, + .tracing_data = perf_event__process_tracing_data, + .build_id = perf_event__process_build_id, + .ordered_events = true, + .ordering_requires_timestamps = true, + }, + }; + struct ctf_writer *cw = &c.writer; + int err = -1; + + /* CTF writer */ + if (ctf_writer__init(cw, path)) + return -1; + + /* perf.data session */ + session = perf_session__new(&file, 0, NULL); + if (!session) + goto free_writer; + + /* CTF writer env/clock setup */ + if (ctf_writer__setup_env(cw, session)) + goto free_session; + + /* CTF events setup */ + if (setup_events(cw, session)) + goto free_session; + + err = perf_session__process_events(session, &c.tool); + if (!err) + err = bt_ctf_stream_flush(cw->stream); + + fprintf(stderr, + "[ perf data convert: Converted '%s' into CTF data '%s' ]\n", + file.path, path); + + fprintf(stderr, + "[ perf data convert: Converted and wrote %.3f MB (%" PRIu64 " samples) ]\n", + (double) c.events_size / 1024.0 / 1024.0, + c.events_count); + + /* its all good */ +free_session: + perf_session__delete(session); + +free_writer: + ctf_writer__cleanup(cw); + return err; +} diff --git a/tools/perf/util/data-bt.h b/tools/perf/util/data-bt.h new file mode 100644 index 000000000000..8fd419c46c5e --- /dev/null +++ b/tools/perf/util/data-bt.h @@ -0,0 +1,8 @@ +#ifndef __DATA_BT_H +#define __DATA_BT_H +#ifdef HAVE_LIBBABELTRACE_SUPPORT + +int bt_convert__perf2ctf(const char *input_name, const char *to_ctf); + +#endif /* HAVE_LIBBABELTRACE_SUPPORT */ +#endif /* __DATA_BT_H */ diff --git a/tools/perf/util/debug.c b/tools/perf/util/debug.c index ba357f3226c6..c936084b1a18 100644 --- a/tools/perf/util/debug.c +++ b/tools/perf/util/debug.c @@ -19,6 +19,7 @@ int verbose; bool dump_trace = false, quiet = false; int debug_ordered_events; +int debug_data_convert; static int _eprintf(int level, int var, const char *fmt, va_list args) { @@ -144,7 +145,8 @@ static struct debug_variable { int *ptr; } debug_variables[] = { { .name = "verbose", .ptr = &verbose }, - { .name = "ordered-events", .ptr = &debug_ordered_events}, + { .name = "ordered-events", .ptr = &debug_ordered_events }, + { .name = "data-convert", .ptr = &debug_data_convert }, { .name = NULL, } }; diff --git a/tools/perf/util/debug.h b/tools/perf/util/debug.h index 6944ea3a119b..e7778785b607 100644 --- a/tools/perf/util/debug.h +++ b/tools/perf/util/debug.h @@ -11,6 +11,7 @@ extern int verbose; extern bool quiet, dump_trace; extern int debug_ordered_events; +extern int debug_data_convert; #ifndef pr_fmt #define pr_fmt(fmt) fmt -- 1.8.3.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 4/5] perf tools data: Add a 'perf' prefix to the generic fields 2014-08-06 12:13 [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa ` (2 preceding siblings ...) 2014-08-06 12:13 ` [PATCH 3/5] perf tools data: Add perf data to CTF convertion support Jiri Olsa @ 2014-08-06 12:13 ` Jiri Olsa 2014-08-06 12:13 ` [PATCH 5/5] perf tools data: Add tracepoint events fields CTF convertion support Jiri Olsa 2014-08-19 15:22 ` [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa 5 siblings, 0 replies; 11+ messages in thread From: Jiri Olsa @ 2014-08-06 12:13 UTC (permalink / raw) To: linux-kernel Cc: Sebastian Andrzej Siewior, Arnaldo Carvalho de Melo, David Ahern, Dominique Toupin, Frederic Weisbecker, Jeremie Galarneau, Jiri Olsa, Mathieu Desnoyers, Namhyung Kim, Paul Mackerras, Peter Zijlstra, Tom Zanussi From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Some of the tracers bring their own id or pid fields and we can end up having two of them. This patch adds a "perf_" prefix to the 'generic' fields so we avoid a clash of the member names. The change is visible in the babeltrace output: Before: $ babeltrace ./ctf-data/ [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 } [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 } ... Now: $ babeltrace ./ctf-data/ [03:19:13.962131936] (+0.000001935) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 8 } [03:19:13.962133732] (+0.000001796) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 114 } ... Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: David Ahern <dsahern@gmail.com> Cc: Dominique Toupin <dominique.toupin@ericsson.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Tom Zanussi <tzanussi@gmail.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by: Jiri Olsa <jolsa@kernel.org> --- tools/perf/util/data-bt.c | 42 ++++++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/tools/perf/util/data-bt.c b/tools/perf/util/data-bt.c index 67d7be0e76bc..2f2ebfcdc9ef 100644 --- a/tools/perf/util/data-bt.c +++ b/tools/perf/util/data-bt.c @@ -147,60 +147,62 @@ static int add_generic_values(struct ctf_writer *cw, */ if (type & PERF_SAMPLE_IP) { - ret = value_set_u64_hex(cw, event, "ip", sample->ip); + ret = value_set_u64_hex(cw, event, "perf_ip", sample->ip); if (ret) return -1; } if (type & PERF_SAMPLE_TID) { - ret = value_set_s32(cw, event, "tid", sample->tid); + ret = value_set_s32(cw, event, "perf_tid", sample->tid); if (ret) return -1; - ret = value_set_s32(cw, event, "pid", sample->tid); + ret = value_set_s32(cw, event, "perf_pid", sample->tid); if (ret) return -1; } if ((type & PERF_SAMPLE_ID) || (type & PERF_SAMPLE_IDENTIFIER)) { - ret = value_set_u64(cw, event, "id", sample->id); + ret = value_set_u64(cw, event, "perf_id", sample->id); if (ret) return -1; } if (type & PERF_SAMPLE_STREAM_ID) { - ret = value_set_u64(cw, event, "stream_id", sample->id); + ret = value_set_u64(cw, event, "perf_stream_id", sample->id); if (ret) return -1; } if (type & PERF_SAMPLE_CPU) { - ret = value_set_u32(cw, event, "cpu", sample->cpu); + ret = value_set_u32(cw, event, "perf_cpu", sample->cpu); if (ret) return -1; } if (type & PERF_SAMPLE_PERIOD) { - ret = value_set_u64(cw, event, "period", sample->period); + ret = value_set_u64(cw, event, "perf_period", sample->period); if (ret) return -1; } if (type & PERF_SAMPLE_WEIGHT) { - ret = value_set_u64(cw, event, "weight", sample->weight); + ret = value_set_u64(cw, event, "perf_weight", sample->weight); if (ret) return -1; } if (type & PERF_SAMPLE_DATA_SRC) { - ret = value_set_u64(cw, event, "data_src", sample->data_src); + ret = value_set_u64(cw, event, "perf_data_src", + sample->data_src); if (ret) return -1; } if (type & PERF_SAMPLE_TRANSACTION) { - ret = value_set_u64(cw, event, "transaction", sample->transaction); + ret = value_set_u64(cw, event, "perf_transaction", + sample->transaction); if (ret) return -1; } @@ -276,34 +278,34 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel, } while (0) if (type & PERF_SAMPLE_IP) - ADD_FIELD(event_class, cw->data.u64_hex, "ip"); + ADD_FIELD(event_class, cw->data.u64_hex, "perf_ip"); if (type & PERF_SAMPLE_TID) { - ADD_FIELD(event_class, cw->data.s32, "tid"); - ADD_FIELD(event_class, cw->data.s32, "pid"); + ADD_FIELD(event_class, cw->data.s32, "perf_tid"); + ADD_FIELD(event_class, cw->data.s32, "perf_pid"); } if ((type & PERF_SAMPLE_ID) || (type & PERF_SAMPLE_IDENTIFIER)) - ADD_FIELD(event_class, cw->data.u64, "id"); + ADD_FIELD(event_class, cw->data.u64, "perf_id"); if (type & PERF_SAMPLE_STREAM_ID) - ADD_FIELD(event_class, cw->data.u64, "stream_id"); + ADD_FIELD(event_class, cw->data.u64, "perf_stream_id"); if (type & PERF_SAMPLE_CPU) - ADD_FIELD(event_class, cw->data.u32, "cpu"); + ADD_FIELD(event_class, cw->data.u32, "perf_cpu"); if (type & PERF_SAMPLE_PERIOD) - ADD_FIELD(event_class, cw->data.u64, "period"); + ADD_FIELD(event_class, cw->data.u64, "perf_period"); if (type & PERF_SAMPLE_WEIGHT) - ADD_FIELD(event_class, cw->data.u64, "weight"); + ADD_FIELD(event_class, cw->data.u64, "perf_weight"); if (type & PERF_SAMPLE_DATA_SRC) - ADD_FIELD(event_class, cw->data.u64, "data_src"); + ADD_FIELD(event_class, cw->data.u64, "perf_data_src"); if (type & PERF_SAMPLE_TRANSACTION) - ADD_FIELD(event_class, cw->data.u64, "transaction"); + ADD_FIELD(event_class, cw->data.u64, "perf_transaction"); #undef ADD_FIELD return 0; -- 1.8.3.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 5/5] perf tools data: Add tracepoint events fields CTF convertion support 2014-08-06 12:13 [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa ` (3 preceding siblings ...) 2014-08-06 12:13 ` [PATCH 4/5] perf tools data: Add a 'perf' prefix to the generic fields Jiri Olsa @ 2014-08-06 12:13 ` Jiri Olsa 2014-08-19 15:22 ` [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa 5 siblings, 0 replies; 11+ messages in thread From: Jiri Olsa @ 2014-08-06 12:13 UTC (permalink / raw) To: linux-kernel Cc: Sebastian Andrzej Siewior, Arnaldo Carvalho de Melo, David Ahern, Dominique Toupin, Frederic Weisbecker, Jeremie Galarneau, Jiri Olsa, Mathieu Desnoyers, Namhyung Kim, Paul Mackerras, Peter Zijlstra, Tom Zanussi From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Adding support to convert tracepoint event fields into CTF event fields. We parse each tracepoint event for CTF conversion and add tracepoint fields as regular CTF event fields, so they appear in babeltrace output like: $ babeltrace ./ctf-data/ ... [09:02:00.950703057] (+?.?????????) sched:sched_stat_runtime: { }, { perf_ip = ... SNIP ... common_type = 298, common_flags = 1, \ common_preempt_count = 0, common_pid = 31813, comm = "perf", pid = 31813, runtime = 458800, vruntime = 52059858071 } ... Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: David Ahern <dsahern@gmail.com> Cc: Dominique Toupin <dominique.toupin@ericsson.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jeremie Galarneau <jgalar@efficios.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Cc: Tom Zanussi <tzanussi@gmail.com> Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- tools/perf/util/data-bt.c | 246 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 246 insertions(+) diff --git a/tools/perf/util/data-bt.c b/tools/perf/util/data-bt.c index 2f2ebfcdc9ef..13ceb99802b3 100644 --- a/tools/perf/util/data-bt.c +++ b/tools/perf/util/data-bt.c @@ -126,6 +126,179 @@ FUNC_VALUE_SET(s64) FUNC_VALUE_SET(u64) __FUNC_VALUE_SET(u64_hex, u64) +static struct bt_ctf_field_type* +get_tracepoint_field_type(struct ctf_writer *cw, struct format_field *field) +{ + unsigned long flags = field->flags; + + if (flags & FIELD_IS_STRING) + return cw->data.string; + + if (!(flags & FIELD_IS_SIGNED)) { + /* unsigned long are mostly pointers */ + if (flags & FIELD_IS_LONG || flags & FIELD_IS_POINTER) + return cw->data.u64_hex; + } + + if (flags & FIELD_IS_SIGNED) { + if (field->size == 8) + return cw->data.s64; + else + return cw->data.s32; + } + + if (field->size == 8) + return cw->data.u64; + else + return cw->data.u32; +} + +static int add_tracepoint_field_value(struct ctf_writer *cw, + struct bt_ctf_event_class *event_class, + struct bt_ctf_event *event, + struct perf_sample *sample, + struct format_field *fmtf) +{ + struct bt_ctf_field_type *type; + struct bt_ctf_field *array_field; + struct bt_ctf_field *field; + const char *name = fmtf->name; + void *data = sample->raw_data; + unsigned long long value_int; + unsigned long flags = fmtf->flags; + unsigned int n_items; + unsigned int i; + unsigned int offset; + unsigned int len; + int ret; + + offset = fmtf->offset; + len = fmtf->size; + if (flags & FIELD_IS_STRING) + flags &= ~FIELD_IS_ARRAY; + + if (flags & FIELD_IS_DYNAMIC) { + unsigned long long tmp_val; + + tmp_val = pevent_read_number(fmtf->event->pevent, + data + offset, len); + offset = tmp_val; + len = offset >> 16; + offset &= 0xffff; + } + + if (flags & FIELD_IS_ARRAY) { + + type = bt_ctf_event_class_get_field_by_name( + event_class, name); + array_field = bt_ctf_field_create(type); + bt_ctf_field_type_put(type); + if (!array_field) { + pr_err("Failed to create array type %s\n", name); + return -1; + } + + len = fmtf->size / fmtf->arraylen; + n_items = fmtf->arraylen; + } else { + n_items = 1; + array_field = NULL; + } + + type = get_tracepoint_field_type(cw, fmtf); + + for (i = 0; i < n_items; i++) { + if (!(flags & FIELD_IS_STRING)) + value_int = pevent_read_number( + fmtf->event->pevent, + data + offset + i * len, len); + + if (flags & FIELD_IS_ARRAY) + field = bt_ctf_field_array_get_field(array_field, i); + else + field = bt_ctf_field_create(type); + + if (!field) { + pr_err("failed to create a field %s\n", name); + return -1; + } + + if (flags & FIELD_IS_STRING) + ret = bt_ctf_field_string_set_value(field, + data + offset + i * len); + else if (!(flags & FIELD_IS_SIGNED)) + ret = bt_ctf_field_unsigned_integer_set_value( + field, value_int); + else + ret = bt_ctf_field_signed_integer_set_value( + field, value_int); + if (ret) { + pr_err("failed to set file value %s\n", name); + goto err_put_field; + } + if (!(flags & FIELD_IS_ARRAY)) { + ret = bt_ctf_event_set_payload(event, name, field); + if (ret) { + pr_err("failed to set payload %s\n", name); + goto err_put_field; + } + } + bt_ctf_field_put(field); + } + if (flags & FIELD_IS_ARRAY) { + ret = bt_ctf_event_set_payload(event, name, array_field); + if (ret) { + pr_err("Failed add payload array %s\n", name); + return -1; + } + bt_ctf_field_put(array_field); + } + return 0; + +err_put_field: + bt_ctf_field_put(field); + return -1; +} + +static int add_tracepoint_fields_values(struct ctf_writer *cw, + struct bt_ctf_event_class *event_class, + struct bt_ctf_event *event, + struct format_field *fields, + struct perf_sample *sample) +{ + struct format_field *field; + int ret; + + for (field = fields; field; field = field->next) { + ret = add_tracepoint_field_value(cw, event_class, event, sample, + field); + if (ret) + return -1; + } + return 0; +} + +static int add_tracepoint_values(struct ctf_writer *cw, + struct bt_ctf_event_class *event_class, + struct bt_ctf_event *event, + struct perf_evsel *evsel, + struct perf_sample *sample) +{ + struct format_field *common_fields, *fields; + int ret; + + common_fields = evsel->tp_format->format.common_fields; + fields = evsel->tp_format->format.fields; + + ret = add_tracepoint_fields_values(cw, event_class, event, + common_fields, sample); + if (!ret) + ret = add_tracepoint_fields_values(cw, event_class, event, + fields, sample); + + return ret; +} + static int add_generic_values(struct ctf_writer *cw, struct bt_ctf_event *event, struct perf_evsel *evsel, @@ -246,11 +419,78 @@ static int process_sample_event(struct perf_tool *tool, if (ret) return -1; + if (evsel->attr.type == PERF_TYPE_TRACEPOINT) { + ret = add_tracepoint_values(cw, event_class, event, + evsel, sample); + if (ret) + return -1; + } + bt_ctf_stream_append_event(cw->stream, event); bt_ctf_event_put(event); return 0; } +static int add_tracepoint_fields_types(struct ctf_writer *cw, + struct format_field *fields, + struct bt_ctf_event_class *event_class) +{ + struct format_field *field; + int ret; + + for (field = fields; field; field = field->next) { + struct bt_ctf_field_type *type; + unsigned long flags = field->flags; + + pr2(" field '%s'\n", field->name); + + type = get_tracepoint_field_type(cw, field); + if (!type) + return -1; + + /* + * A string is an array of chars. For this we use the string + * type and don't care that it is an array. What we don't + * support is an array of strings. + */ + if (flags & FIELD_IS_STRING) + flags &= ~FIELD_IS_ARRAY; + + if (flags & FIELD_IS_ARRAY) + type = bt_ctf_field_type_array_create(type, field->arraylen); + + ret = bt_ctf_event_class_add_field(event_class, type, + field->name); + + if (flags & FIELD_IS_ARRAY) + bt_ctf_field_type_put(type); + + if (ret) { + pr_err("Failed to add field '%s\n", field->name); + return -1; + } + } + + return 0; +} + +static int add_tracepoint_types(struct ctf_writer *cw, + struct perf_evsel *evsel, + struct bt_ctf_event_class *class) +{ + struct format_field *common_fields, *fields; + int ret; + + common_fields = evsel->tp_format->format.common_fields; + fields = evsel->tp_format->format.fields; + + ret = add_tracepoint_fields_types(cw, common_fields, class); + if (!ret) + ret = add_tracepoint_fields_types(cw, fields, class); + + return ret; +} + static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel, struct bt_ctf_event_class *event_class) { @@ -328,6 +568,12 @@ static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel) if (ret) goto err; + if (evsel->attr.type == PERF_TYPE_TRACEPOINT) { + ret = add_tracepoint_types(cw, evsel, event_class); + if (ret) + goto err; + } + ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class); if (ret) { pr("Failed to add event class into stream.\n"); -- 1.8.3.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-08-06 12:13 [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa ` (4 preceding siblings ...) 2014-08-06 12:13 ` [PATCH 5/5] perf tools data: Add tracepoint events fields CTF convertion support Jiri Olsa @ 2014-08-19 15:22 ` Jiri Olsa 5 siblings, 0 replies; 11+ messages in thread From: Jiri Olsa @ 2014-08-19 15:22 UTC (permalink / raw) To: Jiri Olsa Cc: linux-kernel, Arnaldo Carvalho de Melo, David Ahern, Dominique Toupin, Frederic Weisbecker, Jeremie Galarneau, Mathieu Desnoyers, Namhyung Kim, Paul Mackerras, Peter Zijlstra, Sebastian Andrzej Siewior, Tom Zanussi On Wed, Aug 06, 2014 at 02:13:54PM +0200, Jiri Olsa wrote: > hi, > this is initial post of the CTF converter for perf. > Basically adding 'perf data convert' command to allow > conversion of perf data file into CTF [1] data. > > To convert perf data file run: > $ perf data convert --to-ctf=./ctf-data/ > [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ] > [ perf data convert: Converted and wrote 11.268 MB (100230 samples) ] > > To display converted CTF data run [2]: > $ babeltrace ./ctf-data/ > [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 } > [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 } > [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 } > [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 } > [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 } > [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 } > [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 } > [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 } > [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 } > > It has still some limitations (noted in patches), but it has > reached the point of the useful perf feature. More features > and supports are about to come and it'll be better to continue > publicly. > > Also reachable in here: > git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git > perf/core_ctf_convert so I've put perf converted CTF stream into the eclipse CTF trace viewer and got it displayed with the 'Generic CTF trace' type The other type 'LTTng kernel trace' seems to set some rules for the CTF fields/format the trace has to obey. Is this described somewhere? thanks, jirka ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <53F38C74.4030300@voxpopuli.im>]
* Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion [not found] <53F38C74.4030300@voxpopuli.im> @ 2014-08-20 9:28 ` Jiri Olsa 2014-08-20 19:14 ` Alexandre Montplaisir 0 siblings, 1 reply; 11+ messages in thread From: Jiri Olsa @ 2014-08-20 9:28 UTC (permalink / raw) To: Alexandre Montplaisir Cc: linux-kernel, Dominique Toupin, Mathieu Desnoyers, Sebastian Andrzej Siewior, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo On Tue, Aug 19, 2014 at 01:42:12PM -0400, Alexandre Montplaisir wrote: > Hi Jiri, > > (sorry for breaking the thread, I didn't have the original email, but was > forwarded it) np, some other folks to the CC > > I work on the Eclipse viewer (a.k.a. Trace Compass). > > > so I've put perf converted CTF stream into the eclipse CTF trace viewer > and got it displayed with the 'Generic CTF trace' type > > The other type 'LTTng kernel trace' seems to set some rules for the CTF > fields/format the trace has to obey. Is this described somewhere? > > To identify an LTTng kernel trace, we check if the domain defined in the > metadata is "kernel" [1]. I don't think we documented this anywhere though, > we probably should! ok, easy enough ;-) so I'm guessing this governs the expected CTF layout for event/stream headers/contexts, right? Also judging from the trace display, you have hardcoded specific displays/actions for specific events? That's all connected/specific under trace type? > > Once we have some views or analysis specific to perf CTF traces, we could > definitely add a separate trace type for those too. I guess tracepoints and breakpoints should display something like the standard kernel trace. As for HW events it's usual to display profile infomation as the perf report does: https://perf.wiki.kernel.org/index.php/Tutorial#Sampling_with_perf_record I tried to record/display lttng event perf:cpu:cycles, but got nothing displayed in eclipse. Looks like this data provides only summary count of the event for the workload? thanks, jirka ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-08-20 9:28 ` FW: " Jiri Olsa @ 2014-08-20 19:14 ` Alexandre Montplaisir 2014-11-03 17:58 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 11+ messages in thread From: Alexandre Montplaisir @ 2014-08-20 19:14 UTC (permalink / raw) To: Jiri Olsa Cc: linux-kernel, Dominique Toupin, Mathieu Desnoyers, Sebastian Andrzej Siewior, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo On 08/20/2014 05:28 AM, Jiri Olsa wrote: > > ok, easy enough ;-) so I'm guessing this governs the expected > CTF layout for event/stream headers/contexts, right? Correct, if the domain is "kernel" we then assume that the rest of the trace contains the expected elements of a kernel trace. Of course, one could craft a CTF trace to advertize itself as "kernel" or "ust", and not actually have the layout of that trace type, in which case it would fail parsing later on. > Also judging from the trace display, you have hardcoded specific > displays/actions for specific events? That's all connected/specific > under trace type? Yes the trace type is the main "provider" of functionality. I could go into more details, but we use Eclipse extension points to define which columns to put in the event table, which views are available, etc. for each supported trace type. >> Once we have some views or analysis specific to perf CTF traces, we could >> definitely add a separate trace type for those too. > I guess tracepoints and breakpoints should display something like > the standard kernel trace. As for HW events it's usual to display > profile infomation as the perf report does: > https://perf.wiki.kernel.org/index.php/Tutorial#Sampling_with_perf_record Interesting, I haven't tried the perf CTF output yet, but I could see it using the Statistics view (which by default just prints the % of events, per event type) to print the results of the different "perf reports", calculated from the CTF events. Eventually with pie charts! > I tried to record/display lttng event perf:cpu:cycles, but got nothing > displayed in eclipse. Looks like this data provides only summary count > of the event for the workload? Just to be sure I understand, you recorded an LTTng kernel trace, in which you enabled the "perf:cpu:cycles" context? Did this trace display anything in Babeltrace? It should display the same in the Eclipse viewer, the value of the context will be visible in the "Contents" column in the the table (and in the Properties view), although for now we don't make any use of it. From what I understand, yes, the value of the different perf:* contexts represents the value of the perf counter at the moment the tracepoint was hit. So you cannot get perf data "in between" trace events when using this method. Cheers, Alexandre ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-08-20 19:14 ` Alexandre Montplaisir @ 2014-11-03 17:58 ` Sebastian Andrzej Siewior 2014-11-04 1:20 ` Alexandre Montplaisir 0 siblings, 1 reply; 11+ messages in thread From: Sebastian Andrzej Siewior @ 2014-11-03 17:58 UTC (permalink / raw) To: Alexandre Montplaisir, Jiri Olsa Cc: linux-kernel, Dominique Toupin, Mathieu Desnoyers, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo On 08/20/2014 09:14 PM, Alexandre Montplaisir wrote: > On 08/20/2014 05:28 AM, Jiri Olsa wrote: >> >> ok, easy enough ;-) so I'm guessing this governs the expected >> CTF layout for event/stream headers/contexts, right? > > Correct, if the domain is "kernel" we then assume that the rest of the > trace contains the expected elements of a kernel trace. So that is one thing that I had to add. > Of course, one could craft a CTF trace to advertize itself as "kernel" > or "ust", and not actually have the layout of that trace type, in which > case it would fail parsing later on. > >> Also judging from the trace display, you have hardcoded specific >> displays/actions for specific events? That's all connected/specific >> under trace type? > > Yes the trace type is the main "provider" of functionality. I could go > into more details, but we use Eclipse extension points to define which > columns to put in the event table, which views are available, etc. for > each supported trace type. > >>> Once we have some views or analysis specific to perf CTF traces, we >>> could >>> definitely add a separate trace type for those too. >> I guess tracepoints and breakpoints should display something like >> the standard kernel trace. As for HW events it's usual to display >> profile infomation as the perf report does: >> >> https://perf.wiki.kernel.org/index.php/Tutorial#Sampling_with_perf_record > > Interesting, I haven't tried the perf CTF output yet, but I could see it > using the Statistics view (which by default just prints the % of events, > per event type) to print the results of the different "perf reports", > calculated from the CTF events. Eventually with pie charts! I did on the linux side: |perf record \ | -e sched:sched_switch \ | -a This gave me perf.data trace. No with the new extension I converted it into CTF data stream (perf data convert -i perf.data --to-ctf ctf) and imported it into eclipse as a new trace. By setting 'domain = "kernel"' I managed to get it accepted as a kernel trace. Additionally I had to: - rename sched:sched_switch -> sched_switch (I dropped the other events) - rename "perf_cpu" to "cpu_id" and move it within "packet context" (had a patch for that but we wanted to merge this later) - I added "prev_tid" with the content of "prev_pid" and I added "next_tid" with the content of "next_pid". Now exclipse does not "freeze" after loading the first entry - I prefixed every entry with _ (so "prev_tid" becomes "_prev_tid") and now it seems to be recognized by the tracing plugin. puh. Now I have something in "cpu usage", "control flow" windows. The cpu_id field change will be addressed soon on our side. Now, the remaining things: The "domain = kernel" thingy (or another identifier if desired) is something we could add. What do we do on the naming convention? The converter takes basically the event names the way they come from perf. That is why we have a "system" prefix. For the member fields, we take all the specific ones as perf serves them. The only exception is for the generic fields which get a "perf_" prefix in order not to clash with the specific ones. And there is this _ prefix in front of every data member. Now that I identified the differences between the CTF from lttng and perf, any suggestions / ideas how this could be solved? > Cheers, > Alexandre Sebastian ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-11-03 17:58 ` Sebastian Andrzej Siewior @ 2014-11-04 1:20 ` Alexandre Montplaisir 2014-11-05 12:50 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 11+ messages in thread From: Alexandre Montplaisir @ 2014-11-04 1:20 UTC (permalink / raw) To: Sebastian Andrzej Siewior, Jiri Olsa Cc: linux-kernel, Dominique Toupin, Mathieu Desnoyers, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo Hi Sebastian, On 11/03/2014 06:58 PM, Sebastian Andrzej Siewior wrote: > [...] > I did on the linux side: > > |perf record \ > | -e sched:sched_switch \ > | -a > > This gave me perf.data trace. No with the new extension I converted it > into CTF data stream (perf data convert -i perf.data --to-ctf ctf) and > imported it into eclipse as a new trace. By setting 'domain = "kernel"' > I managed to get it accepted as a kernel trace. > > Additionally I had to: > - rename sched:sched_switch -> sched_switch (I dropped the other events) > - rename "perf_cpu" to "cpu_id" and move it within "packet context" > (had a patch for that but we wanted to merge this later) > - I added "prev_tid" with the content of "prev_pid" and I added > "next_tid" with the content of "next_pid". Now exclipse does not > "freeze" after loading the first entry > - I prefixed every entry with _ (so "prev_tid" becomes "_prev_tid") and > now it seems to be recognized by the tracing plugin. > > puh. Now I have something in "cpu usage", "control flow" windows. This is really great! Initially, I had believed that we would have needed to add a separate parser plugin, and to consider "perf traces" as a completely different beast from LTTng traces. However if you can get this close to they way LTTng presents its data, then we can probably re-use most of the existing code. In which case we could rename the "LTTng Kernel Trace" type in the UI to simply "Linux Kernel Trace". And that would cover both LTTng kernel traces and CTF-perf traces. > The cpu_id field change will be addressed soon on our side. > Now, the remaining things: > The "domain = kernel" thingy (or another identifier if desired) is > something we could add. Unless the event data is exactly the same, it would be easier to use a different name. Like "kernel-perf" for instance? From the user's point of view, both would still be Linux Kernel Traces, but we could use the domain internally to determine which event/field layout to use. Mathieu, any thoughts on how CTF domains should be namespaced? > What do we do on the naming convention? > > The converter takes basically the event names the way they come from > perf. That is why we have a "system" prefix. > For the member fields, we take all the specific ones as perf serves > them. The only exception is for the generic fields which get a "perf_" > prefix in order not to clash with the specific ones. > And there is this _ prefix in front of every data member. > > Now that I identified the differences between the CTF from lttng and > perf, any suggestions / ideas how this could be solved? I suppose it would be better/cleaner if the event and field names would remain the same, or at least be similar, in the perf.data and perf-CTF formats. If the trace events from both LTTng and perf represent the same thing (and I assume they should, since they come from the same tracepoints, right?), then we could just add a wrapper on the viewer side to decide which event/field names to use, depending on the trace type. Right now, we only define LTTng event and field names: http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/internal/lttng2/kernel/core/LttngStrings.java But if you could for example tell me the perf equivalents of all the strings in that file, I could hack together such wrapper. With that, in theory, perf traces should behave exactly the same as LTTng traces in the viewer! Cheers, Alexandre ps. fyi, we have recently moved the trace viewer code to its own project, now called "Trace Compass". We will still support the Linux Tools plugins for the time being, but all new developments are done in Trace Compass. If you want to check it out: http://eclipse.org/tracecompass ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-11-04 1:20 ` Alexandre Montplaisir @ 2014-11-05 12:50 ` Sebastian Andrzej Siewior 2014-11-05 17:21 ` Mathieu Desnoyers 0 siblings, 1 reply; 11+ messages in thread From: Sebastian Andrzej Siewior @ 2014-11-05 12:50 UTC (permalink / raw) To: Alexandre Montplaisir Cc: Jiri Olsa, linux-kernel, Dominique Toupin, Mathieu Desnoyers, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo * Alexandre Montplaisir | 2014-11-04 02:20:10 [+0100]: >Hi Sebastian, Hi Alexandre, >On 11/03/2014 06:58 PM, Sebastian Andrzej Siewior wrote: >This is really great! Initially, I had believed that we would have >needed to add a separate parser plugin, and to consider "perf traces" >as a completely different beast from LTTng traces. However if you can >get this close to they way LTTng presents its data, then we can >probably re-use most of the existing code. In which case we could >rename the "LTTng Kernel Trace" type in the UI to simply "Linux >Kernel Trace". And that would cover both LTTng kernel traces and >CTF-perf traces. we have now CTF here. So lets see what we do about the naming convention. >>The cpu_id field change will be addressed soon on our side. >>Now, the remaining things: >>The "domain = kernel" thingy (or another identifier if desired) is >>something we could add. > >Unless the event data is exactly the same, it would be easier to use >a different name. Like "kernel-perf" for instance? Some kind of a namespace / identifier is probably not wrong. The lttng tracer added a tracer version probably in case the format changes between version for some reason. Perf comes with the kernel so for this the kernel version should sufficient. >From the user's point of view, both would still be Linux Kernel >Traces, but we could use the domain internally to determine which >event/field layout to use. > >Mathieu, any thoughts on how CTF domains should be namespaced? > >>Now that I identified the differences between the CTF from lttng and >>perf, any suggestions / ideas how this could be solved? > >I suppose it would be better/cleaner if the event and field names >would remain the same, or at least be similar, in the perf.data and >perf-CTF formats. Yes, that would be cool. Especially if we teach perf to record straight to CTF. >If the trace events from both LTTng and perf represent the same thing >(and I assume they should, since they come from the same tracepoints, >right?), then we could just add a wrapper on the viewer side to >decide which event/field names to use, depending on the trace type. > >Right now, we only define LTTng event and field names: >http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/internal/lttng2/kernel/core/LttngStrings.java Okay. So I found this file for linuxtools now let me try tracecompass. The basic renaming should do the job. Then I have to figure out how to compile this thingy… There is this one thing where you go for "tid" while perf says "pid". I guess I could figure that out once I have the rename done. We don't have lttng_statedump_process_state, this look lttng specific. I would have to look if there is a replacement event in perf. I have no idea what we could do about the "unknown" events, say someone enbales skb tracing. But this is probably something for once we are done with the basic integration. >But if you could for example tell me the perf equivalents of all the >strings in that file, I could hack together such wrapper. With that, >in theory, perf traces should behave exactly the same as LTTng traces >in the viewer! Oooh, that would be awesome. So I installed maven but didn't get much further. Let me gather this for you. So first the renaming: diff --git a/LttngStrings.java b/LttngStrings.java --- a/LttngStrings.java +++ b/LttngStrings.java @@ -27,17 +27,17 @@ public interface LttngStrings { /* Event names */ static final String EXIT_SYSCALL = "exit_syscall"; - static final String IRQ_HANDLER_ENTRY = "irq_handler_entry"; - static final String IRQ_HANDLER_EXIT = "irq_handler_exit"; - static final String SOFTIRQ_ENTRY = "softirq_entry"; - static final String SOFTIRQ_EXIT = "softirq_exit"; - static final String SOFTIRQ_RAISE = "softirq_raise"; - static final String SCHED_SWITCH = "sched_switch"; - static final String SCHED_WAKEUP = "sched_wakeup"; - static final String SCHED_WAKEUP_NEW = "sched_wakeup_new"; - static final String SCHED_PROCESS_FORK = "sched_process_fork"; - static final String SCHED_PROCESS_EXIT = "sched_process_exit"; - static final String SCHED_PROCESS_FREE = "sched_process_free"; + static final String IRQ_HANDLER_ENTRY = "irq:irq_handler_entry"; + static final String IRQ_HANDLER_EXIT = "irq:irq_handler_exit"; + static final String SOFTIRQ_ENTRY = "irq:softirq_entry"; + static final String SOFTIRQ_EXIT = "irq:softirq_exit"; + static final String SOFTIRQ_RAISE = "irq:softirq_raise"; + static final String SCHED_SWITCH = "sched:sched_switch"; + static final String SCHED_WAKEUP = "sched:sched_wakeup"; + static final String SCHED_WAKEUP_NEW = "sched:sched_wakeup_new"; + static final String SCHED_PROCESS_FORK = "sched:sched_process_fork"; + static final String SCHED_PROCESS_EXIT = "sched:sched_process_exit"; + static final String SCHED_PROCESS_FREE = "sched:sched_process_free"; static final String STATEDUMP_PROCESS_STATE = "lttng_statedump_process_state"; /* System call names */ I have no idea how exit_syscall is different from irq_handler_exit and I think we have to skip for now on STATEDUMP_PROCESS_STATE. For the syscalls: - static final String SYSCALL_PREFIX = "sys_"; It is bassicaly: syscalls:sys_enter_ syscalls:sys_exit_ depending what you are looking for. - static final String COMPAT_SYSCALL_PREFIX = "compat_sys_"; I haven't found this. Could it be that we don't record the compat_sys at all? - static final String SYS_CLONE = "sys_clone"; here we have syscalls:sys_enter_clone syscalls:sys_exit_clone I guess the enter is what you are looking for. For the fields, this is one event with alle the members we have. Please note that lttng saves the members with the _ prefix and I haven't seen that prefix in that .java file. The members of each event: event { name = "raw_syscalls:sys_enter"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; integer { size = 64; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } id; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } args[6]; }; event { name = "raw_syscalls:sys_exit"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; integer { size = 64; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } id; integer { size = 64; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } ret; }; event { name = "irq:irq_handler_entry"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } irq; string { encoding = UTF8; } name; }; event { name = "irq:irq_handler_exit"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } irq; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } ret; }; event { name = "irq:softirq_entry"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } vec; }; event { name = "irq:softirq_exit"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } vec; }; event { name = "irq:softirq_raise"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } vec; }; event { name = "sched:sched_switch"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; string { encoding = UTF8; } prev_comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } prev_pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } prev_prio; integer { size = 64; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } prev_state; string { encoding = UTF8; } next_comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } next_pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } next_prio; }; event { name = "sched:sched_wakeup"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; string { encoding = UTF8; } comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } prio; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } success; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } target_cpu; }; event { name = "sched:sched_wakeup_new"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; string { encoding = UTF8; } comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } prio; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } success; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } target_cpu; }; event { name = "sched:sched_process_fork"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; string { encoding = UTF8; } parent_comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } parent_pid; string { encoding = UTF8; } child_comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } child_pid; }; event { name = "sched:sched_process_exit"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; string { encoding = UTF8; } comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } prio; }; event { name = "sched:sched_process_free"; integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id; integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags; integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid; string { encoding = UTF8; } comm; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } pid; integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } prio; }; and here are some example from babeltrace output. One example per event: raw_syscalls:sys_enter [03:37:07.579969498] (+?.?????????) raw_syscalls:sys_enter: { cpu_id = 2 }, { perf_ip = 0xFFFFFFFF81020EBC, perf_tid = 30004, perf_pid = 30004, perf_id = 382, perf_period = 1, common_type = 76, common_flags = 0, common_preempt_count = 0, common_pid = 30004, id = 16, args = [ [0] = 0xE, [1] = 0x2400, [2] = 0x0, [3] = 0x0, [4] = 0xA20F00, [5] = 0xA1FDA0 ] } raw_syscalls:sys_exit [03:37:07.579971763] (+0.000002265) raw_syscalls:sys_exit: { cpu_id = 2 }, { perf_ip = 0xFFFFFFFF81020FAD, perf_tid = 30004, perf_pid = 30004, perf_id = 390, perf_period = 1, common_type = 75, common_flags = 0, common_preempt_count = 0, common_pid = 30004, id = 16, ret = 0 } irq:irq_handler_entry [03:37:08.088041486] (+0.000000211) irq:irq_handler_entry: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF810B8C30, perf_tid = 0, perf_pid = 0, perf_id = 396, perf_period = 1, common_type = 112, common_flags = 9, common_preempt_count = 0, common_pid = 0, irq = 47, name = "ahci" } irq:irq_handler_exit [03:37:08.088049506] (+0.000000508) irq:irq_handler_exit: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF810B8C03, perf_tid = 0, perf_pid = 0, perf_id = 404, perf_period = 1, common_type = 111, common_flags = 9, common_preempt_count = 0, common_pid = 0, irq = 47, ret = 1 } irq:softirq_entry [03:37:07.583335617] (+0.000001100) irq:softirq_entry: { cpu_id = 3 }, { perf_ip = 0xFFFFFFFF8106A6DC, perf_tid = 0, perf_pid = 0, perf_id = 415, perf_period = 1, common_type = 110, common_flags = 16, common_preempt_count = 0, common_pid = 0, vec = 1 } irq:softirq_exit [03:37:07.583336555] (+0.000000619) irq:softirq_exit: { cpu_id = 2 }, { perf_ip = 0xFFFFFFFF8106A6AC, perf_tid = 30004, perf_pid = 30004, perf_id = 422, perf_period = 1, common_type = 109, common_flags = 16, common_preempt_count = 0, common_pid = 30004, vec = 1 } irq:softirq_raise [03:37:07.583333374] (+0.000002067) irq:softirq_raise: { cpu_id = 3 }, { perf_ip = 0xFFFFFFFF8106AA7C, perf_tid = 0, perf_pid = 0, perf_id = 431, perf_period = 1, common_type = 108, common_flags = 9, common_preempt_count = 0, common_pid = 0, vec = 1 } sched:sched_switch [03:37:07.583339259] (+0.000000046) sched:sched_switch: { cpu_id = 3 }, { perf_ip = 0xFFFFFFFF81508B48, perf_tid = 0, perf_pid = 0, perf_id = 439, perf_period = 1, common_type = 312, common_flags = 1, common_preempt_count = 0, common_pid = 0, prev_comm = "swapper/3", prev_pid = 0, prev_prio = 120, prev_state = 0, next_comm = "rcu_sched", next_pid = 7, next_prio = 120 } sched:sched_wakeup [03:37:07.583336717] (+0.000000162) sched:sched_wakeup: { cpu_id = 3 }, { perf_ip = 0xFFFFFFFF81092112, perf_tid = 0, perf_pid = 0, perf_id = 447, perf_period = 1, common_type = 314, common_flags = 53, common_preempt_count = 0, common_pid = 0, comm = "rcu_sched", pid = 7, prio = 120, success = 1, target_cpu = 3 } sched:sched_wakeup_new [04:16:12.328543282] (+0.000005791) sched:sched_wakeup_new: { cpu_id = 2 }, { perf_ip = 0xFFFFFFFF810944B2, perf_tid = 2819, perf_pid = 2819 , perf_id = 486, perf_period = 1, common_type = 313, common_flags = 1, common_preempt_count = 0, common_pid = 2819, comm = "bash", pid = 32604, prio = 120, success = 1, target_cpu = 7 } sched:sched_process_fork [04:16:12.328537491] (+?.?????????) sched:sched_process_fork: { cpu_id = 2 }, { perf_ip = 0xFFFFFFFF810649D0, perf_tid = 2819, perf_pid = 28 19, perf_id = 494, perf_period = 1, common_type = 306, common_flags = 0, common_preempt_count = 0, common_pid = 2819, parent_comm = "bash", parent_pid = 2819, child_comm = "bash", child_pid = 32604 } sched:sched_process_exit [04:16:12.350957386] (+0.022414104) sched:sched_process_exit: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF81067FED, perf_tid = 32604, perf_pid = 32604, perf_id = 500, perf_period = 1, common_type = 309, common_flags = 0, common_preempt_count = 0, common_pid = 32604, comm = "ps", pid = 32604, prio = 120 } sched:sched_process_free [04:16:12.382175653] (+0.031218267) sched:sched_process_free: { cpu_id = 2 }, { perf_ip = 0xFFFFFFFF810663FD, perf_tid = 18, perf_pid = 18, perf_id = 510, perf_period = 1, common_type = 310, common_flags = 16, common_preempt_count = 0, common_pid = 18, comm = "ps", pid = 32604, prio = 120 } The complete babeltrace output from my sample trace is at https://breakpoint.cc/perf-ctf >Cheers, >Alexandre Sebastian ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-11-05 12:50 ` Sebastian Andrzej Siewior @ 2014-11-05 17:21 ` Mathieu Desnoyers 2014-11-06 4:53 ` Alexandre Montplaisir 2014-11-13 19:24 ` Sebastian Andrzej Siewior 0 siblings, 2 replies; 11+ messages in thread From: Mathieu Desnoyers @ 2014-11-05 17:21 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Alexandre Montplaisir, Jiri Olsa, linux-kernel, Dominique Toupin, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo ----- Original Message ----- > From: "Sebastian Andrzej Siewior" <bigeasy@linutronix.de> > To: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im> > Cc: "Jiri Olsa" <jolsa@redhat.com>, linux-kernel@vger.kernel.org, "Dominique Toupin" <dominique.toupin@ericsson.com>, > "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>, "Tom Zanussi" <tzanussi@gmail.com>, "Jeremie Galarneau" > <jgalar@efficios.com>, "David Ahern" <dsahern@gmail.com>, "Arnaldo Carvalho de Melo" <acme@redhat.com> > Sent: Wednesday, November 5, 2014 7:50:28 AM > Subject: Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion > > * Alexandre Montplaisir | 2014-11-04 02:20:10 [+0100]: > > >Hi Sebastian, > Hi Alexandre, Hi! Sorry for jumping in late in the discussion. I really wanted to consider the various impact of tracepoint semantic before answering. > > >On 11/03/2014 06:58 PM, Sebastian Andrzej Siewior wrote: > >This is really great! Initially, I had believed that we would have > >needed to add a separate parser plugin, and to consider "perf traces" > >as a completely different beast from LTTng traces. However if you can > >get this close to they way LTTng presents its data, then we can > >probably re-use most of the existing code. In which case we could > >rename the "LTTng Kernel Trace" type in the UI to simply "Linux > >Kernel Trace". And that would cover both LTTng kernel traces and > >CTF-perf traces. > > we have now CTF here. So lets see what we do about the naming > convention. > > >>The cpu_id field change will be addressed soon on our side. > >>Now, the remaining things: > >>The "domain = kernel" thingy (or another identifier if desired) is > >>something we could add. > > > >Unless the event data is exactly the same, it would be easier to use > >a different name. Like "kernel-perf" for instance? > > Some kind of a namespace / identifier is probably not wrong. The lttng > tracer added a tracer version probably in case the format changes > between version for some reason. Perf comes with the kernel so for this > the kernel version should sufficient. Yes, using the kernel version for Perf makes sense. I reach a similar conclusion for LTTng: we should add tracepoint semantic versioning somewhere in the CTF metadata, because the semantic of an event can change based on the LTTng version, and based on which kernel version LTTng is tracing. A very good example is the semantic of the sched_wakeup event. It has changed due to scheduler code modification, and is now called from an IPI context, which changes its semantic (not called from the same PID). Unfortunately, there is little we can do besides checking the kernel version to detect the semantic change from the trace viewer side, because neither the event nor the field names have changed. The trace viewer could therefore care about the following information to identify the semantic of a trace: - Tracer name (e.g. lttng or perf), - Domain (e.g. kernel or userspace), - Tracepoint versioning (e.g. kernel version for Perf). Because CTF supports both kernel and userspace tracing, we also want to solve this semantic detection problem both for the kernel and userspace. Therefore, we should consider how the userspace tracepoints could save version information in the user-space metadata too. Since we have traces shared across applications (per user-ID buffers) in lttng-ust, the semantic info, and therefore the versioning, should be done on a per-provider (or per-event) basis, rather than trace-wide, because a single trace could contain events from various applications, each with their own set of providers, therefore each with their versioning info. So if we apply this description scheme to the kernel tracing case, this would mean that each event in the CTF metadata would have version information. For Perf, this could very well be the kernel version that we simply repeat for each event metadata entry. For LTTng-modules, we would have our own versioning that is independent of the kernel version, since the semantic of the events we expose can change for a given kernel version as lttng-modules evolves. In summary, for perf it would be really easy: just repeat the kernel version in a new attribute attached to each event in the metadata. For LTTng we would have the flexibility to have our own version numbers in there. This would also cover the case of userspace tracing, allowing each application to advertise their tracepoint provider semantic changes through versioning. > > >From the user's point of view, both would still be Linux Kernel > >Traces, but we could use the domain internally to determine which > >event/field layout to use. > > > >Mathieu, any thoughts on how CTF domains should be namespaced? (see above) > > > >>Now that I identified the differences between the CTF from lttng and > >>perf, any suggestions / ideas how this could be solved? > > > >I suppose it would be better/cleaner if the event and field names > >would remain the same, or at least be similar, in the perf.data and > >perf-CTF formats. > > Yes, that would be cool. Especially if we teach perf to record straight > to CTF. > > >If the trace events from both LTTng and perf represent the same thing > >(and I assume they should, since they come from the same tracepoints, > >right?), then we could just add a wrapper on the viewer side to > >decide which event/field names to use, depending on the trace type. I think we might want to keep a different semantic namespace for perf and lttng, because LTTng has the luxury to change event semantic mapping between minor LTTng versions in order to add/remove/tweak event content as necessary, and Perf is really tied to each kernel version it is shipped with. > > > >Right now, we only define LTTng event and field names: > >http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/internal/lttng2/kernel/core/LttngStrings.java > > Okay. So I found this file for linuxtools now let me try tracecompass. > The basic renaming should do the job. Then I have to figure out how to > compile this thingy… > > There is this one thing where you go for "tid" while perf says "pid". I > guess I could figure that out once I have the rename done. LTTng uses the semantic presented to user-space to identify threads and processes. What you find in /proc is what you find in a LTTng trace. The tracepoint semantic used by perf and ftrace uses the kernel-internal meaning of pid = thread ID, pgid = process ID, which differs from what is visible from user-space. I guess it's up to you to decide if you want to stick to the kernel-internal semantic, or switch to the user-visible (/proc) semantic for perf traces. > We don't have lttng_statedump_process_state, this look lttng specific. I > would have to look if there is a replacement event in perf. Not that I am aware of. Perf tends to add fields to each records to keep track of extra state. LTTng can also do that by dynamically attaching context information, but it also supports dumping the initial system state, thus allowing trace viewers to reconstruct the system state by reading the trace, starting with the state dump events at the beginning. > > I have no idea what we could do about the "unknown" events, say someone > enbales skb tracing. But this is probably something for once we are > done with the basic integration. > > >But if you could for example tell me the perf equivalents of all the > >strings in that file, I could hack together such wrapper. With that, > >in theory, perf traces should behave exactly the same as LTTng traces > >in the viewer! Ideally, the Trace Compass views should only care about a model of the OS. Populating this model can be done by various "state gathering" plugins, e.g. one for lttng, one for perf, which know about versioning and semantic of the events contained in each trace. [...] > For the fields, this is one event with alle the members we have. Please > note that lttng saves the members with the _ prefix and I haven't seen > that prefix in that .java file. The members of each event: Yeah, the _ prefix for event names. This is one decision I would like to find a way to revert, but we'll have to live with it unfortunately for CTF 1.8. The issue it's trying to fix is to allow having fields named "event" that don't clash with the "event" reserved keyword. When I added the _ prefix, I did it like this in the CTF spec: "Replacing reserved keywords with underscore-prefixed field names is recommended. Fields starting with an underscore should have their leading underscore removed by the CTF trace readers." Unfortunately, this introduces semantic corner-cases for event names that would indeed start with an underscore, unless they are prefixed with double-underscore in the metadata. So far, the only fix I see to this situation is to eventually do a CTF 1.9, and add the notion of a $ prefix to the grammar (which is not part of the symbols accepted for an identifier) to be used as a field name prefix that ensures there is no clash with reserved keywords. I'm very open to suggestions there through, and I'm really not in a hurry to release a new CTF spec version (we should only do so when we have a batch of changes that are required, because it will require all trace readers to be updated). Thanks! Mathieu > >Cheers, > >Alexandre > > Sebastian > -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-11-05 17:21 ` Mathieu Desnoyers @ 2014-11-06 4:53 ` Alexandre Montplaisir 2014-11-13 19:24 ` Sebastian Andrzej Siewior 1 sibling, 0 replies; 11+ messages in thread From: Alexandre Montplaisir @ 2014-11-06 4:53 UTC (permalink / raw) To: Mathieu Desnoyers, Sebastian Andrzej Siewior Cc: Jiri Olsa, linux-kernel, Dominique Toupin, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo Hi Mathieu, On 11/05/2014 06:21 PM, Mathieu Desnoyers wrote: > [...] >>>> The cpu_id field change will be addressed soon on our side. >>>> Now, the remaining things: >>>> The "domain = kernel" thingy (or another identifier if desired) is >>>> something we could add. >>> Unless the event data is exactly the same, it would be easier to use >>> a different name. Like "kernel-perf" for instance? >> Some kind of a namespace / identifier is probably not wrong. The lttng >> tracer added a tracer version probably in case the format changes >> between version for some reason. Perf comes with the kernel so for this >> the kernel version should sufficient. > Yes, using the kernel version for Perf makes sense. I reach a similar > conclusion for LTTng: we should add tracepoint semantic versioning > somewhere in the CTF metadata, because the semantic of an event can > change based on the LTTng version, and based on which kernel version > LTTng is tracing. > > A very good example is the semantic of the sched_wakeup event. It has > changed due to scheduler code modification, and is now called from an > IPI context, which changes its semantic (not called from the same > PID). Unfortunately, there is little we can do besides checking the > kernel version to detect the semantic change from the trace viewer > side, because neither the event nor the field names have changed. > > The trace viewer could therefore care about the following information > to identify the semantic of a trace: > > - Tracer name (e.g. lttng or perf), > - Domain (e.g. kernel or userspace), > - Tracepoint versioning (e.g. kernel version for Perf). Sounds good. So perf-CTF traces could still use the "kernel" domain, but the CTF environment metadata would also mention the tracer, which could be so far either lttng or perf. For now we only look at the domain to infer the trace type, but we could also look at the tracer, and tracer version, to determine which event and field naming to use for the analysis. I can also see how in general, versioning the "instrumentation" of an instrumented program could be useful. For example, LTTng changed the name of their syscall events in 2.6. The event still represents the same thing from an analysis's point of view, only the name changed. > Because CTF supports both kernel and userspace tracing, we also want > to solve this semantic detection problem both for the kernel and > userspace. Therefore, we should consider how the userspace > tracepoints could save version information in the user-space metadata > too. > > Since we have traces shared across applications (per user-ID buffers) > in lttng-ust, the semantic info, and therefore the versioning, should > be done on a per-provider (or per-event) basis, rather than trace-wide, > because a single trace could contain events from various applications, > each with their own set of providers, therefore each with their > versioning info. Hmm, where would this per-tracepoint version come from? From the version of the application? From a new "instrumentation version" defined somewhere? Or would the maintainers of the application have to manually version every single tracepoint in their program? Per-tracepoint versioning, at first glance, seems a bit heavy. I'd have to understand more about it to make an informed opinion though ;) But this seems to be a problem for userspace traces only, right? Because with kernel traces 1) the tracers put the kernel version in the environment metadata and 2) you can't have more than one kernel provider in the same CTF trace (can you?) But from a trace viewer's analysis point of view, I think it would make sense. If events in the trace supply a version (in addition to its name/type), then the analysis may decide to handle different versions of an event in different ways. > > So if we apply this description scheme to the kernel tracing case, > this would mean that each event in the CTF metadata would have > version information. For Perf, this could very well be the kernel > version that we simply repeat for each event metadata entry. For > LTTng-modules, we would have our own versioning that is independent > of the kernel version, since the semantic of the events we expose > can change for a given kernel version as lttng-modules evolves. > > In summary, for perf it would be really easy: just repeat the > kernel version in a new attribute attached to each event in the > metadata. For LTTng we would have the flexibility to have our own > version numbers in there. This would also cover the case of > userspace tracing, allowing each application to advertise their > tracepoint provider semantic changes through versioning. > >> >From the user's point of view, both would still be Linux Kernel >>> Traces, but we could use the domain internally to determine which >>> event/field layout to use. >>> >>> Mathieu, any thoughts on how CTF domains should be namespaced? > (see above) > >>>> Now that I identified the differences between the CTF from lttng and >>>> perf, any suggestions / ideas how this could be solved? >>> I suppose it would be better/cleaner if the event and field names >>> would remain the same, or at least be similar, in the perf.data and >>> perf-CTF formats. >> Yes, that would be cool. Especially if we teach perf to record straight >> to CTF. >> >>> If the trace events from both LTTng and perf represent the same thing >>> (and I assume they should, since they come from the same tracepoints, >>> right?), then we could just add a wrapper on the viewer side to >>> decide which event/field names to use, depending on the trace type. > I think we might want to keep a different semantic namespace for > perf and lttng, because LTTng has the luxury to change event semantic > mapping between minor LTTng versions in order to add/remove/tweak event > content as necessary, and Perf is really tied to each kernel version > it is shipped with. > >>> Right now, we only define LTTng event and field names: >>> http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/internal/lttng2/kernel/core/LttngStrings.java >> Okay. So I found this file for linuxtools now let me try tracecompass. >> The basic renaming should do the job. Then I have to figure out how to >> compile this thingy… >> >> There is this one thing where you go for "tid" while perf says "pid". I >> guess I could figure that out once I have the rename done. > LTTng uses the semantic presented to user-space to identify threads and > processes. What you find in /proc is what you find in a LTTng trace. The > tracepoint semantic used by perf and ftrace uses the kernel-internal > meaning of pid = thread ID, pgid = process ID, which differs from what is > visible from user-space. > > I guess it's up to you to decide if you want to stick to the kernel-internal > semantic, or switch to the user-visible (/proc) semantic for perf traces. This is something I will have to look more into. We do use TIDs for most of the kernel analysis, because that is what LTTng is usually providing, but we also track PID's, with events like the statedump and fork's. We just need to make sure we match the field values to the right thing. > >> We don't have lttng_statedump_process_state, this look lttng specific. I >> would have to look if there is a replacement event in perf. > Not that I am aware of. Perf tends to add fields to each records to keep > track of extra state. LTTng can also do that by dynamically attaching > context information, but it also supports dumping the initial system > state, thus allowing trace viewers to reconstruct the system state by > reading the trace, starting with the state dump events at the beginning. > >> I have no idea what we could do about the "unknown" events, say someone >> enbales skb tracing. But this is probably something for once we are >> done with the basic integration. >> >>> But if you could for example tell me the perf equivalents of all the >>> strings in that file, I could hack together such wrapper. With that, >>> in theory, perf traces should behave exactly the same as LTTng traces >>> in the viewer! > Ideally, the Trace Compass views should only care about a model of the OS. > Populating this model can be done by various "state gathering" plugins, > e.g. one for lttng, one for perf, which know about versioning and semantic > of the events contained in each trace. Exactly, the "wrapper" I was talking about previously would be something like an interface that only exposes the *concepts* present in the application, in this case the Linux kernel. It would then be up to the support of each tracer (or tracer version) to provide which events and fields to use for each of those concepts. Cheers! Alexandre > > [...] > >> For the fields, this is one event with alle the members we have. Please >> note that lttng saves the members with the _ prefix and I haven't seen >> that prefix in that .java file. The members of each event: > Yeah, the _ prefix for event names. This is one decision I would like to > find a way to revert, but we'll have to live with it unfortunately for > CTF 1.8. The issue it's trying to fix is to allow having fields named > "event" that don't clash with the "event" reserved keyword. When I added > the _ prefix, I did it like this in the CTF spec: > > "Replacing reserved keywords with underscore-prefixed field names is > recommended. Fields starting with an underscore should have their leading > underscore removed by the CTF trace readers." > > Unfortunately, this introduces semantic corner-cases for event names that > would indeed start with an underscore, unless they are prefixed with > double-underscore in the metadata. > > So far, the only fix I see to this situation is to eventually do a > CTF 1.9, and add the notion of a $ prefix to the grammar (which is not > part of the symbols accepted for an identifier) to be used as a field > name prefix that ensures there is no clash with reserved keywords. I'm > very open to suggestions there through, and I'm really not in a hurry > to release a new CTF spec version (we should only do so when we have > a batch of changes that are required, because it will require all trace > readers to be updated). > > Thanks! > > Mathieu > >>> Cheers, >>> Alexandre >> Sebastian >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-11-05 17:21 ` Mathieu Desnoyers 2014-11-06 4:53 ` Alexandre Montplaisir @ 2014-11-13 19:24 ` Sebastian Andrzej Siewior 2014-11-14 15:51 ` Mathieu Desnoyers 1 sibling, 1 reply; 11+ messages in thread From: Sebastian Andrzej Siewior @ 2014-11-13 19:24 UTC (permalink / raw) To: Mathieu Desnoyers Cc: Alexandre Montplaisir, Jiri Olsa, linux-kernel, Dominique Toupin, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo On 11/05/2014 06:21 PM, Mathieu Desnoyers wrote: > A very good example is the semantic of the sched_wakeup event. It has > changed due to scheduler code modification, and is now called from an > IPI context, which changes its semantic (not called from the same > PID). Unfortunately, there is little we can do besides checking the > kernel version to detect the semantic change from the trace viewer > side, because neither the event nor the field names have changed. > > The trace viewer could therefore care about the following information > to identify the semantic of a trace: > > - Tracer name (e.g. lttng or perf), > - Domain (e.g. kernel or userspace), > - Tracepoint versioning (e.g. kernel version for Perf). this sounds reasonable. That means for "domain" I switch to kernel from kernel-perf that I am using now. And then I need to add tracer_name. > In summary, for perf it would be really easy: just repeat the > kernel version in a new attribute attached to each event in the > metadata. For LTTng we would have the flexibility to have our own > version numbers in there. This would also cover the case of > userspace tracing, allowing each application to advertise their > tracepoint provider semantic changes through versioning. So what you are saying is that I need something like: event { name = "sched:sched_process_fork"; id = 1; stream_id = 0; => version = "3.16"; fields := struct { integer { … } perf_ip; integer { … } perf_tid; … } align(8); }; where the line marked "=>" is that one I should add. >>> Right now, we only define LTTng event and field names: >>> http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/internal/lttng2/kernel/core/LttngStrings.java >> >> Okay. So I found this file for linuxtools now let me try tracecompass. >> The basic renaming should do the job. Then I have to figure out how to >> compile this thingy… >> >> There is this one thing where you go for "tid" while perf says "pid". I >> guess I could figure that out once I have the rename done. > > LTTng uses the semantic presented to user-space to identify threads and > processes. What you find in /proc is what you find in a LTTng trace. The > tracepoint semantic used by perf and ftrace uses the kernel-internal > meaning of pid = thread ID, pgid = process ID, which differs from what is > visible from user-space. > > I guess it's up to you to decide if you want to stick to the kernel-internal > semantic, or switch to the user-visible (/proc) semantic for perf traces. I am happy if I can record and pass unchanged perf data :) >> We don't have lttng_statedump_process_state, this look lttng specific. I >> would have to look if there is a replacement event in perf. > > Not that I am aware of. Perf tends to add fields to each records to keep > track of extra state. LTTng can also do that by dynamically attaching > context information, but it also supports dumping the initial system > state, thus allowing trace viewers to reconstruct the system state by > reading the trace, starting with the state dump events at the beginning. I see. So if this is really a must-have for trace compass there would need to be a similar event added once we start perf. But from what I read in Alexandre's email it is not that tragic. >> For the fields, this is one event with alle the members we have. Please >> note that lttng saves the members with the _ prefix and I haven't seen >> that prefix in that .java file. The members of each event: > > Yeah, the _ prefix for event names. This is one decision I would like to > find a way to revert, but we'll have to live with it unfortunately for > CTF 1.8. The issue it's trying to fix is to allow having fields named > "event" that don't clash with the "event" reserved keyword. When I added > the _ prefix, I did it like this in the CTF spec: > > "Replacing reserved keywords with underscore-prefixed field names is > recommended. Fields starting with an underscore should have their leading > underscore removed by the CTF trace readers." > > Unfortunately, this introduces semantic corner-cases for event names that > would indeed start with an underscore, unless they are prefixed with > double-underscore in the metadata. > > So far, the only fix I see to this situation is to eventually do a > CTF 1.9, and add the notion of a $ prefix to the grammar (which is not > part of the symbols accepted for an identifier) to be used as a field > name prefix that ensures there is no clash with reserved keywords. I'm > very open to suggestions there through, and I'm really not in a hurry > to release a new CTF spec version (we should only do so when we have > a batch of changes that are required, because it will require all trace > readers to be updated). Aha. I haven't seen this underscore prefix in babeltrace examples so I wasn't aware for this. Thanks for explaining. Now should I add the prefix to perf by all means or is okay keep it as is? > Thanks! > > Mathieu > >>> Cheers, >>> Alexandre Sebastian ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [RFC 0/5] perf tools: Add perf data CTF conversion 2014-11-13 19:24 ` Sebastian Andrzej Siewior @ 2014-11-14 15:51 ` Mathieu Desnoyers 0 siblings, 0 replies; 11+ messages in thread From: Mathieu Desnoyers @ 2014-11-14 15:51 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Alexandre Montplaisir, Jiri Olsa, linux-kernel, Dominique Toupin, Tom Zanussi, Jeremie Galarneau, David Ahern, Arnaldo Carvalho de Melo ----- Original Message ----- > From: "Sebastian Andrzej Siewior" <bigeasy@linutronix.de> > To: "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com> > Cc: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>, "Jiri Olsa" <jolsa@redhat.com>, linux-kernel@vger.kernel.org, > "Dominique Toupin" <dominique.toupin@ericsson.com>, "Tom Zanussi" <tzanussi@gmail.com>, "Jeremie Galarneau" > <jgalar@efficios.com>, "David Ahern" <dsahern@gmail.com>, "Arnaldo Carvalho de Melo" <acme@redhat.com> > Sent: Thursday, November 13, 2014 2:24:51 PM > Subject: Re: [RFC 0/5] perf tools: Add perf data CTF conversion > > On 11/05/2014 06:21 PM, Mathieu Desnoyers wrote: > > A very good example is the semantic of the sched_wakeup event. It has > > changed due to scheduler code modification, and is now called from an > > IPI context, which changes its semantic (not called from the same > > PID). Unfortunately, there is little we can do besides checking the > > kernel version to detect the semantic change from the trace viewer > > side, because neither the event nor the field names have changed. > > > > The trace viewer could therefore care about the following information > > to identify the semantic of a trace: > > > > - Tracer name (e.g. lttng or perf), > > - Domain (e.g. kernel or userspace), > > - Tracepoint versioning (e.g. kernel version for Perf). > > this sounds reasonable. That means for "domain" I switch to kernel from > kernel-perf that I am using now. And then I need to add tracer_name. Yes, > > > In summary, for perf it would be really easy: just repeat the > > kernel version in a new attribute attached to each event in the > > metadata. For LTTng we would have the flexibility to have our own > > version numbers in there. This would also cover the case of > > userspace tracing, allowing each application to advertise their > > tracepoint provider semantic changes through versioning. > > So what you are saying is that I need something like: > > event { > name = "sched:sched_process_fork"; > id = 1; > stream_id = 0; > => version = "3.16"; > fields := struct { > integer { … } perf_ip; > integer { … } perf_tid; > … > } align(8); > }; > > where the line marked "=>" is that one I should add. Typically we don't use strings for this. This makes it easier for trace analysis to check on version ranges. We should also define a clear semantic for what constitutes compatible versions. We do have an issue here through. We've had various cases in the past where commits that changed the event layout or semantic were backported to kernel stable versions (e.g. between a x.y.0 and x.y.1 kernel). There is also the question of distribution vendor kernels to consider, where some backport commits without kernel patchlevel increments. The more I look into this problem, the more I start thinking that we might want to add fields to TRACE_EVENT that specify the major/minor version of the event per se. If the content of existing event fields change, we bump the major number. If new fields are added to the event, but the semantic of all existing fields stay the same, we bump the minor number. This would make it really easy for trace viewers to track event semantic changes, without ending up with a mess of incompatible traces generated by kernels with same version but behaving differently due to stable kernels and distribution backports. > > >>> Right now, we only define LTTng event and field names: > >>> http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/internal/lttng2/kernel/core/LttngStrings.java > >> > >> Okay. So I found this file for linuxtools now let me try tracecompass. > >> The basic renaming should do the job. Then I have to figure out how to > >> compile this thingy… > >> > >> There is this one thing where you go for "tid" while perf says "pid". I > >> guess I could figure that out once I have the rename done. > > > > LTTng uses the semantic presented to user-space to identify threads and > > processes. What you find in /proc is what you find in a LTTng trace. The > > tracepoint semantic used by perf and ftrace uses the kernel-internal > > meaning of pid = thread ID, pgid = process ID, which differs from what is > > visible from user-space. > > > > I guess it's up to you to decide if you want to stick to the > > kernel-internal > > semantic, or switch to the user-visible (/proc) semantic for perf traces. > > I am happy if I can record and pass unchanged perf data :) > > >> We don't have lttng_statedump_process_state, this look lttng specific. I > >> would have to look if there is a replacement event in perf. > > > > Not that I am aware of. Perf tends to add fields to each records to keep > > track of extra state. LTTng can also do that by dynamically attaching > > context information, but it also supports dumping the initial system > > state, thus allowing trace viewers to reconstruct the system state by > > reading the trace, starting with the state dump events at the beginning. > > I see. So if this is really a must-have for trace compass there would > need to be a similar event added once we start perf. But from what I > read in Alexandre's email it is not that tragic. Indeed, trace compass should be able to deal with "missing" info. > > >> For the fields, this is one event with alle the members we have. Please > >> note that lttng saves the members with the _ prefix and I haven't seen > >> that prefix in that .java file. The members of each event: > > > > Yeah, the _ prefix for event names. This is one decision I would like to > > find a way to revert, but we'll have to live with it unfortunately for > > CTF 1.8. The issue it's trying to fix is to allow having fields named > > "event" that don't clash with the "event" reserved keyword. When I added > > the _ prefix, I did it like this in the CTF spec: > > > > "Replacing reserved keywords with underscore-prefixed field names is > > recommended. Fields starting with an underscore should have their leading > > underscore removed by the CTF trace readers." > > > > Unfortunately, this introduces semantic corner-cases for event names that > > would indeed start with an underscore, unless they are prefixed with > > double-underscore in the metadata. > > > > So far, the only fix I see to this situation is to eventually do a > > CTF 1.9, and add the notion of a $ prefix to the grammar (which is not > > part of the symbols accepted for an identifier) to be used as a field > > name prefix that ensures there is no clash with reserved keywords. I'm > > very open to suggestions there through, and I'm really not in a hurry > > to release a new CTF spec version (we should only do so when we have > > a batch of changes that are required, because it will require all trace > > readers to be updated). > > Aha. I haven't seen this underscore prefix in babeltrace examples so I > wasn't aware for this. Thanks for explaining. Now should I add the > prefix to perf by all means or is okay keep it as is? If you can eventually have field names such as "event", "trace", or such names that clash with existing keywords, then you should prefix at least those field names with underscore. In LTTng, we simply prefix every field name with underscore. Thanks, Mathieu > > > Thanks! > > > > Mathieu > > > >>> Cheers, > >>> Alexandre > > Sebastian > -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-11-14 15:51 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-06 12:13 [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa
2014-08-06 12:13 ` [PATCH 1/5] perf tools: Add feature check for libbabeltrace Jiri Olsa
2014-08-06 12:13 ` [PATCH 2/5] perf tools: Add new perf data command Jiri Olsa
2014-08-06 12:13 ` [PATCH 3/5] perf tools data: Add perf data to CTF convertion support Jiri Olsa
2014-08-06 12:13 ` [PATCH 4/5] perf tools data: Add a 'perf' prefix to the generic fields Jiri Olsa
2014-08-06 12:13 ` [PATCH 5/5] perf tools data: Add tracepoint events fields CTF convertion support Jiri Olsa
2014-08-19 15:22 ` [RFC 0/5] perf tools: Add perf data CTF conversion Jiri Olsa
[not found] <53F38C74.4030300@voxpopuli.im>
2014-08-20 9:28 ` FW: " Jiri Olsa
2014-08-20 19:14 ` Alexandre Montplaisir
2014-11-03 17:58 ` Sebastian Andrzej Siewior
2014-11-04 1:20 ` Alexandre Montplaisir
2014-11-05 12:50 ` Sebastian Andrzej Siewior
2014-11-05 17:21 ` Mathieu Desnoyers
2014-11-06 4:53 ` Alexandre Montplaisir
2014-11-13 19:24 ` Sebastian Andrzej Siewior
2014-11-14 15:51 ` Mathieu Desnoyers
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
Powered by JetHome