[RFC PATCH v7 00/37] perf tools: filtering events using eBPF programs

[Wang Nan <wangnan0@huawei.com>]

This is the 7th version which tries to introduce eBPF programs to perf.
It enables 'perf record' to filter events using eBPF programs like:

 # perf record --event bpf-file.c sleep 1

and

 # perf record --event bpf-file.o sleep 1

This patch series is based on tip/perf/core (028c63b).

Compare with v6 patch, this series totally refactor code for llvm
compiling '.c' into BPF object file using LLVM. Including:

 1. A specific file tools/perf/util/llvm-utils.c is introduced for
    holding all llvm/clang related work, instead of putting them into
    bpf-loader.c.

 2. Automatically detect kernel include options by embedded shell
    script.

 3. Use command line template to generate compiler commands instead of
    assemble cmdline with printf, passing variables using environment.
    Which enable users to define their own compiler commands.

 4. Introduce '[llvm]' section to perf default config.
    5 options can be configured, but all of them can be omitted:

  [llvm]
        # Path to clang. If omit, search it from $PATH.
	clang-path = "/path/to/clang"

        # Cmdline template. Following line shows its default value.
        # Environment variable is used to passing options.
	clang-bpf-cmd-template = "$CLANG_EXEC $CLANG_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -working-directory $WORKING_DIR  -c $CLANG_SOURCE -target bpf -O2 -o -"

        # Option passed to clang, will be passed to cmdline by $CLANG_OPTIONS.
	clang-opt = "-Wno-unused-value -Wno-pointer-sign"

        # kbuild directory. If not set, use /lib/modules/`uname -r`/build.
        # If set to "" deliberately, skip kernel header auto-detector.
	kbuild-dir = "/path/to/kernel/build"

        # Options passed to 'make' when detecting kernel header options.
	kbuild-opts = "ARCH=x86_64"

    Command line options '--clang-path' and '--clang-opt' is appended to
    'perf record'.

 5. LLVM and BPF opening testcase is adding into 'perf test':

    # perf test  37
    37: Test LLVM searching and compiling                        : Ok

 6. Warning and error messages improvement.

 7. Issue clang command suit for newest clang which support
    '-target bpf', instead of calling
    'clang -emit-llvm | llc -march=bpf' pipe.

Other improvements including:

 1. Add ../lib/bpf into TAG_FOLDERS in tools/perf/Makefile.perf.
 2. Fix dependency problem: when NO_LIBBPF=1 is set don't compile
    libbpf.
 3. Suppress misleading messages printed during probing.
 4. In perf record, when bpf__probe failed, goto out_symbol_exit instead
    of directly return.

In this series:
  Patch 1/37 adds a feature check to check version of eBPF API.

  Patch 2/37 - 21/37 introduce libbpf, which first parse eBPF object
      files then load maps and programs into kernel. These patches are
      already received 'Acked-by' from Alexei Starovoitov except patch
      5/32, which enables opening a memory-based object file image using
      bpf_object__open_buffer().

  Patch 22/37 - 37/37 are perf related. Patch 23/37 - 27/37 are
      llvm-utils, which does most of new stuffs in this series.
      Patch 29/37 use functions defined previously to dynamically
      compile scriptlets.

  Patch 36/37 suppresses misleading messages printed during probing.

  Patch 37/37 adds new options to 'perf record'.

Wang Nan (37):
  tools build: Add feature check for eBPF API
  bpf tools: Introduce 'bpf' library to tools
  bpf tools: Allow caller to set printing function
  bpf tools: Open eBPF object file and do basic validation
  bpf tools: Read eBPF object from buffer
  bpf tools: Check endianess and make libbpf fail early
  bpf tools: Iterate over ELF sections to collect information
  bpf tools: Collect version and license from ELF sections
  bpf tools: Collect map definitions from 'maps' section
  bpf tools: Collect symbol table from SHT_SYMTAB section
  bpf tools: Collect eBPF programs from their own sections
  bpf tools: Collect relocation sections from SHT_REL sections
  bpf tools: Record map accessing instructions for each program
  bpf tools: Add bpf.c/h for common bpf operations
  bpf tools: Create eBPF maps defined in an object file
  bpf tools: Relocate eBPF programs
  bpf tools: Introduce bpf_load_program() to bpf.c
  bpf tools: Load eBPF programs in object files into kernel
  bpf tools: Introduce accessors for struct bpf_program
  bpf tools: Introduce accessors for struct bpf_object
  bpf tools: Link all bpf objects onto a list
  perf tools: Make perf depend on libbpf
  perf tools: Introduce llvm config options
  perf tools: Call clang to compile C source to object code
  perf tests: Add LLVM test for eBPF on-the-fly compiling
  perf tools: Auto detecting kernel build directory
  perf tools: Auto detecting kernel include options
  perf record: Enable passing bpf object file to --event
  perf record: Compile scriptlets if pass '.c' to --event
  perf tools: Parse probe points of eBPF programs during preparation
  perf probe: Attach trace_probe_event with perf_probe_event
  perf record: Probe at kprobe points
  perf record: Load all eBPF object into kernel
  perf tools: Add bpf_fd field to evsel and config it
  perf tools: Attach eBPF program to perf event
  perf tools: Suppress probing messages when probing by BPF loading
  perf record: Add clang options for compiling BPF scripts

 tools/build/Makefile.feature    |    6 +-
 tools/build/feature/Makefile    |    6 +-
 tools/build/feature/test-bpf.c  |   18 +
 tools/lib/bpf/.gitignore        |    2 +
 tools/lib/bpf/Build             |    1 +
 tools/lib/bpf/Makefile          |  195 ++++++++
 tools/lib/bpf/bpf.c             |   84 ++++
 tools/lib/bpf/bpf.h             |   23 +
 tools/lib/bpf/libbpf.c          | 1037 +++++++++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf.h          |   85 ++++
 tools/perf/MANIFEST             |    3 +
 tools/perf/Makefile.perf        |   19 +-
 tools/perf/builtin-probe.c      |    2 +-
 tools/perf/builtin-record.c     |   43 +-
 tools/perf/config/Makefile      |   19 +-
 tools/perf/tests/Build          |    1 +
 tools/perf/tests/builtin-test.c |    4 +
 tools/perf/tests/llvm.c         |   85 ++++
 tools/perf/tests/make           |    4 +-
 tools/perf/tests/tests.h        |    1 +
 tools/perf/util/Build           |    2 +
 tools/perf/util/bpf-loader.c    |  302 ++++++++++++
 tools/perf/util/bpf-loader.h    |   46 ++
 tools/perf/util/config.c        |    4 +
 tools/perf/util/debug.c         |    5 +
 tools/perf/util/debug.h         |    1 +
 tools/perf/util/evlist.c        |   51 ++
 tools/perf/util/evlist.h        |    1 +
 tools/perf/util/evsel.c         |   17 +
 tools/perf/util/evsel.h         |    1 +
 tools/perf/util/llvm-utils.c    |  370 ++++++++++++++
 tools/perf/util/llvm-utils.h    |   39 ++
 tools/perf/util/parse-events.c  |   16 +
 tools/perf/util/parse-events.h  |    2 +
 tools/perf/util/parse-events.l  |    6 +
 tools/perf/util/parse-events.y  |   29 +-
 tools/perf/util/probe-event.c   |   76 +--
 tools/perf/util/probe-event.h   |    6 +-
 38 files changed, 2567 insertions(+), 45 deletions(-)
 create mode 100644 tools/build/feature/test-bpf.c
 create mode 100644 tools/lib/bpf/.gitignore
 create mode 100644 tools/lib/bpf/Build
 create mode 100644 tools/lib/bpf/Makefile
 create mode 100644 tools/lib/bpf/bpf.c
 create mode 100644 tools/lib/bpf/bpf.h
 create mode 100644 tools/lib/bpf/libbpf.c
 create mode 100644 tools/lib/bpf/libbpf.h
 create mode 100644 tools/perf/tests/llvm.c
 create mode 100644 tools/perf/util/bpf-loader.c
 create mode 100644 tools/perf/util/bpf-loader.h
 create mode 100644 tools/perf/util/llvm-utils.c
 create mode 100644 tools/perf/util/llvm-utils.h

-- 
1.8.3.4