From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f197.google.com (mail-dy1-f197.google.com [74.125.82.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2709035838A for ; Sat, 26 Sep 2026 06:20:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.197 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790403656; cv=none; b=L+Q2+una+Rh99qlvSw1zzZ8uJ9HMKMfGEsoXQrTSYE3HGxj/3vb/V4JiRdTp4VGJyDKeNcV0YVNm2Zy+iYLHO9zRhGG+2OsztbksZSrY3+wbhwZIQncvXVkUhhfMuURUMqn4/hK9WAUPbuX5v0qDAIYzats4lhkdHairTzJmft4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790403656; c=relaxed/simple; bh=/yBtzBxMwxDVBY4qQNL3WJEhfcz/2fXfmWa9ndOYAqE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kT3OjvJWo6UXIhGOtYsVp2F8/yCDWT55bV2x8gjOYrAiqkGg+qfUa6rt2fPnHsos0O98OqPqyE7s2HRlfftvnH157gPVclJyw+1YzE5GwzjcBtKZu4woPVY6NRskxSNnkTqKY9K8bosnl+jgb+RkDNoNWtOQxRHXTCcjeXO5kI4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OZzsx+5A; arc=none smtp.client-ip=74.125.82.197 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--irogers.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OZzsx+5A" Received: by mail-dy1-f197.google.com with SMTP id 5a478bee46e88-34316295d86so649029eec.0 for ; Fri, 25 Sep 2026 23:20:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1790403652; x=1791008452; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9KKT+iNDmCFltB4QEX7b+8xc6cvEZlJ5n2PYdoZqOPo=; b=OZzsx+5AR216cg9Hs/41sMNearAFUuZfGSLha9kxIquJBhXgM40A3K4iQE1CjnDY0h 9nvlfsYPCAxPWfwSr4v8dgClgo0qZ/2jBh/NZu/xnzM7cnx74EHvBH0RDQ7ydp35QFc6 6mEoA8D9pnJW0dxKD3LMmUNqngSkCtKDRgfZC0BPPioNcumL8Yx4cBFNr/9vqYEms1+N 92GQZ1Vaemx3/KXQXUrtHwTezYLIWYfYkEHwxu+8tNx1QKn+0yjmmZuIF1iuNAN2NU+n S0cBk1x+HUKWif5DPtojivLT3biHEAuR5M06A81ki0L5VK2z1PL9Jw91C3y9VKzdXN+Y Nm3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790403652; x=1791008452; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9KKT+iNDmCFltB4QEX7b+8xc6cvEZlJ5n2PYdoZqOPo=; b=Td+3AFd+75tS98wBxLT2fvyO/+SCyNN/EjNwoHXoTg/2NRHWQ49ddvu66AQOVJdPdt HbnQqitxnmBTll2ccQoj3+2dpYOOpHt0NSkWBoWsc3bl1cOKA0nt+6/RK6vByORzfaz4 TMIwfuAYABlDfgkAHgugkoYDXqgYyMzDY2MPSJytfOqwcabT2rbu656LTts8vKC+Oojy jL9jSqXGUtkdVWcZsZvsmCWXkoIZHQ9c35O0X62pDMXd9RxfJ3Yd3X0ScNvK0444YSkx kHRIkDytk9Zx5rBX4ufF6uBiJ4RUOKAmCPM7py0xBxf0ANFJjBjNu9/mat5kpFqq6lRM jftg== X-Forwarded-Encrypted: i=1; AKwUvBwKU6ud/3GD7nxl6M1l23eDrTetLyk6Im3/OL+0qqCdbOcfH6m56V/AqJdcX3tumLsxMbVZbdxy5n9LJ3A=@vger.kernel.org X-Gm-Message-State: AFuF++lVSFi2fmVgz+GHUxo3DBzIY5kxLYkQbDqp/y1y56TX8c1+FVld oBRvvZkY+XhhoFlGfE9iK6QRhj/iPvcyuqwxlL4Mk6MbIIiGv2Ug3CYGh7QDfOtEdAUYCGRUYbH McV5dIYJTxw== X-Received: from dlbpu5.prod.google.com ([2002:a05:7022:e885:b0:144:f4b8:ee5e]) (user=irogers job=prod-delivery.src-stubby-dispatcher) by 2002:a05:701b:420e:10b0:143:4450:e1e6 with SMTP id a92af1059eb24-146d039e6f9mr2063326c88.35.1790403651678; Fri, 25 Sep 2026 23:20:51 -0700 (PDT) Date: Fri, 25 Sep 2026 23:19:27 -0700 In-Reply-To: <20260923181213.3032038-1-irogers@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260923181213.3032038-1-irogers@google.com> X-Mailer: git-send-email 2.56.0.rc1.315.gc6ed9934b7-goog Message-ID: <20260926062029.800743-1-irogers@google.com> Subject: [PATCH v4 00/49] perf: Complete transition to standalone Python scripts From: Ian Rogers To: irogers@google.com, acme@kernel.org, alice.mei.rogers@gmail.com, james.clark@linaro.org, leo.yan@linux.dev, namhyung@kernel.org Cc: adrian.hunter@intel.com, dapeng1.mi@linux.intel.com, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, tmricht@linux.ibm.com Content-Type: text/plain; charset="UTF-8" The perf script command has historically supported running Python and Perl scripts by embedding libpython and libperl. That approach has several drawbacks: - high overhead from constructing Python dictionaries for every event (whether used by the script or not), - complex build dependencies on specific Python/Perl libraries, - threading and interpreter lifecycle complications from embedding the interpreter inside perf, - no unified model for standalone scripts such as ilist.py, treport.py, tracepoint.py, and twatch.py. Parts of this series were previously posted as part of the larger "perf: Reorganize scripting support" series: https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ with earlier RFC and mailing list discussion at: https://lore.kernel.org/linux-perf-users/20251029053413.355154-1-irogers@google.com/ https://lore.kernel.org/lkml/CAP-5=fWDqE8SYfOLZkg_0=4Ayx6E7O+h7uUp4NDeCFkiN4b7-w@mail.gmail.com/ >From that earlier series, the prerequisite patches (header dependency cleanups, evlist/evsel reference counting, initial perf.data and perf.session C extension wrappers, callchain/brstack/stat/syscall/config accessors, perf.pyi type stubs, LiveSession, and removal of embedded libperl support) have now been merged. We restarted the version number at v1 for the remaining patches, along with additional C extension capabilities, build-time static analysis (mypy and pylint), and review fixes. Running scripts as standalone Python applications using the perf Python C extension module provides a dramatic speedup. For example, with mem-phys-addr.py: Before (using embedded libpython in perf script): ``` $ perf mem record -a sleep 1 $ time perf script tools/perf/scripts/python/mem-phys-addr.py Event: cpu_core/mem-loads-aux/ Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m3.754s user 0m0.023s sys 0m0.018s ``` After (using standalone Python script with perf module): ``` $ PYTHONPATH=/tmp/perf/python time python3 tools/perf/python/mem-phys-addr.py Event: evsel(cpu_core/mem-loads-aux/) Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m0.106s user 0m0.021s sys 0m0.020s ``` The 49 patches in this series are organized into four groups: 1. Perf Python C Extension & Existing Script Updates (Patches 01-10): - Refine syscall_name()/syscall_id() argument parsing, expose perf.arch_strerrno() from the auto-generated trace beauty errno tables, and update callchain stubs and session.find_thread() lookup in tools/perf/util/python.c and tools/perf/python/perf.pyi. - Clean up pylint warnings and improve `perf script -l` descriptions in existing standalone scripts (ilist.py, treport.py, tracepoint.py, twatch.py). - Expose destination address location attributes (addr_dso, addr_symbol, addr_sym_offset), branch/transaction fields, sym_offset, context_switch callbacks, and Intel PT call_return / itrace synthesis directly in the perf Python extension (many parts previously posted in https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/). - Allow KeyboardInterrupt to propagate cleanly in LiveSession. 2. Build-Time Python Static Analysis (Patches 11-14): - Clean up mypy and pylint warnings in tools/perf/pmu-events/ and tools/perf/tests/shell/lib/. - Make MYPY and PYLINT build checks enabled by default when mypy and pylint (>= 2.16.0) are installed, with NO_MYPY=1 and NO_PYLINT=1 opt-out controls (matching NO_SHELLCHECK=1), so all Python scripts in tools/perf are continuously type-checked and linted at build time. 3. Porting Legacy Python & Perl Scripts to tools/perf/python/ (Patches 15-45): - Previously sent as part of https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ and updated with static type annotations, mypy/pylint cleanups, and accompanying shell tests under tools/perf/tests/shell/. Sashiko-driven review and improvement was undertaken while porting so that the ported scripts are in the best shape possible: * Temporarily install tools/perf/python/ scripts during the transition (Patch 15). * General & profiling scripts: stat-cpi.py, mem-phys-addr.py, stackcollapse.py, flamegraph.py, gecko.py, event_analyzing_sample.py (Patches 16-21). * Syscall & I/O scripts (ported from legacy Python and Perl): syscall-counts.py, syscall-counts-by-pid.py, failed-syscalls-by-pid.py, failed-syscalls.py, sctop.py, rw-by-file.py, rw-by-pid.py, rwtop.py (Patches 22-29). * Scheduling, locking, MM & networking scripts: futex-contention.py, task-analyzer.py, sched-migration.py (with SchedGui.py), wakeup-latency.py, compaction-times.py, net_dropmonitor.py, netdev-times.py, check-perf-trace.py (Patches 30-37). * Hardware trace & architecture-specific scripts: arm-cs-trace-disasm.py, powerpc-hcalls.py, intel-pt-events.py (with libxed.py), and migrating the Intel PT virtual LBR test to the Python API (Patches 38-41). * Database export & viewer utilities: export-to-sqlite.py, export-to-postgresql.py, exported-sql-viewer.py, and parallel-perf.py (Patches 42-45). 4. Removing Embedded libpython & Updating Build, CLI, and Docs (Patches 46-49): - Previously sent as part of https://lore.kernel.org/linux-perf-users/20260428071903.1886173-1-irogers@google.com/ : * Remove embedded libpython support (scripting-engines/trace-event-python.c) and delete the legacy tools/perf/scripts/python/ directory (Patch 46). * Replace the libpython feature check with test-python-module.c and update Python script installation paths (Patch 47). * Update `perf script` (builtin-script.c) to execute standalone scripts directly and remove embedded scripting engine support (Patch 48). * Update perf man pages (perf-script.txt, perf-script-python.txt, perf-check.txt) for standalone Python scripts (Patch 49). Changes in v4: - Rebase onto latest `perf-tools-next`, explicitly `#include ` for `UINT16_MAX` in `tools/perf/util/python.c`, and ensure `Event.ToPerfJson()` in `tools/perf/pmu-events/metric.py` and `IntelFpu()` in `intel_metrics.py` escape event modifier `=` and `,` characters (Patches 01, 11). - Set `symbol_conf.{vmlinux_name,kallsyms_name,symfs}` before calling `perf_session__new()` in `pyrf_session__init()` so `--kcore` directory loads in `perf_session__new()` do not overwrite a user-specified `vmlinux_name`, unconditionally attach `session->itrace_synth_opts` to `perf.session`, and honor `PERF_ITRACE` in `perf.session`, `arm-cs-trace-disasm.py`, `intel-pt-events.py`, and `perf script --itrace=...